This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
(t, x(t)) 0 ( t , x , y , u ( t , x , y ) ) . 0, we
44
Chapter I
2 +
(1.2.43)
(sX
But if we use the usual differentiation rule, then we have
dip(t,x(t}} = hj£?>(*> : L
cr(t,x(t))^(t,x(t))dr,(t)
(1.2.44)
for the differential of a composite function (p (under the condition that x(t) satisfies Eq. (1.2.40) with usual differential dr)(t)).
The outlined computational difficulties disappear if we use the symmetrized form of stochastic integrals and equations. This has already been shown for integration when we calculated the integral (1.2.28). Let us show
that the usual formula of the composite function differentiation (1.2.44) holds for the stochastic process x ( t ) defined by the symmetrized differential equation (that is, by Eq. (1.2.38) with v = 1/2). The proof of this statement is indirect. Namely, we show that formula (1.2.44) for x(t), defined by the symmetrized stochastic equation
dx(t) = a(t, x ( t ) ) dt + a(t, x ( t ) ) d l / 2 r j ( t ) ,
(1.2.45)
implies formula (1.2.43) for x(t], defined by the Ito equation (1.2.40). Indeed, it follows from (1.2.41) that the symmetrized equation equivalent to (1.2.40) has the form
r 1 ^ ~i dx(t) = \a- -a—- \ dt + cr dl/2r](t)
I
2 axj
(the arguments of a and cr are omitted). From this relation and (1.2.44) we
obtain the symmetrized stochastic differential
*•<'"<<» = Now we note that (1.2.27) implies
*(*, z(t)) dor)(t) +
(t, x(t))a(t, x(t)) dt. (1.2.47)
By setting $ = crdtp/dx in (1.2.47), we obtain the Ito stochastic differential (1.2.43) from (1.2.46) and (1.2.47).
Synthesis Problems for Control Systems
45
Now let us consider another problem, which is extremely important from the viewpoint of applications. This is the question whether our mathematical model is adequate to the actual process in the dynamic system with
random perturbations. One of the starting points in the theory of optimal control is the assumption that the equation of motion of a dynamic sys-
tem is given a priori (§1.1). Suppose that the corresponding equation has the form (1.2.2) or (1.2.3). We have already shown that one can construct infinitely many solutions of such equations by choosing one or other form
of stochastic integrals and differentials. Which solution from these infinitely many ones corresponds to the actual stochastic process in the system? Does this solution exist? The answers can be obtained only by analyzing specific physical premises that lead to Eqs. (1.2.2), (1.2.3). Such investigations were performed in [167, 173, 175, 181], whose basic results relative to Eqs. (1.2.2), (1.2.3) we state without details.
If we consider the solution x ( t ) of Eq. (1.2.3) as a continuous model for a stochastic discrete time process Xk = s(fcA), k — 0 , 1 , 2 , . . . , which is computer simulated according to the formula
Zfc+i = xk + a(fcA, xk)A + (r(kA, xk)£k+l
(1.2.48)
(£fc > k — 1, 2 , . . . , is a sequence of independent identically distributed Gaussian random variables with zero mean and variance D£fc = A), then as A —>• 0 the sequence Xk (under the linear interpolation with respect to t between
the points t^ = kA) converges in probability to the solution x ( t ) of (1.2.3), provided that the latter is the Ito equation. If the motion of a dynamic system is given by (1.2.2) (stochastic equations of the form (1.2.2) are called Langevin equations [127]), where £(i) is a sufficiently wide-band stationary stochastic process (for example, the Gaussian Ornstein-Uhlenbeck process with the autocorrelation function
R^(T) = (a/2) exp{-a T\} for large values of a), then the solution of (1.2.2) coincides with the solution of the symmetrized equation (1.2.3), that is, of (1.2.38) with v = 1/2. In particular, each simulation of the Langevin equation (1.2.2) with an "actual" white noise by using analog computers gives
a symmetrized solution of Eq. (1.2.3) (see [37]). In the present monograph, all stochastic equations given in Langevin
form (1.2.2) with the white noise £(i) are understood in the symmetrized sense. In what follows, the symmetrized form of stochastic equations is used rather often, since this is the most convenient form for calculations relative to transformations of random functions, the change of variables, etc. In this connection, we omit the index i> = 1/2 in the stochastic differential.
The subscript 0 in the differential dor](t) in Ito equations is used if and only if the Ito equation and the corresponding symmetrized equation have
46
Chapter I
different solutions. In other cases, just as in symmetrized equations, we write stochastic differentials without subscripts. Stochastic integrals and differentials that correspond to other values of v [191, 192] are basically of theoretical interest. We shall not consider them in what follows.
In conclusion, let us consider some possible generalizations of the results obtained. First we note that all above-mentioned facts for scalar equations
(1.1.2), (1.2.3) can readily be generalized to the multidimensional case, so that the form of (1.2.2), (1.2.3) is preserved, provided that x 6 Rn and £(t) (»?(<)) are n-dimensional vector-columns of phase coordinates and random functions and the functions a and cr are an n-column and an n x n matrix. If necessary, the corresponding systems of equations can be written in more detail, for example, instead of (1.2.2), we can write
Xi = a i ( t , x ) + < T i j ( t , x ) £ j ( t ) ;
i=l,...,n
(1.2.49)
(as usual, the summation is taken over repeated indices if any, that is, in (1.2.49) we have cr^j = YTj=iaij^j}Systems (1.2.2) and (1.2.3) (or (1.2.49)) determine an n-dimensional Markov process x(t) with the vector of drift coefficients Ai(t, x) = ai(t, x) +
°
' X>
t = l,...,n,
(1.2.50)
and the matrix of diffusion coefficients
B(t,x) =
(1.2.51)
If the process x(t) is denned by the Ito equation (1.2.40), then, instead of (1.2.50) and (1.2.51), we have
A(t,x) = a(t,x),
B(t,x) = ( r ( t , x ) < T T ( t , x ) .
(1.2.52)
According to [173], stochastic equations of the more general form (1.2.1) can always be represented in the form (1.2.2). Indeed, as shown in [173], if
random functions £(t) in (1.2.1) have a small correlation time (for example, one can assume that £(<) is an n- vector of independent stochastic processes of Ornstein-Uhlenbeck type with a large parameter a), then Eq. (1.2.1) determines a Markov process with the following drift and diffusion coefficients:
,
(1.2.53)
Synthesis Problems for Control Systems j(t
+ T,x,t(t + T))]dT
47
(1.2.54)
(here K[a,/3] = E(a — Ea)(/3 — E/3) denotes the covariance of random variables a and /3; moreover, the mean Eg; and the correlation functions in (1.2.53) and (1.2.54) are calculated under the assumption that the argument x is a nonrandom fixed vector). Since similar characteristics of the Markov process defined by (1.2.2) (or by (1.2.49)) have the form (1.2.50), (1.2.51), we can obtain the differential equation (1.2.2), which is stochastically equivalent to (1.2.1), by solving system (1.2.50), (1.2.51) with respect to the unknown variables a,- and
. 1 daij a, - Ai - - ~-o-kj\ 2 dxk
. i-l,...,n.
It follows from the preceding that to study Markov processes of diffusion type, without loss of generality, we can consider stochastic equations only in the form (1.2.2.), (1.2.3) or (1.2.40). Therefore, the most general form of differential equations of motion of a controlled system with random perturbations £(t) of the white noise type is given by the equation
(1.2.55) or by the equivalent equation dx(t) - a(t, x(t),u(t)) dt + a(t, x(t), u(t)) drj(t)
(1.2.56)
(in (1.2.55) £(<) is the standard white noise with the characteristics (1.1.34); in (1.2.56) r,(t)= f £(T)dT,
Jo
77(0) = 0,
is the standard Wiener process). In (1.2.55) and (1.2.56) u = u(t) is understood as the control algorithm (1.1.2). The form of this algorithm can be found by solving the Bellman equation.
48
Chapter I §1.3. Deterministic control problems. Formal scheme of the dynamic programming approach
The dynamic programming approach [14] was proposed by R. Bellman
in the fifties as a method for solving a wide range of problems relative to processes of multistage choice. In this section we briefly discuss the main idea of this method applied to synthesis problems for optimal feedback control systems [16, 17]. We begin with deterministic problems of optimal
control and pay the main attention to the algorithm of the method, that is, to the method for constructing the optimal control in the synthesis form.
Let us consider the control problem with free right endpoint of the trajectory, in which the plant is given by system (1.1.5)
x(t)=g(t,x(t),u(t)),
0
z(0) = x 0 ,
(1.3.1)
the performance criterion is a functional of the form (1.1.11)
(1.3.2) and the control vector u may take values at each moment of time in a given
bounded set U C R r , u(t) e U.
(1.3.3)
In problem (1.3. !)-(!. 3. 3) the time interval [0,T] and the initial vector of phase variables XQ are known; it is required to find the control function u*(t): Q
represented in the form
u,(t) =
(1.3.4)
where the current values of the control vector are expressed in terms of the
current values of phase variables of system (1.3.1). The optimal control of the form (1.3.4) is called optimal control in the synthesis form, and formula (1.3.4) itself is often called the algorithm of optimal control. The dynamic programming approach allows us to obtain the optimal control in the synthesis form (1.3.4) for problem (1.3. 1)-(1. 3.3) as follows. We write
U
t
T
-I
C(T, x(r), U(T)) dr + i>(x(T)}
J
.
(1.3.5)
Synthesis Problems for Control Systems
49
The function F(t,xt), called later the loss function,6 plays an important role in the method of dynamic programming. This function is equal to the minimum value of the functional (1.3.2) provided that the control process is considered on the time interval [t,T], 0 < t < T, and the vector of phase variables is equal to x(i] = xt at the beginning of this interval (that is, at time t). In (1.3.5) the minimum is calculated over all possible strategies U(T) =
(a) these functions take values in an admissible set U] (b) for any t € [0,T] the Cauchy problem for system (1.3.1), x(r)=ff(r,z(T),y»(r,x(r))),
t < T < T,
x ( t ) = xt,
has a unique solution X(T) : t < T < T.
The dynamic programming method is based on the Bellman optimality principle [14, 17], which implies that the loss function (1.3.5) satisfies the basic functional equation r ft
F(t,xt)=
min
_
1
/ c(o-,x(o-),u(er)) da + F(t,x7) \,
n(tT)£U
Ut
(1.3.6)
J
t<<7<*
for all t <E [t,T]. For different statements of the optimality principle and comments see [1, 16, 50]. However, here we do not discuss these statements, since to derive Eq. (1.3.6) it suffices to have the definition of the loss function (1.3.5) and to understand that this is a function of time and of the state x(t) = xt of the controlled system (1.3.1) at time t (recall that the control process is terminated at a fixed time T). To derive Eq. (1.3.6), we write the integral in (1.3.5) as the sum Jt = It ~*~ Jf °f *wo integrals and write the minimum as the succession of minima
min = min u(r)6U t
min .
«(
Then we can write (1.3.5) as follows:
. . t<(T
U
t c(er, x(ff),
u(cr)} dcr
t
I c(p,X(p},u(p)}dp+^(x(T)]\. Jt \
(1.3.7)
The function (1.3.5) is also called a value function, a cost function, or the Bellman
function.
50
Chapter I
Since, by (1.3.1), the control u(p) on the interval [I, T) does not affect the solution x(cr} of (1.3.1) on the preceding interval [t,t), formula (1.3.7) takes the form F(t,xt) = min < / c(a;x(a-),u(cr)) da u(
( Jt
t<er
+ mm \ f
c(p,x(p),-u(p))dp+il>(x(T))\\.
u(p)€U Ut t
(1.3.8)
J )
Now, since by (1.3.5) the second term in the braces in (1.3.8) is the loss
function F(t,Xf),
we finally obtain Eq. (1.3.6) from (1.3.8).
The basic functional equation (1.3.6) of the dynamic programming approach naturally allows us to derive the differential equation for the loss
function F(t,x). To this end, in (1.3.6) we set t = t + A, where A > 0 is small, and obtain
F(t,xt) =
min «(
F /"*+A 1 / c(
1
Since the solutions x(t) of system (1.1.3) are continuous, the increments (E*+A —xt) of the phase vector are small for admissible controls u(t) =
F(t, x) is continuously differentiable with respect to all its arguments, we can expand the function F(t + A, K«+A) into its Taylor series about the point (t, Xt) as follows:
(1.3.10) In (1.3.10) dF/dx denotes an n-vector column with components dF/dxi, i— 1, 2, . . . , n; therefore, the third term on the right-hand side of (1.3.10) is the scalar product of the vector of increments (CC<+A — xt) and the gradient of the loss function
dF A. .dF xO —— = ^(Xit+^-Xit) —— . the function o(A) denotes the terms whose order is larger than that of the infinitesimal A. It follows from (1.3.1) that for small A the increment of the phase vector x can be written in the form -xt- g(t, xt, w«)A + o(A).
(1.3.11)
Synthesis Problems for Control Systems
51
Writing the first term in the square brackets in (1.3.9) as t+A
c(tr, x(a), u(cr)) da = c(t, xt, w«)A + o(A),
/.
(1.3.12)
substituting (1.3.10) and (1.3.12) into (1.3.9), and taking into account
(1.3.11), we arrive at T dF F(t, xt) = miu c(<, xt, u t )A + F(<, xt) + -^r(t, «t)A tf t G y LI
C7t
ftW
1
+ f f T (f, x ( , w t )-^-(i, a;t)A + o(A) . (1.3.13) ox j Note that only the first and the fourth terms on the right-hand side of (1.3.13) depend on the control ut. Therefore, the minimum is calculated
only over these terms, the other terms in the brackets can be ignored. Dividing (1.3.13) by A, passing to the limit as A —>• 0, and taking into account the fact that lirriA-KD o(A)/A = 0, we obtain the following differential equation for the loss function F(t, x):
at
,
, ,
,,)jr(t,x)
=0
(1.3.14)
(here we omit the subscript t of the phase vector Xt and the control Ut). Note that the loss function F(t, x) satisfies Eq. (3.1.14) on the entire interval of control 0 < t < T except at the endpoint t = T, where, in view of (1.3.5), the loss function satisfies the condition
F(T,x) = i/>(x).
(1.3.15)
The differential equation (1.3.14), called the Bellman equation, plays the central role in applications of the dynamic programming approach to the synthesis of feedback optimal control. The solution of the synthesis problem, that is, the optimal strategy or the control algorithm u* (t) = y>* (t, x) =
( p f ( t , x ( t ) ) can be found simultaneously with the solution of Eq. (1.3.14). Namely, suppose that we have somehow found the function F(t, x) that satisfies (1.3.14) and (1.3.15). Then the expression in the square brackets in (1.3.14) is a known function of i, x, and u. Calculating the minimum of this function with respect to u, we obtain the optimal control u* — y*(<, x) (it* determines the minimum point of this function in U C R r )If the functions c(t, x, u) and g(t, x, u) and the set of admissible controls
U allow us to minimize the function in the square brackets explicitly, then the optimal control can be written in the form
(1.3.16)
52
Chapter I
where dF/dx is a vector of partial derivatives yet unknown; when we minimize the function in the square brackets in (1.3.14), we assume that this vector is given. Using (1.3.16) and denoting
f , , T, , d F . , mm c(t, x, u) + g1 (t, x, u) ——(t, x) u£U \_
OX
dF
9F\\dF —)J — (1.3.17)
we write (1.3.14) without the symbol "min" as follows:
dF ( dF \ — ( t , x ) + 3>(t,x,—(t,x)}=Q, at \ ox J
Q
1.3.18
To complete the synthesis problem, it is necessary to solve (1.3.18) with regard to (1.3.15), that is, to find the function F(t, x) that satisfies (1.3.18) for 0 < t < T and continuously tends to a given function if>(x) as t —>• T,
and to substitute the function F(t, x) obtained into (1.3.16). In practice, the main difficulty in this synthesis procedure, is related to solving the Bellman equation (1.3.14) or (1.3.18), which is a first-order partial differential equation. The main distinguishing feature of the Bellman equation is that it is nonlinear because of the symbol "min" in (1.3.14), which shows that the function $ in (1.3.18) nonlinearly depends on the components of the vector of the partial derivatives dF/dx. The character of this nonlinearity is determined by the form of the functions c(i, x, u) and g(t, x, u), as well as by the set of admissible controls U. Let us consider some typical illustrative examples.
1°. Suppose that c(t,x,u) = ci(t,x) + uTP(t,x)u, where P is a symmetric r x r matrix positive definite for all x 6 R and t £ [0, T], g(t, x, u) — a(t, aj) + Q(t, x)u (a is an n-vector and Q is an n x r matrix), and the control u is unbounded (that is, U = R r ). Then the expression in the square brackets in (1.3.14) takes the form /) W
f) J?
[•} = Cl(t, x) + aT(t, x)^ + uTQT(t, x)——+ uTP(t, x)u.
(1.3.19)
By differentiating this function with respect to u and solving the system d[-]/du = 0, we obtain 1
/) F
^
(s3j
= --P~l(t,x)QT(t,x)--
(1.3.20)
Synthesis Problems for Control Systems
53
(the matrix P~l is the inverse of P). Substituting (1.3.20) into (1.3.19) instead of u, we obtain Q p\ ap t,x, —— J = c i ( t , a ; ) + a T ^ , x ) ——
(
1 f) W
3W
T - rt-,^rQ(t^}P-\t^}Q (t^}-^C*«K C/tC
(1-3-21)
2°. Suppose that c(i,«,w) = ci(t,x), g(t,x,u) = a(t,x) + Q(t,x)u, and the domain U is an r-dimensional parallelepiped, that is, |w,- < tioi, i = 1,..., r, where the numbers u0» > 0 are given. One can readily see that
dF t, x)——
( L3 - 22 )
9F
where sign A and \A\ are matrices obtained from A by replacing each its element a,-j by signojj and |ctjj|, respectively; {MOII • • • > U 0r} denotes the diagonal r x r matrix with itoi, • • •, «0r on its principal diagonal. 3°. Let the functions c(-) and g(-) be the same as in 2°; for the domain U, instead of a parallelepiped, we take an r-dimensional ball of radius RQ centered at the origin. Then, instead of (1.3.22), we obtain the following expressions for the functions tpo and 4>:
dX
——
"-'
1/2
(1.3.23)
Note that in (1.3.23) and in the following, dF/dxT denotes an n-row-vector with components dF/dxt, i = 1,..., n. Therefore, the function •jjrQQ'1'ffis a quadratic form in components of the gradient vector of the loss function, and the matrix QQT is its kernel. As a rule, the nonlinear character of Bellman equations does not allow one to solve these equations (and the synthesis problem) explicitly. There is only one exception, namely, the so-called linear-quadratic problems of optimal control (LQ-problems). In this case the differential equations (1.3.1) of the plant are linear:
x ( t ) = A(t)x(t) +B(t)u(t)
54
Chapter I
(here A(t) and B(t) are given nxn and nxr matrices), the penalty functions c(t, K, u) and i/'(x) m ^ne optimality criterion (1.3.2) are linear-quadratic forms of the phase variables z and controls u, and there are no restrictions
on the domain of admissible controls (that is, U — Rr in (1.3.3)). Let us solve the synthesis problem for the simplest one-dimensional LQproblem with constant coefficients; in this case, the solution of the Bellman equation and the optimal control can be obtained as finite analytic formulas. Suppose that the plant is described by the scalar differential equation
x = ax + bu,
(1.3.24)
and the optimality criterion has the form [•T
I(u) = clX2(T) + I
Jo
[cx2(t) + hu2(t)} dt
(1.3.25)
(ci > 0, c > 0, T > 0, h > 0, and a and b in (1.3.24) and (1.3.25) are given constant numbers). The Bellman equation (1.3.14) and the boundary condition (1.3.15) for problem (1.3.24), (1.3.25) have the form
9F f dF 1 —— (*,£)+min \cx*+ hu* + (ax + bu)——(t,x}\ =0, x
ot
"
" L
J
F(T,x) = Clx2.
(1.3.26)
(1.3.27)
The expression in the square brackets in (1.3.26) considered as a function of M is a quadratic trinomial. Since h > 0, this trinomial has the single minimum
which can readily be obtained from the relation d[-]/du = 0; this is a necessary condition for an extremum. Substituting M* into (1.3.26) instead of u and omitting the symbol "min", we rewrite the Bellman equation in the form
dF dt
,
dF dx
b2 IdF\2 =o, 4h\dxJ
—— + ex2 + ax _ _ _ _
0
~
v(1.3.29)
'
We shall seek the loss function F(t,x) satisfying Eq. (1.3.29) and the boundary condition (1.3.27) in the form
F(t, x) = p(t)x2,
(1.3.30)
Synthesis Problems for Control Systems
55
where p(t) is the desired function of time. If we substitute (1.3.30) into (1.3.29), then we see that p(t) must satisfy the ordinary differential equation
p+c+2ap-~p2 = 0 h
(1.3.31)
for 0 < t < T. Moreover, it follows from (1.3.27) and (1.3.30) that the function p(t] assumes a given value at the right endpoint of the control interval:
p(T) = ci.
(1.3.32)
Equation (1.3.31) can readily be integrated by separation of variables. The boundary condition (1.3.32) determines the unique solution of (1.3.31). Performing the necessary calculations, we obtain the following function p(i) that satisfies Eq. (1.3.31) and the boundary condition (1.3.32):
P(
'~
(J3- a)h - [62d - h(a
Thus it follows from (1.3.28) and (1.3.30) that the optimal control in the synthesis form for problem (1.3.24), (1.3.25) has the form
«, =
(1.3.34)
where p(t) is determined by (1.3.33). Note that problem (1.3.24), (1.3.25) is one of few optimal control problems, for which the Bellman equation can be solved exactly. In Chapter II we consider some other examples of exact solutions to synthesis problems
of optimal control (for deterministic and stochastic control systems). However, the majority of the optimal control problems cannot be solved exactly. In these cases, one usually employs approximate and numerical synthesis methods considered in Chapters III-VII. We complete this section with some remarks. First we note that we have considered only a formal scheme or, as is said sometimes, the algorithmic essence of the dynamic programming approach.
The described method for constructing an optimal control in the synthesis form (1.3.4) is justified by some assumptions, which are violated sometimes. We need to take into account the following. (1) The loss function F(x,t) determined by (1.3.5) is not always differentiable even if the penalty functions c(t,x,u) and i{>(x) are sufficiently
56
Chapter I
smooth (or even analytic) functions. It is well known that, by this reason, the dynamic programming approach cannot be used for solving many time-optimal control problems [50, 156]. (2) Even in the case where the loss function F(x,t) satisfies the Bell-
man equation (1.3.14), the control u*(i, x] minimizing the function in the square brackets in (1.3.14) may not be admissible. In particular, this control can violate the existence and uniqueness conditions for the solution of
the Cauchy problem for system (1.3.1). (3) The Bellman equation (1.3.14) (or (1.3.18)) with the boundary condition (1.3.15) can have nonunique solutions. Nevertheless, we have the following theorem [1].
THEOREM. Suppose that there exists a unique continuously differentiable solution Fo(t,x) of Eq. (1.3.14) with boundary condition (1.3.15) and there exists an admissible control u*(t,x] such that
[
Ft W 1 3F c(t,x,u)+gT(t,x,u)^-(t,x)\ = c(t,x,ut) + gT(t, x, ut)— —(t, x). ox J ox
Then the control w*(t, x) in the synthesis form is optimal, and the function Fo(t,x) coincides with the loss function (1.3.5).
In conclusion, we point out another fact relative to the dynamic programming approach. The matter is that this method can be used for solving problems of optimal control for which the optimal control ut(t, x) does not exist. For example, such situations appear when the domain of admissible controls U in (1.3.3) is an open set. The absence of an optimal control does not prevent us from deriving the
basic equations for the dynamic programming approach. It only suffices to modify the definition of the loss function (1.3.5). So, if we define the function F(t, xt) as the greatest lower bound of the functional in the square brackets in (1.3.5),
U
T
i
C(T, X(T),U(T)) dr + 1>(x(T))
J
,
(1.3.35)
t
then one can readily see that the function (1.3.35) satisfies the equations
c(a, x(a), u(
(1.3.36)
J
t<<7
3F
r
ft F
~\
ot
utu |_
ox
\
—— ( t , x ) + inf \c(t, x, u)+gT(t, x,u)-—(t, x)\ = 0 ,
(1.3.37)
Synthesis Problems for Control Systems
57
which are similar to Eqs. (1.3.6) and (1.3.14). However, in this case the functions u*(t, x) realizing the infimum of the function in the square brackets in (1.3.37) may not exist. Nevertheless, the absence of an optimal control u*(t,x) is of no fundamental importance in applications of the dynamic programming approach,
since if the lower bound in (1.3.37) is not attainable, one can always construct the so-called £-optimal strategy ue(t, x). If this strategy is used in system (1.3.1), then the performance functional (1.3.2) attains the value I(uc) = -F(0, Xo) + e, where e is a given positive number. Obviously, to construct an actual control system, it suffices to know the e-optimal strategy ue(t, x) for a small e. Here we do not describe methods for constructing £-optimal strategies. First, these methods are considered in detail in the literature (see, for example, [113, 137]). Second (and this is the main point), the optimal control always exists in all special problems studied in Chapters II-VII. This is the reason that, from the very beginning, in the definition of the loss function (1.3.5) we use the symbol "min" instead of a more general symbol "inf." §1.4. The Bellman equations for Markov controlled processes
The dynamic programming approach is widely used for solving stochastic problems of optimal control. In this section we consider the control
problems in which the controlled process is a Markov stochastic process. It follows from the definition of the Markov processes given in §1.1 that the probabilities of future states of a controlled system are completely determined by the current states of the vector of phase variables, which are assumed to be known at any time t.
FIG. 10 One can readily see that the servomechanism shown in Fig. 10 possesses the listed Markov properties if the following conditions are satisfied: (1) the joint vector (y(i),x(t)) of instant values that define the input actions and output variables is treated as the phase vector of the system;
58
Chapter I
(2) the input action y(t) is a Markov stochastic process; (3) the random perturbation £(i) is a white noise type process; (4) the controller C is a noninertial device that forms current values of the control actions u(t) according to the rule
u(t) = v(t,x(t),y(t}).
(1.4.1)
Actually, if the plant P is described by equations of the form (1.2.55)
and y(t) is a Markov process with known probability characteristics, then it follows from (1.2.55) and (1.4.1) that the joint vector x(t) = (x(i),y(t)) is a Markov process. In particular, if y(t) is a diffusion process with drift coef-
ficient A(t, y) and diffusion coefficient B(t, j/), then it follows from (1.2.39), (1.2.55), and (1.4.1) that the vector x(t) satisfies a system of stochastic differential equations of the form (1.2.2), that is, x(t) is a diffusion Markov process.
In this section we deal only with systems of the type shown in Fig. 10. In §1.5 we consider the possibilities of applying the dynamic programming approach in a more general situation with non-Markov controlled process (Fig. 3). Later we shall derive the Bellman equations for various stochastic problems of optimal control that are studied in Chapters II— VII. These problems
were stated in §1.1. 1.4.1. Base problem. Optimal tracking of a diffusion process. As the basic problem we consider the synthesis of the optimal ser-
vomechanism shown in Fig. 10 under the following conditions: (i) the controlled plant P is described by a system of stochastic differential equations of the form x(t)=a(t,x(t),u(t))+a(t,x(t})£(t),
z(0) = xo,
0 < * < T, (1.4.2)
where x G Rn is a vector of controlled output variables, u € Rr is a vector of control actions, ^(t) is the n-dimensional standard white noise
with characteristics (1.1.34), a and a are a given matrix and a vectorfunction, and the initial vector x(Q) = x$ and the time interval [0,T] are specified; (ii) the optimal control is sought in the form (1.4.1), the goal of control
is to minimize the functional
o
J
(1.4.3)
(iii) the restrictions on admissible controls have the form
«(*) G U,
(1.4.4)
Synthesis Problems for Control Systems
59
where U is a given bounded closed subset in the space R r ; (iv) the input stochastic process y(t) is independent of £ ( t ) and is an ra-dimensional diffusion Markov process with a known vector Ay(t,y) of drift coefficients and with matrix By(t, y) of diffusion coefficients;7 (v) there are no restrictions on the phase variables, that is, on the components of the vector x = (a;, y) e R n + m ; the current values of the components of the joint vector x can be measured precisely at any instant of time
te[o,T]. By analogy with (1.3.5) we determine the loss function F(t,xt,yt) for problem (i)-(v) as follows: F(t,xt,yt) = min E u(r)£U
r ,-T / C(X(T), j/(r),u(r)) dr Ut
t
+ i>(x(T),y(T))
x ( t ) = xt,y(t)=y^.
(1-4.5)
The loss function (1.4.5) for stochastic problem (i)-(v) differs from the
loss function (1.3.5) in the deterministic case by the additional operation of averaging the functional in the square brackets in (1.4.5). The averaging in (1.4.5) is performed over the set of sample paths xj = [X(T) : t < r < T],
y? = [y(r): t < T < T], that on the interval [t,T] satisfy the stochastic differential equations (1.4.2) and (*) (see the footnote) with initial conditions x(t) = Xt, y(t) = yt and control function u(r) = tf(r, x(r), y ( r ) j ,
t < T < T. Since the process X(T) — (z(r),j/(r)) is Markov, the result of averaging jF<,,(i, Xt) = F,f,(t, xt, yt) = £[•] in (1.4.5) is uniquely determined by the time moment t, by the state vector of the system xt — (xt, yt) at this moment, and by a chosen algorithm of control, that is, by the vector-function ip(-) in (1.4.1). Therefore, it turns out that the loss function (1.4.5) obtained by
minimizing Fv(t,xt) = F^t, xt, yt) over all admissible controls8 (that is, over all admissible vector-functions >(•)) depends only on time t and the state (xt,yt) of the servomechanism (Fig. 10) at this time moment. As was shown in §1.2, the coefficients Av(t,y) and Bv(t, y) uniquely determine the system of stochastic differential equations
y(t) = a»(t,y(t))+v»(t,y(t))r,(t),
(*)
whose solutions are sample paths of the Markov process y(t); in (*) rj(t] denotes the standard white noise (1.1.34) independent of £(*). 8 Just as in the deterministic case (§1.3) the control in the form (1.4.1) is called
admissible if (i) for all t 6 [0,T), x 6 R n , and y € R m , the vector-function
60
Chapter I
One can readily see that, for any t (E [t,T], the loss function (1.4.5) satisfies the equation
F(t,xt,yt) = mm E\ f c(x(T),y(T),u(T))dT+F(t,xT,W)\ u(r)£U
IJt
(1.4.6) J
t
that is a stochastic generalization of the functional equation (1.3.6). The
averaging in (1.4.6) is performed over the sample paths x\ and y*, and the symbol E[-] in (1.4.6) indicates the conditional expectation E^j 7, [•]. To prove (1.4.6), we write E^T j, 7 <•(•) for the conditional expectation T '^7 1^*'"*
of a functional of phase trajectories denoted by (•). Here we average over
all possible sample paths x'j = [X(T) : t < T < T], y'j = [y(r) :t
writing the integral in (1.4.5) as the sum Jt
= ft + fj
of two integrals
and writing the minimum as the succession of minima min = min t
min ,
t<<7
we can rewrite (1.4.5) as
r
min
min
v(
ft
T,s[[ / c(x(
(1.4.8) It follows from (1.4.1) and (1.4.2) that the controls u(p) on the time interval t < p < T do not affect the stochastic process x(
interval t <
a
t
-> ~t c(x(o-),y(
J t
Synthesis Problems for Control Systems
61
we can rewrite (1.4.8) in the form
F(t,xt,yt)=
min E 7 7,,. u(
i>ytt
U(
| / c(x(a),y(cr),u(cr))
dff
(. Jt
+ min E s T i y T | s T y 7[jr c ( x ( p ) , y ( p ) , u ( p ) ) d p + i > ( x ( T ) , y ( T ) ) ] ^ . t
Since the process (z(£), y(t)} is Markov, the result of averaging in the second term in the braces in (1.4.9) depends only on the terminal state of a fixed sample path (xj, y\}. Thus, replacing E^T^T^T,,! by E^T !,,._,,_ and taking into account the fact that, by (1.4.5), the second term in (1.4.9) is the loss function F ( t , x ^ , i f f ) , we finally obtain the functional equation (1.4.6) from
(1.4.9). Just as in the deterministic case, the functional equation (1.4.6) allows us to obtain a differential equation for the loss function F(t,x,y). By setting
t = t + A, we rewrite (1.4.6) in the form r yt+A
F(t,xt,yt)=
min
E
/
C(X(T), y(r), U(T)) dr
.
(1.4.10)
Assuming that A > 0 is small and the penalty function c(x,y,u) is continuous in its arguments and having in mind that the diffusion processes X(T) and J/(T) are continuous, we can represent the first term in the square brackets in (1.4.10) as
c(x(r), 2/(r), w(r)) dr = c(z t , j/t, ut) A + o(A),
(1.4.11)
where, as usual, the function o(A) denotes infinitesimals of higher order than that of A. Now we assume that the loss function F(t,x,y) has continuous derivatives with respect to t and continuous second-order derivatives with respect to phase variables x and y. Then for small A we can expand the function
62
Chapter I
F(t + A, Xt+Ai Vt+A.) in the Taylor series
= -F^, xt, yt) + A
*9 F
f)F
x T
«+ -yt) + 2
+ o(A) + o(|i« +A - it 2 ) + o(|y t+A - yt\2).
(1.4.12)
Here all derivatives of the loss function are calculated at the point (t, xt, yt),
as usual, dF/dx and dF/dy denote the n- and m-column- vectors of partial derivatives of the loss function with respect to the components of the vectors x and y, respectively; d2F/dxdxT , d2 F / dxdyT , and d2F/dydyT denote the n x n, n x m, and m x ra matrices of second derivatives.
To obtain the desired differential equation for F(t,x,y), we substitute (1.4.11) and (1.4.12) into (1.4.10), average, and pass to the limit as A —>• 0. Note that if we average expressions containing the random increments
(xt+A. — %t) and (y«+A — y t ) , then all derivatives of F in (1.4.12) are considered as constants, since they depend on (t,xt,yt) and the mathematical expectation in (1.4.10) is calculated under the assumption that the values of Xt and yt are known and fixed. The mean values of the increments (xt+& — Xt) can be calculated by integrating Eqs. (1.4.2). However, we can avoid this calculation if we use the results discussed in §1.2. Indeed, if just as in (1.4.11) we assume that the control U(T) is fixed and constant, U(T) = ut, then we see that for t < T < t + A, Eq. (1.4.2) determines a Markov process X(T) such that we can write (see (1.1.54))
E(a; t+ A - x t ) = Ax(t, xt, « t )A + o(A),
(1.4.13)
where Ax(t, xt,Ut) is the vector of drift coefficients of this process. But since (for a fixed u(t) = ut) Eq. (1.4.2) is similar to (1.2.2), it follows from (1.2.50) that the components of this vector have the form9 I
\
< i j . X t
=ai(t,xt,ut) + - ——J-^ ——-a-kj(t,xt), (1.4.14) 9 Recall that formula (1.4.14) holds for the symmetrized stochastic differential equation (1.4.2). But if (1.4.2) is an Ito equation, then we have Ax(t, xt,ut) = a(t,xt,ut) instead of (1.4.14).
Synthesis Problems for Control Systems
63
In a similar way, (1.4.2), (1.1.50) and (1.2.52) imply
E(z t+A - xt)(xt+A - xt)T = Bx(t, x f )A + o(A),
(1.4.15)
where
Bx(t,xt}=a(t,xt)aT(t,xt).
(1.4.16)
The other mean values in (1.4.12) can be expressed in terms of the input Markov process y(t) as follows:
E(J/«+A - yt) = A»(t, yt) A + o(A), (1.4.17) A
°( )-
(1.4.18)
Finally, since the stochastic processes y(t] and £(i) are independent, we have
- Vt)T = o(A).
(1.4.19)
Taking into account (1.4.13)-(1.4.19), we substitute (1.4.11) and (1.4.12) into (1.4.10) and rewrite the resulting expression as follows:
F(t,xt,yt) = min < F(t, xt, yt) + A \c(xt, yt, ut) + -=-
vt£U {
\_
at
For brevity, in (1.4.20) we omit the arguments (t,xt,yt) of all partial derivatives of F and denote the trace of the matrix A — |[oij||™ by Sp^4 = on + a 2 2 + • •• + o nn . By analogy with Eq. (1.3.14), we divide (1.4.20) by A, pass to the limit as A —>• 0, and obtain the following Bellman differential equation for the
loss function F = F(t,x,y):
+ mm\c(x,y,u)+(Ax(t,x,u)f~]
=0.
(1.4.21)
By analogy with (1.3.14), we omit the subscripts of xt, yt, and ut, assuming that the phase variables x, y and the control vector u in (1.4.21) are taken
64
Chapter I
at the current time t. We also note that the loss function F — F(t, x, y) must satisfy Eq. (1.4.21) for 0 < t < T. At the right endpoint of the control interval, this function must satisfy the condition
,y) = i>(x,y),
(1.4.22)
which readily follows from its definition (1.4.5). By using the operator
,
2L
(1.4.23)
oxax1
we can rewrite (1.4.21) in the compact form
mm [Ll,iyF(t, x, y) + c(x, y, u)} = 0.
(1.4.24)
In the theory of Markov processes [45, 157, 175], the operator (1.4.23) is called an infinitesimal operator of the diffusion Markov process x(t) = (*(*)),»(*))•
To obtain the optimal control in the synthesis form ut =
If it is possible to calculate the minimum of the function in the square brackets in (1.4.21) explicitly, then the optimal control can be written as
follows (see §1.3, (1.3.16)-(1.3.18)): u* =
(1.4.25)
and the Bellman equation (1.4.21) can be written without the symbol "min"
Sp
*'<*'»>
+
*'*'*
= °'
(1A26)
where $ denotes a nonlinear function of components of the vector dF/dx 9F
-\=c(x -u *n(t T v —— \\
>x>y>%:)}\ Tir-
(i-4-27)
Synthesis Problems for Control Systems
65
In this case, solving the synthesis problem is equivalent to solving (1.4.26) with the additional condition (1.4.22). After the loss function F(t,x,y) satisfying (1.4.26) and (1.4.22) is found, we can calculate the gradient
dF(t, x, y)/dx = u(t, x, y) and obtain the desired optimal control
u* —
(1.4.28)
Obviously, the main difficulty in this approach to the synthesis problem is to solve Eq. (1.4.26). Comparing this equation with a similar equation (1.3.18) for the deterministic problem (1.3.!)-(!.3.3), we see that, in contrast with (1.3.18), Eq. (1.4.26) is a second-order partial differential equation of parabolic type. By analogy with (1.3.18), Eq. (1.4.26) is nonlinear, but, in contrast with the deterministic case, the nonlinearity of Eq. (1.4.26)
is weak, since (1.4.26) is linear with respect to the higher-order derivatives of the loss function. This is why, in the general theory of parabolic equations [61, 124], equations of type (1.4.26) are usually called quasilinear or semilinear. In the general theory [124] of quasilinear parabolic equations of type
(1.4.26), the existence and uniqueness theorems for their solutions are proved for some classes of nonlinear functions $. The unique solution
of (1.4.26) is selected by initial and boundary conditions on the function F(t,x,y). In our case, condition (1.4.22) that determines the loss function
for t = T plays the role of the "initial" condition. The boundary conditions are determined by the restrictions imposed on the phase variables * and y in the original statement of the synthesis problem. If, as in problem (i)-(v) considered here, there are no restrictions on the phase variables, then it is necessary to solve the Cauchy problem for (1.4.26). In this case, the uniqueness of the solution is ensured by some requirements on the rate of growth of the function F(t, x, y) as x , \y\ —> oo (for details see Chapter III). However, there are no general methods for solving equations of type
(1.4.26) explicitly. Nevertheless, in some specific cases, Eq. (1.4.26) can be solved approximately or numerically, and sometimes, exactly. We describe such special cases in detail in Chapters II-VII. Now let us consider some modifications of problem (i)-(v) that we shall study later. First of all, we trace how the form of the Bellman equation (1.4.21) varies if, in the initial problem (i)-(v), we use optimality criteria that differ from (1.4.3). 1.4.2.
Stationary tracking. We begin by modifying the criterion
(1.4.3), which allows us to examine stationary operating conditions of the servomechanism shown in Fig. 10. We assume that criterion (1.4.3) does not penalize the terminal state
of the controlled system, that is, the penalty function ij}(x,y) = 0 in the
Chapter I
66
functional (1.4.3). Then the servomechanism shown in Fig. 10 can operate in the time-invariant (stationary) tracking mode if the following conditions are satisfied: (1) the input Markov process y(t) is homogeneous in time, namely, its drift and diffusion coefficients are independent of time: Ay(t, y] = Ay (y)
(2) the plant is autonomous, that is, the right-hand sides of Eqs. (1.4.2) do not depend on time explicitly, a(t, x, u) = a(x, u) and cr(t, x) = ff(x)',
3) the system works sufficiently long (the upper integration limit T —>• oo in (1.4.3)).
z(t)
= y(t] - x(t)
FIG.
11
A process of relaxation to the stationary operating conditions is schematically shown in Fig. 11, where the error z(i) = y ( t ) — x(t) between the input action (the command signal) and the controlled value (x and y are scalar variables) is plotted on the ordinate axis. One can see that for large T the operation interval [0, T] can be conventionally divided into two intervals: the time-varying operation interval [0, ti, this correlation disappears, and we can assume that z(t), t 6 [
Synthesis Problems for Control Systems
67
The performance on the time-invariant interval is characterized by the value 7 of mean losses per unit time (the stationary tracking error). If the operation time T increases to T + AT (see Fig. 11), then the loss function (1.4.5) increases by -yAT. Therefore, to study the stationary tracking, it
is expedient, instead of the loss function (1.4.5), to use the loss function f(x, y) that is independent of time and can be written as
f(x,y)=
lim [F(t,x,y)-i(T-t)].
T—*-oo
(1.4.29)
It follows from (1.4.23) and (1.4.24) that function (1.4.29) satisfies the stationary Bellman equation
mm [L« y /(z, y) + c(x, y, u)] = 7,
(1.4.30)
where L™^y denotes the elliptic operator
Obviously, for the optimal control ut =
7=
c(x,y,tpt(x,y))p00(x,y)dxdy
(1.4.32)
and, together with the functions f(x,y) and u* = ipt(x,y), can be found by solving the time-invariant equation (1.4.30). Some methods for solving
the stationary Bellman equations are considered in Chapters II- VI. 1.4.3. Maximization of the mean time of the first passage to the boundary. As previously, we assume that in the servomechanism shown in Fig. 10 the stochastic process y(t) is homogeneous in time and the plant P is autonomous. We also assume that a simply connected closed domain
D C R-n+m is chosen in the (n + ra)-dimensional Euclidean space R n +m of vectors (x,y). It is required to find a control that, for any initial state
(z(0),j/(0)) G D of the system, maximizes the mean time Er during which the representative point (x(t),y(i)') achieves the boundary dD of the domain D (see the criterion (1.1.21) in §1.1). By Wu(t — to, XQ, j/o) we denote the probability of the event that the rep-
resentative point (x, y) does not reach dD during time t— to ifx(to) = XQ and
68
Chapter I
y(to) = 3/0, (£0,2/0) G D, and a control algorithm u(t) — < p ( x ( t ) , y ( t ) ) is chosen. This definition implies the following properties of the function Ww:
if (xo,yo) is an interior point of D;
W«(i-*o,zo,l/b) = 0,
V*>t0,
(1.4.33)
if (s0, 2/0) € 3D.
If t r denotes a random instant of time at which the phase vector x ( t ) = (x(t),2/(f)) comes to the boundary dD for the first time, then the time T = tr — to of coming to the boundary is a random variable and the function
W"(-) can be expressed via the conditional probability
Wu(t~to,x0,yo) = Pu{r>t-t0 \x(t0) = x0,y(t0) = 2/0} = Pn{T>t-t0\x0,y0}.
(1.4.34)
For the mutually disjoint events {T < t—to} and {T > t— to}, the probability addition theorem implies
Pu{r
(1.4.35)
Expressing the distribution function of the probabilities PW{T < t — to \ ^0,2/0} via the probability density wT(a] of the continuous random variable T, we obtain ft — to
P{r
x0,y0} = I Jo
wT(a)du — 1 - Wu(t - t0,x0, t/0)
from (1.4.34) and (1.4.35). Hence, after the differentiation with respect to t, we have
dWu wr(t-t0) = ————(t-to,x0,yo).
(1.4.36)
Using the same notation for the argument of the density and for the random
value, that is, writing wT(t — to) = w(r), from (1.4.33) and (1.4.36) we obtain the mean time ET of achieving the boundary /•OO
=
Jo
,-00
Tw(r)dT = -
Jo
^yyu
T~—(T,x0,y0)dT
dr
/.OO
= / Jo
|»OO
Wu(T,x0,y0)dT=
Wn(t-t0,x0,yoo)dt. )dt.
Jt
°
This formula holds if lim^oo^ - t0)Wu(t - t0, x 0 , yo) = 0.
(1.4.37)
Synthesis Problems for Control Systems
69
The mean time ET depends both on the initial state (XQ, J/Q) of the controlled system shown in Fig. 10 and on a chosen control algorithm. u = f>(x, y). Therefore, the Bellman function for the problem considered is determined by the relation ,.00
Fi(x,y) = max /
WU(T - t, x, y) dr.
(1.4.38)
u(r)£U Jt T>t
By analogy with (1.4.10), for the function (1.4.38), the basic functional equation of the dynamic programming approach has the form
Fi(xt,yt) =
max
r ,t+A -i / WU(T -t,xt,yt)dT +EFi(xt+&.,yt+&)\.
u(r)£U Ut t
J
(1.4.39) The Bellman differential equation for the function Fi(x,y) can be derived from (1.4.39) by passing to the limit as A —>• 0. In this case, the procedure is almost the same as that used for the derivation of Eq. (1.4.21)
for the basic problem (i)-(v). Expanding Fi(xt+&., J/J+A) in the Taylor series around the point (xt,yt), averaging the expansion with respect to the random increments (XJ+A — %t) and (yt+A — yt), taking into account the relation lirriA-+o W"(A, Xt, yt) = I for all (xt,yt) lying in the interior of D, and passing to the limit as A —)• 0, from (1.4.39) with regard to (1.4.13)-(1.4.19), we obtain the Bellman differential equation for the function F-i(x,y): F^x.y)^-!,
(1.4.40)
where the elliptic operator L">y is given by (1.4.31).
We also note that the function Fi(x,y) satisfies Eq. (1.4.40) in the interior of the domain D. It follows from (1.4.33) and (1.4.38) that at the points of the boundary dD the function F\ vanishes,
= 0.
(1.4.41)
[x,y)edD
In the theory of differential equations of elliptic type, the problem of solving Eq. (1.4.40) with the boundary condition (1.4.41) is called the first
interior boundary-value problem or the Dirichlet problem. Thus, solving the synthesis problem for the optimal control that maximizes the mean time of the first passage to the boundary is equivalent to solving the Dirichlet
problem for the semilinear elliptic equation (1.4.40).
70
Chapter I
1.4.4. Minimization of the maximum penalty. Now let us consider the synthesis problem with optimality criterion (1.1.18) for the optimal control system shown in Fig. 10. In this case, it is reasonable to introduce the loss function
F2(t,xt,yt)=
min E max r T C(Z(T), t/(r), W(T)) .
U(T)
L*> >
J
(1.4.42)
t
In (1.4.42) the averaging has the meaning of the conditional mathematical expectation £[•] = £{[•] x(t) = xt,y(t) = yt}. For small A we have the following basic functional relation for the function F2'.
F2(t,xt,yt)
=
min
{max.[c(xt,yt,ut) + o(A),EF2(t + A,
Let us introduce the notation c°(x,y) = min^gjy c(x, y, u). Then it follows from (1.4.43) that either
F2(t,xt,yt) = c°(xt,yt),
(1.4.44)
or
F2(t,xt,yt) = lim
min
A
EF2(t + A, z t + A , j/ t+A ),
(1.4.45)
-*-° u(r)£U t
provided that the function F2(t,X(,yt) > c°(xt,yt) has been obtained from
(1.4.45). Acting by analogy with Section 1.4.1, that is, expanding the function .£2(2 + A, £ ( _|_ A ,y t + A ) in the series (1.4.12), averaging, and passing to the limit as A ->• 0, from (1.4.44) and (1.4.45) we obtain (with regard to (1.4.13)-(1.4.19)) the Bellman equation in the differential form:
f mmLl F2(t,x,y) = Q I u£U ( F2(t, x, y) = c°(x, y)
if F2(t, x, y) > c°(x, y ) , (1.4.46) otherwise,
where Lf^^y denotes the operator (1.4.23). The unique solvability of (1.4.46) implies the condition
F2(T,x,y)=c°(x,y),
(1.4.47)
Synthesis Problems for Control Systems
71
as well as the matching conditions for the function ^2(^1 x, y) on the interface between the domains on which the equations in (1.4.46) are defined. These conditions of "smooth matching" [113] require the continuity of the function F2(t,x,y) and of its first-order derivatives with respect to the
phase variables x and y on the interface mentioned above. If, by analogy with Sections 1.4.2 and §1.4.3, the characteristics of the input process y(t) and of the controlled plant P are time-independent, then it is often expedient to use a somewhat different statement of the problem considered, which allows us to assume that the loss function is independent
of time. In this case, we do not fix the observation time but assume that the optimal system minimizes the functional I[u] = E maxc(x(r), y(r), u(T))e-^T-t},
(1.4.48)
where (3 > 0 is a given number. This change of the mathematical statement preserves all characteristic features of the problem. Indeed, it follows from
(1.4.48) that the time of observation of the function c(x,
y, u) is bounded
and determined by /?. Namely, this time is large for small /3 and small for large (3. For the criterion (1.4.48) the loss function is determined by the formula10
h(x,y) = min E [maxc(a;(T), 2/(r),-u(T)) e -^ ( r - < ) j . «(r)eu L r > * J
(1.4.49)
T>t
Taking into account the relations min E
max c(a;(r), J/(r), u(
u(r)£U
r>t + A
= e-/3A min E max C(X(T), «(r)€l7
r>t+A
we can rewrite Eq. (1.4.43) for the function f z ( x , y ) in the form h(xt,yt) =
min < max[c(a3 t ,J/t,Wt) +o(A), Ef2(xt+/\,yt+^)e~/3^] «(r)eu I
>. J
(1.4.50) 10
As usual £[•] in (1.4.49) is treated as the conditional mathematical expectation
72
Chapter I
By analogy with the previous reasoning, from. (1.4.50) we obtain the Bellman equation for the function /2(x, y)'
( mini" h(x,y) = ( 3 f 2 ( x , y ) I u&u I h(x,y) = c°(x,y)
if f?(x,y) > c°(x,y),
(1.4.51)
otherwise,
where i" is the elliptic operator (1.4.31) that does not contain the derivative with respect to time t. In §2.2 of Chapter II, we solve Eq. (1.4.51) for a special problem of optimal control. 1.4.5. Optimal tracking of a strictly discontinuous Markov process. Let us consider a version of the synthesis problem for the optimal
tracking system that differs from the basic problem (i)-(v) by conditions (i) and (iv). Namely, we assume that (i) the input process y(t) in the servomechanism (see Fig. 10) is given by a strictly discontinuous Markov process (see §1.1) with known characteristics X ( t , y ) and T r ( y , z , t ) determining the intensity of jumps and the density of the transition probability at the
state (t, y) and that (ii) there are no random perturbations £(t) that act on the plant P. In this case, the plant P is described by the system of
ordinary (nonstochastic) differential equations
x(t) = a(t,x(t},u(t)},
x(Q) = xQ,
0
(1.4.52)
It follows from (1.1.68) that for small A the transition probability p(t, yt', t-\-
A,2/ t + A) = p(y(t + A) — y t+ A | y ( t ) = yt) for the input process y(t) is determined by the formula
-yt) AA(t, j/ t )T(*, yt, y < + A ) + o(A).
(1.4.53)
By analogy with the solution of the basic problem, in the case considered, the loss function Fz(t,x,y) is determined by (1.4.5) if £[•] in (1.4.5) is understood as the averaging of the functional [•] over the set of sample paths yf = [y(r) : t < T < T] issued from a given initial point y(t) = yt.
Obviously, Fs(i, x,y) satisfies the functional equations (1.4.6) and (1.4.10). We rewrite Eq. (1.4.10) for F3 as follows:
, «
U
t+A
c(x(T),y(T),u(T))dT
•
(1-4.54)
Synthesis Problems for Control Systems
73
Note that for small A we can explicitly average (1.4.54) by integrating the function in the square brackets multiplied by the transition probability (1.4.53). Since the sample paths of the input process y(t) are discontinuous, the random increments (y*+A ~ Vt) are> generally speaking, not small. Therefore, in our case, instead of (1.4.12), we use the following representation of
F3(t + A, a; t+ A, 2/t+A) as A ->• 0:
dF3 F3(t + A, zt+A, yt+A) = Fsit, xt, yt+&) + A - — (i, xt, o(A)
(1.4.55)
(in (1.4.55) it is assumed that -Fs(<, z,j/) is a continuously differentiable function with respect to t and z). The Bellman equation for F3(t, x, y) can be derived from (1.4.54) in the
standard way. To this end, we substitute expansion (1.4.55) into (1.4.54), average with probability density (1.4.53), and pass to the limit as A —>• 0 in (1.4.54). Using (1.4.53), we obtain
EF3(t,xt,yt+£,) = I F3(t,xt,yt+&)[(l- A\(t,yt))8(yt+& - yt)
+ AA(<, yt)x(t, yt, J/*+A)] dyt+& + o(A) = F3(t, xt, yt) + A
/ F3(t, xt, z)ir(t, yt, z] dz
- F3(t, xt, &)] A(*, yt) + o(A).
(1.4.56)
In a similar way, it follows from (1.4.52) and (1.4.53) that -xt = a(t, xt, ut)A + o(A),
dF* -^(i,i t ,j/ t )+0(A) ) 5F-,
-^(t, s t , yt) + 0(A),
(1.4.57)
(1.4.58) (1.4.59)
/•«+A
E / 1/4
C(X(T), y ( r ) , u ( T ) ) dr = c(xt, yt, w«)A + o(A) (1.4.60)
(in (1.4.58) and (1.4.59) the functions O(A) denote terms of the order of A such that lirriA->.o O(A)/A = AT, where TV is a finite number).
74
Chapter I
Using (1.4.55)-(1.4.60) and passing to the limit as A -> 0 in (1.4.54), we obtain the following Bellman integro-differential equation for the function FS:
^(t,x,y) + X ( t , y) \ f ir(t, y, z)F3(t, x,z)dz- F3(t, x, y) Ot
LJ
+ min \aT(t,x,u)—^-(t,x,y)+c(x,y,u) = 0, 0 < t < T, (1.4.61) ueu |_ ox J F3(T,x,y)=il>(x,y). (1.4.62) If A(t, y) = \(y), 7r(t, y, z) = ?r(y, z), a(i, z, u) = a(x, u), tf>(x, y) = 0, and T —^ oo, then the system shown in Fig. 10 may operate in the stationary tracking mode (see Section 1.4.2). In this case, instead of (1.4.61), we have the stationary Bellman equation A(y)
Tr(y,z)f3(x,z)dz-f3(x,y)\
aT(x, u)--(x,y) +c(x,y, u) =7, ox J where a stationary loss function f3(x,y)
(1.4.63)
is determined by analogy with
(1.4.29) as f3(x, y) = ^ [F3(t, x, y) - 7(T - t)] and the number 7 > 0 determines mean losses per unit time in the stationary tracking mode under the optimal control. The solution of the timeinvariant equation (1.4.63) for a special synthesis problem is given in §2.2. In conclusion, we make some remarks. First, we note that in this section we have considered only the synthesis problems (and the corresponding Bellman equations) that are studied in
the present monograph. The Bellman equations for other stochastic control problems can be found in [1, 3, 5, 18, 34, 50, 57, 58, 113, 122]. Moreover, the ideas and methods of the dynamic programming approach are widely used for solving problems of optimal control for Markov sequences and processes with finitely or countably many states [151, 152], which we do not consider in this book.
We also point out that many arguments and computations in this section are of rather formal character and sometimes correspond to the "physical level of rigor." To justify the optimality principle, the sufficiency of Markov optimal strategies, the validity of Bellman differential equations, and the solvability of synthesis problems rigorously, it is required to have rather complicated and refined mathematical constructions that are beyond the
framework of this book. The reader interested in a closer examination of these problems is referred to the monographs [58, 59, 175], and especially to [113].
Synthesis Problems for Control Systems
75
§1.5. Sufficient coordinates in control problems with indirect observations
We have already noted that the dynamic programming method in, so to say, its "pure" form can be used only for Markov controlled processes. Let Xt be a current phase state of the system. The probabilities of future states -X"t+A (A > 0) of the process X(t) must be completely determined by the last measured value Xt. However, since the time evolution of X(t) depends on random perturbations and control actions, the process X(t) satisfies the Markov property only if the values Ut of the current control are determined by the instant values of the phase variables and time as follows:
ut =
(1.5.1)
The Markov property of the process X(t) allows us to write the basic functional equation of the optimality principle, then to obtain the Bellman equation, etc., that is, to follow the procedure described in §1.4. To implement the control algorithm in the form (1.5.1), it is necessary to measure the phase variables Xt exactly at each instant of time. This possibility is provided by the servomechanism shown in Fig. 10. In this case, the phase variables Xt = Xt — ( x t - , y t ) are the components of the (n + m)-dimensional vector of instant input (assigning) actions and output (controlled) variables. Now let us consider a more general case of the system shown in Fig. 3. At each instant of time, instead of true values of the vectors Xt and y$, we have only the results of measurements J/Q and XQ, which are sample paths of the stochastic processes { y ( s ) : 0 < s < t] and { x ( s ) : 0 < s < t}. These processes are mixtures of "useful signals" y^, x*0 and "random noises" £Q, ?7oOnly these results of measurements can be used for calculating the current values of the control actions Ut', therefore, the desired control algorithm for the system shown in Fig. 3 has the form of the functional ut =
(1.5.2)
To illustrate the computation of the optimal functional ¥>(<, EQ, J/Q), we consider, as an example, the basic synthesis problem (see §1.4, Section 1.4.1) in the case of indirect observations. Assume that the equation of the controlled plant, the restrictions on the control, and the optimality criterion have the form
x(t) = a(t,x(t),u(t))+cr(t,x(t))£(t),
u(t) <E U C R r ,
0 < t < T,
(1.5.3)
(1.5.4)
76
Chapter I
I(u)
= E\ f c(x(t), y ( t ) , u ( t ) ) dt + ^(x(T), y(T))] Uo J
(1.5.5)
(here we use the notation from (1.4.2), (1.4.3), and (1.4.4) in §1.4). The
observed processes x ( t ) and y ( t ) are determined by the relations
x ( t ) = P ( t } x ( t ) + Q(t)r,(t);
y(t) = H(t)y(t) + G(t)C(t),
0
Here P, Q, H, and G are given matrices whose dimensions agree with the dimensions of the vectors x, x, n, y, y, and £. We also assume that the
vectors x and rj (as well as the vectors y and Q are of the same dimension, and the square matrices Q(t) and G(t) are nondegenerate for all t G [0, T].11
We assume that the stochastic process £(i) in (1.5.3) is the standard white noise (1.1.34) and the other stochastic functions y(t), C(^), and r)(t) are Markov diffusion processes with known characteristics (that is, with given drift and diffusion coefficients). The stochastic processes £ ( t ) , y(i), CWi and r](t) are assumed to be independent. We also note that the stochastic process x(t), which is a solution of the stochastic equation (1.5.3), is not Markov, since in this case the control functions u(t) = Ut on the right-hand side of (1.5.3) have the form of functionals (1.5.2) and depend on the history of the process.
Following the formal scheme of the dynamic programming approach, by analogy with (1.4.5), we can define the loss function for the problem considered as follows:
t
U
~. ~ „
T
-, c(z(r), y(r), u(r))dr + j(x(T), y(T)) \ z«, %< . J
„
(L5 7)
-
Since the functions XQ and y^ are arguments of F in (1.5.7), it would be more correct if expression (1.5.7) were called a loss functional; however, both (1.5.7) and (1.4.5) are called loss functions.
In contrast with §1.4, it is essentially new that we cannot write the optimality principle equation of type (1.4.6) or (1.4.10) for the function
(1.5.7), since this function depends on the stochastic processes x ( t ) and y ( t ) , which are not Markov. Formula (1.5.6) immediately shows that x ( t ) and y(t) have no Markov properties, since the sum of Markov processes
is not a Markov process. Moreover, it was pointed out that the process x ( t ) itself is not Markov. Therefore, we can solve the synthesis problem 11
For simplicity, we assume that Q(t) and G(t) are nondegenerafce, but this condition
; not necessary [132, 175].
Synthesis Problems for Control Systems
77
by using the dynamic programming approach only if we can choose new "phase" variables X(t) = Xt for the loss function (1.5.7) so that, on one hand, they were sufficient for computation of minimum future losses in the sense of
F(t,x*,y*)=F(t,Xt) and, on the other hand, the stochastic process X(t) be Markov. Such phase variables Xt are called sufficient coordinates [171] by analogy with sufficient
statistics used in the mathematical statistics [185]. It turns out that there exist sufficient coordinates for the problem considered and Xt is a collection of instant values of observable processes x(t) = Xt and y ( t ) — yt and of the a posteriori probability density p(t,xt,yt) = p(x(t) = xt,y(t) = yt | *o,5o) of the unobserved vectors Xt and yt,
Xt=(xt,yt,p(t,xt,yt)).
(1.5.8)
In what follows, it will be shown that the coordinates (1.5.8) are sufficient to compute the loss function (1.5.7). In the case of an uncontrolled process x(t), the Markov property of (1.5.8) follows from Theorem 5.9 in [175].
To derive the Bellman differential equation, it is necessary to know equations that determine the time evolution of sufficient coordinates. For the first two components of (1.5.8), that is, for the process x ( t ) and y ( t ) , these equations can be assumed to be known, because one can readily obtain them from the a priori characteristics of the processes j/(<), x(t), C(£), n(t) and formulas (1.5.6). Later we derive the equation for the a posteriori probability density p(t, zj, yt). First, we do not pay attention to the fact that the control ut has the form of a functional (1.5.2). In other words, we assume that u(t) in (1.5.3) is a known deterministic function of time. Then the stochastic process x(t) that satisfies the stochastic equation (1.5.3) is a diffusion Markov process whose characteristics (the drift and diffusion coefficients) are uniquely determined
by the vector a(t,x,u) and the matrix cr(t,x) (see §1.2). Thus, in our case, x(t), y(t), C(i), and ^(t) are independent stochastic Markov diffusion processes with given drift coefficients and matrices of diffusion coefficients. In view of formulas (1.5.6) and the fact that the matrices Q(t) and G(t)
are nondegenerate, it follows that the collection ( x ( t ) , y(t), x ( t ) , y(i)~) is a Markov diffusion process whose characteristics can be expressed via given characteristics of the processes x(t), y(t), C(*)> and *?(<)• Indeed, if we
denote the vectors of drift coefficients by Ax(t, x), Ay(t, y ) , A f ( t , C), A^(t, n) and the diffusion matrices of independent Markov processes x ( t ) , y ( t ) , £ ( t ) ) , and r)(t) by Bx(t,x), By(t,y), -B c (i,C), B^(t,rj), then it follows from (1.5.6) that the drift coefficients Ay and Ay for the components x(t) and y(t) are
78
Chapter I
determined by the relations
A-s = P(t)Ax(t,x) + Q(t)An(t, Q~1(t)(x- P ( t ) x ) ) + P(t)x + Q(t)Q-l(t)(x - P(i)z),
(1.5.9)
f = H(t)Ay(t, y) + G(t)Ac (t, G~\t)(y - H(t)y}} G(t)G-1(t)(y-H(t)y)
(1.5.10)
and the matrix B of the diffusion coefficients of the joint process ( x ( t ) , y ( t ) , has the block form
Bz(t,x,x) 0 Bx(t,x)PT(t) 0
0 By(t,y,y) 0 By(t,y)HT(t)
P(t)Bx(t,x) 0 Bx(t,x) 0
0 H(t)B9(t,; 0 By(t,y)
, (1.5.11)
where Bx(t,x,x) and B^(t,y,y) are square matrices12 determined by the relations
(1.5.12) T
By(t,y,y)=H(t)By(t,y)H (t] + G(t)B((t,G-l(t)(y-H(t)y))GT(t). (1.5.13) Now we point out that in the Markov collection of random functions
( x ( t ) , y(f), x(t), yit)) the components x ( t ) and y(t) are observable, but the components x(y) and y ( t ) are not observable. Partially observable Markov processes are often called conditional Markov processes. The rigorous theory of such processes can be found in [132, 175]. Let us consider the conditional (a posteriori) density p(t, xt, yt) = (x(t) = xt, y(t) = yt XQ, y0') of the probability distribution for unobservable com-
ponents of the partially observable Markov process ( x ( t ) , y ( t ) , x ( t ) , y ( t ) ) . It turns out that the a posteriori density p(t, Xt, yt) satisfies a stochastic partial differential equation, first obtained in [175], This is a generalization of the Fokker-Plank equation (1.1.67) to the case of observation. In what follows, we briefly derive this equation. 12 If P, Q, H, and G in (1.5.6) are row matrices, then B~(t,x,x) and B~(t,y,y) are scalar functions.
Synthesis Problems for Control Systems
79
According to [175], we introduce the following notation. We denote the collection of random functions (x(t), y(t), x(i), y ( t ) ) that forms a Markov process by a single letter z(t) and assume that the dimension of the vector z is equal to n. We assume that the unobservable components of the vector z are numbered from 1 to m and the observable components are numbered from ra+1 to n. For convenience, we write xa (I < a < m) for unobservable components and yp (m + 1 < p < n) for observable ones. We also use
three groups of indices: the indices i,k,t,... vary from 1 to n; the indices a, /3, 7, . . . from 1 to m; and /?,
z ( t ) = z} = Ak(t,z); Hm
Azk = zk(t + A) -
E[Azfc AZ, | z(t) =z}= Bkt(t, z);
Urn iE[Az f c l ...Az f c l . | z(t) = z] = 0
(1.5.14) (r > 2).
It is required to obtain an equation for the a posteriori probability p(t, x$) = p(xt | 2/o)i provided that (1.5.14) and the results of observation of y*Q are known. Using the transition probability PA(Z*+A I z*) = PA(a=t+A,^t+A xt,yt) and the probability multiplication theorem, we obtain
p(xt+A,Vt+A,xt | 3/0) =PA(a5t+A,2/«+A
xt,yt)p(xt \ y*0).
(1.5.15)
Integrating (1.5.15) with respect to xt and taking into account (1.1.50), we obtain
I J/o) = / PA(a: t + A,2/t+A ] xt,yt)p(t,xt] dxt.
(1.5.16)
If we write the left-hand side of (1.5.16) in the form p(z«+A,yt+A I 2/o) =P(«t+A | 2/o'2/«+A)p(y*+A
J/Q)>
then we can write (1.5.16) as follows: P(Z*+A
2/o.2/t+A) = -7———j—JT / PA(**+A,y«+A | xt,yt)p(t,xt)dxt.
p(yt+A i %/ ^
(1.5.17)
Integrating (1.5.16) with respect to xt+&, we obtain = //
| xt,yt)p(t, xt)dxtdxt+&.
(1.5.18)
Chapter I
80
Substituting (1.5.18) into (1.5.17) and taking into account the fact that the equality
is valid, since the arguments are continuous, we obtain
xt,yt)p(t,xt)dxt
,
p(t + A,
a;*,
—-—— ~ ——-
+ o(A).
(1.5.19) Equation (1.5.19) for partially observable Markov processes plays the
same role as the Markov (Smoluchovski) equation (1.1.53) for the complete observation. To derive the differential equation for the a posteriori density p(t, Xi] from (1.5.19), we use the same method as for the derivation of the
Fokker-Planck equation in §1.1 (see (1.1. 59)-(l.l. 64)). Let us introduce two characteristic functions of random increments As a , a = ! ) • • • ) m, and Az^, k = 1, . . . , n.,13
= I exp [jua(xat+& - x a ( )]p A (x t + A,J/«+A J
Xt,yt)dxt+&,
(1.5.20)
i, • • • , un,zt)
(1.5.21)
- I exp[JM
The transition probability can be expressed in terms of inverse Fourier transforms as follows:
zt = —71—— / exp [ - juk(zkt+A - zkt)] l^ "; J x 02(ui,...,un,zt)dui,...,dun,
(1.5.22)
zt = -—— I exp [ - jua(zat+& - zat)] \*n)
J
(1.5.23) 13 In (1.5.20) and (1.5.21), as usual, j = \/—1 and the sum is taken over the repeated indices:
Synthesis Problems for Control Systems
81
Using the expansion of In $2(1*! zt) in the Maclaurin series, we can write
(1.5.24)
where Ks [ A z & , . . . , Az r ] denotes the s-order correlation between the components of the vector of increments Az = z(t + A) — z(i) of the Markov process z(t). Using well-known relations between correlations and initial
moments [173, 181], we see that (1.5.14) gives the following representation for (1.5.24):
0 2 (wi,- • -,un,zt) = exp \&jAkuk - —BkiukUi + o(A) Zi
L
(1.5.25)
-i
(for brevity, in (1.5.25) and in the following we do not write arguments of
Ak and Bkt, namely, Ak = Ak(t, zt) and Bkl — Bki(t, zt)). Comparing (1.5.22) and (1.5.23), we see that 0i(ui, ...,um, zt, y t + A ) = (27r)—— / exp [ - ;«<,(^+A - yat}}
x ^2(1*1,..., un,zt) dum+i.. .dun.
(1.5.26)
After the substitution of (1.5.25), we can calculate the integral (1.5.26) explicitly. As the result of integration, for the characteristic function #1 we obtain the formula14
— Kex.p [L(ui,.. .,u r o ,z t ,2/ t + A )A + o(A)],
(1.5.27)
where
= jua Aa + Ba
-
,
(1.5.28)
14
To obtain (1.5.27) and (1.5.28), it suffices to use the well-known formula [67] /
J
ex
P
L
~ ~Bkt(xk ~ mk)(xe-mt)\
2
J
dxi ...dxn = \
*
V detij
which holds for real symmetric positive definite matrices B = j|Sfe^|jr and any constants 771 ft and 7Ti£.
82
Chapter I
and K is a constant that does not influence the final result of calculations. Note that we calculated (1.5.26) under the assumption that the matrix ||.B<7p||ro+i is nondegenerate and we used the notation ]|-FVp|| = H-Bo-pll" 1 Since the exponent in (1.5.27) is small (~ A), we replace the exponential by the first three terms of the Maclaurin series ex = 1 + x + x 2 /2, truncate the terms whose order with respect to A is larger than 1, and obtain
exp [L(ui, . . . , w m , zt, 2/«+A)A + o(A)] = 1 + L(UI, ...,um, zt, 2/ t+A )A
- —(uaupBavFvpBpft
- ArFrpAp) + juaBa^F^pAp^ + o(A).
(1.5.29)
In (1.5.29) we used the relation FapBpT = BapFpr = 8VT, where 8ffT is the Kronecker delta, and the formula
Ay p At/ r = Bpr(t, z t )A + o(A) = BpT A + o(A),
(1.5.30)
which follows from the properties of Wiener processes and is a multidimensional generalization of formula (1.2.8) (for details, see Lemma 2.2 in [175]). Substituting (1.5.28) into (1.5.29) and collecting the similar terms, we obtain
exp [£(ui,...,w m ,z t ,3/ t + A )A + o(A)] = 1 + (jua)(AaA + Ba
(1.5.31)
Using (1.5.23), (1.5.27), and (1.5.31), we calculate the numerator of the fraction on the right-hand side in (1.5.19): /
rr
[
//
II
.',.
(A
A
Jua\-H-a'-l -
+ o(A).
(1.5.32)
Taking into account the formulas (see (1.1.60)-(1.1.64)) 1 ____ m (2 }
/C e-3ui(!C-ft+^.-x-ft)
/ -,'->,
(2 ?r)m J
m
j
j
_
aui...aum—
(,-ju-f(xyt+A.-x^f)
j
j
'''
Synthesis Problems for Control Systems
dS(xat+A — Xat)
TT
f
83
/
(27T)™
we obtain the numerator in (1.5.19) (we omit the constant K, since K and a similar constant in the denominator of (1.5.19) are canceled): r\
p(t, xt+&) - -^——— [(vla A + Ba
32 ———— 5———— [Bapp(t,xt+±)]
+A<7F(7pp(t,xt+&))&yp + o(&)
Q g 23)
(in (1.5.33) (<, zt+A,J/t) are the arguments of the coefficients Afc and Fffp). The denominator of the right-hand expression in (1.5.19) differs from the numerator by integration with respect to Xt+&. We perform this integration, take into account the fact that the normalization condition for the probability density p(t, xt+&) implies p(t, »«+A) dxt+A = 1;
p(t, x) —>• 0,
x) *0 — — » 0 c/xa
as
\x\ —> oo,
and from (1.5.33) obtain the following expression (without K) for the denominator in (1.5.19): I
o(A)
= 1 + &ypEpsAffFap
+ o(A),
(1.5.34)
where E p s (-) denotes the a posteriori averaging f(-)p(t,x)dx. We assume that the elements of the matrix Bap (and of F<,p, respectively) are independent of unobservable components x and take into account (1.5.30). Then we can write
(1.5.35)
84
Chapter I
Multiplying (1.5.33) by (1.5.35) and substituting the result into (1.5.19), we obtain
1
/)2
AaP - PEpsAff - j(Bavp) OX(y
+ o(A). (1.5.36) -*
As A —> 0, the terms denoted by o(A) in (1.5.36) disappear, and the finite increments become differentials. In this case, according to §1.2, it is necessary to point out in which sense stochastic differentials are understood, since the differential equation obtained is stochastic (it contains the differential of the Markov process dyp(t)). Comparing Eq. (1.5.36) (as A —> 0) with the stochastic equation (1.2.3), we see that now the a posteriori probability density p(t, x) in (1.5.36) plays the role of the random function x(t) in (1.2.3), and the vector-function (1.5.37) plays the role of the function
dt [d0yp(t) - EpsAp dt}Fpa. pAa - pEpsAa - -— (5a
d2
dp _ 1
r
m-2dx-^[(Baf}--
dyr
2 yp
dxa
Tf>
dyT
Synthesis Problems for Control Systems f\
a
i -&TP
rt
VT
85 i J? <rp-**-rT **-p
by using coupling formulas between stochastic differentials and integrals (see §1.2);15 in (1.5.39) yp = yp(t) = dyp(t)/dt denotes the formal timederivative of the Markov process yp(t).
Equation (1.5.39) is a generalization of the Fokker-Planck equation to the observation case. It should be noted that if some transformations of random functions p(t,x) are necessary (see formulas (1.5.41)-(1.5.44) below),
then it is more convenient to use Eq. (1.5.39), although it is more cumbersome than a similar Ito equation (1.5.38), since (see §1.2) the symmetrized form allows us to treat random functions (even such singular functions as white noises) according to the same formal rules as for deterministic and
sufficiently smooth functions. We can show [132] that VFap[d0yp(t) - EpsApdt}16 is the differential of the standard Wiener process dr](t) studied in §1.2. Therefore, in view of Eq. (1.5.38), the already cited Markov property of the set (j/t,p(<, x)) can be obtained by not completely rigorous but sufficiently illustrative arguments. Indeed, since the increments [^(t + A) — r),,(i)] of the stochastic pro-
cesses riv(t) = ^/Fap[yp(t)
- J0 EpsAp(T,y(T))dt]
in (1.5.38) are mutually
independent, the future values of the a posteriori probability p(t + A, x) are
completely determined by (x t , yt,p(t, *))• Since the vector xt is unobservable, then the probabilities p(t + A, x) of future values are determined by \ytip(t, x)) and the probability of the current value Xt, that is, by the a posteriori density p(t, x) contained in (yt,p(t, «))• On the other hand, since the process z(t) is of Markov character, the probabilities of future values of the observable process yt+A are completely determined by its current state z t — (xt,yt)i that is, by the same set (yt,p(t, x)), since xt is unobservable. This implies that (y(t),p(t, x)) is a Markov process. Now let us recall that Eqs. (1.5.38) and (1.5.39) were derived under the assumption that the control u(t) in (1.5.3) is a known deterministic
function of time. However, if the control u(t) is given by the functional (1.5.2) (in the new notation, following (1.5.14), this functional has the form
u(t] — ut = p(t, j/o))> then this fact does not affect the Markov properties of (j/t,p(t, a:)), since it is assumed that (j/t,p(i,a:)) is determined by the 15 Here we do not show in detail how to transform the Ito equation (1.5.38) to the symmetrized form (1.5.39); the reader is strongly recommended to do this useful exercise
on his own. 16
\AF(Tp denotes an element of the matrix \/F which is the square root of the ma-
trix F; since the matrix |]Bp
86
Chapter I
entire past history of the observations J/Q = { y ( s ) : 0 < s < t}. Thus, for a given state of (yt,p(t, x)) and any chosen functional if> in (1.5.2), the control Ut is a known vector on which the functions o(i, x, u) in (1.5.3) and the functions Aa and Aa in (1.5.38) depend as on a parameter. Hence it
follows that Eqs. (1.5.38) and (1.5.39) are also valid for controlled processes (provided that the control is given in the form (1.5.2)).
Now let us return to the synthesis problem and the dynamic programming approach. Describing the state of a controlled system at time t by (1.5.8) or, briefly, by (yt,p(t,x)^
(recall that after (1.5.14) we introduced
the new notation: Sj, yt —>• yt and xt, yt —)• xt), we can write the loss function (1.5.7) as F(t,i/Q) = F(t, yt,p(t, z)). Using the Markov property of (yt,p(t, z)), we can write the basic equation of the optimality principle for the function F(t,yt,p(t,x)} as follows:
F(t,yt,p(t, x)) =
rnin
«reu
E< \
"*+A c(xT,yT,uT)dr
t
ytt,,PP(t,x)\. (t,x).
(1.5.40)
Generally speaking, by passing to the limit as A —> 0 in (1.5.40) and using (1.5.14) and (1.5.38) (or (1.5.39)), we can obtain the Bellman differen-
tial equations by analogy with §1.4. However, the equation obtained in this way contains the functional derivatives SF/Sp(t,x), 82F/6p(t,x)Sp(t,x), etc.;
usually it is difficult to solve this equation (as pointed out in §1.4
and §1.5, even the solution of "usual" Bellman partial differential equa-
tions is a rather complicated problem). Therefore, it is more convenient in practice, instead of the a posteriori density p(t, x), to use some equivalent set of parameters as arguments of the function F. We show how to do this. Assume that the a posteriori probability density p(t, x) is a unimodal function of a vector variable x for all t £ [0, T]. By the vector mt = m(t] we denote the maximum of the a posteriori density p(t, x) at time t. Expanding lnp(t, x) in the Taylor series with respect to x around the point m(t), we
obtain the following representation of the a posteriori density p(t, x): p(t, x) = exp < a(t) - y~] —
x
•^ 2J
a,/3,...,C=i
aap...dt)(xc.-ma(t))...(x<;-mf(t))^
} J
(1.5.41)
Synthesis Problems for Control Systems
87
(the scalar function a(t) in (1.5.41) is determined by the normalization condition J p(t, x) dx = 1). Using (1.5.41), we readily obtain the system of equations for the parameters (ma(t),aap(t),aapy(t), . . .) instead of the symmetrized equation (1.5.39). To this end, we rewrite (1.5.39) in the more compact form
at
<
,
p
,
,
.
(1.5.42)
Next we replace the functions Aa and $(a;, y) by their Taylor series,17 substitute (1.5.41) into (1.5.42), and successively set the coefficients of equal powers of (xa — ma), (xa — ma)(xi) — mp), ... on the left- and right-hand sides of (1.5.42) equal to each other; thus we obtain the following system of ordinary differential equations for ma(t), aa/}(t), aap^(i), . . .:
dxa' 2
-
5A7 Qi~jft dx 7P 3
dx( 03^ — -^
Q--JCI
'
, — >> 1 -5—apiS + OXct
(1.5.43)
In (1.5.43) the dot over a variable indicates, as usual, the time derivative (mp = dm/}(t)/dt). Moreover, in (1.5.43) we assume that Bap is independent of x and omit the arguments of the functions A, $, and of their derivatives; we assume_that_the values of these functions are taken at the point x = m, that is, Ap — Ap(t,m,y), d$/dxa = d$(t,m,y)/dxa, etc. It follows from (1.5.41) that the set of the parameters ma(t), aap(t),... uniquely determines the a posteriori probability density p(t, x) at time t. 17
The functions Aa and $(a;,j/) are expanded with respect to x in a neighborhood
of the point ra(£).
88
Chapter I
Thus we can use these parameters as new arguments of the loss function, since F(t, yt,p(t, x ) ) = F(t, yt,mat,aa(3t, • • •)• However, in the general case, system (1.5.43) is of infinite order, and therefore, if we use the new sufficient coordinates (yt, mat, aapt,...) instead of the old coordinates (yt,p(tix)}> then we do not gain considerable advantage in solving special problems.
Nevertheless, there is an important class of problems in which the a posteriori probability density (1.5.41) is Gaussian (conditional Gaussian Markov processes are studied in detail in [131, 132]). We have such processes if [175] (1) the elements of the matrix Bap are constant numbers; (2) the functions Aa linearly depend on x; (3) the function <3>(a;,t/) depends on x linearly and quadratically; (4) the initial probability density (the a priori probability density of unobservable components before the observation) p(0, x) is Gaussian. Under these conditions, we have aa/g7 = aap^t = • • • = 0 in
(1.5.41) and (1.5.43), and system (1.5.43) is closed and of finite dimension:
+ &o:/3 —
^*«7"'po j-"yo
f\
•
a,/3,7,(5= I,...,TO.
(1.5.44)
Now let us consider the synthesis problem corresponding to this case. To avoid cumbersome formulas, we deal with a simplified version of problem (1.5.3)-(1.5.6). Namely, we assume that the input y(t) is absent and the system shown in Fig. 3 does not contain Block 1. Suppose that the plant P is described by the system linear with respect to the output (controlled) variables
x = G(t, u)x + b(t, u) +
(1.5.45)
where x = x ( t ) is an m-vector of output variables, G(t, u) and <j(i) are given m x TO matrices, b(t, u} is a given TO-vector-function, and £(t) is an ra-vector of random perturbations of the standard white noise type (1.1.34). More
explicitly, the vector-matrix equation (1.5.45) has the form a,/3= l , . . . , m . We observe the stochastic process
x ( t ) = P(t)x(t) + Q(t)ri(t),
Q
(1.5.46)
where x and 77 are fc-vectors, P and Q are k x n and k x k matrices, the matrix Q(t) is nondegenerate for all 0 < t < T, and rj(t) is the standard white noise (1.1.34) independent of (,(t).
Synthesis Problems for Control Systems
89
Under the assumption that the admissible control satisfies condition (1.5.4), it is required to find the optimal control uf(t) = <£>(£, ZQ) such that the cost functional
c(x(t),u(t))dt + i,(x(T))
(1.5.47) J
attains its minimum value.
We write y(t) = I X(T) dr. Jo
(1.5.48)
Then (x(t), y(t)) is a Markov stochastic process, and it follows from relations (1.5.45), (1.5.46), and (1.5.48) that the characteristics (1.5.14) of this
process have the form
Aa = Ga/}(t, u)xp + ba(t, u), Bap = Q,
Aa = Paa(t}xa, (
B^ =Qpr(t)Q
' '
'
In (1.5.49) the indices a, /3, 7 take values from 1 to m and the indices p, cr, T from TTi+1 to m + k.
In this case, it follows from (1.5.49) and (1.5.39) that in (1.5.42) we have
Bap = Ba!),
Aa = Aa,
*(x,y) = AaFapyp - -Ap \
Zi
'
(1.5.50)
(Pap is an element of the matrix H-B^H"1 = [Q(t)QT (^)] -1 ). It follows from (1.5.49) and (1.5.50) that in this case system (1.5.43) has the form (1.5.44). Substituting (1.5.49) and (1.5.50) into (1.5.44), we obtain the following system of equations for the parameters ma(t), aa/3(t), a,(3 = 1, . . . , m, of the a posteriori density:
b p ( t , u ) ] + P
- Pp/)(t)mf}),
(t, u)a7/3 - a a7 G 7/3 (t, u)
(1.5.51)
+ P<,a(t)Fffp(t)Ppf)(t). System (1.5.51) can be written in a more compact vector-matrix form. So, introducing the matrix A = ||aa/g|[J* and taking into account the fact that yp = xp according to (1.5.48), we see that (1.5.51) implies
Am = A[G(t, u)m+b(t, u)} + PT (t)[Q(t)QT (t)]-l(x - P(t)m),
A - -Aa(t)crT(t)A - GT(t,u)A - AG(t,u) T
T
+ P (t)[Q(t)Q (t)]^P(t).
(1.5.52)
90
Chapter I
Now we note that the right-hand sides of (1.5.52) do not explicitly depend on y(t), and moreover, the cost functional (1.5.47) is independent of the observable process x(t). Therefore, in this case, the current values of the vector yt do not belong to the sufficient coordinates of problem (1.5.45)(1.5.47), which are the current values of the components of the vector mt and of the elements of the matrix At. If instead of the matrix A we consider the matrix D = A~l of a posteriori covariances, then, multiplying the first equation in (1.5.52) by the matrix D from the left and the second equation in (1.5.52) from the left and from the right and taking into account the formulas
DA = AD = E,
DA + DA = 0,
D = -DAD,
we obtain, instead of (1.5.52), the relations
TO = G(t,u)m+b(t,u) + DPT(t)[Q(t)QT(t)}-1(x(t) T
T
D = v(t)
T
- P(t)m),
(1.5.53)
1
- DP (t)[Q(t)Q (t)]- P(t)D. Equations (1.5.53) are well-known equations of the Kalman filter [1, 5, 58, 79, 132]. As is known, the Kalman filter is a device for optimal filtering of the "useful signal" x(t) that is observed on the background of a random noise. In this case, the vector m(t) is an optimal18 estimate of current values of the components of the unobservable stochastic process x(t) that is the result of observation of XQ = {x(s): 0 < s < t}, provided that the observation process is given by (1.5.46). The matrix D(t) that satisfies the second (matrix) equation in (1.5.53) characterizes the accuracy of the estimation of unobservable components of the process x(t) by the vector
m(t) (see [1, 5, 79]). Equations (1.5.53) play the role of "equations of motion" for the controlled system in the space of sufficient coordinates. Since the process Q~l(t)(x(t) — P(t)mj is a white noise, the first equation in (1.5.53) is a stochastic equation of type (1.5.45), and the second equation is a usual differential (matrix) equation. Therefore, the Bellman differential equation for the loss function F(t,mt, Dt) can be derived by a technique similar to that used in §1.4 to derive Eq. (1.4.21) for the function (1.4.5). 18 The optimality of the estimate m(t) is understood as the minimum of the mean square deviation E|a;(t) — m(t)| 2 ; as is known [167, 175, 181], in the Gaussian case, m(t) coincides with the maximum point of the a posteriori probability density p(t, x) = p(x(t) = x | S0f).
Synthesis Problems for Control Systems
91
After similar calculations, we obtain the Bellman equation of the following form (see also [34, 175]) for the function F(t, m, D) in problem (1.5.45)(1.5.47):
dt
I
ft p
(TOTGT (t, u) + 6 T (t, u)) ——
3F
1
+ Sp ^(VGT(t, u) + G(t,«)D) + c(m, D, u) = 0, (1.5.54)
where dF/drn is an w,-vector with components dF/dma, a = l , . . . , m ; 82F/dmdmT is the mxm matrix of the derivatives d2F/dmadmf), a, (3 = 1,..., TO; dF/dD is the mxm matrix of the partial derivatives dF/dDap, a,/3 = 1,..., TO; and c(m, D, w) denotes the a posteriori mean of the penalty
function c(x,u) in the functional (1.5.47), that is,
= [(2 7 r) T O detD]- 1 / 2
(1.5.55) The loss function F(t, TO, D) satisfies (1.5.54) for 0 < t < T.
At the
terminal instant of time t = T, this function is determined by the relation
F(T,m,D) = EpsTp(x),
(1.5.56)
where, by analogy with (1.5.55), E ps (-) denotes integration of (•) with Gaussian density.
We see that (1.5.56) is a generalization of condition
(1.4.22) to the case of indirect observations. As usual, by solving Eq. (1.5.54) with the additional condition (1.5.56), we simultaneously obtain the optimal control u*(t) = ipi(t,m(t), D(t)) (see §1.3 and §1.4). Thus the desired algorithm of optimal control in the functional form u*(t) = (p(t,Xo) for problem (1.5.45)-(1.5.47) is the superposition of the two operations: the optimal filtering of the observed process ( x ( t ) : 0 < t < T) by means of the Kalman filter (1.5.53) and the formation of the current control ut(t) = tpi(t,m(t), D(t)). This situation is typical of other problems with indirect observations. Therefore, in the general case of the servomechanism shown in Fig. 3, the
Chapter I
92
FIG. 12 controller C actually consists of two blocks that are functionally different (see Fig. 12): the sufficient coordinate block SC that models the corresponding filter and the decision block D whose structure is determined by the solution of the Bellman equation.
Some examples of other Bellman equations obtained by using sufficient coordinates, as well as solutions of these equations, will be considered later
in §3.3, §4.2, §5.4, and §6.1.
CHAPTER II
EXACT METHODS FOR SYNTHESIS PROBLEMS
Exact solutions to synthesis problems of optimal control are of deep theoretical and practical interest. However, exact solutions can be obtained only in some special cases. The point is that exact methods are characterized by rather strict restrictions on the assumptions of the synthesis problem, but these assumptions are seldom satisfied in actual practice. It is well known that, for instance, the Bellman equation can be solved exactly under the following assumptions: (1) the dynamic equations of the plant are linear, (2) the optimality criterion of the form (1.1.11) or (1.4.3) contains only quadratic penalty functions, (3) no restrictions are imposed on the control and on the phase coordinates, (4) random actions (if any) on the system are Gaussian Markov processes or processes of the white noise type. The synthesis problems satisfying (l)-(4) are called linear- quadratic problems of optimal control. An extensive literature is devoted to these problems [3, 5, 18, 24, 72, 112, 122, 128, 132, 168]. In the present chapter we restrict our consideration to an outline of methods for solving such problems (§2.1) and consider in more detail less known results concerning the solution of some special synthesis problems with bounded controls (§§2.2-2.4).
§2.1. Linear-quadratic problems of optimal control (LQ-problems)
2.1.1. First, let us consider an elementary optimal stabilization problem of a first-order system perturbed by a Gaussian white noise (see Fig. 13). Suppose that the plant P is described by a linear scalar equation of the form
x - ax + bu+ A/KW,
(2.1.1)
where a, b, and v are given constants (y > 0) and £(t) is the standard white noise (1.1.31). The performance of this system is estimated by the following functional of the form (1.4.3) with quadratic penalty functions:
T
[cx2(t) + hu2(t)} dt + Cl z 2 (T)
(2.1.2) J
93
Chapter II
94
c
u(t)
FIG.
x(t)
P
13
(here c, ci, and ft, are given positive constants). We do not impose any restrictions on the control u and the phase variable x.
Problem (2.1.1), (2.1.2) is a stochastic generalization of the linear-quadratic problem (1.3.24), (1.3.25) considered in §1.3 and a special case of a more general problem (1.4.2)-(1.4.4). Since the stabilization system shown in Fig. 13 is a specific case of the servomechanism shown in Fig. 8, the Bellman equation for problem (2.1.1), (2.1.2),
- ex2 + min I bu—— + hu2\ = 0,
ax
~dt
'
(2.1.3)
can be obtained from (1.4.21) by setting
Ay=By = 0,
Bx = v,
c(x,y,u) = cx2 + hu2,
Ax = ax + bu.
In (2.1.3) the loss function F = F(t,x), determined, as usual, by
t
satisfies Eq. (2.1.3) in the strip ET = {0 < t < T, -oo < x < 00} and becomes a given quadratic function,
F(T, x) — cix2,
(2.1.5)
for t = T. Condition (2.1.5) readily follows from the definition of the loss function (2.1.4) or from formula (1.4.22) with ip(x, y) — c-^x2. The optimal control M* in the form (1.4.25), which minimizes the expression in the square brackets in (2.1.3), is determined by the condition
d[-]/du — 0 as follows:
—sir
<"•"
Exact Methods for Synthesis Problems
95
Substituting, instead of u, the control u* into the expression in the square brackets in (2.1.3) and omitting the symbol "min", we rewrite Eq. (2.1.3) in the form
?L +ari>L+1L*PL + 2
dt
dx
2 dx
r
2_^lf^V=o
4h\dx)
(217} {
' ' '
(Eq. (2.1.7) is just Eq. (1.4.26) for problem (2.1.1), (2.1.2)). Now to solve the synthesis problem, it remains to find the solution F(t, x)
that satisfies Eq. (2.1.7) in the strip HX and is a continuous continuation of (2.1.5) as t —> T. We shall seek such a solution in the form
F(t,x) =p(t)x2+ r ( t ) ,
(2.1.8)
where p(t) and r ( t ) are some functions of time. We choose these functions so that the solution of the form (2.1.8) satisfy (2.1.5) and (2.1.7). Substituting (2.1.8) into (2.1.7) and setting the coefficient of x2 equal to zero, as well
as the terms independent of a;, we obtain the following equations for the unknown functions p(t) and r(t):
b2 p=-c-2ap+--p2, h r=-vp.
(2.1.9) (2.1.10)
It follows from (2.1.5) that the solutions p(t) and r(t) of (2.1.9) and (2.1.10) attain the values p(T) = d, r(T)=0 (2.1.11) at the terminal time t = T. The system of ordinary differential equations (2.1.9), (2.1.10) with additional conditions (2.1.11) can readily be integrated. As a result, we obtain the following expressions for the functions p(t) and r(t): T
Di-D2eW-
)
,__
v\D2. D3 + DtJW-V =
l
D,
n
(2.1.13) where the constants (3, DI, D2, D3, and D4 are related to the parameters
of problem (2.1.1), (2.1.2) as follows: cb2
—,
DI = c + ci(a + /3), 2
b D3=f3-a+—Ci, n
D2 = c+ Cl (o -/?),
b2 D4=/3 + a-—Cl. h
Chapter II
96
From (2.1.6), (2.1.8), and (2.1.12), we obtain the optimal control law
w,(<, x) - p(t)x,
(2.1.14)
p(t) = -
which is the solution of the synthesis problem for the optimal stabilization
system in Pig. 13. It follows from (2.1.14) that in this case the controller C in Fig. 13 is a linear amplifier in the variable x with variable amplification factor p(t). In the sequel, we indicate such amplifiers by a special
mark ">." Therefore, the optimal system for problem (2.1.1), (2.1.2) can be represented as the block diagram shown in Fig. 14.
FIG. 14 Obviously, the minimum value I[u*] of the optimality criterion (2.1.2) with control (2.1.14) and the initial state z(0) — x is equal to F(0, x). From
(2.1.8), (2.1.12), and (2.1.13), we have
J4e-2^T
"fib
2/3 \D4
n
3
Di
+D 4 + r > 3
^—— ^ .
(2.1.15)
To complete the study of problem (2.1.1), (2.1.2), it remains to prove
that the solution (2.1.12)-(2.1.15) of the synthesis problem is unique. It follows from our discussion that the problem of uniqueness of (2.1.12)(2.1.15) is equivalent to the uniqueness of the solution (2.1.8) of Eq. (2.1.7). The general theory of quasilinear parabolic equations [124] implies that
Eq. (2.1.7) with additional condition (2.1.5) has a unique solution in the class of functions F(t, x) whose growth as x —¥ oo does not exceed that of any finite power of x\. On the other hand, an analysis of properties of the
loss function (2.1.4) performed in [113] showed that, for each t e [0,T] and x £ RI, the function (2.1.4) satisfies the estimate
0
97
Exact Methods for Synthesis Problems
where N(T) is bounded for any finite T. Therefore, the function (2.1.8) is a unique solution of Eq. (2.1.7), corresponding to the problem considered, and the synthesis problem has no solutions other than (2.1.12)-(2.1.15).
REMARK. The optimal control (2.1.14) is independent of the parameter v, that is, of the intensity of random actions on the plant P, and coincides with the optimal control algorithm (1.3.33), (1.3.34) for the deterministic problem (1.3.24), (1.3.25). Such a situation is typical of many other linear-quadratic problems of optimal control with perturbations in
the form of a Gaussian white noise. D The exact formulas (2.1.12)-(2.1.15) allow us to examine the process of relaxation of stationary operating conditions (see §1.4, Section 1.4.2) for the stabilization system in question. To this end, let us consider a special case of problem (2.1.1) in which the terminal state x(T)
is not penalized
(ci = 0). In this case, formulas (2.1.12) and (2.1.13) read
_a
a ) e -2/3(T-t)
(2.1.16)
'
vh, r^ -ln[/3-a + (ft-
a e
1/3. -Sln2/3. (2.1.17)
If the operating time is equal to T > t\ = 3/2/3, then the functions p(t) and r(t) determined by (2.1.16) and (2.1.17) have the form shown in Fig. 15.
vcT
/3-a
T- — T t 2/3
T- — T t
2/3
FIG.
15
The functions p(i) and r(t) are characterized by the existence of two time intervals [0, T—
98
Chapter II
ways. The first interval [0, T — ii] corresponds to the stationary operating mode, that is, p(t) ~ c/(/3 — a) = const for t (E [0, T— ti], the function r(t) linearly decreases as t grows, and on this interval the rate of decrease in r(i) is constant and equal to vc/(f) — a). The terminal interval [T — ii,T] is essentially nonstationary. It follows from (2.1.16) and (2.1.17) that the length of this nonstationary interval is of the order of 3/2/3. Obviously, in the case where this nonstationary interval is a small part of the entire operating time [0, T], the control performance is little affected if, instead of the exact optimal control (2.1.14), we use the control
x
(2.1.18)
that corresponds to the stationary operating mode. It follows from (2.1.18) that for large T the controller C in Fig. 13 is a linear amplifier with constant amplification factor, whose technical realization is much simpler than that of the nonstationary control block described by (2.1.14) and (2.1.12). Formulas (2.1.16) and (2.1.17) show that, for large values of T — t, the loss function (2.1.8) satisfies the approximate relation
F(t,x)~-?—x2 +
-(T-t).
(2.1.19)
Comparing (2.1.19) and (1.4.29), we see that in this case the value 7 of stationary mean losses per unit time, introduced in §1.4, is equal to VC
' ~~ p-a a '
(2.1.20)
that is, coincides with the rate of decrease in the function r(t) on the stationary interval [0, T —
It should be noted that to calculate 7 and the function /(a;), we need not have exact formulas for p(t) and r(i) in (2.1.8). It suffices to use the corresponding stationary Bellman equation (1.4.30), which in this cases has the form
df and to substitute the desired solution in the form /(x) = px2 into (2.1.22). We obtain the numbers p and 7, just as in the nonstationary case, by setting
Exact Methods for Synthesis Problems
99
the coefficients of x2 and the free terms on the left- and right-hand sides in (2.1.22) equal to each other. We also note that if at least one of the parameters a, 6, i/, c, and h of problem (2.1.1), (2.1.2) depends on time, then, in general, there does not exist any stationary operating mode. In this case, one cannot obtain finite
formulas for the functions p(t) and r(t) in (2.1.8), since Eq. (2.1.9) is a Riccati equation and,
in general, cannot be integrated exactly. Therefore,
if the problem has variable parameters, the solution is constructed, as a rule, by using numerical integration methods. 2.1.2. All of the preceding can readily be generalized to multidimensional problems of optimal stabilization. Let us consider the system shown in Fig. 13 whose plant P is described by a linear vector-matrix equation of the form
(2.1.23)
where x = x(t] e Rn is an n- vector-column of phase variables, u G Rr is an r- vector of controlling actions, and £(t) £ RTO is an ra- vector of random perturbations of a Gaussian white noise type with characteristics (1.1.34). The dimensions of the matrices A, B, and a are related to the dimensions of the corresponding vectors and are equal to n x n, n x r, and n x m, respectively. The elements of these matrices are continuous functions of
time1 defined for all t from the interval [0, T] on which the controlled system is considered. For the optimality criterion, we take a quadratic functional of the form
= EJ f
[xT(t)G(t)x(t)+uT(t)H(t)u(t)}dt xTt(T)Qx(T}\.
(2.1.24)
Here Q and G(t) are symmetric nonnegative definite n x n matrices and the symmetric r x r matrix H(t) is positive definite for each t e [0, T]. Just as (2.1.3), the Bellman equation for problem (2.1.23), (2.1.24) follows from (1.4.21) if we set A« = B« = 0, Bx = a(t)(TT(t), Ax = A(t)x + B(t)u, and c(x,y,u] = xTGx + UTHu. Thus we obtain
r,. d2F G(t)x 1
+ min \uTBT(t) —— + uTH(t)u\ = 0. « \_ ax J
(2.1.25)
As was shown in [156], it suffices to assume that the elements of the matrices A(t), .B(t), and rr(t) are measurable and bounded.
100
Chapter II
In this case, the additional condition on the loss function (1.4.22) has the form F(T, x) - xTQx.
The further considerations leading to the solution of the synthesis problem
are similar to those in the one-dimensional case. Calculating the minimum value of the expression in the square brackets in (2.1.25), we obtain the optimal control 1
Q EI
(t)-,
(2.1.26)
which is a vector analog of formula (2.1.6). Substituting the expression
obtained for w* into (2.1.25), we arrive at the equation aP
f) p
i
G(T)x - -B(t)H^(t)BT(t)
= 0. (2.1.27)
We seek the solution of (2.1.27) as the following quadratic form with respect to the phase variables:
F(t,x) = xTP(t)x + r(t).
(2.1.28)
Substituting (2.1.28) into (2.1.27) and setting the coefficients of the quadratic (with respect to x) terms and the free terms on the left-hand side in (2.1.27) equal to zero, we obtain the following system of differential equations for the unknown matrix P(t) and the scalar function r(t):
P + AT(t)P + PA(t) + G(t) - PB(i)H-1(t)BT(t)P T
r + Sp Pa-(t)(T (t) = 0,
= 0, P(T) = Q;
r(T) = 0.
(2.1.29)
If system (2.1.29) is solved, then the optimal solution of the synthesis problem has the form
uf(t,x) = -H-1(t)BT(t)P(t)x
= J>(t)x,
(2.1.30)
which follows from (2.1.26) and (2.1.28). Formula (.2.1.30) shows that the controller C in the optimal system in Fig. 13 is a linear amplifier with n inputs and r outputs and variable amplification factors. Let us briefly discuss the possibilities of solving system (2.1.29). The existence and uniqueness of the nonnegative definite matrix P(i) satisfying the matrix-valued Riccati equation (2.1.29) are proved in [72] under the above assumptions on the properties of the matrices A(t), B(t), G(t), H(t),
Exact Methods for Synthesis Problems
101
and Q. One can obtain explicit formulas for elements of the matrix P(t) only by numerical methods,2 which is a rather complicated problem for large dimensions of the phase vector x. In the special case of the zero matrix G(t) = 0, the solution of the matrix equation (2.1.29) has the form [1, 132]
rT
+ Q/ Jt
-i-l
X(T,s)B(s)H-1(s}BT(s)XT(T,s)ds\
QX(T,t).
(2.1.31)
J
Here X(t,s), t > s, denotes the fundamental matrix of system (2.1.23); sometimes this matrix is also called the Cauchy matrix. The properties of the fundamental matrix are described by the relations
X(t, t) = E,
= A(t)X(t, ,),
= -X(t, s}A(s).
(2.1.32) One can construct the matrix X(t, s) if the so-called integral matrix Z(t) of system (2.1.23) is known. According to [111], a square nxn matrix Z(t) is called the integral matrix of system (2.1.23) if its columns consist of any n linearly independent solutions of the homogeneous system x = A(t)x. If the matrix Z(i) is known, then the fundamental matrix X(t, s) has the form X(t,s) = Z(i)Z-l(s). (2.1.33) One can readily see that the matrix (2.1.33) satisfies conditions (2.1.32). The fundamental matrix can readily be calculated if and only if the elements of the matrix A(t) in (2.1.23) are time-independent, that is, if A(t) = A = const. In this case, we have
and the exponential matrix can be expressed in the standard way [62] either via the Lagrange-Silvester interpolation polynomial (in the case of simple eigenvalues of the matrix A) or via the generalized interpolation polynomial (in the case of multiple eigenvalues and not simple elementary divisors of the matrix A). If the matrix A is time-varying, the construction of the fundamental matrix (2.1.33) becomes more complicated and requires, as a rule, the use of numerical integration methods. 2 There also exist approximate analytic methods for calculating the matrices P(t) [ I , 72]. However, for matrices P(t) of larger dimensions, these methods meet serious
computational difficulties.
102
Chapter II
2.1.3. The results obtained by solving the basic linear-quadratic problem (2.1.23), (2.1.24) can readily be generalized to more general statements
of the optimal control problem. Here we only list the basic lines of these generalizations; for a detailed discussion of this subject see [1, 5, 34, 58, 72,
122, 132]. First of all, note that the synthesis problem (2.1.23), (2.1.24) admits an exact solution even if there are noises in the feedback circuit, that is, if
instead of exact values of the phase variables x(i), the controller C (see Fig. 13) receives distorted information of the form
x ( t ) = N(t)x(t) +
(2.1.34)
where N(t) and
process of the white noise type (1.1.34) or a Gaussian Markov process. In this case, the optimal control algorithm coincides with (2.1.30) in which,
instead of the true values of the current phase vector x = x ( t ) , we use the vector of current estimates m — m(i) of the phase vector. These estimates
are formed with the help of Eqs. (1.5.53) for the Kalman filter, which with regard to the notation in (2.1.23) and (2.1.34) have the form 3
m = [A(t) - B(t)H-i(t)BT(t)P(t)}m + DJV^M^Wr 1 ^) - N(t)m), (2.1.35) T
T
D =
well-known separation theorem [58, 193]. The next generalization of the linear-quadratic problem (2.1.23), (2.1.24) is related to a more general model of the plant. Suppose that, in addition to additive noises (,(t), the plant P is subject to perturbations depending on the state x and control u and to pulsed random actions with Poisson distribution of the pulse moments. It is assumed that the behavior of the plant P is described by the special equation
x = A^x+B^u+o-^x^+o-^u&^+a^^+az^Ott), 3
(2.1.37)
Equations (2.1.35) and (2.1.36) correspond to the case in which 77(4) in (2.1.34) is a
white noise.
Exact Methods for Synthesis Problems
103
FIG. 16 where £i(t) and £2^) are scalar Gaussian white noises (1.1.31), 9(t) is an I-vector of independent Poisson processes with intensity coefficients A,- (i = !,...,£), <TI, 0*2, and 173 are given n x n, n x r, and n x I matrices, and
the other variables have the same meaning as in (2.1.23). For the exact solution of problem (2.1.37), (2.1.24), see [34]. We also note that sufficiently effective methods have been developed for infinitely dimensional linear-quadratic problems of optimal control if the
plant P is either a linear dynamic system with distributed parameters or a quantum-mechanical system. Results corresponding to control of distributed parameter systems can be found in [118, 130, 164, 182] and to control of quantum systems in [12, 13]. All linear-quadratic problems of optimal control, treated examples, are characterized by the fact that isfying the Bellman equation is of quadratic form (a and the optimal control law is a linear function (a
as well as the abovethe loss function satquadratic functional) linear operator) with
respect to the phase variables (the state function). To solve the Bellman equation becomes much more difficult if it is necessary to take into account some restrictions on the domain of admissible control values in the design of an optimal system. In this case, exact analytical results can be obtained, as a rule, for one-dimensional synthesis problems (or for problems reducible to one-dimensional problems). Some of such problems are considered in the following sections of this chapter. §2.2. Problem of optimal tracking a wandering coordinate
Let the input (command) signal y ( t ) in the servomechanism shown in Fig. 2 be a scalar Markov process with known characteristics, and let the plant P be a servomotor whose speed is bounded and whose behavior is
104
Chapter II
described by the scalar deterministic equation
x-u,
\u(t)\
(2.2.1)
(here um determines the admissible range of the motor speed, — um < x < um). Equation (2.2.1) adequately describes the dynamics of a constant
current motor controlled by the voltage on the motor armature under the assumption that the moment of inertia and the inductance of the armature
winding are small [2, 50]. We shall show that various synthesis problems stated in §1.4 can be solved for such servomechanisms. 2.2.1. Let y(t] be a diffusion Markov process with constant drift a and
diffusion B coefficients. We need to calculate the controller C (see Fig. 2) that minimizes the integral optimality criterion f
c (x(t),y(t))dt,
(2.2.2)
Jo
where c(x, y) is a given penalty function.
By setting A* = a, B« = B, ax = u, and Bx = 0 in (1.4.21), we readily obtain the following Bellman equation for problem (2.2.1), (2.2.2):
dF dt
dF dy
Bd2F 2 dy
.
.
. r dF] «-— = 0. |«|<« TO L dx J
— — + a — — + - — —2 - + c z,y + mm
2.2.3
We shall consider the penalty functions c(x, y) depending only on the error signal, that is, on the difference z = y — x between the command input y and the controlled variable x. Obviously, in this case, the loss function F(t,x,y) = F(t,y—x) = F(t,z) in (2.2.3) also depends only on z. Instead of (2.2.3), we have
dF at
dF dz
Bd2F 2 dz
r dFi - u—— \= 0. l«l<«™ L dz J
-5T + a1T- + TlTir 2 + c(z + mm
(2.2.4)
The minimum value of the function in the square brackets in (2.2.4) can be obtained by using the control4
ut — um sign f —— J , 4
(2.2.5)
In (2.2.5) sign
sign
Exact Methods for Synthesis Problems
105
which requires to switch the servomotor speed instantly from one admissible limit value to the opposite value when the derivative dF(t, z)/dz of the loss function changes its sign. Control of the form (2.2.5) is naturally called control of relay type (sometimes, this control is called "bang-ban
Substituting (2.2.5), instead of u, into (2.2.4) and omitting the symbol "min", we reduce Eq. (2.2.4) to the form
3F dF B d2F 8F -57- +0-5- + -^-^+ c(z) -Urn -£- = 0 . dt dz 2 dz2 dz
(2.2.6)
In [113, 124] it was shown that in the strip UT = {0 < t < T, -oo < z < 00} Eq. (2.2.6) has a unique solution F(t,z) satisfying the additional condition F(T, z) = 0 if the penalty function c(z) is continuous and does not grow too rapidly as z\ —>• oo.5 In this case, F(t, z) is a function twice continuously differentiable with respect to z and once with respect to t. In particular, since dF/dz is continuous, the condition
—— ( t , z ) = Q
(2.2.7)
must be satisfied at the moment of switching the controlling action. If c(z) > 0 attains its single minimum at the point z = 0 and does not decrease monotonically as z| —> oo, then Eq. (2.2.7) has a single root z f ( t ) for each t. This root determines the switch point of control. On
different sides of the switch point the derivative dF/dz has opposite signs. If dF/dz > 0 for z > z p (<) and dF/dz < 0 for z < zr(t), then we can write the optimal control (2.2.5) in the form
w»(t,z)=umsign(z-zr(t)).
(2.2.8)
Thus, the synthesis problem is reduced to finding the switch point z (t}. To this end, we need to solve Eq. (2.2.6).
Equation (2.2.6) has an exact solution if we consider the stationary tracking. In this case, the terminal time (the upper limit of integration in (2.2.2)) T —>• oo, and Eq. (2.2.6) for the time-invariant loss function (see (1.4.29)) •t)]
(2.2.9)
becomes the ordinary differential equation
B d2f ^d^
df +a
dz~U"
+ c(z)=
(2.2.10)
5 More precisely, the condition that there exist positive constants A\, A?, and a such that 0 < c(z) < A\ + A? z\a for all z implies constraints on the growth of the
function c(z).
106
Chapter II
which can be solved by the matching method [113, 171, 172]. Let us show how to do this. Obviously, the nonlinear equation (2.2.10) is equivalent to two linear equations
f^+ <-->£+•<•>=* »>v £>»• *
l
(2211)
-
for the functions f i ( z ) and f i ( z ) that determine the function f ( z ) on each side of the switch point z p . The unique solutions to linear equations (2.2.11) are determined by the behavior of /i and /2 as \z\ —>• oo. It follows from the statement of the problem that if we take into account the diffusion "divergence" of the trajectories z(t) for large \z , then we only obtain small corrections to the value of the optimality criterion and, in the limit as \z\ —)• oo, the loss functions f i ( z ) and /2(z) must behave just as the solutions to Eqs. (2.2.11) with B = Q. The corresponding solutions of Eqs. (2.2.11) have the form =
z
2 f°°
-fi JIz B
"' ""'
I c ( z ) - f]
2 [*
dz
„ = —S B
r
ex
\
2K.-a)
1
(2 2 12
L z P \_ - - J —5—— ( ~ z) \ B
>-~+.>r }]f
-- '
\
i i —-r- J t c ( z )-7]exp
i -\~ /
J-oo
„
t-2
Z ) | UZ.
L
According to (2.2.7), we have the following relations at the switch point z p :
~(zr) = -^-(zr) = 0.
(2.2.13)
Substituting (2.2.12) into (2.2.13), considering (2.2.13) as a system of equations with respect to two unknown variables z p and -y, and performing some simple transformations, we obtain the equation for the switch point: /
/ Jo
\cI z -\- ________ ) — r\ ? — ________ I \e~v rlit — fl
O 9 14^ ^Z.Z.ltj
and the expression for the stationary tracking error
^ f zZ +7 - r cC
"Jo
Bp
}c-dvr dv
( r 2(um-a))e
-J0
Cc(z Z
( r
BV
}c-*dv dv
2(um+a))e
' (2.2.15)
Exact Methods for Synthesis Problems
107
To obtain explicit formulas for the switch points and stationary errors, it is necessary to choose some special penalty functions c(z). For example, for the quadratic penalty function c(z) = z 2 from (2.2.14), (2.2.15), we have 5^-j,
(2-2.16)
If c(z) — z|, then we have /
n
\
(2.2.18)
,. 2(um-a) ™ + a ) l n 1 + ——
+um-a
(2.2.19) It should be noted that formulas (2.2.16)-(2.2.19) make sense only under the condition um > a. This is due to the fact that the stationary operating mode in the problem considered may exist only for um > a. Otherwise, (for a > um), the mean rate of increase in the command signal y(t) is larger
than the limit admissible rate of change in the output variable x(t), and the error signal z ( t ) = y(t) — x ( t ) is infinitely growing in time. If the switch point zp is found, then we know how to control the servomotor P under the stationary operating conditions. In this case, according to (2.2.8), the optimal control has the form «„ (z(t)) = um sign ( z ( t ) - z r ) ,
(2.2.20)
and hence, the block diagram of the optimal servomechanism has the form
shown in Fig. 17. The optimal system shown in Fig. 17 differs from the optimal systems considered in the preceding section by the presence of an essentially nonlin-
ear ideal-relay-type element in the feedback circuit. The other distinction between the system in Fig. 17 and the optimal linear systems considered in §2.1 is that the control method depends on the diffusion coefficient B
of the input stochastic process (in §2.1, the optimal control is independent of the diffusion coefficients,6 and therefore, the block diagrams of optimal deterministic and stochastic systems coincide).
If B = 0 (the deterministic case), then it follows from (2.2.16)-(2.2.19) that the switch point zf — 0 and the stationary tracking error 7 = 0. These 6
This takes place if the current values of the state vector x(t) are measured exactly.
Chapter II
108
FIG. 17 results readily follow from the statement of the problem; to obtain these results it is not necessary to use the dynamic programming method. Indeed,
if at some instant of time we have y(t] > x(t) (z(t) > 0), then, obviously, it is necessary to increase x at the maximum rate (that is, at u = +um) till the equality y = x (z = 0) is attained. Then the motor can be stopped. In
a similar way, for y < x (z < 0), the control u — —um is switched on and operates till y becomes equal to x. After y = x is attained and the motor is stopped, the zero error z remains constant, since there are no random actions to take the system out of the state z = 0. Therefore, the stationary tracking "error" is zero.7 If the diffusion is taken into account, then the optimal deterministic
control u*et = um sign z is not optimal. This fact can be explained as follows. Let u — um signz, and let 5 ^ 0 . Then the following two factors affect the trajectories z(t): they regularly move downwards with velocity (um —a) for z > 0 and upwards with velocity (um + a) for z < 0 due to the drift a and control u (see Fig. 18), and they "spread" due to the diffusion B that is the same for all z. As a result, the stochastic process z(t) becomes
stationary (since the regular displacement towards the i-axis is proportional to t and the diffusion spreading away from the i-axis is proportional to \/t) and all sample paths of z ( t ) are localized in a strip of finite width containing the t-axis.8 However, since the "returning" velocities in the upper and lower
half-planes are different, the stationary trajectories of z ( t ) are arranged not It is assumed that the penalty function c(z) attains its minimum value at z = 0 and c(0) = 0. 8 More precisely: if 2(0) = 0, then with probability 1 the values of z(t) lie in a strip of finite width for all t > 0.
Exact Methods for Synthesis Problems
109
z(t)
FIG. 18 symmetrically with respect to the line z = 0, as is conventionally shown in Fig. 19. If the penalty function c(z) is an even function (c(z) — c(—z)), then, obviously, the stationary tracking error 7 = Ec(z) (see (1.4.32)) can be decreased by placing the strip AB (where the trajectories are localized) symmetrically with respect to the axis z = 0. This effect can be reached
by switching the control u at some negative value zr rather than at z = 0. The exact position of the switch point zr is determined by formulas (2.2.14), (2.2.16), and (22.2.18).
FIG. 19 In conclusion, we note that all results obtained in this section can readily be generalized to the case where the plant P is subject to additive noncon-
trolled perturbations of the white noise type (see Fig. 10). In this case,
110
Chapter II
instead of Eq. (2.2.1), we have
um,
(2.2.21)
where £(t) is the standard white noise (1.1.31) independent of the input process y(t) and N > 0 is a given number. In this case, the Bellman equation (2.2.3) acquires the form
dF - ^ dt
dF — dy
Bd2F 2 dy22
Nd2F 2 dx2o
.
i
.
.
mm
r dF
|«|<«
and instead of (2.2.4), we obtain
dF -7 at
dF -5az
B + Nd2F 2 dz2
. .
. r dFi A mm - «—— = 0. l«l<«m L oz
This equation differs from (2.2.4) only by a coefficient of the diffusion term. Therefore, all results obtained for systems whose block diagram is shown in Fig. 2 and whose plant is described by Eq. (2.2.1) are automatically
valid for systems in Fig. 10 with Eq. (2.2.21) if in the original problem the diffusion coefficient B is replaced by B + N. In particular, if noises in the plant are taken into account, then formulas (2.2.16) and (2.2.17) for the stationary switch point and the stationary tracking error take the form Sf =
_(B + N)a ~
2
Note also that the problem studied in this section is equivalent to the synthesis problem for servomechanism tracking a Wiener process of intensity B with nonsymmetric constraints on admissible controls — um + a <
u < um + a, since both these problems have the same Bellman equation
(2.2.4). 2.2.2. Now let us consider the synthesis problem that differs from the problem considered in the preceding section only by the optimality criterion. We assume that there is an admissible domain [^1,^2] for the error
z(i) = y(t) - x(t) (ti and £2 are given numbers such that li < I?). We assume that if z(t) leaves this domain, then serious undesirable effects may occur. For example, the system considered or a part of any other more
complicated system containing our system may be destroyed. In this case,
Exact Methods for Synthesis Problems
111
it is natural to look for controls that keep z(t] within the admissible limits for the maximum possible time. General problems of calculating the maximum mean time of the first passage to the boundary were considered in §1.4. In particular, the Bellman equation (1.4.40) was obtained. In the scalar case studied here, this equation has the form
Bd2Fj
S=-1
dj\
~2~dv^ + a~dy~^ H
(2-2-24)
(Eq. (2.2.24) follows from (1.4.40), (1.4.31), since A« = a, Ax = u, BV = B, Bx = 0). Recall that the function Fi(x,y) in (2.2.24) is equal to the maximum mean time of the first passage to the boundary of the domain of admissible phase variables if the initial state of the system is (x, y). In the case where the domain of admissible values (x, y) is determined by the error signal z = y — x, the function FI depends only on the difference FI(X, y) = Fi(y — x) — F\(z] and, instead of the partial differential equation (2.2.24), we have the following ordinary differential equation for the function FI(Z): „ , 22 f a^+ max - «^ =-1. 2 dz dz |«|<«™ L dz J
(2.2.25)
The function -Fi(z) satisfies Eq. (2.2.25) at the interior points of the domain [£1,^2] °f admissible errors z. At the boundary points of this domain, FI vanishes (see (1.4.41)):
The optimal system can be synthesized by solving Eq. (2.2.25) with the boundary conditions (2.2.26). Just as in the preceding section, one can see that the optimal control u f ( z ) is of relay type and is equal to
ux(z) = -um sign (~rL)•
(2.2.27)
Using (2.2.27), we transform Eq. (2.2.25) to the form
dz
(2.2.28)
The condition of smooth matching (see [113], p. 52) implies that the solution -Fi(z) of Eq. (2.2.28) and the derivatives dFi/dz and d2Fi/dz2 are
Chapter II
112
FIG. 20 continuous everywhere in the interior of [li z1 is determined by the condition
dz
\. Therefore, the switch point
= 0.
(2.2.29)
The same continuity conditions and the boundary conditions (2.2.26), as well as the "physical" meaning of the function F±(z), a priori allow us to estimate the qualitative behavior of the functional dependence -Fi(z). The corresponding curve is shown in Fig. 20. It follows from (2.2.29) that the switch point corresponds to the maximum value of FI(Z). In this case, F ( ( z ) < 0 for z > z1, and F[(z] > 0 for z < z1. In particular, this implies that the optimal control (2.2.27) can be written in the form
u*(z) = -u r a sign(z-zj), which is similar to (2.2.20) and differs only by the position of the switch point. Thus, in this case, if the applied constant displacement —zr is replaced by — z1, then the block diagram of the optimal system coincides with that in Fig. 15. The switch point z1 can be found by solving Eq. (2.2.28) with the boundary conditions (2.2.26). Just as in the preceding section, we replace the nonlinear equation (2.2.28) by the following pair of linear equations for the
Exact Methods for Synthesis Problems
113
function Ff(z), zl < z < £ 2 > and the function F^(z), lj_ < z < z^:
Bd2F+ t .dF+ V^H~+ (a-Wm)-7^ = -l, 2 2 dz dz Bd2F~ .dFr , r
1
z1
Y^- + (a + U ™ ) ^ = -1'
, 2>2>30)
^<*<^
The required switch point z1 can be obtained from the matching conditions for F f ( z ) and F^(z). Since FI(Z) is twice continuously differentiable, it follows from (2.2.27) that these conditions have the form
F?(*i) = fT(*1), v
dz
(2-2-31)
dz
r'
The boundary conditions (2.2.26) and (2.2.32) for F?(z) and J'f(z) imply ,-.0. /
\
^ — <-2
um - a
-^
2(um - a)2 {
exp
-
I B v2(um -a)(z-z1}
— exp
B j.j
n (-1
—— 2
um + a
s
f\ 1 . .
i
.
W
.
' \'
(2.2.33)
It
I
\
'
IJ
2(um + a) 2 |_
*L
B -a)(z-zl
By using (2.2.33) and the continuity condition (2.2.31), we obtain the following transcendental equation for the required point z1:
2umz* = (um + a)£2 + (um - a)li
™
— ,
B\um2 I um + .i._. 4- n.
r
9
.
i 1
(2.2.34)
um — a
In the simple special case a = 0, it follows from (2.2.34) that i
4+4
114
Chapter II
that is, the switch point is the midpoint of the interval of admissible errors z. This natural result can be predicted without solving the Bellman equation. In the other special case where —li — I? = t (I > 0) and a
Zr
oB
~ <
at r i + exp(2Mro£/5)1 +
^ [l-exp(2uml/B)\ '
To find z1 in the other cases, it is necessary to solve the transcendental equation (2.2.34). 2.2.3. Assume that the performance of the servomechanism shown in Fig. 2 is determined by the maximum error z(i] = y(z) ~ x(t] on a fixed
time interval 0 < t < T. Then it is natural to minimize the optimality criterion
/[«] = E max z ( t ) \ = E max \y(t) - x(t)|, 0
0
(2.2.35)
which is a special case of the criterion (1.1.18). For convenience, we shall
use the modification (1.4.48) of the criterion (1.1.18), that is, instead of (2.2.35), we shall minimize I[u] = Emax z(r)| e -^ r -*>.
(2.2.36)
T>t
The parameter (3 > 0 determines the observation time for the stochastic process z(r). We assume that the criteria (2.2.35) and (2.2.36) are equivalent if the terminal time T and the variable (3 are matched, for example, as follows: T = c//3, where c > 0 is a constant. The Bellman equation for the problem considered can be obtained from (1.4.51) with regard to the relation /2(#,y) = f z ( y — x) — /2(z). This equation has the form
min \-u~ =/?/ PJ22 , «|<« m L dz\ '
i fJ/ 2 ( z ) > | z , ^ ' ' '
(2.2.37)
otherwise. Just as in the preceding sections, after the expression in the square brackets is minimized, Eq. (2.2.37) acquires the form = /?/2,
if/2(*)>|*,
(2.2.38)
otherwise. In this case, just as in the preceding sections, the optimal control w*(z) is
of relay type and can be written in the form (2.2.20). The only distinction
Exact Methods for Synthesis Problems
115
is that, in general, the switch point z 2 differs from z p and z1. The point z 2 can be found by solving Eq. (2.2.38). Solving Eq. (2.2.38), we shall distinguish two domains on the z-axis: the domain Z-± where f2(z] > \z\ and the domain Z2 where f2(z) = \z\.
Obviously, if f2(z*) = \z*\ for any z*, then / 2 (z) = z| for any z such that \z\ > z* . In other words, the domain Z2 consists of two infinite intervals
(—oo, z') and (z",+oo). In the domain Z\ lying between the boundary points z' < 0 and z" > 0, we have
Bd2f2 2
2 dz
,
d/2
+ a—— - un
dz
d/2 = /3f2. dz
(2.2.39)
Next, the interval [z', z"] is divided by the switch point z 2 into the following two parts: the interval z' < z < z 2 where Eq. (2.2.39) takes the form
=0
(2-2.40)
and zr2 — < z< z" where —
Thus, in this case, we have seven unknown variables: z', z", z 2 , and the four constants obtained by integrating Eqs. (2.2.40) and (2.2.41). They can be obtained from the following seven conditions:
, c 2} I\ Z / I>, r
( =^ df2
.
2.
_
—;—— V I Z ;I —
/2
(2.2.42) 2
_
—— V ( Z ;I — U .
Formulas (2.2.42) are smooth matching conditions for the solutions f2 (z) and f i ~ ( z ) .
The last three conditions show that the solutions and their
first-order derivatives are continuous at the switch point z 2 (see (2.2.31) and (2.2.32)). The first four conditions show that the solutions and their first-order derivatives are continuous at the boundary points of Z\. By solving (2.2.40) and (2.2.41) with regard to (2.2.42), we readily obtain
116
Chapter II
the following three equations for z', z", and z2: 1
A2{exp[-(A1 + A 2 ) ( Z 2 _ z ' ) ] _ 1
' ~ ^ " ~ l; 1
(2.2.44)
1
-
_e
(2.2.45)
In (2.2.43)-(2.2.45) we have used the notation
= •= um - a + v/(wm - a) 2 + 2/3B , a 1
1 o \
uro + "^("m - a) 2 + 2/3B
The desired switch point z2 can be found by solving the system of transcendental equations (2.2.43)-(2.2.45). Usually, this system can be solved by numerical methods. One can obtain the explicit expression for z1 from
Eqs. (2.2.43)-(2.2.45) only if the problem is symmetric, that is, if a — 0. In this case, the domain Zi is symmetric about the origin, z' = —z", and we have the switch point z2 — 0. However, this is a rather trivial case and of no practical interest.
REMARK. It should be noted that the optimal systems considered in Sections 2.2.2 and 2.2.3 are very close to each other (the switch points nearly coincide, zj Kt z2) if the corresponding parameters of the problem agree well with each other. These parameters can be made consistent in
the following way. Assume that the same parameters a, B, and um are given in problems of Sections 2.2.2 and 2.2.3. Then, choosing a value of
Exact Methods for Synthesis Problems
117
the parameter /3, we can calculate three numbers z' = z'(/3), z" = z"(/3), and z2 — z2((3) in dependence on the choice of (3. Now if we use z' and z" as the boundary values of admissible errors (li = z' (/3), ti — z" (/3)) in the problem considered in Section 2.2.2, then by solving Eq. (2.2.34), we obtain the coordinate of the switch point z* and show that z^((3) tel z^(f3)9 for (3 varying from 1.0 to 10~4. This is confirmed by the numerical experiment
described in [92]. Moreover, in [92] these values of the parameter /3.
it is shown that Fi(z^(f3)}
« (3'1 for
D
2.2.4. Now let us consider the synthesis problem of optimal tracking a discontinuous Markov process. Let us assume that the input process y(t) in the problem of Section 2.2.1 is a pure discontinuous Markov process. As shown in §1.1, such processes are completely characterized by the intensity
\(y) of jumps and the density function ir(y,y') describing the transition probabilities at the jump moments. The one-dimensional density p(t, y) of this process satisfies the Feller equation (see (1.1.71))
dp(t, y) - A(j/)p(t, y) + I A(Z)TT(Z, y)p(t, z) dz = 0. dt
(2.2.46)
From (1.4.61) with regard to (2.2.1) and (2.2.2), we obtain the Bellman equation
ir(y,z)F3(t,x,z)dz-F3(t,x,y) + c(x,y)+
min u - ( t , x, y) = 0. l«l<« m L ox J
(2.2.47)
If we denote the integro-differential operator of Eq. (2.2.46) by L f t y , then this equations can be written in the short form
Lt,yP(t,y)=Q.
(2.2.48)
Comparing Eqs. (2.2.46) and (2.2.47) with the Feller equations (1.1.69) and (1.1.70), we see that, for pure discontinuous processes, the Bellman equation (2.2.47) contains the integro-differential operator L^y of the backward Feller equation; this operator is dual to Ltiy. Therefore, Eq. (2.2.47) can be written in the form , x, y) + c(x, y) + min \u~(t, x, y)} = 0. [«|<« m [_ 9
OX
(2.2.49)
J
The approximate relation z^(&) « z*(P) means that \z*(0)-z^(P)\ < z"(/3)-z'(/3).
118
Chapter II
In what follows, we assume that the input Markov process y(t) is homogeneous with respect to the state variable y, that is, A(y) = A = const and •x(y,y') = n(y — y'). In this case, by using the formal method proposed in [176], we can replace the integro-differential operator L^>y in (2.2.47) and
(2.2.49) by an equivalent differential operator. Let us show how to do this. First, we try to write Eqs. (2.2.46) and (2.2.48) in the form
(2 2 50)
--
where L is the required differential operator and Lp is the density of the probability flow [160, 173]. We apply the Fourier transform to (2.2.50) and
(2.2.46). For the Fourier transform of the probability density oo
/
esyp(t,y)dy,
s = u,
=\
•oo
we obtain the following two equations from the well-known property of the Fourier transform of the convolution of two functions:10
),
(2.2.51)
s)-l),
(2.2.52)
where 7f(s) denotes
f°° s W(s) = I e ^(y}dy.
(2.2.53)
J — oo
Comparing (2.2.51) and (2.2.52), we obtain the spectral representation of the desired operator L(8)
= A^" 1 .
(2.2.54)
s
If the expression on the right-hand side of (2.2.54) is the ratio of polynomials 0
=
Q(s)
l
...
m
qo + qis + - - - + qnsn
V
'
(hi and 5,- are constant numbers), then, as follows from the theory of Fourier
transforms [41], the desired operator L(d / dy) can be obtained from L ( s ) 10
RecaII that A(y) = A and ir(z,y) = Tr(y - z) in (2.2.4
Exact Methods for Synthesis Problems
119
by the formal change s —>• d/dy. Using the operator L(d/dy), we transform the Bellman equation (2.2.49) to the form
(note that if L(d/dy) = H(d/dy)/Q(d/dy), then the Li/> = = Q(d/dy)
!?- =i-c(z)
dz
(2.2.57)
(here the time-invariant loss function /s = fs(z) is determined by analogy with (2.2.9) and 7 is the stationary error.) The optimal control (2.2.58) 3 differs from (2.2.20) only by the^ position puoiuiuii of ui the oiic switch owiu^ii point puniu z 7 , which can
be obtained from the condition dz
v r/
.
To complete the solution of the synthesis problem, we need to calculate z3. By analogy with Section 2.2.1, the switch point z3 divides the z-axis into two parts: the domain z > z3, where df$/dz > 0, it* = um, and
Eq. (2.2.57) has the form
and the domain z < z 3 , where df^/dz the form
w
/ A \
< 0, u* = um, and Eq. (2.2.57) has
AtF + um -^- = 7 - c ( z ) .
(2.2.60)
At the switch point z3 the derivatives of the functions / + (z) and f ~ ( z ) vanish:
120
Chapter II
To solve the linear equations (2.2.59) and (2.2.60) explicitly, we need to specify the form of the linear operator L(d/dz). Assume that the density of the transition probability Tr(y,y') at the jump moment is given by the formula f
1I exp(-fc / L 2 |y I / -2/1)n ffor // y. Calculating the integral (2.2.53), we obtain
^(s) = 71 IT-•——— fc2 —— + Afc s - s22 '
(2.2.63)
A* = fc2
After the change s —> d/dz, we obtain the following expression for the operator L(d/dz) from (2.2.63) and (2.2.54):
(2.2.64) With regard to (2.2.64), we can write Eqs. (2.2.59) and (2.2.60) in the form x(_d__
dz
*,\df£ I dz
/,,
.. d
Introducing the functions
we transform the system (2.2.65) as follows:
=
um ) dz
,
where •U, 7rn/ 1[_
(U,^. 1 "7
(17 U,/(,
JI
Relations (2.2.61) and (2.2.66) imply the following matching conditions for the functions
(k2um + \&k)p+(z3r) = (-k22um + \&k)
(2.2.69)
Exact Methods for Synthesis Problems
121
The characteristic equations corresponding to Eqs. (2.2.67) are
umj
\
um J
=0.
(2.2.70)
By fj,^ and ^ we denote the roots of the characteristic equation for the function (p+ (z) (correspondingly, by /j^ and /j,^ the roots of the characteristic equation for (f>~(z)). A straightforward verification shows that if
A
(2.2.71)
then (1) all roots /uf )2 are real,
(2) each characteristic equation in (2.2.70) has roots of opposite signs (for definiteness, in what follows, we assume that ^ and /j,^ are positive, and respectively, fj,^ < 0). REMARK. Note that condition (2.2.71) must be satisfied, since this is the existence condition for the stationary tracking considered here. Indeed, the expression on the left-hand side in (2.2.71) is equal to the absolute value of the mean rate of the regular displacement in the command signal y(t) caused by random jumps. Obviously, this rate of change cannot exceed the limit admissible speed of the servomotor. Inequality (2.2.71) just confirms
this fact.
D
Taking into account these properties of the roots of the characteristic equations (2.2.70), one can readily see that the solutions of Eqs. (2.2.67) have the form ±
(z)=
X
f r
"?('-*') ± ( '}
'
~ (/if - / 4 ) L / f o o 6 fZ
±, ^ a. 1 ei*2(*-*)c±(z')dz'\.
(2.2.72)
Using (2.2.72), from the matching conditions (2.2.69), we obtain the following equation for the required switch point z3:
(k2um + AAfc) I" r° elt+yc+(z3_ ZT 04 ~ / 4 ) [J-oo6
}
y
, y
f°° rf9+(3_ ZT Jo
] \
e^yc~(z3r~y)dy\.
(2.2.73)
122
Chapter II
For the quadratic penalty function c(z) = z 2 , Eq. (2.2.73) has an exact solution. Actually, taking into account (2.2.68), we can rewrite Eq. (2.2.73) in the form
t-y-
Oh - A*2~)
f°° M~»/o
L./-OO
>
e"*»c(y) dy + / Jo
1 J
e"^c(t/) dy] = 0, J
(2.2.74)
where
c(y) = co + cl2/ + c 2 y 2 ,
CQ = fc 2 (z 3 ) 2 + 2Afcz 3 - 2,
{ 2i.jL. i 0 1
Calculating the integrals in (2.2.74), we obtain
(-k2um+XAk)\
/I
1N
/
c
2
T —:—T———XT— Ko ~
/ / ' I
— ^2 J
L
v \
^i
1
~
2
i *J '
1
4
//~/
,
-
\ ( III ' 1 "
A*2
1/^
Since /Lt^"2 and /z^ 2 satisfy the quadratic equations (2.2.70), after easily transformation, we obtain the following explicit formula for the switch point: /o n
v • •«
Using (2.2.69), (2.2.72), and (2.2.77), we can readily calculate the stationary specific error 32
r
mAk
+ X)
2Afc
3
2
~ ~ ~ ~
.
(2.2.78)
Exact Methods for Synthesis Problems
123
If instead of condition (2.2.71), we have a stronger inequality
A
F~~ I
then we can substantially simplify (2.2.77) and (2.2.78) by expanding these expressions in power series in the small parameter e = XAk/umk2 and taking only the leading terms of these expansions. In this case, instead of (2.2.77) and (2.2.78), we have Afc/2 jfe2
1
'
7 - K3)2-
(
J
(2-2.80)
For the first time, formulas (2.2.79) and (2.2.80) were derived by somewhat different methods in [176]. §2.3.
Optimal control of the population size
Numerous investigations deal with the dynamics of animal and microorganism populations and control of the population size. For example, an
extensive literature on this subject can be found in [51, 73, 87, 89, 133, 142, 186, 187, 189]. Various mathematical evolutional models depending on the environmental conditions of biological populations are used for describing the variations in the population size. We begin with a brief review of such models. The main attention is given to the models that we shall consider in this book later. 2.3.1. Models describing the population dynamics. Apparently, Malthus was the first who considered the following model for the population
dynamics in 1798: J = az.
(2.3.1)
Here x = x ( t ) is the population size11 at time t, and the constant number a, called the growth factor, is determined as the difference between the birthrate and the death-rate factors. If the birth-rate is larger than the death-rate (a > 0), then, according to the Malthus model (2.3.1), the population size must grow infinitely. llr
The variable x is assumed to be continuous, although the number of individuals in
the population can be only integer. However, if the number of individuals is sufficiently large, then the continuous model (2.3.1) can be used. In this case, the variable x is treated as the population density, that is, as the number of individuals per unit area (or volume) of the population habitat.
124
Chapter II
Usually, this result is not confirmed, which shows that the model (2.3.1) is not perfect. Nevertheless, the basic idea of this model, namely, the assumption that the rate of the population variation is proportional to the current population size proved to be very fruitful. Many more realistic
models were constructed on this basis by introducing the corresponding corrections to the growth factor a. So, for example, if we assume that in (2.3.1) the growth factor a depends
on the population size x as = a(x) = -fj,
ln(x/K)
/
x\
or a = a(x) = r (I - — J ,
then we obtain the Gompertz model (1825)
dx
* - '^ or the Verhulst model (1838)
Equation (2.3.3) is often called the logistic equation. The positive constants r and K are usually called the natural growth factor and the capacity of the medium. Models for more complicated systems of interacting populations are also based on the Malthus model (2.3.1). Assume that in the same habitat there are two different populations of sizes xi and x%, respectively. Let each of these populations be described by the Malthus type equations
at
,
at
= bx2.
, . (2.3.4)
Now we assume that individuals in the second population (predators) can exist only if they eat individuals (prey) from the first population.12 In this case, it is natural to assume that the growth factors a and b in (2.3.4) have the form
12
This model is usually illustrated by the assumption that x\ denotes a community
of hares bares and X? a community of wolves. Hares need vegetable food, and wolves feed on hares •es (and (and only onlv on hares). hares)
Exact Methods for Synthesis Problems
125
Thus, we arrive at the two equations
dxi
—-— = (ai — 02^2)*!,
at
dx2
—;— = ( — t>i + ozxzjx^
at
,
,
(2.3.5)
which are well-known Lotka-Volterra equations that model the behavior of the simplest system of interacting populations, the "predator-prey" model. These equations were studied in detail by V. Volterra [187], who found many remarkable properties of their solutions.
The multidimensional generalization of the Lotka-Volterra model has the form
•fcr
/.
^. .. V
r=i)2,...,n.
(2.3.6)
The dynamics of system (2.3.6) depends on the form of the matrix A = \\aij\\™. If this ma trix is antisymmetric, i.e., if a,-j = — oj,- and an = 0, then Eq. (2.3.6) describes a conservative model for the population interaction. If the quadratic form arsxrxs is positive definite, then model (2.3.6) is called dissipative. Further generalizations of models for the population dynamics were related to more detailed descriptions of the interaction between individuals in the population. For example, in many actual situations, the growth factor a depends on the population size at some preceding moment of time rather than on the current population size. In these cases, it is expedient to use the Hutchinson model (1948) M\
h > 0,
(2.3.7)
which is a generalization of the logistic model (2.3.3). In 1976 Gushing proposed the following more general model, in which both discrete and distributed delays are taken into account: f
x(t)=rx(t)\l-
I
f00
Jo
1
x(t-s)dK(s)\,
1
t > 0,
(2.3.8)
where K(s) is a nondecreasing bounded function and the integral on the right-hand side is of the Stieltjes type. In some special cases, it is necessary to take into account the spatial distribution of the population size. In these cases, the state of the population is described by the density function D(t,x,y] at the point (x,y). If the movement of individuals within the habitat is a diffusion process, then instead of (2.3.7), we have the Hutchinson model with diffusion —— (ft x y } =
dt
' '>
126
Chapter II
Equations (2.3.1)-(2.3.9) model the behavior of isolated biological communities (autonomous models). If there are external actions on the population, then some additional terms responsible for these actions appear on the right-hand sides of Eqs. (2.3.1)-(2.3.9) As usual, we distinguish two
types of external actions: purposeful controlled actions, which can be used to control the population size, and noncontrolled random perturbations. Let us consider a population described by the model (2.3.3). If there are external actions, say, some individuals are taken away from the population, then we obtain the controlled logistic model x = r
/ x\ (l-K — \x-qux, V /
(2.3.10)
where the function u = u(t) > 0 is the intensity of the catching process and
the number q > 0 is the catchability coefficient. In this case, the value
ft,
Q=q
u(t)x(t)dt
(2.3.11)
•>«!
gives the number of individuals caught during the time interval [ti,^]-13 In a similar way, the Lotka-Volterra equations can be generalized to the following controlled system:
xi — (ai - a2x2)xi - qiuixi, •
i
X2 = ( — Oi +
i
\
(Z.O.J-.ZJ
Here the control functions «i = ui(t) > 0 and u^ = u-2(t) > 0 are, respectively, the intensities of catching prey and predators. If individuals are removed only from one population, then we have u\(t) = 0 or U2(t) = 0 in
(2.3.12). If the population behavior is substantially influenced by noncontrolled
random perturbations, then the dynamics of the population is described by stochastic differential equations. For example, in some problems, the population behavior can be satisfactory described by the stochastic logistic model
x = r 1- K- x-qux + Wx£(t), (2.3.13) V J where £ ( t ) is the scalar Gaussian white noise (1.1.31) and the number B > 0 determines the intensity of random perturbations. Many other stochastic models used to describe the dynamics of various biological systems can be found in [51, 83]. 13
Note that Eq. (2.3.10) can be used not only in the case of "mechanical" removal
(catching, shooting, etc.) but also in the case where the population size is controlled by treating the habitat with chemical agents.
Exact Methods for Synthesis Problems
127
2.3.2. Statement of the problem. In this section we consider the optimal control of the size of a population described by the controlled logistic model (2.3.10). The statement of this problem is borrowed from the books [35, 68], where this problem is formulated in conformity to the problem of fisheries management.
We shall assume that the state x = x(t) of the controlled system (2.3.10) characterizes the general quantity (or the mean density) of fish at time t in some chosen habitat. We also assume that the intensity of fishing is bounded by a given value um > 0. In this case, the mathematical model for the dynamics of fish population has the form . (, x\ as = r I 1 — — I a? — qux, \ K)
0 < u(t)
t> 0,
(2.3.14)
x(Q) = ZQ > 0.
By p > 0 we denote the price of the unit mass of caught fish and by c > 0 the price of unit "efforts" u for fishing. Then it is natural to estimate the "quality" of control by the functional fT
I(u)= I (pqx(t)-c)u(t)dt, Jo
(2.3.15)
which, with regard to (2.3.11), gives the profit provided by the fishing process defined by the control function u(i): 0 < t < T. The problem is to find the optimal control u*(t): 0 < t < T for which the functional (2.3.15) attains its maximum. Following [35, 68], instead of (2.3.15), we shall estimate the quality of control (i.e., of fishing) by the functional in which the terminal time T —t oo. In this case, an additional "killing" factor appears in the integrand to ensure the convergence, namely, ,.00
I(u) = I Jo
st
e-
(pqx(t)-c)u(t)dt,
(2.3.16)
where S > 0 is a given positive number. As a result, we arrive at the following problem: for an arbitrary initial state x(Q) = x$ of the controlled system (2.3.14), to find a control function 0 < Ut.(t) < um: t > 0 (or
0 < w* ( x ( t ) } < um: t > 0) for which the functional (2.3.16) attains its maximum on the trajectories of system (2.3.14). REMARK. If the initial population size XQ does not exceed the capacity K of the medium, then it follows from Eq. (2.3.14) that for any time moment
128
Chapter II
t > 0 and any admissible control u(t), the population size has the same property, x(t) < K. Therefore, this problem is well posed if the parameters p, q, and c in the functional (2.3.16) satisfy the condition
— = Xl
(2.3.17)
Otherwise, (x± > K), this problem has only a trivial solution u*(t) = 0, t > O.14 Therefore, in what follows, we assume that the inequality (2.3.17) is satisfied. We also assume that qum > r.
D
2.3.3. The solution of problem (2.3.14), (2.3.16). If we define the function F(x) of the maximum future profit by the relation
U
oo
-|
e~st (pqx(t) - c)u(t) dt z(0) = x , j
(2.3.18)
then, using the standard procedure described in §1.3, we obtain the Bellman equation
max (J^U^Ufn
< I j
ra!
(l ~ IT) ~~ qux\ —r— — SF + (pqx — c)u > = 0 ^
-**- '
I CtiC
(2.3.19)
J
corresponding to problem (2.3.14), (2.3.16). It follows from Eq. (2.3.19) that, depending on the current state (the
population size) x of the system (2.3.14), to perform the optimal control we need to choose u*(x) = 0 for all points x £ R1 C R+ at which the function
p(x) =pqx-c-qx—— ax
(2.3.20)
is negative. Conversely, at all points x G R2 C R+ where
need to take the maximum admissible control ut(x) = um. If tp(xt) = 0 at a point cc* (in view of continuity, the point x* separating R1 from R2 is the limit point of these domains), then the optimal control u f ( x * ) for Eq. (2.3.19) is formally undetermined. However, one can see that the choice
of any admissible control 0 < u < um at the point x* does not affect the solution of Eq. (2.3.19).
Now let us consider how the population size in system (2.3.14) varies with time. Let x(0) = XQ < xi. Obviously, in this case, there exists an
initial half-interval [0,t*) at all points of which we must set ut(t) = 0. This statement immediately follows from the fact that the expression pqx — c in
the parentheses in (2.3.16) is negative for x(t] close to XQ. Thus, we have 14
Since x\ > K, we have pqx(t) — c < 0 for all t.
Exact Methods for Synthesis Problems
129
x(t) e R1 for all t £ [0,t«). Hence, it follows from Eq. (2.3.14) with u = 0 that, on the interval [0,**), the population size x(t) increases monotonically up to the value xt = x(t*) that separates the sets R1 and R 2 . At the point x f , as was already noted, the control may take any arbitrary value. It is expedient to take this value equal to
and keep it constant for t > t#. It follows from (2.3.14) that the control (2.3.21) preserves the population size xt. REMARK. For u(xt) ^ MI, the representative point of system (2.3.14), starting from the state a?*, comes either to the set R1 (for u(xt) > MI) or to the set R2 (for u(xt) < «i) during an infinitely small time interval. Then the control u = 0 (or u = um) immediately returns the representative point to the state a;*. Thus, for u(x+) ^ ui, the population size ** is preserved by infinitely rapid switches of control (this is the sliding mode). Though, as follows from (2.3.19), the value of the functional (2.3.16) for this control remains the same as for u(xt) = -MI, the constant control u(t) = u(xt) = HI, t > <*, is more convenient, since in this case the existence problem does not arise for the solution x(t), t > t f , of Eq. (2.3.14). The optimal control 0
for
0 < t < t*
«={(1-V)
fo,
«>«„
(x(t)
< a;*),
<"-22>
realizes the generalized solution x*(t) of Eq. (2.3.14) in the Filippov sense (see §1.1). D Thus, for x(0) = XQ < xi the optimal control (2.3.22) is a piecewise function shown in Fig. 21 together with the plot of the function x*(t), which shows the change of the population size corresponding to this control. It
remains to find the moment tt at which the catching of individuals starts or, which is the same, to find the size (density) xt = xt(tf} of the population that we need to keep constant in the area of active catching. These variables can readily be obtained by calculating the functional (2.3.16) and taking its maximum with respect to t«. Indeed, for the control (2.3.22), the functional (2.3.16) is equal to -
K J
. (2.3.23)
We can calculate its maximum with respect to tt by using the fact that x* = z*(i*) as a function of tt satisfies Eq. (2.3.14) for u = 0. After the
Chapter II
130
FIG. 21 differentiation, from the extremum condition
we obtain the following equation for the optimal size x* of the population: x
t - ± ( ± + ( l - i } K X\ x _ ^ - n • 2\pq + ( 2Mr ' r)") *
(
f2324^ '
This equation has only one positive solution
J
(2.3.25)
which has a physical meaning. Note that, in view of (2.3.17), the value xt determined by (2.3.25) always satisfies the inequality x^ < xt < K.
We also note that the condition
XQ < %i, introduced for the sake of obviousness, does not influence the
Exact Methods for Synthesis Problems
131
choice of the optimal control. This strategy is completely determined by x f , and according to this strategy, we do not catch individuals if the current population density x(t) < xf and start catching with constant intensity (2.3.21) if the population size attains the value xt (2.3.25). We can readily calculate the profit function (2.3.18) corresponding to
this strategy. Integrating the equation in (2.3.14) with a;(0) = x and u = 0, we obtain
-rK
x (w t ) = —————-.——r, x + (K - x)e~Tt
t > 0. ~
v(2.3.26)
'
Using (2.3.26), we see that the condition
xK
— = a;,
X -f {J\ — XJC
(2.3.27)
' '
allows us to find the moment tt. From (2.3.23) and (2.3.27), we explicitly calculate the profit function -Z*
(2.3.28) for x < x*. To solve problem (2.3.14), (2.3.16) completely, it remains to consider the case x(0) = XQ > x*, that is, the case where the initial population size is larger than the optimal size (2.3.25). First, we note that, in view of (2.3.28), the profit function F(x) monotonically increases on the interval 0 < x < x* from zero to the maximum value
(x,) =
dq
(pqx* - c) 1 \ KJ
(2.3.29)
We also note that the function ^(x) has only one maximum point
and since the "killing" factor 6 in (2.3.16) is strictly positive, we always
have the strict inequality x 2 > x*. Now if x(0) = XQ = x2 > £*, then using the constant control (2.3.30)
Chapter II
132
we can keep the population size at a level of x? for which the functional (2.3.16) attains the value
I(u(x2)) = However, the constant control (2.3.30) is not optimal. One can readily see that the functional (2.3.16) can take values larger than l(u(x.2)) = if>(x2) if, instead of (2.3.30), we use the piecewise constant control function
0
(2.3.31)
shown in Fig. 22.
A
x(t)
FIG.
22
We choose a time interval A during which the control um is applied, so that at the end of A the population size attains the value (2.3.25), that is,
«(A) = a;*.15 The time interval A for the control um is determined by the equation , - ________x 2 K(r-qu m )________ «t'* —
p
:
:——————^—;——————TT"-
rx2+[K(r-qum)-rx2\e(iu™-r^
(Z.O.OZ)
15
The inequality I(u&(t)) > 7(«(a;2)) = *l>(xi) is obtained by calculating the func-
tional (2.3.16) with regard to Eq. (2.3.14), where control has the form (2.3.31). Here we do not perform the corresponding elementary but cumbersome calculations and leave
them to the reader as an exercise.
Exact Methods for Synthesis Problems
133
Obviously, control functions of the form (2.3.31) can be used not only for the initial population size x(0) = x% but also for arbitrary initial sizes x(0) = x > xf. In this case, we must perform the change £2 —>• x only in Eq. (2.3.32) for the length A of the initial pulse um. One can easily verify
that (2.3.20) implies tp(x) > 0 for all x > x*. Therefore, the optimal control as a function of the current population size (the synthesizing function) for problem (2.3.14), (2.3.16) has the form
for
for for
0 < x < a;*,
x = x#,
(2.3.33)
x > a;*,
where xt is determined by (2.3.25). Formula (2.3.33) gives the mathematical expression for the control strategy that is well known in the theory of optimal fisheries management [35, 68]. The key point in this strategy is the existence of an optimal size xt of fish population given by (2.3.25). The goal of control is to achieve the optimal size xf of the population as soon as possible and to preserve the size x* by using the constant control (2.3.21). This control strategy maximizes
the profit obtained by fishing if the profit is estimated by the functional (2.3.16). In conclusion, we note that the results presented in this section can be generalized to the case in which the dynamics offish population is subject
to the retarded equation (or the equation with delay) =r(l- ^--V] X(t) - qu(t)x(t); K \ J
here we have studied the controlled Hutchinson model. For the results related to this case, see [99].
The stochastic version of problem (2.3.14), (2.3.16), when the behavior of the population is described by the stochastic equation (2.3.13), will be
described in §6.3. §2.4. Stochastic problem of optimal fisheries management
Now let us consider the problem of optimal fisheries management that differs from the problem considered in §2.3 by the stochastic character of the model used for the description of the population dynamics. We assume that the behavior of a fish population is subject to the stochastic differential equation of the form
x ( t ) = [r- qu(t)]x(t) + V2Bx(t)£(t),
x(0)
= z0,
0 < u(t) < um,
t > 0,
(2.4.1)
134
Chapter II
where £ ( t ) is a scalar Gaussian white noise (1.1.31), B > 0 is a given positive number, the natural growth factor r > 0 and the catchability coefficient q > 0 have the same meaning as similar coefficients in (2.3.10), (2.3.13), and (2.3.14). Equation (2.4.1) is a special case (as K ->• oo) of Eq. (2.3.13) and, in accordance with the classification presented in Section 2.3.1, the model described by Eq. (2.4.1) can be called a controlled stochastic Malthus
model. Just as in §2.3, the size x ( t ) of the fish population is controlled by catching a part of this population. In this case, the catching intensity u(t ) has an upper bound um, and therefore, the set of all nonnegative measurable bounded functions of the form u(t) : [0, oo) —>• [0, um] is considered as the
set of admissible controls. The goal of control is to maximize the functional (2.3.16), which, in view of the random character of the functions x(t) and u(t), is replaced by the corresponding mean value. As a result, we have the problem r
/.oo
I(u) = E \
-|
e~5t (pqx(t) - c}u(t) dt ->•
I JO
J
max
.
(2.4.2)
«(t)6[0,« m ] «>0
In what follows, we assume that the decay index 6 in (2.4.2) satisfies the
condition 8 > r. We shall solve problem (2.4.1), (2.4.2) by using the standard procedure of the dynamic programming approach described in §1.4. We define the profit function for problem (2.4.1), (2.4.2) by the relation F(x)=
r r°° e~
max E / «(«)e[o,« m ] Uo t>o
Si
i
(pqx(t) - c)u(t) dt \ x(Q) = x , \
(2.4.3)
where £[(•) [ x(0) = x] denotes the conditional mathematical expectation of (•). As was shown in [113, 175], the second-order derivative of the profit function (2.4.3) is continuous. It follows from Theorem 3.1.5 in [113] that for all x G R + — [0, oo), this function has the upper bound
F(x)
(2.4.4)
where N > 0 is a constant.
The Bellman equation (Fx - ^, Fxx = f^f) max [Bx2 Fxx + (r + B - qu)xFx - SF + (pqx - c)u] = 0
0<«<« m
(2.4.5)
for the profit function (2.4.3) can be obtained as usual (see §1.4). It should
be pointed out that a symmetrized stochastic integral (see [174] and §1.2)
Exact Methods for Synthesis Problems
135
was used for writing (2.4.5). This led to an additional term B in the parentheses in (2.4.5), that is, in the coefficient of xFx.le Equation (2.4.5) allows us to find the optimal control ut as a function u i f ( x ) of the current states of system (2.4.1). First, we note that, according to (2.4.5), the set of all admissible states of (2.4.1) can be divided into the following two subsets (just as in §2.3):
the subset R 1 , where
the subset R 2 , where f>(x) > 0 and u*(x) = um. The boundary between these two subsets is determined by the relation pqx - c - qxFx - 0.
(2.4.6)
The further calculations show that, in this problem, there exists a unique point x* satisfying (2.4.6). Therefore, the subsets R1 and R2 are the intervals R1 = [0, x*) and R2 = (x*,oo). Thus the optimal control in the synthesis form Ut = u*(x) becomes uniquely determined at all points x <E R+
except for the point x*. It follows from (2.4.6) that we can use any admissible control u(z*) 6 [0,itm] at the point x*. Therefore, the optimal control function M*(X) can be represented in the form ( \ ' ,,
u*(x)=<
(2.4.7)
I Mm,
if
X > X*,
and the final solution of the synthesis problem is reduced to calculating the coordinate of the switch point X*. To calculate x*, we need to solve the Bellman equation (2.4.5). As was already noted, the second-order derivative of the profit function F(x) is continuous, thus the profit function F(x) satisfying (2.4.5) can
be obtained by using the matching method (see §2.2). In what follows, we describe the procedure for solving the Bellman equation (2.4.5) and calculating the coordinate of the switch point xf in detail. By F*(x) and F2(x) we denote the profit function F(x) on the intervals
R1 = [0, x,) and R2 = (a:,, oo). It follows from (2.4.5) and (2.4.7) that the functions Fl and F2 satisfy the linear equations
- SF1 = 0, 2
Bx F
2
2
x
0 < x < x,, 2
+ (r + B - qum)xF - SF + (pqx
- c)um = 0,
(2.4.8) x > a;,.
(2.4.9) 16 If the stochastic differential equation in (2.4.1) is the Ito equation, then the second term in the Bellman equation (2.4.5) has the form (r — qu)xFx.
136
Chapter II
Since the profit function F(x) is sufficiently smooth (recall that the secondorder derivative of F(x) is continuous), both functions F1 and F2 must satisfy the condition (2.4.6) at the switch point x*. Taking into account the fact that F(0) = 0 according to (2.4.1) and (2.4.3), we have the following additional boundary condition for the function F!(x):
Fl(0) = 0.
(2.4.10)
The boundary conditions (2.4.6), (2.4.10), and the upper bound (2.4.4) allow us to obtain the explicit analytic solution of Eqs. (2.4.8) and (2.4.9). Equation (2.4.8) is the well-known homogeneous Euler equation. Its general solution has the form
,
(2.4.11)
where AI and A? are constants, and k\ and k\ satisfy the characteristic equation
Bk(k-l) + (r + B)k-S = 0.
(2.4.12)
The constants AI and A^ are determined by two boundary conditions (2.4.10) and (2.4.6) at the points x = 0 and x = a;,. Since the roots
of Eq. (2.4.12) have opposite signs, we conclude that to satisfy the condition (2.4.10), we need to set A2 equal to zero in (2.4.11). The constant c
~k
can be calculated by substituting Fl(x) = Aixk> into (2.4.6) and taking into account the fact that (2.4.6) is valid at the switch point x*. Thus, the
solution of Eq. (2.4.8) is given by the formula
>
0<x<x*.
(2.4.14)
The inhomogeneous Euler equation (2.4.9) can be solved in a similar way. By using the standard method of variation of parameters, we obtain the general solution PqX
Exact Methods for Synthesis Problems where
137
___________ *? = ^ [««m - r + >/(««m - r) 2 + 4B<5] ,
Y ^2 = ^[?w™ -
r
Lti
___________ ~ V(lum -r)2 + 4BS]
(2.4.16)
satisfy the characteristic equation
Bk2 - (qum -r)k-S = 0, and Ay and A± are arbitrary constants.
Since k2 is positive, we must set the constant A$ equal to zero (otherwise, formula (2.4.15) contradicts the upper bound (2.4.4)). The constant A^ can
be calculated from condition (2.4.6) at the switch point x f . Substituting F2(x) (determined by (2.4.15) with A3 = 0) into (2.4.6) instead of the function F, we obtain A
.
(S-r-B)p
——— ________ 1____________________ '___________ rp
4 — r 7/ r
n
,
\
i_fc*
^
__
*
c
_______ /w
T 7
-kl
2,
*
This implies the following expression for the function F2(x):
6 - r - B + qum
l\(6-r-B)px*
c]fx\k'
+ 72" r————— JT^ ——— ~ T
kl [S-r-B + qum
— )
JJV**/
i
x >x
~
*-
(2.4.17)
The two functions F1(x) (2.4.14) and F2(x) (2.4.17) determine the profit function F(x) satisfying the Bellman equation (2.4.5) for all x £ R+ = [0,oo). These functions contain a parameter a;*, which remains unknown.
We can calculate x* by using the continuity property of the profit function F(x). Each of the functions Fl and F2 is continuous. Hence, to ensure the continuity of F(x), it suffices to satisfy the condition
Fl(x*) = F2(x*)
(2.4.18)
at the switch point x». It follows from (2.4.6), (2.4.8), and (2.4.9) that (2.4.18) is equivalent to the condition
F2,(x.) = F2x(x*) at the switch point x*.
(2.4.19)
138
Chapter II
Calculating the second-order derivative of the functions (2.4.14) and (2.4.17), we derive the following equation for x» from (2.4.19):
p(6 — r — B)
c
Hence, the switch point xt is determined by the explicit formula T _ JLtjfi
c(d — r — tf + qum)
——
/1 I
*\
*
I Zf.rt.ZdJ I
pq[S — r — B + ,ki~k2\qum] Formula (2.4.20) and the optimal control algorithm (2.4.7) constitute
the complete analytic solution of the stochastic problem (2.4.1), (2.4.2) of optimal fisheries management. Some final comments and remarks. It is of interest to compare (2.4.20) and (2.3.25), which is the optimal size of the population in the deterministic problem (2.3.14), (2.3.16) of optimal control considered in §2.3. Denoting (2.3.25) by xf, we may expect that the equality
lim x* = lim xt
K-s-oo
B-X)
(2.4.21)
is valid due to continuity reasons (since the deterministic version of problem (2.4.1), (2.4.2) formally coincides with problem (2.3.14), (2.3.16) as K ->• oo). We can verify (2.4.21) by straightforward calculations of the limits on both sides. Indeed, using (2.3.25), we readily calculate the limit on the left-hand side of (2.4.21) for S > r,
lim i, =
K-+OO
f
pq(S - r)
(2.4.22) ^
'
The same result is obtained by calculating the limit of (2.4.20) as B —>• 0, since fej-l v (S-r)(qum-r) k\ — k\ B->o which follows from (2.4.13) and (2.4.16). Formula (2.4.21) shows how the results obtained in this section for problem (2.4.1), (2.4.2) are related to similar results for problem (2.3.14), (2.3.16) obtained in Section 2.3.3 by quite different methods.
There is another interesting specific feature of problem (2.4.1), (2.4.2). Namely, the standard "classical" approach of the dynamic programming
Exact Methods for Synthesis Problems
139
that leads to the exact solution of the stochastic problem (2.4.1), (2.4.2) does not allow us to solve the synthesis problem (that is, to find the switch point Xf) for the deterministic version of problem (2.4.1), (2.4.2), that is, in the case where there are no random perturbations in Eq. (2.4.1). This fact can readily be verified if we consider the deterministic version of the
Bellman equation (2.4.5). max \(r - qu)xFx - SF + (pqx - c)u] = 0
(2.4.23)
0
and calculate the functions
F'( z ) = l(f>z.-£)(±.J :c I
i •"
d \o-l
,
(2.4.24)
-«'l <- I + T^ i / \7 i \ •*-* / a[(r-qu )(a-l) m
I —
q\ \a;*
* r - qum
(2.4.25)
which, in this case, determine the profit function F(x) on the intervals R1 = [0, »*) and R 2 = [a;*, oo). Contrary to the stochastic case in which the continuity condition (2.4.18) for the functions (2.4.14) and (2.4.17) determines the unique switch point (2.4.20), one can readily verify that the same continuity condition F 1 ( x f ) = F2(x*) for the functions (2.4.24) and (2.4.25) holds for any point xf € (0, oo). Therefore, the control problem considered can serve as an example illustrating the well-known idea (see [113, 175]) that the dynamic programming approach is more suited for solving control problems with stochastic
models of plants (which, by the way, describe the actual reality more adequately) . REMARK. If the equation in (2.4.1) is understood as the Ito stochastic equation, then the Bellman equation for problem (2.4.1), (2.4.2) differs from (2.4.5) and has the form max [Bx2Fxx + (r - qu)xFx — SF + (pqx — c)u] = 0.
0<«<« m
The way for solving this equation is quite similar to the above procedure for solving Eq. (2.4.5). However, the population size xf that determines
the switch point for the optimal control (2.4.7) differs from (2.4.20) and is
140
Chapter II
given by the expressions
_
=
c(S - r + qum)
s B + qum - r - y/(B + qum - r) 2 + 46B . D
CHAPTER III
APPROXIMATE SYNTHESIS OF STOCHASTIC CONTROL SYSTEMS WITH SMALL CONTROL ACTIONS
Various approximate synthesis methods can be useful if the Bellman equation cannot be solved exactly. Chapters III-VI deal with some of these
methods. Approximate methods are usually efficient if the initial statement of the optimal control problem contains a small parameter. Quasioptimal control algorithms are constructed by using either the corresponding procedures of successive approximations or asymptotic expansions of the loss function in powers of a small parameter of the problem. The choice of a method for constructing an approximate solution of the synthesis problem essentially
depends on the choice of a parameter that is considered to be small. For example, in this chapter, the values of control actions are assumed to be small. Chapter IV is about the Bellman equation with small diffusion coefficients. In Chapter V, we consider control problems for oscillating systems with small attenuation decrement. In Chapter VI, the role of small parameters is played by the a posteriori covariances of unknown coefficients in the plant equations. Let us formulate the main idea of the approximate synthesis method studied in this chapter. As was already noted, the method is based on the assumption that control actions on the plant P are relatively small. Prom
the physical viewpoint, this assumption means that the effect of the control actions on the phase trajectories of the system is small, and therefore the system dynamics is similar to noncontrolled motion. In particular, this
assumption holds for control problems with constraints if the noises acting on the plant are of large intensity. Indeed, let us assume that the controlled (unperturbed) plant is a stable
mechanical system. Then large random perturbations lead to large deviations of the system from the equilibrium state. In this case, some "internal" inertial and elastic forces arise in the system. These forces can significantly
exceed the (bounded) control forces whose effects on the system turn out to be relatively small.1 1
Note that in this book we do not consider deterministic synthesis problems for
141
142
Chapter III
From the formal mathematical viewpoint, the fact that control actions are small leads to a small parameter in the nonlinear term in the Bellman equation. To verify this fact, let us consider the synthesis problem for the servomechanism (Fig. 10) governed by the Bellman equation (1.4.21). Assume that the dimensions of the region U of admissible controls are bounded by a small value of the order of e. For definiteness, we assume that U is either an r-dimensional parallelepiped (RT D U = (u: u,-| < umi, i — 1, 2, . . . , r; maxj umi = e)) or an r-dimensional ball of radius e, that is, (R, D U = (u: £J=1 «? < e ) ) . In the first case, according to (1.3.22), the solution of the synthesis problem is given by the formula (the control algorithm)
= - { « m l , - . - , W m r } s i g n ( QT (t) -r— (t, X, y) 1,
\
dx
J
(3.0.1)
where the vector of partial derivatives dF/dx is calculated by solving the equation
LF(t, x,y) = -ci ( x , y ) + eu^ ^
w
~ - r, *,,,
(3.0.2)
Here um denotes the r-vector (column) um/e and L is a linear operator of the form 2
"'^^ u -
(see
32
In the second case (where U is a ball), the optimal control has the form (1.3.23))
systems controlled by small forces. Such systems called weakly controllable in [32] were studied in [32, 137]. 2 Recall that relations (3.0.1) and (3.0.2) follow from the Bellman equation (1.4.21)
with c(x,y,u) = ci(x,y) and Ax(t,x) = a(t,x) + Q(t)u; {uml,.. .,umr} denotes a diagonal (r X r)-matrix; for a column A with components AI ,..., Ar, the expressions sign A and \A\ denote j--columns with components sign-A,- and \A{\ (i = 1,..., r), respectively.
Approximate Synthesis of Stochastic Control Systems
143
where the vector dF/dx is the gradient of the loss function satisfying the equation
(
op
1
ft]?\ ~^
QQT-
•
(3.0.4)
If we denote the nonlinear terms in Eqs. (3.0.2) and (3.0.4) in the same way, then we can write both equations in the form
,
(3.0.5)
where $(f, dF/dx) is a given nonlinear function of its arguments. As a rule, equations of the type (3.0.5) cannot be solved exactly. However, the presence of a small parameter in the nonlinear term of this equation yields a rather natural way of solving this equation approximately. To this end, one can use the method of successive approximations in which the
zero-order approximation Fo(t, x,y) satisfies the equation
LF0 = -c1(x,y)
(3.0.6)
and the successive approximations Fk(t,x,y) can be calculated recurrently by solving the sequence of linear equations of the form
A =1,2,....
(3.0.7)
If we know the solution Fk(t, x, y) of the equation for the /sth approximation (fc = 0, 1, . . .), then we can perform an approximate synthesis of the controlled system by setting the quasioptimal control algorithm as (3.0.8)
In this chapter we consider an approximate method for the synthesis of optimal systems whose "algorithmic" essence is given in formulas (3.0.6)-
(3.0.8).3 Needless to say, the practical use of procedure (3.0.6)-(3.08) in special problems leads to additional problems of constructivity and efficiency of 3 The approximate synthesis algorithm (3.0.6)-(3.08) is a modification of the wellknown Bellman method of successive approximations [14, 16]. This method was used by
W. Fleming for solving some stochastic problems of optimal control [55]. The procedure
(3.0.6)-(3.0.8) is a special case of the Bellman method if the trivial strategy uo(t,x,y) = 0 is used as the initial "generating" control strategy in the Bellman method.
144
Chapter III
this approximate synthesis method. In this chapter we shall discuss these problems in detail. All related material is divided in sections as follows. First (§§3.1-3.3), we consider some methods for calculating the successive approximations for stationary synthesis problems. We write out approximate solutions (that correspond to the first two approximations) for some
special control systems with various types of disturbances affecting the system. In §3.1 and §3.2, we consider random perturbations of the white noise type. In §3.3 the results obtained in §3.1 and §3.2 are generalized to the case of correlated noises. In §3.4 we study nonstationary problems and estimate the error of the approximate synthesis (3.0.6)-(3.0.8) for the first two approximations. In §3.5 we study asymptotic properties of successive approximations (3.0.7), (3.0.8) as k —>• oo. We show that, under some special conditions, as k —> oo the sequence F& is convergent to the exact solution of the Bellman equation, and the corresponding quasioptimal control algorithms (3.0.8) to the optimal control ut(t,x,y). In this case, the convergence Uk -> uf is understood in the sense of convergence of values of the functional to be minimized. Finally, in §3.6 the method of successive approximations (3.0.6)^(3.0.8) is used for approximate synthesis of some stochastic control systems with distributed parameters. §3.1.
Approximate solution of stationary synthesis problems
3.1.1. Let us consider the problem of optimal damping of oscillations in a dynamic system subject to random perturbations of the white noise type (Fig. 13). Let the plant P be described by the following system of linear stochastic differential equations with constant coefficients:
(3.1.1)
Here x = x ( t ) is an n- vector (column) of current phase variables of the system (xi(t), . . . , xn(t )) , u = u(t) is an r-vector (column) of control actions (ui(t), . . . ,u r (t)), £(t) is an n-vector (column) of random perturbations with independent components (£i(<), . . • ,£»»(<)) of the standard white noise type (1.1.31), and A, Q, and
J
(3.1.2)
Approximate Synthesis of Stochastic Control Systems
145
where c(x) > 0 is a given convex penalty function attaining the absolute minimum c(0) = 0 at the point x — 0 (the restrictions on c(z) are discussed in detail in §3.4 and §3.5). Let admissible controls be bounded and small. We assume that all components of the control vector u satisfy the conditions
K < £umi,
i=l,...,r,
(3.1.3)
where e > 0 is a small parameter and w m i , . . . , umr > 0 are given numbers
of order 1. The system shown in Fig. 13 is a special case (the input signal y(t) = 0) of the servomechanism shown in Fig. 10. Therefore, the Bellman equation
for problem (3.1.1)-(3.1.3) readily follows from (1.4.21), and taking into account the relations Ay(t, y) = 0, By(t, y) = 0, Ax(t, x, u) = Ax + Qu, and c(x,y,u) = c(x), we obtain dF(t x}
' +LF(t,X)+ Ot
min
T
gT
+ c(x) = 0.
(3.1.4)
Here L denotes a linear elliptic operator of the form
(3 L5)
-
where, according to (1.4.16), the matrix B = aaT , and, as usual, the sum in the last expression on the right-hand side of (3.1.5) is taken over repeated indices from 1 to n. It follows from (3.0.1) and (3.0.2) that in this case the optimal control has the form
(3.1.6) where the loss function F(t, x) satisfies the equation
dF —— (t, a;) + LF(t, x) - -c(x) + eu
(3.1.7)
Some methods for solving Eq. (3.1.7) will be considered in §3.4. In the present section, we restrict our consideration to stationary operating conditions of the stabilization system in question. It follows from §1.4 that the stationary mode of stabilization (damping) can take place if T —)• oo, where T is the terminal instant of the operation interval (the upper
Chapter III
146
integration limit in (3.1.2)). Obviously, in this case, stationary operating conditions exist only if the unperturbed motion of the plant P is stable, that is, in other words, if the real parts of the eigenvalues of the matrix A in (3.1.1) are negative. In what follows, we assume that these conditions
are satisfied. If we define the stationary loss function f ( x ) by the relation (see (1.4.29), (2.2.9))
then (3.1.7) implies the following time-invariant equation for f ( x ) :
Lf(x)
= 7 - c(x) +
(3.1.8)
where the parameter 7 characterizing stationary "specific losses" together
with the function f ( x ) can be found by solving Eq. (3.1.8). We shall solve Eq. (3.1.8) by the method of successive approximations. The scheme for calculating (3.0.6), (3.0.7), applied to the time-invariant equation (3.1.8) leads to the sequence of equations
(3.1.9)
Lfo = 7° - c ( x ) ,
Lfk = Jk- c(x)
Q1
dx
(3.1.10)
It follows from (3.1.9) and (3.1.10) that each time when we are calculating the next approximation, we need to solve a linear inhomogeneous elliptic equation of the form
Lf(x) =
(3.1.11)
We shall consider a method for solving Eq. (3.1.11) with a given function
in eigenfunctions of some Sturm-Liouville problem [179]. 3.1.2. The passage to the adjoint equation. Let us consider the operator
that is the dual of the operator (3.1.5). The equation
— (<, x) = L*p(t,x)
(3.1.13)
Approximate Synthesis of Stochastic Control Systems
147
is the Fokker-Planck equation (1.1.67) for the n-dimensional Gaussian Markov process x(i) [45, 167, 173]. The assumption that the matrix A is stable implies that this process has a stationary density function po(x) such that
L*p0(x) = 0.
(3.1.14)
In this case the stationary probability density po(x) has the form
po(x) = V
. /
1
«
T ——— exp [L - \(x Px)} J . 2 - i
(3.1.15) '
where P"1 is the matrix of covariances of components of the vector x.
We shall present a possible method for solving Eq. (3.1.13) and calculating the matrix P in (3.1.15). The diffusion Markov process x ( t ) described by (3.1.13) satisfies the system of linear stochastic differential equations
x = Ax +
(3.1.16)
describing the uncontrolled motion of the plant (3.1.1). We pass from x to new variables y related to x by the linear transformation
x = Vy
(3.1.17)
with a nondegenerate matrix V. As a result, instead of (3.1.16), we have the following system for the new variables:
(3.1.18) where
A = V~iAV,
a = V-1a.
(3.1.19)
We choose V so that the matrix A is diagonal
I={A1,A2,...,An}.
(3.1.20)
As is known [62], the matrix V always exists and can readily be constructed if the eigenvalues of the matrix A are simple, that is, if the characteristic equation of the matrix A,
det(j4. - \E) = 0,
(3.1.21)
has different roots AI, A 2 , . . . , A n . In this case, the columns of the matrix V are the eigenvectors vl of the matrix A that satisfy the linear equations
Aff=\ivi,
i=l,...,n.
(3.1.22)
148
Chapter III
The system (3.1.18) can readily be solved in the case of (3.1.20). Indeed, writing (3.1.18) in rows, we obtain
yt = Aiyi + rli(t),
£=l,...,n,
(3.1.23)
where the random functions rjt(i) — &tk(,k(t) are processes of the white noise type and have the characteristics
Eife(*) = 0,
Em(t)rjm(t - T) = BtmS(r)-
£ , m = l , . . . , n . (3.1.24)
Here Btm is an element of the matrix B = cfcfT. obtain
yt(t) = yt0e^-^ +
Solving Eq. (3.1.23), we
e^-r,^') dt',
ylo = W (t 0 ),
(3.1.25)
Jto
and taking into account (3.1.24), derive the following expressions for the means and covariances:
(3.1.26)
which determine the transition probability p(y(t) process y(t). It follows from (3.1.26) that
Ey/ = 0,
Eytym = -
y(to)) of the Gaussian
B m
^ .
(3.1.27)
*t + *m
in the stationary case as t —> oo, since, by assumption, ReA^ < 0 (i = 1, . . . , n) for all roots of the characteristic equation (3.1.21) of the matrix A. It follows from (3.1.27) that the stationary density function po(y) can be written in the form
po(y) =
1
=exp[-i(y 1
r
Py)],
^/(27r)«detP-
where each entry of the matrix P"1 is given by the formula
(3.1.28)
Approximate Synthesis of Stochastic Control Systems
149
The stationary density po(y) satisfies the stationary Fokker-Planck equation
L*Po(y) = --(XiViPo) + ^jj-(BijPo) dyi
2 ayidyj
= 0.
(3.1.30)
Since the random processes y(t) and x(t) are related by the linear transformation (3.1.17), the comparison of (3.1.15) with (3.1.28) yields the formula
P = VTPV,
(3.1.31)
which together with (3.1.29) allows us to calculate the matrix P. Now let us return to the Fokker-Planck equation (3.1.14). If the operator (3.1.5) satisfies the potentiality condition (see §4, Section 5 in [173]), then the operator equality4 POL
= L*PQ
(3.1.32)
readily follows from (3.1.5), (3.1.12), and (3.1.14). However, even if the
potentiality conditions are not satisfied and (3.1.32) does not hold, one can choose an operator L\ satisfying a similar relation PoL
= L\PQ.
(3.1.33)
One can readily see that the operator L\ has the form5
ri = - ( G , ^ +
(^),
(3-1.34)
where the matrix G = ||G,-j||" is similar to the transpose matrix AT from (3.1.1) and (3.1.5),
G=P-1ATP.
(3.1.35)
The similarity transform (3.1.35) employs the matrix P from (3.1.15). Relation (3.1.33) allows us to replace Eq. (3.1.11) by a similar equation
for the dual operator. In other words, it follows from (3.1.11) and (3.1.33) that the problem of finding f ( x ) in (3.1.11) is equivalent to the problem of finding z(x) in the equation
L\z(x) = ip(x),
(3.1.36)
where z(x), i>(x), and the functions /(a;), V>(x) from Eq. (3.1.11) satisfy the relations
z(x)=po(x)f(x), 4
if>(x) = Po(x)if>(x).
(3.1.37)
As usual, the operator equality is understood in the sense that it is an ordinary
r e l a t i o n p g ( x ) L w ( x ) = L*po(x)w(x) for any sufficiently smooth function w(x). 5 The verification of (3.1.33) is left to the reader as an exercise.
150
Chapter III
3.1.3. The solution of equations (3.1.36) and (3.1.11). Let us consider the following problem of finding the eigenfunctions zs (x) and eigenvalues A s of the operator L\ (the Sturm-Liouville problem):
L\zs = \,z,.
(3.1.38)
Since L\ is the Fokker-Planck operator, its eigenfunctions zs must satisfy the zero conditions at infinity (as x —> oo).
By passing from x to new variables y (x = Vy) and acting in a way similar to (3.1.17)-(3.1.31), we can transform the operator (3.1.34) to the form £;
=-(A,.tt)+ I ( B
t f
) ,
(3.1.39)
where Bij is an element of the matrix B = a (TT , "a = V cr, V is a nonde__ _ i __ __ generate matrix such that the transformation V GV makes the matrix G diagonal, G = {Ai, . . ., A n }, and A; are roots of Eq. (3.1.21).6 In the new variables the stationary Fokker-Planck equation has the form
*
- 0-
^-
This equation differs from (3.1.30) only by the matrix of diffusion coefficients; therefore, the stationary probability density p0(y) is determined by the formulas
pQ(y) =
1
_= exp [ - |(j/TPy)] ,
(3.1.41)
y(27r)"detP
1 similar to (3.1.28) and (3.1.29). Differentiating (3.1.40) appropriately many times, we see that
_
+ m 2 A 2 + • • • + mn \n) —^——jr^Po = 0,
(3.1.43)
According fco (3.1.35), the matrix G is similar to the transpose AT. Since all similar and transpose matrices have the same eigenvalues, the characteristic equation det(G — AB) = 0 for the matrix G coincides with (3.1.21)).
Approximate Synthesis of Stochastic Control Systems
151
where mi, m?,..., mn are any arbitrary integers between 0 and oo. It follows from v(3.1.43) that the functions -J^j—Q 1" Pn an-d the num' dy-i ---dyn bers (miAi + • • • + m n A n ) can be treated as the eigenfunctions zs and the
eigenvalues Xs of problem (3.1.38), respectively. By using (3.1.41), we can write the functions zs in more detail as follows:7 Zs
= (_l)™i+"-+™»tf rai ... ra J y )exp [- f(y T Pj/)].
(3.1.44)
Here Hmi,_mn(y) — Hmi...mn(yi: • • -,2/n) denote multidimensional Hermitian polynomials (for instance, see [4]) that, by definition, are equal to Hm^,...,mn(y) = (-l) mi +"+ m « exp [f (j/TPt/)]
"
(3.1.45)
It follows from the general theory [4] for Hermitian polynomials with real variables y that these polynomials form a closed and complete system of functions, and an arbitrary function from a sufficiently large class (these functions grow at infinity not faster than any finite power of \y\) can be expanded in an absolutely and uniformly convergent series in this system of functions. Furthermore, the polynomials H are orthogonal to another
group of Hermitian polynomials G given by the formula
"-|(/i T P"V)]-
(3.1.46)
Here the variables fj, and y satisfy the relation or
=
and the orthogonality condition itself has the form
r00
i"*
J — oo
J — oo
_
MetP
..Sv,m,
(S1/imi is the Kronecker delta). 7
The constant coefficient [(2ir) n detP "'j- 1 / 2 in (3.1.44) is omitted.
(3.1.48)
152
Chapter III
However, we often need to use a complex matrix V for the change of variables x —> y (for instance, see the problem in §3.2.). To pass to complex variables, we need to verify some additional statements from the general
theory [4], which hold for real variables. In particular, it is necessary to verify the orthogonality conditions (3.1.48), which are the most important in practical calculations.
This was verified in [107], where it was shown that all properties of the polynomials H and G remain valid for complex variables if only all functions HVl...Vn(y), Gmi...mn(y), exp[|(yTPy)], and exp[|(/iTP fj,)] are considered as functions of the initial real variables x of the problem. To this end, we need to make the change of variables y = V x in all these functions. In particular, in this case, the orthogonality condition (3.1.48) has the form oo
/
/»oo
... / -oo
e-^xTp^HVl...
J — oo
..£„„„,„,
(3.1.49)
where the matrices PI and P satisfy the relation
~P = VTP1V
(3.1.50)
similar to (3.1.31). Thus, we obtain the following algorithm for constructing the solution f ( x ) of Eq. (3.1.11). First, we seek a stationary density po(x) satisfying (3.1.14) and an operator L\ satisfying (3.1.33). Then we transform prob-
lem (3.1.11) to problem (3.1.36). After this, to find the eigenfunctions and eigenvalues of problem (3.1.38), we need to calculate the matrix V that transforms the matrix G to the diagonal form { A i , . . . , A n } by the siml-
larity transform V GV. Next, using the known A,- and V and (3.1.42), we calculate the matrices P and P that determine the stationary distribution (3.1.41). The expression obtained for p0(y) enables us to find the eigenfunctions zs = zmi...mm (3.1.44) for problem (3.1.38) and the orthogonal polynomials G TOl ... TOn (3.1.46). Finally, we seek the function z(x) satisfying (3.1.36) in the form of the series with respect to the eigenfunctions: oo z x
( ) =
X) mj....m n =0
a
m1...mnZmi. ..mn(x),
(3.1.51)
Approximate Synthesis of Stochastic Control Systems
153
where omi...ran are unknown coefficients; the eigenfunctions zmi,,.mn(x) can be calculated by formulas (3.1.44) with y = V x. If we also represent the right-hand side ij)(x) — po(x)
po(x)
^
bmi...mnzmi...mn(x),
(3.1.52)
where, in view of (3.1.49),
*— r...r
A/det PI (27r) n / 2 mi... !.. .m .,,„„. ^_00 00 n\ J_
Jj.oo _,
T-l i x G r o i ... T O/nT(F~ z)dxi...dz n
(3.1.53)
then we can calculate the unknown coefficients oroi...TOn in (3.1.51) by the formula ^m i...mn
\
ami...mn - T———— ,
\
.
, \
A mi ... mn = Aimi + ••• + Xnmn,
/ o i c x \
(3.1.54)
which follows from (3.1.38) and (3.1.43). Now we see that (3.1.37) implies the expression
for the solution of the initial equation (3.1.11). The algorithm obtained for solving (3.1.11) can be used for calculating
the successive approximations (3.1.9) and (3.1.10) It remains only to solve the problem of how to choose the stationary losses 7^ (k=0,l,2, ... ) in
Eqs. (3.1.9) and (3.1.10). 3.1.4. Calculation of the parameters Vs (k - 0,1,2,...). The structure of the solution (3.1.55) and a natural requirement that the stationary loss function f ( x ) must be finite imply that there is a unique method for choosing 7*. Indeed, since, according to (3.1.54), the eigenvalue AQO...O = 0, the coefficient aoo...o in (3.1.46) is finite if a necessary condition 60o.. .0 = 0 is satisfied, or, more precisely, (in view of (3.1.53) and (3.1.46)) if we have oo
/
/»oo
.../ •oo
J — oo
PQ(X)
154
Chapter III
This relation, (3.1.9) and (3.1.10) imply the following expressions for the stationary losses 7*: OO
/
/»OO
.../ -OO
c(x)pQ(x)dxl...dxn,
(3.1.56)
J — OO
Tdfk-i(x)
Q'
Po(x)dxi...dxn, dx = 1,2,.... (3.1.57)
Thus, we have completely solved the problem of how to calculate the successive approximations (3.1.9), (3.1.10) for the stationary operating conditions
of the optimal stabilization system. If the loss function f k ( x ) in the fcth approximation is calculated, then the quasioptimal control Uk(x) in the feth. approximation is completely defined,
namely, in view of (3.0.8) and (3.1.6), we have
(3.1.58) In the next section, using this general algorithm for approximate synthesis, we shall calculate a special system of optimal damping of random oscillations when the plant is a linear oscillating system with one degree of freedom. §3.2. Calculation of a quasioptimal regulator for the oscillatory plant
In this section we consider the stabilization system shown in Pig. 13, in which the plant P is an oscillatory dynamic system described by the equation
x + /3x + x = u + VB£(t),
(3.2.1)
where the absolute value of the scalar control u is bounded,
\u\ < e,
(3.2.2)
the scalar random process £(t) is the standard white noise (1.1.31), and /?,
B, and s are given positive numbers ((3 < 2). Equations of the type of (3.2.1) describe the motion of a single mass
point under the action of elastic forces, viscous friction, controlling and random perturbations. The same equation describes the dynamics of a direct-current motor controlled by the voltage applied to the armature when
Approximate Synthesis of Stochastic Control Systems
155
the load on the shaft varies randomly. Examples of other actual physical objects described by Eq. (3.2.1) can be found in [2, 19, 27, 136]. For system (3.2.1), (3.2.2), it is required to calculate the optimal regulator (damper) C (see Fig. 13), which will damp, in the best possible way with respect to the mean square error, the oscillations constantly arising
in the system due to random perturbations £(t). More precisely, as the optimality criterion (3.1.2), we shall consider the functional
(x2(t)
=E
x2 (t)) dt] , J
(3.2.3)
which has the meaning of the mean energy of random oscillations in system (3.2.1). Note that the mean square criterion (3.2.3) is used most frequently and this criterion corresponds to the most natural statement of the optimal
damping problem [1, 50]. However, there are other statements of the problem with penalty functions other than the function c(x] = x2 + x2 exploited in (3.2.3). From the viewpoint of the method used here for solving the synthesis problem, the choice of the penalty function is of no fundamental importance. To make the problem (3.2.1)-(3.2.3) consistent with the general statement treated in §3.1, we write Eq. (3.2.1) as the following system of two first-order equations for the phase coordinates x\ and x% (these variables can be considered as the displacement x\ = x and the velocity X2 = x):
(3.2.4)
= —zi —(3x2
Using the vector-matrix notation, we can write system (3.2.4) in the form (3.1.1), where A, Q, and
A=
0
0
Q=
(3.2.5)
According to §3.1, under the stationary operating conditions (T -» oo in (3.2.3)), the desired optimal damper C (Fig. 13) is a relay type regulator described by the equation (see (3.1.6))
u*(xi,x2) = -esign
-— . \OX2'
(3.2.6)
Here / = /(xi^xz) is the loss function satisfying the stationary Bellman equation (see (3.1.8))
df
(3.2.7)
Chapter III
156 where, according to (3.1.5) and (3.2.5),
L — x2-— - (/3x2 +
B d2
-— + IT -5-
(3.2.
ox-2
The equation
21
>.i-j.i.
determines a switching line for the optimal control action (from u — +£ to u = —£ or backwards) on the phase plane (KI, x2). The goal of the present
section is to obtain explicit expressions for the control algorithm (3.2.6) and the switching line (3.2.9). To this end, it is necessary to solve Eq. (3.2.7). We shall solve this equation by the method of successive approximations
discussed in §3.1. First, we shall prepare the mathematical apparatus for calculating the successive approximations. A straightforward verification shows that the stationary distribution with the density function po(x) = po(xi,x2), satisfying the equation (see (3.1.14))
f
=
has the form
(3.2.10) Hence, the matrices P and P~l in (3.1.15) are equal to
2/3 B
B
I 0 0 1
i o o i
(3.2.11)
It follows from (3.2.11) and (3.1.35) that in this case the matrix G of the
operator (3.1.34) coincides with the transpose matrix AT, that is, according to (3.2.5), we have " 0 -1 G= (3.2.12) 1 -0 and the operator (3.1.34) has the form
d
o
B d2 f\ s\ 2
(3.2.13)
One can readily see that the same probability density (3.2.10) satisfies the stationary equation Llpo(xi,x2) = 0. Therefore, in this case, the matrix PI from (3.1.49) and (3.1.50) coincides with the matrix P determined by (3.2.11).
Approximate Synthesis of Stochastic Control Systems
157
The matrix V that reduces (3.2.12) to the diagonal form by the similarity transform is equal to
1
V =
1
—AI
—A 2 (3.2.14)
This expression and formulas (3.1.50) and (3.2.12) imply B
B
Correspondingly, the inverse matrix P
B 2(4-,
2//J
(3.2.15)
-A2
has the form
A2 2//J 2//3 A!
(3.2.16)
The matrices (3.2.15) and (3.2.16) allow us to calculate the two-dimensional Hermitian polynomials .
Gim
r 1 / T7"^" \ T
, e i _,
f)l+m
^
r
1 / T1 ~^ \ i
exp
= (-!)
(3.2.18)
VL = Py.
Then these polynomials must be represented as functions of xi and x? by using the formula x = Vy and expression (3.2.14) for the matrix V. Table 3.2.1 shows some first polynomials H and G. In this case, in view of (3.1.51)-(3.1.55), (3.2.7), (3.2.10), and (3.2.11), the solutions of the equations of successive approximations (3.1.9), (3.1.10) can be written in terms of the Hermitian polynomials H^m(xi,X2) as the
series mA 2 '
(3.2.19) where the coefficients b\m are calculated by the formulas
(3.2.20) £
/V K __
/
-
9
V* ——
^1
9
I
^2 i ^
> 1.
Chapter III
158
TABLE 3.2.1 Polynomials H #00 = 1
-
-
2/3
-H
, p
2/3
//3
\
X2 #10 — -Pnyi + P\iyi — MI — ~TT B »i+ (^-y ) w
-"01 = -i 122/1 + -T22J/2 — M2 — ~^" ±> rr
"p
i
T-T-
n
.
rr
~p
i
'Xl+(P
+js}x,
,,2
-"20 = ~-» 11 + Ml
,,2
-"02 — — •« 22 T P2 #30
— Ml —
~ PllfJ-2
~
#12 =
#03 = A*2 — 3P22/x2 #40 = Ml ~ 3PnPl 2
#31 = MlM2
o-
_ ,,2,,2
rr
3
#"22 — MlM2 3Pl 2 P 2 2
#"13 — PlM2 #04
— fJ-2 ~ 6P22M 2
Polynomials G
GOO = 1
1 2JS 1 2^ Expressions for the polynomials G 2 O j G n , . . . can be obtained from the corresponding expressions for Him by the change Ppq —>• Ppq , fj,p —>• yp. Before we pass to the straightforward calculation of the successive ap-
proximations f k , we make the following two remarks about some singular-
Approximate Synthesis of Stochastic Control Systems
159
ities of the series on the right-hand side of (3.2.19).
REMARK 3.2.1. In practice, the series (3.2.19) is usually replaced by a finite sum. The number of terms of the series (3.2.19) left in this sum is determined by the rate of convergence of the series (3.2.19). Here we do
not discuss this question (see, for example, [26, 166, 179]). However, in our case, the series (3.2.19) cannot be truncated in an arbitrary way, since it contains complex terms such as the polynomials Him and the coefficients
aklm (this follows from (3.2.14)-(3.2.18) and (3.2.20)). At the same time, the loss function /jt(£i, £2) represented by this series has the meaning of a
real function for real arguments. Therefore, truncating the series (3.2.19), we must remember that a finite sum of this series determines a real function
only if the last terms of this sum contain all terms with Him of a certain group (namely, all Htm with l + m = s, where s is the highest order of the
polynomials left in the sum (3.2.19)).
D
REMARK 3.2.2. Equation (3.2.7), as well as the corresponding equations of successive approximations (3.1.9) and (3.1.10), is centrally symmetric (such equations remain unchanged after the substitution (x\, £2) —>• (—£1, —£2))- Therefore, the series (3.2.19) must not contain terms for which the sum (I + TO) is odd, since the polynomials Htm with odd (I + TO) are
not centrally symmetric (see Table 3.2.1). If we take this fact into account, then the body of practical calculations is considerably reduced.
D
In what follows, we present the first two approximations calculated according to (3.1.9) and (3.1.10) and the quasioptimal control algorithms «o(£i,£2) and «i(£i,CB2) corresponding to these approximations. The zero approximation. First of all, let us calculate the parameter 7° of specific stationary losses in the zero approximation. From (3.1.56) with regard to c(x) = x\ + x\ and (3.2.10), we have 0
O
f
r^
II
/•OO
/
2
*) \
F
/3 /
9
9 \ "I
KB JJ_ 0 0 Calculating the integral, we obtain
7° = j.
(3.2.21)
In view of Remark 3.2.2, the first coefficients a°0 and a^ in the series (3.2.19) are equal to zero.8 The coefficients b^Q, b®^ and 6§2 can be calculated by using the formulas for G2o, GII, and G02 from Table 3.2.1 and 8
The same result can be obtained if we formally calculate the coefficients 6j0 and fejjj
by using (3.2.20).
160
Chapter III
(3.2.16). Then, according to (3.2.20), the coefficient b% has the form O<x\
=
f3
x exp [ - f (x2 + x2)} (7° - xl - x2,), (3.2.22) The integral in (3.2.22) can readily be calculated, thus, taking into account (3.2.21) and (3.2.14), we obtain
In a similar way, we can easily find
(3 2 24)
--
All other coefficients b®m with l-\-m > 2 are zero in view of the orthogonality
condition (3.1.49). According to (3.2.19), it follows from (3.2.23) and (3.2.24) that a° =--^-/^-^
(3.2.25) 02
_
**'
IP
,rV
Finally, using the formulas for H20, #11, and H02 from Table 3.2.1 and (3.2.25), we obtain the loss function in the zero approximation t I
\ _
Orj-
| a0 r r
JOv*!) 2-2J — a20-"20 i
ll-"ll +
| a0 r r
02"02
. Ix| _ -B(/? 2 +4)
(3.2.26) This relation and condition (3.2.9) imply the following equation for the zero-approximation switching line F0:
zi+z
2
= 0.
(3.2.27)
Approximate Synthesis of Stochastic Control Systems
161
In this case, the quasioptimal control algorithm UQ(X] in the zero approximation has the form
u0(x) = -g sign (xi + ^x2} .
(3.2.28)
REMARK 3.2.3. The loss function f o ( x i , x 2 ) in the zero approximation (without a constant term) and the parameter of stationary losses (3.2.21) can be calculated in a different way, without using the method considered above. Indeed, if we first seek the solution of the zero approximation equation (L is the operator (3.2.8)) Lf0 = 7° - xl - x\
(3.2.29)
as the quadratic form
fo(xi, x2) = with unknown coefficients /in, hi2, and h22, then, substituting this expression into (3.2.39), we obtain four equations for ha, /ii2, h22, and 7°. However, higher approximations cannot be obtained by this simple reasoning. D
The first approximation. It follows from (3.1.10) and (3.2.26) that in the first approximation we need to solve the equation
2x2 This equation can be solved by analogy with the zero-approximation equation (3.2.29), but the calculations are much more cumbersome due to more complicated expression on the right-hand side.
First, we employ (3.1.57) and (3.2.21) to find the specific stationary losses 1
R
cR
^
f r°°
//
9
f / 3 / 2
2\~l
T
then, after the integral is calculated, we obtain
The coefficients a\0, a\^... in (3.2.19) are calculated by (3.2.19) and (3.2.20) with regard to the formulas for Glm from Table 3.2.1. We omit
162
Chapter III
the intermediate calculations and write the final expression for /I(:EI, £2)Taking only the first terms in the series (3.2.19) up to the fourth order inclusively (that is, omitting the terms for which (I + m) > 4), we obtain the following expression for the loss function in the first approximation: = VX
\ + px\ + puxixi + pmx\ + const . (3.2.31) Here9
ea/3 25
P31 = -^5-,
Pii =
£CX. l+T,
_ ea B
_ ea/3 125
P22 = -=-,
_ ea 165
Pl3 = 77T^, 1
O^O!
^2 = ^ - ^ ,
P04 = T^, ,
/o i->\ — 1 /9
« = (7r/35)
(32.32)
.
The condition f32
(3.2.32). From (3.2.9) and (3.2.31) we obtain the following equation for the switch-
ing line F1 in the first approximation:
It follows from the continuity conditions that for small e the switching
line F1 is close to F° determined by Eq. (3.2.27). Therefore, if we set £ 2 = —(/3/2)xi in the terms of the order of e in (3.2.33), then we obtain errors of the order of e2 in the equation for F1. Using this fact and formulas (3.2.32) and (3.2.33), we arrive at the following expression with accuracy up to the terms of the order of O(e1}:
Figure 23 shows the position of the switching lines F° and F1 on the phase plane (EI, x%). The switching line (3.2.34) determines the quasioptimal control algorithm in the first approximation:
«i(i) = -esign L +
l+
£«a;i -
-Zi •
(3-2.35)
We do not calculate the coefficients v and p and the constant term "const" in (3.2.31) since they do not affect the position of the switching line and the control algorithm in the first approximation.
Approximate Synthesis of Stochastic Control Systems
163
— —e
W* = £
FIG. 23
FIG. 24 This algorithm can easily be implemented with the help of standard blocks of analog computers. The corresponding block diagram of a quasioptimal control system for damping of random oscillations is shown in Fig. 24, where 1 and 2 denote direct-current amplifiers with amplification factors
gg/32 " 45
164
Chapter III
In conclusion, we dwell on another statement that follows from the calculations of the first approximation.
Namely, all expressions with a
small parameter contain this parameter in the form of the product ea = £/^TT/3B, This statement concerns the loss function (3.2.31), the switching line (3.2.34), and the formula (3.2.30) for stationary specific losses, which can be written in the form
or, more briefly,
if the condition /?2
fact, by the parameter e/ ' ^JirfjB. If we recall that, by the conditions of problem (3.2.2), the parameter e determines the values of admissible control, then it turns out that this variable need not be small for the method of successive approximations to be efficient. Only the relation between the limits of the admissible control and the intensity of random perturbations B is important. All this confirms our assertion made at the beginning of this chapter that the method of successive approximations considered here is convenient
for solving problems with bounded controls when the intensity of random perturbations is large.
§3.3. Synthesis of quasioptimal controls in the case of correlated noises
Now we shall show how the method of successive approximations studied in this chapter can be used for constructing quasioptimal controls when random actions on the system are not white noises. Instead of the system shown in Pig. 13, we shall consider a stabilization system of a somewhat more general form (see Fig. 25), where in addition to random actions £(<) on the plant we also take into account the noise r)(t] in the feedback circuit.
Let the controlled plant P be described, just as in §3.1, by the system of linear differential equations with constant coefficients (3.3.1)
Approximate Synthesis of Stochastic Control Systems
165
y(t)
FIG. 25 where XT = (zi, . . ., xn), UT = (MI, . . . , ur), (,T (t) = (&(<), . . .,&«(<)), and the constant matrices vl, Q, and cr are of dimensions n x n, n x r, and
n x ?7i, respectively. Block 1 in Fig. 25 is assumed to be a linear inertialess device described by the equation
y(t) = Cx(t) + Dr](t),
(3.3.2)
where yT = (j/i, . . . , yi, r = (771, . . . , rjt), and C and D are constant matrices of dimensions I x n and I x I, respectively, (det D ^ 0). The goal of control is to minimize a functional of the form - E
c(x(t},u(t}}dt\.
(3.3.3)
We assume that the random perturbations £(<) and r](t) affecting the system are independent diffusion processes with drift coefficients
= Hr),
(3.3.4)
and matrices of local diffusion coefficients B^ and B^ (G and B^ are mxm dimensional constant matrices; H and Bn are £ x £ dimensional constant matrices; the matrices B£ and B^ are symmetric, 5^ is a nonnegative definite matrix and B^ is a positive definite matrix). It is well known that in this case the diffusion processes £(t) and rj(t) are Gaussian. The stated problem is a special case of the synthesis problem treated in
§1.5. This problem is characterized by the fact that the controlled process x(t) is not a Markov process (in contrast, say, with problems considered in §3.1 and §3.2; moreover, x(t) is a nonobservable process), and therefore, to describe the controlled system shown in Fig. 25, we need a special space Xt
Chapter III
166 of states. This space was called the space of sufficient (see
coordinates in §1.5
also [171]). As was shown in §1.5, in this case, as sufficient coordinates,
we must use a system of parameters that determine the current a posteriori probability density of nonobserved stochastic processes:
The a posteriori density (3.3.5) satisfies a stochastic partial differential equation, which is a special case of Eq. (1.5.39). It follows from §1.5 that to write an equation for the density (3.3.5), we need to use a priori probability characteristics of the (n + m + £)-dimensional stochastic Markov process It follows from (3.3.1), (3.3.2), and (3.3.4) that the combined process has the drift coefficients
ax = Ax + Qu + a!;, ay — Ayxx Ayyy +
CQu,
(3.3.6)
and the matrix of local variances
B-
0...0 0...0
0 . . . 00 . . . 0 0...0 0...0
0...0 0...0
0...00...0 0...0
0...0 0...0
0...0
0...0
0...0
(3.3.7)
0...0
B
The matrices introduced in (3.3.6) and (3.3.7) are Ay,
= CA-
Ayy
=
DHD-^,
C,
Ayt: = Ca, By = DBnDT.
(3.3.8)
10 In this case, the control u in (3.3.1) is assumed to be a given known vector at each time instant t.
Approximate Synthesis of Stochastic Control Systems
167
Using (3.3.6) and (3.3.7), we obtain the following equation for the a posteriori probability density (3.3.5):11
dp(t,z)
I
92p(t,z)
ot
2*
Ozoz
(
T D—1
—^—- = -Spg ; — — 1 dy SJV
2\
p C
'
OCLy
d ,
,.
' - Sp^-^(azp(t,z))
-,\ , F . T D - LI /
rp
_l
dy -\- op ——Tfr — C.psd-u &v
^y
_
\
+ \y* B (ay - cEpsay)
uz
r
c
Ody
\
/
*
^y '— ^~p» ^P Tj—^r J P\t) Zi.
^y
J
*'
/ O O O\
{o.o.y ]
Here p(i,z) = pt(x,^) denotes the a posteriori density (3.3.5), z denotes the vector (#,£), az is the vector composed of the vector-columns ax and ct£, the matrix Bz is a part of the matrix (3.3.7) consisting of its first (n + m) rows and columns, Eps denotes the a posteriori averaging of the corresponding expressions (that is, the integration with respect to z with the density p(t, z ) ) .
It follows from (3.3.6)-(3.3.8) that the matrix B2 is constant, the components of the vector az are linear functions of z, and the expression in the square brackets in (3.3.9) linearly and quadratically depends on z. Therefore, as shown in §1.5 (see also [170, 175]), the a posteriori density p(t, z)
satisfying (3.3.9) is Gaussian, that is, (t))TK-l(t)(z
-
- *(<))],
(3.3.10)
if the initial (a priori] density p(0, z) = po(z) is Gaussian (this is assumed in the sequel).
Substituting (3.3.10) into (3.3.9), one can obtain a system of differential equations for the parameters ~z and K~* of the a posteriori probability density (3.3.10). One can readily see that this system has the form
V [y-ay(z,y,u)},
K~l = -2K~1BZK-1 - R-1V - VTK~i - W
(3.3.11)
(3.3.12)
(in our special case, the system (1.5.52) acquires the form (3.3.11), (3.3.12)). If instead of K~[ we use the inverse matrix K (which is the matrix of a posteriori covariances), then the system (3.3.11), (3.3.12) can be written in the form
z = az(z, u) + fflB~\y T
- ay(z, y, u)],
K = 1BZ + VK + K V + KWK. n
(3.3.13) (3.3.14)
To derive (3.3.9) from (1.5.39), we need to recall that, according to the notation
used in (1.5.39), the vector Aa coincides with the vector a z , the vector A& with ay, and
the structure of the diffusion matrix (3.3.7) implies the following relations between the matrices: \\Ba/3\\ = Bz, \\Baa.\\ = 0, \\Fffp\\ = B~l, and ||B,p|| = By.
Chapter III
168 Here
= Ayxkx( A V = 0
a G
yrr _
~
I ^yx-°y AT R"1 ^y A
I
AT R-1 A
»£ »
y*
T
(3.3.15)
"1
AT f!~1A
st y
y
where, in turn, kxx, kx£,... are elements of the block covariance matrix K,
kxx
kxe
(the dimensions of a block are determined by the dimensions of its subscripts; for example, kx£ is of dimension n x TO). The loss function for problem (3.3.1)-(3.3.3)
F(t,zt,Kt) = min Eps
:(X(T),W(T))C!T y*
u(r) t
= min Eps
C X T
= z t , JiC(t) =
u(r) t
(3.3.16) is completely determined by the time instant t and the current values of the parameters ( z t , K f ) of the a posteriori density (3.3.10) at this instant of time. It follows from the definition given in §1.5 that (z(f), K(t)) are sufficient coordinates for problem (3.3.1)-(3.3.3). The Bellman equation (1.5.54) for the function (3.3.16) can readily be obtained in the standard way from the Eqs. (3.3.13), (3.3.14) for the sufficient coordinates. However, it should be noted that, in this case, the system
(3.3.13), (3.3.14) has a special feature that allows us to exclude the a posteriori covariance K(t) from sufficient coordinates. The point is that, in contrast, say, with a similar system (1.5.53), the matrix equation (3.3.14) is independent of controls u and in no way related to the system of differential equations (3.3.13) for the a posteriori means ?(<). This allows us first to solve the system (3.3.14) and calculate the matrix of a posteriori covariances K(t) in the form of a known function of time on the entire control
interval 0 < t < T (we solve (3.3.14) with the initial matrix K(0) = K0, where KQ is the covariance matrix of a priori probability density po(z)). If K(t) is assumed to be known, then in view of (3.3.8) and (3.3.15) we
can also assume that the matrix crz in (3.3.13) is a known function of time, crz(t), and the loss function (3.3.16) depends on the set (i,2t). Therefore,
Approximate Synthesis of Stochastic Control Systems
169
instead of Eq. (1.5.54) for the loss function F(t, 2), we have the Bellman equation of the form
= mn
[c(x,v.)N(z,K(t))dz\ v
\
(3.3.17) (here N(Z,K(t)} denotes the normal probability density (3.3.10) with the vector of mean values ~z and the covariance matrix K(t)).
Just as in §3.1 and §3.2, Eq. (3.3.17) becomes simpler if we consider the stationary operating conditions for the stabilization system shown in
Fig. 25. The stationary operating conditions established during a long operating time interval (which corresponds to large time t) can exist if only there exists a real symmetric nonnegative definite matrix Kf such that
K*WK* + K*VT + VK* + 1BZ = 0
(3.3.18)
and this constant matrix K* is an asymptotically stable solution of (3.3.14). Let us assume that this condition is satisfied. Denoting the mean "control losses" per unit time under the stationary operating conditions, as
usual, by 7, we can define the stationary loss function /(z) = Km [F(t, z) - -j(T - t)], T-KX>
for which, from (3.3.17), we derive the time-invariant Bellman equation
[
rt f
r
aj(z, w ) — + / c(x,u)N(z, K*) dz
In (3.3.19) (TZ> is the matrix az (see (3.3.8) and (3.3.15)), where kxx, k^x,... are the corresponding blocks of the matrix Kf determined by (3.3.18). In some cases, it is convenient to solve Eq. (3.3.19) by the method of successive approximations treated in §3.1 and §3.2. The following example shows how this method can be used. Let us consider the simplest version of
the synthesis problem (3.3.1)-(3.3.3) in which Eqs. (3.3.1), (3.3.2) contain scalar variables instead of vectors and matrices. In (3.3.3) we write the penalty function c(x, u) in the form
c(x,u) = x +e~lu2,
(3.3.20)
170
Chapter III
where e > 0 is a small parameter. From the "physical" viewpoint, this penalty function means that the control actions are penalized much more strongly than the deviations of the phase coordinate x(i) of the control system (3.3.1) from the equilibrium state x — 0. For simplicity, we set Q =
x = — ax -\- £ + u + — [ y — (h — a)x — £ — u • (3.3.21)
£, = —£ + —^[y — (h — a)x — £ — u + /it/], Br,
I
^XX
ff
——
^QifoxX
I
^fox^
I
'
_-j
I
5
-^-,
%£, ——
\®>
' 9)*^x£
ffx = (h - a)kxx + kx^
(3.3.22)
Bn
In this case the Bellman equation (3.3.19) has the form
„..,, s
dxdt
+
4
* eeJ
=
,. (3.3.23,
Here the constants ax* and cr^« are ffx,
= (h- a)k*xx + fc* c ,
where the constant covariances k*x, k*x^ and fc|^ form the stationary solution of the system of differential equations (3.3.22). Passing to the new variables x± = (^/B^/
(3.3.24)
where r =
\+ f x\N(bx1,k*,)dx = f , (3.3.25) J J
Approximate Synthesis of Stochastic Control Systems
171
where b = &x* /\/B^. Taking into account the formulas r°°
I
x\N(bx-L,klx)dx
J — CO
°°
/
x[exp -00
1
--——(v-bx *KXX
V
_ ^rt T.L* ) +I 6' a . 1•*•| Il _ 2 F (\ _ - ^/ rL JT (3.3.26)
27T J -
and minimizing the expression in the square brackets, we obtain from (3.3.25) the optimal control for the stationary stabilization conditions:
uf(xi, x2) = ~~~(xii
a;2
)'
(3.3.27)
where the function /(KI,^) satisfies the nonlinear elliptic equation l, x 2 )
= 7 - "(xi) +
2
-
(3-3.28)
Equation (3.3.28) is similar to Eqs. (3.1.8) and (3.2.7), therefore, in this case, we can use the same method of approximate synthesis as in §3.1 and §3.2. Then the quasioptimal control U]t(xi,X2) in the fcth approximation is determined by the formula
uk(xi,x2) = - — -/^(x1,X2), 20
where the functions fk(xi,x2) approximations
k = 0,1,2,...,
(3.3.29)
satisfy the linear equations of successive
Lfk(xi,x2)=uk(xi,X2),
* = 0,1,2,...,
(3.3.30)
, x2) = U>Q(XI) = 7° -
In this case, the calculations of successive approximations fk(xi,x2) are completely similar to those discussed in §3.1 and §3.2. Therefore, here
Chapter III
172
we restrict our consideration to a brief description of the calculation of fk(xi,xz). We only dwell upon the distinctions in formulas. The operator (3.3.24) can be written in the form (3.1.5) if A = \\Aij\\\ and B = \\Bij\\^ in (3.1.5) are understood as the matrices
A=
1 1 1 1
—a 0
-g
The stationary density po(x) satisfying (3.1.14) has the form (3.1.15), and the matrices P and P~1, as one can readily see, have the form
P ~_ A ,/2
-afj,p
-a/ip -(r +
(3.3.31)
p/ > 1 , = a + g, v — a- g — r, and p = r + 2g). Using (3.3.31), we can find the matrix (see (3.1.35)) p[cf-(r + gY}
p(1a? - 2ar — gr) - g/j,2
(3.3.32)
as well as the matrix p/2gfj, l/2g
V =
(3.3.33)
By the similarity transform, this matrix reduces the matrix (3.3.32) to the diagonal form
\i 0
0 A2
= -a,
A 2 = -g.
(3.3.34)
It follows from (3.1.44), (3.1.51), and (3.1.55) that solutions of the equations of successive approximations (3.3.30) can be represented as the series (3.3.35)
where Him(xi,X2) are two-dimensional Hermitian polynomials calculated __ _ i __ _ -i by the formulas (3.2.17) with y = V x (the matrix V is inverse to (3.3.33)).
Approximate Synthesis of Stochastic Control Systems
173
The coefficients aktm are calculated by the formula (see (3.2.19))
fc
_ ——tfm——_——^m_ ) l\i + m\2 la + mg
(3.3.36)
and the coefficients
,,
00
I 0 /* X* det^P G/ m (/i) ^=T7Tpx ex? [ - ^(xTPx)]uk(x) dXldx2 !m, JJ
(3.3.37) are expressed in terms of the group of Hermitian polynomials Gim(x\, x-z) orthogonal to Him(xi, x-i) and calculated by (3.2.18). Parallel to the calculations of the successive approximations to the loss
function (3.3.35), we calculate specific stationary losses -yfc (corresponding to the Mh approximation) from the condition &QO = 0. So, in the zero approximation we have
hence, performing simple calculations and taking into account (3.3.31), we obtain
g) 2
Next, using the obtained value of 7° and formulas (3.3.26), (3.3.30), (3.3.36), and (3,3.27), we can calculate any desired number of coefficients a°m in the series (3.3.35). With the help of these coefficients, we can construct an approximate expression for the function fo(xi,X2), which allows
us to derive an explicit formula for the quasioptimal control algorithm UQ(XI,X^) in the zero approximation and to calculate the variables -y1, /i (£1,2:2), and MI (3:1,2:2) related to the first approximation. Here we write explicit formulas neither for fo(xi, Xi) nor for /i(a;i, £2), since they are very cumbersome. We only remark that in this case all quasioptimal control algorithms (3.3.29) are nonlinear functions of the phase variables (2:1,2:2); moreover, the character of nonlinearity is determined by
the number of terms left in the series (3.3.35) for the calculations.
Chapter III
174
Thus, from the preceding it follows that the methods for calculations of stationary operating conditions of the stabilization system (Fig. 13) can readily be generalized to the case of a more general system with correlated noise (Fig. 25) if the noise is a Gaussian Markov process. In this case, the
optimal system is characterized by the appearance of an optimal filter in the regulator circuit; this filter is responsible for the formation of sufficient coordinates. In our example (Fig. 25), where x, y, u, £, and f] are scalar, this filter is described by Eqs. (3.3.21). The circuit of functional elements of this closed-loop control system is shown in Fig. 26.
f
——>
" 1 ' -, >
—J
s(i )
Jo
"NT1
M
x(<)
W -*
t( t) ^\
3
Zi
<—
y
5^
Jo
,_ 4<
6
<
7<
f—
d/dt
'
8-
y(*)
!>?(* x
«—
FIG. 26 Blocks P and 1 are units of the initial block diagram (Fig. 25). The rest of the diagram in Fig. 26 determines the structure of the optimal controller. One can see that this controller contains standard linear elements of analog computers such as integrators, amplifiers, adders, etc. and one nonlinear converter NC, which implements the functional dependence (3.3.29).
Units of the diagram marked by ">" and having the numbers 1,2,... ,8 are amplifiers with the following amplification factors Kt:
K5 = -1,
K6 = a - h,
Kr = -1,
K& = h.
Approximate Synthesis of Stochastic Control Systems
175
§3.4. Nonstationary problems. Estimates of the quality of approximate synthesis
3.4.1. Nonstationary synthesis problems. If equations of a plant
are time-dependent or if the operating time T of a system is bounded, then the optimal control algorithm is essentially time-varying, and we cannot find this algorithm by using the methods considered in §§3.1-3.3. In this case, to synthesize an optimal system, it is necessary to solve a time-varying Bellman equation, which, in general, is a more complicated problem. However, if the plant is governed by a system of linear (time-varying) equations, then we can readily write solutions of the successive approximation equa-
tions (3.0.6), (3.0.7) in quadratures. Let us show how this is done. Just as in §3.1,
problem for the stabilization system (Fig.
we consider the synthesis
13) with a plant P described by
equations of the form
x = A(i)x + Q(t)u
+ ar(t)£(t),
(3.4.1)
where x is an n-dimensional vector of phase coordinates, u is an r-dimensional vector of controls, A(t), Q(t), and
n-dimensional standard white noise (1.1.34). To estimate the quality of control, we shall use the following criterion of the type of (1.1.13): (3.4.2)
and assume that the absolute values of the components of the control vector u are bounded by small values (see (3.1.3)):
K <£umi,
i= l,...,r.
(3.4.3)
According to (3.1.6) and (3.1.7), the optimal control u*(t,x) for problem (3.4.1)-(3.4.3) is given by the formula u,(t,x) = -{ew m i,...,£« mr }sign ( QT(t)—-(t,x) ) , \ dx / where the loss function F(t,
(3.4.4)
x) satisfies the equation
LtiXF(t, x) = -c(x)
- £$ (t, —— (<, x)}, \ ox )
(3.4.5)
176
Chapter III
with Lt,x denoting a linear parabolic operator of the form
—+-1
d
fit* L/tt-
*} LJ
For the function 3>(t,dF/dx], we have the expression (3.4.7)
In this case, the function F(t,x) must satisfy (3.4.5) for all x £ R n , 0 < t < T, and be a continuous continuation of the function (3.4.8)
as t -> T (see (1.4.22)). The nonlinear equation (3.4.5) is similar to (3.0.5) and, according to (3.0.6) and (3.0.7), can be solved by the method of successive approximations. To this end, we need to solve the sequence of linear equations
Lt,,F0(t,x) = -c(x),
Lt,,Fk(t, x) = -c(x)
(3.4.9)
- e$ (t, ^±(t, x)}, dx V >
k = 1, 2, . . . (3.4.10)
(all functions Fk(t,x) determined by (3.4.9) and (3.4.10) must satisfy condition (3.4.8)). Next, if we take Fk(t,z) as an approximate solution of Eq. (3.4.5) and substitute Fk into (3.4.4) instead of F, we obtain a quasioptimal control algorithm Uk(t, x) in the fcth approximation. Let us write the solutions Fj. (i, a;), k = 0, 1, 2, . . . , in quadratures. First, let us consider Eq. (3.4.9). Obviously, its solution Fo(t,x) is equal to the value of the cost functional
c(x(T))dT+i>(x(T))
x(t)=x
(3.4.11)
on the time interval [t, T] provided that there are no control actions. In this case, the functional on the right-hand side of (3.4.11) is calculated along the trajectories x(r), t < r < T, that are solutions of the system of stochastic
differential equations
X = A(T)X
+
x ( t ) = x,
describing the uncontrolled motion of the plant (u = 0 in (3.4.1)).
(3.4.12)
Approximate Synthesis of Stochastic Control Systems
177
It follows from §1.1 and §1.2 that the solution of (3.4.12) is a continuous Markov process X(T) (a diffusion process). This process is completely determined by the transitive probability density function p(x, t; z, T), which determines the probability density of the random variable z = X(T) if the stochastic process x(t) was in the state x(f) — x at the preceding time moment t. Obviously, by using p(x,t;z,r), we can write the functional (3.4.11) in the form
FQ(t,x)= f
dr I
Jt
p(x,t-z,T)c(z)dz+
Jnn
f
p(x,t;z,T)4>(z)dz.
(3.4.13)
JRn
On the other hand, we can write the transitive density p(x, t; z, T) for the diffusion process X(T) (3.4.12) as an explicit finite formula if we know the
fundamental matrix X(t, T) for the nonperturbed (deterministic) system z = A(t)z. Indeed, since Eqs. (3.4.12) are linear, the stochastic process X(T) satisfy-
ing this equation is Markov and Gaussian. Therefore, for this process, the transitive probability density has the form
p(x,t;z,T) = [(2ir)B det DJ-^expI-f (z - a)TD~l(z - a)], where a = Ez = E(X(T)
(3.4.14)
x ( t ] = a;) is the vector of mean values and
D = E[(z — Ez)(z — Ez) T ] is the covariance (dispersion) matrix of the random vector z = X(T). On the other hand, using the fundamental matrix X(t,T)12 we can write the solution of system (3.4.12) in the form (the Cauchy formula)
x(r) = X(t,r)x+ f Jt
X(s,T)
Hence, performing the averaging and taking into account properties of the white noise (1.1.34), we obtain the following expressions for the vector a
and the matrix D:
a = Ex(r)=X(t,T)x,
(3.4.15) T T
D = E [ ( x ( r ) - a))((xx((rr))--aa)) ]
!
Jt
X(s,T)o-(s)Et(s)£T(S')o-T(s')XT(s',T)dsds'
12
Recall that the fundamental matrix X(t,r),T > t, is a nondegenerate nxn matrix
whose columns are linearly independent solutions of the system Z ( T ) = A(T)Z(T), so
that X(t, t) = E, where E is the identity matrix. Methods for constructing fundamental matrices and their properties are briefly described on page 101 (for details, see [62, 111]).
178
Chapter III X(s,T)cr(s)6(s'
-s)
X(s,T)B(s)XT(s,T)ds,
=
B(s)=a(s}ffT(s).
J*
(3.4.16)
Formulas (3.4.13)-(3.4.16) determine the solution F0(t,x) of the zeroapproximation equation (3.4.9), satisfying (3.4.8), in quadratures. It follows from (3.4.13)-(3.4.16) that the function F0(t,x) is infinitely many times differentiable with respect to the components of the vector x if the functions c(z) and ij)(z) belong to a rather wide class (it suffices that the functions c ( z ) exp(— |z T D~ 1 z) and ^>(z) exp(— ^zTD~iz) were absolutely integrable [25]). Therefore, by analogy with (3.4.13), we can write the solution Fk(t,x) of the successive approximation equations (3.4.10), satisfying
(3.4.8), in the form
Fk(t,x)=
dr Jt
JRn
= l,2,...
(3.4.17)
To obtain explicit formulas for the functions Fo(t, x), Fi(t, x),. .., which allow us to write the quasioptimal control algorithms Uo(t, x), «i(<, a;), ...
as finite analytic formulas, we need to have the analytic expression of the matrix X(t,r) and to calculate the integrals in (3.4.13) and (3.4.17). For autonomous plants (the case where the matrix A(t) in (3.4.1) and (3.4.12) is constant, A(t) = A = const), the fundamental matrix X(t,r) has the form of a matrix exponential:
X(t,r) = e A ( r -*>
(3.4.18)
whose elements can be calculated by standard methods. On the other hand, it is well known that fundamental matrices of nonautonomous systems can be constructed, as a rule, by numerical methods.13 Thus for A(t) ^ const, it is often difficult to obtain analytical results. If the plant equation (3.4.1) contains a constant matrix A(i) EE A —
const, then formulas (3.4.13) and (3.4.17) allow us to generalize the results obtained in §§3.1-3.3 for the stationary operating conditions to the timevarying case. For example, let us consider a time-varying version of the
problem of optimal damping of random oscillations studied in §3.2. 13 Examples of special matrices A(t) for which the fundamental matrix of the system x = A(t)x can be calculated analytically, can be found, e.g., in [139].
Approximate Synthesis of Stochastic Control Systems
179
Just as in §3.2, we shall consider the optimal control problem (3.2.1)(3.2.3). However, in contrast with §3.2, we now assume that the terminal time (the upper limit T of integration in the functional (3.2.3)) is a finite
fixed value. By writing the plant equation (3.2.1) in the form of system (3.2.4), we see that problem (3.2.1)-(3.2.3) is a special case of problem
(3.4.1)-(3.4.3) if
0
4= a(t) = a =
0
1
Q(t) = Q = 1
-i -p
0
0
C(x) =
o VB
2 2 Xl+X2,
i>(3
(3.4.19) = 0.
Therefore, it follows from the general scheme (3.4.4)-(3.4.10) that in this case the optimal control has the form
u#(t,xi,x2) = -esign f - — ( t , x i , x 2 ) 1,
(3.4.20)
where for 0 < t < T the function F(t,xi,x2) satisfies the equation
Lt>xF(t, xi, x2) = -x\-
(3.4.21)
0z 2 V "
and vanished at the terminal point, that is,
F(T,X!,x2) = Q.
(3.4.22)
According to (3.4.6) and (3.4.13), the operator Lt,x in (3.4.21) has the form
d
d Lt,x = ^T
8
B d2 2 dx\
-—— + — ——r.
dx2
Let us calculate the loss function Fg(t, xi, x2) of the zero approximation. In view of (3.4.9), (3.4.21), and (3.4.22), this function satisfies the linear equation
with the boundary condition
F0(T,xi,x2) = Q.
(3.4.24)
According to (3.4.13), the function F0(t,Xi,x2) can be written in quadratures ,-T
F0(t,xi,x2)=
dr Jt
(3.4.25)
Chapter III
180
where the transition probability density p(x,t;z, T) is given by (3.4.14). It follows from (3.4.15) and (3.4.16) that to find the parameters of the transition density we need to calculate the fundamental matrix (3.4.18). Obviously, the roots AI and A2 of the characteristic equation dei(A — \E) = 0 of the matrix A given by (3.4.19) are
Ai,2 = --
From this and the Lagrange-Silvester formula [62] we obtain the following expression for the fundamental matrix (3.4.18) (here p = (T — t ) ) :
-(A-X1E) -(A-X2E) (Ai-A2 (A2• sin Sp + 6 cos 6p sin 6p — sin 6p 6 cos Sp — | sin 6p (3.4.26)
X(t,r) =
It follows from (3.4.15), (3.4.16), and (3.4.26) that in this case the vector of means a and the variance matrix D of the transitive probability density (3.4.14) have the form
x\ cosSp + |(a:2 + f
(3.4.27)
On
D12
_B
Di 2 B
D 22
25' 6
B
(3.4.28)
Substituting (3.4.14) into (3.4.25) instead ofp(x,t;z,r), integrating, taking into account (3.4.27) and (3.4.28), and performing some easy calcula-
Approximate Synthesis of Stochastic Control Systems
181
tions, we obtain the following final expression for the function Fo(t, x^, x2):
xi
-l-
B(/32+4:); ^2
—e-^sm2S-p(B - 2(3x1 - 4xlX2) 80
(3.4.29)
"? cos 2 W/32 - 4)z? + 40*1*2 + 4zi - 5/3], where ~p = T — t. Let us briefly discuss formula (3.4.29). If we consider the terms on the right-hand side of (3.4.29) as function of "reverse" time ~p — T — t, then these terms can be divided into three groups: infinitely increasing, damping, and independent of ~p as ~p —> oo. These three types of terms have the following physical meaning. The only infinitely growing term (B/(3)~p in (3.4.29) shows how the mean losses (3.4.11) depend on the operating time in the mode of stationary operating conditions. Therefore, the coefficient B//3 has the meaning of the specific mean error 7, which was calculated in §3.2 by other methods and for which we obtained 7° = B//3 in the zero approximation (see (3.2.21)). Next, the terms independent of ~p (in the braces in (3.4.29)) coincide with the expression for the stationary loss function obtained in §3.2 (formula (3.2.26)). Finally, the damping terms in (3.4.29) characterize the deviations of operating conditions of the control system from the stationary ones. Using (3.4.29), we can approximately synthesize the optimal system in the zero approximation, where the control algorithm uo(t, Xi,x2) has the form (3.4.20) with F replaced by FQ. The equation
8F0
=
2x2
— I"
I
2x
+
2cos 2Sp(/3xi + x2) = 0
(3.4.30)
determines the switching line on the phase plane (xi, x 2 ). Formula (3.4.30) shows that this is a straight line coinciding with the K-axis as ~p —)• 0 and rotating clockwise as ~p -> oo (see Fig. 27) till the limit value xi + 2x2/(3 — 0 corresponding to the stationary switching line (see (3.2.27)). Formulas (3.4.29) and (3.4.30) also allow us to estimate whether it is important to take into account the fact that the control algorithm is timevarying. Indeed, (3.4.29) and (3.4.30) show that deviations from the stationary operating conditions are observed only on the time interval lying at
Chapter III
182
FIG. 27 the distance ~ (3~l from the terminal time T. Thus, if the general operating time T is substantially larger than this interval (say, T 3> 3//3), then we can
use the stationary algorithm on the entire interval [0,T], since in this case the value of the optimality criterion (3.2.3) does not practically differ from the optimal value. This fact is important for the practical implementation of optimal systems, since the design of regulators with varying parameters is a rather sophisticated technical problem. 3.4.2. Estimates of the approximate synthesis performance. Up to this point in the present chapter, we have studied the problem of how to find a control syste close to the optimal one by using the method of successive approximations. In this section we shall consider the problem of how the quasioptimal system constructed in this way is close to the optimal
system, that is, the problem of approximate synthesis performance. Let us estimate the approximate synthesis performance for the first two (the zero and the first) approximations calculated by (3.0.6)-(3.0.8). As an example, we use the time-varying problem. (3.4.1)-(3.4.3). We assume that
the entries of the matrices A(t), Q(t), and cr(i) in (3.4.1) are continuous functions of time defined on the interval 0 < t < T. We also assume that the penalty functions c(x) and i}>(x) in (3.4.2) are continuous and bounded for all x £ R n . Then [124] there exists a unique function F(t, x) that satisfies
the Cauchy problem (3.4.5), (3.4.8) for the quasilinear parabolic equation (3.4.5)14 This function is continuous in the strip HT = {\x\ < oo, 0 < t < T} 14
We shall use the following terminology: Eq. (3.4.5) is called a quasilinear (semi-
linear) parabolic equation, the problem of solving Eq. (3.4.5) with the boundary condi-
Approximate Synthesis of Stochastic Control Systems
183
and continuously differentiable once with respect to t and twice with respect to x for 0 < t < T; its first- and second-order derivatives with respect to x are bounded for x 6 HTOne can readily see that in this case
\F(t,x)-F1(t,x)\~e2,
\F(t,x)-F0(t,x)\~e,
(3.4.31)
and hence, for small £, the functions F0(t, x) and Fi(t, x) nicely approximate
the exact solution of Eq. (3.4.5). To prove relations (3.4.31), let us consider the functions So(t,x) = F(t, x) - F0(t, x) and Si(t, x) = F(t, x) - Fi(t, x). It follows from (3.4.5), (3.4.9), and (3.4.10) that these functions satisfy the equations
LS0 = -e* (t, ^-} ,
S0(T, x) = 0,
(3.4.32)
(3.4.33)
Equations (3.4.32) and (3.4.33) differ from (3.4.9) only by the expressions on the right-hand sides and by the initial data. Therefore, according to
(3.4.13), the functions So and Si can be written in the form T
/»
/)/?
dr /
/
S1(t,x)=ef Jt
\
p(x,t;z,T)$ (r,—— (T,Z) ) dz, v
jRn
az
(3.4.34)
'
dr f p (z,i;z,r)[*(r,^)-*(T,^)]dz. v Jnn L v Oz ' J (3 4 35)
Since the function $ is continuous (see (3.4.7)) and the components of the vector dF/dx are bounded, we have |$(T, dF/dz)\ < P for all r 6 [0, T]; hence, we have the estimate
\S0(t ,x)\<e
dr I Jt
p(x,t;z,T)$
dz,
jRn
<eP Ji
dr I p(x,t;z,T)dz = •/R«
(3.4.36)
tion (3.4.8) is called the Cauchy problem, and the boundary condition (3.4.8) itself is sometimes called the "initial" condition for the Cauchy problem (3.4.5), (3.4.8). This terminology corresponds to the universally accepted standards [61, 124] if (as we shall do in §3.5) in Eq. (3.4.5) we perform a change of variables and use the "reverse" time p = T — t instead of t. In this case, the backward parabolic equation (3.4.5) becomes a "usual" parabolic equation, and the boundary value problem (3.4.5), (3.4.8) takes the form of the standard Cauchy problem.
Chapter III
184
The first relation in (3.4.31) is thereby proved. To prove the second relation in (3.4.31), we need to estimate the difference S'0 = (dF/dxi) - (dF0/dxi). To this end, we differentiate (3.4.32) with respect to x;. As a result, we obtain the following equation for the function SQ: (3.4.37)
(in fact, the derivative on the right-hand side of (3.4.37) is formal, since the
function $ (3.4.7) is not differentiable). Using (3.4.13) for SQ, we obtain T
C
dr
(3.4.38)
JRn
Integrating (3.4.38) by parts with respect to z,- and taking into account (3.4.14) and (3.4.15), we arrive at
S'0 = £ f
Jt
dr I
Jnn
p(x,t;z,r)D-j1(zj
-Xjk(t,r)xk}$(r,
-f) dz. (3.4.39)
\ oz /
From (3.4.39) we obtain the following estimate for S10:
\So\ < £P I Jt
dr
I ^i? \zi ~ XJk(t, r)xk\p(x, t; z, T) dz = ePVi, Jnn .
(3.4.40)
J=l Now we note that since Q(i) in (3.4.7) is bounded, the function $(<, satisfies the Lipschitz condition with respect to y:
(3.4.41) Using (3.4.40), (3.4.41), and (3.4.35), we obtain dz <sN f Jt
dr f
p(x,t;z,T}y^\S*0\dz
JRn
i=1
n
<£2NPV(T-t),
V
(3.4.42) 1=1
Approximate Synthesis of Stochastic Control Systems
185
which proves the second relation in (3.4.31). In a similar way, we can also estimate the difference dF/dxi — dFi/dxt = S{. Indeed, just as (3.4.39) was obtained from (3.4.32), we use (3.4.33) to obtain
S{=e ?{=ef Jt
dr dr f
p(x,t-,z,T)D(z-Zkxk)*T,-*T,
dz.
JRn
(3.4.43) This relation and (3.4.40), (3.4.41) for the function S{ readily yield the estimate f> 17
|S1I =
fl Z7
< £2PVViN,
(3.4.44)
which we shall use later. According to (3.0.8), in this case the quasioptimal controls uo(t, x) and MI(<, x) are determined by (3.4.4), where instead of the loss function F(t, x) we use the successive approximations F0(x,t) and Fi(x,t), respectively. By Go(t, a:) and Gi(x,t) we denote the mean values of the functional (3.4.11) calculated on the trajectories of the system (3.4.1)
x = A(T)X + Q(T)MJ(T, a;) +
v(+\ —T \ I — 5
i — 0 1 — ' '
with the use of the quasioptimal controls uo(t, x) and u-\_(t, x). The functions Gi(t, a;), i — 0, 1, estimate the performance of the quasioptimal control algorithms Wi(<, x), i = 0,1. Therefore, it is clear that the approximate synthesis may be considered to be justified if there is only a small difference between the performance criteria Go(t, x) and Gi(i, x) of the suboptimal systems and the exact solution F(t, x} of Eq. (3.4.5) with the initial condition (3.4.8). One can readily see that the functions GO and GI satisfy estimates of type (3.4.31), that is,
\F(t1x)-G0(t1x)\~£,
\F(t,x)-G1(t,x)\~e2.
(3.4.45)
Relations (3.4.45) can be proved by analogy with (3.4.31). Indeed, the functions GO and GI satisfy the linear partial differential equations [45, 157]
LGi(t,x) = -c(x)-euf(t,x)QT(t)-(t,x),
(3.4.46)
Chapter III
186
This fact and (3.4.9), (3.4.10) imply the following equations for the functions H0 = F0 - Go and HI = FI - Gj.:
H0(T, x) = 0,
LH0 =
LH^^W^-*(t^}\,
(3.4.47)
F1(r,,) = o.
(3.4.48)
Since wf Q T '^j£- = $(*, ^")> Eq. (3.4.48) can be rewritten as follows:
(3.4.49) It follows from (3.4.4) that Eqs. (3.1.46), (3.4.49) are linear parabolic equations with discontinuous coefficients. Such equations were studied in [80, 81, 144]. It was shown that if, just as in our case, the coefficients in (3.1.46), (3.1.49) have discontinuities of the first kind, then, under our assumptions about the properties of A(t), Q(t), c ( x ) , and ip(x), the solutions of Eqs. (3.4.46), (3.4.49) and their first-order partial derivatives are bounded.
Using this fact, we can readily verify that the right-hand sides of (3.4.47) and (3.4.49) are of the order of e and e 2 , respectively. For Eq. (3.4.47), this statement readily follows from the boundedness of the components of the
vectors dGo/dx and MO and the elements of the matrix Q. The right-hand side of (3.4.49) can be estimated by the Lipschitz condition (3.4.41) and the inequality
dFi
dF0
8F
dF
which follows from (3.4.40) and (3.4.44). Therefore, for the functions H0 and HI we have
\HO\
•e2.
(3.4.50)
To prove (3.4.45), it suffices to take into account the inequalities |F-Go
FO-GO\< F- FO\ + \HO\,
\F-Gi and to use (3.4.31) and (3.4.50). Thus, relations (3.4.45) show that if the Bellman equation contains a small parameter in nonlinear terms, then the difference between the quasioptimal control system calculated by (3.0.6)-(3.0.8) and the optimal control system is small and, for sufficiently small e, we can restrict our calculations to a small number of approximations. We need either one (the zero)
Approximate Synthesis of Stochastic Control Systems
187
or two (the zero and the first) approximations. This depends on the admissible deviation of the quasioptimal system performance criteria Gi(t, x) from the loss function F(t,x). In conclusion, we make two remarks about (3.4.45).
REMARK 3.4.1. One can readily see that all arguments that lead to the estimates (3.4.45) remain valid for any types of nonlinear functions in (3.4.5) that satisfy the Lipschitz condition (3.4.41). Therefore, in particular, all statements proved above for the function <S> (3.4.7) automatically hold for equations of the form (3.0.4) with r-dimensional ball taken as the set U of admissible controls, instead of an r-dimensional parallelepiped. D REMARK 3.4.2. The estimates of the approximate synthesis accuracy considered in this section are based on the assumption that the solutions of the Bellman equation and their first-order partial derivatives are bounded. At first glance it would seem that this assumption substantially narrows the class of problems for which the approximate synthesis procedure (3.0.6)(3.0.8) can be justified. Indeed, the solutions of Eqs. (3.4.5), (3.4.9), (3.4.10), and (3.4.46) are unbounded for any x G Rn if the functions c(x) and i>(x) infinitely grow as x —>• oo. Therefore, for example, we must eliminate frequently used quadratic penalty functions from consideration. However, if we are interested in the solution of the synthesis problem in a given bounded region XQ of initial states x(0) of the control system, then the procedure (3.0.6)-(3.0.8) can also be used in the case of unbounded penalty functions. This statement is based on the following heuristic arguments. Since the plant equation (3.4.1) is linear and the matrices A(t), Q(t), and ff(t) and the control vector u are bounded, we can always choose a sufficiently large number R such that the probability
P{sup 0 < t < T \x(t)\ > R} becomes arbitrary small [11, 45, 157] for any fixed domain XQ of the initial states x ( Q ) . Therefore, without loss of accuracy, we can replace the unbounded functions c(x) and i{>(x) in (3.4.2) (if, in a certain sense, these functions grow as x\ — R —> oo slower than the probability P{sup 0 < t < T x(i)\ > R} decreases as R —>• oo) by the expressions
c(x] maXj; c(x) \x\ = R,
for for for
ij)(x)
for
\x\ < R, \x\ > R, x
< R, > R,
\x\ = R, for which the solutions of Eqs. (3.4.5), (3.4.9), (3.4.10), and (3.4.46) satisfy the boundedness assumptions. D
188
Chapter III
The question of whether procedure (3.0.6)-(3.0.8) can be used for solving the synthesis problems with unbounded functions c(x) and i/>(x) in the functional (3.4.2) will be rigorously examined in the next section. §3.5. Analysis of the asymptotic convergence of
successive approximations (3.0.6)— (3.0.8) as k —>• oo
The method of successive approximations (3.0.6)-(3.0.8) can also be used for the synthesis of quasioptimal control systems if the Bellman equation
does not contain a small parameter in nonlinear terms. Needless to say that (in contrast with Section 3.4.2 in §3.4) the first two approximations, as a rule, do not approximate the exact solution of the synthesis problem sufficiently well. We only hope that the suboptimal system synthesized on
the basis of (3.0.9) is close to the optimal system for large k. Therefore, we need to investigate the asymptotic behavior as k —>• oo of the functions
Fk(t,x) and U k ( t , x ) in (3.0.6)-(3.0.8). The present section deals with this problem. Let us consider the time-varying synthesis problem of the form (3.4.1)™ (3.4.3) in a more general setting. We assume that the plant is described by
the vector-matrix stochastic differential equation of the form
x = a(t, x) + q(t)u + a(t, x ) £ ( t ) .
(3.5.1)
Here x is an n-dimensional vector of phase coordinates of the system, u is an r-dimensional vector of controls, £(t) is an n-dimensional vector of random actions of the standard white noise type (1.1.34), a(t, x) is a given vectorfunction of the phase coordinates x and time t, and q(t) and
such that for t > t0, 0 < t0 < T, the stochastic equation (3.5.1) has a unique solution x(t) satisfying the condition x(t0) = XQ at least in the weak sense (see §IV.4 in [132]). As an optimality criterion, we take the functional (3.4.2),
U
T
-I
c(x(t))dt + il>(x(T))\. J
(3.5.2)
Here c ( x ) and ij)(x) are given nonnegative scalar penalty functions whose
special form is determined by the character of the problem considered (the requirements on c ( x ) and ij)(x) are given later). The constraints on the domain of admissible controls have the form
(1.1.22), weC/,
(3.5.3)
Approximate Synthesis of Stochastic Control Systems
189
where U C Rr is a closed bounded convex set in the Euclidean space R r . It is required to find a function it, = ut(t,x(t)} satisfying (3.5.3) such that the functional (3.5.2) calculated on the trajectories of system (3.5.1) with the control M* attains its minimum value.
In accordance with the dynamic programming approach, solving this problem is equivalent to solving the Bellman equation that, for problem
(3.5.1)-(3.5.3), reads (see §1.4)
Here a(t, x] is a column of functions with components (see (1.2.48))
-—-^ami, Z Oxm
t=l,...,n.
(3.5.5)
Recall that we assumed in §1.2 that throughout this book all stochastic differential equations written (just as (3.5.1)) in the Langevin form [127] are symmetrized [174].
By definition, the loss function F in (3.5.4) is equal to
F = F(t,x) = min u(T)£U
Here E[(-)
T ( r r E-H / C(X(T))
dr
llJt
x(t) = x] means averaging over all possible realizations of the
controlled stochastic process x(r) =; XU^(T) (r > t) issued from the point
x at T = t. It follows from (3.5.6) that
F(T,x) = j(x).
(3.5.7)
Passing to the "reverse" time p = T — t, we transform Eq. (3.5.4) and the condition (3.5.7) to the form
LF(p,x) = -c(x) - mm \uTqT(p)~(p, x ] \ , F(0,z) = '0(z).
In (3.5.8) we have the following notation:
(3.5.8) (3.5.9)
190
Chapter III
ai(p,x) = a,i(x,T — p), q(p) =q(T — p), b t j ( p , x ) is a general element of the matrix |(T — p, x)'aT (T — p, x) and, as usual, the sum in (3.5.10) (just as in (3.5.5)) is taken over repeated indices from 1 to n. Assuming that the gradient dF/dx of the loss function is a known vector and calculating the minimum in (3.5.8), we obtain
x
c(x).
(3.5.11)
In addition, we obtain the function
u*(p,x) =
(3.5.12)
that satisfies the condition T T, ^^F
dF
~ and solves the synthesis problem (after we have solved Eq. (3.5.11) with
the initial condition (3.5.9)). The form of the functions
Eq. (3.0.5) only by a small parameter (there is no small coefficient e of the function $). Nevertheless, in this case, we shall also use the approximate synthesis procedure (3.0.6)-(3.0.8) in which, instead of the exact solution F(p,x) of Eq. (3.5.11), we take the sequence of functions F Q ( P , X ) , F I ( P , X ) , . . . recurrently calculated by solving the following sequence of linear equations:
) = i>(x),
(3.5.13) (3.5.14)
The successive approximations u0(p, x), Ui(p, x), . . . of control are determined by the expressions
fc = 0 , l , . . .
(3.5.15)
Below we shall find the conditions under which the recurrent procedure (3.5.13)-(3.5.15) converges to the exact solution of the synthesis problem.
Approximate Synthesis of Stochastic Control Systems
191
Let us consider Eq.(3.5.11) with the operator L determined by (3.5.10). The solution F(p, x) and the coefficients bij(p, x) and a,(p, x) of the operator L are denned on UT = {[0, T] x R n } = {(p, x): 0 < p < T, x e R»}. We assume that everywhere in UT the matrix \\bij(p, x)\\" satisfies the con-
dition that the operator L is uniformly parabolic, that is, everywhere in UT for any real vector x
we
have
AM2<X%>,*)x
(3-5.16)
where A. and A are some positive constants. Moreover, we assume that the functions bij(p, x) and o;(/9, x) are bounded in E/r, continuous in both variables (p, x), and satisfy the Holder inequality with respect to x uniformly
in p, that is,
\bij(p,x)-bii(p,x°)\
(3.5.17) 0 < a < 1,
We assume that the functions c ( x ) , if>(x), and 3>(p,dF/dx)
A = const.
are continuous
and that c(x) and if>(x) satisfy the following restrictions on the growth
as x
• oo:
(3.5.18) (h is a positive constant). We also assume that the function $(/?, v) satisfies the Lipschitz condition with respect to v — (vi,...,vn) uniformly in p G
[0,2*], that is,
\*(p,v)-*(p,v°)\
$(p,0) = 0.
(3.5.19)
In particular, the functions $ from (3.4.7) and (1.3.23) satisfy (3.5.19). The following three consequences from the above assumptions are well
known [74]. (1) There exists a unique fundamental solution G(x, p;y,
lim /
G(x, p; y,
(3.5.20)
^P jRn
for any continuous function f ( x ) such that
|/(z)| < const e^l 2 ,
&= -
>
(3.5.21)
192
Chapter III
(here .A is taken from (3.5.16)). (2) Solutions of inhomogeneous equations (3.5.13) and (3.5.14) can be expressed in terms of G(x, p; y,
F0(p,x) =
G(x,p;y,Q)iJ}(y)dy
+
jRn
Fk(p,x)=
f
da Jo
JRn
G(x, p-y,cr)c(y) dy, (3.5.22)
G(x,p-