Abstract Methods in Information Theory

t E JR.

Hence, E(
Taking the suprernum w.r.t.

= aE(/I~) + (3.

a,{3 E JR. on the RHS suitably, we get E(
(13)

E(·I~)

: LP(X) -+ LP(~) is a projection for P 2 1.

For, since J-L is a probability measure and hence LP ~ L 1 , E(·I~) is defined on all LP for P 2 1. If P = 00, then for I E Lex::>(X)

IE(/I~)I

< E(I/II~) < 11/1Iex::>'

Chapter I: Entropy

14

implying that "E(/I~)lloo ~

since cp(t) =

Itl P

11/1100.

If 1 < p <

00,

then it follows from (12) that

is convex, and the desired assertion holds.

(14) E(·I~) = PL2(~), the orthogonal projection of L 2(X) onto L2(~). In fact, for

I

E

L 2 (X) and 9 (f,gh

= =

E L2(~)

i

i

fgdp,

we have

=

i

E(fgllJ)) dp,

E(fIlJ))gdp, = (E(fIlJ)),g)2'

where (., ·)2 is the inner product in L 2(X). This is enough to obtain the conclusion. (15) (Holder's Inequality) If ~

+~

= 1, p, q

2:: 1, then

I

E

LP(X),g E Lq(X).

For, we can assume 0 < E(I/IPI~), E(lglql~)
Ilgl E(lfIPIlJ));; E(lglqllJ)) t 1

1

I/IP

1

Iglq

~ P E(IJIPIlJ)) + qE(lglqllJ))

1

1

since sptq ~ ~sP + -qtq for s, t > O. By taking the conditional expectation E(·I~) it holds that E(I~glllJ)) 1 < 1 + ~ = 1 E(I/IPI~) p E(lglql~) q p q 1

1

or E(lfgll~) ~ E(I/IPI~) p E(lglql~) q, i.e., the desired inequality is obtained.

(16) E(fl~)

0

S = E(f 0

SIS-l~), where

(1 0 S)(x) = I(Sx) for x E X.

For, both sides are S-l~-rneasurable and for A E ~ it holds that

h-1A (E(fIlJ))

0

S)(x) p,(dx)

=

h-1A E(fIlJ))(Sx) p,(dx)

15

1.2. Conditional expectations

EUI!D)(y) J1-(dS-: 1y)

=l

= lEUI!D)dJ1-= l =

r fd(J1-oS-1) = JS-IA r foSdJ1-,

JA

since S is measure preserving. Hence E(fl~) (17) (18) (19) (20) (21)

fdJ1-

0

S = E(f

0

SIS-l~).

0 < P(AI~) < 1. A E ~ =? P(AI~) = lA. A < B =? P(AI~) ~ P(BI~). P(ACI~) = 1

P(AI~).

P(AI2) = jl(A).

(22) {An}

~ X, A j

n A k = 0 (j =F k)

PCQl Anl!D)

= n~l P(Anl!D)·

In fact, the above statements (17) - (22) are almost obvious. Let {~n} be an increasing sequence of a-subalgebras of X such that ~n

t

~,

i.e.,

!Dn ~ !Dn+l (n ;:::: 1) and aCQl !D n) = !D, where a(·) is the a-algebra generated by {.}. We say that {fn} ~ L1(X) is a martingale relative to {~n} if

For instance, if ~n t ~, I E L 1(X) and In = E(/I~n), n ~ 1, then {In} is a martingale relative to {~n}' {In} ~ L1(X) is said to be a submartingale relative to {~n} if In is ~n-rneasurable for ti ~ 1 and

If "~" is replaced by "~" in the above, then {In} is called a supermartingale relative to {~n}' (23) (Martingale Convergence Theorern) If ~n E(/I~n) -+ E(/I~) u-a.e. and in L 1 .

t

~ and

I

E

L 1(X), then

For, let I E L1(X). We can assume that ~ = X and hence E(fl~)

f.We first

prove the convergence in L

1

.

Let

€

> 0 be arbitrary. Since

L 1(X), there exist k ~ 1 and g E L1(~k) such that ~k ~ ~n and E(gl~n) = g, so that

III -

U Ll(~n)

is dense in

n=l gill < e. Then, for n ~ k,

Chapter I: Entropy

16

::; 211f -

gl/1 < 2£.

This shows that E(fl~n) -+ E(fl~) in L 1 . As to u-a.e. convergence, we proceed as follows. For any e > 0

JL ([ li~-:~p IEUI!Dn) - fl >

vie])

< JL( [ ~-:~p {lEU - gl!Dn) - U - g)1 + IE(gl!Dn) - gl} > vie])

~ JL( [li~~p {lEU -

gl!Dn)1 + If - gl} >

vie])

~ JL( [li~S~P lEU - gl!Dn)1 > ~]) + JL( [If - gl > ~]) <~

LIf - gldJL+ ~ LIf- gldJL ~ vie. 4

Therefore E(fl~n) -+ E(fl~) u-a.e. (24) Let ~n t~, f E L 1(X) and ip : IR -+ IR be an increasing convex function. If E L 1(X), then {E(

(25) (Submartingale Convergence Theorem) Let ~n t ~ and {fn} be a submartingale relative to {~n} with sup IIfnl11 < 00, then i« -+ foo t-t-a.e. for some n~1

function foo E L1(~) with

Ilfool11 ::; liminf n-+oo IlfnI11..

For the proof of (24) and (25) we refer to Rao [2].

1.3. The Kolmogorov-Sinai entropy With the preparation given in the previous section we define and study the Kolmogorov-Sinai entropy, Again let (X, X, t-t, S) be a fixed dynamical system and ~ be a a-subalgebra of X. a finite ~-measurable partition 21 of X, i.e., 2{ == {AI, ... ,An} ~ ~,Aj n A k == 0 (j =1= k) and .U A j == X. Denote by P(~) J=1

the set of all

~-partitions.

Let 21, Q3 E

P(~).

21 V Q3 == {A n B : A E 21, B E Q3},

We let

S-121 == {S-l A : A

E

2l},

which are clearly ~-partitions. 21::; Q3 rneans that Q3 is finer than 21, i.e., each A E 21 can be expressed as a union of some elements in Q3.

1.3. The K olmogorov-Sinai entropy

Definition 1. Let 2t. is defined by

17

= {A l , ...

,An} E P(X).

'YInrrt"Q1"Q/"l'YI ...o o ~ _ ~ ~ _ ~ ~ : \ ~ _ o __~

v __

2t.

n

E JL(A log JL(A E JL(A) logJL(A).

H(2t.)

j )

j )

j=l

AE2l

I(2t.)(·) = -

E 1A(·) logJL(A). AE2{

In this case we have

= E(I(2t)) =

H(2t)

.Ix I(2t) dJL.

The conditional entropy function I(2t.I~) and conditional entropy H(2t.I~) are respectively defined by

I(2t.I~)(·) = -

E 1A(·) logP(AI~)(·), AE2{

H(2tI!?))

= E(I( 2tI!?))) =

.Ix I(2tI!?)) dJL.

For 2t. E P(X) we denote by 2t the a-algebra generated by 2t., i.e., a-subalgebras ~l, ~2 we denote ~l V ~2 = a(~l U ~2).

(3.1)

2t =

a(2t.). For

is the same as the Shannon to entropy, H(2t.I~)

= E(I(2t.I~)) = E(E(I(2t.I~)I~)) =-

E r P(AI!?)) log P(AI!?)) dJL, AE2l

Jx

(3.2)

where 2t. E P(X) and ~ is a a-subalgebra of X. Consider a special case where ~ = ~ with ~ E P(X). Then, since P(AI~) = L JL(AIB)lB for A E X, JL(AIB) being BE23

the conditional probability of A given B,

18

Chapter I: Entropy

Ej AE2l

log (

A

E fL(AIB)l

B)

du

BEr.B

E j E IB log J-L(AIB) dJ-L == E J-L(B) E {- J-L(AIB) log J-L(AIB)}. AE2l

A BEr.B

BEr.B

L:

Here we consider -

J-L(AIB) log J-L(AIB) as the conditional entropy of Qt given

AE2l

B E ~ and (3.3) as the average conditional entropy of Qt given ~.

Then the following is fundamental. Theorem 1. Let Qt, ~ E P(X) and ~,~I, ~2 be a-subalqebras of X.

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11)

H(QtI2) == H(Qt). H(Qt V ~I~) == II(QtI~) + H(~IQ( V ~). H(Qt V~) == H(Qt) + H(~IQ(). Qt::; ~ Qt::; ~ ~1

=} =}

2 ~2

H(QtI~)

H(Qt) =}

< H(~I~)·

< ll(~). < H(QtI~2)'

H(QtI~l)

11(QtI~) ::; H(Qt). H(Qt V ~I~) < H(Q(I~) + H(~I~)· H(Qt V ~) < H(Qt) + H(~).

H(S-lQtIS-l~)

==

H(Q(I~).

H(S-lQt) == H(Qt).

Proof. (1) By definition we have

I(QtI2)(·) == -

(3.3)

AE2l

E lAC) logP(AI2)(·) == - E IA(') logP(A) == I(Qt)(.) AE2l

AE2l

and hence

=

H(2t12)

L

I(2t12) dfL

=

L

I(2t) dfL = H(2t).

(2) Observe that I(QtV~I~)==-

E

lclogP(CI~)

CE2lVr.B

EE AE2lBEr.B

lAnBlogP(AnBI~)

19

1.3. The Kolmogorov-Sinai entropy

=-

L

lAlB logP(AI!D) -

A,B

= -

L

~

lA logP(AI~) -

p(AnBI~)

P(AI!D)

LIB 10gP(BI2l V ~), B

we have

L1

P(BIQt !D) = V

and hence

lAlB log

A,B

A

since for B E

L

AE2(

A

P(B n AI!D) J-L-a.e. P(AI!D)

"" P(B n AI~) log P(BIQ( V !D) = L...J lA log P(AI!D) u-a.e. AEQ{

Taking the expectation, we see that the desired equality holds. (3) follows from (1) and (2).

(4) 2l.

< ~ implies 2l ~ ~ and 2l. V ~ =~. H(2l. V ~I~) = H(~I~) = H(2l.I~)

So (2) implies

+ H(~I2l V~) 2

H(2l.I~).

(5) is obtained from (1) and (4). (6) Since ¢(t) = ~t logt is concave it follows from Jensen's Inequality {(12) in Section 1.2} that for A E 2l.

the last equality is by (7) in Section 1.2 since ~ 1 2 ~2. Thus

L E(¢(P(AI~l)))' by (3.2), = L E(E(¢(P(AI~1)1~2))) AEm :s; L E(¢(P(AI~2)))

H(2l.I~l) =

AEm

AEm = H(2l.1~2),

(7) follows from (6) by letting (8) is derived as follows:

~1

=~

by (3.2). and ~2

= 2.

H(2l. V ~I~) = H(2l.I~) + H(~I2l V ~),

:s; H(Q(I~) + H(~I~),

by (2),

by (6).

Chapter I: Entropy

20

When q) = 2, (9) is obtained. (10) Since P(S-lAIS- 1q)) = P(AIq)) H(S-12qS-1q)) =

L

0

S for A E X we have

E(¢>(p(S-lAIS- 1q)))) ,

by (3.2),

AE2t

=

L E(¢>(P(AIq)) S)), L E(¢>(P(AIq)))), by 0

by (16) in Section 1.2,

AE2t

=

jl

0

s:' = u,

AE2t

=

H(2tIq))·

(11) is obvious. Remark 2. (1) Theorern 1 holds if H(2t) and H(2tIq)) are replaced ,by I(2t) and I(2tIq)), respectively, where (10) and (11) of Theorem 1 are then read as: (10) I(S-12tIS- 1q)) = I(2tIq)) 0 S. (11) I(S-l2t) = I(2t) 0 S.

(2) In Theorem 1 (2), if 2t ~ ~ and q) = ~, then 2t V ~ = ~ and Q( V ~ = ~, so that H(2tI~) = O. This means that ~ contains all the information that 2t has, and hence the conditional entropy of 2t given ~ equals o. In particular, H(2tIX) = 0 for every 2t E P(X). Definition 3. Let 2t E P{X).
We have a basic lemma.

Lemma 4. Let 2t E P(X). Then:

(1) H ( n.'1.- 1 s-j2t ) = H(2t) J-O

+ n-1( l: H 2tI .'1k.k=l

J-1

-) . s-j2t

l ) = H(2t) + n-1 -) . (2) H ( n .'1.- s-j2t l: H ( S-k2t Ik-l .'1.- s-j2t J-O

1) (3) I ( n .'1.- s-j2t J-O

k=l

J-O

= I(2t) + n-1 l: I ( 2tI .'1k.k=l

J-1

-) . s-j2t

isdefined by

(3.4)

21

1.3. The K olmogorov-Sinai entropy

(4) I (

n- l

.~

s-jQ() = I(Q()

+ n-l L I ( S-kQ(I .~-

J-O

k 1

k=l

-) . s-jQ(

J-O

Proof. (1) follows from the following computation:

HC~~ S-j2t) = H( s-(n-l)2t V C~: S-j2t) ) = H(s-(n-l)2t) + HC~: S- j 2tls-(n-l)§{) = H(2t) + H (s-(n-2)2t V (;~: S-j2t) Is-(n-l)§{) = H(Q()

+ H(S-C n- 2)Q(ls-C n- 1)2t)

+ H (;~: S- j 2tls-(n-2)§{ V s-(n-l)§{) =

H(Q() + H(Q(IS- l 2t) +

H( ~V3 S-jQ(l. J=O

nVl

S-j2t)

J=n-2

n-l

H(2t) +

LH(2tlj~lS-j§{).

k=1

(2), (3) and (4) can be verified in a similar manner. The following lernma shows that the limit (3.4) exists and H(Q(, S) is well-defined for every Q( E P(X).

Lemma 5. Let Q( E P(X). Then: (1) H(Q(, S) is well-defined and it holds that

H(Q(, S) = lim H(Q(I·.v S-j2t) n-too

= lirn n-too

J=1

H(s-nQ(1 ~V1 S-j2t). J=O

(2) If S is invertible, then

H(Q(, S) = lim H n-too

n . -) 1 (n-1 . ) .V 8 J Q( = lim -H .V 8 J Q( . (Q(IJ=1 n-too n J=O

Proof. (1) By Theorern 1 (6) and (1) we see that

22

Chapter I: Entropy

for n 2:: 1 and hence lim

n-+oo

H(2l.I.V S-i§l) exists. J=l

H(2l., S) = lim

n-+oo

Now by Lemma 4 (1) it holds that

~H( ~V1 S-i2l.) J=O

n

1 n-1 = lim - "" n-+oo

= . lim

n-+oo

k.,.,

H(2l.I.v S-32l.) n Z:: J=l k=l

H(2l.I.V s-i§l). J=l

The second equality is obtained frorn Lemma 4 (2). (2) An application of Theorem 1 (11) and (1) above give

H(2l., S) = lim

n-+oo

=

~ H (~V1 S-i2l.) J=O

n

lim ~H(s-(n-1) ~V1 Si2l.)

n J=O 1 (n-1 . ) = lim -H .V SJ2l. . n-+oo n J=O n-+oo

Similarly, the second equality is obtained. Basic properties of the entropy H(2l., S) are given in the following, which are similar to those of H(2l.).

2l.,

Theorem 6. Let ~ E P(~). Then: (1) < ~ :::} H(2l., S) < H(Q3, S).

2l.

(2) H

(Sm 8-

j2t,

8)

= H(2t, 8)

?: m ?: o. If 8 is invertible, this holds for

for n

n, m E Z == {a, ±1, ±2, ... }. m- 1 ) (3) H ( j~O 8- j2t,8m = mH(Qt, 8) for m ?: 1.

(4) H(2l., S)

< H(Q3, S) + H(2l.I~).

Proof. (1) is immediate from (3.4) and Theorem 1 (5). (2) Observe that

H

.) 1 ( p-1 .)) , . Vn S-J2l., S = lim -H V S- (n . V S-J2l. (J=m p-+oo p k=O J=m k

1 ( = lirn -H p-+oo p

s:» (p+n-m-1 V S-k2l.)) k=O

by definition,

23

1.3. The Kolmogorov-Sinai entropy

=

lim

+n -

p-+oo

m- 1

p

.

1 p

+n -

m - 1

H

(p+n-m-l V S- k) Q{ , k=O

by Theorem 1 (11), = H(Q{, S).

The invertible case is verified by Lemma 5 (2). (3) Since tti ~ 1 we see that by definition

m .) - l S-3Q{,8 H (m .V 3=0

1 (n-l -H V (sm)- k(m-l. .V S-3Q{)) n k=O 3=0 = lim m· -1H (mn-l V S- k) 2t n-+oo mn k=O = mH(Q{, S). = lim

n-+oo

(4) We have the following computation:

H(~Vl S-jQ{)

<.S:

3=0

H((~Vl S-jQ{) V (~Vl S-j~))., 3=0

=

by Theorem 1 (4),

3=0

H(~Vl S-j~) + H(~VlS-jQ{1 nVl S-ksi3) , by Theorem 1 (3), 3=0

3=0

n-l

k=O

~ HC2~ S-ilJ3) + L H(S-iQlI :2~ S-k~),

by Theorem 1 (8),

j=O

n-l

~ HC2~ S-ilJ3) + L H(S-iQlIS-i~), 3

=

by Theorem 1 (6),

j=O

H(~Vl S-j~) + nH(Q{Isi3) , by Theorem 1 (10). 3=0

Thus we have

H(Q{, S)

= lim n-+oo

<.S:

lim

n-+oo

!..H( ~Vl S-jQ{) n 3=0

{!..H(~Vl S-ilJ3) + H(Q{Isi3)} n 3=0

= H(~,

S) + H(Q{Isi3).

The following is a direct consequence of Theorem 6 (2) and (3).

Corollary 7. H(sm) ImIH(S) for m E z.

= mH(S)

for m ~ 1. If S is invertible, then H(sm) =

Chapter I: Entropy

24

In order to state the Kolmogorov-Sinai Theorem we need the following two lemmas, which involve uniform integrability and Martingale Convergence Theorem. Lemma 8. If ~n then

t~

and

2{

P(X), where

E

( sup I(2lI!Dn) djL

i x n?l

supI(2{I~n) and ]?(t)

Proof. Let f

~n

's and ~ are a-subalgebras of X,

< H(2l) + l.

/-L([f>

tJ)

for t ~ 0, where [f

> t]

{x E

n?l

X : f(x)

> t}. Then we see that

L -1 f djL =

and for t

~

00

1

= [- tF(t)]: +

1

00

00

tdF(t)

F(t) dt:::::

F(t) dt

0

F(t)

= jL([~~~ { /-L(A n

== L AEQl

~ lAlogp(AI!D n)} > tJ) [l~\ P(AI!Dn) < e-tJ) -

00

L/-L(AnA~),

== L

AEQln=l

where Ai == [P(AI~l)

n-l

< e- t ] and A; == n

n ~ 2. Note that A~'s are disjoint and

k=l

[P(AI~n)

U A; ==

n=l

[inf

n?l

< e- t , P(AI~k)

P(AI~n) <

~ e- t ] for

e-tJ. Also note

that A~ E ~n and /-L(A n A~) == JA~ P(AI~n) du, so that 00

L /-L(A n A~) n=l 00

Obviously

I:

00


e- t /-L(A~)

< «:',

n=l

/-L(A n A~) ~ /-L(A). Thus we get F(t) ~ min {/-L(A) , e- t } and

n=l

rOO F(t) dt :::::

io

L AEQl

=L AEQl

roo min{jL(A), e-t}dt

io

{i -

log /L(A)

0

jL(A) dt +

1

00

-log /L(A)

e~t dt

}

25

1.3. The K olmogorov-Sinai entropy

L {- JL(A) logJL(A) + JL(A)}

=

AEQ{

= H(2l)

+ 1.

Therefore the desired inequality holds.

Lemma 9. If ~n t ~ and 2l E P(X), then: (1) I(2lI~n) -+ I(2lI~) u-a.e. and in L 1 . (2) H(2lI~n) -!- H(2lI~)·

Proof. (1) It follows from the Martingale Convergence Theorem ((23) in Section 1.2) that P(AI~n) -+ P(AI~) JL-a.e. for A E 2l and hence I(2lI~n) -+ I(2lI~) u-a.e. Since sup I(2lI~n) is integrable by Lemma 8, we see that I(2lI~n) -+ I(2lI~) in L 1 , n~l

too. (2) is derived from (1) and the fact that H(2lI~n) = H(2lI~n) ~ H(2lI~n+l) for n ~ 1 by Theorem 1 (6).

Ix I(2lI~n) dJL

and

Now we have:

Theorem 10 (Kolmogorov-Sinai). If S is invertible and 2l E P(X) is such that

V

n=-oo

sn§t

= X,

Proof. Let 2ln = Observe that for

then H(S)

= H(2l, S).

V Sk2l for n ~ 1. Then H(2ln, S) =

k=-n ~ E P(X)

H(~, S)

< H(2ln, S) + H(~I§tn),

H(2l, S) by Theorem 6 (2).

by Theorem 6 (4),

= H(2l, S) +_H(~I§tn)

-+ H(2l, S)

(n -+ (0)

since H(~I§tn) -!- H(~IX) = 0 by Lemrna 9 (2) and Remark 2 (2). This means that H(~, S) ~ H(2l, S) for ~ E P(X), which implies that

H(2l, S)

= sup {I{(~, S) : ~ E P(X)} = H(S).

An immediate corollary of the above is:

Corollary 11. Let 2l E P(X). Then: 00

-

(1) V s-n2l = X=} H(S) = H(2l, S). n=O

Chapter I: Entropy

26

(2) If S is invertible and Proof. (1) Letting 2tn

V s-n§( = X,

n=O

then H(S)

= H(21, S) = o.

= V 8- k 2t for ti 2: 1,we see that this plays the same role as k=O

in the proof of Theorem 10. (2) By Theorem 10 we have H(2t, 8)

= H(8).

We also have

H(21,8) = lim H(2tI.V 8- i §(), by Lemma 5 (1), =

since X

= 8- 1X =

n-+oo

3=1

H(2tIX),

by Lemma 9 (2)

V 8- k 2i by assumption, which is 0 by Remark 2 (2).

k=l

Consider the case where H(8) o. Note that this implies 8- 1X" = X. For, 1X c X, a proper set inclusion. Then there is an A E X such that suppose that 8A tJ. 8- 1X. Let 2to = {A,AC} E P(X). Then

a contradiction. To consider various dynamical systems and entropies, isomorphisrn among these systems is important, which is defined by

Definition 12. Let (Xi, Xi, J.li, 8 i ) (i = 1,2) be two dynarnical systems. These systems or 8 1 and 8 2 are said to be isomorphic, denoted 8 1 ~ 8 2 , if there exists some one-to-one and onto mapping cP : Xl -+ X 2 such that (1) for any subset Al ~ Xl, ft1 E Xl iff cp(A 1) E X2, and J.l1(A 1) = J.l2(cp(A 1)); (2) cp 0 8 1 = 8 2 0 ip, i.e., cp(8IXI) = 8 2CP(X1) for Xl E Xl. In this case, cp is called an isomorphism.

Theorem 13. 8 1 ~ 8 2 => H(8 1) = H(82). That is, the Kolmogorov-Sinai entropy of measure preserving transformations is invariant under isomorphism. Proof. Let ip : Xl -+ X 2 be the isomorphism between two systems. Let 2t1 E P(X1) and observe that cp(2t1 ) = {cp(A) : A E 2(1} E P(X2). Moreover, H(2t1, 8 1 ) = H(CP(2(l) , 8 2 ) since J.l1(A) = J.l2(cp(A)) for A E 2(1. Hence, H(81 ) ~ H(82). The converse H(8 1) 2: H(82) is also true. Thus H(8 1 ) = H(82). It trivially follows frorn the above theorem that if H(Sl) =1= H(82 ) , then 8 1 '1- 8 2 . In the next exarnple we show how to compute the entropy of Bernoulli shifts.

1.3. The Kolmogorov-Sinai entropy

27

Example 14 (Bernoulli shifts). Let (Xo,p) be a finite scheme, where X o = {al,'" ,all and p = (PI, ... ,Pi) E ill, so that p(aj) = Pj, 1 :::; i : t. Consider the infinite Cartesian product

x = X~ = {x = (... where Z

,X-I,XO,XI,"') : Xk

E

X o, k E Z},

= {O, ±1, ±2, ... }, and the shift S on X given by

A cylinder set is defined by

and let

J.to([X?' .. x~]) = p(x?) ... p(x~). Extend J.to to the a-algebra X generated by all cylinder sets, denoted by u: Note that S is measure-preserving w.r.t. J.t and hence (X, X, u; S) is a dynarnical system. The shift S is called a (PI, . . . ,Pl)-Bernoulli shift. Since 2t = {[xo = all, ... , [xo = all} is a finite partition of X and

00

V

-

n=-oo

sn2t = X by definition, we have by Theorem 10

and Lemma 5 (2) that

H(S) = H(2t,S) = lim

n-+oo

!:H(nyl S-k2t). n k=O

n-l

Now V S-k2t = {[xo ... Xn-l] : Xj E X o, 0 :::; j :::; n - I} and hence k=O

J.t([xo ... Xn-l]) log J.t([xo ... Xn-l])

Xo,··· ,Xn-l EXo

= -

L

J.t([xo]) log J.t([xo]) - ... -

xoEXo

=

since J.t([aj])

L xn-lEXo

nH(2t)

= p(aj) = Pj for

1 :::; j :::; n. This implies that l

H(S) = H(2t) = - LPj logpj. j=l

J.t([Xn-I]) log J.t([Xn-I])

Chapter I: Entropy

28

Thus (~, ~) -Bernoulli shift and ( ~, ~, ~) -Bernoulli shift are not isornorphic. (PI, ... ,Pn)- and (ql, ... ,qm)-Bernoulli shifts have the same entropy, i.e.,

If

m

n

- LPjlogpj == - Lqklogqk, j=l

k=l

then are these isomorphic? This was affirmatively solved by Ornsterin [1](1970) as: Theorem 15. Two Bernoulli shifts with the same entropy are isomorphic. We say that the entropy is a complete invariant among Bernoulli shifts. The proof rnay also be found in standard textbooks such as Brown [3], Ornstein [3] and Walters [1]. Example 16 (Markov shifts). Consider a finite scheme (Xo,p) and the inifinite product space X == X~ with the shift S as in Example 14. Let M == (mij) be an f x f £

stochastic matrix, Le.,

mij

2: 0,

E rnij == 1 for 1 ~ i,j ~ f,

and m == (ml, ...

,m£)

j=l

£

E mimij == mj for 1 < j

~ f. For each i, i, i=l mij indicates the transition probability from the state a; to the state aj and the row vector m is fixed by M' in the sense that mM == m. We always assume that m; > 0 for every i 1, ... ,f. Now we define J-lo on 911, the set of all cylinder sets, by

be a probability distribution such that

11.0

r"

{[a· ... a· ]) -~o

~n

m·~o rn·~O~l . . . . m·~n-l ~n . .

J-lo is uniquely extended to a measure J-l on X which is S-invariant. The shift S is called an (M, m)-Markov shift. To cornpute the entropy of an (M, m)-Markov shift S consider a partition Q{ ==

{[xo

==

al], ... ,[xo == all}

E

P(X), which satisfies

V

n=-oo

sn§( == X. As in Example

14, we see that H

nV - l S-kQ{ ) ==( k=O

Xo,··· ,Xn-l EXo

£

L io,···

£

mio mioi1

•••

mi n-2 in-l

£

== - ' " m·~o logm·~o - (n - 1) L....J '" L....J io=l

log mio rnioil

,in-l=l

i,j=l

m·m· ~ ~J·logm·· ~J

... min-2in-l

1.4. Algebraic models £

since n

L: mimij

i=l -t 00 we

29

£

= mj and

L: mij

= 1 for 1

< i, j < t.

By dividing n and letting

j=l

get £

H(S) = -

L

mirr~ijlogmij.

i,j=l

1.4. Algebraic models In the previous section, we have defined the Kolmogorov-Sinai entropy for measure preserving transforrnations and have seen that two isomorphic dynarnical systems have the .same entropy. In this section, we relax isomorphism to conjugacy, which still has the property that two conjugate dynamical systems have the same entropy, Algebraic models of dynamical systems are good tools to examine conjugacy arnong these systems and these will be fully studied in this section. Let (X, X, J.L, S) be a dynamical system. Two sets A, B E X are said to be J.Lequivalent, denoted A ~ B, if J.L(A~B) = 0, where A~B = (A U B) - (A n B), the symmetric difference. Let A. = {B Ex: A ~ B}, the equivalence class containing A E X, and ~I.£ = {A. : A E X}, the set of all equivalence classes of x, called the measure . algebra of J.L. The measure J.L on ~ 1.£ is of course defined by

J.L(A.) = J.L(A), and is strictly positive in the sense that J.L(A) > 0 if A. :f= 0. For now we consider conjugacy between a pair of probability rneasures.

Definition 1. Let (Xj , Xj, J.Lj) (j = 1,2) be two probability measure spaces with the associated rneasure algebras ~l and ~2, respectively. Then J.Ll and J.L2 are said to be conjugate, denoted J.Ll ~ J.L2, if there exists a measure preserving onto isomorphism T : ~l -t ~2, i.e., T satisfies that BE

~l,

Conjugacy between a pair of probability measures can be phrased in terms of a unitary operator between L 2-spaces as follows.

Chapter I: Entropy

30

Proposition 2. Let (Xj, Xj, J-tj) (j = 1,2) be a pair of probability measure spaces. Then, J-tl c::: J-t2 iff there exists a unitary operator U : L 2(J-tl) -+ L 2(J-t2) such that ULoo(J-tl) ~ Loo(J-t2) and U(fg) = Uf· Ug,

Proof. To prove the "only if' part, assume J-tl c::: J-t2 and let T : Q)l -+ Q)2 be an onto measure preserving isomorphism, where Q)j is the measure algebra of J-tj (j = 1,2). Define an operator U : L 2(J-tl) -+ L 2(J-t2) by B E Q)l.

Note that IIU1BII2 = JJ-t2(TB) = JJ-tl(B) = II1BII2 since T is measure preserving. Then, U can be linearly extended to the set of all Q)l-simple functions and is an isometry. Thus U becomes a unitary operator since T is onto. n

Now, let f

= L:: a j1B j,

j=l disjoint. Then, it holds that

where aj's are complex and B j E Q)l (1

Uf2 = U(

r r

~ajlBj J

~

j ~ n) are

= u(~ajaklBjlBk) J,k

= Lajak1TBjnTBk = La;lTBj' j,k

(Uf)2 =

(~ajlTBj J

=

= ~ajaklTBjlTBk

L aj ak 1T(BjnB

k)

j,k

j

J,k

= L a;lTBj' j

and hence Uf2 = (Uf)2. This implies by the polarization identity that U(fg) = U 1 . U 9 for all Q)l-simple functions I, 9 and then for all i, 9 E LOO (J-tl) by a suitable approximation. Conversely suppose U satisfies the conditions mentioned. If f is an indicator function corresponding to B l E Q)l, then 1 2 = ! and U1 = U1 2 = (U1)2, so that U1 is also an indicator function corresponding to B~ E Q)2, say. Define T : Q)l -+ Q)2 by TB l = B~ for B; E Q)l. Since U is onto, T is also onto. Since U is a unitary operator, T B l = 0 implies B l = 0. To see that T is a measure preserving isomorphism, note that J-tl(Bl) = Ilf"~ = "UI"~ = J-t2(TB l) for B l E Q)l, where 1 is the corresponding element to B l as above. Now T(B lnB2) = TB lnTB2 follows from U(lg) = UI·Ug,

1.4. Algebraic models

31

where 9 corresponds to B 2. Moreover, T(B 1 U B 2) = TB 1 U TB 2 follows from the fact that f + 9 - fg corresponds to B 1 U B 2, and hence

fL1(B1 U B 2) = Ilf + 9 - fgll~ = IIU(f + 9 - fg)ll~

= IIUf + Ug - Uf·

TCQl

nQl

fL2CQl TB

That Bn ) = TBn and verified by a suitable approximation.

UglI~ = fL2(TB1 U TB 2). n)

=

fLl CQl B

n)

for {Bn } s.;; SJ3 1 are

Now let (X, X, fL) be a probability measure space and r(fL) be the set of (equivalence classes of) functions f E L 00 (fL) with If I == 1. Note that I' (fL) is a group with the product of pointwise multiplication, where the complex conjugate 1 is the inverse of f E r(fL). We identify the circle group C = {z E C : Izi = I} with the constant functions of T (fL), so that C c r (fL), where C is the cornplex humber field. For convenience we assume that r(fL) is a group of X-measurable functions f : X -+ C such that f,g E r(fL) and f = 9 u-a.e. imply f(x) = g(x) for every x E X. This is due to the existence of a lifting on Loo(fL) (cf. Tulcea and Tulcea [1]). Let us define a function
cp,..(f) =

L f

du,

Then the following is a basic.

Proposition 3. For a probability measure space (X, X, fL),
Proof. For any n 2:: 1, f1,' .. .i« E r(fL) and a1,' .. ,an E
La/likCP,..(/if/;l) = Lajak j,k

j,k

=

1

fJkdfL

x

LI~ f{ aj

J

du

~ 0,

which implies that
J

L9dfL + i L hdfL = L f dfL = cp,..(f) = 1.

32

Considering 9

Chapter J: Entropy

== g+ - g- with g+, g- 2:: 0, we have 1 = !pp,(f) = L q dp. = L

«: dp, - L g- dp,

< Lg+ du < L implying that Thus f == 9 ==

Igldp, <

Ix g- dM == 0, g- == 0 and Igi == g.

L

Ifl dp, = 1,

Ix Igl du == 1 and Igl == 1.

Hence

Igi == 1.

The conjugacy between two probability measures induces a certain mapping between I'(.)-spaces as is seen below.

Proposition 4. Let (X j , Xj, Mj) (j == 1,2) be a pair of probability measure spaces. If MI ~ M2, then there exists a group isomorphism U : r(MI) -t r(M2) such that (1) Ur(MI) == r(M2); (2) U c == c for c E C; (3) If r ~ r(MI) is a set generating L 2(MI), then ur generates L 2(M2); (4) If r ~ r(MI) is orthonormal in L 2(MI), then ur is orthonormal in L 2(M2); (5) 'PJ1-l (f) == 'PJ1-2 (U I) for f E r(MI)' Proof. Let U : L 2(MI) -t L 2(M2) be a unitary operator realizing the conjugacy between MI and M2 (cf. Proposition 2 above). We show that U restricted to r(MI) is the desired isomorphism. At first, note that IIU flloo == IIflloo for f E r(MI) (see the proof of Proposition 2). Hence IUfl.::; 1 M2-a.e. For n 2:: 1 let

and assume that M2(Bn o )

J IUf1 X2

> 0 for some no 2:: 1. Then, 2

dp,2 =

1e.; IUfI

2

dp,2+

1 IUf1

2

B~o

dp,2

< (1 - ~J P,2(Bno) + P,2(B;o) < M2(X2) == MI (Xl) == ( Ifl 2 dMI'

JX

l

contradiction to /lU 1/12 == 11/112. Thus M2(Bn ) == 0 for every n 2:: 1 and IUII == 1 M2-a.e. This means that U : r(MI) -t r(M2). The properties (1) (5) are easily verified.

1.4. Algebraic models

33

The properties (1) and (5) in Proposition 4 are particularly important and will be used to characterize the conjugacy of measures in terms of their algebraic models. For this goal we need several technical results. Here, a pair of probability measure spaces (Xj,Xj,{tj) (j = 1,2) are fixed. Lemma 5. Let V : L 2({tl) -t L 2({t2) be a linear isometry. If f E LOO({tl) is such that (Vf)n = r: for n ~ 1, then Vf E LOO({t2) and IIVflloo = Ilflloo.

v

Proof. Observe that for n

~

1

{ IVfl 2n dJL2 = {

JX

2

= =

(VJ)n(VJ)n dJL2

JX

2

JX

2

r V fn . V fn d{t2, by assumption, r fn. fn d{tl, since V is an isometry, JX 1

=

r

JX 1

Ifl 2n d{tl,

which irnplies that IIVfl12n = Ilfl12n' Letting n -t Vf E LOO({t2).

00,

we get IIVflloo = Ilflloo and

Lemma 6. Let V : L 2({tl) -t L 2(J.l2) be a linear isometry and A ~ L 2({tl)' If f E LOO({tl) is such that Vf E LOO({t2) and

V(fg) = Vf· Vg,

9 EA,

(4.1)

then (4.1) is true for every 9 E A, the closure of A in L 2({tl). Proof. Take a 9 EA and choose a sequence {gn} ~ A such that Ilgn - gl12 -t O. Then IIVgn - Vgl12 = Ilgn - gl12 -t 0 and IIV(fgn) - V(fg)112 = Ilfgn - fgll2 -t 0 since f is bounded and V is an isometry on L2-spaces. But since V(fgn) = Vf· Vg n for n ~ 1, we have

or V(fg) = Vf· Vg. Thus (4.1) holds for every 9 EA. Lemma 7. Let V : L 2({tl) -t L 2({t2) be a linear isornetry, A ~ LOO({tl) be an additive group and 9 E L 2({tl). If

V(fg) = Vf· Vg,

IIVflloo = Ilflloo

(4.2)

Chapter I: Entropy

34

for every f E A, then (4.2) is true for every f E

A, the

closure of A in Loo(J-Ll).

Proof. Let f E A and choose a sequence {fn} ~ A so that Ilfn - flloo -+ o. Then Ilfng- fgl12 -+ 0 and hence IIV(fng) -V(fg)I/2 -+ 0 because V is an isornetry. Since IIVfn -Vfmlloo == Ilfn - fmlloo -+ 0 as n,m -+ 00 by (4.2) and the choice of {fn}, it follows that IIVfn - Vflloo -+ O. Thus IIVfn· Vg - Vf· Vgl1 2 -+ O. Now since V(fng) == we have by letting n -+

v i.: Vg,

00

IIVfniioo ==

IIfnlloo,

that V(fg) == Vf· Vg and

n ~ 1,

IIVflloo == IIflloo as desired.

Theorem 8. Let A ~ LOO(J-Ll) be such that (1) A is dense in L 2(J-Ll); (2) a, f3 E
f EA. (1) and Lemma 6 imply that

V(fg) == Vf· Vg, We can deduce from Lemma 7 that

V(fg) == Vf· Vg,

IIVflloo == Ilflloo,

(4.3)

f

E

A,

(4.4)

1.4. Algebraic models

35

A is the closure of A in £00(1-£1). Claim 1. If f E A and ¢ is a C-valued continuous function on C, then ¢ f E A. For, note that A is a sub algebra of £00(1-£1) such that 9 E A for 9 E A. Let > Ilflloo. Then there is a sequence {Pn(z, z)} of polynomials in z and z converging

where

0

a uniformly to ¢ on the disk [z] ::; a. Hence {Pn(f(x), l(x))} converges uniforrnly to ¢(f(x)) on X. Thus, ¢ 0 f E A since Pn(f, 1) E A for n ~ 1. Claim 2. V £00(1-£1) ~ £00(1-£2).

For, let f E £00(1-£1) and a > Ilflloo. Choose a C-valued continuous functione on C such that ¢(z) = z for Izi ::; a and 1¢(z)1 ::; a on C. Then, clearly ¢of = f I-£I-a.e. Moreover, choose a sequence {fn} ~ A such that Ilfn - fl12 -+ 0 and t« -+ f I-£I-a.e. Then, hn = ¢ 0 t« E A (n ~ 1) and hn -+ ¢ 0 f = f I-£I-a.e. Since Ihnl ::; I¢I ::; a (n ~ 1), it follows from the Bounded Convergence Theorem that IIVhn - V fJl2 -+ o. Without loss of generality we may assume that Vh n -+ V f 1-£2-a.e. since otherwise we can choose an appropriate subsequence. Then we deduce that IIVflloo ::; a and hence V f E £00(1-£2) since IIVhnii oo = Ilhnll oo ::; a by (4.4). To show (a) we use (4.3) and Lemma 6. We note that

V(fg) = Vf· Vg, since A = £2(1-£1) and each f E £00(1-£1) satisfies the assumptions of Lemma 6, which clearly implies (a). Then, (b) follows from Lemma 5. As to the last statement, if VA is dense in £2(1-£2)' then evidently V£2(1-£1) = £2(1-£2) and V becomes a unitary operator. If we consider V-I: £2(1-£2) -+ £2(1-£1) and apply the part of the theorem proved, we get V- 1£00(1-£2) ~ £00(1-£1) or £00(1-£2) ~ V£oo(I-£I). Thus V£oo(l-£l) = £00(1-£2).

Definition 9. A pair (I',
For instance, if (X, X, 1-£) is a probability rneasure space and if r ~ r(l-£) is a group, then (I',
"ifu

for some '"Y

=1=

1, then we can associate an algebraic measure system

cr, rj;) as follows.

Chapter I: Entropy

36

Let C' == {" E T' : I

,,' E C', "E r.

Let C 1 == {" E I' :
"1"2 1 E C 1 • :f' == r / C 1 , the quotient group and define cp on r by cp(i) ==
Let

"E

where i is the equivalence class containing". Then it is verified that cp is positive cp) is an algebraic measure definite on such that cp(i) == 1 iff i == i, so that system.

r

(r,

Definition 10. Let (X,X,J-L) be a probability measure space and (f,

{t

(Xj/j : (Xj E

c.i,

J=1

f

n

==

L.: O'-jfj j=1

E

A we have that

E

f E f(J-Ll).

I'(J.td,l::; j::; n,n 2:

I} <

L 2(J-Ll ). Then for

1.4. Algebraic models

37

G-ifik<{JJ-t2 (Ufj ut; 1)

= LG-ifik<{JJ-tl(fjf;;l) = L

j,k

j,k

It follows that we can define a mapping Ui, : A -+ L OO(J-L2) unambiguously by Uo (

LOijfj) = 2; Oij ut,, J

since "'i;G-jfj

=0

J

J-L1-a.e. implies that

J

~G-jUfj

=0

J-L2-a.e. Then note that Ui, is

j

a linear rnultiplicative rnapping such that

IIUofl12 = IIflb,

fEA.

It then follows frorn Theorem 8 that Uo can be extended to a unitary operator U1 : L 2(J-L1) -+ L 2(J-L2) such that U1LOO(J-L1) = L OO(J-L2) and

Therefore J-L1

~

J-L2·

Corollary 12. Tuio probability measures J-L1 and J-L2 are conjugate iff algebraic measure systems (r(J-L1), «JJ-tl) and (r(J-L2), «JJ-t2) are isomorphic.

The following theorem states that every algebraic measure system associates a regular Borel measure on some compact abelian group together with the construction of the group. Theorem 13. If (r, «J) is an algebraic measure system, then (F', «J) is an algebraic model for a regular probability measure J-L on a compact abelian group G. Moreover, if «J('Y) = 0 for 'Y 1= 1, then J-L is the IIaar measure of G. Proof. Consider the discrete topology on r and let G = f, the dual group. By Bochner's theorem there is a unique regular Borel measure J-L on G such that

where (x, 'Y) is the duality pair for x E G and 'Y E f. Define J : r -+ L 2(J-L) by J'Y = (', 'Y),

'Y E

r.

Chapter I: Entropy

38

Clearly, J is a homomorphism of f into f(J-L), Jf generates L 2(J-L), and l' E f.

Moreover, J is one-to-one since (-,1') = 1 J-L-a.e. implies


L

(x,,) f-t(dx)

= 1,

so that l' = 1. The assertion about the Haar measure is obvious. Corollary 14. Every probability measure J-L is conjugate to a regular Borel measure u on a compact abelian group.

Proof: (f(J-L) , 'PJL) is an algebraic model for J-L and also an algebraic model for a regular Borel measure u on a compact abelian group G. We invoke Theorem 11 to obtain conjugacy between J-L and u, Theorem 15. Let (X, X, J-L) be a probability measure space. Then J-L is conjugate to a Haar measure A on a compact abelian group G iff there is a group f 1 ~ f(J-L) which is a CONS (= complete orthonormal system) of L 2(J-L).

Proof. Suppose J-L ~ A and let U : L2(A) --t L 2(J-L) be a unitary operator such that ULOO(A) = LOO(J-L) and U(fg) = Uf· Ug, Since the dual group G is a CONS of L 2(A), f 1 = UG is a CONS of L 2(J-L). Conversely, suppose that there is some f 1 ~ f(J-L) that is a CONS of L 2(J-L). Then, (f 1 , 'PJL) is an algebraic model of J-L. Moreover, since 'PJL(I') = 1 iff l' = 1, Theorem 11 implies that (fr, 'PJL) is also an algebraic model of the Haar measure A of G = f 1 , a compact abelian group. Thus J-L ~ A by Theorem 11. Now let (X, X, J-L, 8) be a dynarnical system. Denote by Us the linear isometry on L 2(J-L) defined by Usf = f 0 8, i.e., (Usf)(x) = f(8x) for x E X and f E L2(J-L). Definition 16. Let (X j , Xj , J-Lj , 8 j ) (j = 1,2) be a pair of dynamical systems with measure algebras ~1 and ~2, respectively. Then, 8 1 and 8 2 are said to be conjugate, denoted 8 1 ~ 8 2 , if there exists a measure preserving onto isomorphism T : ~1 --t ~2 such that T8 1 = 8 2T. Let (Xj , Xj, J-Lj, 8j) (j = 1,2) be a pair of dynamical systems. Clearly 8 1 ~ 8 2 implies 8 1 ~ 8 2 , that is, isomorphism implies conjugacy of dynamical systems (cf.

1.4. Algebraic rnodels

39

Definition 3.12). Note that 8 1 ~ 8 2 iff J-L1 and J-L2 are conjugate by means of a unitary operator U : L 2(J-L1) -+ L 2(J-L2) such that UUs l = US2U. This follows from Proposition 2.

Definition 17. A triple (I', cP, U) consisting of an abelian group T, a positive definite function cP and a one-to-one homomorphism U on I' is said to be an algebraic dynarnical system if (I', cp) is an algebraic measure system such that cp(U'"'() = cp('"'() for '"'( E f. Two algebraic dynamical systems (f j , CPj, Uj) (j = 1,2) are said to be isomorphic if (f j, CPj) (j = 1, 2) are isomorphic algebraic measure systems via an onto isomorphism U : f 1 -+ f 2 such that UU1 = U2U. If (X, X, J-L, 8) is a dynamical system, then (f(J-L) , CPp" Us), (0, CPp" Us) and (1, CPp" Us) are algebraic dynamical systems. More generally, if I' ~ r(J-L) is an Us-invariant group, then (I', CPp" Us) is an algebraic dynamical system.

Definition 18. Let (X, X, J-L, 8) be a dynamical system and (I', cP, U) be an algebraic dynamical systern. Then, (I', ip, U) is said to be an algebraic model for 8 or for (X, X, J-L, 8) if (f, cp) is an algebraic model for J-L by means of a one-to-one homomorphism J : T -+ f(J-L) such that JU = UsJ. Then we have the following.

Theorem 19. Tuio dynamical systems are conjugate iff they have isomorphic algebraic models.

Proof. Let (X j , Xj, J-Lj, 8 j ) (j = 1,2) be a pair of dynamical systems with associated isometries USI and US2 on L 2(J-L1) and L 2(J-L2), respectively. Suppose that 8 1 ~ 8 2 . Then, it is easy to see that the algebraic models (f(J-L1), CPp'I' Us I ) and (f(J-L2), CPp'2' Us 2 ) are isomorphic. Conversely, assume that 8 1 and 8 2 have isomorphic algebraic models (f 1 ,
is verified first for f E T 1, then for a linear combination f of functions in T 1, and finally for f E L 2(J-Ll) by a suitable approximation since f l generates L 2(J-Ll)'

Chapter I: Entropy

40

Corollary 20. Two dynamical systems (Xj , X j,l-"j,8j) (j == 1,2) are conjugate iff the algebraic rneasure systems (r(l-"j), CPl-£j' USj) (j == 1, 2) are isomorphic. Theorem 21. An algebraic dynamical system (I", ip, U) is an algebraic model for a dynamical system (G, ~G, 1-", r) consisting of a regular Borel measure I-" on a compact abelian group G with a continuous hornomorphism r, where ~G is the Borel (J'algebra of G. Proof. As in the proof of Theorem 13 we get a regular Borel measure I-" on a compact abelian group G == by equipping a discrete topology on I' such that

r


l

(x,,/,) J.l(dx) ,

By considering a mapping J : I' -+ r(l-") defined by

we have that (r,cp) is an algebraic model for (G,~G,I-"). Now we define a mapping G -+ G by x E G, 'Y E r. (TX, 'Y) == (x, U'Y),

T:

Then it is easily seen that T is a continuous homornorphism of G into G such that JU == UrJ. 1'0 see that T is measure preserving, let us consider a measure v on ~G defined by AE~G'

We then have for 'Y E I'

cp(,/,) = cp(U,/,) =

=

l

l

(x, U,/,) J.l(dx)

(rx,,/,) J.l(dx) =

l

(x,,/,)v(dx).

Since I-" is unique in Bochner's theorem, we must have I-" preserving.

== v and hence

T

is measure

Given two dynamical systems (Xj,X j, I-"j, 8 j) (j == 1,2) with the measure algebra of Xj. If 8 1 ~ 8 2 , then ~1 ~ ~2 by some measure preserving onto isomorphism T : ~1 -+ ~2' For 2t E P(X 1 ) it holds that H(2t) == H(T2t) , H(2t, 8 1 ) == H(T2t, 8 2 ) and H(Sl) == H(S2)' Thus, conjugate dynamical systerns have the same entropy. ~j

41

1.5. Entropy functionals

1.5. Entropy functionals In Section 1.3, we fixed a probability measure space (X, X, Jl) and considered a measure preserving transformation S on it. For a given finite partition 21 E P(X), the entropies H(21) and H(2t, S) are defined and they depend on the probability rneasure Jl. So we denote them by H(Jl,21) and H(Jl, 2t, S), respectively. In this section, however, given a measurable space (X, X) with an invertible measurable transformation (i.e., an automorphism) S, we consider entropies H(Jl, 2t, S) for Sinvariant probability measures u, so that H(·, 21, S) becomes a functional on the set of such measures. Our goal of this section is to extend this functional to the set of all C-valued S-invariant measures, to see that the extended functional is bounded, and to obtain an integral representation of this functional in a special case of interest. To this end we need some notations. Let 2t E P(X) be a finite partition and let n

a, =

V s-j21 E P(X), j=l

2(00

= a(

U

2tn),

n=l

for n ~ 1. denotes the set of all probability measures on X and Ps(X) the set of all S-invariant measures in P(X). M(X) (resp. Ms(X)) stands for the set of all C-valued (resp. S-invariant) measures on X, which is a Banach space with the total variation norm II . II, while M+(X) (resp. M:(X)) stands for the set of all nonnegative elernents in M(X) (resp. Ms(X)). For a probability measure Jl E P(X) and a a-subalgebra ~ of X, PJL(AI~) denotes the conditional probability of A E X relative to ~ under u, For ~ E M+(X) we let ~1 = E P(X) and

nfu

Now we fix a partition 2t E P(X) and consider a functional H(·, 2t, S) on M:(X) given by

where I~(2tI~) =

- L

lA log P~(AI~)·We can rewrite H(~, 2t, S) as follows:

AE2l

L

f p{(AI 2l.,o)logPdAI2too) d/;" by Jx = - L lim f p{(AI2tn ) logP{(AI2tn) d/;" AE2l n--+oo Jx

H(/;" Qt, S) =

-

(3.2),

AE2l

by Lemma 3.9 (2),

Chapter I: Entropy

42

== - n-+oo lim.!. n ::; - L

'" Z::

~(A) 10g~(A),

by (3.4),

AEm v mn-l

~(A) 10g~(A)

+ ~(X) 10g~(X),

by Lemma 3.8.

AEm

For a general ~ EMs (X) we write

(i == yCI) with ~j E M:(X), j H(~, 2(, S)

== 1,2,3,4 and define

== H(~l, 2(, S) -

H(~2, 2(, S) + iH(~3, 2(, S) - iH(~4, 2(, S).

(5.1)

Then, the functional H (., 2(, S) so defined on M; (X) is called the entropy functional for the partition 2( and the automorphism S. Trivially, H(tt, 2(, S) coincides with the Kolmogorov-Sinai entropy for tt E Ps(X) defined in Section 1.3. We shall prove that H(·, 21, S) is linear on M:(X) and on Ms(X), and is bounded on Ms(X) in a series of lemmas. Lemma 1. For any ~,,,, E M:(X) and a,f3 ~ 0 it holds that H(a~ + f3"" 2(,

S) == aH(~,

2(,

S) + f3H(""

2(,

S).

(5.2)

Proof. Let ~,,,, E M:(X) and a,f3 > 0, and observe that

L

1

n AEmvm n -

=

(a~ + .B"1)(A) log(a~ + .B"1)(A) 1

_.!. L (a~( A) + .B"1(A)) log (a~(A) + .B"1(A») n

A

1.

1

== -- L a~(A) 10g~(A) - - Lf3",(A) 10g",(A) nAn A

'" '" ( ~(A) ) ;;:1 '7 a~(A) log ( a +.B ",(A)) ~(A) -;;:1 '7 .B"1(A) log .B + a "1(A) '(5.3)

where in the third and fourth sums on the extreme right we consider ~(A) log (a +

f3e~i~) == 0 if ~(A) == 0 and similarly ",(A) log (f3 + a~~~~) == 0 if ",(A) == ~(A) > 0, then by log(x + 1) ::; x for x > 0

o.

If

1.5. Entropy functionals

43

Hence we have

~~(X)aloga < ~

L a~(A) log (a + /3~~1~)

AE2l V 2l n -

l

1

::; -~(X)a log a n

1

+ -(3'fJ(X). n

Thus it follows from (5.3) that - lim .!. L(a~ + /3T/)(A) n4-00n

A

log(a~ + /3T/)(A)

= -a lim .!. L~(A) log~(A) n4-OO

n

A

/3 lim.!. L n4-OO

n

T/(A) logT/(A),

A

i.e., (5.2) holds. When a{3 = 0, (5.2) is obvious.

Lemma 2. If ~,~','fJ,'fJ' E M:(X) are such that ~ H(~, 2{, S)

'fJ =~'

- 'fJ',

then

- H('fJ, 2{, S) = H(~', 2{, S) - H('fJ' , 2{, S).

(5.4)

= ~' - 'fJ' implies ~ + 'fJ' = ~' + n, we see that

Proof. Since ~ - 'fJ

H(~, 2{, S) + H('fJ', 2{, S)

= H(~ + 'fJ', 2{, S), by Lemrna 1, = H(~' + 'fJ, 2{, S) = H(~', 2{, S) + H('fJ, 2{, S), by Lemma 1.

Thus (5.4) is true. Let M;(X) be the set of alllR-valued measures in Ms(X). By Lemma 2, H(·, is well-defined on M;(X) by H(~, 2{,

where ~

= ~+ -

t:

2{,

S)

S) = H(~+, 2{, S) - H(~-, 2{, S),

with ~+,~- E M:(X) is the Hahn decomposition.

Lemma 3. For any

~,'fJ E

H(a~

M;(X) and a, {3

E lR it holds that

+ (3'fJ, 2{, S) = aH(~, 2{, S) + (3H('fJ, 2{, S).

That is, H (., 2{, S) is a real linear functional on M; (X).

(5.5)

Chapter I: Entropy

44

Proof. It follows from (5.1) and Lemrnas 1, 2 that

eE M;(X). If e = 2: ek, where ek et c: with et, e; E k=l

H(ae, Q(, S) = aH(e, Q(, S),

a ~ 0,

n

Hence the LHS is well-defined.

M:(X) for 1 < k

< n, then

H(~,2t,S) = H( t~t - t~k,2t,S) k=l

=H (

t

k=l

k=l

~t ,a, S) -

H(

t

k=l

~k , 2t, S ) ,

by Lemma 2,

n

=

L

{H'(et, Q(, S) -lI(e;, Q(, S)},

by Lemma 1,

k=l n

=

LH(et - e;, Q(, S),

by Lemma 2,

k=l n

=

L H(ek, Q(, S). k=l

Moreover, for

e= e+ - t: E M;(X) with e+, t: E M:(X) it holds that

H( -e, Q(, S) = H(e- - e+, Q(, S)

= H(e-, Q(, S) - H(e+, Q(, S), =

and hence for any a

~

by Lemma 2,

-H(e, Q(, S),

0

lI(ae, Q(, S) = H(ae+ - ae-, Q(, S) = H(ae+, Q(, S) - H(ae-, Q(, S)

= aH(e+, Q(, S) - aH(e-, Q(, S) = aH(e+ - e-, Q(, S) = aH(e, Q(, S).

Therefore, (5.5) is proved. Lemma 4. Hi-, Q(, S) is a bounded linear functional on M;(X). More fully,

1.5. Entropy functionals

where

12(1

45

is the number of elements in 2(.

Proof. Observe that for ~ E M;-(X) and A E

o~ -

2(

i P~(AI§{oo) logP~(AI§{oo)

df,

::; Ilpe(AI21oo) log Pe(AI21oo ) Ile,oo~(X) 1

< 211~11, where

II· Ik,oo is the ~-ess. sup norm.

Thus we have

H(f" 21, S) <

e: with ~+,~- E M;-(X).

For a general ~ E M;(X) write ~ = ~+ that

IH(f" 21, S)

1~11If,1I. Then we see

I ~ HOf,I, 21, S) ~ 1~11I1f,111 = 1~11If,1I,

as desired. Lemma 5. H(·, 2t., 8) is well-defined and is a bounded linear functional on Ms(X) such that (5.6) IH(~, 2(, 8)1 < 12(111~11,

Proof. If

~ E

M;(X), then it follows from (5.1) that, H(i~,

2(, 8) =

iH(~,

2(, 8).

Hence it is easily seen that H(·, 2(, 8) is well-defined and linear by Lemmas 1, 2, 3 and 4. To see the boundedness of H (., 2(, 8) note that for ~, 'fJ E M; (X)

Then, we obtain

IH(~+i'fJ,2(,8)1 = VIH(~,2(,8)12+ IH('fJ,2(,8)1 2

~ I~I vllf,1I 2 + 1I'1J1I 2 , ::; Thus (5.6) is proved.

12(111~

+ i'fJll·

by Lemma 4,

Chapter I: Entropy

46

Summarizing Lemmas 1 - 5 we have the following theorem.

Theorem 6. The entropy functional H(·, 2(, 8) is a nonnegative bounded linear functional onMs (X) . Remark 7. Here are some properties of the entropy functional. Let' E M;-(X) and 2(, ~ E P(X). Then:

< ~oo =? H(" 2(, 8) < H(',~, 8). (2) 2( < ~ =? H(', 2(,8) ~ H(', ~,8). (3) H(" 8- 12(, 8) = H(" 2(,8). (4) H(" 2( V~, 8) < H(', 2(,8) + H(',~, 8) ~ 2H(', 2( V~, 8).

(1) 2t

The entropy of 8 under' E M;-(X) is defined by

H(" 8 ) = sup {H(', 2(,8) : 2( E P(X)}. If

2( E

P(X) is such that

2too

= X, then

H(~, 8)=

H(" 2(, 8),

by Remark 7 (1). More generally, we have the following.

Theorem 8. Let {2((n)} be a sequence in P(X) such that 2((1)

~

2((2)

~

... ,

Then it holds that

H(e, 8) = n-+oo lim H(" 2((n), 8),

(5.7)

Proof. Let' E M;-(X). For any 2( E P(X) and m, n ~ lone has that

1

k

E

~(A) log~(A)

AEQl v Qlk-l

2n+k

1

E

~--k-·2n+k AEQl(m)

~(A)log~(A)

v Ql(mhn+k-l

- E l»{ lAlogP((AllT(.=~nSjm(m)))d~ AEQl

J

47

1.5. Entropy functionals

= I(k, m, n) + J(m, n), For any e

> 0 we can choose

oS

ma

say.

(5.8)

2: 1 such that

J(rn, n) S J(rn,O) < e,

m 2: ma, n 2: 1.

Hence 1

k

L

~(A) log~(A) < I(k, m, n)

+ 10,

m

2: rna, n 2: 1.

AE2{V 2{k-l

If we let k -+ 00, then I(k, m, n) -+ H((" 2l(m) , S) and the LHS of (5.8)-+ H ((" 2l, S). Thus we have H((" 2l, S) S H((" 2l(m) , S)

Since {H((" 2l(m) , S)}

:=1

+ c,

m 2: ma, n 2: 1.

(5.9)

is monotonely nondecreasing by Remark 7 (2), lim H((" 2l(m) , S)

m-+oo

= He

exists. This and (5.9) imply that

H((" S) S He S H((" S), so that (5.7) holds. In the rest of the section we consider an integral representation of the entropy functionaL Take a /-L E Ps(X) and let MJ-t(X) be the set of all measures (, E M(X) such that (, ~ /-L. MJ-t(X) is a Banach space with the total variation norm II . II and is identified with L 1 (/-L) by I == /-L/ for I E L 1 (/-L), where P,f(A) Note that II/-L/II =

11/11t for I

E

=

L

L 1 (/-L).

f dp"

AEX.

S denotes the operator on M(X)

defined by

A E X, (, E M(X).

A linear functional F on MJ-t(X) is said to be S-stationary if F(S(,) = F((,) for (, E

MJ-t(X).

Let 2l E P(X) be fixed and consider an entropy functional H(·, 2l, S) on Ms(X) n MJ-t(X). For each (, E Ms(X), H((" 2l, S) is represented as

H(~, 21, S) =

Ix h~(x) ~(dx),

(5.10)

48

Chapter I: Entropy

he == -

L {Pel (AI2l

oo )

log Pel (AI2l oo )

Pe2 (AI2loo ) log Pe2 (AI2l oo )

-

AE~

+ iPe (AI2too ) log Pe (AI2too ) 3

3

-

iPe4 (AI2loo ) log Pe4 (AI2loo ) }

.

Our goal is to find a universal function h on X such that (5.10) holds instead of he. The following theorem is a partial solution to this.

Theorem 9. The entropy functional H(·, m, S) on Ms(X) n MJL(X) is extended to a functional H(.,m, S) on M JL (X) so that it is an S-stationary nonnegative bounded linear functional. Moreover, it has an integral representation

ll(("

Q(, S) =

L

h(x) (,(dx),

by some S-invariant nonnegative measurable function h on X. Proof. Let 'JJL be the a-subalgebra of X consisting of S-invariant (mod p,) sets, i.e., 'JJL == {A EX: J-L(S-l A~A) == O}. EJL(·I'JJL) denotes the conditional expectation relative to 'JJL under J-L. We define a functional H(., m, S) on MJL(X) by

ll(("

Q(, S)

= H ( E p (:~

IJ

p)

du, Q(, S),

(-f

Then this functional is well-defined since EJL I'JJL)djl E Ms(X). Moreover, it is linear. The boundedness of H(., m, S) follows from

Ill(("

Q(, S)

I= IH(e; (:~ IJp )djL, Q(, S) I < 1Q(I"Ep(:~IJp)djLll ~ Imlll~11

for ~ E MJL(X). To see that

H(., m, S) is S-stationary, observe that

1.6. Relative entropy and K ullback-Leibler information

49

since d
Ii(~,2(,S) =

Ix h(x)~(dx),

Clearly, h is nonnegative and S-invariant. If we can find a probability measure J.-L which dominates all of Ms(X), i.e., ~ « J.-L for ~ E Ms(X), then we see from the above theorem an integral representation of H(~, 2(, S) (~ E Ms(X)) by a universal function h such that

H(e, 2(, S) =

Ix

h(x) ~(dx),

In this case, h(·) == h(·, 2(, S) is called an entropy junction for 2( and S. This representation can be shown without assuming an existence of a dominating measure in Section 2.7 using Ergodic Theorems and ergodic decompositions.

1!,6. Relative entropy and Kullback-Leibler information In Section 1.1, we defined relative entropy H(plq) for two finite probability distributions p and q. In this section, we extend this definition to an arbitrary pair of probability measures. Some properties of relative entropy are obtained. As an application, we discuss sufficiency of a a-subalgebra in terms of relative entropies of measures. In testing hypotheses, relative entropy is used as a good criterion. We shall discuss these points in sorne detail. Let (X, X) be a measurable space. P(X) and P(X) are the same as before. Let q)be a a-subalgebra of X and take J.-L, v E P(X). Then, the relative entropy of J.-L w.r.t. v relative to q) is defined by

H~ (J.-Llv) == sup {..• L

AE2(

JL(A) log

~i~~

:

2( E

P(!D)},

(6.1)

where P(q)) is the set of all finite q)-measurable partitions of X. If q) == X, then we write HX(J.-Llv) == H(J.-Llv) and call it the relative entropy of J.-L w.r.t. t/, Note that the relative entropy is defined for any pair of probability measures. The following lemma is obvious.

Chapter I: Entropy

50

Lemma 1. Let JL, t/ E P(X) and ~1, ~2 be a-subalqebras. (1) H(JLlv) ~ 0; H(JLlv) =.0 ¢:> J-t = u,

(2) ~1 ~ ~2 When JL

~

=}

Hf.Dl (JLlv) ::; Hf.D2 (JLlv)

< H(J-tlv).

u, relative entropy has an integral form as is shown in the following.

Theorem 2. Let JL, v E P(X). (1) If JL « u, then

(6.2) (2) If JL -;t::.

t/,

then H(JLlv) =

00.

Proof. Since (2) is obvious we prove (1). Since 'ljJ(t) = tlogt (t > 0) is convex, we have for A E X with v(A) > 0 and for a measurable function f ~ 0 that

by Jensen's Inequality (cf. Section 1.2 (12)). If, in particular, JL(A) j1.(A) log v(A) :S

It then holds that for any

2{ E

To show the reverse inequality, let k

=

!fIf;, then

P(X)

Taking the supremum on the LHS, we see that by (6.1)

n,

=

dJL) 1fA (dJL· dv log dv du.

""' JL(A) ""' f (dJL dJL) LJ j1.(A) log v(A) :S ALJ lA dv log dv du = AEm Em

A

f

['5.- < dJL < k + n - du

dJL . ] A n ,n 2 = [ dv ~ n .

1]

n'

lxf

(dJL dJL) dv log dv du.

51

1.6. Relative entropy and K ullback-Leibler information

Then {An,k : 0 ::; k ::; n 2 } E P(X) for each n 2:: 1. Moreover, by the definition of

An,k'S k -v(A n k) ::;

n

for 0 ::; k ::; n

2

'

!

X

lA

k

n,

dJ.L dv::; --v(A k+ 1 -d n k) n

V

'

1 and n 2:: 1. This implies

-

k +1 -k < J.L(An,k) < -. n - V(An,k) -

n

Let K 1 = {k: ~ ::; ~} and K 2 = {k: ~ 2:: ~}. Since 'lj;(t) on (0, ~) and increasing on (~, 1), we have that k + II k + 1 - og - - < J.L(An'k) oIg J.L(An,k) n n - V(An,k) V(An,k) ,

-k Iog -k < J.L(An'k) oIg J.L (An ,k) n n - V(An,k) V(An,k) ,

= tlogt is decreasing

k E K 1,

k E K 2•

Define a function b« for n 2:: 1 by

In(x),

x E

hn(x) = { ( ) 9n x ,

xE

n2- 1

U

An k

U

An k '

kEK2 kEK l

'

'

n2- 1

where In = '" '" ~ klA n n, k and 9n = ~ tl!IA n n, k' It then follows that for n 2:: 1 k=O

k=O

which implies that

H(p,lv) 2: On the other hand, let

I

Ln;

log n; du.

= ~ and observe that for x E X and n 2:: 1

o::; 9n(x) o::; I(x) -

1 n

In (x) ::; -, 1 n

In(x) ::; 9n(X) - In(x) ::; -,

Chapter I: Entropy

52

Then we see that

o :s; If -

hn I :s;

If - f n 11 kEUK2 An ' k + If - gn 11 kEKl U An k , '

n

~

1,

and hence

{ flog f dv = lim ( h n log h n du n~oo lx

lx

< H(/1lv).

Thus the reverse inequality is proved. Therefore (6.2) is true. Let us examine some properties of the relative entropy, which are collected in the following theorem. Theorem 3. (1) Let ~n (n ~ 1) and ~ be a-subolqebras of eX and /1, v E P(X). If ~n t~, then H fD n (/1lv) t HfD(/1l v). (2) Let 0 :s; a < 1 and /1j, Vj E P(X) (j = 1, 2). Then,

II(a/11

+ (1 -

a)/12I a Vl

+ (1

(3) If II/1n - vnll -+ 0 and Ilvn -

a)v2):S; aH(/11Ivl)

vII

+ (1 -

-+ 0 with {/1n, /1, u«,

t/ :

a)H(/12Iv2).

(6.3)

n ~ I} ~ P(X), then

(6.4) (4) If /1, v

E

P(X), then

Proof. (1) By Lemrna 1, {H fD n (/1lv)} is a monotone nondecreasing sequence. only have to show the convergence: H fD n (/1lv) -+ H~ (/1lv). Suppose first /1 ~ t/, For n ~ 1 let /1n = /11~n and V n = vl~n' the restrictions of /1 and ~n, respectively. Then /1 ~ v implies /1n ~ V n for n ~ 1. If we let fn = ~~: f = ~~, then it follows from Theorem 2 that for n ~ 1

Since {fn : n ~ I} is a martingale in L1(v), we have fn -+ f t« log fn -+ flog f u-a.e. Then, Fatou's lemma implies that

HV(fLlv) = [ flogfdv:::; liminf [ fn10gfndv lx n~oo lx

We that v to and

v-a.e. and hence

1.6. Relative entropy and K ullback- Leibler information

53

from which we conclude that H!Dn(JLlv) -+ H!D(JLlv). Now assume that JL 1:- u, Then there is an A E ~ such that v(A) = 0 and JL(A) > O. Let e > 0 be such that JL(A) > 2c. Since ~n t ~, there exist n 2: 1 and B E ~n such that v(A~B) < c. JL(A~B) < c, Now observe that lim H!Dn (JLlv) 2: H!Dn (JLlv) 2: H~(JLlv)

n-+oo

JL( B)

JL( BC)

c

= M(B) log v(B) + M(B ) log v(Bc) ,

(6.5)

where ~ = {B, B C } E P(~n). Note that

v(B) < v(A)

+e=

c.

Thus one has

Therefore, by (6.5) we have lim II9J n (Mlv) 2': !M(A) log M(B) + M(BC ) 2 e -+ 00 as e -+ O.

-

v(B C )

n-+oo

That is, (2) We can assume 0 < a < 1. If JLl 1:- VI or JL2 both sides of (6.3) become 00. So we assume JLl « prove (6.3) it suffices to show that for any 2t E P(X)

1:- V2, VI

then (6.3) is true since and JL2 «V2. In order to

(6.6)

Chapter I: Entropy

54

Let

Cl, C2,

d1 , d2 be nonnegative constants and consider a function sp defined by

cp(x) =

{ XCI

XCI

+ (1 -

X)C2 }

log d X

+ (1-x)c2

1+ (1 - X )d· 2

Then we see that

for

X

E [0,1] since

Cj,

dj 2:: 0 for j = 1,2. Thus,

ip

is convex and hence


aE[O,l],

which proves (6.6). Therefore (6.3) holds. (3) To prove (6.4) it suffices to show 2( E

P(X),

since H(J-Llv) = sup {H§((J-LllJ) : 2( E P(X)}. IIJ-Ln J-LII -t 0 and Ilvn that J-Ln(A) -t J-L(A) and vn(A) -t v(A) for A E X. If v(A) > 0 then

.

J-Ln(A) .

vII

-t

0 imply

J-L(A)

2~~ JLn(A) log vn(A) = JL(A) log v(A) · If v(A)

= 0 and J-L(A) > 0, then . . J-Ln(A) nl~~ JLn(A) log vn(A) =

If v(A)

J-L(A)

00

= JL(A) log v(A) .

= 0 = J-L(A), then liminf JLn(A) log JLn((AA)) n-.+oo

since log x 2:: 1 - ~ for x >

Vn

o.

~ liminf(JLn(A) n-.+oo

vn(A)) = 0

By the definition of H§( (J-Ln Ivn) we see that

(4) Let J-L, v E P(X). By the Hahn decomposition there is some B E X such that AEX.

55

1.6. Relative entropy and Kullback-Leibler information

It follows that

For

2(

= {B, Be} E P(X) it holds that

H2i(J.Llv)

I-£(B)

1 - I-£(B)

= J.L(B) log v(B) + (1- J.L(B)) log 1 _

v(B)'

Now we note that

V

2 { Y log 'It + (1 - y) log 1 - y} x

I-x

~ 2(y -

x),

o < x < y < 1.

(6.7)

This can be seen as follows. Fix y E [0,1] and consider a function p defined by

1-

y

x)2,

p(x)=ylog-+(I-y)log -2(y x I-x Then we have '( ) = (x - y) (2x - 1)

P x

x(l- x)

2

o < x < y.

<0 -,

and hence p(x) ~ p(y) = O. That is, (6.7) is true. Now from (6.7) we see that, by letting x = v(B) and y = I-£(B),

as was to be proved. When the measurable space (X, X) is (R, ~), where lR. is the real line and ~ is the Borel a-algebra, we can consider Kullback-Leibler information. Probability measures on (lR.,23) naturally arise from real random variables on a probability measure space (X, X, 1-£). Let ~ and rJ be real random variables on (X, X, /,,), so that they have probability distributions I-£e and 1-£"., given by AE~,

respectively. Suppose that I-£e and 1-£"., are absolutely continuous w.r. t. the Lebesgue measure dt of lR., so that we have the probability density functions f and g, respectively given by

Chapter I: Entropy

56

In this

the K ullback-Leibler

'1f1l

- - - - - - - -

I(17)

=

is defined by

rrvr-m.nrsrvn

0 _ _

1

(J(t) log Jet) - Jet) logg(t)) dt.

Lemma 5. Let f and g be probability density functions on JR. Then

-1

Jet) log Jet) dt -::;.

-1

Jet) log get)dt

(6.8)

is true if both integrals are finite. The equality holds iff f = 9 a.e. (in the Lebesgue measure). Proof. Since t - 1 2: logt for t > 0 and "=" holds iff t = 1, we have

j(t)(logg(t) -logj(t)) = Jet) log ~~:~ -::;. get) where we define ~

= 00 (a > 0) and

0·00

Jet) a.e.,

= O. This implies that

1

(J(t) log Jet) - Jet) logg(t)) dt

-::;.1 (g(t) -

Jet)) dt = 0,

or (6.8) is true. Obviously, the equality holds iff f(t) = g(t) a.e.

Lemma 6. Let ~ and 'fJ be a pair of real random variables with probability distributions J.le and J.l"" respectively. If J.le and J.l", have probability density functions f and g, respectively, then

Proof. If H(J.lelJ.lTJ) < 00, then J-te I(17)

= = =

«

J.lTJ by Theorem 2. Hence

1 1

(J(t) logj(t)~ Jet) logg(t)) dt

Jet) log ~gj dt

t v« log dJ.le/ dtdt dt

JJR dt

= f d/-L<. JJR dJ.lTJ

dJ.lTJ/

log d/-L<. d/-L1j dJ.lTJ = H(J.lelJ.lTJ)·

1.6. Relative entropy and Kullback-Leibler information

57

Sirnilarly, it can be shown that if I(~I1]) < 00, then J-Le ~ J-L11 and H(J-LelJ-Ll1) = I(~I1])· When ~ is a real Gaussian random variable, the density function f(t) can be written as

f(t)

=

y

1 ~exp 2

21Ta

((t m)2) 2 a

2

'

t E lR

for some m E lR and a > o. In this case, the probability measure J-Le is also said to be Gaussian. As is well-known any two Gaussian measures J-Ll and J-L2 are equivalent (i.e., rnutually absolutely continuous) or singular. That is, J-Ll ~ J.-l2 (Le., J.-ll « J-L2 « J.-ll) or J.-ll .-L J.-l2· This fact can be stated in terms of relative entropy as follows:

Proposition 7. Let J.-ll and J-L2 be a pair of Gaussian probability measures on (lR, ~). Then, J-Ll ~ J-L2 iff H(J-LIIJ-L2) < 00, and J-Ll .-L J.-l2 iff H(J.-lIIJ-L2) = 00. As an application we consider sufficiency and hypotheses testing in connection with relative entropy. We begin with some definitions.

JIL Definition 8. Let 9J1 and SJt be subsets of P(X) . .:LL!§J~~~L!!:J?E2J~~f21WJJ~ uous w.r.t. or dominated by 9J1, denoted SJt ~ 9J1, provided that, if A E eX is such that J-L(A) = 0 for every J.-l E 9J1, then v(A) = 0 for every v E SJt. 9J1 and SJt are said to be equivalent, denoted 9J1 ~ SJt, if VJt -e; SJt ~ 9J1. -Z!J~1'~ill-l2-~-!!:l~~~~ there exists a A E P(X) such that VJt ~ {A}.

Lemma 9. If VJt ~ P(X) is a dominated set, then there exists a countable set SJt ~ P(X) such that VJt ~ SJt.

Proof. Suppose that 9J1 « {A} and let fp, = ~ and tc; = [fp, > 0] for J.-l E VJt. A subset K E eX is called a chunk if there is some J.-l E VJt such that K ~ Kp, and J-L(K) > o. A disjoint union of chunks is called a chain. Since A(X) < 00 and A(K) > 0 for every chunk K, we see that every chain is a countable disjoint union of chunks. We also see that a union of two disjoint chains is a chain and, hence, a countable union of (not necessarily disjoint) chains is a chain. Let a = sup { A(0) : 0 is a chain}. Then, we can find an increasing sequence 00

{On} of chains such that lim A(On) = a. If we let 0 = U On, then 0 is a n-+oo n=l chain and A(O) = a. Moreover, there exists a sequence {Kn } of chunks such that c = n=l U Knwith a sequence of measures {J-Ln} ~ 9J1 such that K n ~ Kp,n and J-Ln(Kn) > 0 for n ~ 1. Let SJt = {J-Ln : n ~ I}. Obviously SJt ~ VJt since SJt ~ VJt. We shall show VJt ~ SJt. Suppose that A E eX satisfies J-Ln(A) = 0 for n ~ 1. Take an arbitrary J.-l E VJt. Since J-L(A\KJ-£) = 0 by the definition of Kp" we can assume that A ~ KJ-£. If J-L(A\O) > 0, then A(A\O) > 0 and hence Au 0 is a chain with A(A U 0) > A(O) = a. This contradicts the maximality of O. Thus J-L(A\C) = o.

Chapter I: Entropy

58

Now observe that

00

A{A n C) ==

:E A{A n K n) == 0 n=l

since 0 == JLn{A) == JL{A n K n) == JAnK n fJ-tn dA and K; ~ KJ-tn imply A{A n K n) == 0 for n 2: 1. Therefore, JL{A) == JL{A\C} + JL{A n C) == O. This means VJt· ~ SJt. We now introduce sufficiency. Definition 10. Let ~ be a a-subalgebra of X and 9J1 ~ P{X). ~ is said to be sufficient for 9J1 if, for any A E X, there exists a ~-measurable function hA such that JL E 9J1. That is, ~ is sufficient for 9J1 if there exists a conditional probability function common to every JL E rot. The following is a fundamental characterization of sufficiency. Theorem 11. Let 9J1 ~ P{X) be a dominated set and ~ be a a-subalqebra of X. Then, ~ is sufficient for 9J1 iff there exists A E P{ X) such that 9J1 ~ {A} and ~ is ~ -measurable JL-a. e. for JL E 9J1.

Proof. Suppose that n 2: 1}. Let

~

is sufficient for 9J1. By Lemma 9 we assume that VJt == {JLn : 1

00

A(A)

= :E 2nJLn(A),

AEX.

n=l

Then, A E P{X) and VJt ~ {A}. Since ~ is sufficient for 9J1, for each A E X there exists a ~-measurable function hA such that

A(A n B)

=

f 2~ 1

Ep,n (lAI!D) dJLn

n=l

=

B

1·

hA

a»,

B

Hence EA{1AI~) == hA A-a.e. Take any JL E 9J1 and let 9 == ~. Then, for any A E X we have

L = L~~ = = Ix Ix = Ix = Ix = Ix = Ix gdA

dA

=

hA dJL

JL(A)

Ep,(lAI!D)dJL

E.x(lAI!D) dJL

E.x(1A 1!D)E.x (gl!D) dA

E.x(lAI!D)gdA

E.x(lAE.x(gl!D)I!D) o.

1.6. Relative entropy and K ullback-Leibler information

59

which implies that 9 = EA(gl~) and 9 is ~-measurable A-a.e. Conversely, suppose that there exists A E P(X) such that rot ~ {A} and ~ is ~-measurable J.-L-a.e. for J.-L E rot. Let J.-L E rot and A E X be arbitrary and let 9 = ~, which is ~-measurable u-a.e. Then, for B E ~ it holds that

L

E),(lAI!D) dtt =

= Since B E

Hence

~

~

L

E),(lAI!D)gd>'

L

lAg o. =

L

E,,(lAI!D) du.

is arbitrary, we see that

is sufficient for rot.

Remark 12. Let rot ~ P(X) and ~ be a a-subalgebra of X. rot is said to be homogeneous if J.-L ~ v for J.-L, lJ E rot. If rot is homogeneous, then ~ is sufficient foroot iff ~~ is ~- measurable for J.-L, lJ E rot. This is seen as follows. Let A be any measure in 001. Then 001 is dominated by A and Theorem 11 applies. We introduce another type of sufficiency.

Definition 13. So let rot pairwise sufficient for m1 if

~ ~

P(X) and ~ ~ X, a a-subalgebra. ~ is said to be is sufficient for every pair {J.-L, lJ} ~ 001.

It is clear from the definition that sufficiency implies pairwise sufficiency. To consider the converse implication, we need the following lemma.

Lemma 14. Let 001 ~ P(X) and ~ ~ X, a a-subalgebra. Then, ~ is pairwise sufficient for 001 iff, for any pair {J.-L, lJ} ~ rot, d(:~V) is ~-measurable (J.-L + lJ)-a.e. Proof. Assume that ~ is pairwise sufficient for 001. Let J.-L, lJ E 001 be arbitrary and let A = ~ E P(X). Since ~ is sufficient for {J.-L, lJ}, we have that ~ and ~~ are ~-measurable by the argument similar to the one given in the proof of Theorem 11. Thus d(:~v) is also ~-measurable. The converse is obvious in view of Theorem 11. Theorem 15. Let rot ~ P(X) be dominated and ~ ~ X, a a-subalgebra. Then, is sufficient for 001 iff ~ is pairwise sufficient for 001.

~

Chapter I: Entropy

60

Proof. The "only if' part was noted before. To prove the "if' part, suppose that is pairwise sufficient for oot. We can assurne that oot = {J.Ln : n 2: I} in view of Lemma 9. Let 00 1 AEX. .\(A) = 2nJln(A), ~

L

n=l

It follows from Lemma 14 that d(~~A) is ~-measurable for J.L E oot. Consequently, for J.L E oot d d(Jl: .\)

)-1

is

~-measurable.

Theorem 11 concludes the proof.

Now sufficiency can be phrased in terms of relative entropies. Theorem 16. Let J.L, v E P(X) and ~ ~ X, a a-subalqebra. (1) If ~ is sufficient for {J.L, v}, then H~ (J.Llv) = H(J.Llv). (2) If H~ (J.Llv) = H(J.Llv) < 00, then ~ is sufficient for {J.L, v}. Proof. We only have to show the case where J.L ~ u, Let J.Lo the restrictions of J.L and v to ~, respectively. Then, H(Jllv) - H!l)(Jllv)

= }fx (dJ.L log dv =

=

J.LI~

and Vo

=

vl~,

dJ.LO)

- log dvo dJl

r dJl/ dv log dJ.LodJl// dvdvo . dJlo du dvo

}x dJ.Lo / dvo

=

Ix

flog f d("

where f = d~j:o and d~ = ~dv. Let 'ljJ(t) = tlogt, t > O. Then, since and hence 0 < f < 00 u-a.e., we can write ¢(J(x)

= ¢(1) + {I(x) = f(x)

- 1+

- 1}¢'(1) + ~ {I (x) - 1}2¢" (g(x)

{f(x) - I} () 2g x

2

u-a.e.,

where g(x) E (1, f(x)) or (f(x), 1) and hence 0 < g <

Ix

Ix f d~ =

¢(J(x))(,(dx)

~0

00

u-a.e. Thus we see that

1

61

1.6. Relative entropy and Kullback-Leibler information

Ix

since f d~ == 1. The equality holds iff f == 1 v-a.e., Le., H(JLlv) == H~(JLlv) iff ~~ == ~~~. Let A == ~. Then, {JL, v} ~ {A} and Theorem 11 irnplies the assertions. To conclude this section we consider a statistical hypothesis testing problem. In a simple hypothesis testing

Hs : p == (p(al),' .. ,p(an ) ) HI: q

== (q (aI), . . . ,q(an)) ,

we have to decide which one is true on the basis of samples of size k, where p, q E ~n and X o == {al,'" ,an}' We need to find a set A ~ X~, so that, for a sample (Xl, ... ,Xk), if (Xl, ... ,Xk) E A, then Hi, is accepted; otherwise HI is accepted. Here, for some E E (0,1), type 1 error probability P(A) satisfies k

L

IIp(Xj) == P(A)

<e

(Xl, ... ,xk)EAj=1

and type 2 error probability Q(A) is given by

L

k

II q(Xj) == 1 -

Q(A) == Q(A C ) ,

(XI, ... ,Xk)(/:. Aj=l

which should be minimized. Under these notations we have the following.

Theorem 17. In a hypothesis testing problem mentioned above, for any and the sample size k ~ 1 let

E

E

(0,1)

(6.9) Then it holds that

lim -k110g (3(k, e)

(6.10)

k-too

Proof. Let ~l,' •• ,~k be independent, identically distributed (i.i.d.) random variables taking values in X o with the probability distribution p == (p(al), ... ,p(an ) ) and let 1~j

< k,

Chapter I: Entropy

62

so that 'fJj '8 are also i.i.d. with the same mean

Ep('fJj)

~

p(aj) q aJ

= L....Jp(aj) log - ( _) = H(plq), j=l

1 ::; j ::; k.

Hence it follows from the weak law of large numbers (cf. e.g. Rao [2]) that for any 8>0 (6.11) For each 8

> 0 let _

E(k,8) -

{

(Xl, ... ,Xk)

i«,

E Xo

' I

k k ~log q(Xj) -

1

p(Xj)

= 1,

8>

H(plq)1 ::; 8.} ·

Then by (6.11) we have that

lim P(E(k,8))

k-+oo

o.

(6.12)

For (Xl, ... ,Xk) E E(k, 8) it holds that k

IT p(Xj) exp { -

k

k(H(plq) - 8)} ~

j=l

IT q(Xj)

j=l k

~ ITp(xj)exp{ -k(H(plq)+8)},

j=l and hence for sufficiently large k

Q(E(k,8))

=

k

IT q(Xj)

L (Xl, ... ,xk)EE(k,d)

<

j=l k

L

ITp(xj)exp{-k(H(plq)-8)}. (Xl, ... ,xk)EE(k,d) j=l

By the definition of (3(k, 8), (6.9), we see that 1

1

k log {3(k, 8) ::; k 10gQ(E(k, 8))

::; -H(plq)

+ 8,

63

Bibliographical notes

which implies that

limSUP~lOg,6(k,8)S; -H(plq) +8.

(6.13)

k-+oo

On the other hand, in view of the definition of E(k, 8) we have

k

L

II q(Xj)

(Xl, ... ,xk)EA c n E (k ,8) j=l k

L

~ (Xl, ...

II p(Xj) exp { -

,Xk )EA c n E

=P(A n E (k,8)) exp { C

Since peA)

~

-k(H(plq)+8)}.

e, (6.12) implies that for large enough k

and hence

k(H(plq) + 8)}

(k ,8) j=l

~

1

1-e

Q(A C ) ~ -2- exp { - k(H(plq) + 8)}.

Since the RHS is independent of A it follows that

1-£

(3(k, 8) ~ -2- exp {

so that

110g{3(k,8) liminfk-e-co k

k(H(plq) + 8)},

~

-H(plq) - 8.

(6.14)

Since 8 > 0 is arbitrary, combining (6.13) and (6.14) we conclude that (6.10) holds.

Bibliographical notes There are some standard textbooks of information theory: Ash [1](1965), Csiszar and Korner [1](1981), Feinstein [2](1958), Gallager [1](1968), Gray [2](1990), Guiasu [1](1977), Khintchine [3](1958), Kullback [1](1959), Martin and England [1](1981), Pinsker [1](1964), and Umegaki and Ohya [1, 2](1983, 1984). As is well recognized there is a close relation between information theory and ergodic theory. For instance, Billingsley [1] (1965) is a bridge between these two theories. We refer to some textbooks in ergodic theory as: Brown [1](1976), Cornfeld, Fomin and Sinai [1](1982),

Chapter I: Entropy

64

Gray [1](1988), Halmos [1, 2](1956, 1959), Krengel [1](1985), Ornstein [2](1974), Parry [1, 2](1969, 1981), Petersen [1](1983), Shields [1](1973) and Walters [1](1982). Practical application of inforrnation theory is treated in Kapur [1](1989) and Kapur and Kesavan [1](1992). The history of entropy goes back to Clausius who introduced a notion of entropy in thermodynamics in 1865. In 1870s, Boltzman [1, 2](1872, 1877) considered another entropy to describe thermodynarnical properties of a physical system in the rnicro-kinetic aspect. In 1928, Hartley [1] gave some consideration of the entropy. 'Then, Shannon came to the stage. In his epoch-making paper [1](1948), he really "constructed" information theory (see also Shannon and Weaver [1](1949)). The history of the early days and development of information theory can be seen in Pierce [1](1973), Slepian [1, 2](1973, 1974) and Viterbi [1](1973).

1.1. The Shannon entropy. Most of the work in Section 1.1 is due to Shannon [1]. The Shannon-Knintchine Axiom is a modification of Shannon's original axiom by Khintchine [1](1953). The Faddeev Axiom is due to Faddeev [1](1956). The proof of (2) =} (3) in Theorem 1.4 is due to Tverberg [1](1958), who introduced a weaker condition than [1°] in (FA). 1.2. Conditional expectations. Basic facts on conditional expectation and conditional probability are collected with or without proofs. For the detailed treatment of this matter we refer to Doob [1](1953), Ash [2](1972), Parthasarathy [3](1967) and Rao [1, 3](1981, 1993). 1.3. The Kolrnogorov-Sinai entropy. KoIrnogorov [1](1958) (see also [2](1959)) introduced the entropy for automorphisrns in a Lebesgue space and Sinai [1](1959) slightly modified the Kolmogorov's definition. As was mentioned, entropy is a complete invariant among Bernoulli shifts, which was proved by Ornstein [1](1970). There are measure preserving transformations, called K -automorphisms, which have the same entropy but no two of them are isomorphic (see Ornstein and Shields [1] (1973)). 1.4. Algebraic models. and Foias [2, 3](1968). Chi section to projective limits are seen in Dinculeanu and

'The content of this section is taken frorn Dinculeanu and Dinculeanu [1](1972) generalized the results in this of measure preserving transformations. Related topics Foias [1](1966) and Foias [1](1966).

1.5. Entropy functionals. Affinity of the entropy on the set of stationary probability measures is obtained by several authors such as Feinstein [3](1959), Winkelbauer [1](1959), Breiman [2](1960), Parthasarathy (1961) and Jacobs [4](1962). Here we followed Breiman's method. Umegaki [2, 3](1962, 1963) applied this result to consider the entropy functional defined on the set of cornplex stationary measures. He obtained an integral representation of the entropy functional for a special case. Most of the work of this section is due to Umegaki [3]. 1.6. Relative entropy and Kullback-Leibler information. Theorem 6.2 is stated

Bibliographical notes

65

in Gel'fand-Kolmogorov-Yaglom [1](1956) and proved in Kallianpur [1](1960). (4) of Theorern 6.4 is due to Csiszar [1](1967). Sufficiency in statistics was studied by several authors such as Bahadur [1](1954), Barndorff-Nielsen [1](1964) and Ghurge [1](1968). Definition 6.8 through Theorem 6.15 are obtained by Halmos and Savage [1](1949). We treated sufficiency for the dominated case here. We refer to Rao [3] for the undorninated case. Theorem 6.16 is shown by Kullback and Leibler [1](1951). Theorem 6.17 is given by Stein [1] (unpublished), which is stated in Chernoff [2](1956) (see also [1](1952)). Hoeffding [1](1965) also noted the same result as Stein's. Related topics can be seen in Blahut [2](1974), Ahlswede and Csiszar [1](1986), Han and Kobayashi [1, 2](1989) and Nakagawa and Kanaya [1, 2](1993).

CHAPTER II

INFORMATION SOURCES

In this chapter, information sources based on probability measures are considered. Alphabet message spaces are reintroduced and examined in detail to describe information sources, which are used later to model information transmission. Stationary and ergodic sources as well as strongly or weakly mixing sources are characterized, where relative entropies are applied. Among nonstationary sources AMS ones are of interest and examined in detail. Necessary and sufficient conditions for an AMS source to be ergodic are given. The Shannon-McMillan-Breiman Theorem is formulated in a general measurable space and its interpretation in an alphabet message space is described. Ergodic decomposition is of interest, which states that every stationary source is a mixture of ergodic sources. It is recognized that this is a series of consequences of Ergodic and Riesz-Markov-Kakutani Theorems. finally, entropy functionals are treated to obtain a "true" integral representation by a universal function.

2.1. Alphabet message spaces. and information sources

In Example 1.3.14 Bernoulli shifts are considered on an alphabet message space. In this section, we study this type of spaces in more detail. Also a brief description of measures on a compact Hausdorff space will be given. Let X o = {al, ... ,ai} be a finite set, so called an alphabet, and X = X~ be the doubly infinite product of X o over Z = {a, ±1, ±2, ... }, i.e., 00

X=X~=

II

Xk,

k=-oo

Each x E X is expressed as the doubly infinite sequence

67

Chapter II: Information Sources

68

defined by

8 : x 1---+ x'

== 8 x == (. ..

,X~ 1 , X~, X~,

... ),

X~

== xk+1, k E Z.

Denote.. a cylinder set by &I

[x?··· xJ] == [Xi == X?, ... ,Xj == xJ] == {X == (Xk) EX: xk == x2, i ~ k ~ j}, where x2 E X o for i ~ k ~ j and call it following properties:

(1) i ~ s ~ t ~ j

can verify the

[x? ... xJ] ~ [x~ ... x~]; (2) [x?··· xJ] f- [y? ... yJ] {:} [x? ... xJ] n [y? ... yJ] == 0;

(3) [x?· .xJ] ==

=}

n {[x2] : i < k ~ j};

(4) i ~ s ~ t ~ j =} [x~·· ·x~] == U {[Xi" 'Xj] : Xk == x2, s ~ k ~ t}; (5) [x?··· xJ]C == U {[Xi' .. Xj] : Xk f- x2 for some k == i, ... ,j};

U{[Xi] : xi E X o} == U{[Xi"

== X. Thus the set 9J1 of all rnessages forms a semialgebra, i.e., (i) 0 E 9J1; (ii) A, B E 9J1 =} n An B E 9J1; and (iii) A E 9J1 =} AC == .U B j with disjoint Bi . ... ,Bn E 9J1. 8 is a (6)

'Xj] : Xi,··· ,Xj

E X o}

J=l

one-to-one and onto mapping such that (7) 8- 1 ((Xk))

== (Xk-1) for (Xk) EX; n (8) 8- [x?· .. xJ] == [Y?+n ... yJ+nJ with y2+n == x2 for i ~ k ~ j and n

E Z.

Let X be the a-algebra generated by all rnessages 9J1, denoted X a(9J1). Then (X, X, 8) is called an alphabet message space. Now let us consider a topological structure· of the alphabet message space (X, X, 8). Letting

d( ') _ ~ dO(Xk' x~) X,X - L..J 21 kl '

X,X' EX,

(1.1)

k=-oo

we see that X is a compact metric space with the product topology and 8 is a homeomorphism on it. Recall that a compact Hausdorff space X is said to be totally disconnected if it has a basis consisting of closed-open (clopen, say) sets. Then we have the following:

Theorem 1. For any nonempty finite set X o the alphabet message space X == X~ is a compact metric space relative to the product topology, where the shift S is a

2.1. Alphabet message spaces and information sources

69

homeomorphism. Moreover, X is totally disconnected and X is the Borel and also Baire a-algebra of x. Proof. The shift S is continuous, one-to-one and onto. Hence it is a horneomorphism. X is totally disconnected. In fact, the set VR of all messages forms a basis for the product topology and each message is clopen. To see this, let U be any nonempty open set in X. It follows from the definition of the product topology that there exists a finite set J = {jl' ... ,jn} of integers such that prk(U) = X k = X o for k t/:. J, where prk(·) is the projection onto the kth coordinate space X k . Let i = min{k : k E J} and j = max{k : k E J}. Then we see that, for any U = (Uk) E U,

[Ui·· ·Uj]


and

U=

U[Ui·· ·Uj]. uEU

This rneans that VR is a basis for the topology. Each message is clearly clopen. In the rest of this section, we consider a compact Hausdorffspace X~:rul its Baire a-algebra X with a measurable transformation S on X. ~~~))and~Xpdenote the Banach spaces of all continuous functions a~ire measurable functIOns on X with sup-norm, respectively. As in Chapter 1, {~(~ denotes the Banach space of all on X. In this case, M(X) is the space of all Baire measures on X. P; (X)) denotes the set of all (resp. S-invariant) probability Each measure J.L E P(X) (or Ps(X)) is a source or every ergodic sources in P; (X). Example 2. Let X o = {al, . . . ,al} be an alphabet with a probability distribution p = (PI, .. . ,P.e)· Consider the alphabet message space X = X~ with a shift S on it. For a rnessage [x? ... xJ] we define

J.Lo([x? .. xJ]) = p(x?) ... p(x~).

(1.2)

Then, J.Lo is defined on the algebra A(VR) generated by VR, the set of all messages, and is S-invariant such that J.Lo(X) = 1. By the Caratheodory extension theorem J.Lo can be extended uniquely to an S-invariant probability measure J.L on X = i.e., J.L E Ps(X). This J.L is called a (PI, . . . ,p.e)-Bernoulli _______ S is called a (PI, ... ,Pl)-Bernoulli shift as in £.JX,cUIlJ.Jle We claim that J.L is ergodic. To see this, suppose that A E X is S-invariant and let e > 0 be arbitrary. Choose B E A(VR) such that J.L(ADt.B) < e and hence ~~"c=c,~~==~~,=__~

_=

k

IJ.L(A) - J.L(B) I < e. Since B = .U B j with disjoint Bl, .. . ,Bk E VR, we can choose J=l

no 2:: 1 such that s-no B have different coordinates from B. This implies that J.L(s-no B n B) = J.L(s-n o B)J.L(B) = J.L(B)2

Chapter II: Information Sources

70

by virtue of (1.2). Then we have

jj(AilS- noB) = jj(s-n oAils-no B),

since A is S-invariant,

= jj(s-no(A~B)) = jj(A~B)

< e,

and hence jj(A~(B

n s» B))

~ jj( (A~B) u (A~s-no B))

< jj(AilB) + jj(A~s-no B) < 2e. Consequently, it holds that Ijj(A)

jj(B n s-noB)/ < 2e and

/jj(A) - jj(A)2/ ~ Ijj(A) - jj(B n s-n oB) / + /jj(B n s-n oB) - jj(A)2/

< 2e + /jj(B)2 - jj(A)21 = 2e + (jj(B) + jj(A)) /jj(B) - jj(A) / < 4e. Therefore jj( A) = jj( A) 2 , or jj(A) = 0 or 1, and jj is ergodic. Moreover, we can see that jj is strongly mixing. This fact and ergodicity of Markov sources will be discussed in Section 2.3.

Remark 3. Let us examine functional properties of P(X) and Ps(X). (1) Observe that M(X) = C(X)* (Riesz-Markov-Kakutani Theorem) by the identification M(X) 3 jj == All- E C(X)* given by

IE C(X) (cf. Dunford and Schwartz [1, IV.6]). Hence, P(X) is a bounded, closed and convex subset of M(X), where the norm in M(X) is the total variation norm "~II = I~I(X) for ~ E M(X). Moreover, it is weak* compact by the Banach-Alaoglu theorem (cf. Dunford and Schwartz [1, V.4]). Since B(X) contains C(X) as a closed subspace, C(X)* = M(X) can be embedded into B(X)*. For each jj E P(X) we have (infinitely many) Hahn-Banach extensions "1 of jj onto B(X), i.e., "1 E B(X)* and "1 = jj on C(X). Among these extensions "1 we can find a unique ji E B(X)* such that In .L f implies ji(ln) ji(/), where ji(f) = I dji for I E B(X). Hereafter, we shall write

+

p,(!)

Ix

=

L

f dp"

p, E M(X),

f

E

B(X).

2.2. Ergodic theorems

f

71

(2) Let us consider the measurable transforrnation S as an operator 8 on functions on X defined by

(8f)(x) = f(Sx),

x

E

X.

Then 8 is a linear operator on C(X), B(X) or LP(X, JL) for p ~ 1 and JL E P(X). For each n E N denote by 8 n the operator on functions f on X defined by

n-1

(SnJ)(x) =

n-1

.!:. ~)Sk J)(x) = .!:. L f(Sk x), n

k=O

n

X

E

X

(1.3)

k=O

Observe that the operator S : P(X) --t P(X) defined by S(JL) = JLoS-1 for JL E P(X) is affine. Suppose that S is continuous in the weak* topology on M(X), i.e., JLn --t JL (weak*) implies SJLn --t SJL (weak*). Then it follows from Kakutani-Markov fixed point theorem (cf. Dunford and Schwartz [1, V.I0]) that there is a JL E P(X) such that SJL = JL ° S-1 = JL, Le., JL E Ps(X). Hence, Ps(X) is nonempty and is also a norm closed and weak* compact convex subset of P(X). By Krein-Milman's theorem (cf. Dunford and Schwartz [1, V.8]) the set exPs(X) of all extreme points of Ps(X) is nonempty and Ps(X) = co [exPs(X)] , the closed convex hull of exPs(X). Here, the closure is w.r.t. the weak* topology and JL E Ps(X) is called an extreme point if JL = 0!'fJ + f3~ for some O!, f3 > 0 with O! + 13 = 1 and 'fJ, ~ E P; (X) imply that

JL

= 'fJ =~.

(3) The operator S on M(X) is continuous in the weak* topology if S is a continuous transformation on X. To see this, first we note that S is measurable. Let f E C(X). Then 8f E C(X) since 8f(·) = f(S·) and S is continuous. If C E X is compact, then there is a sequence {fn}~=1 ~ C(X) such that l-. t Ie as n --t 00 since X is compact and Hausdorff. Thus, le(S·) 81e(·) is Baire measurable, i.e., S-1C E X. Therefore, S is measurable. Now let JLn --t JL (weak*), i.e., JLn(f) --t JL(f) for f E C(X). Then, we have for f E C(X)

SJLn(f)

Ix = Ix =

Ix

f(x) SJLn(dx)

=

f(Sx) JLn(dx)

= JLn(SJ) --+ JL(SJ),

f(x) JLn(dS-1x)

since 8f E C(X), implying SJLn --t SJL (weak*). Therefore, S is continuous in the weak* topology.

2.2. Ergodic theorems Two celebrated Ergodic Theorems of Birkhoff and von Neumann will be stated and proved in this section. We begin with Birkhoff's ergodic theorem, where the operators 8 n 's are defined by (1.3).

Chapter II: Information Sources

72

Theorem 1 (Birkhoff Pointwise Ergodic Theorem). Let j-t E Ps(X) and E LI(X, j-t). Then there exists a unique fs E LI(X, j-t) such that (1) fs = lim Snf j-t-a.e.;

f

n-+oo

(2) Sis = Is u-a.e.; (3)

L L f dp. =

fs df-t for every S-invariant A E X;

(4) IISnl - Is II I,ll- -+ 0 as n -+ 00, II·"I,1l- being the norm in LI(X,j-t). II, in particular, j-t is ergodic, then Isis constant j-t-a. e. Proof. We only have to consider nonnegative I I(x) = lim sup (Snf)(x), n-+oo

E

LI(X, j-t). Let

f(x) = liminf (Snf)(x),

-

xEX.

n-+oo

To prove (1) it suffices to show that

since this implies that

I= I

p-a.e. and (1). Let M > 0 and e > 0 be fixed and

I M(X) = min {lex), M},

xEX.

Define n(x) to be the least integer n 2:: 1 such that n-I

1M(X) ~ (SnJ) (X) + e = !ti :E f(Si x ) + c,

xEX.

j=O

Note that n(x) is finite for each x E X. Since

1 and 1M are

S-invariant, we have

n(x)-I n(x)IM(x):s n(x) [(Sn(x)IM)(X) +e] =

:E

j(sjx) +n(x)e,

j=O

Choose a large enough N 2:: 1 such that

e j-t(A) < M Now we define

with

A = {x EX: n(x) > N}.

j and ii by

lex) =

{ I(x),

x~A

0,

XEA'

ii(x) = { n(x), 1,

x~A

xEA

x E X.

(2.1)

2.2. Ergodic theorems

73

Then we see that for all x E X

n(x)

~

N,

n(x)-l L IM(sjx) j=O

<

by definition,

n(x)-l L !(sjx) + n(x)e, j=O

(2.2)

by (2.1) and S-invariance of 1M' and that

f f dJ-L + f ! dJ-L t,f ! dJ-L = lAc lA

< f f dJ-L + f f dJ-L + f M dJ-L lAc lA i. =

L +l f dp,

Furthermore, find an integer L 2:: 1 so that for each x E X by

nO(x) = 0,

M dp,:5:

L

f dp, + c.

(2.3)

Nfl < e and define a sequence {nk(x)}k=O

nk(x) = nk-l(x) + n(snk(x)x) ,

k 2:: 1.

Then it holds that for x E X

k(x) nk(x)-l L 1M (sjx) = L L IM(sjx) j=O k=l j=nk-l (x)

£-1

£-1

+

L IM(sjx), j=nk(x) (x)

where k(x) is the largest integer k 2:: 1 such that nk(x) ~ L - 1. Applying (2.2) to each of the k(x) terms and estimating by M the last L - nk(x)(x) terrns, we have

k(x) nk(x)-l L 1M (sjx) = L L IM(sjx) j=O k=l j=nk-l (x)

£-1

£-1

+

L 1M(sjx) j=nk(x) (x)

k(x) [ nk(x)-l

]

:5: {; j=n~(x) j(Si x ) + (nk(x) - nk-l(X))c + (L- nk(x) (x))M £-1

~ L!(Sjx)+Le+(N-I)M

j=O since f 2:: 0, 1 M ~ M and L - nk(x)(x) ~ N - 1. If we integrate both sides on X and divide by L, then we get

74

Chapter II: Information Sources

by the S-invariance of f.-L, (2.3) and Nf < E; Thus, letting e -+ 0 and M -+ 00 give the inequality 1df.-L < f df.-L. The other inequality f df.-L < I df.-L can be obtained similarly. (2) is clear, (3) is easily verified, and (4) follows from the Dominated Convergence Theorem. Finally, if f.-L is ergodic, then for any r E lR the set [fs > r] is S-invariant (since fs is S-invariant) and has measure 0 or 1. That is, fs is constant f.-L-a.e.

Ix

Ix

Ix

Ix

Theorem 2 (von Neumann Mean Ergodic Theorem). Let f.-L E Ps(X) and f E L 2(X, f.-L). Then there exists a unique fs E L 2(X, f.-L) such that Sfs = fs u-ti.e. and IISnf - fsll2,JL -+ 0 (n -+ 00), where 1I·112,JL is the norm in L 2(X, f.-L). Proof. Suppose 9 E LOO(X, f.-L). Then, 9 E L 2(X , f.-L) ~ L 1(X, f.-L) and by Theorem 1 Sng -+ gs f.-L- a.e.

for some S-invariant 9s E Ll(X, f.-L). Clearly 9s E LOO(X, f.-L) C L 2(X, f.-L). Since ISn9 - gsl2 -+ 0 f.-L-a.e., it follows from the Bounded Convergence Theorem that

II Sn9 -

(2.4)

9s112,JL -+ 0 (n -+ 00).

Now let f E L 2(X,f.-L) be arbitrary. For any £ > 0 choose 9 E LOO(X,f.-L) such that Ilf - 9112,JL < £. By (2.4) we can find an no ~ 1 such that

n,m

~

no.

Since IISnfIl2,JL ::; IIfIl2,JL for n ~ 1 we see that

+ IISng - Smg1l2,JL + II S m9 9112,JL + e + 119 - f112,JL

II Snf - Smf1l2,JL ::; II Snf - Sng1l2,JL

Smf112,JL

::; Ilf <£+£+£=3£

for n, m ~ no. This means that {Snf}~=l is a Cauchy sequence in L 2(X, f.-L). Hence there is an !s E L 2(X, f.-L) such that II Sn! - !sI12,JL -+ O. To see the S-invariance of !s observe that IIfs - S!s1l2 JL = lim '

n-+oo

II n + 1 Sn+d n

= lim IIfIl2,JL = 0, n-+oo

n

S(SnJ) II 2,JL

2.3. Ergodic and mixing properties

75

implying that Is = SIs u-a.e. This completes the proof. Remark 3. (1) A sharper form of the Pointwise Ergodic Theorem is obtained for an arbitrary measure space (X, X, J-L) based on the Maximal Ergodic Theorem (see e.g. Rao [2]). (2) In the Pointwise Ergodic Theorem, let J be the o-subalgebra of X consisting of S-invariant sets. Then, Is = EI£(/IJ) J-L-a.e., where EI£(·IJ) is the conditional expectation w.r.t. J under the measure J-L. (3) In the Mean Ergodic Theorem, let S be the closed subspace of L 2(X, J-L) consisting of S-invariant functions and Ps : L 2(X, J-L) -+ S be the orthogonal projection. Then, Is = PsI for I E L 2 (X , J-L) (see (5) below). (4) It follows from the proof of the Mean Ergodic Theorem that the Mean Ergodic Theorem holds for every I E LP(X, J-L) with 1 ::; p < 00. That is, if 1 ::; p < 00, J-L E Ps(X) and I E LP(X, J-L), then there is some S-invariant Is E LP(X, JJ) such that IISnl - Isll p ,1£ -+ 0 as n -+ 00, 11·llp,1£ being the norm in LP(X, J-L). (5) The outline of von Neumann's original proof of Theorem 2 is as follows. Let S be as in (3) and 1-l = 6{1 - S/: IE L 2 (X , J-L )}, where 6{··· } is the closed subspace spanned by {... }. Then, the first step is to show that Sand 1-l are orthogonal cornplementary subspaces, i.e., Sffi1-l = L 2 (X , fL). The next step is to prove that Snl -+ 0 in L 2 for I E 1-l. Then, for any I E L 2(X, J-L) write I = 11 + 12 with 11 E Sand 12 E 1-l. Hence we have

IISnl - 11112,1£ = II Sn(/1+ 12) - 11112,1£ = IISnl2112 ,1£ -+ 0, as was desired. This tells us that Theorem 2 holds for an arbitrary measure space.

2.3. Ergodic and mixing properties Let X be a compact Hausdorff space and X be its Baire a-algebra. In this section, ergodicity and mixing properties are considered in some detail. After giving the following lemma we shall characterize ergodicity of stationary sources by using ergodic theorems. Recall that two measures J-L,11 E M(X) are said to be singular, denoted J-L 1- 11, if there is a set A E X such that IJ-LI(A) = IIJ-LII and 111I(AC) = 111111, i.e., J-L and 11 have disjoint supports. Also recall that J-L E Ps(X) is ergodic if each S-invariant set A E X has measure 0 or 1 and Pse(X) denotes the set of all stationary ergodic sources. Lemma 1. II J-L,11 E Pse(X), then either J-L = 11 or J-L 1- 11·

Chapter II: Information Sources

76

Proof. Suppose that J-t

=1=

'rI. Then there is an A

X such that J-t(A)

E

=1=

'rI(A). Let

Ap, = {x EX: lim (Sn 1A)(x) = J-t(A)} , n-+oo

A1] = {x EX: lim (Sn 1A)(x) = 'rI(A)}. n-+oo

Then we see that Ap, and A1] are S-invariant, Ap, n A1] = 0 and J-t(Ap,) by Theorem 2.1 since J-t and 'rI are ergodic. This implies that J-t 1- 'rI.

= 'rI(A1]) = 1

Theorem 2. For a stationary source J-t E Ps(X) the following conditions are equivalent to each other: (1) J-t E Pse(X), i.e., J-t is ergodic. (2) There is some 'rI E Pse(X) such that J-t ~ n, (3) If ~ E Ps(X) and ~ ~ u, then ~ = u, (4) J-t E exPs(X). (5) If f E B(X) is S-invariant J-t-a.e., then f = const J-t-a.e.

(6) fs(x) == lim (Snf)(x) = ( fdJ-t u-a.e. for every f E L1(X,J-t).

Jx

n-+oo

(7) n-+oo lim (Snf,g)2 ,p, = (f, 1)2 , p,(I,g)2, p, for every

i.e

E L 2(X, J-t).

(8) lim J-t((Snf)g) = J-t(f)J-t(g) for every f,g

E

B(X).

(9) lim J-t((Snf)g) = J-t(f)J-t(g) for every f,g

E

C(X).

n-+oo n-+oo

1

n-l

(10) lim n-+oo n

k=O

1

n-l

(11) lim n-+oo n

L: J-t(S-kA n B) L: J-t(S-kA n A)

= J-t(A)J-t(B) for every A, B E X. =

J-t(A)2 for every A

E X.

k=O

Proof. (1) {:} (2) is obvious and (1), (2) =} (3) follows from Lemma 1. (3) =} (4). Suppose (4) is false, i.e., J-t ¢ exPs(X). Then there are 0'., {3 > 0 with 0'. + {3 = 1 and ~,'rI E Ps(X) with ~ =1= 'rI such that J-t = O'.~ + {3'r1. Hence ~ =1= J-t and ~ ~ J-t, i.e., (3) does not hold. (4) =} (1). Assume that (1) is false, i.e., J-t is not ergodic. Then there is an Sinvariant set A E X for which 0 < J-t(A) < 1. Hence J-t can be written as a nontrivial convex combination J-t(.) = J-t(A)J-t(·IA) + J-t(AC)J-t(·IA C), where J-t(·IA) J-t(·IAC) and J-t(·IA), J-t(·IAC) E Ps(X). This rneans that J-t ¢ exPs(X), i.e., (4) is not true. (1) =} (5). Let f E B(X) be real valued and S-invariant and let

AT = {x

EX:

f (x) > r},

r

E JR.

2.3. Ergodic and mixing properties

77

Then A r E X is S-invariant and hence J-l(A r ) = 0 or 1 for every r E lR by (1). This means I = const J-l-a.e. (5) => (6). Let I E Ll(X,J-l). Then, Is is measurable and S-invariant J-l-a.e. by Theorem 2.1. By (5) Is = const J-l-a.e. Hence, Is = Is dJ-l I dJ-l J-l-a.e. 2(X, (6) => (7). Let I, 9 E L J-l). Then, by (6), Is = I dJ-l J-l-a.e. and the Mean Ergodic Theorem implies

Ix Ix

=Ix

(7) => (8) => (9) are obvious since C(X) ~ B(X) ~ L 2(X, J-l) and (9) => (7) can be verified by a simple approximation argument since C(X) is dense in L 2(X, J-l). (8) => (10). Take I = lA and 9 = IB in (8). (10) => (11) is obvious. (11) => (1). Let A E X be S-invariant. Then (11) implies that J-l(A) = J-l(A) 2 , so that J-l(A) = 0 or 1. Hence (1) holds.

Remark 3. (1) Recall that a semialgebra of subsets of X is a set Xo such that (i) 0 E Xo ; (ii) A, B E X o => An B E X o; n (iii) A E X o => AC = .U B j with disjoint B l, . . . ,Bn E X O• J=l

As we have seen in Section 2.1, in an alphabet message space X~, the set VJt of all messages is a semialgebra. Another such example is the set X x ~ of all rectangles, where (Y,~) is another measurable space. (2) Let J-l E P(X) and X o be a semialgebra generating X, i.e., u(Xo) = X. If J-l is S-invariant on Xo, i.e., J-l(S-lA) = J-l(A) for A E X o, then J-l E Ps(X). In fact, let

Then, clearly Xo ~ Xl' It is not hard to see that each set in the algebra A(Xo) generated by X o isa finite disjoint union of sets in X o. Hence A(Xo) ~ Xl. Also it is not hard to see that Xl is a monotone class, i.e., {An}~=l ~ Xl and An t (or An L) imply

U An

n=l

E Xl (or

n An E Xl)'

n=l

Since the o-algebra generated by A(Xo) is

the monotone class generated by A(Xo), we have that X = u(A(Xo)) = Xl' Thus J-l E Ps(X). (3) In view of (2) above, we can replace X in conditions (10) and (11) of Theorem 2 by a semialgebra X o generating X. In fact, suppose that the equality in (10) of

Chapter II: Information Sources

78

Theorem 2 holds for A, B E Xo. Then it also holds for A, B E A(Xo) since each A E A(Xo) can be written as a finite disjoint union of some AI, ... ,An E X o. Now let e > 0 and A, B E X, and choose A o, B o E A(Xo) such that J.t(A~Ao) < e and J.t(B~Bo) < E; Note that for j 2:: 0

(S-j A n B)~(S-j A o n B o) ~ (S-j A~S-jA o) U

(B~Bo)

(S-j(A~Ao)) U (B~Bo)

and hence

since J.t is S-invariant. This implies that j

2::

o.

(3.1)

Moreover, we have that

1J.t(S-jA n B) - J.t(A)J.t(B)

I

::; 1J.t(S-jA n B) - J.t(S-j A o n B o)I + 1J.t(S-jA o n B o) - J.t(Ao)J.t(Bo)I

+ 1J.t(Ao)J.t(Bo) - J.t(A)J.t(Bo)I + 1J.t(A)J.t(Bo) < 4e + 1J.t(S-j A o n B o) - J.t(Ao)J.t(Bo)I,

J.t(A)J.t(B) I (3.2)

which is irrelevant for ergodicity but for mixing properties in Theorem 6 and Remark 11 below. Consequently by (3.1) it holds that 1 1n

I: j=O

/-£(S-j A n B)

/-£(A)/-£(B)/

< /~ ~ {/-£(S-jA n B) -/-£(S-jAo n Bo)}! +

I!I: n

/-£(S-j Ao n Bo)

+ 1J.t(Ao)J.t(Bo) < 4e +

I

/-£(Ao)/-£(Bo)

j=O

J.t(A)J.t(B) I

I~ ~/-£(S-jAo n Bo) -/-£(Ao)/-£(Bo)l,

where the second term on the RHS can be made < e for large enough n. This means that (10) of Theorem 2 holds. (4) Condition (11) of Theorem 2 suggests that the following conditions are equivalent to anyone of (1) - (11) of Theorem 2:

2.3. Ergodic and mixing properties

(7') nl~ (Snf, fh,1-' =

79

IU, Ih'1-'1

= J-L(f)2 (9') lim 1£( (Snf)f) = J-L(f)2 n-+oo (8') lim 1£( (Snf)f) n-+oo

2

for every f E L 2(X,p,);

for every f E B(X); for every f E C(X).

(5) J denotes the er-subalgebra consisting of S-invariant sets in X, i.e., J = {A X: S-lA = A}. For 1£ E P(X) let

E

the set of all J-L-a.e. S-invariant or S-invariant (mod 1£) sets in X. Clearly J ~ J J.L. Then, we can show that 1£ E Ps(X) is ergodic iff J-L(A) = 0 or 1 for every A E JJ.L. In fact, the "if' part is obvious. To prove the "only if' part, let A EaJ.L. First we note that n 2:: o. For, if n 2:: 1, then

s:»A~A ~

n-1

n-1

j=O

j=O

U (S-(j+1) A~S-j A) = Us! is:' A~A)

and hence

Now let A oo

00

= n

00

.U

n=O J=n

.

S-J A = lim sup S"?' A. Then we also note that J-L(A oo ) = J-L(A) n-+oo

since

p,(Aoo~A) = p,( C~OjQn s-j A )~A ) < p,( CQn s~j A)~A) 00

~

E J-L(S-j A~A) = o. j=n

Finally, we note that S-l A oo = A oo , since

n Us-j A = n=Oj=n n US-(j+1)A = n=Oj=n+1 n U s-j A = A n=Oj=n 00

s:' A oo = s:'

00

00

00

00

It follows from ergodicity of 1£ that J-L(A oo ) = J-L(A) = 0 or 1.

00

oo .

Chapter II: Information Sources

80

Example 4. Consider an (M,m)-Markov source J.L, where M = (mij) is an f x f stochastic rnatrix and m is a row probability vector such that m = mM with tti; > 0, 1 ~ i ~ f. If we write Mk = (m~;)) for k 2 1, then this gives the k-step transition probabilities, i.e., 1

< i,j < f.

M or J.L is said to be irreducible if for each i, j there is some k 2 1 such that rn~;) We first claim that

> o.

n-l

N

exists and N

=

=

lim.!.

n-+oo

n

LMk

k=O

(nij) is a stochastic matrix such that

NM=MN=N=N 2 •

In fact, let Ai = [xo = ail, 1 ~ i ~ f and apply the Pointwise Ergodic Theorem to f = 1Ai· Then we have that

exists u-a.e.x and

~ rfs(x)lA-(x)/-L(dx)=~ lim .!.~/-L(S-kAinAj) m; } x m; n-+oo n k=O 3

n-l

= n-+oo lim .!. " m~k) = nij n 'L....J '£)

(3.3)

k=O

for 1 ~ i, j ~ f. Thus N = (nij) is well-defined. The other properties of N are clear. Under this preparation we can characterize ergodicity of Markov sources. For an (M, m)-Markov source J.L the following conditions are equivalent: (1) J.L is ergodic. (2) nij = rnj for every i, j. (3) nij > 0 for every i, j. (4) J.L is irreducible. (1) => (2). By (3.3) we see that for each i, j

81

2.3. Ergodic and mixing properties

while the ergodicity of J.L implies the RHS is also equal to mimj' Hence nij =mj' (2) => (3) is clear since we are assuming m; > 0 for every i. 1 n-l (3) => (4). For any i,j, lim - E m~7) = nij > O. This irnplies that we can find n-too

n k=O

some k ~ 1 such that m~7) > O. That is, J.L is irreducible. It is not hard to show the implications (4) => (3) => (2) => (1) and we leave it to the reader. We now consider mixing properties for stationary sources which are stronger than ergodicity.

Definition 5. A stationary source J.L E Ps(X) is said to be strongly mixing (SM) if A,BEX, and to be weakly mixing (WM) if

A,BEX.

It follows from the definition and Theorem 2 that strong mixing => weak mixing => ergodicity. First we characterize strong mixing.

Theorem 6. For a stationary source J.LE Ps(X) the following conditions are equivalent to each other: (1) J.L is strongly mixing. (2) nl~~(snl,g)2'11 = (I, 1)2,11(1,g)2,11 for every I,g E L 2(X,J.L). That is, snl-+

L

f dp, weakly in L 2(X, p,) for every f E L 2(X, p,).

(3) nl~~(snf,fh,,,=

IU,lh,,,1 2

for every fEL 2(X,p,).

(4) lim J.L(s-nA n A) = J.L(A)2 for every AE X. n-too

(5) lim J.L(s-nA n-too

n A)

= J.L(A)2 for every A E :to, a generating semialgebra.

Proof. (2) => (1) is seen by considering 1 = 1A and 9 = lB. (2) => (3) => (4) => (5) is clear. (5) => (4) follows from (3.1) and (3.2) with A = Band A o = B«.

Chapter II: Information Sources

82

(1) => (2). Let A, B E

.x.

Then by (1) we have

lim (sn1A, 1B)2 p, = lim /-L(s-n A n B) = /-L(A)/-L(B) = (lA,

n-+oo

'

n-+oo

/,

If I =

I:

1)2 ' p,(1, 1n)2,p,.

m

aj

1Aj and g =

j=l

I:

{3k1Bk' simple functions, then

k=l

=

Ej,k aj~k(lAj' 1)2,p,(1, 1Bk)2,JL

Hence the equality in (2) is true for all simple functions I, g. Now let I, g E L 2(X, /-L) and E > o. Choose simple functions 10 and go such that III - loll2,JL < e and Ilg - goI12,p, < e. Also choose an integer no ~ 1 such that

n Then we see that for n

I(sn l,g)2,JL

~

~

no.

no

(/,1)2,JL(1,g)2,p,1

::; I(snl,g)2,p, - (snI0,g)2,p,1 + l(snlo,g)2,p, + I(sn 10,gO)2,p, - (/0, 1)2,p,(1, gO)2,JLI + 1(/0, 1)2,p,(1, gO)2,p, - (I, 1)2,p,(1, gO)2,p, I

(snI0,gO)2,p,1

+ I(I, 1)2,p,(1, gO)2,p, - (I, 1)2,JL(1, g)2,p,1 ::; l(sn(1 - 10),g)2,p,1 + I(snlo,g gO)2,p,1 +e + 1(1 - 10, 1)2,p,11(1,go)2,p,1 + 1(/, 1)2,p,11(1,g - gO)2,p,1 ::; III - 10112,p,lIgI12,JL + 1I/0112,p,lIg - goll2,JL + E + III - loll2,p,llgoll2,p, + 11/112,p,llg - goll2,JL ::; ellgI12,p, + II/0112,p,e + e + ellgoI12,p, + 11/1I2,p,e ::; ellgI12,p, + (11/112,P, + e)e + E + e(llgI12,JL + c) + ell/ll2,p,. It follows that nli~ (sn I,

g)2,p, = (I, 1)2,p,(1, g)2,JL.

(4) => (3) is derived by a similar argument as in the proof of (1) => (2) above.

2.3. Ergodic and mixing properties

(3) => (2). Take any

f

E

83

L 2(X, JL) and let

1-£ = 6 [S" t, c : c E C, n 2: O}, the closed subspace of L 2(X, JL) generated by the constant functions and S" t, n 2: Now consider the set

o.

Clearly 1-£1 is a closed subspace of L 2(X, JL) which contains f and constant functions, and is S-invariant. Hence 1-£1 contains 1-£. To see that 1-£1 = L 2(X, JL) let g E 1-£.1., the orthogonal complement of 1-£ in L 2(X, JL). Then we have

so that g E 1-£1. Thus 1-£.1. ~ 1-£1. Therefore 1-£1

= L 2(X, JL), i.e.,

(2) holds.

In Theorem 6 (2) and (3), L 2(X, JL) can be replaced by B(X) or C(X).

Example 7. Every Bernoulli source is strongly mixing. To see this let JL be a (PI, ... ,Pt)- Bernoulli source on X = X~. Let A = [x?· .. xJ], B = [y~ . . . y~] E VR. Then it is clear that since for a large enough n 2: 1 we have n+i

> t. By Theorem 6 JL is strongly mixing.

In order to characterize weak mixing we need the following definition and lernma.

Definition 8. A subset J ~ Z+ == {O, 1,2, ... } is said to be of density zero if lim

n.-+oo

where I n

= {O, 1,2, ... ,n -

.!.IJ n Jnl = n

I} (n 2: 1) and

Lemma 9. For a bounded sequence

0,

IJn Jnl is the cardinality of J n I n .

{an}~=l of real nurnbers the following conditions

are equivalent: 1 n-l (1) lim - L: lajl = 0; n.-+oo

n

j=O

1 n-l (2) lim - L: lajl2 = 0; n.-+oo

n

j=O

(3) There is a set J ~

z+

of density zero such that

lim

JtJ n.-+oo

an =

o.

Chapter II: Information Sources

84

Proof. If we can show (1)

(3), then (2)

¢}

lim an J'tJ n-+-oo

¢}

= 0 ¢}

(3) is easily verified by noting that

lim a~ J"tJ n-+-oo

= O.

~},

k

So we prove (1) ¢} (3). (1) =} (3). Suppose (1) is true and let

Ek Observe that E 1

~

E2

{n

=

~ •..

E

Z+ : lanl :?:

~

1.

and each Ek has density zero since

as n -+ 00 by (1). Hence for each k = 1,2, ... we can find an integer jk > 0 such that 1 = jo < i, < j2 < ... and (3.4) Now we set J =

jk-l

~

n < jk, then

U (Ek n [jk-l,jk)).

k=l

In I n

We first show that J has density zero. If

[In [O,jk-l)] U [In [jk-l,n)] < [Ek n [O,jk-l)] U [Ek+1 n [O,n)]

=

and hence by (3.4)

This implies that ~IJ n Jnl -+ 0 as n -+ Secondly, we show that lim an = J"tJ n-+-oo lanl < k~l. This gives the conclusion.

00,

o.

i.e., J has density zero. If n > jk and n tf. J, then n

tf. E k

and

2.3. Ergodic and mixing properties

85

(3) ::::} (1). Suppose the existence of a set J ~ Z+ and let e

> O. Then,

Since {an} is bounded and J has density zero, the first term can be made < E for large enough n. Since an -+ 0 as n -+ 00 and n t/. J, the second term can also be made < e for large enough n. Therefore (1) holds. Theorem 10. For a stationary source J-t E Ps(X) the following conditions are equivalent to each other: (1) J-t is weakly mixing. (2) For any A, B E X there is a set J ~ z+ of density zero such that

1 n-l 2 (3) lim - I: 1J-t(S-jA n B) - J-t(A)J-t(B) I = 0 for every A, B E X. n-+oo n j=O 1 n-l (4) lim - I: 1(Sk t, g)2 /1- - (f, 1)2 /1-(1, g)2 /1-1 = 0 for every t, g E L 2(X, J-t). n-+oo n k=O

'

,

,

(5) J-t x J-t is weakly mixing relative to S x S, where J-t x J-t is the product measure on (X x X,XQ9X). (6) J-t x rj is ergodic relative to S x T, where (Y,~, n, T) is an ergodic dynarnical system, i. e., rj E P s e (Y) . (7) J-t x J-t is ergodic relative to S x S. Remark 11. In Theorem 10 above, conditions (2), (3) and (4) may be replaced by (2'),(2"),(3'),(3") and (4'),(4") below, respectively, where X o is a semialgebra generating X: (2') For any A, B E Xo there exists a set J ~ Z+ of density zero such that lim J-t(s-n A n B) = J-t(A)J-t(B). J1:J n-+oo

(2") For any A E X o there exists a set J ~ Z+ of density zero such that lim J-t(s-n A n A) = J-t(A)2.

J1:J n-+oo

I:

1 2 (3') lim ..!:. IJL(S-iA n A) - JL(A)21 = 0 for every A E Xo. n-+oo n j=O 1 n-l . (3") lim - I: 1J-t(S-J A n A) - J-t(A)21 = 0 for every A E X o. n-+oo n j=O

Chapter II: Information Sources

86

Proof of Theorem 10. (1)

¢:>

(2)

¢:>

(3) follows from Lemma 9 with

(1) :::} (4) can be verified first for simple functions and then for L 2 -funct ions by a suitable approximation as in the proof of (1) :::} (2) of Theorem 6. (4):::} (1) is trivial. (2) :::} (5). Let A, B, C, D E X and choose J1, J 2 ~ Z+ of density zero such that

It follows that

(/-L x /-L)((S x s)-n(A x C)

lim JIUJ2f;) n-+oo

==

lim

n (B

x D))

/-L(s-n A n B)/-L(s-nc n D)

J1UJ2f;) n-+oo

== /-L(A)/-L(B)/-L(C)/-L(D) == (/-L x /-L)(A x C)(/-L x /-L)(B x D). Since X x X == {A x B : A, B E X} is a semialgebra and generates X ® X, and J 1 U J 2 ~ Z+ is of density zero, we invoke Lemma 9 and Remark 11 to see that /-L x /-L is weakly mixing. (5) :::} (6). Suppose /-Lx/-L is weakly mixing and (Y,~, T, 'T/) is an ergodic dynarnical system. First we note that (5) implies (2) and hence /-L itself is weakly mixing. Let A,B E X and C,D E~. Then n-1

.!:. :L(/t x 1])(S x T)-i(A x C) n (B x D)) n

j=O

1 n-1

== n

.

:L /-L(A)/-L(B)'T/(T-JC n D) j=O

n-1

+ .!:. :L {/t(S-i A n B) n

j=O

/t(A)/t(B) }7J(T-iC n D).

(3.5)

87

2.3. Ergodic and mixing properties

The first term on the RHS of (3.5) converges to

since

'f}

is ergodic. The second term on the RHS of (3.5) tends to 0 (n -+ (0) since I

~ ~ {JL(S-j A n B) n-l

S;

.!- E n

JL(A)JL(B)}1J(T-

jC

n D)!

IJL(S-j A n B) - JL(A)JL(B) I ~ 0 (n

~ (0),

j=O

since J-L is weakly mixing. Thus J.L x 'f} is ergodic since X x ~ is a semialgebra generating X®~.

(6) =} (7) is trivial. (7)

(3). Let A, B E X and observe that

=}

1

n-l.

n

j=O

1

n-l

.

- E JL(S-J A n B) = ;;: E(JL x JL)((S x S)-J(A x X) n (B x X)) j=O

-+ (J.L x J.L)(A x X)(J.L x J.L)(B x X),

by (7),

= J-L(A)J-L(B), n-l

n-l

.!- E JL(S-jAn B)2 = .!- E(JL x JL)((S x S)-j(A x A) n (B n B)) n

n j=O -+ (J-L x J.L)(A x A)(J.L x J-L)(B x B),

j=O

by (7),

2

= J-L(A) J-L(B)2.

Combining these two, we get n-l

.!- EIJL(S-j A n B) n

2

JL(A)JL(B) 1

j=O

n-l

=

.!- E {JL(S-j An B)2 n

2JL(S-j An B)JL(A)JL(B) + JL(A)2JL(B)2}

j=O

-+ J-L(A)2J.L(B)2

2J.L(A)J.L(B)J-L(A)J.L(B) + J.L(A)2J.L(B)2 =

o.

Thus (3) holds. Example 12. Let J-L be an (M,m)-Markov source on X = X~. J-L or M is said to be aperiodic if there is some n-o ~ 1 such that Mno has no zero entries. Then, the following statements are equivalent:

Chapter II: Information Sources

88

(1) J-L is strongly mixing. (2) J-L is weakly mixing. (3) J-L is irreducible and aperiodic. · m (n) = mj £or every 2,. J.. (4) 1rm n-+oo

ij

The proof may be found in Walters [1, p. 51]. In the rest of this section we use relative entropy to characterize ergodicity and mixing properties. Recall that for a pair of information sources J-L and v the relative entropy H(vlJ-L) of v w.r.t. 1£ was obtained by

~ v(A) log :~~~ : 2l E P(X) }

H(vIJL) = sup { = {

i

log

~: du,

if v

«

JL,

otherwise

00,

(cf. (1.6.1) and Theorem 1.6.2). The following lemma is necessary.

Lemma 13. Let J-Ln (n ~ 1), J-L E P(X). Suppose J.Ln :::; aJ.L for n ~ 1, where a > 0 is a constant. Then, lim J.Ln(A) = J.L(A) uniformly in A E X iff lim H(J-LnIJ.L) = o. n-+oo

n-+oo

Proof. The "if" part follows from Theorem 1.6.3 (4). To see the "only if" part, observe that {~} is uniformly bounded and converges to 1 in probability (w.r.t. 1£) by assumption. Since

t

~

0,

we have {~log ~} converges to 0 in probability. Thus, since {~log ~ } is uniformly bounded, we also have

lim H(JLnIJL) n-+oo

= lim n-+oo

r ddJ.Ln log ddJ.LnJ-L dJ.L

Jx

1£

=

We introduce some notations. Let J-L E P(X). For each n Mn on X Q9 X by 1

Mn(A x B) = n

n-l

L

j=O

o. ~

1 define a rneasure

.

J.L(8- J A n B),

A,BEX.

2.3. Ergodic and mixing properties

89

For a finite partition 2l E P(X) of X, J-L2{ denotes the restriction of J-L to 21 = o-(2l), i.e., J-LS)l == J-Ll m. For 2l, Q) E P(X) denote by A(2l x Q)) the algebra generated by the set {A x B : A E 2l, B E ~} of rectangles. We also let

- (

Il.; !!, Q3

)

=

,,_ ( ) iln(A x B) L..J {tn A x B log (A) (B)' AE2{,BE~ J-L J-L

Hjin(2l x~) = -

E

iln(A x B)logiln(A x B),

AE2{,BE~

HJ1-xJ1-(2l x~) = -

E

J-L(A)logJ-L(A) AES)l = HJ1-(2l) + HJ1-(Q)) , say.

E J-L(B)logJ-L(B) BE~

Now we have the following.

Proposition 14. For a stationary source p E Ps(X) the following statements are equivalent to each other: (1) J-L is ergodic. (2) lim Hn(2l,~) = 0 for every 2l, Q) E P(X). n-+oo

(3) lim Hjin (2l x Q)) = HJ1-xJ1-(2l x ~) for every 2l, ~ E P(X). n-+oo

Proof. (1) {:} (2). By Theorem 2 J-L is ergodic iff

lim iln(A x B)

n-+oo

= J-L(A)J-L(B),

A E 2l, B E

Q),

2l, Q) E P(X).

If we fix 2l, ~ E P(X), then the convergence above is uniform on A(2l x Q)) and

where il~'~ = ilnIA(2{X~), the restriction of iln to A(2l x ~). Thus by Lemma 13 (1) {:} (2) holds. (2) {:} (3) is clear since for n 2: 1 and 2l, Q) E P(X)

To characterize rnixing properties we let for n 2: 1 A,BEX,

90

Chapter II: Information Sources

Hn(2!,~) = H JLn(21

L

JLn(A

AE~,BE~

x~) = -

XB) log JL(~ X(:!, {L

L

{Ln(A X

21, ~ E P(X),

{L

B) log {Ln(A

X

B),

21, ~

E

P(X).

AE~,BE~

Then the following proposition is derived from Proposition 14 by replacing fIn and Hjln by u; and H JLn, respectively.

Proposition 15. For a stationary source J.L E Ps(X) the following statements are equivalent to each other: (1) J.L is strongly mixing. (2) lim Hn(2t,~) = 0 for every 21, ~ E P(X). n-+oo

(3) lim

n-+oo

H JLn (2t X ~) = H JL xl-£(2t X ~)

for every 21, ~ E P(X).

Finally, weak mixing is characterized as follows:

Proposition 16. For a stationary source J.L E Ps(X) the following statements are equivalent to each other: (1) J.L is weakly mixing. 1

(2) lim

n-l

I:

n-+oo n j=O

1

(3) lim n-+oo

n

Hj(2t,~) =

n-l

I:

HJLj

j=O

(21 x

0 for every 21, ~

~) = HJLxJL(21,~)

Proof. (1) ¢:> (2). For any 21, ~ E P(X) and n 1 2a

1

P(X).

E

for every 21, ~ E P(X). ~

1 it holds that

1 n-l.

2

L (A) J.L (B)·;:; L IJL(S-J A n B) - JL(A)JL(B) I AE~,BE~ J.L j=O

JL(A)JL(B)¥=O

1

<-

n-l

LHj(21

x~)

n j=O

<

L AE~,BE~

1

1

(A) (B)· n J.L J.L

n-l

.

L IJL(S-J A n B) - JL(A)JL(B)

j=O

JL(A)I-£(B)¥=O

where a

= max {JL(~) : B 1

E ~,J.L(B)

(t - 1) + 2a (t - 1) 2

:::; t

=f. O}

since

log t :::; It - 11

1

+ 2" (t -

1) 2 ,

tE[O,I].

2 1

,

2.4. AMS sources

91

This is enough to show the equivalence (1) {:} (2). (2) {:} (3) follows from

for n 2: 1 and

2(., ~ E

P(X).

The results obtained in this section will be applied to consider ergodic and mixing properties of stationary and AMS channels in Chapter 3.

2.4. AMS sources Let X be a compact Hausdorff space and X be the Baire a-algebra of X as before. S denotes a measurable transformation on X. Of interest is a class of nonstationary sources for which the Ergodic Theorem holds. Each source in this class is said to be asymptotically mean stationary. Here is a precise definition. Definition 1. A source J.L E P(X) is said to be . ~tJ!!!jr!1£~c:YJL!!~~!!!l1!2!1!1!~~ if, . for each A E X, 1 lim -

n-+oo

n

n-l

:E J.L(S-k A) == ji(A)

(4.1)

k=O

exists. [In this case, ji is a probability measure, i.e., an information source, by the Vitali-Hahn-Saks Theorem (cf. Dunford and Schwartz [1, 111.7]). Moreover, ji E P; (X).] ji is called the stationary mean of J.L. Pa (X) denotes the set of all AMS sources in P(X). Remark 2. (1) If J.L E Pa(X) with the stationary mean ji E Ps(X), then (i) J.L(A) = ji(A) for all S-invariant A E X, (ii) J.L(f) = fl(f) for all S-invariant f E B(X). (2) If, for J.L E P(X), the limit in (4.1) exists for all A E X o, the generator of X, J.L need not be AMS. Lemma 3. (1) Pa(X) is a norm closed convex subset of P(X).

(2) If J.L, rJ E Pa(X), then

liji -1711

~

IIJ.L - rJll·

Proof. (1) Clearly Pa(X) is convex. To see that Pa(X} is norm closed let {J.Ln}~=l ~ Pa(X) be a Cauchy sequence. Then, since P(X) is norm closed there exists a

Chapter II: Information Sources

92

J.l E P(X) such that IIJ.ln - J.lII -t 0 as n -t 00. Hence, lJ.ln(A) - J.l(A) I -t 0 uniformly in A E X since IJ.ln (A) - J.l( A) I == I(J.ln - J.l) (A) I ::; "J.ln - J.l11· That is, for any e

> 0 there is an integer no 2 1 such that n 2 no, A E X.

Thus for A E X and P, q 2 1 we have that

Since J.lno is AMS we can choose an integer Po 2 1 such that the third term of the RHS of the above expression can be made < £ for p, q 2 Po. Consequently it follows that for p, q 2 Po, the LHS of the above expression is < 3£. Therefore, the limit in (4.1) exists for every A E X, so that J.l E Pa(X). (2) Let us set

Ma(X) == {aJ.l + f3rJ : a, f3

E

C, u; "l E Pa(X)},

Ms(X) == {aJ.l + (3"l : a, f3

E

C, u, "l E Ps(X)}.

Note that Ma(X) is the set of all measures J.l E M(X) for which the limit in (4.1) exists, and Ms(X) is the set of all S-invariant J.l E M(X). Define an operator 'I'o : Pa(X) -+ Ps(X) by 'I'oJ.l == ii, and extend it linearly to an operator 'I' on Ma(X) onto Ms(X). Then we see that 'I' is a bounded linear operator of norm 1, since Ma(X) is a norrn closed subspace of M(X) by (1). Hence (2) follows irnmediately.

93

2.4. AMS sources

Definition 4. Let J.L, 'TJ E P(X). That n asymptotically dominates J.L, denoted J.L ~ 'TJ, means that 'TJ(A) = 0 implies lim J.L(s-nA) = o. n-"7OO

The usual dominance implies the asymptotic dominance in the sense that, if J.L E P(X), 'TJ E Ps(X) and J.L ~ rJ, then J.L ~ rJ. In fact, if rJ(A) = 0, then rJ(s-n A) = rJ(A) = 0, which implies that J.L(s-nA) = 0 by J.L ~ rJ for every n E N. Thus lim J.L(s-nA) = 0, implying J.L ~ rJ. Although ~ is not transitive, one has that if n-"7OO a a a J.L« ~«rJ or J.L~~ ~ rJ, then J.L~'TJ. After the next lemma, we characterize AMS sources.

Lemma 5. Let J.L,'TJ E P(X) and for n 2: 0 let SnJ.L = (SnJ.L)a + (SnJ.L)s be the Lebesgue decomposition of SnJ.L w.r.t. n, where (SnJ.L)a is the absolutely continuous part and (snp,)s is the singular part. If In = d(S;:)a is the Radon-Nikodsim a

derivative and J.L« 'TJ, then it holds that lim { sup.IJ.L(s-n A) n-"7OO

AEX

{ fn

JA

drJl} = o.

(4.2)

Proof. For each n = 0,1,2, ... let B n E X be such that rJ(Bn) = 0 and AEX.

Let B

= n=O UBn .

Then we see that rJ(B) = 0 and for any A E X

o < p,(s-n A) -

L

fn d17 == p,(s-n(A n B n ))

::; J.L(s-n(A n B)) as n -+

ex)

< J.L(s-nB) -+ 0

by J.L~rJ. Taking the supremum over A E X, we have (4.2).

Theorem 6. For J.L E P(X), the following conditions are equivalent: (1) J.L E Pa(X), i.e., J.L is AMB. (2) There exists some rJ E Ps(X) such that J.L ~ rl·

(3) There exists some rJ E Ps(X) such that rl(A) =0 and A E Xoo ==

en

n=O

s-nx

imply J.L(A) = O. (4) There exists some rJ E Ps(X) such that rJ(A) = 0 and S-lA = A imply J.L(A) = O.

Chapter II: Information Sources

94

(5) lim Snf = fs J-t-a.e. for f E B(X), where fs is an S-invariant function. n-+oo (6) lim J-t(Snf) exists for f E B(X). n-+oo If one (and hence all) of the above conditions holds, then the stationary mean ji of J-t satisfies: (4.3) f E B(X). ji(f) = lim J-t(Snf) = J-t(fs), n-+oo

Proof. (1) =} (2). Suppose (1) is true and let rJ = ji E Ps(X). Assume ji(A) = 0 and let B = lim sup s:» A = S-k A. Then n-+oo n=l k=n

nu

ji(B) = lim n-+oo

ii( k=n U S-k A) ~ ii( k= U S-k A) 1

00

::; L ii(S-k A) = 0 k=l since ji(S-k A) = ji(A) = 0 for every k 2:: 1. This implies J-t(B) S-invariant. Now we see that lim sup JL(s-n A) ::; JL( lim sup s:»A) n-+oo n-+oo

= 0 since B

is clearly

= JL(B) = 0

by Fatou's lemma. Thus lim J-t(s-nA) = 0, and therefore (2) holds. n-+oo (2) =} (3). Assume that rJ E Ps(X) satisfies J-t~rJ. Take any A E oXoo such that rJ(A) = 0 and find a sequence {An}~l ~ oX with S':" An = A for n 2:: 1. It then follows from Lemma 5 that

where

In =

d(S;:;)a. Since rJ is stationary and '11(A)

= 0, we have

so that J-t(A) = O. Thus (3) holds. (3) =} (4) is immediate since A E oXoo if A E oX is S~invariant. (4) =} (2). Let rJ E Ps(X) satisfy the condition in (4). Take an A E oX with rJ(A) = 0 and let B = lim sup S':" A. Then we see that B is S-invariant and rJ(B) = n-+oo O. That lim J-t(s-nA) = 0 can be shown in the same fashion as in the proof of n-+oo (1) =} (2) above.

95

2.4. AMS sources

(2) => (5). Suppose that J.L ~ 'TJ with r/ E Ps(X), and let f E B(X) be arbitrary. Then the set A == {x EX: (Snf)(x) converges} is S-invariant and 'TJ(A) == 1 by the Pointwise Ergodic Theorern. Thus, that AC is S-invariant and 'TJ(AC) == 0 imply lim J.L(s-nAC) == J.L(AC) == 0 by J.L ~ 'TJ. Consequently, J.L(A) == 1, and lim Snf == fs n-+oo exists and is S-invariant u-a.e. (5) => (6). Let f E B(X) and observe that {Snf}~=l is a bounded sequence in B(X) ~ L 1(X, J.L) such that Snf -+ fs u-a.e. by (5). Then the Bounded Convergence Theorem implies that J.L(Snf) -+ J.L(fs). (6) => (1). We only have to take f == 1A in (6). The equality (4.3) is almost clear. n-+oo

a

Remark 7. (1) Note that J.L« Ii holds for J.L E Pa(X). This follows from the proof of (1) => (2) in the above theorem. (2) If J.L E P(X) and J.L «: 'TJ for some 'TJ E Ps(X), then J.L is AMS. (3) The Pointwise Ergodic Theorem holds for J.L E P(X) iff J.L E Pa(X). More precisely, the following staternents are equivalent for J.L E P(X): (i) J.L E Pa(X). (ii) For any f E B(X) there exists some S-invariant function fs E B(X) such that Snf -+ fs u-a.e. In this case, for f E L 1 (X ,ii), Snf -+ fs ti-a.e., ii-a.e. and in L 1 (X ,ii), and fs == Ej.£(fIJ) == Eji(fIJ) u-a.e. and ii-a.e., where J == {A EX: s:' A == A} is a a-algebra. (4) In (5) and (6) of Theorenl 6, B(X) can be replaced by C(X). (5) Pa(X) is weak* compact with the identification of Pa(X) c M(X) == C(X)*. When S is invertible, we can have some more characterizations of AMS sources. Proposition 8. Suppose that S is invertible. Then, for J.L E P(X) the following conditions are equivalent: (1) J.L E Pa(X). (2) There exists some 'TJ E Ps(X) such that J.L «: 'TJ. (3) There exists some 'TJ E Pa (X) such that J.L «: 'TJ.

Proof. (1) => (2). Let 11 == ii E Ps(X). Since S is invertible, we have SX == X == S-lX and hence Xoo == s-nx == X. Thus the implication (1) => (3) in Theorem 6 n=O implies J.L «: 'TJ == ii. (2) => (3) is immediate. (3) => (1). If'TJ E Pa(X), then 11 «: fj by the proof of (1) => (2). Hence, if J.L «: 'TJ,

n

Chapter II: Information Sources

96 a

then J-t ~ fj. This implies J-t ~ fj as was remarked in the paragraph before Lemma 5. Thus Theorem 6 concludes that (1) is true. Remark 9. In the case where S is invertible, note that the asymptotic dominance implies the usual dominance in the sense that, if J-t E P(X), 'fJ E Ps(X) and J-t ~ n, then J-t «'fJ. This immediately follows from the above proof. Also note again that J-t ~ ji for J-t E Pa(X). Example 10. (1) Take a stationary source 'fJ E Ps(X). By Remark 7 (2), any J-t E P(X) such that J-t ~ 'fJ is AMS. Hence, if I E L1(X, 'fJ) is nonnegative with norm 1, then J-t defined by

JL(A) =

L

AEX

f dn,

is AMS. In this case the stationary rnean ji of J-t is given by 1 ji(A) == lim n-+oo n

n-l

L J-t(S-k A)

k=O n-l

=

.!. L f f d'f/ n-+oo n k=O JS-k A lim

n-l

== lim n-+oo

==

= where

Is

== lim

n-+oo

particular, if J-t

~

j

jA .!.n k=O L f(Skx) 'f/(dx)

lim Snl d'fJ

A n -+ oo

L

fsd'f/,

A E X,

(4.4)

Snl, by the Pointwise Ergodic Theorem since 'fJ is stationary. In ii, then

where Jji == {A EX: ji(S-l A~A) == O}. (2) Take a stationary source 'fJ E Ps(X) and a set B E X such that 'fJ(B) Then, J-t defined by J-t

n B) == (AlB) (A) == 'fJ(A 'fJ(B) 'fJ ,

> O.

2.4. AMS sources

97

is AMS since fL «: "l. In fact, this is a special case of (1) since fL(A) = fA 71(B) d"l for A E X. Similarly, take an AMS source "l E Pa(X) with the stationary mean fj and a set B E X with "l(B) > O. Then the conditional probability fL(') = "l(·IB) of"l given B is also AMS. For, if fj(A) = 0, then "l(B)fL(limsupS- nA) 5: "l(lirnsupS- n A) = 0 n-too

n-too

by tt ~ fj. Hence lirn fL(s-n A) = O. This means that fL ~ fj. Theorem 6 deduce that n-too fL is AMS. It follows that AMS property remains in conditioning while stationarity is lost by conditioning, in general.

DefinUion 11. ~~~~~~~~~~~~~~~~~~~fL(A) =Oor 1 for every S-invariant AMS sources. We have several equivalence conditions of ergodicity for an AMS source.

Theorem 12. For fL E Pa(X) with the stationary mean Ji E Ps(X) the following conditions are equivalent: (1) fL E .Pae(X). (2) Ji E Pse(X).

(3) There exists some "l E Pse(X) such that fL ~ "l.

(4) Is(x) == lim (Snl)(x) = ( I dji Ii-a.e. and u-a.e. lor I

l»

n-too

E

L 1(X,ji).

(5) n-too lim ( s, f, g) 2 ,11- = (f, 1)2 ' ti (1, g) 2 ,11- for t, g E L 2 (X, fL) n L 2 (X, Ji). (6) lim fL((Snf)g) = Ji(f)fL(g) for every f,g

E B(X).

(7) lim fL((Snf)g) = Ji(f)fL(g) for every t,s

E C(X).

n-+oo n-+oo

1

n-l

n

k=O

1

n-l

(8) lim n-+oo

(9) lim -

L

L

n-+oo n k=O

fL(S-k A n B) = Ji(A)fL(B) for every A, B E X. fL(S-k A n A) = Ji(A)fL(A) for every A E X.

Proof. (1) {::} (2) is clear from the definition. (1), (2) =? (3) follows from Remark 7 (1) by taking "l = ii.

be such that fL ~ n. If A E X is S-invariant, then "l(A) = 0 or 1. If "l(A) = 0, then fL(A) = fL(s-n A) ~ 0 by fL ~ 'fJ, i.e., fL(A) = O. Sirnilarly, if "l(A) = 1, then we have fL(A) = 1. Thus fL E Pae(X). The implications (1) =? (4) =? (5) =? (6) =? (7) =? (8) =? (9) =? (1) are shown in much the same way as in the proof of Theorem 3.2.

(3)

=?

(1). Let "l

E Pse(X)

Chapter II: Information Sources

98

Remark 13. In (8) and (9) of Theorem 12, X can be replaced by a semialgebra Xo that generates X. Also in (5), (6) and (7) of Theorem 12, we can take 9 = f. Theorem 14. (1) If J-L E exPa(X), then J-L E Pae(X). That is, exPa(X) ~ Pae(X). (2) If Pse(X) ¥- 0, then the above set inclusion is proper. That is, there is a J-L E Pae(X) such that J-L ~ exPa(X). Proof. (1) This can be verified in exactly the same manner as in the proof of (4) => (1) of Theorem 3.2. (2) Let J-L E Pae(X) be such that J-L ¥- Ii. The existence of such a J-L is seen as follows. Take any stationary and ergodic ~ E Pse(X) (¥- 0) and any nonnegative f E Ll(X,~) with norm 1 which is not S-invariant on a set of positive ~ measure. Define J-L by

JL(A) =

1i d~,

AEX.

We see that J-L is AMS by Example 10 (1) and ergodic because ~ is so. Clearly J-L is not stationary. Hence J-L ¥- Ii. Also note that {i = ~ since for A E X

(i(A) =

1is d~

= ~(A)

by (4.4) andfs = 1 ~-a.e. because of the ergodicity of~. Then 'T/ = ~(J-L + {i) is a proper convex combination of two distinct AMS sources and 'T/(A) = 0 or 1 for S-invariant A E X. Thus 'T/ ~ exPa(X) and 'T/ E Pae(X). Again, if S is invertible, ergodicity of AMS sources is characterized as follows, which is similar to Proposition 8: Proposition 15. If S is invertible, then the following conditions are equivalent for

Pa(X): (1) J-L E Pae(X). (2) There exists some 'T/ (3) There exists some 'T/

J-L E

(4) There exists some 'T/

Pse(X) such that Pae(X) such that

J-L ~ 'T/.

E E

Pae(X) such that

J-L ~ 'TJ.

E

J-L

«

'TJ.

Proof. (1) => (2). Take 'T/ = {i E Pse(X), then J-L ~ {i = 'TJ by Remark 9. (2) => (3) is clear. (3) => (4). Let 'TJ E Pae(X) be such that J-L ~ 'TJ. Then fj E Pse(X) and 'TJ ~ fj. a Hence J-L « fj and J-L ~ fj since fj is stationary.

(4) => (1). Let fJ E Pae(X) be such that J-L~'TJ. Then 'TJ' ~ fj and J-L~fj. Since Pse(X), Theorem 12 concludes the proof.

fj E

99

2.5. Shannon-McMillan-Breiman Theorem

Ergodicity and mixing properties may be defined for nonstationary sources. Let tt E P(X). tt is said to be ergodic if tt(A) = 0 or 1 for every S-invariant A E X. tt is said to be weakly mixing if n-l

! '" IJL(S-jA n B) n-+oo n L...J lim

I = 0,

JL(S-jA)JL(B)

A,BEX

j=O

and strongly mixing if A,BEX.

Clearly, these definitions are consistent with the ones for stationary sources and it holds that strong mixing implies both weak mixing and ergodicity. We can show that if tt is AM8 and weakly mixing, then tt is ergodic. In fact, the condition (8) or (9) of Theorem 12 may be easily verified.

2.5. Shannon-McMillan-Breiman Theorem An ergodic theorem in information theory, the so-called 8hannon-McMillanBreiman Theorem (8MB Theorem), is proved in this section. First we briefly describe practical interpretation of entropy (or information) in an alphabet message space. Then we formulate the 8MB Theorem in a general setting. Let us consider an alphabet message space (X, X, S), where X = X~, X o = {aI, ... ,at} and S is a shift. Let tt be a stationary information source, where we sometimes denote it by [X, ttl. For an integer n ;::: 1, ootn denotes the set of all messages of length n of the form [X~k)

••• x~k~ll,

Le., the messages of length n starting at time O. Note that ootn is a finite partition -

of X, i.e., ootn E P(X), and mtn

=

n-l

.

.V S-Joot1 . Hence the entropy of

J=O

ootn under the

measure tt is given by

where we write the dependence of the entropy on u. If oot~ denotes the set of all messages of length n starting at time i = ±1, ±2, ... ,

100

Chapter II: Information Sources

then the entropy of VJt~ is the same as that of VJtn VJt~ since f.-L is stationary. Hence the information (or entropy) per letter (or symbol) in messages of length n input from the stationary information source [X, f.-L] is

and if we let n -t

00,

then the limit

exists by Lemma 1.3.5, which represents tt!&~~~.!!!!Q!1~ll21~~l~~J2!J!!~ information source This is a interpretation Kolmogorov-Sinai entropy of the shift S, 00 denoted H(f.-L, S), is equal to H(f.-L) by Theorem 1.3.10 since V snVJt1 = X'. n=-oo

To formulate the 5MB Theorem we are concerned with entropy functions. Let (X, X, S) be an abstract measurable space with a measurable transformation Sand f.-L E Ps(X). Recall that for a finite partition 2t E P(X) and a a-subalgebra q) of X', the entropy function 1p,(2t) of 2t and the conditional entropy function 1p,(2t1q)) are defined respectively by Ip,(2t)(·) =

E 1A(·) log f.-L(A) , AEQ(

1p,(2t1q))(·) = -

E 1A(·)logPp,(AIq))(·), AEQ(

where Pp,(·Iq)) is the conditional probability relative to q) under the measure f.-L (cf. Section 1.3). These functions enjoy the following properties: for 2t,~,
(1) 1p,(2t) = 1p,(2t12), where 2 = {0, X}, (2) 1~(2t V ~)

= Ip,(2t) + 1p,(~12t), where 2t = a(2t),

(3) 1p,(2t V ~IQ:) = 1p,(2t1Q:) + 1p,(~12t V Q:), (4) 1p,(2t) 0 S = 1p,(S-12t),

(5) 1p,(2t1~) 0 S = Ip,(S-l2tIS-l~). In fact, (1) - (5) are in Remark 1.3.2. n (6) t; ( .V 2tj

J=l

)

=

Ln

k=l

- ) t; ( 2tk I k-l .V 2t for 2t0 = j

J=O

2, 2t1 ,

•..

,2tn E PCX') and n 2: 1.

For, this is verified by (1),(2),(3) and the mathematical induction.

(7) t; (~~l S-j2t) = 1p,(s-(n-l)2t) J-O

+

f: t; (s-(n-k-l)2tI.~ s-(n- )§(). j

k=l

J-1

101

2.5. Shannon-McMillan-Breiman Theorem

This is obtained from (6) by letting

(8) IJ.t

.'!... s-j2t = IJ.t(2t) (n-l) J-O

0

2tj

= s-(n- j)2t, 1 ~ j

(I

s-:: + I:n IJ.t 2t k=l

< n.

-) .'!k... S-(k-j+l)2t

0

sn-k-l.

J-l

This is immediate from (4) and (7). Now for n = 1,2, ... let

in = IJ.t(n\jl S-k2t) , k=O go

= 11£(2t),

gn

= 11£ ( 2t1 k21 S-ksit) , 9 = 11£ ( 2t1 k~l S-ksit).

Then the equation in (8) can be written as

(9) in ==

n-l I: sn-k-l gk for n 2: 1, where Sgk k=O

(10) H(p" 2t,S) =

gk 0 S.

L

9 dp:

For, by Lemma 1.3.5 it holds that

H(f.-l,

2t, S) =

lim HJ.t

n~oo

r

(2t1 .V

J=l

=

lim 11£ n~oo Jx

=

lim n~oo

S-j2t)

(2tI.V S-j2t) df.-l J=l

r gn df.-l l»

=

r n~oo lim gn du, l»

=

LgdP,.

since gn -+ 9 in L 1 by Lemma 1.3.9,

Similarly, we get

H(f.-l,

2t, S) =

(n-1 .) (n-l S-J2t.) df.-l IJ.t.V

1 lirn -HJ.t .V S-J2t n J=O

n~oo

11

= lim n~oo

= lirn

n x

r!f

n~ooJx n

J=O

n

(5.1)

du,

Ix

We expect that ~in -+ h for some S-invariant function hand Hi p: 2t, S) = h du. The 5MB Theorem guarantees that this is the case, which is given as follows:

Chapter II: Information Sources

102

Theorem 1 (Shannon-McMillan-Breiman). Let Jl E Ps(X) be a stationary source and 2l E P(X) be a finite partition of X. Then there exists an S-invariant function h E L1(X, Jl) such that lim

n-+oo

!/n = n

lim

n-+oo

!IIJ(n

v S-k2l) = h Jl-a.e. and in L

1

1 k=O

n

H(p" 21, S)

=

Ix

h du

>

,

(5.2)

1x9dP,.

If, in particular, Jl is ergodic, then

Ix

h = H(p" 21, S)

Proof. Since

9 dp, u-a. e.

V S-k21T k=l V S-k21 we can apply Lemma 1.3.9 (1) to obtain

k=l

gn

--+ 9 u-a.e. and in L 1 .

(5.3)

It follows from the Pointwise Ergodic Theorem (Theorem 2.1) that there exists an h E L 1(X, Jl) such that Sh = h u-a.e. and 1 n-1

-n E sn-k-1 g = Sng -+ h

Jl-a.e. and in L 1.

(5.4)

k=O

Now observe that

II ~ In - hll l ,1J = 1/ ~ ~sn-k~lgk - hl/

,1J l

~ II~ E(sn-k-l gk -

sn-k-lg)11

: :; ~ E

II ~

k=O

IIgk

glll,1J +

k=O

-+0

1,1-£

I:

+ II~ Esn-k-l g - hll k=O

sn-k-l g -

k=O

1,1-£

hll 1,1-£

asn-+oo

by (5.3) and (5.4). This establishes the L1-convergence. To prove u-a.e. convergence we proceed as follows. We have frorn the above computation that

~ In - hi < ~ Esn-k-ll gk -

I

k=O

gl +

IE k=O

sn-k-l g -

hi.·

103

2.5. Shannon-McMillan-Breiman Theorem

The second term tends to 0 J-L-a.e. by the Pointwise Ergodic Theorem. So we have to show 1 n-l (5.5) lim sup sn-k- 119k - 91 = 0 u-a.e.

E

n k=O

n-+oo

For N = 1,2, ... let G N = sup 19k k~N

91·

Then, GN 4.- 0 u-a.e. since 9k -+ 9 u-a.e., and

o::; Go ::; sup 9n + 9 E L 1(X, J-L) n~l

by Lemma 1.3.8. Let N 2 1 be fixed. If n

~

I:

sn-k-1I gk -

> N, then

(I: + I:) < .! I: sn-k-1Go+.! ~

gl =

k=O

~

sn-k-1I gk -

k=O

n k=O

=

~

gl

k=N

sn-k-1G N

n k=N

N-l

n-N-l

k=O

J=o

""' sn-k- 1 G o + n - N . _1__ nL.-J n n-N

""' SiGN. ~

Letting n -+ 00, we see that the first terrn on the RHS tends to 0 J-L-a.e. and the second term converges to some S-invariant function GN,S E L 1(X, J-L) J-L-a.e. by the Pointwise Ergodic Theorem. Thus we have 1 n-l lim sup sn-k- 119k -

E

n-+oo

Finally, note that G N,S 4.- as N -+

L

91::; GN,S

u-a.e.

n k=O

GN,sd/L

=

00

and by the Monotone Convergence Theorem

L

GNd/L -+ 0

as N -+

00

since G N 4.- 0, which implies that GN,S -+ 0 p-a.e. Therefore (5.5) holds and J-L-a.e. convergence is obtained. (5.2) is clear from (10) above and (5.1). Corollary 2. Let X = X~ be an alphabet rnessage space with a shift Sand J-L E Ps (X) be a stationary source. Then there exists an S-invariant junction h E L1(X, J-L) such that lim

n-+oo

{_.!n

"'" L.-J

XO, •••

,xn-lEXo

1["'0"'''' n -d log /L([xo ... xn-1J)} = h

u-a.e. and in L

1

.

Chapter II: Information Sources

104

If, in particular, J1 is ergodic, then h == H (J1, S) J1-a. e. Proof. Take 2t == rot 1 E P(X) and observe that for n

~

1

i« == t; (nvl S-k2t) == Ip,(rotn), k=O

== -

L

1A log J1(A)

AEVJt n

L

l[xo···x n - d

log J1([xo ... Xn-l])'

xo,··· ,xn-1EXo

Thus the corollary follows from Theorem 1. The following corollary is sometimes called an entropy equipartition property.

Corollary 3. Let X == X~ be an alphabet message space with a shift Sand J1 E

Pse(X) be a stationary ergodic source. For any c > 0 and 8 > 0 there is an integer no

~

1 such that

Hence, for n ~ no, the set rotn of messages of length n starting at time 0 can be divided into two disjoint subsets rotn,g and VJtn,b such that (1) e~nH(p"S)-c < J1(M) < e-nH(p"S)+c for every message M E VJtn,g.

(2)

JL(

U MEVJtn,g

(3)

JL(

U MEVJtn,b

M) ~ 1 M) < O.

s.

Proof. Since a.e. convergence implies convergence in probability the corollary follows immediately from Theorem 1. Remark 4. (1) In Corollary 3, for a large enough n (n ~ no) each message M E VJtn,g has a probability approximately equal to e-nH(p"S) and hence the number of messages in VJtn,g is approximately enH(p"S). Since the number of messages in VJt n is /!n == enlogl and H(J1,S) S log z, it holds that IVJtn,gl ~ IVJtn,bl, i.e., the number of elements in VJtn,g is much larger than that of those in VJtn,b' (2) Another consequence of Corollary 3 is that if we receive a message long enough, then the entropy per letter in the message almost equals the entropy of the information source. A generalized version of Theorem 1 is formulated as follows:

2.5. Shannon-McMillan-Breiman Theorem

105

Corollary 5. Let (X, X) be a measurable space with a measurable transformation S : X -+ X. Let J-L E Ps(X) be a stationary source and 2{ E P(X) be a finite partition of X. Assume that V is a a-subalqebra of X such that S-lV = V and let

In = I~ (:~~ S-k2lI!D), 90

= I~(2lI!D), 9n = I~ (2l1 k21 S-k2i V!D),

9

= I~ (2l1 k~l S-k2i V!D)

for n ~ 1. Then there exists an S-invariant function h E L 1(X, J-L) such that

lirn

n-+oo

If, in particular,

!n I«

1 lim -11£

n-+oo

n

(n-1V S-k2{I) V = h k=O

V ~ 2loo == V S-k21, k=l

H (p"

u-a.e. and in L 1 .

(5.6)

then

2l, S) =

L

h du.

(5.7)

Proof. (5.6) can be verified in a similar rnanner as in the proof of Theorem 1. As to (5.7) assurne that V ~ 2loo . Then we have

H(J-L, 2{, S) = lirn HJ.£(QtI2ln) , n-+oo

where

2{n =

.\1

J=l

s-jQt and

=

HJ.£(QtI21oo )

=

HJ.£(QtI21oo

2tn

=

V

V),

by Lemma 1.3.5,

byassurnption,

u(Qtn ) as before. Thus we get

Since the function h in Corollary 5 depends on J-t E Ps(X) we should denote it by

hJ.£' so that H(p"

2l, S) =

Ln,

dp"

(5.8)

106

Chapter II: Information Sources

This will be applied to obtain an integral representation of the entropy functional in Section 2.7.

2.6. Ergodic decompositions In this section, ergodic decomposition of a stationary source is studied. Roughly speaking, if a measurable space (X, X, S) with a rneasurable transformation S : X -+ X is given, then there is a measurable family {JLx}xEX of stationary ergodic sources, which does not depend on any stationary source, such that each stationary source JL is written as a mixture of this family:

J.t(A) =

L

AEX.

J.tx(A) J.t(dx),

This is, in fact, a consequence of the Krein-Milman Theorem: P, (X) = co Ps e (X) (cf. Section 2.1). Our setting here is that X is a compact metric space, so that X is the Baire (= Borel) a-algebra and the Banach space C(X) is separable. Let {fn}~=l c C(X) be a fixed countable 'dense subset. S is a measurable transformation on X into X as before. Recall the notation Sn (n 2: 1): for any function f on X 1

(Snf)(x) = n

n-l

.

E f(SJ x),

xEX.

j=O

Definition 1. For n 2: 1 and x E X consider a functional Mn,x on B(X) given by

Mn,x(f) = (Snf)(x),

f

E

B(X).

A point x E X is said to be quasi-regular, denoted x E Q, if

exists for every f E C(X). A measurable set A E X is said to have invariant measure 1 if JL(A) = 1 for every JL E Ps(X).

Lemma 2. For each quasi-regular point x E Q there is a unique stationary source JLx E Ps(X) such that JLsx = JLx and

Mx(j) =

L

fey) J.tx(dy),

f

E

C(X).

(6.1)

107

2.6. Ergodic decompositions

Moreover, Q is an S-invariant Baire set of invariant measure 1. Proof. Let x E Q. It is not hard to see that M x ( · ) is a positive linear functional of norm 1 on C(X) since

IIMn,x(·)11 =

sup IMn,x(f)1 IIfll~1

sup I(Snf)(x)1 ~ sup

=

IIfll~1

IIfll~1

Ilfll =

1

and Snf = f for 8-invariant f E C(X). Thus, by Riesz-Markov-Kakutani Theorem there exists a unique source J-Lx E P(X) such that (6.1) holds. To see that J-Lx is stationary, we note that Mx(Sf) = Mx(f) for f E C(X). Hence for f E C(X)

L

f(y) /l-x(dy) = Mx(f) = Mx(Sf)

=

L

f(Sy) /l-x(dy) =

L

f(y) /l-x(dS-1y).

This implies that J-Lx = /-Lx 0 8- 1 or J-Lx is stationary. That J-Lsx = J-Lx is also derived from Msx(f) = Mx(Sf) = Mx(f) for f E C(X). Let {fk}k=1 C C(X) be dense and

Ak =

{x EX:

lim (Snfk)(X) = lim M n x(fk) exists},

n--+-oo

n--+-oo'

k 2:: 1.

00

Then we note that Q = n A k since {fk} is dense in C(X). Also note that A k is an

k=1

Fo-Ii set in X for k 2:: 1 and J-L(A k ) = 1 for every J-L E Ps(X) by the Pointwise Ergodic

Theorem. Now we see that Q

00

= n A k is a Baire set such that J-L(Q) = 1 for every k=1

J-L E Ps(X), or Q has invariant measure 1.

From now on we assume that {J-Lx}xEQ is a family of stationary sources obtained in Lemma 2. Denote by B(X, 8) the set of all 8-invariant functions in B(X). Clearly B(X,8) is a closed subspace of the Banach space B(X). For f E B(X) let

f~(x) = {.

L

f(y) /l-x(dy) ,

x

E

Q,

(6.2)

x¢:. Q.

0,

If f E C(X), then f~ E B(X) since f~(x) = Mx(f) for x E Q and Q E X. For a general f E B(X) choose a sequence {gn}~1 C C(X) such that lim f gn(Y) J-Lx(dy) n--+-oolx = lim Mx(gn) lim g~(x),

t,( f(y) J-Lx(dy) =

n--+-oo

n--+-oo

XEQ.

108

Chapter II: Information Sources

Hence

f Q E B(X). Now we have:

Lemma 3. If f E B(X), then f Q E B(X,B). The mapping A : f I---t jQ is a 1. projection of norm 1 from B(X) onto B(X,B), so that A2 == A and IIAII Moreover it holds that for J-L EPs(X) and f E B(X)

L f(x) p,(dx)

=

k

(L f(y) P"AdY)) p,(dx)

=

kf~

(x) p,(dx).

(6.3)

Proof. Since J-Lsx == J-Lx for x E Q by Lemma 2, we have that

f~(Sx) = L

f(y) p,sx(dy) = L f(y) P,x(dy)

= f~(x),

x

E

Q

and fQ(Bx) == 0 == fQ(x) for x t/. Q. Hence f Q E B(X, B). If f E B(X, B), then clearly f Q == f. Thus A is a projection of norm 1. To see (6.3) we proceed as follows. Let J-L E Ps(X). Then, since J-L(Q) == 1 by Lemma 2 we see that for f E C(X)

( f(x) J-L(dx) == ( lim (Snf)(x) J-L(dx) ,

Jx

Jx n-+oo

=

by the Pointwise Ergodic Theorem,

k

Mx(J) p,(dx)

= k(Lf(y)p,x(dY))P,(dX)

=

kf~(x)

p,(dx).

Let B == {f E B(X) : (6.3) holds for fl. Then B contains C(X) and is a monotone class, which is easily verified. Thus B == B(X) and (6.3) is true for every f E B(X). The following corollary is immediate. Corollary 4. Let J-L E Ps(X). Then, for j E L l (X , J-L),

f Q == Ep,(fIJ) u-a.e., where f Q is defined by (6.2), J == {A EX: B- 1 A == A} and Ep,('IJ) is the conditional expectation w.r.t. J under the measure J-L. Definition 5. A point x E Q is said to be regular if the corresponding stationary source J-Lx is ergodic, i.e., J-Lx E Pse(X). R denotes the set of all regular points. Let J-L* E Pse(X) be a fixed stationary ergodic source and define x

rt R.

109

2.6. Ergodic decompositions

The set {JLx} xEX is called an ergodic decomposition relative to S. We summarize our discussion in the following form.

Theorem 6. Let X be a compact metric space, X the Baire a-alqebra and S : X -+ X a measurable transformation. Then the set R of all regular points is an S-invariant Baire set. Let {JLX}XEX be an ergodic decomposition relative to S. Then, for any stationary source JL E P; (X) the following holds: (1) JL(R) = 1. (2) For f E L 1(X, JL) let f Q be defined by (6.2) and f#(x} = f(y) JLx(dy) if x E Rand = 0 otherwise. Then,

Ix

lim (Snf)(x)

n-+oo

t"

i

f(x) p,(dx) =

= n-+oo' lim M n x(f) = Ell-x (f) J-L-a.e.,

(6.4)

= f Q = EIl-(fIJ)

(6.5)

L

f(x) p,(dx) =

p-a.e.,

L(i

f(y) P,.,(dY))P,(dX).

(6.6)

In particular, if f = 1A (A E X), then

p,(A) =

L

(6.7)

p,.,(A) p,(dx).

Proof. Since JLsx(A) = JLx(A) for x E Q and A E X by Lemma 2, we see that R is S-invariant. Note that by Remark 3.3 (4) R

= {x

E Q: lim J-Lx((Snf)f) n-+oo

= J-Lx(f)2, f

= {x E Q: lim JLx((Snfk)fk) n-+oo

n 00

=

{x

E Q:

E C(X)}

= J-Lx(fk)2, k 2: I}

}~~P,.,(Snfk)!k) = P,.,Uk)2}.

k=l

Ix

Since for each f E C(X), JL(.)(f) = fQ(y) J-L(.) (dy) is X-measurable, R is an Xmeasurable set. Let JL E Ps(X). We only have to verify (1). Observe the following two-sided implications:

JL(R)

= 1 ~ J-Lx ~

E

Pse(X) u-a.e.x

lim (Snfk)(Y) = JLx(fk) J-Lx-a.e.y, u-a.e.x, k 2: 1

n-+oo

110

Chapter II: Information Sources 2

-¢==>

L Ifk,S(Y) - JLx(ik) 1 JLx(dy) = 0 p-a.e.x, k where fk , s(y)

-¢==>

Now for each k

~

~ 1,

= n--+oo lim (Snfk)(Y), Y E X, 2

JLx(fk) 1 JLx(dY))JL(dX) = 0, k

L (L Ifk,S(Y)

~ 1.

1 it holds that 2

L (Llfk'S(Y) - JLx(ik) 1 JLx(dy) )JL(dX)

= ~ (L Ifk,S(Y)1 JLx(dy) -IJLx(fk) 1 ) JL(dx) 2

2

= ~ E/L (Ifk,sl 2IJ) (x) JL(dx)

~ IL

ik dJLxr JL(dx)

= ~ lik,s(x) 1 JL(dx) -

~ If~(x)12 JL(dx)

= ~ Ifk,S(x)1 JL(dx) -

~ Ifk,S(x)1 2 JL(dx) ,

2

2

since f~(x) =0.

= Mx(!k) = !k,S(X) for x E Q,

We should remark that if (X, X) is an abstract measurable space with a countable generator, then for any measurable transformation S : X -+ X we can find an ergodic (6.7) hold. For a detailed decomposition {J-tX}XEX ~ Pse(X) for which (6.4) treatment see e.g. Gray [1].

2.7. Entropy functionals, revisited In Section 1.5 we considered an entropy functional H(·, 2(, S) on the set of all S-invariant C-valued measures on a measurable space (X, X), where 2( E P(X) is a fixed finite partition of X and S is an automorphism on X, and we derived an integral representation of H (., 2(, S) under certain conditions. In this section, we shall show an integral representation of the entropy functional by a universal function by two approaches: functional and measure theoretic ones. Also we shall extend the entropy functional to the space Ma(X) of all AMSmeasures. We begin with a functional approach using ergodic decompositions. Our setting is as follows. Let X be a totally disconnected compact Hausdorff space, S a fixed

111

2. 7. Entropy functionals, revisited

homeomorphism and X the Baire o-algebra, Since an alphabet message space X~ is compact and totally disconnected, the above conditions are fairly general. Take any clopen partition 2( E P(X) consisting of disjoint clopen sets in X. As in Section 1.5, we use the following notations: n

a, =

V s-j2( E P(X), j=l

for n 2: 1. P(X), Ps(X), M(X) and Ms(X) are as before. Since we write the entropy of J-L E Ps(X) relative to 2( and S by

2(

and S are fixed,

H(J-L) = H(J-L, 2(, S)

=-

E Jxr Pp(AIQ{oo) logPp(AIQ{oo) dp.

(7.1)

AE2{

= -

lim 1

n-too

n

'"'" L...J

J-L(A) log J-L(A) ,

AE2{V2{n-l

where Pp,(AI~) is the conditional probability of A relative to a zr-subalgebra ~ of X under the probability measure J-L. To reach our goal we need several lemmas.

Lemma 1. If J-L, 'fJ E Ps(X) and 'fJ« J-L, then for each A E

2(

(7.2) Proof. Let ~ .=

0- ( { sj 2( :

j E Z}). Since ~ has a countable generator, for each

J-L E P, (X), every S- invariant B E ~ belongs to 2100 (mod J-L), i.e., there exists a set B' E 2100 such that J-L(BllB') = O. Let J-L' = J-LI~, the restriction of J-L to ~ for each J-L E Ps(X). Then, J-L' is S-invariant and AE2(

on Cp" the support of J-L. S-invariance of Cp, implies that Cp, E 2l00 • Now let J-L,'fJ E Ps(X) be such that 'fJ ~ J-L. Then, 'fJ'« J-L' and ~ is S-invariant and ~-measurable, hence 2loo -measurable (mod u). Thus, we have that for A E 2{ and B E 2100

112

Chapter II: Information Sources

so that (7.2) is true. Lemma 2. Let J-t E Ps(X). Then there is a bounded, upper semicontinuous and 2t00 -measurable function hl-£ such that

hl-£ == -

L

PI-£(AI2too ) 10gPI-£(AI2t00 ) J-t-a.e.,

(7.3)

AE~

(7.4)

Proof. Note that PI-£(AI2tn) E C(X) for A E 2t. since each B E 2t.n is clopen. For n 2: 1 let (7.5) hl-£,n == PI-£(AI2tn) 10gPI-£(AI2tn)

L

AE~

and observe that hl-£,n E C(X) and hl-£,n 4- by Jensen's Inequality (cf. (12) in Section 1.2). If we let hl-£ == n~oo lim hl-£ ' n, then hl-£ is upper semicontinuous since each hl-£ ,n is continuous, and is 2too-measurable since each hl-£,n is 2tn-measurable. Moreover, {hl-£,n}~=l forms a submartingale relative to {2tn} on (X, X, J-t) and (7.3) is obtained by the Submartingale Convergence Theorern (cf. (25) in Section 1.2).(7.4) follows from (7.3) and (7.1). Lemma 3. The functional H(·) on Ps(X) is weak* upper semicontinuous, where we identify Ps(X) c M(X) == C(X)*.

Proof. For J-t E Ps(X) and n 2: 1 let Hn(p,)

=

L

hp"n dp"

where hl-£,n is defined by (7.5) in the proof of Lemma 2. Since Hn(J-t) -+ H(J-t) as n -+ ex> for each J-t E Ps(X), it suffices to show the weak* continuity of H n(·) on Ps(X), which immediately follows from that Hn(p,)

===

L { lAlogPp,(AI2ln)dp,

AE~JX

L L

AE~BE~n

{J-t(A n B) log J-t(E) - J-t(A n B) log J-t(A n B)}

2. 7. Entropy junctionals, revisited

113

and that A E 2t. and B E 2t.n are clopen. In order to use the ergodic decomposition of 8-invariant probability measures developed in the previous section, we need to introduce a metric structure in X. 00 . Denote by Q3 0 the algebra of clopen sets generated by . u 8 J 2t. and let Q3 = a(Q3o), J=-OO

the a-algebra generated by Q3o. In the Banach space C(X), let C(X, 2t.) be the closed subspace spanned by {IA : A E ~o} ~ C(X). Note that C(X,2t.) has a countable dense subset {jn}~=l since Q3 0 is countable. Define a quasi-metric d on X by x,yEX

and an equivalence relation X

Then the quotient space by

rv

rv

y

by ¢:=>

d(x, y) = 0,

x,yE X.

X = X/ rv becomes a metric space d(x, ii)

= d(x, y),

with the metric d defined

x,yE X,

where x {z EX: Z rv X }, the equivalence class containing x. Moreover, (X,d) is a compact metric space, the canonical mapping x I-t x from X onto X is continuous, and C(X,2t.) is isometrically isomorphic to C(X) with an isometric isomorphism C(X) 3 j I-t j E C(X), where j(x) = j(y),

Hence ~ = {B : B E Q3} with mapping § on X given by

B = {x : x

ss =

S»,

x E X,y E E

x.

B} is the Baire a-algebra of

X.

The

xEX

is well-defined and a homeomorphism. Therefore the triple (X,~, 8) consists of a compact metric space X with the Baire a-algebra ~ and a homeomorphism 8.

Lemma 4. For a positive linear junctional ;\ on C(X,2t.) oj norm 1 there is a probability measure J-LA E P(X, Q3) such that j E C(X,2t.),

where P(X, Q3) is the set of all probability measures on (X, Q3). Moreover, if ;\ is S-invariant, i.e., ;\(8j) = ;\(j) for j E C(X,2t.), then J-LA is S-invariant.

114

Chapter II: Information Sources

Proof. Let A(f) = A(f) for f E C(X,21). Then.:x is a positive linear functional of norm 1 on C(X), and hence there is a probability measure J.LX on (X, ~) such that

i E C(X). By letting J.L>.(B) = J-tx (iJ) for B E 23, we have the desired measure J-t>.. Recall some notations and terminologies from the previous section. For x E X, f E C(X) and n ~ 1 we denote n-l

= (Snf)(x) = ~ E f(Si x) = Mn,x(J).

Mn,x(f)

j=O

Let

Qand R be the sets of quasi-regular and regular points in X, respectively.

Then

Q and R have invariant measure 1, i.e., jj(Q) = jj(R) = 1 for jj E Ps(X) by Lemma 6.2 and Theorem 6.6. If Q = {x EX: x E Q} and R = {x EX: x E R}, then Q and R have invariant measure 1, i.e., J-t(Q) = J-t(R) = 1 for J-t E Ps(X), where Q and R are called the sets of quasi-regular and regular points in X relative to C(X,21), respectively. Thus, for each x E Q, Mx(f) = lim M n x(f) exists for n-+oo ' f E C(X, 21), M x ( ' ) is a positive linear functional of norm 1 on C(X, 21), and there is an S-invariant probability measure J-tx = tiu; on (X,23) such that Mx(f)

=

L

f

f(y) /lx(dy),

E C(X,

21).

Lemma 5. For a bounded 23-measurable function f on X

f~(r) =

L

f(x) /lr(dx) ,

r E

R,j

E

C(X, 21)

is a bounded, 23-measurable and S-invariant function on R and satisfies that

for every J.L E Ps(X). Proof. Let f be bounded and 23-rneasurable on X. By Theorem 6.6 we see that the function 9 on R defined by

9(1')

=

fx lUi;)

/If(dx)

= f~(r),

r ER

115

2.7. Entropy functionals, revisited

is ~- measurable and satisfies

k

g(1') {t(d1') =

S-invariance of

fx

j(i) {t(di),

f Q follows from that of /-Lx for x

E

Q.

Under these preparations, we are now able to prove the integral representation of the entropy functional.

Theorem 6. Let X be a totally disconnected compact Hausdorff space with the Baire cr-algebra X and a homeomorphism S. If 2( E P(X) is a clopen partition of X, then the entropy functional H (.) = H (', 2(, S) on P, (X) has an integral representation with a universal bounded nonnegative S-invariant Baire function h on X:

H(p,) =

Ix

(7.6)

h(x) p,(dx) ,

h(x) = hp.(x) u-a.e.,

/-L E

(7.7)

Ps(X),

where hl-£ is given by (7.3) and h is unique in Ps(X)-a.e. sense. Proof. Let us denote by Ps(X,~) the set of all S-invariant measures in P(X, ~). Define a function h on X by if r E R, if r rJ. R. Clearly h is nonnegative. We shall show that h is bounded, S-invariant and ~­ rneasurable. Note that each hl-£r '(r E R) is obtained by the limit of {hl-£r,n}~=l (cf. (7.5)) as was seen in the proof of Lemma 2: hl-£r,n 4- hl-£r' Let

gn(r)

=

Ix

hl-'r>n(x) P,r(dx)

L L

/-Lr(AnB){log/-Lr(AnB)-log/-Lr(B)}

AEQlBE~n

for r E R. Since /-Lr(O) = Mr(lc) (0 E ~o) is a ~-measurable function of r on R, gn(') is also ~-measurable on R. Hence h is ~-measurable on R since hl-£r,n 4- hl-£r and h(r) = f hl-'r (x) P,r(dx) = lim gn(r), r E R.

l»

n-+oo

This and the definition of h imply that h is bounded and S-invariance of h follows from that of /-Lr'

~-measurable

on X.

116

Chapter II: Information Sources

To show (7.6) let A E 21, B E that on one hand

'I7(A n B)

=

l

=

l (l

2Ioo

and J-t,1] E Ps(X) with

P'1(Alsi(",) d'17 =

l

1]

«: u. Then it holds

PJ.'(AI2too ) d'l7,

by Lemma 1,

PI'(AI2too)(x) J-tr(dX)) 'I7(dr),

by Lemma 5,

and on the other hand

'I7(A n B) = = =

L

1AnB d'17

l l(l

= ll~nB d'17

J-tr(A n B) '17 (dr) ,

by Lemma 5,

Pl'r(A I2too)(X)J-tr(dX)) 'I7(dr).

It follows from these equalities and S-invarianceof J-tr (r E R) that for each A E 21

(7.8) J-tr-a.e. for u-a.e.r, We can now derive (7.6) as follows: for J-t E Ps(X)

H(J-t)

= = = = =

L =l l (L l (L l L hJ.'dJ-t

ht(r) J-t(dr) ,

by Lemma 5,

hl'(x) J-tr(dX)) J-t(dr), hl'r(x) J-tr(dX)) J-t(dr),

by Lemma 5, by (7.8),

h(r) J-t(dr) ,

by the definition of h,

h(x) J-t(dx) ,

by the definition of h.

As to (7.7), let u; rJ E Ps(X) be such that rJ «J-t. Then by Lemmas 1,2 and (7.6)

H('I7)

=

L =L L h'1 d'17

hl'd'l7=

h dn,

and hence for every S-invariant f E L 1(X, J-t)

L

h(x)f(x) J-t(dx)

=

L

hl'(x)f(x) J-t(dx).

117

2.7. Entropy functionals, revisited

Since hand hJ.t are S-invariant, (7.7) holds. The function h obtained above is called a universal entropy function associated with the clopen partition 2l and the homeomorphism S. Of course the functional H(·) is extended to Ms(X) with the same universal function h. Next we formulate an integral representation of the entropy functional by a measure theoretic method. So let (X, X, S) be an abstract measurable space with a measurable transformation S : X -+ X.

Theorem 7. Let 2l E P(X) be a fixed partition. Then there exists an S-invariant nonnegative measurable function h on X such that

(7.9)

(7.10)

h = hJ.t j.l-a.e., where hJ.t is given by (5.8). Proof. With the notation of 2ln

=

.v S-j2l,2{n = a(2l

n)

J=1

and 2{00

= a(

U 2{n) let

n=1

and observe that ~ is a a-subalgebra with S-1~ = ~ and ~ ~ 2{00. Hence by Corollary 5.5, for each j.l E Ps(X), there exists a function hJ.t such that

Let

J-L E

Ps(X) be fixed. Then for any A E 2l V 2ln -

1

we have that

1 k-1 . PJt(AI!D)(x) = lim sup k lA(SJ X ) p,-a.e.

L

k~oo

since the RHS is

~-rneasurable

j=O

and for any B E

~

k-l

k-1

f lim sup ~ L lA(Si x) p,(dx) = f lim sup ~ L IB(x)lA(Si x) p,(dx) JB k-+oo j=O Jx k-e-co j=O = f

lim sup ~

Jx k~oo

= j.l(AnB),

k-1

L lAnB(Six) p,(dx) j=O

(7.11)

118

Chapter II: Information Sources

where we have used the S-invariance of B and the Pointwise Ergodic Theorem. Let for A E X 1 k-l . xEX. fA(X) = lim sup k lA(SJ X),

E

k-+oo

j=O

It then follows from (7.11) that, with the notation in Corollary 5.5,

Since the LHS -t hl-£ I-£-a.e., we see that lim sup ..!:. n-+oo

n

E

lA(X) log fA(X) == h(x) j.L-a.e.

AE2(V2(n-l

Note that h is defined on X, is independent of 1-£, and satisfies (7.9) and (7.10). Moreover, his S-invariant mod 1-£ for 1-£ E Ps(X) since hl-£ is so. Thus we can redefine h so that h is actually S-invariant on X. Let 2{ E P(X) be a fixed partition and h be the S-invariant measurable function obtained in Theorem 7. Then h is called a universal entropy function. Finally we want to extend the entropy functional H(·) = H(·, 2{, S) to Pa(X), the space of all AMS sources, and hence to Ma(X) = {al-£+(3rJ : a, (3 E C, 1-£, rJ E Pa(X)}.

Proposition 8. Assume that (X, X, S) is an abstract measurable space with a measurable invertible transformation S : X -t X. Let 2{ E P (X) be a fixed partition. Then the entropy junctional H (., 2(., S) with a universal entropy junction h can be extended to a junctional H(., 2(., S) on Ma(X) with the same entropy junction h such that

1l(t;,,21,S)

= H(e, 21, S) =

L

hdt;"

Proof. Let ~ E Ma(X) and ~ E Ms(X) be the stationary mean. Since ~ « ~, ~ E Me(X) == {rJ E M(X) : rJ ~ ~} ~ Ma(X). Hence, by Theorem 1.5.9, the

functional H(·, 2{, S) can be extended to a functional H(., 2{, S) on Me(X) with the same entropy function h. But then, since h is invariant we see that

1l(t;,,21,S)

=

L = Lhd~ = H(~,21,S), hdt;,

Bibliographical notes

119

by Remark 4.2. This completes the proof. Results obtained in this section will be applied to derive an integral representation of the transrnission rate of a stationary channel in Section 3.6.

Bibliographical notes 2.1. Alphabet message spaces and information sources. Alphabet message spaces were introduced to formulate information sources and channels by McMillan [1](1953). Theorem 1.1 is shown in Umegaki [4](1964) where he proved that an alphabet message space is not hyper Stonean. 2.2. Ergodic theorems. Birkhoff [1](1931) proved the Pointwise Ergodic Theorem. The proof given here is due to Katznelson and Weiss [1](1982), which does not use the maximal ergodic theorem. von Neumann [1](1932) proved the Mean Ergodic Theorem. See also Akcoglu [1](1975).

2.3. Ergodic and mixing properties. (4) of Theorem 3.2 is obtained by Breiman [2](1960) and Blum and Hanson [1](1960) (see also Farrel [1](1962)). (4) of Theorem 3.6 is proved by Renyi [1](1958). Lemma 3.9 is due to Koopman and von Neumann [1](1932). An example of a measurable transformation that is weakly mixing but not strongly mixing is given by Kakutani [1](1973). Characterization of ergodic and mixing properties by the relative entropy is obtained by Oishi [1](1965) (Lemma 3.13 through Proposition 3.16). Related topics are seen in Rudolfer [1](1969).

2.4. AMS sources. The idea of AMS sources goes back to Dowker [2](1951) (see also [1, 3](1947, 1955)). Jacobs [1](1959) introduced almost periodic sources, which are essentially the same as AMS sources. Lemma 4.3 is proved in Kakihara [4](1991). Lemma 4.5 is due to Gray and Kieffer [1](1980). In Theorem 4.6, (2) and (5) are due to Rechard [1](1956), (3) and (4) to Gray and Kieffer [1] and (6) to Gray and Saadat [1](1984). Proposition 4.8 is shown by Fontana, Gray and Kieffer [1](1981) and Kakihara [4]. In Theorem 4.12, (2) is proved in Gray [1](1988), (8) is given by Ding and Yi [1](1965) (for almost periodic sources), and others are noted here. (1) of Theorem 4.14 is obtained in Kakihara [4] and (2) is in Kakihara [5]. (2) and (3) of Proposition 4.15 are in Kakihara [4] and (4) is in Kakihara [5]. 2.5. Shannon-McMillan-Breiman Theorem. The ergodic theorern in information theory is established in this section. Shannon's original form is Corollary 5.3 given in Shannon [1](1948). McMillan [1] obtained the L1-convergence in the alphabet rnessage space (Corollary 5.2). Breirnan [1](1957, 1960) showed the a.e. convergence. Corollary 5.5 is due to Nakamura [1](1969). There are various types of formulations and generalizations of 5MB Theorem. We refer to Algoet and Cover [1](1988), Barron [1](1985), Chung [1](1961), Gray and Kieffer [1], Jacobs [1], [3](1962), Ki-

120

Chapter II: Information Sources

effer [1, 3](1974, 1975), Moy [1](1960), [2, 3](1961), Ornstein and Weiss [1](1983), Parthasarathy [2](1964), Perez [1, 2](1959, 1964) and Tulcea [1](1960). 2.6. Ergodic decompositions. Ergodic decomposition is obtained by Kryloff and Bogoliouboff [1](1937). Oxtoby [1](1952) gave its comprehensive treatment. The content of this section is mainly taken from these two articles. See also Gray and Davisson [1](1974). 2. 7. Entropy functionals, revisited. The integral representation of an entropy functional by a universal entropy function in the alphabet rnessage space was obtained by Parthasarathy [1](1961) (see also Jacobs [5](1963)). For a totally disconnected compact Hausdorff space Umegaki [4] proved such a representation. Lemma 7.1 through Theorem 7.6 are due to Urnegaki [4]. Nakamura [1] derived an integral representation in a measure theoretic setting without using ergodic decompositions (Theorem 7.7). Proposition 7.8 is noted here. Extension of the entropy functional H(·) to almost periodic sources is shown by Ding and Yi [1] using the result of Jacobs

[1].

CHAPTER III

INFORMATION CHANNELS

In this chapter, information channels are extensively studied in a general setting. Using alphabet message spaces as models for input and output, we formulate various types of channels such as stationary, continuous, weakly mixing, strongly mixing, AMS and ergodic ones. Continuous channels are discussed in some detail. Each channel induces an operator, called the channel operator, which is a bounded linear operator between two function spaces and completely determines properties of the channel. One of the main parts of this chapter is a characterization of ergodic channels. Some equivalence conditions for ergodicity of stationary channels are given. It is recognized that many of these conditions are similar to those for ergodicity of stationary sources. AMS channels are considered as a generalization of stationary channels. Ergodicity of these channels is also characterized. Transmission rate is introduced as the mutual information between input and output sources. Its integral representation is obtained. Stationary and ergodic capacities of a stationary channel are defined and their coincidence for a stationary ergodic channel is shown. Finally, Shannon's first and second coding theorems are stated and proved based on Feinstein's fundamental lemma.

3.1. Information channels Definitions of certain types of channels are given and some basic properties are proved. As in Section 2.1 let X o = {al, ... ,ap } be a finite set,· construct a doubly infinite product space X = X~, and consider a shift transformation S on X. If X is the Baire a-algebra of X, then (X, X, S) is our_input space. Similarly, we consider an output space (Y, ID, T), whereY = Y oZ for another finite set Yo = {b I , ••. ,b q } , ~ is the Baire a-algebra of Y, and T is the shift on Y. The compound space (X x Y, X ®~, S x T) is also constructed, where X ® ~ is the a-algebra generated by {A xC: A E X, C E ~} and is the same as the Baire a-algebra of X x Y. We use notations p(n), Pa(n), ps(n), C(n), B(n), ... etc. for 0. = X, Y or X x Y. 121

Chapter III: Inforrnation Channels

122

Definition 1. A channel with input X = X~ and output Y for which the function v : X x q) -+ [0, 1] satisfies

= YoZ is a triple [X, t/, Y]

(cl ) v(x,') E P(Y) for every x E X; (c2) v(·, C) E B(X) for every CEq). In this case v is called a channel distribution or simply a channel. C(X, Y) denotes the set of all channels with input X and output Y. The condition (cl) says that if an input x E X is given, then we have a probability distribution on the output, and we can know the conditional probability v(x, [Yi' .. Yj]) of a particular message [Yi' .. Yj] received when x is sent. The technical condition (c2) is needed for mathematical analysis. to be if (c3) v(Sx, C) = v(x, T-1C) for every x E X and CEq), which is equivalent to (c3') v(Sx, Ex) = v(x, T- 1 Ex) for every x E X and E E XQ9q), where Ex Y : (x, y) E E}, the x-section of E.

= {y E

Since, in this case, Sand T are invertible, we may write the condition (c3) as (c3") v(Sx, TC) = v(x, C) for x E X and CEq). Cs(X, Y) denotes the set of all stationary channels v E C(X, V). Note that C(X, Y) and Cs(X, Y) are convex, where the convex combination is defined by

x EX, CEq). Let p(bk laj) be the conditional probability of bk received under the condition that aj is sent, where 1 ~ j < p and 1 < k < q. If a channel v is defined by j

(c4) V(X'[Yi"'Yj]) = TIp(y£lx£), where x = (Xl) E X and [Yi"'Yj] C Y is a £=i message, then it is said to be memoryless. The p x q matrix P = (P(bklaj))j,k is called Clearly every memoryless channel is stationary. On the 'V~Utl~~~~'v.L v E C X ,Y is said to have a nite memor r to be an (c5) There exists a positive integer m such that for any message V = [Yi'" Yj] with i ~ . j it holds that v(x, V)

= v(x', V),

x

= (Xk), x' =

(x~) E X

with Xk

= x~ (i - m

~ k ~

j).

In some literature, a finite memory channel is defined to satisfy (c5) above and the finite dependence condition A weaker condition than (c5) is as follows:.~~~~~~~~~~!£J~~~~~~

123

9.1. Information channels

(c5') v{·, [Yi·· ·Yj]) E C{X) for every message [Yi·· ·Yj] C Y, which is equivalent to

(c5") [f(·, y)v(o,dy) E G(X) for every f

E

G(X x Y).

The equivalence will be proved in Proposition 5 below.._------~ (c6) There exists some

'TJ E

P{Y) such that v{x,·)

~ 'TJ

_

for every x E X.

Now let v E C{X, Y) and J.L E P{X), which is called an input source. Then the output source J.LV E P{Y) and the compound source J.L®v E P{X x Y) are respectively defined by

I.LV(G)

=

p, ® v(E)

=

L

v(x, G) p,(dx),

L

v(x, Ex) p,(dx),

GE

!D,

E E X ®!D,

(1.1) (1.2)

where Ex is the x-section of E. (1.2) can also be written as P, ®

v(A x G) = Lv(x, G) p,(dx),

A E X,C E~.

Note that J.L ® v{X x C) = J.Lv{C) and J.L ® v{A x Y) = J.L{A) for A E X and C E ~. All of the above definitions (except (c4), (c5) and (c5')) can be made for a pair of general compact Hausdorff spaces X, Y with Baire a-algebras X, ~ and (not necessarily invertible) measurable transformations S, T, respectively. Or more generally, (cl)-- (c3) and (c6) are considered for channels with input and output of measurable spaces (X, X, S), (Y,~, T). In this case, we consider "abstract" channels v with input X and output Y. In what follows, unless otherwise stated, X and Y stand for general measurable spaces. Note that any output source 'TJ E P{Y) can be viewed as a "constant channel" by letting x E X,C E~. So we may write P{Y) C C{X, Y). In this case,

J.L E P{X). Thus, if 'TJ is stationary, the channel v", is stationary. Consequently, we may write Ps{Y) C Cs{X, Y). A simple and important consequence of the above definitions is:

124

Chapter III: Information Channels

Proposition 2. If v E Cs(X, Y) and J-L E Ps(X), then J-tv E Ps(Y) and J-L Q9 v E P, (X x Y). That is, a stationary channel transforms stationary input sources into stationary output sources and such inputs into stationary compound sources.

Proof. Let v E Cs(X, Y), J-t E Ps(X), A E X and CEq). Then we have J-LQ9 v((S x T)-I(A x C))

= J-tQ9 v(S-IA x T- IC) v(x,T- IC)J-L(dx)

= f

JS-IA

= = where

SJ-L = J-t 0

r

v(Sx, C) p,(dx) =

JS-IA

L

rvex, C) 8p,(dx)

JA

vex, C) p,(dx) = p, ® v(A x C),

S-I. This is enough to deduce the conclusion.

A type of converse of Proposition 2 is obtained as follows:

Proposition 3. Assume that S is invertible and q) has a countable generator q)o, and let J-L E P(X) and v E C(X, Y). If tL Q9 v E Ps(X x Y), then J-L E Ps(X) and u E Cs(X, Y) J-L-a.e. in the sense that there is some stationary VI E Cs(X, Y) such that v(x,·) = Vl(X,·) u-a.e.x E X.

Proof. Since J-t Q9 v is stationary, we have for A E X and CEq) p, ® v(A

x C) =

L

vex, C) p,(dx)

x T)-I(A x C)) = J-L Q9 v(S-1 A x T- IC). = J-L Q9 v((S

If C

= Y,

then (1.3) reduces to AEX,

Le., J-L E Ps(X). Using this, (1.3) can be rewritten as p, e v(S-l A x T-1C) =

=

=

r

JS-I A

L L

vex, T-1C) p,(dx)

V(S-lx, T-1C) 8p,(dx) v(S-lX, T-1C) p,(dx)

(1.3)

125

3.1. Information channels

for every A E X and C E~. Hence v(x, C) = v(S-lx, T-1C) u-a.e.x for every C E~. Let ~o = {C1,C2 , ... } and X n = {x EX: v(x,Cn ) = v(S-lx,T-1Cn ) } for n 2: 1. Then X*

00

= n=l n Xn

E

..

v(x, .) = v(S-'-lx, T- 1.) on t/ is stationary u-a.e.

~o

X is such that /-L(X*)

= 1 and,

for every x E X*,

and hence on ~ since ~o is a generator of~. Thus

The following proposition may be viewed as an ergodic theorem for a stationary channel. Proposition 4. If v E Cs(X, Y) and f-L E Ps(X), then for every E, F E X ® ~ the following limit exists: n-l

lim ! "v(x, [(8 n4-OO

n L...t

X

T)-k En Flo:) u-a.e.x.

k=O ~

In particular, for every C, D E

the following limit exists:

n-l

lim ! n4-OO n

L

v(x, T-kC n D) u-a.e.x,

k=O

Proof. Let J = {E E X ® ~ : (S x T)-l E = E} and E, F E X ® ~ be arbitrary. Since f-L ® u E Ps(X x Y) we have by the Pointwise Ergodic Theorem that n-l

lim !LIE((8xT)k(X,y)) = lim [(S@T)n1E](x,y) n4-OO n n4-OO k=O

= EJL®v(lEIJ)(x, y) f-L ® u-a.e.,

(1.4)

where [(8®T)f](x,y) = f(Sx,Ty) for f E B(XxY), x EX, Y E Yand (8®T)n = ~

n-l

l:: (8

k=O

® T)k. Let Z

= {(x,y)

E X x Y : Limit in (1.4) exists for (x,y)}. Then

Ix

f-L® v(Z) = v(x, Zx) f-L(dx) = 1, so that v(x, Zx) = 1 u-a.e.x, Hence the following limit exists u-a.e.x by the Bounded Convergence Theorem: n-l

lim! "v(x, [(8 n4-OO

n L...t

X

T)-k En Flo:)

k=O n-l

= lim (IF(x,y)! LIE((8xT)k(x,y))II(x,dy) n4-OO }y n k=O

Chapter III: Information Channels

126

=[

IF(x, y)EI'®v(lEIJ)(x, y) v(x, dy).

We need a tensor product Banach space with the least crossnorm A (cf. Schatten [1]). Let E and F be two Banach spaces and £. 0 F be the algebraic tensor product. n

For
= E 4>j 0 'ljJj

the least crossnorm A(
j=l

A(
= sup {/(4)* 0'ljJ*)(*

E

e:»: E F*, /14>*/1::; 1, II'ljJ*/I::; I},

where (4)* 0 'ljJ*) (4> 0 'ljJ) = 4>* (4) )'ljJ* ('ljJ). The completion of £. 0 F w.r. t. A is denoted by £.®.xF and called the injective tensor product Banach space of E and F. Suppose that X, Yare compact Hausdorff spaces. If E = C(X), then C(X) 0 F is identified with the function space consisting of F-valued functions on X: xEX.

Moreover, it is true that

C(X) ®.x F

= C(X ;F),

the Banach space of all F-valued (norm) continuous functions on X with the sup norm. In particular, we have that

C(X) ®.x C(Y) =C(X x Y), where we identify (a0b)(x, y)

= a(x)b(y) for a E C(X), b E C(Y)

Proposition 5. Consider alphabetmessage spaces X v : X x ~ --+ [0,1]. Then: (1) (c5) =>(c5') {::} (c5"). (2) (c5'), (c6) =>(c2).

and x E X, Y E Y.

= X~, Y = YoZ and a junction

Proof. (1) (c5)=>(c5') is clear and (c5'){::}(c5") follows from the fact that each message [Yi ... Yj] is a clopen set and the set rot of all messages forms a topological basis for Y, and the fact that C(X x Y) = C(X) ®.x C(Y), where the algebraic tensor product C(X) 0 C(Y) is dense in it as is noted above. (2) Let b E C(Y). Then there exists a sequence {Sn}~=l of simple functions of kn

the form Sn =

E

O!n,k 1A n ,k ' An,k Erot, n

~ 1 such that

k=l

sup /Sn(Y) - b(y)/ yEY

= /lsn - b/l --+ 0

as n --+

00.

127

3.1. Information channels

This implies that

[ Sn(y) v(x, dy) ----t [b(Y) v(x, dy) uniformly in x,

Jy

Jy

Since sn(Y) v(·, dy) E C(X) by (c5'), we have b(y) v(·, dy) E C(X). Now (c6) implies that the RN-derivative k(x,·) == v~(d:») exists in L 1 (y , rJ) for every fixed x E X and

v(x, C)

=

fc

k(x, y) 'T/(dy) ,

x E X, C E~.

Let C E ~ be given and find a sequence {bn}~=l ~ C(Y) such that

[ Ibn(y) - lc(y)1 'T/(dy) ----t O. Then we have that, for x E X,

v(x, C) = [ lc(y)k(x, y) 'T/(dy)

r lim r bn(y) v(x, dy) l-

== n-+oo lim bn(y)k(x, y) rJ(dy) }y ==

n-+oo

by the Dominated Convergence Theorem. Therefore, v(·, C) is a limit of a sequence of continuous functions, and hence is measurable on X. That is, (c2) holds. For a channel v E C(X, Y) define an operator K; : B(Y)

(Kvb)(x) = [b(Y) v(x, dy),

~

B(X) by

bE B(Y),

(1.5)

where B(X), B(Y} are spaces of all bounded measurable functions on X, Y, respectively. In fact, if b ==

n

E

G-j lej' a simple function on Y, then obviously Kvb E B (X). j=l For a general b E B (Y) we can choose a sequence {bn } of simple functions converging to b pointwise with Ibnl :::; Ibl for n 2: 1. Then the Dominated Convergence Theorem applies to conclude that Kvb E B(X). Now we can characterize continuous channels between alphabet message spaces using the operator K v .

Proposition 6. Consider alphabet message spaces X == X~ and Y == JlQz and a stationary channel t/ E Cs (X, Y). Then the following conditions are equivalent:

Chapter III: Information Channels

128

(1) V is (2) The (3) The M(Y) and weak*.

continuous, i.e., t/ satisfies (c5'); operator K v defined by (1.5) is a linear operator from C(Y) into C(X); dual operator K~ of K; is from C(X)* into C(Y)*, i.e., K~ : M(X) -+ is sequentially weak * continuous, i.e., ~n -+ ~ weak* implies K~en -+ K~~

(4) v(-, k~l[Y}:) ... Y;~)]) E C(X) for n

~

1 and messages [Yik ... Yik]


(1 ::;

k:::; n). Proof. (4)

=* (1) is obvious. We shall show (1) =* (4) =* (2) =* (3) =* (1).

(1) =* (4). We first prove the case where n == 2. If i1 < i 2 , then

U

[y~~) ... YJ~)] n [Y~~) ... YJ~)] ==

[Y~~)·· . YJ~)Yjl +1 ... Yi2-1Y~~) ... YJ~)]

(1.6)

YkEYo jl
and hence

v(., [Y~~) ... yj~)]n[yj~) ... yj~)]) ==

L

v(., [y~~) ... yj~)Yjl+1 ... Yi2-1Y~~) ... yj~)]).

YkEYo jl
(1.7) Since each terrn on the RHS of (1.7) is continuous by (1), the LHS is also continuous. If i 1 :::; i 2 :::; i, :::; i2, then LHS of (1.6)

== [yi~) ... yi~~l] n [yi~) ... yj~)] n [y~~) ... yj~)] n [yj~~l ... yj~)] 2 1 0 if Yk ) =1= Yk ) for some i 2 < k - { [y~l) ... y~1)y~2) ... y~2)] if y(l) == y(2) for i < k < J.. ~1 31 31 + 1 32 k k 2 1

< i,

Thus the LHS of (1.7) is continuous by (1). The other cases are similarly proved. For a general n ~ 2, we can establish (4) by mathematical induction. (4) =* (2). Let Q:o be the algebra generated by the functions of the form l[Yi".Yj] (i :::; i). Then (4) implies that KlIQ: o ~ C(X). Since each b E C(Y) is' uniformly approximated by a sequence {bn } C Q:o, we have Kvb E C(X) because IIKv(b - bn)11 :::; lib - bnll -+ o. Therefore, K; is a linear operator from C(Y) into C(X). (2) =* (3). Suppose that {~n} ~ M(X) == C(X)* is a sequence converging weak* to E M(X), Le.,

e

(a,~n) =

Lad~n L -+

adf, =

(a,~),

a E C(X),

where (-, -) is the duality pair for C(X) and M(X). Then for b E C(Y) we have that

129

3.2. Channel operators

since Kvb E C(X) by (2). This means that {K~~n} ~ M(Y) converges weak* to K~~. Therefore (3) holds. (3) => (1). For x E X we denote by 8x the Dirac measure at x, i.e., 8x(A) = 1 if x E A and = 0 if x tj. A for A E :t. Note that 8x E M(X) for x E X. Assume that {Xn}~=l ~ X converges to x, Le., d(x n , x) -+ 0, where the metric d is defined by (11.1.1). For a E C(X) one has

(a,8x n )

= a(x n ) -+ a(x) = (a,8x ) .

Thus, by (3) we see that for b E C(Y)

If, in particular, we take b = means (1) holds.

I[Yi'.'Yi]'

then the above shows that Kvb E C(X). That

3.2. Channel operators Baire functions. We

'~~l~"~~d'thi~"'ki~~r" of operators.

Let X, Y be a pair of compact Hausdorff spaces with Baire measurable transformations S, T, respectively. Consider the algebras B(X), B(Y) and B(X x Y) of all bounded Baire functions on X, Y and X x Y, respectively, where the product is a pointwise multiplication. These are C*-algebras with unit 1, and B(X) and B(Y) are regarded as *-subalgebras of B(X x Y) by the identification of a = a 0 1 and b = 1 0 b for a E B(X) and b E B(Y). Here the involution f M f* is defined by f*(w) = f(w) for f E B(o'),w E 0, and the norm is the sup-norm, where 0, = X, Y or X x Y. A linear 0 erator A from B X x Y onto B X is said to be an avera in rmerator if for t, 9 E B (X x Y) (al) Al

= 1, A(fAg) = (Af)(Ag);

(a2) f ?:. 0 => Af ?:. 0, which imply: (aI') A is idempotent, i.e., A 2

= A;

(a2') IIAII = 1. That is, A is a norm 1 projection from B(X x Y) onto B(X). An averaging operator A satisfies that Af* = (Af)*,

(Af)*(Af) ~ A(f*f),

f

E B(X

x Y).

Chapter III: Information Channels

130

In fact, the first follows from (a2) and the second is derived using the first, (a1) and (a2): Af)*(f - Af)) = A(f*f) - (Af)*(Af)· Let us denote by B(X) such that (a3) fn

+0

=}

Afn

+o. stauonaru if

a~A

= A(S I8l T).

~~~~ldenotesthe

set of all stationary operators in A(X, Y). Note that A(X, Y) X, Y) are convex sets. We need another set of operators. Let Y) ·denote the set of all bounded linear operators K : B(Y) -+ B(X) such tha an

s

(k1) K1 = 1, Kb (k2) i;

+0

=}

~

Kb n

0 if b ~ 0;

+o.

K E J((X, Y) is said to be stationary if

(k3) KT = SK. J(s(X, Y) denotes the set of all stationary K E J((X, Y). Also note that J((X, Y) and J(s(X, Y) are convex sets. Let u E C(X, Y) be an abstract channel and define operators A : B(X x Y) -+ B(X) and K : B(Y) -+ B(X) respectively by

(Af)(x) = [f(x,y)v(x,dY), (Kb)(x) = [b(Y) v(x,dy),

f

E

B(X x Y),

(2.1)

b E B(Y).

(2.2)

A and K are called channel operators associated with v and sometimes denoted by A v and K v, respectively. Note that A v (l 0 b) = Kvb for b E B(Y), where (a 0 b)(x, y) = a(x)b(y) for a E B(X), b E B(Y) and x E X, Y E Y. First we establish a one-to-one, onto and affine correspondence between C(X, Y) and J((X, Y) as follows, since they are convex:

Theorem 1. There exists a one-to-one, onto and affine correspondence v +-t K between C(X, Y) and J((X, Y) (or Cs(X, Y) and J(s(X, Y)) given by (2.2). Proof. Let v E C(X, Y) and define K : B(Y) -+ B(X)· by (2.2). Then, K satisfies (k1) by (c1) and (c2). (k2) follows from the Monotone Convergence Theorem.

3.2. Channel operators

131

Conversely, let K E IC(X,Y). For each fixed x E X, Px defined by

Px(b) = (Kb)(x),

b E C(Y),

is a positive linear functional of norm 1 on C(Y) by (kl). Hence, there exists uniquely a probability measure V x E P(Y) such that

Px(b)

= [b(Y) IIx(dy),

b E C(Y).

(2.3)

Let 8 1 be the set of all b E B(Y) for which (2.3) holds. Then C(Y) ~ 8 1 and 8 1 is a monotone class by (k2). Thus 8 1 = B(Y). If we take b = Ie for C E ~, then (K1e)(x) = le(Y) vx(dy) = vx(C), x E X and V(.)(C) E B(X). Letting v(x, C) = vx(C) for x E X and C E ~, we see that (cl) and (c2) are satisfied. Thus we have established a one-to-one, onto correspondence v +-+ K between C(X, Y) and IC(X, Y), which is clearly affine. Moreover, if v E Cs(X, Y), then for x E X and b E B(Y) it holds that

Jy

(KTb)(x)

= [K(Tb)] (x) = [b(TY) lI(x, dy) = [b(y)lI(x,dT- 1y) = [b(y)II(Sx,dY) = (Kb)(Sx) = (SKb)(x),

i.e., K E ICs(X, Y). If, conversely, K E ICs(X, Y), then considering b = Ie for C E ~ we see that v(x, T- 1C) = v(Sx, C), x E X by virtue of (k3), i.e., t/ E Cs(X, Y). Therefore, the correspondence v +-+ K is one-to-one, onto and affine between Cs(X, Y) and ICs(X, Y). Using Theorem 1, we can also obtain a one-to-one correspondence between

C(X, Y) and A(X, Y) as follows: Theorem 2. There exists a one-to-one, onto and affine correspondence u +-+ A between C(X, Y) and A(X, Y) (or Cs(X, Y) and As(X, Y)) given by (2.1). Proof. Let v E C(X, Y) be given and define an operator A by (2.1). Al Observe that for t, 9 E B(X x Y) and x E X

[A(j Ag)] (x)

= [(j Ag)(x, y) lI(x, dy) = [f(x, y)(Ag)(x) lI(x, dy)

= 1 is clear.

132

Chapter III: Information Channels

= [f(X, y) vex, dy)(Ag) (x) = (Aj)(x)(Ag)(x).

Hence A satisfies (al ), (a2) is rather obvious and (a3) follows from the Monotone Convergence Theorem. Thus A E A(X, X). If v E Cs(X, Y), i.e., v is stationary, then for j E B(X x Y) and x E X one has

(SAj)(x) = (Aj)(Sx) = [f(Sx, y) v(Sx, dy)

= [f(Sx, y) vex, dT-1y) = [f(Sx, Ty) vex, dy)

= [[(SI2lT)f](x,y)v(x,d y) = [A(SI2lT)f](x). Consequently, (a4) is satisfied and A E As(X, Y). Conversely, let A E A(X, Y) and

(Kb)(x) = A(10 bJ(x),

b E B(Y), x E X.

Then, it is easily seen that K E JC(X, Y). By Theorem 1 there exists a unique channel u E C(X, Y) such that (2.2) holds. Thus, for al, ... ,an E B(X) and bl , ... ,b n E B(Y) we have n

E ak(x)(Kbk)(X) k=1

=

t

ak(x)

k=1

= f

}y

f

}y

bk(y) v(x,dy)

(t ak 8 bk) (x, y) v(x, dy).

(2.4)

k=1

Since the algebraic tensor product space B(X) 0 B(Y) is a monotone class, (2.4) irnplies (2.1). If A E As(X, Y), then for x E X and C E ~ it holds that

v(Sx, C) = [lc(y)V(Sx,dY) = [(1 8 lc)(Sx,y) v(Sx,dy) = [SA(10 Ie)] (x) = [A(S Q9 T)(101e)] (x)

= [l(SX)lc(TY) vex, dy) = [lc(Y) vex, dT-1y) = v(x, T-IC).

133

3.2. Channel operators

Therefore, u E Cs(X,Y).

Remark 3. Let v +-+ K +-+ A, where u E C(X, Y), K E JC(X, Y) and A E A(X, Y). If u is continuous, i.e., v satisfies (c5"), then we can see that K : C(Y) -t C(X) and A : C(X x Y) -t C(X) are positive bounded linear operators of norm 1. Definition 4. Let P be a subset of P(X). (1) to be identical (mod VI(X,·) = V2(X,·) u-a.e.x E X

denoted

VI

== V:

for every J-t E P. In this case, we write VI(X,·) = V2(X,·) P-a.e.x. (2) Two 0 erators K 1 , K 2 E JC(X, Y) are said to be identical mod P, denoted K I == K 2 (mod P), if

for every b E B(Y) and J-t E P. In this case, we write Kib = K 2b P-a.e. (3) Two operators AI, A 2 E A(X, Y) are said to be identical mod P, denoted Al == A 2 (mod P), if

f E C(X x Y), J-t E P, or, equivalently,

J-tEP, where Ai : B(X)* -t B(X x Y)* is the adjoint operator of Ai (i = 1,2). We need the following lernma and proposition.

Lemma 5. Let v E C(X, Y) and J-t E P(X). Then the following holds:

L

(Kb)(x) p,(dx)

L

=

Li

(Af)(x) p,(dx)

b(y) v(x, dy)p,(dx)

= =

Li fro (

=

i

b(y) p,v(dy) ,

(2.5)

f(x, y) v(x, dy)p,(dx) f(x, y) J-t &J v(dx, dy)

(2.6)

JXXY

for b E B(Y) and f E B(X x Y), where J-tV and J-t &J u are output source and compound source defined by (1.1) and (1.2), respectively.

Chapter III: Information Channels

134

Proof. (2.5) and (2.6) can be verified first for simple functions and then for general bounded Baire functions by approximation. Corollary 6. (1) Let VI, V2 E C(X, Y) and P ~ P(X). If VI(X,') «: V2(X,·) Pa.e.x, then MVI «MV2 and M ® VI « M ® V2 for every M E P. (2) If V ++ K ++ A for V E C(X, Y), K E K(X, Y) and A E A(X, Y), then K*M MV and A*M J-t®V for J-L E P(..J(). Proposition 7. Let P ~ P(X) and suppose that Vi E C(X, Y), K, E K(X, Y) and Ai E A(X, Y) (i = 1,2) correspond to each other, i.e., Vi ++ K, ++ Ai for i = 1,2. Then the following statements are equivalent: (1) VI == V2 (modP). (2) K I == K 2 (rnod P). (3) Al == A 2 (mod P). (4) AiJ-L = A 2J-L, i.e., J-L ® VI = J-L ® V2 for every J-L E P.

Proof. We have the following two-sided implications: VI

== V2 (mod P) ¢::::::}

{==}

VI(X,·)

Li L

V2(X,') P-a.e.x

a(x)b(y) vl(x,dy)p,(dx) =

Li

a(x)b(y) v2(x,dy)p,(dx)

for a E C(X), b E C(Y) and J-L E P

{==}

A 1(a 0 b)(x) p,(dx) =

L

(2.7)

A 2(a 0 b)(x) p,(dx)

for a E C(X), b E C(Y) and J-L E P ¢::::::}

¢::::::}

J-L(AI(a 0 b)) = J-L(A 2(a 0 b)) for a E C(X), b E C(Y) and J-L E P Al == A 2 (mod P),

since C(X x Y) = C(X) ®A C(Y). This implies (1) {:} (3). Moreover, we have (2.8)

¢::::::}

J-L(aKIb) = J-L(aK 2b) for a E C(X), b E C(Y) and M E P Kib = K 2b P-a.e. for b E C(Y)

¢::::::}

K I == K 2 (mod P),

¢::::::}

implying (2) {:} (3), and

(2.7) ¢::::::}

J"r a(x)b(y) M® JXXY

VI

(dx, dy)

=

J"rJXXY a(x)b(y) J-L ® v2(dx, dy)

for a E C(X), b E C(Y) and J-L E P ¢::::::} M

®

VI

= M ® V2 for

M

E

P

(2.8)

135

3.2. Channel operators

by Lemma 5, implying (1) {:} (4). The important cases are those where P == Ps(X) and P == Pse(X). Definition 8. Ps(X) is said to be complete for ergodicity provided that if A E X and J-L E Ps(X) are such that J-L(A) > 0, then there exists an ergodic source 'TJ E Pse(X) such that 'TJ(A) > o. As was remarked before (cf. Section 2.2), if S is a continuous transformation on X, then S is measurable, Ps(X) -=I 0 and Pse(X) -=I 0. Proposition 9. Suppose that S is a continuous transformation on the compact Hausdorff space X into X. Then, Ps(X) is complete for ergodicity.

Proof. Suppose that J-L(A) > 0 for some j-t E Ps(X) and A E X. Since J-L is regular, there is a compact set C E X such that C ~ A and J-L( C) > O. Then we can choose a sequence {fn}~=l ~ C(X) such that t« t 1e as n -t 00 since X is compact and Hausdorff. If 0 ::; a ::; 1, then the set Pa == {'TJ E Ps(X) : 'TJ(C) 2: a} is a weak* compact convex set, since

r: == n {'TJ E Ps(X): 'TJ(fn) 2: a} 00

n=l

and Ps(X) is so. Let ao == sup {a : P« -=I 0}. Then, ao 2: j-t(C) > 0 and Pao is a nonempty weak* compact convex set. Hence, there exists some 'flo E ex P ao such that 'TJo(C) == ao > O. We claim that 'flo E exPs(X) == Pse(X). If this is not the case, there exist J-Ll, J-L2 E Ps(X) and (3 E (0,1) such that 'TJo == {3J-Ll + (1 - (3)J-L2 and J-Ll -=I J-L2· Note that by the definition of ao we have J-Ll (C), J-L2 (C) ::; ao. Since ao == 'TJo(C) == (3J-Ll(C) + (1 - (3)J-L2(C), we must have J-Ll(C) == J-L2(C) == ao and hence J-Ll, J-L2 E Pao' This is a contradiction to the extremality of 'flo in Pao' Thus 'TJo E Pse(X). For this 'TJo it holds that 'TJo(A) 2: 'TJo(C) > O. Proposition 10. Suppose that Ps(X) is complete for ergodicity and let Cs (X, Y). Then the following conditions are equivalent:

(1) Vl == V2 (modPs(X)). (2) Vl == V2 (modPse(X)). (3) Vl(X, Ex) == V2(X, Ex) Ps(X)-a.e.xfor every E E XQ9~. (4) vl(x,Ex) == V2(X, Ex) Pse(X)-a.e.xfor every E E XQ9~. (5) J-L Q9 Vl == J-L Q9 V2 for every J-L E P; (X). (6) J-L Q9 Vl == J-L Q9 V2 for every J-L E Pse(X). Proof. (1)

=}

(2), (3)

=}

(4), (5)

=}

(6), (1) {:} (5) and (2) {:} (6) are clear.

Vb

V2

E

Chapter III: Information Channels

136

(1) and (4) =} (3) follow from the completeness for ergodicity of Ps(X). (4). Assume (4) is not true. Then there is some E E X ® q) and some M E Pse(X) such that M({X EX: VI(X, Ex) i= v2(x,Ex)}) > O. We may suppose M({X EX: vI(x,Ex) > v2(x,Ex)}) > O. Then (2) (6)

=}

=}

L.

V

l (x,E x ) P,(dX) >

L.

V

2(x,E x ) P,(dX),

i = 1,2,

it holds that

M® VI (E n (A x which implies that M®

VI

i= M® V2,

Y)) > M® V2 (E n (A x Y)), a contradiction to (6).

To close this section we characterize continuous channels once more in terrns of channel operators, which is an extension of Proposition 1.6.

Proposition 11. Let V E C(X, Y) be a channel and K E JC(X, Y) and A E A(X, Y) be the corresponding channel operators. Then the following conditions are equivalent: (1) v is continuous, i.e., u satisfies (c5"). (2) K : C(Y) -+ C(X) is a linear operator. (3) A : C(X x Y) -+ C(X) is a linear operator. (4) K* : M(X) -+ M(Y) is sequentially weak* continuous. (5) A* : M(X) -+ M(X x Y) is sequentially weak* continuous. Proof. This follows from Proposition 1.6, Remark 3 and Corollary 6. The results obtained in this section will be applied to characterize ergodicity of stationary channels in Section 3.4 and to discuss channel capacity in Section 3.7.

3.3. Mixing channels Finite dependence was defined for channels between alphabet message spaces. As a generalization, asymptotic independence (or strong mixing) and weak mixing are introduced, which are similar to those for stationary information sources. Also ergodicity and semiergodicity are defined for stationary channels and relations among these notions are clarified.

137

3.3. Mixing channels

Consider a pair of abstract measurable spaces (X, X) and (Y,~) with measurable transformations Sand T, respectively. We begin with a notion of ergodicity.

Definition 1. A

n+""'+'i'"\.·.....~......."If ..

if

"".L............L......."".L

i.e., if a stationary ergo_d_i.__~_rce is the input, then the compound source rnust also denotes the set of all stationary ergodic channels. be stationary ergodic. To obtain conditions for ergodicity we first consider channels between alphabet message spaces X = X~ and Y = Yl~, where X o and Yo are finite sets. We have the following with the notation in Section 3.1.

Proposition 2. If

t/

E Cs(X~, YoZ ) is a memoryless channel, then v is ergodic.

Proof. Let P = (p(bla))a,b be the channel matrix of v (cf. Definition 1.1, (c4)). Let J-t E Pse(X) and A = [xo = a],B = [xo = a'] C X and C = [Yo = b],D = [Yo = b'] c Y. Then one has 1

n-l

- LJ-tQ9v((SxT)-k(AxC)n(BxD)) n k=O 1

n-l

= - LJ-tQ9v((S-kAnB) x (T-kCnD)) n

k=O n-l

=

L

1 /L @ V([Xk n k=O 1

= a, Xo = a'] x

[Yk

= b, Yo = b'D

n-l

= -n L J-t([Xk = a, Xo = a'])p(bla)p(b'la') k=O

---+ J-t([xo = a])J-t([xo = a'])p(bla)p(b'la'),

= J-t Q9 v(A

by ergodicity of u,

x C)J-t Q9 v(B x D).

It is not hard to verify that the above holds for general messages A, B c X~ and C, D c Yoz . Hence J-t Q9 u is ergodic by Theorem 11.3.2. Therefore v is ergodic.

Definition 3. A channel v E C(XZ, J:':Z) is said to be finitely dependent or maeuenaeni if n

(c8) There exists a positive integer mEN such that for any n, r, s, tEN with r ~ S ~ t and s- r > m it holds that

~

Chapter III: Information Channels

138

for every x E X and every message Cn,r = [Yn ... Yr], Cs,t

= [Ys ... ytl c Y.

Then we record the following theorem, which immediately follows from Theorem 7 below.

Theorem 4. If a stationary channel v E Cs(X~, yoZ) is finitely dependent, then v is ergodic. Example 5. Consider alphabet message spaces X = X~ and Y = yoZ with messages IDly in Y. Let m 2: 1 be an integer and for each (Xl,... ,Xm ) E X a probability distribution p((XI, ... ,xm)l· ) on Yom is given. Define Vo : X x IDly ~ [0,1] by

o

t-s vo{x,[Ys···Yt])

= IIp((Xs+k-m, ...

,Xs+k-I)I(Ys+k-m, ... ,Ys+k-l))

k=O and extend Vo to v : X X ~ ~ [0,1], so that v becomes a channel. Then, this u is stationary and finitely dependent, and has a finite mernory. This is easily verified and the proof is left to the reader. Finite dependence can be generalized as follows. Consider a general pair of abstract measurable spaces (X, X) and (Y,~) with measurable transformations Sand T as before. First recall that a stationary source JL E P; (X) is said to be strongly mixing (8M) if A,BEX,

and to be weakly mixing (WM) if n-l

.!. L n-too n

IJL(S-kA n B) - JL(A)JL(B)

lim

I = 0,

A,BEX.

k=O

Also recall that JL E Ps (X) is ergodic iff n-l

! ~ JL(S-kA n B) = JL(A)JL(B), n-too n Z:: lim

A,BEX

k=O

(cf. Theorem 11.3.2). It was noted that for stationary sources strong mixing :::} weak mixing :::} ergodicity. As an analogy to these formulations we have the following definition.

I

J

139

9.9. Mixing channels

Definition 6. A channel v E C(X, Y) is said to be asymptotically independent or strongly "mixing (8M) if (c9) For every C,D E

~

lim {vex, T-nc n D) - v{x, T-nC)v(x, D)} = 0 Ps{X)-a.e.x.,

n-+oo

to be weakly mixing (WM) if (c10) For every C,D E

~

1 n-l

lim

n-+oo n

E Iv{x, T-kC n D) -

vex, T-kC}v{x, D)I = 0 Ps(X)-a.e.x,

k=O

and to be semiergodic (8E) if

(ell) For every C,D E 1

lim -

n-+oon

~

n-l

E {vex, T-kC n D) - vex, T-kC)v(x, D)} = 0

Ps(X)-a.e.x.

k=O

Note that if (X, X, S) is complete for ergodicity (cf. the previous section), then in (c9), (c10) and (3.1) below we can replace Ps(X)-a.e. by Pse(X)-a.e. Consider the following condition that looks slightly stronger than (ell): for every C,D E ~ n-l

n-l

lim .!. "v(x, T-kC n D) = lim.!. "v{x, T-kC)v(x, D) Ps{X)-a.e.x. n-+oo n Z:: n-+oo n Z:: k=O k=O

(3.1)

In view of Proposition 1.4, the LHS of (3.1) exists for a stationary channel. Hence (ell) and (3.1) are equivalent for stationary channels. If (3.1) holds for every x E X, then the existence of the RHS means that v(x,') E Pa(Y) and (3.1) itself means that v(x,') E Pae(Y) for every x E X (cf. Theorem 11.4.12). Also we have for (stationary) channels strong mixing ::::} weak mixing ::::} ergodicity ::::} semiergodicity, where the first implication is obvious, the second is seen in Theorem 7 below, and the last will be verified later. Actually, (3.1) is a necessary (and not sufficient) condition for ergodicity of a stationary channel (cf. Theorem 13 and Theorem 4.2). Now we can give a basic result regarding these concepts.

Theorem 7. Let

t/

E

Cs(X, Y) and f-t E Ps(X).

140

Chapter III: Information Channels

(1) If J-L strongly or (2) If J-L (3) If J-L

is ergodic and v is weakly mixing, then J-L ® v is ergodic. Hence, every weakly mixing stationary channel is ergodic. and v are weakly mixing, then J-L ® v is also weakly mixing. and v are strongly mixing, then J-L ® v is also strongly mixing.

Proof. (1) Let A, B E X and C, D E q). Then we have that n-l

r

.!.E n

v(x, T-kC)v(x, D) p,(dx)

k=O}s-kAnB

1 = .!. E 1 n-l

= .!. E

v(x, T-kC)lA(SkX)V(X, D)lB(x) p,(dx)

n k=O X n-l

v(Sk x, C)lA(Skx)v(x, D)lB(x) p,(dx),

n k=O

x

(3.2)

since v is stationary,

L

-+

vex, C)lA(x) p,(dx)

L

vex, D)lB(x) p,(dx),

since J-L is stationary and ergodic, == J-L ® v(A x C)J-L ® v(B x D).

On the other hand, since 1

t/

n-l

is weakly mixing, it holds that

-n E Iv(x, T-kC n D) -

I

v(x, T-kC)v(x, D) ~ 0 u-a.e.x E X

k=O

as n -+

00.

By the Bounded Convergence Theorem we have

n-l

.!.n E

r

I

Iv(x, T-kC n D) - vex, T-kC)v(x, D) p,(dx)

k=O}S-kAnB

1.!. E I n-l

< as n -+

00.

(3.3)

k=O

Cornbining (3.2) and (3.3), we get

I~ ~ p,0 v((S X T)-k(A x C) n (B x D)) =

I

vex, T-kC n D) - vex, T-kC)v(x, D) p,(dx) -+ 0

x n

p,0 v(A x C)p,0 v(B x D)I

I~ ~ p,0 v((S-k A n B) x (T-kC n D)) -

p,0 v(A x C)p, 0 v(B x D)I

141

3.3. Mixing channels

==

I.!.n ~ [ vex, T-kC n D) p,(dx) k=O}S-kAnB

p, 0 v(A

X

C)p, 0 v(B

X

D)

I

n-l

: :; .!. L .[

Iv(x, T-kC n D) - vex, T-kC)v(x, D)I p,(dx)

n k=O}S-kAnB

+ I.!.

E[

v(x,T-kC)v(x,D) p,(dx) - p,0v(A

X

C)p,0v(B

X

D)!

n k=O}S-k AnB

-+ 0 as n -+

00.

Thus, by Theroern 11.3.2 I-" ® v is ergodic. (2) and (3) can be verified in a similar manner as above. In the above theorem we obtained sufficient conditions for ergodicity of a stationary channel, namely, strong mixing and weak mixing. Key expressions there are (3.2) and (3.3). In particular, if'f/ E Ps(Y) is WM, then the constant channel Vn is stationary ergodic. For, if I-" E Pse(X), then I-" ® v"., == I-" x 'f/ is ergodic by Theorem 11.3.10. In the rest of this section, we consider semiergodic channels together with averages of channels. We shall show that there exists a semiergodic channel for which, if a weak mixing source is input, then the compound source is not ergodic. This irnplies that semiergodicity of a channel is weaker than ergodicity.

(c12) v*(Sx,C)

(c13)

L

== v*(x;C) for x

vex,C) p,(dx)

=

L

E X and C E~;

v*(x, C) p,(dx) for A

E

J, C E

!D

and p, E Ps(X).

(c12) means that for each C E ~ the function v*(·,C) is S-invariant, while (c13) means that v*(·, C) == EM (v(., C)13)(.) u-a.e. (3.4) for C E ~ and I-" E Ps(X), where E M(·13) is the conditional expectation w.r.t. 3 under the measure 1-". A sufficient condition for the existence of the average of a given stationary channel will be obtained (cf. Theorem 13). We need a series of lemmas.

Lemma 9. If v

E

Cs(X, Y) has an average u", then for C v*(·, C)

= n-too lim .!. n

E ~

n-l

L v(Sk., C)

k=O

Ps(X)-a.e.

(3.5)

142

Chapter III: Information Channels

Proof. In fact, this follows from (3.4) and the Pointwise Ergodic Theorem. Lemma 10. Let v E Cs(X, Y) with an average t/", If J-L E Pse(X) is such that J-L ® v =f. J-L x (J-Lv), the product measure, then J-L ® v =f. J-L ® u" . Proof. Observe that for A E X and C E

JL ® v*(A x C)

=

i

~

v*(x, C) JL(dx) n-I

= [ lim .! L v(Skx, C) JL(dx), JA n-too n k=O

by (3.5),

n-I

lim .! L [ v(Skx, C)lA(X) JL(dx) n-too n k=O Jx

L

v(x, C) JL(dx)

L

lA(X) JL(dx),

since JL is ergodic,

= J-L(A)J-Lv(C).

Thus, J-L ® v

=f. J-L ® t/"

by assumption.

v;,

Lemma 11. Let VI, V2 E Cs(X, Y) be semiergodic with averages vi, respectively. If vi == v; (modPs(X)) (cf. Definition 2.4), then Vo = !(VI +V2) is also semiergodic. Proof. Since

vi (i = 1,2) exists, we see that for C, D E ~ n-I

lim.! LVi(X,T-kC)Vi(X,D) n-too n k=O

= v;(x,C)vi(x,D) Ps(X)-a.e.x

by Lemma 9 and hence, by semiergodicity of Vi, n-I

lim 1 L Vi(X, T-kC n D) n-too n k=O

= V;(x, C)Vi(X, D) Ps(X)-a.e.x.

Consequently, we have that 1 n-I

lim -

n-toon

L

k=O

{vo(x, T-kC n D) - vo(x, T-kC)vo(x, D)}

1 [1 n-I

= n-too lim - '"" n L..J k=O

-2 VI (x, T-kC n D)

1

+ -V2(X, T-kC n D) 2

(3.6)

143

3.3. Mixing channels

since

VI, V2

are stationary,

= ~{vnx, C)VI(X, D) + vi(x, C)V2(X, D)}

- ~ {vi (x, C) + vi(x, C)} {VI (x, D) + V2(X, D)}, by (3.6), stationarity of

VI, V2,

and the definition of the average,

= ~{vi(x,C) -vi(X,C)}{VI(X,D) -v2(x,D)} == 0

by

Ps(X)-a.e.x,

vi == v 2 (rnodPs(X)).

Lemma 12. If semiergodic.

V

E

Thus Vo is serniergodic.

Cs(X, Y) is semiergodic and has an average u", then u" is also

Proof. Let C, D E ~ and J.L E Ps(X). Then, n-I

n-I

lim 1 "'v*(x,T-kCnD) n-.+oo n L-J k=O

= n-.+oo lim.!.. "'EI'(v(.,T-kCnD)!J)(x) n L-J k=O

= }~~ EI' (~ ~ v(.,T-kC n D)IJ) (x) == E~(v*(.,

by (3.6) with

V

==

Vi

>

C)v(·, D)IJ)(x),

and (10) in Section 1.2, == v*(x,C)E~(v(.,D)IJ)(x) ==

t/"

(x, C)v* (x, D) n-I

= by (cI2), which yields that

t/"

lim .!..

n-.+oo

n

L v*(x, T-kC)v*(x, D)

u-a.e.x,

k=O

is semiergodic.

Now the existence of an average channel of a given stationary channel is considered under SOUle topological conditions.

Chapter III: Information Channels

144

Theorem 13. Let (X, X, S) be an abstract measurable space with a measurable transformation Sand Y be a compact metric space with the Borel a-algebra ~ and a homeomorphism T. Then every stationary channel v E Cs(X, Y) has an average v* E

Cs(X,Y).

Proof. Denote by Mt(Y) c M(Y) = C(Y)* the positive part of the unit sphere and by B(Y) the space of bounded Borel functions on Y. Let v E Cs(X, Y) be given. For each x E X define a functional V x on B(Y) by

v",(J) = [1(y)v(x,dY),

f

E

B(Y).

If we restrict V x to C(Y), then we see that V x E Mt (Y) for each x EX. Let 1) be a countable dense subspace of C (Y) with the scalar multiplication of rational complex numbers and let n-l

A",(J) = lim .!:. n-+oo

n

L VSk",(J),

fE1)

(3.7)

k=O

for each x E X. Since, for each f E C(Y), the function v(o)(f) is bounded and measurable on X, (3.7) is well-defined u-a.e. by the Pointwise Ergodic Theorem for each JL E Ps(X). Let

Xn

= {x EX:

(3.7) exists for all f E 1)}.

It is easily seen that X n E X, Xn is S-invariant, and J-L(Xn ) = 1 for JL E Ps(X) since 1) is countable. Note that, for each x E X, A x ( · ) is a positive bounded linear functional on 1) since for f E 1) IAx(f)1 =

~

I

~

lim .!:. VSk",(J) I n L....t

n-+oo

k=O

n-l

lim n-+oo

.!:. L n

( I/(y)1 v(Skx,dy) < 11111,

k=O}Y

so that Ax (.) can be uniquely extended to a positive bounded linear functional Ax (.) of norm 1 on C(Y). Let us examine some properties of Ax. Ax satisfies that

x E X n , f E C(Y)

(3.8)

since Asx(f) = Ax(f) for x E X n and f E 1). For each f E 1), A(o)(f) is Xmeasurable on Xn, which follows from (3.7). For a general f E C(Y) there exists a sequence {fn}~=l C 1) such that "fn - fll ~ 0, so that

3.3. Mixing channels

145

implying the measurability of A(o)(/) on X1). Moreover, for each x E X1), one can find a probability measure 'l7x on ~ such that

I

Ax(f) = [f(Y) 1Jx(dy),

E C(Y)

by the Riesz-Markov-Kakutani Theorem. One can also verify that rlx is T-invariant, i.e., '17 E Ps(Y) for x E X1), which follows from (3.7) and stationarity of t/. Consider the set

Eo = {f E B(Y) : [ f d1Jx is X-measurable on X:o}. Then we see that C(Y) ~ 8 0 and 8 0 is a monotone class, i.e., if {In}~=l ~ 8 0 and In .,t. I, then I E 8 0 , Hence one has 8 0 = B(Y). Denote by the same syrnbol Ax the functional extended to B(Y). Take any '17 E Ps(Y) and define u" by if x E X1)

v*(x, C) = { 'l7x(C) '17 (C)

if x E X

n

for C E~. We shall show that t/" is the desired average of t/, First note that u" is a stationary channel, u" E Cs(X, Y), by'l7x (x E X1»),'17 E Ps(Y) and (3.8). Now we verify (c12) and (c13) (Definition 8). If x E X1), then Sx E X1) and

v*(x, C) = 'l7sx(C) = 'l7x(C) = v*(x, C), and if x E X

n, then Sx E X n and v*(Sx, C)

= 'I7(C) = v*(x, C),

CE~,

so that (c12) is satisfied. To see (c13), let I-t E Ps(X) be fixed and observe that

L[

fey) vex, dy)p,(dx) =

for I E 11. For, if g(x)

= Jy I(y) v(x, dy)

gs(x) == lim (Sng)(x) n-+oo

and

L

L

[f(Y) v*(x, dy)p,(dx)

(x E X), then

= }y [ fey)

g(x) p,(dx) =

L

v*(x, dy) u-a.e.x

gs(x) p,(dx)

(3.9)

146

Chapter III: Information Channels

by the Pointwise Ergodic Theorem, which gives (3.9). Moreover, (3.9) holds for f E C(Y) since ~ is dense in C(Y). If G E X is S-invariant (GE J) with /-L(G) > 0, then the measure /-La defined by AEX,

is an S-invariant probability measure. We have that for CEq)

fa

v(x, C) p,(dx) = p,(G)

= p,(G)

=

fa

L L

v(x, C) p,a(dx) v*(x, C) p,a(dx)

v*(x, C) p,(dx).

If G E J is such that /-L(G) = 0, then the equality in (c13) also holds. Thus u" satisfies (c13). Therefore t/" is the desired average of u, The following theorem is our main result of this section. Theorem 14. There is a semiergodic channel Va and a strongly mixing input /-La for which the compound source /-La ® Va is not ergodic. Hence, a semiergodic channel is not ergodic. Proof. Consider an alphabet message space X = {O, l}Z with the Baire a-algebra X and the shift S. Let V E C(X, X) be a memoryless channel with the channel matrix p=

(I t)

(cf. Definition 1.1). Then one can directly verify that u is stationary and semiergodic. i/" by Theorem 13 and t/" is semiergodic by Lemma 12, so that

v has an average

1

1 *

Va = "2 v +"2 v

is also semiergodic by Lemma 11 since (z-") * = t/", Now let /-La be a (!' ~)-Bernoulli source (cf. Example II.1.2), which is strongly mixing (cf. Example 11.3.7). Then /-La ® Va is not a direct product measure, so that

by Lemma 10. Since 1

1.

/-La ® Va = "2/-La ® v + "2/-La ® u

*

147

3.4. Ergodic channels

is a proper convex combination, J-to Q9 Vo is not ergodic by Theorem 11.3.2. This completes the proof.

3.4. Ergodic channels In the previous section sufficient conditions for ergodicity were given. In this section, ergodicity of stationary channels is characterized by finding several necessary and sufficient conditions. We shall notice that many of these conditions are similar to those for a stationary source to be ergodic. Also we use channel operators to obtain equivalence conditions for ergodicity. In particular, we realize that there is a close relation between ergodicity and extremality. Let (X, X, S) and (Y,~, T) be a pair of abstract measurable spaces with measurable transformations Sand T, respectively. Recall that~~a",_~,~~,~,~~~~~~J:"e"e~~:~~~~ v is said to be if

(c7) J-t

E Pse(X)

=> J-t Q9 v

E Pse(X

x Y).

Definition 1. Let P be a subset of P(X). A stationary channel v E Cs(X, Y) is said to be extremal in Cs(X, Y) mod P, denoted v E exCs(X, Y) (mod P), if Vl, v2 E Cs(X, Y) and a E (0,1) are such that v == aVl + (1 - a)v2 (mod P), then Vl == V2 (mod P). The following theorem gives some necessary and sufficient conditions for ergodicity of a stationary channel, which are very similar to those for ergodicity of a stationary source (cf. Theorem II. 3.2) .

Theorem 2. For a stationary channel u E Cs(X, Y) the following conditions are equivalent: (1) v E Cse(X, Y), i.e., v is ergodic. (2) If E E XQ9~ is S x T-invariant, then v(x, Ex) = 0 or 1 Pse(X)-a.e.x. (3) There exists an ergodic channel Vl E Cse(X, Y) such that v(x,') ~ Vl(X,·) Pse(X)-a.e.x. (4) If a stationary channel Vo E Cs(X, Y) is such that vo(x, .) « v(x,·) Pse(X)a.e.z, then Vo == v (modPse(X)). (5) v E exCs(X, Y) (modPse(X)). (6) For E, F E X Q9 ~ and J-t E Pse(X) it holds that n-l

lim 1 n-+oo n

L v(x, [(8

k=O

n-l

X

T)-k En F]a:)

.!. L v(x, [(8 X T)-k E]a:)v(x,Fa:) n-+oo n

= lim

k=O

= J-t Q9 v(E)v(x, F x)

u-a.e.x,

Chapter III: Information Channels

148

(7) For A, B E X, C, D E

~

and J-t E Pse(X) it holds that

n-I

lim.!:. n-+oo n

L r.

{v(x, T-kC n D) - v(x, T-kC)v(x, D)} JL(dx)

= O.

k=OJS-k AnB

Proof. (1) =} (2). Let J-t E Pse(X). Then J-t ® v E Pse(X x Y) by (1). Hence, if E E X ® ~ is S x T-invariant, then J-t ® v(E) = 0 or 1, i.e.,

L

v(x, Ex) JL(dx)

=0

or 1.

This implies v(x, Ex) = 0 u-a.e.x or v(x, Ex) = 1 u-a.e.x, Thus (2) holds. (2) =} (1). Let E E X ® ~ be S x T-invariant, J-t E Pse(X) and

A o = {x EX: v(x, Ex)

= o},

Al = {x EX: v(x,Ex) = I}. Note that T-IEsx = Ex since

T- I Es x = =

{Y E Y : Ty E {z

E Y : (Sx, z) E E}}

{y

E

E

Y : (Sx, Ty)

= {y E Y: (x,y) E (S

E}

x T)-IE

E} = Ex.

(4.1)

Henee A o is S-invariant since

S-IA o = {x EX: Sx E A o} = {x EX: v(Sx, Es x) = n} = {x EX: v(x, T-IESx) = O}, because v is stationary,

= {x

EX: v(x, Ex)

= O} = A o.

Similarly, we see that Al is S-invariant. Thus, J-t(A o) = 0 or 1, and J-t(A I) sinee J-t is ergodic. Consequently, we have that

JL @ v(E) =

= 0 or

1

L

v(x, Ex) JL(dx) = 0 or 1.

Therefore, J-t ® v is ergodic, (1) =} (3) is trivial since we can take 11.3.2 and Corollary 2.6 (1).

VI

=v

and (3)

=}

(1) follows from Theorem

3.4. Ergodic channels

149

(1) => (5). Suppose that (5) is false. Then for some Vi, V2 E Cs(X, Y) with Vi t= V2 (modPse(X)) and a E (0,1) we have V == aVl + (1 - a)v2 (modPse(X)). Hence J-l ® Vi :f. J-l ® V2 for some J-l E Pse(X) and p, 0 v(E)

= =

L L[

v(x, Ex) p,(dx) a vl (X,Ex) + (1- a)v2(X'Ex)] p,(dx)

= au Q9 vl(E) + (1 - a)J-l Q9 v2(E ),

E E X Q9~.

Thus J-l ® V = ap. ® Vi + (1 - a) J-l Q9 V2 is a nontrivial convex combination of two distinct stationary sources. Therfore, J-l Q9 u is not ergodic, which contradicts the ergodicity of u, (5) => (2). Assume that (2) is false. Then, there are some J-l E Pse(X) and some S x T-invariant E E XQ9~ such that J-l(A) > 0, where A = {x EX: v(x, Ex) #= 0, 1}. Define Vi and V2 by

v(x, C n Ex) v(x, Ex) , { v(x, C), v(x, C n E~) V2(X, C) = v(x, Ei) , { v(x, C),

Vi(x, C) =

xEA,

xEA,

where C E~. Clearly, Vi and V2 are channels. As in the proof of (2) => (1) we see that Ex = T-1Esx by (4.1) and hence

v(x, Ex) implying that

s:' A =

= v(x, T- l E s x) = v(Sx, Es x),

xE

A,

(4.2)

A. Thus, for x E A and C E ~

_ v(Sx,CnEs x) _ v(x,T-l(CnEsx)) Vi (S x, C) v(x, Ex) v(Sx, Es x) v(x, T- 1C n T- l Es x) v(x, T- 1C n Ex) v(x, Ex) v(x, Ex) = Vi(x, T- 1C). It follows that Vi is stationary. Similarly, one can verify that V2 is stationary. Moreover, we see that

xEA,

Chapter III: Information Channels

150

and hence VI "t V2 (modPse(X)). Note that for x E X and C E ~

Let B = {x EX: v(x,EaJ ~ ~} E

x and define V3 and V4 by xEB, x E BC, xEB, x E BC,

where C E ~. Obviously, V3, V4 E C(X, Y). Observe that S-1 B == Band S-1 BC = B C by (4.2). Then it is easily verified that V3 and V4 are stationary by S-invariance of Band BC, stationarity of VI and V2, and (4.2). Furthermore, we have by the definition of V3 and V4, and by (4.3) that

V4(X, Ex)

{2v(x, Ex) - 1 }Vl (x, Ex) + {2 - 2v(x, Ex)} V2(X, Ex) = 2v(x, Ex) - 1 < 1.

If x E A n BC, then

V3(X, Ex)

= {I - 2v(x, Ex) }V2(X, Ex) + 2v(x, E x)Vl(X, Ex) = 2v(x, Ex) > 0,

while V4(X, Ex) = V2(X, Ex} = o. Thus, V3(X, Ex) -=1= V4(X, Ex) for x E A. This means that V3 "t V4 (modPse(X)) and so V ~ exCs(X, Y) (modPse(X)). Therefore, (5) is not true, (4) =} (2). Suppose (2) is false. Then for some J-t E Pse(X) and E E x®~ with (S x T)-l E = E it holds that J-t(A) > 0, where A = {x EX: 0 < v(x, Ex) < I}. Define Vo by v(x,CnEx) x E A, C E~, vo(x, C) = v(x, Ex) , { lJ(x,C), x E AC, C E ~. Then we see that Vo E Cs(X, Y) and V "t lJo (rnodPse(X)) in a similar way as before. Moreover, we have lJo(x,·) ~ v(x,·) Pse(X)-a.e.x since D E ~ and v(x, D) = 0 imply Vo (x, D) = 0 for x EX. This contradicts (4).

151

3.4. Ergodic channels

(1), (2), (5) =} (4). Thus far we have proved (1) {:} (2) {:} (3) {:} (5). Assume (4) is false. Then there exists some Vo E Cs(X, Y) such that vo(x,·) ~ v(x,·) Pse(X)-a.e.x,

Vo

"# v

(modPse(X)).

Let fL E Pse(X) be arbitrary. By (2), if E E X ® ~ is 8 x T-invariant, then v(x, Ex) = 0 or 1 u-a.e.x.

Let VI = ~v + ~vo. Then, vI(x,Ex) = 0 or 1 u-a.e.x since v(x, Ex) = 0 u-a.e.x implies vo(x, Ex) = 0 u-a.e.x, and v(x, Ex) = 1 u-a.e.x implies vo(x, Ex) = 1 u-a.e.x, Thus VI is ergodic by (2), which contradicts (5). (1) =} (6). Suppose V is ergodic and let fL E Pse(X), Then fL ® v E Pse(X x Y) and hence for every E, F' E X ® ~ n-I

!L n-+oo n lim

p, @ v((8

X

T)-k En F ')

= p, @ II(E)p, @ v(F').

(4.4)

k=O

If we take F ' = F n (A x Y) for F E X ® ~ and A E X, then on the one hand n-I

lim! n-+oo

L Jfx v(x, [(8

X

T)-k En F n (A x Y)]",) p,(dx)

n k=O

= n-+oo lim ! n

n-I

Lp,@v((8 x T)-kEnFn (A x Y)) k=O

=fL®v(E)fL®V(Fn(AxY)),

= p, @ v(E)

L

by (4.4),

v(x,F",) p,(dx),

(4.5)

and on the other hand by Proposition 1.4

!

LHS of (4.4) = lim n-+oo

=

n-I

L

j

v(x, [(8

X

T)-k En F]x) fL(dx)

L v(x, [(8

X

T)-k En F]",) p,(dx).

n k=O A n-I

j

lim!

(4.6)

A n-+oo n k=O

Since (4.5) and (4.6) are equal for every A E X, one has the equation in (6). (6) =} (7). If E = A x C and F = B x D, then the equation in (6) reduces to n-I

! Lls-kAnB(X)V(x, T-kC n D) n-+oo n lim

k=O

== n-+oo lim ! n

n-I

L

k=O

lS-kAnB(X)V(x, T- kC)II(X, D) Pse(X)-a.e.x.

152

Chapter III: Information Channels

Integrating both sides w.r.t. It E Pse(X) over X, we get the equation in (7). (7) =* (1). Let It E Pse(X), A, B E X and C, D E ~. Then n-I

~E

{Jt ® v((S x T)-k(A x C) n (B x D)) - Jt e v(A x C)Jt ® v(B x D)}

k=O n-I

=

.!. E ti

{

k=O JS-k AnB

+~~

{vex, T-kC n D) - vex, T-kC)v(x, D)} Jt(dx)

{L lA(SkX)V(Sk x, C)lB(x)v(x,

-L

lA(X)V(X, C) Jt(dx)

D) Jt(dx)

L

IB(x)v(x, D) Jt(dX)}

-+0 (asn-+oo) by the assumption (7) and the ergodicity of It. Thus It ® v is ergodic. In (7) of Theorem 2, if we replace E = X x C and F =X x D, then (7) reduces to the condition (3.1). We now consider ergodicity of a stationary channel in terms of the channel operators it associates. So assume that X and Yare a pair of compact Hausdorff spaces with homeomorphisms Sand T, respectively.

Definition 3. Let K E JCs(X, Y), A E As(X, Y) and P ~ P(X). Then K is said to be extremal in JCs(X, Y) mod P, denoted K E exJCs(X, Y) (mod P), provided that, if K == aK I + (1 - a)K 2 (mod P) for some K I , K 2 E JCs(X, Y) and a E (0,1), then K I == K 2 (mod P). Similarly, A is said to be extremal in As(X, Y) mod P, denoted A E ex As (X, Y) (mod P), provided that, if A == aA I + (1- a)A 2 (mod P) for some AI, A 2 E As (X, Y) and a E (0,1), then Al == A 2 (mod P). Under these preparations, extremal operators A E As (X, Y) are characterized as follows:

Theorem 4. Let A E As (X, Y). Then the following conditions are equivalent: (1) For every i.s E C(X x Y) lim A(fng)(x) = lim Afn(x)Ag(x) Pse(X)-a.e.x,

n-+oo

n-+oo

1 n-I

where I« = (S®T)nf = - L (S®T)kf for n k=O (2) It E Pse(X) =* A*1t E Pse(X x Y).

ti

2:: 1.

(4.7)

3.4. Ergodic channels

(3) A

E

Proof. (1)

153

ex As (X, Y) (modPse(X)). =}

(2). Let J-L E Pse(X) and f, 9 E C(X x Y). Then

lim A*J-L(fng)

n-+oo

= n-+oo lim J-t(A(fng))

= n-+oo lim J-t(Afn' Ag),

by (1),

= n-+oo lim J-L(Af)nAg) , since A(S ® T) = SA, = J-t(Af)J-t(Ag),

since J-L is ergodic,

= A*J-L(f)A*J-t(g),

where (Af)n = Sn(Af),n ~ 1 (cf. Theorem 11.3.2). Thus A*J-L E Pse(X x Y) by Theorem 11.3.2. (2) =} (1). Suppose that (1) is false. Then it holds that for some J-L E Pse(X) and i.s E C(X x Y)

J-L({x EX: lim A(fng)(x) n-+oo

=I n-+oo lim Afn(x)Ag(x)}) > 0

or, equivalently, for some h E C(X)

(4.8) Since A*J-L is ergodic by (2), we have lim J-L(A(fng)· h)

n-+oo

= n-+oo lim J-t(A(fngh)) ,

by (al),

= lim A*J-t(fngh) n-+oo

= lim A*J-t(fn)A*J-L(gh) n-+oo

= lim J-t(Afn)J-L(Ag· h). n-+oo

(4.9)

Moreover, since J-L is ergodic, we also have

(4.10) (4.9) and (4.10) contradict (4.8). Thus (4.7) holds. (2) =} (3). Suppose A == aA 1 + (1 - a)A 2 (modPse(X)) for sorne AI, A 2 As (X, Y) and a E (0, 1). For J-t E Pse(X)

E

Chapter III: Information Channels

154

is also ergodic by (2). Then by Theorem 11.3.2, AiJL = A 2JL = A * JL E Pse(X x Y). Since this is true for every JL E Pse(X), we have Al == A 2 == A (modPse(X)) by Proposition 2.7. Thus (3) holds. (3) =} (2). Assume that (2) is not true. Then there exists a JL E Pse(X) such that A * JL is not ergodic. Hence there is some S x T-invariant set E E X ® ~ such that 0 < A * JL(E) < 1. Let Al = A * JL(E) and A2 = 1 - AI, and take 'Y so that o < 'Y < rnin{AI' A2}. Let Qi = (i = 1,2) and define operators AI, A 2 on B(X x Y) by

t

Alf = QIA(fIE) + (1

QIAIE)Af,

A 2f = Q2 A(fIEc) + (1 - Q2A1Ec)Af,

f f

E E

B(X xY), B(X x Y).

We shall show AI, A 2 E A(X, Y). All = 1 is clear. AI(fAIg) = (AIf)(AIg) for f, 9 E B(X x Y) is seen from the following computation:

+ (1- QIA1E)Ag}] = QIA[f{QIA(gIE) + (1- QIAIE)Ag}IE] + (1 QIAIE)A[f{QIA(gIE) + (1- QIAIE)Ag}] = Q~A[fIEA(gIE)] + QIA[f IE(l- QIAIE)Ag] + QI(I- QIAIE)A[fA(glE)] + (1- QIAIE)A[f(I- QIAIE)Ag] = Q~A(fIE)A(gIE) + QIA[f 1E(I- QIAIE)]Ag + QI(I QIAIE)Af· A(gIE) + (1- QIAIE)A[f(I- QIAIE)]Ag

AI(fAIg) = Al [f{Q IA(glE)

= Q~A(fIE)A(gIE)

+ QI(I -

+ QIA(f1E)(I - QIAIE)Ag

QIAIE)Af· A(gIE)

+ (1 -

QIAIE)2Af' Ag = [QIA(fIE) + (1- QIAIE)Af] [QIA(gIE) + (1 - QIAIE)Ag] = (AIf)(AIg), where we used A(fAg) = (Af)(Ag). Similarly, we can verify (al) for A 2. Moreover, (a3) is clearly satisfied by AI, A 2, which is seen from their definitions. Thus AI, A 2 E A(X, Y). Furthermore, AI, A 2 E As (X, Y) since for f E B(X x Y) we have AI(S ® T)f

= QIA(((S ® T)f)IE) + (1

QIA1E)A(S ® T)f

= QIA((S ® T)(fIE)) + (1- QIAIE)SAf, since A(S ® T) = SA and E is S x T-invariant, = QISA(fIE) = SAlf

+ (1 -

QIAIE)SAf

3.4. Ergodic channels

155

and similarly A 2 (S ® T)f = SA 2f. Now we show that A 1 t:. A 2 (modPse(X)). In fact, we have

Jlt(A 21Ec - A 11Ec) = Jlt(a2A1Ec

+ (1- a2A1Ec)A1Ec

- a1A(lEc IE) - (1 - a1A1E )A1Ec )

+ A1Ec(a1A1E - a2 A 1Ec)) = a2A*Jlt(EC) + Jlt{A1Ec(a1 A 1E - a2 A 1Ec)) = a2A2 + Jlt(A1Ec )Jlt(a1 A 1E - a2 A 1Ec), since Sn(A1Ec) = AlEc and Jlt is ergodic, = Jlt(a2A1Ec

+ A2(a1A1 - a2 A2) ')' + A2(')' - ')') > O.

= a2A2

=

(4.11)

Finally we see that A1A1 + A2A2 = A since for f E B(X x Y)

A1 A1f + A2A2f = ')'A(f1E) + (A1 - ')'A1E)Af = ')'Af + (1 - ')')Af = Af.

+ ')'A(f1Ec) + (A2

- ')'A1Ec )Af (4.12)

These results imply that A ~ exAs(X, Y) (modPse(X)), a contradiction. As a corollary we have: Corollary 5. Let v E Cs(X, Y) be a stationary channel and K E Ks(X, Y) and A E As(X, Y) be the corresponding channel operators to t/, Then the following

statements are equivalent: (1) v is ergodic. (2) u E exCs(X, Y) (modPse(X)). (3) K E exKs(X, Y) (modPse(X)). (4) A E ex As(X, Y) (modPse(X)). (5) For f, 9 E C(X x Y) it holds that lim A(fng)(x) = lim Afn(x)Ag(x) Pse(X)-a.e.x,

n-too

n-too

where fn = (S ® T)nf, n ~ 1. (6) For t, 9 E B(X x Y) the equality in (5) holds. (7) For E, F E X ® ~ it holds that for Pse(X)-a.e.x n-1

n-1

k=O

k=O

.!. '" v(x, [(8 X T)-k En F]x) = n-too lim .!. '" v(x, [(8 X T)-k E]x)v(x, Fx). n-too n Z:: n L....t lim

Chapter III: Information Channels

156

Proof. (1) ¢:> (2) ¢:> (3) ¢:> (4) ¢:> (5) follow from Theorems 2.1, 2.2, 2 and 4. (5) => (6) is proved by a suitable approximation. (6) => (7) is derived by taking f = IE and 9 IF for E, F E X ®~. (7) => (6) follows from the fact that {IE: E E X ®~} spans B(X x Y). (6) => (5) is trivial. Remark 6. Observe that each of the following conditions is not sufficient for ergodicity of a stationary channel v E Cs(X, Y), where (X, X, S) and (Y,~, T) are a pair of abstract measurable spaces with measurable transformations: (1) v(x,') E Pae(Y) Pse(X)-a.e.x.

(2) v(x,·) E Pse(Y) Pse(X)-a.e.x. (3) v = v", for rJ E Pse(Y), v", being the constant channel determined by n. In fact, if X = Y, rJ E Pse(X) is not WM, and v", is ergodic, then rJ ® v", = rJ x rJ is ergodic, which implies rJ is WM by Theorem 11.3.10, a contradiction.

3.5. AMS channels AMS sources were considered as an extension of stationary sources in Section 2.4. In this section, AMS channels are defined and studied as a generalization of stationary channels. A characterization of AMS channels is obtained as well as that of ergodic AMS channels. Absolute continuity of measures plays an important role in this section. Let X, Y be a pair of compact Hausdorff spaces with Baire a-algebras X, ~ and homeomorphisms S, T, respectively. The invertibility assumption is crucial. Assume that ~ has a countable generator ~o, i.e., ~o is countable and a(Vo) =~. Recall that Pa(O) and Pae(O) stand for the set of all AMS sources in P(!~) and the set of all AMS ergodic sources in Pa(!~) for 0 = X, Y and X x Y, respectively.

Definition 1. A channel v E C(X, Y) is said to be asymptotically mean stationary (AMS) if (cI4) I-t E Pa(X) =>

V

® I-t E Pa(X x Y).

That is, if the input source is AMS, then the compound source is also AMS. Ca(X, Y) denotes the set of all AMS channels. First we need the following two lemmas.

Lemma 2. A channel u E C(X, Y) is AMS iff I-t E Ps(X) => I-t ® v E Pa(X x Y).

Proof. The "only if' part is obvious. As to the "if' part, let I-t

E

Pa(X). Then

157

3.5. AMS channels

Ii

E

Ps(X) and J-L

~

Ii by Remark 11.4.9. Hence, J-L Q!) v

« Ii Q!) v ~ Ii Q!) u,

where the first « is clear and the second follows from Remark 11.4.9 and the assumption since Ii E Ps(X). Thus J-L Q!) v E Pa(X x Y) by Proposition 11.4.8.

Lemma 3. Let u E C(X, Y) and J-L E P(X) be such that J-LQ!)v E Pa(X x Y). Then: (1) J-L E Pa(X) with the stationary mean Ii(') = J-L Q!) v(· x Y) E Ps(X). (2) J-Lv EPa(Y) with the stationary mean J-Lv(,) = J-L Q!) v(X x .) E Ps(Y). (3) v(x,·) ~ J-LV u-o:e.x.

x

Proof. (1) Observe that for A E n-I

~L n-+oo n lim

k=O

n-I

It(S-k A)

= n-+oo lim ~ " It @ v(S n L..,;

x T)-k(A x Y))

k=O

= J-L Q!) v(A x Y). Thus J-L is AMS with the desired stationary mean. (2) is verified similarly. (3) Suppose that J-Lv(C) = O. Then J-Lv(C) = 0 since J-LV «J-Lv. Hence Itv(C)

=

i

v(x, C) It(dx)

= O.

This irnplies that v(x, C) = 0 p-a.e.x, completing the proof. An important consequence of these two lemmas is: Corollary 4. For a channel u E C(X, Y) the following conditions are equivalent: (1) v is AMS, i.e., u E Ca(X, Y). (2) For each stationary J-L E Ps(X) there exists a stationary channel VI E Cs(X, Y) such that (5.1) V(x,·) « VI(X,') u-a.e.x.

(3) For each stationary J-L E Ps(X) there exists an AMS channel such that (5.1) holds.

VI

E

Ca(X, Y)

Proof. (1):::} (2). Suppose v is AMS and let J-L E Ps(X). Then J-LV E Pa(Y) by Lemma 3 (2) since J-L Q!) u EPa(X x Y). If we let VI (x, C) = J-Lv(C). for x E X and C E !D, then VI is a constant stationary channel and (5.1) is true by Lemma 3 (3). (2) :::} (3) is immediate.

Chapter III: Information Channels

158

(3) :::::> (1). Let u E Ps(X) and suppose the existence of VI mentioned. Then VI E Pa(X x Y) and J-t ® V « J-t ® VI by (5.1). Hence J-t ® V E Pa(X x Y) by Proposition 11.4.8. Thus u is AMS by Lemma 2.

J-t ®

Now we want to consider an analogy of stationary means of AMS channels. Suppose that v E Ca(X, Y) and J-t E Ps(X). Then J-t ® v is AMS. Observe the following computation: for A E X and C E ~ n-l

n-l

1 ' " P, ® v((S x T)-k(A x C)) n LJ k=O

= .!.n L p, ® v(S-k A x T-kC) k=O

n-l

= .!. L n

(

vex, T-kC) p,(dx)

k=O}S-kA

n-l

=.!. L

n k=O

1

1

v(S-kx,T-kC)p,(d(S-kx))

A

n-l

= .!. L v(S-k x, T-kC) p,(dx) A

=

n

L

k=O

vn(x, C) p,(dx),

say,

== J-t® vn(A x C) -+ J-t ® v(A x C) (n -+ 00). Clearly, each V n (n 2:: 1) is a channel, but not necessarily stationary. The sequence {v n } is expected to converge to some stationary channel fJ, to be called again a "stationary mean" of u, We shall prove the following.

Proposition 5. A channel v E C(X, Y) is AMS iff for each stationary input source J-t E Ps(X) there exists a stationary channel v E Cs(X, Y) such that for every C E ~ 1 v(x, C) == lim n-+oo n

n-l

L

v(S-k x, T-kC)

u-a.e.x.

(~.2)

k=O

In this case it holds that (5.3)

Proof. Assume that v E Ca(X, Y) and J-t E Ps(X). Let "1(.) == J-tv(·). Then, by Lemma 3, "1 E Ps(Y) and v(x,') «"1 u-a.e.x, Let Xl == {x EX: v(x,·) « "1}, so that J-t(X I ) == 1. Moreover, if X* n S" Xl, then X* is S-invariant and J-t(X*) == 1 nEZ

159

3.5. AMS channels

since 1-£ is stationary. Let

k(x, y)

=

{

v(x,dy) 'f/(dy)'

x E X*,y E Y,

0,

x ¢ X*, Y E Y.

Then we have that k E L 1(X X Y, 1-£ ® v) by Theorem IV.2.5 and Remark IV.2.6, which are independent of this proposition, since 1-£ ® v « 1-£ x "7, and

vex,C)

=

Ie

k(x, y) 'f/(dy) ,

Now one has for x E X* and C E

-1 L

n-1

n

v(S-k x , T-kC)

~

1 n-1 = -

L1

n

k=O

k=O

n-1

= .!:. L

XEX*,CE~.

k(S-k x , y) "7(dy)

T-kC

1

k(S-kx, T-ky) 'f/(dy),

n k=O

since 'f/ is stationary,

C

n-1

= 11c(Y).!:. n

y

L k(S-kx, T-ky) 'f/(dy)

k=O

= [lc(y)[(S-l @T-1)nk](x,Y)'f/(dy).

By the Pointwise Ergodic Theorem there is an S x T-invariant function

(8- 1 ® T-1)nk -+

k*

k*

such that

1-£ ® u-a.e.

since k is jointly measurable. Hence we see that n-1

lim

.!:. L

n--+oo n k=O

since "7(.)

= 1-£ ® v(X

v(S-k x, T-kC)

=

rlc(y)k*(x, y) 'f/(dy)

u-a.e.x

}Y

x .). Take a stationary channel u" E Cs(X, Y) and define D by

D(x,C) = {

Ie

k*(x, y) 'f/(dy) ,

XEX*,CE~,

x¢X*,CE~.

v*(x, C),

Then, clearly Dis a stationary channel and (5.2) holds. By the Bounded Convergence Theorem we have from (5.2) that n-1

p, @ v(A x C)

= n--+oo lim .!:. '" p, @ v((S x T)-k(A x C)) n L.....t k=O

Chapter III: Information Channels

160

for A E X and C E~. Thus, since j.L Q9 1) is stationary, j.L Q9 v is AMS with the stationary mean j.L Q9 1), i.e., (5.3) is established. In Proposition 5, the stationary channel 1) depends on the given AMS channel v and the stationary input source u: We would like to have a single stationary channel independent of input sources such that (5.2) and (5.3) are true for all stationary j.L E Ps(X). This will be obtained using the countable generator ~o of~.

Theorem 6. For any AMS channel v E Ca(X, Y) there is a stationary channel YJ E Cs(X, Y) such that for any stationary input source j.L E Ps(X) 1

YJ(x, C)

= n-too lim n

n-l

L v(S-k x, T-kC)

u-a.e.x, C E ~,

(5.4)

k=O

(5.5)

j.LQ9v = j.LQ9YJ, v(x,·) «YJ(x,·)

u-a.e.x,

(5.6)

Proof. Let v E Ca(X, Y) and X(C)

=

{

x EX: lim

n-too

} .!.n n-l ~ v(S-kx, T-kC) exists L..J

,

CE~O,

k=O

X(v)

=

n

X(C).

CE~o

Then for any stationary j.L E Ps(X) we have j.L(X(v)) = 1

since j.L (X(C)) = 1 for C· E ~o by Proposition 5 and ~o is countable. Take a stationary channel v* E Cs(X, Y) and define a set function YJ on X x ~o by I

_() v x, C

=

n-l

lim - I: v(S-k x, T-kC), n-too n k=O { v*(x, C),

x E X(v),C E ~o, x fJ. X(v),C E ~o.

It is evident that YJ can be extended to X x ~ and becomes a stationary channel. Thus (5.4) is satisfied. (5.5) is shown by the Bounded Convergence Theorem and (5.6) is almost obvious.

Definition 7.. The stationary channel YJ constructed above"is called a stationary mean of the AMS channel v E Ca(X, Y).

161

3.5. AMS channels

The following is a collection of equivalence conditions for a channel to be AMS. Recall the notations s; and (8 ® T)n'

Theorem 8. For a channel u E C(X, Y) the following conditions are equivalent: (1) v E Ca(X, Y), i.e., u is AMB. (2) J-L E Ps(X) => J-L ® v E Pa(X x Y). (3) There is a stationary VI E Cs(X, Y) such that v(x,·) ~ Vl(X,') Ps(X)-a.e.x. (4) There is an AMB VI E Ca(X, Y) such that v(x,·) ~ VI(X,') Ps(X)-a.e.x. (5) There is a stationary Ii E Cs(X, Y) such that for C E ~ 1 n-l lim - "v(S-k x, T-kC) = vi», C) n4-OO n L....J

Ps(X)-a.e.x.

k=O

(6) For f E B(X x Y) and lim n4-OO

J-L E

Ps(X) the following limit exists:

r

l» [Av(S Q9 T)nf] (x) JL(dx).

If any (and hence all) of the above is true, then it holds that

(7) J-L ® v = J-L ® Ii for J-L E Ps(X). (8) v(x,·) ~ Ii(x,') Ps(X)-a.e.x. (9) For J-L E Ps(X) and f E B(X x Y) lim n4-OO

r [Av(S

Jx

Q9 T)nf]

(x) JL(dx)

=

r

Jx (Avf)(x) JL(dx).

Proof. (1) {::} (2) was proved in Lemma 2 and (1) {::} (5) follows from Theorem 6. By taking VI = Ii, (3) is derived from (5). (3) => (4) is immediate and (4) => (1) is proved in Corollary 4. Thus we proved (1) {::} (2) {::} (3) {::} (4) {::} (5). (2) {::} (6). Observe that for f E B(X x Y) and J-L E Ps(X)

L

[Av(S

Q9

T)nf] (x) JL(dx) = =

L[(S j"{

Q9

T)nf(x, y) v(x, dY)JL(dx)

(8 ® T)nf(x, y)

J-L ®

v(dx, dy)

JXXY

by Lemrna 2.5. The equivalence follows from Theorem 11.4.6. (7) and (8) are already noted in Theorem 6. As to (9) we proceed as follows: for f E B(X x Y) and J-L E Ps(X)

L

(Avf) (x) JL(dx)

=

Ll>

y)IJ(x, dY)JL(dx)

Chapter III: Information Channels

162

=

j"( jr(

f(x, y) Jj ® v(dx, dy),

by Lemma 2.5,

f(x, y) p,(ji'y v(dx, dy),

by (7),

JXXY

=

JXXY

= lim n---too

jf"{ (8 ® T)nf(x, y) Jj ® v(dx, dy) JXXY

= n---tooJx lim ( [Av(S Q9 T)nf](x) tL(dx). Example 9. (1) As was rnentioned before, each probability measure 'fJ E P(Y) can be regarded as a channel by letting v'TJ(x,C) = 'fJ(C) for x E X and C E~. If 'fJ E Pa(X), then v'TJ is AMS. In fact, 'fJ «rj since T is invertible, so that v'TJ(x,·) = 'fJ «rj = vi](x,·) for x E X. Moreover, vi] E Cs(X, Y) implies that v'TJ E Ca(X, Y) by Theorem 8 (3). In this case we have v'TJ = vi]. (2) If a channel u E C(X, Y) satisfies that v(x,·) «'fJ

Ps(X)-a.e.x

for some AMS 'fJ E Pa(Y), then v is AMS by Theorem 8 (4) and (1) above. Let

k(

x,y

) = vex, dy)

(x,y) E X x Y,

rj(dy) ,

where rj E P; (Y) is the stationary mean of n. Then v can be written as

v(x,C) and its stationary mean

= [ k(x,y)'fj(dy),

~

v as

v(x, C) = [k*(X, y) 'fj(dy) where k*(x, y)

x E X,C E

= n---too lim (8- 1 ® T-1)nk(x, y)

P.(X)-a.e.x, C Jj

E !l),

® v-a.e. (x, y) for

Jj E

Ps(X).

(3) A channel v E C(X, Y) satisfying the following conditions is AMS: (i) v is dominated, i.e., v(x,·) « 1/ (x E X) for some 'fJ E P(Y); (ii) v(x,·) E Pa(Y) for every x E X. In fact, u is strongly measurable by Corollary IV.2.3 since v is dominated and ~ has a countable generator. Hence v has a separable range in M(Y), so that {v(x n , · ) : n ~ I} is dense in its range for some {x n } ~ X. Let

(.) =

t n=l

v(~:, .).

163

3.5. AMS channels

Then we see that € E Pa(Y) by (ii) and Lemma 11.4.3(1). Thus v is AMS by (2) above since vex, .) ~ € (x EX).

Definition 10. An AMS channel v E Ca(X, Y) is said to be ergodic if (cI5)

J-L E

Pae(X) =>

J-L

® v E Pae(X x Y).

Cae (X, Y) denotes the set of all ergodic AMS channels in Ca(X, Y). After giving a lemma we have the following characterization of AMS ergodic channels. Sorne of the equivalence conditions are similar to those of AMS ergodic sources and some are to those of stationary ergodic channels.

Lemma 11. Let v E Ca(X, Y) be AMS and J-L E Ps(X) be stationary. Then for every E, F E X ® q) the following limit exists p-a.e.x: n-I

lim

n.-+-oo

.!.n "'v(x,[(8xT)-kEnF]x). L....J k=O

Proof. Since v is AMS and J-L is stationary, J-L ® v is AMS with proof parallels that of Proposition 1.4 and we have

J-L

v=

J-L

® 17. The

n-I

lim

n.-+-oo

.!.n E v(x, [(8 X T)-k En F]x) k=O

=

where J

= {E E X® q)

i

IF(X, y)EI'0v(lEIJ)(x, y) vex, dy) u-a.e.x,

: (8 x T)-IE

= E}.

Theorem 12. Let v E Ca(X, Y) with the stationary mean 17 E Cs(X, Y). Then the following conditions are equivalent: (1) v E Cae (X, Y), i.e., v is ergodic. (2) J-L E Pse(X) => J-L ® v E Pae(X x Y). (3) 17 E Cse(X, Y). (4) There is a stationary ergodic VI E Cse(X, Y) such that

v(x,·) «VI(X,')

Pse(X)-a.e.x.

(5.7)

(5) There is an AMS ergodic VI E Cae (X, Y) such that (5.7) is true. (6) If E E X®q) is 8 x T-invariant, then v(x,Ex ) = 0 or 1 Pse(X)-a.e.x.

Chapter III: Information Channels

164

(7) For E, F

E

X Q9 q) and J-t

E

Pse(X)

1 n-l lim - """ v(x, [(8 x T)-kEnF]x) n4-OO n L...J

=

k=O

1 n-l lirn - L17(x, [(8 x T)-kE]x)v(x,Fx) n4-OO n k=O

= J-t Q917(E)v(x, Fx)

u-a.e.x,

Proof. (1)

=} (2) is clear. (2) =} (3). Suppose (2) is true and let J-t E Pse(X). Then J-t Q9 u E Pae(X x Y). Hence J-t ® 17 = J-t v E Pse(X x Y) by Theorem 11.4.12 and Theorem 8 (7). Thus 17 is ergodic. (3) =} (4). Take VI = 17 and invoke Theorem 8. (4) =} (5) is immediate. (5) =} (6). Assume (5) is true. Let E E X Q9 q) be 8 x T-invariant and take J-t E Pse(X). Then J-t ® VI E Pae(X x Y) and J-t Q9 u ~ J-t Q9 VI· If J-t Q9 vl(E) = 0, then 0 = J-t Q9 v(E) = Ix v(x, Ex) J-t(dx) , so that v(x, Ex) = 0 u-a.e.x, Similarly, if J-t ® VI(E) = 1, then we can show that v(x, Ex) = 1 u-a.e.x. (6) =} (1). Let J-t E Pae(X). Suppose that E E X Q9 q) is 8 x T-invariant and v(x, Ex) = 0 ii-a.e.x and hence u-a.e.x, Then

p, @ veE) =

Ix

v(x, Ex) p,(dx)

= o.

If v(x, Ex) = 1 ii-a.e.x, then we have J-tQ9v(E) = 1. Thus J-tQ9V is ergodic. Therefore v is ergodic. (1) =} (7). This is shown in a similar manner as the proof of (1) =} (6) of Theorem 4.2 using Lemma 11. (7) =} (1). Let J-t E Pse(X) and E, F E X Q9 q). By integrating both sides of the equation in (7) w.r.t. J-t over X, we get

Hence J-t Q9 v is AMS ergodic. Therefore v is AMS ergodic. We noted that exPa(X) ~ Pae(X) and the set inclusion is proper (cf. Theorem 11.4.14). Similarly we can prove the following. Theorem 13. (1) If v E exCa(X, Y) (modPse(X)), then v E Cae(X, Y). Thus ex c; (X, Y) ~ Cae (X, Y).

165

3.5. AMS channels

(2) If there exists a weakly mixing source in Pse(Y), then the above set inclusion is proper, in that there exists some AMS ergodic channel v E Cae (X, Y) such that v

rt exCa(X, Y)

(modPse(X)).

Proof. (1) Let v E Ca(X, Y), I1 E Cs(X, Y) be its stationary mean and u B A E A(X, Y) be the corresponding channel operator. Suppose that v rt Cae (X, Y). Then there is some J-L E Pse(X) such that A * J-L rt Pae(X X Y). Hence there is some S x Tinvariant set E E X Q9 ~ such that 0 < Al == A * J-L(E) < 1. Letting A2 = 1 - AI, take ~ > 0 so that 0 < ~ < Inin{Al, A2}. Let ai = (i = 1,2) and define operators AI, A 2 on B(X x Y) by

Xi

= 0!1A(f1E) + (1 - a lA1E)Af, A 2f = a2 A(f1Ec) + (1 - a2A1Ec )Af, Alf

f f

x Y), B(X x Y).

E B(X E

Then as in the proof of (3) => (2) of Theorem 4.4 we see that AI, A 2 E A(X, Y). It follows from (4.11) that Al 1=. A 2 (modPse(X)). Moreover, A is a proper convex combination of Al and A 2 :

which follows from (4.12). Now we want to show that VI and Observe that for f E B(X x Y)

L

V2

are AMS channels, where

Vi B

Ai (i = 1,2).

[A1(S Q9 T)nf](x) /-L(dx)

= =

L

[a1(A(S Q9 T)nf 1E) + (1 ~ a 1 A 1E)A (S Q9 T)nf] (x) /-L(dx)

L

[a1A(S

Q9

T)n(j1E)

because E is S x T-invariant. Since

r,«

+ (1 -

a 1 A 1E)A (S Q9 T)nf] (x) /-L(dx)

v is AMS, n-+oo lim Ix alA(S Q9 T)n(f1E) dJ-L exists

by Theorem 8 (6). Also lim a lA1E)A(S Q9 T)nf dJ-L exists by Theorem 8 (6) n-+oo and the Bounded Convergence Theorem. Thus, we proved that

exists for every f E B(X x Y) and hence VI is AMS by Theorem 8 (6).. Similarly, is AMS. Consequently we see that u rt exCa(X, Y) (modPse(X)).

V2

Chapter III: Information Channels

166

(2) Take an

1]

E

Pse(Y) that is WM and define

~(C) = L9dT/,

eby

CE~,

where 9 E L l(Y,1]) is nonnegative with norm 1 which is not T-invariant on a set of positive 1] measure. Then, as in the proof of Theorem 11.4.14 we see that e E Pae(Y) , i= 1], ~ = 1] and ( == ~(e + 1]) E Pae(Y) isa proper convex combination of two distinct AMS sources. Hence

e

"c ¢ exCa(X, Y)

(modPse(X))

since "c = ~(ve + v.,,), ve, v." E Ca(X, Y) and iJe i= v.". We need to show "c E Cae (X, Y). Clearly v( = v, = v." E Cse(X, Y) since J-t ® Vn = J-t x 1] E Pse(X x Y) for J-t E Pse(X) by Theorem 11.3.10. Thus "c E Cae (X, Y) by Theorem 12. If the output space is an alphabet message space Y = Yl~ with a shift transformation T, then there exists a Bernoulli source that is strongly mixing (cf. Example 11.3.7), so the assumption in Theorem 13 (2) is satisfied.

3.6. Capacity and transmission rate

For a stationary channel we define the transmission rate functional and the stationary and ergodic capacities. An integral representation of the transmission rate functional is given. For a stationary ergodic channel the coincidence of two capacities is shown. First we deal with the alphabet message space case and then the general case. Consider alphabet message spaces X = X~ and Y = Y oZ with shifts Sand T, respectively, where X o = {at, ... ,ap } and Yo = {b l , ... ,b q } . For each n ~ 1, ~(X) denotes the set of all messages in X of length n starting at time i of the form [X~k) .•. X~~n-l], 1 ~ k ~ pn. Similarly, we denote by VJt~ (Y) and VJt~ (X x Y) the sets of all messages in Y and X x

Y of length n starting at time i, respectively. Note that

.

n-l..

~(X) = .V

3=0

S-3VJti(X)

E

P(X) for n ~ 1 and i E Z. Let a channel u E C(X, Y) be given. For each input source J-t E P(X) we associate the output source J-tV E P(Y) and the compound source J-t ® v E P(X x Y). The mutual information In (X, Y) = In(J-t; v) between the two finite schema (VJt~(X), J-t) and (VJt~(Y),J-tv) is given by In (X, Y) = In(J-t; v)

3.6. Capacity and transmission rate

167

(cf. (1.1.2)), where

HJL(VJt~(X)) = -

L

M(A) logM(A),

AEVJt~(X)

etc. Then, ~ In (M ; v) is considered as an average information of one symbol (or letter) when messages of length n in the input source [X, J-L] are sent through the channel u, If the limit

since

V

n=-oo

with a

snVJt~(X) = X, etc. In this case note that I(· ; v) is affine: for a, (3 ~ 0

+ {3 =

1 and M, rJ E Ps(X)

I(aM + {3rJ; v)

= aI(M; v) + {3I(rJ; v),

which follows from Theorem 1.5.6 (or Lemma 1.5.1). The,.. stationary capacity and the are respectively defined by

Cs(v) = sup {I(M; v) : ME Ps(X)}, Ce(v) = sup {I(J-L; v) : ME Pse(X) and MQ9 v

Pse(X x Y)},

E

where if MQ9 v ¢ Pse(X x Y) for every ME Pse(X), then let Ce(v) = O. Let us mention the weak* upper semicontinuity of the entropy functional. For n ~ 1 let Hn(M) = HJL(VJt~(X)). Then we have that for each M E Ps(X)

n n lim Hn(p,) n

n~oo

~

1, k

= H (8). JL

~

1,

~

1,

(6.1)

Chapter III: Information Channels

168

In particular, (6.1) implies that

Hkn(J-L) < Hn(J-L) kn n '

n

2:: 1,k 2:: 1,

so that

H ( ) > H 2(J-L) > H 22(J-L) > ... > H 2n(J-L) > ... , 1 J-L 2 22 2n i.e, H 2;} p,) t Hp,(S). This shows that H(.)(S) is a weak* upper semicontinuous function on Ps(X) c C(X)* since each H n(·) is so. Although this was proved in Lemma II. 7.3, we shall use this method to prove the part (2) (i) of the following theorem, which summarizes the fundamental results regarding transmission rates and capacities in the alphabet message space case.

Theorem 1. Let v E Cs(X~, Yl~) be a stationary channel. (1) 0 ~ Ce(v) ~ Cs(v) ~ logpq, where p = IXol and q = IYol. (2) If v has a finite memory (cf. Definition 1.1 (c5)) and is finitely dependent (cf. Definition 3.3 (c8)), then: (i) 1(· ; v) is upper semicontinuous in the weak* topology of Ps(X) C C(X)*; (ii) Cs(v) = Ce(v); (iii) There exists a stationary ergodic input source J-L* E Pse(X) such that Ce(v) = I(J-L*;v). (3) Cs(v) = Ce(v) if v is ergodic.

Proof. All the statements of Theorem 1 (except (2)(i)) follow from the discussion below (see Theorem 8), where we consider a more general setting. So we prove (2)(i). Since v has finite memory, it is continuous (Definition 3.1, (c5')), so that Hp,v(T) is a weak* upper semicontinuous function of J-L by Proposition 1.6 and Lemma 11.7.3. Hence, it is sufficient to show that Hp,(S) - Hp,®v(S X T) is a weak* upper semicontinuous function of J-L on Ps(X). Since v has finite memory and is finitely dependent, there exists a positive integer m such that (c5) and (c8) hold. By (c5) we have that for any message C = [Ym+l ... Yn](m + 1 ~ n) C Y

v(X, C) = v(x', C),

x,x'

E

A = [Xl'· ·Xn ] .

We denote this common value by v(A, C). For n 2:: 1 and J-L E Ps(X) let

fn(J-L) =

-!. [L J-L ® v(A x C) log J-L ® v(A x C) n

L J-L(A) log J-L(A)] ,

A,C

A

where the sum is taken for A E 9R~(X) and C E 9R:!~(Y). Observe that

fn(P,) =

~ [LV(A,C)P,(A)lOgV(A,C)P,(A) A,C

LP,(A)lOgP,(A)] A

(6.2)

169

3.6. Capacity and transmission rate

1

=-

2: v(A, C)J.L(A) logv(A, C).

(6.3)

n A,C

= [Xl" 'X n] E VJt~(X) let A' = [Xm+l" .xn] E VJt~!~(X) and for C = [Ym+l" 'Yn] E VJt~!~(Y) let C' = [y~ .. 'y!mYm+I" 'Yn] E VJt~(Y). Then one has for J.L E Ps(X) For A

J.L ® v(A xC') :::; J.L ® v(A x C) :::; J.L ® v(A' x C). It follows that for n ~ m

+ 1 and J.L E Ps(X)

2: J.L ® v(A xC') log rz ® v(A xC') : :; 2: J.L ® v(A xC') logJ.L ® v(A x C)

-Hn(J.L ® v) =

A,C'

A,C'

=

2: J.L ® v(A x C) log J.L ® v(A x C) A,C

= Gn(J.L),

(6.4)

say,

< 2: J.L e v(A x C) log®v(A' x C) A,C

=

2: J.L ® v(A' x C) log u ® v(A' x C) 2: J.L ® v(smA' x TmC) log J.L ® v(smA' x TmC)

A',C A',C

= -Hn-m(J.L ® v).

(6.5)

Hence (6.2) and (6.4) imply

fn(P,)

= !n (G n (p,) + Hn(p,» ~ n1 (Hn(p,) - H n (p, ® lI»,

while (6.2) and (6.5) imply

These two yield that liminf fn(P,) n-+oo

~ n-+oo lim !(Hn(p,) n

Hn(p, ® lI»

= Hp,(S) - HJL®v(S X T)

170

Chapter III: Information Channels

lim sup fn(tt) n-+oo

~ n-+oo lim ! (Hn(tt) n

Hn-m(tt ® v))

= lim ~Hn(J-t) - lim n

n = HJ.t(S) n-+oo

n-+oo

HJ.t®v(S

X

n

m. _ l - Hn_ m (J-t ® v) n- m

T).

So we conclude that lim fn(J-t)·= HJ.t(S) - HJ.t®v(S x T),

n-+oo

Note that fn(·) is a weak* continuous function on Ps(X) for each n ~ 1. To prove the weak* upper semicontinuity of HJ.t(S) - Hp,®v(S X T) it suffices to show that {fn(J-t)} contains a monotonely decreasing subsequence. Let f ~ 1 be arbitrary and n = 2(m + f). Denote a message A = [Xl··· Xn] E 9Jt~(X) by A = [Xl· .. Xm+l] n [X m+l+1 ... Xn] = Al n A 2 , say. Similarly, we write C = [Ym+1

Yn]

Ym+l] n [Ym+i+1 ... Y2m+l] n [Y2m+l+1 ... Y2(m+l)] n c' n C 2 , say.

= [Ym+1 = C1

Since v has a finite memory and is finitely dependent, one has

v(A,O) = v(A, 0 1 n Of n O2)

< v(A, C 1 n O2 ) = v(A, 01)v(A, O2), where V(A,Ol n C 2) stands for the common value of v(x, 0 1 n O2) for X E A. Now from (6.3) and the above we see that

fn(J-t) = f2(m+l)(J-t) 1

L L

< 2(m + l)

v(A 1 , C 1)v(A2 , C 2)tt(A1 n A 2 ) logv(A 1 , C1)v(A2 , C 2 )

A1,C1 A 2,C2

=

1 m+.f..

--0

L

v(A 1, C 1 )J-t(A 1 ) logv(A 1, 0 1 )

A1,C1

= fm+l(J-t), so that fm+l(J-t) .~ f2(m+l)(J-t). Since this holds for every f = 1,2, ... , we can choose a monotonely decreasing subsequence {fnk(J-t)}k=l for each J-t E Ps(X).

171

3.6. Capacity and transmission rate

Now we assume that X and Yare totally disconnected compact Hausdorff spaces with the bases X o and ~o of clopen sets and homeomorphisms Sand T, respectively. As before X and ~ stand for the Baire o-algebras of X and Y, respectively. Let 2t E P(Xo) and ~ E P(~o) be fixed clopen partitions of X and Y, respectively. Q: = 2t x ~ denotes the clopen partition {A x B : A E 21,B E ~} of X x Y. We consider three entropy functionals: H 1 (1t) = H(It, 21, B),

H 2 ( ", ) = H("" H3(~)

~,T),

= H(~, c,S x T),

It E Ms(X), '" E ~ E

Ms(Y), Ms(X x Y).

(6.6)

By Theorem 11.7.6 there are B-, T- and B x T-invariant nonnegative measurable functions hi on X, h 2 on Y and h3 on X x Y, respectively such that

H 1(p,)

=

L

h1(x) p,(dx),

H2(11) =

1, 2(y) 11(dy),

H3{~) =

jr(lxxY h3(X,y) ~(dx, dy),

h

~ E

Ms(X x Y).

Definition 2. Let v E Cs(X, Y) be a stationary channel and It E Ps(X) be a stationary source. Then, the transmission rate v:t It; v = v:t It; v, 21, ~ of the channel v w.r.t. by

where HI, H 2 and H 3 are defined by (6.6). Hence v:t(. ; v) is called the transmission rate junctional of v on Ps(X) or on Ms(X). Hereafter, we shall use the letter v:t for the transmission rate functional. We can obtain an integral representation of the transmission rate functional

v:t(. ; v). Proposition 3. Let v E Cs(X, Y) be a stationary channel. Then the transmission rate junctional v:t(. ; It) is a bounded positive linear junctional on M; (X) and there is a universal B-invariant bounded Baire junction t{·) on X such that (6.7)

172

Chapter III: Information Channels

Proof. Observe that for J-t E Ms(X)

H 1(J-t) + H 2(J-tv) - H 3(J-t ® v)

~(J-t; v) =

=

r h(x) p,(dx) + Jyrh(y) P,V(dy)- j JXXY r h(x, y) p, @v(dx ,dy) Jx r

1

i

2

h1(x) p,(dx) +

i[

3

h2(y) v(x, dy)p,(dx)

-i[h (x ,y) v(x ,dY)P,(dX) = l, [h (X) + [h (y) v(x, dY) - [h (x, y) v(x, dY)] P,(dX) 3

1

2

3

by Lemrna 2.5. Hence, letting XEX,

(6.8)

we have the desired integral representation (6.7). S-invariance of 1: follows from S-invariance of h-i, T-invariance of h2 , S x T-invariance of h3 and stationarity of u, Clearly ~(.; v) is a bounded linear functional on Ms(X) since 1: is bounded. We show that ~(. ; v) is nonnegative. Note that

(6.9)

9l(J-t; v) = H 3(J-t x J-tv) - H 3(J-t ® v)

= n-+oo lim 1 n

L [(p, x p,v)(A x B) log(p, x p,v)(A x B) A,B

- J-t ® v(A x B) log u ® v(A x B)]

= n-+oo lim .!.. '" p, @ v(A n L....J

x B)[log(p, x p,v)(A x B) -log p, @ v(A x B)]

A,B

~O

by Theorem 1.1 (1), where the sum is taken over A E 2l. V 2l.n and B E ~ V Q3n, and we have used the following computation:

L A,B

(1-£ xJ-tv)(A xB) log(J-t x J-tv)(A x B)

3.6. Capacity and transmission rate

173

L J-L(A) log J-L(A) + L J-Lv(B) log J-Lv(B) == L J-L ® v(A B) logJ-L(A) + L J-L ® v(A A,B A,B == L J-L ® v(A xB) log(J-L J-Lv)(A B). ==

A

B

X

X

X

B) logJ-Lv(B)

X

A,B

For a general J-L E M;-(X) we can similarly show 9t(J-L; v) 2:: O. The following corollary immediately follows from (6.9).

Corollary 4. For a stationary channel v E Cs(X, Y), the transmission rate functional 9\(. ; v) is wri·tten as

where (1-£ == J-L

X

J-LV - J-L ® u for J-L E M; (X).

To consider the Parthasarathy-type integral representation of transmission rates we use results in Sections 2.6 and 2.7. As in Section 2.7 denote by 2)0 the algebra of 00 . clopen sets generated by . U S-JSll and let 2)x == a(2)o), the a-algebra generated J=-OO

by 2)0' Let C(X, Sll) be the closed subspace of C(X) spanned by {IA : A E 2)0}. Ps(X,2)x) stands for the set of all stationary sources on (X, 2)x), and Pse(X, 2)x) for the set of all ergodic elements from Ps(X, 2)x). As usual, for J-L E P(X), J-L1~x denotes the restriction of J-L to 2)x .

Lemma 5. With the notation mentioned above, (1) Ps(X, Q3x) == {J-LI~x : J-L E Ps(X)}. (2)Pse(X,2)x) == {J-LI~x : J-L E Pse(X)}.

Proof. (1) Let J-LI E Ps(X,2)x) be given. Then J-LI is regarded as a positive linear functional of norm 1 on C(X, Sll) by the Riesz-Markov-Kakutani Theorem, By the Hahn-Banach Theorem there exists an extension of J-LI onto C(X) of norm 1. Let PI be the set of all such extensions. Since PI is weak* compact and convex, the fixed point theorem implies that there is some S-invariant J-L E P(X), i.e., J-L E Ps(X) and J-L1~x == J-LI· (2) Suppose that J-LI E Pse(X,2)x) is ergodic and let PI be as in (1). If J-L E ex [ps(X) n PI]' then J-L is ergodic. For, if J-L == O',rJ + (3€ with a, (3 > 0, 0',+(3 == 1 and n, € E Ps(X), then as functionals J-L == O',rJ + (3€ == J-LI on C(X, 2l). Since J-LI is ergodic, we see that J-L == rJ == € on C(X,2(.) and hence 11, € E Ps(X) n Pl' Thus J-L rJ == € on C(X) because J-L is extremal in Ps(X) n Pl' Therefore J-L is ergodic and J-L1~x == J-LI.

174

Chapter III: Information Channels

It follows from Lemma 5 (1) that

Hence the entropy functional H I ( · ) on P s(X,2() is unambiguously defined by

Recall that R denotes the set of all regular points in X (cf. Section 2.7). For J-t E Ps(X) and r E R, J-tr denotes the stationary ergodic source corresponding to r. By Lemma 5 (2) there is some Pr E Pse(X) such that J-tr = Pr Imx and we define

for a stationary channel u, For partitions ~ and
Proposition 6. Let v E Cs(X, Y) be a stationary channel satisfying that (c2') v(·, C) is

~x

-measurable for CEq:).

Then the function 9\(J-tr; v) of r E R is ~ x -measurable on R such that

Proof. According to the proof of Theorem 11.7.6 the entropy functions hI, h2 and h 3 are ~x-, ~y- and ~xxy-measurable, respectively. Hence the condition (c2') implies that h 2(y) v(·, dy) and h 3 ( · , y) v(·, dy) are ~x-measurable, so that Vl(J-t(.); v) is also ~x-measurable on R. By Lemma 11.7.5 we see that for J-t E Ps(X)

Jy

Jy

!R(JL; v) =

=

L L

l:(x) JL(dx) =

L[L

l:(x) JLr(dX)] JL(dr)

!R(JLr; v) JL(dr).

Definition 7. For a stationary channel t/ E Cs(X, Y) the stationary capacity Cs(v) is defined by Cs(v) = sup {Vl(JL; v) : JL E Ps(X)} and the ergodic capacity Ce(v) by

Ce(v) = sup {Vl(J-t; v) : J-t

E

Pse(X) and JL ® v

E

Pse(X x Y)}.

3.6. Capacity and transmission rate

175

If there is no J-t E Pse(X) with J-t Q9 v E Pse(X x Y), then we let Ce(v) =

o.

Theorem 8. Let v E Cs(X, Y) be a stationary channel satisfying (c2'). (1) Cs(v) = sup {91:(J-t; v) : J-t E Pse(X)}. (2) If v is ergodic, then Cs(v) = Ce(v). (3) If v is ergodic and 91:(. ; v) is weak* upper semicontinuous on Ps(X) c C(X)*, then there exists an ergodic source J-t* E Pse(X) such that Cs(v) = Ce(v) = 91:(J-t*; v). Proof. (1) By Proposition 6 we see that for any J-t E Ps(X)

9l(JL; v)

=

L

9l(JLr; v) JL(dr)

::; sup 91:(J-tr; v) rER

::; sup {91:(J-t; v) : J-t E Pse(X)}. Taking the supremum on the LHS over Ps(X) gives the conclusion. (2) is obvious. (3) Since 91:(.; v) is weak* upper semicontinuous and the set Ps(X) is weak* compact and convex, there exists at least one J-to E P, (X) such that 91:(J-to; v) attains its supremum, i.e., 9t(J-to; v) = Cs(v). Let P1 be the set of all such J-to. Since 91:(.; v) is weak* upper semicontinuous and affine on Ps(X), P1 is weak* compact and convex. By the Krein-Milman Theorem there is at least one extreme point J-t* in Pl. Then J-t* is also extremal in Ps(X). For, if J-t* = cqi; + (1-a)J-t2 for a E (0,1) and J-t1, J-t2 E Ps(X), then

Cs(v)

91:(J-t*; v)

= a91:(J-t1; v) + (1 - a)91:(J-t2; v) ::; Cs(v), which implies 9t(J-t1; v) = 91:(J-t2; v) = Cs(v) and J-tb J-t2 E Pl. By the extremality of J-t* we have J-t* = J-t1 = J-t2, so that J-t* is extremal in Ps(X). Therefore J-t* is ergodic by Theorem 11.3.2. In the rest of the section we discuss the transmission rate and the capacity of a stationary channel in a measure theoretic manner. Let (X, X, S) and (Y,~, T) be a pair of abstract measurable spaces with measurable transformations Sand T. A slightly different type of transmission rate is defined as follows.

Definition 9. Let v E Cs(X, Y) be a stationary channel. (1) The transmission rate 91:(J-t; v) of v w.r.t. J-t E Ps(X) is defined by

91:(J-t; v) = sup {HIJ-(2t, S)

+ HIJ-v(SJ3, T)

- HIJ-@v(2t x SJ3, S x T) :

2t E P(X), SJ3 E P(~)},

Chapter III: Information Channels

176

where Hp,(2(, S) = H(J-t, 2(, S), etc. (2) The stationary capacity Cs(v) and the ergodic capacity Ce(v) are respectively defined as in Definition 7. Remark 10. (1) In the above definition, the equality Cs(v) = Ce(v) for stationary ergodic channel v is not trivial. (2) In the alphabet message space case we have chosen 2( = 9Jt1(X) and ~ = 9Jt1(Y). Since they are the generators in the sense that V sn§{ = X and V Tn~ = nEZ

~,

nEZ

we obtained

by the Kolmogorov-Sinai Theorem. In the discussion from Definition 2 through Theorem 8 we fixed partitions 2( E P(X) and ~ E P(~). (3) Since J-t (8) v ~ J-t x J-tv and

Hp.(m.) + Hp.v(lJ3) - Hp.®v(m. x lJ3)

""

= L...J fL 0

(J-t x J-tv)(A x B)

v(A x B) log fL 0 v(A x 13) ,

A,B

where the sum is taken for A E 2( and B E relative entropy and Theorem 1.6.2 that

~,

it follows from the definition of the

9l(J-t; v) = H(J-t x J-tvlJ-t (8) v) d(J-t (8) v) d(J-t (8) v) = XxY d( J-t x J-tV ) log d( J-t x J-tV ) d(fL x fLV).

11

Note that this quantity is defined for all channels and input sources. Proposition 11. If Ps(X) is complete for ergodicity (cf. Definition 2.8) and v Cse(X, Y) is a stationary ergodic channel, then the equality Cs(v) = Ce(v) holds.

E

Proof. We only have to prove Cs(v) ~ Ce(v) assuming Cs(v) < 00. For an arbitrary e > 0 choose a stationary source J-t E Ps(X) and finite partitions 2( E P(X) and ~ E P(~) such that

Hp,(2(, S)

+ Hp,v(~, T)

- Hp,®v(2( x

~,S

x T) > Cs(v) -

€.

(6.10)

Then by Theorem 11.7.7 we can find transformation invariant bounded measurable functions hI (x), h 2 (y) and h 3(x, y) such that

177

3.6. Capacity and transmission rate

Hp,v(r.I3, T) = [h 2 (Y) p,v(dy),

HJ-t~v(2( X

23, S

X

JrJXXY r h

T) =

3(x,

y) p,@v(dx,dy).

As in the proof of Proposition 3, if we define t(x) by (6.8), then

=

LHS of (6.10)

L

rex) p,(dx).

Since t(·) is S-invariant there exists a sequence {t n (.)} of S-invariant simple functions such that t n t t on X. For each n ~ 1 write t n as kn

tn =

E

CYn,kIAn,k'

k=l

where {An,l,'" ,An,kn} E P(X) with S-lAn,k from the Monotone Convergence Theorem that

=

An,k

for I~ k ~ kn . It follows

Hence by (6.10) we see that

for some no

~

1. Consequently one has for some k o

Since Ps(X) is complete for ergodicity, there exists some 1-£* E Pse(X) such that 1-£* (Ano,ko) = 1. Thus

By the ergodicity of v we obtain Ce(v)

~

Cs(v), completing the proof.

178

Chapter III: Information Channels

3.7. Coding theorems In this section, Shannon's first and second coding theorems are fomulated and proved. Feinstein's fundarnentallemma is also proved. It is used to establish the above mentioned theorems. We use the alphabet message space setting: X = X~ and Y = YoZ for finite sets X o and Yo with the shifts Sand T, respectively. A channel v E C(X, Y) is said to be nonanticipatory if (c16) v(x, [Yn = b]) = v(x', [Yn = b]) for every nEZ, b E Yo, and x = (Xk), x = X with Xk = x~ (k :::; n).

(x~) E

Recall the notations mt~(X) and v(A,D): mt~(X) is the set of all messages of length n starting at time i in X, and v(A, D) is the common value v(x, D) for x E A, where A E X and D E~. Then Feinstein's fundamental lemma is formulated as follows:

Lemma 1 (Feinstein's fundamental lemma). Let v E Cs(X, Y) be a stationary, m-memory, m-dependent and nonanticipatory channel with the ergodic capacity C =

Ceo For any e > 0 there exist positive integers n = n(e) and N = N(e), messages UI , ... ,UN E mt~~n(X) and measurable sets Vb ... ,VN E A(mt~(Y)), the algebra generated by mt~ (Y), such that

(1) VinVj=0 (i#j); (2) u (Ui , Vi) > 1 - e, 1 :::; i :::; N; (3) N > en(C-e) . Proof. By the definition of C e there exists an ergodic source JL E P s e (X) such that e 9l(JL ; v) > C - 2".

In fact, we rnay assume 9l(JL;v) = C in view of Theorem 6.1 (2). Since v is stationary and ergodic by Theorem 3.4, JLV and JLQ9v are also stationary and ergodic. For n ~ 1 we denote

A = [x- mX- m+l · · ·Xn-I]

E VJt~~n(X),

D = [YOYI·· ·Yn-I] E VJt~(Y).

(7.1) (7.2)

Then, by 5MB Theorem (Theorern 11.5.1) 1 --logJL(A), n

1 1 --logJLv(D), --logJLQ9v(AxD) n n converge to HI-£(S) , Hl-£v(T) and HI-£®v(S x T) a.e. and hence in probability, respectively. Hence

..!:.log v(A, D) = ..!:.lo /l Q9 v(A x D) n

JLv(D)

n

g JL(A)JLv(D)

3. 7. Coding theorems

179

tends to Hj.£(S) + Hj.£v(T) - Hj.£Q9v(S x T) = 9t(JL; v) in probability (JL we can choose a large enough n = n(c) ~ 1 for which 1 v(A,D) It Q9 u ([ ;;: log Itv(D)

>C

~]) 2

Q9

v). Thus

> 1- ~2'

(7.3)

where [... ] indicates the set

For each A E oot~+n(X) let

VA =

D) Itv(D) > C U{D E 9.n (Y ) : ;;:1 log v(A, 0

n

e}

2 .

Then we get LHS of (7.3)

= JL Q9 v (U(A

x VA))

=L

A

L JL(A)v(A, VA) > 1 A

JL Q9 v(A x VA)

A

~

(7.4)

2

If A and D are such that ~log:<:.G-~, then v(A,D) Taking a sum w.r.t. D over VA, we have

> JLv(D)en(C-~).

Now choose a UI E oot~+n(X) such that

v(U I , Vu1 ) > 1 - e, which is possible in view of (7.4), and let VI = VUl. If there is a U2 E oot~+n(X) such that v(U2 , VU2 - VI) > 1 c, then we let V2 = VU2 - VI. If, moreover, there is a U3 E

mt~+n(X)

such that

then we let V3 = VUa - (VI U V2 ) . This procedure terminates in a finite number of steps, and we write the so obtained sets as

Chapter III: Information Channels

180

It follows from the construction of these sets that (1) and (2) are satisfied. Furthermore, we have for A E 9Jl;;;'~n(X)

which implies N

V(A,VA) S V(A, }dll~ + V(A, VA -

)

jld N

Vj )

~V(A,UVj)+1-e. J=l

Taking the average over

9Jl;;;'~n(X)

w.r.t. J-L, we get

~ M(A)v(A,VA) S /],V

(jld N

Vj )

+1-

e,

which, together with (7.4), yields that (7.6) Since Vj ~ VUj (1

< j < N), we have by (7.5) that

Thus, invoking (7.6), we see that

if we choose n

= n(e)

~ 1 such that

n(e) ~ ~ log:. Thus (3) holds.

An immediate consequence of the above lemma is: Corollary 2. Let u E Cs(X, Y) be a stationary, nonanticipatory, m-memory and m-dependent channel with the ergodic capacity C = Ceo Then, for any sequence

181

3. 7. Coding theorems

{cn}~=m+l such that

Cn

+0,

there exist a sequence of positive integers {N =

N(n)}~=m+l' a family offinite sets of messages {ui n ) , ..• ,u];)} c m;;"+n(X) (n ~ m + 1) and a family of finite sets of measurable sets {V1(n), . . . , vir)} c A(m~ (Y)) (n ~ m

+ 1)

such that for n ~ m

n ujn)

+1

0 (i =I j); (2) v (Ui(n) , Vi(n)) > 1 - Cn, 1 ::; i (3) N > en(C-c

(1) Ui(n)

=

::; N;

n ) .

To formulate Shannon's coding theorems, we consider another alphabet Xb {a~, ... ,a~} and the alphabet message space (X', X', 8 '), where X' = Xb'L and 8 ' is the shift on X'. A code is a one-to-one measurable rnapping ip : X' -+ X. If we let

vcp (x', A) = 1A (
X'

E

X',A

E

X,

then [X', vCP, X] is called a noiseless channel. This means that, if vcp (x', A) = 1, then we know that x' E X' is encoded as
where r ~ 1 is an integer, called the code length, and
vcp(x', D) = v(
X'

E X',D E~,

then we have a new channel [X', vcp? Y] or Vcp E C(X ', Y), which is called the induced channel. Suppose that v E Cs(X, Y) is a stationary, nonanticipatory, m-rnernory and mdependent channel, and