Counterexamples in Probability, 2nd Edition

RANDOM VARIABLES AND BASIC CHARACTERISTICS

79

where, E ]RI, and G(x), x E ]RI, is non-decreasing left-continuous function of bounded variation and G(-oo) = O. Now let us introduce another notion. The r. v. X, its d.f. F and its ch.f. ¢ are called stable if for every n ~ I there exist constants an and bn > 0 and independent r.v.s X I , ... , X n distributed like X such that

or, equivalently, F

(xt;:n ) = [F(x)]*n, or [¢(t)]n

¢(bnt)einnt.

The basic result concerning stable distributions is as follows (see Chow and Teicher 1978; Zolotarev 1986). The r. v. X with ch.f. ¢ is stable iff ¢ admits the following canonical representation:

(2)

¢(t) = exp {i,t 1

where, E ]R , 0 < a ~ 2,

1f31

cltlO

[1

+ if3I:1 w(t, a)1}

~ 1, c ~ 0 and

( ) _ {tan 11Ta, w t, a (2/71-} log Itl,

if a =1= 1 if a = 1.

Recall that (2) is also known as the Levy-Khintchine representation. In particular, if, = 0, f3 = 0, we obtain the symmetric stable distributions. They have ch.f.s of the type exp( -cltIO) where c ~ 0,0 < a ~ 2. A detailed investigation of the infinitely divisible distributions and the stable distributions can be found in the books by Gnedenko and Kolmogorov (1954), Lukacs (1970), Feller (1971), Linnik and Ostrovskii (1977), Loeve (1978), Chow and Teicher (1978) and Zo]otarev (1986). The next examples illustrate different properties of infinitely divisible and stable distributions. Two examples deal with random vectors.

9.1.

A non-vanishing characteristic function which is not infinitely divisible

Let the r.v. X with ch.f. ¢(t), t E ]RI, be infinitely divisible. Then ¢ does not vanish. The example below shows that in general the converse is not true. Consider the discrete r.v. X which takes the values -1,0, 1 with probabilities ~, krespectively. The ch.f. ¢ of X is

k,

¢(t) = ke-it + ie itO + keit = k(3 + cos t). Obviously ¢( t) > 0 for all t E ]R I, so ¢ does not vanish. Nevertheless, X is not infinitely divisible. To see this, let us assume that X can be written as

(l)

80

COUNTEREXAMPLES IN PROBABILITY

where XI and X2 are i.i.d. r.v.s. Since X has three possible values, it is clear that each of XI and X 2 can take only two values, say a and b, a < b. Let P[XI = a] = p, P[XI = b] = 1 - p for some p, 0 < p < 1. Then XI + X2 takes the values 2a, a + b and 2b with probabilities p2, 2p(I - p) and (I - p)2 respectively. Thus we should have the relations

2a=-I,

a+b=O,

2b=l,

p2=k,

2p(p+l)=~,

(1-p)2=k

which are clearly incompatible. Hence the representation (1) is not possible, implying that X is not infinitely divisible.

9.2.

If IcPl is an infinitely divisible characteristic function, this does not always imply that ¢ is also infinitely divisible

Recall that if cP is an infinitely divisible ch.f. then its absolute value IcPl is so. It is not so trivial that in general the converse statement is false. This was discovered by Gnedenko and Kolmogorov (1954) and we present here their example of a ch.f. cP such that IcPl is infmitely divisible, but ¢ is not. Consider the function

(I)

¢(t)

where 0

= [(1- b)/(I

< a ::; b <

- a)][(1

+ ae-it)/(I

- be it )],

I. Obviously ¢ is continuous, ¢(O)

t E jRl

= 1 and

It follows that ¢ is the ch.f. of a r.v. X with

P[ X

= - 1] = (I

- b) a / (I - a),

P[ X

= k]

= (1 - b) (1

+ ab) bk / (I

- a),

k = 0, 1,2, .... Let us show that ¢ is not infinitely divisible. Indeed, we find that 00

(2)

log¢(t)

= L)(-I)k-lk-lak(e-itk -

1)

+ bkk-I(e itk

- 1)].

k=1

We can also write log ¢(t) in its canonical fonn (see the introductory notes to this section; the Levy-Khintchine fonnula) by taking "I = E~I (b k + (-1 )k a k)/(k 2+ 1) and G(x) to be a function of bounded variation with jumps of size kb k /(k 2 + 1) at x = k and (_I)k-I ka k /(k 2 + I) at x = -k for k = 1,2, .... However, G is not monotone, which automatically implies that ¢ cannot be infinitely divisible. Furthennore, the function

¢(t) = [( I - b)/( 1 - a)][( I

+ ae it )/( 1 -

be-it)]

RANDOM VARIABLES AND BASIC CHARACTERISTICS

81

is also a ch.f. but not infinitely divisible. Our next step is to show that the function

1/J(t) = 1¢(t)12 = ¢(t)¢(t) is infinitely divisible. Note that 1/J is a ch.f. as a product of two ch.f.s. It is easy to write firstly log¢(t) in the fonn (2) and then obtain 10g1/J(t), namely 00

00

Thus in the Levy-Khintchine fonnula for 10g1/J(t) we can take 'Y = 0 and G(x) to be a non-decreasing function with jumps of size k(k 2 + 1)-1 [b k + (-1 )k-I ak] at the points x = ±k, k = 1,2, .... Since this representation of 10g7f!(t) is unique, we conclude that the ch.f. 1/J is infinitely divisible. Moreover I¢I = (1¢12)! is also infinitely divisible despite the fact that ¢ given by (1) is not. Another interesting observation is that the infinitely divisible ch.f. t/J is the product of the two non-infinitely divisible ch.f.s ¢ and

;p.

9.3.

The product of two independent non-negative and infinitely divisible random variables is not always infinitely divisible

(i) Define two independent r.v.s X and Y having values in the sets {O, 1,2,3, ... } and { 1, 1 + e, 1 + 2e, 1 + 3e, ... } respectively where 1 < e < ~. The corresponding probabilities for X and Y are {Po,PI,Pl, ... } and {QO,QI,q2,"'} wherepj > 0, Lpj = 1, Qj > 0, Lqj = 1. Consider the product Z = XY and suppose it is infinitely divisible. Then

(1) where ZI and Z2 are i.i.d. r. v.s. Evidently, the 'first' six possible values of Z are 0, 1,2, 1 + e, 3, 1 + 2e. It follows that 0,1 and 1 + e are among the values of ZI (and hence of Z2). But this implies that 2 + c is a possible value of Z. Since 2 + e < 1 + 2e we get a contradiction. Consequently a relation similar to (I) is not possible. Thus Z cannot be infinitely divisible. Notice that X and Y take their values from different sets. The same answer concerning the non-infinite divisibility ofthe product XY can be obtained in the case of X and Y taking values in the same space. (ii) Let us exhibit now an example in which the reasoning is based on the following (see Katti 1967). Suppose {Pn, n E No} is a distribution with Po > 0 and PI > O. Then {Pn} is infinitely divisible iff the numbers Tko k = 0, 1, ... , defined by n

(2)

(n

+ 1 )Pn+ I = L k=O

T kPn -

k,

n

= 0, 1, 2, ...

82

COUNTEREXAMPLES IN PROBABILITY

are all non-negative. Let us use this result to prove a new and not too well known statement: let ~ and T) be independent r. v.s each having a Poisson distribution Po (.~). Then both, and T) are 'T) is not. infinitely divisible, but the product X Indeed, take n > 1 such that n + 1 is a prime number. Then

=

Pn+1 ::= P[X = n + I] = P['1J = n + I] = P[, = 1,T)::= n + 1] + P[, = n + 1,T) = 1] = 2)..n+l e-2>"/(n

+ I)!.

The number n itself is even and hence n has at least two (integer) factorizations: n = I . n = 2· (n/2). Therefore

Pn = P[X

= n]

= )..n/2+2 e -2>" /(n/2)!. 0, PI = )..2e- 2>" > 0, and so

> 2· ()..2e->" /2!) . ()..n/2e->" /(n/2)!)

Obviously Po = P[X = 0] = I - (1 - e->")2 > ro = PI/Po> 0. Further, suppose that rl, r2, ... ,rn-I in (2) are all non-negative. Let us check the sign of rn. We have

rnPo ~ (n

+ 1)Pn+1

- rOPn

< 2)..n+2e -2>" In!

- ro)..n/2+2e- 2>" /(n/2)!

= e- 2A )..n/2+2 [2)..n/2/n! - ro/(n/2)!].

°

Since).. > is fixed and I/n! goes to zero as n -t 00 faster than 1/(n/2)!, we conclude that for sufficiently large n the number rn becomes negative. This does not agree with the property in (2) that all r n are non-negative. Hence the product T)' of two independent Poisson r. v.s is not infinitely divisible.

9.4.

Infinitely divisible products of non-infinitely divisible random variables

There are many examples of the following kind: if X is a r. v. which is absolutely continuous and infinitely divisible and XI, X2 are independent copies of X, then the product XI X 2 is again infinitely divisible. As a first example take X "" N(O, I). Then Xl X2 has a ch.f. equal to 1/( I + t 2) I /2 and hence X I X 2 is infinitely divisible. As a second example, take X ......, e(O, 1), i.e. X has a Cauchy density f(x) = 1/( 7l'( 1 + x 2)), x E Ii I. If XI and X 2 are independent copies of X, it can be checked that the ch.f. of the product XIX2 is infmitely divisible. These and other examples (discussed by Rohatgi et al (1990» lead to the following question. Suppose XI and X2 are independent copies of the absolutely continuous r. v. X. Suppose further that the product Y = XI X 2 is infinitely divisible. Does this imply that X itself is infinitely divisible? Let Y be a r. v. distributed N(O, I). Then there exists a r. v. X such that by taking two independent copies, XI and X 2 , we obtain XI X 2 4: Y (for details see Groeneboom and Klaassen (1982». Thus P[IYI > x 2] ~ (P[lXI > x])2 which implies that P[IXI > x] ~ (P[lYI > x 2 ]) 1/2 = O(e-

x4 /

4

)

as

x -t

00.

83

RANDOM VARIABLES AND BASIC CHARACTERISTICS

Referring to the paper of Steutel (1973) for details we conclude that X cannot be infinitely divisible. Hence the answer to the above question is negative.

9.5.

Every distribution without indecomposable components is infinitely divisible, but the converse is not true

Following tradition, denote by 10 the class of distributions which have no indecomposable components. Recall that F E 10 means that the ch.f. ¢ of F cannot be represented in the form ¢ ¢1 ¢z where ¢1 and ¢2 are ch.f.s of non-degenerate distributions. Detailed study of the class 10 is due to A. Va. Khintchine. In this connection see Linnik and Ostrovskii (I977) where among a variety of results, the following theorem is proved: the class 10 is a subclass of the class of infinitely divisible distributions. Our purpose now is to show that this inclusion is strong. Indeed, take the following ch.f.:

=

(I)

¢(t)=(I-a)(l-aeit )-"

O
tElRl.

The representation

4>(t) = exp[log(l - a) -log(l - ae")] = exp

[~ann-'(e"n

I)]

shows that ¢ is a limit of products of ch.f.s corresponding to Poisson distributions. Then (see Gnedenko and Kolmogorov 1954; Loeve 197711978) the ch.f. ¢ is infinitely divisible. Further, the identity 1/(1 - x) = f1~0(1 + xZ"').lxl < 1 implies that 00

¢(t) =

II (1 + aZ"'e it2"')/(1 + a2"'). k=O

Recall that (1 + a 21< eitZI< )/(1 + a 21<) is a ch.f. of a distribution concentrated at two points, namely 0 and 2k. However. such a distribution is indecomposable (see Example 9.1). Hence the ch.f. ¢ defined by (1) is infinitely divisible but ¢ does not belong to the class 10.

9.6.

A non-infinitely divisible random vector with infinitely divisible subsets of its coordinates

Let (XI, Xz, X 3) be a random vector and ?/J(tl, t2, t3), tl, t2, t3 E lRl its ch.f.:

?/J(tl, t2. t3)

= E [exp[i(tIX\ + t2X2 + t3X3)]] .

The vector (X l ,X2 ,X3 ) is said to be infinitely divisible jf for each Q' > 0, ?/JO< (tl, t2, t3) is again a ch.f. Obviously this notion can be introduced for random

84

COUNTEREXAMPLES IN PROBABILlTY

vectors in lin with n > 3. We confine ourselves to the three-dimensional case for simplicity. Let us note that if (X], X2, X3) is infinitely divisible, then each subset of its coordinates XI. X 2 • X3 is infinitely divisible. This follows easily from the properties of the usual one-dimensional infinitely divisible distributions. Thus it is natural to ask whether the converse statement is true. Consider two independent r.v.s X and Y each N(O, 1). Let ZI=X 2,

Z2=XY,

Z3=y2.

It is easy to check that each of ZI, Z2. Z3 is infinitely divisible. Moreover, any of the two-dimensional random vectors (ZI, Z2), (ZJ, Z3) and (Z2, Z3) is also infinitely divisible. However, the vector (ZI' Z2, Z3) is indecomposable, it has trivariatc gamma distribution which is not infinitely divisible. For details we refer the reader to works by Levy (1948). Griffiths (1970) and Rao (1984).

9.7.

A non-infinitely divisible random vector with infinitely divisible linear combinations of its components

If (X, Y) is an infinitely divisible random vector, then any linear combination Z = a,X + a2Y, ai, a2 E JRI is an infinitely divisible r.v. The question to be considered is whether the converse is true. This problem, posed by c.R. Rao, has been solved by Ibragimov (1972). The following example shows that the answer is negative. 1 2 For x / and 0 < £ < (XIJ X2) E JR2, Ixi = (XI + define the function

xn

=

ae(x) =

{

I, 0,

if if

-c,

if

i,

Ixl ::; ~ £ or 4 £ < Ixl ::; Ixl > 1 ~ - c < Ixl ::; ~ + c.

1

Let Ae (U). U E JR 2 • be the signed measure with density a e• that is

and also introduce the function

For all sufficiently sma]) £ > 0, tPe is positive definite and hence some d.f. FE in JR 2. Indeed, from (I)

tPe

is the ch.f. of

85

RANDOM VARIABLES AND BASIC CHARACTERISTICS

2 where c = exp[-Ae(1It )]. Thus Fe can be written in the fonn Fe = c(G o + Ge ) where Go is a probability measure with Go({O}) = 1 and Ge is a measure with density Je(x) = E~=I a~(x) In!. Furthennore we can check that for all small E,

ii~(x) 2: O.

ii;(x) = ( ae(x - 11,)ae(u) d11, 2: 0 and

iR2

Hence for n 2: 4 we have a~(x) = ii~n-2) * a;(x) 2: O. For small E, ae(x) is close to the function a6(x). It is easy to see that for! ~ x < ~ we have infx ii~(x) = CI > O. Thus for small E, a;(x) > 2E if E < Ixl :::; + E. Evidently this implies that 2 Je(x) 2: 0 for all x E 1It . Therefore Fe described above is a probability measure in

t-

t

tR 2.

Denote by (XI, X 2 ) a random vector with dJ. Fe. Since Ae is a signed measure (its values are not only positive), Fe cannot be infinitely divisible. It remains to be shown that any linear combination QIX I + Q2X2, QI, Q2 E lIt', has a distribution which is infinitely divisible. Indeed, for s E tR'

rPo:(s) := E{exp[is(Q,XI

+ Q2X 2)]}

= 1fle(QIS,Q2S)

= eX P [k2(e iS (0:,X) -1)dAe(X)].

Denoting (Q, x)

rPo:(s) = exp

= u where u E tR I we can write rPo: (s)

[1:

(e

isu

-

in the fonn

dHo:(11,) =

I) dHo:(U)] ,

1

dAe(x) duo

(o:,x)::; u

Since for sufficiently small E every strip {x : 11,1 :::; (Q, x) :::; 11,2} has positive Aemeasure, we conclude (again see Ibragimov 1972) that the function H 0: (u), u E lIt I, is a d.f. and moreover, rPo:(s) = 1fle(OI.'i, 02S), .'i E tRl, is a chI of a distribution which is infinitely divisible. Thus we have established that any linear combination QIX I + Q2X2 is an infinitely divisible r.v. but (XI, X2) is not an infinitely divisible vector.

9.S.

Distributions which are infinitely divisible but not stable

Usually we introduce and study the class of infinitely divisible distributions and then the class of stable distributions. One of the first observed properties is that every stable distribution is infinitely divisible. Let us show that the converse is not always true. (i) Let X be a r.v. with Poisson distribution, X

P[X where the parameter .\ (I)

= n] =

.\ne->'In!,

"-J

1>0(.\), that is

n = 0, 1,2, ...

> 0 is given. If rP is the chJ. of X rP(t) = exp['\(e it

-

1)],

then

t E tRl.

86

COUNTEREXAMPLES IN PROBABILITY

Since ¢(t) = [¢n{t))n for ¢n(t) = exp[An- 1 (e it - 1)) and ¢n is again a ch.f. (of Po(A/n}) then X is infinitely divisible. However, ¢ from (I) does not satisfy any relation of the type ¢(b l t)¢(b 2t) = ¢(bt)ei-rt (see the introductory notes to this section). This means that the Poisson distribution is not stable despite the fact that it is infinitely divisible. (ii) Let Y be a r.v. with Lap/ace distribution, that is, its density 9 is

(1/2A) exp[-Ix - ttl/A],

g(x)

x E Rl

where J.1 E R I , A > O. For the ch. f. t/J of Y we have

(2) It is not difficult to verify that t/J is infinitely divisible. But t/J from (2) does not satisfy any relation of the type t/J(b, t)t/J(b2 t) = t/J(bt)ei-rt and hence '!/J is not stable. Therefore the Laplace distribution is an example of an absolutely continuous distribution which is infinitely divisible without being stable. (iii) Suppose the gamma distributed r. v. Z has a density

g(x)

(l/v'21T)x- I / 2e- x ,

x >0

=

(and g(x) 0 for x SO). Then by using the explicit form of the ch.f. of Z we can show that Z is infinitely divisible but not stable. An additional example of a distribution which is infinitely divisible but not stable is given in Example 21.S.

9.9.

A stable distribution which can be decomposed into two infinitely divisible but not stable distributions

Let X be a r.v. with Cauchy distribution e(], 0), that is, its density is

f(x)

1/[11"(1 + x 2 )), x E RI.

If ¢ denotes the ch.f. of X, then ¢(t) = e- 1tl , t E RI. It is wen known that this distribution is stable. Let us show that X can be written as

(I) where X I and X 2 are independent r. v.s whose distributions are infinitely divisible but not stable. For introduce the following two functions:

We claim that ¢, and ¢2 are ch.f.s of distributions which are infinitely divisible. This follows from the fact that each of ¢I, ¢2 can be expressed in the fonn

RANDOM VARIABLES AND BASIC CHARACTERISTICS

87

u

exp[- f~ fo 'IjJ(v) dvdu) with a suitable integrand 'IjJ and the only assumption is that 'IjJ is a ch.f. Then our conclusion concerning ¢I and ¢2 is a consequence of a result of Lukacs (1970, Th. 12.2.8). It is easy to verify that ¢I and ¢2 are not stable ch.f.s. Thus we have

(2) Now take two independent r.v.s, say XI and X 2 , whose ch,f.s are ¢I and ¢2 respectively. It only remains to see that (2) implies (1). Therefore we have constructed two r.v.s Xl and Xz which are independent, both are infinitely divisible but not stable and they are such that the sum Xl + X2 has a stable distribution.

SECTION 10.

NORMAL DISTRIBUTION

We say that the r. v. X has a normal distribution with parameters a and a 2 , a E IR I, a > 0, if X is absolutely continuous and has a density 1 exp [ (x - a)2] f(x) -- a-J2ir - 2a2'

(1)

xE

lll)1. J.l\\.

In such a case we use the notation X "" N(a, a 2 ). It is easy to write explicitly the d.L corresponding to (1). Consider the particular case when a = 0, a = I. We obtain the functions

(2) and

(3) These two functions,

, are caBed a standard normal density function, and a standard normal d.f respectively. They correspond to a r. v. N(O, 1). Recall that the r. v. X "" N( a, a 2 ) has EX a, V X a Z and a ch.f ¢(t) exp(iat - ~a2t2). If a 0, then all odd-order moments are zero, that is,a2n+1 E[X2n+l) = 0, while the even-order moments are a2n = E[X 2n J =

=

a 2n (2n

=

=

=

I)!!.

Consider now the random vector X = (Xl"'" Xn). If EXi = ai, i - I , ... , n, then a = (al,"" an) is called a mean value vector (or vector of the expectation) of X. The matrix C = (Cij) where eij = E[(X i - ai)(Xj aj)), i,j = 1, ... ,n is caBed a covariance matrix of X. We say that X has an n-dimensional normal distribution if X possesses a density function

COUNTEREXAMPLES IN PROBABILITY

88

(4)

!(XI, ... ,X n ) = 2 (27r)-n/2I D II / exp

{-i .t

dij(Xi - ai)(Xj - aj )},

(XI, ... , xn) E IRn.

l,J=1

Here the matrix D = (d ij ) is the inverse matrix to C. Clearly, D exists if C is positive definite and IDI :=detD. Note that we could start with the vector a (a), ... , an) E IR n and the symmetric positive definite matrix C = (Cij), then invert C to yield matrix D, and finally use the vector a and the matrix D to write the function! as in (4). This function f is an n-dimensional density and thus there is a random vector, say (XI"'" X n ), whose density is f. By definition this vector is called nonnally distributed, and (4) defines an n-dimensional nonnal density. For some of the examples below we need the explicit fonn of (4) when n = 2. The two-dimensional (or bivariate) nonnal density can be written as

=

(5)

!(XI,X2) =

1 27rCTI O'

2V 1 -

p2

x

I [(XI - al)2 2 (XI - al)(x2 - a2) exp - p { 2(l_p2) O'f 0'10'2

(X2 - a2)2] } + ~----;;,.........:0'1

where 0'1,0'2 > 0 and Ipi < I. If (XI, X2) is a random vector with density (5) then EX I = al,EX2 = a2, VX I = O'r, VX 2 = O'~ andpequalsthecorrelationcoefficient

p(XI' X2).

The nonnal distribution over IR I and IR n is considered in almost all textbooks and lecture notes. We refer the reader to the books by Anderson (1958), Parzen (I960), Gnedenko (I962), Papoulis (1965), Thomasian (1969), Feller (1971), Laha and Rohatgi (1979), Rao (1984), Shiryaev (1995) and Bauer (1996). In this section we have given various examples which clarify the properties of the nonnal distribution.

10.1.

Non-normal bivariate distributions with normal marginals

(i) Take two independent r.v.s ~I and 6, each distributed N(O, ]). Consider the following two-dimensional random vector:

Obviously the distribution of (X I, X2) is not bivariate nonnal, but each of the components XI and X 2 is nonnally distributed. (ii) Suppose h(x), x E IRI, is any odd continuous function vanishing outside the interval [-I, 1] and satisfying the condition Ih(x)l:S (27Te)-1/2. Using the standard

89

RANDOM VARIABLES AND BASIC CHARACTERISTICS

normal density

f(x,y)

( 1)

=
It is easy to check that f(x,y), (x,y) E jR2, is a two-dimensional density function and f(x, y) is not bivariate normal, but the marginal densities fl (x) and Jz(y) are both normal. The function h in (1) can be chosen as follows:

where ][-I,IJ (-) is the indicator function of the interval [-I, I]. (iii) For any number e,

lei

s I, define the function

H(x, y) = <1>(x)<1>(y)[1 + e(l - <1>(x))(l - <1>(y))L

(x, y) E

jR2

(<1> is the standard normal d.f.). It is easy to check that H is a two-dimensional d.f. with marginal distributions <1>( x) and <1>{y) respectively. Obviously, if e f; 0, H is non-normal. Another possibility is to take the function

h(x,y) =
+ t:{2<1>{x)

- I)(2<1>{Y) - I)].

Then h(x,y) is a two-dimensional density function with marginals
(iv) Consider the following function:

f(x,y)

[lj(7f(1 - p2)I/Z)] cxp[-!p-2(x2 - 2pxy + y2)), { 0,

° <°

if xy ~

if xy

where p E (-1, I). It is easy to verify that f is a two-dimensional density function. Denote by (X, Y) the random vector whose density is f(x,y). Obviously the distribution of (X, Y) is not normal, but each of the components X and Y is distributed N(O, 1).

10.2. If (X I, X 2 ) has a bivariate normal distribution then X h X 2 and X I + X 2 are normally distributed, but not conversely Let (XI, X2) have a bivariate normal distribution and f(XI, xz), (Xl, xz) E jRz be its density. Then each of the r.v.s X" Xz and XI + Xz has a one-dimensional normal distribution. We are interested in whether the converse is true. Suppose XI, X 2 are independent r.v.s each distributed N(O, I). Then their joint density is f(Xl, X2) =
90

COUNTEREXAMPLES IN PROBABILITY

SI===========t========== s;

s{ ====1

s~

Figure 2

Firstly, let us draw eight equal squares at a fixed distance from the origin a and located symmetrically about the axes OXI and OX2 as shown in figure 2. Put alternately the signs ( +) and (-) in the squares. Let the small positive number c denote the amount of 'mass' which we transfer from a square with (-) to a square with (+). Now define the function

where Q+ is the union of the squares with (+) and Q- the union of those with (-).

91

RANDOM VARIABLES AND BASIC CHARACTERISTICS

For such squares we can choose c > 0 sufficiently small such that g(Xl' xz) 2: 0 g(Xl, xz)dxl dX2 :=:: 1. for all (Xl, X2) E IRz. From (1) we find immediately that f Hence 9 is a density function of a two-dimensional random vector, say (Yl , Y2 ). Next we want to find the distributions of Y 1 , Y2 and YI + Y 2. The strips drawn in figure 2 will help us to do this. These strips can be arbitrarily wide and arbitrarily located but parallel to OXI or OX2 or to the bisector of quadrants II and IV. Evidently the strips either do not intersect any of the squares, or each intersects just two of them, one signed by (+) and another by (-). Since the total mass in any strip remains unchanged and we know the distribution of (Xl, X2) (recall it is nonnal), then we easily conclude that Y, '"-J N(O, 1), Y2 '"-J N(O, 1), YI + Y2 '"-J N(O, 1). For example. look at the pairs of strips (8,,8D, (82,82), (83,8D. However it is clear that the distribution of (Yl, Y2) given by the density (1) is not bivariate nonnal. Therefore the nonnality of Yi, Y2 and Y, + Y2 is not enough to ensure that (Y" Yz) is nonnally distributed.

fRl

10.3.

A non-normally distributed random vector such that any proper subset of its components consists of jointly normally distributed and mutually independent random variables

We present here two examples based on different ideas. The first one is related to Example 7.3. (i) Let the r.v. X have a distribution N(a, (12) and let f be its density. Take n r.v.S, say XI, ... ,Xn , n 2: 3, and define their joint density gn as follows:

(1)

gn(Xl, ... ,xn )

:=::

[TI~1 f(Xi) 1[1 + TIj=1 (Xj -

a)f(xj)

1'

(Xl,""X n ) E ]Rn.

Firstly we have to check that gn is a probability density. Since in this case we know f explicitly, f(x) = (21r(12)-1/2 exp [_(x - a)2/2(12], we can easily find that f~oo (x - a) f2 (x) dx :=:: O. Then we can derive that gn is non- negative and its integral over IR n is 1. Thus qn, given by (1), is a density and, as we accepted, gn is the density of the vector (Xl"'" Xn). Let us choose k of the variables X" ... ,Xn , 2 S; k :S n 1. Without loss of generality assume that Xl, ... ,Xk is our choice. Denote by gk the density function of (XI,' .. , Xk). From (1) we obtain

(2) (recall that f is the density of X and X '"-J N( a, (12)). Therefore the variables X I, ... ,Xk are jointly normally distributed and, moreover, they are independent. This conclusion holds for all choices of k variables among XI, ... , Xn and, let us repeat, 2 S; k S n - 1. It is also clear that each X j, j = 1, ... , n has a nonnal distribution N(a, (12).

COUNTEREXAMPLES IN PROBABILITY

92

Therefore we have described a set of 71, r. v.s which, according to (1), are dependent but, as it follows from (Z), are (71, 1)-wise independent.

(ii) Let (XI, ... , Xn) be an n-dimensional normaHy distributed random vector. Then its distribution is determined uniquely if we know the distributions of all pairs (Xi, Xj). i, j = I, ... , n. This observation leads to the question of whether (Xl) ... ,Xn ) is necessarily normal if all the pairs (Xi, Xi) are two-dimensional normal vectors. We shall show that the joint normality of aU pairs and even of all (71, - 1)-tuples does not imply that (XI, ... ,Xn ) is normally distributed. (Look at case (i) above.) Firstly, let 11, 2:: 3 and (~], ... ,~n) be such that ~j = ± I and any particular sign vector (YI, ... )Yn) is taken with probability p if n;::::1 Yj == + 1 and with probability q == 2-(n-l) - p if nj::::::l Yj == -1 Here 0 ~ p ~ 2-(n-I). It is not difficult to see that all subsets of 11, - I of the r.v.s ~], ... , ~71 are independent (that is, any 71, - I of them are mutually independent). Moreover, if p i- z-n, all 71, variables are not mutually independent. Indeed, if 1 ~ k ~ 11, the vector (YII , ... , Yi,.) can be extended in zn-k-l ways to a vector (YI,"') Yn) with n;=l Yj == 1 and in as many ways to one for which nj:;;;;;] Yj == 1. Thus

C == P[1,,11

C ]= == Yi,.

Yil , ••• ) I"i,.

Zn-k-l

P + zn-k-l q ==

Z-k

and this equality holds for any k ~ 71, - 1. Hence 6, ... , ~n. are (11, - I) -wise 1) ... ) ~n 1] = P it is obvious that 6) ... ,~n are not independent. Since P[~l 71 independent when p i- 2- • Now take Z,) ... ) Zn to be 11, mutually independent standard normal r. v.s which are independent of the vector (~I"") ~n). Define a new vector (Xl, ... , Xn) where Xj == ~j 1Zj I, j = 1, ... , n. Then clearly the Xj are again standard normal. The independence of the Zj together with the above reasoning concerning the properties of the vector (~I" .. ,~n) imply that aU subsets of 11, - 1 of the variables Xl, ... ,Xn are independent. Thus any (11, I)-tuple out of XI, . .. ,Xn hasan (11,-1 )-dimensional normal distribution. It remains for us to clarify whether all 11, variables Xl, ... ) Xn are independent. It is easy to see that

P[ X I > 0, ... , X n > 0] = P[6 == 1, ... ,~n == 1] == p. We conclude from this that if p ::j:. 2 n then the variables Xl, ... , Xn are not independent and not normally distributed. Let us note finally that in both cases, (i) and (ii), the joint normality and the mutual I of the variables X I, ... , Xn do not imply that the vector independence of any n (Xl) ... , Xn) is normally distributed.

10.4. The relationship between two notions: normality and uncorrelatedness Let (X, Y) be a random vector with normal distribution. Recall that both X and Y are also normally distributed, and if X and Yare uncorrelated, they are independent.

93

RANDOM VARIABLES AND BASIC CHARACTERISTICS

The examples below will show how important the normality of (X, Y) is. (i) Let X '" N(O, I). For a fixed number C

Y _ { X, -X,

2:: 0 define the r.v. Y by if IXI if IXI

:s; c

2:: c.

It is easy to see that Y ,..., N(O, I) for each c. Further,

E[XY]

= E[X2 [(IXI

c)] - E[X2 [(IXI > c)].

=

This implies that E[XY] -I if c = 0 and E[XY] -t I as c -t 00. Since E[XY] depends continuously on c, then there exists Co for which p(X, Y) = E[XY] = O. In X2i.p(x) dx - 1 = 0 fact, Co :;::;:: 1.54 is the only solution of the equation E[XY] = 4 (i.p is the standard normal density). For this Co the r. v.s X and Y are uncorrelated. However, P[X > c, Y > cJ = 0 :j:. P[X > cJP[Y > c] and hence X and Yare not independent.

J;o

(ii) Let i.pl (x, y) and i.p2(X, y), (x, y) E m,2 be standard bivariate normal densities

with correlation coefficients PI and P2 respectively. Define

f(x, y)

= Cli.pl (x, y) + C2i.p2(X, v),

(x, y) E m,2

where CI, Cz are arbitrary numbers, CI ,C2 2:: 0, CI + Cz = 1. One can see that f is non-normal if PI :j:. p2. If we denote by (X, Y) a random vector with density f then we can easily find that X '" N(O, 1), Y '" N(O, 1). Moreover, the correlation coefficient between X and Y is P = CIPI + C2P2. Choosing CI, C2, PI, P2 such that CI PI + C2P2 0, we obtain two normally distributed and uncorrelated r.v.s X and Y. However, they are not independent. (iii) Let (X, Y) be a two-dimensional random vector with density

f(x,y)

= (27rv'3)

I

{exp[-~(x2+xy+y2)] +exp[-~(x2 _xy+y2)]},

(x, y) E Il~? Obviously the distribution of (X, Y) is not bivariate norma1. Direct calculation shows that X ,..., N(O, I). Y ,..., N(O, 1) and E[XYJ = O. Thus X and Yare uncorrelated but dependent. + i6 and Y = 6 + i~4 where i = Ff and (6,6,6,~4) is a normally distributed random vector with zero mean and covariance matrix

(iv) Let X =

6

o

c

[j

-1

o

0 -1 1

0

-~lo . 1

94

COUNTEREXAMPLES IN PROBABILITY

The reader can check that C is a covariance matrix. Since X and Y are complexvalued, their covariance is

E[XY] = E[66 + 6~4] + iE[66 - 6~4] = 0 + i( -1 + 1) = O. Hence X and Y are uncorrelated. Let us see if they are independent. If so, then 6 and ~4 would be independent, and thus uncorrelated. But E[6~4] = -1. This contradiction shows that X and Y are dependent.

10.5. It is possible that X, Y, X + Y, X - Y are each normally distributed, X and Yare uncorrelated, but (X, Y) is not bivariate normal Consider the following function:

where (x, y) E

m. 2 and the constant c > 0 is chosen in such a way that

In order to establish that f is a two-dimensional probability density function and then derive some other properties, it is best to find first the Fourier transform ¢ of f. We have

¢(s, t)

= / L2 exp(isx + ity)f(x, y) dx dy = exp [_~(S2 +t 2 )] + 3'2 st (S2 -

2

t )exp

[-c -

~(S2 +t 2 )]

,

(s,t)

E

m. 2

From this we deduce the following conclusions. (I) (2) (3) (4) (5)

Since ¢(O, 0) = I, f(x, y) is the density of a two-dimensional vector (X, Y). ¢(t,O) = 4>(0, t) = exp( - !t 2 ) => X '" N(O, 1). Y '" N(O, 1). 4>(t, t) = exp( -t 2 ), that is X + Y ,..- N(O, 2). X - Y is also normally distributed. X and Y are uncorrelated.

However, the random vector (X, Y) as defined by the density f(x, y) is not bivariate normal despite the fact that properties (2)-(5) are satisfied.

10.6.

If the distribution of (X, , ... , X n) is normal, then any linear combination and any subset of X I , ... , X n is normally distributed, but

there is a converse statement which is not true This example can be considered as a natural continuation of Examples 10.2 and 10.5. Let us introduce the function

95

RANDOM VARIABLES AND BASIC CHARACTERISTICS

(I)

le(XI,''''Xn) = (27r)-0/2

g

<po (x,)

[I H(xl- xll g

X.I(-I,I)(x.)expG

~x2) 1

where (XI, ... ,Xn) E m. n, IPO(X) = exp(-!x2), 1(_1,1) is the indicator function of the interval ( -1, 1) and the constant c is chosen such that

Under this condition we can check that I~ is a density of some n-dimensional random vector, say (X I, ... ,Xn). Evidently the density Ie defined by (1) is not nonnal. Now let us derive some statements for the distributions of the components of (Xl, ... , Xn). For this purpose we find the ch.f. ¢ of Ie explicitly, namely

where

'IjJ(t)

= { (2ilt2)(sin t -

t cos t),

0,

(iJ(t)

= { (2i It) + (6jt 2)'IjJ( t), 0,

if t ::J: 0 if t = 0, if t ::J: 0 if t = O.

From (2) one can draw the following conclusions. (a) Each of the components XI, ... ,Xn is distributed N(0,1). (b) For each k, k < n, the vector (Xii,"" XiI<) is nonnally distributed. (c) If U = XI ± X2 and V is any linear combination of the variables X 3 , • •• , X n , then U + V is nonnally distributed. (d) If al, ... ,an are real numbers such that ak ::J: 0 for k = 1, ... , nand lall ::J: la21, then 2:;=1 akXk is not nonnally distributed. (e) E[n~=l X k ] = O. For the particular case n = 2 (which can be compared with Example 10.5) we obtain that: (a) ::::} Xl and X2 have standard nonnal distribution; (c) ::::} Xl + X 2 and XI - X2 are nonnally distributed; (e) ::::} XI and X2 are uncorrelated. However (XI, X2) is not nonnal, which follows from (d). Return again to the general case. Let U = XI ± X 2 , U I be a linear combination of anykofthevariablesX3 ,X4 , •.. ,Xn,O::; k::; n-2,andYbealinearcombination of the remaining n - k - 2 of these variables. Then the r.v.S X = U + U I and Y are independent and nonnally distributed. Indeed, X and Y are uncorrelated and normal r. v.s and (c) implies that a countably infinite number of distinct linear combinations of them are distributed nonnally.

96

COUNTEREXAMPLES IN PROBABILITY

10.7.

The condition characterizing the normal distribution by normality of linear combinations cannot be weakened

Let us start with the formulation of the following result (Hamedani and Tata 1975). Suppose {( ak, bk ), k = 1,2, ... } is a countable 'distinct' sequence in ~? such that for each k, akX + bkY is a normal r.v. Then (X, Y) has a bivariate normal distribution. (Here 'distinct' means that the parametric equations t, = akt, t2 = bkt represent an infinite number of lines in ]R2.) We are now interested in whether the condition of this theorem can be weakened. More precisely, let X and Y be r.v.s satisfying the following condition:

(ak,bk),k = 1, ... ,N, N a fixed natural number, the linear combinations ak X + bkY, k = 1, ... , N are normally distributed.

(CN) for given N pairs

The question is of whether (CN) implies that (X, Y) has a bivariate normal distribution. To see this, consider the following function:

(8, t) = exp [-

[g

~(8' + t')1+ exp [-E - ~c(s' + t') 1 (b~8' - a~t')1

where s, t E ]R', c, c E ]R+. Firstly, we shall show that for a suitable choice of c and c, ¢(s, t) is the ch.f. of some two-dimensional distribution. Indeed, denoting by I(x, y) the inverse Fourier transform of ¢(s, t), we obtain:

I(x,y)

=

(27r)-2/L2 exp(-isx-ity)¢(s,t)dsdt

= (27r)-' exp [- ~(x2

he ,c(x, y) .- (27r)-2e-e X

fL2

+ y2)] + he,c(x, y),

exp( -isx - ity) exp [- ~C(s2

+ t 2 )]

n~=, (b~S2 - a~t2) ds dt.

Further, we need an estimate for the function he ,c which has just been introduced. It can be shown (Hamedani and Tata 1975) that for suitably chosen constants c and c (1)

Ihe,c(x,y)l::; (27r)-' exp [-i(x 2 +y2)]

for all (x,y) E

]R2.

Since ¢(s, t) is a continuous function, ¢(O, 0) = I and I(x, y) is real-valued, we conclude that I and ¢ is a pair of functions where I is a two-dimensional density and ¢ its corresponding ch.f. Denote by (~, 17) the random vector whose density is f. Further, the definition of ¢ immediately implies that

¢(akt,bkt) = exp [-~(a~ +b~)t2],

k = I, ... ,N.

Obviously, this means that the r.v. ak~ +b k17 is normally distributed as N(O, a~ +b~), k = 1, ... N. However, ¢ itself is not the ch.f. of a bivariate normal distribution.

RANDOM VARIABLES AND BASIC CHARACTERISTICS

97

Therefore we have constructed a pair of r.v.s" and 11, for which condition (CN) is satisfied. but (',11) is not normaL Thus (CN) is not enough for normality of (',11). It should be noted that condition (1) holds only if N is finite.

10.8.

Non-normal distributions such that all or some of the conditional distributions are normal

(i) Let lex, y), (x, y) E JR2 be a bivariate normal density. Then it is easy to check that each of the conditional densities!1 (xjy) and h(yjx) is normaL This observation leads naturally to the question of whether the converse statement is true. We shall show that the answer is negative. Consider the following function:

(1)

g(x, y) = C exp[ -( 1 + x 2)(1

(x, y) E JR 2.

+ y2)],

Here C > 0 is a norming constant such that ffJR 2 g(x, y) dx dy ]. A standard calculation shows that the conditional densities g[ (xly) and g2 (y Ix) of g(x, y) are expressed as follows: g[ (xly) = (27rap 1/2 exp( -x 2

g2(ylx)

/2ap,

= (27rax)-1/2exp{-y2/2ax)

wherea~ = 1/(2(1 +y2)),a; = 1/(2(1 +x 2)),x E JRl,y E JR l . Obviously gl (xly) and g2(ylx) are normal densities of N{O, a~) and N(O, a;) respectively. However, g( x, y) given by (I) is not a two-dimensional normal density. Therefore the normality of the conditional densities does not imply that the two-dimensional density is normal. Let us note that similar properties hold for any density (non-normal) of the type 2

!i(x,y)

= tex p [-

,2: bijXiyj] 1,J=0

(for details see Castillo and Galambos (1987. 1989». One particular case of 9 is the function 9 given by (I ). (U) Consider now another interesting situation. Let' be a r.v. distributed uniformly on the interval [0,1] and 111,112,113,114,115 be independentr.v.s each with distribution N(O, 1). Suppose additionally that' and 11k, k = 1, ... 5 are independent. Define the r.v.s

It is then not difficult to check that each of XI. X2. X3 has a standard normal distribution N{O, 1). Further, if ¢31 1,2( t) denotes the conditional ch.f. of X3 given Xl.

98

COUNTEREXAMPLES IN PROBABILITY

Xz, then we find

and hence X 3 , conditionally on XI and Xz, has a normal distribution N(O, I). So, given these properties we conjecture that the vector (XI, Xz, X3) has a trivariate normal distribution. Let us check whether or not this is correct. For this purpose we compute the joint ch.f. tj;(tr, tz) of XI and Xz as follows

+ itzXz)} = E{E[exp(itIXI + itzXz)l~]} = E{exp[-!ti~ - !t~~ - !(tl + tz)z(1 - ~)]} = exp[ - !(tl + tz)Z]E[exp(tl tz~)l = (tl tZ)-1 (e t1t2 - 1)e- !(tl+td.

tj;(tl, tz) = E{ exp(itlXI

This form of the ch.f. tj;(tl' tz) shows that the distribution of (Xl, Xz) is not bivariate normal. Therefore the vector (X I, X Z, XJ) cannot have a trivariate normal distribution despite the normality of each of the components Xl, Xz, XJ and the conditional normality of X3 given XI, Xz. Note that under some additional assumptions the conditional normality of the components of a random vector wiJI imply the normality of the vector itself (see Ahsanullah 1985; Ahsanullah and Sinha 1986; Bischoff and Fieger 1991; Hamedani 1992).

10.9.

Two random vectors with the same normal distribution can be obtained in different ways from independent standard normal random variables

Recall that there are a few equ ivalent definitions of a multi-variate normal distribution. According to one of them a set of r. v.s X I, ... , X n with zero means is said to have a multi-variate normal distribution if these variables are linear combinations of independent r.v.s, say 6, ... ,~M, where ~j '" N(O, 1), j = 1, ... , M. That is, we have M

(1)

Xi

=

L Cij~j, i = 1, ... , N. j=1

Note that there is no restriction on M. It may be possible that M

< N, M = N

or

M>N. Suppose we are given the r.v.s ~I, ... '~M which are independent and distributed N(O, 1). Any fixed matrix (Cij) generates by (1) a random vector with a multi-variate normal distribution. Then the natural question which arises is whether different matrices generate random vectors with different (multi-variate normal) distributions. To find the answer we need the following result (see Breiman 1969): if both random vectors (Xl, ... , X N) and (Y1 , ••• , YN ) have a multi-variate normal distribution with the same mean and the same covariance matrix, then they have the same distribution.

RANDOM VARIABLES AND BASIC CHARACTERISTICS

99

Now we shall use this result to answer the above question. According to (1) each of the vectors (Xl, ... ,XN ) and (Yi, ... , YN ) is a transformation of independent N(O, 1) r. v.s obtained by using a matrix. Thus the question to be considered is whether this matrix is unique for both vectors. By a simple example we can show that the answer is negative. Take 6 and 6 to be independent N(O, 1) r.v.s and let

Xl = ~I

+ 6,

X2 =

26 + 6·

Define also

Yi =

V26,

Y2 =

(3/V2) 6 + (l/h) 6·

Thus we obtain two random vectors, (Xl, X2) and (Yi, 1'2). It is easy to see that

(Xl, X2) has zero mean and covariance matrix

(~ ; ) . Further, (Yi, Y2) has zero

mean and the same covariance matrix. Moreover, both vectors, (Xl, X 2 ) and (Yi, Y2 ) are multi-variate normal. By the above result, (Xl, X 2 ) and (Yi, Y2 ) have the same distribution. However, as we have seen, these identically distributed vectors are obtained in quite different ways from independent N(O, 1) r. v.s.

10.10.

A property ofa Gaussian system may hold even for discrete random variables

A set of r.v.s {~I' ... ,~n} is said to be a Gaussian system if any of its subsets has a Gaussian (normal) distribution. Suppose for convenience that each ~j has zero mean and denote Cij = COV(~i' ~j) = E[~i~jl. Then for an arbitrary choice of four indices i, j, k, l (including any possible number of coincidences) the following relation holds:

( 1) Note that a similar property is satisfied also for a larger even number of variables chosen from the given Gaussian system (including coincidences of indices). To prove such a property it is enough (0 use the ch.f. of the random vector whose components are involved in the product. The above property has some useful applications but it is also of independent interest. It is natural to ask if this property holds for Gaussian systems only. If the answer were positive, then (1) would be a property characterizing the given system of r.v.s as Gaussian. It turns out, however, this is not the case. Here is a simple illustration. Consider the sequence 1]1,1]2, ... of i.i.d. r. v.s such that P[1]1 = -V3] = P[1]1 = V3] = 1/6,

P[1][ = 0] = 2/3.

It is easy to see that for all choices of indices (including possible coincidences)

E1]i = 0,

E[1]i1]i] =

Cij

and

E[1]iT/i1]k] = 0,

100

COUNTEREXAMPLES IN PROBABILITY

where Cij

:::

1, if j ::: i and

E[17i77i77i77i]

3,

Cij :::

f:.

0, if j

i. Direct calculations show that for

E[77i1]i77j77j]::: 1

j

f:. i

and

E[1]i1]j1]k1]d::: 0

for all other choices of indices. All these facts taken together justify the following relation (compare with (1»:

E[1]i77j77k1]d :::

CijCkl

+ CikCjl + CUCjk

which is valid for arbitrary indices i, j, k, l. Hence (1) is satisfied for a collection of r. v.s whose distribution is very far from Gaussian.

SECTION 11. THE MOMENT PROBLEM Let {ao ::: 1, ai, a2, ... } be a sequence of real numbers and I be a fixed interval, I C }RI. Suppose there is at least one d.t. F(x), x E I such that

an

=

1

xndF(x),

n::: 0, 1,2 ....

If F is uniquely specified by {an} we say that the moment problem is determinate (the distribution is uniquely determined by its moments). Otherwise the moment problem is indeterminate. Note that the moment problem in the case I = [0,(0) is called the Stieltjes moment problem, while in the case I ::: (- 00, (0) we speak of the Hamburger moment problem. There are sufficient conditions for the moment problem to be determinate or indeterminate. Let us formulate the following three criteria. Criterion (Cd. Let F(x), x E }RI be a d.f. whose ch.f. ¢(t), t E IRI is r-analytic for some r > 0. Then F is uniquely determined by its moment sequence {an} where an = J~oo x tl dF{x). Further, the ch.f. ¢ is r-analytic for some r > iff

°

_._ (a2n)1/(2n) hm

n-+oo

2n

< 00.

(Equivalently, the m.gJ. M(t) ::: E[e tX ] exists for It I < to, to > 0.) Criterion (C2)' Let {ao::: l,al,a2, ... } be the momentsofad.f. F(x), x E IRI and let 00

(1)

I)a2n)-I/(2n) ::: n=l

00

(Carleman condition).

Then F is uniquely determined by {an}. If the d.f. F has as support the interval [0, (0) (instead of (- 00, (0» then a sufficient condition for uniqueness is ,,",<X> la n 1-1/(2n)::: 00 . ~n=1

tot

RANDOM VARIABLES AND BASIC CHARACTERISTICS

Criterion (C3). (a) Suppose the dJ. F(x). x E ]Rl is absolutely continuous with density f(x) > 0, x E ]R I and let F have moments of all orders. If

j <XJ - log+ f(x) x

(2a)

~<XJ

----::=-2-

1

d X

< 00 (Krein condition)

then the distribution F is indeterminate. (b) Let the d.f. F(x), x E ]R+ (F(O) = 0) be absolutely continuous with density f(x) > 0, x > 0 (f(x) - 0, x :s 0) and let F have moments of all orders. If <XJ - log f(x 2 )

1

(2b)

o

--=-~2~

1+ x

dx

< 00 (Krein condition)

then the distribution F is indeterminate. The proof of criteria (Cd and (C2 ) can be found in Shohat and Tamarkin (1943). Criterion (C) was suggested by Krein (1944) and discussed intensively by Akhiezer (1965) and Berg (1995). In these sources, as well as in Kendall and Stuart (1958), Feller (1971), Chow and Teicher (1978) and Shiryaev (1995), the reader will find discussions of these and other related topics. The ex.amples in this section clarify the role of the conditions which guarantee the uniqueness of the moment problem and reveal the relationships between different sufficient conditions.

11.1.

The moment problem for powers of the normal distribution

If ~ is a r.v., ~ N( a, (12), then the distribution of ~ (the normal distribution) as well as that of (X2-distribution) are uniquely determined by the corresponding moment sequences. These facts are well known but also they can be easily checked by, e.g. the Carleman criterion. Thus a reasonable question is: what can we say about higher powers of ~? It turns out 'the picture' changes even for e. The first observation is that all moments E[(e)k], k 1,2, ... , exist, however E[exp(te)] exists only if t = O. Hence the m.g.f. does not exist and we can conclude (see Criterion (CI» that the distribution of e is indeterminate. The case e allows us to make a more detailed analysis. For let us take a r.v. TJ N(O, whose density is 11"-1/2 exp( -x 2), x E ]RI. Then the new r.'1. X TJ3 has the following density:

e

/"oJ

=

/"oJ

!)

By using some standard integrals (fo<XJ(1 + x 2)-1 dx = n/2, fo<XJ[(Iogx)/(I x 2 )]dx = 0, fo<XJ[xO /(1 + x 2 )]dx = n/2cos(6n/2), -1 < t5 < I) we can easily conclude that

i:

[-logf(x)/(l

+ x 2 )] dx <

00.

102

COUNTEREXAMPLES IN PROBABILITY

Hence, according to the Krein criterion (C3), the distribution of the r.v. X = TJ3 is not determined uniquely by the moment sequence {Ok = E[Xk], k = 1,2, ... }. In a similar way we can show that the moment problem is indeterminate for the · dlstn'b' utlOn 0 f any LV. TJ 2n+1 , n = 1, 2, .... Let us return to the r. v. X = TJ3. Knowing that the distribution of TJ3 is indeterminate, we should like to describe explicitly another LV. with the same moments as those of TJ3. One possible way to do this is to consider the following function:

It can be shown that for E E [- ~, ~], Ie (x), x E IR I, is a probability density function. Denote by Xe a r.v. having Ie as its density. Obviously Ie i- I, except for the trivial case E = O. Our further reasoning is based on the equality

This immediately implies that

despite the fact that Xe and X are r.v.s with different distributions since for their densities one holds: Ie i- I, E E [-~,~] (exceptE = 0). It is interesting (and even curious) to note that the distribution of the absolute value IXI is determinate! Indeed, the LV. IXI = ITJI3, where TJ '" N(O, ~), has a density (2/3y'1r)x- 2/ 3 exp( _x 2/ 3 ) for x > 0 (and 0 for x :S 0). Then for the moment ak = E[lXlk] of order k, k = 1,2, ... , we have ak = (1/y0r)r((3k + 1)/2). For large k, ak "-' ck 3/ 2, c = constant, implying that 2:~1(ak)-I/(2k) = 00, i.e. the Carle man condition is satisfied. Therefore in this case the moment problem is determinate.

11.2.

The lognormal distribution and the moment problem

Let X be a LV. such that log X '" N(O, 1). In this case we say that X has a (standard) lognormal distribution. The density I of X is given by

( 1) n2

The moments an = E[xn], n ~ 1, can be calculated explicitly, namely an = e / 2 • It is easy to check that the moments {an} do not satisfy the Carleman condition 2:~1Ianl-I/(2n) = 00. Since this condition is only sufficient, we cannot say whether the sequence {an} determines uniquely the d.f. F of X. Further, we have

RANDOM VARIABLES AND BASIC CHARACTERISTICS

the following relations:

1=

(I +

2 X )

II log xl k dx =

~

i:

103

2y

(1 + e )-llylk e ll dy

fO Iylk e y dy + (Xl Iylkey dy < 00,

1-00

10

k

= 0, 1,2, ....

From this we conclude that the density (1) does satisfy the Krein condition (2b). According to Criterion (C3 ) the lognormal distribution is not determined uniquely by its moments. Alternatively, the same conclusion is derived by referring to Criterion (C,) after showing that E[e tX ] = 00 for each t > 0 meaning that the m.gJ. of X does not exist. Thus we come to the following interesting question: is it possible to find explicitly other distributions with the same moments as those of the lognormal distribution? Two possible answers are given below. (i) Let {fe (x), x E R I, t E [- I, I]} be a family of functions defined as:

fe(x) _ {f(x)[1 0,

(2)

+e:

if x if x

sin(27rlogx)],

>0 ~ 0

where f is given by (1). Obviously, fe(x) ~ 0 for an x E JRI and any E" E [-1,1]. In order to establish other properties of fe. we have to prove that

Indeed, by the substitution log x = (27r)-1/2

i:

exp (

U

= Y + k we reduce

_~u2 + kU) sin(21fu) du = (27r) 1/2exp

(~k2)

i:

Jk

exp

to the integrals

(_~y2) sin(27ry)dy.

The last integral is zero since the integrand is an odd function and the interval (-00,00) is symmetric with respect to O. So, based on (3) we draw the following conclusions concerning the family (2). If k = 0 then for any E E [-1, 1], fe (x), x E JR 1 is a probability density function of some r.v., say Xc' Obviously, if e: = 0, fe and Xc are respectively f and X defined at the beginning of this section. Moreover, we have

E[X:J = E[Xk] for any k, k

1,2, ...

despite the fact that fe f:. f for E f:. O. Therefore we have described explicitly the family {Xc} containing infinitely many absolutely continuous r. v.s having the same moments as those of the r. v. X with

104

COUNTEREXAMPLES IN PROBABILITY

lognonnal distribution. This example, after the paper by Heyde (1963a) appeared, became one of the most popular examples illustrating the classical moment problem. (ii) Now we shall exhibit another family of d.f.s {Ha, a > o} having the same moments as the lognonnal distribution (1). Let us announce a priori that H a , for each a > 0, will correspond to some discrete r.v. Ya . For a > consider the function

°

00

(4)

ha(t)

=

L

n2 a- ne- / 2 exp(iaent),

t E RI.

n=-oo

It is easy to see that the series in (4) is convergent for all t E R I and all a > 0. Moreover, the functions ha(t), i E RI are continuous and positive definite in the standard sense: L:j,k zjzkha(tj - ik) 2: 0, ij, ik E RI, Zj, Zk are complex numbers. By the Bochner theorem (see Feller 1971) the function

is a ch.f. Denote respectively by H a and Ya the d.f. and the r. v. whose ch.f. is 'l/Ja. The explicit fonn (4) of the function ha allows us to describe explicitly the r.v. Ya. We have

The next step is to find the moments an = E[Y:J = L~_oo(aek)npa(k), n = 1,2, .... Since

we can easily obtain

= i- n

L a- k (iae k te-(n +k2 2

)/2 /ha(O)

k

= L a-(k-n)e-(k-n)2 /2 /ha(O) = 1. k

It follows from this that

E[Y:J

= an = en2 / 2 = E[xn],

n

= 1,2, ....

Therefore there is an infinite family {Ya } of discrete r. v.s such that Ya has the same moments as the r.v. X with lognonnal distribution. We refer the reader to papers by Leipnik (1981) and Pakes and Khattree (1992) for more comments about this example and other related topics.

RANDOM VARIABLES AND BASIC CHARACTERISTICS

11.3.

105

The moment problem for powers of an exponential distribution

Let € be a r.v., € '" Exp( 1). The density of € is e- x for x > 0 and 0 for x S; 0, so the moment of order k is Ok = E[e] = k!, k = 1,2, .... The distribution of €, i.e. the exponential distribution is uniquely determined by the moment sequence {Ok}. This follows from the Carleman criterion as well as from the existence of the m.gJ. of € and referring to Criterion eC I ). Now we want to clarify whether powers of € have determinate distributions. For let 6 > 0 and let X = If J is the density of X, we easily find that

e.

I(x)

= (I/6)x 1/ O- 1 exp(-x IN ),

x >0

(of course, I(x) :--: 0 for x S; 0). The moments of X exist, E[Xk] = r(6k + 1), k = 1,2, .... The explicit form of the density I al10ws us to find that oo 2 2 [- (log I (x )) / (1 + x )] dx < 00 iff 6 > 2. Hence for 6 > 2 the distribution of the r.v. X = is indeterminate. Let us show that in this case, with 6 > 2, there is a family of r. v.s all having the Indeed, consider the function: same moments as those of

fo

e

e.

where Co = tan(7r/6) and

1

lEI

e

S; sin(7r/6). The equality

00

xk I(x )[cos(coxIN) - (1/ co) sin(coxIN)] dx = 0,

shows that

Ie is a probability density function of a r. v. Xc

k = 0, 1,2, ...

and

even though Ie f:. I (except the trivial case E = 0). For completeness we have to consider the case 6 E (0,2]. Since Ok = E[(e)k] = r(6k + 1), we can use the properties of the gamma function ro and show that the Carleman condition is satisfied. Thus we conclude that if € '" exp( 1) and 0 < 6 S; 2, then the distribution of is uniquely determined by its moment sequence (also see Example 11.4).

e

11.4.

A class of hyper-exponential distributions with an indeterminate moment problem

Recall first that the one-sided hyper-exponentiaL distribution J(+ (a, b, c), where a, b, c are positive numbers, is given by the density function

(I)

I(x)

= {cob,-a/c(r(a/c))-Ixa-I exp(-xc/b),

if x if x

>0 < o.

106

COUNTEREXAMPLES IN PROBABILITY

(Notice that the gamma distribution ",(a, b), and the exponential distribution £xp(>..) are special cases of the hyper-exponential distributions.) It can be shown (for details see Hoffmann-Jorgensen 1994) that if X is a r.v., X 9{+ (a, b, c), then the quantity E[Xk] does exist for any k 2: 0 (not just for integer k) and, moreover, I"V

Hence the r.v. X has finite moments Ok = E[Xk], k = 1,2, ... , and the question is whether or not the moment sequence {Ok} determines uniquely the distribution of X. Let us take some a > 0, b > 0 and 0 < c < Then we can choose p > 0 such that m := a + p is an integer number, set r = plc,"\ = l' + 11(3, s = tan(crr) and introduce the function

t.

Since e- X ::; rrx-r for all x > 0, we easily see that 11jJ(u)1 ::; 1, U > O. Let k be and a fixed non-negative integer. Then n = k + m is an integer, v = crr E (0, substituting x = U C yields

I)

This implies that for any non-negative integer k and any real holds:

1

00

where

f

1

E

the following relation

00

uk fe(u) du

=

uk f(u) du

is the hyper-exponential density (1) and

fe(x)

= { f(x)[1 + E1jJ(X)], 0,

if x> 0 if x < O.

Since I1jJ( x) ::; 1, x > 0, it is easy to see that for any E E [- 1, 1], fe is a probability density function of a r. v. X e and

despite the fact that fe f:. f (except the trivial case E = 0). Therefore for a > 0, b > 0 and c E (0, the hyper-exponential distribution J(+ (a, b, c) is not determined uniquely by its moment sequence. It can be shown, however, that for a > 0, b > 0 and c 2: ~, the moment problem for J(+ (a, b, c) has a unique solution (Berg 1988).

1)

107

RANDOM VARIABLES AND BASIC CHARACTERISTICS

Since for the exponential distribution Exp(>'), the gamma distribution ,(a, b) and the hyper-exponential distribution :;..c+ (a, b, c) we have

Exp(>')

,(1,1/>'),

,(a, b) = Je+(a,b, 1)

it follows that if X '" exp(>.) or X '" ,(a, b), where>. > 0, a > 0 and b > 0 are arbitrary, then the distribution of X 6 for {} > 2 is not determined uniquely by the corresponding moment sequence. (Also see Example lI.3.) 11.5.

Different distributions with equal absolute values of the characteristic functions and the same moments of all orders

Let us start with the number sequence {ak, k == 1,2, ... } where ak > 0 and 00. We shall consider a special sequence of distributions and study the corresponding sequence of the ch.f.s. The uniform distribution on [-ak, ak] has ach.f. sin(akt)/(akt). Denote by!k (x), x E [- 2ak, 2ak] the convolution of this distribution with itself. Then the ch.f. ¢k of !k is ¢dt) = [sin(akt)/(akt)j2, k - 1,2, ... , t E )Rl. The product ¢j(t) converges, as m -t 00, to the function ¢(t) where

a := L:~I ak <

n;,

n 00

¢(t) :=

(1)

¢j{t),

t

E )RI.

Using the Taylor expansion of sin x for small x, we find that

¢(t) ~ exp (- iiit2 )

as t -t 0

where ii = L:;l aJ. Therefore according to Lukacs (1970, Th. 3.6.1), ¢(t), t E )RI is a ch.f. The d.f. corresponding to ¢ is absolutely continuous. Let its density be denoted by f. Clearly f is an infinite convolution: that is. f == fl * 12 * .. ', By the inversion formula (Feller 1971; Lukacs 1970; Shiryaev 1995) we find that f(O) = (27l')-1 f~oo ¢(t)dt, Since ¢(t) > 0, for all t E )RI, we can construct the density fa by setting fo(x) = (27l'f(0»-1¢(x), x E )RI. If ¢o denotes the ch.f. of fa then ¢OCt) = f~oo exp(itx)fo(x) dx

= (211" f(O»-1 f~oo exp( -itx)¢(x) dx =

f(t)/ f(O).

Note especiaJly that the support of ¢o is contained in the interval [-2a, 2a]. Using the function ¢o and the function ¢ from (1) we define the following four functions:

'1/JJ(t)

= ¢o(t) + H¢o(t + 4a) + ¢o(t -

4a)],

4n(t) = ¢OCt) - H¢o(t + 4a) + ¢o(t - 4a)], 9t(X) = (211"j(0»-I¢(x)(1 +cos4ax), 92(X) = (27l'f(0» l¢(x)(l - cos4ax).

108

COUNTEREXAMPLES IN PROBABILITY

The above reasoning shows that 9 I(x), x E lR I and 92 (x), x E lR I are probability density functions. Moreover, 'l/JI and 'l/J2 given by (2) and (3) are just their ch.f.s. Also I'l/JI (t)1 = 1'l/J2(t) I for each t E lRl. Denote by XI and X 2 r.v.s with densities 91 and 92 respectively. If c;; I := Tff(O) n;=1 a~ then it is easy to derive from (4) that Igl(x)1 S c n lxl- 2n for each n E N. The same estimation holds for 92. Hence both variables XI and X2 possess moments of all orders. As a consequence we obtain (see Feller 1971) that the ch.f.s 'l/JI and 'l/J2 have derivatives of all orders, and, since I'l/JI (t)1 = 1'l/J2(t)l, 'l/JI (t) = 'l/J2(t) for t in a small neighbourhood of 0. This implies that the moments of XI and X 2 of all orders coincide. Looking again at the pairs (91,92), ('l/JI ,'l/J2) and (X I, X 2) we conclude that I'l/JI (t) I = I'l/J2 (t) I for all t E lRl, E[Xn = E[Xf] for each kEN but nevertheless 91 i- 92· 11.6.

Another class or absolutely continuous distributions which are not determined uniquely by their moments

Consider the r.v. X with the following density: f(x)

°

= { c exp( -ax A),

if x > if x SO.

0,

°

Here 0' > 0, < A < ~ and c is a nonning constant. For c E (-1, I) and ,8 = 0' tan ATf define the function fe(x)

fe

= {c exp(-ax A)(1 + csin(J1x A)), 0,

by

°

~f x > If x SO.

Obviously fe(x) ~ 0, x E lRl. Next we shall use the relation

1

00

(I)

xn exp( -ax A) sin(J1x A) dx

Let us establish the validity of (1). If p > ~q > 0, then we use the well known identity

1

= 0.

°

and q is a complex number with

00

Denoting p

1

tp-1e- qt dt = r(p)jqP.

= (n + 1) j A, q = a + ib, t =

x A , we find

1

00

00

xA[(n+I)/A-I] exp[-(a + ib)XA]AXA-I dx = A

1

00

= A

xn exp( -ax n ) COS(bXA) dx - iA

= r((n -+-

1

00

xn exp[-(a + ib)x A]dx

xn exp( -ax A) sin(bx A) dx

l)j A) [a(n+l)/A( 1 -+- i tan ATf)(n+I)/A] .

RANDOM VARIABLES AND BASIC CHARACTERISTICS

The last ratio is real-valued because sin[rr(n

(I

+i

109

+ I)] == 0 and

tanArr)(n+I)/A

= (cos Arr) -(n+I)/ Aei1f (n+l)

= (cos Arr)-(n+I)/A cos1r(n + 1). Thus (1) is proved. Taking n = 0 we see that ff: is a probability density function. Denote by Xf: a r.v. whose density is fe;. The relationship between fe. and f, together with (I), imply that E[X:] = E[xn]

for each n == 1,2 ....

Therefore we have constructed infinitely many r.v.s Xe: with the same moments as those of X though their densities Ie: and f are different (fe: = I only if c = 0). So in this case the moment problem is indeterminate. However, this fact is not surprising because the density f does satisfy criterion (C3).

11.7.

Two different discrete distributions on a subset of natural numbers both having the same moments of all orders

Let q > 2 be a fixed natural number and Mq == {qm : m == 0, 1,2, ... }. Clearly Mq c N for q = 2,3, .... If n E Mq then n has the form qm and we can define Pn by Pn = e-qqm 1m!. It is easy to see that {Pn} is a discrete probability distribution over the set }vfq • Denote by X a r.v. with values in Mq and a distribution {Pn}. In this case we say that X has a log-Poisson distribution. Then the kth-order moment Ok of X is 00

Ok == E[xk]

L e-qqkmqm 1m! = exp[q(l -

I)]

< 00.

m=O

Our purpose now is to construct many other r. v.s with the same moments. Consider the function h(z) := n~, (J - zq-k). Since 2:~1 q-k < 00 for q > I then h(z) is an analytic function in the whole complex plane. Let h(z) = 2::=0 cmz m be its Taylor expansion around O. Taking into account the equality h(qz) = (I - z)h(z) we have the relation Cm I Cm-I = - (qm - I) -I where C{) = I and for m 2: 1 we find

Setting am

= m !cm we see that lam I ::;:

I for all m. This implies that

00

(1)

e- q

L m=1

lmamqmlm!

= e-qh(l+l) == 0

foraU

k

= 0,1,2, ....

110

COUNTEREXAMPLES IN PROBABILITY

N ow introduce the n umber set {p~) , n E M q} where

p~) := Pn(1 + cam) = e-qqm[(m!)-I

+ c( -

I)m((q - l)(q2 - I) ... (qm - 1))-1],

Here c is any number in the interval [-I, I]. Obviously p~) ~ LnEMq

p~) = I for any c

E

n = qm.

°

and (1) implies that

[-I, I]. Therefore {p~)} defines a discrete probability

distribution over the set M q. Let Xe be a r. v. with values in Mq and a distribution {p~:)}. Using (1) again we conclude that E[X:] = E[Xk] for each k = 1,2, ... ,

cE[-l,I]. So, excluding the trivial case c = 0, we have constructed discrete r. v.s X and Xe whose distributions are different but whose moments of all orders coincide.

11.S.

Another family of discrete distributions with the same moments of all orders

Let N = {O, ± 1, ±2, ... } and X be a r.v. with the following distribution:

= e 8m ] =

P[X

ce-

m2

,

c- I =

m E N,

L

"'e-

m2

.

m2 is the sum over all m E N. For any positive integer k we Here and below 2: "'ecan calculate explicitly the moment Q:k = E[Xk] of order k, namely Ok

=

L

"'e8kmce-m2

= e 16k2 L

"'cexp[-(m

+ 4k)2]

= e 16k2 .

Now we shall construct a family consisting of infinitely many r.v.s with the same moments. For any c E (0, 1) define the function if if if if

= ° = m =3

m

1 2

m m

(mod (mod (mod (mod

4) 4) 4) 4).

In the sequel we use the evident properties: for any fixed c, he(m) is an odd function of m, that is he ( -m) = - he (m); he (m) is a periodic function of period 4; he(m + 4k) is an odd function in m for each integer k. The next crucial step is to evaluate the sum Sk where

Sk We have

= L "'e8kmhe(m)ce-m, 2

k = 0,1,2, ....

Sk = cL"'h e(m)exp(8km-m 2 ) = cexp(16k 2) L "'he(m) exp[-(m - 4k)2] = cexp( 16k 2) L: '" he(u

+ 4k) exp( -u 2).

111

RANDOM VARIABLES AND BASIC CHARACfERISTICS

The last sum is zero because he (u + 4k) is an odd function of u for all k = 0, 1,2, .. " Thus we have estabHshed that

(I)

Sk

=

°

for all k

= 0, 1,2, ...

anda11 c E (0, I).

As a consequence of (1) we derive that

for each m E N and any c E (0, 1) and moreover L *qe (m) = 1. This means thatthe set {qe (m), m E N} can be regarded as a discrete probability distribution of some r.v. which will be denoted by Xe. Thus we have constructed a r.v. Xe whose values are the same as those of X but whose distribution is

Since (1) is satisfied for any k = 1, 2, ... we find that

d

Therefore for any c E (0,1) we have Xe same moments of all orders.

11.9.

:I X

but nevertheless Xe and X have the

On the relationship between two sufficient conditions for the determination of the moment problem

(i) Let Z = X log( 1 + Y) where X and Y are independent r. v.s each distributed exponentially with parameter I. Obviously Z is absolutely continuous and takes non-negative values. The nth moment On of Z is

It can be shown (the details are left to the reader) that

(1)

e- 1 log(1

+ n) < v:/ n < clog(1 + n),

c = constant

Thus (Onln!)I/n = v~n

> e-1log(1 + n)

--t 00

as n

--t 00

which implies that the series L~=l ontn In! does not converge for any t :I 0. Therefore we cannot apply Criterion (Cd (see the introductory notes to this section) to decide whether the d.f. of Z is determined by its moments.

112

COUNTEREXAMPLES IN PROBABILITY

From (1) we obtain that for large n,

a!!n ~ e-Inv~jn

< ce-1n 10g(1 + n)

and hence En an Ij(2n) = 00 because the series E:~l I/(mlog(J + m)) is divergent. Therefore, according to Criterion (C2), the dJ. F of the r. v. Z is detennined uniquely by its moments. Note, however, that we used the Carleman criterion (C2) whereas Criterion (C I). based on the existence of the m.g.f. does not help in this case. (ii) In case (i) we considered an absolutely continuous r. v. Here we take a discrete r. v. and tackle the same questions as raised in case (i). So, let Z be a r.v. whose set of values is {3, 4,5, ... } and whose distribution is given by P[Z = m] = cexp( -ml logm), m = 3,4, ...

where c is a norming constant obtained from the condition P[Z ;:: 3] = I. Our purpose is to verify whether Criteria (C I ) and (C2) apply. Firstly we derive suitable upper and lower bounds for the moment a2n of order 2n. Introducing the function

h(x) = xn exp( -xl log x) where x ;:: 3, n ;:: 4, we can easily show that

h' (x) nIl --::::----+--=0 h(x} x logx log2x iff x = Xn where

n=

n logx n

I) <

( 1--logx n

-

Xn .

logx n

Since nand Xn tend to infinity simultaneously, we have (n log xn)lxn --t I whence ~n log n :s: Xn :s; 2n log n for n ~ no. If we define Mn by

Mn = max[x n exp( -xl log x)] = x~ exp( -xnl logx n ) x;::::3

then for n

~

no we obtain the foHowing estimate: 00

a2n = c

L

m 2n e- mj log m S; CM2n+2

m=3

00

L (11m2) m=3

S; CM2n+2 S; c[( 4n

+ 4) log(2n + 2)]2n+2e- 2n - 2.

Therefore for all sufficiently large n

(a2n)lj(2n) S; 4c lj (2n) [(2n (2) S;

+ 2) log(2n + 2)p+(l j n) 8(2n + 2) log(2n + 2).

113

RANDOM VARIABLES AND BASIC CHARACTERISTICS

On the other hand, (a2n)I/(2n) > (cM2n )I/(2n) >e-I(~nlogn)

(3)

for all large n.

Now using (2) and (3) we easily find that 00

lim [(a2n)I/(2n) /(2n)) = 00

n-+oo

and

"'(a2n)-I/(2n) = 00.

L....,

n=1

Therefore the ch.f. of the r.v. Z is not analytic and we cannot apply Criterion (Cd to say whether the moment sequence {ak} determines uniquely the distribution of Z. However, the Carleman criterion (C2) guarantees that the distribution of Z is uniquely determined by its moments.

11.10.

The Carleman condition is sufficient but not necessary for the determination of the moment problem

In two different ways we now illustrate that the Carleman condition is not necessary for the moment problem to be determined. (i) Let FH be a symmetric distribution on ( -00,00) and Fs a distribution on [0,00). (The subscripts Hand S correspond to the Hamburger and the Stieitjes cases.) By the relations 2 F1 ( ) _ { ~[1 + Fs(x )], if x > H x ~[1 - Fs(x 2 )], if x <

°°

we can define a one-one correspondence between the set of symmetric distributions on (-00,00) and the set of distributions on [0,00). It is clear that FH possesses moments {an} of all orders iff Fs possesses moments {an} of all orders. In this case a2n

= an,

a2n+ 1 = 0,

n = 0, I , 2, ....

Thus we conclude that the Hamburger problem for {an} is determinate iff the corresponding Stieltjes problem is determinate. Moreover the Carleman condition 2::(a2n)-I/(2n) = 00 for the determination of the Hamburger case becomes 2::(a n )-I/(2n) = 00 in the Stieltjes case. We shall use this result later but let us now formulate the following result (see Heyde 1963b): if a set {an} of moments corresponds to a determinate Stieltjes problem, the solution of which has no point of discontinuity at the origin, then the set {an} also corresponds to a determinate Hamburger problem. Consider the r. v. X with density f given by

f(x)

= {[I/r(I/,B)]ex p (-x l1 ), 0,

°

if x> if x <

°°

where < ,B < 1. One can show that an = E[xn] = r((n + I)/,B)/r(l/,B), n = 0,1,2, ... , so that a~n Kn l/ 11 for some constant K. Then 2::(a n )-I/(2n) = 00 I"'<..J

COUNTEREXAMPLES IN PROBABILITY

114

!

for ~ /3 < 1, and by the Carleman criterion (for the Stieltjes case) the Stieltjes problem for these moments is determinate for ~ ~ /3 < 1. Since the distribution with density f has no discontinuity at the origin, from the above result we conclude that the Hamburger problem corresponding to the moments an == r( (n + I) 1(3) Ir( 11(3), n = 0, 1,2, ... , 4 /3 < 1 is also determinate. However, it is easy to check that 2:)a2n) 1/(2n) < 00 for 0 < /3 < 1. Hence the Carleman condition is not necessary for the determination of the moment problem (on (-00,00». (ii) Now we shall use the following interesting and intuitively unexpected result

(see Heyde 1963b); let the moments {I, aI, a2, ... } correspond to a determinate Stieltjes problem. After a mass £, 0 < £ < I, has been added at the origin and the distribution has been renormalized. it is possibJe for the new set of moments { J , al ( 1+£) -1, a2 (1 +£) -I, ... } to correspond to an indeterminate Stieltjes problem. So, Jet {l, aI, a2,"'} and {I, al (I +£)-1, a2(I +£)-1, ... }. 0 < £ < l, be sets of moments corresponding respectively to a determinate and an indeterminate Stieltjes problem. Suppose the Carleman condition is necessary for the determination of the moment problem. Then we should have 2:(a2n)-I/(2n) = 00, which is impossible because 2:(a2n)-I/(2n)(1 + £)I/(2n) < 00.

11.11. The Krein condition is sufficient but not necessary for the moment problem to be indeterminate As shown in the introductory notes to this section, the Krein condition is sufficient for the moment problem to be indeterminate. Thus it is useful to consider examples showing that this condition is not necessary. (i) Let X be a r.v., X "" N(O, ~), and d

> O. Then the density of the r.v.IXI 6 is

(/6 (x) 0, if x ~ 0) and it is easy to see that all moments E[(IXI&)k), k == 1,2, ... , exist. Berg (1988) has shown that the distribution of IX 1° is determinate for d ::; 4 and indeterminate for d > 4. However for the density h we find that Jooo{Jogh(x)/(v'x(I + x»)}dx = -00, i.e. the Krein condition is not satisfied, and hence it is not necessary for the moment problem to be indeterminate. (ii) Take the function h(x) == exp( -x'), if x > 0 and h(x) = exp(x), if x ::; O. Here 1 E (0, ~) and let c be a constant such that g,(x) = c,h(x), x E IRI is a probability

=

density. If Y is a r. v. with density g" then all moments E[yk), k J, 2, ... , exist. 2 Moreover, J~oo {logg,(x)/(1 + x )} dx == -00. Hence the Krein condition is not satisfied but the distribution of Y is indeterminate as follows from Example 11.6.

RANDOM VARIABLES AND BASIC CHARACTERISTICS

11.12.

115

An indeterminate moment problem and non-symmetric distributions whose odd-order moments all vanish

In Example 6.5 we described a r. v. Y such that Ct2n+ 1 = E[y2n+ I] = 0 for all n = 0, 1,2, ... but the distribution of Y is non-symmetric. However, we did not discuss tpe reason for this fact. Let us note that the distribution of the r. v. Y in Example 6.5 is not detennined uniquely by its moments. Now we shall show that the vanishing of the odd-order moments of a non-symmetric distribution is closely related to indetenninate Stieltjes problems. From Example ] ] .7 we know that there are indetenninate Stieitjes problems. Let the dJ.s FI and F2 be two distinct solutions of such a problem for a given set of moments {I, Ct I, Ct2, ... }. Then

is a d.f. which evidently is non-symmetric. Moreover, F has the following moments: 1, 0, Ct2, 0, Ct4, 0, .... Therefore any odd-order moment of F is zero despite its non-symmetry. FinaI1y, let US present one additional example based on the lognonnal distribution considered in Example 11.2. Once again, let

f(x) = (27r)-1/2x -1 exp[-!(Iogx)2], fl(x) = f(x)[l - sin(27rlogx)],

x> O.

Denote by Z a r. v. whose density 9 is defined as follows:

g(x) = { tf(x), 2 fl (-x),

if x> 0 if x ::; O.

Then one can check that all the moments of Z are finite, all odd-order moments E[Z2n+l] are zero but Z is non-symmetric.

11.13.

A non-symmetric distribution with vanishing odd-order moments can coincide with the normal distribution only partially

Let us recall that in general no probability distribution is detennined by a finite number of moments. The previous examples show that the distribution cannot be determined uniquely even if we know an (and hence an infinite number of) its moments. However, if we specify the class of distributions, then a member of this class could be detennined by a finite number of moments. For example, a member of the so-called class of Pearson distributions is specified by a knowledge of at most four moments (Feller 1971; Heyde 1975). Certainly we have to indicate the nonnal distribution N(a, (12) which is detennined uniquely by its first two moments only. Thus we come to the following question: does there exist a r. v. X such that for infinitely many k, but not for all k 2: 1, we have E[Xk] = E[ Zk] where Z '" N(a, (12) d

but nevertheless X

~

Z?

COUNTEREXAMPLES IN PROBABILITY

116

Let Z be a r. v. distributed N( 0, I). We shall construct a r. v. X such that

(1)

If (1) holds we can speak about a partial, but not full, coincidence of the distributions of X and Z. So, let YI be a r.v. with density

g(x) ::::::

- e signx sin(clxll/4)],

xE

}RI

> 0, e =F 0, lei <

I. Obviously g is non-symmetric. The moments E[Y/] can be calculated explicitly, namely

where c Qk ::::::

41gC4 exp( -clxP/4)[1

Q2k+l ::::::

0,

Q2k = ~c-gk(8k + 3)!,

k = 0, 1,2, ...

(see also Example 6.5). By choosing c = (11 !/6)1/8 we get Q2 = E[Y?] :::::: 1. Take now a r.v. Y2 which is independent of Yi and takes the values I and -1 with probability! each. For some constant (3, < (3 < 1 which wi11 be specified later, put

°

Clearly the distribution of X is non-normal and non-symmetric, E[X 2k+l]

°

= = E[Z2k+l]

f or each k

°

= )1'2, ... ,

Finally we find

It remains to choose (3 such that E[X4] :::::: 3 :::::: E[Z4]. Indeed, if the kurtosis coefficient of the r.v. Y I is r2 : : : E[yI4J - 3, then take (3 (,2 - y'2 ( 2) /(,2 - 2) Since c was already fixed (c = (11!/ 16) 1/8) then r2 and hence (3 have definite values. Thus we have constructed a r. v. X which coincides partially but not fully with the standard normal r. v. Z in the sense of (1).

SECTION 12.

CHARACTERIZATION PROPERTIES OF SOME PROBABILITY DISTRIBUTIONS

There are probability distributions which can be characterized uniquely by some properties. In such cases it is natural to use the term 'characterization properties'. Let us formulate two important results connected with the most popular distributions, the normal distribution and the Poisson distribution.

117

RANDOM VARIABLES AND BASIC CHARACTERISTICS

Cramer theorem. If the sum X I + X 2 of the r.v.s XI and X 2 is normally distributed and these variables are. independent, then each of Xl and X2 is normally distributed (Cramer 1936), Raikov theorem. If XI and X2 are non-negative integer-valued r.v.s such that XI + X2 has a Poisson distribution and Xl and X2 are independent. then each of XI and X 2 has a Poisson distribution (Raikov 1938). These important theorems, several useful coroUaries and other characterization theorems can be found in the books by Fisz (1963), Moran (1968), FeHer (1971), Kagan et at (1973), Chow and Teicher (1978) and Galambos and Kotz (1978). Let us note that some of the examples dealt with in Section 10 can be compared with the Cramer theorem. In particular this comparison shows that the assumption of the independence of XI and X2 is essential. We present below various examples of discrete and absolutely continuous distributions and clarify whether or not some properties are characterization properties.

12.1. A binomial sum of non-binomial random variables Let the r. v.s. X and Y be non-negative integer-valued and let their sum Z = X + Y have a binomial distribution with parameters (n,p), Z '" 'Bi(n,p). Then the probability generating function of Z is E[ sZ] = (ps + q) n. If additionally we suppose that X and Y are independent, then (ps + q)n = E(sxJE[sYJ. Since alJ factors of the polynomial (ps + q)n have the fonn (ps + q)k, k = 0,1" .. ) n, it foHows that each of the variables X and Y is also binomially distributed. This observation leads to the following question: does this conclusion hold without the hypothesis ofindependence between X and Y? Let us show that the answer is negative. Let ( be any non-negative integer-valued r. v. Suppose ( takes more than two different values. Define the r. v.s and 1] by

e

e=

((/2],

1] = [((

+ 1)/2J

where [xl denotes the 'integer part' of x. Obviously

pre

Moreover, knowing the distribution of ( we can easily compute = k] and P[1] = m] for all possible values of k, m. Since k,1] = m] 0 for those k. m satisfying the relation Ik - ml > I, we see that the r. v.S ( and 1] are not independent. Note that this property holds irrespective of the distribution of (. In particular, suppose ( is binomially distributed. Then neither nor 1] is binomial, but their sum + 1]. which is equal to (, has a binomial distribution. Recall that ( and 1] are dependent.

pre -

e

e

118

12.2.

COUNTEREXAMPLES IN PROBABILITY

A property of the geometric distribution which is not its characterization property

Recall that the r. v. X has a geometric distribution with parameter P, 0 < P < ), if P[X = n] = pqn, q = I - p, n = 0, I, .... Let Xl and X 2 be independent r.v.s each distributed as X. From the definition of a conditional probability we can easily derive that ( 1)

+ X 2 = n]

P[XI = klX l

1 = --1'

n+

k = 0, I, ... ,n.

We are interested now in whether (I) is a characterization property of the geometric distribution. More precisely: suppose Xl, X 2 are integer-valued independent r.v.s which satisfy relation (l), does it follow that Xl, X 2 are geometrically distributed? To find the answer let us consider the set 0 = {Wkn : k = 0, 1, ... , n, n = 1,2, ... } and let Pn, n = 0, I, ... be positive numbers with L~=o Pn = I. Define a probability P on 0 as follows:

P(Wkn)

= Pn/(n + 1).

This means that 0 = U~=o On where On = {Wkn, k = 0, 1, ... n}, P(On) = Pn and each of the outcomes Wkn has probability Pn/ (n + 1). Introduce two r. v.s, Yi and Y2 , such that YI (Wkn) = k, 1'2(Wkn) = n - k. Then for k = 0, 1, .. " n,

pry; = k Iy; 1

I

+

y;: = 2

1=

n

P[Yi = k, Yi + Y2 = n] pry) + 1'2 = n]

_ P(Wkn) _ Pn/(n + 1) _ P(On) Pn n

+I

Thus relation ( I) is true for the r. v.s Yi and 1'2. However, the distribution of Yi is 00

P[YI = k] = P[{Wkn: n = k,k

+ I, ... }] = LPn/(n + l} n=k

and since the Pn are arbitrary (with Ln Pn = 1), P[Yi = k], k = 0, 1, ... can be very different from the geometric distribution. Therefore (I) is not a characterization property of the geometric distribution. Note, however, thal if additionally we suppose that Xl and X 2 are independent and identically distributed, it can be proved that each of these variables has a geometric distribution.

12.3.

If the random variables X, Y and their sum X + Y each have a Poisson distribution, this does not imply that X and Yare independent

(i) Let X and Y be independent r.v.s each with a Poisson distribution. Then their

sum X

+Y

also has a Poisson distribution. We want to know whether the converse

119

RANDOM VARIABLES AND BASIC CHARACTERISTICS

of the above statement is true: if X and Y are integer-valued and each of X, Y and X + Y has a Poisson distrihution, are the variahles X and Y independent? It turns out that the answer to this question is negative. Take two r.v.s, and 1] each with a Poisson distribution of a given rate. Denote 0, I, ... } and {rj,j = 0, I, ... } where their individual distributions by {Qi,i Qi =p[e i] and rj P[1] = j]. Introduce the sets AI {(O, I), (1,2), (2, On and A2 = {(O, 2), (2, I), (I,O)}.Thejoint distribution of and 1], Pij := = i,1] = j] will be defined in the following way:

e

=

=

=

e

Qirj

(I)

Pij

={

+ e,

Qirj -

e,

Qirj,

=

pre

if (i,j) E AI if (i,j~ E A2 otherWise.

Here e is a real number such that lei < minij Qirj, (i,j) E AI U A2. It is easy to check that {Pij, i = 0, I, ... ,j = 0, 1, ... } is a two-dimensional discrete probability distribution. Moreover, using (1) we find that the sum + 1] has a Poisson distribution. By definition and 1] also have Poisson distributions. However, (1) implies that the r.v.s and 1] are not independent.

e

e

e

(ii) Here is another case slightly similar to case (i). For fixed>. > 0 let e be an arbitrary number in the interval (0, e- 2>">.4 /6). Define Pij, i = 0, I, ... , j = 0, 1, ... , as follows: PII P33

= e- 2>">.2 + e, PI3 = e- 2>">.4/6 - e, P31 = e- 2>">.4/6 - e, = e- 2>..>.6/36 + e, Pij = e- 2>">.i+j /i!j! for all other i and j.

Direct calculations lead to the following conclusions:

i, j = 0, I, ... } is a two-dimensional distribution of a random vector, say (X, Y); 2) X Po(>.) and Y Po(>.); 3) X + Y Po(2)'). I)

{Pij,

f"'V

f"'V

f"'V

However the two components X and Y are not independent.

12.4.

The Raikov theorem does not hold without the independence condition

Recall that the independence of the variables X and Y is one of the hypotheses in the Raikov theorem (see the introductory notes). We are now interested in what happens if we do not assume that X and Y are independent. Our reasoning is similar to that used in Example 12.1. Let ( be a r. v. with a Poisson distribution. Define the r. v.s and 1] by

e

e= [(/2],

1]

= [(( + 1)/2]

(here [x] denotes the integer part of x). It is easy to verify that each of an integer-valued r.v., neither nor 1] has a Poisson distribution, and independent, but the sum + 1] = ( has a Poisson distribution.

e

e

e

eand 1] is 1]

are not

COUNTEREXAMPLES IN PROBABILITY

120

Therefore, as expected, the independence condition in the Raikov theorem cannot be dropped.

12.5.

The Raikov theorem does not hold for a generalized Poisson distribution of order k, k ~ 2

We say that the integer-valued non-negative r.v. X has a generalized Poisson distribution of order k and parameter ..x, ..x > 0, if

(1) where the summation is taken over all non-negative integers m\, ... ,mk such that m 1 + 2m2 + ... + kmk = n. Obviously, if k = 1, then (1) defines the usual Poisson distribution. By using (1) we find explicitly the p.g.f. 9( s) = E[ sX] (see Philippou 1983):

(2) Suppose now that Y1, Y2 are independent r. v.s taking values in the set {O, 1,2 ... } and such that the sum Y1 + Y2 has a generalized Poisson distribution of order k. The question is: does it follow from this that each of the variables Yi, Y2 has a generalized Poisson distribution of order k? Note that in the particular case k = 1 the usual Poisson distribution is obtained and it follows from the Raikov theorem that the answer to this question is positive (see Example 12.4). We have to find an answer for k ~ 2. Consider two independent r. v.s, ZI and Zz where ZI has a generalized Poisson distribution of order (k - 1) and a parameter ..x, and Z2 has the following distribution:

P[Zz = m] = e->" ..x m / k /(m/k)!,

m

We shall use the explicit form of the p.g.f.s respectively. Taking (2) into account we find that

= 0, k, 2k, 3k, .. .. gl

(s) and 92 (s) of Z 1 and Zz

On the other hand, direct computation shows that

92(S)

= exp[-..x(l

- Sk)],

lsi

~ 1.

Since ZI and Z2 are independent, the p.g.f. 93 of the sum Z\ gl and 9z. Thus

+ Zz

is the product of

121

RANDOM VARIABLES AND BASIC CHARACTERISTICS

But, looking at (2), we see that 03 is the p.g.f. of a generalized Poisson distribution of order k. Therefore the r. v. Z = ZI + Z2 has just this distribution. Moreover, Z is decomposed into a sum of two independent r. v.s ZI and Z2, neither of which has a generalized Poisson distribution of order k. The Raikov theorem is therefore not valid for generalized Poisson distributions of order k 2:: 2.

12.6. A case when the Cramer theorem is not applicable Recall first that the Cramer theorem can be reformulated in the following equivalent form. Let FI (x), x E lIt I and F2(X), x E lIt I be non-degenerate d.f.s satisfying the relation

(l) where c:I>a.O' is a d.f. corresponding to N(a, (12). Then each of Fr and F2 is a normal d.f. Suppose now that the condition (l) is satisfied only for x $ xo, where Xo is a fixed number, Xo < 00 (i.e. not for all x E lIt I). Is it true in this case that FI and F2 are nonnal d.f.s? The answer follows from the next example. Denote by c:I> = c:I>O,1 the standard nonnal d.f. and define the function:

FJ(x)

= {2~(-1)"(X-n), v(x),

if x <0 if x

>0

where v( x) is an arbitrary non-decreasing function defined for x E (0,00) and such that v(O+) = FI (0) and v{ (0) = 1. It is easy to check that FI is a d.f. and let ~I be a r.v. with this d.f. Funher, let F2 be the d.f. of the r. v. 6 taking two values, 0 and 1, each with probability ~. Then we find that

i.e. condition (1) is satisfied for x $ Xo with Xo = O. However if x (Fr * F2 }{x) i: c:I>(x). Obviously FI and F2 are not normal d.f.s. Hence condition (1) in the Cramer theorem cannot be relaxed.

>

0, then

12.7. A pair of unfair dice may behave like a pair of fair dice Recall first that a standard and symmetric die (a fair die) is a term used for a 'real' material cube whose six faces are numbered by 1, 2, 3, 4, 5, 6 and such that when rolling this die each of the outcomes has probability Suppose now we have at our disposal four dice: white, black, blue and red. The available information is that the white and the black dice are standard and symmetric.

i.

122

COUNTEREXAMPLES IN PROBABILITY

Then the sum X + Y of the numbers of these two dice is a r. v. which is easy to describe. Clearly, X + Y takes the values 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 and 12 with 'l" I 2 3 4 5 6 5 4 3 2 d I . I pro ba bI ItIes 36' 36' 36' 36' 36' 36' 36' 36' 36' 36 an 36' respective y. Suppose we additionally know that the blue and the red dice are such that the sum ~ + 1J of the numbers on these two dice is exactly as the sum X + Y obtained when rolling the white and the black dice (i.e. ~ + 1J takes the same values as X + Y with the same probabilities shown above). Does this information imply that the blue die and the red die are fair, i.e. that each is standard and symmetric? It turns out the answer to this question is negative as can be seen by the following physically realizable situation. Take a pair of ordinary dice changing, however, the numbers on the faces. Namely, the faces of the blue die are numbered 1,2,2,3, 3 and 4 while those of the red die are numbered 1, 3,4, 5, 6 and 8. If ~ and 1J are the numbers appearing after rolling these two dice, we easily find that indeed ~ + 1J d X + Y. Hence, despite the facts that the sum X + Y comes from a pair of fair dice and that X + Y has the same distribution as ~ + 1J, this does not imply that the blue and the red dice are fair. The practical advice is: do not rush to pay the same for a pair of dice with fair sums as for a pair of fair dice!

12.8.

On two properties of the normal distribution which are not characterizing properties

Let X and Y be independent r.v.s distributed N(O, 1). Then the ratio X/Y has a distribution

P[X/Y

~ z] = (2rr)-1

II

exp

(_~x2) exp ( _~y2) dxdy,

zE

jRl.

x/ySz

It is easy to check that

(P[X/Y ~ z])~

= 1/[rr(l + z2)].

Hence X / Y has a Cauchy distribution. Let us call this property (NIN -7C). The presence of the property (NIN -7C) leads to the following question. Suppose X and Y are i.i.d. r.v.s with zero means such that X/Y has a Cauchy distribution. Is it true that X and Yare normally distributed? By examples we show that the answer to this question is negative. (i) Consider two i.i.d. r.v.s

f(x)

~

and 1J having the density

= (J2/rr) /(l + X4),

x

E jRl.

123

RANDOM VARIABLES AND BASIC CHARACfERISTICS

If 9 (z), z E 1R I denotes the density of the ratio {/17 then we easily find

g(z)

=

((2/7r2)!~ f

= ((2/1T')

i:

(l

}x/y
(1

+ x 4)-1 (1 + y4)-ldXdY)

+ y4)

I

x

'( 1 + zV)-'lyl dy )

~ = [1T( 1 + Z')]-I.

Therefore the ratio {/17 has a Cauchy distribution but obviously the variables { and 17 are non-normally distributed. Thus we have established that the property (NIN-tC) is not a characterization property of the normal distribution. (ii) Let us consider another case on the same topic. It can be checked that

2x4 f,(x) = 7r(1 + x 2 )(1 + x4)' is a density function. Take two independent r.v.s XI and YI each with density fl. Then the ratio Z\ = XI/Y\ has a Cauchy distribution (for details see Steck 1959), Clearly X I and Y1 are not normally distributed. It should be noted that the variables X and Y above are independent and this condition is essential for XI Y to be Cauchy distributed. (iii) It turns out the ratio XI Y has a Cauchy distribution for some cases when X and Y are dependent. (We can guess that now the reasoning is more complicated.) Indeed, consider the function

It can be shown that 'l/J is the ch.f. of a two-dimensional random vector, say ({, 17) such that: 1) each of { and 17 is not normally distributed (look at the marginal ch.f.s of f. and 17); 2) f. and 17 are dependent; 3) the ratio f.111 has Cauchy distribution. (For details see Rao 1984.)

(iv) Let XI, ... , Xn be n independent r.v.s each distributed normally N(O, 0'2), n > 2. Define

It is well known (see Feller 1971) that T has a Student distribution with n - 1 degrees of freedom. (Recall that T is often used in mathematical statistics.) Let us consider the converse. SupposeX t ,. " , Xn are Li.d. r.V.s with density f(x), x E 1R I, and we are given that the variable T has a Student distribution. Does it follow from this that f is a normal density? The example below shows that for n = 2 the answer is negative.

124

COUNTEREXAMPLES IN PROBABILITY

Let XI, X 2 be independent r.v.s each with density

f

and let

Our assumption is that T has a Student distribution. Thus the problem is to find the unknown density f. Firstly, let us introduce the functions hi (XI, X2), h 2(x, y) and h 3(z, y) which are the densities of the random vectors (XI, X 2 ), (X, s) and (T, s) respectively. We find that

XI

hI (XI, X2) = f(xi )f(X2), h 2(x,y) = 23/ 2f(x + 2- 1/ 2y)f(x - 2- 1/ 2y), h 3 (z,y) = 2f [y(z +

l)/Vi]

f [y(z -

E

)RI,

X2 E )RI, Y E )R+,

z E

)R I,

YE

X

l)/Vi] ,

E )RI,

)R+.

By the assumption above T has a Student distribution and clearly in this case (of two variables XI and X 2 ) T has a Cauchy distribution, that is the density of T is 1/[7r(1 + z2)], z E )RI. But the density of T can be obtained from h 3(z,y) by integration. Thus we come to the following relation:

It can be shown (see Mauldon 1956) that f is an even function and that the general solution ofCl) is of the form f(x) = 7r- 1/ 2 g(X2), X E )RI, where

10

(2)

00

g(u)g(au) du = 1/(1 + a),

a> o.

Furthermore, the integral equation (2) has an infinitely many solutions. However, for our purpose it is enough to take e.g. only two solutions, namely:

(a)

(b)

g(u)

= }2/7r(1 + u 2 ),

f(x) = ( V2/7r) /(1 + X4),

X E )RI.

Obviously in case (a) the variables XI and X 2 are distributed normally N(O, ~), while in case (b) the distributions of XI and X 2 are both non-normal. Therefore we have constructed two ij.d. r.v.s XI and X 2 whose distribution is non-normal but the variable T has a Student distribution. Finally it is interesting to note that the probability density function f(x) = (Vi/7r)/( 1 + x 4 ) has appeared in both cases (i) and (ii). (v) Recall the definition of the so-called beta distribution of the second kind denoted by /3(2) (a, b). We say that the r. v. X /3(2) ( a, b) if its density equals xa-I(l + x)-a-b/B(a,b), if x > o and 0, if x ~ o. Here a > 0, b > O. r-...J

125

RANDOM VARIABLES AND BASIC CHARACTERISTICS

The following result being of independent interest (see Letac 1995) is used for the reasoning below: Z has a Cauchy distribution e(O, 1) ¢::::::} Z is symmetric and

!).

IZI 2 '" (3(2) (1, Consider now two independent and symmetric r.v.s XI and X2 such that IXrl2"'{3(2)(!,b)

and

IX212 "'{3(2)(!,1 +b)

b>O.

forsome

Then, referring again to the book by Letac (1995) for details, we can show that the quotient X 21XI has a Cauchy distribution e(O, 1). Hence we have described another case of independent r.v.s XI and X2 such that their quotient X2/ XI follows a Cauchy distribution. Obviously, neither XI nor X2 is normally distributed. Note, however, that here we did not make the advance requirement that XI and X2 have the same distribution.

12.9.

Another interesting property which does not characterize the normal distribution

Let us start with the following result (for the proof see Baringhaus et al 1988). If X and Yare independent r.v.s, Z = XYI(X 2 + y2)1/2 and X '" N(O, aT), Y '" N(O, ai), then Z '" N(O, a 2) with a 2 = arai!(al + a2)2. It is interesting to poit out that Z is a non-linear function oftwo normally distributed r.v.s and, as stated, Z itself has also a normal distribution. This leads to the inverse question: if X and Yare independent r.v.s and Z is normally distributed, does it follow that X '" Nand Y '" N? Assume the answer to this question is positive. Thus we can suppose that Z '" N(O, 1). The definition of Z implies that

l/X2

+ l/y2

d

l1Z2.

It is easy to find that the distribution of 1I Z2 has a Laplace transform 1jJ (t) = exp(-V2t), t ~ 0, which means that l1Z 2 has a stable distribution with parameter Let us show that 1I Z2 admits the representation

!.

where UI and U2 are independent non-negative r.v.s such that the distribution of each of them does not have an 'atom' at and does not belong to the class of stable distributions with parameter For this we write 'I/J in the form

°

!.

1jJ(t) = exp [ __1_

roo (1 -

..J2i 10

e- tx )x- 3/2dX],

and introduce the following two functions of x E IR I:

t

>

°

126

COUNTEREXAMPLES IN PROBABILITY

(as usual I A (.) is the indicator function of the set A). Denoting

i.pj(t)

=

1=(] -

e-tX)x-lhj(x)ux,

=

we see that the integrals I 1 x-I hj (x) dx, j tPl (t)

= exp[-i.pl (t)],

t E]Rt

and

j

= 1,2

1,2, are convergent and both

tPz(t) = exp[-i.pz(t)], t E ]RI

are Laplace transforms of an infinitely divisible distribution with support [0,(0) (see FeHer 1971). Since 'I/Jj (t) -t 00 as t -t 00, j = 1,2, these distributions do not have 'atoms' at 0. Suppose now that Uj is a r.v. having Wj as its Laplace transform, j = 1,2. We can take UI and U2 to be independent. Then the Laplace transform of the distribution of UI + U2 equals tPl (t)tP2(t). However1f;1 (t)'l/J2(t) 'I/J(t). This fact and the reasoning above imply that 1I Z2 is the sum of UJ and Uz which are independent but. obviously, the distributions of 1lUI and llUz are not normal as they might be if the answer to the above question were positive. The interesting property described at the beginning is therefore not a characterizing property for a normal distribution.

=

12.10.

Can we weaken some conditions under which two distribution functions coincide?

Let us formulate the following result (see Riedel 1975). Suppose F (x), x E lR I, is an infinitely divisible d.f. and F2(X) = (x), x E ]Rt, where is the standard normal d.f. Then the condition J

(1)

implies that (2)

It is interesting to show the importance of the conditions in this result. In particular, the fonowing question is discussed by Blank (1981): does (1) imply that FI = F2 if we suppose that FI and F2 are arbitrary infinite1y divisible dJ.s, F2(x) > 0, x E ]Rt? By an example we show that the answer is negative. Introduce the functions

G I (x) := e- I

L

11k !,

k<x

and define FI and F2 as convolutions by

G2(X):= e

t

L 2k:S;x

11k!

127

RANDOM VARIABLES AND BASIC CHARACTERISTICS

Then both FI and F2 are infinitely divisible and F2(x) > 0, x E ]RI. Let us now estimate the quantity [FI (x)/ F 2 (x)] - 1 in two ways. We have

F,(x) L:~o(I/k!)(x - k) L:~, (I/k!)(x - k) F2 (x) - 1 = E~o(1/k!)(x _ 2k) - 1 ~ E~,(I/k!)(x - 2k)

< E~, (I/k!)(x - k) -

(x)

~

(x - 1) ~oo / (x) L..."k=1 (1 k!) -+ 0 as x -+

-00.

On the other hand, 1 _ FI (x)

< (x -

F2 (x) -

(x)

2)

E oo k=1

(11 k ') -+ 0 .

as x -+

-00.

Thus limx-+_oo[FI (x)1 F 2 (x)] = 1, that is relation (1) is satisfied, but FI

12.11.

i- F 2•

Does the renewal equation determine uniquely the probability density?

Let us start with a sequence {Xi,i = I,2, ... } of non-negative Li.d. r.v.s with a common d.f. F and density f. It is accepted to interpret Xi as a lifetime, or renewal time and it is important to know the probability distribution, say Ht. of the variable Nt defined as the number of renewals on the time interval [0, t]. In some cases it is even more important to find U (t) = ENt , the average number of renewals up to time t without asking explicitly for H t . We have

(1)

U(t) = F(t)

+

l

F(t - s)dU(s)

and hence the function u(t) = dU(t)/dt (which exists since 1 = F' exists), called a renewal density, satisfies

(2)

u(t) = I(t)

+

lot I(t - s)u(s) ds

for t

> O.

The tenn renewal equation is used for both (1) and (2) and we are interested in . how to find U (oru) in tenns of F (or j), and conversely. If 1* and u* are the Laplace oo oo transfonns of 1 and u (f* (a) = Jo e- at I(t) dt, u* (a) = Jo e-atu(t) dt), we easily find from (2) that

(3)

* U

1*(0:) (0:) = 1 _ 1*(0:) ,

0: ~

o.

Obviously, (3) and (2) imply that 1 detennines u uniquely. Consequently F detennines U uniquely. E.g. if F '" exp(.A), then 1*(0:) = .A/(.A + a), u*(o:) .A/o: :::} u(t) = .A for all t :::} U(t) = .At, a well known result.

COUNTEREXAMPLES IN PROBABILITY

128

Let us now answer the inverse question: does u determine 1 uniquely? It turns out the answer is negative. Recall first that the classical renewal theorem (Feller 1971) states that in general limt ...... = u(t) 1/p" where p, E[Xt] is the average lifetime. Let us show that there is a renewal density u(t) with u(t) -t 1/p, as t -t 00 for u*(a)/[I + u*(a)] found from (3) leads to a same J-' > 0 and such that 1*(0.) function l{t) which may not be a probability density. Indeed, take u (t) = (1 e - p.t) / P, for t > 0 and fixed p,. Obviously, u (t) -t 1/J-' as t -t 00. The Laplace transform u* (a) of this u(t) is

=

=

=

u*(a) = ]/[0.(0. + p,)]

=>

1*(0.)

= 1/(0.2 + ap, + 1).

Suppose now that p, < 2 (by assumption p, > 0). Inverting 1* we find that = e-p.t/2(c/2)-1 sin(ct/2), 0 < t < 00, where c = }4 - p,2 and that 1(t) dt = I. However the function 1 is not a probability density.

I(t)

Jo=

12.12. A property not characterizing the Cauchy distribution Suppose the r. v. X has a Cauchy distribution with density 1(x) = 1/ [7r(I + x 2 )], x E lR I. Then it is easy to check that the r. v. 1/ X has the same density. This property leads naturally to the following question. Let X be a r. v. which is absolutely continuous and let its dJ. be denoted by F(x), x E lR I. Suppose the r. v. 1/ X has the same d.f. F. Does it follow that F is the Cauchy distribution? Clearly, if the answer to this question is positive, then the property X d 1/ X would be a characterizing property of the Cauchy distribution. It turns out, however. that in general the answer is negative. Let us illustrate this by the foHowing example. Suppose X is a r. v. with density

g(x)

1/4, { 1/(4x2),

if if

Ixl ::; 1 Ixl > 1.

Thus X is absolutely continuous and it is not difficult to check that 1/ X has the same density g. Hence X distribution.

1/ X. It is obvious, however, that X does not enjoy the Cauchy

12.13. A property not characterizing the gamma distribution Let X and Y be independent r.v.s each with a gamma distribution "Y(p, a), P a > 0; that is, the common density is

(I)

I(x)

if x ::; 0 if x> O.

> 0,

129

RANDOM VARIABLES AND BASIC CHARACTERISTICS

XI Y has the following density:

Then the ratio Z =

0

(2)

g(z) = { [i/B(p,P)]zP-'(1 + z)-2p ,

if z ~ 0 if z > 0

(beta distribution of the second kind; see Example 12.8). This connection between gamma and beta distributions leads to the next question. Let X and Y be positive independent r. v.s such that the ratio Z XI Y has a density given by (2). Does this imply that each of X and Y has gamma distribution? Let us show that the answer to this question is negative. To see this, introduce the following two functions, where a > 0, p > 0:

f (x)

= { 0,

I

Clx-P-1e-a/x I

0,

h(x)

if x ::; 0 if x > 0,

= { C2XP 1[(1 + X 2 )p+I/2],

if x ~ 0 if x > o.

It is easy to check that with CI

aP/r(p)

and

C2

= 2r(p+ ~)/[r(!p)r(!p+

Dl

and h are density functions. Take two independent r.v.s, say ~I and TIl, each with density fl. Then we can establish that the density gl of the ratio (I = ~I ITII coincides with the density 9 given by (2). Clearly, however, in this case, fl does not have the form (I). The same conclusion can be derive if we start with two independent r.v.s, 6 and Tl2, each having the density h. In this case again the density of (2 = 61112 coincides with (2) while h is not of the form described by (1). fl

12.14. An interesting property which does not characterize uniquely the inverse Gaussian distribution We say that the r. v. X has an inverse Gaussian distribution with parameters /-L and A > 0 if the density f of X is given by

(1)

f(x) =

>0

A) 1/2 [ A (x - Jl.)2] . 27rx3 exp - 2/-L 2 x ,If x > 0 ( { 0,

if x

~

o.

It is easy to see that all moments Q n = E[xn], n = 1,2, ... , exist. Moreover, it can be shown that this distribution is determined uniquely by its moment sequence {Qn,n 1,2, ... }. It is interesting to note that all negative-order moments of X are also finite, that is E[X nl exists for each positive integer n. Further, a standard transformation leads to the following interesting relation:

(2)

130

COUNTEREXAMPLES IN PROBABILITY

This relation and the uniqueness of the moment problem mentioned above motivate the conjecture: if X is a positive r.v. such that all moments E[xn] and E[x-n], n = 1,2, ... , exist and satisfy (2), then X has an inverse Gaussian distribution. It turns out, however, that this conjecture is not correct. Note firstly that EX = JJ and let for simplicity JJ = 1. Then (2) has the form E[x-n] = E[xn+I]. Further, the density (1) satisfies the relation

(3)

x/ex) = (l/x 2 )/(I/x),

x > O.

Thus the density / of the inverse Gaussian distribution can be considered as a solution of the functional equation (3). Let Y be a r. v. whose density 9 satisfies (3). Then it is easy to check that the relation (2) is fulfilled for Y. So it is clear that if(3) has a unique solution, namely f by (1), then our conjecture will be true; otherwise it will be false. To clarify this consider the function 9 given by

(4)

2 I I g(x):::: { -; JX 1 + 0,

,

if x

>0

if x ::; O.

It can be verified directly that 9 is a probability density function which satisfies (3). As a consequence, Y satisfies (2). Therefore we have found two r.v.s, X and y, whose densities (1) and (4) are different, and nevertheless both satisfy relation (2). Thus the relation (2) is not a characterizing property of the inverse Gaussian distribution. Finally we suggest that the reader considers equation (3) and tries to find other solutions to it which will provide new r. v.s satisfying relation (2).

SECTION 13. DIVERSE PROPERTIES OF RANDOM VARIABLES In this section we consider examples devoted to different properties of r. v.s and their numerical characteristics. Some notions are defined in the examp]es themselves.

13.1.

On the symmetry property of the sum or the difference of two symmetric random variables

Recall first that the r. v. X is called symmetric about 0 if X ( - X). In terms of the d.f. F, the density f and the ch.f. r,p this property is expressed as follows: F( -x) :::: I - F(x) for all x ~ 0; f( -x) :::: f(x) for all x E jR'; r,p(t), t E jR' takes only real values. By the examples below we analyse the symmetry and the independence properties under summation or subtraction. (i) If X and Y are identically distributed and independent r. v.s, then their difference

X - Y is symmetric about O. Suppose we know that X

d

Y and that the difference

131

RANDOM VARIABLES AND BASIC CHARACTERISTICS

x-

Y is symmetric. Does it follow that X and Y are independent? To see this consider the random vector (X, Y) defined as follows:

y

1 2 3

X 1

2

I

2

12 12 12 I I T2 0 12

3

4 2 12 0 12 .

It is easy to check that X and Y have the same distribution, each taking the values 1, 2 and 3 with probability equal to ~, ~ and &respectively. Obviously, X and Y are not independent. Further, the difference Z = X - Y takes the values -2, -1, 0, I, 2 with probabilities 366' {6' ~~, {6' 3~' Clearly Z and ( - Z) have the same distribution. X - Y is a symmetric r.v. despite the fact that the variables X In other words, Z and Y are not independent. (ii) If X and Y are symmetric and independent r. V.s, then the sum Z = X + Y is again symmetric. Thus it is of interest to discuss the following question. Suppose X and Y are independent r. v.s and we know that X is symmetric and that the sum Z = X + Y is also symmetric. Is it true that Y is symmetric? Intuitively we could expect a positive answer. It turns out, however, in general the answer is negative. This is illustrated by the following example. Let ~ be a r. v. with the following ch.f. indicating that ~ is symmetric:

1/Jdt ) =

(I)

{Ol,- 21tl,

if if

It I S 1/2 It I > 1/2.

Consider two other ch.f.s:

_{IJ/(4Itl), Itl,

hl(t) -

if

ItI S 1/2

if It I

>

1/2,

h (t) _ { I 2 0,

Itl,

s

if It I I if It I > I.

Introduce now a r. v. 11 with ch.f.1/Jl1 which is the mixture of hI and h2:

1/Jl1 (t) = !eithl (t) + !e- ith 2(t),

t E )RI.

Elementary transformations show that

(2)

_{(Ieit j(8It!) It I) + (lj2)c(t)e-

1/J l1 (t) -

cost,

it

(l

-Itl)'

s

if It I 1/2 if It I > 1/2

wherec(t) = I, ifltl < 1 andc(t) - 0, ifltl > 1. The explicit form (2) of the ch.f. tPl1 shows that the r. v. 11 is not symmetric. Thus we have described two r.v.s, and 11, the first being symmetric while the second is not. Assuming that ~ and 11 are independent we look for the properties of

e

132

COUNTEREXAMPLES IN PROBABILITY

e

the sum ( = 'fl. Since for the ch.f.s 'l/Je, 'l/J1J and 'I/J(, we have 'I/J(, of ( I) and (2), it is not difficult to find that 'I/J (t)

(,

= {(I - 21tl)(1 -Itl)cost, 0,

if if

= 'l/Je'I/J1J' in view

It I ::; 1/2 ItI > 1/2.

Obviously 'I/J(, takes only real values which means that the r. v. ( is symmetric. Therefore the symmetric property of two variables. and ( = + 'fI, together with the independence of and 'fI, do not imply that 'fI is symmetric. Here is another equivalent interpretation-the difference, and hence the sum. of two dependent r. v.s both symmetric, need not be symmetric.

e

e

e

13.2. When is a mixture of normal distributions infinitely divisible? Let G (u), u E IR + be a d.f. Then the function 'I/J( t), t E IR I where

(1) is a ch.f. The d.f. F with ch.f. 'I/J is called a mixture of normal distributions and G a mixing distribution. Note that the density f of F corresponding to (l) has the form (see Kelker 1971): f(x)

= 10

00

(27ru)-1/2 exp( -x2/(2u)) dG(u).

Since the normal distribution is infinitely divisible it is natural to ask whether such a mixture preserves the infinite divisibility. It is easy to check that if G is an infinitely divisible d.f. then 'I/J is an infinitely divisible ch.f. Now we want to answer the converse question: if'I/J is an infinitely divisible ch.f., does it follow that the mixing distribution G is infinitely divisible? Consider the function H(x), x E IRI where H(x) = 0, 0.26, 0.52, 0.48, 0.74 and I respectively in the intervals

(-00, I], (1,2], (2,3], (3,4], (4,5] and (5,00). Clearly H is not a d.f. However, we obtain the interesting and unexpected fact that the convolution H * H is a d.f. Moreover, the function Jooo (2 7ru) - t /2 exp( - x 2 1(2u)) dH (u) is a density. Define G as follows: 00

(2)

G(x} = e-

1

I)k!) k=O

1H*k(x).

133

RANDOM VARIABLES AND BASIC CHARACfERISTICS

We can verify that G given by (2) is a d.f. and find that

f

exp (

-~t'u) dG(u) =

e-

I

[f

~(k!)-I 00

= exp [1

exp

= exp (1OO[cos(tx) -1](27r)-1/21

exp (

(_~t2u) OO

+'u) dH(u)] ,

dH(u)

-1]

u- I / 2ex p[-x 2/(2U)]dH(u)dX) .

The last expression in this chain of equalities coincides with the Kolmogorov canonical representation for an infinitely divisible ch.f. provided oo that fo it- I / 2 exp( -x 2 /(2u)) dH(u) 2: 0 for all x > 0 (see Gnedenko and Kolmogorov 1954). But H satisfies this condition by construction. Therefore 1/J defined by (l), with G given by (2), is an infinitely divisible ch.f. It remains for us to show that G in (2) is not infinitely divisible. This folJows from the Levy-Khintchine representation for the ch.f. of G and from the fact that H is not non-decreasing.

13.3. A distribution function can belong to the class IFRA but not to IFR Let F(x), x 2: 0 be a d.£. with density J. We say that F is an increasing failure rate distribution and write F E IFR, if its failure rate r(x) := J(x)/(l - F(x)) is increasing in x, x > O. In this case - Jog[1 - F(x)] is a convex function in the domain where -log[l - F(x)] is finite. This observation motivates the more general definition: F E IFR if -log[l - F(x)] is convex where finite. However, for some problems it is necessary to introduce a considerably weaker restriction on F. For example, if F has density J and failure rate r such that (1/ x) foX r( it) dit is increasing in x, we say that F has an increasingfai/ure rate average. In this case we write FE IFRA. More generally, FE IFRA if (-l/x) log[l - F(x)] is increasing where finite. Thus we have introduced two classes of distributions, IFR and IFRA, and it is natural to ask what the relationship between them is. According to Barlow and Pros han (1966). if F E IFR and F(O) = 0 then F E IFRA. We are interested now in whether the converse is true. To see this, consider the function

0

F(x) = {

(i - e-X)(l -

It is easy to check that F E IFRA but F

e- kx ),

¢ IFR.

if x ~ 0 if x > 0, k

> 1.

134

13.4.

COUNTEREXAMPLES IN PROBABILITY

A continuous distribution function of the class NBU which is not of the class IFR

A d.f. F (of a non-negative r.v.) is said to belong to the class NBU (new and better than used) if for any x, y ~ 0 we have

F(x + y) :5 F(x)F(y)

(1) If for any y

F = 1 - F.

where

> 0 the function [F(x + y) - F(x)]/ F(x)

is increasing in x, we say that F is of the class IFR (compare this definition with that given in Example 13.3). It is well known that F E IFR =} F E NBU, but in general the converse implication is not true (see Barlow and Proshan 1966). The d.f. F E IFR has the property that it is continuous o'n the set {x : F(x) < I} and, moreover, h(x) = -log F(x) is a convex function. However, the elements of the class NBU need not be continuous. This follows from a simple example. Indeed, Consider the function

F(x)

=

1-2-k

xE (k,k+ I],

for

k=O,I,2, ....

It is easy to check that (I) is satisfied and hence F E NBU. Obviously F is discontinuous and hence F ¢ IFR. Suppose now that F E NBU and F is continuous. Does it follow from these conditions that F E IFR? It turns out that the answer is negative. To see this consider the function _ { sin 2 x, if x E [0, 111'] h( x ) - I( 1) . x E (211', 1 00 ) . 211' X - 211' + I, If It is easy to check that F(x)

h(x

=I-

+ y)

~

e-h(x),

h(x)

x ~ 0 is a dJ. and, moreover, that

+ h(y),

x,y

~

O.

Therefore F E NBU and clearly F is continuous. Nevertheless F h(x) = -log F(x) is not a convex function.

13.5.

¢

IFR since

Exchangeable and tail events related to sequences of random variables

Let {Xn,n ~ I} be an infinite sequence of r.v.s defined on the probability space (0, ~, P). Denote by a{ XI, ... ,Xu} the a-field generated by XI, ... ,Xu' Then clearly U~I a{Xn,Xn+ I , ... ,Xn+d is a field and let a{Xn,Xn+I, ... } be the a-field generated by this field. The sequence of a-fields a{Xn,Xn+ I , .. . }, a{ X n + 1, X n +2, ... }, ... is non-increasing, its limit exists and is a a-field. This limit is denoted by

'J =

n

a{Xn,Xn+" .. . }.

n=1

RANDOM VARIABLES AND BASIC CHARACTERISTICS

135

'I is called the tail a-field of the sequence {Xnl n ~ I}. Any event A E 'I is called a tail event, and any function on n which is measurable with respect to 'I is said to be a tail function. Let us fonnulate the basic result concerning the tail events and functions. Kolmogorov 0-1 law . Let {X n} be a sequence of independent r. v.S and 'I be its tail a-field. Then for any tail event A. A E 'I, either P(A) = 0 or P(A) = 1. Moreover, any tail function is a.s. constant, that is, if Y is a r.v. such that a{Y} C 'I then pry = cJ = 1 with c = constant We now introduce another notion. (Also see Example 7.14). We say that the r. v.s Xl l' .. 1Xn are exchangeable (another tenn is symmetrically dependent) if for each of the n! pennutations {i II i l , ... , in} of {I, 2, ... , n}, the random vectors (Xii 1X iZ )' .. 1Xi n ) and (XI, Xl,· .. 1 Xn) have the same distribution. Further, an infinite sequence {Xnl n ~ I} is said to be exchangeable if for each n the r.v.s Xh ... ,Xn are exchangeable. The ~oo-measurable function g(XJ,X2, ... ) is called exchangeable if it is invariant under all finite pennutations of its g(Xill . .. ) X in ) X n+ 1, •.. ). In particular, arguments: g(X" ... ,Xn , Xn+ I,· .. ) A E a{XJ, X2,"'} is caned an exchangeable event if its indicator function I(A) is an exchangeable function. Let us fonnulate a result concerning the exchangeability.

Hewitt-Savage 0-1 law. Let {Xn) n ~ I} be a sequence of r.v.s which are independent and identically distributed. Then for any exchangeable event A E a{Xl,X21"'} eitherP(A) OorP(A) 1.

=

Note that a detailed presentation of the notions and results given briefly above can be found in the books by Feller (1971). Laha and Rohatgi (1979), Chow and Teicher (1978), Aldous (1985) and Galambos (1988). Obviously tailness and exchangeability are close notions and it would be useful to illustrate by a few examples the relationships between them.

(i) The first question concerns the tail and exchangeable events. Let {Xnl n 2:: I} be a sequence of (real-valued) r.v.s and 'I its tail a-field. If A E 'I, then for any pennutation {ill" .) in} of {I, ... 1 n}, n ~ 1, we can write A in the fonn {(Xn+ll Xn+21"') E Bn+d where Bn+l isa Borel set in !Roo, that is, Bn+l E ~oo. Thus for each n, n n A = {(Xl, Xl,"') E !R x B n+ l } - ((Xip ... ) Xi .. ,Xn+h"') E !R x Bn+d and since Boo := Iltn x Bn+ I is a Borel set in }Roo, this implies that the tail event A is also an exchangeable event. However, there are exchangeable events which are not tail events. The simplest example is to take A {Xn 0 for all n 2:: I}. Obviously A is an exchangeable event. But A ¢ a{ X n, Xn+ I, ... } for every n ~ 1. So A is not a tail event.

(ii) Now let us clarify the possibility of changing some of the conditions in the Hewitt-Savage 0-1 law. Consider the sequence {Xnl n 2:: I} of independent r.v.s

COUNTEREXAMPLES IN PROBABILITY

136

!

=

=

where P[XI 1] P[XI = -1] = and P[Xn OJ = 1 for n 2: 2. The event A = {2: =1 Xj > 0 for infinitely many n} is clearly an exchangeable (but not tail!) event with respect to the infinite sequence of r.v.s {Xn,n 2: I}. Moreover, P(A) P[XJ > 0] = and hence P(A) is neither 0 nor 1 as we could expect. Here the Hewitt-Savage 0-1 law is not applicable since the independent r.v.s X n , n 2: 1 are not identically distributed.

J

!

(iii) Let X n , n

2: 1 be independent r.v.s such that

P[Xn=1]=2-n, and let A - {Xn = 0 for all n Further, we have

P(A)

P[Xn =Oj-I-2- n ,n

1,2, ...

2: I}. Then A is an exchangeable but not a tail event. 00

00

n:-::l

n:::-:I

= II P[Xn = 0] = II (1

- 2-n).

Since 2::'= 1 2 -n < 00, the infinite product n:':::::: I (I - 2- n ) converges to a positive limit which is strictly less than 1 and hence P(A) is neither 0 nor 1. Therefore again, as in case (ii), the Hewitt-Savage 0-1 law does not hold. Note that here the variables X n , n 2: 1, are independent and take the same values but with different probabiJities.

13.6. The de Finetti theorem for an infinite sequence of exchangeable random variables does not always hold for a finite number or such variables Let {X n, n 2: I} be an infinite sequence of exchangeable r. v.s each taking two values, o and 1. Then according to the de Finetti theorem (see FeHer 1971) there is a unique probability measure p on the Borel a-field 'B[O, I] of the interval [0, 1] such that for each n we have

where ej = 0 or I and k = e, + ... + en. In other words, the distribu lion of the number of occurrences of the outcomes 0 and I in a finite segment of length n of the infinite sequence XI, X 2 , •.. of exchangeable variables is always a mixture of a binomial distribution with some proper distribution over the interval [0, 1]. Thus we come to the question of whether this result holds for a fixed finite exchangeable sequence. The answer can be found from the following two examples. (i) Consider the case n = 2 and the r.v.s XI and X 2 :

P[X, - 0, X 2 = 1] = P[X,

= 1,

X2

0]

137

RANDOM VARIABLES AND BASIC CHARACTERISTICS

P[XI

0, X2

0]

P[XI = 1, X 2 = 1] = O.

It is easy to see that XI and X 2 are exchangeable. Suppose a representation like (1)

holds. Then it would foJ]ow that

l

This means that

jJ

p'/J (dp)

0

and

l (\ -

p)'/J (dp)

= O.

puts mass one both at 0 and at 1, which is not possible.

Let Y1 , ••• , Yn be n independent r.v.s with some common distribution. Let Sn YI + ... + Y n and Zk Yk n-ISn for k = 1, ... ,n - 1. Then it is easy to check that the r. v.s ZI, ... , Zn-l are exchangeable but their joint distribution is not of the form (1). Therefore the de Finetti theorem does not always hold for a finite exchangeable sequence. (ii)

=

13.7. Can we always extend a finite set of exchangeable random variables? If {Xn} is a finite or an infinite sequence of exchangeable r. v.s then any subset consists of r.v.s which are exchangeable. Suppose now we are given the set XI"'" Xm of exchangeable r.v.s. We say that XI, ... ) Xm can be extended exchangeable if there is a finite set XI, ... 1 X ml X m+ l ) ... , X m +k , k > 1, or an infinite set XI, ... , X m , X m+ l , X m +2 , ••. of r.v.s which are exchangeable. Thus the question to consider is: can we extend any fixed set of exchangeable r.v.s to an infinite exchangeable sequence? Let us show that in general the answer is negative. Consider the particular case of three r. v.s XI, X 2. X3 each taking the values 0 or 1 with P[Xj = 1] = P[Xj = OJ = ~,j = 1,2,3. Let

P[XI

1,X2 - 1] = P[XI

1,X3

1] = P[X2

I,X3 = 1] =0.2.

It is easy to see that (Xl, X 2 , X 3 ) is an exchangeable set. Assume this set can be extended to an infinite exchangeable sequence Xl, X 2 , X 3 , X4! X5,' ... This would mean that for each 11 ~ 4 the set XI, X 2 , X 3 , X 4, . .. ,Xn consists of exchangeable variables. Then we can easily show that

o :::; =

E[(2:7=1 J(Xj = 1))2]- (E [2:7=1 J(Xj = I)]r 2:7=1 2:~=1 P[Xj 1, Xk = IJ *n 2

~n + Lj~k P[Xj

tn + (0.2)n(n -

0, Xk

1) - tn2

1.] -

!n2

= (0.3)n -

(0.05)n2.

Obviously it follows from this that n must satisfy the restriction n :::; 6. However, this contradicts the definition of an infinite exchangeability and therefore the desired extension of a finite to an infinite exchangeable sequence is not always possible.

138

COUNTEREXAMPLES IN PROBABILITY

Interesting results on exchangeability of finite or infinite sequences of random events or r.v.s can be found in the works of Kendall (1967) and Galambos (1988). Finally let us mention that the variables in an infinite exchangeable sequence are necessarily non-negatively correlated. This follows directly from an examination of the tenns of the variance V[XI + ... + Xn]. However, in the above specific example we have P(Xl' X 2 ) < O.

13.8.

Collections of random variables which are or are not independent and are or are not exchangeable

Let X := {Xn,n ~ 2} be a finite or infinite sequence of r.v.s which are independent and identically distributed. Then X is exchangeable, that is both properties independence and exchangeability hold for X in this case. If, however, Xn (at least two of them) have different distributions then X is not exchangeable regardless of whether X is independent or dependent. Our goal now is to describe a sequence of r. v.s which are totally dependent and exchangeable. For consider a sequence X = {Xn, n ~ 2} of i.i.d. r.v.s each with zero mean and finite variance. Let ~ be another r.v. independent of X. Assume that ~ is non-degenerate with finite variance, that is, 0 < V~ < 00. Let us define a new sequence ~ := {Yn , n ~ 2} where Yn = Xn +~. It is easily seen that ~ is exchangeable. Let us clarify whether or not ~ IS independent. The distribution of Yi l , • • • , Yile is the same for any possible choice of k variables from~, k = 1,2, .... Taking k = 2 we conc1ude that ~ is characterized by a common correlation coefficient, say Po where

po = p(Yi, Yj) = (E[YiYj] - EYiE1j)/(VYiVlj)I/2 for any two representatives Yi and lj

of~.

A simple reasoning shows that

Po = (V~)/(VXI

+ V~)

where V Xl (= E[X1D is the common variance of the sequence X. The assumption o < V~ < 00 implies that Po :/: 0 (in fact Po > 0) and hence Yi and lj cannot be independent because they are not even uncorrelated. In other words, ~ is totally dependent in the sense that there is no pair of variables in ~ which is independent. Hence the sequence ~, finite or infinite, is dependent and exchangeable. The final conclusion is that these two properties, independence and exchangeability, are incompatible.

13.9.

Integrable randomly stopped sums with non-integrable stopping times

Let X and Xl, X2, .. , be i.i.d. r. v.s defined on the probability space (0, ~,P) and {~n, n ~ O} where ~o = {0,0} is an increasing sequence of sub-a-fields of ~.

RANDOM VARIABLES AND BASIC CHARACTERISTICS

139

Recall that the r. v. r with possible values in the set {I, 2, ... ,oo} is said to be a stopping time with respect to frn} if the set [w : T( w) = n] denoted further simply by [r = n] belongs to ~n for each n. If So = 0, Sn = Xl + ... + Xn then ST is the sum of the first r of the variables Xl, X 2 , • • " that is ST = Xl + ... + X T. For many problems it is important to have conditions under which the r. v.s X, rand ST are integrable. Let us formulate the following result (see Gut and Janson 1986). Let r ~ 1 and EX =I- O. Then E[lXn < 00 and E[lSrn < 00 imply that E[rr] < 00. Our aim now is to show that the condition EX =I- 0 is essential for the validity of this result. So, let the r. v. X have EX = O. In particular, take X such that P[X = I] = P[X = -I] = and r = min{n : Sn = I}. Clearly r is a stopping time with respect to {~n} where ~n = a{X I , ••• , X n }. It is easy to check that the r.v. X and the random sum ST have moments of all orders, that is, for any r > 0 we have E[lXn < 00 and E[lSTn < 00. However, E[T I / 2 ] = 00 and therefore E[rr] does not exist for any r ~

!

k.

Part 3

Limit Theorems

Prc)lessor A T. F omenko

M()SC,OW

University.

LIMIT THEOREMS

143

SECTION 14. VARIOUS KINDS OF CONVERGENCE OF SEQUENCES OF RANDOM VARIABLES On the probability space (.0,1', P) we have a given r.v. X and a sequence of r.v.s {Xn' n ~ I}. Important probabilistic problems require us to find the limit of X n as n -+ 00. However. this limit can be understood in a different way. Let us define basic kinds of convergence considered in probability theory. (a) We say that {Xn} converges almost surely (a.s.). or with probability 1, to X as n -+ 00 and write Xn X if

P[w: lim Xn(W) n-+oo

= X(w)] =

1.

(b) The sequence {Xn} is said to converge in probability to X as n € > 0 we have lim P[w: IXn(w) - X(w)1 ~ €J = O. n-+oo

-+

00

if for any

In this case the following notation is used: Xn ~ X or P - limn-+oo Xn X. (c) Let F and Fn be the dJ.s of X and Xn respectively. The sequence {Xn} is called convergent in distribution to X if lim Fn(x) = F(x) n-+oo for all x E lR I for which F(x) is continuous. Notation: Fn ~ F, and X n X. (d) Suppose X and X n • n ~ 1, belong to the space LT for some T > 0 (that is E[lXITJ < 00, E[IXnrJ < (0). We say that the sequence {Xn} converges to X in LT -sense, and write Xn

X, if lim E[lXn n-+oo

Xn = O.

In particular, the LT -convergence with T = 2 is called square mean (or quadratic mean) convergence and is used so often in probability theory and mathematical statistics. Note that the convergence in distribution defined in (c) is closely related to the weak convergence treated in Section 16 (see also Example l4.I(iii». Some notions (complete convergence. weak L I-convergence and convergence of the Cesaro means) are introduced and analysed in Examples 14.14-14.18. Practically all textbooks and lecture notes in probability theory deal extensively with the topics of convergence of random sequences. The reader is advised to consider the following references: Parzen (1960). Neveu (1965). Lamperti (1966), Moran (1968), Ash (1970), Renyi (1970), Feller (1971), Roussas (1973). Neuts (1973), Chung (1974), Petrov (1975), Lukacs (1975). Chow and Teicher (1978), Billingsley (1995), Laha and Rohatgi (1979), Serfling (1980) and Shiryaev (1995).

144

COUNTEREXAMPLES IN PROBABILITY

It is usual for any course in probability theory to justify the following scheme. convergence In probability

convergence a.s.

convergence in distribution

Iconvergence in LT -sense I In this section we consider examples which are different in their content and level of difficulty but all illustrate this general scheme clearly. In particular, we show that the inclusions shown above are all strong inclusions. The relationship between these four kinds of convergence and other kinds of convergence of random sequences is also analysed.

14.1.

Convergence and divergence of sequences of distribution functions

We now summarize a few elementary statements showing that a sequence of d.f.s {Fn , n 2: I} can have different behaviour as n -+ 00. In particular it can be divergent, or convergent, but not to a d.f.

F(x), x E R' be a d.f. which is continuous. Consider two sets of d.f.s, {Fn,n 2: I} and {Gn,n > I} where

(i) Let

Obviously Fn{x) -+ 1 as n -+ 00 for each x E RI. But a function equal to I at all points is not a d.£. Hence {Fn} is convergent but the limit limn-too Fn{x) is not a d.f. Further, G 2n (x) -+ I whereas G2n+1 (x) -+ as n -+ 00 for all x E R'. Clearly the family {G n} does not converge.

°

(ii) Consider the family of d.f.s {Fn' n

{

°

2: I} where

a, 1,

if x if x

°

< >

1 n 1... n

Then Fn{x) -+ F{x) if x:;:. and Fn(O) for each n 2: 1, where F is the d.f. of a degenerate r.v. which is zero with probability 1. Thus limn-too Fn(x) exists but is not a d.f. because it is not right-continuous at x 0. (iii) The following basic result is always used when considering convergence in distribution (see Lukacs 1970; Feller 1971; Chow and Teicher 1978; Billingsley 1995; Shiryaev 1995):

( I)

Fn

~ F.;::::}

[ g(x) dFn(x) -+ [ g(x) dF(x)

JRi

JR i

for all continuous and bounded functions g(x), x E jRl.

145

LIMIT THEOREMS

Despite the fact that (1) contains a necessary and sufficient condition it is useful to show that the assumptions for g cannot be weakened. For example take g bounded and measurable (but not continuous), say

g(x) =

{O,1,

0

~f x ~ If x > O.

Denote by F and Fn the dJ.s of the r.v.s X

=0

=Ijn

and Xn

Fn ~ F and obviously J g dFn = I for each n ~ I but

respectively. Then

J g dF =

O. Therefore (I)

does not hold, as we of course expected. Finally, recall that the integral relation (I) can be used as a definition of the weak convergence of d.f.s (see Example 14.9 and the topics discussed in Section 16).

14.2.

Convergence in distribution does not imply convergence in probability

We show by a few specific examples that in general as n ---7

Xn

d

~

'::/?

X

Xn

P

~

00,

X.

(i) Let X be a Bernoulli variable, that is X is a r. v. taking the values I and 0 with probability ~ each. Let {Xn, n ~ I} be a sequence of r.v.s such that Xn = X for any

n. Since Xn d X then Xn ~ X as n ---7 00. Now let Y = I-X. Thus Xn ~ y because Y and X have the same distributions. However, Xn cannot converge to Yin any other mode since

IX n

-

YI

= I always. In particular,

P[lXn - YI > c] f+ 0 P

for an arbitrary e E (0, I) and therefore Xn

-14 Y

as n ---7

00.

(ii) Let n = {WI, W2, W3, W4}, l' be the a-field of all subsets of nand P the discrete uniform measure. Define the r.v.s X n , n ~ 1, and X where

Xn(WI)

= X n (W2) = 1,

X n (W3)

X(WI) = X(W2) = 0,

= X n (W4) = 0,

n

~ I,

X(W3) = X(W4) = 1.

Then IXn(w) - X(w)1 = I for all wEn and n ~ 1. Hence as in case (i), Xn cannot converge in probability to X as n ---7 00. Further, if F and F n , n ~ I, are the d.f.s of X and X n , n ~ 1, we have

0, F(x) = { ~, 1, Thus Fn(x)

= F(x)

if x ~ 0 if O<x~ I if x > 1,

Fn(x)= {

if x < 0 ifO<x~1 if x > I.

0, ~,

I,

for all x E ~I and trivially Fn(x) ---7 F(x) at each continuity

point of F. Therefore Xn ~ X but, as was shown, Xn

P

---f+ X.

146

COUNTEREXAMPLES IN PROBABILITY

(iii) Let X be any symmetric r.v., for example X '" N(O, 1), and put Xn for each n arbitrary E

14.3.

d

2: 1. Then Xn = X > 0 we have

d

and Xn ---7 X. However, Xn

P

--f-+ X

= -X

because for an

Sequences of random variables converging in probability but not almost surely

n

(i) Let = [0, I], l' = 13[0,1] and P be the Lebesgue measure. For every number n E N there is only one pair of integers, m and k, where m 2: 0, 0 ~ k ~ 2m - 1, such that n = 2m + k. Define the sequence of events An = [k2- m , (k + 1)2- m ) and put Xn = Xn(w) = IA,,(w). Thus we obtain a sequence ofr.v.s {Xn,n 2: I}. Obviously if 0 ~ E < 1 if E > 1. Since n -t

00

iff m -t

00,

we can conclude that

Xn

(1)

P ---70

as n -t

00.

Now we want to see whether in (1) the convergence in probability can be replaced by almost sure convergence. It is easy to show that for each fixed wEn there are always infinitely many n for which Xn(W) = 1 and infinitely many n such that Xn(w) = O. Indeed, w E [k2- m , (k + 1)2- m ) for exactly one k where k = 0,1, ... ,2 m - 1, that is w E A 2,"+I. Obviously, if k < 2m - 1 then also W E A 2"'+k+1 and if k = 2m - 1 (and m 2: 1) then also W E A 2",+I. In other words, wEAn i.o., and also W E n\An i.o. which means that lim sUP n --+ oo Xn = I and lim infn--+oo Xn O. Therefore

=

a.s.

Xn

--f+ 0

(ii) Consider the sequence {X n, n

P[Xn = I] Obviously for any

E,

0

P[lXn -


01

1

= -, n

2: I}

as n -t

00.

of independent r. v.s where

P[Xn

1

= 0] = 1--, n

n > 1.

I we have

> E] = P[Xn = 1] =

!n

-t 0

as

n -t

00

and thus Xn ~ 0 as n -t 00. It turns out, however, that the convergence Xn ~ 0 fails to hold. Let us analyse a more general situation. For given r. v.s X, X n, n 2: 1,

147

LIMIT THEOREMS

define the events

Then

Xn ~ X as n -7

(2)

00

{=::}

P(Bm(c:)) -70 as m -7

00

for all c:

> O.

Indeed, let

C

= {w En: Xn(W)

-7 X(w) as n -7 oo},

A(c:)

= {w En: w E An(C:) i.o.}.

Then P(C) = 1 iff P(A(c:)) = 0 for all c: > O. However {Bm(C:)} is a decreasing sequence of events, Bm(C:) -!. A(c:)asm -7 00 and soP(A(c:)) = OiffP(Bm(c:)) -70 as m -7 00. Thus (2) is proved. Using statement (2) for our specific sequence {Xn} yields

P(Bm(c:)) = I - Mlim P[Xn = 0 for all n such that m -too

~ n ~

M].

By the independence of X n ,

and since the product I1~m (I - k- I ) is zero for each mEN we conclude that P(Bm(.s)) = I for all m, that is P(Bm(.s)) does not tend to zero as (2) indicates. Therefore the sequence {Xn} does not converge almost surely.

14.4.

On the Borel-Cantelli lemma and almost sure convergence

Let {Xn, n 2: I} be a sequence ofr.v.s such that for each c:

> 0,

00

(I)

L P[lXnl >.s] <

00.

n=1

According to the Borel-Cantelli lemma, if {An, n 2: I} is an arbitrary sequence of events and E~, P(An) < 00, then PlAn i.o.] = 0 (see also Example 4.2). This lemma and condition (1) immediately imply that Xn ~ 0 as n -7 00. Moreover, the same conclusion, Xn ~ 0 as n -7 00, holds if for any sequence of numbers {c: n } with C:n -!. 0, we have 00

(2)

L P[lXnl > C: n] < n=1

00.

148

COUNTEREXAMPLES IN PROBABILITY

We now want to clarify whether the converse of the statement is true. For this purpose consider the probability space (.0.,1', P) where.o. = [0,1].1' = ~[O,IJ and P is the Lebesgue measure. Define the sequence of r.v.s {Xn, n ~ I} by I

= X n (w) = {O,

Xn

if 0 S; w S; I - nif 1 - n -I < W S; 1.

1,

Obviously Xn ~ 0 as n -t 00. However, for any En > 0 with en P[lXnl > en] = P[Xn = I] = n- I for sufficiently largen. Thus

t

0 we have

00

L P[lXnl > en] =

00.

n=1

Therefore condition (2) is sufficient but not necessary for the almost sure convergence of X n to zero.

14.5.

On the convergence of sequences of random variables in Lr -sense for different values of r

Suppose X and X n , n ~ I are r.v.s in Lr for some fixed r, r > O. Then X, Xn E L S for each s, 0 < s < r. This follows from the well known Lyapunov inequality (see Feller 1971; Shiryaev 1995), (E[IXlsD'/s S; (E[lXlr])l/r, 0 < s < r, or from the elementary inequality Ixls S; 1 + Ixlr, x E }RI, 0 < s < r (used once before in Example 6.5). In other words Lr

L~

Xn ----+ X => Xn ----+ X

for

0 < s < r.

Let us illustrate the fact that in general the converse is not true. Consider the sequence ofr.v.s {Xn,n ~ I} where

P[Xn

= n] = n-(r+s)/2 = I -

P[Xn

= 0],

n ~

I, 0 < s < r.

Then we find E[X~]

=

n(s-r)/2 -t

which implies that Xn ~ 0 as n -t

00.

L'

14.6.

as

n -t 00

However,

E[X~] = n(r-s)/2 -t

and therefore Xn ----+ 0

0

00

as

n -t

00

Lr

=fo Xn ----+ 0 for all r > s.

Sequences of random variables converging in probability but not in Lr -sense

(i) Let {X n , n ~ I} be r. v.s such lhal

I

P[Xn = en] = -, n

I

P[Xn = 0] = I - -, n

n

> I.

149

LIMIT THEOREMS

Then for any c

> 0 we have < c]

P[IXnl

= P[Xn = 0] = 1 -

and hence Xn ~ 0 as n -t

00.

I

eTn ; ; -t

> 0,

as n -t

00

asn-too

-t 1

However, for each r

E[X~] =

and therefore Xn

~n

00

Lr

-f+ 0 as n

-t

00.

(ii) Consider the sequence {Xn, n ~ I} where Xn has a density

(that is, Xn has a Cauchy distribution). Since for any c P[IXnl

< c]

1

2

<:

=

_<:

> 0,

fn(x) dx = -; arctan(nc) -t I

as n -t

00

p

we conclude that Xn --t 0 as n -t 00. But for any r ~ I, E[IXnIT] = 00 and thus the sequence {Xn} cannot converge to zero as n -t 00 in LT -sense for r ~ I. Moreover, since X n, n ~ I, do not belong to the space L Tit is not sensi ble to speak about LT -convergence. (iii) Define the sequences {Yn , n ~ 2} and {Zn, n ~

P[Yn = 1] = P[Zn

= 0] = 1 p

Then, as n -t

14.7.

00,

n- a ,

II logn = P[Zn L

I] as follows:

I - P[Yn

= 0],

= ±n] = 1/(2n a ),

r

Yn --t 0 but Yn -f+ 0 for an)


0

S; 2. L~

p

> 0; Zn --t 0 but Zn -f+ O.

Convergence in LT -sense does not imply almost sure convergence

(i) Consider again the sequence {Xn, n ~ I} defined in Example 14.3(i): namely, Xn = Xn(w) = IAn (w) for n = 2 m + k and An = [k2- m , (k + 1)2-m). Since wEn = [0, I] and P is the Lebesgue measure then E[IXnl] = E[Xn] = 2- m -t 0 as n --7 00. Thus L' Xn --t 0 as n --7 00. a.s.

Nevertheless, as was shown in Example 14.3(i), Xn

-f+ 0 as n

-t

00.

(ii) Let {Xn, n ~ I} be a sequence of independent r. v.s defined by

P[Xn = n l !(2 T )J = lin,

P[Xn = 0] = 1 - lin,

n> I

150

COUNTEREXAMPLES IN PROBABILITY

where r

> 0 is arbitrary. It is easy to see that E[lXnn == E[X~] =

l (n /(2r)rn

I

= n

1/2

~ 0

as n ~

00.

Lr

Therefore for any r > 0, Xn ---; 0 as n ~ 00. Let A4 < N be positive integers. Since Xn are independent, we find N

for M :::; n

prall Xn - 0

II (I -

< N]

lin).

n==M The continuity of P (BN

t Bo => P(BN) t P(Bo)

implies that N

II (1 -

p{n~M(W : Xn(w) :::; e)] == J~oo

lin)

n==M for arbitrary € > 0 and integer M. Separately we can check that for arbitrary M, n~2(1 - I In) 0 and n:'M(1 - lin) == O. Thus p[n~M(W

: Xn(w) :::; e)]

= O.

Since the r.v.s Xn are non-negative this relation means that the sequence {Xn} cannot converge almost surely. (iii) Let {Yn , n 2:: IJ be a sequence of independent r. v.s given by P[Yn == 0] = I - Ilnl/4,

Then it can be shown that Yn

L:> ---;

±I] = 1/(271,1/4),

P[Yn

0 but Yn

a.s.

-f+ 0 as n

n > I.

~ 00.

el

(iv) Let {Sn, n 2:: I} be a symmetric Bernoulli walk. that is Sn = + ... + en where are Li.d. r.v.s each taking the values (+ I) or (-I) with probability!. Define Xn = Xn{w) == I[Sn=Oj(w),n 2:: I. Thenforeveryr > Owe have

ej

lim E[X~] = O.

n-too

Thus Xn ~ 0 as n ~ 00 for r > O. However, the symmetric random walk {Sn} is recurrent in the sense that Sn crosses the level zero for infinitely many values of n (for details see Feller 1968; Chung 1974; Shiryaev 1995). This means that Xn = I a.s.

i.o. and therefore Xn

14.8.

-f+ 0 as n

~ 00.

Almost sure convergence does not necessarily imply convergence in LT -sense

(i) Define the sequence of r. V.s {Xn, n

P[Xn

= 0] =

I

Iln a ,

P[Xn

2:: I} as follows:

= n] = P[Xn

-nJ ==

1/(2n a ),

a> O.

151

LIMIT THEOREMS

Since E[lXnll/2] = l/n cx - 1/ 2 we find that L:~=I E[lXnll/2] < 00 for any Q > ~. According to the Markov inequality we have P[lXnl > E] S E-I/2E[IXnll/2] and hence L:~=I P[lXnl > c] < 00 for every c > O. Using the Borel~Cantelli lemma as a.s.

in Example 14.4 we conclude that Xn

-f-t 0 as n -+

Further, E[lXnI2] = I /n cx - 2 and hence for any Therefore, if Q E [~, 2], then Xn ~ 0 but Xn

00. L'"

S 2, Xn

Q

-f-t 0 as n -+

00.

L'"

-f-t 0 as n -+

00.

(ii) Let {Yn , n ~ I} be a sequence of r.v.s where Yn takes the values en and 0, with probability n- 2 and I - n- 2 respectively. Since for any c > 0, P[lXnl > c] = P[Xn > c] = P[Xn = en] = n- 2 and

n=1

n=1

we conclude as in case (i) above that Xn ~ 0 as n -+

for any r

14.9.

> O. Therefore, as n -+

00.

Obviously,

Lr

00,

Xn ~ 0 but Xn -f-t 0 for all r > O.

Weak convergence of the distribution functions does not imply convergence of the densities

Let F, Fn , n ~ I be d.f.s such that their densities I, In. n ~ I exist. According to the well known Scheffe theorem (see Billingsley 1968), if In(x) -+ I(x) as n -+ 00 for almost all x E JRI then Fn ~ F as n -+ 00. It is natural to ask whether or not the converse is true. The example below shows that in general the answer is negative. Consider the function

0,

Fn(x)

=

x {

if x

<0

(I - sin2nrrx), if 0< x S I 2nrrx if x > 1.

1,

Then Fn is an absolutely continuous dJ. with density

In(x) = { I - cos2nrrx, 0,

if x E [0, I] otherwise.

Also introduce the functions

0,

F(x)= { x, 1,

if x S 0 ifO<xSI if x > 1,

I(x)

= {OI"

if x E (0,1] otherwise.

152

COUNTEREXAMPLES IN PROBABILITY

Obviously F and I are the d. f. and the density corresponding to a uniform distribution on the interval (0, I] respectively. It is easy to see that

Fn(x) ~ F(x)

as

n ~

00

for all x E 1R1.

However,

In(x)

-It

I(x)

as

n ~

00.

Therefore we have established that in general Fn ~ F f;. In ~ I.

14.10.

The convergence Xn ~ X and Y n ~ Y does not always imply that d Xn + Y n -+X + Y

Let X. X n • n ~ 1 and Y. Yn , n ~ 1 be r.v.s defined on the same probability space. Suppose X n ~ X and Yn ~ Y as n ~ 00. Does it follow from this that

Xn + Yn ~ X + Y as n ~ oo? There are cases when the answer to this question is positive, for example if X nand Yn , n ~ I are independent, or if the joint distribution of X n • Yn converges to that of X, Y (see Grimmett andStirzaker 1982). The examples below aim to show that the answer is negative if we drop the independence condition. (0 Let {Xn, n ~ I} be i.i.d. r.v.s such that Xn d

= lorD with probability! each and

d

put Y n = I X n . Then Xn -+ X and Yn -+ Y as n ~ 00 where each of X and Y takes the values I and 0 with equal probabilities. Further. since Xn + Yn = I, it is obvious that Xn + Yn does not tend in distribution to the sum X + Y which is a r.v. with three possible values, 0, I and 2, with probabilities and ~ respectively.

l, !

(ii) Suppose now the sequences of r.v.s {Xn, n ~ I} and {Yn , n ~ I} are such

that Xn ~ X and Yn ~ Y where X "'" N(O, I). Y '" N(O, I). If for each n, Xn and Yn are independent, then Xn + Yn ~ Z with Z ,. . ., N(O, 2). Moreover, in this case the distribution of (Xn' Yn ) converges to the standard two-dimensional nonnal distribution with zero mean and covariance matrix

(~ ~).

Let us now drop the condition that Xn and Y n are independent. Again take {Xn,n ~ I} such that Xn ~X with X '" N(O, I) and let Y n - Xn for all

n E N. Then Yn Y where Y ,..... N(O, I). Obviously the sum Xn + Yn = 2Xn and it converges in distribution to a r. v. Z where Z '" N(O, 4) but not to a r. v. distributed N(0,2) as expected.

14.11.

The convergence in probability Xn ~ X does not always imply that g(X n ) ~ g(X) for any funelion 9

The following result is well known and is used in many probabilistic problems (see Feller 1971; Billingsley 1995; Serfling 1980).

153

LIMIT THEOREMS

If X, X n , n

2':

1 are r.v.s such that Xn ~ X as n --t

°

and g(x), x E lIt I is a

continuous function, then g(X n ) ~ g(X) as n --t 00. By a specific example we show that the continuity of 9 is an essential condition in the sense that it cannot be replaced by measurability only. To see this, consider the function

g(X)={O,

1,

The sequence {X n , n

°

~fx~O > 0.

If x

2': I} can be taken arbitrarily but so as to satisfy the properties p

°

X n > for all n E N and X n ----t as n --t 00. For example, let X n take the values 1 and n- l with probabilities n- 1 and 1 - n- l respectively. Then obviously Xn ~ X where X = a.s. Moreover, for each n we have g(Xn) = 1. However, g(X) = and hence g(X n ) cannot converge in any reasonable sense to g(X). In particular,

°

°

p

g(Xn) -f+ g(X) as n --t

p

despite the fact that Xn ----t X. We come to the same conclusion by considering the function 9 defined above and the sequence of r.v.s {Xn,n 2': I} where Xn '" N(0,a 2 /n), a 2 > 0. Obviously

Xn ~ X as n --t

00

00

with X =

(X ) = 9

However, g(X) =

14.12.

°

a.s. Since Xn is symmetric, we have for each n,

{O, 1,

n

°

with probability with probability

!

~.

p

a.s. and hence g(Xn)

-f+ g(X)

as n --t

00.

Convergence in variation implies convergence in distribution but the converse is not always true

Let X and Y be discrete r. v.s such that

where ak E lIt I, Pk

2': 0,

qk

2': 0,

k = 1,2, ... ,

Lk Pk = 1,

Lk qk = 1.

If F and G are the dJ.s of X and Y respectively, then the distance in variation, v(F, G), is defined by

(I) If X and Y are absolutely continuous r.v.s whose dJ.s and densities are F, G and f, g, then v(F, G) is defined by

(2)

v(F, G) =

i:

If(x) - g(x)1 dx.

154

COUNTEREXAMPLES IN PROBABILITY

Suppose F. Fn. n > 1, are the dJ.s of the r.v.s X, X n . n 2 1, respectively. If v(Fn' F) -+ 0 as n -+ 00 we write Fn ~ F and also Xn ~ X and say that the sequence {Xn} converges in variation to X as n -+ 00. It is easy to see that convergence in variation implies convergence in distribution, that is, Fn ~ F ~

Fn ~ F. However, as we shall now see, the converse is not true. (i) Let Fn be the dJ. of a r.v. Xn concentrated at the point lin. Then Fn ~ Fo as n -+ 00 where Fo is the d.f. of the r.v. Xo == 0, while the quantity v(Fn, Fo) calculated by (1) does not tend to zero as n -+ 00. (ii) Let F (x), x E IR I be a d.f. with density f (x), x E IR I. Our goal is to construct a

i:

w

i:

sequence of dJ.s {Fn, n ~ I} such that Fn ----+ F but Fn

In

=

2

f(x) cos nxdx,

In

=

v

-1-+ F

as n -+

2

f(x)sin nxdx,

00.

Denote

n> 1.

The obvious identity In + I n = J~oo f(x) dx = 1 implies that the numerical sequences {In, n 2 I} and {Jn, n 2 I} cannot simultaneously tend to zero as n -+ 00. Thus, we can assume that e.g. In f+ 0 as n -+ 00. In such a case we introduce the function

f n (x) = C n f (x) (1

+ cos nx) ,

x E

m. I

where c~ I = J~oo f (x) (1 + cos nx) dx. Then for each n the function f n is a density and let Fn be the corresponding dJ. Let us try to find the limit of the sequence {Fn} as n -+ 00. Since f is a density, then the well known Riemann-Lebesgue lemma (see e.g. Kolmogorov and Fomin 1970)

1

f(x) cosnxdx -+ 0 as n -+

00

holds for any Borel set B E 'B I. Hence

1

fn(x) dx -+

1

f(x) dx,

n -+

as

00

~

w

Fn ----+ F

as

n -+

00.

Let us now calculate the distance in variation v(Fn, F). We have

v(Fn, F)

=

J~oo I/n(x) - f(x)1 dx

= J~oo Icnf(x) cos nx - (1 - cn)f(x)1 dx

2 I J~ou Icnf(x) cos nxl dx - J~ou 1(1 - cn)f(x)1 dxl·

i:

We find that

Cn

-+ 1 as n -+

00

i:

and

f(x)1 cos nxl dx 2

2

f(x) cos nxdx = In

f+ o.

Therefore v(Fn' F) f+ 0 as n -+ 00, i.e. the sequence ofd.f.s {Fn} does not converge in variation to F despite the weak convergence established above.

155

LIMIT THEOREMS

14.13.

There is no metric corresponding to almost sure convergence

It is well known that each of the following kinds of convergence: (i) in distribution; (ii) in probability; (iii) in L r -sense, can be metrized (see Ash and Gardner 1975; Dudley 1976). It is therefore natural to ask whether almost sure convergence can be metrized. Let us show that the answer is negative. Let :R denote a set of r.v.S defined on the probability space (.0,::1, P) and d : :R x :R ~ IR+ a metric on :R, that is, d is non-negative, symmetric and satisfies the triangle inequality. Let us check the correctness of the following statement: For X, XI, X 2 , ••• E:R,

d(Xn, X) ~ 0

iff

Xn ~ X.

Suppose such a function d does exist. Let {X n, n ~ I} be a sequence of r. v.S converging to some r. v. X in probability but not almost surely (see Example 14.3). Then for some 6 > 0 the inequality d(Xn' X) ~ 6 will be satisfied for infinitely many n. Let A denote the set of these n. However, since Xn ~ X there exists a subsequence {Xnk,nk E A} of {Xn,n E A} converging to X almost surely. But this would mean that d(Xnk' X) ~ 0 as nk ~ 00, which is impossible because d(Xnk' X) ~ 6 for each nk E A. Thus the statement given above is incorrect and we conclude that a.s. convergence is not metrizable. Note, however, that this type of convergence can be metrized iff the probability space is atomic (see Thomasian 1957; Tomkins 1975a).

14.14.

Complete convergence of sequences of random variables is stronger than almost sure convergence

The sequence ofr.v.s {Xn, n ~ I} is called completely convergentto 0 if 00

(I)

lim ~ P[lXml

n-+oo

L......

> c] = 0 for every c > O.

m=n

In this case the following notation is used: Xn ---=-+ O. In order to compare this mode of convergence with a.s. convergence, recall that

(2) Since the probability P is semi-additive, we obtain

which immediately implies that Xn ---=-+ 0 => Xn ~ O. However, the converse is not always true. To see this, consider the probability space (.0,::1, P) where .0 = [0,1], ::1 = '.B[O,I] and P is the Lebesgue measure. Take the sequence {Xn, n ~ I} where if 0 < - w < 1 n if 1n -< w -< 1.

156

COUNTEREXAMPLES IN PROBABILITY

Then clearly this sequence converges to zero almost surely but not completely. These two kinds of convergence are equivalent if the r.v.s X n , n ~ I, are independent (Hsu and Robbins 1947).

14.15.

The almost sure uniform convergence of a random sequence implies its complete convergence, but the converse is not true

Recall that the sequence of r.v.s {Xn,n ~ I} is said to converge almost surely uniformly to a r.v. X if there exists a set A E 1" with P(A) = such that Xn = Xn(w) converge uniformly (in w) to X on the complement A C • Note that almost sure uniform convergence implies complete convergence discussed in Example 14.14 (see Lukacs 1975). Thus we come to the question of the validity of the converse statement. To find the answer we consider an example. Let the probability space (n, 1", P) be given by n = [0, 1],1" = '.B[O,l] and P is the Lebesgue measure. Consider the sequence { X n, n ~ I} of r. v. s such that

°

°

if ~ w ~ 1/(2n ) if 1/(2n 2) < w ~ 1 - 1/(2n 2) if I - 1/(2n 2 ) < w ~ I. For arhitrary c > 0, 0 ~ Xn

< c iff wE ((1 -

2

c)/(2n2), I - (1 - c)/(2n 2)). Hence

P[IXnl ~ c] = P[Xn ~ c]

= (I

- c)/n 2

so that 00

00

n=l

n=1

This means that the sequence {Xn} converges completely to zero. Now let us introduce the sets Bn = [0, ~n2) U (1 - ~n2, I]. n ~ I. Clearly P(B n ) = 1/(2n 2 ). Suppose for some set A with P(A) = 0, Xn converges to zero almost surely uniformly on AC. Then there exists a number n~ E N (independent of w) such that /Xn/ ~ c < on AC provided n ~ n~. However, we have Bn n AC = 0 and Bn~ C A. Hence P(A) ~ P(Bn~) = !n;2. This contradiction shows that the sequence {Xn} defined above does not converge almost surely uniformly to zero.

!

14.16.

Converging sequences of random variables such that the sequences of the expectations do not converge

If the sequence of r. V.s {X n} converges in probability or a.s. to some r. v. X, then under additional assumptions we can show that the sequence of the expectations {EXrJ will tend to EX. However, in general such a statement is not true without appropriate assumptions.

157

LIMIT THEOREMS

(i) Let {Xn,n

P[Xn

2

I} be r.v.sdefined by

= -n -

l/(n + 4), P[Xn

4]

4/(n + 4),

= n + 4] = 31 (n + 4).

P[Xn Obviously for any c

= -1] = 1 -

> 0 we have P[lXn - (-1)1> c] = 4/{n

p

and hence Xn --+( -I) as n ---t

00.

EX n = 1 + 4/(n

On the other hand,

+ 1)

and

Therefore

lim EXn

n--+oo

+ 4)

1 f:. -1 - E

lim EXn

n--+oo

[p -

= I.

lim Xn]

n--+oo

and the convergence in probability of Xn to X is not sufficient to ensure that EX" ---t EX. This can be explained by referring to the standard result (see Lukacs 1975; Chow and Teicher 1978): if X n• n 2 E[IXnlk] ---t E[lXlk] for each 0 < k ::; r. (ii) Consider the sequence {Yn , n

Yn(W)

2 I} n2, { 0,

] and X

are L r r.v.s and Xn

".

X then

of r. v.s where ifO::;w::;n-1 if n- I <w::; 1

=

0, w E [0,1]. Then for every w E [0,1] we have and also the r.v. Yew} Yn(w) ---t Yew) as n ---t 00. However. EYn = nand EYn 1+ EY = as n ---t 00. Let us note that in case (ii) EYn is unbounded. while in case (i) EX n is bounded. According to Bi11ingsley (1995). if {Zn} is unifonnly bounded and Iim n --+ oo Zn Z on a set of probability I, then EZ = limn--+oo EZn . Both cases (i) and (ii) show that unifonn boundedness is essential.

14.17.

°

Weak L I-convergence of random variables is weaker than both weak convergence and convergence in L I·sense

Recall that the sequence {X n, n 2 I} of r. V.s in the space L I is said to converge weakly in L I to (he r. v. X iff for any bounded r. v. Y we have

(I)

lim E[XnY] = E[XY].

n--+oo

In this case the following notation is used: Xn w,L; X as n ---t 00. Clearly the limit X belongs to U and it is unique up to equivalence (see Neveu 1965; Chung 1974).

158

COUNTEREXAMPLES IN PROBABILITY

It is of general interest to clarify the connection between this mode of convergence and the others discussed in the previous examples. In particular, if Xn w

w,L;

X, does

L'

it follow that Xn ~ X or that Xn ~ X? Remark. Here the notation ~ is used to denote the so-called weak convergence of Xn to X as n ~ 00 which in this case is equivalent to convergence in distribution. In a more general context weak convergence will be considered in Section 16. To answer these questions consider the probability space (0,1', P) where 0 = [0, 'J, ~ = 'B(O,I] and P is the Lebesgue measure, and take the following sequence of r.v.s Xn{W) = sin2rrnw, n 2 1. Note that {Xn} is not convergent in either

°

sense-weak or LI-sense. Nevertheless we shall show that Xn w,L; in the sense of definition (l). Let Y be any bounded r.v., that is Y = Yew), w E [0,1] is an 1'-measurable function. Then there is a sequence of stepwise functions {y(m) (w), m 2 I} such that y(m) ~ y as m --4 00 (see Loeve 1978). By the Egorov theorem (see Kolmogorov and Fomin 1970; Royden 1968) for any E > we can find an open set Ao C [0, 1] such that the convergence y(m) --4 Y as m --4 00 is uniform forw E A~ = [0, 1]\A o . Here we can also use the Lusin theorem (see Kolmogorov and Fomin 1970) on the existence of a continuous function Y* coinciding with Y on the complement of a set of E-measure. In both cases, for stepwise or continuous Y, we have

°

E[ Xn Y'I =

l'

Y' (w) sin 2"nw dw --> 0 as

n -->

00.

Since Yand Y* are bounded and y* is close to Y, the difference IE[XnY*]-E[XnYll can be made arbitrarily small. Hence E[XnY] --4 0 as n --4 00 for any bounded r.v.

Y. Therefore Xn w Xn ~ 0 or Xn 14.18.

0 as n --4 0 is true.

00.

However, as noted above, neither of the relations

A converging sequence of random variables whose Cesaro means do not converge

Let {Xn' n

(I)

w,L; L' ~

2 I}

be a sequence of r. v.s. Then the following implication holds:

n

1

--4 00 ::::}

-(XI n

+ .. , + Xn)

~0

as

n

--4 00.

This follows from the standard theorem in analysis about the Cesaro means. Our aim now is to show that almost sure convergence in (1) cannot be replaced by convergence in probability. Indeed, consider the sequence of independent r.v.s {en, n 2 I} where en has adJ. Fn given by

Fn(x)

= {~'-I/(x+n),

if x < 0 if x > O.

159

LIMIT THEOREMS

Then for every fixed c

> 0 we have P[I{nl > c] = 1 - Fn(c) = 1/(c + n)

which means that {n ~ 0 as n ~ 00. Let us show that the Cesaro means

1 17n:=

n

p

+ ... + {n) -1+ 0

({I

as

n ~ 00.

Denoting Mn = max{{I, ... , {n} and taking into account the independence of the variables {j we can easily show that for any x > 0

P[Mn

::;

x] ==

(1 __ 1 ) (1 __ 1 ) ... (1 __ I ) < (I __ I) x+1 x+2 x+n x+n

n

Therefore

P[Mn/n::; c] <

(2) Since [Mn/n

> c]

C [1Jn

(I _cn+n 1 )n

> c],

P[Mn/n > c] =s; P[1Jn > c]

or

P[Mn/n =s; c] ~ P[1Jn =s; c].

Combining the last relation with (2) we see that

c

P[ 17n ::; J < (I -

(c; n) 1)

n

and hence lim P[1Jn

n-too

> c]

~ 1

lim P[1Jn S c] ~ 1 - exp( -(c

n-too

+ I)

I)

> O.

This means that 17n does not converge to zero in probability. Therefore in general

{n~O~~({I+'"

{n)~Oasn~oo.

Finally, let us indicate one additional

c~se

leading to the same result. Let

{Xn, n 2: I} be independent r.v.s, Xn taking the values 2n and 0, with probabilities p n- I and I-n- I respectively. ThenX n ~Obut ~(XI + .. ·+Xn ) -I+Oasn ~ 00 (the details are left to the reader).

SECTION 15. LAWS OF LARGE NUMBERS Let {X n, n 2: I} be a sequence of r. v.s defined on the probability space EXk , An = ES n = at + ... + an. Define Sn = XI + ... + X n , ak

(.o,~,

P).

COUNTEREXAMPLES IN PROBABILITY

160

We say that the sequence {Xn} satisfies the weak law oflarge numbers (WLLN) (or that {Xn} obeys the WLLN) if ~Sn - ~An ~ 0 as n ---4 E > 0 we have

Further, if lSn - lAn ~ 0 as n ---4 n n p

00,

00,

that is if for any

that is if

[w: n-too lim (~Sn(W) n

!A n ) =

n

0] = 1

we say that the sequence {Xn} satisfies the strong law oflarge numbers (SLLN) (or that {Xn} obeys the SLLN). Let us formulate some of the basic results concerning the WLLN and the SLLN. It is obvious that either {Xn} is a sequence of identically distributed r. v.s or these variables are arbitrarily distributed. ~

Khintchine theorem. Let {Xn, n E[IX1Il where a

<

00.

I} be a sequence of i.i.d. r.v.s with

Then this sequence satisfies the WLLN and ~Sn ~ a as n

-+

00

= EX 1•

Kolmogorov theorem 1. Let {Xn, n ~ I} be a sequence ofi.i.d. r.v.s. The existence of E[lXlll is a necessary and sufficient condition for the sequence {Xn} to satisfy the SLLN and ~Sn ~ a as n -+ 00 where a = EX 1. ~

Markov theorem. Suppose {Xn, n that the following condition holds:

(1)

(l/n2)V[XI

+ ... + Xnl---4 0

I} is an arbitrary sequence of r.v.s such as n

-+

00

(Markov condition).

Then {X n} satisfies the WLLN.

Kolmogorov theorem 2. Let {X n, n ~ I} be a sequence of independent r. v. s with VXn < 00, n ~ 1. Suppose the following condition is fulfilled:

a~ =

00

(2)

L a~/n2 <

00

(Kolmogorov condition).

n=l

Then the given sequence satisfies the SLLN. In the examples below we refer to (1) and (2) as the Markov condition and the Kolmogorov condition respectively. A detailed presentation of the laws of large numbers can be found in the books by Doob (1953), Gnedenko (1962), Fisz (1963), Revesz (1967), Feller (1971), Chung (1974), Petro v (1975), Laha and Rohatgi (1979), Billingsley (1995) and Shiryaev (1995). In this section we consider examples which illustrate the importance of the conditions ensuring the validity of the WLLN or of the SLLN as well as the relationship hetween these two laws and some other related topics.

LIMIT THEOREMS

15.1.

161

The Markov condition is sufficient but not necessary for the weak law of large numbers

(i) Let {Xn' n :2: I} be a sequence of independent r.v.s such that Xn has a X~­ distribution with n degrees offreedom: that is, Xn has a density

fn(x) = { [2r(n/2)]-1 (x/2)(n-2)/2 exp( -x/2), 0,

if x >.0 otherwise.

Then EX n = n, V Xn = 2n and clearly the Markov condition is not satisfied. Hence we cannot apply the Markov theorem to detennine whether the sequence {Xn} satisfies the WLLN. We use the following result (see Feller (1971) or Shiryaev (1995». If {en, n :2: I} is a sequence ofr. v.s and en has a ch.f. ¢n, then

en ~ 0 as

n --+

00

~ ¢n(t) --+ 1 as

n --+

00

for all

tE

jRI.

The ch.f. 'l/Jn of Xn is 'l/Jn(t) = (1 - 2it)-n/2. Then calculating the ch.f. ~n of (Sn - ESn)/n where Sn = XI + ... + Xn and ESn = ~n(n + 1) we find that ~n --+ 1 as n --+ 00 for all t E m,1 and in view of the above result we conclude that the sequence {Xn} does satisfy the WLLN. Note that analogous reasoning leads to the same conclusion if we consider the sequence of discrete independent r. v.s {Yn , n :2: I} where P[Yn = ± I] = ~ (1 - 2- 71 ) and P[Yn = ±2n] = 2-(n+I). Therefore the Markov condition is not necessary for the WLLN.

(ii) Let {Yn , n

:2: I} be independent r.v.s where Yn has a density

It is easy to show that EYn = 0 and VYn = (]'~. Let us choose (]'~ in the following special way: (]'~ = n 1+0 where 0 ~ t5 < I. Then the Markov condition is not fulfilled. Nevertheless, as will be shown, the sequence {Yn } satisfies the WLLN. However, to prove this statement we need the following result (see Feller 1971): let 11n be independent r. v.s and let

(1) for any positive c., 15, k = 1,2, ... , n and all sufficiently large n. Denote by {17n} the truncated sequence with some constant c > 0, that is 17k = 17k if l11k I ~ c and 17k = c if l11k I > c. Then {17k} obeys the WLLN iff the following two conditions hold for any c. > 0 and c > 0:

(2)

lim

n-+oo

~

L- P [.!.. n I17k I > k=1

c.] = 0

162

COUNTEREXAMPLES IN PROBABILITY

and n 2

lim n- ""V7}k =

(3)

n-+oo

~

o.

k=1

Now for any fixed c > 0 we can easily show that P

[~IYkl

>

c] = exp[-v'2cn/k(l+t5)/2],

k

= 1,2, ... , n

and for sufficiently large n the right-hand side can be made arbitrarily small. Thus condition (1) holds. For given c > 0 and constant c > 0 let N be an integer satisfying the relations: eN ~ c and e(N + 1) > c. Choose n > N. Then n

(4) Since the sum on the right-hand side of (4) contains a finite number of tenns and each tenn tends to zero as n -+ 00, (4) implies (2). It then remains for us to check condition (3). A direct calculation shows that VYk = k l+ 6(1

- exp( -cv'2/k(I+6)/2)] - v'2k(l+6)/2 exp( -cv'2/k(1+ 6)/2).

Using a Taylor expansion we find

where tk includes higher-order terms (their exact expressions are not important). From this we can easily derive (3). Therefore according to the Feller theorem cited above the sequence {Yn } satisfies the WLLN. Again, as in case (i), the Markov condition is not satisfied.

15.2. The Kolmogorov condition for arbitrary random variables is sufficient but not necessary for the strong law of large numbers Consider the sequence {Xn' n ~ I} of independent r. v.s where

Obviously EXn :::::: 0, O'~ :::::: VXn :::::: 1 2- n + 2n so that E~=1 0'~/n2 diverges. Thus the Kolmogorov condition, E~1 0'~/n2 < 00 is not satisfied. Nevertheless we shall show that {X n} obeys the SLLN. Recall that two sequences of r. v.s {'n} and {1]n} are said to be equivalent in the sense of Khintchine if E~=) P('n i= 1]n] < 00. According to ReveSl (1967) two such sequences simultaneously satisfy or do not satisfy the SLLN.

163

LIMIT THEOREMS

Introduce the sequence {Yn , n

2 I} where

EYn and P[Xn =/: Yn] = 2- n , n E N. Since the series :E~=I P[Xn =/: Yn] :E~=I 2- n is convergent, the sequences {Xn} and {Yn } are equivalent in the sense of Khintchine. Further, VYn = 1 - 2- n so that :E~,""I VYnln2 < 00. Thus the Kolmogorov condition is satisfied for the sequence

Clearly EX n

{Yn } and therefore {Yn } obeys the SLLN. By the above result it follows that the sequence {Xn} also obeys the SLLN. Thus we have shown that the Kolmogorov condition for arbitrarily distributed r. v.s is not necessary for the validity of the SLLN. 15.3.

A sequence of independent discrete random variables satisfying the weak but not the strong law of large numbers

Let {Xn, n

2: 2} be independent r.v.S such that

P[Xn = ±n] - 1/(2n logn), P[Xn = 0] = 1 - I/(n logn), n = 2,3, .... Consider the events An

{IX n I 2 n}. n

~ 2. Then

The divergence of the series L~=2 P(An). the mutual independence of the variables Xn and the Borel-Cantelli lemma allow us to conclude that the event [An i.o.] has probability 1. In other words,

P[lXnl 2 n i.o.] = 1 ::} P [ lim Snln =/: 0] n-too

= 1.

Therefore the sequence {Xn, n 2} cannot satisfy the SLLN. Now we shall show that {Xn} obeys the WLLN. Obviously VXk = kllogk. Since the function x I log x has a local minimum at x = e and L~=3 k I log k is a lower Riemann sum for the integral (xl logx) dx. we easily obtain that

ft+ '

t

n 1 + (xl logx) dX]

~ VXk ~ ~ [1 22+ r n k=2 n og J3 <

2

logn

+ (n - 2)(n + 1) n 2 10gn

-t 0

as n -t

00.

Thus the Markov condition for {Xn} is satisfied and therefore the sequence {Xn} obeys the WLLN.

164

COUNTEREXAMPLES IN PROBABILITY

Finally, let us indicate another sequence whose properties are close to those of {X n }. Let {Yn , n ~ 2} be a sequence of i.i.d. r.v.s such that

PlY, = ±nJ = C I(n' I logn), n =

2,3, ... , C =

~ (~(n'IOgn)-I) -I

It can be shown that this sequence obeys the WLLN but does not satisfy the SLLN. The easiest way to do this is to use ch.f.s showing, for example, that 'l/Jn(t) ---+ 1 as n ---+ 00 where 'l/Jn is the ch.f. of ~(Y2 + ... + Yn+d.

15.4.

A sequence of independent absolutely continuous random variables satisfying the weak but not the strong law of large numbers

Let {X n , n ~ I} be independent r.v.s where the density of Xn is given by

= (V2a n )-1 exp(-V2lxl/an ), x E JR I . to show that V Xn = a~. Let us define a~ in the following special way: fn(x)

It is easy a~ = 2n2/(logn)2, n ~ 2. First we shall establish that {Xn} does not obey the SLLN. In fact, if An {IXnl ~ n} then

P(An) = 2(V2a n

)-I[OO exp(-V2x/an )dx = exp [-~V2(logn)2/n].

Since (log n)2 /n ---+ 0 as n ---+ 00, E~=2 P(An) = 00. Using similar reasoning to that in Example 15.3, we conclude that {Xn} does not obey the SLLN. Our purpose now is to show that {Xn} satisfies the WLLN. However, one can check that the Markov condition for {Xn} does not hold. Then the proof uses the Feller theorem cited in Example 15.1. Indeed, we can see that (1)

P

[~IXkl

>

c] = exp(-nc1ogk/k),

k = 2,3, ... ,n

and clearly this probability can be made arbitrarily small for large n. For any truncation level c > 0 and c > 0 we introduce the variables Xk Xk, if IXkl ~ c and Xk c, if IXkl > c. Using (1) we obtain

=

=

tp[~IX.[>cl-40

asn-4oo.

k=2

Similarly to Example 15.1 we can verify that n

(1/n

2

)

LVX

k

---+ 0 as

n ---+

00.

k=2

Thus, by the Feller theorem, the sequence {Xn} satisfies the WLLN.

Xk

where

165

LIMIT THEOREMS

15.5.

The Kolmogorov condition L~=l a~/n2 < 00 is the best possible condition for the strong law of large numbers

Let {Xn,n 2 I} be a sequence of independent r.v.s with finite variances a~ and {b n , n 2 I} be a non-decreasing sequence of positive constants with bn ~ 00. We say that the sequence {Xn} obeys the SLLN with respect to ibn} if b;;lSn - b;;IESn ~ 0 as n --t 00 where Sn = Xl + ... + X n . According to the Kolmogorov theorem the condition L~=l a~/b~ < 00 implies that {Xn} satisfies the SLLN with respect to ibn}. Note that in the classical Kolmogorov theorem bn = n, n 2 1. It is of general interest to understand the importance of the condition L~=l a~/b~ < 00 in the SLLN. We shall now show that this condition is the best possible in the following sense. For simplicity we confine ourselves to the case bn = n, n 2 1. So, let {a~} be a sequence of positive numbers with 00

L a~/n2 =

(1)

00.

n=l We aim to construct a sequence {Yn , n 2 I} of independent r.v.s with VYn = a;t such that {Yn } does not satisfy the SLLN. Let us describe the sequence {Yn}. If a~/n2 ::; 1 then the r.v. Y n takes the values (-n), 0 and n with probabilities a~/(2n2), 1 - a~/n2 and a~/(2n2) respectively. If a~/n2 > 1 then Y n = ±an with probability ~ each. Clearly EYn = 0, VYn = a~. For any e > 0 we have P[lYnl/n > e] = P[Yn =1= 0]

2 2 = { a1,n /n ,

f a~/n2 ::; 1 if a~/n2 > 1.

Suppose the sequence {Yn } does obey the SLLN. Then necessarily Yn/n ~ 0 as n --t 00. From (1) it is easy to derive that L~=l P[lYnl > en] = 00. By the BorelCantelli lemma the events [IYn I > en] occur infinitely often, so the convergence Yn/n ~ 0 as n --t 00 is not possible. Therefore {Yn } does not obey the SLLN.

15.6.

More on the strong law of large numbers without the Kolmogorov condition

Consider the sequence {Xn,n 2 2} of independent r.v.s where

(1)

P[Xn

= ±(n/logn)I/2] = ~.

It is easy to check that the Kolmogorov condition L~=2 a~/n2 < 00 is not satisfied. However, Example 15.2 shows that the SLLN can also hold without this condition. In our specific case the most suitable result which can be applied is the

166

COUNTEREXAMPLES IN PROBABILITY

following theorem (see Revesz 1967): let {~n} be independent r.v.s with E~n and let for some T 2: 1

=0

00

E[I~nI2r]

< 00

L E[I~nI2r]/nr+l < 00.

and

n=1

Then the sequence {~n} satisfies the SLLN. Clearly for the sequence {Xn} defined by (1) it is sufficient to take T = 2 and verify directly the conditions in the Revesz theorem. Thus we arrive at the conclusion that {Xn} obeys the SLLN.

15.7.

Two 'near' sequences of random variables such that the strong law of large numbers holds for one of them and does not hold for the other

Considertwosequencesofr.v.s, {Xn,n 2: 2} and {Yn,n 2: 2} where

P[Xn =n/ log n] = P[Xn = -n/ log n] P[Yn =/3n] = P[Yn =-/3n] with 0

< /3 <

= log n/(2n),

P[Xn = 0] = 1 - log n/n,

= 1/(2/32nlogn),P[Yn=O] = 1- 1/(/32nlogn)

1. Obviously Xn and Yn are symmetric r.v.s with

EX n = EYn = 0,

VXn = VYn = n/logn

and both satisfy the inequalities

IXnl < n

IYnl < n

a.s.,

a.s.,

n = 3,4, ....

We are interested to know whether or not these sequences satisfy the SLLN. We shall show that {Xn} obeys the SLLN while {Yn } does not. For this purpose we introduce Hr where 2r + 1 Hr = 2- 2r L VX n . n=2 r

For any choice of c

+l

> 0 we have exp( -c/ Hr) < exp( -cT log 2/2) implying 00

Lexp(-c/Hr) < 00. r=1

However, this condition is sufficient to conclude that the sequence {Xn} obeys the SLLN (see Prohorov 1950). Suppose now that {Yn } also satisfies the SLLN. Then necessarily

P[Yn/n

-4

0] = 1.

167

LIMIT THEOREMS

It can easily be seen from the definition of {Yn } that 00

L P[IYnl/n

{J]

00.

n=2

Then by the Borel-Cantelli lemma, the events [IYnl = n{J] occur infinitely often. This, however, contradicts the above relation, namely that Yn/n ~ 0 as n ---t 00, and therefore the sequence {Yn } does not obey the SLLN.

15.8. The law of large numbers does not hold if almost sure convergence is replaced by complete convergence Let {Xn, n ~ I} be a sequence of i.i.d. r.v.s, F(x), x E }R' their common d.f. and EX, = J~oo xdF(x) = O. Suppose that {Xn} satisfies the SLLN. Then

(1)

Y n :=

!n (Xl + ... + Xn) ~ 0

as

n ---t

00.

It is natural to ask whether the conditions for the SLLN could guarantee that in (1) almost sure convergence can be replaced by a stronger kind of convergence, in particular by complete convergence (see Example 14.l4). Under the conditions

i:

(2)

x dF(x) = 0,

(J'2 -

i:

x 2 dF(x) < 00

Hsu and Robbins (1947) have shown the convergence of the series L~=' P[lYnl > c] for any c > O. Therefore if condition (2) is satisfied, the sequence {Yn } converges completely. Thus instead of (1) we have Yn ~ 0 as n ---t 00. Suppose now that condition (2) is relaxed a little as follows:

(3)

i:

x dF(x) = 0,

i:

IxlQ dF(x)

< 00,

i:

x 2 dF(x) =

00

where Q' = constant and! (1 + VS) ~ Q' < 2. Then the sequence {Xn} satisfies the SLLN. However, the series L~=l P[lYnl > c] diverges for every c > 0 and hence the relation Yn ~ 0 fails to hold. Therefore there are sequences of Li.d. r. v.s such that the corresponding arithmetic means {}~} converge almost surely but not completely. Finally, it remains for us to indicate a particular case when conditions (3) are satisfied. For example, take X I to be absolutely continuous with density f (x) = Ixl- 3 for Ixl :: 1 and f(x) = 0 otherwise.

15.9.

The uniform boundedness of the first moments of a tight sequence of random variables is not sufficient for the strong law of large numbers

Recall firstly that the sequence {Xnl n ~ I} of real-valued r. v.s is said to be tight if for each c > 0 there exists a compact interval K£ C }R I such thatP[Xn EKe] > l-c

168

COUNTEREXAMPLES IN PROBABILITY

for all n. Let {X n, n ~ I} be a sequence of independent r. v.s. According to a result derived by Taylor and Wei (1979), if {Xn} is a tight sequence and the rth moments with r > I are uniformly bounded (E[lXnn < M = constant < 00) then {Xn} satisfies the SLLN. Is it then possible to weaken the assumption for r, r > 1, replacing it by r = 1 in the above result? By a specific example we show that the answer to this question is negative. Let {X n, n ~ I} be a sequence of independent r. v.s such that

P[Xn

= ±n] = Hn log(n + 2)]-1,

P[Xn

= 0] = 1 -

[n log(n + 2)]-1.

Then EXn = 0, E[IXnl1 = I/Iog(n + 2). So E[lXnlJ are uniformly bounded. and indeed, E[lXnlJ ~ O. Taking into account the relation P[lXnl ~ n] = I /[n log(n + 2)], we conclude that the sequence {Xn} is tight. Further, L~=I P[lXnl ~ n] = 00 and the Borel-Cantelli lemma implies that the event [lXnl ~ n i.o.] has probability I. However, this means that the SLLN cannot be valid for the sequence {Xn}.

15.10.

The arithmetic means of a random sequence can converge in probability even if the strong law of large numbers fails to hold

Let {Xn, n ~ I} be a sequence of i.i.d. r.v.s such that E[lXd] = 00. According to the Kolmogorov theorem this sequence does not satisfy the SLLN, i.e. Yn = Sn/n. where Sn = XI + ... + Xn, is not a.s. convergent as n ~ 00. However we can still ask about the convergence of Yn , the arithmetic means, in a weaker sense, e.g. in probability. This possibility is considered in the next example. Consider the sequence {~n, n ~ I} of i.i.d. r.v.s where P[6 = (-I )k-l k] = 6/(1r 2k 2), k = 1,2, ... The divergence of the harmonic series implies that E[l61J = 00. Hence {~n} does not satisfy the SLLN. Let us show now that the arithmetic means (6 + ... +~n) /n converge in probability to a fixed number as n ~ 00. Our reasoning is based on the following general and very useful result (see e.g. Feller 1971 or Shiryaev 1995): if the ch.f.1/J( t) = E[eit~l), t E ~I of 6 is differentiable at t = 0 and 1/J'(0) = ie, where i = yCT and e E ~ , p

then (~l + ... + ~n)/n ---t e as n ~ r. v. ~I defined above. We have 6

1/J(t)

= 7f2

f; 00

00.

Thus we first have to find the ch.f.1/J of the

(e it )2j-1 (2j - 1)2

6

+

7f2

00

~

(e it )2k (2k)2 .

If we introduce the functions 00

and

h2(U) =

L k=l

2k

(~k)2'

lui

~

I

169

LIMIT THEOREMS

we easily find that they both are differentiable and , I I +u h](u) = -2 log - - , u 1- u

Hence h~ (u) - h~ (u)

= (1/ u) In( I + u) which implies that 1jJ' (0) exists and 6 [h]'( 1) - h2' (I)] = i 6 log 2 1jJ '( 0) = i 2: 2' 1T

1T

Thus we arrive at the final conclusion that I

p

+ ... + ~n) ----+

-(6 n

6 log 2 1T2

as

n ---+

00.

Note that in this case the sequence {~n} satisfies the so-called generalized law of large numbers (see Example 15.12).

15.11.

The weighted averages of a sequence of random variables can converge even if the law of large numbers does not hold

Let {Xk, k ~ I} be a sequence of non-degenerate i.i.d. r. V.s, {Ck, k ~ I} a sequence ofpositivenumbersandletSn = E~=] CkXk andCn = E~=] ck,n ~ 1. The ratios Sn/Cn,n ~ 1 are called weighted averages generated by {Xk,Ck,k ~ I}. We say that the weak (strong) law holds for the weighted averages of {Xk, Ck, k ~ I} iff Sn/Cn converges in probability (a.s.) to some constant as n ---+ 00. Without any loss of generality we can suppose that EX k = 0 for all k ~ 1. We now want to see whether Sn/Cn converges to 0 as n ---+ 00. Obviously if all Ck I then Sn = Xl + .. , + X n , Cn = n and we are in the framework of the c1assicallaws of large numbers. Our aim now is to show that there is a sequence of i.i.d. r.v.s {Xd and a sequence of weights {c n } such that

=

I

-(Xl

n

a.s.

+ ... + Xn) +0

as

n ---+

00.

In other words, the strong law holds for weighted averages but the classical SLLN is not valid. An analogous conclusion can be drawn about the weak law for weighted averages and the classical WLLN. Firstly, consider the strong law. By assumption the variables Xn are identically distributed and in this case the SLLN does not hold iff E[lXll] = 00. Further, we need the following result (see Wright et at): let g(x), x E IR+ be a non-negative measurable function with g(x) ---+ 00 as x -t 00. Then there exists a sequence { X k, Ck, k ~ I} whose weighted averages Sn / C n , n ~ 1, satisfy the strong law and

E[g(Xt)]

= E[g(X))]

=

00.

Actually this result contains all that we wanted, namely a sequence of i.i.d. r. v.s {Xk, k ~ I} with EIX]I = 00 and a sequence of weights {Ck' k ~ I} such that

170

COUNTEREXAMPLES IN PROBABILITY

Sn/Cn ~ 0 as n -+ 00, although the sequence {Xd does not obey the classical SLLN. A similar conclusion can be obtained by using a result of Chow and Teicher (1971) which states that there is a r. v. X with EI X I = 00 such that the sequence {Xd of independent copies of X together with a suitable sequence of weights { Ck} generates weighted averages Sn / Cn which converge a.s. as n -+ 00. Obviously it is impossible 1 since the classical SLLN for {Xd is not satisfied. In this in this case to take Ck connection Chow and Teicher (1971) give two specific examples. The first one arises in the so-called St Petersburg game (Xk = 2k with probability 2- k, and 0 otherwise), while in the second case X has a Cauchy distribution. It is of general interest to compare some consequences of the results cited above. In particular, let us look at the value ofE[IX for different r. Both examples considered by Chow and Teicher are such that

=

n

lim xrp[lXI ~ x] = 0

x-too

0

for all


1

which implies that E[lXn < 00 for all 0 < r < 1. In the result of Wright et af (1977) we can take the function g(x) = (logx)+ and choose a sequence {Xd of i.i.d. r. v.s such that E[IX = 00 for all r > 0 and find weights {cd such that Sn/Cn ~ constant. Clearly the SLLN fails to hold. Consider now the weak law. It is easy to see that if the weak law holds for the weighted averages of {X k, Ck} then {Ck} must satisfy the condition

n

Cn -+

(I)

00,

cn/Cn -+ 0

as

n -+

00.

According to Jamison et af (1965), the weak law holds for any sequence of weights {cd satisfying (1) if Jixl~T x dF(x) -+ a = constant as T -+ 00 and lim TP[lXI ~ T] = 0

(2)

T-too

where F is the d.f of X. This result and a statement by Loeve (1978) allow us to conclude that if X has a fixed distribution (we consider only the case of i.i.d.) then the weak law holds for {X k, cd for any {cd iff {Xd satisfies the classical WLLN (when all Ck 1). However, using the result of Wright et al with g(x) = xr and 0 < r < 1, one can obtain a sequence {Xk, cd for which the weak law holds but condition (2) is not satisfied. In such a case the weak law does not hold for the sequence {X k, l}. Obviously this means that the sequence {Xk} does not obey the classical WLLN in spite of the fact that for some weights {cd. the weighted averages Sn/Cn converge in probability.

=

15.12.

The law of large numbers with a special choice of norming constants

Let {Xn, n ~ I} be a sequence of independent r.v.s and Sn = XI + ... + X n . If for some number sequences { an, n ~ I} and {b n , n ~ I}, with all bn > 0, the following

171

LIMIT THEOREMS

relation holds: (1) and we say that {Xn} satisfies a generalized law of large numbers (LLN). This law is weak or strong depending on the type of convergence in (1). If an = ESn and bn = n we obtain the scheme of the classical LLN. There are sequences of r.v.s for which the classical LLN does not hold, but for some choice of {an} and {b n } the generalized LLN holds. Let us consider an example. In the well known St Petersburg game (also mentioned in Example 15.11), a player wins 2k roubles if heads first appears at the kth toss of a symmetric coin, k = 1,2, .... Thus we get a sequence of independent r. v.s {X k, k 2: I} where P[Xk = 2k] = 2- k = 1 - P[Xk = 0]. It is easy to check that {Xd does not obey the WLLN. However, we can hope that a relation like (]) will hold. Using game tenninology, suppose that a player pays variable entrance fees with a cumulative fee bn = n logn for the first n games. Then the game becomes 'fair' in the sense that

(2)

lim Sn/b n

n-too

=1

in probability.

It is natural to ask whether this game is 'fair' in a strong sense, that is, whether (2) is satisfied with probability 1. Actually we shall show that the St Petersburg game with bn = n log n is 'fair' in a weak but not in a strong sense. In other words, it will be shown that {Xd obeys the weak but not the strong generalized LLN with an = bn = n log n, n 2: 2. The result that Sn/bn ~ 1 as n -+ 00 is left to the reader as a useful exercise. Further, it is easy to see that P[Xn > c] 2: 1/ c for any c > 1 and every n 2: 2. Hence for c = constant> 1 and n 2: 2 we have 00

P[Xn > cbn ] 2: l/{cb n ) = 1/{cn logn)

and

L P[Xn > cbn] =

00.

n=2

This and the Borel-Cantelli lemma imply that P[Xn/b n > c i.o.] = 1. Thus

P[limXn/b n = 00] = 1 and P[limSn/b n = 00]

= 1.

Therefore }:I[ lim Sn/bn = 1] = 0 n-too

showing that (2) is satisfied for convergence in probability but not a.s.

SECTION 16.

WEAK CONVERGENCE OF PROBABILITY MEASURES AND DISTRIBUTIONS

In Section 14 we introduced the notion of convergence in distribution and illustrated it by examples. In particular, we mentioned that this kind of convergence is close to

172

COUNTEREXAMPLES IN PROBABILITY

so-called weak convergence. In this section we define weak convergence and clarify its relationship with other kinds of convergence. Let F n , n 2: 1, and F be d.f.s over the real line ]Rl. Denote by P n and P the probability measures over (]R 1,2) I) generated by Fn and F respectively. Recall that P n and Pare detennined uniquely by the relations Pn(-oo,xl = Fn(x) and P(-oo,xl = F(x), x E ]RI. Since F is continuous at the point x iff P({x}) = 0, then convergence in distribution Fn ~ F means that P n ( -00, xl --+ P( -00, xl for O. Let us consider a more general situation. every x such that P( {x}) For any Borel set A in ]Rl (that is A E 2)'), 8A will denote the boundary of A. Suppose P and P n , n 2: I, are probability measures on (]R' ,2)'). We say that the sequence {Pn} converges weakly to P and write P n ~ P, if for any A E 2) I with P(8A) = 0 we have Pn(A) --+ P(A) as n -+ 00.

=

Now we formulate the following fundamental result.

Theorem 1. The following statements are equivalent: (a) Pn ~P;

(b) lim P n (A) ~ P( A) for any closed set A E 2) t ; n400

(c) lim Pn(A)

2: P(A) for any open set A

E 2)';

n400

(d) For every continuous and boundedfunction 9 on]R' we have

( g(x)Pn(dx) -+ ( g(x)P(dx)

JRi

JRi

as

n -+

00.

Weak convergence can be studied in much more general situations not just for probability measures defined on the real line ]R' . However, convergence in distribution treated in Section 14 is equivalent to weak convergence discussed above. If we work with probability measures, the tenn weak convergence is preferable, while for d.f.s both tenns, weak convergence and convergence in distribution, are used, as well as both notations, Fn ~ F and Fn ~ F. We now fonnulate another fundamental result connecting the weak convergence of d.f.s with the pointwise convergence of the corresponding ch.f.s.

Theorem 2. (Continuity theorem.) Let {Fn' n 2: I} be a sequence of d.fs on ]Rl and {¢n, n 2 I} be the corresponding sequence of the chfs. (a) If Fn ~ F where F is a d.!, then ¢n(t) --+ ¢(t), t E ]Rt where ¢ is the ch.f of F. (b) Iflimn---lo oo ¢n(t) existsforeacht E]Rl and¢(t):= limn---lo{X) ¢n(t) is continuous at t = 0, then ¢ is the ch.! of a d.f F and Fn ~ F as n --+ 00. We refer the reader to the books by Billingsley (1968, 1995), Chung (1974) or Shiryaev (1995) for a detailed proof of Theorems 1 and 2 and of several others. In this section we have included examples illustrating some aspects of the weak convergence of probability measures, distributions, and densities.

173

LIMIT THEOREMS

16.1.

Defining classes and classes defining convergence

Let (n,9) be a measurable space and P, Q probabilities on this space. The class of events A C l' is said to be a defining class, if P= Q

A

on

=}

P= Q

on

1'.

We say that A C l' is a class defining convergence if

=}

Pn(A) --t P{A)

for all sets

A EA

Pn(A) --t P{A)

for aU sets

A E l' with

P(8A):::: 0

with

P{8A)

=0

that is, that P n ~ P as n --t 00. Let us illustrate the relationship between these two notions. (i) Obviously every class defining convergence is a defining class. However, the converse is not always true. Let n :::: [0, I), l' :::: 'B[0,1) and A C l' be the field of all finite sums of disjoint subintervals of the type [a, b) where 0 < a < b < 1. Then A is a defining class but not a class defining convergence. To see this it is enough to consider the probabilities P n and P concentrated at the points I - lin and 0 respectively.

n,

(ii) Let {P n 2:: I}, P and Q be probabilities on (n, 9) where n ~ I , l' :::: 'B I and let A C l' be a defining class. Suppose two conditions are satisfied:

(1)

Pn(A) --t Q(A)

as

n --t

00

for all

as

n --t

00.

A EA

and

(2)

Pn

w

~

P

Since A is a defining class, from (1) and (2) we could expect that P this is not the case. Define P n , P and Q as follows:

Q. However,

It is easy to see that P n ~ P as n --t 00. Further, let B consist of the points 0, 1, ~, 1 + "* where n 2:: 1. Denote by A the field containing all A E l' such that either AB is finite and 0 ¢ A, or A C B is finite and 0 ¢ A c. Then A is a defining c1ass and Pn(A) --t Q(A) as n --t 00 for every A EA. So (1) and (2) are satisfied, but P 1= Q. (iii) Let «:::[0, 1] be the space of all continuous functions on [0, 1] and a-field. For kEN and tJ, ... , tk E [0,1] let 1rt] ... t,.

: «:::[0, 1J t----t Rk

e its

Borel

174

COUNTEREXAMPLES IN PROBABILITY

map the point (function) x E
°

nt, xn(t) =

{

2 - nt, 0,

°

if if if

0< - t -< !n !n 1. n

< t -<
1. n

1.

Since Xn does not converge to uniformly in
°

=

!)

°

=

=

tmin

= min{tj, tj

°

=

=

=1=

o}.

This example shows that weak convergence in the space e[O, I] cannot be characterized by convergence for all finite-dimensional sets (as in ]ROO).

16.2.

In the case of convergence in distribution, do the corresponding probability measures converge for all Borel sets?

Let Fo(x), Fn(x), n

>

1, be dJ.s and /-Lo, /-Ln, n 2: I, their probability measures on

(]R I, 'B I). Suppose Fn ~ Fo as n -+ 0. It follows that /-Ln((-OO, x]) -+ /-Lo((-oo, x]) for every x E ]R I which is a continuity point of Fo. However this is a convergence of /-Ln to /-Lo but for a special kind of sets, namely for infinite intervals which of course belong to 'B I. Thus we arrive at the following question. Is it true that Fn ~ Fo imply /-Ln (B) -+ /-Lo( B) for all B E 'B I? In fact, the negative answer to this question is contained in the definition of convergence in distribution. Perhaps the easiest illustration is to take F n (x) 1[l/n,oo)(x),n 2: 1, and Fo(x) = I[o,oo)(x),x E ]RI.ThenobviouslyFn(x) -+ Fo(x) as n -+ 00 for all x except the only point x = where Fo has a jump (of size 1).

°

Thus in this completely degenerate case we obviously have Fn ~ Fo as n -+ Taking, for example, the Borel set (-00,0], we find

ILn(( -00,0]) = Fn(O) =

°It /-Lo(( -00,0]) = Fo(O)

= 1 as n -+ 00.

00.

175

LIMIT THEOREMS

In the above case the limiting function Fo is discontinuous. Let us assume now that Fo is continuous everywhere on lR', Of course, if Fn ~ Fo and B is a Borel 0, then f..tn(B) -+ f..to(B) as n -+ 00. Let us i1lustrate what we set with f..to(aB) expect if f..to( 8B) i- O. Consider the r.v.s Xo and X 1" n > 1, where Xo is uniformly distributed on the interval (0,1) and Xn is defined by l'[Xn = ~] = for k = 0, ], ... , n - 1, n 2:: 1 (uniform discrete distribution). If Fo and Fn are the dJ.s of Xo and Xn respectively, we have (hy [a] standing for the integer part of a)

=

*

Fo(x) = {

1,

Since l[nx]/n -

~

if x if 0 if x

0, x,

xl

< >

0 x

0, ~

I

Fn(x) =

I,

{

[nx]/n, I,

~ lin for any x E lR' and any n

if x if 0 if x

~

0

<x~ > 1.

2:: 1 we

I

conclude that

Xo as n --+ 00 (equivalently, that Fn -..::., Fo). Denote by Po and Pn the measures on lRl induced by Fo and Fn and let Q be the set of all rational numbers in RI, Then Pn(Q) = I for each n, Po(Q) 0 and hence Xn

lim Pn(Q) = 1 i- 0 = Po(Q).

n-+oo

In this example Po(8Q) = 1, that is the crucial condition Po(8B) = 0 is not satisfied Q. for B Note that the limiting function Fo is not only continuous, it is absolutely continuous with a finite support (uniform distribution on (0,1). A conclusion similar to the above concerning the eventual convergence of Pn to Po can also be derived for absolutely continuous Fo having the whole real line RI as its support. Consider a sequence of independent Bernoulli r.v.s 6,6, ... : P[ei 1] == p, P[ei = 0] = q, q 1 - p, 0< p < 1. Denote by G n the dJ. of the quantity Sn (Sn - np) I (npq) 1/2 where

=

Sn

el + ' .. + en and let 0 be a r.v. distributed normally N(O, I). Then Sn ~ 0,

or equivalently, Gn -..::., q, as n -+ 00 (q, is the standard normal d.f.). If Po and Pn are the measures on lR I induced by q, and G n and the Borel set B is defined by B U~=o{(k - np)/(npq)I/2}, then obviously

Pn(B)

= P[Sn E b] = I fi Po(B) = P[O E B] = 0 as n -+ 00.

Once again this is due to the fact that the condition Po (8B) = 0 is not satisfied.

16.3.

Weak convergence of probability measures need not be uniform

Let Fo(x), Fn(x), x E lRl, n ~ 1 be d.f.s and f..to. f..tn. n ;:::: 1, their corresponding probability measures on (lR I , ~ I). Let us suppose that

(1)

lim f..tn(B) = f..to(B)

n-+oo

for all B E ~I.

176

COUNTEREXAMPLES IN PROBABILITY

It is natura1 to ask if (1) holds uniformly in B. The example below shows that in general the answer is negative even for absolutely continuous dJ.s. Indeed, if Fo. Fn, n ~ 1, have densities 10, In, n ~ 1, respectively, then ( I ) can be written in the form

B E 13 1•

n-'lim t<XJ1[BIn(x) dx = 1B[ lo(x) dx,

(2)

Consider now the following functions:

f ( ) - { 1 + sin (21l'nx) , n x 0,

if x E [0,1] if x ¢ [0, 1].

lo(x)

=

{I,

0,

if x E [0,1] ifx¢[O,I].

It is easy to see that 10 and In for each n ~ 1 are density functions. Clearly 10 is a uniform density on [0,] 1. If Fo and Fn. n ~ ], are the dJ.s of 10 and In, n ~ 1,

then Fn ~ Fo as n --t 00. Moreover, applying the Riemann-Lebesgue theorem (see Rudin 1966; Royden 1968), we conclude that relation (2), and hence (]), is satisfied 1 for this choice of 10. In. n ~ 1, and for all B E 13 • Consider now the sets Bn = {x E [0,1] : In(x) ~ I}. n ~ I. Then

[

1Bn

lo(x) dx =

~,

1

2"

1

+:;,

n

= 1,2, ....

Therefore in general the convergence in (1) and (2) can be non-uniform.

16.4.

Two cases when the continuity theorem is not valid

Let Fo, Fn , n ~ l, be d.f.s with ch.f.s ¢o,
Fn ~ Fo

-¢=:::>


--t

~

1, respectively. The continuity

¢o(t) where
Let us show that the continuity of
°is essential.

(i) Consider the sequence of r.v.s {Xn, n ~ I} where Xn "" N(O, n). Then the ch.f.

¢nofXnisgivenbY¢n(t) as n --t 00 where

exp(-!nt2),t E IR 1.Obviouslywehave¢n(t) --t ¢(t)

¢(t)

= {O,

1,

~f t :I If t

0

= 0.

Thus the limiting function ¢> is discontinuous at 0 and hence the continuity theorem does not hold. On the other hand, we have

Clearly limn-'t<XJ Fn{x) = F(x) exists for all x E IRI but F(x) =: ~ is not a d.f.

177

LIMIT THEOREMS

(ii) Consider the family of functions {Fn, n ~ I} where

0, Fn(x) = (n + x)/(2n), { I,

if x < -n if -n :S x :S n if x ~ n.

Then for each n, Fn is a dJ. and obviously for all x E IR I we have limn-H)o Fn (x) = &. Thus the sequence {Fn} is convergent but its limit, the constant &' is not a d.f. A simple explanation of this fact can be given if we consider the ch.f. ¢n of Fn. Since ¢n(t) = (sinnt)/(nt) then

Again, as in case (i), the limiting function continuity theorem cannot be applied. 16.5.

¢ is discontinuous at 0 and therefore the

Weak convergence and Levy metric

For given two dJ.s F(x), X E 1R1 and G(x), x E IRI the following quantity

L(F,G)

= inf{c > 0:

F(x - c) - c:S G(x) ~ F(x +c) +c,x E 1R1}

is called a Levy metric (distance) between F and G. Note that L(·, .) is a metric in the space of all d.f.s and plays an essential role in probability theory; e.g. the following result is frequently used. Let F and F n , n ~ I be d.f.s. Then, as n --+ 00

Consider now the sequence {Xn, n ~ I} of independent r.v.s. Denote Sn XI + '.' + X n , s~ = VS n and let Fn be the d.f. of Sn/sn. Suppose the variables Xn are such that Fn ~ G as n --+ 00, where G is a d.f. (Actually, G belongs to the class of infinitely divisible distributions.) This is equivalent to saying that for any c > 0 there is an index n E such that for all n ~ n E we have L(Fn, G) < E. Since the quantity L(Fn, G) is 'small', we can suggest that another related quantity, L( Fn , On), is also 'small'. Here Fn is the d.f. of Sn (without normalization!) and Gn(x) = G(xs n ). In several cases such a statement is true, but not always, as in the next example. Let X nj, j = I, ... , n, n ~ I be independent r. v.s where P[Xnj

If Sn as n --+

= X nl 00.

= ±nVs] =

I

2n'

j

= I, ... ,no

+ ... + X nn , then ESn = 0 and s~ = VS n = 5n 2 + n - I --+ 00 For the normalized variable 7Jn = Sn/(nVs) we have E77n = 0 and

178

COUNTEREXAMPLES IN PROBABILITY

= 1 + (n

- 1)/{Sn2 ) implying that VrJn -+ 1 as n -+ 00. Let us find the limit of the d.f. Fn{x) = P[rJn ~ x] as n -+ 00. In this case the best way is to find the ch.f. 'ljJn{t) == E[e it '7n]. By using the structure of the variables X nj and the properties of VrJn

ch.f.s we find that lim t/Jn{t) = t/J{t) = exp(cos t - 1), t E ]RI. n-too

=

exp( cos t - 1) is a ch.f. corresponding to a concrete r. v., say TJO and However t/J{t) 110 = t.1 - 6 with 6 and 6 independent r.v.S each having a Poisson distribution with parameter ~. Hence by the continuity theorem, we have

F n ~ G as n -+

00

with G(x) = P[110 ~ x], x E ]RI, or equivalently limn-too L(Fn, G) = O. Thus the quantity L{Fn , G) is 'small' and we want to see if L(Fn,G n ) is also 'small'. Recall that Fn is the d.f. of Sn itself, while Gn{x) = G(xsn). Note first that Fn and en correspond to discrete r. v.s. Specifically, the values of Sn are in the set {±j, ±kvls ± l : j, k, l = 1, ... , n} and the d.t'. Fn has jumps at all points of this set. Further, en{x) = P[Cn ~ x], where Cn = rJo·nvls and it is obvious thatJn takes I,2, ... } at each point of which G n has a its values in the set {O, ±k·n..J5 : k jump. In particular, P[110 = 0] > 0 which implies that for an odd index n we can find 0]) such that L{Fn, en) 2: c. Hence we a number c > 0 (expressed through P[TJO conclude that in this case the quantity L{Fn, en) is not small.

=

=

16.6.

A sequence of probability density functions can converge in the mean of order 1 without being converging everywhere

Let fo(x),fl{X),h(x), ... , x E ]RI be probability density functions. Here we consider two kinds of convergence of f n to fo: convergence almost everywhere and convergence in the mean of order 1 which are expressed respectively by lim fn{x) = fo{x) almost everywhere n-too

(1) and

lim n-too

(2)

roo Ifn(x) J-oo

fo(x)1 dx

= O.

Let us compare (l) and (2). According to a result by Robbins (1948), (l)=} (2). However, the converse is not always true. Indeed, let

fn{x) = {n/(n - 1), 0,

if (k ~ I)/n otherwIse

< x < kin -

I/n

2

,

k = I,2, ... ,n

179

LIMIT THEOREMS

and let 10 be the unifonn density on the interval (0,1). It is easy to see that for every n, In is a density and if Bn = {x E (0,1) : In(x) > O}, then

_ { I/(n - I), I, Iln(x) - 10(x)1 -

if x E Bn if x E B~

n (0, I).

Since the sets Bn and B~(O, I) have Lebesgue measures (n - 1)ln and lin respectively, we obtain the relation

1 1 o

2 n

Iln(x) - 10(x)1 dx = - => lim

n~oo

11 0

Iln(x) - lo(x) Idx

=

°

that is, In converges to 10 in the mean of order I. It now remains to show that

In(x)

It

lo(x)

= I,

x

E

(0, I).

For any fixed irrational number z there exist infinitely many rational numbers m I k such that mlk - IIk 2 < z < mlk. This fact and the definition of In imply that In(x) = for infinitely many n and for any fixed irrational x E (0,1). Furthennore. if x is a rational number in (0,1), then x = m I k for some positive integers m and k with m < k, and moreover

°

In(x) =0 for n=lk, l= 1,2, .... Thus for any x E (0,1) the densities In (x) cannot converge to lo(x)

16.7.

== 1.

A version of the continuity theorem for distribution functions which does not hold for some densities

Let Xn be a r.v. with d.f. Fn , density In and ch.f. 4>", n ~ 1. The continuity theorem provides necessary and sufficient conditions for the weak convergence of {Fn} in tenns of {4>n}. Now we want to find conditions which relate the ch.f.s {4>n} and the densities {In}. For some r.v. Xo with d.f. Fo, density 10 and ch.f. 4>0 we introduce the following three conditions: lim In(x)

(1)

n~oo

for almost all x E ]RI,

Fn ~ Fo as n

(2) (3)

= lo(x)

-t 00,

lim 4>n(t) = 4>o(t) for all t E ]RI and 4>0 is continuous at 0. n~oo

By the continuity theorem we have (2) {:::::::> (3). According to the Scheffe theorem (see Example 14.9), (I )=>(2). Example 14.9 also shows that in general (2)~(l). Thus

180

COUNTEREXAMPLES IN PROBABILITY

we conclude that (l)::::}(3) and can expect that in general (3)~(l). Let us illustrate by an example that indeed (3)~( 1). Consider the standard normal density tp(x) == (27r)-1/2 exp( _~x2) and its ch.f. ¢o( t) = exp( -

i>.(x) == tp(x)(l

(4)

(5)

!t2). Define the functions

cosAx)/(1 - ¢o(A)),

x E lRl,

W>,(t) = [2¢o(t) - ¢lo(t + A) - ¢lo(t - A)]/[2(1 - ¢O(A))],

tE

jRl

where A is any real number (e.g. take A == n). It is not difficult to check that for each A, 1>.. (x), x E lR I is a probability density function, ¢ >, (t), t E lR I is a ch.f., and moreover, ¢ >.. corresponds to 1>,. Further, we find

(6) where the limiting function ¢o is continuous at 0 and thus (3) is satisfied. However,

(7) and hence condition (1) does not hold. Comparing (6) and (7) we see that in general the pointwise convergence of the ch.f.s ¢n given by (3) is not enough to ensure the convergence (1) of the densities In. At this point the following result may be useful (see Feller 1971). Let ¢n and ¢ be absolutely integrable ch.f.s such that (8) Then the corresponding dJ.s Fn and F have bounded continuous densities In and 1 respectively, and (8) implies that lim In(x) == l(x) uniformly in x, x E lRl.

(9)

n-\o(X)

Obviously. in the above specific example. condition (9) is not satisfied (see (7». It is easy to see that the pointwise convergence given by (6) does not imply the integral convergence (8).

16.8.

Weak convergence of distribution functions does not imply convergence of the moments

Let F and Fn , n ~ 1 be dJ.s. Denote by Ok and oin) their kth moments: Ok

==

i:

xk dF(x),

o~n) =

i:

Xk

dFn(x),

k = 1,2, ....

lSI

LIMIT THEOREMS

According to the Frechet-Shohat theorem (see Feller 1971), if Q~n) --+ Qk as n --+ 00 for all k and the moment sequence {Qk} determines F uniquely, then w

(1)

Fn --+ F as n --+

00.

(For such results also see the works of Kendall and Rao 1950, Lukacs 1970.) Now let us answer the converse question: does the weak convergence (1) imply convergence of the moments Q~n) to Qk? By two examples we show that (1) can hold even if Q~n)

It Qk as n

--+

00

for any k.

(i) Consider the family of d.f.s {Fn, n ~ I} where

Fn(x)

= (1-~)

(211')-1/2 [Zoo e-

u2 /

2du+

2~(1 + l[n,oo)(x)),

x E lRl.

It is easy to see that

lim Fn(x) = (x) for all x E lRl n-too where is the standard normal d.f., that is Fn ~ as n --+ 00. However, the moments Q~n) of any order k of Fn tend to infinity as n --+

00

and hence Q~n) cannot converge to the moments Qk of N(O, 1). Recall that here Q2k-1 = 0, Q2k = (2k - I)!!, k = 1,2, .... (ii) Let Fn be the d.f. of a r.v. Xn distributed uniformly on the interval [0, n] and Fo be the d.f. of a degenerate r.v. X o. for example. Xo = 0. Define

1

Gn(x) = ~Fn(x)

+

(1) Fo(x), x 1- ~

E lR I , n ~ I.

Then {G n , n ~ I} is a sequence of d.f.s. The limit behaviour of {G n } can easily be investigated in terms of the corresponding ch.f.s {1fn}. Since 1jJ n (t) =

1'

1

e t't :z: dGn(x) = _(e ttn - l)/(itn) -00 n 00

+

(1 1- - ) n

we find that limn-too 'l/Jn (t) = 1. t E ~ I which implies that

°

lim Gn(x) = Fo(x) n-too

for all x E lR 1 except x = (the value of Xo; the only point of jump of Fo). It remains for us to clarify whether the moments Q~n) of G n converge to the moments Qk of Fo. We have

Q~n) = [ : xk dGn(x) = n k/(k + 1) --+ 00 for every k, k

as n --+

= 1,2, ... , while the moments Qk of Fo are all zero.

00

182

16.9.

COUNTEREXAMPLES IN PROBABILITY

Weak convergence of a sequence of distributions does not always imply the convergence of the moment generating functions

Recall first a version of the continuity theorem. Suppose {Fn, n = 1,2, ... } are d.f.s and {Mn, n = 1,2, ... }, the corresponding m.g.f.s Mn(z) exist for all Izi :S ro and all n. If F and M is another pair of a d.f. and m.g.f. such that Mn(z) -+ M(z) as n -+ 00 for all Izl :S rJ where rJ < ro, then Fn ~ F. Thus under general conditions the convergence of the m.g.f.s implies the weak convergence of the corresponding d.f.s and this motivates us to ask the inverse question: if Fn ~ F. does it follow that Mn -+ M as n -+ oo? Intuitively we may guess that the answer is 'no' rather than 'yes'. simply because when talking about a m.g.f. we assume at least the existence of moments of any order. The latter is not necessary for the weak distribution. A simple example shows that the answer to the above question is negative. Consider the d.f.s F and Fn. n = 1,2, .... defined by

F(x) =

{O,1,

~f

x <

If x

~

° 0,

0,

Fn(x)

={ !+

Cn

arctan (nx) ,

1,

if x < -n if -n :S x if x ~ n


where Cn = 1/[2arctan(n 2 )]. It is easy to check that Fn(x) -+ F(x) as n -+ 00 at all points of continuity of F. Hence Fn ~ F. Since F is a degenerate distribution concentrated at 0, then its m.g.f. Mn(z) = I for all z. Further, the m.g.f. Mn(z) of Fn ,

= (n

cne zx 1 n 2 2 dx J-n +n x exists for all z. It is almost obvious that Mn(z) -+ M(z) as n -+ 00 only for z If z f: 0, Mn(z) f+ M(z) as n -+ 00 since IMn(z)1 -+ 00 as n -+ 00. Mn(z)

16.10.

Weak convergence of a sequence of distribution functions does not always imply their convergence in the mean

i:

Let F o, F J , F2, ... be d.f.s. Suppose for some (3 (I)

= 0.

nl~~

>

°

the following relation holds:

IFn(x) - Fo(x)li3 dx

= 0.

From here it is easy to derive that Fn ~ F as n -+ 00. Now let us analyse (1) but in the opposite direction. Firstly, suppose that Fn ~ Fo. The question is, under what additional assumptions we can obtain a relation like (1) with a suitable {3 > o? One possible answer is contained in the following result (see Laube 1973). If Fn ~ Fo and for some '"Y > 0, (2)

183

LIMIT THEOREMS

then Fn tends to Fo in the mean of order f3 > I Ir' that is (2) and the weak convergence of Fn to Fo imply (1) with f3 > Ilr' Our aim now is to show that (1) need not be true if we take f3 = II r. To see this, consider the following d.f.s:

Fo(x) = l[o,oo)(x), x E l~'" I

I

Fn(x) = ;;Ir-n,o)(x) + I[o,oo)(x), x E lR, n = 1,2, ....

i:

Then it can be easily seen that lim Fn(x)

Ixl dFn(x) = 1,

n-too

Obviously condition (2) is valid for f3 = I. that is. for f3 = I Ir. since

i:

r =

= 1[0' 00) (x) = Fo(x),

x E lRl.

I. However, relation (I) does not hold for

IFn(x) - Fo(x)1 dx

=1

for all n.

Finally, note that relations like (1) can be used to obtain estimates for the global convergence behaviour in the central limit theorem (CLT) (see Laube 1973).

SECTION 17.

CENTRAL LIMIT THEOREM

Let {X n, n ~ I} be a sequence of independent r. v.s defined on the probability space (0,9", P). As usual, denote

Sn O'~

= Xl + ... + X n ,

ak = EX k , An = ESn = al s~ = V Sn = a? + ... + o'~.

= VX k ,

+ ... + an,

We say that the sequence {X n} satisfies the central limit theorem (CLT) (or, that

{Xn} obeys the CLT) if

nl~~ P[(Sn -

An)1 Sn

~ x] =


iXoo e-

u2 /

2

du for all x E IR' .

Let Fk denote the d.f. of Xk. Clearly, we can suppose that EXk = 0 for all k Now introduce the following three conditions:

(Lindeberg condition); 0'2

(F)

lim

max ~

n-too ':$k:$n s~

=0

~

I.

184

COUNTEREXAMPLES IN PROBABILITY

(Feller condition);

(UAN)

lim n-HXl

max P[lXkn

I :$k:$n'

> c] = 0

-

where Xkn

= Xk! Sn.

(uniform asymptotic negligibility condition (u.a.n. condition». Now we shall formulate in a compact fonn two fundamental results.

Lindeberg theorem. (L)

(CLT)

=}

Lindeberg-Feller theorem. If (F), then (L)

-¢::::::}

( CLT)

(L)

-¢::::::}

(CLT).

or if (UAN). then The proof of these theorems and several other related topics can be found in many books. We refer the reader to the books by Gnedenko (1962), Fisz (1963), Breiman (1968). Billingsley (1968. 1995). Thomasian (1969). Renyi (1970). FelJer (1971). Ash (1972). Chung (1974), Chow and Teicher (1978). Loeve (1978). Laha and Rohatgi (1979) and Shiryaev (1995). The examples below demonstrate the range of validity of the CLT and examine the importance of the conditions under which the CLT does hold. Some related questions are also considered.

17.1. Sequences of random variables which do not satisfy the central limit theorem (i) Let X I , X 2, .•• be independent r. v.s defined as follows: P[ X I = 1] = P[X I - 1] = ~ and for k ~ 2 and some c. 0 < c < 1.

P[Xk

= ±1] = ~(1- c),

P[Xk

= ±k] = 2~2C,

P[Xk

= 0] = (1 -

First let us check if the Lindeberg condition is satisfied. We have

If n is large enough and such that cvln

>

1, c

> 0 is fixed. then we find

:2)

c.

185

LIMIT THEOREMS

Therefore the given sequence {Xk} does not satisfy the Lindeberg condition. However, this does not mean that the CLT fails to hold for the sequence {Xd because the Lindeberg condition is only a sufficient condition. Actually the sequence {Xk} does not obey the CLT. This follows from the fact that Xk/ Sn satisfy the u.a.n. condition. Indeed. if k if k

< cy'n 2:: cy'n.

Thus

Now Sn/ Sn ~ ~ where ~ "'" N(O, I); this and the u.a.n. condition would imply the Lindeberg condition which, as we have seen above, is not satisfied. Thus our final conclusion is that the Lindeberg condition is not satisfied and the CLT does not hold.

!

(ii) Let the r.v. Y take two values, 1 and -I, with probability each, and let {Yk, k 2:: I} be a sequence of independent copies of Y. Define a new sequence {Xk, k 2:: I} where Xk = JISYk/4k and let Sn = XI + ... + X n . Since EY = 0 and V X = 1 we easily find that

ESn

=0

and s~

= VSn = 1 -

Thus s~ ::::: 1 for large n (this is why the factor On the other hand it is obvious that

(1/16)n.

v't5 was involved).

P[lSnl :; ~] = 0 for every n 2:: 1. Therefore the probabilities P[Sn ::; x] cannot converge to the standard nonnal d.f. 4>(x) for all x, so the sequence {Xk} does not obey the CLT. Note that in this ex.ample XI 'dominates' the other tenns. (iii) Suppose that for each n,

Sn = Xnl + Xn2 + ... + Xnn where Xnl, ... ,Xnn are independent r.v.s and each has a Poisson distribution with mean 1/(2n). We could ex.pectthat the distribution ofthe quantity (Sn- ESn ) /VVB:: will tend to the standard nonnal d.f. 4>. However, this is not the case, in spite of the fact that P[Xnk = 0] = e- 1/(2n) ::::: 1 for large n that is each Xnk is 'almost' zero. It is enough to note that for each n the sum Sn has a Poisson distribution with parametefi In particular, P[ Sn = 0] = e -I /2 implying that the distribution of (Sn - ESn) / Sn cannot be close to 4>.

COUNTEREXAMPLES IN PROBABILITY

186

17.2.

How is the central limit theorem connected with the Feller condition and the uniform negligibility condition?

Let {X n , n ~ I} be a sequence of independent r.v.s such that Xn . . . ., N(O, O'~) where O'f 1 and O'i 2k- 2 for k ~ 2. Then Sn = Xl + ... +Xn has variance s~ = 2n - l . Since Xk/Sk ,..,.. N(O,~) we find that

=

=

Sn/ Sn "" N(O, I) for each n and therefore the cLr for {Xk} is satisfied trivially. Further, 0'2

lim

max --.!. 2

n-too 1
2n- 2

I

= n-too lim -n -1 = :f: 0 2 2

and moreover

Hence neither the Feller condition nor the u.a.n. condition holds. This implies that the Lindeberg condition also does not hold. However, despite these facts the sequence {Xn} obeys the CLT.

17.3.

Two 'equivalent' sequences of random variables such that one of them obeys the central limit theorem while the other does not

Consider again the sequence {X n, n ~ I} from Example 17.1: namely, P[ X I ± I] = ~ and for k ~ 2 and 0 < c < 1,

P[Xk

= ±l] = ~(1

c), P[Xk = ±k] =

2~2C,

P[Xk

= 0] =

(1

;2) C.

Using truncation we define the sequence {Xnk, k = 1, ... , n, n ~ I} by

Denote Sn = Xnl VX nk = I - c if k > and thus

+ " . + gnn, s;' == VSn . Since V Xnk = I if k ::; ....jii and ....jii, we find that s~ == [...fii] + (I - c)(n - [...fii]) ~ n(l- c) c))c

-t

O.

Therefore the Lindeberg condition holds and Sn/Sn ~ 1J where 1J is a r.v. distributed N(O, 1). So the sequence {'Ynk} obeys the CLT.

187

LIMIT THEOREMS

We shall show that the sequences {Sn} and {Sn} (not {Xn} and {Xnd) are 'equivalent' in the following sense:

(1) Indeed, n

P[Sn -:f Sn] ~

L P[Xk -:f X nk ] k=1

n

n

~

L P[lXk/ > v'n] ~ L k=1

P[lXk/

= k].

k=[.;n]

Therefore

P[lX.1

= k] = :'

D

(1 -

and

f: :' <

00

k=1

=>

P[Sn '" En] -+ 0 as n -+

00.

However (see Example 17.1) the sequence {Xn} does not obey the CLT. Thus we have constructed two sequences, {Xn} and {Xnd, which are equivalent in the sense of(1) and such that the CLTholds for {Xnk} but does not hold for {X n }. Note again that the Lindeberg condition is valid for {Xnk} but not for {X n }.

17.4.

If the sequence of random variables {X n} satisfies the central limit theorem, what can we say about the variance of Sn/JVS:,.?

Consider two sequences, {Xk' k ~ I} and {Yk, k ~ I}, each consisting of independent r. v.s and such that

Denote

Sn = YI

+ ... + Yn , Sn

= XI

+ ... + X n ·

Obviously the sequence {Yn } obeys the CLT: that is, Sn/.fii ~ ~ where ~ N(O, I). The truncation principle (see Gnedenko 1962; Feller 1971), when applied to the sequence {X k}, shows that Sn/.fii has the same asymptotic behaviour as that of Sn/.fii. Thus we conclude that Sn/.fii ~ 1] as n -+ 00 where 1] N(O, I). f'V

f'V

Then we can expect intuitively that

V[Sn/J1iJ -+ I and V[Sn/J1iJ For the sequence {Yk} we have EYk

-4

1 as n -+

00.

= 0, VYk = I. Thus for each n,

I = V[Sn/v'nl -+ I as n

-400.

188

COUNlEREXAMPLES IN PROBABILITY

On the other hand, for {X/c} we find EX/c = 0, VX/c = 2 - 1/ k 2 and

_ 1 V[Sn/VnJ = - I: n

n

(

k=1

1 1 1) k = 2 I: k --t 2 as n --t n n

2-

2

k=l

2

00

(since E~l (1/ P) < 00), that is V[Sn/ Jii] fi 1 as we assumed. Therefore the CLT does not ensure in general the convergence of the moments of the normed sum Sn/ fo to the moments of the normal distribution N(O, 1). For the convergence of the moments we need some additional integrability conditions. In particular,

17.5.

Not every interval can be a domain of normal convergence

Suppose {X n , n ~ I} is a sequence of Li.d. r.v.s which satisfies the CLT. Denote by Fn the d.f. of (Sn - ESn )/(VSn )1/2 where Sn = Xl + ... + X n. The uniform convergence Fn (x) --t cI>( x), x E IR I implies (1)

. 1 - Fn(x) I1m n-HlO 1 cI>(x)

1 uniformly in x on any finite interval of IR I.

Note that (1) will hold uniformly on intervals of the type [0, bnl whose length btl increases with n. In general, intervals for which (1 ) holds are called domains of normal convergence. Obviously such intervals exist, but we now show that not every interval can be a domain of normal convergence. Consider Xl, X2, .. , independent Bernoulli r.v.s with parameter p: that is, P[XJ = 1] = p = I - P[XI = 0]. Obviously the sequence {Xn} obeys the CLT. If Sn = Xl + ... + Xn then ESn np, s~ = VS n = np(l - p) and

1- Fn(x) = P

= P Hence for an arbitrary x

[(np {l-PW (~Xk -n 1 2 /

p)

>

xl

[~Xk > x(np(1 - p))'/2 + npl.

> (n(l

- p)/p)I/2 we obtain the equality

[I - Fn(x)]j[l - cI>(x)]

=0

which clearly contradicts (1). Therefore (1) cannot hold for any interval of the type [O,O(Vn)]. In particular the interval [O,cpJii], where cp > ((1 - p)/p)I/2 (p is fixed), cannot be a domain of normal convergence. Finally note that intervals of the type [0, o( yin)] are domains of normal convergence. This follows from the well known Berry-Esseen estimates in the CLT (see FeHer 1971; Chow and Teicher 197R; Shiryaev ] 995).

189

LIMIT THEOREMS

17.6. The central limit theorem does not always hold for random sums of random variables Let {X n, n > I} be a sequence of r. v.s which satisfies the CLT. Take another sequence {v n , n ~ I} of integer-valued r.v.s such that Vn ~ 00 as n -t 00 and define Tn = SV n = XI + ... + XVn and b~ - VTn . If lim P[(Tn

n-too

ETn)/bn S x] = (x) ,

x E IRI

we say that the CLT holds for the random sums {SlIn} generated by {Xn} and {vn }. In the next two examples we show that the CLT does not always hold for {Sv n }, In both cases {Xn,n ~ l}isasequenceofLi.d.r.v.ssuchthatP[X I = I] P[XI = 1] = Obviously if Vn = n a.s. for each n, then Tn = Sn = XI + ... + X n ,

!.

b2 = nand Tn/b n ~ Ewhere E"" N(O, I). (i) Define the sequence {vn, n ~

°

Vo = Then

Vn

~

00

O} as follows:

and Vn = min{k

as n -t

00,

> Vn-l : Sk

= (_l)k} forn ~ 1.

b~ = VTn = n 2 and dearly

It follows that the distribution of Tn/bn does not have a limit as n -t the CLT cannot be valid for the random sums {Sv n }.

00

and hence

(ii) Let {v n , n ~ l} be independent r. v.s such that Vn takes the values nand 2n. with probabilities p and q = I - p respectively. Suppose additionally that {v n } is independent of {X n }. Then

b~

= VTn = pE[S~]

qE[S?n] = (I + q)n.

It is easy to check that Tn/bn does not converge in distribution to a r.v. E,. . , N(O, I). More precisely, P[Tn/b n S x] converges to the mixture of the distributions of two r.v.s, EI '"" N(O, (I + q)-2) and 6 -- N(O, 2( I + q)-2) with weights p and q respectively.

17.7. Sequences of random variables which satisfy the integral but not the local central limit theorem Let {Xu, n ~ I} be a sequence ofindependentr.v.s. Denote by Fnand in respectively the dJ. and the density of (Sn - ESn)/sn where as usual Sn = XI + .. , + X n• s~ - VS n. Let us set down the following relations:

(I)

lim Fn(x) = (x) = (27r) 1/2 n-too

j

x

-00

2

e- u /2 du,

x E IRI;

COUNTEREXAMPLES IN PROBABILITY

190

(2) Recall that if (1) holds we say that the sequence {Xn} obeys the integral CLT, while in case (2) we say that {Xn} obeys the local CLT (for the densities). It is easy to see that (2)=}(1). However, in general weak convergence does not imply convergence of the corresponding densities (see Example 14.9). Note that in (1) and (2) the limit distribution is N{O, 1). Hence in this particular case we could expect that the implication (1 )=}(2) is true. Two examples will be considered where (l )~(2). In the first example the variables are identically distributed while in the second they have different distributions. (i) Let X be a r. v. with density

(3)

0,

I(x)

{ 1/(2Ixllog2Ixl),

if Ixl 2:: e- I if Ixl < e- I .

Since X is a bounded r.v., the sequence {Xn' n 2:: I} of independent copies of X satisfies the (integral) CLT. So the aim is to study the limit behaviour of the density

J;

-1

fn of (XI + ... + Xn)/(aVn) where a 2 = VX = (x/ log2 x) dx. If g2 is the density of the sum X I + X2 then g2 is expressed by the convolution -1

92(X) =

iee_

f(u)f(x - u) du. 1

Let us now try to find a lower bound for g2. It is enough to consider x in a neighbourhood of 0; in particular we can assume that Ixl < e -I, and, even more, that 0 < x < e- I . Then g2(X) 2:: J~x f(u)f(x - u) duo Since f(x - u) reaches its minimum in the domain lui::; x at u = 0, we have

1 g2(X) > - 2x log2 x

jX

I

I

2 du = 2xl log3 xl . -x 21ullog lui

Analogously we establish that in a neighbourhood of 0 the density g3 of the sum XI + X2 + X3 satisfies the inequality

g3(X) >

C3

4'

xlog x

In general, jf gn is the density of XI

gn(x) >

e3

= constant> O.

+ ... + Xn we find that around 0,

en

I I n+1 I' x og x

en

= constant> O.

Thus for each n, gn(X) takes an infinite value at x = O. Since the density fn is obtained from gn by suitable nonning, we conclude that In(x) cannot converge to
191

LIMIT THEOREMS

Therefore the sequence {Xn} defined by the density (3) does not obey the local CLT a1though the integral CLT holds.

Oi) Let {Xnl n ~ I} be independent r.v.s where Xn has density if _2- n - 2 :$; X :$; 2- n- 2 or 1 - 2- n- 3 otherwise.

4 In X = {2n, () () 0,

< Ixi < ] + 2- n- 3

!

It is easy to see that EX n = 0, V Xn + 5/(3·2 2k +7 ). Then for an arbitrary k > 1, ~ < V X k < 1, the Lindeberg condition is satisfied and hence the sequence {Xn} obeys the (integral) CLT. Denote by 9k(X), x E rn:.! the density of the sum Sk = XI + ... + Xk. Then for k = 2,92 is the convolution of II and 12, that is

92(X) = (II

* h)(x)

i:

II (u)h(x - u) duo

= t. By (4) we have 15 !8 -< u -< 18 or T6 < Iu I < 15 ?6:$; u:$; J~ or 32 < Iu I <

Let us find the value of 92(X) at the point x

b(u) ~ 0,

12 (t - u)

if

~ 0,

if

17

16'

17 32'

Comparing the intervals where II ~ 0 and 12 ~ 0 we see that92(~) = O. Analogously we find that 00

92(U)h(x _ u)

dul

/ -00

= 0

x=!

t) =

and, more generaBy, that 9n ( 0 for aU n ~ 2. It is not difficult to see that 9n(X) = 0 for al1 x of the fonn x = ~(2m + 1), m = 0, ±I, ±2, ... and finally that 9n(X) = 0 for all x = !(2m + 1) + 6 where m = 0, ± I, ±2, ... and 161 < ThesumSn = XI + .. ·+Xn hasESn = oand VSn = s~ = 1n+ 1~'i2(J _2- 2n ). Since the density 9n of Sn and the density Pn of Sn/ Sn satisfy the relation Pn(x) = Sn9n(XSn), we have to study the behaviour of the quantity Sn9n(X8 n ) as n -t 00. Again, take x Then

!.

= i.

I ) -_ [I'2 n + 1152 5 (1 Sn9n ( iSn

-

2- 2n )] 1/2 '9n (I2 [I2n + 1152 5 (1

-

2- 2n )] 1/2)

.

If n is of the fonn n = 2(2N + 1)2, then the argument of 9n becomes

t(2N + 1)[1

+

11552(1-

2- 2.2(2N+I)2)(2N

!

For large N this expression takes the fonn (2N + I) + properties of 9n established above we conclude that

+ 1)-2].

6 with 161 <

Sn9n (1sn) = 0 for sufficiently large n.

!. From the

COUNTEREXAMPLES TN PROBABILITY

192

This implies that lim Pn

n-too

(4) =

o.

However. rp(!) =I 0 and thus relation (2) is not possible. Therefore the sequence {Xn} defined by the densities (4) does not obey the local CLT. General conditions ensuring convergence of both the d.f.s and the densities are described by Gnedenko and Kolmogorov (1954).

SECTION 18.

DIVERSE LIMIT THEOREMS

In this section we have collected examples dealing with different kinds of limit behaviour of random sequences. The examples concern random series, conditional expectations, records and maxima of random sequences, versions of the law of the iterated logarithm and net convergence. The definitions of some of the notions are given in the examples themselves. For convenience we formulate one result here and give one definition.

Kolmogorov three-series theorem. Let {Xnl n ~

)} be a sequence of

independentr.v.sandX~c) = XnI!lxnl~c)forsomec > o. A necessary condition for the convergence OfL~=l Xn with probability 1 is that the series 00

L

E[XAc)],

00

00

L V[X~c)], L P[IXnl ~ c]

n=] n=1 converge for every c > O. A sufficient condition is that these series are convergent for some c > O. n=1

The proof of this theorem and some usefuJ corollaries can be found in the books by Breiman (1968), Chow and Teicher (1978), Shiryaev (1995). Now let us define the so-called net convergence (see Neveu 1975). Let T be the set of all bounded stopping times with respect to the family (:1n, n EN). Here (:1n) is a non-decreasing sequence of sub-a-fields of:1 and a stopping time T is a function with values in [0, ooJ such that [T = nJ E :1n for each n E N. The family (aT' T E T) of real numbers, called a net, is said to converge to the real number b provided for every [ > 0 (here is TO E T such that for aU T E T with T 2: TO we have la T bl < [, Each of the examples given below contains appropriate references for further reading.

18.1.

On the conditions in the Kolmogorov three-series theorem

(i) Let {Xn, n ~ I} be independent r.v.s with EXn = 0, n

2: 1. Then the condition

L~=I V Xn < 00 implies that L~=I Xn converges a.s. Note that this is one of the simplest versions of the Kolmogorov three-series theorem.

LIMIT THEOREMS

193

Let us show that the condition L~=I V Xn < 00 is not necessary for the convergence of L~=I X n . Indeed, consider the sequence {Xn' n ~ I} of independent r.v.s where

Obviously L~=I V Xn = 00 but nevertheless the series L~=I Xn is convergent a.s. according to the Borel-Cantelli lemma. (ii) The Kolmogorov three-series theorem yields the following result (see Chow and

Teicher 1978): if {X n , n ~ I} are independent r.v.s with EXn = 0, n 2: I and 00

I:: E[X~I[lXn 19]

(I)

+ IXnII[IXn I> 1]] < 00

n=1

then the series L~= I Xn converges a.s. Let us clarify the role of condition (1) in the convergence of L~=I X n . For I} of i.i.d. r. v.s with P[~I = I] = this purpose consider the sequence {~nl n P[~I = -1 J = and define Xn = ~n/ ft, n ~ 1. It is easy to check that for any r > 2 the following condition holds:

!

00

(2)

I:: E[lXnn

< 00.

n=1

Condition (2) can be considered in some sense similar to (I). However the series Xn diverges a.s. This shows that the power 2 in the first term ofthe summands in (1) is essential. Finally let us note that if condition (2) is satisfied for some 0 < r ~ 2 then the series L~=I Xn does converge a.s. (see Loeve 1978). L~=I

18.2.

The independency condition is essential in the Kolmogorov three-series theorem

Let us start with a direct consequence of the Kolmogorov three-series theorem (sometimes called the 'two-series' theorem). If Xm n 2: I are independent r.v.s and the series L~=l an and L~=I (T~, with an = EXn • (T~ = V X n , are convergent, then the random series L~=I Xn is convergent with probability 1. Our goal is to show that the independency property for X n • n 2: I, is essential for this and similar results to hold. (i) Let ~ be a r. v. with E~ = 0 and 0 < V ~ = b2 < 00 (i.e. ~ is non-degenerate). I. Then an = EXn = 0, (T~ V Xn = b2/n2 implying Define Xn = ~/n, n

that

00

I:: an n=1

00

=0

00

and I:: (T~ = b I::(l/n 2

n=1

n=1

2

)

< 00.

COUNTEREXAMPLES IN PROBABILITY

194

Hence two of the conditions in the above result are satisfied and one condition, the independence of X n 1 n 2:: I, is not. Nevertheless the question about the convergence of the random series 2::::=1 Xn is reasonable. Since 00

~(w) L(1ln)

n=1

the series 2::~= 1X n (w) is convergent on the set {w : ~ (w) = O} and divergent on the set AC = {w : e(w) -:f. OJ. If the non-degenerate r.v. ~ is such that peA) = p where p is any number in [0, I), we get a random series 2::~= 1X n which is convergent with probability p (strictly less than I) and divergent with probability 1- P (strictly greater than 0). (ii) In case (i) the dependence among the variables X n , n 2:: 1, is 'quite strong'-any two of them are functionally related. Let us see if the independence of X n , n 2:: 1, can be weakened and replaced e.g. by the exchangeability property. We use the following modification of the Kolmogorov three-series theorem. If X n , n 2:: 1. are Li.d. r.v.s with EX 1 = 0, E[ Xf] < 00 and Cn , n 2:: I are real numbers with 2::~== I c~ < 00, then the random series 2:::=1 cnXn is convergent with probability]. Let us now consider the sequence of i.i.d. r. v.s en, n 2:: I with Eel = 0 and E[efJ < 00 and let 7] be another r.V. with E7] = 0, 0 < E[7]2] < 00 and independent of {en,n 2:: I}. Define the sequenceXn, n;::: I, by

Xn = en + 7], n ;::: 1. Thus X n , n ;::: 1, is a sequence of identical1y distributed r.v.s with EX( - 0 and E[Xf] < 00. Obviously the variables X n , n > 1, are not independent. However X n , n ;::: I, is an exchangeable sequence. (See also Example 13.8.) Our goal is to study the convergence of the series 2::~= I cnX n where Cn . n ;::: ]. satisfy the condition 2::~=1 c~ < 00. Choose Cn. n ;::: 1, such that Cn > 0 for any nand 2:::=1 Cn = 00 (an easy case is Cn = ] In). Since cnXn = cnen + Cn7], we have 00

00

ou

n=1

n==]

n==1

The independence of en, n 2:: I, implies that the series E~= 1 cnen is convergent a.s. Hence, in view of 2::~=1 C n = 00, the series E~=, cnXn is convergent on the set A = {w : new) = O} and divergent on AC = {w : new) -:f. OJ. For preliminary given p, p E [0, I), take the r.v. 7] such that peA) p. Then the random series 2:::=1 cnXn of exchangeable (but not independent) variables is convergent with probability p < I and divergent with probability 1 - P > O. We have seen in both cases (0 and (ij) the role of the independence property for random series to converge with probability 1. The same examples lead to one additional conclusion. According to the Kolmogorov 0-1 law, if X n1 n > ],

=

195

LIMIT THEOREMS

are independent r.v.s, then the set {w 2::'=1 Xn(w) converges} has probability or 1. Hence, if X n , n ~ 1, are not independent we can obtain P[w : 2::'=1 Xn(w) converges] = p for arbitrarily given p E [0,1).

o

18.3.

The interchange of expectations and infinite summation is not always possible

Let us start with the fonnulation of a result showing that in some cases the operations of expectations and summation can be interchanged (see Chow and Teicher 1978). If {Xn' n ~ I} are non-negative r.v.s then

(1) Our aim now is to show that (1) is not true without the non-negativity of the variables Xn even if the series 2::'=1 Xn is convergent. Consider {~n' n ~ I} to be Li.d. r. v.s with P[~I = ± 1] = ~ and define the stopping time T = inf {n ~ 1 : 2:;=1 ~k = I} where inf{0} = 00. Then it is easy to check that P[T < 00] = 1. Setting Xn = ~nI[T~nl' we get from the definition of T that

~ Xn ~ ~~nI[T~n[ ~ ~~n ~ I

=?

E

[~Xn] ~ I.

However, the event [T ~ n] E a{6, ... , En-I}, the r.v.s En and I[T~nl are independent and from the properties of the expectation we obtain

EX n =

E~nEI[T~nl

= 0,

n

>

1.

Thus 2:~=1 EX n = 0 and therefore (1) is not satisfied.

18.4.

A relationship between a convergence of random sequences and convergence of conditional expectations

On the probability space (0,1', P) we have given r.v.s X and X n , n space LT (i.e. r-integrable) for some r ~ 1. Suppose Xn ~ X as n

~

-t

1, all in the

00. Then for

L'

any sub-a-field A C l' we have E[XnIA] ---+ E[XIA] as n -t 00 (e.g. see Neveu 1975 or Shiryaev 1995). This statement is a consequence of the Jensen inequality for conditional expectations. Obviously, we can ask the inverse question and the best way to answer it is to consider a specific example. Let X n , n ~ 1 be i.i.d. r.v.s with P[X = 2c] = P[X = 0] = ~, c=/;O is a fixed real number. Take also (a trivial) r.v. X = c: P[X = c] = ]. Then, if A = a{0, O}, the trivial a-field, we obviously get

E[XnIA]

= EX n = 2c.~ + O.~ = c = E[XIA]

for any n

> 1.

196

COUNTEREXAMPLES IN PROBABILITY

Moreover because X, X n are bounded, then for any r

>

I, one has

L'

E[XnIA] ---t E[XIA] as n -+ However E[IX n - Xlr] L'

hence Xn

18.5.

-1+ X

= E[lXn -

as n -+

en

00.

= ~er + ~er = er for all n

~ I, e

f= 0 and

00.

The convergence of a sequence of random variables does not imply that the corresponding conditional medians converge

Let (0,1", P) be a probability space, 1"0 = {0, o} the trivial a-field and 'D a sub-a-field of 1". If X is a r.v., then the conditional median of X with respect to 'D is defined as a 'D-measurable r. v. M such that

P[X ~ 1\11'D] ~ ~ S P[X < Min] a.s. Usually the conditional median is denoted by I-'(XI'D) (see Example 6.10). If {Xn, n ~ I} is a sequence of r. v.s which is convergent in a definite sense, then it is 10gicaJ to expect that the corresponding sequence of the conditional medians also will be convergent. In this connection let us formulate the following result (see Tomkins 197 Sa). Let {X n, n ~ I} and {Mn, n ~ I} be sequences ofr. v.s such that for a given a-field 'D we have Mn = I-'(Xnl'D) a.s. and there exist r.v.s X and M p

such that Xn X and Mn ---t M as n -+ 00. Then M = I-'(XI'D) a.s. We can now try to answer the question of whether the convergence of {Xn} always implies convergence of the conditional medians {M n}. Let be a r. v. distributed uniformly on the interval ~), 'D = :to (the trivial a-field) and define the sequence {Xu} by

e

!,

It is easy to see that Xn ~ X as n -+ 00. Moreover, X" has a unique median Mn and Mn 0 or ] accordingly as n is odd or even. But clearly the sequence {Mn} does not converge. (It would be useful for the reader to compare this example with the result cited above.)

18.6.

A sequence of conditional expectations can converge only on a set of measure zero

If (O,:t, P) is a complete probability space, (:tn, n E N) an increasing family of sub-a-fields of 1",1"00 = limn-too 1"n and X a positive r. v., the following result holds (see Neveu 1975):

(1)

E(XI1"n] u)E[XI1"oo] outsidetheset {w:E[XI1"nl=oo forall n}.

197

LIMIT THEOREMS

We shall show that this result cannot be improved. More precisely, we give an example of an a.s. finite ~oo-measurable r.v. X such that E[XI~nJ = 00 a.s. for all n E N. Clearly in such a case the convergence in (1) holds only on a set of measure zero. Let n = [0, 1J. ~ = 13[0,1] and P be the Lebesgue measure. Consider the increasing sequence (~n) of sub-a-fields of ~ where ~n is generated by the dyadic partitions {(2- n k,2- n (k + 1)),0 ~ k < 2 n ,n EN}. For each n E N choose a positive measurabJefunctionln : [0, I) t-+ jR+ ofperiod2- n with

10' fn(w) dw =

I

and

10' I 1/. >oj (w) dw = 2-

n

,

Since the sum E:'= I ] [fn >0] is integrable, and hence a.s. finite, then X = E:'= lin is a positive r.v. which is finite a.s. Thus the series E~=I In contains no more than a finite number of non-zero terms for almost all w. On the other hand, for aJJ n E N and all k, 0 ~ k < 2n , we have

by the periodicity of 1m. Therefore we have shown that E[ X I~n1 = in (1) holds only on a set of measure zero.

18.7.

00

for all n E N and the a.s. convergence

When is a sequence oCconditional expectations convergent almost surely?

Let (n,~, P) be a probabiJity space and {~n, n 2 I} an independent sequence of sub-a-fields of~, that is, for k = 1,2,. " and Aj E ~j, 1 j k, we have

s: s:

Let X be an integrable r.v. with EX = a and let Xn = E[XI~nJ. The following result is proved by Basterfield (] 972): if E[IXllog+ IXI]

< 00

then P[X

-t

a as n

-t

00] = ]

(log x is defined for x > 0 and log+ x log x if x > ], and 0 if 0 < x ~ ]). We aim to show that the assumption in this result cannot be weakened, e.g. it cannot be replaced by E[lXI] < 00. To see this, consider a sequence {Anl n > ]} of independent events with P( An) = 1/ n. Define the r. v. X by 00

X =

I: m!{m+2 m=1

198

where

COUNTEREXAMPLES IN PROBABILITY

€m

is the indicator function of the event AI A2 ... Am. Since

we obtain

EX

DO OO( 1 = '""" m!/(m + 2)! = '""" LL- m+l m=1

m=1

1) = -.I

m+2

2

Moreover, it is not difficult to verify that

E[X log+ X]

= 00.

Consider now :fn = {0, An, A~, il} and Xn = E[XI:fn]. We need to check if Xn ~! as n -+ 00. Since E[XIAn] = (l/P(A n )) fAn X dP = n fAn X dP, replacing X by L::=I m!€m+2 we arrive at the equality

E[XIAn] = ~. However, L:~=I P(An) = 00 and, by the Borel-Cantelli lemma, almost all w belong to infinitely many An. Therefore limsupXn = ~

7l-tDO

=I

!= a=

EX.

Thus the condition E[IX\log+ IXI] < 00 cannot be replaced by E[IXIl to preserve the convergence Xn = E[XI:fn] ~ a = EX as n -+ 00.

18.8.

< 00 so as

The Weierstrass theorem for the unconditional convergence of a numerical series does not hold for a series of random variables

Let L:~=I an be an infinite series of real numbers. This series is said to converge unconditionally if L:r=1 an", < 00 for every rearrangement {nl,n2,"'} of {I, 2, ... }. (By rearrangement we understand a one-one map of N onto N.) We say that the series L:~=I an converges absolutely if L:~=I Ian I < 00. According to the classical Weierstrass theorem these two concepts, unconditional convergence and absolute convergence, are equivalent. Thus we arrive at the question: what happens when considering random series L:~=I Xn(w), that is series ofr.v.s? Let {Xn' n ~ I} be a sequence of r.v.s defined on some probability space (il, :f, P). The series L:~=I Xn is said to be a.s. unconditionally convergent if for every rearrangement {nd of N we have L:r=1 X n ", < 00 a.s. If L:~=I IXnl < 00 a.s., the given series is a.s. absolutely convergent. So, bearing in mind the Weierstrass theorem, we could suppose that the concepts a.s. unconditional and a.s. absolute convergence are equivalent. However, as will be seen later, such a conjecture is not generally true.

LIMIT THEOREMS

199

Consider the sequence {rn, n = 0,1,2, ... } of the so-called Rademacher functions. that is rn(W) = sign sin(2 n Trw). ~ W < 1, n 0, ], ... (see Lukacs 1975). Actually rn can also be written in the form

°

rn(w)

={

I, -1,

0,

if 2k/2 n < W < (2k ~f (2k If W -

+ 1)/2 n

+ 1)~2n < W < (2k +})/2n k/2 ,k = 0,1, ... ,2 .

Then {rn} is a sequence of independent r.v.s on the probability space (n, 5", P) with n = [0, 1], 5" = '.B(O, I] and P the Lebesgue measure. Moreover, rn takes the 1. values 1 and -1 with probability! each, Ern 0, Vrn Now take any numerical sequence {an} such that

=

00 2: a~ < 00

n=I

but

=

00 2: lanl = 00.

n=I

For example. an = (_1)n/(n + I). Using the sequence {rn} of the Rademacher functions and the numerical sequence {an} we construct the series (1)

Applying the Kolmogorov three-series theorem we easily conclude that this series is a.s. convergent. If {nk} is any rearrangement of N then the series l:~ 1a nk r nk (w) is also a.s. convergent. However, E:':::::I anTn(w) is not absolutely convergent since Irn(w)1 = 1 and l::'=1 lanl = 00. Therefore the series (1) is a.s. unconditionaUy convergent but not a.s. absolutely convergent, and so these two concepts of convergence of random series are not equivalent.

18.9. A condition which is sufficient but not necessary for the convergence of a random power series

=

Let an an(w), n = 0,1,2, ... , be a sequence of i.i.d. r.v.s. The random power series, that is a series of the type 2::'=:0 an (w )zn, is defined in the standard manner (see Lukacs 1975). As in the deterministic case (when an are numbers), one of the basic problems is to find the so-called radius of convergence r = r(w). This T is a r.v. such that for alllzi < r the series 2::'=oa n (w)zn is a.s. convergent. Moreover, r(w) (Jim sUPn-too Vlan(w)I)-'. Among the variety of results concerning random power series, we formulate here the following (see Lukacs 1975). If {an, n 2 O} are Li.d. r.v.s and the d.f. F(x), x E 1R + of la II satisfies the condition

=

(1)

/00 10gxdF(x) < 00

200

COUNTEREXAMPLES IN PROBABILITY

then the random power series L~=o an (w )zn has a radius of convergence r(w) such that P[r(w) ~ I] = 1. Let us show by a concrete example that condition (1) is not necessary for the existence of r with P[r ~ I] = I. Take ~ as a r. v. distributed unifonnly on the interval [0,1]. Define an by an(w) = exp(I/~(w)). Then the common d.f. of an is if if

X

<e

X

> e.

Clearly

JOO 10gxdF(x) =

00

and condition (I) is not satisfied. However, for any c > P[lim sup [Ianl ~ (I n-too

°

we have

+ ctJ] = P[lim sup [0, I I(n log( I + c))]] = 0. n-too

This relation, the definition of a radius of convergence and a result of Lukacs (1975) allow us to conclude that P[r(w) < x] = for all x E (0, I]. For x = I we get P[r(w) ~ I] = 1. Thus condition (I) is not necessary for the random power series L~=o an(w )zn to have a radius of convergence r ~ I.

°

18.10.

A random power series without a radius of convergence in probability

As before consider a random power series and its partial sums N

00

2:: an(w)zn

2:: an(w)zn

and UN(Z, w) =

n=O

n=O

where the coefficients an(w), n = 0, I, ... , are given r.v.s. If Un(z) are convergent in some definite sense as N ----t 00 then the random power series is said to converge in the same sense. There are several interesting results about the existence of the radii of convergence if we consider a.s. convergence, convergence in probability and LT -convergence. Note that a.s. convergence was treated in Example 18.9. Now we aim to show that no circle of convergence exists for convergence in probability of a random power series. For this let {an (w), n ~ O} be independent r.v.s with

P[ao

= 0] =

I,

Plan

= nn] =

lin,

Plan

= 0] = I -

lin, n ~ 1.

It is easy to check that the power series L~=o an (w)zn is a.s. divergent, that is its radius of convergence is ro = O. Clearly, this series cannot converge in probability or in LT -sense. From the definition of an we find that

LIMIT THEOREMS

201

Define another power series, say L:~=o bn(w)zn, whose coefficients bn are given by bo = ao and bn = an - an-I for n 2: 1. Obviously, N

lim "" bn 1n ~

P-

N-+oo

=P -

n=O

lim aN = O.

N-+oo

Furthermore, we have N

N-I

n=O

n=O

L bn(w)zTl = aN(w)zN + (1 - z) L an(w)zn.

(I)

It is clear that the series L:~=o bn{w)zn converges in probability at least at two points, namely z 0 and z = I. If we suppose that it is convergent for a point z such that z -:p 0 and z -:p 1, we derive from (I) that

=

~ an(w)zn = 1 ~ z

[t.

bn(w)zn - aN (w)zN

1

which must also converge in probability as N -+ 00. However, this contradicts the fact that TO = O. Therefore in general the random power series has no circle of convergence in probability. Finally, note that Arnold (1967) characterized probability spaces in which every random power series has a circle of convergence in probability. Such a property holds iff the probability space is atomic.

18.11.

1\vo sequences of random variables can obey the same strong law of large numbers but one of them may not be in the domain of attraction of the other

Let {Xk, k 2: I} and {Yk, k 2: I} each be a sequence of i.i.d. r.v.s. Omitting subscripts, we say that X and Y obey the same SLLN if for each number sequence {an, n 2: I} with 0 < an t 00 either

n (1)

n

LXk

= o(a n )

k=1

and LYk = o(a n )

a.s.

k=1

or

n (2)

lim sup( 1/ an) L n-+oo

k=1

n

Xk =

00

and limsup(l/a n ) LYk n-+oo

= 00

a.s.

k=1

We also need the following result of Stout (1979): X and Y obey the same SLLN iff

(3)

P[IYI > x]/P[lXI > x] = O(x) as x -+

00.

202

COUNTEREXAMPLES IN PROBABILITY

Note that the statement that two r.v.s X and Y, or more exactly two sequences {X k, k ~ I} and {Yk , k ~ I}, obey the same SLLN is closely related to another statement involving the so-called domain of attraction. Let U and V be r.v.s with d.f.s G and Hand ch.f.s