Asymptotic Statistics

vn(
-v-+

=

cp (e + h) - ¢ (e) =
h � 0.

All the expressions in this equation are vectors of length m, and II h II is the Euclidean norm. The linear map h f-+ ¢� (h) is sometimes called a "total derivative," as opposed to 25

Delta Method

26

partial derivatives. A sufficient condition for ¢ to be (totally) differentiable is that all partial derivatives arp j (x ) I a xi exist for X in a neighborhood of e and are continuous at e . (Just existence of the partial derivatives is not enough.) In any case, the total derivative is found from the partial derivatives. If ¢ is differentiable, then it is partially differentiable, and the derivative map h r--+ ¢� (h) is matrix multiplication by the matrix � (e)

)

axk

a rpm a xk

(e)

If the dependence of the derivative ¢� on e is continuous, then ¢ is called continuously

differentiable.

It is better to think of a derivative as a linear approximation h r--+ ¢� ( h ) to the function h r--+ ¢ ( e + h) - ¢ (e) than as a set of partial derivatives. Thus the derivative at a point e is a linear map. If the range space of ¢ is the real line (so that the derivative is a horizontal vector), then the derivative is also called the gradient of the function. Note that what is usually called the derivative of a function ¢ : Itt r--+ Itt does not com­ pletely correspond to the present derivative. The derivative at a point, usually written ¢' (e) , i s written here as ¢� . Although ¢' (e) i s a number, the second object ¢� i s identified with the map h r--+ rjJ� ( h ) = ¢'(e) h. Thus in the present terminology the usual derivative function e r--+ ¢' (e) is a map from Itt into the set of linear maps from Itt r--+ Itt , not a map from Itt r--+ JEt. Graphically the "affine" approximation h r--+ ¢ (e) + ¢� (h) is the tangent to the function ¢ at e .

Let ¢ : lDrp C Ittk r--+ Ittm be a map defined on a subset of Ittk and dif­ ferentiable at e. Let Tn be random vectors taking their values in the domain of ¢. If rn ( Tn - e) """"' T for numbers rn � oo, then rn ( rfJ C Tn ) - ¢ (e) ) 'V'-7 rjJ� (T). Moreover, the diffe rence between rn (rfJ C Tn ) - rp (e)) and rfJHrn C Tn - e) ) converges to zero in probability. 3.1

Theorem.

Because the sequence rn ( Tn - e) converges in distribution, it is uniformly tight and Tn - e converges to zero in probability. By the differentiability of ¢ the remainder function R (h) = ¢ (e + h) - rp (e) - ¢� (h) satisfies R (h) = o ( ll h II) as h � 0. Lemma 2. 12 allows to replace the fixed h by a random sequence and gives

Proof.

rjJ (Tn) - ¢ (e) - rfJ� ( Tn - e) = R ( Tn - e) = op ( II Tn - e 11 ) .

Multiply this left and right with rn , and note that 0 p ( rn II Tn - e II ) = Op ( 1 ) by tightness of the sequence rn (Tn - e ) . This yields the last statement of the theorem. Because matrix multiplication is continuous, ¢� ( rn (Tn - e) ) ¢� (T) by the continuous-mapping theorem. Apply Slutsky's lemma to conclude that the sequence rn (¢ (Tn) - rp (e)) has the same weak limit. • 'V'--7

A common situation is that Jn(Tn - e ) converges to a multivariate normal distribution Nk (fl.,, L: ) . Then the conclusion of the theorem is that the sequence ,Jn ( ¢ (Tn) - ¢ (e) ) converges in law to the Nm (r/J�fl.,, rjJ� L: (rjJ� ) T ) distribution. Example (Sample variance). The sample variance of n observations X 1 , . . . , Xn is defined as S 2 = n - 1 L7= 1 ( X i - X) 2 and can be written as ¢ ( X , X 2 ) for the function

3.2

3. 1

Basic Result

27

-x 2 . (For simplicity of notation, we divide by n rather than n - 1 .) Suppose that 5 2 is based on a sample from a distribution with finite first to fourth moments a1 , a 2 , a3 , a4 .

cp (x , y) =

y

By the multivariate central limit theorem,

The map ¢ is differentiable at the point () = (a1 , a 2 )T, with derivative ¢c'aI , a2 ) = ( -2a1 , 1 ) . Thus if the vector (T1 , T2 ) ' possesses the normal distribution in the last display, then The latter variable is normally distributed with zero mean and a variance that can be ex­ pressed in a1 , . . . , a4 • In case a1 = 0, this variance is simply a4 - a � . The general case can be reduced to this case, because 52 does not change if the observations Xi are replaced by the centered variables Yi = Xi - a1. Write f.L k = EYik for the central moments of the Xi . Noting that 5 2 = ¢ (Y , Y 2 ) and that ¢ (f.lr , f.l 2 ) = f.l 2 is the variance of the original observations, we obtain In view of Slutsky's lemma, the same result is valid for the unbiased version nj(n - 1)5 2 of the sample variance, because Jli (n/(n - 1) - 1) -+ 0. 0

3.3 Example (Level of the chi-square test). As an application of the preceding example, consider the chi-square test for testing variance. Normal theory prescribes to reject the null hypothesis H0 : f.L 2 :S 1 for values of n 5 2 exceeding the upper a point x; a of the xL 1 distribution. If the observations are sampled from a normal distribution, then the test has exactly level a. Is this still approximately the case if the underlying distribution is not normal? Unfortunately, the answer is negative. For large values of n, this can be seen with the help of the preceding result. The central limit theorem and the preceding example yield the two statements Xn2- 1 - (n - 1 ) v'2n - 2

-v-t

N (O ' 1) '

where K = f.L4 / f.l � - 3 is the kurtosis of the underlying distribution. The first statement implies that ( x ;, a - (n - 1 )) j v'2n - 2) converges to the upper a point Za of the standard normal distribution. Thus the level of the chi-square test satisfies

( r:: ( 52

)

)

(

)

x; a - n za v'2 -+ 1 -

Xn2, a ) = P v n f.l - 1 > ,J1i v'K + 2 · 2 The asymptotic level reduces to 1 -

distribution is 0. This is the case for normal distributions. On the other hand, heavy-tailed distributions have a much larger kurtosis. If the kurtosis of the underlying distribution is "close to" infinity, then the asymptotic level is close to 1 -

Delta Method

28

Level of the test that rejects if nS 2 I M exceeds the 0. 95 quantile ofthe xf9 distribution.

Table 3 . 1 .

Law

Level

Laplace

0.12 0.12

0.95 N (O, 1) + 0.05 N (O, 9)

Note: Approximations based on simulation of 1 0,000 samples.

normal approximation to the distribution of ,.jii ( S 2 I fJ, 2 - 1) the problem would not arise, provided the asymptotic variance K + 2 is estimated accurately. Table 3. 1 gives the level for two distributions with slightly heavier tails than the normal distribution. D In the preceding example the asymptotic distribution of ,Jri(S 2 - CJ 2 ) was obtained by the delta method. Actually, it can also and more easily be derived by a direct expansion. Write vr::n (S 2 - (J 2 )

r:::

= vn

( �{=( 1 -;;_

) -

(X i - 1-i ) 2 - (J 2 - vr:::n (X - fJ,) 2 .

The second term converges to zero in probability; the first term is asymptotically normal by the central limit theorem. The whole expression is asymptotically normal by Slutsky's lemma. Thus it is not always a good idea to apply general theorems. However, in many exam­ ples the delta method is a good way to package the mechanics of Taylor expansions in a transparent way.

3.4 Example. Consider the joint limit distribution of the sample variance S2 and the t -statistic X IS . Again for the limit distribution it does not make a difference whether we use a factor or - 1 to standardize S 2 . For simplicity we use Then ( S 2 , X IS) can be written as ¢ (X, X2) for the map ¢ : Itt2 r--+ Itt2 given by n

n

n.

¢ (x, y)

=

(y - x 2 , (y - :2) 1 /2 ) .

The joint limit distribution of ,.jii ( X - a 1 , X2 - a 2 ) is derived in the preceding example. The map ¢ is differentiable at e = (a l , a 2 ) provided CJ 2 = a 2 - ar is positive, with derivative

It follows that the sequence ,Jri(S2 - CJ 2 , X 1 S - a l /CJ) is asymptotically bivariate normally distributed, with zero mean and covariance matrix,

It is easy but uninteresting to compute this explicitly. D

3. 1

3.5

Basic Result

29

The sample skewness of a sample X 1 , . . , , Xn is defined as n n - 1 L... '\' r = 1 (X · - X) 3 ln /2 . 1 (n - L7= 1 (Xi - X) 2 ) 3 Not surprisingly it converges in probability to the skewness of the underlying distribution, defined as the quotient A. = f..L 3 /C5 3 of the third central moment and the third power of the standard deviation of one observation. The skewness of a symmetric distribution, such as the normal distribution, equals zero, and the sample skewness may be used to test this aspect of normality of the underlying distribution. For large samples a critical value may be determined from the normal approximation for the sample skewness. The sample skewness can be written as ¢ (X, X 2 , X3) for the function ¢ given by c - 3ab + 2a 3 . ¢ (a , b, c) = (b - a 2 ) 3 / 2 The sequence .Jfi.(X - a 1 , X 2 - a 2 , X3 - a 3 ) is asymptotically mean-zero normal by the central limit theorem, provided EXf is finite. The value ¢ (a 1 , a2 , a3 ) is exactly the popu­ lation skewness. The function ¢ is differentiable at the point (a 1 , a 2 , a 3 ) and application of the delta method is straightforward. We can save work by noting that the sample skewness is location and scale invariant. With Yi = (Xi - a 1 ) j C5, the skewness can also be written as ¢ (Y , Y 2 , Y3). With A. = t.L 3 / C5 3 denoting the skewness of the underlying distribution, the Ys satisfy Example (Skewness).

z

( � ) (a, ( �

� y 2 1 """ N K + 3 y3 - A The derivative of ¢ at the point (0, 1 , A.) equals ( - 3 , - 3/.. / 2, 1 ) . Hence, if T possesses the normal distribution in the display, then Jfi(ln - A.) is asymptotically normal distributed with mean zero and variance equal to var(- 3 T1 - 3A.T2 j2 + T3 ) . If the underlying distribution is normal, then A. = f..L s = 0, K = 0 and f..L 6 /C5 6 = 15. In that case the sample skewness is asymptotically N (0, 6) -distributed. An approximate level a test for normality based on the sample skewness could be to reject normality if .Jfi l ln I > .J6 Zaf 2 · Table 3.2 gives the level of this test for different values of n . D 3 .2. Level of the test that rejects if .Jrl l ln I / .J6 exceeds the 0. 975 quantile of the normal distribution, in the case that the observations are normally distributed.

Table

n

Level

10 20 30 50

0.02 0.03 0.03 0.05

Note: Approximations based on simula­ tion of 10,000 samples.

Delta Method

30 3.2

Variance-Stabilizing Transformations

Given a sequence of statistics Tn with Jfi(Tn - 8) -!. N (0, 0" 2 (8)) for a range of values of e ' asymptotic confidence intervals for e are given by

These are asymptotically of level 1 - 2a in that the probability that e is covered by the interval converges to 1 - 2a for every e . Unfortunately, as stated previously, these intervals are useless, because of their dependence on the unknown e . One solution is to replace the unknown standard deviations 0" (8) by estimators. If the sequence of estimators is chosen consistent, then the resulting confidence interval still has asymptotic leve1 1 - 2a . Another approach is to use a variance-stabilizing transformation, which often leads to a better approximation. The idea is that no problem arises if the asymptotic variances 0" 2 (8) are independent of e . Although this fortunate situation i s rare, it i s often possible to transform the parameter into a different parameter 77 = ¢ (8), for which this idea can be applied. The natural estimator for 77 is cp (Tn). If cp is differentiable, then

=

For cp chosen such that ¢'(8)0" (8) 1 , the asymptotic variance is constant and finding an asymptotic confidence interval for 77 = cp (8) is easy. The solution ¢ (8)

=

f -0" (8)1- dB

is a variance-stabililizing transformation. variance stabililizing transformation. If it is well defined, then it is automatically monotone, so that a confidence interval for 77 can be transformed back into a confidence interval for e .

3.6 Example (Correlation). Let ( X1 , Y1 ) , . . . , (Xn , Yn) be a sample from a bivariate nor­ mal distribution with correlation coefficient p . The sample correlation coefficient is defined as

With the help of the delta method, it is possible to derive that Jfi(r - is asymptotically zero-mean normal, with variance depending on the (mixed) third and fourth moments of (X, Y) . This is true for general underlying distributions, provided the fourth moments exist. Under the normality assumption the asymptotic variance can be expressed in the correlation of X and Y. Tedious algebra gives

p)

It does not work very well to base an asymptotic confidence interval directly on this result.

Higher-Order Expansions

3.3

31

Table 3.3. Coverage probability of the asymptotic 95% confidence interval for the correlation coefficient, for two values ofn andfive different values of the true correlation p.

n 15 25

p

=

0

0.92 0.93

p

=

0.2

p

=

0.4

p

0.92 0.94

0.92 0.94

=

0.6

0.93 0.94

p

=

0.8

0.92 0.94

Note: Approximations based on simulation of 10,000 samples.

1

-1 .4

-1 . 0

-0. 8

rh -0. 2

Figure 3.1. Histogram of 1000 sample correlation coefficients, based on 1000 independent samples of the the bivariate normal distribution with c orrelation 0.6, and histogram of the arctanh of these values .

The transformation

¢(p) f 1 1 p2 dp 21 1og 11 + pp arctanh p is variance stabilizing. Thus, the sequence .Jri ( arctanh r -arctanh p) converges to a standard normal distribution for every p. This leads to the asymptotic confidence interval for the correlation coefficient p given by (tanh (arctanh r - Za I -Jfi) , tanh (arctanh r + Za I -Jfi)) . =

_

=

_

=

Table 3.3 gives an indication of the accuracy of this interval. Besides stabilizing the vari­ ance the arctanh transformation has the benefit of symmetrizing the distribution of the sample correlation coefficient (which is perhaps of greater importance), as can be seen in Figure 5.3. D

*3.3

Higher-Order Expansions

To package a simple idea in a theorem has the danger of obscuring the idea. The delta method is based on a Taylor expansion of order one. Sometimes a problem cannot be exactly forced into the framework described by the theorem, but the principle of a Taylor expansion is still valid.

Delta Method

32

In the one-dimensional case, a Taylor expansion applied to a statistic Tn has the form

Usually the linear term (Tn - e)¢>' (e) is of higher order than the remainder, and thus determines the order at which cf> (Tn) - cp (e) converges to zero: the same order as Tn - e . Then the approach of the preceding section gives the limit distribution of ¢> (Tn) - ¢> (e). If ¢>'(e) = 0, this approach is still valid but not of much interest, because the resulting limit distribution is degenerate at zero. Then it is more informative to multiply the difference ¢> (Tn) - ¢> (e) by a higher rate and obtain a nondegenerate limit distribution. Looking at the Taylor expansion, we see that the linear term disappears if cf>' (e) = 0, and we expect that the quadratic term determines the limit behavior of cp (Tn ) .

3.7 Example. Suppose that .jn X converges weakly to a standard normal distribution. Because the derivative of x r--+ cos x is zero at x = 0, the standard delta method of the preceding section yields that Jn (cos X - cos 0) converges weakly to 0. It should be concluded that Jn is not the right norming rate for the random sequence cos - A more informative statement is that -2n (cos X - converges in distribution to a chi -square distribution with one degree of freedom. The explanation is that 0) 2 (cos x) ;�=O · · · . 0)0 cos 0 cos

X 1.

1) + X - = (X - + �(X That the remainder term is negligible after multiplication with n can be shown along the same lines as the proof of Theorem 3.1. The sequence n X2 converges in law to a x � distribution by the continuous-mapping theorem; the sequence -2n (cos X - 1) has the same limit, by Slutsky's lemma. D

A more complicated situation arises if the statistic Tn is higher-dimensional with coor­ dinates of different orders of magnitude. For instance, for a real-valued function ¢> ,

If the sequences Tn,i - ei are of different order, then it may happen, for instance, that the linear part involving Tn,i - ei is of the same order as the quadratic part involving (Tn,j - ej ) 2 . Thus, it is necessary to determine carefully the rate of all terms in the expansion, and to rearrange these in decreasing order of magnitude, before neglecting the "remainder."

*3.4

Uniform Delta Method

Sometimes we wish to prove the asymptotic normality of a sequence .jn(¢> (Tn ) - ¢> (en)) for centering vectors en changing with n, rather than a fixed vector. If .jn(en - e) -* h for certain vectors e and h , then this can be handled easily by decomposing

Moments

3. 5

33

Several applications of Slutsky's lemma and the delta method yield as limit in law the vector tjJ� (T + h) - tjJ� (h ) = tjJ� (T), if T is the limit in distribution of y'ri(Tn - en) . For en � e at a slower rate, this argument does not work. However, the same result is true under a slightly stronger differentiability assumption on ¢.

Let tjJ : JRk f--+ JRm be a map defined and continuously diffe rentiable in a neighborhood of e. Let Tn be random vectors taking their values in the domain of ¢. lf rn (Tn - en) T for vectors en � e and numbers rn � oo, then rn ( t/J (Tn ) tjJ (en)) -v-+ ¢� (T). Moreover, the diffe rence between rn ( ¢ (Tn) - tjJ (en)) and ¢� ( rn (Tn - en)) converges to zero in probability.

3.8

Theorem.

-v-+

­

It suffices to prove the last assertion. Because convergence in probability to zero of vectors is equivalent to convergence to zero of the components separately, it is no loss of generality to assume that tjJ is real-valued. For 0 :::: t :::: and fixed h, define gn (t) = tjJ (en + th). For sufficiently large n and sufficiently small h, both en and en + h are in a ball around e inside the neighborhood on which tjJ is differentiable. Then gn : [0, f--+ lR is continuously differentiable with derivative g� (t) = ¢� + t h (h). By the mean -value theorem, gn - gn (0) = g� (�) for some 0 :=:: � :=:: In other words

Proof

1

1]

"

(1)

1.

By the continuity of the map e f--+ ¢� , there exists for every s > 0 a 8 > 0 such that ll tP� (h) - ¢� (h) I < s ll h ll for every li s - e 11 < 8 and every h. For sufficiently large n and llh l l < 8 / 2, the vectors en + �h are within distance 8 of e , so that the norm I R n (h) I of the right side of the preceding display is bounded by s ll h 11 . Thus, for any 1J > 0,

The first term converges to zero as n by choosing s small. •

� oo.

*3.5

The second term can be made arbitrarily small

Moments

So far we have discussed the stability of convergence in distribution under transformations. We can pose the same problem regarding moments: Can an expansion for the moments of tjJ (Tn) - tjJ (e) be derived from a similar expansion for the moments of Tn - e? In principle the answer is affirmative, but unlike in the distributional case, in which a simple derivative of tjJ is enough, global regularity conditions on tjJ are needed to argue that the remainder terms are negligible. One possible approach is to apply the distributional delta method first, thus yielding the qualitative asymptotic behavior. Next, the convergence of the moments of tjJ (Tn) - tjJ (e) (or a remainder term) is a matter of uniform integrability, in view of Lemma 2.20. If tjJ is uniformly Lipschitz, then this uniform integrability follows from the corresponding uniform integrability of Tn - e . If tjJ has an unbounded derivative, then the connection between moments of tjJ ( Tn) - tjJ (e) and Tn - e is harder to make, in general.

Delta Method

34

Notes

The Delta method belongs to the folklore of statistics. It is not entirely trivial; proofs are sometimes based on the mean-value theorem and then require continuous differentiability in a neighborhood. A generalization to functions on infinite-dimensional spaces is discussed in Chapter 20. PROBLEMS

1.

Find the j oint limit distribution of ( fL) , if and are based on a sample of size from a distribution with finite fourth moment. Under what condition on the underlying distribution are t-L) and asymptotically independent?

,Jfi(X - ,Jfi(S2 - a2)) X S2 n ,Jfi(X ,Jfi(S2 - a2) 2. Find the asymptotic distribution of ,Jn (r - p) if r is the correlation coefficient of a sample of n

bivariate vectors with finite fourth moments. (This is quite a bit of work. It helps to assume that the mean and the variance are equal to 0 and 1, respectively.)

3. Investigate the asymptotic robustness of the level of the t-test for testing the mean that rejects Ho : fL ::::: 0 if is larger than the upper a quantile of the tn - 1 distribution.

,JfiX IS

n - 1 L7= 1 (Xi - X)4 I S4 -

4. Find the limit distribution of the sample kurtosis kn = 3, and design an asymptotic level a test for normality based on kn . (Warning : At least 500 observations are needed to make the normal approximation work in this case.) 5. Design an asymptotic level a test for normality based on the sample skewness and kurtosis jointly. 6. Let be i.i.d. with expectation fL and variance converges in distribution if fL = 0 or fL i= 0.

X 1 , . . . , Xn 1 . Find constants such that a n (X� - bn) 7. Let X 1 , . . . , X n be a random sample from the Poisson distribution with mean e. Find a variance stabilizing transformation for the sample mean, and construct a confidence interval for e based on this.

X 1 , 1. . . , Xn be i.i.d. with expectation 1 and finite variance. Find the limit distribution of ,Jfi(X� - 1) . If the random variables are sampled from f that i s bounded and strictly 1 aoodensity positive in a neighborhood of zero, show that EIX� 1 for every n. (The density of Xn is bounded away from zero in a neighborhood of zero for every n.)

8. Let

=

4 Moment Estimators

The method of moments determines estimators by comparing sample and theoretical moments. Moment estimators are useful for their simplicity, although not always optimal. Maximum likelihood estimators for full ex­ ponentialfamilies are moment estimators, and their asymptotic normality can be proved by treating them as such. 4.1

Method of Moments

Let 1 , be a sample from a distribution that depends on a parameter e, ranging over some set 8. The method of moments consists of estimating e by the solution of a system of equations

X . . . , Xn

Pe

1- Ln fJ(Xt) Ee fj (X), n i=l =

j

=

1, . . . , k,

for given functions f1 , . . . , fk . Thus the parameter is chosen such that the sample moments (on the left side) match the theoretical moments. If the parameter is k-dimensional one usually tries to match k moments in this manner. The choices fJ (x) = x lead to the method of moments in its simplest form. Moment estimators are not necessarily the best estimators, but under reasonable condi­ tions they have convergence rate Jn and are asymptotically normal. This is a consequence of the delta method. Write the given functions in the vector notation f = (!I , . . . , fk ), and let e : 8 r-:>- �k be the vector-valued expectation e( F3 ) = Then the moment estimator en solves the system of equations

j

Pe f .

r?n f

=

1 Ln f (Xt )

-

n i=l

=

e( F3)

=

Pe f.

For existence of the moment estimator, it is necessary that the vector r?n f be in the range of the function e. If e is one-to-one, then the moment estimator is uniquely determined as en = e - 1 Wn f) and A

If r?n f is asymptotically normal and e - 1 is differentiable, then the right side is asymptoti­ cally normal by the delta method. 35

Moment Estimators

36

The derivative of e - 1 at e ( e0) is the inverse e�� 1 of the derivative of e at 80. Because the function e - 1 is often not explicit, it is convenient to ascertain its differentiability from the differentiability of e. This is possible by the inverse function theorem. According to this theorem a map that is (continuously) differentiable throughout an open set with nonsingular derivatives is locally one-to-one, is of full rank, and has a differentiable inverse. Thus we obtain the following theorem.

Suppose that e(e) = Pe f is one-to-one on an open set 8 c JRk and con­ tinuously differentiable at e0 with nonsingular derivative e�a . Moreover, assume that Pea I f 11 2 < oo. Then moment estimators en exist with probability tending to one and satisfy 4.1

Theorem.

�

Continuous differentiability at e0 presumes differentiability in a neighborhood and the continuity of e r--+ e� and nonsingularity of e�a imply nonsingularity in a neighborhood. Therefore, by the inverse function theorem there exist open neighborhoods U of e0 and V of Pea f such that e : U r--+ V is a differentiable bijection with a differentiable inverse e - 1 : V r--+ U . Moment estimators en = e - 1 (Pn f) exist as soon as Pn f E V, which happens with probability tending to by the law of large numbers. The central limit theorem guarantees asymptotic normality of the sequence Jn Wn f Pea f) . Next use Theorem on the display preceding the statement of the theorem. •

Proof.

1

3.1

For completeness, the following two lemmas constitute, if combined, a proof of the inverse function theorem. If necessary the preceding theorem can be strengthened somewhat by applying the lemmas directly. Furthermore, the first lemma can be easily generalized to infinite-dimensional parameters, such as used in the semiparametric models discussed in Chapter 25.

Let 8 c JRk be arbitrary and let e : 8 r--+ JRk be one-to-one and differentiable at a point e0 with a nonsingular derivative. Then the inverse e - 1 (defined on the range of e) is diffe rentiable at e(e0) provided it is continuous at e(e0). Proof. Write ry = e(e0) and l:!.. h = e - 1 (ry + h) - e - 1 (ry) . Because e - 1 is continuous at ry, we have that l:!.. h r--+ 0 as h r--+ 0. Thus 4.2

Lemma.

as h r--+ 0, where the last step follows from differentiability of e. The displayed equation can be rewritten as e�a ( I:!.. h) = h + o( ll l:!.. h II ) . By continuity of the inverse of e�a' this implies that

l:!.. h = e�� 1 (h) + o( ll i:!.. h ll ) . In particular, ll l:!.. h II ( + o( :::: I e�� 1 (h) I = 0 ( II h II) . Insert this in the displayed equation to obtain the desired result that i:!.. h = e�� 1 (h) + o( ll h ll ) . •

1 1) )

Let e : 8 r--+ JRk be defined and differentiable in a neighborhood of a point e0 and continuously diffe rentiable at e0 with a nonsingular derivative. Then e maps every 4.3

Lemma.

Exponential Families

4. 2

37

sufficiently small open neighborhood U of80 onto an open set V and e - 1 : V f---+ U is well defined and continuous. By assumption, e� -? A- 1 := e�0 as f) f---+ 80. Thus Il l - Ae � ll ::::; � for every e in a sufficiently small neighborhood U of 80 . Fix an arbitrary point ry 1 = e(f)I) from V = e(U) (where 81 E U). Next find an c > 0 such that ball(81 , c ) c U, and fix an arbitrary point ry with II ry - rJ1 II < 8 := � II A l l - 1 c . It will be shown that ry = e (f)) for some point e E ball(8 1 , c). Hence every ry E ball(ry 1 , 8) has an original in ball(81 , c). If e is one-to-one on U, so that the original is unique, then it follows that V is open and that e - 1 is continuous at ry1 . Define a function ¢ (8) = f) + A (ry - e (8)) . Because the norm of the derivative ¢� = I - Ae� is bounded by � throughout U, the map ¢ is a contraction on U . Furthermore, if 1 1 8 - 8 1 ll :S c,

Proof

Consequently, ¢ maps ball(81 , c ) into itself. Because ¢ is a contraction, it has a fixed point f) E ball(8 1 , c ) : a point with ¢ (8) = f) . By definition of ¢ this satisfies e (f)) = ry. Any other B with e ( B ) = ry is also a fixed point of ¢. In that case the difference B - f) = ¢ ( B ) - ¢ (8) has norm bounded by li B - 8 11 - This can only happen if B = e . Hence e is one-to-one throughout U . •

�

4.4 Example. Let X 1 , . . . , Xn be a random sample from the beta-distribution: The com­ mon density is equal to

x f---+

rca + fJ) x a -1 (1 - x) {3 - 1 lo<x
r (a)r ({J)

(a, {3 ) is the solution of the system of equations a Xn = Ea R X 1 = -- , a + fJ(a + l)a Xn2 - Ea ,f3 X 21 (a + fJ + l)(a + fJ) The righthand side is a smooth and regular function of (a, {3 ), and the equations can be

The moment estimator for

· �-"

solved explicitly. Hence, the moment estimators exist and are asymptotically normal. D

*4.2

Exponential Families

Maximum likelihood estimators in full exponential families are moment estimators. This can be exploited to show their asymptotic normality. Actually, as shown in Chapter 5, maximum likelihood estimators in smoothly parametrized models are asymptotically normal in great generality. Therefore the present section is included for the benefit of the simple proof, rather than as an explanation of the limit properties. Let X 1 , . . . , Xn be a sample from the k-dimensional exponential family with density

pe (x) = c (f)) h(x)

eeT t(x)_

Moment Estimators

38

Thus h and t = (t 1 , . . . , tk ) are known functions on the sample space, and the family is given in its natural parametrization. The parameter set e must be contained in the natural parameter space for the family. This is the set of e for which Pe can define a probability density. If fJ., is the dominating measure, then this is the right side in

It is a standard result (and not hard to see) that the natural parameter space is convex. It is usually open, in which case the family is called "regular." In any case, we assume that the true parameter is an inner point of 8. Another standard result concerns the smoothness of the function e f--+ c ( 8 ) , or rather of its inverse. (For a proof of the following lemma, see [ 1 00, p. 59] or [17, p. 39] . )

4.5 Lemma. Thefunction () f--+ j h (x ) e e rt(x) dfl,(X) is analytic on the set {() E Ck : Re (3 8 }. Its derivatives can be found by differentiating (repeatedly) under the integral sign: 0

E

for any natural numbers p and i 1 + · · · + ik = p. The lemma implies that the log likelihood £0 (x ) = log p0 (x ) can be differentiated (in­ finitely often) with respect to e . The vector of partial derivatives (the score function) satisfies

.

c c

£0 (x ) = - (8) + t (x) = t (x) - Ee t (X) .

Here the second equality is an example of the general rule that score functions have zero means. It can formally be established by differentiating the identity J Pe dfl, 1 under the integral sign: Combine the lemma and the Leibniz rule to see that =

� i

� pe dfl, = ae

f a aec({)i ) h (X) ee

T t(x) dfl,(X)

+

f c({) ) h(X) ti (X) ee

T t (x) dfl,(X)

.

The left side is zero and the equation can be rewritten as 0 = cj c (e ) + Ee t ( X ) . It follows that the likelihood equations L l e (Xi ) = 0 reduce to the system of k equations 1

n

-n L t ( Xi ) = Ee t (X) . i=l

Thus, the maximum likelihood estimators are moment estimators . Their asymptotic prop­ erties depend on the function e ( 8 ) = Ee t (X) , which is very well behaved on the interior of the natural parameter set. By differentiating Ee t (X) under the expectation sign (which is justified by the lemma), we see that its derivative matrices are given by

e� = Cove t (X) . The exponential family is said to be of full rank if no linear combination L � =l J... jtj (X) is constant with probability 1 ; equivalently, if the covariance matrix of t (X) is nonsingular. In

4. 2

Exponential Families

39

view of the preceding display, this ensures that the derivative e� is strictly positive-definite throughout the interior of the natural parameter set. Then e is one-to-one, so that there exists at most one solution to the moment equations. (Cf. Problem 4.6.) In view of the expression for le , the matrix - ne � is the second-derivative matrix (Hessian) of the log likelihood L 7= 1 ie (X d . Thus, a solution to the moment equations must be a point of maximum of the log likelihood. A solution can be shown to exist (within the natural parameter space) with probability 1 if the exponential family is "regular," or more generally "steep" (see [ 17]); it is then a point of absolute maximum of the likelihood. If the true parameter is in the interior of the parameter set, then a (unique) solution en exists with probability tending to 1 as n f-+ 00, in any case, by Theorem 4. 1 . Moreover, this theorem shows that the sequence ,Jfi(en - 80) is asymptotically normal with covariance matrix

eg1 0 -1 Cove0 t (X) ( eg1 0 - l ) T = ( Cove0 t (X)) - 1 .

So far we have considered an exponential family in standard form. Many examples arise in the form

pe (x ) = d (8) h (x) e Q (B/ t (x ) ,

(4.6)

where Q = ( Q 1 , , Q k ) is a vector-valued function. If Q is one-to-one and a maximum likelihood estimator en exists, then by the invariance of maximum likelihood estimators under transformations ' Q (en) is the maximum likelihood estimator for the natural parameter Q (8) as considered before. If the range of Q contains an open ball around Q (80) , then the preceding discussion shows that the sequence .Jfi ( Q ( en ) - Q (80) ) is asymptotically normal. It requires another application of the delta method to obtain the limit distribution of .J1i (en - 80) . As is typical of maximum likelihood estimators, the asymptotic covariance matrix is the inverse of the Fisher information matrix .

.

•

Ie

=

.

.

Ee ie (X)ie (X) T .

Let 8 c ffi.k be open and let Q : 8 f-+ ffi.k be one-to-one and continuously differentiable throughout 8 with nonsingular derivatives. Let the (exponential) family of densities Pe be given by (4. 6) and be offull rank. Then the likelihood equations have a unique solution en with probability tending to 1 and ,jfi( en - 8) � N (O, Ie-1) for every e. 4.6

Theorem.

According to the inverse function theorem, the range of Q is open and the inverse map Q- 1 is differentiable throughout this range. Thus, as discussed previously, the delta method ensures the asymptotic normality. It suffices to calculate the asymptotic covariance matrix. By the preceding discussion this is equal to

Proof.

By direct calculation, the score function for the model is equal to l e (x ) dj d (8) + ( Q�) T t (x) . As before, the score function has mean zero, so that this can be rewritten as le (x) = ( Q�) T ( t (x) - Ee t (X) ) . Thus, the Fisher information matrix equals Ie = ( Q�) T Cove t (X) Q� . This is the inverse of the asymptotic covariance matrix given in the preceding display. •

Moment Estimators

40

Not all exponential families satisfy the conditions of the theorem. For instance, the normal N (8 , 8 2 ) family is an example of a "curved exponential family." The map Q (8) = (8 - 2 , 8- 1 ) (with t (x) = ( - x 2 / 2, x)) does not fill up the natural parameter space of the normal location-scale family but only traces out a one-dimensional curve. In such cases the result of the theorem may still hold. In fact, the result is true for most models with "smooth parametrizations," as is seen in Chapter 5. However, the "easy" proof of this section is not valid. PROBLEMS 1.

X . . . , Xn

[ -e, e].

Let 1 , be a sample from the uniform distribution on Find the moment estimator of based on X 2 . Is it asymptotically normal? Can you think of an estimator for e that converges faster to the parameter?

e

X 1 , . . . , Xn

2. Let be a sample from a density pe and f a function such that e (e ) = Ee f (X) is differentiable with e ' = Ee t e f for £ e = log pe . (i) Show that the asymptotic variance of the moment estimator based on f equals vare (f) I . cove (f, £ e ) 2 . (ii) Show that this is bigger than Ie- l with equality for all if and only if the moment estimator is the maximum likelihood estimator.

(e)

(X) (X)

e

(iii) Show that the latter happens only for exponential family members .

3. To what extent does the result of Theorem

4. 1 require that the observations are i.i.d.?

4. Let the observations be a sample of size n from the N (tL , a 2 ) distribution. Calculate the Fisher information matrix for the parameter = (tL , a 2 ) and its inverse. Check directly that the maximum 1 likelihood estimator is asymptotically normal with zero mean and covariance matrix I() .

e

5. Establish the formula e(; = Cove t (X) by differentiating e (e ) = Ee t (X) under the integral sign. (Differentiating under the integral sign is justified by Lemma 4.5, because Ee t is the first derivative of

(X)

c(e)- 1 .)

6. Suppose a function e : 8 r+ lltk is defined and continuously differentiable on a convex subset 8 C lltk with strictly positive-definite derivative matrix. Then e has at most one zero in B . (Consider the function g(A.) = 2 ) e (A. l + (1 - A. ) 2 ) for given =f. 2 and 0 :S A. :S 1. I f g(O) = g(l) = 0, then there exists a point A.o with g' (A.o) = 0 by the mean-value theorem.)

(e l - e T e

e

e1 e

5 M- and Z -Estimators

This chapter gives an introduction to the consistency and asymptotic normality of M -estimators and Z-estimators. Maximum likelihood esti­ mators are treated as a special case.

5.1

Introduction

Suppose that we are interested in a parameter (or "functional") e attached to the distribution of observations X 1 , . . . , Xn . A popular method for finding an estimator en = en (X 1 , . . . , Xn) is to maximize a criterion function of the type (5. 1 ) Here m e : X � lR are known functions . An estimator maximizing Mn (e) over 8 is called an M -estimator. In this chapter we investigate the asymptotic behavior of sequences of M -estimators. Often the maximizing value is sought by setting a derivative (or the set of partial deriva­ tives in the multidimensional case) equal to zero. Therefore, the name M -estimator is also used for estimators satisfying systems of equations of the type

n

Wn (e)

1 = -:2: 1/re (X i ) = 0. n i= l

(5.2)

Here 1./le are known vector-valued maps. For instance, if e is k-dimensional, then 1/re typically has k coordinate functions 1/re = (1/re , 1 , , 1/re , k ), and (5.2) is shorthand for the system of equations .

•

.

n

L Yre,j (Xi) = 0, j = 1 , 2, . . . , k. i= l Even though in many examples 1/re ,j is the j th partial derivative of some function me , this

is irrelevant for the following. Equations, such as (5.2), defining an estimator are called estimating equations and need not correspond to a maximization problem. In the latter case it is probably better to call the corresponding estimators Z-estimators (for zero), but the use of the name M -estimator is widespread. 41

42

M- and Z-Estimators

Sometimes the maximum of the criterion function Mn is not taken or the estimating equation does not have an exact solution. Then it is natural to use as estimator a value that almost maximizes the criterion function or is a near zero. This yields approximate M -estimators or Z-estimators. Estimators that are sufficiently close to being a point of maximum or a zero often have the same asymptotic behavior. An operator notation for taking expectations simplifies the formulas in this chapter. We write P for the marginal law of the observations X 1 , . . . , Xn , which we assume to be identically distributed. Furthermore, we write P f for the expectation E f (X) = J f d P and abbreviate the average n - 1 :L 7= 1 f(X J to rl'n f· Thus rl'n is the empirical distribution: the (random) discrete distribution that puts mass 1 In at every of the observations X 1 , . . . , Xn . The criterion functions now take the forms We also abbreviate the centered sums n - 1 1 2 :L 7= 1 ( f(Xi) - Pf) to G n f, the empirical process at f.

5.3 Example (Maximum likelihood estimators). Suppose X 1 , . . . , Xn have a common density Pe . Then the maximum likelihood estimator maximizes the likelihood TI 7= 1 Pe (X i ), or equivalently the log likelihood (J r--+

n L log pe (Xi ). i= 1

Thus, a maximum likelihood estimator is an M-estimator as in (5. 1 ) with me = log pe . If the density is partially differentiable with respect to (] for each fixed x, then the maximum likelihood estimator also solves an equation of type (5.2), with lJ;e equal to the vector of partial derivatives le , j = a I aej log Pe . The vector-valued function le is known as the score function of the model. The definition (5. 1 ) of an M-estimator may apply in cases where (5.2) does not. For instance, if X 1 , . . . , Xn are i.i.d. according to the uniform distribution on [0, (]], then it makes sense to maximize the log likelihood (]

r--+

n

L (log 1 [O, e J (Xi ) - log (] ) . i= 1

=

(Define log 0 -oo.) However, this function is not smooth in (] and there exists no natural version of (5.2). Thus, in this example the definition as the location of a maximum is more fundamental than the definition as a zero. D

5.4 Example (Location estimators). Let X 1 , . . . , Xn be a random sample of real-valued observations and suppose we want to estimate the location of their distribution. "Location" is a vague term; it could be made precise by defining it as the mean or median, or the center of symmetry of the distribution if this happens to be symmetric. Two examples of location estimators are the sample mean and the sample median. Both are Z-estimators, because they solve the equations

n n L (Xi - (]) = 0 ; and L sign( X i - (]) = 0, i= 1 i=1

5. 1

43

Introduction

respectively. t Both estimating equations involve functions of the form 1/J (x - 8) for a function 1/J that is monotone and odd around zero. It seems reasonable to study estimators that solve a general equation of the type n

2: 1/J ( X i - 8) = 0. i=l We can consider a Z-estimator defined by this equation a "location" estimator, because it has the desirable property of location equivariance. If the observations Xi are shifted by a fixed amount a, then so is the estimate: iJ + a solves 2::: 7= 1 1/f(X i + a - 8) = 0 if iJ solves the original equation. Popular examples are the Huber estimators corresponding to the functions if X :S -k, if l x l :S k, if X 2: k . The Huber estimators were motivated by studies in robust statistics concerning the influ­ ence of extreme data points on the estimate. The exact values of the largest and smallest observations have very little influence on the value of the median, but a proportional influ­ ence on the mean. Therefore, the sample mean is considered nonrobust against outliers. If the extreme observations are thought to be rather unreliable, it is certainly an advantage to limit their influence on the estimate, but the median may be too successful in this respect. Depending on the value of k, the Huber estimators behave more like the mean (large k) or more like the median (small k) and thus bridge the gap between the nonrobust mean and very robust median. Another example are the quantiZes. A p th sample quantile is roughly a point e such that pn observations are less than e and (1 - p)n observations are greater than e . The precise definition has to take into account that the value pn may not be an integer. One possibility is to call a pth sample quantile any iJ that solves the inequalities n

-1

<

2: { ( 1 - p) 1 { X i i=l

<

8 } - p 1 {Xi

>

8 })

<

1.

(5.5)

This is an approximate M-estimator for 1/f (x) = 1 - p, 0, -p if x < 0, x = 0, or x > 0, respectively. The "approximate" refers to the inequalities: It is required that the value of the estimating equation be inside the interval ( - 1 , 1), rather than exactly zero. This may seem a rather wide tolerance interval for a zero. However, all solutions turn out to have the same asymptotic behavior. In any case, except for special combinations of p and n, there is no hope of finding an exact zero, because the criterion function is discontinuous with jumps at the observations. (See Figure 5. 1 .) If no observations are tied, then all jumps are of size one and at least one solution iJ to the inequalities exists. If tied observations are present, it may be necessary to increase the interval ( 1 1 ) to ensure the existence of solutions. Note that the present 1/J function is monotone, as in the previous examples, but not symmetric about zero (for p =1- 1/2). -

t

,

The sign-function is defined as sign(x) - 1, 0, 1 if x < 0, x 0 or x > 0, respectively. Also x+ means x v 0 max(x , 0) . For the median we assume that there are no tied observations (in the middle). =

=

=

M- and Z-Estimators

44 l!)

0 C\l 0

0

0 0

0 � '

4

6

10

8

12

-2

-1

0

2

3

Figure 5.1. The functions e H> (e) for the 80% quantile and the Huber estimator for samples of size 15 from the gamma(S , 1) and standard normal distribution, respectively.

Wn

All the estimators considered so far can also be defined as a solution of a maximization problem. Mean, median, Huber estimators, and quantiles minimize 2:7= 1 m (X i - e) for m equal to x 2 , l x l , x 2 l l x l:s k + (2k l x l - k 2 ) 1 1 x l> k and ( 1 - p)x - + px + , respectively. D

5.2

Consistency

If the estimator is used to estimate the parameter e , then it is certainly desirable that the sequence converges in probability to e . If this is the case for every possible value of the parameter, then the sequence of estimators is called asymptotically consistent. For instance, the sample mean X is asymptotically consistent for the population mean EX (provided the population mean exists). This follows from the law of large numbers. Not surprisingly this extends to many other sample characteristics. For instance, the sam­ ple median is consistent for the population median, whenever this is well defined. What can be said about M -estimators in general? We shall assume that the set of possible parameters is a metric space, and write for the metric. Then we wish to prove that e0) 0 for some value e0, which depends on the underlying distribution of the observations. Suppose that the M -estimator maximizes the random criterion function

en

en

n

d

d(Bn, �

Bn

Clearly, the "asymptotic value" of depends on the asymptotic behavior of the functions Under suitable normalization there typically exists a deterministic "asymptotic criterion function" e f--+ M (e) such that

Bn

Mn.

every e .

(5 .6)

For instance, if is an average of the form JID m e as in (5 . 1 ), then the law of large numbers gives this result with M (e) = Pm11 , provided this expectation exists. It seems reasonable to expect that the maximizer of converges to the maximizing value e0 of M. This is what we wish to prove in this section, and we say that is (asymptotically) consistent for e0. However, the convergence (5.6) is too weak to ensure

Mn(e)

n en Mn

Bn

5. 2

45

Consistency

Figure 5.2. Example of a function whose point of maximum is not well separated.

the convergence of fJn. Because the value fJn depends on the whole function e r--+ Mn (e) , an appropriate form of "functional convergence" of Mn to M is needed, strengthening the pointwise convergence (5.6). There are several possibilities. In this section we first discuss an approach based on uniform convergence of the criterion functions. Admittedly, the assumption of uniform convergence is too strong for some applications and it is sometimes not easy to verify, but the approach illustrates the general idea. Given an arbitrary random function e r--+ Mn (e) , consider estimators fJn that nearly maximize Mn , that is,

MnCfJn) :::: supe Mn (e) - Op ( l ) . Then certainly MnCfJn) > Mn (eo) - op (l), which turns out to be enough to ensure con­ sistency. It is assumed that the sequence Mn converges to a nonrandom map M: 8 r--+ JR. Condition (5. 8) of the following theorem requires that this map attains its maximum at a · unique point e0, and only parameters close to e0 may yield a value of M (e) close to the maximum value M(e0) . Thus, e0 should be a well-separated point of maximum of M . Figure 5 .2 shows a function that does not satisfy this requirement.

5.7 Theorem. Let Mn be random functions and let M be a fixed function of e such that for every E > Qt

supiMn (e) - M(e) l � 0,

0 E8

sup

e : d(O,Oo ) ?:: c:

M (e) < M(e0).

Then any sequence of estimators fJn with MnCfJn) bility to eo. t

:=::

(5.8)

Mn (eo) - op (l) converges in proba­

Some of the expressions in this display may be nonmeasurable. Then the probability statements are understood in terms of outer measure.

46

M- and Z-Estimators

By the property of en , we have Mn (e n) � Mn (f3o) - op ( 1 ) . Because the uniform convergence of Mn to M implies the convergence of Mn (610) � M(610) , the right side equals M(610) - op ( 1 ) . It follows that Mn ( 8 n) � M (610) - op ( 1 ) , whence

Proof

M( &o) - M (8 n) :::: Mn ( e n ) - M( e n ) o p ( ) p :=:: sup I Mn - M l ( & ) o p (1 )----+ 0. e by the first part of assumption (5 .8). By the second part of assumption (5. 8), there exists for every fl > 0 a number 17 > 0 such that M( & ) < M(610) - 17 for every e with d( & , 610) � fl . Thus, the event d ( e n , 610) � fl } is contained in the event MC 8 n) < M(610) - 17}. The probability of the latter event converges to 0, in view of the preceding display. •

+ l +

{

{

Instead of through maximization, an M -estimator may be defined as a zero of a criterion function e � \lin ( & ) . It is again reasonable to assume that the sequence of criterion functions converges to a fixed limit:

Then it may be expected that a sequence of (approximate) zeros of \lin converges in prob­ ability to a zero of \ll . This is true under similar restrictions as in the case of maximizing M estimators. In fact, this can be deduced from the preceding theorem by noting that a zero of \lin maximizes the function 61 � - I \lin (61) II ·

5.9 Theorem. Let \lin be random vector-valued functions and let \II be a fixed vector­ valued function of& such that for every f:l > 0 supe E e I \lin (& ) - \II ( & ) I infe : d(e,eo)�c l 'l1(&) 1 > 0 =

� 0, 11 \ll (&o ) l -

Then any sequence of estimators e n such that \lln ( e n) = O p ( 1 ) converges in probability to &o . This follows from the preceding theorem, on applying it to the functions Mn (61) = & - I \lin ( ) I and M (& ) = - 1 \ll ( &) l · •

Proof

The conditions of both theorems consist of a stochastic and a deterministic part. The deterministic condition can be verified by drawing a picture of the graph of the function. A helpful general observation is that, for a compact set 8 and continuous function M or \II , uniqueness of 610 as a maximizer or zero implies the condition. (See Problem 5 .27.) For Mn ( & ) or \lin ( & ) equal to averages as in (5. 1) or (5. 2) the uniform convergence required by the stochastic condition is equivalent to the set of functions {me : e E 8} or {1/fe f 61 E 8, j = 1 , . . . , k} being Glivenko-Cantelli. Glivenko-Cantelli classes of functions are discussed in Chapter 19. One simple set of sufficient conditions is that 8 be compact, that the functions e � m e (x) or e � 1/fe (x) are continuous for every x, and that they are dominated by an integrable function. Uniform convergence of the criterion functions as in the preceding theorems is much stronger than needed for consistency. The following lemma is one of the many possibilities to replace the uniformity by other assumptions.

5. 2

Consistency

47

Let 8 be a subset of the real line and let Wn be random functions and \ll a fixed function of e such that Wn (e) � w (e) in probability for every e. Assume that each map e r--:>- Wn (e) is continuous and has exactly one zero {J n, or is nondecreasing with Wn ( {J n ) = o p (1). Let eo be a point such that \ll (eo - .s) < 0 < \ll ceo + .s) for every .s > 0. p Then en � eo.

5.10

Lemma.

A

Proof

If the map e

r--:>-

\lln Ce) is continuous and has a unique zero at {J n , then

The left side converges to one, because Wn Ceo ± .s) � \ll Ceo ± .s) in probability. Thus the right side converges to one as well, and Bn is consistent. If the map e r--:>- 'lln (e) is nondecreasing and {J n is a zero, then the same argument is valid. More generally, if e r--:>- Wn (e) is nondecreasing, then Wn (eo - .s) < -ry and {J n ::::: eo - .s imply Wn ({J n) < -ry, which has probability tending to zero for every rJ > 0 if {J n is a near zero. This and a similar argument applied to the right tail shows that, for every .s, rJ > 0,

For 2ry equal to the smallest of the numbers -\ll (eo - .s) and \ll (eo + .s) the left side still converges to one. • 5.11

Example (Median).

The sample median 8 n is a (near) zero of the map e

n - 1 L7= 1 sign( Xi - e). By the law of large numbers, Wn (e)

r--:>-

Wn (e)

=

!,. w (e) = E sign(X - e) = P(X > e) - P(X < e ) ,

for every fixed e . Thus, we expect that the sample median converges i n probability to a point e0 such that P(X > e0) = PCX < e0) : a population median. This can be proved rigorously by applying Theorem 5.7 or 5.9. However, even though the conditions of the theorems are satisfied, they are not entirely trivial to verify. (The uniform convergence of Wn to \ll is proved essentially in Theorem 19. 1) In this case it is easier to apply Lemma 5 . 10. Because the functions e r--:>- Wn (e) are nonincreasing, it follows that {J n eo provided that w (e0 - .s) > 0 > w (e0 + .s) for every .s > 0. This is the case if the population median is unique: P(X < e0 - .s) < < P(X < e0 + .s) for all .s > 0. D

!,.

�

*5.2.1

Wald's Consistency Proof

Consider the situation that, for a random sample of variables X 1 , . . . , Xn , M (e)

=

Pme .

In this subsection we consider an alternative set of conditions under which the maximizer Bn of the process Mn converges in probability to a point of maximum e0 of the function M. This "classical" approach to consistency was taken by Wald in 1949 for maximum likelihood estimators. It works best if the parameter set 8 is compact. If not, then the argument must

48

M- and Z -Estimators

be complemented by a proof that the estimators are in a compact set eventually or be applied to a suitable compactification of the parameter set. Assume that the map e f--+ me (x) is upper-semicontinuous for almost all X : For every e a.s . . lim sup men (x) ::: me (x) ' (5. 1 2) en -+ e (The exceptional set of x may depend on 8 .) Furthermore, assume that for every sufficiently small ball U C 8 the function x r--+ sup e EU me (x) is measurable and satisfies P sup me < oo . (5 . 1 3) e EU Typically, the map 8 r--+ Pme has a unique global maximum at a point 80, but we shall allow multiple points of maximum, and write 8o for the set { 8o E 8: Pme0 = supe Pme } of all points at which M attains its global maximum. The set 80 is assumed not empty. The maps me : X r--+ R are allowed to take the value - oo , but the following theorem assumes implicitly that at least Pme0 is finite.

5.14 Theorem. Let 8 r--+ me (x) be upper-semicontinuousfor almost all x and let (5. 1 3) be satisfied. Thenfor any estimators {J n such that Mn (en) :::: Mn (8o) - o p (1)for some 8o E 8o, for every E > 0 and every compact set K C 8, If the function 8 r--+ Pme is identically -oo, then 8o = 8, and there is nothing to prove. Hence, we may assume that there exists 80 E 80 such that Pme0 > - oo , whence P lme0 I < oo by (5. 1 3). Fix some e and let U1 -!- 8 be a decreasing sequence of open balls around 8 of diameter converging to zero. Write m u (x) for supe EU me (x) . The sequence mu1 is decreasing and greater than me for every l . Combination with (5 . 1 2) yields that mu1 -!- me almost surely. In view of (5. 1 3), we can apply the monotone convergence theorem and obtain that Pmu1 -!- Pme (which may be - oo ) . For 8 ¢ 80, we have Pme < Pme0 • Combine this with the preceding paragraph to see that for every 8 ¢ 80 there exists an open ball Ue around 8 with Pmu8 < Pme0 • The set B = {e E K : d (8 , 80) :::: E } is compact and is covered by the balls {Ue : 8 E B}. Let Ue1 , • • • , Uep be a finite subcover. Then, by the law of large numbers, as sup Pnme :=: sup Pnm u8 ----;.. sup Pmu8 < Pme0 • e EB j j =l . ... , p

Proof.

1

1

If {J n E B , then SUPe EB JPlnme is at least JPlnm(jn ' which by definition of en is at least JPlnmeo o p ( 1 ) = Pme0 - op (l), by the law of large numbers. Thus {en

E

B}

c

{ esupEB JPlnme :::: Pme0 - o ( 1 ) } . p

In view of the preceding display the probability of the event on the right side converges to zero as n ----;.. oo . • Even in simple examples, condition (5 . 1 3) can be restrictive. One possibility for relax­ ation is to divide the n observations in groups of approximately the same size. Then (5 . 1 3)

49

Consistency

5. 2

may be replaced by, for some k and every k :::; l < 2k, l

L

P 1 sup me (xi ) < e EU i= l

(5. 15)

oo .

Surprisingly enough, this simple device may help. For instance, under condition (5. 1 3) the preceding theorem does not apply to yield the asymptotic consistency of the maximum likelihood estimator of (fL , O") based on a random sample from the N (fL, 0" 2 ) distribution (unless we restrict the parameter set for O" ), but under the relaxed condition it does (with k = 2). (See Problem 5 .25.) The proof of the theorem under (5 . 1 5) remains almost the same. Divide the n observations in groups of k observations and, possibly, a remainder group of l observations ; next, apply the law of large numbers to the approximately n / k group sums.

5.16 Example (Cauchy likelihood). The maximum likelihood estimator for e based on a random sample from the Cauchy distribution with location e maximizes the map e r+ JPlnme for me (x)

=

- log ( 1 + (x - e) 2 ) .

The natural parameter set lR is not compact, but we can enlarge it to the extended real line, provided that we can define me in a reasonable way for e = ± oo . To have the best chance of satisfying (5. 1 3), we opt for the minimal extension, which in order to satisfy (5 . 1 2) is m _00 (x)

=

lim sup me (x) e f-+- 00

= - oo ;

m00 (x)

=

lim sup me (x) e �-+oo

= - oo .

These infinite values should not worry us: They are permitted in the preceding theorem. Moreover, because we maximize e r+ Pnme , they ensure that the estimator {J n never takes the values ±oo, which is excellent. We apply Wald's theorem with 8 = JR, equipped with, for instance, the metric d (er , e2 ) = I arctg e1 - arctg e2 1 . Because the functions e r+ me (x) are continuous and nonpositive, the conditions are trivially satisfied. Thus, taking K = JR, we obtain that d (en , 80) !,. 0. This conclusion is valid for any underlying distribution P of the observations for which the set 80 is nonempty, because so far we have used the Cauchy likelihood only to motivate me . To conclude that the maximum likelihood estimator in a Cauchy location model is con­ sistent, it suffices to show that 80 = { e0} if P is the Cauchy distribution with center e0. This follows most easily from the identifiability of this model, as discussed in Lemma 5.35. 0

5.17 Example (Current status data). Suppose that a "death" that occurs at time T is only observed to have taken place or not at a known "check-up time" C . We model the obser­ vations as a random sample X 1 , , Xn from the distribution of X = (C, 1 { T :::; C}) , where T and C are independent random variables with completely unknown distribution functions F and G , respectively. The purpose is to estimate the "survival distribution" 1 - F. If G has a density g with respect to Lebesgue measure /.. , then X = (C, £.. ) has a density •

P F (c , 8)

=

•

•

(8 F ( c) + ( 1 - 8) ( 1 - F) (c) ) g (c)

M- and Z-Estimators

50

with respect to the product of A. and counting measure on the set {0, 1 } . A maximum like­ lihood estimator for F can be defined as the distribution function F that maximizes the likelihood F

c-+

n

n ( � i F ( Ci ) + ( 1 - � i ) ( 1 - F ) ( Ci ) ) i =l

over all distribution functions on [0, oo) . Because this only involves the numbers F ( CI ) , . . . , F ( Cn ) , the maximizer of this expression is not unique, but some thought shows that there is a unique maximizer F that concentrates on (a subset of) the observation times C 1, . . . , Cn . This is commonly used as an estimator. We can show the consistency of this estimator by Wald's theorem. By its definition F maximizes the function F c-+ JlDn log p F, but the consistency proof proceeds in a smoother way by setting

2p m F = log FP FF__ = log F---=-+ F F P Po PC + o ) /2 Because the likelihood is bigger at F than it is at � F + � Fo , it follows that JlD nm fr 2:: 0 = J!DnmFo · (It is not claimed that F maximizes F c-+ J!DnmF; this is not true.) Condition (5. 1 3) is satisfied trivially, because mF _:::: log 2 for every F. We can equip the _..:::.__

-

--

set of all distribution functions with the topology of weak convergence. If we restrict the parameter set to distributions on a compact interval [ 0, r], then the parameter set is compact by Prohorov's theorem. t The map F c-+ mF ( c , 8) is continuous at F, relative to the weak topology, for every (c, 8) such that c is a continuity point of F. Under the assumption that G has a density, this includes almost every (c, 8), for every given F. Thus, Theorem 5 . 14 shows that F n converges under Fo in probability to the set :F0 of all distribution functions that maximize the map F c-+ PFo mF, provided Fo E Fa . This set always contains Fo , but it does not necessarily reduce to this single point. For instance, if the density g is zero on an interval [a , b], then we receive no information concerning deaths inside the interval [a , b ] , and there can be no hope that F n converges to F0 on [a , b] . In that case, F0 is not "identifiable" on the interval [a , b ] . We shall show that :F0 i s the set o f all F such that F = Fo almost everywhere according to G . Thus, the sequence F n is consistent for F0 "on the set of time points that have a positive probability of occurring." Because p F = p Fo under PF if and only if F = Fo almost everywhere according to G , it suffices to prove that, for every pair ofprobability densities p and p0, Po log 2 pI (p+ p0) _:::: 0 with equality if and only if p = p0 almost surely under P0. If P0 (p = 0) > 0, then log 2p I (p + p0) = - oo with positive probability and hence, because the function is bounded above, Po 1og 2p l (p + p0) = -oo. Thus we may assume that Po (p = 0) = 0. Then, with f(u) = -u log( � + �u), 0

2p Po log (p + o ) P

t

=

( ) ( )

Po _:::: f P Po Pf P P

=

f( l) = 0,

Alternatively, consider all probability distributions on the compactification [0, oo] again equipped with the weak topology.

5.3

51

Asymptotic Normality

by Jensen's inequality and the concavity of f, with equality only if p0j p under P, and then also under P0. This completes the proof. D

5.3

=

1 almost surely

Asymptotic Normality

Suppose a sequence of estimators en is consistent for a parameter e that ranges over an open subset of a Euclidean space. The next question of interest concerns the order at which the discrepancy en - e converges to zero. The answer depends on the specific situation, but for estimators based on n replications of an experiment the order is often n - 1 1 2 . Then multiplication with the inverse of this rate creates a proper balance, and the sequence .J1i (en - e) converges in distribution, most often a normal distribution. This is interesting from a theoretical point of view. It also makes it possible to obtain approximate confidence sets . In this section we derive the asymptotic normality of M -estimators. We can use a characterization of M -estimators either by maximization or by solving estimating equations. Consider the second possibility. Let X 1 , . . . , Xn be a sample from some distribution P, and let a random and a "true" criterion function be of the form:

\ll (e) = Pl/fe . Assume that the estimator en is a zero of 'lin and converges in probability to a zero eo of \ll . Because en ---+ eo, it makes sense to expand 'lln (en) in a Taylor series around eo. Assume for simplicity that e is one-dimensional. Then where Bn is a point between en and eo. This can be rewritten as r:: A

- .fo \lln (eo) (5. 1 8) A 'lin (eo) 21 (en - eo) 'lin (en) If Pljf� is finite, then the numerator -.fo'lln (e0) = -n - 1 1 2 L 1/fe0 (X i ) is asymptotically normal by the central limit theorem. The asymptotic mean and variance are Pl/fe0 = \ll (eo) = 0 and P1/JJ0 , respectively. Next consider the denominator. The first . term �n (eo) . p is an average and can be analyzed by the law of large numbers : 'lln (eo) ---+ Pl/f e0 , provided the expectation exists. The second term in the denominator is a product of en - e = o p ( 1) and -if; n (B n) and converges in probability to zero under the reasonable condition that �n (fJ n) v ft

(en - eo) =

•

+

••

-

0

(which is also an average) is Op ( l ) . Together with Slutsky's lemma, these observations yield

(5. 19) The preceding derivation can be made rigorous by imposing appropriate conditions, often called "regularity conditions." The only real challenge is to show that �n CfJn) = Op (1) (see Problem 5 .20 or section 5.6). The derivation can be extended to higher-dimensional parameters. For a k-dimensional parameter, we use k estimating equations. Then the criterion functions are maps 'lin : Rk r--+

M- and Z-Estimators

52

IRk and the derivatives �n (610) are (k x k)-matrices that converge to the (k x k) matrix P� eo with entries p a I ae j'l/feo , i . The final statement becomes (5.20) Here the invertibility of the matrix P � eo is a condition. In the preceding derivation it is implicitly understood that the function e f-+ 1/Je (x) possesses two continuous derivatives with respect to the parameter, for every x . This is true in many examples but fails, for instance, for the function 1/Je (x) = sign(x - 61), which yields the median. Nevertheless, the median is asymptotically normal. That such a simple, but important, example cannot be treated by the preceding approach has motivated much effort to derive the asymptotic normality of M -estimators by more refined methods . One result is the following theorem, which assumes less than one derivative (a Lipschitz condition) instead of two derivatives.

For each e in an open subset ofEuclidean space, let x f-+ 1/Je (x) be a mea­ surable vector-valued function such . that, for every 61 1 and 612 in a neighborhood of Bo and . a measurable function 1/J with P1/f 2 < oo, 5.21

Theorem.

Assume that P 11 1/Je0 11 2 < oo and that the map e f-+ P1/Je is differentiable at a zero Bo, with nonsingular derivative matrix Ve0 • lfrl'n 1/Je" = op ( n - 1 1 2 ) , and en .!,. Bo, then

In particular, the sequence Jn(f}n - 610) is asymptotically normal with mean zero and covarzance matrzx ye-o l p ,trr eo ,t,T r e0 ( Ve0- l ) T • ·

·

For a fixed measurable function f, we abbreviate Jn(r?n - P ) f to G f , the empirical process evaluated at f. The consistency of {J n and the Lipschitz condition on the maps e f-+ 1/Je imply that n

Proof.

(5.22) For a nonrandom sequence {J n this is immediate from the fact that the means of these variables are zero, while the variances are bounded by P 11 1/Jen - 1/!e0 f ::S P� 2 ll tln - Bo 11 2 and hence converge to zero. A proof for estimators {Jn under the present mild conditions takes more effort. The appropriate tools are developed in Chapter 19. In Example 19.7 it is seen that the functions 1/Je form a Donsker class. Next, (5.22) follows from Lemma 19.24. Here we accept the convergence as a fact and give the remainder of the proof. By the definitions of {jn and Bo, we can rewrite Gn 1/Je" as ,JfiP(1/!e0 - 1/!e) + o p ( l ) Combining this with the delta method (or Lemma 2. 12) and the differentiability of the map e f-+ p 1/Je we find that .

'

5.3

Asymptotic Normality

53

In particular, by the invertibility of the matrix Ve0 , This implies that (} n is .jll- consistent: The left side is bounded in probability. Inserting this in the previous display, we obtain that JfiVe0 ({}n - eo) = - CGn 1/leo + op (1). We conclude the proof by taking the inverse Ve� 1 left and right. Because matrix multiplication is a continous map, the inverse of the remainder term still converges to zero in probability. • The preceding theorem is a reasonable compromise between simplicity and general applicability, but, unfortunately, it does not cover the sample median. Because the function e f-+ sign(x - e) is not Lipschitz, the Lipschitz condition is apparently still stronger than necessary. Inspection of the proof shows that it is used only to ensure (5.22). It is seen in Lemma 19.24, that (5.22) can be ascertained under the weaker conditions that the collection of functions x f-+ 1/Je (x) are a "Donsker class" and that the map e f-+ 1/Je is continuous in probability. The functions sign(x - e) do satisfy these conditions, but a proof and the definition of a Donsker class are deferred to Chapter 19. If the functions e f-+ 1/Je (x) are continuously differentiable, then the natural candidate for t (x) is supe II t e II , with the supremum taken over a neighborhood of eo . Then the main condition is that the partial derivatives are "locally dominated" by a square-integrable function: There should exist a square-integrable function -(p with I -(p e I .:::: -(p for every e close to eo . If e f-+ ;p e (x ) is also continuous at eo, then the dominated-convergence theorem readily yields that Ve0 = P-(p eo · The properties of M estimators can typically be obtained under milder conditions by using their characterization as maximizers. The following theorem is in the same spirit as the preceding one but does cover the median. It concerns M -estimators defined as maximizers of a criterion function e f-+ IfDnme, which are assumed to be consistent for a point of maximum e0 of the function e f-+ P m e . If the latter function is twice continuously differentiable at e0, then, of course, it allows a two-term Taylor expansion of the form It is this expansion rather than the differentiability that is needed in the following theorem.

5.23 Theorem. For each e in an open subset ofEuclidean space let x f-+ m e (x) be a mea­ surable function such that e f-+ m e (x) is differentiable at e0 for P -almost every x t with derivative me0 (x) and such that, for every e 1 and e2 in a neighborhood of eo and a measur­ able function m with Pm 2 < 00 Furthermore, assume that the map e f-+ Pme admits a second-order Taylor expansion at a point of maximum e0 with nonsingular symmetric second derivative matrix Ve0 • If IfDn m en � SUPe IfDnme - Op (n - 1 ) and en -+p eo, then A

t

Alternatively, it suffices that e

!---+

me

is differentiable at 8o in ?-probability.

M- and Z-Estimators

54

In particular, the sequence Jfi(en - eo) is asymptotically normal with mean zero and covariance matrix Ve� 1 Prhe0rh� Ve� 1 . *Proof. The Lipschitz property and the differentiability of the maps e for every random sequence hn that is bounded in probability,

r-+

m e imply that,

Gn [ Jn (m eo+hnlvfn - me0) - h � rhe0 J !,. 0.

For nonrandom sequences hn this follows, because the variables have zero means, and vari­ ances that converge to zero, by the dominated convergence theorem. For general sequences hn this follows from Lemma 19.3 1 . A second fact that we need and that is proved subsequently i s the Jfi -consistency of the sequence en. By Corollary 5.53, the Lipschitz condition, and the twice differentiability of the map e r-+ P m e , the sequence Jfi({jn - e) is bounded in probability. The remainder of the proof is self-contained. In view of the twice differentiability of the map e r-+ P m e , the preceding display can be rewritten as . + h- ny Gn me0 + Op ( 1 ) . n JP>n (m eo+hnlvfn - m e0 ) = 21 h- ny Ve0hn Because the sequence en i s Jfi -consistent, this is valid both for hn equal to h n = Jfi Cen - eo) and for hn = - Ve� 1 Gnrhe0 • After simple algebra in the second case, we obtain the equations

By the definition of en, the left side of the first equation is larger than the left side of the second equation (up to o p ( 1)) and hence the same relation is true for the right sides. Take the difference, complete the square, and conclude that

� (hn + Ve� 1 Gnrhe0) T Ve0 (hn + Ve� 1 Gnrhe0) + o p ( l )

2:

0.

Because the matrix Ve0 is strictly negative-definite, the quadratic form must converge to zero in probability. The same must be true for llh n + Ve� 1 Gnrhe0 11 . • The assertions of the preceding theorems must be in agreement with each other and also with the informal derivation leading to (5.20). If e r-+ m e (x) is differentiable, then a maximizer of e r-+ lfDn m e typically solves lfDn 1/Je = 0 for 1/Je = rhe . Then the theorems and (5 .20) are in agreement provided that . a a2 Ve = ae 2 P m e = ae Pl/Je = Pl/J e = P m e . This involves changing the order of differentiation (with respect to e) and integration (with respect to x ), and is usually permitted. However, for instance, the second derivative of Pme may exist without e r-+ m e (x) being differentiable for all x, as is seen in the following example.

5.24

Example (Median).

The sample median maximizes the criterion function e

r-+

- L7= 1 1 Xi -e 1 . Assume that the distribution function F of the observations is differentiable

5.3

-0.5

55

Asymptotic Nonnality

0.0

0.5

Figure 5.3. The distribution function o f the s ample median (dotted curve) and its normal approxi­ mation for a sample of size 25 from the Laplace distribution.

at its median e0 with positive derivative j(e0). Then the sample median is asymptotically normal. This follows from Theorem 5.23 applied with me (x) = lx - e I - l x 1 . As a consequence of the triangle inequality, this function satisfies the Lipschitz condition with m (x) 1. Furthermore, the map e f---+ me (x) is differentiable at eo except if X = eo, with mea (x) = - sign(x - e0) . By partial integration, =

eF(O) + J{ (e - 2x) dF(x) - e (1 - F(e)) = 2 lr F(x) dx - e. o (o , e ] If F is sufficiently regular around e0, then Pme is twice differentiable with first derivative 2F(e) - 1 (which vanishes at e0) and second derivative 2j(e). More generally, under the minimal condition that F is differentiable at e0, the function Pme has a Taylor expansion Pmea + � (e - eo) 2 2j (eo) + o(le - eo l 2 ) , so that we set Vea = 2j(eo). Because Pm�a = E1 = 1 , the asymptotic variance of the median is 1/(2j(e0)) 2 . Figure 5.3 gives an Pme

=

impression of the accuracy of the approximation.

D

5.25 Example (Misspecified model). Suppose an experimenter postulates a model {pe : e E 8 } for a sample of observations X 1 , . . . , X n . However, the model is misspecified in that the true underlying distribution does not belong to the model. The experimenter decides to use the postulated model anyway, and obtains an estimate {J n from maximizing the likelihood L log Pe (Xi). What is the asymptotic behaviour of {Jn ? At first sight, it might appear that {Jn would behave erratically due to the use of the wrong model. However, this is not the case. First, we expect that {Jn is asymptotically consistent for a value eo that maximizes the function e f---+ P log pe , where the expectation is taken under the true underlying distribution P . The density P ea can be viewed as the "projection"

M- and Z-Estimators

56

of the true underlying distribution P on the model using the Kullback-Leibler divergence, which is defined as - P log (pe I p) , as a "distance" measure: pe0 minimizes this quantity over all densities in the model. Second, we expect that Jfi({jn - e0) is asymptotically normal with mean zero and covariance matrix

P log pe . The Here fe log pe , and Ve0 is the second derivative matrix of the map e preceding theorem with me log pe gives sufficient conditions for this to be true. The asymptotics give insight into the practical value of the experimenter's estimate {jn · This depends on the specific situation. However, if the model is not too far off from the truth, then the estimated density Pe " may be a reasonable approximation for the true density. D

=

f--+

=

5.26 Example (Exponential frailty model). Suppose that the observations are a random sample (X 1 , Y1 ), . . . , (Xn , Yn) of pairs of survival times. For instance, each X i is the survival time of a "father" and Yi the survival time of a "son." We assume that given an unobservable value Z i , the survival times X i and Yi are independent and exponentially distributed with parameters Zi and e Zi , respectively. The value Zi may be different for each observation. The problem is to estimate the ratio e of the parameters. To fit this example into the i.i.d. set-up of this chapter, we assume that the values Z 1 , . . . , Zn are realizations of a random sample Z 1 , . . . , Zn from some given distribution (that we do not have to know or parametrize). One approach is based on the sufficiency of the variable X i + e Yi for Zi in the case that e is known. Given Zi z, this "statistic" possesses the gamma-distribution with shape parameter 2 and scale parameter z. Corresponding to this, the conditional density of an observation (X, Y) factorizes, for a given z, as he (x , y) ge (x + e y I z) , for ge (s I z) a-density and z 2 se -z s the g

=

amm

he (x , y)

=

e

X

+ ey

.

Because the density of X i + e Y; depends on the unobservable value Zi , we might wish to discard the factor ge (s I z) from the likelihood and use the factor he (x , y) only. Unfor­ tunately, this "conditional likelihood" does not behave as an ordinary likelihood, in that the corresponding "conditional likelihood equation," based on the function h e I he (x , y) a 1 a e log he (x , y), does not have mean zero under e . The bias can be corrected by condi­ tioning on the sufficient statistic. Let

=

= 2e _!_hh e (X, Y) - 2eEe (h_!_he (X, Y) I X + e y) = XX +- ee YY Next define an estimator {} n as the solution of n 1/fe = 0. This works fairly nicely. Because the function e f--+ 1/fe (x , y) is continuous, and de­ creases strictly from 1 to - 1 on (0, oo) for every x , y 0, the equation n 1/le = 0 has a 1/le (X, Y)

.

TID

>

TID

unique solution. The sequence of solutions Bn can be seen to be consistent by Lemma 5. 10. By straightforward calculation, as e -+ eo,

5.3

Asymptotic Normality

57

Hence the zero of e r--+ Pe0 1/!e is taken uniquely at e = eo. Next, the sequence ,Jri({jn - eo) can be shown to be asymptotically normal by Theorem 5.21 . In fact, the functions � e (x, y) are uniformly bounded in x , y > 0 and e ranging over compacta in (0, oo) , so that, by the mean value theorem, the function � in this theorem may be taken equal to a constant. On the other hand, although this estimator is easy to compute, it can be shown that it is not asymptotically optimal. In Chapter 25 on semiparametric models, we discuss estimators with a smaller asymptotic variance. D Suppose that we observe a random sample (X 1 , Y1 ), . . . , (Xn , Yn ) from the distribution of a vector (X, Y) that follows the regression model

5.27

Example (Nonlinear least squares).

Y = fe0 (X) + e,

E(e I X) = 0.

Here fe is a parametric family of regression functions, for instance fe (x) = e 1 + e2 e e3x , and we aim at estimating the unknown vector e. (We assume that the independent variables are a random sample in order to fit the example in our i.i.d. notation, but the analysis could be carried out conditionally as well.) The least squares estimator that minimizes n

e r--+

L (li - fe (Xi) ) 2 i= 1

is an M-estimator for me (x, y) = (y - fe (x)) 2 (or rather minus this function). It should be expected to converge to the minimizer of the limit criterion function

Thus the least squares estimator should be consistent if e0 is identifiable from the model, in the sense that e =I= e0 implies that fe (X) =I= fe0 (X) with positive probability. For sufficiently regular regression models, we have

This suggests that the conditio 1_1 s of Theorem 5.23 are satisfied with Ve0 = 2P j eo j� and rhe0 (x, y) = -2(y - fe0 (x)) f e0 (x). If e and X are independent, then this leads to the asymptotic covariance matrix Ve� 1 2Ee 2 . D Besides giving the asymptotic normality of Jfi({jn - e0), the preceding theorems give an asymptotic representation

If we neglect the remainder term, t then this means that {J n - e0 behaves as the average of the variables Ve� 1 1/Je0 (X i )· Then the (asymptotic) "influence" of the nth observation on the t

To make the following derivation rigorous, more information concerning the remainder term would be necessary.

M - and Z-Estimators

58

value of {j can be computed as n

Because the "influence" of an extra observation x is proportional to Ve- 1 1/Je (x), the function x f--+ Ve- 1 1/Je (x) is called the asymptotic influence function of the estimator {j n . Influence functions can be defined for many other estimators as well, but the method of Z-estimation is particularly convenient to obtain estimators with given influence functions. Because Ve0 is a constant (matrix), any shape of influence function can be obtained by simply choosing the right functions 1/Je . For the purpose of robust estimation, perhaps the most important aim is to bound the influence of each individual observation. Thus, a Z-estimator is called B-robust if the function 1/Je is bounded.

5.28 Example (Robust regression). Consider a random sample of observations (X 1 , Y1 ) , . . . , (Xn , Yn ) following the linear regression model

for i.i.d. errors e 1 , , en that are independent of X 1 , , Xn . The classical estimator for the regression parameter (] is the least squares estimator, which minimizes :L�= 1 (Yi - (] T X i ) 2 . Outlying values of Xi ("leverage points") or extreme values of (X i , Yi) jointly ("influence points") can have an arbitrarily large influence on the value of the least-squares estimator, which therefore is nonrobust. As in the case of location estimators, a more robust estimator for (] can be obtained by replacing the square by a function m (x) that grows less rapidly as x ---+ oo, for instance m (x) = lx I or m (x) equal to the primitive function of Huber's 1/1 . Usually, minimizing an expression of the type 2:: �= 1 m (Yi - (] Xi ) i s equivalent to solving a system of equations .

.

.

•

n

1/I ( Yi L i=1

-

.

•

e r xi)xi =

o.

Because E1jf (Y - el X) X = E1jf (e)EX, we can expect the resulting estimator to be consistent provided E1jf ( e ) = 0. Furthermore, we should expect that, for Ve0 = E1jf ( e )XXT , '

Consequently, even for a bounded function 1/1 , the influence function (x , y) f--+ Ve- 1 1/1 (y e T x)x may be unbounded, and an extreme value of an Xi may still have an arbitrarily large influence on the estimate (asymptotically). Thus, the estimators obtained in this way are protected against influence points but may still suffer from leverage points and hence are only partly robust. To obtain fully robust estimators, we can change the estimating

5.3

59

Asymptotic Normality

equations to

n L o/ (( Yi - e Tx i)v(Xi))w (Xd = 0. i=l Here we protect against leverage points by choosing w bounded. For more flexibility we have also allowed a weighting factor v (X i ) inside 1/J. The choices 1/J (x) = x, v (x) = 1 and w(x) = x correspond to the (nonrobust) least-squares estimator. The solution {j n of our final estimating equation should be expected to be consistent for the solution of

(

)

0 = E o/ ( ( Y - e T X)v (X) )w (X) = E o/ ( e + etf X - e T X) v (X) w (X) .

If the function 1/J is odd and the error symmetric, then the true value e0 will be a solution whenever e is symmetric about zero, because then E 1/J (eO' ) = 0 for every 0' . Precise conditions for the asymptotic normality of Jfi({jn - e0) can be obtained from Theorems 5.21 and 5. 9. The verification of the conditions of Theorem 5.2 1 , which are "local" in nature, is relatively easy, and, if necessary, the Lipschitz condition can be relaxed by using results on empirical processes introduced in Chapter 19 directly. Perhaps proving the consistency of en is harder. The biggest technical problem may be to show that {jn = Op (1), so it would help if e could a priori be restricted to a bounded set. On the other hand, for bounded functions 1/J, the case of most interest in the present context, the functions (x, y) � 1/J ((y - e T x) v(x)) w (x) readily form a Glivenko-Cantelli class when e ranges freely, so that verification of the strong uniqueness of e0 as a zero becomes the main challenge when applying Theorem 5.9. This leads to a combination of conditions on 1/J, v, w, and the distributions of e and X. 0

5.29 Example (Optimal robust estimators). Every sufficiently regular function 1/J defines a location estimator en through the equation L 7= 1 1/f (Xi - e) = 0. In order to choose among the different estimators, we could compare their asymptotic variances and use the one with the smallest variance under the postulated (or estimated) distribution P of the observations. On the other hand, if we also wish to guard against extreme obervations, then we should find a balance between robustness and asymptotic variance. One possibility is to use the estimator with the smallest asymptotic variance at the postulated, ideal distribution P under the side condition that its influence function be uniformly bounded by some constant c. In this example we show that for P the normal distribution, this leads to the Huber estimator. The Z-estimator is consistent for the solution eo of the equation P o/ (· - e) = Eo/(X 1 e) = 0. Suppose that we fix an underlying, ideal P whose "location" e0 is zero. Then the problem is to find 1/J that minimizes the asymptotic variance Po/ 2 /(Po/') 2 under the two side conditions, for a given constant c, sup X

I 1/JPo/(x)' I -<

and Po/ = 0.

c '

The problem is homogeneous in 1/J, and hence we may assume that Po/' = 1 without loss of generality. Next, minimization of Po/ 2 under the side conditions Po/ = 0, Po/' = 1 and I 1/J I � c can be achieved by using Lagrange multipliers, as in problem 14.6 This leads to minimizing 00

M- and Z-Estimators

60

for fixed "multipliers" A and f.1, under the side condition 11 1/J II :::: c with respect to 1/J . This expectation is minimized by minimizing the integrand pointwise, for every fixed x . Thus the minimizing 1/J has the property that, for every x separately, y = 1/J (x ) minimizes the parabola y 2 + A Y + f.l,y (p ' jp) (x) over y E [ c , c] . This readily gives the solution, with [y ]� the value y truncated to the interval [ c, d ] , oo

-

1/J (x)

=

[

1 2

- -A -

1 p' - f.l, - (X) 2 p

]c -c

.

The constants A and f.1, can be solved from the side conditions P l/J = 0 and Pl/J ' = 1 . The normal distribution P = has location score function p'jp(x) = -x, and by symmetry it follows that A = 0 in this case. Then the optimal l/J reduces to Huber's 1/J function. 0

*5.4

Estimated Parameters

In many situations, the estimating equations for the parameters of interest contain prelim­ inary estimates for "nuisance parameters." For example, many robust location estimators are defined as the solutions of equations of the type (5.30) Here a is an initial (robust) estimator of scale, which is meant to stabilize the robustness of the location estimator. For instance, the "cut-off" parameter k in Huber's 1/J-function determines the amount of robustness of Huber's estimator, but the effect of a particular choice of k on bounding the influence of outlying observations is relative to the range of the observations. If the observations are concentrated in the interval [ -k, k] , then Huber's 1/J yields nothing else but the sample mean, if all observations are outside [ -k, k] , we get the median. Scaling the observations to a standard scale gives a clear meaning to the value of k. The use of the median absolute deviation from the median (see. section 2 1 .3) is often recommended for this purpose. If the scale estimator is itself a Z-estimator, then we can treat the pair (fJ , a) as a Z­ estimator for a system of equations, and next apply the preceding theorems. More generally, we can apply the following result. In this subsection we allow a condition in terms of Donsker classes, which are discussed in Chapter 19. The proof of the following theorem follows the same steps as the proof of Theorem 5 .2 1 . 5.31 Theorem. For each () in an open subset oflR k and each 77 in a metric space, let x f-+ 1/Je, ,7 (x) be an 1R k -valued measurable function such that the class offunctions 1/Je . r; : I I () ­ eo II < 8 , d (71 , 7J0) < 8 } is Donsker for some 8 > 0, and such that P 11 1/Je. r; - 1/Jeo. r;o 11 2 -+ 0 as (() , 77) -+ (eo , 77o). Assume that Pl/Jeo . r;o = 0, and that the maps () f-+ Pl/Je. r; are diffe r­ entiable at ()0, uniformly in 77 in a neighborhood of 7Jo with nonsingular derivative matrices p Ve0, r; such that Ve0 . r; -+ Ve0, ,70 • If .Jii fl!n l/Je ,, ry, = o p ( 1 ) and (()n , � n) -+ (eo , 7Jo ), then

{

A

5. 5

Maximum Likelihood Estimators

61

Under the conditions of this theorem, the limiting distribution of the sequence .Jfi({Jn ­

eo) depends on the estimator fin through the "drift" term .Jfi P 1/Je0,r1" . In general, this gives a contribution to the limiting distribution, and fin must be chosen with care. If fin is .Jfi­ consistent and the map rJ f--+ P1/fe0,r1 is differentiable, then the drift term can be analyzed

using the delta-method. It may happen that the drift term is zero. If the parameters e and rJ are "orthogonal" in this sense, then the auxiliary estimators fin may converge at an arbitrarily slow rate and affect the limit distribution of {Jn only through their limiting value rJo .

5.32 Example (Symmetric location). Suppose that the distribution of the observations is symmetric about e0. Let x f--+ 1/f (x ) be an antisymmetric function, and consider the Z-estimators that solve equation (5.30). Because P1/f ((X - e0)ja) = 0 for every a, by the symmetry of P and the antisymmetry of 1/J, the "drift term" due to fi in the pre­ ceding theorem is identically zero. The estimator {Jn has the same limiting distribu­ tion whether we use an arbitrary consistent estimator of a "true scale" a0 or a0 itself. D 5.33 Example (Robust regression). In the linear regression model considered in Exam­ ple 5.28, suppose that we choose the weight functions v and w dependent on the data and solve the robust estimator {Jn of the regression parameters from

This corresponds to defining a nuisance parameter rJ = ( v , w ) and setting 1/fe,v,w (x , y) = 1/! ((y - e T x ) v (x ) ) w (x ) . If the functions 1/!e , v,w run through a Donsker class (and they easily do), and are continuous in (e , v , w ) , and the map e f--+ P1/Je,v , w is differentiable at e0 uniformly in ( v, w ) , then the preceding theorem applies. If E1j; (ea) = 0 for every a , then P 1/fe0, v , w = 0 for any v and w , and the limit distribution of .Jfi({Jn - e0) is the same, whether we use the random weight functions (vn, wn) or their limit ( v0 , w0) (assuming that this exists). The purpose of using random weight functions could be, besides stabilizing the robust­ ness, to improve the asymptotic efficiency of {Jn· The limit ( v 0 , w0) typically is not the same for every underlying distribution P, and the estimators (vn, wn) can be chosen in such a way that the asymptotic variance is minimal. D

5.5

Maximum Likelihood Estimators

Maximum likelihood estimators are examples of M -estimators. In this section we special­ ize the consistency and the asymptotic normality results of the preceding sections to this important special case. Our approach reverses the historical order. Maximum likelihood estimators were shown to be asymptotically normal first by Fisher in the 1 920s and rigor­ ously by Cramer, among others, in the 1940s . General M -estimators were not introduced and studied systematically until the 1960s, when they became essential in the development of robust estimators.

62

M- and Z-Estimators

If X 1 , . . . , Xn are a random sample from a density pe , then the maximum likelihood estimator en maximizes the function (] f-+ L log Pe (Xi ) , or equivalently, the function Pe (X = Pn log Pe . log Mn (f3) = -1 � � n i = l P ea d Pea

(Subtraction of the "constant" L log Pea (Xi ) turns out to be mathematically convenient.) If we agree that log 0 = -oo, then this expression is with probability 1 well defined if Pea is the true density. The asymptotic function corresponding to Mn is t Pe (X) = Pe log Pe . M(f3) = Eea log a Pea P ea The number -M(f3) is called the Kullback-Leibler divergence of Pe and Pea ; it is often considered a measure of "distance" between pe and Pea , although it does not have the properties of a mathematical distance. Based on the results of the previous sections, we may expect the maximum likelihood estimator to converge to a point of maximum of M(f3). Is the true value f30 always a point of maximum? The answer is affirmative, and, moreover, the true value is a unique point of maximum if the true measure is identifiable: every f3 f. f3o.

(5.34)

This requires that the model for the observations is not the same under the parameters f3 and f30. Identifiability is a natural and even a necessary condition: If the parameter is not identifiable, then consistent estimators cannot exist. Lemma. Let { pe : f3 E 8} be a collection of subprobability densities such that (5.34) holds and such that Pea is a probability measure. Then M(f3) = Pea log pe / P ea

5.35

attains its maximum uniquely at f30.

First note that M (f30) = Pea log 1 = 0. Hence we wish to show that M (f3) is strictly negative for f3 f. f3o. Because log x 2( .JX - 1) for every x � 0, we have, writing J.-L for the dominating measure,

Proof

:::

Pea log .!!!_ Pea

:S

2 Pea

- 1 ) 2 J JPe Pea d f.-L - 2 (V{E_ P ea =

::: - f (yfpe - �f d

J.-L .

(The last inequality is an equality if J pe d J.-L = 1 .) This is always nonpositive, and is zero only if pe and Pea are equal. By assumption the latter happens only if f3 = f3o. • Thus, under conditions such as in section 5 .2 and identifiability, the sequence of maxi­ mum likelihood estimators is consistent for the true parameter. t

Presently we take the expectation Pea under the parameter Bo , whereas the derivation in section 5.3 is valid for a generic underlying probability structure and does not conceptually require that the set of parameters 8 indexes a set of under!ying distributions.

5. 5

Maximum Likelihood Estimators

63

This conclusion is derived from viewing the maximum likelihood estimator as an M­ estimator for me = log pe . Sometimes it is technically advantageous to use a different starting point. For instance, consider the function e+ e me = 1o g P P o 2peo ::..._---=. .._

By the concavity of the logarithm, the maximum likelihood estimator fJ satisfies 1 Pe + lP'n -1 log 1 0 = lP'nme lP'nm e :=:: lP'n - log :=:: 0• 2 2 pe0 Even though fJ does not maximize (] f-+ lP'n me , this inequality can be used as the starting point for a consistency proof, since Theorem 5.7 requires that Mn (B ) :=:: Mn ((]0) - op (1) only. The true parameter is still identifiable from this criterion function, because, by the preceding lemma, Pe0 me = 0 implies that (pe + pe0 )/2 = pe0 , or Pe = pe0 • A technical advantage is that me :=:: log(1/2) . For another variation, see Example 5 . 1 7 . Consider asymptotic normality. The maximum likelihood estimator solves the likelihood equations a n - L: log pe (X i ) = 0. ae i = 1 Hence it is a Z-estimator for 1/Je equal to the score function i e = a I ae log Pe of the model. In view of the results of section 5.3, we expect that the sequence ,Jn ( en - (]) is, under (] , asymptotically normal with mean zero and covariance matrix (5.36) Under regularity conditions, this reduces to the inverse of the Fisher information matrix To see this in the case of a one-dimensional parameter, differentiate the identity J pe d JL 1 twice with respect to (] . Assuming that the order of differentiation and integration can be reversed, we obtain J Pe dJL J P e dJL 0. Together with the identities Pe 2 Pe Pe le = le - - ; Pe Pe Pe this implies that Pe l e = 0 (scores have mean zero), and Pe f e = -Ie (the curvature of the likelihood is equal to minus the Fisher information). Consequently, (5.36) reduces to Ie- 1 • The higher-dimensional case follows in the same way, in which we should interpret the identities Pe l e = 0 and Pe f e = - Ie as a vector and a matrix identity, respectively. We conclude that maximum likelihood estimators typically satisfy =

=

·

=

··

_

( )

This is a very important result, as it implies that maximum likelihood estimators are asymp­ totically optimal. The convergence in distribution means roughly that the maximum likeli­ hood estimator B n is N (e , (n /e ) - 1 ) -distributed for every (] , for large n . Hence, it is asymp­ totically unbiased and asymptotically of variance (n /e ) - 1 . According to the Cramer-Rao

64

M- and Z-Estimators

theorem, the variance of an unbiased estimator is at least (n le )- 1 . Thus, we could in­ fer that the maximum likelihood estimator is asymptotically uniformly minimum-variance unbiased, and in this sense optimal. We write "could" because the preceding reasoning is informal and unsatisfying. The asymptotic normality does not warrant any conclusion about the convergence of the moments Ee {J n and vare {J n ; we have not introduced an asymptotic version of the Cramer-Rao theorem; and the Cramer-Rao bound does not make any assertion concerning asymptotic normality. Moreover, the unbiasedness required by the Cramer-Rao theorem is restrictive and can be relaxed considerably in the asymptotic situation. However, the message that maximum likelihood estimators are asymptotically efficient is correct. We give a precise discussion in Chapter 8. The justification through asymptotics appears to be the only general justification of the method of maximum likelihood. In some form, this result was found by Fisher in the 1920s, but a better and more general insight was only obtained in the period from 1 950 through 1 970 through the work of Le Cam and others. In the preceding informal derivations and discussion, it is implicitly understood that the density pe possesses at least two derivatives with respect to the parameter. Although this can be relaxed considerably, a certain amount of smoothness of the dependence (} f-+ Pe is essential for the asymptotic normality. Compare the behavior of the maximum likelihood estimators in the case of uniformly distributed observations : They are neither asymptotically normal nor asymptotically optimal.

5.37 Example (Uniform distribution). Let X 1 , . . . , Xn be a sample from the uniform distribution on [0, (} ] . Then the maximum likelihood estimator is the maximum X (n) of the observations. Because the variance of X en) is of the order O (n- 2 ), we expect that a suitable norming rate in this case is not ,Jn, but n. Indeed, for each x < 0

Thus, the sequence -n (X (n) - 8) converges in distribution to an exponential distribution with mean e . Consequently, the sequence ,Jii ( Xcn) - 8) converges to zero in probability. Note that most of the informal operations in the preceding introduction are illegal or not even defined for the uniform distribution, starting with the definition of the likelihood equa­ tions. The informal conclusion that the maximum likelihood estimator is asymptotically optimal is also wrong in this case; see section 9.4. 0 We conclude this section with a theorem that establishes the asymptotic normality of maximum likelihood estimators rigorously. Clearly, the asymptotic normality follows from Theorem 5.23 applied to me = log pe , or from Theorem 5.21 applied with 1/re = le equal to the score function of the model. The following result is a minor variation on the first theorem. Its conditions somehow also ensure the relationship Pele = -Ie and the twice­ differentiability of the map (} f-+ Pe0 log Pe , even though the existence of second derivatives is not part of the assumptions. This remarkable phenomenon results from the trivial fact that square roots of probability densities have squares that integrate to 1 . To exploit this, we require the differentiability of the maps (} f-+ �. rather than of the maps (} f-+ log Pe . A statistical model (Pe : (} E 8) is called differentiable in q uadratic mean if there exists a

5. 5 Maximum Likelihood Estimators

measurable vector-valued function le0 such that, as e

---+

65

e0, (5 .38)

This property also plays an important role in asymptotic optimality theory. A discussion, including simple conditions for its validity, is given in Chapter 7. It should be noted that 1 a 1(a ) e = log Pe §e. = P ae 5e 25e a e 2 a e a

Thus, the function fe0 in the integral really is the score function of the model (as the notation suggests), and the expression le0 Pe0fei,� defines the Fisher information matrix. However, condition (5 .38) does not require existence of a;ae pe (x) for every x.

=

Suppose that the model (Pe : e E 8 ) is differentiable in quadratic mean at an inner point e0 of 8 c ffi.k . Furthermore, suppose that there exists a measurable function .£ with Pe0l 2 < oo such that, for every e 1 and e2 in a neighborhood ofe0, 5.39

Theorem.

If the Fisher information matrix le0 is nonsingular and {jn is consistent, then

In particular, the sequence y'ri(Bn - e0) is asymptotically normal with mean zero and covariance matrix Ie� 1 . This theorem is a corollary of Theorem 5.23. We shall show that the conditions of the latter theorem are satisfied for m e log pe and Ve0 - le0 • Fix an arbitrary converging sequence of vectors hn ---+ h, and set *Proof

=

=

By the differentiability in quadratic mean, the sequence ,fii, Wn converges in L 2 (Pe0) to the function h T le0 • In particular, it converges in probability, whence by a delta method

vfn (log Peo+hnl y'n - log Peo)

= 2vfn log ( 1 + � Wn ) � h T le0 •

In view of the Lipschitz condition on the map e f---+ log pe , we can apply the dominated­ convergence theorem to strengthen this to convergence in L 2 (Pe0). This shows that the map e f---+ log P e is differentiable in probability, as required in Theorem 5.23. (The preceding argument considers only sequences ell of the special form eo + hn/ y'n approaching eo. Because hn can be any converging sequence and Jn+I; Jn ---+ 1 , these sequences are actually not so special. By re-indexing the result can be seen to be true for any ell ---+ e0.) Next, by computing means (which are zero) and variances, we see that

M- and Z-Estimators

66

Equating this result to the expansion given by Theorem 7 .2, we see that

Hence the map e r--+ Pe0 log Pe is twice-differentiable with second derivative matrix - Ie0, or at least permits the corresponding Taylor expansion of order 2. • Suppose that we observe a random sample (X 1 , Y1 ), . . . , (Xn , Y11) consisting of k-dimensional vectors of "covariates" X; , and 0- 1 "response variables" Y; , following the model

5.40

Example (Binary regression).

Here \lf : ffi. r--+ [0, 1] is a known continuously differentiable, monotone function. The choices \lf (e) = 1 1 ( 1 + e- e ) (the logistic distribution function) and \lf = (the normal distribution function) correspond to the logit model and probit model, respectively. The maximum likelihood estimator e n maximizes the (conditional) likelihood function n n Y e r--+ n pe (Y; I X i) : = n \lf (e T X;)yi (1 - \lf (e T X i) ) I - ; .

i =l

i =l

The consistency and asymptotic normality of en can be proved, for instance, by combining Theorems 5.7 and 5 .39. (Alternatively, we may follow the classical approach given in sec­ tion 5 .6. The latter is particularly attractive for the logit model, for which the log likelihood is strictly concave in e , so that the point of maximum is unique.) For identifiability of e we must assume that the distribution of the X i is not concentrated on a (k - I)-dimensional affine subspace of ffi.k . For simplicity we assume that the range of Xi is bounded. The consistency can be proved by applying Theorem 5.7 with me = log( pe + pe0)12. Because pe0 is bounded away from 0 (and oo), the function me is somewhat better behaved than the function log pe . By Lemma 5.35, the parameter e is identifiable from the density pe . We can redo the proof to see that, with ;S meaning "less than up to a constant,"

This shows that eo is the unique point ofmaximum of e r--+ Pe0me . Furthermore, if Pe0mek -? Pe0me0, then e{ X !,. e[ X. If the sequence ek is also bounded, then E( (ek - eo) T X) 2 -? 0, whence ek r--+ eo by the nonsingularity of the matrix EX x T . On the other hand, llek I cannot have a diverging subsequence, because in that case e{ I II ek I x !,. o and hence ek f II ek II -? o by the same argument. This verifies condition (5 .8). Checking the uniform convergence to zero of supe IIfDnme - Pme I is not trivial, but it becomes an easy exercise if we employ the Glivenki-Cantelli theorem, as discussed in Chapter 19. The functions x r--+ \lf (e r x ) form a VC-class, and the functions me take the form me (x ' y) = ¢ ( \lf (e T X ) , y , \lf eel X) ) , where the function ¢ ( y ' y ' T}) is Lipschitz in its first argument with Lipschitz constant bounded above by 1 I rJ + 1 I (1 - TJ ) . This is enough to

5. 6

67

Classical Conditions

ensure that the functions me form a Donsker class and hence certainly a Glivenko-Cantelli class, in view of Example 1 9.20. The asymptotic normality of Jfi(Bn 8) is now a consequence of Theorem 5.39. The score function -

is uniformly bounded in x , y and e ranging over compacta, and continuous in 8 for every x and y . The Fisher information matrix is continuous in 8, and is bounded below by a multiple of EX x r and hence is nonsingular. The differentiability in quadratic mean follows by calculus, or by Lemma 7 .6. D

*5.6

Classical Conditions

In this section we discuss the "classical conditions" for asymptotic normality of M -estima­ tors. These conditions were formulated in the 1 930s and 1940s to make the informal deriva­ tions of the asymptotic normality of maximum likelihood estimators, for instance by Fisher, mathematically rigorous . Although Theorem 5.23 requires less than a first derivative of the criterion function, the "classical conditions" require existence of third derivatives. It is clear that the classical conditions are too stringent, but they are still of interest, because they are simple, lead to simple proofs, and nevertheless apply to many examples . The classical conditions also ensure existence of Z -estimators and have a little to say about their consistency. We describe the classical approach for general Z-estimators and vector-valued parame­ ters. The higher-dimensional case requires more skill in calculus and matrix algebra than is necessary for the one-dimensional case. When simplified to dimension one the argu­ ments do not go much beyond making the informal derivation leading from (5 . 1 8) to (5 . 19) rigorous. Let the observations X 1 , . . . , Xn be a sample from a distribution P , and consider the estimating equations \ll (8)

=

P 1/Je .

The estimator B is a zero of \lln, and the true value 80 a zero of \ll . The essential condition of the following theorem is that the second-order partial derivatives of 1/Je (x) with respect to e exist for every x and satisfy n

for some integrable measurable function 1f . This should be true at least for every 8 in a neighborhood of 80•

68

M- and Z-Estimators

5.41 Theorem. For each e in an open subset of Euclidean space, let e � 1/fe (x ) be twice continuously differentiable for every x. Suppose that P1/fe0 = 0, that P 1 1/le0 11 2 < oo and that the matrix P'if; e0 exists and is nonsingular. Assume that the second-order partial derivatives are dominated by a fixed integrable function 1f; (x) for every e in a neighborhood ofeo. Then every consistent estimator sequence {jn such that 'lin ({}n) = Ofor every n satisfies

In particular, the sequence ,Jfi(fJn - e0). is asymptotically normal with mean zero and . covariance matrix ( P 1/1 e0 )- 1 P 1/leo 1/1eTo ( P 1/1 e0 )- 1 •

By Taylor's theorem there exists a (random) vector iJ n on the line segment between eo and {jn such that

Proof

The first term on the right 'lln (eo) is an average of the i.i.d. random vectors 1/Je0 (X i ), which have mean P1/Je0 = 0. By the central limit theorem, the sequence ,Jfi'll n (e0) converges in distribution to a multivariate normal distribution with mean 0 and covariance matrix P1/Je0 1/f[.0 The derivative �n (eo) in the second term is an average also. By the law of large numbers it converges in probability to the matrix V = P 1/f eo . The second derivative � n (iJn) is a k-vector of (k x k) matrices depending on the second-order derivatives 1/; e . By assumption, there exists a ball B around e0 such that 1/; e is dominated by 11 1/; II for every e E B . The probability of the event { {} n E B } tends to 1 . On this event •

This is bounded in probability by the law of large numbers. Combination of these facts allows us to rewrite the preceding display as

because the sequence ({}n - e0) Op (1) = op (1) 0 p ( 1 ) converges to 0 in probability if {}n is consistent for e0. The probability that the matrix Ve0 + op (1) is invertible tends to 1 . Multiply the preceding equation by Jn and apply ( V + o p ( l ) ) - 1 left and right to complete the proof. • In the preceding sections, the existence and consistency of solutions {}n of the estimating equations is assumed from the start. The present smoothness conditions actually ensure the existence of solutions. (Again the conditions could be significantly relaxed, as shown in the next proof.) Moreover, provided there exists a consistent estimator sequence at all, it is always possible to select a consistent sequence of solutions.

5.42 Theorem. Under the conditions ofthe preceding theorem, the probability that the eq­ uation lP'n 1/fe = 0 has at least one root tends to 1, as n -+ oo, and there exists a sequence of roots {} n such that {} n -+ eo in probability. If 1/1e = me is the gradient of some function

5. 6

Classical Conditions

69

me and &o is a point of local maximum of& f--:>- Pme, then the sequence {jn can be chosen to be local maxima of the maps e f--:>- JPlnme. Proof point {j

Integrate the Taylor expansion of e f--:>- 1/.re (x ) with respect to x to find that, for a = {j (x) on the line segment between &o and e'

B y the domination condition, II PVie- 11 i s bounded by P II Vi ll < 00 i f e i s sufficiently close to 80 • Thus, the map \ll (&) = P1/re is differentiable at 80. By the same argument \ll is differentiable throughout a small neighborhood of &0, and by a similar expansion (but now to first order) the derivative P � e can be seen to be continuous throughout this neighborhood. Because P � eo is nonsingular by assumption, we can make the neighborhood still smaller, if necessary, to ensure that the derivative of \ll is nonsingular throughout the neighborhood. Then, by the inverse function theorem, there exists, for every sufficiently small o > 0, an open neighborhood Go of &0 such that the map \ll : G0 f--:>- ball(O, o) is a homeomorphism. The diameter of G0 is bounded by a multiple of o, by the mean-value theorem and the fact that the norms of the derivatives (P� e )- 1 of the inverse w- 1 are bounded. Combining the preceding Taylor expansion with a similar expansion for the sample version Wn (&) = Yln 1/.re , we see sup ll wn (&) - \II (&) I

e EGs

:S

op ( l ) + oop ( l ) + o 2 0p ( 1 ) ,

where the op ( 1 ) terms and the Op ( l ) term result from the law of large numbers, and are uniform in small o . Because P ( op ( 1 ) + oop ( l ) > � o ) � 0 for every o > 0, there exists On .,[,. 0 such that P (op ( 1 ) + on op ( 1 ) > � on ) � 0. If Kn,o is the event where the left side of the preceding display is bounded above by o, then P(Kn,oJ � 1 as n � oo. On the event Kn,o the map e f--:>- e - Wn w- 1 (&) maps ball(O, o) into itself, by the definitions of Go and Kn,D · Because the map is also continuous, it possesses a fixed-point in ball(O, o), by Brouwer's fixed point theorem. This yields a zero of Wn in the set G0, whence the first assertion of the theorem. For the final assertion, first note that the Hessian P � eo of e f--:>- P me at 80 is negative­ definite, by assumption. A Taylor expansion as in the proof of Theorem 5.41 shows that . . . JPln 1fre" - Pn 1/r e0 ::+p 0 for every &n �p &o. Hence the Hessian Yln l/r e" of 8 f--:>- Pn � e at any consistent zero &n converges in probability to the negative-definite matrix P1fr e0 and is negative-definite with probability tending to 1 . • o

�

The assertion of the theorem that there exists a consistent sequence of roots of the estimating equations is easily misunderstood. It does not guarantee the existence of an asymptotically consistent sequence of estimators. The only claim is that a clairvoyant statistician (with preknowledge of &0 ) can choose a consistent sequence of roots. In reality, it may be impossible to choose the right solutions based only on the data (and knowledge of the model). In this sense the preceding theorem, a standard result in the literature, looks better than it is. The situation is not as bad as it seems. One interesting situation is if the solution of the estimating equation is unique for every n. Then our solutions must be the same as those of the clairvoyant statistician and hence the sequence of solutions is consistent.

70

M- and Z-Estimators

In general, the deficit can be repaired with the help of a preliminary sequence of estimators iJ n . If the sequence iJ n is consistent, then it works to choose the root {) n of lJ:Dn 1/re = 0 that is closest to iJn . Because ll(jn - iJn ll is smaller than the distance 118� - iJn ll between the clairvoyant sequence {)� and iJ n , both distances converge to zero in probability. Thus the sequence of closest roots is consistent. The assertion of the theorem can also be used in a negative direction. The point (}0 in the theorem is required to be a zero of (} f----+ P 1/re , but, apart from that, it may be arbitrary. Thus, the theorem implies at the same time that a malicious statistician can always choose a sequence of roots (Jn that converges to any given zero. These may include other points besides the "true" value of (} . Furthermore, inspection of the proof shows that the sequence of roots can also be chosen to jump back and forth between two (or more) zeros. If the function (} f----+ P l/re has multiple roots, we must exercise care. We can be sure that certain roots of (} f----+ lJ:Dn 1/re are bad estimators. Part of the problem here is caused by using estimating equations , rather than maximiza­ tion to find estimators, which blurs the distinction between points of absolute maximum, local maximum, and even minimum. In the light of the results on consistency in section 5.2, we may expect the location of the point of absolute maximum of (} f----+ Jl:Dnme to converge to a point of absolute maximum of (} f----+ Pme . As long as this is unique, the absolute maximizers of the criterion function are typically consistent.

5.43 Example (Weibull distribution). Let X 1 , tribution with density !!._ e Pe, a (x) = (j x - I e -xB / a '

•

.

X

•

, Xn be a sample from the Weibull dis­ >

0, (}

>

0, a

>

0.

(Then a I / e is a scale parameter.) The score function is given by the partial derivatives of the log density with respect to (} and a :

f e, a (x) =

(1 + log x - : log x , w - � + ;: ) .

The likelihood equations I; ie , a (x i ) = 0 reduce to n 1" 1 1� I:7=1 xf log x; a = � x�� ·· "D + - � log x; " n x e = 0. n n L... i =I i i=I i=l The second equation is strictly decreasing in (} , from oo at (} = 0 to log x -log X (n) at (} = oo. Hence a solution exists, and i s unique, unless all x ; are equal. Provided the higher-order derivatives of the score function exist and can be dominated, the sequence of maximum likelihood estimators (en , an) is asymptotically normal by Theorems 5.41 and 5.42. There exist four different third-order derivatives, given by 2 xe 3 = - - log x () 3 (j xe = 2 log 2 x (j a3 £e, a (X) 2x e = - - log x aeaa2 a3 a3 £e , a (X) 2 6x e =- +. aa 3 a 3 a4

-

0

-

------,----- -

5. 7

71

One-Step Estimators

For e and a ranging over sufficiently small neighborhoods of e0 and a0 , these functions are dominated by a function of the form M(x)

=

A ( l + x B ) ( l + l log xl +

·

·

·

+ l log x n ,

for sufficiently large A and B . Because the Weibull distribution has an exponentially small tail, the mixed moment Eeo,cro X P I log Xl q is finite for every p , q ::::_ Thus, all moments of le and ee exist and M is integrable. D

0.

*5.7

One-Step Estimators

The method of Z-estimation as discussed so far has two disadvantages. First, it may be hard to find the roots of the estimating equations. Second, for the roots to be consistent, the estimating equation needs to behave well throughout the parameter set. For instance, existence of a second root close to the boundary of the parameter set may cause trouble. The one-step method overcomes these problems by building on and improving a preliminary estimator iJ The idea is to solve the estimator from a linear approximation to the original estimating equation Wn (e) = Given a preliminary estimator en , the one-step estimator is the solution (in e) to n.

0.

This corresponds to replacing Wn (e) by its tangent at iJ n , and is known as the method of Newton-Rhapson in numerical analysis. The solution e = {Jn is In numerical analysis this procedure is iterated a number of times, taking {Jn as the new preliminary guess, and so on. Provided that the starting point iJ n is well chosen, the sequence of solutions converges to a root of Wn . Our interest here goes in a different direction. We suppose that the preliminary estimator {}n is already within range n - 1 12 of the true value of e. Then, as we shall see, just one iteration of the Newton-Rhapson scheme produces an estimator {J that is as good as the Z -estimator defined by \IIn . In fact, it is better in that its consistency is guaranteed, whereas the true Z-estimator may be inconsistent or not uniquely defined. In this way consistency and asymptotic normality are effectively separated, which is useful because these two aims require different properties of the estimating equations. Good initial estimators can be constructed by ad-hoc methods and take care of consistency. Next, these initial estimators can be improved by the one-step method. Thus, for instance, the good properties of maximum likelihood estimation can be retained, even in cases in which the consistency fails . In this section we impose the following condition on the random criterion functions \1111 • For every constant M and a given nonsingular matrix �0 , n

sup

.Jll i iB-Bo I I < M

1 -Jn ( Wn (e) - Wn (eo)) - �o-Jn (e - eo) I � 0 .

(5.44)

Condition (5 .44) suggests that Wn is differentiable at eo, with derivative tending to �o, but this is not an assumption. We do not require that a derivative �n exists, and introduce

72

M- and Z-Estimators

a further refinement of the Newton-Rhapson scheme by replacing �n (en) by arbitrary estimators. Given nonsingular, random matrices �n .o that converge in probability to �0 define the one-step estimator

Call an estimator sequence en Jfi-consistent if the sequence Jn(en - e0) is uniformly tight. The interpretation is that en already determines the value e0 within n -1 1 2 -range .

Let Jfi\lln (e0) Z and let (5. 44) hold. Then the one-step estimator en, for a given Jfi-consistent estimator sequence en and estimators . p . Wn , O_,. \llo , satisfies

5.45

5.46

Theorem (One-step estimation).

Addendum. Theorem 5. 21 with

Proof.

-v-7

For \lin (e) = fiDn 1/fe condition (5. 44) is satisfied under the conditions of �o = Ve0 , and under the conditions of Theorem 5. 41 with �o = P V, 80 •

The standardized estimator �n ,o Jfi(en - e0) equals

By (5 .44) the second term can be replaced by - �0Jfi(en - e0) + o p (1). Thus the expression can be rewritten as

The first term converges to zero in probability, and the theorem follows after application of Slutsky's lemma. For a proof of the addendum, see the proofs of the corresponding theorems. • If the sequence Jn(en - e0) converges in distribution, then it is certainly uniformly tight. Consequently, a sequence of one-step estimators is Jfi-consistent and can itself be used as preliminary estimator for a second iteration of the modified Newton-Rhapson algorithm. Presumably, this would give a value closer to a root of \lin . However, the limit distribution of this "two-step estimator" is the same, so that repeated iteration does not give asymptotic improvement. In practice a multistep method may nevertheless give better results. We close this section with a discussion of the discretization trick. This device is mostly of theoretical value and has been introduced to relax condition (5 .44) to the following. For every nonrandom sequence en = e0 + O (n -1 1 2 ), (5 .47) This new condition is less stringent and much easier to check. It is sufficiently strong if the preliminary estimators en are discretized on grids of mesh width n -1 1 2 . For instance, en is suitably discretized if all its realizations are points of the grid n -1 1 2 £} (consisting of the points n -1 1 2 (i 1 , . . . , i k ) for integers i 1 , . . . , i k ). This is easy to achieve, but perhaps unnatural. Any preliminary estimator sequence e n can be discretized by replacing its values

5. 7

One-Step Estimators

73

by the closest points of the grid. Because this changes each coordinate by at most n -112, Jn-consistency of en is retained by discretization. Define a one-step estimator Bn as before, but now use a discretized version of the pre­ liminary estimator.

5.48 Theorem (Discretized one-step estimation). Let Jfl\lln (e0) Z and let (5.47) hold. Then the one-step estimator en, for a given ,.Jii- consistent, discretized estimator sequence . . en and estimators \lln , O�p \llo , satisfies -v--7

5.49 Addendum. For \lin (e) = lP'n 1/Je and lP'n the empirical measure of a random sample from a. density pe that is differentiable in quadratic mean ( 5.38 ), condition ( 5.47), is satisfied, ·T with \llo = - Pe0 1/fel- eo ' if, as e � eo,

Proof.

The arguments of the previous proof apply, except that it must be shown that

converges to zero in probability. Fix £ > 0. By the ,.Jii - consistency, there exists M with P(Jn ll en - eo ll > M) < c . If Jll l l en - eo I ::::; M, then en equals one of the values in the set Sn = {e E n-1127J..k: 11 e - eo ll ::::; n - 1 1 2M } . For each M and n there are only finitely many elements in this set. Moreover, for fixed M the number of elements is bounded independently of n . Thus

::::; £ +

L P( II R (en) ll > c ) .

Bn E Sn

The maximum of the terms in the sum corresponds to a sequence of nonrandom vectors en with en = e0 + O (n -11 2) . It converges to zero by (5 .47). Because the number of terms in the sum is bounded independently of n, the sum converges to zero. For a proof of the addendum, see proposition A. 10 in [ 1 39] . • If the score function le of the model also satisfies the conditions of the addendum, then the estimators �n ,o = -Pen 1/Jen i�, are consistent for �o - This shows that discretized one-step estimation can be carried through under very mild regularity conditions. Note that the addendum requires only continuity of e f--+ 1/Je , whereas (5 . 47) appears to require differentiability.

5.50 Example (Cauchy distribution). Suppose X 1 , . . . , Xn are a sample from the Cauchy location family pe (x) = n-1 (1 + (x - e) 2 r 1 . Then the score function is given by _

2_ (x_ -_e_ )_

i e (x ) - 1 (x - e) 2 · + _

M- and Z -Estimators

74

0 0

0 LO

0 0

C';J

0 LO

C';J

0 0

'?

-200

- 1 00

0

1 00

Figure 5.4. Cauchy log likelihood function of a sample of 25 observations , showing three local maxima. The value of the absolute maximum is well-separated from the other maxima, and its location is close to the true value zero of the parameter.

This function behaves like 1 j x for x -+ ±oo and is bounded in between. The second moment of fe (X1 ) therefore exists , unlike the moments of the distribution itself. Because the sample mean possesses the same (Cauchy) distribution as a single observation X 1 , the sample mean is a very inefficient estimator. Instead we could use the median, or another M -estimator. However, the asymptotically best estimator should be based on maximum likelihood. We have . 4(x - 8) ((x - 8)2 - 3) fe (x) = ( 1 + (x - 8)2) 3 .:..._ .._ _=---'-

___

The tails of this function are of the order 1 j x3 , and the function is bounded in between. These bounds are uniform in 8 varying over a compact interval. Thus the conditions of Theorems 5.41 and 5 .42 are satisfied. Since the consistency follows from Example 5 . 1 6, the sequence of maximum likelihood estimators is asymptotically normal. The Cauchy likelihood estimator has gained a bad reputation, because the likelihood equation L.)e (Xd = 0 typically has several roots. The number of roots behaves asymp­ totically as two times a Poisson(ljn) variable plus 1 . (See [126] .) Therefore, the one-step (or possibly multi-step method) is often recommended, with, for instance, the median as the initial estimator. Perhaps a better solution is not to use the likelihood equations, but to deter­ mine the maximum likelihood estimator by, for instance, visual inspection of a graph of the likelihood function, as in Figure 5.4. This is particularly appropriate because the difficulty of multiple roots does not occur in the two parameter location-scale model. In the model with density P e (x fer ) fer , the maximum likelihood estimator for (8 , er) is unique. (See [25].) D

5.51 Example (Mixtures). Let f and g be given, positive probability densities on the real line. Consider estimating the parameter 8 = (M , v , er, r , p) based on a random sample from

the mixture density

X

1---+

pj

75

Rates of Convergence

5. 8

(X -- ) 1 + ( 1 - p)g (X - V) 1 · -a

f-L

-T- -;;

-;;

If f and g are sufficiently regular, then this is a smooth five-dimensional parametric model, and the standard theory should apply. Unfortunately, the supremum of the likelihood over the natural parameter space is oo , and there exists no maximum likelihood estimator. This is seen, for instance, from the fact that the likelihood is bigger than n V 1 f-L 1- n pf ( 1 - p)g -- - . r r a a i= Z If we set J-L = x 1 and next maximize over a > 0, then we obtain the value oo whenever p > 0, irrespective of the values of v and r . A one-step estimator appears reasonable in this example. In view of the smoothness of the likelihood, the general theory yields the asymptotic efficiency of a one-step estimator if started with an initial -fo-consistent estimator. Moment estimators could be appropriate initial estimators. D

(X! - )

*5.8

(Xi - )

Rates of Convergence

In this section we discuss some results that give the rate of convergence of M -estimators . These results are useful as intermediate steps in deriving a limit distribution, but also of interest on their own. Applications include both classical estimators of "regular" parameters and estimators that converge at a slower than .Jli-rate. The main result is simple enough, but its conditions include a maximal inequality, for which results such as in Chapter 1 9 are needed. Let JlDn be the empirical distribution of a random sample of size n from a distribution P , and, for every () in a metric space 8, let x �----+ me (x ) be a measurable function. Let (Jn (nearly) maximize the criterion function () �----+ Pn me . The criterion function may be viewed as the sum of the deterministic map () �----+ P me and the random fluctations 8 t---+ Pnme - Pme . The rate of convergence of en depends on the combined behavior of these maps. If the deterministic map changes rapidly as () moves away from the point of maximum and the random fluctuations are small, then (Jn has a high rate of convergence. For convenience of notation we measure the fluctuations in terms of the empirical process Gnme = -Jfl(JIDn me - Pme) .

5.52 Theorem (Rate of convergence). Assume that forfixed constants C and a every n , andfor every sufficiently small 8 > 0, sup P (me - me0 )

:S

E* sup I Gn (me - me0 ) 1

:S

d(e , eo)
>

fJ, for

- C8a ,

C8 !3 .

If the sequence en satisfies JIDn menA 2:. JIDnmeo - Op (nrx / ( Z/3 - Zrx ) ) and converges in outer probability to 8o, then n 1 / ( Za - Zf3 ) d (8n , 8o) = 0� ( 1 ).

76 Proof.

M- and Z -Estimators

Set rn

=

n 1 1 ( 2a - 2f3 ) and suppose that {jn maximizes the map e

variable R n = Op (r;; a ) .

f--+

lFnme up to a

For each n, the parameter space minus the point e0 can be partitioned into the "shells" SJ,n = {e : 2i - I < rnd(e , eo) :::::: 2i }, with j ranging over the integers. If rn d(en , eo) is larger than 2M for a given integer M, then {jn is in one of the shells S1 , n with j � M. In that case the supremum of the map e f--+ lFnme - lFnmea over this shell is at least R n by the property of en . Conclude that, for every E > 0, -

If the sequence {j n is consistent for e0, then the second probability on the right converges to 0 as n --* oo, for every fixed E > 0. The third probability on the right can be made arbitrarily small by choice of K , uniformly in n . Choose E > 0 small enough to ensure that the conditions of the theorem hold for every 8 :::=: E . Then for every j involved in the sum, we have 2 (j - l )a sup P (me - mea) :::=: -C

e ESJ ,n

For � C2CM - I ) a

�

---

r�

K , the series can be bounded in terms of the empirical process Gn by

by Markov's inequality and the definition of r11 • The right side converges to zero for every

M = Mn --*

oo .

•

Consider the special case that the parameter e is a Euclidean vector. If the map e f--+ P me is twice-differentiable at the point of maximum e0, then its first derivative at e0 vanishes and a Taylor expansion of the limit criterion function takes the form

Then the first condition of the theorem holds with a = 2 provided that the second-derivative matrix V is nonsingular. The second condition of the theorem is a maximal inequality and is harder to verify. In "regular" cases it is valid with f3 = 1 and the theorem yields the "usual" rate of convergence .Jfi. The theorem also applies to nonstandard situations and yields, for instance, the rate n 1 1 3 if a = 2 and fJ = � . Lemmas 1 9.34, 19.36 and 19.38 and corollary 1 9.35 are examples of maximal inequalities that can be appropriate for the present purpose. They give bounds in terms of the entropies of the classes of functions {me - mea : d ( e, eo) < 0}. A Lipschitz condition on the maps e f--+ me is one possibility to obtain simple estimates on these entropies and is applicable in many applications. The result of the following corollary is used earlier in this chapter.

5. 8

Rates of Convergence

77

5.53 Corollary. For each e in an open subset of Euclidean space let x f-+ m e (x) be a measurable function such that, for every e 1 and e2 in a neighborhood ofeo and a measurable function m such that Pm 2 < oo,

Furthermore, suppose that the map e f-+ P me admits a second-order Taylor expansion at the point ofmaximum eo with nonsingular second derivative. Jf lfDnmen 2:: lfDnme0 - 0 p ( n - 1 ) , p then � cen - eo) = 0 p (1 ) , provided that en -+ eo. A

A

By assumption, the first condition of Theorem 5 .52 is valid with a = 2. To see that the second one is valid with f3 = 1 , we apply Corollary 1 9.35 to the class of functions :F = { m e - mea : I e - eo II < This class has envelope function F = m , whence

Proof.

8}.

8

111mi P.28 J

E* sup 1 Gn (me - me0 ) 1 � log N[] (8, :F, L 2 (P)) d8. 0 ll e -Bo l d The bracketing entropy of the class :F is estimated in Example 1 9.7. Inserting the upper bound obtained there into the integral, we obtain that the preceding display is bounded above by a multiple of

111 m 1 P.20 F:(D8) 0

log - d 8. 8

Change the variables in the integral to see that this is a multiple of

8.

•

Rates of convergence different from ,fii are quite common for M -estimators of infinite­ dimensional parameters and may also be obtained through the application of Theorem 5.52. See Chapters 24 and 25 for examples. Rates slower than ,fii may also arise for fairly simple parametric estimates.

5.54 Example (Modal interval). Suppose that we define an estimator en oflocation as the center of an interval of length 2 that contains the largest possible fraction of the observations. This is an M-estimator for the functions me = 1 [ e - 1 ,8+1 1 . For many underlying distributions the first condition of Theorem 5.52 holds with a = 2. It suffices that the map e f-+ P me = P [ e - 1, e + 1] is twice-differentiable and has a proper maximum at some point e0. Using the maximal inequality Corollary 19.3 5 (or Lemma 1 9.38), we can show that the second condition is valid with f3 = � · Indeed, the bracketing entropy of the intervals in the real line is of the order j 8 2 , and the envelope function of the class of functions 1 [e - 1 ,8+1] - 1 [ ()o - 1 ,eo +1] as e ranges over ceo - eo + is bounded by 1 [eo - 1 - 8 ,eo - 1 HJ + 1 [eo +1 - Mo + 1 Hl • whose squared L 2 -norm is bounded by liP ll oo 28 . Thus Theorem 5.52 applies with a = 2 and f3 = � and yields the rate of convergence 1 n 13 • The resulting location estimator is very robust against outliers. However, in view of its slow convergence rate, one should have good reasons to use it. The use of an interval of length 2 is somewhat awkward. Every other fixed length would give the same result. More interestingly, we can also replace the fixed-length interval by the smallest interval that contains a fixed fraction, for instance 1 /2, of the observations. This

8

8, 8)

78

M- and Z-Estimators

still yields a rate of convergence of n 1 13 . The intuitive reason for this is that the length of a "shorth" settles down by a ,Jn-rate and hence its randomness is asymptotically negligible relative to its center. D The preceding theorem requires the consistency of en as a condition. This consistency is implied if the other conditions are valid for every 8 > 0, not just for small values of 8 . This can be seen from the proof or the more general theorem in the next section. Because the conditions are not natural for large values of 8 , it is usually better to argue the consistency by other means.

5.8.1

Nuisance Parameters

In Chapter 25 we need an extension of Theorem 5.52 that allows for a "smoothing" or "nuisance" parameter. We also take the opportunity to insert a number of other refinements, which are sometimes useful. Let x f--+ me, 11 (x) be measurable functions indexed by parameters (8 , YJ), and consider estimators e n contained in a set en that, for a given fin contained in a set Hn , maximize the map The sets en and Hn need not be metric spaces, but instead we measure the discrepancies between e n and 80, and �n and a limiting value YJo, by nonnegative functions 8 f--+ d11 ( 8 , 80) and YJ f--+ d(YJ, YJo), which may be arbitrary.

5.55 Theorem. Assume that, for arbitraryfunctions en : Bn x Hn f--+ lR and c/Jn : (0, oo) f--+ lR such that 8 f--+ c/Jn (8) j8� is decreasing for some f3 < 2, every (8 , YJ) E en x Hn, and every 8 > 0, P (me,11 - me0 , 11) + en (8 , YJ) � -d; (8 , 8o) + d 2 (YJ, YJo) , E* sup 1 Gn (me,11 - me0, 11 ) - Jll en (8 , YJ) I � c/Jn (o ) . d� (B.IJo)
(O,ry)E8n xHn

Let On > 0 satisfy c/Jn (8n) � ,Jn 8� for every n. If P (e n E Bn , fin E Hn ) l!Dnmen .iln :::: l!Dnmeo . iln - Op (o�), then diln ( e n , 8o ) = O � (o n + d ( fl n , YJo ) ) .

---+

1 and

For simplicity assume that l!Dn men . iln =::: l!Dn meo . iln • without a tolerance term. For each n E N, j E Z and M > 0, let Sn , J, M be the set { (8 , YJ) E Bn x Hn : 21 - 1 8n < d17 (8 , 8o ) � 21 8n, d(YJ, YJo) � 2- M d17 (8 , 8o ) } .

Proof.

Then the intersection ofthe events ( e n , fin) E en X Hn , and diln (en , 8o) :::: 2 M ( on +d( fl n , YJo)) is contained in the union of the events { ( e n , �n ) E Sn , J, M } over j =::: M. By the definition of en, the supremum of l!Dn (me, 11 - me0,11) over the set of parameters (8 , YJ) E Sn , J, M is nonnegative on the event { (e11 , fin ) E Sn , J ,M } . Conclude that P* dij. (en , 8o) =::: 2 M (8n + d ( fl n , YJo) ) , (en , fin) E Bn x Hn

(

�

I: r *

j�M

(

(

sup

ti , I) ) ESn 1 M

)

)

l!Dn (me,11 - me0,17) :::: 0 .

Argmax Theorem

5. 9

For every j , (e, rJ )

E

79

Sn,J ,M · and every sufficiently large M,

-d; (e, eo) + d 2 ( rJ , rJo) < - (1 - 2 - 2M ) d1)2 (e ' e0 ) < -2 21 -4 8n2 . From here on the proof is the same as the proof of Theorem 5 . 52, except that we use that ¢>n (c8) ::::; cf3¢>n (8) for every c > 1 , by the assumption on tf>n · • P (m e ,l) - me0,1) ) + en (e , rJ )

::::;

_

*5.9

Argmax Theorem

The consistency of a sequence of M -estimators can be understood as the points of maximum en of the criterion functions e r--+ Mn (e) converging in probability to a point of maximum of the limit criterion function e r--+ M(e). So far we have made no attempt to understand the distributional limit properties of a sequence of M -estimators in a similar way. This is possible, but it is somewhat more complicated and is perhaps best studied after developing the theory of weak convergence of stochastic processes, as in Chapters 1 8 and 19. Because the estimators en typically converge to constants, it is necessary to rescale them before studying distributional limit properties. Thus, we start by searching for a sequence of numbers rn r--+ oo such that the sequence hn = rn (en - e) is uniformly tight. The results of the preceding section should be useful. If en maximizes the function e r--+ Mn (e), then the rescaled estimators hn are maximizers of the local criterion functions

h

r--+

( �) - Mn (eo) .

Mn e +

Suppose that these, if suitably normed, converge to a limit process h r--+ M ( h ) . Then the general principle is that the sequence hn converges in distribution to the maximizer of this limit process. For simplicity of notation we shall write the local criterion functions as h r--+ Mn ( h ) . Let { Mn (h) : h E Hn } be arbitrary stochastic processes indexed by subsets Hn of a given metric space. We wish to prove that the argmax-functional is continuous : If Mn -v-+ M and Hn ----+ H in a suitable sense, then the (near) maximizers hn of the random maps h r--+ Mn (h) converge in distribution to the maximizer h of the limit process h r--+ M (h) . It is easy to find examples in which this is not true, but given the right definitions it is, under some conditions. Given a set B , set

M (B )

=

sup M (h ) . hEB

Then convergence in distribution of the vectors ( Mn (A) , Mn ( B ) ) for given pairs of sets A and B is an appropriate form of convergence of Mn to M . The following theorem gives some flexibility in the choice of the indexing sets. We implicitly either assume that the suprema Mn (B ) are measurable or understand the weak convergence in terms of outer probabilities, as in Chapter 18. The result we are looking for is not likely to be true if the maximizer of the limit process is not well defined. Exactly as in Theorem 5 .7 , the maximum should be "well separated." Because in the present case the limit is a stochastic process, we require that every sample path h r--+ M (h ) possesses a well-separated maximum (condition (5.57)).

80

M- and Z -Estimators

5.56 Theorem (Argmax theorem). Let Mn and M be stochastic processes indexed by sub­ sets Hn and H of a given metric space such that, for every pair of a closed set F and a set K in a given collection K, Furthermore, suppose that every sample path of the process h � M(h) possesses a well­ separated point of maximum h in that, for every open set G and every K E K, M( h ) > M(G c n K n H) ,

if h E G,

a.s ..

If Mn ( h n) :=:: Mn (Hn) - o p ( 1 ) andfor every 8 > 0 there exists K E K) < 8 and P( h � K) < 8, then h n "-""+ h .

K such that sup n

(5 .57) P( h n

�

If h n E F n K , then Mn (F n K n Hn ) :::: Mn (B ) - op (l) for any set B . Hence, for every closed set F and every K E K ,

Proof.

P( h n E F n K)

:S

:S

P(Mn (F n K n Hn) :::: Mn (K n Hn) - op ( l ) ) P(M(F n K n H) :::: M(K n H) ) + o ( l ) ,

by Slutsky's lemma and the portmanteau lemma. If h E F e , then M ( F n K n H ) i s strictly smaller than M (h ) by (5.57) and hence on the intersection with the event in the far right side h cannot be contained in K n H . It follows that lim sup P( h n E F n K)

:S

P(h E F) + P( h

tf.

K n H) .

By assumption we can choose K such that the left and right sides change by less than 8 if we replace K by the whole space. Hence h n "-""+ h by the portmanteau lemma. • The theorem works most smoothly if we can take K to consist only of the whole space. However, then we are close to assuming some sort of global uniform convergence of Mn to M, and this may not hold or be hard to prove. It is usually more economical in terms of conditions to show that the maximizers h n are contained in certain sets K , with high probability. Then uniform convergence of Mn to M on K is sufficient. The choice of compact sets K corresponds to establishing the uniform tightness of the sequence h n before applying the argmax theorem. If the sample paths of the processes Mn are bounded on K and Hn = H for every n, then the weak convergence of the processes Mn viewed as elements of the space .C00 (K) implies the convergence condition of the argmax theorem. This follows by the continuous-mapping theorem, because the map

z � (z (A n K), z (B n K) ) from .C00 (K) to IR2 is continuous, for every pair of sets A and B . The weak convergence in .C00 (K) remains sufficient if the sets Hn depend on n but converge in a suitable way. Write Hn --+ H if H is the set of all limits lim hn of converging sequences hn with hn E Hn for every n and, moreover, the limit h = limi hn, of every converging sequence hn, with hn, E Hn, for every i is contained in H .

5. 9

Argmax Theorem

81

5.58 Corollary. Suppose that M11 "'-"+ M in £00 ( K) for every compact subset K of IRk , for a limit process M with continuous sample paths that have unique points of maxima h . If H11 � H, M11 (h 11) :=: M11 (H11 ) - o p ( l ) , and the sequence h 11 is unzformly tight, then h 11 "'-"+ h . The compactness of K and the continuity of the sample paths h f-+ M (h) imply that the (unique) points of maximum h are automatically well separated in the sense of (5 .57). Indeed, if this fails for a given open set G 3 h and K (and a given w in the underlying probability space), then there exists a sequence h m in G c n K n H such that M( h m ) � M( h ) . If K is compact, then this sequence can be chosen convergent. The limit ho must be in the closed set G c and hence cannot be h . By the continuity of M it also has the property that M (h0 ) = lim M( h m ) = M ( h ) . This contradicts the assumption that h is a unique point of maximum. If we can show that (M11 ( F n Hn) , M11 ( K n Hn)) converges to the corresponding limit for every compact sets F c K, then the theorem is a corollary of Theorem 5.56. If H11 = H for every n, then this convergence is immediate from the weak convergence of M11 to M in .C00(K), by the continuous-mapping theorem. For H11 changing with n this convergence may fail, and we need to refine the proof of Theorem 5.56. This goes through with minor changes if

Proof.

lim sup P(M11 (F n H11 ) - M11 ( K H11) ::: x ) ::: P(M(F n H) - M ( K n H) ::: x ) , 11 � 00 for every x , every compact set F and every large closed ball K . Define functions g11 : £00 (K) f-+ lR by �"',

gn (Z) = sup z (h) - sup z (h) , hEFnH,, hEKnH, and g similarly, but with H replacing H11 • By an argument as in the proof of Theo­ rem 1 8 . 1 1 , the desired result follows if lim sup g11 (Z11) ::: g(z) for every sequence Z n � z in .C00 (K) and continuous function z . (Then lim sup P(g11 (M11) ::: x) ::: P(g(M) ::: x) for every x , for any weakly converging sequence M11 "'-"+ M with a limit with continuous sample paths.) This in turn follows if for every precompact set B c K, sup z (h) ::: lim sup Z11 (h) ::: sup z (h). n � oo hEBnH, hEBnH To prove the upper inequality, select h11 E B n H11 such that sup Z11 (h) = Z11 (h11) + o(l) = z (h11) + o ( l ) . hEBnH,, Because B is compact, every subsequence of h11 has a converging subsequence. Because H11 � H, the limit h must be in B n H. Because z (h11) � z (h), the upper bound follows. To prove the lower inequality, select for given 8 > 0 an element h E B n H such that sup z (h) ::: z (h) + 8. hEBnH

Because H11 � H , there exists h11 E H11 with h11 � h. This sequence must be in B C B eventually, whence z(h) = lim z (h11) = lim Zn (h11) is bounded above by lim inf sup hEBnH, Zn (h) . •

M- and Z-Estimators

82

The argmax theorem can also be used to prove consistency, by applying it to the original criterion functions (] f--+ Mn (e). Then the limit process (] f--+ M(e) is degenerate, and has a fixed point of maximum e0. Weak convergence becomes convergence in probability, and the theorem now gives conditions for the consistency {Jn !,. e0. Condition (5.57) reduces to the well-separation of e0, and the convergence sup

e eFnK ne.

Mn (e) �p

sup

e eFnKne

Mn (e)

is, apart from allowing 8n to depend on n , weaker than the uniform convergence of Mn to

M.

Notes

In the section on consistency we have given two main results (uniform convergence and Wald's proof) that have proven their value over the years, but there is more to say on this subject. The two approaches can be unified by replacing the uniform convergence by "one­ sided uniform convergence," which in the case of i.i.d. observations can be established under the conditions of Wald's theorem by a bracketing approach as in Example 19.8 (but then one-sided). Furthermore, the use of special properties, such as convexity of the 1j; or m functions, is often helpful. Examples such as Lemma 5 . 1 0, or the treatment of maximum likelihood estimators in exponential families in Chapter 4, appear to indicate that no single approach can be satisfactory. The study of the asymptotic properties of maximum likelihood estimators and other M -estimators has a long history. Fisher [48], [50] was a strong advocate of the method of maximum likelihood and noted its asymptotic optimality as early as the 1920s. What we have labelled the classical conditions correspond to the rigorous treatment given by Cramer [27] in his authoritative book. Huber initiated the systematic study of M -estimators, with the purpose of developing robust statistical procedures. His paper [78] contains important ideas that are precursors for the application of techniques from the theory of empirical processes by, among others, Pollard, as in [ 1 17], [ 1 1 8], and [ 1 20] . For one-dimensional parameters these empirical process methods can be avoided by using a maximal inequality based on the L 2 -norm (see, e.g., Theorem 2.2.4 in [146]). Surprisingly, then a Lipschitz condition on the Hellinger distance (an integrated quantity) suffices; see for example, [80] or [94] . For higher-dimensional parameters the results are also not the best possible, but I do not know of any simple better ones. The books by Huber [79] and by Hampel, Ronchetti, Rousseeuw, and Stahel [73] are good sources for applications of M -estimators in robust statistics. These references also discuss the relative efficiency of the different M -estimators, which motivates, for instance, the use of Huber's 1/J -function. In this chapter we have derived Huber's estimator as the solution of the problem of minimizing the asymptotic variance under the side condition of a uniformly bounded influence function. Originally Huber derived it as the solution to the problem of minimizing the maximum asymptotic variance sup P a� for P ranging over a contamination neighborhood: P = ( 1 c:)

Problems

83

their asymptotic efficiency by Le Cam in 1 956, who later developed them for general locally asymptotically quadratic models, and also introduced the discretization device, (see [93]). PROBLEMS 1. Let X1 , . . . , Xn be a sample from a density that is strictly positive and symmetric about some

point. Show that the Huber M -estimator for location is consistent for the symmetry point.

2. Find an expression for the asymptotic variance of the Huber estimator for location if the obser­

vations are normally di stributed. 3. Define 1/f (x ) = 1 - p , 0, p if x e) ::S: p ::S: P(X ::S: e ) .

<

0, 0,

>

0. Show that E1/f (X - e) = 0 implies that P(X

<

4. Let X J , . . . , Xn b e i.i.d. N ( � , cr 2 )-distributed. Derive the maximum likelihood estimator for

(� , cr 2 ) and show that it is asymptotically normal . Calculate the Fisher information matrix for this parameter and its inverse.

5. Let X 1 , . . . , Xn be i.i.d. Poisson ( l j e )-distributed. Derive the maximum likelihood estimator for

e and show that it is asymptotically normal.

6. Let X1 , . . . , Xn be i.i.d. N ( e , e) -distributed. Derive the maximum likelihood estimator for e and show that it is asymptotically normal.

7. Find a sequence of fixed (nonrandom) functions Mn : lR H- lR that converges pointwise to a limit Mo and such that each Mn has a unique maximum at a point en , but the sequence en does not converge to eo . Can you also find a sequence Mn that converges uniformly?

8. Find a sequence of fixed (nonrandom) functions Mn : lR H- lR that converges pointwise but not uniformly to a limit Mo such that each Mn has a unique maximum at a point en and the sequence en converges to eo . 9. Let X 1 , . . . , Xn be i.i.d. observations from a uniform distribution on [0, e ] . Show that the sequence of maximum likelihood estimators is asymptotically consistent. Show that it is not asymptotically normal.

10. Let X1 , . . . , Xn be i.i.d. observations from an exponential density e exp ( - e x ) . Show that the

sequence of maximum likelihood estimators is asymptotically normal. 1 11. Let JF;;- (p) be a pth sample quantile of a sample from a cumulative distribution F on JR. that is 1 differentiable with positive derivative at the population pth-quantile p - (p) = inf x : F (x ) ::=: 1 p . Show that .fii JF;;- 1 (p) - p - (p) is asymptotically normal with mean zero and variance p ( l - p)/f F - I (p)

}

(

(

f

)

{

12. Derive a minimal condition on the distribution function F that guarantees the consistency of the

sample pth quantile. 13. Calculate the asymptotic variance of .jii (en - e ) in Example 5 .26. 14. Suppose that we observe a random sample from the distribution of (X, Y) in the following

errors-in-variables model: X = Z+e

Yi = a + f3 Z + f, where (e, f) is bivariate normally distributed with mean 0 and covariance matrix cr 2 I and is independent from the unobservable variable Z . In analogy to Example 5 .26, construct a system of estimating equations for (a, (3) based on a conditional likelihood, and study the limit properties of the corresponding estimators. 15. In Example 5 .27, for what point is the least squares estimator iJn consistent if we drop the

condition that E(e I X) = 0? Derive an (implicit) solution in terms of the function E(e I X ) . Is it necessarily eo if Ee = 0?

M - and Z-Estimators

84

16. In Example 5 .27, consider the asymptotic behavior of the least absolute-value estimator e that minimizes L: 7= 1 1 Y - ¢ e (X i ) I ·

i

17. Let X 1 , . . . , Xn be i.i.d. with density h. , a (x ) = A.e - f. (x - a) l {x � a ) , where the parameters A. > 0 and a E are unknown. Calculate the maximum likelihood estimator of (A. , a) and derive its asymptotic properties.

(An , fin)

lR

X be Poisson-distributed with density pe (x) = e x e - e jx ! . Show by direct calculation that Ee fe (X) 0 and Eele (X) = - Ee f � (X). Compare this with the assertions in the introduction.

18. Let

=

Apparently, differentiation under the integral (sum) is permitted in this case. Is that obvious from results from measure theory or (complex) analysis?

19. Let X 1 , . . . , Xn be a sample from the N ( e , 1) distribution, where it is known that e � 0. Show that the maximum likelihood estimator is not asymptotically normal under e = 0. Why does this not contradict the theorems of this chapter?

20. Show that (e - eo)\lln (en ) in formula (5 . 1 8) converges in probability to zero if en � eo, and that there exists an integrable function M and 8 > 0 with I Vf e (x ) l :::: M (x) for every x and every

11 e - eo I

<

8.

21. If en maximizes Mn , then it also maximizes M;t . Show that this may be used to relax the conditions of Theorem 5 .7 to sup e I M;t - M + l (e) ---+ 0 in probability (if M(eo) > 0) .

22. Suppose that for every s > O there exists a set 88 with lim inf P(en E 88) � 1 - s . Then uniform convergence of Mn to M in Theorem 5 .7 can be relaxed to uniform convergence on every 88 • 23. Show that Wald 's consistency proof yields almost sure convergence of e n , rather than convergence in probability if the parameter space is compact and Mn (en ) � Mn (Bo) - o ( l ) .

24. Suppose that (X 1 , Y1 ) , . . . , (Xn , Yn ) are i.i.d. and satisfy the linear regression relationship Yi = e T Xi + ei for (unobservable) errors e 1 , . . . , en independent of X 1 , . . . , Xn . Show that the mean absolute deviation estimator, which minimizes I: I Yi - ()Xi I , is asymptotically normal under a mild condition on the error distribution. 25.

(i) Verify the conditions of Wald 's theorem for me the log likelihood function of the N (f.L , distribution if the parameter set for ()

=

(f.L,

a 2 ) is a compact subset of lR

(ii) Extend me by continuity to the compactification of lR Wald's theorem fail at the points ( f.L ,

0).

x

JR + .

x

JR + .

a2)

­

Show that the conditions of

(iii) Replace m e by the log likelihood function of a pair of two independent observations from the N ( f.L ,

a 2 )-distribution. Show that Wald 's theorem now does apply, also with a compactified

parameter set.

IRk

26. A distribution on is called ellipsoidally symmetric if it has a density of the form x �---* T g ( (x - f.L) L: - 1 (x - tL)) for a function g : [0, oo) �---* [0, oo), a vector f.L, and a symmetric positive-definite matrix 2:: . Study the Z-estimators for location fl that solve an equation of the form

i =l for given estimators tn and, for instance, Huber's ljr-function. Is the asymptotic distribution of tn important?

27. Suppose that B is a compact metric space and M : B ---+ lR is continuous. Show that (5 .8) is equivalent to the point eo being a point of unique global maximum. Can you relax the continuity of M to some form of "semi-continuity"?

6 Contiguity

"Contiguity " is another name for "asymptotic absolute continuity." Contiguity arguments are a technique to obtain the limit distribution of a sequence of statistics under underlying laws Qn from a limiting distribution under laws Pn. Typically, the laws Pn describe a null distri­ bution under investigation, and the laws Qn correspond to an alternative hypothesis.

6.1

Likelihood Ratios

Let P and Q be measures on a measurable space (Q , A) . Then Q is absolutely continuous with respect to P if P (A) = 0 implies Q (A) = 0 for every measurable set A; this is denoted by Q « P . Furthermore, P and Q are orthogonal if Q can be partitioned as Q = Qp U QQ with Qp n Q Q = 0 and P (QQ) = 0 = Q(Qp ) . Thus P "charges" only Qp and Q "lives on" the set Q Q , which is disjoint with the "support" of P . Orthogonality is denoted by p j_ Q . In general, two measures P and Q need be neither absolutely continuous nor orthogonal. The relationship between their supports can best be described in terms of densities. Suppose P and Q possess densities p and q with respect to a measure p, , and consider the sets Qp

=

QQ

{p > 0},

=

{q > 0} .

See Figure 6. 1 . Because P (Q� ) = Jp =O p d p, = 0, the measure P is supported on the set Qp . Similarly, Q is supported on Q Q . The intersection Qp n Q Q receives positive measure from both P and Q provided its measure under p, is positive. The measure Q can be written as the sum Q = Qa + Ql_ of the measures Q j_ (A) = Q (A n {p = 0}) . (6. 1 ) A s proved in the next lemma, Q a set A

«

P and Q j_

l_

P . Furthermore, for every measurable

{ 9:__ d P . }A p The decomposition Q = Qa + Q l_ is called the Lebesgue decomposition of Q with respect to P . The measures Qa and Q j_ are called the absolutely continuous part and the orthogonal Q a (A)

=

85

Contiguity

86 Q

p> O q=O

p =O q>O

p> O q>O

p =q= O Figure 6.1. Supports of measures.

part (or singular part) of Q with respect to P , respectively. In view of the preceding display, the function q I p is a density of Qa with respect to P . It is denoted d Q I d P (not: d Qa I d P ), so that dQ q P - a.s. dP p As long as we are only interested in the properties of the quotient q I p under P -probability, we may leave the quotient undefined for p = 0. The density d Q I d P is only P -almost surely unique by definition. Even though we have used densities to define them, d Q I d P and the Lebesgue decomposition are actually independent of the choice of densities and dominating measure. In statistics a more common name for a Radon-Nikodym density is likelihood ratio. We shall think of it as a random variable d Q I d P : Q f-+ [0, oo) and shall study its law under P .

6.2 Lemma. Let P and Q be probability measures with densities p and q with respect to a measure f-L. Then for the measures Q a and Q j_ defined in (22. 30) (i) Q = Q a + Q j_ , Qa « P, Q j_ ..l P. (ii) Q a (A) = JA (q lp) d P for every measurable set A. (iii) Q « P if and only if Q (p = 0) = 0 if and only if j (q I p) d P = 1. The first statement of (i) is obvious from the definitions of Qa and Q j_ . For the second, we note that P (A) can be zero only if p (x ) = 0 for M -almost all x E A. In this case, M ( A n {p > 0}) = 0, whence Q a (A) = Q ( A n {p > 0}) = 0 by the absolute continuity of Q with respect to f-L. The third statement of (i) follows from P (p = 0) = 0 and Q j_ (p > 0) = Q( 0 ) = 0. Statement (ii) follows from q q q dt-L = Q a (A) = p df.-L = - d P . Ap An{ p>O} An{ p>O} p For (iii) we note first that Q « P if and only if Q j_ = 0. By (22.30) the latter happens if and only if Q (p = 0) = 0. This yields the first "if and only if." For the second, we note

Proof.

1

1

1

6. 2

87

Contiguity

that by (ii) the total mass of Qa is equal to Qa (Q) = J (q I p) d P . This is 1 if and only if Qa = Q • .

It is not true in general that J f d Q = J f (d Q I d P) d P . For this to be true for every measurable function f , the measure Q must be absolutely continuous with respect to P . On the other hand, for any P and Q and nonnegative f,

This inequality is used freely in the following. The inequality may be strict, because dividing by zero is not permitted. t 6.2

Contiguity

If a probability measure Q is absolutely continuous with respect to a probability measure P , then the Q-law of a random vector X : Q f---+ IRk can be calculated from the P -law of the pair (X, d Qid P) through the formula EQ J (X)

With p x , v equal to the law of the pair can also be expressed as Q (X

E

= E p f ( X)

(X,

V)

dQ . dP

= (X,

dQ B) = E p l s (X) - = dP

1

d Q id P) under P, this relationship

BxR

v

dP x, v (x , v ) .

The validity of these formulas depends essentially on the absolute continuity of Q with respect to P , because a part of Q that is orthogonal with respect to P cannot be recovered from any P -law. Consider an asymptotic version of the problem. Let (Qn , An) be measurable spaces, each equipped with a pair of probability measures Pn and Qn . Under what conditions can a Qn -limit law of random vectors Xn : Qn f---+ IRk be obtained from suitable Pn-lirnit laws? In view of the above it is necessary that Qn is "asymptotically absolutely continuous" with respect to Pn in a suitable sense. The right concept is contiguity. Definition. The sequence Qn is contiguous with respect to the sequence Pn if Pn (An ) -+ 0 implies Qn CAn) -+ 0 for every sequence of measurable sets An . This is denoted Qn <J Pn · The sequences Pn and Qn are mutually contiguous if both Pn <J Qn and Qn Qn . 6.3

The name "contiguous" is standard, but perhaps conveys a wrong image. "Contiguity" suggests sequences of probability measures living next to each other, but the correct image is "on top of each other" (in the limit) . t

The algebraic identify d Q (d Ql d P) d P is false. because the notation d Ql d P is used as shorthand for d Qa I d P: If we write d Q I d P, then we are not implicitly assuming that Q « P . =

Contiguity

88

Before answering the question of interest, we give two characterizations of contiguity in terms of the asymptotic behavior of the likelihood ratios of Pn and Qn . The likelihood ratios d QnldPn and d Pnld Qn are nonnegative and satisfy d Pn < 1. and EQ n -- d Qn Thus, the sequences of likelihood ratios d Qnl d Pn and d Pn l d Qn are uniformly tight under Pn and Qn, respectively. By Prohorov's theorem, every subsequence has a further weakly converging subsequence. The next lemma shows that the properties of the limit points determine contiguity. This can be understood in analogy with the nonasymptotic situation. For probability measures P and Q, the following three statements are equivalent by (iii) of Lemma 6.2: dQ Ep - = 1 . Q « P, dP This equivalence persists if the three statements are replaced by their asymptotic counter­ parts: Sequences Pn and Qn satisfy Qn <1 Pn , if and only if the weak limit points of d Pn I d Qn under Qn give mass 0 to 0, if and only if the weak limit points of d Qn l d Pn under Pn have mean 1 .

Let Pn and Q n be sequences ofprobability measures on measurable spaces (Qn , An). Then the following statements are equivalent: (i) Qn 0) = 1 . (iii) If d Q n I d Pn � V along a subsequence, then E V = 1 . ( iv) For any statistics Tn : Q n f--+ IRk : If Tn � 0, then Tn £.;. 0. 6.4

Lemma (Le Cam 's first lemma).

The equivalence of (i) and (iv) follows directly from the definition of contiguity: Given statistics Tn , consider the sets An = { II Tn I > £ } ; given sets An, consider the statistics Tn = l A n · (i) :::} (ii) . For simplicity of notation, we write just {n} for the given subsequence along which d Pn ld Qn � U . For given n, we define the function gn ( £ ) = Qn (d Pn ld Qn < c) - P(U < £). By the portmanteau lemma, lim inf g11 (£) � 0 for every £ > 0. Then, for En + 0 at a sufficiently slow rate, also lim inf g11 (£11) � 0. Thus,

Proof.

P(U = 0) = lim P(U < En) ::::; lim inf Qn On the other hand,

(:�: < En ) .

If Qn is contiguous with respect to Pn , then the Qn -probabi1ity of the set on the left goes to zero also. But this is the probability on the right in the first display. Combination shows that P(U = 0) = 0. (iii) :::} (i). If Pn (An) -+ 0, then the sequence l Q" - A " converges to 1 in Pn-probability. By Prohorov's theorem, every subsequence of {n} has a further subsequence along which

6. 2

89

Contiguity

(d Qn/dPn , 1r.lcAJ """' (V, 1) under Pn , for some weak limit V . The function (v, t) f-+ vt is continuous and nonnegative on the set [0, oo) x {0, 1}. By the portmanteau lemma lim inf Qn (Qn - An) :::: lim inf

J 1r.ln -A,. �;: d Pn :::: E1 ·

V.

Under (iii) the right side equals E V = 1. Then the left side is 1 as well and the sequence Qn (An) = 1 - Qn (Qn - An ) converges to zero. (ii) ::::} (iii). The probability measures f.Ln = 1 CPn + Qn) dominate both Pn and Qn, for every n . The sum of the densities of Pn and Qn with respect to f.Ln equals 2. Hence, each of the densities takes its values in the compact interval [0, 2] . By Prohorov's theorem every subsequence possesses a further subsequence along which d Pn & d Qn !::, dPn &, W, V' n := U' W d Qn d Pn df.Ln for certain random variables U, V and W. Every Wn has expectation 1 under f.Ln . In view of the boundedness, the weak convergence of the sequence Wn implies convergence of moments, and the limit variable has mean E W = 1 as well. For a given bounded, continuous function j , define a function g : [0, 2] f-+ IR by g (w) = f( w / (2- w) ) (2- w) for 0 � w < 2 and g (2) = 0. Then g is bounded and continuous. Because d Pn /d Qn = Wn/(2 - Wn ) and d Q n I d f.Ln = 2 - Wn ' the portmanteau lemma yields Pn = Pn d Qn = W (2 - W), EJ.Ln g(Wn) ---+ E j E!Ln j dd Qn E Q ,J dd Qn df.Ln 2- W where the integrand in the right side is understood to be g (2) = 0 if W = 2. By assumption, the left side converges to Ef(U). Thus Ef(U) equals the right side of the display for every continuous and bounded function f. Take a sequence of such functions with 1 :::: fm + 1 {OJ , and conclude by the dominated-convergence theorem that

( )

(

0)

(

=

)

)

E1r OJ (U) = E1r OJ 2 -WW (2 - W) = 2P(W = 0) . By a similar argument, Ef(V) = Ef((2 - W)/ W) W for every continuous and bounded function f, where the integrand on the right is understood to be zero if W = 0. Take a sequence 0 � fm (x) t x and conclude by the monotone convergence theorem that E V = E 2 �W w = E(2 - W) 1w> o = 2P(W > 0) - 1 . P(U

=

( )

(

)

Combination of the last two displays shows that P(U

=

0) + E V

=

1.

•

6.5 Example (Asymptotic log normality). The following special case plays an important role in the asymptotic theory of smooth parametric models. Let Pn and Qn be probability measures on arbitrary measurable spaces such that d Pn £.+ e N ( J.L , a 2 ) d Qn Then Qn <1 Pn . Furthermore, Qn <1 r;;. Pn if and only if f.L = -1cr 2 . Because the (log normal) variable on the right is positive, the first assertion is immediate from (ii) of the theorem. The second follows from (iii) with the roles of P11 and Q11 switched, on noting that E exp N (J.L, cr 2 ) = 1 if and only if J.L = -1cr 2 .

Contiguity

90

A mean equal to minus half times the variance looks peculiar, but we shall see that this sit­ uation arises naturally in the study of the asymptotic optimality of statistical procedures . D The following theorem solves the problem of obtaining a Q11-limit law from a P11 -limit law that we posed in the introduction. The result, a version of Le Cam 's third lemma, is in perfect analogy with the nonasymptotic situation.

6.6 Theorem. Let Pn and Q n be sequences ofprobability measures on measurable spaces (Q11 , A11), and let Xn : Qn r----+ IRk be a sequence of random vectors. Suppose that Qn <1 Pn and Xn , d Q n � (X V ) . d Pn

(

)

'

Then L(B) = El s (X) V defines a probability measure, and Xn f?4 L. Because V � 0, it follows with the help of the monotone convergence theorem that L defines a measure. By contiguity, EV = 1 and hence L is a probability measure. It is immediate from the definition of L that J f dL = Ef(X) V for every measurable indicator function f. Conclude, in steps, that the same is true for every simple function f, any nonnegative measurable function, and every integrable function. If f is continuous and nonnegative, then so is the function (x , v) r----+ f (x) v on IRk x [0, oo) . Thus

Proof.

lim inf E Q n f(X11)

�

lim inf

J f(X11) �;: dPn

�

Ef(X) V,

by the portmanteau lemma. Apply the portmanteau lemma in the converse direction to conclude the proof that Xn f?4 L . •

6.7 Example (Le Cam 's third lemma). The name Le Cam 's third lemma is often reserved for the following result. If

then In this situation the asymptotic covariance matrices of the sequence Xn are the same under Pn and Q11 , but the mean vectors differ by the asymptotic covariance T between Xn and the log likelihood ratios. t The statement is a special case of the preceding theorem. Let (X, W) have the given (k + 1 )-dimensional normal distribution. By the continuous mapping theorem, the sequence (X11 , d Q11 jd P11 ) converges in distribution under Pn to (X, e w ) . Because W is N ( - � a 2 , a 2 )­ distributed, the sequences Pn and Qn are mutually contiguous. According to the abstract t

We set log 0 - oo ; because the normal distribution does not charge the point -oo the assumed asymptotic normality of log d Q11/d P11 includes the assumption that Pn (d Qn/ d Pn 0) -+ 0. =

=

91

Problems

version of Le Cam's third lemma, X n & L with L(B) = Els (X)e w . The characteristic function of L is J e i t T dL (x) = Ee i t T x e w . This is the characteristic function of the given normal distribution at the vector (t, -i). Thus

x

The right side is the characteristic function of the Nk (J.L + T, L:) distribution.

0

Notes

The concept and theory of contiguity was developed by Le Cam in [92] . In his paper the results that were later to become known as Le Cam's lemmas are listed as a single theorem. The names "first" and "third" appear to originate from [7 1]. (The second lemma is on product measures and the first lemma is actually only the implication (iii) ::::} (i) .)

PROBLEMS

1. Let P = N (O, 1) and = N ( �J., , 1 ) . Show that the sequences Pn and contiguous if and only if the sequence is bounded .

n

Qn

Qn

n IJ.,n

Qn

are mutually

2. Let Pn and be the distribution of the mean of a sample of size n from the N (0, 1 ) and the N 1 ) distribution, respectively. Show that P if and only if = 0 ( 1 I ,Jfi) .

(()n,

n <1 1> Qn

Qn

en

3. Let Pn and be the law of a sample of size n from the uniform distribution on [0, 1 ] or [0, 1 + 1 / n ] , respectively. Show that P Is it also true that Pn ? U s e Lemma 6.4 t o derive your answers .

n <1 Qn .

Qn <1

Pn - Qn I 0, where I · II is the total variation distance II P - Q II = sup A I P ( A ) Pn <1 1> Qn. 1 - £ . (The 0 find an example of sequences such that Pn <1 1> Qn, but I Pn - Qn I

4. Suppose that I Show that

Q(A) I ·

---+

­

---+ 5. Given £ > maximum total variation distance between two probability measures is 1 .) This exercise shows that it is wrong to think of contiguous sequences as being close. (Try measures that are supported on just two points .)

n <1 Qn, but it is not true that Qn <1 Pn . Show that the constant sequences { P } and { Q} are contiguous if and only if P and Q are absolutely

6. Give a simple example in which P 7.

continuous .

8. If P « then ---+ 0 implies P ( does this follow from Lemma 6.4?

Q,

Q(An)

An )

---+

0 for every sequence o f measurable sets . How

7 Local Asymptotic Normality

A sequence of statistical models is "locally asymptotically normal " if, asymptotically, their likelihood ratio processes are similar to those for a normal location parameter. Technically, this is if the likelihood ratio processes admit a certain quadratic expansion. An important example in which this arises is repeated sampling from a smooth parametric model. Local asymptotic normality implies convergence of the models to a Gaussian model after a rescaling of the parameter.

7.1

Introduction

Suppose we observe a sample X 1, . . . , Xn from a distribution Pe on some measurable space (X, A) indexed by a parameter e that ranges over an open subset 8 of IRk . Then the full observation is a single observation from the product P; of n copies of Pe , and the statis­ tical model is completely described as the collection of probability measures { Pen : e E 8} on the sample space ( xn , An ). In the context of the present chapter we shall speak of a statistical experiment, rather than of a statistical model. In this chapter it is shown that many statistical experiments can be approximated by Gaussian experiments after a suitable reparametrization. The reparametrization is centered around a fixed parameter 80, which should be regarded as known. We define a local parameter h = ,fii ( 8 - 80) , rewrite P; as P8: +h / .fo ' and thus obtain an experiment with parameter h. In this chapter we show that, for large n, the experiments are similar in statistical properties, whenever the original experiments e �----+ Pe are "smooth" in the parameter. The second experiment consists of observing a single observation from a normal distribution with mean h and known covariance matrix (equal to the inverse of the Fisher information matrix) . This is a simple experiment, which is easy to analyze, whence the approximation yields much information about the asymptotic properties of the original experiments. This information is extracted in several chapters to follow and concerns both asymptotic optimality theory and the behavior of statistical procedures such as the maximum likelihood estimator and the likelihood ratio test. 92

93

Expanding the Likelihood

7. 2

We have taken the local parameter set equal to IRk , which is not correct if the parameter set 8 is a true subset of IRk . If e0 is an inner point of the original parameter set, then the vector e = 80 + hj Jfi is a parameter in 8 for a given h, for every sufficiently large n, and the local parameter set converges to the whole of IRk as n --+ oo . Then taking the local parameter set equal to IRk does not cause errors. To give a meaning to the results of this chapter, the measure Peo +h / Fn may be defined arbitrarily if eo + h / Jfi ¢ 8 . 7.2

Expanding the Likelihood

The convergence of the local experiments is defined and established later in this chapter. First, we discuss the technical tool: a Taylor expansion of the logarithm of the likelihood. Let Pe be a density of Pe with respect to some measure f.L. Assume for simplicity that the parameter is one-dimensional and that the log likelihood £e (x) = log Pe (x) is twice­ differentiable with respect to e , for every x, with derivatives fe (x) and le (x) . Then, for every fixed x , 1 .. Pe+h log (x) = h£e· (x) + - h 2 £e (x) + o (h 2 ) . 2 Pe The subscript x in the remainder term is a reminder of the fact that this term depends on x as well as on h . It follows that --

x

.

..

Here the score has mean zero, Pe£e = 0, and - Pe£e = Pe £· 2e = Ie equals the Fisher information for e (see, e.g., section 5 .5). Hence the first term can be rewritten as h f),n,e , where f),n,e = n - 1 12 L 7=1 f e (Xi) is asymptotically normal with mean zero and variance Ie , by the central limit theorem. Furthermore, the second term in the expansion is asymptotically equivalent to - � h 2 Ie , by the law of large numbers . The remainder term should behave as o(ljn) times a sum of n terms and hopefully is asymptotically negligible. Consequently, under suitable conditions we have, for every h , n Pe +h!Fn n log (Xi) = h f),n ,e - -1 le h 2 + Op8 ( 1 ) . 2 i =1 Pe In the next section we see that this is similar in form to the likelihood ratio process of a Gaus­ sian experiment. Because this expansion concerns the likelihood process in a neighborhood of e , we speak of "local asymptotic normality" of the sequence of models { P; : e E 8}. The preceding derivation can be made rigorous under moment or continuity conditions on the second derivative of the log likelihood. Local asymptotic normality was originally deduced in this manner. Surprisingly, it can also be established under a single condition that only involves a first derivative: differentiability of the root density e r---+ ffi in quadratic mean. This entails the existence of a vector of measurable functions fe = ( f e, 1 , . . . , fe , k ) T such that

h --+ 0.

(7. 1 )

Local Asymptotic Normality

94

If this condition is satisfied, then the model (Pe : e E 8) is called diffe rentiable in quadratic mean at e . .J Pe+h (x ) at h = 0 for Usually, � h T f 8 (x ) .Jp8 (x ) is the derivative of the map h (almost) every x . In this case

f----*

1 a � a Pe (x ) = - log p e (x) . f. e (x ) = 2 -vt.::/71 ae Pe (x) ae Condition (7 .1) does not require differentiability of the map e f----* Pe (x) for any single x, but -

v

rather differentiability in (quadratic) mean. Admittedly, the latter is typically established by pointwise differentiability plus a convergence theorem for integrals. Because the condition is exactly right for its purpose, we establish in the following theorem local asymptotic normality under ( 7. 1 ). A lemma following the theorem gives easily verifiable conditions in terms of pointwise derivatives.

7.2 Theorem. Suppose that 8 is an open subset of rr:f..k and that the model (Pe : e E 8) is differentiable in quadratic mean at e. Then Pefe = 0 and the Fisher information matrix Ie = Pefef� exists. Furthermore, for every converging sequence h n h, as n oo,

---+

---+

---+

Given a converging sequence h n h, we use the abbreviations Pn , p, and g for Pe + h,J.Jii • p8 , and h T fe , respectively. By ( 7. 1 ) the sequence .y'ri(#n - ,jP) converges in quadratic mean (i.e., in L 2 (J.L)) to � g,JP. This implies that the sequence ffn converges in quadratic mean to ,JP. By the continuity of the inner product,

Proof.

The right side equals .y'ri(1 - 1) = 0 for every n, because both probability densities integrate to 1 . Thus Pg = 0. The random variable Wn i = 2[ .JPn! p (X i ) - 1] is with P -probability 1 well defined. By ( 7. 1 ) var

E

(8 n

1 n g ( Xi ) Wn i - .y'ri

8

)

�

E (.Jfi Wn i - g (X i ) ) 2

---+ 0,

t Wni = 2n (f y/p;;y/p dj.L - 1 ) = -n J [Jii;; - v'Pf dj.L ---+ - l Pg2 .

(7.3)

Here P g 2 = J g 2 dP = h T Ie h by the definitions of g and !8 . If both the means and the variances of a sequence of random variables converge to zero, then the sequence converges to zero in probability. Therefore, combining the preceding pair of displayed equations, we find (7.4)

7. 2

Expanding the Likelihood

95

Next, we express the log likelihood ratio in L 7=1 Wni through a Taylor expansion of the logarithm. If we write log(l + x) = x - � x 2 + x 2 R (2x) , then R (x ) --+ 0 as x --+ 0, and

As a consequence of the right side of (7.3), it is possible to write n Wn� = g 2 ( X i ) + Ani for random variables Ani such that E I Ani I --+ 0. The averages An converge in mean and hence in probability to zero. Combination with the law of large numbers yields

By the triangle inequality followed by Markov's inequality, nP ( I Wni I >

c:

J2)

::S

::S

n P (g 2 (Xi ) > n c: 2 ) + nP ( I Ani I > n c: 2 ) c;- 2 Pg 2 { i > n c: 2 } + 8 - 2 E I Ani I --+ 0.

The left side is an upper bound for P (maxl<:: i <:: n I Wni I > c: J2) . Thus the sequence maxl <:: i <:: n I Wni I converges to zero in probability. By the property of the function R , the sequence maxl <:: i <:: n I R (Wni) I converges in probability to zero as well. The last term on the right in (7.5) is bounded by maxl<:: i <:: n I R (Wni) I L 7=1 Wn� . Thus it is o p ( l) 0 p (1), and converges in probability to zero. Combine to obtain that n n l Pn n """ log - (Xi ) = L Wni - - Pg 2 + o p ( l ) . 4 i =l i=l p Together with (7 .4) this yields the theorem. • To establish the differentiability in quadratic mean of specific models requires a conver­ gence theorem for integrals. Usually one proceeds by showing differentiability of the map e f--+ Pe (x) for almost every X plus p,-equi-integrability (e.g., domination). The following lemma takes care of most examples.

7.6 Lemma. For every e in an open subset ofiT?.k let Pe be a J-L-probability density. Assume that the map e f--+ s e (x) = Jpe (x) is continuously differentiablefor every x. If the elements of the matrix Ie = j(fJ e !Pe) (p� fpe) Pe d p, are well defined and continuous in e, then the map e f--+ ffe is differentiable in quadratic mean (7. 1 ) with ie given by P e l Pe ·

By the chain rule, the map e f--+ pe (x) = s� (x) is differentiable for every x with gradient P e = 2sese . Because se is nonnegative, its gradient se at a point at which se = 0 must be zero. Conclude that we can write se = � (P e !Pe ) ffe, where the quotient P e !Pe may be defined arbitrarily if Pe = 0. By assumption, the map e f--+ Ie = 4 J ses[ dJ-L is continuous . Because the map e f--+ Se (x) is continuously differentiable, the difference S()+h (x) - Se (x) can be written as the integral J01 h T S()+uh (x) du of its derivative. By Jensen's (or Cauchy­ Schwarz's) inequality, the square of this integral is bounded by the integral J01 (h T S()+uh (x) ) 2

Proof.

Local Asymptotic Normality

96

du of the square. Conclude that

J(

)

SH t h - S g 2 d fL � �

J1 (h'{ S11+uth, ) 2 du dfL = � lal h'{ lll+uth, h t du , 1

where the last equality follows by Fubini's theorem and the definition of le . For h1 -+ h the right side converges to � h T Ie h = J (h T se ) 2 d JL by the continuity of the map 8 1---+ Ie . By the differentiability of the map 8 1---+ Sg (x) the integrand in

! [ SII+th� - Sg - h Tsg r dJL

converges pointwise to zero. The result of the preceding paragraph combined with Propo­ sition 2.29 shows that the integral converges to zero. •

7.7 Example (Exponential families). The preceding lemma applies to most exponential family models

pe (x) = d(8)h(x) e Q Ce l t (x ) . An exponential family model is smooth in its natural parameter (away from the boundary of the natural parameter space). Thus the maps 8 �----+ Jpe (x) are continuously differentiable if the maps 8 �----+ Q (8) are continuously differentiable and map the parameter set 8 into the interior of the natural parameter space. The score function and information matrix equal

fe (x) = Q� (t (x) - Ee t (X)) ,

Ie = Q�cove t (X) (Q�) T .

Thus the asymptotic expansion of the local log likelihood is valid for most exponential families. 0

7.8 Example (Location models). The preceding lemma also includes all location models { f (x - 8) : 8 E JR.} for a positive, continuously differentiable density f with finite Fisher information for location

If =

J ( j') \x) f (x) dx .

The score function fe (x) can be taken equal to - (f '/f) (x - 8 ) . The Fisher information is equal to I1 for every 8 and hence certainly continuous in 8 . B y a refinement of the lemma, differentiability in quadratic mean can also be established for slightly irregular shapes, such as the Laplace density f (x) = �e - l x l . For the Laplace density the map 8 1---+ log f (X - 8) fails to be differentiable at the single point 8 = X . At other points the derivative exists and equals sign(x - 8 ) . It can be shown that the Laplace location model is differentiable in quadratic mean with score function fe (x) = sign(x - 8). This may be proved by writing the difference J f (x - h) - J7(Xj as the integral J01 � h sign(x - uh) J f (x - uh) du of its derivative, which is possible even though the derivative does not exist everywhere. Next the proof of the preceding lemma applies . 0

7.9 Counterexample (Uniform distribution). The family of uniform distributions on [0, 8] is nowhere differentiable in quadratic mean. The reason is that the support of the

7.3

97

Convergence to a Normal Experiment

uniform distribution depends too much on the parameter. Differentiability in quadratic mean (7. 1) does not require that all densities pe have the same support. However, restric­ tion of the integral (7. 1) to the set {pe = 0} yields

Pe +h (Pe = 0) =

1

PB+ h d f-L = o (h 2 ) .

po =O Thus, under (7. 1) the total mass Pe+h (Pe = 0) of Pe + h that is orthogonal to Pe must "disappear" as h -+ 0 at a rate faster than h 2 . This is not true for the uniform distribution, because, for h ::: 0, Pe+h (Pe = 0) =

1

1

--

[O , e ]c () + h

1 [ 0 , 8 + hJ (x ) d x =

h

-- .

() + h

The orthogonal part does converge to zero, but only at the rate O (h).

7.3

D

Convergence to a Normal Experiment

The true meaning of local asymptotic normality is convergence of the local statistical experiments to a normal experiment. In Chapter 9 the notion of convergence of statistical experiments is introduced in general. In this section we bypass this general theory and establish a direct relationship between the local experiments and a normal limit experiment. The limit experiment is the experiment that consists of observing a single observation X with the N (h , le- l )-distribution. The log likelihood ratio process of this experiment equals log

r dN(h , le- 1 ) 1 r X X -h leh . h le ( ) = 2 dN(O, le- 1 )

The right side is very similar in form to the right side of the expansion of the log likelihood ratio log d P;+h / .,fii I d Pen given in Theorem 7 .2. In view of the similarity, the possibility of a normal approximation is not a complete surprise. The approximation in this section is "local" in nature: We fix () and think of

as a statistical model with parameter h, for "known" () . We show that this can be approxi­ mated by the statistical model (N(h , le- 1 ) : h E Il�n. A motivation for studying a local approximation is that, usually, asymptotically, the "true" parameter can be known with unlimited precision. The true statistical difficulty is therefore determined by the nature of the measures Pe for () in a small neighbourhood of the true value. In the present situation "small" turns out to be "of size 0 ( 1 I ,jn) ." A relationship between the models that can be statistically interpreted will be described through the possible (limit) distributions of statistics. For each n , let Tn = Tn ( X 1 , , X11 ) be a statistic in the experiment (Pen+ h / .,fii : h E �k ) with values in a fixed Euclidean space. Suppose that the sequence of statistics T11 converges in distribution under every possible (local) parameter: •

every h .

.

.

98

Local Asymptotic Normality

Here !;.., means convergence in distribution under the parameter e + hj.jfi, and Le ,h may be any probability distribution. According to the following theorem, the distributions {Le , h : h E IRk } are necessarily the distributions of a statistic T in the normal experiment ( N (h , Ie- 1 ) : h E IRk ) . Thus, every weakly converging sequence of statistics is "matched" by a statistic in the limit experiment. (In the present set-up the vector e is considered known and the vector h is the statistical parameter. Consequently, by "statistics" Tn and T are understood measurable maps that do not depend on h but may depend on e .) This principle of matching estimators is a method to give the convergence of models a statistical interpretation. Most measures of quality of a statistic can be expressed in the distribution of the statistic under different parameters. For instance, if a certain hypothesis is rejected for values of a statistic Tn exceeding a number c , then the power function h c---+ Ph (Tn > c) is relevant; alternatively, if Tn is an estimator of h, then the mean square error h c---+ Eh ( Tn - h) 2 , or a similar quantity, determines the quality of Tn . Both quality measures depend on the laws of the statistics only. The following theorem asserts that as a function of h the law of a statistic Tn can be well approximated by the law of some statistic T . Then the quality of the approximating T is the same as the "asymptotic quality" of the sequence Tn . Investigation of the possible T should reveal the asymptotic performance of possible sequences Tn . Concrete applications of this principle to testing and estimation are given in later chapters. A minor technical complication is that it is necessary to allow randomized statistics in the limit experiment. A randomized statistic T based on the observation X is defined as a measurable map T = T (X, U) that depends on X but may also depend on an independent variable U with a uniform distribution on [0, 1 ] . Thus, the statistician working in the limit experiment is allowed to base an estimate or test on both the observation and the outcome of an extra experiment that can be run without knowledge of the parameter. In most situations such randomization is not useful, but the following theorem would not be true without it. t

7.10 Theorem. Assume that the experiment (Pe : e E 8) is differentiable in quadratic mean (7.1) at the point e with nonsingular Fisher information matrix Ie. Let Tn be statistics in the experiments (Pen+h ! .fii : h E IRk ) such that the sequence Tn converges in distribution under every h. Then there exists a randomized statistic T in the experiment ( N (h , Ie- 1 ) : h E IRk ) such that Tn !;.., T for every h. Proof.

For later reference, it is useful to use the abbreviations J = Ie ,

By assumption, the marginals of the sequence (Tn , ,6. n ) converge in distribution under h = 0; hence they are uniformly tight by Prohorov's theorem. Because marginal tightness implies joint tightness, Prohorov's theorem can be applied in the other direction to see the existence of a subsequence of {n} along which

t

It is not important that U is uniformly distributed. Any randomization mechanism that is sufficiently rich will do.

7.3

99

Convergence to a Normal Experiment

jointly, for some random vector (S, �). The vector � is necessarily a marginal weak limit of the sequence �n and hence it is N (O , f)-distributed. Combination with Theorem 7.2 yields

( Tn , log dPn,h ) o ( S, h T � - -h21 T Jh ) . --

�

dPn,O In particular, the sequence log dPn,h / dPn, o converges to the normal N(- � h T Jh, h T ]h)­ distribution. By Example 6.5, the sequences Pn,h and Pn, o are contiguous. The limit law L h of Tn under h can therefore be expressed in the joint law on the right, by the general form of Le Cam ' s third lemma: For each Borel set B

We need to find a statistic T in the normal experiment having this law under h (for every h), using only the knowledge that � is N (0, f)-distributed. By the lemma below there exists a randomized statistic T such that, with U uniformly distributed and independent of �, t

( T (� , U) , � ) "" (S, �) . Because the random vectors on the left and right sides have the same second marginal distribution, this is the same as saying that T ( 8 , U) is distributed according to the conditional distribution of S given � = 8, for almost every 8. As shown in the next lemma, this can be achieved by using the quantile transformation. Let X be an observation in the limit experiment ( N ( h , J - 1 ) : h E Il�_k) . Then J X is under h = 0 normally N (0, f)-distributed and hence it is equal in distribution to � . Furthermore, by Fu bini's theorem,

Ph ( T ( J X , U) E B) =

f P ( T ( Jx, U)

E

B) e - � (x - h )r l(x - h )

= Eo 1 B ( T ( J X , U) ) e h r J X-� h r J h .

det J (2n ) k

-- dx

This equals L h ( B ) , because, by construction, the vector ( T (J X, U) , IX) has the same distribution under h = 0 as (S, �). The randomized statistic T (J X, U) has law L h under h and hence satisfies the requirements. •

7.11 Lemma. Given a random vector (S, �) with values in �d x �k and an independent uniformly [0, 1] random variable U (defined on the same probability space), there exists a jointly measurable map T on �k x [0, 1] such that ( T (� , U) , � ) and (S, �) are equal in distribution. For simplicity of notation we only give a construction for d = 2. It is possible to produce two independent uniform [0, 1 ] variables ul and u2 from one given [0, 1] variable U. (For instance, construct U1 and U2 from the even and odd numbered digits in the decimal expansion of U. ) Therefore it suffices to find a statistic T = T(� , U1 , U2 ) such that (T, �) and (S, �) are equal in law. Because the second marginals are equal, it

Proof.

t

The symbol � means "equal-in-law."

1 00

Local Asymptotic Normality

suffices to construct T such that T(8, U 1 , U2 ) is equal in distribution to S given t. = 6 , for every 8 E IR.k . Let Q 1 (u 1 I 8) and Q 2 (u 2 I 8 , s t ) be the quantile functions of the conditional distributions and respectively. These are measurable functions in their two and three arguments , respectively. Furthermore, Q 1 (U1 18) has law P 51 1 t.. =o and Q 2 ( U2 1 8 , s 1 ) has law p Sz l t.. =o , S1 =s1 , for every 8 and s 1 • Set Then the first coordinate Q 1 (U1 18) of T (8 , U1 , U2 ) possesses the distribution p S1 1 6= 8 . Given that this first coordinate equals s 1 , the second coordinate is distributed as Q 2 (U2 1 8 , s 1 ) , which has law p S2 1 t.. =o , S1 =s1 by construction. Thus T satisfies the requirements. • 7.4

Maximum Likelihood

Maximum likelihood estimators in smooth parametric models were shown to be asymp­ totically normal in Chapter 5. The convergence of the local experiments to a normal limit experiment gives an insightful explanation of this fact. By the representation theorem, Theorem 7. 10, every sequence of statistics in the local ex­ periments (P;+ h /fo : h E IR.k ) is matched in the limit by a statistic in the normal experiment. Although this does not follow from this theorem, a sequence of maximum likelihood esti­ mators is typically matched by the maximum likelihood estimator in the limit experiment. Now the maximum likelihood estimator for h in the experiment ( N (h , !8- 1 ) : h E IR.k ) is the observation X itself (the mean of a sample of size one), and this is normally distributed. Thus, we should expect that the maximum likelihood estimators hn for the local param­ eter h in the experiments (P;+h / fn : h E IR.k ) converge in distributiAon to X. In terms of the original parameter 8, the local maximum likelihood estimator hn is the standardized maximum likelihood estimator hn = Jn(Bn 8) . Furthermore, the local parameter h = 0 corresponds to the value 8 of the original parameter. Thus, we should expect that under 8 the sequence Jn(Bn 8) converges in distribution to X under h = 0, that is, to the N( O , /8- 1 )-distribution. As a heuristic explanation of the asymptotic normality of maximum likelihood estimators the preceding argument is much more insightful than the proof based on linearization of the score equation. It also explains why, or in what sense, the maximum likelihood estimator is asymptotically optimal: in the same sense as the maximum likelihood estimator of a Gaussian location parameter is optimal. This heuristic argument cannot be justified underjust local asymptotic normality, which is too weak a connection between the sequence oflocal experiments and the normal limit exper­ iment for this purpose. Clearly, the argument is valid under the conditions of Theorem 5.39, because the latter theorem guarantees the asymptotic normality of the maximum likelihood estimator. This theorem adds a Lipschitz condition on the maps 8 f---'3> log Pe (x) , and the "global" condition that Bn is consistent to differentiability in quadratic mean. In the fol­ lowing theorem, we give a direct argument, and also allow that 8 is not an inner point of the parameter set, so that the local parameter spaces may not converge to the full space m.k . -

-

7. 4

101

Maximum Likelihood

X

Then the maximum likelihood estimator in the limit experiment is a "projection" of and the limit distribution of Jfi(Bn 19) may change accordingly. Let 8 be an arbitrary subset of IRk and define Hn as the local parameter space Hn Jn(8 19). Then hn is the maximizer over Hn of the random function (or "process") d Pn h f-+ lo g d Pn If the experiment 19 E 8) is differentiable in quadratic mean, then this sequence of processes converges (marginally) in distribution to the process

-

-

Hh / ..fii e

(Pe :

h

f-+

log

dN ( h, dN ( 0,

li1)1 (X) = - -(X 1 1 h) h) + -X leX. Ie(X 2 2 Ie- ) T

T

If the sequence of sets Hn converges in a suitable sense to a set H, then we should expect, under regularity conditions, that the sequence hn converges to the maximizer h of the latter process over H . This maximizer is the projection of the vector onto the set H relative to the metric d (x , y) (x - yl y) (where a "projection" means a closest point); if k H IR , this projection reduces to itself. An appropriate notion of convergence of sets is the following. Write Hn ---+ H if H is the set of all limits lim hn of converging sequences hn with h n E Hn for every n and, moreover, the limit h limi hn, of every converging sequence hn, with h n, E Hn, for every i is contained in H. t =

=

X

Ie(xX -

=

Suppose that the experiment : 19 E 8) is differentiable in quadratic Furthermore, suppose that for mean at 19o with nonsingular Fisher information matrix every and 192 in a neighborhood of 190 and a measurable function .€ with < oo, 7.12

(Pe

Theorem.

le0 •

81

Pe0 l2

If the sequence of maximum likelihood estimators Bn is consistent and the sets Hn Jn( t9n 19o) Jfi(8 19o) converge to a nonempty, convex set H, then the sequence converges under 190 in distribution to the projection of a standard normal vector onto the H. set

le10/2

le10/2

-

- Pe0 ) L 2 ( Pe0 ) Pe e -le00 • P Pe, 1 e sup nJPln log P o +h!..fii - h Gn f e0 + -h le0 h 2 I I 0. Pea

A

-

=

Let Gn Jfl(JPln be the empirical process. In the proof of Theorem 5.39 it is shown that the map 19 f-+ log pe is differentiable at 19o in with derivative and that the map t9 f-+ log permits a Taylor expansion of order 2 at 19o, with "second-derivative matrix" Therefore, the conditions of Lemma 19.3 1 are satisfied for m log whence, for every M, *Proof.

le0

e

=

=

l l h ii �M

T

·

T

---+

P

By Corollary 5.53 the estimators Bn are Jn-consistent under 190. The preceding display is also valid for every sequence Mn that diverges to sufficiently slowly. Fix such a sequence. By the Jn-consistency of Bn , the local maximum likelihood oo

t

See Chapter 16 for examples.

Local Asymptotic Normality

102

estimators h11 are bounded in probability and hence belong to the balls of radius Mn with probability tending to 1 . Furthermore, the sequence of intersections 11 n ball (0 , M11) converges to as the original sets Thus, we may assume that the h11 are the maximum likelihood estimators relative to local parameter sets that are contained in the balls of radius M11 • Fix an arbitrary closed set If h11 E then the log likelihood is maximal on Hence (h E is bounded above by

H,

H

Hn. F.

Hn

F, P n F) P ( sup n og PeaP+eha/ ::::_ sup n og PeaP+eha! v'fi) P ( sup hT Gn-Cea - � h T lea h ::::_ sup hT Gnfea - � h T lea h + o p ( l ) ) P ( l le� 1 12 Gn iea - 1�:\ F n Hn) l :::: l le� 1 12 Gn iea - 1�:2 Hn II + O p ( l) ) , by completing the square. By Lemma 7 . 1 3 (ii) and (iii) ahead, we can replace Hn by H on both sides, at the cost of adding a further op ( l)-term and increasing the probability. Next, by the continuous mapping theorem and the continuity of the map I - A I for every F.

h E Fni-1, =

1Tll 11

I

v'ii

h EI-1,

1Tll 11

1

___:____c_ _:_ _

h EFni-fn

h EI-1,

=

z �----+

z

set A, the probability is asymptotically bounded above by, with Z a standard normal vector,

1�:2 H

The projection ITZ of the vector Z on the set is unique, because the latter set is convex by assumption and automatically closed. If the distance of Z to n is smaller than its distance to the set then IT Z must be in n Consequently, the probability in the last display is bounded by P(IT Z E The theorem follows from the portmanteau lemma. •

1�:2 H,

2 (F H) 1�: 2 (F H). 1�: 1�: 2 F).

Hn

H

If the sequence of subsets of TJf.k converges to a non empty set and the sequence of random vectors converges in distribution to a random vector then (i) n + O p ( l), for every closed set n (ii) ::::_ n G il + op (l), for every open set G. (iii) n G il

7.13

Lemma.

Xn X, I Xn - Hn l -v'+ I X - H l F. I Xn - Hn F l I XnXn - HH F l I Xn - Hn :::: I (i). Because the map x x - H l is (Lipschitz) continuous for any set H, l we have that I X11 - H I -v'+l l X - H I by the continuous-mapping theorem. If we also show that I X11 - Hn I - I X11 - H I � 0, then the proof is complete after an application of Slutsky's lemma. By the uniform tightness of the sequence X11 , it suffices to show that for x ranging over compact sets, or equivalently that l xxn- HnHnI ---+ l xx- HHI uniformly every converging sequence x11 ---+ x. - l X11for, there l For- everyl ---+fixedl vector exists a vector h n Hn with l xn-Hn I l xn - h n 11 - l f n . Unless l xn - Hn I is unbounded, we can choose the sequence hn bounded. Then every subsequence of hn has a further subsequence along which it converges, to a limit h in H. �----+

Proof

E

::::__

Conclude that, in any case,

Conversely, for every 8 > 0 there exists h

E

H and a sequence hn ---+ h with h11

E

l x - H l ::::_ l x - h l - 8 = lim l xn - hn l - 8 ::::_ lim sup l xn - Hn l - 8.

Hn and

Local Asymptotic Normality

7. 6

-

103

Combination of the last two displays yields the desired convergence of the sequence llxn Hn I to llx - H l l (ii). The assertion is equivalent to the statement P II Xn - Hn n F ll - II Xn - H n F ll > - 8 --+ 1 for every 8 > 0. In view of the uniform tightness of the sequence Xn , this follows if lim inf ll xn - Hn n F II ::: llx - H n F II for every converging sequence Xn -+ x . We can prove this by the method of the first half of the proof of (i), replacing Hn by Hn n F. (iii) . Analogously to the situation under (ii), it suffices to prove that lim sup llxn - Hn n G I ::::: llx - H n G I for every converging sequence Xn -+ x . This follows as the second half of the proof of (i). •

)

(

*7.5

Limit Distributions under Alternatives

Local asymptotic normality is a convenient tool in the study of the behavior of statistics under "contiguous alternatives." Under local asymptotic normality, n d Pe+h ! v'n !, N - -1 h T 1 h h T 1 h . lo g 2 d Pn Therefore, in view of Example 6.5 the sequences of distributions P8n+h ! v'n and Pen are mutually contiguous. This is of great use in many proofs. With the help of Le Cam ' s third lemma it also allows to obtain limit distributions of statistics under the parameters e + hI -Jii, once the limit behavior under e is known. Such limit distributions are of interest, for instance, in studying the asymptotic efficiency of estimators or tests. The general scheme is as follows. Many sequences of statistics Tn allow an approxima­ tion by an average of the type

e

(

e, e)

According to Theorem 7 .2, the sequence of log likelihood ratios can be approximated by an average as well: It is asymptotically equivalent to an affine transformation of is asymptotically multivariate The sequence of joint averages normal under e by the central limit theorem (provided has mean zero and finite second moment). With the help of Slutsky's lemma we obtain the joint limit distribution of Tn and the log likelihood ratios under e :

n - 1 12 .2:: ( 1/re (Xi ), le(Xi)) 1/re

n - 1 12 _L le (Xi ).

Finally we can apply Le Cam ' s third Example 6.7 to obtain the limit distribution of under e + hj -Jfi. Concrete examples of this scheme are discussed in later Jn(Tn chapters.

f-Le )

*7.6

Local Asymptotic Normality

The preceding sections of this chapter are restricted to the case of independent, identically distributed observations. However, the general ideas have a much wider applicability. A

Local Asymptotic Normality

1 04

wide variety of models satisfy a general form of local asymptotic normality and for that reason allow a unified treatment. These include models with independent, not identically distributed observations, but also models with dependent observations, such as used in time series analysis or certain random fields. Because local asymptotic normality underlies a large part of asymptotic optimality theory and also explains the asymptotic normality of certain estimators, such as maximum likelihood estimators, it is worthwhile to formulate a general concept. Suppose the observation at "time" n is distributed according to a probability measure Pn , e , for a parameter () ranging over an open subset 8 of rn;.k . Definition. The sequence of statistical models ( Pn , e : () E 8) is locally asymptoti­ cally normal (LAN) at () if there exist matrices rn and Ie and random vectors t-.. n , e such that

7.14

t-.. n , e � N (O, Ie ) and for every converging sequence hn

log

-+

h

d Pn , e + rn- l h " 1 = h T t-.. 11 e - - h T fe h + O p" d Pn , e 2 '

0

(1).

If the experiment ( Pe : () E e) is differentiable in quadratic mean, then the sequence of models (P; : () E 8) is locally asymptotically normal with norming matrices rn = Jli!. D

7.15

Example.

An inspection of the proof of Theorem 7. 10 readily reveals that this depends on the local asymptotic normality property only. Thus, the local experiments of a locally asymptotically normal sequence converge to the experiment (N (h , /0- 1 ) : h E Il�_k), in the sense of this theorem. All results for the case of i.i. d. observations that are based on this approximation extend to general locally asymptotically normal models. To illustrate the wide range of applications we include, without proof, three examples, two of which involve dependent observations. An autoregressive process {Xr : t E Z} of or­ der 1 satisfies the relationship Xr = () Xr - 1 + Zr for a sequence of independent, identically distributed variables . . . , Z_ 1 , Z0 , Z1 , . . . with mean zero and finite variance. There ex­ ists a stationary solution . . . , X _ 1 , X0 , X 1 , to the autoregressive equation if and only if I() I =f. 1 . To identify the parameter it is usually assumed that I() I 1 . If the density of the noise variables Z1 has finite Fisher information for location, then the sequence of models corresponding to observing X 1 , , Xn with parameter set ( - 1 , 1) is locally asymptotically normal at () with norming matrices r11 = Jli!. The observations in this model form a stationary Markov chain. The result extends to general ergodic Markov chains with smooth transition densities (see [ 1 30]). D 7.16

Example (Autoregressive processes).

•

.

.

<

•

•

•

This example requires some knowledge of time­ series models. Suppose that at time n the observations are a stretch X 1 , . . . , X from a stationary, Gaussian time series {Xr : t E Z} with mean zero. The covariance matrix of n

7.17

Example (Gaussian time series).

n

7. 6

Local Asymptotic Normality

1 05

consecutive variables is given by the (Toeplitz) matrix

Tn(fe) (/_: e i (t -s )Jc fe (A) dA ) s t l =

fe

_

,-

....

,n

.

The function is the spectral density of the series. It is convenient to let the parameter enter the model through the spectral density, rather than directly through the density of the observations. Let n , be the distribution (on �n ) of the vector ( X 1 , . . . , Xn) , a normal distribution with mean zero and covariance matrix Tn The periodogram of the observations is the function

Pe

(fe).

Suppose that fe is bounded away from zero and infinity, and that there exists a vector-valued function : � �d such that, as ----?- 0, . T fe] 2 dA = o ( 2 )

h l hl . f Ue +h - fe - h fe Then the sequence of experiments (Pn , e : e 8) is locally asymptotically normal at () with Ie 4n1 f £8 £8 dA . le r-+

E

= -

·

·

T

The proof is elementary, but involved, because it has to deal with the quadratic forms in the n-variate normal density, which involve vectors whose dimension converges to infinity (see [30]). D Consider estimating a location parameter () based on a sample of size n from the density ( - 8). If is smooth, then this model is differentiable in quadratic mean and hence locally asymptotically normal by Example 7.8. If possesses points of discontinuity, or other strong irregularities, then a locally asymptot­ ically normal approximation is impossible. t Examples of densities that are on the boundary between these "extremes" are the triangular density = (1 and the gamma density = e - 1 > 0}. These yield models that are locally asymptotically normal, but with norming rate Jn log n rather than ,Jn. The existence of singularities in the density makes the estimation of the parameter e easier, and hence a faster rescaling rate is necessary. (For the triangular density, the true singularities are the points - 1 and 1 , the singularity at 0 is statistically unimportant, as in the case of the Laplace density.) For a more general result, consider densities that are absolutely continuous except pos­ sibly in small neighborhoods U1 , . . . , Uk of finitely many fixed points c 1 , . . . , ck . Suppose that f 'I ..jJ is square-integrable on the complement of U 1 u1 , that (c 1) = 0 for every j , and that, for fixed constants a 1 , . . . , ak and b1 , . . . , bk . each of the functions 7.18

Example (Almost regular densities).

fx

f

f

f(x)

f(x) x x {x

- lx lt

f

f

t

See Chapter 9 for some examples.

1 06

Local Asymptotic Normality

is twice continuously differentiable. If I)ai + bi ) > 0, then the model is locally asymp­ totically normal at () = 0 with, for equal to the interval (n - 1 1 (Iog n ) - 1 14 , (log n)- 1 ) around zero, t

2

Vn

n

r

=

Jn log n ,

j

1 {Xi - c Vn} 1 1 ( -----'-- f (x + c d x -- 6.n , O Jn log1 n �i = l � Xi ) J= l The sequence 6.n , o may be thought of as "asymptotically sufficient" for the local parameter h . Its definition of 6.n , o shows that, asymptotically, all the "information" about the parameter is contained in the observations falling into the neighborhoods Vn + c Thus, asymptotically, :;== = --;:.::::=

i

LL

E

Cj

v, ,

.

i)

X

1.

the problem is determined by the points of irregularity. The remarkable rescaling rate Jn log n can be explained by computing the Hellinger distance between the densities f (x - ()) and f (x) (see section 14.5). D

Notes

Local asymptotic normality was introduced by Le Cam [92], apparently motivated by the study and construction of asymptotically similar tests. In this paper Le Cam defines two sequences of models () E 8) and () E 8) to be diffe rentially equivalent if

(Pn , e : (Qn , e : sup I Pn , H h jfn - Qn , Hh jy'n l --+ 0, hEK

Tn

for every bounded set K and every (). He next shows that a sequence of statistics in a given asymptotically differentiable sequence of experiments (roughly LAN) that is asymptotically equivalent to the centering sequence is asymptotically sufficient, in the sense that the original experiments and the experiments consisting of observing the are differentially equivalent. After some interpretation this gives roughly the same message as Theorem 7 . 1 0 . The latter is a concrete example of an abstract result in [95], with a different (direct) proof.

6.n , e

Tn

PROBLEMS 1. Show that the Poisson distribution with mean information.

e satisfies the conditions of Lemma 7.6.

Find the

2. Find the Fisher information for location for the normal, logistic, and Laplace distributions. 3. Find the Fisher information for location for the Cauchy distributions.

4. Let f be a density that is symmetric about zero. Show that the Fisher information matrix (if it exists) of the location scale family f ( (x - � ) I 0' ) I 0' is diagonal. 5. Find an explicit expression for the ope ( 1 ) -term in Theorem 7.2 in the case that of the N 1 ) -distribution.

(e,

pe

is the density

6. Show that the Laplace location family is differentiable in quadratic mean . t

See, for example, [80, pp. 133-1 39] for a proof, and also a discussion of other almost regular situations. For instance, singularities of the form f(x) f(cj ) + lx - Cj 1 1 1 2 at points cj with f(cj ) > 0. �

Problems

1 07

7. Find the form of the score function for a location-scale family f ( (x - fL) I a ) I a with parameter e = (f.L , a ) and apply Lemma 7.6 to find a sufficient condition for differentiability in quadratic mean.

8. Investigate for which parameters k the location family f (x - e) for f the gamma(k , 1) density is differentiable in quadratic mean.

9. Let

Pn,e

be the distribution of the vector

(X 1 , . . . , Xn) if {X1

:

t

E

Z} is a stationary Gaussian

time series satisfying X1 = e X r - 1 + Z1 for a given number 1 e 1 < 1 and independent standard normal variables Z1 . Show that the model is locally asymptotically normal.

10. Investigate whether the log normal family of distributions with density

is differentiable in quadratic mean with respect to e = (� , f.L , a ) .

8 Efficiency of Estimators

One purpose of asymptotic statistics is to compare the performance of estimatorsfor large sample sizes. This chapter discusses asymptotic lower bounds for estimation in locally asymptotically normal models. These show, among others, in what sense maximum likelihood estimators are asymptotically efficient.

8.1

Asymptotic Concentration

1/1 (8)

Suppose the problem is to estimate based on observations from a model governed by the parameter What is the best asymptotic performance of an estimator sequence Tn for

8. (8)? 1/1 To simplify the situation, w e shall in most of this chapter assume that the sequence ,.jii ( Tn -1/1 (8)) converges in distribution under every possible value of 8. Next we rephrase

the question as: What are the best possible limit distributions? In analogy with the Cramer­ Rao theorem a "best" limit distribution is referred to as an asymptotic lower bound. Under certain restrictions the normal distribution with mean zero and covariance the inverse Fisher information is an asymptotic lower bound for estimating in a smooth parametric model. This is the main result of this chapter, but it needs to be qualified. The notion of a "best" limit distribution is understood in terms of concentration. If the limit distribution is a priori assumed to be normal, then this is usually translated into asymp­ totic unbiasedness and minimum variance. The statement that ,.Jii ( Tn converges in distribution to a ) -distribution can be roughly understood in the sense that eventually Tn is approximately normally distributed with mean and variance given by

1/1 (8) 8 =

- 1/1 (8))

N (f.l(8), 0" 2 (8)

and

(8), 1/1 0" 2 (8)

n

Because Tn is meant to estimate optimal choices for the asymptotic mean and variance are 0 and variance as small as possible. These choices ensure not only that the asymptotic mean square error is small but also that the limit distribution is maximally concentrated near zero. For instance, the probability of the interval (-a, a) is maximized by choosing 0 and minimal. We do not wish to assume a priori that the estimators are asymptotically normal. That normal limits are best will actually be an interesting conclusion. The concentration of a general limit distribution Le cannot be measured by mean and variance alone. Instead, we

J.1,(8)

=

J.1,(8)

=

0" 2 (8)

N (f.l(8),
108

8. 1

Asymptotic Concentration

109

can employ a variety of concentration measures, such as

J x 2 dLe(x); J l x l dL e (x) ; J l { lx l > a } dLe(x); J ( l x l /\ a ) dLe(x).

A limit distribution is "good" if quantities of this type are small. More generally, we for a given nonnegative function £. Such a function is called focus on minimizing J £ a loss function and its integral J £ is the asymptotic risk of the estimator. The method of measuring concentration (or rather lack of concentration) by means of loss functions applies to one- and higher-dimensional parameters alike. The following example shows that a definition of what constitutes asymptotic optimality is not as straightforward as it might seem.

dL e

dLe

Tn

8.1 Example (Hodges ' estimator). Suppose that is a sequence of estimators for a real parameter (] with standard asymptotic behavior in that, for each (] and certain limit distri­ butions

L e,

Tn Sn

As a specific example, let be the mean of a sample of size n from the N ((] , 1 )-distribution. Define a second estimator through if if

Tn

ITnl :::: n -- 11 //44 · I Tn l < n

If the estimator is already close to zero, then it is changed to exactly zero; otherwise it is left unchanged. The truncation point n - 1 /4 has been chosen in such a way that the limit behavior of is the same as that of for every (] -=j 0, but for (] = 0 there appears to be a great improvement. Indeed, for every r

Sn

Tn n, rnSn e0 0 n ;) n Le , To see this, note first that the probability that Tn falls in the interval ((] - M n- 1 12 , (]+ Mn - 1 12 ) converges to L e (- M, M) for most M and hence is arbitrarily close to 1 for M and n sufficiently large. For (] 0, the intervals ((] - M n- 1 12 (] + M n - 1 12 and ( -n - 1 14 , n - 1 14 r- ,

v

�

-v-+

�

r:J ) �,

""'

T

-=j

,

)

)

are centered at different places and eventually disjoint. This implies that truncation will rarely occur: = � 1 if (] -=F 0, whence the second assertion. On the other hand the 1 1 12 12 interval (- Mn - , M n - ) is contained in the interval ( -n - 1 14 , n - 1 14 ) eventually. Hence under (] = 0 we have truncation with probability tending to 1 and hence = 0) � 1 ; this is stronger than the first assertion. At first sight, is an improvement on For every (] -=j 0 the estimators behave the same, while for (] = 0 the sequence has an "arbitrarily fast" rate of convergence. However, this reasoning is a bad use of asymptotics. Consider the concrete situation that is the mean of a sample of size n from the normal N ((], !)-distribution. It is well known that = X is optimal in many ways for every fixed n and hence it ought to be asymptotically optimal also. Figure 8. 1 shows why = X 1 { I X I :::: n - 1 /4 } is no improvement. It shows the graph of the risk function (] f---+ - (]) 2 for three different values of n . These functions are close to 1 on most

Pe (Tn Sn) Sn

Sn Ee (Sn

Sn Tn

P0(Sn

Tn.

Tn

1 10

Efficiency of Estimators

l()

-

l()

-

0

-

I

-2

I

I

-1

I

0

I

2

Figure 8.1. Quadratic risk functions of the Hodges estimator based on the means of samples of size 10 (dashed) , 1 00 (dotted) , and 1000 (solid) observations from the I ) -distribution.

N(e,

of the domain but possess peaks close to zero. As n � the locations and widths of the peaks converge to zero but their heights to infinity. The conclusion is that "buys" its better asymptotic behavior at f3 = 0 at the expense of erratic behavior close to zero. Because the values of f3 at which is bad differ from n to n , the erratic behavior is not visible in the pointwise limit distributions under fixed f3. D oo ,

Sn

Sn

8.2

Relative Efficiency

In order to choose between two estimator sequences, we compare the concentration of their limit distributions. In the case of normal limit distributions and convergence rate Jfi, the quotient of the asymptotic variances is a good numerical measure of their relative efficiency. This number has an attractive interpretation in terms of the numbers of observations needed to attain the same goal with each of two sequences of estimators. Let v � be a "time" index, and suppose that it is required that, as v � our estimator sequence attains mean zero and variance 1 (or 1 / v ) Assume that an estimator based on n observations has the property that, as n � oo

oo,

.

oo ,

Tn

Then the requirement is to use at time v an appropriate number nv of observations such that, as v � oo ,

.fo ( Tn"

-

1/J(f3)) ! N( O , 1 ) .

Given two available estimator sequences, let nv , l and nv , 2 b e the numbers of observations

8. 3

111

Lower Bound for Experiments

needed to meet the requirement with each of the estimators. Then, if it exists, the limit

nv 1. lm ­, 2 v -HXJ nv, l is called the relative efficiency of the estimators. (In general, it depends on the para­ meter Because - 1/.r (B )) can be written as rvrn:, - 1j.r (e)), it follows that necessarily nv Thus, the relative efficiency of two and also that nvfv -+ estimator sequences with asymptotic variances is just

e.)

.JV(Tnv -+

2a (e). Fv(Tn" a((e) nv , dv a:}Je) lim . nv , ! / V af(e)

oo,

=

!1 ---HXl

If the value of this quotient is bigger than 1, then the second estimator sequence needs proportionally that many observations more than the first to achieve the same (asymptotic) preCISIOn.

8.3

Lower Bound for Experiments

,.Jii( Tn -

It is certainly impossible to give a nontrivial lower bound on the limit distribution of a standardized estimator 1j.r(e)) for a single Hodges ' example shows that it is not even enough to consider the behavior under every pointwise for all Different values of the parameters must be taken into account simultaneously when taking the limit as n -+ We shall do this by studying the performance of estimators under parameters in a "shrinking" neighborhood of a fixed We consider parameters h j ,Jn for fixed and h ranging over ffi.k and suppose that, for certain limit distributions L e , h , oo.

e+

Tn

e.

e,

e.

e. e

v'n(T" - o/ (8 + �))

' +h_L.fo L0 , ,

every h.

(8.2)

(e)

Then can be considered a good estimator for 1/.r if the limit distributions Le, h are maximally concentrated near zero. If they are maximally concentrated for every h and some fixed then can be considered locally optimal at Unless specified otherwise, we assume in the remainder of this chapter that the parameter set 8 is an open subset of ffi.k , and that 1/.r maps 8 into ffi.m . The derivative of r--+ 1j.r (e) is denoted by ;pe . Suppose that the observations are a sample of size n from a distribution Pe . If Pe depends smoothly on the parameter, then

Tn

e,

e.

e

as experiments, in the sense of Theorem 7 . 1 0. This theorem shows which limit distributions are possible and can be specialized to the estimation problem in the following way. 8.3 Theorem. Assume that the experiment (Pe : E 8) is differentiable in quadratic mean (7. 1) at the point e with nonsingular Fisher information matrix Ie. Let 1/.r be dzf­ ferentiable at Let be estimators in the experiments (P�1 hf .fii : h E JRk ) such that

e.

Tn

e

+

Efficiency of Estimators

1 12

(8.2) holds for every h. Then there exists a randomized statistic T in the experiment

( N(h, le- 1 ) : h E ll�_k) such that T - ;peh has distribution Le , h for every h.

Apply Theorem 7.10 to Sn Jfi (Tn - 1/r (&) ) . In view of the definition of Le , h and the differentiability of Vr , the sequence =

Proof.

converges in distribution under h to Le, h * 8 Vro h • where *Oh denotes a translation by h. According to Theorem 7 . 1 0, there exists a randomized statistic T in the normal experiment such that T has distribution Le , h * 8 Vro h for every h . This satisfies the requirements. • This theorem shows that for most estimator sequences Tn there is a randomized estimator T such that the distribution of Jfi ( Tn - Vr (e + hI Jfi)) under e + hI Jfi is, for large n ' approximately equal to the distribution of T - ;peh under h. Consequently the standardized distribution of the best possible estimator Tn for Vr (& + hi Jfi) is approximately equal to the standardized distribution of the best possible estimator T for ;pe h in the limit experiment. If we know the best estimator T for ;peh, then we know the "locally best" estimator sequence Tn for 1/r (&). In this way, the asymptotic optimality problem is reduced to optimality in the experiment based on one observation X from a N (h , le- 1 )-distribution, in which e is known and h ranges over ffi.k . This experiment is simple and easy to analyze. The observation itself is the customary estimator for its expectation h, and the natural estimator for ;peh is ;pe x. This has several optimality properties: It is minimum variance unbiased, minimax, best equivariant, and Bayes with respect to the noninformative prior. Some of these properties are reviewed in the next section. Let us agree, at least for the moment, that ;pe x is a "best" estimator for ;pe h. The distribution of ;pe x - ;peh is normal with zero mean and covariance ;pe le- I ;pe r for every h. The parameter h 0 in the limit experiment corresponds to the parameter e in the original problem. We conclude that the "best" limit distribution of Jfi(Tn - 1/r (&)) under () is the ;pe le- 1 ;pe r)-distribution. This is the main result of the chapter. The remaining sections discuss several ways of making this reasoning more rigorous. Because the expression ;pe li 1 ;pe T is precisely the Cramer-Rao lower bound for the covariance of unbiased estimators for 1/r(&), we can think of the results of this chapter as asymptotic Cramer-Rao bounds. This is helpful, even though it does not do justice to the depth of the present results. For instance, the Cramer-Rao bound in no way suggests that normal limiting distributions are best. Also, it is not completely true that an N(h, le- 1 )-distribution is "best" (see section 8.8). We shall see exactly to what extent the optimality statement is false. =

N(O,

8.4

Estimating Normal Means

According to the preceding section, the asymptotic optimality problem reduces to optimality in a normal location (or "Gaussian shift") experiment. This section has nothing to do with asymptotics but reviews some facts about Gaussian models.

8. 4

1 13

Estimating Normal Means

Based on a single observation X from a N (h , �)-distribution, it is required to estimate Ah for a given matrix A. The covariance matrix � is assumed known and nonsingular. It is well known that AX is minimum variance unbiased. It will be shown that AX is also best-equivariant and minimax for many loss functions. A randomized estimator T is called equivariant-in-law for estimating Ah if the distri­ bution of T - Ah under h does not depend on h. An example is the estimator AX, whose "invariant law" (the law of AX - Ah under h) is the N (0, A� A T )-distribution. The follow­ ing proposition gives an interesting characterization of the law of general equivariant-in-law estimators: These are distributed as the sum of AX and an independent variable.

The null distribution L of any randomized equivariant-in-law estimator of Ah can be decomposed as L N(O, A � A T ) * M for some probability measure M. The only randomized equivariant-in-law estimator for which M is degenerate at 0 is AX.

8.4

Proposition.

=

The measure M can be interpreted as the distribution of a noise factor that is added to the estimator AX. If no noise is best, then it follows that AX is best equivariant-in-law. A more precise argument can be made in terms of loss functions. In general, convoluting a measure with another measure decreases its concentration. This is immediately clear in terms of variance: The variance of a sum of two independent variables is the sum of the variances, whence convolution increases variance. For normal measures this extends to all "bowl-shaped" symmetric loss functions. The name should convey the form of their graph. Formally, a function is defined to be bowl-shaped if the sublevel sets { x : .e (x) � c } are convex and symmetric about the origin; it is called subconvex if, moreover, these sets are closed. A loss function is any function with values in [0, oo). The following lemma quantifies the loss in concentration under convolution (for a proof, see, e.g., [80] or [1 14] .) 8.5

Lemma (Anderson 's lemma).

For any bowl-shaped loss function .e on JR.\ every prob­

ability measure M on IRk , and every covariance matrix �

J .e d N(O, � ) J .e d [N (O, � ) �

*

M] .

Next consider the minimax criterion. According to this criterion the "best" estimator, relative to a given loss function, minimizes the maximum risk sup Eh .€( T - Ah) , h

over all (randomized) estimators T. For every bow 1-shaped loss function .e, this leads again to the estimator AX.

For any bowl-shaped loss function .e, the maximum risk of any random­ ized estimator T of Ah is bounded below by Eo f ( A X). Consequently, AX is a minimax estimator for Ah. If Ah is real and E0(AX) 2 .€ (AX) < oo, then AX is the only minimax estimator for Ah up to changes on sets ofprobability zero.

8.6

Proposition.

For a proof of the uniqueness of the minimax estimator, see [18] or [80] . We prove the other assertions for subconvex loss functions, using a Bayesian argument.

Proofs.

1 14

Efficiency of Estimators

Let H be a random vector with a normal N (0, A )-distribution, and consider the original N (h , 'E )-distribution as the conditional distribution of X given H = h . The randomization variable U in T (X, U) is constructed independently of the pair (X, H) . In this notation, the distribution of the variable T - AH is equal to the "average" of the distributions of T - Ah under the different values of h in the original set-up, averaged over h using a N(O, A)-"prior distribution." By a standard calculation, we find that the "a posteriori" distribution, the distribution of H given X, is the normal distribution with mean ('E- 1 + A - 1 ) - 1 'E - 1 X and covariance matrix ( 'E - l + A - 1 ) - 1 . Define the random vectors

These vectors are independent, because WA is a function of (X, U) only, and the condi­ tional distribution of G A given X is normal with mean 0 and covariance matrix A ( .'E - 1 + A - 1 ) - 1 A T , independent of X. As A = A I for a scalar A -+ oo, the sequence G A converges in distribution to a N (O, A 'E A T )-distributed vector G. The sum of the two vectors yields T - AH, for every A. Because a supremum is larger than an average, we obtain, where on the left we take the expectation with respect to the original model, sup Eh£(T - Ah) h

�

E£ (T - AH) = E£(GA + WA)

�

E£ (GA) ,

by Anderson's lemma. This is true for every A. The lim inf of the right side as A -+ oo is at least E£(G), by the portmanteau lemma. This concludes the proof that AX is minimax. If T is equivariant-in-law with invariant law L, then the distribution of G A + WA T - AH is L, for every A . It follows that

As A -+ oo, the left side remains fixed; the first factor on the right side converges to the characteristic function of G, which is positive. Conclude that the characteristic functions of WA converge to a continuous function, whence WA converges in distribution to some vector W, by Levy's continuity theorem. By the independence of GA and WA for every A, the sequence ( G A , WA ) converges in distribution to a pair ( G, W) of independent vectors with marginal distributions as before. Next, by the continuous-mapping theorem, the distribution of GA + WA, which is fixed at L, "converges" to the distribution of G + W. This proves that L can be written as a convolution, as claimed in Proposition If T is an equivariant-in-law estimator and T (X) = E ( T (X, U) I X ) , then

8. 4.

is independent of h. By the completeness of the normal location family, we conclude that t - AX is constant, almost surely. If T has the same law as AX, then the constant is zero. Furthermore, T must be equal to its projection t almost surely, because otherwise it would have a bigger second moment than t = AX. Thus T = AX almost surely. •

8. 6

Almost-Everywhere Convolution Theorem 8.5

1 15

Convolution Theorem

An estimator sequence Tn is called regular at () for estimating a parameter 1/J ( fJ) if, for every h,

The probability measure Le may be arbitrary but should be the same for every h. A regular estimator sequence attains its limit distribution in a "locally uniform" manner. This type of regularity is common and is often considered desirable: A small change in the parameter should not change the distribution of the estimator too much; a disappearing small change should not change the (limit) distribution at all. However, some estimator sequences of interest, such as shrinkage estimators, are not regular. In terms of the limit distributions Le,h in (8.2), regularity is exactly that all Le,h are equal, for the given e. According to Theorem 8.3, every estimator sequence is matched by an estimator T in the limit experiment ( N (h , Ie- 1 ) : h E �_k). For a regular estimator sequence this matching estimator has the property .

h

T - 1/Jeh "" Le , every h.

(8.7)

Thus a regular estimator sequence is matched by an equivariant-in-law estimator for 1/Je h . A more informative name for "regular" is asymptotically equivariant-in-law. It is now easy to determine a best estimator sequence from among the regular estima­ tor sequences (a best regular sequence): It is the sequence Tn that corresponds to the best equivariant-in-law estimator T for {f;e h in the limit experiment, which is {;;e X by Proposi­ tion 8.4. The best possible limit distribution of a regular estimator sequence is the law of this estimator, a N(O, {;;e le- 1 {;;e T )-distribution. The characterization as a convolution of the invariant laws of equivariant-in-law estima­ tors carries over to the asymptotic situation.

Assume that the experiment (Pe : () E 8) is differentiable in quadratic mean (7 . 1) at the point () with nonsingular Fisher information matrix Ie . Let 1/J be differentiable at (). Let Tn be an at e regular estimator sequence in the experi­ ments ( Pen : () E 8) with limit distribution Le. Then there exists a probability measure Me such that 8.8

Theorem (Convolution).

Le

=

. . N ( 0, 1/Je li 1 1/Je T ) * Me .

_/n particular, ifLe has covariance matrix I;g, then the matrix I;e - {;;e le- I {fre T is nonnegative­ definite. Apply Theorem 8.3 to conclude that Le is the distribution of an equivariant-in-law estimator T in the limit experiment, satisfying (8.7). Next apply Proposition 8.4. •

Proof.

8.6

Almost-Everywhere Convolution Theorem

Hodges ' example shows that there is no hope for a nontrivial lower bound for the limit distribution of a standardized estimator sequence Jn ( Tn - 1/J (fJ) ) for every e . It is always

116

Efficiency of Estimators

possible to improve on a given estimator sequence for selected parameters. In this section it is shown that improvement over an N (0, ;pe le- t ;pe T )-distribution can be made on at most a Lebesgue null set of parameters. Thus the possibilities for improvement are very much restricted.

Assume that the experiment (Pe : E 8) is differentiable in quadratic mean (7 .1) at every with nonsingular Fisher information matrix Ie. Let 1/f be differentiable at every Let Tn be an estimator sequence in the experiments (P0 : E 8) such that Jn ( T11 - 1/f converges to a limit distribution Le under every Then there exist probability distributions Me such that for Lebesgue almost every 8.9

Theorem.

e.

e

e

(e))

e

e.

e

.

.

In particular, if Le has covariance matrix I:e, then the matrix I:e - 1/fe le- 1 1/fe T is nonnegative definite for Lebesgue almost every

e.

The theorem follows from the convolution theorem in the preceding section combined with the following remarkable lemma. Any estimator sequence with limit distributions is automatically regular at almost every along a subsequence of { n}.

e

Lemma. Let Tn be estimators in experiments (Pn . e E 8 ) indexed by a measurable subset 8 of iR.k . Assume that the map f--+ Pn, e (A) is measurable for every measurable set A and every n, and that the map f--+ 1/f is measurable. Suppose that there exist distributions Le such thatfor Lebesgue almost every 8.10

e e

:e

(e)

e

Then for every Yn � 0 there exists a subsequence of {n} such that, for Lebesgue almost every h), along the subsequence,

(e,

e0

Assume without loss of generality that 8 IRk ; otherwise, fix some and let Pn,e Pn,eo for every not in 8. Write Tn,e rn (Tn There exists a countable collection :F of uniformly bounded, left- or right-continuous functions f such that weak convergence of a sequence of maps Tn is equivalent to Ef (Tn) � J f d L for every f E :F. t Suppose that for every f there exists a subsequence of {n } along which

Proof.

=

e

=

=

- ljf(e)).

A. 2k - a.e.

(e, h) .

Even in case the subsequence depends on f, we can, by a diagonalization scheme, con­ struct a subsequence for which this is valid for every f in the countable set :F. Along this subsequence we have the desired convergence. t

For continuous distributions L we can use the indicator functions of cells ( - oo , c] with c ranging over Qk. For general L replace every such indicator by an approximating sequence of continuous functions. Alternatively, see, e.g., Theorem 1 . 12.2 in [ 146] . Also see Lemma 2.25.

8. 7

1 17

Local Asymptotic Minimax Theorem

e

e

Setting gn ( ) = Ee f (Tn,e) and g ( ) = f f dLe , we see that the lemma is proved once we have established the following assertion: Every sequence of bounded, measurable functions gn that converges almost everywhere to a limit g, has a subsequence along which "A 2k - a.e.

(e, h).

We may assume without loss of generality that the function g is integrable; otherwise we first multiply each gn and g with a suitable, fixed, positive, continuous function. It should also be verified that, under our conditions, the functions gn are measurable. Write p for the standard normal density on Rk and Pn for the density of the N(O, I +Y1' /)­ distribution. By Scheffe's lemma, the sequence Pn converges to p in L 1 . Let 8 and H denote independent standard normal vectors. Then, by the triangle inequality and the dominated­ convergence theorem, E l gn ( 8 + Yn H) - g ( 8 + Yn H ) I

=

J l gn (U ) - g(u) I Pn (u) du � 0.

Secondly for any fixed continuous and bounded function gc: the sequence E l gc: ( 8 + Yn H ) ­ gc: ( 8 ) I converges to zero as n � oo by the dominated convergence theorem. Thus, by the triangle inequality, we obtain E l g( 8 + Yn H ) - g ( 8 ) l

J l g - gc: l (u) (pn + p) (u) du + o(l) = 2 J l g - gc: l (u) p(u) d u + o(l). :S

Because any measurable integrable function g can be approximated arbitrarily closely in L 1 by continuous functions, the first term on the far right side can be made arbitrarily small by choice of gc: . Thus the left side converges to zero. By combining this with the preceding display, we see that El gn ( 8 + Yn H ) - g( 8) I � 0. In other words, the sequence of functions f-+ gn (e + Yn h) - g(e ) converges to zero in mean and hence in probability, under the standard normal measure. There exists a subsequence along which it converges to zero almost surely. •

(e, h)

*8.7

Local Asymptotic Minimax Theorem

The convolution theorems discussed in the preceding sections are not completely satisfying. The convolution theorem designates a best estimator sequence among the regular estimator sequences, and thus imposes an a priori restriction on the set of permitted estimator se­ quences. The almost-everywhere convolution theorem imposes no (serious) restriction but yields no information about some parameters, albeit a null set of parameters. This section gives a third attempt to "prove" that the normal N(O, 1/Je Ie- 1 1/Je T )-distribution is the best possible limit. It is based on the minimax criterion and gives a lower bound for the maximum risk over a small neighborhood of a parameter In fact, it bounds the expression

(

e.

lim lim inf sup e,.e v'fi(Tn o--+ 0 n --+oo llfJ' - fJ I <8 E

1/f(e'))).

e.

This is the asymptotic maximum risk over an arbitrarily small neighborhood of The following theorem concerns an even more refined (and smaller) version of the local maxi­ mum risk.

118

Efficiency of Estimators

Let the experiment (Pe : E 8) be differentiable in quadratic mean at with nonsingular Fisher information matrix Ie . Let 1/1 be diffe rentiable at Let Tn be any estimator sequence in the experiments ( P; : E JRk ). Then for any bowl-shaped loss function £ 8.11

e

Theorem.

e

(7.1)

e.

e

( <J; (e + �))) f e dN(O, {fr0 !0- 1 {fr0 T )

(

s �p l���f ��r Eo +h/ ,/ii e .fii T" -

:>

Here the first supremum is taken over all finite subsets I of ffi.k .

-

We only give the proof under the further assumptions that the sequence ,jn ( Tn is uniformly tight under and that £ is (lower) semicontinuous .t Then Prohorov's theorem shows that every subsequence of {n} has a further subsequence along which the vectors

Proof.

e

'ljJ(e) )

(Jll ( Tn - 'ljJ(e) ), }. L fe (Xi )) e.

7. 2 e

converge in distribution to a limit under By Theorem and Le Cam's third lemma, the sequence ,Jn ( Tn - 1/1 converges in law also under every +h j ,Jn along the subsequence. By differentiability of 1/J, the same is true for the sequence ,Jri (Tn + hj ,jn) , whence is satisfied. By Theorem the distributions Le , h are the distributions of T - ;;;e h under h for a randomized estimator T based on an N(h , lg- 1 )-distributed observation. By Proposition

(e))

(8.2)

-'ljJ(e

8. 3 ,

8. 6 ,

)

J

sup Eh £ ( T - ;;;e h) :::: Eo £ ( ;;;e X) = £ dN(O, ;;;e le- 1 ;;;e T ) . h EIRk It suffices to show that the left side of this display is a lower bound for the left side of the theorem. The complicated construction that defines the asymptotic minimax risk (the lim inf sand­ wiched between two suprema) requires that we apply the preceding argument to a carefully chosen subsequence. Place the rational vectors in an arbitrary order, and let h consist of the first k vectors in this sequence. Then the left side of the theorem is larger than

(

�)))·

( - 'ljJ (e

inf sup Ee +h/.fo e v�n rn R : = lim lim + k --'>oo n --'>oo h Eh vn There exists a subsequence {nd of {n} such that this expression is equal to

We apply the preceding argument to this subsequence and find a further subsequence along which Tn satisfies For simplicity of notation write this as {n'} rather than with a double subscript. Because £ is nonnegative and lower semicontinuous, the portmanteau lemma gives, for every h,

(8. 2 ).

'�1'1/� Eoc, 1 -Re t

(

(

# T" - o/

(e + J.,))) J e d L o , h :>

See, for example, [146, Chapter 3. 1 1] for the general result, which can be proved along the same lines, but using a compactification device to induce tightness.

8. 8

Shrinkage Estimators

119

Every rational vector h is contained in h for every sufficiently large k. Conclude that R

2:

sup h EQk

J £ dLe,h = hsupEQk Eh£ (T - {pe h ) .

The risk function in the supremum on the right is lower semicontinuous in h, by the continuity of the Gaussian location family and the lower semicontinuity of £. Thus the expression on the right does not change if Ql is replaced by �k . This concludes the proof. •

*8.8

Shrinkage Estimators

The theorems of the preceding sections seem to prove in a variety of ways that the best possible limit distribution is the N(O, {pe ie- 1 {pe T )-distribution. At closer inspection, the situation is more complicated, and to a certain extent optimality remains a matter of taste, asymptotic optimality being no exception. The "optimal" normal limit is the distribution of the estimator {pe X in the normal limit experiment. Because this estimator has several optimality properties, many statisticians consider it best. Nevertheless, one might prefer a Bayes estimator or a shrinkage estimator. With a changed perception of what constitutes "best" in the limit experiment, the meaning of "asymptotically best" changes also. This becomes particularly clear in the example of shrinkage estimators. Example (Shrinkage estimator). Let X 1 , . . . , Xn be a sample from a multivariate normal distribution with mean fJ and covariance the identity matrix. The dimension k of the observations is assumed to be at least This is essential ! Consider the estimator

8.12

3.

Because X n converges in probability to the mean e , the second term in the definition of Tn is 0 p (n - 1 ) if fJ =j:. In that case ,Jfi(Tn - X n) converges in probability to zero, whence the estimator sequence Tn is regular at every e =j:. For e = hj ,Jn, the variable ,JriXn is distributed as a variable X with an N(h , I)-distribution, and for every n the standardized estimator ylri(Tn - hj ,Jfi) is distributed as T - h for

0.

0.

X . II X II 2 This is the Stein shrinkage estimator. Because the distribution of T - h depends on h, the sequence Tn is not regular at e The Stein estimator has the remarkable property that, for every h (see, e.g., [99, p. T(X) = X - (k - 2)

-

0. 300]), =

Eh ii T - h ll 2 < Eh ii X - h ll 2 = k. It follows that, in terms of joint quadratic loss £(x) = ll x 11 2 , the local limit distributions Lo,h of the sequence ,Jri(Tn - hj ylri) under e = hj ,Jn are all better than the N (O, I)-limit distribution of the best regular estimator sequence X n . 0 The example of shrinkage estimators shows that, depending on the optimality criterion, a normal N {pe Ie- 1 {pe T )-limit distribution need not be optimal. In this light, is it reasonable

(0,

1 20

Efficiency of Estimators

to uphold that maximum likelihood estimators are asymptotically optimal? Perhaps not. On the other hand, the possibility of improvement over the N ;pe !8- 1 ;pe T )-limit is restricted in two important ways. First, improvement can be made only on a null set of parameters by Theorem Second, improvement is possible only for special loss functions, and improvement for one loss function necessarily implies worse performance for other loss functions. This follows from the next lemma. Suppose that we require the estimator sequence to be locally asymptotically minimax for a given loss function f in the sense that

(0,

8. 9 .

This is a reasonable requirement, and few statisticians would challenge it. The following lemma shows that for one-dimensional parameters local asymptotic minimaxity for even a single loss function implies regularity. Thus, if it is required that all coordinates of a certain estimator sequence be locally asymptotically minimax for some loss function, then the best regular estimator sequence is optimal without competition.

'ljr(e)

Assume that the experiment Pe E 8) is diffe rentiable in quadratic mean at e with nonsingular Fisher information matrix !8 . Let Vr be a real-valued map k that is differentiable at Then an estimator sequence in the experiments E IR ) can be locally asymptotically minimax at for a bowl-shaped loss function f such that < f x 2 f(x) dN O , ;pe l8- 1 ;pe T ) x ) < oo only if Tn is best regular at

8.13

(7.1)

( :e

Lemma.

e.

0

(

(

(P8n : e

e

e.

We only give the proof under the further assumption that the sequence ,Jn(Tn is uniformly tight under Then by the same arguments as in the proof of Theo­ rem every subsequence of {n} has a further subsequence along which the sequence y'n ( Tn - Vr + hI y'ri)) converges in distribution under + hI y'n to the distribution Le,h of T - ;pe h under h, for a randomized estimator T based on an N(h , /8- 1 )-distributed observation. Because Tn is locally asymptotically minimax, it follows that

Proof.

'ljr(e) ) 8.11,

e.

(e

e

Thus T is a minimax estimator for ;pe h in the limit experiment. By Proposition 8 .6, T = ;p8 x, whence Le,h is independent of h. •

*8.9

Achieving the Bound

If the convolution theorem is taken as the basis for asymptotic optimality, then an estimator sequence is best if it is asymptotically regular with a N(O, ;p8 !8- 1 ;p8 T )-limit distribution. An estimator sequence has this property if and only if the estimator is asymptotically linear in the score function. 8.14

mean

E

Assume that the experiment (Pe : e 8) is differentiable in quadratic (7.1) at e with nonsingular Fisher information matrix !8 . Let be differentiable at Lemma.

Vr

8. 9

Achieving the Bound

e. Let Tn be an estimator sequence in the experiments (P; : ()

121 E

JRk ) such that

Then Tn is best regular estimator for 1/J (()) at (). Conversely, every best regular estimator sequence satisfies this expansion. The sequence 11n. e = n - 1 12 L Ce ( X i ) converges in distribution to a vector !':!.. e with Ie )-distribution. By Theorem the sequence log d P;+h !-fo/ d P; is asymptotically equivalent to h T 11n.e - �h T Ieh. If Tn is asymptotically linear, then ,Jn(Tn - 1/f (())) is asymptotically equivalent to the function .(p0 /0- 1 !':!.. n , e . Apply Slutsky's lemma to find that

Proof. a N (0,

7. 2

(� (Tn - 1/f(()) ) , log dP��:Jn ) !!.. ( .(pe /0- 1 !':!..e , h T !':!..e - � h T Ieh ) ""--' N

(( - 1 h0T Ieh) ( .(p01/JehJ0- 1 {p0T T 2

0

The limit distribution of the sequence ,Jn(Tn - 1/f (())) under () + h/ ,Jn follows by Le Cam's third lemma, Example 6. and is normal with mean {p0 h and covariance matrix {p0 I0- 1 .(p0 T . Combining this with the differentiability of 1/J , we obtain that Tn is regular. Next suppose that Sn and Tn are both best regular estimator sequences. By the same arguments as in the proof of Theorem it can be shown that, at least along subsequences, the joint estimators ( Sn , Tn) for ( 1/J (()), 1/J (() ) ) satisfy for every h

7,

8.11

for a randomized estimator (S, T) in the normal-limit experiment. Because Sn and Tn are best regular, the estimators S and T are best equivariant-in-law. Thus S = T = .(p0 X almost surely by Proposition 8.6, whence ,Jfi(Sn - Tn) converges in distribution to S - T = Thus every two best regular estimator sequences are asymptotically equivalent. The second assertion of the lemma follows on applying this to Tn and the estimators

0.

Sn = 1/f (()) +

1 1/Je le- 1 11n,O ·

,Jn

°

Because the parameter () is known in the local experiments (P;+h / Jn : h E lRk ), this indeed defines an estimator sequence within the present context. It is best regular by the first part of the lemma. • Under regularity conditions, for instance those of Theorem hood estimator Bn in a parametric model satisfies

5. 3 9, the maximum likeli­

Then the maximum likelihood estimator is asymptotically optimal for estimating () in terms of the convolution theorem. By the delta method, the estimator 1/f (Bn) for 1/J (() ) can be seen

1 22

Efficiency of Estimators

to be asymptotically linear as in the preceding theorem, so that it is asymptotically regular and optimal as well. Actually, regular and asymptotically optimal estimators for e exist in every parametric model (Pe : e E 8) that is differentiable in quadratic mean with nonsingular Fisher infor­ mation throughout 8, provided the parameter e is identifiable. This can be shown using the discretized one-step method discussed in section (see

5. 7 [93]).

*8.10

Large Deviations

Consistency of an estimator sequence Tn entails that the probability of the event This is a very weak require­ d ( Tn , l/J (e) ) > E tends to zero under e , for every E > ment. One method to strengthen it is to make E dependent on n and to require that the probabilities Pe (d ( Tn , 1/f (e) ) > En ) converge to or are bounded away from for a given sequence E n -+ The results of the preceding sections address this question and give very precise lower bounds for these probabilities using an "optimal" rate En r;; 1 , typically

0.

0,

0.

1,

=

n- 1 /2 .

Another method of strengthening the consistency is to study the speed at which the probabilities Pe (d ( Tn , 1/f (e) ) > c ) converge to for a fixed E > This method appears to be of less importance but is of some interest. Typically, the speed of convergence is exponential, and there is a precise lower bound for the exponential rate in terms of the Kullback-Leibler information. We consider the situation that Tn is based on a random sample of size n from a distribution Pe , indexed by a parameter e ranging over an arbitrary set 8. We wish to estimate the value of a function 1/J : 8 f-+ ][J) that takes its values in a metric space.

0

8.15

Theorem.

0.

Suppose that the estimator sequence Tn is consistentfor 1/J (e) under every

0

e. Then, for every E > and every eo,

(

lim sup - ..!_ log Pe0 d ( Tn , 1/J (eo) ) > c n --+co n

)

:S

inf

e : d(l/f( e ) , l/f (eo)) >c

- Pe log

P eo _ Pe

If the right side is infinite, then there is nothing to prove. The Kullback-Leibler information - Pe log P eo I Pe can be finite only if Pe « Pea . Hence, it suffices to prove that - Pe log P eo I Pe is an upper bound for the left side for every e such that Pe « Pea and d ( 1/f (e ) , 1/f (eo) ) > E . The variable An = (n - 1 ) I: 7= 1 log( p e i Pea ) ( X i ) is well defined (possibly - oo ) For every constant M,

Proof

.

(

Pe0 d ( Tn , 1/f (eo) ) > c

)

:=::. :=::. ::::_

An < M ) ( Ee l { d ( Tn , 1/f (eo) ) > An < M } e -n An e - n M Pe (d ( Tn , 1/f (eo) ) > An < M) . Pea d ( Tn , 1/J (eo) ) >

E,

E,

E,

Take logarithms and multiply by - ( l i n ) to conclude that

-� log Pea (d (Tn , 1/J (eo)) >

E

)

:S

M

- � log Pe (d ( Tn , 1/J (eo) ) > 1

E,

)

An < M .

For M > Pe log pe i Pe0 , we have that Pe (An < M) -+ by the law of large numbers . Furthermore, by the consistency of Tn for 1/J (e ) , the probability Pe ( d ( Tn , 1/f (eo) ) > c )

Problems

123

converges to 1 for every e such that d ( 1/f (8) , 1/f (80)) > s . Conclude that the probability in the right side of the preceding display converges to 1 , whence the lim sup of the left side is bounded by M. •

Notes

Chapter 32 of the famous book by Cramer [27] gives a rigorous proof of what we now know as the Cramer-Rao inequality and next goes on to define the asymptotic efficiency of an estimator as the quotient of the inverse Fisher information and the asymptotic variance. Cramer defines an estimator as asymptotically efficient if its efficiency (the quotient men­ tioned previously) equals one. These definitions lead to the conclusion that the method of maximum likelihood produces asymptotically efficient estimators, as already conjectured by Fisher [48, 50] in the 1 920s. That there is a conceptual hole in the definitions was clearly realized in 195 1 when Hodges produced his example of a superefficient estimator. Not long after this, in 1 953, Le Cam proved that superefficiency can occur only on a Lebesgue null set. Our present result, almost without regularity conditions, is based on later work by Le Cam (see [95] .) The asymptotic convolution and minimax theorems were obtained in the present form by Hajek in [69] and [70] after initial work by many authors . Our present proofs follow the approach based on limit experiments, initiated by Le Cam in [95].

PROBLEMS 1. Calculate the asymptotic relative efficiency of the sample mean and the sample median for estimating based on a sample of size n from the normal N 1 ) distribution.

e,

(e,

2. As the previous problem, but now for the Laplace distribution (density p (x )

=

! e - l x l ).

3. Consider estimating the distribution function P(X ::::: x ) at a fixed point x based o n a sample X , . . . , X11 from the distribution of X . The "nonparametric" estimator is n - 1 #(Xi ::::: x ) . If it 1 is known that the true underlying distribution is normal N ( , 1), another possible estimator is (x X ) . Calculate the relative efficiency o f these estimators .

e

-

4. Calculate the relative efficiency of the empirical p-quantile and the estimator - 1 (p) +X for the estimating the p-th quantile of the distribution of a sample from the normal N (f.L , 0" 2 )­ distribution. 5. Consider estimating the population variance by either the sample variance 2 (which is unbiased) or else n - 1 L7= (Xi - :X)2 = (n - 1 ) / n S 2 . Calculate the asymptotic relative efficiency. 1 6. Calculate the asymptotic relative efficiency of the sample standard deviation and the interquartile range (corrected for unbiasedness) for estimating the standard deviation based on a sample of size n from the normal N (/L , 0" 2 ) -distribution.

Sn

n

S

e],

en )

7. Given a sample of size n from the uniform distribution on [0, the maximum X of the observations is biased downwards . Because Ee (e - X Ee X ( l ) • the bias can be removed by adding the minimum of the observations. Is X ( 1 ) + X a good estimator for e from an asymptotic point of view?

en ) ) = en )

S11 based on the mean of a sample from the N (e , I )-distribution. Jn(Sn - en)� - oo, if en ---+ 0 in such a way that n 1 14en ---+ 0 and n 1 12 en ---+ oo. Show that Sn is not regular at e = 0.

8. Consider the Hodges estimator (i) Show that

(ii)

124

Efficiency of Estimators (iii) Show that SUP-8 <e <8 Pe sufficiently slowly.

9. Show that a loss function � for a nondecreasing function

f:

(

Jni Sn

r--+

fo.

10. Show that a function of the form

-

eI

>

kn)

--+

1 for every

kn

that converges to infinity

� is bowl-shaped if and only if it has the form

f(x) = f o (lx I)

f(x) = fo (llx II) for a nondecreasing function fo is bowl-shaped .

11. Prove Anderson's lemma for the one-dimensional case, for instance b y calculating the derivative of + h) dN (0, Does the proof generalize to higher dimensions ?

J f(x

1)(x). Lemma 8.1 3 imply

12. What does about the coordinates of the S tein estimator. Are they good estimators of the coordinates of the expectaction vector?

13. All results in this chapter extend in a straightforward manner to general locally asymptotically normal models. Formulate Theorem and Lemma for such models.

8.9

8. 14

9 Limits of Experiments

A sequence of experiments is defined to converge to a limit experiment if

the sequence of likelihood ratio processes converges marginally in dis­ tribution to the likelihood ratio process of the limit experiment. A limit experiment serves as an approximation for the converging sequence of experiments. This generalizes the convergence of locally asymptotically normal sequences of experiments considered in Chapter 7. Several ex­ amples of nonnormal limit experiments are discussed.

9. 1

Introduction

This chapter introduces a notion of convergence of statistical models or "experiments" to a limit experiment. In this notion a sequence of models, rather than just a sequence of estimators or tests, converges to a limit. The limit experiment serves two purposes. First, it provides an absolute standard for what can be achieved asymptotically by a sequence of tests or estimators, in the form of a "lower bound": No sequence of statistical procedures can be asymptotically better than the "best" procedure in the limit experiment. For instance, the best limiting power function is the best power function in the limit experiment; a best sequence of estimators converges to a best estimator in the limit experiment. Statements of this type are true irrespective of the precise meaning of "best." A second purpose of a limit experiment is to explain the asymptotic behaviour of sequences of statistical pro­ cedures. For instance, the asymptotic normality or (in)efficiency of maximum likelihood estimators. Many sequences of experiments converge to normal limit experiments. In particular, the local experiments in a given locally asymptotically normal sequence of experiments, as considered in Chapter converge to a normal location experiment. The asymptotic representation theorem given in the present chapter is therefore a generalization of Theo­ rem (for the LAN case) to the general situation. The importance of the general concept is illustrated by several examples of non-Gaussian limit experiments. In the present context it is customary to speak of "experiment" rather than model, al­ though these terms are interchangeable. Formally an experiment is a measurable space (X, A) , the sample space, equipped with a collection of probability measures (Ph : h E H). The set of probability measures serves as a statistical model for the observation, written as X. In this chapter the parameter is denoted by h (and not 8), because the results are typi­ cally applied to "local" parameters (such as h = 80)). The experiment is denoted

7,

7.10

Jfi(e

-

1 25

Limits of Experiments

1 26

by (X, A, Ph : h E H) and, if there can be no misunderstanding about the sample space, also by (Ph : h E H) . Given a fixed parameter h0 E H, the likelihood ratio process with base h0 is formed as

Each likelihood ratio process is a (typically infinite-dimensional) vector of random variables dPh/dPho (X) . According to the results of section the right side of the display is Ph0 almost surely the same for any given densities Ph and Pho with respect to any measure J.L. Because we are interested only in the laws under Pho of finite subvectors of the likelihood processes, the nonuniqueness is best left unresolved.

6.1,

Definition. A sequence En = (Xn , An , Pn,h : h E H) of experiments converges to a limit experiment £ = (X, A, Ph : h E H) if, for every finite subset l C H and every ho E H,

9.1

dPn ,h ( dPn,h ( Xn) ) o

h E!

�

dPh ( dPh (X) ) o

h E!

.

The objects in this display are random vectors of length I I 1 . The requirement is that each of these vectors converges in law, under the assumption that h0 is the true parameter, in the ordinary sense of convergence in distribution in Tit! . This type of convergence is sometimes called marginal weak convergence: The finite-dimensional marginal distributions of the likelihood processes converge in distribution to the corresponding marginals in the limit experiment. Because a weak limit of a sequence of random vectors is unique, the marginal distributions of the likelihood ratio process of a limit experiment are unique. The limit experiment itself is not unique; even its sample space is not uniquely determined. This causes no problems. Two experiments of which the likelihood ratio processes are equal in marginal distributions are called equivalent or of the same type. Many examples of equivalent experiments arise through sufficiency. 9.2 Example (Equivalence by sufficiency). Let S : X r-+ Y be a statistic in the statistical experiment (X, A, Ph : h E H) with values in the measurable space (Y, B) . The experiment of image laws (Y, B, Ph s - 1 : h E H) corresponds to observing S. If S is a sufficient statistic, then this experiment is equivalent to the original experiment (X, A, Ph : h E H) . This may be proved using the Neyman factorization criterion of sufficiency. This shows that there exist measurable functions gh and f such that Ph (x) = g h ( S(x ) ) f (x ) , so that the likelihood ratio Ph/ Ph o (X) is the function gh/ gho (S) of S. The likelihood ratios of the measures Ph s - 1 take the same form. Consequently, if (Ph : h E H) is a limit experiment, then so is (Ph s- 1 : h E H). A very simple example that we encounter frequently is as follows: For a given invertible matrix J the experiments ( N ( J h , J) : h E IRd ) and ( N (h, J - 1 ) : h E IRd ) are equivalent. D o

o

o

9.2

Asymptotic Representation Theorem

In this section it is shown that a limit experiment is always statistically easier than a given sequence. Suppose that a sequence of statistical problems involves experiments

9. 3

1 27

Asymptotic Normality

En = (Pn,h : h E H) and statistics Tn . For instance, the statistics are test statistics for testing certain hypotheses concerning the parameter h, or estimators of some function of h. Most of the quality measures of the procedures based on the statistics Tn can be expressed in their laws under the different parameters. For simplicity we assume that the sequence of statistics Tn converges under a given parameter h in distribution to a limit Lh , for every parameter h. Then the asymptotic quality of the sequence Tn may be judged from the set of limit laws {Lh : h E H}. According to the following theorem the only possible sets of limit laws are the laws of randomized statistics in the limit experiment: Every weakly converging se­ quence of statistics converges to a statistic in the limit experiment. One consequence is that asymptotically no sequence of statistical procedures can be better than the best procedure in the limit experiment. This is true for every meaning of "good" that is expressible in terms of laws. In this way the limit experiment obtains the character of an asymptotic lower bound. We assume that the limit experiment E = (Ph : h E H) is dominated: This requires the existence of a O"-finite measure J.L such that Ph « J.L for every h. Recall that a randomized statistic T in the experiment (X, A, Ph : h E H) with values in IRk is a measurable map T : X x [ , 1] f-+ IRk for the product O"-field A x B orel sets on the space X x [ , 1]. Its law under h is to be computed under the product measure Ph x uniform[O, 1 ] .

0

0

Let En = (Xn , An , Pn,h : h E H) be a sequence of experiments that conver­ ges to a dominated experiment E = (X, A, Ph : h E H). Let Tn be a sequence of statistics in En that converges in distribution for every h. Then there exists a randomized statistic T in E such that Tn � T for every h. 9.3

Theorem.

The proof of the theorem starting from the definition of convergence of experi­ ments is long and can best be broken up into parts of independent interest. This goes beyond the scope of this book. The proof for the case of local asymptotic normal sequences of experiments is given in Chapter (It is shown in Theorem 9.4 that such a sequence of experiments converges to a Gaussian location experiment.) Many other examples can be treated by the same method of proof.t •

Proof

7.

9.3

Asymptotic Normality

7

As in much of statistics, normal limits are of prime importance. In Chapter a sequence of statistical models (Pn,e : e E 8) indexed by an open subset 8 c IRd is defined to be locally asymptotically normal at e if the log likelihood ratios log dPn,e + r;l hn / dPn,e allow a certain quadratic expansion. This is shown to be valid in the case that Pn,e is the distribution of a sample of size n from a smooth parametric model. Such experiments converge to simple normal limit experiments if they are reparametrized in terms of the "local parameter" h. This follows from the following theorem. Theorem. Let En = (Pn,h : h E H) be a sequence of experiments indexed by a subset H d IR of (with E H) such that 9.4

0

log t

dPn h ' = h T fj,n n, O

dP

--

-

1

-h T l h + Opn . (1),

2

For a proof of the general theorem see, for instance, [ 141].

0

1 28

Limits of Experiments

0

for a sequence of statistics � n that converges weakly under h = to a N Then the sequence En converges to the experiment (N(Jh, I) : h E H).

(0, J) -distribution.

The log likelihood ratio process with base h0 for the normal experiment has coordinates

Proof

log

1 1 d N ( J h ' I) (X) = (h - h0) T X - -h T Ih + -h 0T Ih0 . dN(Jh0 , I)

2

2

If I is nonsingular, then this follows by simple algebra, because the left side is the quotient of two normal densities. The case that I is singular perhaps requires some thought. By the assumption combined with Slutsky's lemma, the sequence log Pn , h I P n , o is under h = asymptotically normal with mean - �h T Ih and variance h T I h). This implies con­ tiguity of the sequences of measures Pn ,h and Pn , o for every h, by Example Therefore, the probability of the set on which one of P n , o , Pn , h , or P n , h o is zero converges to zero. Outside this set we can write

0

6. 5 .

1

n

n

Pn , h o

P n ,O

--

P ,h P ,h og -= log - log P

n , ho

Pn , O

.

Because this is true with probability tending to 1 , the difference between the left and the right sides converges to zero in probability. Apply the (local) asymptotic normality assumption twice to obtain that 1 1 Pn h - = (h - ho) T �n - -h T Ih + -h 0T I h o + Op ( 1 ) . log -'

2

Pn ,h o

2

ll ,

h0

On comparing this to the expression for the normal likelihood ratio process, we see that it suffices to show that the sequence �n converges under ho in law to X: In that case the vector (P n , h i P n , h o h E I converges in distribution to (dN(Jh, I)ldN(O, I) ( X ) ) h E J ' by Slutsky's lemma and the continuous-mapping theorem. By assumption, the sequence (�11 , h'5 �11) converges in distribution under h = to a vector (� , h'5 �), where � is N(O, I) -distributed. By local asymptotic normality and Slutsky's lemma, the sequence of vectors (�n , log P n ,ho I P n , o) converges to the vector (� , h'5 � - �h6 Iho) . In other words

0

( �n , log -- ) -v--+0 N( ( _ l h0T ih0 ) ' ( h TII Pn , h o Pn ,O

2 0

0

Iho h'5 Iho

) )·

6. 5 ,

By the Gaussian form of Le Cam's third lemma, Example the sequence �n converges in distribution under h0 to a N(Jh0 , I)-distribution. This is equal to the distribution of X under h0 . •

Let 8 be an open subset ofiR.d, and let the sequence of statistical models (Pn.e : () E 8) be locally asymptotically normal at () with nanning matrices rn and a nonsingular matrix Ie . Then the sequence of experiments (Pn , e +r;; l h : h E JR. d) converges to the experiment (N(h , Ie- 1 ) : h E IR.d ) . 9.5

Corollary.

9. 4

9.4

129

Uniform Distribution Uniform Distribution

[0,

The model consisting of the uniform distributions on 8] is not differentiable in quadratic mean (see Example 9.) In this case an asymptotically normal approximation is impossible. Instead, we have convergence to an exponential experiment.

7.

Let P; be the distribution of a random sample of size n from a uniform distribution on 8]. Then the sequence of experiments (P;_ h / n : h E IR) converges for each fixed e > to the experiment consisting of observing one observation from the shzfted exponential density z e-Cz - h ) /e 1 {z > h}ltl. t 9.6

Theorem.

Proof

[0, 0

r--+

If Z is distributed according to the given exponential density, then

dPhz e - ( - h )/:--e 1_{_Z_>_h_}I_B_ (Z) - _:-(-=Z---,h:-o :e- Z - ) / e 1 { Z > ho}IB dPh�

=

e (h - ho ) /e l { Z > h} '

almost surely under h0, because the indicator 1 {z > h0} in the denominator equals 1 almost surely if h0 is the true parameter. The joint density of a random sample X 1, . . . , Xn from the uniform 8] distribution can be written in the form ( 1 1 8 ) n 1 {X en ) :S 8}. The likelihood ratios take the form

[0,

Under the parameter e - hoi n, the maximum of the observations is certainly bounded above by e - hoi n and the indicator in the denominator equals 1 . Thus, with probability 1 under e - ho l n, the likelihood ratio in the preceding display can be written

(e (h - ho ) / e + o(l)) 1 { -n (X (n) - 8)

�

h}.

B y direct calculation, -n(X (n ) -8) !!:;!. Z. B y the continuous-mapping theorem and Slutsky's lemma, the sequence of likelihood processes converges under e - ho l n marginally in distribution to the likelihood process of the exponential experiment. • Along the same lines it may be proved that in the case of uniform distributions with both endpoints unknown a limit experiment based on observation of two independent ex­ ponential variables pertains . These types of experiments are completely determined by the discontinuities of the underlying densities at their left and right endpoints. It can be shown more generally that exponential limit experiments are obtained for any densities that have jumps at one or both of their endpoints and are smooth in between. For densities with discontinuities in the middle, or weaker singularities, other limit experiments pertain. The convergence to a limit experiment combined with the asymptotic representation theorem, Theorem 9.3, allows one to obtain asymptotic lower bounds for sequences of estimators, much as in the locally asymptotically normal case in Chapter We give only one concrete statement.

8.

t

Define Pe arbitrarily for e

<

0.

Limits of Experiments

130

Let Tn be estimators based on a sample X 1 , . , Xn from the uniform distribution on e] such that the sequence n (Tn - e) converges under e in distribution to a limit Le, for every e. Then for Lebesgue almost-every e we have f l x l dLe (x) � EI Z - med Z l and J x 2 dLe (x) � E(Z - EZ) 2 for the random variable Z exponentially distributed with mean e. 9. 7

Corollary.

.

[0,

8.10,

.

By Lemma the estimator sequence Tn is automatically almost reg­ ular in the sense that n (Tn - e + hjn) converges under e hjn in distribution to Le for Lebesgue almost every e and h, at least along a subsequence. Thus, it is matched in the limit experiment by an equivariant-in-law estimator for almost every e. More precisely, for almost every e there exists a randomized statistic Te such that the law of Te ( Z + h , U) - h does not depend on h (if Z is exponentially distributed with mean e). By classical statistical decision theory the given lower bounds are the (constant) risks of the best equivariant-in-law estimators in the exponential limit experiment in terms of absolute error and mean-square error loss functions, respectively. •

Proof (Sketch).

-

In view of this lemma, the maximum likelihood estimator X (n ) is asymptotically ineffi­ cient. This is not surprising given its bias downwards, but it is encouraging for the present approach that the small bias, which is of the order ljn, is visible in the "first-order" asymp­ totics. The bias can be corrected by a multiplicative factor, which, unfortunately, must depend on the loss function. The sequences of estimators

n + log 2 X (n ) and n

---

n+l

--

n

X (n )

are asymptotically efficient in terms of absolute value and quadratic loss, respectively. 9.5

Pareto Distribution

The Pareto distributions are a two-parameter family of distributions on the real line with parameters a > and fJ., > and density

0

0

This density is smooth in a, but it resembles a uniform distribution as discussed in the preceding section in its dependence on fJ.,. The limit experiment consists of a combination of a normal experiment and an exponential experiment. The likelihood ratios for a sample of size n from the Pareto distributions with parameters (a + gjJn, fJ., + hjn) and (a + go/Jn, fJ., + h0jn), respectively, is equal to

9. 6

131

Asymptotic Mixed Normality

Here, under the parameters (a + g0j,Jii , JL + h0jn) , the sequence 1 n 1 X log -i - D.. n = M a Jfi i = l

--L(

)

converges weakly to a normal distribution with mean g0ja 2 and variance 1 ja 2 ; and the sequence Zn = n (X cl ) - JL) converges in distribution to the (shifted) exponential distri­ bution with mean JL/a + h0 and variance (JJ-/a) 2 . The two sequences are asymptotically independent. Thus the likelihood is a product of a locally asymptotically normal and a "locally asymptotically exponential" factor. The local limit experiment consists of observ­ ing a pair ( D. , Z) of independent variables D. and Z with a N (g, a 2 )-distribution and an exp(a/ JL) + h-distribution, respectively. The maximum likelihood estimators for the parameters a and JL are given by A

an =

n

L 7= I log(X d X o ) )

,

and fl n = X cl ) ·

The sequence Jfi(&n - a ) converges in distribution under the parameters (a + gj Jfi , JL + h j n) to the variable D. - g . Because the distribution of Z does not depend on g, and D. follows a normal location model, the variable D. can be considered an optimal estimator for g based on the observation (D. , Z) . This optimality is carried over into the asymptotic optimality of the maximum likelihood estimator &n . A precise formulation could be given in terms of a convolution or a minimax theorem. On the other hand, the maximum likelihood estimator for JL is asymptotically inefficient. Because the sequence n (fln - JL - hjn) converges in distribution to Z - h, the estimators fl n are asymptotically biased upwards. 9.6

Asymptotic Mixed Normality

The likelihood ratios of some models allow an approximation by a two-term Taylor ex­ pansion without the linear term being asymptotically normal and the quadratic term being deterministic. Then a generalization of local asymptotic normality is possible. In the most important example of this situation, the linear term is asymptotically distributed as a mixture of normal distributions. A sequence of experiments (Pn , e : e E 8) indexed by an open subset 8 of IR.d is called locally asymptotically mixed normal at e if there exist matrices Yn , e ---+ such that

0

lo g

dPn, e +Yn ,B hn 1 = h T D. n , e - -2 h T Jn,e h + 0 Pn,e ( 1) ' dPn,e

for every converging sequence hn ---+ h, and random vectors D.. n ,e and random matrices ln , e such that ( D.. n ,e , ln,e) � ( D.. e , le) for a random vector such that the conditional distribution of D.. e given that le = J is normal N J) . Locally asymptotically mixed normal is often abbreviated to LAMN. Locally asymp­ totically normal, or LAN, is the special case in which the matrix le is deterministic. Se­ quences of experiments whose likelihood ratios allow a quadratic approximation as in the preceding display (but without the specific limit distribution of ( D.. n ,e , ln,e)) and that are

(0,

Limits of Experiments

132

such that Pn, !J+ y, eh <1 t> Pn,e are called locally asymptotically quadratic, or LAQ. We note that LAQ or LAMN requires much more than the mere existence of two derivatives of the likelihood: There is no reason why, in general, the remainder would be negligible. 9.8 Theorem. Assume that the sequence of experiments (Pn,e : () E 8) is locally asymptot­ ically mixed normal at 8. Then the sequence of experiments (Pn , !J+ y, ,eh : h E .!Rd ) converges to the experiment consisting ofobserving a pair ( !1 , J) such that J is marginally distributed as le for every h and the conditional distribution of !1 given J is normal N (J h, J).

Write Pe,h for the distribution of ( !1 , J) under h. Because the marginal distribution of J does not depend on h and the conditional distribution of !1 given J is Gaussian

Proof

By Slutsky's lemma and the assumptions, the sequence dPn,fJ + y, ,e h/ dPn,e converges under () in distribution to exp ( h T t1e - �h T leh) . Because the latter variable has mean one, it follows that the sequences of distributions Pn,!J + y, ,eh and Pn,e are mutually contiguous. In particular, the probability under () that dPn.fJ+y,,eh is zero converges to zero for every h, so that dPn, 8 + Yn . eh dPn,e + y, , eh o dPn, 8 + Yn ,eh = lo g log log + 0 P, , e dpn, 8 + y, ,eho dPn,e dPn,e

(1)

_

Conclude that it suffices to show that the sequence (t:... n ,e , ln,e) converges under () + Yn,eho to the distribution of (6. , J) under h0. Using the general form of Le Cam's third lemma we obtain that the limit distribution of the sequence (t111,e , ln,e) under () + Yn,eh takes the form

0

noting that the distribution of ( 6. , J) under h = is the same as the distribution of (t:... e , le ) , we see that this is equal to Eo s ( !1 , J) dPe,h/ dPe.o (t1 , 1) = Ph ( (6. , 1) E B) . •

On

1

It is possible to develop a theory of asymptotic "lower bounds" for LAMN models, much as is done for LAN models in Chapter Because conditionally on the ancillary statistic 1, the limit experiment is a Gaussian shift experiment, the lower bounds take the form of mixtures of the lower bounds for the LAN case. We give only one example, leaving the details to the reader.

8.

9.9

Corollary. E

Let T11 be an estimator sequence in a LAMN sequence of experiments

8) such that y,;J (Tn - 1/1 (8 + Yn,eh)) converges weakly under e ve ry () + Yn,eh to a limit distribution Le , for every h. Then there exist probability distributions M1 (or rather kernel) such that Le = EN (O, .,fr 8 18- 1 .,fr�) * Mfe · In particular, cove Le =::: . a1 Markov . E1j; e le- 1/1 eT · (Pn,e : ()

9. 6

Asymptotic Mixed Normality

133

We include two examples to give some idea of the application of local asymptotic mixed normality. In both examples the sequence of models is LAMN rather than LAN due to an explosive growth of information, occurring at certain supercritical parameter values . The second derivative of the log likelihood, the information, remains random. In both examples there is also (approximate) Gaussianity present in every single observation. This appears to be typical, unlike the situation with LAN, in which the normality results from sums over (approximately) independent observations . In explosive models of this type the likelihood is dominated by a few observations, and normality cannot be brought in through (martingale) central limit theorems. In a Galton-Watson branching process the "nth generation" is formed by replacing each element of the (n - 1 )-th generation by a random number of elements, independently from the rest of the population and from the preceding generations. This random number is distributed according to a fixed distribution, called the offspring distribution. Thus, conditionally on the size Xn _ 1 of the (n - 1)th generation the size Xn of the nth generation is distributed as the sum of Xn - 1 i.i.d. copies of an offspring variable Z. Suppose that X 0 = 1 , that we observe (X 1 , . . . , Xn), and that the offspring distribution is known to belong to an exponential family of the form

9.10

Example (Branching processes).

Pe (Z = z)

=

az 82c(8 ) ,

z

= 0,

1,

2, . . . ,

for given numbers a0, a 1 , The natural parameter space is the set of all e such that 1 c(e) = L a2 82 is finite (an interval). We shall concentrate on parameters in the interior of the naturalz parameter space such that f.L(8) : = Ee Z > 1 . Set a 2 (8) = vare Z. The sequence X 1 , X 2 , . . . is a Markov chain with transition density .

•

.

•

Pe ( Y I x) = Pe (Xn

=

y I Xn - 1

=

x

times

x) = � eYc(e) x .

To obtain a two-term Taylor expansion of the log likelihood ratios, let fe (y I x) be the log transition density, and calculate that

f e ( Y I x ) = Y - �f.L(e) ,

.. fe (y I X)

=

y - xf.L(e) x�(e) e2 e Pe (Z = z) has derivative zero yields -

(The fact that the score function of the model e f--+ the identity f.L(8) = -8 (cjc) (8), as is usual for exponential families.) Thus, the Fisher information in the observation (X 1 , . . . , Xn) equals (note that Ee (X j I X j - 1 ) = X j - 1 f.L(8))

For f.L(8) > 1, this converges to infinity at a much faster rate than "usually." Because the total information in (X 1 , . . . , X11) is of the same order as the information in the last observation X11, the model is "explosive" in terms of growth of information. The calculation suggests the rescaling rate Yn,e = f.L(e) - n l2 , which is roughly the inverse root of the information.

Limits of Experiments

1 34

A Taylor expansion of the log likelihood ratio yields the existence of a point en between e and e + Yn , e h such that

This motivates the definitions

Because Ee (Xn I Xn -1 , . . . , X 1 ) = Xn -1 /k(e), the sequence of random variables {L(e) - n Xn is a martingale under e . Some algebra shows that its second moments are bounded as n --* oo. Thus, by a martingale convergence theorem (e.g., Theorem 10.5 .4 of [42]), there exists a random variable V such that fk(e) - n Xn --* V almost surely. By the Toeplitz lemma (Problem 9.6) and again some algebra, we obtain that, almost surely under e, 1 n 1 n 1 /k(e) x --* v V. --* X " " 1 -· n n fk(e) - 1 ' /k(e) - 1 fk(e) f=; 1 {L(e) f=I J --

It follows that the point en in the expansion of the log likelihood can be replaced by e at the cost of adding a term that converges to zero in probability under e . Furthermore,

Jn , e

r--+

j.J,(e) v e (!k(e) 1 ) , _

Pe -almost surely.

It remains to derive the limit distribution of the sequence 6.n , e . If we write X j for independent copies Zj,i of the offspring variable Z, then

=

:2:: ��1 1 Zj,i

for independent copies zi of z and Vn = :2:: 7= 1 X j -1 · Even though 2 1 ' Zz , . . . and the total number Vn of variables in the sum are dependent, a central limit theorem applies to the right side: conditionally on the event { V > 0} (on which Vn --* oo), the sequence v,:;- 1 12 :2:: �: 1 ( Zi - /k(e)) converges in distribution to (]" (e) times a standard normal variable G. Furthermore, if we define G independent of V, conditionally on { V > 0}, t

(

(]" (e) G , j.1, (8 ) V (6.n ,e , ln , e) -v-t -- r-v� e y e ( !k(e) 1 )

t

See the appendix of [8 1] or, e.g., Theorem 3.5.1 and its proof in [146].

_

)

(9. 1 1 ) ·

9. 6

Asymptotic Mixed Normality

135

It is well known that the event { V = 0} coincides with the event {lim Xn = 0} of extinction of the population. (This occurs with positive probability if and only if a0 > 0.) Thus, on the set { V = 0} the series l....: � 1 X 1 converges almost surely, whence ll n,e -+ 0. Interpreting zero as the product of a standard normal variable and zero, we see that again is valid. Thus the sequence (lln,e , ln, e ) converges also unconditionally to this limit. Finally, note that a 2 (e)je = jL(e) , so that the limit distribution has the right form. The maximum likelihood estimator for J.L (e) can be shown to be asymptotically efficient, (see, e.g., or 0

(9 .11)

[29] [81]).

The canonical example of an LAMN sequence of exper­ iments is obtained from an explosive autoregressive process of order one with Gaussian innovations. (The Gaussianity is essential.) Let 1 e I > and c: 1 , c: 2 , be an i.i.d. sequence of standard normal variables independent of a fixed variable X0. We observe the vector (X o , X 1 , . . . ' Xn ) generated by the recursive formula x t = ex t - 1 + E t · The observations form a Markov chain with transition density p ( · I x1_ 1 ) equal to the N (ex t -l , ) -density. Therefore, the log likelihood ratio process takes the form 9.12

Example (Gaussian AR).

1

.

.

•

1

1

n n Pn, HYn,o h � � 2 2 2 log (Xo, . . . , X n ) = h Yn, e � (X t - e x t - l ) X t - 1 - l h Yn, e � X t 1 · Pn, e t =1 t =1 This has already the appropriate quadratic structure. To establish LAMN, it suffices to find the right rescaling rate and to establish the joint convergence of the linear and the quadratic term. The rescaling rate may be chosen proportional to the Fisher information and is taken Yn, e = e - n · By repeated application of the defining autoregressive relationship, we see that t r e - x t = Xo + L: e - 1 c: 1 -+ V : = X o + L: e - 1 c: 1 , ) =1 )= 1 00

almost surely a s well a s in second mean. Given the variable X0, the limit i s normally distributed with mean X0 and variance (e 2 - ) - 1 . An application of the Toeplitz lemma (Problem yields

9. 6)

1

The linear term in the quadratic representation of the log likelihood can (under e) be rewritten as e -n :z.=;=1 c:1Xr - h and satisfies, by the Cauchy-Schwarz inequality and the Toeplitz lemma,

It follows that the sequence of vectors ( ll n, e , ln, e ) has the same limit distribution as the ) For every n the vector (e- n :z.=;=1 c:1 sequence of vectors (e -n :z.=;=1 c:1e t - 1 V, V 2 I (e 2

- 1) .

136

Limits of Experiments

e t - I , V) possesses, conditionally on X0, a bivariate-normal distribution. As n --* oo these distributions converge to a bivariate-normal distribution with mean (0, X0) and covariance matrix I / (8 2 Conclude that the sequence (�n.e , ln,e) converges in distribution as required by the LAMN criterion. D

1).

9.7

Heuristics

9. 3 ,

The asymptotic representation theorem, Theorem shows that every sequence of statistics in a converging sequence of experiments is matched by a statistic in the limit experiment. It is remarkable that this is true under the present definition of convergence of experiments, which involves only marginal convergence and is very weak. Under appropriate stronger forms of convergence more can be said about the nature of the matching procedure in the limit experiment. For instance, a sequence of maximum like­ lihood estimators converges to the maximum likelihood estimator in the limit experiment, or a sequence of likelihood ratio statistics converges to the likelihood ratio statistic in the limit experiment. We do not introduce such stronger convergence concepts in this section but only note the potential of this argument as a heuristic principle. See section for rigorous results. For the maximum likelihood estimator the heuristic argument takes the following form. If hn maximizes the likelihood h f--+ dPn h , then it also maximizes the likelihood ratio process h f--+ dPn,h fdPn,h o · The latter sequence of processes converges (marginally) in distribution to the likelihood ratio process h f--+ dPh/ dPh o of the limit experiment. It is reasonable to expect that the maximizer hn converges in distribution to the maximizer of the process h f--+ dPh/dPho ' which is the maximum likelihood estimator for h in the limit experiment. (Assume that this exists and is unique.) If the converging experiments are the local experiments corresponding to a given sequence of experiments with a parameter () , then the argument suggests that the sequence of local maximum likelihood estimators hn = rn (en - 8) converges, under 8 , in distribution to the maximum likelihood estimator in the local limit experiment, under h = 0. Besides yielding the limit distribution of the maximum likelihood estimator, the argu­ ment also shows to what extent the estimator is asymptotically efficient. It is efficient, or inefficient, in the same sense as the maximum likelihood estimator is efficient or ineffi­ cient in the limit experiment. That maximum likelihood estimators are often asymptotically efficient is a consequence of the fact that often the limit experiment is Gaussian and the maximum likelihood estimator of a Gaussian location parameter is optimal in a certain sense. If the limit experiment is not Gaussian, there is no a priori reason to expect that the maximum likelihood estimators are asymptotically efficient. A variety of examples shows that the conclusions of the preceding heuristic arguments are often but not universally valid. The reason for failures is that the convergence of experiments is not well suited to allow claims about maximum likelihood estimators. Such claims require stronger forms of convergence than marginal convergence only. For the case of experiments consisting of a random sample from a smooth parametric model, the argument is made precise in section 7 .4. Next to the convergence of experiments, it is required only that the maximum likelihood estimator is consistent and that the log density is locally Lipschitz in the parameter. The preceding heuristic argument also extends to the other examples of convergence to limit experiments considered in this chapter. For instance, the maximum likelihood estimator based on a sample from the uniform distribution on [0, 8]

5. 9

137

Problems

is asymptotically inefficient, because it corresponds to the estimator Z for h (the maximum likelihood estimator) in the exponential limit experiment. The latter is biased upwards and inefficient for every of the usual loss functions . Notes

This chapter presents a few examples from a large body of theory. The notion of a limit experiment was introduced by Le Cam in [95] . He defined convergence of experiments through convergence of all finite subexperiments relative to his deficiency distance, rather than through convergence of the likelihood ratio processes. This deficiency distance in­ troduces a "strong topology" next to the "weak topology" corresponding to convergence of experiments. For experiments with a finite parameter set, the two topologies coincide. There are many general results that can help to prove the convergence of experiments and to find the limits (also in the examples discussed in this chapter). See [82] , [89] , [96] , [97], [ 1 15], [ 1 38], [142] and [144] for more information and more examples. For nonlocal ap­ proximations in the strong topology see, for example, [96] or [1 10] . PROBLEMS 1. Let

X 1 , . . . , Xn be an i.i.d. sample from the normal N (hiJn, 1 ) distribution, in which h

E

JR.

The corresponding sequence of experiments converges to a normal experiment by the general results . Can you see this directly? 2. If the nth experiment corresponds to the observation of a sample of size n from the uniform

[0, 1 - h I n], then the limit experiment corresponds to observation of a shifted exponential variable Z . The sequences -n (X (n) - 1) and .jii ( 2Xn - 1) both converge in distribution under every h . According to the representation theorem their sets of limit distributions are the distributions of randomized statistics based on Z. Find these randomized statistics explicitly. Any implications regarding the quality of X (n) and Xn as estimators ?

3. Let the nth experiment consist o f one observation from the binomial distribution with parameters

n and success probability h I n with 0 < h < 1 unknown. Show that this sequence of experiments converges to the experiment consisting of observing a Poisson variable with mean h .

4 . Let the nth experiment consists of observing an i.i.d. sample o f size n from the uniform

[ - 1 - h l n , 1 + h l n] distribution. Find the limit experiment.

5. Prove the asymptotic representation theorem for the case in which the nth experiment corresponds

to an i.i.d. sample from the uniform [0, e - h I n] distribution with h of this theorem for the locally asymptotically normal case.

>

0 by mimicking the proof

L an = oo and Xn x an L J = l a j X j IL} = l a j converges to --+

6. (Toeplitz lemma.) If an is a sequence of nonnegative numbers with

arbitrary converging sequence of numbers, then the sequence x as well. Show this.

7. Derive a limit experiment in the case of Galton-Watson branching with /L (e) 8. Derive a limit experiment in the case of a Gaus sian AR( l ) process with e

=

<

1.

1.

9 . Derive a limit experiment for sampling from a U [u, r ] distribution with both endpoints unknown.

10. In the case of sampling from the U [O, e ] distribution show that the maximum likelihood estimator

for e converges to the maximum likelihood estimator in the limit experiment. Why is the latter not a good estimator?

1 1 . Formulate and prove a local asymptotic minimax theorem for estimating e from a sample from

a U [O, e ] distribution, using

£(x)

=

x 2 as los s function.

10 Bayes Procedures

In this chapter Bayes estimators are studied from afrequentist perspec­ tive. Both posterior measures and Bayes point estimators in smooth parametric models are shown to be asymptotically normal. 10.1

Introduction

In Bayesian terminology the distribut�on Pn,e of an observation X n under a parameter () is viewed as the conditional law of X n given that a random variable Gn is equal to () . The distribution [] of the "random parameter" B n is called the prior distribution, and the conditional distribution of en given X n is the posterior distribution. If Bn possesses a density n and Pn,e admits a density Pn,e (relative to given dominating measures), then the density of the posterior distribution is given by Bayes' formula

This expression may define a probability density even if n is not a probability density itself. A prior distribution with infinite mass is called improper. The calculation of the posterior measure can be considered the ultimate aim of a Bayesian analysis. Alternatively, one may wish to obtain a "point estimator" for the parameter (), using the posterior distribution. The posterior mean E(Gn I X n) = f () Pe" 1 x" (()) d() is often used for this purpose, but other location estimators are also reasonable. A choice of point estimator may be motivated by a loss function. The Bayes risk of an estimator Tn relative to the loss function e and prior measure n is defined as

Here the expectation Ee £ (Tn - ()) is the risk function of Tn in the usual set-up and is identical to the conditional risk E ( £(Tn - Gn ) I en = e) in the Bayesian notation. The corresponding Bayes estimator is the estimator Tn that minimizes the Bayes risk. Because the Bayes risk can be written in the form EE ( £ C Tn - Gn) I X n ) , the value Tn = Tn (x ) minimizes, for every fixed x, the "posterior risk" -:;-

....

E (£(Tn - Gn) I Xn

_ -

X

)

-

j £ (Tn - ()) pn,e (x ) d f1 ( fl ) . J Pn,e (x ) d[] (()) 138

1 0. 1

Introduction

139

Minimizing this expression may again be a well-defined problem even for prior densities of infinite total mass. For the loss function .£ (y) = I y f, the solution Tn is the posterior mean E(8n I X n ), for absolute loss .£ (y) = II Y II , the solution is the posterior median. Other Bayesian point estimators are the posterior mode, which reduces to the maximum likelihood estimator in the case of a uniform prior density; or a maximum probability estimator, such as the center of the smallest ball that contains at least posterior mass (the "posterior shorth" in dimension one). If the underlying experiments converge, in a suitable sense, to a Gaussian location experiment, then all these possibilities are typically asymptotically equivalent. Consider the case that the observation consists of a random sample of size n from a density Pe that depends smoothly on a Euclidean parameter e . Thus the density Pn,e has a product form, and, for a given prior Lebesgue density n , the posterior density takes the form

1/2

Typically, the distribution corresponding to this measure converges to the measure that is degenerate at the true parameter value e0, as n � oo. In this sense Bayes estimators are usually consistent. A further discussion is given in sections and To obtain a more interesting limit, we rescale the parameter in the usual way and study the sequence of posterior distributions of JfiC8 n e0) , whose densities are given by

10. 2 10. 4 .

-

If the prior density n is continuous, then n (eo + hj Jfi), for large n, behaves like the constant n (e0) , and n cancels from the expression for the posterior density. For densities Pe that are sufficiently smooth in the parameter, the sequence of models (Peo + h! -fii : h E JR1.k ) is locally asymptotically normal, as discussed in Chapter This means that the likelihood ratio processes h r----+ n7=1 Peo +h! -fii / Pea (Xi ) behave asymptotically as the likelihood ratio process of the normal experiment ( N (h , Ie� 1 ) : h E IR1. k ) . Then we may expect the preceding display to be asymptotically equivalent in distribution to

7.

dN (h ' /e-o 1 ) (X) J dN(h, Ie� 1 ) (X)dh

=

dN(X ' Ie-o 1 ) (h) '

where dN(JL, I; ) denotes the density of the normal distribution. The expression in the preceding display is exactly the posterior density for �he experiment ( N (h , Ie� 1 ) : h E IR1.k ) , relative to the (improper) Lebesgue prior distribution. The expression on the right shows that this is a normal distribution with mean X and covariance matrix Ie� 1 . This heuristic argument leads us to expect that the posterior distribution of Jfi(8 n e0) "converges" under the true parameter e0 to the posterior distribution of the Gaussian limit experiment relative to the Lebesgue prior. The latter is equal to the N (X, Ie� 1 )­ distribution, for X possessing the N Ie� 1 ) -distribution. The notion of convergence in this statement is a complicated one, because a posterior distribution is a conditional, and hence stochastic, probability measure, but there is no need to make the heuristics precise at this point. On the other hand, the convergence should certainly include that "nice" Euclidean­ valued functionals applied to the posterior laws converge in distribution in the usual sense. -

(0,

140

Bayes Procedures

Consequently, a sequence of Bayes point estimators, which can be viewed as location functionals applied to the posterior distributions, should converge to the corresponding Bayes point estimator in the limit experiment. Most location estimators (all reasonable ones) map symmetric distributions, such as the normal distribution, into their center of symmetry. Then, the Bayes point estimator in the limit experiment is X, and we should expect Bayes point estimators to converge in distribution to the random vector X, that is, to a N /8� 1 )-distribution under e0 . In particular, they are asymptotically efficient and asymptotically equivalent to maximum likelihood estimators (under regularity conditions). A remarkable fact about this conclusion is that the limit distribution of a sequence of Bayes estimators does not depend on the prior measure. Apparently, for an increasing number of observations one's prior beliefs are erased (or corrected) by the observations. To make this true an essential assumption is that the prior distribution possesses a density that is smooth and positive in a neighborhood of the true value of the parameter. Without this property the conclusion fails . For instance, in the case in which one rigorously sticks to a fixed discrete distribution that does not charge e0, the sequence of posterior distributions of Gn cannot even be consistent. In the next sections we make the preceding heuristic argument precise. For technical reasons we separately consider the distributional approximation of the posterior distributions by a Gaussian one and the weak convergence of Bayes point estimators. Even though the heuristic extends to convergence to other than Gaussian location exper­ iments, we limit ourselves in this chapter to the locally asymptotically normal case. More precisely, we even assume that the observations are a random sample X 1 , . . . , Xn from a distribution Pe that admits a density Pe with respect to a measure fJ., on a measurable space (X, A) . The parameter e is assumed to belong to a measurable subset 8 of IR k that contains the true parameter e0 as an interior point, and we assume that the maps (e , x ) f-+ Pe (x) are jointly measurable. All theorems in this chapter are frequentist in character in that we study the posterior laws under the assumption that the observations are a random sample from Pe0 for some fixed, nonrandom e0. The alternative, which we do not consider, would be to make probability statements relative to the joint distribution of (X 1 , . . . , X n , Gn) , given a fixed prior marginal measure for Bn and with Pen being the conditional law of (X 1 , . . . , Xn) given Gn .

(0,

10.2

Bernstein-von Mises Theorem

The heuristic argument in the preceding section indicates that posterior distributions in dif­ ferentiable parametric models converge to the Gaussian posterior distribution N (X, Ii/ ) . The Bernstein-von Mises theorem makes this approximation rigorous and actually yields the approximation in a stronger sense than discussed so far. In Chapter it is seen that the observation X in the limit experiment is the asymptotic analogue of the "locally sufficient" statistics

7

where fe is the score function of the model. The Bernstein-von Mises theorem asserts that the total variation distance between the posterior distribution of yln(8n - e0) and the random distribution N (l:!.n,e0 , /8� 1 ) converges to zero. Because l:!.n,eo X, this has as a -v--+

1 0. 2

Bernstein-von Mises Theorem

141

consequence that the posterior distribution of .Jii( Gn - 80) converges, in any reasonable sense, in distribution to N (X , 18� 1 ) . The conditions of the following version of the Bernstein-von Mises theorem are re­ markably weak. Besides differentiability in quadratic mean of the model, it is assumed that there exists a sequence of uniformly consistent tests for testing H0 : e = 80 against In other words, it must be possible to separate the H1 : 118 - 80 11 ::::_ 8, for every 8 > true value 80 from the complements of balls centered at 80. Because the theorem implies that the posterior distributions eventually concentrate on balls of radii Mn f .Jn around 80, for every Mn --+ oo, this separation hypothesis appears to be very reasonable. Even more so, since, as is noted in Lemmas and under continuity and identifiability of the model, separation by tests of Ho : (} = eo from H1 : II (} - eo II ::::. 8 for a single (large) 8 > already implies separation for every 8 > Furthermore, if 8 is compact and the model continuous and identifiable, then even the separation condition is superfluous (because it is automatically satisfied). t

0.

10. 4 10. 6 , 0.

0

Let the experiment (Pe : (} E 8) be diffe rentiable in quadratic mean at 8o with nonsingular Fisher information matrix le0, and suppose that for every 8 > there exists a sequence of tests
Theorem (Bernstein-von Mises).

0

sup P; ( l -
11 8 - 8o ll :>-s

0.

(10. 2)

Furthermore, let the prior measure be absolutely continuous in a neighborhood of 80 with a continuous positive density at 80. Then the corresponding posterior distributions satisfy

Throughout the proof we rescale the parameter (} to the local parameter h = .jn(e - 8o) . Let On be the corresponding prior distribution on h (hence Dn (B) = 0 (8o + B j .Jii ) ), and for a given set C let n; be the probability measure obtained by restricting nn to C and next renormalizing. Write Pn,h for the distribution of Xn = (X 1 , . . . , Xn) under the original parameter 8o + hj .Jii, and let Pn, c = f Pn,h dn; (h) . Finally, let H n .jn(811 - 8o) , and denote the posterior distributions relative to On and n; by Pfin 1 x" and pH!::n I Xn , respectively. The proof consists of two steps. First, it is shown that the difference between the posterior measures relative to the priors nn and n;" , for Cn the ball with radius Mn , is asymptotically negligible, for any Mn --+ oo. Next it is shown that the difference between N (t:.... n ,e0 , /8� 1 ) and the posterior measures relative to the priors n;" converges to zero in probability, for some Mn --+ oo. For U, a ball of fixed radius around zero, we have Pn, u <1 r>Pn,o, because Pn,h n <1 r>Pn ,O for every bounded sequence hn , by Theorem Thus, when showing convergence to zero in probability, we may always exchange Pn,o and Pn, u . Proof.

=

-

7.2.

t

Recall that a test is a measurable function of the observations taking values in the interval [0, 1 ] ; in the present context this means a measurable function ¢n : xn r-+ [0, 1].

Bayes Procedures

142

Let Cn be the ball of radius Mn . By writing out the conditional densities we see that, for any measurable set B , PH

n

I X (B) - PH�" 1 xn

n

n

(B) = PH

n

I X (C� n B ) n

PH

n

I X (C� ) PH�" I X- (B) . n

n

n

Taking the supremum over B yields the bound The right side will be shown to converge to zero in mean under Pn , u for U a ball of fixed radius around zero. First, by assumption and because Pn, u <J Pn , o , Manipulating again the expressions for the posterior densities, we can rewrite the first term on the right as

For the tests given in the statement of the theorem, the integrand on the right converges to zero pointwise, but this is not enough. By Lemma there automatically exist tests ¢n for which the convergence is exponentially fast. For the tests given by the lemma the preceding display is bounded above by

10.3,

{ e - cC I h ll zAn ) d On (h) . On (U) Jll h ii�M"

1

Here nn (U) = n (Ito + u I Jn) is bounded below by a term of the order 1/ Jnk , by the positivity and continuity of the density n at 80 . Splitting the integral into the domains sufficiently small that n (tl) is uniformly Mn ::::: l l h II ::::: D,Jfi and ll h I 2: D,Jfi for D bounded on 1111 - 110 11 ::::: D, we see that the expression is bounded above by a multiple of

::::: 1

This converges to zero as n , Mn -+ oo. In the second part of the proof, let C be the ball of fixed radius M around zero, and let c N (f-L, :E ) be the normal distribution restricted and renormalized to C. The total variation distance between two arbitrary probability measures P and Q can be expressed in the form l i P - Q ll = j ( l - pjq) + d Q . It follows that

2

H N c ( t. n , eo , Ie� l ) = <

-

PlL xJ

+ d N C ( t. n,eo , /-1 e0 ) (h) ( ) 1 f l c (h)Pn , h (Xn)nn (h)j fc Pn, g (Xn) nn (g ) dg d P�" I X_ (h) !! ( 1 - Pn, g (Xn) nn (g )dN c ( t.n , eo , le� 1 ) (h) ) + d N c ( t.n,eo /-eo ! ) (g ) d P� _

-+

-+

Pn,h (Xn) nn (h) d N C ( t. n,e0 , le-o 1 ) ( g ) -+

H

"

'

H " I X_ " (h) '

1 0. 2 Bernstein-von Mises Theorem

143

(1

because - EY ) + :::; E ( l - Y ) + . This can be further bounded by replacing the third occurr ence of N c ( D. n,e0 , Ie� 1 ) by a multiple of the uniform measure A c on C. By the dominated-convergence theorem, the double integral on the right side converges to zero in mean under Pn, C if the integrand converges to zero in probability under the measure -+

(Note that Pn, c is the marginal distribution of Xn under the Bayesian model with prior []� .) Here []� is bounded up to a constant by A c for every sufficiently large n. Because Pn,h <1 1> Pn,o for every h, the sequence of measures on the right is contiguous with respect to the measures "A c (dh) Pn,o (dx) "A c (dg ) . The integrand converges to zero in probability under the latter measure by Theorem 7.2 and the continuity of n at e0 . This is true for every ball C of fixed radius M and hence also for some Mn � oo. •

Under the conditions of Theorem 10. 1, there exists for every Mn � oo a sequence of tests such that, for every sufficiently large n and every 10.3

Lemma.

0

11 e - eo ll :::: Mnf y'ii,

P�
0,

We shall construct two sequences of tests, which "work" for the ranges Mn/ y'ii :::; I e - eo I ::::: E and I e - eo II > E' respectively, and a given E > Then the such that the matrix Pe0 l� l� is nonsingular. Fix such an L and define

Proof

0.

[-

By the central limit theorem, Pe: Wn � triangle inequality,

0

0, so that Wn satisfies the first requirement. By the

Because, by the differentiability of the model, Pel� - Pe0 #0 = ( Pe0 l� l� + o ( l ) ) (e - eo ) , the first term on the right i s bounded below b y c 11 e - e0 II for some c > for every e that is sufficiently close to e0, say for 11 e - e0 II < E . If Wn = then the second term (without the minus sign) is bounded above by .j Mnfn. Consequently, for every c 11 e - eo I :::: 2.j Mnfn, and hence for every 11 e - e0 II :::: Mn/ y'ii and every sufficiently large n,

0,

0,

[117]),

by Hoeffding's inequality (e.g., Appendix B in for a sufficiently small constant C . Next, consider the range 11 e - e0 II > E for an arbitrary fixed E > B y assumption there exist tests
P�
0,

sup P; ( l -
lle - ea ll > c:

0.

0.

Bayes Procedures

1 44

It suffices to show that these tests can be replaced, if necessary, by tests for which the convergence to zero is exponentially fast. Fix k large enough such that P� ¢k and Pt ( 1 - ¢k ) for every ll 6l - 6lo II > 8. Let n = mk + r for :::: r < k, and define are smaller than Yn, I , . . . , Yn, m as ¢k applied in turn to X I , . . . , Xb to Xk + I , . . . , X 2b and so forth. Let Y n, m be their average and then define Wn = 1 { Yn, m � 1 2 } . Because Ee Yn,J � for every > 8 and every j , Hoeffding's inequality implies that 6l 80 I ll < e- 2m ( � - � f < e- m /8 . pen ( 1 - wn ) = Pe (Y n, m < 1 2) -

0

1/4

/

3/4

/

Because m is proportional to n , this gives the desired exponential decay. Because Ee0 Yn,J :::: 1 j the expectations P� Wn are similarly bounded. •

4,

The Bernstein-von Mises theorem is sometimes written with a different "centering se­ quence." By Theorem 8. 14 any sequence of standardized asymptotically efficient estimators y'ri (en - 6l) is asymptotically equivalent in probability to �n . e · Because the total variation distance is bounded by a multiple of I �n . e - y'ri (en - 6l) II , any such sequence y'ri(en - 6l) may replace �n . e in the Bernstein-von Mises theorem. By the invariance of the total variation norm under location and scale changes, the resulting statement can be written

Under regularity conditions this is true for the maximum likelihood estimators en . Com­ bining this with Theorem we then have, informally,

5. 3 9

and since conditioning en on en = e gives the usual "frequentist" distribution of en under e . This gives a remarkable symmetry. Le Cam's version of the Bernstein-von Mises theorem requires the existence of tests that are uniformly consistent for testing H0 : 6l = 80 versus HI : ll 6l - 80 11 � 8, for every 8 > Such tests certainly exist if there exist estimators Tn that are uniformly consistent, in that, for every 8 >

0.

0,

sup Pe ( II Tn - e I � 8 ) --*

e

0.

In that case, we can define ¢n = 1 { II Tn - 6lo I � 8 j2 } . Thus the condition of the Bernstein­ von Mises theorem that certain tests exist can be replaced by the condition that uniformly consistent estimators exist. This is often the case. For instance, the next lemma shows that this is the case for a Euclidean sample space X provided, for Fe the distribution functions corresponding to the Pe ,

e ' l >c II l e -inf

Fe - Fe , ll oo >

0.

10.2 Bernstein-von Mises Theorem

145

For compact parameter sets, this is implied by identifiability and continuity of the maps (] f-+ Fe . We generalize and formalize this in a second lemma, which shows that uniformity on compact subsets is always achievable if the model (Pe : f3 E 8) is differentiable in quadratic mean at every f3 and the parameter f3 is identifiable. A class of measurable functions :F is a uniform Glivenko-Cantelli class (in probability) if, for every c; > 0, sup Pp ( II JPln - P ll .r > 8) --* 0. p

Here the supremum is taken over all probability measures P on the sample space, and I I Q ll .r = sup / EF I Qf l . An example is the collection of indicators of all cells ( -oo, t] in a Euclidean sample space. 10.4

Lemma.

every c; > 0,

Suppose that there exists a uniform Glivenko-Cantelli class :F such that,for inf I Pe - Pe ' ll .r > 0. d (B, e ' ) > s

(10. 5 )

Then there exists a sequence of estimators that is uniformly consistent on 8 for estimat­ ing e. 10.6 Lemma. Suppose that 8 is CJ -compact, Pe f3 f-+ Pe are continuous for the total variation

f. Pe ' for every pair f3 f. fJ', and the maps

norm. Then there exists a sequence of estimators that is uniformly consistent on every compact subset of 8.

For the proof of the first lemma, define en to be a point of (near) minimum of the map (] f-+ II JPln - Pe ll .r· Then, by the triangle inequality and the definition of en , II Pe" - Pe ll .r :::: 2 11 JPln - Pe ll .r + ljn, if the near minimum is chosen within distance ljn of the true infimum. Fix c; > 0, and let o be the positive number given in condition ( 1 0.5). Then

Proof.

By assumption, the right side converges to zero uniformly in f3 . For the proof of the second lemma, first assume that 8 i s compact. Then there exists a uniform Glivenko-Cantelli class that satisfies the condition of the first lemma. To see this, first find a sequence A 1, A 2 , . . . of measurable sets that separates the points Pe . Thus, for every pair fJ , (]' E 8, if Pe (Ad = Pe , ( A d for every i , then f3 = fJ'. A separating collection exists by the identifiability of the parameter, and it can be taken to be countable by the continuity of the maps f3 f-+ Pe . (For a Euclidean sample space, we can use the cells ( -oo, t] for t ranging over the vectors with rational coordinates. More generally, see the lemma below.) Let :F be the collection of functions x f-+ i - 1 1 A , (x). Then the map h : 8 f-+ £00 (F) given by f3 f-+ ( Pe f) fEF is continuous and one-to-one. By the compactness of 8, the inverse h - 1 : h ( 8 ) f-+ 8 is automatically uniformly continuous. Thus, for every c; > 0 there exists o > 0 such that

ll h(fJ) - h(fJ') ll .r :::: o implies d(fJ, (]') :::: c:.

Bayes Proudurt!s

1 46 This

means that ( I 0.5) is satis fied . The class :F is

because by Chebyshev's inequality,

This co nc lud e s the

also a u n i fonn Gl ivenko-Cantclli class,

proo f of the second Jemma for compact 0.

To remove the compactness condition. wri te e as the union of an i ncreasing sequence

of compact sets K, c Kz c T,,,, that is unifonnly co n s i st e n t

· · · .

fixed m ,

a,.. m

6,.

=

O t: K.,

the p reced i ng

(d(T,,,m, 0) .!.) 2:

111

sequ e nce m,J _.. oo such that a11 .m,. Tn.m,. sat i s fi es the requirements� •

Then the re e x i st s that

: = s up Po

f-or every m there cxisrs

on Km . by

a

a

seq u en ce of e�timators .

argu ment.

-.

0,

�

0 as " __.

ll _..

Thus. for every

00.

oo.

It is not h a rd

to see

s econd lemma, if there cx isrs a sequence of tests t/Jn such that . ( J 0.2) holds fo r some t: > 0, th e n it holds for every & > 0. In t h a t ca� we can replace the g i \'e n sequence r/Jn by t h e minimum of t/J, and the tes t s I { nT,. - Oo ll 2: £/2 J for a sequ e nce of estimators T,, that is un i fonn ly consistent on a s uftici en r J y large subset of B. As

a

consequence of the

1 0.7 !..emma. Let the .fel ofprobabilit)' mt•a.wre.'i P mr a uu•a.wrablt• ,'ifWCe (,\'. A> be .'it>parable for tlu.• total \'tlriation llorm. Then the re exists ll countable .mb.H•t Ao C A .m ch tlwt P1 = P2 011 Ao implies Pa = 1'2 for f!\ el) Pa . P!. E P. '

Proof.

'

The set P can be i de n t i fi e d with a subset of L a (Jl )

for a su it able prob abil i ty m�asurc

11. . For i n s t a n ce. 11. cim be t a ke n a con\'ex li near co mb i nat i o n of a cou nt abl e dense set. Let

Po be a cou n t_abl e dense·subset, and Jet Ao be the set of all finite intersections of the sets p -• ( B) for p ran g i ng over a choice of densi ties of rhc set Po c L 1 (J.I. ) and R ranging O\'er

a

countable generator of the Borel seas in lR.

l11en every density p E Po is O"(Ao)- mcasumblc by construction. A den sity of a measure

P E P - P0 can be approximated in L 1 (Jt ) by a seq u e n ce from Pu and hence can be chosen a (A0)-measurable. without loss of ge nc m l i ty .. Because Ao is intersection-stable (a rr-systcm ) two probabi lity measures th�1t &Jgrec on Ao a uto m at i cal ly agree on the a -ficJd a (Ao ) generated by A0• Then t hey also gi\'c the same ex pec t a t io n to every a CAo)·mcasurable fu nct ion f : X .- [0, 1 1. If the measures have u (.A0)-measurabJ c d e n s i ti e s then they must agree on A, because P(t\ ) = E,, J " p = E,, E,. ( I 11 I a (..40)) p if p is a (.Ao)-measumble. • .

..

.

.

1 0.3

Point

:Estimators

The Bcmstcin-von M ises t he orem shows that the post e rio r laws con ve rge in distribution to a G a u ss i a n posterior l aw in total variation d i s ta n ce As a con seq ue n ce, any loca t i on functional .

that is sui tably cont inuous re l is t i ve to the total v�lriation norm appl ied

to the sequence of

10.3 posterior laws c onverge s

Poilll Estimators

1 47

to the same location functional applied to the limiring G"ussian X, or a N (0. Ii. ' )-distribution.

pos erior distribution. For most choices thi s means to

t

this sect ion we cons i der more general Bayes point est i mators that arc de fi ned as the e u i s relati\'e to some l os s fu nction. For a given loss function l : R4 H> [0, oo), le t Tn . for fixed X 1 X,. minimize the posrerior risk In

mini mizers of th posterior risk f nct on

•

, H>

• • • •

f e( Jti
the minimizing values T,, can be selected as a measurable is an im plic it assumption, or ot herwise the statem� n ts arc lo be understO 0, It is not imme d at ely clear that

i

fun cti n of the o bse rva ti on s . This

o

.

-

sup JhU::M

t'(lz ) ::;

inf ((h ). lh11�2M

with strict i nequality for at least one AI. t TI1is is tr e, for instance, for loss f n c o ns

u

u ti

of

= lo(lllr n) for a nondecre as i n g function to : [0. 00) H- [0, 00) that is not (0, oo). Fu n herm orc, we suppose that t. grows at most poly no m ia lly: For some constant p � 0,

the form l(h)

con s ta n t on

t !: 1 +

n'z ll ''·

10. 1 hold, ami /el l saris/)· the conclition.'i as .jii( TN - Oo) (·om·ery:es tmder Oo in distribution to the minimi:.er ofl � f ((1 - h ) d N (X, 1,� 1 )(h ). for X possessing 1 the N (0, /� )-cli.ftribution, prm•iclecl rlrat cmy two minimi�ers ofrhis process c:oindde almost .wrt!ly. In particular, for C\'er_\· nun :.ero, .wbcom·t•x loss fwtc·ticm it (.'On\'('rges w X. 10.8

Theorem. Let tile conditions ofThearem

a J ne Jl'' cl n {0 )

list(•d, for a p .'illc:ll tlz l

*Proof.

We ad op the notation as

t

< 00.

Then the .f('CJIIt!IICI!

li�ted in the fi rst paragraph of the proof of Theorem I 0. 1 .

Th e last assertion of the theorem is a consequence of Anderson's l emm a, Lemma 8.5. The standardized e st i m at or

where t, is t h

.jii( T,,

-

0o) minimizes

the function

e funct ion II ._,. l(t - h ). TI1c proof consists of t h ree parts. First it is shown

that i n te grals over the �els llll II � M, can be neglected for every M, _,

that t h e

sequence J,j( T,,

proc e sses 1 � the process

Next, it is proved

- 0o) is unifonn ly t ight. Fi n a l l y, it is shown that the stochastic Z,(l ) conve rge in distribution in the space t'"' ( K), for every compact K. to

I �

t

oo.

Z(t) =

J

l (t - h ) cl

Tho! 2 i s for cunvcnicm:c. any other numhcr would do.

N(X. 1,� 1 )(!z).

Bayes Procedures

148

t,

The sample paths of this limit process are continuous in in view of the subexponential growth of and the smoothness of the normal density. Hence the theorem follows from the argmax theorem, Corollary Let Cn be the ball of radius Mn for a given, arbitrary sequence Mn --+ oo. We first show that, for every measurable function f that grows subpolynomially of order p ,

.e

5. 5 8.

(1 0 .9 ) To see this, we utilize the tests
Here Tin ( U) is bounded below by a term of the order 1 j Jnk , by the positivity and continuity at (}0 of the prior density n . Split the integral over the domains Mn I :::; D,jii and I I ::: D,jii, and use the fact that J II (} d TI ((}) < oo to bound the right side of the display by terms of the order e - A M; and ,jnk +P e -B n , for some A , B > 0. These converge to zero, whence ( 1 0.9) has been proved. Define .C(M) as the supremum of £(h) over the ball of radius M, and f_(M) as the infimum over the complement of this ball. By assumption, there exists 8 > 0 such that ry : = f_(28) - l(8) > 0. Let U be the ball of radius 8 around 0. For every I :=: 3Mn and sufficiently large Mn , we have .C ( - h) - £(-h) :=: 'f] if h E U, and .C ( - h) - £(-h) :=: fo_(2Mn) - .C(Mn) ::: 0 if h E uc n Cn , by assumption. Therefore,

lh

:::; l h

liP

t

ti l

t

[(.e (t -

h) - £(-h)) ( 1 u + 1 ucn c" + 1 c�) ] Zn (t) - Zn (O) = PH" I Xn ::: ry Piln i X" (U) - pil" l x " (£ (-h) 1 cJ . Here the posterior probability Piln 1 X n (U) of U converges in distribution to N(X, !8� 1 ) (U) , by the Bernstein-von Mises theorem. This limit is positive almost surely. The second term in the preceding display converge� to zero in probability by ( 1 0 .9 ) . Conclude that the infimum of Zn - Zn (0) over the set of t with II I :=: 3Mn is bounded below by variables that converge in distribution to a strictly positive variable. Thus this infimum is positive with probability tending to one. This implies that the probability that H> Zn has a minimizer in the set II ::: 3Mn converges to zero. Because this is true for any Mn --+ oo, it follows that the sequence .jn (Tn - (}0) is uniformly tight. Let C be the ball of fixed radius M around 0, and fix some compact set K c �k . Define stochastic processes

(t)

t

t

lit

N (li n ,e0 , l8� 1 ) (£ , 1 c ) , WM = N(X, /8� 1 ) (£ 1 1 c ) .

Wn, M

=

(t)

1 0.4

Consistency

149

The function h 1---+ C (t - h) l c (h) is bounded, uniformly if t ranges over the compact K . Hence, b y the Bernstein-von Mises theorem, Zn , M - Wn,M � 0 in £00(K) as n -+ oo, for every fixed M. Second, by the continuous-mapping theorem, Wn,M WM in £00(K), as n -+ oo, for fixed M. Next WM � Z in £00(K) as M -+ oo, or equivalently C t JR.k . Conclude that there exists a sequence Mn -+ oo such that the processes Zn,M" Z in £00(K). Because, by ( 1 0.9), Zn (t) - Zn,M" (t) 0, we finally conclude that Zn Z in £00 (K) . a '¥'7

'¥'7

�

*10.4

'¥'7

Consistency

A sequence of posterior measures Pe" 1 x1 , . . . , x" is called consistent under () if under Pe00probability it converges in distribution to the measure 8e that is degenerate at (), in proba­ bility; it is strongly consistent if this happens for almost every sequence X 1 , X2 , . . . . Given that, usually, ordinarily consistent point estimators of () exist, consistency of posterior measures is a modest requirement. If we could know () with almost complete accuracy as n -+ oo, then we would use a Bayes estimator only if this would also yield the true value with similar accuracy. Fortunately, posterior measures are usually consistent. The following famous theorem by Doob shows that under hardly any conditions we already have consistency under almost every parameter. Recall that 8 is assumed to be Euclidean and the maps () 1---+ Pe (A) to be measurable for every measurable set A. 10.10 Theorem (Doob 's consistency theorem). Suppose that the sample space ( X , A) is a subset of Euclidean space with its Borel CJ -field. Suppose that Pe =f. Pw whenever () =f. ()'. Then for every prior probability measure n on 8 the sequence of posterior measures is consistent for n -almost every ().

On an arbitrary probability space construct random vectors 8 and X 1 , X 2 , . . . such that 8 is marginally distributed according to n and such that given 8 = () the vectors X 1 , X2 , are i.i.d. according to Pe . Then the posterior distribution based on the first n observations is Pe l X t , . . . , Xn . Let Q be the distribution of (X 1 ' X2 , . . . ' 8) on x oo X e . The main part o f the proof consists of showing that there exists a measurable function h : X00 1---+ 8 with

Proof.

.

.

•

Q-a.s .. Suppose that this is true. Then, for any bounded, measurable function f : 8 Doo b 's martingale convergence theorem,

(10. 1 1 ) 1---+

JR., by

E ( / (8) I X 1 , . . . , Xn ) -+ E ( / (8) I X 1 , X 2 , . . . ) f (h ( X l , X 2 , . . . ) ) , Q-a.s . .

2. 25

B y Lemma there exists a countable collection :F of bounded, continuous functions f that are determining for convergence in distribution. Because the countable union of the associated null sets on which the convergence of the preceding display fails is a null set, we have that Q-a. s . .

IJuy�s Procedures

I SO

This statement refers to the marginal distribution of

(X 1 , X 2,

• • •

)

under

Q.

We wish

to

translate it intO a State ment concerning the P800-mea� urcs. Let C C X� X 0 be th� inter­

section of the sets

on

which the weak convergence hoJds and on which

Fubini's theorem

( 15.9)

is valid. By

Ct, = {x : (.t, 0) E C} is the horizontal section of C at height 0. It follows PtJ00 (CH ) = J for n -al most every 0. For e\'cry 8 such that P,]"�(C0) = J . we have (x, 0) e C for P�-al most every sequence x 1 , x1 • • • • and hence where

This is lhe assertion of the theorem.

In

order 10 establ ish

( 1 5.9),

call a measurable function

exi sts a sequence of measumblc functions

hn: XH

II jh,(x) - f(8)j l\

I

�

I:e

�-+

R such t hat

lhat that

R m·cessibl� i f there

dQ(:r, 0) -+ D.

(Here we abuse notation in viewing II, also as a measurable function on x� x

there also exi sts a {sub)scquence with h H (.t ) -+

H.)

Then

/(0 ) al most surely under Q. whence every I is almost everywhere equal to an Ax. x {0. 0}-mcasurable fu nction. This is a measurable function of :c = (.t1 , x2 , ) alone. If we can show c hat rhc functions f (0 ) = 0; arc accessible, then ( 15.9) follows. \Ve s ha ll in fact show that every Borel accessible function

• • •

measurable function is accessi ble.

L�::l

By the strong law of la�Je numbers, hn (x) =

P/f'.

for every

theorem.

0 and

mcasumble set A.

II lhn(X) -

J " (x, )

-

PH (A ) al most surely under

Consequent ly, by th� domin ated convergence

Po (A )

J dQ(:c. 0)

__.,

0.

of the functions 0 � ��(A ) is accessible. (..\-, A) is Eucl ide<J n by ass umption, there ex ists a countnble measure-deter· m i ning subcollccti on Ao C A. The fu nctions 0 H- PH ( A ) are measur.1ble by a!\sumpt ion and scpararc the points of (-) as A ranges O\'Ct .Ao. in view of the choice of Ao and 1hc Thus each

Because

ident ifiability of the parameter 0.

a-field on e. in view of Lemma

This implies that these fu nctions gcnera1e the Borel

1 0. 1 2.

The proof is complete once it

is

shown rhat every function that is measurable in the

u -fi dd generated by the accessible functions (which is the Borel u - field) is accessible.

From lhe de finition it follows ea�ily that the

set

of accessible fu nctions

is a

veclor .space.

contains the constant functions, is closed under mon otone li mits. and is a lattice.

desired result therefore follows by a monotone class argument. as

1llc merit of the preceding theore m is that it imposes hardly

drawback is that i t gives the

consistency o n ly

up

to

in

Lemma I 0. J 3.

any

The

•

conditions. but its

null sets of possible para mercrs (de­

pending on the prior). In certain ways these null sets can be quite larg�. and examples ha\'C

10.4

ConJi.'lteiiQ'

151

been co n st ru cted where Bayes estimators behave badly. To guarantee co ns i ste n cy under

every parameter it is n ece s sary to impose some fu rt h e r cond it ions. Because in this c ha p ter we arc m a i n l y concerned with asymptotic n omm l i ty of B aye s esti mators (which implies c o ns i st e ncy w i t h a ra te), we omit a di scus..'\ion.

1 0.12

[..emma. Let :F be a cmmtab/e co//('CiiOII ofmeu.mmblefiml'tions I : e c R.l:

that .'i('[mratl'S the points of e. Then the Borel a-field and the a

coincide.

.... ..

R

field generated by F Olt e

·

Proof. By assumption, the m a p h : E-J �--+ IR.F defined by h {O ) f = /(0) is mea s u rabl e and one-to-one . B eca use F is c ou n tabl e . the B ore l a - field on RF (for the product topology) is equal to the a-field gcnernted by the coordinate projections. Hence the a-fields generated by It an d F (viewed as Borel mt!a.'\urabJe maps in ;rtF an d lR.. respectively) on e are idcnrica l. Now ,- • . defi ned on the range of h. is automatically Borel measur.Ihlc, by Pro pos i t i o n in (24}. and hen ce <� and h ( (-) ) arc B o re l isomorph ic. •

8.3.5

F bt.• U lilll'Gr :rub.'ipace of r.. ( n ) with the properties (i) if f. g E :F. tht.•ll I " g E :F: ( ii ) ifO � f• !: l!. � . . e F mul J,� t f E .C , ( n ). then f e :F: (iii) I E :F.

10. 13

Lemma.

Let

·

Then :F coma ins e1·ery (t (:F) -m£•a.'iurttble jimction in

£1 ( n ).

Proof. Because a n y a (F )-measurable n onn egat i ve function is the mo n o t o n e l i mit of a seq u e n ce of s i m p l e fu nctions. it suffices to p ro ve that I A e :F tor every t1 e a (F ). Define An = {A : 1 A e F} . Then An is an intersect ion-stable Dynkin .system a n d h e nce a l1 -field. Furthermore. for C\'Cry f E F a n d a E R. the fu n c t i o n s n C / - ar+- 1\ I arc contai ned in F and i ncrease poi n tw i se to l r r > u t · I t follows that ( .f' > a } E A0 • l ienee a (F ) c A1• •

Notes

The Bern stein-von Miscs t he o re m has that n a me , because. ;.l s Le Cam and Yang (97 ) write. it was fi rst discovered by Lapl ace.

The th�orem th01t is prese nt e d in t h b c hap t e r is consi daably more e lcgu n t than t h e results _by these early authors. and also m uc h better than the result in Le Ca m (91 J. who revi ved the thcorcin in o rd er to prove resu lts on superefficiency. \Vc adapted it from Le C am 1 96) and Lc Cam and Ya ng (97 1. lb rag i mov and Hasmi nski i (80] discuss the convergence of Bayes poi nt c st im < o rs i n greater gencmlity, an d also cover non-Gaussian l i m i t experi ments. b u t their discussion of the i.i.d. case as discussed i n the p re se n t ch a pt e r is li mited to bo u n de d parameter sets and re q u i re s st ro n ge r a ssum pt i on s . Our t re at m e n t uses .some c l e me n t s of their proof. but is heav ily based on Lc C6l m·s Bernstei n-von Mises theorem. I nspection of t he proof shows thnt tht.! co nd it i ons on t h e loss fu n ct i o n cun be relaxed s ig n i l i cu m l y. for i n st ance a l l ow i n g e x po n e n t i a l growth. Doob•s theorem origi n t c n ti :a l null se t s of in co n si s t en cy that it leaves o�n re a lly exist in some s i t u ut i o n s pa rt i c u l ar ly if the param e te r set is infinite di men sio n al .

152

Bayes Procedures

and have attracted much attention. See [34] , which is accompanied by evaluations of the phenomenon by many authors, including Bayesians.

PROBLEMS 1. Verify the conditions of the B ernstein-von Mises theorem for the experiment where Pe is the

Poisson measure of mean

e.

2. Let Pe be the k-dimensiona1 normal distribution with mean

e

and covariance matrix the identify. Find the a posteriori law for the prior IT = N ( r, A) and some nonsingular matrix A . Can you see directly that the B ernstein-von Mises theorem is true in this case?

3. Let Pe be the Bernoulli distribution with mean

e. Find the posterior distribution relative to the

beta-prior measure, which has density

4. Suppose that, in the case of a one-dimensional parameter, we use the loss function £ (h)

1 (- 1 ,2) (h) .

Find the limit distribution of the corresponding Bayes point estimator, assuming that the conditions of the B ernstein-von Mises theorem hold.

11 Projections

A projection of a random variable is defined as a closest element in a given set offunctions. We can use projections to derive the asymptotic distribution of a sequence of variables by comparing these to projections of a simple form. Conditional expectations are special projections. The Hajek projection is a sum of independent variables; it is the leading term in the Hoeffding decomposition.

1 1.1

Proj ections

A common method to derive the limit distribution of a sequence of statistics Tn is to show that it is asymptotically equivalent to a sequence Sn of which the limit behavior is known. The basis of this method is Slutsky's lemma, which shows that the sequence Tn = Tn - Sn + Sn converges in distribution to S if both Tn - Sn � 0 and Sn S. How do we find a suitable sequence Sn ? First, the variables Sn must be of a simple form, because the limit properties of the sequence Sn must be known. Second, Sn must be close enough. One solution is to search for the closest Sn of a certain predetermined form. In this chapter, "closest" is taken as closest in square expectation. Let T and { S : S E S} be random variables (defined on the same probability space) with finite second-moments. A random variable S is called a projection of T onto S (or L 2 -projection) if S E S and minimizes "-"7

s

E

S.

Often S is a linear space in the sense that a 1 S1 + a 2 S2 is in S for every a 1 , a 2 E R, whenever S 1 , S2 E S. In this case S is the projection of T if and only if T - S is orthogonal to S for the inner product (S 1 , S2 ) = ES 1 S2 . This is the content of the following theorem. Let S be a linear space of random variables with finite second moments. Then S is the projection of T onto S if and only if S E S and

11.1

Theorem.

E(T - S) S

=

0,

every S

E

S.

Every two projections of T onto S are almost surely equal. If the linear space S contains the constant variables, then ET = ES and cov (T - S, S) = 0 for every S E S. 153

1 54

Projtcrions

Figure J l.J. A variable T and its projcclion S on :. linear sp::�ce. For any

Proof.

S and S in S,

E(T - S)2 If

=

E(T - S)2 + 2E(T - S)(S - S) + E(S - s)2 •

S satisfies the orthogonality condition, then the middle tenn is zero, and we conclude E(T - S)2 ?: E(T - S)2• with suict i nequal ity un less E(S - S)'2 0. Thus. the

that

=

orthogonality condition impl ies that Conversely, for any number a,

If

a

S t-)lo

is

a

S is a projection. and also lh
a

proj ectio n, then this �xpression is nonneg tive for every a.

a2ES2 -

E(T - S)S

=

2aE(T - S)S

0 is satisfied.

B ut the pambora

is nonne gativ

e if and on ly if the orthogon al ity condition

If the constants are in S, then the o rthogon al ity

a

the l st asse rtions of the theorem follow.

•

condition implies E(T - S)c

=

0, whence

The theorem does n ot assert that projections always exi st. This is not true: The in­ fimum infs

E(T - S)'2 need not be achieved.

A sufficie nt condition for existence is that

S is closed for the second- moment norm, but existence is usuaJJy more easily establ ished

i

d rectly. The orthogonality of

(Sec Figure 1 1 . 1 .)

T - S an d S yie lds the Pythagorean ru le ET2

=

E(T - S)2 + E.f!.

If the constants arc contained in S. then this is also true for variances

instead of second moments.

Tn and linear spaces S,. is given. For each n, let S, t T,, follows from that of Sn , and vice versa. provided the quotient varT,/varSn converges to I . Now sup pose a .sequence of �t t i stics

a

be the proj ecti on of Tn on S,. . Then the l imi i ng behavior of the sequence

I I .2

Theorem. Let

S, be linear J1Jt1Ces of random

l-'ariables with fin ite secot1d moments

that comain tlze cmzsltmts. Let Tn be random variables with projections

varT,,jvarS, -.

I

tlze11

Tn - ET,. sd Tn

_

S ,. ���� � O. sd Sn

S,

onto S, . If

J 1.2

Conditional E.xpectatimr

1 55

Proof. \Ve shall prove convergence in second mean, which is stronger. The expectation of the d i fference is zero. lls variance is eq ual to 2

_

2

S,. ) . sd T" sd S,,

cov(T,..

" :!

By the o rthogon al i ty of T,. - S, and S,. , it follows that ET,,S,. = ES,. . Because the constants arc in S,,, this implies that cov(T,, , S, ) = varS,., an d the the ore m follows. • A

A

A

The condition v:.arT,./varS,. --. 1 in the theorem implies that the projections S,. are asymptotically of the same size as the original Tn . This explains that "nothing is Josf' in the limit� and that the difference between T,, and its projection converges to zero. In the preceding theorem it is essential that the S, arc the projections of the variables T,. , because the condition varT11/varSn --. I for general sequences Sn and T, does not imply anything.

1 1 .2

Conditional Expectation

1l1e expectat ion EX of a random variable X minimizes the quadrJtic form a H> E(X - a) :! over the real n u mbers a. This may be expressed as fo l lows: EX is the best prediction of X. given a quadratic loss function, and in the absence of additional infom1ation. The cmulitimwl expectation E(X I Y) of a random variable X given a random \'ector Y is defined as the best '"prediction"' of X given knowledge of Y. Form al l y, E( X l Y) is a measurable function Ro( Y) of Y that minimizes

over all measurable functions g. In the tem1inology of the preceding section, E(X I Y) is the projection of X onto the l inear space of all measurable functions of Y. It follows that the conditional expectation is the unique measurable function E(X 1 Y) of Y that satisfies the orthogonality relation

E(X - E< X I Y)) �: C Y) = 0.

e\'ery g.

IfE(X I Y) = go( Y}, then it is customary to write E(X I Y = )' ) for Ro(y). This is i nterpreted as the expected value of X given that Y = y is observed. B y Theorem l l . l the projection is unique only up to changc:s on set-; of probabi lity zero. This means that the function g0(y) is un ique up to sets B of values·.,, such that P( Y e B) = 0. (These could be very hig sel<>.) The fo l lowing examples gi\'c some propert ies und also describe the rci:Jtionship with conditional densities. 1 1 .3

Example. The orthogonal ity relationship with g

EE(X I Y).

Thus, ··the expectation

= I yi el d s the formu la EX o f a conditional expectation is the expectation."' 0

/( Y) for a measur.tble fu nction f� th e n ECX I Y) = X. This foll ows immediately from the defin ition. in which the minimum can be reduced to zero. The interpretation is that X is perfectly predictable given knowledge of Y. 0 1 1 .4

Example. If X

=

156

Projections

11.5 Example. Suppose that (X, Y) has a joint probability density f (x , y) with respect to a a- -finite product measure J1, x v , and let f (x I y) = f (x , y) jfy (y) be the conditional density of X given Y = y . Then

E(X I Y) =

f xf (x I Y) d f.l,(x) .

(This i s well defined only if fy (Y) > 0.) Thus the conditional expectation a s defined above concurs with our intuition. The formula can be established by writing E ( X - g (Y) ) 2 =

J [! (x - g(y) ) 2 f(x I y) d f.l,(X ) ] fy (y) d v (y) .

To minimize this expression over g, it suffices to minimize the inner integral (between sq­ uare brackets) by choosing the value of g(y) for every y separately. For each y, the integ­ ral j (x - a ) 2 f (x I y) d f.l,(x) is minimized for a equal to the mean of the density x !----'? f (x I y). D If X and Y are independent, then E(X I Y) = EX. Thus, the extra knowl­ edge of an unrelated variable Y does not change the expectation of X . The relationship follows from the fact that independent random variables are uncorre­ lated: Because E(X - EX)g (Y) = 0 for all g, the orthogonality relationship holds for go (Y) = EX . D 11.6

Example.

Example. If f is measurable, then E ( f (Y) X I Y ) = f ( Y)E(X I Y) for any X and Y. The interpretation is that, given Y, the factor f(Y) behaves like a constant and can be "taken out" of the conditional expectation. Formally, the rule can be established by checking the orthogonality relationship. For every measurable function g, 1 1.7

E ( / (Y)X - f(Y)E(X I Y) ) g(Y) = E ( X - E(X I Y) ) f (Y)g ( Y) = 0, because X - E(X I Y) is orthogonal to all measurable functions of Y, including those of the form f(Y)g (Y) . Because f(Y)E(X I Y) is a measurable function of Y, it must be equal to E ( / (Y)X I Y ) . D If X and Y are independent, then E ( f (X, Y) I Y = y ) = Ef(X, y) for every measurable f. This rule may be remembered as follows: The known value y is substituted for Y; next, because Y carries no information concerning X, the unconditional expectation is taken with respect to X. The rule follows from the equality 11.8

Example.

E ( / (X, Y) - g(Y) ) 2 =

ff ( / (x , y) - g (y)) 2 d Px (x ) d Py (y) .

Once again, this i s minimized over g by choosing for each y separately the value g(y) to minimize the inner integral. D 11.9

Example.

For any random vectors X, Y and Z, E (E(X I Y, Z) I Y ) = E(X I Y) .

11.4

Hoeffding Decomposition

157

This expresses that a projection can be carried out in steps: The projection onto a smaller set can be obtained by projecting the projection onto a bigger set a second time. Formally, the relationship can be proved by verifying the orthogonality relationship E (E( X I Y, Z) - E( X I Y) ) g ( Y) = 0 for all measurable functions g. By Example 1 1 .7, the left side of this equation is equivalent to EE ( Xg(Y) I Y, z) - EE ( g(Y)X I Y ) = 0, which is true because conditional expectations retain expectations. D 1 1 .3

Proj ection onto Sums

Let X 1 , . . . , Xn be independent random vectors, and let S be the set of all variables of the form

n

L gi (Xi ),

i=1 for arbitrary measurable functions gi with Eg[ (Xi )

<

oo. This class is of interest, because

the convergence in distribution of the sums can be derived from the central limit theorem. The projection of a variable onto this class is known as its Hajek projection.

Let X 1 , . . . , Xn be independent random vectors. Then the projection of an arbitrary random variable T with finite second moment onto the class S is given by n S = L E(T I Xd - (n - 1)ET. i=1 11.10

Lemma.

The random variable on the right side is certainly an element of S . Therefore, the assertion can be verified by checking the orthogonality relation. Because the variables X; are independent, the conditional expectation E (E(T I Xi) I Xi ) is equal to the expectation EE(T I X; ) = ET for every i -=1- j . Consequently, E(S I X1 ) = E(T I X1 ) for every j , whence

Proof.

This shows that T - S is orthogonal to S.

•

Consider the special case that X 1 , . . . , Xn are not only independent but also identically distributed, and that T = T (X 1 , . . . , Xn) is a permutation-symmetric, measurable function of the X; . Then

E(T I Xi = x) = ET (x , X 2 , . . . , Xn) . Because this does not depend on i , the projection S is also the projection of T onto the smaller set of variables of the form L 7= 1 g(Xi ), where g is an arbitrary measurable function. * 1 1 .4

Hoeffding Decomposition

The Hajek projection gives a best approximation by a sum of functions of one X; at a time. The approximation can be improved by using sums of functions of two, or more, variables. This leads to the Hoeffding decomposition.

Projections

1 58

B ecau se a projection onto a sum of orthogonal spaces i s the sum of the projections onto the individual spaces, it is convenient to decompose the proposed projection space into a su m of orthogonal sp aces . Given independent variables X� o , X, and a su bset A C { I • . . . • 11 }. let 11" denote the set of all squ are-integrable mndom variables of the ty� • • •

g,.. (X, : i

E

A).

for measurable fu nctions g,.. o f lA I argument� such that

E(gA (Xl : i E A ) I Xj : j E B) = 0,

every

B : IBI

<

l A I.

(Define E(T ) �) = E T. ) By the independence of X 1 , • • • • X, the condi tion in the last d i sp lay is automatically valid for any B c { I . 2 , "} that docs not contain A. Con­ sequently, the spaces HA. when A ranges o\'er all subset� of { I , n}, are pairwise orthogonal. Stated in i ts present fonn. the condit ion reflects the i n ten ti on to bu ild ap­ proximations of increasing complexity by project ing a gi\'en vari able in tum onto the spaces • . . .

. . . •

[ 1 ).

....

where 81i J € Hr; J . gli.jl e 1/ri. J t . - and so forth. Each new space is chosen orthogonal to the precedi ng s paces . Let P,. T denote the project ion of T onto HA . Then, by the orthogonality of the //" , th·e projection onto the sum of the fi rst r spaces is the sum 2:1,.. 1 �, PA T of the projections onto the i nd ivi du al spaces. The projection onto the sum of the fi rst two spaces is the Hajek projection. 1\-fore general ly. the projections of zero, first. and seco nd order can be seen to be

PttT = ET, Plit T = E ( T I X; ) - ET, PII.J I T = E(T I X; . Xj ) - E ( T I X1> - E(T I X1 ) + ET. Now the general fonnula given by the fol lowing lemma should n ot be surprisi ng. Lemma. Let X 1 ,

1 1.1 1

, Xn be independelll random \'a riables, and let T be till arbi· ET2 < oo. Th en the projection of T onto HA i.r gi,·en by

• • •

trary random ''ariable u.oith

PA T = L: C - 1 )1"1-IBIE(T I X, : i

E

B ).

BCA

If T .L 1/8 for every subset

B C A

of a given set

quently. the sum of the .'ipaces H8 with

(Xt : i

Proof.

E

A).

BCA

A.

then E( T I X1 : i e

A)

=

0. Collse­ of

contains all square- imegrab/efunctions

8A (X; : i E A) to g,.. . By the inde­ B) = E( T I A n B) for every su bsets A

Abbre\'iate E(T I Xs : i E A) to E(T I A) and pendence of X1 • • • • • Xn i t follows that E(E(T I A ) I

1/oeffi/ing Decompo.'iitimr

1 1.4

and B of ( 1 • . . . , n J . Thus. for in A,

1 59

P�t T as defined in the lemma and a set C strictly contained

E( PA T 1 C) = 2: <- 1 )1.41-1" 1 E< T 1 B n C> BCA IAI -1<"1 IA I = L L ( - I )lltJ-1/) 1 - j

(

DCC

)

� ICI E(T I D). J

J=O

By the binomial fonnu la, the inner sum is zero for every D. Thus the left side is zero. I n view o f the fonn of P�t T, i t was not a loss o f generality t o assume that C c A. Hence P" T is contai ned in 1/A . Next we ve ri fy the orthogonal ity relationship. For any measurable function g,, .

E(T - Pit T)g" = E{T - E(T I A ))g" -

L (- 1 > 1 " 1 - 1 81 EE( T I B)E(glt I 8).

IleA IJ 'fo/1

This is zero for any g" E HIt . This concludes the proof that P" T is as given. \Ve pro ve the second assertion of the lemma by induction on r = lA J. If T .1 H.., . then E(T I �) = ET = 0. Thus the assertion is true for r = 0. Su ppose that it is true for 0, r - I , and conside r a set A of r clements. If T J. H11 for e\'cry 8 c A. then certainly T .L He for every C c B. Consequent ly. the induct ion hypothesis shows that E(T 1 /l) = 0 for every B C A of r - I or fewer elements. The fonnula for PA T now shows_ that P,, T = E(T I A). By assumption the left side is zero. This concl udes the induction argument. The final assertion of the lemma follows i f the variable T,,. : = T - LBc A PH T is zero for every T that depends on (X1 : i e 1\ ) only. But in this ca.se T,. depends on (X, � i e A ) only and hence equals Err,, t A). which i s zero. because T�t l. H11 for every B c A. • . . . •

If T = T(X. , . . . . X, ) is pcnnutat ion-symmctric and X1 X, a rc independent and identically distributed. then the Hocffding decomposition of T can be simpli fied to •

,

• • • •

,

T

=

L L g,(X; : i

r=O I AI=r

E

A).

for

g, (x ,

•

. . .

, x,)

=

L

o c u . .. . . rJ

(- l )r- I B I ET (.t,

E

B, X; (j

B).

The inner s u m in the representation of T is for each r a U-statistic of order r (as discussed in the Chapter 1 2). with dege nerate kernel. All terms in the sum are orthogonal, whence the variance of T can be found as var T = L�- • (;)Eg!(X� o . . . . X, ).

Notes Orthogonal projections in H ilbert spaces (complete inner product spaces) arc a classical sub­ ject in fu nctional analys is. We have limited our discussion to the li ilbert space L2{Q, U. P) of all square-integrable random variables on a probabi lity space. Another popu lar method to

Projections

1 60

introduce conditional expectation is based on the Radon-Nikodym theorem. Then E(X I Y) is naturally defined for every integrable X. Hajek stated his projection lemma in [68] when proving the asymptotic normality of rank statistics under alternatives. Hoeffding [7 5] had already used it implicitly when proving the asymptotic normality of U -statistics. The "Ho­ effding" decomposition appears to have received its name (for instance in [15 1]) in honor of Hoeffding's 194 8 paper, but we have not been able to find it there. It is not always easy to compute a projection or its variance, and, if applied to a sequence of statistics, a projection may take the form L gn (Xi ) for a function gn depending on n even though a simpler approximation of the form L g (Xi ) with g fixed is possible.

PROBLEMS 1. Show that "projecting decreases second moment" : If S is the projection of T onto a linear space, 2 then ES :=:: ET 2 • If S contains the constants, then also var S :=:: varT.

2. Another idea of projection is based on minimizing variance instead of second moment. Show that var(T - S) is minimized over a linear space S by S if and only if cov( T - S , S) = 0 for every S E S .

3 . I f X :=::

Y almost surely, then E ( X I Z )

::::_ E ( Y I Z ) .

4 . For an arbitrary random variable X :=:: 0 (not necessarily square-integrable), define a conditional expectation E(X I

Y) by limM-+oo E(X 1\ M I Y).

(i) Show that this is well defined (the limit exists almost surely) . (ii) Show that this coincides with the earlier definition i f EX 2 < oo .

(iii) I f E X

<

oo show that E(X - E ( X I

(iv) Show that E ( X 1

Y) )g (Y)

= O for every bounded, measurable function g.

Y) is the almost surely unique measurable function of Y that satisfies the

orthogonality relationship of (iii) .

How would you define E(X I

Y)

for a random variable with E I X I

<

oo ?

5 . Show that a projection S o f a variable T onto a convex set S is almost surely unique. 6. Find the conditional expectation E(X I

Y) if (X, Y) possesses a bivariate normal distribution.

7. Find the conditional expectation E(X1 I X(n) ) if X 1 , . . . , Xn are a random sample of standard uniform variables. 8. Find the conditional expectation E(X 1 I X n ) if X 1 , . . . , Xn are i.i.d.

9. Show that for any random variables S and T (i) sd(S + T) :=:: sd S + sd T , and (ii) I sd S - sd T I :=:: sd(S - T ) .

10. I f Sn and Tn are arbitrary sequences o f random variables such that var(Sn - Tn ) /varTn then Tn - ETn

---

Moreover, varSn fvarTn

11. Show that PA h (Xj : Xi

---+ E

sd Tn

1 . Show this.

p ---+

0.

B ) = 0 for every set B that does not contain A .

---+

0,

12 U-Statistics

One-sample U -statistics can be regarded as generalizations of means. They are sums of dependent variables, but we show them to be asymptoti­ cally normal by the projection method. Certain interesting test statistics, such as the Wilcoxon statistics and Kendall 's T -statistic, are one-sample U -statistics. The Wilcoxon statistic for testing a difference in location between two samples is an example of a two-sample U-statistic. The Cramer-von Mises statistic is an example of a degenerate U-statistic.

12.1

One-Sample U-Statistics

Let X 1, . . . , X n be a random sample from an unknown distribution. Given a known function h, consider estimation of the "parameter"

In order to simplify the formulas, it is assumed throughout this section that the function h is permutation symmetric in its r arguments. (A given h could always be replaced by a symmetric one.) The statistic h ( X 1 , , X r ) is an unbiased estimator for 8 , but it is unnatural, as it uses only the first r observations. A U-statistic with kernel h remedies this; it is defined as 1 .

u= )

C

•

.

� h (Xf31 ' . . . ' Xf3J ,

where the sum is taken over the set of all unordered subsets f3 of r different integers chosen from { 1 , . . . , n}. Because the observations are i.i.d., U is an unbiased estimator for 8 also. Moreover, U is permutation symmetric in X 1 , . . . X n , and has smaller variance than h ( X 1 , . . . , X r ). In fact, if Xc l) , Xcnl denote the values X 1 , . . . , Xn stripped from their order (the order statistics in the case of real-valued variables), then •

.

.

.

Because a conditional expectation is a projection, and projecting decreases second mo­ ments, the variance of the U -statistic U is smaller than the variance of the naive estimator h (X 1 , . . . , X r ) .

161

1 62

U-Statisrks

In this section it is shown that the sequence J,i(U - 6) the condition lh3t Elr 2 (X. , . . . , X, ) < oo.

I is a mean , - • 'L?.11l (X;). The asserted asymptotic no nn al i ty is then just the central limit theorem. 0

1 2.1

Example. A

U-statistic of d eg ree r

is a sy m ptot i cal l y nonnal under

=

4<x•

Example. For the kernel h (.t� w X:! ) = - xz)2 of degree 2, the pammetcr 0 = Elz(X 1 , X2) = var X 1 is th e variance of the observations. The corres pon d ing U -statistic can be calc u lated to be 1 2.2

Thus, the sample variance

is a

U -s ta t i s ti c of order 2.

0

The asymptotic normality of a sequence of U -statistics, if 11 --.. oo '"' ith the kernel rem a i ni ng fixed, can be established by the projection method. The projection of U - 9 onto the set of all statistics of the form L:�... 1 .�; (X1) is given by·

where the function II 1

is

given by

h1 (:r) = Eh(x, X2

• . . .

, X,) - 0.

The first equ3Jity in the formula for {; is t he IUjck projection principle. The second equality

is established in the proof below. The sequence of projeCtions {; is a�ymptolically nomtal by the central limit thcorein, provided Elri(X 1 ) < oo. The difference between U - 0 and its projection is asymptotically negligible. Theorem.

/f Eir2(X � o . . . ,

.jii( U - .0

D) ..!'. 0.

Consequelll(\� the :requence ,Jii( U - 0) is asymptotically normal with mean 0 and \'llriance r2(', , where, with X l ' • • • ' Xr. XI ' . . . ' x; denoting i.i. cl. \"Clriab/es, 1 2.3

(I

=

Xr)

cov (h ( X l ,

< oo,

tlzetr

-

X2, . . . , X,). h(X 1 , x; . . . . . x;)).

\Ve first veri fy the fonnula for the projection U. It su ffi ce s to show that E(U () I X1) = h 1 (X1 ). By the independence of the observations and pcnnutation symmetry of Proof. Jr.

E (h (XtJ• • • • • • X,,, ) - 8 I Xi

=X

)

=

{

h l (:c) 0

if i e {J if i f. fJ.

To ca l c u l ate E(U - 0 1 X;). we take the average over all {3. Then the first case occurs for (� : :> of the vectors fJ in the definition of U. The factor rfn in the fonn u J a for the projection

(J arises as rIn = (� : l )/(�).

12. 1

163

Ollt'-Samplt' U-Sraristirs

The projection (J has mean zero, and \'ariancc equal to .,

"

var U

r·

=

.,

- Ehj ( X I ) II

rl

= -

n

J

E{lt(x. X2

• • • • •

X,) - 9)Eir (x.

X2

r2

• . . . •

x: ) dPx. (x) = -�� · n

finite, the sequence .Jii D converges weakl y to the N(O. r2�1 )-distribution limit theorem. By Theorem 1 1 .2 and Slutsky's lemma, the sequence �(U 0 - U) converges in probability to zero, provided \'ar Ufvar 0 -+ I . In view of the permutation symmetry of the keme I h. an expression of the type CO\' (Ir (Xp1 • • • • • X� ), h(Xf.l; • • • • , Xf.!; )) depends· only on the number of vari abl es X, that arc common to Xp1 , • • • Xp, and X11; • • • • , Xp; . Let ('(" be this covariance if c variables are in common. Then Because this

is

by the central

,

va r U

=

=

(; )

-2

L L cov (lr ( XJf,

• . . .

, Xp, ), /r ( X11j •

(;t�b (�) (; =�) �,.

• • • Xf.l; )) •

The last step follows, because a pair ({3. /J') with c indexes in common can be chosen by first choosing the r indexes in {J. next the c common indexes from {J. and finally the re maining r - c indexes in /J' from { I • . . . • n } f1. Th e expression can be simpl ified to

-

� var U = � r•l

r !2

,

c ! (r - c) !..

(n - r)(11 - r - I )

· ·

· (n -

2r + c + I )

n (n - I ) · · · (II - r + I }

,

(

.

.

sum the first term is 0 ( 1 /n), the second term is 0 ( 1 /11 2 ), and so forth. Be­ cause n t i mes the first tem1 converges to r2('1 • the desired limit result \'ar U f\'ar D � In this

follows.

•

Example (Sig11cd rank statistic). The parameter 0 = P(X 1 + X2 > to the kemel lz (:r1 • x2) = I {x1 + x1 > 0} . 1lte co rrespo nd i n g U-statistic is 1 2 .4

u

-( ) L L l {X, I

=

"

2

I <J

+

Xj

>

0) corresponds

0).

TI1is statistic is the average number of pairs (X, , X1 ) with positive sum X; + X1 > 0. and can be used as a test statistic for invcstig•tting whether the distribution of the observations is located at zero. If m any pairs (X, . X1 ) y ie l d a positive sum (relative to the total number of pairs), then we have an indication that the distribution is centered to the right of zero. The sequence .jii( U - 0) is asymptoticalJy nomtal with mean zero and variance 4{1 • I f F denotes the cumulative distribution function of the observations. then the projection of U - 0 can be written

TI1is formula is useful in subsequent discussion and is also convenient to express the asymp­ totic variance in F.

1 64

U-Statistics

The statistic is particularly useful for testing the null hypothesis that the underlying distribution function is continuous and symmetric about zero: (x) x) for every Under this hypothesis the parameter e equals e and the asymptotic variance reduces to 4 var because is uniformly distributed. Thus, under the null hypothesis of continuity and symmetry, the limit distribution of the sequence Jn ( U is normal N (O, independent of the underlying distribution. The last property means that the sequence is asymptotically distribution free under the null hypothesis of symmetry and makes it easy to set critical values. The test that rejects H0 if ,J3,i(U :=:: Za is asymptotically of level for every in the null hypothesis. This test is asymptotically equivalent to the signed rank test of Wilcoxon. Let ..., K; denote the ranks of the absolute values of the observations: k means that is the kth smallest in the sample of absolute values. More precisely, Then Suppose that there are no pairs of tied observations :S: the signed rank statistic is defined as 0} . Some algebra shows that

X.

F(X 1 ) = 1/3, 1/3), Un

= 1/2,

F(Xd

F = 1 - F (-

- 1 /2)

1/2) Rt, Rt = Rt = X; = X1.

F

a

l XII , I Xn I •

I X; I L�= I 1{1X1 I I X; 1} .

.

.

.

w+ = L 7= I Rt1{X; > w+ = (;) � 1{X; > O}. u+

The second term on the right is of much lower order than the first and hence it follows that ""' N(O, D

n -312 (W+ - EW+ )

1/12) .

Example (Kendall's r ).

The U -statistic theorem requires that the observations X I, . . . , Xn are independent, but they need not be real-valued. In this example the observations are a sample of bivariate vectors, for convenience (somewhat abusing notation) written as (XI, YI), . . . , (Xn, Yn) · Kendall 's T -statistic is 4 r= I: I: l {(Y1 - Y;)(XJ - X;) > 0 } n (n This statistic is a measure of dependence between X and Y and counts the number of concordant pairs (X;, Yi ) and (X 1, Y1) in the observations. Two pairs are concordant if the indicator in the definition of r is equal to 1. Large values indicate positive dependence (or concordance), whereas small values indicate negative dependence. Under independence of X and Y and continuity of their distributions, the distribution of r is centered about zero, and in the extreme cases that all or none of the pairs are concordant r is identically 1 or - 1,Therespectively. statistic r 1 is a U -statistic of order 2 for the kernel 12.5

- l)

l <]

- 1.

+

1 - 2P((Y2 - YI)(X2 - XI) > F1(x, = P(X

Hence the sequence Jn(r + o)) is asymptotically normal 4 with mean zero and variance si · With the notation y) < x, < y) and p r (x , y) y y), the projection of u - e takes the form

= P(X > X, >

Y

X and Y are independent and have continuous marginal distribution functions, then Er = 0 and the asymptotic variance 4si can be calculated to be 4/ 9, independent of the If

12.2

Two-Sample U-statistics

1 65

y

()

concordant X

Figure 12.1. Concordant and discordant pairs of points .

marginal distributions. Then ylrir -v--7 N (0, 4/9) which leads to the test for "independence": Reject independence if .fti174 1 r I > Za/2 · D 12.2

Two-Sample U-statistics

Suppose the observations consist of two independent samples X 1 , . . . , Xm and Y1 , , Yn , i.i.d. within each sample, from possibly different distributions. Let h (x 1 , . . . , Xr , y 1 , . . . , Ys ) be a known function that is permutation symmetric in x 1 , . . . , Xr and Y l , . . . , Ys separately. A two-sample U -statistic with kernel h has the form .

•

.

where and f3 range over the collections of all subsets of r different elements from {1, , m } and of s different elements from { 1 , . . . , n } , respectively. Clearly, U is an unbiased estimator of the parameter a

2, . . .

2,

The sequence Um, n can be shown to be asymptotically normal by the same arguments as for one-sample U -statistics. Here we let both m ---+ oo and n ---+ oo, in such a way that the number of Xi and Y1 are of the same order. Specifically, if N = m + n is the total number of observations we assume that, as m, n ---+ oo,

m

n

0 < A < l. - ---+ A ' - ---+ 1 - A ' N N To give an exact meaning to m , n ---+ oo, we may think of m = mv and n = n v indexed by a third index v E N. Next, we let mv ---+ oo and nv ---+ oo as v ---+ oo in such a way that mv/ Nv A. The projection of U -e onto the set of all functions of the form L�=l ki (Xi ) + L: ) = 1 l1 ( Y1 ) is given by ---+

1 66

U.Staristics

where the fu nctions 11 1 •0 and h0• 1 h � ,o(.t)

=

ho. J (y)

=

are

defined

by

Elt (x, X2 . . . . , X, , Y� t . . . . Y" ) - 0, Eh (X� . , X,, y, Y2· · · · · Y, ) - 0. • • •

This follows, as before, by first ap p ly i ng the Hajek projection lem m a , and next expressing

E( U I X; ) and E( U I Yj ) in the kernel fu nction. If the kernel is square-integrable, then the sequence {J is asymp tot i caJJy nonnal by the centml Ji mit theorem. The di fference between {J a nd U - 0 is asymptoticaJJy neg l ig i b le . 1 2.6 Theorem. I/ Eh2(X1 , • • • , X, , Y1 • • • • , Y.. ) < oo. then the sequence Jil(U 8 0 ) com·erges in probability· to zero. · consequellll)� the sequence Jil(U - 0) £·om·erges in di.t:tribution to the non1wl /aw with meall zero mu/ J.•ariance r2��,0/). + s2("o. d( l - .A.). ,....here. with the X; being i. i.d. l'ariable:r illdepelldent of the i.i.d. l'llritlbles r,. -

�'".J

=

cov (hCX� t

. • •

-

, X, Y� t • • • , Y.. ) .

h ( X. , • . • • Xr ,

x;+l • . . . . x; . r. . . . . . YJ, Y�+ . . . . . . r;>).

Proof. The argu m e nt is sim i l ar to the one given previously for one-sample U - st at i st ic s. The variances of U and it s proje cti o n are given by

var U

=

(;);(:)2 t.t. (;) (�) ('; ::;) (�) W (;=�) �··.d·

It can be checked fro m this that both the sequence Nvar D and the sequence Nvar U co nverge to the number r 2 �1 •0!). + s2�o. J /( I - l). • 12.7

Example (t.famr- U1titllej' statistic). Th e kcmcJ for the

lt (x, y) = I ( X ::;

parameter 0 = P(X ::; Y) is Y }, which is of order I i n both x and )'. The correspondi ng U - s tat i M i c is

The statistic mn U is known as the },famr- \\'ltimey statistic a nd is used to test for a di fference

i n location between the two sa mp l e s . A large value indicates that the Y1 arc '"stochastically larger" than the X, . If the X1 and r1 have c u m ul ati ve distribution functions F a nd G. resp ec t i vel y, then the p rojec t i on of U - 0 ca n be written

It is easy to obrain the limir distribur ion of the projections iJ (and hence of U) from this for­ mula. In particular. under the null h ypothesis that the poole d sam p le X 1 • • • • • Xm , r• . . . . . Y, is i.i.d. with conti nuous dislribu tion fu nction F = G. the sequence Jl2imz/N(U - 1 /2)

12.3

Dt!gmt!mtt" U-Swrisric.f

1 67

converges to a standard nonnal distribution. (The pammetcr equals 0 = 1 /2 and {0• 1 = {J.o = 1 / 1 2.) If no observations in the pooled sample arc tied. then mnU + 4n
* 1 2.3

Degenerate U-Statistics

A

sequence of U-statistics (or, better, their kernel fu nction) is cal led degenerate if the aliymptotic va ri a nce r2{1 (fou nd in Theorem 1 2.3) is zero. The fonnula for the variance of a U -statistic (in the proof of Theorem 1 2.3) shows that var U is of the order ,- r if {1 = · · · = �r- l = 0 < {c· In this case. the sequence n''f2(U11 - 0) is asymptotically tight. In this section we derive its limit distribution. Consider the Hoeffding decomposition as discussed in section 1 1 .4. For a U -Matistic U11 with kernel It of order based on observations X 1 X,• • this can be simpl ified to

r,

U,.

'

=

•

1

L L (" L PA h ( Xtfp I A I=(' )

c=O

IIere, for each 0 !::

r

c

!::

Jf

•

•

•

•

Xifr ) =

•

• • •

L '

(r) £'

U, , r

(say).

c·=O

r. the variable U,,,. is a U -statistic" of order c with kernel h r(X • ·

. • • •

X(' )

=

Pn .. ... (·th( X 1 , .

• • •

X, ).

To sec this� fix a se t A with c clements. Because the space HA is orthogonal to all functions g(Xj : j e B) (i.e the space LccB lie) for every set B that docs not contai n A, the projection PA/z(Xp1 Xp. ) is zero unless A c fJ = (fJ1 fJ, }. For the remaining p the projection PA h ( Xp 1 • • • • • XI'. ) docs not depend on jJ (i.e on the r - c clements of jJ - A) and is a fixed fu nction It,. of ( X1 : j e A). This fol lows by symmetry. or expl icitly from the fommla for the projections in section 1 1 .4. The function It(" is indeed the fu nction as given previously. There arc (� : �) vectors {J that contain the set A. The claim th at U,• . r is a U-statistic with kernel hr: now follows by simple algebra. using the fact that (� : �-)/(�.><:> = 1 /(�). By the defining properties of the space 1/f l.�c·J· it follows that the kemel /r� is degenerate for c � 2. In fact. it is strongly degenerate in the sense that the conditional expectation of hc(X1 X�) given any strict su bset of the variabks X 1 • • • • • X(' is zero. In other words. the integral J h(:c. X2 . , Xc> d P(:c) with respect to any single argument vanishes. By the same reasoni ng. U,,,. is u ncorrclatcd with every measurable fu nction that depends on strictly fewer than ,. elements of X I X11 • V.'e shall show that the sequence n''/2 Un.(' converges in d istribution to a l i mit with variance c! FJz;(x 1 • • • • • X(') for every c � I . Then it follows that the sequence n('12 (U,, - 0) converges i n distribution for c equal to the smallest value such that /r(' ¢ 0. For c � 2 the limit distribution is not nonnal but is known as Gaussian chaos. Because the idea is simple, bu t th� statement of the theor�m (apparently) necessarily compl icated, first consider a special case: c = 3 and a "'product kernel .. of the. form .•

•

• • • •

•

• • • •

.•

•

•

•

•

•

•

• •

•

• • • •

168

U-Statistics

A U -statistic corresponding to a product kernel can be rewritten as a polynomial in sums of the observations. For ease of notation, let Pn i = n - 1 L.7= 1 i(Xi ) (the empirical mea­ sure), and let Gn i = Jll Wn - ) i (the empirical process), for the distribution of the observations X 1 , . . . , X n . If the kernel h 3 is strongly degenerate, then each function ii has mean zero and hence Gn fi = JllPn ii for every i . Then, with (i 1 , i2, i 3 ) ranging over all triplets of three different integers from { 1 , . . . , n } (taking position into account),

P

P

Pi

By the law of large numbers, Pn � almost surely for every i, while, by the central limit theorem, the marginal distributions of the stochastic processes i 1---+ Gn i converge weakly to multivariate Gaussian laws. If G i : i E L 2 (X, A, } denotes a Gaussian process with mean zero and covariance function EGiGg = (a ?-Brownian bridge process), then Gn G. Consequently,

{

P) Pig - Pi Pg

-v-+

3

The limit is a polynomial of order in the Gaussian vector (GJI , Gf2, Gf3) . There is no similarly simple formula for the limit of a general sequence of degenerate U­ statistics. However, any kernel can be written as an infinite linear combination of product kernels. Because a U -statistic is linear in its kernel, the limit of a general sequence of degenerate U -statistics is a linear combination of limits of the previous type. To carry through this program, it is convenient to employ a decomposition of a given kernel in terms of an orthonormal basis of product kernels. This is always possible. We assume that L 2 (X, A, is separable, so that it has a countable basis.

P)

12.8

P),

Example (General kernel).

If 1

=

io , JI , f2, . . . is an orthonormal basis of L 2 (X, ike with (k 1 , . . , kc) ranging over the nonnegative c c c

x A, then the functions ik 1 x integers form an orthonormal basis of L 2 (X , A , p ). Any square-integrable kernel can be written in the form hc (X 1 , . . . , Xc) = L a (k 1 , . . . , kc ) fk 1 X x ike for a(k 1 , . . . , kc) = (h e , ik 1 x x ike ) the inner products of h e with the basis functions. D ·

·

·

.

·

·

·

·

·

·

In the case that c = 2, there is a choice that is spe­ cially adapted to our purposes. Because the kernel h 2 is symmetric and square-integrable by assumption, the integral operator K : L 2 (X, A, 1---+ L 2 (X, A, defined by Ki (x) = J h 2 (x , y) i (y) d ( y ) is self-adjoint and Hilbert-Schmidt. Therefore, it has at most count­ ably many eigenvalues A o , A 1 , , satisfying L_ A� and there exists an orthonormal basis of eigenfunctions io , i1 , . . . . (See, for instance, Theorem VI. 1 6 in [ 1 24].) The kernel h 2 can be expressed relatively to this basis as 12.9

Example (Second-order kernel).

P

P)

P)

.

•

< oo,

.

h 2 (x , y)

=

00

L A k ik (x) fk (y) . k =O

For a degenerate kernel h 2 the function 1 is an eigenfunction for the eigenvalue 0, and we can take io = 1 without loss of generality.

12.3

169

Degenerate U-Statistics

The gain over the decomposition in the general case is that only product functions of the type f x f are needed. D The (nonnormalized) Hermite polynomial H1 is a polynomial of degree j with leading coefficient xi such that J Hi (x) H1 (x) ¢ (x) dx = 0 whenever i 1- j . The Hermite polyno­ mials of lowest degrees are H0 = 1 , H1 (x) = x , H2 (x) = x 2 - 1 and H3 (x ) = x 3 - 3x . Let he : xe 1--+ 1Ft be a permutation-symmetric, measurable function of c arguments such that Eh� (X 1 , . . . , Xe) < oo and Ehe (X l , . . . , Xe- 1 , X c ) 0. Let 1 = fo , fi , h , . . . be an orthonormal basis of L2 (X, A., P). Then the sequence of U­ statistics Un,e with kernel h e based on n observations from P satisfies d (k ) X he, X ik f n e/ 2 Un , e kt . . . ( J n Hai (k) (!Gl/fi (k)) . i= l 12.10

Theorem.

=

-v'-7

Here !G is a P -Brownian bridge process, the functions 1/f 1 (k) , . . . , lfd (k ) (k) are the different elements in fk l ' . . . , ike' and ai (k) is number of times lfi (k) occurs among fk 1 , , fke · The variance of the limit variable is equal to c ! Eh� (X 1 , . . . , Xe). •

•

•

The function he can be represented in L2(Xc , A.e , P e ) as the series L k ( he , fk 1 x . . . , fkJ fh x · · · x ike . By the degeneracy of he the sum can be restricted to k = (k 1 , . . . , ke) with every k1 ::::_ 1 . If Un , c h denotes the U-statistic with kernel h (x 1 , . . . , Xe) , then, for a pair of degenerate kernels h and g,

Proof.

cov(Un ' eh, Un ' c g)

=

c! P c hg. n (n - 1) · · · (n - c + 1)

This means that the map h 1--+ n e f2 �cf Un,eh is close to being an isometry between L 2 (P c ) and L 2 (P n ) . Consequently, the series Lk ( he, fk 1 x · · · x fk) Un,efk1 x · · · x fke con­ verges in L 2 (P n ) and equals Un, c h c = Un,e · Furthermore, if it can be shown that the finite-dimensional distributions of the sequence of processes { Un,efk1 x · · · x fke : k E r�:n converge weakly to the corresponding finite-dimensional distributions of the process { n:�k{ Hai (k) (!Gl/fi (k)) : k E then the partial sums of the series converge, and the proof can be concluded by approximation arguments. There exists a polynomial Pn,e of degree c, with random coefficients, such that

P�F},

(See the example for c = 3 and problem 12. 1 3). The only term of degree c in this polynomial is equal to !Gn fk1 !Gn fkz · · · !Gn fke . The coefficients of the polynomials Pn,e converge in probability to constants. Conclude that the sequence n e 2 c ! Un,e fk1 x · · · x ike converges in distribution to Pc (!G fkp . . . , !G fkJ for a polynomial Pc of degree c with leading term, and only term of degree c, equal to !G fk1 !G fk2 !G ike . This convergence is simultaneous in sets of finitely many k. It suffices to establish the representation of this limit in terms of Hermite polynomials. This could be achieved directly by algebraic and combinatorial arguments, but then the occurrence of the Hermite polynomials would remain somewhat mysterious. Alternatively,

1

•

•

•

U-Sratb:rics

1 70

the representation can be derived from the defi n ition of the Hermite polyno m ial s and co­ variance calculations. By the degeneracy of the kerne l J,.. , x x fi:. . the U stati stic U,,(·/t, x · · · x /t, is orthogonal to all me as u rable fu nc t i o ns of c.· - I or fewer dements of X I • • X,. . and t he i r linear combinations. Th i s includes the functions n; (Gn.'.'r ) for a rbh ra ry fun ct ions g; a n d nonn egati ve i nt egers a; w it h L: a, < c. Tak ing l i m i t s. we con­ c l u de that PcCGJ,., , . Gft.. ) mu s t be orth ogon a l to every polyno m i al in Gjj, 1 • • • • Gfs.. of degree Jess than c - I . By th e orthonormal ity of the basis f, , the variables G f, are i ndepe nde n t s£andard n orm a l variables. Because the Hermite polynomials fonn a b�1sis for the po lyn om i al s in one variable, their ( tensor) prod ucl" fo rm a bas is fo r the polynomials o f more than one argument. The polyno m i al P,. can be wri tten as a J incar combination of c l e m e n ts from this bas is. By the orthogonality. t he cocfficienrs of base clements of d eg ree < ( vanish. From the base elcmenl" of deg ree c.· only the product as in the t heorem can occ ur, as follows from considerat ion of the lead i ng tenn of P.· . • •

·

•

..

"'

0 0 .

o • •

0

'

1 2. 1 1 Example. For c = 2 and a basis I hz, we obtain a l i mi t of the form L,,.
s ta nd ard nonnal variables.

1 2. 1 2

= x

(Zf -

0

/o. /1 • • • • of eigenfunctions of the kernel fd lll.(GJJ J . By the orth o normali ty of t h e I ) for Z 1 • Z:z • • • . a sequ e nce of independent

Example (Samp le �·aria11ce). The kernel h (x 1 • x2 )

=

� (x1 -x2 ) 2 yields the samp le

v ari ance s,; . Because this has asymptot i c vari anc e 11� - Jl � (sec Examp l e 3.2). t he kernel is dege ne ra te if and on ly if JA� = Jt3. Th i s can happen onl y if ( X 1 - a1 )2 is con st a n t . for i:r t = EX, . If we center the observations. so th at 0'1 = 0. then t hi s means that X 1 only t3kes the va lu es -u and a = .j'iG. each with probability J /2. Th i s is a very degenerate si tuat ion. and ir is easy to find t h e limit distribution directly. but perhaps it is i nstructive to apply the ge ne ral theorem. The kemels "� take the forms (Sec s ect io n I J .4).

flo =

E! < X t - X2) � = a2•

ll . c.r.> = E! <x1 - X2)2 - u :! .

h1(x, , .\'.2)

=

! (xr - X;! )2 - E� (x, - X:!)2 - E� ( X 1 - X:! ):! + a 2 •

The ke mel is degenerate if h 1

lt�(x 1 . x2) =

! Cx1 - x2 ) 2. -

=

u '!..

0 a l m os t �urcJy,

an

d

then th� seco nd-orde r kernel is

B L-ca u sc the u n derlying di�tribution has on ly two sup­

o , the eigenfunctions f of the corresponding in tegral operator ca n be iden ti fied w ith vectors (/(-a ). f
degeneracy the kernel allows the decomposition

\Ve can conclude rhar lh� sequence

1 2. 1 3

F..xamp le

n(s,; - Jl � ) con ve rge s in dis t ribution to -a2(Zf - I ).

(Cramir-l·on JUises).

distribution function of a random sa mple

0

Let F, (.r ) , - r L::'= 1 I { .\', � .d be the e mp i.r i ca l X 1 , • • • , X,, of r�;•l-valued nrndom \'ariahlcs. The =

Cmmer-\�m /.-fi.\'t!S statistic for tes t i n g the (nu l l ) hypothes i s t ha t t h e unde rly i n g cumu lative

Probl�m.\' distribution is a given

171

function F is given by

The double sum restricted to the off-diagonal terms i s a U-stat istic. with. under llo. a degenerate kernel. Thus, this s ta t i st i c converges to a nondcgcnc mte li mit distribution. The di agon a l tenns contribute the constant J F( I - F) d F to the limit distribution. by the law of large numbers. If F is uni form. then the kernel of the U-.srati�tic is

h(x. y)

=

�x2 + !Y:! - x v y + . �·

To find the e ige nva l u e s of the corresponding integ ral operator K . we differentiate the identity Kf

=

l�f twice, to find the eq uation .A..f"+ f

=

f f(s) ds.

Because the kerne l is degenerate,

the constants arc eigenfunctions for the eigenvalue 0. The eigenfunctions corresponding to nonzero eigenvalues arc ort hogo nal to this e i ge nspace . whence J f(s ) cis = 0. The eq u a t i on >.f '' + f = 0 has solutions cos a.r und sin ax for a:! = ).- I . Rei nsert ing these in the original equ�ttion or uti lizing the relation J f(s) ds = 0, we find that the nonzero eigenvalues are equal to· j -2rr-2 for j e N, with eigenfunctions J2 cos rrj.t . Tims. 1he Cramer-Von �1ises s tat i s t i c converges in distribut ion to 1 /6 + I ). For an othe r derivation of the li mit distribution. sec Chapter I 9. D

E7=1 j-'!rr-2
Notes The main part of this chapter has its roots in 1he paper by 1-locffding [76] . Because the asymp­ t oti c variance is s ma l le r than the true vari ance of a U -stati stic. l loeffd ing recommends to apply a s ta nd ard nonnal approxi mation to (U - EU)/ sd U . Degene rate U-stati stics were con s i de red . among others. in [ 1 3 1 ) within the context of more general linear combina­ tions of symmetric kemds. A rco ne s and Gi ne (2 1 have s t u d i ed the we<1k c o nve rgence of ··u-processes··. stochastic processes i ndexed by classes of kernels. in spaces of bou nded functions as discussed i n Cha pte r 1 8. PROHLE!\·IS

1 . De: rive the asy mptotic d i M ri b ut ion of Giui'.v mt'atl tlij}""t' ft'IICt'. w h ich is deli ned as <2>- 1

IX, - X1 1.

L L; < 1

2. De rive the proj ec t i on of the sample variance.

3. Find

.a. Find

a kcrnd for the parametcr O a

=

E( X - EX) :l .

kernel for t h e p;u··o.� mclcr 0 = CU\'(X, }' ). Show lhat the corresponding U-stat isric is the .L:;'= 1 ( X� - X ) ( Y; - Y)/(u - I ).

sample covariance

=(2)- 1 L Li ..:J f Y1 - Y, HX1 - X, ). Let U, 1 and fl,2 be U -stat ist ics with ke rne l s II 1 and /1 :! · rc:-.p.:ctivc ly. Derive the joint a symptoti c

5. Find the limil di �tribution oi' U

6.

distribution of (U, I • Un�J.

7. Suppose EX r

Derive the asymptotic d is t ri bu ti on of the seq ue nce , - I L E,�1 X; Xj . Can y ou g i ve a two l i ne proof wi tho u t using the U-:\tat istk the o re m '! What happens i f EX 1 = 0'! <

oo .

8. (Mann's h.-st agaht!it trend.) To lc:st the: null h ypoth c:s i s that a sample X I • the alternative hypothes i s th;lt the di �tri bu t ions uf lhc

• • • •

X, is i.i.d. a�ain!.l

X; arc �roch;•srically increasi ng in i. �1ann

1 72

U-Statistics suggested to reject the null hypothesis if the number of pairs (Xi , Xi ) with i

is large. How can we choose the critical value for large n ?

<

j and Xi

<

Xi

9. Show that the U -statistic U with kernel 1 {x1 + x2 > 0} , the signed rank statistic w + , and the positive-sign statistic S = = 1 1 {Xi > 0} are related by w + = G)U + S in the case that there are no tied observations. 10. A V -statistic of order 2 is of the form n - 2 2::: = 1 :L'f = 1 h (Xi , Xi ) where h (x , y) is symmetric in x and y . Assume that Eh 2 (X1 , X 1 ) < oo and Eh 2 (X1 , X 2 ) < oo . Obtain the asymptotic distribution of a V -statistic from the corresponding result for a U -statistic.

2:::7

7

1 1 . Define a V -statistic of general order r and give conditions for its asymptotic normality.

12. Derive the asymptotic distribution of n ( s; - t.Jd in the case that M4 = f.J.,� by using the delta­ method (see Example 1 2 . 1 2). Does it make a difference whether we divide by n or n - 1 ?

13. For any (n

x

c) matrix aij we have

L ai , i

,

1

·

·

·

ai , c c

=

n

L fl ( - 1) I B I l ( B I - 1 ) ! L fl aij . -

i=1 }EB

B B EB

Here the sum o n the left ranges over all ordered subsets ( i 1 , . . . , i c ) of different integers from { 1 , . . . , n } and the first sum on the right ranges over all partitions B of { 1 , . . . , c} into nonempty sets (see Example [ 1 3 1]).

14. Given a sequence of i.i.d. random variables X 1 , X 2 , . . . , let An be the a -field generated by all functions of (X 1 , X 2 , . . . ) that are symmetric in their first n arguments . Prove that a sequence Un of U -statistics with a fixed kernel h of order r is a reverse martingale (for n :=:: r) with respect to the filtration Ar ::) Ar + 1 ::) · · · .

15. (Strong law.) If E j h (X 1 , · · · , Xr) J

< oo,

then the sequence

converges almost surely to Eh (X 1 , · · · , Xr ) . (For r

>

Un

of

U -statistics with kernel h

1 the condition is not necessary, but a

simple necessary and sufficient condition appears to be unknown.) Prove this. (Use the preceding

problem, the martingale convergence theorem, and the Hewitt-Savage 0- 1 law.)

13 Rank, Sign, and Permutation Statistics

Statistics that depend on the observations only through their ranks can be used to test hypotheses on departuresfrom the null hypothesis that the ob­ servations are identically distributed. Such rank statistics are attractive, because they are distribution-free under the null hypothesis and need not be less efficient, asymptotically. In the case of a sample from a symmetric distribution, statistics based on the ranks of the absolute values and the signs of the observations have a similar property. Rank statistics are a special example ofpermutation statistics.

13.1

Rank Statistics

The order statistics XN(1) ::::; XNc2J ::::; · · • ::::; XN (N) of a set of real-valued observations X 1 , , X N i th order statistic are the values of the observations positioned in increasing order. The rank RNi of Xi among X1 , . . . , XN is its position number in the order statistics. More precisely, if X 1 , . . . , XN are all different, then R Ni i s defined by the equation .

•

.

If Xi is tied with some other observations, this definition is invalid. Then the rank RNi is defined as the average of all indices j such that Xi = XN(j) (sometimes called the midrank), or alternatively as L:: 7= 1 {X1 ::::; Xd (which is something like an up rank). In this section it is assumed that the random variables X 1 , . . . , XN have continuous distribution functions, so that ties in the observations occur with probability zero. We shall neglect the latter null set. The ranks and order statistics are written with double subscripts, because N varies and we shall consider order statistics of samples of different sizes. The vectors of order statistics and ranks are abbreviated to X No and RN , respectively. A rank statistic is any function of the ranks. A linear rank statistic is a rank statistic of the special form L : 1 aN (i, RNi ) for a given (N X N) matrix (aN ( i, n) . In this chapter we are be concerned with the subclass of simple linear rank statistics, which take the form N L CN i aN, RN, . i=1 Here ( cN1 , . . . , CN N ) and (aN1 , . . . , aNN) are given vectors in ffi.N and are called the coeffi­ cients and scores, respectively. The class of simple linear rank statistics is sufficiently large

1

173

1 74

Rank. Sign, and P�rmmmilm Srati.ui,·.fl

to contai n i nte res tin g statistics for test ing a variety of hypotheses.

part icula r,

In

\ve s hal l

sec that it contains all .. locally most powerful" rank stat istics, whic h i n a not h e r c h a pt e r are

shown to be asy mptot ically efficient wi thi n the class of all tests.

Some e le m ent ary propcnics of ranks and order statistics arc ga t here d in the following

lemma.

13.1 /,.emma. Ut X I X N be a random sample ftvm a continuous distribmionfimc­ rion F with density f. Then (i) the vectors XNc , and RN are independelll: < .rN : (ii) rhe vector X N o has density N ! nf:: 1 f(x; ) o11 the set x1 < •

• • • •

·

·

·

N I (iii) rite \'ariable XNCo has densir.v NC�:t') F(x)1- 1 {I - F<x>) - f(x): for F the wu'· form diJtriburlon 011 [0. 1 ). it has mean ij ( N + I ) and \'ariance i (N - i + 1 )/ (CN + 1 )1 (N + 2 )): (i\•) the \'telor R,.. is uniformly distributed on the set ofall N! pt•rmutarion'i of I. 2 . . . . •

N:

(l') for any statiJtic T aizd pennutation r E(T(Xr

•

.

.

.

= (r1 ,

•

• • •

r"' ) of I , 2

, X.v ) l R.v = r) = ET( X.,•cra J ·

(�·i) for em}" simple linear rank .fla tistit: T

=

•

• • • •

• • .

, N.

X,\'fr"· � ) :

Ei� 1 cN;a.v.R.,

.• •

S ta te m ents (i) through { iv) are wel l-known and e l e me nta ry. For the proof of ( v). it is he lpful to write T( X1 XN) as a function of the ranks an d the order statistics. Nex t. we app ly (i), For the proof of statement (vi). we u se that the d i st ributions of the vari ab l es R,w and the \'ectors CRt�; . RN, ) for i -I= j arc u niform on th e sets I = ( I N ) and (i. j) e 1 2 : i i: j respectively. Furthermore. a doubl e sum of the form Li'FI (b, - b) (bi - h) is

Proof.

•

• • • •

• . . . •

J.

equal to -

'L, (b; - b)'·.

{

•

It follows that rank st at i st ic s arc di.�rribmion-free over the set of all models in which the

observations arc indepe ndent and identica l ly distributed . On the one hand. this makes them

�tatistically useless in si tuati o n s i n which the observations are. indee d a random s a mp le .

from some di stribution. On the othe r hand, it makes them of great int ere st to detect cena in

differences in distribution between the obse rva tion s . such as i n the two-sample pro b l em .

If

the null hypothesi s is taken to assert that the observations are i de nt ical l y distributed. then

the critical values for a rank test can be chose n in such a way that the p robabi lity of an

error of the f i rst k i nd is eq ua l to a g iven Jc\'cl a. for any probabi lity distri bution in the nul l hypothes i s. Somewhat s u rprisin g ly. this gain is not necess aril y counteracted by a lo�s i n

asymptot ic effi c ie ncy a s w e sec in Ch a pt e r .

13.2

1 4.

Example (Two·sample location problem). Suppose that the total set of observ at io n s

consists of two i ndependent random samples. inconsistently with the precedi ng notat ion

X1 X the pooletl sample X 1 written as

•

•

•

•

•

m

and Y

• • • •

1

•

•

• • •

, X,. . Y� o

Y" . Set N

. . . •

Yn .

=

m +n

and

let RN rn:

the rank vec to r of

I 3. 1

Rank Stari.titic.,·

1 75

\Ve arc interested in resting t he null hypothesis that the two samples arc identically dis­ tributed (according to a continuous distribut ion) ag ai n st the alternative that the distribution of the second sample is stochastically larger than the distribution of the fir�t sample. Even wit hour a more precise description of the alternat ive hypothesis. we can discuss a tion of usefu l rank stat istics. I f the Y1 arc a sample from a stochastically larger distribu­ tion, then the ranks of the Y1 in the pooled sample should be rclati\'ely large. Thus. any measure of rhe size of the ranks RN.m + J · • • • • RN .v can be used as a test statistic. It will be distribution-free under the null hypothesis. TI1e most popu lar choice in this prob l e m is the Wilcoxon .'ilatistic

collec­

N

W =

L

i=lll +- I

RNr ·

This i s a si mple linear rank stati�tic with coe flicients c = (0, . . . , 0, I , . . . , I }, and scores a = (I, , N ) . The null hypot he s is is rej ected for large values of the \Vi lcoxon �tatiMic. (The \Vi lcoxon statistic is equi \ a l e nt to the J-fmm- \Vhit11ey stati.vtic U = Lr. J I (X1 � Y1 } in that W = U + !n (ll + 1 ). ) There arc many mher reasonable choices of rank stat istics. some o f which are o f special interest and h ave names. For instance, the ''an dt•r U�1erden stati.\'lic is defi ned as . • •

'

,\'

L

r .=m + I

- 1 ( RN, ).

Here - 1 is the standard normal quanti lc function. We s ha ll sec ahead that this statistic is part icul arl y attract ive if it is bel ieved that the underlying distribution of the obse rvat i on s is approxi mately normal. A ge n era l method to ge n cn1 te useful mnk statistics is discussed below. 0 A cri t i ca l value for a rest based on a (distribution-free) rnnk statistic can be found by simply tabulating its null distri bution. For a l arge number of observations this is a bit ted ious. In most cases it is a so unnecessary. bc.."Causc there ex ist accurate asymptotic approx imations. The remainder of this section i s concerned with proving asymptot ic normality of si mple l i near r.mk statistics under the null hypothesi�. Apart from bei ng useful for fi nd ing crit ical values. the theorem is used su bsequ e ntly to �t udy the &�symptotic efliciency of rank tests. Consider a rank statistic of the fonn T.v = L� 1 c,.,, a.v.N., ,· For a sequence of this type to be asy mptot i ca ll y normal, some restrictions on the coeflicicnts c and scores a arc necessary. I n most cases of interest. the scores arc ge nerated .. through a g i ve n fu nction t/J : [0, I J � R in one of two ways. Either

l

"

where UNC J i o UNcN• distribution on (0. I ) ; or • • • •

arc

the order stat istics of

a

sample

"N• = ¢(N� l }

( 1 3.3) of size N from the ·unifonn

( 13.4 )

For \Vei l -behaved function� c/J. these defi nit ions ure closely related and al most ident ica l. because i I ( N + 1 ) = EUN c 1 • · Scores of the first type correspond to the locally most

1 76

Rank, Sign, and Permutation Statistics

powerful rank tests that are discussed ahead; scores of the second type are attractive in view of their simplicity. 13.5 Theorem. Let R N be the rank vector of an i. i.d. sample X 1 , . . . , XN from the continuous distribution function F. Let the scores aN be generated according to ( 13.3) for 1 a measurable function ¢ that is not constant almost everywhere, and satisfies J0 ¢ 2 (u) du < oo. Define the variables N N t N TN = a (cNi - cN )¢ (F(X i ) ) . = + N CNi N RN cNZiN i =l i =l

L

,

, '

L

Then the sequences TN and TN are asymptotically equivalent in the sense that ETN = ETN and var (TN - TN) jvar TN ---+ 0. The same is true if the scores are generated according to ( 13.4) for a function ¢ that is continuous almost everywhere, is nonconstant, and satisfies N - 1 "'£ � 1 ¢ 2 (i /(N + 1)) ---+ f01 ¢ 2 (u) du < oo. Set Ui = F ( X i ) , and view the rank vector R N as the ranks of the first N elements of the infinite sequence U1 , U2 , . . . . In view of statement (v) of the Lemma 1 3 . 1 the definition (13.3) is equivalent to

Proof.

This immediately yields that the projection of TN onto the set of all square-integrable functions of R N is equal to TN = E(TN I R N ) . It is straightforward to compute that N var aN , RN I 1 / (N - 1) "'£ (cNi - eN ) 2 "£ (aN i - ZiN ) 2 N - 1 var ¢ (U1 ) L ( CN i - CN ) 2 var cp (UI ) If it can be shown that the right side converges to 1 , then the sequences TN and TN are asymptotically equivalent by the projection theorem, Theorem 1 1 .2, and the proof for the scores (13.3) is complete. Using a martingale convergence theorem, we shall show the stronger statement

( 1 3.6) Because each rank vector R j - l is a function of the next rank vector R j (for one observation more), it follows that aN, RN ! = E (¢ ( U1 ) I R 1 , . . . , RN ) almost surely. Because ¢ is square­ integrable, a martingale convergence theorem (e.g., Theorem 10.5.4 in [42]) yields that the sequence aN, R N 1 converges in second mean and almost surely to E (¢ (U1 ) I R 1 , R2, . . . ) . If ¢ (U1 ) is measurable with respect to the a--field generated by R 1 , R 2 , . . . , then the condi­ tional expectation reduces to ¢ (UI ) and ( 13.6) follows. The projection of U1 onto the set of measurable functions of R N I equals the conditional expectation E(U1 I RN 1 ) = R N i f(N + 1). By a straightforward calculation, the sequence var (R N I / (N + 1)) converges to 1 / 1 2 = var U1 . By the projection Theorem 1 1 .2 it follows that R N i f (N + 1 ) ---+ U 1 in quadratic mean. Because R N I is measurable in the a- -field generated by R 1 , R 2 , . . . , for every N, so must be its limit U1 . This concludes the proof that ¢ (U1 ) is measurable with respect to the a--field generated by R 1 , R 2 , . . . and hence the proof of the theorem for the scores 13.3.

13. 1

bNi

Rank Statistics

177

a Ni

Next, consider the case that the scores are generated by ( 13 .4). To avoid confusion, write = ¢ ( 1 / (N + 1 ) ) and let these scores as be defined by (1 3.3) as before. We shall prove that the sequences of rank statistics SN and TN defined from the scores and bN, respectively, are asymptotically equivalent. Because RNJ !(N + 1) converges in probability to U1 and ¢ is continuous almost ev­ erywhere, it follows that ¢ (RNJ / (N + 1) ) � ¢ (U1 ) . The assumption on ¢ is exactly that E¢ 2 (RNJ / (N + 1) converges to E¢ 2 (U 1 ) . By Proposition 2.29, we conclude that ¢ ( RNJ/(N + 1) � ¢ (U 1 ) in second mean. Combining this with ( 1 3 .6), we obtain that 2 b Nt ) 2 = E R, ¢ -+ ,

aN

)

)

� t (a m -

(a N

-

(::\ ) ) 0.

By the formula for the variance of a linear rank statistic, we obtain that - CaN - b N ) ) 2 var (S TN) 2:=� 1 (a Ni � ' var TN (a )2 because var aN, R N 1 � var ¢ ( U1 ) > This implies that var SN fvar TN � 1 . The proof is complete. •

N

- bNi N Li= l Ni - aN

-

0.

0

iaN,

Under the conditions of the preceding theorem, the sequence of rank statistics l:= cN RN , is asymptotically equivalent to a sum of independent variables. This sum is asymptotically normal under the Lindeberg-Feller condition, given in Proposition 2.27 . In the present case, because the variables ¢ ( F(X J ) are independent and identically distributed, this is implied by max l -si -sN C cN i - eN ) 2 ( 1 3 .7) N ( - ) 2 � o. c N i cN This is satisfied by the most important choices of vectors of coefficients.

Li= l

If the vector of coefficients CN satisfies ( 1 3.7), and the scores are genera­ ted according to ( 1 3 .3) for a measurable, nonconstant, square-integrable function ¢, then the sequence of standardized rank statistics (TN ETN ) /sd TN converges weakly to an N(O, I)-distribution. The same is true if the scores are generated by ( 1 3 .4) for a function ¢ that is continuous almost everywhere, is nonconstant, and satisfies N - 1 2:=� 1 ¢ 2 ( i / (N + 1) � ¢ 2 (u) du. 13.8

Corollary.

-

) fol

13.9 Example (Monotone score generating functions). Any nondecreasing, nonconstant function ¢ satisfies the conditions imposed on score-generating functions of the type (13.4) in the preceding theorem and corollary. The same is true for every ¢ that is of bounded variation, because any such ¢ is a difference of two monotone functions. To see this, we recall from the preceding proof that it is always true that RNJ ! (N + 1) � U1 , almost surely. Furthermore,

E¢ 2

i ) N 1 N l(i + l)j ( N + l) 2 ( NRNl 1 ) = -N1 LN ¢2 ( -i =l N 1 :::: N Li = l i /(N+ l l ¢ (u) du . --

+

+

+

--

The right side converges to J ¢ 2 (u) du. Because ¢ is continuous almost everywhere, it follows by Proposition 2.29 that ¢ (RNJ / (N + 1) � ¢ (UJ ) in quadratic mean. 0

)

Rank, Sign, and Permutation Statistics

178

In a two-sample problem, in which the first m observations constitute the first sample and the remaining observations n = N - m the second, the coefficients are usually chosen to be 13.10

Example (Two-sample problem).

i = 1, . , m CNi = 1 l = m + 1 , . , m + n .

{0

.

.

.

. .

In this case eN = nj N and L � 1 (cNi -eN ) 2 = mnj N. The Lindeberg condition i s satisfied provided both m ---+ and n ---+ D oo

oo.

The function ¢ ( u) = u generates the scores aNi = i / (N + 1). Combined with "two-sample coefficients," it yields a multiple of the Wilcoxon statistic. According to the preceding theorem, the sequence of Wilcoxon statistics WN = L �m + 1 RNi / (N + 1) is asymptotically equivalent to 13.11

Example (Wilcoxon test).

The expectations and variances of these statistics are given by EWN var WN = mnf(12(N + 1) ) , and var WN = mnj(12N). D

n/2,

13.12 Example (Median test). The median test is a two-sample rank test with scores of the form aN i = ¢ (i j(N + 1)) generated by the function ¢ (u) = 1 co, 1 ;21 (u). The corresponding test statistic is

N

i

J;_

}

N+1 1 RNi ::::; -- . 2 1

{

This counts the number of Y1 less than the median of the pooled sample. Large values of this test statistic indicate that the distribution of the second sample is stochastically smaller than the distribution of the first sample. D The examples of rank statistics discussed so far have a direct intuitive meaning as statistics measuring a difference in location. It is not always obvious to find a rank statistic appropriate for testing certain hypotheses. Which rank statistics measure a difference in scale, for instance? A general method of generating rank statistics for a specific situation is as follows. Suppose that it is required to test the null hypothesis that X 1 , . . . , X N are i.i.d. versus the alternative that X 1 , . . . , X N are independent with Xi having a distribution with density fcNJh for a given one-dimensional parametric model e �----+ fe . According to the Neyman-Pearson lemma, the most powerful rank test for testing H0 : e = against the simple alternative H1 : e = e rejects the null hypothesis for large values of the quotient Pe (R N - r) = N ! Pe (RN = r ) . Po (RN = r )

0

__ _ _

Equivalently, the null hypothesis is rejected for large values of P8 (RN = r). This test depends on the alternative e , but this dependence disappears if we restrict ourselves to

179

13. 1 Rank Statistics

alternatives e that are sufficiently close to 0. Indeed, under regularity conditions,

Pe(RN = r) - Po(RN = r) =

r . L �, (fv'"'' (x, ) - !] to(x, ) ) dx l

N 1 = 8 - L cNi Eo N .,

i= I

dxN

( _Qjfo (Xi ) I RN = r ) + o (&) .

Pe ( R r) TN 2..::{: 1 R aNi = Eo �� ( XN<• l ) = E �� ( F0- 1 ( UN<, J ) .

Conclude that, for small e > 0, large values of N = correspond to large values of the simple linear rank statistic = C N i aN , N , , for the vector aN of scores given by

(j

F0- 1 .

These scores are of the form ( 1 3 .3), with score-generating function ¢ = 0 /fo) Thus the corresponding rank statistics are asymptotically equivalent to the statistics e Nd olfo (X J . Rank statistics with scores generated as in the preceding display yield locally most pow­ erfu l rank tests. They are most powerful within the class of all rank tests, uniformly in a sufficiently small neighbourhood (0, c) of 0. (For a precise statement, see problem 1 3 . 1). Such a local optimality property may seem weak, but it is actually of considerable im­ portance, particularly if the number of observations is large. In the latter situation, any reasonable test can discriminate well between the null hypothesis and "distant" alterna­ tives. A good test proves itself by having high power in discriminating "close" alternatives.

LiN=I .

13.13

Example (Two-sample scale).

o

To generate a test statistic for the two-sample scale

problem, let fe (x) = e- B f (e - 8 x) for a fixed density f. If xi has density fcN , B and the

vector c is chosen equal to the usual vector of two-sample coefficients, then the first m observations have density fo = f; the last n = N - m observations have density fe . The alternative hypothesis that the second sample has larger scale corresponds to e > 0. The scores for the locally most powerful rank test are given by

For instance, for f equal to the standard normal density this leads to the sank statistic with scores

L �=m + I aN,RN,

The same test is found for f equal to a normal density with a different mean or variance. This follows by direct calculation, or alternatively from the fact that rank statistics are location and scale invariant. The latter implies that the probabilities p.,a, B = of the rank vector of a sample of independent variables N with Xi distributed according to e - B f (e- 8 (x - J-L)/CJ ) /CJ do not depend on (J-L, CJ). Thus the procedure to generate locally most powerful scores yields the same result for any (J-L, CJ). 0

RN

X1 , . . . , X

P (RN r)

Rank, Sign, and Permutation Statistics

1 80

13.14 Example (Two-sample location). In order to find locally most powerful tests for location, we choose fe (x) = f (x - 8) for a fixed density f and the coefficients c equal to the two-sample coefficients. Then the first m observations have density f (x) and the last n = N - m observations have density f (x - 8). The scores for a locally most powerful rank test are

For the standard normal density, this leads to a variation of the van der Waerden statistic. The Wilcoxon statistic corresponds to the logistic density. D 13.15 Example (Log rank test). The cumulative hazardfunction corresponding to a con­ tinuous distribution function F is the function A = -log(l - F). This is an important modeling tool in survival analysis. Suppose that we wish to test the null hypothesis that two samples with cumulative hazard functions Ax and Ay are identically distributed against the alternative that they are not. The hypothesis of proportional hazards postulates that Ay = 8Ax for a constant 8 , meaning that the second sample is a factor e more "at risk" at any time. If we wish to have large power against alternatives that satisfy this postulate, then it makes sense to use the locally most powerful scores corresponding to a family defined by Ae = 8A 1 . The corresponding family of cumulative distribution functions Fe satisfies 1 - Fe = (1 - F1 ) e and is known as the family of Lehmann alternatives. The locally most powerful scores for this family correspond to the generating function

¢ (u)

=

a ae

a ax

- log - ( 1 - Fe ) (x) 1e_- 1 ' x - F-I I (u )

=

1 - log( l - u).

It is fortunate that the score-generating function does not depend on the baseline hazard function A 1 . The resulting test is known as the log rank test. The test is related to the Savage test, which uses the scores

N 1 a N , i = L -: J= N -i + 1 1

R:!

( -i ) . N 1

- log 1 -

+

The log rank test is a very popular test in survival analysis. Then usually it needs to be extended to the situation that the observations are censored. D 13.16 Example (More-sample problem). Suppose the problem is to test the hypothesis that k independent random samples X1 , . . . , XN1 , X N1 + 1 , . . . , XN2 , . . . , X Nk_ 1 + 1 , . . . , XNk are identical in distribution. Let N = Nk be the total number of observations, and let R N be the rank vector of the pooled sample X 1 , . . . , X N . Given scores aN inference can be based on the rank statistics

Nk N1 N2 TN1 = L aN , RNr ' TN 2 = L aN ,RNr ' . . . ' TNk = L aN ,RN, . i=1

i= N 1 +1

i =Nk-1 +1

The testing procedure can consist of several two-sample tests, comparing pairs of (pooled) subsamples, or on an overall statistic. One possibility for an overall statistic is the chi-square

13.2

Signed Rank Statistics

181

statistic. For n j = Nj - Nj _ 1 equal to the number of observations in the jth sample, define

� (TN1 - n/(iN) 2 2 = eN L...t = n j var ¢ (U 1 ) j 1

If the scores are generated by (13.3) or (13.4) and all sample sizes n j tend to infinity, then every sequence TN1 is asymptotically normal under the null hypothesis, under the conditions of Theorem 1 3.5. In fact, because the approximations N1 are jointly asymptotically normal by the multivariate central limit theorem, the vector TN = (TN1 , . . . , TNk) is asymptotically normal as well. By elementary calculations, if n i f N ---+ Ai ,

T

TN - ETN

v'ii sd ¢ (UI )

-'A2'A 1

-'A 1 'A2 'A2 ( 1 - 'A2 )

-'A 1 'Ak -'A2'Ak

-'Ak'A 1

-'Ak'A2

'Ak ( l - 'Ak )

)q (1

� Nk 0,

- 'A I )

This limit distribution is similar to the limit distribution of a sequence of multinomial vectors. Analogously to the situation in the case of Pearson's chi-square tests for a multinomial distribution (see Chapter 17), the sequence C1 converges in distribution to a chi-square distribution with k - 1 degrees of freedom. There are many reasonable choices of scores. The most popular choice is based on ¢ (u) = u and leads to the Kruskal-Wallis test. Its test statistic is usually written in the form

(-

)

k 12 N+1 2 n j R j - -- ' . 2 N(N - 1 )

j;

This test statistic measures the distance of the average scores of the k samples to the average score (N + 1 )/2 of the pooled sample. An alternative is to use locally asymptotically powerful scores for a family of distribu­ tions of interest. Also, choosing the same score generating function for all subsamples is convenient, but not necessary, provided the chi-square statistic is modified. D

13.2

Signed Rank Statistics

The sign of a number x, denoted sign(x), is defined to be - 1 , 0, or 1 if x < 0, x = 0 or x > 0, respectively. The absolute rank R ti of an observation Xi in a sample X 1 , , X N is defined as the rank of I Xi I in the sample of absolute values I X 1 1 , . . . , I X N 1 . A simple linear signed rank statistic has the form .

.

•

The ordinary ranks of a sample can always be derived from the combined set of absolute ranks and signs. Thus, the vectors of absolute ranks and signs are together statistically more informative than the ordinary ranks. The difference is dramatic if testing the location of a symmetric density of a given form, in which case the class of signed rank statistics contains asymptotically efficient test statistics in great generality.

1 82

Rank, Sign, and Permutation Statistics

The main attraction of signed rank statistics is their simplicity, particularly their being distribution-free over the set of all symmetric distributions. Write l X I , Rt, and signN (X) for the vectors of absolute values, absolute ranks, and signs.

Let X 1 , . . . , XN be a random sample from a continuous distribution that is symmetric about zero. Then (i) the vectors ( l X I , Rt) and signN (X) are independent; (ii) the vector Rt is uniformly distributed over { 1 , . . . , N}; N (iii) the vector signN (X) is uniformly distributed over { - 1, 1} ; (iv) forany signed rank statistic, var 'L;: 1 aN, , sign(Xi ) = L ;: 1 a�i · 13.17

Lemma.

Rt

Consequently, for testing the null hypothesis that a sample is i.i.d. from a continuous, symmetric distribution, the critical level of a signed rank statistic can be set without further knowledge of the "shape" of the underlying distribution. The null hypothesis of symmetry arises naturally in the two-sample problem with paired observations. Suppose that, given independent observations (X 1 , Y1 ) , . . . , (XN , YN), it is desired to test the hypothesis that the distribution of Xi - Yi is "centered at zero." If the observations (Xi , Yi) are exchangeable, that is, the pairs (Xi , Yi) and (Yi , Xi ) are equal in distribution, then Xi - Yi is symmetrically distributed about zero. This is the case, for instance, if, given a third variable (usually called "factor"), the observations Xi and Yi are conditionally independent and identically distributed. For the vector of absolute ranks to be uniformly distributed on the set of all permutations it is necessary to assume in addition that the differences are identically distributed. For the signs alone to be distribution-free, it suffices, of course, that the pairs are inde­ pendent and that P(Xi < YJ P(Xi > Yi ) � for every Consequently, tests based on only the signs have a wider applicability than the more general signed rank tests. However, depending on the model they may be less efficient.

=

i.

=

Let X 1 , . . . , XN be a random sample from a continuous distribution that is symmetric about zero. Let the scores aN be generated according to ( for a measurable function ¢ such that f01 ¢2 ( u) d u For p+ the distribution function of I X 1 l , define N N + '""" TN = i 1 aN sign(Xi) , TN = L ¢ ( F ( I Xi l ) ) sign(Xd . i 1 1 var (TN Then the sequences TN and TN are asymptoticall y equi v alent in the sense that N - 1 12 TN is asymptotically normal with mean zero TN) 0. Consequentl y , the sequence N 1 and variance J0 ¢ 2 (u) d u. The same is true if the scores are generated -accordi ng2 to 1 for a function ¢ that is continuous almost everywhere and satisfies N L;: 1 ¢ (i j ( N + 1 2 1) ) fo ¢ (u) du Because the vectors signN (X) and ( l X I , Rt) are independent and E signN (X) = 0, the means of both TN and TN are zero. Furthermore, by the independence and the 13.18

Theorem.

< oo.

L =

'

---+

---+

13. 3)

R+

Nt

=

(13.4)

< oo.

Proof

orthogonality of the signs,

13.4

1 83

Rank Statistics for Independence

The expectation on the right side is exactly the expression in ( 1 3.6), evaluated for the special choice U1 = p + ( I X 1 1 ) . This can be shown to converge to zero as in the proof of Theorem 1 3 .5. •

Wilcoxon signed rank statlstzc u.

The Large = 2.:= � 1 R t i sign(Xi ) is obtained from the score-generating function ¢ ( u ) = values of this statistic indicate that large absolute values I Xi I tend to go together with pos­ itive Xi . Thus large values of the Wilcoxon statistic suggest that the location of the Xi is larger than zero. Under the null hypothesis that X 1 , . . . , X N are i.i.d. and symmetrically distributed about zero, the sequence is asymptotically normal N(O, 1 13). The variance of is equal to N(2 N + 1 ) (N + 1 ) 1 6. The signed rank statistic is asymptotically equivalent to the U -statistic with kernel h (x 1 , x 2 ) = 1 {x 1 + x2 > 0}. (See problem 12.9.) This connection yields the limit distri­ bution also under nonsymmetric distributions. 0 13.19

WN

Example (Wilcoxon signed rank statistic).

N -312 WN

WN

Signed rank statistics that are locally most powerful can be obtained in a similar fashion as locally most powerful rank statistics were obtained in the previous section. Let f be a symmetric density, and let X 1 , , XN be a random sample from the density f(- -e). Then, under regularity conditions, .

Pe ( signN (X)

= = =

.

•

s, R t N r ) - Po ( signN (X) s, R t =

=

j ' ( I Xi l ) { signN (x) -e Eo L sign(Xi ) f i= 1 N 1 f' -e -N-, L iEo - ( I Xi l ) I R t i f 2 N . i =1

s (

r)

=

=

s, R t r } o (e ) ) o (e ) . ri =

=

+

+

In the second equality it is used that f' If (x) is equal to sign(x) f' If ( l x I) by the skew symmetry of f' If. It follows that for testing · f against f ( -e) are obtained from the scores

locally most poweiful signed rank statistics

These scores are of the form (1 3.3) with score-generating function ¢ = - (f' 1f) (F + ) - 1 , whence locally most powerful rank statistics are asymptotically linear by Theorem 1 3 . 1 8. By the symmetry of F, we have (F + ) - 1 = p- 1 ( + 1) 12) . o

(u)

(u

Example. The Laplace density has score function f' lf (x) = sign(x) = 1 , for x :::= 0. This leads to the locally most powerful scores 1 . The corresponding test statistic is the TN = 2.:= � 1 sign(Xi ) . Is it surprising that this simple statistic possesses an optimality property? It is shown to be asymptotically optimal for testing Ho : e = 0 in Chapter 1 5 . 0 13.20

sign statistic

13.21

aNi

=

Example. The locally most powerful score for the normal distribution are N i + 1) 12) . These are appropriately known as the normal (absolute) scores. 0

E