This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
0 such that I
If(x!>
X2,
e) - f(x 1,
X2'
k3/K) I ::;;
8
+ 2{ IX11 >
K}
+ 2{ IX21 >
K}.
With the integer k running over the finite range needed for these intervals to cover [ -1, IJ, the functions f(x 1 ,x 2 ,k3/K)
± 8 ± 2{lx 11>
K}
± 2{lx21 >
K}
provide the upper and lower approximations required by Theorem 2. As noted following Theorem 3, the uniform strong laws also apply to empirical measures constructed from stationary ergodic sequences. Accordingly, (6)
sup IQnf(·,·, e) - Qf(.,., e)l-+ 0 almost surely, 101 :51
that is, sup IHn(e) - H(e)l-+ 0 almost surely. 10 1:5 1
Provided en lies in the range [ -1, IJ, we can deduce from (6) that H(8 n) -+ 0 almost surely. It would be a sore embarrassment if the estimate of the autoregressive parameter were not in this range. Usually one avoids the embarrassment by insisting only that Hn(e n) -+ 0, with en in [ -1, 1]. Such a en always exists because HnCe*) -+ 0 almost surely. Convergence results for en depend upon the form of HO. We know en gets forced eventually into the set {I H I < 8} for each 8 > O. If this set shrinks to e* as 8 ! 0 then en must converge to e*, which necessarily would have to be the unique zero of HO. If we assume that H does have these properties we get the consistency result for the generalized M-estimator. 0
11.3. The Combinatorial Method Since understanding of general methods grows from insights into simple special cases, let us begin with the best-known example of a uniform strong law of large numbers, the classical Glivenko-Cantelli theorem. This asserts that, for every distribution P on the real line, (7)
sup Ipnc -
00,
tJ -
P( -
00,
tJ I -+ 0
almost surely,
when the empirical measure Pn comes from independent sampling on P. The ideas that will emerge from the treatment of this special case will later be expanded into methods applicable to other classes of functions. To facilitate back reference, break the proof into five steps.
14
11. Uniform Convergence of Empirical Measures
Keep the notation tidy by writing 11·11 to denote the supremum over the class f of intervals (- 00, t], for - 00 < t < 00. We could restrict the supremum to rational t to ensure measurability. FIRST SYMMETRIZATION. Instead of matching P n against its parent distribution P, look at the difference between P n and an independent copy, P~ say, of itself. The difference P n - P~ is determined by a set of 2n points (albeit random) on the real line ; it can be attacked by combinatorial methods, which lead to a bound on deviation probabilities for IlPn - P~II. A symmetrization inequality converts this into a bound on IlPn - PII deviations. 8 Symmetrization Lemma. Let {Z(t): t ET} and {Z'(t): t ET} be independent stochastic processes sharing an index set T. Suppose there exist constants [J > 0 and a > 0 such that IP{ IZ'(t) I :::; a} 2:: [J for every t in T. Then (9)
IP{s~p IZ(t) I > 8}:::; P-IIP{S~p IZ(t) -
Z'(t) I >
8- a}
PROOF. Select a random 7: for which IZ(7:) I > 8 on the set {sup /Z(t)1 > 8}. Since 7: is determined by Z, it is independent of Z'. It behaves like a fixed index value when we condition on Z: IP{IZ'(7:)I :::; a/Z} 2:: [J. Integrate out.
[JIP{s~p IZ(t)1 > 8}:::; IP{IZ'(r)1 :::; a, IZ(r) I > e} :::; IP{ IZ(7:)- Z'(7:) I > :::;
IP{s~p IZ(t) -
8 -
Z'(t) I >
a} 8 -
a}
D
Close inspection of the proof would reveal a disregard for a number of measure-theoretic niceties. A more careful treatment may be found in Appendix C. For our present purpose it would suffice if we assumed T countable; the proof is impeccable for stochastic processes sharing a countable index set. We could replace suprema over all intervals (- 00, t] by suprema over intervals with a rational end point. For fixed t, Pn( - 00, t] is an average of the n independent random variables {~i :::; t}, each having expected value P( - 00, t] and variance P( - 00, t] (P( - 00, t])2, which is less than one. By Tchebychev's inequality, IP{IP~( -00, t] - P( -00, t]1 :::; !8} 2::!
if n 2:: 88- 2 .
15
11.3. The Combinatorial Method
Apply the Symmetrization Lemma with Z = P n class f as index set, rx = 1e, and fJ = 1-
(10)
W{llPn - PII > e} ::;; 2W{llPn -
-
P and Z'
P~II >!e}
=
P~ - P,
the
if n;:::: 8e- 2 .
SECOND SYMMETRIZATION.
The difference P n - P~ depends on 2n observations. The double sample size creates a minor nuisance, at least notationally. It can be avoided by a second symmetrization trick, at the cost of a further diminution of the e. Independently of the observations ~l"'" ~n' ~'l"'" ~~ from which the empirical measures are constructed, generate independent sign random variables (Jl"'" (In for which W{(Ji = + 1} = W{(Ji = -1} = !. The symmetric random variables {~i::;; t} - {~; ::;; t}, for i = 1, ... , nand - 00 < t < 00, have the same joint distribution as the random variables (JH~i ::;; t} - g; ::;; t}]. (Consider the conditional distribution given {(JJ.) Thus
W{llPn -
P~II > !e} = w{s~p !n-1itl (JH~i::;; t} - {~;::;; t}]! > !e} ::;;
itl
w{s~p !n- 1 (Ji{~i::;; t}! >!c} + w{s~p !n-
1
itl(Jig;::;;
t}! > !cl
Write P~ for the signed measure that places mass n-1(Ji at ~i' The two symmetrizations give, for n ;:::: 8e - 2, (11)
W{llPn - PII > e} ::;;
4W{IIP~11 >
!c}.
To bound the right-hand side, work conditionally on the vector of observations ;, leaving only the randomness contributed by the sign variables. MAXIMAL INEQUALITY.
Once the locations of the; observations are fixed, the supremum IIP~II reduces to a maximum taken over a strategically chosen set of intervals I j = (- 00, t j ], for j = 0, 1, ... , n. Of course the choice of these intervals depends on ;; we need one tj between each pair of adjacent observations. (The to and tn are not really necessary.) With the number of intervals reduced so drastically, we can afford a crude bound for the supremum. n
(12)
W{IIP~II >
iel;} ::;; I
W{IP~Ijl >
iel;}
j~O
::;; (n
+ 1) max W{IP~Ijl > iel;}. j
16
11. Uniform Convergence of Empirical Measures
This bound will be adequate for the present because the conditional probabilities decrease exponentially fast with n, thanks to an inequality of Hoeffding for sums of independent, bounded random variables. EXPONENTIAL BOUNDS.
Let Y1,"" y" be independent random variables, each with zero mean and bounded range: ai::;; Yi ::;; bi' For each ry > 0, Hoeffding's Inequality (Appendix B) asserts IP{IY1
+ ... +
Y"I;:::: ry} ::;;2exp [ -2ry2!J1(b i
-
aY].
Apply the inequality with Yi = ai{~i ::;; t}. Given ~, the random variable takes only two values, ± {~i ::;; t}, each with probability l
Yi
IP{IP~(-CX), t]l;:::: hl~}::;; 2ex p [ -2(ne/4)2lt4{~i::;; t}] ::;; 2 exp( - ne 2 /32), because the indicator functions sum to at most n. Use this for each I j in inequality (12). IP{ 11 P~ 11 > I~} 2(n + 1) exp( - ne 2/32).
te
:;
Notice that the right-hand side now does not depend on
~.
INTEGRATION.
Take expectations over
~.
IP{IJPn - PII > e} ::;; 8(n
+ 1) exp( -ne 2/32).
This gives very fast convergence in probability, so fast that 00
L IP{IIP
n -
PII > e} < CX)
n =1
for each e > 0. The Borel-Cantelli lemma turns this into the full almost sure convergence asserted by the Glivenko-Cantelli theorem.
11.4. Classes of Sets with Polynomial Discrimination We made use of very few distinguishing properties of intervals for the proof of the Glivenko-Cantelli theorem in Section 3. The main requirement was that they should pick out at most n + 1 subsets from any set of n points. Other classes have a similar property. For example, quadrants of the form ( - 00, t] in JR2 can pick out fewer than (n + 1)2 different subsets from a
11.4. Classes of Sets with Polynomial Discrimination
17
set of n points in the plane-there are at most n + 1 places to set the horizontal boundary and at most n + 1 places to set the vertical boundary. (Problem 8 gives the precise upper bound.) With (n + 1)2 replacing the n + 1 factor, we could repeat the arguments from Section 3 to get the bivariate analogue of the Glivenko-Cantelli theorem. The exponential bound would swallow up (n + 1)2, just as it did the n + 1. Indeed, it would swallow up any polynomial. The argument works for intervals, quadrants, and any other class of sets that picks out a polynomial number of subsets. 13 Definition. Let qj) be a class of subsets of some space S. It is said to have polynomial discrimination (of degree v) if there exists a polynomial p(.) (of degree v) such that, from every set of N points in S, the class picks out at most peN) distinct subsets. Formally, if So consists of N points, then there are at most peN) distinct sets of the form So (\ D with D in qj). Call pC) the discriminating polynomial for qj). D
When the risk of confusion with the algebraic sort of polynomial is slight, let us shorten the name "class having polynomial discrimination" to "polynomial class," and adopt the usual terminology for polynomials of low degree. For example, the intervals on the real line have linear discrimination (they form a linear class) and the quadrants in the plane have quadratic discrimination (they form a quadratic class). Of course there are classes that don't have polynomial discrimination. For example, from every collection of N points lying on the circumference of a circle in IR 2 the class of closed, convex sets can pick out all 2N subsets, and 2N increases much faster than any polynomial.
The method of proof set out in Section 3 applies to any polynomial class of sets, provided measurability complications can be taken care of. Appendix C describes a general method for guarding against these complications. Classes satisfying the conditions described there are called permissible. Every specific class we shall encounter will be permissible. As the precise details of the method are rather delicate-they depend upon properties of analytic sets-let us adopt a naive approach. Ignore measurability problems from now on, but keep the term permissible as a reminder that some regularity conditions are needed if pathological examples (Problem 10) are to be excluded. Problems 3 through 7 describe a simpler approach, based on the more familiar idea of existence of countable, dense subclasses.
18
11. Uniform Convergence of Empirical Measures
14 Theorem. Let P be a probability measure on a space S. For every permissible class f0 of subsets of S with polynomial discrimination,
sup IPnD - PDI--+ 0
almost surely.
~
PROOF. Go back to Section 3, change ~ to f0, replace the n + 1 multiplier by the polynomial appropriate to f0, and strike out the odd reference to D interval and real line. Which classes have only polynomial discrimination? We already know about intervals and quadrants; their higher-dimensional analogues have the property too. Other classes can be built up from these. 15 Lemma. If
~
and f0 have polynomial discrimination, then so do each of:
(i) {D c :DEf0}; (ii) {C u D: C E (iii) {C n D: C E
~ ~
and D and D
E E
f0}; f0}.
PROOF. Write cO and dO for the discriminating polynomials. We may assume them both to be increasing functions of N. From a set So consisting of N points, suppose ~ picks out subsets S 1, ... , Sk with k s c(N). Suppose Si consists of Ni points. The class f0 picks out at most deN) distinct subsets from Si. This gives the bound d(N 1 ) + ... + d(N k ) for the size of the class in (iii). The sum is less than c(N) deN). That proves the assertion for (iii). The other two are just as easy. D The lemma can be applied repeatedly to generate larger and larger polynomial classes. We must place a fixed limit on the number of operations allowed, though. For instance, the class of all singletons has only linear discrimination, but with arbitrary finite unions of singletons we can pick out any finite set. Very quickly we run out of interesting new classes to manufacture by means of Lemma 15 from quadrants and the like. Fortunately, there are other systematic methods for finding polynomial classes. Polynomials increase much more slowly than exponentials. For N large enough, a polynomial class must fail to pick out at least one of the 2N subsets from each collection of N points. Surprisingly, this characterizes polynomial discrimination. Some picturesque terminology to describe the situation has become accepted in the literature. A class f0 is said to shatter a set of points F if it can pick out every possible subset (the empty subset and the whole of F included); that is, f0 shatters F if each of the subsets of F has the form D n F for some D in f0. This conveys a slightly inappropriate image, in which F gets broken into tiny fragments, rather than an image of a diligent f0 trying to pick out all the different subsets of F; but at least it is vivid.
19
II.4. Classes of Sets with Polynomial Discrimination
For example, the class of all closed discs in IR 2 can shatter each three-point set, provided the points are not collinear. But from no set of four points, no matter what its configuration, can the discs pick out more than 15 of the 16 possible subsets. The discs shatter some sets of three points; they shatter no set of four points. 16 Theorem. Let So be a set of N points in S. Suppose there is an integer V ~ N such that ::0 shatters no set of V points in So. Then ::0 picks out no more than (~) + (~) + ... + (v i'I. 1) subsets from So· PROOF. Write F 1, .•. , F k for the collection of all subsets of V elements from So. Of course k = (~). By assumption, each Fi has a "hidden" subset Hi that ::0 overlooks: D n Fi 1= Hi for every Din ::0. That is, all the sets of the form D n So, with D in ::0, belong to ~o
= {C
S;
So: C n Fi 1= Hi for each i}.
It will suffice to find an upper bound for the size of ~ 0 .
In one special case it is possible to count the number of sets in ~o directly. ~o can contain an F i ; no C can contain a set of V points. In other words, members of ~o consist of either 0, 1, ... , or V-I points. The sum of the binomial coefficients gives the number of sets of this form. By playing around with the hidden sets we can reduce the general case to the special case just treated. Label the points of So as 1, ... , N. For each i define H; = (Hi u {I}) n F i ; that is, augment Hi by the point 1, provided it can be done without violating the constraint that the hidden set be contained in F i • Define the corresponding class
If Hi = Fi for every i then no C in
~1 =
{C
S;
So: C n Fi 1= H;
for each i}.
The class ~ 1 has nothing much to do with ~ o. The only connection is that all its hidden sets, the sets it overlooks, are bigger. Let us show that this implies ~ 1 has a greater cardinality than ~o. (Notice: the assertion is not that ~o S; ~1.)
Check that the map CH C\{I} is one-to-one from ~O\~l into ~l\~O' Start with any C in ~o \ ~1' By definition, C n Fi 1= Hi for every i, but C n F j = Hi for at least one j. Deduce that H j 1= Hi, so 1 belongs to C and F j and Hi, but not to H j • The stripping of the point 1 does define a one-to-one map. Why should C\ {I} belong to ~ 1 \ ~0 ? Observe that (C\{I}) n F j = Hi\{I} = H j ,
which bars C\{I} from belonging to ~o. Also, if Fi contains 1 then so must H;, but C\{I} certainly cannot; and if Fi doesn't contain 1 then (C\{1}) n Fi = C n Fi 1= Hi = H;.
In either case (C\{I}) n Fi 1= H;, so C\{I} belongs to
~1'
20
H. Uniform Convergence of Empirical Measures
Repeat the procedure, starting from the new hidden sets and with 2 taking over the role played by 1. Define Hi = (H; u {2}) n Fi and ~2
= {C S So: C n Fi
=F Hi' for each i}.
The cardinality of ~2 is greater than the cardinality of ~ 1. Another N .- 2 repetitions would generate classes ~ 3, ~4, ... , ~N with increasing cardirtalities. The hidden sets for ~N would fill out the whole of each F i : the special 0 case already treated. 17 Corollary. If a class shatters no set of V points, then it must have polynomial discrimination of degree no greater than V - 1. 0
All we lack now is a good method for identifying classes that have trouble picking out subsets from large enough sets of points. 18 Lemma. Let <'§ be a finite-dimensional vector space of real functions on S. The class of sets of the form {g 2 O},for gin <'§, has polynomial discrimination of degree no greater than the dimension of <'§. PROOF. Write V-I for the dimension of<'§. Choose any collection {Sb . .. ,sv} of distinct points from S. (Everything reduces to triviality if S contains fewer than V points.) Define a linear map L from <'§ into lR v by
L(g)
= (g(S1),.·., g(sv)).
Since L<'§ is a linear subspace oflR v of dimension at most V-I, there exists in JR'" a non-zero vector Y orthogonal to L<'§. That is,
L Yig(SJ = i
° for each g in <'§,
or (19)
L Yig(Si) = L (-yJg(sJ {+)
for each g.
{-I
Here { + } stands for the set of those i for which Yi 2 0, and { - } for those with Yi < 0. Replacing Y by -Y if necessary, we may assume that {-} is non-empty. Suppose there were a g for which {g 2 O} picked out precisely those points Si with i in { + }. For this g, the left-hand side of (19) would be 2 0, but the right-hand side would be < 0. We have found a set that cannot be picked out. 0 Many familiar classes of geometric objects fall within the scope of the lemma. For example, the class of subsets of the plane generated by the linear space of quadratic forms ax 2 + bxy + cy2 + dx + ey + f includes all closed discs, ellipsoids, and (as a degenerate case) half-spaces. More complicated regions, such as intersections of 257 closed or open half-spaces, can be built up from these by means of Lemma 15. You can feed them into
21
HA. Classes of Sets with Polynomial Discrimination
Theorem 14 to churn out a whole host of generalizations of the classical Glivenko-Cantelli theorem. Uniform limit theorems for polynomial classes of sets have one thing in common: they hold regardless of the sampling distribution. This happens because the number ~n(~) of subsets picked out by the class from the sample {~1, ... , ~n} can be bounded above by a polynomial in n, independently of the configuration of that sample. Without the uniform bound the inequality (12) would be replaced by (20)
IP{IIP~II > tcl~} ~ 2~nC~) exp( -n8 2 /32).
Write w" for the minimum of 1 and the right-hand side of (20). Then the argument from the INTEGRATION step gives the sharper bound
IP{llPn - PII > 8}
~
4IPw"
for all n large enough. Thus a sufficient condition for IlPn - PII to converge in probability to zero is: IPw" ~ 0 for each 8> O. Equivalently, because o ~ w" ~ 1, we could check that log ~n(~) = open). Theorem 16 helps here. ~n(~) ~
BnCV - 1) = (0)
+ ... + (v~ 1),
where V = v,,(~1' ... , ~n) is the smallest integer such that f» shatters no collection of V points from {~1' ... ' ~n}. Set k = V - 1. If k ~ all the terms in the sum for Bn(k) are less than (k):
tn,
n -1 log Bn(k)
~
n - 1 log[(k
+
l)n !f(n - k)! k!].
Three applications of Stirling's approximation and some tidying up reduce the right-hand side to -(1 - kin) 10g(1 - kin) - (kin) log(kln)
+ 0(1),
which tends to zero as kin ~ O. It follows that both n -1 log ~n ~ 0 and lIPn- PII ~ 0 in probability, if V In ~ 0 in probability. If we don't know how fast Vln converges to zero, we can't use the BorelCantelli lemma to deduce from these inequalities that IlPn - PII converges almost surely to zero. But there is another reason why the convergence in probability implies the stronger result. Symmetry properties would force IIPn- PII to converge almost surely to some constant, no matter how V In behaved. Given P n , the unordered set {~1, ... , ~n} is uniquely determined, but there's no way of deciding the order in which the observations were generated. Given P n+ 1, we know slightly less about {~1> ... , ~n}; it could be any of the (n + 1) possible subsets of size n obtained by deleting one of the support points of P n + 1. (Count coincident observations as distinct support points.) The conditional distribution of P n given P n + 1 must be uniform on one of these (n + 1) subsets, each subset being chosen with probability (n + 1)-1. The conditional expectation of P n given P n + 1 (in the intuitive sense of the average over the n + 1 possible choices for P n) must be P n+ 1. The extra information carried
n.
22
Uniform Convergence of Empirical Measures
by P n+ 2 , P n+ 3 , ••• adds nothing more to our knowledge about P n; the conditional expectation of P n given the a-field generated by P n+ 1 , P n+ 2 , ••• still equals Pn+ l' That is, the sequence {P n} is a reversed martingale, in some wonderful measure-valued sense. Apply Jensen's inequality to the convex function thattakes P nonto lIPn - PII to deduce that {lIP n - PII} is a bounded, reversed submartingale. (Problem 11 arrives at the same conclusion in a slightly more rigorous manner.) Such a sequence must converge almost surely (Neveu 1975, Proposition V-3-13) to a limit random variable, W. Since W is unchanged by finite permutations of {~;}, the zero-one law of Hewitt and Savage (Breiman 1968, Section 3.9) forces it to take on a constant value almost surely. The only question remaining for the proof of a uniform strong law oflarge numbers is whether the constant equals zero or not: convergence in probability to zero plus convergence almost surely to a constant gives convergence almost surely to zero. 21 Theorem. Let:» be a permissible class of subsets of S. A necessary and sufficient condition for sup IPnD - PDI- 0
almost surely
~
is the convergence ofn-q;;, to zero in probability, where v" = v,,(~1"'" ~n) is the smallest integer such that :» shatters no collection of v" points from
{~l""'~n}' PROOF. You can formalize the sufficiency argument outlined above; necessity is taken care of in Problem 12. D Because 0::; n- 1v,,::; 1, convergence in probability of n- 1v" to zero is equivalent to n- 1 IPv" _ O. This has an appealing interpretation. The uniform strong law of large numbers holds if and only if, on the average, the class of sets behaves as if it has polynomial discrimination with degree but a tiny fraction of the sample size. 22 Example. Let's see how easy it is to check the necessary and sufficient condition stated in Theorem 21. Consider the class Cfl of all closed, convex subsets of the unit square [0, 1]2. We know that there exist arbitrarily large collections of points shattered by Cfl. Were we sampling from a non-atomic
23
HA. Classes of Sets with Polynomial Discrimination
distribution concentrated around the rim of a disc inside [0, 1J2, the class ~ could always pick out too many subsets from the sample. Indeed, there would always exist a convex C with P n C = 1 and PC = 0. But such configurations of sample points should be thoroughly atypical for sampling from the uniform distribution on [0, 1]2. Theorem 21 should say something useful in that case. How large a subcollection of sample points can ~ shatter? Suppose it is larger than the size requested by Theorem 21. That is, for some 8 > 0, IP{n-1v" ;;:::
8} ;;::: 8
infinitely often.
This will lead us to a contradiction. A set of k points is shattered by ~ if and only if none of the points can be written as a convex combination of the others; each must be an extreme point of their convex hull. So there exists a convex set whose boundary has empirical measure at least kin, which seems highly unlikely because P puts zero measure around the boundary of every convex set. Be careful of this plausibility argument; it contains a hidden appeal to the very uniformity result we are trying to establish. An approximation argument will help us to avoid the trap. Divide [0, 1J2 into a patchwork of m 2 equal subsquares, for some fixed m that will be specified shortly. Because the class .91 of all possible unions of these subsquares is finite,
IP{s~ IPnA -
PA I ;;:::
~8} < ~8
for all n large enough.
The ~8 here is chosen to ensure that, for some n, IP{n-1v" ;;:::
8
and sup IPnA - PAl < ~8}
>!a-
d
Since a set with positive probability can't be empty, there must exist a sample configuration for which ~ shatters some collection of at least n8 sample points and for which IPnA - PAl < ~8 for every A in d. Write H for the convex hull of the shattered set, and AH for the union of those subsquares that intersect the boundary of H. The set AH contains all the extreme points of H, so PnAH ;;::: 8; it belongs to .91, so IPnAH - PAHI < !a- Consequently PA H > k which will give the desired tontradiction if we make m large enough.
24
H. Uniform Convergence of Empirical Measures
Experiment with values of m equal to a power of 3. No convex set can have boundary points in all nine of the subsquares; the middle subsquare would lie inside the convex hull of four points occupying each of the four corner squares. For every convex C the P measure of the union of those
subsquares intersecting its boundary must be less than !. Subdivide each of the nine sub squares into nine parts, then repeat the same argument eight times. This brings the measure of squares on the boundary down to (!)2. Keep repeating the argument until the power of! falls below 1c. That destroys the claim made for A H • 0
11.5. Classes of Functions The direct approximation methods of Section 2 gave us sufficient conditions for the empirical measure Pn to converge to the underlying P uniformly over a class of functions, sup IP nf
- Pf I -4 0
almost surely.
fi'
The conditions, though straightforward, can prove burdensome to check. In this section a transfusion of ideas from Sections 3 and 4 will lead to a more tractable condition for the uniform convergence. The method will depend heavily on the independence of the observations {~i}' but the assumption of identical distribution could be relaxed (Problem 23). Throughout the section write 11-11 to denote sUpfi' 1·1. Let us again adopt a naive approach towards possible measurability difficulties, with only the word permissible (explained in Appendix C) to remind us that some regularity conditions are needed to exclude pathological examples. A domination condition will guard against any complications that could be caused by $' containing unbounded functions. Call each measurable F such that If I ::;; F, for every f in $', an envelope for !F. Often F will be taken as the pointwise supremum of If lover $', the natural envelope, but it will
25
lI.5. Classes of Functions
be convenient not to force this. We shall assume PF < 00. With the proper centering, the natural envelope must satisfy this condition (Problem 14) if the uniform strong law holds. The key to the uniform convergence will again be an approximation condition, but this time with distances calculated using the 21 seminorm for the empirical measures themselves. This allows us to drop the requirement that the approximating functions sandwich each member of :F. 23 Definition. Let Q be a probability measure on Sand :!i' be a class of functions in 2 1 (Q). For each s > 0 define the covering number N 1(s, Q,:!i') as the smallest value of m for which there exist functions g b . . . , gm (not necessarily in :!i') such that min j Q If - gj I ::;; s for each f in :F. For definite0 ness set N 1(s, Q, :!i') = 00 if no such m exists.
If :!i' has envelope F we can require that the approximating functions satisfy the inequality Igjl ::;; F without increasing N 1 (s, Q, :!i'): replace gj by
max{ -F, min[F, gjJ}. We could also require g j to belong to :!i', at the cost of a doubling of s: replace gj by an jj in:!i' for which QIjj - gjl ::;; s. 24 Theorem. Let :!i' be a permissible class of functions with envelope F. Suppose PF < 00. If P n is obtained by independent sampling from the probability measure P and if log N 1 (s, Pn,:!i') = op(n) for each fixed s > 0, then sup? IPnf - Pfl-+ 0 almost surely.
PROOF. Problem 11 (or the slightly less formal symmetry argument leading up to Theorem 21 in Section 4) shows that {llPn - PII} is a reversed submartingale; it converges almost surely to a constant. It will suffice if we deduce from the approximation condition that {llPn - PII} converges in probability to zero. Exploit integrability of the envelope to truncate the functions back to a finite range. Given £ > 0, choose a constant K so large that PF {F > K} < £. Then sup IPnf - Pfl ::;; sup IPnf{F ::;; K} - Pf{F::;; K}I ~
,~
+ sup Pnlfl{F >
K}
+ sup Plfl{F >
K} .
§
.'F
Because I f I ::;; F for each f in :!i', the last two terms sum to less than PnF{F> K}
+ PF{F >
K}.
This converges almost surely to 2PF {F > K}, which is less than 2£. It remains for us to show that the supremum over the functions f {F ::;; K} converges in probability to zero. As truncation can only decrease the 2\P n ) distance between two functions, the condition on log covering numbers also
26
H. Uniform Convergence of Empirical Measures
holds if eachfis replaced by its truncation; without loss of generality we may assume that If I ::; K for each f in JF In the two SYMMETRIZATION steps of the proof of the Glivenko-Cantelli theorem (Section 3) we showed that IP{llPn - PII > s} ::; 4IP{IIP~11 >
h}
n ~ 8s- 2 ,
for
where 11·11 denoted a supremum over intervals ( - 00, tJ of the real line. The signed measure P~ put mass ±n- 1 on each observation ~l' ••• '~n' the random ± signs being decided independently of the {~i}. The argument works just as well if 11·11 denotes a supremum over g;; the interpretation adopted in the current section. The only property of the indicator function ( - 00, tJ needed in the SYMMETRIZATION steps was the boundedness, which implied var(PnC - 00, tJ) ::; n- 1 . This time an ~xtra factor of K2 would appear in the lower bound for n. With intervals we were able to reduce IIP~ 11 to a maximum over a finite collection; for functions the reduction will not be quite so startling. Given ~,choose functions gl' ... ' gM' where M = N 1(is, P n , fiP), such that min P n If
- gj I ::; is
for each f in JF
j
Write f* for the gj at which the minimum is achieved. Now we reap the benefits of approximation in the :£l(Pn ) sense. For any function g,
IP~gl =
1 1 \n- it1 ±g(O\::; n-
J1 19(OI
=
Pnlgl·
Choose 9 = f - f* for each f in turn. IP{suJ
IP~fl > *sl~}::; IP{s~p [IP~f*1 + Pnlf ::; IP{m:x
f*IJ >
hl~}
IP~gjl > isl~} because Pnlf -
f*l::; is
::; N1(is, Pn , fiP) max IP{IP~gjl > isl~}. j
Once again Hoeffding's Inequality (Appendix B) gives an excellent bound on the conditional probabilities for each gj.
IP{IP~gjl > isl~} =
IP{\it1
± gi~;)1 >
insl~}
::; 2 exp [ - 2(ins)2lt1 ::; 2 exp( -ns 2 j128K 2 )
(2gi~i))2 ] because Igjl ::; K.
n.s.
27
Classes of Functions
When the logarithm of the covering number is less than n8 2/256K 2, the inequality IP{IIP~II > i81~}
: :; 2 exp[log N 1(-k8, Pn , $') -
n8 2 /128K 2 ]
will serve us well; otherwise use the trivial upper bound of 1. Integrate out. IP{IIP~II > i8} :::;; 2 exp( - n8 2/256K 2) + IP{log Ni (-k8, P , $') > n8 2/256K2}. n
Both terms on the right-hand side of the inequality converge to zero.
0
For some classes of functions the conditions of the theorem are easily met because N 1(8, P n , $') remains bounded for each fixed 8> O. This happens if the graphs of the functions in $' form a polynomial class of sets. The graph of a real-valued function f on a set S is defined as the subset GJ = {Cs, t): 0:::;; t :::;; f(s)
or
f(s):::;; t:::;; O}
of S ® JR. We learn something about the covering numbers of a class$' by observing how its graphs pick out sets of points in S ® JR. 25 Approximation Lemma. Let$' be a class of functions on a set S with envelope F, and let Q be a probability measure on S with 0 < QF < 00. lithe graphs of functions in $' form a polynomial class of sets then
N 1(8QF, Q, $') :::;; A8- W
for
0<
8
< 1,
where the constants A and W depend upon only the discriminating polynomial of the class of graphs. PROOF.
Let fl' ... ,fm be a maximal collection of functions in $' for which
Q I!; - .fj I > 8QF
if i =P j.
Maximality means that no larger collection has the same property; each f must lie within 8QF of at least one.fj. Thus m;::: N 1(8QF, Q, $'). Choose independent points (Si' t 1 ), ... , (Sk, t k ) in S ® JR by a two-step procedure. First sample Si from the distribution Q(. F)!Q(F) on S. Given Si> sample ti from the conditional distribution Uniform [ - F(Si), F(Si)]. The value of k, which depends on m and 8, will be specified soon. F
28
H. Uniform Convergence of Empirical Measures
The graphs Gland G 2, corresponding to 11 and 12, pick out the same subset from this sample if and only if everyone of the k points lands outside the region G 1 1:::,. G2' This occurs with probability equal to k
IT
[1 - IPIP{(Si' ti) E G1 1:::,. G2lsi}] = [1 - IP(II1(sl) - I2(sl)I/2F(sj))]k
i= 1
= [1 - Q I11 - I21/2Q(F)]k ::; (1 - !S)k ::; exp( -
! ks ).
Apply the same reasoning to each of the G') possible pairs of functions j; and fj. The probability that at least one pair of graphs picks out the same set of points from the k sample is less than (;) exp( - !ks) ::;
! exp(2 log m -
!ks).
Choose k to be the smallest value that makes the upper bound strictly less than 1. Certainly k ::; (1 + 4 log m)/s. With positive probability the graphs all pick different subsets from the k sample; there exists a set of k points in S ® IR from which the polynomial class of graphs can pick out m distinct subsets. From the defining property of polynomial classes, there exist constants B and V such that m ::; Bk v for all k ;::::: 1. Find no so that (1 + 4 log nt ::; n 1 / 2 for all n ;::::: no. Then either m < no or m::; Bm 1 / 2 s- V • Set W = 2Vand A = max(B2, no). D To show that a class of graphs has only polynomial discrimination we can call upon the results of Section 4. We build up the graphs as finite unions and intersections (Lemma 15) of simpler classes of sets. We establish their discrimination properties by direct geometric argument (as for intervals and quadrants) or by exploitation of finite dimensionality (as in Lemma 18) of a generating class of functions.
26 Example. Define a center of location for a distribution P on IR m as any value 8 minimizing the criterion function
H(B, P) = PcP(lx -
(1),
where cP(·) is a continuous, non-decreasing function on [0, (0) and 1·1 denotes the usual euclidean distance. If PcP( Ix I) < 00 and cPO does not increase too rapidly, in the sense that there exists a constant C for which cP(2t) ::; CcP(t) for all t, then the function H(·, P) is well defined:
H(8, P) ::; P[cP(2181){lxl ::;
181} + CcP(lxl){lxl > 181}] <
If trivial cases are ruled out by the requirement (27)
P {x: cP( Ix I) < cP( 00 -
)}
> 0,
00.
29
II.5. Classes of Functions
the minimizing value will be achieved (Problem 21); extra regularity conditions on P, which are satisfied by distributions such as the multivariate normal, ensure uniqueness (Problem 22). For this example, let us not get bogged down by the exact conditions needed; just assume that H(·, P) has a unique minimum at some (}o. Estimate (}o by any value (}n that minimizes the sample criterion function H(·, Pn). To show that (}n converges to (}o almost surely, it will suffice to prove that H((}n, P) ~ H((}o, P) almost surely, because H(B, P) is bounded away from H((}o, P) outside each neighborhood of (}o. The argument follows the same pattern as for k-means (Example 4). First show that (}n eventually stays within a large compact ball {I x I s K}. Choose the K greater than I(}o I and large enough to ensure that
s
tK} > P
which is possible by (27): as K tends to infinity the left-hand side converges to
s
tK}
for every (} with I(} I > K. If we prove uniform almost sure convergence of P n to P over the class ff
= {
then we can deduce almost surely that H((}n, P)
~
H((}o, P) from
H((}n, P n) - H((}n, P)
~
0,
H((}n, P n) S H((}o, Pn)
~
H((}o, P) S H((}n, P).
Here's our chance to apply Theorem 24. The class ff has envelope
f3. x +
ys
+ f>
with parameters f3 in and y, f> in IR. By Lemma 18, the sets {g ~ O}, for gin '§, pick out only a polynomial number of subsets from {Zi}; those sets corresponding to functions in '§ with f3 = - 2(}, y = 1, and f> = I(} 12 pick out even fewer subsets from {z;}. The graphs of functions
30
11. Uniform Convergence of Empirical Measures
Buried within the argument of the last example lies a mini lemma relating finite dimensionality of a class of functions to discrimination properties of the graphs. It is perhaps worth noting.
28 Lemma. Let fF be a finite-dimensional vector space of real functions on S. The class of graphs of functions in fF has polynomial discrimination. PROOF. Define on S
® IR the vector space '1J of real functions gfjs, t)
= f(s) - rt.
Define
G1 = {gf,l :;::: O} = {Cs, t):f(s):;::: t} G2 = {g-f,-l :;::: O} = {Cs, t):f(s)::; t} Observe that Gf
= G1{t:;::: O}
u G2 {t ::; O}.
o
Invoke Lemmas 18 and 15.
29 Example. Now let's have another try at the k-means problem introduced in Example 4. There we met the class of functions of the form fa,b(X) = Ix - al 2 /\ Ix - bl 2 with (a, b) ranging over the subset C of IR 2 • We know that SUPe fa, b ::; F for anFwithPF < 00,providedPlxl 2 < 00. The graphs of functions Ix - a 12 form a class with polynomial discrimination, by Lemma 28. Intersect pairs of such graphs in all possible ways to get the graphs of all functionsfa,b' Apply Lemma 15 (to handle the intersections), then the Approximation Lemma (to bound covering numbers), then Theorem 24: sup e
iPn fa, b -
Pfa,bl-4 0 almost surely.
Compare this with the direct approximation argument of Example 4.
0
II.6. Rates of Convergence Theorem 24 imposed the condition log N I (a, P n , fF) = oin) on the rate of growth of the covering numbers. Many classes meet the condition easily. For example, if the graphs of functions from fF have only polynomial discrimination, the covering numbers stay smaller than a fixed polynomial in a - 1. The method of proof will deliver a finer result for such a class; we can get good bounds not just for a fixed a deviation but also for an an that decreases to zero as n increases. That is, we get a rate of convergence for the
31
11.6. Rates of Convergence
uniform strong law of large numbers. The method will also allow the class of functions to change with n, provided the covering numbers do not grow too rapidly. If the classes are uniformly bounded, and if the supremum of Pf 2 over the nth class tends to zero as n increases, this will speed the rate of convergence. Consider the effect upon the two key steps of the argument for Theorem 24 if we let both Band $' depend on n. As before, replace P n - P by the· signed measure P~ that places mass ± n - 1 at each of ~ 1> ••• , ~n' The symmetrization inequality
IP{s~: IPnf -
(30)
Pfl > 8Bn} ::::;
4IP{S~: IP~fl > 2Bn}
still holds provided var(Pn f)/(4Bn)2 ::::; ! for eachfin ~. The approximation argument and Hoeffding's Inequality still lead to (31)
IP{s~: IP~fl > 2Bnl;}::::; 2N (Bn,Pn, $'n) ex p [ -!nB;/(m~xPngJ)l 1
where the maximum runs over all N 1 (B n, Pn, $'n) functions {gJ in the approximating class. If the supremum over $'n of Pf2 tends to zero, one might expect that the maximum over the {PngJ} should converge to zero at about the same rate. The next lemma will help us make th~ idea precise if the approximating {gj} are chosen from fi'". As squares of functions are involved, covering numbers need to be calculated using 22 seminorms rather than the 21 seminorms of Definition 23. 32 Definition. Let Q be a probability measure on Sand $' be a class of functions in 2 2(Q). For each B > 0 define the covering number NiB, Q, $') as the smallest value of m for which there exist functions gl' ... , gm (not necessarily in $') such that minj(Q(f - g)2)1/2 ::::; B for each f in $'. For 0 definiteness set N 2(B, Q, $') = 00 if no such m exists.
As before, if $' has envelope F we can require that 1gj 1 ::::; F; and we could require gj to belong to ff, at the cost of a doubling of B, by substituting for gj an ij in $' such that (Q(ij - g)2)1/2 ::::; B. 33 Lemma. Let $' be a permissible class of functions with 1 f (PF)1/2 ::::; <5 for eachfin fi'. Then
IP{S~P(PnF)1/2 > 8<5}::::; 4IP[N 2(<5, P
n,
$') exp( -n<5 2 )
/\
1 ::::;
1 and
1].
Let P~ be an independent copy of P n. Write Z(f) for (Pnf2)1/2 and Z'(f) for (P~f2)1/2. From the Symmetrization Lemma of Section 3,
PROOF.
(34)
IP{s~p IZ(f)1 > 8<5}::::; j-IP{s~p IZ(f) -
Z'(f)1 > M}
32
n. Uniform Convergence of Empirical Measures
because, for each 1 in ff,
The intrusion of the square-root into the definition of Z and Z' would complicate reduction to the P~ process. Instead, construct P n and P~ by a method that guarantees equal numbers of observations for both empirical measures. Sample 2n observations XI, ... , X 2n from P. Independently of the vector X of these observations, generate independent selection variables Tb"" Tn with IP{T(i) = 1} = IP{T(i) = O} = ! . Use these to choose one observation from each of the pairs (X 2i-l, X 2i), for i = 1, ., ., n. Construct P n from these observations, and P~ from the remaining observations. Formally, set ~i=X2i-l+t(i) and ~;=X2i-t(i)' then put mass n- 1 on each point ~i for Pn, and put mass n- 1 on each ~; for P~. Set S2n = !(Pn + P~). It has the same distribution as P 2n' Temporarily write p(.) for the :£2(S2n) seminorm: p(f) = (S2"12)1/2. Given X, find functions gb ... , gM, where M = N 2 (j2b, S2n, g;), for which min p(f - g) ::;; j2b
for every f in g;.
j
We may assume that Igjl ::;; 1 for every j. The awkward j2 will disappear at the end when we convert to :£2(Pn) covering numbers. From the triangle inequality for the ff'2(P n) seminorm, and the bound 2S 2n for P n' deduce for each 1 and 9 that IZ(f) - Z(g) 1 ::;; Z(f - g)::;; (2S 2'(1 - g)2)1/2
=
j2p(f - g)
and similarly for Z'. For 1 in g; set 9 equal to the gj that minimizes p(f - g). Then IZ(f) - Z'(f) I ::;; Z(f - g) + IZ(g) - Z'(g) I + Z'(gj - f) ::;; 4b + IZ(gj) - Z'(g)l, whence
IP{s~PIZ(f) -
Z'(f) I >
MIX}::;; IP{m:xIZ(9) -
2'(g) I > 2b 1
X}
::;; MmaxIP{IZ(gj) - Z'(gj) 1 > 2bIX}. j
Fix a 9 with Igl ::;; 1. Bound IZ(g) - Z'(g) 1 by IZ(g? - Z'(g)21/[Z(g) which is less than
+ Z'(g)J
33
II.6. Rates of Convergence
thanks to the inequality a 1/2 + b 1/2 ~ (a Hoeffding's Inequality (Appendix B).
+ b)1/2,
for a, b ~ O. Apply
IP{IZ(g) - Z'(g) 1 > 2<5I X }
J1 ±
~ IP{ I
[g2(X 2i-1) - g2(X 2JJ I > 2n<5(2S2ng2)1/21
~ 2 exp [ -16n2<52S2ng2lt1 4[g2(X 2i-1) ~
1
I
g2(X 2i)J2 ]
2 exp( - 2n<5 2)
because the inequality g n
X}
1
~
1 implies n
[g2(X 2i _ 1) - g2(X2JJ 2 ~ Ig 2(X 2i - 1)
+ g2(X2J
=' 2nS2ng 2.
i= 1
i= 1
Notice how the S2ng 2 factor cancelled. That happened because we symmetrized Z instead of P n' Setting g equal to each gj in turn, we end up with W
{s~p
1
Z(f) - Z'(f) 1 >
MIX} ~ 2N (j2<5 , S2n, g;) exp( - 2n<5 2) 2
Decrease the right-hand side to the trivial upper bound of 1, if necessary, then average out over X: (35)
IP{s~PIZ(f) -
Z'(f) > M} 1
~ IP[2Nzej2<5, S2n, g;)exp(-2n<5
2
) /\
1J
The presence of the S2n is aesthetically unpleasing, especially since both <5 and g; will always depend on n in applications. Problem 24 allows us to replace it by Pn' at a small cost: IP[2N 2 (j2<5, S2n, g;) exp( -2n<5 2) /\ 1J ~ IP[2N 2 (<5, P n, g;)N 2 (<5, P~, g;) exp( -2n<5 2) /\ 1J ~ IP[2N 2(<5,p n,g;)exp(-n<5 2) /\ 1J + IP[N (<5, P~, g;) exp( -n<5 2) /\ 1J 2
by virtue of the inequality xy /\ 1 ~ (x /\ 1) + (y /\ 1) for x, y ~ O. The empirical measures P n and P~ have the same distribution; the sum of expectations is less than 3IP[N 2 (<5, Pn, g;) exp( -n<5 2) /\ 1]. Combine the last bound with (34) and (35) to complete the proof.
D
The bounds we have for .:e 1 covering numbers can be converted into bounds for .:e 2 covering numbers. For the sake of future reference, consider
34
11. Uniform Convergence of Empirical Measures
a class ff with envelope F. Set F equal to a constant to recover theinequalities for a uniformly bounded ff.
36 Lemma. Let ff be a class offunctions with strictly positive envelope F, and Q be a probability measure with QF 2 < 00. Define P(·) = Q(·F 2)/Q(F 2) and '§ = {f /F: f E ff}. Then: (i) N i6(QFZ)1/2, Q, ff) ::; N i6, P, '§) ::; N 1(i6 Z , P, '§); (ii) if the class of graphs offunctions in ff has only polynomial discrimination then there exist constants A and W, not depending on Q and F, such that Nz(6(QFZ)1/Z, Q, ff)::; A6- W for 0 < 6 ::; 1. PROOF.
For every pair of functions f1' f2 with If1 I ::; F and Ifzl ::; F, (QF Z)-lQlf1 - fzl2 = PlfdF - fz/Fl z ::; 2PlfdF - fz/FI·
Assertion (i) follows from these connections between the seminorms used in the definitions of the covering numbers. The graph of f covers a point (x, t) if and only if the graph off /F covers (x, t/F(x)); the graphs offunctions in '§ also have polynomial discrimination. Invoke the Approximation Lemma (11.25) for classes with envelope 1. N 1 (i6 Z , P, '§) ::; A(i6 Z ) - w.
Rechristen A2 W as A and 2W as W.
D
It is now an easy matter to prove rate of convergence theorems for uniformly bounded classes of functions. As an example, here is a result tailored to classes whose graphs have polynomial discrimination. (Remember that the notation Xn ~ Yn means xn/Yn ...... 00.)
37 Theorem. For each n, let covering numbers satisfy
ff"
be a permissible class of functions whose
sup N 1(8, Q, ff,,) ::; A8- W
for
0<
8
< 1
Q
with constants A and W not depending on n. Let {lXn} be a non-increasing sequence of positive numbers for which n6;1X; ~ log n. If If I ::; 1 and (PP)l/Z ::; 6n for each f in ffn' then sup IPnf - Pfl ~ 6;lXn
almost surely,
:lJ'n
PROOF.
Fix
8
> O. Set 8n = 86;lX n. Because var(Pn f)/(48 n)Z ::;
(16n8Z6;1X~)-1 ~
(log n)-l
the symmetrization inequality (30) 'holds for all n large enough:
IP{s~: IPnf -
Pfl > 88n }::;
4IP{S~: IP~fl > 28
n}
35
11.6. Rates of Convergence
Condition on ~. Find approximating functions {gj} as in (31). We may assume that each gj belongs to fF". (More formally, we could replace gj by an jj in :#in for which P nIjj - gj I ::; 8, then replace 8 by 28 throughout the argument.) From (31),
JP{s~: IP~fl > 28n}::; 2A8;w exp( -!n8~/646~) + JP{s~: Pnf2 > 646~} The first term on the right-hand side equals 2Ae- w exp[W log(1/6;an)
-
n8 2 6;a;/128],
which decreases faster than every power of n because log(1/6;an) increases more slowly than log n, while n6;a; increases faster than log n. Lemma 33 bounds the second term by 4A(86;a n )-W exp( -n6;)
which converges to zero even faster than the first term. An application of the Borel-Cantelli lemma completes the proof. D Specialized to the case of constant {an}, the constraint placed on {6 n} by Theorem 37 becomes 6; ~ n- 1 log n. This particular rate pops up in many limit theorems involving smoothing of the empirical measure because (Problem 25) it corresponds to the size of the largest ball containing no sample points. 38 Example. Let P be a probability measure on lRd having a bounded density p(.) with respect to d-dimensionallebesgue measure. One theoretically attractive method for estimating p(.) is kernel smoothing: convolution of the empirical measure Pn with a convenient density function to smear out the point masses into a continuous distribution. The estimate is
Pn(x) = Pna-dK x ""
where Kx,.,.(Y) = K[a-1(y - x)]
for some density function K on lR d and a scaling factor a that depends on n, Note that the a- d is not part of Kx,.,.. The traditional method for analyzing Pn compares it with the corresponding smoothed form of p, p(x) = JPpnCx) = Pa-dK x ,.,..
The difference Pn - P breaks into a sum of a random component, Pn - p, and a bias component p - p. The smaller the value of a, the smaller the bias (Problem 26); the slower a tends to zero, the faster Pn - Pconverges to zero. These two conflicting demands determine the rate at which Pn - P can tend to zero.
36
11. Uniform Convergence of Empirical Measures
If Pn - P is to converge uniformly to zero, we must not allow (J to decrease too fast. Otherwise we might somewhere be smoothing P n over a region where it puts too little mass, making IPn - P I too large. Theorem 37 wi1llet cl decrease almost as fast as n - 1 log n, the best rate possible. For concreteness take K to be the standard normal density, which enjoys the uniform bound 0 :s:; Kx,(J :s:; 1 for all x and (J (the constant (2n)-d/2 is too awkward for repeated use.) The class of graphs of all Kx, (J functions has polynomial discrimination (Problem 28 proves this for a whole class of kernel functions). Assume also that the density p(.) is bounded, say 0 :s:; pO :s:; 1. In that case
because
f
K((y - x)/(J)p(y) dy
= (Jd
f K(t)p(x + (Jt) dt.
=
Everything is set up for Theorem 37. Put (Xn = 1 and 6; = (Jd. Provided ~ n-1log n,
cl
sup IPnKx,(J - PKx,(J1 ~ (Jd
almost surely,
x
that is, sup IPn(x) - p(x) I ~ 0 almost surely. x
Smoothness properties of p(.) determine the rate at which the bias term converges to zero (Problem 26). For example, one bounded derivative would give maximum bias of order O((J) in one dimension. We would then want something like (J3 ~ n- 1 log n to get a comparable rate of convergence 0 from Theorem 37 for Pn - p.
NOTES
Uniform strong laws oflarge numbers have a long history, which is described in the first section of the survey paper by Gaenssler and Stute (1979). Theorem 2 comes from DeHardt (1971), but the idea behind it is much older. Billingsley and Tops0e (1967) and Tops0e (1970, Sections 12 to 15) developed much deeper results for the closely related area of uniformity in weak convergence. Hartigan (1975) is a good source for information about clustering. Hartigan (1978) and Pollard (1981b, 1982b, 1982c) have proved limit
Notes
37
theorems for k-means. Engineers know the method of k-means by the name quantization. The March 1982 IEEE Transactions on Information Theory was devoted to the topic. Denby and Martin (1979) proposed the generalized M -estimator of Example 5. It has long been appreciated that comparison oftwo independent empirical distribution functions transforms readily into a combinatorial problem. Gnedenko (1968, Section 68), for example, reduced the analysis of twosample Smirnov statistics to a random walk problem. The method in the text has evolved from the ideas introduced by Vapnik and Cervonenkis (1971). Their method of conditioning turned calculations for a single fixed set into an application of an exponential bound for hypergeometric tail probabilities. Classes with polynomial discrimination are often called VC classes in the literature. The symmetrization described in Section 3 is a wellknown method for proving central limit theorems in Banach spaces (Araujo and Gine 1980, Section 2; Gine and Zinn 1984, Section 1). Steele (1975,1978) discovered subadditivity properties of empirical measures that strengthen the Vapnik-Cervonenkis convergence in probability results to necessary and sufficient conditions for almost sure convergence. Pollard (1981 b) introduced the martingale tools and the randomization method described in Problem 12 to rederive Steele's conditions. Theorem 21 and Example 22 are based on Steele (1978); Flemming Tops0e explained to me the ~-trick for convex sets. See Gaenssler and Stute (1979) for more about the history of this example. The proof of Theorem 16, which is often called the Vapnik-Cervonenkis lemma, is adapted from Steele (1975). Sauer (1970) was the first to publish the inequality in precisely this form, although he suggested that Shelah had also established the result. (1 am unable to follow the two papers of Shelah that Sauer cited.) Vapnik and Cervonenkis (1971) proved an insignificantly weaker version of the inequality. Dudley (1978, Section 7) has dug further into the history. Lemma 18 is due to Steele (1975) and Dudley (1978). The sum of binomial coefficients in Theorem 16 and the randomization method of Problem 12 suggest that a direct probabilistic path might lead to the necessary and sufficient conditions of Theorem 21. Does there exist a set of n independent experiments that can be performed to decide whether a particular subset of a collection of n points can be picked out by a particular class of sets? Or maybe the experiments could be linked in some way. For a class that picks out only subsets with fewer than V points the solution is easy-it lies buried within the proof of Theorem 16. The notes to Chapter VII will give more background to the concept of covering number. Vapnik and Cervonenkis (1981) have found necessary and sufficient conditions for uniform almost sure convergence over bounded classes of functions. They worked with fr"" and 2'1 distances between functions. Gine and Zinn (1984) applied chaining inequalities and gaussian-process methods (see Chapter VII) to deduce 2'2 necessary and sufficient conditions. The square-root trick in Lemma 33 comes from Le Cam (1983) via Gine and
38
II. Uniform Convergence of Empirical Measures
Zinn. Kolchinsky (1982) and Pollard (1982c) independently introduced the symmetrization method used in Lemma 33. The Approximation Lemma is due to Dudley (1978), who proved it for classes of sets. The extension to classes of functions was proved ad hoc by Pollard (1982d), using an idea of Le Cam (1983). The density estimation literature is enormous. Silverman (1978) and Stute (1982b) have found sharp results involving the n -1 log n critical rate. Bertrand-Retali (1974, 1978) proved that (id ~ n -1 log n is both necessary and sufficient for uniform consistency over the class of all uniformly continuous densities on IRd. Universal separability was mentioned in passing by Dudley (1978) as a way of avoiding measurability difficulties. Most of the results in the chapter can be extended to independent, nonidentically distributed observations (Alexander 1984a). PROBLEMS
[1] In Example 4 relax the assumption that W(·, ., P) has a unique minimum; assume the function achieves its minimum for each (a, b) in a region D. Prove that the distance from the optimal (an, bn) to D converges to zero almost surely, provided P does not concentrate at a single point. [The condition rules out the possibility of a minimizing pair for W(·, " P) with one center off at infinity.] [2] Here is an example of an ad hoc method to force an optimal solution into a restricted region. Suppose an estimator corresponds to the fn that minimizes Pnf over a class !F, and that we want to force fn into a region %. Write 1'0 for the infimum, assumed finite, of Pf over iF. Suppose there exists a positive function bO on :F such that, for some 8 > 0, b(.f) ~ 1'0
+8
p[:~~ f Ib(.f) ]
for f in :F\%
> 11'0 I/( I1'0 I + 8).
Show that liminf inf.?"\.lf Pnf > 1'0 almost surely. [Trivial if 1'0 < 0.] Deduce that!" belongs to % eventually (almost surely). Now read the case A consistency proof of Huber (1967). Compare the last part of his argument with our Theorem 3. [3] Call a class:F offunctions universally separable if there exists a countable subclass ~ such that eachfin :F can be written as a pointwise limit of a sequence in~. If :F has an envelope F for which PF < 00, prove that universal separability implies measurability of lIPn - PII. [4] For any finite-dimensional vector space '§ of real functions on S, the class ~ of sets of the form {g ~ O}, for g in '§, is universally separable. [Express each g in '§ as a linear combination of some fixed finite collection of non-negative functions. Let '§O be the countable subclass generated by taking rational coefficients. Fqr each gin '§ there exists a sequence {gn} in '§O for which gn 1 g. Show that {gn ;::: O} 1 {g ;::: O} pointwise.] [5] The operations in Lemma 15 preserve universal separability.
Problems
39
[6J For a universally separable class 2&, the quantity v" defined in Theorem 21 is unchanged if 2& is replaced by its countable subclass 2&0' Prove that v" is measurable. [7J Prove that the class of indicator functions of closed, convex subsets of IRd is universally separable. [Consider convex hulls of finite sets of points with rational coordinates.J [8J Theorem 16 informs us that the class of quadrants in IR2 picks out at most 1 + tN + tN 2 subsets from any collection of N points. Find a configuration for which this bound is achieved. [9J For N ~ 2 the sum of binomial coefficients singled out by Theorem 16 is bounded by NV. [Count subsets of {I, ... , N} containing fewer than V elements by arranging each subset into increasing order then padding it out with enough copies of the largest element to bring it up to a V-tuple. Don't forget the empty set.J [lOJ Let M be a subset of [0, IJ that has inner lebesgue measure zero and outer lebesgue measure one (Halmos 1969, Section 16). Define the probability measure fl as the trace of lebesgue measure on M (the measure defined in Theorem A of Halmos (1969), Section 17). Assuming the validity of the continuum hypothesis, put M into a one-to-one correspondence with the space [0, U) of all ordinals less than the first uncountable ordinal U (Kelley 1955, Chapter 0). Define 2& as the class of subsets of [0, IJ corresponding to the initial segments [0, xJ in [0, U). (a) Show that 2& has linear discrimination. [It shatters no two-point set.J (b) Equip MOO with its product (j-field and product measure floo. Generate observations ~ I' ~2' ... on P = Uniform(O, 1) by taking them as the coordinate projection maps on MOO. Construct empirical measures {Pn} from these observations. Show that sUPii! IPn D - PD I is identically one. (c) Repeat the construction with the same 2&, but replace (MOO, floo) by a countable product of copies of MC equipped with the product measure A00, where A equals the trace of lebesgue measure on MC. This time sUPii! IPnD - PDI is identically zero. [Funny things can happen when 2& has measurability problems. Argument adapted from Pollard (1981a) and Durst and Dudley (1981).J [l1J For independent and identically distributed random elements {O, write gn for the (j-field generated by all symmetric functions of ~I"'" ~N as N ranges over n, n + 1, .... For a fixed function f, apply the usual reversed martingale argument (Ash 1972, page 310) to show that lP(Pnflgn+l ) = Pn+d If P(sup,? If I) < 00, deduce
for every class of functions g; that makes both suprema measurable. [12J Here is one way to prove necessity in Theorem 21. Suppose liPn - PII ---+ 0 almost surely. Construct fl: by placing mass n - I at each ~i for which the sign variable (ji equals + 1; construct fl;; similarly from the remaining ~/s. Notice that P~ = fl: - fl;;· Let N be the number of sign variables (j I, ... , (jn equal to + 1. (a) Prove that (n/N)fln+ has the same distributions as P N • [What if N = O?J (b) Deduce that both Ilfl: - tP11 ---+ 0 and Ilfl;; - tPII ---+ 0, in probability.
40
n. Uniform Convergence of Empirical Measures (c) Deduce that Ilfl: - fl;; II -> 0 in probability. (d) Suppose PJ shatters a set F consisting of at least n1J of the points ~1"'" ~n' Without loss of generality, at least tn1J of the points in F are allocated to fl:. Choose a D to pick out just those points from F. Use independence properties of the {O';} to show that, with high probability, fl;;(D\F) and fl;;(D\F) are nearly equal. [Argue conditionally on Pn and the O'i for those ~i in F.] (e) Show that fl;;(D) - fl;;(D) ~ t1J with high conditional probability. This contradicts (c).
[13] Rederive the uniform strong law for convex sets (Example 22) by the direct approximation method of Theorem 2. [14] Let ff be a permissible class with natural envelope F = sup&>" II I. If IlPn - PII -> 0 almost surely and if sup&>" IPII < 00 then PF < 00. [The condition on sup&>" IPII excludes trivial cases such as ff consisting of all constant functions. From IlPn - PII < e and IlPn-l - PII < e deduce n-ll/(~n) - PII < 2e; almost sure convergence implies
lP{s~ 1/(~n) -
PII
~n
infinitely often} = O.
Invoke the non-trivial half of the Borel-Cantelli lemma, then replace each ~1 to get 00
>
lP(sulI/(~l) -
PII)
~ IPF(~l) -
~n
by
constant.
Noted by Gine and Zinn (1984).] [15] Here is an example of how Theorem 24 can go wrong if the envelope F has PF = 00. Let P be the Uniform(O, 1) distribution and let ff be the countable class consisting of the sequence {fi}, where J;(x) = x-z{(i + 1)-1 ::;; X < i- 1 }. Show that the graphs have polynomial discrimination and that Ph = 1 for every i. But SUPi Pnh -> 00 almost surely. [Find an exn with nex; -> 0, such that [0, exn] contains at least one observation, for n large enough.] [16] Let ff be the class of all monotone increasing functions on IR taking values in the range [0, 1]. The class of graphs does not have polynomial discrimination, but it does satisfy the conditions of Theorem 24 for every P. [If {x;} and {tJ are strictly increasing sequences, the graphs can shatter the set of points (Xl' t 1), ... , (x N, tN)'] [17] For the ff of the previous problem, rewrite Pnl as g PnU ~ t} dt. Deduce uniform almost sure convergence from the classical result for intervals. [Suggested by Peter Gaenssler.] [18] Let ff and ':§ be classes of functions on S with envelopes F and G. Write :7 for the class of all sums I + g with I in ff and g in ':§. Prove that
N;(bQ(F
+ G), Q, ,9»
::;;
N;CbQF, Q, ff)N;CbQG, Q,
':§)
for
i = 1,2.
[19] A condition involving only covering numbers for P would not be enough to give a uniform strong law of large numbers. Let P be Uniform(O, 1). Let PJ consist of all sets that are unions of at most n intervals each with length less than n- z, for n = 1,2, .... Show that sup§) IPnD - PDI = 1, even though N 1 (e, P, PJ) < 00 for each e > O.
41
Problems [20] Deduce Theorem 2 from Theorem 24.
[21] Under the conditions set down in Example 26, the function H(·, P) achieves its minimum. [If H(el> P) converges to the infimum as i -+ 00, use Fatou's lemma to show that the infimum is achieved at a cluster point of {eJ; condition (27) rules out cluster points at infinity.] Notice that only left continuity of cP is needed for the proof. Find and overcome the extra complications in the argument that would be caused if cp were only left-continuous. [22] This problem assumes familiarity with convexity methods, as described in Section 4.2 of Tong (1980). Suppose that the distribution P of Example 26 has a density p(.) whose high level setsD, = {p ;::: t} are convex and symmetric about the origin. Prove that H(e, P) has a minimum at e = O. [By Fubini's theorem, H(e, P) =
=
Iff ff
{O :::; s :::; cp(lx - el)}{O :::; t :::; p(x)} ds dt dx
volume[B(e, IX(S»" n D,] ds dt,
where B(e, r) denotes the closed ball of radius r centered at e. The volume of B(e, r) n D, is maximized at e = 0.] When is the minimum unique? Show that a multivariate normal with zero means and non-singular variance matrix satisfies the condition for uniqueness. [23J Suppose K} are independent, but that the distribution of ~;, call it Q;, changes with i. Write pIn) for the average distribution of the first n observations, pIn) = n -I(QI + ... + Qn). Show for a permissible polynomial class !!fi that sup IPnD - p(n)DI-+ 0
almost surely.
!'J
What difficulties occur in the extension to more general classes of sets, or functions? [Adapt the double-sample symmetrization method of Lemma 33: sample a pair (X Zi -I, X Zi) from Qi; use the selection variable Ti to choose which member of the pair is allocated to Pn , and which to P~.J [24J Show that
Nz(fib, tcQI
+ Qz), $'):::;
Nz(b, QI, $')Nz{b, Qz, $').
[Let hi be the density of Ql with respect to QI functions gl {hi> t} + gz{h 1 :::; t}.]
+ Qz. Consider the approximating
[25] Let P be the uniform distribution on [0, 1]Z. For a sample of n independent observations on P show that IP{some square of area
IXn
contains no observations} -+ 1
if IXn is just slightly smaller than n- I log n. [Break [0, I]Z into N subsquares each with area slightly less than n- I log n. Set Ai = {ith subsquare contains at least one observation}. Show that IP(Ai+ IIAI n··· n Ai) :::; IPA i +1' The probability that each of these subsquares contains at least one point is less than (IP All. BertrandRetali (1978).]
42
11. Uniform Convergence of Empirical Measures
[26J In one dimension, write the bias term for the kernel density estimate as jj(x) - p(x) =
f
K(z)[p(x
+ O"z) -
p(x)J dz.
Suppose p has a bounded derivative, and that SIzIK(z) dz < 00. Show that the bias is of order 0(0"). Generalize to higher dimensions. [If p has higher-order smooth derivatives, and K is replaced by a function orthogonal to low degree polynomials, the bias can be made to depend only on higher powers of O".J [27J The graphs of translated kernels K x ,(1 have polynomial discrimination for any K on the real line with bounded variation. [Break K into a difference of monotone functions.J [28J Let K be a density on IRd of the form h( Ix I), where h(·) is a monotone decreasing function on [0, (0). Adapt the method of Example 26 to prove that the graphs of the functions K x ,(1 have polynomial discrimination. [29J Modify the density estimate of Example 38 for distributions on the real line by choosing K as a function of bounded variation for which S K(z) dz = 0 and SzK(z) dz = 1 and S IzK(z)I dz < 00. Replace Pn by qnCx) = 0"-2PnKx,(1' Show that IPqn(x) converges to the derivative of p. How fast can 0" tend to zero without destroying the almost sure uniform convergence supx Iqn(x) - IPqn(x)l-> O?
CHAPTER III
Convergence in Distribution in Euclidean Spaces ... which runs through some of the standard methods for proving convergence in distribution of sequences of random vectors, and for proving weak convergence of sequences of probability measures on euclidean spaces. These include: checking convergence for expectations of smooth functions of the random vectors; checking moment conditions for sums of independent random variables (the Central Limit Theorems); checking convergence of characteristic functions (the Continuity Theorems for characteristic functions); and reduction to analogous problems of almost sure convergence via quantile transformations.
IlL!. The Definition Convergence in distribution of a sequence {X n} of real random variables is traditionally defined to mean convergence of distribution functions at each continuity point of the limit distribution function: IP{Xn ::; x}
-+
IP{X ::; x}
whenever
IP{X = x} = O.
Although convenient for work with order statistics and quantiles, this definition does have some disadvantages. Distribution functions are not well suited to calculations involving sums of independent random variables. The simplest proofs of the Central Limit Theorem, for example, do not directly check pointwise convergence of distribution functions; they show that sequences of characteristic functions, or expectations of other smooth functions ofthe sums, converge. With the extensions to sequences ofrandom vectors (measurable maps into multidimensional euclidean space IR d), the difficulties multiply. And for random elements of more general spaces not equipped with a partial ordering, even the concept of distribution function disappears. With all this in mind, let us start afresh from an equivalent definition, which lends itself more readily to generalization. 1 Definition. A sequence of random vectors {Xn} is said to converge in distribution to a random vector X, written Xn ~ X, if IPf(Xn) -+ IPf(X) for every f belonging to the class ~(IRk) of all bounded, continuous, real functions on IRk. 0
This notion of convergence does not specify the limit random vector uniquely. If X and Y have the same distribution, that is, if IP{X E A} = IP{Y E A}
for each borel set A,
44
Ill. Convergence in Distribution in Euclidean Spaces
then Xn ~ X means the same as Xn ~ Y. This invites slight abuses of notation. For example, it is convenient to write Xn ~ N(O, 1), meaning that the sequence of real random variables {Xn} converges in distribution to any random variable having a standard normal distribution. The symbol N(O, 1) stands not only for a particular probability measure on the borel (J-field .?8(IR) but also for any random variable having that distribution. Similarly, we can avoid much circumlocution by writing, for example, n- 1 / 2 [Bin(n,
t) - tnJ
~
N(O, t),
instead of:
t
if Xn has a binomial distribution with parameters nand and X has a normal distribution with mean zero and variance t, then n-l/2(Xn - !n) ~ X.
In general, for probability measures on the borel (J-field .?8(IRk), define Pn ~ P to mean convergence in distribution for random vectors having these distributions. This definition is equivalent to the requirement: Pnf -4 Pf for every f in '?6(IRk). Most authors call this weak convergence.
111.2. The Continuous Mapping Theorem Suppose Xn ~ X, as random vectors taking values in IRk, and let H be a measurable map from IRk into IRs. Does it follow that HX n ~ HX? That is, does IPf(HX n) -4 IPf(HX) for every f in '?6(IRS)? If H were continuous, f H would belong to '?6(IRk) for every f in '?6(IRS). The result would be trivially true. The convergence HX n ~ HX also holds under a slightly weaker assumption: it suffices that H be continuous at almost all points of the range of X. This will follow as a simple corollary to the next lemma. 0
2 Convergence Lemma. Let h be a bounded, measurable, real-valued function on IRk, continuous at each point of a measurable set C.
(i) Let {Xn} be a sequence of random vectors converging in distribution to X. If IP{X E C} = 1, then IPh(Xn) -4 IPh(X). (ii) Let {P n} be a sequence of probability measures converging weakly to P. If p(e) = 1, then Pnh -4 Ph. As the two assertions are similar, we need only prove (ii). Consider any increasing sequence {I;} of bounded, continuous functions for which I; :::;; h everywhere and I; i h at each point of C. Accept for the moment that such a sequence exists. Then weak convergence of {Pn} to P implies that PROOF.
(3)
Pnl;
-4
PI;
for each fixed i.
45
IlI.2. The Continuous Mapping Theorem
On the left-hand side bound I; by h. liminf Pnh ;::: PI; for each fixed i. Invoke monotone convergence as i tends to infinity on the right-hand side. (4)
The companion inequality obtained by substituting -h for h combines with (4) to complete the proof. Now to construct the functions {I;}. They must be chosen from the family !F = {fE~(IRk):f ~ h}. If we can find a countable subfamily of!F, say {gb g2," .}, whose pointwise supremum equals h at each point of C, then setting I; = max{gb ... , gJ
will do the trick. Without loss of generality suppose h > 0. (A constant could be added to h to achieve this.) For each subset A of IRk define the distance function d(·, A) by d(x,A) = inf{lx - YI:YEA}. It is a continuous function of x, for each fixed A. For positive integral m and positive rational r define
fm,r(x) = r /\ md(x, {h ~ r}).
---I I h~/ ,1/
I
,J
/
/
.'
I
\
""..,.:~
I
I
I
I I
I
I I I I
I I
, I
~
I
- - 71 'r-"-.../ I
I
I I I I
I
\
----r
1\"
~ I \ fm., I
I
I
I
1
{
\
,
\
\
\
\
Each fm, r is bounded and continuous; it is at most r if h(x) > r; it takes the value zero if hex) ~ r: it belongs to !F. Given a point x in C and an s > 0, choose a positive rational number r with hex) - s < r < hex). Continuity of h at x keeps its value greater than r in some neighborhood of x. Consequently, d(x, {h ~ r}) > andfm,rCx) = r > hex) - s for all m large enough.
°
o
Weak convergence of {P n } was needed only to establish the convergence (3) for the functions {I;}. These functions were, however, not just continuous, but uniformly continuous. The functions from which they were constructed even satisfied a Lipschitz condition: Ifm,r(x) - fm,rCy) I ~ mix - yl·
46
Ill. Convergence in Distribution in Euc1idean Spaces
Thus the lemma could have been proved using only convergence of expectations of bounded, uniformly continuous functions of the random vectors. In particular, such a requirement would imply the convergence Pnh -+ Ph for each h in ~(IRk).
5 Corollary. If Pnf -+ Pf for every bounded, uniformly continuous f then P n ~ P. (And similarly for convergence in distribution of random vectors.) 0 The lemma also provides an answer to the question asked at the start of the section.
6 Continuous Mapping Theorem. Let H be a measurable mapping from IRk into IRs. Write C for the set of points in IRk at which H is continuous. If a sequence {Xn} of random vectors taking values in IRk converges in distribution to a random vector X for which IP{X E C} = 1, then HX n ~ HX. PROOF. For each fixed f in ~(IRS), the bounded function f at all points of C.
0
H is continuous D
Some authors seem to regard this result as trivial and obvious; they scarcely notice that, at least implicitly, they make use of it for many applications. It is better to recognize these covert appeals to the Continuous Mapping Theorem. That way the more general form of the result in Chapter IV will come as no surprise.
7 Example. If the real random variables {Xn} converge in distribution to X then IP{Xn ::; x} -+ IP{X ::; x} at each x for which IP{X = x} = o. That is, the sequence converges at each continuity point x of the distribution function of X. This holds because x is the only point of discontinuity of (the indicator function of) the set ( - 00, x]. Problem 1 shows you how to go the other way, from pointwise convergence of distribution functions to convergence in distribution of the random variables. The same result is true in higher dimensions if the inequalities X n ::; x and X ::; x are taken componentwise and (- 00, x] is interpreted as a multidimensional orthant with vertex at x. Continuity of the multidimensional distribution function of X at x requires that X lands on the boundary of (- 00, x] with zero probability. D 8 Example. Consider the multinomial distribution obtained by independent placement of n objects into k disjoint cells. Write Fn = (F nl , ... , F nk ) for the column vector of observed frequencies-cell i receives Fni objects-and p = (Pl' ... , Pk) for the column vector of cell probabilities. Pearson's chisquare statistic is k
Zn
=
L (F i~
1
ni -
np;)2/npi·
47
III.2. The Continuous Mapping Theorem
Write Zn as a function of a standardized column vector, by setting Xn = n- 1/2 (Fn - np) and Zn = X~L\-2Xn'
JPi
where L\ denotes the diagonal matrix with as its ith diagonal element. By the Multivariate Central Limit Theorem, which will be proved later in this chapter (Theorem 30), the random vectors {Xn} converge to a N(O, V) distribution whose variance matrix V has (i,j)th element Pi - pf if i = j and - PiPj otherwise. Manufacture a random vector with this limit distribution by applying a linear transformation to a column vector W of independent N(O, 1) random variables: Xn ~ L\(Ik - uu')W, where u denotes the unit column vector (jP;, ... , jp,,). The mapping H from IRk into IR defined by
1L\-l x I2
Hx = x'L\-2 x =
is continuous. Apply the Continuous Mapping Theorem. Zn
= HX n ~
HL\(h - uu')W
=
I(h -
uu')WI 2
,...., Xf-1'
because h - uu' represents the projection orthogonal to the unit vector u. (The squared length of the projection of W onto any (k - 1)-dimensional subspace has a chi-square distribution with k - 1 degrees of freedom.) D 9 Example. Suppose el, e2, ... are independent random variables each with a Uniform(O, 1) distribution. Neyman (1937) developed a goodness-of-fit test whose asymptotic properties depended on the behavior of the statistics
Gn = n-
1J1 [tl
1Cj (ei)T,
where 1Co, 1C 1, ... are given polynomials defined on [0, 1J, with the orthonormality property: (1 if i = j, Jo 1C;(Y)1C/Y) dy = if i #- j.
{I°
Explicitly,1Co(y) = 1,1C 1(Y) = ji2(y - t),1C 2(Y) = j5[6(y - t)2 so on. Define random column vectors Xi = (1C1(ei)' ... , 1Ck(ei))
for
The statistic Gn can then be written as n-1/2.±XiI2. I
,= 1
i = 1,2, ....
U and
48
Ill. Convergence in Distribution in Euclidean Spaces
The Multivariate Central Limit Theorem (Theorem 30) and the orthonormality properties of the polynomials ensure that
n-l/2(X 1
+ ... + Xn)
~
N(O, Ik).
The Continuous Mapping Theorem (applied to which map?) allows us to deduce that Gn ~ X~. Neyman used this to determine the approximate critical region for his test. D
111.3. Expectations of Smooth Functions There are two sorts of perturbation of a random vector X that don't affect the expectation IPf(X) of a smooth, bounded function of X too greatly: changes, however gross, that occur with only small probability; and changes that might occur with high probability but which alter X by only small amounts. These effects are easy to quantify when the smooth function f is uniformly continuous. Suppose If(x) - f(z) I < 8 whenever Ix - zl < b. Write 11 f 11 for the supremum of f(·). Then for any random vectors X and Y, whether dependent or not,
IIPf(X) - IPf(X +
(10)
Y) I
:::; IP{ I YI < b} If(X) - f(X + Y)I + IP{I YI ~ b}(lf(X) I + If(X + Y)I) :::; 8 + 211f11IP{ I YI ~ b}. The inequality lets us deduce convergence in distribution of a sequence of random vectors from convergence of slightly perturbed sequences. 11 Lemma. Let {X n}, X and Y be random vectors for which Xn X + u Y for each fixed positive u. Then X n ---> X.
+
uY ~
PROOF. Remember (Corollary 5) we have only to check that IPf(X n) ~ IPf(X) for each bounded, uniformly continuous j. Apply inequality (10) with X replaced by X nand Y replaced by u Y.
sup IIPf(Xn) - IPf(X n
+ uY)1
:::; 8 + 211f11IP{1 YI ~ bu- 1 }.
Similarly
IIPf(X) - IPf(X
+ uY)I:::; 8 + 211f11IP{IYI
~ bu- 1 }.
Choose u small enough to make both right-hand sides less than 28, then invoke the known convergence of {IPf(X n + uY)} to IPf(X + uY) to deduce that limsup IIPf(Xn) - IPf(X) I :::; 48. D Now, instead of thinking of the u Y as a perturbation of the random vectors, treat it as a means for smoothing the functionj. This can be arranged
49
111.3. Expectations of Smooth Functions
by choosing independently of X and the {Xn} a random vector Y having a smooth density function with respect to lebesgue measure-for convenience, take Y to have a N(O,I k ) distribution. Integrate out first with respect to the distribution of Y then with respect to the distribution of X. lPf(X
+ oT) =
lPf,rCX),
where fa(x) =
=
J
(2n)-k I2f(x + O'y) exp( _!IYI2) dy
J(2n0'2)-k I 2f(z) exp( -!Iz - x12j0'2) dz.
The function f has been smoothed by convolution. Dominated convergence justifies repeated differentiation under the last integral sign to prove that fa belongs to the class ~co(lRk) of all bounded real functions on lRk having bounded, continuous partial derivatives of all orders. 12 Theorem. If lPf(Xn) ---+ lPf(X) for every fin ~co(lRk) then Xn ~ X. PROOF. Convergence holds for every fa produced by convolution smoothing. 0 Apply Lemma 11.
For the remainder of the section assume that k = 1. That is, consider only real random variables. As the results of Section 5 will show, no great generality will be lost thereby-a trick with multidimensional characteristic functions will reduce problems of convergence of random vectors to their one-dimensional analogues. For expectations of smooth functions of X, the effect of small perturb ations can be expressed in terms of moments by applying Taylor's theorem. Suppose f belongs to ~CO(lRk). Then, ignoring the niceties of convergence, we can write f(x + y) = f(x) + yf'(x) + !J2j"(X) + ....
Suppose the random variable X is incremented by an independent amount Y. Then, again ignoring problems of convergence and finiteness, deduce (13)
lPf(X + Y) = lPf(X) + lP(Y)lPf'(X) + !lP(y2)lPj"(X) + ....
Try to mimic the effect of the increment Y by a different increment W, also independent of X. As long as lP(Y) = lP(W) and lP(y2) = lP(W2), the expectations lPf(X + Y) and lPf(X + W) should differ only by terms involving third or higher moments of Y and W. These higher-order terms should amount to very little provided both Y and Ware small; the effect of substituting W for Y should be small in that case. This method of substitution can be applied repeatedly for a random variable Z made up of a lot of little independent increments. We can replace
50
Ill. Convergence in Distribution in Euc1idean Spaces
the increments one after another by new independent random variables. If at each substitution we match up the first and second moments, as above, the overall effect on IPf(Z) should involve only a sum of quantities of third or higher order. In the next section this approach, with normally distributed replacement increments, will establish the Liapounoff and Lindeberg forms of the Central Limit Theorem. To make these approximation ideas more precise we need to bound the remainder terms in the informal expansion (13). Because only the first two moments of the increments are to be matched, a Taylor expansion to quadratic terms will suffice. Existence of third derivatives for fwill help to control the error terms. Assume f belongs to the class C6'3(IR) of all bounded real functions on IR having bounded continuous derivatives up to third order. Then the remainder term in the Taylor expansion (14) f(x + y) = f(x) + yf'(x) + h 21"(x) + R(x, y) can be expressed as R(x, y) = iJ31"'(X
+ 81y)
with 8 1 (dependingonxandy)betweenOand 1. Write 111"'11 forthesupremum of If"'(-) I· Then IR(x, y)1 :::;; i-II1"'lllyI3,
(15)
Set C equal to
HI'" 11· Then from (14) and (15),
IIPf(X +
Y) - IPf(X) - IP(Y)IPf'(X) - !IP(y2)IP1"(X) I
:::;; IPIR(X, Y)I :::;; CIP(IYI 3).
Apply the same argument with Y replaced by the increment W, which is also independent of X. Because IP(Y) = IP(W) and IP(y2) = IP(W2), when the resulting expansion for IPf(X + W) is subtracted from the expansion for IPf(X + Y) most of the terms cancel out, leaving, (16)
IIPf(X
+
Y) - IPf(X
+
W)I :::;; IPIR(X, Y)I + IPIR(X, W)I :::;; CIP(I Y1 3 ) + CIP(I WI 3 ).
This inequality is sharp enough for the proof of a limit theorem for sums of independent random variables with third moments.
III.4. The Central Limit Theorem A sum of a large number of small, independent random variables is approximately normally distributed-that is roughly what the Central Limit Theorem asserts. The rigorous formulations of the theorem set forth conditions for convergence in distribution of sums of independent random
51
IlI.4. The Central Limit Theorem
variables to a standard normal distribution. We shall prove two versions of the theorem. To begin with, consider a sum Z = (1 + ... + (k of independent random variables with finite third moments. Write a} for IP(J. Standardize, if necessary, to ensure that IP(j = for each j and ai + ... + at = 1. Independently of the {~), choose independent N(O, aJ)-distributed random variables {1]j}, forj = 1, ... , k. Start replacing the {(j} by the {1]j}, beginning at the right-hand end. Define
°
Sj
=
(1
+ ... + (j-1 + 1]j+1 + ... + 1]k'
Notice that Sk + (k = Z and that Sl + 1]1 has a N(O, 1) distribution. Choose a ~3(IR) function f, as in Section 3. Theorem 12 has shown that convergence for expectations of infinitely differentiable functions of random variables is enough to establish convergence in distribution; convergence for functions in ~3(IR) is more than enough. We need to show that IPf(Z) is close to IPf(N(O, 1)). Apply inequality (16) with X = Sj' Y = (j' and W = 1]j. Because Sj + (j = Sj+ 1 + 1]j+ 1 for j = 1, ... , k - 1, k
(17)
IIPf(Z) - IPf(N(O, 1))1:s;
L
IIPf(Sj
+0 -
IPf(Sj
+ 1])1
j= 1 k
:s;
L
IPIR(Sj' ()I
+ IPIR(Sj' 1])1
j= 1
:s; C
k
k
j= 1
j= 1
L IPI(jI3 + C L IPI1] I3. j
With this bound in hand, the proof of the first version of the Central Limit Theorem presents no difficulty.
18 Liapounoff Central Limit Theorem. For each n let Zn be a sum of independent random variables (nb (n2, ... , (nk(n) with zero means and variances that sum to one. If the LiapounojJ condition, k(n) (19) L IPI(nj 13 as n - 00, j= 1
is satisfied, then Zn
~
°
N(O, 1).
PROOF. Choose and fix an f in ~3(IR). Check that IPf(Zn) - IPf(N(O, 1)). The replacement normal random variables are denoted by 1]n1' ... , 1]nk(n)' The sum 1]n1 + ... + 1]nk(n) has a N(O, 1) distribution. Write a;j for the variance of (nj' and An for the sum on the left-hand side of (19). With subscripting n's attached, the bound (17) becomes k(n) IIPf(Zn) - IPf(N(O, 1))1 :s; CAn + C L a~jIPIN(O, 1W. j= 1
52
Ill. Convergence in Distribution in EucIidean Spaces
By Jensen's inequality, (J~j = (IP~;)3/2 : .: ; IP I~nj 13, which shows that the sum contributed by the normal increments is less than IP IN(O, 1WAn- Two calls upon (19) as n ~ 00 complete the proof. 0 The Liapounoff condition (19) imposes the unnecessary constraint of finite third moments upon the summands. Liapounoff himself was able to weaken this to a condition on the (2 + c5)th moments, for some c5 > o. The remainder term R(x, y) in the Taylor expansion (14) does not increase as fast as lyl2+o though: (20)
IR(x, y)1
= IU(x + y) - f(x) - yf'(x)J - !y 21"(x) I = Ih21"(x + 8 2 y) - h 21"(x) I : .: ; 1I1"IIIyl2
for all x and y.
The new bound improves upon (15) for large IYI, but not for smalllyl. To have it both ways, apply (15) if Iyl < c and (20) otherwise. Increase C to the maximum of illf"'ll and 111"11. Then the bound on the expected remainder is sharpened to IPIR(X, Y)I::.::;; IPCIYI 3 {IYI < c} + IPClYI 2 {IYI;::: : .: ; cCIP(y2) + CIPy2{ I YI ;::: c}.
(21)
d
22 Lindeberg Central Limit Theorem. For each n let Zn be a sum of independent random variables ~nb ~n2' ••• , ~nk(n) with zero means and variances that sum to one. If,for each fixed c > 0, the Lindeberg condition
ken)
L
(23)
IP~;j{l~njl ;::: c} ~ 0
as
n~
00
j= 1
is satisfied, then Zn -. N(O, 1).
Use the same notation as in the proof of Theorem 18. Denote the left-hand side of (23) by Ln(c). Stop one line earlier than before in the application of (17): PROOF.
ken) IIPf(Zn) - IPf(N(O, 1))1 : .: ;
L
ken)
IPIR(Snj, ~n)1
j= 1
+ L
IPIR(Snj,1]n)l·
j= 1
From (21), the first sum is less than
ken) C
L [cIP~;j + IP~;j{l~njl ;::: dJ
j= 1
which equals Cc + CLnCc) because the variance sum to one. For the second sum retain the bound ken)
C
L (J~jIPIN(O, lW
j= 1
53
111.4. The Central Limit Theorem
but, in place of Jensen's inequality, use:
L~ U~jr ~ (m~x UnjrL~: U~jr = max
Z
Unj
j
~ max[e
Z
+ IP~~j{l~njl
~ e)]
j
o The strange-looking Lindeberg condition is not as artificial as one might think. For example, consider the standardized summands for a sequence {y"} of independent, identically distributed random variables with zero means and unit variances: ~nj = n - I/Zl} for j = 1, 2, ... , n. In this case
Lie) = nIPn-IYi{lYII
~ nl/Ze},
which tends to zero as n tends to infinity, by dominated convergence, because Yi is integrable. It is even more comforting to know that the Lindeberg condition comes very close to being a necessary condition (Feller 1971, Section XV.6) for the Central Limit Theorem to hold.
24 Example (A Central Limit Theorem for the Sample Median). Let Mn be the median of a sample {YI , Yz , ... , y"} from a distribution function G with median M. For simplicity, suppose the sample size n is odd, n = 2N + 1, so that Mn equals the (N + 1)st order statistic of the sample. Suppose also that the underlying distribution function G has a positive derivative y at its median. To prove that
nI/Z(Mn - M)
~ N(O,
h- Z),
it suffices to check pointwise convergence of the distribution functions. (25)
IP{nI/Z(M n - M)
~
x}
+ n-I/Zx} = IP {at least N + 1 observations
= IP{Mn
~
M
= IP[N + 1 ~
.f
{l}
~
M
+ n-I/Zx}
~ M + n-I/Zx}].
J= I
Define
Pn = IP{l} bn
=
~nj = tn
=
~
M
+ n- I/2 x} =
G(M
[npnC! - Pn)J I/Z ,
[{l} ~ M + n-I/Zx} - PnJ/b n, (N
+
1 - npn)/b n .
+ n-I/Zx),
54
Ill. Convergence in Distribution in Euclidean Spaces
Check that IP~nj = 0 and IP~~l + ... + IP~~n = 1. Continuity of G at M gives Pn ...... t; differentiability of G at M gives tn ...... - 2xy. The last probability in (25) becomes (26)
As each I~nj I is less than b;; 1, which converges to zero, both the Liapounoff and Lindeberg conditions are easy to check. n
n
1 "~ IPI!'Sn).1 3 < "i....J IP!,2 - bn ~nJ j= 1
= b-n 1
j= 1
n
LIP~~j{l~njl:::::c}=O
if c>b;;l.
j= 1
By either of two routes, n
I
~nj ---. N(O, 1).
j= 1
Problem 13 shows that substitution of - 2xy for tn in (26) leads to the correct result: IP{n l/2 (Mn - M) ::;; x} ...... IP{N(O, 1) ::::: -2xy}.
o
111.5. Characteristic Functions The smoothing argument of Section 3 showed that only expectations of ~co(IRk) functions need be considered when checking convergence in distribution of random vectors. In this section, further analysis of the method of smoothing will show that the class of functions can be narrowed a little more: it suffices to check the convergence IP exp(iy . Xn) ...... IP exp(iy· X)
for each fixed vector y inIRk. That is, pointwise convergence of characteristic functions implies convergence in distribution. Start again from the convolution expression
derived under the assumption that Y has a N(O, I k ) distribution independent of X. This holds for every bounded measurable f. Reverse the order of
55
IlI.5. Characteristic Functions
integration (Fubini) on the right-hand side to show that
lPf(X
+ O"Y) =
f
f(z)J(z) dz,
where (27) The distribution of X + 0" Y has density function J(.) with respect to lebesgue measure. The dependence on 0", which will remain fixed for most of the argument, need not be made explicit. The integrand appearing on the right-hand side of (27) comes from the density function of 0" Y evaluated at (z - X). It is also proportional to the characteristic function of Y evaluated at (z - X)/O":
Invoke Fubini's theorem.
(28)
J(z) =
f (2nO")-klP exp( - iy· X/O") exp(iy • z/O" -
~lyI2) dy
where cPU denotes the characteristic function of X. Formula (28) shows that the characteristic function of X uniquely determines the distribution of X + 0" Y for each fixed 0". Because X + 0" Y converges almost surely to X as 0" tends to zero, this proves that the characteristic function also uniquely determines the distribution of X. A stronger result can be proved by exploiting continuity properties of the dependence of J on cP. 29 Continuity Theorem. A sequence of random vectors {Xn} converges in distribution to a random vector X if and only if the corresponding sequence of characteristic functions converges pointwise:
lP exp(iy . X n)
---+
lP exp(iy • X)
for each fixed y in IRk.
PROOF. Prove necessity by splitting exp(ix. y) into its real and imaginary parts, both of which define functions in ~(IR k).
56
Ill. Convergence in Distribution in Euclidean Spaces
For the proof of sufficiency, denote the characteristic function of X n by CPn, and the density function of X n + 0" Y by 1 n' The domain of integration is IRk. For fixed f and 0", IlPf(Xn
+ O"Y)
- lPf(X
+ O"Y)
J
=
I
:s;
Ilfll
f(z)Jn(z) dz -
J
f(z)l(z) dzl
J l n(Z) - l(z) I dz l
= IlfII[J(1(Z) - In(z))+ dz + J(1(Z)-l n(Z))- dZ] = 211fll J(1(Z) - In(z))+ dz, because 0=1 - 1
= J(1(Z)-l n(z))dZ = J(1(Z) - In(z))+ dz - J(1(Z)-l n(Z))- dz. Dominated convergence applied to (28) with cP replaced by CPn shows that the functions {in} converge point wise to 1. Thus {(1 - In)+} converges pointwise to zero. Notice that (1 - In)+ :s; 1 for each n. Invoke dominated convergence again.
211fll J(1(Z)
- lnC z ))+ dz
Complete the proof by an appeal to Lemma 11.
~ O. D
Perhaps this result should have been proved before we launched into the Central Limit Theorem proofs of Section 4. At least for identically distributed random variables, Theorems 18 and 22 could have been disposed of more rapidly; but the general cases would have required much the same level of effort, applied to exp(y' ix) in place of a general ~3(IR) function. The advantages of working with characteristic functions come from the special multiplicative properties of the complex exponential. A central limit theorem for martingales in Chapter VIII will make use of this property. Theorem 29 contains a bonus: convergence in distribution of random vectors, Xn ~ X, is equivalent to the convergence in distribution of all linear functions of the random vectors: y . Xn ~ Y • X, for every y. (For the
57
111.6. Quantile Transformations and Almost Sure Representations
non-trivial half of the assertion, notice that CPn(t) is the expectation of a bounded continuous function of t· Xn.) Convergence problems for random vectors reduce to collections of convergence problems for random variables; limit theorems for random vectors can be deduced from their univariate analogues. 30 Multivariate Central Limit Theorem. For every sequence IP~j =
dent, identically distributed random vectors with n-1/2(~1
+ ... +
~n) ---+
{~n}
°and
of indepenV,
IP(~j~j) =
N(O, V).
PROOF. For fixed y, the random variables {y. ~n} are independent and identically distributed with zero means and variances equal to y'Vy. If y'Vy = the random variables y. ~j and y. N(O, V) degenerate to zero; the assertion then holds for trivial reasons. Otherwise standardize to unit variance. The standardized sequence satisfies the Lindeberg condition. By Theorem 22,
°
(y'Vy)-1/2y·n-1/2(~1
+ ... + ~n)
---+N(O, 1) = (y'Vy)-1/2 y • N(O, V).
0
111.6. Quantile Transformations and Almost Sure Representations The definition of convergence in distribution by means of pointwise convergence of distribution functions does have its advantages, at least for real random variables. Behind some of these advantages lies a construction that reduces problems involving arbitrary distributions on the real line to the uniform case. For each distribution function F define a quantile function Q(u) = inf{t: F(t)
~
u}
for
°<
u < 1.
Right continuity of F shows that Q(u) sits at the left endpoint of the closed interval of t values for which F(t) ~ u; in other words, F(t)
~
u
if and only if Q(u):::;; t.
If ~ has the Uniform(O, 1) distribution then IP{Q(~)
:::;; t} = IP{F(t)
~
0
= F(t).
That is, Q(~) has distribution function F. The same construction gives a method for representing weakly convergent sequences of probability measures by sequences of random variables that converge almost surely.
58
Ill. Convergence in Distribution in Euclidean Spaces
31 Representation Theorem. Let {P n} be a sequence of probability measures on the real line that converges weakly to P. There exist random variables {Xn} with distributions {P n}, and X, with distribution P, such that Xn ~ X almost surely.
PROOF. Write Fn for the distribution function of P n' and F for the distribution function of P. Denote the quantile function corresponding to Fn by Qn. Choose ~ with distribution Uniform(O, 1) then define Xn = Qn(~). It suffices to show that {Qn(u)} converges for each u outside a countable subset of (0, 1), for then we can define X (almost surely) as the limit of the {X n }. (Problem 21 gives a more concrete representation for x.) Recall that Fn(t) ~ F(t) for each t in the dense set To of points for which IP{X = t} = 0. Each non-empty open interval contains points of To. In particular, if {Qn(u)} does not converge for some fixed u then there exist points t and s in To for which liminf QnCu) < t < s < limsup Qn(u). Since Qn(u) < t implies Fn(t)::::: U, the limit along a subsequence gives F(t) ::::: u. Similarly, Qn(u) > s infinitely often implies that F(s) ::; u. Thus F takes the constant value u throughout the interval et, s]. There can be at most countably many values of u (a set of zero lebesgue measure) for which F has such a flat spot: different u values produce disjoint flat spots, and the real line can accommodate only countably many disjoint intervals. This proves that liminf QnCu) = limsup Qn(u) for almost all values of u in (0, 1). If X is to be defined as the point wise limit of {Xn} we should also exclude the possibility of infinite limits. Given u in (0, 1), choose t in To such that F(t) > u. Convergence at points of To gives Fn(t) > u eventually, which implies QnCu) ::; t eventually, and so limsup QnCu) < 00. The liminf can be handled similarly. 0 Some weak convergence results, such as the one-dimensional case of the Continuous Mapping Theorem, reduce to trivialities when the Representation Theorem is invoked. Similar representations do obtain in higher dimensions and for probability measures on abstract metric spaces, but these require a completely different construction. More about that in Chapter IV. 32 Example. Given a probability measure P on {18(IR) and a bounded sequence of measurable functions {h n } converging pointwise to a limit function, h, under what conditions will Pnhn ~ Ph for every sequence {P n} converging weakly to P?
Consider first the special case where P concentrates at a single point x. Choose Pn concentrated at x n, for any sequence of real numbers {xn} converging to x. In this case the requirement reduces to hnCx n) ~ hex). This shows that something stronger than mere pointwise convergence of hn to
111.6. Quantile Transformations and Almost Sure Representations
59
h is needed. (Notice that if hn = h for every n, the requirement is equivalent to continuity of h at x.) In general it will suffice if hnCx n) ~ hex) for every x in a set of P measure one and every sequence {x n } converging to such an x. Represent {P n } by an almost surely convergent sequence {Xn} of random variables. Then hn(Xn) ~ heX) almost surely. By dominated convergence, IPhnCX n) ~ IPh(X), which implies that Pnhn ~ Ph. D 33 Example. Let G be an open subset of JR. Suppose that P n ~ P. The Representation Theorem can be used to show that liminf PnCG) ;:::: peG). (The proof of the Convergence Lemma also contains the result implicitly.) Switch to almost surely convergent representations Xn and X. For an w at which convergence holds, if X(w) belongs to G then so does XnCw) for all n large enough. That is,
liminf {X n
E
G} ;:::: {X
E
G}
almost surely.
Apply Fatou's lemma. liminf P n G = liminf IP{Xn E G} ;:::: IP liminf{X nE G} ;:::: IP{X E G} = peG).
D
34 Example. Let {cf>n} be the characteristic functions corresponding to a weakly convergent sequence of probability measures {P n}, and cf> be the characteristic function of the limit distribution P. Take {Xn} as the almost surely convergent representing sequence of random variables. For each fixed t, the sequence {exp(itXn)} converges almost surely to exp(itX). Moreover, the convergence is uniform on each bounded interval loft-values:
sup lexp(itXn) - exp(itX) I :::;; sup lexp(it(Xn - X)) - 1)1 I
I
~
° almost surely,
by virtue of the continuity of the exponential function at zero. Because sup lIP exp(itXn) - IP exp(itX) I :::;; IP sup lexp(it(Xn - X)) - 1)1, I
I
the sequence {cf>n(t)} converges uniformly to cf>(t) over bounded intervals.
D
The argument for the proof of the Representation Theorem made little explicit use of the limit distribution function F. The essentials were: (i) existence of lim FnCt) for each t in a dense subset To; (ii) for each fixed u in (0,1) there exists a t in To for which liminf Fn(t) > u, and an s in To for which limsup Fn(s) < u.
60
Ill. Convergence in Distribution in Euclidean Spaces
The second property is equivalent to the existence, for each fixed constant K such that limsup P n [ -K, KJ <
(35)
G
> 0, of a
G.
Of course K depends on G. A sequence {P n} with this property is said to be uniformly tight. The corresponding property for a sequence of random variables {Xn}-that for each G > 0 there exists a K such that limsup IP{ IXnl > K} <
G
-is also called uniform tightness. 36 Theorem. Each uniformly tight sequence of probability measures on the real line has a subsequence that converges weakly.
Use Cantor's diagonalization procedure to select a subsequence {P n,} for which {Fn{t)} converges for each rational t. The subsequence satisfies both of the conditions (i) and (ii) noted above. Construct an almost surely convergent sequence of random variables {X n'} such that X n' has distribution P n, and X n, ~ X almost surely. The sequence {P n,} converges weakly to the distribution of X. 0 PROOF.
By providing a method for constructing probability measures on 86'(JR), this theorem allows specification of the limit distribution to be omitted from some important results. For example, the Continuity Theorem for characteristic functions, as proved (Theorem 29) in Section 5, can be improved upon slightly. 37 Continuity Theorem (General Form). Let {Xn} be a sequence of real random variables whose characteristic functions {CPn} converge pointwise to some function cp. If cP is continuous at the origin then it must be the characteristic function of an X for which X n r-+ X. PROOF. Once we prove that cP is a characteristic function, Theorem 29 will do the rest. Let us show that {Xn} is uniformly tight. Then, by Theorem 36 it will have at least one subsequence that converges in distribution. The limit of any such subsequence must have cP as its characteristic function. The inequality (35), which defines uniform tightness, calls for a constraint to be placed on the tails of the distributions of the random variables {X n }· Here we can use an inequality, valid for any random variable Z with characteristic function p, that relates the tail behavior of a distribution to the values of its characteristic function near the origin: there exists a positive constant a (approximately 6.308) such that
(38)
IP{IZI
~ h- 1}: : ; (a/2h) rh [1
- p(t)] dt
61
Notes
for every positive h. This follows (Fubini's theorem) from the equality (2h)-1
rh
[1 - IP exp(iZt)J dt = IP[1 - (sin hZ)/hZ].
Interpret (sin 0)/0 as 1. Because the integrand is non-negative everywhere and greater than rx - 1 = 1 - sin 1 on the set {I Z I 2: h - I}, the expectation is greater than rx- 1 IP{ IZ I 2: h -I}. Apply (38) with Z = X nand p =
= (rx/2h)
rh
fh
[1 -
[1 -
by dominated convergence. Thanks to the continuity of
=
lim rx[1 -
=
0
n
by choosing h small enough.
o
NOTES
Everything in this chapter is classical. The name Continuous Mapping Theorem seems to have taken hold in the literature, despite its evident inappropriateness. Billingsley (1968, Section 5) has more on the history of the theorem. Some authors refer to P-almost-sure-continuity as P-Riemann-integrability. Convolution with a normal kernel as a means of smoothing is an ancient idea. Weierstrass (1885) used it to prove his famous approximation theorem. Liapounoff (1900) introduced an auxiliary normal variable as a device for smoothing a sum of independent variables. That enabled him to complete the argument of Glaisher (1872), and thereby prove with full rigor a central limit theorem. The Liapounoff condition did not appear in the 1900 paper; instead Liapounoff imposed a stronger requirement on the maximum third absolute moment. In his 1901 paper he improved this to a requirement on the sum of (2 + b)th moments about the mean. (Problem 19 gives a minor variation on his 1901 result.) Section 4 follows in part the lucid exposition of Lindeberg (1922), with a little help from Billingsley (1968, Section 7). Uspensky (1937, Chapter XIV) attributed the Continuity Theorem to Liapounoff (1900); the key idea was implicit in Liapounoff's proof. Levy (1922) proved it explicitly. Both Liapounoff and Levy required uniform convergence on compacta. The dominated convergence trick in the text is often called Scheffe's lemma.
62
Ill. Convergence in Distribution in Euclidean Spaces
PROBLEMS
[1] Suppose X, Xl' Xl"" are real random variables such that IP{Xn :s; x} ..... IP{X :s; x} whenever IP{X = x} = O. Prove that Xn "> X. [For a fixedJin "6'(IR) find points {lXi} with - CIJ < 1X1 < IXz < ... < IXk < CIJ such that IP{X = IX;} = 0 and both of IP{X :s; IXd and IP{X > IXk} are small. Choose the {IX;} close enough together to ensure that J changes by very little over each interval [lXi' lXi+l]. Approximate J by a step function constant on each of those intervals.] Extend the argument to higher dimensions. [2] Let {x n} be a sequence of points in IRk, and {Xn} be a sequence of degenerate random vectors with IP{Xn = xn} = 1. Show that {Xn} converges in distribution if and only if {xn} converges. [3] Let P be a probability measure on the O"-field &6'(IRk). Show that for each e > 0 and each borel set B there exists a closed set F, contained in B and an open set G, containing B such that peG, - F,) < 8. [Prove that the class of all such B forms a O"-field. It contains every closed set, because closed sets can be written as countable intersections of open sets.] Deduce that if IPJ(X) = IPJ(Y) for every bounded uniformly continuous J then X and Y have the same distribution. [Represent open sets as the limits of increasing sequences of uniformly continuous functions.] [4] A sequence of probability measures on &6'(IRk) can converge weakly to at most one limit distribution: if Pn ,... P and Pn ,... Q then P and Q agree for all borel sets. [Use Problem 3.] [5] If X n ,... X then Xn + op(l) ,... X. [Appendix A explains the ovC') notation. This result is sometimes called Slutsky's theorem.] [6] If {Xn} converges in probability to X then Xn
">
X. The converse is false.
[7] If Xn"> X and IP{X = c} = 1, for some constant c, then {Xn} converges in probability to c. [8] Give an example of a continuous function 9 and a sequence of random vectors with X n ,... X for which {IPg(X n)} does not converge to IPg(X). [Don't forget the word bounded in the definition of convergence in distribution.] [9] Give an example of a map H from IRk into itself and a sequence of random vectors in IRk with X n ,... X for which HX n does not converge in distribution to HX. [10] The set D of discontinuity points of a map H from IRk into IRs is borel measurable. [Start from the set Dm n of all those points x for which IHy - Hz I > m-I for at least one pair ofpoint's with Ix - yl < n- l and Ix - zl < n- l . Prove that Dm,n is open, Tops.pe,] [11] A random IRk-vector X and a random IRj-vector Y defined on the same probability space can be combined into a single random IRj+k-vector (X, Y). If Xn "> X, as random vectors in IRk, and y" ,... Y, as random vectors in IRj, it need not follow that (X n , y") ,... (X, Y), But if IP{X = c} = 1 for some constant c the result is true. [Consider expectations of bounded uniformly continuous functions on IRj+k.] Use characteristic functions to prove that the result also holds if X is independent of Y and Xn is independent of Y", for each n,
63
Problems
[12] If Xn has a t-distribution on n degrees of freedom then Xn [Use Problem 11.]
r->
N(O, 1) as n - t
00.
[13] If real random variables {X n} converge in distribution to X and {xn} is a sequence of real numbers converging to an x for which IP{X = x} = 0, prove that IP{Xn 2: xn} - t IP{X 2: x}. [14] If a sequence of random vectors {Xn} converges in distribution then Xn [Appendix A explains the OpO notation.]
=
Op(I).
[15] Let H be a measurable map from IRk into IR S that is differentiable at a point Xo' That is, there exists a linear map L from IRk into IRS such that Hx = Hxo + L(x - x o) + o(x - xo) near Xo. 1/2 If n (Xn - xo) r-> Z, prove that n1/2(HXn - Hxo) -. LZ. [Some authors call this the delta method.]
[16] If Xn ~ Bin(n, 8) for some fixed 8 in (0, 1), find the limiting distribution of nI/2[arsin(Xn/n)1/2 - arsin 81/2 ] as n - t 00. [17] Fory real, define Hn(Y) = exp(iy) - 1 - iy - .,. - (iy)"/n!. Prove that IHn(Y) I ::; lyln+I/(n + I)!. [Proceed inductively, using if~ Hn(s) ds = Hn+l(t) for t > O. Take complex conjugates for t < O. Borrowed from Feller (1971).]
[18] Use the inequality in the previous problem to show that the characteristic function of n- 1/2 (Poisson(n) - n) converges pointwise to exp( - t 2/2). [This is one way to find the characteristic function of the N(O, 1) distribution.] [19] For each n let Zn be a sum of independent random variables zero means and variances that sum to one. If for some 0 > 0, k(n) L IPI~nY+O - t 0 as n - t 00
~nb
... ,
~nk(n)
with
j~1
then Zn ---. N(O, 1). [Apply the Lindeberg Central Limit Theorem. Liapounoff.] [20] Suppose a random vector X has a characteristic function
[Choose a continuous f vanishing outside a bounded region of IRk. Start from the expression for IPf(X + Cl Y) derived in Section 5:
Make a change of variable W = - y/Cl, then take the limit as Cl - t 0 under the integral sign. Adapt Problem 3 to complete the proof.] [21] In the proof of the Representation Theorem write Q for the quanti le function corresponding to F. Show that {Qn(u)} converges to Q(u) for each u that is a point of continuity for Q.
CHAPTER IV
Convergence in Distribution in Metric Spaces ... in which that theory from Chapter III depending only on the metric space properties oflR k is extended to general metric spaces. It is argued that the theory should consider not just borel-measurable random elements. A Continuous Mapping Theorem and an analogue of the almost sure Representation Theorem survive the generalization. A compactness condition-uniform tightness-is shown to guarantee existence of cluster points of sequences of probability measures.
IVJ. Measurability We write a statistic as a functional on the sample paths of a stochastic process in order to break an analysis of the statistic into two parts: the study of continuity properties of the functional; the study of the stochastic process as a random element of a space of functions. The method has its greatest appeal when many different statistics can be written as functionals on the same process, or when the process has a form that suggests a simple approximation, as in the goodness-of-fit example from Chapter I. There we expressed various statistics as functionals on the empirical process Un, which defines a random element of D[O, 1]. Doob's heuristic argument suggested that Un should behave like a brownian bridge, in some distributional sense. Formalization of the heuristic, the task we embark upon in this chapter, requires a notion of convergence in distribution for random elements of D[O, 1]. As for euclidean spaces, the definition will involve convergence of expectations of bounded, continuous functions of the processes. For this we need a notion of distance. Equip D[O, 1] with its uniform metric, which assigns the maximum separation
Ilx - yll = sup Ix(t) - y(t) I as the distance between x and y. We shall find it easiest to prove convergence in distribution of {Un} using this metric, even though it does create some minor measurability difficulties. Chapter VI will examine another metric, for which these difficulties disappear, at the cost of greater topological complexity. An expectation IPf(U n) is well defined only when f(U n) is measurable. If Un lives on a probability space (Q, fff, IP), we can arrange for measurability
IV.1. Measurability
65
by equipping D[O, 1J with a rr-field, f!l> say, then checking Cjf!l>-measurability of V nand f!l>-measurability off. The borel rr-field will not be the best choice for f!l>. The definition of convergence in distribution for random elements of a general metric space anticipates this complication for D[O, 1]. 1 Definition. An Cjd-measurable map X from a probability space (Q, C, IP) into a set X with rr-field .911 is called a random element of X. If X is a metric space, the set of all bounded, continuous, dj[!8(IR)measurable, real-valued functions on X is denoted by ~(X; d). A sequence {Xn} of random elements of X converges in distribution to a random element X, written Xn ~ X, if IPf(Xn) ---+ IPf(X) for each f in ~(X;
d).
A sequence {P n } of probability measures on .911 converges weakly to P, written P n ~ P, if Pnf ---+ Pffor every fin ~(X; d). 0 The borel rr-field [!8(X), the rr-field generated by the closed sets, will always contain d. For those spaces where we need .911 strictly smaller than the borel rr-field, we will usually have it generated by the collection of all closed balls in X. Also the trace of .911 on each separable subset of X will coincide with the trace of the borel rr-field on the same subset. Limit distributions will always be borel measures concentrating on separable, .911measurable subsets of X. We could build these properties into the definition of weak convergence, but it would neither save us any extra work, nor simplify the theory much. 2 Example. If D[O, 1J is equipped with the borel rr-field [!8 generated by the closed sets under the uniform metric, the empirical processes {Vn} will not be random elements of D[O, 1J in the sense of Definition 1. That is, V n is not Cj[!8-measurable. Consider, for example, the situation for a sample of size one. (Problem 1 extends the argument to larger sample sizes.) For each subset A of [0, 1J define GA = {x E D[O, 1J: x has a jump at some point of A}. Each GA is open because Ix(t) - x(t - ) I depends continuously upon x, for fixed t. If V 1 were Cj[!8-measurable, the set {V 1 EG A } = glEA}-would belong to C. A probability measure J.l could be defined on the class of all subsets of [0, 1J by setting J.l(A) = IP{~ 1 E A}. This J.l would be an extension of the uniform distribution to all subsets of [0, 1]. Unfortunately, such an extension cannot coexist with the usual axioms of set theory (Oxtoby 1971, Section 5): if we wish to retain the axiom of choice, or accept the continuum hypothesis, we must give up borel measurability of V 1. The borel rr-field generated by the uniform metric on D[O, 1J contains too many sets. There is a simple alternative to the borel rr-field. For each fixed t, the map V n(·, t) from Q into IR is a random variable. That is, if n t denotes the
66
IV. Convergence in Distribution in Metric Spaces
coordinate projection map that takes a function x in D[O, 1] onto its value at t, the composition 11: t Un is 6"/&8(lR)-measurab1e. Each Un is measurable with respect to the u-fie1d f!jJ generated by the coordinate projection maps (Problem 2). Call [!J! the projection u-fie1d. Problem 4 shows that [!J! coincides with the u-fie1d generated by the closed balls. All interesting functiona1s on D[O, 1] are [!J!-measurab1e. 0 0
Too large a u-fie1d d makes it too difficult for a map into X to be a random element. We must also guard against too small an d. Even though the metric on X has lost the right to have d equal to the bore1 u-fie1d, it can still demand some degree of compatibility before a fruitful weak convergence theory will result. If ~(X; d) contains too few functions, the approximation arguments underlying the Continuous Mapping Theorem will fail. Without that key theorem, weak convergence becomes a barren theory. An extreme example should give you some idea of the worst that might happen. 3 Example. Allow the rea11ine to retain its usual euclidean metric, but change its u-fie1d to the one generated by the intervals of the form en, n + 1), with n ranging over the integers. Call this u-fie1d !!l/. Functions measurable with respect to !!l/ must stay constant over each of the generating intervals. For a continuous function, this imposes a harsh restriction; continuity at each integer forces an !!l/-measurab1e function to be constant over the whole real line. This completely degrades the weak convergence concept: every sequence of !!l/-measurab1e random elements converges in distribution. It bodes ill for a sensible Continuous Mapping Theorem. Consider the map H from the disfigured rea11ine into the real rea11ine (equipped with its usual metric and u-fie1d) defined by Hx = 1 if x<3 and Hx = otherwise. It is a perfectly good !!l/-measurable map, continuous at the point 1. Apply it to random elements {Xn} identically equal to 3, and X identically equal to 1. Even though Xn ~ X in the sense of Definition 1, {HX n } does not converge in distribution to HX. 0
°
°: ;
IV.2. The Continuous Mapping Theorem Suppose Xn ~ X, as .W'-measurab1e random elements of a metric space X, and let H be an d/d'-measurab1e map from X into another metric space X'. If H is continuous at each point of an ,W'-measurab1e set C with IP{X E C} = 1, does it follow that HX n ~ HX? That is, does {IPf(HX n )} converge to IPf(HX) for every f in ~(X'; d')? We found an answer to the analogous question for random vectors in Section I1I.2 by reducing it to an application of the Convergence Lemma. The same approach works here. We need to prove IPh(X n ) ~ IPh(X) for every bounded, ,W'-measurab1e, real-valued h that is continuous at each point of C. Were d equal to the bore1 u-fie1d &8(X), the proof would go
67
IV.2. The Continuous Mapping Theorem
through almost exactly as before, with only a few words difference. For borelmeasurable random elements of metric spaces, the theory parallels the theory in Chapter III very closely, at least as far as the Continuous Mapping Theorem is concerned. Example 3 warns us that non-borel (J-fields require more careful handling. With this in mind, let's rework the Convergence Lemma of Chapter Ill, paying more attention to measurability difficulties. To begin with we assume only that .91 is a sub-(J-field of P-8(Pl). Define
(4)
ff = {f
E
t:(j(Pl; d):
f
So h}.
Last time we constructed a countable subfamily of ff whose pointwise supremum achieved the upper bound h at each point of C. Functions in the subfamily took the form
fm,r(x)
=
r /\ md(x, {h So r})
Continuity of fm" suffices for borel measurability, but it needn't imply d-measurability. We must find a substitute for these functions. This is possible if we impose a regularity condition, which ensures that the pointwise supremum of ff equals h at each point of C. If C is separable (meaning that it has a countable, dense subset), we can then extract from ff a countable subfamily having the same supremum as ff at each point of C. The regularity condition will capture the key property enjoyed by fm". Without loss of generality suppose h > O. Suppose also that h is continuous at a point x. Choose r with 0 < r < hex). Look for an f in ff withf(x) ~ r. Continuity provides a 6 > 0 such that hey) > r on the closed ball B(x, 6) centered at x. If we could find a g in t:(j(Pl; d) with 0 So g So B(x, 6) and g(x) = 1, the function rg would meet our requirements. Notice the similarity to the topological notion of complete regularity (Simmons 1963, Section 27). If .91 happened to contain all the closed balls centered at x, a property enjoyed by the projection (J-field on D[O, 1J (Problem 4), the function
(5)
g(y)
= [1 - 6- 1 d(x, y)J+
would do, because {g ~ 1 - s} = B(x, s6). For general.91 we must postulate existence of the appropriate g. To maintain the parallel with euclidean spaces as closely as possible, strengthen the requirements on g to include uniform continuity. We lose only a scintilla of generality thereby; the special g of (5) still passes the test. 6 Definition. Call a point x in Pl completely regular (with respect to the metric d and the (J-field d) if to each neighborhood V of x there exists a uniformly continuous, .91-measurable function g with g(x) = 1 and g So V.
D You might well object to yet another mathematical notion attaining the status of regularity; the world is already overloaded with instances of
68
IV. Convergence in Distribution in Metric Spaces
"regular" as a synonym for "amenable to our current theory." At least it has the virtue of reminding us of its topological counterpart. (A more sadistic author might have called it T3J...) The terminology would not be wasted if we were to expand our weak 2convergence theory to cover borel measures on general topological spaces, for there topological complete regularity seems just the thing needed for a well-behaved theory.
7 Convergence Lemma. Let h be a bounded, d -measurable, real-valued function on!![. Ifh is continuous at each point of some separable, d-measurable set C of completely regular points, then: (i) Xn ~ X and IP{X E C} = 1 imply IPh(Xn) -+ IPh(X); (ii) P n ~ P and PC = 1 imply Pnh -+ Ph. PROOF. As the arguments for both assertions are quite similar, let us prove (ii) only. Assume that h > 0 (add a constant to h if necessary). Define §P as in (4), but with the continuity requirement strengthened to uniform continuity. At those completely regular points of !![ where h is continuous, the supremum of §P equals h. This applies to points in C. Separability of C will enable us to extract a suitable countable subfamily from §P. Argue as for the classical Lindel6ftheorem (Simmons 1963, Section 18). Let Co be a countable, dense subset of C. Let {gt> g2, ... } be the set of all those functions of the form rB, with r rational, B a closed ball of rational radius centered at a point of Co, and r B ~ f for at least one f in fF. For each gi choose one f satisfying the inequality gi ~ f. Denote it by /;. This picks out the required countable subfamily: (8)
sup /;
=
sup §P
on C.
i
To see this, consider any point z in C and any f in §'. For each rational number r such thatf(z) > r > 0 choose a rational 8 for whichf > r at all points within a distance 28 of z. Let B be the closed ball of radius 8 centered at a point x of Co for which d(x, z) < 8. The function rB lies completely below f; it must be one ofthe {gJ. The corresponding/; takes a value greater than r at z. Assertion (8) follows.
----r
~g;
x
z
f
IV.2. The Continuous Mapping Theorem
69
Complete the argument as for the Convergence Lemma of Section III.2. Assume without loss of generality that j; i h at points of C. Then liminf Pnh 2:: liminf Pnj; = Pj; -+ Ph Replace h by - h limsup.
for each i because Pn ~ P as i -+ 00, by monotone convergence.
+ (a big constant) to get the companion inequality for the 0
9 Corollary. If JPf(X n) -+ JPf(X) for each bounded, uniformly continuous, d -measurable f, and if X concentrates on a separable set of completely regular points, then Xn ~ X. 0 The corollary flows directly from the decision to insist upon uniformly continuous separating functions in the definition of a completely regular point. As with its counterpart for euclidean spaces, it makes some weak convergence arguments just a little bit more straightforward than the corn:sponding arguments with continuous functions. 10 Example. Let f£ be a space equipped with a a-field d and metric d, and ifY be a space equipped with a a-field gg and metric e. Equip f£ ® ifY with its product a-field and the metric a defined by
a[(x, y), (x', y')] = max[d(x, x'), e(y, y')]' Suppose Xn ~ X, as random elements of f!£. IfJP x concentrates on a separable set of completely regular points, and y" -+ Yo in probability for some fixed completely regular point Yo in ifY, then (X n, y,,) ~ (X, Yo), as random elements of the product space f£ ® ifY. Of course the assertion only makes sense if X nand y" are defined on the same probability space. Given that prerequisite, measurability with respect to the product a-field presents no problem, because
(Xn' y")-l(A ® B) = (X; lA) n (y;lB), and similarly for (X, Yo). Write C for the separable set on which JP x concentrates. Then JPx,yO concentrates on the product set C ® {yo}, which is separable. Each point of this set is completely regular: iff(c) = 1 andf = 0 outside the ball of d-radius s, and g(yo) = 1 and g = 0 outside a ball of e-radius s, then the productf(x)g(y) equals 1 at (c, Yo) and vanishes outside a ball of a-radius s. The product is uniformly continuous if both f and g are bounded and uniformly continuous; it is d ® gg-measurable if f is d-measurable and g is gg-measurable. By virtue of Corollary 9, to prove (Xn' y,,) ~ eX, Yo) we have only to check that JPh(Xn' Yn) -+ JPh(X, Yo) for each bounded, uniformly continuous, d ® gg-measurable, real function h on f£ ® ifY. Given s > 0 choose
70
IV. Convergence in Distribution in Metric Spaces
°
> so that Ihex, y) - hex', y') I < s whenever O"[(x, y), (x', y')] < (j. Write kO for the bounded, uniformly continuous, d -measurable function h(·, Yo). Then
(j
IIPh(Xn' y") - IPh(X, Yo)1 .::; s
+ 21IhIIIP*{e(Yn, Yo):2: (j} + IIPk(Xn) - IPk(X)I·
Convergence in probability of Yn to Yo makes the middle term converge to zero. (Notice the outer measure IP*. By definition, IP*Z equals the infimum of IPW over all @"-measurable real functions with W:2: Z. For most applications e(·, Yo) will be d-measurable, in which case IP* can be replaced by IP.) The last term converges to zero because Xn ~ X. 0 11 Example (Convergence in Distribution via Uniform Approximation). Let X, Xl> X 2 , ••• be random elements of f![ with IPx concentrated on a separable set of completely regular points. Suppose, for each s > and (j > 0, there exist approximating random elements AX, AX 1 , AX 2 , ••• such that:
°
(i) IP*{d(X, AX) :2: (j} < s; (ii) limsup IP*{d(Xn' AXn) :2: (iii) AX n ~ AX.
(j}
< s;
Then Xn ~ X. Notice again the use of outer measure to guard against nonmeasurability. We have already met a special case of this result in Lemma III.1l, where AXn = X n + 0" Y. In applications to stochastic processes, the approximations are typically constructed from the values of the processes at a fixed, finite set of index points. For such approximations, classical weak convergence methods can handle (iii). The assumptions (i) and (ii) place restrictions on the irregularity of the sample paths. Chapter V will take up this idea. The convergence X n ~ X follows from convergence of expectations for every bounded, uniformly continuous, ..Gf-measurableJ. If I f(x) - fey) 1< s whenever d(x, y) < (j then IIPf(Xn) - IPf(X) I is less than
IPlf(X n) - f(AXn) I + IIPf(AX n) - IPf(AX) I + IPlf(AX) - f(X)I· The convergence (iii) takes care of the middle term. Handle the first term by splitting it into the contributions from {d(Xn' AXn) :2: (j} and its complement; and similarly for the last term. 0 The Convergence Lemma has one other important corollary, the result that tells us how to transfer convergence in distribution of random elements of f![ to convergence in distribution of selected functionals of those random elements. For substantial applications turn to Chapter V. 12 Continuous Mapping Theorem. Let H be an dld'-measurable map from
into another metric space f!['. If H is continuous at each point of some separable, ..Gf-measurable set C of completely regular points, then Xn ~ X and IP{X E C} = 1 together imply HX n ~ HX. 0 f![
IV.3. Representation by Almost Surely Convergent Sequences
71
IV.3. Representation by Almost Surely Convergent Sequences In Section 111.6 we used the quantile transformation to construct almost surely convergent sequences of random variables representing weakly convergent sequences of probability measures. That method will not work for probabilities on more general spaces; it even breaks down for IR2. But the representation result itself still holds. 13 Representation Theorem. Let {P n} be a sequence of probability measures on a metric space. If P n --+ P and P concentrates on a separable set ofcompletely regular points, then there exist random elements {Xn} and X with distributions {P n} and P such that Xn -> X almost surely. D
The new construction makes repeated use of a lemma that can be applied to any two probability measures P and Q that are close in a weak convergence sense. Roughly speaking, the idea is to cut up the metric space !!C into pieces Bo, Bb"" Bk for which PBi ~ QB i for each i, so that the set B o has small P measure and each of the other B;'s has small diameter. We use these sets to construct a random element Y of !!C, starting from an X with distribution P. If X lands in Bi choose Y in Bi according to the conditional distribution Q('I BJ For i ;::: 1 this forces Y to lie close to X, because Bi doesn't contain any pairs of points too far apart. The random element Y has approximately the distribution Q: k
(14)
IP{YEA} =
L
IP{YEAIXEBJIP{XEB i}
i=O k
=
L Q(A IBJP(BJ i=O k
~
L Q(A
1
Bi)Q(Bi)
;=0
= Q(A). A slight refinement of the construction will turn the approximation into an equality. When applied with Q = P n and partitions growing finer with n, it will generate the sequence {Xn} promised by the Representation Theorem. 15 Lemma. For each G > 0 and each P concentrating on a separable set of completely regular points, the space !!C can be partitioned into finitely many disjoint, d-measurable sets Bo, Bb ... , Bk such that:
(i) the boundary of each Bi has zero P measure (a P-continuity set); (ii) P(B o) < G; (iii) diameter(B i) < 2G for i = 1, 2, ... , k.
72
IV. Convergence in Distribution in Metric Spaces
PROOF. Call the separable set C. To each x in C there exists a uniformly continuous, d-measurable f with f(x) = 1 andf = 0 for points a distance greater than e from x. The open sets of the form {f > IJ(}, for 0 < IJ( < 1, are all d -measurable and of diameter less than 2e. At each point on the boundary of {f > IJ(}, the continuous function f takes the value IJ(. Because P{f = IJ(} can be non-zero for at most countably many different values of IJ(, there must exist at least one IJ( for which the probability equals zero. Choose and fix such an IJ(, then write G(x) for the corresponding set {f > IJ(}. It has diameter less than 2e and is a P-continuity set. The union of the family of open sets {G(x): x E C} contains the separable set C. Extract a countable subfamily {G(x;): i = 1, 2, ... } containing C. (Every open cover of a separable subset of a metric space has a countable subcover: Problem 5.) Because P[Vl G(X J ]
i
P[Dl G(XJ]
~ P(C) =
1,
there exists a k such that P[Vl G(XJ] > 1 - e. Define Bi = G(xJ\[G(Xl) U '" u G(Xi-l)] for i = 1, ... , k and B o = [G(Xl) U ... u G(xk)J, a process known to the uncouth as disjointification. The boundary of Bi is covered by the union of the boundaries of the Pcontinuity sets G(x 1 ), ... , G(Xk)' Each Bi lies completely inside the corresponding G(Xi), a set of diameter less than 2e if i ~ 1. D PROOF OF THEOREM 13. Holding e fixed for the moment, carry out the construction detailed in the proof of the lemma, generating P-continuity sets Bo, Bb ... , Bk as described. The indicator function of Bi is almost surely continuous [P] because it has discontinuities only at the boundary of B i . So by the Convergence Lemma PnCB i) ~ P(BJ When n is large enough, say n ~ nee), (16)
PnCB;)
~
(1 - e)P(B i) for
i
= 0, 1, ... , k.
Write nm for n(2-m). Without loss of generality suppose 1 = n 1 < n2 < .... For nm S n < nm + b construct X n using the {BJ partition corresponding to em = 2- m . Notice that Bi now depends on n through the value of m. Let ~ be a random variable that has a Uniform(O, 1) distribution independent of X. If ~ s 1 - em and X lands in Bb choose X n according to the conditional distribution PnCIBJ So far no Bi has received more than its quota of P n measure, because of (16). The extra probability will be distributed over the space f!l to bring Xn up to its desired distribution Pn. If ~ > 1 - em choose X n according to the distribution fln determined by k
PiA)
=
fliA)IP{~
> 1 - em}
+
I
i=O
Pn(AIB;)(1 - em)P(BJ
IV.3. Representation by Almost Surely Convergent Sequences
73
That is, k
,unCA) =
e;;; 1
I
Pn(A IBJ[Pn(BJ - (1 - em)P(BJ].
;=0
By (16), the right-hand side is non-negative. And clearly ,un:?E = 1. Except on the set Qm = {X E Bo or ~ > 1 - em}, which has measure at most 2em , the random elements X and X n lie within 2em of each other. On the complement of the set {Qm infinitely often}, the sequence {Xn} converges to X. By the Borel-Cantelli lemma IP{Qminfinitely often} = O.
o
The applications of Theorem 13 follow the same pattern as in Section 111.6. Problems of weak convergence transform into problems of almost sure convergence, to which the standard tools (monotone convergence, dominated convergence, and so on) can be applied. 17 Example. Most of the proof of the Convergence Lemma did not use the full force of almost sure continuity for the function h. To get the inequality for the liminf we only needed lower-semicontinuity of h at points of C. (Remember that semicontinuity imposes only half the constraint of continuity: only a lower bound is set on the oscillations of h in a neighborhood of a point. Problem 9 will refresh your memory on semicontinuity.) The Representation Theorem gives a quick proof of the same result. If g is bounded below, lower-semicontinuous, and d-measurable (automatic if d equals the bore! IT-field), then liminf Png ;::: Pg whenever Pn ...-. P with P concentrated on a separable set of completely regular points. To prove it, switch to almost surely convergent representations. Lower-semicontinuity at X(w) plus almost sure convergence of the representing sequence imply
Iiminf g(Xn(w» ;::: g(X(w»
almost surely.
Take expectations. liminf Png = liminf IPg(Xn) ;::: IPg(X) by Fatou's lemma = Pg. A similar inequality holds for upper-semicontinuous, d-measurable functions that are bounded above. As a special case, (18)
limsup PnF S PF
for each closed, d-measurable set F. If inequality (18) holds for all such F then necessarily P n ...-. P (Problem 12). 0 19 Example. Let <§ be a uniformly bounded class of d-measurable, real functions on :?E. Suppose that Pn ...-. P, with P concentrated on a separable
74
IV. Convergence in Distribution in Metric Spaces
set of completely regular points. Suppose also that '!J is equicontinuous at almost all points [P] of :!E. That is, for almost all x and each 8 > 0 there exists ab> 0, depending on x but not on g, such that Ig(y) - g(x) I < 8 whenever d(x, y) < b, for every g in '!J. Then (20) This result underlies the success of most of the functions that have been constructed in the literature to metrize the topology of weak convergence. To prove (20), represent the probability measures by almost surely convergent random elements {Xn}, then deduce from equicontinuity that sup Ig(Xn) - g(X) 1--+
(21)
c§
°
almost surely.
It would be tempting to appeal to dominated convergence to get
sup IJPg(Xn) - JPg(X) I ~ JP sup Ig(X n) - g(X)I--+ 0, c§
c§
but that would assume measurability of the supremum in (21). Instead, note that (20) could fail only if, for some 8 > 0, there were functions {gn} in '!J for which IPng n - Pg nI ;;::: 8 infinitely often. Apply the dominated convergence argument to the countable family '!J 0 = {g b g 2, ... } to reach a con0 tradiction. 22 Example (The Bounded-Lipschitz Metric for Weak Convergence). Suppose that si contains all the closed balls, as in the case of D[O, 1] under its uniform metric. The function fO = r[1 - md(·, z)] +, which serves to separate z from points outside a small neighborhood of z, has the strong uniformity property
I f(x) - fey) I ~ mr d(x, y). A function satisfying such a condition, with mr replaced possibly by a different constant, is called a Lipschitz function. For the proof of the Convergence Lemma, Pnf --+ Pf for each bounded, d-measurable Lipschitz function would have sufficed; convergence for bounded Lipschitz functions implies weak convergence. From Example 19 we draw a sharper conclusion. Define Y to be the set of all d -measurable Lipschitz functions for which If(x) - f(Y)1 ~ d(x, y) and supx I f(x) I ~ 1. The class Y is equicontinuous at each point of :!E. Every bounded Lipschitz function can be expressed as a multiple of a function in Y. Define the distance between two probability measures on d by }'(P, Q) = sup{ IPf - Qf I: fEY}.
You can check that}, has all the properties required of a metric. If P concentrates on a separable set and P n --... P, the distance }'(Pn, P) converges to zero, in obedience to the uniformity result of Example 19. Conversely, the
75
IV.3. Representation by Almost Surely Convergent Sequences
convergence of A(Pn' P) to zero would ensure that Pn f ~ Pf for each bounded Lipschitz function f, which, as noted above, implies weak convergence. 0
23 Example (The Prohorov Metric for Weak Convergence). Suppose !![ is a separable metric space equipped with its borel a-field. For each (j > 0 and each borel subset A of!![ define
Ab = {x E!![: d(x, A) <
(j}.
(Visualize the open set Ab as A wearing a halo of thickness (j.) Define the Prohorov distance between two borel probability measures as
p(P, Q) = inf{ (j > 0: PA b +
(j
2:: QA
for every A}.
This distance has great appeal for robustniks, who interpret the delta halo as a way of constraining small migrations of Q mass and the added delta as insurance against a small proportion of gross changes. To us it will be just another metric for weak convergence. It is not obvious that p is symmetric, one of the properties required of a metric. We need to show that QAii + (j 2:: PA for every A, whenever p(P, Q) < (j. Set B equal to the complement of Ab. We know that QB :s; PBo + (j. Subtract both sides from 1, after replacing Bb by the complement of A, a larger set. (No point of A can be less than (j from a point in B.) We have symmetry. If p(P, Q) = 0 then certainly PFo + (j 2:: QF for every closed F and every (j > O. Hold F fixed but let (j tend to zero through a sequence of values. The sequence {FO} shrinks to F, giving PF 2:: QF in the limit. Interchange the roles of P and Q then repeat the argument to deduce that P and Q agree on all closed sets, and hence (Problem 11) on all borel sets. For the triangle inequality, suppose that p(P, Q) < (j and p(Q, R) < YJ. Temporarily set B = A~. Then
RA :s; QA~
+ YJ =
QB
+ YJ
+ YJ + (j. p(R, P) :s; YJ + (j. :s; PBo
Check that AO+~ contains BO. Deduce that Next, show that weak convergence implies convergence in the p metric. It suffices to deduce that p(P n, P) :s; (j eventually if P n ~ P. For each borel set A define
fA(X) = [1 -
(j-l d(x, A)] +.
Notice that AO 2:: fA 2:: A. Also, because
the class of all such fA functions is equicontinuous. By Example 19, sup IPnfA - PfAI ~ O. A
76
IV. Convergence in Distribution in Metric Spaces
Call this supremum
~n'
Then
P Ab ;;:: PfA ;;:: P nfA - ~n ;;:: P nA - ~n
for every A. Wait until ~n ~ 15 to be able to assert that p(P, Pn) Finally, if p(P n, P) ~ 0 then, for fixed closed F,
~
15.
limsup PnF ~ PF b + 15 for every 15 > O. Let 15 decrease to zero then deduce from Problem 12 that P n ~ P. Convergence in the p metric is equivalent to weak convergence. 0
IV.4. Coupling The Representation Theorems of Sections III.6 and IY.3 both depended upon methods for coupling distributions P nand P. That is, we needed to construct random elements Xn and X, on the same probability space, such that X n had distribution P n and X had distribution P. Closeness of P nand P, in a weak convergence sense, allowed us to choose Xn and X close in a stronger, almost sure sense. This section will examine coupling in more detail. A coupling of probability measures P and Q, on a space !!C, can be realized as a measure M on the product space !!C @ !!C, with X and Y defined by the coordinate projections. The product measure P @ Q is a coupling, albeit a not very informative one. More useful are those couplings for which M concentrates near the diagonal. For example, in the Representation Theorem we put as much mass as possible on the set {(x, y): d(x, y) ~ e}. Roughly speaking, one can construct such couplings in two steps. First treat the desired property-that as much mass as possible be allocated to a particular region D in the product space-as a strict requirement. Imagine building up M slowly by drawing off mass from the P marginal measure and relocating it within D, subject to a matching constraint: to put an amount 15 near (x, y) one must deplete the P supply near x by 15 and the Q supply near y by 15. When as much mass as possible has been shifted into D by this method, forget about the constraint imposed by D. In the second step, complete the transfer of mass from P into the product space subject only to the matching constraint. The final M will have the correct marginals, P and Q. A precise formulation of the coupling algorithm just sketched is easiest when both P and Q concentrate on a finite set of points. The first step can be represented by a picture that looks like a crossword puzzle. Label the points on which Q concentrates as 1, ... , r; let these correspond to rows of a two-way array of cells. Similarly, let 1, ... , c label both the points on which P concentrates and the columns ofthe two-way array. The stack beside row i represents the mass Q puts on point i, and the stack under column j represents the mass P puts on j. The unshaded cells correspond to D. The
77
IV.4. Coupling
aim is to place as much mass as possible in the unshaded cells without violating the constraint that the total mass in a row or column should not exceed the amount originally in the marginal stacks.
Q
p
This formulation makes sense even if the marginal supplies don't both correspond to measures with total mass one. In general we could allow any non-negative masses R(i) and CU) in the supply stacks for row i and column j. We would seek a non-negative allocation M(i, j) of as much mass as possible into the un shaded cells, subject to
L M(i,j) ~ CU)
and
i
L M(i,j) ~ R(i) j
for each i and j. A continuous analogue of the classical marriage lemma (a sort of fractional polygamy) will give the necessary and sufficient conditions for existence of an M that turns the inequalities for the columns into equalities. Treat C and R as measures. Write C(J) for the sum of supply masses in a set of columns J. Denote by DJ the set of rows i for which cell (i, j) belongs to D for at least one columnj in J. It is easy to see that M can have column marginal C only if R(D J) ;::: C(J) for every J, because the rows in DJ contain all the D-cells in the columns of J. Sufficiency is a little trickier. 24 Allocation Lemma. If R(D J) ;::: C(J) for every set of columns J, then there exists an allocation M(i, j) into the cells of D such that
L M(i, j) =
CU)
and
L M(i, j) ~ R(i) j
for every i and j. PROOF. Use induction on the number of columns. The result is trivial for c = 1. Suppose it is true for every number of columns strictly less than c. Construct M by transferring mass from the column margins into D. Shift mass at a constant rate into each of the D-cells in row r. For any mass
78
IV. Convergence in Distribution in Metric Spaces
shifted from C(j) into (r, j) discard an equal amount from R(r). If R(r) becomes exhausted, move on to row r - 1, and so on. Stop when either: (i) some C(j) is exhausted; or (ii) one of the constraints R(D J) ;;:: C(J) would be violated by continuation of the current method of allocation. Here Rand C are used as variable measures that decrease as mass is drawn off; the supply stacks diminish as the allocation proceeds. Notice that the mass transferred at each step can be specified as the largest solution to a system of linear inequalities. If the allocation halts because of (i), the problem is transformed into an allocation for c - 1 columns. The inductive hypothesis can be invoked to complete the allocation. If allocation halts because of (ii), then there must now exist some K for which R(D K ) = C(K). Continued allocation would have caused R(D K ) < C(K). The matching-constraint prevents K from containing every column: the total column supply always decreases at the same rate as the total row supply. Write KC for the non-empty set of columns not in K. K'
K
c If the marginal demands of the columns in K are to be met, the entire remaining supply R(D K ) must be devoted to those columns. With this requirement the problem splits into two subproblems: rows in DK may match only mass drawn off from the columns in K; from the rows D'k not in DK , match mass from the columns in KC. Both subproblems satisfy the initial assumptions of the lemma. For subsets of K this follows because allocation halted before R(D J) < C(J) for any J. For subsets of KC, it follows from R(D J n D'K)
= R(D JuK ) - R(D K ) ;;:: C(J u K) - C(K) = C(J).
Invoke the inductive hypothesis for both subproblems to complete the proof of the lemma. 0
79
IV.4. Coupling
25 Corollary. If Rand C have the same total mass and R(D]) ;;::: C(J) for every J, then the allocation measure M has marginal measures Rand C. 0
The Allocation Lemma applies directly only to discrete distributions supported by finite sets. For distributions not of that type a preliminary discretization, as in the proof of the Representation Theorem, is needed. 26 Example. Let P and Q be borel probability measures on a separable metric space. The Prohorov distance pep, Q) determines how closely P and Q can be coupled, in the sense that pep, Q) equals the infimum of those values of c such that (27) IP{d(X, Y) ;;::: c} ::; c,
with X having distribution P and Y having distribution Q. We can use the Allocation Lemma to help prove this. Half of the argument is easy. From (27) deduce, for every A, QA
= IP{YE A} ::; IP{X EA'} ::; PA' + c,
+ IP{d(X,
Y) ;;::: c}
whence pep, Q) ::; c. For the other half of the argument suppose pep, Q) < c. Construct X and Y by means of a two-stage coupling. Apply the method of Lemma 15 twice to partition the underlying space into sets Bo, B l , . . . , Bk with both QBo < band PB o < b, and diameter(B;) < b for i = 1, ... , k. Choose b as a quantity much smaller than c; it will eventually be forced down to zero while c stays fixed. The requirement that each B; be a Q- or P-continuity set is irrelevant to our present purpose. Set R(i) equal to QB; and C(j) equal to PB j • Into the region D allow only those cells (i, j), for 1 ::; i ::; k and 1 ::; j ::; k, whose corresponding B; and B j contain a pair of points, one in B; and one in Bj , a distance::; c apart. Augment the double array by one more row, call it 00, whose row stack contains mass c + 2b. Include (00,0), ... , (00, k) in the region D.
80
IV. Convergence in Distribution in Metric Spaces
The hypotheses of the Allocation Lemma are satisfied. For any collection of columns J, C(J)
s
PB o +
p( UB
j)
JI{O)
Q( UBj)E + /; b + Q( UBi) + QB o + /;
< b+
JI{O)
S
by definition of D
DJI{co)
< b + R(D J \ {oo}) + b + /;
= R(D J ). Distribute all the mass from the column stacks into D, as in the Allocation Lemma. The 00 row acts as a temporary repository for the small amount of mass that cannot legally be shifted into the desired small-diameter cells. Return the mass in this row to the column stacks, leaving at least 1 - /; - 2b of the original C mass in the desired cells. Strip away the 00 row. Allocate the remaining mass in the column stacks after expanding D to include all cells (i, j), for 0 s i s k and 0 s j s k. So far we have only decided the allocation of masses MO, j) between the cells. Within the cells distribute according to the product measures M(i,j)
Q(·JBJ(8)P(.JB).
The resulting M on PE (8) PE has marginal measures P and Q. For example, within Bo the column marginal is
L MO, O)Q(BiJBi)P(·JB o) =
P(Bo)P(·JB o) = P(·B o)·
i
The M measure concentrates at least 1 - /; - 2b of its mass within the original D, a cluster of cells each of diameter less than b in both row and column directions. For a point (x, y) lying in a cell (i,j) of this cluster, there exists points Zi and Zj with d(Zj,
y) < b,
which gives d(x, y) < /; + 2b. Put another way, if X and Y denote the coordinate projections then IP{d(X, Y) 2: /;
+ 2b} s /; + 215.
As b can be chosen arbitrarily small, and /; can be chosen as close to pep, Q) as we please, we have the desired result. Problem 17 gives a condition under which the bound pep, Q) can be achieved by a coupling of P and Q. 0
IV.S. Weakly Convergent Subsequences
81
IV.5. Weakly Convergent Subsequences A reader not interested in existence theorems could skip this section, which presents a method for constructing measures on metric spaces. The results will be used in Section V.3 to prove existence of the brownian bridge. The method will be generalized in Chapter VII. We saw in Section 111.6 how to modify the quanti le-transformation construction of the one-dimensional Representation Theorem to turn it into an existence theorem, a method for constructing a probability measure as the distribution of the almost sure limit of a sequence of random variables. We had to impose a uniform tightness constraint to stop the sequence from drifting off to infinity. The analogous result for probabilities on metric spaces plays a much more important role than in euclidean spaces, because existence theorems of any sort are so much harder to come by in abstract spaces. Again the key to the construction is a uniform tightness property, which ensures that sequences that ought to converge really do converge. The setting is still that of a metric space f1£ equipped with a sub-a-field .91 of its borel a-field. 28 Definitions. Call a probability measure P on .91 tight if for every e > 0 there exists a compact set K(e) of completely regular points such that PK(e) > 1 - e. Call a sequence {P n } of probability measures on si uniformly tight if for everye > 0 there exists a compact set K(e) of completely regular points such that liminf PnG > 1 - e for every open, si-measurable G containing K(e). D Problem 7 justifies the implicit assumption of si-measurability for the K(e) in the definition of tightness; every compact set of completely regular points can be written as a countable intersection of open, si-measurable sets. If G is replaced by K(e), the uniform tightness condition becomes a slightly tidier, but stronger, condition. It is, however, more natural to retain the open G. If P n ~ P and P is tight then, by virtue of the results proved in Example 17, the liminf condition for open G is satisfied; it might not be satisfied if G were replaced by K(e). More importantly, one does not need the stronger condition to get weakly convergent sub sequences, as will be shown in the next theorem. For the proof of the theorem we shall make use of a property of compact sets: If {xn} is a Cauchy sequence in a metric space, and if d(xn' K) --+ 0 for some fixed compact set K, then {xn} converges to a point of K. This follows easily from one of a set of alternative characterizations of compactness in metric spaces. As we shall be making free use of these characterizations in later chapters, a short digression on the topic will not go amiss.
82
IV. Convergence in Distribution in Metric Spaces
To prove the assertion we have only to choose, according to the definition of d(xn' K), points {Yn} in K for which d(x n, Yn) ~ O. From {Yn} we can extract a subsequence converging to a point Y in K. For if no subsequence of {Yn} converged to a point of K, then around each x in K we could put an open neighborhood Gx that excluded Yn for all large enough values of n. This would imply that {Yn} is eventually outside the union of the finite collection of G x sets covering the compact K, a contradiction. The corresponding subsequence of {xn} also converges to y. The Cauchy property forces {xn} to follow the subsequence in converging to y. A set with the property that every sequence has a convergent subsequence (with limit point in the set) is said to be sequentially compact. Every compact set is sequentially compact. This leads to another characterization of compactness: A sequentially compact set is complete (every Cauchy sequence converges to a point of the set) and totalIy bounded (for every positive s, the set can be covered by a finite union of closed balIs of radius less than s). For clearly a Cauchy sequence in a sequentially compact K must converge to the same limit as the convergent subsequence. And if K were not totalIy bounded, there would be some positive s for which no finite collection of balls of radius s could cover K. We could extract a sequence {xn} in K with Xn+l at least s away from each of Xl' ... ' xn for every n. No subsequence of {xn} could converge, in defiance of sequential compactness. For us the last link in the chain of characterizations will be the most important: A complete, totalIy bounded subset of a metric space is compact. Suppose, to the contrary, that {G;} is an open cover of a totalIy bounded set K for which no finite union of {GJ sets covers K. We can cover K by a finite union of closed balls of radius t, though. There must be at least one such ball, Bl say, for which K n Bl has no finite {G;} subcover. Cover K n Bl by finitely many closed balls of radius i. For at least one of these balls, B2 say, K n Bl n B2 has no finite {G;} subcover. Continuing in this way we discover a sequence of closed balls {Bn} of radii {rn} for which K n Bl n ... n Bn has no finite {G;} cover. Choose a point Xn from this (necessarily non-empty) intersection. The sequence {xn} is Cauchy. If K were also complete, {xn} would converge to some X in K. Certainly x would belong to some Gi , which would necessarily contain Bn for n large enough. A single G i is about as finite a subcover as one could wish for. Completeness would indeed force {G;} to have a finite subcover for K. End of digression. 29 Compactness Theorem. Every uniformly tight sequence of probability measures contains a subsequence that converges weakly to a tight borel measure.
IV.S. Weakly Convergent Subsequences
83
PROOF. Write {P n } for the uniformly tight sequence, and Kk for the compact set K(Ck), for a fixed sequence {Ck} that converges to zero. We may assume that {K k } is an increasing sequence of sets. The proof will use a coupling to represent a subsequence of {P n } by an almost surely convergent sequence of random elements. The limit of these random elements will concentrate on the union of the compact Kk sets; it will induce the tight borel measure on f1£ to which the subsequence {P n } will converge weakly. Complete regularity of each point in Kk allows us to cover Kk by a collection of open d-measurable sets, each of diameter less than Ck' Invoke compactness to extract a finite subcover, {U ki : 1 :::; i:::; id. Define d m to be the finite subfield of d generated by the open sets Uki for 1 :::; k :::; m and 1 :::; i :::; i k • The union of the fields {dm} is a countable subfield d 00 of d. Apply Cantor's diagonalization argument to extract a subsequence of {P n } along which lim PnA exists for each A in d oo • Write AA for this limit. It is a finitely additive measure on the field d 00' Avoid the mess of double-subscripting by assuming, with no loss of generality, that the subsequence is {P n } itself. If {P n} were weakly convergent to a measure P we would be able to deduce that P(interior of A) :::; AA :::; P( closure of A) for each A in ,xl 00' If we could assume further that P put zero mass on the boundary of each such A, we would know the P measure of enough sets to allow almost surely convergent representing sequences to be constructed as in the Representation Theorem. Unfortunately there is no reason to expect P to cooperate in this way. Instead, we must turn to A as a surrogate for the unknown, but sought after, probability measure P. Since Aneed not be countably additive, it would be wicked of us to presume the existence of a random element of f1£ having distribution A. We must take a more devious approach. We can build a passable imitation of d 00 on the unit interval. Partition (0, 1) into as many intervals as there are atoms of d 1> making the lebesgue measure of each interval A equal to the A measure of the corresponding A in ,xl l' These intervals generate a finite field .91 1 on (0, 1). Partition each atom A in .91 1 into as many subintervals as there are atoms of d 2 in A, matching up lebesgue and A measures as before. The sub intervals together generate a second field .912 on (0, 1), finer than .91 1 , Continuing in similar fashion, we set up an increasing sequence of fields {,91 k } on (0, 1) that fit together in the same way as the fields {dd on f1£. The union of the dk's is a countable subfield .91 00 of (0, 1). There is a bijection A +-+ A between d 00 and d 00 that preserves inclusion, maps d k onto db and preserves measure, in the sense that the lebesgue measure of A equals AA. The construction ensures that, if YJ has a Uniform(O, 1) distribution, IP{YJ E A} = AA for every A in d 00' The random variable YJ chooses between the sets in d k in much the same way as a random element X with distribution P would choose between the sets in d k •
84
IV. Convergence in Distribution in Metric Spaces
By definition of A, there exists an n(k) such that (30)
PnA
~
(1 - Gk)AA
for every A in d
k
whenever
n ~ n(k).
Lighten the notation by assuming that n(k) = k. (If you suspect these notational tricks for avoiding an orgy of subsequencing, feel free to rewrite the argument using, by now, triple subscripting.) As in the proof of the Representation Theorem, this allows us to construct a random element X n , with distribution Pn, by means of an auxiliary random variable ~ that has a Uniform(O, 1) distribution independent of 1]: For each atom A of d n' if I] falls in the corresponding A of d nand ~ S 1 - Gn, distribute X n on A according to the conditional distribution Pn(·IA). If ~ > 1 - Gn distribute Xn with whatever conditional distribution is necessary to bring its overall distribution up to Pn' We have coupled each Pn with lebesgue measure on the unit square. To emphasize that Xn depends on 1], ~, and the randomization necessary to generate observations on Pn(·IA), write it as Xn(w, 1], ~). Notice that the same I] and ~ figure in the construction of every X n' It will suffice for us to prove that {Xn(w, 1], ~)} converges to a point X(w, 1], ~) of Kk for every wand every pair (I], ~) lying in a region of probability at least (1 - Gk)2, a result stronger than mere almost sure convergence to a point in the union of the compact sets {K k }. Problem 16 provides the extra details needed to deduce borel measurability of X. For each m greater than k, let Gmk be the smallest open, d m-measurable set containing K k • Uniform tightness tells us that
AGmk
=
liminf PnGmk > 1 - Gb
which implies IP{I] E Gmd > 1 - Gk' Define Gk as the intersection of the decreasing sequence of sets {Gmd for m = k, k + 1, .... The overbar here is slightly misleading, because Gk need not belong to d 00' But it is a borel subset of (0, 1). Countable additivity oflebesgue measure allows us to deduce that IP{I] E Gk } ~ 1 - Gk' Notice how we have gotten around lack of countable additivity for A, by pulling the construction back into a more familiar measure space. Whenever I] falls in Gk and ~ S 1 - Gk' which occurs with probability at least (1 - Gk)2, the random elements Xb Xk+l"" crowd together into a shrinking neighborhood of a point of K k • There exists a decreasing sequence {Am} with: (i) Am is an atom of d m; (ii) Am is contained in Gmk ; (iii) X m(W, 1], ~) lies in Am. Properties (i) and (iii) are consequences of the method of construction for Xm; property (ii) holds because Gk is a subset of Gmk . The set Gmb being the
85
Notes
smallest open, .s1m-measurable set containing Kb must be contained within the union of those Umi that intersect K k • The atom Am must lie wholly within one such Umi' a set of diameter less than em' So whenever 1] falls in Gk and ~ :s; 1 - eb the sequence {X m} satisfies: (i) d(Xm(w, 1], (ii) d(Xm(w, 1],
~), ~),
m :s; em
Xiw, 1], K k ) :s; em
for k :s; m :s; n; for k :s; m.
As explained at the start of the digression, this forces convergence to a point X(w, 1], ~) of K k • 0 NOTES
Any reader uncomfortable with the metric space ideas used in this chapter might consult Simmons (1963, especially Chapters 2 and 5). The advantages of equipping a metric space with a a-field different from the borel a-field were first exploited by Dudley (1966a, 1967a), who developed a weak convergence theory for measures living on the a-field generated by the closed balls. The measurability problem for empirical processes (Example 2) was noted by Chibisov (1965); he opted for the Skorohod metric. Pyke and Shorack (1968) suggested another way out: Xn ~ X should mean IPf(Xn) --7 IPf(X) for all those bounded, continuous f that make f(Xn) and f(X) measurable. They noted the equivalence of this definition to the definition based on the Skorohod metric, for random elements of D[O, 1J converging to a process with continuous sample paths. Separability has a curious role in the theory. With it, the closed balls generate the borel a-field (Problem 6); but this can also hold without separability (Talagrand 1978). Borel measures usually have separable support (Dudley 1967a, 1976, Lecture 5). Alexandroff (1940, 1941, 1943) laid the foundation for a theory of weak convergence on abstract spaces, not necessarily topological. Prohorov (1956) reset the theory in complete, separable metric space, where most probabilistic and statistical applications can flourish. He and LeCam (1957) proved different versions of the Compactness Theorem, whose form (but not the proof) I have borrowed from Dudley (1966a). Weak convergence of baire measures on general topological spaces was thoroughly investigated by Varadarajan (1965). Tops0e (1970) put together a weak convergence theory for borel measures; he used the liminfproperty for semicontinuous functions (Example 17) to define weak convergence. These two authors made clear the need for added regularity conditions on the limit measure and separation properties on the topology. One particularly nice combination -a completely regular topology and a 'L-additive limit measure-corresponds closely to my assumption that limit measures concentrate on separable sets of completely regular points. The best references to the weak convergence theory for borel measures on metric spaces remain Billingsley (1968, 1971) and Parthasarathy (1967).
86
IV. Convergence in Distribution in Metric Spaces
Dudley's (1976) lecture notes offer an excellent condensed exposition of both the mathematical theory and the statistical applications. Example 11 is usually attributed to Wichura (1971), although Hajek (1965) used a similar approximation idea to prove convergence for random elements of C[O, 1]. Skorohod (1956) hit upon the idea of representing sequences that converge in distribution by sequences that converge almost surely, for the case of random elements of complete, separable metric spaces. The proof in Section 3 is adapted from Dudley (1968). He paid more attention to some of the points glossed over in my proof-for example, he showed how to construct a probability space supporting all the {X n }. Here, and in Section 5, one needs the existence theorem for product measures on infinite-product spaces. Pyke (1969,1970) has been a most pers)..lasive advocate of this method for proving theorems about weak convergence. Many of the applications now belong to the folklore. The uniformity result of Example 19 comes from Ranga Rao (1962); Billingsley and Topsq'le (1967) and Topsq'le (1970) perfected the idea. Not surprisingly, the original proofs of this type of result made direct use of the dissection technique of Lemma 15. Prohorov (1956) defined the Prohorov metric; Dudley (1966b) defined the bounded Lipschitz metric. Strassen (1965) invoked convexity arguments to establish the coupling characterization of the Prohorov metric (Example 26). My proof comes essentially from Dudley (1968), via Dudley (1976, Lecture 18), who introduced the idea of building a coupling between discrete measures by application of the marriage lemma. The Allocation Lemma can also be proved by the max-fIow-min-cut theorem (an elementary result from graph theory; for a proof see Bollobas (1979)). The conditions of my Lemma ensure that the minimum capacity of a cut will correspond to the total column mass. Appendix B of Jacobs (1978) contains an exposition of this approach, following Hansel and Troallic (1978). Major (1978) has described more refined forms of coupling. PROBLEMS
[1] Suppose the empirical process U 2 were measurable with respect to the borel a-field on D[O, 1] generated by the uniform metric. For each subset A of (1,2) define J A
as the open set of functions in D[O, 1] with jumps at some pair of distinct points tl and t2 in [0, 1] with tl + t2 in A. Define a non-atomic measure on the class of all subsets of (1,2) by setting yeA) = IP{U 2 E J A}' This contradicts the continuum hypothesis (Oxtoby 1971, Section 5). Manufacture from l' an extension of the uniform distribution to all subsets of (1, 2) if you would like to offend the axiom of choice as well. Extend the argument to larger sample sizes. [2] Write.s;{ for the a-field on a set 2l' generated by a family {j;} of real-valued functions on 2l'. That is, d is the smallest a-field containing fi- 1 B for each i and each borel set B. Prove that a map X from (n, 0') into 2l' is0'/d-measurableifand only if the composition j; X is 0'/~(IR)-measurable for each i. 0
Problems
87
[3J Every function in D[O, IJ is bounded: [x(t n )[--+ CIJ as n --+ CIJ would violate either the right continuity or the existence of the left limit at some cluster point of the sequence {tn}. [4J Write g> for the projection a-field on D[O, IJ and PAo for the a-field generated by the closed balls of the uniform metric. Write n, for the projection map that takes an x in D[O, IJ onto its value x(t). (a) Prove that each n, is PAo-measurable. [Express {x: n,x > cc} as a countable union of closed balls B(x., n), where Xn equals cc plus (n + n- 1 ) times the indicator function of et, t + n-1).J Deduce that PAo contains ~ (b) Prove that the a-field g> contains each closed ball B(x, r). [Express the ball as an intersection of sets {z: [n,x - n,z[ ~ r}, with t rational.J Deduce that g> contains PAo. [5J Let {G;} be a family of open sets whose union covers a separable subset C of a metric space. Adapt the argument of Lemma 7 to prove that C is contained in the union of some countable subfamily of the {G;}. [This is Linde16f's theorem.J [6J Every separable, open subset of a metric space can be written as a countable union of closed balls. [Rational radii, centered at points of the countable dense set.J The closed balls generate the borel a-field on a separable metric space. [7J Every closed, separable set of completely regular points belongs to d. [Cover it with open, d-measurable sets of small diameter. Use Linde16f's theorem to extract a countable subcover. The union of these sets belongs to d. Represent the closed set as a countable intersection of such unions.J [8J Let Co be the countable subset of C[O, IJ consisting of all piecewise linear functions with corners at only a finite set of rational pairs (t i , rJ Argue from uniform continuity to prove that C[O, 1J equals the closure of Co. Deduce that C[O, IJ is a projection-measurable subset of D[O, 1]. [9J A function h is said to be lower-semicontinuous at a point x if, for each M < h(x), h is greater than M in some neighborhood of x. To say h is lower-semicontinuous means that it is lower-semicontinuous at every point. Show that the upper envelope of any set of continuous functions is lower-semicontinuous. Adapt the construction of Lemma 7 to prove that every lower-semicontinuous function that is bounded below can be represented on a separable set of completely regular points as the pointwise limit of an increasing sequence of continuous functions. How would one define upper-semicontinuity? Which sets should have upper-semicontinuous indicator functions? What does a combination of both semicontinuities imply? [10J If Xn ..... X as random elements of a metric space f!£ and d(X n, y") --+ 0 in probability, then Y" ..... X, provided that IPx concentrates on a separable set of completely regular points. [Convergence in probability means IP*{d(Xn, y") > e} --+ 0 for each e > O.J [11 J Let P be a borel measure on a metric space. For every borel set B there exists an open G, containing B and a closed F, contained in B with P(G,\F,) < e. [The class of all sets with this property forms a a-field. Each closed set has the property because it can be written as a countable intersection of open sets.J Deduce that P is uniquely determined by the values it gives to closed sets. Extend the result to measures defined on the a-field generated by the closed balls.
88
IV. Convergence in Distribution in Metric Spaces
[12] Suppose limsup PnF ::s; PF for each P n ...... P by applying the inequalities
~losed,
d-measurable set F. Prove that
00
k- 1
L U~ ilk}
00
::s;
f
::s;
k- 1
+ k- 1 L U~
ilk}
i=l
i=1
for each non-negative f in ~(,q[; d). [The summands are identically zero for all i large enough. Apply the same argument to - f + (a big constant).] [13] If PnB -+ PB for each d-measurable set B whose boundary has zero P measure then P n ...... P. [Replace the levels ilk of the previous problem by levels ti for which PU = t,} = 0.]
[14] The functions in ~(,q[; d) generate a sub-u-field :!Bc of d. A map X from (Q, <9') into,q[ is <9'/:!Bc-measurable if and only if f(X) is <9'/:!B(IR)-measurable for each f in ~(,q[;
d).
[15] (Continued). The trace of :!Bc on any separable set S of completely regular points of,q[ coincides with the borel u-field on S. [Sets of the form U > O} I l S, with f in ~(,q[; d), form a basis for the relative topology on S. Every relatively open subset of S is a countable union of such sets, by LindelOf's theorem.] [16] (Continued). Let {Xm} be a sequence of <9'/d-measurable random elements of,q[ that converges pointwise to a map X. Prove that X is <9'/:!Bc-measurable. If X takes values only in a fixed separable set of completely regular points, then it is <9'/:!B(,q[)-measurable.
[17] Let P and Q be tight probability measures on the borel u-field of a separable metric space f£. There exists a coupling for which IP{d(X, Y) ~ M ::s; d, where d = p(P, Q), the Prohorov distance between P and Q. [From Example 26, there exist measures M n on ,q[ ® gr, wi1th mar-gin~ls ! and Q, for which Mn{(x, y): d(x, y) ~ d
+ n- 1 }
::s; d
+ n- 1 •
The sequence {Mn} is uniformly tight. The limit of a weakly convergent subsequence defines the required coupling. Is separability of,q[ really needed?]
[18] Let,q[ be a compact metric space, and ~(,q[) be the vector space of all bounded, continuous, real functions on ,q[. Let T be a non-negative linear functional on ~(,q[) with T1 = 1. These steps show that Tf = Pf for some borel probability measure P: (a) Given y > 0 find functions gj, ... , gk in ~+(,q[) with diameter{gi > O} < 2y and gl + ... + gk = 1. [Find fx in ~+(,q[) with fix) > 0 and fiy) = 0 for d(y, x) > y. Cover K by finitely many open sets Ux > O}. Standardize the corresponding functions to sum to one everywhere.] (b) Choose Xi with gi(X i) > O. Define P y as the discrete probability measure that puts mass Tg i at Xi' for each i. If h belongs to ~(,q[), show that 1P yh - Th 1-+ 0 as y -+ O. (c) Extract a subsequence of {Py} that converges weakly. Show that the limit measure P has the desired property.
CHAPTER V
The Uniform Metric on Spaces of Cadlag Functions ... in which random elements of metric spaces of cadlag functions-stochastic processes whose sample paths have at worst simple jump discontinuities-are treated. Necessary and sufficient conditions for convergence in distribution are found then specialized to prove limit theorems for empirical processes and processes with independent increments. The two most commonly occurring limit processes, brownian motion and brownian bridge, are studied.
V.I. Approximation of Stochastic Processes The theory developed in Chapter IV justifies its existence by what it has to say about the limiting distributions of functionals defined on sequences of stochastic processes. Processes {Xn(t): t ET} are identified with random elements of some space f![ of functions on T, a space large enough to contain the sample paths of every X n; the functionals are maps defined on f![ to which the Continuous Mapping Theorem can be applied. In this chapter we shall specialize the general theory to the particular function space D[O, 1], under its uniform metric. It will turn out that most applications, especially those that come up with brownian bridges and brownian motions as limit processes, require no fancier setting than this. Recall that for a function defined on a subset T of the extended real line [ - 00, 00] to deserve the title cadlag it must be right continuous ("continue a droite") and have left limits ("limites it gauche") at each point of T. Many of the well-studied stochastic processes-processes with independent increments, or the markov property, or some form of martingale structurehave versions with cadlag sample paths. For T = [0, 1], these processes can be analyzed as random elements of D[O, 1], the set of all real-valued cadlag functions on [0, 1]. Notice that right continuity at 1 and existence of the left limit at 0 become empty requirements for this space. The analysis for other spaces of cadlag functions on compact intervals (such as D[ - 00, 00], the natural space in which to study empirical distribution functions over the real line ), and the extensions to non-compact index sets (Section 5), will call for only slight variations on the methods developed for D[O, 1].
90
V. The Uniform Metric on Spaces of Cadlag Functions
For each finite subset S of [0, 1] write ns for the projection map from D[O, 1] into IR s that takes an x onto its vector of values {x(t): t E S}. Abbreviate nIt} to n t . These projections generate the projection tT-field, fJ>. Questions of measurability in this chapter will always refer to this tT-field. A stochastic process X on (Q, fff, JP) with sample paths in D[O, 1], such as an empirical process, is fff/&'-measurable provided n t X is fff/gQ(IR)-measurable for each fixed t (Problem IV.2). Probability measures on &' are uniquely determined by the values they give to the generating sets {ns 1 B}, with S a finite subset of [0, 1] and B a bore! subset of IRs. Equivalently, the distribution of a random element of D[O, 1] is uniquely specified by giving the distributions of all its finitedimensional projections. As every cadlag function on [0, 1] is bounded (Problem IV.3), the uniform distance 0
d(x, y) =
Ilx - yll = sup{ Ix(t) - y(t) I:
°: ;
t ::; 1}
defines a metric on D[O, 1]. No other metric will be used for D[O, 1] in this chapter. The closed balls for d generate &' but not the larger borel tT-field (Problem IVA). Every point in D[O, 1] is completely regular-see Definition IY.6 and the discussion preceding it. The difficulty with the tT-field, which can be blamed on the lack of a countable, dense subset of functions in D[O, 1], has dissuaded many authors from working with the uniform metric. Chapter IV showed that the difficulty can be surmounted, at least when limit distributions concentrate on separable subsets of D[O, 1]. As compensation for persisting with the uniform metric, we shall find its topological properties much easier to understand and manipulate than those of its main competitor, the Skorohod metric, which will be discussed in Chapter VI. That will make life more pleasant for us in Section 3 when we come to apply the Compactness Theorem. The limit processes for the applications in this chapter will always concentrate in a separable subset of D[O, 1], usually C[O, 1], the set of all continuous, real functions on [0, 1]. As a closed (uniform convergence preserves continuity), separable, &'-measurable subset of D[O, 1] (Problems IY.8), C[O, 1] has several attractive properties. For example, it inherits completeness from D[O, 1], and its projection tT-field coincides with its borel tT-field for the uniform metric (Problem IV.6). How do we establish convergence in distribution of a sequence {Xn} of random elements of D[O, 1] to a limit process X? Certainly we need the finite-dimensional projections {nsXn} to converge in distribution, as random vectors in IRs, to the finite-dimensional projection nsX, for each finite subset S of [0, 1]. Continuity of ns and the Continuous Mapping Theorem make that a necessary condition. The methods of Chapter III usually help here. But that alone could hardly suffice. Continuous functionals such as Mx = Ilxll, a typical non-trivial example, depend on x through more than its values at a fixed, finite S. Indeed, direct examination of that very functional gives the clue to the extra condition needed.
91
V.l. Approximation of Stochastic Processes
Intuitively, it should be possible to approximate MX n by taking the maximum of IXn(s) I over a large, finely spaced, finite subset S of [0,1]. If S expands up to a countable, dense subset of [0, 1] containing 1, then
MsXn = max IXn(s) 1--+ MX n
s for every sample path of X n. The cadlag property assures us of this. (Notice the special treatment demanded by 1.) Given any (j > and e > 0, a large enough S could be found to ensure that
°
(1)
At first sight this seems to have solved the problem. Because MsXn depends continuously on nsXn' it converges in distribution to MsX as n --+ 00. For f bounded and uniformly continuous, IPf(MsXn) --+ IPf(MsX). From (1) with (j chosen appropriately,
IIPf(MsXn) - IPf(MX n) I ~
e + 211flle.
A similar inequality holds for X. Taken together, these seem to add up to convergence in distribution of MX n to MX. But the argument is flawed. The problem lies with the choice of S in (1). If the sample paths of Xn jumped about more and more wildly as n increased, S would have to get bigger with n. That would undermine the convergence of {MsXn} to MsX, since finite-dimensional convergence says nothing about {nsXn} for S varying with n. We need the same S to work for each n. A similar argument can be invoked for any other continuous functional H on D[O, 1]. We approximate HX n by a continuous function of nsXn for some large, finite set S not depending on n. We construct the approximation by applying H to an element AsXn in D[O, 1]. For simplicity, suppose S has been rearranged into increasing order and augmented by the points 0 and 1, if necessary, to form a grid = So < SI < ... < Sk = 1. For x in D[O, 1] define the approximating path Asx by
°
(2)
(Asx)(t)
=
x(sJ
for
Si ~
t <
Si-I,
and (Asx)(1) = x(1). Notice that Asx depends on x only through nsx. In order that H(Asx) be close to Hx, it suffices that IIAsx - xii be small. Consider, for example, a uniformly continuous H. There exists a (j > such that IH(Asx) - Hx 1< e whenever IIAsx - xii < (j; the random variable HX n lies within e of H(AsXn) with probability no less than IP{IIAsXn - Xnll < (j}. Again, the same grid would have to work for every Xn-or at least for every Xn with n large enough-to allow finite-dimensional convergence to imply convergence in distribution of {AsXn} to AsX. (The example ofthe empirical process Un, whose sample paths have jumps of size n - 1/2, shows that uniform approximation of Xn by AsXn can only be required for large values of n, anyway.) If such a map As does exist, the argument sketched above for the supremum functional M will carry over to the functional H, proving that HX n r-> HX. (The argument might seem familiar-it is a special case of Example IV.ll).
°
92
V. The Uniform Metric on Spaces of Cadlag Functions
For the sake of brevity, from now on shorten "finite-dimensional distributions" to fidis and "finite-dimensional projections" to fidi projections. 3 Theorem. Let X, Xl> X z , ... be random elements of D[O, 1J (under its uniform metric and projection (J-field). Suppose IP{X E C} = 1 for some separable subset C of D[O, 1]. The necessary and sufficient conditions for {Xn} to converge in distribution to X are:
(i) the jidis of X n converge to the jidis of X; that is, ns X n '""* ns X for each jinite subset S of [0, IJ; (ii) to each c; > and b > 0 there corresponds a grid 0 = to < t 1 < ... < tm = 1
°
such that
(4) where J i denotes the interval Et;, t i + 1), for i
= 0, 1, ... , m - 1.
PROOF. Suppose that Xn '""* X. The projection map ns is both projectionmeasurable (by definition) and continuous. Condition (i) follows by the Continuous Mapping Theorem. To simplify the proof of (ii) suppose that the separable subset C equals C[O, 1]. Continuity of the sample paths makes the choice of the grid in (ii) easier. (Problem 4 outlines the extra arguments needed for the general case.) Let {so, S1> •.• } be a countable, dense subset of [0,1]. To avoid trivialities, assume that So = and SI = 1. Write Ak for the interpolation map constructed as in (2) from the grid obtained by rearranging so, ... ,Sk into increasing order. For fixed x in C[O, IJ, the distance IIAkx - xii converges to zero as k increases, by virtue of the uniform continuity of x. Applied to the sample paths of X, this shows that IIAkX - XII -> 0 almost surely. Convergence in probability would be enough to assure the existence of some k for which
°
(5)
Choose and fix such a k. Because IIAkx set F =
{x E D[O, 1]:
xii varies continuously with x, the
IIAkx -
xii;::.: 6}
is closed. By Example IV.17 (inequality IV.18 to be precise), limsup IP{Xn E F} :s; IP{X E F}. The left-hand side bounds the limsup in (4). We could require ";::.: 6" rather than" > b" in (4), but that would create a minor complication in the next lemma. Now let us show that (i) and (ii) imply X n '""* X. Retain the assumption that X has continuous sample paths (see Problem 4 for the general case),
93
V.!. Approximation of Stochastic Processes
so that (5) still holds. Choose any bounded, uniformly continuous, projectionmeasurable, real function f on D[O, 1]. Given 8 > find b > such that If(x) - fey) I < 8 whenever Ilx - yll ::; b. Write AT for the approximation map constructed from the grid in (ii) corresponding to this band 8. Condition (4) becomes
°
limsup IP{IIATX n - Xnll > b} <
°
8.
With no loss of generality we may assume that the Ak of (5) equals AT, because if we combine the two underlying grids we have at worst to replace b by 2b to preserve the two approximations. For example, if ti ::; Sj ::; t < t i + 1 then, whenever IIATX n - Xnll < b,
~ ! "--.
/' /'
//'///
:
:
.-~: ---- -:
/ ::
.: . ' . ' . . .
·· ·
'
··
ti
The argument now follows the lines sketched out before. Write the composition f AT as g n T where g is a bounded, continuous function on D[O, 1]-a fancy way of saying that f(ATx) depends on x continuously through the values that x takes at the grid points. 0
0
IIPf(XJ - IPf(X) I
::; IPlf(X n) - f(ATXn)1 + In>f(ATX n) - IPf(ATX) I + IPlf(ATX) - f(X)1 ::; 8 + 211f11IP{IIX n - ATXnl1 > b} + IIPg(nTXJ - IPg(nTX) I + 8 + 211f11IP{IIX - ATXII > b}. The middle term in the last line converges to zero as n fidi convergence nTXn r-> nTX.
---+ 00,
because of the D
We now know the price we pay for wanting to make probability statements about functionals that depend on the whole sample path of a stochastic process: with high probability we need to rule out nasty behavior between the grid points. For uniform approximation of sample paths, a large value of IX net) - Xn(t i) I would be nasty; inequality (4) rules it out. So how do we
94
V. The Uniform Metric on Spaces of Cadlag Functions
control the left-hand side of (4)? It involves the probability of a union of m events, which we may bound by the sum of the probabilities of those events,
~t~ IP{s~p IXit) -
(6)
Xit i ) I > <5}-
Then we can concentrate on what happens in an interval J i between adjacent grid points. For many stochastic processes, good behavior of the increment Xit i + 1 ) - Xn(t i ) forces good behavior for the whole segment of sample path over J i • 7 Lemma. Let {Z(t): 0::;; t::;; b} be a process with cadlag sample paths taking the value zero at t = 0. Suppose Z(t) is IS't-measurable, for some increasing family of (J-jields {lS't: t ::;; b}. If at each point of {IZ(t) I > <5},
°: ;
IP{IZ(b) - Z(t) I ::;; !IZ(t)I!lS't} 2: [3,
where [3 is a positive number depending only on <5, then
IPts~~bIZ(t)1 > <5}::;; [3-1IP{IZ(b)1 > H}· Let S be a finite subset of [0, b] containing the point b. By virtue of right continuity, as S expands up to a countable, dense subset of [0, b],
PROOF.
max IZ(t) I -+ sup IZ(t) I for every sample path of Z. s
[O,~
Notice the minor complication here if" > <5" were replaced by" 2: <5". It is good enough to establish the inequality with the supremum taken over S. Define r as the first point of S for which IZ(t) I > <5 if there is such a t, otherwise set r = 00. The event {r = t} belongs to IS't. For distinct points t and t f in S the events {r = t} and {r = tf} are disjoint. This justifies a firstpassage decomposition. IP{m:x1Z(t)1 > <5}
=
~ IP{r =
::;; p-l I
t}
IP[{r = t}IP{IZ(b) - Z(t) I ::;; !IZ(t)I!lS't}]
s
= p-lI, IP{r = t, iZ(b) - Z(t) I ::;; !IZ(t)l} s
::;; p-l I, IP{r
= t, IZ(b) I > !<5}
s
o For the lemma to help us we need to know something about the conditional distribution of Z(b) - Z(t) given what has happened up to time t. When do we have such information? Here are some possibilities. The distribution of the increment might not depend on the past at all-the
95
V.2. Empirical Processes
process might have independent increments. In that case the requirement imposed by the lemma reduces to something involving the marginal distributions of the increments. Or the distribution of Z(b) - Z(t) might depend on the past only through the value of Z(t)-the markov property. In that case the requirement turns into a condition on the transition probabilities. Or Z(b) - Z(t) might merely have zero conditional expectation-the martingale property. And then we have all those clever stopping time and maximal inequality tricks to play with. In each of these cases it turns out to be quite easy to find sufficient conditions for strengthening fidi convergence to the full convergence in distribution of the processes as random elements of D[O, 1]. Sections 4 and 2 illustrate the first two possibilities; Chapter VIII will play tricks with martingales. In each caSe the limit process will be either a brownian bridge, a brownian motion, or a close relative. 8 Definition. A brownian motion paths and
IS
a process with continuous sample
(i) B(O) = 0; (ii) for ~ t 1 < ... < tk the increments B(tz) - B(t 1), ... , B(t k) - B(t kare mutually independent; each B(tJ - B(t i - 1 ) has a N(O, ti - t i -
°
1) 1)
distribution. A brownian bridge, or tied-down brownian motion, is a process on [0, 1] with continuous sample paths and (i) U(O) = U(l) = 0; (ii) for each finite subset S of [0, 1] the random vector 11:s U has a multivariate normal distribution with zero means and covariances given by IPU(s)U(t) = s(l - t) for s ~ t. D
The name tied-down brownian motion comes from one method for constructing a random element distributed like U. Apply to B the continuous transformation that takes the function x(t) onto the function x(t) - tx(l). That is, tie down the loose end of B at t = 1 to force upon it the constraint (i) placed on U. The tied-down process has gaussian fidi projections whose means and covariances agree with those for the brownian bridge; the processes generate the same distribution on the projection (J-field.
V.2. Empirical Processes From a sequence {~i} of independent random variables, each having a Uniform(O, 1) distribution, construct the empirical process n
Uit) = n- 1 / Z L[{~i ~ t} - t] i= 1
for
°
~ t ~ l.
v.
96
The Uniform Metric on Spaces of Cadlag Functions
For fixed finite S, the projection ns V n is a normed sum of independent, identically distributed random vectors. By the Multivariate Central Limit Theorem (III.30) it converges to a zero-mean multivariate normal distribution with the same covariance structure as the brownian bridge; that is, the fidis of V,; converge to the fidis of V. (No accident of course-that's why V got the covariances it did.) By design, the first condition of Theorem 3 is satisfied. A markov property of V n will strengthen this to convergence in distribution.
9 Empirical Central Limit Theorem (Uniform Case). The empirical processes {V n} constructed by independent sampling from V niform(O, 1) converge in distribution, as random elements of D[O, 1J, to the brownian bridge.
°
PROOF. We need to find a grid = to < t1 < ... < tm = 1 satisfying (4), for fixed e and o. Do this by making the sum in (6), with Xn replaced by Vn, small. Take the {tJ equally spaced. By reasons of symmetry, or stationarity if you prefer, the sum reduces to mlP{sUPo,;r,;b IV n(t)1 > o}, where b = m- 1 • We seek an m that makes it less than e. Take Iffr as the IT-field generated by V n(s)for s ::; t. It tells us how many of the observations ~b . . . , ~n have landed in [0, tJ, and where they lie. Suppose we know that [0, tJ contains exactly k of the observations. Given this information, the other n - k observations distribute themselves uniformly in the interval (t, 1]. Formally, on {VnCt) = n- 1 / 2 (k - nt)} the conditional distribution of V nCb) - V net) given Iffr is
°: ;
n- 1 / 2 [Bin(n - k, 8) - n(b - t)J, where 8 = (b - t)/(l - t). Notice the markov property: the conditional distribution depends only on Vit). Apply Tchebychev's inequality on the set where I Un(t) I > that is, where Ik - ntl;:::: n 1 / 2 0.
(5,
lP{! UnCb) - UnCt)1 >!I Un(t)!llff r} = lP{IBin(n - k, 0) - n(b - t)1 > !!k - nt!} ::; 4[(n - k)O(1 - 8) + [en - k)O - n(b - t)J2]!(k - ntf ::; 4n8/(k - nt)2 + 48 2 ::; [4b/(1 - b)J/(52 + 4b 2 /(l - b)2 ::;! for small enough values of b. Notice that the argument would break down if we replaced IU n(t) I by in the first line. Apply Lemma 7 with f3 = t and m large enough.
t
t(5
mIPt~~~bIUn(t)1 > b}::; 2mlP{IVnCb)1 > to}. With m fixed, let n -+ 00. Invoke fidi convergence to make the right-hand side of the inequality converge to 2mlP{IN(0, b - b 2 )/ >
to} s
2m[(b - b 2 fWj)-4JlPIN(0, S 32m- 1 0- 4lPIN(0, 1)1 4,
which is less than e for m large enough.
1W
o
97
V.2. Empirical Processes
The restriction to samples from the uniform distribution was unnecessary. A similar result holds for the empirical process n
En(r) = n 1 / 2 [Fn(r) - F(r)] = n- 1 / 2
L
[{1]i ~ r} - F(r)],
i= 1
constructed by independent sampling from any distribution function F on the real line. Think of En as a random element of D[ - 00,00], the space of cadlag functions on [ - 00,00]. Right continuity at - 00 can be achieved by setting En( - 00) equal to zero; the left limit at + 00 also equals zero, the natural value for En( + 00 ). It would cause us no great hardship to carry all the theory for D[O, 1] over to D[ - 00, 00] then track through the proof of the general Empirical Central Limit Theorem. The uniform metric is well-defined, because of the boundedness (Problem 6) of functions in D[ - 00, 00]; the projection O"-field plays the same role as in D[O, 1]. This time the limit gaussian process E would have sample paths in D[ - 00, 00] and multivariate normal fidis with zero means and co variance kernel (10)
IPE(r)E(s)
=
F(r) - F(r)F(s)
for
r
~
s.
The E process need not have continuous sample paths; it jumps where F Jumps. Some small complications would arise with the choice of grid points for the general empirical process, but these could be overcome easily (Problem 7). Everything else would go through in much the same way as for the uniform distribution. There is, however, a much simpler way to prove the general Empirical Central Limit Theorem: use the quanti le transformation (Section II1.6) to represent En as a continuous image of Un. 11 Empirical Central Limit Theorem. The empirical processes {En} constructed by independent sampling from a distribution function F converge in distribution, as random elements of D[ - 00, 00], to the gaussian process E with co variance kernel given by (10).
PROOF. Define a map H from D[O, 1] into D[ - 00,00] by setting (Hx)(r) = x(F(r)). It is measurable (both spaces are equipped with their projection O"-fields) and uniformly continuous:
IIHx - Hyll = sup Ix(F(r)) - y(F(r))l
~
Ilx - YII·
By the uniform case of the Empirical Central Limit Theorem and the Continuous Mapping Theorem, HU n ~ HU. Remember from Section III.6 that the quantile function has the property: ~i ~ F(r) if and only if Q(~i) ~ r. n
(HUn)(r) = n- 1 / 2
L [gi ~ F(r)} -
F(r)]
i= 1 n
= n- 1 / 2
L [(Q(~i) ~ r} i= 1
- F(r)].
98
V. The Uniform Metric on Spaces of Cad\ag Functions
The random variables {Q(O} form an independent sample from the distribution F; the last sum has the same distribution as En. You can complete the proof by checking that HU satisfies the defining requirements for E. (This provides one method for proving the existence of E.) 0 Even though the quantile transformation worked like a charm here, you should beware of applying it unnecessarily to force processes that want to live in D[ -00, ooJ into migrating to D[O, 1]. Gratuitous rescaling of the time axis, perhaps with the aim of reducing a problem to a case involving only uniformly distributed random variables, can complicate an otherwise straightforward argument. 12 Example. The Central Limit Theorem for the sample median, proved by direct methods in Section IlI.4, can be deduced from the Empirical Central Limit Theorem. As before, assume that the sampling distribution F has a continuous, positive density j in a neighborhood of its median m. To finesse the problems caused by the jumps in the empirical distribution function F n, regard any random variable mn for which
Fn(m n)
=! + oin - 1/2 )
as a sample median. From the condition on j, F(t) =
! + (t
- m)[f(m)
+ 0(1)].
Because mn must converge to m (Example ILl), we deduce that F(m n) =
! + (mn
- m)[f(m)
+ op(l)].
Combine the expressions for FnCmn) and F(m n) to get En(m n) = -n I/2 (m n - m)[f(m)
+ op(l)] + 0/1),
which rearranges to (13)
n l12 (m n - m) = [ -En(mn) - op(l)]/[f(m)
+ op(l)].
Think of the right-hand side as a function of the four random elements En, mn , op(l), and oil). (Perhaps we should distinguish between the two 0/1) symbols-they stand for different variables.) Better still, think of it as a function of the random element (En' mn, oil), op(l» of D[ - 00, :xlJ @ IR3. Since (m n, oil), op(l» -? (m, 0,0) in probability and En ~ E, and since Problem 8 shows that E concentrates on a separable subset of D[ - 00, ooJ, Example IV.lO lets us deduce that (14) The right-hand side of (13) can be constructed from the left-hand side of (14) through application of the map H from D[ - 00, 00] @ IR 3 into IR defined by H(x, a, p, y) = [ -x(a) - pJ/[f(m)
+ y].
99
V.2. Empirical Processes
H is continuous at (x, m, 0, 0) for every x that has m as a point of continuity. Almost every sample path of E has this property because, as shown by the representation of E given in the proof of Theorem 11, each such sample path can be written in the form u F, where u belongs to C[O, 1]. (Don't forget that F is continuous at m.) Call out the Continuous Mapping Theorem. n 1/2(m n - m) = H(En' mn, 0/1), op(l)) ~ H(E, m, 0, 0) = -E(m)/!(m), 0
D
which has the desired N(O, t!(m)-2) distribution.
15 Example. What happens to the limit distribution of Kolmogorov's goodness-of-fit statistic when parameters are estimated? That is, if Fn is obtained by independent sampling on the distribution function F(·, eo), with eo the true value of an unknown parameter e, what is the asymptotic behavior of the statistic Dn = n l /2 sup I Fn(t) - F(t, en) I constructed using an estimator en for e? Suppose F were uniformly differentiable, in the strong sense (Problem 9) that
for some fixed function Ll(·) in D[ - 00,00]. Then, provided en were one of those nice estimators that converge in towards eo at the D/n-l/2) rate, Dn could be written as Dn = n 1/21IFn(-) - F(·, eo) - (en - eo)LlOII + oin1/2(en - eo))
= liEn - nl/2(en - eo)LlII + op(l). We know how En behaves for large n-like the gaussian process E-but what will be the effect of adding on the extra term n 1/2(e n - eo)Ll? That depends upon the joint distribution of En and en. According to the statistical folklore, good estimators can often be coerced into the form
en
=
eo
+ n- 1
n
'IL(IJ;) i= 1
+ oin- 1 / 2)
°
for some function L satisfying IPL(IJ;) = and IPL(IJi)2 = (j2 < 00. Let's assume that our estimator has such a representation. That makes it much easier to analyze the random elements Zn = (En' n 1/2(e n - eo)) of D[ - 00, 00] ® JR. Look at the fidis. Write (En(t 1), ... , En(t k ), n 1/2(e n - eo)) as
n- 1/2
n
L ({IJi ~ td i= 1
- F(tl> eo),···, {IJi ~ t k }
-
F(tk> eo), L(IJi))
+ oil).
100
V. The Uniform Metric on Spaces of Cadlag Functions
The Multivariate Central Limit Theorem gives convergence in distribution to a zero mean multivariate normal random vector (E(t 1), ... , E(tk ), y), where the covariances amongst the first k components follow (10) and lPE(t)y = lP{1Ji S t}L(1JJ That suggests a limit process (E, y) for Zn, with E and y having a sort of gaussian distribution in D[ - 00, ooJ ® IR. We actually already have all the tools needed to formalize the limit result for Zn. All that maximal inequality and finite-dimensional approximation stuff goes through as for the Empirical Central Limit Theorems; most of the brain work has been carried out, and only very messy details remain. But that would be terribly inelegant. Why not wait for the neater proof in Section VII.S, and just accept the fidi proof as sufficient evidence in the meantime? Elegance will return when the functions {IJi S t} and L(1Ji) are accorded equal status. Check continuity and measurability for the map from D[ - 00, ooJ ® IR into IR that sends (x, r) onto [[xC·) - rLl(· )[[, then bring out the Continuous Mapping Theorem, yet again, to deduce that Dn ~ D = [[E(·) - yLl(·)[[. The parameter estimation propagates through to the limit to add a random drift term yLl(-) onto the gaussian process EC')' If only we could calculate the distribution of D then we would have solved the problem completely. Problem 10 looks at a special case to show what difficulties this can present. Try reworking the example with all the processes in D[ - 00, ooJ rescaled using FC', 80 ) if you want to discover some of the perils of too automatic a recourse to the quanti le transformation. 0
V.3. Existence of Brownian Bridge and Brownian Motion This section proves the existence of gaussian processes satisfying the requirements of Definition 8. The proof uses the Compactness Theorem from Section IV.S. This is neither the simplest nor the fastest method known to mankind; but it will generalize readily into an existence proof for more exotic gaussian processes. Any reader who found Section IV.S too tedious could safely skip onto the next section, with perhaps just a quick look at Problem 11. A good way to construct a process is to set up a sequence of approximations that should converge to the process, if it exists. One then needs to show that the approximating sequence, or one of its sub sequences, actually does converge to something. One hopes this something has all the required properties of the desired process. We have not yet considered any sequence of processes that should converge to a brownian motion, but in the uniform empirical processes {Un} we do have a sequence converging to a close relative, the brownian bridge. Let us extract a brownian bridge as the limit of a convergent subsequence.
101
V.3. Existence of Brownian Bridge and Brownian Motion
16 Theorem. There exists a brownian bridge. PROOF. Check the uniform tightness condition in the Compactness Theorem of Section IV.5. We need to find a compact subset K of D[O, 1J for which
liminfIP{UnEG} > 1 -
(17)
8,
whenever G is an open, projection-measurable set containing K. To force the brownian bridge to live in C[O, 1J, we shall also want K to contain only continuous functions. Our K will be represented as the intersection of a sequence of closed sets {D k}, with Dk a finite union of closed balls of radius 2jk (any sequence ofradii decreasing to zero would suffice). As the digression after the start of Section IV.5 pointed out, such a K is compact: it inherits completeness from D[O, 1J and, by the choice of {Dd, it is certainly totally bounded. The closed balls making up each Dk center themselves on piecewise linear, continuous functions obtained by linear interpolation between a finite set of vertices (0, ro), ... , Ct;, r;), ... , (1, rm):
°
With T standing for the grid = to < ... < tm = 1, write LT for the map from IR T to C[O, 1J that takes the vector r = Cr 0, ... , rm) onto this interpolated function. The map satisfies a Lipschitz condition,
IILT(r) - LT(s) I :s:: max Iri - Sil·
(18)
i
The composition LT 0 7rT defines a map from D[O, 1J into C[O, 1J, a piecewise linear, continuous approximation map for which
liLT 0
7r T (x)
- xii
:s:: 2 max sup Ix(t) - x(t;) I, i
Ji
where J i = [ti' ti+ 1). The right-hand side should look familiar. It is the functional of x, call it hT(x), that lurks behind the convergence criterion of Theorem 3.
102
V. The Uniform Metric on Spaces of Cadlag Functions
For given b > 0 and e > 0, the proof of Theorem 9 came up with a grid T for which limsup IP{hT(U n) > b} < e. You should convince yourself that it was unnecessary to assume the existence of the brownian bridge process to get this inequality. Reinterpret it as a statement of how well Un can be approximated by linear interpolation: given e and b there exists a grid T for which limsup IP{IIL T nT(U n) - Unll > b} < e. 0
Why are there no measurability difficulties here? Now come back to the task of specifying the closed balls for the Dk sets. For each k choose a grid T(k) such that liminfIP{IILT(k)onT(k/Un) - Unll ::; Ijk} > 1 - ej2k+l.
Because the random vectors {nT(k)( Un)} converge in distribution, there exists a compact subset H k of IR T(k) such that liminf IP{nT(k) Un E Hd > 1 - ej2k+ 1 hence liminf IP{LT(k) nT(k)(Un)E LT(k)(Hk )} > 1 - ej2k+l. 0
As a continuous image of a compact set, LT(k)(H k ) is compact, and so can be covered by a finite collection of closed balls with radius Ijk. Set Dk equal to the union of the finite collection of closed balls with the same centers, but radius 2jk. The larger radius allows for a Ijk distance between Un and its linearly interpolated approximation LT(k) nT(klU n): 0
liminf IP{ Un E Dk } > I - ej2k for every k. The ej2k was contrived so that liminf IP{U n E Dl n ... n Dk } > I - e for every k. Of course we cannot just let k tend to infinity now and hope to replace the finite intersections by their limit K. We really do need that open set G in the definition of uniform tightness. Inequality (17) will follow from what we have proved about the {D k } if we show that G ;;:::> Dl n ... n Dk for some k. If there were no such k, we could choose an Xk from each closed set Fk = GC n Dl n .,. n Dk. As shown in the next paragraph, we could then apply Cantor's diagonalization argument to extract a Cauchy subsequence from {Xk}' The limit of the Cauchy subsequence (remember that D[O, 1J is complete) would belong to Fk for every k, an impossibility because
nFk
= GC n K
k
The existence of the desired k would follow.
= 0·
103
V.4. Processes with Independent Increments
Here is how we would construct the Cauchy subsequence. D1 is a finite union of closed balls of radius 2; some subsequence of {xd would lie completely within one of these, B1 say. B1 n D2 would be contained in a finite union of closed balls of radius 1; some sub-subsequence would lie completely within B1 n B 2, with B2 a closed ball of radius 1. B1 n B2 n D3 would be contained in a finite union of closed balls of radius~; a sub-sub-subsequence would lie within B1 n B2 n B 3, with B3 a closed ball of radius l And so on. The desired Cauchy sequence would take the first element of the first subsequence, the second element of the sub-subsequence, the third element of the sub-sub-subsequence, and so on. Every function in K is continuous, a uniform limit of piecewise linear, continuous functions from the sets LT(k)(H k ). By the Compactness Theorem, some subsequence of the uniformly tight {Un} converges in distribution to a probability measure concentrating on a countable union of compact subsets of C[O, 1]. Its fidis, the limits of the corresponding empirical process fidis, identify it as the distribution of the sought-after brownian bridge. 0 There are two ways of constructing brownian motion from the brownian bridge U. One can either set B 1 (t)
= U(t) + tZ for 0:::; t :::; 1,
where Z has a N(O, 1) distribution independent of U, or one can rescale by setting B 2 (t)
= (1 + t)UC : t) for 0:::; t <
00.
In both cases one gets a gaussian process with continuous sample paths. Direct calculation of means and covariances, which uniquely determine multivariate normal distributions, shows that B 1 has the right fidis to identify it as a brownian motion on [0, 1J and B2 has the right fidis for brownian motion on [0, (0).
V.4. Processes with Independent Increments A stochastic process Z indexed by an interval of the real line is said to have independent increments if Z(to), Z(t1) - Z(to), .. ·, Z(t k ) - Z(t k - 1) are mutually independent whenever to < t1 < ... < t k • For random,elements {Xn} of D[O, 1J with independent increments satisfying a mild regularity condition, the criterion for convergence in distribution given in Theorem 3 reduces to a particularly simple form when the limit process has continuous sample paths. Essentially we have only to check convergence for each increment Xn(t) - Xn(s). Just one thing can spoil the nice clean characterization. Because the increments would be unchanged by the addition of an
v.
104
The Uniform Metric on Spaces of Cad Jag Functions
arbitrary constant to X n , they surely cannot determine uniquely its finitedimensional distributions; Xn(O) must be specified. 19 Theorem. Let X, XI, X 2,
... be random elements of D[O, 1], each with independent increments. Suppose X has continuous sample paths. Then Xn ~ X if and only if:
(i) Xn(O) ~ X(O); (ii) for each pair s < t, the increment X n(t) - X n(s) converges in distribution to X(t) - X(s); (iii) given (j > 0 there exist a > 0 and {3 > 0 and an integer no such that IP{ IXit) - Xis) I :S; H} ~ {3 whenever It - si < a and n ~ no· Forgive the mysterious factor of -!- in (iii); it's there to make the notation fit better with Lemma 7. Suppose X n ~ X. Conditions (i) and (ii) follow from convergence of the fidis. Condition (iii) is a simple consequence of the Continuous Mapping Theorem applied to the continuous map from D[O, 1] into IR defined by
PROOF.
Hax = sup{lx(t) - x(s)l:
It - si < a}.
(Take the supremum over rational pairs to check measurability.) At each continuous x the supremum converges to zero as a ..... 0, because continuity on a compact interval implies uniform continuity. As this applies to every sample path of X, there must exist an a > 0 and {3 > 0 such that IP{HaX < H} = 2{3. Because HaXn ~ HaX and because (- 00, -!-(j) is open, liminf IP{HaXn <
H} ~ 2{3
by authority of Example IV.17. For all n large enough, IP{ IXn(t) - Xn(s) I :S;
H}
~ IP{HaXn
< -!-(j} > {3,
whenever It - si < a. Now let us show that the two requirements listed in Theorem 3 follow from conditions (i), (ii), and (iii). Choose 0 = So < SI < ... < Sk. The joint characteristic function of the random vector
factorizes into a product of one-dimensional characteristic functions, because Xn has independent increments. By (i) and (ii), the product converges to the corresponding product for X, the joint characteristic function of the random vector
With a continuous (linear, even) transformation, recover the desired fidi convergence. For the second requirement of Theorem 3 try a grid 0 = to < ... < tm = 1 for which maxi(ti+ 1 - t i) < Y < a, with a as given by (iii). The value of y
lO5
V.4. Processes with Independent Increments
will be specified at the end of the proof. Fix for the moment a value of n greater than no. Write gt for the IT-field generated by {Xn(s): S ::; t}. Suppose t i ::; t < t i+ 1 . Then, on the set {IXn(t) - Xn(t i) I > b}, IP{IXn(ti+l) - XnCt) I ::; !IXn(t) - Xn(tJllgt} 2:: IP{IX n(t i+ 1 ) - Xn(t) I ::; !b} 2:: /3.
Apply Lemma 7 to the process XnO - XnCt i) on each intervaIJ i limsup ::;
IP{m~x s~p IXn(t) -
XnCt i) I > b}
m-l
I
[ti' t i+ 1).
XnCtJI > b}
limsup~t~IP{s~p IXn(t) -
::; p-l
=
limsup IP{IXnCt i+ 1 )
-
Xn(tJI >
H}
i=O
which is less than m-l
p-l
I
IP{ IX(t i+1) - X(t i) I 2:: !b}
i=O
because (X nCt i+ 1), X n(tJ) "+ (X(ti+ 1), X(t i)) and {(u, v) ElR 2: Iu - v I 2:: !b} is closed. Write Eo, ... ,Em - 1 for the independent events appearing in the last summation. From the inequality exp( - IPEJ 2:: 1 - IP Ei deduce that m-l
I
IPEi ::; -log IP
nEi i
i=O
= -IOg(l - IP
yEi)
::; -log(1 - IP{HyX 2::
H})
because max(t i+ 1
-
t;) < y.
i
°
Argue as before that HyX 1 for each sample path, as y 1 0, to see that y could have been chosen to make the logarithmic expression less than /3c. 0 Notice how we used the sample path continuity of the limit process X. The argument would break down if X were allowed to grow by jumps. For example, if X were a poisson process, with X(t i) - X(t i - 1 ) distributed Poisson(A(t i - t i - 1 )), the sum would not decrease as y tended to zero: m-l
m-I
I
i=O
IP{ IX(ti+ 1)
-
X(tJ I 2:: !b} =
I
i=O
[A(t i + 1
-
t i) + O(t i+ 1
-
tJ]
= ..1.+0(1). The uniform metric is inappropriate for such an X. A better metric, which can tolerate a jumpy limit process, will be introduced in Chapter VI.
106
V. The Uniform Metric on Spaces of Cadlag Functions
Theorem 19 really only says something about convergence in distribution to gaussian limit processes. If X(O) has a normal distribution, the combination of independent increments and continuous sample paths forces X to be a rescaled and recentered brownian motion (the gaussian process described in Definition 8). An approximation argument shows why. Because H y(X) ~ almost surely as y ~ 0, there exists a sequence {en} with en 1 and IP{H2/n(X) > en} ~ 0. Set ~ni = Xii/n) - Xn((i - l)/n). Define step-function approximations to X by
° °
n
Xn(t) = X(O)
+ I
{i/n :s; t}~ni{ I~nd :s; en}·
i= 1
Whenever H 2/n(X) :s; en' the X n process lies uniformly within en of X. Thus IIX n - XII ~ in probability. for the variance of the sum appearing in the definition of Fix t. Write XnCt), and an for its expectation. If an ~ 0, then XnCt) = X(O) + an + 0/1) from which we get X(t) - X(O) = constant, a degenerate sort of normality. If, on the other hand, {an} does not tend to zero, then, along some subsequence, (Xn(t) - X(O) - an)/an ~ N(O, 1) by the Lindeberg Central Limit Theorem. (Necessarily en/an ~ along any subsequence for which an is bounded away from zero.) Convergence of types (Brei man 1968, Section 8.8) allows nothing but normality for the distribution of X(t) - X(O). A similar argument works for any other increment of X. Even with the gloss of apparent generality rubbed off the limit distribution, Theorem 19 still has enough content to handle some non-classical problems about sums of independent random variables.
°a;
°
20 Example. Let gni: i = 1, ... , ken); n = 1,2, ... } be a triangular array of random variables satisfying the conditions of the Lindeberg Central Limit Theorem. That is, they are independent within each row, they have zero means, they have variances {a;i} that sum to one within each row, and
(21)
IIP~;i{l~nd ~ e} ~ i
°
for each fixed e > 0. Set Sni = ~nl + ... + ~ni' What is the asymptotic behavior of a random variable such as maxi Sni? We may write it as a functional of the partial-sum process Si')' the random element of D[O, 1J defined by Sn(t) = Sni
for
var Sni :s; t < var Sn.i+l·
The curious choice for the location of the jumps of Si') has the virtue that var Sn(t) ~ t as n ~ 00, for each fixed t, because maxi a;i ~ 0. The increments of SnC·) over disjoint subintervals of [0, 1J are sums over disjoint groups of the {~nJ; the process has independent increments. If we joined up the vertices, to produce an interpolated partial-sum process with sample paths in C[O, 1J, we would destroy this property.
107
V.5. Infinite Time Scales
S,,(.)~
......:....-_-
•
•
•o
.--
• I
For fixed sand t, with s less than t, the increment Sit) - SnCs) is also a sum of the elements in a triangular array of independent random variables. Because var[SnCt) - SnCs)]
-+
t - s
and because (21) also holds for summation over any subset of the {~n;}' the Lindeberg Central Limit Theorem makes Sn(t) -,Sn(s) converge to a N(O, t - s) distribution-one of the conditions we need to make SnO converge in distribution to a brownian motion. What about condition (iii) of Theorem 19? Tchebychev's inequality is good enough. IP{ ISit) - Sn(s) I ::;; !6} 2 1 - var[Sn(t) - Sn(s)J/(!6? -+ 1 - (t - s)/(!6)2, uniformly in t and s. You can figure out what IJ. should be from this. Only our ingenuity in thinking up functionals that are continuous at the sample paths of brownian motion can curtail our supply of limit theorems for the partial sums. One example: max Sni = sup SnCt)
~
sup B(t).
i
Calculation of the limit distribution in closed form awaits us in Section 6, where this and several other functionals of brownian motion and brownian bridge will be examined. D
v.s.
Infinite Time Scales
Both the spaces D[O,lJ and D[ - 00, ooJ have compact intervals of the extended real line as their index sets; the theories for convergence in distribution of random elements of these spaces differ only superficially. For spaces with non-compact index sets some extra complications arise. As a typical example, consider D[O, 00), the set of all real-valued cadlag functions on [0, 00). A function x in D[O, 00) must be right-continuous at each point of [0, 00), with a finite left limit existing at each point of (0, 00). Because the limit point + 00 does not belong to the index set, no constraint is placed on the behavior
v.
lO8
The Uniform Metric on Spaces of Cadlag Functions
of x(t) as t ~ 00; it could diverge or oscillate about in any fashion. Such a function need not be bounded. The uniform distance between two functions in D[O, (0) might be infinite. Even for bounded random elements of D[O, (0), convergence in distribution in the sense of the uniform metric may impose far stronger requirements on their tail behavior (that is, on what happens as t ~ (0) than we can hope to verify. Sometimes the best we can try for is control over large compact subintervals of [0, (0). That corresponds to convergence in the sense of the metric for uniform convergence on compacta. 22 Definition. A sequence of functions {xn} in D[O, (0) converges uniformly on compacta to a function x if SUPt$k Ixn(t) - X(t) I ~ as n ~ 00 for each fixed k. Equivalently, d(x n , x) ~ 0, where
°
00
d(xn' x) =
Lr
k min[l, dk(x , x)] n
k=l
o
dk(x n, x) = suplxn(t) - x(t)l· t$k
In all that follows, D[O, (0) will be equipped with the metric d and the projection a-field. With that combination, each x in D[O, (0) is completely regular (Problem 12). For each k define a truncation map Lk from D[O, (0) into D[O, k]; set Lkx equal to the restriction of x to the interval [0, k]. By construction, {xn} converges to x if and only if {Lkx n} converges uniformly to Lkx for each k. Convergence in distribution has a similar characterization. 23 Theorem. Let X, X!oX 2, ... be random elements of D[O, (0), withIP{X E C} for some separable set C. Then Xn ~ X if and only if LkXn ~ LkX, as random elements of D[O, k],for eachjixed k.
PROOF. Necessity of the condition follows from the Continuous Mapping Theorem, because each Lk is both continuous and measurable. For sufficiency, define a continuous, measurable map Hk from D[O, k] into D[O, (0) by (HkZ)(t) = z(t /\ k). It comes as close as possible to defining an inverse to the map L k: the function Hk Lkx equals x on [0, k]. Deduce that d(x, Hk Lkx) :s; r k for every x in D[O, (0). Suppose LkXn ~ LkX. Because Hk is continuous, Hk LkXn ~ Hk LkX. Pick k large enough to make 2- k < c. Then both IP{d(Xn' Hk 0 LkX n) > e} and IP{d(X, Hk LkX) > e} equal zero. Example IY.ll, with Hk Lk playing the role of the approximation map, does the rest. 0 0
0
0
0
0
0
Typically C equals C[O, (0), the set of all continuous functions in D[O, (0). As in the case of compact time intervals, C[O, (0) sits inside D[O, (0) as a closed, separable, measurable subset (Problem 13). The borel a-field on C[O, (0), generated by the open sets for the d metric, coincides with the
109
V.S. Infinite Time Scales
projection (J-field. The most important of the limit processes living C[O, 00) is brownian motion on [0, 00).
In
24 Example. Let Sn denote the nth partial sum of a sequence ~ 1> ~2' ... of independent, identically distributed random variables for which IP~i = and IP~r = 1. What is the asymptotic behavior of the hitting time
°
1: n
= inf{j: n- 1 / 2 S j > 1}
as n tends to infinity? Define Hx = inf{t ~ 0: x(t) > 1}, with the usual convention that the empty set has infimum + 00. Right continuity offunctions in D[O, 00) allows us to take the infimum over rational t values to verify measurability of H. Express n- l 1: n as the functional HX n of the process Xn(t) = n- 1 / 2 Sj,
j/n ~ t < (j
+
l)/n.
U se Theorem 23 to check that X n ~ B, a brownian motion on [0, 00). For fixed k, the truncated process LkX n has the same form as the partial-sum process of Example 20, but stretched out and rescaled to fit the interval [0, k] instead of [0, 1]. The modifications have little effect on the arguments adduced there to prove convergence to brownian motion; exactly the same idea works in D[O, k]. If we are to use the Continuous Mapping Theorem to prove HX n ~ HB we will need H continuous at almost all sample paths of B. By itself, continuity of a sample path x will not suffice for continuity of Hat x. You can construct sequences of functions {x n } that converge uniformly to a bad continuous x without having HX n converging to Hx. BAD:
The functional H has better success with a good path like this: GOOD:
-
r-b
r+b
-
-
-1 - c
v.
110
The Uniform Metric on Spaces of Cadlag Functions
Here Hx = L. If x(t) :s; 1 - c for t :s; L - b (a continuous function achieves its maximum on a compact interval) and x( L + b) ~ 1 + c, then any y in D[O, 00) with dt+O(x, y) < c must satisfy L - b :s; Hy :s; L + b. That brownian motion has only good sample paths, almost surely, can best be shown by arguments based on a strong markov property, a topic to be taken up in the next section. The distribution of the functional HB of brownian motion will also be derived in that section. For the moment, we must content ourselves with knowing that n-1Ln has a limiting distribution that we could calculate if we were better acquainted with the limit process of the sequence {X n }. D
V.6. Functionals of Brownian Motion and Brownian Bridge From the definition of brownian motion on [0, 00) it is easy to deduce (Problem 14), for each fixed L, that the shifted process Bit)
=
B(L
+ t)
- B(L)
for
o:s; t < 00
°
is a new brownian motion, independent of the a-field Ct generated by the random variables {B(s): :s; s :s; L}. Equivalently, (25)
IPf(Bt)A
=
IPf(B)IP A
for every A in Ct and every bounded measurable f on C[O, 00).
B
B,
The assertion is also valid for a wide class of random L values. The precisely formulated generalization, known as the strong markov property, underlies most of the clever tricks one can perform with brownian motion. It will get us the distributions of a few interesting functionals. The random variable L will need to be a stopping time, that is, a random variable, taking values in [0, 00], for which {L :s; t} belongs to Ct for each t.
111
V.6. Functionals of Brownian Motion and Brownian Bridge
Interpret Iff 00 as the smallest a-field containing every Ifft-we learn everything about the brownian motion if we watch it forever. Define B.(w, t)
=
[B(w, t + ,(w)) - B(w, ,(w))] {,(w) < oo}.
Problem 17 shows that Br is projection-measurable, as a map from n into C[O,oo). The stopping time, determines a a-field Ifft' which captures the intuitive idea of an event being observable before time ,. By definition, an event A belongs to Iff r if and only if A {, :s;; t} belongs to Iff t for every t. The strong markov property asserts that Br is a brownian motion independent of Iffr on {, < oo}. Equivalently, (25) holds for every Iffr-measurable A contained in {, < oo} and every bounded, measurable f on C[O, 00). To prove the assertion it suffices that we check the equality for each bounded, uniformly continuous f. (Apply Problem 15 to the distributions induced on C[O, 00) by B, under IP, and Bt' under IP(·I A).) With such an f, continuity of sample paths implies 00
f(Br)A = lim
L f(Bk/n)A{(k
- l)ln :s;; , < kin}.
k= 1
Of course only one term in the sum will be non-zero for each fixed n. B
The event A {(k - 1)In :s;; , < kin} belongs to Iff kin; the independence asserted in (25) can be invoked for each kin value. 00
IPf(Br)A = lim
L
IPf(Bk/n)A{(k - l)!n :s;; , < kin}
k=l 00
= lim
I
IPf(B)IPA{(k - 1)ln :s;; , < kin}
k=l
= IPf(B)IP A because, < 00 on A. The strong markov property is established.
by (25)
112
V. The Uniform Metric on Spaces of Cadlag Functions
26 Example. The classical reflection principle for brownian motion involves a hidden appeal to the strong markov property. It enables us to find the distribution of the stopping time T = inf{t: B(t) = a} for fixed a> O.
The reflection principle uses the symmetry of brownian motion - the processes Br and -Br have the same distribution-to argue that B should be just as likely to hit a and end up with B(t) > a as it is to hit a and end up with B(t) < a. The probability that B hits a before time t should be twice the probability that B(t) > a. A formal proof works backwards. IP{B(t) > a} = IP{B(t) > a, T < t} = IP{B(T) + Bit - T) > a, T < t} = IP{Bit - T) > 0, T < t} because B(T) = IP[{T < t}IP{Bit - T) > Ol@"r}].
=
a if T <
00
Because T is @"r-measurable it can be treated as a constant inside the conditional probability. (A slightly more formal justification would invoke Fubini's theorem.) On {T < t}, symmetry of the normal distribution allows us to replace the conditional probability by 1, and then deduce IP{B(t) > a} = 1IP{T < t}.
An explicit expression for IP{ T < t} can be found by using the N(O, t) distribution for B(t). For our later purposes it will be more important that we know the Laplace transform LiA) = IP exp( -AT) for A ~ O. Two applications of Fubini's theorem, then a differentiation, will lead to a simple differential equation for La. LiA) = IP
f {T < t < oo}Ae-
At
dt
= {OO Ae-AtIP{T < t} dt
= {XlAe-At2IP{N(0, t) > a} dt
113
V.6. Functionals of Brownian Motion and Brownian Bridge
f Ae-lt{t > rx 2/N(0, 1)2} dt
IP
=
= IP exp( -Arx 2/N(0, 1)2) (2/n)I/2
=
Loo exp( -Arx 2z- 2 -1Z2) dz.
Differentiate with respect to A, then change variables by setting y = (2A)I/2rxz- 1.
L~(A) =
(2/n)I/2
Loo _rx 2 z- 2 exp( -Arx 2z- 2 -1Z2) dz
= -rx(2A)-1/2(2/n)I/2
Loo exp( -tl -
Arx 2y-2) dy
= - rx(2A)- 1/2 La(A). The only solution with LiO) = 1 is La(A) = exp( -rx(2A)I/2).
o
Doob (1949), in the non-heuristic part of his paper, was able to paday the result from Example 26 into an expression for the distribution function of 11 VII, which his heuristic argument suggested as the limit for Kolmogorov's goodness-of-fit statistic. His approach provides a case study in the application of the strong markov property. First one transforms the brownian bridge V by a scale change: B(t)
=
(1
+ t)UC
: t)
for
°
$;
t < 00.
The means and covariances of this gaussian random element of C[O, 00) identify it as a brownian motion (the same trick as in Section 3). Because V(l)
= 0, IP{II VII
~ rx} = IP{supjvC : t) j ~ rx: °$; t < oo} = IP{ IB(t) I = rx(1 + t) for at least one t} = IP{B hits rx + rxt or B hits -rx - rxt}.
v.
114
The Uniform Metric on Spaces of CadJag Functions
To find the probability that B ever leaves the region bounded by the two sloping lines ±(o: + o:t), one first solves the simpler problem where there is only one sloping line. For a fixed p > 0 (and not just p = 0:) set cp(o:) = JP{B hits 0:
+ pt},
the dependence on pbeing suppressed for the while. If B is ever to hit the line 0:1 + 0: 2 + pt, for 0: 1 > 0 and 0: 2 > 0, it must first hit the line 0:1 + pt. Call the stopping time at which this happens a. Invoke the strong markov property. CP(O:l + 0: 2) = W{B hits 0: 1 + 0:2 + pt} = W{a < 00, Ba hits 0: 2 + pt} = W{a < oo}W{B hits 0: 2 + pt}
= CP(0:1)CP(0:2)·
B
The equation CP(O:l + 0:2) = CP(0:1)CP(0:2) has non-negative, decreasing solutions with cp(O) = 1 only of the form cp(o:) = exp( -co:) for 0: > O. The positive constant c remains to be determined. B,
Another strong-markov argument, working with the stopping time which B first hits level 0:, gives the value of the constant.
1:
at
115
V.6. Functionals of Brownian Motion and Brownian Bridge
exp( -ca) = W{B hits a + {3t} = W{T < 00, Bt hits {3T + {3t} = W[{T < oo}W{Bt hits {3T + {3t/St}] = W{ T < 00 }
W{ B hits a
+ {3t}
=
exp( - 2a{3)
for
rX > 0, {3 > O.
Notice that the hitting probability depends on a and {3 only through their product. The probability of B hitting the line ka + k- 1{3t is the same for each k > O. The two barrier problem is slightly harder. For brevity write, temporarily, I for the line a + {3t and II for the line -a - {3t. Then W{ /B(t) / = a
+ {3t for at least one t} =
W{B hits I or B hits II}.
The two events {B hits I} and {B hits II} are not disjoint. An inclusionexclusion argument is needed. The infinite sum {hit I} - {hit I then II} + {hit I then 11 then I} - ... + {hit II} - {hit II then I} + {hit II then I then II} - ...
takes the value 0 if B never hits a barrier, it takes the value 1 if B makes a finite (positive) number of alternating hits, and it is undefined if B makes infinitely many alternating hits. If we prove that
(28)
W{B makes infinitely many alternating hits} = 0
then dominated convergence will justify
(29) W{B hits I or B hits II} = W{hit I} - W{hit I then II} + W{hit I then 11 then I} - ... + W{hit II} - W{hit 11 then I} + W{hit 11 then I then II} - ....
116
V. The Uniform Metric on Spaces of Cadlag Functions
Our success with the calculation for one barrier will extend, in strongly markov fashion, to each of the individual terms in this sum of probabilities. To avoid notational indigestion let us chew on just one of the terms, IP{hit I then I1}, say. Define the stopping time r (different from the last r) as the first t at which B hits I. From the perspective of B" barrier 11 is defined by the line of slope - 13 and intercept - 2(a + f3r). (We can treat r as if it were a constant if we condition on tS',.) By symmetry, B, has the same probability of hitting the line 3(a + f3r) + f3(t - r), which it sees as having slope 13 and intercept 2(a + f3r). Formula (27) allows B, to double the slope and halve the intercept of the line it is trying to hit. That is, IP{B, hits lIltS',}
= IP{B, hits 3(a + f3r) + f3(t - r)ItS',} = IP{B, hits 2(a + f3r) + 2f3(t - r)ItS',} = IP{B, hits 2a + 2f3tltS',}.
B
II
Notice that the new line sits above a event {r < oo}. IP{B hits I then II}
+
f3t for t 2': O. Integrate out over the
= IP{r < 00, B, hits I1} = IP{r < 00, B, hits 2a + 2f3t} = IP{B hits 2a + 2f3t} = exp[ - 2(2a)(2f3)] from (27).
Similar reasoning succeeds with higher numbers of alternating hits. For example, if (J denotes the time of the first hit on 11 after a hit on I, then IP{B hits I then 11 then I}
= = = =
IP{r IP{r IP{r IP{ r
< < < <
00,
(J
< <
00,
Ba hits I}
(J 00, Ba hits -2a - 2f3t} B, hits -2a - 2f3t} CIJ, B, hits 3a + 3f3t} = IP{B hits 3a + 3f3t} = exp[ - 2(3a)(3f3)].
00, 00,
117
Notes
You will observe the significance of the point -al{J on the taxis if you draw a picture for the argument. In general, for a string of m alternating hits on barriers I and 11, the probability is exp( - 2m 2 a{J). Let m tend to infinity to prove (28). Formula (29) with a = {J provides the solution to the problem with which we began: 00
IP{II Vnll
~ a} --+
IP{II VII
~ a}
= 2
L (-lr+ 1 exp( -2m 2 a2 ). m=l
Read Doob's paper.
NOTES
Most authors avoid the uniform metric once they recognize the measurability problems it creates with the empirical process. Billingsley (1968, Section 18) made the case against it clear. Dudley (1966a, 1967a) had already pointed the way around the problem, but his solution has been mostly overlooked in the literature. Dudley (1966a) had even proved a multivariate version of the Empirical Central Limit Theorem. He used a markov property of the empirical process. The idea of interpreting conditions like (4) as a means for constructing simple approximations to stochastic processes appears explicitly in Wichura (1971). Hajek (1965) had already applied the idea to characterize weak convergence in C[O, 1]. Skorohod (1956) based his study of weak convergence in D[O, IJ under its various Skorohod metrics on the same idea. Unfortunately this simple approach seems to have lost out to the uniform tightness approach (exposited by Billingsley (1968), for example), possibly because the approximation method appears to demand a separate proof for existence of the limit process as a random element of a space such as C[O, 1]. Actually, (4) plus fidi convergence imply uniform tightness; the argument in Section 3 is easily generalized. Lemma 7, specialized to processes with independent increments and with H replacing! IZ(t) I in the hypothesis, is sometimes attributed to Skorohod (1957), although it clearly has a strong similarity to Levy's symmetrization inequality. Notice that the ! IZ(t) I does gain us something-one benefit was noted during the proof of the Empirical Central Limit Theorem, in Section 2. Gihman and Skorohod (1974, V1.5) made more systematic use of a form of the lemma in a study of weak convergence for markov processes. Stute (1982a) obtained delicate oscillation results for empirical processes by exploiting their markovian properties. The Empirical Central Limit Theorem usually goes by the name of Donsker's theorem; Donsker (1952) proved it (using the uniform metric) in justifying Doob's (1949) heuristic approach to the Kolmogorov and Smirnov theorems. Donsker (1952) used a poissonization trick in his proof; he got it from Chung (1949), who got it from Kolmogorov (1933). Kac (1949) knew that a poissonized process was easier to analyze, because of its independent
v.
118
The Uniform Metric on Spaces of Cadlag Functions
increments. Breiman (1968, Section 13.6) used another version of the trickrepresentation of uniform order statistics as rescaled points of a poisson process-in relating the empirical process to a partial-sum process. The quanti le-transformation trick belongs to the folklore. The interest aroused when Durbin (1973a) applied weak convergence methods to get limit distributions for statistics analogous to those of Kolmogorov and Smirnov, but with estimated parameters, died down when the intractable limit processes asserted themselves. The papers in the conference on empirical processes (Gaenssler and Revesz 1976), the lecture notes by Durbin (1973b), and Pollard (1980), offer several perspectives on minimum distance methods. Parr (1982) has compiled a bibliography. The existence proof in Section 3 adapts an argument of Dudley (1966a, Proposition 2; 1978, Lemma 1.3). Erdos and Kac (1946, 1947) proved that the limit distributions for certain functions of partial sums of independent random variables depend on the summands only through their first two moments. Their paper led Donsker (1951) to formulate the results as limit theorems for functionals on a partialsum process. Donsker proved convergence of the process to brownian motion. Skorohod (1957) studied processes with independent increments in great detail; a nice account of some of his results has appeared in Gihman and Skorohod (1969, Chapter IX). Infinite time scales with various topologies (uniform or Skorohod) of convergence on compacta often cause minor confusion. The simple theory for C[O, 00) (Whitt 1970) proves remarkably unwilling to carry over to D[O, 00), at least for Skorohod metrics (Lindvall 1973, Stone 1963). The obvious nature of the main theorem (Theorem 23) for uniform metrics makes it hard to appreciate why Skorohod metrics should be any more difficult. More about this in Chapter VI. Billingsley (1968, Section 11) has derived distributions for functionals of brownian motion by taking limits for functionals of simple random walks (partial-sum processes). Stopping-time arguments for brownian motion give other first-exit distributions (Brei man 1968, Section 13.7). The results can also be derived by elegant martingale methods (Loeve 1978, Complements to Section 42). PROBLEMS [1] D[O, 1] is complete under its uniform metric. [If {xn} is a uniform Cauchy sequence
prove pointwise, and then uniform, convergence. Show that uniform convergence preserves the cadlag property.] [2] Let C be a separable subset of D[O, 1] under its uniform metric. There exists a countable subset To of [0, 1] such that functions in C can have discontinuities only at points of To. [A function in D[O, 1] has only finitely many jumps> e, otherwise the cadlag property would fail at a cluster point of those jumps. Take the union over rational e. If x is a uniform limit of {xn} then it can jump only where one of the {xn} jumps.]
119
Problems
[3J Calculate the covariances of the tied-down process B(t) - tB(l) to show that its fidis agree with those of the brownian bridge. [Means and covariances determine a multivariate normal distribution.J [4J Extend the proof of Theorem 3 to limit processes with jumps. [If X has paths in a separable set C, and if the grid points for the approximation maps Ak pick up every jump point of Cas k -> 00, then IIAkX - XII -> 0 almost surely.J [5J Would the argument in the proof of Theorem 9 for bounding Un(t) over [0, bJ work for a different interval, say [1 - b, 1J? This would be necessary if Un were not stationary; direct analysis of the non-uniform empirical process would require such an extension. [Try a different definition for 6",.J [6J Every function in D[ - 00, 00 J is bounded. [If Ix(tn) I ;:: n then x would not have the cadlag property at a cluster point of {tn}.J [7J Give a direct proof of the Empirical Central Limit Theorem for sampling from a general F. [Make sure the jump points of F appear in the sequence of grids from which fidi approximations are calculated.J [8J The image of a separable metric space under a continuous map is separable. The empirical process E concentrates on a separable subset of D[ - 00, 00 J. [The image of a dense subset is dense.J [9J Suppose the distribution function F(x, 8) has a bounded partial derivative ~(x, 8) with respect to 8. If ~(-, .) is uniformly continuous (or even just ~(x, .) equicontinuous), then sup IF(x, 8) - F(x, ( 0 )
-
(8 -
(0)~(x,
8)1 = 0(8 - ( 0 ),
[lOJ Find the limiting distribution of Kolmogorov's goodness-of-fit statistic for sampling from the N(8, 1) distribution, when 8 is estimated by the sample mean. Show that this limit does not depend on the true value 80 ; express it as a functional of the gaussian process with covariance function <1>(s)[l - <1>(t)J - 4J(s)4J(t), where <1> and 4J denote the distribution function and density function of the N(O, 1) distribution. [l1J Here is another construction for the brownian bridge. (a) Temporarily suppose there exists a brownian bridge U. Define, recursively, new processes Yo, Y1 , ••• and Zo, Zb'" by setting Yo = U, y"+1 = Y" - Zn, and 2"
Zn(t)
=
I
hnj(t)Y,,«2j - 1)/2n+1),
j= 1
where hnj is the function whose graph is an isosceles triangle of height one sitting on the base W - 1)/2n,j/2nJ. Show that Y,,+ 1 and Zn are independent. Deduce that the {Zn} are mutually independent. [Calculate covariances. The process Y" would be obtained from the brownian bridge by tying it down at the points {(2j - 1)/2n}; it is made up of 2n independent, rescaled brownian bridges sitting side by side. The covariance calculations all reduce to the same thing.J (b) Show that the process Zo + ... + Zn interpolates linearly between the vertices U/2n+ 1, UU/2 n+ for j = 0, ... , 2n+1.
1»,
v.
120
The Uniform Metric on Spaces of Cadlag Functions
(c) Now run the argument the other way. Construct processes {Xn} with the same distributions as the {Zn}, then recover the brownian bridge as a limit of sums of the {X n}. Let {~nj: j = 1, ... , 2n; n = 0, I, ... } be independent random variables, with ~nj distributed N(O, 1/2n+2). Define 2"
Xn(t) =
I
hnit)~nj·
j=1
At the points {j/2 n+1: j = 0, ... , 2n+1} the sum Xo + ... + Xn has the right fidis for a brownian bridge. [Only the fidis of U, which we know are well defined, were needed to calculate the distributional properties of Zo + ... + Zn·] (d) Show that JP{IIXnll ~ en} :-:;; 2n+1 exp( _2n+1 e;). [Apply the exponential inequality from Appendix B for normal tails: IIXnl1 is a maximum of 2n independent IN(O, 1/2n+2) I random variables.] (e) By choosing en = (2n/2 n+ 1)1/2 and then applying the Borel-Cantelli lemma, Xn(t) converges uniformly in t, almost surely; it defines a show that process X with continuous sample paths, almost surely. (f) At dyadic rational values for t, the series for X contains only finitely many non-zero terms. The fidi projections of X at a dense subset of [0, 1] have the distributions of a brownian bridge. The process X, with a negligible set of sample paths discarded, is a brownian bridge.
I:..o
[12] Every point of D[O, 00) is completely regular. [The distance dix, y) can be expressed as a supremum involving only rational time points. For each x and k, the function dk(x, .) is projection measurable. Use [1 - mdlx, .)]+ as a separating function.] [13] C[O, 00) is a closed, separable subset of D[O, 00). [It is the closure of a countable collection of piecewise linear, continuous functions, each constant over an interval of the form 00).]
et,
[14] For each fixed -r and each finite subset S of [0, 00), the random vector nsBr is independent of tff p because brownian motion has independent increments. Deduce independence of Br and tff r • [The projection maps {ns} generate the projection eT-field on C[O, 00).] [15] Let P and Q be probability measures on the eT-field :J6 0 generated by the closed balls of a metric space. If Pf = Qf for every bounded, uniformly continuous, :J6o-measurable f then P = Q. [Every closed ball is a pointwise decreasing limit of a sequence of such functions. The same is true for the intersection of any finite collection of closed balls. These sets form a generating class for :J6 0 that is closed under the formation of finite intersections.] [16] The borel eT-field on the product f£ ® qy of two separable metric spaces coincides with the product eT-field :J6(f£) ® :J6(qy). [Every open set in the product space is a countable union of sets of the form Gx ® Gy , with both Gx and Gy open. Compare with Problem IY.5.] [17] Show that the shifted process Br is projection measurable for each random time -r. [Prove measurability of B(w, t + -r(w» by writing it as a composition of two
121
Problems
measurable maps: w H (r(w), B(w, .)) from n into [0, 00] ® C[O, 00) and (s, x) H limn x((t + s) 1\ n)[1 1\ (n - s)+] from [0,00] ® C[O, 00) into IR. [18] In the sense of Example 24, almost all brownian motion sample paths are good. If 1" denotes the first time that B hits level 1, prove that IP{Bls)
~
0 for 0
~
s
~ (j} =
O.
Let (j tend to zero through a sequence to show that bad paths belong to a set of probability zero. [You could try letting the level ex in Example 26 sink to zero.]
CHAPTER VI
The Skorohod Metric on D[O, (0) .,. in which an alternative to the metric of uniform convergence on compacta is studied. With the new metric the limit processes need not confine their jumps to a countable set of time points. Amongst the convergence criteria developed is an elegant condition based on random increments, due to Aldous. The chapter might be regarded as an extended appendix to Chapter V.
VI.I. Properties of the Metric The uniform metric on D[O, 1J is the best choice for applications where the limit distribution concentrates on e[O, 1J, or on some other separable subset of D[O, 1]. It is well suited for convergence to brownian motion, brownian bridge, and the gaussian processes that appear as limits in the Empirical Central Limit Theorem. But it excludes, for example, poisson processes and other non-gaussian processes with independent increments, whose jumps are not constrained to lie in a fixed, countable subset of [0, 1]. To analyze such processes, Skorohod (1956) introduced four new metrics, all weaker than the uniform metric. Of these, the J 1 metric has since become the most popular. (Too popular in my opinion-too often it is dragged into problems for which the uniform metric would suffice.) But Skorohod's J 1 metric on D[O, 1J will not be the main concern of this chapter. Instead we shall investigate a sort of J 1 convergence on compacta for D[O, 00), the space where the interesting applications live. With the results from Section V.5 in mind, and even without seeing the J 1 metric defined, you might suspect that convergence Xn ~ X of random elements of D[O, 00) should reduce to convergence of their restrictions to each finite interval [0, T], in the sense of the J 1 metric on D[O, T]. This is almost true. We need to avoid those values of T at which X has positive probability of jumping. The difficulty arises because projection maps are not automatically continuous for J 1 metrics. Both the points and T require special treatment from the J 1 metric on D[O, TJ, whereas only has an a priori right to special treatment in D[O, 00). That tiny distinction makes it slightly more convenient to study D[O, 00) directly than to deduce all its properties from those of D[O, T]. As we shall not be concerned with
°
°
123
VI.1. Properties of the Metric
Skorohod's J 2, M 1, and M 2 metrics, let us drop the J 1 designation from now on. 1 Definition. For each finite T and each pair of functions x and y in D[0, 00) define the distance dT(x, y) as the infimum of all those values of b for which there exist grids 0= to < t1 < ... < tb with tk ;:0: T, and = So < SI < .. , < Sk, with Sk ;:0: T, such that Iti - Si I ~ b for i = 0, ... , k, and
°
Ix(t)-y(s)l~b
if
ti~t
and
Si~S<Si+l'
for i = 0, ... , k - 1. The weighted sum 00
d(x, y) =
L: r
k min[1, dk(x, y)]
k=1
defines the Skorohod metric on D[O, 00).
o
The proof that d is a metric contains no surprises (Problem 1). The requirement tk ;:0: T prevents discontinuities near T from overinfiating the distance dT(x, y): if, say, y jumps just before T then it can be matched by a similar jump in x just a little beyond T. By allowing d T to depend on more than the segment of path over [0, T], we avoid a difficulty that other authors have encountered in defining a metric for J 1 convergence on compacta. The main difference between convergence in the uniform sense and convergence in the J 1 sense appears at the discontinuity points of a limit function. If {x n } converges uniformly on compacta to x and if x has a jump
at to, then each x n , for n large, must have a jump of almost the same magnitude precisely at to. Skorohod's metric still forces each Xn to have ajump of almost the same magnitude, but not precisely at to.
~~ /~
~--------
124
VI. The Skorohod Metric on D[O, OCJ)
Straight from the definition we deduce that convergence of Xn to x under the d metric is equivalent to dT(x n, x) ~ 0 for each finite T. Another equivalent form is (Problem 2): d(xn' x) ~ 0 if and only if there exist continuous, strictly increasing maps {An} from [0, (0) onto itself such that, uniformly over compact sets of t values, An(t) - t ~ 0 and X(An(t» - xit) ~ O.
Sometimes continuity of a functional on D[O, (0) can be verified more easily when convergence is invoked in this form. 2 Example. Consider once again the first passage time functional from Example V.24: for x in D[O, (0),
Hx = inf{t 2 0: x(t) > I}
with the infimum over an empty set defined to be + 00. The functional is continuous, in the sense of the Skorohod metric, at every good path x.
If d(x n, x) ~ 0 then the functions {Yn} constructed from {xn} and {An} by setting Yn(t) = Xn(A; l(t» converge uniformly to x over compact sets of t values. The argument from Example V.24 gives HYn ~ Hx. Complete the proof of continuity of H by checking that HYn = AiHxn). D
As in Chapter V, we shall find a criterion for weak convergence in D[O, (0) by considering the approximations to processes built from the values they take at a fixed, finite grid of points 0 = to < t 1 < ... < tk. Define the interpolation map A from D[O, (0) into D[O, (0) by k-l (Ax)(t)
= X(tk){t k :::;; t < oo} +
I
i=O
x(ti){ti :::;; t < t i+ d·
125
VU. Properties of the Metric
All the jumps ofAx occur at grid points. For a fine grid, x and Ax will be close if x does not vary much between the grid points, except possibly for a single jump in each interval. In that case values of x before the jump are close to the value at the left grid point; values of x after the jump are close to the value at the right grid point: Ax
I
t,
Multiple jumps, though, could be missed completely:
../
/ t,
Bad behavior of x in [tk' (0) affects d(x, Ax) only through those d T with T;:::: t k • Good behavior of x in [0, T] may be quantified by a modulus function. 3 Definition. For each x in D[O, (0) and each finite interval [a, b], modulus function A(x, [a, b]) equals the infimum of those positive 8 which there exists a point s in (a, b] such that Ix(t) - x(a) I < 8 if a:S; t and Ix(t) - x(b) I < 8 if s:S; t :s; b.
the for
<S 0
126
VI. The Skorohod Metric on D[O,
CX»
4 Lemma. Let Ax be the approximation to x constructed from the values
taken at grid points Then
°= to < ... < tk = T, with ti -
dT(x, Ax) :::; b v max i:5:k
~(x,
t i- 1
:::;
b for each i.
[t i- 10 tJ).
PROOF. Write ~ for the maximum ~(x, [t i- 1 , tJ) value. Choose c > 0. The definition of the modulus function gives points {sJ with
and IX(S)-X(ti)I<~+c
Ix(s) - x(t k) I <
~
if Si:::;S<SHl
+c
i=O, ... ,k-l,
for
if Sk:::; s :::; tk.
Possibly Sk will be strictly less than T. Add on one more point sk+ 1 with tk < Sk+ 1 < tk + band Ix(s) - x(tk) I < ~
+c
for
tk :::; S:::; Sk+l'
Set tk+l = Sk+l' The extra points make {sJ and {tJ suitable grids for the calculation of dT(x, Ax). Since (Ax)(t) = x(t i) throughout [ti' t i+1),
I(Ax)(t) - x(S) I < ~ + c for Deduce that dT(x, Ax) :::; b v
(~
t i :::; t < tHl
and
Si:::;
S
< Si+l'
+ c), then let c tend to zero.
o
The lemma shows that the error committed in the approximation of x by Ax depends upon the variability of x, as measured by the modulus function. If we sufficiently refine the grid used to construct Ax, the approximation improves; for a fixed x, we can make the error as small as we please by choosing the grid fine enough. 5 Lemma. For each fixed x in D[O, (0), each fixed T, and each c > 0, there exists ab> such that
°
max
~(x,
[t i- 1 , tJ) < c
i
for every grid
°= to < ... < tk = T with ti -
t i- 1
:::;
b for each i.
PROOF. If ~(x, [0(, 0(']) > c, we get points 0( < , < f3 :::; 0(' with Ix(,) - x(O() I ~
1C
and
Ix(,) - x(f3) I ~
by setting
, = inf{t > 0(: Ix(t) - x(O() I ~ 1c}, f3 = inf{t > ,: Ix(t) - x(,) I ~ 1c}.
1C
vu.
127
Properties of the Metric
So if the assertion were false there would exist sequences of points an < On < f3n ~ T for which f3n - an ~ and
°
IX(on) - X(a n) I ~!8
and
IX(on) - X(f3n) I ~
k
We could extract sub sequences along which an ~ a, On ~ a, f3n ~ a, for some a in [0, T]. This would violate the cadlag property at a: there must exist a (j > for which
°
Ix(s) - x(a-)I
for
Ix(s) - x(a) I
a-
(j ~
a ~ s
s
< a,
< a + (j,
but eventually both members of at least one of the pairs (an' on) or (on' f3n) would have to lie on the same side of a. D Our main use for Lemmas 4 and 5 will be the construction of finitedimensional approximating processes to characterize convergence in distribution for random elements of D[O, 00). They also help us dispose of measurability complications. 6 Theorem. Under its Skorohod metric D[O, 00) is separable. The borel a-field coincides with the projection a-jield. PROOF. Write Do for the countable class of functions that take constant, rational values on each of the intervals [0, I/N), ... , [(NZ - 1)/N, N), [N, 00) for N = 1,2, .... Let us show that the class Do is dense in D[O, 00). Write S(N) for the set of grid points {j/N: j = 0, ... ,NZ}. As usual, nS(N) denotes the map that projects an x in D[O, 00) onto its vector of values at the points of S(N). Define hN as the continuous map from IRS(N) into D[O, 00) that takes a vector r = (ro, ... , rN2) onto the step function with constant value rj on the jth grid interval. If the components of r are rational, hN(r) belongs to Do. Given x and an 8> 0, choose N with 2- N less than 8 and N- 1 less than the (j of Lemma 5. We may assume that (j < 8. Construct the approximation ANX to x using the grid points S(N). That is, set AN = hN 0 nS(N). From Lemma 4, dN(x, ANX) ~ 8. Hence d(x, ANX) < 28. Find rational numbers rj with Irj - xU/N) I < 8. Then hN(r) belongs to Do and d(x, hN(r» < 38. For the moment write f!J for the borel a-field and f!} for the projection a-field on D[O, 00). To prove that f!} s; f!J, observe that x(t o) = lim Hn(x) for fixed to, where Hn(x) = sup x(t)sn(t), sit) = (1 - nit - to - 2Inl)+. For fixed n, the functional Hn is continuous (Problem 3). As a pointwise limit of continuous functions on D[O, 00), the projection nco must be f!Jmeasurable. The projections generate f!JJ.
128
VI. The Skorohod Metric on D[O, CfJ)
To prove that ~ 5; .9, it is enough to establish that each continuous, real function f on D[O, (0) is g>/~(lR)-measurable. (Every closed set in a metric space can be represented asf-1{0} for some continuous f.) We know that d(x, hN nS(N)(x» -+ as N -+ 00. Thusf hN nS(Nlx)-+ f(x) for each x, by continuity of f. The map f hN is continuous from lRS(N) into lR, and hence ~(lRS(N»/~(lR)-measurable. The map nS(N) is, by definition, g>/~(lRS(N»-measurable. Thus their composition f hN nS(N) must be g>/~(lR)-measurable. As a pointwise limit of such functions, f must also 0 be g>/~(lR)-measurable.
°
0
0
0
0
0
0
Needless to say, from now on we shall always equip D[O, (0) with its projection a-field, alias the borel u-field for the Skorohod metric. Every point of D[O, (0) is completely regular under this u-field. 7 Example. The asymptotic theory for maxima of independent random variables bears some similarity to the theory for sums of independent random variables. The role played by the normal is taken over by the extreme-value distributions, whose distribution functions are of the form exp( - G(x» for G(x) equal to one of e- x , or x-"{x ~ O}, or (-x)-"{x ::;; O}, with IX a positive parameter. If the maximum Mn of n independent observations from a distribution function F can be standardized to converge in distribution, then the limit must be one of these: for constants an (positive) and bn ,
This convergence implies a much stronger result for the joint asymptotic behavior of the maxima at different sample sizes, a result analogous to the convergence of the partial-sum process to brownian motion (Example V.20). Define the maxima process Y,,(.) as the random element of D[O, (0) with y"(t) = (M j
-
bn)/an for j/n::;; t < (j
+ l)/n.
The assumption (8) gives convergence for {Y,,(l)}. Using only the facts about the Skorohod metric that we have so far accumulated, we can strengthen this to convergence in distribution of the {y"} process. The method of proof depends upon a representation of Y" as a continuous transformation of a poisson process. To minimize extraneous detail, assume G(x) = x-"{x ~ O}. Trivial modifications of the argument would cover the other two cases. Define a sequence of measures {Rn} on (0, (0) by means of their distribution functions: R 1(0, x] = exp( -G(x»,
RnCO, x] = nexp(-G(x)/n) - (n - l)exp(-G(x)/(n -1).
Calculate their density functions if you doubt that these are well-defined measures. On S = [0, (0) ® (0, (0) generate independent poisson processes
vu.
129
Properties of the Metric
{nn} with intensity measures {A @ Hn}, where A denotes lebesgue measure on [0, (0). The sum (J n = n 1 + ... + nn is also a poisson process. As n tends to infinity, (In increases to a poisson process (J on S with intensity measure ro
I
n
A @ Hi = A @ lim
i=l
n
L Hi
= A @ y,
i=l
the measure Y being determined on (0, (0) by y(x, (0) = limen - n exp( -G(x)/n)] = G(x).
Label the points of (In as (fJni' hnJ, where fJnl < fJn2 < .... The {fJnJ form a poisson process on [0, (0), with intensity nA, independent of the {hnJ; the gaps between adjacent fJni have independent exponential distributions with mean n- 1 • The {hnJ are independent observations on the distribution function exp( - G/n). When subjected to a slight vertical perturbation they will become standardized observations on F. Let Q be the quanti le transformation corresponding to the distribution function F (Section III.6). Define T,,(y) = [Q(exp( - G(y)/n» - bn]/an·
For large n, this transformation hardly disturbs y: T,,(y) = inf{(z - bn)/an : F(z) 2 exp( -G(y)/n)} = inf{x: Fn(anx + b) 2 exp( - G(y»} -+ inf{x: exp( -G(x» 2 exp( -G(y»} by (8)
= y. The transformed variables {an T,,(h ni ) + bn} form a sequence of independent observations on F, because hni has distribution function exp( - G/n): lP{a nT,,(h ni )
+ bn :::; x} =
lP{Q(exp( - G(hni)/n» :::; x} = lP{Uniform(O, 1) :::; F(x)} = F(x).
The T" has the desired effect, in the vertical direction, on the points of (In. Define a random element Zn of D[O, (0) by setting
If Zn had its jumps at n -1, 2n-1, ... instead of at fJnl' fJn2' ... it would be a probabilistic copy of Y". Remedy the defect by applying to the time axis the random, piecewise linear transformation Yn that sends onto andj/n onto fJnj, for j = 1,2, .... The processes Y,,(.) and Zn YnO have the same distribution as random elements of D[O, (0). By the weak law of large numbers, Yn(t) -+ t in probability uniformly on compact intervals (Problem 4). Thus d(Zn' Zn Yn) -+ in probability. The
° °
0
0
°
130
VI. The Skorohod Metric on D[O, (0)
random elements {Zn} themselves converge almost surely to the random element
Z(t) = sup{h;: 11;
~
t},
where (111, h 1 ), (112, h2 ), ••• denote the points of the poisson process (J arranged in order of increasing time coordinate. Deduce that {Zn}, and hence {Y,,}, converges in distribution to z. 0
VI.2. Convergence in Distribution In Section V.l we found a necessary and sufficient condition for convergence in distribution of random elements {Xn} of D[O, 1], under its uniform metric, to a limit process X concentrating on a separable subset. The separability allowed X to have discontinuities only at fixed locations. For the proof we constructed an approximation AXn to each X n based on the values it took at a fixed finite grid on [0, 1]. The conditions we imposed ensured that, with high probability, the AXn process was uniformly close to X n • A similar method of proof will apply for convergence in distribution of random elements of D[O, 00) under d, its Skorohod metric. The constraint on the limit process will disappear, because D[O, 00) itself is separable under d. Each approximation AXn will, with high probability, be close to its Xn in the sense of d distance. ~ut one extra complication will arise because the fidi projections are not automatically continuous. If x belongs to D[O, 00) and XCi) i= XCi - ), the projection map n t is not continuous at x. For example, if xnCt) = x(ntj(n + 1)) then d(xn' x) --* but ntx n --* x( r -) i= ntx. For r a continuity point of x, however, n t is continuous at x (Problem 5). Necessarily, no is continuous at every x, because every increasing A that maps [0, 00) onto itself must set ..1.(0) equal to 0. For a random element X of D[O, 00), the projection n t will be continuous at all sample paths except those that have a jump at r.
°
9 Lemma. For each random element X of D[O, 00) there exists a subset r x of [0, 00) such that [0, oo)\rx is countable and IP{X(t) = X(t-)} = 1 for tin r x. The projection n t is IPx almost surely continuous at each t in r x. PROOF. It is enough to show that if e >
°then
IP{IX(t) - X(t-)I
~
e}
~
e
for at most finitely many t values in each bounded interval [0, T]. Write J(t) for {IX(t) - X(t-)I ~ e}, IfIPJ(tn) ~ e for an infinite sequence {tn} of distinct points in [0, T] then IP{J(tn) infinitely often} ~ e. There would exist an w belonging to infinitely many of the J(tn) sets. At some cluster point t in
131
V1.2. Convergence in Distribution
[0, T] the inequality IX(w, t n) - X(w, tn -) I 2 8 would hold for infinitely many distinct tn values in every neighborhood of t. This would violate the 0 cadlag property of X(w, .) at t.
Because the projection 11:0 is continuous at every x in D[O, (0), let us also admit as a point of r x, even though X(O - ) is not defined.
°
10 Theorem. Let X, Xl, X 2 , ••• be random elements of D[O, (0). Necessary and sufficient conditions for Xn ro+ X, in the sense of the Skorohod metric, are: (i) the fidis of X n corresponding to finite subsets of r x converge to the fidis of X; (ii) for each 8 > 0, each > 0, and each finite Tin r x, there exists a grid 0= to < ... < tK = T of points from r x with
I]
limsup IP{
m~x Ll(X n' [t i -
1,
t;]) >
I]} < 8.
PROOF OF NECESSITY. Appeal to the Representation Theorem (IV.13) for a new sequence {Xn}, with the same distributions as {X n}, and an X with the same distribution as X, for which
(11) For t in
d(Xn(w, .), X(w,
.»
~
°
for almost all w.
r x, XnCw, t) ~ X(w, t)
at almost every w.
Fidi convergence for {X n} at points of r x follows. Find a grid = to < ... < tk = T of points from
°
(12)
IP{
m~x Ll(X, [t
i - 1,
t;]) 2
I]} <
r x with G.
Such a grid exists by virtue of Lemma 5: as a sequence of grids is refined down to a countable, dense subset of [0, T], the maximum value of Ll over the grid intervals converges to zero at each sample path of X. Write LlT(x) as an abbreviation for maxi Ll(x, [t i -1> t;]). Consider an w at which the convergence (11) holds. Write xi·) for Xn(w, .), and x(·) for X(w, .). If we show limsup LlT(x n) ::; 2Ll T(x), then it will follow that limsup IP{LlT(X n) 2 21]} = limsup IP{LlT(X n) 2 21]} ::; IP{limsup LlT(Xn) 2 21]} ::; IP{LlT(X) 2 1]} < 8 by (12) as required by hypothesis (ii).
132
VI. The Skorohod Metric on D[O, CfJ)
Choose <5 > 0. By definition of ~T(X), there exists points {Si} with t i- 1 < Si ::; ti and
Ix(t) - X(t i - 1)1 < ~T(X) + <5 Ix(t) - x(tJI < ~T(X) + <5
for
t i- 1 ::; t < Si'
Si::; t ::; t i· Continuity of x at both t i - 1 and ti allows us to assume that the strict inequality Si < ti holds, and also that the range of validity for the first inequality is t i- 1 - <5' < t < Si' and for the second Si ::;; t < ti + <5', for some <5' > 0. Because d(x n, x) ~ 0, there exists a sequence of continuous, increasing maps {A.n} from [0, (0) onto [0, (0) such that Ait) - t ~
°
and
for
X(An(t» - xn(t) ~
°
uniformly on compacta. When n is large enough, t i - 1 < A; l(Si) < t i • Use si,n = An- 1(S;) as the split point in [t i - 1, ta for bounding ~T(Xn)' If t i - 1 ::; t < Si, n' and n is large enough,
Ixn(t) - Xn(t i- 1) I ::;; IX(An(t» - X(A n(t i- 1» I + <5 ::; IX(AnCt» - x(t i- 1) I + IX(t i- 1) - X(A n(t i - 1» I + <5 <
2~T(X)
+ 3<5,
because, eventually, t i - 1 - <5' < An(ti-1) ::; An(t) < An(Si,n) argument applies to t in [Si n' tJ.
= Si'
A similar
PROOF OF SUFFICIENCY. Let f be a bounded, uniformly continuous, real function on D[O, (0). We need to show that JPf(Xn) ~ JPf(X). Given 8> 0, find 1] > so that If(x) - f(y)1 < 8 whenever d(x, y)::;; 21]. Choose from r x a T large enough to ensure d(x, y) ::;; dT(x, y) + 1] for every pair x, y. Let ~TO have the same meaning as in the proof of necessity. According to hypothesis (ii) there exists a grid on [0, TJ for which
°
JP{~T(Xn)
> I]} <
8
if n;:::: no.
Also we may assume that the grid points are less than I] apart and, by the same reasoning as for (12), that JP{~T(X) > I]} < 8. Lemma 4 shows that the approximations constructed from this grid are, with high probability, close to the sample paths of the processes:
JP{dT(Xn' AXn) > I]} < IP{dT(X, AX) > I]} <
8
if n;:::: no,
8.
Complete the proof in the usual way. Write A for the interpolation map constructed from the values at the grid points.
IIPf(Xn) - IPf(X) I ::; IPlf(Xn) - f(AXn) I + IIPf(AX n) - IPf(AX) I + IPlf(AX) - f(X) I ::;; 8 + 21IfIIIP{d(Xn' AXn) > 21]} + IIPf(AX n) - IPf(AX) 1 + 8 + 2/1f11IP{d(X, AX) > 21]} ::; 8 + 211fl18 + 8 + 8 + 211f118 eventually, because AXn r-+ AX and d(x, Ax) ::;; dT(x, Ax)
+ 1] for every x.
0
133
V1.2. Convergence in Distribution
Roughly speaking, the modulus il(Xn, [t i - I , tJ) will be small if Xn has at worst one large jump in the interval [t i - I , tJ. To prove convergence in distribution, we need some way of stopping large jumps from piling up in a small interval. If Xn has a jump at Ln, which in general will be a random time, then we need
+ t)
JP{IXn(Ln
- XnCL n) I small, for all small t}
~
1,
and the approximation should hold uniformly in n. A maximal inequality seems required. Contrast this vague, formidable task with an elegant sufficient condition due to Aldous (1978): Xn ~ X if the fidis converge and, for each fixed T, (13)
Xn(Pn
+ bn) -
XnCPn)
-+
0 in probability,
whenever {b n } is a sequence of positive numbers converging to zero and {Pn} is a sequence of stopping times taking values in [0, T]. (The stopping time property means that the event {Pn ~ t} should belong to the O'-field generated by the random variables Xn(s), for 0 ~ s ~ t.) An equivalent form of (13) is: for each T, each 1] > 0, and each c > 0, there exists ab> 0 and an no such that (14)
JP{ IXn(Pn + b') - Xn(Pn) I ~ 1]} < c for
n ~ no,
whenever Pn is a stopping time for Xn that takes values in [0, T] and b' is a real number with 0 ~ b' ~ b. The proof of Aldous's result is built up by repeated application of inequality (14). 15 Lemma. Let Z be a random element of D[O, oo)for which
JP{ IZ(p + b') - Z(p) I ~ 1]} <
8
for each real b' in [0, b] and each stopping time P taking values in [0, T]. If 0' and Lare stopping times for which 0' ~ Land IZ(L) - Z( 0') I ~ 21] on {L < oo}, then JP{L ~ T /\ (0' + !b)} < 48. PROOF. Integrate both sides of the inequality for P with respect to lebesgue measure on [0, b], interchange the order of integration, then make a change of variable in the inner integral.
8b > JP
f
{IZ(s) - Z(p) I ~ 1]}{p
~ s ~ P+
b} ds.
Apply this inequality twice with P equal to the stopping times O'T = T /\ and LT = T /\ L, then add. 2cb > JP
f
{IZ(s) - Z(O'T)/
+ {/Z(s)
~ JP
f
~ 1]}{O'T ~ S ~ O'T +
- Z(LT) I ~ 1]}{LT
~ S ~ LT
[{Z(s) - Z(O'T)/
~ 1]} + {/Z(s)
x {LT ~
+ b} ds.
S ~
O'T
b}
+ b} ds
- Z(LT)/
~ 1]}]
0'
134
VI. The Skorohod Metric on D[O, 00)
On the set {T ~ T}, the sum of the two indicators is at least 1, because at least one of the inequalities
must hold if aT = a, TT = T, and /Z(a) - Z(T) / :?: 2Yf. Deduce that
2cb > IP
J~
T}{ T
:?: IP
J~
T
1\
(a
+ !b)}{a + !b ~ s ~
:?: !bIP{T ~ T
1\
(a
+ !b)}.
{T
{T
16 Theorem. Let X, X 10 X 2,
•••
~ S ~ a + b} ds a
+ b} ds
o
be random elements of D[O, oo)for which:
(i) the fidis of X n corresponding to finite subsets of r x converge to the fidis of X; (ii) Aldous's condition (13) holds.
Then Xn
~
X in the sense of the Skorohod metric.
Verify condition (ii) of Theorem 10. Set down a grid 0= to < ... = T of points from r x with the maximum grid interval shorter than
PROOF.
<
tk
ta, for a value of a that will be specified soon.
Fix an n, then define stopping times for X n by To
Tj+ I
= 0, =
inf{t >
Tj : /
Xit) - XiT) / :?: 2Yf}
with the usual convention that the infimum of the empty set equals + 00. We should perhaps add an extra subscript n to each T j • If (t i - I , tJ contains at most one of the {T} then we must have Ll(Xn' [ti-I, tJ) ~ 4Yf. For ifTj_1 ~ t i- I < Tj ~ ti < Tj +1 then
/Xn(t) - Xn(Tj-I)/ < 21] /Xit) - XiT)/
if t i - I ~ t < Tj'
< 2Yf if Tj
~
t
~
ti
and hence
IXit) - Xn(ti-I)I < 41] /Xit) - Xn(tJI
< 4Yf
if t i- I
~
t < Tj,
~
t
~
if
Tj
ti.
Apply the same reasoning to each grid interval.
{m~x Ll(Xn' [ti-I, tJ) > 4Yf} ~
{some (t i -
I,
tJ contains at least two of the {Tj}}
~ {some pair Tj - I, Tj has Tj ~ T
1\
(Tj-I
+ !a)}.
135
V1.2. Convergence in Distribution
Fix an integer m, whose value will be specified soon. Bound the last indicator function by 2m
L: {Tj:S; T
2m
/\ (rj-1
+ !a)} + m- 1 L: {rj:s;
j=1
T /\ (Tj-1
+ T/m)}.
j=1
The reasoning here is: either r2m 2': T, in which case the pair rj-l' rj would be detected by the first sum; or r 2m < T, in which case at least m terms of the second sum must equal one, for otherwise [0, T] would have to contain (m + 1) disjoint intervals (Tj-l, r) oflength greater than T/m. Take expectations. (17)
IP{m~x L\(Xn' [t
i - 1,
tJ) > 41J}
:s; 2m max IP{rj :s; T /\ (rj-l + ta)} js;2m
+ 2 max IP{Tj:S; T /\ (rj-l + T/m)}. jS;2m
Now we choose m and a. Invoke Lemma 15 for Z = Xn with by (14). For n 2': no,
(J
IP{Tj:S; T /\ (rj-l
= rj_ 10 r = rj and the
+ t(j)} <
(j provided
48.
Choose m so that T /m < t(j; hold it fixed. The second term in the bound (17) is then less than the 88 if n 2': no. From (14) find the a for which
°
°
if :s; (j' :s; a and if Pn is a stopping time for Xn with :s; Pn :s; T. Invoke Lemma 15 again, but replace (j by a and 8 by 8/m. For n 2': nI' IP{rj:s; T /\ (rj-l
+ ta)} < 48/m.
The first term in the bound (17) is less than 88 if n 2': n l' In summary:
IP{m~x L\(Xn' [t i - 1, tJ) > 41J} < 168 which completes the proof.
for
n 2': max(no, nl),
o
18 Example. For processes with independent increments, the criterion for convergence in distribution is particularly simple. Under the uniform metric for D[O, 1], Theorem V.19 showed that fidi convergence plus an equicontinuity condition on the increments suffices if the limit process X has continuous sample paths. With a minor variation we get Aldous's condition; essentially, we can drop the constraint on the sample paths of X if we reinterpret the result as convergence in the Skorohod sense.
136
VI. The Skorohod Metric on D[O, CfJ)
°
Suppose that for each e > 0, 11 > 0, and T < 00, there exists a (j > and an no such that IP{/Xn(t + (j') - XnCt)/ ~ 11} < e whenever 0:-::::; t:-::::; T, (j' :-::::; (j, and n ~ no. If Pn is a stopping time that takes only finitely many different values, and these values all lie in [0, TJ, then for n ~ no,
°:- : ;
IP{/Xn(Pn + (j') - X.(Pn) \ ~ 11} =
IIP[{Pn = t}IP{\Xn(t
+ (j')
-
XnCt) \ ~ I1IPn
= t}J
t
< e. Every stopping time can be approximated arbitrarily closely from above by a stopping time that takes only finitely many different values (round up to the next value on a finely spaced grid); the cadlag property of the sample paths carries the inequality over to the stopping times covered by Aldous's condition. 0 Problem VIlI.8 will give a more interesting application of Theorem 16 to convergence of martingales.
NOTES
Much of this chapter draws ideas from Billingsley (1968, Chapter 4) and Gihman and Skorohod (1974, Sections IlI.4 and VI.5). Skorohod (1956) defined on D[O, 1J several metrics, which allowed different sorts of behavior of a convergent sequence near a discontinuity of the limit function. Billingsley (1968) introduced a variation on Skorohod's J 1 metric, thereby making D[O, 1J complete. This is not so important if we consider only convergence to a known limit process, but it does greatly simplify the theory if existence of the limit process must be proved by compactness arguments. The metric of Definition 1 is modeled on the metric of Kolmogorov (1956) for D[O, 1]. Whitt (1980), and Lindvall (1973) have shown how difficult it is to write down a metric for J 1 convergence on compacta, as defined by Stone (1963). The modulus function is a fixed-grid analogue of Skorohod's (1956) flJj, or the flc of Gihman and Skorohod (1974, page 423). Billingsley (1968, Section 14) has defined other moduli for D[O, 1]. If existence of the limit process is not assumed in Theorem 10, the conditions will not guarantee its existence; the conditions of the theorem do not translate directly into a characterization of uniform tightness. Theorem 6 borrows from Parthasarathy (1967, Section VII.6). The elegant argument for inclusion of the borel u-field in the projection u-field was attributed by Straf (1969, page 67) to Wichura. Example 7 is based on the method of Resnick (1975). The idea of embedding the maxima process into a two-dimensional poisson process comes from Pickands (1971).
137
Problems
The convergence criterion of Theorem 10 corresponds to Theorem VI.5.2 of Gihman and Skorohod (1974). Aldous (1978) proved a result slightly different from Theorem 16; he gave a sufficient condition for uniform tightness in D[O, 1]. I found Kurtz's (1981, Chapter 2) rearrangement of the proof helpful. Skorohod (1957) studied convergence of processes with independent increments. His paper contains many fascinating sample-path arguments. PROBLEMS
[lJ Prove that the d of Definition 1 is a metric on D[O, 00). [To show that d(x, y) = 0 implies x = y, fix a t, then choose T larger than t. For each Twith t < T < T, deduce from the definition of d T the existence of a sequence {sn} with Sn -> Tand y(sn) -> X(T). Deduce that either X(T) = yeT) or X(T) = Y(T-). Choose a sequence {TJ strictly decreasing to t. Right con~inuity of Y at t gives both yeTi) -> yet) and Y(Ti-) -> yet); right continuity of x gives X(Ti) -> x(t).J [2J Show that d(xn' x) -> 0 if and only if there exist continuous, strictly increasing maps {An} from [0, 00) onto itself such that, uniformly on compact sets of t values, An(t) - t ->0
and
x(An(t)) - xn(t) -> O.
[Construct An as a piecewise linear map that takes gridpoints for Xn onto gridpoints for x, for pairs of grids chosen according to the definition of dT(x n , x) with T depending on n. ] [3J Prove continuity of the functional Hn that appeared in the proof of Theorem 6. [If d(x, Xi) -> 0, choose continuous, increasing {AJ with x(Ai(t)) - Xi(t) -> 0 and A/t) - t -> 0 uniformly on compact t sets. For i large enough, bound IHn(x) - HnCXi) I by sup Ix(t) Isup Isn(A/ t)) - sn(t) I + sup IX(A-;{t)) - Xi(t) I t'5:c
t$;c
for some constant c.J [4J Let {Yn} be a sequence of random, increasing maps from [0, 00) onto itself such that Yn(t) -> t in probability, for each fixed t. Show that sup IYn(t) - t I -> 0 in probability O::5:t:sT
for each fixed T. [If IYnCs) - si < e and IYn(t) - tl < ethen IYnCu) - ul < e + Is - t I for u between sand t.J [5J Suppose d(xn, x) -+ 0 and that x is continuous at T. Show that XnCT) x(AnCT)) - Xn(T) -> 0 and An(T) -+ T.J
-+
X(T). [Use
[6J If d(xn' x) -+ 0 and x belongs to C[O, 00) then Xn converges to x uniformly on compacta. [7J If X n ,... X in the Skorohod sense, and if X has sample paths in C[O, 00), then Xn ,... X in the sense of the metric for uniform convergence on compacta. [Switch to versions that converge almost surely in the Skorohod sense.J
CHAPTER VII
Central Limit Theorems ... in which the chaining method for proving maximal inequalities for the increments of stochastic processes is established. Applications include construction of gaussian processes with continuous sample paths, central limit theorems for empirical measures, and justification of a stochastic equicontinuity assumption that is needed to prove central limit theorems for statistics defined by minimization of a stochastic process.
VII. I. Stochastic Equicontinuity Much asymptotic theory boils down to careful application of Taylor's theorem. To bound remainder terms we impose regularity conditions, which add rigor to informal approximation arguments, but usually at the cost of increased technical detail. For some asymptotics problems, especially those concerned with central limit theorems for statistics defined by maximization or minimization of a random process, many of the technicalities can be drawn off into a single stochastic equicontinuity condition. This section shows how. Empirical process methods for establishing stochastic equicontinuity will be developed later in the chapter. Maximum likelihood estimation is the prime example of a method that defines a statistic by maximization of a random criterion function. Independent observations ~1"'" ~n are drawn from a distribution P, which is assumed to be a member of a parametric family defined by density functions {pC-, e)}. For simplicity take e real-valued. The true, but unknown, eo can be estimated by the value en that maximizes n i= 1
Let us recall how one proves asymptotic normality for en' assuming it is consistent for eo. Write go (', e) for log p(., e), and 9 1 (', e), g2(', e), gk, e), for the first three partial derivatives with respect to e, whose existence we impose as a regularity condition. Using Taylor's theorem, expand go(·, e) into go(', eo)
+ (e
- eO)g1(', eo)
+ tee
- eo)2g2(-, eo)
+ ice
- eo?gi·, e*)
139
VII. I. Stochastic Equicontinuity
with fJ* between fJo and fJ. Integrate with respect to the empirical measure P n. GnCfJ)
= Gn(fJo) + (fJ - ()O)Pngl + !(fJ - fJ O)2Png2 + RnCfJ).
If we impose, as an extra regularity condition, the domination
forallfJ,
Igk,fJ)I::;H(·)
then the remainder term will satisfy
IRn(fJ) I ::; ilfJ
- fJ o I3P nlg 3(·, fJ*)1 ::; ilfJ - fJol3PnH.
Assume PH < 00 and PIg21 < 00. Then, by the strong law oflarge numbers, for each sequence of shrinking neighborhoods of fJo we can absorb the remainder term into the quadratic, leaving (1)
Gi()
= Gn«()o) + (fJ - fJO)Png 1 + !«()
-
()0)2(Pg 2 + oil)
near fJo·
The op(l) stands for a sequence of random functions of fJ that are bounded uniformly on the shrinking neighborhoods of fJo by random variables of order 0/1). Provided Pg 2 < 0, such a bound on the error of approximation will lead to the usual central limit theorem for {nl/2(fJ n - fJo)}. As a more general result will be proved soon, let us not pursue that part ofthe argument further. Instead, reconsider the regularity conditions. The third partial derivative of go(·, fJ) was needed only to bound the remainder term in the Taylor expansion. The second partial derivative enters (1) only through its integrated value Pg 2 • But the first partial derivative plays a critical role; its value at each ~i comes into the linear term. That suggests we might relax the assumptions about existence of the higher derivatives and still get (1). We can. In place of Pg 2 we shall require a second derivative for Pg o(-, fJ); and for the remainder term we shall invoke stochastic equicontinuity. In its abstract form stochastic equicontinuity refers to a sequence of stochastic processes {Zn(t): t ET} whose shared index set T comes equipped with a semimetric d(·, .). (In case you have forgotten, a semimetric has all the properties of a metric except that des, t) = 0 need not imply that s equals t.) We shall later need it in that generality. 2 Definition. Call {Zn} stochastically equicontinuous at to if for each YJ > 0 and B > 0 there exists a neighborhood U of to for which limsup
IP{s~p IZit) -
Zn(t O) I > YJ} <
B.
o
There might be measure theoretic difficulties related to taking a supremum over an uncountable set of t values. We shall ignore them as far as possible during the course of this chapter. A more careful treatment of measurability details appears in Appendix C. Because stochastic equicontinuity bounds Zn uniformly over the neighborhood U, it also applies to any randomly chosen point in the neighborhood.
140
VII. Central Limit Theorems
If {en} is a sequence of random elements of T that converges in probability to to, then (3)
because, with probability tending to one, en will belong to each U. When we come to check for stochastic equicontinuity the form in Definition 2 will be the one we use; the form in (3) will be easier to apply, especially when behavior of a particular {en} sequence is under investigation. The maximum likelihood method generalizes to other maximization problems, where {log p(-, 8)} is replaced by other families of functions. For future reference it will be more convenient if we pose them as minimization problems. Suppose §' = {f(., t):tE T}, with T a subset of IR\ is a collection of real, P-integrable functions on the set S where P lives. Denote by P n the empirical measure formed from n independent observations on P, and define the empirical process En as the signed measure nl/2(Pn - P). Define F(t) = Pf(·, t), Fn(t) = Pnf(', t).
We shall prove a central limit theorem for sequences {en} that come close enough to minimizing the {FnC')}' Suppose f(', t) has a linear approximation near the to at which F(·) takes on its minimum value: (4)
f(', t)
=
f(', to)
+ (t
- to)'L1(-)
+ It
- tolr(·, t).
For completeness set r(·, to) = O. The L1(.) is a vector of k real functions on S. Of course, if the approximation is to be of any use to us, the remainder function r(·, t) must in some sense be small near to. If we want a central limit theorem for {en}, stochastic equicontinuity of {En r( " t)} at to is the appropriate sense. Usually r(·, t) will also tend to zero in the ,p2(p) sense:PI r(·, tW ~ 0 as t ~ to. That is, f(', t) will be differentiable in quadratic mean. In that case, we may work directly with the ,p2(P) seminorm pp on the set f.!ll of all remainder functions {r(-, t)}. Stochastic equicontinuity of {Enr(-, t)} would then follow from: for each B > 0 and > 0 there exists in f.!ll a neighborhood V of 0 such that
'1
limSUPIP{s~PIEnrl > '1} <
B.
The neighborhood V would take the form {r E f.!ll: pp(r) :5; b} for some b > O. This would be convenient for empirical process calculations. Differentiability in quadratic mean would also imply that PL1 = O. For if PL1 were non-zero the integrated form of (4), - t o),PL1 + o(t - to) near to, would contradict existence of even a local minimum at to. P f(-, t)
=
P f(', to)
+ (t
141
VII.l. Stochastic Equicontinuity
5 Theorem. Suppose {'In} is a sequence of random vectors converging in probability to the value to at which F(·) has its minimum. Define r(·, t) and the vector of functions ~O by (4). If (i) to is an interior point of the parameter set T; (ii) F(·) has a non-singular second derivative matrix V at to;
(iii) Fn('In) = oin-I) + inft Fn(t); (iv) the components of ~(.) all belong to ;£2(p); (v) the sequence {Enr(·, t)} is stochastically equicontinuous at to; then n 1/2 ('In - to) ~ N(O, V-I[P(~~') - (PM(p~)']V-I). Reparametrize to make to equal to zero and V equal to the identity matrix. Then (ii) implies F(t) = F(O) + !ltl 2 + 0(ltI 2) near O.
PROOF.
Separate the stochastic and deterministic contributions to the function F net) by writing P n as the sum P + n- 1/2 E n. Write Zn(t) for Enr(·, t). Stochastic equicontinuity implies Zn('In) = oil). For values of t near zero, (6) Fn(t) - Fn(O) = P[f(', t) - f(', O)J + n- 1/2 E n[f(', t) - f(', 0)] 2 = !ltl + 0(ltI 2) + n-1/2t'En~ + n- 1/2 ItIZ n(t). Invoke (iii). Because F n('In) comes within oin - 1) of the infimum, which is smaller than F ncO), op(n- 1) ~ FnC'In) - Fn(O)
= !I'In12 + op(I'InI2) +
n-1/2'I~En~
+ oin-1/2I'Inl).
The random vector En~ has an asymptotic N(O, P(~~') - (P~)(P~)') distribution; it is of order Op(l). Consequently, by the Cauchy-Schwarz inequality, 'I~En~ ~ - l'InIOil). Tidy up the last inequality. op(n-I)
~
[! - oil)]I'InI2 - n-1/2I'InIOp(l) - oin-1/2I'Ini)
= [! - op(I)][I'Inl - Op(n- 1/2 )]2 - Oin - 1). It follows that the squared term is at most 0 in -1), and hence 'In = 0 in -1 /2). (Look at Appendix A if you want to see the argument written without the oi') and 0i·) symbols.) Representation (6) for t = 'In now simplifies:
FnC'In)
Fn(O) = Fn(O) =
+ !I'In12 + n-1/2'I~En~ + oin-1) + !I'In + n-1/2En~12 - !n-lIEn~12 + oin-I).
The same simplification would apply to any other sequence of t values of order 0 pen -1 /2). In particular, Fn(-n-1/2En~)
= FnCO) -
!n-lIEn~12
+ oin-1).
Notice the surreptitious appeal to (i). We need n- 1/2 En~ to be a point of T before stochastic equicontinuity applies; with probability tending to one as n --+ 00, it is.
142
VII. Central Limit Theorems
Now invoke (iii) again, comparing the values of Fn at 'n and -n- l /zE n!1 to get tl'n + n- l / zEn !11 2 = oin- I ), whence nl/Z'n = -En!1 + ope!). When transformed back to the old parametrization, this gives nl/ZVI/Z('n - to) = - V- I/zE n!1 + oil) ~ V-I/zN(O, P(!1!1') - (P!1)(P!1)'). 0 Examples 18 and 19 in Section 4 will apply the theorem just proved. But before we can get to the applications we must acquire the means for verifying the stochastic equicontinuity condition.
VII.2. Chaining Chaining is a technique for proving maximal inequalities for stochastic processes, the sorts of things required if we want to check the stochastic equicontinuity condition defined in Section 1. It applies to any process {Z(t): tE T} whose index set is equipped with a semimetric d(., .) that controls the increments:
JP{ IZ(s) - Z(t) I > ry} ::;:: Mry, des, t»
for
ry > O.
It works best when !1( ., .) takes the form
!1(ry, b) = 2 exp( _tryZjDZb Z),
with D a positive constant. Under some assumptions about covering numbers for T, the chaining technique will lead to an economical bound on the tail probabilities for a supremum of IZ(s) - Z(t) I over pairs (s, t). The idea behind chaining, and the reason for its name, is easiest to understand when T is finite. Suppose TI , Tz , ... , 1k+1 = T are subsets with the property that each t lies within bi of at least one point in 1i. Imagine each point of 1i+ I linked to its nearest neighbor in 1i, for i = 1, ... , k. From every t stretches a chain with links t = tH b tb ... , t I joining it to a point in T I ·
o
=
point of T,
o
=
point of T2
•
= point of T3
143
VII.2. Chaining
The value of the process at t equals its value at t 1 plus a sum of increments across the links joining t to t 1. The error involved in approximating Z(t) by Z(t 1) is bounded, uniformly in t, by k
L
max 1Z(t i+ 1) - Z(tJ I·
i= 1
If 7i contains Ni points, the maximum in the ith summand runs over N i+ 1 different increments, each across a link of length at most bi • The probability of the summand exceeding l1i is bounded by a sum of N i + 1 terms, each less than Ll(l1i, bJ (7)
IP{m~x IZ(t) -
Z(t 1)1 > 111
+ ... + I1k}::;
itl N i+ 1Ll(l1i, bJ
This inequality is useful if we can choose l1i' bi , and 7i to make both the righthand side and the sum of the {l1i} small. In that case the maximum of 1 Z(s) - Z(t) lover all pairs in T is, with high probability, close to the maximum for pairs taken from the smaller class T1 • When Ll(I1, b) = 2 exp( -1112/D2b2), a good combination seems to be: {b;} decreasing geometrically and {l1i} chosen so that Ni + 1 Ll(l1i , bJ = 2b i , that is,
l1i = Db i[2 10g(N i+ db i)] 1/2. With these choices the right-hand side of (7) is bounded by the tail of the geometric series b;, and the sum of the {11;} on the left-hand side can be approximated by an integral that reflects the rate at which Ni increases as bi decreases.
Li
8 Definition. The covering number N(b), or N(b, d, T) if there is any risk of ambiguity, is the size of the smallest b-net for T. That is, N(b) equals the smallest m for which there exist points t l' . . . , tm with mini d(t, t i) ::; b for every t in T. The associated covering integral is J(b) = J(b, d, T) =
s:
[210g(N(u)2/u)r/2 du
for
0 < b ::; 1.
0
The N(U)2, in place of N(u), will allow us to bound maxima over more than just the nearest-neighbor links from 7i+ 1 to 7i. If we interpret P as standing for the !E 1(P) or !E 2 (P) semimetrics on :F, the notation NI (b, P, :F) and N 2(b, P, :F) used in Chapter 11 almost agrees with Definition 8. Here we implicitly restrict t I , ... , tm to be points of T. In Chapter II the approximating functions were allowed to lie outside :F. They could have been restricted to lie in :F without seriously affecting any of the results. The proof of our main result, the Chaining Lemma, will be slightly more complicated than indicated above. To achieve the most precise inequality,
144
VII. Central Limit Theorems
we replace IJ; by a function of the link lengths. And we eliminate a few pesky details by being fastidious in the construction of the approximating sets 1i. But apart from that, the idea behind the proof is the same. As you read through the argument please notice that it would also work if N(·) were replaced throughout by any upper bound and, of course, J(.) were increased accordingly. This trivial observation will turn out to be most important for applications; we seldom know the covering numbers exactly, but we often have upper bounds for them. 9 Chaining Lemma. Let {Z(t): t E T} be a stochastic process whose index set has a finite covering integral J(.). Suppose there exists a constant D such that,Jor all sand t,
(10)
IP{IZ(s) - Z(t) I > IJd(s, t)} ::; 2 exp( _!ryz/DZ)
for
IJ > O.
Then there exists a countable dense subset T* of T such that,Jor 0 < e < 1, IP{ IZ(s) - Z(t) I > 26D J(d(s, t)) for some s, t in T* with des, t) ::; e} ::; 2e
We can replace T* by T
if Z
has continuous sample paths. PROOF. Write H(u) for [2log(N(u)z/u)J1/ z. It increases as u decreases. Set 6; = e/2 i for i = 1, 2, .... Construct 26;-nets 1i in a special way, to ensure that Tl c:; Tz c:; .... (The extra 2 has little effect on the chaining argument.) Start with any point t l' If possible choose a t z with d(tz, t 1) > 26 1 ; then a t3 with d(t3' t 1 ) > 26 1 and d(t3' t z ) > 26 1 ; and so on. After some tm' with m no greater than N(6 1 ), the process must stop: if m > N(6 1 ) then some pair t;, tj would have to fall into one of the N(6 1 ) closed balls of radius 6 1 that cover T. Take Tl as the set {tl' ... , t m}. Every t in T lies within 26 1 of at least one point in T1 . Next choose tm+l' if possible, with d(tm+l' t;) > 26 z for i::; m; then tm+Z with d(t m + z, t;) > 26 z for i ::; m + 1; and so on. When that process stops we have built Tl up to Tz , a 26 z-net of at most N(6 z) points. The sets T3 , T4 , .•. are constructed in similar fashion. Define T* to be the union of all the {1i}. For the chaining argument sketched earlier (for finite T) we bounded the increment of Z across each link joining a point of 1i+ 1 to its nearest neighbor in 1i. This time 1i+l contains 1i; all the links run between points of 1i+l' With only an insignificant increase in the probability bound we can increase the collection of links to cover all pairs in 1i+ 1, provided we replace the suggested IJ; by a quantity depending on the length of the link. Set A; = {IZ(s) - Z(t) I > Dd(s, t)H(6;) for some s, t in 1i}. It is a union of at most N(6;)Z events, each of whose probabilities can be bounded using (10). IP A; ::; 2N(6;? exp[ -tH(6;)Z] = 26;.
The union of all the {AJ, call it A, has probability at most 2e.
145
VII.2. Chaining
Consider any pair (s, t) in T* for which des, t) ::;; £. Find the n for which bn < des, t) ::;; 2b n. Because the {T;} expand as i increases, both sand t belong to some Tm+1 with m > n. With a chain s = Sm+1, Sm,"" Sn link S to an Sn in T", choosing each Si to be the closest point of T; to Si+1, thereby ensuring that d(si+ l' sJ ::;; 2b i . Define a chain {tJ for t similarly. Break Z(s) - Z(t) into Z(sn) - Z(tn) plus sums of increments across the links of the two chains; IZ(s) - Z(t) I is no greater than m
IZ(sn) - Z(tn) I +
L [I Z(Si+ 1) -
Z(Si) I + IZ(ti + 1) - Z(U I],
i=n
Both Si+ 1 and Si belong to T; + l' On
Ai + 1>
On AC, these bounds, together with their companions for (sn, t n) and (t i+1, tJ, allow IZ(s) - Z(t) I to be at most m
Dd(sn' tn)H(b n)
+ 2 L 2Db i H(b i+1)' i=n
The distance d(sn' t n) is at most m m m des, t) + d(Si+1, Si) + d(t i+1, tJ::;; 2b n + 2 2b i ::;; lObn •
L
L
L
i=n
i=n
i=n
m
IZ(s) - Z(t) I ::;; lODbnH(b n)
+ 4D
L 4(bi+ 1 -
bi+2 )H(bi+ 1)
i=n
::;; lODbnH(b n)
+
16D In f{b i+2 <
::;; lODbnH(b n)
+
16DJ(b n+1)
U::;;
bi+1}H(u)du
::;; 26D J(d(s, t)). If Z has continuous sample paths, the inequality with T* replaced by T is the limiting case of the inequalities for T* with £ replaced by £ + n - 1. D
Often we will apply the inequality from the Chaining Lemma in the weaker form: IP{ IZ(s) - Z(t) I > 26DJ(£) for some s, tin T* with des, t) ::;; £} ::;; 2£. A direct derivation of the weaker inequality would be slightly simpler than the proof of the lemma. But there are applications where the stronger result is needed.
146
VII. Central Limit Theorems
11 Example. Brownian motion on [0, IJ, you will recall, is a stochastic process {B(·, t):O ~ t ~ I} with continuous sample paths, independent increments, B(·, 0) = 0, and B(t) - B(s) distributed N(O, t - s) for t ;:::: s. If we measure distances between points of [0, IJ in a strange way, the Chaining Lemma will give a so-called modulus of continuity for the sample paths of B. The normal distribution has tails that decrease exponentially fast: from Appendix B, IP{IB(t) - B(s) I > IJ} ~ 2 exp( -tlJ 2 /lt - si).
Define a new metric on [0, IJ by setting des, t) = Is - t 11/2. Then B satisfies inequality (10) with D = 1. The covering number N(f>, d, [0, IJ) is smaller than 2f> - 2, which gives the bound J( f»
~
s:
[2 log 4
+ 10 10g(1/u)J 1/2 du
~ (2 log 4)1/2f> + JiO[log(I/f»J -1/2 ~
4f>[log(1/f»J 1/2
s:
10g(1/u) du
for f> small enough.
From the Chaining Lemma, IP{ IB(s) - B(t) I > 26J(d(s, t)) for some pair with des, t)
~
f>}
~
2f>.
The event appearing on the left-hand side gets smaller as f> decreases. Let f> 1 0. Conclude that for almost all w, IB(w, s) - B(w, t)1 ~ 741(s - t) logls - t11 1/2 for Is - t1 1 / 2 < f>(w). Except for the unimportant factor of 74, this is the best modulus possible (McKean 1969, Section 1.6). 0
VII.3. Gaussian Processes In Section 5 we shall generalize the Empirical Central Limit Theorem of Chapter V to empirical processes indexed by classes of functions. The limit processes will be analogues of the brownian bridge, gaussian processes with sample paths continuous in an appropriate sense. Even though existence of the limits will be guaranteed by the method of proof, it is no waste of effort if we devote a few pages here to a direct construction, which makes non-trivial application of the Chaining Lemma. The direct argument tells us more about the sample path properties of the gaussian processes. We start with analogues of brownian motion. The argument will extend an idea already touched on in Example 11.
147
VII.3. Gaussian Processes
Look at brownian motion in a different way. Regard it as a stochastic process indexed by the class of indicator functions $'
= {CO, t]:
°
~ t ~
I}.
The co variance JP[B(·,f)B(·, g)J can then be written as PUg), where P = Uniform[O, 1]. The process maps the subset $' of fil2(P) into the space fil2 (JP) in such a way that inner products are preserved. From this perspective it becomes more natural to characterize the sample path property as continuity with respect to the fil2(p) seminorm pp on $'. Notice that
pp(I[O, sJ - [0, tJI) = (PI[O, sJ - [0, tW)1/2 = Is - tI1/2. It is no accident that we used the same distance function in Example 11. The new notion of sample path continuity also makes sense for stochastic processes indexed by subclasses of other fil2(p) spaces, for probability measures different from Uniform[O, 1].
12 Definition. Let $' be a class of measurable functions on a set S with a (J-field supporting a probability measure P. Suppose $' is contained in fil2 (P). A P-motion is a stochastic process {B p(., f): f E $'} indexed by $' for which: (i) Bp has joint normal finite-dimensional distributions with zero means and covariance JP[Bp(·, f)B p(·, g)J = PUg); (ii) each sample path Bp(w, .) is bounded and uniformly continuous with respect to the fil2(P) seminorm pp(-) on §'. The name does not quite fit unless one reads "Uniform[O, IJ" as "brownian," but it is easy to remember. The uniform continuity and boundedness that crept into the definition come automatically for brownian motion on the compact interval [0, 1]. In general $' need not be a compact subset of fil2 (P), although it must be totally bounded if it is to index a P-motion (Problem 3); uniformly continuous functions on a totally bounded $' must be bounded. We seek conditions on P and $' for existence of the P-motion. The Chaining Lemma will give us much more: a bound on the increments of the process in terms of the covering integral
J(6) = J(6, Pp,
$')
=
s:
[2 10g(N(u, Pp,
$')2 jU)J1/2
duo
Finiteness of J(-) will guarantee existence of Bp. 13 Theorem. Let $' be a subset of fil2(P) with a finite covering integral, J(.), under the fil2(p) seminorm pp(.). There exists a P-motion, Bp, indexed by ff,for which
IBp(w, f)
- Bp(w,
g)1
with 6(w)finitefor every w.
~
26J(pp(f - g))
if pp(f - g) < 6(w),
148
VII. Central Limit Theorems
PROOF. Construct the process first on a countable dense subset ffo = {fJ} of g: Such a subset exists because fi' has a finite b-net for each b > (otherwise J could not be finite). Apply the Gram-Schmidt procedure to ffo, generating an orthonormal sequence of functions {u}. Each f in ffo is a finite linear combination Lj
°
Z(w, f) = L
The sum converges for every w, because only finitely many of the coefficients
IP[Z(-,f)Z(·, g)J = L
= L
=
°
IP{IZ(f) - Z(g) I 2:: IJ} ~ 2 exp( -!IJ 2 jIP[Z(f) - Z(g)J2) = 2 exp( -!IJ 2 jpp(f - g?).
Apply the Chaining Lemma to the process Z on ffo. Because ffo itself is countable we may as well assume the countable dense subset promised by the Lemma coincides with ffo. Let G( b) denote the set of w for which IZ(f) - Z(g) I > 26J(pp(f - g»
for some pair with
pp(f - g)
~
b.
Then IPG(b) ~ 2b for every b > 0. As b decreases, G(b) contracts to a negligible set G(O). For each w not in G(O), IZ(w,f) - Z(w, g)1
~
26J(pp(f - g»
if
pp(f - g) < b(w).
Reduce Q to Q\ G(O). Then each sample path Z( w, .) is uniformly continuous. Extend it from the dense fi'o up to a uniformly continuous function on the whole of fi'. The extension preserves the bound on the increments, because both J and pp are continuous. Complete the proof by checking that the resulting process has the finite dimensional distributions of a P-motion. 0 For brownian motion, continuity of sample paths in the usual sense coincides with continuity in the pp sense, with P = Uniform[O, 1]. The P-motion processes for different P measures on [0, 1J (or on IR, or on IRk)
149
VH.4. Random Covering Numbers
do not necessarily have the same property. If P has an atom of mass ex at a point to, the sample paths of the Bp indexed by intervals {CO, tJ} will all have a jump at to. The size of the jump will be N(O, ex) distributed independently of all increments that don't involve a pair of intervals bracketing to. All sample paths are cadlag in the usual sense. We encountered similar behavior in the gaussian limit processes for the Empirical Central Limit Theorem (V.1l) on the real line. We represented the limit as U(F(· )), with U a brownian bridge and F the distribution function for the sampling measure P. We can also manufacture the limit process directly from the P-motion, in much the same way that we get a brownian bridge from brownian motion. Denote by 1 the function taking the constant value one. Then the process obtained from Bp by setting Ep( .,f)
= BP( .,f) - (P f)B p(" 1),
is a gaussian process analogous to the brownian bridge. 14 Definition. Call a stochastic process Ep indexed by a subclass fi' of f£2(P) a P-bridge over fi' if
(i) Ep has joint normal finite-dimensional distributions with zero means and covariance 1P[EP(·,f)Ep(·, g)] = PUg) - (P f)(Pg); (ii) each sample path Ep(w, .) is bounded and uniformly continuous with respect to the f£2(P) seminorm on Jl'. 0 The P-bridge will return in Section 5 as the limit in a central limit theorem for empirical processes indexed by a class of functions.
VHA.
Random Covering Numbers
The two methods developed in Chapter 11, for proving uniform strong laws of large numbers, can be adapted to the task of proving the maximal inequalities that lurk behind the stochastic equicontinuity conditions introduced in Section 1. The second method, the one based on symmetrization of the empirical measure, lends itself more readily to the new purpose because it is the easier to upgrade by means of a chaining argument. We have the tools for controlling the rate at which covering numbers grow; we have a clean exponential bound for the conditional distribution of the increments of the symmetrized process. The introduction of chaining into the first method is complicated by a messier exponential bound. Section 6 will tackle that problem. Recall that the symmetrization method relates Pn - P to the random signed measure P~ that puts mass ± n- 1 at each of ~l' ... , ~n' the signs being allocated independently plus or minus, each with probability l For central
150
VII. Central Limit Theorems
limit theorem calculations it is neater to work with the symmetrized empirical process E~ = n 1 / 2 P~. Hoeffding's Inequality (Appendix B) gives the clean exponential bound for E~ conditional on everything but the random signs. For each fixed function f,
IP{IE~fl > 111~} = IP
{IJl ±f(~;)1
>
I1nl/21~}
:s; 2 ex p [ - 2(l1n1/2)2/Jl 4f(02 ]
= 2 exp[ -tI1 2 jPn P]. That is, if distances between functions are measured using the !£2(P n) seminorm then tail probabilities of E~ under IP('I~) satisfy the exponential bound required by the Chaining Lemma, with D = 1. For the purposes of the chaining argument, E~ will behave very much like the gaussian process Bp of Section 3, except that the bound involves the random covering number calculated using the !£2(P n) seminorm. Write
Jz(b,
Pn'~) =
I:[210 g(Nz(U, P n, ~)2jU)]1/2 du
for the corresponding covering integral. Stochastic equicontinuity of the empirical processes {En} at a function fa in ~ means roughly that, with high probability and for all n large enough, 1 En f - En fa 1 should be uniformly small for all f close enough to fa. Here closeness should be measured by the !£2(p) seminorm pp. With the Chaining Lemma in hand we can just as easily check for what seems a stronger property -but if you look carefully you'll see that it's equivalent to stochastic equicontinuity for a larger class of functions. Of course we need ~ to be permissible (Appendix C). 15 Equicontinuity Lemma. Let ~ be a permissible class of functions with envelope F in !£2(P). Suppose the random covering numbers satisfy the uniformity condition: for each 11 > 0 and 8 > 0 there exists ay> 0 such that
(16)
limsup IP{Jz(y,
Pn'~)
> 11} <
8.
n
Then there exists ab> 0 for which limsup IP{su p IEn(f - g)1 > 11} <
8,
[0]
where [bJ = {(f, g): f, g E
~
and pp(f - g) :s; b}.
The idea will be: replace En by the symmetrized process [bJ by a random analogue,
PROOF.
Ob) = {(f, g): f, g E
~
E~;
replace
and (Pn(f - g)2)1/2 < 2b};
then apply the Chaining Lemma for the conditional distributions IP('I ~).
151
VHA. Random Covering Numbers
For fixed I and 9 in [15] we have var(Eil - g)) = P(f - g)2 S 15 2. Argue as in the FIRST and SECOND SYMMETRIZATION steps of Section II.3: when 15 is small enough, IP{su p IEM(f - g)1 > [0]
IJ} S 4IP{SUP IE~(f -
g)1 >
tlJ}.
[0]
That gets rid of En. If with probability tending to one the class <2(5) contains [15], we will waste only a tiny bit of probability in replacing [15] by <2(5): IP{sup [0]
IE~(f -
g)1 >
tIJ} S IP{sup IE~(f -
tlJ} + IP{[b] 't <2b)}.
g)1 >
(20)
It would suffice if we showed sUP?21 Pnh - Ph I --+ 0 almost surely, where {(f - g)2: I, g E 3"}. This follows from Theorem II.24 because the condition (16) implies
ffz = (17)
log N 1(15, Pn,
ffz) =
Din)
for each
15 >
o.
Problems 5 and 6 provides the details behind (17). That gets rid of [15]. The reason we needed to replace [15] by Ob) becomes evident when we condition on 1;. Write PnO for the !£2(Pn) seminorm. We have no direct control over Pn(f - g) for functions in [15]; but for Ob), whose members are determined as soon as I; is specified, Pn(f - g) < 2b. Apply the Chaining Lemma. IP{IE~(f
- g)1 > 26Ji2b, Pn, 3") for some (f, g) in <2b)*II;} S 4b.
The countable dense subclass Ob)* can be replaced by <2b) itself, because E~ is a continuous function on 3" for each fixed 1;: IE~(f - g)1 S n 1 / 2 Pnll - gl S n
1 2 /
pnCI - g).
tlJ}
Integrate out over 1;, then choose 15 so that both IP{26J 2(215, Pn, 3") > and 415 are small. 0
Now that we have the maximal inequalities for empirical processes, we can take up again the central limit theorems for statistics defined by minimization of a random process, the topic we left hanging at the end of Section 1. Recall that we need the processes {Enr(·, t)}, which is indexed by the class f!ll = {r(·, t): t E T} of remainder functions, stochastically equicontinuous at to. If 1(·, t) is differentiable in quadratic mean at to, it will suffice if we find a neighborhood V = {r E f!ll: pp(r) S b} for which limsup
IP{s~p IEnrl > IJ} < 8.
Notice that Vs:::: {r1 - r2:pp(r 1 - r2) S b}, because r(·, to) = 0 by definition. Thus we may check for stochastic equicontinuity by showing: (i) The class f!Jl has an envelope belonging to !£2(p).
152
VII. Central Limit Theorems
(ii) f(', t) is differentiable in quadratic mean at to. From (i), this follows by dominated convergence if r(·, t) --+ almost surely [PJ as t --+ to. (iii) Condition (16) is satisfied for $' = :J1l.
°
These three conditions place constraints on the class {f(-, t)}. 18 Example. The spatial median of a bivariate distribution P is the value of e that minimizes M(e) = Plx - el. Estimate it by the en that minimizes M.(e) = Pnlx - el. Example 11.26 gave conditions for consistency of such an estimator. Those conditions apply when P equals the symmetric normal N(O, 12 ), a pleasant distribution to work with because expliQit values can be calculated for all the quantities connected with the asymptotics for {en}. For this P, convexity and symmetry force M(·) to have its unique minimum at zero, so en converges almost surely to zero. Theorem 5 will produce the central limit theorem,
after we check its non-obvious conditions (ii), (iv), and (v). Change variables to reexpress M(e) in a form that makes it easier to find derivatives. M(e) = (2n)-1
flx l exp( -!Ix + (1 2 ) dx.
Differentiate under the integral sign. M'(O) = 0,
of course,
M"(O) = (2n)-1 fix I(xx' - 1 2 ) exp( -!Ix 12) dx.
A random vector X with a N(O, 12 ) distribution has the factorization X = R U where R2 = IX 12 has a x~-distribution independent of the random unit vector U = X/lXI, which is uniformly distributed around the rim of the unit circle.
v=
M"(O) = IP(R 3 UU' - RI 2) = IPR 3 IPUU' - (IPR)I2 = (n/8)1/2 I 2'
Condition (ii) wasn't so hard to check. To figure out the A(x) that should appear in the linear approximation
Ix - el = Ixl + e'A(x) + lelr(x, e), carry out the usual pointwise differentiation. That gives A(x) = x/I x I for x ¥= 0. Set A(O) = 0, for completeness. The components of A(·) all belong to "p2(p). Indeed, PAS = IPUU' = H 2' That's condition (iv) taken care of.
153
VH.4. Random Covering Numbers
Now comes the hard part-or at least it would be hard if we hadn't already proved the Equicontinuity Lemma. Start by checking that the class f!lt of remainder functions r(·, B) has an envelope in f.e2(P). For B =f. 0, Ir(x, B)I
Ilx - BI - Ixl - B'A(x)I/IBI ::; IBI-1(lx - BI2 - IxI 2)/(lx - BI + Ixl) ::; (21 x I + IBI)/( Ix - BI + Ix I) + I ::; 4.
=
+
I
It follows that I· - BI is differentiable in quadratic mean at B = 0. We have only to verify condition (16) of the Equicontinuity Lemma to complete the proof of stochastic equicontinuity. Each r(·, B), for B =f. 0, can be broken into a difference of two bounded functions: r 1(-,
B)
=
B'A(·)!IBI,
r2(', B)
=
(Ix - BI - Ixi)/IBI.
Write f!lt 1 and f!lt 2 for the corresponding classes offunctions. The linear space spanned by ~ 1 has finite dimension; the graphs have polynomial discrimination, by Lemma 11.28; the covering numbers Niu, P n , ~l) are bounded by a polynomial Au- w in u-r, with A and W not depending on P n (Lemma 11.36). The graphs of functions in f!lt2 also have polynomial discrimination, because {(x, t): Ix - BI - Ixl ;;::: IBlt} can be written as {- 2B'x
+ IBI2;;::: 21Bllxlt + IB1 2 t 2 } n
{Ixl
+ IBlt;;::: o} u
{Ixl
+ IBlt <
a}.
This is built up from sets of the form {g ;;::: o} with g in the finite-dimensional vector space of functions 2 g".p,y,b.e,(X, t) = a'x + f3lxl + ylxlt + bt + et +
r
The covering numbers for f!lt 2 are also uniformly bounded by a polynomial in u- 1 • These two polynomial bounds combine (Problem ILl8) to give a similar uniform bound for the covering numbers of ~, which amply suffices for the Equicontinuity Lemma: for each Yi > there exists a y such that J 2(y, P n , ~) ::; Yi for every P n . The conditions of Theorem 5 are all satisfied; the central limit theorem for {Bn} is established. 0
°
19 Example. Independent observations are sampled from a distribution P on the real line. The optimal 2-means cluster centers an, bn minimize W(a, b, Pn) = Pnia,b' where /a,b(X) = Ix - al 2 /\ Ix - b1 2 • In Examples 1104 and 11.29 we found conditions under which an, bn converge almost surely to the centers a*, b* that minimize W(a, b, P) = Pia,b' Theorem 5 refines the result to a central limit theorem.
154
VII. Central Limit Theorems
Keep the calculations simple by taking P as the Uniform[O, 1J distribution. The argument could be extended to other P distributions, higher dimensions, and more clusters, at the cost of more cumbersome notation and the imposition of a few extra regularity conditions. The parameter set consists of all pairs (a, b) with a ::; b ::; 1. For the Uniform[O, 1J distribution direct calculation gives explicitly the values a*, b* that minimize W(a, b, P).
°: ;
W(a, b, P)
=
f {O ::; x < tea + b)} Ix -
a 12
+ H(a + b)::; x::; 1}lx - bl 2 dx = j-a 3 + j-(1 Minimizing values: a* = centers,
=
W(a, b, P)
b? + #b -
a)3.
t, b* = t, as you might expect. Near these optimal
is + i(a - t)2 -
t(a - t)(b -
t) + i(b
-
t)2
+ cubic terms The function!a,b(x) has partial derivatives with respect to a and b except when x = tea + b). That suggests for Ll(x) the two components LlaCx)
=
-2(x - t){O ::; x <
Llb(x)
=
-2(x -
t){t ::; x
t},
::; 1}.
Both functions belong to 5l'2(p). The remainder function is defined by subtraction of the linear approximation from!a,b- Simplify the notation by t = b - t; change h,b to gs,t and r(·, a, b) to R(., s, t). writing s = a -
t
(Isl
+ Itl)R(x, s, t) =
gs)x) - go,o(x)
+ 2t(x
-
+ 2s(x
t){t ::; x
- t){O ::; x
::; 1}
= a piecewise.linear function of x.
envelope~
rR(.,s,t) I
o
/
,
<
t}
155
VII.S. Empirical Central Limit Theorems
The remainder functions are bounded by a fixed envelope in 2 2 (P).
IR(x, s, t)1
~ ~
[I(x - i - s? - (x - ·
Deduce differentiability in quadratic mean offa.b at the optimal centers. The graphs of the piecewise linear functions in fJil have only polynomial discrimination, and they have an envelope in 2 2 (P). Lemma II.36 gives a uniform bound on the covering numbers that ensures J 2 (y, Pn , fJil) ~ I'] for every Pn if "I is chosen small enough. The Equicontinuity Lemma applies; the processes {Enr(·, a, b)} are stochastically equicontinuous at (i, i); the optimal centers obey a central limit theorem (nl/2(a n
where
-
i),
n 1/2(b n
-iJ 3
'4
i)) ~ N(O, V- 1p(.1.1')V- 1),
-
'
P(.1.1')
=
[2;
2~l
D
VII,S, Empirical Central Limit Theorems As random elements of D[O, IJ, the uniform empirical processes {Un} converge in distribution to a brownian bridge. More generally, the empirical processes {En} for observations from an arbitrary distribution on the real line converge in distribution, as random elements of D[ - 00, ooJ, to a gaussian process obtained by stretching out the brownian bridge. Both results treat the empirical measure as a process indexed by intervals of the real line. In this section we shall generalize these results to empirical measures indexed by classes of functions. Convergence in distribution, as we have defined it, deals with random elements of metric spaces. Once we leave the safety of intervals on the real line it becomes quite a problem to decide what metric space of functions empirical process sample paths should belong to. Without the natural ordering of the intervals, it is difficult to find a completely satisfactory substitute for the cadlag property; without the simplification of cadlag sample paths, empirical processes run straight into the measure-theoretic complications we have so carefully been avoiding. Appendix C describes one way of overcoming these complications. A class offunctions satisfying the regularity conditions described there is said to be permissible. Most classes that arise from specific applications are permissible. Let :F be a pointwise bounded, permissible class of functions for which sup$" IP f I < 00. The empirical processes {En} define bounded functions on :F; their sample paths belong to the space !![ of all bounded, real functions
156
VII. Central Limit Theorems
on fF. To avoid some of the confusion that might be caused by the hierarchy of functions on spaces of functions on spaces of functions, call members of :!l' functionals. Equip :!l' with the metric generated by the uniform norm, Ilxll = suP.¥ Ix(f)I· Be careful not to confuse the norm 11-11 on :!l' with the se 2 (P) seminorm pp(.) on :F. The choice of a-field for :!l' is tied up with the measurability problems handled in Appendix C. We need it small enough to make En a measurable random element of :!l', but large enough to support a rich supply of measurable, continuous functions. The limit distributions must concentrate on sets of completely regular points (Section IV.2). That suggests that the a-field should at least contain the balls centered at the functionals that are uniformly continuous for the pp seminorm. 20 Definition. Write C(:F, P) for the set of all functionals x(·) in :!l' that are uniformly continuous with respect to the se 2 (P) seminorm on fF. That is, to each e > 0 there should exist ab> 0 for which Ix(f) - x(g) I < e whenever pp(f - g) < b. Define (lAP as the smallest a-field on :!l' that: (i) contains all the closed balls with centers in C(:F, P); (ii) makes all the finite-dimensional projections measurable. 0
Notice that c(.?F, P) is complete, because it is a closed subset of the complete metric space (:!l', 11·11). Notice also that f!JP depends on the sampling distribution P. Each En is a (lAP-measurable random element of:!l' under mild regularity conditions (Appendix C). The finite-dimensional projections of {En} (the fidis) converge in distribution to the fidis of E p , the P-bridge process over :F (Definition 14). Of course some doubts arise over the existence of Ep; getting a version with sample paths in c(.?F, P) is no simple matter, as we saw in Section 3. Happily, the questions of existence and convergence are both taken care of by a single property of the empirical processes, uniform tightness. Recall from Section IV.5 that uniform tightness for {En} requires existence of a compact set K, of completely regular points in :!l' such that liminf IP{E nE G} > 1 - e for every open, f!JP-measurable set G containing K,. From uniform tightness we would get a subsequence of {En} that converged in distribution to a tight borel measure on :!l'. If c(.?F, P) contained each K" the limit would concentrate in C(.?, P). Its fidis would identify it (Problem 8) as the P-bridge over :F. Uniform tightness of {En} would also imply convergence of the whole sequence to Ep. For if {IPh(E n)} did not converge to IPh(Ep) for some bounded, continuous, f!JP-measurable h on :!l' then jIPh(En) - IPh(Ep) I > e infinitely often
for some e > O. The subsequence along which the inequality held would also be uniformly tight; it would have a sub-subsequence converging to a
157
VII.5. Empirical Central Limit Theorems
process whose fidis still identified it as a P-bridge. That would give a contradiction: along the sub-subsequence, {lPh(En)} would converge to IPh(Ep) without ever getting closer than 8. 21 Theorem. Let g; be a pointwise bounded, totally bounded, permissible subset of :£Z(P). If for each I] > 0 and 8 > 0 there exists a [) > 0 for which
(22)
limsup IP{su p IEn(f - g) I >
I]}
<
8,
[0)
where [6] = {(f, g): f, g E g; and pp(f - g) < 6}, then En ~ Ep as random elements of f![. The limit P-bridge process Ep is a tight, gaussian random element of f![ whose sample paths all belong to C(ff, P). Check the uniform tightness. Given 8 > 0 find a compact subset K of C(g;, P) with liminf IP{E n E G} > 1 - 8 for every open, [!6P-measurable G containing K. Construct K as an intersection of sets D1 , D z , ... , where Dk is a finite union of closed balls of radius k- 1 centered at points of C(ff, P). Every functional in Dk will lie uniformly within k- 1 of a member of C(ff, P); every functional in K will therefore belong to C(.?i', P), being a uniform limit offunctionals in C(.?i', P). The prooffollows closely the ideas used in Theorem V.16 to prove existence of the brownian bridge. Only the continuous interpolation between values taken at a finite grid requires modification. Fix I] > 0 and 8 > 0 for the moment, and choose 6 according to (22). Invoke the total boundedness assumption on.?i' to find a finite subclass.?i'o = Ul' ... ,fm} of .?i' such that each f in g; has an f* in g;;; for which pp(f - f*) < !6. We need to find a finite collection of closed balls in f![, each with a specified small radius and centered on a functional in C(ff, P), such that En lies with specified large probability in the union of the balls. Construct the centers for the balls by a continuous interpolation between the values taken on at each f in g;;; by realizations of an En. For each f, the sequence of random variables {En(f)} converges in distribution. There exists a constant C for which PROOF.
limsup
lP{m~x IEn(f) I >
c}
<
8.
Define On = {W:sup IEn(w,f - g)1 :-:; [0)
I]
and max IEn(w,f)1 :-:; i
c}.
By the choice of [) and C we ensure that liminf IPO n > 1 - 28. Write Sn for the bounded set of all points in lR m with coordinates En< w, f) for an W in On, and S for the union of the {Sn}. If r belongs to S then Ird :-:; C for each coordinate and Iri - rj I :-:; 21] whenever there exists an f in .?i' with pp(f - f) < 6 and pp(f - fj) < 6.
158
VII. Central Limit Theorems
Construct weight functions
~;(.)
on §' by
Vi(f) = [1 - pp(f - !;)/c5] +, ~i(f)
= vi(f)/[v 1 (f) + ... + vm(f)].
Each v;(·) is a uniformly continuous function (under the pp seminorm) that vanishes outside a ball of radius 6 about k For every f there is at least one !;, its f*, for which v;(f) > !; the denominator in the defining quotient for ~i is never less than t. The ~;(.) are non-negative, uniformly continuous functions that sum to one everywhere in fi'. For r in S define an interpolation function by x(J, r) = L:7'= 1 ~i(f)ri' Each x(·, r) belongs to C(§'; P). If p P(f - jj) < 6, all the r i values corresponding to non-zero {~i(f)} satisfy Iri - rj I .: :; 2'1. As a convex combination of these val ues, x(f, r) must also lie within 2'1 of rj ' An En< w, .) corresponding to an w in On has a similar property: IEn<w,J) - En<w,jj)l.:::;; '1
when
pp(f - jj) < 6.
Thus, if w belongs to On and IEn(w,jj) - rjl .:::;; '1 for every j then supIEn(w,f) - x(f, r)l.:::;; 4'1. ff
Choose from the bounded set S a finite subset {r(l), ... , r(p)} for which min max Irj - rik) I .: :; '1 k
for every r in S.
j
Abbreviate x(·, r(k)) to x k ('), for k = 1, ... , p. Then from what we have just proved min sup IEn<w,J) - xk(f) I .: :; 4'1, k
ff
whenever w belongs to On. If we set D equal to the union of the balls B(Xl' 4'1), ... , B(x p , 4'1), then
liminf IP{En E D} > 1 - 28. Repeat the argument with '1 replaced by (4k) - 1 and 8 replaced by 812k + 1, for k = 1, 2, ... , to get each of the Dk sets promised at the start of the proof: Dk is a finite union of closed balls of radius k - 1 and liminf IP{En E Dd > 1 - 812k. The remainder of the proof follows Theorem Y.16 almost exactly. The intersection of the sets D 1, Dz , .. , is a closed and totally bounded subset of the complete metric space C(§'; P); it defines the sought-after compact K. The open G contains some finite intersection Dl (\ .,. (\ D k • If not, there would exist a sequence Y = {yd with Yk in GC (\ Dl (\ ... (\ Dk for each k. Some subsequence Y' of Y would lie within one of the balls making
159
VII.S. Empirical Central Limit Theorems
up D I ; some subsequence Y" of Y' would lie within one of the balls making up D 2 ; and so on. The sequence constructed by taking the first member of Y', the second member of ylI, and so on, would be Cauchy; it would converge to a point y (PI' is complete) belonging to all the closed sets {Gc (J DI (J ••• (J Dd. This would contradict
n GC 00
(J
DI
(J ••• (J
Dk = GC
(J
K
= 0.
k=1
Complete the uniform tightness proof by noting that liminf IP{En E G} if G contains DI
(J ••• (J
~
liminf IP{E n E DI
(J ••• (J
Dk} > 1 -
6
o
Dk •
Condition (22) points the way towards mass production of empirical central limit theorems. The Chaining Lemma makes it easy. For example, from the Equicontinuity Lemma of Section 4, we get conditions on the random covering numbers under which {En} converges in distribution. The next section will describe other sufficient conditions.
23 Example. We left unfinished back in Example V.15 a limit problem for goodness-of-fit statistics with estimated parameters. The empirical processes were indexed by intervals of the real line ; the estimators took the form n
en = eo + n- I
I
L(~i)
+
oin- 1/2 )
i= I
for an L with PL = 0, of
pe
<
Dn = sup IEn( -
00.
00,
We wanted to find the limiting distribution
t] - nl/2(en - eo)~(t) I + oil)
t
the ~(.) being a fixed cadlag function on [ - 00, 00]. Set fJ' equal to {L} u {( - 00, t]: - 00 < t < oo}. Express Dn in terms of a function on the corresponding PI'. Define
H(x) = sup Ix(( -
00,
t]) - x(L)L\(t) I.
Guard against measurability evils by restricting the supremum to rational t values: it makes no difference to En, L\, or the limiting P-bridge. Clearly H( . ) is a continuous function on PI'. You can check condition (22) by means of the Equicontinuity Lemma. The intervals have only polynomial discrimination; the inclusion of the single extra function L has a barely perceptible effect on the covering numbers. Deduce that Dn = H(En) + op(l) ~ H(Ep). 0
160
VII. Central Limit Theorems
VII.6. Restricted Chaining In this section the method of the Chaining Lemma is modified to develop another approach to empirical central limit theorems. The arguments for three representative examples are sketched. You might want to skip over the section at the first reading. The chaining arguments in Section 2 assumed that the increments of the stochastic process had exponentially decreasing tail probabilities,
The inequality held for every 1] > 0 and (j > O. We shall carry the argument further to cover processes, such as the empirical process, for which the inequality holds only in a restricted region f?) of (I], (j) pairs. Suppose f is a bounded function, If I :s;; c. Let (j2 be an upper bound for the variance (J2(f) = (Pf)2. Bennett's Inequality (Appendix B) gives
pr -
(25)
IP{IEnfl >
1]}
=
IP{lit
12 f(O - Pfl > I]n / }
:s;; 2 exp[ _t(1]2 j(j2)B(2C1]j(nl/2(j2))] :s;; 2 exp( -tAI]2j(j2)
if (j2jl]
~
2Cj(n 1/2B- 1(A))
for any fixed A between 0 and 1, because B(·) is a continuous, decreasing function, with B(O) = 1. The restricted range complicates the task of proving maximal inequalities for the stochastic process {Z(t): t ET}. We can chain as in Section 2 as long as the (I]i' (ji) pairs remain within ~ but eventually the chain will hit the boundary of ~ when the links are getting down to lengths less than some tiny a, say. That leaves the problem of how to bound increments of Z across little links from points in T to their nearest neighbors in an a-net for T. Remember the abbreviations N((j), for the covering number N((j, d, T), and l((j), for the covering integral l((j, d, T)
=
1:
[2Iog(N(u)2ju)r/2 duo
The chaining argument will work for maximal deviations down to about l(a). That explains the constraint l(a) :s;; yj12D in the next theorem. The other constraints on a and y are cosmetic. 26 Theorem. Let {Z(t): t ET} be a stochastic process that satisfies the exponential inequality (24)for every I] > 0 and (j > 0 with (j ~ aI]1/2,for some
constant a. Suppose T has afinite covering integral l(·). Let T(a) be an a-net (containing N (a) points) for T; let ta be the closest point in T( a) to t; and let
161
VII.6. Restricted Chaining
[6] denote the set of pairs (s, t) with des, t) ::; 6. Given e > exists a 6 > 0, depending on e, y, and J(.), for which
IP{s~f IZ(s) provided
0( ::;
Z(t) 1 > 5Y}::; 2e
+ IP{s~p IZ(t)
°
and y >
°
there
- Z(ta) 1 > y}
ie and y ::; 144 and J(O() ::; min{y/12D, 3/D}.
PROOF. The argument is similar to the one used for the Chaining Lemma. Write H(u) for [2 log(N(u)2/u)] 1/2, as before. Choose the largest 6 for which 6 ::; ie and J(6) ::; y/12D. The assumptions about 0( ensure 6 ;;::: 0(. Find the integer k for which 6 < 3k O( ::; 36 then define 6i = 3k- iO( and fJi = D6 i H(6 i+1) for i = 0, ... , k. Notice that 6 1 ::; 6 < 60 and 6k = 0(. Also k-1 fJo + ... + fJk-1 = L~D(6i+1 - 6i+2)H(6 i+1)
::;~D
Jo
f{6 i+2 ::; u < 6i+dH(u)du
::; ~DJ(61)
< Y because J(6 1) ::; J(6) ::; y/12D. Choose 6i -nets 7i containing N(6 i) points, making sure that each t to a to in To through a chain of points,
1k
= T(O(). Link
with ti being the closest point of 7i to ti+ l' By this construction, d(t i+l ' t i) ::; 6i . The smallest value of the ratios {6? /fJi}, for i = 0, ... , k - 1, occurs at i = k - 1; all the ratios are greater than 30(/DH(0() ;;::: 30(2/DJ(0() ;;:::
0(2.
The (fJi' 6i) pairs all belong to the region in which the exponential inequality (24) holds. Apply the inequality for increments across links of the chains. IP{max 1 Z(ta) - Z(to) 1 > y} ::; kI1 IP{max 1Z(t i+1) - Z(tJ 1 > fJi} T(a)
,~o
T'+l
k-1
::; L N(6 i+1)2 exp( -tfJ?/D 26f) 00
::; L 2N(6i+ 1) exp[ -log(N(6 i + 1)2/6i+ 1)]
Notice how one of the N(6 i + 1) factors was wasted; both factors will be needed later. The last series sums to less than e because 6 1 ::; ie and the {6;} decrease geometrically.
162
VII. Central Limit Theorems
--~--------------~-----~
- - - - - - - " * - - - - - - + - t . - - T(a)
Join each (s, t) pair in [15] by two chains leading up to To plus a link between So and to· sup IZ(s) - Z(t) I ~ 2 sup IZ(t) - Z(ta) I + 2 max IZ(ta) - Z(to) I [0]
T
T(a)
+ sup IZ(So)
- Z(to) I·
[0]
Partition the 5y correspondingly. IP{
s~r IZ(s) -
Z(t) I > 5Y}
~ IP{ s~p IZ(t) -
Z(ta) I > Y}
+ IP{su p IZ(so) -
+s
Z(to) I >
y}.
[0]
The distance between the is less than
So
and to of each pair appearing in the last term
k-l
~
k-l
L bi + !Y. + 15 + !Y. + L bi i=O
~
315 0
~
1215.
i=O
+ 2!Y. + 15
There are at most N(b o)2 such pairs. The exponential inequality holds for each pair, because 12b/(!Y.yl/2) ;;::: 12y-l/2 ;;::: 1. IP{sup IZ(so) - Z(to) I >
y}
[0]
~
N(b o)Z2 exp( -h 2/144D 2b2)
~
215
because d(so, to) ~ 1215 Z 2 ~ 215 exp[log(N(b)Z /15) - h2/144D b ] because N(b o) ~ N(b) ~ 215 exp[tb- Z(H(b)Zb 2 - y2/144D 2 )]
< s.
because
H(b)b
~
J(b)
~
y/12D
o
163
VII.6. Restricted Chaining
Theorem 21 states a sufficient condition for empirical processes indexed by a pointwise bounded, totally bounded, permissible subclass ff of .;tJ2(p) to converge in distribution to a P-bridge: given I] > 0 there exists a £5 > 0 for which limsup IP{sup IEnU - g) I > I]} < e. [b]
For a permissible class of bounded functions, say 0 S ! s 1, any condition implying finiteness of N 1(', P, ff) or Nz(·, P, ff) will take care of the total boundedness. Finiteness of a covering integral will allow us to apply Theorem 26, leaving only a supremum over the class Ye = {f - fa:! E ff} of little links. It will then suffice to prove SUPK IEnhl = op(l) to get the empirical central limit theorem. Notice that rt., and hence Ye, will depend on n. The next three examples sketch typical methods for handling Ye. 27 Example. Equip ff with the semi metric dU, g) = (PI! - gl)1/ z. (This is the .;tJZ(P) seminorm applied to the function I! - g 11/Z.) The square root ensures that the variance (J2U - g) is less than dU, g)z. If we take A = t, the exponential bound (25) becomes, for dU, g) S £5,
IP{IEnC! - g)1 > I]}
s
2 exp( -i1J z/£5 Z)
if £5zl1J ~ 2/(n 1 / 2 B- 1 (t».
That is, D = 2 and rt. = (2/B- 1(t»1/2 n -1/4 for Theorem 26. The covering numbers for d(·, .) are closely related to the .;tJ1(p) covering numbers: in terms of the covering integral,
If J is finite, Theorem 26 can chain down to leave a class Ye of little links with Ihis 1 and PI his rt. 2 . If we add to this the condition (29)
log N 1(cn- 1/Z, Pn, Ye) = op(n1/Z)
for each c > 0,
the empirical central limit theorem will hold. The methods of Section II.6 work for the class Ye1 / 2 = {I h 11 / Z : hE Ye}. Notice that
N 2 (£5, Pn , Ye1 / Z ) S N 1 (£5 Z , P n , Ye) because Pn(lh 111/2
-
Ih211/2)2 S P nlh 1 - h 2 1. From Lemma II.33,
(30) IP{S,(P n l h l )1/Z > 8rt.} S 4IP[N 1(rt. z, P n , Ye) exp( -nrt. 2 )
= 4IP[exp(log N 1 (rt. Z , Pn , Ye) -+
0
by (29).
/\
1J
- nrt. Z )
/\
1J
164
VII. Central Limit Theorems
Symmetrize. For n large enough,
IP{s~p IEnhl > 4Y} :s; 4IP{S~P IE~hl >
y}.
Condition on 1;. Cover Ye by M = N 1 (tyn- 1 / 2 , Pn , Ye) balls, for the ft'l(P n ) seminorm, with centers gl' ... , gM in .Yf. Then as in Section 11.6,
IP{s~ IE~hl > yll;} :s; M m~x IP{ IE~gjl > hll;}· On the set of I; where suP.Yt' PnIh I :s; 640: 2, Hoeffding's Inequality bounds the right-hand side by 2 exp[log M - ~(!y)2/(640:2)] which is of order oil) because (29) says log M = oin1/2). The central limit theorem follows. 0 31 Example. The direct approximation method of Section 11.2 gave uniform strong laws of large numbers. With a suitable bound on the number of functions needed for the approximations, we get central limit theorems. Define a direct covering number Ll(b, P, Ye) as the smallest M for which there exist functions gl' ... , gM such that, for every h in £,
Ihl :s; gi
°
and
Pgi:S; b
We may assume :s; gi :s; 1. If (32) log Ll(cn- 1 / 2 , P, Ye)
+ Plhl
= 0(n 1 / 2 )
for some i.
for each c > 0,
and if the covering integral (28) from the previous example is finite, then the empirical central limit theorem holds. Given y > 0, choose A in the exponential inequality (25) so that 2/B- 1 (A) = y. The dependence of A on y does not vitiate the chaining argument in Theorem 26; it does ensure that functions in Ye satisfy Plhl :s; 0: 2 = yn- 1/2 • Find gl"'" gM according to the definition of Ll(yn- 1/ 2 , P, Ye). Because Pg i :s; 2yn -1/2 for each i, the contributions of the means to En are small.
IP{s~ IEnhl > 4Y} :s; IP{s~ n1/2Pnlhl > 3Y} :s;
IP{m~x n 1/ Pngi > 3Y} 2
:s; M max IP{Engi > y}
because n1/2Plhl :s; y because Ihl :s; gJor some i because n1/2Pg i :s; 2y
i
:s; M max 2 exp[ _~(y2 /Pg i)B(2y/(n 1/2Pg i))]
from (25)
i
:s; 2 exp[log M -
= 0(1) by (32).
iYn 1/ 2 B(1)]
o
165
Notes
33 Example. In the previous two examples, the method of chaining left links of small 2 1 (P) seminorm at the end of the chain; 21 approximation methods took care of Yt. If we chain instead with 2 2 (P) covering numbers, we need 22 approximation methods for Yt. Set d(·, .) equal to the 2 2(P) semimetric. Because a 2(f - g) s d(!, g)2, the chaining down to £' requires J il, P, $') finite. At the end
Ph 2 S a 2 = (2jB- 1(i»n- 1 / 2 •
Invoke Lemma II.33.
IP{S~(Pnh2)1/2 S
8a}
->
1 as
n
-> 00
if the random covering numbers satisfy log N 2(cn- 1/4, Pn, £') = op(n 1/2) for each c >
o.
This would follow from (34) because oin1/4) = (cn-1/4)-lJ2(cn-1/4, Pn, £') ~ [210g(N 2 (cn- 1 /4, P n , £,)2n 1/4jcW/ 2.
Symmetrize. For all n large enough,
IP{s~p IEnhl > 4Y} s 4IP{S~ IE~hl > Y}. Now we are back to the sort of problem we were solving in Section 4. Condition on ;. On the set of those; for which sup£(P n h 2)1/2 S 8a, chain using the Hoeffding Inequality to bound the tail probabilities. Apply the Chaining Lemma for IP( ·1;), the 2 2 (P n) seminorm, and I> = 8et.
IP{s~ IE~hl > 26J 2(8a, Pn , £')I;} s
16a
if
S~(Pnh2)1/2 S
8a.
Condition (34) and finiteness of J 2(1, P, $') are sufficient for the empirical central limit theorem to hold. 0 NOTES
Theorem 5 draws on ideas from Chernoff (1954), but substitutes stochastic equicontinuity where he placed domination conditions on third-order partial derivatives. The theorem also holds if to is just a local minimum for F(·), or if 'n is a minimum for F nO over a large enough neighborhood of to. Huber (1967, Lemma 3) made explicit the role of stochastic equicontinuity in a proof of the central limit theorem for an M-estimator.
166
VII. Central Limit Theorems
The chaining argument abstracts the idea behind construction of processes on a dyadic rational skeleton. It appears to have entered weak convergence theory through the work of Kolmogorov and the Soviet School; it is closely related to the arguments for construction of measures in function spaces (Gihman and Skorohod 1974, Sections 111.4, 111.5). The Chaining Lemma is based on an arrangement by Le Cam (1983) of an argument of Dudley (1967a, 1973, 1978). Le Cam's approach avoids the complications introduced into Dudley's proof by the nuisance possibility that covering numbers N(b) might not increase rapidly enough as b decreases to zero. Alexander (1984a, 1984b) has refined Dudley's form of the chaining argument to prove the most precise maximal inequalities for general empirical processes to be found in the literature. Theorem 13 is based on Theorem 2.1 of Dudley (1973), but with his modulus function increased slightly to take advantage of Le Cam's (1983) cleaner bound for the error term. The extra (b 10g(1jb»1/2 does not change the order of magnitude of the modulus for most processes. The argument in Section 4 is based on Pollard (1982c), except for the substitution of convergence in probability (condition (16» for uniform convergence. Kolchinsky (1982) developed a similar technique to prove a similar central limit theorem for bounded classes of functions. He imposed finiteness of J 2(·, P, :#') plus a growth condition on N 1( ., Pn,:#') to get results closer to those of my Example 27. Gine and Zinn (1984) have found a necessary and sufficient random entropy condition for the empirical central limit theorem. Brown (1983) sketched the large-sample theory for the spatial median. He referred to Brown and Kildea (1979) and the appendix he wrote for Maritz (1981) for rigorous proofs, which depend on a form of stochastic equicontinuity. The central limit theorem for k-means was proved by Pollard (1982b, 1982d) for a fixed number of clusters in euclidean space. The one-dimensional result was proved by Hartigan (1978), using a different method. Dudley (1978, 1981a, 1981b, 1984) has developed the application of metric entropy (covering numbers) to empirical process theory. These papers extended his earlier work on entropy and sample path properties of gaussian processes (1967b, 1973), and on the multidimensional empirical distribution function (1966a). Dudley (1966a, 1978) introduced most of the ideas needed to prove central limit theorems for empirical processes indexed by sets. He extended these ideas to classes of functions in (1981a, 1981b). His lecture notes (1984) provide the best available overview of empirical process theory, as of this writing. The proof of my Theorem 21 was inspired by Chapter 4 of those lecture notes, which reworked ideas from Dudley and Philipp (1983). If P f = 0 for each f in :#', a standardization that can be imposed without affecting En or E p , the conditions of Theorem 21 are also necessary for the empirical central limit theorem.
167
Problems
The first central limit theorems for empirical processes indexed by classes of sets were proved by the direct approximation method. Bolthausen (1978) worked with the class of compact, convex subsets of the unit square in JR 2 • He applied an entropy bound due to Dudley (1974). Revesz (1976) indexed the processes by classes of sets with smooth boundaries. Earlier work of Sun was, unfortunately, not published until quite recently (Pyke and Sun 1982). Dudley's (1978) Theorem 5.1 imposed a condition on the "metric entropy with inclusion" that corresponds to finiteness of a covering integral. Strassen and Dudley (1969) proved a central limit theorem for empirical processes indexed by classes of smooth functions. They deduced the result from their central limit theorem for sums of independent random elements of spaces of continuous functions. All these theorems depend on existence of good bounds for the rate of growth of entropy functions (covering numbers). For more about this see Dudley (1984, Sections 6 and 7) and Gaenssler (1984). Theorem 26 resets an argument of Le Cam (1983). Such an approximation theorem has been implicit in the work of Dudley. Gine and Zinn (1984) have pointed out the benefits of stripping off the 2 2 (P) chaining argument, to expose more clearly the problem of how to handle the little links left at the end of the chain. They have also stressed the strong parallels between empirical processes and gaussian processes. The examples in Section 6 follow the lead of Gine and Zinn: Example 27 is based on their adaptation of Le Cam's (1983) square-root trick; Example 31 is based on their improvement of Dudley's (1978) "metric entropy with inclusion" method; Example 33 is based on their Theorem 5.5. PROBLEMS
[IJ Prove that the stochastic equicontinuity concept of Definition 2 follows from: Zn( Tn) - Zn(t O) ..... 0 in probability for every sequence {Tn} that converges in probability to to' [Suppose the defining property fails for some IJ > 0 and 8 > O. For a sequence ofneighborhoods {Uk} that shrink to to find positive integers n(1) < n(2) < ... with lP{sup IZn(k)(t) - Zn(k,(t O) I > IJ} > u.
ts
for every k. Choose random elements {Tn} of T such that, for n(k) ::; n < n(k
+
1),
IZnCw, TnCW)) - Zn(w, to) I 2': ! sup IZn(w, t) - ZnCw, to) I u.
and Tn(w) belongs to U k' Appendix C covers measurability of Tn.J [2J Let {f(., t): t ET} be a collection of IRk-valued functions indexed by a subset of IRk. Suppose Plfe-, tW < Cf) for each t. Set F(t) = Pfe-, t) and Fn(t) = Pnf(" t). Let {Tn} be a sequence converging in probability to a value to at which F(to) = O. If (a) Fe-) has a non-singular derivative matrix D at to; (b) FIi(Tn) = Oin-1/2);
168
VII. Central Limit Theorems
(c) {Enf(" t)} is stochastically equicontinuous at to; then n 1/1(Tn - to) ...... N(O, D- 1P[f(" to)f(', to)']D-1). [Compare with Huber (1967).J [3J For a class ff to index a P-motion it must be totally bounded under the 'pl(P) seminorm pp. [First show ff is bounded: otherwise IBp(f,,)I--- CfJ in probability for some {f,,}, violating boundedness of P-motion sample paths. Total boundedness will then follow from: for each e > 0, every f lies within e of some linear combination of a fixed, finite subclass of:F. If for some e no such finite subclass exists, find {f,,} such that n
fn+1
=
gn+1
+
Ianjij, j= 1
where pp(gn+ 1) :2: e and gn+ 1 is orthogonal to f1' ... ,f". Fix an M. Show that there exists aD> 0, depending on M and e, for which
Deduce that JP {suPn B if,,) :2: M} = 1 for every M, which contradicts boundedness of the sample paths. Notice that continuity of the sample paths does not enter the argument. Dudley (1967b).J [4J If sup.? IPfl is finite then ff must be totally bounded under the 'pl(P) seminorm pp if it supports a P-bridge. [Choose Z with a N,(O, 1) distribution independent of Ep. The process B(f) = EP(f) + ZPf is a P-motion with bounded sample paths. Invoke Problem 3. The condition on the means is needed-consider the ff consisting of all constant functions. The P-bridge is unaffected by addition of arbitrary constants to functions in ff; it depends only on the projection of ff onto the subspace of 'pl(P) orthogonal to the constants.J [5J Let J"t'1 be a class offunctions with an envelope H in 'pl(P). Set J"t'l Show that
=
{h l : hE J"t'd.
N 1(4e(QH1)1/1, Q, J"t'l):::; N l (2e, Q, J"t'1)'
[By the Cauchy-Schwarz inequality, Qlhi - h~l:::; Q(2Hlh 1 - hll):::; 2(QH2)1/2(Qlh l
-
h211)I/l
if both Ihll :::; H and Ihll :::; H.J [6J Let ff be a permissible class of functions with envelope F. Suppose Jl(D, P n , ff) = op(n 1 / 2 )
for each
D > O.
[Condition (16) of the Equicontinuity Lemma implies that Jl(D, P n , ff) = Ope!) for each D > O.J Show that J"t'2 = {(f - g)2: f - gEff} satisfies the sufficient condition (Theorem II.24) for the uniform strong law of large numbers: log NI(e, Pn, J"t'2) [Set H
=
2F and J"t'1
=
= open),
for each e > O.
{f - g:J, g E ff}. Show that, for 1 > e > 0,
N 2(2e, P n , J"t'1) :::; Nie, P n, ff)l :::; e exp(Vl(e, P n, ff)l/e l ).
169
Problems
Deduce from this inequality, Problem 5, and the strong law of large numbers for {PnHZ} that, if 1 > e > 0, IP{log N,(4e(2PH2)J/Z, Pn, Jf'z) > nl'/} :0; IP{log NJ(4e(PnHZ)J/z, P n, Jf'z) > nl'/}
+ IP{PnHZ >
2PH Z}
Z Z :0; IP{log N z(2e, P n, Jf',) > nl'/} + IP{PnH > 2PH } z 2 Z :0; IP{tJz(e, P n, ff)z/e > nl'/} + IP{PnH > 2PH } --+ O.
A weaker result was proved by Pollard (1982c).J [7J If ff is totally bounded under the !£,Z(P) seminorm, then the space C(§; P) of bounded, uniformly continuous, real functions on ff is separable. [Suppose IxU) - x(g) I < e whenever pP(f - g) :0; 2b. Choose {f" ... ,fm} as a maximal set with pp(J; - fj) ~ tb. Use the weighting functions L1l·) from the proof of Theorem 21 to interpolate between rational approximations to the {x(J;)}.J [8J Suppose ff is totally bounded under the !£,2(P) seminorm. If two probability measures A and J1 on the o--field fJP have the same fidis, and if both concentrate on C(§; P), then they must agree everywhere on fJP. [Show that A and J1 agree for all finite intersections offidi sets and closed balls with centers in C(ff, P). For example, consider a closed ball B(x, r) with x in C(ff, P). Let {fJJ2, ... } be a countable, dense subset of C(ff, P). Define Bn
=
{z E C(§; P): Iz(J;) - x(!;)1
:0;
r for 1 :0; i
:0;
n}.
Show that J1B(x, r) :0; J1B n = ABn --+ AB(x, r) as n --+ 00. Extend the result to finite collections of closed balls and fidi sets, then apply a generating-class argument.J [9J The property that the graphs have only polynomial discrimination is not preserved by the operation of summing two classes of functions. That is, both ff and '§ can have the property without the class g' = {f + g: f E ff, g E '§} having it. Let !l2 = {DJ, D2, ... } be the set of indicator functions of all finite sets of rational numbers in [0, 1]. Let ff = {2n + Dn: n = 1,2, ... } and '§ = {-2n: n = 1,2, ... }. The graphs from neither class can shatter two-point sets, but g' can shatter arbitrarily large finite sets of rationals in [0, 1]. [The roundabout reasoning used to bound the covering numbers in Example 18 may not be completely unnecessary.J
CHAPTER VIII
Martingales ... in which martingale central limit theorems in discrete and continuous time are proved. An extended non-trivial application to Kaplan-Meier estimatorsestimation of distribution functions from censored data-is sketched.
VIlLI. A Central Limit Theorem for Martingale-Difference Arrays Martingale theory must surely be the most successful of all the attempts to extend the classical theory for sums of independent random variables to cover dependent variables. Many of the classical limit theorems have martingale analogues that rival them for elegance and far exceed them in diversity of application. We shall explore two of these martingale theorems in this chapter. One main change in technique will become apparent. Where proofs for independent summands use truncation to protect against occasional abnormally large increments-the sort of thing implicit in something like the Lindeberg condition-martingale proofs can resort to stopping time arguments. Optional stopping preserves the conditional expectation connection within a martingale sequence as long as the decision to stop is based only on past behavior of the sequence. The prohibition against peering into the future to anticipate abnormally large increments imposes a characteristic feature on martingale theorems. One needs conditions that protect against the behavior of the worst single increment, because the decision to stop can be taken only after that increment has had its effect. In central limit theorems, independence allows one to factorize expectations of products and separate out the contribution of a particular increment from contributions of past and future increments. The martingale property allows a weaker factorization, for expectations conditional on the past alone. Conditional variances take over the role played by variances of independent summands. But apart from that, the arguments for martingales share the
171
VilLI. A Central Limit Theorem for Martingale-Difference Arrays
same inspiration as the proof of the Lindeberg Central Limit Theorem in Section IIl.4. We shall prove asymptotic normality for row sums of martingale-difference arrays. That is, for each n we have random variables ~nl' ... , ~nn (we avoid some messy notation, and lose no generality, by assuming exactly n variables in the nth row) and (J-fields Cno S; ... S; Cnn for which (a) (b)
IP(~njICn,j_1)=Oforj=
~nj
1, ... ,n;
is Cnrmeasurable.
Define conditional variances Vnj=IP(~~jICn,j-1)
for j=l, ... ,n.
Notice that vnj is an Cn,j_1-measurable random variable. Convergence of sums of conditional variances will be the only connection tying together the variables in different rows of the array. 1 Theorem. Let
{~n}
be a martingale-difference array. If as n ~
00,
(i) Lj vnj ~ (J2 in probability, with (J2 a positive constant; (ii) for every I:; > 0, the sum Lj IP( ~~j{ I~nj I > I:;} ICn, j_ 1) converges in probability to zero (a Lindeberg condition); then ~n1
+ ... + ~nn --+ N(O, (J2).
Without loss of generality set (J2 = 1. Let us check pointwise convergence of characteristic functions:
PROOF.
We really will need some of the special multiplicative properties of the complex exponential function, and not just its smoothness by way of three bounded, continuous derivatives as in Section IlI.4. The randomness of the conditional variances fouls up the argument based on successive substitution of matching normal increments, which worked for independent summands. At the risk of notational abuse (more will follow) abbreviate the conditional expectation IP('IC n) to IPk). Write R(x) for the remainder term eix - 1 - ix and Snj for the partial sum ~n1 + ... + ~nj' Define rnj as the conditional expectation IPj - 1R(t~n)' When the gnj} are small, in a sense to be made precise by condition (ii), we shall get rnj ;.: : ; - tVnj' The proof will work by successive conditioning. We pin down the effect of individual increments by evaluating
a layer at a time. Start with the innermost conditional expectation. Factorize out the part depending on Cn,j-1 then expand the remaining exp(it~nn)'
172
VIII. Martingales
The 1 + r nn factor foils the attempt to work the same idea with IP n- 2; it won't cooperate by slipping outside the conditional expectation, leaving exp(itS n,n-i) to enjoy the same treatment from IPn - 2 that exp(itSnn ) received from IPn - i . We could clear away the obstacle by dividing out the offending factor. IPn - 2 IPn - i [(1 + rnn )-i exp(itSnn)J = IP n- 2 exp(itS n,n-i) = exp(itSn,n-2)(1 + rn,n-i)' We could get rid of the (1 + rn,n-i) in a similar fashion. If you pursue this idea through each layer of conditioning, you will see the sense in starting from n
TI (1 + rn)-i exp(itSnn )·
j= 1 The remote possibility that one rnj might get close to -1 could cause minor difficulty when we come to bound the product term. To avoid this, replace (1 + rn)-i by (1 - rn). When rnj : : : ; 0 the change has little effect. Define k
T"k =
TI (1 j=i
rn)
and
Znk = T"k exp(itSnk )·
If we could show IP I7;.. - exp(t t 2) I -+ 0 and IP Znn -+ 1, then it would follow that lIP exp(itSnn ) - exp( -tt 2)1 S exp( -tt 2)[IPI exp(itSnn + tt 2 ) - Znnl + IIPZ nn - IIJ -+ O. We would get the desired results for {T"n} and {Znn} from: (a) Lj r -+ -tt 2 in probability; nj
(b) Lj Ir nj I S t
2
;
(c) max j Irnjl-+ 0 in probability. The second of these requirements need not be satisfied but, without loss of generality, we may act as if it were. We replace enj by enjU S ITn}, where ITn = max{k: LJ=l Vnj S 2}. Interpret ITn as zero if Vni > 2. Because vnj is 6" n,j- i-measurable, the event Us ITn} is 6"n,j_i-measurable; ITn is a stopping time. The new variables are martingale differences. The new row sums have the same asymptotic behavior as the original row sums: IP{Ji enj -=I Ji enjU S ITn} } S IP{n > ITn} = IPLti Vnj >
2}
-+
O.
The quantities wnj = IP j - i R(tenjU S ITn}) satisfy analogues of the requirements (a), (b), and (c). The argument depends on the inequalities (Problem 1) for real x, IR(x)1 S tlxl2; 2 IR(x) + tx 21 S min{lxI ,ilxn·
173
VIlLI. A Central Limit Theorem for Martingale-Difference Arrays
For the analogue of (c): if (j > 0, no Iwnjl exceeds
m:XIPj_1ttZ~;j{j::;; an}::;; ttZ[(jZ + ~IPj-1~;j{l~njl > (j}]. Set (j small, then invoke the Lindeberg condition. For the analogue of (b): n
n
j= 1
j= 1
I IWnj I ::;; tt Z I IPj _ 1~;j{j ::;; an} n
= tt Z I {j ::;; an}vnj because {j ::;; an} is Cn,j_1-measurable j= 1
::;; t
Z
by definition of an'
For the analogue of (a), fix (j > 0: Z
Ijt1Wnj
+ tt J1 Vnj l
::;; IIPj-1IR(t~nj{j ::;; an}) + ttZ~;j{j ::;; an} + ttZ~;j{j > an} I j
::;; IIPj-1[ilt~nj{j::;; an}13{I~njl::;; (j} j
+ tZ~;j{j ::;;
::;; Lj i(jltl
3
anH I~nj I > (j}
+ ttZ~;j{j > an}]
vnj + t Z I IPj-1~;j{l~njl > (j} + ttZ{n > an} I Vnj· j j
The first sum can be made small with high probability, by an appropriate choice of (j; the last two terms converge in probability to zero. We could carry an along throughout the rest of the argument, but that would clutter up the notation. Instead, let us assume that (a), (b), and (c) hold. While we are at it, let us drop the n subscript for variables in the nth row of the array; all calculations take place within the nth row. Our task is to prove that and
(2)
DJ=
where 1k = 1(1 - r) and Zk = 1k exp(itSk )· The path leading from (a), (b), and (c) to the first assertion in (2) is wellworn (Chung 1968, Section 7.1). For complex e, Ilog(l - e)
+ el ::;; lelz if lel::;; t.
Apply the inequality to each rj . When maxj Irjl ::;; t,
IIIOg(1 - r) + ~ rjl : ; Ilrjlz J
J
J
::;; t 2 max Irjl J
by (b).
174
VIII. Martingales
It follows from (a), (c), and continuity of the exponential function that {T,,} converges in probability to exp(-!t 2 ). Each IT" I is bounded, because of (b):
(3)
I T" I :s;
9
(1
+ Ir j I)
:s; exp (
t I j I) r
:s; exp(t
2
).
Boundedness plus convergence in probability imply convergence in L 1. For the proof of the second assertion in (2), bound the errors that accrue during the calculation of conditional expectations layer-by-Iayer.
IPj _ 1 Zj = IP j - 1[Tjexp(itS j _ 1 + it()J = Tj exp(itS j - 1)IP j - 1 exp(it() = (1 - rj)Zj_1[1 + IPj - 1( it O + IP j- 1R(tOJ = (1 - r)Zj_1(1 + r) = Zj-1 - rJZj-1' Thus IIPZj - IPZj _1 1 :s; IPlrJZj-11 :s; exp(t 2 )IPlrJ I, because inequality (3) implies IZ j -11 = I Tj-11 :s; exp(t 2). Sum over j. IIPZn
-
:s; exp(t 2)IP
11
n
L
Irjl2 ~
j=l
°
as
n~
00,
4 Example. A sequence of real random variables X 0, Xl' .. , is called an autoregression of order one if Xn = eO x n - 1 + Un for some fixed eo. The innovations {un} are assumed independent and identically distributed with zero mean and finite variance (J2. The initial value X 0 is independent of the {un}. The least-squares estimator en minimizes n
L (Xj - exj _ 1)2. j= 1
Solving for en and standardizing, we get (5)
12 n / (e n
-
eo) =
12 [n- /
.±
Uj X j - 1J/[n-
J=l
1
.f XJ-1]'
J=l
With the help of Theorem 1 we can prove n 1 / 2 (e n
-
eo) ~ N(O,
1-
e6)
provided Ieo I < 1 and IPX6 < 00. This will follow from convergence results for the denominator and numerator in (5): n
n -1
L xJ-
1
~ (J2/(1 -
e6)
in probability,
j= 1
n
L n-1/2UjXj_1 ~ N(O, (J4/(1 j= 1
Start with the denominator.
-
e6))·
175
VIII. I. A Central Limit Theorem for Martingale-Difference Arrays
Square both sides of the defining equation of the autoregression, then sum over j. n
IX; = Iu; j~1
+ 28 0
n
IU j X j- 1 + 86 IX;-I'
j~1
j~1
j~1
Rearrange then divide through by n. n
(1 - 86)n- 1 IX;-1
(6)
n
=
n- 1 Iu;
j~1
+ 28 0 n- 1
j~1
+
n
IU j X j - 1
j~1
n- 1(X6 - X;).
On the right-hand side, the first term converges almost surely to (Jz, by the strong law of large numbers. The third term converges in L 1 to zero, because repeated application of the equality
IPX; = IPu; + 28 0 IPu n IPX n - 1 + 86IPX;-1 = (Jz + 86IPX;-I, yields IPX; = (Jz(1 + 86 + ... + 86 n - Z) + 86nIPX6 ---+ (Jz/(1 - 86). The n- 1 brings the limit down to zero. The middle term on the right-hand side of (6) converges in L Z to zero: n-ZIP('± Uj X j _ 1)Z J~
1 n
= n -Z "" L., IPUjZXZj-l
independence kills the cross-product terms
1
j~
n
= n- Z I (JZIPX;_1 j~
= O(n-
1
1
)
because {IPXJ} is convergent.
So much for the denominator. Write tffn for the (J-field generated by {Xo, Ul>"" un}. Abbreviate IP('ltff) to IPk). The variables {ujXj_d are martingale differences for {tffj}. Apply Theorem 1 to the sum in the numerator of (5). For condition (i): ( - 1 UjZXZ) Vnj = IP.i-In j-l =n - 1 (J ZXZj-l,
I
n
Vnj
j~1
= (JZn -1
I
X;-1
---+
(J4/(1 - 86) in probability.
j~1
The Lindeberg condition demands a more delicate argument if highermoment constraints on the innovations are to be avoided. n- 1
IPj-lU;X;-I{lujXj_ll > w 1/Z }
I j~
1
::; n- 1
n
I
j~1
= n- 1
n
+ {X;-1 > w 1/Z }]
X;-I IPuHui > w 1/Z } + n- 1
I
j~
IPj _ 1U;X;_I[{U; > Bnl/Z}
1
n
I
j~
(JZX;_I{X;_1 > 1
W
1/Z }.
176
VIII. Martingales
The first sum converges to zero in probability, because ui is integrable. The second sum converges in L 1 to zero because the sequence {X~} is uniformly integrable (Problem 2). 0
VIII. 2. Continuous Time Martingales A stochastic process {Z(t): o:s; t < oo} is said to be a martingale with respect to an increasing family of (T-fields {C t : :s; t < oo} if Z(t) is adapted to the (T-fields (that is, Z(t) is Ct-measurable) and IP(Z(s) ICt) = Z(t) whenever s > t. After some fiddling around with sets of measure zero it can usually be arranged that such a process has cadlag sample paths (Dellacherie and Meyer 1982, Section VU), in which case it may be studied as a random element of D[O, 00). Call Z an L 2-martingale if it has cadlag sample paths and IPZ(t)2 < 00 for each t. The behavior of an L 2-martingale is largely determined by the conditional variances of its increments. The conditional expectation of [Z(t + 6) - Z(t)J2 given Ct plays a role similar to that of the conditional variance vnj in Section 1. The most economical way to explain this uses some deeper results from the Strasbourg theory of stochastic processes. We could avoid the appeal to the deeper theory by building its special consequences into the martingale calculations for each particular application. That would always work for martingales that evolve by discrete jumps; the calculations would be similar to those in Section 1. The theory would be more selfcontained, but it would disguise the unifying concept of the conditional variance process. According to the Doob-Meyer decomposition (Theorem VII.l2 of Dellacherie and Meyer (1982), applied to the supermartingale _Z2), for each L 2- martingale Z, the process Z2 has a unique representation as a sum V + M of a martingale M and an increasing, predictable, conditional variance process V with V(O) = 0. Both M and V have cadlag sample paths. (Strictly speaking, for this decomposition we need the (T-fields {C t } to satisfy the "usual conditions": Co should contain all IP-negligible sets and each Ct should equal the intersection of the (T-fields Cs for s > t.) The adjective "predictable" has the technical meaning that Yew, t) is measurable with respect to the (T-field on Q ® [0, 00) generated by the class of all adapted, left-continuous processes. So V behaves something like a process with left-continuous sample paths; its paths can be predicted a tiny instant into the future. We will need the predictability property only in Lemma 11. If the martingale Z changes only by jumps ~ 1, ~2' ... occurring at fixed times tl < t2 < "', and if Ct = Ctk for tk :s; t < t k+ 1 , then V is just a sum of conditional variances:
°
177
VIII.2. Continuous Time Martingales
whenever tk :s; t < tk+ l' You can check directly that Z2 - V is a martingale and that there exists a sequence of left-continuous, adapted processes converging pointwise on n ® [0, 00) to V. The value of V at tk corresponds to what we would have written as V 1 + ... + V k in Section 1. You might take this as your guiding example for the rest of the section if you wish to avoid all appeals to the Strasbourg theory. The process V carries information about the conditional variances of the increments of Z. If s > t, (8)
IP([Z(s) - Z(t)J 2 1Ct) 2 = IP(Z(s) 1Ct) - Z(t)2 - 2Z(t)IP(Z(s) - Z(t) 1Ct)
= IP(V(s)ICt) + IP(M(s)ICt) - Vet) - M(t) = IP(V(s) - Vet) 1Ct)· For s very close to t, the predictability of V makes yes) - Vet) almost Ct-measurable; the last conditional expectation almost equals Yes) - Vet), in a sense that will receive a more rigorous meaning later. By means of a simple Tchebychev inequality we get from V a bound on the size of the increments of Z in precisely the form required by the stopping time argument of Lemma V.7. The maximal inequality provided by that lemma lies at the heart of any proof for convergence in distribution in spaces of cadlag functions. 9 Lemma. Let {Z(t): 0 :s; t :s; b} be an L 2-martingale with conditional variance process V. 1f,Jor every t, IP(V(b) - Vet) 1Ct) :s; J2/12
almost surely,
then IP{su p IZ(t) - Z(O) 1 > J} :s; 3IP{ IZ(b) - Z(O) 1 > H}· t-s,b
With no loss of generality we may assume Z(O) = O. Write IP t(·) for expectation conditional on Ct • Lemma V.7 invites us to check that
PROOF.
IPt{IZ(b) - Z(t) 1 :s; !IZ(t)l} ~
t
on
{IZ(t)1 > J}.
We shall do this by bounding the conditional probability from below by (10)
which is greater than ~ - 4J-2(J2/12) on the set {IZ(t)1 > J}. Start from the inequality 1Z(b) 1 :s; IZ(b) - Z(t) 1 + IZ(t)1 :s; ~IZ(t)1 {IZ(b) - Z(t) 1 :s; !IZ(t)1} + 3IZ(b) - Z(t) 1{IZ(b) - Z(t) 1 > !IZ(t)I}.
178
VIII. Martingales
Keeping in mind that the absolute value of a martingale is a submartingale, take conditional expectations given tit.
IZ(t)1
~
IPtl Z(b) I
~ ~IZ(t)IIPt{IZ(b) - Z(t) I ~
tIZ(t)l}
+ 3IP t IZ(b)
- Z(t) I{IZ(b) - Z(t) I > ~ ~IZ(t)IIPt{IZ(b) - Z(t) I ~ tIZ(t)[} + 6I Z (t)I- I IP t IZ(b) - Z(tW· On
{I Z(t) I > <5}
tIZ(t)l}
we may divide through by ~ IZ(t) I to get the bound (10).
o
Martingales with cadlag sample paths define random elements of the space D[O, (0) under its projection (J-field. As we shall be concerned only with convergence in distribution to limit processes having continuous sample paths, we equip D[O, (0) with its metric for uniform convergence on compacta (Section V.5). The main theorem will give conditions for a sequence of L 2- martingales to converge in distribution to a stretched-out brownian motion B H , the process constructed from brownian motion by applying a continuous, strictly increasing transformation HO to the time scale: BH(t) = B(H(t)). This new gaussian process is an L 2 -martingale with conditional variance process H(t), provided H(O) = 0. One hypothesis of the theorem will be pointwise convergence of the conditional variance processes to H. It is analogous to the hypothesis in Theorem 1 that the sum of conditional variances converges to one. As in the proof of that theorem, we shall use the hypothesis to introduce stopping times for the martingales. For fullest generality we must draw upon the Strasbourg theory for the properties of predictable stopping times; for the special case of martingales with conditional variance processes as in (7), the argument from Theorem 1 would suffice.
11 Lemma. Let {X n} be a sequence of L 2 -martingales whose conditional variance processes converge pointwise to a fixed, continuous, increasing function: v,,(t) --+ H(t) in probability, for each fixed t. Then there exists a sequence of stopping times {(In} and constants {6 n}, with (In --+ 00 in probability and 6 n 1 0, such that SUPt IVn(t !\ (J n) - H(t !\ (J n) I ~ 6 n almost surely. PROOF.
First fix an 6 > 0, and define a stopping time Tn
= inf{t > 0: I v,,(t) - H(t) I > 6}.
The hypothesis implies that Tn --+ 00 in probability: ifO = So < SI < ... < Sk are chosen so that H(sJ - H(Si-l) < for each i, then (by monotonicity of v" and H)
t6
IP{ Tn < Sk} ~ IP{ I v,,(sJ - H(sJ I >
h
for some i}
--+
0.
179
VIII.2. Continuous Time Martingales
From the definition of 'n'
I v,,(t) - H(t) I ~
(12)
8
for
t < 'n'
'n
Only the time t = might cause trouble; Vn might have a jump there. That is where predictability comes to the rescue. There exists an increasing sequence of stopping times {'nj} with 'nj < and 'nj i almost surely (Dellacherie and Meyer 1978, IV.69, IY.77), because predictability allows us to peer just a little distance into the future for v". Replace by 'n,j(n) for somej(n) such that 'n,j(n) --+ 00 in probability. This lops off the troublesome point t = 'n in (12); the inequality holds for every t in the range [0, 'n, j(n)J. If the argument works for fixed 8 it must also work for a sequence {8 n } decreasing slowly enough. Write O'n(8) for the stopping time 'n,j(n) just identified. There exist integers n(1) < n(2) < ... such that
'n
'n
'n
IP{O'n(k- 1 ) ~ k} < k- 1 for
n ~ n(k).
o Stopping time tricks like the one used in this proof pop up all over the place in martingale limit theory. Sample path properties that hold with probability tending to one can often be made to hold with probability one by enforcing an appropriate stopping rule. Something must be added to the convergence of conditional variance processes. Otherwise we could specialize the result to processes with independent increments, obtaining as a by-product the Central Limit Theorem for sums of independent random variables without having to impose anything like a Lindeberg condition. The extra something takes the form of a constraint on the maximum jump in the sample path. Define the jump functional J T on D[O, 00) by:
JT(x) = max{lx(s) - x(s-)I:O
~
s
~
T}.
It is both continuous and measurable (Problem 4). If Xn ~ BH then certainly JT(X n) ~ JT(B H) = 0; convergence in probability of {JT(X n)} to zero is a necessary condition. The theorem assumes just a little bit more to get a sufficient condition.
13 Theorem. Let {Xn} be a sequence of L 2-martingales with conditional variance processes {v,,}. Let H be a continuous, increasing function on [0, 00) with H(O) = 0. SuffiCient conditions for convergence in distribution of {Xn}, as random elements of D[O, 00), to the stretched-out brownian motion BH are:
°
(i) X n(O) --+ in probability; (ii) v,,(t) --+ H(t) in probability,for each fixed t; (iii) IP J iX n)2 --+ for each fixed k, as n --+ 00.
°
180
VIII. Martingales
PROOF. By virtue of Theorem V.23, we have only to prove convergence in distribution for the truncations of the processes to each compact interval [0, T]. The argument for the typical case T = 1 will suffice. According to Theorem V.3 we shall need to establish (a) Fidi Convergence: the fidis of the {Xn} converge to the fidis of EH (b) The Grid Condition: to each e > 0 and b > 0 there corresponds a grid o = to < t 1 < ... < tm = 1 such that limsup n
IP{m~x J
sup IXn(t) - XnC t ) 1 >
b} < e
[tj,tj+l)
Theorem 1 will take care of (a); Lemma 9, applied to the Xn processes stopped appropriately, will take care of (b). By Lemma 11 there exist stopping times {O"n} and constants {en}, with O"n ~ CX) in probability and en ! 0, such that 1 v,,(t /\ O"n) - H(t /\ O"n) 1 :::; en almost surely. We may assume that X n has at most one jump of size greater than en up to time O"n' Formally, we would replace {en} by a more slowly decreasing sequence such that IP {J k(n)(X n) > en} ~ 0
as
n~
CX)
for some slowly diverging sequence {ken)}. Such sequences exist because of (iii). Then we would replace O"n by
inf{t :::; O"n: IXn(t) - Xn(t-)I > en}· These modifications would not disturb the other properties of {O"n} and {en}. The stopped martingale X:(t) = Xn(t /\ O"n) has conditional variance process V:O = Vn(· /\ O"n). It enjoys (i), (ii), and (iii) in strengthened form: (i)' X:(O) = Xn(O) ~ 0 in probability; (ii), 1V:(t) - H(t /\ O"n)1 :::; en for all t; (iii)' X: has at most one jump of size > en and IP J 1(X:)2 ~ O.
X:
These will make it easy to prove that ~ EH' The required convergence of the truncation of X n to [0, 1J will thep follow because
IP{Xn(t) = X:(t) for 0:::; t:::; I}
~
IP{O"n
~
I}
~
l.
Simplify the notation by dropping the star.
Fidi Convergence
Let us prove only that Xn(1) ~ N(O, H(1». Problem 6 extends the result to higher-dimensional fidis. Because of (i)" we can do this by breaking Xn(1) - Xn(O) into a sum of martingale differences that satisfy the conditions of Theorem l. Focus for the moment on a fixed Xn by setting Z(t) = XnCt) - XnCO) and writing V instead of v" for its conditional variance process. Break Z(1) into
181
VIII.2. Continuous Time Martingales
a sum of increments Z(r) - Z(r j _ 1), with stopping times defined inductively by r j + 1 = inf{t > T j : IZ(t) - Z(r)
1
°= ~
ro < r 1 < ... a sequence of
cn} /\ 1 /\ (rj
+ bn ).
Choose {b n } so that 1H(t + bn ) - H(t) 1 ::; Cn for every t in [0, 1]. Denote expectations given cfftj by IPk); write I1Jfor the incrementf(T) - f(rj-1) of any function f between successive stopping times. Then 00
L I1jZ,
Z(l) =
j= 1
Along any particular sample path of Z all except finitely many of these increments equal zero; there is no problem with convergence of the sum. Check the conditions of Theorem 1 for the martingale differences {l1 j Z}. Strictly speaking the theorem applies only to triangular arrays with a finite number of variables in each row, but there may be infinitely many I1jZ increments. We would need to apply the theorem to a finite sum of I1jZ, for 1 ::; j ::; j(n), with j(n) chosen so that both 00
00
L I1 (Z) j(n)
I
and
j
j(n)
IPj _ 1 (l1j Z)
converge to zero in probability. By property (8) of the conditional variance process,
I
IPj _ 1 (l1j Z)2
=
I
lFj_1(l1 j V).
j
Informally speaking, predictability of V almost makes I1j V measurable with respect to cff tj _ 1 ; the last sum almost equals Ij I1jV, which we know will converge in probability to H(l) as n --+ 00. Formally, Lj (11 j V - IPj - 111 j V) is a sum of martingale differences with zero mean and variance less than
L IP(l1
j
V)2
because the cross-product terms vanish
j
::; L IP[(2c + I1jHC n
/\ O"n))l1j V]
by (ii)'
j
::; 3c n IP( ::; 3cn(c n
°
~ 11 V) j
by the choice of
{b
n}
+ H(l))
--+ as n --+ 00. That takes care of condition (i) of Theorem l. For the Lindeberg condition it suffices to check the stronger L 1 convergence.
IPIIPj-1(l1jZf{ll1jZI > c} = IPL(l1jZ)2{ll1jZI > c} j
j
182
VIII. Martingales
The inequality follows, by the definition of T j , from: IZ(T) - Z(T j _ 1 )1::; IZ(T) - Z(Tj-)1
::; Ijump at
+
IZ(Tj-) - Z(T j - 1 )1
+ en
Tjl
At most one increment !1jZ can exceed 2en in absolute value, and that happens only if Z has its one jump greater than en at T j . An appeal to the second part of (iii)' completes the proof of the fidi convergence. The Grid Condition
Choose 0 = to < ... < tm = 1 so that H(t j +1) - H(t) ::; 15 2 /24 for each j. For n large enough to make 2e n ::; 15 2 /24, the strengthened condition (ii)' implies IP(VnCtj+l) - v,,(t)ICt )::; 15 2 /12
(14)
almost surely
if tj::; t < t j +l ' Invoke Lemma 9. (15)
limsup IP{ n
m~x J
sup IXn(t) - Xn(t) 1 > b} [tj,tj+t>
m-I
::; L
j=O
limsup3IP{I X nC t j +l) - Xn(t) 1 ;;::: n
rn-I
::; L
j=O
3IP{ 1BH(t j + 1) - BH(t) 1 ;;:::
H}
H}
by fidi convergence.
rn-I
=
L
j=O
3IP{IN(O, H(t j +1 )
-
H(t)) 1 ;;::: tb}
rn-I
::; 4815- 4
L
j=O
[H(t j +1 )
-
H(t)J 2 IPIN(O,
lW
::; 4815- 4 max[H(t j +1) - H(t)JH(l)IP IN(O,
lW
j
which is less than e if the grid points are close enough together.
0
VII!'3. Estimation from Censored Data The empirical distribution function based on an independent sample ~ l ' ... , ~n from P is a natural estimator for the distribution function of P. How should one modify it when the observations are subject to censorship? One possibility, the Kaplan-Meier estimator, can be analyzed by the martingale methods from the previous section. What follows is a heuristic account. The notes to the chapter will point you towards more rigorous treatments, which draw on results from the theory of stochastic integration.
183
VII!'3. Estimation from Censored Data
Consider the simplest model for censorship. Independent variables cn' drawn from a censoring distribution C, cut short the natural lifetime (i; we observe the value Yi = (i 1\ Ci . Each Yi has distribution Q, where Q(s, (0) = P(s, 00 )c(s, (0). In addition, we can tell whether the Yi represents natural death, (i < Ci' or a case of death by censorship, (i ;;::: Ci' Cl"'"
---. -0----0 •• Yl Y2
• • • 0)-------. Y9 co
To construct the Kaplan-Meier empirical measure K n , start from the usual empirical measure Qn for the observations YI' ... ,Yn' Working from the left-hand end, distribute successively all the mass from each censored point equally amongst all the {Yi} lying to its right. In the situation pictured, Y I keeps its mass ~; then Y2 surrenders mass x ~ to each of its seven successors Y3, ... , Y9; then Y3 surrenders i x (~ + x ~) to each of Y4' ... ,Y9; and so on. At the last point, Y9' there are no more {y;} to inherit its mass, so dump it all down on a fictitious super-survivor out at + 00. If Y9 had not been censored, it would have kept all its inherited mass. In any case, Kn will distribute its total mass of one amongst the naturally deceased {Yi}, with maybe a little bit on + 00. Notice that Kn[O, tJ ~ Qn[O, tJ for each t. Make the analysis as simple as possible by assuming that both P and C are continuous distributions living on [0, (0), with neither concentrated on a finite interval. Write Ct for the O"-field corresponding to everything we learn up to timet about which (i have died or been censored. Write IPl·) for IP( ·1 Ct)· Calculate (to first-order terms) the conditional distribution of the increment I1K n = Kit, t + hJ given Ct> for tiny positive h. From Ct we learn the value Qn [0, t]. Define m to be nQn [0, t]. The remaining n - m observations in (t, (0) are generated by choosing each (i from the conditional distribution P(·I ( > t), then censoring it by a Ci chosen from c(·1 C > t), Write I1P for P(t, t + hJ and I1K n for Kn(t, t + h]. To first order, each of the n - m observations has conditional probability I1P/P(t, (0) of registering a natural death during the interval (t, t + h]. A single such observation would receive a fraction (n - m)-l of the Kn measure for (t, 00]. Thus, to first order,
+ +
IPt l1K n = (n - m)-IKit, ooJ(n - m)I1P/P(t, (0).
This suggests that
+ lower order terms, which would lead us to believe that log Kit, 00 J - log P(t, (0) is a continIPtI1Kn/Kit, ooJ = I1P/P(t, (0)
uous-time martingale for each n. An attempt to add rigor to the first-order analysis would reveal a few illegal divisions by zero. There is a positive probability that either n - m or Kit, ooJ could equal zero. A suitable stopping time can save us from
184
VIII. Marti ngales
embarrassment. Let {lXn} be a sequence of positive numbers converging to zero. Make sure that nlXn is an integer. Define Pn as the first t for which QnCt, 00) equals IXn' Then certainly
Kn(t /\ Pn, ooJ 2:: Qn(t /\ Pn'
00)
2::
IXn
>
° for all t.
The first-order analysis could be made rigorous enough to show that the process X net) = log KnCt /\ Pn,
00
J-
log pet /\ Pn,
00)
is a continuous-time martingale for each n.: On the set {Pn > t}, the increment in v", the conditional variance process of X n , would be
IPtC.1Xn)Z = (n - m)-Z(n - m).1P/P(t,
00)
+ smaller order terms
On {Pn :s:; t} the increment would be zero. Recover conditional variances (see Problem 5).
v"
as a limit of sums of
v,,(t) = n- 1 f{o:s:; s:S:; t /\ Pn}Qn(s, oo)-lp(S, oo)-lP(ds). By definition of Pn,
v,,(t):s:; (nlXn)-l
f~ pes, oo)-lP(ds).
Thus {v,,} converges in probability to zero uniformly over compact sets. Apply Lemma 9. For b fixed and n large enough,
IP{~~~ IXn(t) I > <5} :s:; 3IP{ IXnCb) I > tb} :s:; 12<5- z IPX n(b)Z = 12<5- z IPv,,(b) ~o.
That is, for each fixed b, sup Ilog Kn(t /\ Pn'
00
J-
log pet /\ Pn,
t :o::b
00) I ~
°
in probability.
Because both Kn and P are probability measures, and because Pn probability, it follows that (16)
sup IKn[O, tJ - P[O, tJ I ~
°
~ 00
in
in probability.
The Kaplan - Meier measure estimates P consistently, in the sense of uniform convergence of distribution functions. Now consider the normalized martingale n1/ZXn(t). It has conditional variance process nv", which converges in probability to the continuous function
185
Notes
That suggests {nl/2Xn} converges in distribution to a brownian motion stretched out by H. Theorem 13 proves it. We already have the first two requirements ofthe theorem holding. What about the maximum jump of n l/2 Xn on an interval [0, tJ? It can only occur at one of the {yJ that died naturally. In the worst case, where all except the largest Yi in [0, tJ are censored, the measure Kn puts mass (nQn(t A Pn' 00)) - 1 at that largest Yi. The corresponding jump J in nl/2Xn cannot exceed n l/2 (na. n) - la.;; 1, which converges to zero provided a. n ~ n -1/4. That is more than enough to force IPJ 2 --+ as n --+ 00. Thus nl/2Xn ~ BH • The effect of the stopping wears off as n --+ 00, because Pn --+ 00 in probability. So we get from the convergence result for {Xn} that n 1/2 10g[Kn(t, ooJ/P(t, oo)J ~ BH
°
as random elements of D[O, 00). Rewrite the left-hand side as n 1/2 10g[1 + (P[O, tJ - Kn[O, tJ)/P(t, oo)J = - n 1/2 (Kn[0, tJ - P[O, tJ)(1
+ op(l))/P(t, 00)
with the ope!) converging in probability to zero uniformly over compact intervals, by virtue of (16). Deduce that n 1/2 (Kn[0, tJ - P[O, tJ)/P(t, 00) ~ B , H
or, equivalently, n 1/2 (Kn[0, tJ - P[O, tJ) ~ pet, oo)BH ·
The covariances of the gaussian process P(·, oo)B H( .) depend on both P and C. NOTES
Martingales have been one of the hottest topics on the weak convergence market for some years now, partly because of a boom in stochastic integral stocks. For discrete-time martingales key papers are Brown (1971), McLeish (1974), and Aldous (1978). Both McLeish and Aldous obtained results sharper than those in Sections 1 and 2. Hall and Heyde (1980, Section 6.4) gave a much fancier version of Example 4. The literature on continuous time martingales has grown rapidly; the Shiryayev (1981) survey, Metivier (1982), the two-volume monograph by Liptser and Shiryayev (1977, 1978), and Gihman and Skorohod (1979), are accessible. Meyer (1966), and Dellacherie and Meyer (1978, 1982), make harder reading. Dellacherie (1972) contains much about subtle a-fields. What I called the conditional variance process usually gets written as (Z, Z). The more widely definable quadratic variation process [Z, ZJ has taken over the role I gave to (Z, Z) in Section 2. In the discrete-time theory, [Z, ZJ would be the sum of squared increments; (Z, Z) would be the sum of conditional variances.
186
VIII. Martingales
Martingales and stochastic integrals simplify the theory for estimation of distribution functions from censored data. Gill (1980) covered this in some detail; Jacobsen (1982) explained the essential prerequisites from the Strasbourg theory of processes. Bremaud (1981) gave applications to point processes. The martingale proof of the central limit theorem for KaplanMeier estimators makes an interesting comparison with a more traditional weak convergence approach of Breslow and Crowley (1974).
PROBLEMS
[1] For y real, define Hn(Y) = eiy - 1 - iy - ... - (iy)n/n!. Prove that IHn(Y)1 :::;; lyln+I/(n + 1)!' [Proceed inductively, using iJ~Hn(s)ds = Hn+l(t) for t > 0.] Deduce that le iy - 1 - iy + fy21 :::;; min{h 2 + fy2, il YI3}. [2] Prove uniform integrability of the squared auto regressive process {XI} from Example 4. Break Uj into asum Vj + w j , wherevj = Uj{ Iujl :::;; M} - IPuj { Iujl :::;; M} for some large truncation level M. Define y"
=
Zn
=
+ BOvn- 1 + ... + Ba-lVI, Wn + BOw n- 1 + ... + BO-Iw l . vn
Show that Xn = Y" + Zn + BoXo. The sequence {Y;} is uniformly bounded. Because IPZ; = IPw; + .,. + B6n-2IPwI, the M can be chosen to make IPZ; < e for every n. [A similar argument would work for {uJ a sequence of martingale differences. Solution provided by Kai Fun Yu.] [3] For the uniform empirical process Un prove that {U.(t)/(l - t): 0:::;; t < 1} is a martingale with respect to an appropriate family of IT-fields. Bound IP{suP,:5bIUn(t)1 > b} for small enough b, then deduce the Empirical Central Limit Theorem. Generalize to En' [Over [a, b] take tff, as the IT-field determined by the locations of sample points in [a, t], for a :::;; t :::;; b.] [4] The jump functional is projection measurable; JT(x) is the limit of maxi Ix(t) - X(t i _ l ) I as the grid 0 = to < ... < tm = T is refined down to a countable dense subset of [0, T]. [5] Let Z be an L 2 martingale on [0, 1] with conditional variance process V. If V has continuous sample paths, adapt the arguments in the Fidi Convergence part of the proof of Theorem 13 to show that
L IP([Z(t i) -
Z(t i _ I )]2I tff';-I)
--+
V(l)
in probability,
i
as the maximum spacing in the grid 0 = to < ... < tm = 1 tends to zero. [The intuitive interpretation of V as a limit of a sum of conditional variances of increments is close to the mark.] [6] Suppose that martingales {Xn} satisfy the conditions of Theorem 13. Prove that their fidis converge to the fidis of BH . [If 0:::;; tl < ... < tk :::;; 1, then y"(t) = L~= I Q(jX net 1\ t) is a martingale with Y.(I) = Lj Q(jXn(t).]
187
Problems
[7J Using the notation from Section 3, show that Z.(t)
=
Kn[O, t
A
PnJ - J{o:s; s:s; t
A
Pn}Kn(s
A
Pn, ooJP(s, oo)-lp(ds)
is a martingale. Deduce from the consistency result for Kn that {n 1 / 2 Zn} converges in distribution to a stretched-out gaussian process. Reprove the central limit theorem for the Kaplan-Meier estimator. [8J Let {Xn} be a sequence of L 2 -martingales with conditional variance processes {Vn }. Suppose the fidis converge to the fidis of a stretched out brownian motion BH , with H continuous. If v,,(t) ...... H(t) in probability for each fixed t, then X n --. BH . [If Pn is a stopping time taking only finitely many different values, then lP[Xn(Pn
+ (in)
- X n(Pn)J2
=
lP[v,,(Pn
+ (in) -
v,,(Pn)].
Check Aldous's condition (VU3) for suitably stopped processes. Use Lemma I1.J
APPENDIX A
Stochastic-Order Symbols
Mann and Wald (1943) defined the symbols OpO and op('), the stochastic analogues of 0(·) and 0('), as a way of avoiding many of the messy details that bedevil asymptotic calculations in probability theory. Let {Xn} and {y"} be sequences of random vectors. Write Xn = 0iY,,) to mean: for each 8 > 0 there exists a real number M such that IP{IXnl ~ MIY"I} < 8 if n is large enough. Write Xn = o/Y,,) to mean: IP{ IXnl ~ 81 y"1} ~ 0 for each 8 > O. Of course Xn and Y" must be defined on the same probability space. Typically {y"} will be non-random. For example: Xn = oil) is another way of writing Xn ~ 0 in probability; a sequence of order 0/1) is said to be stochastically bounded; statistical estimators commonly come within o p(n - 1/2) of the parameters they estimate. With the 0/·), oi') notation, Problem IIU5 (the delta method) has a quick solution. Differentiability of the map H means H(x)
= H(x o) + L(x - x o) + o(x - x o) near x o ,
where L is a fixed s x k matrix. If n 1/2 (Xn - x o) ~ Z then H(Xn)
= H(x o) + L(Xn - x o) + op(Xn - x o) = H(x o) + L(Xn - x o) + op(n- 1/2 ).
Thus n I/2 (H(Xn) - H(x o))
= Ln 1/2 (Xn - x o) + op(1)
~ LZ.
A few hidden assertions here need justification. For example: n 1/2 (Xn - x o) ~ Z implies 0(Op(n- 1/2 )) = 0/n- 1/2 );
if Zn
~
Z
then
Xn -
Xo
LZ n + oil) ~ LZ.
These conceal the asymptotic details of the proof.
= Op(n- 1/2 );
190
A. Stochastic-Order Symbols
For comparison, here is part of the proof of Theorem VII.5 written without the benefit of the notation. Remember that f(', t) = f(', 0) + t'fl(·) + Itlr(·, t), Fn(t)
=
Pnf(·, t),
F(t) = Pf(·, t) = F(O)
+ iltl 2 + het),
where for each a > 0 there exists ab> 0 such that Ih(t)1 ~
c:ltl 2 if
It I ~ b.
By assumption, Fn(Tn) = Xn
(1)
+ inf Fn(t), t
where nXn - 0 in probability as n - 00. Also we know that n l /2(Fn(Tn) - F(Tn» = Enf(', 0) + T~Enfl + ITnl 1';, or FiTn) = F(Tn)
(2)
+ n- I/2E nf(·,
0)
+ n-I/2T~Enfl + n- I / 2 IT nl Y",
where Y" = Enr(., Tn) - 0 in probability. From (1) and (2), Xn ~ Fn(Tn) - Fn(O) = F(Tn) + n-I/2T~Enfl + n-I/2ITnl Y" - F(O) = !ITnI2 + !h(Tn) + n-I/2T~Enfl + n- I /2ITnIYn-
Take absolute values. Because Xn ~ 0, nXn ~ !nlTnl 2 - !n Ih(Tn) I - n l /2 lTnll E nfll - nl /2lTnll Y"I ~ !n(l - a)ITn12 - nl/2ITnl(IEnfll + I y"1) if ITnl ~ b ~ t(1 - c:)(n l /2 ITnl - W n)2 - t(1 - a) W;,
+ 11';,1). For fixed M deduce that JP{(nl/2ITnl- l-Y,,)2 > 2M} ~ JP{ITnl ~ b} + JP{2(1 - a)-lnX. > M}
where
vv" =
(1 - c:)-I(IEnfll
+ JP{W; >
M}.
The first and second terms converge to zero. If M is large enough, the third term is eventually less than any fixed rx because. " well, because vv" is of order Op(l). The mess gets too great if we try to avoid the stochastic-order symbols any longer. If you need further convincing read the beautiful expository article by Chernoff (1956).
APPENDIX B
Exponential Inequalities
The central limit theorem leads one to expect sums of independent random variables to behave as if they were normally distributed; tail probabilities for standardized sums can be approximated by normal tail probabilities. For the limit theorems proved in this book, we need upper bounds rather than approximations. A few such bounds are collected together here. The tails of the normal distribution decay rapidly. For I] > 0,
! _ !3) exp( -11]2) < IP{N(O, 1) ;::::: I]} < ! exp( _try2) (I] I] Fn I] Fn The important factor is the exp( -11]2). We need a similar upper bound for the tail probabilities of a sum of independent random variables Yl, ... , Y". Set S = Y1 + ... + Y". For each t > 0, n
(1)
IP{S ;::::: I]} :::;; exp( -l]t)IP exp(tS) = exp( - I]t)
TI IPexp(tl';). i= 1
The trick will be to find a t that makes the last product small. For the normal distribution it is easy to find the best t directly: IP{N(O, 1) ;::::: I]} :::;; inf exp(1t2 - I]t) = exp( _t1J2). t>O
For other distributions we have to work harder. We must maneuver the moment generating function of 1'; into a tractable form that gives us some clue about which value of t to choose. 2 Hoeffding's Inequality. Let Y1 , Y2 , ... , Y" be independent random variables with zero means and bounded ranges: a i :::;; 1'; :::;; b i • For each I] > 0,
IP{Yl
+ ... +
Y,,;::::: I]}:::;; exp [ -21]2/Jl(b i
-
a i )2J
192
B. Exponential Inequalities
Use convexity to bound the moment generating function of Yi. Drop the subscript i temporarily.
PROOF.
etY :::;; eta(b - Y)/(b - a)
+ etb(y - a)/(b - a).
Take expectations, remembering that IPY
= O.
IPe tY :::;; eta bleb - a) - etb a/Cb - a). Set IX
= I - [3 = -a/Cb - a) and u = t(b - a). Note: IX > obecause a < 0 < b. log IPe tY :::;; 10g([3e - au
+ IXe
PU
)
= - IXU + 10g([3 + IXe
U
).
Write L(u) for this function of u. Differentiate twice.
L'(u) = -IX + IXe !([3 + IXe U
U )
= -IX + IX/(IX + [3e-
U ),
u
L"(u) = IX[3e- /(IX + [3e- )2 = [IX/(IX + [3e- )][[3e- /(IX + [3e- U ) ] u
U
U
The inequality is a special case of: x(I - x) :::;; Taylor's theorem.
i
:::;;
i.
for 0 :::;; x :::;; 1. Expand by
+ uL'(O) + !J2L"(u*) :::;; 1u 2i = kt 2(b - a)2. Apply the inequality to each Yi, then use (1) L(u) = L(O)
IP{Y1 + .. , + Y,. ;:::.: I]} :::;; exp[ -I]t + kt 2
n
L
(b; - aY].
;=1
Set t =
o
41]/L; (b; - a;)2 to minimize the quadratic.
3 Corollary. Apply the same argument to {- Yi} then combine with the inequality for {Yi} to get a two-sided bound under the same conditions:
IP{IY1 +
... +
Y,.I;:::.: I]}:::;; 2ex p [ -21]2/t/b; - aY].
In one special case the proof can be shortened slightly. If two values, ± a;, each with probability 1, then JPe tY , = t[e ta , + e- ta ,] =
00
L (a;t)2k/(2k)!
Yi
0
takes only
:::;; exp(taft 2).
k=O
The rest of the proof is the same as before. We only need the Hoeffding Inequality for this special case. ••• , Y,. be independent random variables with zero means and bounded ranges: I Yil :::;; M. Write (If for the variance of Yi. Suppose V;:::.: (Ii + .,. + (I;. Thenfor each I] > 0, IP{IY1 + ... + Y,.I > I]}:::;; 2exp[-11] 2V- 1 B(MI]V- 1 )],
4 Bennett's Inequality. Let Y1 ,
where B(A) = 2r2[(1
+ A) log(1 + A) - A] for A>
O.
193
B. Exponential Inequalities
PROOF. It suffices to establish the corresponding one-sided inequality. The two-sided inequality will then follow by combining it with the companion inequality for { - Y;}. Bound the moment generating function of Y;. Drop the subscript i temporarily. 00
IPe t ¥
= 1 + tIPY +
I
(t kjk!)IP(y 2 y k- 2)
k=2 00
~
1+
I
(t kjk!)(J2Mk-2
k=2 = 1 ~
+ (J2g(t)
where get) = (e tM
-
1 - tM)jM2
exp[(J2g(t)].
From (1) deduce IP{S 2:: I]} ~ exp[Vg(t) - I]t]. Differentiate to find the minimizing value, t = M- 1 log(1 + M'1V-1), which is positive.
o
The function B(·) is well-behaved: continuous, decreasing, and B(O + ) = 1. When Ii is large, B(Ii) ::::; 21i -1 log Ii in the sense that the ratio tends to one as Ii ~ 00; the Bennett Inequality does not give a true exponential bound for I] large compared to V j M. For smaller '1 it comes very close to the bound for normal tail probabilities. Problem 2 shows that B(Ii) 2:: (1 + tli) - 1 for allli > O. If we replace B(.) by this lower bound we get IP{ ISI 2:: I]} ~ 2 exp[-1'12j(V
+ tMI])],
which is known as Bernstein's inequality. NOTES Feller (1968, Chapter VII) analyzed the tail probabilities of binomial and normal distributions-sharp results obtained by elementary methods. Bennett (1962) and Hoeffding (1963) derived and compared a number of inequalities on tail probabilities for sums of independent random variables. Dudley (1984) noted the simpler derivation of Hoeffding's Inequality when r; takes only values ± a i • Bernstein's inequality apparently dates from the 1920's; it appeared as Problem X.14 in Uspensky's (1937) book. PROBLEMS [lJ For independent N(O, I)-distributed random variables Y1 , Yz , ... , show that (max j:5n 1';)/(2 log n)l/Z converges in probability to one. [Write M n for the maximum. Show IP{M n ~ (21'/ log n)I/2} = [1 - IP{N(O, 1) > (21'/ log n)l/Z}]n, then use the exponential inequalities for normal tails.]
194
B. Exponential Inequalities
[2] For the function BO appearing in Bennett's Inequality prove that
(1
+ t,1)B(,1)
;;::: 1 for all
A > O.
[Apply I'Hopital's rule twice to reduce the left-hand side to
+ t,1*)(1 + ,1*)-1 + tlog(1 + ,1*) for some ,1* less than A. Then use 10g(1 + ,1*);;::: ,1*/(1 + ,1*).] (1
[3] If a random variable Y has zero mean, finite variance u Z, and is bounded above by a constant M, then for t > 0, IPe tY ::;; (aZe tM
+ MZe-tu2/M)/(uZ + MZ).
[Subject to the constraints IPY = 0 and IPY z ::;; u Z, the value of IPe tY is maximized when IPy concentrates on the two values M and -uz/M.] To prove this let rf>(y) be the quadratic e- tu2 /MC-\M _ y)[1 + (C- 1 + t)(y + aZ/M)] + etMC-Z(y + uZ/M)z, where C = M rf> satisfies
+ U Z/M.
Check that the coefficient of yZ is strictly positive and that
Show that rf>(y) ;;::: etY for y ::;; M, with equality at y = M and y = - a 2/M. [The function hey) = e-tYrf>(y) has a local minimum of 1 at -uz/M. Also heM) = 1. If h(y*) were equal to 1 for some y* in the interval ( - U Z/M, y*), the quadratic etYh'(y) would have three real roots: one at -a 2 /M, one in the interval (-a 2 /M, y*), and one in the interval (y*, M).] The distribution IPy concentrated at M and -az/M achieves equality in IPe tY ::;; IPrf>(Y). Bennett (1962, page 42).] [4] For the one-sided form of Bennett's Inequality one needs only zero means and 1'; ::;; M for each i. Reexpress the inequality from the previous problem as
IPe tY ::;; exp[tM wheref(y)
=
1 - y-l
+ 10gf(1 + uZ/Mz)],
+ y-l e -tMy. Prove that dZ/dyZlogf(y) equals
_2y-3 e -tMY[etMy - 1 - tMy - ¥tMy)Z]/f(y) - [f'(y)/f(y)]Z which is less than zero for y ;;::: 1. Deduce from a Taylor expansion to quadratic terms that
logf(y) ::;; 10gf(1)
+ (y
- 1)f'(1)!f(1)
for
y;;::: 1,
whence IPe ty ::;; exp[aZ(etM - 1 - tM)/M2]. Complete the argument as before. [Hoeffding (1963, page 24).]
APPENDIX C
Measurability
The defining properties of a-fields ensure that the usuaLcountable operations in probability theory-countable unions and intersections, pointwise limits of sequences, and the like-cause no measurability difficulties. In Chapters II and VII, however, we needed to take suprema over uncountable families of measurable functions. The possibility of a non-measurable supremum was brushed aside by an assurance that a regularity condition, dubbed permissibility, would take care of everything. This appendix will supply the missing details. The discussion will take as axiomatic certain properties of analytic sets. A complete treatment may be found in Sections IILl to III.20, IIL27 to III.33, and IIL44 to 45, of Dellacherie and Meyer (1978). Square brackets containing the initials DM followed by a number will point to the section ofthat book where you can find the justification for any unproved assertions. Suppose M is a set equipped with a a-field A. The analytic (A-analytic in DM terminology) subsets of M form a class slightly larger than A. Denote it by d(M). If A is complete for some probability measure 11, (that is, A contains all the sets of zero 11 measure) then d(M) = A [DM 33]. For example, the analytic subsets generated by ~[O, 1J contain that a-field properly, but the a-field of lebesgue measurable subsets of [0, 1J coincides with its analytic sets. You see from this example that we should be writing d(A) rather than ,S#(M). The ambiguity is not serious when M is equipped with only one a-field. For metric spaces, we will always choose A to be the borel a-field; for product spaces, it will always be the product a-field. We considered empirical processes indexed by a class of functions. Formally, ~ 1, ~2' ... were measurable maps from a probability space (0, $, JP) into a set S equipped with a a-field Y'. A class ff of 9"/@i(IR)-measurable,
196
C. Measurability
real-valued functions on S was given. The empirical measure Pn attached to each f in fF the real number n
Pnf = n- 1
L f(~i(W». i= 1
We were assuming measurability offunctions of W such as supS' JPnf - Pf I. Let us now consider more carefully the dependence on w, which we emphasize by writing Pn(w, .) instead of PnO.
PERMISSIBLE CLASSES
Suppose that the class fF is indexed by a parameter t that ranges over some set T. That is, fF = {f ( ., t): t E T} . We lose no generality by assuming fF so indexed; T could be fF itself. When more convenient, write fr instead of f(·, t). Assume T is a separable metric space. The metric on T will be important only insofar as it determines the bore! a-field £!leT) on T. 1 Definition. Call the class fF permissible if it can be indexed by a T in such a way that
(i) the function f(·, .) is !:f' ® £!l(T)-measurable as a function from S ® T into the real line; (ii) T is an analytic subset of a compact metric space T (from which it inherits its metric and bore! a-field). 0 Some authors call a T satisfying (ii) a Souslin measurable space [DM l6J. The usual sorts of class parametrized by bore! subsets [DM 12J of an euclidean space are permissible. (Take T as the one-point compactification.) So are fancier classes such as all indicator functions of compact, convex subsets of euclidean space (Problem 2). Assume from now on that fF is permissible and that (0, $, IP) is complete. Here are the properties of analytic sets that make the definition of permissibility a good one for empirical process applications. For every measurable space (M, vii), (a) d(M ® T) contains the product a-field vii ® £!leT); (b) for each H in d(M ® T), and in particular for each vii ® £!l(T)-measurable set, the projection nMH of H onto M is in d(M) [DM 13, DM 9: the set H is also in d(M ® T), because T is analyticJ; (c) for each A in d(M) and each $/vIi-measurable map I] from (0, $, IP) into M, the set {rt E A} is an analytic subset ofO [DM I1J; hence {I] E d} belongs to $, because (0, $, IP) is complete. From these properties we shall be able to deduce measurability for functions defined by certain uncountable operations.
197
C. Measurability
MEASURABLE SUPREMA
Suppose g(., .) is an A ® gg(T)-measurable real function on M ® T. Set G(m) = SUPt gem, t). Then by (a), d(M ® T) contains the set Ha. = {(m, t): gem, t) > IX}. The projection of Ha. onto M is an analytic set, by (b). It consists of all those m for which G(m) > IX. Thus {G > IX} belongs to d(M) for each real IX. If 11 is a measurable map from a complete probability space (n, tt, JP) into M then, by (c), the set {w: G(I1(w)) > IX} is tt-measurable. That is, SUPt g(I1(W), t) is an tt-measurable function of w. If g; is permissible and if P Ifr I < 00 for each t, requirement (i) of Definition 1 plus Fubini's theorem make Pfr a measurable function of t. Apply the argument given above, with M = sn, A = the product O"-field gm, 11 = the vector (~1"'" ~n)' and
g(s, t) = In- 1
it
[f(Si' t) -
PfrJI
to deduce that sup$'" IPnCw,f) - Pf I is a measurable function of w. MEASURABLE CROSS-SECTIONS
The Symmetrization Lemma 11.8 made use of another property of analytic sets. We had a stochastic process {Zt: t E T}; we assumed existence of a random r for which, almost surely, IZT I > S whenever SUPt IZt I > s. For this we need a cross-section theorem [DM44-45]. Under requirement (ii) of Definition 1, and for any complete probability space (M, A, f.1.), (d) if H belongs to d(M ® T) there exists a measurable map h from M into T U {oo} (where 00 is an ideal point added to T) such that: (m, hem)) belongs to H whenever hem) i= 00; and hem) i= 00 for f.1. almost all m in the projection 1rMH. Call h a measurable cross-section for H.
nMH •••..••••••••••••
gj
~_ _ _ _--11 T
CIJ
Write Z(w, t) to emphasize the role of Z as an tt ® 86'(T)-measurable function on n ® T. Let {s j} be a strictly decreasing sequence of real numbers converging to s, with Sl = 00. Set
{w:
s~p IZ(w, t)1 ~ Sj};
Aj
=
Bj
= {(w, t): Sj+1 < IZ(w,t) I ~
Sj+l
<
Sj}'
198
C. Measurability
The sets {A J all belong to tJ. The sets {B j } all belong to tJ ® f!lJ(T), and hence are analytic. Let to be any fixed element of T. Choose a measurable crosssection rj for each B j • Set r equal to rj on A j , and equal to to outside the union of the {A j }. Redefine rem) to be to whenever (d) would set it equal to 00. For almost all m, ifsuPt!Z(m, t)! > Gthen (m, rem)) belongs to BJor some j; that is, Gj+1 < Z(m, r(m))::;; Gj
for somej,
as required. A formal proof of the Symmetrization Lemma n.8 is now possible. Require Z and Z' to be defined on a product space Q ® Q' equipped with product measure JP ® lP', Z(m, m', t) = Z(m, t), Z'(m, m', t) = Z'(m', t). The r constructed above need depend only on m. For almost all m, lP'{m': !Z'(m', rem))! ::;; IX}
~
p.
The rest of the proof goes through as before, with Fubini's theorem formalizing the conditioning argument. SHATTERED SETS
Theorem n.21 placed a condition on the behavior of v" (~ l' . . . , ~n)' the smallest integer k such that ~ shatters no collection of k points from {~1' ... , ~n}' Assume that the indicator functions of sets in ~ form a permissible class $'. Then v" is measurable. For example, here is how to prove that {v" ::;; 2} belongs to tJ. Define g(8 1 , 8 z , t 1 , t z , t 3 , t 4 ) as
+ 1(8 1, t z )[l - 1(8 z , t z )] - 1(8 1 , t 3 )]1(8z, t 3 ) + [1 - 1(8 1 , t 4 )] [1 - 1(8 z , t 4 )].
1(8 1 , t 1 )1(8 z , t 1 )
+ [1
Clearly g is Y'z ® f!lJ(T 4 )-measurable. The function G(m) = max sup g(~;(m), i, j
~/m),
t)
t
is tJ-measurable. The set {v" ::;; 2} equals {G < 4}. COVERING NUMBERS
Functions of the empirical measure that enter into arguments depending on Fubini's theorem demand that we take measurability seriously. But for other functions, such as the random covering numbers appearing in Sections
199
C. Measurability
11.5, 11.6, VIA, and V1.6, we do not really need all the machinery of analytic sets. For example, we could just as well interpret a condition like log N 1 (b, P n , %) = oin) in terms of outer measure: for each e: > 0, IP*{log N 1 (b, P n , %) > ne:}
--+
0.
The proofs go through almost exactly as before. Equivalently, we could interpret the conditions on the covering numbers to mean IP{Zn > ne:} --+ for some measurable random variable Zn greater than log NI (b, P n' %). For permissible classes, another solution would be to replace covering numbers by packing numbers: define M 1 (b, P n , %) as the smallest m for which there exist functions fl' ... ,fm in % with P nI jj - hi> b for j #- k. This is a measurable function of m (and even jointly measurable in m and b); the set
°
{m: M 1 (b, Pn(m, .), %) equals the projection on {(m, t):
~
m}
n of the g ® gg(Tm)-measurable set
~:~ n-
1
J11
f(~i(m), t) - f(~;(m), t
k)
I>
b}-
The packing numbers are closely related to covering numbers: M 1 (2b, P n, %)
~
N 1 (b, P n, %)
~
M 1 (b, P n, %).
Theorems stated in terms of random covering numbers have equivalent versions for random packing numbers. I cannot prove measurability for covering numbers of permissible classes. The set of m where NI (b, P n , %) strictly exceeds m is the complement of a projection of a complement of an analytic set, which apparently need not be measurable. THE FUNCTION SPACE
?£
For the results in Chapter VII we required the class % to be pointwise bounded and have sup.? I Pf I finite. The empirical process, n
En(m, f) = n- 1 / 2
L: [f(~i(m)) -
PfJ
i= 1
had bounded sample paths; the functions En(m, .) all belonged to the set ?£ of bounded, real functions on %. To avoid confusion, we called members of ?£ functionals. We equipped ?£ with the uniform norm, Ilxll = sup.? Ix(f)l. The limit processes had sample paths in the set C(%, P) offunctionals that were uniformly continuous with respect to the !f?2(p) seminorm pp on YT. The (J-field ggP was smallest for which (i) all the closed balls (for 11·11) belonged to ggP; (ii) all the finite-dimensional projections were ggP-measurable.
200
C. Measurability
We assumed in Chapter VII that En is cff/.?JP-measurable. That is true for a permissible g; that is separable under the pp seminorm. Because each EnC, f) is a real random variable, the finite-dimensional projections create no difficulty for the cff/.?J P -measurability. The properties of analytic sets are needed to prove that {cv: 11 En( cv, .) - x(-) 11 :::; r} belongs to cff whenever x(·) belongs to C(&i', P). Introduce the index set T as described in Definition 1. Problem 1 shows that x(ft) is .?J(T)-measurable. Equip sn with its product IT-field yn. The function g(s, t) =
In-
1/2
itl
[f(Si'
t) - Pft] - x(ft) \
is gn ® .?J(T)-measurable. The argument in MEASURABLE SUPREMA establishes cff-measurability of SUPt g(~( cv), t), which equals 11 En< cv, .) - x(·) 11. NOTES Dudley (1978) introduced a condition, which he termed PE-Suslin, as a way of handling the measurability problems for empirical processes indexed by classes of sets. Since then he has refined the definition several times. He has called the latest version (Dudley 1984) of the condition "image admissible Suslin" (via a parameter space); it is almost the same as permissibility. Le Cam (1983) has also imposed a Suslin type of condition. Dudley and Philipp (1983) have systematically replaced measurability assumptions by conditions framed using measurable cover functions (the idea touched on in COVERING NUMBERs-an unfortunate clash of terminology-with the Zn random variables). PROBLEMS [1] Suppose g; has a countable, dense subset {g) under the £,2(p) seminorm pp. Suppose also that g; satisfies condition (i) of Definition 1. If x(·) is a bounded real functional on g; that is pp continuous, show that xC!') is a measurable function of t. [Assume x 2: O. Represent x(J,) as limsup sup x(g){r/t) < n-!}, where r/t)
=
pp(!, - g). Use Fubini's theorem to prove measurability of rj.]
[2] The class ~ of all non-empty, compact, convex subsets of the unit square [0, 1]2 is permissible. [Equip ~ with the metric d(C!, C 2 ) = inf{1> > O:C! <;;
Cl and C 2
<;;
q},
where Ct denotes the set of x less than I> away from at least one point of C. Under this metric, ~ is compact (Eggleston 1969, Section 4.2). The set of (x, C) with x in C is a closed subset of [0, 1]2 ® ~. Take ~ as its own indexing set.] [3] The class of sets {f> O}, for missible.
f running through a permissible class, is itself per-
References
Aldous, D. (1978). Stopping times and tightness. Annals of Probability 6:335-340. Alexander, K. S. (1984a). Probability inequalities for empirical processes and a law of the iterated logarithm. Annals of Probability. (Based on his 1982 Ph.D. dissertation, MIT.) To appear. Alexander, K. S. (1984b). Rates of growth for weighted 'empirical processes. (The Proceedings of the Neyman-Kiefer Memorial Conference, Berkeley, 1983. Belmont, CA: Wadsworth.) To appear. Alexandroff, A. D. (1940). Additive set functions in abstract spaces. Mat. Sbornik 50(NS 8):307-342. (Chapter 1.) Alexandroff, A. D. (1941). Additive set functions in abstract spaces. Mat. Sbornik 51(NS 9): 563-621. (Chapters 2 and 3.) Alexandroff, A. D. (1943). Additive set functions in abstract spaces. Mat. Sbornik 55(NS 13): 169-234. (Chapters 4 and 5.) Araujo, A. and Gine, E. (1980). The Central Limit Theoremfor Real and Banach Valued Random Variables. New York: Wiley. Ash, R. B. (1972). Real Analysis and Probability. New York: Academic Press. Bennett, G. (1962). Probability inequalities for the sum of independent random variables. Journal of the American Statistical Association 57: 33-45 Bertrand-Retali, M. (1974). Convergence uniforme d'un estimateur d'une densite de probabilite dans R S • Comptes Rendus de l' Academie des Sciences, Paris 278: 451-453. Bertrand-Retali, M. (1978). Convergence uniforme d'un estimateur de la densite par la methode du noyau. Revue Roumaine de Mathematiques Pures et Appliques 23:361-385. Billingsley, P. (1968). Convergence of Probability Measures. New York: Wiley. Billingsley, P. (1971). Weak Convergence of Measures: Applications in Probability. Philadelphia: Society for Industrial and Applied Mathematics. (Regional Conference Series in Applied Mathematics # 5.) Billingsley, P. and Tops!lle, F. (1967). Uniformity in weak convergence. Zeitschriftfor Wahrscheinlichkeitstheorie und Verw. Geb. 7: 1-16. Bollobas, B. (1979). Graph Theory: An Introductory Course. New York: Springer-Verlag. Bolthausen, E. (1978). Weak convergence of an empirical process indexed by the closed convex subsets of 12. Zeitschrift for Wahrscheinlichkeitstheorie und Verw. Geb. 43: 173-181.
202
References
Breiman, L. (1968). Probability. Reading, MA: Addison-Wesley. Bremaud, P. (1981). Point Processes and Queues: Martingale Dynamics. New York: Springer-Verlag. Breslow, N. and Crowley, J. (1974). A large sample study of the life table and product limit estimates under random censorship. Annals of Statistics 2:437-453. Brown, B. M. (1971). Martingale central limit theorems. Annals of Mathematical Statistics 42: 59-66. Brown, B. M. (1983). Statistical uses of the spatial median. Journal ofthe Royal Statistical Society, Series B 45: 25-30. Brown, B. M. and Kildea, D. G. (1979). Outlier-detection tests and robust estimators based on signs of residuals. Communications in Statistics-Theory and Methods A8:257-269. Chernoff, H. (1954). On the distribution of the likelihood ratio. Annals of Mathematical Statistics 25:573-578. Chernoff, H. (1956). Large sample theory: parametric case. Annals of Mathematical Statistics 27: 1-22. Chibisov, D. M. (1965). An investigation of the asymptotic power of tests of fit. Theory of Probability and Its Applications 10:421-437. Chung, K. L. (1949). An estimate concerning the Kolmogorov limit distribution. Transactions of the American Mathematical Society 67: 36-50. Chung, K. L. (1968). A Course in Probability Theory. New York: Harcourt, Brace and World. DeHardt, J. (1971). Generalizations of the Glivenko-Cantelli theorem. Annals of Mathematical Statistics 42 :2050-2055. DeIlacherie, C. (1972). Capacites et Processus Stochastiques. Heidelberg-Berlin: Springer-Verlag. DeIlacherie, C. and Meyer, P-A. (1978). Probabilities andPotential, Part A. Amsterdam: North-Holland. Translation of the 1975 French second edition of the first four chapters from Meyer (1966).) DeIlacherie, C. and Meyer, P-A. (1982). Probabilities and Potential, Part B. Amsterdam: North-Holland. Denby, L. and Martin, R. D. (1979). Robust estimation of the first-order autoregressive parameter. Journal of the American Statistical Association 74: 140-146. Donsker, M. D. (1951). An invariance principle for certain probability limit theorems. Memoirs of the American Mathematical Society 6: 1-12. Donsker, M. D. (1952). Justification and extension of Doob's heuristic approach to the Kolmogorov-Smirnov theorems. Annals of Mathematical Statistics 23: 27728t. Doob, J. L. (1949). Heuristic approach to the Kolmogorov-Smirnov theorems. Annals of Mathematical Statistics 20:393-403. Doob, J. L. (1953). Stochastic Processes. New York: Wiley. Dudley, R. M. (1966a). Weak convergence of probabilities on nonseparable metric spaces and empirical measures on euclidean spaces. Illinois Journal of Mathematics 10: 109-126. Dudley, R. M. (1966b). Convergence of Baire measures. Studia Mathematica 27: 251-268. Dudley, R. M. (1967a). Measures on non-separable metric spaces. Illinois Journal of Mathematics 11: 449-453. Dudley, R. M. (1967b). The sizes of compact subsets of Hilbert space and continuity of gaussian processes. Journal of Functional Analysis 1 :290-330. Dudley, R. M. (1968). Distances of probability measures and random variables. Annals of Mathematical Statistics 39: 1563-1572. Dudley, R. M. (1973). Sample functions of the gaussian process. Annals of Probability 1:66-103.
References
203
Dudley, R. M. (1974). Metric entropy of some classes of sets with differentiable boundary. Journal of Approximation Theory 10:227-236. Dudley, R. M. (1976). Probabilities and Metrics.' Convergence of laws on metric spaces, with a view to statistical testing. Aarhus: Aarhus University. (Mathematics Institute Lecture Note Series #45.) Dudley, R. M. (1978). Central limit theorems for empirical measures. Annals of Probability 6: 899-929. (Correction, ibid. 7 (1979): 909-911.) Dudley, R. M. (1981a). Donsker classes of functions. In Csorgo, M., Dawson, D. A., Rao, J. N. K., and Salek, A. K. Md. E. (Editors), Statistics and Related Topics, pages 341-352. Amsterdam: North-Holland. Dudley, R. M. (1981b). Vapnik-Cervonenkis Donsker classes offunctions. In Aspects Statistiques et Aspects Physiques des Processus Gaussiens. CNRS, Paris, St. Flour 1980. Dudley, R. M. (1984). A Course on Empirical Processes. Springer Lecture Notes in Mathematics. (Lectures given at Ecole d'Ete de Probabilites de St. Flour, 1982.) To appear. Dudley, R. M. and Philipp, W. (1983). Invariance principles for sums of Banach space valued random elements and empirical processes. Zeitschrift for Wahrscheinlichkeitstheorie und Verw. Geb. 62: 509-552. Durbin, J. (1973a). Weak convergence of the sample distribution function when parameters are estimated. Annals of Statistics 1: 279-290. Durbin, J. (1973b). Distribution Theory for Tests Based on the Sample Distribution Function. Philadelphia: Society for Industrial and Applied Mathematics. (Regional Conference Series in Applied Mathematics # 9.) Durst, M. and Dudley, R. M. (1981). Empirical processes, Vapnik-Cervonenkis classes and Poisson processes. Probability and Mathematical Statistics (Wrodaw) 1: 109-115. Eggleston, H. G. (1969). Convexity. Cambridge: Cambridge University Press. Erdos, P. and Kac, M. (1946). On certain limit theorems in the theory of probability. Bulletin of the American Mathematical Society 52: 292-302. Erdos, P. and Kac, M. (1947). On the number of positive sums of independent random variables. Bulletin of the American Mathematical Society 53: 1011-1020. Feller, W. (1968). An Introduction to Probability Theory and Its Applications, Vol. I (Third edition). New York: Wiley. Feller, W. (1971). An Introduction to Probability Theory and Its Applications, Vol. Il (Second edition). New York: Wiley. Gaenssler, P. (1984). Empirical Processes. Institute of Mathematical Statistics Lecture Notes-Monograph Series 3. Gaenssler, P. and Revesz, P. (Editors) (1976). Empirical Distributions and Processes. Springer Lecture Notes in Mathematics 566. Gaenssler, P. and Stute, W. (1979). Empirical processes: a survey of results for independent and identically distributed random variables. Annals of Probability 7: 193-243. Gihman, 1. 1. and Skorohod, A. V. (1969). Introduction to the Theory of Random Processes. Philadelphia: Saunders. (Earlier version of the three volumes 1974, 1975, 1979.) Gihman, 1. 1. and Skorohod, A. V. (1974). The Theory of Stochastic Processes, I. New York: Springer-Verlag. Gihman,1. 1. and Skorohod, A. V. (1975). The Theory of Stochastic Processes, Il. New York: Springer-Ver1ag. Gihman, 1. 1. and Skorohod, A. V. (1979). The Theory of Stochastic Processes, Ill. New York: Springer-Verlag. Gill, R. D. (1980). Censoring and Stochastic Integrals. Mathematical Centre Tract 124. Amsterdam: Mathematisch Centrum.
204
References
Gine, E. and Zinn, J. (1984). On the central limit theorem for empirical processes. Annals. of Probability. To appear. Glaisher, J. W. L. (1872). On the law of facility of errors of observations, and on the method of least squares. Memoirs of the Royal Astronomical Society 39: 75-124. Gnedenko, B. V. (1968). The Theory of Probability (Fourth edition). New York: Chelsea. Hajek, J. (1965). Extension of the Kolmogorov-Smirnov test to regression alternatives. In J. Neyman and L. M. Le Cam (Editors), Bernoulli, Bayes, Laplace: Anniversary Volume, pages 45-60. New York: Springer-Verlag. Hall, P. and Heyde, C. C. (1980). Martingale Limit Theory and Its Applications. New York: Academic Press. Halmos, P. R. (1969). Measure Theory. New York: Van Nostrand. Hansel, G. and Troallic, J. P. (1978). Mesures marginales et theoreme de FordFulkerson. Zeitschriftfur Wahrscheinlichkeitstheorie und Verw. Geb. 43:245-251. Hartigan, J. A. (1975). Clustering Algorithms. New York: Wiley. Hartigan, J. A. (1978). Asymptotic distributions for clustering criteria. Annals of Statistics 6: 117-131. Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association 58: 13-30. Huber, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Fifth Berkeley Symposium on Mathematical Statistics and Probability, pages 221-233. Berkeley, CA: University of California. Jacobs, K. (1978). Measure and Integral. New York: Academic Press. Jacobsen, M. (1982). Statistical Analysis of Counting Processes. Springer Lecture Notes in Statistics 12. Kac, M. (1949). On deviations between theoretical and empirical distributions. Proceedings of the National Academy of Science USA 35: 252-257. Kelley, J. L. (1955). General Topology. New York: Van Nostrand. Kolchinsky, V. I. (1982). On the central limit theorem for empirical measures. Theory of Probability and Mathematical Statistics (Kiev) 24:71-82. Kolmogorov, A. N. (1933). Sulla determinazione empirica di una legge di distribuzione. Giornale dell'lnstituto Ita!' degli Attuari 4: 83-91. Kolmogorov, A. N. (1956). On Skorohod convergence. Theory of Probability and Its Applications 1: 213-222. Kurtz. T. G. (1981). Approximation of Population Processes. Philadelphia: Society for Industrial and Applied Mathematics. (Regional Conference Series in Applied Mathematics # 36.) Le Cam, L. (1957). Convergence in distribution of stochastic processes. University of California Publications in Statistics 2: 207-236. Le Cam, L. (1983). A remark on empirical measures. In Bickel, P., Doksum, K., and Hodges, J. (Editors), Festschriftfor E. L. Lehmann, pages 305-327. Belmont, CA: Wadsworth. Levy, P. (1922). Sur la determination des lois de probabilite par leurs fonctions caracteristiques. Comptes Rendus de l'Academie des Sciences, Paris 175:854-856. Liapounoff, A. M. (1900). Sur une proposition de la theorie des probabilites. Bulletin de l' Academie imperiale des Sciences de St. Phersbourg 13: 359-386. Liapounoff, A. M. (1901). Nouvelle forme du theoreme sur la limite de probabilites. Memoires de l'Academie imperiale des Sciences de St. Phersbourg 12: Number 5. Lindeberg, J. W. (1922). Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung. M athematische Zeitschrift 15: 211-225. Lindvall, T. (1973). Weak convergence of probability measures and random functions in the function space D[O, 00). Journal of Applied Probability 10: 109-121. Liptser, R. S. and Shiryayev, A. N. (1977). Statistics of Random Processes, I: General Theory. New York: Springer-Verlag.
References
205
Liptser, R. S. and Shiryayev, A. N. (1978). Statistics of Random Processes, II: Applications. New York: Springer-Verlag. Loeve, M. (1978). Probability Theory, II. (Fourth Edition) New York: Springer-Verlag. Major, P. (1978). On the invariance principle for sums of independent identically distributed random variables. Journal of Multivariate Analysis 8: 487-517. Mann, H. B. and Wald, A. (1943). On stochastic limit and order relationships. Annals of Mathematical Statistics 14:217-226. Maritz, 1. S. (1981). Distribution-Free Statistical Methods. London: Chapman and Hall. McKean, H. P., lr. (1969). Stochastic Integrals. New York: Academic Press. McLeish, D. L. (1974). Dependent central limit theorems and invariance principles. Annals of" Probability 2: 620-628. Metivier, M. (1982). Semimartingales: A Course on Stochastic Processes. Berlin: DeGruyter. Meyer, P-A. (1966). Probability and Potential. (First edition). Boston: Blaisdell. Neveu, 1. (1975). Discrete-Parameter Martingales. Amsterdam: North-Holland. Neyman, 1. (1937). 'Smooth' test for goodness of fit. Skandinavisk Aktuarietidskrift 20: 149-199. (Reprinted in " A Selection of Early Statistical Papers of 1. Neyman," Cambridge: Cambridge University Press, 1967.) Oxtoby, 1. C. (1971). Measure and Category. New York: Springer-Verlag. Parr, W. C. (1982). Minimum distance estimation: a bibliography. Communications in Statistics, A 11: 1511-1518. Parthasarathy, K. R. (1967). Probability Measures on Metric Spaces. New York: Academic Press. Pickands, 1. (1971). The two-dimensional Poisson process and extremal processes. Journal of Applied Probability 8: 745-756. Pollard, D. (1980). The minimum distance method of testing. Metrika 27:43-70. Pollard, D. (1981a). Limit theorems for empirical processes. Zeitschriftfur Wahrscheinlichkeitstheorie und Verw. Geb. 57: 181-195. Pollard, D. (1981b). Strong consistency of k-means clustering. Annals of Statistics 9: 135-140. Pollard, D. (1982a). Beyond the heuristic approach to Kolmogorov-Smirnov theorems. In Gani 1. and Hannan E. 1. (Editors), Essays in Statistical Science. Journal of Applied Probability, Special Volume 19A, pages 359-365. (Festschrift for P. A. P. Moran.) Pollard, D. (1982b). Quantization and the method of k-means. IEEE Transactions on Information Theory IT-28: 199-205. Pollard, D. (1982c). A central limit theorem for empirical processes. Journal of the Australian Mathematical Society (Series A) 33: 235-248. Pollard, D. (1982d). A central limit theorem for k-means clustering. Annals of Probability 10: 919-926. Prohorov, Yu. V. (1956). Convergence of random processes and limit theorems in probability. Theory of Probability and Its Applications 1: 157-214. Pyke, R. (1969). Applications of almost surely convergent constructions of weakly convergent processes. Springer Lecture Notes in Mathematics 89: 187-200. Pyke, R. (1970). Asymptotic results for rank statistics. In M. L. Puri (Editor), Nonparametric Techniques in Statistical Inference, pages 21-37. Cambridge: Cambridge University Press. Pyke, R. and Shorack, G. (1968). Weak convergence ofa two-sample empirical process and a new approach to Chernoff-Savage theorems. Annals of Mathematical Statistics 39: 755-771. Pyke, R. and Sun, T. G. (1982). Weak convergence of empirical processes. (University of Washington Statistics Department Technical Report # 19.) Ranga Rao, R. (1962). Relations between weak and uniform convergence of measures with applications. Annals of Mathematical Statistics 33:659-680.
206
References
Resnick, S. I. (1975). Weak convergence to extremal processes. Annals of Probability 3:951-960. Revesz, P. (1976). Three theorems of multivariate empirical process. Springer Lecture Notes in Mathematics 566: 106-126. Sauer, N. (1972). On the density of families of sets. Journal of Combinatorial Theory (A) 13: 145-147. Shelah, S. (1971). Stability, the f.c.p., and superstability; model theoretic properties of formulas in first-order theory. Annals of Mathematical Logic 3:271-362. Shelah, S. (1972). A combinatorial problem; stability and order for models and theories in infinitary languages. Pacific Journal of Mathematics 41: 247-261. Shiryayev, A. N. (1981). Martingales: recent developments, results and applications. International Statistical Review 49: 199-233. Silverman, B. W. (1978). Weak and strong uniform consistency of the kernel estimate of a density and its derivatives. Annals of Statistics 6: 177-184. Simmons, G. F. (1963). Introduction to Topology and Modern Analysis. New York: McGraw-Hill. Skorohod, A. V. (1956). Limit theorems for stochastic processes. Theory of Probability and Its Applications 1:261-290. Skorohod, A. V. (1957). Limit theorems for stochastic processes with independent increments. Theory of Probability and Its Applications 2: 138-171. Steele, J. M. (1975). Combinatorial Entropy and Uniform Limit Laws. Ph.D. thesis, Stanford. (Reproduced by University Microfilms, Ann Arbor.) Steele, J. M. (1978). Empirical discrepancies and subadditive processes. Annals of Probability 6: 118-127. Stone, C. (1963). Weak convergence of stochastic processes defined on semi-infinite time intervals. Proceedings of the American Mathematical Society 14: 694-696. Straf, M. (1969). A general Skorohod space and its applications to the weak convergence of stochastic processes with several parameters. Ph.D. thesis, University of Chicago. Strassen, V. (1965). The existence of probability measures with given marginals. Annals of Mathematical Statistics 36: 423-439. Strassen, V. and Dudley, R. M. (1969). The central limit theorem and a-entropy. Springer Lecture Notes in Mathematics 89: 224-231. Stute, W. (1982a). The oscillation behavior of empirical processes. Annals of Probability 10:86-107. Stute, W. (1982b). A law of the logarithm for kernel density estimators. Annals of Probability 10:414-422. Talagrand, M. (1978). Les boules peuvent-elles engendrer la tribu borelienne d'un espace metrisable non separable? Seminaire Choquet, Paris 17e annee: C5. Tong, Y. L. (1980). Probability Inequalities in Multivariate Distributions. New York: Academic Press. Tops~e, F. (1970). Topology and Measure. Springer Lecture Notes in Mathematics 133. Uspensky, J. V. (1937). Introduction to Mathematical Probability. New York: McGrawHill. Vapnik, V. N. and Cervonenkis, A. Ya. (1971). On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications 16: 264-280. Vapnik, V. N. and Cervonenkis, A. Ya. (1981). Necessary and sufficient conditions for the uniform convergence of means to their expectations. Theory of Probability and Its Applications 26:532-553. Varadarajan, V. S. (1965). Measures on topological spaces. American Mathematical Society Translations 48: 161-228. (Translation from Mat. Sbornik 55 (97) (1961): 35-100.)
References
207
Weierstrass, K. (1885). Uber die analytische Darstellbarkeit sogenannter willkurlicher Funktionen reeller Argumente. Sitzungsberichte der Preussischen Akademie der Wissenschaften:633-640, 789-906. (Also in Collected Works, Vol. III, pp. 1-37, Berlin: Mayer u. Muller 1903.) Whitt, W. (1970). Weak convergence of probability measures on the function space C[O, 00). Annals of Mathematical Statistics 41 : 939-944. Whitt, W. (1980). Some useful functions for functional limit theorems. Mathematics of Operation Research 5:67-85. (First circulated as preprint in 1970.) Wichura, M. (1971). A note on the weak convergence of stochastic processes. Annals of Mathematical Statistics 42: 1769-1772.
Author Index
Aldous, D. 133, 137, 185, 187 Alexander, K. S. 38, 166 Alexandroff, A. D. 85 Amato, B. R. viii Araujo, A. 37 Ash, R. B. 39
Donsker, M. D. 117-118 Doob, J. L. 3,4,64, 113, 117 Dudley, R. M. vii, 37-39, 85-86, 117-118, 166-168, 193,200 Durbin, J. 118 Durst, M. 39
Barry, D. G. viii Bennett, G. 193-194 Bertrand-Retali, M. 38,41 Billingsley, P. 36,61,85-86,117-118, 136 Bolloblis, B. 86 Bolthausen, E. 167 Boyce, J. S. Vlll Breiman, L. 4,22, 106, 118 Bn!maud, P. 186 Breslow, N. 186 Brown, B. M. 166, 185
Eggleston, H. G. Erdos, P. 118
Cervonenkis, A. Ya. 37 Chernoff, H. 165, 190 Chibisov, D. M. 85 Chung, K. L. 117,173 Crowley, J. 186 DeHardt, J. 36 Dellacherie, C. 176,179,185,195-197 Denby, L. 37
Feller, W.
200
53, 63, 193
Gaenssler, P. vii, 36-37, 40, 118, 167 Gihman, I. I. 4,117-118,136-137,166, 185 Gill, R. D. 186 Gint':, E. 37,40, 166-167 Glaisher, J. W. L. 61 Gnedenko, B. V. 37
Hajek, J. 86, 117 Hall, P. 185 Halmos, P. R. 39 Hansel, G. 86 Hartigan, J. A. viii, 36, 166 Heyde, C. C. 185 Hoeffding, W. 193-194 Huber, P. J. 38, 165, 168
Author Index
210 Jacobs, K.
86
Jacobsen,~.
186
Kac,~. 117-118 Kelley, J. L. 39 Kildea, D. G. 166 Ko1chinsky, V. I. 38, 166 Kolmogorov, A. N. 99, 117, 136, 166 Kurtz, T. G. 137
Le Cam, L. 37-38,85, 166-167,200 Levy, P. 61 Liapounoff, A.~. 61,63 Lindeberg, J. W. 61 Lindvall, T. 118, 136 Liptser, R. S. 185 Loeve,~. 118
~ajor,
P. 86 H. B. 189 ~aritz, J. S. 166 ~artin, R. D. 37 ~cKean, H. P. 146 ~cLeish, D. L. 185 ~etivier,~. 185 ~eyer, P-A. 176,179,185,195-197 ~ann,
Neveu, J. 22 Neyman, J. 47 Nolan, D. A. viii Oxtoby, J. C.
65,86
Parr, W. 118 Parthasarathy, K. R. 85, 136 Philipp, W. 166,200 Pickands, J. 136 Pollard, D. 36-39, 118, 166, 169 Prohorov, Yu. V. 85-86 Pyke, R. vii, 85-86, 167
Ranga Rao, R. 86 Resnick, S. 1. 136 Revesz, P. 118, 167
Sauer, N. 37 Shelah, S. 37 Shiryayev, A. N. 185 Shorack, G. vii-viii, 85 Silverman, B. W. 38 Simmons, G. F. 67-68,85 Skorohod, A. V. 4, 86, 117-118, 122, 136-137, 166, 185 Steele, J.~. 37 Stone, C. 118, 136 Straf,~. 136 Strassen, V. 86, 167 Stute, W. vii, 36-38, 117 Sun, T. G. 167 Talagrand,~. 85 Tong, Y. L. 41 Tops0e,F. 36-37,62,85-86 Troallic, J. P. 86
Uspensky, J. V.
61, 193
Vapnik, V. N. 37 Varadarajan, V. S. 85
Wald, A. 189 Weierstrass, K. 61 Wellner, J. viii Whitt, W. 118, 136 Wichura,~. 86,117,136 Yu, K. F. Zinn, J.
186
37,38,40, 166-167
Subject Index
abuse, of notation 44, 171 adapted 176 Aldous's condition 133 Allocation Lemma 77 almost-sure representation. See Representation Theorem Approximation Lemma 27 autoregression 12, 174 axiom of choice 65, 86 BH . See brownian motion, stretched out Bp. See P-motion []BP
156
Bennett's Inequality 160, 192, 194 Bernstein's inequality 193 bias, of density estimator 35, 42 binomial coefficient 19 bounded variation 42 brownian bridge 64 defined 3, 95 existence 101, 119 brownian motion construction from brownian bridge 103 defined 95 modulus of continuity for 146 stretched-out 178 tied-down 3 C(§', P) 156 C[0,00) 108
C[O,IJ 1,90 cadlag 3, 89, 176 Cauchy sequence 81 censored data, estimation from 182 Central Limit Theorem Empirical, for distribution functions 97 Empirical, uniform case 96 Liapounoff 51 Lindeberg 52 Multivariate 57 central limit theorem autoregression 174 empirical process 157, 163-165 martingale-difference array 171 minimization functional 141 Chaining Lemma 144 chaining 37 restricted 160-163 for stochastic processes 142-145 characteristic function 54 Continuity Theorem 55, 60 inversion formula 63 uniform convergence on compacta 59 uniquely determines distribution 55 chi-square, Pearson's 46 clustering. See k-means combinatorial method 13 compactness in metric spaces 81-82 sequential 82 Compactness Theorem 82
212 completely regular point of metric space 67 topological space 68 conditional variance of increment 171 process 176 continuity set 71 Continuity Theorem, for characteristic functions 55, 60 Continuous Mapping Theorem for euclidean space 46 failure of 66 for metric spaces 70 consistency for autoregression 12 of k-means 9 of optimal solutions 12 convergence in distribution Aldous's condition for, in D[O, (0) 134 in D[O, (0), for Skorohod metric 131 in D[O, (0), for uniform convergence on compacta 108 of empirical process 97, 157 in euclidean space 43 of L 2 martingale 179 in metric spaces 65 process with independent increments 104,135 via semicontinuity 73 via uniform approximation 70 of uniform empirical process 96 Convergence Lemma for euclidean space 44--46 for metric spaces 68 convex sets, shattering 17, 23 convolution 35, 49, 54 coupling 76-80 in Compactness Theorem 84 covering integral for metric space 143 as modulus of continuity 147 covering numbers direct 164 inequalities between 34 2 1 , for classes of functions 25 2 2 , for classes of functions 31 measurability of 198 for metric spaces 143 for polynomial classes 27 random 150 criterion function minimization of 28 random 12
Subject Index
cross-section, measurable crossword puzzle 76
197
D[O, (0) definition of 107 metric of uniform convergence on compacta 108 Skorohod metric 123 D[O, 1] 3, 89-90 ~n@
21
delta method 63, 189 density convolution 55 estimation 35 differentiability, in quadratic mean 140, 151 direct approximation 8, 164 discrimination, linear; polynomial; quadratic 17 distribution function, pointwise convergence of 43, 46, 53 Donsker's theorem. See Central Limit Theorem, Empirical Doob-Meyer decomposition 176 ~-argument 24 En. See empirical process Ep. See P-bridge empirical measure bivariate 12 defined 6 empirical process central limit theorem for 96-97 defined 2,95,97,140 measurability of 65, 199 envelope, for classes of functions 24, 151 equicontinuity of class of functions 74 stochastic 139 Equicontinuity Lemma 150 ergodicity 9, 12 EXPONENTIAL BOUNDS 16 exponential bound. See tail probability extreme-value distribution 128
fidi projection 92. See also projection maps fidis (finite-dimensional distributions) 3,90,92 function space El" 155, 199
213
Subject Index
functional first passage time 109, 124 in function space 156 jump 179 terminology 2, 7 functions, infinitely differentiable 49 gaussian process characterization 106 indexed by functions 146 Glivenko-Cantelli theorem classical 6-7, 13 generalized, for classes of functions 25 generalized, for classes of sets 18,22 goodness-of-fit statistic 2 with estimated parameters 99, 159 Kolmogorov's, limiting distribution of 113
Neyman's 47 graph, of real-valued function 27 grid, approximations constructed from 91, 124 heuristic argument, Doob's 4, 64 Hewitt-Savage, zero-one law of 22 Hoeffding's Inequality 16,26, 31, 150, 164, 191 independent increments convergence in distribution definition 103 innovations 174 INTEGRATION 16,21 interpolation continuous 101 in D[O, 00) 124 in D[O, 1] 92 in a function space 158 J(b)
104, 135
xiv. See also covering integral
Kaplan-Meier estimator 182 kernel smoothing 35 translates of 42 k-means central limit theorem 153 consistency 9, 30 method of 9
Laplace transform, for hitting time 112 Liapounoff condition 51 Lindeberg condition 52, 106 for martingale-difference arrays 171 Lindel6f's theorem 68, 72, 87 Lipschitz condition 45, 74, 101 little links 163 location, center of 28
marginal stacks, with coupling 77 markov property for empirical process 96 marriage lemma 77 martingale continuous time 176 difference array 171 L2 176 L2, convergence to gaussian process 179 reversed 39 reversed, empirical measure as 22 stopped 180 MAXIMAL INEQUALITY 15 maximal inequality for cadlag processes 94 by chaining 144 tor L 2-martingales 177 maximum likelihood 138 measurability, for random elements of metric spaces 64-66 median central limit theorem for 53, 98 consistency 7 spatial, central limit theorem for 152 spatial, consistency of 28 M-estimator generalized 12 stochastic equicontinuity 165 metric bounded-Lipschitz 74 entropy 166 Prohorov 75, 79, 88 Skorohod, on D[O, 00) 123-124 minimization of random criterion function 140 modulus function for D[O, 00) 125 multinomial distribution 46
N(b) xiv. See also covering number 11 ,11· See norm norm 14,24, 156
214 normal distribution characteristic function of 63 multivariate 29 tail pro babili ties 191
optimal centers 10 outer measure 70 See projection u-field P n • See empirical measure P~ 15,26, 149 partial sum process 106, 109 P-bridge 149 permissible classes of functions 24 classes of sets 17 definition 196 'lis. See projection maps P-motion 147 polynomial bound for number of sets picked out 19 class 17,27 discriminating 17 discrimination 17,20 orthogonal 47 poisson process 105, 129 predictable 176 product u-field 69 projection, measurability of 196 projection maps 66, 90 for D[O, 00) 130 projection u-field 66, 90 on D[O, 00) 127 on D[O, I] 87 Prohorov's theorem. See Compactness Theorem [lJ>.
quadratic variation process 185 quantile transformation 57,97, 129 random element of metric space I, 65 random vector characteristic functions 56 convergence in distribution of 43, 56 perturbation of 48 rates of convergence 30 reflection principle, for brownian motion 112
Subject Index
remainder term, in Taylor expansion 50,139 Representation Theorem with random elements 71 with random variables 58 pp xiv robustnik 75
sample path 1 Scheffe's lemma 61 semicontinuity 73, 87 seminorm xiv separability of C[O, 1] 87 of D[O, 00), under Skorohod metric 127 and u-fields 85 for subsets of metric spaces 67 universal 38 shatter 18,21, 198 sigma field. See also fJ6P, [lJ> generated by balls 87 sign variables 15 signed measure. See ~ Slutsky's theorem 62 Souslin 196 square-root trick 32, 37 Stirling's approximation 21 stochastic equicontinuity 139-140 order symbol 141, 189 process I stopping time 110,172 Strasbourg theory 176 strong markov property, for brownian motion III submartingale 178 reversed 22, 25 subsets hidden 19 picked out 18 substitution, of increments 49 symmetrization 32, 149 inequality 14-15 SYMMETRIZATION, FIRST 14 Symmetrization Lemma 14, 198 SYMMETRIZATION, SECOND 15 symmetry, and empirical measures 21
tail probability characteristic function bound exponential bound 160, 191
60
215
Subject Index
tied-down brownian motion. See brownian bridge tight measure on metric space 81 on real line 60 total boundedness of metric space 82 and P-motion 168 triangular array 51; 106 truncation 25 2-means. See k-means
uniform tightness for general empirical processes 156 for uniform empirical processes 102 usual conditions 176
Un. See empirical process, uniform
weak convergence in euc1idean space 44 in metric spaces 65 weight functions 158
uniform convergence of density estimator 36 uniform integrability 176 uniform strong law of large numbers 7 for classes offunctions 25, 168 for classes of sets 18, 22 for convex sets 22
VC class 37 vector space, finite-dimensional 30, 38
fI. See function space
20,
