Uniform
z J '.L
The book shows how the central limit theorem for independent, identically distributed random variables with values in general, multidimensional spaces holds uniformly over some large classes of functions. The book contains, with complete proofs, the Fernique-Talagrand majorizing measure theorem for Gaussian processes, an extended treatment of VapnikCernovenkis combinatorics, the Ossiander L2 bracketing central limit theorem,
the Gine-Zinn bootstrap central limit theorem in probability, the Bronstein theorem on approximation of convex sets, and the Shor theorem on rates of convergence over lower layers. The book incorporates an updated form of the author's 1984 St.-Flour lecture notes and also gives various results of the author's not previously collected in one place. A number of recent results of Talagrand and others are surveyed without proofs in separate sections. The book will interest mathematicians working in probability, mathematical statisticians, and computer scientists working in computer learning theory.
R. M. Dudley is Professor of Mathematics at the Massachusetts Institute of Technology in Cambridge, Massachusetts.
Cambridge Studies in Advanced Mathematics 63 Editorial Board: W. Fulton, D. J. H. Garling, T. tom Dieck, P. Walters Already published 2 K. Petersen Ergodic theory 3 P. T. Johnstone Stone spaces J.-P. Kahane Some random series of functions, 2nd edition 5 7 J. Lambek & P. J. Scott Introduction to higher-order categorical logic H. Matsumura Commutative ring theory 8 9 C. B. Thomas Characteristic classes and the cohomology of finite groups 10 M. Aschbacher Finite group theory 11 J. L. Alperin Local representation theory 12 P. Koosis The logarithmic integral I 13 A. Pietsch Eigenvalues and s-numbers 14 S. J. Patterson An introduction to the theory of the Riemann zeta-function 15 H. J. Baues Algebraic homotopy V. S. Varadarajan Introduction to harmonic analysis on semisimple 16 Lie groups 17 W. Dicks & M. Dunwoody Groups acting on graphs L. J. Corwin & F. P. Greenleaf Representations of nilpotent Lie groups 18 and their applications 19 R. Fritsch & R. Piccinini Cellular structures in topology 20 H. Klingen Introductory lectures on Siegel modular forms 21 P. Koosis The logarithmic integral II 22 M. J. Collins Representations and characters of finite groups 24 H. Kunita Stochastic flows and stochastic differential equations 25 P. Wojtaszczyk Banach spaces for analysts 26 J. E. Gilbert & M. A. M. Murray Clifford algebras and Dirac operators in harmonic analysis 27 A. Frohlich & M. J. Taylor Algebraic number theory 28 K. Goebel & W. A. Kirk Topics in metric fixed point theory 29 J. F. Humpheys Reflection groups and Coxeter groups 30 D. J. Benson Representations and cohomology I D. J. Benson Representations and cohomology 11 31 32 C. Allday & V. Puppe Cohomological methods in transformation groups 33 C. Soule et al. Lectures on Arakelov geometry 34 A. Ambrosetti & G. Prodi A primer of nonlinear analysis 35 J. Palis & F. Takens Hyperbolicity, stability and chaos at homoclinic
bifurcation 37 38 39
40 41
42 43 44 45
46 47 49
Y. Meyer Wavelets and operators 1 C. Weibel Introduction to homological algebra W. Bruns & J. Herzog Cohen-Macaulay rings V. Snaith Explicit Brauer induction G. Laumon Cohomology of Drinfeld modular varieties I E. B. Davies Spectral theory and differential operators
J. Diestel, H. Jarchow, & A. Tonge Absolutely summing operators P. Mattila Geometry of sets and measures in Euclidean spaces R. Pinsky Positive harmonic function and diffusion G. Tenenbaum Introduction to analytic and probabilistic number theory C. Peskine An algebraic introduction to complex projective geometry R. Stanley Enumerative combinatorics I
50 51
52 53 54 55
56 60
I. Porteous Clifford algebras and the classical groups M. Audin Spinning tops V. Jurdjevic Geometric control theory H. Volklein Groups as Galois groups J. Le Potier Lectures on vector bundles D. Bump Automorphic forms and representations G. Laumon Cohomology of Drinfeld modular varieties 11 M. P. Brodmann & R. Y. Sharp Local cohomology
UNIFORM CENTRAL LIMIT THEOREMS R. M. DUDLEY Massachusetts Institute of Technology
CAMBRIDGE UNIVERSITY PRESS
PUBLISHED BY THE PRESS SYNDICATE OF THE UNIVERSITY OF CAMBRIDGE The Pitt Building, Trumpington Street, Cambridge CB2 IRP 4o West 20th Street, New York, NY 10011-4211, USA 10 Stamford Road, Oakleigh, Melbourne 3166, Australia
© Cambridge University Press 1999 First published 1999 Typeface Times 10/13 pt. System LATEX [RW]
A catalog record of this book is available from the British Library
Library of Congress cataloging in publication data
P.
Dudley, R. M. (Richard M.) Uniform central limit theorems / R. M. Dudley. cm. - (Cambridge studies in advanced mathematics: 63) Includes bibliographical references. ISBN 0 52146102 2
1. Central limit theorem. I. Title. H. Series. QA 273.67.D84 1999 519.2-DC21 98-35582 CIP
ISBN 0 521 46102 2 hardback
Transferred to digital printing 2004
To Liza
Contents
Preface
page xiii
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities 1.1 Empirical processes: the classical case 1.2 Metric entropy and capacity 1.3 Inequalities Problems Notes 1
References
1
2 10 12 18
19 21
Gaussian Measures and Processes; Sample Continuity 2.1 Some definitions 2.2 Gaussian vectors are probably not very large 2.3 Inequalities and comparisons for Gaussian distributions 2.4 Gaussian measures and convexity 2.5 The isonormal process: sample boundedness and continuity 2.6 A metric entropy sufficient condition for sample continuity 2.7 Majorizing measures 2.8 Sample continuity and compactness **2.9 Volumes, mixed volumes, and ellipsoids **2.10 Convex hulls of sequences Problems Notes 2
References
23 23
24 31
40 43 52 59 74 78
82 83
86 88
Foundations of Uniform Central Limit Theorems: Donsker Classes 3.1 Definitions: convergence in law 3.2 Measurable cover functions 3
ix
91 91
95
x
Contents 3.3
3.4 3.5 3.6 3.7 3.8 3.9
4 4.1 4.2 *4.3 *4.4 *4.5 4.6 4.7 4.8 **4.9
Almost uniform convergence amd convergence in outer probability Perfect functions Almost surely convergent realizations Conditions equivalent to convergence in law Asymptotic equicontinuity and Donsker classes Unions of Donsker classes Sequences of sets and functions Problems Notes
6.2 6.3 6.4 **6.5 **6.6 **6.7
117 121
122 127 130 132
Vapnik-Cervonenkis Combinatorics Vapnik-Cervonenkis classes Generating Vapnik-Cervonenkis classes Maximal classes Classes of index 1 Combining VC classes Probability laws and independence Vapnik-Cervonenkis properties of classes of functions Classes of functions and dual density Further facts about VC classes Problems Notes
134 134 138 142 145 152 156 159
References
168
Measurability *5.1 Sufficiency 5.2 Admissibility 5.3 Suslin properties, selection, and a counterexample Problems Notes
6.1
111
References
5
6
100 103 106
161 165
166 167
170 171
179 185 191
193
References
194
Limit Theorems for Vapnik-Cervonenkis and Related Classes Koltchinskii-Pollard entropy and Glivenko-Cantelli theorems Vapnik-Cervonenkis-Steele laws of large numbers Pollard's central limit theorem Necessary conditions for limit theorems Inequalities for empirical processes Glivenko-Cantelli properties and random entropy Classification problems and learning theory Problems
196 196 203
208 215 220 223 226 227
Contents
7 7.1
7.2 7.3 **7.4
Notes References Metric Entropy, with Inclusion and Bracketing Definitions and the Blum-DeHardt law of large numbers Central limit theorems with bracketing The power set of a countable set: the Borisov-Durst theorem Bracketing and majorizing measures Problems Notes
228 230 234 234 238 244 246 247 248
References
248
Approximation of Functions and Sets 8.1 Introduction: the Hausdorff metric 8.2 Spaces of differentiable functions and sets with differentiable boundaries 8.3 Lower layers 8.4 Metric entropy of classes of convex sets Problems Notes References Sums in General Banach Spaces and Invariance Principles 9 9.1 Independent random elements and partial sums 9.2 A CLT implies measurability in separable normed spaces 9.3 A finite-dimensional invariance principle 9.4 Invariance principles for empirical processes **9.5 Log log laws and speeds of convergence Problems Notes References 10 Universal and Uniform Central Limit Theorems 10.1 Universal Donsker classes 10.2 Metric entropy of convex hulls in Hilbert space **10.3 Uniform Donsker classes Problems Notes References The Two-Sample Case, the Bootstrap, and Confidence Sets 11 11.1 The two-sample case 11.2 A bootstrap central limit theorem in probability 11.3 Other aspects of the bootstrap 8
xi
250 250 252
264 269 281
282 283 285 286 291 293 301
306 309 310 311
314 314 322 328 330 330 330 332 332 335 357
Contents
xii
** 11.4 Further Gine-Zinn bootstrap central limit theorems Problems Notes References
Classes of Sets or Functions Too Large for Central Limit Theorems 12.1 Universal lower bounds 12.2 An upper bound 12.3 Poissonization and random sets 12.4 Lower bounds in borderline cases 12.5 Proof of Theorem 12.4.1 Problems Notes
358 359
360 361
12
References
Appendix A Differentiating under an Integral Sign Appendix B Multinomial Distributions Appendix C Measures on Nonseparable Metric Spaces Appendix D An Extension of Lusin's Theorem Appendix E Bochner and Pettis Integrals Appendix F Nonexistence of Types of Linear Forms on Some Spaces Appendix G Separation of Analytic Sets; Borel Injections Appendix H Young-Orlicz Spaces Appendix I Modifications and Versions of Isonormal Processes Subject Index Author Index Index of Notation
363 363 365
367 373
384 388 388 389 391
399
402 405 407 413 417 421
425 427
432 435
Preface
Suppose given a probability distribution P on the plane and a random sample of n points, chosen independently with distribution P. For each half-plane H bounded by a line, the number k of sample points in H has a binomial distribution. Suitably centered, taking k - nP(H), and divided by 'In-, k has an asymptotically normal (Gaussian) distribution as n -) oo, by De Moivre's classical central limit theorem. It will be seen herein, as one example of a uniform central limit theorem, that the asymptotic normality holds simultaneously and uniformly over all half-planes. The corresponding property of half-lines in the line was first shown by M. Donsker. Thus classes of sets, or functions, for which a uniform central limit theorem holds are called Donsker classes. It turns out, as will be seen, that rather general classes of sets in, or functions on, general spaces are Donsker classes. This book developed out of some topics courses given at M.I.T. and my lectures at the St.-Flour probability summer school in 1982. The material of the book has been expanded and extended considerably since then. The reader will need to know some real analysis including Lebesgue integration, and probability based on it, including the finite-dimensional central limit theorem. Starred sections are not cited later in the book except perhaps in other starred sections. At the end of some chapters are doubly starred sections. These are surveys, without proofs, on topics not covered in the book, usually because I did not know short enough proofs to include. Also at the end of each chapter are some problems, notes, and references on that chapter. For useful conversations on topics in the book, I'm glad to thank Kenneth Alexander, Niels Trolle Andersen, Miguel Arcones, Patrice Assouad, Erich Berger, Lucien Birge, Igor S. Borisov, Donald Cohn, Yves Derrienic, Uwe Einmahl, Joseph Fu, Evarist Gine, Sam Gutmann, David Haussler, Jt rgen Hoffmann-Jorgensen, Yen-Chin Huang, Vladimir Koltchinskii, Lucien Le Cam, xiii
xiv
Preface
Pascal Massart, James Munkres, Rimas Norvaiga, Walter Philipp, Tom Salisbury, Galen Shorack, Rae Shortt, Michel Talagrand, He Sheng Wu, Joe Yukich, and Joel Zinn. I especially thank Yong Chen, Peter Gaenssler, Evarist Gind, Jinghua Qian, Arvind Sankar, and Franz Strobl for providing lists of corrections and suggestions. I also thank Xavier Fernique and Evarist Gine very much for sending me copies of recent expositions. Cambridge, Massachusetts February 22, 1999
Note. Throughout this book, all references to "RAP" are to the author's book Real Analysis and Probability, Wadsworth and Brooks/Cole, Pacific Grove, Calif. 1989, reprinted with corrections by Chapman and Hall, New York, 1993. Also, "A := B" means A is defined by B, whereas "A =: B" means B is defined by A.
1
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
Let P be a probability measure on the Borel sets of the real line IR with distribu-
tion function F(x) := P((-oo, x]). Here and throughout, ":=" means "equals by definition." Let X1, X2, be i.i.d. (independent, identically distributed) random variables with distribution P. For each n = 1, 2, and any Borel
set A C R, let P,, (A) := ; F-jn-1 Sxj(A), where 8x(A) = lA(x). Then P , X and is called the empirical measure. Let F be the distribution function of P,,. Then F, is called the empirical distribution function. The developments to be described in this book began with the GlivenkoCantelli theorem, a uniform law of large numbers, which says that with probis a probability measure f o r each X1,
ability 1, F converges to F as n -* oo, uniformly on R, meaning that supx I (F - F) (x) I -+ 0 as n oo (RAP, Theorem 11.4.2); as mentioned in the Note at the end of the Preface, "RAP" refers to the author's book Real Analysis and Probability. The next step was to consider the limiting behavior of a := n112(F - F) as n -+ oo. For any fixed t, the central limit theorem in its most classical form, for binomial distributions, says that converges in distribution to N(0, F(t)(1 - F(t))), in other words a normal (Gaussian) law, with mean 0 and variance F(t)(1 - F(t)). Here a law is a probability measure defined on the Borel sets. For any finite set T of values oft, the multidimensional central limit theorem (RAP, Theorem 9.5.6) tells us that a (t) for t in T converges in distribution as n oc to a normal law N(0, CF) with mean 0 and covariance CF(s, t) _
F(s)(1 - F(t)) for s < t. The Brownian bridge (RAP, Section 12.1) is a stochastic process yt (w) defined for 0 < t < 1 and w in some probability space 0, such that for any finite
set S C [0, 1], yt for t in S have distribution N(0, C), where C = CU for the uniform distribution function U(t) = t, 0 < t < 1, and t -- yt(w) is I
2
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
continuous for almost all w. So the empirical process a converges in distribution to the Brownian bridge composed with F, namely t i-+ YF(t), at least when restricted to finite sets. It was then natural to ask whether this convergence extends to infinite sets or the whole interval or line. Kolmogorov (1933) showed that when F is continuous, the supremum sups a,r (t) and the supremum of absolute value, sups l an (t) 1, converge in distribution to the laws of the same functionals of yF. Then, these functionals of yF have the same distributions as for the Brownian
bridge itself, since F takes R onto an interval including (0, 1) and which may or may not contain 0 or 1; this makes no difference to the suprema since yo = y1 = 0. Also, yt -+ 0 almost surely as t 10 or t f 1 by sample continuity; the suprema can be restricted to a countable dense set such as the rational numbers in (0, 1) and are thus measurable. Kolmogorov evaluated the distributions of sups yt and sups I Yr I explicitly (see RAP, Propositions 12.3.3 and 12.3.4). Doob (1949) asked whether the convergence in distribution held for more general functionals. Donsker (1952) stated and proved (not quite correctly) a general extension. This book will present results proved over the past few
decades by many researchers, where the collection of half-lines (-oo, x], x E R, is replaced by much more general classes of sets in, and functions on, general sample spaces, for example the class of all ellipsoids in R3. To motivate and illustrate the general theory, the first section will give a revised formulation and proof of Donsker's theorem. Then the next two sections, on metric entropy and inequalities, provide concepts and facts to be used in the rest of the book.
1.1 Empirical processes: the classical case In this section, the aim is to treat an illuminating and historically basic special
case. There will be plenty of generality later on. Here let P be the uniform distribution (Lebesgue measure) on the unit interval [0, 1]. Let U be its distribution function, U(t) = t, 0 < t < 1. Let U be its empirical distribution functions and a := n1/2(U - U) on [0, 1]. It will be proved that as n -+ oo, a, converges in law (in a sense to be made precise below) to a Brownian bridge process yt, 0 < t < 1 (RAP, before Theorem 12.1.5). Recall that yt can be written in terms of a Wiener process (Brownian motion) xt, namely yt = xt - tx1, 0 < t < 1. Or, yt is xt conditioned on x1 = 0 in a suitable sense (RAP, Proposition 12.3.2). The Brownian bridge (like the Brownian motion) is sample-continuous, that is, it can be chosen such that for all w, the function t H yt (w) is continuous on [0, 1] (RAP, Theorem 12.1.5).
1.1 Empirical processes: the classical case
3
Donsker in 1952 proved that the convergence in law of an to the Brownian bridge holds, in a sense, with respect to uniform convergence in t on the whole interval [0, 1]. How to define such convergence in law correctly, however, was not clarified until much later. General definitions will be given in Chapter 3.
Here, a more special approach will be taken in order to state and prove an accessible form of Donsker's theorem. For a function f on [0, 1], we have the sup norm
Ilfll0 = sup{If(t)I: 0 < t < 1}. Here is the form of Donsker's theorem that will be the main result of this section.
1.1.1 Theorem
For n = 1, 2,
, there exist probability spaces Stn such
that:
(a) On Q,,, there exist n i.i.d. random variables X1, , X, with uniform distribution in [0, 1]. Let an be the nth empirical process based on these X,; (b) On Qn a sample-continuous Brownian bridge process Yn: (t, w) H Yn (t, (o) is also defined;
(c) Ilan - Ynllooismeasurable,andforalls > 0,Pr(Ilan-Ynlloo > E) -* 0 as n --> oo.
Notes. (i) Part (c) gives a sense in which the empirical process an converges in distribution to the Brownian bridge with respect to the sup norm II Il,". (ii) It is actually possible to use one probability space on which X1, X2,
are i.i.d., while Y, = (B1 +
+ Bn)//i, Bj being independent Brownian
bridges. This is an example of an invariance principle, to be treated in Chapter 9, not proved in this section. (iii) One can define all an and Yn on one probability space and make Y, all equal some Y, although here the joint distributions of an for different n will be
different from their original ones. Then an will converge to Y in probability and moreover can be defined so that Ilan - Y 1100 0 almost surely, as will be shown in Section 3.5. Proof For a positive integer k, let Lk be the set of k + 1 equally spaced points,
Lk := {0, 1/k, 2/k,
,
1} c [0, 1].
It will first be shown that both processes an and yt, for large enough n and k, can be well approximated by step functions and then by piecewise-linear interpolation of their values on Lk.
4
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
Given 0 < s < 1, take k = k(s) large enough so that
4k exp (- ks2/648) < s/6.
(1.1.2)
Let Ijk := [j/k, (j + 1)/k], j = 0,
,
k - 1. By the representation yt =
xt - txl, we have Pr{IYt - Yi1kI > 8/6 for some t E Ilk} < P1 + p2, where
P1 := Pr(lxl l > ks/18),
P2 := Pr{lxt - x3/kl > E/9 for some t E Ilk}.
Then pi < 2 exp(-k2s2/648) (RAP, Lemma 12.1.6(b)). For P2, via a reflection principle (RAP, 12.3.1) and the fact that {xu+h - xu}h>o has the same
distribution as {xh }h>o (applied to u = j/k), we have p2 < 4 exp(-ks2/162). Thus by (1.1.2), (1.1.3)
Pr{IYt - Yj/kl > s/6
for some j = 0,
, k - 1 and some t E Ilk} < s/3.
Next, we need a similar bound for a when n is large. The following will help:
1.1.4 Lemma Given the uniform distribution U on [0, 1]:
(a) For 0 < u < 1 and any finite set S C [0, 1 - u], the joint distribution of { Un (u + s) - U (u) }SES is the same as for u = 0. (b) The same holds for an in place of Un.
(c) The distribution of sup[ lan(t + j1 k) - an(j/k)I: 0 < t < 1/k) is the same for all j.
Proof (a) Let S = NJ', where we can assume so = 0. It's enough to consider { Un (u + sj) - Un (u + sj_ t) }iL1, whose partial sums give the desired quantities. Multiplying by n, we get m random variables from a multinomial distribution for n observations for the first m of m + 1 categories, which have
probabilities {sj - sj_111, where sn,+l = 1 (Appendix B, Theorem B.2). This distribution doesn't depend on u.
(b) Since an (u + s) - a (u) = n 1/2 (Un (u + s) - U. (u) - s), (b) follows from (a). (c) The statement holds for finite subsets of Ijk by (b). By monotone convergence, we can let the finite sets increase up to the countable set of rational
numbers in Ilk. Since Un is right-continuous, suprema over the rationals in Ilk equal suprema over the whole interval (the right endpoint is rational), and Lemma 1.1.4 is proved.
1.1 Empirical processes: the classical case
5
So in bounding the supremum in Lemma 1.1.4(c) we can take j = 0, and we need to bound Pr{n 112 Un (t) - t I > s for some t E [0, 1/k]}. Suppose , nr of sample size n = given a multinomial distribution of numbers n 1, , pr. Then for each j, the + nr in r bins with probabilities p I, n1+ conditional distribution of nj+1 given n 1, , nj is the same as that given
- nj trials + nj, namely a binomial distribution for n - n i nj+ with probability pj+l/(P.i+l + + pn) of success on each (see Appendix B, Theorem B.3(c)). It follows that the empirical distribution function U, has the following Markov property: if 0 < tl < < tj < t < u, then the conditional distribution of U (u) given U (ti), , U (tj), U (t) is the same as that given U, (t). Specifically, given that U (t) = m/n, the conditional distribution of
Un(u) is that of (m + X)/n where X has a binomial distribution for n - m trials with success probability (u - t)/(1 - t). To be given U,(t) = m/n is equivalent to being given an (t) = n 1/2 (n - t), and an also has the Markov property. So the conditional distribution of an (u) given m = n Un (t) has mean Am
I--tt
:=n1/2{n
=n1/2\\n -t/
\1// 1
and variance
(n-m)(u-t)(1-u)
u-t
n(1 - t)2
1-t
< u.
So, by Chebyshev's inequality, Pr { Ian (u) - µm 1 > 2u 1/2 1 m } < 1/4.
If u < 1/2, then i-t > 2. Let 0 < S < 1. If an (t) > 8, then n - t > S/n1/2 and µm > 8(i -) > 3/2, so for any y > 8 (such that Pr{an (t) = y} > 0), S
Pr an(u) > 2 - 2u 1/2 an (t) = Y } > 3/4. (For such a y, y = n1/2(n - t) for some integer m.) If u < 82/64, then u < 1 /2 and Pr{an (u) > 8/4 1 an (t) = y) > 3/4.
Let u = 1 / k and S = e/4. Then by (1.1.2), since e-' < 1/24 implies x > 2, we have u < 82/64, so
Pr{an(1/k)>e/161an(t)=y} > 3/4 for y>s/4. Now take a positive integer r and let r be the smallest value of j/(kr), , r, for which an (r) > s/4. Let Ar be the event that such a j exists. Let Arj := {r = j/(kr)}. Then Ar is the union of the
if any, for j = 1,
6
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
disjoint sets Arj for j = 1, , r. For each such j, by the Markov property, Pr{an(1/k) > E/16 I Arj} > 3/4. Thus
Pr{an(1/k) > E/16 I Ar} > 3/4. Let r -+ oo. Then by right continuity of U, and an, we get
Pr{an(t) > E/4 for some t E [0, 1/k]) <
Pr{an(1/k) > E/16). 3
Likewise,
Pr{an(t) < -E/4 for some t E [0, 1/k]) < 3 Pr{an(1/k) < -E/16). Thus by Lemma 1.1.4(c), (1.1.5)
Pr{Ian(t) -an(j/k)I > E/4 forsomet E Ilk and j = 0, 1,
,
k - 11 < (4k/3) Pr(lan(1/k)I > e/16).
As n oo, for our fixed k, by the central limit theorem and RAP, Lemma 12.1.6(b),
Pr{lan(1/k)I > E/16) - Pr{IY1/kI > E/16} < 2 exp (- ks2/512
.
So for n large enough, say n > no = no(E), recalling that k = k(E),
Pr{lan(1/k)l > s/16} < 3 exp (- ks2/512. Then by (1.1.5) and (1.1.2), for n > no, (1.1.6)
Pr{Ian(t) - an(jl k) I > E/4 for some j = 0,
,
k - 1 and t E Ilk} < E/6.
As mentioned previously, the law, say 4 (an ), of {an (i / k) }k o converges by the central limit theorem in IRk+i to that of {Yi/k}k o, say Gk(y). On ][8k+1 put the metric d,,,(x, y) := Ix - yl := maxi I xi - yi 1, which of course metrizes the
usual topology. Since convergence of laws is metrized by Prokhorov's metric p (RAP, Theorem 11.3.3), for n large enough, say n > n 1(E) > no (E), we have p(Gk(an), 4(y)) < E/6. Then by Strassen's theorem (RAP, Corollary 11.6.4), there is a probability measure lcn on Rk+1 x Rk+1 such that for (X, Y) with G(X, Y) = µn, we have (1.1.7)
L(X) = Lk(an),
£(Y) = Lk(Y),
and µn {(x, Y) : Ix - Yloo > E/6} < E/6
(RAP, Section 9.2).
1.1 Empirical processes: the classical case
7
Let Libk (for "linear in between") be the function from Rk+1 into the space
C[0, 1] of all continuous real functions on [0, 1] such that Libk (x) (j/ k) = xj, j = 0, , k, and is linear (affine) on each closed interval Ilk = [j/k, (j + 1)/k], j = 0, , k - 1. For any x, y E Rk+1, Libk(x) Libk(y) is also linear on each Ilk, so it attains its maximum, minimum, and
-
maximum absolute value at endpoints. So for the supremum norm 11 f 11 o0 supo<x<1 I.f(x)I on C[0, 1], Libk is an isometry into:
II Libk(x) - Libk(y) it oo = Ix - YIoo
for all x, y E I[8k+1
Since C [0, 1 ] is a separable metric space (e.g., RAP, Corollary 11.2.5), (f, g) H
Il.f - glloo is jointly measurable in f, g E C[0, 1] (RAP, Proposition 4.1.7). So, from (1.1.7) we get (1.1.8)
µn{IhLibk(x) -Libk(y)Iloo > e/6) < e/6.
For any real-valued function f on [0, 1], let Yrk(f) _ { E Then nk(Libk(x)) = x for all x E Rk+1 For a sample-continuous Brownian bridge process (w, t) i-a yt (w), 0 < t < Rk+1.
1, the map
w i-- it i-- yt(w): 0 < t < 11 E C[0, 1] is measurable for the Borel a-algebra on C[0, 1] (by a simple adaptation of RAP, Proposition 12.2.2). (Recall that in any topological space, the Borel a-algebra is the one generated by the open sets.) If I xi+l - xjl < e then ILibk(x)(t) - xj I < e for all t E Ilk. It follows from (1.1.3) that (1.1.9)
Pr{IIY - Libk(Jrk(Y))Il oo >
e/3} < e/3,
where for each w, we have a function y: t i-+ yt (w), 0 < t < 1. We can take the probability space for each an process as the unit cube In, , xn, with where then i.i.d. uniform variables in defining Un and an are x1, x = (xl, xn) E In. Then
Ak: X H flan(J/k)}l-o is measurable from In into Rk+1 and has distribution Gk(an) on Rk+1 Also, x i-- Libk(Ak(x)) is measurable from In into C[0, 1]. The next theorem will give a way of linking up or "coupling" processes. Recall that a Polish space is a topological space metrizable by a complete separable metric. 1.1.10 Theorem (Vorob'ev-Berkes-Philipp) Let X, Y and Z be Polish spaces with Borel a-algebras. Let a be a law on X x Y and let # be a law on Y x Z.
8
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
Let zry(x, y) := y and ry(y, z) := y for all (x, y, z) E X x Y x Z. Suppose the marginal distributions of a and P on Y are equal, in other words z a o7rY 1 = B o rY1 on Y. Let n12(x, Y, z) :_ (x, y) and 1x23(x, Y, z) :_ (Y, Z). Then there exists a law yon X x Y x Z such that y o7r121 = a and y o7r231 = B
Proof There exist conditional distributions ay for a on X given y E Y, so that for each y E Y, ay is a probability measure on X, for any Borel set A C X, the function y i-± ay(A) is measurable, and for any integrable function f for a,
f fda = fff(xY)day(x)d1)(y) (RAP, Section 10.2). Likewise, there exist conditional distributions fly on Z for P. Let x and z be conditionally independent given y. In other words, define a set function y on X x Y x Z by
y(C) = JJf
lc(x, y,
z) day(x) doy(z) dq (Y)
The integral is well-defined if
(a) C = U x V x W for Borel sets U, V, and W in X, Y, and Z, respectively; (b) C is a finite union of such sets, which can be taken to be disjoint (RAP, Proposition 3.2.2 twice); or
(c) C is any Borel set in X x Y x Z, by RAP, Proposition 3.2.3 and the monotone class theorem (RAP, Theorem 4.4.2).
Also, y is countably additive by monotone convergence (for all three integrals). Soy is a law on X x Y x Z. Clearly y o Jr121 = a and y o 7r-1
Now, let's continue the proof of Theorem I.I.I. The function (x, f) H Ilan - flloo is jointly Borel measurable for x E In and f E C[0, 1]. Also, u i-+ Libk(u) is continuous and thus Borel measurable from Rk+1 into C[0, 1]. So (x, u) i-+ Ilan - Libk(u) II , is jointly measurable on In x 1Rk+1 (This is
true even though an V C [0, 1] and the functions t -* an (t) for different w form a nonseparable space for II Iloo.) Thus x i-a Ilan - Libk(Ak(x))Iloo is measurable on In. From (1.1.6), we then have (1.1.11)
Pr{IIan - Libk(Ak(x))Il oo > E/2} < E/6.
Apply Theorem 1.1.10 to (X, Y, Z) = (In, Rk+1 Rk+1) with the law of (x, Ak(x)) on In x Rk+1, and p,, from (1.1.7) on Rk+1 x Rk+1 both of which induce the law L k (an) on Y = Rk+1 to get a law yn .
1.1 Empirical processes: the classical case
9
Then apply Theorem 1.1.10, this time to (X, Y, Z) = (In X Rk+t Rk+1 C[0, 1]), with yn on X x Y and the law of (zrk(y), y) on Y x Z, where y is the Brownian bridge.
We see that there is a probability measure n on In x C[0, 1] such that if L (Vn, Yn) = cn, then £(Vn) is uniform on In, £(Y,) is the law of the Brownian bridge, and if we take an = an (Vn ), then for n > n 1(E) defined after (1.1.6), Ilan - YnIIoc
<-
Ilan -Libk(7rk(an))Iloo
+ IILibk(nk(an)) - Libk(nk(Yn)) Iloo + IILibk(nk(Yn)) - Yn Iloo
< E/2 + s/6 + E/3 < s except on a set with probability at most 6 + 6 + 3 < E, by (1.1.11), (1.1.8), and (1.1.9), respectively.
Let On := In x C[0, 1], n > 1 . For r = 1, 2, , let nr := nt(1/r). Let N, be an increasing sequence with N,. > nr for all r. For n < Ni, define µn as in (1.1.7) but with 1 in place of E/6 (both times), so that it always
holds: one can take An as the product measure 4 (an) x 4(y). Define on On as above, but with 1 in place of E/m for m = 2, 4, or 6 in (1.1.6) and
(1.1.11). For Nr < n < Nr+t, define µn and as for e = 1/r. Then Pr(Ilan - N loo > 1/r) < 1/r for n > N,., r > 1, and Theorem 1.1.1 is proved.
0
Remarks. It would be nice to be able to say that an converges to the Brownian bridge yin law in some space S of functions with supremum norm. The standard definition of convergence in law, at least if S is a separable metric space, would
say that EH(an) - EH(y) for all bounded continuous real functions H on S (RAP, Section 9.3). Donsker (1952) stated this when continuity is assumed only at almost all values of y in C[0, 1]. But then, H could be nonmeasurable away from the support of y, and EH(an) is not necessarily defined. Perhaps
more surprisingly, EH(an) may not be defined even if H is bounded and continuous everywhere. Consider for example n = 1. Then in the set of all possible functions Ul - U, any two distinct functions are at distance 1 apart Iloo. So the set and all its subsets are complete, closed, and discrete for II oo If the image of Lebesgue (uniform) measure on [0, 11 by the function II x i--* (t i-k 1{x>t} - t) were defined on all Borel sets for II III in its range, for II
or specifically on all complete, discrete sets, it would give an extension of Lebesgue measure to a countably additive measure on all subsets of [0, 1]. Assuming the continuum hypothesis, which is consistent with the other axioms of set theory, such an extension is not possible (RAP, Appendix Q.
10
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
So in a nonseparable metric space, such as a space of empirical distribution functions with supremum norm, the Borel or-algebra may be too large. In Chapter 3 it will be shown how to get around the lack of Borel measurability. Here is an example relating to the Vorob'ev theorem (1.1.10). Let X = Y = Z = {-1, 1). In X x Y x Z let each coordinate x, y, z have the uniform distribution giving probability 1 /2 each to -1, 1. Consider the laws on the
products of two of the three spaces such that y = -x, z = -y, and x = -z. There exist such laws having the given marginals on X, Y and Z. But there is no law on X x Y x Z having the given marginals on X x Y, Y x Z, and Z x X, since the three equations together yield a contradiction.
1.2 Metric entropy and capacity The word "entropy" is applied to several concepts in mathematics. What they have in common is apparently that they give some measure of the size or complexity of some set or transformation and that their definitions involve logarithms. Beyond this rather superficial resemblance, there are major differences. What are here called "metric entropy" and "metric capacity" are measures of the size of a metric space, which must be totally bounded (have compact completion) in order for the metric entropy or capacity to be finite. Metric entropy will provide a useful general technique for dealing with classes of sets or functions in general spaces, as opposed to Markov (or martingale) methods. The latter methods apply, as in the last section, when the sample space is R and the class C of sets is the class of half-lines (-oo, x], x E IR, so that C with its ordering by inclusion is isomorphic to R with its usual ordering.
Let (S, d) be a metric space and A a subset of S. Let s > 0. A set F C S (not necessarily included in A) is called an s-net for A if and only if for each x E A, there is a y E F with d (x, y) < s. Let N(s, A, S, d) denote the minimal number of points in an c-net in S for A. Here N(s, A, S, d) is sometimes called a covering number. It's the number of closed balls of radius s and centers in S needed to cover A. For any set C C S, define the diameter of C by diam C := sup{d(x, y) : x, y E C}. Let N(s, C, d) be the smallest n such that C is the union of n sets of diameter at most 2s. Let D(s, A, d) denote the largest n such that there is a subset F C A
with F having n members and d (x, y) > e whenever x 0 y for x and y in F. Then, in a Banach space, D(2s, A, d) is the largest number of disjoint closed balls of radius s that can be "packed" into A and is sometimes called a "packing number."
1.2 Metric entropy and capacity
11
The three quantities just defined are related by the following inequalities:
1.2.1 Theorem For any E > 0 and set A in a metric space S with metric d,
D(2E, A, d) < N(s, A, d) < N(s, A, S, d)
< N(s, A, A, d) < D(s, A, d). Proof The first inequality holds since a set of diameter 2s can contain at most one of a set of points more than 2e apart. The next holds because any ball B(x, s) := {y: d(x, y) < s} is a set of diameter at most 2E. The third inequality holds since requiring centers to be in A is more restrictive. The last holds because a set F of points more than E apart, with maximal cardinality, must be an s-net, since otherwise there would be a point more than s away from each point of F, which could be adjoined to F, a contradiction unless F is infinite, but then the inequality holds trivially. It follows that as s ,j. 0, when all the functions in the theorem go to 00 unless S is a finite set, they have the same asymptotic behavior up to a factor of 2 in s. So it will be convenient to choose one of the four and make statements about
it, which will then yield corresponding results for the others. The choice is somewhat arbitrary. Here are some considerations that bear on the choice. The finite set of points, whether more than s apart or forming an s-net, are often useful, as opposed to the sets in the definition of N(s, A, d). N(s, A, S, d) depends not only on A but on the larger space S. Many workers, possibly for these reasons, have preferred N(s, A, A, d). But the latter may decrease when the set A increases. For example, let A be the surface of a sphere of radius s
around 0 in a Euclidean space S and let B := A U (0). Then N(s, B, B, d) = 1 < N(s, A, A, d) for 1 < s < 2. This was the reason, apparently, that Kolmogorov chose to use N(s, A, d). In this book I adopt D(s, A, d) as basic. It depends only on A, not on the larger space S, and is nondecreasing in A. If D(s, A, d) = n, then there are n points which are more than s apart and at the same time form an s-net. Now, the s-entropy of the metric space (A, d) is defined as H(s, A, d) log N(s, A, d), and the s-capacity as log D(s, A, d). Some other authors take logarithms to the base 2, by analogy with information-theoretic entropy. In this book logarithms will be taken to the usual base e, which fits for example with bounds coming from moment-generating functions as in the next section, and with Gaussian measures as in Chapter 2. There are a number of interesting sets
of functions where N(s, A, d) is of the order of magnitude exp(s-') as s 4, 0, for some power r > 0, so that the s-entropy, and likewise the s-capacity, have
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
12
the simpler order a-r. But in other cases below, D(e, A, d) is itself of the order of a power of 1/e.
1.3 Inequalities This section collects several inequalities bounding the probabilities that random variables, and specifically sums of independent random variables, are large. Many of these follow from a basic inequality of S. Bernstein and P. L. Chebyshev.
1.3.1 Theorem For any real random variable X and t E R,
Pr{X > t} <
info>oe-tuEeux.
Proof For any fixed u > 0, the indicator function of the set where X > t satisfies 1{x>t) < eu(x-t)so the inequality holds for a fixed u; then take info>o.
For any independent real random variables X1,
, X,,, let S := X1 +
... + Xn. 1.3.2 Bernstein's inequality Let X1, X2, , X be independent real random variables with mean 0. Let 0 < M < oo and suppose that I Xj I < M almost surely for j = 1, , n. Let Q? = Var(Xj) and t,, := Var(SS)
of + (1.3.3)
+ Q,2. Then for any K > 0,
Pr{ISnI > Kn1/2) <
Proof We can assume that t,2 > 0, since otherwise S, = 0 a.s. (where a.s. means almost surely) and the inequality holds. For any u > 0 and j = 1, (1.3.4)
, n,
E exp(uXj) = I+ u2o Fj/2 < exp (6,2 Fju2/2),
or Fj = 0 if 6? = 0. For r > 2, IXjjr < X2Mr-2 a.s., so Fj < 2 Er2 (Mu)r-2/r! < Err=2 (Mu/3)r-2 = 1/(1-Mu/3)forall <3/M.
where Fj := 2O2 Er°02
Let v := Kn112 and u := v/(r,2 + My/3), so that v = Mu/3). Then 0 < u < 3/M. Thus, multiplying the factors on the right side of (1.3.4) by independence, we have
Eexp(uSn) < exp
22
Mu/3)) = exp(uv/2).
1.3 Inequalities
13
So by Theorem 1.3.1, Pr{Sn > v) < e-`12 and e-uv12
= exp ( - v2/(2r,, + 2Mv/3)) = exp ( - nK2/(2rn +2MKn1/2/3))
Here are some remarks on Bernstein's inequality. Note that for fixed K and M, if X, are i.i.d. with variance v2, then as n -+ oo, the bound approaches the normal bound 2 exp(-K2/(2Q2)), as given in RAP, Lemma 12.1.6. Moreover, this is true even if M := Mn -+ oo as n -+ oo while K stays constant, provided that M/n1/2 - 0. Sometimes the inequality can be applied to unbounded variables, replacing them by "truncated" ones, say replacing an unbounded f by fm where fm(x) := f(x)l{I f(x)1<M} , be i.i.d. variables with P(s, = 1) = P(s, -1) = 1/2. Next, let sl, s2, Such variables are called "Rademacher" variables. We have the following inequality.
1.3.5 Proposition (Hoeffding) For any t > 0 and real a1, n
n
ajsj > t
Pr
< exp -t2 2 a
j=1
j=1
Proof Since 1/(2n)! < 2-'In! for n =
we have cosh x
(ex + e-x)/2 < exp(x2/2) for all x. Apply Theorem 1.3.1, where by calculus, infra exp(-ut + y=1 a?u2/2) is attained at u = t/ F-jn=1 aj?, and the result follows. Here are some remarks on Proposition 1.3.5. Let Y1, Y2, , be independent variables which are symmetric, in other words Yj has the same distribution as - Yj for all j. Let s; be Rademacher variables independent of each other and of all the Yj. Then the sequence {sjYj}{ j>1} has the same distribution as {Yj}{ j>1}.
Thus to bound the probability that E 1 Yj > K, for example, we can consider the conditional probability for each Y1, , Yn ,
Pr{ YsjYj > K j=1
Y11..., Ynl
<
exp \-K2/
\2 J=1
Y2
//
by 1.3.5. Then to bound the original probability, integrating over the distribution of the Yj, one just needs to have bounds on the distribution of Fj 1 Y?, which may simplify the problem considerably.
14
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
The Bernstein inequality (1.3.2) used variances as well as bounds for centered variables. The following inequalities, also due to Hoeffding, use only bounds.
They are essentially the best that can be obtained, under their hypotheses, by the moment-generating function technique.
1.3.6 Theorem (Hoeffding) Let X1,
, Xn be independent variables with
0 < Xj < 1 for all j. Let T:= (Xl +
+ Xn)/n and µ := EX Then for
0
fµ+t 1 1µµ
Pr IT -µ>t}
I
-
14
< e-nt28(µ) <
e-2nt2
where
g(µ) := (1 - 2µ)-1 log((1 - µ)/µ)
for 0 < µ < 1/2, or for 1/2 < µ < 1.
:= 1/(2µ(l - µ))
Remarks. Fort > 1 - µ, Pr(X - it > t) < Pr(X > 1) = 0. Fort < 0, the given probability would generally be of the order of 1/2 or larger, so no small bound for it would be expected. Proof For any v > 0, the function f (x) := e"X is convex (its second derivative is positive), so any chord lies above its graph (RAP, Section 6.3). Specifically,
if 0 < x < 1, then e"X < 1-x+xe". Taking expectations gives E exp(vXj) < 1 - µj + µje", where µj := EXj. (Note that the latter inequality becomes an equation for a Bernoulli variable Xj, taking only the values 0, 1.) Let
S, := X, +-..+X,. Then Pr ( X - µ > t) = Pr(S, - ESn > nt) < E exp(v(S, - ESn - nt)) e-un(t+µ)11r1 E exp(vXj)
< e-nv(t+µ)H=1(1-µj+ttje"). The following fact is rather well known (e.g., RAP (5.1.6) and p. 276):
1.3.7 Lemma For any nonnegative real numbers ti,
, tn, the geometric mean is less than or equal to the arithmetic mean, in other words (t, t2 ... tn)lln
. (tl + ... + tn)/n.
1.3 Inequalities
15
Applying the lemma gives n
Pr {
X - µ > t} <
e-nv(t+u) 1
e-nv(t+t0(I
(.El-ttj+Ajev
- it +
µev)n.
To find the minimum of this for v > 0, note that it becomes large as v ---). 00
since t + µ < 1, while setting the derivative with respect to v equal to 0 gives a single solution, where 1 - µ + µev = µev/(t + µ) and ev = 1 + t/(µ(1- µ - t)). Substituting these values into the bounds gives the first, most complicated bound in the statement of the theorem. This bound can be written
as Pr(X -A > t) < exp(-nt2G(t, µ)), where
G(t,µ) := µt2
tlog(µl2 t2
\\mino
1 1µµ -
G (t, µ) = g(µ) as defined in ment of the theorem. For 0 < x < 1 let The next step is to show that
H(x) :=
1-
I \\\
theJe
state-
2) log(1 - x).
x
In aG(t, µ)/at, the terms not containing logarithms cancel, giving
t2 aGat, tt) =
[1_(1_IL)]log(1_1t) [
- L1- t(lµ+t)]log11-µ+t)
H \1tµ/ H \µ+t) . To see that H is increasing in x for 0 < x < 1, take the Taylor series of log(1 - x) and multiply by 1 - x to get the Taylor series of H around 0, all whose coefficients are nonnegative. Only the first order term is 0. Thus
aG/at > 0 if and only if t/(1 - µ) > t/(µ + t), or equivalently t > 1 - 2µ. So if µ < 1/2, then G(t, µ), for fixed µ, has a minimum with respect to t > 0 at t = 1 - 2µ, giving g(µ) for that case as stated. Or if it > 1/2, then G(t, µ) is increasing in t > 0, with limt10 G (t, µ) = g(ls) as stated for that case, using the first two terms of the Taylor series around t = 0 of each logarithm. Now to get the final bound exp(-2nt2), it needs to be shown that the minimum of g(µ) for 0 < µ < 1 is 2. For µ > 1/2, g is increasing (its denominator
is decreasing), and g(1/2) = 2. For µ < 1/2, letting w := I - 2µ, we get g(µ) = w log('+w). From the Taylor series of log(1 + w) and log(1 - w)
16
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
around w = 0, we see that g is increasing in w, and so decreasing in s, and converges to 2 as µ - 1/2. Thus g has a minimum at tt = 1/2, which is 2. The theorem is proved.
For the empirical measure Pn, if A is a fixed measurable set, n Pn (A) is a binomial random variable, and in a multinomial distribution, each n1 has a binomial distribution. So we will have need of some inequalities for binomial probabilities, defined by
(fl)pjqn_j,
B(k, n, p)
0 < q := I - p < 1,
0<j
E(k, n, p)
y (fl)pjqfl_j.
k<j
Here k is usually, but not necessarily, an integer. Thus, in n independent trials with probability p of success on each trial, so that q is the probability of failure, B(k, n, p) is the probability of at most k successes, and E(k, n, p) is the probability of at least k successes.
1.3.8 Chernoff-Okamoto inequalities We have (1.3.9)
E(k, n, p) <
k
Ik (n
-k
if k > n p,
(1.3.10) B(k, n, p) < exp ( - (np - k)2/(2npq)) if k < np < n/2. Proof These facts follow directly from the Hoeffding inequality 1.3.6. For (1.3.10), note that B(k, n, p) = E(n - k, n, 1 - p) and apply the g(µ) case
with it = 1-p. If in (1.3.9) we set x := nq/(n - k) < ex-1, it follows that (1.3.11)
E(k, n, p) < (np/k)k ek-np if k > np.
The next inequality is for the special value p = 1/2.
1.3.12 Proposition If k < n/2 then 2nB(k, n, 1/2) < (ne/k)k. Proof By (1.3.9) and symmetry, B(k, n, 1/2) < (n/2)nk-k(n - k)k-n. Letting y := n/(n - k) < ey-1 then gives the result. A form of Stirling's formula with error bounds is:
1.3 Inequalities
1.3.13 Theorem For n = 1, 2,
,
17
e1/(12n+1)
< n!(e/n)n(27rn)-1/2 <
el/12n
Proof See Feller (1968), vol. 1, Section 11.9, p. 54.
For any real x let x+ := max (x, 0). A Poisson random variable z with parameter m has the distribution given by Pr(z = k) = e-'nmk/k! for each nonnegative integer k.
1.3.14 Lemma For any Poisson variable z with parameter m > 1,
E(z - m)+ > m 1/2/8.
Proof We have E(z - m)+ = Eke,,, a-mmk(k - m)/k!. Let j := [m], meaning j is the greatest integer with j < m. Then by a telescoping sum (which is absolutely convergent), E(z-m)+ = e mmj+1/j!.ThenbyStirling's formula with error bounds (Theorem 1.3.13),
E(z - m)+
e-mmj+1(e/j )j >
(2nj)-112e-11(12j)
(mj+1/jj+2)e-13/12(27r)-1/2 > m1/2/8.
In the following two facts, let X1, X2, , X, be independent random variables with values in a separable normed space S with norm II II . (Such spaces
are defined, for example, in RAP, Section 5.2.) Let Sj := X1 +
j = 1,...,n.
+ Xj for
1.3.15 Ottaviani's inequality If for some a > 0 and c with 0 < c < 1, we have P(IIS,, - Sjll >
P{maxj
P(IlSnll?a)/(1-c).
Proof The proof in RAP, 9.7.2, for S = Rk, works for any separable normed S. Here (x, y) i-- Ilx - fill is measurable: S x S i-- JR by RAP, Proposition 4.1.7.
When the random variables Xj are symmetric, there is a simpler inequality:
1.3.16 P. Levy's inequality Given a probability space (S2, P) and a countable set Y, let X1, X2, be stochastic processes defined on S2 indexed by Y, in other words for each j and y E Y, is a random variable on Q. For any bounded function f on Y, let 11f Il Y := sup{ l f (y) I : y E Y). Suppose that
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
18
the processes Xj are independent, with II Xj II y < 00 a.s., and symmetric, in other words for each j, the random variables {-Xj(y) : y E Y} have the same joint distribution as {Xj(y) : y E Y}. Let Sn := Xl + + Xn. Then for each
n, andM>0, P(maxj
M) <2P(11 Sn11Y>M). Notes. Each II Sj 11Y is a measurable random variable because Y is countable.
Lemma 9.1.9 treats uncountable Y. The norm on a separable Banach space (X, 11
II) can always be written in the form 11 II y for Y countable, via the Hahn-
Banach theorem (apply RAP, Corollary 6.1.5, to a countable dense set in the unit ball of X to get a countable norming subset Y in the dual X' of X, although X' may not be separable). On the other hand, the preceding lemma applies to some nonseparable Banach spaces: the space of all bounded functions on an infinite Y with supremum norm is itself nonseparable. Proof Let Mk(w) := maxj
M < Mk}, k = 1, 2,
, where we set Mo := 0. Then for 1 < m < n, 211Sm11Y < IISnIlY+112Sm-SnIly.SoifIISmIIY>M,then l1Sn11Y> Mor II 2Sm - Sn Il Y > Mor both. The transformation which interchanges Xj and - Xj
just for m < j < n preserves probabilities, by symmetry and independence. Then S, is interchanged with 2Sm - Sn, while Xj are preserved for j < m. So P(Cm fl {IISnIlY > M}) = P(Cm fl {112Sm - Sn11Y > M}) > P(Cm)/2, and
P(Mn>M)=Em=1 P(Cm)<_2P(IISn1IY>M).
Problems 1. Find the covariance matrix on 10, 1/4, 1/2, 3/4, 1) of (a) the Brownian bridge process yr; (b) U4 - U. Hint: Recall that n 1/2(U - U) has the same covariances as yr.
2. Let 0 < t < u < 1.
Let an be the empirical process for the uniform
distribution on [0, 1].
(a) Show that the distribution of an (t) is concentrated in some finite set At.
(b) Let f (t, y, u) := E(an (u) I an (t) = y). Show that for any yin At, (u, f (t, y, u)) is on the straight line segment joining (t, y) to (1, 0).
Notes
19
3. Let (S, d) be a complete separable metric space. Let s be a law on S x S and let 6 > 0 satisfy
p,({(x, y): d(x, y) > 28}) < 38.
Let n2(x, y) := y and P := s o Let Q be a law on S such that p(P, Q) < S where p is Prokhorov's metric. On S x S x S let n21.
7r12(x, y, z) := (x, y) and 1r3(x, y, z) := Z. Show that there exists a law a on S x S x Ssuch that ao7ri21 =µ,ao7T 1 = Q, and
a({(x, y, z): d(x, z) > 33)) < 48. Hint: Use Strassen's theorem, which implies that for some law v on S x S,
if £(Y, Z) = v, then £(Y) = P, C(Z) = Q, and v({d(Y, Z) > S)) < 8. Then the Vorob'ev-Berkes-Philipp theorem applies.
4. LetA=B=C={0,1}.OnAxB,let it :_ (8(0,0) + 28(1,0) + 56(0,1) + 8(1,1)) /9.
On B x C, let v := [8(0,0) + 8(1,0) + 8(0,1) + 38(1,1)]/6. Find a law y on A x B x C such that if y = £(X, Y, Z), then L (X, Y) = ,a and G(Y, Z) = v. 5. Let I = [0, 1] with usual metric d. For e > 0, evaluate D(e, I, d), N(E, I, d), and N(E, I, I, d). Hint: The ceiling function [xl is defined as the least integer > x. Answers can be written in terms of [.1.
6. For a Poisson variable X with parameter ,l > 0, that is, P(X = k) _ e-xAk/k! for k = 0, 1 , 2, , evaluate the moment-generating function EeIX for all t. For M > A, find the bound for Pr(X > M) given by the moment-generating function inequality (1.3.1).
Notes Notes to Section 1.1. The contributions of Kolmogorov (1933), Doob (1949), and Donsker (1952) were mentioned in the text. When it was realized that the formulation by Donsker (1952) was incorrect because of measurability problems, Skorokhod (1956) - see also Kolmogorov (1956) - defined a separable metric d on the space D[0, 1] of right-continuous functions with left limits on [0, 1], such that convergence ford to a continuous function is equivalent to convergence for the sup norm, and the empirical process a converges in law in D[0, 1] to the Brownian bridge; see, for example, Billingsley (1968, Chapter 3). The formulation of Theorem 1.1.1 avoids the need for the Skorokhod topology and deals with measurability. I don't know whether Theorem 1.1.1 has
20
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
been stated before explicitly, although it is within the ken of researchers on empirical processes. In Theorem 1.1.10, the assumption that X, Y and Z are Polish can be weakened: they could instead be any Borel sets in Polish spaces (RAP, Section 13.1). Still more generally, since the proof of Theorem 10.2.2 in RAP depends just on
tightness, it is enough to assume that X, Y and Z are universally measurable subsets of their completions, in other words, measurable for the completion of any probability measure on the Borel sets (RAP, Section 11.5). Shortt (1983) treats universally measurable spaces and considers just what hypotheses on X, Y and Z are necessary. Vorob'ev (1962) proved Theorem 1.1.10 for finite sets. Then Berkes and Philipp (1977, Lemma Al) proved it for separable Banach spaces. Their proof carries over to the present case. Vorob'ev (1962) treated more complicated families of joint distributions on finite sets, as did Shortt (1984) for more general measurable spaces.
Notes to Section 1.2. Apparently the first publication on e-entropy was the announcement by Kolmogorov (1955). Theorem 1.2.1, and the definitions of all the quantities in it, are given in the longer exposition by Kolmogorov and Tikhomirov (1959, Section 1, Theorem IV). Lorentz (1966) proposed the name "metric entropy" rather than "s-entropy," urging that functions should not be named after their arguments, as functions of a complex variable z are not called "z-functions." The name "metric entropy" emphasizes the purely metric nature of the concept. Actually, "s-entropy" has been used for different quantities. Posner, Rodemich, and Rumsey (1967, 1969) define an s, S entropy, for a metric space S with a probability measure P defined on it, in terms of a decomposition of S into sets of diameter at most s and one set of probability at most S. Also, Posner et al. define s-entropy as the infimum
of entropies - F; P(U;) log(P(U;)) where the U; have diameters at most E. So Lorentz's term "metric entropy" seems useful and will be adopted here.
Notes to Section 1.3. Sergei Bernstein (1927, pp. 159-165) published his inequality. The proof given is based on Bennett (1962, p. 34) with some incorrect, but unnecessary steps (his (3), (4), ...) removed as suggested by Gine (1974). For related and stronger inequalities under weaker conditions, such as unbounded variables, see also Bernstein (1924, 1927), Hoeffding (1963), and Uspensky (1937, p. 205). Hoeffding (1963, Theorem 2) implies Proposition 1.3.5. Chernoff (1952, (5.11)) proved (1.3.9). Okamoto (1958, Lemma 2(b')) proved (1.3.10). Inequality (1.3.11) appeared in Dudley (1978, Lemma 2.7) and Lemma 1.3.12 in
References
21
Dudley (1982, Lemma 3.3). On Ottaviani's inequality (1.3.15) for real-valued functions, see (9.7.2) and the notes to Section 9.7 in RAP. The P. Levy inequality (1.3.16) is given for Banach-valued random variables in Kahane (1985, Section 2.3). For the case of real-valued random variables, it was known much earlier; see the notes to Section 12.3 in RAP.
References *An asterisk indicates a work I have seen discussed in secondary sources but not in the original.
Bennett, George (1962). Probability inequalities for the sum of independent random variables. J. Amer. Statist. Assoc. 57, 33-45. Berkes, Istvan, and Philipp, Walter (1977). An almost sure invariance principle for the empirical distribution function of mixing random variables. Z Wahrscheinlichkeitsth. verw. Gebiete 41, 115-137. Bernstein, Sergei N. (1924). Ob odnom vidoizmenenii neravenstva Chebysheva i o pogreshnosti formuly Laplasa (in Russian). Uchen. Zapiski Nauchn.-issled. Kafedr Ukrainy, Otdel. Mat., vyp. 1, 38-48; reprinted in S. N. Bem9tein, Sobranie Sochinenii [Collected Works], Tom IV, Teoriya Veroiatnostei, Matematicheskaya Statistika, Nauka, Moscow, 1964, pp. 71-79. *Bernstein, Sergei N. (1927). Teoriya Veroiatnostei (in Russian), 2d ed. Moscow, 1934. Billingsley, Patrick (1968). Convergence of Probability Measures. Wiley, New York.
Chemoff, Herman (1952). A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. Ann. Math. Statist. 23, 493-507. Donsker, Monroe D. (1952). Justification and extension of Doob's heuristic approach to the Kolmogorov-Smimov theorems. Ann. Math. Statist. 23, 277-281. Doob, J. L. (1949). Heuristic approach to the Kolmogorov-Smirnov theorems. Ann. Math. Statist. 20, 393-403. Dudley, R. M. (1978). Central limit theorems for empirical measures. Ann. Probab. 6, 899-929; Correction 7 (1979), 909-911. Dudley, R. M. (1982). Empirical and Poisson processes on classes of sets or functions too large for central limit theorems. Z Wahrscheinlichkeitsth. verve Gebiete 61, 355-368.
22
Introduction: Donsker's Theorem, Metric Entropy, and Inequalities
Feller, W. (1968). An Introduction to Probability Theory and Its Applications, 3d ed. Wiley, New York.
Gine, Evarist (1974). On the central limit theorem for sample continuous processes. Ann. Probab. 2, 629-641. Hoeffding, Wassily (1963). Probability inequalities for sums of bounded random variables. J. Amer. Statist. Assoc. 58, 13-30. Kahane, J.-P. (1985). Some Random Series of Functions, 2d ed. Cambridge University Press, Cambridge. *Kolmogorov, A. N. (1933). Sulla determinazione empirica di una legge di distribuzione. Giorn. Ist. Ital. Attuari 4, 83-91. Kolmogorov, A. N. (1955). Bounds for the minimal number of elements of an s-net in various classes of functions and their applications to the question of representability of functions of several variables by superpositions of functions of fewer variables (in Russian). Uspekhi Mat. Nauk 10, no. 1 (63), 192-194. Kolmogorov, A. N. (1956). On Skorokhod convergence. Theory Probab. Appl. 1, 215-222. Kolmogorov, A. N., and Tikhomirov, V. M. (1959). s-entropy and s-capacity of sets in function spaces. Amer. Math. Soc. Transls. (Ser. 2) 17 (1961), 277-364 (Uspekhi Mat. Nauk 14, vyp. 2 (86), 3-86). Lorentz, G. G. (1966). Metric entropy and approximation. Bull. Amer. Math. Soc. 72, 903-937. Okamoto, Masashi (1958). Some inequalities relating to the partial sum of binomial probabilities. Ann. Inst. Statist. Math. 10, 29-35. Posner, Edward C., Rodemich, Eugene R., and Rumsey, Howard Jr. (1967). Epsilon entropy of stochastic processes. Ann. Math. Statist. 38, 10001020.
Posner, Edward C., Rodemich, Eugene R., and Rumsey, Howard Jr. (1969). Epsilon entropy of Gaussian processes. Ann. Math. Statist. 40, 12721296.
Shortt, Rae M. (1983). Universally measurable spaces: an invariance theorem and diverse characterizations. Fund. Math. 121, 169-176. Shortt, Rae M. (1984). Combinatorial methods in the study of marginal problems over separable spaces. J. Math. Anal. Appl. 97,462-479.
Skorokhod, A. V. (1956). Limit theorems for stochastic processes. Theory Probab. Appl. 1, 261-290. Uspensky, J. V. (1937). Introduction to Mathematical Probability. McGrawHill, New York. Vorob'ev, N. N. (1962). Consistent families of measures and their extensions. Theory Probab. Appl. 7, 147-163 (English), 153-169 (Russian).
2 Gaussian Measures and Processes; Sample Continuity
Let X1, X2, .. , be independent, identically distributed real-valued random
variables with EX1 = 0 and EX1 = 02 < co. Let S, := Xl +
+ X.
Then the one-dimensional central limit theorem says that the distribution of converges as n -+ oo to the normal distribution N(0, 02), which (if or > 0) has a density a-1(2ir)-1/2 exp(-x2/(202)) with respect to Lebesgue measure on R (RAP, Theorem 9.5.6). Also, if the X; are i.i.d. with values in 1/2 converges to Rk, EX1 = 0, and E I Xl 12 < oo, then the distribution of a normal distribution N(0, C) where C is the covariance matrix of XI (ibid.). This book is mainly about extensions of the central limit theorem to infinitedimensional situations. Here the limit distributions will be normal distributions on infinite-dimensional spaces. Since their behavior is not as simple as in the finite-dimensional case, this chapter is devoted to a study of normal or Gaussian measures.
2.1 Some definitions Let V be a real vector space. Recall that a seminorm is a function II II from X into the nonnegative real numbers such that lix + yll lixII + Ilyll for all x and y in X and Ilcxll = Iclllx11 for all real c and x E X. The seminorm 11 is called a norm if IIx 11 = 0 only for x = 0 in X, and then (X, II 11) is called 11
a normed linear space. A norm defines a metric by d(x, y) := llx - yll. A normed linear space complete for this metric is called a Banach space. As with
any metric space, it is called separable if it has a countable dense subset. A probability distribution P defined on a separable Banach space will be assumed to be defined on the Borel 0-algebra generated by the open sets, unless another 0-algebra is specified. Then P will be called a law. Let (X, 11 11) be a separable Banach space. A law P on X will be called Gaussian or normal if for every continuous linear form f E X', P o f -1 is 23
Gaussian Measures and Processes; Sample Continuity
24
a normal law on R. Recall that a law on a finite-dimensional real vector space is normal if and only if every real linear form is normally distributed (RAP, Theorem 9.5.13).
2.2 Gaussian vectors are probably not very large First, let's have some bounds for one-dimensional Gaussian variables. Let 4) be the standard normal distribution function and 0 its density function, so 0(x) = (2n)-1i2 exp(-x2/2) for all real x, and D(x) = fX,,,, 0(u)du.
2.2.1 Proposition Let X be a real-valued random variable with a normal distribution N(0, a2). Then
(a) for any M > 0, Pr(IXI > M) < exp(-M2/(2a2)); (b) if M/a > 1, then/ M.
< Pr(,XI > M) < M0
I
(M
Proof Replacing X by X/v, we can assume or = 1. For (a) we want to prove
24 (-c) < exp(-c2/2) for any c > 0. This holds for c = 0 and follows by differentiating both sides for 0 < c < (2/sr)1/2. For larger c, it follows from '1 (-c) < 0(c)lc (RAP, Lemma 12.1.6(a)), as does the right side of (b). For the left side of (b), note that 0 is a convex function for x > 1, since there 0"(x) = (x2 - 1)0(x) > 0. Thus, the region between the graph of 0 and the x axis for x > c includes a right triangle with right vertex at (c, 0) and a vertex at (c, 0(c)), and whose hypotenuse is along the tangent line to the graph of 0 at c. It's easily seen that this triangle has area 0(c)/(2c), which finishes the proof.
This section will prove an extension of inequality (a) to infinite-dimensional Gaussian variables such as those taking values in separable Banach spaces. It will be said that a law P on a separable Banach space (X, II II) has mean 0
if f Ilx II dP(x) < oo and f f (x) dP(x) = 0 for each f E X'. Recall the dual norm 11f II' := sup{ l f(x) I
:
IIx II < 1). Here is one of the main results.
Let P be a normal law with mean 0 on a separable Banach space X. For f E X' let a2(f) f f2dP. Then r2 := sup(a2(f): llfll' < 1) < oo and 2.2.2 Theorem (Landau-Shepp-Marcus-Femique)
fexp(allxll2)dP(x) < 00 for any a < 1/(2r2).
2.2 Gaussian vectors are probably not very large
25
By Proposition 2.2.1(a), the theorem holds in the one-dimensional case, and
by the left side of part (b), the conclusion fails when a = 1/(2r2). Before proving the theorem in general, some other facts will be developed.
Definition. Let X be a real vector space and B a a-algebra of subsets of X. Then (X,13) is called a measurable vector space if both
(a) addition is jointly measurable from X x X to X, and (b) scalar multiplication is jointly measurable from R x X to X (for the usual Borel a-algebra on R).
Example. Let X be a topological vector space, namely a vector space with a topology for which (a) and (b) hold with "measurable" replaced by "continuous." Suppose the topology of X is metrizable and separable. For a Cartesian product of two separable metric spaces, since their topologies have countable bases, the Borel a-algebra in the product equals the product a-algebra of the Borel a-algebras in the two spaces (RAP, Proposition 4.1.7). Thus X with its Borel a-algebra is a measurable vector space.
The notion of normal law can't be defined for general measurable vector spaces by way of linear forms, as it was for Banach spaces in the last section, since there exist measurable vector spaces, such as spaces LP[0, 1] for 0 < p < 1, which have nontrivial normal measures but turn out to have no nontrivial measurable linear forms (Appendix F). Fernique (1970) proposed the following ingenious definition.
Definition. A probability measure P on a measurable vector space (X, 8) will be called centered Gaussian if for variables U and V independent with law P (say, coordinates on the product X x X for the product law P x P) and any O with O < 0 < 2ir, U cos O + V sin 9 and -U sin 9 + V cos 9 are also independent with distribution P.
If X = JR, the transformation of (U, V) E R2 in the last definition is a rotation through an angle 0. Normal laws with mean 0 on finite-dimensional real vector spaces are centered Gaussian in this sense, as can be seen from covariances. Conversely, a law on X = JR satisfying the above definition of "centered Gaussian," even for one value of 0 with sin (20) 0, must be normal according to the "Darmois-Skitovic" theorem - see the notes for this section. We will not need the full strength of the latter theorem below, but the following will be proved.
Gaussian Measures and Processes; Sample Continuity
26
2.2.3 Proposition some a2 > 0.
A centered Gaussian law P on JR is a law N(0, a2) for
Proof P x P on JR2 is invariant under all rotations. A rotation through 6 = n
shows that P is symmetric, dP(x) - dP(-x). Let f be the characteristic function of P, f (t) := f 0 e`txdP(x). Then f is real-valued, f (0) = 1, and f (t) = f(-t). Any point (t, u) E R2 can be rotated to a point on a coordinate axis, so f(t) f(u) - f((t2 + u2)1/2). Let h(t) := log f(ItI112) where it is defined and finite, that is, where f > 0, as is true at least in a neighborhood of 0. Then h(t + u) = h(t) + h(u) for t, u > 0 and, perhaps, small enough. Where both sides are defined and finite, we have h(qu) - qh(u) first when q is an integer, then when it is rational, and then for general real q by continuity. Since h thus doesn't become unbounded on finite intervals where it's defined, it is defined and continuous on the whole line, with h(t) = ct for some constant
c = h(l), so f(t) = exp(ct2) for all t, and c < 0 since I f(t)I < 1. Thus P = N(0, a2) where a2 = -2c (RAP, Proposition 9.4.2, Theorem 9.5.1).
Given a normal measure P = N(C) on a finite-dimensional space X and a vector subspace Y of X, it follows from the structure of normal measures (RAP,
Theorem 9.5.7) that P(Y) = 0 or 1. This fact extends to general measurable vector spaces:
2.2.4 Theorem (0-1 law) Let (X, 8) be a measurable vector space and Y a vector subspace with Y E B. Then for any centered Gaussian law P on X, P(Y) = 0 or 1. Proof Let U and V be independent in X with law P. For 0 < 6 < 7r/2, let A(6) be the event
A(O) := {Ucos9+Vsin6EY, If 6
with 0 < 0 < r/2, then
cos 9 sin 0 - sin 9 cos 0 = sin(0 - 6) 0 0, so if Y1 := u cos 6 + v sin 6 E Y and Y2 := u cos 0 + v sin 0 E Y, then u and v can be solved for as linear combinations of yi, Y2, so they are in Y and -u sin 6 + v cos 9 E Y, -u sin 0 + v cos 0 E Y. So the sets A(9) are disjoint for different values of 6 E [0, r/2]. By definition of centered Gaussian, these sets all have the same probability, which thus must be 0. Taking 9 = 0 gives
0 = Pr(U E Y) Pr(V 0 Y) = P(Y)P(X\Y), so P(Y) = 0 or 1.
2.2 Gaussian vectors are probably not very large A measurable function II
27
II from a measurable vector space X into [0, oo]
will be called a pseudo-seminorm if Y := {x E X : 11x 11 < oo} is a vector subspace of X and I I I I is a seminorm on Y, that is, I I cx I I = I c I Ilx II for each real
candx E Y, and so forallx E X,with
Ilxll+llyll for all x, y E Y, and so for all x, y E X. By the 0-1 law (Theorem 2.2.4), for any pseudo-seminorm II II and centered Gaussian P on X, P(II II < oo) = 0 or 1. Likewise, P(11 II = 0) = 0 or 1. A real-valued stochastic process consists of a set T, a probability space (0, A, P) , and a map (t, w) i-+ Xt(w) from T x 0 into R such that for each t E T, Xt is measurable from 0 into R. The process is called Gaussian if for every finite subset F of T, the law G({Xt}tEF) is a normal distribution on 1(S F.
If S is a countable set, then RS, the set of all real-valued functions on S, with product topology, is a separable metric topological linear space, hence a measurable vector space. If P is the law of a Gaussian stochastic process {xt, t e S} on ]IBS, with Ext = 0 for all t e S, then P is centered Gaussian on IRS. The supremum "norm" II { yt, t E S) II := suet lyt I is clearly a pseudoseminorm on IRS.
Here is a step toward proving Theorem 2.2.2:
2.2.5 Lemma (Landau-Shepp-Femique) Let (X, 8) be a measurable vector space, P a centered Gaussian measure, and 11 II a pseudo-seminorm on X with 0. Then for some s > 0, P(Il 11 < oo) >fexp(ailxli2)dP(x)
< oo for 0 < a < E. Proof As noted above, P(11 II < oo) = 1. Let U and V be independent with distribution P in X. The definition of centered Gaussian for 6 = -ir/4 yields, for any real s < t, (2.2.6)
POI PrIII(U-V)/21/211 <S,
II(U+V)/21/211>t}.
Note that 2112min(IIUII, IIVII) > II(U + V)/21/211- 11 (U
-
V)/21V211,
where the event that the right side is undefined, equaling oo - oo, has zero probability and so can be neglected. Thus on the event on the right in (2.2.6), we have IIVII > (t - s)/21/2 and IIVII > (t - s)/21/2. So (2.2.7)
P(11
II < s)P(II II > t)
<-
P(ll
1/2 2 II > (t - s)/2).
28
Gaussian Measures and Processes; Sample Continuity
Choose s with 0 < s < oo large enough so that q := P(Il II < s) > 1/2. Define a sequence t, recursively by to := s, t,+1 := s +21/2tn, n = 0, 1, Then we have by induction
to =
(21/2
+ 1)(2(n+1)/2 -1)s.
So to increases up to +oo with n. Let xn := P(II By induction, we then have P(II
II > tn)/q, so xn < x2n_1.
it > tn) < q((1- q)lq)2n
It follows that E exp (all 112) < geas2 + E P(tn
< 11
I1
<_ to+1) exp (atn+1),
n=0
where the nth term of the latter sum is bounded above by q(q-1
- 1)2n exp [a (21/2 + 1)2(2(n+2)/2 - 1)2s2]
The sum will be finite if exp {2n [log 1 - q + 4(21/2 + 1)2as2J
n=0
l
q
00,
<
which holds for a < (log 1q)/(2452), proving the lemma. Theorem 2.2.2 will be a corollary of the following fact:
2.2.8 Theorem Let (X, l3) be a measurable vector space and P a centered Gaussian law on X. Let (yn }n>_1 be a sequence of measurable linear forms: X H R. Let Il x 11 suPn I Yn (x) 1. Suppose that P (ll x II < oo) > 0. Then r :=
(supra f yn dP)1/2 < 00, and E exp(allxll2) < oo if and only if a < 1/(2t2).
Proof For each n, II ? I Yn I. It's easily checked that P o yn 1 is centered Gaussian and thus by Proposition 2.2.3 is a law N(0, Q,2), an < t2. Now P(I Yn I > Qn) > c > 0 for all n (c > 0.3). Thus if a, are unbounded, II
P(II II > Qn) > c for all n gives a contradiction. Then to prove "only if," we have E exp(lyn l2/(2t2)) = 1/(12 - an2)1/2, or = +00 if a,, = t2. Taking the
supremum over n gives E exp(llx 112/(212)) = +00. Now to prove "if," recall the space £°O of all bounded sequences of real numbers with supremum norm. This is a nonseparable Banach space. With the smallest or-algebra making the coordinates measurable, it is a measurable vector space (by the way, this a-algebra is smaller than the Borel a-algebra for
2.2 Gaussian vectors are probably not very large
29
the supremum norm). Let Y(x) := {yn(x)}n>1. Let S be the vector subspace is finite. Then P(S) = 1 by Theorem 2.2.4 and Y is linear, of X where II II
measurable, and preserves norms from S into £°O. So it will be enough to prove the theorem in £°O with coordinates l y, }.
By Gram-Schmidt orthonormalization in L2(P) (RAP, 5.4.6) we can write yn = m I an jgj for all n, where gj are linear functions on £°O (finite linear combinations of coordinates), are orthonormal in L2(P), and are normally distributed, so they are i.i.d. N(0, 1). If the yn are linearly independent in L2(P), then m(n) = n. Otherwise, m(n + 1) = m(n) + 1 or m(n) according as Yn+1 is or is not linearly independent of the yj for j < n.
Let an j = 0 for j > m (n). Each gi is in turn a linear combination of yl, , yn for the least n such that in (n) > i.
n=1,2,...,let
For each n, f yndP=F_ja
j>k an jgj. Since an j = 0 for j > n, the sum defining Vkn runs over k < j < n and there is no problem of convergence. Let 13-k be the smallest a-algebra for which gj are measurable for all j > k. Then for any 0 < j < k Vkn :=
and n, we have Vkn = E(Vjn 113_k). Let 11 Vk II
supra I Vkn I < +00. Then for
a > 0 we have the inequalities exp (a II Vk II2) = exp (a supra I Vkn 12)
= exp (a supra I E(Vjn I8-k)12) < exp (a{E(supn I Vin I IB-k)}2) = exp (a{E(11 Vj 1118-k)}2)
< E (exp (a II Vj ll2) 18-k)
by conditional Jensen's inequality (RAP, 10.2.7) if the expectations are finite. First taking j = 0, E exp(a II Vo 112) < oo for some a > 0 by Lemma 2.2.5. Then for j = 0 < k, the inequalities hold and give for that a, E exp(a II Vk 112) <
Eexp(alIVoll2) < oo for all k. Thus for almost all y E f°°, Vk(y) {Vkn(y)}n>1 E too. Let W-k := exp(aIIVk112), k = 0, 1, .. Then by the inequalities for general 0 < j < k, {(Wj, Bj) : j = , -2, -1, 01 is a submartingale (RAP, Section 10.3) and in view of its index set, a reversed submartingale. For any s > 0 and finite k, by the Doob maximal inequality (RAP, 10.4.2),
Pr{maxo<j exp(as2)}
< EWo/ exp (as2) < 00
Gaussian Measures and Processes; Sample Continuity
30
by choice of a. Chooses large enough so that EWo exp(-as2) < 1/2. Then by letting k ---o- oo,
Pr (lim sup j, ,,, II Vj 11 > s) < Pr (sup j 11 Vj 11 > s) < 1/2. Now lim supj,,,,, II Vj 11 is measurable for the "tail a-algebra" f j B_ j. Thus
Pr(limsupj,,, IIVjII > s) = 0 by the Kolmogorov 0-1 law (RAP, 8.4.4). Then for 0 < s < 1/2, there is a k(e) < oo such that Pr(II Vk 11 > s) < s fork > k(s), since if Pr(II Vk II ? s) > s for infinitely many values of k, then Pr(lim supk..., 11 Vk II > s) > s. Then by the last line of the proof of Lemma 2.2.5,
(2.2.9)
< oo for 0 <
E(exp (pIIVk112))
24s2
log
\
£
s
/
.
Now take any a with 0 < a < 1/(2r2). Choose y with a < y < 1/(2r2), then e > 0 small enough so that
log((- s/s)
ay
(2.2.10)
(y 1/2 - a1/2)2 -
24s2
Let k = k(s) and Uk(y) := y - Vk(y). Then al/211Y11
a'1211Uk(Y)II +a'/2II Vk(Y)II
(a/Y)112y112IIUk(Y)II + (1Since t i-+ exp(t2) is a convex function (RAP, Section 6.3, see Problem 1), exp(ally1I2)
< (a/Y)1/2eXp(YIIUk112)
+ (1- (a/Y)'/2) exp ( II Vk II2). By (2.2.9) and (2.2.10), E exp( II Vk 112) < oo. Now, by the Cauchy inequality, for each n, )2
k 2
Ukn =
(
(;gi2)(a;i) k
anigi
k
\
j=1
so
k
E( exp (Y II Uk II2))
E exp
k
g?
y
j=1
i=1 k
< Eexp
{(g) t2
l
(1 - 2yr2)k/2 < oo. 111
So Theorem 2.2.8 is proved.
a
supra
2.3 Inequalities and comparisons for Gaussian distributions
31
Proof of Theorem 2.2.2 Let {x,, }n 1 be dense in X. For each n, by the HahnBanach theorem and a corollary (RAP, 6.1.5), there is a yn E X' with II Yn II ' = 1 and I Yn (xn) I = II xn I I . Then for all x E X, we have sup,, l y, (x) l = I I x 11, so the theorem follows from Theorem 2.2.8.
2.3 Inequalities and comparisons for Gaussian distributions The main result of this section will show that if a set of Gaussian random variables is large enough in the sense of metric entropy (as defined in Section 1.2), meaning that the number of variables more than s apart grows rather fast as s 4. 0, then it is almost surely unbounded. The main steps in the proof will be some inequalities, one due to Slepian and another to Sudakov and Chevet.
2.3.1 Slepian's inequality. Let X1, , Xn be real random variables with a normal joint distribution N(0, r) on Rn. Let Pn(r) := Pr{Xj > 0 for all j = 1, , n}. Let q be another covariance matrix, with r = q;; = 1 for all , n. If ri j > qi j for all i and j, then Pn (r) > Pn (q). i = 1, Remarks. Since each Xj has distribution N(0, 1), clearly P, (r) < 1/2, with Pn (r) = 1 /2 when r, j = 1 for all i and j. On the other hand, Pn (r) = 0 if rl j = -1 for some i j. Proof First, suppose that r is nonsingular, so that it is strictly positive definite and N(0, r) has a density gn given by Fourier inversion of its characteristic function as oo
gn(xl,...,xn)
:=
gn(xl,...,xn;r) = (2n)-n Jt n
eXp(-ixjtj-2
n
j=1
k,m=1
1
00
...
t
J
oo
-o0
rkmtktm dt1...dtn
(RAP, Theorem 9.5.4). Since r is symmetric, it is given by the n(n + 1)/2 variables rkm, 1 < k < m < n. The partial derivatives a2gn/axkaxm can be evaluated by differentiating under the integral signs, applying Corollary A.15 (in Appendix A) twice, thus multiplying the integrand by -tktm (RAP, Theorem 9.4.4). The same integral results from taking ag,,/arkm for k < m, where a/arkm can be taken under the integral sign by Proposition A.16 (since r is strictly positive definite). So (2.3.2)
a2gn/axkaxm = agnlarkm,
k
m.
Gaussian Measures and Processes; Sample Continuity
32
Now
r0 P,, (r) =
f00(2.3.3) xn)dxI...dxn, gn(Xi,...
J
0
J0
where g, (x) = (27r)-n/2(detr)-1/2 exp((-r-lx,x)/2) (RAP, 9.5.8). For a positive definite symmetric matrix s, by Proposition A. 16, applied to t = sib
and i(x) = xixj (or xi /2 if i = j), the integral of exp(-(sx, x)/2) over any measurable region in Rn, specifically the positive orthant {0 < xi < 00, i = , n } , can be differentiated under the integral sign with respect to any com-
1,
ponent of s. Then, since the functions r H r-1 and r i-+ (det r) -1/2 are smooth for r in the set of symmetric, (strictly) positive definite matrices, the integral (2.3.3) can be differentiated under the integral sign with respect to any rkm , k <
m. Then by (2.3.2), we need to evaluate fo ... fo a2gn/axkaxm dx1 ... dxn. Since the integrand is absolutely integrable, we can do the integrations in any order (by the Tonelli-Fubini theorem) and replace fo f( dxk dxm by limn", J0M f0M dxk dxm. Here we may as well assume that k = 1, m = 2. Now, since gn is a smooth function,
M M a2gn/ax18x2dxidx2
f0
0
gn (M, M, x3, ... , xn) - gn (M, O, x3, ... , xn) -gn(O,M,X3,...,x,)+gn(0,0,X3,...,xn)
gn (O, O, x3 , ... , xn ) as M
oo. Thus
aPn(r)1ar12 =
o o J 0 ... f o
> 0.
Likewise, aPn(r)/arkm > 0 for all k < m. In the general case, let s > 0 and 0 < ,l < 1. Let I be the identity matrix and p := Ar + (1 - A)q + cI, which is positive definite. Then dPn(p)
dl
_
aP(p) dpkm
_
8P-(p) (rkm - qkm)
k<m
apkm
d),
k<m
0.
apkm
Integrating from 0 to 1 with respect to A gives P,, (r + cI) > P, (q + cI). The laws N(0, r + cI) converge to N(0, r) as e 0. (To prove this, let X and Y be independent with G(X) = N(0, r) and G(Y) = N(0, I). Then G(X + cY) = N(0, r + eI) and ass 1 0, X + cY X a.s., hence in law.) Likewise, N(0, q + cI) converges to N(0, q). Since the rkk and qkk are
2.3 Inequalities and comparisons for Gaussian distributions all equal to 1, the marginal distribution of each Xk, k = 1,
33
, n, is N(0, 1) and
P(Xk = 0) = 0. Thus the boundary of the orthant {Xk > 0 for k = 1, , n} has zero probability for both N(0, r) and N(0, q). Then, by the portmanteau theorem (RAP, 11.1.1(d)), it follows that P,,(r) > Pn(q).
Example. Let r, j = 1, i, j = 1, 2, 3, and let q be the 3 x 3 identity matrix. Let s = r except that S13 = S31 = 0. Then s is not nonnegative definite, in other words it is not a covariance matrix, because sij = 1 for {i, j} # {1, 3} implies X1 = X2 = X3 a.s. with a N(0, 1) distribution, while S13 = S31 = 0 implies X, and X3 are independent. Thus, in the preceding proof, if we change rkn, to qkm for one pair of indices
k, m at a time, it is possible to pass through values of the matrix which are not positive definite. Then, the "normal density" gn (xl, , xn , r) is not welldefined, and integrals of its "characteristic function" diverge. Recall that the correlation (coefficient) r(X, Y) of two nonconstant variables X and Y with finite second moments is defined by
r(X, Y) := E((X - EX)(Y - EY))/(Qxay), where Qx := or (X) is the standard deviation (E(X -
EX)2)1/2.
2.3.4 Corollary Let X1,
, X n and Y1, , Yn be two sets of jointly normally distributed variables with mean 0, or (X1) > 0 and or (Y,) > 0 for all i,
andr(Xi,Xj)>r(Y,,Yj)foralli Pr{X,>0,
Pr{Y,>0,
Proof Replacing each X, by X, /a (X,) and Y, by Yi /o (Y,) does not change the events being considered, or the correlations, and gives covariances to which Slepian's inequality applies.
Let C be a set of Gaussian random variables with mean 0. On C, we take the G2 metric d(X, Y) := (E(X V)2)1/2. Recall that fore > 0, D(e, C) :_ D(E, C, d) := sup{n : f o r some X1, , X, E C, d(X,, Xj) > e, 1 < i < j < n}. In the following, by Theorem 1.2.1, D(E, C) could be replaced equivalently by N(E, C, d). The following theorem and lemma are sometimes called the Sudakov minoration.
2.3.5 Theorem (Sudakov-Chevet) If lim sup,lo E2 log D(E, C) = +oo, then SUP( IXi : X E C) = +oo almost surely.
Gaussian Measures and Processes; Sample Continuity
34
Proof Let 4) be the standard normal distribution function and G(x) := 1 4)(x). Most of the proof is in the next inequality.
2.3.6 Lemma (Sudakov-Chevet) Let X1, , X, be jointly normally distributed variables with mean 0 such that for some M < oc and some a with 0 < e < 1, we have d (0, Xj) < M for all j and d (Xi, Xj) > e for all i j. Let K := 21/2(M2 + 1). Then
G(1) Pr{Xj < 1 , j = 1, < 2-n-1 +
, n} oo
(2n)-1/2 f
exp
(- t2/2)4>(Kt/e)ndt.
0
Proof Let e1, , en be orthonormal variables such that X1, , Xn are in their linear span, by the Gram-Schmidt process (RAP, 5.4.6). Then e1, , en are i.i.d. N(0, 1). Let en+1 be another N(0, 1) variable independent of e1, Let Y1 := Xi - en+1, i = 1, , n. Then
G(1) Pr{Xj < 1 , j = 1,
, en.
, n}
= Pr{en+l > 1 and Xj < 1, 1 < j < n}
< Pr{Yi <0, i = 1, ,n}. Let bi3 := r (Yi ,
Y j) and II Y II
d (O, Y). Let Bid be the angle between Yi and
Yj at 0, so that bid = cos(61 ). Let U := Xi and V := Xj with i # j. Then
u:=IIUJI <M, v:=IIVII <M,andIIU-VII>s.So 2
< IIU-VII2 = u2-2(U,V)+v2,
and 2(U, V) < u2 + v2 - e2. Thus
[(U,V)+1]2 = (U, V)2+2(U,V)+1 u2v2 + u2 + v2 + 1
- e2,
so
bi
<
(U,V)+1 =
[(u2 + 1)(v2 + 1)]1/2
<
1-
E2
L1-
1/2
(u2 + 1)(v2 + 1)
L
E
2(M2 + 1)2
= 1 - E2/K2 < 1/(1 +e2/K2).
Let f := eei/K - en+1, fi := r(f;, f) = 1/(1 + E2/K2) for i # j, bii = f i := 1. Then Slepian's inequality (above), specifically Corollary 2.3.4,
2.3 Inequalities and comparisons for Gaussian distributions
35
gives
Pr{f <0,
Pr{Y; <0,
i= 00
_ (2n)-1/2J
Pr{e; < tK/e, i = 1,
, n} exp
( - t2/2) dt
00 00
(27r)-1/2
_
fi(tK/s)n exp (- t2/2) dt f00 00
_
(2n)-1/2
f
exp o
< 2 -n-1 + (27r )-1/2
(- t2/2)[`)n(Kt/s) + (pn(-Kt/E)] dt 00
f
exp (- t2/2)q)n(Kt/E) dt,
o
proving the lemma. Now to prove Theorem 2.3.5, take Ek 0 such that Ek log D(Ek, C) > k, k = 1, 2, . In Lemma 2.3.6, let E = Ek and n = D(Ek, C). If the variances of the
variables in C are unbounded, let X(n) E C with or (X(n)) > n. Then clearly I X(n) I are unbounded a.s. So we can assume that for some M < oo, a (X) <
MforallXEC. For 1 < B < oo, the probability that IX; I < B for j = 1,
, n is the probability that J Xj/ B I < 1 for each j. To apply Lemma 2.3.6, we can then replace Ek by Ek/B, giving the same bound except for replacing Kt/Ek by s/Ek where s := Kt B.
Since cn = (1 - G)n < e-nG, it will be enough using the dominated convergence theorem to show that D(Ek, C)G(s/Ek) - +oo as k -+ oo for every s > 0. As x -+ +oo, G(x) (27r)-1/2x-1 exp(-x2/2) (RAP, Lemma 12.1.6), so for k large, G(s/Ek) > Ek(3s)-1 exp(-(s/Ek)2/2). So it will be enough to prove that log D(Ek, C) + log Ek - cEk 2 - +oo as k -+ oo for any c < oo. Now log E > -E-2 for small E > 0, so the log Ek term can be removed. By assumption, Ek log D(Ek, C) - c on multiplying by Ek2 -). +oo also.
+oo, and the result follows
Example. Let G,, be i.i.d. N(0, 1) variables and X,, := Gn/(logn)1/2, n > 2. Then d(Xj, Xk) > (log j)-1/2 for j < k, so D(E, {Xj}2<j n - 1
if E < (log(n - 1))-1/2. So for 0 < E < 1, D(E, {Xj}j>2) > [exp(E-2)] > exp(E-2)/2, where [x] denotes the greatest integer < x. So log D(E, (XjJ f>2) > E-2 - log 2 > E-2/2 for 0 < E < 1/2. On the other hand for each n, Proposition 2.2.1 gives Pr{ I Xn I > 2} < exp(-2log n) = 1 /n2 so by the Borel-Cantelli
Gaussian Measures and Processes; Sample Continuity
36
lemma, lim sup,,,,,,, I Xn I < 2 and sup,, I Xn I < +oo a.s. Thus Theorem 2.3.5 is sharp. Here is another comparison of Gaussian laws.
2.3.7 Theorem Let N(0, C) and N(0, D) be two normal measures with mean 0 on R'. Let X = {X}i_1 have law N(0, C) and let Y = {YY}n-1 have law N(0, D). Suppose that for all i, j = 1, , n, we have E((Y1 - Yj)2) < E((Xi - Xj)2), in other words for each i, j, (2.3.8)
Di1+Djj-2D1j < Ci1+Cjj-2C11.
Then
(a) E{maxi
for any t > 0, C + t21 and D + t2I are nonsingular and the hypotheses hold for them, thus the conclusion. Now, N(0, C + t2I) is the law of X + t Z where Z has law N(0, I) and is independent of X, and likewise for D and Y. Then, letting t 4. 0, the theorem follows for C, D. Suppose then that C and D are nonsingular. For 0 < A < 1, let CA AC + (1 - A)D and let g(A) := E maxl 0 for 0 < A < 1, taking a right derivative at 0 and a left derivative at 1. We have g(A) = f maxl
Since C and D are nonsingular, they are strictly positive definite. Thus for
some y > 0, (C)'y, y) > y Iy12 for all y and 0 < A < 1. Thus det CA is bounded below and (det Cx)-1/2 is bounded above for 0 < a, < 1. Here y can also be"chosen so that (Cj 1x, x) > yIx12 for all x E R' and 0 < A < 1. The mappings A r-a CA and ), -* Cj 1 are both smooth (C°O), and the map (C, x) i-+ (det C)-112 exp(-(C-1x, x)/2) is a smooth map on the open set of strictly positive definite symmetric matrices C and all x E R. It follows that the functions fx(x) and their difference-quotients and first partial derivatives with respect to A and x are uniformly integrable on R" with respect to x and remain so if multiplied by any function bounded above in absolute value by a polynomial. Thus by Theorem A.7 of Appendix A, we can differentiate under
2.3 Inequalities and comparisons for Gaussian distributions
37
the integral sign: (2.3.9)
dg(?) =
fmaxi
As in (2.3.2), we have a2gn/axk = 2agn/arkk for k = 1,
, n. By this and
(2.3.2), we have (2.3.10)
afx(x)
_
d(Cx)rj a2fx
1
dk
2 ,j=1
aa,
axi ax1
1,
where d (Cx)i j /d k = Ci j - D, j. Let S := S(118n) be the space of all C°O functions f from 118" into R such that for every polynomial and every n-tuple p:= (pl, , pn) of nonnegative integers with IPI P1 + + pn, sup, I P(x)DP f(x)I < oo where DPf(x) := alPI f(x)/axr axP". After +X,2 , we multiplying P(x) by some power of 1 + Ix12 where Ix12 := xi + see that each P(x)DPf(x) is integrable on 118n. Here S(118n) is Laurent Schwartz's space of rapidly decreasing functions. It is easily seen that any normal density such as fx is in S(R'). Let u(x) := maxi xi and for each i = 1, , n, let v, := vi (x) maxj#i xj. Let dx/dxi := n j#,dxj. For any i j, define a function gij by gij((xr)r#i) := {yr}r=1 where yr := xr for r i and y, := yj. Let Vi j be the measure defined for suitable functions on W1 by f cpdV, j fA(j)O(g,j({xr}r#,)dx/dx,where A(j) is the set of{xr}r,6i such thatxr <xj for all r i, or equivalently vi = xj. In other words, Vi j is a measure on the subset where xr < xi for all r of the (n -1)-dimensional hyperplane (xi = x j ), given by dx/dx, or equivalently by dx/dxj. On A (j), as an image by a linear map gij of Lebesgue measure on Rn-1, V j is a multiple (by 2-1/2) of Lebesgue
measure on the hyperplane {xi = xj }. Thus, Vi j = Vj, for 1 < i < j < n.
2.3.11 Lemma For any 0 E S(118n),
(a) For any i = 1, , n, f u(x)(a(k/ax1) dx = - fx,.>v, . cb(x) dx. (b) For any i 0 j with 1 < i, j < n, (2.3.12)
f u(x)(a2q5/ax,axj)dx
(c) For each i = 1, (2.3.13)
= -J
dVij.
, n,
f u(x)(a2cb/ax?)dx =
f
O(x)Ix;-v; dx/dx,.
Gaussian Measures and Processes; Sample Continuity
38
Proof For (a), we have f u(x)(a¢/axi) dx = f (u - vi)(x)(a¢/axi) dx since vi doesn't depend on xi. Integrating with respect to xi first, we get °O
v;(x)
(xi - vi)
dx dxi-. dxi axi a¢
Integrating by parts in the inner integral, the boundary terms are 0 both at xi = vi and at xi = oo since 0 E S, so (a) follows. Then to prove (b), apply (a) to ao/axj. Integrating first with respect to xj, (b) follows. For (c), applying (a) to a0/axi and integrating first with respect to xi give (c).
Now continuing the proof of Theorem 2.3.7, note that we have
maxi, j(Xi - Xj) = maxi Xi - minj Xj and
minj X, = - max(-Xj). Normal laws with mean 0 are symmetric, so same law. Thus
and
have the
Emaxi,j(Xi-Xj) = 2Emaxi Xi.
(2.3.14)
Then, from equations (2.3.9), (2.3.10), (2.3.14), (2.3.12), and (2.3.13), we get
dg
_
dl
(Dij 1
/
- C; j) If
1 / dx + 2 L.(Cii - Dii) J fxx)Ix,=v; dx
i=1
`
Now, for Lebesgue measure dx/dxi on {xj} j#i, each set {xj = x,.} for j # i ¢ r # j has measure 0, and thus E ji 1,,;=xj = 1 almost everywhere. So
f fx(x)lx,=v,dx/dxi = J fidVij, JTi and
dg da
12 EE(Dij -Cij +Cii - Dii) i=1 joi
f fAdVij
Symmetrizing the sum, interchanging j and i, gives
dg dX
_
n
1: E(Cii+Cjj+2Dij-2Cij-Dii-Djj)f dVij > 0 4 1
i=1 jai
2.3 Inequalities and comparisons for Gaussian distributions
39
by (2.3.8). Thus (a) of Theorem 2.3.7 is proved. Then, since (2.3.14) also holds for Y, (b) follows and Theorem 2.3.7 is proved.
Next, bounds for expected suprema will be developed, closely related to , Z,) be the the Sudakov-Chevet theorem (2.3.5). Let Mn := max(Zi, maximum of n i.i.d. standard normal variables Zi. Then EMn is bounded below as follows.
2.3.15 Lemma For all n > 1, EMn > (log n)1/2/ 12. Remark. The constant 1/12 can be improved to (n log 2)-1/2, by a less elementary proof (Fernique, 1997, (1.7.1) and references given there).
Proof For n = 1 we have 0 > 0. For n = 2, EM2 > (log 2) 1/2/ 12 can be found by a direct calculation (Problem 1). The following proof is for n > 3. Let a := (87r)-1/2. By Proposition 2.2.1, we have
P(Z > (logn)1/2) >
e-logn/2
2(2n logn)1/2
= a(n logn)-1/2 >
n
where log n < n for all n > 1 since x < ex for all x. Now, by its Taylor series,
log(1 - x) < -x for 0 < x < 1, so et)n
= exp(nlog(1
n11 <
exp(n(-n11 = e-".
Then P(Mn > (logn)1/2) > 1 - e-" > 0.18. Recall that for any realvalued function f, we let f + := max(f, 0) and f - :_ - min(f, 0). Thus 0.18(log n) 1/2.
Next, -Mn- is nondecreasing as n increases, so for n > 3, E(-Mn)E(-M;-). Also, M3 = 0 unless Z1, Z2, and Z3 are all negative, which has probability 1/8. We have E(-M3 IM3 < 0) > E(Z1IZ1 < 0) = -(2/,-r)1/2. Thus E(-M3) > -(2/ir)1/2/8 > -0.1. It follows that EMn > (log n)1/2/ 12 for all n > 3 and so for all n > 1. Recall (RAP, Theorem 12.1.4) the isonormal Gaussian process L on a separable, infinite-dimensional Hilbert space H: for any x E H, L (x) is a Gaussian random variable with distribution N(0, lix 112), and for any xl , , xn E H, (L (x 1), , L (xn)) have a jointly normal distribution with covariance given by the inner products (xi, x j ). To end this section, here is a lower bound based on packing numbers.
40
Gaussian Measures and Processes; Sample Continuity
2.3.16 Theorem For any countable subset S of a Hilbert space H with its usual metric, and any s > 0, E supXEs L(x) > 17s(log D(s, S,
Remark.
d))1/2.
The constant 1/17 can be improved to (2n log 2)-1/2 (Fernique,
1997, Theorem 4.1.4).
Proof Fix any E > 0 and let m := D(E, S, d). Take m points xl, , xm E S j and i.i.d. N(0, 1) variables Zi. Let Vi with Ilxi - xjII2 > E2 for i
EZi/21/2. Then E(Vi - Vj)2 = E2 < E((L(xi) 1,
, m, i # j.
- L(xj))2) for all i, j =
It follows then from Lemma 2.3.15 and the comparison
theorem 2.3.7 that
EsupXEsL(x) > Emaxl Emaxl (E/2112)EMm
> (E/21/2)(log m)1/2/12 > E(logD(e,S,d))1/2/17, which finishes the proof.
2.4 Gaussian measures and convexity There are several useful inequalities about normal measures and convex sets. Convex sets were treated in RAP, Sections 6.2 and 6.6.
A set C in a vector space is called symmetric if -C := (-x: x E C} = C.
A function f is called even if f(-x) = f(x) for all x. Thus the indicator function of a set is even if and only if the set is symmetric. For sets A, B in a vector space and a constant c, let cA := {ca: a E Al and
A+B:={a+b: a E A, bEB}. 2.4.1 Theorem Let C be a convex, symmetric set in Rk. Let f be a nonnegative, even function in £1(Rk, 8, V) where V is Lebesgue measure ,kk and 8 is the Borel cr-algebra. Suppose that for every t > 0, Ki := {x : f (x) > t} is convex. Then for 0
f
c
f(x + y) dx.
> Jc
Proof Since f is integrable, both sides of the stated inequality are finite. First
suppose that f is the indicator function of a set K. Then K is convex and
2.4 Gaussian measures and convexity
41
symmetric. We need to prove (2.4.2)
V (C n (K - ay)) > V (C n (K - y)).
For these measures even to make sense, we need to take care of a measurability
problem. Not all convex sets are Borel sets: for example, in R2, the open unit disk together with an arbitrary subset of the boundary unit circle is always convex. But we do have the following fact.
2.4.3 Lemma Any convex set D in Rk is Lebesgue measurable and its boundary 8D has measure 0, V (8D) = 0.
Proof Either D is included in some (k - 1)-dimensional hyperplane, in which
case V (D) = 0 and we are done, or D has a nonempty interior U (RAP, Theorem 6.2.6). Then D and U have the same closure and boundary (RAP, Proposition 6.2.10; also, any point of D is a vertex of a k-dimensional simplex
included in D). Let p E U. By translation we can assume p = 0. A straight line L through 0 can intersect 8U in at most two points: suppose 0, q, r are on L in that order, q r, q, r E 8U. Now D includes a neighborhood W of 0, W = {x : Ixl < 81 for some 8 > 0. Let X := IRI / Irl. Take points r E D with r -+ r. Then the convex combinations Ar + (1 -,l)w, w E W, yield all points in a neighborhood of q, so q E U, a contradiction. Then by spherical coordinates (RAP, Section 4.4, problems 8 and 9), V (8 D) = 0, so D is Lebesgue measurable.
Now continuing the proof of Theorem 2.4.1, let A := (1 + a)/2, so that
A(-y)+(1-A)y=-ay,and 1/2
C n (K-ay) D A(Cn(K-y))+(1-A)(Cn(K+y)) because C is also convex. Thus
V(Cn(K-ay)) > V{A(Cn(K-y))+(1-),)(Cn(K+y))). Since both C and K are symmetric, C n (K - y) = -{C n (K + y)) and V (C n (K - y)) = V (C n (K + y)). If C and K are compact, then the BrunnMinkowski inequality (RAP, 6.6. 1 (b)), with A = Cn(K-y), B = Cn(K+y) and still A = (1 + a)/2 gives (2.4.2) as desired. Recall that if U is any open set in a metric space and r > 0, with B(x, r) (y: d(x, y) < r), then U,.:= {x E U : B(x, r) C U) is always a closed subset of U. It is easily seen that if U is convex and/or symmetric, so is U, for any r > 0. Thus any open (symmetric) convex set is the union of an increasing
42
Gaussian Measures and Processes; Sample Continuity
sequence of compact (symmetric) convex sets Kn := {x E Ul/n: IxI < n}. Thus for any convex set L in Rk, by Lemma 2.4.3, there exist compact convex sets L (n) such that 1L(n) T 1L almost everywhere for Lebesgue measure. Applying this to the different convex sets L = A, B shows that (2.4.2) holds for any convex symmetric sets C and K with V (K) < oo, in which case V (K) = 0 or K is bounded.
Next suppose f is bounded. Then we can assume 0 < f < 1. Take simple functions approaching f, namely n-1 k
E
.fn
k=0
ri
n
1
1{k/n 1{f>j/n}, j=1
since k/n < f < (k + 1)/n if and only if f > j/n for exactly k values of j > 1. So by linearity and (2.4.2), the conclusion holds for fn for each n. By dominated convergence, it then holds for any bounded f Then if f is unbounded, let gn := min(f, n). Then the result holds for gn for each n and gn f f, so by monotone convergence, it holds for f
2.4.4 Theorem
Let X be a random variable with values in Rk whose law has a density f satisfying the hypotheses of Theorem 2.4.1. Let Y be any other JRk-valued random variable independent of X. Then for any convex, symmetric
set C and 0 < a < 1, Pr{X + aY E C} > Pr{X + Y E C}. Proof For P := L(Y), by Theorem 2.4.1, with -y infJ place of y,
Pr{X+aY E C) = ff-ay f(x)dxdP(y) >
ffz
f(z-ay)dzdP(y)
- y) dzdP(y) = Pr(X + Y E C).
E]
2.4.5 Corollary Let Z and X be random variables with values in Rk, L(Z) = N(0, A) and L(X) = N(0, D), where D and A - D are nonnegative definite. Let C be a convex, symmetric set in Rk. Then
Pr(X E C) > Pr(Z E C). Proof LetYbeindependentofXwithL(Y) = N(0, A-D). Then L(X+Y) L(Z), and Theorem 2.4.4 for a = 0 gives the result.
2.5 The isonormal process: sample boundedness and continuity
43
2.4.6 Corollary Let X1,
, X and Y1, , Y both be jointly Gaussian with mean 0 and such that { EY; Yj - EX; Xi} 1 <,1
for any M, Pr {maxl<j M} < Pr {maxi<j M}. Proof Corollary 2.4.5 applies, taking C :_ {t E ]Rn : maxi I tj I < M}, a convex, symmetric set, then taking complements. For any set C in a vector space V, and a vector space W of real linear forms
on V, the polar of C is defined by C*1 := {w E W : w(x) < 1 for all x e C}. If C is symmetric in V, then C*1 is also symmetric in W and equals {w E W: Iw(x)I < 1for all x EC}. For V = Rk, W will be understood to be Rk also, defining linear forms via the usual inner product. Recall the standard normal law N(0, I) on Rk. 2.4.7 Corollary Let A be a linear transformation from Rk into itself with norm II A II < 1 (for the usual Euclidean norm 11 II on Rk), where 11 A 11 := sup{ II Ax II
:
Ilxli < 11. Then for any symmetric subset C of Rk, N(0, I)(A(C)*1) > N(0 I)(C*1). Proof If G(G) = N(0, I), then G E A(C)*1 means (G, Ax) < 1 for all x E C, for the usual inner product, or equivalently (AtG, x) < 1 for all x E C. Now G(AtG) = N(0, At A), where I - AtA is nonnegative definite, and N(0, I)(A(C)*1) = N(0, AtA)(C*1). Since C*1 is a convex, symmetric set, Corollary 2.4.5 applies and gives the result as stated.
2.5 The isonormal process: sample boundedness and continuity Two stochastic processes {Xt, t E T} and {Yt, t (= T} defined on the same index set, but possibly on different probability spaces, will be said to have the same laws, or one will be said to be a version of the other, if for each finite subset F of T, {Xt, t E F} and {Yt, t E F) have the same law. If Xt and Yt are defined on the same probability space, then one is said to be a modification of the other if for each t E T, we have P(Xt = Yt) = 1. On the relationship of versions and modifications, especially for the isonormal process L restricted to a set, see Appendix J. (L was defined just before Theorem 2.3.16.) For any subset A C H, L(A)* will be defined as ess.supXEA L(x), the smallest random variable Y such that Y > L(x) a.s. for all x E A, where
44
Gaussian Measures and Processes; Sample Continuity
Y is determined up to a.s. equality. Here "ess.sup" stands for "essential supremum." Similarly let IL(A)I* := ess.supXEA IL(x)I.
2.5.1 Lemma For any subset A C H, L (A)* and I L (A) I * are well-defined up to almost sure equality. Proof If B is countable, then clearly L(B)* exists and equals SUNXEB L(x).
Let B be any countable dense subset of A. Let Y := L(B)*. Then Y is measurable. If U is any random variable such that U > L (x) a.s. for all x E A,
then clearly U > Y. For each x r= A, there is a sequence yn E B such that Ilyn - x112 < 1/n2 for all n = 1, 2, , and then by Chebyshev's inequality and the Borel-Cantelli lemma, L(yn) -k L(x) a.s. Thus L(x) < Y a.s., and as seen above, Y is the smallest random variable with this property, so L (A)* is well-defined and equals Y a.s. The proof with absolute values is the same.
2.5.2 Lemma Let B be a linear operator from H into itself with JIB 11 < 1 and C a (nonempty) subset of H. Then for any t > 0,
Pr{IL(BC)I* < t} > Pr{IL(C)I* < t}.
Proof If t = 0, then the right side is 0 unless C = (0). Then both sides of the inequality equal 1 and it holds. So assume t > 0. First suppose C is finite. We have I L (C) I * < t if and only if I L (C/ t) I * < 1. Let V be the linear span of C, a finite-dimensional Hilbert space. A version of L restricted
to V is given by L(v) = (v, w) for the given inner product where w has distribution N(0, I) on V. Since the joint distribution of L(x) for X E C is uniquely determined, Corollary 2.4.7 for the set {±x/t : x e C} gives the result.
In general, let finite sets Fn increase up to a countable dense set in C. Then I L (C) I * < t if and only if I L (Fn) I * < t for all n (except possibly on a set of 0 probability) so
Pr{IL(C)I* < t} = limn,, Pr(IL(Fn)I* < t) < limn,00 Pr(IL(BFn)I* < t) = Pr{IL(BC)I* < t}.
Definitions. A set C in H is called a GB-set if I L (C) I * < oo a.s. Also, C will be called a GC-set if it is totally bounded and the restriction of L to C can be chosen so that each of its sample functions x H L (x) (w), X E C, is uniformly continuous on C.
2.5 The isonormal process: sample boundedness and continuity
45
A function f from a set C in a vector space V into R will be called prelinear , cn E C and a 1, , an e R such that a I cl + + a, cn = 0, if for any cl, we have aI
=0.
A GB-set must be totally bounded (by the Sudakov-Chevet theorem 2.3.5). Since a uniformly continuous function on a totally bounded set must be bounded, every GC-set is a GB-set.
2.5.3 Lemma For any prelinear function f on a set C in a real vector space V into R, let g(xlcl+...+xnCn) :=
x1f(cl)+...+xnf(cn)
for any xl, , xn E R and cl, , cn E C. Then g is a well-defined linear function from the linear span of C into R which extends f. Such an extension exists if and only if f is prelinear. Proof To show that g is well-defined, suppose x1cl +
+xncn = yldi +
Y1,..., E C and "' + ymdm for some Ym ER.Then 0, so since f is prelinear, x1 f(x1) + ... + xn f (cn) - yl f(dl) - ... - ym f(dm) = 0, the
apparent candidates to be defined as g(xlcl + + xncn) are equal, and g is well-defined. Then g is clearly linear and extends f. On the other hand if f is not prelinear, suppose x 1 cl + +xncn = 0 but xi f(c1) + +xn f (cn) 0.
Then x1c1 = -x2C2 - ... - xnCn but xl.f(cl)
-x2f(c2) - ... - xn.f(xn),
so f has no linear extension to the linear span of C. Now, a finite-dimensional projection (fdp) will be an orthogonal projection (RAP, end of Section 5.3) of H onto a finite-dimensional subspace of H. For a sequence {nn } of such projections, nn f I will mean that the range of nn is included in that of nn+1 for all n and that the union of all the ranges is dense in H. Since 7rnx is the nearest point to x in the range of nn (RAP, Theorems 5.3.6 and 5.3.8), it follows that 1I Jrnx - x 1l -+ 0 as n - oo for all x E H. For any orthogonal projection jr, let jr-'- := I - ir, the orthogonal projection onto the orthogonal complement of the range of jr (RAP, 5.3.8).
2.5.4 Lemma
Whenever fdp's irn T I, there is an orthonormal basis of H
which includes an orthonormal basis of the range of 7rn for each n.
Proof Begin with an orthonormal basis of the range of 7rI, then recursively given an orthonormal basis of the range of nn, adjoin an orthonormal basis
46
Gaussian Measures and Processes; Sample Continuity
of the range of 7rn+l - nn. For each n, 7tn+1 o nn = 7rn = 7rn o 7rn+1, so Jn 7rn+1 = 7tn+17tn = 7rn+1 - 7rn Also, 7rn+1 = 7rn+1 o (I - 7rn + 7rn) _ (nn+1 - nn) + nn, so the range of 7tn+1 is the direct sum of two orthogonal subspaces, ran(7rn+1 - 7rn) I ran(7rn). Taking the union of the bases over n gives an orthonormal set whose linear span is dense in H and thus is a basis (RAP, Theorem 5.4.9). If C is a totally bounded set in a metric space, then the set V1 of all uniformly continuous real-valued functions on C is a vector space. For any real function f on C, let II f II c := sup{ I f (x) 1: x E C}. Then VI with norm II IIc is naturally isometric to the space C(K) of all continuous functions on the completion K of C, where K is compact, so C(K) is separable for the supremum norm (RAP, Corollary 11.2.5). Let C C H and let V2 be the set of prelinear elements of V1. Each element
h of H defines a function on C by x i-+ (x, h), x E C. Let He be the completion of H for II IIc. Note that each element of He naturally defines a uniformly continuous, prelinear function on C as a uniform limit of uniformly continuous, prelinear functions. Let V3 be the set of functions on C so defined. Then V3 c V2. (Often, V3 = V2, but whether V3 = V2 in all cases will not be settled here.) Let V be a set of functions on C. Say that L on C can be realized on V if
there is a probability measure t on V such that the process (v, x) H v(x),
v E V, X E C, has the joint distributions of L restricted to C: for any x1,
, x, in C, v i-* v(xi) are jointly Gaussian with mean 0 and covariances
(xi, xj), i, j = 1, ... , n. From the definition, t would be defined on the smallest or -algebra Bc making
all evaluations v i-- v(x) measurable for x E C. If D is a countable dense set in C, V is a set of continuous functions on C, and v, w E V, then
Ilv - wIIc := sup{I(v - w)(y): y E C) = sup{I(v - w)(y): y E D}, so v H II v - w II c is BC measurable for w fixed. If also V is a set of bounded functions on C, separable for II IIc, as V1, V2, and V3 are, then all open sets for the II IIc topology are in 13c (RAP, Proposition 2.1.4 and its proof), so l3c equals the Borel or-algebra. Given a set A in a vector space, the symmetric convex hull of A is the smallest
convex set including A and -A = {-x : x E A), and is the set of all finite ai E A U -A, with a,i > 0 and i=1 X, = 1, convex combinations for all positive integers n. The closed symmetric convex hull of A for some topology (in this case the Hilbert norm) is the closure of the symmetric convex hull. Here is a set of characterizations of GC-sets:
2.5 The isonormal process: sample boundedness and continuity
2.5.5 Theorem
47
The following are equivalent for a totally bounded set C
in H : (a) C is a GC-set;
(a) the closed symmetric convex hull of C is a GC-set;
(b) for any s > 0, Pr(IL(C)I* < e) > 0; (c) there exist fdp's nn T I such that lim infn,,, I L (d) for some sequence nn f I offdp's, I L I*
I * = 0 a.s.; 0 in probability; (d') for some sequence nn t I of fdp's, I L (nn C) I * -+ 0 almost surely; (e) for every sequence irn t I of fdp's, I L (irn C) I * -p 0 in probability; (e') for every sequence nn t I of fdp's, I L (irn C) I * 0 almost surely; (f) L can be realized on the completion V3 of H for II 11 C; (g) L on C can be realized on the space V2 of uniformly continuous, prelinear functions; (h) L on C can be realized on the space Vt of uniformly continuous functions.
Proof For any f, g E H and constant c, L (cf + g) = cL (f) + L (g) a.s., since L (cf + g) - cL (f) - L (g) has mean and variance both 0. For any fdp it, the processes L o it and L o irl, when restricted to a countable set, are independent and satisfy L = L o it + L o irl a.s. Let us first consider the properties (d), (d'), (e), and (e'). Clearly (e') implies (e), which implies (d). For the converse(s), the following will be useful. Let Trn
be fdp's with nn T I. Let Xn (f) := L (Jrn (f )) - L (7tn _ t (f )) for f E C, n =
iro(f):=0forall f.
L
(f )) for all f E C. Let Y be a countable dense subset of C. Then nn Y is
countable and dense in Jrn C for each n. On Y, the processes Xj are independent. Now, the P. Levy inequality (1.3.16) applies to the Xj and Y defined above, since Gaussian random variables with mean 0 are symmetric, and when orthogonal, they are jointly independent (RAP, Theorem 9.5.14). So we can infer that
for each n = 1, 2, (2.5.6)
and t > 0,
Pr(maxp t) < 2Pr(IIS,IIY > t)
Further, for k = 1,
, n,
Pr(maxk<jt) < 2Pr(IISn-SkIly>t) So, assuming (d) for the given sequence 7rn, given e > 0, take k large enough so that Pr{IL(irk C)I* > s} < E/2. It follows that for any n > k, Pr{IL((zrn Jrk)C)I* > s} < s/2, by Lemma 2.5.2 with Jrk C in place of C and B := 7rn,
48
Gaussian Measures and Processes; Sample Continuity
noting that 7rn o zrk = nn o (I - Irk) = 77rn - nk Thus by the Levy inequality (Lemma 1.3.16),
Pr(maxk<j e) < s Letting n
oo, monotone convergence gives
Pr Imaxj>k IL((nj - nk)(Y))
> s} < s.
Thus the processes L o 7rj converge uniformly on Y a.s. be an orthonormal basis of H including an orthonormal basis Let el, e2, of the range of nn for each n, by Lemma 2.5.4. Consider the process M(x) :_
>j(x, ej)L(ej) and the approximating processes Mn(x) := >2j{(x, ej)L(ej): ej E range (7rn)}. We have M.(y) = L(nn(y)) a.s. for each y E Y, thus for all y E Y since Y is countable. It will be shown that for each x, Mn (x) converges to M(x) a.s. Let the variables be defined on a probability space (S2, P). In the Hilbert space J := L2(S2, P), the series defining M(x) converges since L(ej) are orthonormal and F j (X, e)2 = IIx 112 < oo. In J, the series converges
in any order. Thus MM(x) converges to M(x) in J, and so in probability by Chebyshev's inequality. Then, since L(ej) are independent, M,(x) are partial sums of a series of independent variables and so converge a.s. by the Levy equivalence theorem (RAP, Theorem 9.7.1). Now, Mn converges uniformly on Y a.s., and since each Mn is uniformly continuous on C a.s., because C is bounded, the limit is uniformly continuous on Y a.s. and is a version
of L on Y, in fact for each y E Y, M(y) = L(y) a.s. Then M extends by uniform continuity to a process defined on C and uniformly continuous there. Since L is continuous in probability, M is then a modification of L on C. Now
M - Mn converges uniformly to 0 on Y and on C a.s., which implies that 0 a.s., proving (d'). Now let Q,n be another sequence of fdp's with Q,n T I. Then for k fixed as above, let el, , e,- be a basis for the range of Irk. Then Qmej -+ 0 in H for each j. Since r is fixed and C is bounded, I L (QmjrkC) I * -- 0 in probability. L (nn C) I *
We also have Pr{IL(QmJrk C)I* > 81 < Pr{IL(nk C)I* > s} by Lemma 2.5.2. The latter is < s by choice of k. Since I L (Qm C) I * < I L (Qm. rkC) I * + I L (Qm7rk C) I * a.s., we have I L (Q,l C) I * -+ 0 in probability as m -+ oo. By
the last paragraph, the convergence is almost sure. So the properties (d), (d'), (e), and (e') are equivalent. These properties clearly imply (c). To see that they imply (f), note that in the above proof, each Mn (co) is the inner product with some element of H, and Mn almost surely converges uniformly on C to M. Each Mn defines a measurable function from 0 into H, thus into V3. Hence M defines a random variable with values in V3 (RAP, Theorem 4.2.2). So (f) follows.
2.5 The isonormal process: sample boundedness and continuity
49
Clearly (f) implies (g) implies (h) implies (a). On the other hand, (d) implies that the Mn converge uniformly also on the closed symmetric convex hull sco(C) of C. In fact, for any fdp 7r,
IL(nlsco(C))I* = IL(n1C)I* a.s., since IL(-x)I = IL(x)I a.s. for any x, and finite convex combinations with rational coefficients of elements of Y U -Y give a countable dense set in sco(C). The limit of the Mn is again M, a modification of L, now on sco(C). So (d) implies (a'), which implies (a) clearly.
Next, to see that (d) implies (b), given e > 0, take an fdp n such that Pr{IL(Jr1C)I* > s/2} < 1/2. Also Pr{IL(JrC)I* < s/2} > 0, since,rC is a bounded set in a finite-dimensional space, and since L o n and L o it can be taken to be independent (on Y), we get Pr{ I L (C) I * < s} > 0, proving (b).
Next it will be shown that (b) implies (c). For e > 0, and fdp's nn T I, Lemma 2.5.2 implies Pr{IL(Jr, C)I* < e} > Pr{IL(C)I* < s} > S for some S > 0 for all n. The event D that I L (Jr,, C) I * < e for infinitely many n, that is nm> t Un>m (I L (7r C) I * < s }, thus has probability at least S. But D is a "tail event," since it depends on the sequence of independent random variables L (ej)
only for j > k for k arbitrarily large. It follows that D has probability 0 or 1 (Kolmogorov's zero-one law, RAP, 8.4.4), thus probability 1. This yields (c).
Next (c) implies (a): for any s > 0, suppose that IL(Jr,lC)I* < 8/2 for some w and n. Then Mn, being linear on the finite-dimensional bounded set TrnC, is uniformly continuous there, so for some y > 0, lix - YII < y implies I Mn (x) - Mn (y) I < s/2, for x, y E Jrn (Y) and thus for x and y in Y, and then since Mn + L o Jr, = L almost surely on Y, I L (x) - L (y) I < E. Thus L (w) is almost surely uniformly continuous on Y, hence again extends to a uniformly continuous function on C which yields a modification of L, giving (a). It will now be enough to prove that (a) implies (d). Given e > 0, take a version of L and S > 0 such that
Pr{sup{IL(x) - L(Y)I: x, y E Y, IIx - YII < S) > s} < E. Take a finite-dimensional linear subspace F of H such that F fl C is within 3/2 of every point of C. We can assume that Y fl F is dense in F fl C. Let Tr be
the orthogonal projection onto F. Since Y is countable, we have L (x - y) =
L (x) - L (y) and L (n 1(x - y)) = L (r'x) - L (r y) almost surely for all x, y E Y. Then by Lemma 2.5.2,
s > Pr {sup I I L(7r'x) - L(ir'y)1: X, Y E Y, IIx - YII < S} > e}
> Pr {sup{IL(n1x)I: xEY}>>s},
50
Gaussian Measures and Processes; Sample Continuity
since for any x E Y there is a y in F fl Y with Ilx - yll < E and ir-Ly = 0. Letting E = 1/n J. 0, n oo, (d) holds. Recall that a Borel probability measure on a separable Banach space B is called Gaussian if every continuous linear form in B' has a Gaussian distribution. It follows that the norm II II on B satisfies some inequalities on the upper tail of its distribution for it (Landau-Shepp-Marcus-Fernique bounds, Theorem 2.2.2). In particular, f IIx I12dµ(x) < oo.
2.5.7 Theorem Let (B, II.11) be a separable Banach space. Let p be a Gaussian probability measure with mean 0 on the Borel sets of B. Then the unit ball Bi := { f : 11 f 11' < 1 } in the dual Banach space B' is a compact GC-set in L2(B, A).
-
Proof A Cauchy sequence {y, } in Bi for the L2 (µ) norm converges in L 2 (µ). Consider the weak-star topology on B', in other words the topology of pointwise convergence on B. The functions in Bi are uniformly equicontinuous, indeed
Lipschitzwiththeuniformbound I f(x)-f(y)I < Ilx-yll, f E Bi, x, y E B. Thus in Bi, pointwise convergence on B is equivalent to convergence on a countable dense set. So the weak-star topology on Bi is metrizable (cf. RAP, Theorem 2.4.4). Any linear function on B is given by its values on B1 := {x E B : Iix 11 11, and pointwise convergence on B is equivalent to pointwise convergence on B1. The set of all functions from Bt into [-1, 1], with the topology of pointwise convergence, is compact by Tychonoff's theorem (RAP, 2.2.8). It's easily
seen that B, is a closed subset of this compact space, so it is also compact. (Compactness of Bi in the weak* topology for a general Banach space B is known as Alaoglu's theorem; see, for example, Dunford and Schwartz, 1958, pp. 424-426.) So {y,} has a subsequence converging pointwise on B to some element y of B. For jointly Gaussian variables, pointwise convergence (convergence in probability) implies L2 convergence, so y is the L2 limit of {y,} and Bi is compact in L2(µ). The natural mapping T of B' into L2(µ) has an adjoint T* taking L2(µ) into B", the dual space of (B', 11 II'). There is a natural map of B into B" given
by x H (h H h(x)) for x E B and h E B'. The map is an isometry (RAP, Corollary 6.1.5, of the Hahn-Banach theorem 6.1.4). So B can be viewed as a linear subspace of B". If it is all of B", then B is called reflexive. In the present case, whether or not B is reflexive, T* actually has values in B:
2.5 The isonormal process: sample boundedness and continuity
51
2.5.8 Lemma Let B be a separable Banach space and µ a measure on B such that f IIx112dIL(x) < oo. Then for the natural mapping T of B' into H := L2(µ), the adjoint T* on H' has values in B. Proof For any h e H and Y E B',
(T*h)(y) = (h, Ty) =
h(x)y(x)dµ(x) = y(u), lB
where u e B is defined by the Bochner integral u = fB h (x )x d it (x) (Appendix E, Theorem E.9). The linear form y can be taken under the integral sign since the Bochner integral, when it exists, equals the Pettis integral (Appendix E).
So let J be the range of T*, a linear subspace of B, and S its closure, a Banach subspace of B. Note that each element of S is uniformly continuous on Bi for the H = L2(µ) norm topology, since it is a limit in the norm II II on B,
and thus uniformly on Bi, of such functions. It will be shown that µ(S) = 1. If S # B, take a countable dense subset {x,n} of B\S. By the Hahn-Banach theorem (RAP, 6.1.4) f o r each m = 1 , 2, .. , there is a u,,, E Bj such that
um =0on Sand um (x)=d(x,,,,S):=inf{llxm-yll: yES}. For any x E B\S, let e := d (x, S) > 0 and take m with Ilx - x, II < 6/2. Then l un, (x) I > d (xn
, S) - e/2 > s - 8/2 - e/2 = 0, so u,n (x) # 0. Thus
S = n,n u,;,t(0). For each m, to show that p(u,n = 0) = I is equivalent to showing that T (urn) = 0. If not, let T (u,n) = v 0. Then 0 < (v, v) = (T (un,) , v) = um (T * v) = 0 since T * v E S, a contradiction. So tt(u,n = 0) _
1 for each of the countably many values of m. It follows that µ(S) = 1. Let K be the closure of the range of T in H. Then K is a Hilbert space. A limit of Gaussian random variables with mean 0 in L2 (µ) is also such a random variable, so K consists of such random variables, and any finite set of them has a joint normal distribution. Thus the identity from K to itself is an isonormal process L. For this L, we can apply Theorem 2.5.5, where S is the space V3 of Theorem 2.5.5(f). It follows that Bi is a GC-set. The next fact is a direct consequence of Theorem 2.5.5.
2.5.9 Corollary For any two GC-sets C and D, their union C U D is also a GC-set.
Proof Condition (e) or (e') in Theorem 2.5.5 holds on C and on D and so, clearly, on C U D.
Gaussian Measures and Processes; Sample Continuity
52
2.6 A metric entropy sufficient condition for sample continuity Recall that a stochastic process Xt (w), t E T, is said to be sample-bounded on T if suptIT I Xt I is finite for almost all w. If T is a topological space, then the process is said to be sample-continuous if for almost all (o, t i-+ Xt (w) is continuous. The isonormal process is not sample-continuous on the Hilbert space H: let {en) be an orthonormal sequence. Then L(en) are i.i.d. N(0, 1) 0 slowly enough, specifically if a n (log n)1/2 -+ 00 variables. Thus if an oo, L (an en) are almost surely unbounded (by Theorem 2.3.5). So not as n all bounded sets or even compact sets are GB-sets or GC-sets. Such sets must be small enough in a metric entropy sense. This section will prove a sufficient condition based on metric entropy (defined in Section 1.2), while Section 2.7 will give a characterization based on what are called majorizing measures. A metric entropy sufficient condition for sample continuity of L will actually
give a quantitative bound for the continuity. Let (T, d) be a metric space. A function f will be called a sample modulus for a real stochastic process {Xt, t E T} if there is a process Yt with the same laws as Xt and such that for almost all w, there is an M(w) < oo such that for all s, t e T, I Ys - Yt 1(w) < M((o) f (d (s, t)).
Whenever f is a sample modulus for L on C C H, and {Xt, t E T} is a Gaussian process with mean 0 and {Xt t E T) = C, then f is also a sample modulus for the process Xt, with the intrinsic pseudometric d (s, t) (E(XS - Xt)2)112 on T. Recall from Section 1.2 the definitions of N(e, C) and H(s, C), for the usual metric d(x, y) := Ilx - yll on H. Now the main theorem of this section can be stated.
2.6.1 Theorem For any C C H, if f0 (log N(t, C))1/2dt < oo, then C is a GC-set, and if
f (x) := f (log N(t, C))1/2dt, x > 0, o
then f is a sample modulus for L on C. Note.
If C is bounded, then N(t, C) = 1 and log N(t, C) = 0 for t
large enough, and N(., C) is a nonincreasing function, so integrability of (log N(t, C))112 is only an issue near t = 0. If f (x) = +oo for some x > 0, then f (x) = +oo for all x > 0, so it still provides a sample modulus but only a trivial one. By Theorem 1.2.1, N(t, C) could be replaced equivalently by D(t, C).
2.6 A metric entropy sufficient condition for sample continuity
53
Proof We have f(l) < oo and can assume C is infinite. Then H(E) := H(E, C) +oo as s 4, 0. Sequences Sn 4. 0 and E(n) := En 4. 0 will be , En, let defined recursively as follows. Let Si := 1. Given E1, Sn
:= 2inf (E: H(E) < 2H(En)}, min(En/3,Sn)
8n+1
Then En < 3(En - En+1)/2. Also, if En+1 = Sn, then
J6(n) (n+1)
H(x)2dx < 2H(En)2En,
while otherwise En+1 = En/3 and
I(n) n+1)
H(x)2dx <
It follows that for any n = 1, 2, 2 °O m=n
, CO
(Em - Em+1)H(Em)1,2 m=n 00
f(En) < 4
n
E.H(Em)1/2.
M =n
So the convergence of the above sums is equivalent to that of the integral defining
f(1), and they all converge.
For each n, there is a set An C C such that for any x E C we have Ilx yll < 2Sn for some y E An, and the number of elements of An is bounded by card(An) < exp(2H(En)). Let Gn := Ix - y: x, y E An-1 U An). Then card(Gn) < 4exp(4H(En)). Let Pn := Pr {max {IL(z)I/IIzII : 0
z E GnI > 3H(En) 1/2).
If is the standard normal distribution function, then for T > 0, 1 - 1(T) < exp(-T2/2) (RAP, Lemma 12.1.6(b)). Then
Pn < 4exp{4H(En)-9H(En)/2} = 4exp(-H(En)/2). Since H(En+2) > H(Sn/3) > 2H(sn), En Pn is dominated for n > 2 by a sum of two geometric series, one for n even and one for n odd, and so converges. Then for almost all w, there is an no (w) such that for all n > no (w), I L (z) I < 3IIZIIH(En)1/2 for all z E Gn.
Either Sm = Em+l or Sm < 2Em = 6Em+1, so Sm < 6Em+1 for all m > 1,
and En>2Sn_1H(en)1/2 < oo. For each x E C choose An(x) E An with
Gaussian Measures and Processes; Sample Continuity
54
I I x - A n (x) I I < 28n. Then I I A n - i (x) -An(X)11 <28_i + 28n , and for almost
all co, L(An(x))(co) is a Cauchy sequence for all x E C. Define a process
M by M(x)(co) := limn,, L(An(x))(c)), X E C, or M(x)(co) := 0 if the is continuous in probability, for sequence doesn't converge. Then since each x E C, M(x) = L(x) a.s., and M and L have the same laws (on C). From the definition of sample modulus, we can then assume that L - M. So except for w in the set of 0 probability where no(w) is undefined, L (An (x)) (w) L (x) (w) as n oo for all x E C. Ifs, t E Canden+1 < Its - til < En, then 11An(s)-An(t)11 < I1s-t1I+48n, and if n > no (w), then IL(s) - L (t) I ((o)
< I L(An(s)) - L(An(t))I (w) 00
+
(I L (Am (s)) - L (Am+l (s)) I (w) m =n
+ I L (Am(t)) - L (Am+1(t))I (w)}
Us - tll +48n) 3H (8n) 1/2 +
88m 3H
(Em+1)112.
m =n
Thus
L(s)-L(t)j(w) < 7511s-tIIH(Ils-tli)112+144T EmH(Em)112 m>n
< 291f(Ils-tI). This is valid for Its - tll < Eno(,)). The hypothesis implies that C is totally bounded, so whenever no((o) < oo, is bounded on C, say by K(w). I
So the conclusion of Theorem 2.6.1 holds with
M(w) := max (291, 2K(w)/f (Eno(w))).
Li
It follows from Theorem 2.6.1 that C is a GC-set if as E J, 0, N(E, C) =
O(exp(e-P)) for some p < 2, or if N(e, C) = O(exp(E-21 log El-')) for some r > 2. On the other hand Theorem 2.3.5 implies that C is not a GB-set
if as E J, 0, eventually N(E, C) > exp(E-P) for some p > 2 or N(e, C) > exp(e-21 log 815) for some s > 0. It turns out that the gap cannot be closed further: if N(e, C) is of the order of exp(E-21loge I-'') for 0 < r < 2, there are examples showing that C may or may not be a GB-set (see problems 14 and 15). So a characterization of the GB-property can't be given in terms of metric
2.6 A metric entropy sufficient condition for sample continuity
55
entropy, although it comes rather close. For a characterization in other terms, see the next section.
Remark. If C is a GC-set, then can be chosen such that for all w, x H L(x)(w) is continuous for x E C. Then for any countable dense subset A of C, L(C)* = SUPXEA L(x) a.s.
Next, the same integral as in Theorem 2.6.1 yields a bound for expectations of certain suprema.
2.6.2 Theorem Let C C H be nonempty and let D := diam C = supx,YEC Ilx - yll Let B
{x - y: x, y E Q. Then for f as in Theorem 2.6.1,
(a) EIL(B)I* < 81 f(D/4), and (b) EL(C)* < 81 f(D/4). Remarks. All three quantities in (a) and (b) are invariant under translation, replacing C by {c + u : c E C) for any fixed u. But E I L (C) I * does not have such invariance, and it becomes unbounded as II u II -+ oo, so for it we cannot
have an upper bound Kf(D), K < oo. If the constant 81 is replaced by a larger one, one can have, instead of the quantities on the left in (a) and (b), Young-Orlicz norms (Appendix H) II where g(x) := exp(x2) - 1; see Theorem 2.6.8 at the end of this section.
IIg
Proof Note that log N(t, C) = 0 for t > D/2, so f(D/2) = f(+oo). If f (x) = +oo for some (and hence all) x > 0, then (a) and (b) hold trivially (under the given definitions). If f (D) < oo, then we can take L samplecontinuous on C by Theorem 2.6.1. We first have:
2.6.3 Lemma Let go(u) := 2uO(4-1(1/(2u)))foru > 1/2, where 0and are the standard normal density and distribution function, respectively. Then go is concave. For any random variable Z with distribution N(0, a2) and any event A with P(A) > 0, (2.6.4)
f
JA
IZI dP < aP(A)go(1/P(A)).
Gaussian Measures and Processes; Sample Continuity
56
Proof Let h(v) := go(v/2) = v4)(-1(1/v)) for v > 1. Then
h'(v) = 0
(1)) \c-1
+v4
\
1
(0-, \v
()) \-(D_1 \v/) 0 ((D-1 (1/v)) 1) + v(D-1 vl
\v
(i)
V2
V
+
1
v
1
V2
J
((D-1 (1/v))
0
V 1
v2
_J
1 v34) (q)-1 (1/v)` < 0
so h is concave for v > 1 and go is concave for u > 1/2. Next, the left side of (2.6.4) is maximized for fixed P(A) > 0 when A is a set {IZI > r} for some r > 0, by the Neyman-Pearson lemma (e.g., Lehmann, 1986, p. 74). Then P(A) = 2(D(-r/a) and
J. I Z I dP = (2/ir)1/2a exp (- r2/(2a2)). Letting x = r/a, we need to prove, for x > 0,
24)(x) < 24 (-x)go(l/(2(D(-x))).
(2.6.5)
Setting u := 11(2(D(-x)), so that x = - -1(1/(2u)), shows that (2.6.5) holds, with equality.
2.6.6 Lemma If Zl,
, ZN are each normally distributed with mean 0 and
variance < a2, then Emaxl<j
fIZjIdP J
E maxi <j
< TaP(Aj)go(1/P(AS)) < ago(N) j=1
by Lemma 2.6.3, from which go is concave.
2.6 A metric entropy sufficient condition for sample continuity
57
2.6.7 Lemma Let gl (u) := K(log(1 + u))1/2 for u > 0 where
K := (2 + [4 + log
4]/(log(3/2)))1/2.
Then gi (u) > go (u) for all u > 1.
Proof "Mills' ratio" satisfies, for x > 0, Komatsu's inequality +4) 1/2)
M(x) :_ ,)(-x)/O(x) > 2/(x + (x2
(RAP, Section 12.1, Problem 7). Thus M(x) > 1/(x2 + 4)1/2. Let x -4)-1(1/(2u)) > 0, so u = 1/(24)(-x)). Then we need to show that
gl(1/(24'(-x))) ? fi(x)/(D(-x) = 1/M(x), 4)1/2. Since (D(-x) < so it will be enough to prove gi(1/(2(D(-x))) > (x2 + exp(-x2/2) for x > 0 (RAP, Lemma 12.1.6(b)), and gi is nondecreasing, it will be enough to show that gi(exp (x2/2)/2) > (x2 +4) 1/2, x > 0.
Letting y := exp(x2/2)/2, we need to show that
gi(y) = K(log(1 + y))1/2 > (4 + 2log(2Y))1/2,
y > 1/2,
or
y > 1/2,
K2log(1 + y) > 4 + 2 log 2 + 2 log y, which follows from the definition of K.
Now to prove Theorem 2.6.2, D = 0 if and only if C consists of a single point. Then both sides of (a) and (b) are 0 and they hold. So assume D > 0. Let Ek := D/2k and Nk := N(Ek/2, C), k = 0, 1, . Then for each k = 0, 1, 2, , there is a set Ck of Nk points xkj, j = 1, , Nk, such that for all x E C, Ilx - xki II < Ek for some j. Then No = 1, so Co = {xoi } for some xoi. For each k = 1, 2, and j = 1, , Nk, choose and fix a point yki = xk_1,i for some i such that Ilxki - Ykj Il < Ek-1 Let Wk be the set of all variables , Nk. Then by Lemma 2.6.6, L (xkj) - L (ykj), j = 1,
Emax(Izl: z E Wk) < Ek-igo(Nk) For any uk E Ck, there is a sequence of points uj E Cj, j = 0, , k. Thus
that L(ug) - L(uj_i) E Wj, j = 1,
k
l
k
Esup jJL(x)-L(y)I: x,YE UCi1 < I
i=1
j=1
,
k, such
58
Gaussian Measures and Processes; Sample Continuity
The union of all the Cl is dense in C, so by sample continuity and monotone convergence, ao
EC := E sup{IL(x) - L(y)I: x, y E C) < 2 E sj_tgo(Nj) j=1 00
4D Y go(Nj)/23 j=1
By Lemma 2.6.7, where K < 4, we get 00
EE < 16D E(log(1 + Nj))1/2/2j. j=1
For all j > 1, Nj > 2, so [log(1+ Nj)/ log Nj] 1/2 < (log 3/ log 2)1/2 < 1.26. Thus
Ec < 20.2D E (log Nj)1/2/2j 00
j=1 00
Ei+t
81
(logN(t, C))1/2dt = 81 f(D/4),
E1+2
proving (a). Then for any fixed y c C, SUPXEC L(x) < L(y) + suPXEC IL(x) - L(y)I, so (b) follows and Theorem 2.6.2 is proved.
Let g be a convex, increasing function from [0, oo) onto itself. If Y is a random variable such that Eg(SY) < oo for some S > 0, let II Y IIg := inf {c > 0 : Eg(I Y I /c) < 11. Then 11 IIg is a seminorm on such random variables (Appendix H). If there is no such S > 0, let II Y 11 g
+oo.
2.6.8 Theorem There is an absolute constant M < oo such that for any subset C of a Hilbert space H, and g(x) := exp(x2) - 1,
IIL(C)*IIg < II IL('')I*IIg < ME(IL(C)I*). Proof If C is not a GB-set, all three expressions will be infinite, so suppose C is a GB-set, which we can then take to be countable. Then by the LandauShepp-Marcus-Femique theorem (2.2.2), 111 L (C) I * 11 g < oo.
2.7 Majorizing measures
59
The first inequality in the theorem is clear. Suppose there is no such M < co. Then there a r e countable GB-sets C j C H with 111 L (Cj) I* II g> j 3 E I L (Cj) I* for j = 1, 2, . By homogeneity, we can assume E I L (C j) I * = 1 for each j.
Let H1, H2,
be infinite-dimensional Hilbert spaces and form the direct sum 1-1 := ®jHj, so that Hj are taken as orthogonal subspaces of R. We can take
Cj C Hj for each j. Let Dj := Cj/j2 for each j. Let D := Uj Dj C R. Then
EIL(D)I* = EmaxjlL(Dj)I* < T EIL(Dj)I* = J:j-2 < 00, l I so D is a GB-set. Thus by the Landau-Shepp-Marcus-Fernique theorem (2.2.2), II IL(D)I*IIg < oo. But for each j, II IL(D)I*Ilg >_ 1 L(Dj)111 g > j, a contradiction; so Theorem 2.6.8 is proved. 11
2.7 Majorizing measures This section will prove a characterization of GB-sets, that is subsets of a Hilbert
space H on which the isonormal process is sample-bounded, in terms of majorizing measures, to be defined next. Problems 14 and 15 at the end of this chapter show that there is no such characterization in terms of metric entropy in general, although there is under further restrictions (Theorem 2.7.4). The majorizing measure characterization and its proof are due to X. Fernique and M. Talagrand. For a metric space (T, d), r > 0, and x E T, the open ball of center x and radius r is B(x, r) :_ f y: d (x, y) < r). Definition. Let (T, d) be a metric space and P(T) the set of all laws (Borel probability measures) on T. For M E P(T) let Ym(T)
5UpxeT
f 0
(log
1
1/2
11dr. \m(B(x, r)))J
(
If y,,, (T) < oo, then m is called a majorizing measure for (T, d). Let y (T)
y(T,d) := inf{ym(T): ME P(T)}. Then y (T, d) < oo if and only if there exists a majorizing measure on T. If m is a majorizing measure on T, then for all x E T, m(B(x, r)) > 0 for all r > 0 and does not approach 0 too fast as r 0. For example, if T is finite, then ym (T) < oo if and only if m ({x)) > 0 for all x E T. On [0, 1] with usual metric, Lebesgue measure ? is a majorizing measure. More generally, so is any law having an absolutely continuous component with a density h > c for some
c>0.
60
Gaussian Measures and Processes; Sample Continuity
First, two theorems will be stated, which together characterize GB-sets as sets in Hilbert space having majorizing measures. Here T = C will be a subset of a Hilbert space with the usual Hilbert metric. Recall L(C)* := ess.supXEc L(x) and I L (C) I * as defined by Lemma 2.5.1.
2.7.1 Theorem (Fernique, 1975) If C is a subset of a Hilbert space H and y(C) < oo, then C is a GB-set. For some absolute constant K, EL(C)* <
Ky(C). Translation of C, replacing it by {c + It : c E C} for some fixed h, preserves all of N(e, C), y(C), and EL(C)* but not EIL(C)I*. The word Notes.
"majorizing" apparently refers to the inequality in Theorem 2.7.1.
2.7.2 Theorem (Talagrand, 1987) If C is a GB-set then y(C) < oo. For some absolute constant K' and all C C H, y(C) < KEL(C)*. Before Theorems 2.7.1 and 2.7.2 are proved, some easier facts will be given.
Recall from Section 1.2 the definition of D(e, C) := D(s, C, d) for the usual metric d(x, y) := Ilx - yll on H. A purely atomic law is a countable sum m = E j a j 8 y(j) for some a j < 0 and points y(j), where 8x is the unit point mass at x.
2.7.3 Proposition
f °O(log D(e, C))1"gds < oo implies y(C) < oo and
ym (C) < oo for a purely atomic m.
Proof That y (C) < oo follows from Theorems 2.6.1 and 2.7.2. Here is a direct proof. Let D k := D(2-k, C). For k = 1 , 2, .. , take a maximal set xik, i = 1, , Dk, With Ilxik - xjk II > 2-k for i j. Set 00
m := E (2kDk)
1E
Sxik
1
k=1
For each k and each x E C, there is some xik with Ilx - xikll < 2-k. So m(B(x, e)) > 1/(2kDk) for 2-k < s < 21-k and 1
f
Jo
00
(log (1/m(B(x,£))))112ds < T, 2-k(log {2kDk}) 1/2 k=1 00
< T2 -k I (k log2)1/2 + (log Dk)1/2}. k=1
2.7 Majorizing measures
61
Recall that for any set A in a metric space (S, d), the diameter is defined by
diam A := supx,yEA d (x, y). Now Ek k112/2k < oo, and Tk(log Dk)112/2k < oo is equivalent to 1
(log D(s, C))112ds < oo
(integral test).
JO
The integral of (log(1/m(B(x, s))))1/2 from 1 to oo is the sum of the integral from diam C to oo, which is 0, plus, if diam C > 1, an integral bounded above by (diamC)(log(1/m(B(x, 1))))1/2 < oo.
2.7.4 Theorem Let (T, d) be a totally bounded metric space. Suppose that m is a law (Borel probability measure) on T such that for some M < oo and all r > 0, supx,T m(B(x, r)) < MinfyET m(B(y, r)). Then the following are equivalent:
(i) m is a majorizing measure for T ;
(ii) y(T) < oo; (iii) fa (log D(s, T))1/2ds < oo. Proof Proposition 2.7.3 shows that (iii) implies (ii), and clearly (i) implies (ii). We can assume M > e. To show that (iii) implies (i), by definition of D(s, T) (Section 1.2), given
s > 0, there are x; E T, i = 1, , n := D(s/2, T), such that T is the , n of the balls B(xi, s). Thus for any x E T, 1 = union f r o m i = 1, m(T) < D(s/2, T)Mm(B(x, E)). Thus 1/m(B(x, s)) < MD(s/2, T) for all x. In the latter inequality we take logarithms of both sides, then square roots, then integrate with respect to e. The integrals are finite and bounded in x, so m is a majorizing measure.
To show that (i) implies (iii), given s > 0, there are xi E T, i = 1, , D(2s, T), such that d(xi, xj) > 2s for i # j, so the balls Bi := B(xi, s) {y: d(xi, y) < s) are disjoint. Now m(T) = 1 > D(2s, T)m(B(x, s))/M for all x. Thus 1/m(B(x, s)) > D(2s, T)/M for all x. In the latter inequality we again take logarithms of both sides, then square roots, then integrate. The integrals are finite, and the factor of 2 doesn't affect the convergence, so (iii) holds. The last step will be to show that (ii) implies (i). Let s be a majorizing measure for T. If in the definition of y we replace log(.) by L max(log(.), 1), the integral can only be increased, but at most by 1, so finiteness of the supremum
Gaussian Measures and Processes; Sample Continuity
62
of integrals is preserved. The function g: t i-k (L(1/t))1/2 is convex on the interval 0 < t < 1, being the maximum of the function (log (1 / t)) 1/2, which has a positive second derivative for 0 < t < e -1/2, and the constant 1, which is larger fore-1
< t < 1. Foreachr > 0, consider the randomvariableX= µ(B(x, r))
where x has distribution m. Evaluating EX by the Tonelli-Fubini theorem, since
y e B(x, r) is equivalent to x E B(y, r), gives EX = f m (B(y, r)) d p (y). Thus for any x E T, m (B(x, r))/M < EX < Mm (B(x, r)). Jensen's inequality (RAP, 10.2.6) gives g(EX/M) < Eg(X/M). Let K := supx,YET d(x, y). Then for any x, using M > e, 00
r
Jo
1
[log (m (B(x, r))
)]1/2
K
dr <
dr
LL \m (B(x, r)) / J
Jo
K
<
1/2
1
(
f
Jo
fJo K
M
1/2
L
lt(B(y, r)) M [log rr
(
dm (y) dr 111/2
l-t(B(y,r))/J
drdm(y)
< K(log M)112 + y, (T) < oo, proving (i).
2.7.5 Corollary Let (T, d) be a metric space such that there is a group G of 1-1 transformations g of T onto itself for which
(a) d is G-invariant: for all s, t E T and g E G,
d(g(s), g(t)) = d(s, t); (b) there is a law (Borel probability measure) m on T which is G-invariant,
that is,mog 1=m forallgEG; (c) G acts transitively on T: forall s, t c- T, there is a g E G with g(s) = t. Then the hypotheses and thus the conclusion of Theorem 2.7.4 hold.
Proof The hypotheses imply that for each r > 0, m(B(x, r)) is the same for all x E T. Thus Theorem 2.7.4 applies for any M > 1. Notes. Fernique (1975) proved that for Gaussian processes satisfying a homogeneity condition like that in Corollary 2.7.5, the metric entropy integral condition (or the corresponding condition on N(e, T), equivalent by (1.2.1)) is necessary and sufficient for sample continuity. Theorems 2.7.1 and 2.7.2, with
2.7 Majorizing measures
63
Theorem 2.7.4, show that the metric entropy integral condition is equivalent to
sample continuity for the isonormal process on T for a subset T of a Hilbert space satisfying the conditions of Theorem 2.7.4. For an example of the situation in Corollary 2.7.5, let T be the unit circle x2 + y2 = 1 in R2, let G be the group of rotations, let d be the usual metric on R2, and let dm(O) = d9/(27r). Likewise, T could be a sphere of any dimension, with the orthogonal group G. To see how Theorem 2.7.4 applies beyond Corollary 2.7.5, suppose one wants to prove sample continuity of a Gaussian process on a locally compact but not compact metric space, such as a Euclidean space or a non-compact manifold. Then it suffices to prove sample continuity on each of a family of compact sets whose interiors form a base for the topology, such as balls or cubes in Euclidean spaces. Then one can often define a measure, such as Lebesgue measure in a Euclidean space, restrict it to a compact set C, and normalize it to have mass 1 to get a law m. Then m (B(x, r)) may not depend on x while B(x, r) is included in the interior of C, but become smaller as x approaches the boundary of C, yet the hypothesis of Theorem 2.7.4 still holds.
For any set A C H, let E(A) := EL(A)*. Then if A is countable, E(A) E supxEA L(x).
Proof of Theorem 2.7.2 The proof below for Talagrand's theorem 2.7.2 will be mainly as given by Fernique (1997). We have:
2.7.6 Lemma Let S C T C H, u > 0, and suppose d (x, y) > 4u for all x =,4 y in S. Let M be the cardinality of S. Then
E(T) >
E[B(s, u) fl T] + 12 (log M)112u.
Proof The statement holds if E(T) = +oo, so we can assume E(T) < oo. This implies that T is totally bounded by Theorem 2.3.5, so S is finite. , sM}. Let L1, , LM be independent copies of L. Let Z1, , ZM be i.i.d. N(0, 1) variables independent of L1, , LM. We can
Let S = {sl,
assume T C UM1 B(sj, u). Let Y(t) := Li(t) - Lj(sj) + uZj for each t E B(sj, u) and j = 1, M. Let v, t E T. If v, t E B(sj, u) for the same j, then dy(t, v)2 := E((Y(v) - Y(t))2) = d(v, t)2 = Ilv - t112. Otherwise, we have d(v, t) > 2u > dy(v, t). Thus dy(v, t) < d(v, t) in all cases, so by Theorem 2.3.7(b), E(T) > E suptET Y(t). To find a lower bound
Gaussian Measures and Processes; Sample Continuity
64
for E suptET Y(t), we have
suptET Y(t) = maxl
, ZM measurable. Then since
E(suptET Y(t)I13Z) > maxt<j<M [uZj + E suptETnB(s;,u) Lf(t)]. Taking the expectation with respect to {Z;}M 1 then gives
E SUptET Y(t) > U. E maxi<j<M Zj+ mint<j<M E(T fl B(sj, u)) 12 (log M)112 + minSES E(T fl B(s, u))
by Lemma 2.3.15, which finishes the proof.
We next have some positive-homogeneity properties of functionals of sets in H.
2.7.7 Proposition Let C be a bounded set in H and let g be either of the two functionals g(C) = y(C) or g(C) = E(C) := EL(C)*. Then for any constant a > 0, we have g(aC) = ag(C). Proof For E(C) this follows from linearity of L. For y(C), multiplication by a transforms any law m on C to an image measure on aC, and the conclusion follows by a linear change of variables in the definition of yn (C). By the last fact, we can assume in Theorem 2.7.2 that (2.7.8)
3/4 < diam T < 1,
since T is bounded by Theorem 2.3.5, while if it has diameter 0, Theorem 2.7.2 holds trivially. Given a set C, a finite partition of C is a collection of disjoint sets Cj, i = 1 , 2, , J for some J < oo such that C = U 1 Cj. A sequence
C('), m > 0, of partitions is nested if for each m and each A E C(nr+1) there
is a B E C(') such that A C B. Recall that B(x, r) := {y: d(x, y) < r} for any x and any r > 0. A main step in the proof is the following:
2.7.9 Lemma Let p := 1/4. If T is a GB-set in H satisfying (2.7.8), there is a nested sequence {C(i°), n > 0) of partitions of T and for each n, a mapping m from C(") into N and a mapping r from C(") into T such that the following hold:
2.7 Majorizing measures
65
(i) For every n E ICY and each C E C(n), C C B(rn (C), pn); (ii) for any positive integer n, every C E C(n-1), and any two sets D # D'
in Cini included in C, we have mn (D) 0 mn (D'), rn (D) and rn (D')
are in C, and d(rn(D), rn(D')) > p'; (iii) for every s E T, if Cn (s) is the element of C(') containing s, then 00
pn[logmn(Cn(S))]112 < 192E(T). n=0
Proof The partitions COO will be defined recursively. Let C(°) :_ IT), mo(T) := 1 and let ro(T) be any fixed element a E T. Then (i) holds for n = 0. Suppose C(i), mj(.), and have been chosen for j = 0, 1, ,n satisfying (i) and (ii). To define C(n+1) each set C E C(') will be decomposed as a disjoint union of subsets which will be in C(n+1) These subsets and the corresponding values of mn+l () and rn+l () will also be chosen recursively on an index r. Let Ao := 0 and Wo := C. Let t1 be an element of C for which
E(B(tl, pn+2)) > suPtEWo E(B(t, pn+2)) - pn+2E(T)
For r > 1, if C is not included in A,. :_ U
pn+l), let tr+l be an
element of Wr := C fl (T \ Ar) such that (2.7.10)
E( B(tr+1, pn+2) fl Wr)
> suP,Ew E(B(t, pn+2) fl Wr) - pn+2E(T) Then d (ti, tj) > pn+l for i # j. For the least k such that C C Ak, let M(C) k and the construction stops. Then M(C) < exp(300E(T)2/p2n+2) by Theorem 2.3.16. For each m = 1, , M(C), let Dm := Dm(C) := B(tm, pn+1) fl Wm-1 Let the partition C(n+1) consist of the sets D. (C) for m = 1, , M(C) and C E C. Then set rn+l (Dm) := tm and mn+1(Dm) := m. This completes the recursive definition. By construction, (i) and (ii) in the lemma hold.
To prove (iii), fix an s E T and an n E N. Let C := Cn (s) be the set in C(n) containing s. Then Cn+1(s) C Cn (s) and Cn+1(s) is Dm for some m = 1 , , M(C), where Dm is a set in C(n+l) included in C, with m = mn+1(Cn+l (s)) and tm = rn+l (Dm) = rn+1(Cn+l (s)). For the points t1, t2, defined for C as above, and I < i < k < m, we have d(t,, tk) > pn+l The sets B(t,, pn+2) fl W;_1 for i = 1, , m are included in C since W;_1 is. Thus by the choice of p = 1 /4 and Lemma 2.7.6,
(2.7.11) min,<m E(B(t,, pn+2) fW,-1) + zpn+2[logm]l/2 < E(C).
Gaussian Measures and Processes; Sample Continuity
66
On the other hand, for each i = 1, , m and since tm E Wm-1 C Wt-1, by (2.7. 10) for r = i - 1 we have pn+2E(T) + E(B(ti, pn+2)) > E (B(tm, pn+2) n Wi-1) 1
>
El( Bl(tm, pn+2) n Wm-1).
Combining the latter with (2.7.11) gives
ECn(s) = E(C) > 12pn+2[logmn+l(Cn+l(s))]112
(2.7.12)
+ E( B(tm, pn+2) fl Wm-1) - pn+2E(T)
Also, since rn+2(Cn+2(s)) E Cn+1(s) = Dm C Wm_1, by (2.7.10) for r = m - 1 we have E( B(tm, pn+2) n Wm-1)
(2.7.13)
> E( B(rn+2(Cn+2(s)), pn+2) fl Wm-i) - pn+2E(T)
Since Cn+2(s) C B(rn+2(Cn+2(s)), pn+2) n Wm_1, and by inequalities (2.7.12) and (2.7.13), we get
E(Cn(s)) >
12pn+2[logmn+l(Cn+1(s))]1/2+E(C1+2(s))-2pn+2E(T)
Fixing an integer N > 0, we get N
E pn+2 [log m n+1(Cn+1(5) )] 1/2 nn=0
< 12 {ECOs+ E(C1(s)) + 2E(T) n=O
n+2 J
48E(T).
J
Thus yn_1 pn+1[logmn(Cn(s))]1"2 < 48E(T), and because mo(Co(s)) mo(T) - 1, the lemma follows. To finish the proof of Theorem 2.7.2, apply Lemma 2.7.9. Let T be a GB-set in H. For each n = 1, 2, , let T (n) be the set of all points rn (C) for C E C(n). A purely atomic law it will be defined by It :_ >n°_0 EtET(n) An(t)8 . To define the measures 12n, first, T(0) = (a). Let µo(a) := 1/2. For n = and s E T, let (¢n (s) := rn (Cn (s)) and mn (s) := mn (Cn (s)). For 0, 1, 2, t E T(n) and n > 1, let
lin-1(On-1(t)) 2[mn(t)]2 F-{mn(u)-2: On_1(u) = 0n-1(t)} Each µn is 0 outside of T (n). Then for each v E T (n - 1), we have (2.7.14) ltn(t)
(2.7.15)
1
{µn(t): t E T(n), 0n-1(t) = v} = An-1(v)12.
2.7 Majorizing measures
67
Thus An :_ {µn(t) : t E T(n)} = 2 E{isn_i(v) : v E T(n - 1)}, s0 An = An_i/2. Then by induction An = 2-a-1 f o r n = 0, 1, , and A is indeed a probability measure. An (on (u)). For any u E T and n > 0, we have d (u, ¢n (u)) < Let A, u
pn, so A(B(u, pn) > An,u. The sum in the denominator of (2.7.14) is, by Lemma 2.7.9(ii), < F-,n>, m-2, which equals 7r2/6 (problem 19). Thus
µn,u ? 37r-2µn-l,u/mn(u)2.
A(- (u, pn))
(2.7.16)
Iterating the latter inequality, and noting that µp,u = 1/2 for all u, we get for all n and u E T, (2.7.17)
µ B (u> p")) >
(3 1"
1
1
n2 J Mn (u)Zmn-1(u)2 ... M 1(u)2 2 .
For each u E T, we have p,(B(u, r)) < p (B (u, r)) for at most countably many r, making no difference in integrals dr. Now, since by (2.7.8) diam T < 1, letting g(u, r) := [log(1/µ(B(u, r)))]112, we have 00
I (t-r, u)
:=
L
1
g(u, r) dr =
00
E j=1
J0
g(u, r) dr 00
pj-1
pj-lg(u, pi)
g(u, r)dr < P
j=1
For any a; > 0, (a1)1I2 < F; ai 12. Thus by (2.7.17), o
I(µ, u) < 4
5j=1
j pj (j log (7T2/3))1/2+ (log2)1/2 + >2(21ogm;(u))1/2 i=1
The (j log (7r 2 /3)) 1/2 and (log 2)1/2 terms clearly yield summable series not
depending on u. Specifically, we have j1/2 < j and 4 F_jtl j pi = 4p/(1 p)2 = 16/9, and (16/9)(log(7r2/3)) 1/2 + (4/3)(log2)1/2 < 3.06. The remaining terms give 00 4 .21/2>2[logm;(u)]1/2p`/(1 - p) < 1449E(T)
by Lemma 2.7.9(iii). It follows that A is a majorizing measure. Now by
(2.7.8), T contains at least two points x, y at least 3/4 apart, so E(T) > E max(L (x), L (y)) = E max(L (z), 0) > 2/7 where z := x - y, L (z) has law N(0, a2), and or > 3/4. Thus 3.06 < 11 E(T) and I (A, u) < 1460E(T) for all u, so y, (T) < 1460E(T) and Theorem 2.7.2 is proved. 0
Gaussian Measures and Processes; Sample Continuity
68
Proof of Theorem 2.7.1 Let C C H and let it be a majorizing measure on H. It will be proved that C is a GB-set. Let {e,) be an orthonormal basis of H. For each x E H we have L (x) >2,, (x, en)L(en) where the series converges almost surely (the set on which it does not converge depends on x). Define a version of L (x) for all x by taking the sum of the series if it converges, and 0 otherwise. The resulting process is jointly measurable in x and cv (RAP, proof of Theorem 4.2.5). Thus we can assume L is jointly measurable, in particular when restricted to x E C. Theorem 2.7.1 will be proved by way of the next fact.
2.7.18 Theorem Let T be a bounded subset of H with diameter D on which there exists a majorizing measure It. Then there exists a version M of L on T
and a random variable Y with 0 < Y((o) < oo and EY < 485 such that for any s, t E T, (2.7.19)
(M(s) - M(t))(c))
\]1/2
d(s,t)
< Y((0)
I[log(p.(B(U))
J
+
u)))] 1/21 [log( µ(B(s, 1
du.
Proof Recall that a measurable space is a pair (W, 8) where W is a set and B is a o--algebra of subsets of W. One part of the proof will be furnished by the following:
2.7.20 Lemma (a) Let x, y, a, and ,B be positive numbers. Then rr
xy < ao Lexp
1/2
(x 2
-a )/ - 11 + a yL I log I 1 + \\\
/IJ
.
(b) Let (W, B) be a measurable space and let Y(x)(co), X E W, cv E 0 be a jointly measurable Gaussian process with mean 0 and EY(x)2 < 1 for all
x E W. Let µ be a law on W. Then there is a random variable z > 0 with Ez < 5/2 such that for each nonnegative function f E G1 (W, µ), not 0 a.s., we have
f
IY(x)(w)I.f(x)dµ(x)
f
/
3z(w)J f(x) [log I 1 +
/p.)]"2d(x).
2.7 Majorizing measures
69
= 1. To apply the
Proof For (a), dividing by a p, we can assume a =
W. H. Young inequality (Theorem H.6 in Appendix H), let 0 (u) := exp(u2) -1,
u > 0, *(V) := ¢-i(v) = [log(1 + v)]1/2, v > 0, c(x) := fox O(u) du, and %P (y) : = fo i (v) d v. Then clearly I (y) < y,' (y) for y > 0. By derivatives, 4) (x) < 0 (x) for x > 0, and (a) follows from Theorem H.6. Now to prove (b), let p < 1/2. Then supXEw E exp(pY(x)2) < oo. Thus for almost all w, x i-+ exp(pY(x) (w)2) is integrable for it by the Tonelli-Fubini
theorem. Let
z := inf I C1 > 0:
J
(exp[a-2Y(x)2] - l)dµ(x) < 1I
.
(Thus z = IIYIIg as defined in Appendix H for g(x) := exp(x2) - 1.) The dominated convergence theorem implies that as a -+ oo,
J
(exp[a-2Y(x)2] - l)dµ(x) -+ 0
almost surely. Thus [a > 0: f (exp[a-2Y(x)2] - 1) dµ(x) < 1} is almost surely nonempty. So z has finite values a.s. It is easily seen that z(w) = 0 if and only if Y(x)(w) = 0 for /.t-almost all x, and then (b) holds. It remains to prove (b) when z(w) > 0. Since f is not 0 a.s., it follows by part (a) with
0<6:= f fda0,
f
IY(x)((o)I.f(x)dµ(x)
a
f fdµ f LeXp (Y(x)2\ - l]dlt(x) a2 /
+a
f f(x)
/
1/2
f
Ilog I 1 + .f(f.xd) µ)I
dµ(x).
This fact, Proposition H.2 of Appendix H, and the definition of z then imply, since z((o) > 0, (2.7.21)
f
IY(x)(w)I f(x)dg(x)
< z1 f f(x) dlt (x) +
f
[log .f (x)
f(x)
(l + I f da)]
1112
dIL(x)
For u > 0 let F(u) := u[log(1 + (u/,B))]1/2. Then F is increasing and
Gaussian Measures and Processes; Sample Continuity
70
convex on [0, oo) by derivatives. Jensen's inequality (RAP, 10.2.6) then gives (log 2)1/2 f
fdg =
F
(f f d i)/// < fF(f)dt fx)
1/2
f(f
f f(x)
['og 11+
dµ
\l
dµ(x).
I
This, with (2.7.21), gives part (b) except for bounding Ez. To do that, we have for any u > 0 and p > 1 by definition of z and then the Chebyshev-Markov inequality that l
(
P(z > u) < P
if
< 2-"E
exp (y(x)2/u2) dµ(x) > 2 } dµ(x)JP
([f exp (Y(x)2/u2)
)
.
By Holder's inequality, multiplying the latter integrand by 1 (or Jensen's inequality), we have
P(z > u) < 2-PE For 1 < p < u2/2, the
\
f exp (PY(x)2/u2) dµ(x))
.
quantity` on the right is finite and, by the Tonelli-
2pu-2)-1/2. For Fubini theorem, bounded above by g(p, u) := 2-P(1 u > uo := (2 + (log 2)-1)1/2, if we set p := u2/2 - 1/(2 log 2), then p > 1
and
g(p, u) = u(e log 2)1/2 exp (- u2(log 2)/2). For any t > 0 and random variable X with law P,
l
00
00
P(X > u) du
f fuo"
dP(x) du = 100
JX du dP(x)
t
00
(x - t) dP(x) = E(Xlx>t) - tP(X > t). It follows that if X > 0, then EX < t + fto° P(X > u) du. Thus
Ez < uo + (e log 2) 1/2
f
2_U2/2du
00 u
up
uo + (e log 2)1/2(log 2)-1 exp (- ua(log 2)/2) < 2.5 as stated, which finishes the proof of the lemma.
2.7 Majorizing measures
71
Now to continue with the proof of Theorem 2.7.18, recall that we have chosen
a jointly measurable version of L()(). It will be shown that on T can be well approximated by its averages over small balls with respect to A. Recall that B(x, r) :_ {y: d(x, y) < r}. For each k = 0, 1, and t E T let
A(B(t, D/2k)),
µk(t) (2.7.22)
Pk(t, )
0-pk(t))-11g(t
D12k) ('),
and Mk(t) := Mk(t, (o) f pk(t, u)L(u) d s(u), which exists and is finite a.s. by the Tonelli-Fubini theorem since for any x, to E T, E I L (x) I < 1I to II + D. For each t E T, we have B(t, D) = T, so M0(t) = f L d p doesn't depend on t. We also have E I L (x) - Mk (x) I < D/2k for each x and k. It follows that for each x in T, a.s. as k -+ oc, (2.7.23)
Mk(x) - L(x).
The rest of the proof will bound the increments of the processes Mk and their rate of convergence to L.
Let Y(u, v) := (L(u) - L(v))/d(u, v) if u v and 0 if u = v. Then Y is jointly measurable on H x H, EY(u, v) = 0, and EY(u, v)2 < 1 for all , letting pi (u) := p; (t, u), u, v E H. For each t E T and k = 0, 1, 2, k
Mk(t) - MO = T, MM(t) - Mj-1(t), j=1
and
IMj(t) - Mj-1(t)I = f Y(u, v)d(u, v)pj(u)pj-l(v)dtc(u)d1t(v)
< 3D.2-i. f I Y(u, v)1pj(u)pj-1(v) dl-L(u) dlt(v) Applying Lemma 2.7.20(b) to W := T x T with f(u, v) := pj(u)pj_1(v)
for j = 1, 2, .. and s x µ gives a random variable z > 0 with Ez < 5/2, not depending on n or t, and a set 21 C 0 with P(21) = 1 such that for all
c0E221,n EN,n> 1, and t E T, I Mn (t, (o) - Mn-1(t, (0)1 <
9D2 ((o)
[log
I+
)] 1/2
lcn (t)µn-1(t)
21/29Dz 2n (co ) clog (1
+
1
t-1n (t)
)]1/2
72
Gaussian Measures and Processes; Sample Continuity
since An (t) < An_1(t) for all n > 1 and t while log(l + x2) < 2log(1 + x) for all x > 0. Now, letting h(t, r) :_ [log(1 + µ(B(t, r))-1)]1/2
log (1 + µk(t)-1) < h(t, r)2
(2.7.24)
for r < D/2k. Applying this for r > D/2k+1 for each k > 1, we get 00
D/2
E I Mn - M,,-11(t, w) < 18z(c))21/2 n=1
h(t, r) dr.
J
For x > 1, we have (log(l +x))112 < (log(2x))1/2 < (log2)1/2 + (log X)112. So for co E 521, the series M0 + Fool Mn(t, (o) - Mn_1(t, (v) converges absolutely and converges to a limit
M(t, (v) := M(t)(c)) := limk,oo Mk(t, (0) Thus by (2.7.23), for each t, M(t, w) = L(t)(w) a.s., so M(., ) gives a version of Then we have D/2
(2.7.25)
IM(t, w) - Mo(cv)I < 18.21/2z(c))
h(t, r)dr.
J0
Next, for any s and tin T, take the unique integer k = k(s, t) > 0 such that (2.7.26)
D/2k+1 < d(s, t) < D/2k.
Then
IM(s) - M(t)I < IM(s) - Mk(s)I + IMk(S) - Mk(t)I + IM(t) - Mk(t)I. The proof of (2.7.25) yields (2.7.27)
I (M - Mk) (t, w) I
< 18.2 1/2Z(CO)
I
2-k-1 D
h(t, r) dr
and likewise for s in place of t. We have (2.7.28)
Mk(S) - Mk(t)
= f Y(u, v)d(u, v)pk(s, u)pk(t, v)dis(u)d,c(v). By the definition of pk (2.7.22) and (2.7.26), where the integrand in (2.7.28) is nonzero, we have d(u, v) < 3D/2k. Thus for all w, by Lemma 2.7.20(b) again,
I Mk(S) - Mk(t)I <
9Dz
[log I 1 +
1
111/2
µk(S)µk(t)l J
2.7 Majorizing measures
73
It follows as in (2.7.24) that
IMk(S) - Mk(t)I D/2k+1
[108(1
++l0) 1+
1/2
g
µ(B(t, r))
µ(B(s, r)) )
For r > 0 let µ(r) := µs,t(r) := min (It (B (t, r), B(s, r))). Then D/2k+1
1/2
1
[log 1+
I Mk(s) - Mk(t)I < 18.2 1/2Z I
)1dr.
For any nonincreasing function f > 0 and S > 0 we have S
S/2
f(r)dr.
(1. f
f(r)dr < 2 fo
Recalling (2.7.27) for s and t, we then have for w E 521,
1
f D/2k+2
GM(s) - M(t) I
< 154zJ
[log (1
1 /2
+ µ(r))J
dr.
Since d(s, t) > D/2k+1 by (2.7.26), for r < D/2k+2 the balls B(t, r) and B(s, r) are disjoint and so µ(r) < 1/2. For x > 2, we have log(1 + x) (log 3) (log 2) -1 log(x) by first derivatives. Thus
[log (1 +
< [log2 log
/t(r))J
log 3
log2)
log 3
1/2
1
i/2
A(r)
1/2
1
Llog
1/2
1
\lt(B(t, r)))]
1
)]1/2
+ [log \it(B(s, r))
Then (2.7.19) follows with Y(w) := 194z(w). Since Ez < 2.5 from Lemma 2.7.20, we get EY < 485 as stated. So, Theorem 2.7.18 is proved. Now to prove Theorem 2.7.1, by Theorem 2.7.18 there is a version M of L such that inequality (2.7.19) holds, and therefore letting t be fixed and s vary,
EL(C)* = EM(C)* and (2.7.29)
EM(C)* <
f
D
1
[log (/2(B(x, r)) )]
1/2
dr.
Since EY(w) < 485, EM(C)* < 970y (C) and the proof of Theorem 2.7.1 is complete. Theorem 2.7.18 also directly implies a sufficient condition for the GC property if the majorizing measure condition can be strengthened, as follows.
Gaussian Measures and Processes; Sample Continuity
74
2.7.30 Corollary Let H be a Hilbert space and C C H. Suppose that there exists a probability measure µ on C such that 1
b
limo1 suPxec Jo
to
g \µ(B(x, r))
1/2
dr = 0.
Then C is a GC-set in H.
2.8 Sample continuity and compactness This section will show that for a Gaussian process Xt indexed by a compact metric space, or other suitable parameter space such as an open or closed set in a Euclidean space, sample continuity reduces to that of the isonormal process on
some subsets, and continuity of the nonrandom function t H EXt (Corollary 2.8.5).
Let (T, T) and (W, U) be two topological spaces. Let {Xt, t E T) be a stochastic process defined over a probability space (0, B, P) with values in W, meaning that for each t c- T and Borel set B C W, Xt 1(B) E 13. (Recall that the or-algebra of Borel sets is generated by the open sets and that it's equivalent to assume Xt 1(U) E 13 for each U E U.) Let {Yt, t E T) be another process with values in W, possibly defined over a different probability space (7', B', P'). Recall that the processes {Xt } and {Yt } have the same laws iff for every n = 1, 2, and t1, t" E T, the law of on the product or-algebra in W" is the same as that of {Ytj} j-1 Recall that two processes with the same laws are said to be versions of each other. A process {Xt}tET will be called version-continuous if there is a process Y with the same laws such that for all co' E S2', t H Yt (w') is continuous from T into W. (Equivalently, continuity need only hold for almost all co' because then without changing the laws, for the set of measure 0 of values of co' for which Yt ((o') is not continuous, one can replace it by a fixed continuous function, say having a constant value in W.) Now let W = ll8 with usual topology and suppose {Xt, t E T) is a Gaussian process. Suppose also that (T, e) is a metric space with the metric topology on T. We have 2.8.1 Theorem A Gaussian process { Xt } indexed by a metric space T, defined on a probability space (0, P), is version-continuous if and only if both
(a) the nonrandom function t H EXt is continuous, and (b) the process {Xt - EXt} is version-continuous. Then, t i--
is continuous into L2(P).
2.8 Sample continuity and compactness
75
Proof "If" is clear. To prove "only if," suppose Xt is sample-continuous. For any sequence t, -+ t E T, sample continuity implies that Xtn -+ Xt almost surely and therefore in probability. For jointly Gaussian random variables, convergence in probability is equivalent to convergence in L2(P), since for the Gaussian variables Y := Xt - Xt to converge to 0 in probability, the means EY must converge to 0, and so must the variances. Thus EXtn -+ EXt. Since T is a metric space, (a) follows. Then by subtracting the continuous function EXt, (b) follows.
So, in studying sample continuity or version-continuity of Gaussian processes, we may as well restrict ourselves to processes with mean 0. Let Xt be such a process, t E T. Each Xt is an element of a Hilbert space H, namely L2(P). Consider the isonormal process L on this H. Then since L is Gaussian, has mean 0, and preserves covariances, we see that L(Xt) has the same laws as Xt. If h(.) is a continuous function from T into a Hilbert space H, with range C :_ {h(t) : t E T}, and if L restricted to C is version-continuous, then the process L o h is clearly version-continuous. Conversely, if (T, e) is compact and h is 1-1, then h is a homeomorphism (RAP, Theorem 2.2.11). Then version continuity of L on C and of L oh on T are equivalent. So for (T, e) compact and t i-* Xt one-to-one, version continuity of the Gaussian process Xt reduces to that of L on a subset C of H. (Theorem 2.8.2 and Corollary 2.8.5 will show that the 1-1 assumption is not actually necessary.) If T is locally compact, for example an open or closed subset of some Rk, then continuity is equivalent to continuity on each compact subset.
Let T be a set and d a pseudometric on T: for all x, y, z E T, d(x, y) _ d(y, x) andd(x, z) < d(x, y)+d(y, z), d(x, x) = 0, butpossiblyd(x, y) = 0 for some x y. Recall that for a set S C T, the diameter (with respect to d) is defined by diam S := diamd S := sup{d(x, y) : x, y E S}. The next fact holds for general, not necessarily Gaussian, processes.
2.8.2 Theorem If (T, e) is a compact metric space, h is a continuous function from T onto a metric space K, and Y(x, w), x E K, co E S2, is a stochastic process on K with values in a complete separable metric space S, then Y o h is version-continuous on T if and only if Y is on K.
Remark. If Y(x, w) - Y(x), so Y a nonrandom function, then the result is a known fact in general topology (RAP, Theorem 2.2.11). The difficulty in the proof here is that if Y o h is version-continuous, it is not clear that the
Gaussian Measures and Processes; Sample Continuity
76
corresponding sample-continuous process X can be written as Y' o h for a process Y' on K. Proof "If" is obvious. Conversely, let Y o h be version-continuous and take a process X on T with the same laws as Y o h and t i-+ X(t, co) := Xt(c)) continuous on T for all co. Let p and be the metrics on K and S, respectively. Let d be the pseudometric on T defined by d(s, t) := p(h(s), h(t)). Let A be a countable dense subset of T, and B := h(A) := {h(a) : a E A), so B is a countable dense subset of K. For y, 8 > 0 and any countable set F C T, define a random variable by
D(F, 3, y) := sup { (Xs, Xt) : s, t E F, d (s, t) < 8, e(s, t) < y). This is measurable since F is countable and S is separable (RAP, Proposition 4.1.7). Let D(F, 8) := D(F, 8, 1 + diame T), and
UC := n00U "0(D(A, 1/m) < 1/n), n=1 m=1
so that UC is measurable. If P(UC) = 1, then the sample functions t H Xt (w) are almost surely uniformly continuous with respect to d on A. Since A is countable and Y o h has the same laws as X, Y o h also has sample functions
almost surely uniformly continuous for d on A. Equivalently, Y has sample functions almost surely uniformly continuous for p on B. For any x E K let Y'(x) := lim{Y(u) : u -+ x, u E B). Almost surely, all these limits exist (by uniform continuity and since S is complete), and x i-+ Y'(x) is continuous. Since X and Y' o h both have continuous sample functions on T and have the same law on A, they have the same laws on T, and so does Y o h by choice of X. It follows that Y' has the same laws as Y on K, so Y is version-continuous as desired. Otherwise, P(UC) < 1. Then for some s > 0,
infs>oP(D(A, S) > 3e) > 3e. Then by inclusion (monotone convergence, with 8 = 1/m), (2.8.3)
P n {D(A, 8) > 3s) I > 3s.
\ s>o
/
On the other hand, continuity oft H Xt and compactness of T imply that for some y > 0 and any countable set C C T, (2.8.4)
P {D (C, 1 + diamdT, y) > e) < E.
2.8 Sample continuity and compactness
(Otherwise, take a countable union of countable sets for y = 1, 2,
77
1/n, n =
, to get a contradiction.)
T is a finite union of e-open sets Tj with diame Tj < y. For each i j, if there are s E T, and t E Tj with h(s) = h(t), then we say (i, j) E G, and let us choose and fix such s = s(i, j) and t = t(i, j). Let C be the union of A and the set of all s(i, j) and t(i, j). Since G is finite, we can assume that XX(i, j) (w) = Xt (t, j) (w) for all co and (i, j) E G. Let
J := n {D(C, 1/n) > 3s} f1 {D(C, 1 + diamd T, y) < e}. 0" n=1
Then (2.8.3) and (2.8.4) give P(J) > 3s - s > E. Fix an co E J and choose sn E C and to E C such that d(s, tn) < 1/n and (X,s,,, Xtn)((o) > 3e. By compactness, we can assume that the sequences sn and tn both converge fore and
hence also ford. Let sn -* s and to - t. Then d (s, t) = 0, XSn (w) -+ X5 (w), and Xt,, (co) Xt (co) as n oo. Lets E T, and t E Tj. If i = j we have, since
co E J, 3e < (X5, Xt)(co) < s, a contradiction. If i # j, then (i, j) E G. For n large enough, s, E T, and to E Tj, so
3s <
(Xn, XtJ (w) <
(XSn, XS(i,j)) ((0) +
(Xt(1,j), X,J ((0)
< s + s = 2e < 3s, again a contradiction. Now recall that a totally bounded set C in a Hilbert space H is called a GC-set if L restricted to C has a version with uniformly continuous sample functions.
2.8.5 Corollary A Gaussian process {Xt, t E T } with mean 0 on a compact metric space (T, e) is version-continuous if and only if both t H Xt EH L2(P) is continuous and its range K is a GC-set.
Proof Apply Theorem 2.8.2 with h(t) := K := h(T), p the usual metric in H, and S = R with its usual metric; again has the same laws as Xt. If Xt is version-continuous, then t i-* X, by Theorem 2.8.1, and the rest follows.
is continuous into H
Example. Let Xt be a Gaussian process defined for t E R, periodic of period 2n, so that Xt Xt+2n for all t, with E((Xt - X5)2) > 0 for Is - tI < 27r. Then we can write the process as Xt = Y(ett) where Y is a process indexed by the unit circle T1 :_ (z: I z I = 1) in the complex plane, which is compact. is 1-1 from Version continuity for X and for Y are equivalent, and z t-a
78
Gaussian Measures and Processes; Sample Continuity
Tt into H := L2(P), so version continuity is equivalent to that of L on the range of Y in H (without needing Theorem 2.8.2 and Corollary 2.8.5). On the other hand, any process indexed by R is version-continuous if and only if it is
so on each compact interval [-N, N], where in this example for N > n, the process is not 1-1 into H. Recall that a sample function of a stochastic process Xt is a function t H Xt (w) for a fixed w. The usual metric on a Hilbert space is the natural one for an isonormal process, but the GC-property holds for other metrics in the following sense:
2.8.6 Theorem Let C be a subset of a Hilbert space H. Then the following are equivalent:
(I) C is a GC-set; (II) L on C has a version with bounded, uniformly continuous sample functions; (III) there exists a metric p on C such that (C, p) is totally bounded, and the sample functions of the isonormal process L on C can be chosen to be p-uniformly continuous a.s.
Proof (I) implies (H) since by definition a GC-set is totally bounded, so a uniformly continuous function on it must be bounded. (II) implies (III) directly where p is the usual metric. Suppose (III) holds. Take a version of L such that on a set of probability 1,
the sample functions of L are p-uniformly continuous on C. Then L extends to a Gaussian process t i-± Xt on the compact completion M of C for p. Here Xt is version-continuous, and so by Corollary 2.8.5, C is included in, and thus is, a GC-set.
**2.9 Volumes, mixed volumes, and ellipsoids There will be no proofs in this section.
2.9.1 Volumes Any closed set C in a finite-dimensional Euclidean space Rk has a well-defined
volume, its Lebesgue measure V(C) := Vk(C) :_ Ak(C), invariant under Euclidean transformations (rotations, reflections, translations, etc.). Let C be a compact set in a Hilbert space H of dimension > k. Then Vk (C)
will be defined by Vk(C) := sup Vk(Jr(C)) where the supremum is over all
* *2.9 Volumes, mixed volumes, and ellipsoids
79
orthogonal projections 7r with k-dimensional range. Define the exponent of volume of C by
EV(C) := lim
(log V (C))/(n log n).
The exponent of entropy of C is defined by
r(C) := lim sup,10 (log log D(e, C))/(log(1/s)).
Thus C is a GC-set if r(C) < 2 by Theorem 2.6.1 and is not a GB-set if r(C) > 2 by the Sudakov-Chevet theorem (2.3.5). Milman and Pisier (1987) (see also Pisier (1989)) proved the following relation between r(C) and EV (C): Theorem (Milman and Pisier) Let C be a convex, symmetric, compact subset of
a Hilbert space. Suppose that for some 3 > 0 and M < oo, Vk(C) < Mk-'-' for all k = 1, 2, .. Then C is a GC-set and
EV(C) _
1
1
r(C)
2
2.9.2 Mixed volumes and mean widths
Let C be a convex set in a Euclidean space Rk and let B be the unit ball
B:={x: IxI <1}.Forr>OletC+rB:={x+ry: xEC, yE B). Then it is known that Vk(C + rB) is a polynomial of degree kin r, namely k
Vk(C+rB) = L.rk_ j(C) j=0
where ,Bj (C) is called the jth mixed volume of C (see, for example, Bonnesen and Fenchel (1934, section 29) or Leichtweiss (1980, Satz 15.4 p. 162); other normalizations of the mixed volumes can be found in the literature). We have fk (C) = Vk (C). Here if C is viewed as a subset of a space of higher dimension,
its mixed volumes may change, so more precisely one can write fj(C) = ,Bj,k(C). If C is a straight line segment, then 01 (C) is proportional to its length.
Let h1(C) := )81,k(C)/4 where k is such that for a line segment S, h1(S) equals the length of the segment for each k. Now if C is a compact, convex set in a separable, possibly infinite-dimensional Hilbert space and 2rm are finitedimensional projections increasing up to the identity, then h (C)) increase
up to a limit h1(C) < +oo.
Gaussian Measures and Processes; Sample Continuity
80
Let Sk-1 be the unit sphere in Rk, Sk-1 :_ {x E Rk : (x) = 1). For any convex set C in IRk and x E Sk-1, C has a width wx(C) in the direction x defined by
wx(C) := suPZEc(x, z) - infyec(x, Y) is the usual inner product. Let cok_1 be the unique rotationally where invariant Borel probability measure on Sk-1 (normalized (k - 1)-dimensional
surface area measure). Then the mean width of C is defined by w(C) := f wx (C)dwk_ 1(x). It is known that for each k, w (C) is proportional to h 1(C) or to #t (C) for convex sets C; see, for example, McMullen (1993, Theorem 5.5 p. 970), who refers for a proof to Hadwiger (1957). Sudakov (1971, Theorem 4, and 1973) discovered that:
Theorem (Sudakov) A compact, convex set C in a Hilbert space is a GB-set if and only if h 1(C) < oo. In fact,
h1(C) = (2n)1/2EL(C)*. Indeed, it is easily seen that for each k, EL (C)* for a convex set C in Rk is proportional to the mean width. Since for a finite-dimensional set C, EL (C)* doesn't depend on the dimension of the space in which C is imbedded, and neither does h 1(C), one can find the proportionality constant between h 1(C) and EL(C)* by evaluating both on a line segment.
2.9.3 Ellipsoids Let H be a separable, infinite-dimensional Hilbert space. Then a subset E of H will be called a (compact) ellipsoid if there exists an orthonormal basis {en }n> 1 of H such that for some sequence an 4. 0, we have 00
E = E({an})
jx=xnenEH: n=1
00
>(x,,/a,,)2 < 1
.
n=1
Then E is called a Schmidt ellipsoid if an2 < oo. Suppose for X E E we write L (x) xn Gn where Gn = L (en) are i.i.d. N(0, 1) variables, and then En xn Gn = En (xn /an )an Gn. We then have a.s. L(E({an}))* = En a2nG,2)112, where "<" holds by the Cauchy inequality and holds since for finite sums, the maximum over x is attained when xn/an is proportional to anGn. It follows easily that E is a GB-set if and only if it is a Schmidt ellipsoid. Approximating ellipsoids by finite-dimensional ones, it is also not hard to show that a Schmidt ellipsoid is a GC-set.
* *2.9 Volumes, mixed volumes, and ellipsoids
81
Let J be a possibly different separable, infinite-dimensional Hilbert space and let A be a bounded linear operator from J into H. Let B1 be the unit ball of
J, B1 := {x E J: IIxII < 11. Let A(B1) := {A(x) : x E B1}. Then A is called a compact operator if A(B1) is compact. The compact ellipsoids are exactly the sets A(B1) for compact operators A. The operator A is called a Hilbert-Schmidt operator if A(B1) is a Schmidt ellipsoid. Suppose that J = L2 (Y, v) and H = L2 (X, s) for some measure spaces (X, A, µ) and (Y, B, v). One such representation is always possible where X and Y are countably infinite sets (orthonormal
bases) and t and v are counting measures so that J and H are f2 spaces. The operator A is a Hilbert-Schmidt operator if and only if it can be written as an integral operator with a kernel K E L2(X X Y, g x v), namely A(f)(x) := f K (x, y) f (y) d v (y). If { fk }k> 1 and {gj }j> 1 are any orthonormal bases of J and H, respectively, the above criterion is equivalent to >k II A fk 112 < oo or to
Ek (A fk, gj)2 < oo. The latter two sums do not depend on the choices of the orthonormal bases. Further, these criteria do not require finding an orthonormal basis {e } of H in which the ellipsoid is diagonalized, as in the definition of Thus, the question whether an ellipsoid is a GB- or GC-set, in other words a Schmidt ellipsoid, is most easily settled directly rather than by computing its metric entropy or finding a majorizing measure. Still it can be interesting to see how the different kinds of criteria fit together. By Theorems 2.7.1 and 2.7.2, an ellipsoid has a majorizing measure on it if and only if it is a Schmidt ellipsoid. Ledoux and Talagrand (1991, pp. 448-453) construct explicit majorizing measures on Schmidt ellipsoids. For metric entropy, there is a specific criterion
for ellipsoids. Let H(s) := log D(s, E). Then the ellipsoid E is a Schmidt ellipsoid if and only if f0 s2dH(s) > -oo. Sudakov (1969, (4)) stated this criterion. It follows from more general results of Mityagin (1961) for ellipsoids in Banach spaces, as Marcus (1974) showed. Integrating by parts and using the Sudakov-Chevet theorem (2.3.5), one can see that an equivalent criterion is
f0 sH(s)ds < oo. Suppose that H(s) = log D(s, C) is of the order l/(s2[log(1/s)]`) ass j, 0 for some constant c. Then for general compact sets C in Hilbert space, c > 0 is necessary for the GB or GC property by the Sudakov-Chevet theorem, c > 2 is sufficient by Theorem 2.6.1, and both conditions are sharp for some families of sets. For ellipsoids we see that c > 1 is necessary and sufficient for the GB and GC properties. This is strictly intermediate between the best necessary (2.3.5) and best sufficient (2.6.1) conditions on metric entropy for the GB property. It follows that on some Schmidt ellipsoids, there is no probability measure m satisfying the approximate homogeneity condition in Theorem 2.7.4.
Gaussian Measures and Processes; Sample Continuity
82
**2.10 Convex hulls of sequences Two facts proved by Talagrand (1987 (5), Theorems 2 and 3), not explicitly in terms of majorizing measures, will be stated without proof, then some related, easier facts will be proved. Let co(A) denote the closed convex hull of a set A (smallest closed, convex set including A), and Lx := max(1, logx).
2.10.1 Theorem (Talagrand, 1987) C is a GB-set if and only if for some M < oo and gn E H with Ilgn II < M for all n,
C C co ({gn/(Ln)1I2}n>1) . 2.10.2 Theorem (Talagrand, 1987) C is a GC-set if and only if the same holds with IIgnII -* 0.
The next fact, an easy part of Theorem 2.10.1, will be proved.
2.10.3 Proposition If C = {gn}, where Ilgnll = O((logn)-1/2), then y(C) < oo, and C and J(C) are GB-sets. Proof Since constant multiples of C preserve y (C) < oo, we can assume Ilgn II < (Ln)-112 for all n. Let mn := c/(n(Ln)2) where c is such that En mn = 1. Let m :_ En mn8gn. To show that m is a majorizing mea-
sure, we need to consider m (B(gn, e)) for each n. Fix n and let c(j) := cj := (Ln)-t/2 + (Lj)-1"2. Then Ilgn - gj II < cj. Ifs > cj, then m(B(gn, e)) >
F-i> j c/(i(Li)2) > c/Lj. Break up the range of integration [0, oo) as the union of the intervals [0, CO, [cn, cn-1), c(n)
fo
[log (1/m (B (gn,
8)))11"2ds
, [c4, c3), [c3, oo). We have
< cn ( log [n(Ln)2/c])1/2
< 2(Ln)-1/2(Ln + 2LLn - log c)112 which is bounded in n. Next, the integral from c/+1 to cj is bounded by (cj -
cj+1)(L(L(j + 1)/c))1/2. By the mean value theorem, we get cj - cj+t = (Lj)-1I2 - (L(j + 1))-1/2 < 1/(2 j(Lj)3/2). The sum from j = 3 ton - 1 of (LL(j + 1) - logc)1/2/(2j(Lj)312) is bounded in n. The integral from c3 to oc is the integral from 2 to oo, which is 0, plus the integral from c3 to 2, which is bounded above by 2(L3/c)1/2 for all n (using logy < y for all y), so m is a majorizing measure.
To show {gn } is a GB-set, Proposition 2.2.1 gives P(I L (gn) I > M) < exp(-M2Ln/2), which is summable inn for M > 21/2, and the Borel-Cantelli
Problems
83
lemma applies. The set Y of finite convex combinations with rational coefficients of the gn is countable and dense in co({gn}), and L is a.s. bounded on Y, so co({g, }) is a GB-set by Lemma 2.5.1 and its proof.
Let en be orthonormal in H and let an
2.10.4 Proposition
,{
0, C =
{anen }n>1. Then the following are equivalent:
(I) C is a GB-set;
(II) an = O((Ln)-1"2);
(III) y(C) < 00.
Proof (I) implies (II) by Theorem 2.3.5. (II) implies (I) and (III) by Proposition
2.10.3. To show that (III) implies (II), let m be a majorizing measure and mn := m{anen}. Then for each n, 00
00 > ym(C) ? f (log(1/m(B(anen, 8))))1/2 de. 0
The integral is bounded below by the integral from 0 to an, which is at least
Let J
Ym(C)2 < oo. Then mn > exp(-J/an), E. m, = 1, and exp(-J/an) is nonincreasing inn, son exp(-J/an) < 1, a(log(1/mn))1/2.
and an <
(J/Ln)1/2.
Problems 1. If X and Y are i.i.d. N(0, 1), evaluate E max(X, Y). 2. Evaluate E exp(a II X 112) (finite or infinite) as a function of a > 0 if
(a) G(X) = N(0,1) in R, IIXII = IXI; (b) G(X) = N(0, C) in R2, C = (0 ), and II (xl , x2) I =
(x1 +x22)1 2.
3. Let H be a Hilbert space with orthonormal basis {ej}j>1. Let Gn be independent with laws N(0, (a) Under what conditions on does F_ Gnen converge almost surely in the norm of H? Hint: Apply the 3-series theorem (RAP, 9.7.3) to suitable real-valued random variables.
(b) If G = En Gnen in H as in (a), where the sum converges almost surely, find for what a > 0 we have E exp(a II G II2) < 00. After doing this directly, compare it with the results of Section 2.2.
4. Let Gn be i.i.d. N(0, 1) variables and an > 0 for each n. Under what conditions on an is Y-n an I Gn I < oo a.s.? Hint: Use the 3-series theorem.
Gaussian Measures and Processes; Sample Continuity
84
5. (a) Show that for any set A in a real vector space V and for any vector space W of linear forms on V, the polar A*1 of A, defined by A*1 := {w E W: w(v) < 1 for all v E A}, is convex in W. (b) Let C be a set and D its convex hull, that is, the smallest convex set including C. Show that D*1 = C*1.
(c) In I[82 let C be the unit square {0 < x < 1, 0 < y < 1). Evaluate the polar C*1.
6. Let H be a Hilbert space with orthonormal basis {en }n> 1. For c > 0 let E((cnIn>1) :_ (En>1 xnen : Y_n>1 xn/cn < 1), an infinite-dimensional ellipsoid. Show that E is a GB-set if and only if n c,2 < oo. 7. With notation as in the previous problem, let C {en /(log n) 1/2 : n > 21. Show that C is not a GC-set (it is a GB-set as shown in the example before Theorem 2.3.7).
8. Let 1/i be a real-valued characteristic function on R, where 1/i(t) _ °0 00 e`xt dP(x) for some symmetric law P on R, P(A) = P(-A) for
f
all Borel sets A. Show that there exists a Gaussian process Xt, t E R, with mean 0 and covariance EXSXt = 1/i(s - t) for all real s, t.
9. (a) For each t > 0 let flit be a "triangle function" on R with *t (0) = 1, and for some t > 0, Tit(s) = 0 whenever Isi > t, while flit is linear on each interval [-t, 0] and [0, t]. Show that 1/!t satisfies the conditions of the previous problem. Hint: Find its (inverse) Fourier transform and show that it is a probability density.
(b) Let 1/i be a continuous real-valued function on R which is even, 1/i(-x) = 1/i(x), 1/i(0) = 1, and on [0, oo), 1 is nonincreasing, nonnegative, and convex. Show that 1 is a characteristic function. Hint: Use a mixture of triangle functions. First consider the case that 1/i is piecewise linear and is 0 outside some finite interval. Take increasing limits of such piecewise linear functions.
10. Force > 0let 1/i(x) = 1 - (log(l/Ix1))-' forx in some neighborhood of 0 (piecewise linear elsewhere). (a) Show that there exists such a 1/i satisfying the conditions of problem 8, assuming problem 9. (b) What can be said about sample-continuity of the Gaussian process
Xt for different values of a? 11. Let p be a majorizing measure on a metric space (T, d). (a) If T is a subset of a Hilbert space H with usual metric, then T is a GB-set by Theorem 2.7.1 and is totally bounded by Theorem 2.3.5. Prove directly, in general, that (T, d) is totally bounded. Hint: If for some r > 0 there are infinitely many disjoint balls B(x1, r),
Problems
85
then inf1 µ(B(x;, r)) = 0, and l/p (B(x1, s)) is nonincreasing in s for each i.
(b) Let p be another probability measure on (T, d). If 0 < A < 1, show that A s + (1 - .k)p is also a majorizing measure on (T, d). 12. Let Xt (w) Fn=_,,o G, (co)etnt where Gn are independent random variables with laws N(0, Q,2) and >n Qn < oo. Show that the process t i-+ Xt is sample-continuous if and only if {Xt () : 0 < t < 2ir } is a GC-set in
L2(0) 13. Let z be as in Lemma 2.7.20. Show that E(z2) < 100/9. 14. Let e be orthonormal in H and C :_ {an (log n)-112en }n>2 where an --> 0 asn --> co. (a) Show that every such set is a GC-set.
0 slowly enough, show that for any r > 0 there (b) By taking a exist GC-sets C with D(e, C) > exp[1/(s2J log ej'')] for s > 0 small enough.
15. (A further extension of problem 10). For c > 0 let fi(x) := 1 (log(1/IxI))-1(log log(1/lxI))-c forx in some neighborhood of O (piecewise linear elsewhere).
(a) Show that there exists such a f satisfying the conditions of problem 8, assuming problem 9. (b) What can be said about sample continuity of the Gaussian process Xt for different values of c? (c) Show that for any r < 2, there exist GC-sets C with D(e, C) < exp(1/(s2l logejT) for s > 0 small enough. Compare this with problem 14, part (b), to see that for 0 < r < 2, one cannot tell from D(e, C) whether C is a GC-set or not. 16. Let vk be the Lebesgue volume of the unit ball in Rk. Then it is known that . For any c, > 0, the ellipsoid Vk = nk/2/ l,(1 + (k/2)) f o r k = 1 , 2, [X: rk1 x2/c? < 11 C Rk has volume VkC1 ck £k E({c; }k1)
for any ci > 0, i = 1,
,
k. For s > 0 let m := D(s, E).
(a) Show that m > cic2 . . . ck/sk.
(b) If cj > s for j = 1, , k, show that m (s/2)k < 2kc1 c2 . . . ck. (c) If c j = j -" f o r j = 1 , 2, , give upper and lower bounds for D(e, E) as s . 0 and k - oo. Hint: Recall Stirling's formula k!/[(k/e)k(22rk)112]
1 ask - oo (Theorem 1.3.13).
17. If g is a Young-Orlicz modulus, x2 = o(log g(x)) as x -> +oo, and Y is a N(0, 1) random variable, show that 11 YIIg = +oo.
86
Gaussian Measures and Processes; Sample Continuity
18. If g is a Young-Orlicz modulus, log g(x)) = 0(x2) as x -+ +oo, and Y is a N(0, 1) random variable, show that II YIIg < oo.
19. Let f(x) = r-xfor0<x
f ^- c + Y, an sin(nx) + bn cos(nx) n=1
in L2((-7r, 7r)) and use it to prove Yn_1 n-2 = n2/6.
Notes
Notes to Section 2.2. Proposition 2.2.1 improves on RAP, Lemma 12.1.6(b), by a factor of 2. Lemma 2.2.5 was first proved by Landau and Shepp (1971), then by Femique (1970), whose note giving a much shorter proof appeared in print earlier. The main theorems 2.2.2 and 2.2.8, giving the best possible upper bound for a, were then proved independently by Marcus and Shepp (1971) and Fernique (1971). The current exposition is based mainly, though not entirely, on Fernique (1975). The Darmois-Skitovic theorem says that if X1, , X, are independent real random variables where for some constants a 1, , an is independent with a,b1 $ 0, then for that i, Xi has a normal distribution; see Darmois (1951) and Skitovic (1954). C. R. Rao (1973, pp. 158-163 and 218) lists various characterizations of normal distributions. Notes to Section 2.3. Slepian (1962, Lemma 1) proved his inequality. Sudakov
(1969) announced a result somewhat weaker than Theorem 2.3.5. Sudakov (1971, Theorem 5) announced a stronger result, corrected in Sudakov (1973) to be exactly Theorem 2.3.5. Lemma 2.3.6 is essentially from Chevet (1970). Fernique (1975, Theoreme 2.1.2, Corollaire 2.1.3) proved Theorem 2.3.7. The proof given is as in Fernique (1997, pp. 59, 63-67), who mentions an idea of Kahane; see Kahane (1986).
Notes to Section 2.4.
Theorems 2.4.1 and 2.4.4 and Corollaries 2.4.5 and 2.4.6, and essentially Corollary 2.4.7, are due to T. W. Anderson (1955). These
facts were to some extent rediscovered by L. Gross (1962). Both used the Brunn-Minkowski inequality in their proofs. See also Borell (1974, 1975a,b), Y. Gordon (1985), and Kahane (1986).
Notes to Section 2.5. Lemma 2.5.2 is essentially due to Anderson (1955). Lemmas 2.5.3 and 2.5.4 are straightforward. In Dudley (1967, Theorem 4.6),
Notes
87
parts of which are due to Gross (1962), most of the conditions in Theorem 2.5.5 were proved equivalent for convex, symmetric sets. Feldman (1971) proved part (a'), about the symmetric convex hull of C.
Notes to Section 2.6. In apparently the first public use of a metric entropy hypothesis to obtain sample continuity of Gaussian processes, V. N. Sudakov, in a talk at the International Congress of Mathematicians in Moscow, 1966, announced that if log N(e, C) = O(e-') as e . 0 for some r < 2, then C is a GC-set. His first publication on the topic was Sudakov (1969). Specific bounds on metric entropy implying sample continuity were suggested by previous work
of Fernique (1964) for processes indexed by a real parameter. The fact that f o" (log N(e, C))1t2ds < oo implies C is a GC-set was given (with an equivalent
series instead of the integral) in Dudley (1967). Theorem 2.6.1 and its proof were given in Dudley (1973). Theorem 2.6.2 and its proof are based (with some changes) on Ledoux and Talagrand (1991, Section 11.1), where the theorem is extended, with different functions go, to a large class of non-Gaussian stochastic processes. Komatsu's inequality, used in the proof of Lemma 2.6.7, is quoted,
with hints for the proof, in Ito and McKean (1974, p. 17), citing as original source Komatsu (1955).
Talagrand's original proof (1987) of Theorem 2.7.2 was quite long. Talagrand (1992) published a proof of the majorizing measure theorem (2.7.2) which is much shorter, although some steps at the end were left to the reader. Talagrand also gave other reasonably short proofs of Theorem 2.7.2. Building on these, Fernique (1997) gave a proof which has largely been followed here. Specifically, Lemma 2.7.6 is Lemme 3.4.5 of Fernique (1997), Lemma 2.7.9 is his Lemme 5.3.5, and so on, up to changes in constants. Specifically, some constants were allowed to become larger to make the proof more self-contained. Fernique (1975) first proved Theorem 2.7.1, namely that existence of a majorizing measure implies the GB property. The proof given here is from Fernique (1997, pp. 100-101, 104-111) with minor changes. Thus, for example, Theorem 2.7.18 is Fernique's Theoreme 5.2.6 and Lemma 2.7.20 is his Lemme 5.2.7. As given here, the proofs of Theorems 2.7.1 and 2.7.2 turn out to be about equally long. The author wrote out the proofs of the much easier facts: Proposition 2.7.3, Theorem 2.7.4, and Corollary 2.7.5.
Notes to Section 2.7.
Notes to Section 2.8. I have no reference for Theorem 2.8.2. The facts in this section on Gaussian processes were to some extent stated, but not proved, in
88
Gaussian Measures and Processes; Sample Continuity
Dudley (1967, 1973). Andersen and Dobric (1988, Lemma 8, (4.6); preprint, 1985) first published proofs of Corollary 2.8.5 and Theorem 2.8.6 according to Gine and Zinn (1986, p. 58).
Note to Section 2.9. The section incorporates its own notes. Notes to Section 2.10. Talagrand (1987) proved Theorems 2.10.1 and 2.10.2. The author (without claiming any priority) proved the easier Propositions 2.10.3 and 2.10.4.
References *An asterisk indicates a work I have seen discussed in secondary sources but not in the original. Andersen, Niels Trolle, and Dobric, Vladimir (1988). The central limit theorem for stochastic processes II. J. Theoret. Probab. 1, 287-303. Anderson, Theodore W. (1955). The integral of a symmetric unimodal function over a symmetric convex set and some probability inequalities. Proc. Amer.
Math. Soc. 6, 170-176. Bonnesen, T., and Fenchel, W. (1934). Theorie der konvexen Kdrper. Berlin, Springer; repr. Chelsea, New York, 1948. Borell, Christer (1974). Convex measures on locally convex spaces. Ark. Mat. 12, 239-252. Borell, C. (1975a). Convex set functions in d-space. Period. Math. Hungar. 6, 111-136. Borell, C. (1975b). The Brunn-Minkowski inequality in Gauss space. Invent. Math. 30, 207-216. Chevet, Simone (1970). Mesures de Radon sur Rn et mesures cylindriques. Ann. Fac. Sci. Univ. Clermont#43 (math., 6° fasc.), 91-158. Darmois, Georges (1951). Sur une propriete caracteristique de la loi de probabilite de Laplace. Comptes Rendus Acad. Sci. Paris 232, 1999-2000.
Dudley, R. M. (1967). The sizes of compact subsets of Hilbert space and continuity of Gaussian processes. J. Funct. Anal. 1, 290-330. Dudley, R. M. (1973). Sample functions of the Gaussian process. Ann. Probab.
1, 66-103. Dunford, N., and Schwartz, J. T. (1958). Linear Operators, Part I: General Theory. Interscience. New York; repr. Wiley, New York (1988). Feldman, Jacob (1972). Sets of boundedness and continuity for the canonical normal process. Proc. Sixth Berkeley Symposium Math. Statist. Prob. 2, pp. 357-367. Univ. of Calif. Press, Berkeley and Los Angeles.
References
89
Fernique, Xavier (1964). Continuite des processus gaussiens. Comptes Rendus Acad. Sci. Paris 258, 6058-6060.
Fernique, X. (1970). Integrabilite des vecteurs gaussiens. Comptes Rendus Acad. Sci. Paris Sir. A 270, 1698-1699. Fernique, X. (1971). Regularite de processus gaussiens. Invent. Math. 12, 304-320. Femique, X. (1975). Regularite des trajectoires des fonctions aleatoires gaussiennes. Ecole d'ete de probabilites de St.-Flour, 1974. Lecture Notes in Math. (Springer) 480, 1-96. Fernique, X. (1985). Sur la convergence etroite des mesures gaussiennes. Z. Wahrscheinlichkeitsth. verw. Geb. 68, 331-336. Fernique, X. (1997). Fonctions aleatoires gaussiennes, vecteurs aleatoires gaussiens. Publications du Centre de Recherches Mathematiques, Montreal.
Gine, Evarist, and Zinn, Joel (1986). Lectures on the central limit theorem for empirical processes. In Probability and Banach Spaces, Proc. Conf. Zaragoza, 1985, Lecture Notes in Math. (Springer) 1221, 50-113. Gordon, Yehoram (1985). Some inequalities for Gaussian processes and appli-
cations. Israel J. Math. 50, 265-289. Gross, Leonard (1962). Measurable functions on Hilbert space. Trans. Amer. Math. Soc. 105, 372-390. Hadwiger, H. (1957). Vorlesungen fiber Inhalt, Oberflache and Isoperimetrie. Springer, Berlin. Ito, Kiyosi, and McKean, H. P. Jr. (1974). Diffusion processes and their sample paths. Springer, Berlin, New York. Kahane, Jean-Pierre (1986). Une inegalite du type de Slepian et Gordon sur les processus gaussiens. Israel J. Math. 55, 109-110. *Komatsu, Y. (1955). Elementary inequalities for Mills' ratio. Rep. Statist. Appl. Res. Un. Japan. Sci. Engrs. 4, 69-70. Landau, H. J., and Shepp, Lawrence A. (1971). On the supremum of a Gaussian process. Sankhya Ser. A 32, 369-378. Ledoux, M., and Talagrand, M. (1991). Probability in Banach Spaces. Springer, Berlin. Lehmann, E. L. (1986). Testing Statistical Hypotheses, 2d ed. Wiley, New York; repr. 1997. Leichtweiss, K. (1980). Konvexe Mengen. Springer, Berlin. Marcus, M. B. (1974). The e-entropy of some compact subsets of V. J. Approximation Th. 10, 304-312.
90
Gaussian Measures and Processes; Sample Continuity
Marcus, Michael B., and Shepp, L. A. (1972). Sample behavior of Gaussian processes. Proc. Sixth Berkeley Symp. Math. Statist. Prob. (1970) 2, 423441. Univ. of Calif. Press, Berkeley and Los Angeles. McMullen, P. (1993). Valuations and dissections. In Handbook of Convex Geometry, vol. B, eds. P. M. Gruber, J. M. Wills, North-Holland, Amsterdam. Milman, V. D., and Pisier, G. (1987). Gaussian processes and mixed volumes. Ann. Probab. 15, 292-304. Mityagin, B. S. (1961). Approximate dimension and bases in nuclear spaces. Russian Math. Surveys 16, no. 4, 59-127. Pisier, G. (1989). The Volume of Convex Bodies and Banach Space Geometry. Cambridge University Press, New York. Rao, C. R. (1973). Linear Statistical Inference and Its Applications, 2d ed. Wiley, New York.
Skitovic, V. P. (1954). Linear combinations of independent random variables and the normal distribution law. Izv. Akad. Nauk SSSR Ser. Mat. 18, 185-200 (in Russian). Slepian, David (1962). The one-sided barrier problem for Gaussian noise. Bell Syst. Tech. J. 41, 463-501. Sudakov, V. N. (1969). Gaussian measures, Cauchy measures and s-entropy. Soviet Math. Dokl. 10, 310-313. Sudakov, V. N. (1971). Gaussian random processes and measures of solid angles in Hilbert space. Soviet Math. Dokl. 12, 412-415. Sudakov, V. N. (1973). A remark on the criterion of continuity of Gaussian sample function. In Proc. Second Japan-USSR Symp. Probab. Theory, Kyoto, 1972, ed. G. Maruyama, Yu. V. Prokhorov; Lecture Notes in Math. (Springer) 330, 444-454. Talagrand, M. (1987). Regularity of Gaussian processes. Acta Math. (Sweden) 159, 99-149. Talagrand, M. (1992). A simple proof of the majorizing measure theorem. Geom. Funct. Anal. 2, 118-125.
3
Foundations of Uniform Central Limit Theorems: Donsker Classes
3.1 Definitions: convergence in law Let (S, 13, P) be a probability space, to be called the sample space. Examples
to have in mind for S are Euclidean spaces such as the plane R2. To form empirical measures, we'd like to take variables X1, X2, , i.i.d. with law P. To do this, take a countable product S°O of copies of (S, 13, P) (RAP, Theorem
8.2.2) and let X; be the coordinates on the product. A product may be taken with another probability space. Throughout the rest of this book, X, will be defined as such coordinates unless something is said to the contrary. This way of defining Xi will be called the standard model. An example showing that use of the standard model makes a difference will be given at the end of Section 5.3. Recall that the product or -algebra is the smallest for which all the coordinates a r e measurable, and that 8 (A) = lA(x). Then, we can form the empirical measures Pn
- rn
x1
i=1
and the empirical process
vn := n1"2(Pn - P). So Pn is a probability measure on S, defined on B and actually on all subsets of S, for any values of X1, , Xn. Each vn is a finite signed measure of total charge 0. As n -* oo, for any set A E 13, vn(A) converges in law to a normal law
with mean 0 and variance P(A)(1 - P(A)). Next, a general analogue of the Brownian bridge will be defined. Let GP be a Gaussian process indexed by the set G2(P) of measurable, square-integrable functions for P, having
mean 0 and covariance EGp(f)Gp(g) = f fgdP - f f dP f gdP. To see 91
92
Uniform Central Limit Theorems: Donsker Classes
that such a Gaussian process exists, we need to check that the given covariance is nonnegative definite, which holds since it is just the covariance for P, in other words
EGp (.f) Gp (g) = (.f g)o, p
ov p (.f, g) Cf(f_ffdP)(g_fgaP)dP ,
and such a covariance is always nonnegative definite.
Let G2(P) be the set of all functions f E L2(p) such that f f dP = 0. Let L2(P) be the set of all equivalence classes of elements of L2(p) for equality a.s. (P). On L2(P), the covariance for Gp reduces to the ordinary G2 semiinner product. In other words, restricted to the subspace L2(P), Gp is an isonormal process. Let C be the one-dimensional space of constant functions c as a subspace of G2(P). Then Gp is 0 on C, while the spaces C and L2(P) are orthogonal complements of each other (RAP, Theorem 5.3.8) in G2(P). For
any f and gin G2(P) and c E 1t8, Gp(cf + g) = cGp(f) + Gp(g) a.s., since as is easily checked, Gp(cf + g) - cGp(f) - Gp(g) has mean and variance both 0.
The covariance for P defines a pseudometric on G2(P) by pp(f, g)
(E(Gp(f) -
Gp(g))2)112. Then pp equals the usual L2(p) pseudometric on
L2 (p). Moreover, with respect to the semi-inner product ( , -)0, p on L2 (p), Gp is the isonormal process and pp the usual pseudometric. Thus, the results of Chapter 2 apply to sample continuity of Gp with respect to pp. The Brownian bridge process yr is a special case of the Gp process where P is Lebesgue measure on [0, 1] and yt = Gp(1[0,(1); it is easily checked that this has the right covariance. The Brownian bridge process can be taken to have continuous sample paths, in other words to be continuous as a function of t for each w (RAP, Theorem 12.1.5). But Gp is not sample-continuous on the whole space G2(P). Recall
that an atom of a measure space (0, S, s) is a set A E S with µ(A) > 0 such that for all S E S, µ(S fl A) = 0 or it (A). A measure having no atoms is called nonatomic. In fact, unless P is concentrated in finitely many atoms, the spaces G2(P) and L2(P) are infinite-dimensional, in the sense that they contain infinite orthonormal sets, and we saw in Section 2.6 that an isonormal process on an infinite-dimensional Hilbert space H is not sample-continuous.
We will be concerned then with suitable subsets of £2(P). A class F C G2 will be called pregaussian if a Gp process (f, (o) i-4 Gp(f)(w) can be defined on some probability space such that for each w, f i-+ Gp (f) (w) is bounded and uniformly continuous for pp from .F into R.
3.1 Definitions: convergence in law
93
Given f e G2(P), let rro(f) := f - f f dP. Then 7ro(f) E G4(P). Given F C G2(P), let Jro(.P) be the set of all functions 7ro(f), f E .P. For any f E G2(P), pp(f, no(f )) = 0, and Gp(f) = Gp(7ro(f )) a.s. I claim that F is pregaussian if and only if rro(.P) is: if iro(.P) is pregaus-
sian, then f r- Gp(Iro(f)) has the desired properties on .P, while if .P is pregaussian, then for any g E 7ro(.F), select arbitrarily (by the axiom of
choice!) an fg E F with Iro(fg) = g. Then g H Gp(fg) has the desired properties.
Now recall the definitions of GB-set and GC-set from Section 2.5. Note that if a set C in a Hilbert space is a GB-set, then it must be totally bounded by Theorem 2.3.5. If C is also a GC-set, then uniformly continuous sample functions of the isonormal process on C must be bounded. Thus since Gp is isonormal on L2 (P), any set 9 C G4(P) is pregaussian if and only if the corresponding set in L2(P) is a GC-set. Recall the notion of prelinear function (Lemma 2.5.3). A Gp process Y on a class F C G2 (S, 8, P) for a probability space (S, 13, P) will be called coherent if for each (o E S, the function f F-+ Y(f) (w) on.F is bounded, pp-uniformly continuous, and prelinear.
3.1.1 Theorem Given a probability space (S, B, P) and F C £2(S, 8, P), .P is pregaussian if and only if there exists a coherent Gp process on F.
Proof "If" follows from the definition of pregaussian. For the converse, note that no above is linear and Gp is isonormal on Go (S,13, P), and apply Theorem 2.5.5.
Now, if a class F C G2(P) is pregaussian we can ask whether v converges in distribution, or in law, to Gp with respect to uniform convergence over F. Recall that for random variables Y, with values in a separable metric space S, convergence in law of Y to Yo is defined classically to mean that Eg(Y,,) Eg(Yo) as n -+ oo for every bounded continuous function g on S. But empirical processes, even empirical distribution functions as treated in Section 1.1, take values in nonseparable metric spaces, and for an arbitrary bounded continuous g, g(v,) may not be measurable. A new definition of convergence in law is needed to take care of this nonmeasurability. The definition will involve the notion of upper integral. Let g be a real-valued, not necessarily measurable function defined on a space X where (X, S, µ) is a measure space. Let R be the set [-oo, oo] of extended real numbers. Then the upper integral is defined
Uniform Central Limit Theorems: Donsker Classes
94
by
f gdµ := inf if h dµ : h > g, h measurable and R-valued I
,
which will be undefined if there exists a measurable h > g with f hdµ = oo-oo undefined, unless there is also a measurable i/r > g with f *dµ = -oo, in which case f * gdµ will also be defined as -oo. There always exists at least one measurable h > g, namely h - +00. We will be dealing often with compositions of functions. If f is a function whose domain includes the range of g then either f (g) or fog will denote the
function such that (f o g)(x) - f (g(x)). A function, which may not be measurable, from a probability space into a metric space will be called a random element. Now here is a definition of convergence in law, where only the limit variable necessarily has a law:
Let (S, d) be any metric space. Let (Q,, A,z , Q,) be probability spaces for n = 0, 1, 2, .. , and Yn , n > 0, functions from Stn into S. Suppose that Yo takes values in some separable Definition (J. Hoffmann-Jorgensen).
subset of S and is measurable for the Borel sets on its range. Then Y, will be Yo, if for every said to converge to Yo in law as n -+ oo, in symbols Y, bounded continuous real-valued function g on S,
fg(Yfl)dQfl
- fg(Yo)dQoasnoo.
For g bounded, f * g(YY) dP is always defined and finite. Then, here is a general definition of when the central limit theorem for empirical measures holds with respect to uniform convergence over a class F of functions. The metric space S will be the space £°O(F) of all bounded real-valued functions on F, with the metric given by the supremum norm IIHII,, := sup{IH(f)I :
fc.T}. Definition. Let (S2, A, P) be a probability space and F c G2(P). Then F will be called a Donsker class for P, or P-Donsker class, or be said to satisfy the central limit theorem (for empirical measures) for P, if F is pregaussian
for P and vn = Gp in Later on, a number of rather large classes F of functions will be shown to be Donsker classes for various laws P. The next few sections develop some of the needed theory.
3.2 Measurable cover functions
95
3.2 Measurable cover functions In the last section, convergence in law was defined in terms of upper integrals. The notion of upper integral is related to that of measurable cover. Let (7, A, P) be a probability space. Then for a possibly nonmeasurable set A C S2, a set B is called a measurable cover of A if A C B, B E A,
and P(B) = inf{P(C) : A C C, C measurable). For A C B E A with P(A), a measurable cover of A is n,, B,,. If B and C are measurable
covers of the same set A, then clearly so is B fl C. It follows that B = C up to a set of measure 0, in other words P(BOC) = 0 where A denotes the symmetric difference, or equivalently P (1 B = 1 c) = 1. For any set A C S2 let P*(A) := inf{P(B) : B measurable, A C B}. Then for any measurable cover B of A, clearly P*(A) = P(B). Let r0 := &A A, P, 9) denote the set of all measurable functions from S2 into R. Then G° is a lattice: for any f, g E G°, f v g := max(f, g) and f n g := min(f, g) are in G°. But this LO is not a vector space since we could have, for example, f = +oo and g = -oo, so f + g would be undefined. The map y i-* tan-1 y is one-to-one from ]l8 onto [-7r/2,7r/21. Then a
metric on R is defined from the usual metric on [-ir/2, r/2] by d (x, y) := I tan-1 x - tan-1 yl. On Lo we have the Ky Fan metric (RAP, Theorem 9.2.2) defined by
d(f,g) := inf{E>0: P(d(f(x),g(x))>s)<E}. Then d (f, g) = 0 if and only if P (f = g) = 1. For any set j C Ge(S2, A, P, R), a function f E Lo is called an essential
infimum of J, or f := ess.inf J, if for all j c- ,7, f < j a.s. and for any g E LO such that g < j a.s. for all j E ,7, we have g < f a.s. If f and g are two essential infima of the same set ,7, then clearly f = g a.s. A set ,7 of functions will be called a lower semilattice if for any f, g E .7, we have
min(f, g) E J. 3.2.1 Theorem For any probability space (0, A, P) and class ,7 C Go(S2, A, P, R), an essential infimum of ,7 exists. If for some function f : S2 i-+ R we have j = [j E Lo : j > f everywhere), then f * := ess.inf ,7 can be chosen so that f * > f everywhere. Also, f f * dP and E*f := f * f dP are both defined and equal if either of them is well-defined (possibly infinite), for example if f * is bounded below. P r o o f Let j i be the class of all functions min (fl ,
, f , , , ) for fl,
, f E .7
and m = 1, 2, . Then JI is a lower semilattice. For f E G°, f = ess.inf ,71 if and only if f = ess.inf J. So we can assume ,7 is a lower semilattice. For
96
Uniform Central Limit Theorems: Donsker Classes
each j E J, tan-1 j is a measurable function with values in [-7r/2, r/2]. Take jn E J such that f tan-1 j f tan-1 j dP. Then min(J1, , j,,,) is in Ji and decreases to g as m - oo for some g E Ge(S2, A, P, ]R). For any h E J, min(h, j1, , jm) J, min(h, g) so f tan- Imin(h, g) = f tan '(g) and g < h a.s., so g satisfies the definition of ess.inf J. If 3 = (h E LO : h > f) then J is a lower semilattice and g > f everywhere. By the definitions, f * f dP < f f * dP if either side is well-defined, and the inequality is an equation by the definition of essential infimum.
Here f * is called the measurable cover function of f. Recall that in Chapter 2, L (A)* was the essential supremum of L (x) for x E A, and so the essential infimum of random variables Y such that for each x E A, Y > L (x) a.s. - a different, although related, notion. If f is real-valued and bounded above by some finite-valued measurable function then f * is a measurable real-valued function. But whenever there exist nonmeasurable sets A 4. 0 with P* 1, as for Lebesgue measure (e.g., RAP, Section 3.4, problem 2), let f := n on An \
Then f is real-valued but f * _ +oo a.s. The next two lemmas on measurable cover functions are basic.
3.2.2 Lemma For any two functions f, g: S2 H (-oo, oo], we have
(a) (f + g)* < f * + g* a.s., and (b) (f - g)* > f - g* whenever both sides are defined a.s. Proof (a) We have -oo < f * < +oo and -oo < g* < +oo everywere, so f * +g* is an everywhere defined, measurable function > f+g, and (a) follows. For part (b), on the measurable set where g* = +oo, where by assumption f is finite a.s., the right side is -oo and the inequality holds. Where g* is finite,
g is also finite and f = (f - g) + g. Then f * < (f - g)* + g* by (a), so f * - g* < (f - g)*, since this holds where (f - g)* < oo and where
(f -g)*=oo. 3.2.3 Lemma Let S be a vector space with a seminorm II II. Then for any two functions X, Y from 0 into S,
IIX+ YII* < (11XII + IIYII)* < IIXII* + IIYII* a.s. and IIcXII* = Ici IIXII* a.s. for any real c. Proof The first inequality is clear, the second follows from Lemma 3.2.2, and the equation is clear (for c = 0 and c # 0).
3.2 Measurable cover functions
97
Next, in some cases of independence, the upper-star operation can be distributed over products or sums.
3.2.4 Lemma Let (2j, Aj, Pj), j = 1,
, n, be any n probability spaces. Let fj be functions from Q j into R. Suppose either
(a) f >0, j= 1, - - , n, or (b) fl = 1 and n = 2. Then on the Cartesian product IInj_1(S2j, Aj, Pj) with x := (xl,
f(x) :=
, xn), if
fj(xj), we have f*(x) = nj=1 f.*(xj) a.s., where 0 oo is set
equal to 0.
(c) Or, if fj(xj) > - ooforallxj, j = fl (XI) + ... + fn (xn ), then g* (xi, ... , xn) = fl* (x i) + .
+ fn (xn )
a.s.
Proof First, for (c), by induction we can assume n = 2. Clearly g*(x, y) < f l* (x)+ f2 (y) a.s., and if equality does not hold a. s., there is a rational t such that
on a set C of positive probability in the product space, g* (x, y) < t < fl* (x) +
f2 (y), and there exist rational q, r with q + r > t such that C can be chosen
with fl*(x) > q and f2 (y) > r for (x, y) E C. Let Cx := {y: (x, y) E C}. By the Tonelli-Fubini theorem there is a set D C 01 with PI (D) > 0 such that
P2(Cx) > 0 for all x E D. If fi < q on D then fl* < q a.s. on D, but for any x E D and y E Cx # 0 we have fl* (x) > q, a contradiction. So choose and fix an x E D with fl (x) > q. Then for any y E Cx, q + f2 (y) < fi (x) + f2 (y) < g* (x, y), so f2 (y) < g* (x, y)-q and f2 (y) < g* (x, y)-q for almost all y E C. For any such y, q + f2 (y) < q +r and f2 (y) < r, a contradiction. So (c) is proved. Now for products, in case (a) or (b), clearly f *(x) < nj=1 f.* (xj) a.s., with 1 * = 1. For the converse inequality we can assume n = 2, by induction in case (a). Suppose f *(x) < fl (xl) f2 (x2) with positive probability. Then for some
rational r, f * (x) < r < fl*(x1) f2 (x2) with positive probability. If fj - 1, this gives f (x) < f * (x) < r < f2 (x2) on a set of positive probability. Then by _
the Tonelli-Fubini theorem, for some x1, f2(x2) < f *(xl, x2) < r < f2 (x2) on a set of x2 with P2 > 0, contradicting the choice of f2*' So assume fi > 0 and f2 > 0. Then as in case (c), there are rationals a, b
with ab > r, a > 0, and b > 0, such that on a set C in the product with positive probability, fl*(xi) > a, f2 (x2) > b, and f*(x1,x2) < r. Again by the Tonelli-Fubini theorem, there is a set D C 01 with PI (D) > 0 and P2(Cu) > 0, u E D, and there is a point u of D where fl (u) > a. Then for any v E C, f2(v) < f *(u, v)/a, so f2 (v) < f *(u, v)/a for almost all
98
Uniform Central Limit Theorems: Donsker Classes
v E G. For such a v we have aft (v) < ab and f2 (v) < b, a contradiction, finishing the proof.
For the next fact here is some notation: given two functions f, g and a a-
algebra S on the range of f, let (f, g)(x) := (f (x), g(x)) and f -1(S) { f -'(A): A E S}. 3.2.5 Lemma Let (0, A, P) = II3_1(521, Si, PI) with coordinate projections 111: 111(x1, x2, x3) := xi, i = 1, 2, 3. Let Si ®S2 denote the product a-algebra on S21 x 522. Then for any bounded real function f on 01 x 523 and 9(x1, x2, x3) := f (x1, x3), conditional expectations of g* satisfy
E(g*1(1-11, 112)-1(S1 (9 S2)) = E(g*I11-1(S1)) a.s.for P.
Proof By Lemma 3.2.4(b), for 22 X (S21 x 523), g* equals P-almost surely a measurable function not depending on x2, thus independent of 11-1(S2). Let S be the collection of all sets A E (111, f12)-1(S1 0 S2) such that g* and h := E(g* 111-1(Si )) have the same integral over A. Then S contains all finite
disjoint unions of sets (Hi, n2)-1(B1 x B2) = IIi 1(B1) n n21(B2), Bi E Si, i = 1, 2, since both g* and h are independent of 17121(S2). Now S is easily seen to be a monotone class, so it equals all of (111, H2)-1(S1 0 S2) (RAP, Theorem 4.4.2).
3.2.6 Lemma Let X be a real-valued function on a probability space (0, A, P). Then for any t E R, (a) P* (X > t) = P(X* > t);
(b) foranye>0, P*(X>t)t)t-e). Proof Clearly {X > t} c {X* > t) and {X > t) c {X* > t}, so we have "<" in (a) and the first inequality in (b).
Take a measurable cover A D {X > t), so P*(X > t) = P(A). Then X* < t a.s. outside A, so P(X* > t) < P(A), proving (a). Thus for 0 < 8 <
e, P(X* > t - 8) < P*(X > t - s). Letting 8 4, 0 proves (b). Let (0, A, P) be aprobability space. For a function f from Q into [-oo, oo]
let f* f dP := sup{ f gdP : g measurable, g < f}. Let f* be the essential supremum of all measurable functions g < f. Then just as for f*, f* is well-defined up to a.s equality and f* f dP = f f* dP whenever either side is defined, as in Theorem 3.2.1.
3.2 Measurable cover functions
99
It's easy to check that f* = -((- f)*) and that f* f dP = -(f * - f dP). So the convergence in law Y, = Yo as defined in Section 3.1 implies that f* g(Y-,) dQn - f g(Yo) dQoNext, here is a one-sided Tonelli-Fubini theorem for starred functions:
3.2.7 Theorem Let (X, A, P) x (Y, B, Q) be a product of two probability spaces. For a real-valued function f > 0 on X x Y, define f * with respect to P x Q. For each x E X let (E* f)(x) := f * f (x, y) dQ(y). Then
EiE2f(x,Y)
(3.2.8)
ff*(x,y)d(PxQ)(x,y),
where Ei = E* with respect to P. Also, if Q is purely atomic, so that >j Q({yj)) = 1 for some yj E Y, and (3.2.9)
f - dQ, then
Ei E2f(X, Y) < E*f(X, Y) = E2Ei f(X, Y).
Proof By the usual Tonelli-Fubini theorem, we have (3.2.10)
f
f*(x, y)d(P x Q)(Y) = ff f*(x, y)d Q(y) dP(x),
and f f *(x, y)d Q(y) is measurable in x. For each x E X let f (y) := f (x, y). Then f(x, y) < f *(x, y), which is measurable in y, so f,, (y) < f *(x, y) for Q-almost all y. Thus since E* fX = Efx by Theorem 3.2.1,
(E*f)(x) =
f
f,*(Y)dQ(Y) <
f
f*(x,Y)dQ(Y),
and (3.2.8) follows. Next, to prove (3.2.9), note that on Y, since Q is purely atomic, all real-valued functions are measurable (for the completion of Q), so E2 - E2, the left inequality follows from (3.2.8), and the equality from (3.2.10), so (3.2.9) is proved. We also have a one-sided monotone convergence theorem with stars:
3.2.11 Theorem Let (S2, A, P) be a probability space and let f be realvalued functions on Q such that f f f, that is, f (x) T f (x) for all x E Q. If
E* fl > -oo then E*fj T E*f as j -+ oo.
Proof By Theorem 3.2.1, E*fj = Ef * for each j, and as j -+ oo, f increases a.s. up to some function g where clearly g < f * almost surely. If
g < f * on some set A with P(A) > 0, then f < g on A implies f < g and so f * < g on A, a contradiction. The theorem follows.
100
Uniform Central Limit Theorems: Donsker Classes
Recall that for any measure space (X, A, µ) and any C C X, the outer measure is defined by s*(C) := inf(µ(A) : A E A, A D C). Note that there exist subsets Aj := A(j) of [0, 1] with outer measure k*(Aj) = I for all j and Aj J, 0 (RAP, Problem 3.4.2). Letting f := IA(.i), we have f 4, 0, and E* f = 1 for all j, so that the monotone convergence theorem fails for E* on decreasing sequences. Next is a Fatou lemma with stars.
3.2.12 Theorem Let (Q, A, P) be a probability space and f any nonnegative real-valued functions on 0. Then E* lim infs.,, f < lim infj,0 E* f j. Proof For 1 < k < m we have infra>k f, < fm and so E* fn < E* fm, thus E* infn>k fn < infra>k E*fm. Taking the supremum over k on both sides and using the previous monotone convergence theorem gives the result.
3.3 Almost uniform convergence and convergence in outer probability In Section 3.1 the definition of convergence of laws was adapted to define convergence in law for random elements which may not have laws defined. In this section the same will be done for convergence in probability and almost sure convergence. Let (0, A, P) be a probability space, (S, d) a metric space, and fn functions
from S2 into S. Then f, will be said to converge to fo in outer probability if
d(f, fo)* - 0 in probability as n - oo, or equivalently, by Lemma 3.2.6,
for every e > 0, P*(d(f , fo) > E) -k 0 as n -+ oo. Also, f, is said to converge to fo almost uniformly if as n - oo, d(fn, fo)* -k 0 almost surely. The following is immediate:
3.3.1 Proposition Almost uniform convergence always implies convergence in outer probability.
If fn are all measurable functions, then clearly convergence in outer probability is equivalent to the usual convergence in probability, and almost uniform convergence to almost sure convergence. Now some definitions will be given for Glivenko-Cantelli properties, which are laws of large numbers for empirical measures.
Definition. If (X, A, P) is a probability space and .P is a class of integrable real-valued functions, F C Gt (X, A, P), then F will be called a strong (resp.
3.3 Convergence without measurability
101
weak) Glivenko-Cantelli class for P if as n -+ oo, 11 P, - PII.r --+ 0 almost uniformly (resp. in outer probability). In the following proposition, part (C) is what is usually called "Egorov's theorem" for almost surely convergent sequences of measurable functions (RAP, Theorem 7.5.1).
3.3.2 Proposition Let (S2, A, P) be a probability space, (S, d) a metric . Then the following space, and f, any functions from 2 into S for n = 0, 1, are equivalent:
(A) fn - fo almost uniformly. (B) For any e > 0, P*{sup,,>m d(fn, fo) > e} 4, 0 as m -* oo. (C) For any 8 > 0 there is some B E A with P(B) > 1 - 8 such that fn -* fo uniformly on B. (D) There exist measurable hn > d(fn, fo) with hn -+ 0 a.s. Proof (A) implies that (SUpn>m d (fn, f0))* < Sup,,>m (d (fn,
f0)*)
4, 0 a.s. I
as m -+ oo,
which implies (B).
Assuming (B), for k = 1, 2, , let Ck := {supra>m(k) d(f, fo) > 1/k), where m (k) is large enough so that P* (Ck) < 2-k. Take measurable covers B k f o r Ck, so C k C Bk, Bk E A, and P(Bk) < 2-k. For r = 1 , 2, , let Ar := S2 \ Uk>r Bk. Then P(Ar) > 1 - 2-' and fn fo uniformly on Ar, so (C) holds. Now assume (C). Take Ck E A, k = 1, 2, , such that P(Ck) T 1 and f n -+ f o uniformly on Ck. W e can take Cl C C2 C . Take Mk such that
d(f, fo) < 1/k on Ck for all n >_ mk. Then d(fn, fo)* < 1/k on Ck, so d(fn, fo)* --> 0 a.s., proving (A). Clearly, (A) and (D) are equivalent.
Example. In [0, 1] with Lebesgue measure P let A 1 D A2 D . . . be sets with P*(A,) = 1 and fn°_1 An = 0 (e.g., RAP, Section 3.4, problem 2; Cohn, 1980, p. 35). Then 1An -* 0 everywhere and, in that sense, almost surely, but not almost uniformly. Note also that lAn doesn't converge to 0 in law as defined in Section 3.1. To avoid such pathology, almost uniform convergence is helpful.
3.3.3 Proposition Let (S, d) and (Y, e) be two metric spaces and (S2, A, P) a probability space. Let f, be functions f r o m 0 into S for n = 1 , 2,
, such
that f, - fo in outer probability as n -+ oo. Assume that fo has separable
Uniform Central Limit Theorems: Donsker Classes
102
range and is measurable (for the Borel or -algebra on S). Let g be a continuous function from S into Y. Then g(fn) g(o) in outer probability.
Note. If f G (fo) dP is defined for all bounded continuous real G (as it must be if fn fo, by definition) then the image measure P o f0 1 is defined on all Borel subsets of S (RAP, Theorem 7.1.1). Such a law does have a separable support except perhaps in some set-theoretically pathological cases (Appen-
dix Q. , let Bk :_ {x E S: d(x, y) < 1/k implies P r o o f Given e > 0, k = 1, 2, e(g(x), g(y)) < s, y E S}. Then each Bk is closed and Bk T S as k -+ oo. Fix k large enough so that P(f01(Bk)) > 1 - e. Then {e(g(fn), g(.f0)) > s) (1 f0 1(Bk) C {d (fn, .fo)
1/k}.
Thus
P*{e(g(fn),g(f0)) > e} < e+P*{d(fn, fo) > 1/k} < 2s for n large enough.
3.3.4 Lemma Let (S2, A, P) be a probability space and {gn }n o a uniformly bounded sequence of real-valued functions on Q such that go is measurable. If gn -* go in outer probability then lim supn,, f * gn dP < f go dP.
Proof Let I g, (x) I < M < oo for all n and all x E 22. We can assume M = 1. Given s > 0, for n large enough P*(I gn - goI > e) < e. Let An be a measurable set on which Ign - gol < e with P(S2 \ An) < e. Then
J
gndP < e+ J*gndP < 2s+ f A
A An
< 3s+ J $odP.
Letting s 1. 0 completes the proof. On any metric space, the a-algebra will be the Borel or -algebra unless something is said to the contrary.
3.3.5 Corollary If f, are functions from a probability space into a metric space, fn --* fo in outer probability, and fo is measurable with separable range, then fn = fo. Proof Apply Proposition 3.3.3 to g = G for any bounded continuous G and Lemma 3.3.4 to gn := G o fn and also to gn = -G o fn.
3.4 Perfect functions
103
3.4 Perfect functions For a function g defined on a set A let g[A] :_ {g(x) : x E A}. It will be useful that under some conditions on a measurable function g and general real-valued
f, (f o g)* = f * o g. Here are some equivalent conditions: 3.4.1 Theorem Let (X, A, P) be a probability space, (Y, l3) any measurable space, and g a measurable function from X to Y. Let Q be the restriction of P o g t to B. For any real-valued function f on Y, define f *for Q. Then the following are equivalent:
(a) For any A E A there is a B E B with B C g[A] and Q(B) > P(A); (b) for any A E A with P(A) > 0 there is a B E 8 with B C g[A] and Q(B) > 0; (c) for every real function f on Y, (f o g)* = f * o g a.s.; (d) f o r any D C Y, OD o 9)* = 1 *D o g a. s.
Proof Clearly (a) implies (b).
To show (b) implies (c), note that always (f o g)* < f* o g. Suppose (f o g)* < f * o g on a set of positive probability. Then for some rational
r, (fog)* 0.Letg[A]DBEB with Q(B) > 0. Then f o g < r on A implies f < r on B, so f * < r on B a.s., contradicting f * o g > r on A. Clearly (c) implies (d). Now to show (d) implies (a), given A E A, let D := Y \ g[A]. Then we can take 1D = lc for some C E B: let C be the set where 10 > 1. Then D C C
andIDog=(lDog)*=Oa.sonA.LetB:=Y\C.Then BCg[A],and
Q(B) = 1 - Q(C) = 1 -
f 1* d(P o g '),
which by the image measure theorem (RAP, Theorem 4.1.11) equals
1-J 1*DogdP = 1- J(1Dog)*dP > P(A). Note. In (a) or (b), if the direct image g[A] E B, we could just set B := g[A]. But, for any uncountable complete separable metric space Y, there exists a complete separable metric space S (for example, a countable product N°O of copies of N) and a continuous function f from S into Y such that f [B] is not a Borel set in Y (RAP, Theorem 13.2.1, Proposition 13.2.5). If f is only required to be Borel measurable, then S can also be any uncountable complete metric space (RAP, Theorem 13.1.1).
104
Uniform Central Limit Theorems: Donsker Classes
A function g satisfying any of the four conditions in Theorem 3.4.1 will be called perfect or P perfect. Coordinate projections on a product space are, as one would hope, perfect:
3.4.2 Proposition Suppose A = X x Y, P is a product probability v x m, and g is the natural projection of A onto Y. Then g is P-perfect.
Proof Here P o g-1 =m. For any B CA let By := {x : (x, y) E B}, y e Y. If B is measurable, then by the Tonelli-Fubini theorem, for C := {y: v(By) >
0}, C is measurable, C C g[B], and P(B) < m(C), so condition (a) of Theorem 3.4.1 holds.
3.4.3 Theorem Let (0, A, P) be a probability space and (S, d) a metric , (Yn,13,,) is a measurable space, gn a space. Suppose that for n = 0, 1, perfect measurable function from 0 into Y,,, and fn a function from Y into S, where fo has separable range and is measurable. Let Qn := P o gn 1 on 13n and suppose fn o gn - fo o go in outer probability as n --* oo. Then fn = fo as n oo for fn on (Yn, 13,,, Qn)
Before this is proved, an example will be given. Recall that for any measure space (X, S, µ) and C C X, the inner measure is defined by /,t. (C)
sup{µ(A) : A E S, ACC}. 3.4.4 Proposition
Theorem 3.4.3 can fail without the hypothesis that gn be
perfect.
Proof Let C C I := [0, 1] satisfy 0 = X*(C) < X*(C) = 1 for Lebesgue measure A (RAP, Theorem 3.4.4). Let P = A*, giving a probability measure on
the Borel sets of C (RAP, Theorem 3.3.6). Let 0 = C, fo = 0, Yn = I, fn :_ II\C, and let gn be the identity from C into Y,, for all n. Then fn o gn = 0 for all n, so f, o gn -+ fo o go in outer probability (and in any other sense). Let 13,, be the Borel or-algebra on Y,, = I for each n. Let G be the identity from I into IR. Then f * dQn = f * fn da, = 1 for n > 1, while f G(fo) dQo = 0, so fn does not converge to fo in law.
After Theorem 3.4.3 is proved, it will follow that the gn in the last proof are not perfect, as can also be seen directly from condition (c) or (d) in Theorem 3.4.1.
fo o go. Let H Proof of Theorem 3.4.3 By Corollary 3.3.5, fn o g, be any bounded, continuous, real-valued function on S. Then by RAP,
3.4 Perfect functions
105
Theorem 4.1.11,
JH(ffl(gfl))dP - f H(fo(go)) dP
f
H(fo) dQo
Also,
f *H(fn(gn))dP =
f H(fn(gn))*dP
by Theorem 3.2.1
f(Hofn)*(gn)dP
by Theorem 3.4.1
f(Hofn)*dQn
(RAP, Theorem 4. 1.11)
f * H(fn) dQn
by Theorem 3.2.1,
and the theorem follows.
In Proposition 3.4.2, X x Y could be an arbitrary product probability space, but projection is a rather special function. The following fact will show that all measurable functions on reasonable domain spaces are perfect.
Recall that a law P is called tight if sup{P(K) : K compact) = 1. A set P of laws is called uniformly tight if for every e > 0 there is a compact K such that P(K) > 1 - e for all P E P. Also, a metric space (S, d) is called universally measurable (u.m.) if for every law P on the completion of S, S is measurable for the completion of P (RAP, Section 11.5). So any complete metric space, or any Borel set in its completion, is u.m.
3.4.5 Theorem
Let (S, d) be a u.m. separable metric space. Let P be a
probability measure on the Borel or -algebra of S. Then any Borel measurable function g from S into a separable metric space Y is perfect for P.
Note. In view of Appendix C, the hypothesis that S be separable is not very restrictive.
Proof Let A be any Borel set in S with P(A) > 0. Let 0 < s < P(A). By the extended Lusin theorem (Theorem D.1, Appendix D) there is a closed set F with
P(F) > 1-s/2 such that g restricted to F is continuous. Since P is tight (RAP, Theorems 11.5.1 and 7.1.3), there is a compact set K C A with P(K) > s. Then C := F fl K is compact, C C A, P(C) > 0, and g is continuous on C,
so g[C] is compact, g[C] C g[A], and (P o g 1)(g[C]) > P(C) > 0, so the conclusion follows from Theorem 3.4.1.
106
Uniform Central Limit Theorems: Donsker Classes
Let (0, A, P) be a probability space and g a measurable function from S2 into Y where (Y, 8) is a measurable space. Let Q := P o g 1 on B. Call g quasiperfect for P or P-quasiperfect if for every C C Y with g -'(C) E A, C is measurable for the completion of Q. Then the probability space (0, A, P) is called perfect if every real-valued function G on 0, measurable for the usual Borel or -algebra on R, is quasiperfect.
Example. A measurable, quasiperfect function g on a finite set need not be perfect: let X:= {al, a2, a3, a4, a5, a6}, U := {al, a2}, V :_ {a3, a4}, W {a5, a6}, A := {0, U, V, W, U U V, U U W, V U W, X), P(U) = P(V) _
P(W) = 1/3, Y := {0, 1, 21, g(al) := g(a3)
0, g(a2) := g(a5) :=
1,
g(a4) := g(a6) := 2. Let B := {0, Y). For C C Y, g-1 (C) E A if and only if C E B, so g is quasiperfect. But P(U) > 0 and g[U] does not include any nonempty set in 13, so g is not perfect.
3.4.6 Proposition Any perfect function is quasiperfect.
Proof Let C C Y, A := 9_1(C) E A. By Theorem 3.4.1 take B C g[A]
with B E 8 and Q(B) > P(A). Then B C C, so Q(B) = P(g 1(B)) < P(g 1(C)) = P(A), and Q(B) = P(A). Thus the inner measure Q,,(C) = P o g 1(C). Likewise, Q. (Y \ C) = (P o g 1) (Y \ C), so Q*(C) = (P o g 1) (C) and C is Q-completion measurable.
U
3.5 Almost surely convergent realizations First let's recall a theorem of Skorokhod (RAP, Theorem 11.7.2): if (S, d) is a complete separable metric space, and Pn are laws on S converging to a law Po, then on some probability space there exist S-valued measurable functions X, such that P, for all n and X -+ Xo almost surely. This section will prove an extension of Skorokhod's theorem to our current setup. Having almost uniformly convergent realizations shows that the definition of convergence in law for random elements is reasonable. The realizations will be useful in some later proofs on convergence in law.
Suppose f, = fo where f, are random elements, in other words functions not necessarily measurable except for n = 0, defined on some probability spaces (52,,, into a possibly nonseparable metric space S. We want to find random elements Y "having the same laws" as f, for each n such that Yo almost surely or better, almost uniformly. At first look it isn't clear Yn what "having the same laws" should mean for random elements fn, n > 1, not
3.5 Almost surely convergent realizations
107
having laws defined on any nontrivial or-algebra. A way that turns out to work is to define Y, = fn o gn where gn are functions from some other probability space 0 with probability measure Q into Stn such that each gn is measurable and Q o gn1 = Q, for each n. Thus the argument of fn will have the same law Qn as before. It turns out, moreover, that the gn should be not only measurable but perfect. Before stating the theorem, here is an example to show that there may really be no way to define a or -algebra on S on which laws could be defined and yield an equivalence as in the next theorem, even if S is a finite set.
Example. Let (Xn, An, Qn) = ([0, 1], B, A) for all n (), = Lebesgue measure, 13 = Borel a-algebra ). Take sets C(n) C [0, 1] with O = A*(C(n)) < .X*(C(n)) = 1/n2 (RAP, Theorem 3.4.4). Let S be the two-point space {0, 11 with usual metric. Then f,, := lC(n) -* 0 in law and almost uniformly, but each "law" On := Qn o fn 1 is only defined on the trivial a-algebra {0, S). The only larger or -algebra on S is 2S, but no On for n > 1 is defined on 2S.
3.5.1 Theorem Let (S, d) be any metric space, (Xn, An, Qn) any probability . Suppose fo has spaces, and fn a function from Xn into S for each n = 0, 1, separable range So and is measurable (for the Borel a -algebra on So). Then fn = fo if and only if there exist a probability space (S2, S, Q) and perfect , such measurable functions gn from (S2, S) to (Xn, An) for each n = 0, 1, fo o go almost uniformly that Q o gn 1 = Qn on An for each n and fn o gn as n -+ oo. Notes. Proposition 3.4.4 and the "if and only if " in Theorem 3.5.1 show that the hypothesis that gn be perfect can't just be dropped from the theorem.
Proof "If" follows from Proposition 3.3.1 and Theorem 3.4.3. "Only if" will be proved very much as in RAP, Theorem 11.7.2. Let S2 be the Cartesian product fl 0Xn x In where each In is a copy of [0, 1]. Here gn will be the natural projection of 0 onto Xn for each n. Let P := Qo o fo 1 on the Borel or -algebra of S, concentrated in the separable subset So. A set B C S will be called a continuity set (RAP, Section 11.1) for P if P(dB) = 0 where 8B is the boundary of B. Then,
3.5.2 Lemma For any e > 0 there are disjoint open continuity sets Uj, j = 1, , J, for some J < oo, where for each j, diam Uj := sup{d(x, y) :
x,yEUj)<e,andwith 1:;J= 1P(Uj)>1-e.
108
Uniform Central Limit Theorems: Donsker Classes
Proof Let {xjf°° 1 be dense in So. Let B(x, r) :_ {y E S: d(x, y) < r} for 0 < r < oo and x E So. Then B(xj, r) is a continuity set of P for all but at most countably many values of r. Choose rj with e/3 < rj < e/2 such that B(xj, rj) is a continuity set of P for each j. The continuity sets form an algebra (RAP, Proposition 11.1.4). Let
U j := B(xj, rj) \ U{y: d(xi, y) < r,}. i<j
Then Uj are disjoint open continuity sets of diameters < e with
1, so there isaJ
1
P(Uj) _
1P(Uj)> 1-e.
Now to continue the proof of Theorem 3.5.1, for each k = 1, 2, - -, by the last lemma take disjoint open continuity sets Ukj := U(k, j) of P for j = 1, 2, .. -, Jk := J(k) < oo, with diam(Ukj) < 1/k, P(Ukj) > 0, and J(k)
E P(Ukj) > 1-2 -k.
(3.5.3)
j=1
For any open set U in S with complement F, let d(x, F) := inf{d(x, y) y E F). For r = 1, 2, - -, let Fr := {x : d (x, F) > 1/r}. Then Fr is closed and Fr T U as r -* oo. There is a continuous hr on S with 0 < hr < 1, hr = 1 on Fr, and hr = 0 outside F2r: let hr(X) := min(l, max(0, 2rd(x, F) - 1)). For each j and k, let F(k, j) := S \ Ukj. Take r := r(k, j) large enough so that P(F(k, f)r) > (1 - 2-k)P(Ukj). Let hkj be the hr as defined above for such an r and Hkj the her. For n large enough, say for n > nk, we have
jhki(fn)dQn > (1-2-k)P(Ukj) and
f* Hkj(fn)dQn < (1+2-k)P(Ukj) for all j = 1, - - -, Jk. We may assume n1 < n2 < ... . For every n = 0, 1, - , let fkjn := (hkj o fn)* for Qn, so that by Theorem - -
3.2.1,
fhki(fn)dQn =
ffkJndQn,
0 < ,fkjn < hkj(fn),
and fkjn is An-measurable. For n > 1 let Bkjn :_ (.fkjn > 0) E A. Let Bkjo
1(Ukj) E Ao. For each k and n, the Bkjn C f-1
for j= 1,---,Jk,andHkj(fn)=Ion Bkjn, soforn>nk, (3.5.4)
(1-2-k)P(Ukj) < Qn(Bkjn) < (1+2-k)P(Ukj),
are disjoint
3.5 Almost surely convergent realizations
109
and Qo(Bk jo) = P(Ukj). Let Tn := X, x In. Let it, be the product law Q, x .X on Tn where A is Lebesgue measure on the Borel or-algebra B in In. For each
k> 1, Ckjn
:= Bkjn x [0, F(k, j, n)] C T,,
Dkjn
:= Bkjo x [0, G(k, j, n)] C To,
where F and G are defined so that
An(Ckjn) = Ito(Dkjn) = min(Qn(Bkjn), Qo(Bkjo))
Then for each k, j, and n > nk,wehave by(3.5.4),since 1/(1+2-k) > 1-2-k,
(3.5.5) 1-2 -k < min(F, G)(k, j, n) < max(F, G)(k, j, n) = 1. Let J(k) Ckon
7'n \ U Ckjn,
J(k)
DkOn = To \ U Dkjn.
j=1
j=1
For k = 0let Jo := J(0) := 0, Coon Tn, Doon := To, and no := 0. For each n = 1, 2, , let k(n) be the unique k such that nk < n < nk+1 Then for n > 1, Tn is the disjoint union of sets Wnj Ck(n)jn, j = 0, 1, (3.5.6)
, Jk(n). We also have
If j > 1 and (v, s) E Wn j then V E Bk(n) jn so fn (v, S) E Uk(n) j
Next, To is the disjoint union of sets Enj := Dk(n) jn. Then µn (Wnj) = Ito(Enj)
for each n and j, and if j > 1 or k(n) = 0, then Ao(En j) > 0. For x in To and each n, (3.5.7)
let j (n, x) be the j such that x E En j .
Let L := (x E To : so(Enj(n,x)) > 0 for all n). Then To \ L C IJi En(i)o for some (possibly empty or finite) sequence n (i) such that po(En(1)o) = 0 for all i. Thus µo(L) = 1. For X E L and any measurable set B C Tn, in other words B is in the product
or-algebra An 0 B, let j := j(n, x), recall that Itn(Wnj) = t o(Enj), and let (3.5.8)
Pnj(B) := An(B fl Wnj)/Ito(Enj),
Pnx
Pnj(n,x).
Then Pnx is a probability measure on An 0 B. Let px be the product measure 1 -100 Pnx on T := IIn° 1 T, (RAP, Theorem 8.2.2).
Uniform Central Limit Theorems: Donsker Classes
110
3.5.9 Lemma For any measurable set H C T (for the infinite product a algebra with A® ® 13 on each factor), x i-+ px(H) is measurable on (To, Ao ®13).
Proof Let H be the collection of all H for which the assertion holds. Given n, Pnx is one of finitely many laws, each obtained for x in a measurable subset
En j. Thus if Y, is the natural projection of T onto Tn and H = Y,- 1(B) for some B E A®®13 then H E H. If H= ITEM Ym(i)(Bi), where Bi E An(i) 0 13 and M is finite, we may assume the m (i) are distinct. Then I
px(H) = niEMpx(Yn(i)(Bi)), so H E R. Then, any finite, disjoint union of such intersections is in 7-l. Such unions form an algebra. If Hn E 7-1 and Hn T H or Hn ,. H, then H E 7-1. Since the smallest monotone class containing an algebra is a a-algebra (RAP, Theorem 4.4.2), the lemma follows.
Now returning to the proof of Theorem 3.5.1, S2 = To x T. For any set C C S2, measurable for the product a-algebra, and x E To, let C, :_ l y E T : (x, y) E C}, and Q(C) := f px (Cx) dµo(x). Here x r-+ px (Cx) is measurable
if C is a finite union of products Ai x Fi where Ai E Ao ® 13 and Fi is product measurable in T. Such a union equals a disjoint union (RAP, Proposition 3.2.2). Thus by monotone classes again, x i-+ px(Cx) is measurable on To for
any product-measurable set C C Q. Thus Q is defined. It is then clearly a countably additive probability measure.
Let p be the natural projection of T onto Xn. Recall that Pnx = Pnj for all x E En j, by (3.5.7) and (3.5.8). The marginal of Q on X,,, in other words Q o gn 1, is by (3.5.8) again J(k(n))
E lso(Ej)Pnj o p-1 = An o
P-1
= Qn
j=0
Thus Q has marginal Qn on Xn for each n, as desired. By (3.5.3), 00
J(k)
E Q0 X 0 \ U f 1(Ukj) < 1: 2-k < 00. k=1
j=1
k
l
So Qo-almost every y E X o belongs to U f0 1(Uk j) for all large enough k. Also if t E 10 and t < 1, then by (3.5.5), t < G(k, j, n) for all j > 1 as soon
3.6 Conditions equivalent to convergence in law
111
as 1 - 2-k > t and n > nk. Thus for µo-almost all (y, t), there is an in such that (y, t) E U (k(n)) En j for all n > m. If x := (y, t) E En j for j > 1, then y E Bk(n) jo, so fo(y) E Uk(n) j. Also, by (3.5.8), Pnx = Pn j is concentrated in Wnj. For (v, s) E Wnj, fn (V) E Uk(n) j by (3.5.6). Since diam(Ukj) < 1/k for
each j > 1, Q*(d(fn(gn), .Po(go)) > 1/k(n) for some n > m)
< µo({(y, t) E Eno for some n > m}) -+ 0 as in -* oo, so fn (gn) -* fo (go) almost uniformly. Lastly, let's show that the gn are perfect. Suppose Q(A) > 0 for some A.
Now Q(A) = f px(Ax) dµo(x). Firstletn > 1. Thenforsomex, px(Ax) > 0. If µo(Eno) = 0, we take x Eno. Now T = Tn x T(n) where T(n) :_ ff1<(0nT,. Then on T, px = Pnx x Qnx for a law Qnx = fImonPmx on T(n). Let A (x) := A. By the Tonelli-Fubini theorem, px (Ax) = ff 1A(x) (u, v) dPnx (u) dQnx (v) Thus for some v, f 1 A(x) (u, v) dPnx (u) > 0. Choose and fix such a v as well as
x. Now Pnx = Pn j for j = j (n, x) with µo(En j) > 0. Let u = (s, t), s E Xn, and t E In. Then since Pn j = Qn x A restricted to a set of positive measure and normalized,
0 < ff1A(X)(stv)dQfl(s)dt. Choose and fix a t with 0 < f IA(x)(s, t, v)dQn(s). Let C IS E Xn (s, t, v) E Ax}. Then Qn(C) > 0. Clearly C C gn[A], so gn is perfect for n > 1 by Theorem 3.4.1. To show go is perfect, we have µo = Qo x ,., and px (Ax) > 0 for x = (y, t)
in a set X with Ao(X) > 0. There is a t E Io such that Qo(C) > 0 where C := {y: (y, t) E X). Then C C go[A], so go is perfect, finishing the proof of Theorem 3.5.1.
3.6 Conditions equivalent to convergence in law Conditions equivalent to convergence of laws on separable metric spaces are given in the portmanteau theorem (RAP, Theorem 11.1.1) and metrization theorem (RAP, Theorem 11.3.3). Here, the conditions will be extended to general random elements for the theory being developed in this chapter. For any probability space (S2, A, P) and real-valued function f on S2 let
E*f := f * f dP, E* f := f* f dP. If (S, d) is a metric space and f is a
112
Uniform Central Limit Theorems: Donsker Classes
real-valued function on S, recall (RAP, Section 11.2) that the Lipschitz seminorm of f is defined by
IIflIL
Sup{If(x) - f(y)Ild(x, y) : x 0 y)
and f is called a Lipschitz function if II f 11L < oo. The bounded Lipschitz := sup., If(x)j. norm is defined by II f II BL II f II L + II f II where Iif II Then f is called a bounded Lipschitz function if II f II BL < oo, and 11 II BL is a norm on the space of all such functions.
The extended portmanteau theorem about to be proved is an adaptation of RAP, Theorem 11.1.1, and some further facts based on the last section (Theorem
3.5.1). The proof to be given includes relatively easy implications, some of which consist of putting in stars at appropriate places in the proofs in RAP.
3.6.1 Theorem Let (S, d) be any metric space. F o r n = 0, 1 , 2, , let (Xn, An, Q,) be a probability space and fn a function from Xn into S. SupQo o f0 1 on S. pose fo has separable range So and is measurable. Let P Then the following are equivalent: (a) fn = fo;
(a') lim supn,, E*G(fn) < EG(fo) for each bounded continuous realvalued G on S; (b) E*G (fn) --* EG (fo) as n -+ oo for every bounded Lipschitzfunction
GonS; (b') (a') holds for all bounded Lipschitz G on S;
(c) sup{IE*G(fn) - EG(fo)I: IIG1IBL < 1) - Oasn - oo; (d) for any closed F C S, P(F) > lim supn,oo Qn (fn E F); (e) for any open U C S, P(U) < liminfn,oo(Qn)*(fn E U);
(f) for any continuity set A of P in S, Q* (fn E A) - P(A) and (Qn)*(fn E A) -* P(A) as n -* oo; (g) there exist a probability space (S2, S, Q) and measurable functions gn from S2 into Xn and hn from S2 into S such that the gn are perfect,
Qogn1 = Qn and Qohn1 = Pforalln, andd(fn ogn,hn) --). 0 almost uniformly.
Moreover, (g) remains equivalent if any of the following changes are made in it: "almost uniformly" can be replaced by "in outer probability"; we can take hn = fo o yn for some measurable functions yn from S2 into X0, which can be taken to be perfect; and we can take yn to be all the same, yn = yl for all n.
3.6 Conditions equivalent to convergence in law
113
Proof Clearly (a) implies (a'). Conversely, by interchanging G and -G, (a') implies
liminfn*cE*G(fn) > liminfn,,,,E*G(fn) > EG(fo), and (a) follows, so (a) and (a') are equivalent.
Clearly (a) implies (b), which is equivalent to (b') just as (a) is to (a'). To show (b) implies (c), let T be the completion of S. Then all the fn take values in T. Each bounded Lipschitz function G on S extends uniquely to such a function on T, and the functions G o fn on Xn are exactly the same. So we can assume in this step that S and So are complete. Let E > 0. By Ulam's theorem (RAP, Theorem 7.1.4), take a compact K C So with P(K) = Qo(fo E K) > 1 - E. Recall that d (x, K) := inf {d (x, y) :
y E K) and KE := Ix: d(x, K) < E}. Let g(x) := max(O, 1-d(x, K)/e) for 1 + 1/E < oo. Clearly 1K < g < 1KE. Since E*g(fn) -). Eg(fo) as n -+ oo, it follows that x E S. Then g is a bounded Lipschitz function with II9IIBL for n large enough (3.6.2)
(Q,,). (f, E K5) > E*g(fn) > Eg(fo) - E > I - 2E.
Let B be the set of all G on S with II G II BL < 1. Then the functions in B are uniformly equicontinuous, so their restrictions to K are totally bounded for the supremum distance over K by the Arzela-Ascoli theorem (RAP, Theorem 2.4.7). Let G I, , G j for some finite J be functions in B such that for each
G E B, supXEK I(G - Gj)(x)I < E for some j = 1, , J. Next, for any G E B, choose such a j. Then (3.6.3)
I E*G(fn) - EG(fo)I < IE*G(fn) - E*Gj(fn)I + I E*(Gj(fn)) - E(Gj(fo))I + I E(Gj(fo) - G(fo))I,
a sum of three terms. For the last term, splitting the integral into two parts according as fo c K or not, the first part is bounded by E and the second by 2E since I G j - G I < 2 everywhere and Qo (fo K) < E. The middle term on the right side of (3.6.3) is bounded above by
maxj<j IE*Gj(fn) - EGj(fo)I, which is less than E for n large enough by (b). The first term on the right in (3.6.3) is a sum of two parts, one over a measurable subset An of Xn where f, E KE with Q, (A,) > 1- 2E by (3.6.2), and the other over the complement of A. Since G and G j E B, and I G - G j I < E on
K, we haveIG - GjI <3s on K6. It follows that I (Go fn)*-(Gjo fn)*I <3E
Uniform Central Limit Theorems: Donsker Classes
114
on An, so the first part is bounded above by 3e. The second part is bounded by 2(1 - Qn(An)) < 46. So for n large enough the left side of (3.6.3) is less than 116 uniformly for G E B, and (c) follows. Clearly (c) implies (b), so (b) and (c) are equivalent. Next, (b) implies (d) by taking bounded Lipschitz functions gk decreasing down to 1 F as k - oo, specifically the functions g in the proof of (b) implies (c) with F in place of K
and 6 = 1/k. (d) and (e) are equivalent by taking complements. (d) and (e) together easily imply (f). For any bounded continuous function G, all but countably many of the sets
f G < t) are continuity sets of P. It follows that G can be approximated uniformly within e by a simple function F_ ti 1 Ai where Ai are continuity sets {ti-1 < G < ti}. So (f) implies (a), and (a) through (f) are all equivalent. Theorem 3.5.1 says that (a) implies (g), in the strongest form where hn
h 1 - fo o yl and yi is perfect. The weakest form of (g), with convergence in outer probability and hn depending on n and not necessarily a composition of fo with any (perfect) function,
will be shown to imply (b'). Given 6 > 0, take n large enough so that
Q*{d(.fn o gn, hn) > 6) < 6. n So on a measurable set An with Q(An) > 1 - e, we have d(fn o gn, hn) < e, and thus for any G E B (in other words II G II BL < 1) we have
IG(fn(gn)) -G(hn)I < 61A
+2. IA'-
It follows that
E*G(fn(gn)) < EG(hn)+36 = EG(fo)+3e (since hn has the same distribution as fo) and
E*G(fn (gn)) = E(((G o fn) o gn)*) = E((G o fn)* o gn) by Theorem 3.2.1 and since gn is perfect, and where all the expectations are with respect to Q. The latter integral, by the image measure theorem (RAP, Theorem 4.1.11), equals E*(G(fn)) (for Qn). Letting e 4. 0, (b') follows, so all of the forms of (g) are equivalent to any of (a) through (f). Theorem 1.1.1, in a special case, proved a form of convergence which in that case is now easily seen to imply the conditions in Theorem 3.6.1: the one in (c), for example. It will be shown that convergence in law is also equivalent to convergence in some analogues of the Prokhorov and dual-bounded-Lipschitz metrics, which
3.6 Conditions equivalent to convergence in law
115
metrize convergence of laws on separable metric spaces as shown in RAP, Theorem 11.3.3. We will have analogues of metrics, rather than actual metrics, because of the asymmetry between the nonmeasurable random elements f, and the limiting measurable random variable fo.
Definitions. Let (X., Am , Qm) be probability spaces, m = 0, 1, and let (S, d) be a metric space. Let fm be functions from Xm into S, m = 0, 1, such that fo is measurable and has separable range. Let P := Qo o f0 1. Then let
P(.fi, fo) := sup{IE*G(.fl) - EG(fo)I :
IIGIIBL <- 11,
and for F ranging over nonempty closed subsets of S,
p(.fi, .fo) := inf{e > 0: SUPF[P(F) - (Qi)*(fi E FE)] < e}. 3.6.4 Theorem For any metric space (S, d), probability spaces (Xm, Am, Qm) and functions fm from Xm into S, where fo has separable range and is measurable, the following are equivalent: (i) .fm
fo;
(ii) $(fm, Jo)
0 as m
(iii) p(fm, fo) - 0 as m
oo; oo.
Proof (i) and (ii) are equivalent just as (a) and (c) are in Theorem 3.6.1. To show that (ii) implies (iii), we have:
For any fl and fo as in the statement of Theorem 3.6.4, p(Ji, .fo) < min(l, (20(ft,
3.6.5 Lemma
.fo))1/2).
Proof Let F be any closed set in S and P = Qo o f01. Given 0 < s < 1, let g(x) := max(0, 1 - d(x, F)/e) as before. Then 11811 BL < 1 + 1/c, and
(Q1). (f, E FE) > jg(Ji)dQi >
fg(Jo)dQo_(JiJo)(l+) > P(F) - s cases,
if (fi, fo) < r2/2. Clearly p(fl, fo) < 1 in all (2p(.fi, .fo))1/2 when the latter is < 1, the lemma follows.
so letting s
Now to continue the proof of Theorem 3.6.4, applying the lemma to fm in place of fl for each in shows that (ii) implies (iii). To show that (iii) implies , let F k := {x : d (x, y) < 1/k (i), let U be an open set in S. For k = 1 , 2,
116
Uniform Central Limit Theorems: Donsker Classes
implies y E U). Then Fk are closed sets and Fk T U as k -f oo. For n large enough, p(fn, fo) < 1/k, and then P(Fk) < (Qn),k(fn E F1Ik) + k < (Qn)*(fn E U) + k, SO
P(Fk) < liminfn->oo(Qn)*(fn E U) + k Then letting k ->. oo gives P(U) < lim infn oo(Qn). (fn E U). This is (e) of Theorem 3.6.1, so the proof of Theorem 3.6.4 is complete. Next is a version of Theorem 3.6.4 where we have two indices.
3.6.6 Theorem
Let (S, d) be a metric space and (S2, S, Q) a probability
space. Suppose that f o r each m, n = 1 , 2, , fmn is a function from Q into S, and fo is a measurable function from Q into S with separable range. Then the following are equivalent:
(i) fm,n = fo, that is, for every bounded continuous real function G on S,
I*G(fm,n)dQ
f G (.f0) dQ asm,n -+ oo;
(ii) f(fmn, fo) - O as m, n -> oo; (iii) p(fmn, fo) -+ 0 as m, n -+ oo. Proof For a double sequence amn of real numbers, let lim SUpm
n,oo amn =
infk SUpm>k,n>k amn,
lim infm, n-#oo amn = SUpk lnfm>k, n>k amn
Then the steps in the proof of Theorem 3.6.1 showing that (a) through (f) are equivalent extend directly to convergence as m -+ oo and n -+ oo if lim supra,,, is replaced by lim Supm,n,oo (not by lim supm,o lim Supra,,, or the iterated lim sup in the reverse order, which may be different), and likewise for lim inf. Thus, the proof of Theorem 3.6.4 also extends.
We have the following:
3.6.7 Continuous mapping theorem Under the conditions of Theorem 3.6.4, if (T, e) is another metric space, G is a continuous function from S into T, and fo, then G(fm) G(fo) .fm Proof This follows directly from the definition of convergence in law =.
3.7 Asymptotic equicontinuity and Donsker classes
117
We also have
3.6.8 Theorem Suppose fm, m > 0, are measurable random variables taking values in a separable metric space S, so that laws G(fm) exist on the Borel or -algebra of S. Then convergence fm
fo is equivalent to convergence
of the laws G(fm) -+ G(o) in the usual sense. Proof For any bounded continuous real-valued function G on S, the functions G(fm) are measurable, so upper integrals reduce to integrals, and the result follows from the definitions.
3.7 Asymptotic equicontinuity and Donsker classes Recall from Section 3.1 the definitions of the empirical measures P, empirical process v,,, pseudometric pp, and the Gaussian process Gp. Recall that a set .F C 'C2 (p) is called pregaussian if a Gp process restricted to F exists whose
sample functions f i-+ Gp (f) ((o) are (almost) all bounded and uniformly continuous for pp on F. Such a Gp process is called coherent if, in addition, its sample functions are prelinear on .F as in Lemma 2.5.3.
3.7.1 Proposition For any probability measure P and pregaussian set F C G2(P), the symmetric convex hull sco(.F) of.F is also pregaussian, and there exists a Gp process defined on the linear span of .F and constant functions which is 0 on the constant functions and coherent on sco(.F). Proof A Gp process is isonormal on the space L2 (P) for the semi-inner product
(f, g)o,p := P(fg) - PfPg, and Gp is 0 on constant functions. So Theorem 2.5.5 on isonormal processes applies and gives the result. For a signed measure v and measurable function f such that f f d v is defined,
letv(f) := f fdv. A class ,F of functions will be said to satisfy the asymptotic equicontinuity
condition for P and a pseudometric r on F, or F E AEC(P, r) for short, if for every e > 0 there is a 3 > 0 and an no large enough such that for n > no,
f,gE.F, t(f,g)<S}>e} < E. Then.F E AEC(P) will mean.F E AEC(P, pp).
Uniform Central Limit Theorems: Donsker Classes
118
3.7.2 Theorem Let .T' C G2(X, A, P). Then the following are equivalent: (I) F is a Donsker class for P, in other words .F is P-pregaussian and vn
Gp in £'(F);
(II) (a) F is totally bounded for pp, and (b) F satisfies the asymptotic equicontinuity condition for P, F E AEC(P); (III) there is a pseudometric r on .F such that .F is totally bounded for r and
.T' E AEC(P, t).
Proof If F is a Donsker class and e > 0, then since F is pregaussian, it is totally bounded for pp by Theorem 2.3.5, so (a) holds. Take 0 < 3 < s/3 such that for any coherent Gp process,
Pr {sup{IGp(f) - Gp(g)I: pp(f, g) < 8} > s/31 < s/2. By almost surely convergent realizations (Theorem 3.5.1), for n > no large enough we can assume Pr*{Ilvn -GpII.p > s/3} < 8/2. If IIvn -GpII,p < s/3 and I Gp (f) - Gp (g) I < s/3 then I vn (f) - vn (g) l < s. So the asymptotic equicontinuity holds with the 3 and no chosen, and (b) holds, so (I) implies (II).
(II) implies (III) directly, with r = pp.
To show (III) implies (I), suppose F is r-totally bounded and F E AEC(P, r). Let UC := UC(F) denote the set of all real-valued functions on F uniformly continuous for T. Then UC is a separable subspace of t°O(F) for II Il.p since F is totally bounded and UC equals, in effect, the space of continuous functions on the compact completion (RAP, Corollary 11.2.5). For any finite subset g of F, by the finite-dimensional central limit theorem (RAP, Theorem 9.5.6) we can let n oo in the asymptotic equicontinuity
condition and so replace v (f - g) by Gp (f) - Gp (g) for f, g E 9. Given s > 0, take a 8 := 8(s) > 0 and no := no(s) < oo from the definition of asymptotic equicontinuity condition. Then 8 and no don't depend on 9 C F. So we can let CJ increase up to a countable dense set 7-l C F. Gp has sample functions almost surely uniformly continuous on H.
If fo E C and fk E 7-l are such that r(fk, fo) -+ 0 then by applying the finite-dimensional central limit theorem to the finite set 9 = { fo, fi, , fk} for each k, applying the asymptotic equicontinuity condition and letting k -+ oo we see that Gp is almost surely uniformly continuous for r on the set { f )j>o.
So Gp(fk) -- Gp(fo) almost surely. Thus for each f E F, the almost sure limit of Gp(hk) as hk - f through 7-l, which exists by uniform continuity, equals Gp (f) a.s., so the almost sure limit defines a Gp process. This Gp has uniformly continuous sample functions for r on F. So by Theorem 2.8.6, since
3.7 Asymptotic equicontinuity and Donsker classes
119
Gp is isonormal for ( , -)o, p, it follows that .F is pregaussian. So Gp has a law µ3 defined on the Borel sets of the separable Banach space UC. Given e > 0, take S > 0 from the asymptotic equicontinuity condition and a finite set 9 c Y such that for each f E F there is a g E 9 such that r (f, g) < S. Then 1189 is the set of all real-valued functions on 9. Let µ2 be the law of Gp on 1R and let tµ23 be the law on 1[89 x UC where Gp on g in R9 is just the restriction of Gp on UC. So µ23 has marginals µ2 and µ3. Let 1t1,n be the law of v on g, so µ1,n is also defined on R9. Then by the finite-dimensional central limit theorem again, the laws µ1,n converge to µ2 on 1[89. So for the Prokhorov metric p, since it metrizes convergence of laws (RAP,
Theorem 11.3.3), p(µt,,, µ2) < e for n large enough. Taken > no also, then fix n. By Strassen's theorem (RAP, Corollary 11.6.4), there is a law A12 on 1[89 x R9 with marginals 1t1,n and µ2 such that µ1,2{(x, Y): Ix - yA > E) < E. By the Vorob'ev-Berkes-Philipp theorem (1.1.10), there is a Borel measure /'t123 on 1[89 x 1[89 x UC having marginals A12 and µ23 on the appropriate spaces.
The next step is to link up v with its restriction to the finite set 9. The Vorob'ev-Berkes-Philipp theorem may not apply here since v on F may not be in a Polish space, at least not one that seems apparent. (About nonmeasurability on nonseparable spaces see the remarks at the end of Section 1.1.) Here we can use instead:
3.7.3 Lemma Let S and T be Polish spaces and (Q, A, P) a probability space. Let Q be a law on S x T with marginal q on S. Let V be a random variable on Q with values in S and law G(V) = q. Suppose there is a real random variable U on S2 independent of V with continuous distribution function FU. Then there is a random variable W : S2 i-). T such that the joint law G(V, W)
of(V,W)isQ. Proof Two metric spaces X, Y are called Borel-isomorphic if there is a one-toone Borel measurable map of X onto Y with Borel measurable inverse. Every Polish space is Borel-isomorphic to some compact subset of [0, 11, either the whole interval, a finite set, or a convergent sequence and its limit (RAP, Theorem 13.1.1). Since the lemma involves only measurability and not topological properties of the Polish spaces, we can assume S = T = [0, 11. Here we need the fact that for any real-valued random variable X with continuous distribution function F, F(X) has a uniform distribution in [0, 1 ], which can be seen as fol-
lows. For -oo < t < oo, the probability that X < t equals F(t). Now X < t implies F(X) < F(t). On the other hand, the probability that F(X) < F(t) is the supremum of probabilities that X < y for y such that F(y) < F(t), and this supremum is at most F(t). So the probability that X < t equals the probability
Uniform Central Limit Theorems: Donsker Classes
120
that F(X) < F(t). Since F is continuous, for any s with 0 < s < 1 there is a t E R with F(t) = s, so the probability that F(X) < s equals F(t) = s, and F(X) is uniformly distributed in [0, 1] as claimed. So, taking FU (U), we can assume U is uniformly distributed in [0, 1]. By way of regular conditional probabilities (RAP, Section 10.2; Bauer, 1981) we can write Q = f Qx dq(x) where for each x, Qx is a probability measure on T, so that for any measurable set A in (the square) S x T, Q(A) = f f 1 A (X, y) dQx (y) d q (x) (RAP, Theorems 10.2.1 and 10.2.2). Let Fx be the distribution function of Qx and
0 < t < 1.
Fx 1(t) := inf{u: Fx(u) > t1,
Then for any real z and 0 < t < 1, Fx 1(t) < z if and only if Fx(z) > t. Now x i-- Fx (z) is measurable for any fixed Z. It follows that x i-+ Fx 1(t) is measurable for each t, 0 < t < 1. For each x, Fx 1 is left-continuous and nondecreasing in t. It follows that (x, t) i-+ Fx 1(t) is jointly measurable. Thus f o r W (co) := F7 ) (U (w)), co H W (co) is measurable. For each x we have the image measure A o (Fx 1)-1 = Qx (RAP, Proposition 9.1.2). So for any bounded Borel function g,
J
gdQ =
f1 f1
JJ 0
g(x, y) dQx(y) dq(x)
by Theorem 10.2.1 of RAP
0
f
= J0
g(x, Fx 1(y)) dy dq (x) by the image measure theorem
0
I fo g (x, Fx 1(y)) d(q x A)(x, y) 1
1
0
by the Tonelli-Fubini
theorem
= E(g(V, FV 1(U))) = Eg(V, W), since U is independent of V and G(V) = q and by the image measure theorem again. So G(V, W) = Q, proving Lemma 3.7.3. Now let (Q, S, Q) be a probability space on which all the empirical processes v, and an independent U are defined, specifically a countable product of copies
of the probability space (X, A, P) times one copy of [0, 1] with Lebesgue measure A. Then Lemma 3.7.3 applies to Q with S = RO and T = V x
UC, where V is v, restricted to 9, and Q = µ123 on S x T. On Q we then have processes v,, and Gp defined on F, which by construction and the asymptotic equicontinuity condition are within 3e of each other uniformly on F except with a probability at most 38 (as in the proof of Donsker's theorem in Section 1.1).
3.8 Unions of Donsker classes
121
Lets j, 0 through the sequence s = 1 / k, k = 1, 2, . Let the approximation just shown hold for n > nk on a probability space (Qk, Sk, Qk). We can assume nk is nondecreasing ink. Let Akn be the vn process defined on S2k and let
Gkn be the corresponding Gp process on Stk. Let no := 1 and let (Qo, So, Qo) be a probability space on which vn processes Aon and Gp processes Go, Go
are defined, where Go is independent of the Aon processes. Let An Akn and Gn := Gk, if and only if nk < n < nk+1 for k = 0, 1 , . Then for all n, An is a vn process and Gn is a Gp process. Also, II A, - Gn CIF 0 in outer probability, so by Theorem 3.6.1, vn z' Gp on .T7, and (III) implies (I), proving Theorem 3.7.2.
3.8 Unions of Donsker classes It will be shown in this section that the union of any two Donsker classes F and
g is a Donsker class. This is not surprising: one might think it was enough, given the asymptotic equicontinuity conditions for the separate classes, for a given s > 0, to take the larger of the two no's and the smaller of the two 8's. But it is not so easy as that. For example, F and G could both be finite sets, with distinct elements of F at distance, say, more than 0.2 apart for pp, and likewise for CG, but there may be some element of F very close to an element of G. So the equicontinuity condition on the union won't just follow from the conditions on the separate families. Given a probability measure P, F C G2(P), s > 0, 8 > 0, and a positive integer no, say that AE(F, no, s, 8) holds if for all n > no,
Pr* {sup{Ivn(f - g)I: f, g E F, pp(f, g) < 8} > s} < E. Then the asymptotic equicontinuity condition, as in the previous section, holds for .P and P if and only if for every s > 0 there is a 8 > 0 and an no such that AE(F, no, s, 8) holds. The asymptotic equicontinuity condition, together with total boundedness of F for pp, is equivalent to the Donsker property of F for P (Theorem 3.7.2).
3.8.1 Theorem (K. Alexander) Let (S2, A, P) be a probability space and let .T'1 and F2 be two Donsker classes for P. Then F := .T'1 U .T72 is also a Donsker class for P.
Proof (M. Arcones) Given s > 0, take 8i > 0 and ni < oo so that AE(Fi, ni, s/3, 8i) holds, i = 1, 2. F1 and F2, being Donsker classes, are pregaussian, and Gp is an isonormal process for ( , )o,p, so F is pregaussian by Corollary
Uniform Central Limit Theorems: Donsker Classes
122
2.5.9. So there is an a > 0 such that for a suitable version of GP,
Pr{sup{JGP(f-g)j: pp(f,g)s/3} < 8/3. Let S := min(31, 82, a/3). Take finite sets hi c Ti, i = 1, 2, such that for
each i and f E Ti there is an h := Ti f E Hi with pp (f, h) < 3. Since 7-l := Ni U N2 is finite, by the finite-dimensional central limit theorem (RAP, Theorem 9.5.6), vn restricted to N converges in law to Gp restricted to H. Let
F(R, a, s/3) be the set of all y E R H such that I y(f) - y(g) I > 8/3 for some f, g E N with pp(f, g) < a. Then F(N, a, e/3) is closed and has probability less than s/3 for the law of Gp on IRx. Thus by the portmanteau theorem (RAP, Theorem 11.1.1), there is an m such that
AE(N, m, s/3, a) holds.
(3.8.2)
Let no := max(n 1, n2, m). It will be shown that AE(.TF, no, e, 3) holds. By the
asymptotic equicontinuity conditions in each Ti and since no > max(n1, n2) and S < min(81, 32), there is a set of probability less than s/3 for each i = 1, 2 such that outside these sets, I v,, (f) - vn (g) I < 8/3 for any f, g in the same
.T'i with pp(f, g) < S. For pairs f, g with pp(f, g) < 8, with f E .T'1 and g E F2, we have pP(ri f, r2$) < a since pp(f, rl f) < 8, pp(g, r2g) < S, and 33 < a. Thus by (3.8.2), I vn (rl f) - vn (r2g) I < s/3 for all such f, g, outside of another set of probability at most e/3. Thus
vn(f) - vn(g)I
Ivn(f) - vn(nif)I + Ivn(rlf) - vn(r2g)I + (vn (r2g) - vn (g) i
< e/3 + s/3 + e/3 = s
for all f E .T'1 and g E F2 except on a set of probability at most 8/3 + s/3 + s/3 = e.
3.9 Sequences of sets and functions This section will show how the asymptotic equicontinuity condition in Theorem 3.7.2 can be applied to prove that some sequences of sets and functions are Donsker classes. In Chapters 6 and 7, other sufficient conditions for the Donsker property will be given that will apply to uncountable families of sets and functions. 3.9.1 Theorem Let (X, A, P) be a probability space and {C,n },n> 1 a sequence of measurable sets. If 00
(3.9.2)
r(P(Cm)(1 - P(Cm)))' < oo forsome r < oo, m=1
3.9 Sequences of sets and functions
123
then the sequence {Cm),,,>1 is a Donsker class for P. Conversely, if the sets Cm are independent for P, then the sequence is a Donsker class only if (3.9.2) holds.
Proof Suppose (3.9.2) holds. Then the positive integers can be decomposed into two subsequences, over one of which P(Cm) 0 and over the other P(Cm) -+ 1. It's enough to prove the Donsker property separately for each subsequence by Theorem 3.8.1. For any measurable set A with complement
-vn(A) and Gp(A`) - -Gp(A). The transformation of these processes into their negatives preserves convergence in law if it holds. So we can assume P(Cm) ,(, 0 as m -- oo, and then E. P(Cm)' < oo. Also, {Cm}m>1 is totally bounded for pp. By Theorem 3.8.1 we can assume pm := P(Cm) < 1/2 for all m. For any i and m such that P(CiACm) = 0, we will have almost surely for any n that P. (Ci) = Pn (Cm ). So we can assume that P(C1 ACm) > 0 for all i 0 M. For any m such that P(Cm) = 0 we will have Pn (Cm) = 0 almost surely for any n, and then vn (Cm) = 0, so we can assume P (Cm) > 0 for all m. Let 0 < s < 1. Suppose we can find M and N such that for all n > N, Ac,
(3.9.3)
Pr {supra>MIVn(Cm)I >s} < s.
Then for J large enough, pm < pM12 form > J. Let y := min{P(Ci ACj :
1 < i < j < J), a := min(y, pM)12. Then sup{Ivn(Ci) - vn(Cj)I: P(CiACj) < a) <_ suP{Ivn(C,) - vn(CC)I: i, j > M}
< 2sup{Ivn(C1)I: j > M} < 2s with probability at least 1- e, proving the asymptotic equicontinuity condition. So it will be enough to prove (3.9.3). For that, recalling the binomial prob-
abilities defined in Section 1.3, it will suffice to find M and N such that for
n > N, 00
(3.9.4)
57, E
< 8/2
m=M and 00
B(npm - sn1 /2, n, pm) < s/2.
(3.9.5) M=M
Uniform Central Limit Theorems: Donsker Classes
124
Let qm := 1 - pm. For (3.9.5), by the Chernoff-Okamoto inequality (1.3.10), B(npm - En 1/2, n1
p.) < exp (- e2/(2pmgm)).
For some K, 1 < K < oo, pm < Km -1/'' for all m. Choose M large enough so that exp
m' e2/(2K)) < E/2,
M=M
and so (3.9.5) holds for all n. The other side, (3.9.4), is harder. Bernstein's inequality 1.3.2 gives, if 2pn1/2 > e and 0 < p < 1/2, with q := 1 - p, that
E(np + En1/2, n, p) < exp ( - e2/(2pq + en-1/2)} < exp (- E2/(6pq)). Then 00
E {E(npm +En1/2, n, pm) : 2pmn1/2 >
(3.9.6)
m=M 00
{exp-E2/(6pm)): 2pmn1/2 > e} M=M 00
exp (- e2m1/rl (6K)) < e/4
< M=M
for all n if M is large enough. It remains to treat the sum, which will be called S2, of (3.9.4) restricted to values of m with 2pmn1/2 < E.
(3.9.7)
For p:= pm, inequality (1.3.11) implies
E(np + en 1/2, n, p) < (npl (np +
En1/2))np+En1/2eenl/2
Lety:= y(n,m,e) :=n1/2pm/e,so y> 0. Let f(x) - (1+x)log(1+x Then
- (np + en1/2) log (1 + e/(pn1/2)) = En1/2 (1 - f(y)). Forx > 0, f(x) < 0. By (3.9.7), y < 1/2, so f(y) > f(1/2) > 3/2. Thus En1/2
1 < 2f(y)/3, En1/2 (1 - f(y)) < -En 1/2 f (y) /3, and 00
{exp(-(En1/2+npm) [log (1+E/(n1/2pm)}]/3): 2pmn112<E1
S2 m=1 00
JJ ((
M=1
{(pmnl/2/e)En1/2l3:
2pmn1/2 < E}
3.9 Sequences of sets and functions
since 1/y < (1 +
y)l+y, so (1 +
125
y)-(l+y) < y. Now since p,, < Km-1I' we
have S2 < S3 + S4 where 00
._
S3
i{
2Km-1/rfn- <
E}
m=1 E,vIn_13 1` m
K
e/nl(3r)
m>G
where G :_ (2K,1_n_/s)r > 2. Then for any n1 > (3r/s)2 and n > n
S3 <
(K,,,fn-/E) 8vfn-13
x-e,Jn_1(3r)dx
fG-1
< (K,//s) ,/3(s ,/
(3r)-1 - 1`-1(G
-
For K, E and r fixed, and n
oo, the logarithm of the latter expression is asymptotic to -En 1/2 (log 2)/3 -oo, so S3 -+ 0. Thus S3 < E/8 for n > n2 for some n2. Finally, define S4 by 00
E
{(Pm'In-/e)r,1rn_l3:
E<
2Km-1/rfn_}
m=1
<
(2K/i/sy2-E,`l3
-0
as n - o0, so S4 < E/8 forn > n3 for some n3. Thus forn > max(n 1, n2, n3), S2 < e/4. This and (3.9.6) give (3.9.4). So (3.9.2) implies that {Cm}m>1 is a Donsker class. For the converse, if the measurable sets {Cm }m>1 are independent for P and form a Donsker class for P, it will be shown that (3.9.2) holds. Note that for each n, P (Cm) are independent random variables for in = 1, 2, . First suppose
lim supm,,,,, pm =: p < 1 and for all n, EM' 1 pm = +oo. Then for each n, Pr{ P (Cm) = 1 for infinitely many in } = 1 by the Borel-Cantelli lemma. Now
Pn(Cm) = 1 implies vn(Cm) = n1/2(1 - pm) > n1/2(1 - p) /2 form large. From Theorem 3.7.2, { 1 C. : m > 1) must be totally bounded for pp and satisfy
the asymptotic equicontinuity condition. It follows that pm -* 0 as m - oo, and by the asymptotic equicontinuity condition, we must have En° 1 pm < 00 for some n. A symmetrical argument applies if lim infra + pm > 0. We can write {Cm )m> l as a union of two subsequences, one for which
P(C,) < 1/2 and another for which P(Cm) > 1/2, both Donsker classes. Thus by the argument just given, for some n < oo, the first subsequence sat-
isfies E. P(Cm)" < oo and the second, F_m(1 - P(Cm))" < oo. So for the original sequence, (3.9.2) holds.
126
Uniform Central Limit Theorems: Donsker Classes
Next, let's consider sequences of functions. For a probability space (A, A, P) and f E £2(A, A, P) let 0 ,2( f := f f 2 dP - (f f dP) 2 (the variance of f). Here is a sufficient condition for the Donsker property of a sequence { fm } which is easy to prove, yet turns out to be optimal of its kind:
3.9.8 Theorem If { fm}m>1 C G2(P) and Fn° 1 ci (fm) < oo, then {fm }m> 1 is a Donsker class for P.
Proof Since v, and Gp are the same on fm - Efm as on fm a.s. for all m, we can assume Efm = 0 for all m. Then fm - 0 in Go, so the sequence { fm} is totally bounded for pp. For any 0 < s < 1, n > 1 and m > 1, by Chebyshev's inequality,
Pr(Ivn(f )I > s/2) < 4 E o (f)/s2 < s j>m
j>m
form > mo for some mo < oo. We have a.s. for all n, vn (f) = 0 for all j such
that a (f) = 0 and vn (f) = v (fk) for all j and k with or p (f - fk) = 0. Let a be the infimum of o (f - f k ) over all j and k such that o (fj - fk) > 0, j < me, and o (f) > 0. Then a > 0 since in some pp neighborhood of fj there are only finitely many fk. Let 8 := min(a, 1). Then the probability that I vn (f) - vn (fk) I > s f o r some j, k with orp2 (f - fk) < 8 is less than s, implying the asymptotic equicontinuity condition and so finishing the proof by Theorem 3.7.2. The following shows that Theorem 3.9.8, although it does not imply the first half of Theorem 3.9.1, is sharp in one sense:
3.9.9 Proposition Let A := [0, 1] and P := U[0, 1] := Lebesgue measure on A. Let am > 0 satisfy Fm 1 am = +oo. Then there is a sequence { fm I C G2(A, A, P) with c,' (f.) < am for all m where { fm } is not a Donsker class. . 0. There exist cm J. 0 such that >m am cm = +oo (see problem 12). In A let Cm be independent sets with P(Cm) = am cm for
Proof We can assume am
1/2
each m (see problem 13). Let fm := cm I Cm. Then oP (fm) < f f,2 dP = am. For each n, almost surely Pn (Cm) > 1/n for infinitely many m. Then vn(Cm) > n-1/2/2 for infinitely many m and supm vn(fm) = +oo a.s., so the asymptotic equicontinuity condition fails and { fm } is not a Donsker class for P.
Problems
127
Problems 1. Let (0, A, P) be a probability space, let (S, d) be a (possibly nonseparable) metric space, and let xn, n = 0, 1, 2, , be points of S. Let fn ((o) = Xn
for all co. Show that fn = f0 if and only if xn - X. 2. In a general (possibly nonseparable) metric space show that if X0 - p E S is a constant random variable then random elements Xn = X0 if and only X0 in outer probability. if X,
3. Let (T, d) be any metric space and S C T a separable subspace. Let Xn, n > 0, be S-valued measurable random variables on some probability space such that Xn -) X0 in law as n --- oc, in S. Show that Xn = X0 as T-valued random elements. 4. Show that C := J(-oo, t] : t E R} is a Donsker class for any law P on R. Hint: Use Section 1.1.
5. Let Ak be independent sets in a probability space (S2, A, P) such that Y-k[P(Ak)(1 - P(Ak))]" = +oo for all n = 1, 2, - -. Show that {Ak}k>1 is not a Glivenko-Cantelli class, that is, SUpk I (Pn - P) (Ak) I doesn't con-
verge to 0 in probability as n - oo. Hint: See the last part of the proof of Theorem 3.9.1.
6. Let (A, A, P) be a probability space and let F C L2 (A, A, P) be a Donsker class for P. (a) Show that the convex hull of F, namely k
co(F) := j E ) , j : f E .P, j = 1, ... , k, Aj > 0, j=1 k
YAj=1, j=1
is a Donsker class. Hint: Use Theorem 2.5.5 to get that co(F) is pregaussian and Gp can be taken to be prelinear on co(F). Then use almost surely convergent realizations (3.5.1). (b) For any fixed k < oo, show that f : f E F for j = 1, , k} is also a Donsker class. Hints: Use induction and Theorem 3.8.1 on unions. (2f : f E F} is Donsker. Thus take k = 2. It is easy to show that 2F Then apply part (a). 7. Let c > 0. For Lebesgue measure A on [0, 1], the Poisson process with intensity measure cl is defined by first choosing n at random having a
Poisson distribution with parameter c, namely P(n = k) = e-`ck/k! for , then setting Yc := 8x, where Xj are i.i.d. with law). on k = 0, 1, [0, 1]. In the Banach space of bounded functions on [0, 1] with supremum
Uniform Central Limit Theorems: Donsker Classes
128
norm, prove that as c
oo (along any sequence) the random functions
t H (Yc - c? )c-1/2[0, t], 0 < t < 1, converge in law to the Brownian motion process xt, 0 < t < 1. (Recall that xt = L(llo,tl).) Hints: For c = ck -+ oo let n = nk be Poisson (ck). For F(t) - t, 0 < t < 1, write Yc(t) := Yc([0, t]) = n F, (t), so c-1/2(Yc(t)
- Ct)
_ (n/c)1/2[n1"2(Fn - F)(t)] + c-112(n - c)t.
By Donsker's theorem take Brownian bridges y(n) such that n 1/2(Fn - F)
is close to y(n) for n large. Also, c-1/2(n - c) is close in law by the central limit theorem and so in probability (Strassen's theorem) to some random N(0, 1) variable Zc. Show that one can take n((o) independent y(n('0)) of X1, X2, , then y( (w) is a Brownian bridge; apply the t w) := Vorob'ev-Berkes-Philipp theorem to take Zc independent of yt, and then yt + Zc t is Brownian. 8. Let P be a law on a separable, infinite-dimensional Hilbert space H such that f IIx X12 dP(x) < oo and with mean 0, so that f (x, h) dP(x) = 0 for all h E H. Let X1, X2, be i.i.d. in H with law P and Sn := X1 + + Xn.
(a) Show that the central limit theorem holds in H, that is, Sn/n1/2 converges in law to some normal measure on H. Hint: Prove, using variances, that the laws of Sn/n1/2 are uniformly tight. (b) Show that the class of functions x r* (x, h) for h E H with 1Ih 11 < 1 (the unit ball of the dual space of H) is a Donsker class of functions on H
for P. Hints: For part (a) let {en ] be an orthonormal basis so that for any x E H,
x = En xnen with lix 112 = En xn. Thus En Exn is finite. Show that for Exn/cn is still finite. For any K let some cn 0 slowly enough, CK be the set of x such that En (xn /cn )2 < K2 (an infinite-dimensional ellipsoid). Show that each CK is compact and that the laws of Sn/n1/2 are uniformly tight by way of the sets CK. Thus subsequences of these laws converge. Show that they all converge to the same Gaussian limit law, since by the Stone-Weierstrass theorem, functions depending on only finitely many xj are dense in the continuous functions on each CK. , X,n, Y1, 9. The two-sample empirical process. Let X1, , Yn be i.i.d. with the uniform distribution U[0, 1] on [0, 1] (Lebesgue measure restricted to [0, 1]). Let F be the empirical distribution function based , X,,,, and Gn likewise based on Y1, , Yn. Show that in the on X1, space of all bounded functions on [0, 1] with supremum norm, mn
(m+n)
1/2
(Fm
- Gn)
y
Problems
129
as m, n -* oo where t i-± yt, 0 < t < 1 is a Brownian bridge process. Hints: As in Donsker's theorem (1.1.1), form, n large, m1I2(F,, - F) is close to a Brownian bridge Y(m), and n 1/2 (Gn - F) is close to an independent Brownian bridge Z (n ). It follows that [(m n) / (m +n)] 1/2 (Fm - Gn) is
close to (n/(m +n))1/2Y(m) - (m/(m +n))1/2Z(n), which is a Brownian bridge.
10. Let fn (x) = cos(27rnx) for0 < x < I with law U[0, 1] on [0, 1]. For real c let Fc be the sequence of functions n-' fn for all n = 1, 2, . For what values of c is Fc pregaussian? A Donsker class? Hints: If c < 0 show easily that the class is not pregaussian and so not Donsker. If c > 0 then it is pregaussian, by metric entropy. If c > 1/2, it is Donsker by 3.9.8. Show that it is Donsker for 0 < c < 1 /2 by the Bernstein inequality and the asymptotic equicontinuity condition. 11. In R, for any law P on IR, show that for any fixed k < oo, k
U (aj, bj]: aj < bj for all j j=1
is a Donsker class, that is, .F := {lc : C E C} is a Donsker class. Hint: Apply Donsker's theorem (1.1.1) and take differences (and sums). For k = 2 reduce to the case of disjoint intervals and apply problem 6. Do induction to get general k.
12. Show that as stated in the proof of Proposition 3.9.9, for any am > 0 with E. an = +oo there are cm 1 0 with >m cmam = +oo. Hint: Take a sequence mk such that Flan : mk < m < mk+1} > k for each k = 1, 2, . Let cm have the same value for Mk < m < mk+1 13. Show that as also stated in the proof of Proposition 3.9.9, in [0, 1] for P = U[0, 1] there exist independent sets Cm with any given probabilities. Hint: Use binary expansions. By decomposing the set of positive integers into a countable union of countably infinite sets, show that ([0, 1], P) is isomorphic as a probability space to a countable Cartesian product of copies of itself. 14. Suppose that {Cm)m>1 are independent for P, 00
P(Cm)(1 - P(Cm)) _ +00, m=1
and cm
oo. Show that (cm I cm )m> 1 is not a Donsker class.
130
Uniform Central Limit Theorems: Donsker Classes
Notes
Notes to Section 3.1.
For any metric space (S, d), let Bb (S, d) be the o-
algebra generated by all balls B(x, r) := {y: d(x, y) < r}, x E S, r > 0. Then Bb (S, d) is always included in the Borel or -algebra 13(S, d) generated by all the open sets, with 13b(S, d)= 13(S, d) if (S, d) is separable. Suppose Y, are functions from a probability space (S2, P) into S, measurable
for Bb(S, d). Then each Y has a law it, = P o Yn 1 on Bb(S, d). Dudley (1966, 1967) defined convergence in law of Y to Yo to mean that f * Hdµ -k f Hd µo for every bounded continuous real-valued function Hon S. HoffmannJorgensen (1984) gave the newer definition adopted generally and here, where the upper integrals and integral are taken over n, not S, so that the laws µ are not necessarily defined on any particular or -algebra in S. Hoffmann-Jorgensen's monograph was published in 1991. Andersen (1985a,b), Andersen and Dobric (1987, 1988), and Dudley (1985) developed further the theory based on Hoffmann-Jorgensen's definition. Notes to Section 3.2. Blumberg (1935) defined the measurable cover function f *, see also Goffman and Zink (1960). (Thanks to Rae M. Shortt for pointing out Blumberg's paper.) Later, Eames and May (1967) also defined f *. Lemmas 3.2.2 through 3.2.6 are more or less as in Dudley and Philipp (1982, Section 2), except that Lemma 3.2.4(c) is new here. Theorem 3.2.1 and its proof are as in Vulikh (1967, pp. 78-79) for Lebesgue measure on an interval (the proof needs no change). Luxemburg and Zaanen (1983, Lemma 94.4 p. 222) also prove existence of essential suprema and infima of families of measurable (extended) real functions.
Note to Section 3.3. This section is based on parts of Dudley (1985). Notes to Section 3.4. Perfect probability spaces were apparently first defined by Gnedenko and Kolmogorov (1949, Section 3), and their theory was carried on among others by Ryll-Nardzewski (1955), Sazonov (1962), and Pachl (1979).
Perfect functions are defined and treated in Hoffmann-Jorgensen (1984, 1985) and Andersen (1985a,b); see also Dudley (1985).
Notes to Section 3.5. The existence of almost surely convergent random variables with a given converging sequence of laws was first proved by Skorokhod (1956) for complete separable metric spaces, then by Dudley (1968) for any separable metric space, with a re-exposition in RAP, Section 11.7, and by Wichura (1970) for laws on the a-algebra generated by balls in an arbitrary metric space
Notes
131
as mentioned in the notes to Section 3.1. The current version was given in Dudley (1985).
Notes to Section 3.6. Hoffmann-Jorgensen (1984), who defined convergence in law in the sense adopted in this chapter, also developed the theory of it as in this section, and partly in a more general form (with nets instead of sequences, and other classes of functions in place of the bounded Lipschitz functions). Andersen and Dobric (1987, Remark 2.13) pointed out that the portmanteau theorem (as in Topsoe, 1970, Theorem 8.1) "can be extended to the nonmeasurable case. The proof of this extension is the same as the ordinary proof." Much the same might be said of other equivalences in this section. Dudley (1990, Theorem A) gave a form of the portmanteau theorem and (Theorem B) of the metrization theorem (3.6.4).
But not all facts or proofs from the separable case extend so easily: for example, in the separable case, there is an inequality for the two metrics, 0 < 2p, in the opposite direction to Lemma 3.6.5, which follows from Strassen's theorem on nearby variables with nearby laws (RAP, Corollary 11.6.5), but Strassen's theorem seems not to extend well to the nonmeasurable case (Dudley, 1994).
Notes to Section 3.7. An early form of the asymptotic equicontinuity condition appeared in Dudley (1966, Proposition 2) and a later form in Dudley (1978).
The equivalence with a different pseudometric r in Theorem 3.7.2 is due to Gine and Zinn (1986, p. 58). Lemma 3.7.3 is essentially contained in the proof of Skorokhod (1976, Theorem 1), as Erich Berger kindly pointed out. See also Erlov (1975). Notes to Section 3.8. Alexander (1987, Corollary 2.7) stated Theorem 3.8.1 but didn't publish a proof of it, although he had written out an unpublished proof several years earlier. He says that the result is "an extension of a slightly weaker result of Dudley (1981)," where F2 is finite, but this author himself doesn't think his 1981 result was only "slightly weaker"! The proof presented was suggested by Miguel Arcones in Berkeley during the fall of 1991, but I take responsibility for any possible errors in it. Apparently van der Vaart (1996, Theorem A.3) first published a proof.
Notes to Section 3.9. Theorem 3.9.1 first appeared in Dudley (1978, Section 2), Theorem 3.9.8 and Proposition 3.9.10 in Dudley (1981), and Proposition 3.9.9 in Dudley (1984).
132
Uniform Central Limit Theorems: Donsker Classes
References Alexander, K. S. (1987). The central limit theorem for empirical processes on Vapnik-Cervonenkis classes. Ann. Probab. 15, 178-203. Andersen, N. T. (1985a). The central limit theorem for non-separable valued functions. Z. Wahrscheinlichkeitstheorie verve. Gebiete 70, 445-455. Andersen, N. T. (1985b). The calculus of non-measurable functions and sets. Various Publication Series no. 36, Matematisk Institut, Aarhus Universitet. Andersen, N. T., and Dobric, V. (1987). The central limit theorem for stochastic processes. Ann. Probab. 15, 164-177. Andersen, N. T., and Dobric, V. (1988). The central limit theorem for stochastic processes II. J. Theoret. Probab. 1, 287-303. Bauer, H. (1981). Probability Theory and Elements of Measure Theory, 2d ed. Academic Press, London. Blumberg, Henry (1935). The measurable boundaries of an arbitrary function. Acta Math. (Uppsala) 65, 263-282. Cohn, D. L. (1980). Measure Theory. Birkhauser, Boston.
Dudley, R. M. (1966). Weak convergence of probabilities on nonseparable metric spaces and empirical measures on Euclidean spaces. Illinois J. Math. 10, 109-126. Dudley, R. M. (1967). Measures on non-separable metric spaces. Illinois J. Math. 11, 449-453. Dudley, R. M. (1968). Distances of probability measures and random variables. Ann. Math. Statist. 39, 1563-1572.
Dudley, R. M. (1978). Central limit theorems for empirical measures. Ann. Probab. 6, 899-929; Correction 7 (1979) 909-911. Dudley, R. M. (1981). Donsker classes of functions. In Statistics and Related Topics (Proc. Symp. Ottawa, 1980), North-Holland, New York, 341-352. Dudley, R. M. (1984). A course on empirical processes. Ecole d'ete de probabilites de St.-Flour, 1982. Lecture Notes in Math. (Springer) 1097, 1-142. Dudley, R. M. (1985). An extended Wichura theorem, definitions of Donsker class, and weighted empirical distributions. In Probability in Banach Spaces V (Proc. Conf. Medford, 1984), Lecture Notes in Math. (Springer) 1153, 141-178. Dudley, R. M. (1990). Nonlinear functionals of empirical measures and the bootstrap. In Probability in Banach Spaces VII (Proc. Conf. Oberwolfach, 1988), Progress in Probability 21, Birkhauser, Boston, 63-82. Dudley, R. M. (1994). Metric marginal problems for set-valued or non-measurable variables. Probab. Theory Related Fields 100, 175-189.
References
133
Dudley, R. M., and Philipp, Walter (1983). Invariance principles for sums of Banach space valued random elements and empirical processes. Z. Wahrscheinlichkeitsth. verw. Gebiete 62, 509-552. Eames, W., and May, L. E. (1967). Measurable cover functions. Canad. Math. Bull. 10, 519-523. Erlov, M. P. (1975). The Choquet theorem and stochastic equations. Analysis Math. 1, 259-271. Gine, E., and Zinn, J. (1986). Lectures on the central limit theorem for empirical processes. In Probability and Banach Spaces (Proc. Conf. Zaragoza, 1985), Lecture Notes in Math. (Springer) 1221, 50-113. Gnedenko, B. V., and Kolmogorov, A. N. (1949). Limit Distributions for Sums of Independent Random Variables. Moscow. Transl. and ed. by K. L. Chung, Addison-Wesley, Reading, MA, 1954; rev. ed. 1968. Goffman, C., and Zink, R. E. (1960). Concerning the measurable boundaries of a real function. Fund. Math. 48, 105-111. Hoffmann-Jt rgensen, Jorgen (1984). Stochastic processes on Polish spaces. Published (1991): Various Publication Series no. 39, Matematisk Institut, Aarhus Universitet. Hoffmann-Jorgensen, Jorgen (1985). The law of large numbers for non-measurable and non-separable random elements. Asterisque 131, 299-356. Luxemburg, W. A. J., and Zaanen, A. C. (1983). Riesz Spaces, vol. 2. NorthHolland, Amsterdam. Pachl, Jan K. (1979). Two classes of measures. Colloq. Math. 42, 331-340. Ryll-Nardzewski, C. (1953). On quasi-compact measures. Fund. Math. 40, 125-130. Sazonov, V. V. (1962). On perfect measures (in Russian). Izv. Akad. Nauk SSSR 26, 391-414. Skorokhod, Anatolii Vladimirovich (1956). Limit theorems for stochastic processes. Theor. Probab. Appl. 1, 261-290. Skorokhod, A. V. (1976). On a representation of random variables. Theor. Probab. Appl. 21, 628-632 (English), 645-648 (Russian). Topsoe, Flemming (1970). Topology and Measure. Lecture Notes in Math. (Springer) 133. van der Vaart, Aad (1996). New Donsker classes. Ann. Probab. 24, 21282140.
Vulikh, B. Z. (1961). Introduction to the Theory of Partially Ordered Spaces (transl. by L. F. Boron, 1967). Wolters-Noordhoff, Groningen. Wichura, Michael J. (1970). On the construction of almost uniformly convergent random variables with given weakly convergent image laws. Ann. Math. Statist. 41, 284-291.
4
Vapnik-Cervonenkis Combinatorics
This chapter will treat some classes of sets satisfying a combinatorial condition. In Chapter 6, it will be shown that under a mild measurability condition to be treated in Chapter 5, these classes have the Donsker property, for all probability measures P on the sample space, and satisfy a law of large numbers (GlivenkoCantelli property) uniformly in P. Moreover, for either of these limit-theorem properties of a class of sets (without assuming any measurability), the VapnikCervonenkis property is necessary (Section 6.4). The present chapter will be self-contained, not depending on anything earlier in this book, except in some examples.
4.1 Vapnik-Cervonenkis classes Let X be any set and C a collection of subsets of X. For A C X let CA C n A := A nC := (Cf1A : C E C). Letcard(A) :_ I A I denotethecardinality (number of elements) of A and 2A :_ [B: B C A). Let AC(A) :_ ICAI. If A n C = 2A, then C is said to shatter A. If A is finite, then C shatters A if and only if AC (A) = 21 A1.
Let mc(n) := max[0C(F) : F C X , IF] = n} for n = 0, 1, IXI < n let mc(n) := mc(IXI). Then mc(n) < 2" for all n. Let V(C)
:
inf in: me (n) < 2")
if this is finite,
+00
if mc(n) = 2" for all n,
, or if
{I
sup {n: mo(n)=2"}, S(C)
:=
if C is empty.
1
Then S(C) - V (C) - 1, and S(C) is the largest cardinality of a set shattered by C, or +oo if arbitrarily large finite sets are shattered. So, V (C) is the smallest n 134
4.1 Vapnik-Cervonenkis classes
135
such that no set of cardinality n is shattered by C. If V (C) < oo, or equivalently if S(C) < oo, C will be called a Vapnik-Cervonenkis class or VC class. If X is finite, with n elements, then clearly 2X is a VC class, with S(2x) = n. Let NC where
(N
\j/
N!/(j!(N - j)!),
j = 0, 1, ... N,
0,
j>N.
Then NC
4.1.1 Proposition NC
- -
, and
N = 1,2,.... Proof /For each j = 1, 2, - -, N, we have by the classical Pascal's triangle, I
Nl
(Ni 11+IN-1
I,
(N 0
I = INO 11 = 1.
Summing /over j, the conclusion follows. If N < k, we\ get 2N/ 2N-1
= 2N-1 +
For a non-VC class C we have mc(n) = 2' for all n. For a VC class, the next fact, which is fundamental in the Vapnik-Cervonenkis theory, will imply that mc(n) grows only as a polynomial rather than exponentially in n.
4.1.2 Theorem (Sauer's lemma) If mc(n) > nC 1, then mc(k) =2k. Hence if S(C) < oo, then mc(n) < nC<s(c) for all n.
Proof The proof is by induction on k and n. For k = 1, nC n, then nC< mc(n), so the assumption implies k < n. Now assume that the theorem holds whenever k < K and n > k, for all C.
Fix k := K + 1. For n < k, as noted, it holds vacuously. For n = k, the hypothesis mc(n) > C nC
Vapnik-Cervonenkis Combinatorics
136
AC(HN) > NC
AC(HN) < NC
(4.1.3)
Let Cn := Hn n C :_ {A fl Hn : A E C}. Call a set E C HN full if both E and E U {xn } belong to Cn . Let f be the number of full sets. Then the map A i-+ A fl HN takes C n H onto C n HN and is two-to-one onto full sets and one-to-one onto non-full sets in C n HN. Thus
OC(H,,) = Ac(HN) + .f
(4.1.4)
Let F be the collection of all full sets. Suppose f = 0-F(HN) > NC
In the remaining case, f < NC
AC(H,,) < NC k + 2, nC
Proof The inequality clearly holds for k = 0, so assume k > 1. By the binomial theorem, nk-1 (k + n) < (n + 1)k, so (4.1.6)
nk-1/(k
- 1)! +nk/k! < (n + 1)k/k!.
The proof will be done by induction on n and k. For k = 1 the inequality is n + 1 < 1.5n, which holds for all n > 2. For n = k + 2, the desired inequality is 2n - n - 1 < 1.5nii-2/(n - 2)! = 1.5(n -
1)nn-1/n!.
This can be checked directly for n = 3, 4, 5, and 6. Stirling's formula with an error bound (Theorem 1.3.13) gives n! < (n/e)n (2nn)1/2e1/(12n)
4.1 Vapnik-Cervonenkis classes
137
So it will be enough to prove
2n (n/e)n (2irn)1/2e11(12n) < 1.5(n -
1)nn-1
n>7,
which follows from (e/2)" > 2n1/2, n > 7; for f(x) := (e/2)x and g(x) :_ 2x1/2 it is straightforward to check that f (7) > g(7), f'(7) > g'(7), f" > 0, and g" < 0, so f (x) > g(x) for all x > 7. Now suppose the proposition has been proved for n = k + i, i = 2, , j, and for n = k + J, J := j + 1, for k = 1, , K, as we have done for j = 2 and for K = 1. We need to prove (4.1.5) for n = k + J and k = K + 1. We
have k+j = K + J and nC
< 1.5(k + j)k/k! + 1.5(K + J)K/K!
< 1.5nk/k!
by Proposition 4.1.1
by induction hypotheses by (4.1.6),
completing the proof.
Combining Theorem 4.1.2 and Proposition 4.1.5 gives mc(n) < 1.5nk/k! for n > k + 2 where k := S(C). To see that Theorem 4.1.2 is sharp, let X be an infinite set and C the collection of all subsets of X with cardinality k. Then S(C) = k and the inequality in the second sentence of the theorem becomes an equality for all n.
Let dens(C) be the infimum of all r > 0 such that for some K < oo, m c (n) < Kn' for all n > 1. Then we have
4.1.7 Corollary For any set X and C C 2X, dens(C) < S(C). Conversely if dens(C) < oc then S(C) < oo. Proof By Theorem 4.1.2 and Proposition 4.1.5, there is a K such that m c (n) <
Kns(c) for all n > S(C) + 2. The same holds for all n > 1, possibly with a larger K, so dens(C) < S(C). Conversely if dens(C) < oo then since mc(n) < Kn' < 2n for n large we have S(C) < oo. Note that S(C) can be determined by one large shattered set while dens(C) has to do with the behavior of C on arbitrarily large finite sets. For example if X is a set with card(X) = n and C = 2X then S(C) = n while dens(C) = 0.
For any set X, it is immediate that if C C D C 2X then S(C) < S(D) and dens(C) < dens(D). The following is straightforward since for any set X, the
138
Vapnik-Cervonenkis Combinatorics
map A r+ X \ A is one-to-one from 2X onto itself, and for any A, B, C C
X, Af1B#CflBifandonlyif(X\A)f1B#(X\C)f1B: 4.1.8 Proposition If X is any set, C C 2X and D := (X \ A : A E C) then for all B C X, AC(B) = AD(B), so m'(n) = mD(n) for all n, S(D) = S(C) and dens(D) = dens(C).
4.2 Generating Vapnik-Cervonenkis classes Let's begin with some examples of non-VC classes for which some limit theorems for empirical measures will fail. First, let X = [0, 1] and let C be the class of all finite subsets of X. Let P be the uniform (Lebesgue) law on [0, 1]. Clearly, S(C) = +oo, and C is not a VC
class. Also, for any possible value of Pn, we will have P,(A) = 1 for some A = {X1, , Xn} E C while P(A) = 0. Thus SUPAEC(Pn - P)(A) = 1 for all n, so C is not a Glivenko-Cantelli class for P; in other words, IIPn - PIIC
SUPAEC I(Pn - P)(A)I
doesn't approach 0 as n oo in outer probability or in any sense, since it is identically 1. It follows that C is also not a Donsker class for P. Note that all functions 1 A for A E C equal 0 almost surely for P. Thus, the whole class .F := {lA : A E C) reduces to the one point 0 in the space L2(P) of equivalence classes for equality almost everywhere of functions in G2(P), that is, measurable, square-integrable functions. Thus for purposes of empirical processes, functions equal a.s. P are not the same, and we need to deal with classes F C G2(P) of actual real-valued functions, not equivalence classes. Then the integral f fd(P, - P) will be well-defined for any f E L2(P). This
integral is linear in f and thus prelinear for f E T for any set J7 C G2(P). For the empirical process v, = n 1/2(P - P) we will not be taking versions or modifications as was done for Gaussian processes (Appendix J). Next, let C2 be the collection of all closed, convex subsets of R2. Let S1 be the unit circle {(x, y) : x2 + y2 = 11. For any finite subset F of S1, the convex polygon with vertices in F (a singleton if IFl = 1, or a line segment if
I F1 = 2) is in C2, and its intersection with S' is F. Thus S(C) = +oo and C is not a VC class. Let P be the uniform law dP(O) = dO/(2n) on S1. Then the Glivenko-Cantelli and Donsker properties fail for P just as in the previous example. Classes with S(C) finite, in other words Vapnik-Cervonenkis classes, can be
formed in various ways. Here is one. Let G be a collection of real-valued
4.2 Generating Vapnik-Cervonenkis classes
139
functions on a set X. Let
pos(g)
Ix: g(x) > 01, nn(g) := IX: g(x) > 0), gEG,
pos(G)
{pos(g): g E G}, nn(G) := {nn(g): g E G),
U(G)
pos(G) U nn(G).
4.2.1 Theorem Let H be an m-dimensional real vector space of functions on
a set X, let f be any real function on X, and let Hi := (f + h : h E H). Then S(pos(Hl)) = S(nn(H1)) = m. If H contains the constants then also S(U(H1)) = M.
Proof First it will be shown that S(pos(H1)) = m. Clearly card(X) > m. If card(X) = in, then H = Hl is the set ]RX of all real-valued functions on X, so the result holds.
Otherwise, let A C X with card(A) = m + 1. Let G be the vector space {a f + h : a E IR, h E H). Let rA : G ra IRA be the restriction of functions in G to A. If rA is not onto, take 0 v E IRA where v is orthogonal to rA(G) for the usual inner product -)A Let A+ := {x E A : v(x) > 0}. We can assume A+ is nonempty, replacing v by -v if necessary. If A+ = A fl pos(g) for some g E G, then (rA(g), V)A > 0, a contradiction. So pos(G) doesn't shatter A.
Suppose instead that rA(G) = RA. Then rA is 1-1 on G, f 0 H, and rA(H1) is a hyperplane in RA not containing 0, so that for some v E RA,
(j, v)A = -1 for all j E rA(H1). Again let A+ :_ {x E A : v(x) > 0). If A+ = A fl pos(f + h) for some h E H, then (rA(f + h), v)A > 0, a contradiction (here A+ may be empty). Thus pos(H1) never shatters A, so S(pos(Hi)) < m. For each x E X, a linear form 8x is defined on H by 3, (h) := h (x), h E H.
Let H' be the vector space of all real linear forms on H. Then H' is m-
dimensional. Let HH{aJxj: be the linear span in H' of the set of all 8,, x E X, (4.2.2)
HH
xj E X, aj E ]I8, r
1, 2,
}.
j-1 The map h H (r i-4 *(h)): h E H, r E HX, is 1-1 and linear from H JJJ
into (HX)', so HH is m-dimensional. Take B = {xl, , xm) C X such that the 8,1 are linearly independent in H. So rB (H) = RB, rB (H1) = JRB, and pos(rB(H1)) = 2B, so S(pos(Hi)) = m. Then S(nn(H1)) = m by taking complements (Proposition 4.1.8). If H contains the constant functions, then the sets nn(f ), f E H1, are the same as
140
Vapnik-Cervonenkis Combinatorics
the sets If > t), f E H1, t E R, and the sets pos(f), f E H1, are the same as the sets If > t), f E H1, t E R. Now for any finite subset A of X, f E Hl and t E R, since f takes only finitely many values on A, there exist s and u
such that AflIf >t}=AflIf >s}and AflIf >t}=AflIf >u}. So in this case S(U(H1)) = m. Examples. (I) Let H := Pd,k be the space of all polynomials of degree at most k on Rd. Then for each d and k, H is a finite-dimensional vector space of functions, so pos(H) is a Vapnik-Cervonenkis class. For k = 2, it follows specifically that the set of all ellipsoids in 1W' is included in a VapnikCervonenkis class and thus is one.
(II) Let X = R. Let H be the 1-dimensional space of linear functions f(x) = cx, x E 1k, c E R. Then S(pos(H)) = S(nn(H)) = 1 by Theorem 4.2.1, but U(H) shatters {0, 1). Since sets in U(H) are convex (half-lines), it follows that S(U(H)) = 2. So the condition that H contains the constants can't just be dropped from Theorem 4.2.1 for U(H). Let X be a real vector space of dimension m. Let H be the space of all real affine functions on X, in other words functions of the form h + c where h is real linear and c is any real constant. Then H has dimension m + 1 and pos(H) is the set of all open half-spaces of X. Letting f = 0 in Theorem 4.2.1 for this H gives a special case known as Radon's theorem. On the other hand, Theorem 4.2.1 for f = 0 with general X and H follows from Radon's theorem via the following stability fact.
4.2.3 Theorem If X and Y are sets, F is a function from X into Y, C C 2Y, S(F-t (C)) < S(C). If F is onto Y and F-1(C) := {F-1(A) : A E C}, then then S(F-1(C)) = S(C).
x j for i # j. Then , xm } where x; Proof Let F-1(C) shatter {xl, F(xi) 0 F(xj) for i ¢ j and C shatters {F(xl), , F(xm)}. So S(F-1(C)) < S(C). If F is onto Y and H C Y with card(H) = m, choose G C X such that F takes G 1-1 onto H. Then if C shatters H, F-1 (C) shatters G, so S(F-1(C)) = S(C). Now let X be any set and G a finite-dimensional real vector space of real functions on X. Then there is a natural map F : x H S from X into the space of linear functions on G. Then by Theorem 4.2.3 one could deduce Theorem 4.2.1 from its special case where X is an m- or (m + 1)-dimensional real vector space and f and all functions in H are affine, so that sets in pos(H1) are open half-spaces.
4.2 Generating Vapnik-Cervonenkis classes
141
Next it will be seen how a bounded number of Boolean operations preserves the Vapnik-Cervonenkis property.
, let C(k) be the 4.2.4 Theorem Let X be a set, C C 2X, and f o r k = 1 , 2, union of all (Boolean) algebras generated by k or fewer elements of C. Then dens(C(k)) < k dens(C), so if S(C) < oo then S(C(k)) < oo.
Proof Let dens(C) = r, so that for any s > 0 there is some M < oo such that mc(n) < Mnr+E for all n. For any A C X we have A n C(k) = (A n C)(k). An algebra A with k generators A1, , Ak has at most 2k atoms, which are those nonempty sets that are intersections of some of the A; and the complements of the rest. Sets in A are unions of atoms, so JAI < 22k. Thus IA n C(k) I < 22k IA n CIk < 22kMkIAlk(r+e). Letting E J, 0 gives dens(C(k)) < k dens(C). If S(C) < 00 then by Corollary 4.1.7, S(C(k)) < 00. The constant 22k is very large if k is at all large. Let C(lk) be the class of all intersections of at most k sets in C. Then C(nk) C C(k). For C(nk) the constant 22k is not needed. Theorems 4.2.1 and 4.2.4 can be combined to generate Vapnik-Cervonenkis classes. For example, half-spaces in Rd form a VC class. Intersections of at most k half-spaces give convex polytopes with at most k faces, so these form a VC class.
Remarks. Let X be an infinite set, r = 1 , 2,
, and let Cr be the collection of all subsets of X with at most r elements. Then clearly dens(Cr) = S(Cr) = r. It's easy to check that D := C(rk) consists of all sets B such that either B or X \ B
has at most kr elements. Thus mD(n) < 2(nC 2kr + 1. So dens(D) = kr since n C<j is a polynomial inn of degree j. Thus the inequality dens(C(k)) < k dens(C) is sharp. But it does not always hold for S(.) in place of dens(-): if C is the collection of open half-spaces in Rd, d > 1, then S(C) = d + 1 by Radon's theorem, while, taking for example the d half-spaces (xj > 0) for j = 1, , d, we see that C(d) shatters a set of 2d points, one in each coordinate orthant, so S(C(d)) > 2d > d(d + 1) for
d>5. Classes with V (C) = 0 or I are easily characterized:
4.2.5 Proposition A class C of subsets of a set X has V (C) = 0, or equivalently S(C) = -1, if and only if C is empty. Also, V (C) = 1, or equivalently
142
Vapnik-Cervonenkis Combinatorics
S(C) = 0, if and only if C contains exactly one set. Thus S(C) > 1 if and only if C contains at least two sets. Proof Clearly C shatters the empty set if and only if C contains at least one set. If C contains at least two sets, then for some A, B E C and X E X, X E A \ B.
Then C shatters {x), so S(C) > 1. Conversely if S(C) > 1, then clearly C contains at least two sets.
Here are two sufficient conditions for S(C) = 1:
4.2.6 Theorem If C is a collection of at least two subsets of a set X, then S(C) = 1 if either (a) C is linearly ordered by inclusion, or (b) any two sets in C are disjoint.
Proof In any case, S(C) > 1. If C is linearly ordered by inclusion, suppose it shatters {x, y). Let A, B E C, A fl {x, y} = {x}, B fl {x, y} = {y}. But A C B or B C A, giving a contradiction. If the sets in C are disjoint, then we can argue as in part (a), taking C E C with {x, y} C C, but C cannot be disjoint from A or B, a contradiction. Section 4.4 will go more into detail about classes of index 1.
*4.3 Maximal classes
Let C C A be classes of subsets of a set X. Then C will be called (A, n)maximal if S(C) = n while if C C D strictly and D C A, then S(D) > n. If A is the class 2X of all subsets of X, then C will be called n-maximal. If C is n-maximal, then clearly C is (A, n)-maximal for any A such that C C A C 2X. In view of Proposition 4.2.4, classes C with S(C) = i, i = -1 or 0, are empty or contain just one set, respectively, and so are always i -maximal. Thus n-maximality becomes interesting only for n > 1.
Examples. 1. For any set X, let C consist of 0 (the empty set) and all singletons (x} for x E X. Then C is clearly 1-maximal. 2. Let X = R. Let £ 7-1 consist of 0, R, and all left half-lines, closed (-oo, x] or open (-oo, x), for x E R. In other words, £7-l is the collection of all subsets A C JR such that whenever x < y e A then also x E A. Then clearly S(G7-l) = 1 since for x < y and A E £7-l, A fl {x, y} # (y}. On the
*4.3 Maximal classes
143
other hand if any subset of JR not in CH is adjoined, then some 2-element set is shattered, so £7-l is 1-maximal.
3. Let X = R and let Co consist of all subintervals of R, namely 0, R, any closed or open, left or right half-line, and any bounded interval, open or closed at either end. In other words, Co is the class of all convex subsets of R. Then S(Co) = 2, in fact Co shatters every 2-element subset of IR, while if x < y < z and A E Co, then A fl {x, y, z} {x, z}. On the other hand if any set not in Co is adjoined to it, its index becomes 3, so Co is 2-maximal. Here is an existence theorem for maximal classes:
4.3.1 Theorem Let X be a set and D C 2X. Suppose that C C D and S(C) = n. Then there exists a (D, n)-maximal class B with C C B.
Proof Zorn's lemma (RAP, Section 1.5) will be applied. Let (,t3a)aEI be such that C C Ba C D for all a in the index set I and such that the Ba are linearly ordered (i.e., form a chain) by inclusion, with S(Ba) = n for all a. Let A := Ua Ba. To show that S(A) = n, suppose A shatters a set F with I Fl = n + 1. Each of the 2"+1 subsets of F is induced by a set in some Ba. Since there are only finitely many of these sets and the Ba are linearly ordered by inclusion, there is some a such that Ba shatters F, a contradiction. So the chain {8a }aEI has an upper bound. So by Zorn's lemma the collection of all B with C C B C D and S(B) = n has a maximal element (for inclusion). The following fact is straightforward:
4.3.2 Proposition For any set X, Y C X, C C 2X, and Cy := C n Y, we have S(Cy) < S(C). Recall that Z2 := to, 1} with addition mod 2, in other words the usual addition except that I + 1 = 0. For any set X, the group Z2 of all functions from X into Z2, with the natural addition (f + g)(x) := f(x) + g(x) in Z2, provides a group structure for the collection 2X of all subsets of X. Addition of indicator functions mod 2 corresponds to the symmetric difference AAB :_ (A \ B) U (B \ A), so that I A + 1 B = I AAB mod 2. For any fixed set A C X, the translation 1 B H I A + 1 B takes Z 2 one-to-one and onto itself. If the functions are restricted to a subset Y C X, translation still takes Z2 one-to-one and onto itself. For any A C Xand C C 2X, let AAAC {AAC: C E C). Then for any finite F C X, C shatters F if and only if AAAC does. It follows that:
144
Vapnik-Cervonenkis Combinatorics
4.3.3 Proposition For any fixed set A C X and class C C 2X, S(C) S(AAAC). Also, C is n-maximal if and only if AAAC is. Next, we have:
4.3.4 Proposition If C is an n-maximal class of subsets of a set X, and n > 1, then UAc
Proof Suppose there is an x such that x A for all A E C. Take any B E C and let D := C U (B U {x}). Then by n-maximality, D must shatter some set F of cardinality n + 1 > 2, and evidently x E F. Thus D contains at least 2" > 2 sets which contain x, but it only contains one, a contradiction. This proves the first conclusion. The second follows on taking complements, setting A = X in Proposition 4.3.3.
On ZX = 2X there is a product topology coming from the discrete topology on Z2. The product topology is compact by Tychonoff's theorem (RAP, Theorem 2.2.8). 4.3.5 Proposition For any set X, any n-maximal class C C 2X is closed and therefore compact in 2X.
Proof Suppose Ca -* C is a convergent net in 2X with Ca E C for all a. Then for any finite set F C X, there is some a with Ca fl F = C fl F, so S(C U {C}) = S(C) and C E C. So C is closed (RAP, Theorem 2.1.3). A class C of subsets of a set X will be called complemented if X \ A E C for every A E C.
4.3.6 Theorem If S(C) = n, C C A strictly, and C is complemented, then C is not (A, n)-maximal.
Proof For any finite set F C X and G C F, G E C n F if and only if F \ G E C n F. Thus if IFl = n + 1, then IC n Fl < 21+1 - 2. So, for any
AEA\C,S(CU{A})=n. If F is a k-dimensional real vector space of real-valued functions on a set X containing the constants, and C is the collection U(F) of all sets {x : f (x) > 0}
or {x : f (x) > 0} for all f E .F, then S(C) = k by Theorem 4.2.1. Since C is complemented, if U(.F) 2X, then C is not k-maximal.
*4.4 Classes of index 1
145
Let X be any set and Ck the collection of all subsets of X with at most k elements. Then clearly S(C) = k. Also, C is k-maximal since if A 0 C, A C X, then I A I > k and if B is any subset of A with I B I = k + 1 then B is shattered by C U {A). For C = Ck we have me (n) C
*4.4 Classes of index 1 In this section the structure of classes C with S(C) = 1 will be treated. Recall that for classes of two or more sets, disjoint classes and classes linearly ordered by inclusion have S(C) = 1 (Proposition 4.2.6). A common extension of these two kinds of classes is given by treelike partial orderings, defined as follows. A binary relation < on a set X will be called a quasi-order if it is transitive:
x < y and y < z imply x < z; and reflexive: x < x for all x e X. The quasi-order is called a partial order if also x < y and y < x imply x = y. For any set S, inclusion ( C ) is a partial order on 2S or any subset of 2S. Let < be a quasi-order on a set X. Then two elements x and y of X are called comparable if at least one of x < y and y < x holds, or incomparable if neither holds. A quasi-order < on X will be called linear if any two elements of X are comparable. A quasi-order < will be called treelike if for any y E X and L Y := {x : x < y), the restriction of < to L Y is linear. 4.4.1 Theorem Let C C 2X contain at least two sets and satisfy, for any x in X, (4.4.2)
y
A fl {x, y) = 0 for some A E C.
Then the following are equivalent:
(a) S(C) = 1; (b) for every Y C X, the partial ordering of Cy := Y n C by inclusion is treelike; (c) for every Y C X with I Y I = 2, the partial ordering of Cy := Y n C by inclusion is treelike.
Note. Clearly, (4.4.2) holds if 0 E C.
146
Vapnik-Cervonenkis Combinatorics
Proof In proving (a) implies (b), since S(Cy) < S(C) and classes with S(C) < 1 contain at most one set and trivially have a treelike ordering by inclusion, we can assume Y = X. If C doesn't have a treelike ordering by inclusion, there is
a set D E C and B C D, C C D such that B and C are not comparable, so there exist some x E B \ C and Y E C \ B. Take A from assumption (4.4.2). But then the sets A, B, C, and D shatter (x, y) and S(C) > 2, a contradiction. So (b) holds. Now (b) implies (c) directly. If (c) holds and I Y I = 2, since 2Y doesn't have a treelike ordering by inclusion, C must not shatter Y, so (a) follows.
4.4.3 Proposition Let X be a set and A C 2X where 0 E A and for any B and C in A, B fl C E A. If C is (A, 1) -maximal and satisfies (4.4.2) for any
x# yin X, then 0ECandBflCECforanyBandCinC. Proof If x
y and (4.4.2) holds for A, then A fl {x, y} = 0 = 0 fl {x, y}, so adjoining 0 to C doesn't induce any additional subsets of sets with two elements, and S(C U 10)) = 1, so by maximality 0 E C. Suppose B, C E C and S(C U {B fl C}) > 1. Then for some x # yin X, B fl C fl (x, y) # D fl {x, y} for all D E C. Then by (4.4.2), we can assume x E B fl C. If {x, y} C B fl C C B then taking D = B would give a contradiction, so y 0 B fl C. Now B fl c fl {x, y} = {x} # D fl {x, y) for D = B or C implies y E B and y E C, again a contradiction. So B fl C E C.
4.4.4 Proposition Let X be a set and C a finite class of subsets of X with S(C) = 1 such that for any x y in X, (4.4.2) holds. Let D := D(C) consist of 0 and all intersections of nonempty subclasses of C. Then S(D) = 1. For each nonempty set D E D there is a C := C(D) E D such that C C D strictly (C 0 D) and if B is any set in D with B C D strictly, then B C C. Proof By Theorem 4.3.1, let C C E where £ is 1-maximal. Then by Proposition 4.4.3 for A = 2X and induction, C C D C E, so S(D) = 1. Clearly, D is finite. For each nonempty D E D, by Theorem 4.4.1, {B E D: B C D} is linearly ordered by inclusion and contains 0, so it has a largest element C(D) other than D itself.
4.4.5 Proposition
Under the hypotheses of Proposition 4.4.4, the sets
D \ C(D) for distinct nonempty D E D are all disjoint and are nonempty.
Proof Let A
B in D. If B C A, then B C C(A), so B and hence
B \ C(B) are disjoint from A \ C(A). Otherwise, A fl B C B strictly, and then
*4.4 Classes of index 1
Af1B C C(B), so again A \C(A) is disjoint from B\C(B). That A\C(A) 0 follows from the definitions.
147
0
for A
A graph is a nonempty set S together with a set E of unordered pairs {x, y} for some x y in S. Then S will be called the set of nodes and E the set of edges of the graph. The graph (S, E) is called a tree if (a) it is connected, in other words for any x and y in S there is a finite n and x; E S, i = 0, 1, , n, such that xo = x, x, = y, and {xk_1, xk} E E for
k= (b) the graph is acyclic, which means that there is no cycle, where a cycle
is a set of distinct xl,
, x E S such that n > 3, and letting xo := x,,,
{xk_1,xk} E Efork= 1, ,n. 4.4.6 Theorem (a) For in nodes, for any positive integer m, there exist connected graphs with in - 1 edges. (b) A connected graph with in nodes cannot have fewer than in - 1 edges. (c) A connected graph with m nodes has exactly m - 1 edges if and only if it is a tree.
Proof (a) is clear. (b) will be proved by induction. It is clearly true for in = 1, 2. Suppose (S, E) is a connected graph with BSI = in, I EI < in - 2, and m > 3. The edges in E contain at most 2m - 4 nodes, counted with multiplicity, so at least four nodes appear in only one edge each, or some node is in no edge, a contradiction. Select a node in only one edge and delete it and the edge that contains it. The remaining graph must be connected, but is not by induction assumption, a contradiction, so (b) holds.
For (c), let (S, E) be a connected graph with ISO = in and IEJ = m - 1. If the graph contains a cycle, we can delete any one edge in the cycle and the graph remains connected, contradicting (b). So (S, E) is a tree. Conversely, let (S, E) be a tree with I SI = m. It will be proved by induction that I E I < in - 1.
This is clearly true for m = 1, 2. Suppose I E I > in > 3. Take a maximal , xk} C S such that the xi are distinct and {xj_1, Xj) E E for set C := {xl, j = 2, , k. Then there is no y 0 X2 with {x1, y) E E: y cannot be any xj, j > 3, or there would be a cycle, and there is no such y C since C and k are maximal. So we can delete the node xl and the edge {xl, x21 from the graph, leaving a graph which is still a tree with in - 1 nodes and at least in - 1 edges, contradicting the induction hypothesis and so proving (c). Let the class D in Propositions 4.4.4 and 4.4.5 form the nodes of a graph G
whose edges are the pairs {C(D), D} for D E D, D # 0.
148
Vapnik-Cervonenkis Combinatorics
4.4.7 Proposition The graph G is a tree. Proof If D has m elements, then there are exactly m - I pairs {C(D), D}, for D E D, D # 0. Starting with any D E D, we have a decreasing sequence of sets D D C(D) D C(C(D)) D , which must end with the empty set, so all sets in D are connected in G via the empty set, and G is connected. Then by Theorem 4.4.6 it is a tree. 4.4.8 Proposition Let X be a finite set. Let C be 1-maximal in X and suppose (4.4.2) holds for all x y in X. Then C = D(C) as defined in Proposition 4.4.4. The sets D\C(D)for nonempty D E C are all the singletons {x}, x E X.
IfIXI=mthen ICI =m+1. Proof C = D(C) by Proposition 4.4.4. Suppose that for some D E C, D\C(D)
has two or more elements. Then for some B, C := C(D) C B C D where both inclusions are strict. It will be shown that S(C U {B}) = 1. If not, then for some x 0 y, C U {B} shatters {x, y}, so B fl {x, y} 54 F fl {x, y} for all F E C. Letting F = C shows that B fl {x, y} 0 0. Likewise, letting F = D shows that B fl {x, y} {x, y}. So we can assume B fl {x, y} = {x}. Taking F = D shows
that y e D. If G n {x, y} = (y) for some G E C, then y E G fl D E C, and G n D C D strictly, so G n D C C and y E C C B, giving a contradiction. So C U { BI does not shatter {x, y}, and S(C U { B)) = 1, contradicting 1-maximality of C.
So, each set D \ C(D) for 0 0 D E C is a singleton. Each singleton {x} equals D \ C(D) for at most one D E C by Proposition 4.4.5. By Proposition
4.3.4, X = UDEC D. For any x E X take D1 E C with x e D. Let . Forsomem, D,,, = 0, and {x} = Dj\C(Dj) D,+1 := C(D,) forn = 1, 2, for some j < m. So all singletons are of the form D \ C(D), D E C. This
gives a 1-1 correspondence between singletons and nonempty sets in D, so there are exactly m such sets and I D I = m + 1. Suppose throughout this paragraph that C is a class of two or more sets such that (4.4.2) holds with 0 replaced by {x, y}. Then the class of complements, N := {X \ C: C E C), satisfies the original hypotheses of Proposition 4.4.1. If C is 1-maximal, so is N by Proposition 4.3.3. So Theorem 4.4.1 and the facts numbered 4.4.3 through 4.4.7 apply to N, and so does Proposition 4.4.8 if X is finite. Then, C itself has a "cotreelike" ordering, where for each C E C, {D E C : C C D} is linearly ordered by inclusion. Propositions 4.4.3 and 4.4.4 apply to C if 0 is replaced by X and intersections by unions; in Proposition 4.4.4, we will have an immediate successor D(C) D C instead of a
*4.4 Classes of index 1
149
predecessor; and sets D(C) \ C instead of D \ C(D) in Propositions 4.4.5 and 4.4.8. The resulting tree (Proposition 4.4.7) then branches out as sets become smaller rather than larger. Next will be several facts in the general case, that is, without the hypothesis (4.4.2).
4.4.9 Theorem Let X be any set and C any collection of subsets with S(C) = 1. Then for any C E C, the collection Cx\c := (B \ C : B E C) satisfies (4.4.2) for any x y as a collection of subsets of X \ C. Likewise, Cc \ := (C \ B : B E C) satisfies (4.4.2) for any x y as a collection of subsets of C, S(Cc V) < 1 and S(CX\C) < 1.
Proof Letting B = C shows that both classes Cc \ and CX\c contain 0, so (4.4.2) holds for them. Both have index S < 1 by Propositions 4.3.2 and 4.3.3.
So, for an arbitrary class C with S(C) = 1, we have by Theorem 4.4.1 a treelike partial ordering by inclusion in one part X \ C of X and a cotreelike ordering in the complementary part C, for any C E C. If also X \ C happens to be in C, both orderings are linear. To see how the two orderings fit together in general, Proposition 4.3.3 gives:
4.4.10 Corollary Let C be any class of sets with S(C) = 1 and A E C. Let D := AAAC. Then S(D) = 1 and 0 E D. If C is 1-maximal, so is D. Then Theorem 4.4.1, Proposition 4.4.3, and if C is finite, Propositions 4.4.4, 4.4.5, 4.4.7, and if X is finite, Proposition 4.4.8, apply to D. The last sentence in Proposition 4.4.8 has a converse and extension:
4.4.11 Proposition Let X be finite with m elements and C C 2X with S(C) _ 1. Then C is 1-maximal if and only if ICI = m + I.
Proof For any fixed C E C, we can replace C by CAiC without loss of generality by Theorem 4.3.3. So we can assume 0 E C, and then (4.4.2) holds for all x, y. Then Proposition 4.4.8 implies "only if."
Conversely, let S(C) = 1 and ICI = m + 1. Let C C D strictly. Then IDS > m + 1, so by Sauer's lemma (Theorem 4.1.2), S(D) > 2. So C is 1-maximal.
Now, m + 1 = mC<1, which is the maximum value of mc(m) for S(C) = 1 by Sauer's lemma (Theorem 4.1.2). The example at the end of Section 4.3
150
Vapnik-Cervonenkis Combinatorics
shows that Proposition 4.4.11 in the form Cl I_ mC 1. Next it will be shown that 1-maximality can be relativized to subsets. For
a set X, a subset Y C X, and a class C c 2X, recall that Cy := C n Y {A n Y: A E C}.
4.4.12 Theorem If C is 1-maximal and 0
Y C X, then Cy is a 1-maximal
class of subsets of Y.
Proof Let A E C. Without changing 1-maximality, C can be replaced by AAAC (Proposition 4.3.3). So we can assume 0 E C. We can also assume that
IXI>2. Case 1. Suppose X is finite, I X I = m < oo. Then by Proposition 4.4.8 or
4.4.11, ICI = m + 1. Letx E Xand Y := X\{x}. Let B := {y E Y: {x, y} E C{x, y} and (y) E C{x, y} }. A set C C Y will be called full if C E C and
CU{x}E C. 4.4.13 Lemma If A C Y and A is full then A = B. Proof Suppose A is full. First suppose y (= A \ B. Then A U {x} E C implies {x, y} E C{x,y} and A E C implies {y} E C{x,y}, contradicting y
B.
Next, suppose y E B \ A. Then {x} = (A U {x}) n {x, y}, 0 = 0 n {x, y}, and y (= B imply that C shatters {x, y}, a contradiction. So A = B, proving the lemma.
El
Continuing the proof of Theorem 4.4.12, any two distinct sets in C have different intersections with Y, except possibly for B and B U {x}. Now Cl S=
m + 1 by Proposition 4.4.11, so ICyl > m and ICY = m since S(Cy) < 1 and I Y I = m - 1. So if m > 2, Cy is a 1-maximal class of subsets of Y by Proposition 4.4.11 again. Then by induction downward, Cy is 1-maximal in Y for any Y C X, Y 0, proving the theorem if X is finite.
Case 2. Let X be general and 0 Y finite. Suppose Cy is not 1-maximal in Y. Then by Case 1, for any finite Z D Y, Cz is not 1-maximal in Z. Let E be the class of all B C Y such that B Cy and S({B} U Cy) < 1. Suppose for each B E £ there is some finite Z(B) D Y such that there is no A C Z(B) with S({A} U CZ(B)) < 1 and A n Y = B. Let Z := UBEE Z(B). Then Z is a finite set including Y. So Cz is strictly included in some class D which is 1-maximal in Z, by Theorem 4.3.1 (with A = 2Z), and Dy is 1-maximal
*4.4 Classes of index 1
151
by Case 1. So B E Dy for some B E E, say D n Y = B for some D E D. Then A := D n Z(B) gives a contradiction.
So, for some B E E and any finite Z D Y, there is an A C Z such that A n Y = B and S(Cz U {A}) < 1. Let Dz be the class of all subsets D of X for which D n z is such a set A. Then Dz is compact in the product topology of 2X (which was treated in Proposition 4.3.5). For any finite U D Y
and V D Y, DU n Dv D DUuV 0 0. So the intersection of all DU is nonempty. Let D E DU for all finite U D Y. Then S(C U {D}) = 1, so
D E C, but D n Y = B
Cy, a contradiction. This finishes the proof in
Case 2.
Case 3. Let X be infinite and U any infinite subset. Let T (U) := {D C U : D n Y E Cy for all finite Y C U}. Then clearly CU C .F(U). Conversely, if D E .F(U), then for each finite Z C X, let Y := Z n U. Let g(Z) := {A E C : A n Y = D n Y}. Then 9(Z) is compact by Proposition 4.3.5, nonempty since D E .F(U), and decreases as Z increases. Thus there is some C E nz g(Z). Then for any finite Z, C n Z n U = D n z n U, so C n U = D n U = D. So D ECU and CU =.F(U). For each finite, nonempty Y C U, Cy is 1-maximal in Y by Case 2. Suppose
.F(U) is not 1-maximal in U. Let A C U, A .F(U), S(E) < 1 where E = (Al U.F(U). For each finite Y C U, Cy C Sy, so Cy = Ey. Let fly :=
{BEC: BnY=AnY}. Then Ry00andasshown above, I
lyfy#0.
Let B E ly Ry. Then A = B n U E Cu = .F(U), a contradiction. So CU is 1-maximal in U. I
For any set X and C C 2X, let x
4.4.14 Theorem If S(C) = 1, UBEC B = X, and C satisfies (4.4.2) for all x y, then
Proof Suppose S(C) = 1 and (4.4.2) holds for all x y. Let x
152
Vapnik-Cervonenkis Combinatorics
C, D E C with x E C \ D, y E D \ C. Then C shatters {x, y), a contradiction. So {x : x
y<x}EC,and0EC,soS(C)=1. Let A C X and suppose A is not linearly ordered by <. Take x, y E A which are not comparable for <. Since 0, Lx and L y are all in C, C U (A) shatters {x, y), and S(C U (A}) > 2.
Let B C X and suppose there exist x< y E B with x V B. Then Lx E C, Ly E C, and B fl {x, y} _ {y} imply that S(C U {B)) > 2. This has now been shown for any B V C, so C is 1-maximal. Example. Let X = 11, 2, 3, 4, 51 and
9=(0,(1),(5), 12,51,11,2,41,11,2,3,41,11,2,3,51,11,2,3,4,5)1. Let C be the complement of g in 2X. Then it can be checked that C is 3-maximal
but for Y = (1, 2, 3, 41, Cy is not 3-maximal in Y. So in Theorem 4.4.12, "1maximal" and "Y 0" cannot be replaced by "3-maximal" and "JYJ > 3" respectively.
*4.5 Combining VC classes Recalling the density as in Corollary 4.1.7, the following is clear:
4.5.1 Theorem For any set X, if A C 2X and C C 2X then dens(A U C) = max(dens(A), dens(C)). For the Vapnik-Cervonenkis index we have instead:
4.5.2 Proposition For any set X, A C 2X and C C 2X, S(A U C) < S(A) + S(C) + 1. This bound is optimal: for any nonnegative integers k and m there exist X and A, C C 2X with S(A) = k, S(C) = m, and S(A U C) _
k+m+1.
*4.5 Combining VC classes
153
Proof By Theorem 4.1.2, if k := S(A), m := S(C), and n > k + m + 2,
m,auc(n) < mA(n)+mC(n) < nC
(n) + j=0
j=n-m
(n)
<
2n,
so S(A U C) < n. It follows that S(A U C) < k + m + 1 as claimed. Conversely, given k and m let n = k + m + 1 and let X be a set with n members. Let A be the set of all subsets of X with at most k members and let C be the set of subsets
of X with at least n - m = k + 1 members. Then S(A) = k, S(C) = m, and A U C = 2X, so S(A U C) = n. Let X be a set and C, D any two collections of subsets of X. Let
CnD
{C n D: CEC, DED},
CuD
{CUD: CEC, DED}.
If A is a class of subsets of another set Y let
C®A := {CXA: CEC, AEA}. For such classes, we have:
4.5.3 Theorem For any C C 2X and D C 2X or 2Y let k := dens(C) and m := dens(D). Then we have dens(C El D) < k + m for El = n, u, or S. Proof For any of the three operations, we have m c E IV (n) < m c (n )m D (n ), and the conclusions follow.
For the Vapnik-Cervonenkis index the behavior of the El operations is not so simple. For k, m = 0, 1, 2, , and D = u, n, or ®, let D(k, m) :_ max{S(C El D) : S(C) = k, S(D) = m}. Here the maximum is taken where X and Y may be infinite sets. Then we have:
4.5.4 Theorem For any k = 0, 1, 2,
and m = 0, 1, 2,
, and El = u,
n, or ®, we have D(k, m) < oo.
Proof By Theorem 4.1.2 and Proposition 4.1.5, if S(C) = k and S(D) = m, then
mcoD(n) < mc(n)mD(n) < 9nk+m1(4k!m!) < 2n for n > no(k, m) large enough.
154
Vapnik-Cervonenkis Combinatorics
4.5.5 Theorem F o r any k, in = 0, 1 , 2,
,
n(k, m) = u(k, m) = ®(k, m).
Proof The first equation follows from taking complements (Proposition 4.1.8). For the second, for given k and in, by Theorem 4.5.4 it is enough to consider
large enough finite sets in place of X and Y, and then we can assume X =
Y. We have n(k, m) < ®(k, m) by restricting to the diagonal in X x X and applying Proposition 4.3.2. In the other direction, let fix and fly be the projections of X x Y onto X and Y respectively. Let C C 2X and A C 2Y.
Let.F:= (IIXI(C): C E C), B = {IIYI(A): A E A}. By Theorem 4.2.3, S(.T) = S(C) and S(B) = S(A). Since I} XI (C) n IIY I (A) = C x A it follows that S(.F n 8) > S(C ® A), and the theorem is proved. Let S(k, m) be the common value of the quantities in Theorem 4.5.5. Theorem 4.5.4 can be improved as follows. For any nonnegative integers j, k let
9(j, k) := sup{r E N: (rC 2r}. Then 9(j, k) < oo for each j, k by Proposition 4.1.5, and we have:
4.5.6 Proposition S(j, k) < B (j, k) for any j, k E N.
Proof Let S(C) = j and S(D) = k. Then for any n > 6(j, k), by Sauer's lemma (Theorem 4.1.2) and Proposition 4.1.5 again, mcuD(n)
< mc(n)m"(n) < (nC<j)(nC
Can the values S(k, m) be computed? It is shown in Dudley (1984, Proposition 9.2.8) that S(1, 1) = 3 (which is not a corollary of Proposition 4.5.2, apparently). Also, after Proposition 9.2.10 there, an example of L. Birge is given showing that S(l, 2) > 5. For classes satisfying stronger conditions, more can be proved:
4.5.7 Theorem Let X and Y be sets, C C 2X and D C 2Y. If C is linearly ordered by inclusion, then S(C ® D) < S(D) + 1. Proof Let m = S(D). We can assume that m < oo and X E C. Let F C X x Y
with IFI = in + 2. Suppose C ®D shatters F. The subsets of IIXF C X induced by C are linearly ordered by inclusion. Let G be the next largest subset
other than I1XF itself, and p E f}XF \ G. Take y such that (p, y) E F. Then all subsets of F containing (p, y) are induced by sets of the form I}XF x D, D E D. Thus H := F \ {(p, y)} is shattered by such sets. Now I H I = in + 1, and IIY must be one-to-one on H or it could not be shattered by sets of the given form. But then D shatters flyH, giving a contradiction.
*4.5 Combining VC classes
155
4.5.8 Theorem For any set X and C, D C 2X, if C is linearly ordered by inclusion, then S(C D D) < S(D) + 1 for D = n or u.
Proof First consider D = n. Again let m = S(D), and we can assume m < oo. Let F C X and I Fl = m + 2. Suppose C n D shatters F. Now CF is linearly ordered by inclusion. We can assume that X E C, so F E CF. Let G be the next largest element of CF and p E F \ G. Each set A C F containing p is of the form C f1 D fl F, C E C, D E D, so we must have C f1 F = F, and A = D fl F. So D shatters F \ (p), a contradiction. Since the complements of a class linearly ordered by inclusion are again linearly ordered by inclusion, the case of unions follows by taking complements (Proposition 4.1.8).
Then by Theorem 4.2.6 and induction we have:
4.5.9 Corollary Let Ci be classes of subsets of a set X and C := {n"=1 C; : Ci E Ci, i = 1, , n}, where each C1 is linearly ordered by inclusion. Then S(C) < n. Definition. For any set X and Vapnik-Cervonenkis class C C 2X, C will be
called bordered if for some F C X, with I Fl = S(C), and x (= X \ F, F is shattered by sets in C all containing x.
4.5.10 Theorem Let Ci C 2x0) be bordered Vapnik-Cervonenkis classes for i = 1, 2. Then S(C1®C2) > S(C1) + S(C2).
Proof Take F C X(i) and xi as in the definition of "bordered." Let H := ({x1} x F2) U (F1 x {x2}). Then IHI = S(C1) + S(C2). For any Vi C F, i = 1, 2, take Ci E Ci with Ci fl Fi = Vi and xi E Ci. Then (C1 x C2) fl H = ({x1 } x V2) U (V1 x {x21), so Ci ®C2 shatters H.
Theorem 4.5.10 extends by induction to any number of factors. One consequence is:
4.5.11 Corollary In JR let ,7 be the set of all intervals, which may be open or closed, bounded or unbounded on each side. In other words, ,7 is the set of all convex subsets of R. In JR let C be the collection of all rectangles parallel , m }. Then S(C) = 2m. to the axes, C :_ {III" 1 J : J i E ,7, i = 1, Let D be the set of all left half-lines (-oo, x] or (-oo, x) for x E R. Let
Vapnik-Cervonenkis Combinatorics
156
T :_ {IIm 1 Hi : Hi E D, i = 1, , m}, so T is the class of lower orthants parallel to the given axes. Then S(C) = m. Proof The class D is linearly ordered by inclusion and is bordered with S(D) =
1, and so is the class of half-lines [a, oo) or (a, oo), a E JR. The class ,7 of all intervals in R is bordered with S(J) = 2. The results now follow from Corollary 4.5.9 with n = 2m and n = m, and Theorem 4.5.10 and induction.
4.5.12 Proposition Let ,7 be the set of all intervals in JR. Let Y be any set and C C 2t' with Y E C. Then in JR x Y, S(,7 ®C) < 2 + S(C).
Proof If S(C) = +oo the result is clear, so suppose m := S(C) < oo. Let F C ] I R x Y with I FI = 3 + m and suppose ,7 ®C shatters F. Let (xi, yi ),
i=
, m + 3, be the points of F. Let u := mini xi and v := maxi xj. Let
1,
p := (u, yi) E F and q := (v, yj) E F. All subsets of F which include 1p, q) must be induced by sets of the form JR x C, C E C. So n y must be one-to-one on F \ { p, q }, and C shatters n y (F \ (p, q }) of cardinality m + 1, a contradiction.
4.6 Probability laws and independence Let (X, A, P) be a probability space. Recall the pseudometric dp (A, B) P(AAB) on A, where AAB := (A \ B) U (B \ A). Recall also that for any (pseudo) metric space (S, d) and s > 0, D(s, S, d) denotes the maximum number of points more than e apart (Section 1.2).
For a measurable space (X, A) and C C A let s(C) := inf{w : there is a K = K(w, C) < oo such that for every law P on A and 0 < s < 1, D(s, C, dp) < Ke-w). Definition.
This index s(C) turns out to equal the density:
4.6.1 Theorem For any measurable space (X, A) and C C A, dens(C) _ S (C).
Proof Let P be a probability measure on A. Suppose A,, , A, E C and be i.i.d. dp(Ai, Aj) > s > 0 for 1 < i < j < m, with m > 2. Let X1, X2, (P), specifically coordinates on a countable product of copies of (X, A, P).
4.6 Probability laws and independence Note that P may have atoms. For n = 1 , 2, Pr{for some i
157
,
j, Xk 0 Al DAB for all k < n)
< in (m - 1)(1 - s)'/2 < 1 forn large enough, n > -log(m(m-1)/2)/log(1-e). LetPn := n FY=18x,. (as usual) be empirical measures for P. For such n, there is positive probability that Pn (Ai AAA) > 0 for all i j, and so Ai and Aj induce different subsets of {X1, , Xn} and mc(n) > m. For any r > dens(C) there is an M =
M(r, C) < oo, where we can take M > 2, such that mc(n) < Mnr for all n.
Note that -log(1 - s) > E. Thus form > 2, m < M(2log(m2))rs-r, or m (log m) -r < M1 8-r for some M1 = Ml (r, C) = 4r M. For any 3 > 0 and ml-g < C large enough, (log m)' < Cm8 for all in > 1. Thus for all m > 0, M28-r for 0 < s < 1 for some M2 = M2(r, C, 8). Thus m < (M28-r)11(1-1)
Letting r ,I, dens(C) and 8 ,{ 0 gives dens(C) > s(C). In the converse direction, let IA I := card(A). Since it is not the case that
mc(n) < knt f o r all n > 1, for r < t < dens(C) and k = 1, 2, , let Ak C X with Ak 0 0 and I Ak n CI > k I Ak I t. Then I Ak I - oo as k oo. Let B0 := Al. Other sets Bj will be defined recursively. Let Bo, , B,_1 be disjoint subsets of X and let C(n) := Uo<j
Let Bn := Ak(n) \ C(n). So all the Bn are defined and disjoint. Each set in Bn n C is induced by at most 21 C(n)I different sets in Ak(n) n C. Thus
IBn n CI > IAk(n) n CI/21C(n)I > 224IAk(n)It > 224IBnIt. Since Bn is nonempty, it follows that I Bn n C I > 22', and hence I Bn I > 2n Let
an := IBnI-t/r,
S:_
IBnl1-t/r
< 00.
n=0
Let P be the probability measure on Un_1 Bn C X giving mass an/S to each point of Bn for each n. The distinct sets in Bn n C are at dp-distance at least an / S apart, and so are a set of elements of C which induce the subsets of Bn. So for all n,
D(an1(2S), C, dp) > I B. n CI > 224 I Bn I t
= 224an r = 224 (2S)-r(an/(2S))-r For s := an/(2S) -* 0 as n -+ oo, this implies D(s, C, dp) > 2 24
(2S)-rs-r
158
Vapnik-Cervonenkis Combinatorics
Since 2" - oo as a" 4. 0 this implies r < s(C). Then letting r f dens(C), the proof is complete. There is a notion of independence for sets without probability. To define it, for
any set X and subset A C X let A l := A and A-1 := X \ A. Sets A l , , A,,, from are called independent, or independent as sets, if for every function {1, , m} into {-1, +1}, n 1 As(j) 0 0. Such intersections, when they are , Am. nonempty, are called atoms of the Boolean algebra generated by A 1 , Thus for A 1, , Am to be independent as sets means that the Boolean algebra they generate has the maximum possible number, 2m, of atoms. If A 1, , Am are independent as sets, then one can define a probability law on the algebra they generate for which they are jointly independent in the usual probability sense and for which P(Ai) = 1/2, i = 1, , m. For example, choose a point in each atom and put mass 1/2m at each point chosen. Or, if desired, given any qi, 0 < qi < 1, one can define a probability measure Q for which the Ai are jointly independent and have Q(Ai) = qi, i = 1, , n. For a set X and C C 2X let I (C) be the supremum of m such that A t ,
are independent as sets for some Ai E C, i = 1,
, Am
, M.
4.6.2 Theorem F o r any set X , C C 2X, and n = 1, 2, , if S(C) > 2" then I (C) > n. Conversely, if I (C) > 2" then S(C) > n. So I (C) < 00 if and only if S(C) < oo. In both cases, 2" cannot be replaced by 2" - 1.
Proof Clearly, if a set Y has n or more independent subsets, then I Y I > 2'. Conversely, if I Y I = 2", we can assume that Y is the set of all strings of n digits
each equal to 0 or 1. Let Aj be the set of strings for which the jth digit is 1. Then the Aj are independent. It follows that if S(C) > 2", then I(C) > n, while if I Y I = 2" - 1 and C = 21', then S(C) = 2" - 1 while I (C) < n, as stated.
Conversely, if Bj are independent as sets for j = 1, , 2", Bj E C, let A(i) := Ai be independent subsets of {1, , 2"} for i = 1, , n. Choose
xi E nJEA(i) Bi n n0A(i)(X \ Bi). Then xi E Bj if and only if j E A(i).
For each setS C {1, ,n},
nAi nn({1,...,2"}\Ai) iES
for some j := is E 11, B; n
i0s , 2" }. Then j E Ai if and only if i E S, and (xi: i E S).
4.7 Vapnik-Cervonenkis properties of classes of functions
159
So C shatters {xl, , xn) and S(C) > n, as stated. If C consists of 2' - 1 (independent) sets, then clearly S(C) < n.
For any set X, C C 2X and Y C X, recall that Cy := Y n c := {Y fl C : Y) be the set of atoms of the algebra of subsets of Y
C E C}. Let At(C
I
generated by Cy, where in the cases to be considered, Cy will be finite because
C or Y is. Let Ac(Y) := IAt(C Y)J be the number of such atoms. Let mc(n) := sup{DA(Y) : A C C, IAA < n} < 2n. Let dens*(C) denote the infimum of all s > 0 such that for some C < oo, me (n) < Cn' for all n. For anyx E XletCX = {A E C: X E A}. LetC'' := {Cy: y E Y}. I
4.6.3 Theorem For any set X and A C C C 2X, with A finite,
(a) AA(X) = AcX (A); (b) for n = 1, 2, ... , me (n) = mcx'(n); (c) S(CX) = I (C); (d) dens*(C) = dens(CX) < I(C).
Proof For any B C A let
a(B) := n B fl n (X \ A). BES
AEA\l3
Then a (B) is an atom, in At(AI X), if and only if it is nonempty. Now y E a (B)
if and only if A fl Cy = B, so (a) follows. Then taking the maximum over A with IAA = n on both sides of (a) gives (b), which then implies (c) and (d). The last inequality follows from Corollary 4.1.7.
4.7 Vapnik-Cervonenkis properties of classes of functions The notion of VC class of sets has several extensions to classes of functions.
Definitions. Let X be a set and.F a class of real-valued functions on X. Let C C 2X. If f is any real-valued function, each set If > t } fort E R will be called a major set of f. The class .F will be called a major class for C if all the major sets of each f E ,F are in C. If C is a Vapnik-Cervonenkis class, then .F will be called a VC major class (for C). The subgraph of a real-valued function f will be the set {(x, t) E X x R:
0 < t < f(x) or f(x) < t < 01. If D is a class of subsets of X x R, and for each f E .F, the subgraph of f is in D, then .F will be called a subgraph class for D. If D is a VC class in X x JR then .F will be called a VC subgraph class.
Vapnik-Cervonenkis Combinatorics
160
The symmetric convex hull of F is the set of all functions E"'_1 c; f for f E F, c; E I[8, any finite m, and Ym1 Ic; I < 1. If 0 < M < oo, let H(F, M) denote M times the symmetric convex hull of F. Let HS (F, M) be the smallest class 9 of functions including H(.F, M) such that whenever gn E 9 for all n and gn (x) -> g(x) as n -+ oo for all x, we have g E tj. A class F of functions such that F C HS(G, M) for some M < oo and a given 9 will be called a VC subgraph hull class if G is a VC subgraph class, and a VC hull class if g = (IC: C E C} where C is a VC class of sets. So there are at least four possible ways to extend the notion of VC class to classes of functions. Some implications hold between these different conditions, but no two of them are equivalent. The next theorem deals with some of the easier cases of implication or non-implication.
4.7.1 Theorem Let F be a uniformly bounded class of nonnegative realvalued functions on a set X. Then
(a) if F is the set of indicators of members of a VC class of sets, then F is also a VC major class, a VC subgraph class, and a VC hull class; (b) if F is a VC major class then it is a VC hull class; (c) there exist VC hull classes F which are not VC major; (d) there exist VC subgraph classes F which are not VC major. Proof (a) The indicators of a VC class C of sets clearly form a VC major class (forCU{0, X}) and aVChull class. Also, the class of sets (Cx[0, 1])U(Xx{0}) for C E C form a VC class, for example by Theorem 4.5.4. So (a) holds. (b) Since .F is uniformly bounded and the definition of VC hull class allows
multiplication by a constant, we can assume 0 < f < 1 for all f E F. For such an f let n-1
A
n
n-1
l{f>j/n} j=1
jn l{ j/n< f<(j+1)/n}
j=0
Then fn (x) -> f(x) as n -+ cc for all x. (In fact, f(x)-1/n < fn (x) < f(x) for all x, so fn -> f uniformly.) So .F is a VC hull class. (c) Let X =1R2. Let C be the set of open lower left quadrants {(x, y) : x < a, y < b} for all a, b E R. Then S(C) = 2 by Corollary 4.5.11. Let F be the set of all sums Y-k 1 lc(k)/2k where C(k) E C. Then clearly .T' is a VC hull class. The sets where f > 0 for f in .F are exactly the countable unions of sets
4.8 Classes of functions and dual density
161
in C. But such unions are not a VC class; for example, they shatter any finite subset of the line x + y = 1. So .F is not a VC major class and (c) is proved. (d) Let f, := n-1 + n-21 B. for n = 1, 2, and for any measurable sets Bn. Then fn 4. 0, so the subgraphs of the functions fn are linearly ordered by inclusion and form a VC class of index 1 (Theorem 4.2.6(a)). Now If, > 1/n) = Bn for each n, and the sequence {Bn} need not form a VC class, for example let Bn be a sequence of independent sets (see Section 4.6). Then If,) is not a VC major class, proving (d).
4.8 Classes of functions and dual density For a metric space (S, d) and e > 0 recall D(s, S, d), the maximum number of points more than E apart. For a probability measure Q and 1 < p < oo we
have theLpmetric dp,Q(f,g):=(f If-gIPdQ)1/p.For aclass FC£"(Q) let D(p) (E, F, Q) := D(e, Jr, dp,Q). Let D(p) (E, F) be the supremum of D(E, F, dp,Q) over all laws Q concentrated in finite sets. If F is a class of measurable real-valued functions on a measurable space
(X, A), let FF(x) := supfEJ I f(x)I. Then a measurable function F will be called an envelope function for.F if and only if FF < F. If F2 is measurable it will be called the envelope function of F. For any law P on (X, A), F* is an envelope function for F, which in general depends on P.
Given .F, an envelope function F for it, E > 0, and 1 < p < oo, let DF i (e, F, Q) be the supremum of m such that ther