Probability for Statisticians

Preface There is a thin self-contained textbook within this larger presentation. To be sure that this is well understoo...

Author: Galen R. Shorack

115 downloads 1598 Views 3MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Preface There is a thin self-contained textbook within this larger presentation. To be sure that this is well understood, I will describe later how I have used this text in the classroom at the University of Washington in Seattle. Let me ﬁrst indicate what is diﬀerent about this book. As implied by the title, there is a diﬀerence. Not all the diﬀerence is based on inclusion of statistical material. (To begin, Chapters 1–6, provide the mathematical foundation for the rest of the text. Then Chapters 7–8 hone some tools geared to probability theory, while Chapter 9 provides a brief introduction to elementary probability theory right before the main emphasis of the presentation begins.) The classical weak law of large numbers (WLLN) and strong law of large numbers (SLLN) as presented in Sections 10.2–10.4 are particularly complete, and they also emphasize the important role played by the behavior of the maximal summand. Presentation of good inequalities is emphasized in the entire text, and this chapter is a good example. Also, there is an (optional) extension of the WLLN in Section 10.6 that focuses on the behavior of the sample variance, even in very general situations. Both the classical central limit theorem (CLT) and its Lindeberg and Liapunov generalizations are presented in two diﬀerent chapters. They are ﬁrst presented in Chapter 11 via Stein’s method (with a new twist), and they are again presented in Chapter 14 using the characteristic function (chf) methods introduced in Chapter 13. The CLT proofs given in Chapter 11 are highly eﬃcient. Conditions for both the weak bootstrap and the strong bootstrap are developed in Chapter 11, as is a universal bootstrap CLT based on light trimming of the sample. The approach emphasizes a statistical perspective. Much of Section 11.1 and most of Sections 11.2–11.5 are quite unusual. I particularly like this chapter. Stein’s method is also used in the treatment of U-statistics and Hoeﬀding’s combinatorial CLT (which applies to sampling from a ﬁnite population) in the optional Chapter 17. Also, the chf proofs in Section 14.2 have a slightly unusual starting point, and the approach to gamma approximations in the CLT in Section 14.4 is new. Both distribution functions (dfs F (·)) and quantile functions (qfs K(·) ≡ F−1 (·)) are emphasized throughout (quantile functions are important to statisticians). In Chapter 7 much general information about both dfs and qfs and the Winsorized variance is developed. The text includes presentations showing how to exploit the inverse transformation X ≡ K(ξ) with ξ ∼ = Uniform(0, 1). In particular, Chapter 7 inequalities relating the qf and the Winsorized variance to some empirical process results of Chapter 12 are used in Chapter 16 to treat trimmed means and L-statistics, rank and permutation tests, sampling from ﬁnite populations, and bootstrapping. (Though I am very fond of Sections 7.6–7.11, their prominence is minimized in the subsequent parts of the text.) At various points in the text choices can be voluntarily made that will oﬀer the opportunity for a statistical example or foray. (Even if the instructor does not exercise a particular choice, a student can do so individually.) After the elementary introduction to probability theory in Section 9.1, many of the classical distributions

viii

PROBABILITY FOR STATISTICIANS

of statististics are introduced in Section 9.2, while useful linear algebra and the multivariate normal distribution are the subjects of Section 9.3 and Section 9.4. Following the CLT via Stein’s method in Section 11.1, extensions in Section 11.2– 11.3, and application of these CLTs to the bootstrap in Sections 11.4–11.5, there is a large collection of statistical examples in Section 11.6. During presentation of the CLT via chfs in Chapter 14, statistical examples appear in Sections 14.1, 14.2, and 14.4. Statistical applications based on the empirical df appear in Sections 12.10 and 12.12. The highly statistical optional Chapters 16 and 17 were discussed brieﬂy above. Also, the conditional probability Sections 8.5 and 8.6 emphasize statistics. Maximum likelihood ideas are presented in Section A.2 of Appendix A. Many useful statistical distributions contain parameters as an argument of the gamma function. For this reason, the gamma and digamma functions are ﬁrst developed in Section A.1. Section A.3 develops cumulants, Fisher information, and other useful facts for a number of these distributions. Maximum likelihood proofs are in Section A.4. It is my hope that even those well versed in probability theory will ﬁnd some new things of interest. I have learned much through my association with David Mason, and I would like to acknowledge that here. Especially (in the context of this text), Theorem 12.4.3 is a beautiful improvement on Theorem 12.4.2, in that it still has the potential for necessary and suﬃcient results. I really admire the work of Mason and his colleagues. It was while working with David that some of my present interests developed. In particular, a useful companion to Theorem 12.4.3 is knowledge of quantile functions. Sections 7.6–7.11 present what I have compiled and produced on that topic while working on various applications, partially with David. Jon Wellner has taught from several versions of this text. In particular, he typed an earlier version and thus gave me a major critical boost. That head start is what turned my thoughts to writing a text for publication. Sections 8.6, 19.2, and the Hoﬀman–Jorgensen inequalities came from him. He has also formulated a number of exercises, suggested various improvements, oﬀered good suggestions and references regarding predictable processes, and pointed out some diﬃculties. My thanks to Jon for all of these contributions. (Obviously, whatever problems may remain lie with me.) My thanks go to John Kimmel for his interest in this text, and for his help and guidance through the various steps and decisions. Thanks also to Lesley Poliner, David Kramer, and the rest at Springer-Verlag. It was a very pleasant experience. This is intended as a textbook, not as a research manuscript. Accordingly, the main body is lightly referenced. There is a section at the end that contains some discussion of the literature.

Contents Use of this Text xiii Deﬁnition of Symbols

xviii

Chapter 1. Measures 1. Basic Properties of Measures 1 2. Construction and Extension of Measures 3. Lebesgue–Stieltjes Measures 18

12

Chapter 2. Measurable Functions and Convergence 1. 2. 3. 4. 5.

Mappings and σ-Fields 21 Measurable Functions 24 Convergence 29 Probability, RVs, and Convergence in Law Discussion of Sub σ-Fields 35

33

Chapter 3. Integration 1. 2. 3. 4. 5.

The Lebesgue Integral 37 Fundamental Properties of Integrals 40 Evaluating and Diﬀerentiating Integrals 44 Inequalities 46 Modes of Convergence 51

Chapter 4. Derivatives via Signed Measures 1. 2. 3. 4.

Decomposition of Signed Measures 61 The Radon–Nikodym Theorem 66 Lebesgue’s Theorem 70 The Fundamental Theorem of Calculus 74

Chapter 5. Measures and Processes on Products 1. 2. 3. 4.

Finite-Dimensional Product Spaces 79 Random Vectors on (Ω, A, P ) 84 Countably Inﬁnite Product Probability Spaces Random Elements and Processes on (Ω, A, P )

86 90

Chapter 6. General Topology and Hilbert Space 1. General Topology 95 2. Metric Spaces 101 3. Hilbert Space 104

x

PROBABILITY FOR STATISTICIANS

Chapter 7. Distribution and Quantile Functions 1. Character of Distribution Functions 107 2. Properties of Distribution Functions 110 3. The Quantile Transformation 111 4. Integration by Parts Applied to Moments 115 5. Important Statistical Quantities 119 6. Inﬁnite Variance 123 7. Slowly Varying Partial Variance 127 8. Speciﬁc Tail Relationships 134 9. Regularly Varying Functions 137 10. Some Winsorized Variance Comparisons 140 11. Inequalities for Winsorized Quantile Functions

147

Chapter 8. Independence and Conditional Distributions 1. 2. 3. 4. 5. 6.

Independence 151 The Tail σ-Field 155 Uncorrelated Random Variables 157 Basic Properties of Conditional Expectation 158 Regular Conditional Probability 168 Conditional Expectations as Projections 174

Chapter 9. Special Distributions 1. 2. 3. 4.

Elementary Probability 179 Distribution Theory for Statistics 187 Linear Algebra Applications 191 The Multivariate Normal Distribution 199

Chapter 10. WLLN, SLLN, LIL, and Series 0. Introduction 203 1. Borel–Cantelli and Kronecker Lemmas 204 2. Truncation, WLLN, and Review of Inequalities 206 3. Maximal Inequalities and Symmetrization 210 4. The Classical Laws of Large Numbers, LLNs 215 5. Applications of the Laws of Large Numbers 223 6. General Moment Estimation 226 7. Law of the Iterated Logarithm 235 8. Strong Markov Property for Sums of IID RVs 239 9. Convergence of Series of Independent RVs 241 10. Martingales 246 11. Maximal Inequalities, Some with Boundaries 247 12. A Uniform SLLN 252

CONTENTS

xi

Chapter 11. Convergence in Distribution 1. 2. 3. 4. 5. 6. 7. 8.

Stein’s Method for CLTs 255 ˜ ˇ Winsorization and Truncation 264 Identically Distributed RVs 269 Bootstrapping 274 Bootstrapping with Slowly Increasing Trimming Examples of Limiting Distributions 279 Classical Convergence in Distribution 288 Limit Determining Classes of Functions 292

276

Chapter 12. Brownian Motion and Empirical Processes 1. Special Spaces 295 2. Existence of Processes on (C, C) and (D, D) 298 3. Brownian Motion and Brownian Bridge 302 4. Stopping Times 305 5. Strong Markov Property 308 6. Embedding a RV in Brownian Motion 311 7. Barrier Crossing Probabilities 314 8. Embedding the Partial Sum Process 318 9. Other Properties of Brownian Motion 323 10. Various Empirical Processes 325 11. Inequalities for the Various Empirical Processes 333 12. Applications 338

Chapter 13. Characteristic Functions 1. 2. 3. 4. 5. 6. 7.

Basic Results, with Derivation of Common Chfs 341 Uniqueness and Inversion 346 The Continuity Theorem 350 Elementary Complex and Fourier Analysis 352 Esseen’s Lemma 358 Distributions on Grids 361 Conditions for φ to Be a Characteristic Function 363

Chapter 14. CLTs via Characteristic Functions 0. 1. 2. 3. 4. 5. 6.

Introduction 365 Basic Limit Theorems 366 Variations on the Classical CLT 371 Local Limit Theorems 380 Gamma Approximation 383 Edgeworth Expansions 390 ¯n ) Approximating the Distribution of h(X

396

xii

PROBABILITY FOR STATISTICIANS

Chapter 15. Inﬁnitely Divisible and Stable Distributions 1. 2. 3. 4.

Inﬁnitely Divisible Distributions 399 Stable Distributions 407 Characterizing Stable Laws 410 The Domain of Attraction of a Stable Law

412

Chapter 16. Asymptotics via Empirical Proceses 0. 1. 2. 3. 4.

Introduction 415 Trimmed and Winsorized Means 416 Linear Rank Statistics and Finite Sampling The Bootstrap 432 L-Statistics 437

426

Chapter 17. Asymptotics via Stein’s Approach 1. U-Statistics 449 2. Hoeﬀding’s Combinatorial CLT

458

Chapter 18. Martingales 1. Basic Technicalities for Martingales 467 2. Simple Optional Sampling Theorem 472 3. The Submartingale Convergence Theorem 473 4. Applications of the S-mg Convergence Theorem 481 5. Decomposition of a Submartingale Sequence 487 6. Optional Sampling 492 7. Applications of Optional Sampling 499 8. Introduction to Counting Process Martingales 501 9. Doob–Meyer SubmartingaleDecomposition 511 · 10. Predictable Processes and 0 H dM Martingales 516 11. The Basic Censored Data Martingale 522 12. CLTs for Dependent RVs 529

Chapter 19. Convergence in Law on Metric Spaces 1. Convergence in Distribution on Metric Spaces 2. Metrics for Convergence in Distribution 540

531

Appendix A. Distribution Summaries 1. 2. 3. 4.

The Gamma and Digamma Functions 546 Maximum Likelihood Estimators and Moments Examples of Statistical Models 555 Asymptotics of Maximum Likelihood Estimation

References Index 575

568

551 563

Use of this Text The University of Washington is on the quarter system, so my description will reﬂect that. My thoughts are oﬀered as a potential help, not as an essential recipe. The reader will note that the problems are interspersed with the text. It is important to read them as they are encountered. Chapters 1–5 provide the measure-theoretic background that is necessary for the rest of the text. Many of our students have had at least some kind of an undergraduate exposure to part of this subject. Still, it is important that I present the key parts of this material rather carefully. I feel it is useful for all of them. Chapter 1 (measures; 5 lectures) Emphasized in my presentation are generators, the monotone property of measures, the Carath´eodory extension theorem, completions, the approximation lemma, and the correspondence theorem. Presenting the correspondence theorem carefully is important, as this allows one the luxury of merely highlighting some proofs in Chapter 5. [The minimal monotone class theorem of Section 1.1, claim 8 of the Carath´edory extension theorem proof, and most of what follows the approximation lemma in Section 1.2 would never be presented in my lectures.] {I always assign Exercises 1.1.1 (generators), 1.2.1 (completions), and 1.2.3 (the approximation lemma). Other exercises are assigned, but they vary each time.} Chapter 2 (measurable functions and convergence; 4 lectures) I present most of Sections 2.1, 2.2, and 2.3. Highlights are preservation of σ-ﬁelds, measurability of both common functions and limits of simple functions, induced measures, convergence and divergence sets (especially), and relating →µ to →a.s (especially, reducing the ﬁrst to the second by going to subsequences). I then assign Section 2.4 as outside reading and Section 2.5 for exploring. [I never lecture on either Section 2.4 or 2.5.] {I always assign Exercises 2.2.1 (speciﬁc σ-ﬁelds), 2.3.1 (concerning →a.e. ), 2.3.3 (a substantial proof), and 2.4.1 (Slutsky’s theorem).} Chapter 3 (integration; 7 lectures) This is an important chapter. I present all of Sections 3.1 and 3.2 carefully, but older, Section 3.3 is left as reading, and some of the Section 3.4 inequalities (Cr , H¨ Liapunov, Markov, and Jensen) are done carefully. I do Section 3.5 carefully as far as Vitali’s theorem, and then assign the rest as outside reading. {I always assign Exercises 3.2.1–3.2.2 (only the zero function), 3.3.3 (diﬀerentiating under the integral sign), 3.5.1 (substantial theory), and 3.5.7 (the Scheﬀ´e theorem).} Chapter 4 (Radon–Nikodym; 2 lectures) I present ideas from Section 4.1, sketch the Jordan–Hahn decomposition proof, and then give the proofs of the Lebesgue decomposition, the Radon–Nikodym theorem, and the change of variable theorem. These ﬁnal two topics are highlighted. The fundamental theorem of calculus of Section 4.4 is brieﬂy discussed. [I would never present any of Section 4.3.] {I always assign Exercises 4.2.1 (manipulating Radon– Nikodym derivatives), 4.2.7 (mathematically substantial), and 4.4.1, 4.4.2, and 4.4.4 (so that the students must do some outside reading in Section 4.4 on their own).}

xiv

PROBABILITY FOR STATISTICIANS

Chapter 5 (Fubini; 2 lectures) The ﬁrst lecture covers Sections 5.1 and 5.2. Proving Proposition 5.2.1 is a must, and I discuss/prove Theorems 5.1.2 (product measure) and 5.1.3 (Fubini). The remaining time is spent on Section 5.3. [I rarely lecture from Section 5.4, but I do assign it as outside reading.] {I always assign Exercises 5.3.1 (measurability in a countable number of dimensions) and 5.4.1 (the ﬁnite-dimensional ﬁeld).} Chapter 6 (topology and Hilbert space, 0 lectures) [This chapter is presented only for reference. I do not lecture from it.] The mathematical tools have now been developed. In the next three chapters we learn about some specialized probabilistic tools and then get a brief review of elementary probability. The presentation on the classic topics of probability theory then commences in Chapter 10. Chapter 7 (distribution functions (dfs) and quantile functions (qfs); 4 lectures) This chapter is quite important to this text. Skorokhod’s theorem in Section 7.3 must be done carefully, and the rest of Sections 7.1–7.4 should be covered. Section 7.5 should be left as outside reading. [Lecturing from Sections 7.6–7.11 is purely optional, and I would not exceed one lecture.] {I always assign Exercises 7.1.1 (on continuity of dfs), 7.3.3 (F −1 (·) is left continuous), 7.3.3 (change of variable), and 7.4.2 (for practice working with X ≡ K(ξ)). Consider lecturing on Theorem 7.6.1 (the inﬁnite variance case).} Chapter 8 (conditional expectation; 2 lectures) The ﬁrst lecture covers Sections 8.1 and 8.2. It highlights Proposition 8.1.1 (on the preservation of independence), Theorem 8.1.2 (extending independence from π-systems), and Kolmogorov’s 0-1 law. The other provides some discussion of the deﬁnition of conditional probability in Section 8.4, includes proofs of several parts of Theorem 8.4.1 (properties of conditional expectation), and discusses Deﬁnition 8.5.1 of regular conditional probability. [I never lecture on Sections 8.3, 8.5, or 8.6.] {I always assign Exercises 8.1.2 and 8.1.3 (they provide routine practice with the concepts), Exercise 8.4.1 (discrete conditional probability), Exercise 8.4.3 (repeated stepwise smoothing in a particular example), and part of Exercise 8.4.4 (proving additional parts of Theorem 8.4.1).} Chapter 9 (elementary probability; 0 lectures) Sections 9.1 and 9.2 were written to provide background reading for those graduate students in mathematics who lack an elementary probability background. Sections 9.3 and 9.4 allow graduate students in statistics to read some of the basic multivariate results in appropriate matrix notation. [I do not lecture from this chapter.] {But I do assign Exercises 9.1.8 (the Poisson process exists) and 9.2.1(ii) (so that the convolution formula is refreshed).} Chapter 10 (laws of large numbers (LLNs) and inequalities; 3 lectures for now) Since we are on the quarter system at the University of Washington, this leaves me 3 lectures to spend on the law of large numbers in Chapter 10 before the Christmas break at the end of the autumn quarter. In the ﬁrst 3 lectures I do Sections 10.1 and 10.2 with Khinchine’s weak law of large numbers (WLLN), Kolmogorov’s inequality only from Section 10.3, and at this time I present Kolmogorov’s strong law of large numbers (SLLN) only from Section 10.4. {I always assign Exercises 10.1.1 (Ces`aro summability), 10.2.1 (it generates good ideas related to the proofs), 10.2.3 (as it

USE OF THIS BOOK

xv

practices the important Op (·) and op (·) notation), 10.4.4 (the substantial result of Marcinkiewicz and Zygmund), 10.4.7 (random sample size), and at least one of the alternative SLLN proofs contained in 10.4.8, 10.4.9, and 10.4.10.} At this point at the beginning of the winter quarter the instructor will have his/her own opinions about what to cover. I devote the winter quarter to the weak law of large numbers (WLLN), an introduction to the law of the iterated logarithm (LIL), and various central limit theorems (CLTs). That is, the second term treats the material of Chapters 10–11 and 13–17. I will outline my choices for which parts to cover. Chapter 10 (LLNs, inequalities, LIL, and series; 6 lectures) My lectures cover Section 10.3 (symmetrization inequalities and L´evy’s inequality for the WLLN, and the Ottovani–Skorokhod inequality for series), Feller’s WLLN from Section 10.4, the Glivenko–Cantelli theorem from Section 10.5, the LIL for normal rvs in Proposition 10.7.1, the strong Markov property of Theorem 10.8.1, and the two series Theorem 10.9.2. [I do not lecture from any of Sections 10.6, 10.10, 10.11, or 10.12 at this time.] {I always assign Exercise 10.7.1 (Mills’ ratio).} Chapter 11 (CLTs via Stein’s method; 3 lectures) From section 11.1 one can prove Stein’s ﬁrst lemma and discuss his second lemma, prove the Berry–Esseen theorem, and prove Lindeberg’s CLT. Note that we have not yet introduced characteristic functions. Chapter 13 (characteristic functions (chfs); 6 lectures) I do sections 13.1–13.5. {I always assign Exercises 13.1.1 and 13.1.3(a) (deriving speciﬁc chfs) and 13.4.1 (Taylor series expansions of the chf).} Chapter 14 (CLTs via chfs; 6 lectures) The classical CLT, the Poisson limit theorem, and the multivariate CLT make a nice lecture. The chisquare goodness of ﬁt example and/or the median example (of Section 11.6) make a lecture of illustrations. Chf proofs of the usual CLTs are given in Section 14.2 (Section 13.5 could have been left until now). If Lindeberg’s theorem was proved in Chapter 11, one might do only Feller’s converse now via chfs. Other examples from either Section 14.2 or 11.6 could now be chosen, and Example 11.6.4 (weighted sums of iid rvs) is my ﬁrst choice. [The chisquare goodness of ﬁt example could motivate a student to read from Sections 9.3 and 9.4.] At this stage I still have at least 7 optional lectures at the end of the winter quarter and about 12 more at the start of the spring quarter. In my ﬁnal 16 lectures of the spring quarter I feel it appropriate to consider Brownian motion in Chapter 12 and then martingales in Chapter 18 (in a fashion to be described below). Let me ﬁrst describe some possibilities for the optional lectures, assuming that the above core was covered. Chapter 17 (U-statistics and Hoeﬀding’s combinatorial CLT) Sections 17.1 and 17.2 are independent of each other. The Berry–Esseen potential of Lemma 11.1.1 is required for Section 17.1. Either one or two lectures could then be presented on U-statistics from Section 17.1. The alternative Stein formulation of Motivation 11.1.1 is required for section 17.2. Two additional lectures would give the Hoeﬀding combinatorial CLT and its corollary regarding sampling from ﬁnite populations.

xvi

PROBABILITY FOR STATISTICIANS

Chapter 11 (statistical examples) Sections 11.6, 14.2, and 14.6 contain appropriate examples and exercises. Chapter 11 (bootstrap) Both Sections 11.4 and 11.5 on the bootstrap require only Theorem 11.2.1. Chapters 11 and 19 (convergence in distribution) Convergence in distribution on the line is presented in Sections 11.7 and 11.8. [This is extended to metric spaces in Chapter 19, but I do not lecture from it.] Chapter 11 (domain of normal attraction of the normal df) The converse of the CLT in Theorem 11.3.2 requires the Gin´e–Zinn symmetrization inequality and the Khinchine inequality of Section 13.3 and the Paley–Zygmund inequality of Section 3.4. Chapters 7, 10 and 11 (domain of attraction of the normal df) Combining Sections 7.6–7.8, the Section 10.3 subsection on maximal inequalities of another ilk, Section 10.6, and Sections 11.2–11.3 makes a nice unit. L´evy’s asymptotic normality condition (ANC) of (7.7.14) for a rv X has some prominence. In Chapter 7 purely geometric methods plus Cauchy–Schwarz are used to derive a multitude of equivalent conditions. In the process, quantile functions are carefully studied. In Section 10.6 the ANC is seen to be equivalent to a result akin to a WLLN for the rv X 2 , and so in this context many additional equivalent conditions are again derived. Thus when one comes to the CLT in Sections 11.2 and 11.3, one already knows a great deal about the ANC. Chapter 15 (inﬁnitely divisible and stable laws) First, Section 15.1 (inﬁnitely divisible laws) is independent of the rest, including Section 15.2 (stable laws). The theorem stated in Section 15.4 (domain of attraction of stable laws) would require methods of Section 7.9 to prove, but the interesting exercises are accessible without this. Chapter 14 (higher-order approximations) The local limit theorem in Section 14.3 can be done immediately for continuous dfs, but it also requires Section 13.6 for discrete dfs. The expansions given in Sections 14.4 (gamma approximation) and 14.5 (Edgeworth approximation) also require Exercise 13.4.6. Assorted topics suitable for individual reading These include Section 8.6 (on alternating conditional expectations), Section 10.12 (a uniform SLLN), Section 16.4 (L-statistics), Sections 18.8–18.11 (counting process martingales), and Section 18.12 (martingale CLTs). The primary topics for the spring quarter are Chapter 12 (Brownian motion and elementary empirical processes) and Chapter 18 (martingales). I have never covered Chapter 12 until the spring, but I placed it rather early in the text to make clear that it doesn’t depend on any of the later material. Chapter 12 (Brownian motion; 6 lectures) I discuss Section 12.1, sketch the proof of Section 12.2 and carefully apply that result in Section 12.3, and treat Section 12.4 carefully (as I believe that at some point a lecture should be devoted to a few of the more subtle diﬃculties regarding measurability). I am a bit cavalier regarding Section 12.5 (strong Markov property), but I apply it carefully in Sections 12.6, 12.7, and 12.8. I assign Section 12.9 as outside reading. [I do not lecture on Theorem 12.8.2.] {I always assign Exercises

USE OF THIS BOOK

xvii

12.1.2 (on (C, C)), 12.3.1 (various transforms of Brownian motion), 12.3.3 (integrals of normal processes), 12.4.1 (properties of stopping times), 12.7.3(a) (related to embedding a rv in Brownian motion), and 12.8.2 (the LIL via embedding).} At this point let me describe three additional optional topics that could now be pursued, based on the previous lectures from Chapter 12. Chapter 12 (elementary empirical processes) Uniform empirical and quantile processes are considered in Section 12.10. Straightforward applications to linear rank statistics and two-sample test of ﬁt are included. One could either lecture from Section 12.12 (directly) or 12.11 (with a preliminary lecture from Sections 10.10–10.11, or leave these for assigned reading.) Chapter 16 (trimmed means and/or simple linear rank statistics) Both possibilities listed here require Section 12.10 as well as the quantile inequality of (7.11.3). Asymptotic normality of linear rank statistics and a ﬁnite sampling CLT are derived in Section 16.2, and the bootstrap is presented in Section 16.3. The general CLT (Theorem 16.1.1) and asymptotic normality of trimmed means (Theorem 16.1.2, but only present the β = 0 case) are derived in Section 16.1; this will also require stating/proving the equivalence of (16.1.3) and (7.6.4), which is shown in Theorem 7.1.1. Chapter 18 (martingales; 10 lectures) I cover most of the ﬁrst seven sections. {I always assign Exercises 18.1.4 (a counting process martingale), 18.3.2 (a proof for continuous time mgs), 18.3.7, and 18.3.9 (on Lr -convergence).} Appendix A (maximum likelihood estimation) I see this as being of considerable interest in conjunction with statistical pursuits, rather than as a direct part of a course on probability theory.

Deﬁnition of Symbols ∼ = means “is distributed as” ≡ means “is deﬁned to be” a = b ⊕ c means that |a − b| ≤ c Un =a Vn means “asymptotically equal” in the sense that Un − Vn →p 0 X∼ = (µ, σ2 ) means that X has mean µ and variance σ2 X∼ = F (µ, σ2 ) means that X has df F with mean µ and variance σ2 ¯ n is the “sample mean” and X ¨ n is the “sample median” X (Ω, A, µ) and (Ω, A, P ) denote a measure space and a probability space σ[C] denotes the σ-ﬁeld generated by the class of sets C ¯ for the Borel sets B and B¯ ≡ σ[B, {+∞}, {−∞}] F(X) denotes X −1 (B), ξ will always refer to a Uniform(0, 1) rv means “nondecreasing” and ↑ means “strictly increasing” 1A (·) denotes the indicator function of the set A “df” refers to a distribution function F (·) “qf” refers to a quantile function K(·) ≡ F −1 (·) ˜ The “tilde” symbol denotes Winsorization ˇ The “h´ aˇcek” symbol denotes Truncation λ(·) and λn (·) will refer to Lebesgue measure on the line R and on Rn See page 119 for “dom(a, a )” Brownian motion S, Brownian bridge U, and the Poisson process N The empirical df Fn and the empirical df Gn of Uniform(0, 1) rvs → is associated with convergence in the LIL (see page 235) “mg” refers to a martingale “smg” refers to a submartingale > = means “≥” for a submartingale and “=” for a martingale > The symbol “=” is paired with “s-mg” in this context. Prominence Important equations are labeled with numbers to give them prominence. Thus, equations within a proof that are also important outside the context of that proof are numbered. Though the typical equation in a proof is unworthy of a number, it may be labeled with a letter to help with the “bookkeeping.” Likewise, digressions or examples in the main body of the text may contain equations labeled with letters that decrease the prominence given to them. Integral signs and summation signs in important equations (or suﬃciently complicated equations) are large, while those in less important equations are small. It is a matter of assigned prominence. The most important theorems, deﬁnitions, and examples have been given titles in boldface type to assign prominence to them. The titles of somewhat less important results are not in boldface type. Routine references to theorem 10.4.1 or deﬁnition 7.3.1 do not contain capitalized initial letters. The author very speciﬁcally wishes to downgrade the prominence given to this routine use of these words. Starting new sections on new pages allowed me to carefully control the ﬁeld of vision as the most important results were presented.

Chapter 1

Measures 1

Basic Properties of Measures

Motivation 1.1 (The Lebesgue integral) The Riemann integral of a continuous function f , we will restrict attention to f (x) ≥ 0 for a ≤ x ≤ b for convenience, is formed by subdividing the domain of f , forming approximating sums, and passing b to the limit. Thus the mth Riemann sum for a f (x) dx is deﬁned as RSm ≡

(1)

m

f (x∗mi ) [xmi − xm,i−1 ],

i=1

where a ≡ xm0 < xm1 < · · · < xmm ≡ b (with xm,i−1 ≤ x∗mi ≤ xmi for all i) satisfy meshm ≡ max[xmi −xm,i−1 ] → 0. Note that xmi −xm,i−1 is the measure (or length) of the interval [xm,i−1 , xmi ], while f (x∗mi ) approximates the values of f (x) for all xm,i−1 ≤ x ≤ xmi (at least it does if f is continuous on [a, b]). Within the class C+ of all nonnegative continuous functions, this deﬁnition works reasonably well. But b b it has one major shortcoming. The conclusion a fn (x) dx → a f (x) dx is one we often wish to make. However, even when all fn are in C + and f (x) ≡ lim fn (x) b actually exists, it need not be the case that f is in C+ ; and thus a f (x) dx may not even be well-deﬁned. A diﬀerent approach is needed. (Note ﬁgure 1.1.) The Lebesgue integral of a nonnegative function is formed by subdividing the b range. Thus the mth Lebesgue sum for a f (x) dx is deﬁned as LSm ≡

(2)

m m2

i=1

k−1 × measure 2m

k−1 k x : m ≤ f (x) < m 2 2

b

,

and a f (x) dx is deﬁned to be the limit of the LSm sums as m → ∞. For what class M of functions f can this approach succeed? The members f of the class M will need to be such that the measure (or length) of all sets of the form k−1 k (3) x : m ≤ f (x) < m 2 2 1

2

CHAPTER 1. MEASURES

can be speciﬁed. This approach leads to the concept of a σ-ﬁeld A of subsets of [a, b] that are measurable (that is, we must be able to assign to these intervals a number called their “length”), and this leads to the concept of the class M of measurable functions. This class M of measurable functions will be seen to be closed under passage to the limit and all the other operations that we are accustomed to perb b forming on functions. Moreover, the desirable property a fn (x) dx → a f (x) dx for functions fn “converging” to f will be broadly true. 2

xm0

xm1

…

xm,i-1

xmi

…

Riemann sums The domain of f(·) is equally divided.

1 2m

Lebesgue sums The range of f(·) is equally divided. Figure 1.1 Riemann sums and Lebesgue sums.

xmm

1.

BASIC PROPERTIES OF MEASURES

3

Deﬁnition 1.1 (Set theory) Consider a nonvoid class A of subsets A of a nonvoid set Ω. (For us, Ω will be the sample space of an experiment.) (a) Let Ac denote the complement of A, let A ∪ B denote the union of A and B, let A ∩ B and AB both denote the intersection, let A \ B ≡ ABc denote the set diﬀerence, let A B ≡ (Ac B ∪ ABc ) denote the symmetric diﬀerence, and let ∅ denote the empty set. The class of all subsets of Ω will be denoted by 2Ω . Sets A and B are called disjoint if AB = ∅, and sequences of sets An or classes of sets At ∞ are called disjoint if all pairs are disjoint. Writing A + B or 1 An will also denote a union, but will imply the disjointness of the sets in the union. As usual, A ⊂ B denotes that A is a subset of B. We call a sequence An increasing (and we will nearly always denote this fact by writing An ) when An ⊂ An+1 for all n ≥ 1. We call the sequence decreasing (denoted by An ) when An ⊃ An+1 for all n ≥ 1. We call the sequence monotone if it is either increasing or decreasing. Let ω denote a generic element of Ω. We will use 1A (·) to denote the indicator function of A, which equals 1 or 0 at ω according as ω ∈ A or ω ∈ A. (b) A will be called a ﬁeld if it is closed under complements and unions. (That is, A and B in A requires that Ac and A ∪ B be in A.) [Note that both Ω and ∅ are necessarily in A, as A was assumed to be nonvoid, with Ω = A ∪ Ac and ∅ = Ωc .] (c) A will be called a σ-ﬁeld if it is closed under complements and countable unions. (That is, A, A1 , A2 , . . . in A requires that Ac and ∪∞ 1 An be in A.) (d) A will be called a monotone class provided it contains ∪∞ 1 An for all increasing sequences An in A and contains ∩∞ 1 An for all decreasing sequences An in A. (e) (Ω, A) will be called a measurable space provided A is a σ-ﬁeld of subsets of Ω. (f) A will be called a π-system provided AB is in A for all A and B in A; and A will be called a π ¯ -system when Ω in A is also guaranteed. If A is a ﬁeld (or a σ-ﬁeld), then it is closed under intersections (under countable ∞ c c intersections); since AB = (Ac ∪ B c )c (since ∩∞ 1 An = (∪1 An ) ). Likewise, we could have used “intersection” instead of “union” in our deﬁnitions by making use ∞ c c of A ∪ B = (Ac ∩ B c )c and ∪∞ 1 An = (∩1 An ) . Proposition 1.1 (Closure under intersections) (a) Arbitrary intersections of ﬁelds, σ-ﬁelds, or monotone classes are ﬁelds, σ-ﬁelds, or monotone classes, respectively. [For example, ∩{Fα : Fα is a ﬁeld under consideration} is a ﬁeld.] (b) There exists a minimal ﬁeld, σ-ﬁeld, or monotone class generated by (or, containing) any speciﬁed class C of subsets of Ω. We call C the generators. For example, (4) σ[C] ≡ {Fα : Fα is a σ-ﬁeld of subsets of Ω for which C ⊂ Fα } is the minimal σ-ﬁeld generated by C (that is, containing C) . (c) A collection A of subsets of Ω is a σ-ﬁeld if and only if it is both a ﬁeld and a monotone class. ∞ n ∞ Proof. (c) (⇐) ∪∞ 1 An = ∪1 (∪1 Ak )) ≡ ∪1 Bn ∈ A since the Bn are in A and are . Everything else is even more trivial. 2

4

CHAPTER 1. MEASURES

Exercise 1.1 (Generators) Let C1 and C2 denote two collections of subsets of the set Ω. If C2 ⊂ σ[C1 ] and C1 ⊂ σ[C2 ], then σ[C1 ] = σ[C2 ]. Prove this fact. Deﬁnition 1.2 (Measures) Consider a set function µ : A → [0, ∞] (that is, µ(A) ≥ 0 for each A ∈ A) having µ(∅) = 0. (a) If A is a σ-ﬁeld and µ is countably additive (abbreviated c.a.) in that (5)

∞ ∞

An = µ(An ) µ n=1

for all disjoint sequences An in A,

n=1

then µ is called a measure (or, equivalently, a countably additive measure) on (Ω, A). The triple (Ω, A, µ) is then called a measure space. We call µ ﬁnite if µ(Ω) ∞. < ∞ We call µ σ-ﬁnite if there exists a measurable decomposition of Ω as Ω = 1 Ωn with Ωn ∈ A and µ(Ωn ) < ∞ for all n. [If A is not a σ-ﬁeld, we will still call µ a measure on (Ω, A), provided that (5) holds for all sequences An for which ∪∞ 1 An is in A. We will not, however, use the term “measure space” to describe such a triple. We will consider below measures on ﬁelds, on certain π ¯ -systems, and on some other collections of sets. A useful property of a collection of sets is that along with any 1 , . . . , Ak it also include all sets sets A of the type Bk ≡ Ak Ack−1 · · · Ac2 Ac1 ; then n1 An = n1 Bk is also included.] (b) We call µ a ﬁnitely additive measure (abbreviated f.a.) on (Ω, A) if n n (6) µ( 1 An ) = 1 µ(An ) for all disjoint sequences An in A for which n1 An is also in A. Deﬁnition 1.3 (Outer measures) Consider a set function µ∗ : 2Ω → [0, ∞]. (a) Suppose that µ∗ satisﬁes the following three properties. Null: µ∗ (∅) = 0. Monotone: µ∗ (A) ≤ µ∗ (B) forall A ⊂ B. ∞ ∞ Countable subadditivity: µ∗ ( 1 An ) ≤ 1 µ∗ (An ) for all An . Then µ∗ is called an outer measure. (b) An arbitrary subset A of Ω is called µ∗ -measurable if (7)

µ∗ (T ) = µ∗ (T A) + µ(T Ac )

for all subsets T ⊂ Ω.

Sets T used in this capacity are called test sets. (c) We let A∗ denote the class of all µ∗ -measurable sets, that is, (8)

A∗ ≡ {A ∈ 2Ω : A is µ∗ -measurable}.

[Note that A ∈ A∗ if and only if µ∗ (T ) ≥ µ∗ (T A) + µ∗ (T Ac ) for all T ⊂ Ω, since the other inequality is trivial by the subadditivity of µ∗ .] Motivation 1.2 (Measure) In this paragraph we will consider only one possible measure µ, namely the Lebesgue-measure generalization of length. Let CI denote the set of all intervals of the types (a, b], (−∞, b], and (a, +∞) on the real line R, and for each of these intervals I we assign a measure value µ(I) equal to its length, thus ∞, b − a, ∞ in the three special cases. All is well until we manipulate the sets

1.

BASIC PROPERTIES OF MEASURES

5

in CI , as even the union of two elements in CI need not be in CI . Thus, CI is not a very rich collection of sets. A natural extension is to let CF denote the collection of all ﬁnite disjoint unions of sets in CI , where the measure µ(A) we assign to each such set A is just the sum of the measures (lengths) of all its disjoint pieces. Now CF is a ﬁeld, and is thus closed under the elementary operations of union, intersection, and complementation. Much can be done using only CF and letting “measure” be the “exact length”. But CF is not closed under passage to the limit, and it is thus insuﬃcient for many of our needs. For this reason the concept of the smallest σ-ﬁeld containing CF , labeled B ≡ σ[CF ], is introduced. We call B the Borel sets. But let us work backwards. Let us assign an outer measure value µ∗ (A) to every subset A in the class 2R of all subsets of the real line R. In particular, to any ∞subset A we assign the value µ∗ (A) that is the inﬁmum of all possible numbers n=1 µ(An ), in which each An is in the ﬁeld CF (so that we know its measure) in which the An ’s form and ∞ a cover of A (in that A ⊂ ∪∞ µ(A n ) is a natural upper 1 An ). Thus each number 1 bound to the measure (or generalized length) of the set A, and we will specify the inﬁmum of such upper bounds to be the outer measure of A. Thus to each subset A of the real line we assign a value µ∗ (A) of generalized length. This value seems “reasonable”, but does it “perform correctly”? Let us say that a particular set A is µ∗ -measurable (that is, it “performs correctly”) if µ∗ (T ) = µ∗ (T A) + µ∗ (T Ac ) for all subsets of the real line R, that is, if the A versus Ac division of the line divides every subset T of the line into two pieces in a fashion that is µ∗ -additive. This is undoubtedly a combination of reasonableness and ﬁne technicality that took some time to evolve in the mind of its creator, Carath´eodory, while he searched for a condition that “worked”. In what sense does it “work”? The collection A∗ of all µ∗ -measurable sets turns out to be a σ-ﬁeld. Thus the collection A∗ is closed under all operations that we are likely to perform; and it is big enough, in that it is a σ-ﬁeld that contains CF . Thus we will work with the restriction µ∗ |A∗ of µ∗ to the sets of A∗ (here, the vertical line means “restricted to”). This is enough to meet our needs. There are many measures other than length. For an and right-continuous function F on the real line (called a generalized df) we deﬁne the Stieltjes measure of an arbitrary interval (a, b] (with −∞ ≤ a < b ≤ ∞) in CI by µF ((a, b]) = F (b) − F (a), and we extend it to sets in CF by adding up the measure of the pieces. Reapplying the previous paragraph, we can extend µF to the µ∗F -measurable sets. It is the important Carath´eodory extension theorem that will establish that all Stieltjes measures (including the case of ordinary length, where F (x) = x as considered in the ﬁrst paragraph) can be extended from CF to the Borel sets B. That is, all Borel sets are µ∗ -measurable for every Stieltjes measure. One further extension is possible, in that every measure can be “completed” (see the end of section 1.2). We note here only that when the Stieltjes measure µF associated with the generalized df F is “completed”, its domain of deﬁnition is extended from the Borel sets B (which all Stieltjes measures have in common) to a larger collection BˆµF that depends on the particular F . It is left to section 1.2 to simply state that this is as far as we can go. That is, except in trivial special cases, we ﬁnd that BˆµF is a proper subset of 2R . (That is, it is typically impossible to try to deﬁne the measure of all subsets of Ω in a suitable fashion.) 2

6

CHAPTER 1. MEASURES

Example 1.1 (Some examples of measures, informally) (a) Lebesgue measure: Let λ(A) denote the length of A. (b) Counting measure: Let #(A) denote the number of “points” in A (or the cardinality of A). (c) Unit point mass: Let δω0 (A) ≡ 1{ω0 } (A), assigning measure 1 or 0 as ω0 ∈ A or not. 2 Example 1.2 (Borel sets) (a) Let Ω = R and let C consist of all ﬁnite disjoint unions of intervals of the types (a, b], (−∞, b], and (a, +∞). Clearly, C is a ﬁeld. Then B ≡ σ[C] will be called the Borel sets (or the Borel subsets of R). Let µ(A) be deﬁned to be the sum of the lengths of the intervals composing A, for each A ∈ C. Then µ is a c.a. measure on the ﬁeld C, as will be seen in the proof of theorem 1.3.1 below. (b) If (Ω, d) is a metric space and U ≡ {all d-open subsets of Ω}, then B ≡ σ[U] will be called the Borel sets or the Borel σ-ﬁeld . (c) If (Ω, d) is (R, | · |) for absolute value | · |, then σ[C] = σ[U] even though C = U. [This claim is true, since C ⊂ σ[U] and U ⊂ σ[C] are clear. Then, just make a trivial appeal to exercise 1.1.] ¯ ≡ [−∞, = ∞] denote the extended real line and let B¯ ≡ σ[B, {−∞}, {+∞}]. (d) Let R 2 Proposition 1.2 (Monotone property of measures) measure space. Let (A1 , A2 , . . .) be in A. (a) If An ⊂ An+1 for all n, then ∞ (9) µ( 1 An ) = lim µ(An ).

Let (Ω, A, µ) denote a

n→∞

(b) If µ(An0 ) < ∞ for some n0 , and An ⊃ An+1 for all n, then µ( ∞ (10) n=1 An ) = lim µ(An ) . n→∞

[Letting Ω denote the real line R, letting An = [n, ∞), and letting µ denote either Lebesgue measure or counting measure, we see the need for some requirement.] (c) (Countable subadditivity) Whenever (A1 , A2 , . . .) and ∪∞ 1 An are all in A, then ∞ ∞ µ( 1 Ak ) ≤ 1 µ(Ak ) ; and this also holds true for a measure on a ﬁeld or on a π ¯ -system. Proof.

(p)

(a) Now,

∞ µ( ∪∞ with A0 ≡ ∅ 1 An ) = µ( 1 (An \ An−1 )) ∞ = 1 µ(An \ An−1 ) by c.a. n = lim k=1 µ(Ak \ Ak−1 ) n n = lim µ( k=1 (Ak \ Ak−1 )) by f.a. n

(q)

= lim µ(An ). n

1.

BASIC PROPERTIES OF MEASURES

7

(b) Without loss of generality, redeﬁne A1 = A2 = · · · = An0 . Let Bn ≡ A1 \An , so that Bn . Thus, on the one hand, lim µ(Bn ) = µ( ∪∞ 1 Bn ) n

by (a)

c = µ( ∪∞ 1 (A1 ∩ An )) c = µ(A1 ∩ ∪∞ 1 An ) c = µ(A1 ∩ (∩∞ 1 An ) )

= µ(A1 ) − µ( ∩∞ 1 An ).

(r)

On the other hand, lim µ(Bn ) = lim{ µ(A1 ) − µ(An )} n

(s)

n

by f.a.

= µ(A1 ) − lim µ(An ). n

Equate (r) and (s); since µ(A1 ) < ∞, we can cancel it to obtain the equality µ(∩∞ 1 An ) = limn µ(An ). (c) Let B1 ≡ A1 , B2 ≡ A2 Ac1 , . . . , Bk ≡Ak Ack−1 · · · Ac1 . Then these newly n deﬁned sets Bk are disjoint, and ∪nk=1 Ak = k=1 Bk . Hence [a technique worth remembering] n n n µ( k=1 Ak ) = µ( k=1 Bk ) = k=1 µ(Bk ) (11) n n where k=1 Ak = k=1 Bk for Bk ≡ Ak Ack−1 · · · Ac1 ≤ (t)

≤

n

1 µ(Ak )

∞ 1

µ(Ak )

by monotonicity.

Let n → ∞, and use part (a) to get the result.

2

Deﬁnition 1.4 (Continuity of measures) A measure µ is continuous from below (above) if µ(lim An ) = lim µ(An ) for all An (for all An , with at least one µ(An ) ﬁnite). We call µ continuous in case it is continuous both from below and from above. The next result is often used in conjunction with the Carath´eodory extension theorem of the next section. View it as a converse to the previous proposition. Proposition 1.3 (Continuity of measures) If a ﬁnitely additive measure µ on either a ﬁeld or σ-ﬁeld is either continuous from below or has µ(Ω) < ∞ and is continuous from above at ∅, then it is a countably additive measure.

8 Proof.

(a) (b) (c)

CHAPTER 1. MEASURES Suppose ﬁrst that µ is continuous from below. Then ∞ n µ( 1 An ) = µ( lim 1 Ak ) n by continuity from below = lim µ( 1 Ak ) n = lim 1 µ(Ak ) by f.a. (where we used only that A is a ﬁeld) ∞ = 1 µ(An ),

giving the required countable additivity. Suppose next that µ is ﬁnite and is also continuous from above at ∅. Then f.a. (even if A is only a ﬁeld) gives n ∞ n ∞ µ( ∞ 1 An ) = µ( 1 Ak ) + µ( n+1 Ak ) = 1 µ(Ak ) + µ( n+1 Ak ) ∞ (d) → 1 µ(Ak ) + 0, ∞ ∞ where µ( n+1 Ak ) → µ(∅) = 0 by continuity from above at ∅, since n+1 Ak ∅ and µ is ﬁnite. 2 Deﬁnition 1.5 (liminf and limsup of sets) Let ∞ ∞ lim An ≡ n=1 k=n Ak = {ω : ω is in all but ﬁnitely many An ’s} (12) ≡ {ω : ω ∈ An a.b.f.}, where we use a.b.f. to abbreviate in all but ﬁnitely many cases. Let ∞ limAn ≡ ∞ n=1 k=n Ak = {ω : ω is in an inﬁnite number of An ’s} (13) ≡ {ω : ω ∈ An i.o.}, where we use i.o. to abbreviate inﬁnitely often. [It is important to learn to read these two mathematical equations in a way that makes it clear that the verbal description is correct.] Note that we always have lim An ⊂ lim An . Deﬁne (14)

lim An ≡ lim An

whenever lim An = lim An .

We also let lim inf An ≡ lim An and lim sup An ≡ lim An , giving us alternative notations. Deﬁnition 1.6 (lim inf and lim sup of numbers) Recall that for real number sequences an one deﬁnes lim an ≡ lim inf an and lim an ≡ lim sup an by lim inf n→∞ an ≡ limn→∞ (inf k≥n ak ) = supn≥1 (inf k≥n ak ) (15)

and

lim supn→∞ an ≡ limn→∞ supk≥n ak = inf n≥1 supk≥n ak ,

and these yield the smallest limit point and the largest limit point, respectively, of the sequence an .

1.

BASIC PROPERTIES OF MEASURES

9

Deﬁnition 1.7 (“Little oh”, “big oh”, and “at most” ⊕) an ≡ o(rn )

if

an /rn → 0 ,

an ≡ O(rn )

if

lim |an /rn | ≤ (some M ) < ∞.

if

|an − bn | ≤ cn .

We write:

(16) We write (17)

an = bn ⊕ cn

This last notation allows us to string inequalities together linearly, instead of having to start a new inequality on a new line. (I use it often.) Proposition 1.4 Clearly, lim An equals ∪∞ 1 An when An is an sequence, and lim An equals ∩∞ 1 An when An is a sequence. Exercise 1.2 We always have µ(lim inf An ) ≤ lim inf µ(An ), while the inequality lim sup µ(An ) ≤ µ(lim sup An ) holds if µ(Ω) < ∞. Exercise 1.3 (π-systems and λ-systems) A class D of subsets is called a λ-system if it contains the space Ω and all proper diﬀerences (A \ B, for A ⊂ B with both A, B ∈ D) and if it is closed under monotone increasing limits. [Recall that a class is called a π-system if it is closed under ﬁnite intersections, while π ¯ -systems are also required to contain Ω.] (a) The minimal λ-system generated by the class D is denoted by λ[D]. Show that λ[D] is equal to the intersection of all λ-systems containing D. (b) A collection A of subsets of Ω is a σ-ﬁeld if and only if it is both a π-system and a λ-system. (c) Let C be a π-system and let D be a λ-system. Then C ⊂ D implies that σ[C] ⊂ D. Proposition 1.5 (Dynkin’s π-λ theorem) Let µ and µ be two measures on the measurable space (Ω, A). Let C be a π ¯ -system, where C ⊂ A. Then (18) Proof. (19)

¯ -system C µ = µ on the π

implies

µ = µ on σ[C].

We ﬁrst show that σ[C] = λ[C]

when C is a π-system.

Let D ≡ λ[C]. By the easy exercise 1.3(a)(b) it suﬃces to show that D is a π-system (that is, that A, B ∈ D implies A ∩ B ∈ D). We ﬁrst go just halfway; let (a)

E ≡ {A ∈ D : AC ∈ D for all C ∈ C}.

Then C ⊂ E. Also, for A, B ∈ E with B ⊂ A and for C ∈ C we have (since both AC and BC are in D) that (A \ B)C = (AC \ BC) ∈ D, so that A \ B ∈ E. Thus E = D, since D was the smallest such class. We have thus learned of D that (b)

AC ∈ D for all C ∈ C, for each A ∈ D.

10

CHAPTER 1. MEASURES

To go the rest of the way, we deﬁne (c)

F ≡ {D ∈ D : AD ∈ D for all A ∈ D}.

Then C ⊂ F, by (b). Also, for A, B ∈ F with B ⊂ A and for D ∈ D we have (since both AD and BD are in D) that (A \ B)D = (AD \ BD) ∈ D, so that A \ B ∈ F. Thus F = D, since D was the smallest such class. We have thus learned of D that (d)

AD ∈ D for all A ∈ D, for each D ∈ D.

That is, D is closed under intersections; and thus D is a π-system. We will now demonstrate that G ≡ {A ∈ A : µ(A) = µ (A)} is a λ-system on Ω. First, Ω ∈ G, since Ω is in the π ¯ -system C. Second, when A ⊂ B are both in G we have the equality (e)

µ(B \ A) = µ(B) − µ(A) = µ (B) − µ (A) = µ (B \ A),

giving B \ A ∈ G. Finally, let An A with all An ’s in G. Then proposition 1.2(i) yields the result (f)

µ(A) = lim µ(An ) = lim µ (An ) = µ (A),

so that A ∈ G. Thus G is a λ-system. Thus the set G on which µ = µ is a λ-system that contains the π ¯ -system C. Applying (19) shows that σ[C] ⊂ G. 2 The previous proposition is very useful in extending independence from small classes of sets to large ones. The next proposition is used in proving the Carath´eodory extension theorem, Fubini’s theorem, and the existence of a regular conditional probability distribution. Proposition 1.6 (Minimal monotone class; Halmos) The minimal monotone class M ≡ m[C] containing the ﬁeld C and the minimal σ-ﬁeld σ[C] generated by the same ﬁeld C satisfy (20)

m[C] = σ[C]

when C is a ﬁeld.

Proof. Since σ-ﬁelds are monotone classes, we have that σ[C] ⊃ M. If we now show that M is a ﬁeld, then proposition 1.1.1(c) will imply that σ[C] ⊂ M. To show that M is a ﬁeld, it suﬃces to show that (a)

A, B in M implies AB, Ac B, AB c are in M.

Suppose that (a) has been established. We will now show that (a) implies that M is a ﬁeld. Complements: Let A ∈ M, and note that Ω ∈ M, since C ⊂ M. Then A, Ω ∈ M implies that Ac = Ac Ω ∈ M by (a).

1.

BASIC PROPERTIES OF MEASURES

11

Unions: Let A, B ∈ M. Then A ∪ B = (Ac ∩ B c )c ∈ M. Thus M is indeed a ﬁeld, provided that (a) is true. It thus suﬃces to prove (a). For each A ∈ M, let MA ≡ {B ∈ M : AB, Ac B, AB c ∈ M}. Note that it suﬃces to prove that (b)

MA = M

for each ﬁxed A ∈ M.

We ﬁrst show that (c)

MA is a monotone class.

Let Bn be monotone in MA , with limit set B. Since Bn is monotone in MA , it is also monotone in M, and thus B ≡ limn Bn ∈ M. Since Bn ∈ MA , we have ABn ∈ M, and since ABn is monotone in M, we have AB = limn ABn ∈ M. In like fashion, Ac B and AB c are in M. Therefore, B ∈ MA , by deﬁnition of MA . That is, (c) holds. We next show that (d)

MA = M

for each ﬁxed A ∈ C.

Let A ∈ C and let C ∈ C. Then A ∈ MC , since C is a ﬁeld. But A ∈ MC if and only if C ∈ MA , by the symmetry of the deﬁnition of MA . Thus C ⊂ MA . That is, C ⊂ MA ⊂ M, and MA is a monotone class by (c). But M is the minimal monotone class containing C, by the deﬁnition of M. Thus (d) holds. But in fact, we shall now strengthen (d) to (e)

MB = M

for each ﬁxed B ∈ M.

The conditions for membership in M imposed on pairs A, B are symmetric. Thus for A ∈ C, the statement established above in (d) that B ∈ M(= MA ) is true if and only if A ∈ MB . Thus C ⊂ MB , where MB is a monotone class. Thus MB = M, since (as was earlier noted) M is the smallest such monotone class. Thus (e) (and hence (a)) is established. 2

12

2

CHAPTER 1. MEASURES

Construction and Extension of Measures

Deﬁnition 2.1 (Outer extension) Let Ω be arbitrary. Let µ be a measure on a ﬁeld C of subsets Ω. For each A ∈ 2Ω deﬁne ∞ ∞ ∗ (1) µ(An ) : A ⊂ An with all An ∈ C . µ (A) ≡ inf n=1

n=1

∗

Now, µ is called the outer extension of µ. The sequences A1 , A2 , . . . are called Carath´eodory coverings. [There is always at least one covering, since Ω ∈ C.] Theorem 2.1 (Carath´ eodory extension theorem) A measure µ on a ﬁeld C can be extended to a measure on the σ-ﬁeld σ[C] generated by C, by deﬁning (2)

µ(A) ≡ µ∗ (A)

for each A in A ≡ σ[C].

If µ is σ-ﬁnite on C, then the extension is unique on A and is also σ-ﬁnite. Proof. The proof proceeds by a series of claims. Claim 1: µ∗ is an outer measure on (Ω, 2Ω ). Null: Now, µ∗ (∅) = 0, since ∅, ∅, . . . is a covering of ∅. Monotone: Let A ⊂ B. Then every covering of B is also a covering of A. Thus µ∗ (A) ≤ µ∗ (B). Countably subadditive: Let all An ⊂ Ω be arbitrary. Let > 0. For each An there is a covering {Ank : k ≥ 1} such that ∞ ∗ n (3) since µ∗ (An ) is an inﬁmum. 1 µ(Ank ) ≤ µ (An ) + /2 , [The choice of a convergent series (like /2n ) that adds to is an important technique for the reader to learn.] Now ∪n An ⊂ ∪n ∪k Ank ). Thus since µ∗ is monotone µ∗ ( n An ) ≤ µ∗ ( n k Ank ) ≤ n k µ (Ank ) since the Ank ’s form a covering of the set n k Ank ≤ n [ µ∗ (An ) + /2n ] = n µ∗ (An ) + . But > 0 was arbitrary, and thus µ∗ ( n An ) ≤ n µ∗ (An ). Claim 2: µ∗ |C = µ (that is, µ∗ (C) = µ(C) for all C ∈ C) and C ⊂ A∗ . Let C ∈ C. Then µ∗ (C) ≤ µ(C), since C, ∅, ∅, . . . is a covering of C. For the other direction, we let A1 , A2 , . . . be any covering of C. Since µ is c.a. on C, and since ∪∞ 1 (An ∩ C) = C ∈ C, we have from proposition 1.1.2(c) that ∞ ∞ µ(C) = µ( ∞ 1 (An ∩ C)) = 1 µ(An ∩ C) ≤ 1 µ(An ), and thus µ(C) ≤ µ∗ (C). Thus µ(C) = µ∗ (C). We next show that any C ∈ C is also in A∗ . Let C ∈ C. Let > 0, and let T be given. There exists a covering {An }∞ 1 ⊂ C of T such that

2. (a)

CONSTRUCTION AND EXTENSION OF MEASURES

13

∞ µ∗ (T ) + ≥ 1 µ(An ) since µ∗ (T ) is an inﬁmum ∞ c = ∞ 1 µ(CAn ) + 1 µ(C An ) since µ is c.a. on C with C and An in C

(b)

≥ µ∗ (CT ) + µ∗ (C c T )

since CAn covers CT and C c An covers C c T.

But > 0 is arbitrary. Thus C ∈ A∗ . Thus C ⊂ A∗ . Claim 3: The class A∗ of µ∗ -measurable subsets of Ω is a ﬁeld that contains C. Now, A ∈ A∗ implies that Ac ∈ A∗ : The deﬁnition of µ∗ -measurable is symmetric in A and Ac . And A, B ∈ A∗ implies that AB ∈ A∗ : For any test set T ⊂ Ω we have the required inequality µ∗ (T ) = µ∗ (T A) + µ∗ (T Ac )

since A ∈ A

= µ∗ (T AB) + µ∗ (T AB c ) + µ∗ (T Ac B) + µ∗ (T Ac B c ) since B ∈ A∗ with test set T A and with test set T Ac ≥ µ∗ (T AB) + µ∗ (T AB c + T Ac B + T Ac B c ) since µ∗ is countably subadditive (c)

= µ∗ (T AB) + µ∗ (T (AB)c ).

Thus A∗ is a ﬁeld. Claim 4: µ∗ is a f.a. measure on A∗ . Let A, B ∈ A∗ be disjoint. Finite additivity follows from µ∗ (A + B) = µ∗ ((A + B)A) + µ∗ ((A + B)Ac ) since A ∈ A∗ with test set A + B (d)

= µ∗ (A) + µ∗ (B).

Trivially, µ∗ (A) ≥ 0 for all sets A. And µ∗ (∅) = 0, since φ, φ, . . . is a covering of φ. Claim 5: A∗ is a σ-ﬁeld, ∞and it contains σ[C]. n We will show that A ≡ 1 An ∈ A∗ whenever all An ∈ A∗ . Now, 1 Ak ∈ A∗ , since A∗ is a ﬁeld. We thus have n n n µ∗ (T ) = µ∗ (T ( 1 An )) + µ∗ (T ( 1 An )c ) since 1 An ∈ A∗ n ≥ µ∗ (T ( 1 An )) + µ∗ (T Ac ) since µ∗ is monotone and ( n1 An )c ⊃ Ac n (e) ≥ 1 µ∗ (T Ak ) + µ∗ (T Ac ) by claims 3 and 4. Letting n → ∞ gives ∞ (f) µ∗ (T ) ≥ 1 µ∗ (T An ) + µ∗ (T Ac ) (g)

≥ µ∗ (T A) + µ∗ (T Ac )

Thus A ∈ A∗ .

since µ∗ is countably subadditive.

14

CHAPTER 1. MEASURES

Claim 6: µ∗ is c.a. on A∗ . ∗ Replace T by A in (f) to get µ∗ (A) ≥ ∞ 1 µ (An ), and then countable subadditivity gives the reverse inequality. It remains only to establish the uniqueness, for which we will appeal to the minimal monotone class result of proposition 1.1.6. This proposition is interesting in its own right. Claim 7: Uniqueness holds when µ is a ﬁnite measure. Let µ1 and µ2 denote any two extensions of µ. Let M ≡ {A ∈ σ[C] : µ1 (A) = µ2 (A)} denote the class where they are equal. We will ﬁrst show that (h)

M is a monotone class.

Let An be monotone in M. Then (i)

µ1 ( lim An ) = lim µ1 (An )

by propositions 1.1.4 and 1.1.2

(j)

= lim µ2 (An )

since µ1 = µ2 on M

(k)

= µ2 ( lim An )

by propositions 1.1.4 and 1.1.2.

Thus (h) holds. Since C ⊂ M, the minimal monotone class result of proposition 1.1.6 implies that σ[C] ⊂ M. Thus µ1 = µ2 on σ[C] (and possibly on even more sets than this). Thus the claimed uniqueness holds. [Appeal to proposition 1.1.6 could be replaced by appeal to Dynkin’s π-λ theorem of proposition 1.1.5.] Claim 8: Uniqueness holds when µ is a σ-ﬁnite measure (label the sets of the measurable partition as Ωn ). We must again demonstrate the uniqueness. Fix n. We will consider µ, µ1 , µ2 on C, on σ[C] ∩ Ωn , and on σ[C ∩ Ωn ]. We ﬁrst show that (l)

σ[C] ∩ Ωn ≡ {A ∩ Ωn : A ∈ σ[C]} = σ[C ∩ Ωn ] ≡ σ[{C ∩ Ωn : C ∈ C}].

Now, C is a subset of the σ-ﬁeld D ≡ {A ∈ σ[C] : A ∩ Ωn ∈ σ[C ∩ Ωn ]}, and hence σ[C] ⊂ D, so that σ[C] ∩ Ωn ⊂ σ[C ∩ Ωn ]. The other direction is even more obvious. Now, µ1 |σ[C ∩ Ωn ] = µ2 |σ[C ∩ Ωn ], since µ|σ[C ∩ Ωn ] is a ﬁnite measure on the ﬁeld C ∩ Ωn . Thus for A ∈ σ[C] we have µ1 (A) = ∞ by c.a. 1 µ1 (AΩn ) ∞ = 1 µ2 (AΩn ) since µ1 = µ2 on σ[C ∩ Ωn ] = σ[C] ∩ Ωn , by claim 7 (m)

= µ2 (A)

completing the proof.

by c.a., 2

Question We extended our measure µ from the ﬁeld C to a collection A∗ that is at least as big as the σ-ﬁeld σ[C]. Have we actually gone beyond σ[C]? Can we go further?

2.

CONSTRUCTION AND EXTENSION OF MEASURES

15

Deﬁnition 2.2 (Complete measures) Let (Ω, A, µ) denote a measure space. If µ(A) = 0, then A is called a null set. We call (Ω, A, µ) complete if whenever we have B ⊂ (some A) ∈ A with µ(A) = 0, we necessarily also have B ∈ A. [That is, all subsets of sets of measure 0 are required to be measurable.] Exercise 2.1 (Completion) (4)

Let (Ω, A, µ) denote a measure space. Show that

Aˆµ ≡ {A : A1 ⊂ A ⊂ A2 with A1 , A2 ∈ A and µ(A2 \ A1 ) = 0}

(5)

= {A ∪ N : A ∈ A, and N ⊂ (some B) ∈ A having µ(B) = 0}

(6)

= {A N : A ∈ A, and N ⊂ ( some B) ∈ A having µ(B) = 0},

and that Aˆµ is a σ-ﬁeld. Deﬁne µ ˆ on Aˆµ by (7)

µ ˆ(A ∪ N ) = µ(A)

for all A ∈ A and for all N ⊂ (some B) ∈ A having µ(B) = 0. Show that (Ω, Aˆµ , µ ˆ) is a complete measure space for which µ ˆ |A = µ. [Note: A proof must include a demonstration that deﬁnition (7) leads to a well-deﬁned µ ˆ. That is, whenever A1 ∪ N1 = A2 ∪ N2 we must have µ(A1 ) = µ(A2 ), so that µ ˆ(A1 ∪ N1 ) = µ ˆ(A2 ∪ N2 ).] Deﬁnition 2.3 (Lebesgue sets) The completion of Legesgue measure on (R, B, λ) is still called Lebesgue measure. The resulting completed σ-ﬁeld Bˆλ of the Borel sets B is called the Lebesgue sets. Corollary 1 When we complete a measure µ on a σ-ﬁeld A, this completed measure µ ˆ is the unique extension of µ to Aˆµ . [It is typical to denote the extension by µ also (rather than µ ˆ).] Corollary 2 Thus when we begin with a σ-ﬁnite measure µ on a ﬁeld C, both ˆ [C]µ are unique. the extension to A ≡ σ[C] and the further extension to Aˆµ ≡ σ ∗ ∗ ˆ Here, we note that all sets in Aµ = σ ˆ [C]µ are in the class A of µ -measurable sets. Proof. Consider corollary 1 ﬁrst. Let ν denote any extension to Aˆµ . We will demonstrate that (a)

ν(A ∪ N ) = µ(A)

for all A ∈ A, and all null sets N

(that is, ν = µ ˆ). Assume not. Then there exist sets A ∈ A and N ⊂ (some B) in A with µ(B) = 0 such that ν(A ∪ N ) > µ(A) [necessarily, ν(A ∪ N ) ≥ ν(A) = µ(A)]. For this A and N we have µ(A) < ν(A ∪ N ) = ν(A ∪ (Ac N )) (b)

where Ac N ⊂ Ac B = (a null set)

= ν(A) + ν(Ac N ) ≤ ν(A) + ν(B) since ν is a measure on the completion

(c)

= µ(A) + µ(B)

since ν is an extension of µ.

Hence µ(B) > 0, which is a contradiction. Thus the extension is unique.

16

CHAPTER 1. MEASURES

We now turn to corollary 2. Only the ﬁnal claim needs demonstrating. Suppose A is in σ ˆ [C]µ . Then A = A ∪ N for some A ∈ A and some N satisfying N ⊂ B with µ(B) = 0. Since A∗ is a σ-ﬁeld, it suﬃces to show that any such N is in A∗ . Since µ∗ is subadditive and monotone, we have (d)

µ∗ (T ) ≤ µ∗ (T N ) + µ∗ (T N c ) = µ∗ (T N c ) ≤ µ∗ (T ),

because µ∗ (T N ) = 0 follows from using B, ∅ ∅, . . . to cover T N . Thus equality holds in this last equation, showing that N is µ∗ -measurable. 2 Exercise 2.2 Let µ and ν be ﬁnite measures on (Ω, A). (a) Show by example that Aˆµ and Aˆν need not be equal. (b) Prove or disprove: Aˆµ = Aˆν if and only if µ and ν have exactly the same sets of measure zero. (c) Give an example of an LS-measure µ on R for which Bˆµ = 2R . Exercise 2.3 (Approximation lemma; Halmos) Let the σ-ﬁnite measure µ on the ﬁeld C be extended to A = σ[C], and also refer to the extension as µ. Then for each A ∈ A (or in Aˆµ ) such that µ(A) < ∞, and for each > 0, we have (8)

µ(A C) <

for some set C ∈ C.

[Hint. Truncate the sum in (1.2.1) to deﬁne C, when A ∈ A.] Deﬁnition 2.4 (Regular measures on metric spaces) Let d denote a metric on Ω, let A denote the Borel sets, and let µ be a measure on (Ω, A). Suppose that for each set A in Aˆµ , and for every > 0, one can ﬁnd an open set O and a closed set C for which both C ⊂ A ⊂ O and µ(O \ C ) < . Suppose also that if µ(A) < ∞, one then requires that the set C be compact. Then µ is called a regular measure. [Note exercise 1.3.1 below.] Exercise 2.4 (Nonmeasurable sets) Let Ω consist of the sixteen values 1, . . . , 16. (Think of them arranged in four rows of four values.) Let C1 = {1, 2, 3, 4, 5, 6, 7, 8}, C3 = {1, 2, 5, 6, 9, 10, 13, 14},

C2 = {9, 10, 11, 12, 13, 14, 15, 16}, C 4 = {3, 4, 7, 8, 11, 12, 15, 16}.

Let C = {C1 , C2 , C3 , C4 }, and let A = σ[C]. (a) Show that A ≡ σ[C] = 2Ω . (b) Let µ(Ci ) = 14 for 1 ≤ i ≤ 4. Show that Aˆµ = A, and that it thus contains 24 = 16 sets. (c) Let µ(Ci ) = 13 for 1 ≤ i ≤ 3, with µ(C4 ) = 0. Then show that Aˆµ contains 27 = 128 sets. (d) Illustrate proposition 2.2 below in the context of this exercise. Proposition 2.1 (Not all sets are measurable) Let µ be a measure on A ≡ σ[C], with C a ﬁeld. If B ∈ Aˆµ , then there are inﬁnitely many measures on σ[Aˆµ ∪ {B}] that agree with µ on C. [Thus the σ-ﬁeld Aˆµ is as far as we can go with the unique extension process.] (We merely state this observation for reference, without proof.) [To exhibit a subset of R not in B requires the axiom of choice.]

2.

CONSTRUCTION AND EXTENSION OF MEASURES

Proposition 2.2 (Not all subsets are Lebesgue sets) that is not in Bˆλ .

17

There is a subset D of R

Proof. Deﬁne the equivalence relation ∼ on elements of [0, 1) by x ∼ y if x−y is a rational number. Use the axiom of choice to specify a set D that contains exactly one element from each equivalence class. Now deﬁne Dz ≡ {z + x (modulo 1) : x ∈ D} for each rational z in [0, 1), so that [0, 1) = z Dz represents [0, 1) as a countable outer measure; union of disjoint sets. Moreover, all Dz must have the same call it a. Assume D = D is measurable. But then 1 = λ([0, 1)) = λ(D ) = 0 z z z a gives only z a = 0 (when a = 0) and z a = ∞ (when a > 0) as possibilities. This is a contradiction. Thus D ∈ Bˆλ . 2

Proposition 2.3 (Not all Lebesgue sets are Borel sets) There necessarily exists a set A ∈ Bˆλ \ B that is a Lebesgue set but not a Borel set. Proof.

This proof follows exercise 7.3.3 below.

Exercise 2.5

2

Every subset A of Ω having µ∗ (A) = 0 is a µ∗ -measurable set.

Coverings Earlier in this section we encountered Carath´eodory coverings. Exercise 2.6 (Vitali cover) (a) We say that a family V of intervals I is a Vitali cover of a set D if for each x ∈ D and each > 0 there exists an interval I ∈ V for which x ∈ I and λ(I) < . (b) (Vitali covering theorem) Let D ⊂ R be arbitrary. Let V be a Vitali cover of D. Then there exists a ﬁnite number of pairwise disjoint intervals (I1 , . . . , Im ) in V for which Lebesgue outer measure λ∗ satisﬁes m (9) λ∗ (D \ j=1 Ij ) < . [Lebesgue measure λ will be formally shown to exist in the next section, and λ∗ will be discussed more fully.] [Result (9) will be useful in establishing the Lebesgue result that increasing functions on R necessarily have a derivative, except perhaps on a set having Lebesgue measure zero.] Exercise 2.7 (Heine–Borel) If {Ut : t ∈ T } is an arbitrary collection of open sets that covers a compact subset D of R, then there exists a ﬁnite number of them U1 , . . . , Um that also covers D. [We are merely stating this well-known and frequently used result in the disguise of an exercise so that the reader can easily contrast it with the two other ideas of Carath´eodory covering and Vitali covering introduced in this chapter.]

18

3

CHAPTER 1. MEASURES

Lebesgue–Stieltjes Measures

At the moment we know only a few measures informally. We now construct the large class of measures that lies at the heart of probability theory. Deﬁnition 3.1 (Lebesgue–Stieltjes measure) A measure µ on the real line R assigning ﬁnite values to ﬁnite intervals is called a Lebesgue–Stieltjes measure. [The measure µ on (R, 2R ) whose value µ(A) for any set A equals the number of rationals in A is not a Lebesgue–Stieltjes measure.] Deﬁnition 3.2 (gdf) A ﬁnite function F on R that is right-continuous is called a generalized df (to be abbreviated gdf). Then F− (·) ≡ limy· F (y) denotes the left-continuous version of F . The mass function of F is deﬁned by ∆F (·) ≡ F (·) − F− (·),

while

F (a, b] ≡ F (b) − F (a) for all a < b

is called the increment function of F . We identify gdfs having the same increment function. Only one member F of each equivalence class obtained by such identiﬁcation satisﬁes F− (0) = 0, and this F can (and occasionally will) be used as the representative member of the class (also to be called the representative gdf). Example 3.1 We earlier deﬁned three measures on (R, B) informally. (a) For Lebesgue measure λ, a gdf is the identity function F (x) = x. (b) For counting measure, a gdf is the greatest integer function F (x) = [x]. (c) For unit point mass at x0 , a gdf is F (x) = 1[x0 ,∞) (x). 2 Theorem 3.1 (Correspondence theorem; Lo` eve) (1)

µ((a, b]) = F (a, b]

The relationship

for all − ∞ ≤ a < b ≤ +∞

establishes a 1-to-1 correspondence between Lebesgue–Stieltjes measures µ on B and the representative members of the equivalence classes of generalized dfs. [Each such µ extends uniquely to Bˆµ .] Notation 3.1 We formally establish some notation that will be used throughout. Important classes of sets include: (2)

CI ≡ {all intervals (a, b], (−∞, b], or (a, +∞) : −∞ < a < b < +∞}.

(3)

CF ≡ {all ﬁnite disjoint unions of intervals in CI } = (a ﬁeld) .

(4)

B ≡ σ[CF ] ≡ (the σ-ﬁeld of Borel sets).

(5)

Bˆµ ≡ (the σ-ﬁeld B completed for the measure µ).

(6)

B¯ ≡ σ[B, {−∞}, {+∞}].

2

3.

LEBESGUE–STIELTJES MEASURES

19

Proof. Given an LS-measure µ, deﬁne the increment function F (a, b] via (1). We clearly have 0 ≤ F (a, b] < ∞ for all ﬁnite a, b, and F (a, b] → 0 as b a, by proposition 1.1.2. Now specify F− (0) ≡ 0, F (0) ≡ µ({0}), F (b) ≡ F (0) + F (0, b] for b > 0, and F (a) = F (0) − F (a, 0] for a < 0. This F (·) is the representative gdf. Given a representative gdf, we deﬁne µ on the collection I of all ﬁnite intervals (a, b] via (1). We will now show that µ is a well-deﬁned and c.a. measure on this collection I. Nonnegative: µ ≥ 0 for any (a, b], since F is . Null: µ(∅) = 0, since ∅ = (a, a] and F (a, a] = 0. ∞ Countably additive andwell-deﬁned: Suppose I ≡ (a, b] = ∞ 1 In ≡ 1 (an , bn ]. ∞ ∞ We must show that µ( I ) = µ(I ). n n 1 1 n ∞ First, we will show that 1 µ(In ) ≤ µ(I). Fix n. Then 1 Ik ⊂ I, so that (relabel if necessary, so that I1 , . . . , In is a left-to-right ordering of these intervals) n n (a) 1 µ(Ik ) = 1 F (ak , bk ] ≤ F (a, b] = µ(I). Letting n → ∞ in (a) gives the ﬁrst claim. ∞ Next, we will show that µ(I) ≤ 1 µ(In ). Suppose b−a > > 0 (the case b−a = 0 is trivial, as µ(∅) = 0). For each n ≥ 1, use the right continuity of F to choose an

n > 0 so small that (b)

F (bn , bn + n ] < /2n ,

and deﬁne Jn ≡ (an , cn ) ≡ (an , bn + n ).

These Jn form an open cover of the compact interval [a + , b], so that some ﬁnite number of them are known to cover [a + , b], by the Heine–Borel theorem. Sorting through these intervals one at a time, choose (a1 , c1 ) to contain b, choose (a2 , c2 ) to contain a1 , choose (a3 , c3 ) to contain a2 , . . . ; ﬁnally (for some K), choose (aK , cK ) to contain a + . Then (relabeling the subscripts, if necessary) K K K F (a + , b] ≤ F (aK , c1 ] ≤ 1 F (ak , ck ] ≤ 1 F (ak , bk ] + 1 /2k ∞ (c) ≤ 1 µ(Ik ) + . Let 0 in (c) to obtain the second claim as ∞ (d) µ(I) = F (a, b] ≤ ∞ 1 F (ak , bk ] = 1 µ(Ik ). We will now show that µ is a well-deﬁned c.a. measure on the given ﬁeld CF . If A = n In ∈ CF with each In of type (a, b], then we deﬁne µ(A) ≡ n µ(In ). If we also have another representation A = m Im of this set, then we must show (where the subscripts m and n could take on either a ﬁnite or a countably inﬁnite number of values) that (e) n µ(In ) = m µ(Im ). Now, Im = A ∩ Im = n In Im and In = AIn = m Im In , so that (f) m µ(Im ) = m n µ(In Im ) = n m µ(In Im ) = n µ(In ) (g)

= µ(A).

Finally, a measure µ on CF determines a unique measure on B, as is guaranteed by the Carath´eodory extension of theorem 1.2.1. 2

20 Exercise 3.1 measures.

CHAPTER 1. MEASURES Show that all Lebesgue–Stieltjes measures on (R, B) are regular

Probability Measures, Probability Spaces, and DFs Deﬁnition 3.3 (Probability distributions P (·) and dfs F (·)) (a) In probability theory we think of Ω as the set of all possible outcomes of some experiment, and we refer to it as the sample space. The individual points ω in Ω are referred to as the elementary outcomes. The measurable subsets A in the collection A are referred to as events. A measure of interest is now denoted by P ; it is called a probability measure, and must satisfy P (Ω) = 1. We refer to P (A) as the probability of A, for each event A in AˆP . The triple (Ω, A, P ) (or (Ω, AˆP , Pˆ ), if this is diﬀerent) is referred to as a probability space. (b) An right-continuous function F on R having F (−∞) ≡ limx→−∞ F (x) = 0 and F (+∞) ≡ limx→+∞ F (x) = 1 is called a distribution function (which we will abbreviate as df). [For probability measures, setting F (−∞) = 0 is used to specify the representative df.] Corollary 1 (The correspondence theorem for dfs) Deﬁning P (·) on all intervals (a, b] via P ((a, b] ) ≡ F (b) − F (a) for all −∞ ≤ a < b ≤ +∞ establishes a 1-to-1 correspondence between all probability distributions P (·) on (R, B) and all dfs F (·) on R. Exercise 3.2

Prove the corollary.

Chapter 2

Measurable Functions and Convergence 1

Mappings and σ-Fields

Notation 1.1 (Inverse images) Suppose X denotes a function mapping some ¯ ≡ R ∪ {±∞}; we denote this by X : Ω → R. ¯ set Ω into the extended real line R Let X + and X − denote the positive part and the negative part of X, respectively: X(ω) if X(ω) ≥ 0, (1) X + (ω) ≡ 0 else, (2)

X − (ω) ≡

−X(ω) if X(ω) ≤ 0, 0 else.

Note that (3)

X = X+ − X−

and |X| = X + + X − = X + 2 X − = 2 X + − X .

We also use the following notation: (4)

[ X = r ] ≡ X −1 (r) ≡ { ω : X(ω) = r }

for all real r,

(5)

[ X ∈ B ] ≡ X −1 (B) ≡ { ω : X(ω) ∈ B }

for all Borel sets B,

(6)

X −1 (B) ≡ { X −1 (B) : B ∈ B } .

We call these the inverse images of r, B, and B, respectively. We let (7)

B¯ ≡ σ[ B, {+∞}, {−∞}] .

Inverse images are also well-deﬁned when X : Ω → Ω∗ for arbitrary sets Ω and Ω∗ . 2 21

22

CHAPTER 2. MEASURABLE FUNCTIONS AND CONVERGENCE

For A, B ∈ Ω we deﬁne A B ≡ ABc ∪ Ac B and A \ B ≡ AB c . There is use for the notation (8)

X ≡ sup |X(ω)| , ω∈Ω

and we will also reintroduce this sup norm in other contexts below.

Proposition 1.1 Let X : Ω → Ω and Y : Ω → Ω . Let T denote an arbitrary index set. Then for all A, B, At ⊂ Ω we have (9) (10)

X −1 (B c ) = [ X −1 (B) ]c , X −1 ( t∈T At ) = t∈T X −1 (At ),

X −1 (A \ B) = X −1 (A) \ X −1 (B), X −1 ( t∈T At ) = t∈T X −1 (At ).

For all sets A ⊂ Ω , the composition Y ◦ X satisﬁes (11) Proof.

(Y ◦ X)−1 (A) = X −1 (Y −1 (A)) = X −1 ◦ Y −1 (A) . 2

Trivial.

Proposition 1.2 (Preservation of σ-ﬁelds)

Let X : Ω → Ω . Then:

(12)

X −1 (a σ-ﬁeld A of subsets of Ω ) = (a σ-ﬁeld A of subsets of Ω ) .

(13)

X −1 (σ[C ] ) = σ[X −1 (C )]

for any collection C of subsets of Ω .

{A : X −1 (A ) ∈ (a speciﬁc σ-ﬁeld A of subsets of Ω)} (14) Proof.

= (a σ-ﬁeld A of subsets of Ω ) . Now, (12) is trivial from proposition 1.1. Consider (14). Now: A ∈ A

(a)

implies X −1 (A ) ∈ A

A n ’s ∈ A (b)

implies X −1 (A c ) = [X −1 (A )]c ∈ A implies A c ∈ A , implies X −1 (A n )’s ∈ A

implies X −1 ( n An ) = n X −1 (An ) ∈ A

implies

n An

∈ A .

This gives (14). Consider (13). Now, (c)

X −1 (σ[C ] ) = (a σ-ﬁeld containing X −1 (C )) ⊃ σ[X −1 (C )] .

Then (14) shows that (d)

A ≡ { A : X −1 (A ) ∈ σ[X −1 (C )] } = (a σ-ﬁeld containing C ) ⊃ σ[C ] ,

so that (e)

X −1 (σ[C ] ) ⊂ X −1 (A ) ⊂ σ[X −1 (C )] .

Combining (c) and (e) gives (13). [We apply (13) below to obtain (2.2.6).]

2

1.

MAPPINGS AND σ-FIELDS

23

Roughly, using (12) we will restrict X so that X −1 (B¯ ) ⊂ A for our original ¯ B¯ ), while (14) tells us (Ω, A, P ), so that we can then “induce” a distribution on (R, that the collection A is such that we can always induce a distribution on (Ω , A ). We do this in the next section. First we generalize our deﬁnition of Borel sets to n dimensions. Example 1.1 (Euclidean space)

Let

Rn ≡ R × · · · × R ≡ {(r1 , . . . , rn ) : each ri is in R} . Let Un denote all open subsets of Rn , in the usual Euclidean metric. Then (15)

Bn ≡ σ[Un ] is called the class of Borel sets of Rn .

Following the usual notation, B1 ×· · ·×Bn ≡ {(b1 , . . . , bn ) : b1 ∈ B1 , . . . , bn ∈ Bn }. Now let n (16) i=1 B ≡ B × · · · × B ≡ σ[{B1 × · · · × Bn : all Bi are in B}]. Now consider (17)

σ[ {(−∞, r1 ] × · · · × (−∞, rn ] : all ri are in R} ].

Note that these three σ-ﬁelds are equal. Just observe that each of these three classes generates the generators of the other two classes, and apply exercise 1.1.1. [Surely, we can deﬁne a generalization of area λ2 on (R2 , B2 ) by beginning with λ2 (B1 × B2 ) = λ(B1 ) × λ(B2 ) for all B1 and B2 in B, and then extending to all sets in B2 . We will do this in theorem 5.1.1, and call it Lebesgue measure.] 2

24

2

CHAPTER 2. MEASURABLE FUNCTIONS AND CONVERGENCE

Measurable Functions

We seek a large usable class of functions that is closed under passage to the limit. This is the fundamental property of the class of measurable functions. Propositions 2.2 and 2.3 below will show that the class of measurable functions is also closed under all of the standard mathematical operations. Thus, this class is suﬃcient for our needs. Deﬁnition 2.1 (Simple functions, etc.) Let the measure space (Ω, A, µ) be given and ﬁxed throughout all of our discussion. The indicator function 1A (·) of the set A ⊂ Ω is deﬁned by 1 if ω ∈ A, (1) 1A (ω) ≡ 0 else. A simple function is of the form (2)

X(ω) ≡

n

xi 1Ai (ω)

for

n

Ai = Ω with all Ai ∈ A, and xi ∈ R.

1

i=1

An elementary function is of the form (3)

X(ω) ≡

∞

xi 1Ai (ω)

for

i=1

∞

¯ Ai = Ω with all Ai ∈ A, and xi ∈ R.

i=1

Deﬁnition 2.2 (Measurability) Suppose that X : Ω → Ω , where (Ω , A ) is a measurable space. We say that X is A -A-measurable if X −1 (A ) ⊂ A. We also denote this by writing (4)

X : (Ω, A) → (Ω , A )

or

X : (Ω, A, µ) → (Ω , A )

(or even X : (Ω, A, µ) → (Ω , A , µ ) for the measure µ “induced” on (Ω , A ) by ¯ B¯ ), the mapping X, as will soon be deﬁned). In the special case X : (Ω, A) → (R, we simply call X measurable; and in this special case we let F(X) ≡ X−1 (B¯ ) denote the sub σ-ﬁeld of A generated by X. ¯ Suppose σ[C ] = B. ¯ Proposition 2.1 (Measurability criteria) Let X : Ω → R. Then measurability can be characterized by: (5)

X is measurable

if and only if X −1 (C) ⊂ A,

(6)

X is measurable

if and only if X −1 ( [−∞, x] ) ∈ A for all x ∈ R ;

and note that we could replace [−∞, x] by any one of [−∞, x), [x, +∞], or (x, +∞].

Proof. (a) (b)

Consider (5). Now, X −1 (B¯ ) = X −1 (σ[C ] ) = σ[X −1 (C)] ⊂A

by proposition 2.1.2

since X −1 (C) ⊂ A, and A is a σ-ﬁeld.

2.

MEASURABLE FUNCTIONS

25

The other direction is trivial. Thus (5) holds. To demonstrate (6), we need to show that B¯ satisﬁes (c)

¯ σ[ {[−∞, x] : x ∈ R } ] = B.

Since B = σ[CI ] for CI as in (1.3.2) and (d)

(a, b] = [−∞, b] ∩ [−∞, a]c ,

(e)

{−∞} = ∩n [−∞, −n],

[−∞, b) = ∪∞ 1 [−∞, b − 1/n],

{+∞} = ∩n [−∞, n]c ,

the equality (c) is obvious. The rest is trivial.

etc.,

2

Proposition 2.2 (Measurability of common functions) Let X, Y , and Xn ’s be measurable functions. Consider cX with c > 0, −X, inf Xn , sup Xn , lim inf Xn , lim sup Xn , lim Xn if it exists, X 2 , X ± Y if it is well-deﬁned, XY where 0 · ∞ ≡ 0, X/Y if it is well-deﬁned, X + , X − , |X|, g(X) for continuous g, and the composite function g(X) for all measurable functions g. All of these are measurable functions. Proposition 2.3 (Measurability via simple functions) (7)

Simple and elementary functions are measurable.

(8)

¯ is measurable X :Ω→R if and only if X is the limit of a sequence of simple functions.

Moreover: (9)

If X ≥ 0 is measurable, then X is the limit of a sequence of simple functions that are ≥ 0 and .

[The Xn ’s and Zn ’s that are deﬁned in the proof below are important.] Proof.

The functions in proposition 2.2 are measurable, since: [−X < x] = [X > −x].

(a)

[cX < x] = [X < x/c],

(b)

[inf Xn < x] = ∪[Xn < x],

sup Xn = − inf(−Xn ).

(c)

lim inf Xn = sup( inf Xk ),

lim sup Xn = − lim inf(−Xn ).

n

(d)

k≥n

(e)

lim Xn = lim inf Xn , provided that lim Xn (ω) exists for all ω. √ √ √ √ [X 2 < x] = [− x < X < x ] = [X < x ] ∩ [X ≤ − x ]c .

(f)

[X + Y > x] = ∪r {[X > r] ∩ [Y > x − r] : r is rational}.

(g)

X − Y = X + (−Y ),

(h)

X/Y = X × (1/Y ),

XY = [(X + Y )2 − (X − Y )2 ]/4 .

26

CHAPTER 2. MEASURABLE FUNCTIONS AND CONVERGENCE

where [1/Y < x] = [Y > 1/x] for x > 0, and [1/Y < x] = [Y < 1/x] for x < 0 and [1/Y < 0] = [Y < 0]. X+ = X ∨ 0

(i)

and

X − = (−X) ∨ 0 .

For g measurable, (g ◦ X)−1 (B¯ ) = X −1 (g −1 (B¯ )) ⊂ X −1 (B¯ ) ⊂ A. Then continuous g are measurable, since g −1 (B) = g −1 (σ[ open sets ] ) = σ[g−1 ( open sets )] ⊂ σ[ open sets ] ⊂ B¯ ,

(j)

¯ where we now apply the and both g−1 ({+∞}) = ∅ ∈ B¯ and g −1 ({−∞}) = ∅ ∈ B, result for measurable g. We now prove proposition 2.3. Claim (7) is easy. Consider (8). Deﬁne simple functions Xn by n2n k−1 Xn ≡ × 1 k−1 − 1 k−1 (10) [ 2n ≤ X < 2kn ] [ 2n ≤ −X < 2kn ] 2n k=1

+ n × {1[ X≥n ] − 1[ −X≥n ] } . Since |Xn (ω) − X(ω)| ≤ 2−n for |X(ω)| < n, we have Xn (ω) → X(ω)

(k)

as n → ∞

for each ω ∈ Ω.

Also, the nested subdivisions k/2n cause Xn to satisfy Xn

(l)

when X ≥ 0.

We extend proposition 2.3 slightly by further observing that (11)

Xn − X → 0

as n → ∞,

if X is bounded.

Also, the elementary functions ∞ k−1 (12) Zn ≡ × 1 k−1 − 1 k−1 k k [ 2n ≤X< 2n ] [ 2n ≤−X< 2n ] 2n k=1

+ ∞ × {1[ X=∞ ] − 1[ X=−∞ ] } are always such that (13)

(Zn − X) × 1[−∞<X<∞ ] ≤ 1/2n → 0

as n → ∞.

2

Proposition 2.4 (The discontinuity set is measurable; Billingsley) If (M, d) and (M , d ) are metric spaces and ψ : M → M is any function (not necessarily a measurable function), then the discontinuity set of ψ deﬁned by (14)

Dψ ≡ { x ∈ M : ψ is not continuous at x }

is necessarily in the Borel σ-ﬁeld Bd (that is, the σ-ﬁeld generated by the d-open subsets of M ).

2.

MEASURABLE FUNCTIONS

Proof. (a)

27

Let A,δ ≡ {x ∈ M : d(x, y) < δ, d(x, z) < δ and d (ψ(y), ψ(z)) ≥ for distinct y, z ∈ M }.

Note that A,δ is an open set, since {u ∈ M : d(x, u) < δ0 } ⊂ A,δ will necessarily occur if δ0 ≡ {δ − [d(x, y) ∨ d(x, z)] }/2; that is, the y and z that work for x also work for all u in M that are suﬃciently close to x. (Note: The y that worked for x may have been x itself.) Then ∞ ∞ (b) Dψ = i=1 j=1 Ai ,δj ∈ Bd , where 1 , 2 , . . . and δ1 , δ2 , . . . both denote the positive rationals, since each A,δ is an open set. 2

Induced Measures Example 2.1 (Induced measures) We now turn to the “induced measure” previewed above. Suppose X : (Ω, A, µ) → (Ω , A ), so that X is A -A-measurable. We deﬁne µX ≡ µ by (15)

µX (A ) ≡ µ (A ) ≡ µ(X −1 (A ))

for each A ∈ A .

Then µX ≡ µ is a measure on (Ω , A ), called the induced measure. This is true, since we verify that (a)

µ (∅) = µ(X −1 (∅)) = µ(∅) = 0, µ (

(b)

∞

=

1

An ) = µ(X −1 (

∞ 1

∞

µ(X −1 (An )) =

1

An )) = µ(

∞ 1

∞ 1

X −1 (An ))

µ (An ).

Note also that (c)

µ (Ω ) = µ(X −1 (Ω )) = µ(Ω).

Thus if µ is a probability measure, then so is µX ≡ µ . Note also that we could regard X as an A -F(X)-measurable transformation from the measure space (Ω, F(X), µ) to (Ω , A , µX ). Suppose further that F is a generalized df on the real line R, and µF (·) is the associated measure on (R, B) satisfying µ((a, b] ) = F (b) − F (a) for all a and b (as was guaranteed by the correspondence theorem (theorem 1.3.1)). Thus (R, B, µF ) is a measure space. Let (16)

X(ω) = ω

for all ω ∈ R.

Then X is a measurable transformation from (R, B, µF ) to (R, B) whose induced measure µX is equal to µF . Thus for any given df F we can always construct a measurable function X whose df is F . 2

28

CHAPTER 2. MEASURABLE FUNCTIONS AND CONVERGENCE

Exercise 2.1 Suppose (Ω, A) = (R2 , B2 ), where B2 denotes the σ-ﬁeld generated by all open subsets of the plane. Recall that this σ-ﬁeld contains all sets B × R and R × B for all B ∈ B; here B1 × B2 ≡ {(r1 , r2 ) : r1 ∈ B1 , r2 ∈ B2 }. Now deﬁne measurable transformations X1 ((r1 , r2 )) = r1 and X2 (r1 , r2 )) = r2 . Then deﬁne Z1 ≡ (X12 + X2 )1/2 and Z2 ≡ sign(X1 − X2 ), where sign(r) equals 1, 0, −1 according as r is > 0, = 0, < 0. The exercise is to give geometric descriptions of the σ-ﬁelds F(Z1 ), F(Z2 ), and F(Z1 , Z2 ). Proposition 2.5 (The form of an F(X)-measurable function) Suppose that Y is a measurable function on (Ω, A) that is F(X)-measurable. Then there must exist a measurable function g on (R, B) such Y = g(X). Proof. [The approach of this proof is to consider indicator functions, simple functions, nonnegative functions, general functions. This approach will be used again and again. Learn it!] Suppose that Y = 1D for some set D ∈ F(X), so that Y is an indicator function that is F(X)-measurable. Then we can rewrite Y as Y = 1D = 1X −1(B) = 1B (X) = g(X), for some B ∈ B. Thus the proposition holds for indicator functions. It holds for simple functions, since when all Bi ∈ B, m m m Y = 1 ci 1Di = 1 ci 1X −1 (Bi ) = 1 ci 1Bi (X) ≡ g(X). Let Y ≥ 0 be F(X)-measurable. Then there do exist simple F(X)-measurable functions Yn such that Y ≡ limn Yn = limn gn (X) for the simple B-measurable functions gn . Now let g = lim gn , which is B-measurable, and note that Y = g(X). For general Y = Y + − Y − , use g = g + − g − . 2 Exercise 2.2 (Measurability criterion) Let C denote a π ¯ -system of subsets of Ω. Let V denote a vector space of functions (that is, X + Y ∈ V and αX ∈ V for all X, Y ∈ V and all α ∈ R). (a) Suppose that: (17)

1C ∈ V

for all C ∈ C,

(18)

If An A

with 1An ∈ V,

then 1A ∈ V.

Show that 1A ∈ V for every A ∈ σ[C]. (b) It then follows trivially that every simple function m (19) Xn ≡ 1 αi 1Ai is in V ; m here m ≥ 1, all αi ∈ R, and 1 Ai = Ω with all Ai ∈ σ[C ]. (c) Now suppose further that Xn X for Xn ’s as in (19) implies that X ∈ V. Show that V contains all σ[C]-measurable functions.

3.

CONVERGENCE

29

3

Convergence

Convergence Almost Everywhere Deﬁnition 3.1 (→a.e. ) Let X1 , X2 , . . . denote measurable functions on (Ω, A, µ) ¯ B¯ ). Say that the sequence Xn converges almost everywhere to X (denoted to (R, by Xn →a.e. X as n → ∞) if for some N ∈ A for which µ(N ) = 0 we have Xn (ω) → X(ω) as n → ∞ for all ω ∈ N . If for all ω ∈ N the sequence Xn (ω) is a Cauchy sequence, then we say that the sequence Xn mutually converges a.e. and denote this by writing Xn − Xm →a.e. 0 as m ∧ n → ∞. ¯ B¯ ). Exercise 3.1 Let X1 , X2 , . . . be measurable functions from (Ω, A, µ) to (R, ˜ ˜ (a) If Xn →a.e. X, then X = X a.e. for some measurable X. (b) If Xn →a.e. X and µ is complete, then X itself is measurable. Proposition 3.1 A sequence of measurable functions Xn that are a.e. ﬁnite converge a.e. to a measurable function X that is a.e. ﬁnite if and only if these functions Xn converges mutually a.e. [Thus we can redeﬁne such functions on null sets and make them everywhere ﬁnite and convergent.] Proof. The union of the countable number of null sets on which ﬁniteness or convergence fails is again a null set N . On N c , the claim is just a property of the real numbers. 2 Proposition 3.2 (The convergence and divergence sets are measurable) Consider the ﬁnite measurable functions X, X1 , X2 , . . . (perhaps they have been redeﬁned on null sets to achieve this); thus, they are B-A-measurable. Then the convergence and mutual convergence sets are measurable. In fact, the convergence set is given by ∞ ∞ ∞ 1 |Xm − X| < (1) [Xn → X] ≡ ∈ A, k n=1 m=n k=1

and the mutual convergence set is given by ∞ ∞ ∞ 1 (2) [Xn − Xm → 0] ≡ |Xm − Xn | < ∈ A. k n=1 m=n k=1

Proof. Just read the right-hand side of (1) as, for all ≡ 1/k > 0 there exists an n such that for all m ≥ n we have |Xm (ω) − X(ω)| < 1/k. (Practice saying this until it makes sense.) 2 Taking complements in (1) allows the divergence set to be expressed via ∞ ∞ ∞ ∞ 1 c (3) [ Xn → X] = |Xm − X| ≥ ≡ Ak with Ak in k, k n=1 m=n k=1

where (4)

Ak =

∞

n=1 Dkn ,

k=1

where the Dkn ≡

∞

m=n [ |Xm − X|

≥ 1/k ] are in n.

30

CHAPTER 2. MEASURABLE FUNCTIONS AND CONVERGENCE

Proposition 3.3 (i) We have

(5)

Consider ﬁnite measurable Xn ’s and a ﬁnite measurable X.

if and only if Xn →a.e. (such an X) ∞ for all > 0. µ( ∞ n=1 m=n [ |Xm − Xn | ≥ ] ) = 0 ,

[A ﬁnite limit X exists if and only if the Cauchy criterion holds; and we want to be able to check for the existence of a ﬁnite limit X without knowing its value.] (ii) (Most useful criterion for →a.e. ) When µ(Ω) < ∞, we have

(6)

if and only if Xn →a.e. (such an X) ∞ µ( m=n [ |Xm − Xn | ≥ ] ) → 0, for all > 0,

(7)

µ( [ max |Xm − Xn | ≥ ] ) ≤ for all N ≥ n ≥ (some n ), for all > 0.

Proof.

if and only if

n≤m≤N

This is immediate from (3), (4), and proposition 1.1.2.

2

Remark 3.1 (Additional measurability for convergence and divergence) Suppose we still assume that X1 , X2 , . . . are ﬁnite measurable functions. Then the following sets are seen to be measurable:

(8) (9)

¯ ]c = [ lim inf Xn < lim sup Xn ] [ω : Xn (ω) → X(ω) ∈ R = rational r [ lim inf Xn < r < lim sup Xn ] ∈ A, [ lim sup Xn = +∞ ] = ∞ m=1 [ lim sup Xn > m ] ∈ A.

These comments reﬂect the following fact: If Xn (ω) does not converge to a ﬁnite number, then there are several diﬀerent possibilities; but these interesting events are all measurable. 2

Convergence in Measure Deﬁnition 3.2 (→µ ) A given sequence of a.e. ﬁnite and measurable functions X1 , X2 , . . . is said to converge in measure to the measurable function X taking ¯ (to be denoted by Xn →µ X as n → ∞) if values in R (10)

µ( [ |Xn − X| ≥ ] ) → 0

as n → ∞,

for all > 0.

[This implies that X must be ﬁnite a.s., as ∞ [ |X| = ∞] ⊂ { k=1 [ |Xk | = ∞]} ∪ [ |Xn − X| ≥ ] shows.] We say that these Xn converge mutually in measure, which we denote by writing Xm − Xn →µ 0 as m ∧ n → ∞, if µ( [ |Xm − Xn | ≥ ] ) → 0 as m ∧ n → ∞, for each > 0.

3.

CONVERGENCE

Proposition 3.4 Proof. (a)

31

˜ then X = X ˜ a.e. If Xn →µ X and Xn →µ X,

For all > 0 ˜ ≥ 2 ] ) ≤ µ( [ |Xn − X| ≥ ] ) + µ( [ |Xn − X| ˜ ≥ ] ) → 0. µ( [ |X − X|

Thus (b)

˜ ) = µ( [ |X − X| ˜ ≥ 1/k ] ) ≤ ∞ µ( |X − X| ˜ ≥ 1/k ) = ∞ 0 , µ( [ X = X] 1 1 k

as claimed.

2

Exercise 3.2 (a) Show that in general →µ does not imply →a.e. . (b) Give an example with µ(Ω) = ∞ where →a.e. does not imply →µ . Theorem 3.1 (Relating →µ to →a.e. ) ﬁnite a.e.

Let X, X1 , X2 , . . . be measurable and

(11)

Xn →a.e. (such an X)

if and only if

Xn − Xm →a.e. 0.

(12)

Xn →µ (such an X)

if and only if

Xn − Xm →µ 0.

(13)

Let µ(Ω) < ∞. Then Xn →a.e. (such an X) implies Xn →µ X.

(14)

(Riesz) If Xn →µ X, then for some subsequence nk we have Xnk →a.e. X.

(Reducing →µ to →a.e. by going to subsequences) Suppose µ(Ω) < ∞. Then

(15)

Xn →µ X if and only if each subsequence n has a further n on which Xn →a.e. (such an X).

Proof. Now, (11) is proposition 3.1, and (12) is exercise 3.3. Result (13) comes from the elementary observation that ∞ (a) µ( [ |Xn − X| ≥ ] ) ≤ µ( m=n [ |Xm − X| ≥ ] ) → 0, by (6). To prove (14), choose nk ↑ such that (b)

µ(Ak ) ≡ µ( [ |Xnk − X| > 1/2k ] ) < 1/2k ,

with µ( [ |Xn − X| > 1/2k ] ) < 1/2k for all n ≥ nk . Now let ∞ ∞ (c) Bm ≡ k=m Ak , so that µ(Bm ) ≤ k=m 2−k ≤ 1/2m−1. ∞ c On Bm = m Ack we have |Xnk − X| ≤ 1/2k for all k ≥ m, so that (16)

c |Xnk (ω) − X(ω)| ≤ 1/2k → 0 as k → ∞ for each ω ∈ Bm ,

with µ(Bm ) < 1/2m−1 . Thus (d)

Xnk (ω) → X(ω) as k → ∞ for each ω ∈ C ≡

∞

c m=1 Bm

,

32

CHAPTER 2. MEASURABLE FUNCTIONS AND CONVERGENCE

where (e)

µ(C c ) = µ(

∞

m=1 Bm )

≤ lim sup µ(Bm ) ≤ lim 1/2m−1 = 0

as required. (For exercise 3.3 below, replace X above by Xnk+1 , and claim a.s. convergence to some X on this subsequence. Then show that the whole sequence converges in measure to this X.) Consider the unproven half of (15) by assuming that every n contains a further n as claimed. Assume that Xn →µ fails. Then for some > 0 and some n (f)

limn µ( [ |Xn − X| > ] ) = (some a) > 0.

But we are given that some further subsequence n has Xn →a.e. X, and thus Xn →µ X by (13). Thus (g)

limn µ( [ |Xn − X| > ] ) = 0;

but this is a contradiction of (f).

2

Exercise 3.3 As in (12), show that Xn →µ X if and only if Xm − Xn →µ 0. [Hint. Adapt the proof of (16).] Exercise 3.4 Suppose that µ(Ω) < ∞ and g is continuous a.e. µX (that is, g is continuous except perhaps on a set of µX measure 0). Then Xn →µ X implies g(Xn ) →µ g(X).

4.

PROBABILITY, RVS, AND CONVERGENCE IN LAW

4

Probability, RVs, and Convergence in Law

33

Deﬁnition 4.1 (Random variable and df ) A probability space (Ω, A, P ) is just a measure space for which P (Ω) = 1. Now, X : (Ω, A, P ) → (R, B) will be called a random variable (to be abbreviated rv); thus it is a B-A-measurable ¯ B¯ ), then we will call X an extended rv. The function. If X : (Ω, A, P ) → (R, distribution function (to be abbreviated df) of a rv is deﬁned by (1)

FX (x) ≡ P (X ≤ x)

for all − ∞ < x < ∞.

We recall that F ≡ FX satisﬁes (2)

F is and right continuous, with F (−∞) = 0 and F (+∞) = 1.

We let CF denote the continuity set of F , that contains all points at which F is continuous. [That F is trivial, and the other three properties all follow from the monotone property of measure, since (∞, x] = ∩∞ n=1 (−∞, x + an ] for every sequence ∞ an 0, ∩∞ n=1 (−∞, −n] = ∅, and ∪n=1 (−∞, n] = R.] (If F is and right continuous with F (−∞) ≥ 0 and F (+∞) ≤ 1, then F will be ¯ B¯ )) will be denoted by PX , called a sub df.) The induced measure on (R, B) (or (R, and it satisﬁes (3)

PX (B) = P (X −1 (B)) = P (X ∈ B)

for all B ∈ B

(for all B ∈ B¯ if X is an extended rv). We call this the induced distribution of X. We use the notation X ∼ = F to denote that the induced distribution PX (·) of the rv X has df F . We say that rvs Xn (with dfs Fn ) converge in distribution or converge in law to a rv X0 (with df F0 ) if (4)

Fn (x) = P (Xn ≤ x) → F0 (x) = P (X0 ≤ x)

at each x ∈ CF0 .

We abbreviate this by writing either Xn →d X0 , Fn →d F0 , or L(Xn ) → L(X0 ). Notation 4.1 Suppose now that {Xn : n ≥ 0}, are rvs on (Ω, A, P ). Then it is customary to write Xn →p X0 (in place of Xn →µ X0 ) and Xn →a.s. X0 (as well as Xn →a.e. X0 ). The “p” is an abbreviation for in probability, and the “a.s.” is an abbreviation for almost surely. Anticipating the next chapter, we let Eg(X) denote g(X) dµ, or g(X) dP when µ is a probability measure P . We say that Xn converges to X0 in rth mean if E|Xn − X0 |r → 0. We denote this by writing Xn →r X0 or Xn →Lr X0 . 2 Proposition 4.1 Then Xn →d X.

Suppose that the rvs X ∼ = F and Xn ∼ = Fn satisfy Xn →p X.

Proof. (This result has limited importance. But the technique introduced here is useful; see exercise 4.1 below.) Now, (a) (b)

Fn (t) = P (Xn ≤ t) ≤ P (X ≤ t + ) + P ( |Xn − X| ≥ ) ≤ F (t + ) +

for all n ≥ some n .

34

CHAPTER 2. MEASURABLE FUNCTIONS AND CONVERGENCE

Also, Fn (t) = P (Xn ≤ t) ≥ P (X ≤ t − and |Xn − X| ≤ ) ≡ P (AB) ≥ P (A) − P (Bc ) = F (t − ) − P ( |Xn − X| ≥ ) ≥ F (t − ) −

for n ≥ (some n ).

Thus for n ≥ (n ∨ n ) we have (c)

F (t − ) − ≤ lim Fn (t) ≤ lim Fn (t) ≤ F (t + ) + .

If t is a continuity point of F , then letting → 0 in (c) gives Fn (t) → F (t). Thus Fn →d F . 2 The following elementary result is extremely useful. Often, one knows that Xn →d X, but what one is really interested in is a slight variant of Xn , rather than Xn itself. The next result was designed for just such situations. Theorem 4.1 (Slutsky) Suppose that Xn →d X, while the rvs Yn →p a and Zn →p b as n → ∞ (here Xn , Yn , and Zn are deﬁned on a common probability space, but X need not be). Then (5)

Yn × Xn + Zn →d aX + b

as n → ∞.

Exercise 4.1

Prove Slutsky’s theorem. [Hint. Recall proposition 4.1.]

Exercise 4.2

Let c be a constant. Show that Xn →d c if and only if Xn →p c.

Remark 4.1 Suppose X1 , X2 , . . . are independent rvs with a common df F . Then Xn →d X0 for any rv X0 having df F . However, there is no rv X for which Xn converges to X in the sense of →a.s. , →p , or →r . 2

5.

DISCUSSION OF SUB σ-FIELDS

5

Discussion of Sub σ-Fields

35

Consider again a sequence of rvs X1 , X2 , . . . where each quantity Xn is a measurable transformation Xn : (Ω, A, P ) → (R, B, Pn ), with Pn denoting the induced measure PXn . Each of the rvs Xn is B-F(Xn )-measurable, with F(Xn ) a sub σ-ﬁeld of A. Even though the intersection of any number of σ-ﬁelds is a σ-ﬁeld, the union of even two σ-ﬁelds need not be a σ-ﬁeld. We thus deﬁne the sub σ-ﬁeld generated by X1 , . . . , Xn as n −1 (Bn ) ≡ (X1 , . . . , Xn ) , (1) F(X1 , . . . , Xn ) ≡ σ[ k=1 F(Xk ) ] = X for X where the equality will be shown in proposition 5.2.1 below. Note that F(X1 , . . . , Xn ) ⊂ F(X1 , . . . , Xn , Xn+1 ), so that these necessarily form an increasing sequence of σ-ﬁelds of A. Also, deﬁne ∞ (2) F(X1 , X2 , . . .) ≡ σ[ k=1 F(Xk ) ]. It is natural to say that such Xn ’s are adapted to the F(X1 , . . . , Xn ). In fact, if F1 ⊂ F2 ⊂ · · · is any sequence of σ-ﬁelds for which F(X1 , . . . , Xn ) ⊂ Fn for all n, then we say that the Xn ’s are adapted to the Fn ’s. Think of F(X1 , . . . , Xn ) as the amount of information available at time n from X1 , . . . , Xn ; that is, you have available for inspection all of the probabilities (3)

P ((X1 , . . . , Xn ) ∈ Bn ) = P ((X1 , . . . , Xn )−1 (Bn )) = P(X1 ,...,Xn ) (Bn ),

for all Borel sets Bn ∈ Bn . Rephrasing, you have available for inspection all of the probabilities (4)

P (A), for all A ∈ F(X1 , . . . , Xn ).

At stage n+1 you have available P (A) for all A ∈ F(X1 , . . . , Xn , Xn+1 ); that is, you have more information available. [Think of Fn \ F(X1 , . . . , Xn ) as the amount of information available to you at time n that goes beyond the information available from X1 , . . . , Xn ; perhaps it comes from other rvs not yet mentioned, but it is available nonetheless.] Suppose we are not given rvs, but rather (speaking informally now, based on your general feel for probability) we are given joint dfs Fn (x1 , . . . , xn ) that we think ought to suﬃce to construct probability measures on (Rn , Bn ). In (2.2.16) we saw that for n = 1 we could just let (Ω, A, µ) = (R, B, µF ) and use X(ω) = ω to deﬁne a rv that carried the information in the df F . How do we deﬁne probability measures Pn on (Rn , Bn ) so that the coordinate rvs (5)

Xk (ω1 , . . . , ωn ) = ωk

for all (ω1 , . . . , ωn ) ∈ Rn

satisfy (6)

Pn (X1 ≤ x1 , . . . , Xn ≤ Xn ) = F (x1 , . . . , xn )

for all (x1 , . . . , xn ) ∈ Rn ,

and thus carry all the information in Fn ? Chapter 5 will deal with this construction. But even now it is clear that for this to be possible, the Fn ’s will have to satisfy some kind of consistency condition as we go from step n to n + 1. Moreover, the consistency problem should disappear if the resulting Xn ’s are “independent”.

36

CHAPTER 2. MEASURABLE FUNCTIONS AND CONVERGENCE

But we need more. We will let R∞ denote all inﬁnite sequences ω1 , ω2 , . . . for which each ωi ∈ R. Now, the construction of (5) and (6) will determine probabilities ∞ on the collection Bn × k=n+1 R of all subsets of R∞ of the form ∞ Bn × k=n+1 R (7) ≡ {(ω1 , . . . , ωn , ωn+1, . . .) : (ω1 , . . . , ωn ) ∈ Bn , ωk ∈ R for k ≥ n + 1}, with Bn ∈ Bn . Each of these collections is a σ-ﬁeld (which within this special probability space can be denoted by F(X1 , . . . , Xn )) in this overall probability space (R∞ , B∞ , P∞ ), for some appropriate B∞ . But what is an appropriate σ-ﬁeld B∞ for such a probability measure P∞ ? At a minimum, B∞ must contain ∞ ∞ ∞ (8) σ[ n=1 {Bn × k=n+1 R}] = σ[ n=1 F(X1 , . . . , Xn )], and indeed, this is what we will use for B∞ . Of course, we also want to construct the measure P∞ on (R∞ , B∞ ) in such a way that n ∞ (9) P∞ (−∞, x ] × R = Fn (x1 , . . . , xn ) for all n ≥ 1 k k=1 k=n+1 and for all x1 , . . . , xn . The details are given in chapter 5. Until chapter 5 we will assume that we are given the rvs X1 , X2 , . . . on some (Ω, A, P ), and we will need to deal only with the known quantities F(X1 , . . . , Xn ) and F(X1 , X2 , . . .) deﬁned in (1) and (2). This is probability theory: Given (Ω, A, P ), we study the behavior of rvs X1 , X2 , . . . that are deﬁned on this space. Now contrast this with statistics: Given a physical situation producing measureθ ments X1 , X2 , . . ., we construct models {(R∞ , B∞ , P∞ ) : θ ∈ Θ} based on various θ plausible models for Fn (x1 , . . . , xn ), θ ∈ Θ, and we then use the data X1 , X2 , . . . and the laws of probability theory to decide which model θ0 ∈ Θ was most likely to have been correct and what action to take. In particular, the statistician must know that the models to be used are well-deﬁned. We also need to extend all this to uncountably many rvs {Xt : t ∈ T }, for some interval T such as [a, b], or [a, ∞), or [a, ∞], or (−∞, ∞), . . . . We say that rvs Xt : (Ω, A, P ) → (R, B) for t ∈ T are adapted to an sequence of σ-ﬁelds Ft if Fs ⊂ Ft for all s ≤ t with boths, t ∈ T and if each Xt is Ft -measurable. In this situation we typically let RT ≡ t∈T Rt and then let (10) Ft ≡ F(Xs : s ≤ t) ≡ σ[ s Xs−1 (B) : s ≤ t and s ∈ T ] for all t ∈ T. This is also done in chapter 5 (where more general sets T are, in fact, considered). The purpose in presenting this section here is to let the reader start now to become familiar and comfortable with these ideas before we meet them again in chapter 5 in a more substantial and rigorous presentation. (The author assigns this as reading at this point and presents only a very limited amount of chapter 5 in his lectures.)

Chapter 3

Integration 1

The Lebesgue Integral

Let (Ω, A, µ) be a ﬁxed measure space and let X, Y, Xn , . . . denote measurable n ¯ B¯ ). If Ω = functions from (Ω, A, µ) to (R, 1 Ai where A1 , . . . , An are in A, then A1 , . . . , An is called a partition (or measurable partition) of Ω. n Deﬁnition 1.1 (Lebesgue integral X dµ or X) If X = i=1 xi 1Ai is a simple function (in that all real xi ≥ 0 and A1 , . . . , An is a partition of Ω), then (1)

X dµ ≡

n

xi µ(Ai ).

i=1

If X ≥ 0, then (2) X dµ ≡ sup Y dµ : 0 ≤ Y ≤ X and Y is a simple function . [Of course, we must show that the value of representation of X that is speciﬁed.] For general measurable X, + X dµ ≡ X dµ − X − dµ, (3)

X dµ in (1) is independent of the

X + dµ and X − dµ is ﬁnite. We let L1 ≡ L1 (Ω, A, µ) ≡ {X : |X| dµ < ∞}, + L+ 1 ≡ L1 (Ω, A, µ) ≡ {X ∈ L1 : X ≥ 0}, Lr ≡ Lr (Ω, A, µ) ≡ Lr (µ) ≡ {X : |X|r dµ < ∞}, for each r > 0.

provided that at least one of

(4)

If X (which is not equals a measurable function Y on a set A having measurable) µ(Ac ) = 0, then X dµ ≡ Y dµ. [It is trivial that the value X dµ is independent 37

38

CHAPTER 3. INTEGRATION

of the choice of Y and A.] If X is measurable and X dµ is ﬁnite, then X is called integrable. For any A ∈ A, (5) X dµ ≡ X 1A dµ. A

We also use the notation (especially in proofs, to save space) (6) X ≡ X dµ ≡ (the integral of X) ≡ EX ≡ (the expectation of X) . For ordinary Lebesgue measure µ on R, we often write

X dµ =

X(r) dr .

It needs to be demonstrated that the above deﬁnition makes sense and that X dµ satisﬁes the following elementary properties.

Proposition 1.1 (Elementary properties of the integral) It holds that deﬁnition 1.1 of the integral is unambiguous. Now suppose that the functions X and Y are measurable, that X dµ and Y dµ are well-deﬁned, and that their sum (the number X dµ + Y dµ) is a well-deﬁned number in [−∞, +∞]. Then (7) (X + Y ) dµ = X dµ + Y dµ and cX dµ = c X dµ, (8)

X ≥0

implies

X dµ ≥ 0.

Proof. Consider case of simple functions. ﬁrst the m m xi µ(Ai ) for simple functions X = 1 xi 1Ai makes Step 1: Deﬁning X dµ = 1 X dµ well-deﬁned for such simple functions. Suppose that we also have Y = n1 yj 1Bj . Then m n n m (a) i=1 xi j=1 1Ai Bj = X = j=1 yj i=1 1Ai Bj , so that xi = yj if Ai Bj = ∅. Thus m m n m n i=1 xi µ(Ai ) = i=1 xi j=1 µ(Ai Bj ) = i=1 j=1 xi µ(Ai Bj ) = =

m n i=1

n

j=1 yj

j=1

since xi = yj if Ai Bj = ∅

yj µ(Ai Bj )

m

i=1 µ(Ai Bj )

=

n j=1

yj µ(Bj ) ;

and since the two extreme terms represent the two diﬀerent deﬁnitions of X dµ, we see that X dµ is well-deﬁned. Step 2: The integral m behaves linearly n for simple functions. m n Suppose X = 1 xi 1Ai and Y = 1 yj 1Bj . Then X +Y = 1 1 (xi +yj ) 1Ai Bj . We thus have m n (X + Y ) dµ = 1 1 (xi + yj ) µ(Ai Bj ) m n m n = 1 1 xi µ(Ai Bj ) + 1 1 yj µ(Ai Bj )

1.

THE LEBESGUE INTEGRAL n m µ(Ai Bj ) + 1 yj 1 µ(Ai Bj ) n X dµ + Y dµ, = m 1 xi µ(Ai ) + 1 yj µ(Bj ) = =

m

39

1

xi

n 1

which establishes the additivity. Step 3: Even in general, the monotonicity is trivial; and so is cX = c X. The proof for general X ≥ 0 and Y ≥ 0 is included in the proof of the monotone convergence theorem (MCT) (that is, the ﬁrst theorem of the next section). That is, we will prove the MCT using linearity just for simple functions, and then we will use the MCT to obtain the current linearity for any functions X ≥ 0 and Y ≥ 0. (The ﬁnal linearity step is then trivial. Just write X = X + −X − and Y = Y + −Y − and do algebra.) 2

40

2

CHAPTER 3. INTEGRATION

Fundamental Properties of Integrals

Theorem 2.1 (MCT, the monotone convergence theorem) Xn X a.e. for measurable functions Xn ≥ 0. Then (1) 0 ≤ Xn dµ X dµ .

Suppose that

Proof. By redeﬁning on null sets if necessary, we may assume that Xn X for all ω. Thus X is measurable, by proposition 2.2.2. Also, X n is , and so a ≡ lim Xn exists in [0, ∞]. Moreover, Xn ≤ X implies Xn ≤ X; and so we conclude that a ≤ X. Let Y be an arbitrary simple function satisfying 0 ≤ Y ≤ X. Fix 0 < θ < 1. Then note that An ≡ [Xn ≥ θ Y ] Ω (since 0 ≤ θ Y ≤ X on [X = 0] and 0 ≤ θ Y < X on [X > 0] are both trivial). We thus have (a) θ Y × 1An = θ Y × 1An ≤ Xn × 1An ≤ Xn ≤ a; and passing to thelimit as n → ∞ gives θ Y ≤ a (by proposition 3.1.1) for each 0 < θ < 1, so that Y ≤ a. Since 0 ≤ Y ≤ X is arbitrary, this gives X ≤ a. This completes the proof of the MCT. 2 Proof. We now return to the linearity of the integral for general measurable functions X ≥ 0 and Y ≥ 0. Let Xn X and Yn Y be the measurable simple functions of (2.2.10). Then Xn +Yn X +Y . Thus the MCT twice, the linearity of the integral for simple functions, and then the MCT again give the general linearity of the integral X + Y = lim Xn + lim Yn = lim ( Xn + Yn ) (a) = lim (Xn + Yn ) by simple function linearity (b) = (X + Y ) by the MCT. 2

Theorem 2.2 (Fatou’s lemma) For Xn ’s measurable, (2) lim Xn dµ ≤ lim Xn dµ, provided that Xn ≥ 0 a.e. for all n. Proof. (3) so that (a) (b)

Now, a.e. we have Xn ≥ Yn ≡ inf Xk lim Xn , k≥n

or

lim Xn = lim Yn with Yn ,

lim Xn = lim Yn = lim Yn by the MCT = lim Yn ≤ lim Xn since Yn ≤ Xn . 2

2.

FUNDAMENTAL PROPERTIES OF INTEGRALS

41

Theorem 2.3 (DCT, the dominated convergence theorem) Suppose now that |Xn | ≤ Y a.e. for all n, for some dominating function Y ∈ L1 ; and suppose either (i) Xn →a.e. X or (ii) Xn →µ X. Then (4) (that is, Xn →L1 X). |Xn − X| dµ → 0 as n → ∞ [Note that (supn≥1 |Xn | ) would make a suitable dominating function.] Corollary 1 Note that (4) implies Xn dµ → X dµ (5) (that is, (6)

EXn → EX),

Xn dµ − X dµ → 0. sup

A∈A

A

A

Proof. (i) Suppose that Xn →a.e. X. Then Zn ≡ |Xn − X| →a.e. 0, with 0 ≤ Zn ≤ 2Y a.s., of the functions 0 and where both 2Y are in L1 . Thus Fatou’s lemma gives 0 = 0 dµ = lim inf Zn dµ ≤ lim inf Zn . Now apply Fatou’s lemma to the rvs 2Y − Zn , and conclude that (a) (2Y − 0) ≤ lim (2Y − Zn ) ≤ lim (2Y − Zn ) by Fatou (7) = lim ( 2Y − Zn ) = 2Y − lim Zn . Hence, lim sup Zn ≤ 0 = 0. Combining the two results gives (b) 0 ≤ lim inf Zn ≤ lim sup Zn ≤ 0 ; so lim Zn = 0, as claimed. (ii) Suppose Xn →µ X. Let a ≡ lim sup Zn ≥ 0. Let n be a subsequence such that Zn → a. But Zn →µ 0, so theorem 2.3.1 gives a further subsequence n such that Zn →a.e. 0, while we still have Zn → a. But Zn → 0 by case (i). Thus a = 0. Thus (c) 0 ≤ lim sup Zn = a = 0, or Zn → 0 . (iii) Consider the corollary. We have | Xn − X| ≤ |Xn − X|, and thus (d) | A Xn − A X| ≤ A |Xn − X| ≤ |Xn − X| → 0 uniformly in all A ∈ A. Theorem 2.4

∞ 1

2 Xn dµ =

∞

Proof. Note that 0 ≤ Zn ≡ MCT to the Zn ’s. 2

1

n 1

Xn dµ

Xk Z ≡

if Xn ≥ 0 a.e., for all n. ∞ 1

Xk a.e., and now apply the

42

CHAPTER 3. INTEGRATION

Theorem 2.5 (Absolute continuity of the integral) (8) |X| dµ → 0 as µ(A) → 0.

Fix X ∈ L1 . Then

A

That is,

A

Proof. (a)

|X| dµ < , provided only that µ(A) < (an appropriate δ ).

Now,

|X| 1[ |X|≤n]

|X| 1[ |X|>n] ≤ /2

|X| by the MCT, so we may claim that

for n ≥ N ≡ (some N ) .

Thus (b)

A

|X| ≤

A

|X| 1[ |X|≤N ] +

provided that µ(A) ≤ /(2N ) .

|X| 1[ |X|>N ] ≤ N × µ(A) + /2 ≤ ,

2

Deﬁnition 2.1 (Induced measure) Suppose that X : (Ω, A, µ) → (Ω , A ) is a measurable function. Recall from (2.2.15) that (9)

µ (A ) ≡ µX (A ) = µ(X −1 (A ))

for all A ∈ A ,

and µ is a measure on (Ω , A ), called the induced measure of X. Theorem 2.6 (Theorem of the unconscious statistician) First, the induced measure µX (·) of the rv X determines the induced measure µg(X) for all measurable ¯ B¯ ). Second, functions g : (Ω , A ) → (R, (10) g(X(ω)) dµ(ω) = g(x) dµX (x) for all A ∈ A , X −1 (A )

A

in the sense that if either side exists, then so does the other and they are equal. Proof. (a)

Now, g(X) is measurable. By (2.1.11) and (2.2.5) we see that µg(X) (B) = µ( [g(X) ∈ B] ) = µ(X −1 ◦ g −1 (B)) = µX (g −1 (B))

is well-deﬁned, since g−1 (B) ∈ A . Thus the ﬁrst claim holds. We now turn to the second claim. It suﬃces to prove this when A = Ω and X −1 (Ω ) = Ω, since we need only replace g by g × 1A , noting that 1A (X(ω)) = 1X −1 (A ) (ω). Case 1. g = 1A : Then 1A (X) dµ = 1X −1 (A ) dµ = µ(X −1 (A )) = µX (A ) = 1A dµX . n n Case 2. g = i=1 ci 1Ai , where i=1 Ai = Ω with Ai ∈ A : Then n n g(X) dµ = i=1 ci 1Ai (X) dµ = i=1 ci 1Ai (X) dµ (b)

(c)

=

n

i=1 ci

1Ai dµX =

g dµX .

2.

FUNDAMENTAL PROPERTIES OF INTEGRALS

43

Case 3. g ≥ 0: Let gn ≥ 0 be simple with gn g : Then g(X) dµ = lim gn (X) dµ by the MCT, since gn (X) g(X) = lim gn dµX by case 2 (d) by the MCT. = g dµX Case 4. g is measurable, and either g(X)+ dµ or g(X)− dµ is ﬁnite: Using g = g + − g − , we note that g(X)+ = g + (X) and g(X)− = g − (X). Then g(X) dµ = g(X)+ dµ − g(X)− dµ = g + (X) dµ − g − (X) dµ = g + dµX − g − dµX by case 3 (e) = g dµX . In the arguments (b), (c), (d), (e) one should start from the end that is assumed to exist, in order to make a logically tight argument. 2

Exercise 2.1 (Only the zero function) Show that (11) X ≥ 0 and X dµ = 0 implies µ( [X > 0] ) = 0 . Exercise 2.2 (Only the zero function) = 0, (12) X dµ = for all A ∈ A ≥0 A

Show that implies

X=

= 0 a.e., ≥ 0 a.e.

Exercise 2.3 Consider a measure space (Ω, A, µ). Let µ0 ≡ µ|A0 for a sub σ-ﬁeld A0 of A. Starting with indicator functions, show that X dµ = X dµ0 for any A0 -measurable function X.

44

3

CHAPTER 3. INTEGRATION

Evaluating and Diﬀerentiating Integrals

Let (R, Bˆµ , µ) denote a Lebesgue–Stieltjes measure space that has been completed. If g is Bˆµ -measurable, then g dµ is called the Lebesgue–Stieltjes integral ofg; and if F is the generalized df corresponding to µ, then we also use the notation g dF . b Also, a g dF ≡ (a,b ] g dF = 1(a,b ] g dF . Theorem 3.1 (Equality of LS and RS integrals) Let g be continuous on b [a, b]. Then the Lebesgue–Stieltjes integral a g dF equals the Riemann–Stieltjes integral. [And since the LS-integral equals the RS-integral, we can continue to evaluate most LS-integrals using the methods learned in elementary calculus.] Proof. We ﬁrst recall the classical setup associated with the deﬁnition of the RS-integral. Consider any sequence of partitions a ≡ xn0 < · · · < xnn ≡ b such that the partition Xn ≡ {xn0 , xn1 , . . . , xnn } is a reﬁnement of Xn−1 in the sense that Xn−1 ⊂ Xn . Then if meshn ≡ max1≤k≤n (xnk − xn,k−1 ) → 0, and if x∗nk ’s are such that xn,k−1 < x∗nk ≤ xnk , we have (letting gn (a) be deﬁned by right continuity) n (a) gn ≡ k=1 g(x∗nk ) 1(xn,k−1 ,xnk ] → g uniformly on [a, b ], since g is uniformly continuous. Thus for all such sequences the LS-integral of section 3.1 satisﬁes b b g dF = lim a gn dµ by the DCT, bounded by a constant a n n = lim 1 g(x∗nk ) µ(xn,k−1 , xnk ] = lim 1 g(x∗nk ) F (xn,k−1 , xnk ] (b)

= lim {a Riemann–Stieltjes sum for the integral of g}

(c)

= {the Riemann–Stieltjes integral of g},

and this holds for all partitions and x∗nk ’s as above, provided only that meshn → 0. b Thus the LS-integral a g dF also equals the RS-integral. 2 Exercise 3.1

We state a few additional facts here, just for completeness:

(1)

g is RS-integrable with respect to F g is continuous a.e. µF (·).

if and only if

(2)

If g is RS-integrable with respect to F, b then the RS and LS-integrals a g dF are equal.

Let D(F ) and D(g) denote the discontinuity sets of F and g. Then (3)

g is not RS-integrable when D(F ) ∩ D(g) = ∅.

Exercise 3.2 Suppose that the improper RS-integral of a continuous function b g on R, deﬁned by RS( g dF ) ≡ lima→−∞,b→∞ (RS a g dF ) exists ﬁnitely. Then b lima→−∞,b→∞ (LS a |g| dF ) need not be ﬁnite. Thus the fact that an improper RS-integral exists does not imply that the function is LS-integrable. Construct an example. 2

3.

EVALUATING AND DIFFERENTIATING INTEGRALS

45

Exercise 3.3 (Diﬀerentiation under the integral sign) Suppose that the function X(t, ·) is a measurable function, for each t ∈ [a, b ]. Suppose also that for ∂ a.e. ω the partial derivative ∂t X(t, ω) exists for all t in the nondegenerate interval [a, b], and that ∂ for all t ∈ [a, b ], where Y ∈ L1 . X(t, ω) ≤ Y (ω) ∂t Then (using one-sided derivatives at the end points) d ∂ (4) X(t, ω) dµ(ω) = X(t, ω) dµ(ω) dt Ω Ω ∂t

for all t ∈ [a, b ].

46

4

CHAPTER 3. INTEGRATION

Inequalities

Convexity We begin by brieﬂy reviewing convexity. A real-valued function f deﬁned on some interval I of real numbers is convex if (1)

f (αx + (1 − α)y) ≤ α f (x) + (1 − α) f (y) for all x, y in I and all 0 ≤ α ≤ 1.

If f is convex on an interval, then f is continuous on the interior of the interval. Also, the left and right derivatives exist and satisfy D− (x) ≤ D+ (x) at each point in the interior of the interval. The following is useful. Convexity on the interval I holds if and only if (2)

f ((x + y)/2) ≤ [f (x) + f (y)]/2 for all x, y in I, provided that f is also assumed to be bounded on I.

[There exist functions satisfying the inequality in (2) that are not continuous, but they are unbounded in every ﬁnite interval. Thus requiring (1) for all 0 ≤ α ≤ 1 is strictly stronger then requiring it to hold only for α = 12 .] We need a simple test for convexity, and so note that f is convex if (3)

f (x) is for all x

or

f (x) ≥ 0 for all x .

We call f strictly convex if strict inequality holds in any of the above. If f is convex, then there exists a linear function l such that f (x) ≥ l(x) with equality at any prespeciﬁed x0 in the interior of the domain I of f ; this function is called the supporting hyperplane. 2 Deﬁnition 4.1 (Moments) [The following deﬁnitions make sense on a general measure space (Ω, A, µ), and are standard notation on aprobability space (Ω, A, P ).] Recall from (3.1.6) that Eh(X) = h(X(ω)) dµ(ω) = h(X) dµ = h(X). Let (4)

µ ≡ EX ≡ (the mean of X),

(Note two diﬀerent uses of µ)

σ2 ≡ Var[X] ≡ E(X − µ)2 = (the variance of X), (5) σ ≡ StDev[X] ≡ (the standard deviation of X) . We will write X ∼ = (µ, σ2 ) if EX = µ and Var[X] = σ2 < ∞. We will also write 2 X∼ = F (µ, σ ) if X also has df F (·). Now let (6)

EX k ≡ (kth moment of X),

for k ≥ 1 an integer,

(7)

E|X|r ≡ (rth absolute moment of X),

for r > 0,

(8)

Xr ≡ {E|X|r }1/r ≡ (rth norm of X),

for r > 0,

(9)

µk ≡ E(X − µ)k ≡ (kth central moment of X),

for k ≥ 1,

(10)

Cov[X, Y ] ≡ E[(X − µX )(Y − µY )] = (the covariance of X and Y ).

Note that Cov[X, X] = Var[X]. (We only need the case when µ(Ω) = 1.)

4.

INEQUALITIES

47

Proposition 4.1 (Ls ⊂ Lr ) We require µ(Ω) < ∞. Then Ls ⊂ Lr whenever 0 < r < s. [So if E|X|r < ∞, then E|X|r and EX k are ﬁnite for all 0 ≤ r , k ≤ r.]

Proof. Now, |x|r ≤ 1 + |x|r ; and integrability is equivalent to absolute integrability. Note that µ(Ω) < ∞ was used to claim 1 ∈ L1 . 2 Proposition 4.2 Let µ(Ω) < ∞. Then σ2 < ∞ holds if and only if EX 2 < ∞. In this case σ2 = EX 2 − µ2 . Proof. Suppose EX 2 < ∞. Then EX 2 − µ2 = E(X 2 ) − E(2µX) − E(µ2 ) = 2 E(X − µ) = Var[X]. Note that proposition 4.1 was used for EX. Thus µ(Ω) < ∞ was used. Suppose that σ2 < ∞. Then E{(X − µ)2 + 2µ(X − µ) + µ2 } = EX 2 . 2

Inequality 4.1 (Cr -inequality) E|X + Y |r ≤ Cr E|X|r + Cr E|Y |r , where Cr = 2r−1 for r ≥ 1 and Cr = 1 for 0 < r ≤ 1. Proof. There are no restrictions on µ. Case 1. r > 1: Then |x|r is a convex function of x, since its derivative is ↑ . Thus |(x + y)/2|r ≤ [|x|r + |y|r ]/2; and now take expectations. Case 2. 0 < r ≤ 1: Now, |x|r is concave and for x ≥ 0; just examine derivatives. Thus |x + y|r − |x|r ≤ |y|r , and now take expectations. 2 Inequality 4.2 (H¨ older’s inequality) E|X Y | ≤ E1/r |X|r E1/s |Y |s for r > 1, where s is deﬁned by 1/r + 1/s = 1. Alternatively, X Y 1 ≤ Xr × Y s . When expectations are ﬁnite we have equality if and only if there exist constants A and B (not both 0) for which A |X|r = B |Y |s a.e. Proof. The result is trivial if E|X|r = 0 or ∞. Likewise for E|Y |s . So suppose that both expectations are in (0, ∞). Since f (x) = ex is convex, it satisﬁes (1) with α ≡ 1/r and 1 − α = 1/s, x ≡ r log |a| and y ≡ s log |b| for some a and b; thus (1) becomes (11)

exp( 1r x +

1 s

y) ≤

1 r

ex +

1 s

ey ,

or

Young’s inequality (12)

|ab| ≤

|a|r |b|s + r s

for all a, b

with equality only when |b| = |a|1/(s−1) .

Now let a = |X|/Xr and b = |Y |/Y s , and take expectations. Equality holds if and only if |Y |/Y s = (|X|/Xr )1/(1−s) a.e. (that is, all mass is located at equality in (a)) if and only if s/s−1 |Y |s |X| |X|r (a) = = a.e. E|Y |s Xr E|X|r

48

CHAPTER 3. INTEGRATION

Equality extends to the initially excluded cases also if and only if there exist A and B (not both 0) such that A |X|r = B |Y |s a.e. This also gives the next inequality as an immediate corollary. 2 Exercise 4.1 (Convexity inequality) (13)

α

u v

1−α

≤ α u + (1 − α)

Show that

for all 0 < α < 1 and all u, v ≥ 0.

Use this to reprove H¨ older’s inequality. Inequality 4.3 (Cauchy–Schwarz inequality) ( E|XY | )2 ≤ EX 2 EY 2 . 2 2 If both √ EX and EY √ take values in (0, ∞), then equality holds if and only if |X| / EX 2 = |Y | / EY 2 a.e. Example 4.1 (Correlation inequality) For rvs X and Y having positive and ﬁnite variances, it holds that (14)

−1 ≤ ρ ≤ 1,

for the correlation ρ of X and Y deﬁned by (15)

Cov[X, Y ] ρ ≡ ρX,Y ≡ Corr [X, Y ] ≡ . Var[X] Var[Y ]

2

Exercise 4.2 Consider rvs X and Y . Show that ρ = +1 if and only if X − µX = a (Y − µY ) for some a > 0, ρ = −1 if and only if X − µX = a (Y − µY ) for some a < 0. Thus ρ measures linear dependence, not general dependence. Inequality 4.4 (Liapunov’s inequality) (16)

E1/r |X|r

Let µ(Ω) < ∞. Then

is in r for all r ≥ 0.

Proof. Without loss of generality, set µ(Ω) = 1. Suppose E|X|b < ∞. We use µ(Ω) < ∞ via proposition 4.1 for existence of the [0, b] moments. Let 0 ≤ a ≤ b. Apply H¨ older’s inequality with 1/r, 1/s, X, Y replaced by a/b, (b − a)/b, |X|a , 1 to obtain E( |X|a · 1) ≤ ( E|X|b )a/b ( E 1)(b−a)/b . This gives ( E|X|a )1/a ≤ ( E|X|b )1/b for 0 ≤ a ≤ b.

2

Exercise 4.3 (Littlewood’s inequality) Let mr ≡ E|X|r denote the rth absolute moment. Then for r ≥ s ≥ t ≥ 0 we have ms−t mr−s ≥ msr−t . In particular, t r (17)

m4 m21 ≥ m32 .

4.

INEQUALITIES

49

Inequality 4.5 (Minkowski’s inequality) for all r ≥ 1.

E1/r |X + Y |r ≤ E1/r |X|r + E1/r |Y |r

Proof. This is trivial for r = 1. Suppose r > 1, and note that s = r/(r − 1). Then for any measure µ we have E{|X + Y |r } ≤ E{|X| |X + Y |r−1 } + E{|Y | |X + Y |r−1 }

(a)

≤ ( Xr + Y r ) |X + Y |r−1 s

by H¨ older’s inequality

= (Xr + Y r ) E1/s |X + Y |(r−1)s = (Xr + Y r ) E1/s |X + Y |r . If E|X + Y |r = 0, the result is trivial. If not, we divide to get the result. Inequality 4.6 (Basic inequality) all measurable X we have (18) Proof. (a) (b)

2

Let g ≥ 0 be on [0, ∞) and even. Then for

µ( |X| ≥ λ ) ≤ E g(X)/g(λ)

for all λ > 0 .

Now,

E g(X) = [ |X|≥λ] g(X) dµ + [ |X|<λ] g(X) dµ ≥ [ |X|≥λ] g(X) dµ ≥ g(λ) [ |X|≥λ] 1 dµ = g(λ) µ( |X| ≥ λ ). 2

The next two inequalities are immediate corollaries. Inequality 4.7 (Markov’s inequality)

µ(|X| ≥ λ) ≤ E|X|r /λr for all λ > 0.

Inequality 4.8 (Chebyshev’s inequality) (19)

µ( |X − µ| ≥ λ) ≤ Var[X]/λ2

for all λ > 0 . Any rv X ≥ 0 with ﬁnite mean satisﬁes

Inequality 4.9 (Paley–Zygmund) (20) Proof.

P (X > λ) ≥ [(EX − λ) ] /EX + 2

2

for each λ > 0 .

Now, EX = E(X 1[X≤λ]) + E(X 1[X>λ]) ≤ λ +

E(X 2 ) P (X > λ)

by Cauchy–Schwarz. Rearranging gives the inequality.

2

Inequality 4.10 (Jensen’s inequality) Suppose g is convex on (a, b), where −∞ ≤ a < b ≤ ∞. Then if µ(X ∈ (a, b)) = µ(Ω) = 1 and if EX is ﬁnite (and hence a < EX < b), then (21)

g( EX ) ≤ Eg(X) .

For strictly convex g we have equality if and only if X = EX a.e.

50 Proof. (a)

CHAPTER 3. INTEGRATION Let l(·) be a supporting hyperplane to g(·) at EX. Then Eg(X) ≥ El(X)

(b)

= l(EX)

since l(·) is linear and µ(Ω) = 1

(c)

= g(EX)

since g(·) = l(·) at EX .

Now g(X) − l(X) ≥ 0. Thus Eg(X) = El(X) if and only if g(X) = l(X) a.e. µ if and only if X = EX a.e. µ . 2

Inequality 4.11 (Bonferroni) n n n i =j P (Ai Aj ) i=1 P (Ai ) ≥ P (∪i=1 Ai ) ≥ i=1 P (Ai ) − for any collection of events Ak on a probability space (Ω, A, P ). ˜ equal Exercise 4.4 Suppose E|X| < ∞, and let µ ≡ EX. Fix c < d. Let X ˜ Show that c, X, d according as [X < c], [c ≤ X ≤ d], [d < X], and set µ ˜ ≡ EX. ˜ −µ E|X ˜|r ≤ E|X − µ|r for all r ≥ 1. Exercise 4.5 (Hardy) Suppose h ∈ L2 (0, 1) and deﬁne T h(u) = u−1 for 0 < u ≤ 1. Use the Cauchy–Schwarz inequality to show that 1 1 {T h(u)}2 du ≤ 4 0 h2 (u) du. 0 u [Hint: Write T h(u) = u−1 0 h(s)sα s−α ds for some α.]

u 0

h(s) ds

Exercise 4.6 (Wellner) Suppose that T ∼ = Binomial(n, p). [Thus P (T = k) = n k n−k p (1 − p) for 0 ≤ k ≤ n, and the measure associated with T has mean np k and variance np(1 − p).] Then use inequality 4.6 with g(x) = exp(rx) and r > 0, to show that (22)

P (T /n ≥ p ) ≤ exp(−np h( )),

where h( ) ≡ (log( ) − 1) + 1. Exercise 4.7 (Geometric mean) whenever all xk ≥ 0.

Show that (x1 ×· · ·×xn )1/n ≤ (x1 +· · ·+xn )/n

5.

MODES OF CONVERGENCE

51

5

Modes of Convergence

Deﬁnition 5.1 (Modes of convergence) Let X and Xn ’s be measurable and a.e. ﬁnite from the measure space (Ω, A, µ) to (R, B). (a) Recall that Xn converges a.e. to X (denoted by Xn →a.e. X) if (1)

Xn (ω) → X(ω) for all ω ∈ A,

where µ(Ac ) = 0.

(b) Also, recall that Xn converges in measure to X (denoted by Xn →µ X) if (2)

µ( [ ω : |Xn (ω) − X(ω) | ≥ ] ) → 0

for each > 0.

(c) Now (rigorously for the ﬁrst time), Xn converges in rth mean to X (denoted by Xn →r X or Xn →Lr X) if (3)

E|Xn − X|r → 0

for Xn ’s and X in Lr ;

here, r > 0 is ﬁxed. [Note from the Cr -inequality that if Xn − X and one of X or Xn is in Lr , then the other of Xn or X is also in Lr .] Recall from chapter 2 that Xn →a.e. (some a.e. ﬁnite X) holds if and only if Xn − Xm →a.e. 0 as m ∧ n → ∞ . Likewise, in chapter 2 we had Xn →µ (some X) if and only if µ( |Xn − Xm | ≥ ) → 0 as m ∧ n → ∞ for all > 0. Exercise 5.1 (Completeness of Lr ) (a)

(I) Let Xn ’s be in any Lr , for r > 0.

(Riesz–Fischer) Xn →r (some X ∈ Lr ) if and only if Xn − Xm →r 0.

That is, Lr is complete with respect to →r . Prove (a), using (2.3.16). Then show that (Lr , · r ) is a complete and separable metric space (provided that we identify X and X whenever X = X a.e.). (II) Let µ(Ω) < ∞. Then: (b)

If Xn →r X, then Xn →r X for all 0 < r ≤ r.

(c)

Show by example that Xn →r X does not imply that Xn →a.e. X.

(d)

Show by example that Xn →a.e. X does not imply that Xn →L1 X.

[Hint: Use Fatou’s lemma in (a) and H¨ older’s inequality in (b).] Summary Let X and Xn ’s be as in deﬁnition 5.1. Then

(4)

Xn converges a.e., in measure, or in Lr to some X if and only if Xn is Cauchy a.e., in measure, or in Lr .

52

CHAPTER 3. INTEGRATION

Consequences of Convergence in Distribution on (Ω, A, P ) Notation 5.1 Suppose now that µ really denotes a probability measure, and so we will label it P . Recall that Xn converges in distribution to X (denoted by Xn →d X, Fn →d F or L(Xn ) → L(X) with L(·) referring to “law”) when the dfs F and Fn of X and Xn satisfy (5)

Fn (x) → F (x)

as n → ∞

for each continuity point x ∈ CF of F (·) .

[Note that Fn ≡ 1[1/n,∞) →d F ≡ 1[0,∞) , even though Fn (0) = 0 → 1 = F (0).] The statement →d will carry with it the implication that F corresponds to a probability measure P , which can be viewed as the PX = µX of an appropriate rv X. Theorem 5.1 (Helly–Bray) Consider some (Ω, A, P ). Suppose Fn →d F , and suppose that g is bounded and is continuous a.s. F . Then (6) E g(Xn ) = g dFn → g dF = E g(X) . Conversely, Eg(Xn ) → Eg(X) for all bounded, continuous g implies that Fn →d F . Theorem 5.2 (Mann–Wald) Consider some (Ω, A, P ). Suppose Xn →d X, and let g be continuous a.s. F . Then g(Xn ) →d g(X). Proof. We really leave this proof to the next exercise, but we also give a “look-ahead” proof here. (See theorem 7.3.2 below.) Skorokhod theorem If Xn →d X, there are Y and Yn on some (Ω, A, P ) having (7)

Yn ∼ = Xn and Y ∼ = X and especially

Yn → Y a.s. PX (·) .

Note that A1 ≡ {ω : Yn (ω) → Y (ω)} has P (A1 ) = 1. Also, (a)

P (A2 ) ≡ P ({ω : g is continuous at Y (ω)})

(b)

= PY ({y : g is continuous at y} ) = 1.

Thus A ≡ A1 ∩ A2 has P (A) = 1. Especially, (c)

g(Yn (ω)) → g(Y (ω))

for all ω ∈ A,

with P (A) = 1 .

Since g is bounded, applying the DCT to (7) gives the Helly–Bray claim that (d) g dFn = g(Yn ) dP → g(Y ) dP = g dF. We note additionally that since (7) implies g(Yn ) →a.s. g(Y ), it also implies g(Yn ) →d g(Y ). Since g(Xn ) ∼ = g(Yn ) and g(X) ∼ = g(Y ), we can conclude that g(Xn ) →d g(X). This argument did not use the boundedness of g, and so proves the Mann–Wald theorem. [The Helly–Bray theorem will be used later in this section (in proving Vitali’s theorem). Proving it as indicated in the next exercise would have been possible now, but the proof based on Skorokhod’s theorem is more in keeping with the spirit of this book.]

5.

MODES OF CONVERGENCE

53

Consider the converse. Let g (·) equal 1, be linear, equal 0 on (−∞, x − ], on [x − , x], on [x, ∞); and let h (·) equal 1, be linear, equal 0 on (−∞, x], on [x, x + ], on [x + , ∞). Then F (x − ) ≤ E g (X) = lim E g (Xn ) ≤ lim Fn (x) ≤ lim Fn (x) ≤ lim E h (Xn ) = lim E h (X) ≤ F (x + ),

(e)

so that Fn (x) → F (x) at all continuity points of F . Deﬁnition 5.2 (Limit-determining class) continuous functions g on the line. If Eg(Xn ) → Eg(X) for all g ∈ G

2

Let G denote a class of bounded Xn →d X ,

implies

then call G a limit-determining class. [The proof of the converse half of Helly–Bray exhibited one such class of particularly simple functions. (See section 11.8.)] Exercise 5.2 Prove the Mann–Wald and Helly–Bray theorems for all bounded continuous g, without appeal to theorem 7.3.2 of Skorokhod. [Truncate the real line at large continuity points ±M of F , and then use the uniform continuity of g on the interval [−M, M ] to obtain a simple proof in this special case of bounded and continuous functions g.]

General Moment Convergence on (Ω, A, µ) Theorem 5.3 (Moment convergence under →r )

E|Xn |r → E|X|r

(8) Proof.

and EXnk → EX k

for all 0 ≤ k, r ≤ r.

Let (Ω, A, µ) be arbitrary and 0 < r < 1. The Cr -inequality gives E|Xn |r ≤ E|Xn − X|r + E|X|r

(a)

If Xn →r X, then

and E|X|r ≤ E|X − Xn |r + E|Xn |r ,

so that | E|Xn |r − E|X|r | ≤ E|Xn − X|r → 0

(9)

when 0 < r ≤ 1.

Suppose r ≥ 1. Then using Minkowski’s inequality twice (as in (a)) gives 1/r (10) when r ≥ 1. E |Xn |r − E1/r |X|r ≤ E1/r |Xn − X|r → 0, Combining (9) and (10) shows that E|Xn |r → E|X|r . Apply exercise 5.1(b) for r . Now, |Xn+ −X + | equals |Xn −X|, |Xn −0|, |0−X|, |0−0| just as [Xn ≥ 0, X ≥ 0] [Xn ≥ 0, X < 0], [Xn < 0, X ≥ 0], [Xn < 0, X < 0]. Thus (11)

|Xn+ − X + | ≤ |Xn − X|,

and

|Xn− − X − | ≤ |Xn − X|

also. Hence Xn+ →r X + and Xn− →r X − , so that E(Xn+ )r → E(X + )r and E(Xn− )r → E(X − )r . Cross-product terms are 0, since X + (ω)X − (ω) = 0, so (b)

EXnk = E(Xn+ )k + (−1)k E(Xn− )k → E(X + )k + (−1)k E(X − )k = E(X k ) . 2

54

CHAPTER 3. INTEGRATION

Uniform Integrability and Vitali’s Theorem Deﬁnition 5.3 (Uniformly integrable) A collection of measurable Xt ’s is called integrable if sup t E|Xt | < ∞. Further, a collection of rvs {Xt : t ∈ T } is said to be uniformly integrable (and is abbreviated u.i.) if (12)

sup E[ |Xt | × 1[ |Xt |≥λ ] ] → 0

as λ → ∞ .

t∈T

Remark 5.1 Suppose that |Xt | ≤ Y a.s. for some Y ∈ L1 . Then for some null set Nt , we have [|Xt | ≥ λ] ⊂ [|Y | ≥ λ] ∪ Nt , and µ(|Xt | ≥ λ) ≤ µ(|Y | ≥ λ) → 0 uniformly in t as λ → ∞ (use Markov’s inequality). Also, supt∈T E|Xt | ≤ E|Y |. Finally, for each ﬁxed t, |Xt | dµ ≤ [ |Y |≥λ ] Y dµ → 0 uniformly in t as λ → ∞ [ |Xt |≥λ ] by the absolute continuity of the integral of Y in theorem 3.2.5 and by Markov’s inequality. Thus, (13)

if |Xt | ≤ Y for some Y ∈ L1 ,

then the Xt ’s are uniformly integrable .

Exercise probability measure P . (a) Let X ≥ 0 have df F . Then ∞ 5.3 Consider a∞ EX = 0 P (X ≥ y) dy = 0 [1 − F (y)] dy will be shown to hold in (7.4.11) below. For now, we shall just use this fact. (The claimed formula can also be established for simple functions by summing by parts; and then the full claim follows from the MCT. This cumbersome proof is possible now. It constitutes this exercise.) (b) Use the result of (a) to show that for X ≥ 0 and λ ≥ 0 we have ∞ X dP = λ P (X ≥ λ) + λ P (X ≥ y) dy. [X≥λ] (c) Suppose there is a Y ∈ L1 such that P ( |Xn | ≥ y) ≤ P ( Y ≥ y) for all y > 0 and all n ≥ 1. Then use (b) to show that {Xn : n ≥ 1} is uniformly integrable. Exercise 5.4 (Uniform integrability criterion) If supt E|Xt |r < ∞ for some r > 1, then the Xt ’s are uniformly integrable. (Compare this to theorem 5.6 of de la Vall´ee Poussin below, by letting G(x) = xr .) Theorem 5.4 (Uniform absolute continuity of integrals) Let µ(Ω) < ∞. A family of measurable Xt ’s is uniformly integrable if and only if both (14)

sup E|Xt | < ∞

(the collection is integrable)

t

(15)

µ(A) < δ

implies

(a)

|Xt | dµ <

sup t

Proof.

and

(uniform absolute continuity).

A

Suppose (12) holds. Then for λ large enough, (12) gives E|Xt | ≤ λ µ(Ω) + [ |Xt |≥λ ] |Xt | dµ ≤ λ µ(Ω) + 1

for all t;

5.

MODES OF CONVERGENCE

55

thus the collection is integrable. Suppose µ(A) < δ. Then A |Xt | dµ = A |Xt | × 1[ |Xt |<λ ] dµ + A |Xt | × 1[ |Xt |≥λ ] dµ (b) ≤ λ × µ(A) + |Xt | 1[ |Xt |≥λ ] dµ ≤ /2 + /2 =

for a suﬃciently large ﬁxed λ and for δ ≤ /(2λ). Thus (15) holds. Suppose the conditions hold. Then Markov’s inequality gives (c)

µ( |Xt | ≥ λ) ≤ E|Xt | /λ ≤ supt E|Xt | /λ < δ

for λ large enough. Thus (15) applied to the sets [ |Xt | ≥ λ ] yields (d) supt |Xt | × 1[ |Xt |≥λ ] dµ ≤ , so that (12) holds.

2

Theorem 5.5 (Vitali) Let r > 0. Suppose that Xn ’s ∈ Lr satisfy Xn →µ X. The following are equivalent when µ(Ω) < ∞: (16)

{|Xn |r : n ≥ 1} are uniformly integrable rvs.

(17)

Xn →r X.

(18)

E|Xn |r → E|X|r .

(19)

lim E|Xn |r ≤ E|X|r < ∞.

Exercise 5.5 (a) We need only omit “(16) implies (17)” in Vitali’s theorem when we instead have µ(Ω) = ∞. (b) Give an example to demonstrate the previous claim. (c) Show that t ∈ [0, ∞) may replace n ∈ {0, 1, 2, . . .} in Vitali’s theorem. Remark 5.2 (Vitali’s theorem in L1 ) Let r = 1, µ(Ω) < ∞, and Xn →µ X. Then (16) – (19) are equivalent by Vitali’s theorem. As a preliminary (from Fatou’s lemma and Xn →a.e. X on some subsequence n ), (20)

E|X| = E| lim Xn | = E lim |Xn | ≤ lim E|Xn | ≤ lim E|Xn | ≤ lim E|Xn |

always holds. Thus lim sup E|Xn | ≤ E|X| < ∞ implies that E|Xn | → E|X| < ∞. Hence E|Xn − X| → 0

if and only if

E|Xn | → E|X|

if and only if

(21) {|Xn | : n ≥ 1} are uniformly integrable rvs.

2

Corollary 1 (L1 -convergence) Let X and Xn ’s be measurable and a.e. ﬁnite. Let µ(Ω) < ∞. Then Xn →1 X if and only if both Xn →µ X and {|Xn | : n ≥ 1} are uniformly integrable rvs.

56

CHAPTER 3. INTEGRATION

Proof. Suppose (16) holds. Now, Xn →a.s. X for some subsequence by theorem 2.3.1. Thus E|X|r = E(lim inf |Xn |r ) ≤ lim inf E|Xn |r < ∞ using Fatou’s lemma and then (14). Thus X ∈ Lr . Now, the Cr -inequality gives |Xn − X|r ≤ Cr {|Xn |r + |X|r }, so that the rvs |Xn − X|r are clearly uniformly integrable. Thus E|Xn − X|r = E{|Xn − X|r × 1[ |Xn −X|> ] } + E{|Xn − X|r × 1[ |Xn −X|≤ ] } ≤ + r × µ(Ω)

(a)

for n large enough;

the in (a) is from (15), since µ(|Xn − X| ≥ ) → 0 by hypothesis. Thus (17) holds. Now, (17) implies (18) by theorem 5.3. Also (18) trivially implies (19). Suppose (19) holds. Deﬁne fλ to be a continuous function on [0, ∞) that equals |x|r , 0, or is linear, according as |x|r ≤ λ, |x|r ≥ λ + 1, or λ ≤ |x|r ≤ λ + 1. Then (graphing fλ (x) and xr on [0, λ + 1]) limn [ |Xn |r >λ+1] |Xn |r dµ = limn { |Xn |r dµ − [ |Xn |r ≤λ+1] |Xn |r dµ} (b) ≤ E|X|r − limn [ |Xn |r ≤λ+1] |Xn |r dµ by (19) ≤ E|X|r − lim E fλ (Xn )

(d)

= E|X|r − E fλ (X) ≤ [ |X|≥λ ] |X|r dµ

(e)

→0

(c)

implying (16).

by the Helly–Bray theorem

as λ → ∞ since X ∈ Lr , 2

We now generalize exercise 5.4. In fact, we are able to obtain a characterization of uniform integrability. Theorem 5.6 (de la Vall´ee Poussin) Let µ(Ω) < ∞. A family of L1 -integrable functions Xt is uniformly integrable if and only if there exists a convex function G on [0, ∞) for which G(0) = 0, G(x)/x → ∞ as x → ∞ and (22) Proof. (a)

supt E G(|Xt |) < ∞ .

For λ so large that G(x)/x ≥ c for all x ≥ λ we have |Xt | dµ ≤ 1c [ |Xt |≥λ ] G(|Xt | ) dµ ≤ 1c supt EG(|Xt | ) <

[ |Xt |≥λ ]

for c suﬃciently large. Thus (22) implies {Xt : t ∈ T } is uniformly integrable. Now we show that {Xt : t ∈ T } u.i. implies (22) for some G. We deﬁne x G(x) = 0 g(y) dy where (with a sequence bn having b0 = 0, to be speciﬁed below) we set g(x) = bn for all n ≤ x < n + 1, n ≥ 0. Deﬁne an (t) ≡ µ( |Xt | ≥ n). We note that

(b)

E G( |Xt | ) ≤ b1 µ(1 ≤ |Xt | < 2) + (b1 + b2 ) µ(2 ≤ |Xt | < 3) + · · · = ∞ n=1 bn an (t).

5.

MODES OF CONVERGENCE

57

∞ It thus suﬃces to choose bn ∞ such that supt 1 bn an (t) < ∞. By the deﬁnition of uniform integrability, we can choose integers cn ↑ ∞ such that (c) supt [ |Xt |≥cn ] |Xt | dµ ≤ 1/2n . Thus for all t we have ∞ 1/2n ≥ [ |Xt |≥cn ] |Xt | dµ ≥ i=cn i µ(i ≤ |Xt | < i + 1) = ≥ (d)

=

∞ i=cn

∞ j=cn

∞ j=cn

i j=1

∞ i=j

µ(i ≤ |Xt | < i + 1 ) µ(i ≤ |Xt | < i + 1) =

∞ j=cn

µ( |Xt | ≥ j )

aj (t).

Thus, interchanging the order of summation, ∞ ∞ −n (e) 1= ∞ ≥ supt ∞ n=1 2 n=1 j=cn aj (t) = supt j=1 bj aj (t) for bj ≡ (the number of integers n such that cn is ≤ j). As was noted above, the proof is given by (e). 2 Exercise 5.6 (a) Let ξ ∼ = Uniform(0, 1), and let Xn ≡ (n/ log n)1[0,1/n] × (ξ) for n ≥ 3. Show that these Xn are uniformly integrable and Xn dP → 0, even though these rvs are not dominated by any ﬁxed integrable rv Y . (b) Let Yn ≡ n1[0,1/n] (ξ) − n1[1/n,2/n](ξ). Show that these Yn are not uniformly integrable, but that Yn dP → 0.

Summary of Modes of Convergence Results Theorem 5.7 (Convergence implications) be measurable and a.e. ﬁnite. (Note ﬁgure 5.1.)

Let 0 < r ≤ r. Let X and Xn ’s

(i) If Xn →a.e. X and µ(Ω) < ∞, then Xn →µ X. (ii) If Xn →µ X, then Xn →a.e. X on some subsequence n . (iii) If Xn →r X, then Xn →µ X and {|Xn |r : n ≥ 1} are uniformly integrable. (iv) If Xn →µ X and {|Xn |r : n ≥ 1} are uniformly integrable rvs, then Xn →r X. If Xn →µ X and lim E|Xn |r ≤ E|X|r < ∞, then Xn →r X. (v) If Xn →r X and µ(Ω) < ∞, then Xn →r X. (vi) If Xn →p X, then Xn →d X. (vii) If µ(Ω) < ∞, then Xn →µ X if and only if every subsequence {n } contains a further subsequence {n } for which Xn →a.e. X. (viii) If Xn →d X, then Yn →a.e. Y for Skorokhod rvs with Yn ∼ = Xn and Y ∼ = X.

58

CHAPTER 3. INTEGRATION

5

Xn →r X

|Xn|r u.i. 4

Xn →a.e. X

1 if µ(Ω) < ∞

6

if µ(Ω) < ∞

Xn →r′ X for all r′ ≤ r

3

2

X n →µ X

if µ = P

7

Xn →d X

Some Xn′ →a.e. X

if µ(Ω) < ∞

Every n′ contains an n˝ for which Xn˝ → a.e. X

Figure 5.1 Convergence implications.

Proof.

See theorem 2.3.1 for (i) and (ii). Markov’s inequality gives µ( |Xn − X| ≥ ) ≤ E|Xn − X|r / r → 0,

and then Vitali’s theorem gives (iii). Vitali’s theorem also gives (iv). Liapunov’s inequality gives (a)

E|Xn − X|r ≤ {E|Xn − X|r (r/r ) }r /r = {E|Xn − X|r }r /r ,

as claimed in exercise 5.1(b); so (v) holds. Proposition 2.4.1 gives (vi). And then theorem 2.3.1 gives (vii). The Skorokhod construction of theorem 7.3.2 was stated below in (7); and (7) gives (viii). 2

5.

MODES OF CONVERGENCE

59

Approximation of Functions in Lr by Continuous Functions Let Cc denote the class of continuous functions on R that vanish outside a compact (∞) set, and then let Cc denote the subclass that has an inﬁnite number of continuous derivatives. Let Sc denote m the class of all step functions on R, where such a step function is of the form 1 yj 1Ij for disjoint ﬁnite intervals Ij . Further, let F denote a generalized df, and let µ(·) ≡ µF (·) denote the associated Lebesgue–Stieltjes measure. Let X denote a rv on (Ω, A) = (R, B). Theorem 5.8 (The continuous functions are dense in Lr , r ≥ 1) Suppose throughout that X ∈ Lr , for some r ≥ 1. (a) (Continuous functions) Then there exists a bounded and continuous function Y in Cc for which |X − Y |r dµF < , for each > 0. Thus the class Cc is -dense within the class of integrable functions Lr . (∞) (b) We may even insist that the function Y belong to Cc . (c) (Step functions) Such a close approximation may also be found within the step functions Sc , making them -dense also. (d) All this extends to rvs on (Ω, A) = (Rn , Bn ). + Proof. Let r = 1 ﬁrst. We need only consider X ; and we approximate it by a n simple function X = 1 xi 1Ai so closely that |X −X | dµF < /3. For each such xi > 0 we must have µ(Ai ) < ∞, since X dµF < ∞. Now, the approximation lemma of exercise 1.2.3 guarantees sets B1 , . . . , Bn made up of a ﬁnite disjoint union of intervals of the form (a, b] for which

µ(Ai Bi ) < /(3n |xi |), (p)

X ≡

n 1

xi 1Bi satisﬁes

and so

|X − X | dµF < /3.

m Rewrite X = 1 yj 1Cj with C j = (aj , bj ]. Then X is the step function called m for in part (c). Deﬁne δ ≡ /(3 1 yj ). Now approximate 1Cj by the continuous function Yj that equals 1, is linear, equals 0 according as x ∈ [aj , bj ], as x ∈ [aj − δ, aj ]∪ [bj , bj + δ], as x ∈ (−∞, aj − δ] ∪ [bj + δ, ∞). Then let Y ≡ m 1 Yj , which has |X − Y | dµF < /3. Thus |X − Y | dµF < , as called for in part (a). For (b) we consider the function ψ(x/δ) [where 1 (23)

ψ(x) ≡ x1 0

exp(−1/((s(1 − s))) ds exp(−1/((s(1 − s))) ds

for 0 ≤ x ≤ 1,

with ψ(x) equal to 1 or 0 according as x ≤ 0 or x ≥ 1] is easily seen to have an inﬁnite number of continuous derivatives on R. Use it on [aj − δ, aj ] and [bj , bj + δ] (∞) to connect 0 to 1, instead of linear connections. The result is a function in Cc . For general r ≥ 1 we need only know the trivial fact that |a − b|r ≤ |ar − br | for all a, b ≥ 0 and r ≥ 1. Just note example 5.1.1(d) below for part (d). 2

60

CHAPTER 3. INTEGRATION

Miscellaneous Results Exercise e’s theorem) (a) If f0 , f1 , f2 , . . . are ≥ 0 on (Ω, A, µ) and 5.7 (Scheﬀ´ satisfy Ω fn dµ = 1, and fn →a.e. f0 with respect to µ, then (24) f0 dµ → 0 as n → ∞. sup fn dµ − A∈A

A

A

[The name derives from pointing out its usefulness in probability theory.] (We can think of this conclusion as the uniform convergence of measures.) (b) In fact, we require only that Ω fn dµ → Ω f0 dµ = 1 and fn →µ f0 . [Hint. Note Vitali’s theorem.] Exercise 5.8 (→a.u. ) (i) We deﬁne Xn →a.u. X (which is an abbreviation for almost uniform convergence) to mean that for all > 0 there exists an A with µ(A ) < such that Xn →uniformly X on Ac . (ii) (Egorov) If µ(Ω) < ∞ and Xn →a.s. X, then Xn →a.u. X. If |Xn | ≤ Y a.s. for all n where Y ∈ Lr with r > 1, then we may omit the hypothesis µ(Ω) < ∞. (iii) (a) If Xn →a.u. X, then both Xn →a.e. X and Xn →µ X. (b) If Xn →µ X, then Xn →a.u. X on some subsequence n . Exercise 5.9 (Lusin) Let X be measurable on (Ω, A, µ), where Ω is a locally compact Hausdorﬀ space [every point has a neighborhood whose closure is compact, such as the real line R with the usual Euclidean metric] and A = σ[{open sets}]. Suppose X(ω) = 0 for all ω ∈ A, where µ(A) < ∞. Let > 0. Then there exists Y , where Y (ω) = 0 for all ω ∈ B, with the set B compact, and where Y is continuous, sup |Y | ≤ sup |X|, and µ({ω : X(ω) = Y (ω)}) < . [That is, a measurable function is “almost equal” to a continuous function.] Exercise 5.10 (Lo`eve) Suppose X1 , X2 , . . . are integrable on (Ω, A, µ). Deﬁne φn (A) ≡ A Xn for all A ∈ A, and suppose φn (A) converges to a ﬁnite number for all A ∈ A. Deﬁne |φ|n (A) ≡ A |Xn |. Then supn |φ|n (Ω) < ∞. Moreover, sup |φ|n (A) → 0 n

as either µ(A) → 0 or A 0.

Finally, there exists an integrable function X (that is unique a.e. µ) for which φn (A) → φ(A) for all A ∈ A, with φ(A) ≡ A X. (Relate this to the absolute continuity of measures introduced below.)

Chapter 4

Derivatives via Signed Measures 1

Decomposition of Signed Measures

Deﬁnition 1.1 (Signed measure) A signed measure on a σ-ﬁeld ﬁeld) A (or a is a set function φ : A → (−∞, +∞] for which φ(∅) = 0 and φ( An ) = φ(An ) for all countable disjoint sequences of An ’s in A (that have An in A ). When additivity is required only on ﬁnite unions, then φ is called a ﬁnitely additive (f.a.) signed measure. (If φ≥ 0, it is a measure or f.a. measure.) If |φ(Ω)| < ∞, then φ is ∞ called ﬁnite. If Ω = 1 Ωn with all Ωn ∈ A and all |φ(Ωn )| < ∞, then φ is called σ-ﬁnite. Proposition 1.1 (Elementary properties) (i) If φ(A) is ﬁnite and B ⊂ A, then φ(B) is ﬁnite. ∞ Thus φ(Ω) ﬁniteis∞equivalent to φ(A) being ﬁnite for all A ∈ A. (ii) If |φ( 1 An )| < ∞, then 1 φ(An ) is absolutely convergent. Proof. (a)

(i) Now, (a ﬁnite number ) = φ(A) = φ(B) + φ(A \ B)

implies that φ(B) and φ(A \ B) are both ﬁnite numbers. φ(An ) is≥ 0 or < 0. Let A− An or (ii) Let A+ n equal An or n equal ∅ as + + ∅ as + φ(An ) is ≤0 or > 0. Then φ(A ) = φ( A ) < ∞ by (i), since A ⊂ An . n n − n Likewise, φ(A− ) = φ( A ). Now, convergent series of numbers in [0, ∞) may n n be rearranged at will. Thus |φ(An )| = φ(A+ φ(A− 2 n)− n ) is ﬁnite. Example 1.1 (The prototypical example) Let X be measurable. Then (1) φ(A) ≡ X dµ is a signed measure if X − ∈ L1 . A

Note that φ is ﬁnite if X ∈ L1 . Also, φ is σ-ﬁnite if X is a.e. ﬁnite and µ is σ-ﬁnite. 61

62 Proof. (a)

CHAPTER 4. DERIVATIVES VIA SIGNED MEASURES Now, φ(∅) = 1∅ dµ = 0 dµ = 0. Also, φ(A) = A X + dµ − A X − dµ ≥ − A X − dµ > −∞

for all A ∈ A. Finally, ∞ φ( 1 An ) = Σ An X = Σ An X + − Σ An X − + − = by the MCT, twice An X − An X ∞ (b) = ( An X + − An X − ) = 1 φ(An ). An X = Thus φ is a signed measure. Note that|φ(A)| = | A X| ≤ A |X| ≤ |X| < ∞ for all A, if X ∈ L1 . Let Ω ≡ n Ωn be a measurable decomposition for the σ-ﬁnite µ. Then the sets Ωnm ≡ Ωn ∩ [m ≤ X < m + 1] and Ωn,±∞ ≡ Ωn ∩ [x = ±∞], for n ≥ 1 and for all integers m, is a decomposition showing φ to be σ-ﬁnite. 2 Deﬁnition 1.2 (Continuous signed measure) A signed measure φ is continuous from below (above) if φ( lim An ) = lim φ(An ) for all An (for all An , with at least one φ(An ) ﬁnite). We call φ continuous in case it is continuous both from below and from above. Proposition 1.2 (Monotone property and continuity of signed measures) A signed measure on either a ﬁeld or a σ-ﬁeld is ﬁnitely additive and continuous. Conversely, if a ﬁnitely additive signed measure on either a ﬁeld or σ-ﬁeld is either continuous from below or is ﬁnite and continuous from above at ∅, then it is a countably additive signed measure. Proof. This result has nearly the same proof as does the corresponding result for measures; see proposition 1.1.2 and proposition 1.1.3. 2 Theorem 1.1 (Jordan–Hahn) Let φ be a signed measure on (Ω, A). Then Ω can be decomposed as Ω = Ω+ + Ω− , where (2)

Ω+ is a positive set for φ, in that φ(A) ≥ 0 for all A ⊂ Ω+ ,

(3)

Ω− is a negative set for φ, in that φ(A) ≤ 0 for all A ⊂ Ω− .

Moreover, we deﬁne measures on (Ω, A) via (4)

φ+ (A) ≡ φ(A ∩ Ω+ )

and

φ− (A) ≡ −φ(A ∩ Ω− ) ,

with φ+ a measure and φ− a ﬁnite measure on (Ω, A). Of course, φ+ (Ω− ) = 0 and φ− (Ω+ ) = 0. We will call φ+ , φ− , and |φ|(·)| ≡ φ+ + φ− the positive part, the negative part, and the total variation measure associated with φ; thus (5)

|φ(·)| ≡ φ+ (·) + φ− (·)

is the total variation measure on (Ω, A),

Moreover, the following relationships hold: (6)

φ+ (A) ≡ sup{φ(B) : B ⊂ A, B ∈ A} , φ− (A) ≡ − inf{φ(B) : B ⊂ A, B ∈ A}.

1.

DECOMPOSITION OF SIGNED MEASURES

Exercise 1.1

63

If φ and ψ are signed measures, then so is φ + ψ.

Exercise 1.2 Identify φ+ , φ− , |φ|, and |φ(Ω)| in the context of the prototypical situation of example 1.1. Proof. Let B denote some set having φ(B) < 0. [That φ(B) > −∞ is crucial; this proof will not work on the positive side.] (If no such set exists, let Ω+ ≡ Ω, giving |φ| = φ+ = φ and φ− ≡ 0.) We now show that (a)

B contains a negative set C.

If B is a negative set, use it for C. If not, then we will keep removing sets Ak with φ(Ak ) > 0 from B until only a negative set C is left. We will choose sets Ak with φ(Ak ) ≥ 1 as many times as we can, then φ(Ak ) ≥ 12 as many times as we can, . . . . To this end, let n1 ≡ min{i : φ(A1 ) ≥ 1/i for some A1 ⊂ B, with A1 ∈ A}, ··· (b) nk ≡ min{i : φ(Ak ) ≥ 1/i for some Ak ⊂ B \ ∪k−1 i=1 Ai , with Ak ∈ A} ··· . Let C ≡ B \ k Ak , where the union is inﬁnite (unless the process of choosing nk ’s terminates) and where only ﬁnitely many sets Ak exist for each 1/i [else proposition 1.1(i) would be violated]. The c.a. of φ then gives (c) 0 > φ(B) = φ(C) + k φ(Ak ) ≥ φ(C) > −∞. Moreover, C is a negative set, since no subset can have measure exceeding 1/i for any i. Now we know that we have at least one negative set. So we let (d)

d ≡ inf{φ(C) : C is a negative set } < 0,

and deﬁne Ω− ≡ ∪k Ck ,

where Ck denotes a sequence of negative sets for which φ(Ck ) d. Now, Ω− is also a negative set (else one of the Ck ’s would not be), and thus φ(Ω− ) ≥ d, because it must exceed the inﬁmum of such values. But φ(Ω− ) ≤ d also holds, since φ(Ω− ) = φ(Ck ) + φ(Ω− \ Ck ) ≤ φ(Ck ) for all k gives φ(Ω− ) ≤ d. Thus φ(Ω− ) = d. Then Ω+ is a positive set, since if φ(A) < 0 for some A ⊂ Ω+ , then the set Ω− ∪ A would have φ(A ∪ Ω− ) < d (which is a contradiction). 2 + Exercise 1.3 The set Ω+ is essentially unique, in that if Ω+ 1 and Ω2 both satisfy + + the theorem, then |φ|(Ω1 Ω2 ) = 0.

Lebesgue Decomposition Deﬁnition 1.3 (Absolute continuity of measures) Let µ and φ denote a measure and a signed measure on a σ-ﬁeld A. Call φ absolutely continuous with respect to µ, denoted by φ µ, if φ(A) = 0 for each A ∈ A having µ(A) = 0. We say φ is singular with respect to µ, denoted by φ ⊥ µ, if there exists a set N ∈ A for which µ(N ) = 0 while |φ|(N c ) = 0.

64

CHAPTER 4. DERIVATIVES VIA SIGNED MEASURES

Theorem 1.2 (Lebesgue decomposition) Suppose the measure µ and the signed measure φ are σ-ﬁnite on the space (Ω, A). Then there exists a unique decomposition of φ as (7)

φ = φac + φs

where φac µ and φs ⊥ µ,

with φac and φs σ-ﬁnite measures. Moreover, φac (A) = (8) Z0 dµ for all A ∈ A A

for some ﬁnite A-measurable function Z0 , which is unique a.e. µ. Proof. By σ-ﬁniteness and the Jordan–Hahn decomposition, we need only − give the proof if µ and φ are ﬁnite measures; just consider φ+ Ωn and φΩn separately ∞ (n = 1, 2, . . .) of the σ-ﬁnite decomposition Ω = 1 Ωn . (To give the details would be pedantic.) We now establish the existence of the decomposition in the reduced problem when φ and µ are ﬁnite measures. Let (a) Z ≡ {Z : Z ≥ 0, Z ∈ L1 and A Z dµ ≤ φ(A) for all A ∈ A}. Now, Z = ∅, since Z ≡ 0 is in Z. Case 1: φ µ. The ﬁrst step is to observe that (b)

Z1 , Z2 ∈ Z implies Z1 ∨ Z2 ∈ Z.

With A1 ≡ {ω ∈ A : Z1 (ω) > Z2 (ω)} and A2 ≡ AAc1 , we have (c) (Z1 ∨ Z2 ) dµ = A1 Z1 dµ + A2 Z2 dµ ≤ φ(A1 ) + φ(A2 ) = φ(A). A Thus (b) holds. Now choose a sequence Zn ∈ Z such that (d) Z dµ → C ≡ supZ∈Z Ω Z dµ ≤ φ(Ω) < ∞. Ω n Replace Zn by Z1 ∨ · · · ∨ Zn via (b), and thus take the functions Zn in (d) to be an sequence of functions; letZ0 ≡ lim Zn . The MCT then gives (for any A ∈ A, and also for Ω) (e) so that Z0 ∈ Z , A Z0 dµ = lim A Zn dµ ≤ lim φ(A) ≤ φ(A), (f) Z dµ = lim Ω Zn dµ = C, allowing also that Z0 ≥ 0 is ﬁnite. Ω 0 We now deﬁne (g) φac (A) ≡ A Z0 dµ

and φs (A) ≡ φ(A) − φac (A)

for all A ∈ A.

Then φac is a ﬁnite measure, which can be seen by applying example 1.1 with C ﬁnite; and φac µ. Moreover, φs ≡ φ − φac ≥ 0 (since Z0 ∈ Z), so that φs is a ﬁnite measure by exercise 1.1. If φs (Ω) = 0, then φ = φac and we are done, with φs ≡ 0.

1.

DECOMPOSITION OF SIGNED MEASURES

65

Suppose φs (Ω) > 0. Then (since µ(Ω) is ﬁnite) there is some θ > 0 for which (h)

φs (Ω) > θ µ(Ω).

Let Ω+ and Ω− denote the Jordan–Hahn decomposition for φ∗ ≡ φs − θ µ. Then (i)

µ(Ω+ ) > 0;

µ(Ω+ ) = 0 would imply φs (Ω+ ) = 0 [since φs = φ − φac with φ µ assumed for Case 1 and with φac µ obvious from example 4.1.1], which would then contradict (h) via (j)

φs (Ω) − θ µ(Ω) = φ∗ (Ω) = φ∗ (Ω− ) ≤ 0.

Now, φs (AΩ+ ) ≥ θ µ(AΩ+ ) (since φ∗ (Ω+ ) ≥ 0). Thus φ(A) = φac (A) + φs (A) ≥ A Z0 dµ + φs (AΩ+ ) ≥ A Z0 dµ + θ µ(AΩ+ ) (k) ≥ A (Z0 + θ 1Ω+ ) dµ for all A ∈ A. This implies both Zθ ≡ Z0 + θ 1Ω+ ∈ Z and Ω Zθ dµ = C + θ µ(Ω+ ) > C. But this is a contradiction. Thus φs (Ω) = 0. Thus φ equals φac and satisﬁes (8), and the theorem holds in Case 1. The a.s. µ uniqueness of Z0 follows from exercise 3.2.2. (This also establishes the Radon–Nikodym theorem below.) Case 2: General φ. Let ν ≡ φ + µ, and note that both φ ν andµ ν. Then by Case 1 (l) φ(A) = A X dν and µ(A) = A Y dν for all A ∈ A for ﬁnite ν-integrable functions X ≥ 0 and Y ≥ 0 that are unique a.e. ν. Let D ≡ {ω : Y (ω) = 0}, and then Dc = {ω : Y (ω) > 0}. Deﬁne φs (A) ≡ φ(AD) and φac (A) = φ(ADc ). Since µ(D) = D Y dν = D 0 dν = 0 and φs (Dc ) = φ(DDc ) = φ(∅) = 0, we have φs ⊥ µ. If µ(A) = 0, then A Y dν = 0; and thus Y = 0 a.e. ν in A by exercise 1.2.1. But Y > 0 on ADc , and so ν(ADc ) = 0 follows. Then φac (A) = φ(ADc ) = 0, since φ ν. Thus φac µ. Consider the uniqueness of the decomposition. If φ = φac + φs = φ¯ac + φ¯s , then ψ ≡ φac − φ¯ac = φ¯s − φs satisﬁes both ψ ⊥ µ and ψ µ. Thus ψ ≡ 0. 2 (m)

66

2

CHAPTER 4. DERIVATIVES VIA SIGNED MEASURES

The Radon–Nikodym Theorem

Recall that the absolute continuity φ µ means that (1)

φ(A) = 0

µ(A) = 0 with A ∈ A.

whenever

Theorem 2.1 (Radon–Nikodym) Suppose both the signed measure φ and the measure µ are σ-ﬁnite on a measure space (Ω, A, µ). Then φ µ if and only if there exists uniquely a.e. µ a ﬁnite-valued A-measurable function Z0 on Ω for which (2) φ(A) = Z0 dµ for all A ∈ A . A

Moreover, φ is ﬁnite if and only if Z0 is integrable. The function Z0 of (1) is often denoted by very suggestive notation: dφ dµ for all A ∈ A. φ(A) = A dµ

dφ dµ ,

so that we also have the following

The function Z0 is called the Radon–Nikodym derivative of φ with respect to µ. Proof. The Lebesgue decomposition theorem shows that such a Z0 necessarily exists. The suﬃciency is just the trivial example 4.1.1. The “moreover” part is also a trivial result. 2 Theorem 2.2 (Change of variable theorem) Let µ ν with µ a σ-ﬁnite measure on (Ω, A). If X dµ is well-deﬁned, then dµ (3) X dµ = X dν for all A ∈ A , dν A A One useful special case results from b b · (4) f dG = f g dH when G ≡ g dH for a generalized df H, a

a

where we agree that

b a

≡

a

(a,b]

.

Case 1. X = 1B , for B ∈ A: Then the Radon–Nikodym theorem gives dµ dµ (a) A 1B dµ = µ(AB) = AB dν dν = A 1B dν dν . n Case 2. X = 1 ci 1Bi , for a partition Bi : Case 1 and linearity of the integral give Proof.

dµ dµ dν = A X dν . dν dν Case 3. X ≥ 0: Let Xn ≥ 0 be simple functions that to X. Then the MCT twice gives dµ dµ (c) X dµ = lim A Xn dµ = lim A Xn dν = A X dν . A dν dν (b)

A

X dµ =

n

i=1 ci

A 1Bi

dµ =

n 1

ci

A

1Bi

2.

THE RADON–NIKODYM THEOREM

67

Case 4. X measurable and at least one of X + , X − in L1 : Then X dµ = A X + dµ − A X − dµ A dµ dµ dµ dν − A X − dν = A X dν , dν dν dν so long as one of A X + dµ and A X − dµ is ﬁnite. 2 (d)

=

A

X+

Exercise 2.1 Let µ and ν be σ-ﬁnite measures on (Ω, A). Let φ and ψ be σ-ﬁnite signed measures on (Ω, A). Then (5)

dφ dψ d(φ + ψ) = + a.e. µ dµ dµ dµ

if φ µ and ψ µ ,

(6)

dφ dφ dµ = · a.e. ν dν dµ dν

if φ µ and µ ν.

Note that theorem 3.2.6 (of the unconscious statistician) is another change of ¯ A¯ ) and g : (Ω, ¯ A¯ ) → (R, ¯ B¯ ), then variable theorem. That is, if X : (Ω, A) → (Ω, (7) g(X) dµ = g dµX = y dµg(X) (y) for all B ∈ B, (g◦X)−1 (B)

g−1 (B)

B

when one of the these integrals is well-deﬁned. (See also exercise 7.3.3 below.) Exercise 2.2 Let Pµ,σ2 denote the N (µ, σ2 ) distribution. Let P be Cauchy. (a) Show that Pµ,1 P0,1 and compute dPµ,1 /dP0,1 . (b) Show that P0,σ2 P0,1 and compute dP0,σ2 /dP0,1 . (c) Compute dP/dP0,1 and dP0,1 /dP . Exercise 2.3 Flip a coin. If heads results, then X is a Uniform(0, 1) outcome; but if tails results, then X is a Poisson(λ) outcome. The resulting distribution on R is labeled P . (a) Let µ denote Lebesgue measure on R. Find the Lebesgue decomposition of P with respect to this µ; that is, write P = Pac + Ps . (b) Let µ denote counting measure on {0, 1, 2, . . .}. Determine the Lebesgue decomposition of P with respect to this µ. [If need be, see the deﬁnitions of various distributions in chapter 9.] Exercise 2.4 If µ1 , µ2 , . . . are ﬁnite measures on some (Ω, A), then there exists a ﬁnite measure µ on (Ω, A) such that µk µ for each k ≥ 1. Exercise 2.5 Let µ and ν be signed measures on (Ω, A). Show that the following are equivalent: (a) µ ν. (b) µ+ ν and µ− ν. (c) |µ| |ν|. Exercise 2.6 Let µ and ν be ﬁnite measures on (Ω, A). Then µ ν if and only if for every > 0 there exists δ > 0 such that ν(A) < δ implies µ(A) < .

68

CHAPTER 4. DERIVATIVES VIA SIGNED MEASURES

Exercise 2.7 (Halmos) Suppose µ, µ1 , µ2 , . . . and ν, ν1 , ν2 , . . . are ﬁnite measures on some (Ω, A) for which µk νk for each k ≥ 1. Suppose also that n n µ(A) = lim and ν(A) = lim k=1 µk (A) k=1 νk (A) n→∞

n→∞

for all A ∈ A. Show that the following hold a.e. ν : {d nk=1 νk } /{dν} 1 and {d nk=1 µk } /{dν} {dµ /dν} , (8) n n {d k=1 µk } /{d k=1 νk } → {dµ /dν} . These can be thought of as theorems about Radon–Nikodym derivatives, about absolute continuity of measures, or about change of variables. Exercise 2.8 Let A denote the collection of all subsets A of an uncountable set Ω for which either A or Ac is countable. Let µ(A) denote the cardinality of A. Deﬁne φ(A) to equal 0 or ∞ according as A is countable or uncountable. Show that φ µ. Then show that the Radon–Nikodym theorem fails. Exercise 2.9

For a σ-ﬁnite measure µ and a ﬁnite measure ν on (Ω, A), let

φ(A) ≡ µ(A) − ν(A)

for all A ∈ A .

(a) Show that φ is a signed measure. φ(A) = A (f − g) d(µ + ν),

(b) Show that

for measurable functions f and g with g ∈ L+ 1 (µ + ν). (Note example 4.1.1.) (c) Determine φ+ , φ− , and |φ|; and determine |φ|(Ω) in case µ is also a ﬁnite measure. Exercise 2.10 (Total variation distance between probability measures) Deﬁne P and Q to be probability measures on (Ω, A). (a) Show that the total variation distance dT V (P, Q) between P and Q satisﬁes (9) dT V (P, Q) ≡ {supA∈A |P (A) − Q(A)|} = 12 | p − q | dµ for any measure µ dominating both P and Q (that is, P µ and Q µ). (b) Note speciﬁcally that the choice of dominating measure µ does not aﬀect the value of dT V (P, Q). (c) Use the previous exercise to show that dT V (P, Q) = |P − Q| (Ω) /2 . Exercise 2.11 (Hellinger distance between probability measures) Let P and Q denote probability measures on (Ω, A). Deﬁne the Hellinger distance H(P, Q) by √ √ (10) H 2 (P, Q) ≡ 12 [ p − q ]2 dµ for any measure µ dominating both P and Q. Show that the choice of dominating measure µ does not aﬀect the value of H(P, Q).

2.

THE RADON–NIKODYM THEOREM

69

Exercise 2.12 Let φ be a σ-ﬁnite signed measure. Deﬁne X dφ = X dφ+ − X dφ− when this is ﬁnite. Show that | X dφ| ≤ |X| d|φ| . Exercise 2.13 Let µ be a σ-ﬁnite measure on (Ω, A), and deﬁne φ(A) ≡ for all A ∈ A for some µ-integrable function X. Show that for all A ∈ A. |φ|(A) = A |X| dµ

A

X dµ

Exercise 2.14 Let (Ω, A) be a measurable space, and let M denote the collection of all ﬁnite signed measures µ on (Ω, A). Let µ ≡ |µ|(Ω). Thus µ1 − µ2 = |µ1 − µ2 |(Ω). Show that (M · ) is a complete metric space.

70

3

CHAPTER 4. DERIVATIVES VIA SIGNED MEASURES

Lebesgue’s Theorem

Theorem 3.1 (Lebesgue) Suppose F is an function on [a, b]. Then F has a measurable derivative F that exists a.e. λ on [a, b]. Proof.

Consider the Dini derivates D F (x) ≡ lim suph→0+ [F (x + h) − F (x)]/h , D− F (x) ≡ lim suph→0+ [F (x) − F (x − h)]/h , D+ F (x) ≡ lim inf h→0+ [F (x + h) − F (x)]/h , D− F (x) ≡ lim inf h→0+ [F (x) − F (x − h)]/h . +

Trivially, D+ F (x) ≥ D+ F (x) and D− F (x) ≥ D− F (x). All four derivates having the same ﬁnite value is (of course) the deﬁnition of F being diﬀerentiable at x, with the common value of the derivates being called the derivative of F at x and being denoted by F (x). Let

(a)

A ≡ {x : D+ F (x) > D− F (x)} ≡ r,s Ars ≡ r,s {D+ F (x) > s > r > D− F (x)},

where the union is over all rational r and s. It suﬃces to show that all Ars have outer Lebesgue measure zero, in that λ∗ (Ars ) = 0. To this end, let U be an open set for which Ars ⊂ U with λ(U ) < λ∗ (Ars ) + . For each x ∈ Ars we can specify inﬁnitely many and arbitrarily small h for which [x − h, x] ⊂ U and [F (x) − F (x − h)]/h < r. This collection of closed intervals covers Ars in the sense of Vitali (see exercise 1.2.6). Thus some ﬁnite disjoint collection of them m has interiors I1 ≡ (x1 −h1 , x1 ), . . . , Im ≡ (xm −hm , xm ) for which Brs ≡ Ars ∩( i=1 Ii ) has λ∗ (Brs ) > λ∗ (Ars ) − . Then m m ∗ (b) i=1 [F (xi ) − F (xi − hi )] < r i=1 hi ≤ r λ(U ) < r [λ (Ars ) + ] . For each y ∈ Brs we can specify inﬁnitely many and arbitrarily small h for which [y, y + h] ⊂ (some Ii ) and [F (y + h) − F (y)]/h > s. This collection covers Brs in the sense of Vitali. Thus some ﬁnite disjoint collection of them has interiors J1 ≡ (y1 , y1 + h1 ), . . . , Jn ≡ (yn , yn + hn ) for which Crs ≡ Brs ∩ ( nj=1 Jj ) has λ∗ (Crs ) > λ∗ (Brs ) − . Then n n ∗ ∗ (c) j=1 [F (yj + hj ) − F (yj )] > s j=1 hj ≥ s [λ (Brs ) − ] > s [λ (Ars ) − 2 ]. Moreover, since the disjoint union of the Jj ’s is a subset of the disjoint union of the Ii ’s, results (b) and (c) yield m r [λ∗ (Ars ) + ] > i=1 [F (xi ) − F (xi − hi )] (d) ≥ nj=1 [F (yj + hj ) − F (yj )] > s [λ∗ (Ars ) − 2 ] for every > 0. That is, r λ∗ (Ars ) ≥ s λ∗ (Ars ). But r < s. Thus λ∗ (Ars ) = 0 for all rational r and s. Thus λ∗ (A) = 0. The sets where any two other Dini derivates take on unequal values also have measure zero, by analogous arguments. That is, F has all four derivates equal to some value F (x) for a.e. x in [a, b].

3.

LEBESGUE’S THEOREM

71

Now, the measurable function diﬀerence quotients (e)

Dn F (x) ≡ n[F ((x + 1/n) ∧ b) − F (x)]

on [a, b] converge a.e. λ to F (x) on [a, b], so that F (x) is measurable. Applying Fatou’s lemma to the Dn F (which are ≥ 0, since F is ) gives b b b (f) F (x) dx = a [ lim Dn F (x)] dx ≤ lim a Dn F (x) dx a = lim

b a

[F ((x + 1/n) ∧ b) − F (x)] dx

b+1/n

n F (b) dx −

(g)

= lim [

(h)

≤ F (b) − F (a),

b

a+1/n a

n F (x) dx ]

where we use monotonicity of F in the last step. Thus F is integrable, and is also ﬁnite a.e. λ, since we just showed that b (1) whenever F is on [a, b]. a F (x) dx ≤ F (b) − F (a) That is, F is diﬀerentiable a.e. λ, and its derivative F satisﬁes (1). [The Lebesgue singular df in example 7.1.1 below will show that equality need not hold in (1). An example in Hewitt and Stromberg (1965, p. 278) shows that F (x) = 0 is possible for all x, even with a ↑ F. ] 2 Theorem 3.2 (Term-by-term diﬀerentiation n of series) Let gk be on [a, b] for each k ≥ 1, and suppose that Sn (x) ≡ k=1 gk (x) converges at x = a and x = b. Then Sn (x) → S(x) for all x in [a, b], for some ﬁnite-valued function S(x). Mainly, S (·) exists a.s. λ and is given by (2)

S (x) =

∞

gk (x).

k=1

n Corollary 1 If the power series S(x) ≡ ∞ n=1 an (x − a) converges absolutely for x = a + R, then for all |x − a| < R we may diﬀerentiate S(x) term by term. Moreover, this is true for any number of derivatives of S. Proof.

Note that Sn (a) is a convergent sum. Now write n Sn (x) = Sn (a) + [Sn (x) − Sn (a)] = Sn (a) + k=1 [gk (x) − gk (a)] .

Since sequences bounded above converge, the convergence at x = a and x = b gives convergence at all x in the interval. We may assume all gk ≥ 0 on [a, b] with g(a) = 0; else replace gk by gk (·) − g(a). Since S and all Sn are , the derivatives S and all Sn exist a.e. λ by theorem 3.1 (of Lebesgue). Now, (a)

Sn (x) ≤ Sn+1 (x) ≤ S (x)

a.e. λ ;

72

CHAPTER 4. DERIVATIVES VIA SIGNED MEASURES

both essentially follow from ∞

(b)

Sn (x + h) − Sn (x) gk (x + h) − gk (x) S(x + h) − S(x) = + h h h n+1 ≥

Sn (x + h) − Sn (x) . h

From (a) we see (without having made use of gk (a) = 0) that (c)

Sn (·) converges a.e. λ with lim Sn ≤ S a.e. λ.

Because Sn , it suﬃces to show that Sn i →a.e. S for some subsequence ni . Since Sn (b) S(b), we may specify ni so large that 0 ≤ S(b) − Sni (b) < 2−i , and then ∞ ∞ (d) 0 ≤ S(x) − Sni (x) = ni +1 gk (x) ≤ ni +1 gk (b) = S(b) − Sni (b) < 2−i , for all x ∈ [a, b]. Thus ∞ ∞ (e) 0 ≤ i=1 [S(x) − Sni (x)] ≤ i=1 2−i = 1

for all x ∈ [a, b],

where the series in (e) has summands (f)

hi (x) ≡ S(x) − Sni (x)

that are in x.

Thus conclusion (c) also applies to these hi ’s (not just the gk ’s), and we thus conclude from (c) that the series n (g) Tn ≡ i=1 hi converges a.e. λ . But a series of real numbers can converge only if its nth term goes to 0; that is, (h)

S (x) − Sn i (x) = hi (x) → 0

a.e. λ.

As noted above, this suﬃces for the theorem.

Exercise 3.1

2

Prove the corollary.

Example 3.1 (Taylor’s expansion) Suppose g(·) is deﬁned in a neighborhood of a. Let x∗ denote a generic point between x and a. Let (3)

P1 (x) ≡ g(a) + g (a)(x − a) ,

(4)

P2 (x) ≡ P1 (x) + g (a)(x − a)2 /2! ,

(5)

P3 (x) ≡ P2 (x) + (x − a)3 /3! , . . . ,

(6)

R1 (x) ≡ [g(x) − g(a)]/(x − a)

or g (a),

as

x = a or x = a,

(7)

R2 (x) ≡ 2! [g(x) − P1 (x)]/(x − a)2

or g (a),

as

x = a or x = a,

(8)

R3 (x) ≡ 3! [g(x)−P2 (x)]/(x−a)3

or g (a),

as

x = a or x = a .

3.

LEBESGUE’S THEOREM

73

Then l’Hospital’s rule gives (provided g (a), g (a), g (a), . . . exist, respectively) (9) (10)

lim R1 (x) = g (a) = R1 (a),

x→a

g (x) − P1 (x) g (x) − g (a) = lim = g (a) = R2 (a), x→a x→a x−a x−a

lim R2 (x) = lim

x→a

2! [g (x) − P2 (x)] g (x) − P2 (x) = lim 2 x→a x→a (x − a) x−a

lim R3 (x) = lim

x→a

(11)

g (x) − g (a) = g (a). x→a x−a

= lim

Thus we ﬁnd it useful to use the representations (with g(k) (a) abbreviating that g (k) (·) exists at a, and with g(k) (·) abbreviating that g(k) (x) exists for all x in a neighborhood of a) P1 (x) + [R1 (x) − g (a)](x − a) if g (a), g(x) = (12) ∗ P1 (x) + [g (x ) − g (a)](x − a) if g (·) ,   if g (a),  P2 (x) + [R2 (x) − g (a)](x − a)/2! (13) g(x) = P2 (x) + [g (x∗ ) − g (a)](x − a)2 /2!   = P (x) + g (x∗ )(x − a)2 /2! if g (·) , 1

(14)

 3   P3 (x) + [R2 (x) − g (a)](x − a) /3! g(x) = P3 (x) + [g (x∗ ) − g (a)](x − a)3 /3!   = P (x) + g (x∗ )(x − a)3 /3! 2

if g (a), if g (·) .

∞ k k Exercise 3.2 Let f (x) = ∞ 0 ak x / 1 bk X in some interval. Suppose that all ak , bk > 0 and ak /bk ↑. Then f (x) > 0 for all x in that interval. (This result is useful in conjunction with the monotone likelihood ratio principle.)

74

4

CHAPTER 4. DERIVATIVES VIA SIGNED MEASURES

The Fundamental Theorem of Calculus

Deﬁnition 4.1 (Bounded variation) Let F denote a real-valued function on [a, b]. The total variation of F over [a, b] is deﬁned by Vab F ≡ V[a,b] F (1)

≡ sup

n

|F (xk ) − F (xk−1 )| : a ≡ x0 < x1 < · · · < xn ≡ b, n ≥ 1 .

k=1

We say that F is of bounded variation (BV) on [a, b] if Vab F < ∞. It is clear that (2)

Vab F = Vac + Vcb

for a ≤ c ≤ b and F of BV.

Deﬁnition 4.2 (Absolutely continuous functions) A real-valued function F on any subinterval I of the line R is said to be absolutely continuous if for all > 0 there exists a δ > 0 such that (3)

n

|F (dk ) − F (ck )| <

whenever

k=1

n (dk − ck ) < δ k=1

and the (ck , dk ) are disjoint subintervals of I. Deﬁnition 4.3 (Lipschitz condition) A real-valued function F on any subinterval I of R is said to be Lipschitz if for some ﬁnite constant M we have (4)

|F (y) − F (x)| ≤ M |y − x|

for all x and y in I.

We ﬁrst establish some elementary relationships among the Lipschitz condition, absolute continuity, bounded variation, and the familiar property of being . These concepts have proven to be important in the study of diﬀerentiation. We will soon proceed further in this direction, and we will also consider the relationship between ordinary derivatives and Radon–Nikodym derivatives. We ﬁrst recall from theorem 1.3.1 (the correspondence theorem) that every generalized df F can be associated with a Lebesgue–Stieltjes measure µF via µF ((a, b]) ≡ F (b) − F (a). Proposition 4.1 (The basics) (i) If F is of BV on [a, b], then

We establish some relationships.

(5)

with F1 (x) ≡ Vax F

F (x) = F1 (x)−F2 (x)

and F2 (x) ≡ Vax F −F (x)

both being on [a, b]. (ii) If F is absolutely continuous, then it is of BV. The F1 and F2 in (i) are both absolutely continuous and , and F = F1 − F2 a.s. λ ≡ Lebesgue measure. (iii) Lipschitz functions are absolutely continuous.

4.

THE FUNDAMENTAL THEOREM OF CALCULUS

75

Proof. Consider (i). Now, F1 (x) = Vax F is obviously ; use (2). Then F2 is also , since for x ≤ y we have F2 (y) − F2 (x) = [Vay F − F (y)] − [Vax F − F (x)] (a)

= Vxy F − [F (y) − F (x)] ≥ 0.

Consider (ii). Suppose that the df F (·) is absolutely continuous. Letting = 1 and choosing δ1 so small that equally spaced a ≡ x0 < x1 < · · · < xn ≡ b have mesh ≡ (b − a)/n < δ1 , we have from (2) that n n (b) Vab F = k=1 V[xk−1 ,xk ] F ≤ k=1 1 = n; and thus F is of BV. Since F1 and F2 are , their derivatives exist a.e. by theorem 4.3.1 (Lebesgue’s theorem). But we must still show that F1 is absolutely continuous if F is. So we n suppose that F is absolutely continuous, and specify that 1 (dk − ck ) < δ/2 for some choice of n, ck ’s, and dk ’s. Then for each ﬁxed k with 1 ≤ k ≤ n and the tiny number /(2n), the deﬁnition of the BV of F gives mk (c) V[ck ,dk ] F < j=1 |F (ankj ) − F (ank,j−1 )| + ( /2n) for some choice of ck ≡ ank0 < · · · < ankmk ≡ dk . These add to give n n n k=1 |F1 (dk ) − F1 (ck )| = k=1 |V[a,dk ] − V[a,ck ] | = k=1 V[ck ,dk ] F n

mk

(d)

≤

(e)

≤ ( /2) + ( /2) =

k=1

j=1

|F (ankj ) − F (ank,j−1 )| +

n k=1

( /2n)

by absolute continuity of F , since n mk n (f) k=1 k=1 (dk − ck ) < δ/2 j=1 (ankj − ank,j−1 ) = was speciﬁed. Consider (iii). Being Lipschitz implies absolute continuity with δ = /M .

Exercise 4.1 Vab F ±

2

If F is of BV on [a, b], deﬁne F1 ≡ Vab F + and F2 ≡ Vab F − , where n ≡ sup{ k=1 [F (xk ) − F (xk−1 )]± : a ≡ x0 < · · · < xn ≡ b, n ≥ 1}.

Verify that F = F1 − F2 with F1 and F2 both (providing an alternative to (5)). x Exercise 4.2 Let f be continuous on [a, b], and deﬁne F (x) = a f (y) dy for each a ≤ x ≤ b. Then F is diﬀerentiable at each x ∈ (a, b) and F = f on (a, b). [Since f is continuous, we need only the Riemann integral. Can we extend this to the Lebesgue integral? Can we reverse the order, and ﬁrst diﬀerentiate and then integrate? The next theorem answers these questions.]

76

CHAPTER 4. DERIVATIVES VIA SIGNED MEASURES

Theorem 4.1 (Fundamental theorem of calculus) (i) Let F be absolutely continuous on [a, b], and let λ denote Lebesgue measure. Then the derivative F (·) of F (·) exists a.e. λ on [a, b], and is integrable with respect to λ (using only that F is of BV). Moreover, x (6) F (x) − F (a) = F (y) dy for all x ∈ [a, b]. a

x (ii) Conversely, if F (x) − F (a) = a f dλ for some f that is integrable with respect to λ on [a, b], then F is of BV and F = f a.e. λ on [a, b]. Moreover, F is absolutely continuous on [a, b]. Remark 4.1 The fundamental theorem of calculus can be summarized by saying that F is absolutely continuous if and only if it is the integral of its derivative. The ordinary derivative F is, in fact, also a Radon–Nikodym derivative of the signed measure naturally associated with F ; proposition 4.2 below makes this even clearer. [The Lipschitz condition represents “niceness with a vengeance,” as it guarantees that all diﬀerence quotients are uniformly bounded.] 2 b Proof. Consider the converse. If F (x) ≡ F (a)+ a f (y)dy for a ≤ y ≤ b, then F is absolutely continuous by the absolute continuity of the integral theorem. Then F is of bounded variation on [a, b] and F exists a.e. λ in [a, b], by proposition 4.1(ii). But does F = f a.e. λ? Case 1: Suppose |f | is bounded by some ﬁnite M on [a, b]. We could consider f + and f − separately, but we will simply assume without loss of generality that f ≥ 0. Then x+1/n the diﬀerence quotient Dn F (x) ≡ n x f (y)dy of F also satisﬁes |Dn F | ≤ M on [a, b], and Dn F (x) → F (x) a.e. Applying the DCT (with dominating function identically equal to M ) once for each ﬁxed x ∈ [a, b] gives x x x a F (y) dy = a lim Dn F (y) dy = lim a n [F (y + 1/n) − F (y)] dy x+1/n a+1/n = lim [n x F (y) dy − n a F (y) dy ] = F (x) − F (a) (a)

=

x a

by continuity of F

f (y)dy.

Thus F (y) = f (y) a.e. on [a, b] by exercise 3.2.2 (refer also to the prototypical example 4.1.1). Case 2: Suppose f is integrable. Again, f ≥ 0 may be assumed. Let fn (·) ≡ n∧f (·), x with f − fn ≥ 0. Now, a fn has derivative fn a.e. on [a, b], by case 1. Thus d x d x d x (b) F (x) = dx a f (y)dy = dx a fn (y)dy + dx a [f (y) − fn (y)]dy ≥ fn (x) + 0 for all n, and hence F (x) ≥ f (x) a.e. on [a, b]. Thus (c)

b a

F (x)dx ≥

b a

f (x)dx = F (b) − F (a), which is ≥

b a

F (x)dx

4.

THE FUNDAMENTAL THEOREM OF CALCULUS

by (4.3.1). Thus b (d) a [F (x) − f (x)] dx = 0

77

with F (x) − f (x) ≥ 0 a.e.,

so that F = f a.e. on [a, b] by exercise 3.2.1. Consider the direct half. Now, F exists a.e. on [a, b] by proposition 4.1(ii), and it is integrable by (4.3.1). Also, x F (x) − F (a) = a f dλ (e) where f ≡ dµF /dλ by proposition 4.2 below, and it is integrable by (4.3.1). Now apply the converse half of the present theorem to conclude that F = f a.e. on [a, b]. 2 Suppose F is and absolutely continuous on R. Use µF ((a, b]) ≡ F (b) − F (a) and the correspondence theorem to associate a Lebesgue–Stieltjes measure µF with F (which is a generalized df). Proposition 4.2 Let F be and absolutely continuous on R. Then µF λ with Radon–Nikodym derivative dµF /dλ, and x (7) F (x) − F (a) = a f dλ where f ≡ dµF /dλ. Proof. Let µ ≡ µF and ﬁx an interval [a, b]. Given > 0, let δ > 0 be as in the deﬁnition (3) of absolute continuity. Let A ∈ B be a subset of [a, b] having λ(A) < δ /2. Recalling our deﬁnition (1.2.1) of λ via Carath´eodory coverings, we can claim that Lebesgue measure satisﬁes ∞ (a) λ(A) = inf{ ∞ n=1 λ(An ) : A ⊂ n=1 An for An ’s in the ﬁeld BF }. Thus for some ﬁxed choice of sets in the ﬁeld BF , we can write ∞ ∞ (b) A ⊂ 1 (cn , dn ] , where 1 (dn − cn ) < λ(A) + δ /2 < δ (recall that each An in the ﬁeld BF is a ﬁnite disjoint union of intervals (c, d] with c and d ﬁnite, with ﬁniteness following from A ⊂ [a, b]). Thus ∞ ∞ µF (A) ≤ µF ( 1 (cn , dn ]) ≤ 1 µF ((cn , dn ]) ≤

∞ 1

[F (dn ) − F (cn )] ≤ lim m

m 1

[F (dn ) − F (cn )]

≤ lim = , m

m using F absolutely continuous with 1 (dn − cn ) < δ in the last line. Thus µF (A) < whenever λ(A) < δ , so that µF (A) = 0 whenever λ(A) = 0. Apply this with A = [k, k + 1] for every −∞ < k < ∞. Adding up on k gives µF (A) = 0 whenever λ(A) = 0, for any A ∈ B. Thus µF λ. Now apply the Radon–Nikodym theorem to obtain f as in (7). 2

78

CHAPTER 4. DERIVATIVES VIA SIGNED MEASURES

Densities of Transformed Random Variables Example 4.1 (Change of variable) Suppose that X is a rv on (Ω, A, P ) with df FX λ ≡ (Lebesgue measure), with density fX . Let (8)

Y ≡ g(X)

where g−1 is ↑ and absolutely continuous.

Then FY (y) ≡ P (Y ≤ y) = P (g(X) ≤ y) = P (X ≤ g−1 (y)) = FX (g −1 (y)) , where the composition FY = FX (g −1 ) of these absolutely continuous functions is absolutely continuous. So the fundamental theorem of calculus tells us that FY is the integral of its derivative. We can then compute this derivative from the ordinary chain rule. Thus b b −1 d FY (b) − FY (a) = a FY (r) dλ(r) = a [FX (g (r)) dr g −1 (r)] dλ(r) for all a ≤ b. Thus FY λ with density (9)

fY (y) = fX (g −1 (y))

d dy

g −1 (y)

on the line. Call (d/dy) g−1 (y) the Jacobian of the transformation.

2

Exercise 4.3 (a) Show that the (well-deﬁned) composition of two absolutely continuous functions is absolutely continuous. (b) Suppose that h : [a, b] → (0, ∞) is absolutely continuous on [a, b]. Show that log h is also absolutely continuous on [a, b]. Exercise 4.4 Let F be right continuous and of BV on R, with F (−∞) = 0. Let µ denote the associated LS-measure. Then µ λ if and only if F is an absolutely continuous function on R. Exercise 4.5 (Speciﬁc step functions that are dense in L2 ) Let h ∈ L2 ([0, 1], B, λ). Consider the following two approximations to h(·). Let ¯ m (t) ≡ m i/m ˇ m (t) ≡ h(i/(m + 1)) h and h (i−1)/m h(s) ds for (i − 1)/m < t ≤ i/m and m ≥ 1. Show that: (10)

¯ m → h a.s. and L2 . h

ˇ m → h a.s. and L2 h provided that h is . 1 ¯ m − h)2 dt = 1 (h2 − h ¯ 2 ) dt, and then [Hint. Show that 0 ≤ 0 (h m 0 i/m ¯ m (t) = m(i/m − t) { h h ds/(i/m − t)} t t + m(t − (i − 1)/m) { (i−1)/m h ds/(t − (i − 1)/m)} → h(t) a.s. (11)

leads easily to a.s. convergence. Alternatively, use the fact that the continuous functions are dense in L2 .]

Chapter 5

Measures and Processes on Products 1

Finite-Dimensional Product Measures Suppose (Ω, A) and (Ω , A ) are measurable

Deﬁnition 1.1 (Product spaces) spaces. Deﬁne (1)

A × A ≡ σ[F] where F ≡

m (Ai × Ai ) : m ≥ 1 Ai ∈ A and Ai ∈ A , i=1

(2)

F0 ≡ {A × A : A ∈ A and A ∈ A }.

Here A × A ≡ {(ω, ω ) : ω ∈ A, ω ∈ A } is called a measurable rectangle. The σ-ﬁeld A × A ≡ σ[F] is called the product σ-ﬁeld. (Ω × Ω , A × A ) is called the product measurable space. The sets A × Ω and Ω × A are called cylinder sets. Proposition 1.1

F is a ﬁeld (as ﬁgure 5.1 below shows).

A1′ A2′

A1 A2 Figure 1.1 The field F.

Theorem 1.1 (Existence of the product measure) Let (Ω, A, µ) and (Ω , A , ν) be σ-ﬁnite measure spaces. Deﬁne φ on the ﬁeld F via m m

(3) (Ai × Ai ) = µ(Ai ) × ν(Ai ). φ i=1

i=1

79

80

CHAPTER 5. MEASURES AND PROCESSES ON PRODUCTS

Then φ is a well-deﬁned and σ-ﬁnite measure on the ﬁeld F. Moreover, φ extends uniquely to a σ-ﬁnite measure, called the product measure and also denoted by φ, on (Ω × Ω , A × A ). Even when completed, this measure is still unique and is still referred to as the product measure φ. Proof. (See the following exercise. Although the proof is “obvious”, it still requires some tedious detail.) But we will give a better proof herein very soon. 2

Exercise 1.1 Verify that φ is well-deﬁned on F0 , and that φ is countably additive on F0 . Then verify that φ is well-deﬁned on F, and that φ is countably additive on F. Thus φ is a σ-ﬁnite measure on F, so that the conclusion of theorem 1.1 follows from the Carath´eodory extension of theorem 1.2.1 and its corollary. Exercise 1.2

Use induction to show that theorem 1.1 extends to n-fold products.

Example 1.1 (Lebesgue measure in n dimensions, etc.) (a) We deﬁne n n ¯ n , B¯n ) ≡ ¯ ¯ (Rn , Bn ) = i=1 (R, B) and (R i=1 (R, B ) to be the n-fold products of the real line R with the Borel sets B and of the extended ¯ with the σ-ﬁeld B¯ ≡ σ[B, {+∞}, {−∞}], respectively. Recall from real line R example 2.1.1 that Bn = σ[Un ], where Un denotes all open subsets of Rn . We will refer to both Bn and B¯n as the Borel sets. ¯ B¯ ) by (b) Let λ denote Lebesgue measure on (R, B), as usual. We extend λ to (R, the convention that λ({+∞}) = 0 and λ({−∞}) = 0. Then n ¯n , B¯n , λn ) ≡ n (R, ¯ B, ¯ λ) (4) and (R (Rn , Bn , λn ) ≡ i=1 (R, B, λ) i=1 provides us with a deﬁnition of n-dimensional Lebesgue measure λn as the natural generalization of the concept of volume. It is clear that (5)

(Rm × Rn , Bm × Bn , λm × λn ) = (Rm+n , Bm+n , λm+n ),

and that this holds on the extended Euclidean spaces as well. (It is usual not to add the symbol in dealing with the completions of these particular measures.) (c) Now, λ is just a particular Lebesgue–Stieltjes measure on (R, B). Any Lebesgue– ¯ B¯ ) yields an obvious n-fold product on either Stieltjes measure µF on (R, B) or (R, ¯n , B¯n ), which could appropriately be denoted by µF × · · · × µF . (Rn , Bn ) or (R Further, we will let Fn denote the ﬁeld consisting of all ﬁnite disjoint unions of sets of the form I1 × · · · × In where each Ik is of the form (a, b],(−∞, b] or (a, +∞) when considering (Rn , Bn ) (or of the form (a, b], [−∞, b], or (a, +∞] when considering ¯ n , B¯n )). (That is, in the case of (Rn , Bn ) there is the alternative ﬁeld Fn that (R also generates the σ-ﬁeld Bn ; and this Fn is made up of simpler sets than is the ﬁeld B × · · · × B used in deﬁnition 1.1.) (d) The approximation lemma of (1.2.8) now shows that if (µF × · · · × µF )(A) < ∞ and if > 0 is given, then (µF × · · · × µF )(A C) < for some C in (the simpler ﬁeld) Fn . That is, the simpler ﬁeld gives us a nicer conclusion in this example, because its sets C are simpler. 2

1.

FINITE-DIMENSIONAL PRODUCT MEASURES

81

Exercise 1.3 The measures on (Rn , Bn ) introduced above in example 1.1 are regular measures. Show this. Deﬁnition 1.2 (Sections) (a) Let X denote a function on Ω × Ω . For each ω in Ω, the function Xω (·) on Ω deﬁned by Xω (ω ) ≡ X(ω, ω ), for each ω in Ω , is called an ω-section of X(·, ·) . An ω -section of X(·, ·) is deﬁned analogously. (b) Let C be a subset of Ω×Ω . For each ω in Ω, the set Cω = {ω : (ω, ω ) is in C } is called the ω-section of C. An ω -section of C is deﬁned analogously. Theorem 1.2 (Product measure) Let (Ω, A, µ) and (Ω, A , ν) denote ﬁnite measure spaces. Let C ∈ A × A . Then: (6) (7)

Every Cω ∈ A and every Cω ∈ A whenever C ∈ A × A , φ(C) ≡ µ(Cω ) dν(ω ) = ν(Cω ) dµ(ω) for every C ∈ A × A , Ω

Ω

and this φ is exactly the product measure φ = µ × ν of theorem 1.1 . Proof. We ﬁrst show (6). This result is trivial for any C in F0 , or any C in F. Now let S denote the class of all sets C in A × A for which (6) is true. Then S is trivially seen to be a σ-ﬁeld, using (a)

(n)

(∪n C (n) )ω = ∪n Cω

and (C c )ω = (Cω )c .

But since F ⊂ S, we have that A × A = σ[F] equals S. Let M denote the collection of all sets C in A × A for which (7) holds. Clearly, M contains F0 and F. Note that if Cn converges monotonically to some C, then 1Cn converges monotonically to 1C and (b)

every section of 1Cn converges monotonically to the corresponding section of 1C .

We will now use this fact to show that the class M of all subsets of A × A for which (7) holds is a monotone class; it will then follow by proposition 1.1.6 that M = σ[F] = A × A . Let Cn denote a sequence of sets in the class M that converge monotonically (we will consider only the case, since we only need to take complements in the case), and we give the name C to the limiting set. Since 1Cn 1C , the function 1C is (A × A )-measurable, and thus every section of 1C is measurable by (6). Now, for ﬁxed ω the number h(ω ) ≡ µ(Cω ) = 1 (ω)dµ(ω) is (by the MCT and (b)) the limit of the sequence of numbers Ω C,ω hn (ω ) ≡ µ(Cn,ω ) = Ω 1Cn ,ω (ω)dµ(ω), for each ω in Ω . Thus the function h on Ω is the limit of the functions hn on Ω ; and since Cn is in M, the functions hn are A -measurable; thus h is A -measurable by proposition 2.2.2. Moreover, the ﬁnite numbers φ(Cn ) are bounded above by µ(Ω) ν(Ω ), and thus converge to some number; call it φ(C). That is, (c) φ(C) ≡ lim φ(Cn ) ≡ lim Ω { Ω 1Cn ,ω (ω)dµ(ω)} dν(ω ) when Cn is in M n

n

82

CHAPTER 5. MEASURES AND PROCESSES ON PRODUCTS

(d)

= lim n

= (e)

=

Ω

hn dν =

hn } dν Ω {lim n

(ω)dµ(ω)} dν(ω )

Ω

{

Ω

µ(Cω ) dν(ω ) .

Ω 1C,ω

=

Ω

by the MCT and hn h

hdν

by the deﬁnition of h

(Since φ(C) is ﬁnite, we see that h is ν-integrable. Thus h(ω ) is ﬁnite for a.e. [ν] ω .) The argument for each ﬁxed ω is symmetric, and it gives the second equality in (7). Thus C is in M, making M the monotone class A × A ; and (b) holds. [Thus the result (7) holds for the set function φ. But is φ a measure?] In this paragraph we will show that the product measure φ of theorem 1.1 exists, and is deﬁned by (e). To this end, let D1 , D2 , . . . be pairwise disjoint sets in A × A , n ∞ and let Cn ≡ 1 Dk C ≡ 1 Dk . Then linearity of both single integrals shows (in the second equality) that n ∞ n (f) 1 φ(Dk ) = lim 1 φ(Dk ) = lim φ( 1 Dk ) = lim φ(Cn ) = φ(C) n

n

n

by (c) through (e) ∞ = φ( 1 Dk ),

(g)

so that φ is c.a., and a measure on A × A . We have just veriﬁed that the product measure of (3) exists on A × A , and is given by (7). That is, we have just proven theorem 1.1 and given the representation (7) for φ(C). Note that the product measure φ also satisﬁes φ(C) = Ω×Ω 1C (ω, ω ) dφ(ω, ω ). 2 Theorem 1.3 (Fubini) Let (Ω, A, µ) and (Ω , A , ν) be σ-ﬁnite measure spaces. Let φ = µ × ν on (Ω × Ω , A × A ). Suppose that X(ω, ω ) is φ-integrable (i.e., X −1 (B¯ ) ⊂ A × A and Ω×Ω X dφ is ﬁnite). Then: (8)

All ω -sections Xω (·) of X are A-measurable functions on Ω.

(9)

For a.e. [ν] ﬁxed ω , the function Xω (·) ≡ X(·, ω ) is µ-integrable. The function h(ω ) ≡ Xω (ω) dµ(ω) is a ν-integrable function of ω .

(10)

(11) Ω×Ω

Ω

X(ω, ω )dφ(ω, ω ) = [ X(ω, ω )dµ(ω)]dν(ω ) =

Ω

Ω

h(ω )dν(ω ).

Ω

[Setting X equal to 1C in (11) for C ∈ A × A shows how the value φ(C) of the product measure φ at C was deﬁned as an iterated integral; recall (7).] Corollary 1 (Tonelli) Let X be A × A -measurable and suppose either (12) |X| dµ dν < ∞ or |X| dν dµ < ∞ or X ≥ 0. Then the claims of Fubini’s theorem are true, including (13) X dφ = X dµ dν = X dν dµ.

1.

FINITE-DIMENSIONAL PRODUCT MEASURES

83

Corollary 2 (µ × ν null sets) A set C in A × A is (µ × ν)-null if and only if almost every ω-section of C is a ν-null set. That is, for C ∈ A × A we have (14)

µ × ν(C) = 0

if and only if

ν(Cω ) = 0 for a.e. [µ] ω in Ω.

Proof. By using the σ-ﬁniteness of the two measures to decompose both Ω and Ω , we may assume in this proof that both µ and ν are ﬁnite measures. We begin by discussing only measurability questions. We will ﬁrst show that (a)

all ω -sections of an (A × A )-measurable function X are A-measurable.

The previous theorem shows that (b)

all ω -sections Xω of X are A-measurable whenever X = 1C for some C ∈ A × A .

¯ Now let X denote any (A × A )-measurable function. Then for any B in B, (c)

−1 Xω−1 (B)} (B) = {ω : X(ω, ω ) ∈ B} = {ω : (ω, ω ) ∈ X

is the ω -section of the indicator function of the set C = X −1 (B); so (b) shows that any arbitrary ω -section of this X is A-measurable, and so establishes (a) and (8). We now turn to all the other claims of the Fubini and Tonelli theorems. By theorem 1.2 they hold for all (A × A )-measurable indicator functions. Linearity of the various integrals shows that the theorems also hold for all simple functions. Applying the MCT to the various integrals shows that the theorems also hold for all (A × A )-measurable X ≥ 0. Then linearity of the integral shows that the theorems also hold for all X for whichever of the three integrals exists ﬁnitely (the double integral or either iterated integral). Corollary 2 follows immediately by applying (13) and exercise 3.2.2 (only the zero function) to the integral of the function 1C . 2

84

2

CHAPTER 5. MEASURES AND PROCESSES ON PRODUCTS

Random Vectors on (Ω, A, P )

We will now treat measurable functions from a probability space (Ω, A, P ) to a Euclidean space (Rn , Bn ), with n ≥ 1. Let x ≡ (x1 , . . . , xn ) denote a generic vector in the Euclidean space Rn . ≡ (X1 , . . . , Xn ) is such that Deﬁnition 2.1 (Random vectors) Suppose X X : Ω → Rn is Bn -A-measurable. Then X is called a random vector (which is also by abbreviated rv ). Deﬁne the joint distribution function (or just df) of X n F (x) ≡ FX1 ,...,Xn (x1 , . . . , xn ) = P ( i=1 [Xi ≤ xi ] ) . Write x ≤ y to denote that xi ≤ yi for all 1 ≤ i ≤ n; and now deﬁne the basic rectangles (x, y ] ≡ ×ni=1 (xi , yi ] whenever x ≤ y. Let n F (x, y ] ≡ P ( i=1 [xi < Xi ≤ yi ] ) (1) for all x ≤ y . ≡ (X1 , . . . , Xn ) : Ω → Rn or R ¯n Proposition 2.1 (Measurability) Now, X is such that Bn -A-measurable B-A-measurable X is if and only if each X is i ¯ B¯n -A-measurable B-A-measurable. Thus, a random vector is measurable if and only if each coordinate rv is measurable. Proof.

We give the details for ﬁnite-valued functions. (⇒) Now, [Xi ≤ xi ] = Xi−1 ((−∞, xi ] ) = X −1 (R × · · · × R × (−∞, xi ] × R × · · · × R) ∈ A.

≤ x ] = ∩n [Xi ≤ xi ] ∈ A, since each Xi is measurable, where (⇐) Also, [X i=1 σ[ {all (−∞, x1 ] × · · · × (−∞, xn ]} ] = Bn . Moreover, −1 (A∗ ) ∈ A} = (a σ-ﬁeld) A∗ ≡ {A∗ : X by proposition 2.1.2. Thus X −1 (Bn ) ⊂ A.

Exercise 2.1 (Joint df) (2)

2

A joint df F is and right continuous and satisﬁes

All F (x1 , . . . , xi−1 , −∞, xi+1 , . . . , xn ) = 0

and

F (x1 , . . . , xi−1 , +∞, xi+1 , . . . , xn ) (3)

= FX1 ,...,Xi−1 ,Xi+1 ,...,Xn (x1 , . . . , xi−1 , xi+1 , . . . , xn )

for all j = 1, . . . , n and x1 , . . . , xn .

F (∞, . . . , ∞) = 1,

2.

RANDOM VECTORS ON (Ω, A, P )

85

Exercise 2.2 Suppose F : Rn → R is and right continuous and satisﬁes (2) and (3). Then there exists a unique probability measure P ≡ PF on Bn that satisﬁes (4)

P ((x, y ] ) = F (x, y ]

for all x ≤ y .

This is a generalization of the correspondence theorem to n > 1. Now note that the identity function X(ω) ≡ ω, for each ω ∈ Rn , is a random vector on (Rn , Bn ) that has as its joint df the function F above. Thus, given any joint df F , there is a having F as its joint df. This is in the spirit of example 2.2.1. random vector X Exercise 2.3 (Hoeﬀding–Fr´echet bounds) (i) Show that Ha (u, v) ≡ u ∧ v and Hb (u, v) ≡ (u + v − 1)+ are dfs on [0, 1]2 with Uniform(0, 1) marginal dfs. Determine the sets on which the densities of Ha and Hb are positive. (ii) Suppose that F (x, y) is a df on R2 with marginal dfs G(x) ≡ F (x, ∞) and H(y) ≡ F (∞, y), respectively. Show that Hb (G(x), H(y)) ≤ F (x, y) ≤ Ha (G(x), H(y)) for all (x, y) ∈ R2 .

86

3

CHAPTER 5. MEASURES AND PROCESSES ON PRODUCTS

Countably Inﬁnite Product Probability Spaces

We now begin to carry out the program discussed in section 5. That is, we will extend the notion of rvs and product probability measures to a countably inﬁnite number of dimensions. Notation 3.1 (R∞ and B∞ ) Let ∞ (1) R∞ ≡ n=1 R ≡ {(x1 , x2 , . . .) : xn ∈ R for all n ≥ 1}. Let I denote an interval of the type (c, d ], (−∞, d], or (c, +∞). By an n-dimensional rectangle we will mean any set of the form I1 × · · · × In × R × R × · · ·, where each interval In is of the type above. A ﬁnite-dimensional rectangle is an n-dimensional rectangle, for some n ≥ 1. A cylinder set is a set of the form Bn × R × R × · · · with Bn in Bn for some n ≥ 1. Thus: (2)

CI ≡ {all ﬁnite-dimensional rectangles} = {I1 × · · · × In × R × R × · · · : n ≥ 1, all In as above},

(3)

CF ≡ {all ﬁnite disjoint unions of ﬁnite-dimensional rectangles},

(4)

C∞ ≡ {all cylinder sets} ≡ {Bn × R × R × · · · : n ≥ 1, Bn ∈ Bn }.

Both CF and C∞ are ﬁelds, and a trivial application of exercise 1.1.1 shows that (5)

B∞ ≡ σ[CI ] = σ[CF ] = σ[C∞ ].

Thus, extending a measure from CI to B∞ will be of prime interest to us. We ﬁrst extend the criterion for measurability from n dimensions to a countably inﬁnite number of dimensions. 2 ≡ (X1 , X2 , . . .) : Ω → R∞ Proposition 3.1 (Measurability on B∞ ) (a) Now, X is B∞ -A-measurable if and only if each Xn is B-A-measurable. is B∞ -A-measurable and if (i1 , i2 , . . .) is an arbitrary sequence of integers, (b) If X then Y ≡ (Xi1 , Xi2 , . . .) is B∞ -A-measurable. Exercise 3.1

Prove proposition 3.1.

Notation 3.2

We will use the notation

(6)

F(Xi ) ≡ Xi−1 (B)

and

−1 F(Xi1 , Xi2 , . . .) ≡ Y −1 (B∞ ) = σ[ ∞ n=1 Xin (B)]

to denote the minimal sub σ-ﬁelds of A relative to which the quantities Xi and ≡ (Xi1 , Xi2 , . . .) are measurable. Y 2 Now suppose that Pn is a probability measure on (Rn , Bn ), for each n ≥ 1. The question is, When can we extend the collection {Pn : n ≥ 1} to a measure on (R∞ , B∞ )? Reasoning backwards to see what conditions the family of ﬁnitedimensional distributions should satisfy leads to the following deﬁnition.

3.

COUNTABLY INFINITE PRODUCT PROBABILITY SPACES

87

Deﬁnition 3.1 (Consistency) Finite-dimensional distributions {(Rn , Bn , Pn )}∞ n=1 are consistent if for every n ≥ 1, every B1 , . . . , Bn ∈ B, and every 1 ≤ i ≤ n, Pn−1 ((X1 , . . . , Xi−1 , Xi+1 , . . . , Xn ) ∈ B1 × · · · × Bi−1 × Bi+1 · · · × Bn ) (7)

= Pn ((X1 , . . . , Xi−1 , Xi , Xi+1 , . . . , Xn ) ∈ B1 ×· · ·×Bi−1 ×R×Bi+1 ×· · ·×Bn ).

Theorem 3.1 (Kolmogorov’s extension theorem) An extension of any to a probability P (·) consistent family of probability measures {(Rn , Bn , Pn )}∞ n=1 on (R∞ , B∞ ) necessarily exists, and it is unique. We will ﬁrst summarize the main part of this proof as a separately stated result that seems of interest in its own right. Theorem 3.2 ((R∞ , B∞ ) extension theorem; Breiman) Let P on CI satisfy: (a) P ≥ 0 and P (R∞ ) = 1. m (b) If D = m j=1 Dj for n-fold rectangles D and Dj , then P (D) = 1 P (Dj ). (c) If D denotes any ﬁxed n-dimensional rectangle, then there exists a sequence of compact n-dimensional rectangles Dj for which Dj D and P (Dj ) P (D). [That is, P is well-deﬁned and additive on n-dimensional rectangles and satisﬁes something like continuity from below.] Then there exists a unique extension of P to B∞ . Proof.

(Recall the continuity result of proposition 1.1.3.) Now, CF ≡ {all ﬁnite disjoint unions of ﬁnite-dimensional rectangles} = {a ﬁeld generating B∞ }.

(p) For A =

m 1

Dj ∈ B0 , deﬁne P (A) ≡

m j=1

P (Dj ).

m m First, we will show that P is well-deﬁned on CF . Let A = 1 Dj = 1 Dk . m m Now, Dk = Dk A = 1 Dk Dj and Dj = Dj A = 1 Dj Dk . Thus (q) (r)

P (A) = =

m 1

P (Dj ) =

m m 1

1

m 1

P(

P (Dk ∩ Dj ) =

m 1

m 1

Dj Dk ) =

P(

m 1

m m 1

1

Dk ∩ Dj ) =

P (Dj Dk ) m 1

P (Dk ) = P (A).

Next, A1 , . . . , Am ∈ CF be such that we will show that P is f.a. on CF . So we mlet i A≡ m 1 Ai ∈ CF also. Then, by writing Ai = 1 Dij , m mi m mi m (s) P (A) = P ( m 1 Ai ) = P ( 1 1 1 P (Ai ), 1 Dij ) = 1 P (Dij ) = (using condition (b) in each of the last two equalities), since P is well-deﬁned. We will now show that P is continuous from above at ∅. Let An ’s in CF be such that An ∅. We must show that P (An ) 0. Assume not. Then ∞ P (An ) > 0; and by going to subsequences, we may assume that An = A∗n × n+1 R, where each A∗n is a ﬁnite union of disjoint rectangles (repeat some members of the sequence if

88

CHAPTER 5. MEASURES AND PROCESSES ON PRODUCTS

necessary in order to have A∗n ⊂ Rn ). By condition (c), choose Bn∗ ⊂ A∗n such that Bn∗ is a ﬁnite union of compact disjoint rectangles in Rn with ∞ (t) where Bn ≡ Bn∗ × n+1 R. P (An \ Bn ) < /2n+1, n ∞ Let Cn = 1 Bk ≡ Cn∗ × n+1 R, with Cn∗ compact in Rn (the Bn ’s need not be , but the Cn ’s are). Then we observe that Cn ∅, since Cn ⊂ Bn ⊂ An with An ∅; but we also have P (Cn ) ≥ /2, since n n n P (An \ Cn ) ≤ k=1 P (An \ Bk ) ≤ k=1 P (Ak \ Bk ) ≤ 1 /2k+1 ≤ /2. (u) But Cn with P (Cn ) ≥ /2 for all n is not compatible with the conclusion that (n) (n) Cn ∅: Let x(1) ∈ C1 , . . . , x(n) ∈ Cn , . . . , where x(n) ≡ (x1 , x2 , . . .). Choose an (N1 ) → (some x1 ) ∈ C1∗ ; then choose a further initial subsequence N1 such that x1 (N2 ) (N2 ) subsequence N2 such that (x1 , x2 ) → (some (x1 , x2 )) ∈ C2∗ ; . . . . Along the (N ) diagonal subsequence, say N , we have xj → xj , for all j. Now, x = (x1 , x2 , . . .) ∈ Cn for all n. Hence Cn ∅. But this is a contradiction, and thus allows us to claim that P (An ) 0 for any An ’s in CF that satisfy An ∅. Now apply the continuity of measures in proposition 1.1.3, and then apply the Carath´eodory extension of theorem 1.2.1 to complete the proof. 2 Proof. (v)

We now turn to the Kolmogorov extension theorem. The P deﬁned by P (B1 ×· · ·×Bn ×· · ·) ≡ Pn (B1 ×· · ·×Bn ) = Pm+n (B1 ×· · ·×Bn ×R×· · ·×R)

is a well-deﬁned f.a. probability on CI = {all ﬁnite-dimensional rectangles}; this follows from the consistency condition (7). Thus (a) and (b) of theorem 3.2 hold. We will now verify (c). Fix n. Let Dn be an arbitrary but ﬁxed n-dimensional rectangle. It is clearly possible to specify compact n-dimensional rectanglesDnj for ∞ which Dnj Dn as j → ∞. Write Dj = Dnj × ∞ n+1 R and D = Dn × n+1 R, so that Dj D. Thus, by the continuity of signed measures in proposition 1.1.3, (w)

P (Dj ) = Pn (Dnj ) Pn (Dn ) = P (D),

since Pn is a measure on (Rn , Bn ). Thus (c) holds. The conclusion follows from theorem 3.2. 2 Example 3.1 (Coordinate rvs) Once consistent probability measures Pn (·) on (Rn , Bn ) have been extended to a probability measure P (·) on (R∞ , B∞ ), it is appropriate then to deﬁne Xn (x1 , x2 , . . .) = xn , for each n ≥ 1. These are rvs on the probability space (Ω, B, P ) ≡ (R∞ , B∞ , P ). Moreover, −1 (Bn × ∞ R)) P ((X1 , . . . , Xn ) ∈ Bn ) = P ((X1 , . . . , Xn )−1 (Bn )) = P (X n+1 ∞ (8) = P (Bn × n+1 R) = Pn (Bn ) ≡ (X1 , X2 , . . .) : Ω → R∞ for all Bn ∈ Bn . We thus have a realization of X that is B∞ -A-measurable, and each (X1 , . . . , Xn ) induces the distribution Pn on (Rn , Bn ). This is the natural generalization of example 2.2.1 and the comment below exercise 5.2.2. 2

3.

COUNTABLY INFINITE PRODUCT PROBABILITY SPACES

89

Theorem 3.3 (The ﬁnite dfs deﬁne probability theory) = (X1 , X2 , . . .) denote any random element on (R∞ , B∞ ). Then P itself Let X and whether or not there exists a ﬁnite rv X such that Xn converges to X in the sense of →a.s. , →p , →r , or →d can be determined solely by examination of the ﬁnite-dimensional distributions of X. Proof. Let C denote the π ¯ -system of all sets n1 (−∞, xi ], for some n ≥ 1 and all xi ∈ R. The ﬁnite-dimensional distributions (even the ﬁnite-dimensional dfs) determine P∞ on C, and hence on B∞ = σ[C] (appeal to Dynkin’s π-λ theorem of proposition 1.1.5). To emphasize the fact further, we now consider each convergence mode separately. →d : Obvious. →r : E|Xn − X|r → 0 if and only if E|Xn − Xm |r < for all n, m ≥ some N . →p : Xn →p X if and only if P (|Xn − Xm | > ) < for all n, m ≥ some N . →a.s. : Xn →a.s. X if and only if ∞ 1 = P (∪∞ n=1 ∩m=n [|Xm − Xn | ≤ ])

for all > 0

= lim lim P (∩N m=n [|Xm − Xn | ≤ ]) = lim lim{a function of FXn ,...,XN }. n

N

n

N

The proof is complete 2

Example 3.2 (Equivalent experiments) Perhaps I roll an ordinary die n times with the appearance of an even number called “success.” Perhaps I draw a card at random n times, each time from a freshly shuﬄed deck of standard playing cards, with “red” called “success.” Perhaps I ﬂip a fair coin n times with “heads” called “success.” Note that (X1 , . . . , Xn ) has the same distribution in all three cases. Thus, if I report only the data from one of these experiments, you can not hope to determine which of the three experiments was actually performed. These are called equivalent experiments. 2

90

4

CHAPTER 5. MEASURES AND PROCESSES ON PRODUCTS

Random Elements and Processes on (Ω, A, P )

Deﬁnition 4.1 (Projections and ﬁnite-dimensional subsets) Let MT denote a collection of functions that associate with each t of some set T a real number denoted by either xt or x(t). [T is usually a Euclidean set such as [0, 1], R, or [0, 1] × R. The collection MT is often a collection of “nice” functions, such as the continuous functions on T .] For each integer k and all (t1 , . . . , tk ) in T we let πt1 ,...,tk denote the projection mapping of MT into k-dimensional space Rk deﬁned by (1)

πt1 ,...,tk (x) ≡ (x(t1 ), . . . , x(tk )).

Then for any B in the set of all k-dimensional Borel subsets Bk of Rk , the set πt−1 (B) is called a ﬁnite-dimensional subset of M . 1 ,...,tk Exercise 4.1 Show that the collection M0T of all ﬁnite-dimensional subsets of MT is necessarily a ﬁeld. (This is true no matter what collection MT is used.) Deﬁnition 4.2 (Measurable function spaces, ﬁnite-dimensional distributions, random elements, and normal processes) We let MT denote the σ-ﬁeld generated by the ﬁeld M0T . We call M0T and MT the ﬁnite-dimensional ﬁeld and the ﬁnitedimensional σ-ﬁeld, respectively. Call the measurable space (MT , MT ) a measurable function space over T . Given any probability space (Ω, A, P ) and any measurable space (Ω∗ , A∗ ), an ∗ A -A-measurable mapping X : Ω → Ω∗ will be called a random element. We denote this by X : (Ω, A) → (Ω∗ , A∗ ) or by X : (Ω, A, P ) → (Ω∗ , A∗ ), or even by X : (Ω, A, P ) → (Ω∗ , A∗ , P ∗ ), where P ∗ denotes the induced probability on the image space. A random element X : (Ω, A, P ) → (MT , MT , P ∗ ) in which the image space is a measurable function space will be called a process. The ﬁnite-dimensional distributions of a process are the distributions induced on (Rk , Bk ) by the projection mappings πt1 ,...,tk : (MT , MT , P ∗ ) → (Rk , Bk ). If all of the ﬁnite-dimensional distributions of a process X are multivariate normal (see section 9.3 below), then we call X a normal process. Deﬁnition 4.3 (Realizations and versions) If two random elements X and Y (possibly from diﬀerent probability spaces to diﬀerent measurable function spaces) have identical induced ﬁnite-dimensional distributions, then we refer to X and Y as diﬀerent realizations of the same random element and we call them equivalent random elements. We denote this by agreeing that X∼ means that X and Y are equivalent random elements. =Y [We will see in chapter 12 that a process called Brownian motion can be realized on both (R[0,1] , B[0,1] ) and (C, C), where C ≡ C[0,1] denotes the space of all continuous functions on [0, 1] and C ≡ C[0,1] denotes its ﬁnite-dimensional σ-ﬁeld.] If X and Y are deﬁned on the same probability space and P (Xt = Yt ) = 1 for all t ∈ T , then X and Y are called versions of each other. (In chapter 12 we will see versions X and Y of Brownian motion where X : (Ω, A, P ) → (R[0,1] , B[0,1] ) and Y : (Ω, A, P ) → (C[0,1] , C[0,1] ). Of course, this X and Y are also diﬀerent realizations of Brownian motion.)

4.

RANDOM ELEMENTS AND PROCESSES ON (Ω, A, P )

91

Deﬁnition 4.4 (Finite-dimensional convergence, →f d ) Suppose X, X1 , X2 , . . . denote processes with image space (MT , MT ). If the convergence in distribution (2)

πt1 ,...,tk (Xn ) = (Xn (t1 ), . . . , Xn (tk )) →d (X(t1 ), . . . , X(tk )) = πt1 ,...,tk (X)

holds for all k ≥ 1 and all t1 , . . . , tk in T , then we write Xn →f d X as n → ∞, and we say that the ﬁnite-dimensional distributions of Xn converge to those of X.

The General Stochastic Process Notation 4.1 (RT , BT )) (3)

We now adopt the convention that

(RT , BT ) denotes the measurable function space with RT ≡

t∈T Rt ,

where each Rt is a copy of the real line. Thus RT consists of all possible realvalued functions on T , and BT is the smallest σ-ﬁeld with respect to which all πt are measurable. We call a process X : (Ω, A, P ) → (RT , BT ) a general stochastic process. We note that a general stochastic process is also a process. But we do not yet know what BT looks like. A set BT ∈ BT is said to have countable base t1 , t2 , . . . if (4)

BT = πt−1 (B∞ ) 1 ,t2, ...

for some B∞ ∈ B∞ ;

here B∞ is the countably inﬁnite-dimensional σ-ﬁeld of section 5.3. Let BC denote the class of countable base sets deﬁned by (5)

BC ≡ {BT ∈ BT : BT has a countable base}.

[Recall F(X1 , X2 , . . .) and F(Xs : s ≤ t) measurability from section 2.5.]

2

Proposition 4.1 (Measurability in (RT , BT )) Now, BC is a σ-ﬁeld. In fact, BC is the smallest σ-ﬁeld relative to which all πt are measurable; that is, (6)

BT = B C .

Also (generalizing proposition 5.2.1), (7)

X is BT -A-measurable if and only if Xt is B-A-measurable for each t ∈ T.

Proof. Clearly, BT is the smallest σ-ﬁeld containing BC ; so (6) will follow from showing that BC is a σ-ﬁeld. Now, C is closed under complements, since c πt−1 (B∞ )c = πt−1 (B∞ ). Suppose that B1 , B2 , . . . in BC have countable bases 1 ,t2 ,... 1 ,t2 ,... ∞ T1 , T2 , . . ., and let T0 = ∪m=1 Tm . Then using the countable set of distinct coor∞ ∞ dinates in T0 , reexpress each Bm as Bm = πT−1 (Bm ) for some Bm ∈ B∞ . Then 0 −1 ∞ ∞ ∞ ∪m=1 Bm = πT (∪m=1 Bm ) is in BC . Thus BC is closed under countable unions. Thus BC is a σ-ﬁeld. Now to establish (7): Suppose X is BT -A-measurable. Thus (a)

Xt−1 (B) = X −1 (πt−1 (B)) ∈ A for B ∈ B,

so that each Xt is B-A-measurable. Suppose that each Xt is B-A-measurable. Then exercise 5.3.1 shows that (Xt1 , Xt2 , . . .) is B∞ -A-measurable for all sequences t1 , t2 , . . . of elements of T . That is, X −1 (BC ) ⊂ A. Since BT = BC , we thus have X −1 (BT ) ⊂ A, and hence X is BT -A-measurable. 2

92

CHAPTER 5. MEASURES AND PROCESSES ON PRODUCTS

Remark 4.1 (Consistency of induced distributions in (RT , BT )) Any general stochastic process X : (Ω, A, P ) → (RT , BT ) has a family of induced distributions (8)

(Bk )) Pt∗1 ,...,tk (Bk ) = P (X −1 ◦ πt−1 1 ,...,tk

for Bk ∈ Bk

for all k ≥ 1 and t1 , . . . , tk ∈ T . These distributions are necessarily consistent in the sense that (9)

Pt∗1 ,...,tk (B1 × · · · × Bi−1 × R × Bi+1 × · · · × Bk ) = Pt∗1 ,...,ti−1 ,ti+1 ,...,tk (B1 × · · · × Bi−1 × Bi+1 × · · · × Bk )

for all k ≥ 1, all B1 , . . . , Bk ∈ B, all 1 ≤ i ≤ k, and all t1 , . . . , tk ∈ T . [The next result gives a converse. It is our fundamental result on the existence of stochastic processes with speciﬁed distributions.] 2 Theorem 4.1 (Kolmogorov’s consistency theorem) Given a consistent set of distributions as in (9), there exists a distribution P on (RT , BT ) such that the identity map X(ω) = ω, for all ω ∈ RT , is a general stochastic process X : (RT , BT , P ) → (RT , BT ) whose family of induced distributions is the Pt∗1 ,...,tk of (9). Exercise 4.2 Prove theorem 4.1. [Deﬁne P ∗ (B) = P (πT−1 (B)) for B ∈ BC and i each countable subset Ti of T . Use notational ideas from the proof of proposition 4.1 to show easily that P ∗ (·) is well-deﬁned and countably additive.] Example 4.1 (Comment on (R[0,1] , B[0,1] )) The typical function x in RT has no smoothness properties. Let T = [0, 1] and let C denote the subset of R[0,1] that consists of all functions that are continuous on [0, 1]. We now show that (10)

C ∈ B[0,1] .

Let (Ω, A, P ) denote Lebesgue measure on the Borel subsets of [0, 1]. Let ξ(ω) = ω. Now let X : (Ω, A, P ) → (R[0,1] , B[0,1] ) via Xt (ω) = 0 for all ω ∈ Ω and for all t ∈ T . Let Y : (Ω, A, P ) → (R[0,1] , B[0,1] ) via Yt (ω) = 1t (ω). Now, all ﬁnite-dimensional distributions of X and Y are identical. Note, however, that [ω : X(ω) ∈ C] = Ω, while [ω : Y (ω) ∈ C] = ∅. Thus C cannot be in B[0,1] . 2

Smoother Realizations of General Stochastic Processes Suppose now that X is a process of the type X : (Ω, A, P ) → (RT , BT , P ∗ ). As the previous example shows, X is not the unique process from (Ω, A, P ) that induces the distribution P ∗ on (RT , BT ). We now let MT denote a proper subset of RT and agree that MT denotes the σ-ﬁeld generated by the ﬁnite-dimensional subsets of MT . Suppose now that X(ω) ∈ MT for all ω ∈ Ω. Can X be viewed as a process X : (Ω, A, P ) → (MT , MT , P˜ ) such that (MT , MT , P˜ ) has the same ﬁnitedimensional distributions as does (RT , BT , P ∗ )? We now show that the answer is necessarily yes. Interesting cases arise when the functions of the MT above have smoothness properties such as continuity. The next result is very important and useful.

4.

RANDOM ELEMENTS AND PROCESSES ON (Ω, A, P )

93

Theorem 4.2 (Smoother realizations of processes) Consider an arbitrary measurable mapping X : (Ω, A, P ) → (RT , BT , P ∗ ). (i) Let MT ⊂ RT . Then we can view X as a process X : (Ω, A) → (MT , MT ) if and only if every sample path X· (ω) = X(·, ω) is in MT and every Xt (·) ≡ X(t, ·) is a random variable. (ii) Let X(Ω) ⊂ MT ⊂ RT . Then X : (Ω, A, P ) → (MT , MT , P˜ ), where the ﬁnitedimensional distributions of (MT , MT , P˜ ) are the same as those of (RT , BT , P ∗ ). Proof. (11)

Note ﬁrst that AT ≡ X −1 (MT ) ≡ X −1 (MT ) ∩ X −1 (BT ) = X −1 (MT ∩ BT )

is a σ-ﬁeld

(though none of the subsets of AT may be in A). Then (Ω, AT , PT ) is a probability space where (12)

PT (X −1 (MT ) ∩ X −1 (BT )) ≡ P (X −1 (BT ))

for all BT ∈ BT .

(i) (⇐) For any t1 , . . . , tk in T and any Bt1 , . . . , Btk in B we have (Bt1 × · · · × Btk )) = X −1 (πt−1 (Bt1 ) ∩ · · · ∩ πt−1 (Btk )) X −1 (πt−1 1 ,...,tk 1 k = ∩ki=1 X −1 (πt−1 (Bti )) = ∩ki=1 Xt−1 (Bti ) i i (a)

∈A

since each Xti is a random variable.

Thus X −1 (MT ) = X −1 (σ[ all πt−1 (Bt1 × · · · × Btk ) ∩ MT ] 1 ,...,tk = σ[ all X −1 (πt−1 (Bt1 × · · · × Btk ) ∩ MT )] 1 ,...,tk = σ[a collection of subsets of AT ] (b)

by proposition 2.1.2

by (a)

⊂ AT ,

so that X is MT -A-measurable. (⇒) This is now trivial, since X −1 (MT ) ∩ Xt−1 (B) = X −1 (MT ∩ πt−1 (B)) ∈ A for each B ∈ B. (ii) This is now clear, and it summarizes the most useful part of this theorem. 2

Remark 4.2 It is interesting to consider the case where MT is a countable or ﬁnite set. The resulting (MT , MT , P˜ ) is the natural probability space. 2

Chapter 6

General Topology and Hilbert Space 1

General Topology

Only the deﬁnitions and the statements of the major results are presented. Deﬁnition 1.1 (Topology, open set, neighborhood, and boundary) (a) A topological space (or a space) is a pair (M, U) where M is a set and U is a family of subsets that satisﬁes (1)

∪V ≡ ∪V ∈V V ∈ U, for any V ⊂ U , ∩V ∈ U for any ﬁnite V ⊂ U, and

∅ and M are in U.

U is called a topology, and the sets in U are called open sets. The complements of all open sets are called closed sets. An open set U containing a point x ∈ M is called a neighborhood of x. (b) The interior A◦ of A is int(A)≡ A◦ ≡ ∪{U : U ⊂ A with U open}. The closure A¯ of A is deﬁned by cl(A)≡ A¯ ≡ ∩{F : F ⊃ A with F closed}. The boundary ∂A of A is ∂A ≡ A¯ \ A◦ . (c) Call x an accumulation point of A if A \ {x} contains x. Exercise 1.1 (a) Show that A¯ = {x : every neighborhood U of x has U ∩ A = ∅}. ¯ while A ∩ B ⊂ A¯ ∩ B. ¯ (b) Show that A ∪ B = A¯ ∪ B, (c) Show that A◦ = M \Ac , and that (A∩B)◦ = A◦ ∩B ◦ , while (A∪B)◦ ⊃ A◦ ∪B ◦ . (d) Show that ∂A = A¯ ∩ Ac , M \ ∂A = A◦ ∪ (Ac )◦ , A¯ = A ∪ ∂A, and A◦ = A \ ∂A. Deﬁnition 1.2 (Bases and subbases) (a) Call B a base for the topology U if B ⊂ U and if each nonvoid U ∈ U satisﬁes U = ∪V for some V ⊂ B. We then say that B generates U. (b) A family of sets S is a subbase for U if ∪S = M and the family of all ﬁnite intersections of members of S is a base for U. 95

96

CHAPTER 6. GENERAL TOPOLOGY AND HILBERT SPACE

Exercise 1.2 (Bases and subbases) (a) B is a base for a topology U on M ≡ ∪B if and only if for each x ∈ M and each neighborhood U ∈ U of x there exists a B ∈ B such that x ∈ B ⊂ U . (b) B is a base for some topology on M ≡ ∪B if and only if whenever B1 and B2 are members of B and x ∈ B1 ∩ B2 , there is a B3 ∈ B such that x ∈ B3 ⊂ B1 ∩ B2 . (c) Any collection S of subsets of M having ∪S = M is a subbase for a unique topology on M . Exercise 1.3 (Relative topology) Let (M, U) be a topological space. First let N ⊂ M , and then let V ≡ {U ∩ N : U ∈ U}. (a) Then V is a topology (the relative topology) for N . (b) Moreover, B ⊂ N is open (closed) with respect to V if and only if there is an A ⊂ M that is open (closed) with respect to U for which B = A ∩ N . ¯ ∩ N , while the interior (c) The closure cl(B|N ) of an arbitrary B in (N, V) equals B c c int(B|N ) of an arbitrary B in (N, V) equals ( B ) ∩ N .

Convergence Deﬁnition 1.3 (Sequences) A sequence is a function x whose domain is the natural numbers {1, 2, · · ·}, and we let xn ≡ x(n). Suppose x takes values xn in some space (M, U). Then the sequence is said to converge to some x0 ∈ M if for each neighborhood U of x0 there exists a natural number nU such that xn ∈ U for all n ≥ nU . We write xn → x0 or limn xn = x0 to denote that the sequence xn converges to x0 . Deﬁnition 1.4 (Nets) A partial ordering is a relationship R that is reﬂexive (xRx), antisymmetric (xRy and yRx implies x = y), and transitive (xRy and yRz implies xRz). A directed set is a set I under a relationship for which, for any i, j ∈ I there is a k ∈ I satisfying i k and j k. A net is any function whose domain is such a directed set I (with xi ≡ x(i)) taking values in some topological space (M, U). The net {xi }i∈I is said to converge to a value x0 in M if for each neighborhood U of x0 there is an iU ∈ I for which xi ∈ U for all i iU . [The set of all neighborhoods of a point constitutes a directed set. So does the set of all partitions of an interval.]

Compactness Deﬁnition 1.5 (Compactness) (a) A collection A of subsets of M covers the subset B of M if B ⊂ ∪A; and A is called an open cover of B if all sets A in A are open. A set A is called compact if every open cover A of A has a ﬁnite subcover V (that is, A ⊂ ∪V, where V ⊂ A contains only ﬁnitely many subsets of A). [This is called the Heine–Borel property.] (b) A ⊂ M is compact (relatively compact) if it (its closure) is compact in the relative topology. (c) The collection A has the ﬁnite intersection property if ∩F = ∅ for each ﬁnite subcollection F of subsets of A.

1.

GENERAL TOPOLOGY

97

Exercise 1.4 (a) A closed subset of a compact space (M, U) is compact. (b) Let B be a base for (M, U), and let A ⊂ M . Then A is compact if and only if every cover of A by members of the base B admits a ﬁnite subcover. (c) (Alexander’s lemma) Let S be a subbase for the topological space (M, U). Then M is a compact space if and only if each cover of M by members of the subbase S admits a ﬁnite subcover. (d) A space (M, U) is compact if and only if each family of closed sets with the ﬁnite intersection property has a nonempty intersection. (e) (M, U) is compact if and only if every net on M has at least one accumulation point in M .

Separation Deﬁnition 1.6 (Separation) (a) (M, U) is called a Hausdorﬀ space if for each x = y in M there are disjoint open sets U and V such that x ∈ U and y ∈ V . (b) (M, U) is called a normal space if for each pair of disjoint closed sets A and B there are disjoint open sets U and V having A ⊂ U and B ⊂ V . (c) (M, U) is called separable if it contains a countable subset D that is dense in M ; ¯ = M. here D is called dense if D (d) (M, U) is called ﬁrst countable if for each x ∈ M there is a family Nx of neighborhoods Nx of x such that for each neighborhood U of x we have x ∈ Nx ⊂ U for some neighborhood Nx in Nx . (e) (M, U) is called second countable or perfectly separable if the topology U admits a countable base B. Exercise 1.5 (a) A sequence with values in a Hausdorﬀ space converges to at most one point. ¯ (b) If all values xn are in A and xn → x0 , then x0 ∈ A. Exercise 1.6 (a) A perfectly separable (M, U) is separable. (b) (Lindel¨ of) An open cover of a perfectly separable space necessarily admits a countable subcover. Exercise 1.7 (a) A compact subset of a Hausdorﬀ space (M, U) is closed. (b) A compact Hausdorﬀ space is normal. (c) A space (M, U) is normal if and only if for each closed C and open U with C ⊂ U there exists an open V with C ⊂ V ⊂ V¯ ⊂ U . (d) A subset of a Hausdorﬀ (normal) space (M, U) is Hausdorﬀ (normal) in its relative topology.

Continuity Deﬁnition 1.7 (Continuous functions and homeomorphisms) (a) Let f : M → N for topological spaces (M, U) and (N, V). Call the function (or mapping) f continuous at x if for each neighborhood V of f (x) there is a neighborhood U of x for which f (U ) ⊂ V .

98

CHAPTER 6. GENERAL TOPOLOGY AND HILBERT SPACE

(b) A 1-to-1 function that maps onto the image space is called a bijection. (c) A 1-to-1 bicontinuous function f (that is, both f and f −1 are continuous and onto) is called a homeomorphism. Exercise 1.8 (Conditions for continuity)

The following are equivalent:

(a)

f is a continuous function.

(b)

Inverse images of all open sets are open.

(c)

Inverse images of all closed sets are closed.

(d)

Inverse images of all subbasic open sets are open.

(e)

For each x ∈ M and for each net {xi }i∈I converging to this x, {f (xi )}i∈I → f (x) also converges.

(f)

f (A¯ ) ⊂ f (A)

(g)

¯ for all subsets B of N . f −1 (B) ⊂ f −1 (B)

(h)

f −1 (B ◦ ) ⊂ [f −1 (B)]◦

for all subsets A of M.

for all subsets B of N .

Exercise 1.9 (a) (Dini) If fn are continuous real-valued functions on a compact (M, U) for which fn (x) f (x) for each x ∈ M , then fn converges uniformly to f . (b) A uniform limit of bounded and continuous functions fn is also bounded and continuous. Exercise 1.10 (Continuity and compactness) (a) If f : M → N is continuous and M is compact, then f (M ) is compact. [Thus [0, 1] cannot be mapped continuously onto R.] (b) The composition of continuous functions is continuous. (c) If f : M → N is 1-to-1 and continuous (that is, it is a bijection) where M is compact and N is Hausdorﬀ, then f is a homeomorphism. Exercise 1.11 (Urysohn’s lemma) (M, U) is normal if and only if for each pair of disjoint closed sets A and B there is a continuous function f : M → [0, 1] having f (x) = 0 for all x ∈ A and f (x) = 1 for all x ∈ B. Exercise 1.12 (Tietze) Let (M, U) be a normal space, and let A be a closed subset. If the function f : A → [−1, 1] is continuous, then f admits a continuous extension from M to [−1, 1].

1.

GENERAL TOPOLOGY

99

Product Spaces Deﬁnition 1.8 (Product topology) Let P {Ma } ≡ a∈A Ma denote the product set associated with the individual topological spaces (Ma , Ua ) over the index set A; this product set is deﬁned to be {x : x is a function on A with xa ≡ x(a) ∈ Ma }. The a0 th projection function πa0 : P {Ma } → Ma0 is deﬁned by πa0 (x) = xa0 . Let S ≡ {πa−1 (U ) : U ∈ Ua and a ∈ A}. Then S is the subbase for a topology called the product topology. Exercise 1.13 (a) Projection functions are continuous in the product topology. (Recall deﬁnition 5.4.1 for the projection maps πt1 ,...,tk (·).) (b) (Tychonoﬀ) P {Ma } with the product topology is compact if and only if each (Ma , Ua ) is compact. (c) The product of countably many separable topological spaces is a separable space in the product topology. ¯a } for all choices Ba ⊂ Ma , and thus the product of closed sets (d) P {Ba } = P {B is closed in the product topology. (e) (P {Ba })◦ = P {Ba◦ } for all choices Ba ⊂ Ma , provided that A is ﬁnite. (f) If each (Ma , Ua ) is Hausdorﬀ, then the product topology is Hausdorﬀ. (g) Let (M, U) be a topological space and suppose fa : M → Ma is a continuous function for each a ∈ A. Deﬁne f : M → P {Ma } by πa f (x) = fa (x) for each x ∈ M . Then f is continuous in the product topology.

Local Compactness Deﬁnition 1.9 (Locally compact) Let (M, U) be a topological space. Let x ∈ M . The space is locally compact at x if x has a neighborhood with compact closure (that is, for each U ∈ U having x ∈ U there exists a V ∈ U with x ∈ V ⊂ U with V¯ compact). The space is locally compact if it is locally compact at each point. [Rn and (0, 1) are locally compact.] Exercise 1.14 (Locally compact Hausdorﬀ spaces are nice) Let M be a locally compact Hausdorﬀ space. Let C ⊂ U ⊂ M , where C is compact and U is open. (a) Then there is an open set V whose closure is compact having C ⊂ V ⊂ V¯ ⊂ U . (b) (Urysohn’s lemma) There is a continuous function f : M → [0, 1] for which f (x) equals 1 for x ∈ C and 0 for x ∈ U c . (c) There is an open set V and a compact set D where C ⊂ V ⊂ D ⊂ U with V a countable union of compact sets and D a countable intersection of open sets. Sets of the type V form a base for the topology. Exercise 1.15 (One-point compactiﬁcation) The one-point compactiﬁcation of any locally compact Hausdorﬀ space (M, U) makes it a compact Hausdorﬀ space. ¯ ≡ M ∪ {p}, and designate the open sets U ¯ to be all open sets U That is, let M c ¯ , U) ¯ is plus all sets {p} ∪ C with C compact and {p} the one new point. Then (M a compact Hausdorﬀ space.

100

CHAPTER 6. GENERAL TOPOLOGY AND HILBERT SPACE

Category Deﬁnition 1.10 (Category) A subset A of (M, U) is nowhere dense if (A¯ )◦ = ∅. If A can be written as a countable union of nowhere dense sets, then it is said to be of the ﬁrst category; otherwise, it is said to be of the second category. Exercise 1.16 (Baire category theorem) second category.

A locally compact Hausdorﬀ space is

Topological Properties Deﬁnition 1.11 (Topologically equivalent) If there exists a homeomorphism between the two topological spaces (M, U) and (N, V) (that is, there exists a 1-to-1 bicontinuous mapping between them), then there exist 1-to-1 correspondences both between points and between open sets. We then say that these two topological spaces are topologically equivalent. Thus any properties that are deﬁned solely in terms of the open sets either hold or fail simultaneously in the two spaces. Such concepts are called topological concepts. [These include closed set, closure, interior, boundary, accumulation point, compactness, separability, perfect separability, local compactness, category, continuity, and being either Hausdorﬀ or normal.] We are about to introduce metric spaces. We will soon see that two diﬀerent metrics can induce the same topology. Thus distance in a metric space is not a topological property. A homeomorphism f that leaves distance unchanged (thus, d2 (f (x), f (y)) = d1 (x, y)) is called an isometry.

2.

METRIC SPACES

2

Metric Spaces

101

Deﬁnition 2.1 (Metric space) (a) A semimetric space (M, d) consists of a set M together with a semimetric d that satisﬁes (1)

d(x, y) = d(y, x) ≥ 0,

d(x, z) ≤ d(x, y) + d(y, z),

and d(x, x) = 0

for all x, y, z ∈ M . If d(x, y) = 0 implies x = y, then d is called a metric and (M, d) is called a metric space. [If all d(x, y) ≤ (some ﬁnite c), then d is called a bounded metric.] Call the set (2)

Sr (x) ≡ {y ∈ M : d(x, y) < r} an open sphere or ball about x of radius r.

(b) The collection of all such spheres is the base for a topology, called the metric topology, and is denoted by Ud . The abbreviation (M, d, Ud ) refers to the metric space with its metric topology, and we call it a topological metric space. (c) Let (M, U) be a topological space. Suppose there exists a metric d on M whose open balls form a base for the topology U. Then the topology U is said to be metrizable. Exercise 2.1 (Equivalent metrics) (a) Two metrics d and ρ on a set M lead to equivalent topologies if and only if for each x ∈ M and each > 0 there is a δ > 0 such that for all y ∈ M , d(x, y) < δ implies ρ(x, y) <

and

ρ(x, y) < δ implies d(x, y) < .

[If only the ﬁrst holds, then Ud ⊂ Uρ , meaning that the ρ topology is ﬁner.] Such equivalent metrics lead to the same continuous functions. If both sets of δ values do not depend on x, then the metrics are called uniformly equivalent metrics and lead to the same uniformly continuous functions. (b) Deﬁne four equivalent metrics on Rn . (c) ρ ≡ d/(1 + d) deﬁnes a metric equivalent to the d metric on (M, d); and ρ is a bounded metric. Exercise 2.2 (a) The closed sphere {y : d(x, y) ≤ r} equals Sr (x) with respect to the metric topology Ud . (b) Any metric space is homeomorphic to a metric space with a bounded metric. (c) A metric space is perfectly separable if and only if it is separable. (d) A compact metric space is separable and perfectly separable. (e) Any metric space is Hausdorﬀ, normal, and ﬁrst countable. (f) The product of countably many metrizable in the product spaces is metrizable n topology, with the bounded metric d(x, y ) ≡ ∞ 1 ψ(dn (xn , yn ))/2 , where we deﬁne ψ(t) ≡ t/(1 + t). If each is separable, then so is the product. (g) The relative topology on a subset A of the metric space (M, d, Ud ) is the same as the metric topology for (A, d). (h) Every subspace of a separable metric space is separable in its relative topology. (i) A separable metric space is locally compact if and only if it is the union of U1 ⊂ U2 ⊂ · · · with each U¯n a compact subset of Un+1 .

102

CHAPTER 6. GENERAL TOPOLOGY AND HILBERT SPACE

Completeness Deﬁnition 2.2 (Complete) (a) We will call a sequence x in a metric space (M, d) a Cauchy sequence if d(xm , xn ) → 0 as m ∧ n → ∞. (b) The topological metric space (M, d, Ud ) is called complete when every Cauchy sequence x having values xn in M converges to a member of M . Exercise 2.3 (a) A sequence x taking values in a metric space (M, d) converges to a member x0 of M if and only if d(xn , x0 ) → 0. (b) Every compact metric space is complete, separable, and perfectly separable. (c) Every closed subspace of a complete metric space is complete. (d) A metric space in which every sequence has a convergent subsequence is perfectly separable.

Compactness Deﬁnition 2.3 (a) A metric space (M, d) is totally bounded if for every > 0 there is a ﬁnite subset F of M such that for each x ∈ M there exists a y ∈ F suﬃciently close to x to have d(x, y) < . (b) Let diam(A) ≡ sup{d(x, y) : x, y ∈ A} denote the diameter of A. The metric space is called bounded if diam(M ) is ﬁnite. Exercise 2.4 (Compactness) The following conditions are equivalent to the compactness of a subset K of the metric space (M, d, Ud ). (a)

(Heine–Borel property = compactness, by deﬁnition): Every open cover of K has a ﬁnite subcover.

(b)

(Bolzano–Weierstrass property): Every inﬁnite subset of K has a limit point in K.

(c)

(Sequential compactness (or relative compactness)): Every sequence in K has a subsequence converging to a point in K.

(d)

(Countable compactness): Every countable open cover of K has a ﬁnite subcover.

(e)

K is totally bounded and complete.

(f)

K is totally bounded, and any sequence of closed spheres Sn in K whose diameters → 0 has ∞ 1 Sn = {some singleton x} ∈ K.

Exercise 2.5

A subset of Rn is compact if and only if it is closed and bounded.

Exercise 2.6 If the closed subset A and the compact subset B are disjoint, then we have d(A, B) ≡ inf{d(x, y) : x ∈ A, y ∈ B} > 0.

2.

METRIC SPACES

103

Deﬁnition 2.4 (Covering numbers) Let (M, d) be a totally bounded metric space. Let > 0. Let N ( , M ) denote the minimum n for which M = ∪n1 Ak for sets Ak all having diam(Ak ) ≤ 2 . Let D( , M ) be the largest number m of points xk in M having d(xi , xj ) > for all i = j. These are called covering numbers. Exercise 2.7

Show that N ( , M ) ≤ D( , M ) ≤ N ( /2, M ).

Spaces of Functions Exercise 2.8 (Uniform norm) Let Cb (M ) denote the collection of all bounded and continuous functions from the topological space (M, U) to R, and deﬁne the uniform norm · on the functions of Cb (M ) by f ≡ sup{|f (x)| : x ∈ M }. (a) A uniform limit of bounded, continuous functions is bounded and continuous. (b) Moreover, (Cb (M ), · ) deﬁnes a complete metric space. (c) Let (M, U) be a compact Hausdorﬀ space. Then (Cb (M ), · ) is separable if and only if (M, U) is metrizable. Deﬁnition 2.5 (Uniform continuity) A mapping f from a metric space (M1 , d1 ) to a metric space (M2 , d2 ) is uniformly continuous if for all > 0 there exists δ > 0 for which d2 (f (x), f (y)) < whenever d1 (x, y) < δ . Deﬁnition 2.6 (Equicontinuity) (a) A collection F of bounded and continuous functions from a topological space (M, U) to a metric space (N, d) is equicontinuous if for every > 0 and each x ∈ M there is a neighborhood U of x for which d(f (x), f (y)) < for all y ∈ U and all f ∈ F. (b) In fact, if (M, ρ, U = Uρ ) is a metric space for which d(f (x), f (y)) < for all f ∈ F whenever ρ(x, y) < δ , then the functions of F are called uniformly equicontinuous. Exercise 2.9 A continuous function (an equicontinuous family of functions) from a compact metric space into another metric space is uniformly continuous (is uniformly equicontinuous). Exercise 2.10 (Ascoli’s theorem) Let F be an equicontinuous class of functions from a separable metric space (M, d) to a metric space (N, ρ). Let fn denote a sequence of such functions for which cl({fn (x) : n ≥ 1}) is compact for each x ∈ M . Then there is a subsequence fn that converges pointwise to an f that is continuous on M , and this convergence is uniform on any compact subset K of M . Exercise 2.11 (Arzel` a’s theorem) (a) A collection F of real-valued, bounded, and continuous functions f on a compact topological space (M, U) is a totally bounded subset of (Cb (M ), ·) if and only if it is a collection of uniformly bounded and uniformly equicontinuous functions. (b) A subset F of (Cb ([a, b]), · ) is compact if and only if F is closed, bounded, and equicontinuous with respect to · .

104

3

CHAPTER 6. GENERAL TOPOLOGY AND HILBERT SPACE

Hilbert Space

Deﬁnition 3.1 (i) (Vector space) Let V be a real (or a complex) vector space; that is, x + y ∈ V and c x ∈ V for all x, y in the commutative group V and for all real scalars c ∈ R (or all complex scalars c ∈ C). Moreover, c(x + y) = cx + cy, (c + d)x = cx + dx, c(dx) = (cd)x and 1x = x must be satisﬁed. (ii) (Inner product space) A vector space H is an inner product space if for each pair x, y ∈ H there is a real (or a complex) number x, y ≡ (the inner product of x and y) satisfying: (a)

c x, y = cx, y

for all x, y ∈ H, and all c ∈ R

(b)

y, x = x, y

( or satisfying, y, x = x, y ),

(c)

x + y, z = x, z + y, z

(d)

x, x ≥ 0

( or all c ∈ C ),

for all x, y, z ∈ H,

for all x ∈ H, with equality only if x = 0.

(iii) (Norm) We use (d) to deﬁne the norm (denoted by · ) of x; thus (e) so that x = x, x. x2 ≡ x, x , (iv) (Orthogonal, and orthogonal complement) We agree that (f)

x⊥y

means that x, y = 0,

and we then say that x is orthogonal to y. We also let (g)

M ⊥ ≡ {x : x ⊥ y for all y ∈ M },

and we call it the orthogonal complement of M . (v) (Hilbert space) If H is an inner product space that is complete (with respect to the metric deﬁned by the norm x − y ), then H is a Hilbert space. Let µ denote a ﬁxed positive measure. Example 3.1 (L2 is a Hilbert space) Then let L2 (µ) ≡ {f : f 2 dµ < ∞}, which is a Hilbert space with inner product f, g = f g dµ. Thus for any probability measure P on a measurable space (Ω, A), the collection of rvs L2 (P ) is a Hilbert space with X, Y ≡ E(XY ). This follows from the completeness that was presented in exercise 3.5.1. 2 Proposition 3.1 (Elementary properties) Let x, y ∈ H. (a)

(Cauchy–Schwarz)

|x, y| ≤ x y.

(b)

(Triangle inequality)

x + y ≤ x + y.

(c)

(Pythagoras theorem)

x + y2 = x2 + y2

(d)

(Parallelogram law)

x + y2 + x − y2 ≤ 2 [ x2 + y2 ].

(e)

(Polarization identity)

4x, y = x + y2 − x − y2

(f)

(Linear functionals) x → x, y, x → y, x, are uniformly continuous linear functionals on H.

(g)

The only continuous linear functionals on H are those in (f).

if x ⊥ y.

when c’s are real.

and x → x

3.

HILBERT SPACE

105

Deﬁnition 3.2 (Subspaces) We say that M ⊂ H is a subspace of the vector space H if M is also a vector space. [It is enough if x + y and c x are in M for all x, y ∈ M and all scalars c.] We say that M ⊂ H is a closed subspace if it is a subspace that contains all of its own limit points. Exercise 3.1 Let M be a subspace of the Hilbert space H. (a) Then M ⊥ is a closed subspace of M . (b) (M ⊥ )⊥ = M . The following theorem gives fundamental properties of a general Hilbert space. Theorem 3.1 (Orthogonal projections) Let M be a closed subspace of H. (i) Unique mappings P and Q on H exist such that (1)

x = P x + Qx

for all x ∈ H

with P x ∈ M and Qx ∈ M⊥ .

This P and Q are called the orthogonal projections of H onto M and M⊥ . (ii) Speciﬁcally and additionally, P and Q satisfy: (2)

x2 = P x2 + Qx2 ,

(3)

x − P x = inf{ x − y : y ∈ M}

(4)

P and Q are linear maps.

for each x ∈ H,

(iii) If M is a proper subset of H, then there exists a y = 0 in H such that y ⊥ M . Moreover, the space spanned by M and y is closed. Exercise 3.2

Prove theorem 3.1.

Example 3.2 (HD is a closed subspace of H) Let (5)

HD ≡ {X ∈ L2 (P ) : F(X) ⊂ D},

where D denotes a ﬁxed sub σ-ﬁeld of the basic probability space (Ω, A, P ). Then exercise 3.5.1 shows that HD is a closed subspace of H ≡ L2 (P ). 2 Exercise 3.3

Present the details for the assertion of example 3.2.

Deﬁnition 3.3 (Independent, span, orthonormal, Fourier coeﬃcient, and basis) (i) Call vectors {x1 , . . . , xn } in V linearly independent if c1 x1 + · · · + cn xn = 0 implies c1 = · · · = cn = 0. Call the set Z linearly independent if all of its ﬁnite subsets are linearly independent. Let the span of Z (denoted by S[Z]) denote the set of all ﬁnite linear combinations of elements of the set Z. (ii) If Z ≡ {za : a ∈ A} is a subset of H indexed by A, then the elements of Z are called orthonormal if za , zb = 0 for all a, b ∈ A, while za = 1 for all a ∈ A. Moreover, the values (6)

xA (a) ≡ x, za , for all a ∈ A, are called the Fourier coeﬃcients of x

relative to Z . A maximal orthonormal set is called an orthonormal basis.

106

CHAPTER 6. GENERAL TOPOLOGY AND HILBERT SPACE

Theorem 3.2 (Coordinates) Let vectors {z1 , . . . , zn } be orthonormal in the Hilbert space H. n (i) If x = 1 ci zi , then for each 1 ≤ k ≤ n, (7)

ck = x, zk , the z1 , . . . , zn are linearly independent, and x2 =

n

2 1 |ck |

.

(ii) For each x ∈ H and scalars c1 , . . . , cn we have n n x − 1 x, zk zk ≤ x − 1 ck zk , (8) with equality if and only if ck = x, zk . Moreover, n (9) 1 x, zk zk is the orthogonal projection of x onto the span S[z1 , . . . , zn ]. n Call d(x, S[z1 , . . . , zn ]) ≡ inf{x − 1 ck zk : ck ’s are scalars} the distance from x to the span S[z1 , . . . , zn ]; then (10) d2 (x, S[z1 , . . . , zn ]) = x2 − n1 |x, zk |2 . (iii) (Bessel’s inequality) For any orthonormal set {z1 , . . . , zn }, 2 2 2 (11) a∈A |xA (a)| = a∈A |x, za | ≤ x . Thus each x ∈ H has at most a countable number of nonzero Fourier coeﬃcients. (iv) Let 2 (A) denote L2 (A, 2A , counting measure). We rewrite Bessel’s inequality as xA ∈ 2 (A) with xA 2 ≤ x. (Then, of course, xA − yA 2 ≤ x − y.) Theorem 3.3 (Orthonormal bases) The following are equivalent:

Let {za : a ∈ A} be an orthonormal set.

(a)

{za : a ∈ A} is an orthonormal basis,

(b)

The span of these za ’s is · -dense in H, x2 = a∈A |xA (a)|2 for all x ∈ H, (Parseval’s identity) x, y = a∈A xA (a) yA (a)

(c) (d)

for all x, y ∈ H.

Theorem 3.4 (H is isomorphic to 2 ) (a) Suppose the Hilbert space H has an orthonormal basis {za : a ∈ A}. Then the mapping x → xA maps H onto 2 (A). And it is an isomorphism in that it preserves inner products via the correspondence (12) x, y = xA , yA 2 ≡ xA (a)yA (a) for all x, y ∈ H. a∈A

(b) Every Hilbert space possesses an orthonormal basis. Thus every Hilbert space H is isomorphic to some 2 (A). (c) Every orthonormal basis for the Hilbert space H has the same cardinality. (d) H is separable if and only if H contains an orthonormal basis that is ﬁnite or countably inﬁnite. Exercise 3.4 (Gram–Schmidt) From any n linearly independent vectors one can deﬁne n orthonormal vectors that have the same span.

Chapter 7

Distribution and Quantile Functions 1

Character of Distribution Functions

Let X : (Ω, A, P ) → (R, B) denote a rv with distribution function (df ) FX , where (1)

FX (x) ≡ P (X ≤ x)

for − ∞ < x < ∞.

Then F ≡ FX was seen earlier to satisfy (2)

F is and right continuous, with F (−∞) = 0 and F (+∞) = 1.

Because of the following proposition, any function F satisfying (2) will be called a df. [If F is , right continuous, 0 ≤ F (−∞), and F (+∞) ≤ 1, we earlier agreed to call F a sub-df. As usual, F (a, b] ≡ F (b) − F (a) denotes the increments of F , and ∆F (x) ≡ F (x) − F− (x) is the mass of F at x.] Proposition 1.1 (There exists an X with df F ) If F satisﬁes (2), then there exists a probability space (Ω, A, P ) and a rv X : (Ω, A, P ) → (R, B) for which the df of X is F . We write X ∼ = F. Proof. The corollary to the correspondence theorem (theorem 1.3.1) shows that there exists a unique probability distribution P on (Ω, A) ≡ (R, B) for which P ((a, b]) = F (b) − F (a) for all a ≤ b. Now deﬁne X(ω) = ω for all ω ∈ R to be the identity function on R. 2 Theorem 1.1 (Decomposition of a df) (3)

Any df F can be decomposed as

F = Fd + Fc = Fd + Fs + Fac = (Fd + Fs ) + Fac ,

where Fd , Fc , Fs , and Fac are the unique sub-dfs of the following types (unique among those sub-dfs equal to 0 at −∞): (4) Fd is a step function of the form j bj 1[aj ,∞) (with all bj > 0). 107

108

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

(5)

Fc is continuous.

(6)

Fs is singular, with its measure orthogonal to Lebesgue measure. · Fac = −∞ fac (y) dy for some fac ≥ 0 that is ﬁnite, measurable, and unique a.e. λ.

(7)

Proof. Let {aj } denote the set of all discontinuities of F , which can only be jumps; and let bj ≡ F (aj ) − F− (aj ). There can be only a countable number of jumps, since the number of jumps of size exceeding size 1/n is certainly bounded by n. Now deﬁne Fd ≡ j bj 1[aj ,∞) , which is obviously and right continuous, since Fd (x, y] ≤ F (x, y] 0 as y x (the inequality holds, since the sum of jump sizes over every ﬁnite number of jumps between a and b is clearly bounded by F (x, y], and then just pass to the limit). Deﬁne Fc = F − Fd . Now, Fc is , since for x ≤ y we have Fc (x, y] = F (x, y] − Fd (x, y] ≥ 0. Now, Fc is the diﬀerence of right-continuous functions, and hence is right continuous; it is left continuous, since for x y we have (a) Fc (x, y] = F (x, y] − x
∆Fd (a) − ∆Gd (a) = 0,

contradicting the continuity of Gc − Fc = Fd − Gd . Thus Gd = Fd , and hence Fc = Gc . This completes the ﬁrst decomposition. We now turn to the further decomposition of Fc . Associate a measure µc with Fc via µc ((−∞, x]) = Fc (x). Then the Lebesgue decomposition theorem shows that µc = µs + µac , where µs (B) = 0 and µac (B c ) = 0 for some B ∈ B; we say that µs and µac are singular. Moreover, this same Lebesgue theorem implies the claimed uniqueness and shows x that fac exists with the uniqueness claimed. Now, Fac (x) ≡ µac ((−∞, x]) = −∞ fac (y) dy is continuous by Fac (x, y] ≤ µac (x, y] → 0 as y → x or as x → y. Thus Fs ≡ Fc − Fac is continuous, and Fs (x) = µs ((−∞, x]). 2

Example 1.1 (The Lebesgue singular df) Deﬁne the Cantor set C by ∞ (8) C ≡ {x ∈ [0, 1] : x = n=1 2an /3n , with all an equal to 0 or 1}. [Thus the Cantor set is obtained by removing from [0, 1] the open interval ( 13 , 23 ) at stage one, then the open intervals ( 19 , 29 ) and ( 79 , 89 ) at stage two, . . . .] Finally, we deﬁne F on C by ∞ ∞ (9) F ( n=1 2an /3n ) = n=1 an /2n .

1.

CHARACTER OF DISTRIBUTION FUNCTIONS

109

1 7 8 3 4 5 8 1 2 3 8 1 4 1 8 0 0

1 9

2 9

1 3

2 3

7 9

8 9

1

Figure 1.1 Lebesgue singular function.

Now note that {F (x) : x ∈ C} = [0, 1], since the right-hand side of (9) represents all of [0, 1] via dyadic expansion. We now deﬁne F linearly on C c (the ﬁrst three components are shown in ﬁgure 1.1 above). Since the resulting F is and achieves every value in [0, 1], it must be that F is continuous. Now, F assigns no mass to the 1/3 4 “ﬂat spots” whose lengths sums to 1 since 13 + 29 + 27 + · · · = 1−2/3 = 1. Thus F c is singular with respect to Lebesgue measure λ, using λ(C ) = 1 and µF (C c ) = 0. We call this F the Lebesgue singular df. [The theorem in the next section shows that removing the ﬂat spots does, even for a general df F , leave only the essentials.] 2 Exercise 1.1 Let X ∼ = N (0, 1) (as in (9.1.22) below), and let Y ≡ 2X. (a) Is the df F (·, ·) of (X, Y ) continuous? (b) Does the measure µF on R2 have a density with respect to two-dimensional Lebesgue measure? [Hint. Appeal to corollary 2 to Fubini’s theorem.] Deﬁnition 1.1 Two rvs X and Y are said to be of the same type if Y ∼ = aX + b for some a > 0. Their dfs are also said to be of the same type.

110

2

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

Properties of Distribution Functions

Deﬁnition 2.1 The support of a given df F ≡ FX is deﬁned to be the minimal closed set C having P (X ∈ C) = 1. A point x is a point of increase of F if every open interval U containing x has P (X ∈ U ) > 0. A realizable t-quantile of F , for 0 < t < 1, is any value z for which F (z) = t. Theorem 2.1 (Jumps and ﬂat spots) Let 0 ≤ t ≤ 1. Let Ut denote the maximal open interval of x’s for which F (x) = t. The set of t’s for which these Ut ’s are nonvoid is at most countable (these are exactly those t’s that have more than one realizable t-quantile). Moreover: (a) C ≡ ( 0≤t≤1 Ut )c is a closed set having P (C) = 1. (b) C is equal to the set of all points of increase. (c) C is the support of F . (d) F has at most a countable number of discontinuities, and these discontinuities are all discontinuities of the jump type. (e) F has an at most countable number of ﬂat spots (the nonvoid Ut ’s). [We will denote jump points and jump sizes of F by ai ’s and bi ’s. The t values and the λ(Ut ) values of the multiply realizable t-quantiles will be seen in the proof of proposition 7.3.1 below to correspond to the jump points cj and the jump values dj of the function K(·) ≡ F −1 (·), and there at most countably many of them.] Proof. (a) For each t there is a maximal open set Ut (possibly void) on which F equals t. Now, P (X ∈ Ut ) = 0 using proposition 1.1.2. Note that C ≡ (∪t Ut )c is closed (since the union of an arbitrary collection of open sets is open). Hence C c = ∪0
3.

THE QUANTILE TRANSFORMATION

111

3

The Quantile Transformation

Deﬁnition 3.1 (Quantile function) For any df F (·) we deﬁne the quantile function (qf ) (which is the inverse of the df) by (1)

K(t) ≡ F −1 (t) ≡ inf{x : F (x) ≥ t}

for 0 < t < 1.

K(·)

F(·)

Figure 3.1 The df F(·) and the qf K(·) = F –1(·).

Theorem 3.1 (The inverse transformation)

Let

X ≡ K(ξ) ≡ F −1 (ξ),

where ξ ∼ = Uniform(0, 1).

(3)

[X ≤ x] = [ξ ≤ F (x)]

for every real x,

(4)

1[X≤·] = 1[ξ≤F (·)]

on R, for every ω,

(5)

X ≡ K(ξ) ≡ F −1 (ξ)

has df F ,

(6)

1[X<·] = 1[ξ
on R, for a.e. ω.

(2) Then

112

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

Proof. Now, ξ ≤ F (x) implies X = F −1 (ξ) ≤ x by (1). If X = F −1 (ξ) ≤ x, then F (x + ) ≥ ξ for all > 0; so right continuity implies F (x) ≥ ξ. Thus (3) holds; (4) and (5) are then immediate. If ξ(ω) = t where t is not in the range of F , then (6) holds. If ξ(ω) = t where F (x) = t for exactly one x, then (6) holds. If ξ(ω) = t where F (x) = t for at least two distinct x’s, then (6) fails; theorem 7.2.1 shows that this can happen for at most a countable number of t’s. (Or: Graph a df F that exhibits the three types of points t, and the rest is trivial with respect to (6), since the value of F at any other point is immaterial. Speciﬁcally, (6) holds for ω unless F has a ﬂat spot at height t ≡ F (ξ(ω)). Note ﬁgure 3.1.) 2

Deﬁnition 3.2 (Convergence in quantile) Let Kn denote the qf associated with df Fn , for each n ≥ 0. We write Kn →d K0 to mean that Kn (t) → K0 (t) at each continuity point t of K0 in (0, 1). We then say that Kn converges in quantile to K0 , or Kn converges in distribution to K. Proposition 3.1 (Convergence in distribution equals convergence in quantile) (7)

Fn →d F

if and only if

Kn →d K.

Proof. Suppose Fn →d F . Let t ∈ (0, 1) be such that there is at most one value x having F (x) = t (that is, there is not a multiply realizable t-quantile). Let z ≡ F −1 (t). First: We have F (x) < t for x < z. Thus Fn (x) < t for n ≥ (some Nx ), provided that x < z is a continuity point of F . Thus Fn−1 (t) ≥ x for n ≥ Nx , provided that x < z is a continuity point of F . Thus lim inf Fn−1 (t) ≥ x, provided that x < z is a continuity point of F . Thus lim inf Fn−1 (t) ≥ z, since there are continuity points x that z. Second: We also have F (x) > t for z < x. Thus Fn (x) > t, and hence Fn−1 (t) ≤ x for n ≥ (some Nx ), provided that x > z is a continuity point of F . Thus lim sup Fn−1 (t) ≤ x, provided that x > z is a continuity point of F . Thus lim sup Fn−1 (t) ≤ z, since there are continuity points x that z. Summary Fn−1 (t) → F −1 (t) for all but at most a countably inﬁnite number of t’s (namely, for all but those t’s that have multiply realizable t-quantiles; these correspond to the heights of ﬂat spots of F , and these ﬂat spot heights t are exactly the discontinuity points of K). That is, Kn →d K. The proof of the converse is virtually identical. 2

Exercise 3.1 (Left continuity of K) Show that K(t) = F −1 (t) is left continuous on (0, 1). [Note that K is discontinuous at t ∈ (0, 1) if and only if the corresponding Ut is nonvoid (see theorem 7.2.1). Likewise, the jump points cj and the jump sizes dj of K(·) are equal to the t values and the λ(Ut ) values of the multiply realizable t-quantiles.] [We earlier agreed to use ai and bi for the jump points and jump sizes of the associated df F .]

3.

THE QUANTILE TRANSFORMATION

Exercise 3.2 (Properties of dfs) F ◦ F −1 (t) ≥ t

113

(i) For any df F we have

for all 0 ≤ t ≤ 1,

and equality fails if and only if t ∈ (0, 1) is not in the range of F on [−∞, ∞]. (ii) (The probability integral transformation) If X has a continuous df F , then F (X) ∼ = Uniform(0, 1). In fact, for any df F , P (F (X) ≤ t) ≤ t

for all 0 ≤ t ≤ 1 ,

with equality failing if and only if t is not in the closure of the range of F . (iii) For any df F we have F −1 ◦ F (x) ≤ x

for all − ∞ < x < ∞ ,

and equality fails if and only if F (y) = F (x) for some y < x. Thus P (F −1 ◦ F (X) = X) = 0

whenever X ∼ = F.

(iv) If F is a continuous df and F (X) ∼ = Uniform(0, 1), then X ∼ = F. Proposition 3.2 (The randomized probability integral transformation) Let X denote an arbitrary rv. Let F denote its df, and let (aj , bj )’s denote an enumeration of whatever pairs (jump point, jump size) the df F possesses. Let η1 , η2 , . . . denote iid Uniform(0, 1) rvs (that are also independent of X). Then (8) ξ˙ ≡ F (X) − j bj ηj 1[X=aj ] ∼ = Uniform(0, 1), (9)

˙ = K(ξ). X = F −1 (ξ)

[We have reproduced X from a Uniform(0, 1) rv that was deﬁned using both X and some independent extraneous variation. Note ﬁgure 3.1.] Proof. We have merely smoothed out the mass bj that F (X) placed at F (aj ) by subtracting the random fractional amount ηj bj of the mass bj . 2 Exercise 3.3 (Change of variable) Suppose that rvs X ∼ = F and Y ∼ = G are −1 related by G(H) = F and X = H (Y ), where H is right continuous on the real line with left-continuous inverse H −1 . (a) Then set g, X, µ, µX , A in the theorem of the unconscious statistician equal to g, H −1 , G, F, (−∞, x] to conclude that (10) g(H −1 ) dG = g dF, (−∞,H(x)]

(−∞,x]

since (H −1 )−1 ((−∞, x]) = {y : H −1 (y) ≤ x} = (−∞, H(x)]. (b) Making the identiﬁcations G = I, H = F , and Y = ξ ∼ = Uniform(0, 1) gives especially (via part (a), or via (2) and (3)) −1 (11) g(F (t)) dt = g dF [0,F (x)]

(−∞,x]

for arbitrary df F and any measurable g.

114

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

Proof. We now prove proposition 1.2.3. Let D be a subset of [0, 1] that is not Lebesgue measurable; its existence is guaranteed by proposition 1.2.2. Let B ≡ F −1 (D). Then B is a subset of the Cantor set C of example 7.1.1. Since λ(C) = 0 and B ⊂ C, it follows that B is a Lebesgue set with λ(B) = 0. We now assume that B is Borel set (and look for a contradiction). We note that F (B) is also a Borel set, since F (being ) is Borel measurable. However, F is 1-to-1 on C, and so F (B) = D. That is, D is a Borel set, and hence D is a Lebesgue set. This is the contradiction we sought. 2

The Elementary Skorokhod Construction Theorem Let X0 , X1 , X2 , . . . be iid F . Then Xn →d X0 , but the Xn do not converge to X0 in the sense of →a.s. , →p , or →r . However, when general Xn →d X0 , it is possible to replace the Xn ’s by rvs Yn having the same (marginal) dfs, for which the stronger result Yn →a.s. Y0 holds. Theorem 3.2 (Skorokhod) Suppose that Xn →d X0 . Deﬁne ξ(ω) = ω for each ω ∈ [0, 1] so that ξ ∼ = Uniform(0, 1) on (Ω, A, P ) ≡ ([0, 1], B ∩ [0, 1], λ), for Lebesgue measure λ. Let Fn denote the df of Xn , and deﬁne Yn ≡ Fn−1 (ξ) for all n ≥ 0. Let DK0 denote the at most countable discontinuity set of K0 . Then both Yn ≡ Kn (ξ) ≡ Fn−1 (ξ) ∼ = Xn ∼ = Fn

for all n ≥ 0

and

(12) Yn (ω) → Y0 (ω) Proof.

for all ω ∈ DK0 .

This follows trivially from proposition 3.1.

2

Exercise 3.4 (Wasserstein distance) For k = 1 or 2, deﬁne Fk ≡ {F : F is a df, and |x|k dF (x) < ∞} , dk (F1 , F2 ) ≡

1 0

|F1−1 (t) − F2−1 (t)|k dt

for all F1 , F2 ∈ Fk .

Show that both (Fk , dk ) spaces are complete metric spaces, and that (13)

dk (Fn , F0 ) → 0 Fn →d F0

and

(with all {Fn }∞ 0 ∈ Fk ) k |x| dFn (x) → |x|k dF0 (x) .

if and only if

4.

INTEGRATION BY PARTS APPLIED TO MOMENTS

4

Integration by Parts Applied to Moments

115

Integration by Fubini’s theorem or “integration by parts” formulas are useful in many contexts. Here we record a few of the most useful ones.

Integration by Parts Proposition 4.1 (Integration by parts formulas) Suppose that both the leftcontinuous function U and the right-continuous function V are monotone functions. Then for any a ≤ b we have both U+ (b)V (b) − U (a)V− (a) = (1) U dV + V dU and [a,b]

[a,b]

U (b)V (b) − U (a)V (a) =

(2)

U dV + (a,b]

V dU, [a,b)

written as where U+ (x) ≡ limy x U (y) and V− (x) ≡ lim yx V (y). [Symbolically, d(U V ) = U− dV + V+ dU , it means also that h d(U V ) = h [U− dV + V+ dU ] for any measurable h ≥ 0.] 1[x′ < y′]

b 1[x′ ≥ y′] a a

b

Figure 4.1 Integration by parts.

Proof.

We can apply Fubini’s theorem at steps (a) and (b) to obtain [U+ (b) − U (a)] [V (b) − V− (a)] = [a,b] { [a,b] dU } dV

(a)

=

(b)

= =

[a,b]

[a,b]

[1[x
[a,b]

[U (y) − U (a)] dV (y) +

[a,b]

U dV − U (a) [V (b) − V− (a)] +

[a,b]

[V (x) − V− (a)] dU (x) [a,b]

V dU − V− (a) [U+ (b) − U (a)].

Algebra now gives (1). Subtract U (a)∆V (a) + V (b)∆U (b) from each side of (1) to obtain (2). 2

Exercise 4.1 How should the left side of (1) be altered if we replace [a, b] in both places on the right side of (1) by (a, b), or by (a, b], or by [a, b)? [Just plug in a+ or a− as well as b+ or b− on both sides of the equation d(U V ) = U− dV + V+ dU so as to include or exclude that endpoint; this will give the proper formulation.]

116

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

Useful Formulas for Means, Variances, and Covariances If ξ ∼ = Uniform(0, 1) and F is an arbitrary df, then the rv X ≡ F −1 (ξ) has df F . Thinking of X as F −1 (ξ) presents alternative ways to approach problems. We will do this often! Note that this X satisﬁes (3) X= F −1 (t) d1[ξ≤t] and X= x d1[X≤x] , (0,1)

(−∞,∞)

where 1[ξ≤t] is a random df that puts mass 1 at the point ξ(ω) and 1[X≤x] is a random df that puts mass 1 at the point x. If X has mean µ, then (depending on which representation of X we use) −1 µ= (4) F (t) dt and µ= x dF (x). (0,1)

(−∞,∞)

Moreover, when µ is ﬁnite we can combine the two previous formulas to write −1 (5) F (t) d(1[ξ≤t] − t) = − (1[ξ≤t] − t) dF −1 (t) X −µ= (0,1)

or (6)

(0,1)

X −µ=

x d(1[X≤x] − F (x)) = − (−∞,∞)

(1[X≤x] − F (x)) dx. (−∞,∞)

The ﬁrst formula in each of (5) and (6) is trivial; the second follows from integration t by parts. For example, (5) is justiﬁed by |t F −1 (t)| ≤ | 0 F −1 (s) ds| → 0 as t → 0 1 −1 when E|X| = 0 |F (t)| dt < ∞, and the analogous result (1 − t)F −1 (t) → 0 as t → 1. For (6), we note that x[1 − F (x)] ≤ (x,∞) y dF (y) → 0 as x → ∞ if E|X| < ∞. Thus when Var[X] < ∞, Fubini’s theorem gives both −1 −1 Var[X] = E (1[ξ≤s] − s) dF (s) (1[ξ≤t] − t) dF (t) (0,1)

= (0,1)

(7)

(8)

E{(1[ξ≤s] − s)(1[ξ≤t] − t)} dF −1 (s) dF −1 (t)

(0,1)

= (0,1)

(0,1)

[s ∧ t − st] dF −1 (s) dF −1 (t)

(0,1)

[s ∧ t − st] dK(s) dK(t)

= (0,1)

(0,1)

via (5), and the parallel formula ∞ ∞ (9) Var[X] = [F (x ∧ y) − F (x)F (y)] dx dy −∞

−∞

via (6). Of course, we already know that 1 −1 2 (10) [F (t) − µ] dt = Var[X] = 0

∞

−∞

(x − µ)2 dF (x).

4.

INTEGRATION BY PARTS APPLIED TO MOMENTS

117

Proposition 4.2 (Other formulas for means, variances, and covariances) (i) If X ≥ 0 has df F , then (11)

∞

1

F −1 (t) dt.

(1 − F (x)) dx =

EX = 0

0

(ii) If E|X| < ∞, then (12)

E(X) = −

0

∞

(1 − F (x)) dx =

F (x) dx + −∞

0

1

F −1 (t) dt.

0

(iii) If X ≥ 0, then (13)

1

r

E(X ) =

[F

−1

r

∞

(t)] dt = r

0

xr−1 (1 − F (x)) dx.

0

In fact, one of the two integrals is ﬁnite if and only if the other is ﬁnite. (iv) If (X, Y ) has joint df F with marginal dfs FX and FY , and if G and H are and left continuous, then ∞ ∞ (14) Cov[G(X), H(Y )] = [F (x, y) − FX (x)FY (y)] dG(x) dH(y). −∞

−∞

(Note the special case G = H = I.) (v) If K is and left continuous and ξ ∼ = Uniform(0, 1) (perhaps K = h(F −1 ) for an left-continuous function h, and for X ≡ F −1 (ξ) for a df F ), then (15)

1

∞

∞

= −∞

1

[s ∧ t − st] dK(s) dK(t) 0

(16)

Var[K(ξ)] =

−∞

0

[F (x ∧ y) − F (x)F (y)] dh(x) dh(y) = Var[h(X)] .

(vi) If X ≥ 0 is integer-valued, then EX = ∞ and k=1 P (X ≥ k) (17) ∞ EX 2 = k=0 (2k + 1) P (X > k). Exercise 4.2

Prove formulas (11)–(13).

Exercise 4.3

Prove formulas (14)–(16).

Exercise 4.4

Prove formula (17).

Exercise 4.5 value, say k.

Give an extension of (13) to arbitrary rvs in case r takes an integral

118

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

Exercise 4.6 (Integration by parts formulas) We showed in proposition 4.1 earlier that d(U V ) = U− dV + V+ dU . (i) Now show (noting that dU− = dU+ ) that (18)

dU 2 = U− dU + U+ dU = (2U + ∆U )dU

for ∆U ≡ U − U− .

(ii) Apply proposition 4.1 to 1 = U · (1/U ) to obtain (19)

d(1/U ) = −{1/(U+U− )} dU = −{1/(U (U + ∆U ))} dU.

(iii) Show by induction that for k = 1, 2, . . . we have k−1 i k−i−1 (20) dU k = ( i=0 U+ U− ) dU. Exercise 4.7 (21)

Show that for an arbitrary df F and any measurable h ≥ 0,

d(F/(1 − F )) = {1/((1 − F )(1 − F− ))} dF.

Exercise 4.8 For any df F we have [F (x + θ) − F (x)] dx = θ for each θ ≥ 0. ∼ (0, σ2 ) with df F . Then g(x) ≡ Exercise 4.9 (Stein) Suppose X = x is a density. (And g(x) = − −∞ y dF (y)/σ2 is also true.)

∞ x

y dF (y)/σ2

5.

IMPORTANT STATISTICAL QUANTITIES

5

Important Statistical Quantities

119

˘ ˜ ˇ Notation 5.1 (Trimming, Winsorizing, and Truncating, and dom(a, a)) Let dom(a, a ) denote [0, 1 − a ), (a, 1], or (a, 1 − a ) according as X ≥ 0, X ≤ 0, ˜ a,a (·) denote K(·) Winsorized outside dom(a, a ). Thus when or otherwise. Let K K+ (a) < 0 < K(1 − a ), (1)

˜ a,a (t) equals K+ (a), K(t), K(1 − a ) K as 0 < t ≤ a, a < t < 1 − a , 1 − a ≤ t < 1

˜ (while a ≡ 0 and K(a) ≡ K(0) if X ≥ 0, etc). Let ξ denote a Uniform(0, 1) rv. Let ˜ a,a (ξ) ≡ 1 K ˜ a,a (t) dt, µ ˜(a, a ) ≡ EK (2) 0 which is the (a, a )-Winsorized mean of the rv K(ξ), and let 2 ˜ a,a (ξ)] = 1 K ˜ a,a σ ˜ 2 (a, a ) ≡ Var[K ˜(a, a )2 (t) dt − µ 0 (3)

=

1 1 0

0

˜ a,a (s) dK ˜ a,a (t) [s ∧ t − st] dK

denote the (a, a )-Winsorized variance. For general X, let (4)

µ ˜(a) ≡ µ ˜(a, a) ,

σ ˜2 (a) ≡ σ ˜ 2 (a, a) ,

and

˜ a (·) ≡ K ˜ a,a (·) ; K

but µ ˜(a) ≡ µ ˜0,a if X ≥ 0, etc. We now let 0 ≤ kn < n − kn ≤ n denote integers, and then let an ≡ kn /n and an ≡ kn /n ,

so that 0 ≤ an < 1 − an ≤ 1 .

˜ n (·) denote K(·) Winsorized outside dom(an , an ). Let Let K 1−a µ ˇn ≡ µ ˇK (an , an ) ≡ an n K(t) dt, µ ˘n ≡ µ ˇK (an , an ) ≡ µn /(1 − an − an ), (5) ˜ n (ξ) ≡ 1 K ˜ n (t) dt, ˜K (an , an ) ≡ EK µ ˜n ≡ µ 0

so that µ ˘n is the (an , an )-trimmed mean, µ ˇn is herein called the (an , an )-truncated (6)

µ ˜n is the (an , an )-Winsorized mean, and mean of the rv K(ξ). Then let 2 ˜ n (ξ)] = 1 1 [s ∧ t − st] dK ˜ n (s) dK ˜ n (t) σ ˜n2 ≡ σ ˜K (an , an ) ≡ Var[K 0 0

denote the (an , an )-Winsorized variance. When they are ﬁnite, the mean µ and variance σ2 satisfy 1 µ ≡ µK = 0 K(t) = x dF (x) = E(X) = EK(ξ), 1 1 2 σ2 ≡ σK = 0 0 [s ∧ t − st] dK(s) dK(t) = EX 2 − µ2 = EK 2 (ξ) − µ2 . Let a· ≡ inf{t : K(t) ≥ 0}, and let a◦ ≡ a· ∧ (1 − a· ). (But a· ≡ 0 if X ≥ 0, and a· ≡ 1 if X ≤ 0.) Now, (K − µ ˜n )+ and (K − µ ˜n )− denote the positive and negative parts of K − µ ˜n , and let (7)

¯ n ≡ [K − µ ¯ n2 ≡ −[(K − µ K ˜ n ] and K ˜n )− ]2 + [(K − µ ˜n )+ ]2

on (0, 1).

120

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

In this context, we may wish to assume that both (8)

(kn ∧ kn ) → ∞

(an ∨ an ) → 0;

and

and perhaps we will also assume an /an → 1 and/or (kn − kn )/(kn ∧ kn ) → 0 . We will refer to kn , kn as the trimming/Winsorizing numbers and an , an as the trimming/Winsorizing fractions. Describe the case of (8) as slowly growing to ∞. Now suppose that Xn1 , . . . , Xnn is an iid sample with df F and qf K. Let Xn:1 ≤ · · · ≤ Xn:n denote the order statistics (that is, they are the ordered values of Xn1 , . . . , Xnn ). Let Kn (·) on [0, 1] denote the empirical qf that equals Xn:i on ((i − 1)/n, i/n], for 1 ≤ i ≤ n, and that is right continuous at zero. Now let ¯n ≡ 1 X Xnk = µKn (0, 0) n

1 ¯ n )2 = σ 2 (0, 0) (Xnk − X Kn n

n

(9)

n

Sn2 ≡

and

k=1

k=1

denote the sample mean and the “sample variance.” We also let

(10)

n−k 1 n ˇ Xn:i = µ ˇKn (an , an ) , Xn ≡ n

˘n ≡ X

i=kn +1

1 n − kn − kn

n−kn

Xn:i ,

i=kn +1

(11)

n−kn 1 ˜ Xn ≡ Xn:i + kn Xn−kn = µ ˜Kn (an , an ) kn Xn:kn +1 + n i=kn +1

denote the sample (an , an )-truncated mean, the sample (an , an )-trimmed mean, ˜ n:1 , . . . , X ˜ n:n denote the (an , an )and the sample (an , an )-Winsorized mean. Let X ˜ n . Now note that Winsorized order statistics, whose empirical qf is K 2 ˜ n:i = µ ˜ (an , a ); let S˜2 ≡ 1 ˜ n:i − X˜n )2 = σ ˜n = 1 X X (X ˜K ˜n (an , an ) n n Kn n i=1 n i=1 n

(12)

n

denote the sample (an , an )-Winsorized variance. Let σ ˘n2 ≡ σ ˜n2 /(1 − an − an )2

and

S˘n2 ≡ S˜n2 /(1 − an − an )2 .

¯ n , Sn , X ˘ n , S˜n estimate µ, σ, µ Of course, X ˘n , σ ˜n . We also deﬁne the standardized estimators √ ¯ √ ˘ √ ˇ n (Xn − µ) n (Xn − µ ˘n ) n (Xn − µ ˇn ) (13) Zn ≡ and Zˇn ≡ = , σ σ ˘n σ ˜n and the studentized estimators √ ¯ n (Xn − µ) (14) Tn ≡ and Sn

√ T˘n ≡

˘n − µ n (X ˘n ) = S˘n

√

ˇn − µ n (X ˇn ) . S˜n

[The ﬁrst formula for T˘n is for statistical application, while the second formula is for probabilistic theory.] We will very often assume that these independent rvs

5.

IMPORTANT STATISTICAL QUANTITIES

121

Xn1 , . . . , Xnn having df F and qf K are deﬁned in terms of independent Uniform(0, 1) rvs ξn1 , . . . , ξnn via (see above (7.4.3)) (15)

Xnk ≡ K(ξnk )

for 1 ≤ k ≤ n.

(If we started with iid K rvs X1 , . . . , Xn and then deﬁned Xnk ’s via (15) for the ξ˙k ’s of (7.3.8), then the Xnk ’s of (15) are just the original Xk ’s.) Thus the device of (15) is broadly useful. ¨ n to equal Xn:(n+1)/2 or (Xn:n/2 + Xn:n/2+1 )/2, We deﬁne the sample median X according as n is odd or even. In the previous context, let ξn:1 < · · · < ξn:n denote the order statistics of iid n ≡ (Rn1 , . . . , Rnn ) denote the ranks of these ξn1 , . . . , ξnn , Uniform(0, 1) rvs. Let R n is a and let Dn ≡ (Dn1 , . . . , Dnn ) denote their antiranks. Thus the rank vector R random permutation of the vector (1, 2, . . . , n) , while Dn is the inverse permutation; and these satisfy (16)

ξnDnk = ξn:k

and

ξnk = ξn:Rnk .

We will learn later that (17)

(ξn:1 , . . . , ξn:n ) and (Rn1 , . . . , Rnn ) are independent random vectors.

Such notation is used throughout the remainder of this book.

2

The Empirical DF Notation 5.2 (Empirical dfs and processes) Let X1 , X2 , . . . be iid with df F and qf K. The empirical df Fn of (X1 , . . . , Xn ) is deﬁned by (18)

Fn (x) ≡

1 1 1(−∞,x] (Xk ) = 1[Xk ≤x] n n n

n

k=1

k=1

for − ∞ < x < ∞.

This is a step function on the real line R that starts at height 0 and jumps by height 1/n each time the argument reaches another observation as it moves from left to right along the line. We can think of Fn as an estimate of F . The important study of the empirical process √ (19) En (x) ≡ n [Fn (x) − F (x)] for x ∈ R will allow us to determine how this estimator Fn of F performs. We also let ξ1 , ξ2 , . . . be iid Uniform(0, 1), with true df the identity function I on [0, 1] and with empirical df (20)

Gn (t) ≡

1 1 1[0,t] (ξk ) = 1[ξk ≤t] n n n

n

k=1

k=1

for 0 ≤ t ≤ 1.

The corresponding uniform empirical process is given by √ (21) Un (t) ≡ n [Gn (t) − t] for t ∈ [0, 1].

122

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

If we now deﬁne an iid F sequence X1 , X2 , . . . via Xk ≡ F −1 (ξk ) = K(ξk ), then the empirical df and empirical process of these (X1 , . . . , Xn ) satisfy (22)

(Fn − F ) = [Gn (F ) − I(F )] on R valid for every ω,

and

En = Un (F )

on R,

as follows by (7.3.3). (If we use the ξ˙k ’s of (7.3.8), then the Fn on the left in (22) is everywhere equal to the Fn of the original Xk ’s.) Thus our study of properties of En can proceed via a study of the simpler Un , which is then evaluated at a deterministic F . (Recall also in this regard theorem 5.3.3 about probability being determined by the ﬁnite dimensional distributions.) 2

6.

INFINITE VARIANCE

6

Inﬁnite Variance

123

Whenever the variance is inﬁnite, the Winsorized variance σ ˜2 of the previous section ˜ a,a denote K completely dominates the square µ ˜2 of the Winsorized mean. Let K Winsorized outside (a, 1 − a ). Theorem 6.1 (Gnedenko–Kolmogorov) Every nondegenerate qf K satisﬁes 1−a 1−a 2 (1) |K(t)| dt}2 / a K (t) dt = 0 whenever EK 2 (ξ) = ∞, lim sup{ a a∨a →0

(2)

˜ 2 (ξ) → 1 as (a∨a ) → 0 ˜ a,a (ξ)]/EK Var[K a,a

whenever

EK 2 (ξ) = ∞ .

Proof. Let h be positive, continuous, symmetric about t = 12 , ↑ to ∞ on 1 1 [ 2 , 1), and suppose it satisﬁes Ch ≡ 0 h2 (t) dt < ∞. Let b ≡ 1 − a . Then Cauchy–Schwarz provides the bound b b b b { a |K(t)| dt}2 = { a h(t)|K(t)/h(t)| dt}2 ≤ a h2 (t) dt a [K 2 (t)/h2 (t)] dt. (a) Fix c ≡ c so close to zero that Ch /h2 (c) < . Fix c , and let a ∨ a → 0. Then b b b b { a |K(t)| dt}2 / a K 2 (t) dt ≤ Ch a [K 2 (t)/h2 (t)] dt/ a K 2 (t) dt

(c)

c b K 2 (t) dt ( a + 1−c ) K 2 (t) dt b 2 K (t) dt ≤ Ch + a h2 (1/2) h2 (c) 1−c b ≤ Ch { c K 2 (t) dt/h2 (1/2)}/ a K 2 (t) dt +

(d)

< 2

1−c

(b)

c

for a and b near enough to 0 and 1, since EK 2 (ξ) = ∞.

2 Then (2) follows from [aK+ (a) + a K(1 − a )]2 / [aK+ (a) + a K 2 (1 − a )] → 0.

2

Exercise 6.1 (Comparing contributions to the variance) Let K(·) be arbitrary. Establish the following elementary facts: =0 if EK 2 (ξ) < ∞, 2 2 2 (a) lim sup [aK+ (a) + a K (1 − a )]/˜ σ ≤1 if EK 2 (ξ) = ∞. a∨a →0 1−a <∞ always, (b) lim sup a K 2 (t) dt/˜ σ2 ≤1 if EK 2 (ξ) = ∞. a∨a →0 =0 always, (c) lim sup{a|K+ (a)| + a |K(1 − a )|}/˜ σ = 0 if EK 2 (ξ) = ∞. a∨a →0 1−a <∞ always, (d) lim sup a |K(t)| dt/˜ σ =0 if EK 2 (ξ) = ∞. a∨a →0 <∞ always, ˜ (e) lim sup E|Ka,a (ξ)|/˜ σ = 0 if EK 2 (ξ) = ∞. a∨a →0

124

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

Exercise 6.2 (3)

Let 0 < r < s. Show that for every nondegenerate qf K we have 1−a 1−a |K(t)|s dt = 0 if E|K(ξ)|s = ∞. lim sup { a |K(t)|r dt}s/r a a∨a →0

1 Exercise 6.3 (An added touch) Let K ≥ 0 be on (0, 1) with 0 K(t) dt < ∞. Then there exists a positive function h(·) on (0, 1) with h(t) → ∞ as t → 0 for 1 which 0 h(t)K(t) dt < ∞. Exercise 6.4 (a) Show that for iid rvs X1 , X2 , . . . having Var[X] = ∞ one has ¯ n ]2 Xn2 →p 0 . [X (b) Generalize exercise 6.2 to other sample moments.

The CLT and Slowly Varying Functions Let Xn1 , . . . , Xnn be iid with distribution function (df) F (·) and quantile function (qf) K(·) ≡ F −1 (·). The classical central limit theorem √ ¯ (CLT) states that when this distribution has a ﬁnite variance σ2 , then Z¯n ≡ n [X n − µ]/σ →d N (0, 1). When σ2 is ﬁnite, it is true that (4)

2 t [K+ (ct) ∨ K 2 (1 − ct)] / σ2 → 0

as t → 0,

for each ﬁxed c > 0.

(Note the calculation just below (7.4.6).) But what if σ2 is inﬁnite? Let K denote the ﬁxed qf. Let a > 0 be tiny. We agree that dom(a, a) denotes [0, 1 − a), (a, 1], or (a, 1 − a) according as X ≥ 0, ˜ a,a (·) denotes K Winsorized outside dom(a, a). X ≤ 0, or general X, and that K ˜ a,a (·) [For example, when X takes on both positive and negative values, then K equals K+ (a), K(t), K(1 − a) according as t ≤ a, a < t < 1 − a, 1 − a ≤ t.] Let ξ denote a Uniform(0, 1) rv. Let X ≡ K(ξ), so that X has df F , and let ˜ ˜ ˜ a,a (ξ). Then X(a) has mean (called the (a, a)-Winsorized mean) given X(a) ≡K 1 ˜ by µ ˜(a) ≡ 0 Ka,a (t) dt and variance (called the (a, a)-Winsorized variance) given by the particularly useful expression 1 1 ˜ a,a (r) dK ˜ a,a (s) σ ˜ 2 (a) = 0 0 [r ∧ s − rs] dK (5) = dom(a,a) dom(a,a) [r ∧ s − rs] dK(r) dK(s) , which increases to σ2 as a 0, whether σ2 is ﬁnite or inﬁnite. And this expression makes no reference to any mean (such as µ or µ ˜(a)). This is nice! It turns out that √ ˜n − µ (6) Z˜n ≡ n [X ˜(1/n)] / σ ˜ (1/n) →d N (0, 1) if and only if (7)

2 t [K+ (ct) ∨ K 2 (1 − ct)] / σ ˜ 2(t) → 0 2

as t → 0,

for each ﬁxed c > 0.

And condition (7) holds if and only if σ (t) does not grow too fast, that is, if it is a slowly varying function of t as t → 0 (in the sense of the deﬁnition below). Thus it is appropriate to examine slow variation in the inﬁnite-variance case. Should we

6.

INFINITE VARIANCE

125

examine it in the df domain or the qf domain? We shall do both, in the next section. And since the problem is so “central”, we shall do it very carefully. Often this problem is made rather diﬃcult, by examining this problem in the context of the CLT and in the context of the general theory of slowly varying functions. But the equivalence of all the conditions in the next section follows in an elementary fashion from simple pictures and a dash of Cauchy–Schwarz. In fact, we will not even mention the CLT in the next section. The situation also becomes simpler if one ignores the general theory of slowly varying functions and uses the fact that F and K are monotone, and so no properties of slowly varying functions will be used in the next section either. Somewhere we want to connect with probability theory by showing that each of the equivalent conditions in the next section is also equivalent to (6) and to n ¯ n )2 / n = Xn2 − (X ¯ n )2 . Sn2 / σ (8) ˜ (1/n)2 →p 1, where Sn2 ≡ (Xk − X 1

The reader is also referred to section 11.3, where a major CLT presentation is made. (Actually, quite general necessary and suﬃcient conditions for normality are given, but all the proofs given are quite simple.) And the reader is referred to section 10.6, where the consistency of sample moments is studied in the inﬁnitemoment case. By the time all this is put together, a huge list of equivalencies has been developed that includes (6), (7), (8), and all the conditions of section 7.7, section 10.6, and sections 11.2 – 11.3. Many are very interesting and very useful. Roughly, it is easier to develop new conditions in the context of the weak law of 2 large numbers (WLLN) [for the rvs Xnk ] than in the context of the CLT. Many of the conditions are also new. Finally, we can use the list of equivalencies developed in the next section to make simpler proofs in sections 10.6 and 11.3. The next theorem is just a teaser; it is established in section 11.3. Theorem 6.2 (A studentized CLT) Let Xn1 , . . . , Xnn ’s be row independent, with nondegenerate df F . Then the following are equivalent: n 2 (9) /n ]/νn →p 1 for any one speciﬁc νn > 0 for which n νn . [ 1 Xnk (10) (11)

Sn2 /˜ σn2 →p 1 . √ ˜n − µ n [X ˜(1/n)]/˜ σ(1/n) →d N (0, 1) .

When any/all of these hold, then both √ ¯n − µ (12) n [X ˜(1/n)]/˜ σ(1/n) →d N (0, 1) √ ¯ (13) n [Xn − µ ˜(1/n)]/Sn →d N (0, 1) .

and

Deﬁnition 6.1 (Slowly varying functions) (a) Call L(·) > 0 slowly varying at 0 (written L ∈ R0 or L ∈ L) if it satisﬁes L(ct)/L(t) → 1 as t → 0, for each positive number c > 0. (b) The function l(·) > 0 on (0, ∞) is called slowly varying at ∞ (written as l ∈ U0 ) if it satisﬁes l(cx)/l(x) → 1 as x → ∞, for each positive number c > 0. Note that L(t) = log(1/t) and l(x) = log(x) are slowly varying. They are the prototypes. Note also that when σ2 is ﬁnite, the Winsorized variance function σ ˜2 (t) 2 2 2 2 is always slowly varying, since σ ˜ (ct)/˜ σ (t) → σ /σ = 1 for all c > 0.

126

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS Warning

Let us discuss the rest of this chapter. Strong forms of the classical WLLN and the CLT require some understanding of contributions to the mean and variance from the various parts of the distribution, and sections 7.6 – 7.11 are speciﬁcally geared to this. The author recommends reading only the current section 7.6 now, and that is just to have some rough idea as to what is here. Look quickly at the statement of theorem 7.7.1, but no more for now. Sections 7.6 and 7.7 are used “lightly” in section 10.6, on the consistent estimation of “moments.” But no other section in chapter 10 appeals to these results at all. Thus you are encouraged to wait until section 10.6 to see whether you wish to read them, even if you do eventually read them carefully. Even when you get to section 10.6, read that section ﬁrst. The appeal to the present sections is purposefully minimized. When X has a ﬁnite variance, the classical CLT will be seen to hold. Even when X has inﬁnite variance, we may still be able to claim a CLT. One approach is to see for which inﬁnite-variance distributions the CLT still holds with only the minor modiﬁcation of changing the centering and scaling constants. This is done in chapter 11 using the direct approach methods of analysis that we have already learned; we ﬁrst establish the Berry–Esseen theorem, the Lindeberg–Feller theorem, and the “asymptotic normality condition” without any appeal whatsoever to results of sections 7.6 – 7.11. [In chapter 14 we shall again derive some of the CLT results, now using solely the characteristic function approach (that is, an approach based on the Fourier-analytic method to be studied in chapter 13).] Another approach to the CLT is to “trim” the sample observations Xn1 , . . . , Xnn . ˇ n is the average of the entire trimmed sample, and µ Roughly, if X ˇn and σ ˜n are the truncated moment and the Winsorized standard deviation deﬁned above, then √ ˇ n [Xn − µ ˇ n ]/˜ σn →d N (0, 1) for a huge class of dfs, provided only that we trim a slowly growing fraction of the observations. This is contained in chapter 16, and is based on the empirical process methods of chapter 12; it appeals to the companion inequalities of section 7.11 and the ﬁrst part of section 7.10 also. In chapter 16, uniformity in the CLT conclusion is emphasized. The reader is again encouraged to wait. That is, read in chapter 16 until you feel the need to come back here. Section 7.8 is devoted to one result. It connects the section 7.7 qf domain t-results with its df domain x-results. Section 7.9 is a self-contained unit, used in chapter 15 on classic stable law results. Roughly, with a stable law the contributions from the two extreme tails must either balance or exhibit domination by one tail. If you trim the extremes away (in chapter 16), then you need to worry only about the moderately extreme observations. And this type of thing is done carefully at the end of section 7.10. I have been very conscious of downplaying the material of this chapter so that most results presented do not depend on it. In the end only sections 10.6, 11.2, and 11.3 (depending only on sections 7.6 – 7.7) and chapters 15 and 16 do. But if the reader wants to understand all the jots and tittles of the very best results in these cited locations, then this material does enter in. In fact, I hope that many of you will ﬁnd this material very interesting and useful. The text is organized so that all general qf material is in just the present chapter. 2

7.

SLOWLY VARYING PARTIAL VARIANCE

7

Slowly Varying Partial Variance

127

Many facts about slowly varying functions can be learned from simple pictures. We concentrate here on just such facts. L´evy’s condition (14) has emerged as the necessary and suﬃcient condition of choice for the CLT; and we will now derive many equivalent ways to demonstrate it. [Necessary and suﬃcient conditions for consistent estimation of the variance parameters V (1/n) and σ ˜2 (1/n) as deﬁned below are also equivalent to the conditions on the current list.] (Note (20) below.) Notation 7.1 Let Y denote an arbitrary rv (with df F and qf K); let X ≡ |Y |. Let FX and KX denote the df and qf of X. For 0 < t < 1, let xt ≡ KX (1 − t). Let Yt denote Y Winsorized outside [−xt , xt ]. Deﬁne 2 (1) v(t) ≡ KX (1 − t) , V (t) ≡ (t,1] v(s) ds , σ ˜2 (t) ≡ Var[Yt ] , (2)

q(t) ≡ KX (1 − t) ,

(3)

U (x) ≡

[0,x]

m(t) ≡

y 2 dFX (y) , M (x) ≡

(t,1]

q(s) ds ,

[0,x]

y dFX (y) ,

µ ˜(t) ≡ EYt , V˜ (t) ≡ EYt2 .

Theorem 7.1 (Partial variance, with symmetric Winsorizing) (A): The following [also referred to as (4)(A)–(12)(A)] are equivalent (as t → 0): (4)

[˜ σ 2 (ct) − σ ˜ 2 (t)]/˜ σ2 (t) → 0

for all 0 < c < 1.

(5)

[V˜ (ct) − V˜ (t)]/V˜ (t) → 0

for all 0 < c < 1.

(6)

r(t) ≡ t v(t)/V (t) → 0 .

(7)

[V (ct) − V (t)]/V (t) → 0

for all 0 < c < 1.

(8)

t [v(ct) − v(t)]/V (t) → 0 √ d(t) ≡ t [q(ct) − q(t)]/ V (t) → 0

for all 0 < c < 1.

(9)

(11)

t v(ct)/V (t) → 0 [m(ct) − m(t)]/ t V (t) → 0

(12)

r˜(t) ≡ t v(t)/˜ σ2 (t) → 0 .

(10)

for all 0 < c < 1. for all 0 < c < 1. for all 0 < c < 1.

(B): Specify a sequence an 0 as n → ∞ that satisﬁes lim sup an /an+1 < ∞. Conditions (4)(B)–(12)(B) are obtained by replacing t by an in (4)(A)–(12)(A). These conditions (4)(B)–(12)(B) are also equivalent to condition (4)(A). (C): Conditions (5)(C)–(11)(C) are obtained by replacing V (an ) or V˜ (an ) by σ ˜ 2 (an ) in the denominators of (5)(B)–(11)(B). Then (5)(C)–(11)(C) are also equivalent to the condition (4)(A). (D): The most useful choices are an ≡ /n (equivalently, an ≡ 1/n with c ≡ ), or the alternative an ≡ n /n with n 0 subject to lim sup n / n+1 < ∞.

128

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

(E): The following are equivalent (as x → ∞) to the previous conditions. [Any sequence xn to be speciﬁed below is assumed to satisfy lim(xn+1 /xn ) < ∞) .] (13)

U ∈ U0

(that is, U is slowly varying at ∞).

(14)

R(x) ≡ x2 P (X > x)/U (x) → 0

(equivalently, R(xn ) → 0).

(15)

x[M (cx) − M (x)]/U (x) → 0

for any one (or, all) ﬁxed c > 1.

(16)

˜ (x) ≡ U (x) + x2 P (X > x) U

deﬁnes a function in U0 .

Theorem 7.2 (Partial variance, equal fractions) Consider an arbitrary rv ˜ a,a (·) denote K(·) Winsorized outside dom(a, a), and Y with df F and qf K. Let K now redeﬁne − v(t) ≡ {[K+ (17) (t)]2 + [K + (1 − t)]2 } and V (t) ≡ (t,1] v(s) ds ,

(18)

− q(t) ≡ [ K+ (t) + K + (1 − t) ]

and m(t) ≡

(19)

˜ t,t (ξ)] µ ˜(t) ≡ E[K

˜ t,t (ξ)] . and σ ˜2 (t) ≡ Var[K

(t,1]

q(s) ds ,

All equivalences stated in parts (A), (B), and (C) of the previous theorem are still 2 ˜ t,t valid for the new deﬁnitions of q, v, σ ˜2 , and V˜ (t) ≡ E[K (ξ)]. [In the previous theorem we had q2 (t) = v(t). In the present theorem we have v(t) ≤ q2 (t) ≤ 2v(t), which is still convenient and meets all our needs. What is crucial is that m(·), q(·), v(·), and V (·) have an appropriate meaning. Note that this V (t) is the truncated (with respect to (t, 1 − t)) second moment.] Note Theorem 7.1 Winsorizes symmetrically about zero, while the companion theorem 7.2 Winsorizes equal fractions from each tail. Remark 7.1 (Why all these equivalencies?) In section 10.6 we have a suitable elementary bridge between these two sets of equivalencies that will allow us to combine the lists. Here is how we will do it. For Yn1 , . . . , Ynn iid with arbitrary 2 df F and qf K, the quantity Tn ≡ [maxk Ynk ]/[nYn2 ] can be equivalently seen (with respect to →p ) either as T1n based on Xnk ≡ K|Y | (ξnk )’s or as T2n based on Ynk ≡ KY (ξnk )’s. Now, the list L1 of conditions in theorem 7.1 contains (14), while the list L2 of conditions in theorem 7.2 contains (6)(B) (based on (17)). In theorem 10.6.1 we show that T1n →p 0 is equivalent to a list L3 that contains condition (14) of L1 . In theorem 10.6.2 we show that Tn2 →p 0 is equivalent to a list L4 that contains condition (6)(B) (based on (17)) of L2 . Thus Tn →p 0 is on both L3 and L4 . Thus all conditions on the lists L1 , L2 , L3 , and L4 are equivalent. In theorem 10.6.4 we add to the combined list the condition that Sn2 /˜ σ 2 (1/n) →p 1; and we will also add In theorem 11.3.1 we will add to the list the condition √ (7.6.9). ˜ n) − µ (7.6.6) that Z˜n ≡ n [X(a ˜(an )] / σ ˜ (an ) →d N (0, 1) (for an = 1/n, and other appropriate an ). Knowing these last two facts, it is indeed very interesting to have a large list. 2

7.

SLOWLY VARYING PARTIAL VARIANCE

129

Proof. Clearly, all of (4)(A)–(12)(C) (in the context of both theorem 7.1 and theorem 7.2), hold when σ2 < ∞. So from here on we always assume that σ2 = ∞. (A): Now, (5) implies (6), since for each d > 1 ﬁgure 7.1(a) demonstrates that (a)

t v(d t) + t [v(t) − v(d t)] t v(t) ≤ V (t) V (d t) + (d − 1) t v(d t)

(b)

≤

1 d t v(d t) + d t [v(t) − v(d t)] d−1 V˜ (d t)

(c)

≤

1 d V˜ (dt/d) − V˜ (d t) + d−1 d−1 V˜ (d t)

(d)

→ 1/(d − 1) + 0

by (5).

Since this holds for any d > 1, it gives (6).

v(·)

t

v(·)

dt (a)

ct

v(·)

t

ct ct 2

(b)

t (c)

Figure 7.1 Comparison of areas.

Next, (6) implies (7), since for each ﬁxed 0 < c < 1 ﬁgure 7.1(b) shows that (e)

[V (ct) − V (t)]/V (ct) ≤ [(1 − c)/c][ctv(ct)/V (ct)] → 0 .

Suppose (7), that V ∈ R0 . This implies [V (ct/2) − V (t)]/V (t) → 0, and ﬁgure 7.1(c) then demonstrates that (f)

(ct/2) [v(ct) − v(t)]/V (t) ≤ [V (ct/2) − V (t)]/V (t) → 0 ;

and so (8) holds. Supposing (8) about v(·), we will establish (9) about d(·). Now, √ (g) t[v(ct) − v(t)]/V (t) = d(t) × [d(t) + 2 t q(t)] / V (t) ] ≥ d2 (t) ; and so d(t) → 0, as required for (9), is implied.

130

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS Then (9) implies (5), since ﬁgure 7.1(b) also shows that V˜ (ct) − V˜ (t) t[v(ct) − v(t)] ≤ ˜ V (t) + tv(t) V (t) √ √ √ t [q(ct) − q(t)] t [q(ct) − q(t)] 2 tq(t) = × + V˜ (t) V˜ (t) V˜ (t) √

(h)

≤

t [q(ct) − q(t)] × V (t)

√

√ t [q(ct) − q(t)] 2 tq(t) + V (t) tv(t)

= d(t){d(t) + 2} → 0 . We have closed the (5)–(9) circle. It is now trivial that (6) and (8) imply (10), while (10) even more trivially implies (6). We obtain (11) implies (6) via t √ 1/2 tv(t) t q(t) m(ct) − m(t) ct q(s) ds (i) (1 − c) ≤ = → 0. = (1 − c) V (t) V (t) t V (t) t V (t) Then (7) implies (11) via Cauchy–Schwarz in t t (1 − c)t ct q 2 (s) ds 1/2 m(ct) − m(t) ct q(s) ds = ≤ t V (t) t V (t) t V (t) (j)

≤

V (ct) − V (t) V (t)

1/2 → 0.

We next show that (4) is equivalent to the simpler (5). Suppose (4) holds, so that σ ˜ 2 (·) ∈ R0 . We use Gnedenko and Kolmogorov’s theorem 7.6.1 to write 2 σ ˜ (ct) − σ V˜ (ct) − V˜ (t) ˜ 2 (t) µ ˜2 (ct) − µ ˜2 (t) σ ˜ 2 (ct) σ ˜ 2 (t) = + × 2 × 2 2 σ ˜ (ct) σ ˜ (ct) σ ˜ (t) V˜ (t) V˜ (t) (k)

=

σ ˜ 2 (ct) − σ ˜ 2 (t) σ ˜ 2 (ct) + o(1) + o(1) × 2 × [1 + o(1)] = o(1) ; 2 σ ˜ (ct) σ ˜ (t)

and this implies V˜ (·) ∈ R0 , which is (5). Now suppose that (5) holds, so that V˜ (·) ∈ R0 . The same Gnedenko–Kolmogorov theorem now gives " 2 ! σ ˜ 2 (ct) − σ ˜ 2 (t) V˜ (ct) − V˜ (t) µ ˜ (t) − µ ˜2 (ct) V˜ (ct) = + σ ˜ 2 (t) V˜ (t)[1 + o(1)] V˜ 2 (ct) V˜ (t)[1 + o(1)] (l)

=

V˜ (ct) − V˜ (t) [1 + o(1)] + [o(1) + o(1)][1 + o(1)] = o(1) ; V˜ (t)

and this implies σ ˜2 (·) ∈ R0 , which is (4).

7.

SLOWLY VARYING PARTIAL VARIANCE

131

That (6) and (12) are equivalent is immediate from (7.6.2), which gives (m)

r˜(t) =

t v(t) t v(t) t v(t) r(t) ∼ = . = σ ˜ 2 (t) 1 + r(t) V˜ (t) − µ ˜2 (t) V˜ (t)

(B): We next show that (6)(B) implies (6)(A). Suppose that (6)(B) holds for even one sequence an 0 having lim an /an+1 < ∞. We are thus given that rn ≡ an v(an )/V (an ) → 0. So (6)(A) holds via (n)

sup

an+1 ≤t≤an

≤

t v(t) V (an+1 ) an an+1 v(an+1 ) V (an+1 ) an rn+1 ≤ = V (t) an+1 V (an+1 ) V (an ) an+1 V (an )

an rn+1 an+1

#

1−

since (o)

an − 1 rn+1 → 0 , an+1

V (an ) 1≥ = 1− V (an+1 )

(an+1 ,an ]

v(t) dt

V (an+1 )

≥ 1−

an+1 v(an+1 ) an −1 → 1. V (an+1 ) an+1

Since(6)(A) trivially implies (6)(B), the condition (6)(B) is now on the list. That (5)(B) implies (5)(A) follows from (p)

1≤

sup

an+1 ≤t≤an

V˜ (ct) V˜ (c an+1 ) V˜ ((c/M )an ) ≤ → 1 × 1 = 1, ≤ V˜ (t) V˜ (an ) V˜ (an )

since lim(an /an+1 ) < M < ∞. We next show that (6)(B) implies (7)(B). Now, (6)(B) (since it implies (6)(A)) allows use of (e) with t = an to conclude (7)(B). The arguments of (f)–(h) can be repeated with t = an to show that (7)(B) implies (8)(B), which implies (9)(B), which implies (5)(B). Now, (5)(B) (which was shown in (p) to imply (5)(A), and thus it also implies (6)(A)) implies (6)(B) by using the argument of (a)–(d) (which is valid by using (6)(A)). Also, (10)(B) can be added by the same argument used for (10)(A). The arguments of (i) and (j) with t = an allow us to add (11)(B). Finally, the equivalence of (4)(B) and (5)(B) still follows from (k) and (l) with t = an . Next observe that (12)(B) is equivalent to (6)(B), since (m) gives (q)

r˜(an ) =

an v(an ) r(an ) an v(an ) ∼ = . σ ˜ 2 (an ) V (an ) + an v(an ) 1 + r(an )

Thus (12)(B) is on our list. (C): Note that (6)(C) is exactly (12)(B), and so it is on our list, and thus it implies (12)(A). We may reread (e) (with the denominator V (an ) replaced by σ ˜ 2 (an ) throughout) to see that (6)(C) (that is, its (12)(A) consequences) implies (7)(C). Then rereading (f)–(h) (with the same denominator replacement) shows that (7)(C) implies (8)(C), which implies (9)(C), which implies (5)(C). We now close the circle on (5)(C)–(9)(C) by noting that (5)(C) implies (5)(B), again using the Gnedenko and Kolmogorov result (7.6.2). We can add (10)(C) by the same trivial argument as before. Rereading (i)–(j) (with the new denominator) then allows us to add (11)(C).

132

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

The proof of theorem 7.2 is nearly identical. All but lines (h) and (j) are identical; line (i) is identical because m is still the integral of q. But (h) and (j) are not identical because we no longer have q2 = v. But we do have v2 = [q + ]2 +[q − ]2 , where q = q + +q − , and that is enough. Just factor the two pieces separately in (h) and (j), and apply the trivial inequalities (a + b)2 ≤ 2(a2 + b2 ) and a ∨ b ≤ a + b ≤ 2(a ∨ b). 2 Proof. (E): We prove this part separately. We ﬁrst show that (14) implies (13). So suppose that (14) holds, so that R(x) ≡ x2 P (X > x)/U (x) → 0 as x → ∞. If c < 1, then [U (x) − U (cx)]/U (cx) = (cx,x] y 2 dFX (y)/U (cx) (a)

≤ c−2 [(cx)2 P (X > cx)/U (cx)] → 0;

and for c > 1 it is analogous that [U (cx) − U (x)]/U (x) → 0. Thus, U is slowly varying, as in (13). Suppose (13) holds. Then for all x ≥ (some x ), 2x 2 y dFX (y) x2 P (x < X ≤ 2x) U (2x) − U (x) (b) ≤ x ≤ < , U (x) U (x) U (x) (c)

x2 P (2x < X ≤ 4x)

U (2x) < ≤ [(1 + )/4] , U (x) 4 U (x)

(d)

x2 P (4x < X ≤ 8x)

U (4x) U (2x) < 2 ≤ [(1 + )/4]2 , . . . . U (x) 4 U (2x) U (x)

So for x ≥ x we add these to get (e)

R(x) = x2 P (X > x)/U (x) ≤

∞

k=0 [(1

+ )/4]k ≤ ≡ /(1 − (1 + )/4) .

Thus R(x) → 0 as x → ∞. That is, (13) implies (14). We will obtain (15) implies (14) analogously to (b)–(d). (To keep notation nearly identical to (b)–(d), we shall pretend that c = 2; but really, 4 = c2 , 8 = c3 , . . . .) Now, for all x ≥ (some x ), (15) gives 2x x x dFX (y) x2 P (x < X ≤ 2x) x[M (2x) − M (x)] (f) ≤ ≤ < , U (x) U (x) U (x) (g)

x2 P (2x < X ≤ 4x)

U (2x)

2x[M (2x) − M (x)]

≤ ≤ {1 + } ≤ (1 + 2 ) , U (x) 4 U (x) 4 U (x) 4 x2 P (4x < X ≤ 8x)

U (4x) U (2x) ≤ 2 U (x) 4 U (2x) U (x)

(h)

≤

4x[M (4x) − M (2x)]

(1 + 2 ) 1 + ≤ 2 (1 + 2 )2 . 42 U (2x) 4

Add these to get R(x) → 0 as x → ∞, as in (e). Thus (15) implies (14).

7.

SLOWLY VARYING PARTIAL VARIANCE

133

We obtain (13) implies (15) via cx 2 y dFX (y) [U (cx) − U (x)] x[M (cx) − M (x)] (i) ≤ x ≤ → 0. U (x) U (x) U (x) Now, (13) implies (16) follows from (j)

˜ (cx) − U ˜ (x) U U (cx) − U (x) (c2 + 1)x2 P (X > x) + → 0, ≤ ˜ (x) U (x) U (x) U

since (13) now brings (14) with it. Then (16) implies (14), since ˜ (cx) − U ˜ (x)]/U ˜ (x) [U cx ˜ (x) = [ x y 2 dFX (y) + (cx)2 P (X > cx) − x2 P (X > c)]/U ˜ ≥ {[x2 [P (X > x) − P (X > cx)] + c2 x2 P (X > cx) − x2 P (X > x)}/U(x) (k)

= [(c2 − 1)/c2 ]/ {1 + 1/ [(cx)2 P (X > cx)/U (cx)]} ,

and the extreme left term going to 0 forces (cx)2 P (X > cx)/U (cx) → 0. The second condition in (14) suﬃces, since 2 x2 xn P (X > xn ) x2 P (X > x) (l) sup ≤ lim n+1 ≤ O(1)×R(xn ) → 0 . 2 U (x) xn U (xn ) xn ≤x≤xn+1 Thus (13)–(16) are equivalent. That (14) and (6)(A) are equivalent is the subject of the next section. At that point (13)–(16) will be on our list. 2 Though already established, we will still give a simple proof that V (·) ∈ L implies V˜ (·) ∈ L. If V ∈ L, a simple graph shows that 0 ≤ (1 − c)av(a)/V (a) ≤ [V (ca) − V (a)]/V (a) → 0 , implying that tv(t)/V (t) → 0 as t → 0. Then the equation # V˜ (ct) V (ct) tv(t) (0 + 1) × 1 ctv(ct) = +1 × +1 → =1 ˜ V (ct) V (t) V (t) 0+1 V (t) shows that V˜ ∈ R0 . We also note that when V ∈ L, then [V˜ (ct) − V˜ (t)]/V˜ (t) ≤ t[v(ct) − v(t)]/V (t) → 0 , showing that V˜ (·) ∈ L follows. Exercise 7.1 (20)

Verify that an equivalent condition in theorem 6.2 is

2 t [K+ (ct) ∨ K 2 (1 − ct)] / σ ˜ 2(t)

→0

as t → 0,

for each ﬁxed c > 0.

(This condition is singled out for heavy use in chapter 16. It is purposely left to the exercises to encourage chapter 16 readers to put some eﬀort into these proofs.)

134

8

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

Speciﬁc Tail Relationships

We list two relationships that will prove important, the ﬁrst concerning the CLT and variance estimation and the second the WLLN. They compare the height of the qf with the magnitude of a partial moment. Theorem 8.1 (The (x, t)-tail equivalence; CLT) Consider any X ≥ 0, with qf K. Then (1) lim sup tK r (1−t)/ [t,1] K r (1−s) ds = lim sup xr P (X > x)/ [0,x] y r dF (y) , x→∞

t→0

for each r > 0. The same is true for the lim inf, and for the lim (if it exists). Proof. (2)

By the change of variable of (7.3.11), r r for all x. [0,F (x)] K (s) ds = [0,x] y dF (y)

Deﬁne t = 1 − F (x). Then (a)

r(t) ≡

t K r (1−t) [0,1−t]

if

K r (s) ds

= R(x) ≡ x

r

P (X>x)

[0,x]

y r dF (y)

t K r (1 − t) = xr P (X > x).

But examination of ﬁgure 8.1 shows that (from an x point of view) (3)

t K r (1 − t) = xr P (X > x)

fails

when and only when (4)

x is any point in the domain of some ﬂat spot of F, other than the left endpoint.

Also, R is continuously across any such ﬂat spot, and R(·) approaches a limit at both ends of each ﬂat spot. So far, we have considered all values of x, but we have omitted values of t not in the range of the F (x)’s. But equality fails in (3) (from a t point of view) when and only when (5)

t is any point in the domain of some ﬂat spot of K, other than the left endpoint.

Also, r is continuously across any such ﬂat spot as t ; and r(·) approaches a limit at both ends of each a ﬂat spot. Finally, note that all right and left limits achieved by R(·) across ﬂat spots are achieved as left and right limits of r(·), and vice versa. This suﬃces for the claim. [Note also ﬁgure 10.4.1 below. And bare in mind that (a ∨ b) ≤ a + b ≤ 2(a ∨ b) for all a, b > 0.] 2

Deﬁnition 8.1 (Order-r qfs) (6)

A qf K is of order r (with r > 0) if

t{|K+ (t)|r + |K(1 − t)|r } → 0

as t → 0.

)

)( )( )

0

t-values

x-values

1

0

where

(

( (

is equal to or xP(X > x) is not equal to ∫ K(s) ds [0,x] y dF (y)

)(

tK(1 – t)

)(

[0,1 – t]

∫

(

(

Figure 8.1 (x, t)-tail equivalence comparison, when r = 1.

1

)

0

t

SPECIFIC TAIL RELATIONSHIPS

(

)

Not Equal

8.

K(1 – t)

F (x)

135

Not

Equal

x

136

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

Theorem 8.2 (The (x, t)-order equivalence; WLLN) Consider any X ≥ 0, with qf K. Then (7)

lim sup t|K(1 − t)|r = lim sup xr P (X > x) , t→0

x→∞

for each r > 0. The same is true for lim inf, and lim (if it exists). Proof. As ﬁgure 8.1 shows, the values (or limit of the values) of t|K(1 − t)|r r and x P (X > x) are equal at the associated pairs of endpoints related to any ﬂat spot of F or any ﬂat spot of K; and they are monotone across the ﬂat spots. At all nonﬂat-spot pairs of associated points, they are equal. That is, at all of the key values (i.e., local extremes) the quantities in question are equal. 2 Remark 8.1 Clearly, if E|X|r < ∞ for X ∼ = K, then K is of order r, since for some ﬁnite M , (8)

g(x) ≡ |x|r F (x)[1 − F (x)] ≤ M on R, while g(x) → 0 as |x| → ∞,

(9)

h(t) ≡ t[|K(t)|r + |K(1 − t)|r ] ≤ M on (0, 1), while h(t) → 0 as t → 0. 2

9.

REGULARLY VARYING FUNCTIONS

137

9

Regularly Varying Functions

Deﬁnition 9.1 (Regularly varying functions, at 0) Call V (·) > 0 regularly varying at 0 with characteristic exponent r (written V ∈ Rr ) if L(t) ≡ t−r V (t) satisﬁes L(ct)/L(t) → 1 for each c > 0 (such a function L was called slowly varying, and we agreed to write L ∈ L ≡ R0 ). [Clearly, shifting a qf up or down has absolutely no eﬀect on whether or not it is varies regularly.] 1 Theorem 9.1 Let V (t) ≡ t v(s)ds, where v ≥ 0 and v on some 0 < t < a◦ . Consider also V˜ (t) ≡ V (t) + t v(t). Let β ≥ 0. (Monotone density theorem) (1)

V ∈ R−β

iﬀ

tv(t)/V (t) → β as t → 0.

(2)

V ∈ R−β

implies

v ∈ R−(β+1) ,

when β > 0.

(Karamata theorem) (3)

v ∈ R−(β+1)

implies

V ∈ R−β ,

(4)

V ∈ R−β

implies

V˜ (t) ∈ R−β

Proof.

Consider the “if” half of (1). For 0 < b < a < 1 we note that at (a − b) t v(at) ≤ bt v(s)ds = V (bt) − V (at) ≤ (a − b) t v(bt).

(a)

when V (0) = ∞. and t v(t)/V˜ (t) → β/(1 + β).

Also, V (bt) − V (at) V (bt) b−β L(bt) V (at) a−β L(at) = × − × −β −β −β t L(t) (bt) L(bt) L(t) (at) L(at) L(t) (b)

→ (b−β − a−β ).

The left side of (a) thus gives (c)

lim sup t 0

(b−β − a−β ) d −β t v(at) ≤ − , and is also ≤ − b =β t−β L(t) b−a db b=1

by setting a = 1 and letting b 1. The right side of (a) analogously gives (d)

lim inf t 0

t v(bt) (b−β − a−β ) d ≥− , and is also ≥ − b−β =β −β t L(t) b−a db b=1

by setting b = 1 and letting a 1. Combining (c) and (d) gives t v(t)/V (t) → β, so the “if” half of (1) holds. Then (2) and (4) are immediate. Consider the converse part of (1). Let r(t) ≡ t v(t)/V (t), so that the ratio r(t) ∈ [β − , β + ] for all 0 < t ≤ (some t ). Now, V is absolutely continuous on every closed subinterval of (0, t ] by the fundamental theorem of calculus. And thus

138

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

log V is absolutely continuous on the same closed subintervals (see (4.4.3)(b)), and so we may claim that (e)

(log V ) (s) = −r(s)/s

a.e. on (0, t ].

Thus for any 0 < t ≤ t , we can integrate to get t (f) log V (t) − log V (t ) = t [ r(s)/s ] ds, and this gives t [(ct)β V (ct)]/[tβ V (t)] = c β e t

ct

[r(s)/s] ds

t = e β log c e

ct

[r(s)/s] ds

[(r(s)−β)/s] ds

(g)

=e

(h)

= (some value between c− and c ) for all 0 < t ≤ t .

ct

But > 0 is arbitrary. Thus V ∈ R−β . Moreover, (f) shows that V may be written in the form d (5) V (t) = c t−β exp( t [r(s) − β]/s ds) for some constants c, d and for r(t) ≡ t v(t)/V (t) → β as t → 0. Consider (3). Fix 0 < r < 1. Let L(·) ∈ L. Then for all w ≤ (some t0 ≡ tr ), (i)

(1 − )L(w) ≤ L(rw) ≤ (1 + )L(w).

(Recall that a = b ⊕ c means that |a − b| ≤ c.) We write rt t V (rt) − V (rt0 ) = rt0 v(u) du = r t0 v(rw) dw = r−β

t0 t

with u ≡ r w

w−(β+1) L(rw) dw

(j)

= r−β [1 ⊕ ]

(k)

= r−β [1 ⊕ ]

t0 t

t0 t

w−(β+1) L(w) dw

using (i)

v(w) dw = r−β [1 ⊕ ] [V (t) − V (t0 )].

Thus (l)

(1 − ) r−β ≤

V (rt) − V (rt0 ) ≤ (1 + ) r−β . V (t) − V (t0 )

Since V (t) → ∞ as t → 0, we have from (l) that V ∈ R−β .

2

Deﬁnition 9.2 (Regularly varying functions, at ∞) A function U > 0 on (0, ∞) is called regularly varying with exponent ρ (written U ∈ Uρ ) when l(x) ≡ x−ρ U (x) satisﬁes l(cx)/l(x) → 1 for each c > 0 (such a function l(·) was called slowly varying at ∞).

9.

REGULARLY VARYING FUNCTIONS

139

Theorem 9.2 (Partial variance, β > 0) [Let α ≡ 2/(β + 1) and β = (2 − α)/α.] (i) Based on the deﬁnitions in (7.7.1) and (7.7.3), the following are equivalent: (6)

U ∈ U2 β/(1+β) = U2−α .

(7)

x2 P (X > x)/U (x) → β

as x → ∞.

(8)

t v(t)/V (t) → β

as t → 0.

(9)

V ∈ R−β .

(10)

v ∈ R−(β+1) .

(ii) Based on the deﬁnition in (7.7.17), the conditions (6)–(10) are equivalent. Proof. The equivalence of (8), (9), and (10) follows from theorem 9.1. Then add (7) using theorem 8.1. That (6) is also equivalent is the subject of the next exercise. 2 Exercise 9.1 Establish the initial equivalence (6) in theorem 9.2. [Hint. Use the integration by parts result P (X > x) = (x,∞) x−2 dU (x).] − Exercise 9.2 (De Haan) Let v− (t) ≡ [K+ (t)]2 and v+ (t) ≡ [K + (1 − t)]2 . Let t t V − ≡ 0 v − (s) ds and V + ≡ 0 v + (t) ds. If v− (t)/v+ (t) → c ∈ [0, ∞] as t → 0, then V − (t)/V + (t) → c as t → 0, provided that both integrals are inﬁnite on (0, 1).

The following exercises summarize some standard general regular variation results. [That is, h is not assumed to be monotone. Karamata’s result is now harder.] Exercise 9.3 (Regular variation holds uniformly) (a) Let h denote any function that is regularly varying at 0, of order r ∈ R. Then for 0 < a < b < 1 we have the uniform convergence of both: (11)

sup |h(ct)/h(t) − cr | → 0 a≤c≤b

(12)

sup a≤u≤v≤b

as t → 0,

v v | u h(ct) dc/h(vt) − u cr dc/v r | → 0

as t → 0.

(b) Prove (11) assuming h . [Hint. Use (5) for (b). That (12) follows from (11) is easy, even in general.] Exercise 9.4 (Karamata) Let h be regularly varying at 0, of order r ∈ R. (i) Suppose r ≤ −1. Then 1 (13) h(s) ds/[th(t)] → −1/(r + 1) as t → 0. t 1/2 (ii) Suppose r > −1 or both r = −1 and 0 h(s) ds < ∞. Then t (14) as t → 0. 0 h(s) ds/[th(t)] → 1/(r + 1) Question Theorem 9.1 shows that when β > 0, the collection of quantile functions {K : V (·) ∈ R−β } form a subset of the collection {K : V˜ (·) ∈ R−β }. Are these two collections actually the same collection?

140

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

10

Some Winsorized Variance Comparisons

Notation 10.1

(1)

We agreed in notation 7.5.1 that for 0 ≤ a ≤ 1 − a ≤ 1,

˜ ≡K ˜ a,a (·) denotes K(·) Winsorized outside dom(a, a ), K(·) ˜ ˜ ˜ 2 (ξ), with ξ ∼ µ ˜ ≡ EK(ξ), σ ˜ 2 ≡ Var[K(ξ)], V˜ ≡ EK = Uniform(0, 1), 2 ˜ ˜ q˜ ≡ K(1 − a ) − K(a) and V ≡ dom(a,a ) K (t) dt .

˜ ˜ a,a (t) equals K+ (a), =K Thus whenever K+ (a) < 0 < K(1 − a ) we note that K(t) K(t), K(1 − a ) according as t is in (0, a], (a, 1 − a ), [1 − a , 1), while only the right tail of K(·) is modiﬁed if X ≥ 0, and only the left tail of K(·) is modiﬁed if X ≤ 0. (Recall also that ao ≡ a. ∧ (1 − a. ) for a. ≡ inf{t : K(t) ≥ 0}, from just above (7.5.7).) In any case, 1 1 ˜ a,a (s) dK ˜ a,a (t) (2) σ ˜2 = σ ˜ 2 (a, a ) ≡ 0 0 (s ∧ t − st) dK for 0 ≤ a < 1 − a ≤ 1, which shows that σ ˜2 (a, a ) always Var[K(ξ)] as a 0 and a 0. Setting a = a = 0 gives a valid representation of the variance in [0, ∞] without mention of µ.

K0(·)

0

a0 1 – a0

1

Figure 10.1 If K ∈ K, then the graph of (K – µK)/σK lies entirely in the shaded region.

Fix a qf K0 having mean 0 and ﬁnite variance σ02 , and (for some 0 < a0 < ﬁxed) deﬁne the class of qfs (3)

1 2

K ≡ {K(·) : |K − µK |/σK ≤ |K0 | on (0, a0 ] ∪ [1 − a0 , 1)}

to be all qfs whose standardized form is bounded in the tails by the ﬁxed qf K0 . (See ﬁgure 10.1.) Let µ ≡ µK and σ ≡ σK for each K ∈ K. 2

10.

SOME WINSORIZED VARIANCE COMPARISONS

Inequality 10.1 (Uniform Winsorized variance comparisons) as (a ∨ a ) → 0,

141 Note that

σ 2 − σ 2 |/˜ σ2 → 0 , supK∈K |˜ √ supK∈K |˜ (5) µ − µ|/(˜ σ a ∨ a ) → 0 (and µ ˇ may replace µ ˜), √ √ (6) ˜| + a |K(1 − c a ) − µ ˜|}/˜ σ→0 and supK∈K { a |K+ (ca) − µ √ for every ﬁxed 0 < c, c < 1. [Thus supK∈K n |ˇ µ−µ|/˜ σn → 0 when an = an = 1/n.] (4)

Proof. Let K ∼ = (0, 1) with K ∈ K. The following bounds are uniform over K ∈ K and all 0 ≤ (a ∨ a ) ≤ (some a ). For the ﬁrst moment comparison we use 1 a (7) |µ − µ ˜|/σ ≤ 0 |K0 (t)| dt + 1−a |K0 (t)| dt + a |K0+ (a)| + a |K0 (1 − a )| (a) since

≤

√

a ∨ a × ,

√ a √ |K0 (t)| dt ≤ || t K0 (t)||a0 × 0 t−1/2 dt = o( a ) √ follows from t K0 (t) → 0 as t → 0 whenever σ2 < ∞. Comparing second moments shows that for all 0 ≤ (a ∨ a ) ≤ (some a ) we have 1 2 1 2 ˜ (t) dt /σ 2 0 K (t) dt − 0 K (8)

(9) Finally,

(10)

a 0

≤

a 0

+

1 1−a

2 (a) + a K02 (1 − a ) ≤ . K02 (t) dt + aK0+

√ √ { a |K+ (ca) − µ ˜| + a |K(1 − c a ) − µ ˜|}/σ √ √ √ √ ≤ 2 ca |K0+ (ca)|/ c + 2 c a |K0 (1 − c a )|/ c ≤ .

Simple algebra completes the proof.

2

Inequality 10.2 (Basic Winsorized variance inequality) We suppose that 0 ≤ a ≤ 1 − a ≤ 1 (with a = 0 allowed only if X ≥ 0, and a = 0 allowed only if ˜ c (·) ≡ K ˜ ca,c a (·), and let µ X ≤ 0). Fix 0 < c, c < 1. Let K ˜c , σ ˜c2 , V˜c , and q˜c denote the same quantities for this new qf. It is immediate from ﬁgure 10.2 (using |˜ µ| ≤ q˜) that for a ∨ a suﬃciently small, (11)

0≤σ ˜c2 − σ ˜ 2 = (V˜c − V˜ ) − (˜ µ2c − µ ˜2 ) = {V˜ ⊕ (a + a ) q˜c2 − V˜ } + {(˜ µ ⊕ (a + a ) q˜c )2 − µ ˜2 }

(12)

≤ 4(a + a ) q˜c2 .

142

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

Inequalities Aimed at the Inﬁnite-Variance Case Suppose the qf K(·) satisﬁes K+ (a) < 0 < K(1 − a ). Fix 0 < c, c < 1. Deﬁne and

d ≡

√ a σ ˜

[K(1 − a) − µ ˜] ,

− K+ (ca)|

and

D ≡

√ a σ ˜

[K(1 − c a ) − K(1 − a )] ,

2 2 [K+ (ca) − K+ (a)]

and

˜ ≡ D

a σ ˜2

(13)

d ≡

√ a σ ˜

(14)

D≡

√ a σ ˜ [K+ (a)

(15)

˜≡ D

a σ ˜2

[µ ˜ − K+ (a)]

[K 2 (1 − ca ) − K 2(1 − a )] .

All of these are then ≥ 0. (Note ﬁgure 10.2.)

K(·)

˜ K c

˜ K

0 ˜→ µ

ca

a

˜ K

˜ K c

˜ and K˜ (·). Figure 10.2 The qfs K(·) c

a′

c′a′

1

10.

SOME WINSORIZED VARIANCE COMPARISONS

Inequality 10.3 (Tail relationships) (16)

143

Suppose K+ (a) < 0 < K(1 − a ). Then

0 ≤ d ∨ d ≤ 1 ,

(19)

√ √ 2 (˜ µc − µ ˜)2 /˜ σ 2 ≤ ( a D + a D )2 ≤ 2 (a D2 + a D ), ˜ + c D ˜ ≤ (V˜c − V˜ )/˜ ˜ +D ˜ , cD σ2 ≤ D √ √ ˜ = (D )2 + 2D a K(1 − a )/˜ ˜ = D2 + 2D a |K+ (a)|/˜ σ and D σ, D

(20)

˜ 2 )/˜ σ 2 ≤ D (D + 2d) + D (D + 2d ) ≤ D (D + 2) + D (D + 2) , (˜ σc2 − σ

(21) (22)

q˜ ≤ |K+ (a) − µ ˜| + |K(1 − a ) − µ ˜| ≤ 2 q˜ , √ √ √ a |K+ (ca)|/˜ σ ≤ D + a |K+ (a)|/˜ σ ≤ D + d + aµ ˜/˜ σ,

(23)

D + D ≤ 2 [(a ∨ a )/(a ∧ a )]1/2 × (˜ qc − q˜) / q˜ .

(17) (18)

Proof. Figure 10.2 immediately gives all but (20) and (23). For (23) observe 1 ˜ that σ ˜2 ≥ 0 (K(t) −µ ˜)2 dt ≥ (a ∧ a ) (˜ q /2)2 . Since ν = µY minimizes E[(Y − ν)2 ], 1 1 ˜ c (t) − µ ˜ (a) (˜ σc2 − σ ˜ 2 )/˜ σ 2 ≤ ( 0 (K ˜)2 dt − 0 (K(t) −µ ˜)2 dt)/˜ σ2 a 1 (b) ≤ ( 0 + 1−a )(same) dt/˜ σ2 ≤ D2 + 2Dd + (D )2 + 2D d . 2 There are many equivalent ways of expressing that a df or qf is in the domain of attraction of a normal or stable distribution. The next inequality enables us to go back and forth between various of these equivalent conditions and to establish new ones. We now prove that (24)–(28) below are equivalent. [We write (24)u –(28)u to denote uniformity of the inequalities in the qfs over some collection Ku of qfs K(·).] It matters not here whether the variance of K(·) is ﬁnite or inﬁnite. Inequality 10.4 (Tail equivalencies) (A) Let the qf K be arbitrary. With ﬁxed 0 < c, c < ∞ (bounded away from 0 and ∞), as (a ∨ a ) → 0 the following are equivalent: (24)

D ∨ D → 0 ,

(25)

[˜ σc2 − σ ˜ 2 ]/˜ σ2 → 0 , ˜ ∨D ˜ → 0 , D

(26) (27) (28)

[V˜c − V˜ ]/˜ σ2 → 0 , [V˜c − V˜ ]/V˜ → 0 .

(B) These are also equivalent for a speciﬁc c ≡ ca → 1 and c ≡ ca → 1 as the maximum (a ∨ a ) → 0. (C) The condition a ∨ a → 0 may be replaced by a speciﬁc an ∨ an → 0 as n → ∞. (D) If any one of (24)–(27) holds uniformly over a class Ku of qfs K, then all of them hold uniformly over the same class Ku . (E) Everywhere in (24)–(27) that a σ ˜ 2 appears in a denominator it may be replaced ˜ everywhere simultaneously by V . (F) Suppose X ≥ 0. We may let a ≡ 0, and claim everything above with respect only to a .

144

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

Proof. [The proofs are written assuming 0 < c, c < 1, with only minor adjustments needed otherwise.] Now, (27) is equivalent to (26) by (18), and (26) implies (24) by (19) (all implications holding with the claimed uniformity). Then (24) implies (25) by (20) (also with the claimed uniformity). We will show in the next paragraph that (25) implies (27) and (24) (also with the claimed uniformity). ˜ From the deﬁnition of K(·) we have (for rectangular regions R1 , R2 , R3 , R4 and corner regions R5 , R6 , R7 , R8 ) (a)

(b)

˜ 2 ]/˜ σ2 [˜ σc2 − σ 1−c a 1−c a 1−a 1−a = ( ca − a ) (s ∧ t − st) dK(s) dK(t)/˜ σ2 ca a = Σ81 Ri (s ∧ t − st) dK(s) dK(t)/˜ σ2 ≥{

(c)

a

1−a σ }{ a (1 ca sdK(s)/˜

+ (d)

a a

ca ca (s

− t)dK(t)/˜ σ}

∧ t − st)dK(s)dK(t)/˜ σ2 + {(six terms}

√ ˜ + (ξ)/˜ ≥ {c aD}{EK σ } + {(c/2)D2 } + {(six analogous terms)}

˜ + (ξ) term by parts). for a ∨ a small enough (integrate the EK Suppose (25) holds. From (d) (along with the other six analogous conclusions) we learn that the (25) conclusion [˜ σc2 − σ ˜ 2 ]/˜ σ 2 → 0 (uniformly) implies both √ ˜ (29) D a E|K(ξ)|/˜ σ → 0 (uniformly) and D → 0 (uniformly). Thus when [˜ σc2 − σ ˜ 2 ]/˜ σ 2 → 0 (uniformly) this gives (recall (20), too) σ ˜2 V˜c − V˜ |˜ µ2c − µ ˜2 | |˜ µc − µ ˜| |˜ µc − µ ˜| |˜ µ| ˜c2 − σ (e) = + 2 − ≤ σ ˜2 σ ˜2 σ ˜2 σ ˜ σ ˜ σ ˜ (30)

√ √ √ √ ˜ ≤ [ aD + a D ]{[ aD + a D ] + 2E|K(ξ)|/˜ σ} → 0

(uniformly), and so assuming [˜ σc2 −˜ σ 2 ]/˜ σ 2 → 0) implies |V˜c − V˜ |/˜ σ2 → 0 (uniformly). That is, (25)u implies (27)u . Thus (24)u –(27)u are equivalent. Since |(V˜c − V˜ )/V˜ | ≤ |V˜c − V˜ |/˜ σ2 , we have (27)u implies (28)u . Finally, V˜ − V˜ V˜c − V˜ |V˜c − V˜ | µ ˜2 c (f) − × ; = σ ˜2 σ ˜2 V˜ V˜ but µ ˜2 /˜ σ 2 is bounded as a ∨ a → 0 whenever σ2 ∈ (0, ∞), and it likewise goes to zero whenever σ 2 = ∞ by (7.6.2). Thus (f) shows that (28) implies (27). Note also that (28)u implies (27)u in any class Ku in which (31)

lima∨a →0 supK∈Ku µ ˜2 /˜ σ2 < ∞ .

Claim (E) also follows from (7.6.2) when EK 2 (ξ)2 = ∞, and is trivial when this is ﬁnite. Claims (B), (C), and (G) are now trivial. 2

10.

SOME WINSORIZED VARIANCE COMPARISONS

145

The Statistical Domain of Attraction Deﬁnition 10.1 (Domains of attraction) Let K be a qf, and deﬁne V˜ (t) ≡ − ˜ t,t (ξ)] as in (1). Let v− (t) ≡ [K+ V˜ (t, t) = Var[K (t)]2 and v+ (t) ≡ [K + (1 − t)]2 . − + Call K(·) balanced if v (t)/v (t) → (1 − p)/p for some p ∈ [0, 1]. Now deﬁne (32)

D ≡ ∪β≥0 {all K : V (·) ∈ some R−β , and K(·) is balanced if β > 0},

and call D the classical total domain of attraction. [It is customary to focus on the relationship α ≡ 2/(β + 1) instead.] Let (33)

˜ ≡ ∪{all qf’s K : V˜ ∈ R−β for some β ≥ 0}, D

˜ the statistical domain of attraction. [In trimming observations the and we call D two extreme tails do not have to be in balance, as they will be thrown away. That is, the natural CLT for the applied statistician can apply more generally than the probabilist’s natural CLT. We are preparing for this.] Theorem 10.1 (Tail comparisons for regularly varying qfs) Suppose the qf ˜ ≡ {all qfs K : V˜ (·) ∈ R−β for some β ≥ 0}. Suppose further K is in the class D that the followimnmg two conditions hold: (i): Fix c, c (34)

(if β = 0)

or

(ii): c ≡ ca → 1 and c ≡ ca → 1 as (a ∨ a ) → 0 (if β > 0), (i): 0 < lim a /a ≤ lim a /a < ∞ (if β = 0)

(35)

or

(ii): a /a → 1 (if β > 0).

Then all of the conclusions (24)–(28) hold. Proof. Because of inequality 10.4, we need only establish (28). We suppose that 0 < a, a < a◦ ≡ (a· ∧ (1 − a· )). We assume a < a in deriving an inequality (the other case of a < a is symmetric). Let V˜ (a) ≡ V˜ (a, a), etc. Now, [V˜ (ca, c a ) − V˜ (a, a )] V˜ (a, a ) (a)

[V˜ (ca) − V˜ (a)] [V˜ (c a ) − V˜ (a )] ≤ + × V˜ (a) V˜ (a )

$

[V˜ (a ) − V˜ (a)] 1+ V˜ (a)

% .

Suppose V˜ ∈ R−β , with β ≥ 0. Then L(a) ≡ aβ V˜ (a) ∈ L by deﬁnition of R−β . For β > 0 we have

(b)

[V˜ (ca) − V˜ (a)] (ca)−β L(ca) − a−β L(a) = a−β L(a) V˜ (a) L(ca) −β =c − 1 + (c−β − 1) → 0 , L(a)

146

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

since L ∈ L, a → 0, and c → 1 as (a ∨ a ) → 0, while [V˜ (ca) − V˜ (a)]/V˜ (a) → 0 directly when β = 0 and V˜ ∈ R0 . Of course, [V˜ (c a ) − V˜ (a )]/V˜ (a ) is identical to the previous claim, since c → 1. For β > 0 (with 1 − θ ≡ a /a → 1 as (a ∨ a ) → 0) the other term in (a) satisﬁes (c)

(d)

[a(1 − θ)]−β L(a(1 − θ)) − a−β L(a)] [V˜ (a ) − V˜ (a)] = a−β L(a) V˜ (a) L(a(1 − θ)) −β = (1 − θ) − 1 + [(1 − θ)−β − 1] → 0, L(a)

while [V˜ (a ) − V˜ (a)]/V˜ (a) → 0 directly for β = 0. Note that if β = 0, we needed only 0 < a/M < a < M a < ∞ to conclude that V˜ (a )/V˜ (a) → 1. 2 Deﬁnition 10.2 (Uniformity class Ku ) qfs deﬁned by

Consider the uniformity classes Ku of

(36)

Ku ≡ {K : K(1 − t) = −K(t) = t−β for 0 < t < 1/2, for all 0 < β ≤ β0 } ,

(37)

Ku ≡ {K(t) ≡ [−t−β · 1[ta· ] ] Q(t) : 0 < β ≤ β0 , Q ∈ Q} ,

where Q is any class of qfs Q contained in L = R0 that are uniformly slowly varying at 0 and at 1. Theorem 10.2 (Uniformity class Ku ) classes Ku above that (38)

As a ∨ a → 0, we have for either of the

supK∈Ku D ∨ D → 0 ,

provided that either (39)

0 < lim inf(c ∧ c ) ≤ lim sup(c ∧ c ) < ∞ (if β = 0)

(40)

c ≡ ca → 1 and c ≡ ca → 1 as a ∨ a → 0 (if β > 0).

or

Thus all of (24)–(28) hold uniformly over such classes Ku . Proof. We now prove this for the class of qfs (37), with the result for the class of qfs (36) as a corollary. If we bound 4(a ∨ a )/(a ∧ a ) by M 2 , then (23) gives (a) (b)

D ≤ M |K+ (ct) − K+ (t)|/|K+ (t)| ≤ 2M |(ct)−β Q(ct) − t−β Q(t)|/t−β |Q(t)| = c−β |Q(ct) − Q(t)|/|Q(t)| + (c−β − 1) → 0 .

2

11.

INEQUALITIES FOR WINSORIZED QUANTILE FUNCTIONS

147

11

Inequalities for Winsorized Quantile Functions

The key to the smoothly functioning power of the following inequalities is the formulation of the tail terms in (3) and (7) below. The inequalities look clumsy, but they work extremely eﬃciently. Note that the upper bounds that appear in equations (1) and (6) do not depend at all on the unknown qf K. Roughly, parts (i), (ii), and (iii) prepare for the central limit theorem (CLT), the law of large numbers (LLN) (whether for second moments or for ﬁrst moments), and general integral variance estimators. [If σ ˜n = 0, then just multiply through by this symbol. We agree that b a ≡ (a,b) throughout the course of this section.] Use the notation of section 7.5. All qfs are assumed to be nondegenerate. Inequality 11.1 (Winsorized variance inequalities) Let K(·) be arbitrary. Let 0 < c ≤ an < 1 − an ≤ 1 − c < 1. Let 1/n ≤ r/n ≤ 12 ≤ 1 − r /n ≤ 1 − 1/n. (i) For the CLT Fix ν ∈ (0, 12 ). The following statements hold: 1−an −ν 9/ν 1/2−ν (1) n [t(1 − t)] dK(t)/˜ σn ≤ . an (kn ∧ kn )ν an −ν √ (2) n [t(1 − t)]1/2−ν dK(t)/˜ σn ≤ an |K+ (c) − K+ (an )| /˜ σn . c

1−an

(3)

−ν

n an

where

√ γn ≡ √ γn ≡

[t(1 − t)]

1/2−ν

9/ν dK(t)/˜ σn ≤ + γn + γn , [ (kn ∨ r) ∧ (kn ∨ r )]ν

r |K+ (an ) − K+ (r/n)| √ × 1[an
and

r |K(1 − an ) − K(1 − r /n)| √ × 1[an
(ii) For the WLLN and variance estimation Fix 0 < ν < 1. Recall (7.5.7) ¯ n2 . Then for the distribution function K 1−an 2 ¯ 2 (t)/˜ (4) n−ν [t(1 − t)]1−ν dK σn2 ≤ + γ˜n + γ˜n , n [(kn ∨ r) ∧ (kn ∨ r )]ν an where γ˜n ≡

2 2 r |K+ (an ) − K+ (r/n)| × 1[an
γ˜n ≡

r |K 2 (1 − an ) − K 2 (1 − r /n)| × 1[an
and

Additionally, √ √ 1−an 1−a 8 t(1 − t)K(t)an n r r −ν 1−ν √ √ (5) n [t(1−t)] dK(t) ≤ + γ + γ . n ν [(kn ∨ r) ∧ (kn ∨ r )]ν n n n an

148

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

(iii) For variance estimation involving double integrals Let gn (s, t) ≡ (ns(1 − s))−ν0 (nt(1 − t))−ν1 (s ∧ t − st)

for s, t ∈ (0, 1),

where we set 0 ≤ ν0 , ν1 < 12 with 0 < ν0 + ν1 < 1 (to “dominate” the diﬀerence between the “limiting” and estimated variances). Then 1−an 1−an (6) gn (s, t) dK(s) dK(t)/˜ σn2 ≤ 2/(kn ∧ kn )ν0 +ν1 . an an Since we also have 1/n ≤ r/n ≤ 12 ≤ 1 − r /n ≤ 1 − 1/n, then 1−an 1−an gn (s, t) dK(s) dK(t)/˜ σn2 an

(7)

an

≤

√ √ 2 + 4 r γ + 4 r γn . n [(kn ∨ r) ∧ (kn ∨ r )]ν0 +ν1

˜ ≡K ˜ a,a denote K Winsorized outside (a, 1 − a ), µ ˜ ˜ ≡ E K(ξ), and Proof. Let K 2 ˜ σ ˜ ≡ Var[K(ξ)]. Integration by parts, then Cauchy–Schwarz, and the elementary inequality (|a| + |b| + |c|)2 ≤ (3 max(|a|, |b|, |c|))2 ≤ 9(a2 + b2 + c2 ) give (with a ˜· ≡ inf{t : K(t) − µ ˜ ≥ 0}) that 1−a 1−a [t(1 − t)]1/2−ν dK(t) = a [t(1 − t)]1/2−ν d[K(t) − µ ˜] a ≤ a1/2−ν |K+ (a) − µ ˜| + a1/2−ν |K(1 − a ) − µ ˜| 1−a + a |K(t) − µ ˜| (1/2 − ν) |1 − 2t| [t(1 − t)]−(1/2+ν) dt √ √ ˜|} + (a )−ν { a |K(1 − a ) − µ ˜|} ≤ a−ν { a |K+ (a) − µ 1−a √ +(1/2 − ν) { a [K(t) − µ ˜]2 dt}1/2 (a ∧ a )−ν 2ν / ν 1−a √ ≤ (3/ ν ) (a ∧ a )−ν { a [K(t) − µ ˜]2 dt

(8)

+a [K+ (a) − µ ˜]2 + a [K(1 − a ) − µ ˜]2 }1/2} √ ≤ (3/ ν )(a ∧ a )−ν σ ˜K (a, a ) for any 0 < ν < 12 .

This gives (1). Also, 1−a [t(1 − t)]1−ν dK(t) a ≤ a1−ν |K+ (a)| + (a )1−ν |K(1 − a )| +

1−a a

|K(t)/[t(1 − t)]ν | dt

1−a ≤ (a ∧ a )−ν {a|K+ (a)| + 2 a |K(t)| dt + a |K(1 − a )|} (9)

˜ ≤ 2 E|K(ξ)|/(a ∧ a )ν

for any 0 < ν < 1.

¯ 2 ≡ −[(K − µ Apply (9) to K ˜)− ]2 + [(K − µ ˜)+ ]2 to get (4) via 1−a ¯ 2 (t)/˜ (10) [t(1 − t)]1−ν dK σ2 ≤ 2 (a ∧ a )−ν for any 0 < ν < 1. a

11.

INEQUALITIES FOR WINSORIZED QUANTILE FUNCTIONS

149

Integration by parts gives (5) via 1−a a

≤

[t(1 − t)]1−ν dK(t)

|a(1 − a)K+ (a)| |a (1 − a )K(1 − a )| + [a(1 − a)]ν [a (1 − a )]ν + t(1 −

(11)

1−a t)K(t)a

1−a

|1 − 2t| dt [t(1 − t)]1+ν

a

≤ (8/ν) (a ∧ a )−ν t(1 − t)K(t)1−a a

for any 0 < ν < 1.

Now observe that for 0 < c ≤ r/n we have (since r ≥ 1) (12) n−ν [t(1 − t)]1/2−ν dK(t) ≤ r/n |K+ (c) − K+ (r/n)|. (c,r/n] Letting (c, r/n] ≡ (c, an ] in (12), we obtain (2). There is a symmetric right tail result. Now let (c, r/n] = (an , r/n] in (12), and assume an ≤ r/n < 1 − r /n ≤ 1 − an also now holds. Then −ν [t(1 − t)]1/2−ν dK(t) (an ,1−a ) n n

≤ ≤

(13)

+ (an ,r/n]

(r/n,1−r /n)

(r/n,1−r /n)

+

[1−r /n,1−an )

n−ν [t(1 − t)]1/2−ν dK(t)

n−ν [t(1 − t)]1/2−ν dK(t) + γn + γn

√ ≤ (2/ ν ) σ ˜K (r/n, r /n) /(r ∧ r )ν + γn + γn √ ≤ (2/ ν ) σ ˜K (an , an ) /[(r ∨ kn ) ∧ (r ∨ kn )]ν + γn + γn

using σ ˜K (a, a ) as a and a . This gives (3). Consider part (iii). Now, 1−r /n 1−r /n r/n

(a)

r/n

gn (s, t) dK(s) dK(t)

≤ 2 (r ∧ r )−(ν0 +ν1 )

1−r /n

1−r /n

(s ∧ t − st) dK(s) dK(t) r/n

(b)

r/n

2 ≤ 2σ ˜K (r/n, r /n) /(r ∧ r )(ν0 +ν1 ) .

Setting r = kn and r = kn in (b) gives (6). If 0 ≤ s ≤ t ≤ 1, then (c) (d)

gn (s, t) =

1 1 s1−ν0 1 × × × ν1 × (1 − t)1−ν1 (1 − s)ν0 nν0 tν1 n

≤ 2 × (r/n)ν0 × (r/n)1−ν0 −ν1 × (r/n)ν1 × 1 ≤ 2r/n

if an ≤ s ≤ r/n;

150

CHAPTER 7. DISTRIBUTION AND QUANTILE FUNCTIONS

and this bound extends to an ≤ t ≤ s ≤ r/n by symmetry. Thus g(s, t) dK(s) dK(t) σ ˜n2 (an ,r/n] (an ,1−an ) (e)

≤ 2r ×

|K+ (an ) − K+ (r/n)| |K+ (an )| + |K(1 − an )| √ √ × , nσ ˜n nσ ˜n

and the integral over (an , 1 − an ) × (an , r/n] gives the same value. Whenever 1 − r /n ≤ t ≤ 1 − an , then both the integral over (an , 1 − an ) × [1 − r /n, 1 − an ) and that over [1 − r /n, 1 − an ) × (an , 1 − an ) give equal values symmetric to (e). Adding the four bounds, together with (7.8.5) and (6), then gives (7). 2 Exercise 11.1 Suppose kn ∞, but an ≡ kn /n → 0. Let 0 < ν < 14 be given. Deﬁne ln ≡ kn1+2ν , and let en ≡ ln /n. Show that for every nondegenerate qf K we necessarily have √ 1−an −ν en (14) + n [t(1 − t)]1/2−ν dK/˜ σn ≤ 2/knν . an 1−en [Thus the tails will be separated from the vast middle by a vanishingly small piece that is inconsequential.]

Chapter 8

Independence and Conditional Distributions 1

Independence

The idea of independence of events A and B is that the occurrence or nonoccurrence of A has absolutely nothing to do with the occurrence or nonoccurrence of B. It is customary to say that A and B are independent events if (1)

P (AB) = P (A)P (B).

Classes C and D of events are called independent classes if (1) holds for all A ∈ C and all B ∈ D. We need to deﬁne the independence of more complicated objects. Deﬁnition 1.1 (Independence) Consider a ﬁxed probability space (Ω, A, P ). (a) Consider various sub σ-ﬁelds of A. Call such σ-ﬁelds A1 , . . . , An independent σ-ﬁelds if they satisfy (2) P (A1 ∩ · · · ∩ An ) = n1 P (Ai ) whenever Ai ∈ Ai for 1 ≤ i ≤ n. The σ-ﬁelds A1 , A2 , . . . are called independent σ-ﬁelds if A1 , . . . , An are independent for each n ≥ 2. (b) Rvs X1 , . . . , Xn are called independent rvs if the σ-ﬁelds F(Xi ) ≡ Xi−1 (B) (for 1 ≤ i ≤ n) are independent. Rvs X1 , X2 , . . . are called independent rvs if all X1 , . . . , Xn are independent. (c) Events A1 , . . . , An are independent events if σ[A1 ], . . . , σ[An ] are independent σ-ﬁelds; here we let (3)

σ[Ai ] ≡ {φ, Ai , Aci , Ω}.

The next exercise is helpful because it will relate the rather formidable deﬁnition of independent events in (3) back to the simple deﬁnition (1).

151

152 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS Exercise 1.1 (a) Show that P (AB) = P (A)P (B) if and only if {∅, A, Ac , Ω} and {∅, B, B c , Ω} are independent σ-ﬁelds. [Thus we maintain the familiar (1).] (b) Show that A1 , . . . , An are independent if and only if P (Ai1 · · · Aik ) = kj=1 P (Aij ) (4) whenever 1 ≤ i1 < · · · < ik ≤ n with 1 ≤ k ≤ n. Remark 1.1 When discussing a pair of possibly independent events, one should draw the Venn diagram as a square representing Ω divided into half vertically (with respect to A, Ac ) and into half horizontally (with respect to B, Bc ) creating four cells (rather than as the familiar two-circle picture). Also, if one writes on the table the probability of each of the four combinations AB, ABc , Ac B, Ac B c , one has the contingency table superimposed on the picture. (See ﬁgure 1.1.) [This extends to two partitions (A1 , . . . , Am ) and (B1 , . . . , Bn ), but not to three events.] 2

A

Ac

B

P (AB)

P(AcB)

Bc

P (AB c)

P(AcB c)

Figure 1.1 The 2 × 2 table.

Theorem 1.1 (Expectation of products) Suppose X and Y are independent rvs for which f (X) and g(Y ) are integrable. Then f (X) g(Y ) is integrable, and (5)

E[f (X) g(Y )] = Ef (X) Eg(Y ) .

Proof. The assertion is obvious for f = 1A and g = 1B . Now hold f = 1A ﬁxed, and proceed through simple and nonnegative g. Then with g held ﬁxed, proceed through simple, nonnegative, and integrable f . Then extend to integrable g. 2

1.

INDEPENDENCE

153

Proposition 1.1 (Extending independence on π-systems) (a) Suppose the π-system C and a class D are independent. Then σ[C] and D

are independent.

(b) Suppose the π-systems C and D are independent. Then σ[C] and σ[D]

are independent.

(c) If C1 , . . . , Cn are independent π-systems (see (2)), then σ[C1 ], . . . , σ[Cn ] Proof. (i)

are independent σ-ﬁelds.

(a) Fix D ∈ D, and deﬁne CD ≡ {A ∈ σ[C] : P (AD) = P (A)P (D)}.

We now demonstrate that CD is a λ-system. Trivially, Ω ∈ CD . If A, B ∈ CD with A ⊂ B, then P ((A \ B)D) = P (AD \ BD) = P (AD) − P (BD) (j)

= P (A)P (D) − P (B)P (D) = P (A \ B)P (D);

and this implies that A \ B ∈ CD . If An A with all An ∈ CD , then (k)

P (AD) = P (lim An D) = lim P (An D) = lim P (An )P (D) = P (A)P (D);

and this implies that A ∈ CD . Thus CD is a λ-system, and it contains the π-system C. Thus CD ⊃ σ[C] by (1.1.19). Just apply part (a) to the π-system D and the arbitrary class σ[C] to obtain (b). The now minor (c) is left to exercise 1.3. 2 Theorem 1.2 Let X1 , X2 , . . . be independent rvs on (Ω, A, P ). Let i1 , i2 , . . . and j1 , j2 , . . . be disjoint sets of integers. (a) Then (6)

F(Xi1 , Xi2 , . . .) and F(Xj1 , Xj2 , . . .)

are independent σ-ﬁelds.

(b) This extends immediately to countably many disjoint sets of integers. Corollary 1 (Preservation of independence) Any rvs h1 (Xi1 , Xi2 , . . .), h2 (Xj1 , Xj2 , . . .), . . . (that are based on disjoint sets of the underlying independent rvs Xi ) are themselves independent rvs, for any choice of the B-B∞ -measurable functions h1 , h2 , . . . . Proof. Let C denote all sets of the form C ≡ [Xi1 ∈ B1 , . . . , Xim ∈ Bm ], for some m ≥ 1 and for B1 , . . . , Bm in B. Let D denote all sets of the form D ≡ [Xj1 ∈ B1 , . . . , Xjn ∈ Bn ] for some n ≥ 1 and sets B1 , . . . , Bn in B. Both C and D are π ¯ -systems, and σ[C] = F(Xi1 , Xi2 , . . .) and σ[D] = F(Xj1 , Xj2 , . . .). It

154 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS follows that (p)

n P (C ∩ D) = P ({∩m k=1 [Xik ∈ Bk ]} ∩ {∩l=1 [Xjl ∈ Bl ]}) n = m by independence k=1 P (Xik ∈ Bk ) l=1 P (Xjl ∈ Bl ) n = P (∩m k=1 [Xik ∈ Bk ]) P (∩l=1 [Xjl ∈ Bl ])

(q)

by independence

= P (C) P (D),

so that C and D are independent classes. Thus σ[C] and σ[D] are independent by proposition 1.1(b), as is required for (6). The extension to countably many disjoint sets of indices is done by induction using proposition 1.1(c), and is left to the exercises. [The corollary is immediate.] 2

Exercise 1.2

Prove theorem 1.2(b).

Exercise 1.3

Prove proposition 1.1(c).

Criteria for Independence Theorem 1.3 (7)

The rvs (X1 , . . . , Xn ) are independent rvs if and only if

FX1 ,...,Xn (x1 , . . . , xn ) = FX1 (x1 ) · · · FXn (xn )

for all x1 , . . . , xn .

Proof. Clearly, independence implies that the joint df factors. For the converse we suppose that the joint df factors. Then for all x1 , . . . , xn we have P (X1 ≤ x1 , . . . , Xn ≤ xn ) = P (X1 ≤ x1 ) · · · P (Xn ≤ xn ). That is, the classes Ci ≡ {[Xi ≤ xi ] : xi ∈ R} are independent, and they are π-systems, with σ[Ci ] = F(Xi ). Independence of X1 , . . . , Xn then follows from proposition 1.1(c). 2

Exercise 1.4 Rvs X, Y that take on only a countable number of values are independent if and only if P ([X = ai ][Y = bj ]) = P (X = ai )P (Y = bj ) for all i, j. Exercise 1.5 Show that rvs X, Y having a density f (·, ·) are independent if and only if the density factors to give f (x, y) = fX (x)fY (y) for all x, y. Remark 1.2 That rv’s X, Y are independent if and only if their characteristic function factors appears as theorem 13.3.3 below.

2.

THE TAIL σ-FIELD

155

2

The Tail σ-Field

Deﬁnition 2.1 (The tail σ-ﬁeld) Start with an arbitrary random element ≡ (X1 , X2 , . . .) from (Ω, A, P ) to (R∞ , B∞ ). Then T ≡ ∞ F(Xn , Xn+1 , . . .) X n=1 is called the tail σ-ﬁeld, and any event D ∈ T is called a tail event. Theorem 2.1 (Kolmogorov’s 0-1 law) If X1 , X2 , . . . are independent rvs, then P (D) equals 0 or 1 for all tail events D in the tail σ-ﬁeld T . = X −1 (B∞ ). By the approximation Proof. Specify a set D ∈ T ⊂ F(X) lemma of exercise 1.2.3 and the introduction to section 5.5, there exists an integer −1 m n and a set Dn in the nth member of ∪m F(X1 , . . . , Xm ) = ∪m X (Bm ) = (a ﬁeld) such that P (Dn ∆D) → 0. Thus both P (D ∩ Dn ) → P (D) and P (Dn ) → P (D) occur. Happily, D ∈ T ⊂ F(Xn+1 , . . .), so that D and Dn ∈ F(X1 , . . . , Xn ) are independent. Hence P (D)2 ← P (D) P (Dn ) = P (D ∩ Dn ) → P (D), yielding P (D)2 = P (D). Thus P (D) = 0 or 1.

Note that for any Borel sets B1 , B2 , . . . in B,

Remark 2.1 (1) since

[Xn ∈ Bn i.o.] equals lim [Xn ∈ Bn ] = ∞

m=n [Xm

∞ ∞ n=1

m=n [Xm

∈ Bm ] ∈ T ,

∈ Bm ] ∈ F(Xn , . . .). Also,

lim [Xn ∈ Bn ] = (2)

2

∞ ∞ n=1

m=n

[Xm ∈ Bm ] = (

∞ ∞ n=1

m=n

c c [Xm ∈ Bm ])

c c = ( lim [Xm ∈ Bm ]) ∈ T .

Note also that [ω : Xn (ω) → (some ﬁnite X(ω))] (3)

=

∞ ∞ ∞ k=1

Likewise, if Sn =

n=1

n i=1

m=n

[ω : |Xm (ω) − Xn (ω)| ≤ 1/k] ∈ T .

Xi , then

[ω : Sn (ω) → (some ﬁnite S(ω))] (4)

=

∞ ∞ ∞ k=1

n=1

m=n

m [ω : | i=n+1 Xi (ω)| ≤ 1/k] ∈ T .

2

156 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS

The Symmetric σ-Field Deﬁnition 2.2 (Symmetric sets) Let π denote any mapping of the integers onto themselves that (for some ﬁnite n) merely permutes the ﬁrst n integers. Let ≡ (X1 , X2 , . . .) be B∞ -measurable, and set X π ≡ (Xπ(1) , Xπ(2) , . . .). Then X −1 (B) for A ≡ X −1 (B) for some B ∈ B∞ is called a symmetric set if A = X π all such π. Let S denote the collection of all symmetric sets. Exercise 2.1 (Hewitt–Savage 0-1 law) ≡ (X1 , X2 , . . .) have iid coordinates Xk . Let X (a) Show that P (A) equals 0 or 1 for every A in S. (b) Show that S is a σ-ﬁeld, called the symmetric σ-ﬁeld. (c) Show that the tail σ-ﬁeld T is a subset of the symmetric σ-ﬁeld S. (d) Give an example where T is a proper subset of S. [Hint. Use the approximation lemma of exercise 1.2.3 as it was used above.]

3.

UNCORRELATED RANDOM VARIABLES

3

Uncorrelated Random Variables

157

Recall from deﬁnition 3.4.1 that X ∼ = (µ, σ2 ) denotes that X has mean µ and variance σ 2 that is assumed to be ﬁnite. Deﬁnition 3.1 (Correlation) If X1 , . . . , Xn have ﬁnite variances, then we call them uncorrelated if Cov[Xi , Xj ] ≡ E{(Xi − EXi )(Xj − EXj )} = 0 for all i = j. Deﬁne the dimensionless quantity Cov[Xi , Xj ] Corr[Xi , Xj ] ≡ Var[Xi ]Var[Xj ] ≡ (X1 , . . . , Xn ) , then the n × n to be the correlation between Xi and Xj . If X covariance matrix of X is deﬁned to be the matrix Σ ≡ |[σij ]| whose (i, j)th element is σij ≡ Cov[Xi , Xj ]. Proposition 3.1 Proof.

Independent rvs with ﬁnite variances are uncorrelated.

Now, Cov[X, Y ] = E[(X − µX )(Y − µY )] = E(X − µX )E(Y − µY ) = 0 · 0 = 0,

(a) where

0 ≤ {Cov[X, Y ]}2 ≤ Var[X] Var[Y ] < ∞

(b)

by Cauchy–Schwarz.

2

Note from (b) (or recall from (3.4.14)) that (1)

| Corr[X, Y ] | ≤ 1

for any X and Y having ﬁnite variance.

Proposition 3.2 If (X1 , . . . , Xn ) are uncorrelated and Xi ∼ = (µi , σi2 ), then n n n

(2) ai Xi ∼ ai µi , a2i σi2 . = 1

1

1

In particular, suppose X1 , . . . , Xn are uncorrelated (µ, σ2 ). Then n σ2

√ ¯n ≡ 1 ¯ n − µ)/σ ∼ (3) X Xi ∼ , while n(X = µ, = (0, 1), n 1 n provided that 0 < σ < ∞. Moreover m n m n Cov ai Xi , bj Yj = Cov ai (Xi − µXi ), bj (Yj − µYj ) i=1

(4)

=

n m

j=1

i=1

ai bj Cov[Xi , Yj ] .

i=1 j=1

Note that if (5) Proof.

Y = AX,

ΣY = A ΣX A .

then

This is trivial.

2

j=1

158 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS

4

Basic Properties of Conditional Expectation

Illustrations will follow the deﬁnitions. Deﬁnition 4.1 (Conditional expectation) Let (Ω, A, P ) denote a probability space. Let D denote a sub σ-ﬁeld of A. Let Y be a rv on (Ω, A, P ) for which E|Y | < ∞. By E(Y |D)(·) we mean any D-measurable function on Ω such that (1) E(Y |D)(ω) dP (ω) = Y (ω) dP (ω) for all D ∈ D. D

D

[Such a function exists and is unique a.e. P , as is seen below.] If X is another rv on (Ω, A, P ), then (2)

E(Y |X)(ω) ≡ E(Y |F(X));

¯ for the Borel subsets B of the real line R. we recall that F(X) ≡ X −1 (B) Justiﬁcation of deﬁnition 4.1. First let Y ≥ 0. Deﬁne a measure ν on D by ν(D) ≡ D Y dP (a) for all D ∈ D. Now, ν is a measure on (Ω, D) by example 4.1.1, and the restriction of P to D (denoted by P |D) is another measure on (Ω, D). Moreover, that ν P |D is trivial. Thus the Radon–Nikodym theorem guarantees, uniquely a.e. P |D, a D-measurable function h such that (b) for all D ∈ D. ν(D) = D h d(P |D) = D h dP Now, being D-measurable and unique a.s. P |D implies that the function h is unique a.s. P . Deﬁne E(Y |D) ≡ h. For general Y we let E(Y |D) ≡ E(Y + |D) − E(Y − |D). Radon–Nikodym derivatives are only unique a.e., and any function that works is called a version of the Radon–Nikodym derivative. 2 Proposition 4.1 Suppose that Z is a rv on (Ω, A) that is F(X)-measurable. Then there exists a measurable function g on (R, B) such that Z = g(X). Proof.

This is just proposition 2.2.5 again.

2

Notation 4.1 Since E(Y |X) = E(Y |F(X)) is F(X)-measurable, the previous proposition shows that h ≡ E(Y |X) = g(X) for some measurable function g on (R, B). The theorem of the unconscious statistician gives X −1 (B) g(X) dP = g dPX , where we have written the general set D ∈ F(X) as D = X −1 (B) for B some B ∈ B. Thus we may deﬁne E(Y |X = x) = g(x) to be a B-measurable function on R for which (3) E(Y | X = x) dPX (x) = Y (ω) dP (ω) for all B ∈ B. B

X −1 (B)

(This function exists by the above, and it is easily seen to be unique a.e. PX .) We summarize: (4)

If g(x) ≡ E(Y | X = x),

then h(ω) ≡ E(Y | X)(ω) = g(X(ω)).

4.

BASIC PROPERTIES OF CONDITIONAL EXPECTATION

159

Deﬁnition 4.2 (Conditional probability) Since P (A) = E1A for standard probability, deﬁne the conditional probability of A given D, denoted by P (A|D), by (5)

P (A|D) ≡ E(1A |D).

Thus P (A|D) is a D-measurable function on Ω satisfying P (A ∩ D) = (6) P (A|D) dP for all D ∈ D, D

and it exists and is unique a.s. P . Also, P (A|X) ≡ P (A|F(X)). Thus P (A|X)(ω) = g(X(ω)), where g(x) ≡ P (A|X = x) is a B-measurable function satisfying −1 (7) P (A ∩ X (B)) = P (A|X = x) dPX (x) for all B ∈ B. B

This function exists and is unique a.s. PX . Exercise 4.1 (“Discrete” conditional probability) Suppose Ω = i Di , and then deﬁne D = σ[D1 , D2 , . . .]. Show that (whether the summation is ﬁnite or countable) the diﬀerent expressions needed on the diﬀerent sets Di of the partition D can be expressed together via (8)

P (A|D) =

P (ADi ) i

P (Di )

1Di ,

i) where (just for deﬁniteness) PP(AD (Di ) ≡ P (A) if P (Di ) = 0. For general Y ∈ L1 show that the function E(Y |D) takes the form 1 (9) E(Y |D) = Y dP 1Di , P (Di ) Di i

with the term in braces deﬁned to be 0 if P (Di ) = 0. Remark 4.1

Note that

(10)

0 ≤ P (A|D) ≤ 1

(11)

P(

(12)

P (∅|D) = 0

(13)

A1 ⊂ A2

∞ 1

Ai |D) =

a.s. P, ∞ 1

P (Ai |D)

a.s. P,

a.s. P, implies

P (A1 |D) ≤ P (A2 |D)

a.s. P.

[To see these, just apply parts (16)(monotonicity) and (17)(MCT) of theorem 4.1 appearing below.] These properties remind us of a probability distribution. 2

160 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS Discussion 4.1 Given that the event B has occurred (with P (B) > 0), how likely to occur now is the event A. The classic elementary approach deﬁnes the conditional probability of A given B by P (A|B) ≡ P (AB)/P (B). Thus we have taken a revisualized view of things, while regarding B as the updated sample space. For any event B, only that portion AB of the event A is relevant (as B was known to have occurred). Thus all that matters is the probabilistic size P (AB) of AB relative to the probabilistic size P (B) of B. The resulting P (·|B) is a probability 2 distribution over AB ≡ {AB : A ∈ A}. Discussion 4.2 Let us consider the current approach to conditional probability. We will illustrate it in a special case. Let A ∈ B2 denote a two-dimensional Borel ≡ (X 2 + X 2 )1/2 , so that T = t deﬁnes in the plane the circle set. Let T ≡ T (X) 1 2 Ct ≡ {(x1 , x2 ) : x21 + x22 = t2 }. Let B ∈ B1 denote a one-dimensional Borel set, and then let D ≡ T −1 (B) = ∪{Ct : t ∈ B}. Requirements (6) and (7) become P (AD) = P (A ∩ (∪{Ct : t ∈ B})) = ∪{Ct :t∈B} hA (x ) dPX (x ) = B gA (t) dPT (t). So if g(·) is given a value at t indicating the probabilistic proportion of A ∩ Ct that belongs to Ct (or h(x ) is given this same value at all x ∈ Ct ), then the above equation ought to be satisﬁed. [When densities exist, such a value would seem to be gA (t) = Ct 1A (x ) pX (x ) dx / Ct pX (x ) dx, while hA (x ) would be assigned this same value at each x ∈ Ct .] (Note how we have approached the problem from a diﬀerent point of view than that in discussion 4.1.) 2 Remark 4.2 (Elementary texts) For discrete rvs, a conditional distribution is speciﬁed by g(y|x) ≡ P (Y = y |X = x) ≡ P (Y = y and X = x)/P (X = x). (This is in line with discussion 4.1.) One “natural approximation” of this approach for continuous rvs considers x+h fX,Y (r, y) dr g(y|x) = lim x−h . x+h h→0 x−h fX (r) dr But making this approach rigorous fails without suﬃcient smoothness, and leads to a tedious and limited theory. So elementary texts just suggest the even more blatant and “less rigorous” imitation of the discrete result via . fX,Y (x, y) ∆x∆y . fX,Y (x, y) ∆y g(y|x) ∆y = = . fX (x) ∆x fX (x) But in discussion 4.2 we would instead let hA (x, y) = A∩Lx f (x, u) du/ Lx f (x, u) du at each point of the line Lx ≡ {(x , y ) : x = x}. In example 8.5.1 below, we will see that it is only the present rigorous approach of discussion 4.2 that leads us eventually to the simplest solution involving this last g(y|x) = fX,Y (x, y)/fX (x).

4.

BASIC PROPERTIES OF CONDITIONAL EXPECTATION

161

Moreover, if (x(t), y(t)), a ≤ t ≤ b, parametrizes a smooth curve (imagine a circle about the origin, or a line of slope 135◦ ), it is deﬁnition 4.2 that leads rigorously to formulas of the type f (x(t), y(t)) (dx/dt)2 + (dy/dt)2 for a ≤ t ≤ b b 2 2 a f (x(t ), y(t )) (dx/dt ) + (dy/dt ) dt for the conditional density at the point t given that one is on the curve.

2

Example 4.1 Let an urn consist of six balls identical except for the numbers 1, 2, 2, 3, 3, 3. Let X1 and X2 represent a sample of size two drawn with replacement, and set Y = X2 and S = X1 + X2 . Consider ﬁgure 4.1 below. In the ﬁgure (a) we see the sample space Ω of (X1 , X2 ) with the values of S superimposed, while the ﬁgure (b) superimposes the probability function on the same representation of Ω. In the ﬁgure (c) we picture the ﬁve “diagonal sets” that generate D ≡ S−1 (B). The three-part ﬁgure (d) depicts P (Y = i |D)(·) as a D-measurable function on Ω for each of the three choices [Y = 1], [Y = 2], and [Y = 3] for A, while the ﬁgure (e) depicts E(Y |D)(ω) as a D-measurable function. [Had we used the elementary deﬁnition of P (Y = · |S = k) as a function of y for each ﬁxed k, then the conditional distributions would have been those shown along the ﬁve diagonals in the ﬁgure (f), while E(Y |S = k) is shown at the end of each diagonal.] 2 Exercise 4.2 (a) Repeat example 4.1 and the accompanying ﬁgure, but now in the context of sampling without replacement. (b) Make up an interesting second example. Exercise 4.3 Let Y denote a rv on some (Ω, A, P ) that takes on the eight values 1, . . . , 8 with probabilities 1/32, 2/32, 3/32, 4/32, 15/32, 4/32, 1/32, 2/32, respectively. Let C ≡ F(Y ), and let Ci ≡ [Y = i] and pi ≡ P (Ci ) for 1 ≤ i ≤ 8. Let D ≡ σ[{C1 +C5 , C2 +C6 , C3 +C7 , C4 +C8 }], E ≡ σ[{C1 +C2 +C5 +C6 , C3 +C4 +C7 +C8 }], and F ≡ {Ω, ∅}. (a) Represent Ω as a 2 × 4 rectangle having eight 1 × 1 cells representing C1 , . . . , C4 in the ﬁrst row and C5 , . . . , C8 in the second row. Enter the appropriate values of Y (ω) and pi in each cell, forming a table. Evaluate E(Y ). (b) Evaluate E(Y |D). Present this function in a similar table. Evaluate E(E(Y |D)). (c) Evaluate E(Y |E). Present this function in a similar table. Evaluate E(E(Y |E)). (d) Evaluate E(Y |F). Present this function in a similar table. Evaluate E(E(Y |F)). For the most part, the use of regular conditional probabilities (as deﬁned in the next section) can be avoided by appeal to the following theorem.

162 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS

Y = X2

3

4

5

6

3 36

6 36

9 36

2

3

4

5

2 36

4 36

6 36

1

2 1

3 2

4 3

1 36

2 36

3 36

X1

(a)

(b)

(c)

3 10

0

0

4 10

1 2

0

3 10

1 2

1

1 2

3 10

0

1 2

4 10

1 2

0

3 10

1 2

1

1 2

3 10

0

1 2

4 10

0

0

3 10

P (Y = 1|D)(·)

P (Y = 2|D)(·) P (A |D)(·) =

Σ k

P (Y = 3|D)(·)

P (A ∩ Dk) 1Dk(·) P (Dk)

(d)

2

5 2

3

3 10

1 2

1

3

3 2

2

5 2

1 2

4 10

1 2

5 2

1

3 2

2

1

1 2

3 10

E (Y |D)(·)

1

3 2

2

(e)

(f )

Figure 4.1 Conditional probability and conditional expectation.

4.

BASIC PROPERTIES OF CONDITIONAL EXPECTATION

163

Theorem 4.1 (Properties of conditional expectation) Let X, Y, Yn be integrable rvs on (Ω, A, P ). Let D be a sub σ-ﬁeld of A. Let g be measurable. Then for any versions of the conditional expectations, the following hold: (14)

(Linearity) E(aX + bY |D) = a E(X|D) + b E(Y |D) a.s. P (or, a.s. P |D)).

(15)

EY = E [ E(Y |D)].

(16)

(Monotonicity) X ≤ Y a.s. P implies E(X|D) ≤ E(Y |D) a.s. P.

(17)

(MCT) If 0 ≤ Yn Y a.s. P, then E(Yn |D) E(Y |D) a.s. P.

(18)

(Fatou) If 0 ≤ Yn a.s. P, then E( lim Yn |D) ≤ lim E(Yn |D) a.s. P.

(19)

(DCT) If all |Yn | ≤ X and Yn →a.s. Y, then E(Yn |D) →a.s. E(Y |D) .

(20)

If Y is D-measurable and XY ∈ L1 (P ), then E(XY |D) =a.s. Y E(X|D) .

(21)

If F(Y ) and D are independent, then E(Y |D) = EY a.s. P.

(22)

(Stepwise smoothing). If D ⊂ E ⊂ A, then E[E(Y |E)|D] = E(Y |D) a.s. P.

(23)

If F(Y, X1 ) is independent of F(X2 ), then E(Y |X1 , X2 ) = E(Y |X1 ) a.s. P.

(24)

Cr , H¨ older, Liapunov, Minkowski, and Jensen inequalities hold for E(·|D). Jensen:

(25) (26)

g(E(Y |D)) ≤a.s. E[g(Y )|D] for g convex with g(Y ) integrable.

If Yn →Lr Y for r ≥ 1, then E(Yn |D) →Lr E(Y |D). hD (·) is a determination of E(Y |D)

if and only if

E(XY ) = E(XhD ) for all D-measurable rvs X. (27)

If P (D) = 0 or 1 for all D ∈ D, then E(Y |D) = EY a.s. P.

Proof.

We ﬁrst prove (14). Now, by linearity of expectation [ a E(X|D) + b E(Y |D)] dP = a D E(X|D) dP + b D E(Y |D) dP D = a D X dP + b D Y dP = D [ a X + b Y ] dP for all D ∈ D, as required.

(a)

To prove (15), simply note that (b) E Y = Ω Y dP = Ω E(Y |D) dP = E [ E(Y |D) ].

164 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS For (16), use (14) for the ﬁrst step of D {E(Y |D) − E(X|D)} dP = D E(Y − X|D) dP (c) is a measure ν(·). = D (Y − X) dP ≡ ν(D) ≥ 0 Then E(Y |D) − E(X|D) is a Radon–Nikodym derivative of ν(·), and is ≥ 0 a.s. Statement (17) follows easily from (16), since we have (d)

E(Yn |D) ≤ E(Yn+1 |D) ≤ E(Y |D)

a.s., for all n.

Thus limn E(Yn |D) exists a.s., and (e) D lim E(Yn |D) dP = lim D E(Yn |D) dP n

n

= lim n

=

D

D

Yn dP =

D

Y dP

by the MCT

by the MCT

E(Y |D) dP ;

and we can appeal to the uniqueness of Radon–Nikodym derivatives, or apply exercise 3.2.2. Now we use (16) and (17) to prove (18). Thus E( lim Yn |D) = E( lim inf Yk |D) n k≥n

(f)

by the deﬁnition of lim

= lim E( inf Yk |D)

a.s., by the MCT of (17)

≤ lim inf E(Yk |D)

by the monotonicity of (16)

n

k≥n

n k≥n

= lim E(Yn |D)

by the deﬁnition of lim.

To prove (19), apply the Fatou of (18) to Yn + X to get E(Y |D) + E(X|D) = E(X + Y |D) = E( lim (X + Yn )|D) (g)

≤ lim E(X + Yn |D) = lim E(Yn |D) + E(X|D).

Canceling the a.e. ﬁnite E(X|D) from both ends of (g) gives E(Y |D) ≤ lim E(Yn |D) ≤ lim E(Yn |D); (h)

≤ E(Y |D)

by applying the Fatou of (18) again, to X − Yn .

To prove (20) we proceed through indicator, simple, nonnegative, and then general functions, and each time we apply exercise 3.2.2 at the ﬁnal step. Case 1: Y = 1D∗ . Then Y E(X|D) dP = D∩D∗ E(X|D) dP D = D∩D∗ X dP = D Y X dP = D E(Y X|D) dP.

4.

BASIC PROPERTIES OF CONDITIONAL EXPECTATION

165

n Case 2: Y = 1 ai 1Di . Then n 1 ai D 1Di E(X|D) dP D Y E(X|D) dP = n = 1 ai D 1Di X dP by case 1 = D Y X dP = D E(Y X |D) dP. Case 3: Y ≥ 0. Let simple functions Yn Y where Yn ≥ 0. Suppose ﬁrst that X ≥ 0. Then we have E(Y X|D) dP = D E(lim Yn X|D) dP D n

= = =

D

lim E(Yn X|D) dP

by the MCT of (17)

D

lim Yn E(X|D) dP

by case 2

D

Y E(X|D) dP

n

n

by the MCT.

For general X, use X = X + − X − and the linearity of (14). Case 4: General Y . Just write Y = Y + − Y − . To prove (21), simply note that 1D Y dP = E (1D ) E(Y ) D E(Y |D) dP = D Y dP = = P (D) E(Y ) = D E(Y ) dP. Assertion (22) is proved by noting that D E[ E(Y |E) |D] dP = D E[ Y |E] dP = D Y dP = D E(Y |D) dP.

since D ∈ D ⊂ E

The integrands of the two extreme terms must be equal a.s. by the exercise 3.2.2. Consider (23). Now, F(X1 , X2 ) = σ[D ≡ D1 ∩ D2 ≡ X1−1 (B1 ) ∩ X2−1 (B2 ) : B1 , B2 ∈ B] . Then E(Y |X1 , X2 ) dP = D Y dP = 1D1 1D2 Y dP = P (D2 ) D1 Y dP = P (D2 ) D1 E(Y |X1 ) dP = D1 1D2 E(Y |X1 ) dP

ν1 (D) ≡

=

D

D

E(Y |X1 ) dP ≡ ν2 (D).

Since these two measures ν1 and ν2 agree on all sets in the π ¯ -system consisting of all sets of the form D = D1 ∩ D2 , they agree on the σ-ﬁeld F(X1 , X2 ). Thus the integrands satisfy E(Y |X1 , X2 ) = E(Y |X1 ) a.s.

166 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS We next prove (25), leaving most of (24) and (26) to the exercises. We have E |E(Yn |D) − E(Y |D)|r = E |E(Yn − Y |D)|r (i)

≤ E[ E(|Yn − Y |r |D) ] = E|Yn − Y |r

by the conditional Jensen inequality of (24)

by (15)

→ 0. To prove (27), note that for all D ∈ D we have E(Y ) if P (D) = 1 (j) E(Y |D) dP = Y dP = = D E(Y ) dP. D D 0 if P (D) = 0 We now turn to part of the Jensen inequality of (24). N Case 1: Suppose that X is the simple function deﬁned by X = 1 ai 1Ai . Since N N g( 1 ai 1Ai ) = 1 g(ai ) 1Ai , we have a.s. that N N E [ g(X)|D] = E[ g( 1 ai 1Ai )|D] = E[ 1 g(ai ) 1Ai |D] N

g(ai ) E(1Ai |D)

(k)

=

(l)

≥ g(

(m)

= g( E(

1

N 1

where

ai E(1Ai |D))

N 1

N 1

E(1Ai |D) = E(

N 1

1Ai |D) = 1 a.s.

by convexity, and the usual Jensen

ai 1Ai |D)) = g( E(X|D))

by the linearity of (14).

Case 2: General X. Associate with X the rv n2n k−1 (n) 1[ k−1 . Xn ≡ − 1[ k−1 k k 2n ≤X< 2n ] 2n ≤−X< 2n ] 2n k=1

We consider only the case that g(0) = 0 with g for x > 0 and g for x < 0, and leave other cases to the following exercise. Note that (o)

0 ≤ g(Xn ) ≤ g(X) ∈ L1

with

g(Xn ) → g(X) a.s.

by the continuity of g (that follows from convexity), and that (p)

|Xn | ≤ |X| ∈ L1

with

Xn → X a.s.

Thus (q)

E[ g(X)|D] ← E[ g(Xn )|D]

a.s., by (o) and the DCT of (19)

(r)

≥ g( E(Xn |D))

by (m)

(s)

→ g( E(X|D))

by (p), the DCT of (19), and continuity of g.

The proofs of the remaining parts are left to exercise 4.4.

2

4.

BASIC PROPERTIES OF CONDITIONAL EXPECTATION

Exercise 4.4

Prove (24) and (26) of theorem 4.1.

Exercise 4.5 (Dispersion inequality) rvs with µY = 0. Show that (28)

167

Suppose that X and Y are independent

E|X| ≤ E|X + Y |.

[Hint. Apply E(·|X) to X + Y via Jensen’s inequality, with g(·) = | · | .] Note At this point one can go directly to the next chapter.

168 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS

5

Regular Conditional Probability

For ﬁxed A, the function P (A|D)(·) is a D-measurable function on Ω that is only a.s. unique. We wish that for each ﬁxed ω the set function P (·|D)(ω) were a probability measure. But for each disjoint sequence of Ai ’s there is a null set where (8.4.10)– (8.4.12) may fail, and there typically are uncountably many such sets. The union of all such null sets need not be null. In the most important special cases, though, we may assume that P (·|D)(ω) behaves as we would like, where the nonuniqueness of P (A|D)(·) also provides the key, by allowing us to make whatever small changes are required. [For added useful generality, we will work on a sub σ-ﬁeld A˜ of the basic σ-ﬁeld A.] Deﬁnition 5.1 We will call P (A|D)(ω) a regular conditional probability on a sub σ-ﬁeld A˜ of the σ-ﬁeld A, given the σ-ﬁeld D, if (1)

˜ the function P (A|D)(·) of ω satisﬁes deﬁnition 4.2, for each ﬁxed A ∈ A,

(2)

˜ for each ﬁxed ω, P ω (·|D) ≡ P (·|D)(ω) is a probability measure on A.

Exercise 5.1 Verify that the discrete conditional probability of exercise 8.4.1 is a regular conditional probability. When a regular conditional probability exists, conditional expectation can be computed by integrating with respect to conditional probability, and we ﬁrst show this general theorem 5.1. In theorem 5.2 and beyond we shall show speciﬁcally how to construct such conditional probabilities in some of the most important examples. ˜ and let Theorem 5.1 Let P (A|D)(ω) be a regular conditional probability on A, ˜ Y ∈ L1 (Ω, A, P ). Then a version of the conditional expectation of Y given D is formed by setting (3) for each ﬁxed ω. E{Y |D}(ω) = Y (ω ) dP ω (ω |D) , Proof. (a) (b)

If Y = 1A , then (3) follows from Y dP ω = 1A (ω ) dP ω (ω |D) = A dP ω (ω |D) = P ω (A|D) = P (A|D)(ω) =a.s. E{Y |D}(ω),

no matter which version of the latter is used,

with the various steps true by deﬁnition. Thus (3) is trivial for simple functions Y . If Y ≥ 0 and Yn are simple functions for which Yn Y , then for any version of E(Y |D)(·) we have Y (ω ) dP ω (ω |D) = lim Yn (ω ) dP ω (ω |D) (c)

=a.s. lim E{Yn |D}(ω) =a.s. E{Y |D}(ω)

using the MCT (ordinary, and of (17)) in the ﬁrst and last steps. Finally, let Y = Y + − Y −. 2

5.

REGULAR CONDITIONAL PROBABILITY

169

Regular conditional probabilities need not exist; the null sets on which things fail may have a nonnull union. However, if Y : (Ω, A) → (R, B) is a rv, then things necessarily work out on (R, B), and this will be generalized to any “Borel space”. We will now start from scratch with regular conditional probability, and will choose to regard it as a measure over the image σ-ﬁeld. Deﬁnition 5.2 (Borel space) If Z : (Ω, A) → (M, S) and if a 1-to-1 bimeasurable mapping φ from (M, S) to a measurable subset E of (R, B) exists, then (M, S) is called a Borel space. Exercise 5.2 (a) Show that (Rn , Bn ) is a Borel space. (b) Show that (R∞ , B∞ ) is a Borel space. (c) The spaces (C, C) and (D, D) to be encountered below are also Borel spaces. (d) Let (M, d) be a complete and separable metric space having Borel sets M, and let M0 ∈ M. Then (M0 , M0 ∩ M) is a Borel space. [This exercise is the only mathematically diﬃcult part of the chapter that we have encountered so far.] Deﬁnition 5.3 (Regular conditional distribution) Suppose that Z : (Ω, A) → (M, S). Let A˜ ≡ Z −1 (S), and let D be a sub σ-ﬁeld of A. Then PZ (S|D)(ω) will be called a regular conditional distribution for Z given D if (4)

for each ﬁxed S ∈ S, the function PZ (S|D)(·) is a version of P (Z ∈ S|D)(·) on Ω,

(5)

for each ﬁxed ω ∈ Ω, the set function PZ (·|D)(ω) is a probability distribution on (M, S);

and PZω (·|D) ≡ PZ (·|D)(ω) will denote this probability distribution. Theorem 5.2 (Existence of a regular conditional distribution) Suppose Z : (Ω, A) → (M, S) with (M, S) a Borel space. Then the existence of a regular conditional probability distribution PZ (S|D)(ω) is guaranteed. Proof. Case 1: Suppose ﬁrst that Z : (Ω, A) → (R, B). Let r1 , r2 , . . . denote the set of rational numbers. Consider P (Z ≤ ri |D) and note that except on a null set N , all of the following hold: (a) (b) (c) (d)

ri ≤ rj

implies P (Z ≤ ri |D) ≤ P (Z ≤ rj |D).

lim P (Z ≤ rj |D) = P (Z ≤ ri |D).

rj ri

lim P (Z ≤ rj |D) = P (Ω|D) = 1.

rj ∞

lim P (Z ≤ rj |D) = P (∅|D) = 0.

rj −∞

170 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS Now deﬁne, for an arbitrary but ﬁxed df F0 , limrj z P (Z ≤ rj |D)(ω) (e) F (z|D)(ω) = F0 (z)

if ω ∈ N, if ω ∈ N.

Then for every ω, the function F (·|D)(ω) is a df. Also, (e) and the DCT of theorem 8.4.1 show that F (z|D)(·) is a version of P (Z ≤ z|D)(·). Now extend P (Z ≤ ·|D)(ω) to a distribution [labeled PZ (B|D)(ω)] over all B ∈ B. We now deﬁne M ≡ {C ∈ B : PZ (C|D) is a version of P (Z ∈ C|D)}. Now, M contains all (a, b] and all m 1 (ai , bi ], and M is closed under monotone limits. Thus M = B, by the minimal monotone class result of proposition 1.1.6, completing the proof in this case. Case 2: Let Y ≡ φ(Z), so Y is a rv. Thus a regular conditional distribution PY (B|D) exists by case 1. Then for S ∈ S, deﬁne PZ (S|D) ≡ PY (φ(S)|D). 2 (f)

Example 5.1 (Elementary conditional densities) Suppose that P ((X, Y ) ∈ B2 ) = f (x, y) dx dy for all B2 ∈ B2 , B2

where f ≥ 0 is measurable; and then f (x, y) is called the joint density of X, Y . Letting B2 ≡ B × R, we can conclude that PX λ ≡ (Lebesgue measure), with dPX (x) (6) = fX (x) ≡ f (x, y) dy ; dλ R we call fX (x) the marginal density of X. We ﬁrst deﬁne f (x, y)/fX (x) for all x with fX (x) = 0, (7) g(y|x) ≡ an arbitrary density f0 (y) for all x with fX (x) = 0, and then deﬁne (8)

P (Y ∈ A|X = x) =

g(y|x) dy

for A ∈ B.

A

Call g(y|x) the conditional density of Y given X = x, and this P (Y ∈ A|X = x) will now be shown to be a regular conditional distribution (if modiﬁed appropriately on the set where fX (x) = 0). Moreover, if E|h(Y )| < ∞, then ∞ (9) E{h(Y )|X = x} = h(y) g(y|x) dy a.s. PX . −∞

Thus (8) (more speciﬁcally, (10)) fulﬁlls theorem 5.2, and (9) will be seen to fulﬁll theorem 5.3. (Note that this example also holds for vectors x ∈ Rm and y ∈ Rn .) Proof. (a)

By Fubini’s theorem, P (X ∈ B) = B×R f (x, y) dx × dy = B [ R f (x, y) dy] dx = B fX (x) dx.

Moreover, Fubini’s theorem tells us that fX (x) is B-measurable. Let S ≡ {x : fX (x) = 0}. We may assume that (Ω, A) = (R2 , B2 ). Let A˜ = −1 Y (B). We will verify that (8.4.6) holds. For A ∈ B and for [Y ∈ A] ∈ A˜ ≡ R × B, we note that for all B ∈ B we have (writing f (x) for fX (x))

5.

REGULAR CONDITIONAL PROBABILITY B

[

= =

(b)

A

g(y|x) dy] dPX (x)

B

[

A

B∩S

[

g(y|x) dy]f (x) dx =

A

f (x, y) dy] dx =

B∩S

B×A

A

171

f (x, y) dy f (x) dx f (x)

f (x, y) (dx × dy)

= P ((X, Y ) ∈ B × A) = P ([Y ∈ A] ∩ X−1 (B)) = B P (Y ∈ A|X = x) dPX (x).

Thus (c)

P (Y ∈ A|X = x) =

A

g(y | x) dy

a.s. PX ,

and so (c) works as a version. Now, for any ﬁxed set A ∈ B we note that c (10) A g(y|x) dy = 1S (x) × [ A f (x, y) dy / f (x)] + 1S (x) × A f0 (y) dy is a measurable function on (R, B). It is clear that for each ﬁxed x the function of (10) acts like a probability distribution. Thus (10) deﬁnes completely a regular conditional probability distribution. Suppose that E|h(Y )| < ∞. Then [ R h(y)g(y|x) dy] dPX (x) = B∩S [ R h(y)g(y|x) dy]f (x) dx B = = (d)

=

B∩S

[

R

h(y)f (x, y) dy] dx =

B×R h(y)f (x, y) (dx

X −1 (B)

so that (9) holds.

B

[

R

h(y)f (x, y) dy] dx

× dy)

h(y) dPX,Y (x, y) =

B

E(h(Y )|X = x) dPX (x),

2

Theorem 5.3 (Conditional expectation exists as an expectation) Given a random element Z : (Ω, A) → (M, S), where (M, S) is a Borel space, consider a transformation φ : (M, S) → (R, B) with E|φ(Z)| < ∞. Then a version of the conditional expectation of φ(Z) given D is formed by setting E{φ(Z)|D}(ω) = (11) φ(z) dPZ (z|D)(ω) for all ω. M

Exercise 5.3

Prove theorem 5.3. (Note how (9) relates to (8), for comparison.)

Exercise 5.4 Let ξ1 and ξ2 be independent Uniform(0, 1) rvs. Let Θ ≡ 2πξ1 and Y ≡ − log ξ2 . Let R ≡ (2Y )1/2 . Now let Z1 ≡ R cos Θ and Z2 ≡ R sin Θ. Determine the joint distribution of (Y, Θ) and of (Z1 , Z2 ).

172 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS Example 5.2 (Suﬃciency of the order statistics) Let X1 , . . . , Xn be iid with df F in the class Fc of all continuous dfs. Let T (x) ≡ (xn:1 , . . . , xn:n ) denote the vector of ordered values of x, and let T ≡ {x : xn:1 < · · · < xn:n }. Exercise 5.5 ∈ T ) = 1 for all F ∈ Fc . Let X below asks the reader to verify that PF (T (X) denote those x ∈ Rn having distinct coordinates. Let A and B denote all Borel subsets of X and T , respectively. Then D ≡ T −1 (B) denotes all symmetric subsets of A (in that x ∈ D ⊂ D implies π(x ) ∈ D for all n! permutations π(x ) of x ). (n) Let dPF (·) denote the n-fold product measure dF (·) × · · · × dF (·). Suppose φ(·) (n) is PF -integrable. Then deﬁne 1 φ0 (x) ≡ n! (12) all n! permutations φ(π(x )), (n)

which is a D-measurable function. Since PF is symmetric, for any symmetric set D ∈ D we have (n) (n) x ) dPF (x ) = D φ(π(x )) dPF (x ) for all n! permutations π(·) D φ( (13)

=

(n)

D

φ0 (x ) dPF (x )

for every D ∈ D.

But this means that (14)

EF {φ(x )|T }(x ) = φ0 (x )

(n)

a.s. PF (·) . (n)

Now, for any A ∈ A, the function 1A (·) is PF -integrable for all F ∈ Fc . (Thus conclusion (14) can be applied.) Fix F ∈ Fc . For any ﬁxed A ∈ A we have (n)

PF (A|T )(x ) = EF {1A (x )|T }(x ) = (15)

=

1 1A (π(x )) n! all π

(the # of times π(x ) is in A) . n!

Note that the right-hand side of (15) does not depend on the particular F ∈ Fc , and so T is said to be a suﬃcient statistic for the family of distributions Fc . (Note discussion 4.2 once again.) 2 n = (Rn1 , . . . , Rnn ) and the Example 5.3 (Ranks) Consider the ranks R n = (Dn1 , . . . , Dnn ) in a sample from some F ∈ Fc (see the previous antiranks D example, and (7.5.17)). Let n denote the set of all n! permutations of (1, . . . , n). n takes on values in when F ∈ Fc (since ties occur with probability 0). Now, R n By symmetry, for every F ∈ F we have (n) (16) PF (R r ) = 1/n! for each r ∈ n . n = is equivalent to (T, R n ), in the sense that each determines the other. Note that X Note also that (17)

n are independent rvs (for each ﬁxed F ∈ F), T and R

5.

REGULAR CONDITIONAL PROBABILITY

in that for all B ∈ B and all π ∈

n

173

we have

(n) n = r ] ) PF ([T ∈ B] and [R

= (18)

1 n!

×

(n)

T −1 (B)

(n)

(n)

n = r ) × P (T ∈ B) n! dPF (x ) = PF (R F (n)

n = r ) × P (T ∈ B), = P (R F

n = r ) does not depend on the particular F ∈ Fc . Since the ranks are since P (R n independent of the suﬃcient statistic,they are called ancillary rvs. (Note that D is also equally likely distributed over n , and that it is distributed independent of the order statistics T .) 2 Exercise 5.5 Suppose that the n observations are sampled from a continuous distribution F . Verify that with probability one all observations are distinct. [Hint. Use corollary 2 to theorem 5.1.3.] Exercise 5.6 Suppose X1 , . . . , Xn are iid Bernoulli(p) rvs, for some p ∈ (0, 1). n Let T ≡ 1 Xk denote the total number of successes. Show that this rv T is suﬃcient for this family of probability distributions (that is, T is “suﬃcient for p”).

174 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS

6

Conditional Expectations as Projections

In this section we develop the view of conditional expectations as projections or projection operators in the Hilbert space L2 (P ). Then we prove the alternating projection theorem of von Neumann, and indicate extensions and applications to statistics.

Conditional Expectation as a Projection Operator Consider the collection of rvs (1)

H ≡ L2 (P ) ≡ {X on (Ω, A, P ) : EX 2 < ∞}.

Recall that H is a Hilbert space (that is, it is a complete inner product space with inner product X, Y ≡ E(XY )). Let D be a sub σ-ﬁeld of A, and let HD be the subspace of all D-measurable elements of L2 (P ). That is, (2)

HD ≡ {X ∈ L2 (P ) : F(X) ⊂ D}.

Deﬁne a linear operator PD on L2 (P ) by (3)

PD Y ≡ E(Y |D)

Exercise 6.1 rvs Z ∈ HD .

for each Y ∈ L2 (P ).

Setting Z ≡ E(Y |D) minimizes E(Y −Z)2 among all D-measurable

Proposition 6.1

PD has the following properties. (so PD is the projection of H onto HD ).

(a)

PD is idempotent: PD2 = PD

(b)

⊥ I − PD is the projection onto HD .

(c)

PD is self-adjoint: PDT = PD ,

meaning that for all Y in H we have PD Y, Z = Y, PD Z for all Z ∈ H. Moreover, (d)

PD Y ≤ Y ,

and PD = 1 if dim(HD ) ≥ 1,

where PD ≡ sup[Y ∈H with

Y =0] PD Y

/Y ).

Proof. Linearity of PD follows from (8.4.14). That PD is idempotent follows from PD2 Y = E(E(Y |D)|D) = E(Y |D) (p)

by the stepwise smoothing of (8.4.22)

= PD Y.

Then I − PD is orthogonal to HD , since for any Z ∈ HD we have (I − PD )Y, Z = Y − PD Y, Z = E((Y − E(Y |D))Z) = E(Y Z) − E(E(Y Z|D)) (q)

= E(Y Z) − E(Y Z) = 0.

by (8.4.15)

by (8.4.20)

6.

CONDITIONAL EXPECTATIONS AS PROJECTIONS

175

To see that PD is self-adjoint, let Z ∈ HD . Then (r)

PD Y, Z = E(Z E(Y |D)) = E(E(ZY |D)) = E(Z Y )

(s)

by (8.4.20)

by (8.4.15)

= E(Y E(Z|D))

by (8.4.20)

= Y, PD Z. By Jensen’s inequality (8.4.24) for conditional expectations, it follows that (t)

PD Y 2 = E(E(Y |D)2 ) ≤ E(E(Y 2 |D)) = E(Y 2 ) = Y 2 ;

this proves the ﬁrst part of (d), and shows that PD is a bounded operator (that is, PD ≤ 1). But for any Y = 0 that is D-measurable (8.4.20) gives (u)

PD Y 2 = E(E(Y |D)2 ) = E(Y 2 ) = Y 2 > 0,

so that PD = 1.

2

Projection on a Sum Space by Alternating Projections: ACE Now let X1 , X2 , . . . be rvs deﬁned on (Ω, A, P ), and consider the corresponding subspaces and projection operators (4)

H1 ≡ HF (X1 ) ,

P1 Y ≡ E(Y |X1 ),

(5)

H2 ≡ HF (X2 ) ,

P2 Y ≡ E(Y |X2 ),

for Y ∈ H ≡ L2 (P ). We often want to project onto the sum subspace (6)

H+ ≡ H1 + H2 ≡ {g1 (X1 ) + g2 (X2 ) : gi is measurable, Egi2 (Xi ) < ∞}.

Let P+ ≡ P H1 +H2 denote the projection onto H+ . There is one easy case in which P+ can be written down explicitly. Proposition 6.2 If F(X1 ) and F(X2 ) are independent (or equivalently, if the rvs X1 and X2 are independent, or equivalently, if H1 and H2 are orthogonal), then (7) Proof.

P+ Y = E(Y |X1 ) + E(Y |X2 ) ≡ P1 Y + P2 Y

for all Y ∈ H ≡ L2 (P ).

Let Z = g1 (X1 ) + g2 (X2 ) ∈ H+ . Then Y − E(Y |X1 ) − E(Y |X2 ), Z = Y − E(Y |X1 ) − E(Y |X2 ), g1 (X1 ) + g2 (X2 ) = Y − E(Y |X1 ), g1 (X1 ) − E(Y |X2 ), g1 (X1 ) +Y − E(Y |X2 ), g2 (X2 ) − E(Y |X1 ), g2 (X2 ) =0−0+0−0=0

by proposition 6.1(d), independence of X1 and X2 , and Egi (Xi ) = 0.

2

176 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS If X1 , X2 are not independent (i.e., H1 and H2 are not orthogonal), then P+ can be obtained iteratively as follows. Let Q+ ≡ I − P+ denote the projection onto ⊥ H+ , and let Q1 ≡ I − P1 , Q2 ≡ I − P2 denote the projections onto M1 ≡ H1⊥ and M2 ≡ H2⊥ , respectively. Since (H1⊥ ∩ H2⊥ )⊥ = (H1 + H2 ) by the following exercise, it follows immediately that Q+ = PM1 ∩M2 = PH⊥ ⊥. 1 ∩H2 Exercise 6.2

Show that (H1⊥ ∩ H2⊥ )⊥ = (H1 + H2 ), as claimed above.

Consider the two sequences of operators: (8)

σ1 : Q1 , Q2 Q1 , Q1 Q2 Q1 , Q2 Q1 Q2 Q1 , . . . ,

(9)

σ2 : Q2 , Q1 Q2 , Q2 Q1 Q2 , Q1 Q2 Q1 Q2 , . . . .

Theorem 6.1 (von Neumann) The two sequences of operators σ1 and σ2 both converge to the projection operator Q+ = PM1 ∩M2 . So if {An } denotes either of the sequences σ1 or σ2 , we have (10)

An Y →L2 Q+ Y = PM1 ∩M2 Y

as n → ∞

for each Y ∈ H = L2 (P ). Thus, in particular, (11)

(Q1 Q2 )n Y →L2 Q+ Y

as n → ∞,

(12)

(Q2 Q1 )n Y →L2 Q+ Y

as n → ∞.

Corollary 1 We have (13)

{I − [(I − P1 )(I − P2 )]n }Y →L2 P+ Y = PH1 +H2 Y.

This is the “inner loop” of the (population version of) the alternating conditional expectation algorithm (or ACE algorithm) of Breiman and Friedman (1985). Proof. (a)

Let An be the nth operator in the sequence σ1 . Then for Y, Z ∈ H Am Y, An Z = An+m− Y, Z,

where

1 if m + n is even, 0 if m + n is odd. For example, proposition 6.1(b) gives

=

A4 Y, A3 Z = Q2 Q1 Q2 Q1 Y, Q1 Q2 Q1 Z = Q1 Q2 Q1 Q2 Q1 Q2 Q1 Y, Z = A7 Y, Z, while A2 Y, A4 Z = Q2 Q1 Y, Q2 Q1 Q2 Q1 Z = Q1 Q2 Q1 Q22 Q1 Y, Z = Q1 Q2 Q1 Q2 Q1 Y, Z = A5 Y, Z.

6.

CONDITIONAL EXPECTATIONS AS PROJECTIONS

177

We want to show that limn→∞ An Y exists for any Y ∈ H. But for some k we have Am Y − An Y 2 = Am Y − An Y, Am Y − An Y = Am Y, Am Y − Am Y, An Y − An Y, Am Y + An Y, An Y = A2m−1,Y + A2n−1 Y, Y − 2Am+n− Y, Y (b)

= A2m−1 , Y + A2n−1 Y, Y − 2A2k−1 Y, Y ,

since m + n − is always odd. By (b), it suﬃces to show that limk→∞ A2k−1 Y, Y exists. Now, A2k−1 Y, Y = Ak Y, Ak Y = Ak Y 2 ≥ 0, and similarly, A2k+1 Y, Y = Ak+1 Y 2 ≥ 0. But Ak+1 Y could equal either Q1 Ak Y or Q2 Ak Y , so Ak+1 Y ≤ Ak Y , since QY ≤ Y by proposition 6.1 (c). Hence A2k−1 Y, Y ≥ A2k+1 Y, Y , or A1 Y, Y ≥ A3 Y, Y ≥ · · · ≥ 0. Thus (c)

lim A2k−1 Y, Y exists;

k→∞

then (b) and the completeness of H give that limn An Y exists. Call this limit Y ∗ . Deﬁne AY ≡ Y ∗ . Clearly, A is a linear operator on H, and it remains only to show that A = Q+ . We have shown that for any Y, Z in H we have AY, AZ = = (d)

lim Am Y, An Z m∧n→∞

lim Am+n− Y, Z m∧n→∞

= AY, Z.

Letting Y = Z in (d) shows that Z = AZ + (Z − AZ), where AZ ⊥ (Z − AZ). It follows that A is of the form of a projection operator A = PM , for some M. If Y is in M1 ∩M2 , then Q1 Y = Q2 Y = Y , An Y = Y , and hence AY = Y . Thus Y ∈ M by theorem 6.3.1, and it follows that M1 ∩ M2 ⊂ M. Since Q1 A2i = Q2i+1 and Q2 A2i−1 = A2i , it follows by the continuity of Q1 and Q2 (implied by (8.4.24)) that Q1 A = A and Q2 A = A. Letting Z ∈ H and setting Y ≡ AZ yields (e)

Q1 Y = Q1 AZ = AZ = Y

and

Q2 Y = Y,

so that range(A) = M ⊂ M1 ∩ M2 . Hence M = M1 ∩ M2 and A = PM1 ∩M2 = Q+ . By interchanging Q1 and Q2 , the same argument shows that σ2 has limit A∗ = PM1 ∩M2 = Q+ . 2

178 CHAPTER 8. INDEPENDENCE AND CONDITIONAL DISTRIBUTIONS

Extensions We now state a generalization of theorem 6.1 to k > 2 subspaces H1 , . . . , Hk . Let us suppose that X1 , . . . , Xk are rvs on (Ω, A, P ) and let (14)

Hi ≡ HF (Xi ) ,

Pi Y ≡ E(Y | Xi )

for i = 1, . . . , k

for Y ∈ H ≡ L2 (P ). Deﬁne (15)

H+ ≡ H1 + · · · + Hk .

As when k = 2, let P+ denote the projection onto H+ , Q+ ≡ I−P+ , and Qi ≡ I−Pi , i = 1, . . . , k. Exercise 6.3 (Halperin) (16)

Deﬁne

Q = Q1 , . . . , Qk .

Then (17)

Qn Y →L2 Q+ Y = PH⊥ ⊥ Y. 1 ∩···∩Hk

Other generalizations and extensions of theorem 6.1 are known, including rates of convergence. We are simply stating some known results in this section as an “exercise”. Exercise 6.4 (Rota–Burkholder) Y ∈ Lr (P ) for some r > 1, then (18)

(Q1 Q2 )n Y →Lr

and a.s

Let Q1 and Q2 be as in theorem 6.1. (i) If

(some rv in Lr (P )).

(ii) If E{|Y | log+ |Y |} < ∞, then (19)

(Q1 Q2 )n Y →L1

and a.s

(some rv in L1 (P )).

Conversely, if convergence holds in (19) for all choices of the projections Q1 and Q2 , then we have E{|Y | log+ |Y |} < ∞. The alternating methods developed here have many applications in statistics. One important application is to the additive regression models.

Chapter 9

Special Distributions 1

Elementary Probability

Independent Bernoulli Trials If P (X = 1) = p = 1 − P (X = 0), then X is said to be a Bernoulli(p) rv. We refer to the event [X = 1] as “success”, and [X = 0] as “failure”. Let X1 , . . . , Xn be iid Bernoulli(p), and let Tn ≡ X1 + · · · + Xn denote the number of successes in n independent Bernoulli(p) trials. Now, P (Xi = xi for 1 ≤ i ≤ n) = p

n 1

xi

n

(1 − p)n−

1

xi

if all xi equal 0 or 1;

this formula gives the joint distribution of X1 , . . . , Xn . From this we obtain n k (1) P (Tn = k) = p (1 − p)n−k for 0 ≤ k ≤ n, k since each of the nk diﬀerent possibilities that place k of the 1’s in speciﬁc positions in an n-vector containing k outcomes 1 and n − k outcomes 0 has probability pk (1−p)n−k , from the earlier display. We denote this by writing Tn ∼ = Binomial(n, p) when (1) holds. Note that Binomial(1, p) is the same as Bernoulli(p). Let X1 , X2 , . . . be iid Bernoulli(p). Let Y1 ≡ W1 ≡ min{n : Tn = 1}. Since we can rewrite the event [Y1 = k] = [X1 = · · · = Xk−1 = 0, Xk = 1], we have (2)

P (Y1 = k) = (1 − p)k−1 p

for k = 1, 2, . . . .

We write Y1 ∼ = GeometricT(p). Now let Wm ≡ min{n : Tn = m}. We call Wm the waiting time to the mth success; Wm counts the number of turns until the mth success. We let Ym ≡ Wm − Wm−1 for m ≥ 1, with W0 ≡ 0, and we call the Ym ’s the interarrival times. Note that [Wm = k] = [Tk−1 = m − 1 and Xk = 1]. Hence k−1 m (3) P (Wm = k) = p (1 − p)k−m for k = m, m + 1, . . . . m−1 We write Wm ∼ = Negative Binomial Turns(m, p) ≡ NegBiT(m, p). [We agree that NegBiF(m, p) denotes the distribution of Wm − m, and that this “F” connotes “failures”; the rv Wm − m counts the number of failures prior to the mth success.] 179

180

CHAPTER 9. SPECIAL DISTRIBUTIONS

Exercise 1.1

Explain why Y1 , Y2 , . . . are iid GeometricT(p).

Since the number of successes in the ﬁrst n1 +n2 trials is the same as the number of successes in the ﬁrst n1 trials plus the number of successes in the next n2 trials, it is clear that (4)

T1 +T2 ∼ = Binomial(n1 +n2 , p)

for independent rvs Ti ∼ = Binomial(ni , p).

Likewise, waiting for m1 successes and then waiting for m2 more successes is the same as waiting for m1 + m2 successes in the ﬁrst place. Hence, (5)

W1 +W2 ∼ = NegBiT(m1 +m2 , p) for independent rvs Wi ∼ = NegBiT(mi , p).

Urn Models Suppose an urn contains N balls that are identical, except that M bear the number 1 and N − M bear the number 0. Thoroughly mix the balls in the urn. Draw one ball at random. Let X1 denote the number on the ball drawn. Then X1 ∼ = Bernoulli(p) with p ≡ M/N . Now replace the ball in the urn, thoroughly mix, and draw at random a second ball with number X2 . Continue the process. This is the sampling with replacement scheme. Then Tn ≡ X1 + · · · + Xn ∼ = Binomial(n, p), where p = M/N represents the probability of success in n independent Bernoulli(p) trials. Suppose now that the same scheme is repeated, except that the balls are not replaced. In this sampling without replacement scheme X1 , . . . , Xn are dependent Bernoulli(p) rvs with p = M/N . Also, M N −M (6)

P (Tn = k) =

k

Nn−k

,

provided that the value k is possible.

n

We write Tn ∼ = Hypergeometric(M, N − M ; n). Suppose now that sampling is done without replacement, but the N balls in the urn bear the numbers a1 , . . . , aN . Let X1 , . . . , Xn denote the numbers on the ﬁrst n balls drawn, and let the general ﬁnite Tn ≡ X1 + · · · + Xn . We call this N 2 ¯)2 /N sampling model. Call a¯ ≡ N 1 ai /N the population mean and σa ≡ 1 (ai − a 2 the population variance. Note that Xi ∼ ) for all 1 ≤ i ≤ n, since we now (¯ a , σ = a assume n ≤ N . From (7.3.4), we have N (7) 0 = Var[ 1 Xi ] = N Var[X1 ] + N (N − 1)Cov[X1 , X2 ], N with the 0 valid, since 1 Xi is a constant. Solving (7) yields (8)

Cov[X1 , X2 ] = −σa2 /(N − 1).

As in (7), and using (8), Var[Tn ] = nσa2 − n(n − 1)σa2 /(N − 1). Thus 1 2 n−1 (9) Var[Tn /n] = σa 1 − , n N −1 where [1 − (n − 1)/(N − 1)] is called the correction factor for ﬁnite population sampling.

1.

ELEMENTARY PROBABILITY

Exercise 1.2

181

Verify (8) and (9).

Exercise 1.3 Suppose that T1 ∼ = Binomial(m, p) and T2 ∼ = Binomial(n, p) are independent. Then the conditional distribution of T1 given that T1 + T2 = k is Hypergeometric(k, m + n − k; m).

The Poisson Process Suppose now that Xn1 , . . . , Xnn are iid Bernoulli(pn ), where npn → λ as n → ∞. Let Tn ≡ Xn1 + · · · + Xnn , so that Tn ∼ = Binomial(n, pn ). An easy calculation shows that (10)

P (Tn = k) → λk e−λ /k!

for k = 0, 1, . . . .

When P (T = k) = λk e−λ /k! for k = 0, 1, . . . , we write T ∼ = Poisson(λ). The above can be used to model the following Geiger counter experiment. A radioactive source with large half-life is placed near a Geiger counter. Let N(t) denote the number of particles registered by time t. We will say that {N(t) : t ≥ 0} is a Poisson process. [Do note that our treatment is purely informal.] Physical considerations lead us to believe that the increments N(t1 ), N(t1 , t2 ], . . . , N(tk−1 , tk ] should be independent rvs; here, the increment N(ti−1 , ti ] ≡ N(ti ) − N(ti−1 ) is the number of particle counts across the interval (ti−1 , ti ]. We say that N has independent increments. We now deﬁne (11)

ν ≡ E N(1) ≡ [the intensity of the process].

Let M denote the number of radioactive particles in our source, and let Xi equal 1 or 0 depending on whether or not this ith particle registers by time t = 1. It seems possible to assume that X1 , . . . , XM are iid Bernoulli. Since N(1) = X1 + · · · + XM has mean ν = E N(1) = M EX1 , this leads to N(1) ∼ = Binomial(M, ν/M ). By the ﬁrst paragraph of this section, N(1) is thus approximately a Poisson(ν) rv. We now alter our point of view slightly, and agree that we will use this approximation as our model. Thus N(1) is a Poisson(ν) rv. Because of the stationary and independent increments we thus have (12)

N(s, t] ≡ N(t) − N(s)∼ = Poisson(ν(t − s))

(13)

N has independent increments.

for all 0 ≤ s ≤ t,

Agree also that N(0) ≡ 0. (This is actually enough to rigorously specify a Poisson process.) Let Y1 ≡ W1 ≡ inf{t : N(t) = 1}. Since (14)

[Y1 > t] = [N(t) < 1] = [N(t) = 0],

we see that 1 − FY1 (t) = P (Y1 > t) = P (N(t) = 0) = e−νt by (12). Thus Y1 has df 1 − exp(−νt) for t ≥ 0 and density (15)

fY1 (t) = νe−νt

for t ≥ 0;

we write Y1 ∼ = Exponential(ν). Now let Wm ≡ inf{t : N (t) = m}; we call Wm the mth waiting time. We call Ym ≡ Wm − Wm − 1, m ≥ 1, the mth interarrival time.

182

CHAPTER 9. SPECIAL DISTRIBUTIONS

In light of the physical properties of our Geiger counter model, and using (13), it seems reasonable that (16)

Y1 , Y2 , . . . are iid Exponential(ν) rvs.

Our assumption of the previous sentence could be expressed as follows: (17)

Y1 and N1 (t) ≡ N(Y1 , Y1 + t] = N(Y1 + t) − N(Y1 ) are independent, N1 is again a Poisson process, with intensity ν.

We will call this the strong Markov property of the Poisson process. Additionally, (18)

[Wm > t] = [N(t) < m],

m−1 k −νt so that 1 − FWm (t) = P (Wm > t) = /k! ; the derivate of this k=0 (νt) e expression telescopes, and shows that Wm has density (19)

fWm (t) = ν m tm−1 e−νt /Γ(m)

for t ≥ 0.

We write Wm ∼ = Gamma(m, ν). Since waiting for m1 counts and then waiting for m2 more counts is the same as waiting for m1 + m2 counts in the ﬁrst place, (20)

Z1 + Z 2 ∼ = Gamma(m1 + m2 , ν)

for independent Zi ∼ = Gamma(mi , ν).

It is true that (19) is a density for any real number m > 0, and the property (20) still holds for all positive mi ’s. Exercise 1.4

Verify (10).

Exercise 1.5

Verify (19)

Exercise 1.6

Verify that (20) holds for arbitrary real mi > 0.

Exercise 1.7 If X ∼ = Poisson(ν1) and Y ∼ =Poisson(ν2), then the conditional distribution of X given that X + Y = n is Binomial(n, ν1 /(ν1 + ν2 )). Exercise 1.8 Use Kolmogorov’s extension theorem to show that a Poisson process N exists on (R[0,∞) , B[0,∞) ). Then apply the smoother realizations theorem 5.4.2.

Location and Scale If a > 0, then FaZ+b (x) = P (aZ + b ≤ x) = P (Z ≤ (x − b)/a) = FZ ((x − b)/a) holds for any FZ (·). Thus for any density fZ (·), the rv aZ + b has density (21)

faZ+b (x) = a1 fZ ( x−b a )

for − ∞ < x < ∞.

1.

ELEMENTARY PROBABILITY

183

Normal Distributions Suppose the rv Z has density (22)

fZ (x) =

√1 2π

2

exp(− x2 )

for − ∞ < x < ∞;

then Z is said to be a standard normal rv. So the rv X ≡ µ + σZ ∼ = (µ, σ2 ) has density (23)

√1 2π σ

2 exp(− 12 ( x−µ σ ) )

for − ∞ < x < ∞,

and we write X ∼ = Normal(µ, σ2 ), or just X ∼ = N (µ, σ2 ). Exercise 1.9 Show that the formula fZ (·) of (22) is a density. Then show that this density has mean 0 and variance 1. [Transform to polar coordinates to compute ( fZ (x) dx)2 = 1.] The importance of the normal distribution√derives from the following theorem. ¯ − µ)/σ ∼ Recall that if X1 , . . . , Xn are iid (µ, σ2 ), then n (X = (0, 1) for the sample ¯ average Xn ≡ (X1 + · · · + Xn )/n. This is only a statement about moments. But much more is true. The result we now state will be proved in chapter 11. We will use it in the meantime for motivational purposes. Theorem 1.1 (Classical CLT) √ ¯ n − µ) →d N (0, σ2 ) (24) n (X

If X1 , . . . , Xn are iid (µ, σ2 ), then as n → ∞.

Let σ > 0. Then the Zn below is asymptotically normal, in that √ ¯ n − µ)/σ →d N (0, 1) (25) Zn ≡ n (X as n → ∞. Suppose that Z is N (0, 1). Then (26)

√ √ √ √ FZ 2 (x) = P (Z 2 ≤ x) = P (− x ≤ Z ≤ x ) = FX ( x ) − FZ (− x );

thus Z 2 has density (27)

√ √ 1 fZ 2 (x) = √ [fZ ( x ) − fZ (− x )] 2 x

for x ≥ 0.

[Note that formula (27) is true for any density fZ (·).] Plugging into (27) for this Z shows that (28)

fZ 2 (x) = (2πx)−1/2 exp(−x/2)

for x ≥ 0;

this is called the Chisquare(1) distribution. Note that Chisquare(1) is the same as Gamma( 12 , 12 ). Thus (20) establishes that (29)

if X1 , . . . , Xm are iid N (0, 1), then

m i=1

1 where Chisquare(m) ≡ Gamma( m 2 , 2 ).

Xi2 ∼ = Chisquare(m),

184

CHAPTER 9. SPECIAL DISTRIBUTIONS

Uniform and Related Distributions We write X ∼ = Uniform(a, b) if (30)

fX (x) =

1 (b−a) 1[a,b] (x)

=

1 (b−a)

on [a, b].

By far the most important special case is Uniform(0, 1). A generalization of this is the Beta(c, d) family. We write X ∼ = Beta(c, d) if (31)

fX (x) =

1 β(c,d)

xc−1 (1 − x)d−1 1[0,1] (x) =

1 β(c,d)

xc−1 (1 − x)d−1

on [0, 1],

where β(c, d) ≡ Γ(c)Γ(d)/Γ(c + d). Suppose that ξ1 , . . . , ξn are iid Uniform(0, 1). Let 0 ≤ ξn:1 ≤ · · · ≤ ξn:n ≤ 1 denote the ordered values of the ξi ’s; we call the ξn:i ’s the uniform order statistics. It seems intuitive that ξn:i equals x if (i − 1) of the ξi ’s fall in [0, x), 1 of the ξi ’s is equal to x, and n − i of the ξi ’s fall in (x, 1]. There are n!/[(i − 1)!(n − i)!] such designations of the ξni ’s, and for each such designation the chance of the rv’s falling in the correct parts of [0, 1] is xi−1 (1 · dx)(1 − x)n−i . Thus (32)

fξn:i (x) =

n! xi−1 (1−x)n−i 1[0,1] (x), or ξn:i ∼ = Beta(i, n−i+1). (i − 1)!(n − 1)!

Exercise 1.10 Give a rigorous derivation of (32) by computing 1 − Fξn:i (x) and then diﬀerentiating it. Exercise 1.11 Chose a point at random on the surface of the unit sphere (with probability proportional to area). Let Θ denote the longitude and Φ denote the latitude (relative to some ﬁxed axes) of the point so chosen. Determine the joint density of Θ and Φ.

The Cauchy Distribution We say X ∼ = Cauchy(b, a) if (33)

fX (x) = 1/{aπ[1 + (x − b)2 /a2 ]}

on (−∞, ∞).

By far the most important special case is Cauchy(0, 1); we then say simply that X ∼ = Cauchy, and its density is given by 1/[π(1 + x2 )] on (−∞, ∞). Verify that E|X| = ∞. We will see below that if X1 , . . . , Xn are iid Cauchy, then the sample ¯ n ≡ (X1 + · · · + Xn )/n ∼ average X = Cauchy. These two facts make the Cauchy ideal for many counterexamples.

Double Exponential and Logistic Distributions We say X ∼ = Double Exponential(b, a) when (X − b)/a has density 12 exp(−|x|) on the line. We say X ∼ = Logistic(b, a) when (X − b)/a has density ex /(1 + ex )2 = −x/2 x/2 2 1/(e + e ) on the line. Exercise 1.12 Now, X ≡ F −1 (ξ) has df F by the inverse transformation. So, compute F −1 for the Logistic(0, 1) and the Double Exponential(0, 1) distributions.

1.

ELEMENTARY PROBABILITY

185

Rademacher Random Variables and Symmetrization Many problems become simpler if the problem is symmetrized. One way of accomplishing this is by the appropriate introduction of Rademacher rvs. We say that

is a Rademacher rv if P ( = 1) = P ( = −1) = 12 . Thus ∼ = 2 Bernoulli( 12 ) − 1. ∼ We say that X is a symmetric rv if X = −X. If X and X are iid, then s X ≡ (X − X ) ∼ = (X − X) = −(X − X ) = −X s ; hence X s is a symmetric rv. Exercise 1.13 If X is a symmetric rv independent of the Rademacher rv , then X∼ = X always holds.

The Multinomial Distribution Suppose that B1 + · · · + Bk = R for Borel sets Bi ∈ B; recall that we call this i ≡ (Xi1 , . . . , Xik ) a partition of R. Let Y1 , . . . , Yn be iid rvs on (Ω, A, P ). Let X ≡ (1B1 (Yi ), . . . , 1Bk (Yi )) for 1 ≤ i ≤ n, and set (34)

n n n n T ≡ (T1 , . . . , Tk ) = ( Xi1 , . . . , Xik ) = ( 1B1 (Yi ), . . . , 1Bk (Yi )). i=1

i=1

i=1

i=1

Note that X1j , . . . , Xnj are iid Bernoulli(pj ) with pj ≡ P (Yi ∈ Bj ), and thus Tj ∼ = Binomial(n, pj ) (marginally). Note that T1 , . . . , Tn are dependent rvs. The joint distribution of (T1 , . . . , Tn ) is called the Multinomial(n, p ) distribution. We now derive it. The number of ways to designate n1 of the Yi ’s to fall in B1 , . . . , Bk , and nk of the Yi ’s to fall in Bk is the multinomial coeﬃcient n! n (35) ≡ , where n1 + · · · + nk = n. n1 . . . nk n1 ! · · · nk ! k Each such designation occurs with probability 1 pi ni . Hence for each possible n, n (36) P (T = n) ≡ P (T1 = n1 , . . . , Tk = nk ) = pn1 1 · · · pnk k . n1 . . . nk It is now a trivial calculation that (37)

Cov[Xij , Xil ] = E1Bj (Yi )1Bl (Yi ) − E1Bj (Yi )E1Bl (Yi ) = −pj pl

if j = l.

Thus (38)

Cov[Tj , Tl ] = −n pj pl

for all j = l.

Thus (with Dp a diagonal matrix having each dii = pi )  (39)

     T1 p1 p1 (1 − p1 ) −p1 pk . . .  .   .      ∼     . . , n . . .   =     = ( p , n [Dp − p p ] ).       . . . . . Tk pk −pk p1 pk (1 − pk )

186

CHAPTER 9. SPECIAL DISTRIBUTIONS

Assorted Facts Stirling’s Formula For all n > 1 we have √ n! = ean nn+1/2 e−n 2π, (40) where 1/(12n + 1) < an < 1/(12n). Eulers’s Constant n (41) i=1 1/i − log n ↑ γ ≡ 0.577215664901533 . . . . Exercise 1.14

If

∞ 1

an < ∞, there exists a cn ↑ ∞ such that

∞ 1

cn an < ∞.

Elementary Conditional Probability One deﬁnes the conditional probability of the event A given that the event B has occurred via P (A|B) ≡ P (AB)/P (B) when P (B) = 0. One then calls A and B independent if P (A|B) = P (A), because the probability of A is then unaﬀected by whether or not B occurred. Thus both of the following statements hold: (42)

Deﬁnition: P (A|B) ≡ P (AB)/P (B) leads to Theorem: P (AB) = P (B)P (A|B).

(43)

Deﬁnition: Independence means P (A|B) = P (A) leads to Theorem: P (AB) = P (A)P (B) if A and B are independent.

The big advantage of computation of P (A|B) via the theorem of (42) is that one can often revisualize P (A|B) in the context of a much simpler problem. Thus the probability of drawing two Reds when drawing at random without replacement from an urn containing 6 Reds and 4 Whites is P (R1 R2 ) = P (R1 )P (R2 |R1 ) = (6/10) × (5/9), where we revisualized to an urn containing 5 Reds and 4 Whites to compute P (R2 |R1 ) = 5/9. [Had we used sampling with replacement, our answer would have been (6/10) × (6/10) via (43).] [In the next exercise, revisualization works superbly to trivialize the problem.] Exercise 1.15 Determine the probability that the “shooter” wins in the game of craps. [When trying to convert a “point” of “ten” (say), we can revisualize and say that on the turn on which the game ends the dice will be showing either one of the 3 tens or one of the 6 sevens, and the probability of this conversion is clearly 3/(3 + 6).]

2.

DISTRIBUTION THEORY FOR STATISTICS

187

2

Distribution Theory for Statistics

Convolution If X and Y are independent rvs on (Ω, A, P ), then FX+Y (z) = P (X + Y ≤ z) = x+y≤z dFX (x) dFY (y) =

∞ z−x −∞

(1)

∞

= −∞

−∞

dFY (y) dFX (x)

FY (z − x) dFX (x) ≡ FX ∗ FY (z)

is a formula, called the convolution formula, for FX+Y in terms of FX and FY (the symbol ∗ deﬁned here stands for “convolution”). In case Y has density fY with respect to Lebesgue measure, then so does X + Y . In fact, since z ∞ ∞ z −∞ −∞ fY (y − x) dFX (x) dy = −∞ [ −∞ fY (y − z) dy] dFX (x) =

∞

F −∞ Y

(z − x) dFX (x) = FX+Y (z),

we see that X + Y has a density given by ∞ (2) fX+Y (z) = fY (z − x) dFX (x). −∞

In case both X and Y have densities, we further note that ∞ (3) fX+Y (z) = fY (z − x)fX (x) dx ≡ fY ∗ fX (z). −∞

Exercise 2.1 Use (2) to show that for X and Y independent: (i) X ∼ = N (µ1 , σ12 ) and Y ∼ = N (µ2 , σ22 ) implies X + Y ∼ = N (µ1 + µ2 , σ12 + σ22 ). ∼ ∼ (ii) X = Cauchy(0, a1 ) and Y = Cauchy(0, a2 ) has X + Y ∼ = Cauchy(0, a1 + a2 ). (iii) X ∼ = Gamma(r1 , θ) and Y ∼ = Gamma(r2 , θ) has X + Y ∼ = Gamma(r1 + r2 , θ). Exercise 2.2 (i) If X1 , . . . , Xn are√iid N (0, 1), then the normed sample average necessarily satisﬁes (X1 + · · · + Xn )/ n ∼ = N (0, 1). (ii) If X1 , . . . , Xn are iid Cauchy(0, 1), then (X1 + · · · + Xn )/n ∼ = Cauchy(0, 1). If X and Y are independent rvs taking values in 0, 1, 2, . . ., then clearly (4)

P (X + Y = k) =

k

P (X = i)P (Y = k − i)

for k = 0, 1, 2, . . . .

i=0

Exercise 2.3

Use (3) to show that for X and Y independent:

X∼ = Poisson(λ1 ) and Y ∼ = Poisson(λ2 ) has X + Y ∼ = Poisson(λ1 + λ2 ) .

188

CHAPTER 9. SPECIAL DISTRIBUTIONS

A fundamental problem in probability theory is to determine constants bn and an > 0 for which iid rvs X1 , . . . , Xn , . . . satisfy (5)

(X1 + · · · + Xn − bn )/an →d G,

as n → ∞

for some nondegenerate df G. Exercise 2.2 gives us two examples of such convergence; each was derived via the convolution formula. Except in certain special cases, such as exercises 2.1 – 2.3, the various convolution formulas are too diﬃcult to deal with directly. For this reason we need to develop a more oblique, but ultimately more convenient, approach if we are to solve problems of the form (5). This is taken up in chapters 11, 13, 14, and 15.

Other Formulas Exercise 2.4 Suppose that X and Y are independent with P (Y ≥ 0) = 1. Show that products and quotients of these rvs satisfy ∞ FXY (z) ≡ P (XY ≤ z) = 0 FX (z/y) dFY (y) (6) for all z, (7)

FX/Y (z) ≡ P (X/Y ≤ z) =

∞ 0

FX (zy) dFY (y)

for all z.

If FX has a density fX , then changing the order of integration above shows that FXY and FX/Y have densities given by ∞ (8) fXY (z) = 0 y −1 fX (z/y) dFY (y) for all z, (9)

fX/Y (z) =

∞ 0

y fX (yz) dFY (y)

for all z.

∼ N (0, 1), U ∼ Exercise 2.5 Let Z = = χ2m , and V ∼ = χ2n be independent. (a) Establish these classically important results: (10)

Z ∼ = Student’s tm . U/m

(11)

U/m ∼ = Snedecor’s Fm,n . V /n

(12)

U ∼ = Beta(m/2, n/2) . U +V

Here (13)

Γ((m + n)/2) 1 ftm (x) ≡ √ 2 π m Γ(m/2) (1 + x /m)(m+n)/2

(14)

fFm,n (x) ≡

Γ((m + n)/2) (m/n)m/2 xm/2−1 Γ(m/2)Γ(n/2) (1 + mx/n)(m+n)/2

for − ∞ < x < ∞, for 0 < x < ∞.

(b) Compute the kth moment of each of these three distributions.

2.

DISTRIBUTION THEORY FOR STATISTICS

189

Exercise 2.6 (15)

If Y1 , . . . , Yn+1 are iid Exponential(θ), then (Y1 + · · · + Yi )/(Y1 + · · · + Yn+1 ) ∼ = Beta(i, n − i + 1).

Exercise 2.7 Let X1 , . . . , Xn be iid N(µ, σ2 ). √ ¯ ∼ (a) Show that Wn ≡ n (X n − µ)/ σ = N(0, 1). n 2 2 ¯ n )2 / σ 2 ∼ (b) Show that (n − 1)Sn / σ ≡ 1 (Xk − X = χ2n−1 . (c) Show that Wn and Sn2 are independent rvs. √ ¯ ∼ (d) Show that Tn ≡ n (X n − µ)/ Sn = Student’s tn−1 . √ [Hint. Let Γ ≡ |[ γij ]| be an orthogonal matrix with all γ1j = 1/ n. Now let ≡ Γ (X − µ 1 )/ σ. This yields iid N(0, 1) rvs Z1 , . . . , Zn , with Wn = Z1 ∼ Z = N(0, 1) n 2 ∼ 2 and (n − 1)S 2 = Z =χ . Apply exercise 2.5.] n

2

k

n−1

Statistical Conﬁdence Intervals Example 2.1 Suppose we model the performances of n independent repetitions X1 , . .√. , Xn of an experiment as iid N (µ, σ2 ) rvs. The previous exercise shows ¯ n − µ)/σ is a N (0, 1) rv independent of the sample variance estimator n (X that n 2 ¯ n )2 /(n − 1) of σ2 , and that Sn /σ ∼ Sn ≡ 1 (Xk − X = {χ2n−1 /(n − 1)}1/2. Thus √ ¯ n − µ]/Sn ∼ (16) Tn ≡ n [X = Tn−1 ≡ Student tn−1 . Specify tp/2 such that P (−tp/2 ≤ Tn−1 ≤ tp/2 ) = 1 − p; perhaps p = .05. Then with the “large” probability of 1 − p = .95 we have √ ¯ n − µ]/Sn ≤ tp/2 ) (17) 1 − p = P (−tp/2 ≤ Tn ≤ tp/2 ) = P (−tp/2 ≤ n [X √ √ ¯ n ≤ µ + tp/2 Sn / n ≤ X ¯n) (18) = P (µ − tp/2 Sn / n ≤ X √ √ ¯ n − tp/2 Sn / n ≤ µ ≤ X ¯ n + tp/2 Sn / n) . (19) = P (X That is: (20)

¯ n ± tp/2 Sn /√n The random interval X will contain the unknown value of µ an average of (1 − p) × 100% of the time.

So when we apply this to the data values√x1 , . . . , xn , we can have (1 − p) × 100% conﬁdence that the interval x ¯n ± tp/2 sn / n did enclose the true (but unknown) value of µ. We say that √ ¯ n ± tp/2 Sn / n provides a (1 − p) × 100% conﬁdence interval (21) X for the unknown mean µ. Or we say that √ (22) x¯n ± tp/2 sn / n provides a (1 − p) × 100% numerical conﬁdence interval for the unknown mean µ. There is a probability of 1−p (or a (1−p)×100% chance) that the former will contain the unknown value of µ when the X-experiment is repeated n times. There is a (1 − p) × 100% conﬁdence (or degree of belief) that the latter did contain the unknown value of µ after the X-experiment √ was repeated n times giving the actual data values x1 , . . . , xn . We call tp/2 sn / n the numerical margin for error exhibited by our experiment. 2

190

CHAPTER 9. SPECIAL DISTRIBUTIONS

Transformations of Random Variables Exercise 2.8 Suppose X has density fX (·) with respect to Lebesgue measure λn (·) on n-dimensional Euclidean space Rn . (a) Let Y ≡ AX denote a linear transformation with A a nonsingular matrix. The Jacobian of this linear transformation is ∂(old) ∂x i with |J|+ = |A−1 | = 1/|A|. J ≡ ≡ = A−1 , ∂(new) ∂yj Verify that the rv Y has a density fY (·) with respect to Lebesgue measure that is given by fY (y) = fX (A−1 y)/ |A| on Rn . (b) Suppose now that X has density fX (·) with respect to Lebesgue measure on a region RX in Rn . Suppose the 1-to-1 transformation Y ≡ g(X) from RX to the region RY ≡ g(RX ) has a nonsingular Jacobian with continuous elements at each point of the region. Show that Y has a density given by fY (y) = fX (g −1 (y)) × |[∂(old)/∂(new)]|. (Any “nice” transformation is locally linear.) Exercise 2.9 Suppose that U ≡ XY and V ≡ X/Y for rvs having joint density fXY (·, where x > 0 and y > 0. The inverse transformation is √·) on the region X = U V and Y = U/V with a “nice” Jacobian that is equal to 2v. Thus the joint density of U, V is √ 1 fU V (u, v) = 2v fXY ( uv, u/v )) on the appropriate (u, v)-region, provided that the transformation is 1-to-1. Now evaluate fU V (·, ·) and fV (·) in the following cases. (a) X and Y are independent Exponential(1). (b) X and Y are independent with density 1/(xy)2 on x, y ≥ 1. Evaluate fU (·). (c) X and Y are independent N(0, 1). [Note that this transformation is not 1-1.] (d) X ∼ = N(0, 1) and Y ∼ = Uniform(0, 1) are independent. [This exercise demonstrates vividly the important role played by the regions RX and RY .]

3.

LINEAR ALGEBRA APPLICATIONS

191

3

Linear Algebra Applications

Notation 3.1 (Mean vector and covariance matrix) Let X ≡ (X1 , . . . , Xn ) be a rv. Then E(X) ≡ µ ≡ (µ1 , . . . , µn ) , where µi ≡ E(Xi ) is called the mean vector. And Σ ≡ |[ σij ]| ≡ |[ Cov[Xi , Xj ] ]| is called the covariance matrix. (By the Cauchy–Schwarz inequality, both of µ and Σ are well-deﬁned provided that each of 2 σii ≡ Var[Xi ] ≡ Cov[Xi , Xi ] is ﬁnite.) Deﬁnition 3.1 (Linear algebra) We will operate on n-dimensional space Rn with n × n matrices and n × 1 vectors. (i) A matrix Γ with column vectors γi (that is, Γ = [γ1 , . . . , γn ] ) is called orthogonal if Γ Γ = I. [Thus γi γj equals 1 or 0 according as i = j or i = j; when γi γj = 0 we say that these vectors are orthogonal, and we write γi ⊥ γj .] Under the orthogonal transformation of Rn onto itself deﬁned by y = Γx, the image of each γi is the standardized basis vector ei ≡ (0, . . . , 0, 1, 0, . . . , 0) with the 1 in the ith slot. (ii) Call a symmetric matrix A positive deﬁnite (written A > 0) if x Ax > 0 for all vectors x = 0. Call it nonnegative deﬁnite (written A ≥ 0) if x Ax ≥ 0 for all vectors x = 0. (iii) If A is symmetric and idempotent (that is, if AA = A), then A is called a projection matrix (the symbol P is often used for a projection matrix). (iv) Let Da be the diagonal matrix with dii = ai (and dij = 0 for all i = j). (v) Let R[A] denote the column space of A; that is, it is the set of all vectors that can be written as linear combinations of the column vectors of A. n n (vi) Call x Ax = 1 1 xi aij xj a quadratic form in the vector x. What follows is the statistician’s main result from linear algebra. We simply state it, then interpret it geometrically in discussion 3.1, and then put it into a very useful format in discussion 3.2. Theorem 3.1 (Principal axes theorem) Let A denote an arbitrary real and symmetric matrix of rank r. (a) There exists an orthogonal matrix Γ ≡ [γ1 , . . . , γn ] and a diagonal matrix D for which we have the representation (1)

A = Γ D Γ

and/or

Γ A Γ = D

with rank(D) = r.

The γi are called eigenvectors, while the corresponding dii are called eigenvalues. (See (39) below for further comments.) (b) If A > 0 (A ≥ 0), then all dii > 0 (dii ≥ 0). We can specify Γ such that d11 ≥ · · · ≥ drr > 0 = dr+1,r+1 = · · · = dnn . (c) If P is a projection matrix, then all dii = 1 or 0. Moreover, we must have n r ≡ rank(A) = tr(D) = tr(A) = 1 aii . Discussion 3.1 (Spectral decomposition) Consider a projection matrix P of rank r. Then the transformation y = P x can be broken down as n (2) P x = Γ D Γ x = [γ1 , . . . , γn ] D [γ1 , . . . , γn ] x = 1 dii (γi x) γi ,

192

CHAPTER 9. SPECIAL DISTRIBUTIONS

where (γi x) γi is the projection of x onto γi in the direction of γi , and where this term is present when dii = 1 and is absent when dii = 0. Also, P x ⊥ (I − P )x, where n (3) (I − P )x = i=1 [1 − dii ] (γi x) γi projects onto R[γr+1 , . . . , γn ] = R⊥ [γ1 , . . . , γr ]. Finally, P x = ri=1 dii (γi γi ) x = [ ri=1 Pi ] x (4) with Pi ≡ γi γi . This is called the spectral decomposition of the transformation y = P x. 2 Exercise 3.1 (5)

(a) Show that for compatible matrices B and C,

tr(BC) = tr(CB)

and rank(BC) ≤ rank(B) ∧ rank(C),

giving rank(A Γ) = rank(A) above. (b) Prove theorem 3.1(b)(c) using theorem 3.1(a). (c) Show that R[A] = R[AA ] and R[A ] = R[A A]. Proposition 3.1 (Properties of E(·)) (6)

E(A X B + C) = A E(X) B + C

(a) It holds that and Cov[A X, B Y ] = A Cov[X, Y ] B .

(b) Any covariance matrix ΣX ≡ |[ Cov[Xi , Xj ] ]| satisﬁes ΣX ≥ 0. Exercise 3.2

Prove proposition 3.1.

Discussion 3.2 (Versions of Σ− and Σ−1/2 ) Let X ∼ = (µ, Σ). According to the principal axes theorem, we can make the decomposition (for any orthogonal matrix ∆ whatsoever) 1/2 1/2 D 0 D 0 Σ = Γ D Γ = Γ ∆ ∆ Γ 0 0 0 0 (7) 1/2 1/2 D 0 D 0 = Γ Γ = ( Γ D1/2 )( D1/2 Γ ) ≡ AA , 0 0 0 0 1/2

where D1/2 has the numbers dii on its diagonal and where A is n×k. The presence of ∆ ∆ (which equals I ) shows that this decomposition is not unique. Continuing on gives (8)

Σ = Γ D Γ = (Γ D Γ ) (Γ D Γ ) ≡ Σ1/2 Σ1/2 ,

(9)

Σ−1/2 ≡ Γ D−1/2 Γ ,

where

dii

(10)

Σ− ≡ Γ D− Γ ,

where

D− ≡ D−1/2 D−1/2 .

−1/2

≡

0 √1 dii

if dii = 0, if dii > 0,

3.

LINEAR ALGEBRA APPLICATIONS

193

Note that (11)

Σ Σ− Σ = Σ

(12)

Σ1/2 Σ−1/2 =

and

Σ1/2 Σ−1/2 Σ1/2 = Σ1/2 ,

Ik 0 = Σ−1/2 Σ1/2 0 0

and Σ Σ− =

Ik 0 = Σ− Σ . 0 0

These last two results are in keeping with the deﬁnition of generalized inverses. Recall that the generalized inverse B− of the matrix B is deﬁned to be any matrix B − that satisﬁes B B − B = B. A generalized inverse always exists. It has the following interpretation. Fix the matrix B and the vector c. Then (13)

B β = c (with any c ∈ R[B] ) has the solution βˆ = B − c .

Stated this way, it is clear that the generalized inverse does always exist.

2

Theorem 3.2 (Properties of covariance matrices) (a) The following results are equivalent for real matrices: (14)

Σ is the covariance matrix of some rv Y.

(15)

Σ is symmetric and nonnegative deﬁnite.

(16)

There exists an n × n matrix A such that Σ = AA .

(Recall (7) for A.)

(b) The matrix ΣX is positive deﬁnite (that is, ΣX > 0) if and only a vector c = 0 and a constant b do not exist for which c X = b a.s. Proof. Now, (14) implies (15): Σ is symmetric, since EYi Yj = EYj Yi . Also, a Σa = Var[a Y ] ≥ 0 for all vectors a, so that Σ ≥ 0. Also, (15) implies (16): Just recall (7). Also, (16) implies (14): Let X ≡ (X1 , . . . , Xn ) , where X1 , . . . , Xn are independent N (0, 1). Let Y ≡ AX. Then Y has covariance matrix Σ = AA by (6). 2

Exercise 3.3

Prove theorem 3.2(b).

Exercise 3.4 Let X ∼ = (θ, Σ) and let B be symmetric. (a) E{(X − b) B (X − b)} = tr(B Σ) + (θ − b) B (θ − b). n (b) If Σ = σ2 I, then tr(B Σ) = σ2 tr(B) = σ2 i=1 bii . (c) If Σ = σ2 I and B is idempotent, then tr(B Σ) = σ2 tr(B) = σ2 rank(B). Exercise 3.5 For symmetric A there exists an upper (or lower) triangular matrix H for which A = H H . If A > 0 (or A ≥ 0), we may suppose that all hii > 0 (or hii ≥ 0).

194

CHAPTER 9. SPECIAL DISTRIBUTIONS

Discussion 3.3 (Best linear predictor and multiple correlation) the partitioned random vector Y0 ∼ 0 σ00 σ0 (17) , with |Σ| = 0 . = Y 0 σ0 Σ

Consider

The best linear predictor of Y0 based on Y is (18)

α0 Y ≡ σ0 Σ−1 Y

(or α0 ≡ Σ−1 σ0 ) ,

where “best” is in the sense that (19)

Var[ Y0 − β Y ] ≥ Var[ Y0 − α0 Y ] = σ00 − σ0 Σ σ0

for all β.

In parallel with this, (20)

Corr[ Y0 , β Y ] ≥ Corr[ Y0 , α0 Y ]

for all β.

The maximized value of the correlation (that is, the multiple correlation coeﬃcient) is given by , σ0 Σ−1 σ0 (21) , ρ0·1,...,n ≡ Corr[ Y0 , α0 Y ] = σ00 and the variance of the best linear predictor is also easily seen to equal (22)

Var[ Y0 − α0 Y ] = σ00 − σ0 Σ−1 σ0 = σ00 (1 − ρ20·1,...,n ) .

[Proof. The ﬁrst holds, since Var[ Y0 − β Y ] = Var[ (Y0 − α0 Y ) + (α0 − β)Y ] = Var[ Y0 − α0 Y ] + 2 · 0 + (α0 − β) Σ (α0 − β) ≥ Var[ Y − α0 Y ] . The second holds, since Corr2 [ Y0 , β Y ] =

σ Σ−1 σ0 (β σ0 )2 ≤ 0 , σ00 β Σ β σ00

with equality only at β = c Σ−1 σ0 (as follows from application of Cauchy–Schwarz).] Simple linear regression model We now want the best linear predictor of Y based on X. The conditional distribution of Y given that X = x is given by 2 σXY σXY 2 (23) Y X = x ∼ µ + (x − µ ), σ − = (α + β x, σ2 ) , = Y X Y 2 2 σX σX expressing the moments in terms of σXY σY σXY (24) ρ≡ , β≡ρ , α ≡ µY − µX 2 , σ2 ≡ σY2 (1 − ρ2 ) . σX σY σX σX This leads directly to the simple linear regression model that conditionally on X = x the observations Yi satisfy (25) Yi = α + β xi + i where i ∼ = (0, σ2 ) are iid

with (26)

β=ρ

σY , σX

σ2 ≡ σY2 (1 − ρ2 ) ,

α ≡ µY − µX β .

2

3.

LINEAR ALGEBRA APPLICATIONS

195

Discussion 3.4 (Conditional moments and projections) (1) (1) µ Σ11 Σ12 Y ∼ (27) , . Y = = Y (2) µ(2) Σ21 Σ22

Suppose that

Then the moments of the conditional distribution of Y (1) given that Y (2) = y (2) are summarized in (2) (28) Y (1) Y (2) = y (2) ∼ − µ(2) ), Σ11 − Σ12 Σ−1 = (µ(1) + Σ12 Σ−1 22 (y 22 Σ21 ). To see this, just deﬁne (1) (1) (2) − µ(1) ) − Σ12 Σ−1 − µ(2) ) Z (Y 22 (Y Z≡ (29) ≡ . Y (2) − µ(2) Z (2) It is a minor calculation that 0 Σ11 − Σ12 Σ−1 22 Σ21 0 (30) Z∼ , . = 0 0 Σ22 The exercises will show that (31)

|Σ| = |Σ22 | |Σ11 − Σ12 Σ−1 22 Σ21 | .

[Proof. It is easy to compute ΣZ,12 = Σ12 − Σ12 Σ−1 22 Σ22 = 0. Trivially, ΣZ,22 = Σ22 . Then (a) (b)

−1 −1 ΣZ,11 = Σ12 − 2Σ12 Σ−1 22 Σ22 + Σ12 Σ22 Σ22 Σ22 Σ21

= Σ12 − Σ12 Σ−1 22 Σ21 .

(2) Since Y (1) = µ(1) + Z (1) + Σ12 Σ−1 with Y (2) = Z (2) , we see that conditionally 22 Z (2) (c) Y (1) Y (2) = y (2) ∼ = µ(1) + Z (1) + Σ12 Σ−1 22 z (2) ∼ , ΣZ,11 ) = (µ(1) + Σ12 Σ−1 22 z (2) = (µ(1) + Σ12 Σ−1 − µ(2) ), ΣZ,11 ) , 22 (y

as required.] [See exercise 3.7 below for (31).]

2

Exercise 3.6 Consider the rvs Z (1) and Z (2) in (29). Suppose µ(1) = 0 and µ(2) = 0. Let H1 and H2 denote the Hilbert spaces generated by the rv subsets Y (1) ≡ (Y1 , . . . , Yk ) and Y (2) ≡ (Yk+1 , . . . , Yn ), respectively. Show that Z (1) is the projection of Y (1) into the Hilbert space H2⊥ . (See (6.3.1).) Discussion 3.5 (Partitioned matrices) Let 11 12 A11 A12 A A (32) A≡ , and write A−1 ≡ A21 A22 A21 A22 when the inverse exists. We agree that A11 is k × k.

196

CHAPTER 9. SPECIAL DISTRIBUTIONS

Exercise 3.7 (a) If |A22 | = 0, show that |A| = |A22 | |A11 − A12 A−1 22 A21 |. (b) If |A| = 0, show that |A + x y | = |A| (1 + y A−1 x) for all vectors x, y. [Hint. Appeal to 0 C 0 I = |C| |D| and work with B≡ ED −A12 A−1 22 I for appropriate choices.] Exercise 3.8

(a) Show that for a symmetric A having |A11 | = 0 and |A22 | = 0 :

(33)

−1 A11 = (A11 − A12 A−1 22 A21 )

22 and A12 = −A−1 11 A12 A .

(34)

−1 A22 = (A22 − A21 A−1 11 A12 )

11 and A21 = −A−1 22 A21 A .

[Hint. Start multiplying the partioned form of A A−1 = I.] (b) Obtain analogous formulas from A−1 A = I. (c) Show that (35)

A11 A11 + A12 A21 = I

Exercise 3.9 (36)

and A11 A12 + A12 A22 = 0 .

2

Show that for symmetric A,

∂ [ β A β] = 2 A β . ∂β

Discussion 3.6 (Simultaneous decomposition) For a real symmetric matrix A that is nonnegative deﬁnite (that is, A ≥ 0) we wrote (37)

A = Γ D Γ

and

Γ A Γ = D

with d11 ≥ · · · ≥ drr > 0. (A) We note that (38)

|A − λ I| = |Γ| |D − λ I| |Γ | = |D − λ I| = 0

all have the same solutions d11 , . . . , drr , 0, and thus d11 , . . . , drr are indeed the nonzero eigenvalues of A. Moreover, (37) gives (39)

AΓ = ΓD

or A γi = dii γi

for 1 ≤ i ≤ r,

so that γ1 , . . . , γr are the corresponding eigenvectors. (B) Suppose A > 0 and B ≥ 0. Then (40)

|B − λ A| = 0 , |A−1/2 B A−1/2 − λ I| = 0 , have the same solutions λ.

|A−1 B − λ I| = 0

[Just note that |B − λ A| = |A1/2 | |A−1/2 B A−1/2 − λ I| |A1/2 | and |A−1 | |B − λ A| = |A−1 B − λ I| .] Writing A−1/2 B A−1/2 = ∆ Dθ ∆ with ∆ orthogonal gives (41)

B = (A1/2 ∆) Dθ (∆ A1/2 ) and A = (A1/2 ∆) (∆ A1/2 ) .

This last formula is called the simultaneous decomposition of A and B.

2

3.

LINEAR ALGEBRA APPLICATIONS

Discussion 3.7 (42)

197

(a) (Cauchy–Schwarz) For all vectors x, y:

(x y)2 ≤ x2 y2 ,

with equality (for y = 0) if and only if x = cy for some constant c. (b) For any real symmetric matrix A > 0 (43)

a A a = d11 (A) a a

max a =0

(as follows immediately from (1), with d11 (A) the largest eigenvalue of A). (c) Let A > 0, and ﬁx C ≥ 0 and b = 0. Then (44)

(x y)2 ≤ (x A x) (y A−1 y),

with equality (when y = 0) if and only if x = c A−1 y for some c. Also, (45)

min {a A a} = 1/(1 A−1 1),

a 1=1

with equality only at a0 ≡ A−1 1/(1 A−1 1). Also, (46)

max a =0

(a b)2 = b A−1 b, a A a

with equality only when a = (some c) A−1 b. Also, max a =0

(47)

a C a (a A1/2 ) (A−1/2 C A−1/2 ) (A1/2 a) = max a =0 a A a (a A1/2 ) (A1/2 a)

= d11 (A−1/2 CA−1/2 ) = d11 (C A−1 ) ,

with (a C a) = (a b)2 an important special case (also approachable via (46)). (d) Let A > 0, let Bk×n have rank(B) = k, and let bk×1 = 0. Then (48)

min {a A a} = b [B A−1 B ]−1 b,

B a=b

is achieved at a0 ≡ B [B A−1 B ]−1 b .

Exercise 3.10 Prove (42)–(47) (the equality in (42) needs some attention). [The harder (48) is proven below.] Proof.

Consider (48). Now, a A a ≥

(a y)2 y A−1 y

for all y = 0

(a)

=

[a B [B A−1 B ]−1 b ]2 b [B A−1 B ]−1 (B A−1 B )[B A−1 B ]−1 b

(b)

=

[b [B A−1 B ]−1 b ]2 b [B A−1 B ]−1 b

if y ≡ B [B A−1 B ]−1 b

for all a, using B a = b

= b [B A−1 B ]−1 b yielding a bound not depending on a, which proves (48).

2

198

CHAPTER 9. SPECIAL DISTRIBUTIONS

Exercise 3.11 (Minimum variance unbiased linear estimators) (a) Let X1 , . . . , Xn 2 be uncorrelated with common mean n µ and common ﬁnite variance σ . All linear n estimators T ≡ 1 ai Xi having 1 ai = 1 are unbiased estimators of µ (that is, ET = µ). Show that the choice with all ai = 1/n has minimum variance within this class of linear unbiased estimators. (b) Determine the minimum variance unbiased linear estimator of the common mean µ when the variances are σ2 /c1 , . . . , σn2 /cn , with the ck being known constants.

4.

THE MULTIVARIATE NORMAL DISTRIBUTION

4

The Multivariate Normal Distribution

199

Deﬁnition 4.1 (Jointly normal) Call Y = (Y1 , . . . , Yn ) jointly normal with 0 means if there exist iid N (0, 1) rvs X1 , . . . , Xk and an n × k matrix A of known , constants for which Y = AX. [We again write Y in this section, rather than Y when the context seems clear.] Note that the n × n covariance matrix ΣY ≡ Σ of the random vector Y is (1)

Σ ≡ ΣY = E Y Y = EAXX A = AA .

The covariance matrix of X is the k × k identity matrix Ik . We will write X ∼ = N (0, Ik ), and we will write Y ∼ = N (0, Σ). Then write Y ∼ = N (µ, Σ) if Y − µ ∼ = N (0, Σ). Call Y multivariate normal with mean vector µ and covariances matrix Σ. Call Y nondegenerate when |Σ| = 0 (that is, the determinant of Σ is not equal to 0). Say that Y1 , . . . , Yn are linearly independent if (rank Σ) = n. Of course, this means that (2)

Y is nondegenerate if and only if rank(A) = n.

Now, Σ is symmetric. Also aΣa = Var[aY ] ≥ 0 for all vectors a. When aΣa ≥ 0 for all vectors a, the symmetric matrix Σ is called nonnegative deﬁnite, and one writes Σ ≥ 0. Theorem 4.1 (Densities) If Y ∼ = N (0, Σ) is nondegenerate, then Y has density (with respect to Lebesgue measure on Rn ) given by 1 fY (y) = (3) exp(−y Σ−1 y/2) for all y ∈ Rn . (2π)n/2 |Σ|1/2 [Note that each possible normal distribution is completely determined by µ and Σ.] ∼ N (0, Ik ). Proof. Now, Y = XA, where AA = Σ, (rankA) = n, |A| = 0, X = It is trivial that (a) with fX (x) ≡ (2π)−n/2 exp(−x x/2). P (X ∈ Bn ) = 1Bn fX (x)dx Thus X = A−1 Y gives

P (Y ∈ Bn ) = P (AX ∈ Bn ) = P (X ∈ A−1 Bn ) = 1A−1 Bn (x)fX (x) dx ∂x = 1A−1 Bn (A−1 y)fX (A−1 y)| | dy ∂y ∂x = 1Bn (y)(2π)−n/2 exp(−(A−1 y) (A−1 y)/2)| | dy ∂y = (2π)−n/2 |Σ|−1/2 exp(−y Σ−1 y/2) dy,

(b)

Bn −1

) (A−1 ) = (AA )−1 = Σ−1 and ∂x (c) | | = |A−1 | = |A−1 ||A−1 | = |Σ−1 | = 1/ |Σ|. ∂y This is the required statement. 2

since (A

200

CHAPTER 9. SPECIAL DISTRIBUTIONS

Theorem 4.2 (Characteristic functions and representations) (a) If we are given a random vector Y = An×k X k×1 where X ∼ = N (0, Ik ), we have (4)

φY (t) ≡ E eit Y = exp(−t Σt/2)

with Σ ≡ AA and rank(Σ) = rank(A). (b) If Y has characteristic function φY (t) ≡ EeitY = exp(−t Σt/2) with Σ ≥ 0 of rank k, then Y ∼ (5) with (rank A) = k and X ∼ = An×k X k×1 = N (0, I). (Thus the number of independent rvs Xi ’s needed is equal to the rank of A.) Proof. Our proof will use the fact that the characteristic function φY of any rv Y is unique (as will be shown below in chapter 13.) [When a density function does not exist, one can use this characteristic function for many of the same purposes.] We observe that φY (t) = E exp(it AX) = E exp(i(A t) X) (a)

= exp(−(A t) (A t)/2)

since E eitX j = exp(−t2 /2)

by example 13.1.2 below (b)

= exp(−t (AA )t/2).

The converse follows from (9.3.7).

2

Even when a multivariate normal rv Y does not have a density, the characteristic function can often be manipulated to establish a desired result. Theorem 4.3 (Marginals, independence, and linear combinations) Suppose that Σ11 Σ12 Y = (Y1 , . . . , Yk , Yk+1 , . . . , Yn ) ∼ . with Σ≡ = N (0, Σ) Σ21 Σ22 (i) The marginal covariance matrix of (Y1 , . . . , Yk ) is the k × k matrix Σ11 , and (Y1 , . . . , Yk ) ∼ (6) = N (0, Σ11 ). (ii) If Σ12 = 0, then (Y1 , . . . , Yk ) and (Yk+1 , . . . , Yn ) are independent. (iii) If (Y1 , Y2 ) is a jointly normal rv, then Y1 and Y2 are independent if and only if they have the zero covariance Cov[Y1 , Y2 ] = 0. (iv) Linear combinations of normals are normal. Proof. (i) Use the ﬁrst k coordinates of the representation Y = AX. (ii) Use the fact that one can factor 1 Σ11 0 φY (t) = exp(− t t) . 0 Σ22 2 (iii) Just apply (ii), as the other direction is trivial. (iv) Z m×1 ≡ B m×n Y n×1 = B(AX) = (BA)X. 2

4.

THE MULTIVARIATE NORMAL DISTRIBUTION

201

Theorem 4.4 (Conditional distributions) If (1) (1) Y µ Σ11 Σ12 ∼ Y = (7) N , , = Y (2) µ(2) Σ21 Σ22 then (8)

(2) − µ(2) ), Σ11 − Σ12 Σ−1 Y (1) Y (2) = y (2) ∼ = N (µ(1) + Σ12 Σ−1 22 (y 22 Σ21 ).

Note that (9)

|Σ| = |Σ22 ||Σ11 − Σ12 Σ−1 22 Σ21 |.

Proof.

The vector (1) (1) (2) Z − µ(1) ) − Σ12 Σ−1 − µ(2) ) (Y 22 (Y Z≡ ≡ Z (2) Y (2) − µ(2)

(10)

is just a linear combination of the Yi ’s, and so it is normal. We need only verify the means and variances. But we did this in discussion 9.3.4. 2 Exercise 4.1 Show that (Y1 , Y2 ) can have normal marginals without being jointly normal. [Hint. Consider starting with a joint N (0, I) density on R2 and move mass in a symmetric fashion to make the joint distribution nonnormal, but still keeping the marginals normal.]

Quadratic Forms Exercise 4.2 Let Y n×1 ∼ = N (0, I), and suppose that A is symmetric and of rank r. Then Y AY ∼ = χ2r if and only if A is a projection matrix (that is, A2 = A). Exercise 4.3 Let Y n×1 ∼ = N (0, I). Suppose that A and B are symmetric and both Y AY and Y BY have chisquare distributions. Show that Y AY and Y BY are independent if and only if AB = 0. Exercise 4.4 Suppose A and B are n × n projection matrices with ranks rA and rB , and suppose A B = 0 and I − A − B ≥ 0. Then: (a) I − A is a projection matrix of rank n − rA . (b) I − A − B is a projection matrix of rank n − rA − rB . Exercise 4.5 Suppose Y n×1 ∼ = N (0, Σ), and let A be an arbitrary symmetric matrix of rank r. Show that Y AY ∼ = χ2r if and only if AΣA = A.

The Multivariate CLT The following result is theorem 14.1.3, but we also list it here for convenient referral. Theorem 4.5 Suppose that the random vectors X1 , . . . , Xn are iid (µ, Σ). Then √ ¯ n − µ) →d N (0, Σ) (11) n (X as n → ∞.

202

CHAPTER 9. SPECIAL DISTRIBUTIONS

Normal Processes To specify a normal process, we must specify consistent distributions (in the sense of Kolmogorov’s consistency theorem). But µ and Σ completely specify N (µ, Σ), while the marginals of N (µ, Σ) are N (µ(1) , Σ11 ). Thus a normal process exists, provided only that the mean value function µ(·) on I and the covariance function Cov(·, ·) on I × I are well-deﬁned and are such that Cov(·, ·) is nonnegative deﬁnite (meaning that every n-dimensional covariance matrix formed from it is nonnegative deﬁnite). We call {S(t) : 0 ≤ t < ∞} a Brownian motion if S is a normal process having (12)

ES(t) = 0

and

Cov[S(s), S(t)] = s ∧ t

for all s, t ≥ 0.

Since this covariance function is nonnegative deﬁnite, a version of the process S exists on (R[0,∞) , B[0,∞) ) by the Kolmogorov consistency condition. Then (13)

U(t) ≡ −[S(t) − t S(1)] for all 0 ≤ t ≤ 1 is called Brownian bridge.

It is a normal process on (R[0,1] , B[0,1] ) for which (14)

EU(t) = 0

and

Cov[U(s), U(t)] = s ∧ t − st

for all 0 ≤ s, t ≤ 1.

Chapter 10

WLLN, SLLN, LIL, and Series 0

Introduction

This is one of the classically important chapters of this text. The ﬁrst three sections of it are devoted to developing the speciﬁc tools we will need. In the second section we also present Khinchin’s weak law of large numbers (WLLN), which can be viewed as anticipating both of the classical laws of large numbers (LLNs). Both the classical weak law of large numbers (Feller’s WLLN) and classical strong law of large numbers (Kolmogorov’s SLLN) are presented in section 10.4, where appropriate negligibility of the summands is also emphasized. This section is the main focus of the chapter. Some applications of these LLNs are given in the following section 10.5. Then we branch out. Suppose a sample moment is formed when the underlying true moment is inﬁnite. Can a parameter be deﬁned that captures the behavior of this estimator? This is the subject of section 10.6, and it ties in rather closely with section 7.7. The law of the iterated logarithm (LIL), the strong Markov property, and convergence of inﬁnite series are treated in sections 10.7 – 10.9. The choice was made to be rather speciﬁc in section 10.4, with easy generalizations in section 10.9. The usual choice is to begin more generally, and then specialize. Martingales (mgs) are introduced brieﬂy in section 10.10, both for limited use in chapter 12 and so that the inequalities in the following section 10.11 can be presented in appropriate generality. The ﬁnal section 10.12 contains a uniform version of the earlier SLLN.

203

204

1

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Borel–Cantelli and Kronecker lemmas

The ﬁrst three sections will develop the required tools, while applications will begin with the LLNs (the ﬁrst of which appears in section 10.3). We use the notation ∞ ∞ [An i.o.] = [ω : ω ∈ An inﬁnitely often] = n=1 m=n Am = limn An . (1) This concept is important in dealing with convergence of various random elements. The following lemmas exhibit a nice dichotomy relative to sequences of independent events. Lemma 1.1 (Borel–Cantelli lemma) ∞

(2)

P (An ) < ∞

implies

For any events A n ,

P (An i.o.) = 0.

n=1

Lemma 1.2 (Second Borel–Cantelli lemma) events A1 , A2 , . . . , we have the converse ∞

(3)

P (An ) = ∞

implies

For a sequence of independent

P (An i.o.) = 1.

n=1

Thus ∞ independent events A1 , A2 , . . . have P (An i.o.) equal to 0 or 1 according as 1 P (An ) is ﬁnite or inﬁnite. Proof.

We use proposition 1.1.2 freely. Now, ∞ ∞ ∞ (a) P (An i.o.) = P ( limn n Am ) = limn P ( n Am ) ≤ limn n P (Am ) = 0 ∞ whenever 1 P (An ) < ∞. Also, ∞ ∞ ∞ P ([ lim An ]c ) = P ( n=1 m=n Acm ) = limn P ( m=n Acm ) = limn limN P (

N

c m=n Am )

N

= limn limN

(c)

N ≤ limn limN exp(− m=n P (Am ))

(d)

∞ = limn exp(− m=n P (Am )) = limn exp(−∞) = limn 0 = 0,

using

∞ 1

P (An ) = ∞.

m=n [1

− P (Am )]

(b)

by independence since 1 − x ≤ exp(−x)

2

Remark 1.1 (Kolmogorov’s 0-1 law) In theorem 8.2.1 we considered the tail σ-ﬁeld T ≡ ∩∞ n=1 F(Xn , Xn+1 , . . .) of an arbitrary sequence of independent rvs X1 , X2 , . . . . We learned that P (D) = 0 or 1 for all D ∈ T . 2

1.

BOREL–CANTELLI AND KRONECKER LEMMAS

205

Lemmas About Real Numbers There is an important bridge going from the convergence of series to the convergence of averages. [An alternative bridge is provided by the monotone inequality 10.11.1 (see exercise 10.4.10).] Lemma 1.3 (Kronecker’s lemma) Let x1 , x2 , . . . be arbitrary real numbers. If bn ≥ 0 and ∞, then (4)

n

xk → r

implies

1

Proof. (a)

1 bn

k Let sk ≡ 1 xk with s0 ≡ 0 and b0 ≡ 0. Summing by parts gives n 1 n 1 n−1 (bk − bk+1 ) sk + b1n bn sn 1 bk xk = bn 1 bk (sk − sk−1 ) = bn 0 =−

(c)

=−

(d)

n 1

n 1

ak sk−1 + sn

where

ak ≡

n bk − bk−1 ≥ 0 with 1 ak = 1 bn

ak (sk−1 − r) + (sn − r).

n 1 n N bn 1 bk xk ≤ 1 |ak (sk−1 − r)| + N +1 |ak (sk−1 − r)| + |sn − r| N

(e)

bk xk /bn → 0.

1

(b)

Thus

n

≤

1

≤ 3

(bk − bk−1 )|sk−1 − r| + +

bn

for N suﬃciently large, but ﬁxed

for n suﬃciently larger than N ,

so that thelemma holds. n n [Since 1 xk → r, we must have xk → 0. Note that 1 bk xk /bn puts large weight only on the later terms.] 2 Exercise 1.1 (Ces` aro summability) If Sn ≡

n k=1

xk → r, then

n k=1

Sk /n → r.

Exercise 1.2 (Toeplitz) Let ank (for 1 ≤ k ≤ kn , with kn → ∞) be such that: kn (i) For every ﬁxed k, we have ank → 0. (ii) k=1 |ank | ≤ c < ∞, for every n. kn Let xn ≡ k=1 ank xk . Then (a) xn → 0 implies kn If k=1 ank → 1, then

xn → 0.

implies xn → x. n In particular, if bn ≡ k=1 ak ∞, then n (c) entails xn → x ﬁnite k=1 ak xk /bn → x. (b)

xn → x

[This exercise will not be employed anywhere in this text.]

206

2

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Truncation, WLLN, and Review of Inequalities

Truncated rvs necessarily have moments, and this makes them easier to work with. But it is crucial not to lose anything in the truncation. Deﬁnition 2.1 (Khinchin equivalence) Two sequences of rvs X1 , X2 , . . . and ∞ Y1 , Y2 , . . . for which n=1 P (Xn = Yn ) < ∞ are called Khinchin equivalent. Proposition 2.1 Let X1 , X2 , . . . and Y1 , Y2 , . . . be Khinchin equivalent rvs. (a) If Xn →a.s. or p , then Yn →a.s. or p , and the limits are equal. n n (b) If Sn ≡ 1 Xk →a.s. or p , then Tn ≡ 1 Yk →a.s. or p . (c) If Sn /bn →a.s. or p and bn → ∞, then Tn /bn →a.s. or p , with equal limits. Proof. (p)

The Borel–Cantelli lemma gives P (Xn = Yn i.o.) = 0.

Thus the a.s. statements for Xn and Sn are trivial. Moreover, since Xn (ω) = Yn (ω) for all n ≥ some n(ω), we have (q) (r)

Sn(ω) + Sn − Sn(ω) Sn(ω) + Tn − Tn(ω) Sn(ω) − Tn(ω) Tn Sn = = = + bn bn bn bn bn = o(1) + Tn /bn

using bn → ∞. Since a sequence (such as Xn , Sn or Sn /bn ) converges in probability if and only if each subsequence n contains a further subsequence n on which the convergence is a.s., the in probability statements follow directly from the a.s. statements. 2 Inequality 2.1 (Sandwiching E|X|) (1)

∞

P (|X| ≥ n) ≤ E|X| ≤

n=1

∞

For any rv X we have (see ﬁgure 2.1)

P (|X| ≥ n).

n=0

If X is a rv with values 0, 1, 2, . . . , then (2)

E(X) =

∞

P (X ≥ n).

n=1

∞ Proof. If X ≥ 0, then EX = 0 [1 − F (x)] dF (x) by (7.4.11). If X ≥ 0 is integer valued, then ∞ k E(X) = ∞ k=0 k P (X = k) = k=1 n=1 P (X = k) ∞ ∞ = n=1 k=n P (X = k) (a) = ∞ n=1 P (X ≥ n). For the greatest integer function [·], an arbitrary rv satisﬁes (b)

[|X|] ≤ |X| ≤ [|X|] + 1.

2.

TRUNCATION, WLLN, AND REVIEW OF INEQUALITIES

207

Moreover, (a) shows that ∞ ∞ (c) E[|X|] = n=1 P ([|X|] ≥ n) = n=1 P (|X| ≥ n) , with (d)

E{ [|X|] + 1 } =

∞

n=1 P (|X|

≥ n) + 1 =

ˇ ˜ Example 2.1 (Truncating and Winsorizing) truncate and Winsorize the Xn by deﬁning (3)

∞

n=1 P (|X|

≥ n) + P (|X| ≥ 0).2

Let X1 , X2 , . . . be iid as X. Let us

ˇ n = Xn ×1[|X |
From (2) we see that (4)

E|X| < ∞ if and only if ∞ ∞ ˇ ˜ 1 P (Xn = Xn ) < ∞ if and only if 1 P (Xn = Xn ) < ∞ ,

ˇ n ’s and X ˜ n ’s are Khinchin equivalent to the Xn ’s if and only if the so that these X absolute moment E|X| < ∞. (Do not lose sight of this during the SLLN.) 2 Proof. Using inequality 2.1, then iid, and then the Borel–Cantelli lemmas, we obtain that E|X| < ∞ if and only if ∞ 1 P (|X| ≥ n) < ∞ if and only if ∞ P (|X | ≥ n) < ∞ if and only if P (|X | ≥ n i.o.) = 0. We emphasize that n n 1 (5)

E|X| < ∞

if and only if

P (|Xn | ≥ n i.o.) = 0.

2

∞ Exercise 2.1 (a) Show that X ≥ 0 gives EX 2 = k=0 (2k + 1)P (X > k). Thus ∞ (6) if and only if EX 2 < ∞ n=1 nP (|X| ≥ n) < ∞. ˇ n or X ˜ n result when Xn is truncated (b) Now let X1 , X2 , . . . be√iid √ as X. Now let X or Winsorized outside (− n, n ). Show that (7)

EX 2 < ∞ if and only if ˇ n ’s and X ˜ n ’s are Khinchin equivalent to the Xn ’s. these new X 1 F|X|(·)

0

1

Figure 2.1 The moment E|X| =

∞

∫0

[1 – F |X|(x)]dx is sandwiched.

208

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Khinchin’s WLLN We begin with an easy result that illustrates our path rather clearly. As we improve our technique, we will be able to improve this result. Theorem 2.1 (WLLN; Khinchin) Let X1 , . . . , Xn be iid with mean µ. Then (8)

¯ n →p µ . X

Proof. Truncate via Ynk ≡ Xk × 1[−n≤Xk ≤n] , with µn ≡ EYnk . Note that µn → µ. Now, for any > 0, Chebyshev’s inequality gives √ 2 P ( |Y¯n − µn | > ) ≤ n1 2 n Var[Yn1 ] ≤ 1n E(Yn1 (a) )

2 (b) ≤ 1n E Yn1 1[|X1 |≤ √n ] + 1[|X1 |> √n ] 2 n n

+

(d)

≤ +

(e)

≤ 2

(c)

≤

1

n n

[

[|x|>

√ n<|x|≤n ]

√ n]

|x| dFX (x)

|x| dFX (x)

for all ≥ (some n ),

since E|X1 | < ∞ implies that the integral in (d) goes to 0. Thus Y¯n − µn →p 0. And since µn → µ, this gives Y¯n →p µ. Thus, by the Khinchin equivalence of ¯ n →p µ. 2 proposition 2.1(c) and by example 2.1 we have X Exercise 2.2 (More general WLLN) Suppose that Xn1 , . . . , Xnn are independent. Truncate as before via Ynk ≡ Xnk × 1[−n≤Xnk ≤n] , and let µnk ≡ E(Ynk ). Then set µ ¯n ≡ n1 µnk /n. (i) Show that the same proof as above gives (a)

¯n − µ X ¯ n →p 0,

even in this non iid case,

provided that both n (b) and 1 P (|Xnk | > n) → 0 n 1 (c) 1 E {|Xnk | × 1[|Xnk |≤n ] } → 0 n

for all > 0 .

¯ n − µn →p 0 still holds even if (c) is weakened appropriately. (ii) Show that X Remark 2.1 There are two natural ways to proceed to improve Khinchin’s WLLN ¯ n →a.s. µ ; and this is in the iid case. One way is to obtain the conclusion X done in Kolmogorov’s SLLN (theorem 10.4.2 below). Another way is to relax the assumption of a ﬁnite mean and center diﬀerently; and this is done in Feller’s WLLN (theorem 10.4.1 below). [Other possibilities and other approaches will be outlined in the exercises of section 10.4.] In section 10.3 we will develop a number of inequalities (so called “maximal inequalities”) to help us to the stated goal. (At the end of this section the reader could go directly to section 10.4, and then go to section 10.3 for the inequalities as they are needed.) 2

2.

TRUNCATION, WLLN, AND REVIEW OF INEQUALITIES

209

Review of General Inequalities from Measure Theory As we have completed the transition from measure theory to probability theory, we take this opportunity to restate without comment a few of the most important inequalities presented earlier. Inequality 2.2 (Review) Then:

Let X and Y be rvs on a probability space (Ω, A, P ).

(9)

Cr -inequality: E|X + Y |r ≤ Cr {E|X|r + E|Y |r } for r > 0, Cr ≡ 2(r∨1)−1 .

(10)

H¨ older:

E|XY | ≤ (E|X|r )1/r (E|Y |s )1/s

for r > 1, 1r +

(11)

Liapunov:

(E|X|r )1/r is in r,

for r ≥ 0.

(12)

Markov:

P (|X| ≥ λ) ≤ E|X|r /λr

(13)

Dispersion :

E|X| ≤ E|X + Y |

(14)

Jensen:

g(EX) ≤ Eg(X)

for all λ > 0;

1 s

= 1.

requires r > 0.

for independent rvs with 0 means.

if g is convex on some (a, b) ⊂ R having P (X ∈ (a, b)) = 1, and if EX is ﬁnite. Deﬁnition 2.2 (“Big ohp ”, and “little ohp ”, =a , and “at most” ⊕) (a) We say that Zn is bounded in probability [and write Zn = Op (1)] if for all > 0 there exists a constant M for which P (|Zn | ≥ M ) < . For a sequence an , we write Zn = Op (an ) if Zn /an = Op (1); and we say that Zn is of order an , in probability. (b) If Zn →p 0, we write Zn = op (1). We write Zn = op (an ) if Zn /an →p 0. (c) This notation (without subscript p) was also used for sequences of real numbers zn and an . For example, zn = o(an ) if zn /an → 0. (Note that o(an ) = op (an ).) (d) Write Un =a Vn if Un −Vn →p 0; and call Un and Vn asymptotically equal. (This is eﬀectively a passage to the limit that still allows n to appear on the right-side.) (e) We write a = b ⊕ c if |a − b| ≤ c. (This can be used in the same fashion as op (·), but it allows one to keep track of an absolute bound on the diﬀerence. Especially, it allows inequalities to be strung together more eﬀectively.) Exercise 2.3 For independent rvs X and Y and r ≥ 1, we have E|X + Y |r is ﬁnite if and only if E|X|r and E|Y |r are ﬁnite. Exercise 2.4 γn Vn →p 0.

If Vn = Op (1) and γn = op (1) are rvs on the same (Ω, A, P ), then

210

3

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Maximal Inequalities and Symmetrization

Sums of independent random variables play an important role in probability and statistics. Our goal initially in this section is to develop probability bounds for the maximum of the ﬁrst n partial sums. Such inequalities are called maximal inequalities. The most famous of these is Kolmogorov’s inequality. For symmetric rvs, L´evy’s inequality is an extremely clean and powerful version of such a maximal inequality; it does not require the underlying rvs to have any moments. Neither does the Ottavani–Skorokhod inequality, which is true for arbitrary rvs, though it is not nearly as clean. Inequality 3.1 (Kolmogorov) Let X1 , X2 , . . . be independent, with Xk ∼ = (0, σk2 ). Let Sk ≡ X1 + · · · + Xk . Then n (1) P max |Sk | ≥ λ ≤ Var[Sn ]/λ2 = σk2 /λ2 for all λ > 0. 1≤k≤n

k=1

[This contains Chebyshev’s inequality that P (|Sn | ≥ λ) ≤ Var[Sn ]/λ2 for all λ > 0.] n Proof. Let Ak ≡ [max1≤j
2

Deﬁnition 3.1 (Symmetric rvs) A rv X is called symmetric if X ∼ = −X. Note that this is equivalent to its df satisfying F (−x) = 1 − F− (x) for all x ≥ 0. Suppose X∼ = X are independent rvs; then X s ≡ X − X is called the symmetrization of the initial rv X. Deﬁnition 3.2 (Medians) Let X be an arbitrary rv. Then m ≡ median(X) is any number for which P (X ≥ m) ≥ 12 and P (X ≤ m) ≥ 12 . [One median of the symmetrization X s of any rv X is always 0. And (2) below shows that the tails of X s behave roughly the same as do those of X.] Inequality 3.2 (Symmetrization inequality) Suppose that Xs ≡ X − X ∼ where X = X with X and X independent. Then for every real number a we have (2)

2−1 P (|X − median(X)| ≥ λ) ≤ P (|X s | ≥ λ) ≤ 2 P (|X − a| ≥ λ/2)

for all λ > 0. We may replace ≥ by > in the three events in (2).

3.

MAXIMAL INEQUALITIES AND SYMMETRIZATION

211

Let m ≡ median(X). Now, the ﬁrst inequality comes from

Proof.

P (X s ≥ λ) = P [(X − m) − (X − m) ≥ λ] ≥ P (X − m ≥ λ) P (X − m ≤ 0) ≥ P (X − m ≥ λ)/2.

(a)

The second inequality holds, since for any real a, P (|X s | ≥ λ) = P (|(X − a) − (X − a)| ≥ λ) ≤ P (|X − a| ≥ λ/2) + P (|X − a| ≥ λ/2) = 2P (|X − a| ≥ λ/2).

(b)

2

Inequality 3.3 (L´ evy) Let X1 , . . . , Xn be independent and symmetric rv’s. Let Sn ≡ X1 + · · · + Xn . Then (3) for all λ > 0, P max |Sk | ≥ λ ≤ 2 P (|Sn | ≥ λ) 1≤k≤n

(4)

P

max |Xk | ≥ λ ≤ 2P (|Sn | ≥ λ)

1≤k≤n

for all λ > 0.

Proof. Let Ak ≡ [max1≤j
(b)

by independence of X1 , . . . , Xk from Xk+1 , . . . , Xn ≥ n1 P (Ak )/2 by symmetry

(c)

= P (max1≤k≤n Sk ≥ λ)/2.

Combine this with the symmetric result, and achieve the ﬁrst claim. Now let Ak ≡ [ max1≤j
P (Ak ) ≤ P (Ak ∩ [|Sn | ≥ λ]) + P (Ak ∩ [|Sno | ≥ λ]) = 2P (Ak ∩ [|Sn | ≥ λ]).

So summing on k gives P (A) ≤ 2P (A ∩ [|Sn | ≥ λ]) ≤ 2P (|Sn | ≥ λ). An alternative proof: Let M ≡ XK , where K ≡ min{k : |Xk | = max1≤j≤n |Xj |}. Let T ≡ Sn − XK . Then, for all four choices of + or − signs, the rvs (±M, ±T ) have the same distribution. Then we require both P (M ≥ λ) ≤ P (M ≥ λ and T ≥ 0) + P (M ≥ λ and T ≤ 0) (e)

= 2P (M ≥ λ and T ≥ 0)

(f)

≤ 2P (M + T ≥ λ) = 2P (Sn ≥ λ)

and the symmetric result. [See exercise 3.1 below for more on (4).]

2

212

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Remark 3.1 Kolmogorov’s inequality is a moment inequality. Since the rv Sn /StDev[Sn ] ∼ = (0, 1) is often approximately normal (2π)−1/2 exp(−x2 /2) on the line, and since - ∞ 1 2 x 2 √ exp(−x /2) dx ≤ exp(−x2 /2) dx P (|N (0, 1)| > λ) = π λ 2π λ [|x|≥λ] 21 (6) for all λ > 0, exp(−λ2 /2) ≤ πλ both L´evy’s inequality and the Ottavani–Skorokhod inequality to follow oﬀer the hope of a much better bound. 2 Inequality 3.4 Let Sk ≡ X1 + · · · + Xk for independent rvs Xk . (Ottavani–Skorokhod) For all 0 < c < 1 we have P (|Sn | ≥ cλ) (7) P max |Sk | ≥ λ ≤ for λ > 0. 1≤k≤n [1 − max1≤k≤n P (|Sn − Sk | > (1 − c)λ)] √ (8) ≤ 2 P (|Sn | ≥ cλ) for all λ ≥ 2 StDev[Sn ]/(1 − c). (Etemadi) Alternatively, P (max1≤k≤n |Sk | ≥ 4λ) ≤ 4 max1≤k≤n P (|Sk | ≥ λ)

for all λ > 0.

n Proof. Let Ak ≡ [S1 < λ, . . . , Sk−1 < λ, Sk ≥ λ], so that k=1 Ak = [max1≤k≤n Sk ≥ λ]. Thus k is the smallest index for which Sk exceeds λ. (This is now the third time we have used this same trick.) Note that (a) (b)

a ≡ min1≤k≤n P (Sn − Sk ≤ (1 − c)λ) = 1 − max1≤k≤n P (Sn − Sk > (1 − c)λ) ≥ 1 − max1≤k≤n Var[Sn − Sk ]/[(1 − c)λ]2

(c)

by Chebyshev’s inequality

≥ 1 − Var[Sn ]/[(1 − c)λ]2 √ ≥ 12 if λ ≥ 2 StDev[Sn ]/(1 − c)

allows us to “improve” (7) to (8). Meanwhile, (7) comes from n (d) a × P (max1≤k≤n Sk ≥ λ) ≤ k=1 P (Sn − Sk ≤ (1 − c)λ)P (Ak ) (e) = nk=1 P (Ak ∩ [Sn − Sk ≤ (1 − c)λ]) by independence (f)

≤ P (Sn ≥ cλ).

Combining (f) and (b) with the symmetric result completes the proof.

Exercise 3.1

Prove Etemadi’s inequality.

2

3.

MAXIMAL INEQUALITIES AND SYMMETRIZATION

213

Exercise 3.2 Consider the independent symmetrized rvs Xks ≡ Xk − Xk , for 1 ≤ k ≤ n, where each Xk is an independent copy of Xk . Let mk denote a median of Xk , and let a denote an arbitrary number. Show that for all λ > 0 we have (9)

2−1 P (max |Xk − mk | ≥ λ) ≤ P (max |Xks | ≥ λ) ≤ 2P (max |Xk − a| ≥ λ/2).

[Then P (max |Xks | ≥ λ) ≤ 2P (|Sns | ≥ λ) ≤ 4P (|Sn − a| ≥ λ/2) (for any real a), by inequality 3.3 and inequality 3.2.]

Inequalities for Rademacher RVs Inequality 3.5 (Symmetrization; Gin´ e–Zinn) Let X1 , . . . , Xn be iid rvs, and let 1 , . . . , n denote an independent sample of iid Radamacher rvs (that satisfy P ( k = ±1) = 12 ). Then n . / . / n 1 1 P √ (10) for all λ > 0 .

i Xi > 2λ ≤ sup 2 P √ Xk > λ n n n≥1 k=1

Proof. (a)

By conditioning on the Rademacher rvs we obtain n P (n−1/2 | 1 k Xk | > 2 λ) ≤ P (n−1/2 | k:k =1 k Xk | > λ) + P (n−1/2 | k:k =−1 k Xk | > λ) ≤ E P (n−1/2 |

(b)

k=1

k:k =1

≤ 2 supk≤n P (n−1/2 | ≤ 2 supk≤n P (k−1/2 |

(c)

k 1

k

≤ 2 supk≤∞ P (k−1/2 |

as required.

Xk | > λ) + E P (n−1/2 |

1

k:k =−1

Xk | > λ)

Xk | > λ) Xk | > λ)

k 1

Xk | > λ) ,

2

Exercise 3.3 (Khinchin inequality) Suppose 1 , . . . , n are iid Rademacher rvs. Let a1 , . . . , an be real constants. For p ≥ 1 it holds that n 2 1/2 n n 2 1/2 p 1/p (11) Ap ≤ E | 1 ak k | ≤ Bp 1 ak 1 ak √ for some constants Ar and Br . Establish this for p = 1, with A1 = 1/ 3 and B1 = 1. [Hint. Use Littlewood’s inequality with r, s, t equal to 4, 2, 1.] Exercise 3.4 Let X1 , . . . , Xn be independent with 0 means, and independent of the iid Rademacher rvs 1 , . . . , n . Let φ be and convex on R. Then n n n (12) Eφ(| 1 k Xk |/2) ≤ Eφ(| 1 Xk |) ≤ Eφ(2| 1 k Xk |) . [Hint. The left side is an average of terms like Eφ(| n1 ek (Xk − EXk )|/2), for independent Xk ∼ = Xk and with each ek equal to ±1.]

214

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Maximal Inequalities of Another Ilk Discussion 3.1 (Weak negligibility) Let Yn1 , . . . , Ynn be independent with dfs Fn1 , . . . , Fnn . Let θ > 0 be given. Deﬁne xθn by requiring [−xθn , xθn ] to ¯n ≡ n Fnk /n assigns be the smallest closed and symmetric interval to which F 1 n probability at least 1 − θ/n. Let P¯n (x) ≡ n1 1 P (|Ynk | > x) denote the average tail probability, and then let Kn denote the qf of the df 1 − P¯n (·). Note the quantile relationship xθn = K n (1 − θ/n). For any > 0, let pnk ≡ P (|Ynk | > ). Now, the maximum Mn ≡ [max1≤k≤n |Ynk | ] satisﬁes n n n (13) 1 − exp(− 1 pnk ) ≤ P (Mn > ) = 1 − 1 (1 − pnk ) ≤ 1 pnk . [The equality is simple, and the bounds follow from it using ﬁrst 1 − x ≤ exp(−x) and then ∩n1 Ack = [∪n1 Ak ]c .] This gives the standard result that n (14) if and only if nP¯n ( ) = 1 pnk → 0 for all > 0. Mn →p 0 Since K n (1 − θ/n) = inf{x : 1 − P¯n (x) ≥ 1 − θ/n} = inf{x : P¯n (x) ≤ θ/n}, we have (15)

nP¯n ( ) ≤ θ

K n (1 − θ/n) ≤ .

if and only if

Suppose 0 < ≤ 1 and θ > 0 are ﬁxed, and that we are considering all n exceeding some n,θ . Thus conclusions (14) and (15) give (the seemingly new emphasis) (16)

Mn →p 0

xθn = K n (1−θ/n) → 0 for all θ > 0. 2

if and only if

Discussion 3.2 (Weak negligibility in the LLN context) Let νn > 0 be constants. Applying the previous paragraph to the rvs |Ynk |/n νn , whose θ/nth quantile is xθn /n νn , gives the equivalencies (17)

Mn ≡ maxk |Ynk |/n νn →p 0 ,

(18)

xθn /n νn → 0 n 1 P (|Ynk |/n νn > ) → 0

(19)

for all θ > 0 , for all 0 < ≤ 1 .

Useful choices for νn might be the truncated absolute moment [ |y|≤x1n ] |y| dF¯n (y) and the Winsorized absolute moment ν1n + x1n P¯n (x1n ). (Here x1n equals xθn with θ = 1.) 2 Inequality 3.6 (Daniels’ equality) With high probability there is a linear bound on the uniform empirical df Gn . Thus, for each 0 < λ < 1, (20)

Proof. (a) (b)

(c)

P (Gn (t) ≤ t/λ for 0 ≤ t ≤ 1) = P (ξn:k ≥ λk/n for 1 ≤ k ≤ n) = 1 − λ . Now, (ξn:1 , . . . ξn:n ) has joint density n! on its domain. Thus P (Gn (t) ≤ t/λ for 0 ≤ t ≤ 1) = P (ξn:k ≥ λk/n for 1 ≤ k ≤ n) 1 tn t3 t2 = λ λ(n−1)/n · · · λ2/n λ/n n! dt1 · · · dtn = ··· = n = n! tn! −

λtn−1 n!

1 = 1−λ. λ

2

4.

THE CLASSICAL LAWS OF LARGE NUMBERS, LLNS

215

4

The Classical Laws of Large Numbers, LLNs

It is now time to present versions of the laws of large numbers under minimal hypotheses. The weak law of large numbers (WLLN) will establish →p , while the ¯n. strong law of large numbers (SLLN) will establish →a.e. of a sample average X Theorem 4.1 (WLLN; Feller) Let Xn1 , . . . , Xnn be iid with df F and qf K, ¯ n ≡ (Xn1 + · · · + Xnn )/n. The following are equivalent: for each n. Let X (1)

¯ n − µn →p 0 X

for some constants µn .

(2)

τ (x) ≡ xP (|X| > x) → 0

as x → ∞ .

(3)

t{|F+−1 (t)| + |F −1 (1 − t)|} = t{|K+ (t)| + |K(1 − t)|} → 0

(4)

Mn ≡ [ n1 max |Xnk | ] →p 0.

as t → 0 .

1≤k≤n

1−1/n K(t) dt When (1) holds, possible choices include µn ≡ [−n,n] x dF (x), νn ≡ 1/n ¯ and median(Xn ). If E|X| < ∞, then (1) holds and µn = µ ≡ EX works. Theorem 4.2 (SLLN; Kolmogorov)

Suppose X, X1 , X2 , . . . are iid. Then:

(5)

E|X| < ∞

implies

¯ n →a.s. µ ≡ EX. X

¯ n →L1 EX.] [Moreover, X

(6)

E|X| = ∞

implies

¯ n | =a.s. ∞. limn→∞ |X

(7)

E|X| < ∞

if and only if

¯ n | < ∞ a.s. lim |X

(8)

E|X| < ∞

if and only if

Mn ≡ [ n1 max |Xk | ] →a.s. 0. 1≤k≤n

Conditions (4) and (8) show the sense in which these LLNs are tied to the size of the maximal summand. This is an important theme, and the reader should should not lose sight of it. We now give a symmetric version of condition (4). Theorem 4.3 (The maximal summand) Let Xn1 , . . . , Xnn , n ≥ 1, be iid row independent rvs with df F . We then let Xnk ≡ Xnk −Xnk denote their symmetrized versions. Fix r > 0. [Most important is r = 1.] Then (9) (10)

τ (x) ≡ xr P (|X| > x) → 0 iﬀ τ s (x) ≡ xr P (|X s | > x) → 0 iﬀ max |Xnk − a|/n1/r →p 0 for each real a iﬀ

1≤k≤n

s max |Xnk |/n1/r →p 0.

1≤k≤n

Proof. We ﬁrst consider the SLLN. Let Yn ≡ Xn × 1[ |Xn |
iﬀ the Xn ’s and Yn ’s are Khinchin equivalent rvs.

∞

[1 – F (y)]dy → 0

x[1 –F(x)]

∫x

x

[1 – F (y)]dy

∫ t1 F –1(s)ds → 0.

∞ x

∫

iff

iff (1 – t)F –1(t) → 0.

F (·)

0

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

t

F –1(·)

(1 – t)F –1(t)

1

∫ t F –1(s)ds

These two sum to

Figure 4.1 Conditions for the WLLN and for the SLLN. (Use (a∨b) ≤ a+b ≤ 2(a∨b) for general X.)

0

SLLN iff

WLLN iff x[1 – F (x)] → 0

Let X ≥ 0 have df F (·).

216

1

4.

THE CLASSICAL LAWS OF LARGE NUMBERS, LLNS

217

¯ n →a.s. µ if and only if Y¯n →a.s. µ. Now let µn ≡ EYn and σn2 ≡ Var[Yn ]. Thus X Then we can write n (c) ¯n , where µ ¯n ≡ (µ1 + · · · µn )/n → µ Y¯n = [ 1 (Yi − µi )/n] + µ by the Ces` aro summability exercise 10.1.1 (since µn = EYn = (−n,n) x dF (x) → µ). n It thus suﬃces to show that 1 (Yi − µi )/n →a.s. 0. Thus by Kronecker’s lemma it suﬃces to show that (d) Zn ≡ ni=1 (Yi − µi )/i →a.s. (some rv Z). But Zn →a.s. (some Z), by proposition 2.3.3, if for all > 0 we have (e)

pnN ≡ P (maxn≤m≤N |Zm − Zn | ≥ ) m = P (maxn≤m≤N | i=n+1 [(Yi − µi )/i] | ≥ ) → 0.

Then Kolmogorov’s inequality yields (e) via N N pnN ≤ −2 n+1 Var[(Yi − µi )/i] ≤ −2 n+1 σi2 /i2 2 2 (f) ≤ −2 ∞ for all N n+1 σi /i →0

as n → ∞,

provided that ∞ ∞ 2 2 2 (g) n=1 σn /n = n=1 Var[Yn ]/n < ∞. Now, this last is seen to be true via the Kolmogorov type argument ∞ ∞ ∞ 2 2 2 2 2 1 Var[Yn − µn ]/n ≤ 1 EYn /n = 1 [|x|
= 2 E|X| < ∞.

¯ n →a.s. µ. Thus we do have Zn →a.s. (some rv Z), and so X Assume E|X| = ∞. Then inequality 10.2.1 gives (for any large constant C) ∞ ∞ (l) n=0 P (|Xn | ≥ nC) = n=0 P (|X|/C ≥ n) ≥ E|X/C| = ∞ for all C > 0, so that applying the second Borel–Cantelli lemma to (l) gives (m)

P (|Xn | ≥ nC i.o.) = 1

for all C > 0.

Since Sn = Sn−1 +Xn , (m) implies (using the fact that |Sn | < nC/2 and |Xn | ≥ nC implies |Sn−1 | > nC/2 > (n − 1)C/2) that (n)

P (|Sn | ≥ nC/2 i.o.) = 1

for all C > 0.

That is, lim |Sn |/n ≥ C/2 a.s. for all C. That is, lim |Sn |/n = ∞ a.s.

218

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES ¯ n →L1 µ in (5).] Thus (5)–(7) hold. [Apply Vitali with exercise 4.16 below for X

Consider (8). Suppose Mn →a.s. 0. Then a.s. for all n ≥ (some nω ) we have (o)

[ max1≤k≤n |Xk | ] /n < ,

and hence

|Xn |/n < .

We merely repeat this last statement, writing (p)

An ≡ [ |Xn |/n ≥ ]

satisﬁes

P (An i.o.) = P (|Xn |/n ≥ i.o.) = 0.

Thus inequality 10.2.1 (by applying iid, and then the second Borel–Cantelli) gives ∞ ∞ (q) E|X|/ = E|X/ | ≤ n=0 P (|X/ | ≥ n) = n=0 P (|Xn |/n ≥ ) < ∞. Conversely, suppose E|X| < ∞. Then Xn /n →a.s. µ by the SLLN. Since Xn Sn − nµ n − 1 Sn−1 − (n − 1)µ µ (r) = − + →a.s. 0 − 1 · 0 + 0 = 0, n n n n−1 n we have a.s. that (s)

|Xn |/n ≤

for all n ≥ (some nω ).

Thus for all n exceeding some even larger nω we have 1 |Xk | k |Xk | (11) max ≤ max · ≤ max |Xk | ∨ max 1≤k≤n n 1≤k≤n n k≥nω k n 1≤k≤nω (t)

≤ n−1 [a ﬁxed number depending on ω] + ≤ 2

|Xk | k

using (s),

where we will have to increase the speciﬁcation on nω for (t). Thus Mn →a.s. 0. From (7.4.13) we see (note ﬁgure 4.1) that 1 −1 ∞ (12) P (|X| > x) dx < ∞ iﬀ |F (t)| dt < ∞ . 2 E|X| < ∞ iﬀ 0 0 Suppose X1 , . . . , Xn are independent, with Xi ∼ = (0, σi2 ). Then n ∼ (0, n σ2 ), while X ¯ n ≡ Sn /n ∼ Sn ≡ X1 + · · · + Xn = = (0, 1 σi2 /n2 ). 1 i

Remark 4.1

Chebyshev’s inequality and Kolmogorov’s inequality give n (a) P (|Sn | ≥ λ) ≤ Var[Sn ]/λ2 = i=1 σi2 /λ2 for all λ > 0, (13) (b) P (max1≤k≤n |Sk | ≥ λ) ≤ Var[Sn ]/λ2 for all λ > 0. ¯ n →p µ, by Chebyshev’s For X1 , X2 , . . . iid (µ, σ2 ), the inequality (13)(a) gives X ¯ inequality. But the WLLN conclusion Xn →p µ should not require the variance σ2 to be ﬁnite, as this cheap proof based on (13)(a) requires. Indeed, Khintchine’s WLLN of theorem 10.2.1 didn’t. Exercise 4.8 below outlines one very cheap proof of the SLLN using “only” the Borel–Cantelli lemma, and exercise 4.9 outlines a slightly improved version that also uses Kolmogorov’s inequality. Kolmogorov’s proof of the full SLLN made the key step of incorporating truncation. Exercise 4.10 describes an elementary way to avoid use of Kronecker’s lemma. 2

4.

THE CLASSICAL LAWS OF LARGE NUMBERS, LLNS

219

Proof. Consider the WLLN. Suppose (2) holds. Deﬁne Ynk ≡ Xnk × 1[|Xnk |≤n] and µn ≡ EYnk = [−n,n] x dF (x). Now, ¯ n − µn | ≥ P (|X

√

¯ n − EY¯n | ≥

) = P (|X

(14)

≤ P (|Y¯n − EY¯n | ≥

(15)

≤

(a)

1 = − n

1

√

) +

n

1 P (Ynk

Var[Y¯n ]+nP (|X| > n) ≤ [0,n]

1 n

√

)

= Xk )

2 E Yn1 + τ (n) (Truncation inequality)

x2 dP (|X| > x) + τ (n)

n+ = −1 −n−1 x2 P (|X| > x)0− + n−1 [0,n] P (|X| > x) 2x dx + τ (n) using integration by parts = 1 [−τ (n) + 0 + 2 n−1 [0,n] τ (x) dx] + τ (n) (16)

≤

2 n

τ (x) dx [0,n]

for 0 < ≤ 1. Note that τ (x) ≤ x, and choose M > 0 so large that τ (x) < 2 /4 for x > M . Applying these to (16) gives n M n 2 2 2 M2 (b) n 0 τ (x) dx ≤ n { 0 τ (x) dx + M 4 dx} ≤ n + 2 ≤

for all n ≥ 2M 2 / 2 . Combining (16) and (b), it follows that √ ¯ n − µn | ≥ ) ≤

(c) P (|X for n ≥ (some N ) . ¯ n −µn →p 0. We also have X ¯ n −median(X ¯n ) →p 0, since the symmetrization Thus X inequality 10.3.2 gives (d)

¯ n )| ≥ ) ≤ 4P (|X ¯ n − µn | ≥ /2) → 0. ¯ n − median(X P (|X

[The acceptability of the third exhibited choice for the centering constant is left to exercise 4.1 below.] In any case, we have shown that (2) implies (1). The equivalence of (2) and (3) follows as in the proof of theorem 7.8.1. Note also ﬁgure 4.1, bearing in mind that (a ∨ b) ≤ a + b ≤ 2(a ∨ b). Consider the equivalence of (4) and (2). We know from (10.3.14) that Mn →p 0 if and only if nP (|X| > n) → 0, that is, if and only if τ ( n) = nP (|X| > n) → 0 for all > 0. Thus Mn →p 0 if and only if τ (x) → 0 as x → ∞. We still neeed to show that (1) implies (2), but we’ll wait a paragraph for this. Consider next theorem 4.3. We need only provide a proof with r = 1, as we may just replace |X| by |Y | ≡ |X|r , after raising |X| to the power r. Now, τX (x) → 0 implies τ s (x) → 0 by the right-hand side of inequality 10.3.2 with a = 0, while the left-hand side then gives τX−med (x) → 0. The equivalence of (4) and (2) then gives max |Xnk − med|/n →p 0, which trivially gives max |Xnk | →p 0, which gives τX (x) → 0 by the equivalence of (4) and (2). We are done with theorem 4.3.

220

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Finally, we prove that (1) implies (2). Suppose that there exists some constant ¯ n − µn →p 0. Let S s = Sn − S , where S ≡ µn such that Sn /n − µn = X n n n ∼ Xn1 + · · · + Xnn with Xnk = Xnk and with Xnk ’s and Xnk ’s independent. Thus s Sns /n →p 0. Then Mn ≡ max1≤k≤n |Xnk |/n →p 0 by the (10.3.4) L´evy inequality. Thus τ (x) → 0 by theorem 4.3. 2 Exercise 4.1 Verify that the choice νn ≡ WLLN as a centering constant in (1).

1−1/n 1/n

K(t) dt also works in the

We have just seen that good inequalities lead to good theorems! In sections 10.9 and 12.11 we will add to our growing collection of good inequalities. Some will be used in this text, and some will not. But the author thinks it important to illustrate these possibilities. ¯n| = ∞ Exercise 4.2 When E|X| = ∞, the SLLN above showed that limn |X a.s. Show the following stronger result. If X1 , X2 , . . . are iid with E|X| = ∞, then ¯ n − cn | =a.s. ∞ limn→∞ |X for every sequence of constants cn . Exercise 4.3 (Erickson)

If EX − < ∞ but EX + = ∞, then lim Sn /n =a.s. +∞.

Exercise 4.4 (Marcinkiewicz–Zygmund) Let X1 , X2 , . . . be iid. Let 0 < r < 2. Establish the equivalence (17)

E|X|r < ∞

if and only if

n

n 1 (Xk − c) →a.s. 0 for some c. 1/r k=1

If so, then c = EX when 1 ≤ r < 2, while c is arbitrary (so c = 0 works) when 0 < r < 1. [Hint. Truncate via Yn ≡ Xn × 1[|Xn |≤n1/r ] .] Let X1 , X2 , . . . be iid with E|X| = ∞. If an /n ↑, then ∞ = 0 a.s., < ∞, (18) lim |Sn |/an = according as P (|Xn | ≥ an ) = n=1 = ∞ a.s., = ∞. ∞ [Note that P (|Xn | ≥ an i.o.) equals 0 or 1 according as 1 P (|Xn | ≥ an ) is ﬁnite or inﬁnite.] Exercise 4.5 (Feller)

Exercise 4.6

Clarify the overlap between (17) and (18).

Exercise 4.7 (Random sample size) (a) Suppose X1 , X2 , . . . are iid and (0, σ2 ). Let Nn ≥ 0 be any positive integer-valued rv satisfying Nn /n →p c ∈ (0, ∞). Then (19) SNn /Nn − µn →p 0, for µn ≡ [−n,n] x dF (x). (b) Suppose X1 , X2 , . . . are iid and µ ≡ EX is ﬁnite. Let Nn ≥ 0 be any positive integer-valued rv satisfying Nn /n →a.s. c ∈ (0, ∞). Then (20)

SNn /Nn →a.s. µ.

4.

THE CLASSICAL LAWS OF LARGE NUMBERS, LLNS

221

Exercise 4.8 Let X1 , X2 , . . . be independent rvs with means 0 and suppose that all EXk4 ≤ (some M ) < ∞. ¯ n ≡ Sn /n ≡ (X1 + · · · + Xn )/n satisﬁes (a) (Cantelli’s inequality) Verify that X P (|Sn | ≥ λ) ≤ 3M n2 /λ4

for all λ > 0.

¯ n →a.s. 0. (b) (A very weak SLLN) Show that under these circumstances X Exercise 4.9 (A weak SLLN) If X1 , X2 , . . . are independent and Xk ∼ = (0, σk2 ) 2 ¯ with σk ≤ (some M ) < ∞ for all k, then Xn →a.s. 0 as n → ∞. Prove this by ¯ n2 | = |Sn2 /n2 | > 0 i.o.) = 0, and showing P (|Sn | ≥ n ) ≤ M/(n 2 ) so that P (|X then showing that the “block maximum” ∆n ≡ maxn2 i.o.) = 0. Exercise 4.10 (Alternative proof of the SLLN) Apply either the H´ ajek–R´enyi inequality (inequality 10.11.3) or the monotone inequality (inequality 10.11.1) to replace use of the Kronecker lemma in the SLLN proof. Exercise 4.11 (St. Petersburg paradox) Let X1 , X2 , . . . be iid rvs for which P (X = 2m ) = 1/2m for m ≥ 1. Show that (Sn − bn )/an →p 0 for bn ≡ n log2 n and an ≡ n log2 n (where log2 · is used here to denote logarithm to the base 2). Exercise 4.12 (Spitzer) (21)

EX = 0

Show that ∞ if and only if n=1 P (|Sn | ≥ n )/n < ∞ for all > 0.

Exercise 4.13 If X1 , X2 , . . . are iid Exponential(1), then lim Xn / log n = 1 a.s. and Xn:n / log n → 1 a.s. Exercise 4.14

√ If X1 , X2 , . . . are iid N(0, 1), then Xn:n / 2 log n →p 1.

Exercise 4.15 (a) Does the WLLN hold for the Cauchy distribution? (b) Does the WLLN hold when P (|X| > x) = 2/[x log x] for x ≥ e, X symmetric? (c) Make up one more example of each of these two types. Exercise 4.16 (Uniform integrability of sample averages) Let X1 , X2 , . . . be iid, ¯ n ≡ (X1 +· · ·+Xn )/n. Then the rvs {Xn : n ≥ 1} are uniformly integrable and let X ¯ n : n ≥ 1} are uniformly integrable. if and only if the rvs {X Exercise 4.17 (a) Let row independent rvs Xn1 , . . . , Xnn be iid with the df F (·) (having mean µ) for all n ≥ 1. If Mn ≡ [max1≤k≤n |Xnk |/n] →p 0 (and using the fact that EMn ≤ E|X|), show that (22)

EMn = E[max1≤k≤n |Xnk |/n] → 0.

(b) For iid X1 , X2 , . . . show that E|X| < ∞ if and only if Mn →L1 0; see theorem 4.2.

222

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Generalizations of the LLNs Our results allow simple generalizations of both the WLLN and SLLN. Theorem 4.4 (General WLLN and SLLN) Let X1 , X2 , . . . be independent. Then n 2 2 n (23) implies 1 σk /bn → 0 1 (Xk − µk )/bn →p 0, ∞ 2 2 n (24) implies 1 σk /bk < ∞ with bn ∞ 1 (Xk − µk )/bn →a.s. 0. Proof. The ﬁrst claim is immediate from Chebyshev’s inequality. Also, (f) in the SLLN proof shows that ∞ 2 2 n (a) implies 1 σk /bk < ∞ 1 (Xk − µk )/bk →a.s. (some rv S). n Then Kronecker’s lemma gives 1 (Xk − µk )/bn →a.s. 0. 2

5.

APPLICATIONS OF THE LAWS OF LARGE NUMBERS

223

5

Applications of the Laws of Large Numbers

Let X1 , X2 , . . . be iid F . Let Fn denote the empirical df of X1 , . . . , Xn , given by (1)

Fn (x) ≡ Fn (x, ω) ≡

1 1 1(−∞,x] (Xk (ω)) = 1[Xk ≤x] n n n

n

k=1

k=1

Theorem 5.1 (Glivenko–Cantelli) (2)

Fn − F ≡

sup

−∞<x<∞

for all real x.

We have

|Fn (x) − F (x)| →a.s. 0

as n → ∞.

[This is a uniform SLLN for the random function Fn (·).] Proof. Let Xk = F −1 (ξk ) for k ≥ 1 be iid F , with the ξk ’s iid Uniform(0, 1). Let Gn denote the empirical df of the ﬁrst n of these ξk ’s, and let Fn denote the empirical df of the ﬁrst n of these Xk ’s. Let I denote the identity function. Then (3)

(Fn − F ) = [Gn (F ) − I(F )]

on (−∞, ∞)

for every ω

by (7.3.4). Thus by theorem 5.3.3, it will suﬃce to prove the result in the special case of uniform empirical df’s Gn ’s. (Recall the remark in bold above (7.4.3) that the representation of X as F −1 (ξ) allows alternative ways to approach problems.) Now, Gn (k/M ) − k/M →a.s. 0 as n → ∞ for 0 < k ≤ M by the SLLN applied to the iid Bernoulli(k/M ) rv’s 1[0,k/M ] (ξi ). We now assume that M is so large that 1/M < . Then for (k − 1)/M ≤ t ≤ k/M , with 1 ≤ k ≤ M , we have both k k−1 k k 1 (a) Gn (t) − t ≤ Gn M +M and − M ≤ Gn M −M (b)

Gn (t) − t ≥ Gn

k−1 M

−

k M

≥ Gn

k−1 M

−

k−1 M

−

1 M.

These combine to give (c) (d)

k 0 sup0≤t≤1 |Gn (t) − t| ≤ max0≤k≤M Gn M −

1

k M

+

1 M

→a.s. 0 + 1/M < .

Since > 0 is arbitrary, we have shown that sup0≤t≤1 |Gn (t) − t| →a.s. 0. That is, (4)

Fn − F = Gn (F ) − F ≤ Gn − I →a.s. 0 ,

as claimed.

2

Example 5.1 (Weierstrass approximation theorem) If f is continuous on [0, 1], then there exist polynomials Bn such that Bn − f = sup0≤t≤1 |Bn (t) − f (t)| → 0 as n → ∞.

224 Proof. (5)

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES (Bernstein) Deﬁne the Bernoulli polynomials n k n k Bn (t) = f t (1 − t)n−k for 0 ≤ t ≤ 1 n k k=0

(a)

= Ef (T /n)

where T ∼ = Binomial(n, t).

Since f is continuous, f is bounded by some M , and f is uniformly continuous on [0, 1] having |f (x) − f (y)| < whenever |x − y| < δ . Then |f (t) − Bn (t)| = nk=0 [f (t) − f (k/n)] nk tk (1 − t)n−k ≤ | {k:|k/n−t|<δ } same | + | {k:|k/n−t|≥δ } same | (b)

< + 2M P (|T /n − t| ≥ δ ) ≤ + 2M t(1 − t)/nδ2

(c)

≤ + 2M/4nδ2 ≤ 2

by uniform continuity of f

for all t for n ≥ some N .

Note that this is an application of the WLLN (that is, of Chebyshev’s inequality). 2

Example 5.2 (Borel’s normal numbers) A number x in [0, 1] is called normal to base d if when expanded to base d, the fraction of each of the digits 0, . . . , d − 1 converges to 1/d. The number is normal if it is normal to base d for each d > 1. We are able to conclude that a.e. number in [0, 1] is normal with respect to Lebesgue measure λ. [Comment: 13 = 0.010101 . . . in base 2 is normal in base 2, but 13 = 0.1000 . . . in base 3 is not normal in base 3.] [This was a historically important example, which spurred much of the original development.] Proof. (a)

Let (Ω, A, P ) = ([0, 1], B ∩ [0, 1], λ). Let ∞ ω = n=1 βn (ω)/dn deﬁne rvs β1 , β2 , . . . .

Note that the βn ’s are iid discrete uniform on 0, 1, . . . , d − 1. Thus, letting ηnk = 0 or 1 according as βn = k or βn = k, we have n (b) λ(Ad,k ) ≡ λ({ω : n−1 j=1 ηjk → 1/d}) = 1 by the SLLN. Thus Ad ≡ ∩d−1 k=0 Ad,k has λ(Ad ) = 1; that is, a.e. ω in [0, 1] is normal to base d. Then trivially, A ≡ ∩∞ d=1 Ad has λ(A) = 1. And so, a.e. ω in [0, 1] is normal. 2

5.

APPLICATIONS OF THE LAWS OF LARGE NUMBERS

225

Example 5.3 (SLLN for random sample size) Let Nn be positive-integer valued rvs for which Nn /n →a.s. c ∈ (0, ∞), and let X1 , X2 , . . . be iid with mean µ. (a) Then (6)

SNn /n →a.e. µc

as n → ∞.

(b) If X1 , X2 , . . . are iid Bernoulli(p) and Nn (ω) ≡ min{k : Sk = n}, then the waiting times Nn satisfy Nn /n →a.s. 1/p. Proof. (a) Now, Sn /n →a.s µ by the SLLN, and thus Nn →a.s. ∞ implies SNn /Nn →a.s. µ. Thus SNn /n = (SNn /Nn ) (Nn /n) →a.s. µc, using Nn →a.s. ∞ by c > 0. (b) We also have 1 = SNn /n = (SNn /Nn ) (Nn /n),

so

Nn /n = 1/(SNn /Nn ) →a.s. 1/p,

completing the proof. Note that we could also view Nn as the sum of n iid Geometric(p) rvs, and then apply the SLLN. 2 Exercise 5.1 (Monte Carlo estimation) Let h : [0, 1] → [0, 1] be continuous. (i) Let Xk ≡ 1[h(ξk )≥Θk ] , where ξ1 , ξ2 , . . . Θ1 , Θ2 , . . . are iid Uniform(0, 1) rvs. Show that this sample average is a strongly consistent estimator of the integral; that is, ¯ n →a.s. 1 h(t) dt. show that X 0 1 (ii) Let Yk ≡ h(ξk ). Show that Y¯n →a.s. 0 h(t) dt. ¯ n ] and Var[Y¯n ]. (iii) Evaluate Var[X

226

6

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

General Moment Estimation

Notation 6.1 (General case on [0, ∞)) Let Yn1 , . . . , Ynn denote row indepenn dent rvs having dfs Fn1 , . . . , Fnn . Let P¯n (x) ≡ 1 P (|Ynk | > x)/n. Let the qf K n (·) denote the inverse of the df 1 − P¯n (x). Let xθn denote the θ/nth quantile of the df 1 − P¯n . [Thus x1n ≡ K n (1 − 1/n) is the upper (1/n)-quantile of 1 − P¯n , while [−x1n , x1n ] is the shortest interval containing at least 1 − 1/n of the probability.] r r + · · · + Xnn )/n. Let the Let Xnk ≡ |Ynk | for 1 ≤ k ≤ n. Fix r > 0. Consider (Xn1 rth partial absolute moment, or truncated absolute moment, Un (·) be deﬁned by (1) |y|r dF¯n (y) for each x > 0 . Un (x) ≡ [ |y|≤x]

Let uθn ≡ Un (xθn ). The heaviness of the tails of F¯n (·) is measured with Rn (·) deﬁned by (2) Rn (x) ≡ xr P¯n (x)/ [ |y|≤x] |y|r dF¯n (y) for each x > 0. 2 Theorem 6.1 (Consistency of the absolute moments) (A) Conditions (4)–(7) are equivalent, and imply (3), where (3)

Xnr / u1n →p 1,

where u1n = Un (x1n ) ,

(4)

xrθn /n uθn → 0

for each 0 < θ ≤ 1 ,

(5)

for each 0 < θ ≤ 1 ,

(6)

xrθn /n u1n → 0 n r 1 P (Xnk /n u1n > ) → 0

(7)

[ max1≤k≤n

1 n

for each 0 < ≤ 1 .

r Xnk ] / u1n →p 0 .

(B) Fix θ > 0. Then Rn (xθn −) → 0 implies xrθn /n uθn → 0, as in (4). (C) The Winsorized moment u ˜1n ≡ u1n + xr1n P¯n (x1n ) satisﬁes u ˜1n /u1n → 1 when condition (5) holds. Corollary 1 Suppose Ynk ≡ Xnk × 1[−n,n] (Xnk ) satisﬁes ¯ n →p 1. and EY¯n → 1 and Var[Y¯n ] → 0. Then X Exercise 6.1

n 1

P (Ynk = Xnk ) → 0,

Prove the previous corollary. Can you prove a converse?

Proof. It suﬃces to prove this theorem for r = 1 (reduce the problem by transforming from X to X r ). Fix θ > 0. Let µθn ≡ [ |y|≤xθn ] |y| dF¯n (y) = Un (xθn ). Also, (5)–(7) were shown equivalent in (10.3.17)–(10.3.19). Now, (5) trivially implies (4) (since uθn ≥ u1n ), and the converse holds, since (a)

xθn xθn (xθn /n µ1n ) ≥ = . n µθn n [µ1n + xθn /n] 1 + (xθn /n µ1n )

6.

GENERAL MOMENT ESTIMATION

227

Consider (5) implies (3) next. Deﬁne Znk to equal Xnk if Xnk ≤ xθn and to equal 0 otherwise. Note that EZ¯n = µθn . Fix > 0. Then for some ﬁxed tiny θ ≡ θ , we have 1≥ (b)

µ1n + n1 xθn − µ1n µθn − µ1n xθn µ1n ≥1− ≥1− =1− µθn µθn µθn n µθn

≥1−

xθn ≥ 1− . n µ1n

Also, the truncation inequality (10.4.15) (that is, the steps (c) and (d) below) gives ¯ n − µθn | > µθn ) ≤ P (|Z¯n − µθn | > µθn ) + n P (Xnk = Znk ) P (|X (c) 1 n 2 ≤ 1 E(Znk (d) )/( 2 µ2θn n2 ) + nP¯n (xθn ) = [ |y|≤xθn ] y 2 dF¯n (y)/( 2 n µ2θn ) + nP¯n (xθn )

|y| dF¯ (y)/( 2 n µ2θn ) + θ

(e)

≤ xθn

(f)

= −2 {xθn /nµθn } + θ ≤ −2 {xθn /nµ1n } + θ ≤ 2

[ |y|≤xθn ]

for all n exceeding some suﬀﬁciently large n (by (5)). Thus for n large enough, | Xnr − µ1n | µθn Xnr − µθn µ1n 1 (g) = · +1− ≤ [ + ] µ1n µ1n µθn µθn 1 −

with probability at least 1 − 2 . Thus (3) holds. Now (C) is trivial, since nP¯n (x1n ) ≤ 1. Observe that (h)

P¯n (x1n ) ≤ 1/n = θ −1 (θ/n) ≤ θ−1 P¯n (xθn −) ,

and hence (B) holds via (i)

xθn 1 xθn P¯n (xθn −) 1 ≤ = Rn (xθn −) → 0 . n µθn θ [ |y|≤xθn ] |y| dF¯n (y) θ

2

Notation 6.2 (Absolute moments in the iid case) Let Y and all rvs Ynk now have common df F (·) ≡ FY (·) and qf K(·) ≡ KY (·). [Let X ≡ |Y |. The rvs Xnk ≡ |Ynk | may be represented as KX (ξnk ) for row-independent Uniform(0, 1) rvs r ξn1 , . . . , ξnn .] Then P (·) ≡ 1 − FX (·), and deﬁne m(·) ¯ ≡ KX (1 − ·). The rth partial ¯ absolute moment, or truncated absolute moment, M(·) and U (·) are deﬁned by r ¯ (t) ≡ M KX (s) ds on [0, 1], U (x) ≡ [0,x] y r dFX (y) on [0, ∞). [0,1−t) ¯ (θ/n) and uθn ≡ U (xθn ), where xθn ≡ KX (1 − θ/n) is deﬁned to be Let m ¯ θn ≡ M the (1 − θ/n)th quantile. The Winsorized absolute moment ˜¯ 1n ≡ m ¯ 1n + n1 m(1/n) ¯ equals u˜1n ≡ u1n + x1n P (x1n ) , m even though the lead terms may not be equal.

2

228

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Theorem 6.2 (Consistency of absolute moments in the iid case) rvs the conditions (3)–(15) are equivalent, where

For iid

(8)

Xnr / νn →p 1

for some constants νn > 0 with n νn ,

(9)

(a) Xnr / m ¯ 1n →p 1 ,

(b) Xnr / u ˜ 1n →p 1 ,

(10)

R(x) ≡ xr P (x)/U (x) = xr P (X > x)/ [0,x] y r dFX (y) → 0 or R(x−) → 0,

(11)

¯ (t) = t K r (1 − t)/ r¯(t) ≡ t m(t)/ ¯ M X

(12)

r¯(an ) → 0

for any one an 0 with lim(an+1 /an ) < ∞ ,

(13)

n P (X r > n m ¯ 1n ) → 0

for all > 0 ,

(14)

[ max1≤k≤n

1 n

r Xnk ]/m ¯ 1n →p 0 ,

(15)

[ max1≤k≤n

1 n

r Xnk ] / Xnr →p 0 .

[0,1−t)

(c) Xnr / u1n →p 1 ,

r KX (s) ds → 0

as t → 0 ,

˜¯ 1n /νn → 1 under (8). Of course, m ¯ 1n /νn → 1, u1n /νn → 1, and u ˜1n /νn = m Proof. Without loss of generality, set r = 1. Suppose (8). Let c > 1. Note ¯ n /νn ≥ c) → 0 ﬁrst that P (maxk Xnk < cnνn )= P (all Xnk < cnνn ) → 1; else P (X is true, and (8) fails. But the conclusion P (maxk Xnk < cnνn ) → 1 implies by (10.3.19) that we have (the general background result) that for all c > 1, n (a) 1 P (Xnk ≥ c nνn ) → 0 {and P ([maxk Xnk /n νn ] ≥ c) → 0} . Consider (8) implies (10). Let > 0 be tiny. Let c > 1. Truncate the Xnk ’s via (b) Znk ≡ Xnk 1[Xnk
[0, c n νn )

x2 dFX (x) ≤ c n νn µn we obtain

¯ n − µn | > νn ) ≤ Var[Z¯n ]/( νn )2 + nP (c n νn −) P (|X (c)

≤ (c/ 2 ) µn /νn + nP (c n νn −) ,

where nP (c n νn −) → 0 by (a). We next show that nP (c n νn −) → 0 implies that (d)

lim µn /νn > 0

(equivalently, lim νn /µn < ∞).

Suppose µn /νn → 0 were true on some subsequence n . Then (c) would give (e)

¯ n /νn − µn /νn = X ¯ n /νn − o(1) →p 0 X

but this is a contradiction. Thus (d) holds. Thus

(or

¯ n /νn →p 0) ; X

6. (f)

GENERAL MOMENT ESTIMATION

229

c n νn P (c n νn −) lim ≤ c [ lim (νn /µn )] [ lim nP (c n νn −)] [0, c n νn ) y dFX (y) = O(1) o(1) → 0 .

So, (10) holds on the subsequence xn = cnνn . Thus [as n P (Xn,n+1 /n νn ≥ c) → 0 follows from equation (a)], (g)

¯ ¯ n+1 X n νn Xn Xn,n+1 νn = + {1 + 2 + op (1)} ≤ 1 + op (1) = νn+1 n + 1 νn+1 νn n νn νn+1

shows that lim νn+1 /νn ≤ 3. Thus (since nνn ) sup

(h)

[c n νn ≤x≤c (n+1) νn+1 ]

xP (x−) ≤ y dFX (y) [0, x)

n+1 n

c νn+1 c n νn P (c n νn −) →0 c νn y dFX (y) [0, c n νn )

by (f), so that (10) holds. In fact, the stronger R(x−) → 0 was just veriﬁed. We appeal also to theorem 6.1(A)(B). It has now been shown that (8) implies (10) with R(·−), which implies (4), which implies (3), which implies (9)(c), which implies (8). Purely geometric arguments in section 7.7 show that (10), (11), and (12) are equivalent. Thus (3), (8), (9)(c), (10) with R(·−), (11), and (12) have been added to the list (4)–(7). [And (9)(b) then follows from (11).] We next show that (11) implies (9)(a); and since (9)(a) trivially implies (8), this will add (9)(a) to the list. Deﬁne (i)

Znk ≡ KX (ξnk ) 1[ξnk <1−θ/n)

and m ¯ θn ≡ EZnk =

[0,1−θ/n)

KX (s) ds .

Fix > 0. Then for a ﬁxed tiny θ ≡ θ , the truncation inequality (10.4.15) gives (j)

¯n − m P (|X ¯ θn | > m ¯ θn ) ≤

n 1

2 EZnk /( n m ¯ θn)2 + n(θ/n)

(k)

≤

≤

2 KX (s) ds [0,1−θ/n) 2 2

nm ¯ θn

+θ

(θ/n) KX (1 − θ/n) 1 (θ/n) m(θ/n) ¯ +θ = 2 + θ < 2

2 ¯ θ m ¯ θn θ

M (θ/n)

for n suﬃciently large, by (11). In like fashion (l)

m ¯ 1n (1 − θ) (θ/n) KX (1 − θ/n) [0,1−1/n) KX (s) ds 1≥ = ≥1− ≥1−

m ¯ θn θ [0,1−θ/n) KX (s) ds [0,1−θ/n) KX (s) ds

for all n suﬃciently large, by (11). Thus, for n ≥ (some n ) we have

230 (m)

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES ¯n − m m ¯ θn |X ¯ 1n | = m ¯ 1n m ¯ 1n

¯ X m ¯ 1n 1 ¯ θn 2

n−m + 1 − ≤ [ + ] ≤ m ¯ θn m ¯ θn 1 −

1−

¯ n /m ¯ 1n →p 1, giving (9)(a). with probability at least 1 − 2 . Thus X Next, (11) implies (13). Let > 0. Observe that n n n (n) ¯ θn) ≤ 1 P (Znk > 2 nm ¯ θn)+ 1 P (Xnk = Znk ) < 2

1 P (Xnk > 2 nm for all large n, as in bounding ¯ 1n to replace the right side of (j). Then (l) allows m m ¯ θn in (n), which implies n1 P (Xnk > 3 nm ¯ 1n ) < 2 for all large n. This gives condition (13). Consider (13) implies (12) with t = an = 1/n. Fix 0 < θ ≤ 1. State (13) as (o)

P (KX (1 − ξ)/(n m ¯ 1n ) > ) < θ/n

This implies KX (1 − θ/n)/(n m ¯ 1n ) ≤ for all all n ≥ nθ ]; that is, (12) holds with t = an Then (10.3.17) and (10.3.19) show that (13) Finally, we add (15) to the list. Now (14) (p)

for all n ≥ (some nθ ) . n ≥ nθ [entailing r¯(1 − 1/n) ≤ for = 1/n. Thus (13) has been added. is equivalent to (14). implies (15), via

¯ n = {[max Xnk /n ]/ m ¯n} [max Xnk /n ]/ X ¯ 1n } × {m ¯ 1n /X = op (1) [1/(1 + op (1))] →p 0 ;

using (14) implies (9)(a). Now, suppose (15) holds. We will verify (12). Daniels’ inequality 10.3.6 shows that the event An ≡ [ ξn:k ≥ (k/n)/λ for all 1 ≤ k ≤ n] has P (An ) ≥ 1 − for all n ≥ (some n ), provided λ is speciﬁed large enough (with order statistics ξn:1 ≤ · · · ≤ ξn:n ). Thus (letting 1n denote the indicator of An ) n ¯ n × 1n = 1n × n m(ξ X (q) ¯ n] k=1 ¯ n:k )/n ≤ λ [ k=1 m((k/n)/λ)/λ ≤ λ{[m(1/nλ)/nλ] ¯ + (1/nλ,1/λ] m(t) ¯ dt} ¯ ≤ 3λ (1/2nλ,1/λ] m(t) ¯ dt ≤ 3λM(1/2nλ) , with probability exceeding 1 − . This gives (r)

¯ (1/2nλ)] ≤ 3λ [maxk KX (1 − ξnk )]/[n X ¯n] 1n × [maxk KX (1 − ξnk )]/[nM →p 0,

and since P ([min ξnk ] ≥ 1/2nλ) = (1 − 1/2nλ)n → exp(−1/2λ) > 0, this gives (s)

¯ m(1/2nλ)/[n ¯ M(1/2nλ)] → 0,

which is (12) with an = 1/2nλ. We showed that (8) implies the stronger R(x−) → 0. And theorem 7.7.1 shows that the weaker R(x) → 0 implies (11); and this was seen above to imply (9)(a), and hence (8). Thus R(x) → 0 and R(x−) → 0 are equivalent. 2

6.

GENERAL MOMENT ESTIMATION

231

Notation 6.3 (Moments in the iid case) Let Yn1 , . . . , Ynn be a triangular array of row independent rvs, all with common df F and qf K. Now deﬁne the rvs + − Ynk ≡ Ynk × 1[Ynk ≥0] and Ynk ≡ −Ynk × 1[Ynk <0] . Fix r > 0. Let − m− (t) ≡ |K+ (t)|r ,

M (t) ≡ m− 1n ≡

(t,1]

m(s) ds

(1/n,1]

m− (s) ds ,

m+ (t) ≡ |K + (1 − t)|r , m ≡ m− + m+ ,

where m+ 1n ≡

(1/n,1]

m+ (s) ds ,

(16) m± 1n

+ + − simultaneously denote any one of m− 1n , m1n , or m1n ± m1n ,

− Y¯rn ≡

1 n

± Y¯rn

n k=1

− r [Ynk ] ,

+ Y¯rn ≡

1 n

n k=1

+ r [Ynk ] ,

− + Y¯rn ≡ Y¯rn + Y¯rn ,

− ¯+ + − simultaneously denote any one of Y¯rn , Yrn or Y¯rn ± Y¯rn .

¯ ≡ KX (·) denote the qf of the rv X ≡ |Y |, and then let Let K(·) ¯ ¯ (t) ≡ ¯ r (1 − t) , K(s) ds = (t,1] m(s) (17) M ¯ ds, where m(t) ¯ ≡K [0,1−t) as was considered in Theorem 6.2. 2 Theorem 6.3 (Consistency of moments in the iid case) Fix an r > 0. Just some of the possible equivalent conditions are as follows: n (18) [ n1 k=1 |Ynk |r ]/m1n →p 1, where m1n ≡ M (1/n) . (19) as t → 0 . r(t) ≡ t m(t)/M (t) = t m(t)/ (t,1] m(s) ds → 0 (20)

r(an ) → 0

(21)

[max1≤k≤n

(22)

[max1≤k≤n

for any one speciﬁc an 0 having lim (an /an+1 ) < ∞ . |K(ξnk )|r ]/ m1n →p 0 n r r 1 1 k=1 |K(ξnk )| ] →p 0 . n |K(ξnk )| ] / [ n 1 n

Moreover, these are also equivalent to the list of theorem 6.2, which includes n ¯ (1/n) . (23) [ n1 k=1 |Ynk |r ]/m ¯ 1n →p 1, where m ¯ 1n ≡ M When any one of (3)–(15) and (18)–(23) holds, we also have both (24)

± [Y¯rn − m± 1n ]/m1n →p 0

(25)

m1n / m ¯ 1n → 1 .

and

Proof. Let r = 1. That (19) implies (18) is virtually the same as (i)–(m) in the proof of theorem 6.2, but now with Znk ≡ |K(ξnk )| 1(θ/n<ξnk <1−θ/n) . Now note that (18) is exactly (8) with νn = m1n . But (8) implies (15), while (15) implies (22) (since they are identical in content). The proof that (22) implies (21)

232

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

is virtually the same as the proof that (15) implies (14). That (21) implies (20) with an ≡ θ/n is just a two-tailed copy of (13) implies (12). That (20) implies (19) is contained in theorem 7.7.1. Thus the equivalence of (18)–(22) and (8) and (15) has been established. Having (15) on this list allows us to add all equivalences in theorem 6.1, and this is what (23) exempliﬁes. Consider (24). For the Ynk of the previous paragraph, we now deﬁne (a)

± ≡ m± (ξnk ), Ynk

± m± n ≡ EYnk ,

+ − Ynk ≡ Ynk +Ynk ,

mn ≡ E|Ynk | .

Using the truncation inequality (10.4.15) yet again gives (b)

¯ ± − m± | ≥ mn ) ≤ EY 2 /n( mn )2 + . P (|X n n n1

Now proceed to establish (24), as in (i)–(m) of the proof of theorem 6.1. Finally, combining (18) with (23) gives (25). 2

Proposition 6.1 Suppose X ≥ 0 and EX s = ∞, and 0 < r < s. (i) Let Xns ≡ n1 Xks /n for iid rvs X1 , X2 , . . . distributed as X, etc. Then (26)

[ Xnr ]s/r / Xns →a.s. 0,

(27)

2 ˜ a,a (ξ)]/EK ˜ a,a Var[K (ξ) → 1

as (a ∨ a ) → 0 .

¯ r and m ¯ s denote the m, ¯ functions of (11) for the rth and sth (ii) Let m ¯ r, M ¯ s, M ¯ M powers, respectively. Then the condition (28)

¯ s (t) → 0 implies t m ¯ r (t) → 0. tm ¯ s (t)/M ¯ r (t)/M

Proof. Consider (26). We follow the proof of (27) given in Gnedenko– Kolmogorov’s theorem 7.6.1, and give the proof for s = 2 and r = 1. Let Kn denote the empirical qf. Let h be positive, continuous, symmetric about t = 12 , ↑ to ∞ on 1 [ 12 , 1) and suppose it satisﬁes Ch ≡ 0 h2 (t) dt < ∞. Deﬁne a = 1 − b = 1/(2n). Then Cauchy–Schwarz gives b b b b (a) ( a |Kn (t)| dt)2 = ( a h(t)|Kn (t)/h(t)| dt)2 ≤ a h2 (t) dt a [K2n (t)/h2 (t)] dt. Let c ≡ c be ﬁxed so close to zero that Ch /h2 (c) < /8. Fix c . Then b b { n1 n1 |Xk | }2 / Xn2 ≤ 4{ a |Kn (t)| dt}2 / Xn2 ≤ 4 Ch a [K2n (t)/h2 (t)] dt/ Xn2 c b K2n (t) dt ( a + 1−c ) K2n (t) dt + / Xn2 h2 (1/2) h2 (c)

1−c (b)

≤ 4 Ch

(c)

≤ 4 Ch {

(d)

<

c

1−c c

K2n (t) dt/h2 (1/2)}/ Xn2 + /2

6.

GENERAL MOMENT ESTIMATION

233

for all n exceeding some n . The numerator of the leading term in (c) converges a.s. to a ﬁnite number, while the denominator has an a.s. lim sup at least as big as 1/

times the numerator. Use H¨ older instead of Cauchy–Schwarz for general 0 < r < s. Raising to a power increases a maximum more than an average, and so (e) (f)

¯ r (1 − t) tK t ¯ r (1 − u) du ≤ 1 − t K (t,1]

tm ¯ r (t) ¯ r (t) = M → 0.

1 1−t

¯ s (1 − t) tm ¯ s (t) K ¯ s (t) ¯ s (1 − u) du = M K (t,1]

2

Theorem 6.4 (Equivalencies pertaining to estimation of the variance) Let Y and Yn1 , . . . , Ynn be iid with df F (·) and qf K(·). Let F|Y | (·) and K|Y | (·) denote the df and qf of |Y |. Let dom(a, a) denote one of (a, 1], [0, 1 − a), or (a, 1 − a) − according as Y ≤ 0, Y ≥ 0, or Y is general. Deﬁne v(t) ≡ [K+ (t)]2 + [K + (1 − t)]2 . The following are just some of the possible equivalent conditions: (29) x2 P (|Y | > x)/U (x) → 0, where U (x) ≡ [ |y|≤x] y 2 dF (y) .

(30)

2 t K|Y | (1 − t)/V|Y | (t) → 0,

where

V|Y | (t) ≡

(31)

2 t [K+ (t) ∨ K 2 (1 − t)]/V (t) → 0,

where

V (t) ≡ dom(t,t) K 2 (s) ds .

(32)

an v(an )/V (an ) → 0

(33)

Yn2 / u1n →p 1,

where u1n ≡ U (x1n ) .

(34)

Yn2 / u ˜1n →p 1,

(35)

Yn2 / v¯1n →p 1,

where u ˜1n ≡ U (x1n ) + x1n P (|Y | > x1n ) . 2 where v¯1n ≡ [0,1−1/n) K|Y | (s) ds = V|Y | (1/n) .

(36)

Yn2 / v1n →p 1,

where v1n ≡ V (1/n) = dom(1/n,1/n) K 2 (s) ds .

(37)

Yn2 /νn →p 1 for any one speciﬁc νn > 0 for which n νn .

(38)

[maxk

1 n

2 Ynk ] / ν1n →p 0 where ν1n is any one of v1n , v¯1n , u ˜1n or u1n .

(39)

[maxk

1 n

2 Ynk ] / Yn2 →p 0 .

(40)

Dn ≡ [maxk

(41)

˜n2 →p 1. Sn2 / σ

1 n

[0,1−t)

2 K|Y | (s) ds .

for any one an 0 having lim (an /an+1) < ∞ .

|Ynk − Y¯n |2 ] / Sn2 →p 0

where Sn2 ≡ Yn2 − (Y¯n )2 .

Of course, (29) implies u1n /v1n → 1, v¯1n /v1n → 1, and u˜1n /v1n = v˜¯1n /v1n → 1. [Theorems 7.7.1 and 7.7.2 present well more than 48 other equivalencies similar to (29)–(32), all of which follow from using just geometrical considerations and Cauchy–Schwarz.]

234

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Proof. We note that the equivalence of (29) through (39) (and many other equivalencies) have already been established. We were able to accomplish this by combining the lists of theorems 6.2 and 6.3, which was possible since (38) appeared on both lists. Suppose (39). Suppose E(Y 2 ) = ∞ (since the other case is trivial). Then ¯ (Yn )2 / Yn2 →p 0 by (26), which makes (40) trivial. Suppose (40). Now, 2 2 Y ¯n | ¯n max |Ynk −Y Y nk √ max √ 2 − √ 2 n Yn2 nYn nYn (a) = ; Dn = 1 − (Y¯n )2 / Yn2 1 − (Y¯n )2 / Yn2 here 0 ≤ (Y¯n )2 / Yn2 ≤ 1 always holds, and |Y¯n |/ n Yn2 →a.s. 0 always holds. Thus Dn (ω) → 0 for a.e. ﬁxed (Yn1 (ω), Yn2 (ω), . . .) implies that the numerator of (a) goes to zero for a.e. ﬁxed ω. Thus [max |Ynk | ]/ nYn2 converges in probability to zero, as required for (39) (by going to subsequences). That (41) is equivalent to (34) follows using proposition 6.1. 2 2 Remark 6.1 Let σ ˜|Y | (x) denote the variance of the rv Y Winsorized outside [−x, x]. 2 Let σ ˜ (t) denote the variance of the rv KY (ξ) Winsorized outside dom(t, t). Note 2 2 that σ ˜|Y ˜ 2 (1/n). Thus σ ˜|Y ˜ 2 (1/n)/v1n → 1 when (29) holds | (x1n ) = σ | (x1n )/v1n = σ with Var[Y ] = ∞; we just apply both (32) and (27). 2

7.

LAW OF THE ITERATED LOGARITHM

235

7

Law of the Iterated Logarithm

Theorem 7.1 (LIL; Hartman–Wintner; Strassen) Let X1 , X2 , . . . be iid rvs. Consider the partial sums Sn ≡ X1 + · · · + Xn . (a) If EX = 0 and σ2 ≡ Var[X] < ∞, then (1)

lim sup √ n→∞

Sn = σ a.s., 2n log log n

while

lim inf √ n→∞

Sn = −σ a.s. 2n log log n

(b) In fact, (2)

√

Sn →a.s. [−σ, σ]. 2n log log n

√ [That is, for a.e. ω the limit set of Sn / 2n log log n is exactly [−σ, σ] ]. (c) Conversely, if (3)

lim sup √ n→∞

|Sn | < ∞ a.s., 2n log log n

Theorem 7.2 (The other LIL; Chung) (4)

lim inf max

n→∞ 1≤k≤n

then

EX = 0 and σ2 < ∞.

If X1 , X2 , . . . are iid (0, σ2 ), then

|Sk | 2 log log n √ = π/2 a.s. nσ

[We state this for fun only, as it has seen little application.] Versions of both theorems are also known for cases other than iid. The classical proof of theorem 7.1 in full generality begins with truncation, and then carefully uses exponential bounds for bounded rvs. A more modern proof relies upon Skorokhod embedding of the partial sum process in Brownian motion. This general proof is outlined in the straightforward exercise 12.8.2, after embedding is introduced. But the proof below for the special case of normal rvs contains several of the techniques used in the classical proof of the general case (and in other related problems). And it is also a crucial component of the general case in exercise 12.8.2. Proposition 7.1 Let Z1 , Z2 , . . . be iid√N (0, 1) rvs. Let Sn ≡ Z1 + · · · + Zn and √ bn ≡ 2 log log n. Then lim supn→∞ Sn / n bn = 1 a.s. Proof. (5)

Let > 0. We will use the exponential bound √ exp[−(1 + )λ2 /2] ≤ P (Sn / n ≥ λ) ≤ exp[−(1 − )λ2 /2]

for all λ > λ

(for some λ ) [see Mills’ ratio exercise 7.1 below], and the L´evy maximal inequality (a)

P ( max Sk ≥ λ) ≤ 2P (Sn ≥ λ) 1≤k≤n

for all λ > 0.

Let nk ≡ [ak ] for a > 1; a suﬃciently small a will be speciﬁed below. Now,

236

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES Ak ≡

nk−1 ≤m≤nk

[Sm ≥

√

m(1 + 2 )bm ]

⊂

(b) since (c)

√

max

nk−1 ≤m≤nk

Sm ≥ (1 + 2 )

√ nk−1 bnk−1 nk , nk

n is and bn is ; so that for k suﬃciently large, √ nk−1 by (a) P (Ak ) ≤ 2P Snk / nk ≥ (1 + 2 ) bnk−1 nk 1 2 1−

≤ 2 exp − (1 − )(1 + 2 ) 2 log k by (5) 2 a ≤ 2 exp(−(1 + ) log k) = 2/k 1+

(d)

for a suﬃciently close to 1

= (a convergent series).

Thus P (Ak i.o.) = 0 by Borel–Cantelli. Since > 0 is arbitrary, we thus have (e)

Sn lim sup √ ≤ 1 a.s. n bn n→∞

Also, (e) yields an extra result that will be useful below. Thus (f)

P (Ak i.o.) = 0

for any large positive a.

We must now show that the lim in (e) is also ≥ 1 a.s. We will still use nk ≡ [ak ], but a will be speciﬁed suﬃciently large below. We write Snk = Snk−1 +(Snk −Snk−1 ), so that Snk−1 Snk − Snk−1 nk−1 bnk−1 Snk (g) = + √ √ √ nk bnk nk bnk nk−1 bnk−1 nk bnk (h)

Snk−1 Snk − Snk−1 1 ∼ √ ·1· √ + √ . a nk−1 bnk−1 nk bnk

Now, the independent events (i) have (j)

Bk ≡ [Snk −Snk−1

√ Snk − Snk−1 (1 − 2 ) nk bnk √ ≥ (1−2 ) nk bnk ] = √ ≥ √ nk − nk−1 nk − nk−1

1 nk P (Bk ) ≥ exp − (1 + )(1 − 2 )2 b2nk 2 nk − nk−1 1 2 (1 + )a ≥ exp − (1 + )(1 − 2 ) 2 log k 2 a−1 ≥ exp (−(1 − ) log k)

(k)

for a suﬃciently large

= 1/k 1− = (a series with inﬁnite sum),

by (5)

7.

LAW OF THE ITERATED LOGARITHM

237

so that P (Bk i.o.) = 1 by the second Borel–Cantelli lemma. But P (Ak i.o.) = 0 and P (Bk i.o.) = 1 means that (l)

P (Ack ∩ Bk i.o.) = 1.

Moreover, on Ack ∩ Bk we have, using (h), (i), and the symmetric version of (b), (m)

√

Snk (1 + 2 )(1 + ) √ ≥− + (1 − 2 ) ≥ (1 − 3 ) nk bnk a

for the constant a speciﬁed suﬃciently large. Thus, even on the subsequence nk , since > 0 was arbitrary, (n)

lim sup √ k→∞

Snk ≥ 1 a.s. nk bnk

Combining (e) and (n) gives the proposition.

Exercise 7.1 (Mills’ ratio) (6)

λ2

2

(a) For all λ > 0

λ 1 1 1 √ exp(−λ2 /2) < P (N (0, 1) > λ) < √ exp(−λ2 /2) , + 1 2π λ 2π

which can be rewritten as (7)

1 λ φ(λ) < 1 − Φ(λ) < φ(λ) , λ2 + 1 λ

where φ and Φ denote the standard normal N (0, 1) density and df, respectively. Thus if λn ∞, we have (with δn 0) (8)

P (N (0, 1) > λn ) = exp(−λ2n (1 − δn )/2)

as n → ∞.

[This clean exponential bound for normal rvs is the key to a simple LIL proof in proposition 7.1. The classic Hartman–Wintner proof uses truncation to achieve a reasonable facsimile.] (b) (Ito–McKean) Show that (9)

√

2 2 φ(λ) < 1 − Φ(λ) < √ φ(λ) . λ2 + 4 + λ λ2 + 2 + λ

Exercise 7.2 In place of (c) in the LIL proof of proposition 7.1, use Mills’ ratio √ √ to bound P (An ) ≡ P (Sn / n ≥ (1 + 2 ) 2 log n). Use that bound directly to show √ √ that lim sup |Sn |/( n 2 log n) ≤ 1 a.s. [This “poor” result will show the value of using the “block of indices” in the deﬁnition of Ak in the proof we gave.] Exercise 7.3 Suppose arbitrary events An and Bn satisfy P (An i.o.) = 1 and P (Bnc i.o.) = 0. Show that P (An ∩ Bn i.o.) = 1.

238

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Summary Suppose X, X1 , X2 , . . . are iid (µ, 1). Then: n k=1 (Xk − µ) →a.s. 0 (10) by the SLLN. n n (11)

k=1 (Xk n1/r

− µ)

→a.s. 0

for all 1 ≤ r < 2, by Marcinkiewicz–Zygmund.

n (Xk − µ) √k=1 by the LIL. →a.s. [−1, 1] 2n log log n √ Suppose we go all the way to n in the denominator. Then the classical CLT gives (12)

n (13)

k=1 (Xk

√

− µ)

n

→d N (0, 1)

by the CLT,

even though we have divergence to ±∞ for a.e. ω (by the LIL).

2

Exercise 7.4 (rth mean convergence theorem) Let X, X1 , X2 , . . . be iid, and consider the partial sums Sn ≡ X1 + · · · + Xn . Let 0 < r < 2 (and suppose EX = 0 in case 1 ≤ r < 2). The following are equivalent: (a)

E|X|r < ∞.

(b)

Sn /n1/r →a.s. 0.

(c)

E|Sn |r = o(n).

(d)

E(max1≤k≤n |Sk |r ) = o(n).

8.

STRONG MARKOV PROPERTY FOR SUMS OF IID RVS

239

8

Strong Markov Property for Sums of IID RVs

≡ (S1 , S2 , . . .). Let X1 , X2 , . . . be iid and let Sn ≡ X1 + · · · + Xn . Let S Deﬁnition 8.1 The rv N is a stopping time for the sequence of rvs S1 , S2 , . . . if [N = k] ∈ F(S1 , . . . , Sk ) for all k ≥ 1. It is elementary that (1) (2)

FN ≡ F(Sk : k ≤ N ) : A ∩ [N = k] ∈ F(S1 , . . . , Sk ) for all k ≥ 1} = (a σ-ﬁeld), ≡ {A ∈ F(S)

since it is clearly closed under complements and countable intersections. Proposition 8.1

Both N and SN are FN -measurable.

Proof. Now, to show that [N ≤ m] ∈ FN we consider [N ≤ m] ∩ [N = k] equals this implies [N ≤ m] ∈ FN . Likewise, [N = k] or ∅, both of which are in F(S); (a)

[SN ≤ x] ∩ [N = k] = [Sk ≤ x] ∩ [N = k] ∈ F(S1 , . . . , Sk ),

implying that [SN ≤ x] ∈ FN .

2

Theorem 8.1 (The strong Markov property) the increments continuing from the random time (3)

S˜k ≡ SN +k − SN ,

If N is a stopping time, then

k ≥ 1,

have the same distribution on (R∞ , B∞ ), as does Sk , k ≥ 1. Moreover, deﬁning S˜ ≡ (S˜1 , S˜2 , . . .) , (4)

˜ ≡ F(S˜1 , S˜2 , . . .) is independent of FN (hence of N and SN ). F(S)

Proof.

Let B ∈ B∞ and A ∈ FN . Now, ∞ P ([S˜ ∈ B] ∩ A) = n=1 P ([S˜ ∈ B] ∩ A ∩ [N = n]) ∞ = n=1 P ([(Sn+1 − Sn , Sn+2 − Sn , . . .) ∈ B] ∩ (A ∩ [N = n]))

(a)

with A ∩ [N = n] ∈ F(S1 , . . . , Sn ) ∞ = n=1 P ([(Sn+1 − Sn , Sn+2 − Sn , . . .) ∈ B]) P (A ∩ [N = n]) ∈ B)∞ P (A ∩ [N = n]) = P (S n=1 (b)

∈ B) P (A). = P (S

Then use P (S˜ ∈ B) = P (S = B) to Set A = Ω in (b) to conclude that S˜ ∼ = S. rewrite (b) as (c)

P ([S˜ ∈ B] ∩ A) = P (S˜ ∈ B) P (A),

which is the statement of independence.

2

240

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Deﬁnition 8.2

(5)

Deﬁne waiting times for return to the origin by

W1 ≡ min{n : Sn = 0} · · · Wk ≡ min{n > Wk−1 : Sn = 0}

with W1 = +∞ · · · with Wk = +∞

if the set is empty,

if the set is empty.

Then deﬁne Tk ≡ Wk − Wk−1 , with W0 ≡ 0, to be the interarrival times for return to the origin. Proposition 8.2 are, in fact, iid.

If P (Sn = 0 i.o.) = 1, then T1 , T2 , . . . are well-deﬁned rvs and

Proof. Clearly, each Wk is always an extended-valued rv, and the condition P (Sn i.o.) = 1 guarantees that Wk (ω) is well-deﬁned for all k ≥ 1 for a.e. ω. Now, T1 = W1 is clearly a stopping time. Thus, by the strong Markov property, (1) T1 is independent of the rv S˜(1) ≡ S˜ with kth coordinate S˜k ≡ S˜k ≡ ST1 +k − ST1 Thus T2 is independent of the rv S˜(2) with kth coordinate and S˜(1) ≡ S˜ ∼ = S. (2) (1) (1) Continue with S˜(3) , S˜k ≡ S˜T2 +k − S˜T2 = ST1 +T2 +k − ST1 +T2 and S˜(2) ∼ = S˜(1) ∼ = S. etc. [Note the relationship to interarrival times of a Poisson process.] 2

Exercise 8.1 (Wald’s identity) (a) Suppose X1 , X2 , . . . are iid with mean µ, and N is a stopping time with ﬁnite mean. Show that Sn ≡ X1 + · · · + Xn satisﬁes (6)

ESN = µEN.

(b) Suppose each Xk equals 1 or −1 with probability p or 1 − p for some 0 < p < 1. Then deﬁne the rv N ≡ min{n : Sn equals −a or b}, where a and b are strictly positive integers. Show that N is a stopping time that is a.s. ﬁnite. Then evaluate EN . [Hint. [N ≥ k] ∈ F(S1 , . . . , Sk−1 ), and is thus independent of Xk , while SN = ∞ X k 1[N ≥k] .] k=1

9.

CONVERGENCE OF SERIES OF INDEPENDENT RVS

241

9

Convergence of Series of Independent RVs

In section 10.4 we proved the SLLN after recasting it (via Kronecker’s lemma) as a theorem about a.s. convergence of inﬁnite series. In this section we consider the convergence of inﬁnite series directly. Since the convergence set of a series is a tail event (recall remark 10.1.1), convergence can happen only with probability 0 or 1. Moreover, the ﬁrst theorem below seems to both limit the possibilities and broaden the possible approaches to them. All proofs are given at the end of this section. Let X1 , X2 , . . . be independent. Then, for some rv S, we have

Theorem 9.1 (1)

Sn ≡

n

Xk →a.s. S

k=1

iﬀ

n

Xk →p S

iﬀ

k=1

n

Xk →d S.

k=1

[We will show the ﬁrst equivalence now, and the second in chapter 14.] Theorem 9.2 (The 2-series theorem) Xk ∼ = (µk , σk2 ). We then have (2)

n

µk → µ and

k=1

∞

Let X1 , X2 , . . . be independent, with

σk2 < ∞ implies Sn ≡

k=1

n

Xk →a.s. (some rv S).

k=1

If further |Xk | ≤ c for all k, then (including a converse) (3)

S0,n ≡

n

(Xk − µk ) →a.s. (some rv S0 )

∞

if and only if

k=1

(4)

Sn ≡

n

σk2 < ∞,

k=1

Xk →a.s. (some rv S) iﬀ

k=1

n

∞

µk → µ and

k=1

σk2 < ∞.

k=1

Moreover, in all cases, S = S0 + µ and ∞ ∞ (5) Var[S] = σ2 ≡ k=1 σk2 , ES = µ ≡ k=1 µk

and

Sn →L2 S.

If a series is to converge, the size of its individual terms must be approaching zero. Thus the rvs must be eﬀectively bounded. Thus truncation should be particularly eﬀective for series. Theorem 9.3 (The 3-series theorem) Let X1 , X2 , . . . be independent rvs. (c) (a) Deﬁne Xk to be the trimmed Xk that equals Xk or 0 as |Xk | ≤ c or as |Xk | > c. Then n (6) Sn ≡ Xk →a.s. (some rv S) k=1

if and only if for some c > 0 the following three series all converge: (7)

Ic ≡

∞ k=1

P (|Xk | > c),

IIc ≡

∞ k=1

(c)

Var[Xk ],

IIIc ≡

∞

(c)

EXk .

k=1

(b) The condition (7) holds for some c > 0 if and only if it holds for all c > 0. n (c) If either Ic , IIc , or IIIc diverges for any c > 0, then k=1 Xk diverges a.s.

242

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Example 9.1 Suppose X1 , X2 , . . . are independent and are uniformly bounded. They are assumed to be independent of the iid Rademacher rvs 1 , . . . , n . Then ∞ 2 n (8) if and only if k=1 k Xk →a.s. (some rv S) k=1 σk < ∞. ∞ Moreover, S ∼ σ2 ). [This is immediate from the 2-series theorem.] 2 = (0, k=1

k

Exercise 9.1 X1 , X2 , . . . are iid with P (Xk = 1) = P (Xk = −1) = 12 . ∞ Suppose n Show that n=1 Xn /3 →a.s. (some S), and determine the mean, variance, and the name of the df FS of S. Also determine the characteristic function of S (at some point after chapter 13). n Exercise 9.2 (a) Show that k=1 ak Xk →a.s. (some S) when X1 , X2 , . . . are independent with Xk ∼ = Uniform(−k, k) for k ≥ 1, and where 0 < a < 1. (b) Evaluate the mean and variance of S. Exercise 9.3 LetX1 , X2 , . . . be arbitrary rvs withall Xk ≥ 0 a.s. Let c > 0 n be arbitrary. Then ∞ k=1 E(Xk ∧ c) < ∞ implies that k=1 Xk →a.s. (some rv S), and conversely. Exercise 9.4 ∞

(a) Let Z1 , Z2 , . . . be iid N (0, 1) rvs. Show that

2 2 k k=1 [Z2k−1 + Z2k ]/2 1 ∞ 0n n+k n=1 k=1 Zk /2

→a.s. (some rv), →a.s. (some rv),

and determine (if possible) the mean, variance, and distribution of the limiting rvs. ∞ (b) Let Y1 , Y2 , . . . be iid Cauchy(0, 1) rvs. Does k=1 Yk /2k →a.s. (some rv)? If so, what is the distribution of the limit?

Proofs Proof. Consider theorem 9.1. Now, →a.s. always implies →p . So we turn to the converse. Suppose Sn →p S (which is equivalent to Sm − Sn →p 0). To establish Sn →a.s. , it is equivalent to verify of the convergence set (2.3.7) that (9)

lim lim P ( max |Sm − Sn | ≥ ) = 0 n

N

n≤m≤N

for all > 0.

But Ottavani–Skorokhod’s inequality 10.3.4 gives (a) (b)

m P (maxn≤m≤N |Sm − Sn | ≥ ) = P (maxn≤m≤N | n Xk | ≥ ) N N ≤ P (| n Xk | ≥ /2)/[1 − maxn≤m≤N P (| m Xk | > /2)]

(c)

≤ P (|SN − Sn | ≥ /2)/[1 − maxn≤m≤N P (|SN − Sm+1 | > /2)]

(d)

= o(1)/[1 − o(1)] < θ

for all n, N ≥ (some n,θ ),

using SN − Sm →p 0 for (d). Thus (9) holds, and Sn →a.s. (some rv S ). The a.s. limit S equals S a.s. by proposition 2.3.4. 2 The proof of the 2-series theorem above will require a converse of Kolmogorov’s inequality that is valid for bounded rvs.

9.

CONVERGENCE OF SERIES OF INDEPENDENT RVS

243

Inequality 9.1 (Kolmogorov’s other inequality) Consider independent zero-mean rvs Xk , and set Sk ≡ X1 + · · · + Xk for 1 ≤ k ≤ n. Suppose |Xk | ≤ (some M ) < ∞ for all k. Then (10) P ( max |Sk | ≤ λ) ≤ (λ + M )2 / nk=1 σk2 for all λ > 0. 1≤k≤n

Proof. Let Ak ≡ [max1≤j
1

k

(b)

1Ak ) + 2 · 0 + P (Ak ) E(Sn − Sk )2 } ≤ (λ + M )2 n1 P (Ak ) + n1 P (Ak ) nj=k+1 σj2

(c)

≤ {(λ + M )2 + Var[Sn ]}P (Mn > λ),

=

2 1 {E(Sk

by independence

where in step (b) we take advantage of |Sk | ≤ |Sk−1 | + |Xk | ≤ λ + M on Ak . We also note that E Sn2 1n A = ESn2 − E{Sn2 1[Mn ≤λ] } 1

(d)

k

≥ Var[Sn ] − λ2 P (Mn ≤ λ) = Var[Sn ] − λ2 [1 − P (Mn > λ)],

using |Sn | ≤ λ on the event [Mn ≤ λ] to obtain (d). Combining (c) and (d) and doing algebra gives (e) (f)

P (Mn > λ) ≥ ≥1−

Var[Sn ] − λ2 (λ + M )2 = 1 − (λ + M )2 + Var[Sn ] − λ2 (λ + M )2 + Var[Sn ] − λ2

(λ + M )2 . Var[Sn ]

2

Proof. Consider theorem 9.2. We ﬁrst verify (2), and without loss of generality we assume that all µk = 0. By theorem 9.1, we need only show that Sm − Sn →p 0. But this follows immediately from Chebyshev’s inequality, since ∞ P (|Sm − Sn | ≥ ) ≤ Var[Sm − Sn ]/(n 2 ) ≤ n+1 σk2 /(n 2 ) <

for all suﬃciently large n. Thus (2) holds. Consider the forward half of (3), again taking all µk = 0. Since Sn →a.s. , for some suﬃciently large λ we have (a)

0 < P (supn |Sn | ≤ λ)

(b)

= limN P (max1≤n≤N |Sn | ≤ λ) by the monotone property of a measure 2 ≤ limN (λ + c)2 / N by Kolmogorov’s other inequality 9.1 1 σk

(c)

= (λ + c)2 /

∞ 1

Note that (c) implies that

σk2 . ∞ 1

σk2 < ∞, giving (3).

244

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Consider the forward half of (4). Suppose that Sn →a.s. S. The plan is ﬁrst to symmetrize, so that we can use (3) to prove (4). Let Xn ’s be independent, and independent of the Xn ’s with Xn ∼ = Xn ; then Xns ≡ Xn − Xn denotes the symmetrized rv. Since →a.s. depends only on the ﬁnite-dimensional distributions, n the given fact that Sn →a.s. S implies that Sn ≡ 1 Xks →a.s. (some rv S ) ∼ = S. We can thus claim that n (d) Sns ≡ 1 Xks →a.s. S s ≡ S − S . ∞ s Now, |Xns | ≤ 2c; thus (d) and (3) imply that 1 Var[Xn ] < ∞. Thus ∞ 2 ∞ s (e) 1 σn = 1 Var[Xn ]/2 < ∞. n Now, (e) and (2) imply that 1 (Xk − µk ) →a.s. (some rv S0 ) with mean 0. Thus n n n (f) 1 µk = ( 1 Xk ) − [ 1 (Xk − µk )] →a.s. S − S0 . Thus (4) holds, and S = S0 + µ. We next verify (5) that Var[S] = σ2 and ES = µ. Now, (g)

Var[S] = ≤

(h)

1 2

1 2

E(S − S )2 =

lim E(Sn − Sn )2

= lim

n 1

σk2 =

∞ 1

1 2

E[ lim(Sn − Sn )2 ] =

1 2

E[( lim (Sn − Sn )2 )]

by Fatou’s lemma σk2 = σ 2 .

And Var[S] ≥ σ2 , since for all n Var[S] = 12 E(S − S )2 = 12 Var[ (i)

≥

n 1

n 1

Xns ] + 12 Var[

∞

s n+1 Xn ]

σk2 + 0.

Thus Var[S] = σ2 , and hence ES = µ by theorem 3.5.3 on moment convergence. 2 Proof. Consider the 3-series theorem. Consider (a) and (b) in its statement: Suppose that the 3 series converge for at least one value of c. Then II and III imply (c) that n1 Xk →a.s. by (2). Thus n1 Xk →a.s. by proposition 10.2.1, since I < ∞ (c) (c) implies that X1 , X 2 , . . . and X1 , X2 , . . . are Khinchin equivalent sequences. n Suppose that 1 Xk →a.s. . Then for all c > 0 we have P (|Xn | > c i.o.) = 0, so that I < ∞ holds for all c > 0 by the second Borel–Cantelli lemma. Thus n (c) (c) (c) 1 Xk →a.s. for all c, since I < ∞ implies that X1 , X2 , . . . and X1 , X2 , . . . are Khinchine equivalent sequences. Thus II < ∞ and III < ∞ for all c by the 2-series theorem result (4). Consider (c). Kolmogorov’s 0-1 law shows that Sn either converges a.s. or else diverges a.s.; and it is not convergent if one of the three series fails to converge. 2

9.

CONVERGENCE OF SERIES OF INDEPENDENT RVS

245

Other Generalizations of the LLNs Exercise 9.5 The following (with r = 1) can be compared to theorem 10.4.4. If X1 , X2 , . . . are independent with 0 means, then ∞ 2r r+1 (11) < ∞ for some r ≥ 1 implies Sn /n →a.s. 0. 1 E|Xn | /n Exercise 9.6 (Chung) We give another variation of theorem 10.4.4. Suppose φ > 0 is even and continuous and either φ(x)/x but φ(x)/x2 or else φ(x) but φ(x)/x . Let bn ∞. Suppose X1 , X2 , . . . are independent with 0 means. Then ∞ Eφ(Xn )/φ(bn ) < ∞ implies both n=1 n (12) ∞ X /b → and X a.s. n=1 n n 1 k /bn →a.s. 0. The WLLN is taken up again after the characteristic function tool has been introduced.

246

10

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Martingales

Deﬁnition 10.1 (Martingales) (a) A sequence of rvs S1 , S2 , . . . on a probability space (Ω, A, P ) that is adapted to an sequence of σ-ﬁelds A1 ⊂ A2 ⊂ · · · is called a martingale (abbreviated mg) if E|Sk | < ∞ for all k, and (1)

E(Sk |Ai ) =a.s. Si

for all i ≤ k in the index set.

If (Sk , Ak ), k ≥ 1 is a mg, then the increments Xk ≡ Sk − Sk−1 are called the martingale diﬀerences. ¯ A collection {St : t ∈ I} (b) Let I denote a subinterval of the extended real line R. of rvs on some (Ω, A, P ) that is adapted to an family of σ-ﬁelds {At : t ∈ I} is called a martingale if E|St | < ∞ for all t ∈ I, and (2)

E(St |Ar ) =a.s. Sr

for all r ≤ t in I.

(c) If “=” is replaced by “ ≥” in either of (1) or (2), then either of {Sk : k ≥ 1} or {St : t ∈ I} is called a submartingale (or submg). Example 10.1 (The prototypical example) Let X1 , . . . , Xn denote independent rvs with 0 means, and set Sk ≡ X1 +· · ·+Xk and Ak ≡ σ[X1 , . . . , Xk ] for 1 ≤ k ≤ n. Then the sequence of partial sums satisﬁes (3)

(Sk , Ak ), 1 ≤ k ≤ n, is a mg,

while (4)

(Sk2 , Ak ), 1 ≤ k ≤ n, is a submg.

The ﬁrst claim is trivial, and the second holds, since E(Sk2 |Ai ) = E{Si2 + 2Si (Sk − Si ) + (Sk − Si )2 |Ai } (a)

≥ Si2 + 2Si E{Sk − Si |Ai } + 0 = Si2 + 0 + 0 = Si2 ,

using (8.4.20) and (8.4.16).

2

Exercise 10.1 (Equivalence) (a) Show that (St , At ), t ∈ I, is a martingale if and only if (5) St dP = Sr dP for all Ar ∈ Ar and all r ≤ t with r, t ∈ I. Ar

Ar

(b) For a submartingale, just replace “=” by “≥” in (5). Notation 10.1 We will use  and =  mg submg and ≥ (6)  > s-mg and =

the following notational system: for a martingale. for a submartingale. for a s-mg (mg or submg, as the case may be).

Thus exercise 10.1 states that (St , At ), t ∈ I, is a s-mg if and only if > (7) St dP = Sr dP for all Ar ∈ Ar , and for all r ≤ t with r, t ∈ I.2 Ar

Ar

11.

MAXIMAL INEQUALITIES, SOME WITH BOUNDARIES

11

Maximal Inequalities, Some with Boundaries

247

Inequality 11.1 (Monotone inequality) For arbitrary rvs X1 , . . . , Xn and for constants 0 < b1 ≤ · · · ≤ bn we let Sk ≡ X1 + · · · + Xk and obtain k X |Sk | i ≤ 2 max (1) max . 1≤k≤n bk 1≤k≤n bi i=1

If all Xi ≥ 0, then we may replace 2 by 1. [This also holds in higher dimensions, when properly formulated. See Shorack and Smythe(1976).] Proof. (a)

Deﬁne b0 = 0, X0 = 0, Yj = Xj /bj , and Tk = Sk =

k

j=1 bj ∆Tj

=

k j=1

∆Tj

j i=1

∆bi =

k

k i=1

j=1 Yj .

Then

Tik ∆bi ,

k k where ∆bj ≡ bj −bj−1 , ∆Tj ≡ Tj −Tj−1 , and Tik ≡ j=i Yj . As i=1 (∆bi /bk ) = 1 with each ∆bi /bk ≥ 0, we have k (b) max |Sk |/bk ≤ max { i=1 |Tik | (∆bi /bk )} 1≤k≤n

1≤k≤n

(c)

≤ max

(d)

≤2

1≤k≤n

max |Tik |

1≤i≤k

since an average does not exceed the maximum

max |Tk | .

1≤k≤n

Note that 1 can replace 2 in step (d) if all Xi ≥ 0.

2

Martingale Maximal Inequalities Inequality 11.2 (Doob) Let (Sk , Ak ), 1 ≤ k ≤ n, be a submg and deﬁne the maximum Mn ≡ max1≤k≤n Sk . Then (2) λP (Mn ≥ λ) ≤ Sn dP ≤ ESn+ ≤ E|Sn | for all λ > 0, [Mn ≥λ]

(3)

P (Mn ≥ λ) ≤ inf E(erSn )/erλ r>0

for all λ > 0.

If (Sk , Ak ), 1 ≤ k ≤ n is a zero-mean mg, then (Sk2 , Ak ), 1 ≤ k ≤ n is a submg. This allows the maximum to be bounded by (4)

P (Mn ≥ λ) ≤ Var[Sn ]/λ2

for all λ > 0.

[This last is Kolmogorov’s inequality, valid for zero-mean mgs.]

248

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Proof. (a)

Since E(Sn |Ak ) ≥ Sk a.s. by the deﬁnition of a submg, we have for all Ak ∈ Ak Ak Sn dP = Ak E(Sn |Ak ) dP ≥ Ak Sk dP

by (8.4.1) in the deﬁnition of conditional expectation. Now let (b)

Ak ≡ [max1≤j
so that k is the ﬁrst index for which Sk is ≥ λ. Then n n λP (Mn ≥ λ) = λ 1 P (Ak ) ≤ 1 Ak Sk dP (c)

≤

(d)

=

n 1

Ak

[Mn ≥λ]

Sn dP Sn dP ≤

using (a) [Mn ≥λ]

Sn+ dP ≤ Sn+ dP ≤ |Sn | dP,

as claimed. In a s-mg context, these are called “ﬁrst passage time” proofs. nTo this end, set τ equal to k on Ak for 1 ≤ k ≤ n, and set τ equal to n + 1 on ( 1 Ak )c . Then τ is the ﬁrst passage time to the level λ. That {(exp(rSk ), Ak ), 1 ≤ k ≤ n} is also a submg for any r > 0 follows from Jensen’s inequality for conditional expectation. Applying Doob’s ﬁrst inequality (2) above thus gives (3). [This in often sharper than (2), though it requires the existence of the moment generating function E exp(rSn ).] When (Sk , Ak ) is a mg, then (Sk2 , Ak ) is also a submg (by another application of the same Jensen’s inequality), so that (2) applied to the latter submg gives (4). 2 Inequality 11.3 (H´ ajek–R´ enyi) Let (Sk , Ak ), 1 ≤ k ≤ n, be a mg with all ESk = 0. Let 0 < b1 ≤ · · · ≤ bn . Then $ n % N 4 2 2 2 2 (5) P max |Sk |/bk ≥ λ ≤ 2 σk /bn + σk /bk for all λ > 0. n≤k≤N λ k=1

k=n+1

Proof. (We give the proof for independent rvs.) The monotone inequality bounds the maximum partial sum via / . k S Xi n + (6) max |Sk |/bk ≤ 2 max . n≤k≤N n≤k≤N bn bi i=n+1

Applying Kolmogorov’s inequality (4) to (6) gives N (a) P max |Sk |/bk ≥ λ ≤ (λ/2)−2 Var[Sn /bn ] + k=n+1 Var[Xk ]/b2k n≤k≤N

(b)

= (4/λ2 )

n 1

σk2 /b2n +

2 2 σ /b n+1 k k .

N

(A diﬀerent proof can eliminate the factor 4.)

2

11.

MAXIMAL INEQUALITIES, SOME WITH BOUNDARIES

249

Exercise 11.1 To complete ajek–R´enyi inequality for mgs, one the proof of the H´ can show that Tk ≡ Sn /bn + kn+1 Xi /bi is such that (Tk , Ak ), n ≤ k ≤ N , is also a mg, and that Var[SN ] is equal to the right-hand side of (b). Do it. Inequality 11.4 (Birnbaum–Marshall) Let (S(t), A(t)), 0 ≤ t ≤ θ, be a mg having S(0) = 0, ES(t) = 0, and ν(t) = ES 2 (t) ﬁnite and continuous on [0, θ]. Suppose that paths of S are right (or left) continuous. Let q(·) > 0 on (0, θ] be and right (or left) continuous. Then θ θ −2 (7) P (S/q0 > λ) ≤ 4λ [q(t)]−2 dν(t) for all λ > 0. 0

Proof. (a)

Because of right (or left) continuity and S(0) = 0, we have θ n n P (S/q0 ≤ λ) = P maxn |S(θi/2 )|/q(θi/2 ) ≤ λ for all n ≥ 1 0≤i≤2

= lim P

maxn |S(θi/2 )|/q(θi/2 ) ≤ λ n

n

0≤i≤2

≥ lim{1 − λ−2

2n 1

by proposition 1.2.2

E[S 2 (θi/2n ) − S 2 (θ(i − 1)/2n )]/q 2 (θi/2n )}

by H´ ajek–R´enyi = 1 − 4λ−2 lim (b)

→ 1 − 4λ−2

θ 0

2n 1

q −2 (θi/2n ) [ν(θi/2n ) − ν(θ(i − 1)/2n )]

[q(t)]−2 dν(t)

using the MCT.

2

Inequality 11.5 (Doob’s Lr -inequality) (i) Let (Sk , Ak ), for 1 ≤ k ≤ n, be a submg. Consider Mn ≡ max1≤k≤n Sk+ . Let r > 1. Then (8)

r r E Mnr ≤ ( r−1 ) E{(Sn+ )r } .

(ii) Let (Sk , Ak ), for 1 ≤ k ≤ n, be a mg. Consider Mn ≡ max1≤k≤n |Sk |. Let r > 1. Then (9)

r r E Mnr ≤ ( r−1 ) E{|Sn |r } .

Proof. Now, (Sk+ , Ak ), for 1 ≤ k ≤ n, is also a submg, by the conditional version of Jensen’s inequality. (Or refer to (18.1.7).) [And refer to (18.1.6) for case (ii).] Thus in case (i) we have ∞ (a) E Mnr = 0 rλr−1 P (Mn > λ) dλ by (7.4.13) ∞ (b) ≤ 0 rλr−1 λ−1 E{Sn+ 1[Mn ≥λ] } dλ by Doob’s inequality 11.2 (c)

= E{Sn+

Mn 0

rλr−2 dλ}

by Fubini

250

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES r = E{ Sn+ ( r−1 ) Mnr−1 }

(d)

r ) (E{(Sn+ )r })1/r (E{Mnr })(r−1)/r ≤ ( r−1

by H¨ older’s inequality,

where r−1 + s−1 = 1 implies that s = r/(r − 1). So (e)

r ) (E(Sn+ )r )1/r , (E Mnr )1−(r−1)/r ≤ ( r−1

which gives the results. (Just change Sn+ to |Sn | for case (ii).)

2

Hoﬀman–Jorgensen Inequalities The following inequalities show that “in probability” control of the overall sum and of the maximal summand actually gives control of moments of sums of independent rvs. Inequality 11.6 (Hoﬀmann–Jorgensen, probability form). Let X1 , . . . , Xn be independent rvs, and let Sk ≡ X1 + · · · + Xk for 1 ≤ k ≤ n. Let λ, η > 0. Then (10)

P ( max |Sk | > 3λ + η) ≤ {P ( max |Sk | > λ)}2 + P ( max |Xi | > η) . 1≤k≤n

1≤i≤n

1≤k≤n

If the Xi ’s are also symmetric, then both (11)

P ( max |Sk | > 3λ + η) ≤ {2P (|Sn | > λ)}2 + P ( max |Xi | > η)

(12)

P (|Sn | > 2λ + η) ≤ {2P (|Sn | > λ)}2 + P ( max |Xi | > η) .

1≤k≤n

1≤i≤n

and

1≤i≤n

Inequality 11.7 (Hoﬀmann–Jorgensen, moment form). Let the rvs X1 , . . . , Xn be independent, and let Sk ≡ X1 + · · · + Xk for 1 ≤ k ≤ n. Suppose that each Xi ∈ Lr (P ) for some r > 0. Then r r 4 r (13) E max |Sk | ≤ 2 (4t0 ) + 2 · 4 E max |Xi | , 1≤k≤n

1≤i≤n

where t0 ≡ inf{t > 0 : P (max1≤k≤n |Sk | > t) ≤ 1/(2 · 4r )}. If the Xi ’s are also symmetric, then r r r r (14) E|Sn | ≤ 2 (3t0 ) + 2 · 3 E max |Xi | , 1≤i≤n

where t0 ≡ inf{t > 0 : P (|Sn | > t) ≤ 1/(8 · 3r )}. Proof. Consider inequality 11.6. Let τ ≡ inf{k ≤ n : |Sk | > λ}. Then [τ = k] n depends only on X1 , . . . , Xk , and [maxk≤n |Sk | > λ] = k=1 [τ = k]. On [τ = k], |Sj | ≤ λ] if j < k, and for j ≥ k, (a)

|Sj | = |Sj − Sk + Xk + Sk−1 | ≤ λ + |Xk | + |Sj − Sk | ;

11.

MAXIMAL INEQUALITIES, SOME WITH BOUNDARIES

251

hence (b)

max1≤j≤n |Sj | ≤ λ + max1≤i≤n |Xi | + maxk<j≤n |Sj − Sk | .

Therefore, by independence, P (τ = k, max1≤k≤n |Sk | > 3λ + η) (c)

≤ P (τ = k, max1≤i≤n |Xi | > η) + P (τ = k) P (maxk<j≤n |Sj − Sk | > 2λ) .

But maxk<j≤n |Sj − Sk | ≤ 2 max1≤k≤n |Sk |, and hence summing over k on both sides yields (d)

P (max |Sk | > 3λ + η) ≤ P (max |Xi | > η) + {P (max |Sk | > λ)2 } . k≤n

i≤n

k≤n

The second inequality follows from the ﬁrst by L´evy’s inequality 10.3.3. For the symmetric case, ﬁrst note that (e)

|Sn | ≤ |Sk−1 | + |Xk | + |Sn − Sk | ,

so that P (τ = k, |Sn | > 2λ + η) (f)

≤ P (τ = k, max1≤i≤n |Xi | > η) + P (τ = k) P (|Sn − Sk | > λ) ;

and hence summing over k then yields P (|Sn | > 2λ + η) (g)

≤ P (max |Xi | > η) + P (max |Sk | > λ) P (max |Sn − Sk | > λ) . i≤n

k≤n

k≤n

The third inequality again follows from L´evy’s inequality.

2

Proof. Consider inequality 11.7. Here is the proof of (14); the proof of (13) is similar. Let u > t0 . Then, using (12) for (i), u ∞ (h) E|Sn |r = 3r ( 0 + u ) P ( |Sn | > 3t) d(tr ) by (7.4.11). ∞ ∞ (i) ≤ (3u)r + 4 · 3r u P ( |Sn | > t)2 d(tr ) + 3r u P (max1≤i≤n |Xi | > t)d(tr ) ∞ (j) ≤ (3u)r +4·3r P ( |Sn | > u) u P ( |Sn | > t) d(tr )+ 3r E(max1≤i≤n |Xi |r ) . Since 4 · 3r P (|Sn | > u) ≤ (k)

1 2

by our choice of u, applying (7.4.11) again (to (j)) gives

E|Sn |r ≤ 2 · (3u)r +

1 2

E|Sn |r + 2 · 3r E(max1≤i≤n |Xi |r ) .

Simple algebra now gives (14).

Exercise 11.2

2

Provide the details in the case of (13)

252

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

12

A Uniform SLLN

There are many other extensions and generalizations of the SLLN. A generalization that handles dependent but stationary sequences of rvs is the ergodic theorem, which is presented in many texts. Another direction of generalization is to allow the (realvalued) rvs to be replaced by random elements in a Banach space such as C[0, 1] or D[0, 1]. In fact, the Glivenko–Cantelli theorem can be treated alternatively by theorems of this kind. Another extension involves the uniformity in F (or P ) of the convergence. Theorem 12.1 (Chung) Suppose that X1 , X2 , . . . are iid P ∈ P, where the collection P satisﬁes the uniform integrability condition sup EP {|X| × 1[|X|≥λ]} = sup [|x|≥λ] |x| dP (x) → 0 (1) as λ → ∞. P ∈P

P ∈P

Then, for every > 0, ¯ m − EX| ≥ → 0 (2) sup P sup |X P ∈P

Proof.

as n → ∞.

m≥n

It follows easily from (1) that supP ∈P EP |X| ≤ (some M ) < ∞;

(a)

the constant M will appear repeatedly in the following calculations. Let > 0. We will prove (2) by producing an N0 ≡ N0 ( ) such that the left side in (2) is less than

for n ≥ N0 . Here is the choice that works. First, by (1), we can choose N ≡ N ( ) so large that 16 2 √ (b) + 6 sup |x| dP (x) + sup [|x|≥N ] |x| dP (x) < /4. 1/4 ] [|x|≥N 2

N P ∈P P ∈P For this N = N ( ) we can then choose N0 = N0 ( ) ≥ N ( ) so large that both 4N 2 (c) N0 supP ∈P [|x|≥N0/4N ] |x| dP (x) < /2 and (d)

NM N0

+ supP ∈P

[|x|≥N ]

|x| dP (x) < /2.

To show that this choice of N and N0 works, we set (e)

Xk ≡ Xk 1[|Xk |
and

Xk ≡ Xk − EXk ,

and note that (f)

E[(Xk )2 ] ≤ E[(Xk )2 ] =

[|x|
x2 dP (x).

Then we can write P (|Sn − ESn |/n > for some n ≥ N0 )

12.

A UNIFORM SLLN n N = P n1 k=1 Xk + n1 k=N +1 (Xk − EXk )

n + n1 k=N +1 EXk − n1 ESn > for some n ≥ N0

X | >

/4 for some n ≥ N ≤ P | n1 N k 0 k=1 1 n

+P | n k=N +1 (Xk − EXk )| > /4 for some n ≥ N0

(g)

≡ I + II ,

(h) since | n1

(i)

253

n

k=N +1 EXk

− n1 ESn |

n N ≤ | n1 k=N +1 EXk − EXk | + | n1 k=1 EXk | ≤ [|x|≥N ] |x| dP (x) + NnM ≤ /2

for all n ≥ N0 ,

uniformly in P, by (d).

Now we bound the two terms in (h). The ﬁrst term is easy, since (j) I ≤ P (| N10 N k=1 Xk | > /4) 4N ≤ N P (|X| > N0 /4N ) ≤ N N [|x|>N0 /4N ] |x| dP (x) 0 (k)

< /2

uniformly in P, by (c).

To handle II, ﬁrst note that it follows from the proof of Kronecker’s lemma that uniformly in P, P (| n1 nk=N +1 (Xk − EXk )| > /4 for some n ≥ N0 ) → 0 , provided uniformly in P that n P (| k=N +1 (Xk − EXk )/k | > /4 for some n ≥ N0 ) → 0 . Since N0 ≥ N , it suﬃces to show that uniformly in P we have n P (AN, ) ≡ P (| k=N (Xk − EXk )/k | > /4 for some n ≥ N ) → 0. (l) Now, (m)

P (AN, ) = P (AN, ∩ [Xk = Xk for all N ≤ k < ∞])

(n)

+P (AN, ∩ [Xk = Xk for some N ≤ k < ∞]) n ∞ ≤ P (| k=N (Xk − EXk )/k| > /4 for some n ≥ N ) + k=N P (|Xk | ≥ k) ∞ 2 1 ≤ 16 k=N k2 [|x|
(o)

by Kolmogorov’s inequality, and (q)–(r) below √2 + 6 ≤ 16 |x| dP (x) + [|x|≥N ] |x| dP (x) 2 1/4 [|x|≥N ] N by (s)–(u) below

(p)

< /4

uniformly in P, by (b).

254

CHAPTER 10. WLLN, SLLN, LIL, AND SERIES

Inequality (n) holds, since ∞ ∞ ∞ (q) k=N P (|X| ≥ k) = k=N j=k P ([|X|] = j) = (r)

≤

∞

j

j=N

∞ j=N

P ([|X|] = j)

k=N

j P ([|X|] = j) ≤

[N,∞) x dF (x).

Inequality (o) holds by interchanging the summations and then breaking up the resulting sum into three regions as follows: ∞ 2 1 (s) k=N k2 [|x|
(t)

=

∞

=

∞

= ≤

1 k=N k2

j=0

j=0

[N 1/4 ]−1 j=0

2 N

2 [j≤|x|<j+1] x

2 [j≤|x|<j+1] x

+

(u)

k−1

+

N −1

2 [|x|
∞

2 j=N j+1

≤

√2 N

+4

≤

√2 N

+6

dP (x)

j=[N 1/4 ]

+

dP (x) +

2 N

∞

1 k=max{N,j+1} k2

∞

j=N

(ditto)

N −1

2 [j≤|x|<j+1] x

dP (x)

j=[N 1/4 ]

dP (x)

dP (x)

[|x|≥N 1/4 ]

|x| dP (x) + 2

[|x|≥N 1/4 ]

|x| dP (x)

2 [j≤|x|<j+1] x

[|x|≥N ]

|x| dP (x)

Thus (o) holds. Combining (k) and (p) into (h) completes the proof. See also (16.1.37).

2

Chapter 11

Convergence in Distribution 1

Steins Method for CLTs

Notation 1.1 Is W asymptotically normal? Let Φ denote the df of a N (0, 1) rv Z, let φ ≡ Φ , and let Φh ≡ Eh(Z) for an appropriate function h. Fix ∈ (0, 1). For each ﬁxed real z we deﬁne hz, (·) to equal 1, to be smooth with 0 ≤ hz, (x) ≤ 1, to equal 0 according as the value of x is in (−∞, z], [z, z + ], [z + , ∞). Here “smooth” will always imply at least absolute continuity. Then (1)

FW − Φ = {supz |P (W ≤ z) − Φ(z)|} ≤ {supz |Ehz, (W ) − Φhz, |} + √2π

follows trivially from 1(−∞,z] (·) ≤ hz, (·) ≤ 1(−∞,z+] (·). We now further require that hz, (·) is either made up of a ﬁnite number of linear pieces or else has |h (x)| bounded for all x on the line. In only these two cases will we proceed and also write h ; let H ≡ {all such hz, (·)}. Let H0 denote the subset for which the hz, (·) are linear on [z, z + ], as in ﬁgure 1.1 below. 2 Motivation 1.1 (Stein’s approach) Let Hc denote the class of all continuous h on the line for which Φ|h| ≡ E|h(Z)| < ∞. Now deﬁne f (·) on the line via x 2 2 (2) f (x) = ex /2 −∞ [h(u) − Φh] e−u /2 du . Note that this f solves Stein’s equation (3)

f (x) − xf (x) = h(x) − Φh

for all real x,

and for it f (·) is continuous. (Just do the trivial diﬀerentiation to conﬁrm this.) Thus, for any hz, (·) ∈ H (and any > 0), we have (4)

FW − Φ ≤ {supz |Ehz, (W ) − Φhz, |} +

√ 2π

≤ {supz |E[fz, (W ) − W fz, (W )]|} +

.

√ 2π

Nice! More generally, for any h ∈ Hc for which both E|h(Z)| and E|h(W )| are ﬁnite, we have (5)

|Eh(W ) − Eh(Z)| = |E{f (W ) − W f (W )}| . 255

256

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

We will soon show how useful the reformulation shown in (4) of the task (1) can be. But the next lemma will ﬁrst establish some useful properties of any solution fz, (·) of Stein’s equation that is associated with some hz, (·) ∈ H as above. And great care will be given to the case where hz, (·) is linear on [z, z + ]. It is the H0 case that counts! 2

hz – ε,ε(·)

hz,ε(·)

z

z+ε

Figure 1.1 The function hz,ε(·).

Lemma 1.1 (Stein’s lemma) Fix ∈ (0, 1), and ﬁx a real number z. Fix h ≡ hz, in H, and then let f ≡ fz, denote the solution of Stein’s equation (2) for this h. For any such h we have f (x) = [1 − Φh] × Φ(x)/φ(x)

and

according as x ≤ z or z + ≤ x; and (6) |xf (x)| ≤ 1, |f (x)| ≤ π/2 < 1.3,

f (x) = Φh × [1 − Φ(x)]/φ(x) and |f (x)| ≤ 2

for all x.

Using (3) for all real x and w (6) gives |f (w + x) − f (w)| = |xf (w + x) + w[f (w + x) − f (w)] + [h(w + x) − h(w)]| (7)

≤ |x|(1.3 + 2|w|) + d(w)

with

d(w) ≡ |h(w + x) − h(w)| ∈ [0, 1] .

Especially: The length of the interval of w’s on which d(w) > 0 does not exceed |x| + . Proof. Now, (2) gives |f (x)| ≤ Φ(x)/φ(x). Then |f (x)| ≤ |1 − Φ(x)|/φ(x), ∞ since f (x) = − exp (x2 /2) x [h(u) − Φh] exp (−u2 /2) du follows from (2). Thus |xf (x)| ≤ 1 follows from Mills’ ratio (10.7.7). This then gives |f (x)| ≤ 1 for |x| ≥ 1 and |f (x)| ≤ Φ(0)/φ(0) = π/2 for |x| ≤ 1. Then |f (x)| ≤ xf (x) + h − Φh ≤ 1 + 1 = 2. Consider d(w) ≡ |hz, (w + x) − hz, (w)|. Now, d(·) will be 0 or 1 as indicated on the four regions shown in ﬁgure 1.2, and it is in [0, 1] on the two narrow strips. Thus for ﬁxed values of z and x, the length of the interval of w’s on which d(w) is nonzero is at most |x| + (and there 0 < d(w) ≤ 1); this seems new and slick, and it is crucial! 2

1.

STEINS METHOD FOR CLTS

257

w

|1 – 0| = 1

|0 – 0| = 0 z+ε z

|1 – 1| = 0

|0 – 1| = 1

x Figure 1.2 Values of d(w) ≡ |h(w + x) – h(w)|.

Lemma 1.2 (i) (Stein) Fix h ∈ H, and consider the associated f of (2). Then both f (x) → 0 and f (x) → 0 as x → ±∞ and (8)

|f (y) − f (x)| ≤ 2 h |x − y|

for all x and y.

(Note that h = 1/ for any h linear on [z, z + ].) The bound (8) holds, since (9)

f (x) = f (x) + xf (x) + h (x) = (1 + x2 )f (x) + x[h(x) − Φh] + h (x)

holds whenever h (x) exists; and this leads both to f (x) → 0 as x → ±∞ and to (10)

|f (x)| ≤ 2 h

whenever h (x) exists.

[Carefully introduce any h term below, as only an n−1/4 theorem is then possible, but not an n−1/2 theorem.] (ii) The choice h(x) ≡ ψ((x − z)/ ) on [z, z + ] with 1 1 (11) ψ(t) ≡ t exp(−1/(s(1 − s))) ds/ 0 exp(−1/(s(1 − s))) ds for 0 ≤ t ≤ 1 has h = ψ / < 2.61/ < 3/ . This h z, (·) has an inﬁnite number of bounded and continuous derivatives on the whole line.

258

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Proof. l’Hospital’s rule shows that the expressions for f go to 0 as x → ±∞. Diﬀerentiate f to get f on [z, z + ]c . l’Hospital’s rule and the above limits for f at ±∞ show that f (x) → 0 as x → ±∞. Then (9) is trivial, by diﬀerentiating (3). And (9) shows that for x ≤ z we have f (x) = (1 − Φh)[(1 + x2 )Φ(x)/φ(x) + x]; and this goes to 0 as x → −∞, by l’Hospital’s rule. Likewise, f (x) → 0 as x → ∞. The Lipschitz condition (8) will follow from the fundamental theorem of calculus and (10), since f (·) is absolutely continuous by (3). Finally, (ii) is straightforward to show. Just plot ψ (·) to obtain the bound 2.61. ¯ ≡ 1 − Φ(·), we have Thus, only (10) remains. Letting Φ(·) ∞ h(x) − Φh = −∞ [h(x) − h(y)]φ(y) dy = (a)

=

x −∞

x ∞ y [ y h (z) dz]φ(y) dy − x [ x h (z) dz]φ(y) dy

x

−∞ h

(z)Φ(z) dz −

∞ x

¯ h (z)Φ(z) dz .

Thus f (w) = (b)

w −∞

x ∞ ¯ [ −∞ h (z)Φ(z) dz − x h (z)Φ(z) dz]φ(x) dx/φ(w)

w ∞ ¯ ¯ = −{Φ(w)/φ(w)} h (z)Φ(z) dz − {Φ(w)/φ(w)} w h (z)Φ(z) dz . −∞

Finally, plugging (a) and (b) into (9) gives (c)

¯ f (w) = h (w) − {(1 + w2 )[Φ(w)/φ(w)] − w} × −{(1 + w2 )[Φ(w)/φ(w)] + w} ×

∞ w

w −∞

h (z)Φ(z) dz

¯ h (z)Φ(z) dz ,

where the two term within braces {· · ·} in (c) are positive for all w. (This positivity is made clear simply by plotting these two functions (using Splus, say).) It then follows, after changing the order of integration, that (making a third plot to obtain the bound 2 in (e) below) (d)

¯ − w}[wΦ(w) + φ(w)] |f (w)| ≤ h {1 + {(1 + w 2 )[Φ(w)/φ(w)] ¯ +{(1 + w2 )[Φ(w)/φ(w)] + w}[−wΦ(w) + φ(w)]}

(e)

≤ 2 h .

2

Remark 1.1 We will soon prove the CLT by showing (essentially) the convergence of expectations for just the class H0 of those hz, (·) functions that are linear on [z, z + ], with any > 0. It would also suﬃce to show such convergence for other classes of functions. Deﬁnitions of other appropriate classes G will be given in section 11.7 and section 11.8. The study of characteristic functions in chapters 13 and 14 is merely an appeal to yet a diﬀerent G class. Immediately after sections 11.7 and 11.8 the reader could turn to chapter 19, where such topics are treated on general metric spaces for various G classes. 2

1.

STEINS METHOD FOR CLTS

259

2 Theorem 1.1 (Berry–Esseen CLT) Suppose the rvs Xnk ∼ ), for = (µnk , σnk n 2 1/2 1 ≤ k ≤ n, are independent. Let Yk ≡ (Xnk − µnk )/( 1 σnk ) . Now deﬁne n 2 3/2 n n ∼ (0, 1) (12) ) . W ≡ k=1 Yk = and γ ≡ ( 1 E|Xnk − µnk |3 )/( 1 σnk

There is uniform convergence to normality when γ ≡ γn → 0, in that (13)

FW − Φ ≤ 9 γ .

Proof. Let Wk ≡ W − Yk = j =k Yj . Further, let Y1 , . . . , Yn be independent copies of the rvs Y1 , . . . , Yn . Specifying that h ≡ hz, is linear on [z, z + ], for the associated f we have E[ W f (W )] = n1 E[ Yk f (W )] n (a) = 1 E{Yk [f (Wk )+Yk f (Wk +Yk∗ )]} for some Yk∗ between 0 and Yk (b) = n1 [0×Ef (Wk )]+ n1 E{Yk2 f (Wk +Yk∗ )} as Yk ,Wk are independent n = 1 E{Yk2 f (Wk + Yk∗ )} (14) for Yk∗ between 0 and Yk . We can also write (this is cute, crucial, and trivial—it uses W ∼ = Wk + Yk ) n n (15) Ef (W ) = ( 1 EYk2 )E[f (W )] = 1 E[ Yk2 f (Wk + Yk )] . We thus obtain from (4) that (c)

(16)

FW − Φ ≤ supz |Eh(W ) − Φh| + √2π ≤ supz |E[f (W ) − W f (W )]| + √2π n ≤ supz 1 E Yk2 [f (Wk + Yk ) − f (Wk )] + [f (Wk ) − f (Wk + Yk∗ )] +

(17)

≤ + supz

n 1

E Yk2 (|Yk | + |Yk |)(1.3 + 2|Wk |)

+ E[hz, (Wk +

(18) (19)

(d)

Yk )

−

hz, (Wk )|Yk

] + E[hz, (Wk +

Yk∗ )

− hz, (Wk )|Yk ]

by applying (7) twice, and its d(·) bound twice in the next step n = 1 E{Yk2 {(|Yk | + |Yk |) (1.3 + 2|Wk |) + |Yk | + + |Yk | + }} +

n n ≤ 1 {E|Yk |3 + E(Yk2 ) E|Yk |}(1.3 + 2E|Wk | + 1) + 2 ( 1 EYk2 ) +

by the independence of Yk and Wk n ≤ 1 {E|Yk |3 (1.3 + 2 + 1) + (E|Yk |3 )2/3 (E|Yk |3 )1/3 (1.3 + 2 + 1)} + 2 +

n ≤ 8.6 1 E|Yk |3 + 3 ≤ 9 γ + 3

with > 0 arbitrary. [It is my “especially” remark following Stein’s lemma that makes this proof so nice (and the Lindeberg–Feller proof below).] 2

260

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

2 Theorem 1.2 (Lindeberg–Feller CLT) Suppose Xnk ∼ ) are = Fnk (µnk , σnk n 2 n 2 independent, for 1 ≤ k ≤ n. Let sdn ≡ 1 σnk . Deﬁne Zn ≡ 1 (Xnk − µnk )/sdn . The following are equivalent:

(20)

FZn − Φ → 0

(21)

LFn

≡

n k=1

[max1≤k≤n P (|Xnk − µnk |/sdn > )] → 0 .

and

[|x−µnk |/sdn ≥]

x − µnk sdn

2 dFnk (x) → 0

for all > 0 .

[Rvs that satisfy the second condition in (20) will be called uniformly asymptotically negligible (and abbreviated uan).] [Condition (21) is Lindeberg’s condition; Feller established its necessity.] Moreover, Lindeberg’s condition implies both (22)

Mn →p 0

and

2 [ max1≤k≤n σnk /sdn ] → 0 ,

for the maximum Mn ≡ [max1≤k≤n |Xnk − µnk |/sdn ]. Proof. Let the n Yk ≡ (Xnk − µnk )/sdn , and let Fk denote n df of the rv Yk . Then W ≡ Zn = k=1 Yk ∼ = (0, 1). Deﬁne the average df F¯n ≡ 1 Fk /n. Lindeberg’s (21) gives 2 (a) σnk /sd2n ≤ [ [|y|≤] y 2 dFk (y) + [|y|≥] y 2 dFk (y) (b)

≤ 2 + LFn ≤ ,

for n ≥ (some n ) .

2 Thus [maxk σnk /sd2n ] → 0, giving the uan condition. Since we easily calculate that (c) LFn ≥ 2 n1 P (|Xnk − µnk |/sdn ≥ ) ,

Lindeberg’s condition also implies Mn →p 0 via (10.3.14). (·) associated with linear hz, (·) are uniformly Lipschitz Fix > 0. Since the fz, on R (by (8)), the term in (16) is bounded by 5 , provided that all |Yk | ∨ |Yk | never exceeds the appropriate δ = 2 (note (8) again). Thus (16) and f ≤ 2 give the upper bound (23) FW − Φ ≤ 4 n1 [|y|≥2 ] y 2 dFk (y) + 4 n1 EYk2 P ( |Yk | ≥ 2 ) + 5

(d)

≤4

n 1

[|y|≥2 ]

y 2 dFk (y) + 4 [maxk P ( |Yk | ≥ 2 )] + 5 .

Thus Lindeberg’s LFn -condition (21) (which implies uan) implies FW − Φ ≤ 6

for n ≥ (some n ), for > 0 arbitrary. Thus FW − Φ → 0. Note the nice format n (e) LF = y 2 dF¯n (y) when all µnk = 0 . [|y|≥]

The necessity will be established in chapter 14, using characteristic functions. However, we will use the necessity in this chapter. 2

1.

STEINS METHOD FOR CLTS

261

Many interesting statistics can be put in the form W + ∆, where W is a “nice” rv. For us, nice will mean that we have already established a useful bound on FW − Φ. Can we then claim a useful bound on FW +∆ − Φ as well? The previous methods are also well suited for establishing such bounds, as the next lemma shows. This applies beautifully to U -statistics; see section 17.1. [The reader will have suﬃcient background after this section to read all of chapter 17 except for the short subsections on the SLLN and the functional CLT for U -statistics.] Lemma 1.3 (Berry–Esseen potential)

For any rvs W and ∆ we have

FW +∆ − Φ ≤ FW − Φ + 4 E|W ∆| + 4 E|∆| . √ √ [The ∆-terms are bounded by 4 EW 2 E∆2 + 4 E∆2 ; and often EW 2 ≤ 1 will provide an easy upper bound.] (24)

Proof.

Recall that a = b ⊕ c means that |a − b| ≤ c. Now, FW +∆ − Φ ≤ FW +∆ − FW + FW − Φ ,

where we take the diﬀerence again for FW +∆ − FW = [FW +∆ − Φ] − [FW − Φ] (a)

= E[f (W + ∆) − (W + ∆)f (W + ∆)] − E[f (W ) − W f (W )] ⊕

by (4), twice = E[f (W + ∆) − f (W )] − E(W [f (W + ∆) − f (W )] − E[∆f (W + ∆)] ⊕ .

Using Stein’s lemma on the ﬁrst term gives FW +∆ − FW (b)

≤ E{|∆| (1.3 + 2|W |) + |∆| + } + f E|W ∆| + f E|∆| +

≤ (2 + f ) E|W ∆| + (2.3 + f ) E|∆| + 2

(c)

≤ 4 E|W ∆| + 4 E|∆| + 2 ,

with > 0 arbitrary. (Note also a related (13.5.7).)

Exercise 1.1 (25)

2

Let rvs X and ∆ be arbitrary. Suppose G (·) = g(·). Then

FX+∆ (·) − G(·) ≤ FX (·) − G(·) + g + P (|∆| > ) for > 0 .

[To avoid loss, a strong bound like P (|∆| > ) ≤ K is needed, but not moments.]

262

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

and Motivation 1.2 (Alternative Stein formulation) Consider both W ≡ W (X) Y ), and deﬁne W ≡ W + V . Suppose that W and W are coupled via V ≡ V (X, (26)

(W, W ) ∼ = (W , W )

(a) :

= −λW (b) : E(V |X)

(cc) : EV 2 = 2λ

for some 0 < λ < 1. Thus (W )] = − 1 E[V f (W )] E[W f (W )] = − λ1 E[E(V |X)f λ =

1 2λ

E{V [f (W ) − f (W )]}

by the symmetry of the exchangeability (26)(a) (27)

=

1 2λ

E{V 2 [f (W + V ∗ )}

for some V ∗ between 0 and V . Then for any linear h ≡ hz, , plugging the associated f ≡ fz, into (5) gives |Eh(W ) − Φh| = |E{f (W ) − W f (W )}|

(28)

(29)

=

1 2λ

|E{E(V 2 ) f (W ) − V 2 f (W + V ∗ )}|

≤

1 2λ

− EV 2 }f (W )] | + | E [{E(V 2 |X)

1 2λ

| E [V 2 {f (W + V ∗ ) − f (W )}] |

≤ f

1 2λ

2 E |E{V 2 |X}−EV |+ 21λ E {V 2 [|V |(1.3+2|W |)+|V |+ ]} by (7)

≤ f

1 2λ

− EV 2 | + E |E{V 2 |X}

≤

1 λ

− EV 2 | + E |E{V 2 |X}

1 λ

2 2λ

{1.15 E |V |3 + E |V 3 W |} +

2λ

EV2

{1.15 E |V |3 + { E V 4 }3/4 {E W 4 }1/4 } +

2λ

.

And since > 0 is arbitrary, the recipe of (4) leads to (30)

− EV 2 | + 1.15 E |V |3 + { E V 4 }3/4 {E W 4 }1/4 } . FW − Φ ≤ λ1 { E |E{V 2 |X}

This bound (30) is useful in additional cases. [In the context of the Berry–Esseen theorem, and exploiting the independence still available at line (28), the author can obtain the bound 14γ via this approach.] One can replace the second term in (28) by 21λ f E |V |3 , by (8). This yields FW − Φ ≤ (31)

≤

1 λ

1 λ

− EV 2 | + E |E{V 2 |X}

− EV 2 | + E |E{V 2 |X}

1 λ×

1 (2π)1/4 λ1/2

E |V |3 +

√ 2π

{E |V |3 }1/2 ,

√ by specifying 2 ≡ 2π E|V |3 /λ. [We replaced h by 1/ , since these h’s are linear on [z, z + ].] Recall my “especially” remark at the end of lemma 1.2(i), noting how the

terms in the numerator and denominator had to be “balanced” in (31). The result was an inferior compromise. (Read exercise 1.2 below.) 2

1.

STEINS METHOD FOR CLTS

263

Exercise 1.2 (Comparing diﬀerent Stein-type strategies) (a) Establish a bound on (28) in the context of the Berry–Esseen theorem. (b) Establish a bound on (30) in the context of the Berry–Esseen theorem. [This costs a fourth-moment assumption, as opposed to a third-moment assumption.] (c) Establish a bound on (31) in the context of the Berry–Esseen theorem. [Now, you will get convergence only at the rate 1/n1/4. This is your penalty for ignoring my “especially” remark in the last sentence of lemma 11.1.1. (Note also the diﬀerence between (17.2.6) and (17.2.7) in Hoeﬀding’s combinatorial CLT below.)] (Read the paragraph below (31).) Exercise 1.3 Suppose Zn →d Z0 , and let Pn (B) ≡ P (Zn ∈ B) for all Borel sets B and all n ≥ 0. Let C denote an open subset of the real line. Show that lim inf n Pn (C) ≥ P0 (C). In this entire section we have used only the self-contained discussion 10.3.1 with (10.3.14) from chapter 10.

264

2

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

˜ ˇ Winsorization and Truncation

Deﬁnition 2.1

(a) Call the rvs Xn1 , . . . , Xnn weakly negligible if

Mn ≡ [ max1≤k≤n |Xnk | ] →p 0 . (b) Call them uniformly asymptotically negligible (or uan) if max1≤k≤n P ( |Xnk | ≥ ) → 0

for all > 0.

Notation 2.1 (Weak negligibility in the CLT context) Let Xn1 , . . . , Xnn be independent rvs having dfs Fn1 , . . . , Fnn . Let θ > 0 be given. Deﬁne xθn by requiring [−xθn , xθn ] to be the smallest closed and symmetric interval to which F¯n ≡ 1 n 1 n ¯ 1 Fnk assigns probability at least 1 − θ/n. Let Pn (x) ≡ n 1 P (|Xnk | > x) n denote the average tail probability, and then let K n denote the qf of the df 1− P¯n (·). Note the quantile relationship xθn = K n (1 − θ/n). ˜ nk denote Xnk Winsorized outside [−xθ0 n , xθ0 n ] (often, θ0 ≡ 1). Let µ ˜nk Let X 2 and σ ˜nk denote the resulting means and variances, and then set both µ ˜n ≡ n1 n1 µ ˜nk n 2 √ and σ ˜n2 ≡ n1 1 σ ˜nk . Applying discussion 10.3.1 to the rvs |Xnk |/ n σ ˜n , whose √ θ/nth quantile is xθn / n σ ˜n , shows that the following are equivalent conditions for weak negligibility: √ (1) ˜n →p 0 . Mn ≡ [ max |Xnk | ]/ n σ 1≤k≤n

(2) (3)

√ xθn / n σ ˜n → 0 for all 0 < θ ≤ θθ0 . √ n ˜n ) → 0 for all 0 < ≤ 1 . 1 P (|Xnk | > n σ

2

Theorem 2.1 (CLT using Winsorization) Let Xn1 , . . . , Xnn be independent rvs having dfs Fn1 , . . . , Fnn . If any/all of (1), (2), and (3) holds, then both √ ¯n − µ (4) Z¯n ≡ n [X ˜n ]/˜ σn →d N (0, 1) and √ ˜ n ≡ [max |Xnk − µ (5) M ˜nk | ]/ n σ ˜n →p 0 . k

˜ θ denote Xnk Winsorized outside [−xθn , xθn ]; Notation 2.2 For 0 < θ ≤ θ0 let X nk θ θ let µ ˜nk and σ ˜nk denote the resulting means and standard deviations, and deﬁne n θ n θ 2 2 ˜ θn ≡ 1 n X ˜ θ , and let both µ ˜θn ≡ n1 1 µ ˜nk and σ ˜θn ≡ n1 1 (˜ σnk ) . Let X nk 1 n √ θ ˜ θn − µ ˜θ − µ ˜ θn ]/˜ σθn . Deﬁne third central moments γ˜nk ≡ E |X ˜ θnk |3 , Z˜θn ≡ n [X nk n θ ˜ θn ≡ [ maxk |Y˜ θ | ] denote the maximal summand. and let γ˜θn ≡ 1 γ˜nk . Let M nk √ θ θ θ ˜ ˜ Call the rvs Y ≡ (X − µ ˜ )/ n σ ˜θn the associated rvs. 2 nk

nk

nk

˜ n →p 0 by the Proof. Now, all µ ˜nk ∈ [−xθ0 n , xθ0 n ]. Thus Mn →p 0 implies M triangle inequality. n n θ ˜ )), we have Observe that with δn ≡ P ( 1 Xnk = 1 X nk √ ¯ ˜ (a) P (Zn ≤ z) = P (Zθn × (˜ σθn /˜ σn ) + n(˜ µθn − µ ˜ n )/˜ σn ≤ z) ⊕ δn .

2.

˜ ˇ WINSORIZATION AND TRUNCATION

265

n n θ 3 ˜ ) ≤ θ is trivial, and since F ˜ − Φ ≤ 9 γ˜θn /√n σ Since P ( 1 Xnk = 1 X ˜θn nk Zθn by the Berry–Esseen theorem, (4) will follow immediately from showing that (6) below holds. [Recall that a = b ⊕ c means |a − b| ≤ c.] 2 Inequality 2.1 (6)

γ˜θn (i) √ 3 nσ ˜θn

For each 0 < θ ≤ θ0 , the inequalities (7), (8), and (9) lead to √ n|µ ˜θn − µ ˜n | σ ˜2 → 0, (j) θn → 1 , (k) → 0. 2 σ ˜n σ ˜n

˜n = σ ˜1n . Bounding Proof. For convenience, set θ0 = 1. We then note that σ θ 3 θ ˜θ − µ one power of |X ˜ | by 2 x in the integrand of each γ ˜ gives θn nk nk nk (7)

x x γ˜ √ θn3 ≤ 2 √ θn ≤ 2 √ θn → 0, nσ ˜θn nσ ˜θn nσ ˜n

using (2). Since the probability outside [−x1n , x1n ] is at most 1/n, √ xθn n|µ ˜θn − µ ˜n | (8) ≤√ →0 σ ˜n nσ ˜n using (2). We need some notation before turning to (6)(j). Let V˜θn denote the ˜ θ , and set V˜n ≡ V˜1n . Now, F¯n assigns at most 1/n probability second moment of X nk to the complement of the interval [−x1n , x1n ], and on this complement the integrand of V˜θn never exceeds 4x2θn . And so (note also inequality 7.10.2) (a) (b)

2 2 2 2 0 ≤ [˜ σθn −σ ˜n2 ]/˜ σθn = {V˜θn − V˜n }/˜ σθn − {˜ µ2θn − µ ˜ 2n }/˜ σθn 2 2 = {V˜n ⊕ (2 xθn )2 /n − V˜n }/˜ σθn − {[ µ ˜ n ⊕ xθn /n]2 − µ ˜2n }/˜ σθn 2 2 ≤ 4x2θn /[n˜ σθn ] + 2 [ |˜ µn | xθn ]/[n˜ σθn ]+0

(9)

2 2 ≤ 6 {x2θn /n˜ σθn } ≤ 6 {x2θn /n˜ σ1n }→0

¯ by (2). Thus (6) holds, and it gives the normality √ of Zn in (4). [We have actually established (4) under the slightly weaker xθn / n σ ˜θn → 0 for all 0 < θ < θ0 .] 2 Exercise 2.1

Show that Sn /˜ σn →p 1 holds in the context of theorem 2.1.

Theorem 2.2 (Asymptotic normality) Let Xn1 , . . . , Xnn be independent rvs. ˇ nk be the truncated rv that equals Xnk or 0 according Let σ > 0. Fix c > 0. Let X as |Xnk | < c or as |Xnk | ≥ c. Let Mn ≡ [max1≤k≤n |Xnk | ]. The following are equivalent: n 2 (10) for uan rvs Xn1 , . . . , Xnn . k=1 Xnk →d N (µ, σ ) n for all > 0 (equivalently, Mn →p 0) and k=1 P (|Xnk | ≥ ) → 0 (11) n n 2 ˇ nk → µ ˇ nk ] → σ2 . µ ˇn ≡ EX and σ ˇn ≡ Var[X k=1

k=1

If (11) holds for one c > 0, then it holds for all c > 0. [These results also hold with ˜ nk , µ ˇ nk , µ Winsorized quantities X ˜n , σ ˜n2 replacing the truncated quantities X ˇn , σ ˇn2 .]

266

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

n n n ˇ Proof. Suppose (11). Then P ( 1 Xnk = 1 Xnk ) ≤ 1 P (|Xnk | ≥ c) → 0. ˇ nk − µ Thus we need only show that Zˇn ≡ ( n1 X ˇn )/ˇ σn →d N (0, 1). It suﬃces to ˇ n of (11.1.21) satisﬁes LF ˇ n → 0. We will do so presently. verify that Lindeberg’s LF ˇ nk | →p 0. Thus First observe that Mn →p 0, by (10.3.14). Thus max |X (a)

ˇ nk | → 0, m ˇ n ≡ max |ˇ µnk | ≤ E max |X

by the DCT with dominating function c. Thus for n ≥ (some n ) we have (b)

m ˇ n ≤ σ/8,

and also

σ ˇn ≥ σ/2 .

ˇ n of (11.1.21) satisﬁes Then the Lindeberg–Feller quantity LF ˇ ≡ 12 n (c) LF [x − µ ˇnk ]2 dFnk (x) n k=1 [|x−ˇ σ ˇ µnk |≥ σ ˇn ]∩[|x|≤c] n

c+ σ/8 σ/2

(d)

≤

(e)

≤

(f)

→0

c+ σ/8 σ/2

2 n

k=1 [|x−ˇ µnk |≥ σ ˇn ]∩[|x|≤c]

2 n k=1

P (|Xnk | ≥ σ/4)

dFnk (x)

for n ≥ n , by (b)

by (b) again

by (11).

˜ nk and Thus (10) holds. [For the Winsorized version, just recopy this proof using X dF˜nk .] We will not prove the converse. 2

Analogous Results for QFs This subsection primarily contains technical details to be used below. Notation 2.3 Let Xn1 , . . . , Xnn be row-independent rvs having dfs Fnk and qfs ˜ nk denote Knk Winsorized outside Knk . Let an ≡ θn /n where θn 0, and let K (an , 1 − an ); so it equals Knk+ (an ), Knk (t), Knk (1 − an ) according as t ≤ an , an < ˜ nk ≡ K ˜ nk (ξnk ), for row-independent Uniform(0, 1) t < 1 − an, 1 − an ≤ t. Deﬁne X 2 ∼ ˜ rvs ξnk . Writing Xnk = (˜ µnk , σ ˜nk ) with third absolute central moment γ˜nk and ﬁrst absolute central moment m ˜ nk deﬁnes our notation for these various moments. Let n 2 1/2 n n ˜sdn ≡ [ 1 σ ˜nk ] and γ˜n ≡ 1 γ˜nk . We next deﬁne µ ˜n ≡ n1 k=1 µ ˜nk and then n 2 1/2 2 1 1 2 let σ ˜n ≡ n ˜sdn = n [ 1 σ ˜nk ] . We further let n n ˜ nk − µ (12) Z¯n ≡ k=1 (Xnk − µ ˜ nk )/ ˜sdn and Z˜n ≡ k=1 (X ˜nk )/ ˜sdn . To measure the potential outliers of each Xnk , we deﬁne (13)

qnk ≡ Knk (1 − an ) − Knk+ (an )

and

qn ≡ [max1≤k≤n qnk ] .

2

Now, µ ˜nk is necessarily contained in the obvious interval of length qnk . It is ˜ nk − µ important and trivial (since sup |X ˜nk |/˜sdn ≤ qnk /˜sdn ) that (14)

γ˜n / ˜sd3n ≤ [max1≤k≤n qnk ]/ ˜sdn = qn / ˜sdn .

2.

˜ ˇ WINSORIZATION AND TRUNCATION

267

The simple inequality (12) gives useful results. [The basic idea we employ is the ˜ nk = Xnk for all 1 ≤ k ≤ n) ≥ (1 − 2θn/n)n → 1. Thus trivial observation that P (X Z˜n will have the same limiting distribution as Z¯n . We will study Z˜n instead of Z¯n , because it is normed and because (14) is applicable.] The normalized summands ˜ nk − µ ˜nk )/˜sdn Y˜nk ≡ (X of Z˜n will be called the equivalent summands. These are particularly appealing to statisticians because of their natural normalization, and because the worst potential outliers have been eliminated. Applying Berry–Esseen’s theorem 11.1.1 via (14) immediately gives part (a) of the following theorem. We will establish part (b) in the remarks that follow the theorem. Part (b) is a useful result, and in the iid case it is a good result. [Our goal in this subsection is remark 2.3 below.] (The case where all θn = θ is also covered by the results below; and since (1 − θ/n)n → exp(−2θ) > 1 − 2θ, this case can also be made to work.) 2 Theorem 2.3 (a) Let Xnk denote any array of row-independent rvs, and appeal to notation 2.2. Whenever the equivalent summands Y˜nk above satisfy γ˜n /˜sd3n → 0 (which is implied by qn /˜sdn → 0, as in (14)), it follows that (15) Z¯n ≡ nk=1 (Xnk − µ ˜ nk )/ ˜sdn →d N (0, 1) and Z˜n →d N (0, 1) . (b) If Z˜n →d N (0, 1) for the uan equivalent summands Y˜nk above, then n 2 (16) an k=1 qnk /˜sd2n → 0 . Remark 2.1 (Necessity) Fix 0 < < 1, and ﬁx 0 < θ < θ0 . Line (i) of the general necessity proof of the Lindeberg–Feller theorem in section 14.2 will establish from the LFn condition that appears in (11.1.21) that for all n ≥ (some n,θ ) we have the bound n n 2 1 [|y|≥] y dFnk (y) ≤ θ + θ 1 [|y|≥] dP (Ynk ≤ y) (17) n = θ{1 + 1 P (|Ynk | ≥ )} . [Thus bounding lim n1 P (|Ynk | ≥ ) will imply that the Lindeberg–Feller term on the left-hand side has limit zero.] When that same necessity argument is read in qf notation (that is, replace x, dFnk (x), and the old interval In in line (e) of that ˜ nk (t), dt, and the new interval In ≡ (an , 1 − an )), that same necessity proof by K argument (for arbitrary 0 < an < 12 ) leads (via lines (j) and (l) of that proof) to n n ˜nk )/˜sdn ]2 dt ≤ + 1 (an ,1−an )c dt 1 (an ,1−an )c [(Knk (t) − µ (18) = {1 + 2 θn } . [A suitable choice for an will prove to be θ/n, for any 0 < θ < θ0 , or θn /n with θn 0 (and perhaps requiring lim sup(θn /θn+1 ) < ∞ also).] We will appeal to these inequalities in the present section. (A reader familiar with characteristic functions can read the straightforward proof of (15) and (16) in theorem 14.2.2 now.) 2

268

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Remark 2.2 (Technical comment) We wish to observe the truth of (22) below. Deﬁne the tail contributions 2 (19) d2n ≡ an n1 [ Knk+ (an )−˜ µnk ]2 /˜sd2n , d n ≡ an n1 [ Knk (1−an )−˜ µnk ]2 /˜sd2n . Now note that n 2 ˜ (20) ˜nk ]2 dt/˜sd2n = d2n + d n . 1 (an ,1−an )c [Knk (t) − µ Applying the inequality (a2 + b2 ) ≤ (|a| + |b|)2 ≤ 2 (a2 + b2 ) termwise allows us to n 2 2 sandwich an 1 [Knk (1−an )−Knk+(an )]2 /˜sd2n between (d2n +d n ) and 2 (d2n +d n ). Thus n 2 2 (21) d2n + d n → 0 if and only if an 1 qnk /˜sd2n → 0 . We now turn to theorem 2.3. Suppose the Z˜n of (12) and (15) satisﬁes Z˜n →d N (0, 1) for a particular sequence an ≡ θn /n having θn → 0. Appealing to (18) with ˜ nk ’s gives K 2 2 d2n ∨ d n → 0 (equivalently an n1 qnk /˜sd2n → 0 ) whenever (22) ˜ ˜ Zn →d N (0, 1) and the equivalent summands Ynk are uan, since the right side of (18) equals (1 + 2 θn ) → , for all arbitrarily small . Thus theorem 2.3(b) holds. This argument is also valid when we set all θ = 1. That is, if we set all θ = 1 and ˜ verify that both the resulting Y˜ nk are uan and that the resulting Zn →d N (0, 1), n 2 2 then we can conclude that an 1 qnk /˜sdn → 0 for the value resulting from the choice an = 1/n. That is, (22) also holds when an = 1/n for all n. 2 Remark 2.3 (Summary for iid rvs) When the Ynk are iid and √ θn 0, the condition in theorem 2.3(a) is implied by the requirement that qn / n σ ˜n → 0 ; and this is equivalent to [an qn2 /˜ σn2 ]/θn → 0. Also, the condition in theorem 2.3(b) reduces to the requirement that [an qn2 /˜ σn2 ] → 0 . We will apply this formally in the next section as theorem 11.3.1(e). (All we need for these to be equivalent is “an added touch” to close the gap. We close it in the next section.) Obtaining this remark was the main goal of this subsection. However, what has already been established is suﬃcient for our treatment of the bootstrap in section 11.4. 2

3.

IDENTICALLY DISTRIBUTED RVS

269

3

Identically Distributed RVs

Notation 3.1 (Slowly varying σ ˜2 (·)) Let K be a ﬁxed qf. Let a > 0 be tiny. We agree that dom(a, a) denotes [0, 1 − a), (a, 1], or (a, 1 − a) according as X ≥ 0, ˜ a,a (·) denotes K Winsorized outside dom(a, a). X ≤ 0, or general X, and that K ˜ a,a (·) [For example, when X takes on both positive and negative values, then K equals K+ (a), K(t), K(1 − a) according as t ≤ a, a < t < 1 − a, 1 − a ≤ t.] − − (t). Then deﬁne v(t) ≡ [K+ (1 − t)]2 + [K+ (t)]2 . Deﬁne q(t) ≡ K + (1 − t) + K+ 2 [Note that v(t) ≤ q (t) ≤ 2 v(t).] Let ξ be a Uniform(0, 1) rv. Let X ≡ K(ξ), and ˜ ˜ a,a (ξ). We also deﬁne notation for various moments by agreeing that X(a) ≡K (1)

˜ ˜ a,a (ξ) ∼ X(a) ≡K µ(a), σ ˜2 (a)) = (˜

with

˜ γ˜ (a) ≡ E|X(a) −µ ˜(a)|3 .

Now deﬁne the rvs (Xn1 , . . . , Xnn ) ≡ (K(ξn1 ), . . . , K(ξnn )) for row-independent Uniform(0, 1) rvs ξn1 , . . . , ξnn ; thus they are row-independent with df F and qf K. Our interest is in √ ¯ ˜n ]/˜ σn , where µ ˜n ≡ µ ˜(1/n) and σ ˜n ≡ σ ˜ (1/n) , Z¯n ≡ n [X n−µ (2)

˜ n ) ≡ √n [X ˜n − µ Z(a ˜(an )]/˜ σ (an ),

˜n ≡ where X

1 n

n ˜ 1 Xnk (an ),

¯n − µ ¯ n ) ≡ √n [X ˜(an )]/˜ σ (an ) . Z(a This is enough notation for the following theorem, but we will also prepare now for its aftermath. Let σ ˜n ≡ σ ˜1n ≡ σ ˜ (1/n). Let v1n ≡ V (1/n), where V (t) ≡ dom(t,t) K 2 (s) ds on [0, 12 ] and (3) 1 m(t) ≡ dom(t,t) |K(s)| ds on [0, 2 ] , which for all t suﬃciently small equal dom(t,t) v(s) ds and dom(t,t) q(s) ds. The Winsorized second moment [when Winsorized outside dom(1/n, 1/n)] is given by ˜2 v˜1n ≡ EK 1/n,1/n (ξ). Let |X| have df F|X| (·) and qf K|X| (·). Deﬁne partial moments or truncated moments in terms of both dfs and qfs via 2 V|X| (t) ≡ [0,1−t) K|X| (s) ds on [0, 1] , (4) U (x) ≡ [|y|≤x] y 2 dF (y) on [0, ∞) . Let v¯1n ≡ V|X| (1/n) and u1n ≡ U (x1n ), where we deﬁne xθn ≡ K|X| (1 − θ/n) to be the (1 − θ/n)-quantile of the rv |X|. Thus [−xθn , xθn ] is the shortest interval symmetric about the origin that contains at least proportion θ/n of the X probability. The Winsorized second moment in the present context is u ˜1n , where 2 u˜1n ≡ u1n + x21n P ( |X| > 1/n) equals v˜¯1n ≡ v¯1n + K|X| (1 − 1/n)/n, 2 (x) denote the variance of even though the lead terms may not be equal. Let σ ˜|X| 2 an X Winsorized outside [−x, x], and note that σ ˜|X| (x1n ) = σ ˜n2 = σ ˜ 2 (1/n).

2

270

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Theorem 3.1 (Studentized CLT for σ ˜2 ∈ L) Suppose r(t) ≡ tq2 (t)/˜ σ2 (t) → 0 as t → 0. [Any n 0 below satisﬁes lim sup( n / n+1) → 0.] Then:

(b)

Z¯n →d N (0, 1) whenever ˜n in (2) whenever µ ˇn ≡ (1/n,1−1/n) K(t) dt can replace µ

r(t) → 0.

(c)

˜ Z(1/n) →d N (0, 1)

if and only if

r(t) → 0.

(d)

¯ n /n) →d N (0, 1) Z(

if and only if r(t) → 0.

(e)

˜ n /n) →d N (0, 1) Z(

if and only if r(t) → 0.

(f)

Xn2 /νn →p 1

if and only if r(t) → 0.

(a)

for one νn > 0 with n νn

r(t) → 0.

Conditions (6)–(29) below are all equivalent. (Conditions (6), (9), (15), (19) (21), (22), (26), (27), (28), and (29) are especially interesting.) Proof. Consider (a). We are Winsorizing the qf on [t, 1 − t], not the df on [−xθn , xθn ]. Nevertheless, we essentially repeat the argument used for (10.2.4). Let √ n ˜ nk ( /n)] − µ (m) µ ˜n ≡ µ ˜( /n) , σ ˜n ≡ σ ˜ ( /n) , Z˜n ≡ n { n1 [ 1 X ˜n }/˜ σn . (Now, a = b ⊕ c means that |a − b| ≤ c.) First observe, the probability that n with n ˜ nk ( /n)), that the two rvs diﬀer bounded by n ≡ P ( k=1 Xk = k=1 X √ P (Z¯n ≤ z) = P (Z˜n × (˜ (n) σn /˜ σn ) + n(˜ µn − µ ˜n )/˜ σn ≤ z) ⊕ n . n n ˜ nk ( /n)) ≤ is trivial, and since the Berry–Esseen Since n = P ( 1 Xk = 1 X √ 3 theorem gives FZ˜n − Φ ≤ 9 γ˜n / n σ ˜n , the current theorem will follow from showing that for each > 0 we have √ σ ˜ ( /n) n |µ ˜( /n) − µ ˜(1/n) | γ˜n (5) → 1 , (k) → 0. (i) √ 3 → 0 , (j) nσ ˜n σ ˜ (1/n) σ ˜ (1/n) − Recall from above that q(a) ≡ K + (1 − a) + K+ (a). Then when r(t) → 0, we have √

/n q( /n) γ˜ 1 √ n3 ≤ √ × (o) = r( /n)/ → 0 using r(t) → 0, nσ ˜n

σ ˜ ( /n)

(p)

(q)

σ ˜ 2 ( /n)/˜ σ 2 (1/n) → 1 in analogy with (11.2.9), using r(t) → 0, √ √ n|µ ˜( /n) − µ ˜(1/n) |/˜ σ(1/n) ≤ {q( /n) − q(1/n)}/ n σ ˜ (1/n) √ ≤ q( /n)/ n σ ˜ (1/n) √ ≤ {r( /n)/ } {˜ σ ( /n)/˜ σ(1/n)} → 0 by (o) and (p).

Thus (5), and thus (a). Just (5)(i)( = 1) gives the normality in (c). [Temporarily set aside normality in (e).] Consider the implications of normality in (c) and (e). Remark 2.2 shows that the asymptotic normality implies (11.2.16) (with either

3.

IDENTICALLY DISTRIBUTED RVS

271

an = 1/n for (c) or an = n /n for (e)). Also, r(t) → 0 gives (b). See theorems 7.7.1 and 7.7.2 and theorems 10.6.1 – 10.6.4 for proposition 3.1. Then (f) is (23). Then (d) and (e) are equivalent, since P (Z˜n ( /n) = Z¯n ( /n)) ≥ (1 − 2 /n)n → 1. ˜ n /n) →d N (0, 1). For the normality in (e), we will use (10) to establish that Z(

And we will do so by showing that (5)(i) (in the present circumstances √ we need √ an n , rather than an ) satisﬁes γ˜nn ≤ r( n /n)/ n → 0 for r(t) ≡ t q(t)/˜ σ (t) and for some choice of n 0. (Recall again remark 11.2.3.) Well, in the worst situation r(t) → 0 rather slowly. So we deﬁne r¨(t) ≡ sup{r(s) : 0 < s ≤ t}, so that r¨(t) 0. This allows (r) r(t r¨(t))/ r¨(t) ≤ r(t)/ r¨(t) ≤ r¨(t)/ r¨(t) ≤ r¨(t) 0 , and thus the choice n ≡ r¨(1/n) works! Finally, it is trivial that (s)

lim ( n / n+1) = lim{¨ r (1/n)/¨ r(1/(n + 1))} ≤ 1.

2

Proposition 3.1 (Conditions equivalent to the slow variation of σ ˜2 (·)) The following contains just the “best” items from our list of equivalencies. When any one (hence all) of these equivalencies hold, we will write either K ∈ D(Normal) or F ∈ D(Normal). We will then say that K or F is in the domain of attraction of the normal distribution. [We require one speciﬁc an 0 having lim (an /an+1 ) < ∞) in (10) and in (15); and the choices an = /n or an = n /n turn out to be the most useful.] The following are equivalent: (6)

σ ˜ 2 (·)

is slowly varying at zero.

(7)

V|X| (·)

is slowly varying at zero.

(8)

V (·)

is slowly varying at zero.

(9) (10)

r2 (t) ≡ t q2 (t)/˜ σ2 (t) → 0 √ an q(an )/˜ σ (an ) → 0

(11)

2 t K|X| (1 − t)/V|X| (t) → 0

for any one speciﬁc an 0 . 2 where V|X| (t) ≡ [0,1−t) K|X| (s) ds .

(12)

2 t K|X| (1 − ct)/V|X| (t) → 0

for all 0 < c ≤ 1 .

(13)

2 t [K+ (t) ∨ K 2 (1 − t)]/V (t) → 0

where V (t) ≡

(14)

for all 0 < c ≤ 1 .

(17)

2 t [K+ (ct) ∨ K 2 (1 − ct)]/˜ σ 2 (t) → 0 √ q( /n) / [ n σ ˜ (1/n)] → 0 √ [q( /n) − q(1/n)] / [ n σ ˜ (1/n)] → 0 √ n [m( /n) − m(1/n)] / σ ˜ (1/n) → 0

(18)

[v( /n) − v(1/n)] / [n σ ˜n2 (1/n)] → 0

(15) (16)

as t → 0 .

for all > 0. for all > 0. for all > 0. for all > 0 .

dom(t,t) K

2

(s) ds .

272 (19)

CHAPTER 11. CONVERGENCE IN DISTRIBUTION U (·)

is slowly varying at inﬁnity,

where U (x) ≡

2

dF (y).

|y| dF (y).

(20)

x [M (cx) − M (x)] /U (x) → 0

(21)

R(x) ≡ x2 P (|X| > x)/ U (x) → 0

or x2 P (|X| ≥ x)/ U (x−) → 0 .

(22)

Xn2 /νn →p 1

for a speciﬁc νn > 0 with nνn .

(23)

Xn2 / νn →p 1,

(24)

[maxk

(25)

n P (X 2 > c n V|X| (1/n)) → 0 √ xθn / [ n σ ˜n ] → 0

(26)

1 n

[|y|≤x]

for νn one of v1n , v˜1n , u1n , v¯1n or u ˜1n = v˜¯1n .

2 Xnk ] / νn →p 0

for νn one of v1n , v˜1n , u1n , v¯1n or u ˜1n = v˜¯1n .

(28)

2 [maxk Xnk ] / n Xn2 →p 0. √ ¯ n | ]/ n Sn →p 0 . [maxk |Xnk − X

(29)

˜n2 →p 1 . Sn2 / σ

(27)

for all c > 0; M (x) ≡

[|y|≤x] y

for all c > 0. for all 0 < θ ≤ 1.

Proof. The equivalence of conditions (6)–(29) was speciﬁcally developed earlier (as cited in the previous theorem), where much longer lists of equivalences are given. And theorem 3.1 needed only to tie the CLT into the combined list. As the introduction points out, many of these equivalences are found in the literature. (Be aware that having a diﬀerent denominator in a condition, or being required to verify it only on some sequence of values, can be very valuable.) 2 √ ¯ Remark 3.1 (Natural parameters) The conclusion n [X n − νn ]/τn →d N (0, 1) for any νn and τn > 0 could be shown (with heavy-duty mathematics) to imply that σ ˜2 (·) ∈ L in the context of theorem 3.1.] Granted this result, the theorem of types (theorem 13.7.1) then gives √ σ (1/n) → 1 and n | νn − µ ˜(1/n) |/˜ σ(1/n) → 0 τn /˜ √ ¯ (30) whenever n[Xn − νn ]/τn →d N (0, 1). √ ¯ Then µ ˜(1/n) and σ ˜ (1/n) could be used, giving n [X σ(1/n) →d N (0, 1). n −µ(1/n)]/˜ ˜ n ) used in (2) allowed us to avoid all that in a natural way. Introducing the Z(a Moreover, statisticians want to use means and standard deviations. 2

A Converse of the CLT Theorem 3.2 (Domain of normal of the normal df ) n attraction √ iid rvs X1 , . . . , Xn , and set Zn = 1 Xk / n. Then: implies

Consider

EX1 = 0 and E(X12 ) < ∞.

(31)

Zn = Op (1)

(32)

EX1 = 0 and Var[X1 ] < ∞

imply

Zn →d N (0, Var[X1 ] ).

3.

IDENTICALLY DISTRIBUTED RVS

273

Proof. (Gin´e and Zinn) Now,√(32) was established previously. Consider (31). Fix t > 0. Let Zn ≡ n1 k Xk / n. By Gin´e–Zinn symmetrization of (10.3.10), we have (a)

P (Zn > 2λ) ≤ 2 P (Zn > λ) ;

and thus P (Zn > λ) = Op (1) by our hypotheses. Also, Khinchin’s inequality in exercise 10.3.3 (regarding the Xk ’s as ﬁxed constants) gives (b)

E |Zn | ≥ A1 Xn2

1/2

= c Sn

with Sn2 ≡ Xn2 .

Applying Paley–Zygmund’s inequality 3.4.9 to Zn (conditioned on ﬁxed values of the Xk ’s) for the ﬁrst inequality and (b) for the second yields 2 2 2 (EZn − t)+ (cSn − t)+ t 2 (c) P (|Zn | > t) ≥ ≥ = c 1 − Sn cSn (E{(Zn )2 })1/2 (d)

≥ (c2 /4) 1[Sn>2t/c] .

Taking expectations across the extremes of this inequality with respect to the Xk ’s gives the bound (e)

P (|Zn | > t) ≥ (c2 /4) P (Sn > 2t/c) .

Thus Sn = Op (1), by combining (e), (a), and the hypothesis. Fix M > 0. The SLLN gives n 2 2 1 (f) 1 Xk 1[Xk2 ≤M ] →a.s. E{X1 1[X12 ≤M ] } . n But →a.s. implies →d . Thus, applying exercise 11.1.3 to the open set (t, ∞) gives n 2 1 1(0,E(X 2 1 2 (g) 1 Xk 1[Xk2 ≤M ] > t) )) (t) ≤ lim n P ( n 1

(h)

[X1 ≤M ]

≤ sup P ( n1 n

n 1

Xk2 1[Xk2 ≤M ] > t),

for each t > 0.

It follows that (i)

sup 1(0,E(X 2 1 2 (t) ≤ sup sup P ( n1 1 [X ≤M ] )) M >0 1 M >0 n n 2 1 ≤ sup P ( n 1 Xk > t)

n 1

Xk2 1[Xk2 ≤M ] > t) .

n

(j)

≤ sup P (Sn2 > t) . n

Since Sn =Op (1), we have Sn2 =Op (1); and this implies that we can specify a t value of t0 in (j) so large that the right-hand side of (j) at t0 is less than 12 . But this implies that for this t0 the indicator function in (i) must equal zero uniformly in M . This means that (k)

sup E(X12 1[X12 ≤M ] ) ≤ t0 .

M >0

But this last supremum equals E(X12 ), and hence we must have E(X12 ) ≤ t0 < ∞. To complete the proof, we must now show that E(X1 ) = 0. Since EX12 < ∞, ¯ n →p EX1 . But the hypothesis that Zn = Op (1) implies that the WLLN√gives X ¯ n = Zn / n →p 0. Combining these gives EX1 = 0. 2 X

274

4

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Bootstrapping

Suppose X1 , . . . , Xn are an iid sample from F . Denote the empirical df of the n ≡ (X1 , . . . , Xn ) by Fn (·). This empirical df Fn has mean X ¯ n and sample X ∗ ≡ (X ∗ , . . . , X ∗ ) denote an iid sample from Fn , called the variance Sn2 . Let X n n1 nn ¯ n∗ and Sn∗ denote the mean and the standard deviation of bootstrap sample. Let X the bootstrap sample. Since the moments of Fn exist, we will work with normed ∗ ¯ n | ]/√n Sn ]. summands. The maximum normed summand is Mn∗ ≡ [maxk |Xnk −X (Note that the normed summands always constitute a uan array.) Now consider √ n ) ≡ [ maxk |Xk − X ¯ n | ]/ n Sn . (1) Dn ≡ D(X √ (For all 0 < θ ≤ θ0 ≡ 12 the quantity xθn / n σ ˜n in (11.2.2) has value Dn . Note that Dn is formed from the original sample, while Mn∗ is formed from the bootstrap sample. The following theorem is in the spirit of (11.2.4).) (Note, moreover, that 0 ≤ Mn∗ ≤ Dn , while P (Mn∗ = Dn ) ≥ 1 − (1 − 1/n)n → 1 − 1/e > 0 also holds.) The “standardized” rv of theorem 1.2.1(i) is now equal to √ √ ¯ n∗ − X ¯ n ]/Sn , ¯ n∗ − X ¯ n ]/Sn∗ . Z¯n∗ ≡ n [X (2) and also deﬁne Tn∗ ≡ n [X Agree that the weak bootstrap holds if Z¯n∗ →d N (0, 1) for the joint probability on Ω × Ω∗n . Agree that the strong bootstrap holds if for a.e. given sequence of values of X1 , X2 , . . . it is true that the conditional distribution of Z¯n∗ on Ω∗n satisﬁes Z¯n∗ →d∗ N (0, 1). Theorem 4.1 (Bootstrapping) Consider Z¯n∗ in the iid case. (a) The weak bootstrap for Z¯n∗ is equivalent to both Dn →p 0

and/or

σ ˜ 2 (·) ∈ L .

(b) The strong bootstrap for Z¯n∗ is equivalent to both Dn →a.s. 0

and/or

Var[X1 ] = σF2 ∈ (0, ∞) .

Corollary 1 (a) The weak bootstrap holds for Tn∗ whenever Dn →p 0. (b) The strong bootstrap holds for Tn∗ whenever Dn →a.s. 0. Proof. Consider the normality. (b) The actual (11.2.2) requirement [i.e. that n ] is implied by requiring Dn → 0 Dn → 0 for a.e. ﬁxed sequence of values of X a.s. (a) The proof given for (11.2.4) established √ a uniform rate of convergence to normality. Thus whenever θ < /2 and {xθn / n σ ˜n } < δ for an appropriate value of δ , we could conclude that FZ¯n − Φ ≤ . Now consider the present √ situation. For each ω ∈ Ω it is true for all 0 < θ ≤ θ0 that the quantity {xθn / n σ ˜n } is bounded by Dn , and thus we need only show that for the appropriate value δ we have P (Dn ≤ δ ) > 1 − /2 for all n ≥ (some n ). That is, it suﬃces to show that Dn →p 0.

4.

BOOTSTRAPPING

275

Consider the converse. Suppose Z¯n∗ →d N (0, 1) for a ﬁxed array (xn1 , . . . , xnn ). The summands are necessarily uan. Thus (11.2.22) (with all θ = 1) gives Dn → 0 for this same ﬁxed array. Thus Dn →a.s. 0 is implied by the strong bootstrap, and Dn →p 0 is implied (by going to subsequences) by the weak bootstrap. Now (11.3.28) and (11.3.6) show that Dn →p 0 is equivalent to σ ˜ 2 (·) ∈ L. And it is known from Kolmogorov’s SLLN in theorem 10.4.1 that Dn →a.s. 0 is equivalent to Var[X1 ] < ∞. Consider the corollary. Use that (10.6.5) implies (10.6.3) (with both r = 2 and r = 1), and use (10.6.26) (with r = 1 and s = 2). Conclude that we analogously have both (3)

Sn∗ /Sn →p×p∗ 1 if Dn →p 0

(4)

Sn∗ /Sn →p∗ 1 for a.e. value of (X1 , X2 , . . .) if

Exercise 4.1

and Dn →a.s. 0.

Establish all the details of (3) from theorem 10.6.1.

2

276

5

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Bootstrapping with Slowly Increasing Trimming

Let kn and kn denote trimming numbers, with kn ∧ kn → ∞. But suppose the trimming fractions (an ∨ an ) ≡ (kn ∨ kn )/n → 0. Bootstrapping such a trimmed mean from an iid sample always works! Notation 5.1 Let Kn ≡ Fn−1 denote the qf associated with some ﬁxed nondegenerate df Fn . We can always specify that dom(an , an ) ≡ (an , 1 − an ) for any n(an ∧ an ) → ∞. However, if Fn− (0) = 0, then specifying that dom(0, an ) ≡ [0, 1 − an ) is preferable; and if Fn (0) = 1, then specifying that dom(an , 0) ≡ (an , 1] is preferable. So we agree to let an ∧an denote an ∧an , or an , or an according to the ˜ n (·) denote Kn (·) Winsorized outside of dom(an , a ), and let µ ˜n scheme used. Let K n 2 ˜ n (ξ) . Then X ˜ nk ≡ K ˜ n (ξnk ) are (unoband σ ˜n denote the mean and variance of K ˜ n when ξnk are row-independent Uniform(0, 1) servable) row-independent with qf K n ˜ ˜ ˜ ˜ ˜ 2 rvs. Let Xn ≡ (Xn1 + · · · + Xnn )/n and S˜n2 ≡ 1 (X nk − Xn ) /n. Let √ √ ˜n − µ ˜n − µ (1) ˜n ]/˜ σn and Zˆn ≡ n [X ˜n ]/S˜n . Z˜n ≡ n [X ˜ n1 − µ ˜ n |3 . Let Fn denote the collection of all dfs Fn for which σ ˜n > 0. Let γ˜n ≡ E|X 2 Theorem 5.1 (Universal studentized CLT) Suppose the trimming fractions satisfy n(an ∧ an ) = (kn ∧ kn ) → ∞. Then uniformly in Fn : √ 3 (2) FZ˜n − Φ ≤ 9 γ˜n / n σ ˜n ≤ 9/ n(an ∧ an ) → 0 . (3)

P (|S˜n /˜ σn − 1| ≥ ) → 0.

(4)

FZˆn − Φ → 0.

Requiring (an ∨ an ) → 0 guarantees that every nondegenerate df F is eventually in all further Fn . Example 5.1 Let all Fn be Bernoulli(10−10 ). Then n must be huge before σ ˜n (an ) > 0. 2 √ 3 ˜n is immediate from the Berry–Esseen Proof. That FZ˜n − Φ ≤ 9 γ˜n / n σ ˜ n (t)− µ ˜ n (t)− µ theorem. Maximizing |K ˜n |1 in the integrand of γ˜n (but leaving |K ˜n |2 to integrate) gives √ 3 √ − γ˜n / n σ ˜n ≤ [ |K + (1 − an ) − µ ˜n | ∨ | K+ (an ) − µ ˜n | ]/ n σ ˜n (5) ≤ 1/ n (an ∧ a n ) → 0 , as claimed. Let qn ≡ K(1 − an ) − K+ (an ), as usual. Moreover, n ˜nk − µ ˜n − µ (a) S˜n2 /˜ σn2 = { [ 1 (X ˜n )2 /n]/˜ σn2 } − {(X ˜n )/˜ σn }2 ≡ {I2n } − {I1n }2 , where Chebyshev’s inequality gives both P (|I1n | ≥ ) ≤ 1/( 2 n) → 0 and

5.

BOOTSTRAPPING WITH SLOWLY INCREASING TRIMMING n

(b) (c)

P (|I2n − 1| ≥ ) ≤ ≤

1

277

˜ nk − µ E[(X ˜n )4 ] (an ∧ an ) qn2 1 ≤

2 n2 σ ˜n4

2 n(an ∧ an ) σ ˜n2

2 → 0.

2 n(an ∧ an )

Thus (3) holds. Writing √ ˜n − µ Zˆn = n (X (d) ˜n )/S˜n = Z˜n + Z˜n (˜ σn /S˜n − 1) , we obtain (4); note that P (An ) ≡ P ([|Z˜n | ≤ M ]) can be made uniformly small, even though this set depends on F , while P (|˜ σn /S˜n − 1| ≥ ) → 0 uniformly. 2 n ≡ (Xn1 , . . . , Xnn ) denote an iid sample from the qf Kn , Notation 5.2 Let X 2 ¯ n , Sn , Gn , and Kn (·) denote its sample mean, sample variance, sample and let X ˜ ≡ (X ˜ n1 , . . . , X ˜ nn ) denote third absolute central moment, and sample qf. Let X n the (kn , kn )-Winsorized sample, for integers kn and kn . Let an ≡ kn /n and an ≡ kn /n for kn ∧ kn going to ∞ (here kn ∧ kn will denote either kn ∧ kn , or kn , or kn as ˜ n denote the sample mean, ˜ n , S˜n , G ˜ n , and K in the scheme of notation 5.1). Let X sample variance, sample third central moment, and sample qf of the population ˜ . Let X ∗ ≡ (X ∗ , . . . , X ∗ ) denote the iid bootstrap sample from K ˜ n (·), and X n n1 nn n ∗ 3 ∗2 2n∗ and S ∗ let X n be the sample mean and sample variance of Xn . Let Pn denote the bootstrap probability distribution. Now deﬁne the statistics of interest √ √ 2∗ ≡ n [ X 2∗ − X ˜ n ]/S˜n 4∗ ≡ n [ X 2∗ − X ˜ n ]/S 2∗ . (6) Z and Z n n n n n [We saw in the previous section that the sample mean and sample variance Xn∗ and √ ¯ n ]/Sn Sn∗2 of an iid bootstrap sample from Kn are such that Z¯n ≡ n [ Xn∗ − X satisﬁes the strong (or the weak) bootstrap if and only if Var[X] ∈ (0, ∞) (or F ∈ D(Normal)). But next we see the glories of Winsorizing! Winsorizing does do what Winsorizing was supposed to do. The bootstrap always works, provided that we just trim a bit.] 2 Theorem 5.2 (Universal bootstrap CLT) Suppose the trimming fractions are n such that n(an ∧ an ) → ∞ in the context of notation 5.2. Then uniformly in all X n , conditional on X n , for which S˜n > 0 we have that for a.e. X √ 3 ˜ n / n S˜n ≤ 9/ n(an ∧ an ) = 9/ kn ∧ kn → 0 , (7) FZ2∗ − Φ ≤ 9 G n 2∗ − S˜n |/S˜n ≥ X n ) → 0 , (8) P∗n (|S n (9)

FZ4∗ − Φ → 0,

(10)

lim S˜n > 0 if we also specify that (an ∨ an ) → 0 , with F nondegenerate .

Proof.

n

This is immediate from the previous theorem.

2

278

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Remark 5.1 If we knew how to Winsorize correctly in theorem 5.1, it would be a useful theorem. The point is, we always know how to Winsorize correctly in the bootstrap of theorem 5.2. But should we instead do bootstrap sampling from the empirical qf Kn itself, ˜ n , and then Winsorize this sample? No! Sampling from K ˜ n gives rather than K us the analog of theorem 5.1, while sampling from Kn (it can be shown) does not. ˜ of (7.10.33) provided Sampling from Kn does work for any qf K in in the class D that (kn ∧ kn )/ log2 n → 0 with an ∨ an → 0 (as can be shown by the methods of chapter 16). 2 Let ξ1 , ξ2 , . . . be iid Uniform(0, 1) rvs. Let Xnk ≡ Fˇn−1 (ξk ) with −1 −1 1(0,1/2)(t) + (1 − t) ∨ (1/n) 1[1/2,1) (t). Fˇn−1 (t) ≡ − t ∨ (1/n) ¯ n ≡ n Xnk /n. Compute Vn , as well as the higher Let Vn ≡ Var[Xnk ], and let X k=1 4 moments E|Xnk |3 and EXnk . √ ¯ √ (a) Show that Zn ≡ n X n / Vn →d N (0, 1) by verifying the Lindeberg condition. (b) What conclusion does the Berry–Esseen theorem imply for Zn ? n 2 /n satisﬁes Xn2 /Vn →p 1. (c) Show that Xn2 ≡ k=1 Xnk √ ¯ 2 1/2 → N (0, 1). (d) Of course, this immediately implies that Tn ≡ n X n /( Xn ) d 2 (e) Show that (E|Xnk |) /Vn → 0. Exercise 5.1

Exercise 5.2

Formulate and solve another example in the spirit of exercise 5.1.

n ¯ n )2 /(n − 1) satisﬁes Sn2 /˜ Exercise 5.3 Verify that Sn2 ≡ k=1 (Xnk − X σn2 →p 1, and do it by verifying a Lindeberg type condition in the context of theorem 11.2.1.

6.

EXAMPLES OF LIMITING DISTRIBUTIONS

279

6

Examples of Limiting Distributions

Example 6.1 (Delta method) (a) Suppose cn [Wn − a] →d V where cn → ∞, and suppose g(·) is diﬀerentiable at a (recall (4.3.7)). Then (as in the chain rule proof of calculus) immediately (1)

cn [ g(Wn ) − g(a)] =a {g (a)} · cn [ Wn − a] →d {g (a)} · V .

[Recall that Un =a Vn means that Un − Vn →p 0.] (b) The obvious vector version of this has the conclusion (2)

n ) − g(a )] =a { ∇g(a)} · cn [ W n − a ] . cn [ g(W

2

Example 6.2 (Asymptotic normality of the sample variance) Suppose the rvs X1 , . . . , Xn are iid (µ, σ2 ) with µ4 ≡ EX 4 < ∞ and σ2 > 0. Then (3)

Sn2

n 1 ¯ 2 = (the sample variance). ≡ (Xk − X) n−1 k=1

For a useful phrasing of conclusions, deﬁne Zk ≡ (Xk − µ)/σ ∼ = (0, 1) , (4)

Yk ≡ Zk2 = [(Xk − µ)/σ]2 ∼ = (1, µσ44 − 1) = (1, 2 (1 + γ2 /2)) ,

where γ2 ≡ (µ4 − 3σ 4 )/σ4 ≡ (the kurtosis) measures the tail heaviness of the distribution of X. We will show that as n → ∞ both √ √ (5) n [Sn2 − σ 2 ] √21σ2 =a √12 n [ Y¯n − 1] →d N (0, 1 + γ2 /2) and (6)

√

n [Sn − σ] σ2 =a

Proof.

n [ Y¯n − 1] →d

√

2

2 N (0, 1 + γ2 /2) .

Now, 2 Sn σ2

(a)

√

=

=

1 n−1

n

n 1 n−1 n

k=1

¯ n )2 (Xk −X σ2

n k=1

=

n 1 n−1 n

n k=1

¯

[( Xkσ−µ ) − ( Xnσ−µ )]2

[Zk − Z¯n ]2 .

Then note from (a) that √ √ 6 √ n (Sn2 − σ 2 ) n 1 5√ ¯ n √ √ (b) = n ( Yn − 1) − n Z¯n2 − √ 2 n−1 2 2σ 2 (n − 1) (c)

=a

√1 2

√

n ( Y¯n − 1) →d N (0, Var[ Y ]/2).

2

280

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Exercise 6.1 (a) Determine the joint limiting distribution (of appropriately nor¯ n and Sn ≡ [ n (Xk − X ¯ n )2 /(n − 1)]1/2 in the iid case. (Consider the malized) X 1 representation of Sn in (6) as a normed sum of the rvs Yk .) What condition on the moments is required for the result? (b) Determine the asymptotic distribution of the (appropriately normalized) rv ¯ n (the coeﬃcient of variation) in the iid case. Obtain a similar representaSn /X tion. (You may suppose that the Xk ’s now take values in (0, ∞).) (c) Note that (6) provides a stronger conclusion than just asymptotic normality, in that it forms a superb starting point for the further asymptotic work in (a) and (b). Note also (13) below. ¯ n and Sn2 ) Let X1 , . . . , Xn be iid. Note/show Exercise 6.2 (Moments of X 2 ¯n ∼ that (provided that µ or σ is well-deﬁned) X = (µ, σ2 ) and ESn2 = σ 2 . Show that (provided that µ3 or µ4 is well-deﬁned): (7)

¯ n − µ)3 = E(X

µ3 n2

(8)

¯ n − µ)4 = E(X

3 σ4 n2

(9)

Var[Sn2 ] =

1 n

.

{ µ4 −

+

µ4 −3 σ4 n3

n−3 n−1

σ4 }

. and

¯ n , Sn2 ] = Cov[X

1 n

µ3 .

√ ¯2 2 2 Exercise 6.3 If X1 , . . . , Xn are iid (µ, σ2 ), then n [X n − µ ] →d 2 µ × N (0, σ ) 2 ¯ (by the delta method). But what is the asymptotic distribution of nXn when µ = 0? √ Exercise 6.4 (Two sample problems) If m (Sm − θ) →d N (0, 1) as m → ∞ √ and n (Tn − θ) →d N (0, 1) as n → ∞ for independent rvs Sm and Tn , then mn m+n (Sm − Tn ) →d N (0, 1) as m ∧ n → ∞. [Hint: Consider what happens under the additional hypothesis that λmn ≡ m/(m + n) → λ ∈ [0, 1]. Then show that this hypothesis can be dropped. This is useful for the two-sample t-test and F -test.] Exercise 6.5 (Simple linear rank statistics) Let TN ≡ √1N N 1 ci aπ(i) , where (π(1), . . . , π(N )) achieves each of the N ! permutations of (1, . . . , N ) with probability 1/N ! . Here, the ci and ai are constants. Show that: N Eaπ(i) = a ¯N , Var[aπ(i) ] = σa2 ≡ N1 1 (ai − a ¯N )2 , (10) Cov[ aπ(i) , aπ(j) ] = − N1−1 σa2 for all i = j . (11)

√1 N

[Hint. Var[

E TN = c¯N · a ¯N ,

N 1

Var[TN ] =

N N −1

σc2 σa2 .

aπ(i) ] = 0, as in (9.1.8).]

¨n) Example 6.3 (The median X Let X1 , . . . , Xn be iid with df F (· − θ), for 1 some θ ∈ R, where F (0) = 2 and F (0) > 0 exists and exceeds zero. [Thus ¨ n denote Xi ∼ = θ + i , for i ’s that are iid F (·) with a unique median at 0.] Let X any sample median. (a) Then √ ¨ n − θ] →d Z1 ∼ (12) Z1n ≡ n [X = N (0, 4[F 1(0)]2 ) .

6.

EXAMPLES OF LIMITING DISTRIBUTIONS

281

√ ¯ 2 ∼ (b) If F (·) ∼ = (µ, σ2 ) also, then Z2n ≡ n [X n − (µ + θ)] →d Z2 = N (0, σ ). In fact, the joint distribution of (Z1n , Z2n ) →d (Z1 , Z2 ), where the covariance of the limiting distribution is given by (13)

E{[ − µ] × [1[>0] − 1/2]/F (0)} .

2

Proof. The “mid-value”, or (population) median, of the distribution of the Xk ’s is any value θ satisfying P (X ≤ θ) ≥ 12 and P (X ≥ θ) ≥ 12 ; in the present case its value is θ. The ordered values of the Xk ’s are denoted by Xn:1 ≤ · · · ≤ Xn:n , and ¨ n is deﬁned to be Xn:m or any are called the order statistics. The sample median X point in the interval [(Xn:m , Xn:m+1)] according as n equals 2m + 1 or 2m is odd or even. Let Xnl and Xnr denote the left and right endpoints of the interval of possible sample medians (of course, Xnl = Xnr =√Xn:m+1 if n = 2m + 1 is odd). n By the event equality [Xnr − θ ≤ y/ n ] = [ 1 1[Xi −θ≤y/√n ] > n/2], we have √ √ n P ( n [Xnr − θ] ≤ y) = P (Xnr − θ ≤ y/ n ) = P ( 1 1[Xi −θ≤y/√n ] > n/2) (14)

= P ( n1

n

= P ( √1n (a)

1[i ≤y/√n ] > 1/2)

1

n 1

since i ≡ Xi − θ ∼ = F (·)

{1[i >y/√n ] − 12 } < 0)

= P (Wn + An < 0) .

Here (15)

Wn ≡

√1 n

n 1

{1[i>0] − P ( i > 0)} →d Z1 ∼ = N (0, p(1 − p))

with p ≡ P ( i > 0) = 12 , and (as we will now show) (16) An ≡ √1n n1 {1[i >y/√n ] − 1[i >0] } →p −y F (0) . Note that all terms in the summation in An are of the same sign. Then (b)

An →p −y F (0±)

according as y > 0 or y < 0,

since EAn = (c)

√

√ √ √ n [ P ( > y/ n ) − P ( > 0)] = −y [F (y/ n ) − F (0)]/ [y/ n ]

→ −y F (0±)

[provided only that both F (0±) exist]

and (d)

√ √ Var[An ] = [F (y/ n ) − F (0)]{1 − [F (y/ n ) − F (0)]} → 0 .

Thus Wn + An →d Z1 − yF (0) via (15), (16), and Slutsky. By →d we then have √ P ( n [Xnr − θ] ≤ y) = P (Wn + An < 0) (e)

→ P (Z1 − yF (0) ≤ 0) = P (Z1 /F (0) ≤ y) for each y.

282

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

That is, √

∼ N (0, p(1 − p)/ [F (0)]2 ) . n [Xnr − θ] →d Z1 /F (0) = √ In like fashion, [ n [Xnl − θ] ≤ y] = [ n1 1[i ≤y/√n ] ≥ n/2], so that (f)

(g)

√ P ( n [Xnl − θ] ≤ y) = P ( n1 1[i ≤y/√n ] ≥ n/2) = P (Wn + An ≤ 0) .

Thus the same argument as before gives √ (h) n [Xnl − θ] →d Z1 /F (0) . ¨ n in between, via Now we squeeze the general X √ √ √ ¨ n − θ] ≤ y) ≤ P ( n [Xnl − θ] ≤ y) , (i) P ( n [Xnr − θ] ≤ y) ≤ P ( n [X where both ends converge to P (Z1 /F (0) ≤ y). This completes the proof. In the course of this proof we have demonstrated the events √ ¨ [ ω : n (X n (ω) − θ) ≤ y ] and [ ω : Wn (ω) ≤ y F (0) ] (17) diﬀer by a probabilistically negligible amount. For the joint result, apply (17) and the multivariate CLT to (Wn , Z2n ).

2

Exercise 6.6 (Joint asymptotic normality of quantiles) For 0 < p < 1, the pth quantile xp of F is now deﬁned as xp ≡ F −1 (p). (a) Show that if F has a derivative F (xp ) > 0 at xp , then √ (18) n [ Xn:[np] − xp ] →d N (0, p(1 − p)/ [F (xp )]2 ) as n → ∞. (b) Establish joint normality for pi and pj quantiles, where the covariance matrix of the asymptotic distribution has (i, j)th entry σij ≡ [(pi ∧ pj ) − pi pj ]/ [F (xpi )F (xpj )] . Write out the analogue of (17), and use it. Exercise 6.7

What happens when you try to apply (12) to:

(a) F (x) = exp(−|x|)/2 ? or (b) F (x) = Show that

√

1 2

1[−1,0) (x) +

1 4

1[0,2] (x) ?

¨ n − θ] →d (a rv) in both cases. (In case (b) it is not normal.) n [X

Exercise 6.8 Use (11.1.13) to verify (12) both for n = 2m + 1 odd and for ¨ n ≤ Xn:m+1 , n = 2m even. Since Xn:m ≤ X √ n ¨ n − θ) ≤ z) ≤ P ( n 1[ ≤y/√n ] ≥ n ]) . P ( 1 1[i ≤y/√n ] > n2 ]) ≤ P ( n (X 1 i 2 (The right side is an equality when n is odd.)

6.

EXAMPLES OF LIMITING DISTRIBUTIONS

283

Exercise 6.9 Consider (with hypothesis as weak as possible) the asymptotic distribution of (appropriately normalized forms of) both n n 1 1 ¯ ¨ (19) and 1 |Xk − Xn | 1 |Xk − Xn | n n for iid samples X1 , . . . , Xn from a df F (µ, σ2 ) having median ν. Exercise 6.10 Let X1 , X2 , . . . be independent with Xk ∼ = Uniform(−k, k). Then establish that Sn /σn →d N (0, 1). Exercise 6.11 Determine the limiting distribution of n 2 n 3/2 k=1 (Xk − X2n+1−k ) k=1 (Xk − X2n+1−k )/n where X1 , X2 , . . . are iid (µ, σ2 ) rvs. Exercise 6.12 Determine the .95-quantile of the limiting distribution of √ n −Xk / n , k=1 Uk for independent rvs with Xk ∼ = Double Exponential(0, 1) and Uk ∼ = Uniform(0, 1). Example 6.4 (Weighted sums of iid rvs) Suppose that rvs Xn1 , . . . , Xnn are row independent and iid (µ, σ2 ). Let cn ≡ (cn1 , . . . , cnn ) for n ≥ 1, and set n n 2 c¯n ≡ k=1 cnk /n and σcn ≡ k=1 (cnk − c¯n )2 /n ≡ SScc /n . Suppose we have the uan condition (20)

0 1 max1≤k≤n (cnk − c¯n )2 /n max1≤k≤n (cnk − c¯n )2 Dc ≡ D(cn ) ≡ = →0 n 2 σcn ¯n )2 k=1 (cnk − c

as n → ∞. Then (21)

n (cnk − c¯n ) Xnk − µ √ →d N (0, 1). n σcn σ

k=1

[We need not center the cnk ’s if the Xnk ’s have mean 0.] Proof. Without loss of generality, set µ = 0. Now, Lindeberg’s condition holds, as we demonstrate via 2 n (cnk − c¯n ) 2 √ x dF (x) √ σ n σcn [|cnk −¯ cn | |x|≥ σ n σcn ] k=1

(a)

≤ σ −2 · 1 ·

since Dc → 0 and

√ [|x|≥σ/ Dc ]

x2 dF (x) → 0,

x2 dF (x) < ∞.

2

The preceding example is useful in regression situations, and in connection with the projection techniques of section 9.3. See section 14.2.

284

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Exercise 6.13 (Monte Carlo estimation) Let h : [0, 1] → [0, 1] be a measurable 1 function, and let θ = 0 h(t) dt. Let X1 , Y1 , X2 , Y2 , . . . be iid Uniform(0, 1) rvs. Deﬁne two diﬀerent estimators of θ by T1n ≡

n k=1

h(Xk )/n

and

T2n ≡

n

k=1 1[Xk ≤h(Yk )] /n.

(a) Show that both T1n and T2n are unbiased estimators of θ, and determine which estimator has the smaller variance. Indicate how the variance of each estimator could be estimated. (b) Determine the joint asymptotic distribution of appropriately normalized forms of T1n and T2n . Exercise 6.14 (An analogue of the student-t statistic based on quartiles) Let X1 , . . . , Xn be iid with df F (·). Let m ≡ [n/4], for the greatest integer [·]. Let Un ≡ Xn:m ,

¨ n ≡ (the median), Vn ≡ X

Wn ≡ Xn:n+1−m

denote the quartiles and median of the sample. Make appropriate assumptions regarding F (·). (a) Determine the joint asymptotic distribution of √ √ ( n [Vn − ν], n [Wn − Un − µ] ) for appropriately deﬁned µ and ν. (b) Simplify this if the Xi are symmetrically distributed about 0. (c) Determine the asymptotic distribution under symmetry of the (student-t like) statistic (formed from three sample quantiles) Tn ≡

√

n [Vn − ν]/ [Wn − Un ].

Exercise 6.15 Let the Xk ’s be iid Cauchy(0, 1) in the previous exercise. (d) Evaluate F (x) = P (X ≤ x) for x ∈ R. (e) Solve p = F (x) for xp ≡ F −1 (p), when 0 < p < 1. (f) Express your answers to (b) and (c) of the previous exercise in the present context. Exercise 6.16 (Poisson estimation) Let X1 , . . . , Xn be iid Poisson(θ). ¯ (a) Reasonable estimators of θ include the nsample mean T1n ≡ Xn , the sample n 2 variance T2n ≡ Sn , and T3n ≡ 1 kXk / 1 k (which puts more emphasis on the more recent observations). Evaluate lim Var[Tin ] for i = 1, 2, 3. ¯ n2 − X ¯ n /n and T5n ≡ Xn2 − X ¯ n are both unbiased estimators (b) Verify that T4n ≡ X of θ2 . Evaluate lim Var[Tin ] for i = 4, 5. √ ¯ 2 ¯ (c) Determine the asymptotic distribution of Dn ≡ n [ X n − Sn ]/ Xn when the observations really do follow a Poisson distribution. (d) What is the asymptotic distribution of Dn when the observations Xk actually follow a NegBiT(r, p) distribution?

6.

EXAMPLES OF LIMITING DISTRIBUTIONS

285

Theorem 6.1 (Doeblin’s CLT for a random number of rvs) Consider iid (0, σ2 ) rvs X1 , X2 , . . . . Let {νn }∞ n=1 be integer-valued rvs such that the proportion νn /n →p c ∈ (0, ∞) as n → ∞. Let Tn ≡ X1 + · · · + Xn denote the total. Then √ (22) Tνn / νn →d N (0, σ2 ). [Note that νn and X1 , X2 , . . . need not be independent.] Proof.

Now, ,

% $ Tνn − T[cn] T[cn] [cn] + (23) . νn [cn] [cn] Note that T[cn] / [cn] →d N (0, 1) and [cn]/νn = paragraph we will show that (a) (Tνn − T[cn] )/ [cn] →p 0. Tν √n = νn

[cn] c cn νn /n

→p 1. In the next

The theorem then follows from Slutsky’s theorem. We now let An ≡ [|Tνn − T[cn] |/ [cn] > ], and note that (b) P (An ) = ∞ k=1 P (An ∩ [νn = k]) = P (An ∩ [νn = k]) + {k:|k−[cn]|≤3 cn}

(c)

≡

+

1

P (An ∩ [νn = k])

{k:|k−[cn]|>3 cn}

2.

Since νn /[cn] →p 1, for n suﬃciently large we have νn 3 3 cn − 1| >

2 ≤ P (|νn − [cn]| > cn) ≤ P | [cn] [cn] νn (d) ≤P | − 1| > 3 < . [cn] Also, applying Kolmogorov’s inequality twice, ≤ P max |T − T | >

[cn] k [cn] 1 3 |k−[cn]|≤ cn

≤P

max 3

[cn]≤k≤ cn

|Tk − T[cn] | > [cn]

+P

max 3

[cn]− cn≤k≤[cn]

|Tk − T[cn] | >

[cn]

[cn]+3 cn

≤2

Var[Xk ]/ 2 [cn] ≤ 2 3 cnσ2 / 2 [cn]

k=[cn]+1

(e)

≤ 4 σ2

for all n suﬃciently large.

Combining (d) and (e) into (c) shows P (An ) → 0, as required.

2

286

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Exercise 6.17 (Sample correlation coeﬃcient Rn ; Cram´er and Anderson) Let us suppose that Xi 0 1ρ are iid , for 1 ≤ i ≤ n, Yi 0 ρ1 √ and that the Σ below has ﬁnite entries. Consider n [Rn − ρ], where Rn is the sample correlation coeﬃcient. Thus Rn ≡ SSXY /{ SSXX SSY Y }1/2 for the sums n ¯ n )(Yi − Y¯n ), etc. of squares SSXY ≡ 1 (Xi − X (a) Reduce the case of general means, variances and covariances to this case. (b) Note that  1 n    √ 1 (Xi Yi − ρ) n Z1  √1 n (X 2 − 1)  (24) = N (0, Σ)  n 1  →d  Z2  ∼ i n 2 Z √1 3 (Y − 1) i 1 n

with



 E(X 2 Y 2 ) − ρ2 E(X 3 Y ) − ρ E(XY 3 ) − ρ (25) Σ ≡  E(X 3 Y ) − ρ EX 4 − 1 E(X 2 Y 2 ) − 1  . 3 2 2 E(XY ) − ρ E(X Y ) − 1 EY 4 − 1 √ ∼ (c) Then show that n [Rn − ρ] →d Z1 − ρ2 Z2 − ρ2 Z3 √ = N (0, τ 2 ), and evaluate τ 2 . (d) Show that when X and Y are independent, then n [Rn − ρ] →d N (0, 1). (e) If the (Xi , Yi ) are jointly normal, show that   1 + ρ2 2 ρ 2 ρ (26) Σ =  2ρ 2 2 ρ2  . 2 ρ 2 ρ2 2 Then simplify the expression for τ 2 and obtain √ (27) n [Rn − ρ] →d N (0, (1 − ρ2 )2 ) . √ (f) Show that n [g(Rn ) − g(ρ)] →d N (0, 1) for g(t) ≡ 12 log( 1+t 1−t ). √ ρ (g) Approximating n − 3 [g(Rn ) − g(ρ) − 2(n−1) ] by N (0, 1) is excellent. (See example 14.6.1.) ¯ n , Yi − Y¯n ] = (1 − 1 ) Cov[Xi , Yi ]. (h) Show that Cov[Xi − X n Exercise 6.18 (Extreme value quantiles) Let X and X1 , . . . , Xn be iid with the Weibull(α, β) density f (x) = (βxβ−1 / αβ ) exp(−(x/α)β ) on x ≥ 0. Now, (X/α)β ∼ = Exponential(1), and thus Y ≡ log X satisﬁes Y ∼ =ν+τ W

where ν ≡ log α and τ ≡ 1/β

and W has the Extreme Value density for minima given by exp(w−ew ) on (−∞, ∞). Let Yn:1 ≤ · · · ≤ Yn:n denote the order statistics of the rvs Yk ≡ log Xk . First, let 0 < p1 < p2 < 1, and then deﬁne Un ≡ Yn:[np1 ] and Vn ≡ Yn:[np2 ] . We seek values p1 and p2 such that √ n (V − ν) n √ →d N (0, Σ). n (Vn − Un − τ )

6.

EXAMPLES OF LIMITING DISTRIBUTIONS

287

(a) Let 0 < p < 1. Evaluate yp ≡ FY−1 (p), fY (yp ) and p(1 − p)/fY2 (yp ). (b) Determine values of p1 and p2 that achieve the objective. (c) Establish the claimed asymptotic normality, and evaluate Σ both symbolically and numerically. Exercise 6.19 (Estimating a common normal mean) Consider independent rvs X1 , . . . , Xm and Y1 , . . . , Yn from N (θ, σ2 ) and N (θ, τ 2 ). When γ ≡ σ2 /τ 2 is known, the unbiased estimator of θ that has minimum variance (for all possible values of the parameters within this model) is known to be θˆ ≡ θˆmn ≡

¯ m +γ·nY ¯n mX m+γ·n

.

2 ˆ ≡ α ˆmn (SX , SY2 ) depend only on the Deﬁne α ≡ αmn ≡ m/(m +γ · n). Let α m 2 2 ¯ two sample n variances SX ≡ 1 (Xi − Xm ) /(m − 1) = SSXX /(m − 1) and also SY2 ≡ 1 (Yj − Y¯n )2 /(n − 1) = SSY Y /(n − 1), and suppose that α ˆ is a rv with values in [0, 1]. We hypothesize that

α ˆ2 / α2 →p 1

as m ∧ n → ∞.

(All limits below are to be taken as m ∧ n → ∞.) Then deﬁne ¯ n + (1 − α) θˆ ≡ θˆmn = α ˆX ˆ Y¯n , 2 ≡ vo2 ≡ vomn

1 m

α2 σ2 +

2 Vˆ 2 ≡ Vˆmn ≡

1 m

2 α2 SX +

α ˜≡α ˜ mn ≡

m 2 /S 2 m+nSX Y

1 n

(1 − α)2 τ 2 ,

1 n

(1 − α)2 SY2 ,

2 = mSY2 /(mSX + nSY2 ) .

2 Note that γ˜ ≡ γ˜mn ≡ SX /SY2 →p 1, and α ˜ 2 / α2 →p 1 is indeed true. ˆ (a) Show that Eθ = θ. (b) Show that (θˆ − θ)/ vo →d N (0, 1). (c) Show that |Vˆ 2 − vo2 |/ vo2 →p 0. (d) Thus (θˆ − θ)/ Vˆ →d N (0, 1). 2 ˆ in terms of Eˆ (e) Evaluate v2 ≡ vmn ≡ Var[θ] α2 and E(1 − α) ˆ 2. 2 2 2 (f) Attempt to show that |Vˆ − v |/ v →p 0 (view it as a ﬁnal step to claim the normality in (d), if one ﬁrst shows that (θˆ− θ)/ v →d N (0, 1)). [The author believes that you will be required to assume that α/ ˆ α →L2 1 in order to achieve this end.] (There is a lesson here! Note (g).) (g) Does α/ ˜ α →L2 1?

Exercise 6.20 (Exponential estimation) Let X1 , . . . , Xn be iid Exponential(θ). ¯ n . Another unbiased esThe minimum variance estimator of θ is known to be X ¯ n / Γn (1 + 1/n), where G ¯ n ≡ ( n Xk )1/n denotes the getimator of θ is Tn ≡ G 1 ometric mean of the observations. Evaluate the limiting ratio of the variances ¯ n ]/ Var[Tn ]. lim Var[X Exercise 6.21 Let X1 , . . . , Xn be iid Poisson(λ). Show the moment convergence ¯ n − λ|3 → E|N (0, 1)|3 . E|X

288

7

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Classical Convergence in Distribution

Deﬁnition 7.1 (Sub-dfs) (a) Suppose we have rvs Xn ∼ = Fn and X. We now wish to allow the possibility that X is an extended rv. In this case, we assume that H is a sub-df (we will not use the notation F in this context), and we will write X ∼ = H. The interpretation in the case of an extended rv X is that H(−∞) = P (X = −∞), H(x) = P (−∞ ≤ X ≤ x) for all −∞ < x < ∞, and 1 − H(+∞) = P (X = +∞). The set CH of all points at which H is continuous is called the continuity set of H. (b) If Fn (x) → H(x) as n → ∞ at each x ∈ CH of a sub-df H, then we say that Xn (or Fn ) converges in sub-df to X (or H), and we write Xn →sd X (or Fn →sd H) as n → ∞. [What has happened in the case of sub-df convergence is that amounts H(−∞) and 1 − H(+∞) of mass have escaped to −∞ and +∞, respectively.] (c) We have agreed that Fn , F , etc. denote a bona ﬁde df, while Hn , H, etc. may denote a sub-df. Thus Fn →d F (with letter F rather than letter H) will still imply that the limit is necessarily a bona ﬁde df. [The next deﬁnition provides a condition that guarantees (in a totally obvious way, on R at least) that any possible limit is a bona ﬁde df.] Deﬁnition 7.2 (Tightness) A family P of distributions P on R is called tight if for each > 0 there is a compact set (which for one-dimensional rvs is just a closed and bounded set) K with (1)

P (K ) = P (X ∈ K ) ≥ 1 −

for all dfs P ∈ P.

Theorem 7.1 (Helly–Bray) If Fn →d F and g is bounded and continuous a.s. F , then the expectations satisfy (2) Eg(Xn ) = g dFn → g dF = Eg(X). Conversely, if (2) holds for allbounded continuous g, then Fn →d F . [Thus Fn →d F if and only if g dFn → g dF for all bounded and continuous g.] Theorem 7.2 (Continuous mapping theorem; Mann–Wald) Suppose that Xn → X and suppose that g is continuous a.s. F . Then g(Xn ) →d g(X). How do we establish that Fn →d F ? We already have the necessary and suﬃcient condition of Helly–Bray’s theorem 7.1 (or 3.5.1). (We should now recall our deﬁnition of the determining class used in the context of the proof of theorem 3.5.1.) We can also show convergence in distribution of more complicated functions of rvs via Mann–Wald’s continuous mapping theorem 7.2 (or 3.5.2) [for example, Zn →d Z implies that Zn2 →d Z 2 ∼ = χ21 ]. We introduced the concept of tightness above to trivially guarantee that any possible limit is necessarily a bona ﬁde df. This becomes more important in light of the next theorem. Theorem 7.3 (Helly’s selection theorem) Let F1 , F2 , . . . be any sequence of dfs. There necessarily exists a subsequence Fn and a sub-df H for which Fn →sd H as n → ∞. If the subsequence of dfs is tight, then the limit is necessarily a bona ﬁde df.

7.

CLASSICAL CONVERGENCE IN DISTRIBUTION

289

Corollary 1 Let F1 , F2 , . . . be any sequence of dfs. Suppose there is one ﬁxed sub-df H such that for every sd-convergent subsequence {Fn }, we have Fn →sd (this same H). Then the whole sequence satisﬁes Fn →sd H. [In particular, let F1 , F2 , . . . be any sequence of dfs. Suppose there is a ﬁxed df F˜ such that for every subsequence Fn that converges in distribution, we have Fn →d F˜ for this same F˜ . Then Fn →d F˜ .] Proof. Let r1 , r2 , . . . denote the rationals. Using Bolzano–Weierstrass, choose a subsequence n1j such that Fn1j (r1 ) → (some a1 ). A further subsequence n2j also satisﬁes Fn2j (r2 ) → (some a2 ). Continue in this fashion. The diagonal subsequence njj converges to ai at ri for all i ≥ 1. [This Cantor diagonalization technique is important. Learn it!] Deﬁne H on the rationals via H(ri ) = ai . Now deﬁne H on the reals via H(x) = inf{H(ri ) : ri > x}; this H is a sub-df, since it clearly is , takes values in [0, 1], and is right-continuous. Then the diagonal subsequence, we will now refer to it as n , is such that Fn →sd H. Consider the corollary. Fact: Any bounded sequence of real numbers contains a convergent subsequence; and the whole original sequence converges if and only if all subsequential limit points are the same. We showed that every subsequence of Fn contains a further subsequence Fn for which Fn (x) → H(x) for each ﬁxed x ∈ CH . Thus the whole sequence has Fn (x) → H(x) for each x ∈ CH . Thus Fn →sd H. 2 Exercise 7.1 (Convergence of expectations and moments) (a) Suppose Fn →sd H and Fn− (a) → H− (a) and that both Fn (b) → H(b) for some −∞ < a < b < ∞ in CH . Then (3) g dFn → [a,b] g dH for all g ∈ C[a,b] ≡ {g : g is continuous on [a, b]}, [a,b] (4) g dFn → g dH for all g ∈ C0 , where C0 ≡ {g : g is continuous on R and g(x) → 0 as |x| → ∞}. [Actually, g continuous a.s. H suﬃces in (3) and (4).] (b) Let g be continuous a.s. H. If Fn →sd H, then lim inf |g| dFn ≥ |g| dH. (c) Suppose Fn →d F . If g and h are continuous functions such that |g(x)| → ∞ as |x| → ∞ and |h(x)/g(x)| → 0 as |x| → ∞, then lim sup |g| dFn < ∞ implies h dFn → h dF as n → ∞. (d) If E|Xn |r0 < (some M ) < ∞ for all n, then Fn →d F implies (5)

E|Xn |r → E|X|r and EXnk → EX

Exercise 7.2 (P´ olya’s lemma) (6)

for all 0 < r < r0 and all 0 < k < r0 .

If Fn →d F for a continuous df F , then

Fn − F → 0.

Thus if Fn →d F with F continuous and xn → x, then Fn (xn ) → F (x). Exercise 7.3 (Verifying tightness) Suppose Xn ∼ = Fn . Show that {Fn : n ≥ 1} r is tight if either (a) lim E|Xn | < ∞ for some r > 0, or (b) Fn →d F .

290

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Equivalent Deﬁnitions of Convergence in Distribution The condition Fn (x) → F (x) can be rewritten as Pn ((−∞, x]) → P ((−∞, x]), and as E1(−∞,x](Xn ) → E1(−∞,x] (X). Thus →d is reduced to computing expectations of the particularly simple function 1(−∞,x] ; but these simple functions have the disadvantage of being discontinuous. Deﬁnition 7.3 (Closure, interior, and boundary) The closure of B is deﬁned to ¯ ≡ ∩{C : B ⊂ C and C is closed}, while B0 ≡ ∪{U : U ⊂ B and U is open} is be B ¯ is the smallest closed set called the interior of B. These have the property that B containing B, while B0 is the largest open set contained within B. The boundary of ¯ \ B 0 . A set B is called a P -continuity set if P (∂B) = 0. B is deﬁned to be ∂B ≡ B (These deﬁnitions are valid very generally, not just on R.) Theorem 7.4 (→d equivalencies) Let F, F1 , F2 , . . . be the dfs associated with the probability distributions P1 , P2 , . . . . Let Cb denote all bounded, continuous functions g on R, and then let Cbu denote all bounded and uniformly continuous functions g on R. The following are equivalent: (7)

Fn →d F.

(8)

Fn (x) → F (x)

(9)

Eg(Xn ) = g dFn → g dF = Eg(X) for all g in Cb .

(10)

for all x in a dense set.

g dFn → g dF

for all g in Cbu .

(11)

lim Pn (B) ≤ P (B)

for all closed sets B.

(12)

lim Pn (B) ≥ P (B)

for all open sets B.

(13)

lim Pn (B) = P (B)

for all P -continuity sets B.

(14)

lim Pn (I) = P (I)

(15)

L(Fn , F ) → 0 for the L´evy metric L (see below).

for all (including unbounded) P -continuity intervals I.

Exercise 7.4 That (7)–(10) are equivalent is either trivial, or done previously. Cite the various reasons. Then show that (11)–(15) are also equivalent to →d . Exercise 7.5 (L´evy’s metric) between F and G) (16)

For any dfs F and G deﬁne (the 45◦ distance

L(F, G) ≡ inf{ > 0 : F (x − ) − ≤ G(x) ≤ F (x + ) + for all x}.

Show that L is a metric and that the set of all dfs under L forms a complete and separable metric space. Also show that Fn →d F is equivalent to L(Fn , F ) → 0.

7.

CLASSICAL CONVERGENCE IN DISTRIBUTION

291

Convergence of Types Deﬁnition 7.4 (Type) When Y ∼ = (X − b)/a for some a = 0, we say that X and Y are of the same type. [Suppose that Xn →d X where X is not degenerate. Then if an → a = 0 and bn → b, we know from Slutsky’s theorem that (Xn − bn )/an →d Y ∼ = (X − b)/a.] Theorem 7.5 (Convergence of types) Suppose that both Xn →d X and Yn ≡ (Xn − bn )/an →d Y , where an > 0 and X and Y are nondegenerate. Then there exists a > 0 and a real b such that an → a, bn → b and Y ∼ = (X − b)/a. Remark 7.1 The classical CLT implies that if X1 , X2 , . . . are iid (0, σ2 ), then √ Sn / n →d N (0, 1). The above theorem tells us that no matter how we normalize Sn , the only possible nondegenerate limits in distribution are normal distributions. Moreover, √ if Sn /an →d (some rv), the limiting distribution can be nondegenerate only if an / n → (some constant) ∈ (0, ∞). Exercise 7.6 Prove theorem 7.5 on the convergence of types. [Hint. Just consider continuity points x, x of the df FX (·) and continuity points y, y of the df FY (·) for which 0 < FX (x) < FY (y) ≤ FY (y ) < FX (x ) < 1. Then for all n large enough one will have x ≤ an y + bn ≤ an y + bn ≤ x .]

Higher Dimensions If X, X1 , X2 , . . . are k-dimensional random vectors with dfs F, F1 , F2 , . . ., then we say that Xn converges in distribution to X if (17)

Fn (x) → F (x)

for all x ∈ CF ,

just as in one dimension. The Helly–Bray theorem, the Mann–Wald theorem, Helly’s selection theorem, and Polya’s lemma all hold in k dimensions; generalizations of the other results also hold. Moreover, if Xn denotes the ﬁrst j coordinates of Xn , with 1 ≤ j < k, then Xn →d X implies Xn →d X . Exercise 7.7 Prove the k-dimensional Helly–Bray theorem (along the lines of exercise 3.5.2) using Helly’s selection theorem and P´ olya’s lemma. Prove that Xn →d X implies Xn →d X . After reading section 11.8, prove the k-dimensional version of the Mann–Wald theorem. Exercise 7.8

Prove that theorem 7.4 holds in k dimensions.

See also theorem 13.3.2 and theorem 14.1.3 below.

292

8

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Limit Determining Classes of Functions

We can approximate the functions 1(−∞,z] (·) to an arbitrary degree of accuracy within various classes of particularly smooth functions. Within these classes of functions we do not have to worry about the continuity of the limiting measure at z, and this will make these classes more convenient. Indeed, section 11.1 considered the highly specialized class H0 of this type. Deﬁnition 8.1 (Limit determining class) A set G of bounded, continuous functions g is called a determining class if for any dfs F˜ and F , the requirement ˜ that gdF = g dF for all g ∈ G implies F˜ = F . (b) If Eg(Xn ) → Eg(X) for all g ∈ G implies that Xn →d X, then G is called a limit determining class. Deﬁnition 8.2 (Various classes of smooth functions) (i) Let C (let Cb ) [let Cbu ] denote the class of continuous (bounded and continuous) [bounded and also (k) (∞) uniformly continuous] functions on R. Let Cb (let Cb ) denote the subclasses with k (with all) derivatives bounded and continuous. (ii) An extra c on these classes will indicate that all functions vanish outside some compact subset of R. (iii) Let C0 denote the subclass of C that converge to 0 as |x| → ∞. (iv) Let H0 denote the class of all hz, with z real and > 0; here hz, (x) equals 1, is linear, equals 0 according as x is in (−∞, z], is in [z, z + ], is in [z + , ∞) (this class was introduced in the proofs of Helly–Bray’s theorem 3.5.1 and Stein’s lemma 11.1.1). (v) Let G 0 denote the class of all functions ga,b, with a < b and > 0; here ga,b, (x) equals 0, is linear, equals 1 according as x is in (−∞, a − ] ∪ [b + , ∞), is in [a − , a] ∪ [b, b + ], is in [a, b]. Theorem 8.1 (Criteria for →d ; a kinder and gentler Helly–Bray) (i) Let F1 , F2 , . . . be tight. Let G be a limit determining class. (a) If g dFn → (some #g ) for each g ∈ G, then Fn →d F . Moreover, it must be that #g = g dF . (b) Conversely: If Fn →d F , then g dFn → g dF for each g ∈ G. (k) (∞) (ii) Each class C0 , Cb , Cbu , Cb with k ≥ 1, Cb , H0 , and G 0 is both a determining class and a limit determining class. (iii) So, too, if we add an extra subscript c to the various C-classes in (ii). [For some proofs in the literature, functions g with sharp corners are unhandy.] Exercise 8.1

Prove the previous theorem.

Exercise 8.2 (Higher dimensions) Show that the natural extension of each of the results of this section to Rk is valid. Exercise 8.3 Exhibit at least one more limiting determing class.

8.

LIMIT DETERMINING CLASSES OF FUNCTIONS

293

Moments as a Determining Class for a Moment Unique Limit Theorem 8.2 (CLT via moments; Fr´echet–Shohat) (a) Suppose F is the unique df having the speciﬁc moments µk = xk dF (x), for all integers k ≥ 1. Then Fn →d F whenever µnk ≡ xk dFn (x) → µk ≡ xk dF (x) (1) for all k ≥ 1. (b) Any normal df is determined by its moments. Proof. By the Helly selection theorem we have Fn →sd H for some subsequence n and some sub-df H. However, lim E|Xn |2 < ∞, so that {Fn : n ≥ 1} is tight by Markov’s inequality. Thus H is a bona ﬁde df, and Fn →d H. Also, for all k ≥ 1 k x dF (x) = lim xk dFn (x) by hypothesis k = x dH(x) (a) by exercise 11.7.1(d). Thus xk dH(x) = xk dF (x) for all k ≥ 1; and since only F has these moments, we conclude that H = F . Thus Fn →d F . Moreover, Fn →d (this same F ) on any convergent subsequence n . Thus Fn →d F , by the corollary to theorem 11.7.3. See exercise 7.5 for part (b) of the theorem. 2 In general, moments do not determine a distribution uniquely; thus {xk : k ≥ 1} is not a determining class. This is shown by the following exercise. Exercise 8.4 (Moments need not determine the df; Heyde) log X ∼ = N (0, 1); thus √ 2 fX (x) = x−1 e−(log x) /2 / 2π for x > 0.

Suppose that the rv

For each −1 ≤ a ≤ 1, let Ya have the density function fa (y) = fX (y)[1 + a sin(2π log y)] for y > 0. Show that X and each Ya have exactly the same moments. [Knowing that these particular distributions have this property is not worth much; it is knowing that some dfs have this property that matters.] Though we have just seen that moments do not necessarily determine a df, it is often true that a given df F is the unique df having its particular moments (name them {µk : k ≥ 1}). Here is an “exercise” giving various suﬃcient conditions. Exercise 8.5 (When moments do determine a df) Suppose any one of the following hold: ∞ −1/(2k) (a) =∞ (Carleman’s condition). 1 µ2k ∞ k (b) in some interval. 1 µ2k t /(2k)! < ∞ (c)

lim |µk |1/k /k < ∞.

294

CHAPTER 11. CONVERGENCE IN DISTRIBUTION

Then at most one distribution function F can possess this particular set of moments µk = xk dF (x). [Note Feller(1966, pp. 227–228) and/or Breiman(1968, p. 182). Carleman’s condition is actually necessary and suﬃcient.] The exercise is to show that (c) suﬃces. Exercise 8.6 Show that the N (0, 1) distribution is uniquely determined by its moments (by appealing to the previous proposition). Summary The methods of this section that establish →d by verifying the moment condition that Eg(Xn ) → Eg(X) for all functions g in some given determining class G can be extended from the present setting of the real line to more general settings. Also, Chapter 13 carefully considers all the results associated with the particular determining class G ≡ {gt (·) ≡ eit· : t ∈ R}. We can let φ(t) ≡ EeitX = Egt (X) for t ∈ R denote the expectations of all of these gt (X)’s; we call this φ(·) the characteristic function of the rv X. Chapter 13 is a specialized study of the characteristic function. Chapter 14 will apply this characteristic function tool to the CLT. 2

Chapter 12

Brownian Motion and Empirical Processes 1

Special Spaces

General Metric Spaces Let (M, d) denote an arbitrary metric space and let Md denote its Borel σ-ﬁeld (that is, the σ-ﬁeld generated by the collection of all d-open subsets of M ). Let MB d denote the σ-ﬁeld generated by the collection of all open balls, where a ball is a subset of M of the form {y : d(y, x) < r} for some x ∈ M and some r > 0; call this the Baire σ-ﬁeld. [The important concept of weak convergence is best described in the context of metric spaces.] Exercise 1.1 (1)

Now, MB d ⊂ Md , while

MB d = Md

if (M, d) is a separable metric space.

The Special Spaces (C, C) and (D, D) For functions x, y on [0, 1], deﬁne the uniform metric (or supremum metric) by (2)

x − y ≡ sup |x(t) − y(t)| . 0≤t≤1

Let C denote the set of all continuous functions on [0, 1]. Then (3)

(C, ) is a complete and separable metric space.

Here C will denote the σ-ﬁeld of Borel subsets of C; then CB will denote the σ-ﬁeld of subsets of C generated by the open balls, and C will denote the σ-ﬁeld generated by the ﬁnite-dimensional subsets of C (that is, all πt−1 (Bk ) for which 0 ≤ t1 ≤ · · · ≤ tk ≤ 1 and Bk ∈ Bk ). It can be shown that (4)

C

= CB = C . 295

296

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Let D denote the set of all functions on [0, 1] that are right continuous and possess left-hand limits at each point. (In some applications below it will be noted that D is also used to denote the set of all left-continuous functions on [0, 1] that have right-hand limits at each point. This point will receive no further mention. In some cases we will admit to D, and/or to C, only functions X having X(0) = 0, etc. This, too, will receive little, if any, further mention.) Then (5)

(D, ) is a complete metric space that is not separable.

Here D will denote the Borel σ-ﬁeld of subsets of D, then DB will denote the σ-ﬁeld of subsets of D generated by the open balls, and D will denote the σ-ﬁeld generated by the ﬁnite-dimensional subsets of D. It can be shown that (6)

D = DB ,

and both are proper subsets of D ,

and moreover, (7)

C ∈D

and

C = C ∩D.

We now digress brieﬂy. The proper set inclusion of (6) caused diﬃculties in the historical development of √ the theory of empirical processes (note that the uniform empirical process Un = n(Gn − I) takes values in D). To circumvent these diﬃculties, various authors showed that it is possible to deﬁne a metric d on D that has nice properties (see exercise 1.4 below); thus there is a d(·, ·) for which (8)

(D, d) is a complete and separable metric space

whose Borel σ-ﬁeld Dd satisﬁes (9)

Dd = D.

Moreover, for all x, xn in D the metric d satisﬁes (10)

xn − x → 0

implies

d(xn , x) → 0,

while (11)

d(xn , x) → 0 with x ∈ C

implies

xn − x → 0.

The metric d will not be important to us. We are able to replace d by in our theorems; however, we include some information on d as an aid to the reader who wishes to consult the original literature. Exercise 1.2

Verify (3) and (4).

Exercise 1.3 (i) Verify (5). [Hint. For each 0 ≤ t ≤ 1 deﬁne a function xt in D by letting xt (s) equal 0 or 1 according as 0 ≤ s ≤ t or t ≤ s ≤ 1.] (ii) Verify (6). [Hint. Consider ∪{Ot : 0 ≤ t ≤ 1} where Ot is the open ball of radius 13 centered at xt .] (iii) Verify (7).

1.

SPECIAL SPACES

Exercise 1.4 (12)

297

Consult Billingsley (1968, chapter 3) to verify (8)–(11) for

d(x, y) ≡ inf{x − y ◦ λ ∧ λ − I : λ ∈ Λ},

where Λ consists of all ↑ continuous maps of [0, 1] onto itself. [Roughly, this metric measures how closely x and a slightly perturbed (via λ) y line up, where too much perturbation is penalized.] Exercise 1.5 (13)

Verify that

C is both -separable and d-separable, viewed as a subset of D.

[We will require the -separability below.] Let q ≥ 0 be positive on (0, 1). For functions x, y on [0, 1] we agree that (14)

(x − y)/q is the · /q-distance between x and y,

when this is well-deﬁned (that is, when x/q and y/q are ﬁnite). Exercise 1.6 It is useful to be able to view C∞ ≡ C[0,∞) as a metric space; of course, this denotes the class of all continuous functions on [0, ∞). (We may sometimes require a subclass, such as the one consisting of functions that equal zero at zero; and we will make no further mention of this.) Let C∞ ≡ C[0,∞) denote the ﬁnite-dimensional σ-ﬁeld. Consider (C∞ , C∞ ) = (C[0,∞) , C[0,∞) ). (a) For functions x and y on [0, ∞), deﬁne (15)

ρ∞ (x, y) ≡

∞ k=1

2−k

ρk (x, y) , 1 + ρk (x, y)

where ρk (x, y) ≡ sup0≤t≤k |x(t) − y(t)|. Show that (C[0,∞) , ρ∞ ) is a metric space. (b) Show that ρ∞ (x, y) → 0 if and only if ρk (x, y) → 0 for each 0 < k < ∞. (c) Show that (C[0,∞) , ρ∞ ) is a complete and separable metric space. Moreover, the σ-ﬁeld Cρ∞ of Borel subsets is the same as the σ-ﬁeld C[0,∞) of ﬁnite-dimensional B subsets, as is C∞ . (d) Verify that (D[0,∞) , ρ∞ ) is a complete metric space, and that the Borel σ-ﬁeld Dρ∞ satisﬁes C[0,∞) ∈ Dρ∞ and Cρ∞ = Dρ∞ ∩ C[0,∞) . Also, D[0,∞) = DρB∞ is a proper subset of Dρ∞ . (e) Other spaces of continuous and right-continuous functions are analogously treated. They will receive no speciﬁc mention.

Independent Increments and Stationarity If T is an interval in (−∞, ∞), then we will write (16)

X(s, t] ≡ X(t) − X(s)

for any s, t ∈ T,

and we will refer to this as an increment of X. If X(t0 ), X(t0 , t1 ], . . . , X(tk−1 , tk ] are independent rvs for all k ≥ 1 and all t0 < · · · < tk in T , then we say that X has independent increments. If X(s, t] ∼ = X(s + h, t + h] for all s, t, s + h, t + h in T with h ≥ 0, then X is said to have stationary increments. If (X(t1 + h), . . . , X(tk + h)) ∼ = (X(t1 ), . . . , X(tk )) for all k ≥ 1, h ≥ 0, and all time points in T , then X is said to be a stationary process.

298

2

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Existence of Processes on (C, C) and (D, D)

When dealing with processes, we would like to work with the smoothest version possible. This is the version that best models physical reality. It is important at this point to recall theorem 5.4.2 on the existence of smoother versions of processes. Roughly, if all of the sample paths of a process are shown to lie in a (useful) subset of the current image space, then we can restrict ourselves to that subset. Theorem 2.1 (Existence of processes on (C, C)) Begin with a process X : (Ω, A, P ) → (R[0,1] , B[0,1] , PX ) . Suppose that for some a, b > 0 the increments of X satisfy (1)

E|X(s, t]|b ≤ K · F (s, t]1+a

for all 0 ≤ s, t ≤ 1,

where F is a continuous df concentrated on [0, 1] and F (s, t] ≡ F (t) − F (s). Then there exists an equivalent version Z : (Ω, A, P ) → (R[0,1] , B[0,1] , PZ ) for which (2)

Z : (Ω, A, P ) → (C, C, PZ ),

with Z(t) = X(t) a.s., for each t in [0, 1].

Corollary 1 (Sample path properties) For any 0 < δ < a/b and any > 0, there exists a constant K ≡ K,δ,a,b for which the process Z of (2) satisﬁes (3)

P ( |Z(s, t] | ≤ K · F (s, t]δ for all 0 ≤ s ≤ t ≤ 1 ) ≥ 1 − .

Proof. Case 1. Suppose that the df F of (1) is F (t) = t on [0, 1]. Let 0 < δ < a/b be ﬁxed. Let λ ≡ (a/b − δ)/2. Deﬁne tni ≡ i/2n for 0 ≤ i ≤ 2n and n ≥ 1. For n ≥ 0 deﬁne processes Z n : (Ω, A, P ) → (C, C) by letting (a)

Z n(t) ≡ X(tni ) + 2n (t − tni ) [X(tn,i+1 ) − X(tni )]

for tni ≤ t ≤ tn,i+1 ,

for each 0 ≤ i ≤ 2n − 1; thus Z n(·) equals X(·) at each tni and Z n(·) is linear on the intervals between these points. Deﬁne (b)

Uni ≡ | X(tn,i−1 , tni ] |

for 1 ≤ i ≤ 2n .

If we deﬁne (c)

∆n (t) ≡ Z n(t) − Z n−1(t)

for 0 ≤ t ≤ 1,

then for tn−1,i ≤ t ≤ tn−1,i+1 we have |∆n (t)| ≤ | [X(tn,2i ) + X(tn,2i+2 )]/2 − X(tn,2i+1 ) | (d)

= |X(tn,2i , tn,2i+1 ] − X(tn,2i+1 , tn,2i+2 ]|/2 ≤ [Un,2i+1 + Un,2i+2 ]/2

(e)

≤ [Un,2i+1 ∨ Un,2i+2 ]

for all n ≥ 1. Thus for all n ≥ 1 we have (f)

∆n ≤ Vn ≡ [max1≤i≤2n Uni ] .

2.

EXISTENCE OF PROCESSES ON (C, C) AND (D, D)

299

Let θ > 0 be arbitrary but ﬁxed, and deﬁne (g)

pn ≡ P (∆n δ > 2 θ 2−nλ ) ≡ P (|∆n (s, t]|/(t − s)δ > 2 θ 2−nλ for some 0 ≤ s ≤ t ≤ 1).

Recalling (f) shows that (h)

|∆n (s, t]| ≤ 2 Vn

for all 0 ≤ s ≤ t ≤ 1.

Thus (i)

for 2−n ≤ t − s,

|∆n (s, t]|/(t − s)δ ≤ 2 Vn 2nδ

while |∆n (s, t]|/(t − s)δ ≤ [|∆n (s, t]|/(t − s)] (t − s)1−δ ≤ [Vn 2n ] 2−n(1−δ) (j)

for 0 ≤ t − s ≤ 2−n

= Vn 2nδ

(to see this, consider |∆n (s, t]/(t − s)| when s and t are both points in some [tn,i−1 , tni ]). Thus for all n ≥ 1, we have pn ≤ P (2 Vn 2nδ > 2 θ 2−nλ ) ≤ P (Vn > θ 2−n(δ+λ) ) by (g) 2n by (f) ≤ i=1 P (Uni > θ 2−n(δ+λ) ) 2n b ≤ i=1 E Uni /[θ 2−n(δ+λ) ]b by Markov’s inequality ≤ 2n [K2−n(1+a)]/[θ 2−n(δ+λ) ]b

by (1)

= K θ−b 2−n(a−b(δ+λ)) = K θ−b 2−nb(a/b−δ−λ) (k)

= K θ−b 2−nλb

since a/b − δ = 2λ > 0.

Since 0 ≤ t − s ≤ 1, we also have p0 ≡ P ( |Z 0(s, t]| > 2 θ (t − s)δ for some 0 ≤ s ≤ t ≤ 1) ≤ P ( |X(0, 1]| > 2 θ) ≤ E|X(0, 1]|b /(2θ)b (l)

≤ K θ−b = K θ−b 2−0·λ b .

Now, λb = (a − δb)/2 > 0, and so 2−λ b < 1; hence ∞ n=0 pn < ∞. Thus for arbitrarily small θ, we have for m suﬃciently large (recall (g) for · δ ) that P (maxm≤k<∞ Z k − Z mδ > θ) = lim P (maxm≤k≤n Z k − Z mδ > θ) n→∞

≤ lim P (maxm≤k≤n Z k − Z mδ > 2 θ

n

n→∞

for

∞ m+1

2−kλ < 1

≤ lim P (maxm≤k≤n n→∞

≤ lim

n→∞

(m)

≤

n i=m+1

∞

i=m+1

2−kλ )

m+1

pi ≤

k i=m+1

∆i δ > 2 θ

n i=m+1

2−iλ )

P (∆i δ > 2 θ 2−iλ ) ∞ i=m+1

Kθ−b 2−iλ b → 0

as m → ∞,

300

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

so that (n)

Z n(t) ≡ Z 0(t) +

n k=1

∆k (t)

converges uniformly on [0, 1] for a.e. ω; call the limit function Z(t). Since the uniform limit of continuous functions is continuous, ∞ (o) Z = n=0 ∆n = lim Z n is a continuous function on [0, 1] for a.e. ω. Now, Z = lim Z n, and since Z n equals X at each tni , we have (4)

Z(t ni) = X(tni )

at each tni = i/2n with 0 ≤ i ≤ 2n and n ≥ 0.

Thus all ﬁnite-dimensional distributions with diadic rational coordinates are equal. For other t, we pick diadic rationals t1 , t2 , . . . such that tm → t. Then X(tm ) →p X(t) as m → ∞ by (1) and Chebyshev, while Z(t m) →a.s. Z(t) as m → ∞, since Z has continuous sample paths. Thus Z(t) = X(t) a.s. by proposition 2.3.4. By redeﬁning Z ≡ 0 on the null set of (n), we may assume (p)

Z : (Ω, A) → (C, C)

by theorem 5.4.2. So ﬁnite-dimensional distributions agree: In particular, we have PZ ([x ∈ C : xt ∈ B]) = PX ([x ∈ R[0,1] : xt ∈ B]) for all sets B ∈ Bk and for all t ∈ [0, 1]k for any k ≥ 1. Case 2. General F . Deﬁne (q)

Y(t) ≡ X(F −1 (t))

for 0 ≤ t ≤ 1,

where F −1 (t) ≡ inf{x ∈ [0, 1] : F (x) ≥ t}. Then for 0 ≤ s ≤ t ≤ 1, E|Y(s, t]|b = E|X(F −1 (s), F −1 (t)] |b ≤ K [ F ◦ F −1 (t) − F ◦ F −1 (s)]1+a (r)

= K (t − s)1+a ,

since F ◦ F −1 = I for continuous F by exercise 7.3.2. Now use case 1 to replace Y ¯ : (Ω, A) → (C, C). Then deﬁne by an equivalent process Y (s)

¯ )∼ Z = Y(F = X(F −1 ◦ F )

by (q).

Now, F −1 ◦ F (t) = t, unless F (t − ) = F (t) for some > 0; see exercise 7.3.2. But in this case equation (1) shows that ∆X is 0 across that same interval. Thus X(F −1 ◦ F ) ∼ = X. For the corollary, in case 1 we have (using (o) in line 2, (g) and (k) in line 3) P (| Z(s, t] |/(t − s) δ > 2 θ/(1 − 2−λ ) for some 0 ≤ s ≤ t ≤ 1) ∞ ∞ = P (| k=0 ∆k (s, t] |/(t − s)δ > 2 θ n=0 2−nλ for some 0 ≤ s ≤ t ≤ 1) ∞ ∞ ≤ n=0 pn ≤ n=0 K θ−b /2n λ b = K θ−b /(1 − 2−λ b ) (t)

→0

as θ → ∞.

Take K to be an appropriately large value of θ. Use the transformation F −1 again in case 2. 2

2.

EXISTENCE OF PROCESSES ON (C, C) AND (D, D)

301

Exercise 2.1 Prove (2), by simplifying the proof of theorem 2.1 as much as possible with this simpler goal in mind. We merely state an analogous result for the existence of processes on (D, D). Theorem 2.2 (Existence of processes on (D, D))

Consider a process

X : (Ω, A, P ) → (R[0,1] , B[0,1] , PX ) . Suppose that for some K > 0, b > 0, and a > (5)

1 2

E|X(r, s] X(s, t] |b ≤ K · F (r, s]a F (s, t]a

we have for all 0 ≤ r ≤ s ≤ t ≤ 1,

where F is any df concentrated on [0, 1]. Then there exists an equivalent process Z : (Ω, A, P ) → (R [0,1] , B[0,1] , PZ ), which in fact satisﬁes (6)

Z : (Ω, A, P ) → (D, D, PZ ),

[See Billingsley (1968, pp. 130, 134).]

with Z(t) = X(t) a.s., for each t ∈ [0, 1] .

302

3

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Brownian Motion and Brownian Bridge

We deﬁne {S(t) : 0 ≤ t ≤ 1} to be a Brownian motion on [0, 1] if S is a normal process having the moment functions (1)

ES(t) = 0

and

Cov[S(s), S(t)] = s ∧ t

for all 0 ≤ s, t ≤ 1.

This covariance function is nonnegative deﬁnite (in the sense of section 9.3), and these distributions are consistent; thus Kolmogorov’s consistency theorem shows that the process S exists as a random element on (R [0,1] , B[0,1] ). Modifying this S on a set of measure zero (as in theorem 12.2.1), we may may create a version of S that satisﬁes (2)

all sample paths of S are continuous functions on [0, 1] that equal 0 at 0.

Thus (as with the smoother realizations of theorem 5.4.2) there is a nice realization of S having smoother paths; that is, (3)

S exists as a process on (C, C).

So, Brownian motion exists as the coordinate map St (ω) ≡ ωt for some distribution P on (Ω, A) = (C, C). This is a more convenient realization of S (than is the one guaranteed by Kolmogorov’s consistency theorem). For either realization (4)

S has stationary and independent increments.

In fact, its sample paths satisfy (5)

P (|S(s, t]| ≤ K (t − s)δ for all 0 ≤ s ≤ t ≤ 1) ≥ 1 −

for some K,δ , for any ﬁxed > 0 and for any ﬁxed 0 < δ < 12 . This follows from theorem 12.2.1 and its corollary, since for any k ≥ 1, (6)

E S(s, t] 2k = [1 · 3 · · · (2k − 1)] (t − s)k

for all 0 ≤ s ≤ t ≤ 1,

and since a/b = (k − 1)/(2k) 12 as k → ∞. (Note that (5) would allow a further application of the smoother realizations theorem using just this smaller subset of such functions in C.) [No appeal has been made to section 1.] Let us now deﬁne (7)

V(t) ≡ S(t) − t S(1)

and

U(t) ≡ −V(t)

for 0 ≤ t ≤ 1.

Then both U and V are obviously normal processes on (C, C) and satisfy (5); just observe that V(t) is a simple linear combination of two normal rvs. Moreover, trivial calculations give (8)

EU(t) = 0

and

Cov[U(s), U(t)] = s ∧ t − st

for all 0 ≤ s, t ≤ 1.

Call U a Brownian bridge. And V is also a Brownian bridge. In a similar fashion we establish the existence of Brownian motion on (C∞ , C∞ ). In particular, a Brownian motion on (C∞ , C∞ ) is given by (9)

S(t) = (1 + t) U(t/(1 + t)),

0 ≤ t < ∞.

3.

BROWNIAN MOTION AND BROWNIAN BRIDGE

303

Recall the proposition 10.7.1 LIL result. In section 12.8 we will establish the companion LIL result for Brownian motion that √ lim sup |S(t)|/[ t b(t)] = 1 a.s. (10) (the LIL for S at inﬁnity), t→∞

where b(t) ≡ 2 (1 ∨ log log t). (We will use it in very minor ways in the meantime.) The next exercise similarly deﬁnes some additional normal processes. These may provide a useful revisualization device that enables calculation. Exercise 3.1 (Transformations of Brownian motion) Let Z ∼ = N(0, 1) and the Brownian bridges V, U(1) , and U(2) be independent. Fix a > 0. Show that: (11) (12) (13) (14)

S(t) = V(t) + tZ, 0 ≤ t ≤ 1, √ S(at)/ a, 0 ≤ t < ∞,

is a Brownian motion.

S(a + t) − S(a), t ≥ 0, √ √ 1 − a U(1) ± a U(2)

is a Brownian motion.

is a Brownian motion.

is a Brownian bridge.

(15)

√ 1 Z(t) ≡ [U(1) (t) + U(2) (1 − t)]/ 2, 0 ≤ t ≤ , 2

is a Brownian motion.

(16)

U(t) = (1 − t) S(t/(1 − t)),

is a Brownian bridge;

0 ≤ t ≤ 1,

use the LIL at inﬁnity of (10) to show that this U(·) converges to 0 at t = 1. (17)

t S(1/t),

0 ≤ t < ∞,

is a Brownian motion;

apply the LIL of (10) to verify that these sample paths converge to 0 at t = 0. Exercise 3.2 (LIL for S and U at 0) Use (10), (17), and then (7) to show that √ √ lim sup |S(t)|/[ t b(1/t)] = 1 a.s.; and lim sup |U(t)|/[ t b(1/t)] = 1 a.s. (18) t→0

t→0

Exercise 3.3 (Integrals of normal processes are normal rvs) (a) Suppose X is a normal process on (C, C) with mean function m(·) and covariance function Cov(·, ·) that are both continuous on [0, 1]. Let the weight function g(·) also be continuous. Suppose X/q10 = Op (1) for a function q ≥ 0 that is on (0, 12 ] and on [ 12 , 1). Let K(·) be an and left continuous function. Suppose 1 also that 0 q |g| dK < ∞. Show that the integrated process 1 X(t) g(t) dK(t) 0 (19) 1 1 1 ∼ Cov(s, t) g(s)g(t) dK(s) dK(t)) , = N ( m(t) g(t) dt, 0

0

0

provided that both its mean and variance are well-deﬁned. 1 (b) Determine the distribution of 0 U(t) dt. 1 (c) Develop results for 0 S g dK, for appropriate functions g and K. [Hint. (a) The Riemann sums are normally distributed. (b) Perhaps, use (18).]

304

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

√ Exercise 3.4 Let Z0 , Z1 , Z2 , . . . be iid N (0, 1). Let fj (t) ≡ 2 sin(jπt), for j ≥ 1; these are orthogonal functions. Verify that U(t) ≡ ∞ (20) 0 ≤ t ≤ 1, is a Brownian bridge. j=1 Zj fj (t) / jπ, Thus the process S(t) ≡ U(t) + t Z 0 is a Brownian motion on [0, 1]. Moreover, 1 ∞ W 2 ≡ 0 U2 (t) dt ∼ (21) = j=1 Zj2 /(j 2 π 2 ). This rv has a distribution that is well tabled (the asymptotic null distribution of the Cram´er–von Mises statistic). Show that Z is a Brownian motion on [0, 1], where t for 0 ≤ t ≤ 1. Z(t) ≡ U(t) + 0 [U(s)/(1 − s)] ds

Exercise 3.5 (22)

Exercise 3.6 (White noise) (a) Suppose that h and ˜h on [0, 1] are in L2 . View white noise as an operator dS that takes the function h into a rv [0,1] h(t) dS(t) in the sense of →L2 . Deﬁne this integral ﬁrst for step functions, and then use exercise 4.4.5 to deﬁne it in general. Then show that [0,1] h(t) dS(t) exists as such an →L2 limit for all h in L2 . (b) In case h has a bounded continuous derivative h on [0, 1], show that 1+ (23) [0,1] h(t) dS(t) ≡ hS| 0− − [0,1] S(t) h (t) dt . ˜ dS(t). (c) Determine the joint distribution of [0,1] h(t) dS(t) and [0,1] h(t) (d) Deﬁne [0,1] h(t) dU(t), and obtain the marginal and joint distributions of all three of the rvs in (c) and (d). Exercise 3.7 (Conditional Brownian motion) Let 0 ≤ r < s < t. Determine the conditional distribution of S(s) given that S(r) = y and S(t) = z. Put your answer in a format that allows some insight to be oﬀered. Exercise 3.8 Find the solution V (t) of the stochastic diﬀerential equation with V (t) = −kV (t) + σS (t). Determine its covariance function. (Think of a tiny particle suspended in a liquid whose velocity is impeded by the viscosity of the liquid and is additionally subjected to random changes from collisions with particles in the medium.) [Hint. Rewrite the equation ﬁrst as ekt [V (t) + kV (t)] = σ ekt S (t), then transform it to t V (t) = V (0)e−t + σ 0 e−σ(t−s) dS(s) , and then use integration by parts to give meaning to dS(·).]

4.

STOPPING TIMES

4

Stopping Times

305

We ﬁrst paraphrase the main result of this section. If we observe a right-continuous process at a random time that depends on the process only through its past, then the result is a rv (that is, it is measurable). Notation 4.1 Let (Ω, A, P ) denote our basic probability space. We suppose that our time set is a linearly ordered set such as [0, 1], [0, ∞), [0, ∞], {0, 1, 2, . . .}, {0, 1, 2, . . . , ∞}. Let X denote a process with such an index set, deﬁned on (Ω, A, P ). We now suppose that the At ’s are an collection of sub σ-ﬁelds of A, in that As ⊂ At whenever s < t. If it further holds that each Xt is an At -measurable rv, then we say that the process is adapted to the At ’s. The minimal such collection of σ-ﬁelds is the histories σt ≡ σ[Xs−1 (B) : s ≤ t]. Roughly, σt denotes all events for the process up to time t. We let At+ ≡ ∩∞ n=1 At+1/n ; and if At+ = At for all t ≥ 0, then we call the σ-ﬁelds At right continuous. Let At− ≡ σ[As : s < t]. Then 2 let A∞ ≡ σ[∪t<∞ At ]. Deﬁnition 4.1 (Stopping times) An (extended) rv τ ≥ 0 will be called an (extended) stopping time with respect to the At ’s if [τ ≤ t] ∈ At for all t ≥ 0. (That is, one can determine whether τ ≤ t using only current knowledge At .) Roughly, whether τ “stops” or “occurs” by time t or not depends only on those events At with probabilities within our knowledge base up through time t. We deﬁne the pre-τ σ-ﬁeld Aτ ≡ {A ∈ A : A ∩ [τ ≤ t] ∈ At for all t ≥ 0};

(1)

roughly, at any instant we can decide whether or not A has occurred yet. Note that if τ (ω) ≡ t for all ω, then Aτ = At ; that is, the ﬁxed time t is a stopping time whose σ-ﬁeld is At . We now develop some other technical properties. Proposition 4.1 (Preservation of stopping times) stopping times. Then: and T1 ∧ T2

Suppose the rvs T1 , T2 , . . . are

(2)

T1 ∨ T2

(3)

If Tn , then T ≡ lim Tn is a stopping time.

(4)

If Tn and At ’s are right continuous, then T ≡ lim Tn is a stopping time.

are stopping times.

This proposition is also true for extended stopping times. Proof.

Note that these four rvs satisfy

(a)

[T1 ∨ T2 ≤ u] = [T1 ≤ u] ∩ [T2 ≤ u] ∈ Au ,

(b)

[T1 ∧ T2 ≤ u] = [T1 ≤ u] ∪ [T2 ≤ u] ∈ Au ,

(c)

∞ [T ≤ u] = ∩∞ n=1 [Tn ≤ u] = ∩n=1 (events in Au ) ∈ Au ,

306

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

(d)

∞ ∞ [T ≤ u] = ∩∞ m=1 ∪n=1 [Tn ≤ u+1/m] = ∩m=1 (Au+1/m events) ∈ Au+ = Au .

Thus each of these is a stopping time. (No change is needed for extended stopping times.) 2 Proposition 4.2 (Integral stopping times) Integer-valued T ≥ 0 is a stopping time if and only if [T = n] ∈ An for all 0 ≤ n < ∞. This result is also true for extended stopping times. Proof. If T is a stopping time, then [T = n] = [T ≤ n] ∩ [T ≤ n − 1]c is in An , c since [T ≤ n − 1]c ∈ An−1 ⊂ An . Also, [T = ∞] = (∪∞ n=1 [T ≤ n]) ∈ A. Going the other way, [T ≤ n] = ∪m≤n [T = m] ∈ An , since [T = m] ∈ Am ⊂ An . Also, [T ≤ ∞] = Ω ∈ A∞ . 2 Exercise 4.1 (Properties of stopping times) times. Then: (5)

Let T1 , T2 , . . . be extended stopping

T1 + T2 is a stopping time for right-continuous At ’s.

[Hint. Form a (limit of a) countable union involving the rationals.] (6)

A ∈ AT1 implies A ∩ [T1 ≤ T2 ] ∈ AT2 .

[Hint. [T1 ∧ t ≤ T2 ∧ t] ∈ At .]

(7)

T1 ≤ T2 implies AT1 ⊂ AT2 .

(8)

If Tn T0 and the At ’s are right continuous, then AT0 = ∩∞ n=1 ATn .

Deﬁnition 4.2 (Progressively measurable) Let {X(t) : t ≥ 0} be a process. Let Bt denote the Borel subsets of [0, t]. Call X progressively measurable, and denote this type of measurability by writing X : ([0, t] × Ω, Bt × At ) → (R, B)

for each t ≥ 0,

provided that for each t in the index set we have (9)

{(s, ω) : 0 ≤ s ≤ t and X(s, ω) ∈ B} is in Bt × At .

Proposition 4.3 (Right-continuous processes) Suppose X : (Ω, A, P ) → (D[0,∞) , D[0,∞) ) is a process adapted to the At ’s and suppose that τ is a stopping time with respect to the At ’s. Then: (10)

Aτ is a σ-ﬁeld.

(11)

τ is Aτ -measurable.

(12)

X is progressive measurable.

(13)

The histories σt generated by X are right continuous.

(14)

X(τ ) is Aτ -measurable.

We may replace [0, ∞) by any [0, θ) or [0, θ] with 0 < θ < ∞. [Essentially, one can work nicely with right-continuous processes.] [Right-continuous X may be replaced by progressively measurable X.]

4.

STOPPING TIMES

307

Proof. That Aτ is a σ-ﬁeld and that τ is Aτ -measurable are trivial for any image space. (For example, for each real number r we have [τ ≤ r] ∈ Aτ , since [τ ≤ r] ∩ [τ ≤ t] = [τ ≤ r ∧ t] ∈ Ar∧t ⊂ At for each t ≥ 0.) Now, (12) holds, since X = lim Xn (by right continuity), where X((k + 1)/2n ∧ t, ω) for k/2n ≤ s < (k + 1)/2n , k ≥ 0, s < t, (a) Xn (s, ω) ≡ X(t, ω) for s = t clearly satisﬁes Xn : ([0, t] × Ω, Bt × At ) → (R, B) is measurable. That is, Xn is progressively measurable; thus the process X is also, by proposition 2.2.2. That is, X : ([0, t] × Ω, Bt × At ) → (R, B) is measurable. Likewise, the histories of these Xn ’s are right continuous, and X inherits this property from the Xn ’s having it. The following type of truncation argument with stopping times is common; learn it. We must show that [X(τ ) ∈ B] ∩ [τ ≤ t] ∈ At , for all Borel sets B ∈ B. But setting τ ∗ ≡ τ ∧ t, we see that this set is equal to [X(τ ∗ ) ∈ B] ∩ [τ ≤ t], and hence it suﬃces to show that [X(τ ∗ ) ∈ B] ∈ At . Now, the mapping ω → (τ ∗ (ω), ω) is a measurable mapping from (Ω, At ) to ([0, t] × Ω, Bt × At ), since for A ∈ At and the identity function I we have (for sets [0, s] × A generating Bt × At ) (b)

[(τ ∗ , I) ∈ [0, s] × A] = [τ ∗ ≤ s] ∩ A = [τ ∧ t ≤ s] ∩ A ∈ At .

Combining this (τ ∗ , I) : (Ω, At ) → ([0, t] × Ω, Bt × At ) measurability with the progressive measurability X : ([0, t] × Ω, Bt × At ) → (R, B) shown above, we see that the composition map ω → X(τ ∗ (ω), ω) is B-At -measurable. We express this by writing [X(τ ∗ ) ∈ B] ∈ At . 2 Exercise 4.2 Let Ta ≡ inf{t : S(t) = a} for a Brownian motion S on (C, C). Then Ta is a stopping time, but Ta − a is not. Let 0 < a < b. Now A ≡ [S ≥ a] is in ATa , but B ≡ [S ≥ b] is not. Exercise 4.3

Write out the full details for the truth of (13).

308

5

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Strong Markov Property

We now extend the strong Markov property (which was proved for discrete-time processes in section 10.7) to processes with stationary and independent increments. Theorem 5.1 (Strong Markov property) Consider the stochastic process X : (Ω, A, P ) → (D[0,∞) , D[0,∞) ) adapted to right-continuous At ’s. Suppose that X(0) = 0, X has stationary and independent increments, and suppose that the increment X(t + s) − X(t) is independent of At for all s ≥ 0. Let τ be an extended stopping time for the At ’s, and suppose P (τ < ∞) > 0. For some t ≥ 0 we deﬁne X(τ + t) − X(τ ) on [τ < ∞] , (1) Y (t) ≡ 0 on [τ = ∞] .

Then Y : [τ < ∞] ∩ Ω, [τ < ∞] ∩ A, P ( · [τ < ∞]) → (D[0,∞) , D[0,∞) ) and (2)

P Y ∈ F [τ < ∞] = P (X ∈ F )

for all F ∈ D[0,∞) .

Moreover, for all F ∈ D[0,∞) and for all A ∈ Aτ , we have

(3) P [Y ∈ F ] ∩ A[τ < ∞] = P ( [X ∈ F ] ) × P A[τ < ∞] . Thus if P (τ < ∞) = 1, then X and Y are equivalent processes and the process Y is independent of the σ-ﬁeld Aτ . Proof. That Y : (Ω ∩ [τ < ∞], A ∩ [τ < ∞], P (·|[τ < ∞]) → (D[0,∞) , D[0,∞) ) follows from proposition 12.4.3. This proposition and exercise 12.4.1 show that (4)

At ≡ Aτ +t are and right continuous, with Y adapted to the At ’s.

Case 1. Suppose the ﬁnite part of the range of τ is a countable subset {s1 , s2 , . . .} of [0, ∞). Let t1 , . . . , tm ≥ 0, let B1 , . . . , Bm be Borel subsets of the real line, and let A ∈ Aτ . Then P ([Y (t1 ) ∈ B1 , . . . , Y (tm ) ∈ Bm ] ∩ A ∩ [τ < ∞]) = = =

k

P ([Y (t1 ) ∈ B1 , . . .] ∩ A ∩ [τ = sk ])

k

P ([X(t1 + sk ) − X(sk ) ∈ B1 , . . .] ∩ A ∩ [τ = sk ])

k

P (X(t1 + sk ) − X(sk ) ∈ B1 , . . .) P (A ∩ [τ = sk ])

= P (X(t1 ) ∈ B1 , . . .) (a)

k

P (A ∩ [τ = sk ])

= P (X(t1 ) ∈ B1 , . . . , X(tm ) ∈ Bm ) P (A ∩ [τ < ∞]),

where the third equality holds as A ∩ [τ = sk ] = (A ∩ [τ ≤ sk ]) ∩ [τ = sk ] is in Ask , and is thus independent of the other event by the independent increments of X.

5.

STRONG MARKOV PROPERTY

309

Putting A = [τ < ∞] in (a) yields P (Y (t1 ) ∈ B1 , . . . , Y (tm ) ∈ Bm |[τ < ∞]) (b)

= P (X(t1 ) ∈ B1 , . . . , X(tm ) ∈ Bm );

substituting (b) into (a) and dividing by P (τ < ∞) yields P ([Y (t1 ) ∈ B1 , . . . , Y (tm ) ∈ Bm ] ∩ A|[τ < ∞]) (c)

= P (Y (t1 ) ∈ B1 , . . . , Y (tm ) ∈ Bm |[τ < ∞]) P (A|[τ < ∞]).

Thus (b) and (c) hold for the class G of sets of the form [Y (t1 ) ∈ B1 , . . . , Y (tm ) ∈ Bn ] and for all sets A in Aτ . Since G generates Y −1 (D[0,∞) ), equation (b) implies (2). Since G is also closed under ﬁnite intersections (that is, it is a π ¯ -system), (c) and proposition 8.1.1 imply (3). Case 2. Now consider a general stopping time τ . For n ≥ 1, deﬁne $ k/n for (k − 1)/n < τ ≤ k/n and k ≥ 1, τn ≡ 1/n for τ = 0, (d) ∞ for τ = ∞ . Note that τn (ω) τ (ω) for ω ∈ [τ < ∞]. For k/n ≤ t < (k + 1)/n we have [τn ≤ t] = [τ ≤ k/n] ∈ Ak/n ⊂ At (so that τn is a stopping time), and also for A in Aτ that A ∩ [τn ≤ t] = A ∩ [τ ≤ k/n] ∈ Ak/n ⊂ At (so that Aτ ⊂ Aτn ). Deﬁne (e)

Yn (t) = X(τn + t) − X(τn )

on [τn < ∞] = [τ < ∞],

and let it equal 0 elsewhere. By case 1 results (b) and (c), both (f)

P (Yn ∈ F |[τ < ∞]) = P (X ∈ F )

(g)

P ([Yn ∈ F ] ∩ A |[τ < ∞]) = P (Yn ∈ F |[τ < ∞]) P (A |[τ < ∞])

and

hold for all F in D[0,∞) and all A in Aτ (recall that Aτ ⊂ Aτn as above, and [τ < ∞] = [τn < ∞]). Let (r1 , . . . , rm ) be any continuity point of the joint df of (Y (t1 ), . . . , Y (tm )), and deﬁne

(h)

Gn G G∗ H

≡ ≡ ≡ ≡

[Yn (t1 ) < r1 , . . . , Yn (tm ) < rm , τ < ∞], [Y (t1 ) < r1 , . . . , Y (tm ) < rm , τ < ∞], [Y (t1 ) ≤ r1 , . . . , Y (tm ) ≤ rm , τ < ∞], [X(t1 ) < r1 , . . . , X(tm ) < rm ].

By the right continuity of the sample paths, Yn (t) → Y (t) for every t and every ω in [τ < ∞]; thus (i)

G ⊂ lim Gn ⊂ lim Gn ⊂ G∗ .

310

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Thus P (G|τ < ∞) ≤ P (lim Gn |[τ < ∞]) ≤ lim P (Gn |τ < ∞) by (i), then DCT = P (H) = lim P (Gn |τ < ∞)

by using (f) twice

by the DCT and (i) ≤ P ( lim Gn |τ < ∞) ≤ P (G∗ |τ < ∞) m ≤ P (G|τ < ∞) + i=1 P (Y (ti ) = ri |τ < ∞) (j)

= P (G|τ < ∞),

since (r1 , . . . , rm ) is a continuity point. Thus (j) implies (k)

P (G|τ < ∞) = P (H),

and this is suﬃcient to imply (2). Likewise, for A ∈ Aτ ⊂ Aτn , P (G ∩ A|[τ < ∞]) ≤ P ( lim Gn ∩ A|τ < ∞)

by (i)

≤ lim P (Gn ∩ A|τ < ∞) by the DCT = lim P (Gn |τ < ∞) P (A|τ < ∞) = P (G|τ < ∞) P (A|τ < ∞)

by (j)

= lim P (Gn |τ < ∞) P (A|τ < ∞) = lim P (Gn ∩ A|τ < ∞)

by (c), with [τ < ∞] = [τn < ∞]

by (j)

by (c), with [τ < ∞] = [τn < ∞]

≤ P ( lim Gn ∩ A|τ < ∞) ≤ P (G∗ ∩ A|τ < ∞) by the DCT, then (i) m ≤ P (G ∩ A|τ < ∞) + i=1 P (Y (ti ) = ri |τ < ∞) (l)

= P (G ∩ A|τ < ∞),

since (r1 , . . . , rm ) is a continuity point. Thus (l) implies (m)

P (G ∩ A|τ < ∞) = P (G|τ < ∞) P (A|τ < ∞);

and using proposition 8.1.1 again, we see that this is suﬃcient to imply (3). The ﬁnal statement is immediate, since when P (τ < ∞) = 1 we necessarily have P (A|τ < ∞) = P (A) for all A ∈ A. 2

Exercise 5.1 Formulate (9.1.17) rigorously.

6.

EMBEDDING A RV IN BROWNIAN MOTION

311

6

Embedding a RV in Brownian Motion

Let a, b > 0. For a Brownian motion S on (C ∞ , C∞ ), we deﬁne τ ≡ τab ≡ inf{t : S(t) ∈ (−a, b) c }

(1)

to be the ﬁrst time S hits either −a or b. Call τ a hitting time. [Show that τ is a stopping time.] Note ﬁgure 6.1.

b

(·)

0

τab

–a

Figure 6.1 The stopping time τab .

Theorem 6.1 (Embedding via τab )

Let τ ≡ τab . Then:

(2)

ES(τ ) = 0 .

(3)

P (S(τ ) = −a) = b/(a + b)

(4)

Eτ = a b = ES 2(τ ) .

(5)

Eτ r ≤ r Γ(r) 22r ES 2r(τ ) ≤ r Γ(r) 22r ab (a + b)2r−2

and

P (S(τ ) = b) = a/(a + b) .

for all r ≥ 1 .

Deﬁnition 6.1 (Martingale) A process {M (t) : t ≥ 0} is a continuous parameter martingale (mg) if E|M (t)| < ∞ for all t, M is adapted to the At ’s, and (6)

E{M (t) |As } =a.s. M (s)

for all 0 ≤ s ≤ t.

Deﬁnition 6.2 (Stopping time) If τ is a random time (just a rv that is ≥ 0) for which the event [τ ≤ t] ∈ At for all t, then we call τ a stopping time.

312

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Future theorem Let τ be a stopping time. With appropriate regularity conditions on a mg M , we can claim that (7)

EM (τ ) = EM (0) .

Our present applications are simple special cases of a result called the optional sampling theorem for mgs. The general version will be proven in chapter 18. We will use it for such simple special cases now. 2 Proof. The independent increments of S lead to satisfaction of the mg property stated in (6). Also, S is suitably integrable (we will see later) for (7) to hold. Thus, with p ≡ P (S(τ ) = b), we have (a)

0 = ES(τ ) = bp − a(1 − p),

or

p = a/(a + b) .

Also, the process (8)

{S 2(t) − t : t ≥ 0}

is a mg adapted to the σ-ﬁelds At ≡ σt ,

since E{S 2(t) − t|As } = E{[S(t) − S(s) + S(s)] 2 − t|As } = E{[S(t) − S(s)] 2 + 2 S(s)[S(t) − S(s)] + S 2(s) − t|As } = E{[S(t) − S(s)] 2} + 2 S(s)E{S(t) − S(s)} + S 2(s) − t = t − s + 2 S(s) · 0 + S 2(s) − t = S 2(s) − s .

(b)

This process is also suitably integrable, so that optional sampling can be used to imply E[S(τ ) 2 − τ ] = 0. Thus (c)

Eτ = ES 2(τ ) = (−a)2 · b/(a + b) + b2 · a/(a + b) = ab .

We leave (5) to exercise 12.7.3 below.

2

Theorem 6.2 (Skorokhod embedding of a zero-mean rv) Suppose X is a rv with df F having mean 0 and variance 0 ≤ σ2 ≤ ∞. Then there is a stopping time τ such that the stopped rv S(τ ) is distributed as X; that is, (9)

S(τ ) ∼ = X.

Moreover, (10)

Eτ = Var[X],

and for any r ≥ 1 we have (11)

Eτ r ≤ Kr E|X|2r

with Kr ≡ r Γ(r) 22r .

6.

EMBEDDING A RV IN BROWNIAN MOTION

313

Proof. For degenerate F , just let τ ≡ 0. Thus suppose F is nondegenerate. Let (A, B) be independent of S, with joint df H having (12)

for a ≥ 0, b > 0 .

dH(a, b) = (a + b) dF (−a) dF (b)/EX+

The procedure is to observe (A, B) = (a, b) according to H, and then to observe τab , calling the result τ . (Clearly, τab = 0 if a = 0 is chosen.) Note that [τ ≤ t] can be determined by (A, B) and {S(s) : 0 ≤ s ≤ t}, and hence is an event in At ≡ σ[A, B, S(s) : 0 ≤ s ≤ t}. For t ≥ 0,

(a)

P (S(τ ) > t) = E(P {S(τ ) > t|A = a, B = b}) = [0,∞) (0,t] 0 · dH(a, b) + [0,∞) (t,∞) (a/(a + b)) dH(a, b)

(b)

=

(c)

= 1 − F (t) ,

(t,∞)

[0,∞)

a dF (−a) dF (b) / EX + =

(t,∞)

by (3)

dF (b) EX − / EX +

since EX = 0 with X nondegenerate implies EX + = EX − . Likewise, for t ≥ 0, (d) P (S(τ ) ≤ −t) = [0,t) (0,∞) 0 · dH(a, b) + [t,∞) (0,∞) (b/(a + b)) dH(a, b)

(e)

=

(f)

= F (−t) .

[t,∞)

(0,∞) b dF (b) dF (−a) / EX

+

=

[t,∞)

dF (−a)

Thus S(τ ) ∼ = X. Moreover, Eτ = E(E{τ |A = a, B = b}) = E(E{S 2(τ )|A = a, B = b}) = ES 2(τ ) (g)

= EX 2 = Var[X] .

Note that (a + b)2r−1 ≤ 22r−2 [a2r−1 + b2r−1 ] by the Cr -inequality. Thus Eτ r = E(E{τ r |A = a, B = b}) (h)

(i) (j)

≤ 4 r Γ(r) E(AB(A + B)2r−2 )

by (5)

≤ 4 r Γ(r) E(AB(A + B)2r−1 /(A + B)) B A A2r + B 2r ≤ 4 r Γ(r) 22r−2 E A+B A+B = Kr E(E{S 2r(τ )|A = a, B = b}) = Kr E(S 2r(τ )) = Kr EX 2r ,

as claimed.

2

314

7

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Barrier Crossing Probabilities

For −a < 0 < b we deﬁned the hitting time (1)

τab ≡ inf{t : S(t) ∈ (−a, b) c },

where S denotes Brownian motion on (C ∞ , C∞ ). We also considered the rv S(τ ab), which is called Brownian motion stopped at τab . We saw that it took on the two values b and −a with the probabilities p ≡ a/(a + b) and q ≡ 1 − p = b/(a + b). For a > 0 we deﬁne the stopping time (the hitting time of a) (2)

τa ≡ inf{t : S(t) ≥ a} .

[Now, [τa ≤ c] = ∩q q] (over rational p and q) shows that τ a is a stopping time.] The LIL of (12.3.7) shows that both τab and τa are ﬁnite a.s. Theorem 7.1 (The reﬂection principle; Bachelier) (3)

(4) (5)

Both

P (sup0≤t≤c S(t) ≥ a) = P (τ a ≤ c) = 2 P (S(c) ≥ a) √ = 2 P (N (0, 1) ≥ a/ c ) for a > 0 and ∞ P (S 10 > a) = 4 k=1 P ((4k − 3) a < N (0, 1) < (4k − 1) a) ∞ 4 (−1)k (2k + 1)2 π 2 =1− exp − for a > 0. π 2k + 1 8 a2 k=0

Proof. Deﬁne the stopping time τa ≡ τa ∧ c, and note that τa = τa on the event [S(c) > a]. Now, [τ a ≤ c] ∈ Aτ is independent of the Brownian motion {Y(t) ≡ S(τ + t) − S(τ ) : t ≥ 0}, by strong Markov. In (b) below we will use that S(τ a ) = a on [S(c) > a]. We have P (τa ≤ c) = P (τa ≤ c) + 0 (a)

= P ([τa ≤ c] ∩ [S(c) > a]) + P ([τ a ≤ c] ∩ [S(c) < a]) + 0

(b)

= P ([τa ≤ c] ∩ [S(c) − S(τ a ) > 0]) + P ([τa ≤ c] ∩ [S(c) − S(τ a ) < 0])

(c)

= 2 P ([τa ≤ c] ∩ [S(t) − S(τ a) > 0])

(d)

= 2 P (S(t) > a),

using the strong Markov property

since the events in (c) and (d) are identical. The two-sided boundary of formula (4) follows from a more complicated reﬂection principle. Let A+ ≡ [S + > a] = [S exceeds a somewhere on [0, 1]] and A − ≡ [S − > a] = [S falls below −a somewhere on [0, 1]]. Though [S > a] = A + ∪A− , we have P (S > a) < P (A + ) + P (A− ), since we included paths that go above a and then below −a (or vice versa) twice. By making the ﬁrst reﬂection in ﬁgure 7.1, we see that the probability of the former event equals that of A+− = [S + > 3a], while that of the latter equals that of A−+ = [S − > 3a]. But subtracting out these probabilities from P (A+ ) + P (A− ) subtracts out too much, since the path may then have recrossed the other boundary; we compensate for this by adding back in the probabilities of A+−+ ≡ [S + > 5a] and A−+− ≡ [S − > 5a], which a second reﬂection shows to be equal to the appropriate probability. But we must continue this process ad inﬁnitum. Thus

7.

BARRIER CROSSING PROBABILITIES P (S 10 > a) =

(e)

315

P (A+ ) − P (A+− ) + P (A+−+ ) − · · · + P (A− ) − P (A−+ ) + P (A−+− ) − · · ·

by symmetry = 2 [P (A+ ) − P (A+− ) + P (A+−+ ) − · · ·] ∞ by (3) = 2 k=1 (−1)k+1 2 P (N (0, 1) > (2k − 1) a) ∞ = 4 k=1 P ((4k − 3) a < N (0, 1) < (4k − 1) a)

(f)

(g)

as claimed. The ﬁnal expression (5) is left for the reader; it is reputed to converge more quickly. 2 Exercise 7.1

Prove (5). (See Chung (1974, p. 223).)

The second reflection 3a The first reflection 2a a The path 0 (·) –a

Figure 7.1 The reflection principle for Brownian motion.

Theorem 7.2 (The reﬂection principle for linear boundaries; Doob) the line c t + d with c ≥ 0, d > 0. Then: (6) (7) Proof. (8)

Consider

P (S(t) ≥ ct + d for some t ≥ 0) = exp(−2 c d) . k+1 P (|S(t)| ≥ c t + d for some t ≥ 0) = 2 ∞ exp(−2 k2 c d) . k=1 (−1) Now, for any θ = 0 the process {V (t) ≡ exp([θ S(t) − θ 2 t/2]) : t ≥ 0}

is a mg (with V (0) ≡ 1).

This holds with σt ≡ σ[S(s) : s ≤ t] (using the mgf of a normal rv), since E{V (t)|σs } = exp([θS(s) − θ 2 s/2] + θ [S(s, t] − θ(t − s)/2]) (a)

= V (s) exp(−θ2 (t − s)/2) E{exp(θ N (0, t − s))}

(b)

= V (s).

316

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Thus if we now redeﬁne τab ≡ inf{t : X(t) ≡ S(t) − θt/2 ∈ (−a, b) c } where we have a > 0, b > 0, then V (t) = eθ X(t) . Hence (c)

1 = EV (τab ) = P (X(τab ) = −a) e−θa + P (X(τab ) = b) eθb ,

so that (9)

P (X(τab ) = b) = (1 − e−θa )/(eθb − e−θa )

(d)

→ e−θb

if θ > 0 and a → ∞

(e)

= e−2cd

if θ = 2c and b = d.

But this same quantity also satisﬁes (f)

P (X(τab ) = b) → P (X(t) ≥ b for some t)

as a → ∞

= P (S(t) − θt/2 ≥ b for some t) = P (S(t) ≥ θt/2 + b for some t) = P (S(t) ≥ ct + d for some t)

(g)

if c = θ/2 and d = b.

Equating (g) to (e) (via (f) and (9)) gives (6).

Exercise 7.2

2

Prove (7). (See Doob (1949).)

Theorem 7.3 (Kolmogorov–Smirnov distributions)

Both

P (U± > b) = exp(−2b2 ) for all b > 0 k+1 exp(−2 k2 b2 ) P (U > b) = 2 ∞ k=1 (−1)

and

(10) (11)

for all b > 0 .

Now, U− ∼ = U+ and

Proof.

P (U+ > b) = P (U(t) > b for some 0 < t < 1) = P ((1 − t) S(t/(1 − t)) > b)

(12)

for some 0 ≤ t ≤ 1,

= P (S(r) > b + r b for some r ≥ 0) = exp(−2 b2 )

(a)

letting r = t/(1 − t)

by theorem 7.2.

Likewise, (b) (c)

P (U > b) = P (|S(s)| > b + s b for some s ≥ 0) ∞ = 2 k=1 (−1)k+1 exp(−2 k2 b2 )

by theorem 7.2.

2

by (12.3.16)

7.

BARRIER CROSSING PROBABILITIES

317

Exercise 7.3 (a) Prove (12.6.5) for r = 2. (b) Prove (12.6.5) for integral r. [Hint. The Vθ of (8) are martingales on [0, ∞). Diﬀerentiate under the integral sign in the martingale equality (13) for all A ∈ As . A Vθ (t) dP = A Vθ (s) dP Then conclude that [∂k /∂θk Vθ (t)]θ=0 is a martingale for each k ≥ 1. For k = 4 √ this leads to S 4(t) − 6tS 2(t) + 3t2 = t2 H4 (S(t)/ t ) being a martingale on [0, ∞); here H4 (·) is the fourth Hermite polynomial (deﬁned in (14.5.15)). We need to work only with this single speciﬁc martingale in part (a); the rest of this hint is simply an intuitive explanation of how this martingale arises.]

318

8

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Embedding the Partial Sum Process

The Partial Sum Process Let Xn1 , . . . , Xnn be row-independent rvs having a common F (0, 1) distribution, and let Xn0 ≡ 0. We deﬁne the partial sum process Sn on (D, D) by 1 1 S n(t) ≡ √ Xni = √ Xni n i=0 n i=0 [nt]

(1)

k

for

k k+1 ≤t< , 0≤k≤n n n

(or for all k ≥ 0, in case the nth row is Xn1 , Xn2 , . . .). Note that (2)

[ns] [nt] Cov[S n(s), S n(t)] = i=1 j=1 Cov[Xni , Xnj ]/n = [n (s ∧ t)]/n for 0 ≤ s, t ≤ 1

for the greatest integer function [·]. We suspect that Sn “converges” to S. We will establish this shortly.

Embedding the Partial Sum Process Let {S(t) : t ≥ 0} denote a Brownian motion on (C ∞ , C∞ ). Then √ Z n(t) ≡ n S(t/n) for t ≥ 0 is also such a Brownian motion.

Notation 8.1 (3)

By using the Skorokhod embedding technique of the previous section repeatedly on the Brownian motion Z n, we may guarantee that for appropriate stopping times τn1 , . . . , τnn (with all τn0 ≡ 0) we obtain that (4)

Xnk ≡ Z n(τn,k−1 , τnk ],

for 1 ≤ k ≤ n,

are iid F (0, 1) rvs.

Let S n denote the partial sum process of these Xnk ’s. Then, for t ≥ 0 we have

[nt] τ S n(t) = √1n k=1 Xnk = √1n Z n(τn,[nt] ) = S n,[nt] n (5)

[nt] = S n1 k=1 Tnk = S(I n(t)) with Tnk ≡ (τnk − τn,k−1 ) and In (t) ≡ n1 τn,[nt] . Observe that: (6)

Xn1 , . . . , Xnn

are iid F (0, 1), in each row.

(7)

Tn1 , . . . , Tnn

(8)

r ETnk ≤ Kr · E|Xnk |2r ,

are iid with means = 1 = Var[X], in each row. with

Kr ≡ r Γ(r) 22r .

2

Theorem 8.1 (Skorokhod’s embedding theorem) The partial sum process S n on (D, D) of row-independent F (0, 1) rvs formed as above satisﬁes (9)

S n − S → p 0

as n → ∞.

8.

EMBEDDING THE PARTIAL SUM PROCESS

319

Notice: The joint distributions of any Sm, S n in theorem 8.1 are not the same as they would be if formed from a single sequence of iid rvs. In fact, we have no idea of what these joint distributions may be. However, the partial sums of an iid sequence do not generally converge to their limit in the sense of →p , so we have gained a great deal via the embedding. Theorem 8.2 (Embedding at a rate) Suppose that EX 4 < ∞. Let I denote the identity function. Then for each 0 ≤ ν < 14 , the process S n of (5) satisﬁes (10)

nν (S n − S)/I 1/2−ν 11/n = Op (1).

Proof. Consider theorem 8.1. Let I denote the identity function. Suppose we now show that (a)

In − I10 = sup0≤t≤1 |τn,[nt] /n − t| →p 0.

Then on any subsequence n where →p 0 in (a) may be replaced by →a.s. 0, the continuity of the paths of S will yield (b)

S n (·) − S(·) = S(I n ) − S → a.s. 0,

and thus (9) will follow. This is a useful argument; learn it. It therefore remains to prove (a). The WLLN gives (c)

In (t) = τn,[nt] /n →p t

for any ﬁxed t.

Using the diagonalization technique, we can extract from any subsequence a further subsequence n on which (d)

In (t) →a.s. t

for all rational t.

But since all functions involved are monotone, and since the limit function is continuous, this implies that a.s. (e)

In (t) → t uniformly on [0, 1] .

Thus (a) follows from (e), since every n has a further n with the same limit. Thus the conclusion (9) holds. In the proof just given, the conclusion (9) can trivially be replaced by (f)

sup0≤t≤m |S n(t) − S(t)| → p 0.

Appealing to exercise 12.1.6(b) for the deﬁnition of · ∞ , we thus obtain (11)

ρ∞ (S n, S) → p 0

on (C∞ , C∞ ) ,

provided that the rvs Xn1 , Xn2 , . . . are appropriately iid (0, σ2 ). [We consider the proof of theorem 8.2 at the end of this section.] 2 Let g : (D, D) → (R, B) and let ∆g denote the set of all x ∈ D for which g is not · -continuous at x. If there exists a set ∆ ∈ D having ∆g ⊂ ∆ and P (S ∈ ∆) = 0, then we say that g is a.s. · -continuous with respect to the process S.

320

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Theorem 8.3 (Donsker) Let g : (D, D) → (R, B) denote an a.s. · -continuous mapping that is D-measurable. Then g(S n) : (Ω, A, P ) → (R, B), and both (12)

g(S n) →p g(S)

as n → ∞

for the S n of (5)

(13)

g(S n) →d g(S)

as n → ∞

for any S n having the same distribution.

and

(D-measurability is typically trivial, and hypothesizing it avoids the measurability diﬃculties discussed in section 12.1.) [Theorem 8.2 allows (13) for D-measurable functionals g that are continuous in other · /q-metrics.] Proof. Now, S n −S is a D-measurable rv, and S n −S → p 0 for the S n of (5). Thus any subsequence n has a further subsequence n for which S n − S → 0 for all ω ∈ A , where P (A ) = 0. Moreover, (a)

P (A ∪ [S ∈ ∆]) ≤ P (A ) + P (S ∈ ∆) = 0 ,

and if ω ∈ A ∪[S ∈ ∆], then g(S n(ω)) → g(S(ω)) holds, since S n(ω)−S(ω) → 0 and g is -continuous at S(ω). Thus g(S n) →p g(S) as n → ∞ for the S n of (5). Thus g(S n) →d g(S) for the S n of (5), and hence of (13) also. [Note that we are dealing only with functionals for which the compositions g(Sn ) and g(S) are (Ω, A, P ) → (R, B) measurable.] 2

Example 8.1 (14)

Since the functionals · and ·+ are a.s. · -continuous,

+ S + n →d S

and

S n →d S .

The limiting distributions are those given in theorem 7.1.

2

Exercise 8.1 Let X0 ≡ 0 and X1 , X2 , . . . be iid (0, σ2 ). Deﬁne Sk ≡ X1 +· · ·+Xk for each integer k ≥ 0. (a) Find the asymptotic distribution of (S1 + · · · + Sn )/cn for an appropriate cn . (b) Determine a representation for the asymptotic distribution of the “absolute area” under the partial sum process, as given by (|S1 | + · · · + |Sn |)/cn .

The LIL Recall the (10.7.1) LIL for a single sequence of iid F (0, 1) rvs X1 , X2 , . . . with partial sums Sn ≡ X1 + · · · + Xn ; that is (15) limn→∞ |Sn |/ 2 n log log n = 1 a.s. The two LILs for Brownian motion (recall (12.3.7) and (12.3.18)) are (16) lim t→∞ |S(t)|/ 2 t log log t = 1 a.s., (17)

lim t→0 |S(t)|/

2 t log log(1/t) = 1 a.s.

8.

EMBEDDING THE PARTIAL SUM PROCESS

Notation 8.2 which the rvs (18)

321

Deﬁne stopping times T1 , T2 , . . . (with T0 = 0) having mean 1 for

Xk ≡ S(τ k−1, τk ] are iid as F .

Let τk ≡ T0 + T1 + · · · + Tk for k ≥ 0, and deﬁne the partial sums n (19) Sn ≡ k=1 Xk = S(τ n) = S(n) + [S(τ n) − S(n)]. [Note that this embedding diﬀers from that in notation 8.1. This one is based on a single sequence of rvs X1 , X2 , . . . .] 2 Exercise 8.2 (The LIL) (a) First prove (15), while assuming that (16) is true. [Hint. By proposition 10.7.1, we want to show (roughly) that √ |S(τ n) − S(n)|/ 2 n log log n →a.s. 0 or that √ (20) |S(τ [t]) − S(t)|/ 2 t log log t →a.s. 0. We will now make rigorous this approach to the problem. First apply the SLLN to τ [t]/t as t → ∞. Then deﬁne ∆k ≡ sup{|S(t) − S(t k)| : tk ≤ t ≤ tk+1 }, with tk ≡ (1 + a)k for some suitably tiny a > 0. Use a reﬂection principle and Mills’ ratio to show that P (∆k ≥ (an appropriate ck )) < ∞. Complete the proof using Borel–Cantelli.] (b) Now that you know how to deal with the “blocks” ∆k , model a proof of (16) on the proof of proposition 10.7.1.

Proof for Embedding at a Rate Proof. Consider theorem 8.2. Let d2 ≡ Var[T ]. Let Log k ≡ 1 ∨ (log k). Let M ≡ M be speciﬁed below, and deﬁne √ k (a) Acn ≡ [max1≤k≤n | i=1 (Tni − 1)|/(d k Log k) ≥ 2M/d]. Then the monotone inequality gives √ k Acn ⊂ [max1≤k≤n | i=1 {(Tni − 1)/(d i Log i)}| ≥ M/d] (b) ≡ [max1≤k≤n | ki=1 Yni | ≥ M/d], where the Yni ’s are independent with mean 0 and variance (i Log2 i)−1 . Thus the Kolmogorov inequality gives n n P (Acn ) ≤ (d/M )2 Var[ 1 Yni ] = (d/M )2 1 (i Log2 i)−1 ∞ (21) ≤ (d/M )2 1 (i Log2 i)−1 ≡ (d/M )2 v 2 < 2 if M > dv/

(c)

< .

Thus (d)

. P (Bn ) ≡ P

max

1≤k≤n

nν | S(

k

1 Tni /n) − (k/n)1/2−ν

S(k/n)|

2M ≥√ dv

/

322

(e)

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES ≤ P (Bn ∩ An ) + P (Acn ) k n | S( 1 Tni /n)−S(k/n)| √ ≤ k=1 P [2M k ( Log k)/n]1/2 ≥ n

2M k1/2−ν 1 √ √ √ n dv [2M k ( Log k)/n]1/2

∩ An +

√ | S(r + k/n) − S(k/n)|/ a ≥ b) +

(f)

≤

(g) (h)

and with ≥ b as on the right in (e) √ n using (k) below ≤ 3 k=1 P (sup 0≤r≤a | S(t, t + r]|/ a ≥ b/3) +

n ≤ 12 k=1 P (N (0, 1) ≥ b/3) +

by the reﬂection principle

(i)

≤ 12

(22)

k=1 P (sup 0≤|r|≤a

√ with a ≡ 2M k (Log k)/n (as in An in (a)),

≤ 12

n k=1 n k=1

(j)

exp(−(b/3)2 /2) +

by Mills’ ratio

M k 1/2−2ν exp − +

9dv Log k

< 2 ,

if M ≡ M is large enough and if 0 ≤ ν < 14 (this ﬁnal step holds, since we have ∞ δ 0 exp(−c x ) dx → 0 as c → ∞). The inequality (g) used sup 0≤|r|≤a |S(r + k/n) − S(k/n)| (k)

≤ sup 0≤r≤a | S(r+k/n)−S(k/n)|+2 sup 0≤r≤a | S(r+k/n−a)−S(k/n−a)|

[with t in (g) equal to k/n or k/n − a, and with a as above (see (f))]. Now, P (Bn ) ≤ 2 shows that (9) is true, provided that the sup over all of [1/n, 1] is replaced by the max over the points k/n with 1 ≤ k ≤ n. We now “ﬁll in the gaps”. Thus (even a crude argument works here) √ P ( n max1≤k≤n−1 sup 0≤t≤1/n | S(t + k/n) − S(k/n)| / k 1/2−ν ≥ M ) √ n−1 1/n ≤ k=1 P ( S 0 ≥ M k 1/2−ν / n ) n−1 ≤ 4 k=1 P (N (0, 1) ≥ M k1/2−ν ) by the reﬂection principle n−1 ≤ 4 k=1 exp(−M 2 k 1−2ν /2) (23) by Mills’ ratio (l)

< ,

if M ≡ M is large enough and if 0 ≤ ν < Exercise 8.3 (24)

1 2

(even).

2

Suppose EX 4 < ∞. Show that the process S n of (5) satisﬁes

(n1/4 / log n) S n − S = O p (1) .

[Hint. Replace nν /(k/n)1/2−ν by n1/4 / log n in the deﬁnition of Bn in (d). Now determine the new form of the bounds in (20) and (21).] [While interesting and often quoted in the literature, this formulation has little value for us.]

9.

OTHER PROPERTIES OF BROWNIAN MOTION

9

Other Properties of Brownian Motion

323

Here we collect some selected sample path properties of Brownian motion, just to illustrate a sample of what is known. Some proofs are outlined in the exercises. Deﬁnition 9.1 (Variation)

For a sequence of partitions

Pn ≡ {(tn,k−1 , tnk ] : k = 1, . . . , n} of [0, 1] (with 0 ≡ tn0 < · · · < tnn ≡ 1), deﬁne the rth variation of S corresponding to P n by n (1) Vn (r) ≡ k=1 |S(t nk) − S(t n,k−1)|r . We call these partitions nested if Pn ⊂ Pn+1 for all n ≥ 1. We further deﬁne the mesh of the partitions to be Pn ≡ sup1≤k≤n |tnk − tn,k−1 |. Theorem 9.1 (Nondiﬀerentiability) (a) Almost every Brownian path is nowhere diﬀerentiable. (b) In fact, Vn (1) → ∞ a.s. if Pn → 0. (c) (Finite squared variation) Vn (2) →L2 1 if Pn → 0. (d) (Finite squared variation) Vn (2) →a.s. 1 ∞ if either (i) n=1 Pn < ∞ or (ii) The Pn are nested with mesh approaching 0. (e) (Dudley) Vn (2) →a.s. 1 if and only if (log n) Pn → 0. Theorem 9.2 (L´evy) (2)

The H¨ older condition is true:

|S(t) − S(s)| = 1 a.s. lim sup 2 a log(1/a) {0≤s
Theorem 9.3 (The zeros of S in [0, 1])

Deﬁne

Zeros(ω) ≡ {t ∈ [0, 1] : S(t, ω) = 0}. For almost all ω, the set Zeros(ω) is a closed and perfect set of Lebesgue measure zero. [A set A is called dense in itself if every point x of A is such that every neighborhood of x contains another point of A beyond x. A compact set that is dense in itself is called perfect.] √ Theorem 9.4 (Strassen) Let Z n(t) ≡ S(nt)/ 2n log log n for 0 ≤ t ≤ 1. Let K be the set of absolutely continuous functions f on [0, 1] with f (0) = 0 and 1 2 [f (t)] dt ≤ 1; equivalently, 0 t 1 K ≡ f ∈ C[0, 1] : f (0) = 0, f (t) = f (s) ds, [f (s)]2 ds ≤ 1 . 0

0

For almost all ω the sequence {Z n(·, ω) : n = 3, 4, . . .} visualized within C[0, 1] is relatively compact with limit set K. That is, (3)

P ( limn Z n − K = 0) = 1

and

P ( ∩f ∈K [ limn Z n − f = 0]) = 1.

We will write this conclusion symbolically as Z → K. [This can be used to establish a LIL for various functionals g of S n, by determining the extreme values of g(K).]

324

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Exercise 9.1

Prove theorem 9.1(a).

(a)

(α) Let Z ∼ = N(0, 1). Let r > 0. Show that: √ Cr ≡ E|Z|r = 2r/2 Γ( r+1 2 )/ π.

(b)

|S(t n,k−1, tnk ] |r ∼ = ( Cr |tnk − tn,k−1 |r/2 , (C2r − Cr2 ) (tnk − tn,k−1 )r ) .

Exercise 9.2

(β) Now show that E Vn (2) = 1 and Var[Vn (2)] ≤ (C2r − Cr2 ) Pn , giving ∞ 2 −2 ∞ (c) 1 P ( |Vn (2) − 1| ≥ ) ≤ (C2r − Cr )

1 Pn < ∞. (γ) Finally, demonstrate the truth of theorem 9.1(d), case (i). Exercise 9.3 Prove theorem 9.1(b) when all tnk = k/2n . [Hint. Let 0 < λ < 1. The Paley–Zygmund inequality gives P (Vn (1) > λE Vn (1)) ≥ (1 − λ)2 E2 Vn (1)/E(Vn2 (1)) → (1 − λ)2 , where E Vn (1) → ∞.]

10.

VARIOUS EMPIRICAL PROCESS

325

10

Various Empirical Process

Suppose that ξn1 , . . . , ξnn are iid Uniform(0, 1). Their empirical df Gn is deﬁned by 1 1[0,t] (ξnk ) n n

(1)

Gn (t) ≡

for 0 ≤ t ≤ 1

k=1

for ξn:k ≤ t < ξn:k+1

= k/n

(2)

and 0 ≤ k ≤ n ,

where 0 ≡ ξn:0 ≤ ξn:1 ≤ · · · ≤ ξn:n ≤ ξn:n+1 ≡ 1 are the order statistics; see ﬁgure 10.1. Note that nGn (t) ∼ = Binomial(n, t) ∼ = (t, t(1 − t)). The Glivenko–Cantelli theorem shows that Gn converges uniformly to the true df I; that is, (3)

Gn − I →a.s. 0

(even for the present triangular array of ξnk ’s).

(The Cantelli proof of the SLLN based on fourth moments shows that Gn (t) →a.s. t for each ﬁxed t; even for triangular arrays. The rest of the proof is identical.) The uniform empirical process Un is deﬁned by (4)

Un (t) ≡

√

1 n [Gn (t) − t] = √ [1[ξnk ≤t] − t] n n

for 0 ≤ t ≤ 1.

k=1

This process is also pictured in ﬁgure 10.1. The means and covariances of Un are the same as those of Brownian bridge U, in that (5)

EUn (t) = 0

and

Cov[Un (s), Un (t)] = s∧t−st

for all 0 ≤ s, t ≤ 1;

this follows easily from (6)

Cov[1[0,s] (ξnk ), 1[0,t] (ξnk )] = s ∧ t − st

for 0 ≤ s, t ≤ 1.

{Moreover, for any dk ’s and ek ’s we have immediately from this that n n n (7) Cov[ 1 dk 1[0,s] (ξnk ), 1 ek 1[0,t] (ξnk ))] = ( 1 dk ek ) × [s ∧ t − st] ; we would have n1 E[dk , ek ] instead, if these were rvs independent of the ξnk ’s.} We note that G−1 n (t) ≡ inf{x ∈ [0, 1] : Gn (x) ≥ t} is left continuous, with (8)

G−1 n (t) = ξn:k for (k − 1)/n < t ≤ k/n

and G−1 n (0) = 0, as in ﬁgure 10.1. The uniform quantile process Vn is deﬁned by √ (9) Vn (t) ≡ n [G−1 for 0 ≤ t ≤ 1. n (t) − t] The key identities relating Un and Vn are (with I the identity function) the trivial √ (10) Un = −Vn (Gn ) + n [G−1 on [0, 1] , n ◦ Gn − I] √ −1 (11) Vn = −Un (G−1 on [0, 1] . n ) + n [Gn ◦ Gn − I] Note that (12)

Gn ◦ G−1 n − I = 1/n

and

G−1 n ◦ Gn − I = [max1≤k≤n+1 δnk ] ;

here δnk ≡ (ξn:k − ξn:k−1 ), for 1 ≤ k ≤ n + 1, denotes the kth of the n + 1 uniform spacings.

326

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

¨ n and G ¨ −1 deﬁned by It is sometimes convenient to use the smoothed versions G n ¨ n (ξn:k ) = k/(n + 1) and G ¨ −1 (k/(n + 1)) = ξn:k (13) G for 0 ≤ k ≤ n + 1, n connected linearly between points. Sometimes the smoothed uniform quantile pro¨ −1 (t) − t] is a useful variation on Vn . The Glivenko–Cantelli ¨ n (t) ≡ √n [G cess V n theorem implies that ¨ n −I →a.s. 0, G ¨ −1 −I →a.s. 0 ; Gn −I →a.s. 0, G−1 −I →a.s. 0, G (14) n

n

see ﬁgure 10.1. Coupling these with the identities (10) and (11) shows that (15)

Un − U →p 0

Vn − V →p 0 .

if and only if

Let cn ≡ (cn1 , . . . , cnn ) denote a vector of known constants normalized so that n n 2 cn· ≡ n1 k=1 cnk = 0 and σc,n ≡ n1 k=1 (cnk − cn· )2 = 1 , and let (16) n c4n· ≡ n1 k=1 [cnk − cn· ]4 . We suppose that these constants also satisfy the uan condition √ √ (17) max |cnk − c¯n |/[ n σc,n ] = max |cnk |/ n → 0 1≤k≤n

1≤k≤n

as n → ∞.

The weighted uniform empirical process is deﬁned by n 1 (18) Wn (t) ≡ √ cnk [1[ξnk ≤t] − t] for 0 ≤ t ≤ 1. n k=1

The Wn process is pictured in ﬁgure 10.1. It is trivial from (7) that (19)

Cov[Wn (s), Wn (t)] = s ∧ t − st

for 0 ≤ s, t ≤ 1.

It is easy to show that Wn →f d W , where W denotes another Brownian bridge, one that is independent of U. n ≡ (Rn1 , . . . Rnn ) denote the ranks of ξn1 , . . . , ξnn ; and then denote the Let R n ≡ (Dn1 , . . . , Dnn ) . Then R n is a random permutation of the antiranks by D vector (1, . . . , n) , while Dn is the inverse permutation. These satisfy (20)

ξnDnk = ξn:k

and

ξnk = ξn:Rnk .

As observed in example 8.5.3, (21)

(ξn:1 , . . . , ξn:n ) and (Rn1 , . . . , Rnn )

are independent rvs.

The empirical ﬁnite sampling process Rn is deﬁned by (22)

[(n+1)t] 1 Rn (t) ≡ √ cnDnk n

for 0 ≤ t ≤ 1.

k=1

The Rn process is also pictured in ﬁgure 10.1. The key identities are ¨ −1 ) (23) Wn = Rn (Gn ) or Rn = Wn (G on [0, 1]. n These identities give (24)

Wn − W →p 0

if and only if

Rn − R →p 0,

as with (15). Because of (21), we see that (25)

Rn

and Vn

are independent processes.

10.

VARIOUS EMPIRICAL PROCESS

1

1 n

n–1 n

–1 n

ξn:n

…

… 1 n 0

327

ξn:1

…

0 ξn:1

ξn:n 1

0

…

1 n

n–1 1 n

n

n

n

CnDnn

ξn:1 CnDn1

ξn:n 1 CnDn2

Figure 10.1

n,

–1 n ,

1 n+1

CnDnn

CnDn1

n 1 n+1 CnDn2

n,

n

and

n.

328

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES We reiterate that

(26)

W

and V = −U

are independent Brownian bridges; n n this is further corroborated, since (7) with 1 dk ek = 1 cnk /n = 0 imply that the cross covariance (27)

Cov[Un (s), Wn (t)] = 0

for all 0 ≤ s, t ≤ 1 .

We will prove only part of theorems 10.1 and 10.3 (namely, that (28) holds). For the believability of the rest, we will rely on (28), our earlier proof that S n can be embedded at a rate, and the proof of theorem 10.2. (Shorack(1991) contains these proofs, written in the current style and notation.) See section 12.11 for proofs of theorems 10.2 and 10.4. Theorem 10.1 (Convergence of the uniform processes) We can deﬁne independent Brownian bridges U = −V and W and row-independent Uniform(0, 1) rvs ξn1 , . . . , ξnn on a common probability space (Ω, A, P ) in such a way that (28)

Un − U →p 0

and

Vn − V →p 0 ,

(29)

Wn − W →p 0

and

Rn − R →p 0 ,

provided that the cnk ’s are uan with c¯n = 0, σc2 = 1, and c4n < ∞. Theorem 10.2 (Pyke–Shorack) Let q > 0 on (0, 1) be on [0, 12 ], on [ 12 , 1], 1 and have 0 [q(t)]−2 dt < ∞. Then: (30)

(Un − U)/q →p 0

and

(Vn − V)/q →p 0 .

(31)

(Wn − W )/q →p 0

and

(Rn − W )/q →p 0 .

Corollary 1 (Cs¨ org˝ o–R´ev´esz) We may replace 1/q in the previous theorem by K, for any qf K having Var[K(ξ)] < ∞. Theorem 10.3 (Weighted approximation of the uniform processes) The embeddings of the previous theorem are such that for any 0 ≤ ν < 14 we have (a) (M. Cs¨ org˝ o, S. Cs¨ org˝ o, Horv´ ath, Mason) 1−1/n

(32)

∆νn ≡ nν (Un − U)/[I ∧ (1 − I)]1/2−ν 1/n

(33)

¯ νn ≡ nν (Vn − V)/[I ∧ (1 − I)]1/2−ν 1−1/2n = Op (1) . ∆ 1/2n

= Op (1) ,

(b) (Shorack) Suppose lim c4n < ∞. Then (34)

˙ νn ≡ nν (Wn − W )/[I ∧ (1 − I)]1/2−ν 1−1/n = Op (1) , ∆ 1/n

(35)

¨ νn ≡ nν (Rn − W )/[I ∧ (1 − I)]1/2−ν 1−1/2n = Op (1) . ∆ 1/2n

[The supremum limits in (32) and (34) may be changed to c/n and 1 − c/n for any constant c > 0. This relates to exercise 10.3 below.]

10.

VARIOUS EMPIRICAL PROCESS

329

Theorem 10.4 (Weighted approximation of Gn ; Mason) Gn , any n ≥ 1, any 0 < ν < 12 , and all λ > 0 we have = = = nν (Gn − I) = 0 = = Op (1) . = (36) ∆νn ≡ = [I ∧ (1 − I)]1−ν =

For any realization of

¨ −1 in (36). We may replace Gn by G n Example 10.1 (R-statistics) Consider the simple linear rank statistics n n 1 1 Rnk i (37) cnk K =√ K cnDnk Tn ≡ √ n+1 n+1 n n k=1

(38)

=

1 0

K dRn = −

k=1

1 0

Rn dK ,

where the last step holds if K = K1 − K2 with each Ki and left continuous on (0, 1). As in (12.3.19), this suggests that 1 (39) Tn →p 0 W dK ∼ = N (0, Var[K(ξ)]), provided that the uan condition holds and provided that Var[Ki (ξ)] < ∞ for i = 1, 2. Indeed, this can be shown to be true. (Writing 1 1 Tn = − 0 W dK ⊕ (Rn − W )/q 0 [q(t)]−2 d|K| provides a simple proof in case this integral is ﬁnite for some square integrable function q and for total variation measure d|K|.) We will return to this in chapter 16 below. 2 Proof. Consider Vn . We will represent our uniforms rvs as a normed sum of Exponential(1) rvs. Thus we really begin with a Skorokhod embedding of iid Exponential(1) rvs. Let F (x) = 1 − exp(−(x + 1)) for x ≥ −1, so that F is a (0, 1) df; and if X ∼ = F, then X + 1 ∼ = Exponential(1). According to Skorokhod’s embedding theorem, there exist row-independent rvs Xn1 , . . . , Xnn with df F such that the partial sum process S n of the nth row satisﬁes S n − S → p 0 for some Brownian motion S. We now deﬁne (40)

ηnk ≡ k + Xn1 + · · · + Xnk

and ξn:k ≡ ηnk /ηn,n+1

for 1 ≤ k ≤ n + 1.

It is an elementary exercise below to show that these ξnk ’s are distributed as n row-independent Uniform(0, 1) order statistics. Let Gn denote their empirical df and Un their uniform empirical process. The key identity relating Vn to S n is √ k ηk k (a) Vn−1 = n−1 − n−1 ηn n−1 √ n n − 1 ηk − k k ηn − n k k √ √ = − − n−1 − ηn n n n−1 n n n

330

(41)

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES n = ηn

-

1 k n−1 [S n(k/n) − (k/n) S n(1) ] − √ , n n−1 n

so that for 0 ≤ t ≤ 1, (42)

n Vn−1 (t) = ηn

-

n−1 1 [ S n(In (t)) − In (t) S n(1) ] − √ In (t), n n−1

where In (t) ≡ k/n for (k − 1)/(n − 1) < t ≤ k/(n − 1) and 1 ≤ k ≤ n − 1 with In (0) ≡ 0 satisﬁes In − I → 0. Note that ηn /n →p 1 by the WLLN and that Gn − I →p 0. Thus (43)

Vn − V →p 0

for

V ≡ S − I S(1)

follows from the identity (42), In − I → 0, and the fact that (b)

S(I n) − S ≤ S(I n) − S(I n) + S(I n) − S ≤ S n − S + S(I) − S →

p

0,

by continuity of all sample paths of the S process. All sample paths of√V are continuous, and the maximum jump size of |Vn − V| is bounded above by [ n max1≤i≤n+1 δni ]; so Vn − V →p 0 and (12) imply √ √ (44) nG−1 as n → ∞. n ◦ Gn − I = [ n max1≤k≤n+1 δni ] →p 0 Thus

√

(c)

n [Gn ◦ G−1 n − I ] + V √ ≤ Vn (Gn ) − V(Gn ) + V(Gn ) − V + n Gn ◦ G−1 n − I

(d)

≤ Vn − V + V(Gn ) − V + op (1)

Un − U = − Vn (Gn ) +

= V(Gn ) − V + op (1) (e)

= op (1),

using Gn − I|| →p 0 and uniform continuity of the sample paths of V. We will prove Mason’s theorem in the next section. 2 Exercise 10.1

Establish the claim made just below (40).

Example 10.2 (The supremum functionals) · -continuous. Then (45)

g(Un ) →p g(U)

and

Suppose g : (D, D) → (R, B) is a.s.

g(Vn ) →p g(V)

for the special constructions of theorem 10.1. Moreover, convergence in distribution holds for any versions of these processes. Letting # denote +, −, or | · |, we can thus claim the convergence in distribution 1 2 1 # U# (46) and U (t) dt →d 0 U2 (t) dt n →d U 0 n for any versions of these processes. These limiting distributions were given in section 12.7. 2

10.

VARIOUS EMPIRICAL PROCESS

331

Exercise 10.2 (The two-sample uniform process) (i) Let Gm and Hn be the empirical dfs of two independent Uniform(0, 1) special constructions. Let √ √ Um = m(Gm − I) and Vn ≡ n(Hn − I) denote the corresponding empirical process, and let λmn ≡ n/(m + n). Then mn (Gm − Hn ) = ( λmn Um − 1 − λmn Vn ) Wmn ≡ m+n has (47)

Wmn − W0mn →p 0 as m ∧ n → ∞, where √ √ W0mn ≡ ( λmn U − 1 − λmn V) is a Brownian bridge.

# We thus have W# mn →d W , for Brownian bridge W . Write out the details. (ii) Now use a discrete reﬂection principle to compute the exact distribution of P (W+ nn ≥ a), and pass to the limit in the resulting expression to obtain (12.7.9). (This provides an alternative to the earlier method.) [Hint. Go through the order statistics of the combined sample from smallest to largest. If it is from sample 1, step up one unit as you go to the right one unit. If it is from sample 2, step down one unit as you go to the right one unit. In this way, perform a random walk from (0, 0) to (2n, 0). What is the chance you ever cross a barrier of height a?]

Example 10.3 (The Kolmogorov–Smirnov and Cram´er–von Mises statistics) Let ξn1 , . . . , ξnn be the iid Uniform(0, 1) rvs of the special construction. Let F denote an arbitrary df. Then Xnk ≡ F −1 (ξnk ), 1 ≤ k ≤ n, are iid F . Let Fn denote the empirical √ df of Xn1 , . . . , Xnn and let En denote the empirical process deﬁned by En (x) ≡ n [Fn (x) − F (x)]. Now, En = Un (F ). Thus (28) implies En − U(F ) ≤ Un − U →p 0, where equality holds if F is continuous. Thus √ # √ # (48) nDn ≡ n (Fn − F )# = U# if F is continuous. n →d U Likewise, a change of variable allows elimination of F , and gives 1 1 (49) Wn2 ≡ n(Fn −F )2 dF = 0 U2n (t) dt →d 0 U2 (t) dt if F is continuous. √ These statistics are used to test whether or not F is really the true df, and nDn# and Wn2 all measure how far the estimate Fn of the true df diﬀers from the √ hypothesized df F . [The percentage points of the asymptotic distributions of nDn# and Wn2 , under the null hypothesis when F is really the true df, are available.] (i) (i) Consider now the two-sample problem in which the rvs Xnj ≡ F −1 (ξnj ), for i = 1, 2 and 1 ≤ j ≤ ni , of independent special constructions have empirical dfs (1) (2) Fn1 and Fn2 . Note that for independent uniform empirical processes (2) [F(1) n1 − Fn2 ] =

(50)

n1 n2 n1 +n2

(51)

=a Wn1 ,n2 (F )

n2 n1 +n2

U(1) n1 (F ) −

n1 n1 +n2

U(2) n2 (F ) ≡ Wn1 ,n2 (F )

if F is continuous,

where (52)

Wn1 ,n2 ≡

n1 n1 +n2

U(1) −

n2 n1 +n2

U(2) ∼ =U

for all n1 and n2 .

332

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

This gives the asymptotic null distribution for the various supremum and integral functionals with which we have dealt, no matter which version of these processes is considered. 2 Exercise 10.3

Show that n ξn:n →d Exponential(1).

1 Exercise 10.4 ( 0 g dUn ) Suppose Var[g(ξ)] and Var[h(ξ)] are ﬁnite. 1 1 (a) Show that for some rvs (to be labeled 0 g dU and 0 h dW ) we have 1 1 1 1 (53) g dUn →p 0 g dU and h dWn →p 0 h dW . 0 0 (b) Show also that 1 1 (54) 0 g dVn →p − 0 g dU Exercise 10.5 (Mason) (55)

and

1 0

h dRn →p

1 0

h dW .

Consider the ∆nν of (32). For some a > 0,

supn≥2 E exp(a ∆nν ) < ∞.

[Hint. This is too hard to be an “exercise”, but it is a very nice bound.]

11.

INEQUALITIES FOR VARIOUS EMPIRICAL PROCESSES

11

Inequalities for Various Empirical Processes

333

We wish to apply the Birnbaum–Marshall and H´ ajek–R´enyi inequalities to various martingales (mgs) associated with the processes of the previous section. Proposition 11.1 (Various martingales) (1)

{Un (t)/(1 − t) : 0 ≤ t < 1}

is a mg.

(2)

{Wn (t)/(1 − t) : 0 ≤ t < 1}

is a mg.

(3)

{U(t)/(1 − t) : 0 ≤ t < 1}

is a mg.

(4)

{Vn (k/(n + 1)) : (1 − k/(n + 1)) : 0 ≤ k ≤ n}

is a mg.

(5)

{Rn (k/(n + 1))/(1 − k/n) : 0 ≤ k ≤ n − 1}

is a mg.

Proof. (a)

Let At ≡ σ[1[ξ≤s] : 0 ≤ s ≤ t]. Then E{1[ξ≤t] − t|As } = 1[ξ≤s] + = 1[ξ≤s] +

(b)

=

t−s 1−s {1

1−t 1−s {1[ξ≤s]

t−s 1−s 1[ξ>s]

−t

− 1[ξ≤s]} − t

− s} ,

so that (6)

[1[ξ≤t] − t]/(1 − t),

0 ≤ t ≤ 1,

is a mg.

Noting (12.10.7), summing (6) shows that (1) and (2) hold. Let At ≡ σ[U(s) : 0 ≤ s ≤ t]. Then (9.3.8) gives (c)

E(U(t)|As ) = {[s(1 − t)]/[s(1 − s)]}U(s),

since U(t)|U(s) is normal with mean µt + Σts Σ−1 ss [U(s) − µs ]. Thus (3) holds. Let Znk ≡ Rn (k/(n + 1))/(1 − k/n), and set ∆Znk ≡ Znk − Zn,k−1 for integers 1 ≤ k ≤ n − 1. Then

n k n k−1 n−k ∆Znk = n−k − n−k R n+1 R n−k+1 n+1 (d)

=

n √1 n n−k

cnDnk +

1 n−k+1

k−1 j=1

cnDnj .

Let Ak ≡ σ[Dn1 , . . . , Dnk ]. Then E(∆Znk |Ak−1 ) = (e)

=

n √1 n n−k

= 0,

n √1 n n−k

1 n−k+1

[E(cnDnk |Ak−1 ) +

n j=k

since c¯n = 0.

cnDnj +

1 n−k+1

1 n−k+1

k−1 j=1

k−1 j=1

cnDnj

cnDnj ]

334

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

Apply the ﬁnite sampling results (9.1.8) and (9.1.9) to (d) to conclude that

2 σ2 (k−1)−1 k−1 n 2 k−1 Var[∆Znk ] = c,n (f) 1 − n−k+1 n n−k n−1 + (n−k+1)2 1 − n−1 (7)

=

2 σc,n n2 . n − 1 (n − k)(n − k + 1)

Let Ak ≡ σ[ξn:1 , . . . ξn:k ]. Then (g)

E(ξn:k |Ai ) −

k n+1

= ξn:i +

k−i n−i+1

[1 − ξn:i ] −

k n+1

=

n−k+1 n−i+1

ξn:i −

i n+1

,

since the conditional distribution of ξn:k given ξn:i is that of the (k − i)th order statistic in a sample of size n − i from Uniform(ξn:i , 1), and (9.1.32) can be applied. Thus (4) holds. 2 ¨n, Inequality 11.1 (Pyke–Shorack) Let X denote one of the processes Un , V Wn , Rn , or U. Let q > 0 on [0, θ] be and right continuous. Then for all λ > 0 we have the probability bound θ (8) P (X/qθ0 ≥ λ) ≤ (16/λ2 ) 0 [q(t)]−2 dt . Proof. Let X denote any one of Un , Wn , or U. Then X(t)/(1 − t) is a mg with mean 0 and variance ν(t) ≡ t/(1 − t). Thus the Birnbaum–Marshall inequality gives P ( X(t)/q(t) θ0 ≥ λ) = P ( [X(t)/(1 − t)]/[q(t)/(1 − t)] θ0 ≥ λ) (a)

≤ (4/λ)2

θ 0

[(1 − t)/q(t)]2 d [t/(1 − t)] = (4/λ)2

θ 0

[q(t)]2 dt .

Let X denote Rn . Then, with bk ≡ q(k/(n + 1)) and m ≡ [(n + 1)θ], P ( Rn (t)/q(t) θ0 ≥ λ) = P (max1≤k≤m | Rn (k/(n + 1)) |/bk ≥ λ)

≤ P max1≤k≤m |R n(k/(n+1))/(1−k/n)| ≥λ bk /(1−k/n) (b)

(c)

m

Var[∆Znk ] k=1 [bk /(1−k/n)]2

≤

4 λ2

≤

4 λ2

≤

4 1 λ2 n

by (7) and H´ ajek–R´enyi

m

(n−k)2 n2 1 k=1 n−1 (n−k)(n−k+1) n2 b2k

m k=1

b−2 k ≤

16 λ2

θ 0

by (7)

[q(t)]−2 dt .

(We can improve (a) and (c) by a factor of 4, as stated in the H´ ajek–R´enyi inequality, but there is no real point to this.) 2

Exercise 11.1

¨n. Verify (8) for V

11.

INEQUALITIES FOR VARIOUS EMPIRICAL PROCESSES

Inequality 11.2 (In probability linear bounds on Gn and G−1 n )

> 0 there exists λ ≡ λ so small that the event An on which

335 For all

(9)

Gn (t) ≤ t/λ on [0, 1],

Gn (t) ≥ λt on [ξn:1 , 1],

(10)

Gn (1−t) ≤ 1−λ(1−t) on [0, ξn:n ), √ |Gn (t) − t| ≤ 1/(λ n ) on [0, 1]

Gn (1−t) ≥ 1−(1−t)/λ on [0, 1] ,

(11)

has P (An ) ≥ 1 − for all n ≥ 1. Let 1n denote the indicator function of An . (These conclusions hold for any realization of Gn and G−1 n .) (Note that linear bounds on G−1 n are also established by this result.) Proof.

Now, (ξn:1 , . . . ξn:n ) has joint density n! on its domain. Thus P (Gn (t) ≤ t/λ for 0 ≤ t ≤ 1) = P (ξn:k ≥ λk/n for 1 ≤ k ≤ n) 1 tn t3 t2 = λ λ(n−1)/n · · · λ2/n λ/n n! dt1 · · · dtn = · · ·

(a)

= n!

(b)

tn n!

−

λtn−1 n!

1 =1−λ λ

≥ 1 − /3

(c)

for all λ ≤ λ ≡ /3,

and for all n. This gives the upper bound of (9). And (10.3.20) gives Daniels’ equality P (Gn /I ≤ λ) = P (Gn (t) ≤ t/λ for 0 ≤ t ≤ 1) (12)

= P (ξn:k ≥ λk/n for 1 ≤ k ≤ n) = P (G−1 n (t) ≥ λt for all 0 ≤ t ≤ 1) =1−λ

for all 0 ≤ λ ≤ 1 .

We now turn to the lower bound of (9). Now, (13)

Sk ≡ n ξn:k+1 /k,

1 ≤ k ≤ n − 1,

is a reversed mg,

as a rearrangement of E(ξn:k |ξn:k+1 ) = [k/(k + 1)] ξn:k+1 shows. So, immediately, (d)

S˜k ≡ S(n−1)−k+1 = Sn−k = n ξn:n−k+1 /(n − k)

is a mg for 1 ≤ k ≤ n − 1.

Now calculate 1 − P (Gn (t) ≥ λt everywhere on [ξn:1 , 1]) (e)

[or = P (I/Gn 1ξn:1 > 1/λ)]

= P (ξn:k+1 > (k/n)/λ for some 1 ≤ k ≤ n − 1) = P (max1≤k≤n−1 Sk > 1/λ) = P (max1≤k≤n−1 S˜k > 1/λ) = P (max1≤k≤n−1 exp(rS˜k ) > exp(r/λ))

(f)

˜

≤ inf r>0 [e−r/λ EerSn−1 ] = inf r>0 [e−r/λ Eern ξn:2 ] by Doob’s (10.10.3)

336

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES 1

(g)

e−r/λ ernt n(n − 1) t(1 − t)n−2 dt n = inf r>0 e−r/λ 0 ers n(n − 1) (s/n) (1 − s/n)n−2 ds/n n ≤ inf r>0 e−r/λ 0 sers (1 − s/n)n−2 ds n since 1 − s/n ≤ e−s/n ≤ inf r>0 e−r/λ 0 s ers e−s e2 ds ∞ ≤ e2 inf r>0 e−r/λ 0 s exp(−(1 − r)s) ds

(h)

= e2 inf r>0 e−r/λ /(1 − r)2

(i)

= (e2 /4λ2 ) exp(−1/λ)

(j)

< /3

= inf r>0

0

from the mean of an exponential density since diﬀerentiation gives r = 1 − 2λ

for λ ≡ λ small enough. Thus the lower bound in (9) holds. Then (10) follows from (9) by symmetry. Finally, (11) holds since Un = Op (1). In fact, we have Chang’s inequality 1 P (I/G−1 n ξn:1 ≤ x) = P (Gn (t) ≥ t/x on all of [ξn:1 , 1])

(14)

≥ 1 − 2 x2 e−x

for all x > 1.

2

Proof. Consider Mason’s theorem 12.10.4. Apply the Pyke–Shorack inequality with divisor q(t) ≡ (a ∨ t)1−ν to obtain

(a)

(b)

P (nν (Gn (t) − t)/t1−ν ba ≥ λ) = P ( Un /t1−ν ba ≥ λ n(1/2)−ν ) b ≤ P (Un /qb0 ≥ λn1/2−ν ) ≤ 4 0 (a ∨ t)−(2−2ν) dt/(λ2 n1−2ν ) =

4 λ2 (an)1−2ν

+

4 λ2 n1−2ν

=

4 λ2 (an)1−2ν

−

4 λ2 n1−2ν

b a

·

t−(2−2ν) dt

1 b (1−2ν)t1−2ν |a

≤ 8(1 − 2ν)−1 /[λ2 (an)1−2ν ] .

Using a = 1/n, b = 12 and the symmetry about 12 gives .= / = = nν [Gn (t) − t] =1−1/n 16 1 = = (15) P = ≥λ ≤ . = 1−ν [t ∧ (1 − t)] (1 − 2ν) λ2 1/n But [0, 1/n] is easy (and [1 − 1/n, 1] is symmetric), since on [0, 1/n] we have (c)

nν |Gn (t) − t|/t1−ν ≤ (nt)ν [1 + Gn (t)/t] ≤ 1 + Gn (t)/t ;

and thus (9) gives .= / = = nν [Gn (t) − t] =1/n 1 = (16) P = ≥λ ≤ = [t ∧ (1 − t)]1−ν = λ−1 0

for λ > 1.

11.

INEQUALITIES FOR VARIOUS EMPIRICAL PROCESSES

337

¨ −1 and V ¨ n replacing Gn and We can repeat this same proof up to (13) with G n ¨ −1 , as the Un , because of the Pyke–Shorack inequality. Then (0, 1/n] is trivial for G n values on this whole interval are deterministically related to the value at 1/n. 2

Exercise 11.2 Prove the Pyke–Shorack theorem 12.10.2. [Hint. Model your proof on (a) of the previous proof, with a = 0 and b suﬃciently small, and with theorem 12.10.1 suﬃcient on [b, 1 − b].]

338

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

12

Applications

Theorem 12.1 (Donsker) Let g : (D, D) → (R, B) denote an a.s. · -continuous mapping that is D-measurable. Then g(Un ) : (Ω, A, P ) → (R, B), and (1)

g(Un ) →p g(U)

as n → ∞

for the Un of (12.10.28),

(2)

g(Un ) →d g(U)

as n → ∞

for an arbitrary Un .

[These conclusions hold for D-measurable functionals g that are continuous in other · /q-metrics as well.] Exercise 12.1

Write out the easy details to prove this Donsker theorem.

Example 12.1 (Tests of ﬁt) (i) Call F stochastically larger than F0 whenever PF (X > x) ≥ PF0 (X > x) for all x (with strict inequality for at least one x), and write F ≥s F0 . To test the null hypothesis H0 that F = F0 is true against the alternative hypothesis Ha that F ≥s F0 it is reasonable to reject the H0 claim that ∞ √ for large values of Birnbaum’s statistic Zn ≡ −∞ n [ Fn (x) − F0 (x) ] dF0 (x). Now suppose that H0 is true, with a continuous df F0 . Then ∞ 1 1 1 (3) Zn ∼ ). = −∞ Un (F0 ) dF0 = 0 Un (t) dt →d Z ≡ 0 U(t) dt ∼ = N (0, 12 (ii) Alternatively, one could form the Cram´er–von Mises statistic ∞ √ (4) Wn ≡ −∞ { n [ Fn (x) − F0 (x) ]}2 dF0 (x) ∼ = = (5)

∞

2 −∞ Un (F0 ) dF0

1 0

→d =

U2n (t) dt

1 0

by (7.5.22)

when F0 is continuous, by (7.3.10)

U2 (t) dt

1 ∞ ∞ 1 1 k=1 φk (t) πk Zk }{ j=1 φj (t) πj Zj } dt 0{ for the orthonormal functions φk (t) ≡

√

(see below)

2 sin(πkt) on [0, 1]

and iid N(0, 1) rvs Zk

(6)

=

∞ ∞

=

∞

k=1

1 j=1 π 2 jk

1 k=1 π 2 k2

Zj Zk

1 0

φk (t)φj (t) dt

Zk2 .

This shows that Wn is asymptotically distributed as an inﬁnite weighted sum of independent χ21 rvs. This representation of the limiting distribution has been used to provide tables. If Wn ≡ Wn (F0 ) is computed but a diﬀerent df F is true, then ∞ ∞ 2 2 1 (7) n Wn = −∞ [ Fn − F0 ] dF0 →a.s. −∞ [ F − F0 ] dF0 > 0 .

12.

APPLICATIONS

339

[In statistical parlance, this shows that the Wn -test is consistent against any alternative F = F0 .] (iii) A third possibility is the Anderson–Darling statistic ∞ √ 1 2 { n [ Fn − F0 ]}2 Un (t) (8) dF0 = dt An ≡ F (1 − F ) t(1 − t) 0 0 −∞ 0 for F0 continuous 1 2 ∞ U (t) 2 1 →d dt ∼ = k(k+1) Zk . 0 t(1 − t)

(9)

2

k=1

Proof. (i) Consider Birnbaum’s Zn . Method 1: By (7.5.22) and then the change of variable theorem of (7.3.10) (with identity function H) one obtains the ﬁrst two steps of (3). Apply Donsker for the third step. Appeal to (12.3.19) for the →d to a normal rv Z. Finally, appeal to Fubini’s theorem for both 1 1 1 (a) EZ = E 0 U(t) dt = 0 E(U(t)) dt = 0 0 dt = 0 and 11 E(Z 2 ) = E{ 0 0 U(s) U(t) ds dt} 11 11 = 0 0 E{U(s) U(t)} ds dt = 0 0 [s ∧ t − s t] ds dt 1t (b) = 0 0 s(1 − t) ds dt = 1/12 . Method 2: Apply (12.10.28) for 1 1 (c) |Un (t) − U(t)| dt ≤ 0 1 dt × Un − U →p 0 0

to replace step three in the above. Thus Zn →d Z. The rest of the justiﬁcation of example 12.1 is outlined in exercise 12.2 and exercise 12.3. 2 Exercise 12.2 Consider the Cram´er–von Mises statistic Wn . (I) Verify step (5). Use (12.10.28). (II) We now seek to justify the step representing U as an inﬁnite series. To this end formally write ∞ U(t) = 1 φk (t) π1k Zk (p) for iid N(0, 1) rvs Zk and the orthonormal functions φk (·). First recall the group of trigonometric identities

(q)

sin(A + B) = sin A cos B + cos A sin B , 2 sin A cos B = sin(A + B) + sin(A − B) , cos(A + B) = cos A cos B − sin A sin B , 2 cos A sin B = sin(A + B) − sin(A + B) .

340

CHAPTER 12. BROWNIAN MOTION AND EMPIRICAL PROCESSES

1 Use these to verify that 0 φj (t)φk (t) dt equals 0 or 1 according as j = k or j = k. [Think of this formal U(t) as an odd function on [−1, 1], and thus only these φk (·) are needed.] Then note that the Fourier coeﬃcients and the Fourier series are 1 (r) U, φk = 0 U(t) φk (t) dt = π1k Zk , (s)

U=

∞

k=1 U, φk φk

=

∞

1 k=1 φk π k

Zk .

So, verify that the series in (p) converges a.s. and then everything so far for the formal U is rigorous. Then Parseval’s identity (note theorem 6.3.3) gives 1 2 ∞ ∞ 1 2 2 2 (t) 1 |U, φk | = 1 π 2 k 2 Zk . 0 U (t) dt = U = Finally, one needs to verify the step (u) below in the identity ∞ ∞ ∞ E{ j=1 φj (s) π1j Zj × k=1 φk (t) π1k Zk } = 1 π21k2 φk (s) φk (t) = (u)

2 π2

∞ 1

1 k2

sin(π k s) sin(π k t)

= s∧ t −st,

and thus the (originally formal) process U is in fact a Brownian bridge. Where did this idea come from? Verifying that 1 1 (v) on [0, 1] 0 Cov[s, t] φk (s) ds = π k φk (t) shows that Cov[s, t] ≡ Cov[U(s), U(t)] = s∧t−st has eigenvalues π1k with associated eigenfunctions φk (·) for k = 1, 2, . . . . [Recall the spectral decomposition of matrices in (9.3.2)–(9.3.4).] Exercise 12.3 Verify the results claimed for the Anderson–Darling statistic An . [Verifying →d will be a little trickier this time, since (12.10.30) will now be needed in place of (12.10.28).] The rest is roughly similar in spirit, but the details are now a geat deal more complicated. Fundamentally, one must now represent the covariance function Cov[s, t] = (s ∧ t − s t)/ s(1 − s) t(1 − t) as a convergent inﬁnite series of orthonormal functions. (Hopefully, at least the approach is now clear. Providing the details is hard work.)

Chapter 13

Characteristic Functions 1

Basic Results, with Derivation of Common Chfs

Elementary Facts Deﬁnition 1.1 (Characteristic function) Let X be an arbitrary rv, and let F denote its df. The characteristic function of X (abbreviated chf ) is deﬁned (for all t ∈ R) by ∞ φ(t) ≡ φX (t) ≡ EeitX = −∞ eitx dF (x) (1) ∞ ∞ ≡ −∞ cos(tx) dF (x) + i −∞ sin(tx) dF (x). With dF replaced by h dµ, we call this the Fourier transform of the signed measure h dµ. [We note that φX (t) exists for −∞ < t < ∞ for all rvs X, since |eitX(ω) | ≤ 1 for all t and all ω.] Proposition 1.1 (Elementary properties) (a)

φ(0) = 1 and |φ(t)| ≤ 1

(b)

φ(·) is uniformly continuous on R.

(c)

φaX+b (t) = eitb φX (at)

(d)

¯ φ(−t) = φ(t)

(e)

φ is real-valued if and only if X ∼ = −X.

(f) (g)

Let φ denote an arbitrary chf.

for all t ∈ R.

for all t ∈ R.

for all t ∈ R.

|φ(·)|2 is a chf [of the rv X s ≡ X−X , with X and X iid with chf φ]. n φn Xi (·) = i=1 φXi (·) when X1 , . . . , Xn are independent. 1

Proof. Now, (a), (c), (d), and (g) are trivial. (f) If X and X are independent with characteristic function φ, then φX−X = φX φ−X = φφ¯ = |φ|2 . (e) If X ∼ = −X, 341

342

CHAPTER 13. CHARACTERISTIC FUNCTIONS

then φX = φ¯X ; so φX is real. If φ is real, then φX = φ¯X = φ−X ; so X ∼ = −X by the uniqueness theorem below. For (b), we note that for all t |φ(t + h) − φ(t)| = | [expi(t+h)x −eitx ] dF (x)| ≤ |eitx | |eihx − 1| dF (x) ≤ |eihx − 1| dF (x) → 0 (a) as h → 0, by the DCT with dominating function 2. The converse of (g) is false. Let X1 ≡ X2 and X3 be two iid Cauchy(0, 1) rvs. We will see below that φCauchy (t) = exp(−|t|), giving φ2X1 (t) = φX1 +X2 (t) = φX1 +X3 (t) for all t. 2 Motivation 1.1 (Proving the CLT via chfs) In this chapter we present an alternative method for establishing Fn →d F . It is based on the fact (to be demonstrated below) that the complex exponential functions eit· on R, indexed by t ∈ R, form a limit determining class. Saying this another way, the chf φ determines the distribution P , or the df F (or the density f , if there is one). Thus (as is shown in the continuity theorem below) we can establish that Fn →d F by showing that φn (·) → φ(·) on R. Indeed, using just the elementary properties listed above, it is trivial to give an informal “proof” of the classical CLT. Thus, we begin by expanding the chf of one rv X as √ √ (a) φ(X−µ)/√n (t) = φX−µ (t/ n ) = Eeit(X−µ)/ n (b)

it (it)2 = E{1 + √ (X − µ) + (X − µ)2 /2 + o(t2 /n)} n n it (it)2 = 1 + √ E(X − µ) + E(X − µ)2 /2 + o(t2 /n) n n

(c)

= 1 + 0 − t2 σ2 /2n + o(t2 /n) = 1 − t2 [σ2 + o(1)]/2n.

(In section 13.4 we will make such expansions rigorous, and in section 13.5 we will estimate more carefully the size of the errors that were made.) Then the standardized sum of the iid rvs X1 , . . . , Xn is √ √ ¯ n − µ) = n (Xk − µ)/ n , (d) Zn ≡ n(X 1 and it has chf (e) (f) (g)

n φZn (t) = k=1 φ(Xk −µ)/√n (t) = [φ(X−µ)/√n (t)]n n 2 2 t2 [σ2 + o(1)] = 1− → e−t σ /2 2n = φN (0,σ2 ) (t)

as will be shown below.

Since φZn (·) → φZ (·) on R, where Z ∼ = N (0, 1), the uniqueness theorem and the continuity theorem combine to guarantee that Zn →d Z. In principle, this is a rather elementary way to prove the CLT.

1.

BASIC RESULTS, WITH DERIVATION OF COMMON CHFS

343

Think of it this way. To have all the information on the distribution of X, we must know P (X ∈ B) for all B ∈ B. We have seen that the df F also contains all this information, but it is presented in a diﬀerent format; a statistician may well regard this F format as the “tabular probability calculating format.” When a density f exists, it also contains all the information about P ; but it is again presented in a diﬀerent format, which the statistician may regard as the “distribution visualization format.” We will see that the chf presents all the information about P too. It is just one more format, which we may well come to regard as the “theorem proving format.” 2

Some Important Characteristic Functions Distribution Binomial(n, p) Poisson(λ) GeometricF(p) Normal(µ, σ2 ) Exponential(θ) Chisquare(n) Gamma(r, θ) Uniform(0, 1) Double Exp(θ) Cauchy(0,1) de la Vall´ee Poussin Triangular

Density

n k n−k ;0 ≤ k ≤ n k p (1 − p) −λ k e λ /k!; k ≥ 0 pqk ; k ≥ 0 2 2 √ e−(x−µ) /2σ / 2πσ e−x/θ /θ on R+ x(n/2)−1 e−x/2 /[2n/2 Γ(n/2)] xr−1 e−x/θ /[θr Γ(r)] on R+ 1[0,1] e−|x|/θ /2θ 1/[π(1 + x2 )] (1 − cos x)/(πx2 ) on R [1 − |t|] × 1[−1,1] (t)

Chf [1 + p(eit − 1)]n exp(λ(eit − 1)) p(1 − qeit )−1 exp(itµ − σ2 t2 /2) (1 − itθ)−1 (1 − 2it)−n/2 (1 − itθ)−r [exp(it) − 1]/it 1/(1 + θ 2 t2 ) e−|t| [1 − |t|]1[−1,1] (t) 2(1 − cos x)/(x2 ) on R

Table 1.1

Review of Some Useful Complex Analysis A function f is called analytic on a region (a connected open subset of the complex plane) if it has a derivative at each point of the region; if it does, then it necessarily has derivatives of all orders inthe region. If z0 is an at each point m n −n , then isolated singularity of f and f (z) = ∞ n=0 an (z − z0 ) + n=1 bn (z − z0 ) k ≡ (the residue of f at z0 ) = b1 . Thus if f has a pole of order m at z0 (that is, bn = 0 for n > m in the expansion above), then g(z) ≡ (z − z0 )m f (z) is such that b1 = g (m−1) (z0 )/(m − 1)! . Thus b1 = k = (residue of f at z0 ) = lim (z − z0 )f (z) for a simple pole at z0 . z→z0

We also note that a smooth arc is described via equations x = φ(t) and y = ψ(t) for a ≤ t ≤ b when φ and ψ are continuous and not simultaneously zero. A contour is a continuous chain of a ﬁnite number of smooth arcs that do not cross the same point twice. Closed means that the starting and ending points are identical. (See Ahlfors (1953, pp. 102, 123) for what follows.)

344

CHAPTER 13. CHARACTERISTIC FUNCTIONS

Lemma 1.1 (Residue theorem) If f is analytic on a region containing a closed contour C, except for a ﬁnite number of singularities z1 , . . . , zn interior to C at which f has residues k1 , . . . , kn , then (for counterclockwise integration over C) n g(z) (2) f (z) dz = 2πi kj = dz = 2πi g(z0 ) for one simple pole . C C z − z0 j=1 Lemma 1.2 Let f and g be functions analytic in a region Ω. Suppose that f (z) = g(z) for all z on a set that has an accumulation point in Ω. We then have the equality f (z) = g(z) for all z ∈ Ω.

Evaluating Various Characteristic Functions Example 1.1 (Derivation of the Cauchy(0, 1) chf) Let t > 0. Let C denote the upper semicircle centered at the origin with radius R parametrized counterclockwise; and let A (for arc) denote C without its base. Then the Cauchy chf satisﬁes 1 eitz 1 1 (a) 2πi × Residue at i = eitz dz π 1 + z2 π 1 + z2 C (b)

(c)

1 = π →

1 π

R

1 1 eitx dx + 1 + x2 π

−R

∞

−∞

A

1 eitz dz 1 + z2

1 eitx dx + 0 = φ(t) 1 + x2

as R → ∞,

since the second integral in (b) is bounded in absolute value by 1 1 1 1 dz = πR → 0 as R → ∞. π A R2 − 1 π R2 − 1 Moreover, we know from (2) that the left-hand side of (a) equals (d)

2πi(z − i)

eitz |z=i = e−t π(1 + iz)(1 − iz)

for t > 0.

Since the Cauchy is symmetric, φ(−t) = φ(t) = exp(−|t|); or integrate the contour clockwise with t < 0. 2 Example 1.2 (Derivation of the N (0, 1) chf) √ ∞ 2 φ(t) = −∞ eitx (e−x /2 / 2π ) dx.

Let X be N (0, 1). Then

Let us instead think of φ as a function of a complex variable z. That is, √ ∞ 2 φ(z) = −∞ eizx e−x /2 dx/ 2π. Let us deﬁne a second function ψ on the complex plane by ψ(z) ≡ e−z

2

/2

.

1.

BASIC RESULTS, WITH DERIVATION OF COMMON CHFS

345

Clearly, φ and ψ are analytic on the whole complex plane. Let us now consider the purely imaginary line z = iy. On this line it is clear that ψ(iy) = ey

2

/2

,

and since elementary calculations show that √ √ ∞ ∞ 2 2 2 2 φ(iy) = −∞ e−yx e−x /2 dx/ 2π = −∞ e−(x+y) /2 dx ey /2 / 2π = ey /2 , we have ψ = φ on the line z = iy. Thus lemma 1.2 implies that ψ(z) = φ(z) for all z in the plane. Thus φ(t) = ψ(t) for all real z = t. That is, φ(t) = exp(−t2 /2)

for all real t

as claimed. (A similar approach works for the gamma distribution in exercise 1.3 below.) 2 Exercise 1.1 Derive the N (0, 1) chf via the residue theorem. Then extend to N (µ, σ2 ). [Hint. Let C denote a closed rectangle of height t with base [−R, R] on the x-axis.] Exercise 1.2 (a) Derive the Poisson(λ) chf (by summing power series). (b) Derive the Geometric(p) chf. (c) Derive the Bernoulli(p), Binomial(n, p), and NegBiT(m, p) chfs. Exercise 1.3 (a) Derive the Gamma(r, θ) chf. [Hint. Note example 1.2.] (b) Determine the Exponential(θ), Chisquare(n), and Double Exponential(θ) chfs. (c) Determine the Logistic(0, 1) chf. Exercise 1.4

Show that the real part of a chf (or Re φ(·) ) is itself a chf.

346

2

CHAPTER 13. CHARACTERISTIC FUNCTIONS

Uniqueness and Inversion

For the chf to be a useful tool, there must be a 1-to-1 correspondence between dfs and chfs. The fact that this is so is called the uniqueness theorem. We give a simple proof of the uniqueness theorem at the end of this subsection. But the simple proof does not establish an inversion formula that expresses the df as a function of the chf. In order to establish an inversion formula, we will need some notation, and an inversion formula useful for other purposes will require a hypothesis on the chf that is strong enough to allow some useful simpliﬁcation. Let U denote a rv with continuous density fU (·), and let W denote a rv with a bounded and continuous density fW (·) and with chf φW (·); and suppose we are lucky enough to determine a complementary pair that (for some constant c) satisfy the relationship (1)

fU (t) = c φW (−t)

for all real t.

(Complementary pair)

We give three examples of such pairs. Let Z ∼ = N (0, 1), T ∼ = Triangular(0, 1), and let D have the de la Vall´ee Poussin density. Then examples of (1) are √ (2) U =Z and W = Z, with c = 1/ 2π, (3)

U =T

and

W = D,

with c = 1,

(4)

U =D

and

W = T,

with c = 1/2π.

(The Cauchy(0, 1) and the Double Exponential(0, 1) then lead to two additional complementary pairs.) (The beauty of this is that we can nearly eliminate the use of complex analysis.) An arbitrary rv X, having df FX (·) and chf φX (·), may not have a density. Let us recall from the convolution formula (9.2.2) that (if U has a density) a slightly perturbed version Xa of X is smoother than X, in that (5)

Xa ≡ X + aU always has a density fa (·) ; and Xa →d X as a → 0

by Slutsky’s theorem, since a U →p 0 as a → 0. Thus F (·) = lim Fa (·) at each point in the continuity set CF of F . This is the key to the approach we will follow to establish an inversion formula. Theorem 2.1 (Uniqueness theorem)

Every df on the line has a unique chf.

Theorem 2.2 (Inversion formula) If an arbitrary rv X has df FX (·) and chf φX (·), we can always write r2 (6) FX (r2 ) − FX (r1 ) = lim fa (t) dt for all r1 < r2 in CFX , a→0

r1

where the density fa (·) of the rv Xa ≡ X + aU of (5) [with U as in (1)] is given by ∞ (7) e−itv φX (v)cfW (av) dv for all t ∈ R. fa (t) = −∞

2.

UNIQUENESS AND INVERSION

347

Theorem 2.3 (Inversion formula for densities) that satisﬁes the integrability condition ∞ (8) |φX (t)| dt < ∞,

If a rv X has a chf φX (·)

−∞

then X has a uniformly continuous density fX (·) given by ∞ 1 (9) fX (x) = e−itx φX (t) dt. 2π −∞ Remark 2.1 The uniqueness theorem can be restated as follows: The set of complex exponentials G ≡ {eitx for x ∈ R : t ∈ R} is a limit determining class. This is so because knowing all values of φX (t) = EeitX allows the df F to be determined, via the inversion formula. 2 Proof.

(a)

From the convolution formula (9.2.2) we have ∞ fa (t) = −∞ a1 fU ( t−x a ) dFX (x) =c

∞

1 −∞ a

φW ( x−t a ) dFX (x)

by (1)

∞ ∞ = (c/a) −∞ −∞ ei(x−t)w/a fW (w) dw dFX (x) ∞ ∞ = (c/a) −∞ e−itw/a fW (w) −∞ ei(w/a)x dFX (x) dw (b)

by Fubini

∞ ∞ = (c/a) −∞ e−itw/a φX (w/a) fW (w) dw = c −∞ e−itv φX (v) fW (av) dv.

Since Xa →d X, at continuity points r1 < r2 of F we have (with Xa ∼ = Fa (·)) r2 (c) FX (r2 ) − FX (r1 ) = lim {Fa (r2 ) − Fa (r1 )} = lim r1 fa (t) dt. a→0

a→0

This establishes theorems 2.1 and 2.2. The particular formula given in (c) is essentially useless, but the mere fact that one can recover FX from φX via some formula is enough to establish the important property of uniqueness. We now turn to theorem 2.3, in which we have added a hypothesis that allows the previous formula to be manipulated into a simple and useful form. Suppose that (8) holds, so that applying the DCT to (b) (using a constant times |φX (·)| as a dominating function) gives [recall the hypothesis on the fW (·) of (1)] as a → 0 that ∞ (d) fa (t) → f (t) ≡ [cfW (0)] −∞ e−itv φX (v) dv, since fW is bounded and is continuous at 0. Note that uniform continuity of f follows from the bound ∞ |f (t) − f (s)| = [cfW (0)] | −∞ [e−itv − e−isv ]φX (v) dv| (e)

≤ [cfW (0)]

∞

−∞ |e

−i(t−s)v

− 1| |φX (v)| dv → 0

as t − s → 0,

348

CHAPTER 13. CHARACTERISTIC FUNCTIONS

by applying the DCT (with dominating function [cfW (0)]|φX (·)|. The uniform convergence of fa to f on any ﬁnite interval involves only an |fW (0) − fW (av)| term under the integral sign. That f really is the density of FX follows from applying this uniform convergence in (c) to obtain r (f) FX (r2 ) − FX (r1 ) = r21 f (t) dt . Specifying U = W = Z gives [cfW (0)] = 1/(2π) (as it always must), so that the conclusion (9) holds. 2 Esseen’s inequality 13.5.1 below provides an important extension of theorem 2.2 by showing that if two chfs are suﬃciently close over most of their domain, then the corresponding dfs will be uniformly close over their entire domain. Exercise 2.1 Show that setting W = Z in line (c) of the previous proof leads, for any rv X, to the alternative inversion formula ∞ −itr2 2 2 1 e − e−itr1 (10) FX (r2 ) − FX (r1 ) = lim φX (t) e−a t /2 dt a→0 2π −∞ −it at all continuity points r1 < r2 of FX (·). [This is one possible realization of (6).] Exercise 2.2 Derive the chf of the Triangular(0, 1) density via an elementary integration. Then use theorem 2.3 to derive the chf of de la Vall´ee Poussin density, while simultaneously verifying that the nonnegative and real integrable function (1 − cos x)/(πx2 ) really is a density. Following section 13.4, determine E|X| when X has the de la Vall´ee Poussin density. Exercise 2.3 (Kernel density estimator) Since the rv X having df FX (·) and chf φX (·) may not have a density, we choose instead to estimate the density fa (·) of (5) and (7) using ∞ fˆa (x) ≡ c (11) e−itx φˆX (t)fW (at) dt −∞

[where fU (·) = cφW (−·), and where we now insist on EU = 0] with the empirical chf φˆX (·) deﬁned by ∞ n 1 itXj ˆ φX (t) ≡ (12) eitx d Fn (x) = e for − ∞ < t < ∞. n j=1 −∞ (a) Verify that fˆa (·) is actually a kernel density estimator, meaning that it is expressible as n 1 ∞ x−y 1 1 x − Xj ˆ (13) fa (x) = fU d Fn (y) = fU . a −∞ a n j=1 a a [This has statistical meaning, since we are averaging densities centered at each of the observations.]

2.

UNIQUENESS AND INVERSION

349

(b) Show that fˆa (x) is always unbiased (in that it has mean fa (x)) and has a variance we can calculate; thus for all x ∈ R we have (14) (15)

Efˆa (x) = fa (x) , ∞ x−y 1 1 2 2 ˆ dF (y) − [fa (x)] . Var[fa (x)] = f n a2 −∞ U a (2)

(c) Supposing that FX (·) has a density f (·) ∈ Cb , determine the order of the mean squared error (16)

MSE{fˆa (x)} ≡ Bias{fˆa (x)} + Var[fˆa (x)] ≡ {E(fˆa (x)) − f (x)}2 + Var[fˆa (x)]

of fˆa (x), viewed as an estimator of f (x). Show that this MSE is of order n−4/5 when a is of order n−1/5 , and that this is the minimal attainable order. (d) Note that the choice U = Z (or U = T ) leads to an fˆa (·) that is the sum of n normal (or triangular) densities that are centered at the n data points and that have a scale parameter directly proportional to a. (e) Obtain an expression for lima→0 a4/5 MSE {fˆa (x)} in terms of f (x), f (x), and f (x) when a = n−4/5 (and obtain it for both of the choices U = Z and U = T ). (f) We could also motivate this idea of a kernel density estimator based on (13) alone. How much of what we have done still carries over for a general kernel? What properties should a good kernel exhibit? What can you prove in this more general setting? Exercise 2.4 Use the table of chfs above to show in what sense the sums of independent Binomial, Poisson, NegBiT, Normal, Cauchy, Chisquare and Gamma rvs have distributions that again belong to the same family. (Recall section 9.2, noting that chfs have allowed the complicated operation of convolution of dfs or densities to be replaced by the simple operation of multiplication of chfs.)

350

3

CHAPTER 13. CHARACTERISTIC FUNCTIONS

The Continuity Theorem

Theorem 3.1 (Continuity theorem for chfs; Cram´ er–L´ evy) (i) If φn → φ where φ is continuous at 0, then φ is the chf of a bona ﬁde df F and Fn →d F . (ii) Fn →d F implies φn → φ uniformly on any ﬁnite interval |t| ≤ T . Inequality 3.1 (Chf bound on the tails of a df) 1/λ (1) P (|X| ≥ λ) ≤ 7λ 0 [1 − Real φ(t)] dt Proof. (a)

For any df F we have for all λ > 0.

Now, 1/λ 1/λ ∞ λ 0 [1 − Real φ(t)] dt = λ 0 −∞ [1 − cos(tx)] dF (x) dt ∞ 1/λ = −∞ λ 0 [1 − cos(tx)] dt dF (x) 1/λ ∞ = −∞ (λ/x)[xt − sin(xt)]0 dF (x) ∞ = −∞ [1 − sin(x/λ)/(x/λ)] dF (x) ≥ [|x|/λ≥1] [1 − sin(x/λ)/(x/λ)] dF (x)

(b)

= inf [|y|≥1] [1 − sin(y)/y] P (|X| ≥ λ) = [1 − sin(1)] P (|X| ≥ 1/ )

(c)

= (.1585 . . . )P (|X| ≥ λ) ≥ P (|X| ≥ λ)/7,

as claimed. [It may be instructive to compare this to Chebyshev’s inequality.]

2

Proof. Consider theorem 3.1. (i) The uniqueness theorem for chfs shows that the collection G of complex exponential functions form a determining class, and the expectations of these are hypothesized to converge. It thus suﬃces (by the kinder and gentler Helly–Bray theorem (theorem 11.8.1(i)(a))) to show that {Fn : n ≥ 1} is tight. Now, 1/λ (a) limn→∞ P (|Xn | ≥ λ) ≤ limn→∞ 7λ 0 [1 − Real φn (t)] dt 1/λ = 7λ 0 [1 − Real φ(t)] dt by the DCT, with dominating function 2 (b)

→0

as λ → ∞ ,

so that {Fn : n ≥ 1} is tight. (ii) Now replacing Xn →d X by versions Yn →a.s. Y (and using Skorokhod’s construction) gives for |t| ≤ T that (c) |φn (t) − φ(t)| ≤ |eitYn − eitY | dP ≤ |eit(Yn −Y ) − 1| dP ≤ sup|t|≤T |eit(Yn −Y ) − 1| dP (d)

→0

by the DCT, with dominating function 2.

2

3.

THE CONTINUITY THEOREM

351

Higher Dimensions ≡ (X1 , . . . , Xk ) from If X1 , . . . , Xk are rvs on (Ω, A, P ), then the Bk -A-mapping X is Ω to Rk induces a measure PX on (Rk , Bk ). The characteristic function of X (2)

φX ( t ) ≡ E eit

X

= E ei [t1 X1 +···+tk Xk ]

for t ≡ (t1 , . . . , tk ) ∈ Rk .

Without further explanation, we state simply that the uniqueness theorem and the Cram´er–L´evy continuity theorem still hold, based on minor modiﬁcations of the previous proof. We also remark that all equivalences of →d in theorem 11.7.1 are still valid. But we now take up an extremely useful approach to showing convergence in distribution in higher dimensions. is The characteristic function of the one-dimensional linear combination λ X (3)

φλ X (t) = E ei [tλ1 X1 +···+tλk Xk ]

for t ∈ R.

Comparison of this with (2) shows that knowing φX ( t ) for all t ∈ Rk is equivalent to knowing φλ X (t) for all t ∈ R and λ ∈ Rk with |λ| = 1. This immediately yields the following useful result. Theorem 3.2 (Cram´ er–Wold device) (4)

φλ X n (t) → φλ X (t)

for all t ∈ R

If Xn ≡ (Xn1 , . . . , Xnk ) satisfy and for each λ ∈ Rk ,

n →d X. (It suﬃces to show (4) for all unit vectors λ in Rk .) [In fact, what then X n →d λ X for all such λ (no matter what method we use to we require is that λ X show it).] Theorem 3.3 The rvs kX1 , . . . , Xk are independent if and only if the joint chfs satisfy φX (t1 , . . . , tk ) = 1 φXi (ti ). Exercise 3.1

Prove theorem 3.3.

352

4

CHAPTER 13. CHARACTERISTIC FUNCTIONS

Elementary Complex and Fourier Analysis

Lemma 4.1 (Taylor expansions of log(1 + z) and eiz ) [Note that log z is a many-valued function of a complex z = reiθ ; any of (log r) + i[θ + 2πm] for m = 0, ±1, ±2, . . . will work for log z. However, when we write log z = log r + iθ, we will always suppose that −π < θ ≤ π. Moreover, we denote this unique determination by Log z; this is the principal branch.] The Taylor series expansion of Log (1 + z) gives ∞ m−1 |Log (1 + z) − k=1 z k /k| = | k=m z k /k| (1) m |z|m ≤ |z|m (1 + |z| + |z|2 + · · ·) ≤ m(1−|z|) for |z| < 1. Thus (2)

|Log (1 + z) − z| ≤ |z|2 /(2(1 − θ))

for |z| ≤ θ < 1.

From another Taylor series expansion we have for all z that (3)

|ez −

m−1 k=0

∞ |z|j ∞ j! |z|m e|z| z k /k! | = | k=m z k /k! | ≤ |z|m j=0 ≤ . j! (j + m)! m!

Lemma 4.2 (Taylor expansion of eit ) Let m ≥ 1 and 0 ≤ δ ≤ 1, and require that m + δ > 0. Then for all real t we have m δ21−δ it (it)k (4) |t|m+δ ≡ Km,δ |t|m+δ . e − ≤ k! (m + δ) · · · (2 + δ)(1 + δ)(0 + δ) k=0

Proof. The proof is by induction. For m = 0 we have both |eit −1| ≤ 2 ≤ 2 |t/2|δ for |t/2| ≥ 1, and t |t| (a) |eit − 1| ≤ | 0 ieis ds| ≤ 0 ds = |t| ≤ 2 |t/2|δ for |t/2| ≤ 1; so that (4) holds for m = 0. We now assume that (4) holds for m − 1, and we will t verify that it thus holds for m. Since the formula eit − 1 = i 0 eis ds holds and we m−1 t k k also have m 1 (it) /k! = i k=0 0 [(is) /k!] ds, then it m t is m−1 k k e − = i (it) /k! e − (is) /k! ds k=0 k=0 0 |t|

(b)

≤ Km−1,δ

(c)

= Km,δ |t|m+δ .

0

sm−1+δ ds

by the induction step

[The next inequality is immediate. See Chow and Teicher (1997).]

2

Inequality 4.1 (Moment expansion inequality) Suppose E|X|m+δ < ∞ for some m > 0 and 0 ≤ δ ≤ 1. Then m (it)k (5) EX k ≤ Km,δ |t|m+δ E|X|m+δ for all t. φ(t) − k! k=1

4.

ELEMENTARY COMPLEX AND FOURIER ANALYSIS

353

Some Alternative Tools Lemma 4.3 (The ﬁrst product lemma) satisfy the following conditions: n (a) as n → ∞. θn ≡ 1 θnk → θ (b) (c)

δn ≡ [ max1≤k≤n |θnk | ] → 0. Mn ≡ nk=1 |θnk | satisﬁes

For all n ≥ 1, let complex θn1 , . . . , θnn

δn Mn → 0.

Then (6)

n >

(1 + θnk ) → eθ

as n → ∞.

k=1

Proof. As soon as n is so large that 0 < δn ≤ 12 (and we are on the principal branch), (2) gives n n n (p) | k=1 Log (1 + θnk ) − k=1 θnk | ≤ k=1 |θnk |2 ≤ δn Mn → 0. Thus (q)

n k=1

Log (1 + θnk ) → θ

as n → ∞.

Moreover, (p) shows that n n (r) Log ( k=1 (1 + θnk )) = k=1 Log (1 + θnk ). Because of (r), (q) implies (6). [See Chung (1974).]

2

Lemma 4.4 (The second product lemma) If z1 , . . . , zn and w1 , . . . , wn denote complex numbers with modulus at most 1, then n n n > > (7) zk − wk ≤ |zk − wk | . k=1

Proof. (a) (b)

k=1

k=1

This is trivial for n = 1. We will use induction. Now, n−1 n−1 | nk=1 zk − nk=1 wk | ≤ zn n−1 k=1 zk − k=1 wk + |zn − wn | k=1 wk n−1 n−1 n−1 n−1 ≤ k=1 zk − k=1 wk +|zn − wn |· k=1 1 ≤ k=1 |zk − wk |+|zn − wn |

by the induction step. [See most newer texts.]

2

Inequality 4.2 (Moment expansions of chfs) Suppose E|X|m < ∞ for some m ≥ 0. Then its chf satisﬁes m (it)k (8) EX k |t|m → 0 as t → 0. φ(t) − k! k=0

354 Proof. (a) (b)

CHAPTER 13. CHARACTERISTIC FUNCTIONS Use the real expansions for sin and cos to obtain eitx = cos(tx) + i sin(tx) = =

m k=0

(itx)k k!

+

(itx)m m!

m−1 k=0

(itx)k k!

+

(itx)m m!

[cos(θ1 tx) + i sin(θ2 tx)]

[cos(θ1 tx) + i sin(θ2 tx) − 1].

Here, we have some θ1 , θ2 with 0 ≤ |θ1 | ∨ |θ2 | ≤ 1. Then (8) follows from (b) via (9)

limt→0 E|X m [cos(θ1 tX) − 1 + i sin(θ2 tX)]| = 0,

by the DCT with dominating function 3|X|m . [See Breiman (1968).]

Inequality 4.3 (Summary of useful facts) gives the highly useful φ(t) − [1 − σ2 t2 /2] = o(t2 )

2

Let X ∼ = (0, σ2 ). Result (8) then

as t → 0,

while applying (5) twice gives (10)

φ(t) − [1 − σ2 t2 /2] ≤ σ 2 t2 ∧ E|X|3 |t|3 /6

for all t ∈ R.

Exercise 4.1 (Distributions determined by their moments) (a) Suppose that E|X|n < ∞. Then the nth derivative φ(n) (·) is a continuous that EX n = i−n φ(n) (0). function given by φ(n) (t) = in E(X n eitX ), so ∞ (b) If all moments are ﬁnite, then φ(t) = n=0 (it)n EX n /n! within the radius of convergence. (c) If the radius of convergence in (b) is strictly positive, then the distribution having the stated moments is uniquely determined by its moments. (d) The normal distribution is uniquely determined by its moments. (e) The series in (b) converges for |t| < r if and only if E exp(t |X|) < ∞ for |t| < r. Exercise 4.2 (a) If φ (0) is ﬁnite, then σ2 is ﬁnite. Prove this. (b) In fact, if φ(2k) (0) is ﬁnite, then EX 2k < ∞. Prove this. Exercise 4.3 (Bounds on (1 − x/n)n ) 0 ≤ t < 1 at t = x/n to show that 0 ≤ e−x − (1 − nx )n ≤

x2 n

e−x

(i) Use (1 + t) ≤ et ≤ 1/(1 − t) for for 0 ≤ x < n .

(ii) (Hall and Wellner) Show that 2 e−2 ≤ n supx≥0 e−x − (1 − x/n)n 1[0,n] (x) ≤ (2 + n−1 )e−2 for all n ≥ 1.

4.

ELEMENTARY COMPLEX AND FOURIER ANALYSIS

355

Results from Fourier Analysis On some occasions we will need to know the behavior of φ(t) for |t| large. Lemma 4.5 (Riemann–Lebesgue lemma)

∞

(11) −∞

eitx g(x)dx → 0

If

∞ −∞

|g(x)|dx < ∞, then

as t → ∞.

m Proof. Now, Ψ ≡ {ψ ≡ 1 ci 1(ai ,bi ] : ai , bi , ci ∈ R and m ≥ 1} is dense in L1 ∞ by theorem 3.5.8; that is, if −∞ |g(x)|dx < ∞, then there exists ψ ∈ Ψ such that ∞ ∞ |g − ψ| dx < . Thus γ(t) ≡ | −∞ eitx g(x)dx| satisﬁes −∞ γ(t) ≤

∞

≤ +

(a)

−∞ |e

m 1

itx

∞ | |g(x) − ψ(x)| dx + | −∞ eitx ψ(x) dx|

b |ci | | aii eitx dx|.

It thus suﬃces to show that for any a, b in R we have

b

eitx dx → 0

(12)

as |t| → ∞.

a

A quick picture of sines and cosines oscillating very fast (and canceling out over the interval) shows that (12) is trivial. (Or write eitx = cos(tx) + i sin(tx) and compute the integrals.) 2

Lemma 4.6 (Tail behavior of chfs) If F has density f with respect to Lebesgue measure, then φ(t) → 0 as |t| → ∞. If F has n+1 integrable derivatives f, f , . . . , f (n) on R, then (13)

tn φ(t) → 0

as |t| → ∞.

Proof.

The ﬁrst statement is immediate. Then use φ(t) = eitx f (x) dx = f (x) d(eitx /it) ∞ = (eitx /it) f (x)−∞ − eitx f (x) dx/(it)

(a)

= − eitx f (x) dx/(it)

with

f (·) ∈ L1 .

Applying the Riemann–Lebesgue lemma to (a) gives tφ(t) → 0 as |t| → ∞. Keep on integrating by parts and applying the Riemann–Lebesgue lemma. 2

356

CHAPTER 13. CHARACTERISTIC FUNCTIONS

Other Alternative Tools Then chf always exists, so it can always be used. However, if X ≥ 0 or if X is integer valued, then Laplace transforms or probability generating functions oﬀer more elementary tools. Exercise 4.4 (Laplace transform) Let F + denote the class of all dfs F having + F− (0) = 0. For any df F ∈ F we deﬁne the Laplace transform L of F by ∞ L(λ) = E e−λX = 0 e−λx dF (x) (14) for λ ≥ 0. (a) Establish an analogue of proposition 1.1(a), (b), (c), and (g). (b) (Uniqueness) Show that each df in F+ has a unique Laplace transform. (c) (Continuity) Let Xn ∼ = Fn ∈ F + . If Ln (λ) → (some L(λ)) for all λ ≥ 0 with L(·) right continuous at 0, then L is the Laplace transform of a df F ∈ F + for which the convergence in distribution Fn →d F holds. (d) Establish analogues of inequality 4.1 on moment expansions. Exercise 4.5 (Probability generating function) Let F I denote the class of all dfs F assigning mass 1 to the integers 0, 1, 2, . . . . For any df F ∈ F I we deﬁne the probability generating function g of F by ∞ (15) g(z) = E z X = k=0 pk z k for all complex z having |z| ≤ 1. (a) Establish an analogue of proposition 1.1. (b) (Uniqueness) Show that each df F in FI has a unique generating function. (c) (Continuity) Let Xn ∼ = Fn ∈ F I . If gn (z) → (some g(z)) for all |z| ≤ 1 with g(·) continuous at 1, then g is the generating function of a df F in FI for which Fn →d F . Exercise 4.6 (Cumulant generating function) ψX (·) of a rv X is deﬁned by (16)

The cumulant generating function

ψX (t) ≡ Log φX (t) = Log E(eitX ),

and is necessarily ﬁnite for t-values in some neighborhood of the origin. Temporarily suppose that all moments of X are ﬁnite. Let µk ≡ E(X − µ)k , for k ≥ 1. Then when µ = EX = 0 and with σ2 ≡ µ2 , we have φX (t) = 1 − t2 σ2 /2 + (it)3 µ3 /3! + (it)4 µ4 /4! + · · · ≡ 1 + z . We would thus have ψX (t) = Log φX (t) = Log (1 + z) = z − z 2 /2 + z 3 /3 + · · · = (it)2 µ2 /2! + (it)3 µ3 /3! + (it)4 (µ4 − 3µ22 )/4! + · · · (17) (18)

= (it)2 σ2 /2! + (it)3 µ3 /3! + t4 (µ4 − 3σ 4 )/4! + · · · ∞ ≡ j=2 (it)j κj /j! ,

4.

ELEMENTARY COMPLEX AND FOURIER ANALYSIS

357

where κj is called the jth cumulant of X. Note that for independent rvs X1 , . . . , Xn , n n (the jth cumulant of k=1 Xk ) = k=1 (the jth cumulant of Xk ), (19) which is √ nice. In the iid case, the third and fourth cumulants of the standardized ¯ n − µ)/σ are rv Zn ≡ n(X √ √ γ1 / n ≡ (µ3 /σ 3 )/ n (20) and γ2 /n ≡ (µ4 /σ 4 − 3)/n, where γ1 measures skewness and γ2 measures tail heaviness. [This is particularly ¯ n ) of skewness disappears at nice; it√ shows that the eﬀect (on the distribution of X rate 1/ n, while the eﬀect of tail heaviness disappears at rate 1/n.] Finally, if only E|X|m < ∞ for some m ≥ 1, then in a suﬃciently small neighborhood of the origin j m m (21) |ψ(t) − m j=2 κj (it) /j! | ≤ cm |t| E|X| δm (t), where δm (t) 0 as t 0, and m−1 (22) |ψ(t) − j=2 κj (it)j /j! | ≤ c¯m |t|m E|X|m for some universal constant c¯m . The exercise is to establish carefully that all of this is true.

358

5

CHAPTER 13. CHARACTERISTIC FUNCTIONS

Esseen’s Lemma

Let G denote a ﬁxed function having G(−∞) = 0, G(+∞) = 1, having derivative bounded by some constant M , having ∞ g on the real line for which |g(·)| is ∞ xg(x) dx = 0, and then let ψ(t) ≡ eitx g(x) dx. Let F denote a general df −∞ −∞ having mean 0, and let φ denote its characteristic function. We wish to estimate F − G = sup−∞<x<∞ |F (x) − G(x)| in terms of the distance between φ and ψ. Roughly speaking, the next inequality says that if φ and ψ are suﬃciently close over most of their domain, then F − G will be small. [In the initial application of this Esseen’s lemma, we will take G, g, ψ to be the N (0, 1) df, √ density, and chf. In this context, the constant of (1) is 24g/π = 24M/π = 24/( 2ππ) = 3.047695 . . . .] Inequality 5.1 (Esseen’s lemma) Let F and G be as above. For any a > 0 we have the uniform bound 1 a φ(t) − ψ(t) 24 g (1) F − G ≤ dt + . π −a t πa Proof. The key to the technique is to smooth by convolving F and G with the df Ha whose density ha and characteristic function γa are given by 1 − cos(ax) 1 − |t|/a if |t| ≤ a, ha (x) = (2) on R and γ (t) = a 0 if |t| > a. πax2 Let Fa and Ga denote the convolutions of F and G with Ha . We will now show that (3)

F − G ≤ 2Fa − Ga + 24 g/(πa).

Let ∆ ≡ F − G. Now, ∆(x) = ∆+ (x) and ∆− (x) exist for all x; thus there exists x0 such that either D ≡ F − G = |∆(x0 )| or D = |∆− (x0 )|. Without loss of generality, we suppose that D = |∆(x0 )| (just replace X, Y by −X, −Y if not). Note ﬁgure 5.1. Without loss of generality, we act below as though ∆(x0 ) > 0 (if ∆(x0 ) < 0, then z < x0 rather than z > x0 ). Now, since F is and g is bounded by M , we have (a)

∆(z − x) ≥ D/2 + M x

for |x| ≤ ,

where ≡ D/2M and z ≡ x0 + . Trivially (since D was the supremum), (b)

∆(z − x) ≥ −D

for |x| > .

Thus, with ∆a ≡ Fa − Ga , using (a) and (b) gives ∞ (c) Fa −Ga ≥ ∆a (z) = −∞ ∆(z−x)ha (x) dx (d)

≥

−

by the convolution formula

[D/2 + M x] ha (x) dx − D [|x|>] ha (x) dx

= (D/2)[1 −

[|x|>] ha (x) dx]

since xha (x) is odd

+M ·0−D

[|x|>] ha (x) dx

5.

ESSEEN’S LEMMA

359

= (D/2) − (3D/2) (e)

[|x|>] ha (x) dx

≥ (D/2) − (12M/πa)

= F − G/2 − (12M/πa),

(which is (3)), since ∞ 2 (f) [|x|>] ha (x) dx ≤ 2 (2/πax ) dx = 4/(πa ) = 8M/(πaD). F y = D – Mx

D

G D/2

ε x0

ε z

x

Figure 5.1 Bounds for Esseen’s lemma.

We now bound Fa − Ga . By the Fourier inversion formula, Fa and Ga have bounded continuous “densities” that satisfy a (4) fa (x) − ga (x) = −a e−itx [φ(t) − ψ(t)] γa (t) dt/(2π). From this we suspect that a 1 φ(t) − ψ(t) ∆a (x) = γa (t) dt . (5) e−itx 2π −a −it That the integrand is a continuous function that equals 0 at t = 0 (since F and G have 0 “means”, inequality 13.4.1 gives this) makes the right-hand side well-deﬁned, and we may diﬀerentiate under the integral sign by the DCT [with dominating function γa (·)] to get the previous equation (4). Thus ∆a (x) can diﬀer from the right-hand side of (5) by at most a constant; but this constant is 0, since obviously ∆a (x) → 0 as |x| → ∞, while the right-hand side does the same by the Riemann– Lebesgue lemma. Equation (5) gives a φ(t) − ψ(t) 1 dt (6) |∆a (x)| ≤ for all x. 2π −a t Combining (3) and (6) gives (1).

2

360

CHAPTER 13. CHARACTERISTIC FUNCTIONS

Corollary 1 (Stein) Suppose that instead of convolving F and G with the Ha of (2), we convolve with an arbitrary df H instead. In this situation we obtain (7) Proof. (d)

F − G ≤ 2 F ∗ H − G ∗ H + 8 g E|H −1 (ξ)| . Picking up at line (d) of the previous proof (with Y ∼ = H), we obtain F ∗ H − G ∗ H ≥ [−,] [D/2 + M y] dH(y) − D P (|Y | > ) ≥ (D/2) [1 − P (|Y | > )] − M E|Y | − D P (|Y | > ) ≥ (D/2) − (3D/2) P (|Y | > ) − M E|Y |

(e)

≥ (D/2) − 4M E|Y |

using Markov’s inequality and ≡ D/2M in the last step.

2

6.

DISTRIBUTIONS ON GRIDS

6

Distributions on Grids

361

Deﬁnition 6.1 We say that a rv X is distributed on a grid if there exist real numbers a, d such that the probabilities pn ≡ P (X = a + nd) satisfy ∞ −∞ pn = 1. We call d the span of the grid. The maximal span is sup{|d| : |d| is a span}. Proposition 6.1 If t0 = 0, the following are equivalent: (a) |φ(t0 )| = 1. (b) |φ| has period t0 ; that is, |φ(t + nt0 )| = |φ(t)| for all n and t. (c) The rv X is distributed on a grid of span d = 2π/t0 . Proof. Suppose that (a) holds. Then φ(t0 ) = eiα for some real α. That is, it x 0 e dF (x) = eiα , or ei(t0 x−α) dF (x) = 1. Taking real parts gives ∞ (p) [1 − cos(t0 x − α)] dF (x) = 0. −∞ Since the integrand is nonnegative for all X, this means that 1 − cos(t0 x − α) = 0

(q)

a.s. F ;

that is, (r)

t0 X − α ∈ {2πm : m = 0, ±1, ±2, . . .}

a.s.

That is, X ∈ {α/t0 + (2π/t0 )m : m = 0, ±1, ±2, . . .} a.s.; so (c) holds. Suppose (c) holds. Then (b) holds, since ∞ |φ(t + nt0 )| = | m=−∞ pm ei(t+nt0 )(a+dm) | ∞ = |ei(t+nt0 )a | | m=−∞ pm ei(t+nt0 )dm | =| (s)

=|

∞ m=−∞

∞ m=−∞

pm ei(t+2πn/d) dm| pm eitdm | |ei2πnm | = |φ(t)|.

Suppose that (b) holds. Then (t)

1 = |φ(0)| = |φ(0 + t0 n)| = |φ(0 + t0 1)| = |φ(t0 )|,

so that (a) holds.

2

Corollary 2 If a = 0 in (c), then we may replace |φ| by φ in (a) and (b), and proposition 6.1 will still hold. Proposition 6.2 One of the following possibilities must hold: (d) |φ(t)| < 1 for all t = 0. (e) |φ(t)| < 1 for 0 < t < 2π/d and |φ(2π/d)| = 1. Thus, X has maximal span d. (f) |φ(t)| = 1 for all t. And so φ(t) = eiat for all t and P (X = a) = 1, for some a.

362

CHAPTER 13. CHARACTERISTIC FUNCTIONS

Proof. Clearly, either (d), (e), or (f) holds, or else |φ(tn )| = 1 for some sequence tn → 0. In this latter case, |φ(mtn )| = 1 for all m, for each n by proposition 6.1. Since {mtn : n ≥ 1, m = 0, ±1, ±2, . . .} is dense in R and since φ, and thus |φ|, is continuous, we must have case (f) again. It remains to establish the consequences of (e) and (f). Consider (e). Proposition 6.1 shows that (e) holds if and only if both d is a span and no number exceeding d is a span. In the case of (f), we have |φ(t1 )| = 1 = |φ(t2 )| for some t1 and t2 having t1 /t2 = (an irrational number). But |φ(t1 )| = 1 and |φ(t2 )| = 1 imply that both 2π/t1 and 2π/t2 are spans. Thus if at least two points have positive mass, then the distance between them must equal m1 2π/t1 for some integer m1 and it must equal m2 2π/t2 for some integer m2 . That is, 2πm1 /t1 = 2πm2 /t2 , or t1 /t2 = m1 /m2 = (a rational number). This contradiction shows that there can be at most one mass point a. 2

Exercise 6.1 (Inversion formula for distributions on Let X be distributed a grid) it(a+dn) on a grid with pn = P (X = a + dn). Then φ(t) = ∞ p e . Show that n −∞ (1)

pm =

d 2π

π/d −π/d

φ(t) e−it(a+dm) dt.

7.

CONDITIONS FOR φ TO BE A CHARACTERISTIC FUNCTION

7

Conditions for φ to Be a Characteristic Function

Example 7.1

363

Now, for a > 0,

Fa (x) = [1 − cos(ax)]/(πax2 )

for x ∈ R

is a de la Vall´ee Poussin density function with chf φa (t) = (1 − |t|/a) 1[−a,a] (t)

for t ∈ R.

Let Fa denote the df. Then F ≡

n

pi Fai

with

pi ≥ 0,

n 1

pi = 1,

and 0 < a1 < · · · < an

1

is a df with characteristic function n φ = 1 pi φai . Thus any even function φ ≥ 0 with φ(0) = 1 whose graph on [0, ∞) is a convex polygon is a chf. 2 Proposition 7.1 (P´ olya) Let φ ≥ 0 be an even function with φ(0) = 1 whose graph on [0, ∞) is convex and ↓. Then φ is a chf. Proof. Pass to the limit in the obvious picture, using the continuity theorem to complete the proof. 2 Bochner’s theorem below gives necessary and suﬃcient conditions for a function to be a chf. We merely state it, as a background fact. Its proof can be found in a number of the standard texts. Deﬁnition 7.1 A complex-valued function φ(·) on R is nonnegative deﬁnite if for any ﬁnite set T and any complex-valued function h(·) we have ¯ (1) s,t∈T φ(s − t)h(s)h(t) ≥ 0. Theorem 7.1 (Bochner) A complex-valued function φ(·) is a chf if and only if it is nonnegative deﬁnite and continuous.

Chapter 14

CLTs via Characteristic Functions 0

Introduction

The classical CLT states that if X1 , X2 , . . . are iid (µ, σ2 ), then √ ¯ n − µ) →d N (0, σ2 ) n (X as n → ∞. Chapters 11 and 14 consider the following generalizations: (i) Nonidentically distributed rvs. (ii) Triangular arrays of row-independent rvs Xn1 , . . . , Xnn for n ≥ 1. (iii) The speed of convergence of the dfs to the limiting df. (iv) Convergence of the density functions as well. (v) Random sample sizes. (vi) Certain dependent rvs. (vii) The multidimensional case. (viii) Nonnormal limits (with both the degenerate WLLN and the Poisson). In chapter 15 we will consider situations that lead to stable and inﬁnitely divisible rvs as limits. Chapters 16, 17, and 18 all contain additional extensions leading to asymptotic normality.

365

366

1

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

Basic Limit Theorems

The goal of this section is to use a chf approach to present the classical central limit theorems for sums of iid random variables in R and in Rk . We also compare and contrast the central limit theorem with the Poisson limit theorem.

The Classical CLT Theorem 1.1 (Classical CLT) For each n ≥ 1, let Xn1 , . . . , Xnn be iid F (µ, σ2 ); this denotes that the df F (·) of the Xnk ’s has mean µ and ﬁnite variance σ2 . Deﬁne ¯ n ≡ Tn /n. Then as n → ∞, the total Tn ≡ Xn1 + · · · + Xnn and the average X √ n 2 ¯ n − µ) = √1 (Tn − nµ) = √1 (1) n (X k=1 (Xnk − µ) →d N (0, σ ) . n n Proof. (a)

(b)

Now, for ﬁxed t we have [with r(t) → 0 as t → 0] √ n φ√n (X¯ n −µ) (t) = k=1 φ(Xnk −µ)/√n (t) = [φXnk −µ (t/ n )]n ! 2 2 "n σ2 t t t √ + √ r √ by inequality 13.4.2 = 1− 2 n n n n σ2 t2 t2 t = 1− . + ×r √ 2n n n

The ﬁrst product lemma (lemma 13.4.3) with θ = −σ2 t2 /2 trivially applies. Thus φ√n (X¯ n −µ) (t) → e−σ t /2 = φN (0,σ2 ) (t) , √ ¯ 2 using table 13.1.1. Thus n (X er–L´evy continuity n − µ) →d N (0, σ ) by the Cram´ theorem (theorem 13.3.1) and the uniqueness theorem (theorem 13.2.1). Had we chosen to appeal to the second product lemma (lemma 13.4.4) instead, we would have instead claimed that φ√n (X¯ −µ) (t) − (1 − σ2 t2 /2n)n n n n (d) = k=1 φ(Xnk −µ)/√n (t) − k=1 (1 − σ 2 t2 /2n) n φ(X −µ)/√n (t) − (1 − σ2 t2 /2n) (e) ≤ 2 2

(c)

k=1

(f)

≤

n

nk

2

k=1 (t

√ √ /n)r(t/ n ) = t2 r(t/ n ) → 0.

But (1 − σ2 t2 /2n)n → exp(−σ2 t2 /2) = φN (0,σ2 ) (t), so the continuity theorem and the uniqueness theorem complete the proof. 2

Degenerate Limits Exercise 1.1 (WLLN, or classical degenerate convergence theorem) Use ¯ n →p µ as n → ∞. chfs to show that if X1 , X2 , . . . are iid with mean µ, then X Equivalently, (2)

¯ n →d (the degenerate distribution with mass 1 at µ). X

1.

BASIC LIMIT THEOREMS

367

The Classical PLT Theorem 1.2 (Classical Poisson limit theorem; the PLT) For each n ≥ 1, suppose that Xn1 , . . . , Xnn are independent Bernoulli(λ ) rvs for which the values n nk n of the parameters satisfy λn ≡ k=1 λnk → λ while k=1 λ2nk → 0. (This is true if λn1 = · · · = λnn = λn /n for all n, with λn → λ). Then (3)

Tn ≡ Xn1 + · · · + Xnn →d Poisson(λ)

Proof.

From table 13.1.1 we have φXnk (t) = 1 + λnk (eit − 1). Thus n n φTn (t) = k=1 φXnk (t) = k=1 [1 + λnk (eit − 1)]

(a) (b)

→ exp(λ(eit − 1))

(c)

= φPoisson(λ) (t)

as n → ∞.

by the product lemma by table 13.1.1.

Now apply the Cram´er–L´evy continuity theorem and the uniqueness theorem. Exercise 1.2 (Poisson local limit theorem) (4)

P (Tn = k) → P ( Poisson(λ) = k)

2

Show that as n → ∞, for k = 0, 1, . . . ,

when λn1 = · · · = λnn in the PLT. Show that this implies dT V (Pn , P ) ≡ sup{|Pn (A) − P (A)| : A ∈ B} → 0, where Tn ∼ = Pn and Poisson(λ) ∼ = P . [This is improved in exercise 14.4.4 below.] √ Exercise 1.3 Show that if Tλ ∼ = Poisson(λ), then (Tλ − λ)/ λ →d N (0, 1) as the parameter λ → ∞. (5)

A Comparison of Normal and Poisson Convergence Exercise 1.4 (6)

Mn ≡

(a) Suppose the hypotheses of the classical CLT hold. Show that √ max |Xnk − µ|/ n →p 0 .

1≤k≤n

(b) Suppose the hypotheses of the classical PLT hold. Show that (7) Mn ≡ max |Xnk | →d Bernoulli(1 − e−λ ) . 1≤k≤n

(c) There is something fundamentally diﬀerent regarding the negligibility of the corresponding terms in these two cases! The CLT involves summing many tiny pieces, but the PLT arises from very occasionally having a “large” piece. Remark 1.1 Let Xn1 , . . . , Xnn be independent. Let pnk ≡ P (|Xnk | ≥ ). Recall equation (10.3.14) for n Mn ≡ max |Xnk | →p 0 if and only if (8) pnk → 0 for all > 0, 1≤k≤n

(9)

k=1

n n 1 − exp(− k=1 pnk ) ≤ P (Mn ≥ ) ≤ k=1 pnk .

2

368

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

The Multivariate CLT n ≡ (Xn1 , . . . , Xnk ) , n ≥ 1, Theorem 1.3 (Classical multivariate CLT) Let X be a sequence of iid ( µ, Σ) random vectors. Then (10)

Proof. (a)

n 1 (Xj − µ ) →d Nk (0, Σ) 1/2

n

as n → ∞.

j=1

For any λ ∈ Rk the rvs j − µ Yj ≡ λ (X ) ∼ = (0, λ Σλ)

are iid for j = 1, . . . , n.

Thus the classic CLT gives √ (b) n Y¯n →d N (0, λ Σλ). n ≡ n−1/2 n (X j − µ That is, Z ) satisﬁes 1 (c)

φλ Z n (t) = φ√n Y¯n (t) → exp(−λ Σλ t2 /2).

∼ ∼ Now, if Z = Nk (0, Σ), then λ Z = N (0, λ Σλ ); and hence (d)

φλ Z (t) = exp(−λ Σλ t2 /2).

Thus (c) and (d) give φλ Zn (t) → φλ Z (t) for all t ∈ R, for each λ ∈ Rk . Thus the n →d Z. Cram´er–Wold theorem (theorem 13.3.2) shows that Z 2 √ Exercise 1.5 (Empirical process; Doob) Let Un ≡ n [Gn − I] be the uniform empirical process of sections 7.5 and 12.10, and let U denote the Brownian bridge of section 9.4. Show that Un →f d U as n → ∞; that is, show that for any set of points 0 < t1 < · · · < tk < 1 we have (Un (t1 ), . . . , Un (tk )) →d (U(t1 ), . . . , U(tk ))

as n → ∞.

(Essentially, all results in chapter 12 derive from this example—via a suggestion of Doob(1949).) Exercise 1.6 (Partial sum process of iid rvs) Let S n denote the partial sum process of iid (0, 1) rvs (see (11) below) and let S denote Brownian motion. Show that S n →f d S as n → ∞. [Hint. Set things up cumulating from the left, and then transform. Or note that the random element you must consider can be written in a form equivalent to something simpler. Or use the Cram´er–Wold device. One of these methods is much simpler then the others.] Exercise 1.7 (Partial sum process) Suppose that Xn1 , . . . , Xnn are independent and satisfy Lindeberg’s condition. Deﬁne S n on [0, 1] by letting (11)

S n(t) =

with s2nk ≡ motion.

k

k

i=1 Xni /snn

2 i=1 σni

for

s2n,k+1 s2nk ≤ t < , s2nn s2nn

0 ≤ k ≤ n,

and s2n0 ≡ 0. Show that S n →f d S, where S denotes Brownian

1.

BASIC LIMIT THEOREMS

369

Example 1.1 (Chisquare goodness of ﬁt statistic) Suppose Ω = Now let X1 , . . . , Xn be iid on (Ω, A) with all pi ≡ P (X ∈ Ai ) > 0. Let (12)

Nni ≡

n

1Ai (Xj ) ≡ ( the number of Xj ’s that fall in Ai )

k i=1

Ai .

for 1 ≤ i ≤ k.

j=1

√ (a) Now, (Z1j , . . . , Zkj ) , with Zij ≡ (1Ai (Xj ) − pi )/ pi , has mean vector 0 and covariance matrix Σ = |[σii ]| with σii = 1 − pi and σii = − pi pi for i = i . √ n ≡ n Zj / n →d W ∼ (b) Thus W = Nk (0, Σ) as n → ∞, by theorem 1.3. 1 (c) The usual chisquare goodness of ﬁt statistic is (13)

Qn ( p) ≡

k (Nni − npi )2 i=1

npi

W =W n n →d W W

=

k (Observedi − Expectedi )2 Expectedi i=1

by the Mann–Wald theorem

)∼ ) (ΓW = (ΓW = Chisquare(k − 1) ; √ √ √ here Γ is k × k and orthogonal with ﬁrst row p , so that Γ Σ Γ = Γ[I − p p ]Γ = I − (1, 0, . . . , 0) (1, 0, . . . , 0). This has diagonal elements (0, 1, 1, . . . , 1) with all ∼ oﬀ-diagonal elements 0, and then ΓW = N (0, ΓΣΓ ) (by (8.3.5) and (9.3.6)). We also use (9.1.29) for (16). [If a value of Expected is unknown, it should be replaced ˆ by an appropriate estimator Expected.] (See exercise 14.2.13 below.) (This statistic is just a quadratic form.) 2 (14)

Exercise 1.8 (Independence in an I × J table) Suppose both Ω = J and Ω = j=1 Bj represent partitions of Ω. (a) Let pij ≡ P (Ai Bj ) = pi· p·j , where pi· ≡ P (Ai ) and p·j ≡ P (Bj ). Let

I

i=1 Ai

Nij ≡ (the number of iid observations X1 , . . . , Xn that fall in Ai Bj ). J I Let pˆi· ≡ j=1 Nij /n and pˆ·j ≡ i=1 Nij /n. Show that (15) Qan ≡ Ii=1 Jj=1 (Nij − nˆ pi· pˆ·j )2 /(nˆ pi·pˆ·j ) →d Chisquare((I − 1)(J − 1)). (b) Let pi|j ≡ P (Ai |Bj ). Let n ≡ n·1 + · · · + n·J . For each 1 ≤ j ≤ J, let (j) (j) Nij ≡ (the number of iid P (·|Bj ) observations X1 , . . . , Xnj that fall in Ai Bj ). J Let pˆi|j ≡ j=1 Nij /n·j . Show that when σ[A1 , . . . , AI ] and σ[B1 , . . . , BJ ] are independent, the chisquare statistic satisﬁes (16) Qbn ≡ Ii=1 Jj=1 (Nij − nj pˆi|j )2 /(nj pˆi|j ) →d Chisquare((I − 1)(J − 1)) as n1 ∧ · · · ∧ nJ → ∞. (c) Suppose that both sets of marginal totals n1· , . . . , nI· and n·1 , . . . , n·J are ﬁxed, and that both sum to n. Suppose that n balls are assigned to the IJ cells at random without replacement, subject to the side conditions on the marginal totals stated above. Let Nij denote the number assigned to the (i, j)-th cell. Then I J (17) Qcn ≡ i=1 j=1 (Nij −ni· n·j /n)2 /(ni· nj· /n) →d Chisquare((I − 1)(J − 1))

370

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

as (n1· ∧ · · · ∧ nI· ) ∧ (n·1 ∧ · · · ∧ n·J ) → ∞. [Suppose I = 5 diﬀerent social groups are at work in broadcasting, where the sum of the I group sizes ni· of our data is n = 250. The number whose salaries fall in each decile (thus J = 10) of the observed salaries is necessarily n·j = n/J = 25. The statistic in (17) can be used to test for independence of group and salary level.] 2 Exercise 1.9 Let Vn2 now denote the sample variance. Show, in the context of Doeblin’s CLT, that Tνn /Vνn →d N (0, 1) as n → ∞. Exercise 1.10 Prove a version of Doeblin’s theorem for Xnk ’s independent but not iid; assume the Lindeberg condition and νn /n →p c ∈ (0, ∞). [Revert to the Liapunov condition, if necessary.]

Limiting Distributions of Extremes Exercise 1.11 (a) Let ξn1 , . . . , ξnn be iid Uniform(0, 1) rvs. Then the sample minimum ξn:n satisﬁes nξn:n → Exp(1). (b) Now, ξn:n is the sample maximum. Determine the joint asymptotic distribution of nξn:1 and n(1 − ξn:n ). Exercise 1.12 (Special cases of Gnedenko’s theorem) of an iid sample X1 , . . . , Xn from F (·). Then:

Let Xn:n be the maximum

(a)

P (Xn:n − log n ≤ y) → exp(−e−y ) when 1 − F (x) = e−x for x ≥ 0.

(b)

P (n1/b Xn:n ≤ y) → exp(−|y|b ) for all y < 0, when 1 − F (x) = |x|b for − 1 ≤ x ≤ 0, with b > 0.

(c)

P (Xn:n /n1/a ≤ y) → exp(−y−a ) for all y > 0, a when 1 − F (x) = 1/x for x ≥ 1, with a > 0.

for all y ∈ R,

[Distributions that are “suitably similar” to these prototypes yield the same limiting results, and these limits are the only possible limits.]

2.

VARIATIONS ON THE CLASSICAL CLT

371

2

Variations on the Classical CLT

Notation 2.1 Let Xnk , 1 ≤ k ≤ n for each n ≥ 1, be row-independent rvs having 2 means µnk and variances σnk , and let γnk ≡ E|Xnk − µnk |3 < ∞ denote the third absolute central moments . Let n 2 (1) and γn ≡ nk=1 γnk , and sdn ≡ k=1 σnk (2)

Zn ≡

n

1

sdn

[Xnk − µnk ] .

k=1

Let φnk (·) denote the chf of (Xnk − µnk )/sdn .

2

Theorem 2.1 (Rate of convergence in the CLT) Consider the rvs above. The df FZn of the standardized Zn is uniformly close to the N (0, 1) df Φ, in that (3)

FZn − Φ ≤ 13 γn /sd3n .

Corollary 1 (Liapunov CLT) (4)

Zn →d N (0, 1)

whenever

γn /sd3n → 0.

Corollary 2 (Berry–Esseen for iid rvs) F (µ, σ2 ) with γ ≡ E|X − µ|3 < ∞. Then (5)

FZn − Φ ≤

Let Xn1 , . . . , Xnn be iid rvs with df

8γ √ . σ3 n

Proof. [There is considerable advantage to treating theorem 2.1 by giving a careful proof of the rate of convergence to normality based on Esseen’s lemma and lemma 13.4.1.] Without loss of generality, we assume that all µnk = 0. We now note that (recall that a = b ⊕ c means |a − b| ≤ c) 2 2 n φZn (t) − e−t /2 = k=1 φnk (t) − e−t /2 ≤ e−t

2

(6)

≡ e−t

2

n Log k=1 e

/2

/2

φnk (t) +t2 /2

|ez − 1| ≤ e−t

2

/2

|z| e|z|

− 1 by (13.4.3)

for all z, where n n 2 i2 t2 σnk 2 (a) |z| = Log φnk (t) + t /2 = Log (1 + [φnk (t) − 1]) − 2 2 sd n k=1 k=1 n 2 i2 t2 σnk 2 (7) ≤ [φnk (t) − 1 + ] ⊕ |φnk (t) − 1| 2 sd2n k=1

provided that |φnk (t) − 1| ≤ 12 , using (13.4.2)

372

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS |t|3 γn ≤ + 3 6 sdn k=1 n

(b)

.

|t|3/2 E|Xnk |3/2 sd3/2 n

/2 2 where K1,1/2 =

K1,1/2

8 9

using (13.4.4) [with m = 2 and δ = 1, then with m = 1 and δ = 12 ] (c)

≤

1 2a

|t|3

19

a ≡ sd3n /γn

with

9

using Liapunov’s inequality for (E|Xnk |3/2 )2 ≤ γnk (d)

≤

1 2 4 t

on |t| ≤

9 38

a.

But validity of (7) required that all |φnk (t) − 1| ≤ 12 . However, (13.4.4) with m = 1 and δ = 1 reexpresses this requirement as (e)

2 |φnk (t) − 1| ≤ (t2 σnk /2 sd2n ), and must be ≤

1 2

.

For any ﬁxed t, the bound on |z| in (c) goes to 0 whenever 1/a = γn /sd3n → 0. More2 over, (e) always holds when γn /sd3n → 0, since (max σnk /sd2n )3/2 ≤ (max γnk /sd3n ) ≤ (γn /sd3n ) → 0. Thus φn (t) → exp(−t2 /2) and Zn →d N (0, 1) by (6), whenever we have γn /sd3n → 0. That is, corollary 1 holds. [This is already a good CLT!] Consider the requirement of (e) again. If |t| ≤ a1/3 , then (f) (g)

2 |φnk (t) − 1| ≤ (t2 /2) · σnk /sd2n ≤ (1/2) a2/3 [ γn /sd3n ]2/3

≤

1 2

on 0 ≤ |t| ≤ a1/3 .

9 a we have not yet justiﬁed the requirement made But on the interval a1/3 < |t| ≤ 38 in (e), which gave (d); so we will now show that (d) is still true, even for the larger range a1/3 ≤ |t| ≤ 38 a. Now, |φn (t)|2 is the chf of the symmetrized rv Zns ≡ Zn − Zn (and this rv has mean 0, variance 2, and third absolute moment bounded above by 8γn /sd3n (via the Cr -inequality)). Thus

|φZn (t)| ≤ [ |φn (t)|2 ]1/2 ≤ [1 + 0 − (h)

≤ exp(−t2 [ 12 −

(i)

≤ exp(−t2 /4)

2|t| 3

γn / sd3n ])

2t2 2!

+

|t|3 3!

8γn / sd3n ]1/2

using 1 − x ≤ e−x

for |t| ≤ (3/8) (sd3n /γn ),

as was desired. This leads to (j)

|φZn (t) − e−t

2

/2

| ≤ 2 e−t

2

/4

≤ 2( |t|3 /a) e−t

2

/4

for a1/3 ≤ |t| ≤ 38 a.

Key chf inequality Combining (6), (d), (c), and (g) with (j) gives 2 2 (8) for 0 ≤ |t| ≤ 38 sd3n /γn . φZn (t) − e−t /2 ≤ (2 |t|3 γn /sd3n ) e−t /4

2.

VARIATIONS ON THE CLASSICAL CLT

373

We apply (8) to Esseen’s lemma and get (since we know the variance of a normal distribution) FZn − Φ ≤

(k)

(3/8)a

1 2|t|3 −t2 /4 3.04769 dt + e π|t| a (3/8) a

−(3/8)a

3.04769 1 2 ∞ 2 −t2 /4 t e dt + ≤ a π −∞ 3/8 √ . = [8/ π + (8/3) 3.04769]/a = 12.641/a ≤ 13/a.

(l)

In the iid case use K1,1 = |t|3 β |z| ≤ √ + n 6 n (m)

≤

t2 σ2 2nσ 2

5 β √ |t|3 12 n

1 2

and β ≡ E|X|3 /σ 3 ≥ 1 in (b), and obtain

2

|t|3 β t4 β 2 ≤ √ + 6 n 4n

for all |t| ≤

√

n/β,

with (d) necessarily valid. Thus (8) can be replaced in the iid case by (9)

|φZn (t) − e−t

2

/2

|≤

2 5 γ √ |t|3 e−t /12 3 12 σ n

on 0 ≤ |t| ≤

√ this yields 8γ/ n σ3 when the steps leading to (l) are repeated.

Theorem 2.2 (Lindeberg–Feller) (10)

Zn →d N (0, 1)

(11)

LFn

≡

nσ3 /γ;

2

The following are equivalent: [max1≤k≤n P (|Xnk − µnk |/sdn > )] → 0 .

and

n k=1

√

[|(x−µnk )|≥ sdn ]

x − µnk sdn

2 dFnk (x) → 0

for all > 0 .

[Condition (11) implies that Mn ≡ [max1≤k≤n |Xnk −µnk |/sdn ] →p 0, via (14.1.9).] Proof. (Lindeberg) We prove the suﬃciency here, with the necessity considered in the following separate proof. We note that the moment expansion inequality (inequality 13.4.1) gives bounds on βnk (t), where (12)

φnk (t) ≡ 1 + θnk (t) ≡ 1 −

2 σnk t2 + βnk (t) sd2n 2

deﬁnes θnk (t) and βnk (t). Moreover (in preparation for the product lemma) n > n n σ2 t2 (13) φZn (t) = k=1 φnk (t) = k=1 [1 + θnk (t)] = 1 − nk2 + βnk (t) sdn 2 k=1

374

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

where (14)

θn (t) ≡

n

θnk (t) = −

k=1

n n n 2 t2 σnk 2 + β (t) = −t /2 + βnk (t) . nk 2 sd2n k=1 k=1 k=1

The moment expansion inequality of (13.4.10) (compare this with (7)) gives n n 2 σnk t2 (a) βnk (t) = φnk (t) − 1 − 0 + 2 sdn 2 k=1

(15)

≤

(b)

≤

k=1

n it(x − µnk ) [it(x − µnk )]2 it (x−µnk )/sdn dFnk (x) − 1+ + e 2 sdn 2 sd n k=1 n k=1

(c)

+

[|(x−µnk )|< sdn ]

n k=1

(16)

≤

3 1 it(x − µnk ) dFnk (x) 6 sdn

|t|3 6

[|(x−µnk )|≥ sdn ]

n

2 k=1 σnk sd2n

+ t2

it(x − µnk ) 2 dFnk (x) sdn

n k=1

[|(x−µnk )|≥ sdn ]

x − µnk sdn

2 dFnk (x) .

Thus normality holds, provided that the integral in (16) goes to 0 for all > 0. Note that [max1≤k≤n |θnk (t)| ] → 0 as required by the product lemma, since we can actually claim that 2 (d) σnk /sd2n ≤ [ [|x−µnk |≤ sdn ] (x − µnk )2 dFnk (x)/sd2n + (e)

[|x−µnk |> sdn ]

≤ 2 + o(1) ≤ ,

(x − µnk )2 dFnk (x)/sd2n

for n ≥ (some n ) .

2

Proof. (Feller) We proved suﬃciency in the previous proof; we now turn to necessity. Suppose that condition (10) holds. Applying (13.4.2) [since the easy exercise 14.2.9 below applied to our uan rvs shows that the terms znk = φnk (t) − 1 converge uniformly to 0 on any ﬁnite interval] gives n n n (17) | k=1 Log φnk (t) − k=1 [φnk (t) − 1] | ≤ k=1 |φnk (t) − 1|2 n 2 (18) ≤ [max1≤k≤n |φnk (t) − 1|] × (t2 /2) × [ k=1 σnk /sd2n ] by (13.4.5) (a)

≤ o(1) × (t2 /2) × 1 → 0,

using (10).

We thus have n n (b) Log 1 φnk (t) = 1 [φnk (t) − 1] + o(1).

2.

VARIATIONS ON THE CLASSICAL CLT

375

But we also know that (c)

Log

n 1

φnk (t) → −t2 /2,

since we have assumed asymptotic normality. [Recall that a = b ⊕ c means that |a − b| ≤ c.] Combining (b) and (c) shows that for every tiny > 0 and every huge M > 0 we have (d)

−t2 /2 = Real(−t2 /2) = Real{

n 1

[φnk (t) − 1]} ⊕ for |t| ≤ M

for all large n; that is, for n ≥ (some nM ) we have (e)

t2 /2 =

n 1

[1 − cos(t (x − µnk )/sdn )] dFnk (x) ⊕ on |t| ≤ M.

Deﬁne yk ≡ (x − µnk ). We further deﬁne In ≡ [|x − µnk | < sdn ]. Note that (f)

0 ≤ 1 − cos(ty/sdn ) ≤ (t2 y 2 /2 sd2n ).

Thus for all |t| ≤ M we have for all n ≥ nM that (g)

(t2 /2)

n

= (t2 /2) −

k=1

(19)

=

(i)

≤2

(j)

≤ (2/ 2 )

(k)

≤ 2/ 2 + .

k=1

c [1 In

n

c In

k=1

In

n k=1

In

(yk2 /sd2n ) dFnk (x)]

(t2 yk2 /2 sd2n ) dFnk (x)

In

n

≤ (t2 /2) − k=1

n

(h)

n

(yk2 /sd2n ) dFnk (x) = (t2 /2) [1−

c In

k=1

[1 − cos(tyk /sdn )] dFnk (x)

− cos(tyk /sdn )] dFnk (x) ⊕

by (f)

by (e)

[the key step]

dFnk (x) +

n k=1

2 c (yk In

sd2n ) dFnk (x) +

Specifying t2 = M 2 = 4/( 2 × θ) in (g) (for some 0 < θ < 1) shows that for all n ≥ nθ we have (l)

sd−2 n

n 1

c In

yk2 dFnk (x) ≤ 2 θ,

where θ > 0 is arbitrary. Thus, the Lindeberg condition (11) holds.

2

376

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

Remark 2.1 (Lindeberg’s condition) still be true that (a)

Sn /σn →d N (0, a2 )

with

(i) If Lindeberg’s condition fails, it may a2 < 1 and [max1≤k≤n σk2 /σn2 ] → 0.

Let the rv’s Un be independent (0, 1) with Un equal to −cn, ∞ 0, cn with probabilities ∞ 1/(2n2 ), 1 − 1/n2 , 1/(2n2 ). Since 1 P (|Un | ≥ ) = 1 n−2 < ∞, the Borel– Cantelli lemma shows√that for a.e. ω the rv sequence Un satisﬁes Un = 0 only ¯n →p 0 follows. Now let Y1 , Y2 , . . . be iid (0, 1) rvs, ﬁnitely often. Thus n U √ so that n Y¯n →d N (0, 1) by the CLT. For n ≥ 1 set Xn ≡ Yn + Un , and let Sn ≡ X1 + · · · + Xn . Note that σn2 ≡ Var[Sn ] = (1 + c2 )n. So, by Slutsky’s theorem, √ √ ¯n )/ 1 + c2 Sn /σn = ( n Y¯n )/ 1 + c2 + ( n U →d N (0, 1)/ 1 + c2 + 0 ∼ = N (0, 1/(1 + c2 )) (b)

= N (0, a2 )

with a2 = 1/(1 + c2 ) < 1.

√ 2 ˇnk /ˇ σn2 ] → 0. But, even so, Lindeberg’s (One could also let c = n.) Note that [max σ condition fails, since a n √ LFn = x2 dFXk (x) n k=1 [|x|≥ n/a ] (c)

∼

a n

√

k:ck≥

c2 (kc)2 + o(1) → > 0; k2 (1 + c2 ) n/a

the nonzero contribution shown in the last step is due to Uk , whereas we do already know that the contribution due to the Yk ’s is o(1). This example shows that it is possible to have Xn →d X without having Var[Xn ] → Var[X]. Note that Var[N (0, 1/(1 + c2 )] = 1/(1 + c2 ) < 1 = lim 1 = limVar[Sn /σn ] (via the Fatou lemma and Skorokhod’s theorem). (ii) Note that if Xn1 ∼ = N (0, pn), for some 0 < p < 1, Xnk ≡ 0 for 2 ≤ k ≤ [pn], and Xnk ∼ N (0, 1) for pn < k ≤ n for independent rvs Xnk , then Sn /σn →d N (0, 1), = 2 while Lindeberg’s condition fails and [max1≤k≤n σnk /σn2 ] → p > 0. 2 Remark 2.2 It is known that the constant 8 in (5) can be replaced by 0.7975. 3 It is also known in the iid case √ with E|X| < ∞ that the “limiting distribution measure” d(F, Φ) ≡ limn→∞ nFZn − Φ exists, and moreover that it takes on the √ √ value supF (σ3 /γ) d(F, Φ) = ( 10 + 3)/(6 2π ) = 0.409. This sup is achieved by √ √ c [Bernoulli(a) − a], where c = ( 10− 3)/2 and a = (4 − 10 )/2. Thus the constant 0.7975 cannot be greatly improved. Many other improvements and reﬁnements of the Berry–Esseen theorem are possible. The books by Bhattacharya and Rao (1976, pp. 110, 240) and Petrov(1977) both give many. We list three as “exercises” in exercise 2.11 below. 2 Exercise 2.1 Beginning with (15), obtain the Berry–Esseen bound (but with a diﬀerent constant) by appeal to the second product lemma.

2.

VARIATIONS ON THE CLASSICAL CLT

377

Exercise 2.2 (Liapunov’s (2 + δ)-CLT) Deﬁne γnk = E|Xnk |2+δ for every value 0 < δ ≤ 1. Suppose we have Liapunov’s (2 + δ)-condition that n 2+δ /σn2+δ → 0 for some 0 < δ ≤ 1. k=1 E|Xnk | Show that Sn /σn →d N (0, 1). Exercise 2.3 Construct an example with iid X1 , X2 , . . . for which the Lindeberg condition holds, but for which Liapunov’s (2 + δ)-condition fails for each 0 < δ ≤ 1. Exercise 2.4 Establish the (2 + δ)-analogue of theorem 2.1. [Hint. Use both 2 + δ and 1 + δ moments in line (b) of the theorem 2.1 proof, via lemma 13.4.2.] Exercise 2.5 (Liapunov-type WLLN) Let Xn1 , . . . , Xnn , n ≥ 1, be a triangular array of row-independent rvs with 0 means. Then n 1+δ /n1+δ → 0 for some 0 < δ ≤ 1 k=1 E|Xnk | ¯ n →p 0 as n → ∞. implies that X Exercise 2.6

Formulate a WLLN in the spirit of the Lindeberg–Feller theorem.

Exercise 2.7 (Pitman) Exercise 2.8

¯ n →p a holds if and only if φ (0) = ai. For iid rvs, X

Complete the proof of theorem 10.9.1.

Exercise 2.9 (Some uan equivalences)

The following are equivalent:

(a)

|Xnk |’s are uan [meaning that max1≤k≤n P (|Xnk | ≥ ) → 0 for all > 0].

(b)

[max1≤k≤n |φnk (t) − 1|] → 0 uniformly on every ﬁnite interval of t’s. max1≤k≤n α(x) dFnk (x) → 0 for α(x) ≡ x2 ∧ 1.

(c)

¯ n for a single iid Exercise 2.10 Let Zn denote an appropriately normalized X sequence of rvs. Prove the following statement: Z¯n →d N (0, 1) if and only if the symmetrized rvs Z¯ns →d N (0, 2) (provided it’s true). Exercise 2.11 (a) (Petrov) Suppose Xn1 , . . . Xnk are row independent rvs for n 2 2 which Xnk ∼ ), and set σn2 ≡ k=1 σnk and Fn (x) ≡ P (Sn /σn ≤ x). Then = (0, σnk for some universal constant C we have 0 1 3 2 (20) Fn −Φ ≤ C σn−3 nk=1 EXnk 1[|Xnk |<σn ] + σn−2 nk=1 EXnk 1[|Xnk |≥σn ] . 2 (b) (Petrov) If E[Xnk g(Xnk )] < ∞ for 1 ≤ k ≤ n where

(21)

g ≥ 0 is even and for x > 0, and x/g(x) is for x > 0,

then for some absolute constant C we have the very nice result n 2 (22) Fn − Φ ≤ C 1 E[Xnk g(Xnk )]/σn2 g(σn ).

378

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

(c) (Nagaev) Bounds on |Fn (x) − Φ(x)| that decrease as |x| → ∞ are given (in the iid case) in the expression √ (23) for all real x. |Fn (x) − Φ(x)| ≤ C(E|X|3 /σ 3 )/( n(1 + |x|3 ) (d) (Bernstein) Let r > 2. Consider row-independent rvs Xn1 , . . . , Xnn for which n 2 we have Xnk ∼ ). Let Zn ≡ k=1 (Xnk − µnk )/σn . = (0, σnk n r/2 (α) Suppose k=1 |Xnk |r /σn → 0. Then we have Zn →d N (0, 1) as well as the conclusion that E|Zn |r → E|N (0, 1)|r . 2 /σn2 ] → 0. (β) The converse holds if we also have [max1≤k≤n σnk (e) See Petrov (1975, pp. 118, 113, 125, 103) for (a), (b), (c), and (d).

Statistical Applications Exercise 2.12 (Simple linear regression) Consider the simple linear regression model of (9.3.25); thus we are assuming that (24)

Ynk = α + β xnk + k

for iid rvs k ∼ = (0, σ2 )

and for known constants cnk for 1 ≤ k ≤ n. The least squares estimators (LSEs) α ˆ n and βˆn of α and β are deﬁned to be those values of a and b that minimize the sum of squares n1 [ Ynk − (a + b xnk ) ]2 . (a) Show that the LSEs are given by n n ¯n ) Ynk 1 (xnk − x ˆ ¯ (25) α ˆn = Yn and βn = ≡ k=1 dnk Ynk . n 2 (x − x ¯ ) nk n 1 n (b) Let SSxx ≡ 1 (xnk − x ¯n )2 and dn = (dn1 , . . . , dnn ) , (26) ¯n |2 D(xn ) ≡ max |xnk − x SSxx = max dnk = D(dn ) . 1≤k≤n

1≤k≤n

Use the Cram´er–Wold device and the weighted sums of example 11.6.4 to show that √ n [ˆ αn − α] (27) √ →d N (0, σ 2 I), provided that D(dn ) = D(xn ) → 0 SSxx [βˆn − β] (recall (11.6.20)). [Note also that the LSE βˆn of β is given by βˆn = SSxY /SSxx .] Deﬁnition 2.1 (Noncentral distributions) (a) Let X1 , . . . , Xm be independent, m and suppose that Xi ∼ = N (θi , σ2 ). Let θ2 ≡ 1 θi2 , and deﬁne δ via δ2 ≡ θ2 / σ 2 . Show that the quadratic form m m U ≡ 1 Xi2 / σ 2 ∼ (28) = (Z1 + θ)2 + i=2 Zi2 , where Z1 , . . . , Zm are iid N(0, 1) rvs. Denote this distribution by (29)

U∼ = χ2m (δ 2 /2),

and say that U is distributed as noncentral chisquare with m degrees of freedom and noncentrality parameter δ.

2.

VARIATIONS ON THE CLASSICAL CLT

379

(b) Let Y ∼ = N (θ, 1), U ∼ = χ2m (δ 2 /2) and V ∼ = χ2n be independent rvs. We deﬁne the noncentral Student-tn (θ) distribution and the noncentral Snedecor-Fm,n (δ 2 /2) distribution by writing (30)

Tn (θ) ≡

Y ∼ n U U/m ∼ = = Student-tn (θ) and = Snedecor-Fm,n (δ 2 /2) . V /n m V V /n

(Note that Tn2 (θ) ∼ = F1,n (θ2 /2).) Proposition 2.1 (Form of the noncentral distributions) Consider the rvs U , V , and Y of the previous deﬁnition. Let y > 0. (a) The rv U of (28) satisﬁes ∞ (31) P (χ2m (δ 2 /2) > y) = k=0 P (Poisson(δ2 /2) = j) × P (χ2m+2j > y) . Here, Poisson(λ) denotes a Poisson rv with mean λ, and χ2r denotes an ordinary chisquare rv with r degrees of freedom. (b) It is thus trivial that 2 (32) P (Fm,n (δ 2 /2) > y) = ∞ k=0 P (Poisson(δ /2) = j) × P (Fm+2j,n > y) . √ (c) For Cn ≡ 2(n+1)/2 Γ(n/2) π n we have

(33)

P (Tn (δ) > y) ∞∞ = C1n y 0 u(n−1)/2 e−u/2 exp(− 12 (v( un )1/2 − δ)2 ) du dv .

Exercise 2.13

Prove proposition 2.1.

Exercise 2.14 (Chisquare goodness of ﬁt, again) (a) (Local alternatives) We suppose that the statistic Qn ≡ Qn ( p0 ) of (14.1.13) is computed, but that in √ reality the true parameter vector is now pn ≡ p0 + a/ n (with k1 ai = 0, so that the coordinates pni add to 1). Let pˆni ≡ Nni /n estimate pni for 1 ≤ i ≤ k. Show that the vector √ √ nk×1 ≡ |[ n (ˆ + d , (34) W pni − p0i )/ p0i ]| →d W √ √ ∼ where W = N (0, I − p p ) and di ≡ ai / p0i for 1 ≤ i ≤ k. Thus (35)

. W n →d Q ≡ (W + d ) (W + d ) ∼ Qn = W = χ2k−1 (d d/2) n

(b) (Fixed alternatives) Suppose that Qn ≡ Qn ( p0 ) is computed, but a ﬁxed p is true. Show that k 2 1 (36) i=1 (pi − p0i ) / p0i . n Qn →a.s. Exercise 2.15 (37) (38)

∼ Suppose X = N (θ, Σ), with rank(Σ) = r. Show that

Σ− X = Y Y ∼ X , = χ2r (θ Σ− θ/2)

where Ir 0 −1/2 Y ≡ Σ−1/2 X = (ΓD−1/2 Γ )X ∼ N Σ θ, . = 0 0

380

3

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

Local Limit Theorems

Recall from Scheﬀ´e’s theorem that if fn and f are densities with respect to some dominating measure µ, then (1)

fn (x) → f (x)

a.e. µ

implies that (2)

dT V (Pn,P ) ≡ sup |P (Xn ∈ B) − P (X ∈ B)| → 0

as n → ∞.

B∈B

Thus convergence of densities implies convergence in total variation distance, which is stronger than convergence in distribution. We will now establish (1) in a CLT context, for summands that are either suitably continuous or else are distributed on a grid. Theorem 3.1 (Local limit theorem, continuous case) Let X, X1 , X2 , . . . be ∞ √ iid (0, σ2 ) with −∞ |φX (t)| dt < ∞. Then Sn / n has a density fn (·) for which (3)

sup

−∞<x<∞

|fn (x) − (1/σ)fZ (x/σ)| → 0

as n → ∞,

for the N (0, 1) density fZ (·). Theorem 3.2 (Local limit theorem, discrete case) Let X1 , X2 , . . . be iid (0, σ2 ) rvs that take values on the grid a ± md for m = 0, ±1, ±2, . . . . Now √ √ let x = (na + md)/(σ n ), m = 0, ±1, ±2, . . . , denote a possible value of S /(σ n ), n √ and let pn (x) ≡ P (Sn /(σ n ) = x). Then √ (4) as n → ∞, sup |(σ n/d ) pn (x) − fZ (x)| → 0 −∞<x<∞

for the N (0, 1) density fZ (·). Example 3.1 (5)

Let X1 , . . . , Xn be iid Bernoulli(θ) rvs. Then (by (4)) . / √ 1 m − nθ sup n P (Sn = m) − fZ → 0. nθ(1 − θ) nθ(1 − θ) 0≤m≤n

Exercise 3.1

Verify (5) by direct computation.

Exercise 3.2

Give an example where Xn →d X, but (2) fails.

2

Proof. Consider theorem 3.1. Without loss of generality we may suppose that σ = 1. Notice that √ √ (a) |φSn /√n (t)| dt < ∞, since |φSn /√n (t)| = |φX (t/ n )|n ≤ |φX (t/ n )|. Thus the Fourier inversion formula of (13.2.9) gives ∞ (b) fn (y) ≡ fSn /√n (y) = (1/2π) −∞ e−ity φSn /√n (t) dt.

3.

LOCAL LIMIT THEOREMS

381

This same formula also holds for the distribution of a N(0, 1) rv Z. Thus ∞ √ 2 (c) 2π|fn (x) − fZ (x)| ≤ −∞ |φnX (t/ n ) − e−t /2 | dt = (d)

+ [|t|≤a]

√ [a<|t|<δ n ]

+

√ [|t|≥δ n ]

√ 2 |φnX (t/ n ) − e−t /2 | dt

≡ I1n + I2n + I3n .

We ﬁrst specify δ > 0 so small that |φ(t)| ≤ exp(−t2 /4) for |t| ≤ δ. This is possible, since |φ(t)| = |1 − 0 − t2/2| + |o(t2 )| ≤ 1 − t2 /4 ≤ exp(−t2 /4) in some neighborhood |t| ≤ δ, by inequality 13.4.2. √ √ 2 Thus for a speciﬁed large enough (since |φX (t/ n )|n ≤ e−t /4 for |t| < δ n ) we have 2 I2n ≤ [|t|>a] 2e−t /4 dt < . (e) For this ﬁxed large a we have (f)

for n ≥ ( some n1 ),

I1n <

since the Cram´er–L´evy theorem implies that the convergence of these chfs is uniform on any |t| ≤ a. Now, X is not distributed on a grid (we have a formula for the density). Thus |φX (t)| < 1 for all t = 0, by proposition 13.6.2. Moreover, |φX (t)| → 0 as |t| → ∞, by the Riemann–Lebesgue lemma, giving |φX (t)| < (some θ) < 1 for |t| > (some λ). Thus, since θ < 1, ∞ √ 2 I3n < θn−1 −∞ |φX (t/ n )| dt + [|t|>δ√n ] e−t /2 dt =

√

n θn−1

(6)

= o(n−r )

(g)

<

∞

−∞ |φX (t)| dt

√ 2 + 2(δ n )−1 e−nδ /2

for any r > 0 whenever lim|t|→0 |φX (t)| < 1.

for n ≥ (some n2 ).

Combining (e), (f), and (g) into (d) establishes the claim made in the theorem.

2

Proof. Consider theorem 3.2. By the inversion formula (13.6.1) given for distributions on grids, √ √ πσ√n/d √ σ n σ n d √ (h) pn (x) = φnX (t/ n )e−itx dt. √ d d σ n 2π −πσ n/d By the inversion formula (13.2.9) given for densities, ∞ 2 (i) fZ (x) = (1/2π) −∞ e−t /2 e−itx dt.

382

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

Thus (j)

√ |(σ n/d) pn (x) − fZ (x)| ≤ (1/2π)

πσ√n/d

+(1/2π) (k)

=

√ −πσ n/d

√ 2 |φnX (t/ n ) − e−t /2 | dt

2 √ e−t /2 [|t|>πσ n/d ]

+ [|t|≤a]

√ [a<|t|<δ n ]

+

dt

√ √ [δ n<|t|<πσ n/d ]

√ 2 |φnX (t/ n ) − e−t /2 | dt

+o(n−r ) (l)

≡ I1n + I2n + I3n + o(n−r ).

The proof of theorem 3.1 applies, virtually verbatim; the only thing worthy of note √ is that 0 < θ < 1 now holds, since πσ n/d is only 12 of the period. 2

4.

GAMMA APPROXIMATION

4

Gamma Approximation

383

If the underlying summands Xnk in a CLT approximation are symmetric, then a normal approximation may seem particularly appropriate. But what if the underlying distribution is positively skewed? (If X is negatively skewed, we just consider √ −X instead.) Consider the rv Gr ≡ [Gamma(r) − r]/ r ∼ = (0, 1), where r is chosen to make the the third cumulants match. Might this not give a better approximation for small n? And since Gr →d N (0, 1) as r → ∞, there is no contradiction at the limit. We will show that this recipe works. The fact that the sum of independent gammas is again gamma is crucial to the technical details of the proof. Can we go one step further and match the ﬁrst four cumulants? Yes, because sums of independent rvs that are distributed as “gamma-gamma” again belong to the same family. [This will also work within our ability to match up the cumulants.] The skewness γ1 ≡ µ3 /σ 3 = E(X − µ)3 /σ 3 is deﬁned to be the third cumulant of the standardized rv, and the tail heaviness (or kurtosis) is the fourth such cumulant deﬁned by γ2 ≡ µ4 /σ 4 − 3 = E(X − µ)4 /σ 4 − 3. We will use these formulas with µ = 0 and σ2 = 1, so that the skewness becomes µ3 and the tail heaviness becomes µ4 − 3. For the standardized gamma, the ﬁrst four cumulants are given by Gamma(r) − r ∼ 2 6

√ (1) Gr ≡ . = 0, 1; √ , r r r For the diﬀerence of two independent gammas we let p + q = 1 and c, d > 0 and set u ≡ p/c and v ≡ q/d, and then further deﬁne r = c2 n and s = d2 n and set

3/2 3/2 2 2 √ √ Gr,s ≡ − p Gc2 n + q Gd2 n ∼ = 0, 1; √2n (− p c + q d ), n6 ( pc2 + dq 2 ) (2)

√ √ = 0, 1; √2n (− p u + q v), n6 (u2 + v 2 ) . [This parametrization can match all (µ3 , µ4 ) pairs for which µ23 ≤ 23 (µ4 − 3), all of which have heavier tails than normal distributions.] Theorem 4.1 (Gamma approximation; the GLT) Let Xn1 , . . . , Xnn be iid as an X having df F with cumulants (0, 1; µ3 , µ4 − 3), where µ3 ∈ (0, ∞), so that √ (3) Zn ≡ √1n nk=1 Xnk ∼ = (0, 1; µ3 / n, (µ4 − 3)/n) . Assume either (4)(a) for the df results below or (4)(b) for the density results, where: ∞ (4) (a) |φX (t)| → 0 as |t| → ∞ or (b) −∞ |φX (t)| dt < ∞ . √ ∼ √ 2 (i) Let r ≡ 4n/µ3 , so that Gr ≡ [Gamma(r)−r]/ r = (0, 1; µ3 / n, 3 µ23 /2n). Then for some constants CF or CF,n → 0 (that may depend on the df of (X − µ)/σ), and fZn − fGr ≤ CF /n when µ4 < ∞, √ (6) FZn − FGr ≤ CF,n / n and fZn − fGr ≤ CF,n / n when µ3 ∈ (0, ∞). √ (ii) Suppose r and s can be speciﬁed so Gr,s ∼ = (0, 1; µ3 / n, (µ4 −3)/n). Then n3/2 √ can replace n in (5) when µ5 < ∞. And n can replace n in (6) when µ4 < ∞. (iii) The density gr (·) of Gr (·) may replace the N (0, 1) density in the local limit theorems of section 14.3. (5)

FZn − FGr ≤ CF /n

√

384

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

Proof.

We initially approximate the distribution of Zn by that of √ (a) r ) − r¯]/ r¯ with r¯ ≡ n/(2 µ23 ) Z¯n ≡ √12 N (0, 1) + √12 [Gamma(¯ √ √ ∼ r) = (0, 1; 1/ 2¯ r, 3/2¯ r) = √12 (0, 1; 0, 0) + √12 (0, 1; 2/ r¯, 6/¯ √ ∼ (b) matching (3) to three cumulants = (0, 1; µ3 / n, 3 µ23 /n) n 1 n 1 1 (c) ≡ √n 1 [ √2 Nk + √2 Wk ] ≡ √1n 1 Yk , √ where the Nk ∼ = N (0, 1) and the Wk ∼ = [Gamma(a) − a]/ a with a = 1/(2µ23 ) are independent. Let φY (t) ≡ EeitY and ψY ≡ log φY , with φX and ψX deﬁned analogously. Then √ √ n )−ψY (t/ n )]

|φZn (t) − φZ¯n (t)| = |φZ¯n (t)| × |en[ψX (t/

(d)

− 1|

≡ |φZ¯n (t)| × |ez − 1| √ √ ≤ |φN (0,1) (t/ 2 )| × |φ Wk /√n (t/ 2 )| × |z|e|z|

× 1 × |z|e|z| . √ Here (provided that |t|/ n is suﬃciently small for the expansion of (13.4.22) to be valid) the inequality (13.4.22) then gives (since the ﬁrst three cumulants of X and Y match) √ √ (f) |z| = n |ψX (t/ n ) − ψY (t/ n )| ≤ t4 c¯4 [EX 4 + EY 4 ]/n (e)

≤ e−t

2

/4

3/2

≤ t4 c¯4 [(3 + (µ4 − 3)) + (3 + 3 µ23 ) ]/n ≤ t4 7 c¯4 µ4 /n (g)

≡ t4 c2 /n

(h)

≤ t2 /9

for |t| ≤

√

n/3c .

Plugging both (g) and (h) into (e) gives (i)

|φZn (t) − φZ¯n (t)| ≤ (c2 t4 /n) exp (−(5/36) t2 )

√ for |t| ≤ (some d ) n ,

where c and d depend on the β2 value of (X −µ)/σ (see (A.2.26)). [The speciﬁcation in (a) that Z¯n has a normal component is not natural or practically useful, but it delivers the technically lovely exponential bound component exp (−t2 /4) in both (d) and (e). Since (a) is not useful for a practical approximation, we will overcome this objection by doing the approximation (a) again—to diﬀerence it out. (I believe this whole approach to be new.)] √ Let Gn1 , . . . , Gnn be iid as [Gamma(b) − b]/ b for b ≡ 4/µ23 . Then n ∼ [Gamma(r) − r]/√r Gr ≡ √1 (j) Gnk = with r ≡ 4n/µ2 n

(k)

3

1

√

∼ = (0, 1; µ3 / n, (3/2) µ23 /n),

matching (3) to three cumulants.

We now approximate Gr by the Z¯n of (a) [just as earlier we approximated Zn by the Z¯n of (a)]. This gives (with generic constants c and d) √ (l) |φGr (t) − φZ¯n (t)| ≤ (c2 t4 /n) exp (−(5/36) t2) for |t| ≤ d n .

4.

GAMMA APPROXIMATION

385

Combining (i) and (l) gives (7)

|φZn (t) − φGr (t)| ≤ (c t4 /n) exp (−(5/36) t2 )

√ for |t| ≤ d n ,

where the generic c and d may depend on the df of (X − µ)/σ. Consider (5)(b). The inversion formula (using (13.4.22) to expand) gives ∞ 1 |fZn (x) − fGr (x)| = 2π | −∞ e−itx [φZn (t) − φGr (t) ] dt | ≤ I1n + I2n + I3n (m) ≡ [ |t|≤d√n ] |φZn (t) − φGr (t) | dt +

√ [|t|>d n ]

|φZn (t)| dt +

√ [|t|>d n ]

|φGr (t)| dt .

Now (7) bounds the integrand of the lead term to give (n) I1n ≤ (c t4 /n) exp (−(5/36)t2 ) dt ≤ c/n . Since |φX (t)| dt < ∞ by (4)(b), the density inversion formula (13.2.9) shows that X has a density fX (·). Since X is thus not distributed on a grid (and likewise Y ), proposition 13.6.2 gives ∞ θ ≡ φX ∞ d ∨ φW d < 1 .

Thus the second term in (m) satisﬁes √ √ I2n = [|t|>d√n ] |φX (t/ n )|n dt ≤ n [|s|>d ] |φX (s)|n ds (o)

≤ θn−1

√ ∞ n −∞ |φX (s)| ds = o(n−r ),

for any r > 0,

since the θn term goes to 0 geometrically. Likewise, I3n = o(n−r ), for any r > 0, since |φG |k satisﬁes (4)(b), for some k. Combine (n) and (o) into (m) to get (5)(b). Consider (5)(a). We will apply Esseen’s lemma. Thus

(p) (q)

FZn (x) − FGr (x) ≡ I1n + I2n + I3n 1 ≤ { [|t|≤d√n ] + [d√n<|t|≤d n ] } |t| |φZn (t) − φGr (t)| dt + 24 gr / π d n

√ ≤ (c|t|3 /n) exp (−(5/36)t2 ) dt + 2(dn/d n )θn + 24gr /(πdn) = O(1/n) ,

where (4)(a) is now used to obtain θn . Consider (6), when µ4 is not assumed ﬁnite. Use of (13.4.22) at line (f) must be replaced √ by use of (13.4.21). The |t|3 E|X|3 in (13.4.22) would give √ a bound of only CF / n at line (m) of the current proof; but the added δ (t/ n ) term in 3 √ (13.4.21) that is valid on |t| ≤ d n (now with a tiny d ) allows a CF to be replaced by a CF,n → 0. Dominated convergence is used for this, with dominating function guaranteed by E|X|3 < ∞. If we knew any appropriate two-parameter family closed under convolution, we could choose those two parameters to match both third and fourth cumulants. Then CF /n3/2 under µ5 < ∞ and CF,n /n under µ4 < ∞ would be possible. The proof is

386

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

essentially unchanged, and needs no further comment. The diﬀerence of two gammas can be speciﬁed in several diﬀerent ways. All work. The only question is which has the greatest coverage of the (skewness, kurtosis)-plane. Using gammas, we seem stuck with positive kurtosis, which leaves out some of the least important situations. [Edgeworth expansions allow us to cover the whole (skewness, kurtosis)-plane, but they have some other deﬁciencies. For instance, the Edgeworth approximation to a df or density is not necessarily a df or density itself.] 2

Poisson Approximation Most discrete distributions we care about live on the integers, and the write-up here will reﬂect that fact and make this case ﬁt our notation with the least eﬀort. Rather than approximating sums of such rvs X by an appropriate gamma with a continuity correction, we will use a nice discrete analogue of the gamma. For the standardized Poisson, the ﬁrst four cumulants are Poisson(r) − r ∼ 1 1

√ Gr ≡ (8) . = 0, 1; √ , r r r For the diﬀerence of two Poissons we let p + q = 1 and c, d > 0 and set u ≡ p/c and v ≡ q/d, and then deﬁne r = c2 n and s = d2 n and set √ √ ∼ (0, 1; √1 (− p3/2 + q3/2 ), 1 ( p22 + q22 )) Gr,s ≡ − p Gc2 n + q Gd2 n = c d n c d n (9) √ √ = (0, 1; √1n (− p u + q v), n1 (u2 + v 2 )) . [This approach can (multiply) match all (µ3 , µ4 ) pairs for which µ23 ≤ (µ4 − 3).] Theorem 4.2 (Poisson approximation) Consider a rv X on the integers and let Zn be as in (3). Let r ≡ n/µ23 , so that √ √ Gr ≡ [Poisson(r) − r]/ r ∼ = (0, 1; µ3 / n, µ23 /n). (i) Then for some constants CF and CF,n → 0 (that may depend on the df of the standardized rv (X − µ)/σ): (10)

pZn − pGr ≤ CF /n3/2

when µ4 < ∞ .

(11)

pZn − pGr ≤ CF,n /n

when only µ3 ∈ (0, ∞) .

√ [Most probabilities that one computes involve summing over the appropriate M n number of terms that are each of the type pZn (·).] √ (ii) Suppose r and s can be speciﬁed so that Gr,s ∼ = (0, 1; µ3 / n, (µ4 − 3)/n). Then n2 can replace n3/2 in (10), provided that µ5 < ∞. And n3/2 can replace n in (11), provided that µ4 < ∞. Proof. The appropriate inversion formula now (for a distribution on the grid am + b) is given by a (12) pm ≡ P (X = m) = 2π [ |t|≤π/a ] exp (−it(am + b)) φX (t) dt .

4.

GAMMA APPROXIMATION

387

By the previous proof (including the previous step (a) normal component, but now appearing in step (u)) yields √ (u) |φZn (t) − φGr (t)| ≤ (c2 t4 /n) exp (−(5/36) t2) for |t| ≤ d n , for c and d that may depend on the df of (X −µ)/σ. Applying the inversion formula in (12) now gives √ √ √ (v) n |P (Zn = m/ n ) − P (Gr = m/ n ) | √ √ √ √ = 2 π n√n | [ |t|≤π√n ] e−itm/ n [φZn (t/ n ) − φGr (t/ n )] dt| √ √ (w) ≤ [ |t|≤d√n ] |φZn (t/ n ) − φGr (t/ n )| dt √ √ + [d√n<|t|≤π√n ] |φZn (t/ n )| dt + [ d√n<|t|≤π√n ] |φGr (t/ n )| dt (x)

≤ c/n + o(n−r ) + o(n−r ) ,

as√is now easily shown with the same arguments as before (because θ < 1, since π n never reaches a full period of φX ; recall proposition 13.6.2). 2 Exercise 4.1

Verify part (iii) of theorem 3.1.

Exercise 4.2 some k > 0.

Show that the chf φ of the Gr of (1) satisﬁes

|φ(t)|k dt < ∞ for

Exercise 4.3 We can replace the Poisson by the NegBiT(r, p) distribution with the moment structure NegBiT(r, p) − r/p ∼ 1 1 + q 1 1 + 4q + q 2 (13) 0, 1; √ √ , . Gr ≡ = r q r q rq/p2 This is probably more useful than the previous theorem. (a) Verify the claim. (b) Provide some numerical work to compare Poisson and NegBiT approximations to a situation of interest. Remark 4.1 (Gamma approximation or Edgeworth approximation?) In the next section we will derive the classical Edgeworth approximations. The ﬁrst-order Gamma (or Poisson, or NegBiT) approximations of the current section are of the same order as the ﬁrst-order Edgeworth approximations. Moreover, approximation by the Gr -distribution is an approximation by a probability distribution; but this is not true of the Edgeworth approximation. Happily, gamma approximations are easily and accurately implemented in S-plus, say. The situation is similar regarding the two second-order approximations, provided that the ﬁrst two cumulants of the underlying rv can be matched within the family of Grs -distributions. However, the Grs -distributions are not available within any set of computer-generated routines I know, so that this would be hard to implement at present. However, this would seem to make a nice project for a computer-oriented statistician. 2

388

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

Examples ¯ n and Sn2 ) Suppose X1 , . . . , Xn is Example 4.1 (Sampling distribution of X a random sample from a population whose ﬁrst four cumulants are (µ, κ2 ; κ3 , κ4 ). [Let K&S denote Kendall and Stuart (1977, Vol. I).] How do we apply a gamma approximation? (a) Consider ﬁrst an inﬁnite population, in which √ κ2 = σ 2 , κ3 = µ3 , and κ4 = µ4 −3. ¯ n − µ) as Then (13.4.20) gives the ﬁrst four cumulants of n (X √ ¯ n − µ) ∼ (14) n (X = (0, σ2 ; √1n κ3 , n1 κ4 ) . Now, unbiased estimators κ ˆj of these κj are given (see K&S (p. 297, 300)) by (15)

κ ˆ2 ≡

n n−1

m ˆ 2, κ ˆ3 ≡

n2 (n−1)(n−2)

m ˆ 3,

2

n {(n + 1)m ˆ 4 − 3 (n − 1)m ˆ 22 }, κ ˆ 4 ≡ (n−1)(n−2)(n−3) ¯ n )j /n. We will combine these facts with the theorems of where m ˆ j ≡ n1 (Xj − X √ ¯ the previous sections to approximate the distribution of n(X n − µ). Additionally (by K&S (p. 306-307), √ 2 3 2n 12 8n (16) n (Sn2 − σ 2 ) ∼ κ22 ; n1 κ6 + n−1 κ4 κ2 + 4(n−2) = (0, κ4 + n−1 (n−1)2 κ3 + (n−1)2 κ2 ) ,

where correcting for skewness in (16) should probably be ignored. An unbiased estimator of the variance in (16) (unbiasedness veriﬁable from K&S (p. 296)) is (17)

n−1 n+1

n [ˆ κ4 + 2 n−1 κ ˆ 22 ] .

(b) Finally (by K&S (p. 327)), (18)

Corr[ˆ κ1 , κ ˆ 2 ] = γ1 /{γ2 +

2n 1/2 . n−1 }

(c) In approximating the bootstrap distribution of Xn∗ , it is exactly true that √ ¯n) ∼ (19) n ( Xn∗ − X ˆ 2 ; √1n κ ˆ 3 , n1 κ ˆ4 ) . = (0, κ (d) Now consider a ﬁnite population X1 , . . . , XN whose second, third, and fourth ˆj true cumulants Kj are given by (15), with N replacing n. Unbiased estimators K are also given by (15), now with n again (see K&S (p. 320)). It is also true (by K&S (p. 321-322)) that √ 2 +2n2 ¯n − X ¯N ) ∼ (20) n (X K3 , = 0, NN−n K2 ; N −3nN n2 N 2 {α3 − 4 α2 /N + 6 α1 /N 2 −

3(N −1) N (N +1)

2 −1 2 α21 } K4 + 3 N N +1 α1 K2 ,

where αr ≡ (n−r − N −r ). Finally (by K&S (p. 323)), √ 2 ∼ n (21) n (Sn2 − SN ) = (0, K4 + 2 n−1 K22 ) . Then (by K&S (p. 323)) an unbiased estimator of this last variance is given by (17) ˆ j replacing κ ˆ j ). (Though straightforward, the results cited from K&S are (with K somewhat cumbersome.) 2

4.

GAMMA APPROXIMATION

389

A noncentral chisquare rv χ2n (δ) satisﬁes n [χ2n (δ) − (n + δ)] ∼ 1 23/2 (1 + 3 δ/n) ∼ T ≡ Xk = 0, 1; µ3 ≡ √ . =√ n 1 n (1 + 2 δ/n)3/2 2(n + 2δ)

Example 4.2 (Hall) (22)

So we approximate this distribution by Gr with r ≡ (n + 2 δ)3 /(2(n + 3 δ)2 ). Then . (23) P (χ2n (δ) ≤ x) = P (Gamma(r) ≤ r + [x − (n + δ)](n + 2 δ)/2(n + 3 δ)) (24)

= (1 − α)

if x ≡ (n + δ) + (γ α − r)2(n + 3 δ)/(n + 2 δ) ,

where γ α denotes the upper 1 − α percentage point of Gamma(r). This is easy to implement in Splus, for example. (Hall found that the accuracy seemed quite good, especially in relation to previous proposals.) 2 Exercise 4.4 (Poisson approximation of the generalized binomial) We suppose Xn1 , . . . , Xnn are independent rvs with Xnk ∼ = Bernoulli(pnk ). Suppose further that Ynk ∼ ) are independent for 1 = Poisson(pnk n ≤ k ≤ n. Let Pn and Qn denote the n distributions of Xn ≡ 1 Xnk and Yn ≡ 1 Ynk . Show that the total variation distance between Pn and Qn satisﬁes n (25) dT V (Pn , Qn ) ≡ sup{|Pn (A) − Qn (A)| : A ∈ B} ≤ k=1 p2nk . If pnk = λk /n for 1 ≤ k ≤ n, then the bound becomes λ2 /n. [Hint. The ﬁrst step is to replace the original Bernoulli(pnk ) rvs by diﬀerent Bernoulli(pnk ) rvs, to be denoted by Xnk also. To this end we now deﬁne the new Znk ∼ = Bernoulli(1 − (1 − pnk )epnk ) rvs that are independent for 1 ≤ k ≤ n (and they are also independent of the Ynk ’s). Now deﬁne (26)

Xnk ≡ 1[Ynk ≥1] + 1[Ynk =0] 1[Znk =1] ,

and verify that it is indeed a Bernoulli(pnk ) rv. (This choice of the jointly distributed pair (Xnk , Ynk ) maximizes the mass on the diagonal x = y of an (x, y)-coordinate system.) Now verify that n n dT V (Pn , Qn ) ≤ P (Xn = Yn ) ≤ 1 P (Xnk = Ynk ) ≤ 1 p2nk . (27) This type of proof is called a coupling proof, in that the (Xnk , Ynk ) pairs are coupled together as closely as possible.] Exercise 4.5 (a)–(d) Derive the claims made in example 4.1(a)–(d).

390

5

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

Edgeworth Expansions

The Setup Let F0 , f0 , and φ0 denote the N (0, 1) df, density, and chf. Thus (1)

f0 (x) ≡ e−x

2

/2

√ / 2π

and

φ0 (t) ≡ e−t

2

/2

on the real line. These are related via the inversion formula for chfs as ∞ (2) f0 (x) = −∞ e−itxφ0 (t) dt/(2π). Diﬀerentiating f0 gives

f0 (x) = −x f0 (x),

f (x) = (x2 − 1) f0 (x),

(3)

f (iv) (x) = (x4 − 6x2 + 3) f0 (x), f0 (x) = −(x3 − 3x) f0 (x), f (v) (x) = −(x5 − 10x3 + 15x) f0 (x), f (vi) (x) = (x6 − 15x4 + 45x2 − 15) f0 (x);

and in general, (4)

f (k) (x) = (−1)k Hk (x)f0 (x)

deﬁnes the kth Hermite orthogonal polynomial Hk . Equating the derivatives in (3) to derivatives of the right-hand side of (2) gives ∞ (5) (−1)k Hk (x)f0 (x) = −∞ e−itx (−it)k φ0 (t) dt/(2π), which expresses Hk f0 as the inverse Fourier transform of (it)k φ0 (t). This gives the key result that (6)

(it)k φ0 (t)

is the Fourier transform of

Hk (·) f0 (·).

Now suppose that X1 , . . . , Xn are iid where X∼ has chf φ(·). = (0, σ2 ) n We let Sn ≡ k=1 Xk , and agree that (7)

(8)

Fn (·)

denotes the df of

√ Sn /(σ n ).

The idea is to expand Fn in terms of the orthogonal polynomials Hk . However, we choose instead to obtain a ﬁrst-order or second-order approximation, together with an error analysis. Also, √ (9) fn (·) denotes the density of Sn /(σ n ), if it exists. In this latter case we will also seek to expand fn . The expansions we will derive for fn and Fn are known as Edgeworth expansions.

5.

EDGEWORTH EXPANSIONS

391

Edgeworth Expansions for Densities Instead of just assuming that fn exists, we assume instead that the chf φ of the rv X∼ = (0, σ2 ) satisﬁes ∞ m (10) for some positive integer m. −∞ |φ(t)| dt < ∞, This guarantees both that fn exists for all n ≥ m, and that it can be found from the Fourier inversion formula (13.2.9). Theorem 5.1 Suppose condition (10) holds. Let γ1 ≡ E(X/σ)3 denote the skewness, and let γ2 ≡ E(X/σ)4 − 3 denote the tail heaviness of X ∼ = F (0, σ2 ). (a) Then = = = = =fn (·) − f0 (·) {1 + γ√1 H3 (·)}= = = 3! n (11)

√ = o(1/ n )

(as a function of E|X/σ|3 and FX/σ ) [if E|X|3 < ∞]

or (12)

= O(1/n)

(as a function of E|X/σ|4 and FX/σ ) [if EX 4 < ∞].

(b) Moreover, = = 2 = = =fn (·) − f0 (·) 1 + √1 γ1 H3 (·) + 1 γ2 H4 (·) + γ1 H6 (·) = = = n 4! 2(3!)2 n 3! (13)

= o(1/n)

(as a function of E|X/σ|4 and FX/σ ) [if EX 4 < ∞]

or (14)

= O(1/n3/2 ) (as a function of E|X/σ|5 and FX/σ ) [if E|X|5 < ∞].

We speciﬁcally write out that

(15)

H1 (x) ≡ x, H3 (x) ≡ x3 − 3x, H5 (x) ≡ x5 − 10x3 + 15x,

H2 (x) ≡ x2 − 1, H4 (x) ≡ x4 − 6x2 + 3, H6 (x) ≡ x6 − 15x4 + 45x2 − 15,

for use in the current set of theorems. The previous theorem was for densities. The next is for dfs. Condition (10) is used to control the extreme tails in the Fourier inversion formulas for densities. In proving analogues of (11) and (12) for dfs, we will be able to use Esseen’s lemma to control these tails instead. However, the analogues of (13) and (14) run into other problems, and these are again overcome via a (now weaker) restriction on φ. All proofs are at the end of this section. Exercise 5.1

Determine H7 (·).

392

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

Edgeworth Expansions for Distribution Functions Consider the potential hypothesis (16)

lim sup |φ(t)| < 1. |t|→∞

(This is weaker than (10). The Riemann–Lebesgue lemma shows that (16) holds if F has an absolutely continuous component, a` la theorem 7.1.1.) Theorem 5.2 Suppose that X is not distributed on a grid. Let γ1 ≡ E(X/σ)3 denote the skewness, and let γ2 ≡ E(X/σ)4 − 3 denote the tail heaviness of the rv X∼ = F (0, σ2 ). (a) Then = = = = γ1 =Fn (·) − F0 (·) + f0 (·) √ H2 (·) = = = 3! n (17)

√ = o(1/ n ) or

(18)

= O(1/n)

(as a function of E|X/σ|3 and FX/σ ) [if E|X|3 < ∞] (additionally requiring (16) for (18)) (as a function of E|X/σ|4 and FX/σ ) [if EX 4 < ∞].

(b) Moreover, when (16) holds, = = 2 = = =Fn (·) − F0 (·) + f0 (·) √1 γ1 H2 (·) + 1 γ2 H3 (·) + γ1 H5 (·) = = = n 3! n 4! 2 · (3!)2 (19)

= o(1/n)

(as a function of E|X/σ|4 and FX/σ ) [if EX 4 < ∞]

or (20)

= O(1/n3/2 ) (as a function of E|X/σ|5 and FX/σ ) [if E|X|5 < ∞].

√ Exercise 5.2 Let Zn ≡ Sn /σ n as above, and let fn denote its density under √ condition (10). Let Zn∗ ≡ [Gamma(r) − r]/ r, with r ≡ 4n/γ12 , and let gn (·) denote its density. Show that fn − gn = O(1/n) as a function of E|X/σ|4 and FX/σ when E|X|4 < ∞.

The Proofs We deﬁned and expanded the cumulant generating function ψ(·) ≡ Log φ(·) in exercise 13.4.6. The ﬁrst few cumulants of the standardized rv X/σ were seen to be 0, 1, γ1 , and γ2 . Proof. Consider theorem 5.1(a). Without loss of generality we suppose that σ = 1. We now agree that 1 γ1 (a) dn ≡ fn − 1 + √ H3 f0 n 3!

5.

EDGEWORTH EXPANSIONS

393

denotes the diﬀerence between the true fn and our ﬁrst approximation to it. Note from (6) that dn has Fourier transform √ 1 γ1 (21) (it)3 φn (t) ≡ [φ(t/ n )]n − φ0 (t) 1 + √ n 3! √ /2+nψ(t/ n )

(b)

= e−t

/2

{et

(c)

≡ e−t

/2

[ez+ − (1 + z)].

2

2

2

√ − [1 + (γ1 /3!)(it)3 / n ]}

Thus the Fourier inversion formula (13.2.9) gives ∞ (d) dn (x) = −∞ eitx φn (t) dt/(2π), ∞ since (10) implies that −∞ |φn (t)| dt < ∞. Thus for any ﬁxed θ > 0 and all x > 0 we have ∞ ∞ 2 |dn (x)| ≤ −∞ |φn (t)| dt = −∞ e−t /2 |ez+ − (1 + z)| dt (e)

=

√ [|t|≤θ n/E|X|3 ]

e−t

2

/2

|ez+ − (1 + z)| dt + o(n−r )

for any r > 0, as in (14.3.6). Now, (f)

|φn (t)| = e−t

2

√ n )−(−t2 /2)

|enψ(t/

≡ e−t

/2

|ez+ − (1 + z)| = e−t

≤ e−t

/2

[| | e|| e|z| + z 2 e|z| /2]

2

(g)

/2

2

2

√ − [1 + (γ1 /3!)(it)3 / n ]|

/2

|ez (e − 1) + (ez − (1 + z))|

√ using (13.4.3). Note that for all |t| ≤ θ n/E|X|3 we have √ (h) |z| ≤ (|t|3 /6) E|X|3 / n ≤ θt2 /6 ≤ t2 /8 if θ ≤ 34 , √ √ | | = |nψ(t/ n ) − [−t2 /2 + (γ1 /6) (it)3 / n ]| (i)

√ ≤ c3 |t|3 E|X|3 δ(θ)/ n ≤ c3 t2 θ δ(θ) ≤ t2 /8

if θ is small enough,

where δ(·) denotes the function δ3 (·) function of (13.4.21) associated with the rv X/σ. Using (h) and (i), the bound in (e) becomes (for some θ small enough) 2 |dn (x)| ≤ [|t|≤θ √n/E|X|3 ] e−t /2 {| | e|| e|z| + z 2 e|z| /2} dt + o(n−r ) (j)

√ ∞ 2 ≤ c3 δ(θ) [E|X|3 / n ] −∞ |t|3 e−t /4 dt +[(E|X|3 )2 /(72n)]

(k)

= o(n−1/2 )

∞ −∞

|t|6 e−3t

uniformly in x,

2

/8

dt + o(n−r )

394

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

since a tiny δ(θ) results from a suﬃciently tiny θ. Thus (11) holds. For (12), we replace the bound in line (i) above by √ (l) | | ≤ c¯4 t4 EX 4 /n ≤ t2 /8, which is valid for |t| ≤ n/ 8 c¯4 E|X|4 , as (13.4.22) guarantees. We then use (l) instead of (i) during (j) (now integrated √ over the interval [|t| ≤ n/ 8/¯ c4 EX 4 ]). We now turn to (13), and then (14). We ﬁrst redeﬁne 1 γ2 γ12 1 γ1 H (m) H3 + H4 + . dn ≡ fn − f0 1 + √ 6 n 4! 2 · (3!)2 n 3! Taking the inverse of its Fourier transform φn (·) gives (as in (e)) that for any ﬁxed value of θ > 0 and all x, ∞ |dn (x)| ≤ −∞ |φn (t)| dt √ 2 2 (n) = [|t|≤θ √n/EX 4 ] e−t /2 |et /2+nψ(t/ n ) − {1 + z + z 2 /2}| dt + o(n−r ) 2 = [|t|≤θ √n/EX 4 ] e−t /2 |ez+ − (1 + z + z 2 /2)| dt + o(n−r ) (o) for each ﬁxed r > 0, with 1 γ2 1 γ1 (p) z≡√ (it)3 + (it)4 . n 3! n 4! The ﬁnal details are nearly the same as before. Exercise 5.3 Proof. (q)

2

Finish the details of the previous proof of theorem 5.1(b).

Consider theorem 5.2(a). We note that 1 γ 1 γ1 1 Dn ≡ Fn − F0 + f0 √ H2 H3 , has Dn = dn ≡ fn − f0 1 + √ n 3! n 3!

where dn is as in line (a) of the previous proof (just use x H2 (x) − 2x = H3 (x) to verify this). Esseen’s lemma then gives √ 1 |φn (t)| 24 f0 [1 + (γ1 /3!)H3 / n ] √ (r) Dn ≤ dt + , π [|t|≤a√n/E|X|3 ] |t| π a n/E|X|3 where φn is the same φn appearing in (21). Since the norm in the√second term on the right of (r) is bounded, the second term in (r) is less that / n whenever a ≡ a( , FX/σ ) chosen large enough. Fix this a in the limits of integration √ of (r), and then break this integral into two pieces: the integral over [|t| ≤ θ n/E|X|3 ] √ 3 with θ as in√(i), and the integral over [|t| > θ n/E|X| ]. The integral over the set [|t| > θ n/E|X|3 ] is o(n−r ), for any r > 0 (` a la (14.4.6), as before at line √ (e)). Finally, the value of the integral over the set [|t| ≤ θ n/E|X|3 ] is bounded by a term like the right-hand side of (j) (in which |t|3 and t6 are replaced in those integrals by t2 and |t|5 , to account for division by |t| in the integrand of (r)). This completes the proof of (17) when X is not distributed on a grid. For (18), the initial region of integration in (r) must be [|t| ≤ an/EX 4 ], and then an/EX 4 will also appear below the norm term. Moreover, we will now use θ for a, since only O(1/n) is required.

5.

EDGEWORTH EXPANSIONS

395

Consider theorem 14.5.2(b). We note that 1 γ1 1 γ2 γ12 √ H2 (·) + H3 (·) + Dn (·) ≡ Fn (·)−F0 (·)+f0 (·) (s) H5 (·) n 3! n 4! 2 · (3!)2 has derivative Dn (x) = dn (t)

(u)

= fn (x) − f0 (x) + f0 (x)

1 γ1 √ [2x − xH2 (x)] n 3!

1 γ2 γ12 2 4 2 + f0 (x) [3(x − 1) − xH3 (x)] + [5(x − 6x − xH5 (x)] n 4! 2(3!)2 1 γ1 1 γ2 γ12 √ = fn (x)−f0 (x)+f0 (x) H3 (x) + H4 (x) + H6 (x) , n 4! 2(3!)2 n 3!

and this is the same dn as in (m) of the previous proof. Thus the ﬁnal details are nearly the same as before. 2

Exercise 5.4

Complete the details in the previous proof of theorem 5.2(b).

Exercise 5.5 Consider a non-iid case in which all dfs Fnk have third and/or fourth moments that are of the same order. Then all of the previous results still obtain. Exercise 5.6 (Large deviations) Suppose the moment generating function (or mgf ) MX (t) ≡ EetX of the rv X √is ﬁnite for 0 ≤ |t| < . Let X1 , X2 , . . . be iid ¯ n − µ) and let F0 (·) denote the N (0, 1) df. (0, σ2 ). Let Fn (·) denote the df of n(X Show that (22)

[1 − Fn (xn )]/[1 − F0 (xn )] → 1,

provided that

xn = o(n1/6 ).

396

6

CHAPTER 14. CLTS VIA CHARACTERISTIC FUNCTIONS

¯n) Approximating the Distribution of h(X

Example 6.1 Suppose the basic rv X satisﬁes X ∼ = (µ, σ2 ), with third and fourth central moments µ3 and µ4 . Let X1 , X2 , . . . denote iid repetitions. We consider the ¯ n ), for some h(·) satisfying distribution of h(X (1)

h(x) = h(µ) + h (µ) (x − µ) + 12 h (µ) (x − µ)2 +

1 6

h (µ) (x − µ)3

1 + 24 h (µ) (x − µ)4 + · · ·

(2)

≡ c0 + c1 (x − µ) + c2 (x − µ)2 + c3 (x − µ)3 + c4 (x − µ)4 + · · · .

Then we have the following approximations: ¯ n ) = c0 + 1 1 c2 σ2 + 12 [ 1 c3 µ3 + Eh(X (3) n 2

(4)

n

6

¯ n ) − Eh(X ¯ n ) = c1 (X ¯ n − µ) + h(X

1 2

1 8

c4 σ4 ] + · · · .

¯ n − µ)2 − c2 [(X

¯ n − µ)3 − + 16 c3 [(X

1 n2

1 n

σ2 ]

µ3 ] + · · · .

(5)

¯ n )] = Var[h(X

c21 σ2 +

1 n2

[c1 c2 µ3 + ( 12 c22 + c1 c3 ) σ4 ] + · · · .

(6)

¯ n ) − Eh(X ¯ n )]3 = E[h(X

1 n2

[c31 µ3 + 3 c21 c2 σ4 ] + · · · .

(7)

¯ n ) − Eh(X ¯ n )]4 − 3 Var[h(X ¯ n )] = E[h(X

1 n

0 n2

+··· .

We call h(·) a normalizing transformation if the lead term on the right of (6) equals 0. We call h(·)√a variance stabilizing transformation if the variance of the limiting ¯ n ) − h(µ)] does not depend on the parameter in some family distribution of n [h(X being used to model the distribution of X. (See Bickel and Doksum (1977) and Kendall and Stuart(1977).) 2 Proof. (8)

The ﬁrst two claims are immediate from exercise 11.6.2: ¯ n − µ)3 = 12 µ3 . E(X

(9)

¯ n − µ)4 = E(X

n

1 n3

[E(X − µ)4 + 3(n − 1) σ 4 ] =

1 n2

3 σ4 +

1 n3 (µ4

− 3σ 4 ) .

Squaring the equation in (4) gives ¯ n − µ)2 + c1 c2 [E(X ¯ n − µ)3 − 0] ¯ n )] = c2 E(X Var[h(X 1 (a) ¯ n − µ)2 − 1 σ2 ]2 + 1 c1 c3 [E(X ¯ n − µ)4 − 0] + · · · + 41 c22 E[(X n 3 (b)

=

1 n

c21 σ2 + c1 c2

(c)

=

1 n

c21 σ2 +

1 n2

1 n2

µ3 +

[c1 c2 µ3 +

1 4 1 2

c22 [ n12 3 σ 4 −

(e)

= c31

1 n2

µ3 + 3 c21

1 2

σ4 ] +

1 3

c1 c3 [ n12 3 σ 4 ] + · · ·

c22 σ4 + c1 c3 σ4 ] + · · · .

Cubing the equation in the second claim gives ¯ n )−Eh(X ¯ n )]3 = c3 E(X ¯n −µ)3 +3 c2 (d) E[h(X 1

1 n2

1 1 2

¯ n −µ)4 − 1 σ2 1 σ2 ]+· · · c2 [E(X n n

c2 [ n12 3 σ 4 − n12 σ4 ] + · · · =

1 n2

[c31 µ3 + 3 c21 c2 σ4 ] + · · · .

[Clearly, rigorous bounds via Taylor series and (13.4.8) would allow the error terms above to be declared to be o(·) of a bound on the last indicated term.] 2

6.

¯N ) APPROXIMATING THE DISTRIBUTION OF H(X

Example 6.2 (Wilson–Hilferty approximation) (10)

{Vn1/3 − (1 −

2 9n )}/

2 9n

397

Let Vn ∼ = Chisquaren /n. Then

is very nearly N (0, 1) .

This follows from the example above with X ≡ Chisquare1 , µ = 1, σ2 = 2, µ3 = 8, h(x) = x1/3 , c0 = 1, c1 = 1/3, c2 = −2/9, c3 = 10/27, c4 = −80/81. Then ¯n) = 1 − Eh(X

2 9n

+

0 n2

+···,

¯ n )] = Var[h(X

2 9n

+

0 n2

+ ···,

(11) ¯ n ) − Eh(X ¯ n )]3 = E[h(X

0 n2

+··· .

This approximation is known to work very well, and the calculations of (11) should give some insight into this. There is another way to look at this. Consider the family of transformations h(x) = xγ , for some γ. We wish to choose γ so that ¯ n ) is 0 up to terms of order 1/n2 . Now, h (x) = γ xγ−1 and the skewness of h(X γ−2 h (x) = γ(γ − 1) x , while µ = 1, σ2 = 2, and µ3 = 8. Thus the lead term is h (µ)3 µ3 + 3 [h (µ)]2 h (µ) σ4 = γ 3 · 8 + 3 γ 2 · γ(γ − 1) · 4 = 4 γ 3 [2 + 3 (γ − 1)] , which equals 0 if γ = 13 .

2

Example 6.3 (Normalizing transformation for the Poisson) Suppose that the rv ¯n X ∼ = Poisson(λ). Let X1 , X2 , . . . be iid Poisson(λ). What transformation h of X 2 has skewness 0 for the term of order 1/n ? To answer this, suppose that h(x) = xγ , so that h (x) = γ xγ−1 and h (x) = γ(γ − 1) xγ−2 . Since µ = λ, σ2 = λ, and µ3 = λ, the coeﬃcient of 1/n2 in the expression for the third central moment of the ¯ n ) under consideraton is rv h(X h (µ)3 µ3 + 3 [h (µ)]2 h (µ) σ4 = [γ λγ−1 ]3 · λ + 3 [γ λγ−1 ]2 · γ(γ − 1) λγ−2 · λ2 = γ 3 λ3 γ−2 [1 + 3(γ − 1)] , and this equals 0 if γ = 23 . Note that the resulting transformation h(x) = x2/3 diﬀers from the variance stabilizing transformation h(x) = x1/2 . 2 Exercise 6.1 Powers of x do not always suﬃce. (a) Consider the binomial, which √ is a sum of iid Bernoulli(p) rvs. Show that the transformation h(x) = arcsin( x ) is variance stabilizing, while the transformation x h(x) = 0 [t(1 − t)]−2/3 dt is a normalizing transformation. (b) For the sample correlation coeﬃcient based on bivariate normal observations, show that Fisher’s transformation h(x) = 12 log((1 + x)/(1 − x)) is both variance stabilizing and normalizing.

Chapter 15

Inﬁnitely Divisible and Stable Distributions 1

Inﬁnitely Divisible Distributions

Deﬁnition 1.1 (Triangular arrays, and the uan condition) A triangular array is just a collection of rvs Xn1 , . . . , Xnn , n ≥ 1, such that the rvs in the nth row are independent. Call it a uan array if the uniform asymptotic negligibility condition holds, that is (1)

max P ( |Xnk | ≥ ) → 0

1≤k≤n

as n → ∞ for all > 0.

The uan condition is a natural one for preventing one term from dominating the whole sum. The Problem: Let Sn ≡ Xn1 + · · · + Xnn denote the nth row sum of a uan array. (i) Find the family of all possible limit laws of Sn . (ii) Find conditions for convergence to a speciﬁed law of this form. Find specialized results for further restrictions on the uan array. (a) Suppose variances exist. (b) Suppose the limit law is normal or Poisson. (c) Consider Sn = [(X1 +· · ·+Xn )−Bn ]/An for a singly subscripted sequence of iid rvs X1 , . . . , Xn , . . . . Some of the results in this chapter are stated with only indications of the proofs. The goal in this chapter is simply to develop some rough understanding of the subject. We will see in this section that the set of all possible limit laws of row sums Sn ≡ Xn1 + · · · + Xnn of a uan array of Xnk ’s is exactly the class of inﬁnitely divisible laws, which we now deﬁne. 399

400 CHAPTER 15. INFINITELY DIVISIBLE AND STABLE DISTRIBUTIONS Deﬁnition 1.2 (Inﬁnitely divisible) Call both the rv Y and its distribution inﬁnitely divisible (id ) if for every value of n it is possible to decompose Y into n iid components as (2)

Y ∼ = Yn1 + · · · + Ynn

for some iid rvs Yn1 , · · · , Ynn .

We denote the class of all id distributions by I; the subclass with ﬁnite variance is denoted by I2 . (We remark that the Yni ’s of this deﬁnition form a uan array, but this needs to be shown.) Exercise 1.1 (Chf expansions for the uan array Xn1 + · · · + Xnn ) uan array of rvs Xnk , as in (1). (a) Let Fnk and φnk denote the df and the chf of Xnk . Show that (3)

[max1≤k≤n |φnk (t) − 1|] → 0

Consider a

uniformly on every ﬁnite interval.

[Hint. Integrate over |x| < and |x| ≥ separately to obtain |φnk (t) − 1| ≤ δ + 2P (|Xnk | ≥ ), from which point the result is minor.] We then deﬁne (as will be useful regarding an expansion of Log (1 + (φnk (·) − 1)) below) n (4)

n (t) ≡ k=1 |φnk (t) − 1|2 . n 2 2 ) , and σn2 ≡ 1 σnk ≤M <∞ (b) Verify the elementary fact that if Xnk ∼ = (0, σnk 2 with [max1≤k≤n σnk ] → 0, then n (t) → 0 uniformly on each ﬁnite interval. Exercise 1.2 (Chf expansions for the uan array Y ∼ = Yn1 + · · · + Ynn ) (a) Let φ and φn denote the chf of Y and of the Ynk ’s, respectively, in (2). Show that these Yn1 , . . . , Ynn form a uan array, and that Log φn = (1/n) Log φ. (Recall lemma 13.4.1 about the principal branch of Log (·).) (b) If φ is id, then Log φ(t) = 0 for any t. (c) If Ym →d Y where the rvs Ym are id, then Y is id. Motivation 1.1 (Limits of uan arrays) Suppose that Sn ≡ Xn1 + · · · + Xnn for some uan array. Let Fnk and φnk denote the df and chf of the rv Xnk . Then for the function n (·) of (4) we necessarily have for n suﬃciently large (recall that a = b ⊕ c means that |a − b| ≤ c) that Log φSn (t) = n1 Log φnk (t) = n1 Log (1 + (φnk (t) − 1)) n = 1 [φnk (t) − 1] ⊕ n (t) with n (·) as in (4) n ∞ (5) = k=1 −∞ (eitx − 1) dFnk (x) ⊕ n (t). If we further assume that all the Xnk ’s have 0 means and ﬁnite variances, then we can rewrite (5) to obtain n ∞ itx 1 e − 1 − itx 0 2 Log φSn (t) = x dFnk (x) ⊕ n (t) . 2 x −∞ k=1

1.

INFINITELY DIVISIBLE DISTRIBUTIONS

401

Thus (6)

∞ n itx Log φSn (t) = −∞ { e −1−itx } [ x2 1 dFnk (x) ] ⊕ n (t) x2 ∞ ≡ −∞ { φ(x, t) } [ dKn (x) ] ⊕ n (t) .

Observe additionally that (7)

φ(x, t) is continuous on the (x, t)-plane with each φ(·, t) bounded and continuous, and equal to − t2 /2 at x = 0 .

Moreover, take the point of view that x (8) Kn (x) = −∞ y 2 d nk=1 Fnk (y) = (the contribution to variance up to x). It is natural to hope that Sn will converge in distribution to a rv Y whose Log chf is of the form ∞ Log φY (t) ≡ −∞ φ(x, t) dK(t) , (9) provided that Kn →d K. When means and variances need not exist, we deﬁne (10)

α(x) ≡ x2 ∧ 1

and

β(x) ≡ (|x| ∧ 1) sign(x).

We then note that (5) can also be manipulated to give " ! ∞ itx n (e − 1 − itβ(x)) Fnk (x) Log φSn (t) = α(x) d α(x) −∞ k=1

+ it [ (11)

≡

∞

∞ −∞

−∞ φ(x, t) dHn (x)

β(x)d nk=1 Fnk (x)] ⊕ n (x) + it βn ⊕ n (x),

where this new φ(x, t) still satisﬁes (7), and where we now deﬁne x n . (12) Hn (x) = −∞ α(u) d k=1 Fnk (u) = (contribution to variance up to x), at least for x near 0, which is where all the action is in any uan array. It is natural to hope that Sn will now converge in distribution to a rv Y whose chf is of the form ∞ (13) Log φY (t) ≡ itβ + −∞ φ(x, t) dH(x), provided that Hn →d H. We are thus particularly interested in the behavior of Kn and/or Hn (and n (·)), both for the general uan Xnk ’s of (1) and for the special uan Ynk ’s of (2). 2 The next example enables us to show that any chf of the form (9) or (13) is the chf of some id distribution. The details are left to the easy exercise 1.3.

402 CHAPTER 15. INFINITELY DIVISIBLE AND STABLE DISTRIBUTIONS Example 1.1 (Generalized Poisson and compound Poisson distributions) suppose that the rvs Xn1 , . . . , Xnn are iid with aj with probability pnj , for 1 ≤ j ≤ J, Xnk = 0 otherwise,

We

where npnj → λj ∈ (0, ∞) as n → ∞. Then J J (14) φSn (t) → φY (t) ≡ exp( j=1 λj (eitaj − 1)) = j=1 exp(λj (eitaj − 1)). J Thus the limiting distribution is that of Y ≡ j=1 aj Yj for independent Poisson(λj ) rvs Yj . This is called the generalized Poisson distribution. Note also that the chf on the right-hand side of (14) satisﬁes J (15) φY (t) = exp(λ j=1 pj (eitaj − 1)) = exp(λ(φW (t) − 1)) = Eeit(W1 +···+WN ) , J where λ ≡ j=1 λj , pj ≡ P (W = aj ) = λj /λ for 1 ≤ j ≤ J, φW is the chf of W , and W1 , W2 , . . . are iid as W and N ∼ = Poisson(λ). The distribution of the rv Y is called the compound Poisson distribution, and is distributed as a Poisson sum of independent Multinomial(k; λ1 /λ, . . . , λk /λ) rvs. The compound Poisson distribution of (15) is obviously id, as is clearly seen by using λ/n in place of λ in (15) for the iid Ynk ’s of (2). And thus the generalized Poisson distribution in (14) is also id. It is in the compound Poisson format that we recognize that this distribution is id, but it will be in its generalized Poisson format that we will put it to work for us. 2 Exercise 1.3 Our heuristics have suggested that if Y is id, then its chf φ is of the form (13) (the reader should also note (29), where the normal component of the limit is removed). They have not yet suggested the converse. However, ∞ mn iβt + −∞ φ(x, t) dν(x) ← j=1 {[eitxj − 1 − itβ(xj )]/α(xj )}ν(Ij ) + iβt n itxj = m (16) − 1) λj + itβn j=1 (e n with λj ≡ [ν(Ij )/α(xj )] and βn ≡ β − m j=1 ν(Ij )β(xj )/α(xj ) and appropriate intervals Ij (with xj , Ij and λj all depending on n) → (a limit of generalized Poissons) (17)

= (a limit of id rvs) = id,

since the limit under →d of id rvs is also an id rv (as was stated in exercise 1.2(c)). [The present exercise is to make all this rigorous.] Theorem 1.1 (Kolmogorov’s representation theorem) Let the rv Y have chf φ and ﬁnite variance. We will use the symbol K to denote a generalized df with 0 = K(−∞) < K(+∞) < ∞. Now, Y is id if and only if ∞ (18) Log φ(t) = iβt+ φ(x, t) dK(x) for some such gdf K and some real β, −∞

with φ(x, t) as in (6) (and (7)). The representation is unique. Moreover, it holds that β = EY and K(+∞) = Var[Y ] .

1.

INFINITELY DIVISIBLE DISTRIBUTIONS

403

Theorem 1.2 (Bounded variances limit theorem for I2 ) (a) We start with a 2 triangular array of row-independent rvs Xnk ∼ ) having = (µnk , σnk n 2 2 (19) →0 and σn2 ≡ k=1 σnk ≤M <∞ max σnk 1≤k≤n

(such a triangular array is necessarily uan). Then (20)

Sn ≡

n

Xnk →d Y,

k=1

where the limiting rv necessarily satisﬁes ∞ itx ∞ e − 1 − itx (21) φ(x, t) dK(t) Log φY (t) = iβt + dK(x) = itβ + x2 −∞ −∞ with K(+∞) ≤ lim sup σn2 , if and only if (22)

µn ≡

n k=1

µnk → β

and

Kn (·) ≡

n k=1

·

−∞

y 2 dFnk (y + µnk ) →sd K(·).

If Kn (∞) = ≤ M < ∞ is replaced by Kn (+∞) = σn2 → Var[Y ] < ∞, then we can claim that Var[Y ] = K(+∞) = lim σn2 , and we will write Kn →d K. (b) The family of all possible limit laws of such Sn is the family I2 of all possible inﬁnitely divisible laws that have ﬁnite variance. σn2

Proof. (If one grants exercises 1.1 and 1.3, then the proof we give will be complete. These exercises are straightforward.) Any chf of the form (18) or (21) is id, by exercise ∞ 1.3. Diﬀerentiating twice in (21) shows that this chf has mean β and variance −∞ 1 dK = K(+∞). For the uniqueness of the representation, just diﬀerentiate (21) twice, and thus obtain −(Log φ(t)) = eitx dK(x) subject to 0 = K(−∞) < K(+∞) = Var[Y ]; then applying the ordinary inversion formula to itx e dK(x) gives K in terms of −(Log φ) . It remains to show that any id Y with mean 0 and ﬁnite variance has a chf of the form (21) with β = 0. Reconsider (6). For the special uan Ynk ’s of (2) we have Var[Ynk ] = Var[Y ]/n, so that exercise 1.1(b) implies that n (t) → 0 uniformly on all ﬁnite intervals. Moreover, the family of functions φ(·, t) in (7) are bounded and continuous functions that converge to 0 as |x| → ∞. Applying the Helly–Bray exercise 12.1.1 to each φ(·, t) in (6) shows (for the ﬁrst equality we need only recall that Y ∼ = Tn ≡ Yn1 + · · · + Ynn ) that (a) Log φY (t) = Log φTn (t) → φ(x, t) dK(x), provided that Kn →sd K. So we must show that Kn →sd K. Now, every subsequence n has a further n on which Kn →sd (some K). But every resulting such φ(x, t) dK(x) with a limiting K inserted must equal Log φY , and thus the ﬁrst paragraph of this proof implies that K has 0 = K(−∞) < K(+∞) = Var[Y ]. In fact, all possible subsequential limits K must be equal, by the uniqueness of the representation in equation (21). Thus Kn →sd K on the whole sequence n. Thus Log φY = φ(x, t) dK(t). This completes the proof of theorem 1.1.

404 CHAPTER 15. INFINITELY DIVISIBLE AND STABLE DISTRIBUTIONS We now turn to the proof of theorem 1.2. Under the basic hypothesis (19), we have in (6) that n (t) → 0 uniformly on all ﬁnite intervals (by exercise 1.1(b)). Thus whenever Kn →sd (some K) and µn → µ, we have by applying Helly–Bray to each φ(·, t) in (7) that Log φSn (t) → Log φ(t) for each t, for the φ of (21) with β = µ. Thus Sn →d Y for the id Y with chf given by (21). Suppose Sn →d Y . We argue (as in the proof of theorem 1.1) that each subsequence n has a further n on which the Kn of (6) satisﬁes Kn →sd (the same K) and µn → (the same µ). Thus Kn →sd K and µn → µ, using theorem 1.1 for this uniqueness. That is, Log φSn (t) → Log φ(t) for the φ of (21) having this K and µ. But Log φSn (t) → Log φY (t) also. Thus Log φY = Log φ. Moreover, n 2 K(+∞) = Var[Y ] ≤ lim inf Var[Sn ] = lim inf 1 σnk ≤ M (using Fatou and a Skorokhod representation for which Sn∗ →a.s. Y ∗ ). If (b)

Kn (+∞) = Var[Sn ] = σn2 → Var[Y ] = K(+∞) ,

then Kn →sd K reasonably becomes Kn →d K.

2

Example 1.2 (Normal convergence) (i) (Representation) The N (0, 1) chf φ has Log φ(t) = −t2 /2. Thus µ = 0 and K = 1[0,∞) . 2 (ii) (Lindeberg–Feller theorem) Suppose the triangular array with Xnk ∼ ) = (µnk , σnk 2 2 satisﬁes µnk = 0 and σn ≡ σnk = 1 for all n. Then 2 (23) Sn ≡ nk=1 Xnk →d N (0, 1) and [max1≤k≤n σnk ]→0

if and only if n 2 (24) k=1 [|x|≥] x dFnk (x) → 0 Exercise 1.4

for all > 0.

2

Verify example 1.2.

Example 1.3 (Poisson convergence) (i) (Representation) The Poisson(λ) chf φ has Log φ(t) = λ(eit − 1) = itλ + λ(eit − 1 − it). Thus µ = λ and K = λ1[1,∞) . 2 (ii) (Convergence) Suppose the triangular array with Xnk ∼ ) satisﬁes = (µnk , σnk n 2 2 [max1≤k≤n σnk ] → 0, and 1 σnk → λ ∈ (0, ∞). Then (25) Sn ≡ n1 Xnk →d Poisson(λ)

if and only if n (26) k=1 µnk → λ

and

n k=1

2 [|x−1|≥] x

dFnk (x + µnk ) → 0.

2

1.

INFINITELY DIVISIBLE DISTRIBUTIONS

Exercise 1.5

405

Verify example 1.3.

Exercise 1.6 (Decomposition of normal and Poisson distributions) Suppose that X∼ = X1 + X2 , where X1 and X2 are independent I2 rvs. Then: (27)

X normal implies that X1 and X2 are both normal.

(28)

X Poisson implies that X1 and X2 are both Poisson.

[This is also true if I replaces I2 .] Exercise 1.7 If φ is a chf, then exp(c(φ − 1)) is an id chf for all c > 0. Thus any chf such as the one in (15) represents the log of an id chf. (We do not make explicit use of this anywhere.) From here to the end of this section we mainly just state results, mostly by analogy. Theorem 1.3 (L´ evy–Khinchin representation theorem) Let Y have chf φ. Then Y is inﬁnitely divisible (id) if and only if ∞ ∞ 2 2 Log φ(t) = iβt + (29) φ(x, t) dH(x) = iβt − σ t /2 + φ(x, t) dν(x), −∞

−∞

where σ2 = ∆H(0) and ν ≡ H − σ2 1[0,∞) , for φ(x, t) as in (11) (and (7)). The representation is unique. (We write Y =r (β, H) to denote this representation. We will think of iβt−σ2 t2 /2 as the normal component of the limit law.) Theorem 1.4 (General limit theorem for I) triangular array. Then (30)

Sn ≡

n

Xnk =

k=1

n k=1

where necessarily (31)

(Xnk − bnk ) +

bnk →d Y,

k=1

Log φY (t) = iβt +

n

Let the rv’s Xnk form a uan

∞

−∞

[(eitx − 1 − itβ(x))/α(x)] dH(x),

if and only if for some ﬁnite-measure generalized df H and for some real β we have (32)

βn → β

where βn ≡

n

(33) Hn (·) =

Hn →d H

and

k=1

[bnk +

n k=1

∞

· −∞

−∞

β(x) dFnk (x + bnk )]

and

α(y) dFnk (y + bnk ) ,

and where bnk ≡ EXnk 1(−δ,δ) (Xnk ), with δ > 0 arbitrary but ﬁxed. The family of possible limit laws of such Sn is the family I of all possible inﬁnitely divisible distributions.

406 CHAPTER 15. INFINITELY DIVISIBLE AND STABLE DISTRIBUTIONS Theorem 1.5 (Normal limits of uan arrays) (a) Let Sn ≡ Xn1 + · · · + Xnn for iid Xnk ’s, and suppose that Sn →d S. Then (34)

Mn →p 0

if and only if

S is normal.

(b) Let Sn ≡ Xn1 + · · · + Xnn for independent Xnk ’s, and suppose Sn →d S. Then (35)

Mn →p 0

if and only if

S is normal and the Xnk ’s are uan.

(Compare this to theorem 11.2.1.) Exercise 1.8 (a) Show that all Gamma(r, ν) rvs are inﬁnitely divisible. (b) Show that an inﬁnitely divisible rv can not be concentrated on a ﬁnite interval. Exercise 1.9 Use theorem 1.4 to prove the asymptotic normality condition of theorem 11.2.2. Exercise 1.10

Use theorem 1.4 to prove theorem 1.5.

Exercise 1.11 Use chfs to determine necessary and suﬃcient conditions under which the WLLN will hold.

2.

STABLE DISTRIBUTIONS

407

2

Stable Distributions

Deﬁnition 2.1 (Domain of attraction) Let X, X1 , X2 , . . . be iid F . Suppose an An > 0 and a Bn exist such that Sn ≡ X1 + · · · + Xn satisﬁes (1)

(Sn − Bn )/An →d Y ∼ = G.

Then F is said to belong to the domain of attraction of G, and we write F ∈ D(G). We also say that G possesses a domain of attraction. [If Var[X] < ∞ above, then √ necessarily (Sn − EX)/ n →d N (0, Var[X]) by the ordinary CLT; thus the only new and interesting cases have EX 2 = ∞.] Deﬁnition 2.2 (Stable law) an > 0 and bn with (2)

Sn ∼ = an X + bn ,

Call a df G stable if for all n there exist constants

where X, X1 , . . . , Xn are iid as G.

We call G strictly stable if we may take bn = 0 in (2). Theorem 2.1 (Only stable dfs have domains of attraction) A df G will possess a domain of attraction if and only if G has a stable distribution. Moreover, the an of (2) must satisfy (3)

for some 0 < α ≤ 2.

an = n1/α

We call α the characteristic exponent of G. [To compare with section 7.9 results for R−β with β > 0, we deﬁne (4)

β ≡ (2/α) − 1

or

α = 2/(β + 1),

where 0 < α ≤ 2 and 0 ≤ β < ∞.] Deﬁnition 2.3 (Basic domain of attraction) The df F is said to belong to the basic domain of attraction of G (or the domain of normal attraction of G), which is denoted by writing F ∈ DN (G), provided An = (constant) × n1/α works in (1). Remark 2.1 (a) Suppose that G1 and G2 are both of the same type (in the sense of deﬁnition 11.7.4). Then: D(G1 ) = D(G2 )

and

DN (G1 ) = DN (G2 ),

(5) G ∈ D(G)

if G is stable.

(Thus there is a domain of attraction for the normal type, but this is not so for a particular normal df.) 2 ∼ an X + bn , or (Sn − bn )/an ∼ Proof. Suppose G is stable. Then Sn = =X∼ =G for all n. Thus G ∈ D(G). Suppose G possesses a domain of attraction. Thus there exists X1 , X2 , . . . iid F where F ∈ D(G). Hence for some An > 0 and some Bn we have (a)

Tn ≡ (X1 + · · · + Xn − Bn )/An →d Y ∼ = G.

408 CHAPTER 15. INFINITELY DIVISIBLE AND STABLE DISTRIBUTIONS Replace n by nk in (a) and obtain Tnk =

(Xn(k−1)+1 + · · · + Xnk ) − Bn An (X1 + · · · + Xn ) − Bn = +···+ An An Ank

(b)

X1 + · · · + Xnk − Bnk Ank

−(Bnk − k Bn )/ Ank , which can be rewritten in the more useful format Ank Bnk − k Bn Tnk + An An Xn(k−1)+1 + · · · + Xnk − Bn X1 + · · · + Xn − Bn +··· + = An An (c)

→ Y1 + · · · + Yk

for Y1 , . . . , Yk iid as G.

Let k be ﬁxed. Recall that the convergence of types theorem states that if (d)

Tnk →d Y (true, from (a))

and also

ank Tnk + bnk →d Y1 + · · · + Yk

(true, from (c)),

with ank ≡ Ank /An and bnk ≡ (Bnk − k Bn )/An , then it must be that (e)

ank → (some ak ) ∈ (0, ∞),

(f)

Y1 + · · · + Yk ∼ = ak Y + bk .

bnk → (some bk ) ∈ (−∞, ∞),

where

From (f), we see that G is stable. This completes the proof of the ﬁrst statement. Now we further exploit equation (e). From it we determine that (g)

amk = lim

n→∞

Anmk Anmk Anm = lim = ak am n→∞ An Anm An

for all m, k ≥ 1.

We now let Z ≡ Y − Y and Zn ≡ Yn − Yn , where Y, Y , Y1 , Y1 , . . . are iid as G. Then the rvs Z and Zn are symmetric, and (1) shows that (h)

Z 1 + · · · + Zn ∼ = an Z

for the same an ’s as in (2).

Thus for some x > 0 ﬁxed we have P (Z > (am /am+n ) x) = P (am+n Z > am x) = P (Z1 + · · · + Zm+n > am x)

by (h)

= P ((Z1 + · · · + Zm ) + (Zm+1 + · · · + Zm+n ) > am x) = P (am Z1 + an Z2 > am x)

by (h)

≥ P (an Z2 ≥ 0 and am Z1 > am x) = P (Z2 ≥ 0)P (Z1 ≥ x) (i)

≥ (some δ) > 0

for all m and n.

2.

STABLE DISTRIBUTIONS

409

Thus am /am+n stays bounded away from ∞ for all m and n. Letting m = kI and m + n = (k + 1)I , we have from (h) that (ak /ak+1 )I = akI /a(k+1)I = am /am+n (j)

≤ ( some M ) < ∞

for all k and I.

Thus (ak /ak+1 ) ≤ 1 for all k; that is, (k)

an is ,

where amk = am ak

for all m, k ≥ 1,

was shown in (a). Thus exercise 2.1 below shows that an = n1/α for some α > 0. Suppose √ √ that α > 2. Then Var[Y ] < ∞ by exercise 2.2 below. We √ can thus claim that n(Y¯ − µ) →d N (0, 1) by the ordinary CLT. Thus An = n and ak = k work above. By the convergence of types theorem, there are no other choices. That is, when α > 2, then only α = 2 works. 2 Exercise 2.1 Suppose that an with a1 = 1, and suppose that amk = am ak for all k, m ≥ 1. Then show that necessarily an = n1/α for some α ≥ 0. Exercise 2.2 (Moments) exponent α. Then (6)

Suppose that Y ∼ = G is stable with characteristic

E|Y |r < ∞ for all 0 < r < α.

[Hint. Use the inequalities of section 10.3 to show that n P (|X| > an x) is bounded in n, where an ≡ n1/α . Then bound the appropriate integral.] Exercise 2.3 (Strictly stable dfs) Suppose that G is stable with characteristic exponent α = 1. Then there is a number b such that G(· + b) is strictly stable. [Hint. Show that b must satisfy bn ≡ bn + (an − n)b = 0 for all n, and specify b such that b2 = 0. Or else b2 = b = 0 immediately.] Example 2.1 (Hitting time as a stable law) Watch Brownian motion S until it ﬁrst attains height a. The time this takes is denoted by Ta . According to the strong Markov property, (7)

Tna ∼ = Ta(1) + · · · + Ta(n)

with

Ta(1) , . . . , Ta(n)

iid as Ta .

Checking the covariance functions, we see that S(a 2·)/a ∼ = S on [0, ∞), and thus Ta ∼ = a2 T1 . Putting these last two equations together gives (8)

Ta(1) + · · · + Ta(n) ∼ = Tna ∼ = n2 a2 T1 ∼ = n2 Ta ;

√ thus Ta is strictly stable with α = 12 . From (12.7.3), P (Ta < t) = 2[1 − Φ(a/ t )] for the N (0, 1) df Φ. Thus the df Fa and the density fa of Ta are given by √ 2 a (9) Fa (x) = 2[1 − Φ(a/ x )] and fa (x) = √ e−a /(2x) for x > 0. 2 3/2 2πx

410 CHAPTER 15. INFINITELY DIVISIBLE AND STABLE DISTRIBUTIONS

3

Characterizing Stable Laws

Theorem 3.1 (General stable chfs) Suppose that Y ∼ = G is stable. Then either Y is a normal rv or there is a number 0 < α < 2 and constants m1 , m2 ≥ 0 and β for which ∞ 1 dx Log φ(t) = iβt + m1 0 [ eitx − 1 − itα(x) ] x1+α (1) 0 1 +m2 − ∞ [ eitx − 1 − itα(x) ] |x|1+α dx. For 0 < α < 1 or 1 < α < 2 this can be put in the form (2)

Log φ(t) = idt − c |t|α × (1 + iθ Cα sign(t)),

where d ∈ (−∞, ∞), c > 0, |θ| ≤ 1, and Cα ≡ tan(π α/2). For α = 1 the form is (3)

Log φ(t) = idt − c |t| × (1 + iθ sign(t) log |t|)

with c, d, θ as above. In fact, θ = (m1 − m2 )/(m1 + m2 ) measures skewness, while the constants d and (1/c)(1/α) are just location and scale parameters. Then let p ≡ m1 /(m1 + m2 ) = (1 + θ)/2. Corollary 1 (Symmetric stable chfs) φ(·) is the chf of a nondegenerate and symmetric stable distribution with characteristic exponent α if and only if (4)

φ(t) = exp(−c|t|α )

for some 0 < α ≤ 2 and some c > 0.

Proof. We give only a direct proof of the corollary. This keeps things simple. Let Y ∼ G be strictly stable with chf φ and let ψ ≡ Log φ. Since Sn ∼ = = an Y , we have φn (t) = φ(an t). Thus (modulo 2πi) nψ(t) = ψ(an t) = ψ(am (an /am )t) = mψ((an /am )t) (a)

= mψ((n/m)1/α t)

by (15.2.3).

Thus for all rationals r > 0 we have shown that (5)

rψ(t) = ψ(r1/α t)

modulo 2πi,

and by continuity, (5) also holds for all real r > 0. Set t = 1 and r = τ α in (5) for ψ(τ ) = c τ α (b)

for all τ > 0,

with c ≡ ψ(1).

≡ (−c1 + ic2 ) τ α .

It must be true that c1 > 0; if c1 < 0 were true, then we would have the impossible situation that φ(τ ) → ∞ as τ → ∞, while c1 = 0 would imply that |φ(τ )| = 1 for all τ , and that Y is degenerate by proposition 13.6.2. Thus for some c1 > 0, for t > 0 we must have (c)

φ(t) = exp[(−c1 + ic2 ) tα ],

with φ(−t) = φ(t).

3.

CHARACTERIZING STABLE LAWS

411

We can summarize the two equations in (c) as (d)

φ(t) = exp(−c1 |t|α × [1 − i(c2 /c1 ) sign(t)] ) .

Since G is symmetric, φ is real, and so c2 = 0. Thus (5) holds. All that remains is to be sure that φ is a valid chf. This follows from the next two exercises. If α = 1, then exercise 15.2.3 shows that φ(t) exp(idt) is a strictly stable chf for some real d. Thus it satisﬁes (d), which is almost (2). 2

Exercise 3.1 (6)

Suppose that X ∼ = F with chf φ that satisﬁes

φ(t) = 1 − c |t|α + O(t2 )

as t → 0

for some ﬁxed α ∈ (0, 2). Suppose now that X1 , . . . , Xn are iid as F . Show that the properly normed sum satisﬁes Sn /n1/α →d Z where φZ (t) = exp(−c|t|α ). Exercise 3.2 Suppose X has density f (x) = (α/2) |x|−(α+1) × 1(1,∞) (|x|) with the constant α ∈ (0, 2). Show that φ satisﬁes (5). Exercise 3.3 (Holtzmark–Chandrasekar) Let Xn1 , . . . , Xnn be iid Uniform(−n, n). We now let 0 < 1/p < 2 and M > 0 be ﬁxed. Let n n Zn ≡ k=1 Znk ≡ k=1 (M sign(Xnk )/|Xnk |p . Regard Zn as the sum of forces exerted on a unit mass at the origin by n stars of mass M that are uniformly distributed on (−n, n) in a universe where an inverse pth power of attraction is operating. Show that Zn →d Z, where the chf of Z is given by φZ (t) = exp(−c|t|α ) for appropriate c and α. Exercise 3.4

Show that (1) can be put into the form (2) or (3).

412 CHAPTER 15. INFINITELY DIVISIBLE AND STABLE DISTRIBUTIONS

4

The Domain of Attraction of a Stable Law

We now merely state some results that assert when convergence in distribution to a general stable law takes place. Theorem 4.1 (Stable domain of attraction with 0 < α < 2) (a) Now, F ∈ D(G) for some stable G with characteristic exponent α ∈ (0, 2) if and only if (as x → ∞) both (1) [for α = 2/(β + 1)] U (x) ≡ [|y|≤x] y 2 dF (y) ∈ U2−α

(2)

(or equivalently) 1−t V (t) ≡ t K 2 (s) ds ∈ R−β

[for β ≡ (2 − α)/α])

and also (3)

P (X > x)/P (|X| > x) → (some p) ∈ [0, 1].

Moreover, α and p determine G up to type, as follows from the theorem of types. (b) The constants An of (Sn − Bn )/An →d Y ∼ = G necessarily satisfy (according to the theorem of types) (4)

nU (An )/A2n ∼ nA−α n L(An ) → 1

as n → ∞.

(c) The following are equivalent (for some constant 0 < α < 2, and then for some c > 0 and some 0 ≤ p ≤ 1): (5)

F ∈ DN (G)

for some stable G with characteristic exponent α.

(6)

xα P (X > x) → c p

and

xα P (X < −x) → c (1 − p)

(7)

t|K(1 − t)| → cp

and

t|K(t)| → c(1 − p)

α

α

as x → ∞.

as t → 0.

Exercise 4.1 Use section 7.9 to show that for 0 < α < 2 the following are equivalent conditions: (8)

U ∈ U2−α .

(9)

P (|X| > x) ∈ Uα . [x2 P (|X| > x)/ [|y|≤x] y 2 dF (y)] →

(10)

2−α α

as x → ∞.

Other characterizations in terms of K can be found in or derived from section 7.9. The theorems and this remark can also be proved via the L´evy–Khinchin theorem and results about regularly varying functions. Exercise 4.2 (a) State necessary and suﬃcient conditions on F for F ∈ D(Cauchy). (b) Do the same for F ∈ DN (Cauchy). (c) Show by example that DN (Cauchy) is a proper subset of D(Cauchy). (d) Observe that a symmetric df F (·) is in D(Cauchy) if and only if the tail function deﬁned by xP (X > x) = x(1 − F (x)) is slowly varying. (Recall the tail function τ (·) of Feller used in the WLLN in (10.4.2).)

4.

THE DOMAIN OF ATTRACTION OF A STABLE LAW

413

Exercise 4.3 (a) Show by example that the domain of normal attraction of the normal law DN (N (0, 1)) is a proper subset of the domain of attraction of the normal law D(N (0, 1)). To this end, let X1 , X2 , . . . be iid with density f (x) = |x|−3 × (2 log |x|) × 1[1,∞) (|x|) √ and consider Sn /( n log n). (b) Give a second example that works. (c) For both examples, determine an An that works. Exercise 4.4 (a) Consider the context of theorem 15.4.1. The constants An used for (Sn − Bn )/An →d Y ∼ ˜n2 /A2n → 1 (equivalently, it must = N (0, 1) must satisfy n σ 2 hold that n U (An )/An → 1 as n → ∞) as follows from the theorem of types. (b) It is also true that if and only if σ2 < ∞. √ ¯ (When σ2 ∈ (0, ∞), we know already that n(X n − µ)/σ → N (0, 1).) (11)

F ∈ DN (Normal)

Chapter 16

Asymptotics via Empirical Processes 0

Introduction

In section 16.1 we derive both the usual CLT and its studentized version for iid samples from any distribution in the domain of attraction of the normal distribution. We then obtain the corresponding pair of results for a trimmed mean valid for samples from any df in any domain of attraction provided only that the number of observations trimmed from each tail grows to inﬁnity so slowly that the fraction of observations trimmed from each tail goes to zero. When the qfs of all distributions in a class are bounded by an appropriate envelope qf, then all of these convergence results are uniform across the class of qfs. This uniformity allows random trimming. In section 16.4 complete analogs are developed for L-statistics, including uniformity of a studentized CLT. In section 16.2 similar results are derived for linear rank tests and permutation tests, and a uniform studentized CLT is given for sampling from a ﬁnite population. (Two very interesting ways of creating normality are discussed in this section.) Necessary and suﬃcient conditions are given in section 16.3 for both the weak and the strong bootstrap, and the Bayesian bootstrap is treated also. All of the results are based on the empirical process construction of section 12.10 combined with the quantile method1 inequalities from the second half of chapter 7. We will frequently obtain conclusions of the form Tn →p Z for a special construction version of an interesting statistic Tn and its limiting normal rv Z. Or, we may even obtain supK |Tn − Z(K)| →p 0 for a family of normal rvs Z(K) indexed by the qf K in a class K. These strong →p conclusions are only true for the special construction versions of the statistic Tn . However, Tn →p Z for the special construction Tn implies that Tn →d Z for any version of Tn . In like fashion, supK |Tn − Z(K)| →p 0 for a special construction version of Tn implies that the rate at which Tn →d Z is uniform across the entire class of qfs in K for any version of the statistic Tn . 1 My interest in this area grew from my collaboration with David Mason. But let me mention S. Cs¨ org˝ o, Haeusler, and Mason (1989), especially. Working and learning with David on a joint paper and reading the cited paper is what really got me turned on to quantile functions.

415

416

1

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

Trimmed and Winsorized Means

Let Xn1 , . . . , Xnn be iid with df F . Let Xn:1 ≤ · · · ≤ Xn:n denote their order 1 statistics, with empirical qf denoted by Kn . Deﬁne µ ≡ µK ≡ 0 K(t) dt = EK(ξ) 2 ¯ n = n Xni /n denote and σ2 ≡ σK ≡ Var[K(ξ)], when these exist. We also let X 1 n ¯ n )2 /n denote the “sample variance”. For the sample mean and Sn2 = 1 (Xni − X ˜ n (·) trimming numbers kn ∧ k ≥ 1 and for an ≡ kn /n and a ≡ k /n, we let K n

n

n

denote K(·) Winsorized outside (an , an ) (see notation 7.5.1) and deﬁne 1−a µ ˇn ≡ µ ˇK (an , an ) ≡ an n K(t) dt = E{K(ξ) × 1(an ,1−an ) (ξ)} , 11 2 ˜ n (s) dK ˜ n (t) = Var[K ˜ n (ξ)] , σ ˜n2 ≡ σ ˜K (an , an ) ≡ 0 0 [s ∧ t − st] dK (0) ˇn ≡ X ˇ n (an , a ) ≡ 1 n−kn Xn:i = µ ˇKn (an , an ), X n i=kn +1 n S˜2 ≡ σ ˜ 2 (an , a ). n

Kn

n

Here S˜n ≡ σ ˜Kn (an , an ) denotes the sample (an , an )-Winsorized standard deviation, ˇn = µ and the rv X ˇKn (an , an ) is being called the sample (an , an )-truncated mean, ˘n ≡ X ˇ n /(1 − an − a ) denotes the sample (an , a )-trimmed mean. Also, and X n n µ ˘n ≡ µ ˇ n /(1 − an − an ) is the population trimmed mean, and σ ˘n ≡ σ ˜n /(1 − an − an ). ˇ ˘ n , since Now, Xn is our vehicle of convenience for studying the trimmed mean X √ √ ˇn − µ ˘n − µ (1) n (X ˇ n )/˜ σn = Z˘n ≡ n (X ˘n )/˘ σn .

˘ n has statistical meaning, while X ˇ n does not; but X ˇ n is much [Said another way, X more convenient to work with notationally and probabilistically.] For kn ∧ kn ≥ 1 we always have (integrating Brownian bridge U, which for each ﬁxed ω is just a continuous function) 1−an 1 ˜ ˜ n /˜ (2) Z(Kn ) ≡ − U dK/˜ σn = − U dK σn ∼ = N (0, 1). 0

an

˜ t,t (·) denotes K(·) Winsorized outside (t, 1−t), and σ ˜ t,t (ξ)]. Generally, K ˜2 (t) ≡ Var[K Recall that Rn =a Sn denotes that Rn − Sn →p 0. Note We now specify that throughout this chapter Xni ≡ K(ξni ) = F −1 (ξni ), for 1 ≤ i ≤ n, for the ξni described in notation 1.1 below. (Recall the sentence above (7.4.3) where we noted that this representation of rvs allows alternative methods of proof.) [However, the conclusions in (4), (6), (17), and (18) below are true only for these particular rvs F −1 (ξni ) that are iid F (·), but the implied →d conclusion is true for any iid rvs Xn1 , . . . , Xnn having df F (·).] Theorem 1.1 (The CLT) (3)

Let K ∈ D(Normal). Equivalently, suppose

2 t [K+ (ct) ∨ K 2 (1 − ct)] / σ ˜ 2 (t) → 0

as t → 0, kn

for each ﬁxed c > 0.

(Note proposition 11.3.1 regarding (3).) Deﬁne kn ≡ ≡ 1, µ ˇn ≡ µK (1/n, 1/n) ˜ and σ ˜n ≡ σ ˜K (1/n, 1/n), and let Kn (·) denote K(·) Winsorized outside (1/n, 1−1/n). Then iid rvs Xn1 , . . . , Xnn with qf K(·) satisfy √ ∼ N (0, 1) ¯n − µ ˜ n) = (4) Zn = n (X ˇn )/˜ σn =a Z(K and Sn /˜ σn →p 1 .

1.

TRIMMED AND WINSORIZED MEANS

417

Now suppose the integers kn and kn satisfy (5)

(kn ∧ kn ) → ∞,

(an ∨ an ) → 0,

and

an /an → 1.

˜ n (·) now denotes K(·) Winsorized outside (an , 1 − an ). Of course, K Theorem 1.2 (CLT for trimmed means) Suppose the qf K(·) is such that ˜ t,t (ξ)] is in one of the regularly varying classes the partial variance σ ˜ 2 (t) ≡ Var[K ˜ in the R−β , for some β ≥ 0. [This union of all the R−β classes was labeled as D deﬁnition (7.10.33). R0 = D(Normal). Note inequality 7.10.4.] If (5) holds, then √ ˇn − µ ˜ n) ∼ (6) ˇn )/˜ σn =a Z(K and S˜n /˜ σn →p 1. Z˘n = n (X = N (0, 1) If β = 0, we can weaken (5) to an ∨ an → 0 when all 0 < an /M < an < M an < ∞. Corollary 1 (Trimming ﬁxed fractions) Suppose an → a and an → a for 0 < a < 1 − a < 1. Then (4) holds for any qf K(·) continuous at both a and 1 − a . Notation 1.1 Be clear that we are working on the speciﬁc probability space (Ω, A, P ) of theorem 12.10.3 on which are deﬁned a ﬁxed Brownian bridge U and a triangular array of rvs whose nth row members ξn1 , . . . , ξnn are iid Uniform(0, 1) with order statistics 0 ≤ ξn:1 ≤ · · · ≤ ξn:n ≤ 1, empirical df Gn , and empirical √ process Un = n [Gn − I] that not only satisfy Un = U →p 0, but in fact, for each ﬁxed 0 ≤ ν < 14 , satisfy (7)

= =1−1/n = = nν (Un − U) = ∆νn ≡ = = Op (1) . = [I ∧ (1 − I)](1/2)−ν = 1/n

For any ξn1 , . . . , ξnn (and hence for this realization also), for each ﬁxed 0 < ν < 1, = = = nν (Gn − I) =1 = (8) ∆0νn ≡ = 2 = [I ∧ (1 − I)]1−ν = = Op (1). 0 Remark 1.1 We will prove these theorems in such a way that the uniformity available is also apparent, and we will claim this uniformity in theorem 1.3, (14), and proposition 1.1 below. One of the values of the qf inequalities of section 7.11 is that they can be easily applied to the empirical qf. Remark 1.5 considers random kn and kn , which is useful to applied statisticians. 2 Proofs. Integration by parts (with (d,c] ≡ − (c,d] if c < d ) [a.s., dK puts no mass at any ξn:i ] yields √ ˇn − µ Z˘n = n (X ˇn )/˜ σn √ (a) = n{ [ξn:k +1 ,ξ K dGn − (an ,1−a ) K(t) dt}/˜ σn ] n

n:n−kn

n

418 (b)

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES =

√

n{KGn ξ

n:kn +1 −

√ + n{−tK

+ an +

(c)

=a.s. − − −

(an ,1−an )

√ (ξn:kn +1 ,an ]

−

[ξn:kn +1 ,ξn:n−k ]

ξ + Gn dK + KGn n:n−kn }/˜ σn

n

(an ,1−an ) t dK

˜ n /˜ U dK σn −

1−a − − tK n }/˜ σn

(an ,1−an )

˜ n /˜ [Un − U] dK σn

n [Gn (t) − an ] dK/˜ σn

[1−an ,ξn:n−k )

√

n [Gn (t) − (1 − an )] dK/˜ σn

n

(9)

˜ n ) + γ1n + γ2n + γ ≡ Z(K 2n

ˇn − µ ¯n − µ {+ γ3n + γ3n when replacing X ˇ n by X ˇn during theorem 1.1}, √ √ ˜n and γ3n ≡ K(ξn:n )/ n σ ˜n . [We require the added where γ3n ≡ K(ξn:1 )/ n σ terms γ3n and γ3n only for the theorem 1.1 identity, because in that case only we have inserted the inappropriate mean value µ ˇ(1/n, 1/n) (we could not use the “correct” µ = µ ˇ(0, 0) in the theorem 1.1 case, as it may be inﬁnite).] We begin an examination of the various γ-terms. Concerning γ1n , we note that 1−an (10) |γ1n | ≤ ∆νn × n−ν [t(1 − t)]1/2−ν dK(t)/ σ ˜n ≡ ∆νn × Mνn (K), an

and it is this sort of factorization into a random term times a deterministic term that is key to the proof. The randomness refers only to the Uniform(0, 1) distribution, and the only reference to K(·) is in the deterministic term (to which inequality 7.11.1 applies). In particular, ∆νn = Op (1) by (7). Then (7.11.3) gives (the second term appearing in (d) is the γn of (7.11.3), and the γn term in (d) is the symmetric term from the right tail) √ √ r |K+ (an ) − K+ (r/n)| (3/ ν ) √ (d) Mνn (K) ≤ + 1[an
√ the integrand of γ2n is uniformly bounded by |Un (an )| = an En , √ where it is trivially true that En ≡ |Un (an )/ an | = Op (1). We also let 1n denote the indicator function of an event, having probability exceeding 1 − , on which (with λ ≡ λ small enough) we can specify our choice of one of the following: (e)

(f)

(i) (ii) (iii)

λan ≤ ξn:kn ≤ ξn:kn +1 ≤ an /λ. √ √ ξn:kn +1 lies between an ∓ λ kn /n = an (1 ∓ λ/ kn ). Bound ξn:kn +1 by its lower and upper /2-quantiles t∓ n .

[The bounds in (f) are derived as follows: For (ii), the Chebyshev (second moment) inequality with beta rv moments of ξn:kn +1 . Or for (i), the Markov (ﬁrst moment) inequality, or from the in probability linear bounds of inequality 12.11.2. Or for choice (iii), the exact distribution of ξn:kn +1 .]

1.

TRIMMED AND WINSORIZED MEANS

419

Consider theorem 1.1, when a√ n = an = 1/n. Now, γ1n is controlled via (d) (at the left end) solely by |K+ (an )|/ n σ ˜n (just choose r ﬁrst to be very large in the ﬁrst term in the bound of (d)). To summarize, γ1n →p 0 whenever √ (11) σn → 0 . { an |K+ (an )| + an |K(1 − an )| }/˜

For γ2n and γ3n we apply (f)(i) in chosing 1n in (f) to get (when K+ (0) < 0) √ √ (g) σn ≤ 2En × | an K(λan )/˜ σn | , |γ2n | × 1n ≤ En × an (λan ,an /λ) dK/˜ √ √ |γ3n | × 1n ≤ 2 × |K+ (λan )| /( n σ (h) ˜n ) ≤ 2 × | an K(λan )/˜ σn | . Thus the CLT claim of (4) holds, provided only (adding the symmetric condition in the right tail) that √ (12) σn → 0 { an |K+ (λan )| + an |K((1 − λan ))| }/˜ for each 0 < λ ≤ 1, and it holds uniformly in any class of qfs in which (12) holds uniformly. But (12) follows from (3) (or any of the equivalent conditions (11.3.14), or (11.3.8), or (11.3.9), or (11.3.29), or (10.6.31), or (7.7.6), or (7.6.7)). Thus the normality in theorem 1.1 is established. [Note that (the crude upper bound) condition (12) implies (11), (g), and (h).] Consider theorem 1.2. The ﬁrst term in (d) converges to 0, and the other two terms that appear in (d) equal zero. Thus, again γ1n →p 0 whenever kn ∧ kn → ∞ (but now, it converges uniformly in all qfs). In the present context the γ3n term always equals 0. Thus, only γ2n must be shown to be negligible; but we must now be much more careful than the crude bound (12). Now, using (f)(ii) in the deﬁnition of 1n in (f), we obtain √ (i) |γ2n | × 1n ≤ En × an Iλn dK/˜ σn , √ √ where Iλn ≡ (an (1 − λ/ kn ), an (1 + λ/ kn )). Thus the CLT claim of (6) holds, provided that (with symmetric requirements in the right tail) √ √ (13) an K(an (1 ∓ λ/ kn )) − K+ (an ) /˜ σn ≡ an I ∓ dK/˜ σn → 0 λn

for each λ > 0. Thus, normality holds uniformly in any class Ku in which both (13) and its right tail analogue hold uniformly; call such a class Ku a uniformity class. Whenever kn ∧ kn → ∞, √ ˇ n (an , a ) − µ ˜ n ) →p 0 supK∈Ku n [X ˇn ]/˜ σn − Z(K n

Summary (14)

for any class Ku in which both (13) and its right tail analogue hold uniformly. Also, we may replace σ ˜n by S˜n in (14) under this same requirement (as will be shown in our variance estimation proof below). Now, (13) does hold for a ﬁxed K whenever both K is in any R−β and (5) holds (by theorem 7.10.1), and this gives theorem 1.2. (Two uniformity classes Ku are exhibited in theorem 7.10.2.) Aside from variance estimation, the proofs of theorems 1.1 and 1.2 are complete. But ﬁrst we consider uniformity of the asymptotic normality, via making (12) hold uniformly. 2

420

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

Uniform CLT for Trimmed Means Notation 1.2 (K and A) (i) Specify a ﬁxed qf K0 (·) which has ﬁnite variance. Fix 0 < a0 < 12 . Deﬁne the class (recall (7.10.3)) (15)

K ≡ {K : |K − µK |/σK ≤ |K0 | on (0, a0 ] ∪ [1 − a0 , 1)}

to be all qfs whose standardized form is bounded in the tails by the ﬁxed qf K0 (·). (ii) Let b(t) = b(1 − t) ≡ 1 ∨ [ 2 log2 (1/t) ]1/2 = 1 ∨ [ 2 log(− log t)]1/2 for 0 < t ≤ 12 , and suppose further in regard to the qf K0 (·) of (15) that

1

(16) 0

b(t) |K0 (t)| dt < ∞. t ∧ (1 − t)

(iii) Let A ≡ {(an , an ) : n ≥ 1, an ∨ an ≤ (some ﬁxed a0n ), where a0n → 0}.

2

Theorem 1.3 (Uniform studentized CLT for means and trimmed means) Suppose K as in (15) satisﬁes (16). Then √ ˇn − µ n (X ˇn ) ˜ (17) sup − Z(Kn ) →p 0 and ˜ ˜n or σK A,K∈K Sn or σ (18)

√ ¯ n − µ) n (X ˜ n ) →p 0 . sup − Z(K ˜n or σK K∈K Sn or σ

˜ n ) depends on n.] [Of course, kn = k = 1 in (18).] [The “limiting” N (0, 1) rv Z(K n The conclusions using σ ˜n in (17) and (18) only require K0 to have ﬁnite variance; they do not require (16). Nor is (16) required if supA,K and supK are omitted. ˜ n ) by Z(K)) Proposition 1.1 (Replacing Z(K (19)

(20)

sup

A,K∈K

˜ n )| = Op (1), |Z(K

sup |Z(K)| = Op (1),

sup

A,K∈K

and

K∈K

where Z(K) ≡ −

1 0

sup K∈K

When (16) holds, we also have

˜ n ) − Z(K)| →p 0, |Z(K

a0n 0

+

1 1−a0n

˜ n) ≡ − U dK/σK ∼ = N (0, 1) and Z(K

|U| dK/σK = op (1) ,

1−an an

U dK/˜ σn ∼ = N (0, 1).

Proof. Consider proposition 1.1. First, let g(t) ≡ b(t) t ∧ (1 − t). Then U/g = Op (1) follows from applying the LIL for U found in (12.3.18). Second, use |K − µ|/σK ≤ |K0 | and (16) after integration by parts to ﬁnd that (j)

supK

1 0

K g(t) d K(t)−µ ≤ supK σK

1 0

|K0 (t)| d|g|(t) =

1 0

|K0 | d|g| < ∞

for total variation measure d|g|. (Exercise 1.1 asks for all details of (j).) Combining these two results gives

1.

TRIMMED AND WINSORIZED MEANS

421

1 K supK∈K |Z(K)| = supK∈K | 0 U d K−µ σK (k)

≤ U/g × supK |

1 0

K g(t) d K(t)−µ ≤ U/g × σK

1 0

|K0 | d|g| = Op (1).

That is, the ﬁrst half of (20) holds; and the same argument on the tails clearly yields the second half of (20). (Exercise 1.1 again asks for all details.) Then (21)

n (K) ≡ (˜ σn − σK )/σK

supA,K∈K | n (K)| → 0

satisﬁes

under (15)

by (7.10.4). (Exercise 1.1 again asks for all details.) Since (l)

˜ n ) − Z(K) = Z(K = ( σ1K −

(m) (n)

= (1 − =

1 σ ˜n )

σK σ ˜n

×

1 σK

1 0

1 0

U dK −

1 σ ˜n

U dK + σσ˜Kn ×

) × Z(K) ⊕ (1 − (1 −

1−an an

U dK

(an ,1−an )c σK σ ˜n

U dK/σK

a 1 )) × ( 0 0n + 1−a )|U| dK/σK 0n

1 a

n (K) 1 × Z(K) ⊕ × ( 0 0n + 1−a ) |U| dK/σK , 0n 1 + n (K) 1 + n (K)

combining (21) with the two halves of (20) gives the second half of (19). The ﬁrst half of (19) then follows from the ﬁrst half of (20) and the second half of (19). A workable alternative to (16) requires that (22)

sup

1

K∈K

0

K g d K−µ <∞ σK

and

lim sup

a→0 K∈K

(a,1−a)c

K g d K−µ = 0. 2 σK

Proof. Consider theorem 1.3. Without loss of generality, we may suppose that the qf K has mean 0 and variance 1. Then (7.10.4)–(7.10.6) show that the fundamental requirement (12) holds uniformly in the class K of (15) and in A. Thus the σ ˜n result stated in (17) is true. Moreover, in the σ ˜n result of (17) with an = an = 1/n, we can replace µ ˇn by µ to obtain (18) by (7.10.5). [In fact, (23)

sup

A,K∈K

n k0n

|ˇ µn − µ| /˜ σn → 0

under (15)

by (7.10.5), but (23) does not typically allow µ to replace µ ˇ in (17).] Consider the algebraic identity √ ˇn − µ ˜ n) n (X ˇ n )/σK − Z(K (24)

√ ˇn − µ ˜ n )] + [˜ ˜ n) = [˜ σn /σK ] × [ n (X ˇ n )/˜ σn − Z(K σn /σK − 1] × Z(K

(o)

= [Op (1)] × [op (1)] + [op (1)] × [Op (1)] ,

where the terms are Op (1) and op (1) uniformly over K ∩ A by (19), (21), and the σ ˜n versions of (17) and (18). This, and (23), gives the σK versions of (17) and (18).

422

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

We are leaving the proof of all results regarding the various variances untill the following separate proof. So for now we merely state that (25)

σn − S˜n )/S˜n

ˆn (K) ≡ (˜

satisﬁes

sup |ˆ

n (K)| → 0

A,K∈K

under (15).

(But we will use (25) right now.) Consider the algebraic identity √ ˇn − µ ˜ n) n (X ˇ n )/S˜n − Z(K √ ˇn − µ ˜ n )] + [˜ ˜ n) = [˜ σn /S˜n ] × [ n (X (26) ˇn )/˜ σn − Z(K σn /S˜n − 1] × Z(K (p)

= [Op (1)] × [op (1)] + [op (1)] × [Op (1)] ,

where the terms are Op (1) and op (1) uniformly over K ∩ A by (19), (25), and σ ˜n versions of (17) and (18). This gives the S˜n versions of (17) and (18). 2 Proof. We now turn to the various claims about variance estimation made above. We will show that Sn /˜ σn →p 1, and we will also obtain uniform versions of the same result. Let 1−a mjn ≡ an n K j (t) dt and (27) n−k j K j dGn = n1 i=knn+1 Xn:i m ˆ jn ≡ [ξn:k +1 ,ξ ] n:n−kn

n

for j = 1, 2. Trivial modiﬁcations in the basic identity (9) give (q) σnj = { [ξn:k +1 ,ξ K j dGn − (an ,1−a ) K j (t) dt}/˜ σnj (m ˆ jn − mjn )/˜ ] n

(r)

n

= − (an ,1−a ) [Gn (t) − t] dK j (t)/˜ σnj n − (ξn:k +1 ,an ] [Gn (t) − an ] dK j (t)/˜ σnj n − [1−a ,ξ [Gn (t) − (1 − an )] dK j (t)/˜ σnj ) n

(28)

n:n−kn

n:n−kn

≡ γj;1n + γj;2n + γj;2n

{+ γj;3n + γj;3n for the proof of theorem 1.1}

where γj;3n ≡ K j (ξn:1 )/(n˜ σnj ) and γj;3n ≡ K j (ξn:n )/(n˜ σnj ). Note that γ1;1n is −1/2 −1/2 Op (n ) while γ1;2n and γ1;3n are op (n ), as was actually shown during the normality proof given earlier (that is, our earlier proof corresponded to j=1). In the earlier proof (with j=1, while considering γ2n = γ1;2n ), we showed that √ √ |Un (an )/ an | × | an |K(ξn:kn +1 ) − K+ (an )|/˜ σn | ≡ En × Dn

(29)

= Op (1) × op (1) →p 0.

Now (with j=2, while considering γ2;2n ) we can claim that (s) (30)

|γ2;2n × 1n ≤

1 an

|Gn (an ) − an | ×

an 2 σ ˜n

2 ˜n × D ˜n |K 2 (ξn:kn +1 ) − K+ (an )| ≡ E

˜n × [Dn (Dn + 2 d˜n )] = Op (1) × op (1) →p 0 =E

1.

TRIMMED AND WINSORIZED MEANS

423

whenever (31)

2 d˜2n ≡ an K+ (an )/˜ σn2 = Op (1),

using E˜n = Op (1) by inequality 7.10.3, (which is also true uniformly in A by exercise 1.4 below, as a random sample size would require). So, γ2;2n →p 0 has 2 →p 0. We just been shown to hold under (13) and (31). And trivially, γ2;3n = γ1;3n will show that γ2;1n is uniformly Op (1) in the next paragraph. We ﬁrst observe that the additional contributions to S˜n2 from the Winsorized tails are negligible; thus, j j an |K+ (ξn:kn +1 ) − K+ (an )|/˜ σnj

(t) ˜ n = op (1) for j=2, equals D

and

equals

√

√ an Dn = op ( an ) for j=1.

All contributions considered so far (that is, all contributions except γ2;1n ) were shown to be uniformly neglible in both in Ku or in A∩K. [Recall (7.10.4)–(7.10.6).] ¯ 2 ≡ −[(K − µ For γ2;1n , we obtain (with K ˜ n )− ]2 + [(K − µ ˜n )+ ]2 , see (7.5.7)) n (u)

¯ 2 (K ¯ 2 ) = Op (1) × M ¯ 2 (K ¯ 2 ) →p 0 |γ2;1n | ≤ ∆0νn × M νn n νn n

using (8) for ∆0νn and mimicking (10), since we next use (7.11.4) to obtain 2 ¯ νn ¯ n2 ) ≡ 1−an n−ν [t(1 − t)]1−ν dK ¯ n2 /˜ M (K σn2 an (32)

2 2 ≤ 2/[(kn ∨r)∧(kn ∨r )]ν +r|K+ (an )−K+ (r/n)|/n σ ˜n2 ×1[an
This holds uniformly in A ∩ K by (12), and uniformly in all qfs under condition (5). We have thus established all variance estimation claims. [In particular, having shown the negligibility of (S˜n2 − σ ˜n2 )/˜ σn2 , we have thus established the negligibility of the (˜ σn − S˜n )/S˜n claimed in (25). Add these details to exercise 1.1.] For theorem 1.2 and (14) we choose to assume in this proof that all µ ˜n = 0. Then (31) is not required, as all d˜2n ≤ 1 is now true. 2 ˜ of qfs K having σ Remark 1.2 The class D ˜ 2 (·) ∈ R−β for some β ≥ 0 is strictly bigger then the stable laws; the stable laws also require that v + (t)/v− (t) → (some (1 − p)/p) ∈ [0, 1] so that both the contributions from the two extreme tails {(from 1 to kn ) and (from n − kn + 1 to n)} can be balanced. (Recall the De Haan result in exercise 7.9.2.) But we do not need to balance them, we threw them away. 2 Exercise 1.1 Supply the complete details in the four places requested in the proof of theorem 1.3. Exercise 1.2

√ ˜n . Verify that sup{Un (t)|/ an : c an ≤ t ≤ an /c} = Op (1), for E

Exercise 1.3

Supply complete details for (f) (i) and (ii) in the theorem 1.1 proof.

424

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

Exercise 1.4 Write out the identity (c) in case an < ξn:kn +1 and an < ξn:n−kn in the proof of theorem 1.1. Remark 1.3 (Heavy trimming) Suppose −K(t) = K(1−t) = exp(1/t) on (0, 12 ), and kn = kn . Then (recall (7.10.23)) |K+ (ca) − K+ (a)|/|K+ (a)| = exp((1/a)(1/c − 1)) − 1 if n2/3 /kn → 0 . ≤ exp((n/kn )(2λ/ kn ) − 1 → 0 That is, we can even bring back exponential growth of K by trimming heavily enough. Clearly, there is a myriad of such examples, but the required rate will depend on K(·). We will not attempt to characterize them here. The statistician does not want to throw away too much data, and would likely be more comfortable ˜ 2 assuming (5) in a preplanned insurance policy that works throughout all D. Remark 1.4 (Random trimming) However, the data analyst throws away data based on the sample, using random sample sizes κn and κn . Suppose σ ˜2 (·) is in R−β for some β ≥ 0. If for some ﬁxed kn and kn as in (5), we have (33)

(κn − kn )/kn ≡ (ˆ an − an )/an = op (1)

and (κn − kn )/kn = op (1),

then |κn /kn − 1| ≤ θn → 0 on an appropriate 1n , and thus our theorem 7.10.1 proof goes through as before. (We must now strengthen the observation below (e) √ earlier that En = |Un (an )|/ an = Op (1) to the conclusion that √ (34) sup{|Un (t)|/ an : c an ≤ t ≤ an /c} = Op (1) , but this is still true; see exercise 1.2.) Then (6) and (14) still hold, with random centering a` la √ ˇ n (κn /n, κ /n) − µK (κn /n, κ /n)]/Wn =a Z(K ˜ n) (35) n [X n n for a Wn equal to either σ ˜n or S˜n (using either kn and kn , or κn and κn ). For 2 σ ˜ (·) ∈ R0 , this argument requires only a less stringent Op (1) in (33); the sample version of the requirement (7.7.6) that av(a)/V (a) → 0 as a → 0 could be used regarding a test of this requirement. 2 Remark 1.5 Our proof that the m ˆ nj /mnj →p 1 (see (27)) oﬀers an alternative proof of the suﬃciency in theorem 10.6.2. 2 Exercise 1.5 (a) Determine the quantities F (·), K(·), σ ˜2 (·), and t K 2 (ct)/˜ σ2 (t) for the Cauchy(0, 1) distribution. (b) Determine also the asymptotic order of √ an |K(an (1 − M/ kn )) − K(an (1 + M/ kn ))|/˜ σ (an ) . Exercise 1.6 Repeat the previous exercise for the qf K(1 − t) = −K(t) ≡ t−β for 0 < t < 12 , with β > 0.

1.

TRIMMED AND WINSORIZED MEANS

425

Exercise 1.7 (Rossberg) Suppose 0 ≤ ξn:1 ≤ · · · ≤ ξn:n are the order statistics of the ﬁrst n of the inﬁnite sequence of independent Uniform(0, 1) rvs ξ1 , ξ2 , . . . . Let αn ≡ {j : 1 ≤ j ≤ kn }, δn ≡ {j : kn + 1 ≤ j ≤ ln }, βn ≡ {j : ln + 1 ≤ j ≤ n − ln }, δn ≡ {j : n − ln + 1 ≤ j ≤ n − kn }, and αn ≡ {j : n − kn + 1 ≤ j ≤ n}. Show that the collections of rvs {ξn:j : j ∈ αn }, {ξn:j : j ∈ βn }, and {ξn:j : j ∈ αn } are asymptotically independent when kn ∞ with an ≡ kn /n → 0 and ln ≡ kn1+2ν for 0 < ν < 14 (with analogous conditions in the right tail). Remark 1.6 Consider asymptotic normality of the sample mean. Let Un , Vn , Wn , Vn , and Un denote the sum of those Xn:j = K(ξn:j ) for which j is in αn , δn , βn , δn , and αn , respectively. The previous exercise shows that Wn , Un , and Un are asymptotically independent. Exercise 7.11.1 shows that Vn and Vn are always asymptotically negligible. We saw in chapter 11 that the condition σ ˜2 (·) ∈ L is also necessary for (4). Since the vast middle is “nearly always” normal, one needs to determine what is happening only in the extreme tails (as the mid-tails were always negligible). This asymptotic independence is at the heart of the very general ¯ n found in S. Cs¨ asymptotics for X org˝ o, Haeusler and Mason (1989). 2 √ ˜ ˜n )/˜ σn for the sample Exercise 1.8 (Winsorized mean) Let Z˜n ≡ n (X n − µ ˜ n of the Winsorized sample X ˜ n . For the N (0, 1) rv Z(K ˜ n ) of (2), the mean X identity (9) becomes ˜ n ) + γ1n + γ˜2n + γ˜ , Z˜n = Z(K 2n √ √ where γ˜2n ≡ n In Gn dK/˜ σn and γ˜2n ≡ n I Gn dK/˜ σn n

and γ1n is as before. Here In is equal to [ξn:kn +1 , an ] or (an , ξn:kn +1 ) according as ξn:kn +1 < an or ξn:kn +1 ≥ an , and In is equal to [1−an , ξn:n−kn ] or (ξn:n−kn , 1−an ) according as 1 − ξn:n−kn < an or 1 − ξn:n−kn ≥ an . Show that this quantity γ˜2n for the Winsorized mean essentially exceeds the γ2n of the trimmed mean proof by √ the factor of kn . This is just enough to prevent analogues of the theorems for trimmed means that we proved in this chapter. (a) Prove what you can for the Winsorized mean. (b) Graph a typical K on (0, 1). Locate an and ξn:kn +1 near the 0 endpoint (and suppose K(0) < 0). Obtain the natural graphical upper bounds on the magnitudes of γ2n and γ˜2n , and note how the second quantity is inherently larger than the ﬁrst (pictorially, a “trapezoid” versus a “triangle”). (c) Nevertheless, recall theorems 11.5.1 and 11.5.2. (In that previous context, trimming did not work as well as Winsorizing.) Exercise 1.9 (Uniform WLLN) (36)

t (|K0 (t)| + |K0 (1 − t)|) → 0

Suppose the qf K0 (·) is of order one, in that as t → 0 .

Then one can claim the uniform WLLN 1−1/n ¯ (37) sup Xn − K(t) dt →p 0. K∈K0 1/n Now K0 ≡ {K : |K| ≤ |K0 | on (0, a0 ] ∪ [1 − a0 , 1)}, for some 0 < a0 ≤ 12 .

426

2

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

Linear Rank Statistics and Finite Sampling

Example 2.1 (Linear rank statistics) Consider the RN process of section 12.10, with the same notation and assumptions as were made there. Thus (for cN i ’s with 2 mean c¯N = 0, standard deviation σcN = 1, and standardized fourth central moment 4 4 cN /σcN ≤ M < ∞ for all N ) we deﬁne (1)

1 RN (t) ≡ √ N

[(N +1)t]

i=1

cN DN i − cN · 1 =√ σcN N

[(N +1)t]

cN DN i

on [0, 1],

i=1

¨ νN = Op (1), as in (12.10.35). The known constants and this process satisﬁes ∆ cN ≡ (cN 1 , . . . , cN N ) are called regression constants. Let aN ≡ (aN 1 , . . . , aN N ) 2 specify known scores. Let aN · , σaN > 0, and µ4 (aN ) denote their mean, variance, and fourth central moment. The class of simple linear rank statistics is deﬁned by (2)

N 1 aN i − aN · TN ≡ TN (aN ) ≡ √ cN DN i . σaN N i=1

Now, ETN = 0 and Var[TN ] = NN−1 by exercise 2.1 below. Assume (for convenience only) that the scores are ordered as aN 1 ≤ · · · ≤ aN N , and we deﬁne an left-continuous qf KN on [0, 1] by (3)

KN (t) = aN i − aN ·

for (i − 1)/N < t ≤ i/N, and 1 ≤ i ≤ N,

with KN (0) ≡ aN 1 . Note that 1 (4) TN ≡ KN d RN /σaN = − 0

1

RN dKN /σaN .

0

The basic probability space and rvs are as deﬁned in notation 16.1.1. Let 1 (5) ZN ≡ Z(aN ) ≡ − W dKN /σaN . 0

Clearly, ZN is a normal rv with mean 0, since it is just a linear combination of the jointly normal rvs W (1/N ), . . . , W (1 − 1/N ). Fubini’s theorem gives 1 1 1 2 Var[ZN ] = Var − 0 W dKN /σaN = 0 0 [s ∧ t − st] dKN (s) dKN (t)/σaN (6)

2 = Var[KN (ξ)]/σaN = 1.

We will also consider the sum of independent rvs given by 1 1 1 TN0 ≡ TN0 (aN ) ≡ σ−1 0 WN dKN = σaN 0 KN dWN aN (7) = √1N ni=1 [ cN i KN (ξN i ) ]/ σaN ∼ = (0, 1). We next show that TN =a TN0 =a ZN ∼ = N (0, 1) under rather mild conditions. Exercise 2.1

Show that ETN = 0 and Var[TN ] = N/(N − 1).

2

2.

LINEAR RANK STATISTICS AND FINITE SAMPLING

Deﬁnition 2.1 (D(aN )-negligiblity) (8)

D(aN ) ≡

427

Call such aN a negligible array if

max1≤i≤N |aN i − aN · | √ ≤ (some N ) 0 . N σaN

We let A denote any collection of such arrays that uses the same N ’s. When the aN i are random, we call them p-negligible if D(aN ) →p 0

and

a.s.-negligible if D(aN ) →a.s. 0 .

Theorem 2.1 (Uniform CLT for linear rank statistics) Suppose that the regression constants satisfy c4N ≤ M < ∞ for all N . Let A be a collection of uniformly negligible arrays, in that N ≡ supA D(aN ) → 0. Then (9)

supA |TN (aN ) − Z(aN )| →p 0

Proof.

Fix 0 ≤ ν < 14 . Now, γ(aN ) ≡ TN − ZN satisﬁes N/(N +1) |γ(aN )| = |TN − ZN | = | − 1/(N +1) (RN − W ) dKN /σaN |

(a) (b)

and

= ν = = N (RN − W ) =N/(N +1) = = ≤= × [I(1 − I)]1/2−ν = 1/(N +1)

(10)

supA |TN0 (aN ) − Z(aN )| →p 0 .

N/(N +1)

N −ν [t(1−t)]1/2−ν dKN (t)/σaN

1/(N +1)

¨ νN × MνN (K ˜ N ), ≡∆

¨ νN = Op (1) by (12.10.35). Thus (7.11.3) gives where ∆ ¨ νN × |TN − ZN | ≤ ∆ $ % √ |KN (1/(N + 1)) − aN · | √ |KN (N/(N + 1)) − aN · | 9/ν √ √ + r + r rν N σaN N σaN (c) (11)

√ √ ¨ νN × { 9/ν /rν + 2 r [max1≤i≤N |aN i − aN · |/ N σaN ]} ≤∆ √ ¨ νN × { 9/ν /rν + 2 r D(aN )}. ≤∆

By ﬁrst choosing r large and then letting N → ∞, we see that (d)

|TN − ZN | = Op (1) × o(1) →p 0 .

Note that we have separated the randomness from properties of the aN i ’s, so that the convergence is uniform over A. [The rate for TN in (9) depends on the sequence ¨ νN ≥ M ) ≤

N → 0, and on the cN i ’s only through the M ’s in the statement P (∆ for all N .] Since (see (12.10.34)) N/(N +1) ˙ νN × MνN (K ˜ N ), (e) |TN0 − ZN | = | 1/(N +1) (WN − W ) dKN /σaN | ≤ ∆ comparing (e) with (a) and (10) shows that the proof is the same in this case. [One can also allow random regression constants cN and scores aN that are N and for which c4 = Op (1) and sup D(aN ) →p 0.] 2 independent of the D A N

428

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

Example 2.2 (Creating normality) The following are known, and intuitive. (A) (Using normal regression constants) When lim c4N ≤ ∞, present methods give (12)

TN →d N (0, 1) if and only if either D(aN ) → 0 or cN DN 1 →d N (0, 1).

Result 1 Thus, with absolutely no hypotheses, (13)

the choice cN i ≡ Φ−1 (i/(N + 1)) always gives TN →d N (0, 1) for every choice of aN i ’s.

(B) (Winsorizing a ﬁnite sample) Let a ˜N · , σ ˜aN , T˜N , and Z˜N be deﬁned as before, ˜N consisting of but now based on the (kN , kN )-Winsorized population a (14)

; aN,N −k , . . . , aN,N −k . aN,kN +1 , . . . , aN,kN +1 ; aN,kN +1 , . . . , aN,N −kN N N

Of course, theorem 2.1 also applies to T˜N . But note that now √ ˜N ) = max |˜ aN i − a ˜N · |/ N σ ˜aN D( a 1≤i≤N

1 √ − a = |aN,kN +1 − a ˜N · | ∨ |aN,N −kN ˜N · | /( N σ ˜aN ) 0

− a |aN,kN +1 − a ˜N · | ∨ |aN,N −kN ˜N · | ≤5 61/2 + 1)(a − a ˜N · )2 + (kN ˜N · )2 (kN + 1)(aN,kN +1 − a N,N −kN

) + 1, ≤ 1/ (kN ∧ kN

(15)

. provided only that aN,kN +1 < aN,N −kN

˜N ) → 0 whenever Thus D( a (16)

→ ∞, and aN,kN +1 < aN,N −kN for all N suﬃciently large. kN ∧ kN

Result 2 Suppose c4N ≤ M < ∞ for all N . For ﬁxed kN ∧ kN → ∞, we have ˜ aN ) : all arrays in A with aN,kN +1 < aN,N −k →p 0. (17) sup T˜N (aN ) − Z( N A

Summary: Asymptotic normality is guaranteed by Winsorizing a number that slowly increases to inﬁnity, provided only that we do not collapse the whole sample. 2 Exercise 2.2

Argue heuristically why (13) should be true. [See Shorack(1996).]

Example 2.3 (Permutation statistics) Suppose X1 , . . . , XN are iid rvs with N ≡ (X1 , . . . , XN ) denote the full nondegenerate df F on (Ω, A, P ). Then let X population of observed values, having order statistics XN :1 ≤ · · · ≤ XN :N , antiranks ¯ N , sample variance S 2 , empirical df FN , and (DN 1 , . . . , DN N ), sample mean X N ˜ −1 empirical qf KN ≡ FN . Let 0 ≤ kN < N − kN ≤ N , and let X N denote the ˜ N :1 ≤ · · · ≤ X ˜ N :N (as in (14)) whose parameters (kN , kN )-Winsorized population X 2 ˜N . are the Winsorized mean T˜N , the Winsorized variance S˜N , and empirical qf K We note that XN :1 ≤ · · · ≤ XN :N and (DN 1 , . . . , DN N ) are independent rvs, (18) if tied Xi ’s are randomly assigned their ranks.

2.

LINEAR RANK STATISTICS AND FINITE SAMPLING

429

We also recall (from theorem 11.4.1) that (19)

N ) →a.s. 0 D(X

if and only if 0 < Var[X] < ∞.

(20)

N ) →p 0 D(X

if and only if F ∈ D(Normal).

Moreover, from (16), (21)

˜ [making X N a.s. negligible] for a.e. X1 , X2 , . . . ,

˜ D(X N ) →a.s. 0

provided only that (22)

kN ∧ kN →a.s. ∞

− XN :k +1 ) > 0 a.s. lim (XN :N −kN N

and

for kN and kN that are either ﬁxed integer sequences or integer-valued rvs that are N . Condition (22) necessarily holds if independent of the antiranks D

(23)

F is any nondegenerate df , and if kN ∧ kN → ∞ while (kN ∨ kN )/N → 0.

[We shall maintain the order statistics (which are on (Ω, A, P )), but we can replace the independent antiranks by a realization (on some (Ω∗ , A∗ , P ∗ ) independent of ¨ ∗ = Op (1) on (Ω∗ , A∗ , P ∗ ) for some Brownian bridge W . (Ω, A, P )) for which ∆ νN This is possible whenever c4N ≤ M < ∞ for all N (see theorem 12.10.3).] By a permutation statistic we mean a rv of the form (24)

N N 1 N ) ≡ √1 TN ≡ TN (X cN i Xi /SN = √ cN DN i XN :i /SN , N i=1 N i=1

2 2 ¯ 2 with SN = σXN = N cN ≡ (cN 1 , . . . , cN N ) population 1 (Xk − XN ) /N and with a that is standardized. [Note that the distribution of TN is unaltered by using this diﬀerent realization of the antiranks.] 2 Theorem 2.2 (Permutation tests)

¨ ∗ = Op (1) (as when lim c4 < ∞), If ∆ νN N

N) ∼ TN =a Z(X = N (0, 1) (25)

  on (Ω∗ , A∗ , P ∗ ) for a.e. X1 , X2 , . . . if cN i = Φ−1 (i/N + 1), holds on (Ω∗ , A∗ , P ∗ ) for a.e. X1 , X2 , . . . if 0 < Var[X] < ∞,  on (Ω × Ω∗ , A × A∗ , P × P ∗ ) if F ∈ D(Normal).

N ) →p or [The convergence is uniform over classes F of dfs F in which D(X uniformly.] Also, whenever (22) (or (23)) holds a.e. we have (26)

˜ ∼ T˜N =a Z(X N ) = N (0, 1)

a.s.

0

on (Ω∗ , A∗ , P ∗ ) for a.e. X1 , X2 , . . . .

N ) and T˜N (Y N ), where Y N ≡ (YN 1 , . . . , YN N ) with Similar results hold for TN (Y (27)

YN i ≡ gˆN (Xi )

N. for any function gˆN (·) independent of the antiranks D

430 Proof. (a) (28) (b)

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES Equation (11) now becomes (recall (10) for MνN (·)) N )| ≡ |TN − Z(X N )| = | − 1 (RN − W ) dKN /SN | ≤ ∆ ¨ ∗ × MνN (KN ) |γ(X νN 0 √ ¨ ∗νN × { 9/ν /rν + 2 r D(X N )} ≤∆ = Op (1) × o(1) →p 0,

as in (25),

N ). So, (25) using either (19), (20) on subsequences, or the proof of (13) on D(X ˜ 2 holds. Likewise, conclusion (26) holds by using (22) to apply (15) to D(X N ).

Sampling from Finite Populations Example 2.4 (Simple random sampling) Let X1 , . . ., Xn be a random sample ¯n without replacement from an aN ≡ (aN 1 , . . . , aN N ) population. As usual, let X 2 2 2 ¯ and Sn ≡ Xn − Xn denote the sample mean and “sample variance”. Suppose that aN 1 ≤ · · · ≤ aN N , that n ≡ nN , and the D(aN ) of (8) satisfy both (29) (30)

0 < lim n/N ≤ lim n/N < 1

and √ sup D(aN ) = sup { max |aN i − aN · |/ N σaN } → 0 . A

A

1≤i≤N

[Prior to normalizing, the cN i ’s consist of n values of 1 and m ≡ N − n values of 0, 2 with cN · = −n/N and σcN = mn/N 2 . After normalizing, c4N = (m3 + n3 )/(mnN ) ≤ 2(m ∨ n)/(m ∧ n) . Thus (29) implies that all c4N ≤ (some M ) < ∞. Since cN DN 1 →d N (0, 1) clearly fails, (12) shows that TN →d N (0, 1) if and only if D(aN ) → 0. The limiting rv (as 1 in (5)) will be ZN ≡ Z(aN ) ≡ − 0 W dKN /σaN ∼ = N (0, 1). Now deﬁne √ ¯ 1 n (Xn − aN · ) (31) TN ≡ TN (aN ) ≡ = − 0 RN dKN /σaN ∼ = (0, N/(N − 1)) , σaN 1 − n/N √ ¯ n (Xn − aN · ) (32) TˆN ≡ TˆN (aN ) ≡ . Sn 1 − n/N Theorem 2.3 (Simple random sampling) Suppose (29) holds, and suppose the arrays A are uniformly negligible with supA D(aN ) → 0 (as in (30)). Then (33)

supA |TN (aN ) − Z(aN )| →p 0,

(34)

1 supA | σaN Sn − 1| →p 0,

(35)

TˆN (aN )−Z(aN ) →p 0;

while

supA |TˆN (aN )−Z(aN )| →p 0 if (36).

The uniform convergence conclusion in (35) holds if N/(N +1) N· (36) supN 1/(N +1) g(t) d KN (t)−a <∞ σaN for g(t) = b(t) [t ∧ (1 − t)]1/2 and b(t) = b(1 − t) = 1 ∨ [2 log2 1/t]1/2 for t ∈ [0, 1/2].

2.

LINEAR RANK STATISTICS AND FINITE SAMPLING

431

Exercise 2.3 (a) Show that (30) and (36) both hold whenever the qfs KN (·) have a uniformly bounded 2+δ moment for any δ > 0. (b) Devise a “logarithmic moment” that will suﬃce. Proof. Now, (33) follows from (9). Consider (34). Let dN ≡ m/nN. Simple algebra (start on the right) gives ¯ 2 1 n 2 2 2 Xn − aN · Sn2 − σaN 1 (Xi − aN · ) − σaN 2 n − (a) = ≡ I2n − I1n . 2 2 σaN σaN σaN Using Chebyshev’s inequality with the ﬁnite sampling variance of (9.1.9) yields (b)

¯ n ]/ 2 σ2 = [1 − P (|I1n | ≥ ) ≤ Var[X aN

2 n−1 N −1 ]/n

→ 0.

Letting Yi ≡ (Xi − aN · )2 , we use (9.1.9) again for (c) (d)

2 4 P (|I2n | ≥ ) = P (|Y¯n − µY | ≥ σaN ) ≤ Var[Y¯n ]/ 2 σaN

≤

σY2 n − 1 E(X − aN · )4 m 1− ≤ 4 4 2 n σaN N −1 n 2 σaN N −1 N

= (e)

1

2 (Xi − aN · )4 /N m 2m σaN 2 ≤ D ( a ) N 4 2 2 n σaN N −1 σaN 2 n

≤ D2 (aN )

2m →0

2 n

by (29).

Thus (34) holds. Then (33) and (34) gives the ﬁrst claim in (35) via Slutsky’s theorem. (Note that (29) uniformly bounds the ratio m/n ). The second claim made in (35) will now follow from the identity (f)

(TˆN − ZN ) = (TN − ZN ) × {[(σaN /Sn ) − 1] + 1} + ZN × [(σaN /Sn ) − 1]

provided we show that (37)

supN |Z(aN )| = Op (1).

From (k) in the proof of proposition 16.1.1 we see that (36) implies (37).

2

432

3

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

The Bootstrap

Notation 3.1 We will herein justify the standard bootstrap and the Bayesian bootstrap of the sample mean. Let Xi ≡ F −1 (ξi ), where ξ1 , . . . , ξn are independent Uniform(0, 1) on (Ω, A, P ), so that X1 , . . . , Xn are iid F . Let Gn denote the usual empirical df of ξ1 , . . . , ξn and let Fn denote the empirical df of X1 , . . . , Xn . We also let K ≡ F −1 and Kn ≡ F−1 n . Note that −1 Fn ≡ Gn (F ) and Kn ≡ F−1 n = K(Gn ). √ Of course, Un (t) ≡ n [Gn (t)−t] on [0, 1] denotes the empirical process of ξ1 , . . . , ξn . ∗ ∗ We now let ξn1 , . . . , ξnn be independent Uniform(0, 1) rvs on an (Ω∗ , A∗ , P ∗ ) independent of the (Ω, A, P √ ) of the preceding paragraph. Let G∗n denote their ∗ empirical df and let Un (t) ≡ n [G∗n (t) − t] on [0, 1] denote their empirical process. For the bootstrap sample, we note that

(1)

(2)

∗ ∗ ∗ Xni ≡ F−1 n (ξni ) = Kn (ξni ),

F∗n

denote the empirical df of We let the empirical qf. Note that (3)

F∗n = G∗n (Fn )

and

1 ≤ i ≤ n, is an iid sample from Fn (·). ∗ ∗ Xn1 , . . . , Xnn ,

and we let K∗n ≡ F∗n −1 denote

∗ −1 K∗n = Kn (G∗n −1 ) = K(G−1 )). n (Gn

Example 3.1 (The bootstrap principle) the df F . We go further and set (4) θ ≡ θ(F ), θˆn ≡ θ(Fn ), and

2

Let θ(F ) denote some functional of θn∗ ≡ θ(F∗n ).

Suppose that √ (5) n (θˆn − θ) →d N (0, V 2 ), where V ≡ V (F ) is some functional on the class of dfs. The bootstrap principle is the claim that √ (6) n (θn∗ − θˆn ) →d N (0, V 2 ) on (Ω∗ , A∗ , P ∗ ) for a.e. X1 , X2 , . . . . The less interesting weak bootstrap principle is the claim that √ (7) n (θn∗ − θˆn ) →d N (0, V 2 ) on (Ω × Ω∗ , A × A∗ , P × P ∗ ). The weak bootstrap principle tells us that most of the data sets we might have obtained in our original experiment necessarily lead to an accurate Monte Carlo approximation. The bootstrap principle tells us that (with probability 1) the one data set we are actually observing leads to an accurate Monte Carlo approximation. We let E∗ and Var∗ denote means and variances computed for a true df of Fn . 2 Notation 3.2 It is convenient to require (see (12.10.32)) that we are using a Brownian bridge U∗ for which one can claim = =1−1/n = ν ∗ 1 = ∗ 2 −ν = (8) ∆∗νn ≡ = n (U − U )/[I(1 − I)] = Op (1) on (Ω∗ , A∗ , P ∗ ) n = = 1/n

1 4.

for any ﬁxed 0 < ν < [We use this construction to prove the theorems, but all the →d conclusions derived below hold without it being true.] 2

3.

THE BOOTSTRAP

433

Example 3.2 (Bootstrapping the mean) 1 µ ≡ µ(F ) ≡ 0 K(t) dt . (9)

The true mean µ (or θ) is given by

¯ n satisﬁes The sample mean X ¯ n ≡ 1 n Xi = 1 Kn (t) dt = 1 K dGn , X i=1 n 0 0 (10) 1 ∗ 1 ∗ ∗ ¯ Xn = 0 Kn (t) dt = 0 Kn dGn

while

is the mean of the bootstrap sample. Note also that n ∗ ¯ n∗ ) = E∗ Xn1 ¯n. (11) = i=1 Xi /n = X E∗ (X We consider also the sample variance n ¯ n )2 /n = 1 [Kn (t) − 1 Kn (s) ds]2 dt. Sn2 ≡ i=1 (Xi − X (12) 0 0 ¯ n and S 2 are the mean and variance of the df Fn . Now, Note that X n √ ∗ ∗ ∗ ¯n − X ¯ n )/Sn = (13) n (X (0,1) Kn dUn /Sn = − (0,1) Un dKn /Sn =−

1 0

U∗ dKn /Sn −

1 0

(U∗n − U∗ ) dKn /Sn

n ) − γn . ≡ Z(X

(14)

n ) is just a ﬁnite linear combination of the jointly Now, for each ﬁxed ω, the rv Z(X ∗ ∗ n ) is normal with mean 0. From normal rvs U (1/n), . . . , U (1 − 1/n), and thus Z(X Fubini’s theorem, (15)

n )] = [the variance of Fn ]/S 2 = 1. Var∗ [Z(X n

Thus (16)

n) ∼ Z(X = N (0, 1)

on (Ω∗ , A∗ , P ∗ ).

Just as Sn2 is the variance of Fn , the variance Sn∗2 of F∗n can be represented as n ∗ (17) Sn∗2 ≡ i=1 (Xni − Xn∗ )2 /n. Let us deﬁne (18)

∗ Mni ≡ G∗n (i/n) − G∗n ((i − 1)/n)

for 1 ≤ i ≤ n.

Note that (19)

∗ ∗ ∗ Mni ≡ (Mn1 , . . . , Mnn )∼ = n−1 Multinomial(n; 1/n, . . . , 1/n).

It is interesting to observe (compare to (25) below) that 1 n ∗ ∗ (20) Xn = Kn dG∗n = Xn:i Mni . 2 0

i=1

434

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

Theorem 3.1 (Bootstrapping the sample mean) n) ∼ (a) Whenever EX 2 < ∞, we have (with Z(X = N (0, 1)) √

(21)

Zn∗ ≡

¯∗ − X ¯ n )/Sn =a Z(X n ) on (Ω∗ , A∗ , P ∗ ) for a.e. X1 , X2 , . . . , n (X n

(22)

Sn∗ /Sn →p∗ 1

on (Ω∗ , A∗ , P ∗ ) for a.e. X1 , X2 , . . . .

(b) Whenever F ∈ D(Normal) we have √

¯ ∗ −X ¯ n )/Sn =a Z(X n) ∼ n (X = N (0, 1) n

(23)

Zˆn∗ ≡

(24)

Sn∗ /Sn →p×p∗ 1

on (Ω×Ω∗ , A×A∗ , P ×P ∗ ), on (Ω×Ω∗ , A×A∗ , P ×P ∗ ).

(c) Note that (22) and (24) allow Sn∗ to replace Sn in (21) and (23), as is necessary in order to bootstrap situations in which the true “variance” is unknown. [There is n ) →a.s. 0 uniformity in these conclusions across families F of dfs F over which D(X n ) →p 0 uniformly (regarding (b)).] uniformly (regarding (a)) or D(X Theorem 3.2 (Converse) (a) If P (Zn∗ ≤ z |X1 , X2 , . . .) →a.s. P (N (0, 1) ≤ z), then Var[X] < ∞. (b) If P (Zn∗ ≤ z |X1 , X2 , . . .) →p∗ P (N (0, 1) ≤ z), then F ∈ D(Normal). Example 3.3 (Bayesian bootstrap) The so-called Bayesian bootstrap results when we set n ≡ n − 1 and deﬁne (compare to (20)) (25)

Xn∗

1

≡ 0

Kn dG∗−1 n + =

n

∗ ∗ Xn:i [ξn−1:i − ξn−1:i−1 ];

i=1

∗ this inverse empirical G∗−1 n + is here deﬁned to equal ξn−1:i on [i/(n+1), (i+1)/(n+1)) ∗ ∗ for 0 ≤ i ≤ n, with ξn−1:0 ≡ 0 and ξn−1:n ≡ 1. √ Let us deﬁne the quantile process V∗n (t) = n [G∗−1 n + (t) − t] on (0, 1), and let V∗ ≡ −U∗ . Now, √ ¯ n )/Sn = (26) n ( Xn∗ − X Kn dV∗n /Sn = − V∗n dKn /Sn (0,1)

=− = (27)

1

1 0

0

V∗ dKn /Sn −

U∗ dKn /Sn −

n ) − γn . ≡ −Z(X

1 0

1 0

(0,1)

(V∗n − V∗ ) dKn /Sn

(V∗n − V∗ ) dKn /Sn

2

Corollary 1 (Bayesian bootstrap) Analogues of (21) and (23) hold for the Bayesian bootstrap (or any other bootstrap satisfying (c) in the proof below).

3.

THE BOOTSTRAP

Proof. (a)

435

Consider theorem 3.1. Consider γn in (14). Noting (8), |γn | ≤ ∆∗νn ×

−ν

[1/n,1−1/n] n

1

¯ n ]/Sn [t(1 − t)] 2 −ν d[Kn (t) − X

% $ ¯ n | √ |Kn ((1 − 1/n)+ ) − X ¯n| √ 9/ν (1/n) − X |K n √ √ ≤ ∆∗νn × + r + r rν n Sn n Sn (28)

√ √ ¯ n |/ n Sn ]} ≤ ∆∗νn × { 9/ν /rν + 2 r [ max |Xi − X 1≤i≤n

gives (b)

|γn | → 0,

√ n ) = max |Xi − X ¯ n |/ n Sn → 0. D(X

provided that

1≤i≤n

Thus, proving (21) and (23), we cite (16.2.19) and (16.2.20) (or theorem 11.4.1) for the conclusions (29)

γn →p∗ 0

(30)

γn →p×p∗ 0

for a.e. ω ∈ Ω

n ) → 0 a.s. P when EX 2 < ∞, since D(X n ) →p×p∗ 0 when F ∈ D(Normal) . since D(X

Consider the Bayesian bootstrap in (27). Then (28), (29), and (30) also hold for γn , if we just replace ∆∗νn by = =n/(n+1) = ν ∗ = = n (Vn − V∗ ) = ∗ ∗ = (c) ∆νn ≡ = , for which ∆νn = Op (1). 1 = = = [t(1 − t)] 2 −ν = 1/(n+1)

Thus the analogues of (21) and (23) hold for the Bayesian bootstrap. Obviously, we can replace G∗n in (20) (or G∗−1 n + in (25)) by any random function ∗ ∗ ∗ Hn of ξn1 , . . . , ξnn for which H∗n equals 0 in neighborhoods of 0 and 1 (so integration by parts leaves no endpoint terms) and for which = =n/(n+1) = ν√ = ∗ ∗ = = = n [ n (Hn − I) − U ] = (d) ∆∗H ≡ = Op (1), νn = 1 = = = [I(1 − I)] 2 −ν 1/(n+1)

and still claim the analogues of (21) and (23). Now consider (22) and (24). Let (e)

¯ N ≡ K −X ¯n K

and

¯ 2 ≡ −[(Kn − X ¯ n )− ]2 + [(Kn − X ¯ n )+ ]2 . K n

Using (3), (3), and then (7.11.4) at (i) gives n ∗ ¯ 2 n ¯ n )2 /n]−(X ¯ ∗ −X ¯ n )2 }/S 2 (31) (Sn∗2 −Sn2 )/Sn2 = {[ 1 (Xni −Xn ) /n− 1 (Xni −X n n 1

¯ n ]2 d[G∗ (t) − t]/S 2 } − { [Kn (t) − X n n

(32)

={

(f)

2 ≡ I2n − I1n .

0

1 0

¯ n ] d[G∗ (t) − t]/Sn }2 [Kn (t) − X n

436

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

n ) and (12.10.36) for ∆0∗ ), Happily (see (16.2.9) for D(X νN 1 ∗ 2 2 ¯ /S (g) |I2n | ≤ 0 |Gn − I| dK n n 1

¯ 2 /S 2 } n−ν [t(1 − t)]1−ν dK n n

(h)

≤ ∆0∗ νn × {

(i)

ν 2 ¯ 2 ≤ ∆0∗ νn × {2/r + 2r [ max1≤i≤n |Xni − Xn | /nSn ]}

(33)

ν 2 = ∆0∗ νn × {2/r + 2r D (Xn )} ,

0

while simultaneously 1 ¯ n ]/Sn (j) |I1n | ≤ 0 |G∗n (t) − t| d[Kn (t) − X (34)

√ ν ≤ ∆0∗ νn × {2/r + 2r D(Xn )/ n }.

The rest is now immediate using ∆0∗ νN = Op (1) by (12.10.36), and (16.2.19) and (16.2.20). 2 N , the Proof. Consider the statements of theorem 3.2. Conditional on X √ ∗ ¯ ¯ normalized rv N (XN − XN )/SN has mean 0 and variance 1. (a) Is this conditional distribution asymptotically N (0, 1)? If so, then it must be (conditionally) that the Lindeberg–Feller condition is satisﬁed; this is true, since the second term in (14.2.10)√is always (conditionally) bounded by a larger quantity 2 N ) < and the region of /sdn = 1/ N → 0. But either D(X that satisﬁes σnk integration for LFN is void, or else we necessarily have √ ¯ N |/ N SN ]2 dFN (x) ≥ D(X N ) . (35) LFN = N [|x−X¯ N |/√N SN ≥] [ |x − X N ) → 0 for a.e. It is thus required by LFN → 0 for a.e. X1 , X2 , . . . that D(X X1 , X2 , . . . . Such convergence thus requires Var[X] < ∞ for →a.s. , by (16.2.19) (or by (10.4.8) applied to the iid rvs Xk2 ). (b) We can go to subsequences regarding (2.3.15). We thus require LFN →p∗ 0, N ) →p∗ 0. Such convergence thus requires which by (a) means that we require D(X F ∈ D(Normal), by (16.2.20) (or by (10.6.40)). 2

4.

L-STATISTICS

4

L-Statistics

437

Let K ≡ F −1 , and deﬁne Xni ≡ K(ξni ), for 1 ≤ i ≤ n, in terms of the Uniform (0, 1) rvs ξn1 , . . . , ξnn of notation 16.1.1. Then Xn1 , . . . , Xnn are iid F , and we let Xn:1 ≤ · · · ≤ Xn:n denote the order statistics. Suppose the statistician speciﬁes a known and left-continuous function h, known constants cn1 , . . . , cnn , and known integers 0 ≤ kn < n − kn ≤ n. We wish to establish the asymptotic normality of the trimmed L-statistic

Ln ≡

(1)

Ln (kn , kn )

n−k n−k 1 n 1 n ≡ cni h(Xn:i ) = cni H(ξn:i ), n n i=kn +1

i=kn +1

−1

where H ≡ h(F ) = h(K) is also and left continuous. [Other useful cases such as h(x) = x2 are dealt with by considering (H − )2 and (H + )2 separately, and then adding the results. Here H − ≡ −H · 1[H≤0] and H + ≡ H · 1[H≥0] denote the negative and positive parts of H. Thus there is no theoretical loss in now assuming that h(X) = X and H = F −1 .] Now, Gn and Un denote the empirical df and the empirical process of those specially constructed ξni ’s of notation 16.1.1, whose empirical process Un converges pathwise to the Brownian bridge U in the manner described. This will ﬁgure heavily in our proofs and in =a claims, but not in any →d claims. We need a centering constant µn for Ln . We deﬁne (2)

Jn (t) = cni

for (i − 1)/n < t < i/n and 1 ≤ i ≤ n,

where the value of Jn at the i/n points is totally inconsequential. Suppose that Jn “converges” to J in some sense. Deﬁne an ≡ kn /n, an ≡ kn /n as before, and then deﬁne centering constants 1−a 1−a (3) µn ≡ an n Jn (t)H(t) dt and µ0n ≡ an n J(t)H(t) dt 1 i/n n−k with µ0 ≡ µ0n (0, 0) = 0 J(t)H(t) dt. Note that µn = i=knn+1 cni (i−1)/n H(t) dt, which means that kn = 0 and cn1 > 0 (that kn = 0 and cnn > 0) entails the added requirement that EH − (ξ) be ﬁnite (that EH + (ξ) be ﬁnite), for ξ ∼ = Uniform (0,1). [Our main interest is in µn , while µ0n is secondary; µn is the data analysis constant, while µ0n is just a constant for theory.] It is convenient to assume that on (0, 1) Jn ≥ 0,

(4)

J ≥ 0 is continuous,

and H is and left continuous.

[More generally, we can apply our results for two separate J functions and then just subtract the results.] Now specify a· ∈ (0, 1) to satisfy H+ (a· ) = 0, and deﬁne (5) K(t) ≡ J(s) dH(s),

(a· ,t)

where (a· ,t) ≡ − [t,a· ] . (But set a· = 0 if H(·) ≥ 0, and use [0,t) in (5); and set a· = 1 if H(·) ≤ 0, and use [t,1] in (5).) Thus (in case (5)) (6)

K is and left continuous on (0, 1) with

K+ (a· ) = 0,

438

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

where K+ is the right-continuous version, and ∆K ≡ K+ − K. [Since Ln − µn is invariant under vertical shift, there is actually no theoretical loss in also assuming as we did above that H satisﬁes H+ (a· ) = 0.] Since K is a qf, the unobservable rvs (7) Yni ≡ K(ξni ), for 1 ≤ i ≤ n, are iid with qf K, and let Y¯n ≡ n1 ni=1 Yni . The most historically important case obtains when (8)

σ2 ≡ Var[K(ξ)] ∈ (0, ∞),

µ ≡ EK(ξ),

and

kn = kn = 0.

In this case we would desire to show that (on the same probability space where the special ξni ’s above are deﬁned) for some N (0, 1) rv that we will denote by ZK (or 1 alternatively, and suggestively, we will also denote by 0 K dU/σ) we have 1 √ √ (9) n [Ln (0, 0)−µn (0, 0)]/σ =a n (Y¯n −µ)/σ =a ZK ≡ 0 K dU/σ ∼ = N (0, 1). We would also like a consistent estimator of σ, and we might want to be able to 1 replace µn (0, 0) by µ0 = 0 J(t)H(t) dt. Of course, Jn will have to approximate J suﬃciently closely; see (23) below. ˜ n denote K Winsorized outside (an , 1 − a ), and deﬁne the unobservable Let K n Winsorized rvs (10)

˜ n (ξni ) Y˜ni ≡ K

for 1 ≤ i ≤ n.

˜ n and mean µ Then Y˜n1 , . . . , Y˜nn are iid with qf K ˜n and variance σ ˜n2 given by 1 ˜ n (t) dt and σ ˜ n (ξ)] . (11) µ ˜n ≡ EY˜ni = 0 K ˜n2 ≡ Var[Y˜ni ] = Var[K (We can allow only kn = 0 or kn = 0 if the variance double integral is ﬁnite.) Let 1 n ˜ n dGn . (12) Y˜n· ≡ Y˜n (an , an ) ≡ n1 i=1 Y˜ni = 0 K In this case it is our desire to show that 1 1 √ √ ˜ n dUn /˜ ˜ n /˜ (13) n (Ln − µn )/˜ σn =a n (Y˜n· − µ ˜ n )/˜ σn = 0 K σn = − 0 Un dK σn (14)

=a ZK (an , an ) ≡

1 0

˜ n dU/˜ K σn ∼ = N (0, 1) .

We also seek an appropriate estimator of σ ˜n , and we may want to be able to 1 ˜ n dU to replace µn by µ0n . Whenever kn ∧ kn ≥ 1, we always deﬁne the symbol 0 K 1 ˜ mean − 0 U dKn (that is, pathwise integration for each ω) for all qfs, and even if kn ∧ kn = 0, this holds under condition (16.1.16). Make throughout without further comment the rather modest assumptions that an and an satisfy lim inf(1 − an − an ) > 0 and lim inf σ ˜n = lim inf σK (an , an ) > 0 for the df F or F0 under consideration at the particular moment. The ﬁrst says that we will deal with averaging, rather than just “quantiles” (though we could easily have added in ﬁxed quantiles had we chosen to do so). The second says that the statistician has at least enough insight to avoid removing all the variation. Let Fn be the empirical df of Xn1 , . . . , Xnn , and let F−1 n denote the empirical qf. Because of our theoretical convention (made below (1)) that h(x) = x, we have −1 the identity Hn ≡ h(F−1 n ) = Fn .

4.

L-STATISTICS

439

Estimation and Studentization Desiring an invariant estimator θˆn of a location parameter θn , we consider n−k θˆn ≡ Ln /Cn , θn ≡ µn /Cn , τ˜n ≡ σ (15) ˜n /Cn , where Cn ≡ i=knn+1 cni /n. But it is suﬃcient (and more convenient) to study Ln instead, because √ ˆ √ (16) n (θn − θn )/˜ τn = n (Ln − µn )/˜ σn . In asymptotic theory we may wish to replace Cn by a “limiting” value, but not in data analysis. We now introduce a fundamentally important statistical idea. Since the Y˜ni ’s of (10) are unobservable, we now deﬁne an observable pseudo-sample Yˆn:i of the order statistics of these Y˜ni ’s, by estimating them. Roughly, we want to estimate . K(·) − θn via dK = Jn dHn , where Jn takes on the n values cni and where dHn would give n increments, provided that we were to add θˆn to the n values Xn:i to form n + 1 values with n increments (and I will denote the index where θˆn falls among the Xn:i ’s). Once we have estimated K, it is trivial to studentize Ln or θˆn . Using essentially this idea, an ordered pseudo-sample of size n from K(·) − θn is deﬁned by  i cn,I+1 (Xn:I+1 − θˆn ) + j=I+2 cnj (Xn:j − Xn:j−1 )       for I + 1 ≤ i ≤ n, ˆ (17) Yn:i ≡  −c (θˆ − X ) − I−1 c (X  nI n n:I n:j+1 − Xn:j )  j=i nj    for 1 ≤ i ≤ I, where (note that θˆn necessarily satisﬁes Xn:kn +1 ≤ θˆn ≤ Xn:n−kn , as it is a weighted average) we require (18)

I to be the maximum integer (not exceeding n − kn ) for which Xn:I ≤ θˆn < Xn:I+1 .

˜ n denote Kn Let Kn denote the qf of this ordered pseudo-sample, and then let K ˜ Winsorized outside (an , 1−an). Thus Kn is the qf of the (kn , kn )-Winsorized ordered pseudo-sample (19)

Yˆn:kn +1 = · · · = Yˆn:kn +1 ≤ · · · ≤ Yˆn:n−kn = · · · = Yˆn:n−kn .

This Winsorized ordered pseudo-sample has observable mean and variance 1 ˜ ˜ n (t) dt = 1 {kn Yˆn:k +1 + n−kn Yˆn:i + k Yˆn:n−k } (20) Yˆ n:· ≡ 0 K n n i=kn +1 n n and V˜n2 ≡ V˜n2 (kn , kn ) ≡ (21)

=

1 n {kn

1 0

2 ˜ ˜ n (t)]2 dt − ( 1 K [K 0 n (t) dt)

n−k ˜ˆ 2 2 2 2 Yˆn:k + i=knn+1 Yˆn:i + kn Yˆn:n−k } − Y n:· . n +1 n

Let Vn2 denote V˜n2 when kn = kn = 0.

440

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

√ Roughly, (13) that n (Ln −µn ) acts asymptotically like the unobservable √ states sample average n (Y˜n· − µ ˜ n ), while V˜n provides an observable studentization of the observable estimator Ln of µn . Thus (9), (13), (14), (17), (21), and (22) below are important statistical statements in this section. (And since our asymptotic results will be highly uniform in F , kn , and kn , they can even cover both random trimming based on the data and bootstrapping.) Summary Let us summarize the statistical aspects of what we have done so far. Specify kn and kn to meet our perceived needs/fears. The statistic θˆn = Ln /Cn is a natural estimator of the parameter θn = µn /Cn . To studentize this estimator we form √ the estimated residuals Yˆn:i of (17). Since (14) (once we prove it) will imply that n (Ln − µn )/σn behaves asymptotically as does the normalized sample mean Y˜n· of the truncated rvs Y˜ni of (10), we similarly truncated the Yˆn:i ’s and formed ˜ n = V˜n /Cn . Once proved, the the sample variance V˜n2 of them as in (21). Let W conclusion √ √ ˜ n = n (Ln − µn )/V˜n →d N (0, 1) (22) n (θˆn − θn )/W will allow asymptotic conﬁdence intervals for θn . Choosing weights cni that greatly broaden this asymptotic normality by decreasing the inﬂuence of the extreme order statistics is one realistic possibility. Indeed, trimming a slowly growing number of ˜ into the domain of asymptotic normality of such Ln order statistics brings all D (recall (7.10.33)).

Smoothness of the Scores How well must Jn approximate the limiting J function? At ﬁrst sight you will think that the following condition is dirty and ugly. Do not be deceived. First, example 4.1 below shows that this smoothness condition holds in all the standard examples. Second, the condition was formulated with the following point of view in mind. The statistician typically chooses J, its approximation Jn , and the function h. Nature chooses only F −1 , which is necessarily and left continuous. Thus we want to put all the burden of smoothness requirements on J and Jn . Our requirement (23) is tailored to be the minimum that the proofs require. Indeed all our theorems follow in a very eﬃcient fashion. The reason that the condition is so very eﬃcient is that as soon as we apply it, both the asymptotically important term and the asymptotically negligible terms depend on J, Jn , h, and F −1 only in terms of K. The behavior of the lead term is governed by well-known necessary and suﬃcient conditions, and under these same conditions the remainder terms are shown to be asymptotically negligible. We now assume that Jn approximates J closely enough that for each 0 < λ ≤ 1 we can choose Mλ and M so large that for all n we have  |Jn (s) − J(t)| ≤ Mλ [t(1 − t)]−1 J(t)[|s − t| + 1/n]     for all λan ≤ t ≤ 1 − λan  and all s with λt ≤ s ≤ t/λ, λ(1 − t) ≤ 1 − s ≤ (1 − t)/λ, (23)   and |s − t| ≤ 1/λ,    and also cn1 ≤ M J(t) and cnn ≤ M J(1 − t) for 0 ≤ t ≤ 1/n and all n.

4.

L-STATISTICS

441

(On any [δ, 1 − δ] with 0 < δ < 12 we can opt to replace Mλ [t(1 − t)]−1 J(t) in (23) by just Mλ , as we point out at the appropriate place in the proof. This option is a useful possibility in case J(t) = 0 at some point t interior to (0, 1), because at such a point the bound on the right-hand side of (23) vanishes.) Basically, think of Mλ [t(1 − t)]−1 J(t) as a very crude upper bound on |J | after the mean value theorem is applied; but in fact, only the “Lipschitz condition” of (23) is needed on any [δ, 1 − δ], rather than mean value theorem requirements. The term 1/n is typically needed in [|s − t| + 1/n] only in case s and t are in equal or adjacent intervals in the partition (0, 1/n, . . . , i/n, . . . , 1). Note that the ﬁnal line in (23) implies that cn1 = 0 in case J(0) = 0 and cnn = 0 in case J(1) = 0. Example 4.1 (Suﬃcient condition on Jn (·) and J(·)) Key fact We never deal with (23) again, outside the proof, because any J(·) and Jn (·) treated in this example satisfy (23). Suppose that (24)

cni = Jn (i/n) = J(tni ) where 2 n max1≤i≤n |tni − i/(n + 1)| < 1 for all large n.

The common choices for tni like i/(n + 1) and (3i − 1)/(3n + 1) all satisfy (24). Then (23) holds if either (25) or (26) below holds, as we now note. (a) First, (23) clearly holds if (25)

J is Lipschitz on [0, 1].

Note also that if 0 ≡ a0 < a1 < · · · < am < am+1 ≡ 1 and J(ai ) ≥ 0 with J linear on all [ai−1 , ai ], then (25) necessarily holds. One such choice is J(t) = 12 − |t − 12 |. (b) Instead of (25), suppose J has a continuous derivative J on (0, 1) and for each 0 < λ < 1 there exists and Mλ for which (26)

|J (s)| ≤ Mλ [t(1 − t)]−1 |J(t)| for all λt ≤ s ≤ t/λ and 0 < t ≤ δ and 1 − δ ≤ t < 1

with J Lipschitz on [δ, 1 − δ]. Then (23) again holds. i/n (c) Condition (23) also holds for integrated scores, i.e., when cni ≡ n (i−1)/n J(t) dt (i+1)/(n+2) under (25), or when cni ≡ (n + 2) i/(n+2) J(t) dt under (26), etc. From the Karamata theorem of section 7.9, we will have (26) if a continuous J is regularly varying at 0 and 1. Also, for J the inverse of the N (0, 1) df Φ, Mills’ ratio shows that J = Φ−1 is slowly varying at 0 and 1, and so J = Φ−1 satisﬁes (26). 2

Asymptotic Normality Theorem 4.1 (CLT for L-Statistics) Suppose J(·) as in (4) is approximated by a Jn (·) as in (23) and kn = kn = 0 (recall example 4.1), so that Ln and Vn are untrimmed. Suppose also that Y ≡ K(ξ) ∼ ˜n and = (µ, σ2 ) with σ2 ∈ (0, ∞). Let µ σ ˜n be as in (11) with an = an = 1/n. Then (as in notation 16.1.1) 1−1/n √ √ (27) n (Ln − µn )/σ =a n (Y¯n − µ)/σ =a Zn (K) ≡ 1/n U dK/˜ σn ∼ = N (0, 1) , (28)

Vn /σ →p 1.

442

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

Theorem 4.2 (CLT for trimmed L-statistics) Suppose that J(·) as in (4) is approximated by a Jn (·) as in (23). Suppose the statistician protects himself by specifying trimming numbers kn and kn for which kn ∧ kn → ∞, while an ∨ an → 0 ˜ with an /an → 1, and suppose that K is in the statistical domain of attraction D (recall (7.10.33), (16.1.3), and proposition 11.3.1). Then (as in notation 16.1.1) √ √ √ (29) n (θˆn − θn )/˜ τn = n (Ln − µn )/˜ σn =a n (Y˜n· − µ ˜n )/˜ σn =− (30) (31)

1−an an

Un dK/˜ σn = −

1 0

˜ n /˜ Un dK σn ,

1−a 1 ˜ n /˜ σn = − 0 U dK σn ∼ =a Zn ≡ ZK (an , an ) ≡ − an n U dK/˜ = N (0, 1) , ˜ n /˜ W τn = V˜n /˜ σn →p 1 .

[If K ∈ D(Normal), only (kn ∧ kn ) ≥ 1 is required and an /an → 1 may be omitted.]

Uniform CLT for L-Statistics Uniformity also holds in these results. It is completely analogous to the uniformity established in section 16.1. Let H0 denote a ﬁxed qf, and then deﬁne K0 as in (5) for this H0 . Again, all rvs are as in notation 16.1.1. Recall example 4.1. Theorem 4.3 (Uniform CLT for L-statistics) Suppose that the J(·) of (4) is approximated by a Jn (·) as in (23). Let K0 , K, and A be as in notation 16.1.2 (subject to (16.1.15), but not to (16.1.16)). Now let F denote all dfs F (·) for which the K(·) of (5) falls in this class K of qfs. For the rvs of notation 16.1.1, we deﬁne 1−a 1 ˜ n /˜ (32) ZK (an , an ) ≡ − an n U dK/˜ σn = − 0 U dK σn ∼ = N (0, 1) (that is, pathwise integration necessarily makes sense, even if kn = kn = 0). The ˜ n of θn and τn given in (15)–(22), with estimators θˆn and W √ √ ˜ n /˜ (33) n (θˆn − θn )/˜ τn = n (Ln − µn )/˜ σn and W τn = V˜n /˜ σn , satisfy (34)

√ √ sup sup | n (θˆn − θn )/˜ τn − n (Y˜n· − µ ˜ n )/˜ σn | →p 0

(35)

√ sup sup | n (Y˜n· − µ ˜n )/˜ σn − ZK (an , an )| →p 0 .

A F ∈F

and

A F ∈F

If, further, condition (16.1.16) is imposed, then (36)

˜ n /˜ sup sup |W τn − 1| →p 0 ,

(37)

√ √ ˜ n − n (Y˜n· − µ sup sup | n (θˆn − θn )/W ˜ n )/˜ σn | →p 0 ,

(38)

√ ˜ n − ZK (an , a )| →p 0 sup sup | n (θˆn − θn )/W n

A F ∈F

A F ∈F

A F ∈F

(Especially!).

4.

L-STATISTICS

443

Proofs We initially prove only (27), (29), and (30), and then add (33) and (34) at the end. The rest can be found in Shorack (1997b), where (7.11.7) is applied to establish results (28) and (31), as well as (36)–(38). Proofs. (The central part of all the normality proofs) Suppose ﬁrst that kn ∧ kn ≥ 1. Let t (a) Ψn (t) ≡ 1/2 Jn (s) ds for 0 ≤ t ≤ 1. Then (b) (c)

√

n−k n [ n1 i=knn+1 cni Xn:i − µn ]/˜ σn 1−an √ H dΨ (G ) − H dΨ = n [ξn:k +1 ,ξ /˜ σn . n n n an ]

ηn ≡

n:n−kn

n

Integration by parts (plus a bit of algebra) gives (since H a.s. has no mass at ξn:kn +1 , the integral over (ξn:kn +1 , an ] relative to dH equals that over [ξn:kn +1 , an ] relative to dH) the fundamental identity ηn = − (an ,1−a ) Un J dH/˜ σn n

− − −

(an ,1−an )

√ { n [Ψn (Gn ) − Ψn ] − Un J} dH/˜ σn

(ξn:kn +1 ,an ]

√ n [Ψn (Gn ) − Ψn (an )]dH/( n σ ˜n )

(d)

≡−

(39)

=

[1−an ,ξn:n−k ) n

√

(an ,1−an )

√ n [Ψn (Gn ) − Ψn (1 − an )] dH/( n σ ˜n )

Un dK/˜ σn + γ0n + γ2n + γ2n

n (Y˜n· − µ ˜ n )/˜ σn + γ0n + γ2n + γ2n

using integration by parts in the last step. Let An denote the set (and let 1n denote its indicator function) on which Gn (t) ≥ λt on [ξn:1 , 1], (e)

Gn (t) ≤ t/λ on [0, 1],

Gn (1 − t) ≤ 1 − λ(1 − t) on [0, ξn:n )

Un ≤ 1/λ,

and

Gn (1 − t) ≥ 1 − (1 − t)/λ on [0, 1]; then P (An ) ≥ 1− /4 provided that λ ≡ λ is small enough (as in inequality 12.11.2). Note that on An we have (f)

|Jn (s) − J(t)| ≤ Mλ [ |Gn (t) − t| + 1/n] [t(1 − t)]−1 J(t)

for all s between Gn (t) and t and all λ an ≤ t ≤ 1 − λ an , as applying (e) to (23) demonstrates. (This is the reason (23) was formulated as it was.) Rewrite (f) as ν n |Gn (t) − t| 1 J(t) (g) |Jn (s) − J(t)| × 1n ≤ Mλ + 1−ν 1−ν [t(1 − t)] [nt(1 − t)] [nt(1 − t)]ν

444

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES ≤ {Mλ (∆0νn + 2/λ )} J(t)/[nt(1 − t)]ν ≡ Bn × J(t)/[nt(1 − t)]ν

(h)

for all s between Gn (t) and t

for all s between Gn (t) and t

when λ an ≤ t ≤ 1 − λ an and kn ∧ kn ≥ 1, and for ∆0νn as in notation 16.1.1. Consider γ0n . Applying (h) and letting ρ ≡ (1 − ν)/2 we have √ 1−a G (t) σn 1n × |γ0n | ≤ 1n × n an n | t n |Jn (s) − J(t)| ds| dH(t)/˜ (i)

≤ Bn ×

1−an an

|Un (t)/[nt(1 − t)]ν | dK(t)/˜ σn

= (1−ν)/2 = 1−a n =n [Gn (t) − t] = = = = Bn × = n−ν/2 [t(1 − t)](1−ν)/2 dK(t)/˜ σn = 1−(1−ν)/2 [t(1 − t)] an (j)

= Bn × ∆0ρn ×

1−an an

n−ν/2 [t(1 − t)](1−ν)/2 dK(t)/˜ σn .

Now, Bn and ∆0ρn are both Op (1), and neither depends on K. We can thus claim ˜ n ) ≡ 1−an n−ν/2 [t(1 − t)](1−ν)/2 dK(t)/˜ (40) σn → 0 . |γ0n | →p 0 if MνN (K an [See how extremely clean and tidy (40) became under condition (23)!] (Note the remark following (23) that says a bound of 1 on [δ, 1 − δ] suﬃces. Then replacing [t(1 − t)]−1 J(t) above by 1 also leads to (40).) Let us digress. (Condition (f) in the proof of theorem 16.1.1 is now established + in detail.) Let t− n and tn denote the /2 and 1 − /2 quantiles of ξn:kn +1 . Simply restate this as (k)

+ A0n ≡ [t− n ≤ ξn:kn +1 ≤ tn ]

P (A0n ) ≥ 1 − ;

has

let 10n denote the indicator function of A0n . We now examine the relationship + among t− n , tn , and an . Using the known moments of beta rvs, kn + 1 1 kn + 1 kn + 1 (l) ξn:kn +1 ∼ , (1 − ) . = n+1 n+2 n+1 n+1 Using Chebyshev’s inequality we choose c /2 so large (independent of the values of n and kn ) that

√ kn kn +1 c √ 1 n +1 n +1 an − c n ≤ n+1 − 2 n+2 kn+1 1 − kn+1 ≤ t− n (m)

≤

t+ n

≤

kn +1 n+1

+

c √ 1 2 n+2

From (m) we see that − (n) n [t+ n − tn ]/ kn ≤ 2 c

kn +1 n+1

1−

kn +1 n+1

≤ an + c

√ kn n

.

for all n and kn .

The leftmost term in (m) may be negative, so we remark further (from (e)) that (o) t− for all n and all choices of kn . n ≥ (an (1 − c / kn )) ∨ (λ an )

4.

L-STATISTICS

445

Because it is nonnegative with mean given in (n), − (p) n [Gn (t+ n ) − Gn (tn )]/ kn = Op (1). Consider γ2n . If ξn:kn +1 < an , then (q)

0 ≤ Ψn (Gn (t)) − Ψn (an ) ≤ Ψn (Gn (an )) − Ψn (an ) for ξn:kn +1 ≤ t ≤ an ,

while if an < ξn:kn +1 , then (r)

Ψn (Gn (an )) − Ψn (an ) ≤ Ψn (Gn (t)) − Ψn (an ) ≤ 0 a.s. H for an ≤ t < ξn:kn +1 .

Thus (with Bn as in (h)) we have a.s. that √ 1n 10n |γ2n | ≤ 1n 10n (ξn:k +1 ,an ] n [Ψn (Gn (an )) − Ψn (an )] dH/( n σ ˜n ) n

= 1n 10n (ξn:k

n

n +1 ,an ]

≤

(s) (t)

+ (t− n ,tn )

|Un (an )|

Gn (an ) an

J dH + σ ˜n

√ [Jn (s) − J(t) + J(t)] ds dH(t)/( n σ ˜n )

+ (t− n ,tn )

Bn |Un (an )| J dH [nt(1 − t)]ν σ ˜n

− Un (an ) √an |K(t+ 2Bn n ) − K(tn )| ≤ √ 1+ + ν an σ ˜n [n (t− n ∧ (1 − tn )) ] √ − = {Op (1)} × { an |K(t+ σn } × {Op (1)} . n ) − K(tn )|/˜

+ Note that neither Op (1) term depends on the qf K. Thus (with t− n and tn denoting the lower /2 and upper /2 percentiles of the rv ξn:kn +1 ) √ (u) |γ2n | →p 0, provided that max an |K(t∓ σn → 0. n ) − K± (an )|/˜ +−

+ By symmetry (with t− n and tn denoting the lower /2 and upper /2 percentiles of the rv 1 − ξn:n−kn ) |γ2n (v) | →p 0, provided that max an |K(1 − t∓ σn → 0. n ) − K∓ (1 − an )|/˜ +−

Compare the condition in (40) with the conditions in (u) and (v). Because of the quantile inequality (7.11.3), we see that the condition in (40) is redundant. We can thus claim (when kn ∧ kn ≥ 1) that √ (w) |ηn − n (Y˜n· − µ ˜n )/˜ σn | ≤ |γ0n | + |γ2n | + |γ2n | →p 0, provided that the left tail satisﬁes √ (x) an |K(t∓ σn → 0 n ) − K± (an )|/˜ and the right tail satisﬁes the symmetric condition (y) an |K(1 − t∓ σn → 0. n ) − K∓ (1 − an )|/˜

446

CHAPTER 16. ASYMPTOTICS VIA EMPIRICAL PROCESSES

If kn → ∞, then (o) shows that condition (x) holds if √ (41) an |K(an (1 ∓ c/ kn )) − K+ (an )|/˜ σn → 0

for all real c.

Appealing again to (o) we see that (x) also holds (for arbitrary kn ≥ 1) if √ (42) an |K(can )|/˜ σn → 0 for all 0 < c ≤ 1. 1−a σn as in (30) and (32), we next consider With ZK (an , an ) = − an n U dK/˜ √ |γ1n | ≡ | n (Y˜n· − µ ˜n )/˜ σn − ZK (an , an )| 1−an

(z)

=|−

(43)

≤ ∆νn ×

an

(Un − U) dK/˜ σn |

1−an an

n−ν [t(1 − t)]1/2−ν dK(t)/˜ σn ,

for ∆νn as in (12.10.32). The ﬁrst term in (43) is Op (1) (and does not depend on the qf K(·)), while the second term in (43) appeared earlier in (40). Thus (A)

|γ1n | →p 0

under (x) and (y).

(Compare (41) and (42) to (16.1.13) and (16.1.12).) Let us summarize everything proved so far. If (4) and (23) hold, then √ |ηn − n (Y˜n· − µ ˜n )/˜ σn | →p 0 and (44) √ | n (Y˜n· − µ ˜n )/˜ σn − ZK (an , an )| →p 0 provided that kn ∧ kn ≥ 1 and (x) and (y) hold. Thus (29) and (30) follow from (41) and (42) just as the trimmed means results of theorem 16.1.1–theorem 16.1.3 followed from (16.1.13) and (16.1.12) (again, theorem 7.10.1 comes in to play). Suppose now that kn = 0 (with kn = 0 also, only for the convenience of our notation). The identities (d) and (39) now become 1 1 (B) ηn = − 0 Un dK/σ + γ0n + 0 + 0 = 0 K dUn + γ0n 1 √ n (Y¯n − µ)/σ − 0 { n [Ψn (Gn ) − Ψn ] − Un J} dH/σ, n 2 ∼ where Y¯n = 1 Yi /n and Yi = K(ξni ) = (µ, σ ). We note that kn = 0 has − introduced the requirement that Var[K0 (ξ)] < ∞. Let 1n be as in (e), and let c ≡ c ≡ λ ≤ 1. Noting that γ0n involves integration over all of (0, 1), we let γ00n denote the contribution to γ0n obtained by integrating over (0, c/n) for the c ≡ c above. Note that on the event An of (e) we have Gn (c/n) = 0. Thus (using the last phrase of (23) at line (C) below) c/n √ √ 1n × |γ00n | = | 0 { n [Ψn (0) − Ψn (t)] − n (0 − t) J(t)} dH(t)/σ (45)

=

√

=|

c/n √ 0

n {−t Jn (1/n) + t J(t)} dH(t)|/σ

1/n √ ≤ ( n/σ) × 0 t [Jn (1/n) + J(t)] dH(t) .

4. (C)

L-STATISTICS

447

1/n √ ≤ (2M n/σ) × 0 t dK(t) 1/n 1/n √ ≤ (2M n/σ) × { |tK(t) | + 0 |K(t)| dt} 0

√ √ √ √ ≤ 2M |K(1/n)|/( n σ) + 2M n lim r | r K(r)/σ| r→0

√

√ 1/n 1/n + t K(t)/σ0 × 2M n 0 t−1/2 dt (D)

→ 0,

√ 1/n since t K(t)/σK (0, an ))0 → 0 when Var[K − (ξ)] < ∞. Moreover, if we compare 1n × |γ0n − γ00n | to (f)–(40) and then (x), we obtain √ (E) 1n × |γ0n − γ00n | →p 0 whenever |K+ (c/n) − K+ (1/n)|/ n σ ˜n → 0 . Note (D) and (E) (and (42)) for |γ0n | →p 0 with kn = 0 if Var[K − (ξ)] < ∞. We thus claim that √ (46) |ηn − n (Y¯n − µ)/σ| →p 0, with kn = kn = 0, when Var[K(ξ)] < ∞. In all cases, we have now shown that 1 √ ˜ σn − 1 U dK ˜ n /˜ (47) | n (Y¯n − µ ˜n )/˜ σn − ZK (an , an )| = | 0 Un dK/˜ σn | →p 0. 0 So, comparing (41) and (42) to (16.1.13) and (16.1.12) (and comparing (D) to (16.1.15) (taking µK = 0 without loss of generality)), we see that everything so far is uniform over K’s in any K as in (16.1.15) or in any Ku as in (16.1.14). That is, as in (34), √ √ (48) sup sup | n (Ln − µn )/˜ σn − n (Y¯n − µ ˜n )/˜ σn | →p 0. A

F ∈F

As in theorem 16.1.3, we may use proposition 16.1.1 to obtain √ ˜ n )/V˜n − ZK (an , an )| supA supF ∈F | n (Y¯n − µ (49) 1 1 ˜ n /˜ ˜ n /˜ = supA supF ∈F | 0 Un dK σn − 0 U dK σn | →p 0 when the integral condition (16.2.16) also holds. The proofs of the present variance claims are quite similar to those given in section 16.1, but appeal must now be made to (7.11.7), since the method of proof expresses the variance estimators as double integrals. 2

Chapter 17

Asymptotics via Stein’s Approach 1

U-Statistics

Let X, X1 , . . . , Xn be independent with df F . Consider the one sample U-statistic 1 (1) Un ≡ Un (H) ≡ n H(Xi , Xj ), i<j

2

where H(·, ·) is a symmetric function (without loss of generality) for which (2)

EH 2 (Xi , Xj ) < ∞

for i = j.

The symmetric function H is called the kernel of this U-statistic. Also, ∞ ∞ (3) θ ≡ EUn = EH(X1 , X2 ) = −∞ −∞ H(x1 , x2 ) dF (x1 ) dF (x2 ) .

Projection and the CLT for U-Statistics We introduce the notation H·· ≡ θ = EH(Xi , Xj ), (4)

Hij ≡ H(Xi , Xj ),

and

Hi· ≡ H·i ≡ H1 (Xi ) ≡ E{H(Xi , Xj )|Xi )} .

The rv H1 (Xi ) is key, so we give it the special name Yi . We note that (5)

Yi ≡ H1 (Xi ) = E{H(Xi , Xj )|Xi } has EYi = θ and σ12 ≡ Var[Yi ] < ∞ .

Fundamental Identity Our presentation is based on 1 Un − θ = n(n−1) i =j (Hij − H·· ) 1 = n(n−1) i =j [(Hij − Hi· − H·j + H·· ) + (Hi· − H·· ) + (H·j − H·· )] n n = n1 i=1 (Hi· − H·· ) + n1 j=1 (H·j − H·· ) 1 + n(n−1) i =j (Hij − Hi· − H·j + H·· ) (6)

= 2(Y¯n − θ) + U0n , 449

450

CHAPTER 17. ASYMPTOTICS VIA STEIN’S APPROACH

where (7)

U0n = Un (h)

with

h(Xi , Xj ) ≡ hij ≡ H(Xi , Xj )−H1 (Xi )−H1 (Xj )+θ .

Moreover, (8)

Cov[Y¯n , U0n ] = 0,

since (9)

Cov[Hk· − θ, Hij − Hi· − H·j + θ] = 0

for all k and all i = j

(if k = i and k = j, it is trivial, while if k = i = j, just average the product conditionally over Xj ﬁrst to get 0 for the integral of the second expression). Also, 2 Var[U0n ] = i =j k =l Cov[hij , hkl ]/[n(n − 1)] = 2 n(n − 1) Cov[h12 , h12 ]/[n(n − 1)]2 (10)

using the method of (9)

= 2 Var[h12 ]/[n(n − 1)] ≡ 2 σ12 /[n(n − 1)] ,

where σ12 ≡ Var[hij ] = Var[h(Xi , Xj )]. Thus (6), (8), and (10) imply that (11)

4 σ12 2 σ12 Var[Un ] = 4 Var[Y¯n ] + Var[U0n ] = + . n n(n − 1)

The following is immediate from (6), (10), and Slutsky’s theorem. Theorem 1.1 (CLT for U-statistics; Hoeﬀding) When EH 2 (X1 , X2 ) < ∞, √ (12) n (Un − θ) →d N (0, 4σ12 ) and StDev[Un ]/2σ1 → 1 . Motivation 1.1 (Projection) Consider the collections of rvs W ≡ {Wn ≡ n1 ni=1 ψ(Xi ) : Eψ2 (X1 ) < ∞} and (13) U ≡ {Un (H) : (1) and (2) hold}. Represent U geometrically as a full-dimensional space, and represent W as a lowerdimensional plane within it. Then 2(Y¯n − θ) is the projection of Un (H) − θ from U onto W, and U0n is the orthogonal vector (of residuals) left over. Note ﬁgure 1.1. [This explains the similarity to analysis of variance identities that is shown by relationships (6), (8), and (9).] 2 Exercise 1.1 (Berry–Esseen for U-statistics) easy Berry–Esseen type result (14)

F√n (Un −θ)/2σ1

E|Y1 − θ|3 − FN (0,1) ≤ √ 3 + n σ1

Use lemma 11.1.2 to verify the √

32 StDev[h(X1 , X2 )] √ . n − 1 σ1

1.

U-STATISTICS

451

U

U0n Un(H) – θ

2(Yn – θ)

W

Figure 1.1 Projection of the U-statistic Un(H ) – θ.

Example 1.1 (Sample variance) Let H(x, y) = (x − y)2 . We suppose the iid rv’s X1 , . . . , Xn have mean µ, variance σ2 , and ﬁnite fourth central moment µ4 ≡ E(X − µ)4 in (0, ∞). Then θ = 2 σ2 and Yi = E{H(Xi , Xj )|Xi } = E{(Xi − Xj )2 |Xi } (a)

= (Xi − µ)2 + E(Xj − µ)2 − 2 (Xi − µ) E{(Xj − µ)|Xi } = (Xi − µ)2 + σ 2 .

Hence (b)

σ12 ≡ Var[Yi ] = µ4 − σ 4

and

σ12 ≡ Var[h(Xi , Xj )] = Var[−2(X1 − µ)(X2 − µ)] = 4σ4 .

Note that Un is related to the sample variance Sn2 via n n 2 2 1 Un = n(n−1) i=1 j=1 (Xi + Xj − 2 Xi Xj ) (c)

=

2 n−1

n ¯ n2 ) = 2 Sn2 . ( i=1 Xi2 − n X

[Recall also (11.6.5).] Thus (11) and (12) give √ √ √ n (Sn2 − σ 2 ) =a 12 n (Un − θ) =a n (Y¯n − θ) (d) →d N (0, σ12 ) = N (0, µ4 − σ 4 ). 2 √ Exercise 1.2 (a) Evaluate a constant CF in the Berry–Esseen bound CF / n produced by (14) for the sample variance. (b) Determine that the exact variance of Sn2 is [µ4 − σ 4 (n − 3)/(n − 1)]/n, via (11).

452

CHAPTER 17. ASYMPTOTICS VIA STEIN’S APPROACH

The LIL for U-Statistics We now consider U0n in greater detail. We let Zn ≡ n(n − 1) U0n = (15) i =j h(Xi , Xj ) = i =j hij . Note that

(16)

n−1 ∆n ≡ Zn − Zn−1 = i=1 (hin + hni ) n = 2 n−1 i=1 hin = 2 i=1 [Hin − Hi· − H·n + θ] .

Let An ≡ σ[X1 , . . . , Xn ] denote the σ-ﬁeld generated by X1 , . . . , Xn . Averaging conditionally on Xn ﬁrst, as in establishing (9), we see that E(∆n |An−1 ) = 0. Thus (17)

{Zn , An }, for n ≥ 2, is a martingale,

and the martingale diﬀerences ∆n have (18)

Var[∆n ] = 4(n − 1) Var[h12 ] = 4(n − 1) σ12 ,

since E(hin , hjn ) = 0 for all i = j follows as in the veriﬁcation of (9) and/or (10). This √ is suﬃcient to establish a LIL for U-statistics, as we will now show. Letting ajek–R´enyi inequality for martingales gives bj ≡ j (j − 1) and M ≡ 2k−1 + 1, the H´ √ P ((max2k−1 <m≤2k m |U0m | ≥ ) √ = P ((max2k−1 <m≤2k |Zm |/ m (m − 1) ≥ ) m ≤ P (max2k−1 <m≤2k j=2 ∆j /bm ≥ ) ≤ (4/ 2 ) {Var[ZM ]/b2M + = (4/ 2 ) {4 σ12 (19)

M (M −1) M (M −1)2

= (4/ 2 ) {4 σ12 /2k−1 +

2k

2 j=M +1 Var[∆j ]/bj }

+

2k j=M +1

2k j=2k−1 +2

4 σ12

by H´ ajek–R´enyi

(j−1) j(j−1)2 }

4 σ12 /j(j − 1)}

= (the kth term of a convergent series). √ Thus the Borel–Cantelli lemma gives P ( m |U0m | ≥ i.o.) = 0 for all > 0, so that √ (20) n U0n →a.s. 0 as n → ∞. Thus (6) gives the LIL result that whenever EH 2 (X1 , X2 ) < ∞, √ √ √ √ n (Un − θ) 2 n (Y¯n − θ) n U0n 2 n (Y¯n − θ) = + =a.s. + o(1) 2 log2 n 2 log2 n 2 log2 n 2 log2 n (21)

→ [−2 σ1 , 2 σ1 ]

a.s.

(see (10.7.2) for →)

by using the ordinary LIL for Y¯n . Of course, the SLLN that Un →a.s. θ is an immediate corollary to this, whenever σ12 is ﬁnite.

1.

U-STATISTICS

453

A Functional CLT for Un In analogy with partial sums, the U-statistic process of the past Wn on [0, 1] is √ (22) for k/n ≤ t ≤ (k + 1)/n and 2 ≤ k ≤ n, Wn (t) ≡ k(Uk − θ)/(2 σ1 n ) with Wn (t) = 0 for 0 ≤ t < 2/n. Consider also the particular process S n of the Yk ’s deﬁned by √ k (23) k/n ≤ t < (k + 1)/n and 2 ≤ k ≤ n S n(t) = i=1 2 (Yi − θ)/(2 σ1 n ) with S n(t) = 0 for 0 ≤ t < 2/n. We now assume σ1 > 0. Suppose now that (24)

ψ ≥ 0 is on (0, δ] with 0 < δ ≤ 1.

Then (6) and (15) give

(25)

√ (Wn − S n) ψ δ0 = [nI] U0,[nI] ψ δ0 /(2 σ1 n ) √ = max2≤k≤[nδ] |ψ(k/n) Zk /(k − 1)|/(2 σ1 n ) .

Application of (16), (18), using k ≤ 2 (k − 1), and the H´ ajek–R´enyi inequality for martingales gives [nδ] P ( (Wn − S n) ψ δ0 ≥ ) ≤ (4/4 2 σ12 n) k=2 ψ 2 (k/n) Var[∆k ]/(k − 1)2

(26)

≤

8 Var[h12 ] 1 [nδ] ψ 2 (k/n) 8 Var[h12 ] δ ψ 2 (t) ≤ dt k=2 2 1/n 2

σ1 n n k/n

2 σ12 n t

=

8 σ12 n2ν 2 2 (1 − 2ν) σ1 n2ν

when ψ(t) = nν t−((1/2)−ν) with 0 ≤ ν <

Thus in setting ψ(t) = nν t−(1/2−ν) , δ = (27)

1 2

1 2

.

and = λ = (big), this inequality gives

nν (Wn − S n)/I 1/2−ν = Op (1) ,

for each 0 ≤ ν < 12 ,

and this gives a so-called functional CLT for Wn , when combined with (12.8.10).

The SLLN for U-Statistics Let An ≡ σ[Un , Xn+1 , Xn+2 , . . .], which is an sequence of σ-ﬁelds. Then

(28)

Un = E{H(X1 , X2 )|An } = E{H(X3 , X4 )|An } = i<j E{H(Xi , Xj )|An }/ n2 = E{Un |An }

for all n ≥ 2, by symmetry. Moreover, just as in the context of example 18.4.1 below, {Un , An }∞ n=2 is a reversed martingale. And a totally trivial copying of that example yields E{Un |An } → E{Un |σ[φ, Ω]} = EUn = θ, so that (29)

Un → θ

a.s. and L1 as

n → ∞.

454

CHAPTER 17. ASYMPTOTICS VIA STEIN’S APPROACH

More Examples of U-Statistics Example 1.2 (One-sample Wilcoxon statistic)

Let

H(x, y) ≡ 1[x+y≥0] . When the df F of the iid sample X1 , . . . , Xn is symmetric and continuous, we have θ = EH(Xi , Xj ) = P (Xj > −Xi ) =

1 2

,

Yi = E{H(Xi , Xj )|Xi } = 1 − F (−Xi ) = F (Xi ) 1 ∼ ). = Uniform(0, 1) ∼ = ( 12 , 12

As in (6), we can write Un − 12 = n1 i<j 1[Xi +Xj ≥0] − (2)

1 2

= 2 (Y¯n − 12 ) + U0n .

Let Vn ≡

1 n2

n n 1

1

1[Xi +Xj ≥0] .

The reader is asked in exercise 1.3 below to check the simple claim that n n2 Vn = 2 n2 Un + n1 1[Xi ≥0] = 2 1 1[Xi ≥0] i≤j 1[Xi +Xj ≥0] − n (30) = 2 Wn − 1 1[Xi ≥0] ; here Wn denotes the Wilcoxon signed rank statistic n Wn ≡ i=1 Ri 1[Xi >0] , with Ri denoting the rank of |Xi | among |X1 |, . . . , |Xn |. Then (11), (12), and the n negligibility of 1 1[Xi +Xj ≥0] (since its variance is of order n) give √ √ (31) n (Un − 12 ) →d N (0, 13 ) and n (Vn − 12 ) →d N (0, 13 ) . 2 Exercise 1.3

Verify (31). Then establish Berry–Esseen bounds for Wn and Vn .

Exercise 1.4

Derive the asymptotic distribution of Gini’s mean diﬀerence |Xi − Xj |/ n2

i<j

for an iid sample from a df F satisfying conditions imposed by the reader. Exercise 1.5 (Sample skewness) Consider the U-statistic Un having the symmetric kernel (x1 − x2 )3 . Describe all the preceding results of this section in the context of this particular U-statistic. Determine a second U-statistic that will provide a consistent estimator of the asymptotic variance of the given Un . Exercise 1.6 Repeat part one of the previous exercise with the symmetric kernel associated with tail heaviness, H(x1 , x2 ) = (x1 − x2 )4 .

1.

U-STATISTICS

455

Exercise 1.7 (Sample covariance) Let Xi ≡ (Vi , Wi ), for 1 ≤ i ≤ n, be iid rvs with (joint) df F (·). [The theory above requires no change in case the rvs Xi are vectors.] Consider the kernel H(Xi , Xj ) ≡ (Vi − Vj )(Wi − Wj )/2, and verify that n 1 2 ¯ ¯ Un = n−1 1 (Vi − Vn )(Wi − Wn ). Determine the quantities θ, Yi , σ1 , h(Xi , Xj ), 2 and σ12 . Let µ2,2 ≡ E{(V1 − µV )2 (W1 − µW )2 }. Establish the variance formula 2 Var[Un ] = {µ2,2 − [(n − 2)σV2 W − σV2 σW ]/(n − 1)}/n.

Exercise 1.8 (Kendall’s τ ) Let Xi ≡ (Vi , Wi ), for 1 ≤ i ≤ n, be independent with df F having a density f . Letting H(Xi , Xj ) ≡ sign{(Vi − Vj )(Wi − Wj )}, the resulting U-statistic τn is called Kendall’s coeﬃcient of concordance. [We note that n2 τn is the number of “concordant pairs” minus the number of “discordant 2 . Verify that when each Vi and Wi pairs.”] Determine θ, Yi , σ12 , h(Xi , Xj ), and σ12 is independent, then Var[τn ] = {2(2n + 5)}/{9n(n + 1)}. Exercise 1.9 Consider two U-statistics of the type (1), with kernels H(Xi , Xj ) ¯ i , Xj ); the observations X1 , . . . , Xn are the same in both cases. Determine and H(X ¯n . the covariance of the resulting U-statistics Un and U Exercise 1.10 Derive all the corresponding theory for a U-statistic of the form Un ≡ H(Xi , Xj , Xk )/ n3 , where the summation is understood to be over all n3 distinct indices. is a Example 1.3 (Hodges–Lehmann estimator) Suppose Dn (t) ≡ Dn (t; X) right-continuous function of t. Suppose that when t assumes its “true” value θ, the = v], and suppose quantity Dn (θ) “tries to achieve” the value v [perhaps EDn (θ; X) a.s. that the range of Dn (t; X) over all possible t necessarily straddles v. Let θˆn denote that value of t at which Dn (t) changes sign (or average the endpoints of the interval where it equals zero, if need be). We let θ n and θn denote the leftmost and rightmost values of t being averaged to form θˆn . We wish to obtain the asymptotic distribution of θˆn . We have (a)

[ θn < t] ⊂ [Dn (t) < v] ⊂ [ θn ≤ t] and [ θ n > t] ⊂ [Dn (t) > v] ⊂ [ θ n ≥ t],

so that P ( θn ≤ t) = P (Dn (t) < v) and P ( θ n ≤ t) = P (Dn (t) ≤ v) both hold, √ provided the distributions of θ n and θn are continuous. Thus, letting t ≡ θ + z/ n, √ Pθ (Dn (θ + z/ n ) < v) = Pθ (Dn (t) < v) = Pθ (θn ≤ t) (b) ≤ Pθ (θˆn ≤ t) ≤ Pθ (θ n ≤ t) = Pθ (θ n < t) √ = Pθ (Dn (t) ≤ v) = Pθ (Dn (θ + z/ n ) ≤ v). √ Thus knowing the asymptotic distribution of Dn (θ + z/ n ) when θ is true gives √ ˆ the asymptotic distribution of n (θn − θ) via √ √ (32) P ( n (θˆn − θ) ≤ z) = Pθ (θˆn ≤ t) = lim Pθ (Dn (θ + z/ n ) ≤ v), √ provided that the limiting value of Dn (θ + z/ n ) exists and is continuous in v (and provided θ n and θ n have continuous distributions). 2

456

CHAPTER 17. ASYMPTOTICS VIA STEIN’S APPROACH

Exercise 1.11 (Hodges–Lehmann) Suppose the rv’s X1 , . . . , Xn are iid with df Fθ (·) ≡ F (· − θ), for some df F that has a density f (·) that is bounded, continuous, and symmetric on the line. Deﬁne θˆn to be the “solution” of the equation Un (t) ≡ Establish that

√

1

is equal to 12 .

i<j 1[Xi +Xj >2 t]

n 2

( )

n (θˆn − θ) →d N (0, 1/[12 ( f 2 (x) dx)2 ]).

Two-Sample U-Statistics Exercise 1.12 (Two-sample U-statistics) (a) Suppose that X1 , . . . , Xm are iid F and Y1 , . . . , Yn are iid G. Deﬁne the two-sample U-statistic (33)

Umn =

1 mn

m n i=1

j=1

H(Xi , Yj ) .

The projections onto the X-space and the Y -space are given in θ ≡ EH(X1 , Y1 ) , Vi ≡ Hi· ≡ E{H(Xi , Yj )|Xi }

Wj ≡ H·j ≡ E{H(Xi , Xj )|Yj } .

and

Establish that (34)

mn m+n

(Umn − θ)/σmn →d N (0, 1)

as m ∧ n → ∞

when EH 2 (Xi , Xj ) < ∞, where we use the deﬁnition (35)

2 σmn ≡

mn m+n

{Var[V1 ]/m + Var[W1 ]/n} =

n m+n

Var[V1 ] +

m m+n

Var[W1 ] .

(b) Determine the natural analogue of (6). Exercise 1.13 (a) Establish the LIL and the Berry–Esseen result for the twosample U -statistics of the previous problem. Make the assumptions that seem appropriate for this statistic. (b) Likewise, establish a functional LIL of the past. Exercise 1.14 (Mann–Whitney statistic) Deﬁne H(Xi , Yj ) ≡ 1[Xi ≤Yj ] in the context of exercise 1.12, so that Umn is an unbiased estimator of θ ≡ P (X1 ≤ Y1 ). Suppose that the dfs F and G are continuous. 1 2 (a) Show that V1 = 1 − G(Xi ) and W1 = F (Yj ), and that σmn = 12 when F = G. ˆ (b) Use the Hodges–Lehmann idea to deﬁne an estimator θmn of θ, and obtain an appropriate asymptotic normality result. Exercise 1.15 Present the two-sample theory for a statistic having a kernel of the form H(x1 , x2 ; y1 , y2 ). Present an appropriate interesting example.

1.

U-STATISTICS

457

Related Statistics Example 1.4 (V-Statistics) Also of interest is the V-statistic deﬁned by n n Vn ≡ n12 i=1 j=1 H(Xi , Xj ) . (36) Noting that Vn = we have

(37)

√

n−1 n

Un +

1 1 nn

n

i=1 [H(Xi , Xi )

− EH(Xi , Xi )] +

n (Vn − θ) √ = n−1 n (Un − θ) n n 1 √1 + n n i=1 [H(Xi , Xi ) − EH(Xi , Xi )] +

√1 n

1 n

EH(X, X) ,

[EH(X, X) − θ] .

Thus the CLT (12), the LIL (21), and the functional CLT (27) all hold for such V-statistics as well as for U-statistics, provided that EH 2 (Xi , Xj ) < ∞ for all i, j. [The Vn of example 1.2 is a V-statistic.] 2 Exercise 1.16 (Incomplete U-statistics) (a) Suppose that the design Dn is a set of m distinct pairs of indices i < j that is balanced (in the sense that each index i appears in Dn exactly r times). Then nr = mk. An incomplete U-statistic is then deﬁned to be the restricted sum ¯n ≡ 1 (38) U (i,j)∈Dn H(Xi , Xj ) . m ¯n . Show that the previous theory carries over to such U ¯ (b) Think of Un as a simpliﬁed version of Un requiring less calculation. Give an ¯n , and work out the details. interesting example of such U Exercise 1.17 (Generalized L-statistics) We agree that WM :1 ≤ · · · ≤ WM :M are the M ≡ n2 ordered values of the H(Xi , Xj ). Consider a linear combination Ln ≡ M 1 cM i WM :i of these ordered values. You can mimic the outline of the proof of theorem 16.4.1, using exercise 1.9 to determine the covariance function of the empirical process of the WM :i , to state the desired conclusion regarding asymptotic normality of such statistics.

458

2

CHAPTER 17. ASYMPTOTICS VIA STEIN’S APPROACH

Hoeﬀding’s Combinatorial CLT

Let |[cij ]| denote an N × N matrix of constants cij ≡ cN ij . Let N H ≡ HN ≡ i=1 ciπ(i) , (1) where π ≡ (π(1), . . . , π(N )) is a random permutation of (1, . . . , N ) . We will show presently that N N N N EH = N c·· ≡ N1 i=1 j=1 cij (2) and Var[H] = N1−1 i=1 j=1 dij2 (the mean is obvious), where (3)

dij ≡ [cij − ci· − c·j + c·· ] ≡ cij − N1

N

j=1 cij

−

1 N i=1 cij N

+ N12

N N 1

1

cij .

¯ ≡ (H − EH)/StDev[H] satisﬁes Thus the result we wish to establish is that H (4)

¯ = H

√

N −1

N diπ(i) N σdN i=1

(with

2 σdN ≡

1 N2

N N 1

1

d2ij )

is asymptotically normal under appropriate hypotheses on the cij . [We always assume that the cij ’s are such that Var[H] > 0.] To state results, for real r > 0 and integral k > 0 we deﬁne N N k r (5) Mr ≡ N12 N and Mk0 ≡ N12 N i=1 j=1 |dij | i=1 j=1 dij . [Note that for “smooth” scores dij the Mr ’s should be approximately constant.] The asymptotic normality that follows is due to Hoeﬀding, but the rates of convergence are as cited. Theorem 2.1 (Hoeﬀding’s combinatorial CLT) ¯ N →d N (0, 1) if any of the bounds below goes to zero. (a) (Hoeﬀding) Now, H (b) (Shorack, herein) Suppose cij = ai bj for all 1 ≤ i, j ≤ N . Then (6)

FH¯ − Φ ≤

M4 85 N 1/2 M22

.

(c) (Stein) Consider general cij . Then (7)

FH¯ − Φ ≤

9 N 1/4

M4 M22

.

(d) (Bolthausen) Consider general cij . For some absolute constant K (we state without proof that) (8)

FH¯ − Φ ≤

M3 K N 1/2 M 3/2 2

.

Proof. Only (6) and (7) will be proved, via the alternative Stein formulations 2 of (11.1.26) and (11.1.31). It is useful to let σd2 ≡ σdN , and work with N (9) W ≡ i=1 diπ(i) = H − EH

2.

HOEFFDING’S COMBINATORIAL CLT

459

for the dij of (3). Noting that di· = d·j = 0 for all i, j gives (10)

Ediπ(i) = di· = 0

Var[diπ(i) ] = Ed2iπ(i) =

and

1 N

N

2 i=1 dij

;

while for i = j we have (11)

E[diπ(i) djπ(j) ] =

1 N (N −1)

djn = − N (N1−1)

m =n dim

N

m=1 dim djm .

Thus W = H − EH satisﬁes Var[H] = Var[W ] =

(a)

N N

=

1 N

=

1 N

=

1 N N 1 1 N

=

1

N N 1

EW 3 =

1

1

1

E d2iπ(i) +

N

i =j

E[diπ(i) djπ(j) ]

d2ij −

1 N (N −1)

d2ij −

1 N (N −1)

d2ij +

N N 2 1 m=1 i=1 dim N (N −1)

N N

1 N −1

Exercise 2.1 (12)

1

N

1

d2ij .

i =j

m=1

N

dim djm

N

m=1 {(

N

i=1 dim )(

j=1

djm ) −

N

2 i=1 dim }

since di· = 0

2

For a GLT approximation (recall section 14.4), show that W has N3 (N −1)(N −2)

M30

and

γ1 (W ) =

√1 N

N (N −1) M30 (N −2)2 M 3/2 2

.

(This is not needed in the proof below.) Proof. (a)

Consider (6). Simplify notation by renormalizing the constants so that N N 2 2 1 (that is, NN−1 M2 = 1). i=1 j=1 dij = 1 N −1

Then (b)

EW = 0,

Var[W ] = 1,

and Mr =

1 N2

N N 1

1

|dij |r .

Also, if eij ’s are arbitrary constants, then (2) yields (see also exercise 2.2 below) N N N N N Var[ 1 eiπ(i) ] = N 1−1 1 1 (eij − ei· − e·j + e·· )2 ≤ N1−1 1 1 eij2 . (c) To apply Stein’s key idea we let I and J be independent rvs that are uniformly distributed on {1, . . . , N } and are independent of the random permutation π . Let (d)

V ≡ −dIπ(I) − dJπ(J) + dIπ(J) + dJπ(I) .

Figure 2.1 clearly shows that W ∼ = W ≡ W + V . The ﬁgure also makes clear that N N ¯ (2) ≡ N N diπ(j) djπ(i) ∼ (e) W = 1 1 diπ(i) djπ(j) = W 2 . 1 1

460

CHAPTER 17. ASYMPTOTICS VIA STEIN’S APPROACH

Also (with λ ≡ 2/N ), N (f) E(V |π ) = 2[− N1 1 diπ(i) ] + 2 × 0 = − N2 W = −λW , using di· = 0. Multiplying out gives (g)

V 2 = [d2Iπ(I) + d2Jπ(J) ] + [d2Iπ(J) + d2Jπ(I) ] + 2 dIπ(I) dJπ(J) + 2 dIπ(J) dJπ(I) −2[dIπ(I) dIπ(J) + dIπ(I) dJπ(I) + dJπ(J) dIπ(J) + dJπ(J) dJπ(I) ] .

(h)

– dIπ(I )

+dIπ(J )

+dJπ(I )

– dJπ(J )

Figure 2.1 Stein’s random I, J interchange.

Since it is clear that E(dIπ(I) dIπ(J) |π ) = EI (dIπ(I) EJ (dIπ(J) |I, π )|π ) = EI (0) = 0 (and similarly for the other three terms in (h)), we obtain from (g) that E(V 2 |π ) − = { N2

N 1

+{ N22 (i)

=

2 N

4(N −1) N2

N N

N 1

2 diπ(i) −

1

1

2(N −1) N2 }

dij2 −

2 (diπ(i) −

2(N −1) N2 }

N −1 N2 )

+0+

+

2 2 N2 W

2 2 N2 W

+

2 N N 1 1 N2

2 ¯ (2) N2 W

+

;

diπ(j) djπ(i)

recall (a) and (e).

Applying (e) and (a)–(b) to (i) gives (using 4(N − 1)/N 2 = (4/N ) − (4/N 2 )) 2 2 N −1 2 2 ¯ (2) ) + E|E(V 2 |π ) − N4 | ≤ N2 E| N 1 (diπ(i) − N 2 )| + N 2 E(W ) + N 2 E(W (j)

=

2 N

E|

N 1

2 (diπ(i) −

N −1 N 2 )|

+

2 N −1

M2 +

2 N −1

M2 +

4 N2

.

Now, if we deﬁne eij ≡ dij2 , then (k)

E[

N 1

(eiπ(i) −

N −1 N 2 )]

=

1 N

N N 1

1

dij2 −

N −1 N

=0

by (a) ;

4 N2

2.

HOEFFDING’S COMBINATORIAL CLT

461

so that Liapunov’s inequality and (c) give 2 N E|

(l)

=

N 1

2 (diπ(i) −

√ 2 N −1

√

N −1 N 2 )|

≤

2 N

Var[

N 1

eiπ(i) ] =

2 N

1 N −1

N N 1

1

√ 10 √ M4 N −1

.

dij4

M4 .

Plugging (l) into (j) gives (m)

E|E(V 2 |π ) −

4 N|

≤

√ 2 M4 √ N −1

+

4M2 N −1

+

√ 2 M4 √ N −1

≤

4 N2

+

8M2 N −1

≤

Also, applying (k), (b), and (e) to (i) yields (n)

EV 2 =

4(N −1) N2

+0+0+

2 N2

+

2 N2

=

4 N

= 2λ.

Now, W ∼ = W ≡ W + V , (f) with λ ≡ 2/N , and (n) combine to give (11.1.26). Then both N N and E|dIπ(I) |3 = N1 i=1 N1 j=1 |dij |3 = M3 3 1 N N 1 N i=1 j=1 N k=1 |dik | N2

E|dIπ(J) |3 =

= M3 .

Applying the Cr -inequality twice in (d) gives both N N E|V |3 ≤ 64 M3 = N642 1 1 |dij |3 ≤ 64 M2 M4 (o)

and E V 4 ≤ 256 M4 .

Armed with (m), (n), and (o), we will now appeal to the alternative Stein formulation given in (11.1.30) (using h’s that are linear on [z, z + ]). This gives us the bound (with λ = 2/N by (f))

(13)

FH¯ − Φ $ 3/4 1/4 % E |E {V 2 |π } − E(V 2 )| E |V |3 EV 4 EW4 N ≤ 2 + 1.15 + σ2 σ3 σ4 σ4 σ2 ≡ Var[H] = 1 =

with (p)

≤

N 2 σ2

√ 10 √ M4 + N −1

1.15

N 64 2 σ3

N2 N −1

√

M2

M2 M4 +

N 64 2 σ3

3/4

M4

K

M4 M22

1/4

provided that E W 4 / σ 4 ≤ KM4 / M22 (q)

(14)

≤

√5 N

≤

5+37 √ N

≤

1/4 M4 42+32K √ M22 N

M4 M22 M4 M22

for some value of K.

+ 1.15 N2 64 +

M4 M22

3/2 { NN−1 + 2 }

M4 M22

3/2 32 K 1/4 N { NN−1 2 }

32√ K 1/4 M4 M22 N M4 provided that E W 4 / σ 4 ≤ K M 2 , 2

462

CHAPTER 17. ASYMPTOTICS VIA STEIN’S APPROACH We necessarily have (see the easy exercise 2.6 below)

(r)

EW 4 /σ 4 ≤ 3 M4 / M22 for all N ≥ 4 whenever cij = ai bj for all 1 ≤ i, j ≤ N.

Thus whenever cij = ai bj for all 1 ≤ i, j ≤ N we necessarily have (for N ≥ 4) (15)

FH¯ − Φ ≤

85 M4 √ N M22

,

M4 E W 4 / σ4 ≤ 3 M 2 .

using

2

For general cij we appeal instead to (11.1.31) and obtain the poorer bound M4 M3 M4 3.6 9 FH¯ − Φ ≤ N 51/2 (16) + . 2 2 1/4 3/2 ≤ N 1/4 M M2 N M2

2

2

Example 2.1 (Simple scores in Hoeﬀding’s CLT) (17)

cij = ai bj

implies

Note that

dij = (ai − a· )(bi − b· ) ,

N where a· ≡ 1 ai /N , etc. Note exercise 2.6 regarding (15). Thus for the resulting simple scores R-statistic, (18)

85 M4 √ N M22

=

85 A4 B4 √ N A22 B22

3 √K M3/2 N M2

and

=

B3 3 √K A3/2 N A2 B23/2

are the bounds of (6) and (8), where (19)

Ar ≡

N

i=1 |ai

− a· |r /N

and

Br ≡

N i=1

|bi − b· |r /N .

In this case (20)

N ai bπ(i) − N a· b· H − EH ¯ H≡ = i=1 = Var[H] A2 B2 N 2 /(N − 1)

N (ai bπ(i) − a· b· ) i=1 A2 B2 /(N − 1)

1 N

¯ N →d N (0, 1) as N → ∞, provided that any of the bounds goes to zero. 2 and H Example 2.2 (Spearman rank correlation coeﬃcient) (21)

κ≡

N

ai bπ(i) ,

ai ≡ bi ≡ (i −

where

Consider

N +1 2 )/

N 2 −1 12 .

i=1

The reader can show that a· = b· = 0 and A2 = B2 = 1. Then example 2.1 (using the ﬁrst bound in (18)) gives (22)

Fκ − Φ ≤

√1 N

85 A4 B4 ≤

√1 N

85( 95 )2 ≤

276 √ N

,

using (9.1.8). In particular, κ →d N (0, 1) as N → ∞.

2

2.

HOEFFDING’S COMBINATORIAL CLT

463

Example 2.3 (Correlation coeﬃcient) (a) Let (Xi , Yi ), 1 ≤ i ≤ N , be an iid sample from a bivariate distribution. Consider the sample correlation coeﬃcient RN of exercise 11.6.17. Thus N 1 ¯ ¯ SSXY i=1 (Xi − XN )(Yi − YN ) N (23) ≡ . RN ≡ SSXX SSY Y N 1 ¯ N )2 1 N (Yi − Y¯N )2 (X − X i 1 1 N N Let ρ ≡ Cov[X, Y ]/ Var[X]Var[Y ] denotes the population correlation coeﬃcient. From that exercise, √ (24) N (ˆ ρN − ρ) →d N (0, τ 2 ), where τ 2 was evaluated. In particular, τ 2 = 1 if the Xi and Yi are independent. (b) We suppose again that the joint distribution of the (Xi , Yi )’s factors so that Xi and Yi are themselves independent. Then each value of any Yio could just as well be associated with any of the Xi ’s. This causes us to regard (X1 , Y1 ), . . . , (XN , YN ) as ﬁxed numbers, and consider N ¯ N )/SSXX and bi ≡ (Yi − Y¯N )/SSY Y . (25) TN ≡ i=1 ai bπ(i) with ai ≡ (Xi − X In exercise 2.4 below we ask the reader to use Hoeﬀding’s combinatorial CLT to show that if E|X|3 < ∞ and E|Y |3 < ∞, then almost every inﬁnite sequence (X1 , Y1 ), (X2 , Y2 ), . . . is such that (26)

TN →d N (0, 1).

(c) Consider a simple linear regression model in which the slope coeﬃcient has true value zero. Then the permutation distribution of the regression coeﬃcient 2 2 βˆN ≡ SSXY /SSXX is “asymptotically” a N (0, SSY2 Y /SSXX ) distribution for a.e. 3 3 inﬁnite sequence, provided that E|X| < ∞ and E|Y | < ∞. 2 Exercise 2.2 Verify that N N 2 1 1 eij (27)

= N 2 e2·· + N +

N N 1

1

N 1

(ei· − e·· )2 + N

N 1

(e·j − e·· )2

(eij − ei· − e·j + e·· )2 .

[Many readers will recognize this identity from the analysis of variance.] Exercise 2.3 Prove theorem 17.2.1 as Hoeﬀding did, by verifying that all of the ¯ N converge to the corresponding N (0, 1) moments. [Add hypotheses, moments of H if needed, on the constants cij ’s.] Exercise 2.4

Verify (26).

Exercise 2.5 (Wilcoxon scores) Verify the claims of example 2.2. Include a demonstration that a· = 0 and A2 = 1, and evaluate A3 and A4 .

464

CHAPTER 17. ASYMPTOTICS VIA STEIN’S APPROACH

Exercise 2.6 Verify that EW 4 ≤ 3 M4 (as required for (15)) in the case when 0 cij = ai bj simple scores. [Hint. Let denote the sum over all distinct N N 2 0 2 ai aj ak , start by expanding 0 = N indices. To evaluate 1 1 1 ai aj ak , and plug inthe other moments you already know. Use this same approach to 0 evaluate ai aj ak al .] Example 2.4 (Finite sampling CLT) Let an urn contain N balls labeled with a1 , . . . , aN . Let X1 , . . . , Xn denote the result of n random draws without ¯ n ≡ n Xi /n. Then the formulas of the previous section replacement, and let X i=1 can be specialized. Thus and bn+1 = · · · = bN = 0, b 1 = · · · = bn = 1 N n so that H = 1 ai bπ(i) ∼ = 1 Xi . With m ≡ N − n we have

(28)

(29)

b· = n/N

and

B2 =

1 N {n(1

−

n 2 N)

+ m(0 −

n 2 N) }

=

mn N2 .

¯ is thus The standardized version of H (30)

¯ = (X ¯ n − a· )/ A2 [1 − H

n−1 N −1 ]/n .

Moreover, (31)

B3 =

1 N {n(1

−

n 3 N)

+ m(0 −

n 3 N) }

=

mn(m2 +n2 ) N4

,

(32)

B4 =

1 N {n(1

−

n 4 N)

+ m(0 −

n 4 N) }

=

mn(m3 +n3 ) N5

,

(33)

B √ 33/2 N B2

(34)

√ B4 2 N B2

=

=

mn(m2 +n2 ) N3 √1 N4 m3/2 n3/2 N

mn(m3 +n3 ) N 4 √1 N5 m2 n2 N

=

=

m√2 +n2 N mnN

m3 +n3 mnN 3/2

≤

≤

√m∨n mnN

(m∨n)2 √ mn N

≤

√

≤

√ 1 , m∧n

N

m∨n mn

=

√ N m∧n .

Thus the two bounds of (18) become √

(35)

N 85 m∧n A4 /A22

and

√K m∧n

3/2

A3 /A2

.

2

Exercise 2.7 Equation (10) shows that Cov[Xi , Xj ] = −A2 /(N −1) for all i = j. N Derive this fact again, by starting from 0 = Var[ 1 Xi ]. Exercise 2.8 Smith and Jones enjoy playing the following game. They ﬁll an urn with 500 balls [having 25 purple (value $10 each), 25 blue (value $8 each), 50 green (value $5 each), 50 yellow (value $2 each), 100 red (value $1 each), and 250 white (value $0 each)]. Each then chooses 25 balls at random without replacement. Let U denote the value of Smith’s sample and let V denote the value of Jones’ sample. A payoﬀ is made to the holder of the larger value only if |U − V | > 10. Determine the probability that Smith wins. Exercise 2.9 Show how to treat the Wilcoxon two-sample test statistic and/or the permutation distribution of the two-sample t-statistic via these ﬁnite sampling methods.

2.

HOEFFDING’S COMBINATORIAL CLT

465

Exercise 2.10 Consider rN random draws XN 1 , . . . , XN rN without replacement from an urn containing 2N + 1 balls numbered −N, . . . , −1, 0, 1, . . . , N . Suppose √ rN 2 that rN /N → 0 as N → ∞. Prove that TN ≡ k=1 XN k / rN →d (some rv T ), and determine the distribution of T . Exercise 2.11 Suppose that an urn contains N balls labeled with a1 , . . . , aN . the sampling result of m + n random draws Let X1 , . . . , Xm and Y1 , . . . , Yn denote ¯ m ≡ m Xi /m and Y¯n ≡ n Yj /n. Consider without replacement. Let X i=1 j=1 ¯ m + bY¯n . Normalize this statistic appropriately and give conditions under which aX the normalized statistic will converge in distribution to a N(0, 1) rv.

Chapter 18

Martingales 1

Basic Technicalities for Martingales

Notation 1.1 We will work with processes on the following time sets I: {0, . . . , n}, {0, 1, . . .}, {0, 1, . . . , ∞}, {. . . , −1, 0}, {−∞, . . . , −1, 0} in the discrete case and [0, t], [0, ∞), [0, ∞], (−∞, 0], [−∞, 0] in the continuous case. In the continuous cases we will consider only processes of the type X : (Ω, A, P ) → (DI , DI ) that are adapted to an sequence of sub σ-ﬁelds At of A. We will use the notation {an }∞ n=0 , ∞ 0 0 {an }n=0, {an }n=−∞ , {an }n=−∞ to denote sequences over {0, 1, . . .}, {0, 1, . . . , ∞}, {. . . , −1, 0}, {−∞, . . . , −1, 0}, respectively. 2 Deﬁnition 1.1 (Martingale and submartingale) Suppose E|Xt | < ∞ for all t. Call {Xt , At }t∈I a martingale (abbreviated mg) if (1)

E(Xt |As ) = Xs

a.s.

for each pair s ≤ t in I.

Call {Xt , At }t∈I a submartingale (abbreviated submg) if (2)

E(Xt |As ) ≥ Xs

a.s.

for each pair s ≤ t in I.

(If the inequality in (2) is reversed, the process {Xt , At }t∈T is then called a supermartingale.) When the index set I is a subset of the negative numbers [−∞, 0], we refer to such a process as a reversed martingale or reversed submartingale. [Most results in the ﬁrst seven sections of this chapter are due to Doob.]

Basic Technicalities Proposition 1.1 (Equivalence) Now, {Xt , At }t∈I is a submg if and only if for every pair s ≤ t we have (3) for all A ∈ As . A (Xt − Xs ) dP ≥ 0 Similarly, {Xt , At }t∈I is a mg if and only if equality holds in (3). Notation As in section 10.10, we combine these two statments by writing > (4) {Xt , At }t∈I is a s-mg iﬀ (Xt − Xs ) dP = 0 for all A ∈ As . A

467

468

CHAPTER 18. MARTINGALES

Proof. (a)

As in section 10.10, for each pair s ≤ t, > (Xt − Xs ) dP = A E (Xt − Xs |As ) dP = 0 for all A ∈ As A >

if and only if E(Xt |As ) − Xs = E(Xt − Xs |As ) = 0 a.s. As .

2

Deﬁnition 1.2 (a) Call {Xt , At }t∈I integrable if sup{E|Xt | : t ∈ I} < ∞. (b) If {Xt2 , At }t∈I is integrable, then {Xt , At }t∈I is called square-integrable. Let φ : (R, B) → (R, B) have E|φ(Xt )| < ∞ for all t ∈ I.

Proposition 1.2 (a)

If φ is convex and {Xt , At }t∈I is a mg, then {φ(Xt ), At }t∈I is a submg.

(b)

If φ is convex and and {Xt , At }t∈I is a submg, then {φ(Xt ), At }t∈I is a submg.

Proof. (a) (b)

Clearly, φ(Xt ) is adapted to At . Let s ≤ t. For the mg case, E[φ(Xt )|As ] ≥ φ(E(Xt |As ))

by the conditional Jensen inequality

= φ(Xs ) a.s.

For the submg case, (c) (d)

E[φ(Xt )|As ] ≥ φ(E(Xt |As ))

by the conditional Jensen inequality

≥ φ(Xs ) a.s.,

since φ is and E(Xt |As ) ≥ Xs a.s. Example 1.1

2

Let {Xt , At }t∈I be a martingale. Then:

(5)

{|Xt |r , At }t∈I is a submg, for any r ≥ 1 having E|Xt |r < ∞ for all t ∈ I,

(6)

{Xt− , At }t∈I is a submg,

(7)

{Xt+ , At }t∈I is a submg (even if {Xt , At }t∈I is only a submg).

[Note that φ(x) = |x|r and φ(x) = x− are convex, while φ(x) = x+ is also .] 2 Proposition 1.3 (8)

If {Xt , At }t∈I and {Yt , At }t∈I are s-mgs, then (trivially)

{Xt + Yt , At }t∈I

are s-mgs.

Exercise 1.1 If {Xtc , At }t∈I is a submg for all c in some index set C, then the maximum {Xtc1 ∨ Xtc2 , At }t∈I is necessarily a submg for any c1 , c2 ∈ C. Likewise, {supc∈C Xtc , At }t∈I is a submg, provided that E|supc∈C Xtc | < ∞ for all t ∈ I.

1.

BASIC TECHNICALITIES FOR MARTINGALES

469

Deﬁnition 1.3 (Filtration) Let (Ω, A, P ) be a complete probability space. Let N ≡ {N ∈ A : P (N ) = 0}. Let {At : t ≥ 0} be such that the At ’s are an sequence of σ-ﬁelds with At = Aˆt = At+ for all t ≥ 0 (here Aˆt ≡ σ[At , N ] and At+ ≡ ∩{Ar : r > t}). [That is, they are complete and right continuous.] Such a collection of σ-ﬁelds is called a ﬁltration. Notation 1.2 (Completeness assumption) Throughout this chapter we will make the assumption that the σ-ﬁelds At form a ﬁltration. If {Xt , At }t∈I is a s-mg, then {Xt , Aˆt }t∈I is also a s-mg; note exercise 1.2 below. We assume throughout the present chapter that this completion has already been performed on the σ-ﬁelds labeled At . Thus, from exercise 12.4.1 on properties of stopping times we see that S ≤ T a.s. implies AS ⊂ AT ; note exercise 1.2 below. 2 Exercise 1.2

Verify the two claims made in the previous assumption.

Exercise 1.3 If X is a process on (D, D) or (D[0,∞) , D[0,∞) ), then the histories σt ≡ σ[Xs : s ≤ t] are right continuous, as are the σ ˆt ≡ σ[σt ∪ N ]. (Recall proposition 12.4.3.) Remark 1.1 All deﬁnitions and results in this section make sense for processes on the measurable space (RI , BI ). 2

Some Examples Example 1.2 (Sums of iids) Let X1 , X2 , . . . be iid with E(Xi ) = 0, and let Sn ≡ X1 + · · · + Xn and An ≡ σ[S1 , . . . , Sn ]. Then E|Sn | ≤ ni=1 E|Xi | < ∞ and so {Sn , An }∞ 2 n=1 is a mg. Example 1.3 As in example 1.2, but now assume that E(Xk2 ) ≡ σ2 < ∞. 2 Let Yn ≡ Sn2 − nσ 2 . Then {Yn , An }∞ n=1 is a mg. Note that Sn is a submg by proposition 1.2, and that we have written Sn2 = (Sn2 − nσ 2 ) + nσ2 = (martingale) + (increasing process). This is an example of the Doob decomposition of a submartingale, which we will establish in section 5. 2 Example 1.4 Suppose µ ≡ E(Xi ) > 0 in example 1.2. Then the partial sums {Sn , An }∞ 2 n=1 form a submg. Example 1.5 (Wald’s mg) Consider again example 1.2, but now suppose that the Xk ’s have a mgf φ(t) = E exp(tX). Let Yn ≡ exp(cSn )/φ(c)n . Then {Yn , An }∞ n=1 is a mg. Note that the mg of example 1.2 is recovered by diﬀerentiating once with respect to c and setting c = 0; the mg of example 1.4 is recovered by diﬀerentiating twice with respect to c and setting c = 0. 2 Example 1.6 (Brownian motion) Let {S(t) : t ≥ 0} be standardized Brownian motion, and let At ≡ σ[S(s) : s ≤ t]. Then {S(t), A t : t ≥ 0} is a mg. 2

470

CHAPTER 18. MARTINGALES

Example 1.7 Let Y (t) ≡ S(t) 2 − t in example 1.6. Then the Brownian motion transform {Y (t), At : t ≥ 0} is a mg. 2 Example 1.8 (The exponential mg for Brownian motion) As in example 1.6, let S denote standard Brownian motion, and much as in example 1.5, set Y (t) = exp(cS(t))/ exp(c 2 t/2) = exp(cS(t) − c 2 t/2) . Diﬀerentiating once with respect to c and setting c = 0 yields the mg of example 1.6; diﬀerentiating twice with respect to c and setting c = 0 yields the mg that appears in example 1.7; higher-order derivatives yield mgs based on the Hermite polynomials. (Recall (12.7.13) and (14.5.15).) 2 Example 1.9 2

Let X ∈ L1 and An be σ-ﬁelds. Then Yn ≡ E(X|An ) is a mg.

Example 1.10 (A simple counting process) Suppose that Y ≥ 0 has df F . Let N(t) ≡ 1[Y ≤t] and A ≡ σ[N(s) : 0 ≤ s ≤ t]. Then N is , and hence is a submg. The reader will soon be asked to show (giving a decomposition) that t (9) M (t) ≡ N(t) − −∞ 1[Y ≥s] dΛ(s) is a mg, x 2 where Λ(x) ≡ 0 [1 − F− ]−1 dF is called the cumulative hazard function. Example 1.11 (Another counting process) Suppose that the rvs ξ1 , ξ2 , . . . are iid Uniform(0, 1), and let Nn (t) ≡ nGn (t) ≡ (the number of ξi ’s ≤ t). Then Nn is a counting process, since it is , and increases only by jumps upward of size +1. Hence it is a submartingale. The reader will be asked to show (giving another decomposition) that t (10) Mn (t) ≡ Nn (t) − 0 {n[1 − Gn− (t)]/(1 − t)} dt is a martingale. 2 Example 1.12 (Poisson process) Suppose N(t) is a Poisson process with rate λ > 0. It is a counting process and hence a submartingale. Moreover, the process 2 M (t) ≡ N(t) − λt is a martingale, and the process M (t) − λt is also a martingale. 2 Example 1.13 (Likelihood ratios) Let (Ω, A, P ) and (Ω, A, Q) be probability spaces for Q and P . Suppose that An is an sequence of sub σ-ﬁelds of A. Suppose Qn and Pn denote the measures Q and P respectively restricted to An , and suppose that Qn Pn . Let Xn ≡ dQ n /dPn . Then for A ∈ An and n > m we have A Xn dP = Qn (A) = Qm (A) = A Xm dP , so that A (Xn − Xm ) dP = 0. This shows that (11)

{Xn , An }∞ n=1 is a mg of likelihood ratios.

2

Example 1.14 (Kakutani’s mg) Let X1 , X2 , . . . be independent rvs with each n Xk ≥ 0 and E Xk = 1. Let Mn ≡ 1 Xk , for 1 ≤ k ≤ n, with M0 ≡ 1. Then Mn is a mg with all E Mn = 1. 2

1.

BASIC TECHNICALITIES FOR MARTINGALES

Exercise 1.4

Verify the martingale claim made in example 1.10.

Exercise 1.5

Verify the martingale claim made in example 1.11.

471

Exercise 1.6 Find the exponential martingale that corresponds to the mg M (t) in example 1.12. Then diﬀerentiate this twice with respect to c, set c = 0 each time, and obtain the two mgs given in the example.

472

2

CHAPTER 18. MARTINGALES

Simple Optional Sampling Theorem

The following proposition gives a particularly simple special case of the optional sampling theorem. Proposition 2.1 (Optional sampling) If {Xn , An }∞ n=0 is a s-mg and stopping times S and T satisfy 0 ≤ S ≤ T ≤ N a.s. for some ﬁxed integer N , then (1)

E(XT |AS ) ≥ XS

a.s.

Thus (X0 , A0 ), (XS , AS ), (XT , AT ), (XN , AN ) is a s-mg. Proof.

Case 1: 0 ≤ T − S ≤ 1. Let A ∈ AS . Then A (XT − XS ) dP = A∩[T =S+1] (XT − XS ) dP =

(a)

= =

(b)

N i=0

N i=0

A∩[T =S+1]∩[S=i]

A∩[S=i]∩[T =i+1] (Xi+1

N i=0

(XT − XS ) dP

Ai

− Xi ) dP

(Xi+1 − Xi ) dP

where Ai ≡ A ∩ [S = i] ∩ [T = i + 1]

>

=0

by (18.1.4), provided that we show that Ai ∈ Ai . We have (c) (d)

A ∩ [S = i] ∩ [T = i + 1] = (A ∩ [S ≤ i]) ∩ ([S = i] ∩ [T ≤ i]c ) = (an event in Ai ) ∩ (an event in Ai ) ∈ Ai .

Thus (b) holds, and case 1 is completed. Case 2: 0 ≤ S ≤ T ≤ N a.s. Deﬁne the stopping times (e)

Rk ≡ T ∧ (S + k)

for k = 0, 1, . . . , N ;

recall that sums and minima of stopping times are stopping times. Note that (f)

0 ≤ S = R0 ≤ · · · ≤ Ri ≤ Ri+1 ≤ · · · ≤ RN = T ≤ N

where 0 ≤ Ri+1 − Ri ≤ 1 for all i. Thus, (g)

E(XT |AS ) = E(XRN |AR0 ) = E(E(XRN |ARN −1 )|AR0 ) >

= E(XRN −1 |AR0 ) >

>

by case 1

= · · · = E(XR0 |AR0 ) = XR0 (h)

= XS .

2

holds a.s.,

3.

THE SUBMARTINGALE CONVERGENCE THEOREM

3

The Submartingale Convergence Theorem

473

Theorem 3.1 (S-mg convergence theorem) Let {Xn , An }∞ n=0 be a s-mg. + + (A) Suppose EXn M < ∞ (i.e., the Xn -process is integrable). Then (1)

Xn → X∞

for some X∞ ∈ L1 .

a.s.

(B) For uniformly integrable Xn ’s (equivalently, Xn →L1 X∞ ), this X∞ is said to close the s-mg in that ∞ {Xn , An }∞ n=0 is a s-mg with EXn EX∞ , and with A∞ ≡ σ[∪n=1 An ]. < [Closing the s-mg means that A Xn dP = A X∞ dP for all A ∈ An and all n, with the limiting rv X∞ ∈ L1 .]

(2)

(C) If {Xn , An }∞ n=0 is actually a mg, then the following statements are equivalent:

(3)

(a) Xn ’s are uniformly integrable,

(b) Xn →L1 X∞ ,

(c) Some rv Y closes the mg,

(d) X∞

closes the mg,

(e) Xn ’s are integrable and supn E|Xn | ≤ E|X∞ |. (D) In all the above, if {Xt , At }t∈[0,∞) is a process on (D[0,∞) , D[0,∞) ), then (4)

n, Xn , An , {0, 1, . . .}

may be replaced by

t, Xt , At , [0, ∞).

[Closing an X-martingale on [0, ∞) by X∞ is the same as closing an X-martingale on [0, θ) by the limiting rv Xθ .] Notation 3.1 If a sequence does not converge to an extended real value, then it must necessarily be oscillating. If it does so oscillate, then some interval must be “upcrossed” inﬁnitely often. We seek to take advantage of this. Let X1 , X2 , . . . be a sequence of rvs. Let a < b. Then: (n)

U[a,b] (ω) ≡ (the number of upcrossings of [a, b])

(5) (6)

number of integer pairs (i, j) with 0 ≤ i < j ≤ n having , ≡ Xi (ω) ≤ a, a < Xk (ω) < b for i < k < j, and Xj (ω) ≥ b (∞)

(n)

U[a,b] (ω) ≡ lim U[a,b] (ω) . n→∞

2

Inequality 3.1 (Upcrossing inequality; Doob) Let {Xk , Ak }nk=0 be a submg. Then 1 1 (n) (7) EU[a,b] ≤ {E(Xn − a)+ − E(X0 − a)+ } ≤ {EXn+ + |a|}. b−a b−a If {Xn , An }∞ n=0 is a submg, then (8)

(∞)

EU[a,b] ≤

1 1 + {E(X∞ − a)+ − E(X0 − a)+ } ≤ {EX∞ + |a|}. b−a b−a

474

CHAPTER 18. MARTINGALES

Proof. The number of upcrossings of [a, b] made by Xk and of [0, b − a] made by (Xk − a)+ are identical; since (Xk − a)+ is also a submg, we may assume that Xk ≥ 0 and a = 0 in this proof. Deﬁne T0 ≡ 0, T1 ≡ min{n ≥ 0 : Xn = 0}, T2 ≡ min{n > T1 : Xn ≥ b}, T3 ≡ min{n > T2 : Xn = 0}, . . . , Tn+2 ≡ n ;

(a)

here we use the convention that min ∅ ≡ n. Clearly, these are stopping times that do satisfy 0 = T0 ≤ T1 ≤ · · · ≤ Tn+2 = n. Thus proposition 18.2.1 shows that the process {XTi , ATi }n+2 i=0 is a submg. Now, Xn − X0 = n+2 i=1 (XTi − XTi−1 ) (b)

=

(same) +

i odd

i even

(same) ≡ I1 + I2 ,

and since XTi is a submg, we have EI1 ≥ 0

(c)

EI2 ≥ 0.

and (n)

Now suppose that U[0,b] (ω) = k ; then I2 (ω) = [XT2 (ω) (ω) − XT1 (ω) (ω)] + · · · + [XT2k (ω) (ω) − XT2k−1 (ω) (ω)] + · · · (n)

≥ [b] + · · · + [b] + 0 = b U[0,b] (ω).

(d)

Thus (recall (2.1.3)) (n)

E(Xn − X0 ) = EI1 + EI2 ≥ EI2 ≥ b EU[0,b] ,

(e)

and this is (7) in disguise (since Xk really denotes (Xk − a)+ ). Finally, note the positive part inequality (Xn − a)+ ≤ Xn+ + |a|. (n)

(∞)

Now, 0 ≤ U[a,b] U[a,b] . Thus, (f)

(∞)

(n)

E U[a,b] = lim E U[a,b]

by the MCT

(g)

1 ≤ lim b−a {E(Xn − a)+ − E(X0 − a)+ }

(h)

≤

1 b−a

by (7)

{E(X∞ − a)+ − E(X0 − a)+ } ,

since E(Xn − a)+ ≤ E(X∞ − a)+ follows from {(Xn − a)+ , An }∞ n=0 being a submg. 2 Proof.

Consider the theorem. Now, [ω : lim Xn (ω) exists as a number in [−∞, ∞] ]c = [ lim Xn < lim Xn ]

(a)

=

{r<s

rational} [ lim Xn < r < s < lim Xn ] ≡

{r<s

rational} Ars .

3.

THE SUBMARTINGALE CONVERGENCE THEOREM

475

It suﬃces to prove that P (Ars ) = 0 for all r, s. Now, (b)

(∞)

Ars ⊂ Brs ≡ [ω : U[r,s] = ∞].

It thus suﬃces to show that P (Brs ) = 0 for all r, s. But (c)

(∞)

(n)

(n)

E U[r,s] = E lim U[r,s] = lim E U[r,s]

(d)

1 ≤ lim s−r {EXn+ + |r|}

(e)

≤ (some M ) < ∞

by the MCT

by (7)

since the Xn+ ’s are integrable;

hence we must have P (Brs ) = 0. Thus X∞ ≡ lim Xn exists a.s. with values in [−∞, ∞], and X∞ is A∞ -measurable, since all Xn ’s are. Now (recall (2.1.3)), (f)

E|Xn | = 2EXn+ − EXn ≤ 2EXn+ − EX0 ≤ M < ∞.

Thus Fatou’s lemma implies (g)

E|X∞ | = E(lim |Xn |) ≤ lim E|Xn | ≤ M < ∞;

thus X∞ ∈ L1 with its values in (−∞, ∞) a.s. Thus (A) holds. Vitali shows that uniform integrability is equivalent to L1 convergence (since Xn → X∞ a.s. by part (A), in either s-mg case). Thus for n ≥ m, we have from (18.1.4) and L1 -convergence that < (h) X dP = A Xn dP → A X∞ dP for all A ∈ Am . A m Thus {Xn , An }∞ n=0 is a s-mg by condition (18.1.4). That is, X∞ closes the s-mg (and (B) holds). Consider (C). For (a)–(d), we lack only that (c) implies (a). For a mg, (i) (j)

E { |Xn | × 1[|X n |≥λ] } ≤ E { |Y × |1[|X n |≥λ] } →0

since Y closes the mg

by absolute continuity of the integral,

since the sets satisfy (k)

P (|Xn | ≥ λ) ≤ E|Xn |/λ ≤ E|Y |/λ → 0

uniformly in n as λ → ∞.

See exercise 3.1 below for the remaining (e) implies (a). This completes the proof in the case of discrete time. Consider (D). (Continuous time) Our preliminaries will not assume the s-mg structure. Now, lim Xt (ω) could be +∞ for some ω’s, and this will cause diﬃculties with the present approach. Thus (following Doob) deﬁne (l)

Yt (ω) = (2/π) tan−1 (Xt (ω))

to transform the range space from [−∞, ∞] to [−1, 1]. For each m choose rational numbers tm1 , . . . , tmkm in [m, ∞) so that (remember, X : (Ω, A, P ) → (D, D) ) (m)

P (supt∈[m,∞) Yt − suptmj ∈[m,∞) Ytmj >

1 m)

<

1 2m .

476

CHAPTER 18. MARTINGALES

This is possible, since the sup over all rationals r in [m, ∞) equals the sup over all reals t in [m, ∞), and the rationals are the limit of {r1 , . . . , rk } for any ordering {r1 , r2 , . . .} of the rationals; thus, 0 = P (supt∈[m,∞) Yt − suprj ∈[m,∞) Yrj >

1 m)

(n) = limk→∞ P (supt∈[m,∞) Yt − suprj ∈[m,∞);j≤k Yrj >

1 m) .

We may assume that the tmj ’s were chosen so as to simultaneously satisfy (o)

1 P (inf t∈[m,∞) Yt − inf tmj ∈[m,∞) Ytmj < − m )<

1 2m .

Thus, if we (p)

let t1 < t2 < · · · denote an ordering of ∪∞ m=1 { tm1 , . . . , tmkm }

(which does exist, since all tmj ≥ m), then P (supt∈[m,∞) Yt − supti ∈[m,∞) Yti >

1 m)

<

1 2m

1 )< P (inf t∈[m,∞) Yt − inf ti ∈[m,∞) Yti < − m

1 2m

and

(q) .

Letting Am and Bm denote the events in (q) we see that ∞ ∞ m 1 P (Am ∪ Bm ) ≤ 1 2/2 < ∞ , so that P (Am ∪ Bm i.o.) = 0. Thus (r)

lim t→∞ Yt = lim i→∞ Yti a.s.

and

lim t→∞ Yt = lim i→∞ Yti a.s.

Now, transforming back via Xt (ω) = tan((π/2) Yt (ω)), (r) implies the next lemma: Lemma 3.1 For every X : (Ω, A, P ) → (D[0,∞) , D[0,∞) ) there exist rational numbers t1 < t2 < · · · such that (9)

lim t→∞ Xt = lim i→∞ Xti a.s. and lim t→∞ Xt = lim i→∞ Xti a.s.

[Note that ti → ∞ in (9) could be replaced by ti θ for any ﬁnite θ.] Armed with the (9) “lemma”, it is now easy to use the discrete version of this theorem to prove the continuous version. We will refer to the continuous versions of conclusions (1)–(3) as (1 )–(3 ). We return to the proof of the theorem. (Now, we again assume s-mg structure in what follows.) Let Yi ≡ Xti and A˜i ≡ Ati for the ti ’s in (9). Then (Yi , A˜i )∞ i=0 is a s-mg to which the discrete theorems can be applied. Thus, (s)

lim t→∞ Xt = lim i→∞ Xti a.s.

(t)

= (a.s., some X∞ in L1 )

(u)

= lim i→∞ Xti

by (1)

= lim t→∞ Xt

by (9),

by (9) by (1) applied to (Yi , A˜i )∞ i=1

3.

THE SUBMARTINGALE CONVERGENCE THEOREM

477

so that (v)

Xt → X∞ a.s.

where X∞ ∈ L1 .

That is, (1 ) holds. The rest is left to the reader in exercise 3.2.

2

Exercise 3.1

Complete the proof of the discrete part of theorem 3.1.

Exercise 3.2

Comple the proof of the continuous part of theorem 3.1.

Exercise 3.3 (a) Let · · · ⊂ A−1 ⊂ A0 ⊂ A1 ⊂ · · · be sub σ-ﬁelds of A. Let X ∈ L1 (Ω, A, P ). Let Yn ≡ E(X|An ). Then (Yn , An )∞ n=−∞ is a uniformly integrable mg. (b) Let Yt ≡ E(Y |Dt ), for an arbitrary collection of sub σ-ﬁelds Dt of (Ω, A, P ). These Yt ’s are necessarily uniformly integrable. Exercise 3.4 Let {Xn , An }∞ n=0 be a submg for which Xn ≤ 0. Then (1) holds, and Y ≡ 0 closes the submg. Exercise 3.5 Let {Xn , An }∞ n=0 be a submg with Xn ≥ 0. Let r > 1. Then the Xnr ’s are uniformly integrable if and only if the Xnr -process is integrable. Exercise 3.6 Let {Xn , An }∞ n=0 be a submg. The following are equivalent: (a): The Xn+ ’s are uniformly integrable. (b): There exists a rv Y that closes the submg. When these hold, then X∞ (which necessarily exists a.s., and must be in L1 ) closes (a) the submg. [Hint. Do what you can with Xn+ . Then apply it to Yn ≡ (Xn ∨a)+|a|, and let a → −∞.] Exercise 3.7 (Martingale Lr -convergence theorem) Let {Xn , An }∞ n=0 be a mg sequence. Let r > 1. Then the following are equivalent: (10)

The |Xn |r -process is integrable.

(11)

Xn →Lr X∞ .

(12)

The Xn ’s are uniformly integrable (thus Xn →a.s. (some X∞ ) and X∞ ∈ Lr ).

(13)

The |Xn |r ’s are uniformly integrable.

(14)

r r {|Xn |r , An }∞ n=0 is a submg and E|Xn | E|X∞ | .

Exercise 3.8 (a) Show that t, Xt , At , [0, ∞) may replace n, Xn , An , {0, 1, . . .} in all of exercise 3.3–exercise 3.7. [Also, [0, θ) may replace [0, ∞).] (b) Prove (D) of theorem 3.2 below.

478

CHAPTER 18. MARTINGALES

Deﬁnition 3.1 (Reversed s-mg) LetXn be adapted to An with n ∈ {. . . , −1, 0}, and deﬁne the σ-ﬁeld A−∞ ≡ ∩0n=−∞ An . The process {Xn , An }0n=−∞ is a reversed s-mg (as deﬁned earlier) if (15)

<

Xn = E(Xm |An ) a.s.

for all n ≤ m.

(This is like the second law of thermodynamics run backward, since An brings more stability as n .) Theorem 3.2 (Reversed s-mg convergence theorem) be a s-mg sequence. (A) It necessarily holds that (16)

Let {Xn , An }0n=−∞

Xn → X−∞ a.s. for some X−∞ ∈ [−∞, ∞) a.s., that is A−∞ -measurable .

(B)–(C) Furthermore, the following are equivalent (and all yield an X∞ ∈ L1 ): (17)

EXn M > −∞

as n → −∞.

(18)

Xn ’s are uniformly integrable.

(19)

Xn →L1 X−∞ .

(20)

{Xn , An }0n=−∞ is a s-mg,

(This is trivial for any mg.)

where A−∞ ≡ ∩0n=−∞ An .

(D) In all the above, if {Xt , At }t∈(−∞,0] is a process on (D(∞,0] , D(−∞,0] ), then (21)

n, Xn , An , {. . . , − 1, 0}

may be replaced by

t, Xt , At , (−∞, 0].

(n)

Proof. Consider (16). Let U[r,s] now denote the upcrossings of [r, s] by the process X−n , . . . , X−1 , X0 . Replace line (d) of the proof of theorem 3.1 by (a)

(∞)

EU[r,s] ≤

1 s−r

(EX0+ + |r|) ≤ M < ∞,

and conclude that X−∞ ≡ lim Xn exists a.s. with values in [−∞, +∞]. Since the sequence {Xn+ , An }0n=−∞ is necessarily a submg, we obtain from Fatou that (b)

+ EX−∞ = E(lim Xn+ ) ≤ lim EXn+ ≤ lim EX0+ = EX0+ < ∞.

Thus X−∞ takes values in [−∞, +∞) a.s., and is A−∞ -measurable. Thus (16) does hold.

3.

THE SUBMARTINGALE CONVERGENCE THEOREM

479

Suppose that (17) holds. Then (b) and (17) give E|X−∞ | = E[ lim |Xn | ] ≤ lim E|Xn | = lim [2 EXn+ − EXn ] ≤ 2 EX0+ − M < ∞;

(c)

thus X−∞ takes values in (−∞, ∞) a.s., and X−∞ ∈ L1 (closing the s-mg, as in (h) of the previous proof). Also, from (c), (d)

P (|Xn | ≥ λ) ≤ E|Xn |/λ ≤ [2 EX0+ − M ]/λ → 0

uniformly in n ∈ {−∞, . . . , − 1, 0 } as λ → ∞. Thus (an analogous Xn− proof works only in the case of a mg), (e)

<

+ [Xn ≥λ]

Xn+ dP =

+ + [Xn ≥λ] X0

dP

by (18.1.3)

implies the Xn+ ’s are uniformly integrable. Now for n < m we have 0 ≥ − [Xn ≤−λ] Xn− dP = E (Xn − Xm ) + EXm − [Xn >−λ] Xn dP >

= E(Xn − Xm ) + EXm − = E(Xn − Xm ) + ≥ − +

(f)

[Xn ≤−λ]

[Xn >−λ] Xm

[Xn ≤−λ] Xm

dP

dP

Xm dP

for all n ≤ (a ﬁxed m that is large enough), since EXn M ≥ − −

[Xn ≤−λ]

≥ −2

(g)

|Xm | dP

for λ large enough, as in (d), with m now ﬁxed.

Thus, the Xn− are uniformly integrable. Thus, the Xn are uniformly integrable; that is, (18) holds. Then (18) implies (19) by Vitali. Suppose (19) holds. For any n ≤ m we have from (18.1.4) that Xn dP → A X−∞ dP for all A ∈ Am , since L1 -convergence gives |Xn − X−∞ | dP → 0. Thus {Xn , An }0n=−∞ is a s-mg, so (20) holds. Note that (20) trivially implies (17). The extension of this theorem from n to t uses (9) to extend (16), just as in the case of theorem 3.1. Then the proof that the t versions of (16)–(20) hold adds nothing new. 2 (h)

>

A

Xm dP =

A

480

CHAPTER 18. MARTINGALES

Exercise 3.9 (a) Let {Xn , An }0n=−∞ be a mg. If E|X0 |r < ∞ for some r ≥ 1, then necessarily (22)

Xn →Lr X−∞

as n → −∞.

{Xn , An }0n=−∞

(b) Let be a submg. If Xn ≥ 0 and E|X0 |r < ∞ for some r ≥ 1, then (22) again holds. Theorem 3.3 Let {An }+∞ n=−∞ be sub σ-ﬁelds of A. Suppose that the rv X is integrable, in that X ∈ L1 (Ω, A, P ). Then: (23) (24) Proof. (25)

E(X|An ) →a.s. and E(X|An ) →a.s. and

L1

E(X|A∞ )

as n → ∞,

L1

E(X|A−∞ )

as n → −∞.

Now, {Yn , An }∞ n=−∞ is a mg

for

Yn ≡ E(X|An ),

since E(Ym |An ) = E{E(X|Am )|An } = E(X|An ) = Yn for n ≤ m. Moreover, (26)

these Yn = E(X|An ) are uniformly integrable,

since the tails yield 0 ≤ [±Yn ≥λ] ± Yn dP = [±Yn ≥λ] ± E(X|An ) dP (a) = [±Yn ≥λ] ± X dP since [±Yn ≥ λ] ∈ An ≤ [ |Yn |≥λ] |X| dP (b)

→0

as λ → ∞ , by absolute continuity of the integral,

using P (|Yn | ≥ λ) ≤ E|Yn |/λ = E|E(X|An )|/λ ≤ E|X|/λ → 0 as λ → ∞. Thus Y±∞ ≡ limn→±∞ Yn

exists a.s.,

(27) {Yn , An }+∞ n=−∞ is a mg, and Yn →L1 Y±∞

as n → ±∞.

We just applied the s-mg convergence theorem (theorem 3.1) as n → ∞ and the reversed s-mg convergence theorem (theorem 3.2) as n → −∞. We must now show that Y∞ = E(X|A∞ ). Now, for A ∈ ∪∞ n=1 An we have (c) Y dP = A Yn dP by (18.1.4) A ∞ (d) = A X dP by deﬁnition of E(·|An ). ∞ That is, A Y∞ dP = A X dP for all A ∈ ∪∞ n=1 An , where the collection ∪n=1 An is a monotone ﬁeld that generates A∞ ; thus equality also holds for all A ∈ A∞ , by the Carath´eodory extension theorem. Thus Y∞ = E(X|A∞ ). We must also show that Y−∞ = E(X|A−∞ ). Now, if A ∈ A−∞ , then (e) Y dP = A Yn dP by (18.1.4) A −∞ = A X dP by deﬁnition of E(·|An ), since A ∈ An (f) = A E(X|A−∞ ) by deﬁnition of E(·|A−∞ ); and thus Y−∞ = E(X|A−∞ ).

2

4.

APPLICATIONS OF THE S-MG CONVERGENCE THEOREM

481

4

Applications of the S-mg Convergence Theorem

The following examples give just a few selected applications to show the power of the various s-mg convergence theorems. Example 4.1 (SLLN) Let X1 , X2 , . . . be iid µ. Then the partial sum process Sn ≡ X1 + · · · + Xn satisﬁes (1)

¯ n ≡ Sn /n → µ a.s. and L1 X

as n → ∞.

(Recall also the SLLN for U -statistics) in (17.1.29).] Proof. (a)

2

Let A−n ≡ σ[Sn , Sn+1 , . . .] = σ[Sn , Xn+1 , Xn+2 , . . .].

Now, Y−n ≡ E(X1 |A−n ) is a reversed mg on . . . , −2, −1 and (b)

E(X1 |A−n ) → E(X1 |A−∞ )

a.s. and L1

as n → −∞, by theorem 18.3.3. Now, E(X1 |A−n ) = E(X1 |Sn , Xn+1 , · · ·) = E(X1 |Sn ) =

by (8.4.23)

n

k=1 E(Xk |Sn )/n

by symmetry

= E(Sn |Sn )/n (c)

= Sn /n.

Combining (b) and (c) gives (d)

Sn /n = E(X1 |A−n ) → E(X1 |A−∞ )

a.s. and L1

as n → ∞.

But lim(Sn /n) is a tail rv, and thus it is a.s. a constant by the Kolmogorov 0-1 law; hence E(X1 |A−∞ ) is a.s. a constant, by (d). But E[E(X|A−∞ )] = µ, so that the constant must be µ ; that is E(X1 |A−∞ ) = µ a.s. Thus (d) implies Sn /n → µ a.s. and L1 . 2 Exercise 4.1 (SLLN for U -statistics) Let Y−n ≡ Un be a U -statistic based on X1 , X2 , . . . , with a symmetric kernel h for which Eh(X1 , X2 ) is ﬁnite. Consider n:n , Xn+1 , Xn+2, . . .], for the vector X n:n of the ﬁrst n order the σ-ﬁeld A−n ≡ σ[X statistics of the sequence. (a) Show that {Yn , An }2n=−∞ is a reversed mg. (b) Use this to show that Un →a.s. and L1 Eh(X1 , X2 ). (c) Extend this to higher-dimensional kernels.

482

CHAPTER 18. MARTINGALES

Example 4.2 (Kolmogorov’s 0-1 law) Suppose that Y1 , Y2 , . . . are iid rv’s and let An ≡ σ[Y1 , . . . , Yn ]. Suppose that A ∈ T ≡ (tail σ-ﬁeld) = ∩∞ n=1 σ[Yn+1 , Yn+2 , . . .]. Since An is independent of T , P (A) = P (A|An ) = E(1A |An ). But by theorem 18.3.3 we have E(1A |An ) →a.s. E(1A |A∞ ) =a.s. 1A . 2

Thus P (A) = 0 or 1.

Example 4.3 (Approximation of L1 and L2 functions) 1 f ∈ L1 ([0, 1], B, Lebesgue); thus 0 |f (u)| du < ∞. Let

Fix the function

An ≡ σ[((i − 1)/2n , i/2n] : i = 1, . . . , 2n } = B[0, 1] . Deﬁne X ≡ f (ξ), where ξ ∼ = Uniform(0, 1). Now let 2n Xn ≡ E(X|An ) = E(f (ξ)|An ) = k=1 Cnk (f ) 1[(k−1)/2n <ξ≤k/2n ] , with Cnk (f ) ≡ 2n

k/2n (k−1)/2n

f (u) du. Since E|X| < ∞, theorem 18.3.3 gives

Xn → E(X|A∞ ) = E(X|B) = X

a.s. and L1 .

Summary Let f ∈ L1 and deﬁne the step function fns (·) by k/2n n k (2) fns (t) ≡ 2n f (u) du for (k−1) 2n < t ≤ 2n and 1 ≤ k ≤ 2 . (k−1)/2n

Then for every f ∈ L1 , (3)

fns (·) → f (·) a.s. Lebesgue

and

1 0

|fns (t) − f (t)| dt → 0 .

Now suppose that f ∈ L2 . Then f ∈ L1 also, and so (3) still holds; and this implies (18.3.12). Thus (the equivalent) (18.3.11) gives (4)

fns →L2 f

for every f ∈ L2 .

[So fns (·) can be thought of as approximating the derivative of the indeﬁnite integral x F (x) ≡ 0 f (t) dt.] 2 Example 4.4 (Kakutani’s mg) Let X1 , X2 , . . . be independent with each Xk ≥ 0 and E Xk = 1. Deﬁne n (5) Mn ≡ 1 Xk , for 1 ≤ k ≤ n, with M0 ≡ 1. Then {Mn , An }∞ 1 is a mg for which all E Mn = 1, where An is an appropriate sequence of σ-ﬁelds (such as the histories). Since Mn is bounded in the space L1 , the sm-g convergence theorem of (18.3.1) shows that (6)

Mn →a.s. M∞

4.

APPLICATIONS OF THE S-MG CONVERGENCE THEOREM

483

for the appropriate rv M∞ . We now show that the following are equivalent: (7)

c∞ ≡ EM∞ = 1,

(8)

Mn ’s are uniformly integrable,

(9)

Mn →L1 M∞ ,

(10)

∞

(11)

∞

1

1

where an ≡ E(Xn1/2 ) ≤ 1,

an > 0,

(1 − an ) < ∞.

Whenever one (hence all) of these equivalent statements fails, then necessarily (12)

P (M∞ = 0) = 1

and

c∞ = 0.

2

Proof. Equivalence of (10) and (11) is an easy exercise. We now show that condition (10) implies (9). Deﬁne the normalized product Nn ≡

n 1

(13)

1/2

Xk

E(Nn2 ) = 1/(

/

n 1

n 1

ak ,

ak )2 ≥ 1/(

with ∞ 1

ak )2 < ∞ .

Thus {Nn , An }∞ 1 is a mean-1 mg that is bounded in L2 . Since all Doob’s L2 -inequality (inequality 10.11.5) and the MCT give (a)

n 1

ak ≤ 1,

E(supn Mn ) ≤ lim E( sup1≤k≤n Mk ) n

(b)

≤ lim E( sup1≤k≤n Nk2 ) × 1

(c)

2 2 ≤ ( 2−1 ) E(Nn2 ) < ∞ .

n

Thus M ∗ ≡ supn Mn is a rv in L1 for which 0 ≤ Mn ≤ M ∗ . Hence the rvs {Mn : 1 ≤ n ≤ ∞} are uniformly integrable. The rest of the claimed equivalence of conditions (7)–(9) follows from theorem 18.3.1. Conclusion (11) follows, since Nn still converges a.s. (it is bounded in L2 ), while (d)

Mn = (

n 1

ak ) Nn → 0 a.s.

implies that M∞ = 0 a.s. and thus that c∞ = 0.

2

Exercise 4.2 Suppose that one of (7)–(9) hold (and hence all of them hold). Show that this implies that (10) also holds.

484

CHAPTER 18. MARTINGALES

A Branching Process Model Example 4.5 (Branching Processes) Let X denote the number of oﬀspring of a particular type of individual, and let pk ≡ P (X = k) for k = 0, 1, . . . . We start at generation zero with a single individual Z0 = 1, and it produces the individuals in a ﬁrst generation of size Z1 . These in turn produce a second generation of size Z2 , and so forth. Thus, (14)

Zn+1 ≡

Zn

Xnj

for n ≥ 0,

with Z0 ≡ 1,

j=1

where Xnj denotes the number of oﬀspring of the jth individual present in the nth generation. We assume that all Xnj ’s are iid as the X above. Also, we suppose ∞ (15) m ≡ EX = k=0 k pk < ∞, with p0 > 0, and p0 + p1 < 1. We call this a simple branching process model. Let (16)

Wn ≡ Zn /mn

Proposition 4.1 (17)

(18)

Var[Wn ] =

An ≡ σ[W1 , . . . , Wn ].

The process

{Wn , An }∞ n=0

and

and

is a mg nσ2 1−m−n σ2 m(m−1)

with mean

EWn = 1 ,

if m = 1, if m = 1

provided that σ2 ≡ Var[X] < ∞. Proof. (a)

(b)

We note that

EZn+1 = E[E(Zn+1 |Zn )] = ∞ k=0 E(Zn+1 |Zn = k) P (Zn = k) k ∞ = ∞ k=0 E( j=1 Xnj ) P (Zn = k) = k=0 m k P (Zn = k) = m E(Zn ) = · · · = mn+1 ,

while the mg property follows from (c)

E(Wn+1 |An ) = m−(n+1) E(Zn+1 |Zn ) = m−(n+1) mZn = Wn .

We leave (18) to the following exercise. Exercise 4.3

2

Verify the variance formula (18). Verify (20) below.

Notation 4.1 (19)

We deﬁne the generating functions f and fn of X and Zn by ∞ k ∞ f (s) ≡ k=0 s pk and fn (s) ≡ k=0 sk P (Zn = k).

It is easy to verify that (20)

fn+1 (s) = fn (f (s)) = f (fn (s))

for |s| ≤ 1.

4.

APPLICATIONS OF THE S-MG CONVERGENCE THEOREM

Theorem 4.1 (Branching process) σ2 ≡ Var[X] < ∞. Then (21)

485

(i) Suppose that m = EX > 1 and also

Wn →a.s. and L2 W∞ ∼ = (1, σ2 /[m(m − 1)]),

where (W∞ , A∞ ) closes the mg. Also, (22)

P (W∞ = 0) = (the probability of ultimate extinction) = π,

where (23)

π ∈ (0, 1)

is the unique solution of

f (π) = π.

Moreover, the chf φ of W∞ is characterized as the unique solution of (24)

φ(mt) = f (φ(t))

for t ∈ R subject φ(0) = 1 and φ (0) = i m.

(ii) If m ≤ 1, then Wn →a.s. 0 as n → ∞. Proof. (i) Now, EWn2 ≤ 1+σ 2 /[m(m−1)] for all n, so that the mg {Wn , An }∞ n=1 is square-integrable. Thus the mg Lr convergence of exercise 18.3.7 gives (21). We let π∗ ≡ P (W∞ = 0). Then ∞ ∞ (a) π∗ = k=0 P (W∞ = 0|Z1 = k) P (Z1 = k) = k=0 P (W∞ = 0)k pk ∞ (b) = k=0 π∗ k pk = f (π∗ ). Now, f (0) = p0 > 0, f (1) = 1, f (1−) = m > 1, and f (s) is in s for 0 < s < 1; draw a ﬁgure. Thus f (π) = π has a unique solution in (0, 1). The solution π = 1 is ruled out by Var[W∞ ] > 0, since π = 1 would imply W∞ ≡ 0. (Note that (22) also follows from (20).) We now turn to (24). Now, ∞ (c) φn+1 (t) ≡ E eitWn+1 = j=0 E(exp(itZn+1 /mn+1)|Z1 = j)P (Z1 = j) ∞ = j=0 φn (t/m)j pj (d)

= f (φn (t/m)).

Since Wn →a.s. W∞ implies Wn →d W∞ , we have φn → φ on R. Applying this to the identity (b) gives φ(t) = lim φn+1 (t) = lim f (φn (t/m)) since f is continuous on |r| ≤ 1

= f (lim φn (t/m)) (e)

= f (φ(t/m)).

Suppose now that ψ satisﬁes ψ(t) = f (ψ(t/m)). Then γ(t) ≡ [ψ(t) − φ(t)]/t = [(ψ(t) − 1) − (φ(t) − 1)]/t → ψ (0) − φ (0) (f)

=0

if ψ(0) = 1

if ψ (0) = i m

486

CHAPTER 18. MARTINGALES

as t → 0. Also, |t m| × |γ(tm)| = |ψ(tm) − φ(tm)| = |f (ψ(t)) − f (φ(t))| ≤ |f (t∗ )| × |ψ(t) − φ(t)|

for t∗ ∈ (0, 1) by the mean value theorem

≤ m |ψ(t) − φ(t)| (g)

= |tm| × |γ(t)|,

and iterating (e) gives (h)

|γ(t)| ≤ |γ(t/m)| ≤ · · · ≤ |γ(t/mn )| → 0,

so that γ(t) = 0 for all t = 0. Trivially, γ(t) = 0 for t = 0. Thus, (24) holds. (ii) Set s = 0 in (20) to get P (Zn+1 = 0) = f (P (Zn = 0)), where P (Zn = 0) is necessarily . Passing to the limit gives π = lim P (Zn = 0) = f (π). But if m ≤ 1, then π = 1 is the only solution of π = f (π). 2

5.

DECOMPOSITION OF A SUBMARTINGALE SEQUENCE

5

Decomposition of a Submartingale Sequence

487

Deﬁnition 5.1 (Predictable process) A predictable process {An , An }∞ n=0 is one in which each An is An−1 -measurable for each n ≥ 0; here A0 is a constant (or A0 is {∅, Ω}-measurable). [Especially interesting are and predictable processes, since any submg can be decomposed as the sum of a mg and such a predictable process.] Theorem 5.1 (Decomposition of a submg) Then Xn can be decomposed as (1)

Let {Xn , An }∞ n=0 be a submg.

Xn = Yn + An = [mg] + [ and predictable],

where {Yn , An }∞ n=0 is a 0-mean mg and An is a predictable process satisfying (2)

A0 ≡ EX0 ≤ A1 ≤ · · · ≤ An ≤ · · ·

a.s.

This decomposition is a.s. unique. Conversely, if Xn = Yn + An as above, then {Xn , An }∞ n=0 is a submg. [Call An the compensator.] Proof. (Doob) Suppose that {Xn , An }∞ n=0 is a submg. Let A0 ≡ EX0 , and deﬁne (the compensator candidate) (3)

An ≡

n

[E(Xk |Ak−1 ) − Xk−1 ] + EX0

for n ≥ 1.

i=1

Clearly, An is and is An−1 -measurable. It remains to show that Yn ≡ Xn − An is a mg. Now, E(Yn |An−1 ) = E(Xn |An−1 ) − E(An |An−1 ) = E(Xn |An−1 ) − An (a)

= E(Xn |An−1 ) − [E(Xn |An−1 ) − Xn−1 ] − An−1

(b)

= Xn−1 − An−1 = Yn−1 .

Consider the uniqueness. Suppose Xn = Yn + An is one such decomposition that works. Elementary computations give (c)

E(Xn |An−1 ) = E(Yn |An−1 ) + An = Yn−1 + An

a.s.;

but the speciﬁcation of the decomposition also states that (d)

Xn−1 = Yn−1 + An−1

a.s.

Subtracting (d) from (c) gives uniqueness via (e)

An − An−1 = E(Xn |An−1 ) − Xn−1

a.s.

The converse holds, since (f)

E(Yn + An |An−1 ) = Yn−1 + An ≥ Yn−1 + An−1

as required.

2

a.s., for n ≥ 1

488

CHAPTER 18. MARTINGALES

Exercise 5.1 If the X-process is integrable, then the A-process is uniformly integrable (in either theorem 5.1 or theorem 5.2). Theorem 5.2 (Decomposition of a reversed submg) Let {Xn , An }0n=−∞ be a submg such that limn→∞ E(Xn ) = M > −∞ (thus, the Xn -process is integrable). Then Xn can be decomposed as (4)

Xn = Yn + An = [mg] + [ ≥ 0 , , and predictable],

where {Yn , An }0n=−∞ is a mg and An is an An−1 -measurable function with (5)

0 = A−∞ ≡ lim An ≤ · · · ≤ An ≤ · · · ≤ A0 n→−∞

a.s.

This decomposition is a.s. unique. Conversely, if Xn = Yn + An as above, then {Xn , An }0n=−∞ is a submg. [Call An the compensator.] Proof. (6)

(Doob) We deﬁne An ≡

n

[E(Xk |Ak−1 ) − Xk−1 ]

for n ≤ 0;

k=−∞

then An is clearly ≥ 0, , and An−1 -measurable, provided that it can be shown to be well-deﬁned. Now m E(Am − An ) = n+1 E(Xk − Xk−1 ) = EXn − EXm ≤ EX0 − EXm (a) (b)

≤ ( some M ) < ∞,

by hypothesis. Also, (c)

m A˜m ≡ lim (Am − An ) = lim n+1 [E(Xk |Ak−1 ) − Xk−1 ] , n→−∞

n→−∞

so that the MCT gives (d)

EA˜m = lim E(Am − An ) = EXm − lim EXn = EXm − M < ∞ n→−∞

n→−∞

with a well-deﬁned ﬁnite limit. Since A˜m ≥ 0 and EA˜m < ∞, we know that A˜m is ﬁnite a.s.; so (6) is well-deﬁned. The A˜m ’s are and bounded below by 0. Thus A−∞ ≡ limm→∞ A˜−m exists a.s., and it is ≥ 0. Moreover, the equalities in (d) show that EA˜−m → 0 as m → ∞; just use the MCT via (e)

EA−∞ = lim EA˜−m = lim EXm − M = M − M = 0. m→∞

m

Thus A−∞ = 0 a.s., and each A˜m = Am a.s. Let Yn ≡ Xn −An . Lines (a)–(f) of the previous proof complete this proof, using (18.3.20) about mgs for the existence of Y−∞ . 2

5.

DECOMPOSITION OF A SUBMARTINGALE SEQUENCE

489

Example 5.1 (Predictable variation, or conditional variance) Suppose 2 2 ∞ {Xn , An }∞ denotes a 0-mean mg with each EX < ∞. Then {X , A } n n=0 n n n=0 is a submg by proposition 18.1.2. By theorem 5.1 we see that (7)

Zn ≡ Xn2 − An

is a 0-mean mg adapted to the An ’s for n ≥ 0.

Here An is the predictable process (with A0 ≡ EX02 = Var[X0 ] ≥ 0) deﬁned by n 2 } + EX02 An ≡ k=1 {E(Xk2 |Ak−1 ) − Xk−1 (8) = nk=1 E{(Xk − Xk−1 )2 |Ak−1 } + EX02 (9)

=

n

E{(∆Xk )2 |Ak−1 } + EX02 ,

i=1

where ∆Xk ≡ Xk − Xk−1 and ∆X0 = X0 . The compensator term An (of the Xn2 process) given in (9), is called the conditional variance or the predictable variation of the Xn -process. Note that (for Xn ≡ An ), (10) EXn2 = Var[Xn ] = nk=0 Var[Xk ] = EAn = EXn , since we agree to also use the notation Xn for the predictable variation process An that corresponds to the mg {Xn , An }∞ n=0 . Thus (to summarize), (11)

Xn ≡ An =

n

E{(∆Xk )2 |Ak−1 } + EX02

k=1

is the predictable variation (or conditional variance) of a 0-mean mg {Xn , An }∞ n=0 , and then the conditionally centered process (12)

Zn ≡ Xn2 − Xn

is a 0-mean mg with respect to the An ’s, for n ≥ 0 ,

provided that we have ﬁnite EXn2 ’s.

2

Martingale Transforms Deﬁnition 5.2 (H-transforms) Let {Hn }∞ n=0 be a predictable process with ∞ respect to the ﬁltration {An }n=0. [Think of Hn being the amount a gambler iwill wager at stage n, based only on observation of the winnings/game up through time n − 1 (but not, of course, through time n).] For some other process {Xn }∞ n=0 , deﬁne the H-transform of X (to be denoted by {(H · X)n }∞ n=0 ) by (13)

(H · X)n ≡

n k=1

Hk (Xk − Xk−1 ) + H0 X0 =

n

Hk ∆Xk .

k=0

(We agree that ∆X0 ≡ X0 , and that H0 is a constant.) Theorem 5.3 (S-mg transforms) (i) Let {Xn , An }∞ n=0 be a s-mg (or supermg). ∞ If {Hn }n=0 is predictable with each Hn ≥ 0 and bounded, then {(H · X)n , An }∞ n=0 is a s-mg (or supermg). (The supermartingale case shows that there is no system for beating the house in an unfavorable game.) ∞ (ii) If {Xn , An }∞ n=0 is a mg and {Hn }n=0 is predictable and bounded, then the ∞ process {(H · X)n , An }n=0 is a mg with mean H0 EX0 .

490 Proof. (a)

CHAPTER 18. MARTINGALES We compute E[(H · X)n+1 |An ] = (H · X)n + E[Hn+1 (Xn+1 − Xn )|An ]

(b)

= (H · X)n + Hn+1 E[∆Xn+1 |An ]

(c)

= (H · X)n ,

>

>

since Hn+1 ≥ 0 and E(∆Xn+1 |An ) = 0. Note that E∆X0 = EX0 in the mg case. (The supermg case just reverses the inequality.) 2 Corollary 1

If {Xn , An }∞ n=0 is a s-mg and T is a stopping time, then:

(a)

Hn ≡ 1[T ≥n]

is ≥ 0, bounded, and predictable.

(b)

(H · X)n = XT ∧n

is a s-mg.

Proof. Now, Hn is predictable, since [T ≥ n] = [T ≤ n − 1]c ∈ An−1 for a stopping time T . Furthermore, (H · X)n = nk=0 1[T ≥k] (Xk − Xk−1 ) = nk=0 1[T ≥k] ∆Xk = XT ∧n , (a) so the conclusion follows from theorem 5.3.

2

∞ Now, suppose {Hn }∞ n=0 is predictable and bounded, and {Xn , An }n=0 is a mg 2 2 ∞ with EXn < ∞, so that {Xn }n=0 is a submg with predictable variation process 2 {Xn }∞ n=0 . Then we know that {(H · X)n } is a submg. We will give the form taken by its predictable variation process H · Xn . Also, we will summarize everything so far in one place. (Let “wrt” abbreviate “with respect to.”)

Theorem 5.4 (Martingale transforms) Suppose {Xn , An }∞ n=0 is a 0-mean 2 ∞ mg with EXn < ∞ for each n, and let {Hn }n=0 be bounded and predictable. The predictable variation process Xn is given by (14)

Xn ≡ An =

n

E{(∆Xk )2 |Ak−1 } + EX02 .

k=1

Then the conditionally centered process (15)

Zn ≡ Xn2 − Xn

is a 0-mean mg with respect to the An ’s,

for n ≥ 0. The martingale transform n Wn ≡ (H · X)n ≡ k=0 Hk ∆Xk (16) is a mg with mean H0 EX0 with respect to the An ’s, for n ≥ 0. Its predictable variation process W n is (17)

W n ≡ H · Xn =

n

Hk2 E{ (∆Xk )2 |Ak−1 } + H02 E X02 .

k=1

Moreover, for n ≥ 0, the sequence (18)

Ln ≡ Wn2 − W n ≡ {(H · X)n }2 − (H · X)n is a 0-mean mg with respect to the An ’s.

5.

DECOMPOSITION OF A SUBMARTINGALE SEQUENCE

491

Proof. Recall example 5.1 and theorem 5.3 for the ﬁrst parts. Then by straightforward calculation from (13), we have H · Xn = nk=0 E{[∆(H · X)k ]2 |Ak−1 } (a) n = k=0 E{[Hk (Xk − Xk−1 )]2 |Ak−1 } n since Hk is Ak−1 -measurable = k=0 Hk2 × E{(∆Xk )2 |Ak−1 } (b) = nk=0 Hk2 ∆Xk . Note that L0 = [H0 ∆X0 ]2 − H02 E([∆X0 ]2 |A0 ) has mean 0, while Ln is a mg by example 5.1. 2

492

6

CHAPTER 18. MARTINGALES

Optional Sampling

We now extend the simple optional sampling theorem of section 18.2. Our aim will be to relax the restrictive assumption used there, that the stopping times are bounded.

Discrete Time Notation 6.1 (1)

Suppose that

{Xn , An }∞ n=0 is a s-mg

and that (2)

0 ≤a.s. T0 ≤a.s. T1 ≤a.s. · · ·
for stopping times T0 , T1 , . . . .

We deﬁne (3)

˜ n ≡ XTn X

and

A˜n ≡ ATn ,

˜ n ’s are adapted to the A˜n ’s. We would like to prove that {X ˜ n , A˜n }∞ so that the X n=0 is a s-mg, but this requires hypotheses. The weakest such hypotheses presented are (4) (5)

˜ n | < ∞ for all n, E|X Xk+ dP = 0 lim inf k→∞

for each n.

[Tn >k]

(But the conditions (4) and (5) need to be replaced by useful conditions that are more easily veriﬁable.) Theorem 6.1 (Optional sampling theorem) Let (1)–(3) deﬁne the sequence ˜ n , A˜n }∞ with respect to the s-mg {Xn An }∞ . Suppose (4) and (5) hold. {X n=0 n=0 Then the optionally sampled process (6)

˜ n , A˜n }∞ {X n=0

is a s-mg

for which (7)

<

<

<

<

<

˜ 0 = · · · = EX ˜ n = · · · = supk EXk ≤ ∞. EX0 = EX

Corollary 1 (A) Condition (4) holds if {Xn , An }∞ n=0 is integrable. (B) Conditions (4) and (5) both hold if any of the following conditions hold: (8)

Each Tn is a.s. bounded by some ﬁxed integer Nn .

(9)

The Xn ’s are uniformly integrable.

(10)

Xn ≤a.s. (some M ) < ∞

(11)

ETj < ∞ for all j, and there exists a constant K such that for all j, E(|Xn − Xn−1 |An−1 )(ω) ≤ K a.s., for all n ≤ Tj (ω).

for all n.

6.

OPTIONAL SAMPLING

493

Notation 6.2 The theorem becomes much cleaner if our s-mg includes an entry at ∞ that closes the s-mg. Suppose (12)

∞ ¯ {Xn , An }n=0

is a s-mg

and (13)

0 ≤a.s. T0 ≤a.s. T1 ≤a.s. · · · ≤ T∞ ≤a.s. ∞

for extended stopping times T0 , T1 , . . . , T∞ . We again deﬁne (14)

˜ n ≡ XTn X

A˜n ≡ ATn ,

and

˜ n ’s are adapted to the A˜n ’s (for 0 ≤ n ≤ ∞). so that the X Theorem 6.2 (Optional sampling theorem) (15)

˜ n , A˜n }∞ {X n=0 <

Suppose (12)–(14) hold. Then

is a s-mg

<

<

<

<

with EX0 = EXT0 = EXT1 = · · · = EXT∞ = EX∞ .

Continuous Time Theorem 6.3 (Optional sampling theorem) Suppose that the X-process is integrable and satisﬁes X : (Ω, A, P ) → (D[0,∞) , D[0,∞) ), and that it is adapted to some ﬁltration {At }t∈[0,∞) . Then (16)

n, Xn , An , {0, 1, . . .}, { 0, 1, . . . , ∞}, k t, Xt , At , [0, ∞), [0, ∞], s

may be replaced by

in theorem 6.1, theorem 6.2, and corollary 1 (only (11) must be omitted from the list of things that carry over with no change).

Proofs Consider theorem 6.1. Let A ∈ ATn−1 . It suﬃces to show that

Proof. (a)

<

A

XTn−1 dP =

A

XTn dP.

Basically, we wish to use proposition 18.2.1 and the DCT. To this end we deﬁne (b)

Tn(k) = Tn ∧ k = (a bounded stopping time) Tn .

Now, A ∩ [Tn−1 ≤ k] ∈ AT (k) , since for each m ≥ 0 we have n−1

(k)

A ∩ [Tn−1 ≤ k] ∩ [Tn−1 ≤ m] = A ∩ [Tn−1 ≤ k] ∩ [Tn−1 ∧ k ≤ m ∧ k] = A ∩ [Tn−1 ≤ m ∧ k] ∈ Am∧k ⊂ Am . Thus, for n ﬁxed we have

494 (c)

CHAPTER 18. MARTINGALES A∩[Tn−1 ≤k]

= <

(d)

=

(e)

=

<

XTn−1 dP

A∩[Tn−1 ≤k]

XT (k) dP

as the integrands are equal on the set

A∩[Tn−1 ≤k]

XT (k) dP

by proposition 18.2.1

n−1

n

A∩[Tn−1 ≤k]∩[Tn ≤k]

XTn dP +

A∩[Tn−1 ≤k]∩[Tn >k]

Xk dP ,

since XT (k) = Xk on [Tn > k]. Let k → ∞ in (c) and (e); and since Tn < ∞ a.s. n and since E|XTn−1 | and E|XTn | are ﬁnite, the DCT implies (recall that a = b ⊕ c means that |a − b| ≤ c) (f) X dP A Tn−1 <

(g)

=

(h)

=

A

XTn dP ⊕ lim

A

XTn dP ⊕ lim ak

[Tn >k]

Xk+ dP

where lim k→∞ ak = 0 by (5).

Equate the terms in (f) and (h) on a subsequence k having ak → 0 to obtain < (i) X dP = A XTn dP for all A ∈ ATn−1 . A Tn−1 This is equivalent to (6) by (18.1.4). Letting A = Ω in (i) shows that EXTn is . Introducing the new stopping time < τ ≡ 0 ≤ T0 and applying this result shows that EX0 ≡ EXτ = EXTn . < It remains to show that EXTn = supk EXk . Now, (j) E XTn = [Tn ≤k] XTn dP + [Tn >k] XTn dP = (k)

<

=

[Tn ≤k]

XT (k) dP +

[Tn ≤k]

Xk dP +

n

[Tn >k]

[Tn >k]

XTn dP

XTn dP

by proposition 18.2.1, since [Tn ≤ k] ∈ AT (k) n

(l)

<

= EXk ⊕

[Tn >k]

Xk+ dP +

[Tn >k]

XTn dP.

For the second term in (l) we recall (5). For the third term in (l) we note that E|XTn | < ∞ and that Tn < ∞ a.s. We thus conclude from (l) that (m) (n)

<

EXTn = lim k→∞ EXk + 0 + 0 <

= supk EXk .

This gives (7).

2

6.

OPTIONAL SAMPLING

495

Proof. Consider the ﬁrst claim made in the corollary. That is, we verify (k) that (4) holds if supk E|Xk | < ∞. Let Tn ≡ Tn∧k . Now, both {Xn , An }∞ n=0 and (k) + ∞ {Xn , An }n=0 are submgs. Since 0 ≤ Tn ≤ k, proposition 18.2.1 implies that both (a)

and X0+ , X +(k) , Xk+

X0 , XT (k) , Xk

Tn

n

are smgs.

Thus, (b) (c)

E|XT (k) | = E[2 X +(k) − XT (k) ] Tn

n

n

≤ 2 EXk+ − EX0

since |x| = 2 x+ − x

using (a)

≤ 2 E|Xk | − EX0 ≤ 2 supk E|Xk | − EX0 (d)

≤ (some M ) < ∞

by hypothesis.

Thus, Fatou’s lemma and then (d) gives (e)

˜ n | = E|XTn | = E| lim X (k) | E|X T

(f)

≤ lim E|XT (k) | ≤ M < ∞.

n

n

2

˜ n | ≤ Nn E|Xj | implies (4), and (5) is trivial, Proof. Consider (8). Now, E|X 0 since the integral equals 0 for k ≥ Nn . 2 Proof. Consider (9). Uniformly integrable Xn ’s are uniformly bounded in L1 and are uniformly absolutely continuous, by theorem 3.5.4. Uniformly bounded in L1 means supk≥1 E|Xk | < ∞; hence by part (A) of the corollary, we have (4). Since Tn < ∞ a.s., we have P (Tn > k) → 0 as k → ∞, and hence uniform absolute continuity implies (5). 2 Proof. (a)

Consider (10). Now, (5) is trivial, since Xk+ ≤ M , and (4) holds, since 0 ≤ E(M − XTn ) = E[lim(M − XT (k) )] n

(b)

≤ lim E(M − XT (k) )

(c)

≤ lim E(M − X0 )

(d)

= M − E(X0 ) < ∞,

n

giving EX0 ≤ EXTn ≤ M .

by Fatou, since all M − Xn ≥ 0 by proposition 18.2.1

2

496

CHAPTER 18. MARTINGALES

Proof. Consider (11). Let Y0 ≡ |X0 | and Yj ≡ |Xj − Xj−1 | for j ≥ 1. Let n T Zn ≡ 0 Yj and Z˜n ≡ 0 n Yj . Then (4) holds since ∞ k ˜ n | ≤ E|Z˜n | = ∞ (a) E|X k=0 [Tn =k] Zk dP = k=0 [Tn =k] j=0 Yj dP (b)

=

∞

(c)

=

∞

(d)

since [Tn ≥ j] = [Tn ≤ j − 1]c ∈ Aj−1 ∞ ≤ K j=0 P (Tn ≥ j) = K (1 + ETn ) using (11) and then (10.2.1)

(e)

< ∞.

j=0

j=0

[Tn ≥j] Yj [Tn ≥j]

dP

by Fubini

E(Yj |Aj−1 ) dP

Also, (5) holds, since ETn < ∞ implies P (Tn > k) → 0 as k → ∞; hence EZ˜n < ∞ (as follows from (e)) gives ˜ (f) as k → ∞ . 2 [Tn >k] |Xk | dP ≤ [Tn >k] Zn dP → 0

Proof. Consider theorem 6.2. Since {Xn , An }∞ n=0 is a s-mg, we see from the s-mg convergence theorem that the Xn ’s are uniformly integrable and Xn → X∞ a.s., for some X∞ ∈ L1 . Now, for any n ∈ {0, 1, . . . , ∞} we have E|XTn | ≤ E( lim |XT (k) | + |X∞ |) n→∞

n

≤ lim E|XT (k) | + E|X∞ | n

≤ lim E|Xk | + E|X∞ | < ∞,

(a)

since Tn(k) ≡ Tn ∧ k Tn

by Fatou by proposition 18.2.1

since the process is integrable and X∞ ∈ L1 .

Let A ∈ ATn−1 ; recall that Tn−1 could now equal +∞. Even so, we do have A ∩ [Tn−1 ≤ k] ∈ AT (k) was shown at the start of the proof of theorem 6.1 . Thus, (b)

n−1

A∩[Tn−1 ≤k]

= <

= (c)

=

XTn−1 dP

(k)

A∩[Tn−1 ≤k]

XT (k) dP

since Tn−1 = Tn−1 on [Tn−1 ≤ k]

A∩[Tn−1 ≤k]

XT (k) dP

by proposition 18.2.1

n−1

n

A∩[Tn−1 ≤k]∩[Tn >k]

Xk dP +

A∩[Tn−1 ≤k]∩[Tn ≤k]

XTn dP ,

since XT (k) = Xk on [Tn > k]. Let k → ∞ in (b) and (c), since (a) shows that n

E|XTn−1 | and E|XTn | are ﬁnite, the DCT gives

6. (d)

OPTIONAL SAMPLING A∩[Tn−1 <∞] <

= lim k→∞

497

XTn−1 dP A∩[Tn−1 ≤k]∩[Tn >k]

(e)

=

A∩[Tn−1 <∞]∩[Tn =+∞] X∞

(f)

using Xk →L1 X∞ = A∩[Tn−1 <∞] XTn dP.

Xk dP +

dP +

A∩[Tn <∞] XTn

A∩[Tn <∞]

dP

XTn dP

We add to each of (d) and (f) the equal terms of the equation (g) A∩[Tn−1 =∞] XTn−1 dP = A∩[Tn−1 =∞] XTn dP, and obtain < (h) A XTn−1 dP = A XTn dP

for all A ∈ ATn−1 .

The proof of the previous paragraph also shows that < (i) X dP = A XT∞ dP for all A ∈ ATn . A Tn Finally, (h), (i), and (18.1.4) show that {XTn , ATn }∞ n=0 is a s-mg. Add in Ta ≡ 0 and Tb ≡ ∞ for the expectation claim, with A = Ω. 2 Proof. Consider theorem 6.3. We ﬁrst consider the extension of theorem 6.2. It suﬃces to consider the stopping times a pair at a time; we will do so, relabeling them so that S ≤ T a.s. Let Dn ≡ {k/2n : k = 0, 1, . . . , }, and note that (a)

{Xt , At }t∈Dn

is a s-mg.

Deﬁne stopping times T (n) by (b)

T (n) (ω) = k/2n

whenever (k − 1)/2n < T (ω) ≤ k/2n and k ≥ 0;

and likewise for S (n) . Note that (c)

S (n) ≤ T (n) ,

S ≤ S (n) ,

T ≤ T (n)

a.s.

We can apply theorem 6.1 to S (n) and T (n) to conclude < (d) X dP = A XT (n) dP for all A ∈ AS ⊂ AS (n) . A S (n) [Now, T (n) satisﬁes (5), since [T (n) > k/2n ] = [T > k/2n ],

and

lim inf k

[T >k/2n ]

+ Xk/2 n dP → 0

+ by hypothesis. Also, T (n) satisﬁes (4) (using that supk E|Xk/2 n | < ∞), by the proof of the ﬁrst statement of the corollary.] Now, by right continuity of the paths

(e)

XS (n) →a.s. XS

and

XT (n) →a.s. XT

as n → ∞.

498

CHAPTER 18. MARTINGALES

Thus Vitali’s theorem allows us to pass to the limit in (d) and obtain < (f) for all A ∈ AS A XS dP = A XT dP provided that we show that (g)

the XT (n) ’s (and analogously the XS (n) ’s) are uniformly integrable.

Since 0 ≤ T (n) ≤ T (n−1) with both taking values in Dn , theorem 6.1 gives (h)

<

XT (n) = E(XT (n−1) |AT (n) ) a.s.

Thus (17)

{Yn , Bn }0n=−∞ is a reversed s-mg, where Yn ≡ XT (−n) and Bn ≡ AT (−n) . <

From this we need only the rather minor fact (since EX0 = EXT (n) for this s-mg pair) that (i)

>

lim EYn = lim E(XT (−n) ) = EX0 > −∞,

n→−∞

n→−∞

by theorem 6.1. Thus the reversed s-mg theorem implies (g). Let A = Ω in (f) for (j)

<

<

<

−∞ < EX0 = EXS = EXT = EX∞

if we also apply (f) to T0 ≡ 0 and T∞ ≡ ∞. Then (f), (k), and (18.1.4) complete the proof. 2

Exercise 6.1

Prove theorem 6.3 (the case of integrable Xt in theorem 6.1 only).

Exercise 6.2

Prove theorem 6.3 (the corollary to theorem 6.1 case).

7.

APPLICATIONS OF OPTIONAL SAMPLING

499

7

Applications of Optional Sampling

Example 7.1 (Gambler’s ruin) Suppose Y1 , Y2 , . . . are iid with p ≡ P (Y1 = 1) and with q ≡ P (Y1 = −1). Let Sn ≡ Y1 + · · · + Yn . Let −a < 0 < b be integers, and deﬁne the stopping time (1)

τ ≡ inf{n : Sn = −a or b}.

Deﬁne An ≡ σ[X1 , . . . , Xn ]. Let φ(t) = pet + qe−t denote the mgf of Y . Let c0 ≡ log(q/p), and note that φ(c0 ) = 1. We now apply the examples of section 18.1. When p = q = 12 : (2)

Sn

is a mean-0 mg,

(3)

Zn(1) ≡ Sn2 − n

is a mean-0 mg.

For general p and q : (4)

Zn(2) ≡ Sn − n(p − q)

is a mean-0 mg,

(5)

Zn ≡ (q/p)Sn = exp(c0 Sn ) = exp(c0 Sn )/φn (c0 )

is a mean-1 mg.

We now make the claim (see exercise 7.1 below) (6)

ESτ = 0,

EZτ(1) = 0,

EZτ(2) = 0,

EZτ = 1.

With probability 1, the rv Sτ takes on one of the values −a or b. Now, τ ∧ m τ a.s. and Sτ ∧m → Sτ a.s., while (τ ∧ m) is a bounded stopping time to which proposition 18.2.1 or theorem 6.1 necessarily applies. Thus, for p = q = 12 we can conclude that 0 = limm 0 = limm ESτ ∧m = E Sτ = −aP (Sτ = −a) + b[1 − P (Sτ = −a)] by proposition 18.2.1 and the DCT with dominating function a + b; and Eτ = lim E(τ ∧ m) = ESτ2∧m → ESτ2 = a2 P (Sτ = −a) + b2 P (Sτ = b) by the MCT, proposition 18.2.1 and the DCT. Solving these gives (7)

P (Sτ = −a) = b/(a + b) and Eτ = ab

when p = q = 12 .

(2)

(1)

Justifying the other two equations in (6) (Zn is analogous to Zn , while Zn uses condition (18.6.11)), (8)

P (Sτ = −a) =

1 − (p/q)b 1 − (p/q)a+b

if p = q

and, with µ ≡ p − q, (9)

Eτ =

b b + a 1 − (p/q)b − µ µ 1 − (p/q)a+b

if p = q.

Note that if µ ≡ p − q < 0, then [max 0≤n<∞ Sn ] ∼ = Geometric(p/q). That is, (10)

P (max 0≤n<∞ Sn ≥ b) = (p/q)b

for all integers b,

when p < q.

(Just let a → ∞ in the formula for P (Sτ = −a) to obtain the complementary probability.) 2

500

CHAPTER 18. MARTINGALES

Example 7.2 (Gambler’s ruin for Brownian motion) Suppose that S µ is Brownian motion with drift: S µ(t) = S(t) + µt for t ≥ 0. Deﬁne the stopping time τab ≡ τ ≡ inf{t ≥ 0 : S µ(t) = −a or b}, where −a < 0 < b. An easy argument will show that Eτ < ∞. Observe ﬁrst that (11)

S 20(t) − t,

S 0(t),

S µ(t) − µt

are 0-mean mgs.

Then set θ = −2µ, and recall (12.7.8) to conclude that (12)

exp(θ[S µ(t) − µt] − θ2 t/2) = exp(−2µ[S(t) + µt])

is a mean-1 mg.

Applying the optional sampling theorem to (11) and (12), we obtain (13)

P (S(τ ) = −a) = b/(a + b)

if µ = 0 ,

(14)

Eτ = ab

if µ = 0 ,

(15)

P (S µ(τ ) = −a) =

(16)

Eτ =

1 − e2µb 1 − e2µ(a+b)

b a + b 1 − e2µb − µ µ 1 − e2µ(a+b)

if µ = 0 , if µ = 0 .

∼ Note that if µ < 0, then Sµ∞ 0 = Exponential(2|µ|). Let a → ∞ in (14) to obtain (17)

P (S µ∞ 0 ≥ b) = exp(−2 |µ| b) = exp(−θ b)

Note the complete analogy with example 7.1.

for all b > 0. 2

Exercise 7.1

Give all details in justifying the ﬁnal two equalities in (6).

Exercise 7.2

Verify completely the claims of example 7.2.

Exercise 7.3 Derive an analogue of the previous example 7.2 that is based on the Poisson process {N(t) : t ≥ 0}.

8.

INTRODUCTION TO COUNTING PROCESS MARTINGALES

501

8

Introduction to Counting Process Martingales

Heuristic Treatment of Counting Process Martingales Suppose now that {M (x), Ax }x∈R is a 0-mean martingale. Then for any increment M (x + h) − M (x) we have E{M (x + h) − M (x)|Ax } = 0. Operating heuristically, this suggests that (1)

E{dM (x)|Ax− } = 0 a.s.

for any martingale {M (x), Ax }x∈R ,

where Ax− is the σ-ﬁeld generated by everything up to (but not including) time x. With this background, we now turn to our problem. Suppose now that (2)

N (x)

is a counting process;

a counting process is (informally) an process that can increase only by taking jumps of size +1. Incremental change is modeled via (3)

E{dN (x)|Ax− } = dA(x) a.s.;

here dA(x) is Ax− -measurable.

It then seems that M (x) ≡ N (x) − A(x) for x ∈ R is a martingale; we call A(·) = (−∞,x] dA(y) the compensator of N . Note that A(·) is an and Ax− -measurable process. We compute the predictable variation process M of the martingale M (as suggested by (18.5.3) or (18.5.8), and using integration by parts) via (4)

(5)

dM (x) ≡ E{dM 2 (x)|Ax− } = E{M− (x) dM (x) + M (x) dM (x)|Ax− } = E{2 M− (x) dM (x) + [dM (x)]2 |Ax− } = 2 M− (x) E{dM (x)|Ax− } + E{[dM ]2 |Ax− } = 2 M− (x) · 0 + E{[dM ]2 |Ax− }

(6)

= E{[dM (x)]2 |Ax− }

by (1)

as is also suggested immediately by (18.5.11)

so, the heuristics of both (18.5.8) and (18.5.11) give the same thing = E{[dN (x) − dA(x)]2 |Ax− } = E{[dN (x)]2 − 2[dA(x)] [dN (x)] + [dA(x)]2 |Ax− } = E{[dN (x)]2 |Ax− } − 2 dA(x) E{dN (x)|Ax− } + E{[dA(x)]2 |Ax− } = E{dN (x)|Ax− } − 2 [dA(x)]2 + [dA(x)]2 by (3) and [dN (x)]2 = dN (x) (7)

= dA(x) − [dA(x)]2

(8)

= [1 − ∆A(x)] dA(x)

by (3) where ∆A(x) ≡ A(x) − A− (x) ≡ A(x) − A(x−);

502

CHAPTER 18. MARTINGALES

note that [dN (x)]2 = dN (x), since dN (x) takes on only the values 0 and 1. This suggests that x (9) M (x) = −∞ [1 − ∆A(y)] dA(y). Thus (note (5)), the process M 2 (x) − M (x)

has

E{d[M 2 (x) − M (x)]|Ax− } = 0,

which suggests that, provided that each E M 2 (x) < ∞, (10)

M 2 (x) − M (x)

for x ∈ R

is a 0-mean mg with respect to the Ax .

Summary Starting with a 0-mean martingale M (·) having all EM 2 (x) < ∞, (11)

M (·) is the predictable variation of the submartingale {M 2 (x), Ax }t∈R .

That is, M (·) is the , ≥ 0, and A−x -measurable process whose existence is guaranteed by the Doob–Meyer decomposition theorem (theorem 18.9.2 below). [Note that we “discovered” this without using said Doob–Meyer theorem; that theorem will merely guarantee that our heuristic guess-and-verify approach (assuming that we can make it rigorous) gives us the “right answer.” This is typical.] Consider the martingale transform x W (x) ≡ −∞ H(y) dM (y) (12) where H(x, ·) is Ax− -measurable for all x. Then E{dW (x)|Ax } = E{H(x)dM (x)|Ax− } = H(x)E{dM (x)|Ax− } = 0 by (1), so (13)

{W (x), Ax }x∈R is a martingale,

provided that each E|W (x)| < ∞.

Moreover, its predictable variation is given by dW (x) = E{[dW (x)]2 |Ax− )

appealing directly to (6) this time

= E{[H(x) dM (x)]2 |Ax− } = H 2 (x) E{[dM (x)]2 |Ax− } = H 2 (x) dM (x) suggesting that (14)

W (x) =

x −∞

since H(x) is Ax− -measurable

by (6),

H 2 dM .

This also suggests, provided that each EW 2 (x) < ∞, that (15)

{L(x) ≡ W 2 (x) − W (x), Ax }x∈R ,

is a 0-mean mg.

Processes H(·) that are Ax− -measurable have the property that H(x) = E{H(x)|Ax− }, or that H(x) can be determined by averaging H(·) over the past; such an H is thus called predictable. The martingale transform statement (15) can be summarized as x (16) −∞ [predictable] d [martingale] = [martingale] provided that expectations exist.

8.

INTRODUCTION TO COUNTING PROCESS MARTINGALES

503

Suppose now that we have a sequence of martingales Mn whose increments satisfy a type of Lindeberg condition; this suggests that any limiting process M (·) ought to be a normal process. From the martingale condition we hope that Cov[M (y) − M (x), M (x)] = lim E{[Mn (y) − Mn (x)] Mn (x)} n

= lim E{Mn (x) E{Mn (x, y]|Ax }} = lim E{Mn (x) · 0} = 0; n

n

and for a normal process M (·), uncorrelated increments also mean independent increments. The variance process of M (·) should be EM 2 (x) = limn EMn2 (x) = limn EMn (x) by (6). So it seems reasonable to hope that [recall (11.1.15)] (17)

Mn →d M ∼ = S(V )

on (D R , DR , ρ∞ )

as n → ∞

for a Brownian motion S, provided that (18)

the increments of Mn satisfy a type of Lindeberg condition,

and provided that (note (9)) (19)

Mn (x) →p [some V (x)]

as n → ∞, for each x ∈ R,

where (20)

V is and right continuous with V (−∞) = 0.

As noted above, (21)

it often holds that

V (x) = lim EMn (x) = lim E Mn2 (x) = E M 2 (x). n

n

Of course, the original martingales Mn need to be square integrable. This “quasi theorem” is roughly Rebolledo’s CLT. One other bit of heuristics seems in order. Suppose now that we have several counting processes Ni (x) and that we perform the above x calculations and determine martingales Mi (x) = Ni (x) − Ai (x) with Mi (x) = −∞ [1 − ∆Ai ] dAi . Now, for Ax− -measurable functions ci (·), n (22) Mn (x) ≡ i=1 ci (x) Mi (x) is also a martingale. We note from (6) that

(23)

dMn (x) = E{[dMn (x)]2 |Ax− } n = i=1 c2i (x) E{[dMi (x)]2 |Ax− } + i =j ci (x) cj (x) E{[dMi (x)] [dMj (x)]|Ax− } n 2 = i=1 ci (x) Mi (x),

provided that the (24)

Mi (x, y]

and Mj (x, y]

are uncorrelated, given Ax− .

In fact, conditions under which all of the previous heuristics are actually true are given below. Even without these, we can use these heuristics as the ﬁrst step in a guess-and-verify approach.

504

CHAPTER 18. MARTINGALES

The Guess-and-Verify Approach in a Single Sample Example 8.1 (Single-sample martingale) (25)

Ni (x) ≡ 1[Xi ≤x]

Suppose now that

for x ∈ R,

for X1 , . . . , Xn iid F on R. Let Ax ≡ σ[1[Xi ≤y] : y ≤ x, 1 ≤ i ≤ n]. Then Ni is a counting process with E{dNi (x)|Ax− } = P (dNi (x) = 1|Ni(x−) = 0) = dA(x) ≡ 1[Xi ≥x] dΛ(x), where dΛ(x) ≡ [1 − F− (x)]−1 dF (x) with ∆Λ(x) = [∆F (x)]/[1 − F− (x)] ≤ 1, so x (26) Mi (x) ≡ Ni (x) − Ai (x) = Ni (x) − −∞ 1[Xi ≥y] dΛ(y) for x ∈ R satisﬁes (27)

{Mi (x), Anx }x∈R

is a 0-mean mg. x The predictable variation process is (letting −∞ ≡ (−∞,x] ) x Mi (x) = −∞ [1 − ∆Ai (y)] dAi (y) = (28)

=

x −∞

[1 − 1[Xi ≥y] × ∆Λ(y)] × 1[Xi ≥y] dΛ(y)

x

−∞ 1[Xi ≥y]

× [1 − ∆Λ(y)] dΛ(y).

Thus, Cov[Mi (x), Mi (y)] = E Mi2 (x ∧ y) since Mi is a mg x∧y = E Mi (x ∧ y) = E −∞ 1[Xi ≥t] × [1 − ∆Λ(t)] dΛ(t) = = (29)

x∧y −∞

x∧y −∞

E(1[Xi ≥t] × [1 − ∆Λ(t)] dΛ(t) [1 − ∆Λ] dF

= V (x ∧ y),

where (30)

V (x) ≡

x −∞

[1 − ∆Λ] dF

for x ∈ R.

Since the sum of martingales is also a martingale, we have n (31) is a 0-mean mg on R with respect to the Anx Mn (x) ≡ √1n i=1 Mi (x) √

n Fn (x) −

(32)

=

(33)

= Un (F (x)) +

x −∞

x

√

n [1 − Fn− ] dΛ

−∞ Un (F− ) dΛ.

8.

INTRODUCTION TO COUNTING PROCESS MARTINGALES

505

Moreover, (23) tells us that x Mn (x) = −∞ [1 − Fn− ] × [1 − ∆Λ] dΛ x →p −∞ [1 − F− ] × [1 − ∆Λ] × [1 − F− ]−1 dF x (34) = −∞ [1 − ∆Λ] dF = V (x). Our heuristic Rebolledo CLT suggests that (35)

Mn →d S(V )

on (D R , DR , ρ∞ )

for Brownian motion S. ˜ n of the It is clear from theorem 12.10.1 that a Skorokhod embedding version M original process Mn satisﬁes (36)

˜ n − M || →p 0 M

as n → ∞,

where (note exercise 8.1 below) · (37) M ≡ U(F ) + −∞ U(F− ) dΛ ∼ = S(V ).

2

Exercise 8.1 Suggest, by computing the means and covariances, that M ∼ = S(V ).

The Random Censorship Model Example 8.2 (Random Censorship) Suppose now that X1 , . . . , Xn are iid nondegenerate survival times with df F on [0, ∞). However, we are not able to observe the Xi ’s due to the following random censoring. Let Y1 , . . . , Yn be iid censoring times whose df G is an arbitrary df on [0, ∞]. Suppose also that the Yi ’s are independent of the Xi ’s. All that is observable is (38)

Zi ≡ Xi ∧ Yi

and

δi ≡ 1[Xi ≤Yi ]

for i = 1, . . . , n.

Let Ax ≡ σ[{1[X≤y] , 1[Y ≤y] , δ1[X≤y] , δ1[Y ≤y] : y ≤ x, 1 ≤ i ≤ n}]. Let the ordered values be denoted by Zn:1 ≤ · · · ≤ Zn:n (we agree that in a tied group of Zn:i ’s, those with a δ of 1 are given smaller subscripts than those with a δ of 0), and let δn:i correspond to Zn:i . Now, Z1 , . . . , Zn are iid H where 1 − H = (1 − F )(1 − G), and we let Hn denote the empirical df of the Zi ’s. Thus, (39)

Hn (t) ≡

n 1 1[Zi ≤t] n i=1

and

1 − H = (1 − F ) (1 − G) .

uc Let Huc denote the empirical df and the true df of the uncensored rvs; so n and H 1 n Huc and n (t) ≡ n i=1 1[Zi ≤t, δi =1] (40) H uc (t) ≡ E1[Zi ≤t, δi =1] = [0,t] (1 − G− ) dF .

The basic counting process here is (41)

Ni (t) ≡ 1[Zi ≤t] δi

for t ≥ 0.

506

CHAPTER 18. MARTINGALES

Since (42)

dAi (t) ≡ E{dNi (t)|At− } = 1[Zi ≥t]

1 − G− (t) dF (t) = 1[Zi ≥t] dΛ(t), 1 − H− (t)

the basic martingale is (43)

M1i (t) ≡ Ni (t) − Ai (t) = 1[Zi ≤t] δi −

t 0

1[Zi ≥s] dΛ(s)

for t ≥ 0.

And the predictable variation process is M1i (t) = [0,t] 1[Zi ≥s] × [1 − ∆Λ(s)] dΛ(s). (44) Now, observe (with (45)

Mn (t) ≡ =

√

t

√1 n

≡

0

[0,t] )

n i=1

that

M1i (t) =

√

n [ Huc n (t) −

t

√

uc n [ Huc n (t) − H (t) ] + 0

(46)

t

= Euc n (t) + 0

t 0

(1 − Hn− ) dΛ]

n [ Hn− − H− ] dH uc 1 − H−

En− dH uc , 1 − H−

where we have deﬁned √ uc (47) Euc n [ Huc n ≡ n −H ]

and

En ≡

√

n [ Hn − H]

on [0, ∞).

We would expect that (48)

Mn − M →p 0

as n → ∞

where, for appropriately deﬁned Euc and E , · E uc (49) M ≡E + dH uc on [0, ∞). 1 − H− 0 Also, (50)

Mn (t) =

t 0

[1 − Hn (s)] × [1 − ∆Λ(s)] dΛ(s)

and (51)

V (t) ≡ EMn (t) =

t 0

(1 − G− ) (1 − ∆Λ) dF =

t 0

(1 − H− ) × [1 − ∆Λ] dΛ.

As would now be expected, it holds that (52)

M∼ = S(V )

on (D [0,∞) , D[0,∞) ) .

The cumulative hazard function Λ(·) and the Aalen–Nelson cumulative hazard function estimator Λn (·) are deﬁned by t t 1 − G− 1 (53) Λ(t) ≡ dF and Λn (t) ≡ d Huc n 0 1 − H− 0 1 − Hn−

8.

INTRODUCTION TO COUNTING PROCESS MARTINGALES

507

for all t ≥ 0. This is motivated by the deterministic halves of (39) and (40). We now use this to form an estimator of the df F , that is, a type of instantaneous life table. The Kaplan–Meier product-limit estimator of the survival function 1 − F is given by δn:i ˆn (t) ≡ 1−F (54) for t ≥ 0. s≤t [1 − ∆Λn (s)] = Zn:i ≤t [1 − 1/(n − i + 1)] We deﬁne (55)

Xn ≡

√

n [Fn − F ],

Bn ≡

√

n [Λn − Λ],

Zn ≡

√ [Fˆn − F ] n 1−F

on [0, ∞); and, with (56)

T ≡ Zn:n

and

Jn (·) ≡ 1[0,T ] (·) = 1[0,Zn:n ] (·) = 1[Hn− (·)<1]

(where Jn (·) is predictable), we further deﬁne (57)

XTn ≡ Xn (T ∧ ·),

BTn ≡ Bn (T ∧ ·),

Z Tn ≡ Z n(T ∧ ·)

on [0, ∞).

Now, note that for t ≥ 0, t √ t BTn (t) = n [ 0 1−H1n− d Huc n − 0 dΛ] =

√ t n 0

(58)

t

= 0

1 1−Hn−

d [ Huc n −

Jn Mn = 1 − Hn−

t 0

(1 − Hn− ) dΛ]

t

[predictable] d[martingale] = [martingale](t) . 0

(b) Evaluate BnT .

Exercise 8.2

(a) Verify (43)–(52).

Exercise 8.3

Determine Cov[Euc n (s), En (t)] for s, t ≥ 0.

Use integration by parts to show that for 0 ≤ t < τ ≡ F −1 (1), t ˆn− Xn (t) 1−F 1 T 1[0,Zn:n ] dMn Z n (t) = = 1 − F (t) 1 − F 1 − Hn− 0

Exercise 8.4 (59)

(60)

t

=

[predictable] d[martingale] = [martingale](t) . 0

Now, our heuristics suggest (it can be proved by either mg or empirical process methods) that ! "2 t 1 − Fˆn− 1 T (61) Z n (t) = 1[0,Zn:n ] dM n for 0 ≤ t < τ 2 1 − F (1 − H n− ) 0 =

!

t

1[0,Zn:n ] 0

ˆn− 1−F 1−F

"2

1 × [1 − ∆Λ] dΛ 1 − Hn−

508

CHAPTER 18. MARTINGALES

(62)

→ D(t) ≡ 0

t

1 1 dΛ, 1 − H− 1 − ∆Λ

since (63)

[(1 − F− )/(1 − F )]2 = [1/(1 − ∆Λ)]2 ,

and (64)

Z Tn →d S(D)

and

X Tn →d (1 − F ) S(D) ,

both on (D[0,∞) , D[0,∞) , ρ∞ ). Theorem 8.1 (65) Proof. (66)

Let F denote an arbitrary df F on [0, ∞). Then for V as in (51),

BTn →d S(V )

on (D [0,∞) , D[0,∞) , ρ∞ ).

We will show that for all 0 ≤ r ≤ s ≤ t < ∞, we have E{ BTn (r, s]3/2 BTn (s, t]3/2 } ≤ 35 V (r, s]3/4 V (s, t]3/4 ,

so that theorem 12.2.2 gives relative compactness. We leave convergence of the ﬁnite-dimensional distributions to an exercise below. Deﬁne Yt ≡ BTn , and note that Yt is of the form (a) Yt = [0,t] H dM for H predictable and M a mg; in fact, H ≡ Jn /(1 − Hn− )

(b)

and

M ≡ Mn .

We will require two basic results. It holds that (c)

E{Y (s, t]2 |As } ≤ 2 βn V (s, t]

for all 0 ≤ s ≤ t < ∞,

where (d)

n:n βn ≡ (1 − H− )/(1 − Hn− ) Z 0

has

Eβnk ≤ 1 + e k Γ(k + 1).

These will be established below. However, we ﬁrst use them to give a proof of (66). Now, for all 0 ≤ r ≤ s ≤ t < ∞ we have E{ |Y (r, s] |3/2 |Y (s, t] |3/2 } = E{ |Y (r, s] |3/2 E{|Y (s, t] |3/2 |As } } (e)

≤ E{ |Y (r, s] |3/2 E{Y (s, t]2 |As }3/4 }

(f)

≤ E{ |Y (r, s] |3/2 (2 βn V (s, t])3/4 }

(g)

≤ 23/4 V (s, t]3/4 E { (Y (r, s]2 )3/4 βn3/4 }

by conditional Liapunov by (c)

≤ 23/4 (Eβn3 )1/4 V (s, t]3/4 (E{Y (r, s]2 })3/4

by H¨ older’s inequality

≤ 23/4 (Eβn3 )1/4 V (s, t]3/4 (E{E{Y (r, s]2 |Ar }})3/4

8. (h)

INTRODUCTION TO COUNTING PROCESS MARTINGALES ≤ 23/4 (Eβn3 )1/4 V (s, t] 3/4 ({ E2 βn V (r, s]})3/4

509

by (c)

= 23/2 (Eβn3 )1/4 (Eβn )3/4 V (r, s]3/4 V (s, t]3/4 (i)

≤ 23/2 (Eβn3 )1/2 V (r, s]3/4 V (s, t]3/4

(j)

≤ 35 V (r, s]3/4 V (s, t]3/4

by Liapunov’s inequality

by (d),

as claimed in (66). We now establish (c). Now, E{ Y (s, t]2 |As } = E{ Yt2 − 2Ys Yt + Ys2 |As } (k)

= E{ Yt2 − Ys2 |As }

since {Yt , At }[t≥0] is a mg

= E{ Yt2 − Y t |As } − (Ys2 − Y s ) + E{Y t − Y s |As }

(l)

= E{ Y t Y s |As } since {Yt2 − Y t , At }[t≥0] is a mg so far, holding very generally = E{ (s,t] H 2 dM |As }

(m)

= E{

(n)

=

(o)

= Jn (s)

(p)

Jn (s.t] (1−Hn− )2

(s,t]

(1 − Hn− ) × [1 − ∆Λ] dΛ}

n Jn (v) E{ n [1−H |As } × [1 − ∆Λ(v)] dΛ(v) n− (v)]

(s,t]

n E{

1[Z(v)>0] Z(v) }

by Fubini

× [1 − ∆Λ(v)] dΛ(v)

where Z(v) ∼ = Binomial(n[1 − Hn− (s)], [(1 − H− (v)]/[1 − H− (s)]) 2n 1 − H− (s) ≤ Jn (s) × [1 − ∆Λ(v)] dΛ(v) (s,t] n[1 − Hn− (s)] 1 − H− (v) by (67) below = 2Jn (s)

(q)

1 − H− (s) 1 − Hn− (s)

(s,t]

1 × [1 − ∆Λ] dΛ 1 − H−

≤ 2 βn V (s, t],

as claimed in (c). Step (o) used the fact that (67)

E{Z −1 1[Z>0] } ≤ 2/(mp)

when Z ∼ = Binomial(m, p).

This is true, since Z −1 1[Z>0] < 2/(Z + 1) and m k m−k m! 1 E(Z + 1)−1 = k=0 k+1 k!(m−k)! p q = (r)

1 (m+1)p

m+1

(m+1)! k+1=1 (k+1)!((m+1)−(k+1))!

≤ 1/((m + 1)p) ≤ 1/(mp).

pk+1 q (m+1)−(k+1)

510

CHAPTER 18. MARTINGALES We now turn to (d). Now, by (7.4.13) and (12.11.14) we have ∞ ∞ Eβnk = 0 P (βn ≥ x)kxk−1 dx ≤ 1 + 1 2x2 e−x kxk−1 dx

(s)

≤ 1 + 2k Γ(k + 2)

(t)

≤ 145

as claimed in (d).

for general k

for k = 3, 2

Exercise 8.5 Complete the proof of theorem 8.1, by showing convergence of the ﬁnite-dimensional distributions.

9.

DOOB–MEYER SUBMARTINGALE DECOMPOSITION

9

Doob–Meyer Submartingale Decomposition

511

It is not necessary to actually understand the rather sophisticated deﬁnition 9.1 and theorem 9.1 below. But we will “apply it” to obtain some more understandable theorems. Finally, our examples are all of the guess and verify type, so that these preliminaries are merely for guidance anyway. Let I denote [0, θ) and/or [0, θ] for 0 < θ ≤ ∞, and let I¯ = [0, θ]. Let (Ω, A, P ) and the ﬁltration {At }t∈I be given. A process {At , At }t∈I is called increasing (denoted by ) if As ≤ At a.s. for all s ≤ t with s, t ∈ I. “Predictable” processes (those which possess a certain kind of measurability) will play a fundamental role, but we delay the precise deﬁnition just a bit. Deﬁnition 9.1 Let {Mt , At }t∈I be an adapted process on the space (DI , DI ). Let 0 ≤ T1 ≤ T2 ≤ · · · < θ a.s. denote stopping times having Tn → θ a.s.; these are called localizing times. (i) Then the M -process is called a local martingale if {MTn ∧t , At }t∈I is a uniformly integrable martingale, for some choice of localizing times Tn . Call any {MT ∧t }t∈I a stopped process. (ii) [An A-process is integrable if EAθ < ∞, with Aθ ≡ limt→θ At .] If {ATn ∧t }t∈I is integrable for some choice of localizing times Tn , then the A-process is called locally integrable. (iii) The M -process is said to be of the D-class whenever the collection of rvs {XT : T is a stopping time} is uniformly integrable. [Recall that all stopping times T have P (T < ∞) = 1.] Exercise 9.1 Let the {Mt , At }t∈I be a local mg. Show that {Mt : t ∈ I} is a uniformly integrable mg if and only if the M -process is of class D. Theorem 9.1 (Doob-Meyer) (i) Let {Xt , At }t∈I be a submg of class D on (DI , DI ). Then we may decompose the X-process so that: (1)

Xt = Mt + At

¯ for t ∈ I,

(2)

{Mt , At }t∈I¯

is a uniformly integrable 0-mean mg,

(3)

{At , At }t∈I¯

is an and At -predictable process,

with A(0) = EX0 and EAθ < ∞. (ii) Let {Xt , At }t∈I be a submg. We may decompose as Xt = Mt + At for t ∈ I, where the X-process is a local mg and the A-process is , locally integrable, and At -predictable with A0 = EX0 . (iii) These decompositions will be unique in that if Xt = Mt0 + A0t as above, then P (Mt = Mt0 ) = 1 and P (At = A0t ) = 1for all t ∈ I¯ . [We call the A-process the compensator of the submg X-process.]

512

CHAPTER 18. MARTINGALES

Proof. We give no proof, since this does not give a recipe for the compensator. But it does give a useful target to aim for in speciﬁc examples, and we use it only for this purpose. See Lipster and Shiryayev (1989, p. 36). 2 Theorem 9.2 Let {Mt , At }t∈I be a 0-mean square-integrable martingale on (DI , DI ) (and so supt∈I EMt2 < ∞) [and thus {Mt2 , At }t∈I¯ is a closed submg]. Then we can decompose Mt2 as ¯ (4) Mt2 = Zt + M t for t ∈ I; here {Zt , At }t∈I¯ is a 0-mean mg and {M t }t∈I¯ is an At -predictable process that is and ≥ 0 with M 0 = EM02 . Now, {M t , At }t∈I¯ is the compensator of the M 2 -process, and is called the predictable variation process of the M -process. Since EZt = 0 and EMt = 0 in (4), ¯ (5) EM t = EM 2 = Var[Mt ] for all t ∈ I. t

Proof. This is immediate from theorem 10.1, since a uniformly integrable closed ¯ since mgs can be closed under mg is clearly of class D. Everything extends to I, uniform integrability. 2 ˜ , At }t∈I be 0-mean square-integrable mgs Theorem 9.3 Let {Mt , At }t∈I and {M ˜ and M D ≡ M − M ˜ are also. Let M S and on (DI , DI ) for which M S ≡ M + M D M denote their predictable variation processes. Then we can decompose the ˜ t in the following fashion: product Mt M ˜ t − M, M ˜ t ¯ (6) Zt ≡ M t M is a 0-mean mg for t ∈ I. (7)

˜ t ≡ 1 {M S t − M D t } M, M 4

¯ is an At -predictable process for t ∈ I.

(8)

˜ is the diﬀerence of two ≥ 0, , M, M and At -predictable ﬁnite-mean processes.

˜ t is a.s. unique, for each t. Then (6) and EMt = 0 = EM ˜ t give The process M, M ˜ t = E(Mt M ˜ t ) = Cov[Mt , M ˜ t] ¯ (9) EM M for t ∈ I. ˜ t , At }t∈I the predictable covariation process of the Deﬁnition 9.2 Call {M, M ˜ ˜ ˜ are called mgs M and M. If M, M is the identically 0 process, then M and M orthogonal. Proof. Now, M S and M D both possess predictable variation processes, called S M and M D . Note that ˜ t = 1 {(M S )2 − (M D )2 } (a) Mt · M for t ∈ I, 4

where (7) satisﬁes (8), and ˜ t −M, M ˜ t = 1 { [(M S )2 −M S t ]−[(M D )2 −M D t ] } for t ∈ I (b) Zt ≡ M t M t t t 4 is clearly a 0-mean mg, as (6) requires.

2

9.

DOOB–MEYER SUBMARTINGALE DECOMPOSITION

513

Predictable Processes Processes that are predictable will play an important role in the development of stochastic integration for martingales. We now need to deﬁne “predictability” more precisely. Deﬁnition 9.3 We deﬁne P(At : t ≥ 0) to be the σ-ﬁeld over Ω × I generated by the measurable rectangles of the form As × (s, t]

(10)

such that As ∈ As and 0 ≤ s ≤ t with s and t in I.

(Note that the s in As is at the left end of the time interval (s, t].) We agree to call P(At : t ∈ I) the predictable σ-ﬁeld. Now let {Ht }t∈I be a process such that H0 is A0 -measurable and such that the mapping (ω, t) → Ht (ω) from Ω × I into R is B − P(At : t ∈ I)-measurable; such a process is then called At -predictable. Remark 9.1 Now, by (18.1.4) we know that if for a.e. ω the path M· (ω) is of bounded variation on [0, t] for all t ∈ I, then {Mt , At }t∈I is a mg if and only if E[ I Hr dMr ] = 0 (11) for all H of the form H = 1A 1(s,t] for all A ∈ As and 0 ≤ s ≤ t with s, t ∈ I. But since condition (11) holds for enough H’s to generate the At -predictable processes, then it should hold for all At predictable processes for which expectations exist. Thus if H and M are suﬃciently regular so that Wt ≡ [0,t] H dM is a.s. well-deﬁned for each t ∈ I and so that E|Wt | < ∞, then {Ws , As }s∈[0,t] will be a mg for t ∈ I. That is, the measurability problems are easily handled for At -predictable H, and this is the largest σ-ﬁeld for which we can expect measurability requirements necessarily to hold. Deﬁnition 9.4 We say that a process {Mt , At }t∈I is of ﬁnite variation on I (1) (2) if M = M − M where each of the M (i) is ≥ 0 and with EM (i) (0) < ∞, and has sample paths in DI . Thus M (s, t] = (s,t] dM , and the total variation measure of the M process is well-deﬁned by |M |(s, t] ≡ (s,t] d[M (1) + M (2) ) for all s ≤ t in the index set I. [Thus E|Mt | < ∞ for such an M -process if and only if (1)

E[Mt

(2)

+ Mt ] < ∞.]

Proposition 9.1

Let {Ht }t∈I denote any At -predictable process. Then:

(12)

Every sample path H(ω, ·) is a B-measurable function on I,

(13)

Every rv Ht (·) ≡ H(·, t) is At -measurable, so that H(·) is At -adapted.

Proposition 9.2 (14)

Let {Ht }t∈I be adapted to the ﬁltration {At }t∈I . Then

H is At -predictable if H : (Ω, A, P ) → (DI− , DI− )

(that is, a left-continuous process having trajectories in DI− is predictable).

514

CHAPTER 18. MARTINGALES

Proposition 9.3 Let {Ht }t∈I be At -predictable. Let g denote a deterministic B-measurable function on I. Let {Mt }t∈I be a process adapted to the At ’s, and suppose a.e. sample path of M is a generalized df on I. Then: (15) (16)

The process {gt Ht }t∈I is At -predictable, { [0,t] g H dM }t∈I is a process on (DI , DI ) adapted to the At ’s,

provided that it is a.e. well-deﬁned. Proposition 9.4 Suppose {Mt , At }t∈I is of ﬁnite variation on I, and let d|M |t denote integration with respect to the total variation measure on I, for each ω in Ω. Suppose H is At -predictable. It follows that the process {Wt ≡ [0,t] g H dM }t∈I satisﬁes: (17)

{Wt , At }t∈I is a process on (DI , DI ), if b1 (t) ≡ E [0,t] |H| d|M | < ∞ for all t ∈ I.

(18)

The |Wt |k ’s are uniformly integrable, if bk ≡ E{( I |H| d|M |)k } < ∞ (for some k ≥ 1).

Proof. (a)

Consider (12). Let C ≡ {C : C = Aν × [(ν, ∞] ∩ I] with ν ≥ 0 and Aν ∈ Aν },

and note that (b)

σ[C] ≡ P ≡ P(At : t ∈ I) = [the predictable σ-ﬁeld].

We let E denote the class of all subsets E of Ω × I whose every section 1E (ω, ·) is a measurable function on I. Using 1E c (ω, ·) = 1 − 1E (ω, ·) and proposition 2.2.2, we see that E is a σ-ﬁeld. Since E ⊃ C is obvious, E ⊃ σ[C]. That is, (c)

1A (ω, ·) is a B-measurable function on I for every ω ∈ Ω and A ∈ σ[C].

Thus, every section of every simple predictable function is measurable on I. Thus, proposition 2.2.3 shows that (d)

every section H· (ω) of a predictable process H is measurable on I.

Consider (13). Just interchange the roles of t and ω in the proof presented in the previous paragraph. Consider (14). Now (as in proposition 12.4.3), for all ω and for all t ∈ I, (e)

H(t) = limn H (n) (t) ≡ limn {

n 2n−1 k=0

Hk/2n 1[k/2n n] } .

Now, for each n and each 0 ≤ k ≤ n2n − 1 the mapping (f)

(ω, t) → Hk/2n 1[k/2n
is B − P(At : t ∈ I)-measurable,

9.

DOOB–MEYER SUBMARTINGALE DECOMPOSITION

515

and likewise, (g)

(ω, t) → Hn (ω) 1[t>n]

is B − P(At : t ∈ I)-measurable.

Thus, the H (n) , and their limit H, is B − P (At : t ∈ I)-measurable. Consider (15). We merely note that a ﬁxed measurable function is predictable: The predictable σ-ﬁeld contains Ω × B, and these are enough sets to demonstrate the measurability of g. Clearly, the product of measurable functions is measurable, and so g H is thus At -predictable. Consider (16). That this is adapted to the At ’s follows from (13), because the integral of both the positive and negative parts of g H are deﬁned by the MCT. The right continuity of the paths follows from the absolute continuity of the integral, once the discontinuities of M (ω, ·) are removed and then added back afterwards. Consider (17) and (18). Now, |H| is measurable on I by (12), and the integrals deﬁning Wt and Wt2 are a.s. ﬁnite when the stated is ﬁnite. Uniform expectation integrability trivially follows from A |Wt |k dP ≤ A { I |H| d|M |}k dP for all t ∈ I, and the absolute continuity of the outer integral. 2

516

10

CHAPTER 18. MARTINGALES

Predictable Processes and

·

0 H dM Martingales Notation 10.1 Recall that I equals [0, θ) or [0, θ] for some 0 < θ ≤ ∞, and I¯ ≡ [0, θ]. Suppose {Mt , At }t∈I is a 0-mean mg on (DI , DI ) that is of ﬁnite variation on I, and let {Ht }t∈I be an At -predictable process. Then bk ≡ E({ I |H| d|M |}k ) = supt∈I bk (t) , (1) where bk (t) ≡ E({ [0,t] |H| d|M |})k (by the MCT). These quantities are well-deﬁned, but may be inﬁnite. We deﬁne Wt ≡ [0,t] H dM (2) for t ∈ I, and then apply propositions 18.9.3 and 18.9.4 to conclude that: (3)

Wt is a process on (DI , DI ) adapted to the At ’s, provided that b1 (t) < ∞ for all t ∈ I.

(4)

The |Wt |k ’s are uniformly integrable, provided that bk < ∞ (for this ﬁxed k ≥ 1).

The next theorem will focus on the mg structure of {Wt , At }t∈I . Deﬁnition 10.1 Saying that the M -mg possesses a predictable variation process will here be taken to mean that there exists an , ≥ 0, and At -predictable process {M t }t∈I on (DI , DI ) for which {Mt2 − M t , At }t∈I is a 0-mean mg. Theorem 10.1 Let the 0-mean martingale M on (DI , DI ) be of ﬁnite variation on the given interval I. (A) Suppose b1 ≡ E I |H| d|M | < ∞. Then (5)

{Wt , At }t∈I¯

is a uniformly integrable mg,

with

EWt = E(H0 M0 ).

[If only b1 (t) < ∞ for each t ∈ I, we can still claim that {Wt , At }t∈I is a mg.] (B) Suppose EMt2 < ∞ for t ∈ I, and that M possesses a predictable variation process {M t , At }t∈I . Now, b2 ≡ E({ I |H| d|M |}2 ) < ∞ if and only if we have b2 = E I H 2 dM < ∞. Then (6)

{Wt , At }t∈I¯ is a square-integrable closed mg with means E(H0 M0 ),

and possesses a predictable variation process {W t , At }t∈I¯ on (DI¯, DI¯) given by ¯ (7) W t = [0,t] H 2 dM that is uniformly integrable, for t ∈ I, with W 0 = H02 M 0 . Moreover, {Lt , At }t∈I¯ deﬁned by (8)

Lt ≡ Wt2 − W t

¯ is a uniformly integrable closed mg with 0 means on I.

[If only b2 (t) < ∞ for each t ∈ I, we can still claim that (6), (7), and (8) hold on I (but without claiming closure, uniform integrability, or square integrability).]

10.

PREDICTABLE PROCESSES AND

· 0

H DM MARTINGALES

517

˜W ˜ θ, ˜ ,L ˜ both satisfy the hypotheses of Corollary 1 Suppose H, θ, W, L and H, ˜ ˜ part (ii) of the theorem, in that b2 < ∞ and b2 < ∞ and M (·) exists on [0, θ ∧ θ]. Deﬁne the predictable covariation process by ˜ ˜ ˜ t ≡ (9) for t ∈ [0, θ ∧ θ]. W, W [0,t] H H dM Then (as the name would suggest), (10)

˜ t − W, W ˜ t , At } {Wt W ˜ t∈[0,θ∧θ]

is a closed mg with 0 means

having (11)

˜ t ] = EW, W ˜ s∧t Cov[Ws , W

˜ for all s, t ∈ [0, θ ∧ θ].

Remark 10.1 The key to having a clean and strong conclusion in this theorem was hypothesizing the existence of the M -process. And, typically, we do know the form of the M -process. Thus the Doob-Meyer theorem 18.9.1 was downplayed. Exercise 10.1 Let r ≤ s ≤ t be in I, in theorem 10.1(ii). Verify that 2 (12) for all Ar ∈ Ar . Ar {M (s, t] − M (s, t]} dP = 0 Let 0 ≡ t0 < t1 < · · · < tn ≡ t in the corollary. Verify that: ˜ t = n E{M (ti−1 , ti ] M ˜ (ti−1 , ti ]}, EM, M k=1 n ˜ ˜ i=1 M (ti−1 , ti ] M (ti−1 , ti ] →p M, M t

Exercise 10.2 (13) (14)

as meshn ≡ max{ti − ti−1 : 1 ≤ i ≤ n} → 0. Proof. Let H denote the class of all At -predictable processes for which the claims of the theorem holds. Claim 1: H contains {1C : C ∈ C} for C deﬁned by (a)

C ≡ {C : C ≡ Av × [(v, ∞] ∩ I]

with v ∈ I and Av ∈ Av }.

(Note that C ∪ {Ω × I} is a π ¯ -system.) Let H ≡ 1C for some C ∈ C, and then for t ∈ I deﬁne the process Wt ≡ [0,t] H dM = 1Av 1(v∧t,t] (r) dM (r) = 1Av M (v ∧ t, t] . (b) Note that the process (c)

{Wt , At }t∈I

has the same integrability properties as does {Mt , At }t∈I ;

use E|Wt | ≤ 2 E|Mt | or EWt2 ≤ 4 EMt2 , as needed. Furthermore, for s < t in I and As ∈ As , we have by (b) As (Wt − Ws ) dP = As 1Av {M (v ∧ t, t] − M (v ∧ s, s]} dP (d) by (18.1.4); = As ∩Av M ((v ∨ s) ∧ t, t] dP = 0 this holds, since As ∩ Av ∈ A(v∨s)∧t if v ≤ t, while M ((v ∧ s) ∧ t] = 0 if v > t. Thus,

518

CHAPTER 18. MARTINGALES

{Wt , At }t∈I is a mg by (18.1.4). Also by (18.1.4), (e)

EWt = E{1Av E(M (v ∧ t, t] |Av )} = E(1Av · 0) = 0

if v ≤ t,

while EWt = 0 if v > t, since Wt = 0 in this case. Note that W0 ≡ 0 for all of the current W -processes; thus we will now write W(0,t] rather then Wt . Thus, (c), (d), and (e) yield that (f)

{W(0,t] , At }t∈I

is a 0-mean mg that satisﬁes (c);

and thus it is integrable when b1 < ∞. Now suppose EMt2 < ∞ for each t ∈ I and suppose the M -process possesses the predictable variation process {M t , At }t∈I . Deﬁne W t = [0,t] H 2 dM = 1Av 1(v∧t,t] dM = 1Av M (v ∧ t, t], (g) which is clearly a , ≥ 0, and At -predictable process (since M is such a process) with W 0 = 0. Then (h)

Lt ≡ Wt2 − W t = 1Av {M 2 (v ∧ t, t] − M (v ∧ t, t]} .

Furthermore, for s < t in I and for As ∈ As we may conclude that (as t ≤ v, or as s ≤ v ≤ t, or as v ≤ s) (Lt − Ls ) dP As    As 0 dP = As ∩Av {M 2 (v, t] − M (v, t]} dP  2 2  As ∩Av {M (v, t] − M (v, t] − M (v, s] + M (v, s]} dP

(i)

 0 = 0  0

trivial by exercise 10.1 by exercise 10.1 and square M 2 (v, t] = {M (v, s] + M (s, t]}2 ,

so that {Lt , At }t∈I is a mg by (18.1.4). Then (j)

ELt = E{1Aν E{M 2 (v ∧ t, t] − M (v ∧ t, t] |Aν } } = E{1Aν · 0} = 0,

using (h), (g), and (b) and the existence of M . Also, E|Lt | ≤ EWt2 + EW t ≤ 8 EMt2

for all t,

and this {Lt , At }t∈I is integrable when M is square integrable. To summarize: (k)

{Lt , At }t∈I

is a 0-mean mg, which is integrable when b2 < ∞;

and this statement veriﬁes that W t deﬁned in (g) gives the predictable variation process of W .

10.

PREDICTABLE PROCESSES AND

· 0

H DM MARTINGALES

519

By the mg L2 -convergence theorem of exercise 18.3.7, the square integrability of W yields the conclusions: ¯ are uniformly integrable, {Wt2 : t ∈ I}

(l)

Wt →a.s. and

(m)

{Wt2 , At }t∈I¯ is a submg, with EWt2 EWθ2 .

L2

(some Wθ ),

Now, for each ω the W t are (with W t ≥ 0), so that EW θ ≡ limtθ EW t = limtθ EWt2 (n)

by the MCT (giving W t →L1 W θ )

since the Lt ’s form a mg on I

= EWθ2 ≤ 4 supt EMt2

by (l), and the comment following (c),

so that (l) through (n) combine (using the L2 and L1 convergence) to show that (o)

Lθ ≡ Wθ2 − W θ

closes the mg.

Thus, we also state speciﬁcally (note also theorems 18.3.1 and 18.3.2) that (p)

{Wt2 }t∈I¯,

{Lt }t∈I¯,

and {W t }t∈I¯ are all uniformly integrable.

Claim 2: 1Ω×I ∈ H. This merely adds W0 = H0 (∆M0 ) = H0 M0 to the mg W (0, t] of case 1 that was based on C ≡ Ω × [(0, ∞] ∩ I], and correspondingly adds H02 M0 to the predictable variation. Thus, we see that for any H ∈ H we will have (q)

EWt = E(H0 M0 ) and ELt = 0 for all t,

while W 0 = H02 M0 .

It thus suﬃces to work on (0, t], rather than [0, t], in what follows. Claim 3. If 1An ∈ H where An A, then A ∈ H. Now, Hn (ω, ·) ≡ 1An (ω, ·) is measurable on [0, ∞] ∩ I by (18.9.14), and thus its limit H(ω, ·) ≡ 1A (ω, ·) (which is dominated by the 1 function, during case 3) is also (by proposition 2.2.2). [We will reread this proof later (in case 6) for more general Hn and H. This accounts for various “during case 3” remarks.] Write M = M (1) − M (2) , and review deﬁnition 18.9.4. Now, (1) (2) | (s,t] Hn dM | ≤ (0,t] H d|M | < ∞ a.s. (≤ Mt + Mt , during case 3). Applying the DCT once for each ω, with dominating function H (the 1 function, in case 3), gives (r)

(i)

fn,t (ω) ≡ (i)

(i)

(s,t]

Hn dM (i) → ft (ω) ≡ (i)

where both fn,t and thus its limit ft

(s,t]

H dM (i)

for i = 1, 2 and s ∈ I,

are At -integrable (also, b1 < ∞ gives closure).

520

CHAPTER 18. MARTINGALES

We also have (1) (2) 0 = As [ (s,t] Hn dM ] dP = As [fn,t − fn,t ] dP → (s)

=

(1)

As

As

[ft

(2)

− ft ] dP

[ (s,t] H dM ] dP

so (18.1.4) yields that (t) Wt ≡ (0,t] H dM

(i)

by the DCT twice, with the ft

dominating

for each t ≥ s and all As ∈ As ,

is a 0-mean mg with respect to the At ’s, for t ∈ I.

Now, suppose of M that it is square integrable and possesses predictable varia(1) (2) (1) (2) tion process M t . We deﬁne fn,t ≡ fn,t − fn,t and ft ≡ ft − ft , and then 2 Ln,t ≡ ( (0,t] Hn dM )2 − (0,t] Hn2 dM ≡ fn,t − ψn,t (u)

→a.s. Lt ≡ ft2 − ψt ≡ ( (0,t] H dM )2 − (0,t] H 2 dM ,

by applying the DCT once for each ω, with dominating function H (the 1 function, in case 3). Now, Ln is a 0-mean mg (as Hn ∈ H), and thus (18.1.4) and then (u) combine to give (v) 0 = As (Ln,t − Ln,s ) dP → As (Lt − Ls ) dP and 0 = ELn,t → ELt by the DCT, with dominating functions on Ω given ﬁrst by ( (0,t] |H| d|M |)2 and (1) (2) then by (0,t] H 2 dM ) (which in case 3 are dominated by Mt + Mt and M t ). Thus, (18.1.4) and (u) show that (w)

{Lt , At }t∈I

is a 0-mean mg.

Moreover, (l)–(p) still hold, so that this martingale can be closed. ˜ are in H, then H + H ˜ and c H are in H. Claim 4. If H and H S D ˜ ˜ Let H ≡ H + H and H ≡ H − H, so that for t ∈ I ∩ I˜ we can write D ˜ t and WtD ≡ ˜ (x) WtS ≡ (0,t] H S dM = Wt + W (0,t] H dM = Wt − Wt . ˜ let Now, for t ∈ I ∩ I, (15) W S t ≡ (0,t] (H S )2 dM

and

W D t ≡

(0,t] (H

) dM .

D 2

˜ then it is also ﬁnite for Note that if any b value in (1) is ﬁnite for both H and H, both H S and H D (use the triangle inequality for k = 1 and Cauchy–Schwarz for k = 2). That c H ∈ H is trivial. Thus H is a vector space. m Claim 5. Every simple function 1 xi 1Ai with m ≥ 1, xi ∈ R, and Ai ’s in σ[C] m having 1 Ai = Ω × I is in H. This is just the ﬁrst conclusion of the exercise 2.2.2 measurability criterion, and σ[C] is exactly the predictable σ-ﬁeld.

10.

PREDICTABLE PROCESSES AND

· 0

H DM MARTINGALES

521

Claim 6. H ∈ H for any H ≥ 0 satisfying the hypotheses of this theorem. Choose simple functions Hn H, for Hn ’s as in claim 5. Now reread the proof of claim 3 to see that H ∈ H. (This is why Hn and H were written in place of 1An and 1A in the proof of claim 3.) Claim 7. The theorem holds. Let H = H + − H− for H + and H − as in claim 6, and apply claim 4. Claim 8. b2 = E({ I |H| d|M |}2 ) < ∞ is equivalent to b2 = E I H 2 dM < ∞. We can suppose H ≥ 0, else we use H = H + − H − ; and we may suppose M is , else we use M = M (1) − M (2) . Then E( I H dM )2 = E(lim( I Hn dM )2 ) with Hn as in claim 6 (y) = E(lim I Hn2 dM ) since Hn ∈ H = E I H 2 dM

by the MCT

demonstrates the claim. Claim 9. The corollary holds. Let H S , H D , W S , and W D be as in claim 4. Deﬁne ˜ t ≡ 1 {W S t − W D t } = ˜ W, W (16) 4 [0,t] H H dM

˜ for t ∈ [0, θ ∧ θ].

It then follows that (17)

{Zt , At }t∈[0,θ∧θ] ˜

is a 0-mean mg,

where the Z-process is deﬁned by (18)

˜ t − W, W ˜ t = 1 {[(W S )2 − W S t ] − [(W D )2 − W D t ]} ; Zt ≡ Wt W t t 4

just apply (8) to W S and W D . Then ˜ t ) = E(Ws W ˜ t ) + E(Ws E{W ˜t − W ˜ s |A}) E(Ws W (z)

˜ s ) + E(Ws · 0) = E(Ws W ˜ s ) = EW, W ˜ s = E(Ws W

by deﬁnition (16).

2

522

11

CHAPTER 18. MARTINGALES

The Basic Censored Data Martingale

Let X denote the failure time on [0, ∞) of a particular mechanism. However, the value of X may be unobservable, because there is also a censoring time Y at which our ability to observe the mechanism ceases. Our interest is in the survival function 1 − F associated with the df F of X. We now deﬁne (1)

Z ≡X∧Y

and

δ ≡ 1[X≤Y ] ,

where δ equals 1 or 0 according as the observable rv Z is equal to the failure time X or the censoring time Y . The cumulative hazard (or cumulative net hazard ) of X is deﬁned by 1 Λ(t) ≡ (2) dΛ ≡ dF (x) for t ≥ 0. 1 − F− [0,t] [0,t] The cumulative crude hazard of X is deﬁned by P (Y ≥ x|X = x) Λ∗ (t) ≡ dF (x) P (Z ≥ x) [0,t] (3)

= [0,t]

P (X ≥ x)P (Y ≥ x|X = x) dΛ(x) P (X ≥ x, Y ≥ y)

for t ≥ 0, which reduces to Λ(t) in case X and Y are independent rvs. However, (4)

Λ∗ = Λ is a more general statement then the independence of X and Y ,

since P (X = Y = 0) = 12 while P (X > x, Y > y) = exp(−λ(x + y)/2) implies that Λ = Λ∗ . Our interest is in F . But knowing only the joint distribution of the observable rvs Z and δ for a completely general joint distribution of (X, Y ), as in (3), is not enough to uniquely determine the df of F . That is, F is a nonidentiﬁable parameter in that situation. Nevertheless, Λ∗ is a well-deﬁned population parameter in the context of knowing the null model of the joint distribution of (X, Y ). Thus the following results from probability theory are correct. But the statistician who uses only (Z, δ) information will not be able to estimate F in a nonidentiﬁable model. For t ≥ 0 let (5)

Nt ≡ 1[Z≤t, δ=1] ,

and

Yt ≡ 1[Z≥t] ,

(6)

Mt ≡ Nt − At

where

At ≡

(7)

M t ≡

(8)

Ft ≡ σ[1[X≤s] , 1[Y ≤s] , 1[X≤s] δ , 1[Y ≤s] δ : 0 ≤ s ≤ t] .

[0,t] Y ×

[0,t]

Y dΛ∗ = Λ∗ (Z ∧ t),

[1 − ∆Λ∗ ] dΛ∗ ,

Now, ∆F ≡ F − F− , and ∆Λ and ∆Λ∗ are analogous. As in proposition 18.9.3, both Λ and Λ∗ are right-continuous generalized dfs. None of the jumps of Λ can

11.

THE BASIC CENSORED DATA MARTINGALE

523

exceed 1 (since ∆Λ ≤ ∆F/(1 − F− ) ≤ (F − F− )/(1 − F− ) ≤ 1). Also, ∆Λ∗ ≤ 1 for all t. Now, ∆Λ∗ (t) > 0 only if ∆F (t) > 0. But in this case ∆Λ∗ (t) = [P (Y ≥ t and X = t)/P (Y ≥ t and X ≥ t)] ∆F ≤ [1] ∆F ≤ 1 . To summarize, (9)

∆Λ(t) ≤ 1 and ∆Λ∗ (t) ≤ 1 for all t ≥ 0, for the gdfs Λ and Λ∗ .

Theorem 11.1 (The censored data mg) (10)

(Mt , Ft )t∈[0,∞]

is a square-integrable mg with 0 means.

(11)

(M t , Ft )t∈[0,∞]

is its predictable variation process.

(12)

(M2t − M t , Ft )t∈[0,∞]

is a closed mg with 0 means.

(13)

EM2 (t) = EM t =

(14)

≤

[0,t]

[0,t] P (Z

≥ s) [1 − ∆Λ∗ (s)] dΛ∗ (s)

P (Y ≥ x|X = x) dF (x) = P (Y ≥ X and 0 ≤ X ≤ t) ≤ F (t) ≤ 1.

(15)

M0 = 1[X=0] − P (X = 0)

and

(16)

M∞ = 1[δ=1] − Λ∗ (Z)

and

Now, deﬁne (17) Wt ≡

M 0 = P (X = 0)[1 − P (X = 0)] . M ∞ = [0,∞) Y × [1 − ∆Λ∗ ] dΛ∗ .

H dM

and

[0,t]

˜t ≡ W

˜ dM H

for all t ≥ 0,

[0,t]

˜ Let I1 denote the largest interval for for some Ft -predictable processes H and H. which we have ﬁniteness of the quantity |E I1 H Y dΛ∗| = |E I1 H dN| = | I1 H(x) P (Y ≥ x|X = x) dF (x) | . (18) Note that I1 must be of the form [0, θ1 ) or [0, θ1 ] for some 0 ≤ θ1 ≤ ∞; just consider H = H + − H − . We further deﬁne (19) W t ≡ H 2 dM = H 2 Y [1 − ∆Λ∗ ] dΛ∗ for all t ≥ 0. [0,t]

[0,t]

Let I2 , necessarily of the form [0, θ2 ) or [0, θ2 ] for some 0 ≤ θ2 ≤ θ1 ≤ ∞, be the largest interval for which we have (20) EW t = E(H 2 Y) [1 − ∆Λ∗ ] dΛ∗ < ∞ . I2

Now, (18) and (20) guarantee that W and W are a.s. well-deﬁned on I¯1 and I¯2 , respectively. [This example motivated our introduction of b1 , b2 , b2 , and b1 (t) and ˜ also hold. b2 (t) above.] Analogous deﬁnitions and remarks for W

524

CHAPTER 18. MARTINGALES

Theorem 11.2 (Transforms of the censored data mg) (21)

(Wt Ft )

is a mg with 0 means on I1 .

(22)

(Wt , Ft )

is a square-integrable mg with 0 means on [0, θ2 ] .

(23)

(W t , Ft )

is its predictable variation process on [0, θ2 ] .

W0 = H0 × ∆M0 = H0 [1[X=0] − P (X = 0)]

and

(24) W 0 = H02 × P (X = 0) [1 − P (X = 0)] .

(26)

˜ t − W, W ˜ t ) is a closed mg with 0 means on [0, θ2 ∧ θ˜2 ], where (Wt W ˜ t ≡ ˜ dM = ˜ [1 − ∆Λ∗ ] dΛ∗ . W, W HH HH [0,t] [0,t]

(27)

˜ t ] = EW, W ˜ s∧t = Cov[Ws , W

(25)

[0,s∧t]

˜ Y) [1−∆Λ∗ ] dΛ∗ on [0, θ2 ∧θ˜2 ]. E(H H

Remark 11.1 Corollary 1 to theorem 18.10.1, theorem 11.2, and theorem 11.3 below also allow conclusions on I1 [or on I2 ] when only b1 (t) < ∞ on I1 [or b2 (t) < ∞ on I2 ] is hypothesized. Model the appropriate conclusions on theorem 18.10.1. 2 Remark 11.2 All results hold true if the rvs have dfs on (−∞, ∞) instead of [0, ∞). Only trivial and obvious changes are needed in the statements. 2 Proof.

(a)

Clearly, Mt is Ft -measurable. Also, M is integrable, since E|Mt | ≤ ENt + [0,t] P (Z ≥ s) dΛ∗ (s) ≤ F (t) +

[0,t]

P (Y ≥ x|X = x) dF (x) ≤ 2F (t) ≤ 2.

To verify the mg property we will show that (b) E(Nt − Ns |Fs ) = E{ (s,t] 1[Z≥r] dΛ∗ (r)|Fs } . Now, agreeing that 0/0 = 0, we have E(Nt − Ns |Fs ) = E(1[ss] P (s < Z ≤ t, δ = 1|Z > s) = 1[Z>s] P (s < X ≤ t, δ = 1) /P (Z > s) = 1[Z>s] (s,t] P (Y ≥ x|X = x) dF (x) /P (Z > s) (c)

= 1[Z>s]

[s,t]

P (Z ≥ x) dΛ∗ (x) /P (Z > s) ,

11. while

(d)

THE BASIC CENSORED DATA MARTINGALE

525

E( (s,t] 1[Z≥r] dΛ∗ (r)|Fs ) = (s,t] E(1[Z≥r] |Fs ) dΛ∗ (r) = 1[Z>s] (s,t] P (Z ≥ r|Z > s) dΛ∗ (r) = 1[Z>s] (s,t] P (Z ≥ r) dΛ∗ (r) /P (Z > s) .

But (c) and (d) imply (b), showing that M is a mg on [0, ∞). The mg M has 0 means, since (e) EM0 = EN0 − E {0} 1[Z≥r] dΛ∗ (r) = ∆F (0) − ∆F (0) = 0 . We recall from section 4 the integration by parts formulas (f) U dV = U+ (t) V (t) − U (s) V− (s) − (s,t] V dU and (s,t] (g) U dV 2 = (s,t] U V dV + (s,t] U V− dV = (s,t] U (2 V − ∆V ) dV (s,t] for the left-continuous U and the right-continuous V . Now note that E{M2t |Fs } = E{(Nt − At )2 |Fs } = E{A2t 1[Z>t] |Fs } + E{12[Z≤t , δ=1] |Fs } (h) −2 E{At 1[Z≤t , δ=1] |Fs } + (E{A2t 1[Z≤t] |Fs } = 1[Z>s] Λ∗ (t)2 P (Z > t|Z > s) +1[Z>s] (i)

(s,t]

−2 1[Z>s]

P (Y ≥ x|X = x) dP (X ≤ x|Z > s) + 1[Z≤s , δ=1]

(s,t]

Λ∗ (x) P (Y ≥ x|X = x) dP (X ≤ x|Z > s)

−2 1[Z≤s , δ=1] Λ∗ (Z) + {−1[Z>s]

(s,t]

Λ∗ (x)2 dP (Z ≥ x|Z > s)

+1[Z≤s] Λ∗ (Z)2 } = 1[Z>s] Λ∗ (t)2 P (Z > t)/P (Z > s) +1[Z>s] (j)

(s,t] P (Z

+1[Z>s] (−2)

(s,t]

≥ x) dΛ∗ (x)/P (Z > s) + 1[Z≤s , δ=1] P (Z ≥ x) Λ∗ (x) dΛ∗ (x)/P (Z > s)

−2 1[Z≤s] , δ=1] Λ∗ (Z) +1[Z>s] (−1) (s,t] Λ∗ (x)2 dP (Z ≥ x)/P (Z > s) + 1[Z≤s] Λ∗ (Z)2 = 1[Z>s] (s,t] P (Z ≥ x) [1 − ∆Λ∗ (x)] dΛ∗ (x)/P (Z > s) + 1[Z≤s , δ=1] (k) +1[Z>s] Λ∗ (s)2 − 2 1[Z≤s , δ=1] Λ∗ (Z) + 1[Z≤s , δ=1]

526

CHAPTER 18. MARTINGALES

using the integration by parts results (f) and (g) for (k) via − (s,t] Λ∗ (x)2 dP (Z ≥ x)

(l)

= −Λ∗ (t)2 P (Z > t) + Λ∗ (s)2 P (Z > s) +

(m)

= −Λ∗ (t)2 P (Z > t) + Λ∗ (s)2 P (Z > s) + (s,t] P (Z ≥ x) {2Λ∗ (x) − ∆Λ∗ (x)} dΛ∗ (x) .

Also, E{M t |Fs } = = 1[Z>s] (n)

[0,t]

+1[Z>s] +1[Z≤s]

[0,s]

(s,t]

P (Z ≥ x) dΛ∗ (x)2

P (Z ≥ r|Fs ) [1 − ∆Λ∗ (r)] dΛ∗ (r)

[1 − ∆Λ∗ ] dΛ∗

(s,t] P (Z

[0,Z]

≥ r)[1 − ∆Λ∗ (r)] dΛ∗ (r)/P (Z > s)

[1 − ∆Λ∗ ] dΛ∗ .

Subtracting (n) from (k) gives E{M2t − M t |Fs } (o)

= {1[Z≤s , δ=1] − 2 1[Z≤s , δ=1] Λ∗ (Z) + (1[Z≤s] Λ∗ (Z)2 + 1[Z>s] Λ∗ (s)2 )} −{1[Z≤s]

(p)

[0,Z]

[1 − ∆Λ∗ ] dΛ∗ − 1[Z>s]

[0,s]

[1 − ∆Λ∗ ] dΛ∗ }

= M2s − M s .

Thus, M2 − M satisﬁes the mg condition (18.1.4) on [0, ∞). Clearly, (15) holds. Thus, M2t − M t has mean 0 at t = 0, and thus it has mean 0 at all t, by taking expectations in (o) and (p) with s = 0. Thus (13) and (14) are trivial. Now, M2 is square integrable by (14), so the (M2t )t∈[0,∞] are uniformly integrable by the mg Lr -convergence theorem of exercise 18.3.7, where Mt →a.s. M∞ is trivial from Nt N∞ ≡ 1[δ=1] a.s. and At A∞ ≡ Λ∗ (Z) a.s. Also, (q) M t M ∞ ≡ [0,∞) Y × [1 − ∆Λ∗ ] dΛ∗ a.s., where EM ∞ < ∞ by (14); and thus the (M t )t∈[0,∞] are uniformly integrable. The s-mg convergence theorem shows that M2∞ − M ∞ deﬁned above (as in (16)) closes this mg, giving (12). The process M of (7) is Ft -predictable by proposition 18.9.3, since Y is left continuous and Λ∗ is a generalized df (with 0 ≤ ∆Λ∗ ≤ 1); and M is clearly ≥ 0 and . Thus, (12) shows that M is the predictable variation process, as claimed in (11). Claim (10) was shown in (b), (e), and (14). We now turn to theorem 11.2. But it is an immediate corollary to theorem 18.10.1 and its corollary. 2

11.

THE BASIC CENSORED DATA MARTINGALE

527

The One-Sample Problem Suppose (X1 , Y1 ), . . . , (Xn , Yn ) are independent, though not necessarily identically distributed. But do suppose that X1 , . . . , Xn are iid with df F and cumulative hazard Λ. The cumulative crude hazards Λ∗i may be diﬀerent in each pair (Xi , Yi ), and so may the marginal dfs Gi of Yi . That is, we are allowing a diﬀerent censoring mechanism in each performance of the experiment Xi , so long as it does not aﬀect the marginal df F . Let (28)

Zi ≡ Xi ∧ Yi

and

δ ≡ 1[Xi ≤Yi ] .

The basic one-observation processes are, for t ≥ 0, (29)

N1i (t) ≡ 1[Zi ≤t , δi =1]

(30)

M1i (t) ≡ N1i (t) − A1i (t)

Y1i ≡ 1[Z≥t] ,

and

where

A1i (t) ≡

Y1i dΛ∗i = Λ∗i (Zi ∧ t) ,

[0,t]

(31)

Y1i [1 − ∆Λ∗i ] dΛ∗i ,

M 1i (t) ≡ [0,t]

(32)

Ft1i ≡ σ[1[Xi ≤s] , 1[Yi ≤s] , 1[Xi ≤s] δi , 1[Yi ≤s] δi : 0 ≤ s ≤ t] .

These are in accord with the earlier results of this section. Now suppose n 2 (33) i=1 cni = 1 for constants cni , and deﬁne n n (34) N ≡ Nn ≡ i=1 cni N1i and Y ≡ Yn ≡ i=1 cni Y1i , n M ≡ Mn ≡ i=1 cni M1i where (35) n n A ≡ An ≡ [0,·] i=1 cni Y1i dΛ∗i = i=1 cni Λ∗i (Zi ∧ ·) , (36)

M ≡ Mn ≡

n

c2ni Y1i [1 − ∆Λ∗i ] dΛ∗i ,

[0,·] i=1

i=1

(37)

n

c2ni M1i =

Ft ≡ Ftn ≡ ∩[Ft1i : 1 ≤ i ≤ n] .

˜ we now deﬁne, for t ∈ I, For Ft -predictable processes H and H ˜ ˜ ˜ dM , (38) W ≡ Wn ≡ H dM and W ≡ Wn ≡ H [0,·]

[0,·]

(39)

W ≡ Wn ≡ [0,·]

(40)

E

n I i=1

H 2 dM =

n

c2ni H 2 Y1i [1−∆Λ∗i ] dΛ∗i

[0,·] i=1

c2ni E(H 2 Y1i) [1 − ∆Λ∗i ] dΛ∗i < ∞ .

where

528

CHAPTER 18. MARTINGALES

Theorem 11.3 (One-sample censoring) Suppose (X1 , Y1 ), . . . , (Xn , Yn ) are independent rv’s, and all Xi ∼ = F . Suppose that (40) holds on the interval I ˜. ˜ holds on the interval I˜ for W for statements regarding W , and that (40) (A) It then follows that: (41)

(Mt , Ft )t∈[0,∞]

(42)

(M t , Ft )t∈[0,∞]

(43)

(M2t − M t , Ft )t∈[0,∞]

(44)

EM2t = EM t = M0 =

is a square-integrable mg with 0 means. is its predictable variation process.

n

(45) M 0 =

is a closed mg with 0 means.

n [0,t]

i=1 cni [1[Xi =0]

n

2 i=1 cni P (Xi

[Recall (36).]

2 i=1 cni P (Zi

≥ s) [1 − ∆Λ∗i (s)] dΛ∗i (s) ≤ F (t) ≤ 1 .

− P (Xi = 0)]

and

= 0) [1 − P (Xi = 0)] .

(46)

¯ and (Wt , Ft ) is a square-integrable mg with 0 means on I,

(47)

¯ (W t , Ft ) is its predictable variation process on I.

W 0 = H02 M 0 . [In fact, (Wt , Ft ) is a 0-mean mg on I whenever |E [0,t] H Y1i dΛ∗i | < ∞ for t ∈ I .] (48)

W0 = H0 M0

[Recall (39).]

and

(B) It also holds that: (49)

(M1i · M1j , Ft )t∈[0,∞]

(50)

M1i , M1j ≡ 0

(51)

¯˜ where ˜ t − W, W ˜ t ) is a closed mg with 0 means on I¯ ∩ I, (Wt · W

(52)

˜ t ≡ W, W

is a closed mg with 0 means, provided i = j .

on [0, ∞], provided that i = j .

˜ dM = HH

[0,t]

˜ HH

[0,t]

(53)

˜ t ] = EW, W ˜ s∧t = Cov[Ws , W

n

c2ni Y1i [1 − ∆Λ∗i ] dΛ∗i .

i=1 n

˜ Y) [1 − ∆Λ∗i ] dΛ∗i c2ni E(H H

[0,s∧t] i=1

¯˜ for all s, t in I¯ ∩ I. Exercise 11.1 Prove theorem 11.3. [Hint. All but (49) and (50) follow trivially n from theorem 11.1 and theorem 11.2, once you verify that M = i=1 c2ni M1i . Verifying this involves cross-product terms. Thus, you need only verify that (49) and (50), and then use them to show M = ni=1 c2ni M1i .]

12.

CLTS FOR DEPENDENT RVS

529

12

CLTs for Dependent RVs

Let {Xnk : k = 1, 2, . . . and n = 1, 2, . . .} be an array of rvs on a basic probability space (Ω, A, P ). For each n we suppose that Xn1 , Xn2 , . . . are adapted to an sequence of σ-ﬁelds An0 ⊂ An1 ⊂ An2 ⊂ · · · ⊂ A. For each n we suppose that κn in an integer-valued stopping time with respect to these (Ank )∞ k=0 . We now introduce Pk−1 (·) ≡ P (· |An,k−1 ) , (1) Ek−1 (·) ≡ E(· |An,k−1 ) ,

and

Vark−1 [·] ≡ Var[· |An,k−1 ] .

Our interest is in the sum n (2) Sn ≡ κk=1 Xnk . We will have reason to consider (3)

Xnk ≡ Xnk 1[ |Xnk |≤1]

and Xnk ≡ Xnk 1[ |Xnk |>1] .

What follows is the most basic CLT in this monograph. For row-independent rvs and κn ≡ n it reduces to the asymptotic normality condition of theorem 11.2.2, with c = 1. The second theorem implies the ﬁrst, and is very much in the spirit of the Lindeberg theorem. Theorem 12.1 (Basic dependent CLT) Now, Sn →d N (0, 1) if κn for all > 0 k=1 Pk−1 (|Xnk | ≥ ) →p 0 (4) (equivalently, max1≤k≤κn |Xnk | →p 0) , κn (5) (i.e., partial sums of the Xnk are nearly a mg), k=1 Ek−1 (Xnk ) →p 0 κn (6) k=1 Vark−1 [Xnk ] →p 1 . Theorem 12.2 We have Sn →d N (0, 1) if κn 2 (7) for all > 0, k=1 Ek−1 (Xnk 1[ |Xnk |≥] ) →p 0 κn (8) k=1 Ek−1 (Xnk ) →p 0 , κn (9) k=1 Vark−1 [Xnk ] →p 1 . If one replaces (5) by κn 2 (10) k=1 |Ek−1 (Xnk )| →p 1 , then (6) may be replaced by κn 2 (11) k=1 Ek−1 (Xnk ) →p 1 . If (4) holds, then (11) is equivalent to κn 2 (12) k=1 Xnk →p 1 .

530

CHAPTER 18. MARTINGALES

If Xn1 , Xn2 , . . . are mg diﬀerences, then (10) and (5) are implied by κn (13) k=1 |Ek−1 (Xnk 1[ |Xnk |>1] )| →p 0 . Moreover, (13) is implied by either (7) or (14)

E(max1≤k≤κn |Xnk |) →p 0 .

We now summarize these last claims. Theorem 12.3 (MG CLT) Suppose the Xn1 , Xn2 , . . . are mg diﬀerences, for each n. Then Sn →d N (0, 1), provided that any of the following occurs: (15)

Conditions (7) and (9) hold,

(16)

Conditions (12) and (14) hold,

(17)

Conditions (4) , (12) and (13) hold.

Chapter 19

Convergence in Law on Metric Spaces 1

Convergence in Distribution on Metric Spaces

Many results for convergence in distribution generalize to probability measures on a general metric space (M, d) equipped with its Borel σ-ﬁeld Md . We call such measures Borel measures. Instead of using convergence of dfs to deﬁne convergence in distribution (or law), we use directly the Helly–Bray idea embodied in the →d equivalences of theorem 11.7.4, and that theorem is here extended to general metric spaces. Skorokhod’s construction is also generalized to complete and separable metric spaces. Section 12.1 gave speciﬁc information on the two very important metric spaces that gave rise to (C, C) and (D, D). In section 19.2 the dual bounded Lipschitz metric is introduced, along with Hellinger, Prohorov, and total variation metrics. These are useful on function spaces. Deﬁnition 1.1 (Convergence in distribution) probability measures on (M, d, Md ) satisfying (1)

If {Pn : n ≥ 1} and P are

g dPn →

g dP

for all g ∈ Cb (M )

[where Cb (M ) ≡ { all bounded and d-continuous functions g from M to R}, and Cbu (M ) denotes those functions that are additionally d-uniformly continuous], then we say that Pn converges in distribution (or law) to P , or that Pn converges weakly to P ; and we write Pn →d P or Pn →L P . Similarly, if Xn , X are random elements in M for which (2)

E g(Xn ) → E g(X)

for all g ∈ Cb (M ),

then we write Xn →d X, Xn →L X or L(Xn ) → L(X).

531

532

CHAPTER 19. CONVERGENCE IN LAW ON METRIC SPACES

Theorem 1.1 (Portmanteau theorem; Billingsley) For probability measures {Pn : n ≥ 1} and P on any metric space (M, d, Md ) the following are equivalent: (3) Pn →d P [i.e., g dPn → g dP for all g ∈ Cb (M )]. (4) g dPn → g dP for all g ∈ Cbu (M ). (5)

lim Pn (B) ≤ P (B)

for all closed sets B ∈ Md .

(6)

lim Pn (B) ≥ P (B)

for all open sets B ∈ Md .

(7)

lim Pn (B) = P (B)

for all P -continuity sets B ∈ Md .

Proof. Clearly, (3) implies (4). Consider (4) implies (5): Suppose that (4) holds and that B is closed. Let

> 0. Then for integral m large enough the set Bm ≡ {x : d(x, B) < 1/m} satisﬁes (a)

P (Bm ) ≤ P (B) + ,

since Bm B as m → ∞. Let gm (x) ≡ ψ(md(x, B)) = max{0, (1 − m d(x, B))}, where ψ(t) is equal to 1, 1 − t, 0 according as t has t ≤ 0, 0 ≤ t ≤ 1, 1 ≤ t. Then (b)

1B ≤ gm ≤ 1Bm ,

and for each m ≥ 1, gm is Lipschitz and uniformly continuous. Hence, by (4) and also (a) and (b), (c) lim Pn (B) ≤ lim gm dPn = gm dP ≤ P (Bm ) ≤ P (B) + . Since > 0 was arbitrary, (5) follows. Equivalence of (5) and (6) is easy to show by taking complements. Consider (5) implies (3): Suppose that g ∈ Cb (M ) and that (5) holds. Now, transform g linearly so that 0 ≤ g(x) ≤ 1. Fix k ≥ 1, and deﬁne the closed set (d)

Bj ≡ {x ∈ M : j/k ≤ g(x)}

Then it follows that k+1 j−1 j=1 k P (x : (e)

≤

g dP <

j−1 k

k+1

for

j = 0, . . . , k + 1.

≤ g(x) < kj )

j j=1 k

P (x :

j−1 k

≤ g(x) < kj ).

Rewriting the sum on the right side and summing by parts gives k k (f) j=1 (j/k) [P (Bj−1 ) − P (Bj )] = (1/k) + (1/k) j=1 P (Bj ), which together with a similar summation by parts on the left side yields (g) (1/k) kj=1 P (Bj ) ≤ g dP ≤ (1/k) + (1/k) kj=1 P (Bj ).

1.

CONVERGENCE IN DISTRIBUTION ON METRIC SPACES

533

Applying the right side of (g) to Pn and the left side to P and using (5) for the closed sets Bj gives k lim n g dPn ≤ lim n k1 + k1 j=1 Pn (Bj ) (h)

≤

1 k

+

1 k

k 1

P (Bj ) ≤

1 k

+ g dP ,

and letting k → ∞ in (h) yields (i) lim n g dPn ≤ g dP. Applying (i) to (the nontransformed) −g yields (j) lim n g dPn ≥ g dP. Combining (i) and (j) gives (3). Consider (5) implies (7): Since (5) and (6) are equivalent, we have, with B0 ¯ its closure, the interior of any set B ∈ M and B (k)

¯ ≤ P (B). ¯ P (B 0 ) ≤ lim Pn (B 0 ) ≤ lim Pn (B) ≤ lim Pn (B) ≤ lim Pn (B)

¯ = P (B0 ), so the extreme If B is a P -continuity set, then P (∂B) = 0 and P (B) terms in (k) are equal; thus lim Pn (B) = P (B), as required by (7). Consider (7) implies (5): Since ∂{x : d(x, B) ≤ δ} ⊂ {x : d(x, B) = δ}, the boundaries are disjoint for diﬀerent δ > 0, and hence at most countably many of them can have positive P -measure. Therefore, for some sequence δk → 0, the sets Bk ≡ {x : d(x, B) < δk } are P -continuity sets and Bk B if B is closed. It follows that from B ⊂ Bk and then (7) that (l)

lim Pn (B) ≤ lim Pn (Bk ) = P (Bk ).

Then (5) follows from the monotone property of P , since Bk B as k → ∞.

2

Proposition 1.1 Pn →d P if and only if each subsequence {Pn } contains a further subsequence {Pn } such that Pn →d P . Proof. This is easy from deﬁnition 1.1 (and the fact that a sequence of real numbers has xn → x if and only if each {xn } contains a further subsequence {xn } such that xn → x), as in the corollary to Helly’s selection in theorem 11.7.3. 2

Theorem 1.2 (Slutsky’s theorem) Suppose that Xn , Yn are random elements taking values in a separable metric space (M, d, Md ), both deﬁned on some Ωn . (a) Show that d(Xn , Yn ) is a rv whenever (M, d) is separable. (b) If Xn →d X and d(Xn , Yn ) →p 0, then Yn →d X.

534 Proof. (a) (b)

CHAPTER 19. CONVERGENCE IN LAW ON METRIC SPACES For a closed set B and δ > 0 let Bδ ≡ {x : d(x, B) < δ}. Then P (Yn ∈ B) = P (Yn ∈ B, d(Xn , Yn ) < δ) + P (Yn ∈ B, d(Xn , Yn ) ≥ δ) ¯δ ) + P (d(Xn , Yn ) ≥ δ). ≤ P (Xn ∈ B

The second term on the right side of (a) goes to zero, since d(Xn , Yn ) →p 0 (note the following exercise). Then Xn ∼ = Yn and Xn →d X gives ¯δ ) lim sup P (Xn ∈ B) = lim sup P (Yn ∈ B) ≤ P (X ∈ B (c) ¯δ B as δ 0, since B is closed, so the via the portmanteau theorem (5). Then B result follows from the portmanteau theorem. 2 Exercise 1.1

Prove theorem 1.2(a).

Theorem 1.3 (Continuous mapping theorem) Let Xn →d X on (M, d, Md ), ¯ is another metric space) is continuous a.s. ¯ (where (M ¯ , d) and suppose g : M → M with respect to P ≡ PX (that is, P (X ∈ Cg ) = 1 for the continuity set Cg of g). Then, necessarily, g(Xn ) →d g(X). Proof. We simply note that this is essentially no diﬀerent from the proof of the Mann–Wald theorem. (Only now we apply the general Skorokhod construction of the following theorem, instead of the elementary Skorokhod theorem.) [This proof however, requires that the metric space be complete and separable. The next exercise asks the reader to provide a general proof.] 2 Exercise 1.2 Prove the continuous mapping theorem above by appeal to (5) and then proposition 2.2.4 (that the discontinuity set of a transformation between metric spaces is always measurable). This proof requires neither completeness nor separability! (And this is actually a much more elementary proof.)

Skorokhod’s Construction We earlier established the elementary form of Skorokhod’s theorem: If random variables Xn →d X0 , then there exist random variables {Yn : n ≥ 0} deﬁned on a common probability space satisfying Yn ∼ = Xn for all n ≥ 0 and Yn →a.s. Y0 . That proof relied on the inverse transformation. We now turn to the extension of the elementary Skorokhod theorem from R to a complete and separable metric space (M, d), whose open sets generate the Borel σ-ﬁeld. [The ﬁrst step in the proof will be to establish a preliminary result for just one P on (M, d, Md ).] Theorem 1.4 (Skorokhod construction) Suppose (M, d, Md ) is a complete and separable metric space and the measures {Pn : n ≥ 0} satisfy Pn →d P0 . Then there exist random elements {Xn : n ≥ 0} taking values in the space M (thus Xn (ω) ∈ M for all ω ∈ Ω) and all deﬁned on the common probability space (Ω, A, P ) ≡ ([0, 1], B([0, 1]), Lebesgue), with Xn ∼ = Pn and satisfying (8)

d(Xn (ω), X0 (ω)) → 0

for each ω ∈ Ω.

1.

CONVERGENCE IN DISTRIBUTION ON METRIC SPACES

535

Proposition 1.2 Suppose P is a probability measure on (M, d, Md ). Then there is a random element X deﬁned on (Ω, A, P ) = ([0, 1], B[0, 1], Lebesgue) and taking values in M that has distribution P . Proof. For each k, decompose M into a countable number of disjoint sets Ak1 , Ak2 , . . . whose diameter is diameter less than 1/k. Then arrange it so that Ak+1 reﬁnes Ak ≡ {Ak1 , Ak2 , . . .}. Make a corresponding decomposition of the unit interval as Ik ≡ {Ik1 , Ik2 , . . .} where the subintervals Ikj satisfy P (Akj ) = λ(Ikj ) and where the decomposition Ik+1 reﬁnes Ik . Let xkj be a point in Akj , and deﬁne (a)

if ω ∈ Ikj ⊂ [0, 1].

Xk (ω) = xkj

Since {Xk (ω), Xk+1 (ω), . . .} ⊂ (some one Akj ), its diameter is bounded by 1/k. Thus, {Xk (ω)} is Cauchy for each ω, limk Xk (ω) ≡ X(ω) exists, and as k → ∞. d(X(ω), Xk (ω)) ≤ 1/k → 0 ∗ For a given set B, write ≡ {j:Ajk ∩B =∅} , and similarly for unions of sets. Then (9)

P (Xk ∈ B) ≤ P (Xk ∈ = (b)

∗

λ(Ikj ) =

∗

∗

Akj ) =

∗

P (Xk ∈ Akj )

P (Akj )

≤ P (B 1/k ),

where B δ ≡ {x : d(x, B) ≡ inf y∈B d(x, y) < δ}. For a closed set B we have ∞ 1/k = B, and so k=1 B (c)

limk P (Xk ∈ B) ≤ P (B)

for closed sets B,

and hence the distribution of Xk converges to P . It follows from Slutsky’s theorem (with Yk = X for all k) that X ∼ = P. 2 Proof. Consider Skorokhod’s theorem. First construct the decompositions Ak of the proof of the previous proposition, but now do it in a way that makes each Akj a P -continuity set. Because ∂{y : d(x, y) < δ} ⊂ {y : d(y, x) = δ}, the spheres about x are P -continuity sets for all but countably many radii; hence M can be covered by countably many P -continuity sets all with diameter at most 1/k. The usual disjointiﬁcation procedure preserves P -continuity because ∂(B ∩ C) ⊂ (∂B) ∪ (∂C). Consider the decompositions Ik as before, and, for each n, construct successively (n) (n) (n) (n) ﬁner partitions Ik = {Ik1 , Ik2 , . . .} with λ(Ikj ) = Pn (Aj ). Inductively arrange (n)

(n)

the indexing so that Iki < Ikj if and only if Iki < Ikj ; here I < J for intervals I and J means that the right endpoint of I does not exceed the left endpoint of J. (1) (2) In other words, we ensure that for each k the families Ik , Ik , Ik , . . . are ordered similarly.

536

CHAPTER 19. CONVERGENCE IN LAW ON METRIC SPACES Deﬁne Xk as before, where xkj ∈ Akj , and deﬁne

(a)

(n)

(n)

if ω ∈ Ikj .

Xk (ω) = xkj

(n)

Again Xk (ω) converges to an X(ω) satisfying (9), and Xk (ω) converges to an X (n) (ω) satisfying (b)

(n)

d(X (n) (ω), Xk (ω)) ≤ 1/k → 0

as k → ∞.

And againX has distribution P and X (n) has distribution Pn . Since j [P (Akj ) − Pn (Akj )] = 0, it follows that (c)

j |λ(Ikj )

=2 =2

(d)

(n)

− λ(Ikj )| =

j [P (Akj )

j [P (Akj )

j |P (Akj )

− Pn (Akj )|

− Pn (Akj )] − Pn (Akj )]+ ,

where the next to the last sum extends over those j for which the summand is positive. Each summand goes to 0 as n → ∞ because the Akj are P -continuity sets, and it follows by the DCT (with dominating function identically equal to the constant function with value 1) that (n) (e) lim j |λ(Ikj ) − λ(Ikj )| = 0. n

(n)

Fix k and j0 , let α and αn be the left endpoints of Ikj0 and Ikj0 respectively, and let indicate summation over the set for which Ikj < Ikj0 , which is the same (n) (n) as the set for which Ikj < Ikj0 . Then (d) implies (f)

α=

j

λ(Ikj ) = lim n

j

(n)

λ(Ikj ) = lim αn . n

(n)

Similarly the right endpoint of the interval Ikj converges as n → ∞ to the right endpoint of the interval Ikj . (n) Hence, if ω is interior to Ikj , then ω lies in Ikj for all n large enough, so that the conclusions (9) and (b) give (g)

d(X(ω), X (n) (ω)) ≤ 2/k.

Thus, if ω is not an endpoint of any Ikj , then for each k we have that (g) holds for all suﬃciently large n. In other words, limn X (n) (ω) = X(ω) if ω is not in the set of end-points of the Ikj . This last set, being countable, has Lebesgue measure 0; thus if X (n) (ω) is redeﬁned as X(ω) on this set, X (n) still has distribution Pn and there is now convergence for all ω. This completes the proof, with X (n) for Xn and with X for X0 . 2 Exercise 1.3 Recall the partial sum process S n = {S n(t) : t ≥ 0} deﬁned earlier. Now, S n →d S by Donsker’s theorem (theorem 12.8.3), where S is a Brownian motion process in M = C[0, 1]. Consider the following four functions:

1.

CONVERGENCE IN DISTRIBUTION ON METRIC SPACES

537

(a) g(x) = sup0≤t≤1 x(t), 1 (b) g(x) = 0 x(t) dt, (c) g(x) = λ({t ∈ [0, 1] : x(t) > 0}), where λ denotes Lebesgue measure, (d) g(x) = inf{t > 0 : x(t) = b}, with b > 0 ﬁxed. For each of these real-valued functionals g of x ∈ C[0, 1], ﬁnd the discontinuity set Dg of g. [If we can show that the P measure of these discontinuity sets is zero, where P denotes the measure of S on C[0, 1], then it follows immediately from the continuous mapping theorem that g(Sn) →d g(S).] Exercise 1.4 Let S0 ≡ 0, S1 ≡ X1 , Sk ≡ X1 + · · · + Xk for k ≥ 1 be the n partial sums of the iid (0, 1) rvs that go into the deﬁnition of Sn. Represent n−3/2 k=1 |Sk | as a functional g(S n). Is the resulting g continuous?

Tightness and Relative Compactness The notion of tightness comes into play in a crucial way in the general theory of convergence in distribution on a metric space (M, d), since there now are more ways to “leave” the space than simply for mass to drift oﬀ to inﬁnity. Deﬁnition 1.2 (Tightness) Let P0 denote a collection of probability measures on (M, d, Md ). Then P0 is tight (or uniformly tight) if and only if for every > 0 there is a compact set K ⊂ M with (10)

P (K ) > 1 −

for all P ∈ P0 .

Deﬁnition 1.3 (Sequential compactness, or relative compactness) Let P0 be a family of probability measures on (M, d, Md ). We call P0 relatively compact (or sequentially compact) if every sequence {Pn } ⊂ P0 contains a weakly convergent subsequence. That is, every sequence {Pn } ⊂ P0 contains a subsequence {Pn } with Pn →d (some probability Q) (not necessarily in P0 ). Proposition 1.3 Let (M, d, Md ) be a separable metric space. (a) If {Pn }∞ n=1 is relatively compact with limit set {P }, then Pn →d P . (b) If Pn →d P , then {Pn }∞ n=1 is relatively compact. (c) We have thus related convergence in distribution to relative compactness. Proof. See proposition 1.1 for both(a) and (b). (That is, we have merely rephrased things we already know.) 2 Theorem 1.5 (Prohorov) Let P0 denote a collection of probability measures on the metric space (M, d, Md ). (a) If P0 is tight, then it is relatively compact. (b) Suppose that (M, d, Md ) is separable and complete. If the collection P0 is relatively compact, then it is tight. (c) We have thus related relative compactness to tightness, at least on complete and separable metric spaces.

538

CHAPTER 19. CONVERGENCE IN LAW ON METRIC SPACES

Proof. A full proof progresses from M = Rk to R∞ , to sigma compact M , and ﬁnally to general M , at each step using the next simpler case. We present only the proof of (a) for the case M = Rk . If {Pn } is any sequence in P0 , then Helly’s selection theorem implies that the corresponding sequence of dfs {Fn } deﬁned by Fn (x ) ≡ Pn (−∞, x ] contains a subsequence {Fn } satisfying (p)

Fn (x ) → F (x )

for all x ∈ CF ,

where the sub df F is continuous from above. Now, there is a measure P on Rk such that P (a, b ] equals F diﬀerenced around the vertices of the k-dimensional rectangle (a, b ]. Now Pn →d P will follow if we can show that P (Rk ) = 1. Given > 0, choose K ⊂ Rk compact with Pn (K) > 1 − for all n ; this is possible by tightness of P0 . Now choose a, b in Rk such that K ⊂ (a, b ] and all 2k vertices of (a, b ] are continuity points of F (we can do this because at most a countable number of parallel (k − 1)-dimensional hyperplanes can possibly have positive P -measure. Since Pn (a, b ] equals Fn diﬀerenced around (a, b ], (a) yields Pn (a, b ] ≥ Pn (K) ≥ 1 − , so P (a, b ] ≥ 1 − . Since was arbitrary, P (Rk ) = 1. Hence P is a probability measure, Pn →d P , and P0 is relatively compact. 2

Convergence in Distribution on (D, D) We phrase results carefully in this subsection, so as to mention primarily the familiar metric · on D (while limiting mention of the contrived metric d of exercise 12.1.4 for which (D, d) is both complete and separable with Dd = D). Recall that D denotes the ﬁnite-dimensional σ-ﬁeld. Theorem 1.6 (Criterion for →d on (D, D); Chentsov) Let each Xn denote a process on (D, D). Suppose that for some a > 12 and b > 0 the increments of the Xn processes satisfy (11)

E |Xn (r, s] Xn (s, t] |b ≤ [ µn (r, s] × µn (s, t] ]a

for all 0 ≤ r ≤ s ≤ t ≤ 1

for some ﬁnite measure µn on the Borel subsets of [0, 1]. (Typically, a = 1 and b = 2, and note the relaxation of this condition in the remark below.) Suppose that µ is a continuous measure on the Borel subsets of [0, 1], and that either (12)

µn (s, t] ≤ µ(s, t]

for all 0 ≤ s ≤ t ≤ 1 and for all n ≥ 1

(13)

µn /µn ([0, 1]) →d µ/µ([0, 1])

or

as n → ∞.

(a) Then for the metric d of exercise 12.1.4 we have that (14)

{Xn : n ≥ 1} is relatively compact on (D, d).

(b) Especially, if we further have Xn →f.d. (some X for which P (X ∈ C) = 1, then (15)

g(Xn ) →d g(X) for all D-measurable and a.s. · -continuous g : D → R.

1.

CONVERGENCE IN DISTRIBUTION ON METRIC SPACES

539

Remark 1.1 For processes like the partial sum process S n the condition (11) is troublesome; but since S n is constant on the intervals [(i − 1)/n, i/n), it should be enough to verify (11) for r, s, t restricted to be of the form i/n. We now make this rough idea precise. For m ≥ 1 we let Tm ≡ {0 ≡ tm0 < tm1 < · · · < tmkm ≡ 1}, and measure the coarseness of this partition by deﬁning mesh(Tm ) ≡ max{tmi −tm,i−1 : 1 ≤ i ≤ km }. Let x ∈ D and let Am (x) denote the function in D that equals x(tmi ) at tmi for each 0 ≤ i ≤ km and is constant in between these points. We agree to call Am (x) the Tm -approximation of x. Suppose now that one of the two conditions (12) or (13) holds, that Xn = An (Xn ) with mesh(Tn ) → 0 as n → ∞, and that (11) holds for all r, s, t in Tn . Then both (14) and (15) hold. [That Xn = An (Xn ) means that Xn is equal to its own Tn -approximation; and this clearly holds for S n when 2 all tni = i/n.] “Proof ”. By analogy with Helly’s selection theorem, it should suﬃce to show the tightness. Thus, for each > 0 we must exhibit a compact set K of functions on a’s theorem (see [0, 1] for which P (Xn ∈ K ) > 1 − for all n. According to Arzel` exercise 6.2.11), a compact set of functions consists of a uniformly bounded set of functions that have a uniform bound on their “wiggliness.” A complex and delicate argument (slightly in the spirit of Kolmogorov’s inequality) based on (6) can be given to bound this “wiggliness.” Since the details are long and hard and only good for the present theorem, they will be skipped. See Billingsley (1968). 2 Exercise 1.5 (Prohorov) For any real-valued function on [0, 1] we now deﬁne the modulus of continuity ωδ (x) of x by ωδ (x) ≡ max{|xt − xs | : |t − s| ≤ δ} for each δ > 0. Let X, X1 , X2 , . . . denote processes on (C, C). Then Xn →d X on (C, · ) if and only if both Xn →f.d. X and limδ→0 lim supn→∞ P (ωδ (Xn ) > ) = 0 for all

> 0. (The modulus of continuity also measures the “wiggliness” of the process; and Prohorov’s condition implies that the processes are “not too wiggly”.) Exercise 1.6 (Doob) Use theorem 1.6 to establish that g(Un ) →d g(U) for all D-measurable and a.s. · -continuous functionals g on D. Exercise 1.7 (Donsker) Use theorem 1.6 to establish that g(Sn) →d g(S) for all D-measurable and a.s. · -continuous functionals g on D. Exercise 1.8 (Prohorov) Linearize S n between the i/n-points so as to make it a process on (C, · ), and then use exercise 1.3 to show that this linearized process converges in distribution to Brownian motion.

540

2

CHAPTER 19. CONVERGENCE IN LAW ON METRIC SPACES

Metrics for Convergence in Distribution

Deﬁnition 2.1 (Prohorov metric) [The L´evy metric of exercise 11.7.5 extends in a nice way to give a metric for →d more generally.] For any Borel set B ∈ Md and > 0, deﬁne B ≡ {y ∈ M : d(x, y) < for some x ∈ B}. Let P, Q be two probability measures on (M, d, Md ). If we set (1)

ρ(P, Q) ≡ inf{ : Q(B) ≤ P (B ) + for all B ∈ Md },

then ρ is the Prohorov metric (see exercise 2.1). (We note that this deﬁnition is not formed in a symmetric fashion.) Deﬁnition 2.2 (Dudley metric) (i) Label as BL(M, d) the set of all real-valued functions g on the metric space (M, d) that are bounded and Lipschitz (in the sense that both of the quantities (2)

g∞ ≡ sup |g(x)|

and

x∈M

gL ≡ sup [|f (x) − f (y)|/d(x, y)] x =y

are ﬁnite). For functions g in BL(M, d) we deﬁne (3)

gBL ≡ gL + g∞ ,

and so BL(M, d) = {g : gBL < ∞}. (ii) Now let P, Q be two probability measures on (M, Md ), and set (4) β(P, Q) ≡ sup{| g dP − g dQ| : gBL ≤ 1}. Then β is called the dual bounded Lipschitz distance (or Dudley distance) between the probability distributions P and Q. Proposition 2.1 Let P ≡ {all probability distributions P on (M, M) }. Then both ρ and β are metrics on P. (Equation (12) below will show that Mρ = Mβ , which we abbreviate here as M.) Exercise 2.1

Prove the previous proposition.

The following theorem says that both ρ and β metrize →d on (M, d, Md ) just as the L´evy distance L metrized the convergence in distribution →d of dfs on R1 . Theorem 2.1 (Metrizing →d ; Dudley) For any separable metric space (M, d) and probability measures {Pn : n ≥ 1} and P on the Borel σ-ﬁeld Md , the following are equivalent conditions:

(6)

Pn →d P, g dPn → g dP

(7)

β(Pn , P ) → 0,

(8)

ρ(Pn , P ) → 0.

(5)

for all g ∈ BL(M, d),

2.

METRICS FOR CONVERGENCE IN DISTRIBUTION

Theorem 2.2 (Ulam) (M, Md ) is tight.

541

For (M, d) complete and separable, each single P on

Proof. Let > 0. By the separability of M , for each m ≥ 1 there is a sequence Am1 , Am2 , . . . of open 1/m spheres covering M . Choose im so that ∞ P ( i≤im Ami ) > 1 − /2m . Now, the set B ≡ m=1 i≤im Ami is totally bounded in M , meaning that for each > 0 it has a ﬁnite -net (that is, a set of points xk with d(x, xk ) < for some xk , for each x ∈ B). [Nice trick!] By completeness of M , ¯ is complete and is also compact; see exercises 6.2.3(c) and 6.2.4(e). Since K≡B ∞ m c ¯ c ) ≤ P (B c ) ≤ ∞ P ([ (a) P (K c ) = P (B i≤im Ami ] ) < m=1 1 /2 = , the conclusion follows.

2

Proof. We now prove theorem 2.1, since Ulam’s theorem is in hand, but only under the additional assumption that M is complete. Clearly, (5) implies (6). We will now show that (6) implies (7). By Ulam’s theorem, for any > 0 we can choose K compact so that P (K) > 1 − . Let f |K denote f restricted to K. Now, the set of functions F ≡ {f |K : f BL ≤ 1} forms a · -totally bounded subset of a theorem of exercise 6.2.11(a)). Thus, for every the functions Cb (K) (by the Arzel`

> 0 there is some ﬁnite k ≡ k and functions f1 , . . . , fk ∈ F such that for any f ∈ F there is an fj with (a)

sup |f (x) − fj (x)| ≤ ;

moreover,

x∈K

sup |f (x) − fj (x)| ≤ 3 ,

x∈K

since f and fj are in F (note the second half of (2)). Let g(x) ≡ max{0, (1 − d(x, K)/ )} ;

then g ∈ BL(M, d) and 1K ≤ g ≤ 1K . Thus, g dPn → g dP , so that for n large enough we have (b) Pn (K ) ≥ g dPn > g dP − ≥ 1K dP − = P (K) − > 1 − 2 . Hence, for any f ∈ F we have from (a), (2), (b), and P (K) > 1 − that | f d(Pn − P )| = | (f − fj ) d(Pn − P ) + fj d(Pn − P )|

(c)

≤ | (f − fj ) dPn | + | (f − fj ) dP | + | fj d(Pn − P )| ≤ (3 + 2 × 2 ) + ( + 2 × ) + | fj d(Pn − P )| ≤ 11

for n chosen large enough. Hence (7) holds. We next show that (7) implies (8). Suppose a Borel set B and an > 0 are given. We let f (x) ≡ max{0, (1 − d(x, B)/ )}. Then f ∈ BL(M, d), fBL ≤ 1 + −1 , and 1B ≤ f ≤ 1B . Therefore, for any P and Q on M we have from (4) that (d) Q(B) ≤ f dQ ≤ f dP + (1 + −1 ) β(P, Q) ≤ P (B ) + (1 + −1 ) β(P, Q),

542

CHAPTER 19. CONVERGENCE IN LAW ON METRIC SPACES

and it follows that (e)

ρ(P, Q) ≤ max{ , (1 + −1 ) β(P, Q)}.

Hence, if β(P, Q) < 2 , then ρ(P, Q) < + 2 < 2 . Hence, for all P, Q we have ρ(P, Q) ≤ 2 β(P, Q). Thus, (7) implies (8). [It is also possible to establish the inequality β(P, Q)/2 ≤ ρ(P, Q). This would give (f) β(P, Q)/2 ≤ ρ(P, Q) ≤ 2 β(P, Q), showing that ρ and β are equivalent metrics (see (12) and exercise 2.3 below).] Finally, we will show that (8) implies (5). Suppose (8) holds. Let B denote a P -continuity set, and let > 0. Then for 0 < δ < with δ small enough, we have P (B δ \ B) < and P ((Bc )δ \ B c ) < . Then for n large enough we have (g)

Pn (B) ≤ P (Bδ ) + δ ≤ P (B) + 2

and also (h)

Pn (B c ) ≤ P ((Bc )δ ) + δ ≤ P (B c ) + 2 .

Combining these yields (i)

|Pn (B) − P (B)| ≤ 2 ,

and hence Pn (B) → P (B). By the portmanteau theorem, this yields (5).

2

Strassen’s Coupling Theorem Suppose Pn →d P0 on a separable metric space (M, d). Then theorem 2.1 gives ρ(Pn , P0 ) → 0, for Prohorov’s metric ρ, while Skorokhod’s theorem gives existence of random elements Xn on a common probability space satisfying d(Xn , X0 ) →a.s. 0. Claiming less than is true, d(Xn , X0 ) →p 0; or P (d(Xn , X0 ) ≥ ) → 0 as n → ∞. We are then naturally led to ask the question: How rapidly does this convergence occur? It turns out that this is essentially determined by ρ(Pn , P ). Alternatively, the following theorem can be used to bound ρ(Pn , P ), provided that a rate is available regarding Skorokhod. Theorem 2.3 (Strassen) Suppose that P and Q are measures on the Borel sets of a separable metric space (M, d). Then ρ(P, Q) < if and only if there exist X and Y deﬁned on a common probability space with X ∼ = P and Y ∼ = Q and coupled closely enough that P (d(X, Y ) ≥ ) ≤ . Proof.

See Dudley (1976, section 18).

2

2.

METRICS FOR CONVERGENCE IN DISTRIBUTION

543

Some Additional Metrics As shown in theorem 2.1, both the Prohorov metric ρ and also the dual-bounded Lipschitz metric β metrize weak convergence (→d ). Other stronger metrics are also available and often useful. Deﬁnition 2.3 (Total variation metric) the measurable space (M, Md ), let (9)

For probability measures P and Q on

dT V (P, Q) ≡ 2 sup{|P (A) − Q(A)| : A ∈ Md };

dT V is called the total variation metric. The total variation metric dT V is equal to dT V (P, Q) = |f − g| dµ = 2[1 − f ∧ g dµ],

Proposition 2.2 (10)

where f = dP/dµ, g = dQ/dµ, and µ is any measure dominating both P and Q (for example, P + Q). Proof.

Note that |f − g| = (f ∨ g) − (f ∧ g) = (f + g) − 2(f ∧ g).

2

Deﬁnition 2.4 (Hellinger metric) For probabilities P and Q on (M, Md ), let d2H (P, Q) ≡ [f 1/2 − g 1/2 ]2 dµ = 2[1 − (11) f g dµ], where f = dP/dµ, g = dQ/dµ, and µ is any measure dominating both P and Q (for example, P + Q); then dH is called the Hellinger metric. Exercise 2.2

dH does not depend on the choice of µ.

Here is a theorem relating these metrics and the Prohorov and bounded Lipschitz metrics. The inequalities in (12) show that ρ and β induce identical topologies. The inequalities in (13) show that the total variation metric dT V and the Hellinger metric dH induce identical topologies. Moreover, (14) shows that the β and ρ topology is the ﬁner topology (with more open sets and admitting more continuous functions). [Recall exercise 6.2.2 dealing with equivalent metrics.] Theorem 2.4 (Inequalities among the metrics) (a) For P and Q probability measures on (M, Md ), the following inequalities necessarily hold: (12) β(P, Q)/2 ≤ ρ(P, Q) ≤ 2 β(P, Q), (13)

d2H (P, Q) ≤ dT V (P, Q) ≤ dH (P, Q) {4 − d2H (P, Q)}1/2 ,

(14)

ρ(P, Q) ≤ dT V (P, Q).

(b) For dfs F, G on R (or Rk ) we have the following: (15)

L(F, G) ≤ ρ(F, G) ≤ dT V (F, G),

(16)

L(F, G) ≤ dK (F, G) ≤ dT V (F, G),

where dK (F, G) ≡ F − G∞ ≡ supx |F (x) − G(x)| is the Kolmogorov distance.

544 Exercise 2.3

CHAPTER 19. CONVERGENCE IN LAW ON METRIC SPACES Prove the ﬁrst inequality in (12).

Exercise 2.4 [Hint. To prove the left inequality, establish the √ Prove (13). inequality f g dµ ≥ f ∧ g dµ and second equality in (11). To show √ use√the √ √ the right inequality, write |f − g| = | f − g | | f + g |. Exercise 2.5

Prove (14).

Exercise 2.6 (Statistical interpretation of the dT V metric) Consider testing P versus Q. Find the test that minimizes the sum of the error probabilities, and show that the minimum sum of errors is P ∧ Q ≡ f ∧ g dµ. Note that P and Q are orthogonal if and only if P − Q ≡ dT V (P, Q) = 2 if and only if P ∧ Q = 0 if √ √ and only if f g dµ ≡ dP dQ = 0.

Appendix A

Distribution Summaries In this appendix we consider many of the common statistical distributions. First, we care about the maximum likelihood estimation of their parameters. For some distributions, such as the Weibull and Gamma, terms of the form Γ(α) enter into the maximization process. Diﬀerentiating Γ(·) or log Γ(·) leads naturally to a study of the gamma and digamma functions. This is done in section A.1. (Whittaker and Watson (1963) proves a very useful guide.) A summary of the main results for maximum likelihood is contained in section A.2. Second, we care about the shape of these distributions, as measured by their ﬁrst four cumulants. This is considered in section A.3.

545

546

1

APPENDIX A. DISTRIBUTION SUMMARIES

The Gamma and Digamma Functions

Deﬁne Euler’s constant γ by " ! n 1 (1) − log n = 0.577215664901533 . . . . γ ≡ lim n→∞ k k=1

Weierstrass deﬁned the gamma function Γ(·) in 1856 via

1 ∞ −z/k z (2) 1 + ) e ; = z exp (γ z) k=1 k Γ(z) clearly, Γ(z) is analytic except for simple poles at z = 0, −1, −2, . . . . [A function is analytic at z if it has a derivative at all points in a neighborhood of z.] (We will establish equations (1) and (2) below.) From (2) we obtain ∞ − log Γ(z) = log z + γ z + k=1 [ log(1 + kz ) − kz ] (3) for z = 0, −1, −2, . . . . Hence the digamma function ψ(z) ≡ ψ(z) ≡ (4)

d dz

d dz

log Γ(z) satisﬁes (via diﬀerentiation)

1 1 log Γ(z) = − z1 − γ − ∞ − k=1 z+k k

∞ 1 1 =− −γ+z z k(k + z)

for z = 0, −1, −2, . . . ,

k=1

and this is a useful computational formula. Diﬀerentiating term by term again gives (5)

ψ (z) =

∞ k=0

1 (k + z)2

for z = 0, −1, −2, . . . .

We can keep on diﬀerentiating to obtain the various polygamma functions. Thus, ∞ 1 (6) ψ (n) (z) = (−1)n+1 n! k=0 (k+z) for z = 0, −1, −2, . . . . n+1 In exercises below we ask the reader to verify that the gamma function of (2) satisﬁes (the familiar integral expression of (11) and) the important recursion relationship (7)

for z = 0, −1, −2, . . . ,

Γ(z + 1) = z Γ(z)

with Γ(1) = 0 Γ(0) ≡ limz→0 z Γ(z) = 1. Diﬀerentiating the log of (7) thrice, we obtain the recursion relationships (8)

ψ(z+1) = ψ(z)+ z1 ,

ψ (z+1) = ψ (z)− z12 ,

and ψ (z+1) = ψ (z)+ z23 .

Equating (4) and (8) gives (9)

ψ(z + 1) = −γ + z

∞

1 k=1 k (k+z)

= −γ +

∞

k=1

1 k

−

1 k+z

for z = −1, −2, −3, . . . . From (9) and (5) we obtain (10)

ψ(1) = −γ = −0.5772156649 . . .

and ψ (1) =

π2 6

= 1.6449340668 . . . .

1.

THE GAMMA AND DIGAMMA FUNCTIONS

547

The familiar integral expression for the gamma function is ∞ (11) tz−1 e−t dt for Real(z) > 0. Γ(z) = 0

Thus, the digamma function ψ(·) trivially satisﬁes ∞ z−1 −t d 1 d (12) log Γ(z) = Γ (z)/Γ(z) = Γ(z) e dt ψ(z) ≡ dz dz 0 t (13)

=

1 Γ(z)

∞ 0

(log t) tz−1 e−t dt .

∞ 1 Thus, −γ = 0 (log t) e−t dt. Also, Γ(z) = 0 (− log t)z−1 dt. Repeat and then diﬀerentiate the formula Γ (z) = Γ(z) ψ(z) to obtain (14)

Γ (z) = Γ(z) ψ(z) and Γ (z) = Γ(z) ψ (z) + Γ (z) ψ(z) = Γ(z) {ψ (z) + [ψ(z)]2 } .

Continued diﬀerentiation will produce additional formulas. There is also an easy duplication formula (15)

Γ(2z) =

22z−1/2 Γ(z) Γ(z + 12 ) .

√1 2π

(Just apply √ (7) repeatedly to the left-hand side of (15), and then cancel terms. Use Γ( 12 ) = π, as will be shown to be true below.) Diﬀerentiate (15) twice to obtain (16)

ψ(2 z) = 12 [ψ(z)+ψ(z+ 21 )]+log 2

Now, plug z = (17)

1 2

ψ (2 z) = 14 [ψ (z)+ψ (z+ 12 )] .

and

into (16) to obtain (using (10))

ψ( 12 ) = −γ − log 4

ψ ( 12 ) = 3 ψ (1) =

and

π2 2

.

Applying (8) to ψ(·) over and over yields the useful evaluation formulas 1 (18) ψ(n + z) = ψ(1 + z) + n−1 for z = −1, −2, −3, . . . , k=1 k+z ψ(n) = ψ(1) +

n−1

= −γ +

1 k=1 k

ψ(n + 12 ) = ψ( 12 ) +

n−1

1 k=0 k+1/2

n−1

1 k=1 k

,

= ψ( 12 ) + 2

n−1

1 k=0 2k+1

.

Now, Γ(·), log Γ(·), ψ(·), ψ (·), ψ (·), and ψ (·) are all tabled in Abramowitz and Stegun (1972, p. 267) for 1 ≤ z ≤ 2; these can be used in conjunction with the formulas (18), (19), etc. From (5) we have ∞ 1 (19) ψ (z) = k=0 (k+z) for z = −1, −2, −3, . . . , 2 ψ (n) =

π2 6

−

ψ (n + 12 ) =

n−1

π2 2

1 k=1 k2

−4

,

n

∞ ∞ 1 using k=1 (2k−1) 2 = k=1 simple algebra).

1 k=1 (2k−1)2

1 k2

−

1 4

∞

,

1 k=1 k2

=

π2 6 (1

− 14 ) =

π2 8

(from (10) and

548

APPENDIX A. DISTRIBUTION SUMMARIES

Remark 1.1 We state the following asymptotic formulas from Abramowitz and Stegun. As |z| → ∞ with | arg z| < π, both (20)

ψ(z) ∼ log z −

(21)

ψ (z) ∼

1 z

+

1 2z

1 2z 2

−

+

1 12z2

1 6z 3

−

+

1 120z4

1 30z5

+

−

1 252z6

1 42z7

−

+

1 30z9

1 240z8

−···

and

+··· .

Those authors also include the integrated formulas (22)

ψ(z) = −γ +

∞ 0

e−t −e−zt 1−e−t

and ψ (z) =

∞ 0

te−zt (1−e−t )

dt = z

∞ 0

tz−1 (1−t)

dt .

All of the asymptotic and integrated formulas remain valid when diﬀerentiated repeatedly. 2 Let Γ(z) ≡

Remark 1.2 (Summary of values of some constants) d and ψ(z) ≡ dz log Γ(z) = Γ (z)/Γ(z). Then n

1 k=1 k

− log n] = −

∞

γ = 0.5772156649 . . . = lim [

(24)

Γ(1) = 1,

Γ( 12 ) =

(25)

ψ(1) = −γ,

ψ( 12 ) = −γ − log 4,

ψ(z + 1) = ψ(z) +

(26)

ψ (1) =

ψ ( 12 ) =

ψ (z + 1) = ψ (z) −

(27)

ψ (k) (z) = (−1)k+1 k!

(28)

π2 6 ,

 2  π /6 ∞ = −2 1  4 π /15 Function Γ(z) Γ (z) ψ(z) ψ (z) ψ (z) ψ (z)

1 j3

√

π,

tz−1 e−t dt

Γ(z + 1) = z Γ(z) ,

π2 2 ,

∞

0

0

(log t) e−t dt ,

(23)

n→∞

∞

1 j=0 (z+j)k+1

1 z

, 1 z2

,

for k ≥ 1 and z = 0, −1, −2, . . . .

= 1.6449340668.. . = −2.4041138063.. . = 6.4939394023.. .

$ if

k=1 k=2 k=3

and z = 1, and z = 1, and z = 1.

z = 1/2

z=1

z=2

1.7724538509 −3.4802309069 −1.9635100260 4.9348022005 −16.8287966441 97.4090910340

1.0000000000 −0.5772156649 −0.5772156649 1.6449340668 −2.4041138063 6.4939394023

1.0000000000 0.4227843351 0.4227843351 0.6449340668 −0.4041138063 0.4939394023

Exercise 1.1 Verify the values in the above table. (The last digit of ψ ( 12 ) is suspect, to within four units.)

1.

THE GAMMA AND DIGAMMA FUNCTIONS

549

Remark 1.3 (Digression on Z ∼ (0, 1) really having mean 0 and variance 1.) = N√ 2 Consider the density f (x) ≡ e−x /2 / 2 π. This is indeed a density, since ∞ 2π ∞ 2 2 2 ( −∞ 21π e−(x +y )/2 dx dy)2 = 0 0 21π e−r r dr dθ = 1 . √ Thus, Γ( 12 ) = π, since ∞ ∞ 2 1 = 2 0 √12 π e−x /2 dx = √22 π 0 e−t √12 t dt = √1π Γ(1/2) . Clearly, EN (0, 1) = 0, and E[N (0, 1)]2 =

√2 π

Γ( 32 ) =

√2 1 π 2

Γ( 12 ) = 1. 2

Remark 1.4 (Digression on Beta(r, s)) Now, 1 (29) B(r, s) ≡ 0 tr−1 (1 − t)s−1 dt = Γ(r) Γ(s)/Γ(r + s)

for all r, s > 0 .

This holds, since

∞∞ Γ(r) Γ(s) = 0 0 xr−1 e−x y s−1 e−y dx dy 1 ∞ uy r−1 s−1 −y/(1−u) y 1 = 0 0 ( 1−u ) y e 1−u 1−u dy du letting u ≡ x/(x + y)

=

with 1∞ 0

0

x = uy/(1 − u) and dx = (1 − u)−2 y du (uv)r−1 [(1 − u)v]s−1 e−v v dv du

letting v ≡ y/(1 − u) with y = (1 − u) dv and dy = (1 − u) dv ∞ r+s−1 −v 1 = 0 v e dv · 0 ur−1 (1 − u)s−1 du = Γ(r + s) B(r, s) . With t ≡ x/(1 + x) (with (1 − t) = 1/(1 + x) and dt = ∞ xr−1 (30) B(r, s) = 0 (1+x) 2 r+s dx Proof.

1 (1+x2 )

dx), obtain

Consider (2). Now, | log(1 + nz ) − nz | = | −

1 z2 2 n2

+

1 z3 3 n3

−···|

for |z| ≤ N/2 and n > N ≤ (a)

≤

|z|2 |z| n2 {1 + n 2

1N 2 n2

.

+

|z|2 n2

+ · · ·} ≤

1 N2 1 4 n2 {1 + 2

+

1 22

+ · · ·}

∞ Thus, for |z| ≤ N/2 the function SN (z) ≡ n=N +1 {log(1+ nz )− nz } is an absolutely and uniformly convergent series of analytic functions, and hence SN (·) is analytic on |z| ≤ N/2, too. Consequently, exp(SN (·)) is, too, and hence also exp(S1 (·)). And since N is arbitrary, S1 (·) is analytic for all complex z; hence zeγz S1 (z) also. Thus, expression (2) is valid. Consider (1). Letting ak ≡ k1 − log k+1 k , we have n 1 n n+1 γn ≡ k=1 n − log n = k=1 ak + log n n n+1 = k=1 ( k1 − log k+1 k ) + log n n 1 t (b) = k=1 0 k(k+t) dt + log n+1 n .

550

APPENDIX A. DISTRIBUTION SUMMARIES

So for n > N we have 1 γn − γN ≤ nk=N +1 0 k12 dt + log(1 + n1 ) − log(1 + n (c) = k=N +1 k12 + log(1 + n1 ) − log(1 + N1 ) ,

1 N)

showing that γ ≡ lim γn exists. Perhaps, the following ﬁgure 1.1 is better yet.

2

1 f(x) = x

1/1 1

Figure 1.1

Σkn = 1 k1

1/2 2

1/3 3

1/4 4

1/5 5

6

– log n = (shaded area) → γ = .57721566. . . .

(a) Derive Euler’s formula that for z = −1, −2, . . . , ∞ z Γ(z) = n=1 {(1 + 1/n)z /(1 + z/n)} .

Exercise 1.2 (31)

Begin with the expression 1 lim e(1 + Γ(z) = z

1 2

+··· +

1 m −log m)z

m→∞

lim

m

m→∞

n=1 {(1

+ nz ) e−z/n } .

(b) Use this last expression for 1/z Γ(z) and use (2) for Γ(z + 1), and then show that the ratio converges to 1, and so obtain (7). Exercise 1.3 (Bounds on (1 − x/n)n ) Note that (32)

(1 + y) ≤ ey ≤ 1/(1 − y)

for 0 ≤ y ≤ 1 ,

which yields (33)

1 ≥ ey (1 − y) ≥ (1 − y 2 )

for 0 ≤ y ≤ 1.

Use this (with y = t/n) to show that (34)

0 ≤ e−x − (1 − nx )n ≤

x2 −x n e

for all 0 ≤ x < n.

Exercise 1.4 Establish (11). [Hint. Start by considering ∞ n | 0 e−t tz−1 dt − 0 (1 − nt )n tz−1 dt | . Evaluate the second integral repeatedly using integration by parts, and then apply n formula (31) to establish that 0 (1 − nt )n tz−1 dt → Γ(z) for Re(z) > 0.] Exercise 1.5

Verify the term-by-term diﬀerentiation leading to (4) and (5).

2.

MAXIMUM LIKELIHOOD ESTIMATORS AND MOMENTS

551

2

Maximum Likelihood Estimators and Moments

Motivation 2.1 (Maximum likelihood estimator, MLE) Suppose the outcome X of some experiment is modeled as a random element on the probability space (X , A, Pθ ), where the particular parameter value θ is an unknown point in the ﬁxed set Θ of all possible parameter values. We suppose that each Pθ µ with ≡ (X1 , . . . , Xn ) , density pθ (x) for x ∈ X , for some ﬁxed σ-ﬁnite measure µ. Let X where the Xk ’s denote n independent repetitions of the basic X-experiment. Let us agree to call the joint density function n Lx (θ) ≡ pθ (x ) = 1 pθ (xk ) the observed likelihood function of the observed data value x when we think of the joint density expression as a function of the parameter value θ while holding the experimental outcome x ﬁxed. Then n n (1) θ ≡ (θ) ≡ (θ|x ) ≡ log pθ (x ) = 1 log pθ (xk ) ≡ 1 (θ|xk ) will be called the observed log likelihood function. The maximum likelihood estimate ˆ x ) of the parameter that maximizes the observed likelihood of θ is that value θˆn ≡ θ( This highly intuitive procedure for for the data value x of the experiment X. generating an estimate of the unknown parameter is due to R.A. Fisher. Meanwhile ) is called the theoretical log likelihood, and a formula/algorithm for the θ ≡ (θ|X ˆX ) that maximizes this expression produces the maximum likelihood value θˆn ≡ θ( estimator. Since the value that maximizes the likelihood is the same value that maximizes the log likelihood, and since sums are typically easier to maximize than products, it is usual to think of the MLE θˆn of θ as that value of the parameter that maximizes the log likelihood θ . It is often possible to determine the MLE θˆn by setting the derivative of the log likelihood with respect to θ equal to zero and solving the resulting likelihood equation for an expression for the MLE θˆn . Suppose now that the parameter of the model is actually an m dimensional ) is vector θ ≡ (θ1 , . . . , θm ) . Associated with the theoretical log likelihood (θ|X ˙ ˙ X ) whose rth coordinate is given by the m × 1 score vector ˙θ ≡ (θ) ≡ (θ| n ˙ ∂ ≡ ∂ log pθ (X ) = n (2) ˙r (θ) ≡ ˙r (θ|X) k=1 ∂θr log pθ (Xk ) = k=1 r (θ|Xk ) . ∂θr ˆX ) of the vector θ is now typically determined The formula for the MLE θˆn ≡ θ( by stating that the likelihood equation (it’s an m × 1 vector equation) (3)

0 = ˙θ

has as its solution the MLE θˆn .

˙ X ) = 0. Suppose our model is “suﬃciently regular” for the following Usually, Eθ (θ| to exist. Deﬁne the m × m Fisher information matrix for one observation X as (4)

T ˙ ˙ Iθ ≡ Eθ {(θ|X) (θ|X) } = |[ Eθ {˙r (θ|X) ˙s (θ|X)} ]| = |[ Eθ {−¨rs (θ|X)} ]|

where ¨rs (θ|X) ≡

∂ ∂ ∂θr ∂θs

(θ|X) (and where the last equality is typically true). The ˆX ) (with its corresponding numerical fundamental result about the MLE θˆn ≡ θ( ˆ x )) is that in a “suﬃciently regular” model, θˆn satisﬁes data values θˆn = θ( √ (5) n [θˆn − θ] →d N (0, Iθ−1 ) . 2

552

APPENDIX A. DISTRIBUTION SUMMARIES

Location and Scale Models Suppose we model an experimental outcome Y via a location-scale model as (6)

Y =ν+τ W

where W has a ﬁxed density f (·) on the line, with −∞ < ν < ∞ and τ > 0. Let (7)

g(·) ≡ − log f (·)

g (·) = −f (·)/f (·) .

with

Let wp ≡ F −1 (p) denote the pth quantile of W , so that Y has pth quantile (8)

yp ≡ ν + τ wp

for each 0 < p < 1.

It is assumed throughout this section that the df F (·) satisﬁes suﬃcient regularity conditions for everything that follows to make sense. We will loosely refer to these cases as regular models. Let θ ≡ (ν, τ ) and Θ ≡ R × R+ . Then one observation Y has (9)

y−ν y−ν θ = log pθ (y) = log τ1 f ( y−ν τ ) = − log τ + log f ( τ ) = − log τ − g( τ ) .

The ﬁrst partials (that is, the coordinates of the score vector) are given by (10)

y−ν ˙τ = − τ1 + y−ν τ2 g ( τ ) = = ˙ν = τ1 g ( y−ν τ )

1 τ 1 τ

[w g (w) − 1]

=

[g (w)]

=

1 τ 1 τ

[−1 − w f (w)/f (w)] , [−f (w)/f (w)] .

The two quantities in square brackets in (10) have 0 means, both here and in (12) below; they are called the optimal score functions for scale and for location. The second partials are given by 2

(11)

(y−ν) y−ν −¨τ τ = − τ12 + 2 y−ν g ( y−ν τ3 g ( τ ) + τ4 τ ), 1 y−ν ¨ −νν = τ 2 g ( τ ) , y−ν −¨τ ν = 12 g ( y−ν ) + y−ν ), 3 g ( τ

τ

τ

τ

or (in the more revealingly w-format) are given by

(12)

−¨τ τ = −¨νν = −¨τ ν =

1 τ2 1 τ2 1 τ2

{w2 g (w) + 2[w g (w) − 1] + 1} , {g (w)} , {w g (w) + [g (w)]} .

˙ X) = 0) that We observe from (10) (and also from Eθ (θ; (13)

E {g (W )} = 0

and

E {W g (W )} = 1 .

Then the Fisher information matrix is given from (12) and (13) by 1 −Eθ ¨τ τ −Eθ ¨τ ν E{W 2 g (W ) + 1} EW g (W )} (14) Iτ ν ≡ Iθ = = −Eθ ¨τ ν −Eθ ¨νν E{W g (W )} E{g (W )} τ2 ! " f (W ) 2 f (W ) 2 1 E{[−1 − W f (W ) ] } E{W [ f (W ) ] } 1 1 c00 c01 (15) = 2 ≡ C ≡ . τ τ2 τ 2 c01 c11 E{W [ f (W ) ]2 } E{[ f (W ) ]2 } f (W )

f (W )

Note that E{W g (W )} = 0 when W is symmetrically distributed. Moreover, the matrix C does not depend on θ.

2.

MAXIMUM LIKELIHOOD ESTIMATORS AND MOMENTS

553

Suppose now that an iid sample Y1 , . . . , Yn is observed. Then the MLEs τˆ ≡ τˆn and νˆ ≡ νˆn are typically obtained by jointly solving 0 = ˙τ = nτ { n1 ni=1 Wi g (Wi ) − 1} = nτ { W g (W ) − 1} , n (16) 0 = ˙ν = n { 1 g (Wi ) − 1} = n { g (W ) − 1} . τ

n

i=1

τ

That is, the solutions τˆ and νˆ are such that when the Wi themselves are replaced by the residuals Wi ≡ (Yi − νˆ)/ˆ τ in these likelihood equations in (16), then these expressions do indeed yield the intended value of 0. We estimate I(τ, ν) by the observed information matrix Iˆn which is now deﬁned by ! " ˆ 2 g (W ˆ g (W ˆ )+1 W ˆ) 1 W (17) Iˆn ≡ 2 ˆ g (W ˆ) ˆ) τˆ W g (W 1 cˆ00 cˆ01 1 (18) ≡ 2 ≡ 2 Cˆn . cˆ01 cˆ11 τˆ τˆ The fundamental theorem about the solutions of the likelihood equations τˆn and νˆn 1/2 in location-scale models is that under suﬃcient regularity (recall (9.3.9) for Cˆn ) √ n (ˆ τn − τ )/ˆ τn 1/2 ˆ √ (19) Cn →d N (0, I2 ) n (ˆ νn − ν)/ˆ τn (for the 2 × 2 identity I2 ), and also √ τn − τ )/ˆ τn √ n (ˆ (20) →d N (0, C −1 ) . n (ˆ νn − ν)/ˆ τn ˆ n ≡ |[dˆij ]| ≡ Cˆ −1 . We note the conventional We deﬁne D ≡ |[dij ]| ≡ C −1 and D n wisdom that the approximation in (19) is typically somewhat more accurate than the approximation in (20). Note also that the solutions τˆ and νˆ of the likelihood equations are invariant, in that for all a > 0 and all real b we have (21)

+ b 1 ) = a νˆ(Y ) τˆ(aY

and

+ b 1 ) = a νˆ(Y ) +b. νˆ(aY

In case the true density of W is modeled to belong to some family of densities fγ (·) indexed by the parameter γ, then we need to expand the likelihood equations to include the appropriate terms. Thus, we let (22)

˙γ = −g( ˙ y−ν τ ),

ˆ ) = 0 at the MLEs. and note that g( ˙ W

The MLEs will still be invariant, now satisfying (for a > 0, b real, and all γ) (23)

+ b 1 ) = γˆ (Y ) , τˆ(aY + b 1 ) = a τˆ(Y ) , νˆ(aY + b 1 ) = a νˆ(Y ) +b. γˆ (aY

The Fisher information matrix now becomes  − τ1 E{W g˙ (W )} E{¨ g(W )} 1 (24) Iγ,τ,ν ≡  − τ E{W g˙ (W )} τ12 E{W 2 g (W ) + 1} 1 − τ1 E{g˙ (W )} τ 2 E{W g (W )}

 − τ1 E{g˙ (W )} 1 . τ 2 E{W g (W )} 1 E{g (W )} τ2

(Note how τ appears in this matrix. The expectations do not depend on τ or ν.)

554

APPENDIX A. DISTRIBUTION SUMMARIES

Moments Deﬁnition 2.1

For a rv W , let µ ≡ EW and σ2 ≡ Var[W ]. Then let 1/2

(25)

µ3 ≡ E(W − µ)3

and

γ1 ≡ β1

(26)

µ4 ≡ E(W − µ)4 ,

β2 ≡ µ4 /σ 4 ,

and

≡ µ3 /σ 3 ,

γ2 ≡ β2 − 3 .

Recall that for a rv W we agreed that φ(t) ≡ EeitW denotes the characteristic function (or chf) and that M (t) ≡ EetW denotes the moment generating function (or mgf). We also agreed that ψ(t) ≡ Log φ(t) denotes the cumulant generating function (or cgf). The statement of exercise 13.4.6 contains all the useful facts about this cgf. However, we will also call C(t) ≡ log M (t) the cgf (for those values t for which it is well deﬁned). The so called method of moments estimator (or MME) θ¯ ≡ θ¯n of θ is computed ¯ n and also ¯ n for θ¯n (or by jointly solving µ(θ) = X by solving the equation µ(θ) = X 2 µ2 (θ) = Xn for θ ≡ (θ1 , θ2 ) in the two dimensional case), etc.

3.

EXAMPLES OF STATISTICAL MODELS

555

3

Examples of Statistical Models

Discrete Distributions Exercise 3.1 (Binomial) (1)

µ = np,

(2)

γ1 =

Show that for the Binomial(n, p) distribution, µ3 = npq(q − p),

σ2 = npq,

q−p √ npq

and

γ2 =

1−6pq npq

µ4 = 3(npq)2 + npq(1 − 6pq) ,

.

This distribution has mgf M (t) = [1 + p(et − 1)]n for all t. Exercise 3.2 (Poisson) (3)

Show that the cgf of the Poisson(λ) distribution is ∞ C(t) = λ(et − 1) = 0 λtk /k! ,

and so the mth cumulant κm equals λ for all m. Show that (4)

µ = σ2 = µ3 = λ,

µ4 = λ(1 + 3λ),

Exercise 3.3 (Negative binomial)

γ1 =

√1 , λ

γ2 =

1 λ

.

Show that the NegBiT(r, p) has

(5)

µ = r/p, σ2 = rq/p2 , µ3 = rq(1+q)/p3 , µ4 = [3(rq)2 +rq(1+4q +q 2 )]/p4 ,

(6)

γ1 =

1+q √ rq

and

γ2 =

1+4q+q2 rq

.

The only change for NegBiF(r, p) is that the mean is now rq/p = r/p − 1. Exercise 3.4 (Discrete uniform distribution of ranks) Show that the Discrete Uniform(1, N ) distribution that takes on each value 1, . . . , N with probability 1/N has moments given by N +1 2 ,

(7)

µ=

(8)

γ1 = 0,

− 1),

σ2 =

2 1 12 (N

γ2 =

3 3N 2 −7 5 N 2 −1 ,

µ4 =

2 1 240 (3N

− 7)(N 2 − 1),

µ6 =

4 1 112 (3N

− 18N 2 + 31)(N 2 − 1) .

All odd central moments equal 0. We state for convenient reference that: n 2 1 n 1 1 k = 2 n(n + 1) , 1 k = 6 n(n + 1)(2n + 1) , (9) n 3 1 2 n 4 1 2 2 1 k = 4 n (n + 1) , 1 k = 30 n(n + 1)(2n + 1)(3n + 3n − 1) . Exercise 3.5 (Hypergeometric) The Hypergeometric(pN, qN ; n) distribution has: µ = np, γ1 =

q−p √ npq

σ2 = npq [1 − ×

1−

n−1 N −1

(N −1)(N +6) β2 = 3 (N −2)(N −3) +

n−1 N −1

×

1 N pq

],

N −2n N −2

µ3 = npq(q − p) NN−2n −2 , ,

[1 − 6 NN+1 (pq +

n(N −n) )] . N2

556

APPENDIX A. DISTRIBUTION SUMMARIES

Classical Normal Theory Distributions Exercise 3.6 (Normal) Let Z ∼ = N (0, 1). Then γ1 = 0 and γ2 = 0. Thus, γ1 and γ2 take the standardized values 0 and 0 for any normal distribution. Also, √ (10) E|Z|r = 2r/2 Γ( r+1 for all r > 0. 2 )/ π , Thus, E|Z| = 2/π and EZ 4 = 3. The mgf of N(µ, σ2 ) is M (t) = exp(µt + σ2 t2 /2). Exercise 3.7 (Gamma) Verify what follows. (a) If Y ∼ = Gamma(r, 1), then √ (11) µ = r, σ2 = r, γ1 = 2/ r , γ2 = 6/r . Also, E(1/Y ) = 1/(r − 1) for r > 1. The mode is located at (α − 1)β. (b) The Gamma(α, β) distribution has cgf C(t) = −α log(1 − βt) for |t| < 1/β, and so the mth cumulant is given by κm = (m − 1)! αβ m . Thus √ (12) µ = αβ, σ2 = αβ 2 , γ1 = 2/ α, γ2 = 6/α , (13)

µ2 = (α + α2 )β 2 ,

µ3 = 2αβ 3 ,

µ4 = (3α2 + 6α)β 4 .

Let θ ≡ (α, β) . The log likelihood θ ≡ log pθ (x) and its partial derivatives are log pθ (x) = − log Γ(α) − α log β + (α − 1) log x − x/β , (14)

˙α = −ψ(α) − log β + log x and ˙β = −¨αα = ψ (α) ,

−¨ββ =

1 β3

α β

+

[2x − αβ] ,

x β2

=

1 β2

[x − βα] ,

−¨αβ =

1 β

.

˙ X) = 0), we learn that From this (and Eθ (θ; (15) (16)

Eθ (log X) = ψ(α) + log β and Eθ X = αβ , ψ (α) β1 1 α −β −1 and Iθ = Iα,β = Iθ = . α 1 α ψ (α) − 1 −β β 2 ψ (α) β β2

Two other natural parameterizations of the Gamma involve the choice α, ν ≡ 1/β or the choice α, µ ≡ αβ. These yield α ψ (α)−1 0 ψ (α) − ν1 α (17) Iα,ν = or I = . α,µ α α − ν1 0 µ2 ν2 Suppose tht V ∼ = χ2α /χ2β , where α, β > 0. Verify that

Exercise 3.8

β−2r α β EV r = B( α+2r 2 , 2 )/B( 2 , 2 )

whenever α > 0 and β > 2r with r > 0. Verify that r

V α+2r β−2r+2s E{ (1+V )/B( α2 , β2 ) )s } = B( 2 , 2

whenever α > 0, β > 0, α + 2r > 0 and β − 2r + 2s > 0.

3.

EXAMPLES OF STATISTICAL MODELS

557

Exercise 3.9 (Student-t) Verify what follows. (a) If T ∼ = Student-tβ , then √ β Γ((β − 1)/2) √ (18) E|T | = for β > 1 , π Γ(β/2) β β−2

(19)

Var[T ] =

for β > 2 ,

(20)

γ1 = 0 for β > 3

and

γ2 =

6 β−4

for β > 4 .

(b) Suppose that pθ (x) = fβ ((x−θ1 )/θ2 ), where fβ (·) denote the Student-tβ density subject to θ2 > 0 with θ1 real. Verify that: (21) (22) (23)

−

f (x) (β + 1)x = f (x) β + x2

∞ −∞

∞

−∞ {−[1 +

Iθβ =

xf (x)/f (x)]}2 dx = 2β/(β + 3) ,

2β 0 β+3 β+1 0 β+3

=

iscale 0

n n−2

σ2 =

γ2 =

8(n − 4) m(m + n − 2)

γ1 =

.

for n > 4 ,

1/2

2m + n − 2 n−6

for n > 6 ,

12{(n − 2)2 (n − 4) + m(m + n − 2)(5n − 22)} m(n − 6)(n − 8)(m + n − 2)}

r r+s 2(s − r) r+s+2

β2 = 3

for n > 8 .

(a) If X ∼ = Beta(r, s) with r, s > 0, then

Exercise 3.11 (Beta) µ=

ilocation

for n > 2 ,

2 n2 (m + n − 2) m(n − 2)2 (n − 4)

γ1 =

0

If X ∼ = Snedecor-Fm,n , then

Exercise 3.10 (Snedecor-F ) µ=

xf (x) β(x2 − 1) = , f (x) x2 + β

{−f (x)/f (x)}2 dx = (β + 1)/(β + 3) ,

(24)

−1−

and

and

σ2 =

r+s+1 rs

1 r s , r+s+1 r+s r+s

1/2 ,

r + s + 1 (r + s)rs + 2(r2 − rs + s2 ) . r+s+3 (r + s + 2)rs

558

APPENDIX A. DISTRIBUTION SUMMARIES

(b) Let θ ≡ (α, β) . The log likelihood θ ≡ log pθ (x) and its partial derivatives are log pθ (x) = log Γ(α + β) − log Γ(α) − log Γ(β) + (α − 1) log x + (β − 1) log(1 − x) , (25)

˙α = −ψ(α + β) − ψ(α) + log x and ˙β = −ψ(α + β) − ψ(β) + log(1 − x) , −¨αα = ψ (α + β) + ψ (α), −¨ββ = ψ (α + β) + ψ (β), −¨αβ = ψ (α + β).

˙ X) = 0) we learn that From this (and Eθ (θ; (26) (27)

Eθ (log X) = ψ(α)+ψ(α+β) and Eθ (log(1−X)) = ψ(β)+ψ(α+β) , ψ (α) + ψ (α + β) ψ (α + β) . I(α,β) ≡ Iθ = ψ (α + β) ψ (β) + ψ (α + β)

Other Distributions Exercise 3.12 (Double Exponential) (a) Show that the Double Exponential(0, 1) distribution with density f (w) = 12 e−|w| on the line has mean 0, variance 2, skewness γ1 = 0, tail heaviness γ2 = 3, β2 ≡ µ4 /σ 4 = 4!/22 = 6, and 4th cumulant κ4 ≡ µ4 − 3σ 4 = 12. Note that all odd moments of W equal 0 and that |W | ∼ = Exponential(1). (b) Consider a location-scale model Y = ν+τ W with W ∼ = Double Exponential(0, 1). Determine the MLEs τˆ and νˆ. Show next that log f (w) = −|w| − log 2, with score functions −f (w)/f (w) = sign(w) for w = 0, and −[1 + wf (w)/f (w)] = |w| − 1 for w = 0. Show that the Fisher information matrix equals τ12 I2 . Exercise 3.13 (Logistic) Verify the following facts. (a) The Logistic(0, 1) distribution with density (28)

f (w) = e−w /(1 + e−w ) = 1/(e−w/2 + ew/2 )2

on (−∞, ∞)

has mean 0, variance π 2 /3, skewness γ1 = 0, tail heaviness γ2 = 1.2 and 4th cumulant κ4 = 2π 4 /15. All of its odd moments equal 0. In fact, the moment generating function M (·) and the cumulant generating function C(·) are given by (29)

M (t) = Γ(1 − t) Γ(1 + t)

and

C(t) = log Γ(1 − t) + log Γ(1 + t)

for |t| < 1. Thus, the mth cumulant equals 0 for m odd and it equals 2ψ(m−1) (1) for m even. (b) Consider a location-scale model Y = ν + τ W with W ∼ = Logistic(0, 1). Show how to determine the MLEs τˆ and νˆ. Verify that −f (w)/f (w) = 2F (w) − 1. Then show that the Fisher information matrix is diagonal, where c11 = 1/3 is easy and where c00 = π12 + 13 requires substantial eﬀort. Exercise 3.14 (Uniform) Show that the Uniform(−1, 1) distribution has mean 0, variance 13 , γ1 = 0 and γ2 = −1.2.

3.

EXAMPLES OF STATISTICAL MODELS

559

Exercise 3.15 (Log Gamma(α) distribution) Suppose X ∼ = Gamma(α, 1) with density fX (x) = xα−1 e−x /Γ(α) for x ≥ 0, for some α > 0. Then W ≡ log X has the Log Gamma distribution with density function (30)

1 Γ(α)

f (w) =

exp(α y − ey )

on (−∞, ∞).

Verify the following facts. The mode is located at log α. (a) The cumulant generating function of W is (31)

C(t) = log{Γ(t + α)/Γ(α)} = log M (t)

for |t| < α .

Thus, the mth cumulant of W is given by dm (32) = ψ (m−1) (α) κm = dt m C(t) t=0

for the digamma function ψ(·). From this one obtains (33)

σ2 = ψ (α)

µ = ψ(α),

γ2 = ψ (α)/[ψ (α)]2 .

3/2

γ1 = ψ (α)/[ψ (α)]

(b) Recalling (14), we obtain E{eY } = M (t) t=1 d Y E{Y e } = dt M (t) t=1 (34) d2 2 Y E{Y e } = dt2 M (t)

= Γ(α + 1)/Γ(α) = α , = Γ (α + 1)/Γ(α) = α ψ(α + 1) , = Γ (α + 1)/Γ(α)

t=1

= α{ψ (α + 1) + [ψ(α + 1)]2 } . Then note that (35)

g(w) ≡ log Γ(α) − α w + ew ,

g (w) = −α + ew

and g (w) = ew .

Thus the location-scale model Y = ν + τ W has Fisher information given by (A.1.14), which simpliﬁes to E{Y 2 eY + 1} E{Y eY } 1 (36) ≡ τ12 C (α) Iτ,ν ≡ τ 2 E{Y eY } E{eY } (37)

=

1 τ2

1 + α{ψ (α + 1) + [ψ(α + 1)]2 } α ψ(α + 1)

α ψ(α + 1) α

.

Specialize equations (A.2.16) and (A.2.17) to the present model. Exercise 3.16 (Extreme Value) The so called Extreme Value(0, 1) distribution for minimums corresponds to the special case of the previous exercise with α = 1. Thus, W ≡ log X has the Extreme Value distribtuion with density (38)

f (w) = exp(y − ey )

on (−∞, ∞).

560

APPENDIX A. DISTRIBUTION SUMMARIES

Verify the following facts. (a) The cumulant generating function is (39)

C(t) = log Γ(t + 1) = log M (t)

for |t| < 1 .

From (32) one obtains µ = ψ(1) = −γ (40)

2 = π6 3/2

2

σ = ψ (1) γ1 = ψ (1)/σ

γ2 = ψ (1)/σ4

= −0.5772157 . . . = 1.644934 . . . ∞ = −2 1 i13 /(π2 /6)3/2 = −1.139547 . . . =

π 4 /15 (π 2 /6)2

= 2.4000 . . .

∞ using 1 i−3 = 1.2020569032 . . . . This distribution trivially has its mode at 0. The df equation p = F (y) = 1 − exp(−ey ) then yields a pth quantile given by (41)

1 1 yp = log2 ( 1−p ) = log(log 1−p )

for 0 < p < 1.

Note that two particularly useful values are y.5 = log2 2 = −0.3665129 . . . and yp = 0 for p = 1 − 1/e = .3621206 . . . . (b) Recalling (34), we obtain E{eY } = M (t) = Γ(2) = 1.0000000 , t=1 d E{Y eY } = dt M (t) = Γ (2) = 1 ψ(21) = 1 − γ = 0.422784 . . . , (42) t=1 d2 E{Y 2 eY } = dt = Γ (2) = {ψ (2) + [ψ(2)]2 } = 0.823681 . . . . 2 M (t) t=1

Then note that (43)

g(w) ≡ − log f (w) = −w + ew ,

g (w) = −1 + ew ,

and g (w) = ew .

Thus, the location-scale model Y = ν + τ W has Fisher information given by (37), and this simpliﬁes to 2 π 1−γ + (1 − γ)2 (44) Iτ,ν = τ12 6 ≡ τ12 C (α) 1 1−γ (45)

=

1.823681 . . . 0.422784 . . .

1 τ2

with (46)

−1 Iτ,ν = τ2

(47)

= τ2

6 π2

1 −(1 − γ)

0.607927 . . . −.473999 . . .

0.422784 . . . 1.000000 . . .

,

−(1 − γ) π2 2 6 + (1 − γ) −.473999 . . . 0.977502 . . .

Specialize equations (16) and (17) to the present model.

3.

EXAMPLES OF STATISTICAL MODELS

Exercise 3.17 (Weibull) (48)

By X ∼ = Weibull(α, β) we mean that

β xβ−1 exp(−(x/α)β ) αβ

fX (x) =

561

for x ≥ 0 .

This distribution is often used to model lifetimes. The survival function or reliability function R(x) ≡ P (X > x) is given by (49)

R(x) = 1 − F (x) = P (X > x) = exp(−(x/α)β )

for x ≥ 0 .

Verify the following claims. (a) It is easy to compute that (50)

µ = α Γ(1 + 1/β)

σ2 = α2 {Γ(1 + 2/β) − Γ2 (1 + 1/β)} .

and

Note that (51)

V ≡ (X/α)β ∼ = Exponential(1) and W ≡ log V ∼ = Extreme Value(0, 1).

It thus holds that Y ≡ log X satisﬁes the location-scale model (52)

Y =ν+τ W

with ν ≡ log α and τ = 1/β ,

where W has the Extreme Value(0, 1) density f (w) = exp(w − ew ) on the line. Note 1 }1/β of X and yp of Y satisfy that the quantiles xp = α{log 1−p (53)

1 β(yp − log α) = β(xp − log α) = log2 ( 1−p ) = log(log

1 1−p )

for 0 < p < 1.

n ) “should” be a straight Thus a plot of the ordered values log Xn:i vs log2 ( n−i+1/2 line of slope 1/β and horizontal intercept log α at pα ≡ 1 − 1/e = 0.6321206 . . . . (b) Show that the log likelihood for one observation satisﬁes

(54)

˙α =

β α

[V − 1]

and

˙β =

1 α

[1 + V − V log V ] ,

and also satisﬁes (55)

−¨αα = −¨ββ = −¨αβ =

β α2 {(β + 1)V − 1} , 2 1 β 2 {1 + V log V } , 1 α {1 − V − V log V } .

Thus, the Fisher information is given by β 2 /α2 −(1 − γ)/α Iαβ = −(1 − γ)/α [(π2 /6) + (1 − γ)2 ]/β 2 (56) β 2 /α2 −0.4227843 . . . /α = −0.4227843 . . . /α 1.823681 . . . /β 2

.

The determinant of Iαβ equals (π2 /6)/α2 = 1.644934 . . . /α2 . Note also that 0.607927 . . . β 2 −0.257022 . . . α −1 (57) Iαβ = . −0.257022 . . . α 1.108665 . . . α2 /β 2

562

APPENDIX A. DISTRIBUTION SUMMARIES

Then note that (58)

E{V log V } = ψ(2) = 1 − γ , E{V log2 V } = ψ (2) + [ψ(2)]2 = −1 + (π 2 /6) + (1 − γ)2 .

(c) Show that the MLEs α ˆ and βˆ satisfy both (59)

α ˆ=

X βˆ

1/βˆ and

ˆ , 0 = H(β)

where (60)

H(β) ≡

X β log X Xβ

− log X −

1 β

.

Use Cauchy-Schwarz to verify that H (β) > 0 for all β > 0. Show that the NewtonRaphson iteration (61)

βˆm = βˆm−1 − H(βˆm−1 )/H (βˆm−1 )

for m = 1, 2, . . .

will necessarily converge quickly to a solution, starting from an initial value of n (62) βˆ0 ≡ (π2 /6)/ n1 1 (log Xi − log X )2 . Develop a formula for the observed information Iˆn . Exercise 3.18 (Pareto) Develop results similar to exercise 3.15(a) for the Pareto density f (w) = β w−(β+1) on w > 1, for β > 0.

4.

ASYMPTOTICS OF MAXIMUM LIKELIHOOD ESTIMATION

563

4

Asymptotics of Maximum Likelihood Estimation

The Model (M0) We suppose that the basic experiment X under consideration is modeled as having distribution Pθ for some value of θ in the set Θ ⊂ Rm . The distributions Pθ are known, but the true value of θ is not. We thus let (X , A, Pθ ) denote a probability space, for each θ ∈ Θ ⊂ Rm , and we suppose that the true distribution of the experimental outcomes X is one of the members Pθ of this family. Suppose also that each Pθ (·) has a density pθ (·) with respect to some ﬁxed σ-ﬁnite measure µ on the measurable space (X , A). These Pθ ’s are assumed to be identiﬁable (that is, θ = θ implies Pθ = Pθ ). We now suppose that X1 , . . . , Xn are ≡ (X1 , . . . , Xn ) . iid repetitions of the basic X-experiment distributed as Pθ . Let X Frequently, it will be convenient to let θ0 denote the true value of the parameter θ. Notation 4.1 We agree that (for one repetition X of the basic experiment) ≡ n L(θ|Xk ), (1) L(θ|x) ≡ Lx (θ) ≡ pθ (x), L(θ) ≡ L(θ|X) k=1 n ≡ (2) (θ|X) ≡ log pθ (X), (θ) ≡ (θ|X) k=1 (θ|Xk ), (3)

(B|X) ≡ supθ∈B (θ|X),

≡ supθ∈B (θ|X), (B) ≡ (B|X)

(4)

˙r (θ|X) ≡

˙ ˙ X) ≡ ˙r (θ|X) m×1 , (θ) ≡ (θ|

(5)

¨rs (θ|X) ≡

∂ ∂θr

(θ|X),

∂ ∂ ∂θr ∂θs

¨ ≡ (θ| ¨ X) ≡ ¨rs (θ|X) m×m . (θ)

(θ|X),

We further agree that (6)

˙ (θ|X) ≡ (˙1 (θ|X), . . . , ˙m (θ|X)) ≡ (the score vector) ,

(7)

I(θ) ≡ Eθ {˙r (θ|X) ˙s (θ|X) } ≡ (Fisher’s information matrix),

(8)

¨ ¯ I(θ) ≡ Eθ {−¨rs (θ|X)} ≡ E{−(θ|X)},

(9)

= L(Θ|X). θˆn ≡ θˆn (X) is a MLE if L(θˆn |X)

(The MLE θˆn may not exist, may not be unique, or may be diﬃcult to determine.) The likelihood ratio statistic (LR-statistic) for testing H0 : θ = θ0 vs H1 : θ = θ0 is deﬁned as a comparison of the likelihoods via (10)

≡ L(Θ)/L(θ0 ); Λn ≡ Λn (X)

and thus

2 log Λn = 2[(Θ) − (θ0 )]. 2

The natural approach to ﬁnding the MLE θˆn is to solve the likelihood equations (11)

0 = ˙r (θ|X)

for 1 ≤ r ≤ m.

If we are unsure the solution determines a maximum of the likelihood, we will denote it by θ˜n and call it a solution of the likelihood equations. Let θ0 be true. Let √ ˙ 0 |X) (12) Zn ≡ Zn (θ0 ) ≡ √1n (θ and Dn ≡ n(θ˜n − θ0 ), (13)

Iˆn (θ) ≡ −

1 n

m×m ¨rs (θ|X)

and

˜ n ≡ L(θ˜n )/L(θ0 ). Λ

Call Iˆn (θ˜n ) the observed information matrix. Typically, Zn →d Z ∼ = N (0, I(θ0 )).

564

APPENDIX A. DISTRIBUTION SUMMARIES

Basic Assumptions (M1) The parameter set Θ in Rm contains an open neighborhood of θ0 in which: (i) For a.e. x the log likelihood (θ|x) is twice continuously diﬀerentiable in θ. ... (ii) For a.e. x the third order partials rsγ are continuous in θ and satisfy ...

| rsγ (θ|x)| ≤ (someM (x)) having Eθ0 M (X) < ∞ (for all 1 ≤ r, s, γ ≤ m). (M2) (i) Eθ0 {˙r (θ0 |X)} = 0 for each 1 ≤ r ≤ m. (ii) Eθ0 {[˙r (θ0 |X)]2 } < ∞ for each 1 ≤ r ≤ m. ¯ 0 ) ≡ − Eθ0 {¨rs (θ0 |X)} > 0 (iii) I(θ0 ) ≡ Eθ0 {˙r (θ0 |X) ˙s (θ0 |X)} = I(θ ¯ 0 ) and I(θ0 ) is positive deﬁnite). (that is, both I(θ0 ) = I(θ (M3) (i) For a.e. ω and for all n ≥ (some nω ), a solution θ˜n of the likelihood equations (11) exists having θ˜n →a.s. θ0

(with respect to Pθ0 );

that is, a strongly consistent solution θ˜n of (11) exists. (ii) A solution θ˜n of (11) exists for which θ˜n →p θ0

(with respect to Pθ0 );

that is, a weakly consistent solution θ˜n of (11) exists. (M4) The support {x : pθ (x) > 0} does not depend on θ. ˜ n) Theorem 4.1 (Asymptotic distributions of θ˜n and Λ (A) Suppose that conditions (M0), (M1)(i)(ii) and (M2)(i)(ii)(iii) hold, and that a strongly consistent estimator θ˜n exists as in (M3)(i). Suppose θ0 is true. Then both Iˆn (θ˜n ) ≡ − n1 nk=1 ¨rs (θ˜n |Xk ) =a I(θ0 ), and (14) ˜ I(θn ) =a I(θ0 ) if I(·) is assumed continuous at θ0 ; the same is true for the inverses, since · −1 is a continuous function. Also, √ Dn ≡ n (θ˜n − θ0 ) =a I −1 (θ0 ) Zn →d D ≡ I −1 (θ0 ) Z ∼ = N (0, I −1 (θ0 )) (15) with Z ∼ = N (0, I(θ0 )), √ √ Wn ≡ [ n (θ˜n − θ0 )] Iˆn (θ˜n ) [ n(θ˜n − θ0 )] =a Dn I(θ0 ) Dn (16) =a Rn ≡ Zn I −1 (θ0 ) Zn →d Z I −1 (θ0 ) Z ∼ = χ2m , (17) (18)

˜ ˜ n = 2 log L(θn ) =a Dn I(θ0 ) Dn →d Z I −1 (θ0 ) Z ∼ 2 log Λ and = χ2m , L(θ0 ) √ √ [ n (θ˜n −θ0 )] I(θ˜n ) [ n (θ˜n −θ0 )] =a Dn I(θ0 ) Dn →d Z I −1 (θ0 ) Z ∼ = χ2m ,

since I(θ) is continuous at θ0 . (B) When θ = θ0 is in reality true (and for Kullback-Leibler information K(Pθ , Pθ0 )) 2 ˜ n →p 2 K(Pθ , Pθ0 ) ≡ 2 Eθ log pθ (X) > 0, with respect to Pθ . (19) log Λ n pθ0 (X)

4.

ASYMPTOTICS OF MAXIMUM LIKELIHOOD ESTIMATION

565

(C) We can replace (M3)(i) by (M3)(ii) in this theorem; just go to subsequences. [By Efron and Hinkley (1978), (16) and (17) are more accurate than (15) and (18).] Exercise 4.1 Show that (M1)(i)(ii), (M2)(i)(ii)(iii), and (M4) imply (M3)(ii). Exercise 4.2 When θ = θ0 in theorem 4.1, we have (in addition to (19)) (20)

1 n

Wn →p (θ − θ0 ) I(θ) (θ − θ0 ) > 0

for Wald’s Wn

(21)

1 n

˙ 0 |X)} I −1 (θ) Eθ {(θ ˙ 0 |X)} Rn →p Eθ {(θ

for Rao’s Rn .

and

Corollary 1 Suppose that q(θ) is also diﬀerentiable at θ0 . Then, when θ0 is true and with Q(θ) ≡ ∂θ∂r qi (θ)K×m for q(θ) = (q1 (θ), . . . , qK (θ)) , we have √

n [ q(θ˜n ) − q(θ0 )] =a Q(θ0 ) Dn =a Q(θ0 ) I −1 (θ0 ) Zn

(22) →d N (0, Q(θ0 ) I −1 (θ0 ) Q (θ0 )). Proof. For a.e. ω and for all n ≥ nω , we use (M1)(i) and (M3)(i) with 0 ≤ t∗ ≤ 1 for the mean value expansion of the continuously diﬀerentiable function of a real and obtain variable f (t) ≡ ˙r (θ0 + t(θ˜n − θ0 )|X), √ = √1 ˙r (θ0 |X) − n m (θ˜ns − θ0s ) [− 1 ¨rs (θ∗ |X)], (a) 0 = √1n ˙r (θ˜n |X) n s=1 n n for 1 ≤ r ≤ m, where θn∗ = θ0 + t∗ (θ˜n − θ0 ). Or, 0 = Zn − Iˆn (θn∗ ) Dn . From (M2)(i)(ii) and the multivariate CLT we have n ˙ ∼ = √1 (23) Zn ≡ √1n ˙r (θ0 |X) k=1 r (θ0 |Xk ) →d Z = N (0, I(θ0 )). n From (M2)(ii)(iii) and the SLLN, Iˆn (θ0 ) = − (24)

1 n

=− ¨rs (θ0 |X)

1 n

n

¨

k=1 rs (θ0 |Xk )

→a.s. Eθ0 {−¨rs(θ0 |X)} = Eθ0 {˙r (θ0 |X) ˙s (θ0 |X)} = I(θ0 ).

Use (M1)(ii) with 0 ≤ t∗∗ ≤ 1 for the mean value expansion of the continuously to obtain diﬀerentiable function f (t) ≡ ¨rs (θ0 + t(θn∗ − θ0 )|X) + 1 ¨rs (θ0 |X)| = | 1 n [¨rs (θ0 |Xk ) − ¨rs (θ∗ |Xk )]| | − n1 ¨rs (θn∗ |X) n k=1 n n = | n1

n

m

k=1

m

γ=1 (θ0γ

n |θ0γ − θ˜nγ |} × { n1 k=1 M (Xk )}

(b)

≤{

(c)

→a.s. {0} × {Eθ0 M (X)} = 0

γ=1

...

∗ − θnγ ) rsγ (θ0 + t∗∗ (θn∗ − θ0 )|Xk )|

using (M1)(ii)

566

APPENDIX A. DISTRIBUTION SUMMARIES

by (M3)(i), and by (M1)(ii) and the SLLN. Combining (24) and (c) (on each of the m × m coordinates) gives (d)

−

1 n

→a.s. I(θ0 ) ¨rs (θn∗ |X)

[that is, Iˆn (θn∗ ) →a.s. I(θ0 )] ;

and since · −1 is a continuous function of its argument, for all n ≥ (some nω ) the −1 exists and satisﬁes inverse − n1 ¨rs (θn∗ |X) (e)

−

1 n

−1 →a.s. I −1 (θ0 ) ¨rs (θn∗ |X)

[that is, Iˆn−1 (θn∗ ) →a.s. I −1 (θ0 )] .

We can now apply (e) and (23) to (a) (for all n ≥ (some nω )) to conclude that √ −1 Zn =a I −1 (θ0 ) Zn →d I −1 (θ0 ) Z, (f) n (θ˜n − θ0 ) = − n1 ¨rs (θn∗ |X) which gives (15). One permissible choice for θn∗ in (d) and (e) is θ˜n ; thus the conclusions (d) and (e) give (g)

Iˆn (θ˜n ) →a.s. I(θ0 )

Iˆn−1 (θ˜n ) →a.s. I −1 (θ0 ) ,

and

as claimed in (14). Thus (f) and (g) give (16). And (18) follows likewise, since I(θ˜n ) →a.s. I(θ0 ) in that case using the continuity of I(·) in (14). Of course, the ﬁnal claim of (14) is trivial. We now turn to (17). For a.e. ω and for n ≥ nω we now Taylor expand as = (θ˜n |X) + m (θ0r − θ˜0r ) ˙r (θ˜n |X) (θ0 |X) (h) r=1 + 12

m m r=1

˜

s=1 (θnr

− θ0r )(θ˜ns − θ0s ) ¨rs (θn∗ |X)

for some θn∗ = θ˜n + t∗ (θ0 − θ˜n ) with 0 ≤ t∗ ≤ 1. Thus ˜ n = 2 [(θ˜n |X) − (θ0 |X)] 2 log Λ = (i)

n

r=1 0

+2×

1 2

√

n (θ˜n − θ0 ) × −

1 n

× ¨rs (θn∗ |X)

√

= Dn Iˆn (θn∗ ) Dn

for Dn and Iˆn (·) as in (12) and (13)

=a Dn I(θ0 ) Dn

using (d)

→d D I(θ0 ) D

where D ∼ = N (0, I −1 (θ0 )) by (15)

∼ = χ2m .

(j)

Finally, we consider (19). When θ = θ0 is really true, we have 2 n

(k)

n (θ˜n − θ0 )

˜n = log Λ =

2 n

=

2 n

2 n

− (θ0 |X)] [(θ˜n |X)

− (θ0 |X)] + [(θ|X) n k=1

k) log ppθθ (X (Xk ) + 0

2 n 2 n

− (θ|X)] [(θ˜n |X) − (θ|X)] . [(θ˜n |X)

4.

ASYMPTOTICS OF MAXIMUM LIKELIHOOD ESTIMATION

567

The ﬁrst term in (k) converges a.s. Pθ to 2 K(Pθ , Pθ0 ) by the SLLN. The second − (θ|X)] =a 1 χ2 →p 0 using (j); the hypotheses still term in (k) has n2 [(θ˜n |X) n m must hold in a neighborhood of the truth, which is now θ. Thus (19) holds. Corollary 1 follows immediately by the delta method, since q(·) is diﬀerentiable. We now turn to corollary 2 below. As in (h), for t ≡ (t1 , . . . , tn ) we have √ ˙ 0 |X) − (θ0 |X) = √t (θ + 1 √t ¨rs (θ0 + √t∗ |X) √t (l) (θ0 + t/ n |X) 2

n

= t Zn −

1 2t

(m)

=a t Zn −

(n)

→d t Z −

Iˆn (θ0 +

1 2 t 1 2 t

n

n

n

t∗ √ )t n

ˆ 0) t I(θ

I(θ0 ) t ∼ = N (− 21 σ02 , σ02 )

with σ02 ≡ t I(θ0 ) t .

2

Corollary 2 Suppose (M0),(M1)(i)(ii) and (M2)(i)(ii)(iii) hold. Then √ − (θ0 |X) =a t Zn − 1 t I(θ0 ) t →d N (− 1 σ02 , σ02 ), (25) (θ0 + t/ n |X) 2

2

with = t I(θ0 ) t. (These local alternatives are said to be contiguous to the null hypothesis θ0 .) σ02

Notation 4.2 If the likelihood equations are diﬃcult to solve or have multiple roots, then a one-step approximation is possible. We thus suppose that θ¯n is a preliminary estimator of θ that satisﬁes n1/4 (θ¯n − θ0 ) →p 0 √ (and this does hold if n (θ¯n − θ0 ) = Op (1)). We now iterate once to obtain n (27) θˇn ≡ θ¯n + √1n Iˆn−1 (θ¯n ) Zn (θ¯n ) where Zn (θ) ≡ √1n k=1 ˙r (θ|Xk ) .

(26)

(Newton-Raphson suggests 0 = 2

√1 n

. ˙ θ˜n |X) = (

√1 n

˙ θ¯n |X) + √n (θ˜n − θ¯n ) Iˆn (θ¯n ).) (

Theorem 4.2 (One-step approximation) Suppose (M1)(i)(ii), (M2)(i)(ii)(iii) and condition (26) hold. Then (28)

θˇn may replace θ˜n throughout theorem 4.1 and corollary 1.

(Note that one can (in fact) iterate any ﬁxed ﬁnite number of times.) Proposition 4.1 (Kullback–Leibler information number) Let P and Q denote a probability measure and a subprobability measure (that is, Q(X ) ≤ 1) on (X , A) having densities p(·) and q(·) with respect to a σ-ﬁnite measure µ on (X , A). Then p(X) a number in [0, ∞] always, (29) K(P, Q) ≡ EP log = 0 if and only if P=Q . q(X) Exercise 4.3 Prove the previous one-step approximation theorem. Exercise 4.4 Prove the previous Kullback-Leibler proposition.

568

References I would like to use this discussion of the literature to say a very heartfelt “Thank you!” to a number of people who have ﬁgured prominently in my professional life. Especially, I want to thank my professors Fred Andrews (University of Oregon) and Lincoln Moses (Stanford University), whose volunteered eﬀorts on my behalf had far-reaching consequences on most aspects of my life. I shall oﬀer some thoughts on my own personal history as well as the subject matter of this book. My view is strongly aﬀected by how I came to learn about these things. Others have undoubtedly had diﬀerent experiences. Measure theory This text begins with ﬁve chapters devoted to measure theory. Halmos (1950) has had a major inﬂuence on what future books on measure theory and real analysis would contain and how they would present the subject. Other books on measure theory and real analysis that I have found to be especially useful include Royden (1963), Hewitt and Stromberg (1965), Rudin (1966), and the nicely simpliﬁed presentation of Bartle (1966). Many theorems in this introductory part are to some degree recognizable from several of these sources (and/or from the other sources listed in the probability section below). Certainly, Halmos’s book was a popular one while I was getting my M.S. degree in mathematics at the University of Oregon, 1960–1962. My own introduction to “real and abstract analysis” came from a beautiful course taught by Karl Stromberg. Later, Edwin Hewitt was a colleague at the University of Washington. So it is a real pleasure for me to cite their work at various points. Lou Ward taught the topology course that I took at Oregon. He gave us a list of theorems, and we had to come up with proofs and present them. That was the most fun I ever had in the classroom. A good deal of chapter 6 reﬂects what I learned in his course. Kelly (1955), Copson (1968), and Housain (1977) are useful published sources. Watching over the Oregon graduate students in mathematics was Andrew Moursand, chairman. He really cared about all of us, and I owe him my thanks. Probability Lo´eve’s (1977-78, originally 1955) presentation has been a very inﬂuential work on probability, certainly from the pedagogical point of view. To me, it reﬁnes and specializes much general analysis to probability theory, and then treats a broad part of this subject. Clearly, many learned probability from his text. Also, many seem to follow notational conventions used in his book. But I was rather late in learning from it. My original training was at Stanford from lectures that became Chung (1974), and those lectures also reﬂected Chung’s eﬀorts regarding translation of Gnedenko and Kolmogorov (1954). I truly enjoyed Chung’s course, and his book. Breiman’s (1968) style coincided most closely with my own. I particularly liked his treatment of partial sum and empirical processes, as one would suspect from my own research. I have sometimes used his text as a “permanent reference” to stand beside my own notes in my courses on probability theory. My choice of notation has been most inﬂuenced by Lo´eve and Breiman. Feller (1966) has a diﬀerent ﬂavor from most probability texts, and it includes various interesting approaches not found

REFERENCES

569

elsewhere. And it is informative on rates of approximation. Billingsley (1968) created some excitement and spawned much interesting work, and a bit of that is included here. Doob’s (1954) work on martingales has had a huge inﬂuence on the subject. I ﬁnd Meyer (1966) and Hall and Heyde (1980) particularly signiﬁcant. Lectures by Tom Fleming that led to Fleming and Harrington (1991) sparked part of my martingale presentation here. Whittaker and Watson (1963) is still a superb source for the gamma function. Lehmann (1959) has greatly inﬂuenced my view of conditional probability and expectation. This brings me back to the University of Oregon, and to Fred Andrews. Fred “recruited me to statistics” and then taught a year-long course out of Lehmann’s book (even though I was the only student), and he was one of those who lined me up for a National Science Foundation fellowship that made it possible for me to go to Stanford University. Don Truax also ﬁgured heavily in this. He cared about me, and I learned a lot from him. Thank you both! The scene shifts southward. My years at Stanford were very fruitful, and I met some ﬁne people. Ingram Olkin is fun, a good teacher, and he went out of his way to be helpful to me. The multivariate topics in chapter 9 represent what I learned from him. Lincoln Moses was my thesis advisor. This relationship grew out of a course in nonparametric statistics that I took from him. One of the topics in his course was Charles Stein’s approach to the central limit theorem. Lin spoke on it for three days, even though he had to leave a couple of well-acknowledged gaps in his presentation—because he believed it was good work. That gave me a profound respect for him as a teacher. The topic caught my imagination, and chapters 11 and 17 reﬂect this. Lin was also my assigned advisor when I arrived at Stanford. His second sentence to me was, “OK, Shorack, what’s important to you in life?” My answer had a lot to do with the geography of the Paciﬁc Northwest. Two months before I graduated he responded on my behalf to a University of Washington opening. Wow! At Washington I had a chance to teach courses in probability and statistics. And I learned a lot from my association with Ron Pyke, and later with Jon Wellner. The presentations in parts of chapters 12, 16, and 19 reﬂect this to varying degrees. Fritz Scholz got me started on gamma approximations in the central limit theorem. Likewise, work with David Mason on quantile functions, embedding, and trimmed means is reﬂected in parts of chapters 7, 10, and 16. I oﬀer them all my thanks. Obviously, I also owe a huge debt to “the literature” in regard to all these topics, and I will list some of those sources below. However, this is a textbook. It is not a research monograph. My emphasis is on presentation, not attribution. Often, my citation concerns where I learned something rather than who did it originally. And in some areas (especially, chapters 7, 12, and 16) I have oﬀered only a sampling of the citations that I could have given. Moreover, I have often chosen to cite a book instead of an original paper. My own work is cited “too heavily” because it is written in the same style and notation as this book. The bibliography contains quite a number of other books on probability theory, and many are very good books. But it is the ones listed above that have had the most inﬂuence on me. I hope that the reader will ﬁnd that my book also has a somewhat diﬀerent ﬂavor—a statistical ﬂavor.

570

PROBABILITY FOR STATISTICIANS

Special thanks to Chari Boehnke, Roger and Heather Shorack, the Michael Boehnke family, the Barbara Aalund family, Kathleen Shorack, David Mason, Michael Perlman, Fritz and Roberta Scholz, Jon Wellner, the Jan Beirlant family, Piet Groeneboom, Frits Ruymgaart, Derek Dohn, and Pauline Reed for enabling me to write this book. CHAPTERS 1-6 Ash, R. (1972) Real Analysis and Probability. Academic Press, New York. Bartle, R. (1966) The Elements of Integration. John Wiley & Sons, New York. Breiman, L. (1968) Probability. Addision-Wellsey, Reading, MA. Chow, Y. and H. Teicher (1997) Probability Theory: Independence, Interchangeability, Martingales 3rd ed. Springer-Verlag, New York. Cohn, D. (1980) Measure Theory. Birkh¨ auser, Boston. Copson, E. (1968) Metric Spaces. Cambridge University Press, Cambridge. Halmos, P. (1950) Measure Theory. Van Nostrand, Princeton, NJ. Hewitt, E. and K. Stromberg (1965) Real and Abstract Analysis. Springer-Verlag, New York. Husain, T. (1977) Topology and Maps. Plenum Press, New York. Lo` eve, M. (1977–78) Probability Theory. Springer-Verlag, New York. Meyer, P. (1966) Probability and Potentials. Blaisdell, Waltham, MA. Royden, H. (1967) Real Analysis. Macmillan, New York. Rudin, W. (1966) Real and Complex Analysis. McGraw-Hill, New York.

CHAPTER 8 Breiman, L. (1968) Probability. Addision-Wellsey, Reading, MA. Breiman, L. and J. Friedman (1985) Estimating optimal transformations for multiple regression and correlation. J. Amer. Statist. Assoc. 80 580-619. Lehmann, E. (1959) Testing Statistical Hypotheses. John Wiley & Sons, New York.

CHAPTER 10 Breiman, L. (1968) Probability. Addision-Wellsey, Reading, MA. Chow, Y. and H. Teicher (1997) Probability Theory. Independence, Interchangeability, Martingales 3rd ed. Springer-Verlag, New York. Chung, K. (1974) A Course in Probability Theory 2nd ed. Academic Press, New York. Feller, W. (1966) An Introduction to Probability Theory and Its Applications. Vol. 2. John Wiley & Sons, New York. Gin´ e, E., F. G´ otze, and D. Mason (1998) When is the student t-statistic asymptotically standard normal? Ann. Probab. 25 1514-1531. Gnedenko, B. and A. Kolmogorov (1954) Limit Distributions for Sums of Independent Random Variables. Addison-Wesley, Cambridge, Mass. Hoﬀman–Jorgensen, J. (1974) Sums of independent Banach space valued random variables. Studia Math. 52 159-186.

REFERENCES

571

Kallenberg, O. (1997) Foundations of Modern Probability. Springer-Verlag, New York. Lo` eve, M. (1977-78) Probability Theory. Springer-Verlag, New York. Resnick, S. (1999) A Probability Path. Birkh¨ auser, Boston. Strassen, V. (1967). Almost sure behavior of sums of independent random variables and martingales. Proceeding of the Fifth Berkeley Symposium on Mathematical Statistics and Probability 2 315-343, University of California Press, Berkeley CA.

CHAPTER 11 Anderson, T. (1984) Introduction to Multivariate Statistical Analysis. John Wiley & Sons., New york. Billingsley, P. (1968) Convergence of Probability Measures. John Wiley & Sons, New York. Cs¨ org˝ o, S. and D. Mason (1989) Bootstrapping empirical functions. Ann. Statist. 17 1447–1471. Gin´ e, E. and J. Zinn (1990) Necessary conditions for the bootstrap of the mean. Ann. Statist. 17 684–691. Hall, P. (1990) Asymptotic properties of the bootstrap for heavy tailed distributions. Ann. Statist. 18 1342-1360. Lehmann, E. and G. Casella (1998) Theory of Point Estimation 3rd ed. SpringerVerlag, New York. Lo` eve, M. (1977-78) Probability Theory. Springer-Verlag, New York. Reinhert, G. (1977) Couplings for normal approximations with Stein’s method. To appear in Conference Proceedings, Microsurveys in Discrete Probability. Shorack, G. (1998) On Stein’s appraoch to the central limit theorem. Univ. Washington Dept. of Statistics Technical Report 339. Stein, C. (1986) Approximate Computation of Expectations. IMS, Hayward, CA.

CHAPTERS 7, 12, and 16 Billingsley, P. (1968) Convergence of Probability Measures. John Wiley & Sons, New York. Breiman, L. (1968) Probability. Addision-Wellsey, Reading, MA. Chung, K. (1974) A Course in Probability Theory 2nd ed. Academic Press, New York. Cram´ er, H. and M. Leadbetter (1967) Stationary and Related Stochastic Processes. John Wiley & Sons, New York. Cs¨ org˝ o, M., S. Cs¨ org˝ o, L. Horv´ ath and D. Mason (1986) What portion of the sample makes a partial sum asymptotically stable or normal? Probab. Theory Related Fields 72 1–16. Cs¨ org˝ o, M., S. Cs¨ org˝ o, L. Horv´ ath and D. Mason (1986) Weighted empirical and quantile processes. Ann. Probab. 14 31–85. Cs¨ org˝ o, S., E. Haeusler, and D. Mason (1988) The asymptotic distribution of trimmed sums. Ann. Probab. 16 672-699.

572

PROBABILITY FOR STATISTICIANS

Cs¨ org˝ o, S., E. Haeusler, and D. Mason (1989) A probabilistic approach to the asymptotic distribution of sums of independent identically distributed random variables. Adv. in Appl. Math. 9 259-333. Doob, J. (1949) Heuristic approach to the Kolmogorov–Smirnov theorems. Ann. Math. Statist. 20 393-403. Freedman, D. (1971) Brownian Motion and Diﬀusion. Holden-Day, San Francisco. Haeusler, E. and D. Mason (1989) A law of the iterated logarithm for modulus trimming. Colloquia Math. Soc. J. Bolyai, 57 Limit Theorems in Probability and Statistics. Mason, D. (1999) An exponential inequality for weighted approximation to the uniform empirical process with applications. University of Deleware Technical Report. Mason, D. and G. Shorack (1990) Necessary and suﬃcient conditions for asymptotic normality of trimmed L-statistics. J. Statist. Plan. Inf. 25 111-139. Meyer, P. (1966) Probability and Potentials. Blaisdell, Waltham, MA. Ross, S. (1997) An Introduction to Probability Models, 6th ed., Academic Press, San Diego. Shorack, G. (1998) The asymptotic normality condition. Univ. Washington Dept. of Statistics Technical Reports 323, 336, 337, and 338. Shorack, G. (1991) Embedding the ﬁnite sampling process at a rate. Ann. Probability 19 826-842. Shorack, G. (1996) Linear rank statistics, ﬁnite sampling, permutation tests and Winsorizing. Ann. Statist. 24 1371-1385. Shorack, G. (1997a) Inequalities for quantile functions with a uniform studentized CLT that includes trimming. Nonparametric Statist. 8 307-335. Shorack, G. (1997b) Uniform CLT, WLLN, LIL and bootstrapping in a data analytic approach to trimmed L-statistics. J. Statist. Plan. Inf. 60 1–44. Shorack, G. (1998) Applications of weighted approximations via quantile inequalities. Asymptotic Methods in Probability and Statistics, A Volume in Honor of Mikl´ os Cs¨ org˝ o. B. Szyszkowicz, Ed., Elsevier, Amsterdam. Shorack, G. and J. Wellner (1986) Empirical Processes with Applications to Statistics. John Wiley & Sons, New York. CHAPTERS 13-15 Alfors, L. (1979) Complex Analysis McGraw-Hill, Inc., New York. Bickel, P. and K. Doksum (1977) Mathematical Statistics: Basic Ideas and Special Topics. Holden-Day, San Francisco. Bhattacharya, R. and R. Rao (1976) Normal Approximation and Asymptotic Expansions. John Wiley & Sons., New York. Billingsley, P. (1986) Probability and Measure. John Wiley & Sons, New York. Breiman, L. (1968) Probability. Addision-Wellsey, Reading, MA. Durrett, R. (1996). Probability: Theory and Examples. Wadsworth, Belmont, CA. Feller, W. (1966) An Introduction to Probability Theory and Its Applications. Vol. 2. John Wiley & Sons, New York.

REFERENCES

573

Galambos, J. (1995) Advanced Probability Theory 2nd ed. Marcel Dekker, New York. Gnedenko, B. and A. Kolmogorov (1954) Limit Distributions for Sums of Independent Random Variables. Addison-Wesley, Cambridge, Mass. Hall, P. (1983) Chi squared approximations to the distribution of a sum of independent random variables. Ann. Statist. 11 1028-1036. Kendall, M. and A. Stuart (1977) The Advanced Theory of Statistics Vol. 2. Macmillan, New York. L´ evy, P. (1937) Th´eorie de l’Addition des Variables Al´eatoires. Bautier-Villars, Paris. Lo` eve, M. (1977-78) Probability Theory. Springer-Verlag, New York. Petrov, V. (1975) Sums of Independent Random Variables. Springer-Verlag, New York. Shiryayev, A. (1984) Probability. Springer-Verlag, New York.

CHAPTER 17 Barbor, A. and P. Hall (1984) Stein’s method and the Berry–Esseen theorem. Austral. J. Statist. 26 8-15. Bolthausen, E. (1984) An estimate of the remainder in a combinatorial central limit theorem. Z. Wahrsch. Verw. Geb. 66 379-386. ˇ H´ ajek, J. and V. Sidak (1967) Theory of Rank Tests. Academic Press, New York. Hoeﬀding, W. (1948) A class of statistics with asymptotically normal distribution. Ann. Math. Statist. 19 293–325. Hoeﬀding, W. (1951) A combinatorial central limit theorem. Ann. Math. Statist. 22 558-566. Reinhert, G. (1977) Couplings for normal approximations with Stein’s method. To appear in Conference Proceedings, Microsurveys in Discrete Probability. Serﬂing, R. (1980) Approximation Theorems of Mathematical Statistics. John Wiley & Sons, New York. Stein, C. (1986) Approximate Computation of Expectations. IMS, Hayward, CA.

CHAPTER 18 Andersen, P., O. Borgan, R. Gill and N. Keiding (1993) Statistical Models Based on Counting Processes. Springer-Verlag, New York. Breiman, L. (1968) Probability. Addision–Wellsey, Reading, MA. Doob, J. (1953) Stochastic Processes. John Wiley & Sons, New York. Fleming, T. and D. Harrington (1991) Counting Processes and Survival Analysis. John Wiley & Sons, New York. Hall, P. and C. Heyde (1980) Martingale Limit Theory and Its Application. Academic Press, New York. Liptser, R. and A. Shiryaev (1977) Statistics of Random Processes I. SpringerVerlag, New York. Meyer, P. (1966) Probability and Potentials. Blaisdell, Waltham, MA. Williams, D. (1991) Probability with Martingales. Cambridge University Press, Cambridge.

574

PROBABILITY FOR STATISTICIANS

Chapter 19 Billingsley, P. (1968) Convergence of Probability Measures. John Wiley & Sons, New York. Dudley, R. (1976) Convergence of laws on metric spaces, with a view to statistical testing. Matematisk Institut Aarhus Universitet. Dudley, R. (1989) Real Analysis and Probability. Wadsworth & Brooks/Cole, Paciﬁc Grove, CA. Skorokhod, A. (1956) Limit theorems for stochastic processes. Theor. Probab. Appls. 1 261-290.

APPENDIX A Abramowitz, M. and I. Stegun (1964) Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. National Bureau of Standards Applied Mathematics Series, 55 U.S. Government Printing Oﬃce, Washington, DC. Efron, B., and D. Hinkley (1978) Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information. Biometrika 65 457-482. Johnson, N., A. Kotz, and N. Balakrishnan (1994) Continuous Univariate Distributions Volumes 1 and 2. John Wiley & Sons, New York. Rao, C. (1965) Linear Statistical Inference and Its Applications 2nd ed. John Wiley & Sons, New York. Whittaker, E. and G. Watson (1963) A Course of Modern Analysis. Cambridge University Press, Cambridge.

Index Brownian motion, 202, 235, 302, 311, 318, 469, 500, 536 integrals of, 303 LIL, 303, 320 path properties, 323 reﬂection principle, 314, 315, 321, 322, 331, 409 transformations, 303, 318

a.b.f., 8 absolute continuity, 255, 258 fundamental theorem of calculus, 76, 78, 137 of functions, 74, 76, 78, 323 of log f , 78, 138 of measures, 60, 63, 64, 66–68, 78, 108 of the integral, 42, 54, 66, 76, 480 Radon–Nikodym, 66 adapted, 35, 36, 305 added touch, 124, 268 ancillary statistic, 173 approximation by continuous functions, 59, 60 by elementary functions, 26 by simple functions, 26 by step functions, 59, 78 ¯ n )), 396 approximation for L(h(X approximation lemma, for sets, 16, 80, 155, 156 associated rvs, 264

Cantor set, 108, 114 Carath´eodory covering, 12, 17, 77 extension theorem, 5, 10, 12, 80, 88, 480 censored data, 523, 527 chain rule, 67, 78 change of variable, 78, 113, 134 Radon–Nikodym, 66–68 unconscious statistician, 42, 67, 113 chf, 242, 294, 341, 351 complementary pair, 346 continuity theorem, 342, 350, 351, 366, 381 distributions on grids, 361 empirical, 348 Esseen’s lemma, 358, 371, 373, 385, 391, 394 examples, 343–345, 349 expansion for uan array, 400 Fourier, Laplace, etc., 341, 356 inversion formula, 346–348, 359, 362, 380, 381, 385, 386, 393 moment expansions, 352–354, 371, 374 multivariate normal, 200 tail behavior, 355

best linear predictor, 194 bootstrap, 274, 277, 388, 432 Bayesian, 434 Borel sets, 5, 6, 17, 18, 23 ¯ 18 B,Bˆµ,B, generators of B, 24 generators of Bn , 23, 35, 80 generators of B∞ , 36, 86 countable base BC of BT , 91 Borel–Cantelli, 204, 217, 218, 236, 237, 244, 321, 376 bounded variation, 74–76 total variation measure, 62 branching processes, 484 Brownian bridge, 202, 302, 326 575

576 uniqueness theorem, 342, 346, 347, 350, 351, 366 chisquare tests, 369, 379, 389 CLT, 125, 126, 147 asymptotic normality condition, 265, 406, 529 basic dependent, 529 Berry–Esseen, 259, 261, 263, 267, 270, 276, 278, 371, 376, 450 Berry–Esseen for U-statistics, 454 bootstrap, 274, 434 classical, 183, 238, 342, 366 connection with variance estimation, 125, 233, 270, 271 creating normality, 428 delta method, 279 Doeblin, 285, 370 Edgeworth (see also), 391 examples, 279, 280, 282, 283, 286, 287, 368, 369 ﬁnite sampling, 388, 430, 464, 465 for general iid uan arrays, 406 Fr´echet–Shohat via moments, 293 gamma approximation, 383, 388, 389, 459 Hoeﬀding’s combinatorial, 458 L-statistics, 441, 442 Liapunov, 371, 377 Lindeberg–Feller, 260, 266, 267, 373, 404, 529 local limit (see also), 380 martingale, 530 multivariate, 201, 368 negligibility, 215, 233, 234, 367 Poisson approximation, 386 R-statistics, 427 random sample size, 285 Rebolledo, 503, 505 slowly varying σ ˜ 2 (·), 233, 268, 270– 272 Stein’s approach, 255, 262, 263, 459 Studentized CLT for σ ˜2 (·) ∈ L, 270, 272 trimmed means, 416, 417, 420 U-statistics, 450, 453

PROBABILITY FOR STATISTICIANS universal bootstrap CLT, 277 universal studentized CLT, 276 Winsorized, 264, 270 compensator, 487, 488, 501, 511 complete measure, 15, 18, 29 space Lr , 51 completeness assumption, 469 conditional expectation, 158, 159, 168, 171, 246, 248 properties of, 163 conditional probability, 159, 160, 186, 201 regular, 10, 168–170 contiguity, 567 convergence a.e., 29–31, 51 a.s., 33, 78, 89, 114, 241 almost uniform, 60 Ces` aro summability, 205, 217 DCT, 41 fd convergence →f d , 91 in distribution, 33, 52, 89, 112, 114, 241, 288, 290, 291, 293, 342, 531 in L2 , 78 in measure, 30, 31, 51 in probability, 33, 51, 89, 241 in quantile, 112 in rth mean, 33, 51, 89 in sub-df, 288 MCT, 40 modes, 57 mutual, 29–31, 51 set, 29, 217, 242, 252 uniform, 60 convex, 46, 48, 49, 56, 468 convolution formula, 187, 347–349 correlation coeﬃcient, 286, 463 multiple, 194 Spearman, 462 counting process, 470, 501 coupling, 262, 389, 542 Cram´er Cram´er–L´evy continuity theorem, 350

INDEX Cram´er–von Mises, 304, 331, 338, 339 Cram´er–Wold device, 351, 368, 378 cumulant gf, 356, 371, 384, 387, 388, 392 cumulative hazard function, 470, 506, 522, 527 decomposition of Normals, 405 decomposition of Poissons, 405 dense Bernoulli polynomials, 223 continuous functions, 59, 78 in Hilbert space, 106 step functions, 59, 78, 482 density estimation, 348 derivative diﬀerentiable, 70, 75, 76 Dini derivates, 70 Lebesgue theorem, 70, 71, 78 Radon–Nikodym, 66–68, 74, 76– 78, 158 series, 71 Taylor’s theorem, 72, 352 under integral sign, 45, 317 determining class, 347 (k) (∞) C,C0 ,Cb ,Cbu ,Cb ,Cb , 292 chf, 294 G,G0 ,G1 ,G2 , 292 limit, 53 moments, 293, 294, 354 df, 20, 33, 107 absolutely continuous, 108 decomposition of, 107 generalized, 18, 19, 27 joint, 84, 85, 154 jumps and ﬂat spots, 110, 134 Lebesgue singular, 71, 108 properties, 113 singular, 108 sub, 33, 107, 288 support, 110 diagonalization, 289 discontinuity set, 26 distribution Bernoulli, 179

577 Beta, 184, 549 Binomial, 179, 555 Cauchy, 184, 187, 242, 284, 342– 344, 412 Chisquare, 183 Compound Poisson, 402 de la Vall´ee Poussin, 343, 346, 348, 363 Discrete Uniform, 555 Double Exponential, 184, 283, 345, 558 Exponential, 181, 287 Extreme Value, 286, 559 Gamma, 182, 383, 392, 406, 556 Geometric, 179 hitting time, 409 Hypergeometric, 180, 555 Log Gamma, 559 Logistic, 184, 345, 558 Multinomial, 185, 369, 402 Multivariate Normal, 199, 286 NegBiT, 179, 387, 555 Noncentral, 378, 379, 389 Normal, 183, 287, 405, 549, 556 Pareto, 562 Poisson, 181, 284, 287, 386, 405, 555 sampling without replacement, 180 Snedecor’s Fm,n , 188, 557 stable laws (see also), 407 Student’s tm , 188, 557 Triangular, 343, 346, 348 Uniform, 184, 558 Weibull, 561 divergence set, 29 domain of attraction, 145 D(Normal), 267, 268, 271, 272, 277 DN (Normal), 413 D(G) of the stable law G, 407, 412 DN (G) of the stable law G, 407, 412 ˜ 145, 417, 442 statistical D, total D, 145 Edgeworth expansions, 386, 391, 392

578 eigenvalues, 191, 196, 340 embedding Cs¨ org˝ o, Cs¨ org˝ o, Horv´ ath, Mason, 328 Shorack, 328 Skorokhod, 235, 312, 318, 329 empirical chf, 348 df Fn , 121, 223, 325 df Gn , 121, 223, 325, 417 process En , 121 process Un , 121, 325, 368, 417 qf Kn , 120 quantile process Vn , 325 two-sample process Wm,n , 331 weighted process Wn , 326 ﬁnite sampling process Rn , 326 equivalent summands, 267 Euclidean space, 23 Euler’s constant, 186, 546, 550 expectation, 38 of products, 152 extremes, 370 ﬁltration, 469, 489, 493, 511 ﬁnite-dimensional convergence →f d , 91 distributions, 90 projection mappings πt1 ,...,tk , 90 rectangle, 86, 89 σ-ﬁeld, 90 subsets, 90 function elementary, 24–26 indicator, 3, 24 measurable, 24 ψ-function, 59 simple, 24–26, 28, 37, 166 step, 59 gambler’s ruin, 499, 500 gamma approximation; the GLT, 383 gamma function, 546 digamma, 546 generalized inverse, 193 Σ− and Σ−1/2 , 192, 379 generators, 4

PROBABILITY FOR STATISTICIANS ¯ of the induced F(X) ≡ X −1 (B), 24 of the induced F(X1 , . . . , Xn ), 35 of the induced F(X1 , X2 , . . .), 35 regarding X −1 , 22 regarding independence, 9 σ-ﬁeld, 3, 10, 24 various for Borel sets B, 24 various for Borel sets Bn , 23, 35, 80 various for Borel sets B∞ , 36, 86 Glivenko–Cantelli, 223, 325, 326 Hermite polynomials, 317, 390, 391, 470 Hewitt–Savage zero-one law, 156 Hilbert space, 104, 174 2 -isomorphism, 106 alternating projections (ACE), 175 Fourier coeﬃcients, 105, 106, 340 Gram–Schmidt, 106 idempotent operator, 174 inequalities, 104, 106 L2 is a Hilbert space, 104 orthogonal projections, 105, 174 orthonormal basis, 105, 106 projection operator, 176 self-adjoint operator, 174 subspaces, 105 hitting time, 311, 314, 409 Hodges–Lehmann estimator, 455, 456 i.o., 8, 204 increment function, 18 independence tests, 369 independent, 163 rvs, 151, 153, 154, 157, 189, 200, 201, 351 σ-ﬁelds, 151, 153 indicator function proof, 28, 38, 42, 66, 81, 83, 164, 166 induced distribution, 23, 24, 27, 33, 42, 52, 92 distribution consistency, 87 inequality basic, 49

INDEX Birnbaum–Marshall, 249, 333 Bonferroni, 50 bounds on (1 − x/n)n , 354, 550 Cantelli, 221 Cauchy–Schwarz, 48, 197 Chang, 336 Chebyshev, 49, 218 chf bound on tails, 350 chf key inequality, 372 convexity, 48 correlation, 48 Cr -inequality, 47, 163, 209, 313, 372 Daniels’ equality, 214, 230, 335 dispersion, 167, 209 Doob, 247, 249, 335 Etemadi, 212 geometric mean, 50 Gin´e–Zinn symmetrization, 213, 273 H´ ajek–R´enyi, 221, 248, 333, 452, 453 Hardy, 50 Hilbert space inequalities, 104, 106 Hoeﬀding–Fr´echet, 85 Hoﬀman–Jorgensen, 250 H¨ older, 47, 48, 163, 209 Jensen, 49, 163, 166, 167, 209, 248, 468 Khinchin, 213, 273 Kolmogorov, 210, 217, 218, 247, 248, 253, 285, 321 Kolmogorov’s other, 243 L´evy, 211, 220, 235 Liapunov, 48, 163, 209, 371 Littlewood, 48 Markov, 49, 209 Mills’ ratio, 235, 237, 321, 322 Minkowski, 49, 53, 163 moment expansions of chfs, 352, 354, 371, 374 monotone, 205, 221, 247, 248, 321 Ottavani–Skorokhod, 212, 242 Paley–Zygmund, 49, 273, 324 Pyke–Shorack, 328, 334, 336

579 sandwiching the mean, 206, 207, 215, 217, 218 Shorack, 141, 143, 147, 214, 256, 458 Shorack–Smythe, 247 symmetrization, 210, 219 truncation, 219, 227–229, 232 upcrossing, 473 Wellner, 50 Winsorized variance, 141, 143, 147 Young, 47 inﬁnitely divisible, 400 limits of, 400 log chf is never zero, 400 subclass I2 , 400 information, 35 integrable, 38 collection, 54 L1 , 37 L2 , 78 Lr , 37, 47 L+ r , 37 product, 82 uniformly, 54–56, 221, 252, 473, 477, 478, 488, 492, 511, 516 integral, 38 improper, 44 Lebesgue, 37 Lebesgue–Stieltjes, 44 linearity, 38, 163 Riemann, 1, 75 Riemann–Stieltjes, 44 integration by parts formulas, 115, 118 inverse image, 21, 22 of σ-ﬁelds, 22 inverse transformation, 111 Jacobian, 78, 190 Khinchin equivalent rvs, 206, 207, 215, 244 Kolmogorov consistency theorem, 92, 302 extension theorem, 87 Gnedenko–Kolmogorov theorem, 123, 232 inequality (see also), 210

580 Kolmogorov–Smirnov, 316, 331 representation theorem for I, 402 SLLN (see also), 215, 275 zero-one law, 155, 204, 482 Kullback-Leibler information, 564, 567 large deviations, 395 Lebesgue decomposition theorem, 64, 66, 108 integral, 37 Lebesgue–Stieltjes measure, 4, 18, 20, 77, 78, 80 measure, 6, 23, 38, 67 measure λn , 80 sets, 15, 17 singular df, 71, 108 sums, 1 theorem re derivatives, 70, 71, 78 likelihood ratios, 470 LIL, 235, 238, 321, 323 see also Brownian motion, 303 U-statistics, 452 limit determining class, 292 limit theorem general uan terms, 405 uan terms with negligible variances, 403 Lindeberg’s LFn , 260, 266, 267, 278, 283, 373, 375, 376, 436 linear algebra, 191 Lipschitz condition, 74, 540 LLN, 126, 148, 218, 222, 224–226 Glivenko–Cantelli (see also), 223 negligibility, 215, 226 random sample size, 220 SLLN of Kolmogorov, 208, 215, 220, 275, 481 strong, 221, 224, 238, 245, 325 U-statistics, 452, 453, 481 uniform SLLN of Chung, 252 weak, 208, 224, 226, 228, 245, 366, 377, 406, 425 WLLN of Feller, 208, 215, 220 WLLN of Khinchin, 208 local limit theorem, 380

PROBABILITY FOR STATISTICIANS Mann–Whitney statistic, 456 martingale, 246, 311, 317, 467 closes, 473, 477, 478, 493 CLT, 530 convergence theorem, 473, 477, 478 counting process, 470 decomposition, 487, 488, 502, 511, 512 equivalence, 246, 467 examples re empiricals, 333 exponential, 470, 499, 500 integrable, 468 Kakutani, 470, 482 local, 511 > = notation, 246 optional sampling theorem, 312, 472, 492, 493 reversed, 335, 453, 478, 481, 498 s-mg, 246, 467 square-integrable, 468 sub mg, 246, 247, 467 transform, 489, 490, 502, 513, 516 U-statistic, 452 Wald, 469 Mason theorem, 329, 332, 336 maximum likelihood, 551, 563 measurability criterion, 24, 25 measurable, 24 as a limit, 25, 29 A -A-measurable, 24 Bn -measurable, 84 B∞ -measurable, 86 common functions are, 25 C-measurable, 298 D-measurable, 301, 320 Aτ -measurable, 306 F(X)-measurable, 24, 28, 86, 158 F(X1 , . . . , Xn )-measurable, 35 F(X1 , X2 , . . .)-measurable, 35, 86, 91 Ft -measurable, 36 F(Xs : s ≤ t)-measurable, 36, 91 ˜ F(S)-measurable, 239 measurability criterion, 28 non, 16, 92

INDEX partition, 37 progressively, 306 set, 26, 29 σ[C]-measurable, 24, 28 space, 3 measurable function spaces, 90 (C[0,∞) , C[0,∞) ), 297 (M, d) with Md or MB d , 295 (C, C), 90, 295, 298 (D, D), 295, 298, 469 general space (MT , MT ), 90 (R[0,1] , B[0,1] ), 90 (RT , BT ), 91 measure, 4 absolute continuity, 63, 64, 66– 68, 78, 108 Borel, 531 complete, 15, 18, 29 continuous, 7, 62, 88 counting, 6, 67 ﬁnite, 61 induced, 23, 24, 27, 33, 42, 52, 92 Lebesgue, 4, 6, 23, 38, 67 Lebesgue λn , 80 Lebesgue–Stieltjes, 4, 18, 20, 77, 78, 80 monotone property, 6, 62 motivation, 4 outer, 4 outer extension, 12 positive part, 62, 64 positive set, 62 probability, 20 product, 80, 81 regular, 16, 20 σ-ﬁnite, 12, 14, 61, 64, 66, 80 signed, 61, 64, 66, 76 singular, 63, 64, 71, 108 space, 4 total variation, 62, 513 uniform convergence, 41, 54, 60 mesh, 44 metric space, 101, 295 Arzel` a theorem, 103, 539, 541 Ascoli theorem, 103 compact, 102

581 compactness equivalents, 102 complete, 102, 537 covering numbers, 103 discontinuity set, 26 equicontinuity, 103 equivalent metrics, 101 properties, 101 regular measure, 16 relatively compact, 537 separable, 295, 537 sup norm, 103 totally bounded, 102 uniform continuity, 103 metrics d on (D, D), 297 Dudley, 540 Hellinger, 68, 543 Kolmogorov, 543 L´evy, 290 Prohorov, 540 total variation, 68, 380, 543, 544 modulus of continuity, 539 moment, 46, 47, 117 conditional, 195 consistent estimation, 226, 228, 231, 270, 272, 275 convergence of, 53, 244, 289, 293, 376 correlation, 48, 157, 463 covariance, 46, 117, 157, 199 cumulant, 357, 388 generating function, 395 mean, 46, 116, 117, 119 partial, 127, 128, 134, 226–228, 231, 269 partial variance, 145 skewness γ1 and tail heaviness γ2 , 279, 357, 383, 386–388, 391, 392, 554 standard deviation, 46 variance, 46, 116, 117, 119, 233 moments determine the normal, 293 moments of stable laws, 409 Monte Carlo, 225 natural parameters, 272

582 negligibility, 213–215, 217–221, 226, 228, 231, 233, 264, 274, 275, 367, 427, 430 uan, 260, 264, 283, 326, 377, 399 nonnegative deﬁnite, 193, 202, 363 norm L2 -norm, 104 q-norm, 329 rth mean, 46 sup, 22, 103, 295, 330 null set, 15, 83 oh big Op , 209 big O, 9 little op , 209 little o, 9 o-plus ⊕, or “at most”, 9 optional sampling, 472, 492, 499 order statistics, 120, 172, 173, 184, 281, 325 orthogonal, 104, 191, 338, 339, 369, 379, 512 partial sum process S n, 318, 319, 368, 536, 539 partition, 37 P∞ , 36 PLT, 367, 404 negligibility, 367 Poisson limit theorem, 367, 404 Poisson compound, 402 generalized, 402 Poisson approximation, 386, 389 Poisson process, 181, 470, 500 positive deﬁnite, 191 positive part, 21, 119, 468 predictable, 487, 489, 502, 511, 513, 514, 516 covariation, 512, 517 σ-ﬁeld, 513, 514 variation, 489, 490, 501, 502, 512, 516 probability integral transformation, 113 inverse, 111 process, 90

PROBABILITY FOR STATISTICIANS convergence on (D, D), 538 counting, 470 D-class, 511 empirical process (see also), 121 existence of on (C, C), 298 existence of on (D, D), 301 general, 91 increasing, 511 independent increments, 297 normal, 90, 202 predicatable (see also), 487 realizations (equivalent), 90 realizations (smoother), 93 stationary, 297 stationary increments, 297 versions, 90 product F0 , F, A × A = σ[F], µ × ν, 79 countable, 86 cylinder set, 86 Fubini theorem, 82 integrable, 82 measurable rectangle, 79 measure, 80, 81 measure existence, 80, 82 null sets, 83 sections, 81, 82 σ-ﬁeld, 79 space, 79 topology, 99 product lemma, 353, 366, 373 product limit estimator, 507 projection, 105, 195, 201, 449 conditional expectation, 174 mappings πt1 ,...,tk , 90 U-statistic, 450 qf, 111, 123, 145, 147, 231 order-r, 134, 136 standardized class K bounded by K0 , 140 ˜ 119, 123, 140, Winsorized qf K, 141, 143, 145, 147 quadratic form, 191, 369 quantile, 110, 112, 282, 286 median, 210, 280

INDEX random censorship, 505 random sample size, 225, 285 random variable (see also rv), 33 ranks and antiranks, 121, 172, 280, 326 rationals, 169, 289, 300, 362, 410, 476, 514 regression, 194, 378, 463 regularly varying, 145, 419 qf (see also), 137 De Haan, 139 deﬁnition: at inﬁnity, 138 deﬁnition: at zero, 137 Karamata theorem, 137, 139 monotone density theorem, 137 uniformly, 139 representation theorem Kolmogorov for I2 , 402 L´evy–Khinchin, 405, 412 revisualization, 186, 303 rv, 33 existence of, 27, 85, 88, 92, 107 extended, 33 Rademacher, 185, 213, 242 random vector, 84 random element, 90 random process, 90 sample covariance, 455 mean, 120, 280 median, 121, 280 quantiles, 282, 284 space, 20 trimmed mean, 120 truncated mean, 120 variance, 120, 189, 270, 272, 278– 280, 370, 451 Winsorized mean, 120 Winsorized variance, 120 series, 71 three-series theorem, 241, 244 two-series theorem, 241 set theory, 3 λ-system, 9, 153 limsup (liminf), 8, 155, 204 monotone class, 3, 10, 81, 170

583 π ¯ -system, 3, 6, 9, 28, 153, 165, 309 set diﬀerence A \ B, 3 σ-ﬁeld, 3 symmetric diﬀerence A∆B, 3 σ-ﬁeld, 3 µ∗ -measurable sets A∗ , 4 A = σ[C], 12 Aˆµ , the completed σ-ﬁeld, 15 histories, 305, 469 ¯ 24 induced F(X) ≡ X −1 (B), predictable, 513 preservation of, 22 symmetric, 156 tail, 155 skewness γ1 and tail heaviness γ2 , 383, 454 Skorokhod elementary theorem, 52, 114, 404 embedding (see also), 318 theorem, 376, 534, 542 slowly varying, 124, 125, 127, 137, 233, 268, 269, 271, 272, 419 space Borel, 169 Hilbert, 104, 174 inner product, 104 measurable, 3 measure, 4 probability, 20, 33 sample, 20 vector, 28, 104 spectral decomposition, 191, 340 St. Petersburg paradox, 221 stable laws, 407, 409, 410 strictly stable, 409, 411 symmetric, 410 statistics L-statistics, 437, 441 permutation, 428 R-statistics, 280, 329, 426, 462 U-statistics, 449, 456, 481 V-statistics, 457 Stein’s approach to CLTs, 255, 262, 263, 458, 459 Stirling’s formula, 186

584 stopping time, 239, 305, 311, 318, 321, 469, 490, 492 localizing, 511 strong Markov property, 182, 239, 240, 308, 409 subsequences, 55, 60, 288, 403, 533, 538 criterion relating →a.e. to →µ , 31 suﬃcient statistic, 172, 173 symmetrized rv, 210, 213, 219, 244, 372 tail equivalence, 143, 145 CLT, 134 WLLN, 136, 219 theorem absolute continuity of the integral, 42, 54 Arzel` a, 103, 539, 541 Ascoli, 103 Bochner, 363 Carath´eodory extension (see also), 10, 12 Carleman, 293 Chentsov, 538 chf (see also this topic), 346 continuous mapping, 288, 534 convergence implications, 57 convergence of types, 272, 291, 408, 409, 412 correspondence, 18, 20, 27, 77, 85, 107 DCT, 41, 163 de la Vall´ee Poussin, 54, 56 Donsker, 320, 338, 539 Doob, 368, 487, 488, 539 Dynkin’s π-λ, 9 Egorov, 60 Esseen’s lemma (see also chf), 358 Fatou’s lemma, 40, 55, 71, 163, 244, 376, 404 Fubini, 10, 82, 115, 116, 170 fundamental of calculus, 76, 137, 258 Gnedenko–Kolmogorov, 123, 131, 232 Heine–Borel, 17, 19, 96, 110

PROBABILITY FOR STATISTICIANS Helly selection, 288, 291, 293, 539 Helly–Bray, 52, 56, 288, 291, 403, 404 kinder and gentler, 292 Hilbert space, 105, 106 Jordan–Hahn, 62 Karamata, 137, 139 Kolmogorov (see also), 87 Kronecker’s lemma, 205, 217, 221, 253 Lebesgue (see also), 64 Lo`eve, 60 local limit (see also), 380 Lusin, 60 Mann–Wald, 52, 288, 291, 369 Marcinkiewicz–Zygmund, 220, 238 Mason, 329, 332, 336 MCT, 40, 163 mg convergence (see also), 473 moment convergence, 53, 244, 289 monotone class of Halmos, 10, 81, 170 monotone density, 137 only the zero function, 31, 43, 83, 164 P´ olya’s lemma, 289, 291 portmanteau, 290, 532 principal axes, 191 Prohorov, 537, 539 Radon–Nikodym, 66, 77, 158 residue, 344, 345 Riemann–Lebesgue lemma, 355, 359, 381, 392 Riesz, 31 Riesz–Fischer, 51 rth mean convergence, 238 Scheﬀ´e, 60, 380 Skorokhod (see also), 114 Slutsky, 33, 34, 285, 346, 376, 533 smoother realization of a process, 93 Stein’s lemma, 256 Strassen, 235, 323, 542 supporting hyperplane, 46, 50 Taylor, 72, 352 Tonelli, 82

INDEX Ulam, 541 unconscious statistician, 42, 67, 113, 158 uniform absolute continuity, 54 Vitali, 218 Vitali (see also), 55 von Neumann, 176 Weierstrass approximation, 223 tightness, 288, 289, 350, 537, 539 topology, 95 base and subbase, 95 boundary, 95, 100, 290 category, 100 compact, 96, 100, 102 continuity, 98, 100 homeomorphism, 97, 100 isometry, 100 locally compact, 99, 100 net, 96 one-point compactiﬁcation, 99 product, 99 relative, 96, 323, 508 separation, 97, 100 triangular array, 365, 399 trigonometric identities, 340 trimmed, 119 fraction, 120, 417 mean, 119, 416 truncated, 207, 218, 226, 228, 235, 265 mean, 119, 226, 228, 416 type, 109, 291 convergence of types, 291 theorem of types, 272 uan, 283, 326, 377, 399 uncorrelated, 157 uniformity class, 146, 419 upcrossing, 473, 475, 478 Vitali covering, 17, 70 theorem, 55, 60, 475, 479, 498 waiting time, 179, 240 Wald’s identity, 240 Wasserstein distance, 114

585 Wilcoxon statistic, 454 Wilson–Hilferty approximation, 397 Winsorized, 119, 207, 277 fraction, 120 mean, 119, 226–228 moment, 269 outside, 119, 123, 127, 128, 140, 141, 143, 264, 266, 269, 276, 416 variance, 119, 123, 140, 141, 143, 145, 147, 232, 269–272, 276

Probability for Statisticians

Probability for Statisticians

Probability for Statisticians

Probability for Statisticians

Numerical analysis for statisticians

Numerical analysis for statisticians

Numerical analysis for statisticians

Numerical analysis for statisticians

Stereology for Statisticians

Stereology for Statisticians

Numerical Analysis For Statisticians

Numerical Analysis For Statisticians

Statistics for Non-Statisticians

Statistics for Non-Statisticians

Numerical Analysis for Statisticians

Numerical Analysis for Statisticians

Stereology for Statisticians

Stereology for Statisticians

A matrix handbook for statisticians

A matrix handbook for statisticians

Numerical Analysis for Statisticians, Second Edition

Numerical Analysis for Statisticians, Second Edition

Twelve British statisticians

Twelve British statisticians

A Matrix Handbook for Statisticians (Wiley Series in Probability and Statistics)

A Matrix Handbook for Statisticians (Wiley Series in Probability and Statistics)

Elementary Probability for Applications

Elementary Probability for Applications

Probability For Dummies

Probability For Dummies

Elementary Probability for Applications

Elementary Probability for Applications

Quantum probability for probabilists

Quantum probability for probabilists

Probability For Dummies

Probability For Dummies

Probability for Risk Management

Probability for Risk Management

Statistical Thinking for Non-Statisticians in Drug Regulation

Statistical Thinking for Non-Statisticians in Drug Regulation

Nonparametric Statistics for Non-Statisticians: A Step-by-Step Approach

Nonparametric Statistics for Non-Statisticians: A Step-by-Step Approach

Statistical Thinking for Non-Statisticians in Drug Regulation

Statistical Thinking for Non-Statisticians in Drug Regulation

Statistical Thinking for Non-Statisticians in Drug Regulation

Statistical Thinking for Non-Statisticians in Drug Regulation

Asymptotic Approximations for Probability Integrals

Asymptotic Approximations for Probability Integrals

Probability theory for statistical methods

Probability theory for statistical methods

Asymptotic approximations for probability integrals

Asymptotic approximations for probability integrals

Subjective Probability Models for Lifetimes

Subjective Probability Models for Lifetimes

Probability

Probability

Probability

Our partners will collect data and use cookies for ad personalization and measurement. Learn how we and our ad partner Google, collect and use data. Agree & close