ADVANCED TEXTS IN ECONOMETYCS General Editors
C. W. J. Granger
G. E. Mizon
STOCHASTIC LIMIT THEORY An lntroduction for Econometricians
JAMES DAVIDSON
(lxford llniversity Press 1994
Oxjord University Press, Walton Street Oxjord oxa Oxford New York Athens Auckland Banghok Bombay Calcutta Cape Tozvn Dar es Salaam Delhi Florence Hong Kong Istanbul Karachi Kuala Lumpur Madras Madrid Melbourne Mexico City Nairobi Jtzrl Singapore Ttzzc Tohyo Toronto and assoclted companies in Berlin Ibadan ,
6Dp
Published in the Unitei States by Oxford University Pyess l'nc., Nec York
(0 JamesDavidson, 1994 All rlk/lfl reserved. No part of this publication may be repvoduced, stored in a rdfrfcrtz/ system, or transmitted, in any or .',Vtz'rly means, without the yrfor permsion 2'A; zvriting t)./ Oxford Ublbcrlf.y Press. Within the UK, exceptions are allouled p respect of any fair dealing .Jorthe Azlr/t?ycoj research or private study, or crl'li'cl'lpl or reviezv, as permitted under the Co/wrr/, Desns and Patents Act, zq'l', or fr the case V veprographic reproduction in accordance zcff/l the terms V the licences issued by the Copyright Licensing Agency. A'r/tzpzrc: concerntg rcyrotucfpn outside these terms and in other countries should be sent to the Rights Depavtment, Oxjord University Press, at the address above ./brv
F/7l'.book f.ssold subject to the condition that it shall not, by zt?tz.'p oj trade or otherwe, be lent, re-sold, Itived out or otheyulise circulated without the publisher's prior consent s any formof binding (?r covey other than that l'A; which 2'J is published and xithout a similar condition i'ac/z/kg this condhion being imposed on the subsequent purchaser srzk./z Librar.y Ctzftz/twlfag in Publication Data Data available Library
of Congress Cataloging in Publication Data
Data available ISBN 0-19-877402-8 ISBN 0-19-877403-6 ('Pbk) 1 3 5 7 9 10 8 6 4 2 Printed in Great Britain on acid-jkee yl/cr by Biddles Ltd., Guildjord and King
's
Lynn
For Lynette, Julia, and Nicola.
what in me is dark Illumine, what is 1ow raise and support, That, to the height of this great argument, 1 may assert Eternal Providence, And justify the ways of God to men.'
& .
.
.
Paradise f.,t?l/, Book 1, 16-20
Conients Preface
xiii
Mathematical
Symbols and Abbreviations
AXA
Part 1: Mathematics 1. Sets 1.1 1.2 1.3 1.4 1.5 1.6
and Numbers Basic Set Theory Countable Sets The Real Continuum Sequences of Sets Classes of Subsets Sigma Fields
3 8 10 12 13 15
2. Limits and Continuity 2.1 The Topology of the Real Line 2.2 Sequences and Limits 2.3 Functions and Continuity 2.4 Vector Sequences and Functions 2.5 Sequences of Functions 2.6 Summability and Order Relations 2.7 Arrays
20 23 27 29 30 31 33
3. Measure 3.1 Measure Spaces 3.2 The Extension Theorem 3.3 Non-measurability 3.4 Ptoduct Spaces 3.5 Measurable Transfcymations 3.6 Borel Functions
36 40 46 48 50 55
4. lntegration 4.1 Construction of the Integral 4.2 Properties of the lntegral 4.3 Product Measure and Multipl 4 4 The Radon-Nikodym
57 61 64 69
lntepals
'theorem
.''' -' y'
-'
7*
(qd qd 5. Metric
Spaces and Metris Distances 1 5. ) xjj 5.2 Separability and tjmptttt t(((...'.! j y.) .. . r), 5.3 Exampls ' ..)
.....
'
.
...
...
E. .
.........y
.. y. . . . . . E)rt)yy),,:yy't,y:r(,q)'yt 2 .,
i.; ! E E. ... . . . .. ... . ... . q.
. r
. j.
. .. q .
.
...
75 78 82
Contents
5.4 Mappings on Metric Spaces 5.5 Function Spaces
84 87
6. Topology 6.1 Topological Spaces 6.2 Countability and Compactness 6.3 Separation Properties 6.4 Weak Topologies 6.5 The Topolpgy of Product Spaces 6.6 Embedding and Metrization
93 94 97 101 102 105
Part 11: Probability 7. Probability Spaces 7. 1 Probability Measures 7.2 Conditional Probability 7.3 Independence 7.4 Product Spaces
111 113 114 115
8. Random Variables 8. 1 Measures on the Line 8.2 Distribution Functions 8.3 Examples 8.4 Multivariate Distributions 8.5 Independent Random Variables
117 117 122 124 126
9. Expectations 9. 1 Averages and Integrals 9.2 Expectations of Functions of X 9.3 Theorems for the Probabilist's Toolbox 9.4 Multivariate Distributions 9.5 More Theorems for the Toolbox 9.6 Random Variables Depending on a Parameter
128 130 132 135 137 140
10. Conditioning 10.1 Conditioning in Product Measures 10.2 Conditioning on a Sigma Field 10.3 Conditional Expectations 10.4 Some Theorems on Conditional Expectations 10.5 Relationships between Subfields 10.6 Conditional Distributions
143 145 147 149 154 157
11. Characteristic Functions 11.1 The Distribution of Sums of Random Variables 11.2 Complex Numbers
161 162
Contents
11.3 The Theory of Characteristic Functions 11.4 The Inversion Theorem 11.5 The Conditional Characteristic Function
f;r
164 168 171
Part 111:Theory of Stochastic Processes 12. Stochastic Processes 12.1 Basic Ideas and Terminology 12.2 Convergence of Stochastic Sequences 12.3 The Probability Model 12.4 The Consistency Theorem 12.5 Uniform and Limiting Properties 12.6 Unifonn lntegrability
177 178 179 183 186 188
13. Dependence 13.1 Shift Transformations 13.2 Independence and Stationarity 13.3 Invariant Events 13.4 Ergodicity and Mixing 13.5 Subfields and Regularity 13.6 Strong and Uniform Mixing
191 192 195 199 203 206
14. Mixing 14.1 Mixing Sequences of Random Variables 14.2 Mixing Inequalities 14.3 Mixing in Linear Processes 14.4 Sufficient Conditions for Strong and Uniform Mixing
209 211 215 219
15. Martingales 15.1 Sequential Conditioning 15.2 Extensions of the Martingale Concept 15.3 Martingale Convergence 15.4 Convergence and the Conditional Variances 15.5 Martingale Inequalities
229 232 235 238 240
16. Mixingales 16.1 Definition and Examples 16.2 Telescoping Sum Representations 16.3 Maximal Inequalities 16.4 Uniform Square-integrability
247 249 252 257
17. Near-Epoch Dependence 17.1 Definition and zxamples 17.2 Near-Epoc Dependence and Mixingales
261 264
Contents
17.3 Near-Epoch Dependence and Transformations 17.4 Approximability
267 273
Part lV: The Law of Large Numbers 18. Stochastic Convergence 18.1 Almost Sure Convergence 18.2 Convergence in Probability 18.3 Transformations and Convergence 18.4 Convergence in Lp Norm 18.5 Examples 18.6 Laws of Large Numbers
281 284 285 287 288 289
19. Convergence in fp-Norm 19.1 Weak Laws by Mean-square Convergence 19.2 Almost Sure Convergence by the Method of Subsequences 19.3 A Martingale Weak Law 19.4 A Mixingale Weak Law 19.5 Approximable Processes
293 295 298 302 304
20. The 20. 1 20.2 20.3 20.4 20.5 20.6
306 311 313 316 318 323
Strong Law of Large Numbers Technical Tricks for Proving LLNS The Case of Independence Martingale Strong Laws Conditional Variances and Random Weighting Two Strong Laws for Mixingales Near-epoch Dependent and Mixing Processes
21. Uniform Stochastic Convergence 21.1 Stochasti Functions on a Parameter Space 21.2 Pointwise and Unifonn Stochastic Convergence 21.3 Stochastic Equicontinuity 21.4 Generic Uniform Convergence 21.5 Unifonu Laws of Large Numbers
Part V: The Central Lint
327 330 335 337 340
Theorem
22. Weak Convergence of Distributions 22.1 Basic Concepts 22.2 The Skorokhod Representation Theorem 22.3 Weak Convergence and Transformations 22.4 Convergence of Moments and Characteristic Functions 22.5 Criteria for Weak Convergence 22.6 Convergence of Random Sums
347 350 355 357 359 361
Contents
23. The 23. 1 23.2 23.3 23.4 24.
CLTS
24.1 24.2 24.3 24.4 24.5
11
Classical Central Limit Theorem The i.i.d. Case lndependent Heterogeneous Sequences Feller's Theorem and Asymptotic Negligibility The Case of Trending Variances
364 368 373 377
for Dependent Processes A General Convergence Theorem The Martingale Case Stationary Ergodic Sequences The CLT for NED Functions of Mixing Processes Proving the CLT by the Bernstein Blocking Method
380 383 385 386 391
25. Some Extensions 25.1 The CLT with Estimated Normalization 25.2 The CLT with Random Norming 25.3 The Multivariate CLT 25.4 En'or Estimation
399 403 405 407
Pa14 V1: The Functional Central Limit Theorem 26. Weak Convergence in Metric Spaces 26.1 Probability Measures on a Metric Space 26.2 Measures and Expectations 26.3 Weak Convergence 26.4 Metrizing the Space of Measures 26.5 Tightness and Convergence 26.6 Skorokhod's Representation
413 416 418 422 427 431
27. Weak Convergence in a Function Space 27.1 Measures on Function Spaces 27.2 The Space C 11.3 Measures on C 27.4 Brownian Motion 27.5 Weak Convergence on C 27.6 The Functional Central Limit Theorem 27.7 The Multivariate Case
434 437 440 442 447 449 453
28. Cadlag Functions 28.1 The Space D 28.2 Metrizing D 28.3 Billingsley's Vetric 28.4 Measures on D
456 459 461 465
Contents
Xlt
28.5 Prokhorov's Metric 28.6 Compactness and Tightness in D 29.
FCLTS
29.1 29.2 29.3 29.4 29.5
for Dependent Variables The Distribution of Continuous Functions on D Asymptotic Independence The FCLT for NED Functions of Mixing Processes Transfonned Brownian Motion The Multivariate Case
467 469 474 479 481 485 490
30. Weak Convergence to Stochastic Integrals 30.1 Weak Limit Results for Random Functionals 30.2 Stochastic Processes in Continuous Time 30.3 Stochastic Integrals 30.4 Convergence to Stochastic Integrals
496 500 503 509
Notes
517
References
519
lndex
Preface
Recent years have seen a marked increase in the mathematical sophistication of econometricresearch. While thetheory of linearparametric models which fonns the backbone of the subject makes an extensive and clever use of matrix algebra, the statistical prerequisites of this theory are comparatively simple. But now that these models are pretty thoroughly understood, research is concentrated increasingly on the less tractable questions, such as nonlinear and nonparametric estimation and nonstationary data generation processes. The standard econometlics texts are no longer an adequate guide to this new technical literature, and a sound understanding of the probabilistic foundations of the subject is becoming less and less of a luxury. The asymptotic theory traditionally taught to students of econometrics is founded on a small body of classical limit theorems, such as Khinchine's weak law of large numbers and the Lindeberg-Lvy central limit theorem, relevant to the stationary and independent data case. To deal with linear stochastic difference equations, appeal can be made to the results of Mann and Wald (1943a),but even these are rooted in the assumption of independent and identically distributed disturbances. This foundation has become increasingly inadequate to sustain the expanding editke of econometric inference techniques, and recent years have seen a systematic attempt to construct a less restrictive limit theory. Hall and Heyde' s Martingale Limit Fdt??'.y and its Application (1980)is an important landmark, as are a series of papers by econometricians including among others Halbert White, Ronald Gallant, Donald Andrews, and Herman Bierens. This work introduced to the econometrics profession pioneering research into limit theot'y under dependence, done in the preceding decades by probabilists such as J. L. Doob, 1. A. Ibragimov, Patrick Billingsley, Robert Serfling, Murray Rosenblatt, and Donald McLeish. These ltter authors devised various concepts of limited dependence for general nonstationary time series. The concept of a martingale has a long history in probability, but it was primarily Doob's Stochastic Processes (1953)that brought it to prominence as a tool of limit theory. Martingale processes behave like the wealth of a gambler who undertakes a succession of fair bets; the differences of a martingale (the net winnings at each step) are unpredictable from lagged information. Powerful limit theorems are available for martingale difference sequences involving no further restrictions on the dependence of the process. Ibragimov and Rosenblatt respectively defined strong mixing and unbrm mixing as charactermemory' or independence at long range. McLeish defined the izations of notion of a mixingale, the asymptotic counterpart of a martingale difference, becomingunpredictable m steps aheadaspz becomeslarge. This is aweakerproperty than mixing because it involves only low-order moments of the distribution, but mixingales oossess most of those attributes of mixine orocesses needed to make
,
Preface
limit theorems work. Very important from the econometrician's point of view is the property dubbed by Gallant and White (1988)near-epoch dependence from a phrase in one of McLeish's papers, although the idea itself goes back to
Billingsley (1968)andlbragimovtlg6z). Themixingproperty may notbepreserved by transformations of sequences involving an intinite number of lags, but nearepoch dependence is acondition under whichthe outputs of adynamic econometric model can be shown, given some further conditions, to be mixingales when the inputs are mixing. Applications of these results are increasingly in evidence in the econometric literature', Gallant and White's monograph provides an excellent survey of the possibilities. Limit theorems impose restzictions on the amount of dependence between sequence coordinates, and on their marginal distributions. Typically, the probability of outliers must be controlled by requiring the existence of higher-order moments, but there are almost always trade-offs between dependence and moment restrictions, allowing one to buy more of one at the price of less of the other. The fun of proving limit theorems has been to see how far out the envelope of sufficient conditions can be stretched, in one direction or another. To complicate matters, one can get results both by putting limits on the rate of approach to independence (the rate of mixing), and by limiting the type of dependence (the martingale approach), as well as by combining both types of constraint (the mixingale approach). Theresults now available areremarkably powerful,judged by the yardstick of the classical theory. Proofs of necessity are elusive and the limits to the envelope are not yet known with certainty, but they probably lie not too far beyond the currently charted points. Perhaps the major development in time-series econometrics in the 1980s has been the theory of cointegration, and dealing with the distributions of estimators when time series are generated by unit root processes also requires a new type of limit theory. The essential extra ingredient of this theory is the functionalcentral limit theorem (FCLT). The proof of these weak convergence results calls for a limit theory for the space of functions, which throws up some interesting problems which have no connterpart in ordinary probability. These ideas were pioneered by Russian probabilists in the 1950s, notably A. V. Skorokhod and Yu. V. Prokhorov. lt turns out that FCLTS hold under properties generally similar to those for the ordinary CLT (thoughwith a crucial difference), and they can be analysed with the same kind of tools, imposing limitations on dependence and outliers. The probabilistic literature wlch deals with issues of this kind has been seen as accessible to practising econometricians only with difticulty. Few concessions are made to the nonspecialist, and the concerns of probabilists, statisticians, and econometlicians are frequently different. Textbooks on stochastic processes (Cox and Miller 1965 is a distinguished example) often give prominence to topics would regard as fairly specialized (e.g. Markov chains, that econometricians processes in continuous time), while the treatment of important issues like nonstationarity gets tucked away under the heading of advanced or optional material if not omitted altogether. Probability texts are written for students of
mathemtics
and ag&ume
n fnmilinritv
wifh
fln/R
fnllrlnr-
nf
tlao
t.lxlor':
'lqne
Preface
econometricians may lack. The intellectual investmnt required is one that students and practitioners are often, quite reasonably, disinclined to make. It is with issues of this sort in mind that the present book has been written.
The t'irstobjective has been to provide a coherent and unified account of modern asymptotic theory, which can function as both a course text, and as a work of reference. The second has been to provide a grounding in the requisite mathematics and probability theory, making the treatment sufticiently self-contained that even readers with limited mathematical training might make use of it. This is not to say that the material is elementary. Even when the mathematics is mastered, the reasoning can be intricate and demand a degree of patience to absorb. Profs for nearly all the results are provided, but readers should never hesitate to pass over these when they impede progress. The book is also intended to be useful as a reference for students and researchers who only wish to know basic things, like the meaning of technical tenns, and the variety of limit results available. But, that said, it will not have succeeded in its aim unless the reader is sometimes if for no better stimulated to gain a deeper understanding of the material reason, because this is a theory abounding in mathematical elegance, and technical ingenuity which is often dazzling. -
Outline of the Work Part I covers almost a11the mathematics used subsequently. Calculus and matrix algebra are not treated, but in any case there is little of either in the book. Most readers should probably begin by reading Chapters 1 and 2, and perhaps the first sections only of Chapters 3 and 4, noting the definitions and examples but skipping a11 but the briefest proofs initially. These chapters contain some difticult material, which does not a11need to be mastered immediately. Chapters 5 and 6 are strictly required only for Chapter 21 and Part VI, and should be passed over on tirst reading. Nearly everything needed to read the probability literature is covered in these chapters, with perhaps one notable exception the theory of normed spaces. Some treatments in probability use a Hilbel't space framework, but it can be pvoided. The number of pplications exploiting this approach seemed currently too small to justify the added technical overhead, although future developments may require this judgement to be revised. Pm 11 covers what for many readers will be more familiar territory. Chapters 7, 8, and 9 contain essential background t be skimmed or studied in more depth, as appropriate. lt is the collections of inequalities in j9.3 and j9.5 that we will have the most call to refer to subsequently. The content of Chapter 10 is probably less familiar, and is very important. Most readers will want to study this chapter carefully sooner or later. Chapter 11 can be passed over initially, but is needed in conjunction with Part V. In Part 11I the main business of the work begins. Chapter 12 gives an introduction to the main concepts arising in the study of stochastic sequences. Chapters 13 and 14 continue the discussion by reviewing concepts of dependence, and Chapters 15, 16, and 17 deal with s/ecializedclasses of sequence whoseproperties -
7771
Preface
make them amenable to the application of limit theorems. Nearly all readers will want to study Chapters 12, 13, and the earlier sections of 14 and 15 before going further, whereas Chapters 16 and 17 are rather technical and should probably be avoided until the context of these results is understood. In Parts IV and V we arrive at the study of the limit theorems themselves. The aim has been to contrast alternative ways of approaching these problems, and to present a general collection of results ranging from the elementary to the very general. Chapter 18 is devoted to fundamentals, and everyone should read this before going further. Chapter 19 compares classical techniques for proving laws of large nmbers, depending on the existence of second moments, with more modern methods. Although the concept of convergence in probability is adequate in many econometric applications, proofs of strong consistency of estimators are increasingly popular in the econometrics literature, and techniques for dependent processes are considered in Chapter 20. Uniform stochastic convergence is an essential cohcept in the study of econometric estimators, although it has only recently been systematically researched. Chapter 21 contains a synthesis of results that have appeared in print in the last year or two. Part V contrasts the classical central limit theorems for independent processes with the modern results for martingale differences and general dependent processes. Chapter 22 contains the essentials of weak convergence theory for random variables. The treatment is re%onably complete, although one neglected topic, to which much space is devoted in the probability literature, is convergence to stable laws for sequences with infinite variance. This material has found few applications in econometlics to date, although its omission is anothet judgement that may need to be revised in the future. Chapter 23 describes the classic CLTS for independent processes, and Chapter 24 treats modern techniques for dependent, heterogeneous processes. Pal't V1 deals with the functional central limit theorem and related convergence results, including convergence to limits that can be identified with stochastic integrals. A number of new mathematical challenges are presented by this theory, and readers who wish to tackle it seriously will probably want to go back and apply themselves to Chapters 5 and 6 first. Chapter 26 is both the hardest going and the least essential to subsequent developments. It deals with the theory of weak convergence on metric spaces at a greater level of generality than we strictly need, apd is the one section where topological arguments seriously intrude. Almost certa in ly one shoul go first to Chapter 27 ? referring back as needed for definitions and statements of the prerequisite theorems, and pursue the rationale for these results further only as interest dictates. Chapter 28 is likewise a technical prologue to Chapers 29 and 30, and might be skipped over at first reading. The meat of this pal't of the book is in these last two chapters. Results are given on the multivariate invariance principle for heterogeneous dependent processes, paralleling the central limit theorems of Chapter 24. A number of the results in the text are, to the author's knowledge, new. These include 14.13/14, 19.11, 20.18/19, 20.21, 24.6/7/14, 29.14/29.18, and 30.13/14, although some have now appeared in print elsewhere.
Preface
A37II
Further Reading There is a huge range of texts in print covering the relevant mathematics and probability, but the following titles were, for one reason or another, the most frequently consulted in the course of writing this book. T. M. Apostol's Mathematical Analysis (2nd edition) hits just the right note for the basic bread-andbutter results. For more advanced material, Dieudonn's Foundations ofModern Analysis and Royden's Real Analysis are well-known references, the latter being the more user-friendly although the treatment is often fairly concise. Halmos's classic Measure Fdt?ry and Kingman and Taylor's Introduction to Measure and Probability are worth having access to. Willard's General Topology is a clear and well-organized text to put alongside Kelley's classic of the same name. Halmos's Naive Set Fct??'y is a slim volume whose main content falls outside our sphere of interest, but is a good read in its own right. Strongly recommended is Borowski and Borwein's Collins Reference Dictionaty ofMathematics; one can learn more about mathematics in less time by browsing in this little book, and following up the cross references, than by any other method 1 can think of. For a stimulating introduction to metric spaces see MichaelBarnsley's popularFmc/l/. Everywhere. For further reading on probability, one might begin by browsing the slim volume that stmed the whole thing off, Kolmogorov's Foundations of the F/ldtp?-yof Probability. Then, Billingsley's Probability and Measure is an inspiration, both authoritative and highly readable. Breiman's Probability has a refreshingly informal style, and just the right emphasis. Chung's A Course in Probability The value of the Fct??'y is idiosyncratic in parts, but strongly recommended. Probability and Fwr.y Love's edition) Feller's An Introdclassic texts, (4th uction to Probability Fct??-y and its Applications (3rd edition) is self-evident, although these are dense and detailed books that can take a little time and patience to get into, and are chiefly for reference. Cramr's Mathetnatical Methods of Statistics is now old-fashioned, but still useful. Two more recent titles are Shiryayev's Probability, and R. M. Dudley's tough but stimulating Real Analysis and Probqbility. Of the more specialized monographs on stochastic convergence, the following titles (in order of publiation date) are a11 important: Doob, Stochastic The fzzw- of fzzrpd Numbers', Parthasarathy, Probability Processes; Rvsz, Measures on Metric Spaces', Billingsley, Convergence of Probability Measures', losifescu and Theodorescu, Random Processes and Learning; Ibragimov and Linnik, Independent and Stationaly Sequences of Random Variables', Stout, Almost Sure Convergence; Lukacs, Stochastic Convergence; Hall and Heyde, Martingale Limit F/lct??'yand its Application; Pollard, Convetgence ofstochastic Processes; Eberlein and Taqqu (eds.),Dependence in Probability and Statistics. Doob is the founding father of the subject, and his book its O1d Testament. Of the rest, Billingsley's is the most original and influential. Ibragimov and Linnik's essential monograph is now, alas, hard to obtain. The importance of Hall and Heyde was mentioned above. Pollard's book takes up the weak convergence
Preface
story more or less where Billingsley leaves off, and much of the material complements the coverage of the present volume. The Eberlein-rraqqu collection contains up-to-date accounts of mixing theory, covering some related topics outside the range of the present work. The literature on Brownian motion and stochastic integration is extensive, butKaratzas and Shreve's BrownianMotion and Stochastic Calculus is arecent and comprehensive source forreference, and Kopp's Martingales and Stochastic Integrals was found useful at several points. These items receive an individual mention by virtue of being between hard covers. References to the journal literature will be given in context, but it is worth mentioningthatthe fourpapers by DonaldMctaeish, appearing between 1974 and 1977, form possibly the most influential single contribution to our subject. Finally, titles dealing with applications and related contributions include Serfling, Approximation Theorems ofMathentatical Statistics', White, Asymptotic Theotyfor Econometricians', Gallant, Nonlinear Statistical Methods; Gallant and White, A Un6ed Fwr.y of Estimation and Inference for Nonlinear Dynamic Models, Amemiya, Advanced Econometrics. Al1of these are highly recommended for forming a view of what stochastic limit theory is for, and why it matters.
Acknowledgements The idea for this book originated in 1987, in the course of writing a chapter of mathmatical and statistical prerequisites for a projected textbook of econometric theory. The initial, vel'y tentative draft was completed during a stay at the University of California (San Diego) Depmment of Economics in 1988, whose hospitality is gratefully acknowledged. lt has grown a great deal since then, and competing academic commitments as getting itnishedhasinvolvedas%gglewith well as the demands of family life. My family deserve special thanks for their
forbearance.
My colleaguepeterRobinson
has been agreat source of encouragement and help,
and has commented on various drafts. Other people who have read portions of the manuscript and provided invaluable feedback, not least in pointing out my errors, include Getullio Silveira, Robertde Jong, and especially Elizabeth Boardman, who took immense trouble to help me lick the chapters on mathematics into shape. l am also most grateful to Don Andrews, G raham Brightwell, Sgren Johansen, Donald
McLeish, Peter Phillips, Hal White, and a number of anonymous referees for helpful conversations, comments and advice. None of these people is responsible for the various flaws that doubtless remain. The book was written using the ChiWriter 4 technical word processor, and after conversion to Postscript format was produced as camera-ready copy on a HewlettPackard Laserlet 451 printer, direct from the original iiles.I must particularly thank Cay Horstmann, of Horstmann Software Design Corp., for his technical assistance with this task. Lrndon, June 1994
Mathematical Symbols and Abbreviations In the text, the symbol (:a is used to terminate examples and definitions, and also theorems and lemmas unless the proof follows directly. The symbol w tenninates proofs. References to numbered expressions are enclosed in parentheses. References to numbered theorems, examples etc. are given in bold face. References to chapter sections are preceded by j. In statements of theorems, roman numbers (i), (ii), (iii),...are used to indicate the parts of a multi-part result. Lower case letters (a), (b), (c),... are used to itemize the assumptions or conditions specified in a theorem, and also the components of a detinition. The page numbers below refer to fuller definitions or examples of use, as aPpropriate.
absolute value fw-norm Euclidean norm;
1,i -+ D
&
-, / S, 2 <, <(
>
i 1x(.) a.e.
AR
ARMA
a.s., a.s.gpt1
also fineness (of a q/rtition) weak convergence (of measures); also implication monotone convergence convergence almost sure convergence convergence in distribution convergence in Lp nol'm convergence in probability mapping, function composition of mappings equivalence in order of magnitude (of sequencesl; also equiyalence in distribution (of r.v.s) addition modulo 1 set difference partial ordering, inequality strict ordering, strict inequality order of magnitude inequality (sequencesl; also absolutely continuous (measures) mutually singular (of measures) indicator function almost everywhere autoregressive process autoregressive moving average process almost surely, (withresp. to p.m. g)
Symbols AC
I (A)40, Gm
dO V Bnbp) CLT
ch-f. c-d.f. C'p,1j C, Q c D
i Z (n4 dxny) ogc,jy D A :A 6
StlP E.4 E. Ix) E. 1V) eSS
H j+
J-
1
FCLT F(.)
9x(.) 9,,, iff inf
i-i-d. i-o. in pr.
LIE LII
,
lim limsup, lim liminf, 1im Ln) AP-NED MA
m.4
complement of A closure of A intedor of A strong mixing coefficient aleph-nought (cardinalityof (N) dfor every' Binomial distribution with parameters n and p Borel field central limit theorem characteristic function cumulative distribution function continuous functions on the unit interval
set containment strict containment chi-squared distribution with n degreeg of freedom distance between x and y cadlag functions on the unit interval dyadic rationals symmetric difference boundary of A set membership essential supremum expectation conditional expectation (on variable x) conditional expectation (on c-field 5')
Lebesgue measure
3
Symbols m.d. m.g.f. m.s.
martingale difference moment-generating function mean square space of measures
N(jt,c2)
Gaussian distribution with mean jt and variance
s(N()
natural numbers
rh, O
m h #!
0(.) t?(.) Op.) ov.4 p-d.f. P.m.
#(.)
#(.IA)
#(. IN)
H,
1-1
7rj(.)
(D
F.V.
/?(0,1j .f R y R R+
F F+
R& S.e. S.S.e.
SLLN suP Sxvz)
!
Jn
c(C)
c(m fdx
K QJ
(0l
intersection minimum of m and n fBig Oh' order of magnitude relation ,
Oh', strict order of magnitude relation stochastic order relation strict stochastic order relation null set probability density function Ktaittle
probability measure probability conditional probability (on event A) conditional probability (on c-field N) product of numbers', also partition of an interval product measure coordinate projection rational numbers random variable real valued functions on g0,11 relation real line non-negative reals extended real line, R t..p (-x,+x1
R+kJ
f+x)
n-dimensional Euclidean space stochastic equicontinuity strong stochastic equicontinuity strong law of large numbers suptemum e-neighbourhood, sphere sum of random variables variance of Sn c-tield generated by collection C c-field generated by r.v. X
J/Wpo fdF
Lebesgue integral Lebesgue-stieltjes integral
fdp z, E
expected value (integralwith resp. to p.m.) sum
ui
(y2
Symbols
Mlt
shift transformation union unifonn distribution on interval (tz,:4 union of c-fields maximum of m and n weak 1aw of lm'ge numbers with probability 1 sample mean of sequence (XrJZ
F
U
kJ,
Uka,bj
w, V
m V
n
MXLN
sv.p.1
Xn
x, X
Cartesian product
Z
ll l 1T,( ) .
'>O
.
.
ll ll (x1 I,:(l a,b) .
(f1,T)
(f,',p,)
(f1,F,F) (f,T;#) (S,* (X,*
c-field of product sets integers set designation', also sequence, array infinite sequences array largest integer S closed interval bounded by a,b open interval bounded by a,b measurable space measure space complete measure space probability space metric space topological space .t
Common usages A,#,C,D,... X,F,Z,... A',F.Z,...
J,#,,... :,,q
B,5'.I
2,C,O,F',.. 9,V,J't,-.. S,5-,X,... 1,,Y,-
-
d.p T
-
sets random variables random vectors functions positive constants bounding constants collections of subsets c-fields spaces measures metrics topology
1
MATHEMATICS
Sets and Numbers
1. 1 Basic Set Theory A set is any specified collection of objects. ln this book the objects in question are often numbers, but they may also be functions, or other sets, or indeed wholly arbitrary, to be determined by the context in which the theory is applied. ln any analysis there is a set which defines the universe of discourse, containing al1 the objects under consideration, and in what follows, sets denoted A, B etc., are subsets of a set X, with generic element x. q A meaning Set membership is denoted by the symbol belongs to the set writes and A #. The usual way B have the same elements, one A' To show sets A descriptive enclosed define is by in braces, so that for statement a to a set example A tx:x e #) defines membership of A in terms of membership of B, and is an alternative way of writingA #. Another way to denote set membership is by labels. If a set has n elements one can write A fxj, i 1,...,n 1, but any set of labels will do. The statement A lau, (x e C) says that A is the set of elements bearing labels a contained in another set C, called the index set for A. The labels (indices)need not be numbers, and can be any convenient objects at tends to be preferred in all. Sets whose elements are sets (theword this context) are denoted by upper-case script characters. A e C denotes that the set A is in the collection C, or using indices one could write C (Aa: a e Cl B is called a subset of A, written B c A, if al1 the elements of B are also elements of A. If B is a proper subset of A, ruling out B A, the relation is written B c A. The union of A and B is the set whose elements belong to either or both sets, written A k.p #. The union of a collection C, the set of elements belongr ing to one or more A e C, is denoted UA es,A, or, alternatively, one can write UaecAz for the union of the collection fAa: (x l C). The intersection of A and B is the set of elements belonging to both, written A rn #. The intersection of a collection C is the set of elements commop to a1l the sets in C, written f-tlsc/t or OascAa. ln pmicular, the union and intersection of (Aj ,A2, Au ) are written U7=1Ajand O7=1Aj.When the index set is implicit or unspecified we may write just U(xAa, OfAf or similar. The dterence of sets A and #, written A B or by some authors A #, is the of two sets set of elements belonging to A but not to B. The symmetric tft//rcrcc (A B) k.p (B -A). X-A is the complement of A in X, also denoted Ac is A A# when X is understood, and we have the general result that A B 4 rn Bc. The null set (or empty set) is O M, the set with no elements. Sets with no elements in common (havingempty intersection) are called disjoint. A partition of a set is a 61
'
<x
.x
,
=
.
=
=
=
=
=
dcollection'
=
.
=
...,
-
=
-
-
=
=
Mathematics
4
collection of disjoint subsets whose union is the set, such that each of its elements belongs to one and only one member of the collection. Here are the basic rules of set algebra. Unions and intersections obey commutative, associative and distributive laws: A t.p B
=
B QJ A,
A f'n B
=
B rn A,
(A t..p B4 tp C
=
A t.p (B
(A fa #) rn C
=
A f-h CBf'A C),
(1.4)
=
(A fa B4 tp (A rn C),
(1.5)
A fa
(# tp C)
?
C),
(A tp #) r'h(A kp C. There are also rules relating to complements known as de Morgan's laws: A tp
(# fa
C)
=
(A tp B4c
=
(A rhB)c
=
AC
fa
Bc
Ac t.p Bc.
(1.8)
Venn diagrams, illustrated in Fig. 1.1, are a useful device for clarifying relationships between subsets.
Fig. The distributive and de Morgan laws extend to general collections,
1.1 Theorem Let C be a collection of sets, and B a set. Then
(i) (ii)
UA
rn B
OA
tp B
=
AGC
U (A
=
AGC
f-lzt
AeC
#),
r)
(A kp B),
dgC C
(iii)
f-h
AGN
zc
(-Jic,
AEC
as follows.
Sets and Numbers C
(iv)
U
A
OAc. u
=
AeC
AeC
The Cartesian product of two sets A and #, written A x B, is the set of a1l possible ordered pairs of elements, the first taken from A and the second from #; we write A x B (@,y): e A, y G #) For a collection of n sets the Cartesian product is the set of al1 the n-tuples (orderedsets of n elements, with the th element drawn from A/), and is written .:'
=
.
n
Af
.(
=
(x1x2, ,
/=1
...,
xi e Af
Ak):
,
1-
1,...,n )
=
(1.9)
.
If one of the factor sets Af is empty, X2=1Xfis also empty. Product sets are important in a variety of different contexts in mathematics. Some of these are readily appreciated', for example, sets whose elements are s-vectors of real numbers are products of copies of the real line (see j1.3). But prodct sets are also central to the mathematical formalization of the notion of relationship between set elements. Thus: a relation R on a set A is any subset of A xA. If @,y)e #, we usually write x#y. R is said to be reflexive iff xRx, symmetric iff xR y implies yRx, antisymmetric iff xRy and y Rx implies x y, transitive iff xRy and yRz implies xR z, each in where case the indicated condition holds for every x, y, and z e A, as the and only if means case may be. (Note: relation relation equivalence that is reflexive, symmetric, and transiis An a tive. Given an equivalence relation R on A, the equivalence class of an element x e A is the set Ex (y e A: xRy If Ex and Ey are the equivalence classes of elements x and y, then either Ex fa Ey 0, or Ex Fy. The equality relation y is the obvious example of an equivalence relation, but by no means the only one. A partial ordering is any relation that is reflexive, antisymmetric, and transitive. Partial orderings are usually denoted by the symbols f or k, with the understanding that k y is the same as y f x. To every partial ordering there corresponds a strict ordering, defined by the omission of the elements @,x)for a11 x e A. Strict orderings, usually denoted b: < or >, are not reflexive or antisymmetric, but thy are transitive. A set A is said to be linearly ordered by a partial ordering f if one of the relations x < y, x > y, and x y hold forr every pair @,y)e A xA. If there exist elements a e A and b G A such that a < x for al1 x e A, or x S b for all x e A, a and b are called respectively the smallest and largest elements of A. A linearly ordered set A is called well-orderd if every subset of A contains a smallest element. It is of course in sets whose elements are numbers thai the ordering concept is most familiar. Copsider two sets X and F, which can be thought of as representing the universal sets fr a pqir of related problems. The following bundle of definitions contains =
Kiff'
=
.)
.
=
.x
=
.x
=
=
Mathematcs
b
the basic ideas about relationships between the elements of such sets. A (or transformation or function) T: X
F->
tnapping
F
is a rule that associates each element of X with a unique element of F; in other words, for each a7 e X there exists a specified element y e F, denoted F@). X is called the domain of the mapping, and F the codomain. The set Gw
(@,y):
e X, y
.z'
=
is called the graph of T. For T(4)
is called the image of
,4
:4
e /t
Jq
under F, and for B
F -1(#)
=
(1.10)
Xx F
the set
? X,
(F@): x
=
W@))
=
fx: F(x)
e
(1 11)
F
.
1', the set
c
#J c
X
(1.12)
is called the inverse rrlzwe of B under F. The set T(m is called the range of F, and if F(m 1,the mapping is said to be from X onto F, and otherwise, into K If each y is the image of one and only one x e X, so that F(x1) Tx if and only the mapping is said to be one-to-one, or 1-1. if xl =
=
=
.n,
The notions of mapping and graph are really interchangeable, and it is permissable to say that the graph is the mapping, but it is convenient to keep a distinction in mind between the rule and the subset of Xx F which it generates. The term function is usually reserved for cases when the codomain is the set of real numbers (seej1.3). The term correspondence is used for a nlle connecting -1 elements of X to elements of F where the latter are not necessarily unique. T is a con'espondence, but not a mapping unless F is one-to-one. However, the term one-toone correspondence is often used specitically, in certain contexts that will arise below, to refer to a mapping that is both 1. 1 and onto. lf partial orderings are defined on both X and F, a mapping is called order-preserving if F(x1) G Tx iff On the other hand, if X is partially ordered by f, a 1- 1 mapping induces .;q S
.
1.2 Theorem (i) For a collection (Aa c aY), F UAa (:
(ii) for a collection (#a i F), T-' (iii) for #
i', F -1(sc) .
UF(Xa);
=
'
Ufa
=
UF-1(#a);
w-1(s)c. ,
(iv) for A i X, A g F -1(z(u)). #. u (v) for B g F, F(F -1(#)) ,
Here, T -1(B)c means X- F-1(#). Using de Morgan's laws, properties
(ii) and
Sets and Numbers
(iii) are easily extended to the inverse images of intersections and differences', for example, we may show that the inverse images of disjoint sets are also F(A)C F(AC), # in general. Pal'ts (iv) and (v) are disjoint. However, F- F(A) illustrated in Fig. 1.2, where X and i' both correspond to the real line, A and B are intervals of the line, and F is a function of a real variable. =
Fig. 1.2
When F is a 1-1 con-espon dene (1-1 and onto) so is F-1 These properties then hold symmetrically, and the inclusion relations of parts (iv)and (v) also become X and B c K lf Z is a third set, and equalities for a1l A .
U1 F
F--> Z
is a further mapping, UoF: X
F--
the composite
mapping
Z
takes each x e X to the element U(F(x)) e Z. UoT operates as a simple transform-1 from X to Z, and 1.2 applies to this case. For C c Z, (&J07) c' ation -1 =
-1
F
(&
(C)).
0- x E .
j
i
.
.
.
.
.
'
.
' .
.
' .
' .
' .
.
j
F(z4c )
1
+
'
.
)Y
X1Z
Fig. 1.3
1.3 Example lf X 0- x E is a product space, having as elements the ordered pairs .x = (0,(), the mapping F: 0- x E i- E =
,
Mathematics
8
defined by F(0,() (, is called the projection tnapping onto E. The projection of 0- x E onto E (respectively,0-) is the set consisting of the second a set A (resp., first) members of each pair in A. On the other hand, for a set B e E, 0- x # It is a useful exercise to verify 1.2 for this case, and also to F -1 B4 T(AC) in general. In Fig. 1.3, 0- and E are line segments and check that F(A # F(A)C is the union of the indicated line 0- x E is a rectangle in the plane. Here, whereas T(A9 E. u segments, =
=
.
=
The number of elements contained in a set is called the cardinality or cardinal in this context is not a primitive one, number of the set. The notion of but can be reduced to fundamentals by what is called the principle. A set A is said to be equipotent with a set B if there exists a 1-1 correspondence connectingA and #. Think in terms of taking an element from each set and placing the pair in a pigeon-hole. Equipotency means that such a procedure can never exhust one set before the other. Now, think of the number 0 as being just a name for the null set, 0. Let the number 1 be the name for the set that has a single element, the number 0. Let 2 denote the set whoseelements are the numbers 0 and 1.And proceeding recursively, 1et n be the name for the set (0,...,n 1 1. Then, the statement that a set A has n elements, or has cardinal number n, can be interpreted to mean that A is equipotent with the set n. The set of natural numbers, denoted N, is the collection (n: n = 1,2,3,... 1. This collection is well ordered by the relation usually denoted <, where n S m actually means the same as n m under this definition of a number. %number'
dpigeon-hole'
-
1.2 Countable Sets Set theory is trivial when th: number of elements in the set is finite, but formalization becomes indispensable for dealing with sets having an intinite number of elements. The set of natural numbers (Nis a case in point. If n is a member so is n + 1, and this is true for every n. None the less, a cardinal number is formally assigned to iN, and is represented by the symbol Mtl When the elements of an infinite set can be put into a one-to-one correspondence with the natural numbers, the set is said to have cardinal number d(), but, more commonly, to be countable, or equivalently, denumerable. Countability of a set requires that a scheme can be devised for labelling each element with a unique element of (N.This imposes a well-ordering on the elements, such that there is a 'first' element labelled 1, and so on, although this ordering may have significance or be arbitrary, depending on the circumstances. It is the pigeon-hole principle that matters here, that each element has its own unique label. With infinite sets, everyday notions of size and quantity tend to break down. Augmenting the natural numbers by the number 0 defines the set (Ntl (0,1,2,3,... 1. The commonplace observation that (Ntlhas more' element than ENis contradicted by the fact that INand (Nnare equipotent (labeln 1 e Ntlby n e IN).Still more suprisingly, the set of even numbers, E = (2n, n e IN) also has an obvious labelling scheme demonstrating equipotency with EN.The nave idea that there are (Kaleph-nought').
=
-
,
df.xxzlr.o
eac
meanxz'
xlomontc
ln
N
re
1n IF- ;t
Axzithnllf
lfatyinnl fnllnclntlnn
Sets and Numbers
9
infinite subset A of ENhas a natural well-ordering and is equipotent with EN itself, the label of an element x e A being the cardinal number of the set (y e A:
A' V A l than EN consider the integers, I Turning to sets apparently 1,2,... ), the set containing the signed whole numbers and zero. These ( are linearly ordered although not well ordered. They can, however, be paired with scheme: the natural numbers using the .
Elarger'
=
,
...,-1,0,
Ezig-zag'
(1,0),(2,1),(3,-1), (4,2),...,(n, (n/21(-1)''),...,
where gxJdenotes the largest whole number below x. Thus, N and I are equipotent.
Then there are the rational (I)
=
f.x: .x
numbers.
=
alb, a e 1, b e 1, b # 0)
(1.13)
.
We can also show the following.
1.4 Theorem Q is a countable set. Proof We construct a 1-l correspondence between I x J and N. A 1-l correspondence between I x I and I x: is obtained by the methodjust used to show I countable, and one between I x: and ENx: is got by the same method. Then note that the number 2 3b e INis uniquely associated with each pair a,b4 e INxN ; the rule for recovering a and b from 2'3b is g et a as the number of divisions by 2 required to get an odd number, and the number so obtained is 3 ' The collection f 2 '3b : a e ENb G EN) c ENis equipotent with ENitself as shown in the preceding paragraph. The composition of a1l these mappings is the desired correspondence. w <
.
,
,
Generalizing this type of argument leads to the following fundamental result.
1.5 Theorem The union of a countable collection of countable sets is a countable set. n The concept of a sequence is fundamental to a11 the topics in this book. A sequence can be thought of as a mapping whose domain is a wellrordered countable set, the index set. Sinc there is always an order-preserving 1-1 mapping from IN to the index set, there is usually no loss of generality in considering the composite mapping and thinking of N itself as the domain. Another way to characterize a sequence is as the graph of the mapping, that is, a countable collection of pairs having the ordering conferred on it by the elements of the domain. The ranges of the sequences we consider below typically contain either sets or real numbers', the associated theory for these cases is to be found respectively in j1.4 and j2.2. The term sequence may also be applied to mappings having I or another linearly ordered set as index set. This usage broadens the notion, since while such sets can be re-indexed by N (see above) this cannot be done while preserving the original ordering.
Mathematics
10 1.3 The Real Continuum
The real-number continuum R is such a complex object that no single statement of definition can do itjustice. One can emphasize the ordinal and arithmetic properties of the reals, or their geometrical interpretation as the distances of points on a line from the origin (the point zero). But from a set-theoretic point of view, the essentials are captured by defining R as the set of countably infinite sequences of decimal digits, having a decimal point inserted at exactly one position in the sequence, and possibly preceded by a minus sign. Thus, the real number x can be written in the form X
'x
,,,(x)1(/)7-)j(.x)10-f
=
(1. 14)
i=1
where the sequence (J1(.x), J2@),...) consists of decimal digits (elementsof the set (0,1,...,9 J), px) e (Ntldenotes the position of the decimal point in the +1 if k 0, and otherwise (the string (the decimal exponent), and mlx) all of the decimal expansion sign). When dix) 0 for but a finite number terms, of x is said to terminate and the final Os are conventionally omitted from the representation. -1
.x
=
=
by (1.14) is not always unique, and there exists a 1-1 correspondence between elements of R and sequences (>,#,#1,#2,#a,... l only after eliminate arbitrary of leading zeros we must the latter are excluded. To certain 0. And since for example, 0.49999... (the stipulate that dj y: 0 unless p sequence of 9s not terminating) is the same number as 0.5, we always take the terminating representation of a number and exclude sequences having di 9 in al1 but a finite number of places. R is of course linearly ordered, and in terms of (1. 14) the ordering corresponds to the lexicographic ordering of the sequences The representation
of
.x
=
=
fmsmpnmdk,mdznmdL...l.
The choice of base 10 in the definition is of course merely conventional, and of the alternative possibilities, the most important is the binary (base2) represn-
tation,
x
=
>(x)
2d*J>7n(x)2-i f=l
,
(1.15)
where the bi are binary digits, either 0 or 1, and qx) is the binary exponent.
The integers have the representation in (1.14) with the strings terminating elements rationals of R, being those which either after px) digits. The are also terminate after a finite number of places, or else cycle repeatedly through a finite sequence of digits beyond a certain tinite point. The real numbers that are not rational are called irrational. The irrational numbers are overwhelmingly more numerous than the rationals, representing a higher order of infinity. The following is the famous argument of Cantor. ddiagonal'
1.6 Theorem The jet R is uncountable. Proof Assume a 1-1 correspondence between R and ENexists. Now construct a real
Sets and Numbers
number in the following way. Let the first digit be different from that of the real number labelled 1, the second digit be different from that of the real number labelled 2, and in general the nth digit be different from that of the real number labelled n, for every n. This number is different from every member of the labelled collection, and hence it has no label. Since this construction can be performed for any labelling scheme, the assumption is contradicted. w c, where c is the cardinal
number
of R The linear ordering on R is of interest to us chiefly for providing the basis for constnlcting the fundamental subsets of R, the intervals. The set A (.x:a < x < &) is called an open interval, since it does not contain the end points, whereas the interval # = (x: a f x S /71is said to be closed. Common notations are gJ,pj, asb), (tz,:j and Lanb) to denote closed, open and half-open intervals. A set containing just a single point a is called a singleton, written ftz) Unbounded defined by a single boundary point, are written intervals such as C = (x: a < (J,+x), (-x,:), and (J,+x), (-x,:j for the open and closed cases respectively, where the infinities +x and are the fictional (notelements of R) with > +x < for al1 R respective and An important example properties x the xe is the positive half-line (0,+x), which we will denote subsequently by R+.
We say that d ()
<
.
=
.
.x1,
Kpoints'
-=
.r
-x,
.
1.7 Theorem Every open intelwal is uncountable. Proof Let the interval in question be anb). lf a < b, there exists n k 0 such that the n + llth term of the sequence msmpntndjsmdzn...) in the expansion of (1.14) defining b exceeds that in the corresponding sequence for a, whereas the t'irst n digits of each sequence are the same. The elements of a,b4 are those reals whose expansions generate the same initial sequence, with the (n+ llth terms not exceeding that of b nor being exceeded by that of a. If a and b are distinct, n is finite. The result follows on applying the diagonal argument in 1.6 tb these expansions, beginning at position n + 2. w Other useful results concerning
R and its intervals include the follpwing.
1.8 Theorem The points of any open interval are equipotent
with R
.
Proof This might be proved by elaborating the argument of 1.7, but it is simpler just to exhibit a 1-1 mapping from R onto asb). For example, the function
y
=
a+ b
2
for x e R fulfils the requirement. 1 9 Theorem The reai plane R2 *
Proof In
+
(b
2(1
-
+
tzlx
(1.16)
IxI) .+
. =
R x R is equipotent with R
,
view of the last theorem, it suffices to show that the unit interval o Givep points x, y e g0,1J, define 1q (0, is equipotent with the unit square (0,11. the point z e (0,1j according to the decimal expansion in (1.14),by the rule
Mathematics
diz)
di-vblllx),i odd =
Jf/2@),
i
1,2,3,...
=
(1.17)
i even
In words, construct z by taking a digit from x and y alternately. Such a z exists for every pair x and y, and, given z, x and y can be uniquely recovered by setting
(1.18) This defines a 1-1 mapping from (0,112 onto gO 1j as required. w ,
,
This argument
can be extended from R 2 to Rn for any n e N.
1.10 Theorem
Every open intelwal contains a rational number.
,
Proof This is equivalent to the proposition that if x < y, there exists with x < r < y. First suppose x 0. Choose q as the smallest integer such that qy > qx + 1, and choose p as the smallest integer 1/(v qy. Then x < (,p 1)/ < y. For the case < 0 choose an integer n > < r < n +y, .x < r- n < y, where r is the rational satisfying n -x),
-x,
.x
-
.-x
above.
rational r
exceeding
exceeding and then found as
.
1.11 Corollary
Every collection
of disjoint open intervals is countable.
Proof Since each open interval contains a rational appearing in no other interval disjoint with it, a set of disjoint open intervals can be placed in 1-1 correspondence with a subset of the rationals. w The supremum of a set A c R, when it exists, is the smallest number y such that x S y for every x e A, written supA. The inpmum of A, when it exists, is the largest number y such that k y for every x e A, written inf A. These may or may a, and supgl,') not be elements of A. In particular, infgtz,hj inftc,#) suptc,:) b. Open intervals do notpossess largest or smallest elements. However, every subset of R which is bounded above (resp.below) has a supremum (resp. intimum). While unbounded sets in R lack suprema and/or infima, itis customary to called the extended real line. In V, every set has define the set V R t.p ( number or +x, and similarly, every set has an either real a tinite a supremum, infimum. The notation 2-+will also be used subsequently to denote R+ k.p (+x). .x
=
=
=
=
-x,+x),
=
l.4 Sequences of Sets will be written, variously, as (An'. n e IN), (A,,)7, Set sequences fA1,A2,A3, or just fAnl when the context is clear. with each member of A monotone sequence is one that is either non-decreasing, n), being contained in its (An the sequence successor c An+l, V or non-increasing, with each member containing its successor (An+1i An, V n4. We also speak of increasing (resp. decreasing) sequences when the inclusion is strict, with c (resp. D) replacing c (resp.Q). For a non-decreasing sequence, define the set A ...)
=
U';=1A,,and for a non-increasing
sequence the set A
=
fV=IA,;
=
(U';=1Aj)f. These
Sets and Numbers
13 '1'
sets are called the limits of the respective sequences. We may write An A and d,, A. 1 A, and also, in general, limrl-yx,As zt and A,, --->
=
1.12 Example The sequence ((0, 1/n1, n e (Nl is decreasing and has as limit the singleton f0). In fact, limn-+x(0,1/n) = (0) also, whereas 1im,,-4x(0,1/n) 0. The decreasing sequence of open intervals, ((tz 1/n, b + 1/n), n e EN), has as its (J, :1. On the other hand, the sequence ((tz+ 1/n, limit the closed interval its limit is b4. a, a b 1/nj, n e IN) is increasing, and =
-
-
is non-increasConsider an arbitrary sequence (zt,,1.The sequence Bn U,a'='=,#4,, = exists. superior is called limit of the This set the ing, so that B limn-oon (Anl, written limsupnAn, and also as limnAa. Similarly, the limit of sequence the non-decreasing sequence Cn F1e;=,,A,,,is called the inferior limit of the written liminfndn, or limnAr,. Formally: for a sequence (A,,, n e EN ), =
=
sequence,
oo
limsupz
O
=
n=1
n
liminfAn
U n=1
=
Cr
UA,s
(1.19)
m=n
OA,u1 .
m=n
/
(1.20)
The limsup is the set of De Morgan's laws imply that liminfnAn = (limsuwzvlf. elementscontained in infinitely rrltwz of the An, while the liminf is the set belongingto all but a Jl/c number of the An, that is, to every member of the sequence from some point onwards. These concepts provide us with a criterion for convergence of set sequences in genergl. LiminfnAs ? limsupnAn, and if these two sets differ, there are elements that belong to infinitely many of the An, but also do not belong to infinitely many of them. Such a sequence is not convergent. On the other hand, if liminfuA,l = limsupnAn A, the elements of A belong to infinitely many of the An and do not belong to at most a finite number of them. Then the sequence (An) is said to converge to A, and A is called the limit of the sequence. =
1.5 Classes of Subsets The set of a11 the subsets of X is called the power set of X, dnoted 2X The power set of a set with n elements has 2 elements, which accounts for its name and representation. ln the case of a countable set, the power set is thought of fprmally as having 2 20 e le ments. One of the fundamental facts of set theory is tht the number of subsets of a given set strictly exceeds the number of its elments. For finite sets this is obvious, but when extended to countable sets it M to the claim that 2 g > do. ?mntj .
n
1 13 Theorem
.'
')'. ()' td
'' -'
.
Prof '
. . .
.. .
q
E
..'
..'
.
.(
;
))t () it)(lit'(. . . . . '.. .:
'
.
c.
.
.
.
' '
' .
,'
i
(..
=
The proposition is proved if we can show that 2 is equipotent with R or, qivalently (in view of 1.8), with the unit interval (0,1j. For a set A G 2 x (.2.1liitsftqt . th sequence of binat'y digits (:1,hz,,... ) according to the rule,
E . ..
.
2B0
:
. . . ... .. .. . .
-.
.
,
Lbn
Mathematics
14
= 1 if n e 4, bn 0 otherwise'. Using formula (1.15)with m 1 ad q 0, let this sequence detine an element XA of (0,1)(the case where bn 1 for a1l n defines 1). On the other hand, for any element x e (0,1q, construct the set Ax G 2N according to the rule, n in Ax if and only if the nth digit in the binal'y expansion of is a 1'. These constructions define a 1-1 correspondence between 2 ENan d (0 11 w =
=
=
=
'include
.x
,
.
When studying the subsets of a given set, particularly their measure-theoretic properties, the power set is often too big for anything very interesting or useful to be said about it. The idea behind the following detinitions is to specify subX sets of 2 t h a t are large enough to be interesting, but whose characteristics may be more tractable. We typically do this by choosing a base collection of sets with known properties, and then specifying certain operations for creating new sets from existing ones. These operations permit an interesting diversity of class members to be generated, but important properties of the sets may be deduced from those of the base collection, as the following examples show.
1.14 Definition A ring S is a nonempty class of subsets of X satisfying (a) O q S. (b) lf A and B e R then A wB e S, A (AB G R and A B e X. n -
One generates a ring by specifying an arbitrary basic collection C, which must include 0, and then declaring that any sets that can be generated by the specified operations also belong to the class. A ring is said to be closed under the operations of union, intersection and difference. Rings lack a crucial piece of structure, for there is no requirement fclr the set X itself to be a member. lf X is included, a ring becomes aheld, or synonymously AC, this amounts to including all complements, and, in an algebra. Since X-A view of the de Morgan laws, specifying the inclusion of intersections and differences becomes redundant. =
1.15 Dennition A tield 5 is a class of subsets of X satisfying (a) X e 5. (b) lf A 6 5 then Ac 6 T. (c) If A and B G 5 then A t..p # e @. n A field is said to be closed under complementation and finite union, and hence under intersections and differences too; none of these operations can take one outside the class. These classes can be very complex, and also very trivial. The simplest case of a ring is (OJ. The smallest possible t'ield is tX,O). Scarcely less trivial is the AC, O ) where A is any subset of X. What makes any class of sets field (-Y,A, interesting, or not, is the collection C of sets it is declared to contain, which for the class. We speak of the smallest tield we can think of as the generated by C'. field containing C as Rings and fields are natural classes in the sense of being defined in terms of the simple set operations, but their structure is rather restrictive for some of '
,
tseed'
tthe
Sets and Numbers
15
the applications in probability. More inclusive definitions, carefully tailored to include some important cases, are as follows. ,/
is a non-empty class of subsets of X satisfying 1.16 Definition A semi-ring (a) O e $. then A /'n B e (b) If A, B e where (c) If A, B e and A i #, H n < cxa such that B A U7=1G', e = f''h for each j'. O j, n and .f.
,/
,/
./
=
-
.
.'
.
More succinctly, condition partition into '-sets.
that the difference of two '-sets has a finite
(c) says ,/
is a class of subsets 1.17 Dennition A semi-algebra (a) X e (b) If A, B e then A chB e (c) If A e $, 3 n < x such that Ac Uyzlcj,where for each j, j'. a
of X satisfying
.f.
.l'.
,/
=
.
e
.t/?
and C) ch Cj'
O
=
A semi-ring containing X is a semi-algebra. R, and consider the class of all the half-open intervals and Iz < together with the empty set. If h (J1,:1) for I (tz,!?J (J2,:1q, f-'h and 0. And if fl = (tza,:2),then fl h is. one of h, h, (tz1,:2), c 11 (tz2,J11, (:1,:2j, so that a) f tz1 and bj f b1, then h h is one of 3, semi-ritzg conditions defining therefore and The sat(az,atj t.p (:1,&2J, h. a are isfied, although not those detining a ring. If we now let R be a member of the class and follow 1.17, we find that the halfand (tz,+x),plus open iniervals, plus the unbounded intervals of the form In O artd R, constitute a semi-algebra.
1-18 Example
Let X
=
=
a S b
<
-n
+C,o,
=
-
-xtbl
1.6 ji gTa Fields .
. .. .
.
.
Whn we' say tht a field contains the complements ;;!,' t!)'--,$@. ,;' y' .j' r' ;'(k,-':' ..,' iip'.l!:d @'. j,' jr'r' (@.-)--81'* :)'. ):'. '''#' r', jjli'. yrd jd r'. ;d 11,;::2:r11.-*. :-'. i'-' t:'.-'-ll)ll'-' qrrrdkiy--i---.d-,jtls'ltdkdi.lhlikirrstipp.:fl.rd pFq'. yy.;'. ,;r'j-. y' yjd ()..' jt'. ijd k' j'yy.'-r)'.' 133*.* j'i'.i (,'. ;);.;-' ;'@@' r' (y'. j.' )' i@'. j..' t'' )'()' y'; g' q'd ('. (';' kt'qi'. ''Ikiii!r:-'. ((' t,,(,t'' ?' L' ..' yjjjjjyryjjj.,,d :::*-2)1-.* Ft@-'. !!)p'.(' py'i'l'g )tI-'. ytd yy..,';.-: y;j-yy;',. (';' .d 4j' ('. .LLLq'. .:!1211-----*:11... :I.EIII.-.CEII-.'. 'Iii!'.k.d ''Ikii:rr:-d q!IkE7'r!'k..'. .-'Iki:t:i;'.. 'q'IlkEjIIk,'. (.'. ,''-r.-'II::7!IIk... I'EEEII.'. ijyipl,.dtsil----'k-ljptjs..!rrl ,-jb'L)j-..L'tjLkL33,?.,.q-',i-..-j,qk.. t(' .y)'. tjd t'j,'t)' .'')'''';:E:!!.,-. ''k::E:!:.' k.-'-'''Ill::iEl1I-..t.''-.tt'$'12ii!I.-.'. rpr'y r' .-'. r'.' .-' '1I(ii''' (!11:::1,-* 4111E::::,..*..':!1L.*. it:ipd rjqj-dltdllt.-l'ttd desyrves explanation. It is clear that z4l
and finite unions, the qualifier A,, e 5 implies that U& j=3 A) .#hit lteratlon of slmple by n-fold 5 But, the constructlv: palrwlse unlon. e glven t t: (j jm. tj (jj'I Out W 1* legiti 1 k. tju further 1 mate ttlr th t ti t no e e s on q':j).: .) . ' a s P ul ati O n to r o t? 41@.;: )' .(ir.. ytt,rtlh be taken llmlt. a.,g, the By maklng n.operatlon can thls addltlonal to ))(, j).jt (..... .. . . . 'lk::Ei!lk.. @ E..t!1IIp:,I.i.. 'r '. E. !!I1L., :111:7:!1.-. ' ' ' :..-t.'E!IIk..' . .-'.EEE:!!:. ..,.jy-...'- ..:..'.. '. ' .. . :111:7:!1.-.. r,,;-. Jl'... ' ' ' '..-.. .' ' ''. . rr ... '.r.-'...(. . . .'.'i.7tr. i'.' ..luju,-..t9'.j:;:,.j)..jr, r.... f.tltz... jyy;;.jjy..... jj.lyytjr .. .. ...... .. . ;.g,..yt:jj,j.. . ..*1.: j., A .'.j'tp:('.()) r yjj''.'gt). (tsajgebra)5 ls a class Of subsets Of X satlsfylng ('. f l ...;...: uljjjtl .......,..y 't;. r;J) .j . j ?)...j!.y. -y..F..:yqqy t)?.p:. (y .kly(..yy.yq..:)...q ):;1211E11.... .!kjjj;;r;;)., .. ' ..
.' .'.' ':jls')d .'. '-' -j--!,r,!tk'. -:'. --y!j;;'-j-.' -q' -,.'tp..-.(t. -'k' -(r)'. -8)* y' :')' 24jg* -'. -' j;d ('. !111:(q*;LL.' ;..(' i.'. k' .-'. -'-' --'' -,' j'y' -:tr!)i...ti:,jp':F'--)?;t--''' E' ,-.'itt-,ttd ,'-'.k'(.E ;E-;r'.' ?' y' lrrq))d :':11,*. :L';'' ''. g')j;-',' j';'y' y't':. i--d r' s.d ,'g'' E'. )-''E -;.' ''IItEL.FX -' -,'.-y' -'. .)' .q' ''. k' .:' ,''(''jy ;'))'. ,'..(.. (t' ?.' :y'-' ;;' yyd ((' ;'. :('. F'-j. !': q' (j.,' F' C--qy;'E :'-'E' :)')' L3--'; :;' ''-.' ;,'-'L'' ,-'' )'
,-,' -' .','.:'.. -' .','.-tq'-' -,-' .)r,' ,--))'-' ,,,' 'jjjj:yr:jjj,'; .'.'. ''.-'' .!112::*. .'. ,'.'-'. 4:11::.* '''bii'''bi''-''
'.'. .''''--d .'.' .' . .'r:).. ... ''.' '..'. '. ..' . .' . .
'.
:
...
.
.
....
. E- .
(. '
... .
.
.. ' g' '
.
q
(
.'' . .' '( . . EE
...
.
. .. . . .. . . (j;) :.. . . . :
.
.
E
t
,np
E
.
:
.'
.
'!;'j .L'
'. 7 .!L(.
...;
j, ....j .; ,4k .:f .(a z ( .
.
.
. -
-.
..
....'
j.(
...:;.. .
...
...
-
.
(
.
....
'
.'
.. .
g
...
.
7
.
..
.. .
'
.
.
..
..i
..k
. .
.
.:.
.
..
. . ' .
.
.
.
..
dp
4)
..
-11::::::,11;-.
.11:2:::,...
,11::2::,1.-
.
:.. .
.
:
,
. .
y.
.
.
......'
'lk:!r:-.
,y
..
-
. .. .,k;)jjjtyy........
(lzzllp'l. ' (jj:jjj.. (j;jj..,...
.
y.
.
.
.
.
,,
.
,,
,,
7.
u.....r.
:..,. . .: . ... . ;. . ..;.,k y . (. .;..LtEL. :. . , : ry q..;..y.... jjggggyy;: . .. .-'E . .. .y E E. CE.; . . ;.... : .)Ey. -... . r :. . .E . ---i .. . ,. q .. ..E . . .q ,. jy. ..g .y.. EitttliyitztlrE . . rtjyEiljzqttigryyf ,.. . q - yy q . j . .. . . q . y . . .. j, . . . . koyyyyyyjgy . y. . . . . . yyyjj yy. . yju r . ' i EE yu y . E !' '(' :. q: i E.'rit:-. g l.E; :);y.. g yy g ().. y . .. g y y: .. jy q; ; L y. . . r Lj.,L..LjL; I.t: jL'L'q(jLL'; y. '.t;tEqq;;' gE.y))gy .. y j, . . .... .. ... , . . . . . . ....yy .q :.q( .;).(,LL.;..;.. . . . y y. r.@'ry'.'y..r?i..E.!:..... kjjjyj .;. rykt',.E..yi. ....q .. y.; . y.p; y. . . .'. .)L;jt..).q(y,jj,d!!1i-,; :.( ..E.. -y ....) ( ?.jj,.. y-1... y ' ' E i i. E - rFti... t E .j;. ;!.. E (! . !- E . jygy. . - ...;:.. E..- . E. . (E .. E ! .... . : . .;.. .kyk. . EE t'F-i''' - q (. . -. . . ...rj.. . -: -. -, . kL. . qE. i . -; -y.,.-.g.1.j..-..d..,)....y.;,..y...r.-y.-.st$y-. .. y . .;.. -. .y '. EE.. ; . t; . . . ,.f33f. .E E . . .. .- y . -. ':$l8t,E-.k;#)),pl-(i'.-.'j,b.slb . .....p.-.. .'. --..'-... -.pt -y. ,.)ki-f- --.--; #..... .. . ,,,gt;:t!jj:jjj.j .
dp
:1
.
.
.
..
.
.
q . ..
..
..y).
...
-
41
...
..:..:.
.i'.
1p
...t..
.
...;;.;,,
.
.
. .
...y;'y'y...y(iy(y)(yy:jj..yy..:y(j....
.:jy.yj...
2
.
.
:
.
.'
<.
.
.
.2.
.
...'
....
..r
:';'
'
.
.
'
.
......
:
...
.
..
.
,
-
.q.(
'rt,ywpy
..
. .
...
-
EyEi yt($.?E.r(r . .. ,E 'E. i.E k'JE C4jIIE. ..L 3.k() LLj;L.L .(:.g).-. E ;i.EE-)!'(; . ttjii;. )....y.2..2...,..J): :7.:..
E..y.(.
.
: :E..
LjL.L..kL /. ..
..
.
.
-:--
-. - . ..j .. .j(;E ;.; . (y.jjj.(q.. sl/tq E .
..
. .
'
.E.. . .. j q. ; E E/E.Lq ..L.. . . ... . . ) .. --(j ( . . --. ' . i . -.' --- . -.. . . - . E. .-.. . . -. -:--'. -. ..' ' ... g.. t):;.... ..q/.:I.jLL . .. : g (! .... j ...:k..j..g . . LLL.. ;.r .y y.. . y. gy y . .:yyy ... ' bs? qoujyyjd . ;
...:
.
,
o
.
.
E
.:
...
...
'
'
,.
.
:!)
.:
.
'
; . .j . .. . . ... . g . .. 4 . '-'112::..
iE: ':'E: . -.' ..;E.) '.'.E
...;.
.
. : f . !..p ' ... L. ' '. . L...'...2.r .' ' ;''. ' . ( . (.' ,u..km,,,,o,k,,,.,r+y,,,.,,a, .
.
-
E.'.p.:
)Eq
.
.
: .
.'
.. .
...,
..
.
. .
.
..
.
.
.
. .
.
.r.. . g ... j . ., . ...... . . ... .4: j y. .yyjaj . ts. )j; . j .. jjjj:jjj'. !!!' i.$. jys,lt. . k.. i.. . .ty jyy )- jjy . yyrtyj g '#:'l:'j':.j'.jr:!.k::: .. .yy . yyy gc y i: . . l. .r.)(k!r/.:i..tt'.j:r. .tr(g.(..). r. i)'i;r... yt .tk-. . ttr;: .;... ryki j yy y y. t. r..r) ti).;7..i-tl ... j;... t). . ryli l.t:.. ,:,.,:,, ;)).y;..)-k.t8.l9)-., gy'tt..kq.j4:yy.-.-..,.it::!:j-.,.. L,tjtLL r,.. . 442!2:r ,-L..).;;;.)-L66LLj6q?''' jj:rljjs 41222:; jjzkr:i 44:22,), .(::.-tgy..tr..;y(jtty . ...E .-..(jl. 'l.s' 4(222:, 4(22::; jjjj;, rjrr, ,...EEEEEE. 4k22::; ..j'tj'L. .1k.k,j. (jlr:jl. jljrhhhjj :jjjgCIE1j,. :jjji rjjjq- :jjj;, (jrr:lk. :.1-:.9. (jrrjl. :;:;::::;' ; '. ).. i-.ql..?-' 4(4E ,, IILLL,.:LL..).L (jj j(.@i. .-. .:y. i.-. t; t. p ;. )i. . , E. .. t. El(;. jr. .. r ; ;. tgg j)rpi j-. :18-. (..?- rjq. jyy.i . j-y . t-. ;. .-. j.. . ))-. . . '.i . r. . . .. r,r .' ty. (y;j--.;.kr. . . . ;. .. . . . .. . (. yj).r'!jz'.,..,jp'.'..-''.. . . -,y.,4tE:1'. :lir4:::: 4t::. 1. qlL y.-y1,y-, 4:::,. ....,....,.)s,j.,, 112). 4:2:: 4::2.::;7. ;p';!r 4:::). 1ki2: :5'k 11. llrlu 4(22). 1L )y.j. .l,'.k' .::2). .. L6:-. 1ty!1.. 1:EIL llrqlrrkp IIF:I::IL llrl. :5,k .. .... ..-!1-2Ik. h..dki:r (1. qlrl.IEEIk . .:E:j.. 1t) 1l: IIFI. Jlr!l. ql. Ilrl,) jll:)p .11:)'L.. tdk4(22). !rdtgr11.,1. ki-b.' u. Jlrck!q;l.. . .'... . ' L..;L.-.?-..... k,' qtthk. t.. .r)k.-...y . tl.#-''-it--.. ..----. --L(-q...L.t..-. tl),,:... . . . ytyth, t. til. u Cit-'.' . laws, of countble 1ntersecti on also. A . t.g cqqy 7. tp,y E. E ::.rgn ' .E ..r.;E-.:. ng can EF. jC.y E. . g.. .--.. t.. . ykkE (j,.t, . . y 2 ,t ...E.: j....y t( q.,y...y.. .,p... .(),. 'tj. , . .. .. ,ar. . J?'1'l ,. ;w.. tykk,yyal. @(r,t t7l7)li)'yE--jp:ytq-l'h@i'-.--.j#i.tt)r:-.)k2!l$,j);.. ;,; , , .. t..Lti-.(yy....hk-...'i'..-.-i'.---j;.rls tlot , t ,-,Ljb;LLb6.. j, ) ln the sequel )j.rq( shall t;need -)ykyy.tjj.. t:-.t.-)-t...?t-(..--.-..;-yy..,F.rconcept . -?,--y)k-ttg.-tjE '...-..-.--..:.--r-. we g ..yqE a E ;..-...y..-.:..)...;..y..--.y.y....-........... ),,r.tg2s;-...:,:.5-r):(t..---r@t()kttr;jpki-):.y.t2r).-. ,-.)j.Lb1,,.q--. (;. .y......-.. . .. .. .. . . ,,.tzk r:kjt. 1)4 ? E y,pqy),jj 33:13333,..6,31)3 qtp. 2L2t1jjk1,(1.. yo,i, . .. q l,. ;,?,. q?. y. . j,yj(. lnt:rsectlon (E. . . . .qk .. : :6. :;)..y.,@, of al1 the jyjj,, .( jy .yj ls j.j..j. ,,,. yjyg.. yyt;,y( tygyj;. contmnlng y.y,rty..y..j, ,. . : '! ;(y.tjrj. . (j(:,.tjyyy. .. . y . ty.j .. .. . . . . sj jyjjy.t.yjgy, LILLIL) jjjjjj,yyyyyjy.. yyyyy T. .r.. . .. .)g-jjjy)j.).)4jj..j)j.:.,;j.);.j,j;.jj;.)j,.j.L..,.;..j ..., y. .........j3b;,jbbb;$LLL,).jtfjj;6;3$j(. ..g ...s.sy . ... .. .. .. . . .. . .- . . :7',t.,..;r,,#.-. )),.jjj ). .. ..... . . .jj,.., ...yj.,.. . .'''.. . .j.q,jj. . . . . ' q$3t... ,;-. . . ..' r-,-iiq.-r),-iyr,!!.. j. .=-. $.-<..., . . . );,-':'!'tk. '
. '
.
-
..
.:.
-
.
-.
.
''.
-
-
.
..
.
. . .
-
.;
.
,.);.-')
-
,j,q2..
'
.. . .
tju
..
-
..
,
- ,.
,.-
- ..y;)
-
.
..;.
-,...
..
,..
..
.-.
.. ..
y
:.
.
..
. . .
.
(gjj.. -0. ' s.t.).....,-.y....ytt,..c...y.-....1. ?t..'tlt.t.,t.iy.'t?.'rtA)'.i.' .d.e.. '':.('. e..,4(i!I... ''-. . . %)' 4.) 4)d..,'',,M ttytt/p ''4kk,,.,)'#',,,..)4i yt t 14thls iis 7>1'h: j x,,soy s,: xe.)ts..T?th: ,, .
. . .
.
-.
-
.
-.
,.:r
,.
-
.
.
.-.
-
.
.
.-
.
.
.
.
.
.-
. .
.
.
.
. .-
.
. -
.
.
.j(rtrp1k.
.jry,
,,...
.
.
.
. .q;,..
-iji
.
-.
-..
.-
'
.
-.
. .
-
1k-,,,,-Jj
rjjj;;!;;q
.
.-t;,
. .. .
..
.
.
. .
. ..
.
,,,
-,,
.
.....';-.7.
..:.,2,,
'
..
.
,-,.
c-n'
',,
.;
''
,;j),
''j-.k,
-
. . .
,..
.
.
.
'.
--k...
-. .
.
-
. ..
-
.
.
--,.
..
.
be
,,,
.
.
.
..
-
.,.
,,
't(l!jt)t2),.
..
-@jjjgjjk.:
.,i(-$;ydjy-.. .....
. :
.
'
.
.
:
.
..
.
jlj..
. yyjjggk., ?jjy-.$,,;gjjjr..tyy(,.k;.y;jy;j.-
:jjjy...
.
y
.
..
y.s,.y,.y
.
.
.
, .
.
. ..
..
..
'..-,,ibn
y,.;.y
... ,
.....,.--
.
k
s..
,
...
.
.
,,
.
...)...j....j.,..
,.yyy....y,y.y
..
.
.
.,.)......),),.....,).j
.:
.
-..!it,..
....
.
,q
kf,.
,.
.
.
,,
.yy
,4j,j)),,;jjj,-j)))jj)..,;;,)j,,j...
..
-),-.t..
.
.
.
.)
-,,
,.
.
..
,,
.
....
,
.
.
.
,
..
c-flelds
--
.
-,
C
--
.
16
Mathematics
generated by C, customarily denoted c(C). The following theorem establishes a basic fact about c-fields.
calledthe c-field
1.20 Theorem If C is a finite collection c(C) is tinite, otherwise c(C) is always uncountable. Proof Define the relation R between elements of X by Ry iff x and y are elements of the same sets of C' R is an equivalence relation, and hence defines an equivalence class & of disjoint subsets. Each set of & is the intersection of a11the C-sets containing its elements and the complements of the remainder. (For example, see Fig. 1.1. For this collection of regions of R2 & is the partition defined by the complete network of set boundaries.) lf C contains n sets, & contains at most ln sets and c(C), in this case the collection of all unions of &-sets, contains at most lln sets. This proves the first pm of the theorem. Let C be infinite. If it is uncountable then so is c(C) and there is nothing more to show, so assume C is countable. ln this case every set in & is a countable intersection of C-sets or the complements of C-sets, hence & g c(C), and hence is the collection of a11the countable unions of also U4&) c(C), where flX&) t/(&) uncountable, is &-sets. If we show the same will be true of c(C). We may assume that & is countable, since otherwise there is nothing more to show. So 1et the sets of & be indexed by N. Then every union of &-sets corresponds uniquely with a subset of N, and every subset of ENcorresponds uniquely to a union of 2N, &-sets. ln other words, the elements of U4&) are equipotent with those of which are uncountable by 1.13. This completes the proof. . .
,
R, and let C ((-x,rj, r G (E1 ), the collection of closed with rational called Jltilt/'-ff/r6y endpoints. c(C) is the Borel jield of R, generally denotd B. A number of different base collections generate T. Since countable unions of open intervals can be closed intervals, and vice versa, (compare1.12), the set of open half-lines, ((-x,r), r G (EI), will also serve. Or, letting (rnJ b: a decreasing sequence of rational numbers with rn 2, x,
1.21 Example Let X
=
=
(1.21) Such a sequence exists for any G ER(see2.15), and hence the same c-field is generated by the (uncountable)collection of half-lines with real endpoints, ((-x,x), x e R ) lt easily follows that various other collections generate S, including th open intervals of R, the closed intervals, and the half-open .x
.
intervals. n
1 22 Example Let X given. It is
T
=
F, the extended
real line. The Borel
field of F is easily
# e Sl, where B is the Borel field of R. You can verify that S is a c-field, and is generated by the collection F' of 1.21 augmented by the sets (-x) and V. n =
(#, B tp (+xJ, B t.p (-x4, B QJ (+x) t-p (-x):
1.23 Example Givn an interval f of the line, the class
BI
=
LBf-h f: B e B ) is
Sets and Numbers
called the restriction of T to 1, or the Borel field on 1. ln fact, T; is the c-field generated from the collection C ((-x,r1f-h f: r e () ) n =
.
Notice how c(r) has been defined from the outside' It might be thought that c(r) the inside', in terms of a specified sequence of the operacould be defined tions of complementation and countable union applied to the elements of r. But, despite the constructive nature of the definitions, 1.20 suggests how this may be impossible. Suppose we define .Wlas the set that contains r, together with the complement of every set in T' and a11 the finite and countable unions of the sets is not c(C) because it does not contain the complements of the of r. Of course, be the set containing W1together with all the complements and unions. So let in the same finite and countable unions of the sets in W1.Defining 45, manner, itmight be thoughtthat themonotone sequence (W,,) would approach c(C) x; but in fact this is not so. ln the case of the class fm,l!, for as n it example, can be shown that Wx is strictly smaller than c(C) (see Billingsley IX This fact is 1986: 261. On the other hand, c(C) may be sm aller than for in again fm,1), j3.4. demonstrated, The union of two c-fields (the set of elements contained in either or both of them) is not generally a c-field, for the unions of the sets from one tield with those from the other are not guaranteed to belong to it. The concept of union for c-fields is therefore extended by adding in these sets. Given c-fields ; and N, the smallest c-field containing a11the elements of 5 and all the elements of Wis denoted 5 v V, called the union of 5 and V. On the other hand, 5 r'A W (A:A G . and A e V') is a c-field, although for uniformity the notation 5 ZN V may be used for such intersections. Fonnally, 5 zq N denotes the largest of the c-fields whose elements belong to both 5 and V. Both of these operations generalize to the count1,2,3,... we may define able case, so that for a sequence of c-tields 5n, n l;a. V= n n 15 n and 0* Without going prematurely into too many details, it can be said that a large part of the intellectual labour in probability and measure theory is devoted to proving that particular classes of sets are c-tields. Problems of this kind will arige throughout this book. It is usually not too hard to show that Ac e 5 whenever W e 5, but the requirement to show that a class contains the countable unions can be tough to fulfil. The following material can be helpful in this connection, A monotone class A is a class of sets such that, if tAn) is a monotone sequence withL limit 4, and An e A for a1l n, then A G A. If (Aa) is non-decreasing, then zl U7=1An.lf it is non-increasing, then A O0J=Iz4,,. The next theorem shows that, to dtenine whether or not we ar: dalinf with a c-field, it is sufficient to ,'krd r' y' jr';ty 't'ld r' yyy'.yyjjjjyjjjjjjiy.d k' t'(' y' );' jyyd lr'yyy p' )' 'y,d trytd td ttd t'. ild ctmsl'detwhdherthe limits of monotone sequences belong to it, which should often ' b j esiet to establish than the general case. .
tfrom
.d1
.4z
.44,...
--->
.
=
=
=
=
=
C' y' .'E
-'. .' -' :t' '
.'''
.' '''.'' ..'.'.'....'. ..'.' '. ' . . .'... . .. .. ..(E.
' . .. '
E
' g. ...
' . .'
r . .. : :
1);,4
.
. .lj ' .. . y. .. . . Flrqh:oykm
l q)
l g...(
.. . , . . .
l .
.
'
.
.
. .
.
.
?;; is a c-field iff it is both a field and a monotone class.
.t:)2
E
.
.
:.y
:
.
. . .. . . .r ..Eq(
,,k))
y.
t j' ' .
. . The..ftjjjjy, p tjtjjy .
''
of the
ito
is immediate. For
theorem patt the part y(t.: j. . ;) y y. j.r r ,'tlt:' t.ri.jjryry.yi.,yuit y 1:2j1. :jkq-.j(jjji. :g;pC;: ;(.):t..,.(.,)i'.:..'..y.;tt.t,jiyA......,y '.1t jkj:r qlj?ii. 4, )1:41.. .j1r:lg. 1:;(jj.. jpp:p-;,r 4:;:: ...... jlrrjs 4:2:: 422:: j,qbqf. . 4EEEE! jjr-iylr jjy-plrqjj .. jrrqls ,,krIg. :;; igjl. ji. ji. :. ,(EE L.,;.,jfL:..; ... j . I j!..I t;i:;r ni; (1q ... .g. )......)...... tjjii:;s ,b, 4:22EE jj tlij;r (yj,?)'' yj e. (. . k:j?,. .t.y(). r .., j; (jry-;;;(tj yy ). .s.u.. y... r(j-(.. (.; .i.qLL);,;j . y. . ... y... .;jtytytyt, . . . . tg -L$1t.. ttttt . . .;j )')r'--t)-tzjj i-.-tki -,.-(jkyr))-;-..---,r. y is . rj limit with =1A,, t monotone ;, );. by t sequence ) a ulyytjt U.-----.- e y-):-),::::r,:--.. --..L3LLLi..-. .,.-yt.-..,7r-...(-.......). r ;-)y)-. j/l-rjt--j,-....;.LL;L;.,.;j;;-),.,;b;qj, '''-?'''t''-' )jjt.-. (t)-.)rrtt.,..,)))-y)#)---.r),-.-. t;)--?--.--,r...r-....-.,... ...... ... . . .--. ... ;.,-(.$;.,q;-.,-LLLL1L--t;))plt:'-i yryjyti... E #,,.. .. kr. t).y-j-t,yr ....-..--...- ....-.... ttl'i?.-.-;r-..t...)k.-(--..-t..-.ti..-)-.,)---)'.-.-.t.....-..t-., .t'y4?.i@''. ... . ..Li q-?..'..I3;......;-;t.-b. .!.#,..-t,) llt))k...-.... tiy,-3$bqL-;--....').L ..::?)i..-..$kr'-'t)L' .. .)yyyiyyyi 14441111@11..... i
r
,.tE'jt.ty( : ;,
i E kE'
(
(
.
.
-.yy)j,ytj,.i...y.yE.y;;r.,...myt;y.U,
; -. . g . .
; ;
(
.
E
i..E -
-
q
,.
..
.
.,...
,,
-. .
-
-. .
- . . .
:.
-
.
-.
.E-...
,
q .
. . .
.. -.
.
.
:
E .
.
-
k.tlll.
..
-.
..
..
...-....
.....
..
,.
.
.
..
- ...-.-
.
.
. . ,
...
,,
-;::,---::--
.
-
........
-j
..-.
.,
.
.
..
...
'''--'f'':tt--
.....
define
,
-k,(
..
.....,.
x ' '--------8-'-:'''*''''% . -))
'if'
-jlqilii;,;j;r
.
....
..
-......-....
..
.
.
. .
.
.
.
.
.
..-
..
... ..... -
-p
. .
..
..
-
.
18
Mathetnatics Ur1=lAk, so the theorem follows.
assumption.Uo;=1An =
.
Another useful trick is Dynkin ' s zr- theorem.l To develop this result, we define two new classes of subsets of X.
1.25 Definition A class T is a z-system if A and B e T implies A rn B e T. A class .f is a h-system if (a) X e 1. and B (b) If A and B e c A, then A B e 1. is a non-decreasing sequence and Au A, then A e (c) If fAn e n ,f
-
'1'
,f)
.
Conditions (a)and (b)imply that a l-system is closed under complementation (put .f
m.Moreover, since (b) implies that Bn A,,+1 A,i e for each n, (c) implies that a countable union of disjoint d-sets is in 1. In fact, these implica-
X
=
=
-
tions hold in both directions, and we have the following.
is a l-system if and only if 1.26 Theorem A class (a) X e 1. then Bc e 1. () If B e is a disjoint sequence, then UnAn e (c) If (Aa e ,f
.f
,f)
.
Ia
ln particular, a c-field is a X-system, and moreover, a class that is both a is is a c-tield. This follows by 1.24, because a a-system and a class closed under unions by 1.25(c), and by de Morgan's laws is if a monotone closed under both intersections and complemntation. The following result makes these dehnitions useful. -system
-system
1.27 Dynkin's x-A theorem (Billingsley 1979, 1986: th. 3.2) If T is a a-system, ctfis a l-system, and P c 2, then c(T) c 1. containing P (theintersection of a11 Proof Let kTj denote the smallest containing P), so that in particular, (T) the c M.We show that 3.(:P)is remarks l-system. follow the above, it will then that (T) is a c-field, and By a 2, as required. hence that c(T) c &T) For a set A e 1(T), let Vztdenote the class of sets B such that A rn B e (T). We Clearly, X e Vz, so that condition 1.25(a) is shall show that gk is a satisfied. Let #1, Bz e 5k, and Bj c #2; then A rnfl e 14T) and A fa #2 e (T), and (A r''h#1) c (A fa #2), which implies that -system
-systems
-system.
(A rn #2)
(A t'''h#1)
-
A f'n
=
(#2 -
#1) e 14T).
(1.22)
But this means that Bz #1 e Vx, and condition 1.25(b) is satisfied. Lastly, suppose A f''h Bi e ,4T) for each i 1,2,... and Bi #. Then zt f-h # e (T) by 1.25(c), which means that 1.25(c) holds for Vx, and sk is a as asserted. Suppose A e P. Then B e P implies A PB e T p is a a-system) and since P i and '4T) (T), this further implies B c 5k. Hence P c 5k. Since Nx is a containing have in this smallest also (T) gk P, is the we case. So, c V',A and hence A fa B e (T). whn A e T, # e VT) implies B e We can summarize the last conclusion as: -
'1'
=
-system
-system,
-system
Sets and Numbers (A e P, B e 1(T)) Now defining Vs by analogy
(# e
(T),
yt
with
PB e
=
19
(A OB e
(T)).
(1.23)
9%, so that (T))
=
(A e Vs),
(1.24)
we see that (1.23)and (1.24)together yield P c T. Since Vs is also a by the same argument as held for Vx, and contains T, 1(T) c Ns by definition of (T). Thus, suppose B e (T) and C E (T). Then Ce Vs, which means that B f''h Cq (T). So 1(T) is a n-system as required. . -system
+.
2 Limits and Continuity
2.1 The Topology of the Real Line The purpose of this section is to treat rigorously the idea of as it applies to points of the line. The key ingredient of the theory is the distance between a pair of points, x, y e R, defined as the non-negative real number Ix y l what is formally called the Euclidean distance. ln Chapters 5 and 6 we examine the generalization of this theory to non-Euclidean spaces, and find not only that most aspects of the theory have a natural generalization, but that the concept of distance itself can be dispensed with in their development. We are really studying a special case of a very powerful general theory. This fact may be helpful in making sense of certain ideas, the definition of compactness for example, which can otherwise appear a little puzzling at first sight. An e-neighbourhood of a point e R is a set Sx, fy: Ix-yI < :1, for some s > 0. An open set is a set d c R such that for each e A, there exists for some e > 0 an e-neighbourhood which is a subset of d. The open intervals defined in j1.3 are open sets since if a < < b, : minl 1b l > 0 satisfies the la definition. R and O are also open sets on the definition. The concept of an open set is subtle and often gives beginners some diftkulty. Naive intuition strongly favours the notion that in any bounded set of points to' a point outside the set. But open sets are there ought to be one that is sets that do not have this property, and there is no shortage of them in R. For a complete understanding of the issues involved we need the additional concepts of Cauchy sequence and limit, to appear in j2.2 below. Doubters are invited to suspend their disbelief for the moment and just take the definition at face value. The collection of a1l the open sets of k is known as the topology of R. More precisely, we ought to call this the usual topology on R, since other ways of defining open sets of R can be devised, although these will not concern us. (See Chapter 6 for more information on these matters,) More generally, we can discuss subsets of (Rfrom a topological standpoint, although we would tend to use the term subspace rather than subset in this context. If A c S ? R, we say that A is open in S if for each x G 4 there exists Sxtzj, : > 0, such that .(x,E) f''AS is a subset of A. Thus, the interval (0,) is not open in R, but it is open in (0,1). These sets define the relative topology on S, that is, the topology on S relative to R. The following result is an immediate consequence of the definition. inearness',
-
,
.z7
=
.'r
-x1
.7
=
-x1
,
2.1 Theorem lf 4 is open in R, A r''hS is open in the relative topology on S.
l:n
A closure point of a set A is a point x e R such that, for every E > 0, the set
Limits and Continuity
21
is not empty. The closure points of A are not necessalily elements of A /'7 A, open sets being a case in point. The set of closure points of A is called the c losure of A, and will be denoted X-or sometimes (A)- if the set is defined by an expression. On the other hand, an accumulation point of A is a point x e R wich is a closure point of the set A f.x) An accumulation point has other points of A A, it must also arbitrarily close to it, and if x is a closure point of A and accumulation closure is point. A point that not an accumulation point (the be an of contains x former definition being satisfied because each E-neighbourhood itselfl is an isolated point of A. AC /-7 Sxnz) is A boundar.v point of a set A is a point e X such that the set points of A is denoted ?A, and X not empty for any e:> 0 The set of boundary AO A A. A closed set is one containing A t.p :z4. The interior of A is the set all its closure points, i.e. a set A such that A A. For an open interval A a,b) c R, X' (tz,1. Every point of anb) is a closure point, and a and b are also closure points, not belonging to asbj. They are the boundat'y points of both a,b) and gJ,?7). ,&(x,sl
-
.
.x
.x
.x
=
.
=
-
=
=
=
2.2 Theorem The complement of an open set in R is a closed set.
(:I
This gives an alternative definition of a closed set. According to the definitions, O (theempty set) and R are both open and closed. The half-line (-x,xq is the complement of the open set @,+x)and is hence closed. Extending this result to relative topologies, we have the following.
2.3 Theorem lf
:4
is open in S c R, then S -A is closed in S. cl
In particular, a corollary to 2.1 is that if B is closed in R then S r-hB is closed in S. But, for example, the interval l, 1) is not closed in R, although it is is open in (0,1). closed in the set (0,1),since its complement (0,r1) Some additional properties of open sets are given in the following theorems.
2.4 Theorem
(i) The
union of a collection of open sets is open.
(ii) If A and B are open, then A /-7 B is open. n
This result is be proved in a more general context below, as 5.4. Arbitrary intersections of open sets need not be open. See 1.12 for a counter-example.
2.5 Theorem Every open set A q R is the union of a countable collectin joint open intervals.
of dis-
Proof Consider a collection (&x,:x), x e A h where for each x, Ex > 0 is chosen enough 5'@,:x) Then that A. small i Uxex5'tx,:xlA, but, since necessarily A UxeA5'@,sxl UxeA.@,Exl, it follows that A. This shows that A is a union of open intervals. Now define a relation R for elements of A, such that xRy if there exists an open interval f q A with x e f and y E 1. Every x e A is contained in some interval by the preceding argument, so that xRx for all x e A. The symmetry of R is obvious. Lastly, if x,y e I A and y,z e 1' ? A, I /-7 1' is nonempty and hence I t.p 1' is also an open interval, so R is transitive. Hence R is an equivalence =
Mathematics
22
relation, and the intervals I are an equivalence class partitioning A. Thus, A is a union of disjoint open intervals. The theorem now follows from 1.11. w Recall from 1.21 that f, the Borel field of R, is the c-field of sets generated by both the open and the closed half-lines. Since every interval is the intersection of a half-line (openor closed) with the complement of another half-line, 2.2 and 2.5 yield directly the following important fact.
2.6 Theorem
B contains the open sets and the closed sets of R n .
A collection r is called a covering for a set A i R if A an open set, it is called an open covering.
UsswB. If each
B is
covering theorem If C is any collection of open subsets of R, 2.7 Lindelf's is countable subcollection (Bi e C, i e N ) such that there a X
UB U#. =
f=1
Be N
Proof Consider the collection (Sk Srkbs, rk e Q, sk e Q+) ; that is the collection of a11 neighbourhoods of rational points of R, having rational radii. xQ+ is countable by 1.5, and hence is countable; in other words. The set (E1 show exhausts by We k the set. that, for any open set B c R and point indexing e N a' e B, there is a set Sk G such that e Sk c #. Since has a E-neighbourhood inside B by definition, the desired Sk is found by setting sk to any rational from for s > 0 sufticiently small, and then choosing rk e the open interval (0,1as), 5'(a',/) as is possible by 1.10. Now for each x e UsewB choose a member of say Skp, satisfying x e Skp i B for any B e C. Letting k(x) be the smallest index which satisfies the requirement gives an unambiguous choice. The distinct members of this collection form a set that covers Ussc#, but is a subset of and hence countable. Labelling the indices of this set as k1,k2,..., choose Bi as any member of C containing Ski. and hence also for Usecs. w Clearly, U:'xlffis a countable covering for U7=15k, ,/
=
=
,/
.$
.z'
.x
.$,
,/
lt follows that, if C is a covering for a set in R, it contains a countable subcovering. This is sometimes called the Lindelf property. The concept of a covering leads on to the crucial notion of compactness. A set A is said to be compact if every open covering of A contains afinite subcovering. The words that matter in this definition are and Any open covering that has R as a member obviously contains a finite subcovering. But for a set to be compact, there must be no way to construct an inrducible, infinite open covering. Moreover, every interval has an irreducible infinite cover, consisting of the singleton sets of its individual points; but these sets are not open. Gevery'
Eopen'.
2.8 Example Consider the half-open interval (0,11.An open covering is the eountable collection (41/n,1j, n e NJ lt is easy to see that there is no finite subcollection covering (0,11 in this case, so (0,1j is not compact. EI .
A set A is bounded if zt i 5'(x,s) for some x e A and ir > 0. The idea here is that
Limits and Continuity
23
: is a possibly large but tinite number. In other words, a bounded set must be containable within a finite interval. 2.9 Theorem A set in R is compact iff it is closed and bounded. u This can be proved as a case of 5.12 below, and provides an alternative definition of compactness in R. The sufticiency part is known as the Heine-Borel theorem. A subset B of A is said to be dense in A if B c A q X. Readers may think they know what is implied here after studying the following theorem, but denseness is a slightly tricky notion. See also 2.15 and the remarks following before coming to any premature conclusions.
2.10 Theorem Let A be an interval of R, and C c A be a countable set. Then A C is dense in A. -
Proof By 1.7, each neighbourhood of apoint inA contains an uncountable number of points. Hence for each e A (whetheror not e C), the set (A C) f-h 5'(x,:) is not empty for every e > 0, so that is a closure point of A C. Thus, A C c (A C) QJ C A ? (A C) K .x
.v
-
.x
-
=
-
-
-
.
The k-fold Cartesian product of R with copies of itself generates what is called Euclidean k-space, R The points of Rk have the interpretation of k-vectors, or ordered k-tuples of real numbers, x (x1,.n,...,.q)'.All the concepts defined above for sets in R generalize directly to R The only modification required is to replace the scalars x and y by vectors x and y, and define an e-neighbourhood be the Euclidean distance betweenx andy, where 1Icll in a new way. Let Ilx 11 1/2 IEt,-1 , q is the length of the vector a (J1,...,JJ' and then define S(x,z) < el, for som s > 0. An open set A of R2 is one in which every point (y: Ilx 11 x e A can be contained in an open disk with positive radius centred on x. In R3 the open disk becomes an open sphere, and so on. .
=
.
-y
=
=
=
-y
2.2 Sequences and Limits A real sequence is a mapping from E$into R. The elements of the domain are called the indice- and those of the range variously the terms, members, or coordinates of the sequence. We will denote a sequence either by tak,n e EN), or more briefly by IakI:', or just by (xnJ when the context is clear. (xnl'kois said to converge to a limit x, if for every : > 0 there is an integer Nv for which
Ixn
-
xI
<
e for all n > Nv.
Write xn it ij often x, or = limu-jxak. When a sequence is tendinygo +x or said to diverge, but it may also be said to converge in R, to distinguish those cases when it is does not approach any fixed value, but is always wandering. increasing, non-increasing, or decreasA sequence is monotone (non-decreasing, ing) if one of the inequalities ak f xn+1, xn < xn+1, xn k ak+1, or xn > xnx.l holds for every n. To indicate that a monotone sequence is converging, one may write for emphasis either xn a7 will or xn t, x, as appropriate, although xn -.->
.x
-x
'1'
.x
--
24
Mathematics
also do in both cases. The following result does not require elaboration. 2.11 Theorem Evel'y monotone sequence in a compact set converges.
n
A sequence that does not converge may none the less visit the same point an infinite number of times, so exhibiting a kind of convergent behaviour. If fak,n e N l is real sequence, a subsequence is fxnk,k e INl where fnk, k e N ) is any increasing sequence of positive integers. If there exists a subsequence taksk e (h1 ) and a constant c such that xnk (-1)N, c, c is called a cluster point of the n 1,2,3,...) does not converge, but sequence. For example, the sequence f subsequence obtained by only taking the even values of n converges trivially. c is usually a finite constant, but +x and may be cluster points of a sequence if we If the of notion allow convergence in F. a subsequence. is convergent, then so is ) where fmk3 is an any subsequence of the subsequence, defined as (x,,u,k e EN whose members of fnkl members are also increasing sequence The concept of a subsequence is often useful in arguments concerning convergence. A typical line of reasoning employs a two-pronged attack', first one identities a convergent subsequence (a monotone sequence, perhapsl; then one uses other characteristics of the sequence to show that the cluster point is actually a limit. Especially useful in this connection is the knowledge that the members of the sequence are points in a compact set. Such sequences cannot diverge to infinity, since the set is bounded; and because the set is closed, any limit points or cluster points that exist must be in the set. Specifically, we have two uscful --
=
-x
.
results.
2.12 Theorem Every sequence in a compact set of R has at least one cluster point. Proof A monotone sequence converges in a compact set by 2.11. We show that ) hasamonotonesubsequence.Defineasubsequence (xnk) evel'y sequence (ak,ne EN follows. for 1,2,3,... let if S'et 1, and k there nl xnk..! sup,on:ak as exists a finite ?7z+1 satisfying this condition', otherwise let the subsequence terminate at nk. This subsequence is non-increasing. If it terminates, the sub?u) must contain a non-decreasing subsequence. A monotone sequence (.v, n k subsequence therefore exists in every case. w =
=
=
2.13 Theorem A sequence in a compact set either has two or more cluster points, or it converges. Proof Suppose that c is the unique cluster point of the sequence fxn) but that xn zd-yc. Then there is an infinite set of integers (nz, L e EN) such that Iau c I s for some : > 0. Define a sequence fyk)by setting yz xnk. Since (yz)is also a sequence on a compact set, it has a cluster point c' which by construction is different from c. But c' is also a cluster point of fx,,1, of which (ykl is a subsequence, which is a contradiction. Hence, xn c. w ,
-
=
-->
3 ,...,x
n 2.14 Example Consider the sequence ( 1,x,x2 or more j orma jj y ta.gl where is a real number. In the case IxI< 1, this sequence converges to n e EN()1, The condition specified zero,( I.x''lJ being monotone on the compactinterval g0,11. ,a:
.'t7
,...),
,
Limits and Continuity
25
in (2.2) is satisfied for Nc = logtel/log IxI in this case. If x = 1 it converges it to 1, trivially. lf > 1 it diverges in R, but converges in 2-to +x. If x +1 diverges, but oscillates between cluster points and neither converges nor -1
.x
=
-1.
Finally, if x < 1 the sequence diverges in R, but does not converge in Ea mately, it oscillates between the cluster points +x and -
F. Ulti-
-x.
We may discuss the asymptotic behaviour of a real sequence even when it has no limit. The superior limit of a sequence fxn) is
limsup xn n
=
inf sup xm. n
(2.3)
m kn
(Alternative notation'. limn xn.) The limsup is the eventual upper bound of a sequence. Think of (sup-znau, n 1,2,... J as the sequence of the largest values the sequence takes beyond the point n. This may be +x for every n, but in all cases it must be a non-increasing sequene having a limit, either +x or a finite real number; this limit is the limsup of the sequence. A link with the corresponding concept for set sequences is that if xn supA,, for some sequence of sets (4n c R ) then limsup xn supA, where A limsupsAn. The inferior limit is defined likewise, as the eventual lower bound: =
=
=
,
liminf xn
=
=
-
limsupt-ak)
=
sup inf xm, n
n
(2.4)
mkn
also written limnak. Necessarily, liminfaak ; limsupnak. When the limsup and liminf of a sequence are equal the sequence is convergent, and the limit is equal the sequence converges in V. to their common value. If both equal +x, or The usual application of these concepts is in arguments to establish the value of a limit. lt may not be pennissible to assume the existence of the limit, but the limsup and liminf always exist. The trick is to derive these and show them to be equal. For this purpose, it is sufticient in view of the above inequality to show liminfn xn limsupn xn. We often use this type of argument in the sequel. To determine whether a sequence converges it is not necessary to know what the limit is; 4he relationship between sequence coordinates the tail' (as tl criterion for converbecomes large) is sufficient for this purpose. The Cauchy > iff for real 0 of that Ne such ('E'i states every e: fakl converges sequence gence a that Ixn xm I < : whenever n > Nc and m > Ne. A sequence satisfying this criterion is called a Cauchy sequence. Any sequence satisfying (2.2)is a Cauchy sequence, and conversely, a real Cauchy seqqence must possess a limit in R. The two definitions are therefore equivalent (in R, at least), but the Cauchy condition may be easier to verify in practice. The limit of a Cauchy sequence whose members a1lbelong to a setA is by definition a closure point of A, though it need not itself belong to A. Conversely, for every accumulation point of a set A there must exist a Cauchy sequence in the set whose limit is x. Construct such a sequence by taking one point from each of the sequence of sets, -x,
-
.7
(A rn S(x.$In4, n
=
1,2.3,..1,
26
Mathetnatics
none of which are empty by definition. The term limit point is sometimes used synonymously with accumulation point. The following is a fundamental property of the reals.
2.15 Theorem Every real number is the limit of a Cauchy sequence of rationals. Proof For finite n 1et xn be a number whose decimal expansion consists only of zeros beyond the nth place in the sequence. lf the decimal point appears at position m, with m > n, then xn is an integer. If m f n, removing the decimal /10'J-'N point produces a tinite integer a, and xn a so ak is rational. Given any real x, a sequence of rationals (ak) is obtained by replacing with a zero every beyond the ath, for n = 1,2,... Since digit in the decimal expansion of jxa..l l < 10-/, lak) is a Cauchy sequence and xn x as n x. w =
.v
-->
.--->
-xn
The sequence exhibited is increasing, but a decreasing sequence can also be conIf x is structed, as f-yn) where (yn) is an increasing sequence tending to itself rational, this construction works by putting ak = x for every n, which trivially defines a Cauchy sequence, but certain arguments such as in 2.16 below depend on having xn # x for every n. To satisfy this requirement, choose the terminating' representation of the number', for example, instead of 1 take This does not 0.9999999..., and consider the sequence (0.9, 0.99, 0.999, work for the point 0, but then one can choose f0.1, 0.01, 0.001,...1. of a real One lnterestingcorollary of 2.15 is that, since every E-neighbourhood number must contain a rational, (E)is dense in R. We also showed in 2.10 that R (D is dense in R since Q is countable. We must be careful not to jump to the concluAnother version sion that because a set is dense, its complement must be of this proof, at least for points of the intelwal E0,1), is got by using the binary expansion of a real number. The dyadic rationals are the set -x.
<non-
...)
.
-
,
tsparse'
.
D
=
(ilzn,
i
=
1,...,2* 1, n c EN )
(2.5)
-
.
The dyadic rationals corresponding to a finite n define a covering of E0, 11 by 1/2N, which are bisected each time n is incremented. For any x intervals of width e g0,1J, a point of the set f illn' i = 1,,..,22 1 ) is contained in 5'(x,eJfor : < 2/21 so the dyadic rationals are dense in (0,11. D is a convenient analytic tool when we need to detine a sequence of partitions of an interval that is becoming dense in the limit, and will often appear in the sequel. Another set of useful applications concern set limits in R -
.
2.16 Theorem Every open interval is the limit of a sequence of closed subintervals wiih rational endpoints. Proof If a,b) is the interval, with a < &, choose Cauchy sequences of' rationals an a and bn b, with tzI < l)j (alwayspossible by 1.10). By definition, for for all n k N, and hence every x e anbj there exists N k 1 such that x gtzr,,lul > anb) i liminfnlln,lul. On the other hand, since an a and b > bn, abbjc (Jn,:,,1f for all n 2 1, so that a,bjc i liminfagtzn,hnlc. This is equivalent to limsupagtzn,:nj atb). Hence limnltzn,:nl exists and is equal to a,b). . .1e
''
Limits and Continuit.y
27
This shows that the limits of sequences of open sets need not be open, nor the limits of sequences of closed sets closed (takecomplements above). The only hard and fast rules we may 1ay down are the following corollaries of 2.4(i): the limit of a non-decreasing sequence of open sets is open, and (bycomplements) the limit of a non-increasing sequence of closed sets is closed.
2.3 Functions and Continuity hfunction of a real variable is a mapping J:S F-->T, where S c R, and 'F c R. By specifying a subset of R as the codomain, we imply without loss of generality that /S) = T', such that the mapping is onto 7. Consider the image in F, under f, of a Cauchy sequence fxnlin S converging to .x. lf the image of every such sequence converging to x e S is a Cauchy sequence the function is said to be continuous at Continuity is in 'F converging to formally defined, without invoking sequences explicitly, using the 6r approach. < # is continuous at the point x e S if for any e > 0 D > 0 such that Iy whenever < The choice here depend S. of implies lfl may on x. ye fx) I e, lf f is continuous at every point of S, it is simply said to be continuous on S. Perhaps the chief reason why continuity matters is the following result. .f@),
.x.
-
-xl
-
2.17 Theorem lf J: S F-> 'T is continuous at a1l points of S, f -1(A) j s open ju whenever A is open in 7J, and f -1(4) is closed in S whenever 4 is closed in F.
s
(:a
This important result has several generalizations, of which one, the extension to vector functions, is given in the next section. A proof will be given in a still more general context below; see 5.19. Continuity does not ensure that J(A)is open whenA is open. A mapping with this fAC) J(X)C in general, we # property is called an open mapping, although, since cannot assume that an open mapping is also a closed mapping, taking closed sets to clpsed sets. However, a homeomorphism is a function which is 1-1 onto, continuous, and has a continuous inverse. If f is a homeomorphism so is f and hence by 2.17 it is both an open mapping and a closed mapping. It therefore preserves the structum of neighbourhoods, so that, if two points are close in the domain, their images are always close in the range. Such a transformation amounts to a relabelling of axes. If fx + ) has a limit as h i 0, this is denoted f@+4.Likewise, J@-) denotes the limit of f(x ). It is not necessary to have x e S for these limits to exist, but if fx) exists, there is a weaker lkotion of cpntinuity at x. f is said at the point x e S if, for any s > 0, R 6 > 0 such that to be right-continuous and < 8 whenever 0 f x- h e S, .j
,
-
Iflx-h) fx) I < -
lt is said to be ever 0 < h <
(2.6)
E.
left-continuous at x if, for any e: > 0, 3 8 > 0 and x e S,
wch that
when-
-
If@) J@ ) I < -
-
z.
(2.7)
Mathematics
28
Right continuity at x implies J@) J@+)and left continuity at x implies fx) = the function is continuous at x. fx-). If fx) f(x+) Continuity is the property of a point x, not of the function f as a whole. Despite continuity holding pointwise on S, the property may none the less break down as certain points are approached. =
.f@-),
=
=
2.18 Example Conjider fx)
1/x, with S
=
=
T'
=
(0, x).
iff
<
For 6: > 0, 1
Iflx- ) fx) I -
=
x(x+o
and hence the choice of depends on both 0. l:l 0, but not in the limit as that
J: S
F->
sx
-
flx) is continuous
continuous if for every E
'F is unformly
Ix-yl
6: and x.
j
for all x
>
-->
.x
The function
< E
<
=
Ifx,
>
0 H
>
0 such
-.f(A') 1< e:
(2.8)
for every x,y e S. ln 2.18 the function is not uniformly continuous, for whichever is chosen, we can pick x small enough to invalidate the detinition. The problem arises because the set on which the function is defined is open and the boundary point is a discontinuity. Another class of cases that gives difticulty is the one x. where the dofnain is nbounded, and continuity at x is breaking down as x -->
However, we have the following result.
2.19 Theorem lf a function is continuous everywhere on a compact sets, then it is bounded and uniformly continuous on S. In (For proof, see 5.20 and 5.21.) Continuity is the weakest concept of smoothness of a function. So-called Lipschitz conditions provide a whole class of smoothness properties. A function f is said to satisfy a Lipschitz condition at a point x if, for any y e 5'@,) fot some 8 > 0, H M > 0 such that
If@l fx, I K -
ftlx
-yI )
(2.9)
R+ F-> R+ satisfies hld) 1 0 as d 1, 0. f is said to satisfy a uniform Lschitz condition if condition (2.9)holds, with fixed M, for a1l x,y e S. The Continuity (resp.unifonn type of smoothness imposed depends on the function uniform Lipschitz) property for any continuity) follows from the Lipschitz (resp. choice of h. Implicit in continuity is the idea that some function (.): R+ F-> R+ exists satisfying 6(E) 1 0 as 6: 1 0. This is equivalent to the Lipschitz condition holding for some (.),the case h = 8-1 B y imposig some degree of smoothness on h making it a psitive power of the argument for example we impose a degree of smoothness on the function, forbidding sharp corners' The next smoothness concept is undoubtedly well known to the reader, although differential calculus will play a fairly minor role here. Let a function J: S F-> 'F be continuous at x e S. lf
here
W
:
.
.
-
-
.
29
Limits and Continuity
fx-h) fx) h -
/:(x)
=
lim i0
(2.10)
exists, A'@)is called the left-hand derivative of f at x. The right-hand derivative, /1(x), is detined correspondingly for the case h 0. lf J+'(x) f-'(x),the common value is called the derivative of f at x, denoted f'x) or df/dx, and f is said to be dterentiable at x. lf J': S F-.> R is a continuous function, f is said to be continuously dterentiable on S. A function f is said to be non-decreasing (resp. increasing) if f@) flx) (resp. f@)> Ax))whenever y > x. lt is non-increasing (resp.decreasing) if is non-decreasing (resp.increasing). A monotone function is either non-decreasing or non-increasing. 'When the domain is an interval we have yet another smoothness condition. A function f.. (J,:1 F-+ R is of bounded variation if 3 M < x such that for every < xn b, by tinite collections of points a partition of gt7,/71 xo < x1 < '1'
=
-f
=
...
=
n
< M. lfxi) .f(.xf-.1)I 77 k=1
(2.1 1)
-
2.20 Theorem If and only if f is of bounded variation, there exist non-decreasing functions h and h such that f n .42
.f1.
=
-
(For proof see Apostol 1974: Ch. 6.) A function that satisfies the uniform Lipschitz condition on gJ,hl with ( I.x y I) Ix y I is of bounded variation on =
-
-
(J,:1.
2.4 Vector Sequences and Functions A sequence (xs) of real k-vectors is said to converge to a limit x if for every E > 0 there is an integer Nc for which
-x1I< Ilxn
E
for all H > Ne.
The sequence is called a Cauchy sequence in R iff Ne and m > Nv. A function
fl S
i--y
(2.12)
-auIl< e Ilxn
whenever
n
>
1-,
where S c Rk an d 7 R associates each point of S with a unique point of 7. 1ts graph is the subset of S x'r consisting of the k + ll-vectors (x, fx) ) for each x e S. f is continuous at x l S if for any E > 0 H > 0 such that ,
,
#
IIII <
=
I/.(x+)
-
fx) I < e
(2.13)
whenever x + b e S. The choice of may here depend on x. On the other hand, f is uniformly continuous on S if for any E > 0, 3 > 0 such that
IIII<
c:':>
sup x e S,x+: c S
l/'tx+h)
-.f@)
I<
e.
(2.14)
Mathematics
f (h,...,fm)'of functions of x is called, simply enough, a vector function.2 c on tinuity concepts apply element-wise to f in the obvious way. The A vector
=
function
f. . s
F-.
s sc ,
R -1
is said to be one-to-one if there exists a vector function f : S F-.-:hS, such that f -1(/.1)) x fOr each x e S An example of a 1-1 continuous function is the affine transformation 3 ./12) X2 + b = for constants b k x 1) and A (kx k) with IA l 0, having inverse function/-l -1 does not possess a closed form, A (y b4 ln most other cases the but there is a generalization of 2.17, as follows. =
.
=
./-1/)
:y:
-
.
2.21 Theorem Iff / S F- 1- is continuous, where S i Rk an d 1- i R'/ f-1A) is open in S when A is open in I-, and.f -1(A) is closed in S when A is closed in -T. (:1 ,
2.5 Sequences of Functions 7, 'T c R, be a function, where in tb-is case fl may be an arbitrary set, not necessarily a subset of R. Let f/'n,n e (N) be a sequence of such functions. If there exists f such that, for each e f, and E > 0, R Nvjl such that 1Jnt)l J((J)) l < e:when n > Asro,then fn is said to converge to f,pointwiseon f1. As for real sequences, we use the notations fn f, fn f, or fn 1 J, as appropwhere in the latter case the monoriate, for general or monotone convergence, tonicity must apply for eveyy (.t) e f2. This is a relatively weak notion of convergence, for it does not rule out the possibility that the convergence is breaking down at certain points of f1. The following example is related to 2.18 above. Let
fn1f
F-
-
'1'
-->
2.22 Example Let fnx) (0,x) is 1/x. But
=
nllnx
+
1),
fnlxt -
-
.x
1
e
=
x
(0,x). The
pointwise limit of
fnx) on
1 xtrl-x + 1)
,
and llxNcxx + 1)) < e: only for Nu > (1/a 1)(1/x). Thus for given E, Nvx cxo 0 and it is not possible to put an upper bound on Nu such that as x lfnx) 1/xl < s, n 2 Ncx, for every x > 0. n -->
-
--
-
To rule out cases of this type, we define the stronger notion of unifrm convergence. If there exists a function f juch that, for each : > 0, there exists N such th a t
lfnkbt J((,))I < sup eD -
fn is said to converge
E when
to J unifonnly on f1.
n
>
N,
31
Limits and Continuity
2.6 Summability and Order Relations The sum of the tenns of a real sequence faklTis called a series, written X'izzlxa (or just EAk). The terms of the real sequence (X'=lau,n e (N) are called the partial sums of the series. We say that the series converges if the partial sums converge to a tinite limit. A series is said to converge absolutely if the monotone sequence (Z'=1IxmI n e Ehl converges. ,
T x? This 2.23 Example Consider the geometric series, Zy converges to 1/(1 x) when 1x1< 1, and also converges absolutely. It oscillates between cluster points and for other values of x it diverges. u 0 and 1 for x .1
-
.
-1,
=
2.24 Theorem lf a series converges absolutely, then it converges. Proof The sequence (X;.1lxm I n e EN)is monotone, and either diverges to +x or converges to a finite limit. ln the latter case the Cauchy criterion implies that + Ixn..mI + Ixn..mI 0 as m and n tend to infinity. since Ixn I + lxnl + inequality,4 +xn.vm I by the triangle lak + convergence of (X)=1au,n E EN ) follows by the same criterion. w ,
-->
....
....
....
An alternative tenninology speaks of summability. A real sequence taklT is said to be summable if the series Xzkconverges, and absolutely summable if ( Ixnll7is summable. Any absolutely summablesequenceis summableby 2.24,and any summable sequence must b converging to zero. Convergence to zero does not imply summability (see2.27 below, fof example), but convergence of the tail sums to zero is necessary and suftkient.
2.25 Theorem 1ff (xn)7is summable, Zrlzznx- 0 as n -->
n-1
-->
x.
Proof For necessity, write IZ,,=l.zu x ! f IZ,a=1au I + I m-nxm j sincefor any : > 0 < k there exists N such that IE,7ox,ul e: for n N, it follows that lZp1=laulS ,,=,14% Then n I +: < x.n Conversely, assume summability and let A Eklxn. IE#-1 Z,n=n-Ym X Y,,=1.<,a () as n .-.y x s
sx
.
=
-1
-->
=
.
-
.
A sequence fakJTis Cesro-summable if the sequence This is weaker than ordinary convergence.
i .26 Theorem lf (akl:' cbnverges to x, its Cesro
-1
n converges. f?z Eru=laulq
sum also converges to x. n
But a sequence can be Cesro-summable in spite of not converging. The sequence f (-1)/)7 converges in Cesro sum to zero, whereas the partial sum sequence (r=()(-1)''' J7converges in Cesro sum ttl (compare2.14). Various notations are used to indicate the relationships between rates of diverfJnl!' gence or convergence of different sequences. If (xnlTis any real sequence, is a sequence of positive real numbers, and there exists a constant B < x such that IakIIan ; B for a1l n, we say that xn is (at most) of the order of magnitude of an, and write xn Oa. If takIanl converges to zero, we write xn oan), 4nd s>y that xn is of smaller order of magnitude than an. an can be incfeasing or decr:aing, so this notation can be used to express an upper bound either on the fte of eropth of a divereent seauence. or on the rate of convergence of a =
=
32
Mathematics
sequenceto zero. followsfrom the
Here are some rules for manipulation of 0(.), whose proof definition. lf xn On a) and y,j Onq), then =
.;tn +ya x 'n =
=
=
t?tzynwelxtfts; )
(2. 15)
(?(Itz+f5)
(2.16)
,
xls onsq), whenever =
a'fz is defined.
(2.17)
An alternative notation for the case xn 0 is xn f< an, which means that there is a constant, 0 < B < x, such that xn f Ban for al1 n. This may be more convenient in algebraic manipulations. The notation ak an will be used to indicate that there exist N k 0, and t'inite constants A > 0 and B k A, such that infapvak lan) 2 A and supazxtak Ian) S B. This says that l and (Jnl grow ultimately at the same rate, and is different 0. Some f'rom the relation xn Oan), since the latter does not exclude xn Ian 1. authors use xn an in the stronger sense of xn lan -
txn
.-.-)
=
.-->
-
2.27 Theorem If (ak) is a real positive sequence, and xn 1+a. n (i) if a > then Zpual.u n then Erlxlau log n; (ii) if ('x ()(sl+a) (j mzmxm < then )Lxlau x (iii) if (x < an xx
n,
-
-1
-
,
-1
=
-
-1
(x
.
.
Proot By assumption there exist N 2 1 and constants A > 0 and B k A such that An < xn K Bnz for n k #, and hence Amnzmt < mnzxm S Bxmnzm. The limit x for different values of a defines the Riemann zetahmction of 1x.l?'?ztz as n and its rates of divergence for a k for a < are standard results', see e.g. of Apostol (1974:Sects. 8. 12-8.13). Since the sum terms from 1 to N-1 is tinite, their omission cannot change the conclusions. . ..4
-1
-1,
lt is common pratice to express the rate of convergence to zero of a positive real sequence in tenns of the summability of the coordinates raised to a given power. The following device allows some f'urtherrefinement of summability condiV as v x (0) for tions. Let U(v) be a positive function of v. If &(vx)/&(v) > 0 and < p < +=, U is said to be regularly vatying at inhnity zero). lf a .x x (0), 1 for x > 0 as v positive function Lv) has the property L(%'x)lLM) frl/W/y varying said Evidently, is slowly regularly it at zero). to be any varying function can be expressed in the form &(v) = vpfXvl,where L(v) is slowly varying. While the definition allows v to be a real variable, in the cases of interest we will have v n for n e N, with U and L having the interpretation of positive sequences. --
--
-x
-->
-->
=
2.28 Example (logvltxis slowly varying at infinity, for any
G.
u
On the theol'y of regular vqriation see Feller (1971),or Love (1977).The important property is the following. 2.29 Theorem If L is slowly varying at infinity, then for any k 1 such that
>
0 there exists N
33
Limits and Continuity
v -8
Atvl
<
<
v
8
a 11v
,
N
>
.
(2.18)
(a
Hence we have the following corollary of 2.27, which shows how the notion of a convergent power series can be refined by allowing for the presence of a slowly varying function. 'ixlak
2.30Corollary
< If xn OnLnl) then Ln) which are slowly varying at infinity. u =
x
for a11(x
and al1 functions
-1
<
On the other hand, the presence of a slowly varying component can affect the summability of a sequence. The following result can be proved using the integral test for series convergence (Apostol 1974: Sect. 8.12).
2.31 Theorem If ak
X& m=
l.x,,,
l/gntlog n) 1+8 j w ith
-
>
0 then ,
log 1og n. n
-
E=nzz lak
<
If 6
x.
=
0, then
2.32 Theorem (Feller 1971: 275) lf a positive monotone function &(v) satisfies &(v.x) f7(v) -where D is dense in R+, and 0
v(.x),al1 a7 e <
v(x)<
then
x,
(2.19)
D,
v@) =
.x
P
for
<
-x
p
<
x.
To the extent that (2.19) is a fairly general property, we can conclude monotone functions are as a rule regularly varying.
u that
2.33 Theorem The derivative of a monotone regularly varying function is regularly varying at cra. Proof Given &(v) U' (v) If f7(v)
-->
=
=
vPf-(v),
write
pvP-1f-(v)
+vPfJ(v)
=
vP-1(p1,(v)
+vr(v)).
0 there is no more to show, so assume liminfvfxv)
jfzotjvxxilj
Kdv
which implies f7(vx)/f7(v) U'Mx)
U,(v)
Ly'vl
=
x
...0
-
(v)
--
L
(v) fXv)
>
0. Then
(),
(2.21)
w
(2.22)
1. Thus,
p-lpfatvx) =
f-tv-lj
jf7,(vAl
(2.20)
+
vxc'tvxl
pAtvl +vA
,
(v)
--)
p
a7
.
#
2.7 Arrays Arguments concerning stochastic convergence often involve a double-indexing of elements. An array is a mapping whose domain is the Cartesian product of countable, linearly ordered sets, such as ENx ENor r x (N, or a subst thereof. A real double array, in particular, is a double-indexed collection of numbers, or, alternatively, a sequence whose members are real sequences. We will use notation juch as (fakt,t E z J, n e (N), or just fau) when the context is clear.
Mathematics
34
-?'
A collection of tinite sequences (fau, f 1,...,k ), n e IN) where kn x as n --+ x, is called a triangular array. As an example, consider array elements of the 1,...,n) is a real sequence. The question of form xnt ytln, where tys t whether the series ( :=1.x,,,, n e N l converges is equivalent to that of the Cesro convergence of the original sequence', however, the array formulation is frequently =
,
=
=
the more convenient.
2.34 Toeplitz's lemma Suppose (y,;) is a real sequence and yn .-.-: y. lf ((.'xkf, t= such that 1,...,lk ), n e ENl is a triangula.r array (a) xnt
0 as n
--
for each fixed /,
x
--
kn
(b)
lim n-yoo
X Iak,lS
C <
x,
>1
kn lim A'-lau1, =
n-+x
>1
k then E/i,lxn,y, --- y. Proof
Iyn -y
F or y
=
(c) can be
O ,
fyn) for any s > 0 5 Ne k 1 such that for (c), and then (b) and the triangle inequality,
By assumption on I < 8C. Hence by
,
kn
n
>
Nvs
kn
lim Xxna, -y
n-yx
omitted.
X-v/t-'w
lim
=
>1
-y)
>1
n-yx
Nz
Txntyt y)
< lim
-
+
s
wj
n--kx
in view of (a). This completes the proof, since
E
(2.23)
:,
=
is arbitrary.
x
A particular case of an array fxn/)satisfying the conditions of the lemma is xnt where S = (Z''=1y,)-1y/, ty/lis a positive sequence and Z'lsxlys A leading application of this result is to prove the following theorem, a fundamental tool of limit theory. -->
=.
2.35 Kronecker's Iemma Consider sequences (tz,)';'and x, If Ll=jxtlat C < cxa as n numbers, with at -?'
tx/lTof
positive real
-->
=.
--
1
n
-Xx,
Jn
0.
--)
(2.24)
>1
Proof Detining co 0 and cn Xntcjxtlat for n G N, note that xt atct cr-1), 1,...,n. Also define tzo 1,...,n, so that an 0 and bt at fh-l for t t ntzzkbt. Now apply the identity for arbitrary sequences ao,...,an and co,...,cn, =
=
=
77(c,,.-1 Abel's known
c,-1)
as
=
=
-
-
=
n
n
(This is
=
=
=
77(tz,-1 zflcr-l -
+
ancn
-
zoco.
,-1
partial
summation formula.) We obtain
(2.25)
Limits and Continuit.v
1 n -- X xt an >1
n
1
X atlc t an
=
-
>1
= cn
35
1
-
-
ct- 1)
w-xn
--Lbtct-j an
-->
C- C
=
(2.26)
0
>1
is by the Toeplitz lemma, setting xnt
where the convergence
=
bt lan.
.
The notion of array convergence extends the familiar sequenceconcept. Consider subsequences, collection of full generality k (N for a (fxmnk,e l m e EN), an array where fnk, k e INl is an increasing sequence of positive integers. If the limit xm limz--ycxaAun: exists for each m e IN,we would say that the array is convergent; and = its limit is the infinite sequence (x,,,,m e N 1.Whether this sequence converges is a separate question from whether it exists at all. Suppose the array is bounded, in the sense that supkmlxmpulf # < x. We know by 2.12 that for each m there exists at least one cluster point, say xm, of the inner sequence (xaw,k e IN1. An important question in several contexts is this: is it valid to say that the array as a whole has a cluster point?
2.36 Theorem Corresponding to any bounded array f(xrm,k e ENJ, m e N ) there k e N J m e EN exists a sequece (x,,l, the limit of the array ( tx,ug, J as k x, subsequence n:) of f for each m. where (rl:) is the same ,
-->
,
Proof This is by construction of the required subsequence. Begin with a conver1 be a su bsequence o f (nz) such that sj gent subsequence for m 1', let (nkJ x1. Next, consider the sequence (.n,n)J. Like f-n,s:J,this is on the bounded interval (-#,#), and so contains a convergent subsequence. Let the indices of this 1 be denoted f42:) and note that latter subsequence, drawn from the members of (?1k), Proceeding in the same way for each m generxl,nj xl as well as 12,4 ates an array (fn, k e IN1, m e INJ, having the property that lxjw, k e EN) is a convergent sequence for 1 S i S m. k k e ENJ in other wotds, take the first member of Now consider the sequence fns 1 th e secon d member of (rl2k), and so on, For each m, this sequence is a subtlkl, pzth point of the sequence onwards, and hence the sequence of (n'2Jfrom the k e EN ) sequence f.x,,,4,k k PZJis cnkergent. This means that the sequence (xmmj, k = setting rl11 requirement isfies of the theorem. w is convergent, so the f fnzl sat ...-)
.n
=
,
-->
.n.
--
'
,
is called the
,
.
3 s4easure
3. 1 Measure Spaces A measure is a set function, a mapping which associates a (possiblyextended) real number with a set. Commonplace examples of measures include the lengths, areas, in and volumes of geometrical tigures,but wholly abstract sets can be an analogous way. Formally, we have the following definition. <measured'
3.1 Definition Given a class 5 of subsets of a set f, a measure
jt:
; F->
R-
is a function having the following properties: (a) g,(A) k 0, al1 A e %. 0. (b) g,(0) (c) For a countable collection (Aj e F, j e ENl with Aj DA./' and UjAj e ;, =
p. UXy
=
)'
=
O for j
j'
#
X g,(Xy). C1 j
The particular cases at issue in this book are of course the probabilities of random events in a sample space f; more of this in Chapter 7. Condition (a) is optional and set functions taking either sign may be referred to as measures (see e.g. j4.4), but non-negativity is desirable for present purposes. A measurable space is a pair (f1,T) where f is any collection of objects, and 5 is a c-field of subsets of f. When (,T) is a measurable space, the triple (,T,g) is called a measure space. More than one measure can be associated with the measurable space (f1,F), hence the distinction between measure space and measurable space is important. Condition 3.1(c) is called countable additivity. lf a set function has the property
p,tA kJ#)
=
(3.2)
p.(z4)+p,(f)
for each disjoint pair A,#, a property that extends by iteration to finite collections A1,..., An, it is said to be jnitely additive. In 3.1 5 could be a field, but the possibility of extending the properties of g, to the corresponding c-tield, by allowing additivity over countable collections, is an essential feature of a measure.
If g,(f1) < cxn the measure is said to be hnite.And if Q UjLh where lfl is a countable collection of Y-sets, and g.tfl < x for each j, g. is said to be ; G-jinite. In particular, if there is c(./) and Lh l a collection .Y'such that =
,$
=
sieasure ./
(A fa #: # e for each j, g is said to be c-finite on (ratherthan on ;). If is a measurable space and (A,?L,g) is a measure space 11 for some A q @,(A,8L) g,(AC) restriction If in of (f,T,g,) to A. 0 (equivalentto this case called the g(A) gtf1l when p.tfl < cx)l A is called a supportof the measure. When A supports 2, the sets of ?L have the same measures as the corresponding ones of 5. A point e f with the property g,(( )) > 0 is called an atom of the measure. =
=
=
3.2 Example The case closest to everyday intuition is Lebesgue measure, m, on the measurable space (R,O), where tB is the Borel field on R Generalizing the b a to an notion of length in geometry, Lebesgue measure assigns r?z((J,:1) (J,!71. plausible Additivity is an intuitively interval property if we think of measuring the total length of a collection of disjoint intervals. .
=
-
Lebesgue measure is atomless (see3.15 below), every point of the line taking ((J,:1, O(u,,j,m) denote the restriction of measure 0, but ??z(R) x. Letting (R,f,pz) to a finite interval, m is a finite measure on (t7,:1.Since IR can be partitioned into a countable collection of finite intervals, m is c-finite. n =
additional properties may be deduced from the definition:
Sole
3.3 Theorem For arbitrary T-sets A, #, and (Ay,j e N J (i) A c # = g,(A) S g,(#) (monotonicity). (ii) gtA k.p #) + g,(A ch B4 g.(A)+ g(#). (iii) p,(UyAj) f Xg,(Ay) (countablesubadditivity).
,
=
Proof To show (i) note that A and B -A are disjoint sets whose union is #, by hypothesis, and use 3.1(a) and 3.1(c). To show (ii),use A wB A QJ (B -A) and B = (A chB) k.pCB A), where again the sets in each union are disjoint. The result Al and Bn follows on application of 3.1(c). To show (iii), detine #1 UP-14 An j. Note that the sets Bn are disjoint, that Bn ? An, and that U==lfj j UT=IAy. Hence, J =
-
=
-
./
=
=
=1
X
G)
U p,/=1
=
p,
X
oo
Xp,(l U#, /=1 =
j=1
< Xp,(A?).
(3.3)
.
j=1
This proof illustrates a standard technique of measure theory, converting a sequence of sets into a disjoint sequence having the same union by taking differences. This trick will become familiar in numerous later applications. The idea behind 3.3(ii) can be extended to give an expression for the measure of any tinite union. This is the inclusion-exclusion formula'. n
UA/
p,j=
#
n
Xp,(A)
-
1
j= 1
-
77p,(+rnAzl k+j
+
p.tAycAAkchA,l 77 k:gy:y;l + jttx
f'-'h 1 r''hA:2,
...
-
...
r''hXrj),
(3.4)
where the sign of the last term is negative if n is even and positive if n is odd, and there are ln 1 tenns in the sulp in total. The proof of (3.4)is by induction from 3.3(ii), substituting for the second term on the right-hand side of -
Mathematics
38
(,U.-j+) p.(-.-)
g, repeatedly, for n
-
+
p,
-j,'-s
-U.-,'.,) p.
(,
-.-)
(
-
(3.5)
f..h
,
1. n 2,..., 1. IN be n e ) a monotone sequence of T-sets with limit A e T. A set function on g, is said to be continuous if g,(z4,,) g.(A). -
-
(AU,
Let
.--.-:
3.4 Theorem A finite measure is continuous. Proof First 1et (An) be increasing, with A,,-I cAn, and A U*a=IA,,.The sequence (Bj, j G ) where #1 Aj, and Bj Aj -Ay-I for j > 1 is disjoint by eonstrucand tion, with Bj e @, Aa U7=1#j, =
=
,
=
=
p,(An)
=
Xp,(). j=1
The real sequence (g,(A,,)) is therefore monotone, and converges since it is bounded g,(A). above by g,4Q)< x. Countable additivity implies LT) g,(U7 O';=1A,,. Consider Alternatively, 1et (An) be decreasing, with z4n-l Q An and the increasing sequence lA)) determine g,(49 by the same argument, and use finite additivity to conclude that g(A) = gtf1l g,(A9 is the limit of g(An) =l#y.)
=1g,(#j)
=
:4
=
=
,
=
-
P.(fX -
P.(XnC).
K
The finiteness of the measure is needed for the second part of the argument, but the result that g,tAul g(A) when ytu A actually holds generally, not excluding the case g,(A) x. This theorem has a partial converse: '
-->
=
3.5 Theorem A non-negative uous is countably additive.
set function g, which is finitely additive and contin-
Proof Let (#n) be a countable, disjoint sequence. If A,, U7=l, the sequence (An ) is increasing, Bn f'nAn-l 0, and so gtAnl g,(#n)+ g,(An-I) for every n, by finite additivity. Given non-negativity, it follows by induction that (g,(An)Jis ZTJ =Ip,(), whereas continuity implies that g,(A) monotone. lf A U7=tBj, g(A) LLIUTJ * lB =
=
=
=
=
=
=
.
Arguments in the theory of integution often turn on the notion of a set. In a measure space (f1,T,g), a set ofmeasure zero is (simplyenough) a set M e 5 with g.(M) 0. A condition or restriction on the elements of Q is said to occur almost everywhere (a.e.)if it holds on a set E and f E has measure zero. lf more than one measure is assigned to the same space, it may be necessary to indicate which measure the statement applies to, by writing a.e.lgl or a.e.gvq as the case may be. fnegligible'
=
-
3.6 Theorem (i) If M and N are T-sets, M has measure 0 and N i M, then N has measure 0. (ii) lf (A&) is a countable sequence with g(Mj) 0, V j, then g(UjMj4 0. (iii) If fe)) is a countable sequence with g,(r.'pl 0, V j, then g,((UjA))9 = 0. .
=
=
=
Measure Proof
39
(i) is
an application of monotonicity', (ii) is a consequence of countable additivity', and (iii) follows likewise, using the second de Morgan law. .
In j3.2 and j3.3 we will be concerned about the measurability of the sets in a given space. We show that, if the sets of a given collection are measurable, the sets of the c-field generated by that collection are also measurable (the Extension Theorem). For many purposes this fact is sufficient, but there may be sets outside the c-field which can be shown in other ways to be measurable, and it might be desirable to include these in the measure space. In particular, if g(A) g(A) wheneverd c E c #. This is g,(#) it would seem reasonable to assign jtt'l equivalent to assigning measure 0 to any subset of a set of measure 0. The measure space (f1,T,jt) is said to be complete if, for any set E e 5 with g.(F) = 0, a11 subsets of E are also in T. According to the following result, every measure space can be completed without changing any of our conclusions except in respect of these negligible sets. =
=
3.7 Theorem Given any measure space (f,;,jt), there exists a complete measure TB, (f,TB,g), called the completion of (f1,F,p), such that 5 and g(F) space c F(F) for al1 E e %. In =
Notice that the completion of a space is defined with respect to a particular has a different completion for each measure measure. The measurable space (f1,@) that can be defined on it. Proof Let
AB J;B
denote the collection of a11 subsets of i-sets of pmmeasure0, and LF
=
c
D: FA F e
AB
for some E e
.41
.
0, any set F c E satisfies the criterion of (3.7)and so is in FB as the definition requires. For F e @B, let F(#) g,(F), where E is any F-set satisfying E F e AB. To show that the choice of E is immaterial, let Ej and E1 be two such If jt(F)
=
=
sets, and note that
p,(A'lA&) Since jttFl t-p F2)
=
g,((FAF1)A (FAF2))
=
=
0.
g(F1 A F2), we must conclude
p,(F1 r'hE
+
g,tflr-hE
2 p.tfl k p,(F1 r'hF2)
(3.8) that
for i 1 and 2, or, g,(F1) p.(&). Hence, the measure is unique. When F e T, we AB, confirming that the measures agree on T. can choose E F, since FAF= O e TB is a c-tield colgaining ?F.Choosing E F in (3.7) It remains to show that AFC FB. 8/ shows If F e ;B, thenFA F e ABfor E e 5 and hence Ec for Fe 5 c TB. And finally AB where Ec if Fj e J;B foj e (N,there EF e 5, and so Fc e e exist Ej e 5 for j e EN,such that Ejh Fj e AB. Hence =
=
=
=
=
e U1 A U/) c UIF/AFJ ?
? by 3.6(ii). This means that
AB,
j
Ojb 6
TB,
and completes the proof. w
(3.10)
Mathematics
40 3.2 The Extension Theorem
You may wonderwhy, in thedefinition of ameasurable space, Fcould not simply be the set of all subsets; the power set of D. The problem is to tind a consistent method of assigning a measure to every set. This is straightforward when the space has a tinite number of elements, but not in an infinite space where there is no way, even conceptually, to assign a specitic measure to each set. It is necessary to specify a rule which generates a measure for any designated set. The problem of measurability is basically the problem of going beyond constructive methods without running into inconsistencies. We now show how this problem can be solved for c-fields. These are a sufticiently general class of sets to cope with most situations arising in probability. One must begin by assigning a measure, to be denotedgo, to the members of some basic collection C for which this can feasibly be done. For example, to construct the measure b a. Lebesgue measure we started by assigning to each interval (tz,&1 We then reason from the properties of go to extend it from this basic coltection to a11 the sets of interest. C must be rich enough to allow go to be uniquely defined by it. A collection C c 5 is called a determning class for (f,F) if, whenever jt and v are measures on 5, jt(A) v(A) for all A e C implies that g v. Given C, we must also know how to assign gmvalues to any sets derived from rby operations such as union, intersection, complementation, and difference. For disjoint sets A and B w have g)tz4QJ #) go(A) + go(#) by finite additivity, and when B c A, gotA B) go(A) 04#). We also need to be able to determine gotA fa #), which will require specitic knowledge of the relationship between the sets. When such assignments are possible for any pair of sets whose measures are themselves known, the measure is thereby extended to a wider class of sets, to be is closed and C are the same collection, but in any event denoted 'Y'.Often under various finite set operations, and must at least be a semi-ring. In the is typically either a field (algebra)or a semi-algebra. Example applications 1.18 is a god case to keep in mind. However, cannot be a c-field since at most a finite number of operations are permitted to determine go(A) for any A e .Y'.At this point we might pose the opposite question to the one we started with, and ask why might not be a rich enough collection for our needs. ln fact, events of interest frequently arise which cannot contain. 3.15 below illustrates the necessity of being able to go to the limit, and consider events that are expressible only as countably infinite unions or intersections of C-sets. Extending to the events 5 c(#) proves indispensable. We have two results, establishing existence and uniqueness respectively. -
=
=
=
-
=
-
./
-t/'
,/
'f
,/
,$
=
3.8 Extension ,/.
theorem (existence)Let $ be a semi-ring, and let go:
lf 5 = c(,$). there exists a measure jt on measure on ,YI. jL0t'l for each E e n
./
F--y
F+be
(D,?D,such that jt(F)
a =
Although the proof of the theorem is rather lengthy and some of the details are fiddly, the basic idea is simple. Take an event A c f to which we wish to assign a
Measure ./,
,/,
consider choosing a finite we have g,(A) go(A). If A measure g,(z4).If A G that is, a selection of sets Ej e or countable covering for A from members of OjEj. The object is to tind as 1,2,3,... such that A c ./, j a covering as possible, in the sense that Xgo(f;)is as small as possible. The outer measure of A is =
,/,.
teconomical'
=
P,*(X)
inf X.to(F./),
=
(3.11)
where the infimum is taken over a11tinite and countable coverings of zl by y-sets.
If no such covering exists, set g,*(A) x. Clearly, g7(A) gotAl for each A E g,* is called the outer measure because, for any eligible definition of g,(A), g,*(A) 2
Xg,(l
2
p,
j
Uf j
.9.
=
=
2 p,(A), for Ej
,Y'.
E
(3.12) ,/
The first inequality here is by the stipulation that p(Ej) goIFJfor Ej e in side majorant where exists, is infinite. The the case a covering or else the second and third follow by countable subadditivity and monotonicity respectively, because p, is a measure. We could also construct a minimal covering for Ac and, at least if the relevant outer measures are tinite, define the inner measure of 4 as jt+ (4) jt(AC) p,*(z4C). and g,*(A9 k jttAfl by (3.12), Note that since p,(z4) g,tfl g,*(f) =
=
=
-
-
p,(A) 2 g,+(A).
(3.13)
lf g,*(4) g+44), it would make sense to call this common value the measure of A, and say that A is measurable. ln fact, we employ a more stringent criterion. A set A g f is said to be measurable if, for any B c f, =
g,*(Af-'h B4
+
g,*tz4f chB)
=
(3.14)
jt*(#).
This yields g,*(A) p,+(A) as a special case on putting B f1, but remains valid even if g,tfll x. Let A denote the collection of all measurable sets, those subsets of f) satisfying (3.14).Since g,*(A) go(A) forA e and /1.:40) 0, putting 4 O in (3.14) gives the trivial equality g,*(#) g,*(#). Hence O e A, and since the definition implies that Ac E A if 4 e A, f E A too. The next steps are to detennine what properties the set function jt*: A !-+ F shares with a measure. Clearly, =
=
=
,/
=
=
=
=
jt*(A)
0 for all 4
C f1.
(3.15)
Another property which follows directly from the definition of p,*is monotonicity: X1 I X2 =
F*(X1) f >*(X2), fOr A1, 42 i
.
(3.16)
Our goal is to show that countable additivity also holds for g,* in respect of A-sets, but it proves convenient to begin by establishing countable Juhadditivity. 3.9 Lemma If fAy,j
E
EN J is any sequence of subsets of f, then
42
Mathematics
g,* UX./ f
/7 >*(X./).
(3.17)
Proof Assume g*(Ay) < x for each j. (If not, the result is trivial.) For each j, 1et fEjkq denote a countable covering of Aj by y-sets, which satisfies p.*(A>+ 2-:
< Xpo(,) k
: > 0. Such a collection always exists, by the definition of g*. Since LkkEjk, it follows by definition that U#.j ?
for any
p.*UA/ noting Z7=12S ity is strict. .
=
1.
f XpotA>l
<
j,k
j
(3.17)
Xp,*()+ e,
(3.18)
./
now follows since : is arbitrary and the last inequal-
The following is an immediate consequence of the theorem, since subadditivity supplies the reverse inequality to give (3.14).
3.10 Corollary A is measurable if, for any B c D, g,*(AC r''h#) K g*(#). n t *(A r''h ) +
(3.19)
The following lemma is central to the proof of the extension theorem. lt yields countable additivity as a corollary, but also has a wider purpose.
3.11 Lemma A is a monotone
class.
Proof Letting fAj, j e INJ be an increasing sequence of A-sets converging to 4 U#.j, we show A e A. For n > 1 and E e f1, the definition of an A-set gives =
g7(An fa E)
=
g*(4n-1 ra (An rh A=))+ g*(4k-1 f-'h (Anf''h F))
= g*(An-1 rhEI+ g7(& rn E). where Bn g,*(A()rn E)
= =
An -An-1, and the sequence 0', then by induction, ke(An chF)
=
(3.20)
(BjI is disjoint.
X pz*( rn E)
Put
ztttl
=
O so that (3.21)
y=1
holds for every n. The right-hand side of (3.21)for n e INis a monotone real x. Now, since An e A, g,*(4rh A>)as n sequence, and g,*(An(hE) .--)
-->
jt*(F)
r1*(Anrnm
+
h1*(Xrn)
k jt*(An(nE)
+
g*tAc rnfl,
=
using the monotonicity of g,*and the fact that from the foregoing argument that
AC
(3.22) Anf.Taking the limit, we have
sleasure jt*(A7 k g7(A r'A F)
+
43
g.*(AC
f'-h A'),
(3.23)
so that 4 e A by 3.10. For the case of a decreasing sequence, simply move to the complements and argue as above. . Since () is a disjoint sequence, countable additivity emerges as a by-product of the lemma, as the following corollary shows.
3.12 Corollary
If
()
is a disjoint sequence of A-sets,
g,*U j Proof Immediate on putting E
-
Xp,*().
(3.24)
j
in (3.21) and letting n
=
--> x,
noting VjBj
A. w
=
Notice how we needed 3.10 in the proof of 3.11, which is why additivity has been derived from subadditivity rather than the other way about. Proof of 3.8 We have established in (3.15)and (3.24)that g* is a measure for the elements of A. If it can be shown that 5 q A, setting g,(A) g,*(A)for al1 A e 5 will satisfy the existence criteria of the theorem. implies The t'irst step is to show that c A or, by 3.10, that 4 e =
.t/7
,/
g,*(A'r7A) + g,*(A'r-hAf)< g*(F)
(3.25)
,$1
denote a finite or countable covering of E such that Lj gotAyl < p,*(F) + E, for s > 0. If no such covering exists, g,*(A-) cxl by definis a ition and (3.25)holds trivially. Note that E r'A A c U/A; r-7A), and since /''AAC f-7WC), and by simple semi-ring the sets Aj ('''AAare in ,Y'.Similarly, E c Uyt>j algbra definition of and the set a semi-ring,
for any E i (. Let lAy e
=
,/
Aj r'Aytf
=
Aj
Aj chAl
-
=
UG:
(3.26)
k
where the Cjk are a finite collection of yksets, disjoint with each other and also with Aj r'hA. Now, applying 3.9 and the fact that g,*(#) k)(#) for B e we find .$,
=
g,*(A'faA) +
g*(Ff-$AC)
f
X jtot/j chdl X XgotGl +
j
j
k
p,*(eD =Xw)(z4y) ? <
+ :,
(3.27)
is tinitely additive, and Aj DA where the equality follows from (3.26)becau and the Cjk are mutually disjoint. Since s is arbitrary, (3.25)follows. Next, we show that A is a c-field. We have only to show that A is a field, because 3.11 implies it is also a c-field, by 1.24. We lready know that e A and A is closed under complementation, so it remains to show that unions of A-sets are in M. Suppose that Al and A2 are A-sets and E c f. Then .to
:,*(13 =
>*(A1 nA') + F*(Af rhEI
Mathematics
=
jt*(z42 c7z4jra e) +
g*(z4qrndl chE) + g,*(A2/'nAk ra A')+ jt*(A rnAf
f-h
E)
p,*(A2f-7Al ra A')
+ g*((Aq r7Al ra F)
f-h F)) rn E) k.p (Aj/-AAC
k.p (A2 chAf
+p7((42ra4j)cfaF), = jt*((A2rn41) cjE)
(3.28)
where the inequality is by subadditivity, and the rest is set algebra. By 3.10 this is sufficient for A1 f'742 e A, and hence also for A1kJA2 G A, using closure under complementation. It follows that A is a c-tield containing $, and since 5 is the smallest such c-field, we have that 5 i X, as required. . Notice that (3.28)was got by using (3.14)as the relation defining measurability. The proof does not go through using p.*(A) gx(A) as the definition. The style of this argument tells us some important things about the role of .Y'. Any set that has no covering by '-sets is assigned the measure x, so for finite measures it is a requisite that f q LbEj for a tinite or countable collection fEj o? e #') The measure of a union of y-sets must be able to approximate the measure any F-set arbitrarily well, and the bagic content of the theorem is to establish that a semi-ring has this property. To complete the demonstraiion of the extension, there remains the question of uniqueness. To get this result we need to impose c-finiteness, which was not needed for existence. =
.
3.13 Extension theorem (uniqueness)Let jt and g,' denote measures on a space (f1,T), where 5 c(.$), and is a semi-ring. If the measures are c-finite on and g,(A') jt'(F) for a1l E e .T, then g(An) g'(F) for a1l E e %. .$
,/
=
=
=
Proof We
tirst prove
the theorem for the case of finite measures, by an applitheorem. Define .J= fE e F: p.(F) = p,'(AD ) Then $z ,J by hypothBy 1.27, the proof is completed esis. If is a semi-ring, it is also a l-system. contains c(#). show hence if Fe can that .J is a l-system, and and condition 1.26(a) holds. Additivity When the measure is finite, f e implies that, for A e
cation of
the 7:-
.
./
.4
.4,
F(X3
so that 1.26(b) additivity,
=
F(f)
-
XX)
=
LUIf) -
$(A)
=
>'(A3,
(3.29)
holds. Lastly, 1et fA.j) be a disjoint sequence in co
p,
UAp
j=3
X
oo -
77p,(A> j=
-
.4.
By countable
G)
Xp,-+) -
j=1
p,' UAk
(3.30)
,
y=1
and 1.26(c) holds. It follows by 1.26 and the 7:- theorem that 5 c(./) ? W. VjBj where Bj e and pBp Now consider the c-finite case, Let ( g,'(#j) < co. % fBj rnA: A e 5 J is a c-field, so that the (Bj,5) are measurable spaces, =
./
=
=
=
Measure
45 ./
fe'h Bj. The preceding argument on which jt and $ are tinite measures agreeing on showed that, for A e 5, g(#j rnz4) bt'Bj rnA) only if g and g,' are the same =
measllre.
Consider the following recursion. By 3.3(ii) we have p.(Ara (#1t..p#2)) Letting Cn
=
f''h #2) g.(X f''h #1 f''h #2). g,(z4r'7#1) + h1(z4
=
U7=l the
(3.31)
-
same relation yields
(3.32)
rn cn) p.tz4rn Bn) + p,tA ra cn-j) p,tz4rn Bn rn cn-1). g,ta4 =
-
The terms involving Cn-l on the right-hand side can be solved backwards to yield f-'h an expression for g,(A G), as a sum of terms having the general form g,(Ara Bjj rn Bjzra Bj, ra
for some j, say j that gtD ra Bj) that in (3.32) =
=
j, in which ease D
jt'(D ra B? for all D g.(Afa Cn)
olds for any n. Since Cn
This
=
G
=
=
=
g,to ra Bj)
(3.33)
< x
A fn Bh fe'h Bh fa e F. Since we know preceding by 5 the argument, it follows ...
g,'(4 fa Cn4. f as n
--)
F(A)
)
...
-->
x,
(3.34) we obtain in the
F'(A),
limit
(3.35)
the two sides of the equality being either finite and equal, or both equal to This completes th proof, since A is arbitrary. w
+x.
3.14 Example Let A denote the subsets of R which are measurable according to (3.14) when g,* is the outer measure defined on the half-open intervals, whose measures go are taken equal to their lengths. This defines Lebesgue measure m. 'hese sets fol'm a semi-ring by 1.18, a countable collectioh of them covers R, and the extension theorem shows that, given m is a c-finite measure, A contains the Borel field on R (see 1.21), so (R,f,?n) is a measure space. It can be shown (we wonpt) that a11 the Lebesgue-measurable sets not in B are subsets of T-sets of includes a1l measure 0. For any measure g, on (R,f), the complete space @,SB,F) of the Lebzsgue-measurable sets. In The following is a basic property of Lebesgue measure. Notice the need to deal with a countable intersection of intervals to determine so simple a thing as the measure of a point. 3.15 Theorem Any countable set from R hs Lebesgue measure 0. Proof The measure
of a point
(.x) is
zero, since for x e R,
@Q
fx) ppd :
.
'
'
??z((<))q
'
: ''
' '
=
limn-jxl/n
=
=
r)
n=1
(x
-
1/n, xj e B
0. The result follows by 3.6(ii). w
(3.36)
Mathematics
46 3.3 Non-measurability
To give the ideas of the last section their true force. it needs to be shown that D X c 2 i s poss ible in other words, that ( can contain non-measurable subsets. In this section we construct such a set in the half-open unit interval (0,11,a standard counter-example from Lebesgue theory. For m y e (0,11, define the operator ,
.#4.x
A'+
1,
(3.37)
=
y+x- 1,
This is addition modulo 1. Imagine the unit interval mapped onto a circle, like a clock face with 0 at the top. y x is the point obtained by moving a hand clockwise through an angle of 2= from an initial point y on the circumference. For each set A (0,11, and x e (0,1J, detine the set .i.
A J.x
=
(y 4.x:
AJ.
l
y
(3.38)
3.16 Theorem If A is Lebesgue-measurable so is A %x, and m(A 4.a') any
=
mA), for
X.
for any real x such b +x1) b c lna,bj), unions > A-x + of intervals extends < 0 and b 1. The that a x to finite property translated by x. If A is any Lebesgue-measurable subset of (0,11,and A A-x (0,11 where A + x (y+x: y e A J, the construction of the extension similarly is measurable and mA) mA +x). implies that A and A2 A f-h (1 Now let A1 A /'7 (0, 1 11. Then pztdl +x) -(A1) and mAl + x 1) mA1), where the sets on the left-hand sides of these equalities are in each case contained in (0,11. A1 +x and A2 +x 1 are disjoint sets whose union is A %x', and hence Proof For
(J,!71
(0,11,ma
i
+x,
=
=
-
=
..x
=
-x),
-
-x,
=
=
=
=
-
mA 4.x)
rntz4l +x)
=
+
mAz
= rn(A1) + mA
+
=
x
-
1)
m(A). w
(3.39) .i.
Define a relation for points of (0,1q, letting xRy if y = x r for r e :. That is, xR y if y is separated from x by a rational distance along the cifcle. R is an equivalence relation. Defining the equivalence classes Ex
=
fy: y
.i.
=
.x
r, r e
t1),
(3.40)
(fI,
the sets of the collection x e (0,11 ) are either identical, or disjoint. distance is rational from some other point of the interval, these Since every x a sets cover (0,11.A collection formed by choosing just one of each of the identical sets, and discarding the duplicates, is therefore a partition of (0,11. (FA, Write this as x 6 C), where C denotes the residual set of indices. Another example may help the reader to visualize these sets. In the set of integers, the set of even integers is an equivalence class and can be defined as 0 the Ket of inteeers which differ from 0 an even integer. Of course F0 El = =
sfeasure
47
Eln, for any n l Z. The set of odd integers can be detined similarly as F2n+1 E 1 the set o f integers differing by an even integer from 1. F1 = F3 (A'A, for any n e Z. Discarding the redundant members of the collection e 11 F1 0 collection LE partition of the Z. d leaves just J to e tine a Now constnlct a set H by taking an element from Ex for each x e
#
=
=
...
=
,
...
= .x
,
3.17 Theorem H is not Lebesgue-measurable. Proof Consider the countable collection (H # r, r l (I)) We show that this collection is a partition of (0,11. To show disjointness, argue by contradiction. Suppose z e H rl and z e H r2, for rl # r2. This means there are points 1, hl e H, such that .
'i.
.i.
.i.
l rl
.i.
=
z
=
(3.41)
hz r2.
If rl # r1, we cannot have Jll = 2; but if Jll # hl then Jll and hz belong to different equivalence classes by construction of H, and cannot be a rational distance lrl r2 I apart; hence no z satisfying (3.41)exists. On the other hand, LjrH r), and consider any point let H e (0,11. belongs to one of the rational distance of some element of H; equivalence classes, and hence is within a but H* contains al1 the points that are a rational distance r from a point of H, for some r, and hence x e H., and it follows that (0,11c S*. mlH4 for all r. Since mH*) k Suppose mllt exists. Then by 3.16, mH% r?z((0,1J) 1, we must have m+ > 0 by 3.6(ii), but countable additivity then rnzbul'-l r) which is impossible. It follows that m+ does not gives mH.) -
.i.
.&
.x
=
=
=
=
=
=,
exist. w The definition of H involves a slightly controversial area of mathematics, since the set o f equ ivalence classes is uncountable. It is not possible to devise, even in principle, constructive rules for selecting the set C, and elements from Ex for each e C. The proposition that sets like H exist cannot be deduced from the axioms of set theory but must be asserted as an additional axiom, the so-called axiom of choice. lf one chooses to reject the axiom of choice, this counterexample fails. We have made no attempt here to treat set theol'y from the axiomatic standpoint, and the theory in Chapter 1 has been what is technically called nat've is). For us, the problem of (i.e. based on the intuitive notion of what a the axiom of choice reduces to the question: should we admit the existence of a mathematical object that cannot be constructed, even in imagination? The decision is ultimately personal, but suffice it to say that most mathematicians are willing do SO Sets like H do not belong to (0,1j = f# r'h(0,1j, B e B ). It is not difficult to show that a11the sets of (o,1j are Lebesgue-measurable', see 3.14 and restrict m to (0,1) as in 3.2. By sticking with Borel setf we shall not run into measurability difficulties on the line, but this example should serve to make us careful. ln less familiar situations (suchas will arise in Part VI) measurability cah fail in superficially plausible cases. However, if measurability is in doubt one might remember that outer measure g,*is well defined for a1l subsets of f, and coincides vk
tset'
X
to
.
Mathematics
48
with z whenever the latter is defined. Sometimes measurability problems are dealt with by working explicitly with outer measure, and forgetting about them.
3.4 Product Spaces If (f1,T) and (E,T) are two measurable fxE
((,():
=
(t) l
spaces, 1et
f,
(
G
El
(3.42)
be the Cartesian product of f and E, and define ?F@V Spg
=
((Fx G), F e @, G
G
=
c(Spg), where
V).
(3.43.)
a product space, and (f,F) and (E,W) are the coordinate factor spaces, or spaces, of the product. The elements of the collection Sys are called the measurable rectangles. The rectangles of the Euclidean The space
(f1x E, 5 * V) is called
plane R x R
=
R2
(produc ts
of intervals) are a familiar case.
3.18 Example Rather trivially, consider the two-element setsA B f(1,(21e N. The corresponding rectangle is
=
(1,(1
e @,and
=
A XB
=
1), (t,)1,t2), (*2,(1),()2,t2)l. 1(t,)1,('
The sets f(1,(1), ((*,,(2))and ((1,(2), (2,(1)) are not rectangles, unions of rectangles and so are elements of 5 @ T. n
but are
Two important pieces of terminology. If E c f x E, the set an(F) (: (,() e F) is called the projection of E onto f1. And if A D, the inverse projection of A is the set =
o-1(A)
A xE
=
=
(4),4):
)
e A,
(
EJ
e
(3.44)
.
A x E is also called a cylinder set in f x E, with base A. The latter terminology is natural if you think about the case = R2 an d E R Cylinder sets with bases in 5 and T are elements of R,e. One might tlnk that if E e 5 @ N, antA-) should be an F-set, but this is not necessarily the case. :tE'f # ln(M) in general (see is not closed 1.3) so that the collection C of projections of 5 (&N-sets onto = notice under complementation. However, that A ao(A x Z) so that 5 i C. The main task of this section is to establish a pair of results required in the construction of measures on product spaces. =
.
3.19 Theorem If C and b are semi-rings of subsets of and E, respectively, then Sws = (CxD: C e C, D e 01 is a semi-ring of
x E.
Proof There are three conditions from 1.16 to be established. First, Ses clerly contains 0. Second, consider C1, Czq C, andDlyDz G 0. Q rn Czq CandDl rhDZ e D, and as a matter of definition, (C1 xD1) rn (Q xD
=
()
e
Q, (
c Dj
) fa (
e
Q,
(
e Dal
Measure
49
= ((o e cl c2, ( e ol = (c1ch ca)x (D1 cjD1) e Rwe. fazhl
fa
Third. assume that C1 xo2)
xDl i Cz xD2, and by a similar argument,
CI
(c1 xD1)
-
(3.45)
((
=
(
e Cz,
= ((C2
-
e D2): either
cl or (
)
D1l
C1) XD1) tp (C1 X (D2 -D1))
tp ((Q
C1) X (D2 -D1)).
-
(3.46)
where the sets in the union on the right-hand side are disjoint. By hypothesis, the sets Cz C1 and D1 DI are finite disjoint unions of C-sets and O-sets respectively, say (47,...,Cn'), and (D(,...,D,,,'). The product of a finite disjoint union of sets is a disjoint union of products; for example, -
-
n
n
U
/=1
C; x D1
(tl),():(t
=
Uts (
e
,
e D1
=
UICJx D1).
(3.47)
./=1
j=1
Extending the same type of argument, we may also write m
n
(ca cllxtoa-ol) -
-
Utclxol) U(c1xoi) k-p
kp
k=1
.j=1
U(cJxo;) (3.48) .
j,k
All of the product seys in this union are disjoint (i.e., a pair ((t),))can appear in at most one of them) and al1 are in Scs. This completes the proof. . The second theorem leads to the useful result that, to extend a measure on a productspace, it suffices to assign measures to the elements of Scs, where C and D are suitable classes of the factor spaces.
3.20 Theorem If 5 = c(C) and T c(O) where C and D are semi-rings of subsets of f and E respectively, then 5 @ V = C @ 0. =
Proof It ij clear that Rwo i Sps, and hence that C (8)D converse, consider the collection of inverse projections, .t/a (J:o-1(#), F e Fj pwg.
5 (8)W.To show the
c
=
,/p
is a c-field of fl x E, and is in fact the smallIt can easily be verified that is a (;o'(C), C e C J i C (>))0. est c-field containing the colletion and since it follows (& is and hence D c-field by the T l-system, a a theorem that c(#c) ? C @)0. Exactly same conclusion holds for #s, the orresponding collection for V. Evel'y element of Sss is the intersection of an element from #v and one from #'g, and it follows that Spg c C @D. But Sys is a l-system by 3.19 and hence a further application of the 7:- theorem gives 5 (8)N ? u gs; , .1
vt/c
,t/c
=
-system,
xt/p
'
=
irlik pztjlptlof > product extends beyond pairs to triples and general n-tuples, and gd jjjajj be jnterested in the nrooerties of Euclidean a-qnnne ffRlh '
y:
,;r-.
..
.. .
. .
.
.
jjtyj)jjjujf
50
Mathetnatics
For tinite n at least, a separate theory is not needed because results can be obtained by recursion. If (Ylf,X)is a third measurable space, then trivially, fxE
x'P
((,(,v):
=
(
e f,
(,)
= f((,(),v): (,() E) xT. = (f x
e E,
ve
e DXE,
v
t1:l
e .P1
(3.49)
Either or both of (f),T) and (E,V) can be product spaces, and the last two theorems extend to product spaces of any finite dimension.
3.5 Measurable Transformations
F: D
F->
(E,N) in a different context, as domain
and
Consider measurable spaces (f,T) and codomain of a mapping E.
if F -1(#) l 5 for a11# l N. The idea is that a to be sl%-measurable measure p, defined on (,T) can be mapped into (E,V), every event B e V being assigned a measure v(#) g,(F -1(.#)) We havejust encountered one example, the projection mapping, whose inverse defined in (3.44)takes each T-set A into a measurable rectangle. Corresponding to a mesurable transformation there is always a transformed in following the sense. measure, F is said
=
.
3.21 Theorem Let jt be a measure on (f,F) and F: f ation. Then P.F -1 i s a measure on (E V) where ,
jLF -1(#)
E a measurable
F-
transfonu-
,
-1
=
(#)), each
g,(F
B e g,
(gj5o
.
.
Proof we check conditions 3.1(a)-(c). Clearly g,F -1(4 ) 1 0 a1l A e Bv. Since holds by definition, T-1(O) T -1(E) O by 1.2(iii) and so g,F-1(3) p,(T -1(O)) g,(O) 0. For countable additivity we must show ,
=
=
=
=
=
g,F-1
for a disjoint collection that the B; are disjoint becomes
U#y j
=
X/tF-1()
(3.51)
j -1
F Bj4, 1.2 shows both Bj,B1,... e E. Letting B) and that F-1(Uy#y) Equation (3.51) therefore Uy#J. =
=
p,Ur j
=
X g,(#J)
(3.52)
./
for disjoint sets Bgq,which holds because g, is a measure. The main result o
general transformations
.
is the following.
3.22 Theorem Suppose F -1(#) l 5 for each B e D where D is an arbitrary class of sets, and N c4D). Then the transfonnation F is T/v-measurable. ,
=
51
Measure
(
$.,
.
.
.
,
.
,
=
,
j'
T-3LjBj)
T-3Bj)
q @,j q s, then proof By 1.2(ii) and (iii), if l then @ @ It follows d if z'-1(#r) z'-1(#)c z'-1(,) E e ?yan
,
k
=
UjF-1()
that the class of
CtS
a is a c-field.
Since D
t,: w-1(s)s s)
.
N c 4 by definition.
.4,
c
.
This result is easily iterated. If (t1#,J't)is another measurable
is a V/x-measurable transformation, then &oF: (
F-+
space and U: E
F->
VIJ
VP
is F/s-measurable, since for C -1
R, W 1(C)
E
-1
(&0D (0
F
=
N an d hence
E
,
-1
(&
(C))
9.
1
(3.53)
E is called a measurable isomorphism if it is An important special case: F: f' 1-1 oflto, and both F and F -1 are measura ble The measurable spaces (2,1) and (E,N) are said to be isomorphic if such a mapping between them exists. The implication is that measure-theoretic discussions can be conducted equivalently in either (f,T) or (E,V). This might appear related to the homeomorphic property of realfunctions, and ahomeomorphismis indeed measurablyisomorphic. Butthereis no implication the other way. -..:
.
r0,11 F--> (0,11,defined
3.23 Example Consider g: #(A)
x+
,1
0 f x f 1 2
x
1
l <
=
by
(3.54)
.
-
2,
1)
x < 1
Note that g is discontinuous, but is 1-1 onto, of bounded variation, -j Sgo,lj/fgo,lj-measurableby 3.32 below, and g g. I::I
and hence
=
The class of measurable transformations most often encountered is where the codomain is (R,O), B being the linear Borel tield. In this case we speak of a function, and generally use the notation f instead of F. A function may also have the extended real line (RlW)as codomain. The measurability criteria are aj
follows.
3.24 Theorem (i) A function /': f
R for which (): J(() f x) e ?Ffor each x E (Dis So is a function foz which (): fk < x) e 5 for each
F-+
T/f-measurable. x e (D (ii) A function fl f.1 F-> 2- for which f(t): Q tp (+x) t.p (-x) is T/f-measurable. .
.f4))
f xl e ?F for each x e -1(#)
Proof For case (i), the sets f(t): flk f xl are of the form f B e T' where C is defined in 1.21. Since B c(C), the theorem follows by 3.22. The other cllection indicated also genertes :B, and the same argument applies. The extension to case (ii) is equally straightforward. . ,
=
52
Mathentatics
The basic properties of measurable functions follow directly.
3.25 Theorem (i) If f is measurable, so are c + f and c#, where c is any constant. (ii) If f and g are measurable, so is f + g. Proof If .X
e
jq
f
K x, then
f+
c S x + c, so
that f
c is measurable
+
by 3.24. Also, for
,
(: (:
c/)
XJ
S
/*()
((t): /)
=
< xIc ),
c
>
0
< xlclc,
c
<
0
c
=
c
=
f1, 0,
where for each of the cases on the right-hand side and each x/c in 5, proving part (i).
If and only if lt follows that
(:
J())
f+g
+.g((t))
<
there exist r e ) such that
.x,
U (2
.z')
<
=
f(l
<
r)
f-n
(:
(3.55)
0 and x k 0 0 and x < 0
f
<
r
<
R the sets are
6
x g -
:(t0) < x- r1.
(see 1.10). (3.56)
rq Q
The countable union of T-sets on the right-hand side is an T-set, and since this holds for every x, pal't (ii) also follows by 3.24(i), where in this case it is convenient to generate tB from the open half-lines. >
Combining parts (i) and (ii) shows that if h,...,fn are measurable functions so is Xhjcjfj, where the cj are constant coefficients. The measurability of suprema, infima, and limits of sequences of measurable functions is important in many applications, especially the derivation of integrals in Chapter 4. These are the main cases involving the extended line, because of the possibility that sequences in R are diverging. Such limits lying in V are called extended functions. 3.26 Theorem Let f/l be a sequence of F/s-measurable functions. Then infn/n, supn/k, liminfln, and limsuw/ are F/W-measurable. Proof For any x e
t:
F (: fnlk
f xl s 5 for each n by assumption. Hence
,
supnlnt)
<
.z')
=
Of):
n=1
fnk
so that supn/ is measurable by 3.24(ii). Since infn/'n obtain (:
infn/t)
.x)
<
(: sup,,(-Jn())> -xJ = (: supn(-Jn())S -xJf =
@,
S xl e
-supnt-ln),
=
(3.57) we also
Measure
53 C
00
O
n=1
la''. -Tn(a')K
-xl
X
=U1e):/()
<
5.
xl e
(3.58)
n=1
To extend this result from strong to weak inequalities, write X
f xl
infn/kt)
(:
O(:
=
infn/kt)
<
x+
?n=1
Similarly to
(3.59)
5,
(3.60)
may show
(3.57),we
1:
1/rlJ e 5.
supknfkt)
f
Ot:
.x1
=
.fn()
S
.x1
e
kkn
and applying (3.59)to the sequence of functions gn (:
limsupn/t)
sums/k
=
yields
K xl e F.
(3.61)
In much the same way, we can also show f x) e @.
( (t): liminfn/t)
(3.62)
condition of 3.24 is therefore satisfied in each case.
The measurability
.
We could add that limn/kt) exists and is measurable whenever limsupn/t) liminfs/kt). This equality may hold only on a subset of f1, but we say fn converges a.e. when the complement of this set has measure zero. The indicatorfunction 1s()) of a set E e 5 takes the value 1s() 1 when (.t)E characteristichmction otherwise. and 1s((l)) 0 Some authors call ls E, of F. the It may also be written as fs or as zE. We now give some useful facts about indicator functions. =
=
=
3.27 Theorem (i) 1s() is 5IB measurable 1 1s(). (ii) 1sc()) (iii)1Ujsj((l)) sup 1sf(). =
if and only if E
G
F.
-
=
i
(iv) lnjsjt)
inf 1s/)
=
i
=
I-l1sj((l)). i
Proof To show (i) note that, for each B e .S,
1F-1
(#)
'Fhse
sts
ithediyte
=
if 0 G B and 1 G B E if 1 e #, 0 # Ec if 0 e B, 1 , B 0, otherwise
(3.63)
are in 5 if and only if E e @. The other parts of the theorem fwm tlw dfinition. .
re
Mathematics
54
Indicator functions are the building blocks for more elaborate functions, constructed so as to ensure measurability. A simplefunction is a T/o-measurable function J: fl F...: R having finite range', that is, it has the fonn
ftl
af, Xaflsjtt9l =1
=
=
tt e Ei,
(3.64)
where the G1,...,(xa are constants and the collection of T-sets F1,...,Fa is a tinite partition of f1. T/f-measurability holds because, for any # 6 B,
J-1(,)
U Ei s
=
ai
@.
(3.65)
B
EE
Simple functions are ubiquitous devices in measure andprobability theory, because many problems can be solved for such functions rather easily, and then generalized to arbitrary functions by a limiting approximation argument such as the following.
l
Fig. 3.1
3.28 Theorem If f is T/o-measurable and non-negative, there exists a monotone n e EN I such thatnlt) f sequence of T/f-measurable simple functions f/'(n), for evel'y (l) e f1. '
1,...,n2'l, consider the sets Ei = f ): i 1)/2' < il2n ) Proof For i f Augment these with the set Enzn.. f : J() k nl This collection corresponds to a n2n + l-fold partition of (0,x) into S-sets, and since f is a function, each and hence belongs to one and only one Ei. The Ff maps into one and only one therefore constitute a partition of f1. Since J is measurable, Ei s 5 for each i. Define a simple function hn)on the Ei by letting i i 1)/2/, for i = 1,..., nln + 1. Then fn) f f, but A+1())k /4)) for every ); incrementing n bisects n+1)()) is equal to either i 1)/2*, each interval, and if n)()) n)(), 1)/2'+1 &+1 > follows that the sequence is 1)/2 2/ It li or ( J. (). =
-
.
=
.
.f4)),
=
=
-
=
-
-
-
sieasure
55
J(). This holds for each ) e f. To extend from monotone, and lims-yxnlt) non-negative to general functions, one takes the positive and negative parts. Define J+ maxlf,ol and ff'b f, so that both f- and J- are non-negative functions. Then if f'Injand j'-n) are the non-negative simple approximations to it is clear that f+ and f- defined in 3.28, and fnj f =
=
=
-
.f+(n)
-(n),
=
-
l/-J(n)I < lJ+-.f+(n)l+ 1.f---(n)I Fig. 3. 1 illustrates the construction for n is a function on the real line.
=
0. w
--.h
2 and the case
(3.66) =
R, so that
.f()
3.6 Borel Fqnctions lf J is a measurable function, and 5'; S g R, 'T g R is a function of a real variable, is the composite function gof measurable? The answerto this question is yes if and only if g is a Borel hmction.Let Ss (# ros: B e S), where B is the Borel field of R. fs is a c-field of subsets of S, and B DS is open (closed)in the relative topology on S whenever B is open (closed).in R (see 2.1 and 2.3). fs is called the Borel field on S. Define fv similarlywith respect to 7. Then g is called a Borel function (i.e., is Borelmeasurable)if g -1 B) e B s for all sets B l f.r.
gl S
F-+
=
3.29 Example Consider gx) Ixl g -1 ta kes e ach point of R+ into the points x is the set For any B E B + (th e res triction of B to R+) the image under and for each a7 e #, which is an element of f. E1 containing the points x and =
.
.g-1
-x.
-x
1 if is rational, 0 otherwise. Note that (D e B (see 3.30 Example Let g(x) -1 3.15), and g i s defined according to (3.63)with E Q, so g is Borel-measurable. I:a .:
=
=
non-measurable ln fact, to construct a following. but the obvious case is the
4plausible'
function is quite difticult,
3.31 Example Take a set A f; for example, let A be the set H defined in 3.17. l Now construct the indicator function 1x@): R F-> (0,1 ) Since 1x - (f 1 1) A gp Shis function is not measurable. Ia .
=
,
Necessary conditions for Borel measurability are hard to pin down, but the following suftkient conditions are convenient.
3.32 Theorem lf g'. S it is Borel-measurable. Pmof
(i) follows
F-+
'T is either
immediately j cotinuity implies that h B) in T, by 2.17. To prove (ii), consider first prperty h@l S @)when y < ..
(i) continuous
or
(ii) of
botlnded variation,
from 3.22 and the definition of a Borel field, since is open (closed)in S whenever B is open (closed) a non-decreasing function h: R F.-y R, having the x; if A (y: h@l S (x)), sup A = x and A is one =
Mathematics
of (-x,x) and (-=A, so the condition of 3.24 is satisfied. So suppose g is nondecreasing on S; applying the last result to any non-decreasing h with the property hx) = x e S, we have also shown that g is Borel-measurable because = -1 h-'B4 5') f''ASe Ss, for each Brh'T e O.r. Since a function of bounded g CBt''h variation is the difference of two non-decreasing functions by 2.20, the theorem now follows easily by 3.25. . '(x):
This result lets us add a further case to those of 3.25.
3.33 Theorem If J and g are measurable, Proof
jg
=
with 3.25(ii).
z1((J+:)2
-
fl
gl)
-
so is
fg.
and the result follows on combining 3.32(i)
.
The concept of a Borel function extends naturally to Euclidean a-spaces, and indeed, to mappings between spaces of different dimepsion. A vector function 7,' S i R 1- R'/ gl S --./
,
is Borel-measurable if g -1(#) e B s for a11B e Sv, where Bs and fv (# r-75': B e f'''J.
=
(# taS:
# e
l
=
3.34 Theorem
lf g is continuous. it is Borel-measurable.
Proof By 2.21. w Finally, note the application of 3.21 to these cases.
3.35 Theorem If jt is a measure on (R Bk) an d g : S s.y is Borel-measurable where S c Rk an d -Erc R'/ gg is a measure on (T,Sv) where 'r
,
-1
,
gg-1(B)
t-1(s))
=
(3.67)
,
for each B e f.r. n R'N
for m < k. If X is A simp le example is where g is the projection of R onto k x 1 with partition X' = (21,7+'.),where X. is m x 1 and X.. is k m) x 1, 1et gl R k F-> R*' be defined by -
r(A') -1
In this case, g,g B)
=
X..
-1
=
P.W(#))
=
jt(#
xR
(3.68) k-m
)
or p s qrrl .
4 Integration
4. 1 Construction of the lntegral The reader may be familiar with the Riemann integral of a bounded non-negative function f on a bounded interval of the line gJ,:J, usually written fdx. The objects to be studied in this chapter represent a heroic generalization of the same idea. lnstead of intervals of the line, the integral is defined on an arb-
itral'y measure space. Suppose
(f1,@,g)is :
l'1
O
a measure space and
2-6
is a T/s-measurable function into the non-negative, extended real line. The integral of f is defined to be the real valued functional
jfdb
Sup
=
tXj
fte)ljpfj
tinsfo
(4.1)
where the supremum is taken over a11finite pmitions of into sets Ei e T, and the supremum exists. lf no supremum exists, the integral is assigned the value +x. 5 T he integral of the function IAJ,where 1x() is the indicator of the set 4 e 5, is called the integral of f over A, and written fdbk. The expression in (4.1) is sometimes called the lower integral. and denoted sfdp. Likewise defining the upper integral of f,
j'ydb -
inf
E i
(supsy/j bte'i)
(4.2)
,
G
f from below and from we should like these two constnzctions, approximating .fdbk J'/W/whenever above, to agree. And indeed, ttis possible to show that f *fdbk oo if either the set > 0) has is bounded and g(f1) < x. However, (: infinite measure, or f is unbounded on sets of positive measure. Definition (4.1) is preferred because it can yield a finite value in these cases. =
.f()
=
4.1 Example A familiar case is the measure space ('R,f,-), where m is Lebesgue where f is a Borel function is the Lebesgue integral of measure. The integral Ifdm f. This is customarily written j'fdx,reflecting the fact that mx, x + #x1) tfx, even though the sets (A'j) in (4.1) need not be intervals. n =
4.2 Example Consider a measure space (R,f,g,) where g, differs from m. The where f is a Borel function, is the Lebesgue-stieltjes integral. integral Ifdbk'
Mathematics
58 The monotone function Fxt
=
p,((-x,x1)
(4.3)
F(b) F(c), and the measure of the interval has the property (x, x + #< can be written JF@). The notation IfdF means exactly the same as J/Wg.,the choice between the p. and F representations being a matter of taste. See j8.2 and j9.1 for details. n g,((J,:1)
=
-
For a contrast with these cases, consider the Riemann-stieltjes integral. For an 1et a partition into subintervals be defined by a set of points H interval Etz,!71, Akl, < < xn with a = xo < b. Another set H' is called a = (x1,..., refinement of Fl if IR c H'. Given functions f and a: R 1-+ R, let .x1
=
...
5lRvas.fl
=
77f(f)(G(A)
-
a(zf-I)),
(4.4)
i=1 bafd%,
here ti e (xf-l,xf). If there exists a number there is a partition IQ with W
(&l-I.a,.f)
'-jajd
such that for every : > 0
b < 6:
l
for all FI Q IQ and every choice of fff), this is called the Riemann-stieltjes integtal of J with respect to a. Recall in this connection the well-known formula for integration by parts, which states thqt when both integrals exist,
fbtub)
=
/J)a(J)
+
jafd ja@f' +
When (x = x and #is bounded this definition yields the ordinary Riemann integral, and when it exists, this always agrees with the Lebesgue integral of f over gtz,:). Moreover, if a is an increasing function of the form in (4.3),this integral is equal to the Lebesgue-stieltjes integral whenever it is defined. There do exist bounded, measurable functions which are not Riemann-integrable (consider3.30 for example) so that even for bounded intervals the Lebesgue integral is the more inclusive concept. However, the Riemann-stieltjes integral is defined for more general classes of integrator function. In particular, if f is continuous it exists for a of bounded not necessarily monotone. These integrals therefore fall variation on gtz,&), the class defined by (4.1), although note that when a is of bounded variaoutside tion, having a representation as the difference of two increasing functions, the Reimann-stieltjes integral is the difference between a pair of Lebesgue-stieltjes
integrals on (J,:1. The best way to understand the general integral is not to study a particular measure space, such as the line, but to restrict attention initially to particular classes of function. The simplest possible case is the indicator of a set. Then, every partition fEiI yields the same value for the sum of terms in (4.1),which is
jgdn
Integration
59
jxdp
(4.6)
BIX),
=
=
5, the integral is undefined.
for any A e 5. Note that if A
Another case of much importance is the following.
4.3 Theorem lf J
0 a.e.lgl, then
=
fdp
Proof The theorem says there exists C e C. For any partition (Fl,...,&J 1et A'; ity of g,,
77 i
inf (l) e
Ei
/.() g,(e)
=
Xi
inf (l) e
F;
=
0.
1, such that /4)) 0 for ) Ei rn C, and A': Ei F;. By additiv-
with
gtc)
=
=
=
=
/.()l g,(e':) +
Xi
inf f e e:
-
g,(rJ)
= 0,
(4.7)
the first sum of terms disappearing because /*4) 0, and the second disappearing =
by
.6(i)
since pE'i) f g,(C')
0 for each i.
=
.
A class of functions for which evaluation of the integral is simple, as their name suggests,is the non-negative simple functions.
4.4 Theorem Let (?() 5 is a partition of f.
=
where aj k 0 for X:=1%1sj()),
i
=
1,...,n, gnd EL,...,En E
Then
tFp, 77
(4. 8 )
ipEi).
=
=1
Proof Consider an arbitrary finite partition of f1, A1,...,4-, and define inftxxyst). Then, using additivity of g, Ip,(Ay) X SX 77 f=1 .j=1 j=3 =
<
i=1
xi
I
=
p,(AyraEi4
jtt/tjc
Ei4
j=L
(4.9)
ipEi),
= f=1
where the inequality uses the fact that I assumes the smallest value of Gj such that Aj ra Ei # 0, by definition. The theorem follows, given (4.1), since (4.9) holds as an equality for the case m n and Ai = Ei, i 1,...,n. w =
=
So for functions with finite range, the integral is the sum of the possible values of f, weighted by the measures of the sets on which those values hold. Look at Fig. 3.1. The Lebesgue integral of the approximating function fz) in the figure is the sum of the areas of the rectangular regions. To compute the Lebesgue-stielties inteeral with resoect to some measure u.. one renlnces the width of the qetq
62
Mathematics
= yaj
a simple function. Hence,
ja-bdb
+ tzj,
s Af ra Bj,
(t)
(4. 17)
X Xtla, +bbbviAil-bB?
-
j
i
=
zxGf7r/ttArnl
=
'7rafgtA
=
i
,xfs>7/ttAf
+
j
j
rn B?
i
bTfbbkB)
+
j
i
ajdp
+
bjydp,
(4.18)
showing that linearity applies to simple functions. Now applying 4.6,
jaf
+
bgldp
ja jygjtj jt?lj1jsyusp, (suspy +
sup
=
bz4dbk
(pf/f'
=a =
+
b
#
ajfdp
+
bjgdp.
(4. 19)
To extend the result to general functions, note that
la.f
1 If I + Ib I Ig l
bg l < a I
+
.
.
(4.20)
,
so (4.19) shows that af + bg is integrable so long f and g are integrable, and a and b are finite. The identity af
+
bg
=
afl'
afl-
-
bg4'b (bg)-
+
(4.21)
-
implies, applying (4.19), that
jaf
bntdb
+
If a k 0, then
ab+dk -
=
jaldb
afl+dbk
aft-dbk
=
J(&T)-*+ Jt/WI+dB
-
-
afl-d
Ia I f-db
=
-
Jtf/'+lt
f+db
-
=
ment applies to the terms in b and g. So
JJ-Jg,)
IJI
-fd
'-jb8t-db.
=
(4.22) ajfdbk, whereas if a < 0, alfdb. The
=
(4.16) holds as required.
same argu.
is a very useful property. The first application is to show the the integral to the behaviour of functions on sets of measure 0, of invariance extending the basic result of 4.3. Linearity
4.8 Lemma Let f and g be integrable functions. (i) If f f g a.e.lg1, then fdkt; hdp. dp. (ii) If f = g a.e.gg,l, then J/Wg,= Proof For (i), consider first the case J 0. lf g 2 0 ekerywhere, directly from (4.1). So suppose # k 0 a,e.rg,l and define =
dbk 2 0
63
Integration
Jl4)) Then h
0 a.e.gp,) but g
=
jg
0 S
fhdbk0andby
since
J('
To prove
JJJF
2 0
=,
#()
<
.
0 4.7,
jgdp jhdpjgdp, +
=
(4.23)
=
0,
(ii), let
h
J(' + hjdbk
=
:()
h k 0 everywhere, and, applying
hjdp
+
0,
4.3. Now replace g by g J in the last argument to show hence dbkk J/Wptby 4.7.
=
-pdp
+
=
-
=
dp
=
f
-
+
0 by 4.3. Then g so that h 0 a.e.gg,l, and hdp hdbk J#JF, Where the second equality is by 4.7. w =
=
=
These results permit the extension to the more commonly quoted version of the monotone convergence theorem.
4.9 Corollary lf fn k 0 and fn
'1'
limn-oxl/nv
J a.e.g.l,
=
jdp. n
T
Another implication of linearity is the following.
j
Proof
j lf 1 jf'
inequality
4.10 Modulus
lJ lJz
Jjt 2
+#-)tfp,
=
j jjj'dp jf jf'bdp .
-#p,
=
+
Jy-#>l jJ/-d: )Jy::1. .
-
-
In the form of 4.9, the monotone convergence
theorem has several other useful
corollaries. 4.11 Fatou's
lemma If
Jn k
0 a.e.Lg1,then
liminf fndp k nMx
j
liminf
Jn Jg,.
n->oo
Proof Let gn infkn/'k, so that (,n) is a non-decreasing sequence, and gn 1% g sides oo 2 Igndp. Letting n both of liminfn/n. Since Jn k gn, Jfntkyt the on inequality gives =
=
--+
liminf fndp 2 lim gndp n.-+txz
=
gdp
liminf
=
n-M
4.12 Dominated convergence theorem If fn
--
that IfnI
fn #p..
(4.24)
w
n--x
S g a.erlp1 for al1 n and
gdbk <
x,
f a.e.lgj, an there exists g such then JlnIg, .-A JJJP,.
Proof According to 4.8(i), dp < x implies JlfnI#g, < =. Let hn Ifn f I such that 0 f hn K 2g a.e.lg,j and hn 0 a.e.ggj. Applying 4.3 to liminfns, linear=
-->
jw.
.t.......:.')..:..
t.............( .) ..... .:
.......
Fatu' s lymma, attl .).( ,
.'). '
..
.(..
.
...
E .y. ... . .
..
.
.. .
j.... ..
.
-
,
64
Mathematics
ljgdp
J
=
=
liminftzg
-
j (lg jhndp
hn) Jjt f liminf
-
hnjdp
N->x
n-/co
jgdp
limsup
-
(4.25)
,
N-+=
where the last equality uses (2.4). Clearly, limsupn-yxf/lnzv k 0 the modulus inequality implies 1im n-e
=
0, and since
jfndp j/'ljt jhndp < 1im
-
=
0.
j'hndb (4.26)
.
n'M
Taking the case where the g is replaced by a tinite constant produces the following version, often more convenient:
4.13 Bounded convergence theorem If fn then limn--jxlfntfg, Ifdt < x. E1
--y
f a.e.gg1 and IA! f
B<
x
for all n,
-->
Theorem 4.7 extends by recursion from pairs to arbitrary finite sums of funcPut fn tions, and in pmicular we may assert that J(Z:=1'i)Jg, = Xl=tgiand fndbk l=tlgid, where the gi are pon-negative functions. Then, if fn -1xf Z7=l'j < x a.e., 4.9 also permits us to assert the following.
Z:=1J##g..
=
=
=
4.14 Corollary
If
f'fl is
a sequence of non-negative functions,
J Ng'
,
-
# jg,
.
(4.27)
o
i= 1
fcz1
By implication, the two sides of this equation are either both infinite, or finite and equal. This has a particular application to results involving c-finite measures. Suppose we wish to evaluate an integral gdkkusing a method that works for finite measures. To extend to the c-finite case, choose a countable partition igi, lffl of f1, such that g(fk) < x for each Letting gi loy, note that g iidp by gdp (4.27). and =
.
=
=
4.3 Product Measure and Multiple lntegrals Let (f,@,g) and (E,V,v) be measure spaces. ln general, tfx E, 5 @ N, 1) might also be a measure space, with 7t a measure o'n the sets of 5 @ V. In this case respectivlly, are measures g andv, defined by g,(#) 7:(Fx E) andvtG) a(f1 x G4 called the marginal measures conrsponding to T:. Alternatively, suppose that g and v are given, and define the set function 1: Sys F-> 2-+, =
=
where Sss denotes the measurable rectangles 7:(Fx G)
=
g,(&v(G).
of the space f x E, by
(4.28)
We will show that 1: is a measure on Syg, called the product measure. >nd has an extension to 5 * T, so that tfx E, 5 (8) V, a) is indeed a measure space. The tirst
Integration
65
step in this demonstration is to define the mapping F(o: E F-> x E by Fo(() (,(), so that, for G e V, F(o(G) f )) x G. For E e 5 @ V, let =
=
Eo
F;1(F)
=
((: (,()
=
e F)
c
E.
(4.29)
The set Eoj can be thought of as the cross-section through E at the element ), any countable collection of 5 @ V-sets (Ej, j e EN
U
((:(t,),t)
=
j
For future reference,
U)
e
Ul(:(,()
=
e
./
j
EjI
=
U().
the following.
note
e 5 (& V. If F=Fx
Proof We must show thate'to e N wheneverf G e N, it is obvious that Eo, =
c(Sps),
(4.30)
j
4.15 Lemma Toj is a W5 @ Vl-measurable mapping for each
Since 5 (&N
For
(t.
G,
E
F
e
F
(l)
e f1.
G for Fq 5 and
e N.
=
3,
(l)
(4.31)
the lemma follows by 3.22. w
The second step is to show the following.
4.16 Theorem
';:
on Sps.
is a measure
0, recalling that Fx O O x G O Proof Clearly JE is non-negative, and <0) for any F e 5 or G e V, and applying (4.28).It remains to show countable additivity. Let fE. e Sps, j e (N) be a disjoint collection, such that there exist sets Fj e 5 and Gj E V with Ej F) x Gj; and also suppose E VjEj e Spg, such that there exist sets F and G with E Fx G. Any point (,() e Fx G belongs to one and only one of the sets Fj x Gj, so that for any (t) e F, the sets of the subcollection fGjI for which ) e Fj. must constitute a partition of G. Hence, applying (4.30)and (4.31), =
=
=
=
=
=
vE.)
=
v
U j
U 3, j
U()(o j
Gj,
=v
v
=
)
e Fj =
(t)
e Fj
Xj 1,())v(%),
(4.32)
where the additivity of v can be applied since the sets Gj appearing in this dcomposition are disjoint. Since we can also write vtfo) v(G)1A)), we find =
l(F) .f....j....L..r
. ... . .
....
.
:
.
..
.
). .
....
.
=
g,tFlvtGl
=
JY(FV>())
=
J
X 1z)(t0)V(f) 1
tf/t*l
66
Mathentatics
= as required,
Xp.(F?)v(t7;)
Xz:(e;.) j
=
j
(4.33)
the penultimate equality is by 4.14. w
where
It is now straightforward
4.17 Theorem ( x E, 5
to extend the measure from R,g to 5
e N, J:)
(& V.
is a measure space. hence Svs, is a semi-ring by
5 and V are c-fields and hence semi-rings; 3.19. The theorem follows from 4.16 and 3.8. . Proof
Iterating the preceding arguments (i.e. letting (,T) and/or (E,V) be product spaces) allows the concept to be extended to products of higher order. In later chapters, product probability measures will embody the intuitive notion of statistical independence, although this is by no means the only application we shall meet. The following case has a familiar geometrical intemretation. Lebesgue measure in the plane, R2 R x R is detined for intervals
4.18Example by
=
-((Jl,:11
x
(J2,&21)
=
b1
-
J1)(:2
,
-
(4.34)
J2).
Here the measurable rectangles include the actual geometrical rectangles (products of intervals), and o the Borel sets of the plane, is generated from these as a 2 is a measure space in consequence of 3.20. By the foregoing reasoning, (R2,B which the measure of a set is given by its area. n 0A,
,m4
We now construct integrals of functions /*((t),() on the product space. The following lemma is a natural extension of 4.15, for it considers what we might think of yielding a function with as a cross-section through the mapping at a point (J) e domain E. ,
4.19 Lemma Let f:D x E F-> R be 5 @)Ws-measurable. Define Then f(0:E F- R is V/o-measurable. fixed ) e
.f(o(j)
=
A,tlfor
.
Proof We can write
f(()
=
f(eD,()
=
f(Fo(())
=
/oF(().
By 4.15 and the remarks following 3.22, the composite function measurable.
(4.35) is MIB-
fo
K
Suppose we are able to integrate f with respect io v over E. There are two quesJE/'IWV tions of interest that arise here. First, is the resulting function gl F/plmeasurable? And second, if g is now integrated over f1, what is the relationship between this integral and the integral Inxzfd:Over ( x E? The affirmative answer to the first of these questions, and the fact that the integral where these exist, are fiterated' integral is identical with the the most important results for product spaces, known jointly as the Fubini theorem. Since iterated integration is an operation we tend to take for g'ranted xxrltla mllltlnlo piemnnn nerhnns the main Doint needin: to be stressed infnyrnlq =
tdouble'
Integration
67
here is that this convenient property of product measures
(and multivariate
Lebesgue measure in particular) does not generalize to arbitrary measures on product spaces. The first step is to let f be the indicator of a set E e 5 (& V. ln this case f is the indicator of the set Efo defined in (4.29),and
J/'olv
v(F(o)
=
+(e)),
=
(4.36)
2-+is well-defined, say. ln view of 4.15, Efo e V and the function #s: fl values unless in the extended although, v is a finite measure, it may take its shown. line, half as .-.:
(& V,
4.20 Lemma Let jt and v be c-finite. For all E e 5 and Jo#FdB TCZ')'
g; is T/p-measurable
(4.37)
=
By implication, the two sides of the equality in or finite and equal.
(4.37)are
either both infinite,
Proof Assume first that the measures are finite. The theorem is proved for this theorem. Let denote the collection of sets E such that gz case using the satisfies (4.37).Spv i ,W,since if E Fx G then, by (4.31), .4
';:-
=
+() and gzdbk fl x E e 1sz- lsj,
=
(4.38)
Clearly g(#)v(G) l(F) as required. We now show is a = and since holds. lsz-sj 1.25(a) lf F1 c Ez, then, Ej,E2 e so -system.
=
.4
Js1/,()JV(()
gE,-EL4
an 5lT measurable
Js1e'1(t0,()#V(()
=
=
-
'sztt,)l
(4.39)
-%(),
function by 3.25, and so, by additivity of
'n,
.
jgjgh-zbdbk
.'
Y,
F e
.4
.4,
' r' 1*
v(G)1F(),
=
l(F2)
=
-
r(F1)
=
zE2
-
F1),
.VL.
t
(4.k0)
''
showing that .Wsatisfies 1.25(b). Finally, If 41 and A2 are disjoint so are (A1)(o + gy). and (A2)(o,and ujwxct)) u1((l)) To establish 1.25(c), let (. e W, with be IN Ej j e J a monotone sequence, F. Define the disjoint collection (Aj) = With A1 q= E$ and Aj Epj A),j > 1, so that E U7=3Ajand Aj e Wby (4.39). additlvity of v, y ountable
(
=
't
=
-
x ','(l
=
Xuytl.
yj -.t. .E )r-. )..' L.k,.j-L);(b. ..r.),).).t - ; )--'.ti't(.. (yyyE (;()t. .(.yy')-....qy . r. tf''Ett..t.q'? ;t-yjjy:jj(j .yjjjy.;gjjyjjjjjt.. .y yy. q jtyjjj yyyj,;( jqqj)q;((. .!k jyy .ryLLI. gjyj)j. jjrjjk yy. ... jj;gjr j.yy.j.jjjyrr jjygj. yjjjyj $kj)4 yryt;y.)...-i.r..t.E)). .E 7..y jj ... y..q-().r-(.t(. ...-.-!;y..:. ,.-. .. gyjj,. . r) r.q.r t. #t-.-y 1l(!(l:l.)(.. ..((r-. )-)t.. . t.-... .. .. ' ;.1 . . . . . . ... . . .. . . . ((
')'
..t.-)
-'.t.t
-..(q..)--r;:,)j);.tr-ry
.
..
..q
..
y..jt..j. . ;fj,jL)qq;... )) jtjj(j)q..... ..y,r(.jk
-.-
.li.
:
--''
-
- - .
-..)-.-.))t:..,.
-..
.
...
.
tj!jj jj.(gjrj
(4.41)
68
Mathematics x
x
Jo
X'At)
tfBtt0l
g
Xj j=1
=
j=j
v
x
o'Atttltfp,tl
X5(X/)
=
=
1(13,
(4.42)
j=j
n
.d
Since Sys is a where the first equality is by 4.14. This shows that is semi-ring it is also a a-system, and 5 @ N c(Sys) c by 1.27. This completes the proof for finite measures. To extend to the c-finite case, 1et J and (Ey) be countable partitions of f tff and E with finite g-measure and v-measure respectively', then the collection tff x Ej e Sys ) forms a countable pmition of x E having finite measures, attf x Ej) = g.tfvtWl.For a set E l 5 @V, write Eq = E r7 tf x Ej). Then by the last argument, -system.
.4
=
ghdbk
=
where gzql fk disjoint and
F-> 's((l))
nEq),
(4.43)
R+ is defined by +,y((l)) v((F(j)(J, (t) e fk. The sets Eq are vjLjjl whep (l) e ff, or =
=
gE
=
X 1f4()X+/).
(4.44)
j
i
The sum on the right need not converge, and in that case g/ F/s-measurability holds by 3.25/3.26, and
jcysdb -
-
Jo
Xjl XyS's
XXJO:S/B i
j
*'
-
'*'l
+x.
=
However,
Jp, XXn(A) i
-
(4.45)
nk,
j
using 4.14 and countable additivity. This completes the proof. Now extend from indicator functions to non-hegative
.
functions:
4.21 Tonelli's theorem Let 7: be a product measure with c-finite marginal measures g and v, and 1et fl x E F-> R+ be (@(&Vl/f-measurable. Detine functions .f:
f1
= R+ f(,(), is T/f-measurable, (i) g JsfA
E
F->
(ii)
b-V(()
jtstcafdx Jo =
and let gk
=
hfuA.
Then
tfB. E1
ln part (ii) it is again understood that the two sides of the equation are either finite and equal, or both infinite. Like the other results of this section, the and (E,N,v), and the complementary results given theorem is symmetric in (f1,@,g) roles interchanging of the marginal spaces do not require a sepapte stateby the ment. The theorem holds even for measures that are not c-tinite, but thij further complicates the proof. Proof This is on the lines of 4.6. For a partition (F1,...,Fn J of 5 @)T let f Xwlsj, and then f Xii ltsjl(sand g = Xiqis'lEil by 4.4. g is F/p-measurable
=
=
Integration
69
by 3.25, and 4.20 gives vedb gdb Xm fl i
Xaine'i)
=
=
fl
=
j
AE
j'dn,
(4.46)
so that the theorem holds for simple functions. For general non-negative f, choose a monotone sequence of simple functions converging to f as in 3.28, show measurability of g in the limit using 3.26, and apply the monotone convergence theorem. w Extending to general
f
requires the additional
assumption of integrability.
4.22 Fubini's theorem Let a be a product measure with c-finite measuresg, and v; 1et #: fl x E >-> R be (; @ Vl/f-measurable with
marginal
Jzxsl.ftt,)'tllllt,ll <
(4.47)
=;
Then and let g define f)1 E >-> R by Jsffzyfv. (i) f is N/f-measurable and integrable for (z) e A f2, with g,tfl A) (ii) g is F/f-measurable, and integrable on A; .f4,();
.406)
=
=
=
-
JsTto'cldvt'l joxsft,clflttt,'l Jo
(iii)
0',
dpt*'
=
Proof Apart from the integrability, 4.19 shows (i) and Tonelli's Theorem shows (ii) and (iii) for the functions f' = maxl,o) and f- = f'b f, where I f+ + f-. But under (4.47),I I < on a set of a-measure 1. With A defined (i), (ii) and (iii) hold for f'b and as the projection of this set onto finite of the equation in sides with both (iii). Since f f'b f (i) extends to f by 3.25, and (ii) and (iii) extend to f by 4.7. . .41
=
-
.f(),()
'x
-,
,
-,
=
-
4.4 The Radon-Nikodym Theorem Consider c-finite measures g. and v on a measurable space (f2,1).g, is said to be absolutely continuous with respect to v if v(A) 0, for E e 5, implies g(A-) 0. This relationship is written as g <( v. If g tt v and v <( go the measures are said to be equivalent. If there exists a partition (A,A ) of f1, such that g(A) 0 and v(AC) 0, then g and v are said to be mutually singular, written g, v. Mutual singularity is symmetric, such that p. v means the same as v g,. The following result defines the Lebesgue decomposition of g, with respect to v. =
=
c
=
.1.
=
.1.
.
4.23 Theorem If p, and v are c-tinite measures, there exist measures g,1and p.2 such that g = gl + ga, g,l v, and p.2
lf there is a function #: fl 1..-> F+ it follows fairly directly (chooseE such that v(F) 0) such that g(F) Js/tfv, that g, << v, and f might be thought of as the derivative of one measure with =
=
70
Mathematics
respect to the other; we could even write f dbjldv. The result that absolute continuity of g, with respect to v implies the existence of such a function is the =
theorem.
Radon-Nikodym
4.24 Radon-Nikodym theorem Let v and g,2be c-finite measures and 1et p,2t< v. There exists a T/p-measupble function fl f F-y 2-+such that g2(F) fdv for all E e 5. u =
f is called the Radon-Nikodym derivative of g, with respect to v. If g is another for all E e 5, then v(.f #: g) 0, otherwise at such function and g,2(A-) Jsgtfv and E1 (: least one of the sets Ej (): ()< gl ) must contrathe detinition. dict =
=
.j'((l))1
.f4)>
=
=
Proof of these results requires the concept of a signed measure.
4.25 Definition A signed measure on (f,F) is a set function z: 5 F-> R-
satisfying 0. (a) :(0) j) (b) z(U#. XgAj) (c) Either z < x or =
=
disjoint collection (Ay e 5
for any countable, u z>
For example, let g, and v be non-negative measures on a space least one of them finite. For a non-negative constant r, define Z(X)
for any
)
.
-x.
=
((, @), with
p,(A) rvtAl
at
(4.48)
-
A e 5. For disjoint (Ay)
,
zj+l (-s) (x/) -
j=3
p,
-
y=1
= Xtg,tll
rv
/=1
-
rvtAjll,
(4.49)
i=1
so that countable additivity holds. If A is a F-set with the property that # e 5 with B c A, A is z(#)k 0 for every complementary negative defined in the called apositive set, a set being manner. A null negative both is positive and is called careful Be that set. to distinset a guish between positive (negative,null) sets, and sets of positive measure (negative measure, measure zero). A set has measure zero if g,(A) rvtz4l in (4.48), but it is not a null set. By the definition, any subset of a positive set is positive. The following theorem defines the Hahn decomposition. :4
=
4.26 Theorem Letx be a signed measure on a measurable space (f,1), having the property x(A) < cxo for a1l 4 e 5. There exists a partition of f into a positive set
A+ and a negative
set A-
Proof Let sup x(A), where the supremum is taken over the positive sets of x. A+ Choose a sequence of positive sets (Anl such that lima-ox z(Aa) o and 1et z4+ A+. measurable consider E Undn.To show that is also a positive set, any Iaetting B, A1 and B- A- -A--1. n > 1. the seuuence $Bnl is disioint, positive ,
=
=
=
=
=
Integration
71
since Sa c A,i for each n, and UnSn = A+. Likewise, if En
Ecj Bn the sequence fEn ) is disjoint, positive since En c #n, and Unk E. Hence zlE') XnzEn) 0, and since E was arbitrary, A+ is shown to be positive. A+ -An being therefore =
=
=
positive, Z(X+)
Z(X2
=
Z(Z+ -A
+
k Z(Xu), a1l n,
(4.50)
z(A+) k 1, implying z(A+) 1. Now let A- f -A+. We show, by contradiction, that A- has no subset E with positive measure. Suppose there exists E c A- with zE4 > 0. By construction E and A+ are disjoint. Every subset of A+kJ E is the disjoint union of a subset of A+ with a subset of F, so if E is a positive set, so is A+ t? E. By definition of
and hence
=
=
X,
,
2
b-)E4 z(z4+
:+z(A),
=
(4.51)
which requires x(A>) 0, so E cannot be a positive set. If F is a subset of F, it is also a subset of A-, and if positive it must have zero measure, by the argument just applied to E. The desired contradiction is obtained by showing that if zE) > 0, E must have a subset F which is both positive and has positive measure. The technique is to successively remove subsets of negative measure from E until what is left has to be a positive set, and then to show that this remainder has positive measure. Let ?z1 be the smallest integer such that there is a subset F1 c and detine F1 E F1. Then let nl be the smallest integer E with z(F1) < In general, for k = 2,3,..., such that there exists E1 i F1 with z(F2) < 1et nk be the smallest positive integer such that Fk-j has a subset Ek satisfying Z(f) < -$Ink, and let =
-1/n1,
=
-
-1/n2.
Fk
E
=
-
U). j=
(4.52)
lf no such set exists for finite nk, let nk +x and Ek = 0. The sequence (Fk) is x. non-increasing. and so must converge to a limit F as k =
---
We may therefore write E Fb..l (U7=1A), where the sets on the right-hand are mutually disjoint, and hence, by countable additivity, =
oo
z(f)
:4#')
=
+
side
Cr
Xz(A)
<
k=1
llnk. zb7 X k=1
(4.53)
-
Since x(F) > 0 it must be the case that x(#)> 0, but since zF < oo by assumptoo as k x. < x, and hence nk This ion, it is also the case that Z7=1(1/ak) with and negative is therefore a positive measure, means that F contains no subset . having positive measure. set -->
--
For any set B e 5, define Z(#) z(A+ rn B4 and z-B) -z(A- fa #), such that x+(S) z-B). It is easy to verify that zB4 z+and x- are mutually singular, non-negative measures on (f1,F). f z- is called the Jordan decomposition of x a s ig ned measure. z+and z- are called the upper variation and lower variation of =
=
=
-
=
a,
wxvxa
+1-.zk
maooxa...x
l .,
I
l,+
-
..u
%,-
lo
-
zxollxcxzl
tlao
tntnl
a/mrizyimn
nf
q;
Tho
Tnrdnn
Mathematics
72
decomposition shows that all signed measures can be represented in the form of (4.48). Signed measures therefore introduce no new technical difticulties. We can integrate with respect to z by taking the difference of the integrals with respect V to x an (j y We are now able to prove the Radon-Nikodym theorem. It is actually most convenient to derive the Lebesgue decomposition (4.23)in such a way that the Radon-Nikodym theorem emerges as a fairly trivial corollary. It is also easiest to begin with tinite measures, and then extend the results to the c-tinite case. -
.
4.27 Theorem Finite, non-negative measures v and g have a Lebesgue decomposition g /1,1+ g2 where gl v and p,2 << v, and there exists an F-measurable R+ F-+ f such that g,2(F) function /': fdv for a11 E e 5. .1.
=
=
Proof 6 L e t G denote the class of al1 F/f-measurable functions gl f F-> R+ for which IEgS S p,('), a11 E e @. 6 is not empty since 0 is a member. Let (x sup#esf'tfv, so that (x f g,tfM< x. We show there is an element of G at which the supremum is attained. Either this element exists, and there is nothing further to show, or it is possible by detinition of a to choose an element gn of G satisfying f a, for each n G IN.Generate a monotone sequence (/k,n e IN) in G tx 1/a f Igndv follows. Put so that fn 2 max(A-1((z)), J1 :1 and define fn by fn as > %()) for n 2,3,..., and then if fn-l A-l. Define the sets Aa (: e G, =
-
,w()),
=
=
.fn-1((l))
=
Xtfv
=
fn-tdv +
=
F
Erun
Fcxq
gndv
g,(FrnAn) + p.tf f'7Ak)
=
g,(f)
(4.54)
so that fn e G. Since fn 1 f, it follows by the monotone convergence theorem that IEf* S h1,(0, and hence f e G. And since fn gn so that fndv --4
a we must conclude that
-
Lln S
J.ftfv a, =
g,2tD
=
gndv S
fndv S a, Now detine g,2 by
as was required. dv, E
C
F'
(4.55)
Evidently g,2 is a non-negative measure (for countable additivity consider the and use 4.14), and also g,2 q v. Define g,1(e')= g,(f) 41,24f), functions which is non-negative by construction of f, and also a measure. It remains to show .
that p.j
=
IV
-
,
.1.
v.
Let (A+n,A-a)be a Hahn decomposition for the measure 1,2,3,.... Then for E e T, g,(Fr7A+) n
=
p.1(Ff-74)
+
-1v(l
>n
ra x+n) +
g.l v/n, for n -
=
g2(F raA+sl
j
sck.:
y:v
=
j
srn,j
-)
.f
+
w,
(4.56)
Integration
73
and hence p.(A')
p,(Ff'3A1) +p.(FrnAa-) 2 p.(Ff7A1)
=
k
J
El'x<
f
1 tv
+
n
Note from this inequality that
(x k
Jf
+
v
+
fdv J Fzn-a
J + n-11Aj -s
1v(A+n) =
=
+
p.2(Fr-7Aa-)
fdv + JE
-)v(F
r-'hA).
(4.57)
e 6 so that ,
-1v(4+n),
a
+
(4.58)
u
implying vAhjlln 0. This holds for each n G N, so if A U';=1A, v(A) 0. Note g.1(AC) v(AC)/l An-for every n, and so for every n. Hence ; that Ac 0* N= 1Aa-i g,1(Af) 0, and-so jt1 v. w =
=
=
=
.1.
=
It remains
to extend this result to the c-finite case.
such Proof of 4.23 By c-tiniteness there exists a countable partition f l of collection with finite for finite each j. If (Ay) is any that vtf#) and p,tf#l are union and whose is letting f11 Lh -Ay-1 for A1 1 > Aj defines j measures such a partition. If different collections with tinite measures are known for v and p,, say lAjzjl and fAvyl the collection containing all the Ax rnAvk for j,k e N, is countable and of finite measure with respect to both v and g,, and after reindexing this collection can generate (f). Consider the restrictions of g, and v to the measurable spaces (Lh,%4, for j e IN,where % (F t''h Lh, E e 5 J By countable additivity, g,(F) XjttA' rn L%lwith similar equalities for g,l, g,z, and v; by implication, the two sides are in each case either finite and equal, or both +x. If v(F rn Lhl 0 implies g,2(F chLhl 0 for each j, then v(F) 0 implies g2(F) 0 for E e 9, and g,2< v. Similarly, let Aj, AJ) define partitions of the Lh such that g,1(Aj) v(A,) 0', then A U#. j, v(AC) = Qv(A,) = Ac unions, 0. p,1(A) Qg4(AJ 0, and and Ugdlare disjoint Hence g,1 v. K ,
=
,
=
,
=
=
.
=
=
=
=
=
=
=
=
=
=
.
The proof of the Radon-Nikodym theorem is now achieved by extending the other conclusion of 4.27 to the c-finite case. Proof of 4.24 In the countable partition of the last proof, 4.27 implies the such that non-negative X/f-measurable
existence of
g,(F rn Lh)
=
p,ltf
f''h
L1jl + g,2(F r-hLhl
where P,2(A't''h f#) Define
J:
F-
Jsyoo *,
=
R+ by .fttl)l
=
al1 E e
X lf/ttl'tt9l.
5.
(4.59) (4.60) -1
This is a function since the Lh are disjoint, and is g/s-measurable since f B) y-1(#) LtEj e 5 where Ej e 5j, for each # e S. Apply 4.14 to give = Uy/' =
Mathematics
74 :2(A3
=
Xp,ztFr7
Lhl
=
j
Js
1tzjfldv. - X
Xj
Js
Jsljdv
fdv
-
.
(4.6 1)
*
Consider the case where g, is absolutely continuous with respect to another gq, v(A) measure v. If the Lebesgue decomposition with respect to v is g, g,l + g,1(AC) since 0 implies 0 which in implies g,l(A) 0. But g,(A) 0 turn g,l v, too, Thus, g,1(f2) 0 and g, ga. The absolute continuity of a measure implies the existence of a Radon-Nikodym derivative f as an equivalent representation of the fOr measure, given v, in the sense that g,(F) Js/'tfv any E e @. An important application of these results is to measures on the line. =
=
=
=
=
.
=
=
=
4.28 Example Let v in the last result be Lebesgue measure, m, and 1et g, be any other measure on the line. Clearly, g,1 m requires that g,1(A'') 0 except when E is of Lebesgue measure 0. On the other hand, absolute continuity of g,2 with length', any countable collection of respect to m implies that any set of isolated points for example, must have zero measure under g2. If g, is absolutely continuous with respect to m, we may write the integral of a measurable function g as =
.
Ezero
j
+x
'(AVP'@)
=
-X
p+x
1 -
gxtfxtdx'
-X
(4.62)
so that all integrals reduce to Lebesgue integrals. Here, f is known as the density functionof the measure g, and is an equivalent representation of g,, with the relation g,(f)
=
jefxtdx
(the Lebesgue integral of f over E4 holding for each E e f. n
(4.63)
5 Metric Spaces
5. 1 Distances and Metrics Central to the properties of R studied in Chapter 2 was the concept of distance. For any real numbers x and y, the Euclidean distance between them is the number Jstx,y) Ix y I e R+. Generalizing this idea, a set (otherwisearbitrary) having a distance measure, or metric, defined for each pair of elements is called a metric space. Let S denote such a set. =
-
A metric is a mapping J: S xs F-> R+ having the properties dx,y), 0 iff if y, 2 dx, dy, (triangleinequality). metl'ic (S,#) is S A space a set paired with metric #, such that conditions (a)-(c) hold for each pair of elements of S. u
5.1 Dqfinition (a) dy.x) (b) dx,yt (c) dx,yt +
=
.'t7
=
=
0, but dx,y) 0 is possible when x # y, we If 5.1(a) and (c) hold, and dxtx) would call d a pseudo-metric. A fundamental fact is that if (A,#) is a metric space and # c A, (#,#) is also a metric space. If (E1is the set of rational numbers, Q c R and (Q,#s) is a metric space', another example is (g0,11,#s). While the Euclidean metric on R is the familiar case, and the proof that dz satisfies 5.1(a)-(c) is elementary, dz is not the only possible metric on R =
=
.
5.2 Example For x,y
E
R 1et
4@,y)
=
!
Ix +
yI
-
jx
..y
j
(5.1)
.
It is immediate that 5.1(a) :nd (b) hold. To show (c), note that Ix l doxtyjlk dx,yj). The inequality tz/(1 a) + /7/(1 b4 2 c/(1 c) simplifies c). We obtain 5.1(c) on putting a doxty), b doy,z), to a + b 2 c + abl c 4)@,z),and using the fact that 0 f do S 1. Unlike the Euclidean metric, Jo is defined for or y = ix. (V,4) is a metric spae on the definition, while Rwith the Euclidean metric is not. n -y
-
-
-
=
-
=
-
=
=
.x
lh the space R2 a 1arger va riety of metrics is found. 5.3 Example The Euclidean distance on R2 is -y1)2
defx-ytIIx-yII E(m =
d (q2r 4) p,lp ,
.
''
,
.
=
-y,)211'2
+(.n
(5.2)
'
.
is the Eglidean '
'
'
plane. An alternative is the
Etaxicab'
metric,
76
Mathematics
dvx,y)
=
Ixl
-
yl I +
1A'2
-
y2 1
(5.3)
.
dz is the shortest distance between two addresses in Manhattan as the crow flies, but dv is the shortest distance by taxi (see Fig. 5. 1). The reader will note that dv and Js are actually the cases for p 1 and p 2 of a sequence of metrics on R 2 He/she is invited to supply the definition for the case p = 3, and so for any cxo is the maximum metric, p. The limiting case as p =
=
.
-..-h
JA,g,y)
=
maxl
-y2l
1.x1yl I 1x2 -
,
1.
(5.4) R'T
Al1 these distance measures can be shown to satisfy 5.1(a)-(c). Letting R x ERx x R for any finite n, they can be generalized in the obvious fasltions (RR,#r), (R'l,#x) and so forth. n to define metric spaces (R&,Js), =
...
t/yg (-,:,)2 )
------------
t2-(-);,)7
)
Fig. 5.1 Metrics dk and dz on a space S are said to be equivalent > 0, there is a > 0 such that
dblx,yq < 8
=
dlx,yq < E
dzlxty) < 8
=
[email protected]') < E
if, for each
-v
e S and :
for each y e S. The idea here is that the two metrics confer essentially tlte same properties on the space, apart from a possible relabelling of points and axes. A metric that is a continuous, increasing function of another metric is equivalent to it; thus, if d is any metric on S, it is equivalent to the bounded metric dll + d4. Js and do of 5.2 are equivalent in R, as are are Js and du in R2 o n the other hand, consider for any S the discrete metric Jo, where for x,y e S, dE dnxby) 0 if are not y, and 1 otherwise. do is a metric, but Jo and equivalent in R ln metric space theory, the properties of R outlined in j2.1 are revealed as a special case. Many definitions are the same, word for word, although other concepts are novel. In a metric space (S,#) the concept of an open neighbourhood in R generalizes to the sphere or ball, a set Sdxn (y: y e S, J(x,y) < :) where x e S and : > 0. We write simply Sx,z) when the context makes clear which .
=
.x
=
.
=
,
Metric Spaces
77
metric is being adopted. ln (R2 d E ) S@ :) is a circle with centre at x and it is a radius E. In (R2 (rotatedsquare) centred on x with e the distance from x to the vertices. In (R2 du ) it is a regular square centred on x, with sides of 2e. For (R3 d s) we 11 think about it! An open set of (S,#) is a set A S such that, for each x e A, (; > 0 such that 5'@,) is a subset of 4. If metrics dj and dz are equivalent, a set is open in (S,#1) iff it is open in (S,J2). The theory of open sets of R generalizes straightforwardly. For example, the Borel field of S is a well-defined notion, the smallest c-field containing the open sets of (S,J). Here is the general version of 2.4. ,
,
,
tdiamond'
,#w)
,
,
,
...
5.4 Theorem (i) If C is any collection of open sets of (S,#), then
c
UA
=
Ae N
is open.
(ii) lf A and B are open in (S,#), then A ch# is open. Sx, c A and A e C, then S.z,z) c C. Since such a ball exists by a11 definition for x E A, a11 A e C, it follows that one exists for all x e C. (ii) If 5'(x,/) and S(x,k) are two spheres centred on x, then
Proof
(i) If
5'@,6k)f''h 5'@,Es)
(5.8)
5'@,E),
=
If x e A, H 5'(x,/) c A with Ex > 0, and if e #, (5S(x,ze4 E > 0. If x e A chB, 5'@,E) I A rn B, with 6: > 0. x
where e minlerx,Es) c B similarly, with
.x
=
.
The important thing to bear in mind is that openness is not preserved under
arbitrary intersections. A closure point of a set A is a point x e S (not necessarily belonging to A) The set of closure points of A, such that for all 6 > 0 ('!y e A with #(x,y) < . of Closure called the closure A. points is denoted A, are also called adherent points, sticking to' a set though nt necessarily belonging to it. If for some 8 x, so that > 0 the definition of a closure point is satisfied only for y &x,) f'-AA (xJ, x is said to be an isolated point of A. A boundary point of A is a point x e X, such that for all > 0 H z e Ac with The set of boundary points of A is denoted :A, and W A Q.J :4. The J(x,z) < interior of A is A0 A PA. A losed set is one containing al1 its closure Points, such that X A. An Qpen st dyjes not contain a1l of its closure points, since the boundary points do not belng to the set. The empty set 3 and the space S are both open and closed. A Subset pf A is said to be dense in A if B c A c X. A collection of sets T' is calld a. bvering for A if 4 Usssf. If each B is =
=
=
.
=
-
=
.
.
...:.;
.
''
.
'
.
.
;
'
.,>
covtink. A jt A is called compact if every open open, it is called an open kqtog. x is said to be relatively compact if covering of A contains a tinite S itsel? 'A,t,tk'.$,j,'..,k4) is said to be a compact space. The cott, is compact. If s is rk.,i tl compaltlktlyj, ,. ttk y))(y,y;(jty)):Vyty,yjkj4g eqqally relevant to the general case. remarks in j2. 1 about jkyyjytyyyjjy yjj yyy, jj 4::11. d!;p dr:;li 4k220 154!r 11::). tjyod jyyyyyktty.jtyz,y,g,jyjyyyyyjxyyyjyjyy,,yyyyyyytyy ii.1b;1b;Citii. 1l:4(E:1. rr.t).:y...;....,. ..,.4!k. ;yjjy .yj < x, sucj.j tjaatx s sxtrj; and also .
x
. .
.p,
,1.)
.))
'
t
.
rjg)
....r
.
.F ..--
.
Mathematics
78
totally bounded (or precompact) if for every e: > O there exists a finite collection of points x1,...,xm (calledan E-net) such that the spheres Sxz), i z4. The Sxi, can be replaced in this definition by = 1,...,- form a covering for noting that Sxi,z) is contained in Sxi, : + ) for all 4'5 their closures Sxinz), z4. > 0. The points of the E-net need not be elements of An attractive mental image is a region of R 2 covered w ith little cocktail umbrellas of radius : (Fig. 5.2). Any set that is totally bounded is also bounded. In certain cases such as ?n,dE) the converse is also true, but this is not true in general. . .k . .
.
.
... . . .
.
.
.
.
4
.
.
.
.
,
.
.
.
k...
.
.
.
.
.
.
.
.k
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
y .
.
.
.
.
'
j
.
.
.
.
.
.
.
.
.
.
.
' .
.'.
'
.. .
.
.
.
. .. '
.
.
.
.
. .
.
.
. '.
...
.
.
...
.
i
.
.
.
.
.
....
.
.
'.'k. .
.
. .
.
'.
.
.
.i; ..
.
...'
. .
.
...
.
.
4 .
.
...
. .
...'.
kj'..'
Fig. 5.2
5.5 Theorem lf a set is relatively
compact, it is totally bounded.
Proof Let A be relatively compact, and eonsider the covering of X'consisting of the e-balls Sxnz) for a1l x e X'.By the detinition this contains a finite subis an e-net for cover Sxitz), i = 1,...,-, which also covers A. Then (x1,...,au) d, and the theorem follows since e: is arbitrary. K The converse is true only when the space is complete', see 5.13.
5.2 Separability and Completeness In thinking about metric spaces, it is sometimes helpful to visualize the analogue R'J with n f 3, and use one's intuitive knowledge of problem for R, or at most for those cases. But this trick can be misleading if the space in question is too alien to geometrical intuition. A metric space is said to be separable if it contains a countable, dense subset. Separability is one of the properties that might be considered to characterize an 4R-like' space. The rational numbers (I) are countable and dense in R, so R is separable, as is R&. An aliernative definition of a separable metric space is a metric spaee for which the Lindelf property holds (see2-7). This result can be given in the following fonn.
5.6 Theorem In a metric space S the following three properties are equivalent: (a) S is separable. (b) Every open set A S has the representation
Metric Spaces
4
UBi, Bi e
=
79
'P,
izu1
where V is a countable collection of open spheres in %. Every (c) open cover of a set in S has a countable subcover.
I:a
A collection V with property (b) is called a base of S, so that separability is equated in this theorem with the existence of a countable base for the space. In topology this property is called second-countabilit.y (see j6.2). (c) is the Lindelf propely. Proof We first show that (a) implies (b). Let V be the countable collection of spheres (5'(1,r):a7e D, r e t1+), where D is a countable, dense subset of S, and Q+ is the set of positive rationals. If A is an open subset of S, then for each x e A, 3 > 0 such that 5'(x,) g A. For any such x, choose xi e D such that dxie < /2 (possiblesince D is dense) and then choose rational ri to satisfy dxip < ri < &2. Define Bi Sxitr e V, and observe that =
S(A7,8)
Bi I
(5.10)
i X.
Since V as a whole is countable, the subcollection fBiI of al1 the sets that satisfy this condition for at least one x e A is also countable, and clearly A c Uff I X, SO X = U#f. Next we show that (b) implies (c). Since V is countable we may index its elements as ( Jj, j E EN). If C is any collection of open sets covering A, cboose a subcollection ( Cp j e LNJ where Cj is a set from C which contains 5. if such by F-sets, as just exists, otherwise let Cj 0. There exists a covering of shown, and each 5. can itself be covered by other elements of V with smaller radii, so that by taking small enough spheres we may always Gnd an element of C to contain them. Thus 4 c UjG,and the Lindelf property holds. Finally, to show that (c) implies (a),consider the open cover of S by the sets (5'(x,1/n), x e S l If there exists for each n a countable subcover (Sxnktllnj, k e ENJ, for ach k there must be one br more indices k' such that dxnkpxnk')< lln. Since this must be true for every n, the countable set fak, k e (N n e EN ) must be . completes ptoof. the This in S. dense ,
:4
=
.
,
The theorem has a useful corollary, lc'tg C' k' ;''j' y' q.rljljjrjtrj'jk. k:5j';;'jll.' j(j('(jy';;)(;(jjjr-:';E' 'i'. 11(F*))7(j)1))*5i'-!tt)@'.tjjj: rjd t;(.',' y' (jd y.'r' )' :'''jt'ty'k). (y'r;. )' jy.yjy-'tyyd yj'(' (' )jjj'Lqjj.j;Lj))(L:'.. ;';)' j;'16'. illj.Fklfll'r!id j()$.tj!y' !l-i@;yy'. jj.'lq. j'td .'.'k(t(jjj.'. --' ,.' -' .'.,' ,'.r:,ggjj4;4* ..t-d '-7*
'''jd 7* ::.tky,(y)jjj4jj44.2* yjyyjj...d ..-tj).'.' ,.t,' jy'. E'. 5* k(j)jt)':. ;'-'kE''. ,':'.#y' ?'
)'jt. jyd (' jjryjjgj;rrjjj'. yy'' jjjjj..d jjj')j(jk)'. r)'k(!@'' )' h' tl'tjjg t'jkd
''. .' 57 @
Corollary A totally bounded space is separable. n .
Another important property is that gbspaces which we show as follows. y
sR
Tuoorem
If 4:.* is a sp' ..
.
.
..
of separable
spaces are separable.
ab ll,!.'pae and A c S, then A,d) is separable. l l '
.
.'
;
'.;
. .y
y.
:
5:.'
:
')'q q.ty ;..' j,.);.j.;Lj. jy yk y. ..) .. a,,#..z'inl.s.
....y.((
. ..
.'
)
;y..
:
.'
.
g
g
...
.
. d'tll(td. Constnlct the countable set E by ) qp;jt,. jytgr yy a, ( '' supposeD is countbl g.;..j y (. y .). . .... y .. ... .. . . yjtliltjt, . q F tnul iyl!q t ' 'y takingone point from each;.$.',tj,.A,, ' (qi r..i q. y ( .( .. jy- jy.j(. ..jjtk(,... rk ;,-j,bj,.,. ...;,;-j-tyjj-yyy.trr,pjrrj--t:. jq j rk.,yjj ' r...'. .(. .,.-bj',::,;;;LL?3b.,L, y ..g.. .. ;f;;,,,;j:;-f);;,-'LL?33'-.,-j;)--Ljj;::-),'b--jj.)L,,t,;b'f?;));j-.
Ivoof
.
y
E
.
:
(
E
(E $2 .'
EE .
.
E ' ly . .
jE. .
?.
.
: gy. q
.ptllko'tyq)
E. q: . .
:
.E
.
,
.
.
l'.r
-i
y..y)..
E' . .. .
.' ' ::EE. . . . . E .-' (Ej.jijjjjjyryjgggggg, .. . . g: E . y y . y, g . . yy . . . y .jjyy... ..j... . y.-.... . ,!j;pjj42)$g,-.,,y,..(y.kpjr)(.y(y(,yyjjyjjyj;jy.-..,...-y.tyjky,
E
. E .. yj' .E. !. - q. . (-. .(j. ; (E r
.
.
.
.
.
-
. -:;..,-:3626,3.-.; -...633333:,31,'.. . . . .
.
j .....-.-,...yjj..... . .-,,.... .....
#.,..- ytlyy.jitr.. )r,y,-.yyjyjjtjj)y.,,. .--..!1k.. ijijLjlijjjiiljl-. . ... .. ... ..
);.ryj-rkji-. 1,:-::,..-,k. .... lttjj?, . . .;(;;yyit..ykrj.-.yy-j-kjtjrqy-yj. .,jjt;-.. ...y'ptj,;,.-p!h!!!qqj--..,.
...-....
,
( !jr r... ..... . . .. :..
: : : . ..
,
-k,yt.yytyyy-y,......--...-
.
,....y
-.
.
;k.
.
..
.
,.
. ,.
.
),
.
..
.
,
..
,;.j,:-.,,j;;)j.yty,r.t y .(.. - .. .... -
.
.
..
.
4::). jjjjj.s ,-
tjggjjj
Mathematics
80
For any x e A and > 0, we may choose y E D such that J(x,y) < 6/2. For every such y, R z e E satisfying z e A fn S@, for r < 6/2, so that d@,z) < &2. Thus
dxa)
f dx,y)
+
J(.y,z)
<
8,
(5.12)
and since x and 8 are arbitrary it follows that E is dense in A. w This argument does not rule out the possibility that A and D are disjoint. The separability of the irrational numbers, R Q, is a case in point. On the other hand, certain conditions are incompatible with separability. A subset A of a metric space (S,J) is discrete if for each x e A, R 6 > 0 such that (5'(x,6) (xJ)DA is empty. In other words, each element is an isolated point. The integers ; are a discrete set of (R,#s), for example. If S is itself discrete, the discrete metric do is equivalent to J. -
-
5.9 Theorem If a metric space contains an uncountable discrete subset, it is not separable. Proof This is immediate from 5.6. Let A be discrete, and consider the open set where Ex is chosen small enough that the specified spheres folnn a Uxez5'@,exl, disjoint collection. This is an open cover of A, and if A is uncountable it has no countable subcover. . The separability question arises when we come to define measures on metric spaces (see Chapter 26). Unless a space is separable, we cannot be sure that a1l of its Borel sets are measurable. The space Dgtz,,jdiscussed below (5.27)is an important example of this difficulty. The concepts of sequence, limit, subsequence, and cluster point all extend from R to general metric spaces. A sequence (x,,)of points in (S,J) is said to converge to a limit x if for al1 e > 0 there exists Ne k 1 such that
dxng)
< :
for a1l n
>
(5.13)
Ne.
Theorems 2.12 and 2.13 extend in an obvious way, as follows.
5.10 Theorem Every sequence on a compact subset of S has one or more cluster points. n 5.11 Theorem If a sequence on a compact subset of S has a unique cluster point, then it converges. Ia The notion of a Cauchy sequence also remains fundamental. A sequence (xnlof points in a metric space (S,tf) is a Cauchy sequence if for a11 e, > 0, 3 Nz such that dxnbxmj < e,whenever n > Nz and m > Nv.The novelty is that Cauchy sequences in a metric space do not always possess limits. It is possible that the point on which the sequence is converging lies outside the space. Consider the space (Q,#s). The sequence fxn), where xn 1 + 1/2 + 1/6 + + 1/n! e Q, is a Cauchy 0', but of course, xn e (the base of sequence since Iak+1 l = 1/4n + 1)! the natural logarithms), an irrational number. A metric space (S,#) is said to be complete if it contains the limits of all Cauchy sequences defined on it. (?,dE) =
...
-->
-xn
--
Metric Spaces
is a complete space, while
81
Q,dE4
is not. Although compactness is aprimitive notion whichdoes notrequirethe concept of a Cauchy sequence, we can nevertheless detine it, following the idea in 2.12, in terms of the properties of sequences. This is often convenient from a practical point of view.
5.13 Theorem The following statements about a metric space (S,(f) are equivalent: (a) S is compact. (b) Every sequence in S has a cluster point in S. (c) S is totally bounded and complete. Ia Notice the distinction between completeness and compactness. In a complete space all Cauchy sequences converge, which says nothing about the behaviour of nonCauchy sequences. But in a compact space, which is also totally bounded, all sequences contain Cauchy subsequences which converge in the space. Proof We show in turn that (a) implies (b),(b) implies (c), and (c) implies (a). Suppose S is compact. Let fxn,n e IN) be a sequence in S, and detine a decreasing sequence of subsets of S by Bn fxk:k k n,J The sets Tn are closed, and the cluster points of the sequence, if any, compose the set C O';=1X,; IICIC. U';=I#,C,, If C 3, S (U= so that the open sets T'nc are a cover for S, and by assumption these contain a t'initesubcover. This means that, for some m < oo, S (O'2=1))C This leads to the contradiction Tm 0, so that C must LKLc n S U'''ol) be nonempty. Hence, (a) implies (b). Now suppose that every sequence has a cluster point in S. Considering the case of Cauchy sequences, it is clear that the space is complete; it remains to show that it is totally bounded. Suppose not: then there must exist an : > 0 for which such that no e-net exists', in other words, no tinite n and points (x1,...,AkJ x k. But letting n in this case, we have found a dxjtx S : for all j sequence with no cluster point, which is again a contradiction. Hence, (b)implies =
.
=
=
=
n=
=
=
=
=
.
:g
-->
(C).
Finally, 1et C be an arbitrary open covr of S. We assume that C contains no finite subcover of S, and obtain a contradiction. Since S is totally bounded it must possss for each n k 1 a tinite cover of the fonu B ni
sxni, 1/2'')i
=
y'
jyjjjjjyyjjjy'
1
=
...,kn.
(5.14)
,
Fixing ?z, choose an 1 for whlch Bni hs' or finite cover by C-sets (at least one such exists by hypothesis) and all thls st Dn. For n > 1, jlBni))=n 1 is also a jl tjjt X h & X n-1 jja.s Sujpcgyeyy COVCf1ng jw p n-1 ap wg ggyj gjyjpjjj Xyr,(,.)!y;yy;jy fojyjj* njj tg F yyjj . kyry,j;. :,, jyzjojjsea sequence of points (xn e D IS nonemp tyi yj.g.(..)t;(( C-sets, and accordlngly n, y .(-(j,jj(. . )y.( . y.kj. s l tontalns ak, and Ds+1 1s of radi us ) since Dn 1s a ball Of radm# .,.. yy,,.tyjjj;,,,,gy; .yr;yy, jj-j jygyrgjs jkgyy jjygj.. . jtyjjy yggjj jkgyy ;yy gjg gjygjs y....jjygjyyj jjyy;yy (j. yj.gj. ikyj. gj. ;jy yyyjy yjs gjygjyyyg jjygjs $j44,, gjs lln- gj and contalns Ak+1 dxngxq-j) .iJ (j. gyjj,, gjjgjj y 0'1: '''''j . !.E ',..j';;jjc''''j;.)',.*.,.,,;.;,)k,q..qL,.j'q,)j-,Ls))jj)q,;t)).L'j,.j !1..2,'... ......1812.....,.... 1t!12$!E;k..
w
,E
,
.
y
y
,..
..f.kj)
L;..
.
E
.
y y t .y..;.. .. y.... zt'i')>t7)Eitu . . .,2:jj,g.
,1
.
-
-
..
,-
...,,q..yyy-
..,,.j.)
.
...
.
y
..
,
....
..ksysj-sssssyyyssy,j.yr
,..syyyy
.,
.
',','t.)y;;r,))',tyyyiy),.,),y!(jjj,y)g.,,yiityy;j,t,y.,,yy,,,jyjj,
y
,
''''
..j.,
,,
..)
=
j,;.jr
..
d,
jj:rr;jj
''
-'::::g
.iltrr;,:r
..
',-
q.g...j-: .E , q..
.(...j!;r';-r:).,..-jj)...
jjggygi
.....
d.
...t.
,
:
ktrriiir'dt:l'.k;,d::r '-,.,,
'-::::::'
...,
g
..,.fj),)),)jj)j)),,
..
,
...
.
..
,
.rqrrty.r. .
.
.g .jjjj. j,.
y..ygyjjjy2jjj;;;.yyjjyygjjg ;,,.q.;u,..,..,,,,r,,..
jjjjjjya.y.
.jjjgjrj...
,4..
..,..
.,r,...r,r,,,.,....)))))jj,jj.j:)jj(j,;..(qjj.
yjyyyjyy.yyyyj;yyj.. ..,...yyyyyj;..;.
..g yt -t-;t4i!)7-.ygjjtt' -
.
yjjjgj.y,-pk@$1r....-,... . yjya
--
.'.
'' ' .' t.ylii. Ji. .;..
:..... ,-..
..
yjjjjjjjjyyjj.g. yy,,yjjjyyyj,... g.yy.. yyy ...yy.,yyyy;
....yyjgyyyyy..
..
.
--$t'-.
.
.yyjjyyjy.
jsggggggggy
,,..).:,.j
...,..y .
,y
..
yyyjy.yygyyjy,
..yjyyyy. a.aayyayay..
y.
..
y,
-;jj-t.yy.
.
-...
--jjy;;-
..
.
-zi
.
....
..
.. ....
.. yjyjy;.;yjjry,;;jy..j(.. .g...j.jgy,...;.g..g........y.y. ..... . rysg..;
jjryrrjyjtr
.
.')' yj..-T!rt)j-ji.. .. . '-ty,--i)... ,,- EEE; ;.,-. ,qjy)j)y(. . E ,;,'.-);,.. ;' E..:-...'.-..?.-.., ..k:, ;.hF(.f.
...
-..
,..y......
,
.g.yyy..
ysy. (gylrqE..jj
.
).
.
jjjjjj,jj),,,j r.:g; .,yjy,4E ..ypj.'q).k(jyj.,j:;II,..: ),j)jj,..t jpkzjlk.g))jjjjjj):...k -(...
.:;,,::7(22,-
bif'
-.;(
)F?jjjgjjj:... r.()
j .. y,y.q.yt. .E ;..... )i):y:.
j!jjjjj).
jtkkgry
jtg
,,'..
.-;p7.
...
.jrr,
jjyyg;jj jjygg;jjjggygi
.
.'r'y)'-'.
,444,.4
.j(r,
!j,
,jjy,
''
.....
.j;,k:rr
.!jjj;,.
.
''.'''''12.12
.
jl,.
-.
.E.'.:.'.y..E.E......:... ),1lpl-t''t,-;-)@--
.,
.
...
.
.tlih'tjjrt.;2).-:.lqt.1r):.
!,-
77?:?,-' .... ;,..-.. b3-,Lt;j,,1'3$;t311L;;,;(;L)3$L;-t..i;'.).j)). .'.ti;;jit)()(.. . ... ;-
1t;;$(yr';)))(;...---.j$)yyk?Ey-..,it). -. . '-'. -(jj(-,-.'. '.
.-..
yjgy;yyjygyyyyy jygyyyyy
.
.
.
..
,. -
.
.
.
. .
.
y;jjjjyy
82
Mathematics
< 9Iln ensures that Dn c Sxn. But this means Dn c A, which is a contradiction since Dn has no finite cover by C-sets, Hence C contains a finite subcover, and (c) implies (a). K In complete spaces, the set properties of relative compactness and precompactness are identical. The following is the converse of 5.5.
5.13 Corollary In a complete metric space, a totally bounded set A is relatively compact. Proof lf S is complete, every Cauchy sequence in A has a limit in S, and a1l such points are closure points of A. The subspace (A,#)is therefore a complete space. It follows from 5.12 that if A is totally bounded, X is compact. .
5.3 Examples The following cases are somewhat more remote from ordinary geometric intuition than the ones we looked at above.
5.14 Example In j12.3 and subsequently we shall encounter R=, that is, inhnitedipensional Euclidean space. lf x (x1,.n,...)e R=, and y @1,y2,...)e R= similarly, a metric for R* is given by =
=
X
d-x-y) >72-Q()(x,,y,), al
(5. 15)
=
where do is detined in (5.1).Like do, t/x is a bounded metric with #xtx,yl f 1 for all x and y. n (RX,Jx) is separable and complete.
5.15 Theorem
Proof To show separability, consider the collection
4
=
xk rational (x (x1,x2,...):
if k S m, xk
=
R=
=
0 otherwise
)
(5.16)
.
(A,u, m Am is countable, and by 1.5 the collection A and e: > 0, H x e Am such that able. For any y e =
R''O
m
=
1,2,... J is also count-
co
d-x,y)S y2-$ -al
+
-t=
5-' 2-4(0,yk)
S
m+ 1
e+ 2--.
(5.17)
Since the right-hand side can be made as small as desired by choice of e:and m, y is a closure point of z4. Hence, A is dense in R=. To show completeness, suppose (xn @1n,x2n,...), n e N ) is a Cauchy sequence in R ' Since dtxknvxkm)S 2J x @n, xm4 for any k, fxa, n e EN ) must be a Cauchy sequence in R. Since =
.
d-xx)
/
57l-kdxkakn) 1 -t=
+
2-'n
(5.18)
Metric Spaces
83
for all m, we can say that xn x @1,x2,...) xk for each k e R= iff xkn completeness of R implies that lxn) has a limit in R=. w 1,2,...., the -->
--)
=
=
5.16 Example Consider the infinite-dimensional cube', g0,1J=;the Cartesians product of an infinite collection of unit intervals. The space (g0,11=,#x) is separable by 5.8. We can also endow (0,11= with the equivalent and in this case bounded metric, X
772-11x: -ytl
Px(x,y) =
(5.19)
u
.
a1
In a metric space (S,*, where Jcan be assumed bounded without loss of generality, define the distance between a point x e S and a subset A ? S as dxl) infysxJtx,y). Then for a pair of subsets h,# of (S,#) define the function du 2 s x zs j..o R+ =
,
W here
2S
is the power set of S, by du(A,B4
max sup d@A), sup d(y,B)
=
xB
(5.20)
.
yGA
du(A,B) is called the Hausdorff distance between sets 4 and #.
5.17 Theorem Letting J'fs denote the compact, nonempty subsets of S, ls,du) is a metric space. satisfies 5.1(b) since the sets of Rs are 0, so that du is only a pseudo-metric for for any x E 4 and any z E C we have, by d is a metric,
Proof Clearly du satisfies 5.1(a). It closed, although note that duA, W) general subsets of S. To show 5.1(c), definition of dx.B) and the fact that =
sup dx,B) K supttftmz)
)
dz,B)
+
Ac#
xcA
(5.21)
.
Since C is compact, the infimum oyer C of the expression in braces on the righthand side above is attained at a point z e C. We can therefore write + dz,B)) < snp tnf(4(x,z) supltms) A ze xed
.xG
supltz,l). sup (;:,6'D z
(5.22)
.#
y', :'y' rtjy'E (;,'. :'. E' j::yt',.'Lli'y.jjy:yjj'lt-.kt4j?iyrj)j('.' y' j'yyjyy'. k'. ,', jjt!rrg;ji!r'd 111:::2;11!!'* :'i'.j,,g.' sgEy-yzgyj',d ;',rjjy't. (' !yj'. !,,' r' ;'y'g y' .L'. ;'. :@),'.,.,,;y.jjjj;g'j;g'kjrt.ilyjrg-7ldrjjklypysjlljpt'jjq'lrsr'd k' j't'yyjy,,, y'y .r' 44::7* jtd iE-':jd )' j'y;d qtd .y' jgy'1!. jydt.yy.rd j'gi i-(' litjjjltsiytj'y'. j);';' 4* r' yyd t':g'33t'. yjjy'. y;il'. y'jy,ytd yrd ytyy'.' jyy'. yrgj'.;yjyjd .'qL);)'. (''.. y' y-'( jy-'). (i'yd ygy';6!r;.'.jygjyj.yry!;(r!. jj',;L?'. .',jy',rg..jjjj. -'y -'
)k',.;jr!j'(;;'. .-' yy.j.-d .;j,' 33666666636663*, r)'... (tg,' ,,,,,,,,,,,,,'
''. .': .'.'.
sl'milarly,supyssltxz)
supzs
..??
...
-<
,-
.
.
:l::)Er!t. ax
s
(x
dr:s
ll.,l. (14:,.,, ..441k.
dEE.
.;)t:r
...1..-.
.
'..
.
..
.
'
:
.
,
.
'
,
and hence,
,
.
E ..',' .j .k.. 2..' . . . . . '...,-...,.'...g...,.....-,. ..-..:!:.......... ..,..k.., -.s. '..';
(2), drr;ir.n
---.11..
(z ) + supyss:
-<
'
.i(:).-, 'i.ii!Eil'.....,...... ..,1:11.. s..,...12.. tii''fL;.:, (;,..qI...sll.,l.jll:), .!147,.. ..(14;,.. :;.
......(2,...... . .
....
..
.
..
i
. . . ' ..
:
:
c: )y)j
'
.
'.'
. .
'
.
'
,,.-,.11..
.
.
..,..
..
:. .i. E
:
C
: E: E j) E ) ,)L.q/;j-i...L,j;...3L;.....L.. :.E.; )...E... ..'q''. . . tlLi: ..;..-..1qs? .... E..E:(:; . ifr: .q. LE . y(..!. . .: q ( q..;..sy . ;LL;.1Lj. y:.Ey.tjyy.tpyit;kr . . . .. .... ' . . E. .. . . LL..L......L. . :Lj.LL..:....j.L.. Ljf,. . .: . . E. . .. .g k!:..(#k:gk....jtt .... L).;):,.;.)..).;.:L.....j;. ....4:j.. :.. j . .: . :....:(. i. .E. .jg (.... .. ..,. .... .. j.; , :. g. . j y: .;....L.. ; . .kj. jr.). yr . j y. .. ;'. ' (yj ( jjyE. k:, y . ..j E :.' E .y . y! :. . .. .y-.L- . :. g.( q j yy.. j.:; . ..r . . r yg . .y . . . .. .. '. yy. ' y;. . . y y yy . : .,),.;.tj..$ : E i: . . . .. . . :yjytrr...,:jygj ;.,y..... y.y... ! gs.tjjtyyyjygjgytjrjry 4)f,Ljj;.L;;;j).);).f..j;q))..)):k.. L...j),)j)):)..,..,.))q,..L4)q);;q;. .. .L.(. j..-.. y y. yy . .yj r j:.j).,.;t,), yyrjyjjjjjj., y.y. y .j.. .g yy. y...
E
.1:7
-
j
.1,.!;
.
..:..jj..y!.)y.g;...::.).j
....(
..
....;)..
dr:;ir-
.
. .
t)t,.g..y t:
..
.. .
.
.
.
)5.jk; qL., ... .j.y.. (. ..tjtr .. ( y.)(.) .. j.jjyt.yjt... (j) )y(jjy,..t;.yy: y.. . . . . jj;, (yg j . .yt;jg yq . (..;.j j,(jj . t' ):b,. qi(gy-. q.. ...:::::::::( ..... yyj . j .: . . . .. ry .yt,. . . . .j.. y. . jy...yry.jj,s .. t)$:.,;jjg:. . i.. yy,y . ItIL. t.y;y. ... y kk))r.' kjl.yy ; (-. .'. jj;.,. ' ...g. Lqjjjf. rjy . yy4tjj,r.. fjjjj.. .. . ; 11k2. dtr:;.!r-'l)p--.--?-j:,,-(-,-----. ----.-b--b-,i-------;-jbv.... di:22:). ...t .-... dk:r:..'...i...---..-).)--..-.---...-..-........---...;...i..f..t--...-;:., dl::::: 4.:2::,. dliz!,.-.. 1lE5'..lI::!'... :llr7.:':1L. 4:7-.. . .-tk-. ,,- :.1. ,--L-.-----?,,--. II. :!i4;. .. t,!!i9;...qlr.... ....-.... .-........-k;!!iiIr... a.)-....-.--...-.,y.,-.;,:.-... 11:::,,. ..
ti!iiE;k ,(:!:; -,-
..
:
..
...)
. ...
.,.,..;yyj:
:(.ygjq
..
.
.
.
.-jji!!!ijiij''
-.-..11......
,j.!!.!!!!Ii;... ' .
....
.
,
.
.
.
. ..
,
. .
.
.
.
.
.
:
. .
.
....
-
.tg,
,
..
.
.
.
.;
,
.,y...
.
-
.
...
j)
..((y.- -.
- '
- . .
tjkj;j ..(.
..
.
,
.
,),))))j.
.
.
. . .
.
.
,,
(:3?'.i!EiI? dE!!!E
. -
.
(5.23)
.
. .
...
.,,,:
,
.
.r
ll.
'1r:.
.
- .
.
,,.
. .
..
.
.
.
...
.
.
.
.yjj.,.
.
-.
Mathematics
84
5.18 Example Let S = R. The compact intervals with the Hausdorff metric define a complete metric space. Thus, ((0, 1 1/n!, n l EN) is a Cauchy sequence which converges in the Hausdolf metric to (0,11. This is the closure of the set g0,1) which we usually regard as the limit of this sequence (compare2.16), but although (0.1) Cts, JVE0,1),(0,1q) 0. u -
=
Another case is where S @2 J s ) and J'fs contains the closed and bounded subsets of the Euclidean plane. To cultivate intuition about metric spaces, a useful exercise is to draw some figures on a sheet of paper and measure the Hausdorff distances between them, as in Fig. 5.3. For compact yt and B, duA,B) = 0 if and B; compare tlzis with another intuitive concept of the only if A between two sets', infxsx,yss#stmy), which is zero if the sets touch or =
,
tdistance
=
intersect.
'' . .. .
. .
. .
'
.
. .
*1'1 .
.
.
.. .
.
.. .
.. .
. .
'
'
.
.
'
' .
.
' .
. '
.
.
.
.. .
'
.:::
.
. .
.
' .
.
D
: : 727::!:::::12::!; :;:: :::7::t:::!:!J1:::: ::'' :: : : : 7t : : : : : : :: :: :: )(72:2 : 7 7 ! : : 2. :: : l : :: L : l : L: : : ' 2: : : 1 ) . t :2:::::::::r2:2E:1:::2r:2:J:;::;::::t:
. .
' .
'
.. : . . . ..
'
.
..
.'
' .
' .
'
. .
'
'
' .
. '
.
'
. '
. '
.
' .
' .
. '
.
' .
' .
.::::.:.:.:.:.:.:.:.::.:::.::l.:::.:::;;:::i::.:.:.::::.:,.. . : .
.
dr;lr
igrjr
(:) a//qik
aj!jjl ,
()h
Fig. 5.3
5.4 Mappings on metric spaces We have defined a function as a mapping which takes set elements to unique points of R, but the tenn is also used where the codomain is a general metric space. Where the domain is another metric space, the results of j2.3 arise as special cases of the theory. Sme of the following properties are generalizations of those given previously, while others are new. The terms mapping, transformation, etc., are again synonyms for function, but an extra usage ksfunctional, which refers to the case where the domain is a space whose elements are themselves functions, with (usually) R as co-domain. An example is the integral defined in j4.1. The function J: (S,#) F-> (T,p) is said to be continuous at x if for all e > 0 H > 0 such that ys
p(.f@),f(x)) < z. sup sdx,n)
(5.24)
Here, may depend on x. Another way to state the condition is that for e > 0 3 > 0 such that
f(&(x,))i
5'p(.f@),e).
(5.25)
Melrfc Spaces
85
where Sd and s'pare respectively balls in (S,J) and (7,p).Similarly, f is said to be unbrmly continuous on a set A i S if for al1 : > 0, 3 8 > 0 such that < :. sup p(.f(.y),J(x)) supy e 5'J(x,)t''4x
(5.26)
xGz4
Theorem 2.17 was a special case of the following important result. 5.19 Theorem ForA I 1-, f -1 (A) is open (closed)in S wheneverA is open in 1-, iff f is continuous at al1 points of S.
(closed)
.f-144).
Proof Assume A is open, and 1et fx) e A for x q for some E > 0. By 1.2(iv) and continuity at x,
f-'fsdx,nbLtj c
Sdxnn)
Wz have Spflxln
? A
J-1(Sp(f@),e)) i f-'(A).
(5.27)
A is closed and f -1(V A) s If A is open then by 1.2(iii), which -1 is closed if f (A) is open. This proves sufficiency. To prove necessity, suppose J -1(A) is open in S whenever A is open in 1-, and in for E > 0 is open in S. Since particu lar J-1(' p (/x),E)) e f-(S p (.j(x),:)), there is a 8 > 0 such that (5.25)holds. Use complements again for the case of closed sets. . -y-1(4)
''
-
=
-
.x
,
This property of inverse images under J provides an alternative characterization of continuity, and in topological spaces provides the primary definition of continuity. The notion of Borel measurability discussed in j3.6 extends naturally to mappings between pairs of metric spaces, and the theorem establishes that continuous transformations are Borel-measurable. The properties of functions on compact sets are of interest in a number of contexts. The essential results are as follows.
5.20 Theorem
The continuous image of a compact set is compact.
Proof We show that, if A c S is compact and Jis continuous, then /'tAlis compact. -1 Let C be an open covering of fA). Continuity of f means that the sets f @),B s C are open by 5.19, and their union covers A by 1.2(ii). Since A is compact, these sets contain a finite subcover, say, J (#1),...,J Bm4.lt follows that .A
.j
y(A)
c /'
.j
c UBj, U.f-'(#y)U.f(.f-'(>) y=1
(5.28)
-
jv1
y=1
is by 1.2(i) and ihe seond inclusion by 1.2(v). Hence, Bj,...,Bm is a finite subcover of fA) by Cij4ts. sinceT' is arbitrary, it follows .jyjyd -),'jLiii'''L. k' 11:222:::;* y'. ;'y'. jjj;jy!r;'. y' yyd !)' -i-E.(E' y'y y'ty' ry'j y;t'11I::2t!):.!q'-?q#').i. q' j-q,,,4fb.'. ..' jy'. g'g'. j';;;r2qpjjy.'. (' jj122222,,r*. y'(!jjt:r!''-, ;'(j1Ii(r:j11:qr:1I,;.. diIt!j:rjt)'. (jii).d (r:ljill;k.d .y' y' rlll:rjjd ..-,;;p9i;j:r'.-' .tjjjrrjjq:!)gry:yjj(,.'. gjjjjjjjjj'.' j';' jyj''' (.(;-'j4' :!jEi;,-' 1;:Ej5)11,,*. j'y .r.j;4'?i (' LjjkLbftjt' 11*. y-' L-t--..,,blfbqk:t''L.' rjjlg)d rljjj;;-d ..' y' yy'.'y yyy'...-y'yjj'y,(;jjry.qjjjyy.rjj(.. (yjjj.'. ty'-tj.ytjjj't. -.-,'-qi;'fqbI;;'.' (111:r::114.*. (jjlyrrr!ll.d .,' .g' ty.-y.jjy'..' t)j)-',. that fA) is compact. w )')66fLLLj. .jjqn,qjv...'k::kf'qktLLLl''
where the equality
''.'.' .'. .'rs .'.'jy,.
-(j(y'-y' -'. --((2:jp,'.' ,:pj(,jg.' -j;yjjjjj,jyjjj,,'. '-jjj::jjrjj;'. ;'F:' ..:,' ';t';--,::;;;jg!!r;y.-j)-' ---')::' -.' ,'ygjjjj)jjjj(yyrjj(y,'yyy -'.' -' ,':' :-'jj(jgj.y.' -'jyj' ;',',' E'jyyyy j'.-kk;'. )jyyjy-.'' t-,'. y'yd jjr:;rr:r;d jy'y'((r)))I''C' .i-'',,j(;)' y' t'.-jjjyjy:jjj,.' yrd .t't(;yy!ryryjjjgr;rryy.k' ('. jr:'-'.:.,,-j,jj;jjjjjjk,:.,y)--.)t;'i.'. ;,,*j3.*313413;*. ---'ibbfLi',,j,'jg'y.yyj-,-l t'yjytt'js j'yd .y' g' (y;'. 41kk2:::;* jjkykzrr:;d 11;j2r!:;'. .it')E y' :',jjjkjg,r;j'. ;'yyllllirrrjlk.'gj .-;'.jjjjjjrjjjjj.'. t.' :''jjj;:rjjjjj...' )' .y' j'yj,d
-,' .--'.'-.;k1ijj(2!t$;'y ..' .11;:2)* '',*4111::)*. .111:2k::1,,*. -' .'. ','.'i;'i---'jkg;jjpjr.. -'j);,4,j)tf,'3t',' jjryjpjiij'j.. .y'yyjjprrtjjj-;' ,'y.jy'-yj -'. ,k-'-
5.21Theorem If the set
Jis
. .
continuous on
.
,
.
.5.(' .
.)
..'
.
. '.. . .
'.
; y.' . j . '. C
.'
.
.
.
fj....
'
'
.
.
.
( (.
.
..
njpd set, it is uniformly continuous on .
.
' '. ' . . lrEi ''.
'.'
.
.
.
..
.
,.
F y. jjl kj.g::( .)()('; k...
E .r:
..j
.
....
.. .
.
.
j
.
.
ryyy,y',.'yt;9y'jr,tty:j(,.i.yyy,rr,,y; tt'tkttyt, d fr eacil x e A, continity '' a.t x S be compact. C'..h,ptt ...!)E;1il):il).l!;)7tj!: ;..r' .'ij.tE itJ))j#15! .. ... ' 4).k6.). -. 7(; (.q-( ... .. . tjq. .)j k;jy'jiI-k::IIL.. y-;:!iii;y-. 5. .. ' :,!iij;;.. y. li .;;,.:ijj,. ..f- lil-. .IIkrIT).. (,11:::,14,.. :!ij;;--:1EE5111,( ...jj...y....-....C4;CIIi:jyjj,yy.. j;j!jrr:jjy. (qjj::--. ,11I::2jII),. jjjjyrr;jj:. iljj;;: rjjygq 11(2rrjjIIl,. ' . .... . ..yy.. y...y(IIE:r:11l,.. y .. jjjjgl:::)).. . (j11(r:)l1... t!jjI(4(:jpt.. ' gg.. ... .. j . jL:jji,,'''. .. : . .. E j . y . . . . '.. i... . j.j y..... .... . Jiii .).. . .(. . . .r....yyj .. y. y jg.. j. . j. ..yy.))j,. .y . . r jjjyy:t.. ,jjjjj)f:63L... ' ' . ., rrrjj;!lL . . .d1EE!i1. .. . .q-.'.rllqq-ll:E),s jj!jjj;j,. 4422::, d;::::: . y j-. . t. j .....j . .44:)( 4(:4):2,j-:, 11E1!2, . .... (Iir11.. . . tlEsll, (Ir:'' IIEE:!'' ;,jt-.,...jr))-...jj;tj.yjy gj .yy;y-(...-.,#.,.yyj,y,k.-.-.-.yy..y?yy)......t-,,..-.yt;y-.. jky.y..,y(y.yy(jyy!..-jyy-.. . j.yy.y ..y .j.... , .) .. y yyy.yjjyt.. jyy.., j.y;jj.. yk,-..-jy.)..;(.,... ... ,yyy.jjj. . .. ... jj-.. )j.,...;,tyy; 44:)( ((;), ((;), ((;), .; ..... jy.. ;y...yjtj;.....,jy..-jj#,,-.j;L... . ,.y.... . .. . . . . . ...j..... . . .
Proof Let A
.1jlgq,
-:rjjgijr,.
,gjj!r:r
,,.
,gyy..t: ,;....
. .
y..
y
.
E
-(:!i-l'
'
--. ..!tr:())yry-
.
.
...
...
-...
. .
.
:.,..
.,-
.
-
.
... ..
.-y. -.
-.74.
;' ):: l;!. ?!-):;)!t27tE.''''. . ,b:3b... :.'.'.,.Ekikkir:....-.. .;
--k::::::
.j;1d:r7
' .
.
.
.
''.
.
.
.
-
.
..''.
..
.
'.
-
-, . -
.,
.
qE-y q. -
E $it)kyy.)jtt.i..rjjjy)y -.-y j. .- ,, . .
,
.
....
. -
-...
..
.
,-jjj,!r!,jjj(j-.
. .
.
y
.
.
.
.
.
..yy)....r.-kt..j). q. ..
....
.
.
.
,
y .:
.
.
;.;(gt;-). .-.
..
,j.
t;y
E
' '-'..
.. .
.
..
.'Erjj,?.
.
!.
.
.. .
:
.
.
, -
''
,g;j/r:'
.
.,
E'7
. .
y
.
jjjyyygyjjj. ...
.
-,yy -.,
y
.
.
y
...
tyy . ,. y..y..;jj,-, r,jj:j:r:;;. . y-jjrrjqyjjjjj-t., - ,)..t,y.-.y,.yj.t,,yt.,yy,(.t, .y . . .
,
,
.
,
..
,-,.
-
..
.
.
y.
.
j
j . j..yy,.-y.y?.-y......-E.-y-.q.-..-.(q.y . ......-j;.;.;..-,:.,j-).j.;k.q.;jr-,y,..y,.yyy.-.t.y4.
...r.
.
.
-
...(j.....
....
.
. .
.
.
.
.
-
y-. y. .y yyy.
.
-
y. .
. . . . yy. g y-. . y-. yj y. -gyy. . . . . . y.. .. . .
y. .
,
-
-.
Mathematics
86
1,...,-. compact they contain a finite subcover, say Sdlxknr, k consider of points x,y e S such that dlx,y) < minlckcmrk, and a pair < zlE, and also e Sdxk,rk) for some kn so that pfx,fyjj dxkvx)S dlxk,yt + dlx,y) f rk + 6 S lrk,
using the triangle
inequality. Hence plflxnfxl)
S
f
.
(5.29)
r1e,and
< e. S p(J(x),f (x1))+ p(Axk),J(y)) p(f@),J@))
Since, 8 independent of x and y,
Let 8 Now, y =
=
(5.30)
is unifonnly continuous on A. w
T' is 1-1 onto, and f and f -1 are cont inuous, J is called a homeomorphism, and S and are said to be homeomorphic if such a function exists. lf S is homeomorphic with a subset of T', it is said to be embedded in 'T by f. If f also preserves distances so that p(f(x4,f@)) #@,y) for each x,y e S, it is called an isometry. Metrics J1 and #2 in a space S are equivalent if and only if the identity mapping from (S,J1) to (S,J2) (themapping which takes each point of S into itselg is an homeomorphism. If
.f:
S
F..+
'r
=
5.22 Example If Jx and poo are the metrics defined in (5.15) and (5.19) respectively, the mapping gl (R=,#x) ((0,1)=,px),where g = (:1,:2,...)and --)
is an homeomorphism.
Xi
1
'(x) 2 -+
=
2(1
dxnyt<
t.tkJ is
Ixfl)
,
i
=
(5.31)
1,2,...
EI
Right and left continuity are spaces, but there is a concept of the range of the function. A semicontinuous at x if for each
If
+
not well defined notions for general metric continuity which is with respect to function f: (S,#) >-> R is said to be upper E > 0 3 5 > 0 such that, for y e S,
=
/'(J)
<
fx)
(5.32)
+:.
0, upper semicontinuity a sequence of points in S and #@n,x) implies limsupa/xn) f fx). The level sets of the form (x: f(x) < aJ are open for a1l (x e R iff f is upper semicontinuous everywhere on S. f is lower semiis upper semicontinuous, and f is continuous at x iff it is both continuous iff upper and lower semicontinuous at x. A function of a real variable is upper semicontinuous at x if it jumps at x with fx) > max(/'(x-),.f(x+)J;isolated discontinuities such as point h in Fig. 5.4 are not ruled out if this inequality is satisfied, On the other hand, upper semicontinuity fails at point #. Semicontinuity is not the same thing as right/left continuity except in the case of monotone functions', if f is increasing, right tlef't) continuity is equivalent to upper (lower)semicontinuity, and the reverse holds for decreasing functions. The concept of a Lipschitz condition generalizes to metric spaces. A function f on s,dj satisfies a Lipschitz conditionat x e S if for > 0 3 M > 0 such that, for any y e Sdx,), -->
-f
Metric Spaces P(.f(A'),.f@)) f Mhdx,yq)
(5.33)
where (.): R+ F-> ER+satisfies hd) 4, 0 as d 0. It satisfies a unrm Lipschitz conditionif condition (5.33)holds uniformly, with tixed M, for all x e S. The remarks following (2.9) apply equally here. Continuity is enforced by this conditionwith arbitrary and stronger smoothness conditions are obtained for specialcases of h. .1,
,
Fig. 5.4
5.5 Function Spaces The non-Euclidean metric spaces met in later chapters are mostly spaces of real functionson an interval of R. The elements of such spaces are graphs, subsets of R2. However, most of the relevant theory holds for functions whose domain is any metric space (S,#), and accordingly, it is this more general case that we will study.Let Cs denote the set of all bounded continuous functions S F-+ R, and .f:
de/ne
duf
g)
spl
=
J(x)
I
gx)
-
(5.34)
.
xe S
dv is a metric.
5.23 Thorem
Proof Conditions 5 1(a) and (b) re immediate. To prove the triangle inequality CN, write, given functions J, g and h .
ttd .yy.(' t'yyty'
t';'j' ;'E y' t'(' )' t''r' y;t'. j':'jyjy' j'. y' (' .);' j'g' .-j;'. -' -'.jt'
.'.'
dfbhj
=
SLIPj
...
Ies
:
'.
'
.
'
..
...
-
hlx)
.'t ' '.
''
.
.
...4(:). 11.,1.. 4:yx) <- 154, jj(:),. ...j),b:. ). ;.3:''.' .x.....j. (.'.(. ).t . 1......#..... . ).I..jI..j .........4(:4.. .......+.,.... .j.. .
j!jj,r
S
IE
<-d
'
.E .E j : ( .. . . . .'' .. .. . ... .g . j.j . .!#.,p
..
q' . . ; g.: . . ... g.. jy . , .. . '
g.. .
,g.)
.....:.(
'
. .' g.. . .
.
.
.
. bhli ..
,
(5.35)
.
;yyr.,
,, .;y:ry.,:tj..t.yjyEy.yj..-.y i.).. ;.jLE:.y. ... ... . j.y j q.. -
( .. E E riy; . E .. ;)(-. E :..
(y)j
4:yx)j
E :'
(
'
. .
. ..
rt).yjy r ; .y( y. .q.. .): .. . r:l).))-.-'jjyyjjgjj(jgj(yjj(ytk.:i'q).;'f'l. . (2yjy y.. . j.3fk.i.. .yy ..,)L-) t!y(j-(..y. .ytj) (j-EJII?). ..' ' @.q .; )jq)j,gL li.ygry j)..)j) j,yy .;yjgjjy;j(jjtj;;y yyjjt yj;( jkjgr jjyi jjjjyj. gjypj:g;yjjyj y;ry..jy.jir.,,.ggyyj.;, .;.g....y....y,j.t-(y.jy.yy. gj. .
.
.
E- .
. ..)...:t;!..k...
:
.
) ;
j
j/ri!r
--....
..
...jt
-i
. . .. - y jEjqy t..t(Ehi.( ..
..
y
r
-y,.. .. ...-
, -,:j;
..
:.r,
-
,
q:,);jj..-L.L,.j;;.,,fjfL.. . . . k; , ....
-
-,
,,
,jfjj),.
jy....(:.j)-ri..(..y,.q jkg;y jjgyr $, gjygjggjj gkyjryyt,..44:. j)yljjk.jj. . y. . gjjg;j .jj. yjjyjyyjjjyj..,.. .. .j ;q f't;jtj.. vy,, t(; , yy t-.-tjj,..--tj:.?-.,gjykjjj,y--,(11). ,1:2:11Lqf ....y)j.jyjJ;;.-,g;yyr; . dlzz!ri ,1:4::,1 (11.-y -:3)q.jL:'6t3. jl:r-.lk:. ,L...tj,j,(.L)L;));;;--):-:fi. yj jII2:bi:.I2:bi:. 66113L32:'......,. jyjy yryj,jjj-.;jjs---, llk.zll. ..--tE!i!t.r.' :i4; Jjiki11:::),. jlr,:lk.ylrrrllr.rlk.tjjgrr)l fyurjyjjy (; ou yj uou s z.lttrit/lltq $:E1,1, ,..).L......j .,,,)#t)-t!s,i(.. yjjy. j.... rlrr:)ks .. . t..-,,.. ,r,.,y.-,1,). ....,,....s....-r......-..k....y,.z..-. tt. .-,,.,,. -,t!)1y),ty,,j;-.,-..... '.y?.-.-. .. . (-v...;;,.--k.,,,. iills. dk,r jj-,......-,,,,,,,,,-,-,,.;2,... ;;,,,,-#.s---,...,.,...,..,.,...=
.-d,d!!qjj-.
--
IS Compact,
.
'1rr.
.
..,j
.k E'ktliikt..
.1rr.
r
..yyj
.E#.(. .'.j)
t
.,,
...
-
-
. .
?
.
y
4:(11. . 14k1(:2: 4(:;1!' 112. 4kj2:ir;lj!'i!r 4k22: Io i!'-l!hl!'i!. llk(1. (1. llr!l. jry;?-21. ma.j
S
(x) (x)+ glx)
.jj.x,
.
,
,
..,;,.yy.:
Cs
k!r)-;,. y.. & 11::,-.
y ut. .#.. . ..g
.
<,
,..
.
.,
. .
,
.-,..1.
.
.
.
)
.
,
,.-y-.
:(q)
s..,..l
....., ,..
.
,
.,
.
functi o ns lf tj. !lr:.L d:::r d:::r Ik,r (1k. l. dlE:,dt:EE::'t!!,; atl (:j,,. m(14:,, a t ev. 'EEE!E as . -,
.
-,k-,.
-,k-,.
,-
.
-r,,--
Mathematics
88
a uniformly continuous restriction to S, and evel'y f e Us has a continuous extension to V, say J, constructed by setting Jx) = J(x) for a' e S and limu/xu) for a sequence fx:.G S l converging to x, for each x e V-S. Note that duf,f). so that the spaces Ck and Us are for any pair fbf' G S, dvf'l but these isometric. There are functions that are continuous on S and not on cannot be uniformly continuous. The following is a basic property of Cs which holds independently of the nature of the domain S. 'x)
=
=
,
5.24 Theorem
(Cs,#g) is complete.
Proof Let (Jnl be a Cauchy sequence in C's ; in other words, for 6: > 0 3 Ns 1 such that dulfnnfm) f E for n,m > Nz. Then for each x e S, the sequences (/@) ) satisfy Ifnx) fmx)l s Jtz(/,Al; these are Cauchy sequences in R, and so have limits fx). In view of the definition of dv, we may say that jn J unifonnly in S. For any x,y 6 S, the triangle inequality gives -
-->
lfx)
-.f(A') l f
1/'(x) fnx) l + l/@) -
fnlL
-
I + Ifn
Fix e > 0. Since fn e Cs, ('i 6 > 0 such that Ifnlx) Also, by unifonn convergence (; n large enough that -
maxfIfx) fnlx)l Ifaly) f@lI l < -
-
,
so that Ifx) J(y)1< z. Hence f e -
!
-f(.y)
(5.36)
.
I< Jn(.y)
Y if dx,y)
13s,
<
.
(5.37)
C's, which establishes that Cs is complete. w
Notice hpw this property holds by virtue of the unifonn metric. It is easy to devise sequencej of continuous functions converging to discontinuous limits, but none of these are Cauchy sequences. lt is not possible for a continuous function to be arbitrarily close to a discontinuous function at every point of the domain. A number of the results to follow call for us to exhibit a continuous function which lies. uniformly close to a function in Us, but is fully specified by a finite collection of numbers. This is possible when the domain is totally bounded.
5.25 Theorem Let (S,J) be a totally bounded metric space. For any f G &s, there existj for any : > 0 a function g e Us, completely specified by points of the such that dvjqj < E. D and rational numbers J1,...,Jm, domain .x1,...,au
We specify rational numbers here, because this will allow ud to assert in applications that the set of a11possible g. is countable.
Proof n By total boundedness of S, R for 8 > 0 a finite ,1) xi, 1et Af = fx: dx,xi) k 26) and Bi (x: dxnxi) K gi: S r- 0, 1J by '
(x1,...,au)For
-net
=
gix)
dxli) =
dxAij
+ dlX,
s i ),
each and detine functions .
,
(5.38)
where dxA) = infysxltmy). dxl) is a uniformly continuous function of by constmction, and gilx) is also uniformly continuous, for the denominator is never .:
Metric Spaces
less than
).
89
Then define
'bl '4x) '7 -lp/xl7f
=
(5.39)
.
-1:/x)
Being a weighted average of the numbers (JfJ,gx) is bounded. Also, since (xflis for S, there exists for every x e S some i such that dxli) k as well a ZT=Ip(x) and < dx,Bi) dx,xi) Therefore, f k hence that gix) such as and unifonn continuity extends from the gi to g. For arbitrary f e &s, fix E > 0 and choose small enough that Ifx) fyl l < ls when #(x,y) < 26, for any x,y e G. Then tix m large enough and choose xf and ai Jfl for each i. 1,...,-, so that the Sxinn) cover S, and lfx < for i Note that if dxtx k 28 then e Aj and gix) = 0, so that in a1l cases -net
,
,
.
-
=
Y
-
-x
gixtlfx) fx I f
.ftxls.
(5.40)
-
Hence
gix)Ifxt
-
ai
I<
lfx) f
gix)
x
-
I+
Ifx
gix)
-
ai I
< gixtz for each x
6
(5.41)
S and each i. We may conclude that
duf,gl
=
sup Ifx)
-:(x)1
x6 5
S sup
ZT-1p(x)I fx)
Jfl
-
<
Z:'-1'/x)
xss
z.
(5.42)
.
The next result makes use of this approximation theorem, and is fundamental. It tells us (recallingthe earlier discussion of separability) that spaces of continuous functions are not such alien objects from an analytic point of view as they might at first appear, at least when the domain is totally bounded.
5.26 Theorem (i) If (S,#) is totally bounded then (&s,Jg) is separable. (ii) If (S,#) is compact then cn,dulis separable. Proof We need only prove part (i), since for part (ii), Cs Us by 5.21 and the same conclusion follows. Fix m and suitable points (xl,...,aulof S so as to define a countable family of functionsAm = fgmk,k e IN), whUre the g.k are defined as in 5.25, and the index k of rationals. For enumerates the countable collectibn of lrvectofs (J1,...,tu) > 0, there exists m large enppgh yhqt,for each f e &s, duf, gm < : for : each 5 ij, tits'ftibi, apd there exists gk e A such that somek. By 1. A limp-xA?nTh@'! 'd.ttlh,gtA 0. : is dense in &s. . < ,, dvy, gk) : for every > yytlyyttttt. yyty d1:::,i-....t).',t)'tt ..jj3i,1.. ,).() !j,, yj,jsy, EIl..jjojoug uyytsy mox geueraj jyy ..... ., To show that we canno :lrr-.d.pir,... j));;.,j);)q.;j...;);)... ;;)))..;....-..; .,.434... yy...;jygr-.j-y--.. .. )j-#).;-y--t..r.-,)g. y)-... yty. ...t-y.y-.)(tyy.y,yy. =
'
.
,
'.
.
=
'
..
.
.
..y,,y.
.)
'ril,.
'
.
' 'ttyljk
,
,,
'tij.'yjy,,yt.#,y,
--.i1friil-.
.1r:. .)ky,yy..yyy-. ..... r;-.)t);t@ -jyyt-r),t-.y.ytyj-,.-.
,typtl-l-y-y-)..
.
... . .
........... ..
-..y
.
-.
.
..
Mathematics
90
function space.
circumstances, we exhibit a nonseparable
5.27 Example For S (tz,:J,an interval of the real line, consider the metric space Dka,bndu) of real, bounded cadlag functions of a real variable. Cadlag is gauche) to describe droite, limites a colourful French acronym continue functions of a real variable which may have discontinuities, but are right continuous at every point, with the image of every decreasing sequence in Ltz,:)containing its limit point; in other words, there is a limit point to the left of every point. Of course, Cp,s) c Dp,x. Functions with completely arbitrary discontinuities form a larger class still, but one that for most purposes is too unstructured to permit a useful theory. =
To show that Dfa,bjsdu) is not separable, conslder the subset with elements
J/f)
0, t
<
0
=
1, t k 0
,
0
(5.43)
EJ,1.
6
This set is uncountable, containing as many elements as there are points in gtz,hJ. But duh,h 1 for 0 # 0', so it is also discrete. Hence Dta,bj,du) is not separable by 5.9. (n =
Let A denote a collection of functions f : (S,#) F-y (T,p). A is said to be equicontinuous at x G S if Y 6: > 0 3 > 0 such that
sup .feA y
sup pfyt,fxj) G
Sx,&)
,4 is also said to be unformly
supa:ssup
feA
s
(5.44)
< :.
if Y 6: > 0 3 8
equicontinuous
sup pfyl,fxl)
>
0 such that
(5.45)
< :.
yc Saxnnl
Equicontinuity is the property of a set of continuous functions (or uniformly continuous functions, as the case may be) which forbids limit points of the set to Cs (&s) but A is not be not (uniformly)continuous. ln the case when (uniformly) equicontinuous, we cannot rule out the possibility that W Cs (Us)k An important class of applications is to countable A, and if we restrict attention to the case A = ffn, n e IN) A c Cs torUs) may not be essential. lf we are willing to tolerate discontinuity in at most a finite number of the cases, the following concept is the relevant one. A sequence of functions f/, n e IN) will be said to be asymptotically equicontinuous at if Y E > 0 3 > 0 such that :4
,
.x
limsup l-yx
and asymptotically
sup
ye Sdxszj
?zzif/brzlfy
p(A(y),Jn(x)) < e,,
equicontinuous if V e, > 0 R
(5 46) >
0 such that
Metric Spaces
limsup sup n--:x
;r
e
sup
91
p(Jn@),n(x)) <
(5.47)
E.
Jx,&)
ye
If the functions fn are continuous for a11n, limsupn-yx can be replaced by supn in (5.46) and similarly for (5.47)when all the fn are uniformly continuous. ln these circumstances, the qualifier asymptotic can be dropped. The main result on equicontinuous sets is the Arzel-Ascoli theorem. This designation covers a number of closely related results, but the following version, which is the one appropriate to our subsequent needs, identifies equicontinuity as the property of a set of bounded real-valued functions on a totally bounded domain which converts boundedness into total boundedness.
5.28 Arzel-Ascoli theorem Let (S,#) be a totally bounded metric space. A set A c Cs is relatively compact under dv iff it is bounded and uniformly equicontinuous. Proof Since Cs is complete, total boundedness of A is equivalent to relative compactness by 5.13. So to prove if' we assume boundedness and equicontinuity, and construct a tinite E-net for A. It is convenient to define the modulus of continuity of f, that is, the function w: C xR+ h- R+ where ,
s
w(T,) Fix s
>
0, and choose 8
=
sup a.s s
(as is
If(.y4 J@)I
sup
.
ysdx,)
equicontinuity)
possible by unifonn
sup w(.f,)
(5.48)
-
such that
(5.49)
< :.
ysA
Boundedness of A under the unifonn metric means that there exist finite real numbers U and L such that inf
L f
/'lA.xe
be (x1,...,xm)
Let
S
#4x)f
sup DA,XC
U.
(5.50)
5
for S, and construct the finite family
-net
a Dm
fx) S
=
l'k
A, k
q
=
1,...,(v + 1)-)
(5.51)
according to the recipe of 5.25, with the constants ai taken from the finite set
where u and v are integers with v exceeding CU L4Ie and u 0,...,v. This set contains v + 1 real values between U and L which are less than 6: apal't, so that Dm has (v+ 1)= members, as indicated. Since the assumptions imply
fL
+
(&- Ljulvj
-
,
=
4 c &s, it follows by 5.25 that for every f E A there exists gk e Dm with < :. This shows that Dm is a e-net for A, and A is totally bounded. dvf,g ito suppose A is relatively compact, and hence totally bounded. To prove Trivially, total boundedness implies boundedness, and it remains to show unifonn equ icontinuity. Consider for E > 0 the set ionly
,
'
Mathematics
#k(:)
(/': w(/',1/k)
=
< :).
(5.52)
Uniform equicontinuity of A is the condition that, for any
large enough that
X'c
Bkz). lt is easily veritied
lw(.f,)
-
w(',*
6: >
0, there exists k
that
I f 2JrXJ,'),
(5.53)
R*,dE)
F--> is continuous. Bkz) is the inso that the function w(.,): (Cs,d verse image under w(.,8) of the half-line (0,:) which is open in R+, and hence
x 0 as k for each f e Bkzj is open by 5.19. By definition of Cs, w(/', 1/k) Cs In other words, w converges to 0 pointwise on Cs which implies that the Cs, and hence forW.But b.collection (#k(:),k e IN) must be an open coverinAfor covering is such of has A X hypothesis compact, every a finite subcover, and so A q Bke) for finite k, as required. . -->
--
.
,
6 Topology
6. 1 Topological Spaces Metric spaces fonn a subclass of a larger class of mathematical objects called topological spaces. These do not have a distance defined upon them, but the concepts of open set, neighbourhood, and continuous mapping are still well defined. Even though only metric spaces are encountered in the sequel (Part VI), much of the reasoning is essentially topological in character. An appreciation of the topological underpinnings is essential for getting to grips with the theory of
weak convergence. 6.1 Definition A topological space (X,I) is a set X on which is defined a topology, a class of subsets called open sets having the following properties: (a) X e T, O e 1. (b) If C i T, then U()scoe T. Oa G T, then 01 f-'h 02 q T. n (c) If 01 G 'r
'r,
These three conditions dehne an open set, so that openness becomes a primitive concept of which the notion of e-spheres around points is only one characterization. A metric induces a topology on a space because it is one way (thoughnot the only way) of detining what an open set is, and a1l metlic spaces are also topological spaces. On the other hand, some topological spaces may be made into metric spaces by defining a metric on them under which sets of are open in the sense defined in j5.1. Such spaces are called metrizable. A subset of a topological space (X,I) has a topology naturally induced on it by (A f''h 0: O G 'IJ is called the the parent space. If A c X, the collection relative topology for A. (A,n) would normally be referred to as a subspace of X. and If two topologies are defined on a space and :1 c T2, then T1 is said to be coarser, or weaker, than T2, whereas 12 is hner stronger) than I1. In particular, the power set of X is a topology, called the discrete topology, whereas (?,x J is called the trivial topology. Two metrics define the same topology on a space if and only if they are equivalent. If two points are close in one space, their images in the other space must be correspondingly close. If a set O is open, its complement Oc on X is said to be closed. The closure X' of an arbitrary set A ? X is the intersection of a1l the closed sets containing A. As for metric spaces, a set A i #, for B c X, is said to be dense in B if B c X. ':
'u
':1
=
.:2
6.2 Theorem The intersection of any collection of closed sets is closed. X and O are both open and closed. n
Mathematics
94
However, an arbitral'y union of closed sets need not be closed, just as an arbitrary intersection of open sets need not be open. For given e X, a collection Vx of open sets is called a base for the point x if for every open O containing x there is a set B e Vxsuch that x e B and B c 0. This is the generalization to topological spaces of the idea of a system of neighbourhoods or spheres in a metric space. A basefor the topology T on X is a and every x 6 0, there exists B e collection V of sets such that, for every O 6 that V such x e B c 0. The definition implies that any open set can be expressed union the of sets from the base of the topology; a topology may be defined for as specifying a base collection, and letting the open sets be defined as a space by the unions and finite intersections of the base sets. ln the case of R, for example, the open intervals form a base. .x
,r,
6.3 Theorem A collection V is a base for a topology I on X iff (a) Ulevf = X. (b) V #1,#2 e F and x e #1 chB1. 3 B5 e V such that a' e #3 c #1 /n %. Proof Necessity of these conditions follows from the definitions of base and open in terms of the base F, as follows: set. For sufficiency, define a collection 'r
O e
'r
iff, for each
.'&'
e 0, H B e F such that x
q
B c 0.
O s4tisfies the condition in (6.1), and X satisfies it given condition (a) of the theorem. lf C is a collection of T-sets, UtxwO e T since (6.1) holds in this case in respect of a base set B corresponding to any set in C which contains x. And if 01,02 G 1, and x e 01 fa O1. then, letting #1 and B1 be the base sets specified in (6.1) in respect of x and 01 and 01 respectively, condition (b) implies that x e B5 c 01 f''h O1, w ltic h shows that I is closed under finite intersections. Hence, is a topology for X. . ':
The concept of base sets allows us to generalize two further notions familiar points) of a set A in a from metric spaces. The closure points (accumulation topological space (X,'r) are the points x e X such that every set in yhe base of a7 contains a point of A (a point of A other than x). An important exercise is to show that x is a closure point of A if and only if is in the closure of z4. We have generalizations of two other familiar concepts. A sequence tak) of points in a topological space is said to converge to x if, for every open set O containing x, H N k 1 such that ak e O for all n k N. And is called a cluster point of (.:kl if, for every open O containing and every N k 1, xn e O for some n k N. ln general topological spaces the notion of a convergent sequence is inadequate for characterizing basic properties such as the continuity of mappings, and is augmented by the concepts of net and///er. Because we deal mainly with metric spaces, we do not require these extensions (seee.g. Willard 1970: Ch. 4). .x
.x
.x
6.2 Countability and Compactness The countability tutppu provide one classitkation of topological spaces according, roughly speaking, to their degree of structure and amenability to the methods
Topology
95
of analysis. A topological space is said to satisfy the first axiom of countability (to be jrst-countable) if every point of the space has a countable base. It if the space as a satisfies the second axiom of countability (is second-countable) whole has a countable base. Every metric space is first-countable in view of the existence of the countable base composed of open spheres, Sx, 1/n) for each x.
More generally, sequences in first-countable spaces tend to behave in a similar manner to those in metric spaces, as the following theorem illustrates.
6.4 Theorem ln a first-countable space, x is a cluster point of a sequence (.v, n e IN) iff there is a subsequence (xa#,k s IN) converging to x. Proof Sufficiency is immediate. For necessity, the definition of a cluster point implies that 3 n k N such that xn e 0, for every open 0 containing x and every N k 1. Let the countable base of be the collection Vx fBi, i G (11), and choose a monotone sequence of base sets (Ak, k G INl containing x (andhence nonempty) with A1 #l, and Ak c Ak-l f''h Bk for k 2,3,...; this is always possible by 6.3. Since x is a cluster point, we may construct an infinite subsequence by taking xnk as th: next member of the sequence contained in 4s for k = 1,2,... For every open set O containing x, R N k 1 such that xnk E Ak 0, for a1l k k N, and hence xnk x, as required. w --> x as k .x
=
=
=
.-..:
The point of quoting a result such as this has less to do with demonstrating a new property than with reminding us of the need for caution in assuming properties we take for granted in metric spaces. While the intuition derived from R-like situations might lead us to suppose that the existence of a cluster point and a convergent subsequence amount to the same thing, this need not be true unless we can establish first-countability. A topological space is said to be separable if it contains a countable dense subset. Second-countable spaces are separable. This fact follows directly on taking a point from each set in a countable base, and verifying that these points are dense in the space. The converse is not generally true, but it is true for metric spaces, where separability, second countability and the Lindelf property (that every open cover of X has a countable subcover) are all equivalent to one another. This is just what we showed in 5.6. More generally, we can say the
following.
6.5 Theorem A
jecond-countable
space is both separable
and Lindelf.
Proof The proof of separability is in the text above. To prove the Lindelf property, let C be an open cover of X, such that UzeCA X.For eachA e C and e A, we can find a base set Bi such thgt x 6 Sj c A. Since UT=1#fX, we may choose a countable subcollection Af, i 1,2,... such that Bi c 4f for each i, and hence U7=1Xf X. K .x
=
=
=
=
A topological space is said to be compact if every covering of the space by open countably compact if each countable sets has a finite subcover. It is said to e coveringhas a finite subcovering. And it is said to be sequentially compact if subsequpce. Somesimes, compacteach sequence on the space has a convrgnt
96
Mathematics
ness is more conveniently characterized in tenns of the complements. The complements of an open cover of X are a collection of closed sets whose intersection is empty', if and only if X is compact, every such collection must have a finite subcollection with empty intersection. An equivalent way to state this proposition is in terms of the converse implication. A collection of closed sets is said to have the Tnite intersection property if no finite subcollection has an empty intersection. Thus: 6.6 Theorem X is compact (countablycompact) if and only if no collection (countable collection) of closed sets having the finite intersection property has an empty intersection. a The following pair of theorems summarize important relationships different varieties of compactness.
between the
6.7 Theorem A first-countable space X is countably compact iff it is sequentially compact. Proof Let the space be countably compact. Let fxn,n G EN ) be a sequence in X, and detine the sets Bn fxn,ak+1,... ) n 1,2,... The collection of closed sets (Tn, n e (N) clearly possesses the finite intersection property, and hence (X'n is nonempty by 6.6, which is another way of saying that fak) has a cluster point. Since the sequence is arbitrary, sequential compactness follows by 6.4. This proves necessity. For sufficiency, 6.4 implies that under sequential compactness, al1 sequences in X have a cluster point. Let ( Ci, i e INl be a countable collection of closed sets having the finite intersection property such that A,y 0Z=IC/ # 0, for every finite n. Consider a sequence fxn) chosen such that xn e A,,, and note since (A,, ) is monotone that xn e Am for all n 2 m; or in other words, Am contains the /?z). Since (ak) has a cluster point x and Amis closed, x e Am. sequence fak,n k This is true for every m e N so that OT=1Q is nonempty, and X is countably 6.6. by compact w =
,
=
=
,
6.8 Theorem A metl'ic space (S,#) is countably Proof
compact iff it is compact.
Sufficiency is immediate. For necessity, we show first that if S is countably compact, it is separable. A metric space is first-countable, hence countable compactness implies sequential compactness (6.7),which in turn implies that every sequence in S has a cluster point (6.4).This must mean that for any 6: > 0 there exists a finite E-net (m,...,auJ such that, for a1l x e S, dx,xk) < E, for some k E ( 1,...,-) ; for otherwise, we can construct an infinite sequence fxn) with dxn,xn') k 6: for n # n', contradicting the existence of a cluster point. Thus, for each n e IN there is a finite collection of points 4,, such that, for is countable and every x e S, #@,y) < 2-n for some y e An. The set D = U*n=1An dense in S, and S is separable. Separability in a metric space is equivalent by 5.6 to the Lindelf property, that every open cover of S has a countable subcover', but countable compactness implies that this countable subcover has a finite subcover in its turn, so that
Topology
compactness is proved.
97
.
Like separability and compactness, the notion of a continuous mapping may be defined in tenns of a distance measure, but is really topological in character. In the mapping /': X F--y f is said to be contina pair of topological spaces X and uous if J -1(#) is open in X when B is open in and closed in X when B is closed in That in metric spaces this definition is equivalent to the more familiar one in terms of s- and 8-neighbourhoodsfollows from 5.19. The concepts of homeomorphism and embedding, as mappings that are respectively onto or into, and 1-1 continuous with continuous inverse, remain well defined. The following theorem gives two important properties of continuous maps. ,
,
.
6.9 Theorem Suppose there exists a continuous space X onto another space (i) If X is separable, T is separable. is compact. (ii) If X is compact,
mapping
f
from a topological
.
problem is to exhibit a countable, dense subset of Consider .f(D4 where D is dense in X. lf J(D) is the closure of J(D), the inverse image f-JfD is closed by continuity of f, and contains f -1(/'(D)), and hence also coptains D by 1.2(iv). Since X'is the smallest closed set containing D, and X c 5, /X) it follows that X c J-1C(D)). But since the mapping is onto, f ff-1(fD))4 c J(D), where the last inclusion is by 1.2(v). J(D) is therefore ' in dense as required. J(D) is countable if D is countable, and the conclusion follows directly. (ii) Let C be an open cover of Then (J -1(#): B e C) must be an open cover of X by the definition. The compactness of X means that it contains a finite subf-1B n ) such that cover, say f -1(B l ) Proof
(i) The
.
=
.
,...,
y'
=
y(x) y =
Uy-'tsp= J y-1 tjsy c /=1 Usy,
y=1
(6.2)
j=L
where the third equality uses 1.2(ii) and the inclusion, 1.2(v). Hence C contains a Gnite svbcover. . in both these results. The extension Note the importance of the stijulation and f is obvious, and can be supplied of (ii) to the case of compact subsets of X reader. the by Completeness, unlike separability, compactness, and continuity, is not a topological property. To define a Cauchy sequence it is necessary to have the concept of a distance between points. One of the advantages of defining a metric on a space is that the relatively weak notion of completeness provides some of the essential features of compactness in a wider class than the compact spaces.
6.3 Separation Properties Another classification of topological spaces is provided by the separation tztppzu, whichin one sense are more primitive than the countabilityaxioms. They are
Mathematics
98
indicators of the richness of a topology, in the sense of our ability to distinguish between different points of the space. From one point of view, they could be said to define a hierarchy of resemblances between topological spaces and metric spaces. Don't confuse separation with separability, which is a different concept altogether. A topological space X is said to be: with x # y 3 an open set containing x but not y and - a Fl-space, iff V x,y e X also an open set containing y but not Fa-) space, iff V x,y e X with x # y 3 disjoint open sets 01 and - a Hausdotffo 01 in X with e 01 and y E O2; regular iff C Hdisjoint open sets 01 and for each closed set C and space - a 02 with x e 01 and C c O2; - a normal space iff, given disjoint closed sets C1 and Cz, 3 disjoint open sets 01 and 01 such that C1 c 01 and (72 c O2. .x;
.x
.x
A regular Fl-space is called a Fg-space, and a nonnal Fl-space is called a F4-space. In a Fl-space, the singleton sets (.:) are always closed. In this case y e (xJf (xJC is the complement of a closed set, and hence open. whenever y # x, where Conversely, if the F1 property holds, evel'y y # x is contained in an open set not containing x, and the union of all these sets, alsp open by 6.1(b), is (x)f. It is easy to see that F4 implies Tz implies Fa implies T1, although the reverse implications do not hold, and without the F1 property, nonnality need not imply regularity. Metric spaces are always F4, for there is no difficulty in constructing the sets specified in the definition out of unions of E-spheres. We have the following important links btween separation, compactness, countability, and metrizability. The first two results are of general interest but will not be exploited directly in this book, so we forgo the proofs. The proof of 6.12 needs some as yet undefined concepts, and is postponed to j6.6 below.
6.10 Theorem A regular Lindelf
space is normal. a
6.11 Theorem A compact Hausdorff space is F4. 6.12 Urysohn's able. u
metrizatlon
(a
theorem A second-countable
Fg-space is metriz-
In fact, the conditions of the last theorem can be weakened, with F4 replaced by Fa in view of 6.10, since we have already shown that a second-countable space is Lindelf (6.5). The properties of functions from X to the real line play an important role in defining the separation properties of a space. The key to these results is the remarkable Urysohn's lemma.
6.13 Urysohn's Iemma A topological space X is nonnal iff for any pair A and B of disjoint closed subsets there exists a continuous function f: X (0,1j such = 0 and J(#) 1. :l that jA) .-+
=
The function
J is
called a separating
function.
Topology
99
Proof This is by construction of the required function. Recall that the dyadic rationals D are dense in (0,1j. We demonstrate the existence of a system of open sets f Ur, r e D l with the properties A c &r;
Vrfa
#
'X c
0',
=
(6.4)
Ur for r
>
(6.5)
s.
Nonnality implies the existence of an open set 171/2(say)such that 171/2contains :4 an d (&1/2)ccontains B. The same story can be told with &f/2 replacing B in the role of C1 to define 171/4,and then again with 171/2replacing A in the role of C1 to define Um. The argument extends by induction to generate sets ( Umnn, m 1,...,2* 1 ) for any n e EN,and the collection ( Ur, r e D ) is obtained on letting n --A x. lt is easy to verify conditions (6.3)-(6.5)for this collection. Fig. 6.l illustrates the construction for n 3 when A and B are regions of the plane. One of the onion' in the limiting case. must imagine countably many more =
-
=
)1
!/3/8
315ls
:/1/2
!/3/4 Llzls
:/1/4
U 1/8
B
,4
Fig. 6.1 Now define
f:
X
(0,11by
--)
inffr
e D: x e &r) x e ,
Urss Ur
x Urso Ur
(6.6)
-
1 for x e #. Because of the monotone property (6.5), where, in particular, we have f:r any a e (0,1) .f(x)
=
(x: fx)
<
al
(x:inf fr e D: x e = (x:H r < a such that =
&r, = r
<
a)
x e &r)
(6.7)
which is open. On the other hand, because D is dense in g0,1qwe can deduce that,
Mathematics
100
for any k$e
0, 1),
(x: flx)
<
I31 (x:inf tr e =
D: x e &rl <
= (x: x e Ur V r
>
jlJ
))
Ur O rn.js
=
=
r)
(6.8)
r
r>f$
which is closed. Here, the final equality must hold to reconcile the following two and second that, for al1 r > I3,there exists (sinceD facts: first that Ur i and Us Ur by (6.5).We have therefore shown that, > > dense) D with is se r s k$ c for 0 S ;$< a ; 1, ,
(x: j
<
fx)
<
a)
(.x:flx) <
=
a)
fa
fx: flx) K
;'$)C
(6.9)
is open, being the intersection of open sets. Since every open set of (0,11is a union of open intervals (see2.5), it follows that f -1(A) is open in X whenever A is open in g0,11,and accordingly f is continuous. It is immediate that f(A4 0 and f(#) 1 as required, and necessity is proved. Sufficiency is simply a matter, given the existence of f with the indicated and J-1((11j), whose images are PrOPerties of citing the two sets J-1(g0,1)) a a, and respectively, in X, disjoint, and contain B so that X is normal. w open =
=
,
,4
It is delightful the way this theorem conjures a continuous function out of thin air! It shows that the properties of real-valued functions provide a legitimate means of classifying the separation properties of the space. In metric spaces, separating functions are obtained by a simple direct construction. If 4 and B are closed and disjoint subsets of a metric space (S,#), the nonnality property implies the existence of > 0 such that infxsz,yswJtmy) k 6. The required function is
fx)
=
Jtx,Al dxv)
+ ty(x,s)
(6. 10)
where dxl) = infysxttx,y), and J(x,S) is defined similarly. The continuity of f follows since dxl) and #(x,#) are continuous in x, and the denominator in (6.10) A similar construction was used in the proof of 5.25. is bounded below by The regularity property can be strengthened by reqtliring the existence of separating functions for closed sets C and points x. A topological space X is said to be completely regular if, for a1l closed C c X and points x C, 3 a continuous 0 and flx) = 1. A completely regular Fl-space function fl X 1- E0,1)with fc' is called a Tychonoff space or Fgj-space. As the tongue-in-cheek terminology suggests, a L-space is F3j (this is immediate from Urysohn's lemma) and a F3j-space is clearly F3, although the reverse implications do not hold. Being F4, metric spaces are always F?j. .
=
Topology
101
6.4 Weak Topologies Now let's go the other way and, instead of using a topology to define a class of real functions, use a class of functions to define a topology. Let X be a space and F a class of functions f : X i-- y, where the codomains y are topological under which spaces. The weak topology induced by F on X is the weakest topology -1 is continuous. Recall, continuity F (A) is means that J open in X every f e whenever A is open in y. We can also call the weak topology the topology generated by the base of sets V consisting of the inverse images of the open sets of the y, under f e F, together with the finite intersections of these sets. The inverse images themselves are called a sub-base for the topology, meaning that the sets of the topology can be generated from them by operations of union and finite
intersection.
increase the number of sets in this base and If we enlarge F we (potentially) get a stronger topology, and if we contract F we likewise get a weaker topology. With a given F, any topology stronger than the weak topology contains a richer collection of open sets, so the elements of F must retain their continuity in this case, but weakening the topology further must by definition force some fq F to be
discontinuous.
The class of cases in which y R for each f suggests using the concept of a weak topology to investigate the structure of a space. One way to represent the richness of a given topology on X is to ask whether contains, or is contained in, the weak topology generated by a particular collection of bounded real-valued functions on X. For example, complete regularity is the minimal condition which makes the sol4 of construction in 6.13 feasible. According to the next result, this is sufficient to allow the topology to be completely characterized in terms of bounded, continuous real-valued functions on the space. =
':
'r
6.14 Theorem lf a topological space (X,':) is completely regular, the topology is the weak topology induced by the set F of the separating functions. ':
Proof Let F denote the collection of inverse images of open sets under the funcr tions of F. And 1et F denote the weak topology induced by F, such that the F-sets, together with their finite intersections, fonn a base for F. We show that F T. For any x e X, let O e be an open set containing x. Then Oc is closed, and by l and complete regularity there exists f e F taking values in (0,1j with J(I) = J(O9 J-1((, 1q) is therefore an is open in g0,1j, and B 0. The set (1z,1) with Oc containing disjoint and that B c 0. Since this holds for x so open set, every such 0, x has a base Vx consisting of inverse images of open sets under functions from F Since x is arbitrary the collection V = ( Vx,x e XJ forms a base for T. lt follows that T ? F. On the other hand, F is by definition the weakest topology under which every f E F is continuous. Since f E F is a separating function and continuous under it also follows that F c 1. . =
'r
=
=
.
'r,
Mathematics
6.5 The Topology of Product Spaces Let X and f be a pair of topological spaces, and consider the product space X x The plane R x R and subsets thereof are the natural examples for appreciating the properties of product spaces, although it is a useful exercise to think up more exotic cases. An example always given in textbooks of topology is C XC where C is the unit circle; this space has the topology of the torus (doughnut). Let the ordered pair (x,y)be the generic element of X x f. The coordinate defined by projections are the mappings 7:x: X x F-) X and z:v: X x f F-> .
,
lx(.Y,')
=
A'
(6.11)
1:(-Y,)/)
=
y.
(6.12)
are topological spaces, the coordinate projections can be used to generate a new topology on the product space. The product topology on X x is the weak topology induced by the coordinate projections. The underlying idea here is very simple. lf A i X and B q f are open sets, the 1 1(#) will be lk - (z4)an d X x B = 1:yset A x B (A x f) rn (X x #), where A x f rectangle called of X The product regarded as open in X x and is x f. an open topology on X x is the one having the open rectangles as a base. This means that and @2,y2) two points @1,y1) are close in X x provided xl is close to x2 in X, antl yl to y2 in f. Equivalently, it is the weakest topology under which the coordinate projections are continuous. If the factors are metric spaces (X,#x) and f,h), several metrics can be constructed to induce the product topology on X x including If X and
=
=
,
,
,
@2,y2)) maxllxtxl,xzl, p((x1,y1), =
Jr(A'1,y2)l
(6.13)
and p'(@1,A'1), @2,y2)) =
#x@1sx2)+#:/1,y2).
An open sphere in the space (X x p), where p is the metric in happens to be an open rectangle, for ,
5'p(@,A'),)
=
Sdsxnnt
X Sdj,vnt',
(6.14) (6.13),
also
(6.15)
but of course, this is not true for every metric. Since either X or may be a product space, the generalization of these results from two to any finite number of factors is straightforward. The generic element of the space X2=1Xfis the n-tuple (x1,...,ak: xi e Xf), and so on. But to deal with infinite collections of factor spaces, as we shall wish to do below, it is necessary to approach the product from a slightly different viewpoint. Let 4 denote an arbitrary index se and fXa, (x l A ) a collection of spaces indexed by is the collection of a1l the mappings x: 4 4. The Cartesian product X Xasxxa F-y Uaszxa such that x(a) e Xa for each (x e A. This definition contains that given in j1.1 as a special case, but is fundamentally more general in character. The coordinate projections are the mappings 1a: Xn, F-> Xa with ,t
=
Topology
103
l(x(.%) Xq),
(6.16)
=
but can also be defined as the images under x of the points a e A. Thus, a point in the product space is a mapping, the one which generates the coordinate projections when it is evaluated at points of the domain z4. In the case of a finite product, A can be the integers 1,...,l. In a countable case A EN,or a set equipotent with EN,and we should call an infinite sequence, an element of =
.x
X'='(say).A familiar uncountable example is provided by a class of real-valued functions x: R F-> R, so that A R. In this case, x associates each point a e R R with a real number x(a), and defines an element of the product R The product topology is now generalized as follows. Let (Xa, a e A ) be an arbitrary collection of topological spaces. The Tychonoff topology (product X topology) on the space X h as as base the finite-dimensional open rectangles, sets where the Oa ? Xa are open sets and Os Xa except for at of the form Xas.4otx, most a finite number of coordinates. These basic sets can be written as the interkections of tinite collections of cylinders, say =
.
=
B
-1 aajtoaj)
=
-1
f-h
chzramto oo ;
...
,
o j'y; .
for indices a1,...,a,a e A. Let be a topology under which the coordinate projections are continuous. If Oz and hence contains the Tychonoff topology. Since is open in Xa, aa-1(O(: ) e this is true for any such 1, we can characterize the Tychonoff topology as the weak topology generated by the coordinate projections. The sets C:a-1(Oa) fonn the sub-base for the topology, whose hnite intersections yield the base sets. Something to keep in mind in these intinite product spaces is that, if any of X the sets Xct are empty, X i s emp ty. Some of our results are true only for nonempty spaces, so for full rigour the stipulation that elements exist is desirable. 'r
'r
':
,
6.15 Example The space hdv) examined in j5.5 is an uncountable product space having the Tychonoff topology; the unifonn metric is the generalization of the maximum metric p of (6.13).Continuous functions are regarded as close to one another upder dv only if they are close at every point of the domain. The subsequent usefulness of this characterization of C,du) stems mainly from the fact that the coordinate projections are known to be continuous. n Thetwoessentialtheorems onproductspacesextendseparability andcompactness from the factor spaces to the product. The following theorem has a generalization to uncountable products, which we shall not pursue since this is harder to prove, and the countable case is sufticient for our purposes.
6.16 Theorem Finite or countable product spaces are separable under the product topology iff the factor spaces are separable. Proof The proof for tinite products is an easy implication of the countable case, hence consider X* XT=fXf.Let Df fdikndizn... l c Xf be a countable dense set by D each and construct a set i, detining for c =
=
X''O
104
Mathematics
Fm
=
Xojx
i=m+ 1
f=l
fl l
(6. 18)
for m = 1,2,..., and then letting D U;=IF-. Fm is equipotent with the set of m-tuples formed from the elements of the countable DI ,...,Dm, and is countable by induction from 1.4. Hence D is countable, as a countable union of countable sets. We will show that D is dense in X=. Let B = X7=IO/ be a non-empty basic set, with Oi open in niand Of niexcept for a finite number of coordinates. Choose m such that Oi Xf fOr i > m, and then =
=
=
m
B ra Fm
=
=
x X ftl X((?/rno/) f=rn+1
=1
)
y,
?.
(6.19)
l,...,m. Since recalling that the dense property implies Oi fa Di # 0, for B fa Fm B ?'7 D, it follows that B contains a point of D; and since B is an arbitrary basic set, D is dense in X7 as required. w =
One of the most powerful and important results in topology is Tychonoff's theorem, which states that arbitrary products of compact topological spaces are also compact, under the product topology. It will suftice here to prove the result for countable products of metric spaces, and this case can be dealt with using a more elementary and familiar line of argument. lt is not necessary to specify the metrics involved, for we need the spaces to be metric solely to exploit the equivalence of compactness and sequential compactness.
6.17 Theorem A finite or countable product of separable metric spaces (Xf,) is compact under the product topology iff the factor spaces are compact. Proof As before, the finite case follows easily from the countable case, so assume X= = X:=1Xj, where the Xf are separable spaces. In a metric space, which is first countable, compactness implies seprability and is equivalent to sequential compactness by 6.8 and 6.7. Since Di is sequentially compact and tirst-countable, every sequence (xfa,n e INl on Xf has a cluster point xi on the space (6.4)k Applying the diagonal argument of 2.36, there exists a single subsequence of integers, (nk,k E l ), such that xink xi, for every i. Consider the subsequence xw,...). In the product topology, xy in X*, lau, k s N ) where xng @1n:, which proves that X* is sequentlally xo...) iff for i, every xink xi .x (.x1, compact. X= can be endowed with the metric px Z7wl4/zf,which induces the product topology. X= is separable by 6.16, and sequential compactness is equivalent to compactness by 6.8 and 6.7, as above. This proves sufficiency. Necessity follows fzom 6.9(ii), by continuity of the projections as before. w -..-9
=
=
--
--
=
6.18 Example The space R= (see5.14) is endowed with the Tychonoff topology if we take as the base sets of a point the collection .x
N(x.,k,
(y: la)
-yjl
=
< :,
=
1,...,k1., k e N, er > 0.
(6.20)
A point in R* is close to a7 in this topology if many of its coorclinates are close
Topology
to those of x; another point is closer if either more coordinates are within 6: of each other, or the same coordinates are closer than e, or both. The metric Jx defined in (5.15)induces the topology of (6.20).If (xn) is a sequence in X, dxlxnvx) 0 iff V z,k 3 N 1 such that xn e N(x;k,e) for a1l n k N. We already know that R= is separable under Jx (5.15),but now we can deduce this as a purely topological property, since R* inherits separability from R by 6.16. (:1 -->
The intinite cube (0,1j=shares the topology (6.20)with R= and is a compact space by 6.17; to show this we can assign the Euclidean metric to the factor spaces (0,1j. The trick of metrizing a space to establish a topological property is frequently useful, and is one we shall exploit again below.
6.6 Embedding and Metrization Let X be a topological space, and F a class of functions F-y tion p'lzw :: X/rfyis the mapping defined by
exjf
y. The evalua-
J: X F->
xt.
=
(6.21)
The class F may be quite general, but if it were finite we would think of d(x) as the vector whose elements are the J(x),J e F (6.21)could also be written z:yoe = f where ay is the coordinate projection. A minor complication arises because f need not be onto y, and :(X) c Swy is possible. If 4 is a set of points in ly -1(4) may contain points not in e(X). We therefore y, the inverse projection need to express the inverse image of A under f, in tenns of c. as .
:-1(ay-1(4)
f -144) (xyo:)-1(4) =
=
f...j c(x))
c
y.
(6.2:)
The importance of this concept stems from the fact that, under the right conditions, the evaluation map embeds X in the product space generayed by it. It would be holeomorphic to it in the case c(X) Skyy. =
6.19 Theorem Suppose the class F sparates points of X, meaning that fx) # J@) for some f G F whenever x an d y are distinctpoints of X. If X is endowed with the weak topology induced by F, the evaluation map defines an embedding of X into ly J .
.' ,'--f,,' -' ' :.);' .'...' .'-.' .,'
k' r' ::'i;t' lkd ;'y' j' f'.f..
tjd f'
Proof lt has to be shown that e is a 1-1 mapping from X onto a subset of Xro which is continuous with continuou: inverse. Since F separates points of X, e is 1-1, since c@) # :(.y) w he never flx) #(.f@). for some f e F. To show continuity of -1 X rwheneker A is open in y under the weak e, note first that f (A) is open in topology, and sets of the form ly*(A) i open in X/ywith the product Bltt e (z:y(A)) (ayoc) (A) topology, since the projecons are y yyj ; j lh, under e of sets of the form gs J- (A), so we can conclude that. thte '-' .. ).' .T.'1.). (qgt. .y. .g j(. ..t. ),q:.., yj.).yt) ). tyi,j:lt. . .. . y'yt,jgyy,kytt,jyyyryltysy invrfs unions and intersections t; ay-1(4), 4 c :y, are open. since #.-;y.;.L.L.y ( .r (..eryt yrve y (t. (.'( ). ty t . . se sets of Xfy,which are (see 1.2) the same property extrh.d, kygtk..qjlkyrltlqplttit,tt,i ltjj tjju illwl invfj tuj , product topology, and tjo ) r))t.()),qy:,. jjy y Ljjqj.jlqjtjlllq-kjLb. f-iite intersections of these .. . ..; p cztt' yyyt( j))j;)).jjjy)(4tr.)jk;)r...'...))' yyyjyy, y y ,y...'y#)tyv-y..y,' ,.;;.bitiL.) !!F:1. L.)iLj;;Lb'-j)'3';'bb'' 1t1!4,1(E:,. ,y'').%' ,)'..; .y'' ftt. ,:... jjj7td-jjj,1!2421,:'(E:,..ki:r t::' )8,<-..,s.. ..''.y)1I)k-jyjj)'. &.-,. llr.. 154, 421,: d:::: 1ki::1l; :1Er' .kE,: l.rlijjif..' :El2 #:#*'f,ty.-')j.A.yy .rt;...'.7...,tj;-rtr 1lr1. d:E:,h 11. 11. llrl. .'. (.).,j.y,tty. gllr)p %.. ( g .. . jyyy. 1l; 1 :55;4:2::4:::,,llr11l2417:1. tjjb1'3+. )'t*..111!111k!!1-.. k(!r7qp).... $44)14. 4::2ii.'.......11#)i)... #I!2l-y.. jiii 154) ll:ll.i.1Ir71L 1ki!: 4k12: 4:::,, 11.21. .... 1l: ql::)p );.r'#..'y)yy: . ...k' 1bd;i.lq- 4:y# . p.. L3ffL'. ;EE1g (1(2..-, j:''yt,.;Ayz,t., ii. 11.,1. #.:, ()iiL' - ii. . . ''. ; 'lri
Lkwise
qtjt
,$
'tihuoj.
,
,
y
q
yk
y
,
g
,,k;
.'.
:
..
...'...'j;.ji(...
:.i:t
,, '
.
..)
=
,
tyrtj., .;i
..
=
;
,ytl,ii1.
-.)..
.:.
E...
.
.
..
.
..
.,p,y,
..
..
.'
( jy)t,,.E.j
(II'E .E!' '.j2.t(). .:E. .. . :. ,,/'::t Eyyyhtttyyjt.ttjlyyyyyyyyjyyjrjyl,y.yyyt'jtb,t,,.,) ..
..
.
(..))t
.
.
,
,))
,yttytyytyy;y
,
,y(
.:/E.:,tytlr.-)()jtEj4jjkjp.. .Ei...E -jy ;;y.yjE(j..)y.....y''E,#.
''.-.';'.7ti1ir7.
'''1.j.
..
.
4:
..'
... . ..
,
jj riEjptl-l,
'-('
,-t.k.
....
yja
.ij'
.
-
.
..
...t..
.'....TiTi.tj;)))1t-? -yy.'':;)'r.y(..y)j-.,$y.,',.y,yr.;..,-yj:..,.-p.. .
-.7
-:
...:
-,
.
.
.
E : -:-
-:
. '..
.
-
.: !.
;.
-ki. . . . . . . . .. ..
-.
,.
-,.
E .?).!-
,. ..
. .
.. -
.
-
.
.
Mathematics
106
set of the fomf -1(A) where A is open in y. Since F defines the topology on x we know this set to be open, and the finite intersections of such sets form a base for X by assumption. Since e is 1-1 and e -1 a mapp ing, it will suffice to verify that their images under e are open in e(X). Noting that B is a set of the type shown in (6.22),eB) ay-1(A) r''hd(x), but since ly-1(A) is open, eB) is open in '(x) as required. . ,
=
The following is the (forus)mostimportant case of Urysohn
's
embedding theorem.
6.20 Theorem A second-countable F4-space (X,5) can be embedded in (0,1)*.(:I The proof requires
the sufticiency
part of the following lemma.
6.21 Lemma Let x e X, and 1et O X be any open set containing x. 1ff X is a regular space, there exists an open set U with e c 0. -t7
.x
Proof Let X be regular. If O is open nd x e 0, there exist disjoint open sets U OC CC CC 0. Since & by the disand C such that e U and c C, and hence CC closed, Cc is jo intness and we have V ? OC g O and sufficiency is proved. To necessity, is a closed set not containing x, and suppose x e U and Vc 0. prove OC where U and 'P are disjoint open sets. Hence X is regular. w c .x
,
-t7'
Since the space is T4 it is F? and Proof of 6.20 Let V be a countable base for hence reglar. For any x e X and B e V containing x, we have by 6.21 a U e such that G U c #, and also by detinition of a base 3 A e V with e A c U c V, and hence x e X'c V c B. (Xis the smallest closed set containing A, note.) Since V is countable, the collection of all such pairs, say .
'1
.z7
.z'
.4
=
((A,#): A e F, B e F;
X'c BI,
(6.23)
is countable, and so we can label its elements A,B4i (Aj,Sj), i = 1,2,... Every x e X lies in Aj fr some i e IN. Since the space is normal, we have by Urysohn's lemma a separating function : X F-+ (0,IJ for each element of .W,such that fixi) 1 and hB 0. For each a' e X and closed set C such that # C, choose (Aj,#Jsuch that x e Xic Bi c Cc, 0. These separating functions form a countable and then h.@) 1 and ficj class F, a subset of U(X). Since the space is F1, C can be a point (yl so that F separates points. And since the space is F3j and hence completely regular, is the weak topology induced by F, by 6.14. It follows by 6.19 that the evaluation map for F embeds X into (0,1)*.. =
=
=
.x
=
=
':
IJ= endowed with the metric px defined in (5.19)is a compact r0, metric space. lt follows that :(X), which is homeomorphic to X under the evalRecall that
uation mapping by F, is a totally bounded metric space. lt further follows that is metrizable, since, among other possibilities, it can be endowed with the and y of X is taken to be metric under which the distance between points have metrization theorem, 6.12. We Urysohn px(c@),c@)). therefore proved the The topology induced by this metric on g0,11= is the Tychonoff topology. A base for a point p e g0,1)*in this topology is provided by sets of the (p1,p2,...) (X,I)
.x
=
Topology
107
fonn Nlp-nkte.t
=
14./e l0,11*: I#/ -
qi I
i
< E,
=
(6.24)
1,...,/c),
for some finite k, and 0 < 6: < 1, which is the same as (6.20).The topology induced on X by the embedding is accordingly generated by the base sets Nx,.kte)
=
ly e
I @) fily)I <
X:
-
:,
i
=
1,...,/c1,
(6.25)
which can be recognized as finite intersections of the inverse images, under functions from F, of e-neighbourhoods of R; this is indeed the weak topology induced by F. This further serves to remind us of the close link between product topologies and weak topologies. Since metric spaces are F4, separable metric spaces can be embedded in (0,lj= by 6.20. In this case the motivation is not metrization, but usually compactitication that is, to show that separable spaces can be topologized as totally bounded spaces. Both metrization and compactification are techniques with important applications in the theory of weak convergence, which we study in Chapter 26. Although the following theorem is a straightforward corollary of 6.20, the result is of suftkient interest to deserve its own proof; the main interest is to see how in metric spaces there always exists a ready-made collection of functions to define the weak topology. -
A separable metric space (S,#) is homeomorphic to a subset of
6.22 Theorem
(0,11*.
Proof Let Jo J/(1 + #), which satisfies 0 f Jo f 1 and is equivalent to d, so that (X,A)is homeomorphic to (X,#). By separability there exists a countable set of points (zf,i e EN) which is dense in X. Let a countable family of functions be 1,2,..., and define an evaluation map : X F-, defined by fix) dx,zi), i =
=
=
x jj y g0,11
hx)
(%(mz1),J0(x,z2),...). where px(,#) that h is an embedding in (g0,1J=,px)
(6.26)
=
We show 7=1Ihk #kl llk. lf (x,,) is a squence in X converging to x, then for each k, dolxntz 4(x,zk)k Accordingly, V k,z H N 2 1 such that xn e Nxik,e) for a11n k N, poohxnlvhxjj --> 0, and h is continuous at x. On th other hand, if xn zz...hx, there exists : > 0 such that Y N k 1, djxntx) 2 6: for some n k N. Since (z:) is dense in X, there is and J0(mzk) < doxvz I 2 so that 1doxnnz a k for which doxnnz k 1: and hence =
-
--
Y
px((x,,),(x)) 2
/,
/2
1+1
-
(6.27)
k
Since this holds for some n k N for every N k 1, it holds for infinltely many n, and hxn) zz-y (x). We have therefore shown that hxn) (x) if and only (Jxn --).x. This is the property of a 1-1 contingous function with cntinuous inverse. . --
But note too the alternative approach of transforming th distance functions into ting functions as in (6t10),@nd applying6.20.
separa
11 PROBABILITY
7 Probability Spaces
Probability Measures A random experiment is an action or observation whose outcome is uncertain in advance of its occurrence. Tosses of a coin, spins of a roulette wheel, and observations of the price of a stock are familiar examples. A probability space, the triple (D,1,#), is to be thought of as a mathematical model of a random experiment. f is the sample space, the set of all the possible outcomes of the The collection 5 experiment, called the random elements, individually denoted of random events is a c-field of subsets of f1, the event A e 5 being said to have occurred if the outcome of the experiment is an element of A. A measure # is assigned to the elements of 5, PA) being the probability of A. Fonually, we have the following. .
7.1 Definition A probability measure (p.m.)on a measurable space (f1,1)is a set function #: 5 F--> (0,1j satisfying the axioms of probability (a) #(A) k 0, for a11:4 e T. 1. (b) #(D) (c) Countable additivity: for a disjoint collection tz4j e @,j e IN) =
,
#
Uz4yTPAj).
n
=
Under the frequentist intepretation of probability, #(A) is the limiting case of theproportion of a long run of repeated experimnts in which the outcome is in 4. Alternatively, probability may be viewed as a subjective notion with #(A) said to represent n observer's degree of belief that A will occur. For present purposes, the intrpretation given to the probabilities has no relevance. The theory stands or falls by its mathematical consisteny alone, although it is then up to us to decidewhether the results acord with ur intuition and are useful in the analysis of real-world problems. Additional properties of P follow i om the axioms.
7.2 Theorem lf A, #, and (, j e Eq) am afbitrary T-sets, then (i) #(X) f 1. #(AC) 1 #(A). (ii) t E 0. (iii) PZ) ytiitity). (iv) A c # = #(A) f #() tpij lty, yy, y Ir$r7.y7,#kk. yyE ');.(.y.-- ..ty .... ... ...;. .. 4(:(y.-t,d$p1k-2. .)-.(21,q.y.. .. 44:4)..j!!E111, L-i4qii.. jit' ..ty..-. (v)(A w B) = pA) + .ji!!:::,' y )j)t yq j y,, . . . yy ,y jg tr j, ju j yy, jyky y kjy (vi) PojAj) f L)PAj) (ctllitAb. tyt,tt j j,( y y rytyrjjtyry =
-
'
=
i
,
-t.(..-...y...ty-.t
(2)jI.
..))
.jj!.::;i'
-. .
..-...k.
....
..
-'
...tE.y..-
...
.
-
.
.
,jj
.
$jjy
,tjjyyytyyyylyyy,yyqyjyj.ytlyyttyyyyyy,ly
...),
.. .
..
y...
,y
,
y,y ,
..
112
Probability
(vii) Aj
'r'
t, A
A or Aj
PAj) .-A #(A)
(continuity).cl
Most of these are properties of measures in general. The complementation property (ii) is special to #, although an analogous condition holds for any finite measure, with #(X replacing 1 in the formula. (iii) confinns P is a measure, on the definition. Proof Applying 7.1(a), P(A)
+
(b), and (c), PAc)
PA t.JAc)
=
#(f)
=
1,
=
from which follow (i) and (ii), and also (iii) on setting A by 3.3, and (vii) by 3.4. w
=
(.
(iv)-(vi)follow
To create a prbability space, probabilities are assigned to a basic class of according C, events to a hypothesis about the mechanisms underlying the random outcome. For example, in coin or die tossing experiments we have the usual hypothesis of a fair coin or die, and hence of equally likely outcomes. Then, provided C is rich enough to be a determining class for the space, (f,@,#) exists by 3.8
(extension theorem) where ?F = c(C).
7.3 Example
Let Sp,1)
=
(# fn g0,1), #
e f), where T is the Borel field on R.
Then ((0,1), fm,1J, m), where m is Lebesgue measure, is a probability space, 1. The random elements of this space are real numbers between 0 since ,7:(:0,1)) and 1, an a drawing from the distribution is called a random variable. It is said to be distributed rfnt/r/zl/y on the unit interval. The inclusion or exclusion of the endpoints is optional, remembering that /1((0, 1J) r?2440,1)) = 1. n =
=
The atoms of a p.m. are the outcomes (singletonsets of ) that have positive probability. The following is true for tinite measures generally but has special importance in the theory of distributions.
7.4 Theorem The atoms of a p.m. are at most countable. Proof Let l be an atom satisfying P((1 J) #(( J)for al1 (0 6 f1, 1et (t'/2 satisfy #(f tl)2)) k #4(1) for all e f(t)1 ), and so forth, to generate a sequence with -
#(l1l)
2 #(tt%l)
2 #(t2l)
...
(7.3)
The partial sums Z7=1#(fj)) form a monotone sequence which cannot exceed #(D) 1, and therefore converges by 2.11, implying by 2.25 that limn-hxftf.t)a 1) 0. All points with positive probability are therefore in the countable set f(, i e Nl . =
=
.
Suppose a random experimentrepresented by the space (fk9,#) is modified so as to confine the possible outcomes to a subset of the sample space, say A c f1. For example, suppose we switch from playing roulette with a wheel having a zero slot to one without. The restricted probability space is derived as follows. Let 5% denote the collection fEcj A, E e @) 5%is a c-tield (compare1.23) called 5h and #(m/#(A) for E e Fa, Ph can be is called the trace of % on A. Defining #a() (A,Fa,#A) is called the trace of (f1,F,#) on A. triple verified to be a p.m. .
=
'lhe
Probability Spaces This is similar to the restriction of a measure space to a subspace, except that the measure is renormalized so that it remains a p.m. In everyday language, we are inclined to say that events may be or certain' lf such events are none the less elements of @, and hence technically by random, we convey the idea that they will occur or not occur with here assigning them probabilities of zero or one. The usage of the tenn marks is deliberately loose, as the quotation suggest. To say an event cannot it has probability different from saying it cannot occur because is zero occur because the outcomes it contains are not elements of f1. Similarly, to say an event has probability 1 is different from saying it is the event f1. In technical discussion we therefore make the nice distinction between sure, which means the latter, and almost sure, which means the former. An event E is said to occur almostsurely fl E has prob(a.s.), or equivalently, with probability one (w.p.1) if M abilitymeasurezero.This terminologyis synonymouswithlfv/l/eveqwercta.e.) in the measure-theoretic context. When there is ambiguity about the p.m. being considered, the notation may be uded. Aimpossible'
.
tcertainty' tcertain'
=
-
4a.s.(#j'
7.2 Conditional Probability A central issue of probability is the treatment of relationships. When random experiments generate a multi-dimensioned outcome (e.g.a poker deal generates several different hands) questions always arise about relationships between the different aspects of the experiment. The natural way to pose such questions is:
0. #(. IA) satisfies the event B as #(#IA) probability axioms as long as P does and #(A) > 0. In particular, #(A IA) 1, and P(Bc IA) 1 PB lA), since B /-74 and Bc DA are disjoint, and their union is 4. The space (A,@x,fk), the trace of the set A on (D,T,#), models the random experiment from the point of view of an observer who knows that (l) e A. Events A and # are said to be dependent when PB IA) # #(#). In certain respects the conditioning concept seems a little improper. A context in which the components of the random outcome are revealed sequentially to an observer might appear relevant only to a subjective interpretation of prpbability, and lead a sceptical reader to call the neutrality of the mathematical theory ipto question. We might also protest that a random event is random, and hak no business Gthe
=
=
=
-
Probability
114
defining a probability space. ln practice, the applications of conditional probability in limit theory are usually quite remote from any considerations of subjectivity, but there is a serious point here, which is the difficulty of constructing a rigorous theory once we depart from the restricted goal of predicting random outcomes a priori. The way we can overcome improprieties of this kind, and obtain a much more powerful theol'y into the bargain, is to condition on a class of events, a c-subfield of 9. Given an event B e F, let the set function F-> (0,11 /74#1 ): '
represent the contingent probability to be assigned to B after drawing an event A from N, where V i @.We can think of V'as an infonuation set in the sense that, for each A G V, an observer knows whether or not the outcome is in A. Since the elements of the domain are random events, we must think of #(# lN) as itself a random outcome (arandom variable, in the terminology of Chapter 8) derived from the restricted probability space (f1,V,#). We may think of this space as a model of the action of an observer possessing information V, who assigns the conditional probability #(# lA) to B when be observes the occurrence of A, viewed from the standpoint of another observer who has no prior infonnation. N is a c-tield, because if we know an outcome is in A we also know it is not inAc, and if we know whether or not it is in Aj for each j 1,2,3,..., we know whether or not it is in U#.j. The more sets there are in V the larger the volume of information, all the way from the trivial set 5 (f,0) (completeignorance, with 8#1 T) #(#) a.s.) to the set 5 itself, which orresponds to almost sure knowledge of the outcome. In the latter case, P@ 11) 1 a.s. if ( e B, and 0 otherwise. lf you know whether or not (J) G A for every A e @, you eftctively know (J). 8 =
=
=
=
7.3 Independence A pair of events A, B e 5 is said to be independent if #(A ch#) equivalently, if PB jA)
=
#(#).
=
#(A)#(#), or,
(7.4)
lf, in a collection of events C, (7.4)holds for every pair of distinct sets A and B from the cotlection, C is said to be pairwise indepenttent. In addition, C is said to be totally independent if (.D-,A)
#
-
J,'tA'
(7.5)
containing two or more events. This is a stronger condition for every subset # i thanpairwise independence. Suppose Cconsists of setsA, #, and C. Knowing thatS has occurred may not influence the probability we attach to A, and similarly for C) but thejoint occunrnce of B and C may none the less imply something about A. Pairwiseindependence implies thatfA fa B) =PIAIPCBI,PCA rn C) =#(A)#(C), and PB rn C' PB)PC4, but total independence would also require P(A f'n B rn C) = =
P(A)PB)PC).
Probability Spaces
115
Here are two useful facts about independent events. In each theorem let C be a totally independent collection, satisfying ptf)xs .gA) l7ze.y#(A) for each subset =
#' I C. 7.5 Theorem The collection C' which contains independent.
yt
and Ac for each A e C is totally
Proof It is sufficient to prove that the independence of A and B implies that of Ac and B, for B can denote any arbitrary intersection of sets from the collection AC. This is certainly true, since if and (7.5) will be satisfied, for either A or #(A)#(#), then #(A f''h B) =
PCAC
r'A #)
=
PAc ra #)
= #(#)
-
+
P(A fa #)
P(A4PB4
-
#(A)#(#)
PAPB4.
=
(7.6)
w
7.6 Theorem Let () be a countable disjoint collection, and let t e collections consisting of Bj and the sets of C be totally independent for each-j. Then, if B LjBj, the collection consisting of B and C is als independent.
=
Proof Let #' be any subset of C. Using the disjointness of the sr-ts of B, and
countable additivity, P
j
rn
)
xOs ,A
-
P
sA)
zOs ) (#?
(Uy Bj ra ,A xOs XyP -
=Xy
PCB?P
7.4 Product Spaces
sA)
tj
=
Plhj
ra
,PA4.
.
and independence arise when multiple random eyperiments run in parallel, and product spaces play a natural role in the analysis of these issues. Let (f x E, 5 @ V, #) be a probability space where 5 (*)V is the
Questionsof dependence
c-fieldgenerated by the measurable rectangles of (-1x E, and PL x E) 1. The randomoutcome is a pair (,4). This is no more than a case of the general theory of j7.1 (wherethe nature of (l) is unspecified) except that it becomes possible to askquestions about the part of the outcome represented by ) or ( alone. #04/7 =
=
#(Fx E) for F e F, and #E(G) Pl x G4 for G e N, are called the marginal probabilities.(1,1,#0) and (E,T,#E) are probability spaces representing an incompletelyobserved random experiment, with ) or k, respectively, being the only things observed in an experiment generating (,(). On the other hand, suppose we observe ( and subsequently consider the experiment'of observing (t). Knowingtmeans thatforeachf x Gweknowwhetherornot (,() is in it. The conditional probabilities generated by tlzis two-stage experimentcan be wr itten by a slight abuse of notation as #(Fl V), although strictly Fx E, and the elemts of the speakingthe re 1evan t events are the cylinders uzrln enHzw+lxlv.lllv,qia 'n uzn n'zala' rmfiold nre 0./ G for G G W. Cnn ditinnine =
Probability
116
#(Fx E lfl x T). ln this context, product measure assumes a special role as the model of independence. In (f x E, 5 (& V, #), the coordinate spaces are said to be independent when PFx
G)
=
#0(#)#s(G)
for each F e 5 and G e T.
Unity of the notation is preserved since Fx G (Fx E) fa (( x G). We can also write #(Fx EIf x G) /704/-), or with a further slight abuse of notation #(FIG) #n(/'), for any pair Fq 5 and G e N. Independence means that knowing ( does not affect the probabilities assigned to sets of ;. Since the measurable rectangles are a determining class for the space, the p.m. P is entirely detennined by the =
=
marginal measures.
=
8 Random Variables
8. 1 Measures on the Line a probability space. A real random variable (r.v.) is an 5lBf'unction X: ( F.-: R 9 That is to say, aY4)) induces an inverse mapping measurable from B to 5 such that X -1(#) l 5 for every B e B where B is the linear Borel field. The term T-measurable r.v.' may be used when the role of B is understood. The symbol g. will be generally used to denote a p.m. on the line, reserving P for the p.m. on the underlying space. Random variables therefore live in the space (R,S,g,), where g. is the derived measure such that g(#) PX -1(#)) = #( X e B) The term distribution is synonymous with measure in this context. The properties of nv.s are special cases of the results in Chapter 3; in particular, the contents of j3.6 should be reviewed = in conjunction with this chapter. If g: R F-> R is-1a Borel function, then gox is also a r.v., having derived p.m. g,: according to 3.21. p(A)) If there is a set S e B having the property g(& 1, the trace of (R,S,g) on S is equivalent to the original space in the sense that the same measure is assigned to B and to B r''hS, for each # e S. Which space to work with is basically a matter of technical convenience. If X is a r.v., it may be more satisfactory to say that the Borel function X 2 is a r v. distributed on R+ than that it is distributed on R but takes values in R+ almost surely. One could substitute for (M,S,g,) the extended space (F,S,g) (see 1.22), but note that assigning a positive probability to infinity does not lead to meaningf'ul results. Random variables must be finite with probability 1. Thus (R,f,g), the trace of (F,S,g) on R, is equivalent to it for nearly a1l purposes. However, while it is always finite a.s., a r.v. is not necessarily bounded a.s.; there may exist no constant # such that 1X(l I f B for a1l (l) e C, with #(D C4 0. The essential supremum of X is
Let
(f1,1,#) be
.
,
=
.
=
,
.
=
-
ess sup X
=
inf (x: #41XI > x)
and this may be either a finite number, or
0),
=
+x.
8.2 Distribution Functions The cumulative distribution
@.b.f.)Of
f'Ifrlcft??z
X is the function F:
where
2- F-> g0,1J,
< xj, x s F. (8.2) l ( ( fty/, j'q..(( gn the values 0 and 1 to )( jj:t.:.' tk..j.,.((y. we take the domain to be R-since it' iq.(... (('..t'jr(t.t:.....t( .. .. . y)':)j)t..: .i-) ... (j).jL?, f;. ji,j.. . F(x)
=
g((-x, x1)
x
=
.
.
..
.
.
:
'
:
:.)'
.-
..'.''
....
.
..)
.... .
.
'
'
.t.
.
.:$j.i.
(Ej
.
.
....j
... y.. q .))(-.3E,l'( y j.y)y(y)jgE.t..-'(.:,.t.yy.. .t
...yr.j(,?:jyj.jy.
,,;.j..j;...(..j.
..
.
-(.y.'.,
.
. .
..
.
.
.
.
c
..
.
r)().... .. .. . y.....) ,.E.
. .
.
Probability
118
F(-x) and F(+x) respectively. No other values are possible so there is no contradiction in contining attention to just the points of R. To specify a distribution for X it is sufticient to assign a functional form for F; p, and F are equivalent representations of the distribution, each useful for different purposes. To represent g,(A) in terms of F for a set A much more complicated than an interval would be cumbersome, but on the other hand, the graph of Fis an appealing way to display the characteristics of the distribution. To see how probabilities are assigned to sets using F, start with the half-open interval (x,y) for x < y. This is the intersection of the half-lines (x,y) and (-x, (.x,+x).Let A (-=, x) and B = (-x,y), so that g(A) F@) and g(#) F(y); then -xzf
=
=
BIAC
g,(@,)/1) =
=
ra #)
= 1 (g(A) -
1 p,tA
=
+
-
1 g(#))
tJ#C)
g(#)
=
-
=
-
g(z4)
=
F@) F@), -
A and Bc being disjoint. The half-open intervals form a semi-ring (see1.18), and from the results of j3.2 the measure extends uniquely to the sets of f. a) for x c R As an example of the extension, we detennine g,((x)) PX will (compare 3.15). Putting a7 y in (8.3) not yield this result, since A cjAc = 0, not (xl We could obtain fxl as the intersection of (-x, x) and (-x, xlc, but then there is no obvious way to tind the probability for the open interval (-x, x) (-x, xl f.xl The solution to the problem is to consider the monotone sequence of half-lines (-x, x 1/n) for n e N. Since @ 1/p, (-x, aj, 1/n), according have 1/n,<) F(x) lln Fx ra g,(@ to we (-x, x = and where 1/n, x), Fx) F@-), Since T g,((x)) (8.3). (a7Je fx) 0':nltx F@-) is the left limit of F at x. Fx) exceeds Fx-j (i.e. F jumps) at the atoms of the distribution, points with g,((xJ)> 0. We can deduce by the same kind of F(y-4 Fx), g((x,y)) = F@-) F(x-), and that, reasoning that p,(@,y)) generally, measures of open intervals are the same as those of closed intervals unless the endpoints are atoms of the distribution. Certain characteristics imposed on the c.d.f. by its definition in tenns of a measure were implicit in the above conclusions. The next three theorems establish these properties. =
=
=
=
rx,+x)
.
=
-
.
.:71
-
=
-
=
=
-
-
-
-
-
-
.z:
=
-
-
1, 8.1 Theorem F is non-negative and non-decreasing, with F(-x) 0 and F(+x) and is increasing at e R iff every open neighbourhood of x has positive measure. =
=
.x
Proof These are all direct consequences of the definition. Non-negativity is from (8r2), and monotonicity from 7.2(iv). F is increasing at x if Fx + :) > F@ E) for each E > 0. To show the asserted sufficiency, we have for each such E, -
F@ + :)
-
F@ :) k F((x
+ 8)-)
-
For the necessity, suppose g,(5'(x,E))
g,(5'(x,e))= F(@ + e,)-) F@ -
-
=
E)
-
F(x
-
6,)
=
g,(5'@,y)).
(8.4)
0 and note that, by monotonicity of F, k Fx
+ :/2)
-
F@- :/2).
w
The collection of points on which F increases is known as the sunnort of
(8.5) 11
IfA
Random
Variables
119
complement in R, the largest set of zero measure, consists of points that must a11 liein open neighbourhoods of zero measure, and hencemustbe open.-rhe supportof g, is accordingly a closed set. F is right-continuous
8.2 Theorem Proof For x
l
everywhere.
R and n 2 1, additivity of the p.m. implies
g((-x,
.v
+
1/nJ) g,((-x, =
x1)
+
g,(@,x+ 1/n1).
(8.6)
g,(4-x, x+ 1/aJ)1, g.((-x, x1) by continuity of the measure, and hence limn-yxjtttx, + 1/n1) 0. lt follows that for er > 0 there exists Ne such that g,(@,x + 1/nJ) < E, and, accordingly, As n
.-.+
x,
.v
P,((-=,
=
.Y1)
f t((-=,
for n k Ne. Hence F(x+)
=
x+ 1/n1)
<
p,tt-x, x1) +
(8.7)
E,
F(x), proving the theorem since x was arbitrary.
.
If F@) had been defined as g,((-x, x)), similar arguments would show that it was left ontinuous in that case.
'W
Fig. 8.1
8.3 Theorem F has the decomposition F(x)
whereF'@) jumps, and
=
F'x)
+
F'x)
(8.8)
is a right-continuous step function with at most a countable number of r'(x) is everywhere continuous.
Proof By 7.4, the jump points of F are at most countable. Letting
denote these
points,
f-x1,x2, ...)
Probability
120 F'@)
=
X xisx
Fxi,
Fxi-j)
-
(8.9)
is a step function with jumps at the points xi, and F'x)
F(xi-)
=
Flxi) at each xi and is continuous everywhere.
Fx)
=
-
F'x)
has
*
Fig. 8. 1 illustrates the decomposition. This is not the only decomposition of F. The Lebesgue decomposition of g with respect to Lebesgue measure on R (see4.28) is g = p,1+ g.2where g,l is singular with respect to m (is positive only on a set of Lebesgue measure 0) and g2 is absolutely continuous with respect to Lebesgue measure. Recall that g2(A) = fxjdxo A e B where Tis the associated Radon-Nikodym derivative (density function). If we decompose F in the same way, such that Ff(x) w((-x,x1) for i = JQ.f(()J), implying that f@) dFz /#( I(=x.This = 1 and 2, we may write Flx) must hold for almost all x (Lebesgue measure), and we call F2 an absolutely continuous f'unction,meaning it is differentiable almost everywhere on its domain. F' S F1 since F1 may increase on a set of Lebesgue measure 0, and such sets can be uncountable, and hence larger than the set of atoms. It is customary to summarize these relations by decomposing F' into two additive components, the absolutely continuous part &, and a component F3 = F'- F2 which is continuous and also singular, constantexcept on a setof zero Lebesgue measure. This componentcan in most cases be neglected. The collection of half-lines with rational endpoints generates B (1.21) and should be a determining class for measures on @,f). The following theorem establishes the fact that a c.d.f. defined on a dense subset of R is a unique representation of g,. ,
=
=
8.4 Theorem Let z be a tinite measure on (R,f) and D a dense subset of R. The function G defined by 81)
Gx)
=
IX(-=,-Y1),
e D
.<
=
x e R D
F(x+),
-
(8.10)
is identical with F. Proof By definition, R q X'and the points of R D are a11closure points of D. For each x e R, not excluding points in R D, there is a sequence of points in D converging to x (e.g. choose a point from Sx,lln) r'AD for n g N). Since F is F(x+) for each x e R -D. w right-continuous everywhere on R, p,((-x,x)) -
-
=
Finally, we show that every F corresponds
to some g,, as well as every g to an F.
8.5 Theorem Let F: V --A E0,1) be a non-negative, non-decreasing, right=0 and F(+x) 1. There exists a uniquep.m. g on continuous function, with F(-x) (R,S) sqch that Fx) g((-x,x1) for a11a7 G R. u =
=
Right continuity, as noted above, correspopds to the convention of defining F by (8.2). If instead we defined F@) = g((-x,x)), a left-continuous non-decreasing F
Random Variables
121
would represent a p.m.
#: (0,11F- 2', defined
Proof Consider the function (4If)
=
inf (x: u < F(x) )
by
(8. 11)
.
4)can be thought of as the inverse of F; (40) (41) +x, and since F is nonIight-continuous, decreasing and # is non-decreasing and left-continuous; # is Borel-measurable by According to 3.21, we may define a meas3.32(ii). therefore ?n4 -1 (#) for each B e f, where m is Lebesgue measure on theBorel ure on (R,f) by of sets (0,11. ln particular, consider the class C of the half-open intervals (J,:1 for a11a,b e R with a < b. This is a semi-ring by 1.18, and c(C) B by 1.21. Note that t)-1((J, 4) (/g:inf (x: u K F(x)) e (tz,h1 ) (F((7),Fblj. (8.12) =
-x,
=
=
=
=
For each of these sets define the measure P,((J,#1)
=
/19
-1((J, :j))
=
Fb)
-
(8.13)
Fa).
The fact that this is a measure follows from the argument of the preceding paragraph. C is a determining class for (R,O), and the measure has an extension by 3.8. It is a p.m. since g.(R) 1, and is unique by 3.13. x =
The neat construction used in this proof has othef applications in the theory of random variables, and will reappear in more elaborate form in 922.2.The g'raph of () is found by rotating and reflecting the graph of F, sketched in Fig. 8.2; to see the fonner with the usual coordinates, turn the page on its side and view in a
mirror.
Fig. 8.2 If F has a discontinuity at x, then 4) x on the interval (F@-), F(x)1, and -1(lxl) = (F(x-), F(x)j. Thus, g,((x)) -((F(x-), F@)1) F(x) F(x-), as 41 required.On the other hand, if an interval (J,!4 has measure 0 under F, F is =
=
=
-
Probability
122
F(b) = c (say).4) constant on this interval and # has a discontinuity at Fa) -1 takes the value a at this point, by left continuity. Note that () (c) = (J,>1, so that g,((J,:J) mc) = 0, as require. =
=
8.3 Examples Most of the distributions met with in practice are either discrete or continuous. A discrete distribution assigns zero probability to a11 but a countable set of points, with F' = 0 in te decomposition of 8.3.
8.6 Example The Bemoulli (orbinal'y) nv. takes values 1 and 0 with fixed probaThink of it as a mapping from any probability space containbilities p and 1 elements. and tFailure'
isuccess'
;
1
8.7 Example The binomial distribution with parameters n and p (denotedBn,pl) is the disibution of the number of ls obtained in n independent drawings from the Bernoulli distribution, having the probability function P(X
=
x)
4
=
x
p
x
(1
n-x
-p)
x
,
=
8.8 Example The limiting case of (8.14) with p distribution, having probability function P(X
=
x)
-w
cp
1
0,...,p.
hln. as
=
(8.14)
Ia
n
-..-y
x,
is the Poisson
-h,
.
X'.
x ,
x
=
() j ,
,
z
(8.15)
,...
This is a discrete distribution with a countably infinite set of outcomes.
D
In a continuous distribution, F is absolutely continuous with FI = 0 in the Lebesgue decomposition of the c.d.f. The derivative f #F/#x exists a.e.(>1 on R and is called the probability densit.v function(p.d.f.)of th p.m. According to the Radon-Nikodym theorem, the p.d.f. has the property that for each E e f, =
g,(f)
=
jyflvdx.
,
(S' 16)
8.9 Example For the unkf'orm distribution on g0,11 (see 7.3), 0, x F@)
<
x, 0 f
=
1, x
>
0 .x
S 1.
(8.17)
1
The p.d.f is constant at 1 on the interval, but is undefined at 0 and 1.
E1
8.10 Example The standard normal or Gaussian distribution has p.d.f. f(x)
=
1
/-2a
-x2/2
e
-x
'
<
x
< +.=
,
(8.18)
Random Variables
whose graph is the well-known
123
bell-shaped curve with mode at zero. n
8.11 Example The Cauchy distribution has p.d.f.
fxq
1
=
<
-cx,
,
+x2)
zr(1
which, like the Gaussian, is symmetric
x
(8.19)
< +x,
with mode at 0. n
When it exists, the p.d.f. is the usual means of characterizing a distribution. A particularly useful trick is to be able to derive the distribution of gzk' from that of X, when g is a function of a suitable type.
8.12 Theorem Let #: S F--> 1- be a 1-1 function onto 7, where S and 1- are open subsets of R, and let h g -1 be con tinuously differentiable with dhldy # 0 for all y e 7. lf X is continuously distributed with p.d.f. fx, and F #(.X), then F is continuously distributed with p.d.f. =
=
),I
.fx(,-1(,))
s(,) -
.
(8.20)
u
The proof is an easy exercise in differential calculus. This result illustl-ates 3.21, but in most other cases it is a great deal harder than this to derive a closed form for a transformed distribution.
8.13 Example Generalize the uniform distribution (8.9)from (0,1) to an arbitrary interval gtz,:1.The transformation is linear, y so that fyy)
=
b a) -1 on
a
=
(J,:),
-
F@)
b
+
by
(.y
=
-
(8.21)
Jlx,
-
(8.20).The
allb
c.d.f. is d e jjxd
on
a).
-
gc,)j by (8.22)
Membership of the uniform family is denoted by X
-
&(c,:1. n
8.14 Exapple Linear transformations of the standard Gaussian r.v., X
=
jt + cZ,
c
>
0,
(8.23)
generate the Gaussian family of distributions, with p.d.f.s
1 e fx'' g.,0-) /--2ac
-tx-jtlz/acz
'-=
=
,
<
.7
< +=
.
(:I
(8.24)
have better-known designatiops as The location parameter ytand scaleparameter and 9.7 9.4 below. distribution; Membership of the Gaussian of see moments the 2 family is denoted by X #(g,,c ). -
A family of Cauchy distributions is generated from the standard Cauchy by linear transformations X v + Z, > 0. ne family of p.d.f.s with locationparameter v and scale parnmeter take the form
8.15 Example
=
Probability
k=N
J@;v,)
1
1
=
k''g1+
r(x -
<
-=
,
v)/) z
< +x.
.z7
(8.25)
n
8.16 Example Consider the square of a standard Gaussian nv. wit p.= 0 and c = 1. Since the transfonnation is not monotone we cannot use 8.12 to detennine the density, density, strictly speaking. But consider the Khalf-normal'
zlt&l =
2J(ff), u
k 0
0,
<
u
(8.26)
0
where f is given by (8.18).This is the p.d.f. of the absolute value of a Gaussian variable. The transformation g Iu I) = u 2 is 1-1 so the p.d.f. of Z2 is ,
1 e fz24&) W2V
-1/2
-un
u
=
applying(8.20).This is the ch-squared or z241).It is a member (withp (x =
G(?z; ',pj W
herej-+4
=
f!
() < u
,
=
and 1R?zl (n llF(n
-
(8.27)
,
-1 -ea
((xI/)P
vpe
-
.
distribution with one degree of freedom, ) of the gnmma family,
,
0
<
u
< x,
is the gammafunction, J7(P-1e-Q(
=
=
<
.
a
>
0, p
>
0,
(8.28)
having the properties F()
=
W2'V,
1). c!
8.4 Multivariate Distributions In Euclidean k-space R k the k-dimensional Borel field Tk is c(S$ where Rk denotes the measurable rectangles of Rk th e se ts of the form #1 x B1 x x Bk where Sf e B for f = 1,...,k. In a space (D,@,#), a random vector (msXi,...,.V)' = X is a measurable mapping X1 fl Rk ,
,
,
...
.-+
.
lf t is the derived measure such that g,(A) = PE) for A G T k an d E s 5 the mu ltivariate c.d.f., F: Fk (0,1), is defined for (x1,...,x' = x by ,
..-+
F@)
=
g,((-=,x11 x
...
x (-x, aJ).
(8.29)
The extension proceeds much like the scalar case.
8.17 Example Consider the random pair (X,D.Let F(x,y) = g((-x, XJx (-x, y1). The measure of the half-open rectangle (x,x+ Ax1x (z,y +Ay) is M7x, y)
=
F@ + Ax, y + Ay)
-
Fx
+
Ax,y)
-
F@, y + Ay) + Fx, y4.
(8.30)
To show this, consider the four disjoint sets of R 2 illustrated in Fig. 8.3:
Random Variables
125
:4
=
@,x + AXJx @,y + AyJ,
B
=
(-x, x) x (.y,y +Ay),
C
=
(x,x+AXJx (-x, yq,
D
=
(-x,x) x (-x,yj.
A is the set whose probability is sought. Since #(A wB tp Ck.)D) Fx + Ax,y), and #(D) Fx+ Ay + Ay), #(# QJ D) = Fxby + Ay), PCCwD) result probability is immediate from axioms. the Fx, y), the n =
= =
Fig. 8.3 Extending the approach of 8.17 inductively, the measure of the k-dimensional , xi + AQ can be shown to be rectang le Xil@f, AF(I1,...,
A'k)
=
77(:f..6/),
(8.31)
j
where the sum on the right has 2k terms, an d the Fj are the values of F at each of the verices of the k-dimensional rectangle extending from @1,....q)'with sides 1,...,k. The sign pattern depends on k', if k is odd, the Fj of length Mj, i having as arguments even numbers of upper vertices (pointsof the form xi + A'&)) take negative signs, and the others positive; wilile if k is even, the Fj with odd numbers of upper vertices as arguments are negative. Generalizing the monotonicbe ity of the univmiate c.d.f., F must satisfy the condition that AF@1,..., (R)+ non-negative for every chice of (x1,..., ' e R an d (A.x1,..., Axk)' s Applying 3.19 inductively shows that the class of half-open rectangles is a semi-ring, so that the measure defined by F extends to the sets of T k. hence (R Bk is a probability space derived from L1,5,P). If the distribution is continuous with p.d.f. flx), Fubini's iheoremgives =
.q)
.q)
.
-dimensional
,
,g)
,
Fx)
.f((1,...,(z)t:61
.dlk
=
..
(-x,xl1x,.,x(-x,xk)
=
xl
xk ...
J(k1
.,4:2)t2)1
...#(k.
,..
Theorem 8.12 has the following generalization. A dteomorphism
(8.32)
(also,coor-
Probability
126
dinate transformationj is a function g: S F- -T(S and 7 open subsets of Rk) w lli c h # 0 for all x e S, is 1-1 onto and continuously differentiable with etglx') whose element glx' Jacobian matrix is is the gllxi for i,j where (f,.j)th 1,...,1. The inverse of a diffeomorphism is also continuously differentiable. =
8.18 Theorem If F
where g is a diffeomorphism,
g
=
the p.d.f. of F is
h%l flg-1(.:))1yj
(g ag)
=
whereJ
detl:
=
-1
)j
y
'j
.
u
.
This is a standard result in the theory of multiple Lebesgue integrals Apostol 1974: 15.10-15.12).
8.19 Example Letting
denote the standard Gaussian p.d.f.
.f
k
lzi) 1-1
9(z) =
(zzo-Wexpf
=
(see8.10),
(see e.g. consider
-z1(z'z)).
(8.34)
i=1
This is a k-dimensional p.d.f., and the corresponding random vector (z1,...,Zk)', is the standard Gaussian vector. The affine transformation X
AZ+ !z,
=
Z
=
(8.35)
whereA (kx k nonsingular) and M kx 1) are constants, is 1-1 continuous with inversez A -1(X p.) having J IA-1 l 1/lA I Detine E 4A' such that (A -1) x-1 and j j4 j j jEI the positive square root being ( nderstood.Applying 8.18 produces =
=
ix,
,
=
,
-
jx I
lnl-m 14-1I =
(21)-1/2 jx j
,
1
9(A -1@ p.))
j
=
.
-1/2
-1
=
=
=
=
-
,4:4..)-1
.:4.-1
j
expf
-12
-yx
-
Jz)'(A-l)'A-1(x
-
Izlj.
.s;,x-1(.x
t
exp
-j(x .s)
j.
(8.36)
This is the multinormal p.d.f., depending on parametets p. and X. Every such distribution is generated by an affine transform applied to Z. Membership of the multinormal family is denoted X N#,X). n -
8.5 lndependent Random Variables Suppose that, out of a pair of r.v.s (.X,1')on @2T1 we are interested only in situation of interest X. In this the events are the cylinder sets in predicting R 2 having the form B xR, B e S. The marginal distribution of X is detined by (R,S,g,x) where ,g),
,
,
p,x(X)
=
t(X X R)
for W e T. The associated marginal c.d.f. is Fx
(8.37)
F@? +x). The notion of independence defined in 97.4specializes in the following way.xand =
Rattdom Variables F are called independent r.v.s iff pt(XX #)
>x(A)g.l'(S)
=
(8.38)
for al1 pairs of events A,B e f, where g.x is detined by (8.37)and g.y is analogous. Equivalently, g, is the product measure generated by g,x and yty. 8.20 Theorem X and J' are independent iff for each x,y Fxnyq
G
R
(8.39)
Fx@)Fy(X.
=
lf the distribution is continuous the p.d.f. factorizes as x,L
fxxjhb.
=
(8.40)
Proof Obviously, (8.39)is true only if g, satisties (8.38).The problem is to show that the former condition is also suftkient. Consider the half-open rectangles,
C
=
((tz,:)x (c,#1,a.c
e
R-,b,d
e R ).
lf ahd only if (8.39)holds, p,((J,/8
X (c,#1)=
= =
Fb,dl
F(:,c)
-
Fxbq
-
-
Fa,dl +Fa,c4
Fxabttdj
-
Fv)
(8.41)
g,x((J,:1)p,i'((c,tf1),
where the first equality is by 8.17 C is a determining class for (R2 S2) and g is detined by the extension of the measure satisfying (8.41)for ordinary rectangles or, equivalently, satisfying (8.39).The Extension Theorem (uniqueness partl shows that this is identical with the product measure satisfying (8.38).The extension to p.d.f.s follows directly from the detinition. . ,
.
,
With more than two variables there are alternative independence concepts k (compare j7.3). Variables m,...,Xk distributed on the space (R- pk are sa itj to be totally independent if ,g.)
,
1
XAf
p,f=1
k
=
l-lp,xJAd)
(8.42)
f=1
for all k-tuples of events A1,...,Akl 0. By contrast, pairwise independence can hold between each pair Xi&. without implying total independence of the set. Another way to think of total independence is in terms of a partitioning of a vector X = X3....,X' into subvectors .Y1 (jx 1) and Xz k x 1) for 0 < j < k. Under total independence, the measure of X is always expressible as the product measure of the two subvectors, under a11orderings and partitionings of the e lements. -j4
9 Expectations
9. 1 Averages and lntegrals or mean, of a r,v. X()l in a probability space
When it exists, the expectation, L1,5,P) is the integral EX4
Joxtovptl'
=
(9.1)
F(m measures the central tendency of the disibution of X. It is sometimes identified with the limiting value of the sample average of realized values xt drawn in n identical random experiments, xn
=
1
n
(9.2)
.t,
-
>1
as n becomes large. However, the validity of this hypothesis depends on the method of repeating the experiment. See Part IV for the details, but suffice it to say at this point that the equivalence certainly holds if F(A') exists and the random experiments are independent of one another. The connection is most evident for simple random variables. lf X = jxjjej where the ()) are a partition of f1, then by 4.4, osllj). /?(r) !F7 j cr
When the probabilities are interpreted as relative frequencies of the events Ej (: X'4(0) z)) in a large number of drawings from the disibution, (9.2) wlth large n should approximate (9.3). The values xj will appear in the sum in a proportion roughly equal to their probability of occurrence. F(m has a dual characterization, as an abstract integral on the parent probability space and as a Lebcsgue-stieltjes integral on the line, under the derived distribution. lt is equally conrct to write either (9.1) or =
=
+x
EX4
=
xdFxx).
(9.4)
Which of these representations is adopted is mainly a matter of convenience. lf 1x() is the indicator function of a set A G T, then
'(1,4m =
jgxldlk =
Jx(x).x#F@),
(9.5)
Expectations
129
where X(A) e B is the image of A under X. Here the abstract integral is obviously the more direct and simple representation, but by the same token, the Stieltjes fonn is the natural way to represent integration over a set in B.
If the distribution is discrete, X is a simple function and the formula in applies directly. Under the derived distribution,
(9.3)
'k-xj
F(m
where xj, j
=
(9.6)
p,(ta)l),
=
1,2,..., are the atoms of the distribution.
9.1 Example If X is a Bernoulli variable (8.6),EX) = 1.# + 0.(1
-#)
=
p. n
9.2 Example If X is Poisson (8.8), F(m
=
x
e-
ax x Wx
x
e-,E (x
=
vl
a
z-1
'A'
'A'
.
l ) 1.
-
vl
=
,.
u
(9.7)
For a, continuous distribution, the Lebesgue-stieltjes integral of a' coincides with the integral in ordinary Lebesgue measure of the function xfx).
9.3 Example For the uniform distribution on the interval gtz,)1(8.13), F(m
b
1
=
b a
xdx
=
1a(c +
-
a
b). n
(9.8)
9.4 Example For the Gaussian family (8.19), F(A3
1
=
-(x-jt)2/2o.2
#?
xe
Wfic
=
g.
(9.9)
This can be shown by integration by parts, but for a neater proof see 11.8.
I:a
ln a mixed continuous-discrete distzibution with atoms x1,x2,..., we can use the decomposition F F1 + F2 where F1(x) = Zouxgltlayl)and F1(x) is absolutely continuous with derivative hx). Then =
EX)
77.)
p,1(tA)l)
=
+
jxhxtdx.
(9.10)
The set of atoms has Lebesgue measure zero in R, so there is no need to exclude these from the integral on the right-hand side of (9.10). Some random variables do not have an expectation.
9.5 Example Recall the condition for integrability in (4.15),and note that for the Cauchy distribution (8.11),
1
+u
jxj
J-a(
-a
1+
x)2
dx
--
'>9
as
f'
-->
=.
El
(9.11)
130
Probability
9.2 Expectations of Functions of X lf X is a r.v. on the probability space @,f,g), and gl R F-- R is a Borel func-1), as note d in j8.1. This leads is a nv. on the space t'R,f tion, goX to the following dual characterization of the expectation of a function. =
'(m
,g.'
9.6 Theorem If g is a Borel function, F('(A'))
jglxtdbkxb /dB#-1t3')'
=
=
&n):R+
Proof Define a sequence of simple functions m i
z(a)(x)77I =
-
1
(9. 12) F-
R+ by
(9. 13)
1s,.@),
2n
1,...,-. Then, Z(n)(x) 1 and Bi = (2-&( 1), 2-'') for for x k 0, by arguments paralleling 3.28. According to 3.21, (R,S,pw -1 ) j s a g(#-1(#)) for B e S, and so by the monotone measurespace where g' -1(B)
where m
=
nln
't
+
.x
=
-
=
convergencetheorem,
-l(y) y;m
JZ(n)(A')t*'
1
-
.
f=1
zn
jydvg-t).
-1(sj)
.-.y
>'
(9. 14)
Consider first the case of non-negative g. Let 1s@) be the indicator of the set B G 0, and then if g is Borel, so is the composite function
1, glx)
(1so')@)
=
6
B
0, glx) e
B
=
1#-l(s)(x).
(9. 15)
Hence, consider the simple function (Z(,,)o#)@) =
m
77
i
1
-
2n
=1
m
(1sfo#)(x) =
By the same arguments as before, Znjog However, m
Eznjog)
=
77
j
j=1 m i
.
j
2n -
1
gn =X =1
'1'
77 f=I
i
-
2n
1
1#-1(sp@).
g, and Elznlog)
=
Eg)
=
JWg.
.j
g,(# Bi))
-
j
g,# Bi)
=
ztalllfp.'
-
1
(A),
and (9.12)follows from (9.14). To extend the result to general g, consider the non-negative g'b g separately. lt is immediate that maxf ',0J and g-
-->
(9.16)
(9. 17)
functions g'
=
Expectations E(Zn)og+4 Eznjog-) -
m
j.j
m
j-1
X =1
=
Eg-)
-
Eg),
=
(9. 18)
..j
>7 =1 2/
=
Eg')
--
of this limit separately.
so consider each component Eznlog-)
131
p,((#+) Bi)4 (9.19) jooycyg-jo),
-1
ptt# (wi ;;
2*
-..y
0
-1
-1
second equality holds because
where the g +) (Bi) g Bi) for i k g sjnce the 1 the tel'm elements of Bi are a11 positive for these cases, whereas for i disappears. Similarly, -Zn(x) 1 x for < 0, and =
=
.z'
?rj
-7r
-znjog-)
j.j
-
f=1
2/
m
i- 1
-:7
-
zn
j=1
.j
p.((#-)
Bi4)
0
j--ydv.g
-1
p,t, (#7))
--->
..j
(T),
(9.20)
1)1, and in this case the second equality holds where B; (-2-&, because (g- ) (#j) g (-Bi) j or j u z uence -2-''(f
=
-
-1
-1
=
.
Ezn,og
+) -
Ezn,og-)
j-
--
ydjw-lol,
(9.21)
and the theorem follows in view of (9.18).w The quantities F(X), for integer k 2 1, are called the moments of the distribution of X, and for k > 1 the central moments are defined by EX
-
Exlk
=
k
p
X
-
j
Eujh-kjxllk-i
(9.22)
-
A familiar case is the variance, Vartm E(X- F(m) 2 EX ) F(A') the usua j measure of dispersion about the mean. When the distribution is symmetric, with PX- F(A') e A) #(F(m X e 4) for each A e tB,the odd-order central moments are all zero. =
=
9.7 Exampl.
=
2
-
2
,
-
For the Gaussian case
(8.14),the
k'sk EX
-
g) k
=
2
kll
'
(kll)!
central
k even,
moments are
(9.23)
k odd. This formlamay be derived after some manipulation from equation (11.22) below. Vartm c 2 an d a11 the finite-order moments exist although the sequence increases monotoniclly. n =
,
Probability
132
The existence of a moment of given orderrequires the existence of the corresponding absolute moment. If FlXlP < =, for any real p > 0, X is sometimes said to belong to the set Lp (of functions Lebesgue-integrable to order p), or otherwise, to be Lp-bounded.
9.8 Example For X
N(0,c2), we have, by
-
FlXl
2(2a
=
(8.26),
y)-1/2j*.x e-xllloldx ()
(9.24)
l?2 = (2/1) c. n 'raking the corresponding root of the absolute moment is convenient forpumoses of comparison (see 9.23) and for X e Lp, the Lp-ttorm of X is defined as
(FI#I')'* 11111 p
(9.25)
=
The Gaussian disibution possesses a11finite-order moments according to (9.23), but its support is none the less the whole of R, and its p-norms are not unlrmly x, it coincides with the essential bounded. If IIXII, has a finite limit as p supremum of X, so that arandom variable belonging tofax is bounded almost surely. --
9.3 Theorems for the Probabilist's Toolbox The following inequalities for expected values are exploited in the proof of innumerable theorems in probability. The first is better known as Chebyshev inequalf@ for the special case p = 2. 's
9.9 Markov's
For er > 0 and # > 0,
inequality
#( l11 k Proof
tpp
l.Y1
Epjj
:)
=
Al
f:)
Fj aYjf
(9.26)
.
:
yvdFx) K
P
Jjzjzalxl PdFx) f E
1115.*
This inequality does not bind unless F1X1PIZP < 1, but it shows that if Fj -Yjr < x. The order of the tail probabilities converg to zero at the rate z-P as f: fv-boundedness measures the tendency of a distribution to generate outliers. The Markov inequality is a special case of (at least) two more general inequalities. =,
...-4
9.10 Corollary For any event A e F, E
Equivalently,
#((:
j
Arnt
IX()) I
Proof Obvious from 9.9.
lllkcl :1
dp fa
u
j IxI,g,. A
A) f F(1z
(9.27)
lXI#)/e'.
.
9.11 Corollary Let g: R F-> R be a function with the property that x ga) > 0, for a given constant a. Then gxj
a implies
Expectations
PX k a) <
Proof gajpx
k a4
&'(m) .
.
(9.28)
.
ga)
gajxyadFx)
=
133
jxyggxldFx)
S
K
EgT).
w
An increasing function has the requisite property for all a > 0. Let 1 ERbe any interval. A function (: I F-.: R is said to be convex on 1 if
9((1 )x +
,)?)
-
for a11 x, y e I and
.
(0,11. lf
e
(1 )(x)
f
+
-
-9
is convex
*(.y) on t # is
(9.29) on 1.
concave
9.12 Jensen's inequality lf a Borel function (1)is convex on an interval 1 containing the support of an integrable r.v. X, where ((m is also integrable,
(9.30)
9('(A')) f f@(A')). For a concave function the reverse inequality holds. n
The intuition here is easily grasped by thinking about a binary r.v. taking values with probability 1 A convex 4)is illustrated in xl with probability p and whereas F@(A')) p#(x1) + (1 This Fig. 9. 1. #(m px3 + (1 vertical chord the axis by point is mapped from F(m onto the joining and on 9, while 9(F(m) is mapped from the same point by (j itself. -p.
,:2
,-pjxz,
-p)9(x2).
=
=
-:2
.x1
Fig. 9.1 A proof of the inequality is obtaied interior of 1.
9.13 Lemma If and y e 1,
4)is convex
from the following lemma. Let 10 denote the
there exists a function A@) such that, for a11 e 10
X(-:)@
.x
-x4
f #(A') -
9(*.
(9.31)
Proof A convex function possesses right and left derivatives at all points of I0.
Probability
134
(9.29)implies
This follows because (x +
kh)
9(x)s
-
kh
(j(.7 +
for h h) h
-
>
0 that
#(x)
N s ((),jj.
'
(9.32)
+ 1/n) The sequence n e N ) is decreasing, and has a limit 9;(x). (T)(x)), In the case < 0 the inequality in (9.32)is reversed, showing the existence of 4,-/(x)as the limit of an increasing sequence. Note that ()-'@) < (J(x).Tnking the + h gives limit as 2. 4. 0 with > 0 tixed in (9.32)and y
tn(#tx
-
.z7
=
t)-'@)tJ'
S K(-:')(J'
-x4
-xt
S t)(A')
9(.1)
-
<
0 gives, for y
;(.z7)@ S #-'@)1
; th(A') t)@)
whereas the parallel argument with h -x)
-x)
F(m, y
=
X in
=
A('(m)(X-f(m) Taking expectations side has expectation
x,
(9.34)
-
lnequality (9.31)is therefore satisfied with (say)A@) Proof of 9.12 Set x
<
(9.33)
=
#+'(x)..
give
(9.31)to
#(m #(-)).
f
(9.35)
-
of both sides gives inequality
(9.30),since the
left-hand
zero. w
Next, we have an alternative approach to bounding tail probabilities which yields the Markov inequality as a corollary.
9.14 Theorem If X is a non-negative r.v. and r EX
rj-xr-zpx
r)
-
(9.36)
xvx.
>
0
Proof lntegration by parts gives, for some b joxrdFx)
0,
>
=
-
bryb) rjo
= r
-
>
0,
rjoxr-ypdx
Fpldx V--l(s(,) -
x 0
r-lp
dx.
(.x< x u
The theorem follows on letting b tend to infinity.
.
If the left-hand side of (9.37)diverges, so does the theorem is true whether or not EX) is finite.
9.15 Corollary
lf X is non-negative
J-
-xdF
-
spx k
(9.37)
light,
and in this sense the
and integrable,
o
+
j-px
> xvx.
-
(9.38)
Expectations Proof Apply 9.14 Fith r
135
1 to the r.v. 1(xk:)X. This gives
=
J- J-tlfxkslx = Jo 'dF
=
>
xvx
>
xltfx
o
$(1(xk:)X
+
j*PX
>
xldx
:
=
#(X k
E ejodx
jvP(X c
+
>
xjdx.
.
(9.39)
Not only does (9.38)give the Markov inequality on replacing non-negative X by IXI; for p > 0 and arbitral'y X, but the error in the Markov estimate of the tail probability is neatly quantified. Noting that P( IXI E) = 81.11#2 zP),
s ##( I-Yl k e)
j-
=
e
#
IXIJWF
= FIXl# where both the subtracted
j
-
j-
-
gP
P
IA-I'>
#
IXIPJF
px
-
0
I J
xldx
#(IXIP
eP
terms on the right-hand
>
xqdx,
(9.40)
side are non-negative.
9.4 Multivariate Distributions From one point of view, the integral of a function of two or more random variables presents no special problems. For example, if
gl R2
F-y
IR
is Borel-measurable, meaning in this case that g -1 B4 e Bl for every B e B, then #(#((l)),F()) is just a F/f-measurable r.v., and () =
F('(X,F))
=
Jjatldptl
(9.41)
is its expectation, which involves no new ideas apart from the particular way in which the t.v. hlk hppens to be defined. Alternatively,
the Lebesgue-stieltjes Eg-)
=
form is
jpzgx'ybdFx'L'
(9.42)
where dFxny) is to be thought of as the limiting case of &F@,yl defined in (8.30) as the rectangle tends the differential of area. When the distribution is with respect to continuous, the integral is the ordinary integral of '(x,y)J@,y) Lebesgue product measure. According to Fubini's theorem, it is equivalent to an iterated integral, and may be written +x
&'(X,l?))
=
+x
j-yqx,ytx'ybdydx. J.x
But caution must be exercised with formula
(9.43)
(9.42)because
this is not a double
Probability
136
integral in general. lt might seem more appropriate to write d lFx y) instead of JF(x,y), but except in the continuous case this would not be correct. The abstract notation of (9.41)is often preferable, because it avoids these ambiguities. In spite of these caveats, the expectation of a function of (say)X alone can in every case be constructed with respect to either the marginal distribution or te joint distribution. ,
EglxL
9.16 Theorem
jvz
#(X#F(mA')
=
jpgxjdklxj.
=
Proof Define a function +: R2 b..A (R
g
by setting g *@,y)= glx), a 11y e R :*-14#) is a cylinder in R2 with base :-1(#) e B for B e S, and g * i s Sz/s-measurable. For non-negative g, let .
m
* #(a) where m nl n+ 1 an d E i = = l-nli .j xR where Af (x: m
X
=
f=1
i
2
n
(9.44)
ls i
ln
=1
2-''(f 1) S #*(x,y) < 2-NI') e Bl Since Ei 1) K #(x) < 2-% l and tx(Af) = g,(&), -
,
-
.
=
,
j
-
j
.
(@y):
=
F(#*(,'))
j.
=
m
g,t&l
X
=
i
=1
j
-
2
n
p,x(A)
=
Elgnlj.
(9.45)
By the monotone convergence theorem the left and right-hand membrs of (9.45) *x,yldFx,y) and Egj xjdFxxj respectively. Extend converge to E(g.) non-negative general complete the from proof. * to g to =
=
The means and variances of X and F are the leading cases of this result. We also have cross moments, and in particular, the covariance of X and 1' is Cov(X,F)
=
E((X-
&m)(1'- F(F)))
=
Fubini's theorem suggests a characterization
ECXT F(mF(F).
(9.46)
-
of pairwise independence:
9.17 Theorem lf X and F are independent nv.s, Cov@(m,v(F)) of integrable Borel functions ( and
=
0 for al1 pairs
v.
Proof Fubini's theorem gives F(#(mv(F))
=
=
jp,
#@)v(y)#F@,y)
jpxtdlxjvlsbdFkl-bq
= &#(m)F(v(F)).
w
(9.47)
The condition is actually suffictent as well as necessary for independence, although this cannot be shown using the present approach; see 10-25 below.
Expectations
137
Extending from the bivariate to the general k-dimensional case adds nothing of substance to the above, and is mainly a matter of appropriate notation. lf X is a random k-vector, F(m
=
jxdFx)
(9.48)
denotes the k-vector of expectations, EXi) for 1,...,k. The variance of a scalar r.v. generalizes to the covariance matrix of a random vector. The k x k matrix, =
2 X1X2 X1
XX'
.
.
.
X3Xk
2 X2X1 X2
(9.49)
=
2
Xkxj
Xk
is called the outer product of #, and ECXX') is the k x k positive semi-detinite matl'ix whose elements are the expectations of the elements of XX'. The covariance matrix of X is Vartm
ELCX F(m(#
=
-
-
F(m)'q
=
ECXX') ExTEX4'.
(9.50)
-
is positive semi-detinite, generalizing the non-negative property of a scalar variance. lt is of full rank (notwithstanding that XX' has rank 1) unless an element of X is an exact linear function of the remainder. The following generalizes 4.7, the proof being essentially an exercise in interpreting the matl'ix formulae. Vartm
9.18 Theorem lf F= BX+c where X is an k-vector with EX4 g and Vartm = E, and B and c are respectively an m x k constant matrix and a constant v-vector, then (i) ElY) B# +r. BSB'. a (ii) Vartp =
=
=
Note that it m
>
k Vartl'l is singular, having rank k.
9.19 Example If a random vector Z (Z1,...,Zk)' is standard Gaussian (8.19),it Ik. Applying 9.18 to 0 and EZZ') is easy to verify, applying 9.17, that Ezj = the transformation in (8.35)produces F(A') g and =
=
Vartm
=
EX
-
p)(#-
g4'
=
F(AZZ'A')
=
=
AF(ZZ')A'
=
M'
=
E. n
9.5 More Theorems for the Toolbox Thefollowingcollection of theorems, together with thelensen and Markovinequalities of j9.3, constitute the basic toolbox for the proof of results in proba-
Probability
bility. The student will tind that it will suffice to have his/her thumb in these pages to be able to follow a gratifyingly large number of the arguments to be encountered in subsequent chapters. 9.20 Cauchy-schwartz
inequality
2<
Ex
with equality
attained when F
EXl)Eyl)
(9.51)
,
cX, c a constant.
=
Proof By linearity of the integral, ELaX+ F)2)
a
=
lEXl)
+ E(Yl) ZXIEIXLI
+
for any constant a. (9.51)follows on setting a equality if and only if aX+ F 0. .
=
O
-EIXYIIECXl ), an d holds as an
=
The correlation
c/elcdaf,
rxy
=
1/2 accor dingjy Cov(X,F)/(Var(mVar(D)
the interval (-1,+11. The Cauchy-schwartz inequality is a special case (forp 9.21 Hlder's p/(p
=
-
=
2) of the following.
For any p 2 1,
inequalil
11-Y11,111'11z?.
FI-YFIK where q
jies in
,
1) if p > 1, and q
Proof The proof for the case p
>
if p
x
=
1 requires
=
(9.52)
1.
a small lemma.
9.22 Lemma For any pair of non-negative numbers a,b, ab S
tzp bq
-
P
+
(9.53)
-.
q
If either a or b are zero this is trivial. lf both are oositive, let s = etlq ay = & and t = q log b. Inverting these relations gives a es y rlogtz S/PTf# d an d (9 53) follows from the fact that ex is a convex function of x, noting 3Iq 1 1/p and applying (9.29).* Proof
=
,
,
.
=
Choose a
.
,
=
-
)XI/II-Yllp,b
=
F11t?.For lr1/11
FIXFI
XIp
F
q
= Eab)
these choices, Eap)
S 1/# +
$Iq
=
=
Ebq)
1.
=
1, and
(9.54)
S Fl Xl ess sup F, which holds For the case p = 1, the inequality reduces to FI-YFI since F < ess sup i' a.s., by definition. w
The
lder inequality spawns a range of useful corollaries and special cases, includingthe following.
9.23 Liapunov's
inequality
Proof Let Z F:zlXIF, F = 1, s
(norm inequality) =
r/p. Then,
If r > p > 0, then
(9.52)gives FIZFIf
jlxllr I1-YIlp.
or IIzII,II FIIs(,-1),
Expectations E ( 1X 1#) < E 111
pslls
=
rfr
I11
E
(9.55)
.
.
This result is also obtainable as a corollary of the Jensen inequality.
9.24 Corollary
For each A e
jg IXI'IdP Proof ln
and 1/p + jIq
.
1/p
Jx
p
l71 PdP
<
1,
=
Jx
1Iq
11'IqdP
X by Xlx and i' by i'1x. x
(9.52),replace
probabilistic in character.
Alternative variants of the result are not explicitly
9.25 Corollary n
Let x1,...,x,, and y1,...,yn be any sequences n
77 izz1
1/,
77 1 izzz
lxafl S
771 IyfI
Then
for 1/p + Iq
=
1. n
fzz
be Lebesgue-integrable
functions of a real
'''-(J-
1/,+
(j-
I?f)I,z,)
'-ftkbgtkbkdt s
of numbers.
1/<
n
1x/1J'
and gl
9.26 Corollary Let fl variable. Then
J-
.
I,,(,)l-?,)''
or
lq
-
1. a
Proofs are left as an exercise. The sequences in 9.25 and the functions in 9.26 can be either real or complex-valued (see jl 1.2).
9.27 Minkowski's Proof For r
=
inequality
For r k 1, IIX+ i'llr <
i'llr. + 11 IIXIIr
1 this follows direct from the triangle inequality,
IX+ i'I < I-YI+ I1'l on taking expectations. For r
>
(9.56)
,
1, note that r-1)
FIx+ FIr F(I#+ FII-Y+ FI S E IXI+ IFI)IX+ l'I ) r-1). r-1) + E Irl I x+ I'I = E IxIIX+ l'I =
r-1
inequality to the two right-hand-side
Applying the Hlder
'
E IX+
FI
<
r)1-1/r.
+ 11 X+ l'I FIIr)(FI (II#IIr
Cancelling A'IaY+FI and rearranging r
application to the sum of m variables, generalizes directly to K r
terms yields
(9.58)
gives the result. w
By recursive
Yxi f=1
(9.57)
77Ilmllr =1
for r k 1. For an intinite series pne can write
the Minkowski inequality
(9.59)
Probability
140
77m =1
IImlIr+77xi X i=?n+1 f=1
:
r
If
0 as IIZ7=,,,+1XjIIr --
m
it is permissable to conclude that
tx.
--
(9.60) r
o/
X
i= 1
xi s
f=1
r
IIx)Ilr,
(9.6 1)
not ruling out the possibility that the right-hand side is infinite. 9.28 Love's
cr inequality
For r Fq
e' where cr
=
Txi
>
r
m
s
f=1
1 when r S 1, and cr
0,
=
r, cr XA'1 Xi I
(9.62)
i=1
m r-l wj)en r
j
.
Proof This goes by proving the inequality m
r
m
<-
ai /=1
cr f=1
Iai Ir
(9.63)
for real numbers JI,...,tza, then substituting random variables and taking expectations. Since 1Z0,zzkai lr S (E't', Iai I)r, it will suffice to 1et the ai be non-nega1, tive. For the case 0 < r S 1, 0 f zi f 1 implies z; k zi and hence if Z'?, ctajj. XTt k 1. (9.63)follows on putting zi = ai /(Z0./ For r k 1, on the other hand, convexity implies directly that .1
xlzf
=
=lz(,
j -
m
X icz:l
r
ai
K
-
1
m
XX.
*
izz,1
9.29 Theorem If X, 1', and Z are non-negative r.v.s satisfying X f tz(F+Z) a.s. for a constant a > 0, then, for any constant M > 0, FtltxyvlA')
f
zJtftlf
y>M/2c)F)+F(1(z>ArJw)Z)).
(9.64)
Proof lf we can prove the almost sure inequality 1(x>M)X S 2J(1ty>&z/2u)F+1(z>M/2u)Z),a.s.,
(9.65)
the theorem will follow on taking expectations. 1(x>M)Xis the r.v. that is equal to X if X > M and 0 otherwise. If X K M (9.65)is immediate. At least one of the inequalities F k X/M, Z k Xlla must hold, and if X > M, (9.65) is no less obviously true. w
9.6 Random Variables Depending on a Parameter Let G(,0): f x 0- F-> R, O i R, denote a random function of a real variable 0, or n other words, a family of random variables indexed on points of the real line.
141
Expectations The following results, due to Cramr nated convergence theorem.
easy consequences of the domi-
(1946),are
9.30 Theorem Suppose that for each (l) e C, with #4C) 1, G4,0) is continuous at apoint %,and IG((t),0) I < F((t))for each 0 in an open neighbourhood No of % where =
F(i')
<
Then
x.
lim EG(44
(9.66)
F(G(0o)).
=
8-+%
Proof Passage to a limit 00 through a continuum of points in 0- as indicated in (9.66), is implied by the convergence of a countable sequence in 0- Let (0v,v e NJ be such a sequence, in No, converging to 0o. Putting Gv((t)) G(,0v) defines a countable sequence of r.v.s. and limsupvGvt) and liminfvGvt) are r.v.s by 3.26. By continuity, they are equal to each other and to G(,%) for (l) e Q in other G(%) a.s. The result follows from the dominated convergence words, G(0v) ,
.
=
-->
theorem.
.
9.31 Theorem 1f, for each
)
e C with #(C)
G(),%
+
h4
1, (JG/J0)((t)) exists at a point
=
G(,%)
-
n((o)
< for 0 f
< Jl1, where F(l'1)
KdoEa)
<
j
(x,
and
is independent of
l
ttjG()..)
-
(.,
I
e.
% and
),
then
(9.67)
.
Proof The argument goes like the preceding one, by considering a real sequence (v ) tending to zero through positive values and hence the sequence of r.v.s (SVJ where Hv gG(0o+/lv) G(%)1/v, whose limit H S(0()) exists by assumption. . =
=
-
The same sort of results hold for integrals. Fubini's Theorem provides the extension to general double integrals. The following result for Riemann integrals on intervals of the line is no more than a special case of Fubini, but it is useful to note the requisite assumptions in common notation with the above.
9.32 Theorem Suppose thatforeach (l) E C, withf'to a t'initeopen interval atb), and IG(,0) l < 1'2() for
=
1, G(,0) is continuous on 0 < b, where F(F2) < x.
q <
Then
s(js(e)ts)
J-s(c(e)):e -
lf J7'IG(,0) lJ0 < 1'3(t,))for (t) E C and Ek) and b +=. of a =
-x
(9.68)
.
-
<
cx),
(9.68)holds
for either or both
=
Proof For the case of t'initea and b, consider Hlsb,tl JJG(,0)#0. This has the I lG(),f) I < F2(), for each propertiesIH% I < b J)F2(t,)), and I dHld EH) exists and by /, each 9.31, for t e a, b4 Hence, =
-
.
=
Probability
l42
d EH) E(Gt))
is continuous
=
E
dH
=
dt
EGtj).
on a,b4 by the a.s continuity
a(,) -
jtEol4d
(9.69) of G and 9.30, and hence
-s(s(?))
(9.70)
J
is differentiable on (a,b4,and dhldt = 0 at each point by (9.69).But by defi0, and hence hb) 0 for e C, so that A(J) 0 which is equivnition, H,a) alent to (9.68). =
=
=
JX
G(,0)#0 exists and is Under the stated integrability condition on G(,0), = J!xG(t0,0)#0 is well defined and has an expectation for finite on C. Hence Htt) and and/or b = x. > th argument above goes through with a = all t e R, -X
-.x
10 Conditioning
10.1 Conditioning in Product Measures It is difficult to do justice to conditioning at an elementary level. Without resort to some measure-theoretic insights, one can get only so far with the theory before nznning into problems. There are none the less some relatively simple results which apply to a resicted (albeitimportant) class of distributions. We approach, and so demonstrate the diftiintroduce the topic by way of this arise, before going culties that on to see how they can be resolved. In the bivariate context, the natural question to pose is usually: we know X what of lX,l') F?' For a random real pair is the best predictor on (,T, #) we = x, class conditional evidently define of functions for distribution j7.2) a (see can K For any A e B such that PX e A) > 0, let 'naive'
dif
F@ lX
G
A)
PX e A, F < y) PX s A)
=
( 10. 1)
.
This corresponds to the idea of working in the trace of (f1,T,#) with respect to A, once A is known to have occurred. Proceeding in this way, we can attempt to construct a theory of conditioning for random variables based on the c.d.f. We may tentatively define the conditional distribution function,F(.y Ix),when it exists, as a mapping from R2 to R w hich for tixedx q R is a non-decreasing, right-continuous function of y with F(-x lx) 0 and F(+x lx)= 1, and which for fixed y e R satisties the equation =
Px
e
,4,
J' < y)
jgFl.y
=
Ix)#Fx(x)
(10.2)
in y-space for any A e B (compareRao 1965: j2a.8). Think of the graph of F(.yIx) through the sudace of the joint distribution funcas the protile of a tion, parallel to the y-axis, at the point x. However, much care is needed in interpreting this construction. Unlike the ordinm'y c.d.f., it does not represent a probability in general. If we try to interpret it as P(Y S y IX x), we face the possibility that P(X x) 0, as in a continuous distribution. Since the integral of F@ lx) over a set in the marginal distribution of X yields a probability, as in (10.2),it might even be treated as shows that we would need a type of density function. Taking A (.Y f islice'
=
=
=
.x)
=
,(x,y)j-,'(,I()#,k(t) J-j' g,'(vI():,'x(t) -
-
ex
ex
ex
(10.3)
Probability
144
to hold. Since F(x,y) is an integral over R 2 F u bini's theorem implies that F@ 1x) is well detined only when the integrals in (10.3)are with respect to a product ,
meastlre.
If Xand Fare independent we can say unambiguously (butnot very usefully) that FLyjx) Fy%). F(? j.x)is also well defined for continuous distributions. Let Sx denote the support of X (the set on which fx > 0); the conditional p.d.f. is =
J# Ix)
fx,yj .fx@)
=
,
.x
e
sx,
(10.4)
where jxx) is the marginal density of X. We may validly write, for A e B
px
e
x, r s
jgj'
.ftvlo/'axltxo
y)
-
-
f''h
Sx,
j z-twylxlyxt.xlts (10.5) #
-x
where F(..lxl
j' Avlxlfo.
-
(10.6)
-K
The second equality of (10.5)follows by Fubini's theorem, since the function fxny) is integrated with respect to Lebesgue product measure. However, (10.6) appears to exist by a trick, rather than to have a firm relationship with our intuition. The problem is that we cannot work with the trace (A,Ta,#a) when A = (: X()) xl and #(A) 0, because then #a = #/#(A) is undefined. lt is not clear the case when (,Y = x) has occurred' when this event what it means to fails to occur almost surely. Except in special cases such as the above, the factorization dF(y,xl = dF@ Ix)#Fx@) is not legitimate, but with this very important caveat we can define the mean and other moments of the conditional distribution. The conditional expectation of a measurable function #(-Y,i'), given X = x, can be defined as =
=
Ex,n
Ix)
+*
J
=
glx-ybdFytxq,
(10.7)
-X
also written as Elgx,Y) lX = #. The simplest case is where g(X,j') is just F. f(l'Ix) is to be understood in terms of the attempt of an observer to predict F after the realization of Xhas been observed. When Xand Fare independent, &F1x) where EY4 is the ordinary expectation of F, also called the marginal or = E, unconditional expectation. ln this case, the knowledge that X x is no help in predicting K =
10.1 Example These concepts apply to the bivariate Gaussian distribution. From (8.36), the density is -1
fx,yj
1
=
2a
C11 C12
c12c22
j/2
exp
,r
-
a
611 C12
z't.x g.1,y g,21 C12 -
-
C22
/1
X-
F
-
/2
Conditioning
145
(A' g,2) -
-
1
= 2J(cl
1c22
2 cl2 g,1) (.:7 c1l (x -
-
-
c212)
exp
jzl) 2
-
-
l/2
2 c22
zcjj
c212
,
(10.8)
-
c11 the square in the exponent.
where the last equality is got by completing Evidently, fxky) fy Ixjfxx) where =
(A,
p,2) -
-
1I)
fl.y
1
=
2 WIVc2a c12 c11
exp
ja
c12 c11
2
'--fx
-
-
2 c22 -
-
c21 a
M,1)
(10.9)
c11
and fxxj
l exp 2zrc11
=
-
@- B1)1
zo.11
.
(10.10)
Thus, F(Flx)
(T12
----t.r
=
g,2 +
-
c22
p,1),
(10.1 1)
and VartFlxl If c1a
of the
=
0,
fy lx) reduces
marginals,
=
622
-
c212 .
c11
(10.12)
so that the joint density is the product and x and y are independent. n to
fvy),
10.2 Conditioning on a Sigma Field In view of the limitations of working directly with the distribution of (aY,l'), we pursue the approach inoduced in j7.2, to represent partial knowledge of the distribution of F by specifying a c-field of events N g 5 such that. for each G e N, an observer knows whether or not the realized outcome belongs to G. The idea of knowing the value of a random variable is captured by the concept of subfield measurability. A random variable .X()): fl F-> R is said to be measurable with respect to a c-field V c 5 if x-2B)
=
(:
.Y4(0)G #) e V, for al1 B e f.
(10.13)
The implication of the condition V c 5 is that the r.v. X is not a complete
Probabilit.y
representation of the random outcome ). We denote by c(m the intersection of all c-fields with respect to which X is measurable, called the c-fild generated by X.
Jf, on being confronted with the distribution of a random pair (X()),1'())), we learn that X we shall know whether or not each of the events G e ctm has by determining whether X(G) contains x. The image of each G e c(A3 occurred under the mapping .x,
=
(X()), F()))2 D
F--> R
2
is a cylinder set in R 2 an d the p.m. defined on c(A') is the marginal distribution of X. .
10.2 Example The knowledge that xl fk X ; can be represented R). R c(f(-x, x1: x < xl ), ((x, x): x > .x2
by
.x2),
=
Satisfy yourself that for every element of this c-field we know whether or not X belongs to the set; also that it contains all sets about which we possess this knowledge. The closer together a'1 and are, the more sets there are in R. Wben +x, lO,R 1. n J4 = 5 ctm, and when xl .v1 x2, R .x2
=
=
=
-x,
.:2
=
=
The relationships between transformations and subfield measurability are summarized in the next theorem, of which the first part is an easy consequence of the definitions but the second is trickier. lf two random variables are measurable with respect to the same subfield, the implication is that they contain the same information; knowledge of one is equivalent to knowledge of the other. This means tha t evety Borel set is the image of a Borel set under g- This is a stronger condition than measurability, and requires that g is an isomorphism. It suffices for g to be a homeomorphism, although this is not necessary as was shown in 3.23. .
10.3 Theorem Let X be a r.v. on the space (S,fs,p), and let i' gl S F-+ 'T is a Borel function, with S i R and 'T i R. (i) ctl') c ctm. (ii) c(1?) ctm iff g is a Borel-measurable isomorphism.
=
glX) where
=
Proof Each B e f.r has an image in fs under g -1 w hich in turn has an image in c(m under X -1 This proves (i). To prove (ii), define a class of subsets of S, C (.g-1(#): B e B v.). To every -1 A S there corresponds (sinceg is a mapping) a set B 'T such that A = g B), and making this substitution gives ,
.
=
fs
lz4:
'(z4)
e
.r 1
fp-j (#): gg
=
-1
(#)) e s v l
=
g
jj(;
,
.
j4;
where the inclusion is by measurability of g -1 an d the second equality is because -' = gg (#)) B for any B c 'T, since g is 1-1 onto. It follows from (10.14)that ,
fx -1( s -4)
j:
z4
e ls
)
g
(x-1(:-1(,)) c
f: B s f.y).
(10.15)
If F is V-measurable for some c-field V 5 (suchthat V contains the sets of the right-hand member of (10.15)), then X is also V-measurable. ln particular, ctm ctl?). Part (i) then implies c(m = c(F), proving sufficiency of the conditions. = To show the necessity, suppose first that g is not 1-1 and glxL) = y '(.n)
Conditioning
(x1) and (x2)are elements of Ss but not of C, which contains only g -1(fy)) fxl) k.p (x2) Hence : Bs C, and 3 a Os-set X for which G there is no Or-set B having the property g -1(B) A This implies that (say)for xl
y:
The sets
.n.
=
.
.:-1(4)
=
.
c(F), so that c(F) c ctm. We may therefore assume that g is 1- 1 If g is not Borel-measurable, then by C; and again, definition 34 = g -1(B) G B s suc h thatgtA) =B e tB.r,and henced Ss C, so that c(l') c ctm by the same argument. This completes the proof of
ctm but
-1
.
necessity.
.
We should briefly note the generalization of these results to the vector case. A random vector #4): f F-> Rk i s measura b1e with respect to 97 c 5 if X -1 B4
=
f
:
A)l
e #)
G
s vs ,
s pk
(1().j6)
.
If ctAr) is the c-tield generated by X, we have the following result.
10.4 Theorem Letfbe $ c R an d Bks fl =
F
k arandom vector on the probability space (S,fs,p) Sk) and consider a Borel function s: B e
f-h
,
g(m: S
=
w jaere
F--
1', T' g R2'.
(10.17)
(i) G'(1') l c(A'). (ii) If m k and g is 1-1 with Borel inverse, then c(1') =
=
ctm.
Proof This follows the proof of 10.3 almost word for word, with the substitutions of fs k an d Bkr for Ss and S.r, X and F for X and F, and so forth. .
10.3 Conditional Expectations Let i' be an integrable r.v. on (f,T,#) and N a c-tield contained in F. The term conditional expectation, and symbol F(FIN), can be used to refer to any integrable, N-measurable r.v. having the property jgFtrl
U)dP
=
F(FI joj'dp
G)#(G), all G e N.
=
(10.18)
Intuitively, F(FIV)()) represents the prediction of F() made by an observer having information V, when the outcome (.t) is realized. The second equality of (10. 18) supplies the definition of the constant S(l'l G), although this need not exist unless #4G) > 0. The two extreme cases are F(FI@) i' a.s., and F(Jrl 5) EY) a.s., where T denotes the trivial c-field with elements (f,0 ). Note that f s 5', so integrability of F is necessal'y for the existence of F(FIV). The conditional expectation is a slightly bizarre construction, not only a r.v., but evidently not even an integral. To demonstrate that an object satisfying (10.18) actually does exist, consider initially the case F 0, and define =
v(G)
=
JgF##.
10.5 Theorem v is a measure, and is absolutely
=
(10.19) continuouj with respect to #.
Probability
148
0. lt remains to show Clearly, v(G) 2 0, and #(G) = 0 implies v(G) is a disjoint sequence, then countable additivity. If ll Proof
=
vUb
v,o,Ydp
= *'
j
J
=
J
X j
o,Ydp *'
-
'
Xvt(l,
(10.20)
./
where the second equality holds under disjointness.
.
So the implication of (10.18)for non-negative F turns out to be that F(FIT) is the Radon-Nikodym derivative of v with respect to #. The extension from nonnegative to general F is easy, since we can write F F+ F- where F+ and Yare non-negative, and from (10.18),F(Fl V) F(F+IV) E(Y- IN), where both of the right-hand r.v.s are Radon-Nikodym derivatives. The Radon-Nikodym theorem therefore establishes the existence of F(FI5:); at any rate, it establishes the existence of at least one nv. satisfying (10.18).It does not guarantee that there is only one such r.v., and in the event of nonuniquenes, we speak of the different versions of F(l'IV). The possibility of multiple versions is rarely of practical concern since 10.5 assures us that they are al1 equal to one another a.s.g#j, but it does make it necessary to qualify any statement we make about conditional expectations with the tag a.s.', to indicate that there may be sets of measure zero on which our assertions do not apply. ln the bivariate context, F(Fl c(m), which we can write as FtFlm when the context is clear, is interpreted as the prediction of F made by observers who observe X. This notion is related to (10.7)by thinking of F(l'l x) as a drawing from the distribution of F(FIm. =
=
-
-
10.6 Example ln place of (10.11)we write &Flm(t,))
61 2
=
---(X() p,2 + C1 1
-
p.1),
(10.21)
which is a function of X(l, and hence a r.v. defined on the marginal distribution of X. F(FIm is Gaussian with mean g,2 and variance of c2la/cll. E! Making
F(FIx)a point
in a probability space on R circumvents the difficulty
previously with conditioning on events of probability 0, and our is valid for a11distributions. lt is possible to define F(FIG4 when
encountered construction #(G) 0. What is required is to exhibit a decreasing sequence fGn e N) with #(Gn) > 0 for every n, and Gn 1 G, such that the real sequence (F(1'1 Gn)) converges. This is why (10.7)works for continuous distributions. Take Gn gx,x + 1/p1XR s =
=
c(A'), so that G
=
Using
(x)xR.
(10.4)in (10.18),
+co f*x+1/n
f(rl Gn)
J..J
yfl,ytdldy
+x
x
=
--
+.
ujtn
uJ-?'(''.''vjvy
j-giy'xsdy =
&Fl*,
(10.22)
Conditioning x. Fubini's theorem allows us to evaluate these double integrals one as n dimension at a time, and to take the limits with respect to n inside the integrals with respect to y. --->
Conditional probability can sometimes generate paradoxical results, following case demonstrates.
as the
10.7 Example Let X be a drawing from the space (g0,11, p,IJ, m), where m is Lebesgue measure. Let N c fp,1) denote the c-field generated from the singletons fxl, x e (0,1). A1l countable unions of singletons have measure 0, while all complements have measure 1. Since either #(G) 0 or #4G) = 1 for each G e V, it is clear from (10.18)that F(XIN) F(m a.s. However, consider the following weknowwhetheror notxc Gforeach Ge V, weknow argument. 'Since fx) e N, if x. ln particular, V contains knowledge of the outcome. It ought to be the case that F(XIV) X a.s.' n =
=
=
,
=
The mathematics are unambiguous, but there is evidently some difficulty with the idea that V should always represent partial knowledge. lt must be accepted that the mathematical model may sometimes part company with intuition, and generate paradoxical results. Whether it is the model or the intuition that fails is a nice point for debate.
10.4 Some Theorems on Conditional Expectations 10.8 Law of iterated expectations
(LlE)
FgF(J'IN)j Proof lmmediate from
(10.23)
Ek.
=
(10.18),setting
G
=
D.
.
if constant' under The intuitive idea that conditioning variables can be held the conditional distribution is confirmed by the following pair of results. 'as
10.9 Theorem lf X is integrable and V-measurable, then Proof Since X is N-measurable, E
+
then
jyydp
-
=
1): X()
>
F(XIN)()1
F(.YI jg-yxf jyjxN)J#
This contradicts (10.18),so PE*) = (tt):X((t7) < #(xl V)(@) E V.
N))J#
=
=
F(#IN)
>
=
X, a.s.
e N. lf #(F+)
0.
0. By the same argument, P(E-)
>
0,
(10.24) =
0 where E-
*
10.10 Theorem lf Fis F-measurable and intepable, X is V-measurable for V g @, N) a.s. < x, ten &FaY1T) and E IA-FI -YSIFI =
Proof By definition, the theorem follows if %)dP xYFIFI
Let
xn)
=
o
'z-lajlsj
=
J s
XYdP a.s., for al1 G e T.
(10.25)
be a V-measurable simple f.v., with F1,...,Fn a pmition of
Probability
150
f and Ei e V for each i. (10.25)holds for X
X(n) since, for all G e V,
=
jgxtnwtyl cldp nwj ac, s(I,Itvp x -
i= 1
=X-
.ij
yw,
-
GIAEt'
/=1
j x-,vdp,
(10.26)
G
Ei e N when G e V and Ei e V. Let X k 0 a.s., and let (X(,,)J be a monotone sequence of simple N-measurable converging XY a.s. and IX(n)F( K lXFI functions to X as in 3.28. Then XnlY N) where F1 XFI < x by assumption. Similarly, X(n)F(FI #FIl'IV) a.s., and
noting G
fa
,--+
,
--
e-l-Ytnlftrl N) I
=
1-Y(n)1'l I9-))
F1 F(A)n)FI N) I S EE
= FlaYtall'l
e-IA'l'I<
<
(10.27)
x,
where the first inequality is the conditional modulus inequality, shown in 10.14 below, and the second equality is the LlE. It follows by the dominated convergence JcxftFl N)##, and so (10.25)holds for non%)dP theorem that Jc#(n)F(rI negative X. The extension to general N-measurable X is got by putting -->
x where X*
=
maxfmo)
eFXl 9')
=
=
X+ -X-
(10.28)
2 0 and A'- 2 0, and noting
EIYX'P
= X*
-
-
FX- IN)
X-)F(l'IN)
=
ECYX' IN) EYX- lN) -
XFIFIV) a.s.,
=
using (10.33)below and the result for non-negative X.
(10.29) .
X does not need to be integrable for this result, but the following is an important application to integrable X.
10.11 Theorem lf F is F-measurable and integrable and F(rl V) El') for N c @, then Covtmp = 0 for integrable, V-measurable X. =
Proof From 10.8 and 10.10,
ECXYI #(A'(A-FlN)) =
If
F(FIV)
=
EY) a.s.
(a constant),
=
Ft-tftrl
V)J.
= then ECXYI EkEr).
(10.30) w
Note that in general Covt#,F) is defined only for square integrable r.v.s X and K But Cov(X,Y) 0, or ECXYI= FtmFtF), is a property which an integrable pair can =
satisfy.
The following is the result that justifies the characterization of the conditional mean as the optimal predictor of J' given partial information. optimal'is sen to have the specific connotation of minirnizing the mean of the squared
prediction errors.
Conditioning 10.12 Theorem
.:
t $... .(y:,.. y... ).)... ../.
g
Let
denote any V-measurable approximation to F. Then < (1 Y- f'112. jlY- F(rI N) 112
.j;
-
...
151
(10.31)
-
. .. .. .-.
Proo
f (F-
hence
f)2 (F-F(FIN)j2+2(y-F(Fj f)2IT)
F((F-
=
Fg(F-F(Fj
a,q-
V)jgF(J'l T)
=
N))2jNj + (F(yl N)
-
a..v
Fj
-
+
gF(FIN) F)2, and
f12a.s.,
-
(10.32)
noting that the conditional expectation of the cross-product disappears by definition of F(1'1 V), and 10.10. The proof is completed by taking unconditional expectations tluough (10.32)and using the LIE. w The foregoing are results that have no counterpart in ordinary integration theory, but we can often exploit the fact that the conditional expectation behaves expectation, apart from the standard caveat we are dealing with like a nv.s so that diferent behaviour is possible on sets of measure zero. Linearity holds, for example, since %real'
hFIN)
EaX+
=
aExt T) +
:FIFIT),
(10.33)
a.s.,
is a direct consequence of the detinition in (10.18).The followipg are conditional versions of various results in Chapters 4 and 9. The first extends 4.5 and 4.12. 10.13 Lemma V) 0 a.s. (i) If X 0 a.s., then F(aYI (ii) If X f F a.s., then F(.Yl T) S F(Fl V) a.s. (iii) If X F a.s., then F(aYlN) F(FIV) a.s. =
=
=
=
Proof (i) follows diretly from (10.18).To prove (10. 18) and 4.8(i) together imply jaExk
for a11G e V. SinceA
N)##
=
(ii), note that the hypothesis,
jaxdp hk'dp S
=
JgF(Fl
G4dP
T)((t))> 6(FIT)((t))1e V, itfollows f: F(xYI
that #(A) = 0. The proof of (iii) uses 4.8(ii), and is otherwise identical to that of (ii). w
10.14 Conditional
=
IF(FIT) I f
modulus inequality
E Il'l IT) a.s.
Proof Note that IFI F+ + Y-. where F+ and Y- are defined in (10.28).These are non-negative nv.s so that F(F+l N) > 0 a.s. and EY- j N) 0 a.s. by 10-13(i) 1, and (ii). For ) e C with PC4 =
=
lF(F+
-
Y- lN)()1
=
IF(F+l T)()
-
< F(l'+IT)((o) + =
Ev+w y-
/(F-1 97)())1 EY-
I )(),
where both the equalities are by linearity.
.
lN)() (10.34)
Probability
10.15 Conditional monotone convergence theorem lf L f F and L 1- 1' a.s., then F(FnI5') 1' F(1'1 N) a.s. Proof Consider the monotone sequence Zn L K Since Zn f 0 and Zn S Zn+I, 10.13 implies that 4he sequence Ezn lN) ) is negative and non-decreasing a.s., and hence converges a.s. By Fatou's Lernma, =
-
ja limsup Ezn N) dP limsup jaE(Zn U4dP k l I n'-/oo
n'->
= limsup
Zndp
=
G
N-->=
(10.35)
0
for G e V, the tirst equality being by (10.18), and the second by regular monotone convergence. Choose G = f : limsupaAzn lT)() < 0J which is in N by 3.26, and (10.35) implies that #4G) 0. It follows that ,
=
V) lim F(ZnI
0, a.s.
=
(10.36)
.
n'-co
10-16 Conditional
Fatou's
liminf
lemma If Fn k 0 a.s. then
&FnIV) k
E liminf L IN a.s.
influnrk so that l%' is non-decreasing, and converges to F = EY' IW) by 10.15. F,, k Fr, and hence E liminfarr,. Then E'n' 5') l5>)1 E'n' lT) a.s. by 10.13(ii). The theorem follows on letting n . Proof Put
L'
(10.37)
n-'+oo
n-oo
=
--+
--)
,.
Extending the various other corollaries, such as the dominated convergence theorem, follows the pattern of the last results, and is left to the reader.
10.17 Conditional
Markov
inequality
#(1IFl
P E.l IT) S
F(l
rlpl T) EP
,
a.s.
Proof By Corollary 9.10 we have
s'jolfIrlkcll' By definition, for G e N,
jopf I1'l
f
Jgl
Y'PdP, G e @.
k El I*4dP
and jaEf
FIPIUtdp
=
=
Jclt
ll'lk.:)tf#,
Jcl
YPdP.
Substituting (10.39)and (10.40)into (10.38),it follows that
(10.38)
(10.39)
(10.40)
Conditioning JcEEf#(t
l1'I 2
El
153
I5') -F(I F1#1U4dp
(10.41)
f 0.
The contents of the square brackets in (10.41)is a V-measurable r.v. Let G e N 0. . denote the set on which it is positive, and it is clear that #(G) =
10.18 Conditional Jensen's inequality Let a Borel function () be convex on an interval 1 containing the support of a i-measurable r.v. F where F and (/(J') are integrable. Then
4'tftrlT))
< F@tl'')l
Proof The proof applies 9.13. Setting x
N), a.s.
(10.42)
F(l'IV) and
=
y
J' in
=
T)) < :(F) -#(F(1'IN)).
A(F(1'l V))(F-F(Fl
(9.31), (10.43)
V)) is a random variable. It is not certain that However, unlike A(F(D),A(F(FI the left-hand side of (10.43)is integrable, so the proof cannot proceed exactly like that of 9.12. The extra t'rick is to replace F by 1sF, where E ftl): F(FIN)() f #1 for B < x. F(Fl V) and hence also ls are V-measurable random variables, so F(1sFIT) = 1sF(l'IV) by 10.10, and =
N) e'@(1sl')I
=
= Thus, instead of
F(9(F)ls+ lsfttF)l
#(0)1rIT) V)
(1
+
-
1s)#(0).
(10.44)
(10.43),consider
A(F(ls1'IT))(1sF-
lsF(l'l V)) k #(1sF) -#(1sF(l'l
T)).
(10.45)
The majorant side of (10.45)is integrable given that (/(i') is integrable, and hence so is the minorant side. Application of 10.9 and 10.10 establishes that the conditional expectation of the latter term is zero almost surely, so with (10.44) we get
#(1sF(l'IV)) Finally, 1et #
--
x
< 1sF@(1?)l V)+
so that ls
-->
(1
-
(10.46)
1s)9(0), a.s.
1 to complete the proof.
.
The following is a simple application of the last result which will have use
subsequently.
10.19 Theorem Let X be V-measurable and fv-bounded for r k 1. If F is F-measurable, X+ F is also Lr-bounded, and F(Fl V) 0 a.s., then =
FIxY+ FIr k
FI-YIr.
(10.47)
Proof Take expectations and apply the LlE to the inequality F(1-Y+ FIr! N) 1
IEX+
irj N)Ir
=
lx1r
a.s.
.
(10.48)
Finally, we can generalize the results of j9.6. It will suffice to illustrate with the case of differentiation under the conditional expectation.
154
Probability
10.20 Theorem Let a function G(,0)
e'(-J I I
r)
dedl 9'' -
:-o
E
-
G(%)
hv
T
I
8=00
(v, v e
Proof Take a countable sequence of the conditional xpectation, G(0() + h
satisfy the conditions of 9.31. Then
F(G(%
(Nl with hv +
v)
=
lV)
(10.49)
-.s.
,
-
--
0 as v
x.
--
(G(%) IT)
By linearity
(10.50)
a.s.
hv
If Cv 6 V is the set on which the equality in (10.50)holds, with #(Cv) 1, the #(OvCv) a nd limit 1 by 3.6. The in the two sequences agree on the set Ovcv, left-hand side of (10.50)converges a.s. to the left-hand side of (10.49)by assumption, applying the conditional version of the dominated convergence theorem. Since whenever it exists the a.s. limit of the right-hand side of (10.50) is the right-hand side of (10.49)by definition, the theorem follows. . =
=
10.5 Relationships between Subfields TI
5 and Mzc 5 are independent subfields if, for every pair of events Gj e V1 and Gz s N2, #(Gj r7 G
=
#(G1)#(Gc).
(10.51)
Note that if F is measurable on Vl it is also measurable on any collection containing V1,and on ; in particular. Theorems 10.10 and lo-llcover cases where F as well as X is measurable on a subfietd.
10.21 Theorem Random variables X and F are independent iff ctm and ctF) are independent. Proof Under the inverse mapping in (10.13),G1 e c(m if and only if #1 X(G1) i T with a corresponding condition for c(F). It follows that (10.51)holds for each G1 e c(m, G1 G c(F) iff PX* Bj, i'e Bz4 PX e #1)#(Fe Bz) for eachfl X(G1), Bz F(G2). The only ifo of the theorem then follows directly from the follows, given (8.38),from the fact that every Bi e definition of ctm. The B has an inverse image in any subfield on which a r.v. is measurable. w =
=
=
=
'if'
The
only ifo in the first line of this proof is essential. Independence of the subtields always implies independence of X and F, but the converse holds only for the infimal cases, c(m and f5(1').
10.22 Theorem Let F be integrable and measurable on V1. Then 5'I a.s. for a11 V independent of
F(FIV)
=
EY)
.
Proof Define the simple Vl-measurable r.v.s l'(n) G11,...,G1s of ( where G1j e V1, each i, with l%)
On a partition = Z'lzzlyflclf 1 F as in 3.28. Then
Conditioning
jynlb
5')J#
n
X'faFtltyl/l
=
f=1
=
*'
vl,
%''
#(G)XX#(G1f)
-
155
ynyjacjf
c)
ra
=1
V,
all G e
fOr
PG4EYn)4
=
(10.52)
f=1
F(i') by the monotone convergence theorem. F(F(n) IN) is not a simple function, but EYn) IV) F(Fl V) a.s. by 10.15, and
Ejnj)
--
'1'
joEn)
IVV# --
VV#' jcfk''
(10.53)
convergence. Hence for Vl-measurable F,
by regular monotone
JgFIFIM4dP
=
for a1l G e V.
PGjEY)
(10.54)
Frop the second equality of (10.18) it follows that F(FIG) which proves the theorem. w
10.23 Corollary
If X and F are independent, the
Proof Direct from 10.21 and 10.22, putting V
=
&FI
EY4 for a11 G e V,
=
.)
=
E').
ctm and V1
=
c(F).
.
10.24 Theorem Apairof c-fields V1cF and Vac 5 areindependentiff Cov(X,D 0 for every pair of integrable r.v.s X and F such that X is measurable on V1 and F is measurable on N2. =
Proof By 10.22, independence implies the condition of 10.11 is satisfied for N tif' consider X ito To prove lcj, G1 e V1, and F 1cz for V1, proving G2 e N2. X is Nl-measurable and F is Na-measurable. For this case, =
Cov(X,D
=
Cov(X,D =
=
,
.
#(GI
fa
=
G2) #(Gl)#(G2).
(10.55)
-
0 for every such pair implies V1and V2 are independent by (10.51).w
10.25 Corollary Random variables Xand F are independent iff Cov@(m,v(F)) 0 for every pair of integrable Borel functions 4)and v.
=
Proof By 10.3(i), 9(A') is measurable with respect to c(m for al1 and v(F) is ctpimeasurable for a1l v. If and only if al1 these pairs are uncorrelated, it follows by 10.24 that c(A3. and cti') are independent subfields. The result then follows by 10.21. w ,
An alternative proof of the necessity part is given in 9.17. The next result generalizes the law of iterated expectations to subfields. We say that c-fields V1 and V2 are nested if V1 c V2.
10.26 Theorem lf N1 i Va @, then for T-measurable F, N2)1N11 F(l'l N1) a.s. (i) EEF(l'I N1)1T21 F(l'IT1) a.s. (ii)FEe-(1'l =
=
Probability
156 Proof By definition, jLELECY'
V2)IUjjdp
But, since G e T1 implies G jcyE''
J(JIFI 5'2)## for
=
6
V2)1T11##
al1 G e N1.
(10.56)
N2, (10.18)and (10.56)imply that =
jo-dp
all G e T1,
fOr
(10.57)
T2) IV1 is a version of F(Fl N1), proving (i). Part (ii) is by so that A'(F(Fl NI) is a Vc-measurable r.v. w sinceF(FI
10.9,
lf A simple application of the theorem is to a three-variable disibution. 3 (X(),l'(),Z())) is a random point in R measura ble on T, 1etc(Z) and c(F,@ be infimal c-fields on which Z and (F,@ respectively are measurable, and c(/ the c(F,@ 5. Unifying notation by writing F(l'Iz) = F(FIc(z)) and F(rlmz) = F(FIc4-Y,/), 10.26 implies that ,
F(e'(i'I-Y,z)IZ1
=
F(A'(FIz)I-X-,Z)4
=
F(1'1z).
(10.58)
Our final results derive from the conditional Jensen inequality.
10.27 Theorem Let F be a F-measurable r.v. and T1 i Uz Proof
T. lf
9(.)is convex,
F(t(F(FI5-2))) 2 F(9(e'(rI971))(). Applying 10.18 to the Tz-measurable r.v. F(FIW2)gives F@(f(FIT2))1V1) k #(F(F(l'I91)1N1)) #(F(Fl V1)) a.s.
(10.59) (10.60)
=
where the a.s. equality is by 10.26(i). The theorem follows on taking unconditional expectations and using the LlE. . The application of interest here is the comparison of absolute moments. Since lxlP 1, the absolute moments of F(FIT2)exceed thoseof F(FIT1) when is convexforr N1 i N2. ln particular,
(10.61) FgF(rIT2)21 k FgF(rIN1)2j. SinceFti'l NllandFtFl Vzlbothhavemean ofE, T2))k (10.61)impliesVar(F(FI EYl) for i 1 or 2, (the T1)). Also, F(F(FIVj)2)+F((F- F(FIVj))2) Var(F(FI =
=
expected cross-product vanishes by 10.10), so that an equivalent inequality is Fgtr-e'(J'l N2))21 J Fg(r-F(J'j T1))2j. (10.62) The intepretation is simple. V1represents a smaller infonnation set than V2, and if one predictor is based on more information than another, it exhibits more variation and the prediction error accordingly less variation. The exeme cases ;) are &FI = Fand F(FIT) = F(1'), with variances of Vartp and zero respectively, This generalizes a fundamental inequality, that a variance is non-negative, to the partial infonnation case. While (10.61)generalizes from the square to any convex function, (10.62)does
Conditioning
157
not. However, there is the following nonn inequality for prediction errors. 10.28 Theorem If F is T-measurable and N1 i N2 i ;,
IIF-FIFIN2)11,K 2I1F-F(F1Tl)IIp, p k 1. Proof Let
n
n -
F-
=
F(FIV1). Then
F(n lN2)
=
(10.63)
by 10.26(ii),
F- F(r1 N1)-&F1 T2) +&&FIV1)1T2)
= J'-s(rI va).
(10.64)
The theorem now follows, since
IIq F(q IT2)l1p< IIq11,+11F(nIT:2,)11,f 2IIq11p
(10.65)
-
by, respectively, L1E.
the (Minkowsk.i and conditional Jensen inequalities, and the
.
10.6 Conditional Distributions The conditional probability of an eventA e # can evidently be defined as #(4 IN) F(lx 1N), where 1x() is the indicator function of A. But is it therefore meaningful to speak of a conditional distribution on (D,@), which assigns probabilities PA 1N) to each A e @? nere are two ways to approach this question. First, we can observe straightforwardly that conditional probabilities satisfy the axioms of probability except on sets of probability 0 and, in this sense, satisfactorily mimic the properties of true probabilities, just as was found for the expectations. Thus, we have the following. =
10.29 Theorem (i) P(A IN) k 0, al1 A e F. (ii) Pf IN) 1 a.s. (iii) For a countable collection of disjoint sets Aj e @, =
#
UAjIT
=
X#(4jI 9') a.s.
(i),suppose 3 G e N with
Proof To prove
(10.18),
jaudp
=
PA IV)())
Jg#(4 I@4dP <
(10.66) 0 for a11(t) e G. Then, by
<
(10.67)
0,
which is a contradiction, since the left-hand member is a probability. To prove (ii), note that #(f1 lV) is N-measurable and 1et G+ e N denote the set of (l) such G+, that #(fl V)4)) > 1. Suppose #46*) > 0. Then since G+ rn =
#(&*)
=
josdp <
jo.#(QIT)##
=
jawezdp =
#(G+),
(10.68)
Probability
158
which is a contradiction. Hence, #(G+) 0. Repeating the argument for a set G- on 0. For (iii), (10.18)gives, for any which #4f1 I5')() < 1 shows that #(G-) G 6 V, =
=
J(,#(U,A'
jIU4dP
(10.69) Xyjocvbdp' jocvjpdp jvjacpdp =
=
=
since the sets G /'7 Aj are disjoint if this is tl'ue of the Aj. By definition there exists a version of #(l N) such that V G 6 5',
jocujdp Js#(X/IV) dP,
(10.70)
=
and hence
jop-v' /1 T)#/'
=
XJSPIA/IU4dP
=
./
V)) (Xy #(X/I dP.
jo
(10.71)
The left- and right-hand members of (10.71)define the snme measure on W(see 10.5)and hence #(U.?A.j lN) = Q#(AjIT) a.s. by the Radon-Nikodym theorem. w But there is also a more exacting criterion which we should consider. That is, does there exist, for fixed (z), a p.m. g,r,lon (f,F) which satisfies yu(A)
P(A l534)), each A e 5
=
(10.72)
E G where #(C) 1? lf this condition holds, the fact that conditional for all expectations and probabilities behave like regular expectations and probabilities requires no separate proof, since the properties hold for jt(o. lf a family of p.m.s lg,(o, e fJ satisfying (10.72)does exist, it is said to detine a regular conditional probability on V. However, the existence of regular conditioning is not guaranteed in every case, and counter-examples have been constructed (see e.g. Doob 19532 623-4). The = problem is this. ln (10.66),there is allowed to exist for a given collection (A e TJ an exceptional set, say Ca with PCa) 0, on which the equality fails. This in itself does not violate (10.72),but the set Ca is specific to and uncountable countable subsets number of there typically 5, we since are an Poaj would required both be for 0, be that guarantee to a p.m. cannot as g,fo and to satisfy (10.72). =
.4
=
.d,
.4
=
This is not a particularly serious problem because the existence of the family lju) has not been critical to our development of conditioning theory, but for certain purposes it is useful to know, as the next theorem shows, that p.m.s on the line do admit regular condittonal distributions. *
10.30 Theorem Given a space L1,5,P) and a subfield N c @,a random variable F has a regular conditional distribution defined by Fr/l
for
e C with #(C)
N)() =
=
j ,
lT)(), Fg(. IV)()
#((-x,y1
where
y e R, is a c.d.f. for all
(10.73) e f1.
Conditioning
159
Proof Write F(o*(y)to denote a version of #((-x,y1 I9')(). Let Mq denote the set of ) such that F' ri) > F(o*(rj)for ri, rj e :) with ri < (/ Similarly, 1et Ri denote the set of (l) on which limn-yx/trj + 1/n) # FLlril,ri e e. And tinally, 1et L denote the set of those ) for which F(l(+x) # 1 and F(o*(-x) y: 0. Then C (Uj#C is monotone and right-continuous at rn Lc is the set of on which FfLvl rn F(l(+x) with 1 and Ff)(-x) 0. For y e R let a11rational points of the line, .
-&lc
=
=
Fg'@) y e ()
=
,
Fr(. I5')()
=
F.(.y+), y e R
(,)e c,
-0
(10.74)
Otherwise,
G(X, where G is an arbitrary
c.d.f. ln view of 10.29, #(Mj) 0 for each pair i,j, PRi4 0 for each i and PL) 0. (1fneed be, work ln the completion of the space to define these probabilities.) Since this collection is countable, PC) 1, and in view of 8.4, Fy(. IV)() is a c.d.f. which satisfies (10.73),as it was required to show. . =
=
=
=
lt is straightforward, at least in principle, to generalize this argument to multivariate distributions. For # e B it is possible to write F(1s I5')(t,))
=
V)() jedkyf
(10.75)
a.s.,
and the standard lgument by way of simple functions and monotone convergence leads us full circle, to the representation
F(FI9')()
+x
J
=
-X
y#&@l
syy(y,a.,.
(10.76)
ctm, we have constructions to parallel those of j10.1. Since no restriction had to be placed on the distribution to obtain this result, we have evidently found a way around the difficulties associated with the earlier detinitions. However, Fy(. IV)() is something of a novelty, a c.d.f. that is a random element f'rom a probability space. lntuitively, we must attempt to understand this as representing the subjective distribution of F(@ in the mind of the observer who knows whether or not ) e Gfor each G e V. The particular case Fy(. IN)4) is the one of interest to the statistical modeller wheh the outcome ) is realized. Manyrandom variables may begeneratedfromtheelements of (D,1,#), notonly the but also variables outcome itself in the bivariate case the pair F(),X() and A11 have the quantiles of Fv@ these such as F(FIm(), to be thought I of as different aspects of the same random experiment. Let X and F be r.v.s, and N a subfield with V c Rx c(A') and V ? Rv c(F). We say that X 4nd F are independent conditional on V if If V
=
-
-
m().
=
Fxrtmyl 5')
=
Fx@l S'IFI-I.yI V) ms.
=
(10.77)
Probability
160
This condition implies, for example, that f(Xl'IV) g(.,(o) be the conditional measure such that g,(o(lX 6
(--,A1,
i'
(--,A'1 1)
6
=
=
F(XIV)F(FIV) a.s.
Fxvx'y
Let ju
=
1V)().
with (t) fixed this is a regular p.m. by (the bivariate generalization ofl 10.30, and gwotAfa #) g(o(A)g(o(#) for eachA e Rx and B e Xy, by 10.21. ln this sense, the subtields Rx and Ry can be called conditionally independent. =
10.31 Theorem.
lf X and F are independent conditional on V, then
f(l'1 Rx4
F(1'l N) a.s.
=
(10.78)
Proof By independence of Rx and Rv under JxftFl
XaYll/t
=
JxFtfltto
=
u we can write BtX)JFtV' X e Rx.
(10.79)
This is equivalent to
&1z&F1Jx)l 9')(t,)) F(lxl'l 5')() =
T) IT)() = F(1xF(1'I
were tlae tirst equality also with respect to #, noting f
JxF(FIRdp
=
a.s.(#1,
(10.80)
follows from 10.26(i) and 10.10. lntegrating over f e V, using 4.8(ii) and the LIE, we arrive at JxF(Fl U4dP. A e Rx.
jgk'dp =
This shows F(Fl Rx) is a version of
F(FIN), competing
(10.81)
the proof.
F(FIRx) is in principle
.
Xvmeasurable, it is in fact almost surely I#J equal to a V-measurable r.v. Needless to say, the whole argumentis symmetric in X and Rp The idea we are capturing here is that, to an observer who possesses the information in V (knowswhether (t) e G for each G e V), observing Xdoes not yield any additional information that improves his prediction of F, and vice versa. This need not be true for an observer who does not possess prior information. Equation (10.77) shows that the predictors of F based on the smaller and larger information EY4 a.s., so sets are the same a.s.(#), although this does not imply F(Fl R that X apd F are not independent in the ordinary sense. Thus, while
=
11 Charqcteristic Functions
11.1 The Distribution of Sums of Random Variables Let a pair of independent r.v.s X and F have marginal c.d.f.s Fx@) and Fy@). The c.d.f. of the sum J'P X+ F is given by the convolution of Fx and Fs the =
function
Fx*Fy(w)
+x J..Fx(w
=
(11.1)
-ytdFv(9.
11.1 Theorem If r.v.s X and F are independent, then Fx*Fy(w)
#(X+ F f w)
=
FI'*Fx(w).
=
(1 1.2)
Proof Let 1w(my) be the indicator function of the set fmy:x K w y), so that F(1w(X,F)). By independence Fxty) FxxqFvy), so this is #(X+ F < w) -
=
=
Js21w@,A')F@,A')
+x
P+x
J..J
dFyIyl
.x1w@,.')tFx(I)
=
+x
w-y J..JFx@)
J..
=
dFyvl
+x
Fxtw
=
-
y)#Fy(y),
(1 1.3)
-X
where the first equality is by Fubini's theorem. This establishes the first equality in (11.2). Reversing the roles of X and i' in (11.3) establishes the second. w For continuous distributions, the convolution
f
=
fx*fyof
p.d.f.s
fx and h
is
+x
j-.fxw -').fy(')*,
f(w)
=
such that
Jlx/'ttlx
=
(1 1.4)
F(w).
11.2 Example Let X and Fbe independent drawings from the uniform distribu1m,1j@).Applying (11.4)gives tion on 0,11, so that fxx) =
/k+r(w)
=
1
llw-l,wj#y.
()
that the graph of this function forms an isosceles triangle lt is with base (0,21and height 1. n easily veritied
Probability
162
This is the most direct result on the distribution of sums, but the formulae generated by applying the rule recursively are not easy to handle, and other approaches are preferred. The moment generatinghlnction (m.g.f.)of X, when it exists, is jedFx), Eeth Mxtf) t e R, (11.6) =
=
where e denotes the base of natural logarithms. (Integrals are taken over (-x,+x) unless otherwise indicated.) If X and i' are independent, Mx-vvt)
jjetx*'gdt-xx4dll-y,
=
=
jetxdnxjetydFyy)
= Mx(f)Mg(/).
(11.7)
This suggests a simple approach to analysing the distribution of independent sums. The difficulty is that the method is not universal, since the m.g.f. is not defined for every distribution. Considering the series expansion of e all the moments of X must evidently exist. The solution to this problem is to replace the variable f by it, where i is the imaginary number, VX. The characteristic function (ch.f.)of X is defined as ,
9x(f) =
1l
.2
Ee
itX
)
itx
(1 1.8)
e dFx).
=
Complex Numbers
A complex number is z a + ib, where a and b are real numbers and i V-X.a and b are called the real and imaginary parts of the number, denoted a = Re(z) and b lm(z). The complex conjugate of z is the number i' a ib. Complex adthmetic is mainly a matter of carrying i as an algebraic unknown, and replacing 2 j)y j /3 i 4 by 1 e tc wherever these appear in an expression. by One can represent z as a point in the plane with Cartesian coordinates a and b. The modulus or absolute value of z is its Euclidean distance from the origin, =
=
=
=
-
-
,
-,
,
.,
lz l
=
(zD 1/2
=
a
2
+
jp
1/2
)
(1j
.
.
9)
Polar coordinaies can also be used. Let the complex exponential be defined by
efo = cos 0
+ jsin
8
(11.10)
for real 0. All the usual properties of the exponential function, such as multiplication by summing exponents (according to the nzles of complex arithmetic) go through under this definition, and
Ie i l
=
+ sin2e)1/2 (coszo
=
1
(11.11)
Characteristic
Functions
163
for any 0, by a standard trigonometric identity. We may therefore write z Iz Iei where Retz) Iz Icos 0 and 1m(z) Iz Isin 0. Also note, by (11.11),that =
=
=
.Re(z)+Im(z) j Ie z 1 j =
@Re(z)
jef1m(z)j
=
=
@Re(z)
(j.j. jg;
.
.
random If X and i' are real random variables, Z X+ iY is a complex-valued variable. lts distribution is defined in the obvious way, in terms of a bivariate c.d.f., F@,y). In particular, =
EZ)
(11.13)
F(A') + iElY).
=
Whereas EZ4 is a complex variable, FIZI is of course real, and since 1zl < IXl + Il'I by the triangle inequality, integrability of X and F is suftkient for the integrability of Z. Many of the standard properties of expectations extend to the complex case in a straightforward way. One result needing proof, however, is the generalization of the modulus inequality.
11.3 Theorem
If Z is a complex random variable,
Proot Consider a complex-valued
4F(Z) I S FI I .
.
simple nv.
(a./+ lllsy, zn) 7'j=L -
where the xj and I are real non-negative constants and the Ej e 5 for j constitute a partition of f1. Write Pj F(1sy). Then
(11. 14) =
1,...,p
=
2
IEznl) 1
-
2+
77xjpj
=
2
77fjjpj
+ f5,5/,7? 77(a??+ 7777(%a,+ fbbrjn,
( 11.15)
j*k
j
whereas CEIz(n)I)2
=
=
1/2
>7(a,?+ p,?)Pj
2
./ + + jk2)1/2# j pk. X (( + f5?)#? X X (%?+ Y?)'/2(ak2 J
J
J
j*k
./
(11.16)
The modulus inequality holds for Zn) if
0S =
(E IZ(n) I)2
-
2
IF(z(n)) I
p2,)1/2(%a,+ 1%)1#./#,. 7777E(a?+p,?)''2(a,2+ -
J
(1 1.17)
The coefficients of Pjpk in this expression are the differences of pairs of nonnegative terms, and these differences are non-negative if and only if the differences of the squares are non-negative. But as required,
Probability
164
+ p2,) (%a.+ Ip.)2 (a,?+ If)(a2k -
,
a,?p2,+ a%?
=
=
zwakjf'lk
-
(ay;'J. a.I$)2z O
(11 18)
-
.
.
This result extends to any complex r.v. having non-negative real and imaginary Z X+ iY, using 3.28, and invoking the parts by letting Z(n) = Xn) + in) monotone convergence theorem. To extend to general integrable nv.s, splitXand F X* + iY* into positive and negative parts, so that Z = Z+ r, where Z+ X+ 1'+ A?'-+ iT, with X- k 0 and lr- k 0. Noting that 2 0 and k 0, and Zwith '1k
=
=
-
=
IEZ) I < IE(Z+ + r) l < e'lz++ r I FIzl
(11.19)
=
completes the proof.
.
11.3 The Theory of Characteristic Functions We are now equipped to study some of the properties of the characteristic function x(/). The fact that it is detined for any diskibution follows from the fact 1 and Eeth is finite regardless of the 1 for all x; E( Icfl'fl ) that Ie x I distribution of X. The real and imaginary parts of #x(J) are respectively Ftcos t.k and fsin tX4. =
=
k< 11.4 Theorem If E IA-I
x,
then
k
d px(/)
d
=
Eiuk
k itx
e
).
(1 1.20)
Proof
t-bht #x(J) = h
eixt'ht
+=
-
eitx
-
#(x),
h
-x
(1 1.21)
where, using (11.10), e
t-h)
h
-
e
itx
=
cOs
xt
+
h4 h
-
cOs
tx
+
j is n
.7
(/ + 4 -
siu tx .
h
The limits of the real and imaginary terms in this expression as h sin tx and i(x cos /x), so the limit if the integrand in respectively -->
-x
is
xi cos tx
-
sin fx)
=
fccos tx
+
i sin tx)
=
ie
0 are
(11.21)
it: .
since1tfxlefr-lIxl
the integral exists if A'C-Y'I This proves (11.20) for complete 1. To the case k the proof, the same argument can be applied itx inductively to the integrands (ix)k-1e j or k z a s =
,
=
.
,
,...
It follows that the integer moments of the distribution repeated differentiation with respect to t.
can be obtained by
Characteristic lf Fl#! k <
11.5 Corollary
Functions
165
th en
x,
24x(f)
k k = i EX ). n
dtkx
(1 1.22)
An alternative way to approach the last result is to construct a series expansion of the ch.f. with remainder, using Taylor's theorem. Tlzis gives rise to a very useful approximation theorem.
11-6 Theorem lf
FI.Y1k <
then
x,
k
k 77tffytil (1)x(/)j=0 j! Proof A function
,
(1 1.23)
.
differentiable k times has the expansion
which is
f
+1
l I/XI IfXl k! (k+ 1)!
f E min
-
./*(a/)
1 Jz''(0), 3 z'(0) , +
f(f) f(0) + f '(o), + =
where 0 f
; 1. The expansion
(x
of
#
e itx where yk
=
iksgntfx)
k
jwx
(e
ft)
e
=
+
...
:!
p,
)
it?
glves
#
'
itxf
y'l y!
=
+
6
z
+
Itx I yk,
(1 1.24)
k!
j; ayjtj
-
1
sgnt/x
tx k 0
,
=
.
- 1 tx
<
,
0
Applying (11 10) and (11 11), we can show that Iyk I (2 2cos a/x)1/2 < 2. However, by extending the expansion to tenn k + 1, we also have =
.
.
k
e itx
k+1
'
itx/
=
g! .
jzzo
+
ltx I
j or () < (g s j and where zk ik+1sgnt/x) e (1 1.24) and (11.25) hold, we may conclude that ,
k
d
k
'
itxf
itx y.l ./! -
c
min
(1 1.25)
k + 1)! zk,
k+1 ift.T
=
-
2 Itx I k!
jzkj
1. Gjven that both of
=
1+l
,
Itx I
(k+
1)!
The theorem now follows on replacing x with the r.v. #in ations and using the modulus inequality:
(1 1.26)
.
(11.26),taking
expect-
Probability
166 j4e'tx./l k 9x(/)77 !
t ./
-
s
,
e
k
i
-
(/m./ j--,j y!
:7
+1
k
f E min
l I/.X'1 I/-YI k! (k+ 1)! ,
.
(11.27)
w
There is no need for FIXl k+1to ex i st for this theorem to hold, and we can think of it as giving the best approximation regardless of whether I/XI is large or small. To interpret the expectation on the right-hand side of (11.27), note that, for any pair of non-negative, measurable functions g and ,
Ftminf#tm,tml)
inf F(#(m1,4
=
+
Ae 3
tmlxc),
(11.28)
the infimal set being the one containing those points x on which particular,for any s 2 0, the set A ( IXI> EJbelongs to the class the intimum in (11.28)is taken, and we get the further inequality,
< hlx). In
.(x)
=
.
E m1n
2 Itx
k'!
Ik Itxl ,
k+1
k
k+1
2 I/aYI f E 1( jxjlsj k!
k + 1)!
over which
+
2/ E A'1k1f IxI> .:) ) k! I
E
I?-YI lt jxjssj
(k+
1) !
k+1E,k+1
+
It I
k + 1) !
(1 1.29)
S
2/ I/'I F1x'1 kt F(lxI '1. ( IxI>e) ) + 1)! (.+ k! k+1
The second alternative on the light is obtained in view of the fact that k k k+11( IxI
kax+b
=
ibt E efr(cx+h) ) = e #x(J/).
The second is the counterpart variables X and F,
of
(11.7).For
a pair of independent
(1 j ao .
random
jjeitx*'tdlkxtdFy'
#x+y(/) =
=
jeitxdFxxljeivdFvlp
=$x(f)#,'(/).
(11.31)
An interesting case of the last result is F -X' where X' is an independent drawing from the distribution of X. The disibution of X- X' is the snme as that of =
Characteristic X'-X,
Functions
167
symmetric about 0. The ch.f. of X X' is real,
and hence this r.v. is
-
since
#x(f)9x(-f) I4)xtfl I2 =
(11 32) .
,
Ee-ith in v iew o f the fact that bx#aV).It can be verified from the with real ch. f., a1l the existing odd-order moments expansion in (11.23) that a be the trademark distribution. of symmetric zero, must a =
=
:=1Xj where (X1,...,aL Considering more generally a sum S 1 are a totally recursive yields collection, of application independent (11.31) =
(1),(/)f=1 11(l)x(/).
( 11.33)
=
To investigate the distzibution of S, one need only establish the fonnulae linking the ch.f.s with the relevant c.d.f.s (or where appropriate p.d.f.s) which are known for the standard sampling distributions.
11.7 Example For the Poisson distribution (8.8),
9x(f',)
Ee
=
itX
)u
)
x
(j.
77w
--
e
=
;:
X!
e
i,
e
=
)k(eyy..j j .
(1 1.34)
n
11.8 Example In the standard Gaussian case, (8.10), 9z(J)
=
Ee
itLl
)
1 =
Completing the square yields itz
z(f)
=
e-
/2
+oo
/-2l
-
z2/2
-
1
+=
27:
x
e
itz
2
/2
-z
jf)2/2
-(z
=
dz.
-
(1 1.35) /2/2
-
and hence
-(z+ff)2 -/2
dz
e
-x
=
e
.
(1 1.36)
(The integral in the middle member has the value ulV for any choice of t, note.) Accordingly, consider X = cZ+ po whose p.d.f. is given by (8.24).Using (1 1.30), we obtain
(xlt, . poc) Equation With g,
e/w-c2p/z
.
(11.22) can be used to verify the moment =
(11.37)
.
0 the ch.f. is real, reflecting
formulae given in 9.4 and 9.7.
the symmetry of the distribution. n
11.9 Example The Cauchy distribution (8.11)has no integer moments. The ch.f. turns out to be e -lfI w hich is not differentiable at t = 0, as (11.22) would lead us to expect. The ch.f. for the Cauchy family (8.15)is #x(J;
V, : )
=
e
frv-lfl .
E1
The ch.f. is also defined for multivariate disibutions. (??zx 1) the ch.f. is
(j j ag; .
For a random vector X
Probability
168
expfif'fl),
#(f)
=
(11.39)
where t is a v-vector of arguments. This case will be especially important, not least because of the ease with which, by the generalization of (11.30), the ch.f. +.d (kx 1), can be derived for an affine transformation of the vector. Let F BX where B (kx m) and d k x 1) are constants, and then we have =
g(/)
F(exp(ff'F))
=
exp(I'/'#)F(exp(//'#.Y))
=
=
expfit'dklxlB't).
(11.40)
11.10 Example Let X m x 1) be multinormal with p.d.f. as in (8.36).
ch.f. is
rrhe
#x(f;M,E)
1
=
+=
rn/z
(2a) 1x j = exp l it'p
-
+= ...
1/2
-.
expl/'.t
g)'E-1(x
@
-
-
..
-
M)l#-4r
rlfzy.z )
(1 1.41)
.
The second equality is obtained as before by completing the square: 1 g(x it 'x j/zD ,1(x g j&)I-1(x g j&) itp g,)'E-l (x p) -
where it can be shown mI1
(2,n) Ix j
1/2
=
-
-
-
-
-
-
-
-
,
of the last term integrates over R to
that the exponential
n
.
1 1.4 The lnversion Theorem Paired with (11.8) is a unique inverse transformation hom #4/)to F@), so that the ch.f. and c.d.f. are fully equivalent representations of the distribution. The clef step in the proof of this proposition is the construction of the inverse transfonnation, as follows.
11.11 Lemma If
9(f)is defined
Fb)
-
F(J)
by
1 1im 2a F->x
=
(11.8), then yn
--
e
ittl
--
e
-
itli
(dt
it
..w
for any pair a and b of continuity points of F,' with a generalization of this formula is AF(xl
,...,.q)
=
1 m
za
1im T-+x
z'
z' ...
-w
-z
k
-jx./
e
J''I
jzzj
-
e
(11.42) <
b. The multivariate
-j.%+Ax.j)
jjy
(1 1.43) xtxj...xktt,---ntktdtb---dtk. wher AF(x1,...,.q) is defined in (8.31)and the vertices of the rectangle based at the point x1,...,ak, with sides Lxj > 0, are a11continuity points of F. n --
1.,
wxza..1
xrolon:
f.n
/1 1 q0'
Tt onn
lw
xzorifiofl
Ilqin
e (11
10
thnt
Characteristic
169
-itb
-ita
d
Functions
-d
--: b a as t
0.
-->
-
it
The integrals in (11.42)and (11.43) are therefore well defined in spite of including the point t 0. Despite this, it is necessary to avoid writing (11.42) =
as '-ita
Fb)
F(J)
-
'-itb
+x e
=
e
-
(11.44)
(f)Jf,
lnit
because the Lebesgue integral on the right may not exist. Forexample, suppose the a'it.j j random variable is degenerate at the point 0; this means that (/) e and =
F
1 21
-
e
ita
,
-itb
e
-
1 r j. 7: 1 t
log F,
--dt
dt k
it
-w
=
-
-
( 11.45)
so that the criterion for I rbesgue integrability over (-x,+x) fails. However, the limits in (11.42)and (11.43) do exist, as the proof reveals. Proof of 11.11 Only the univariate case will be proved, the multivariate extension being identical in principle. After substituting for () in (11.43) we can interchange the order of integration by 9.32, whose continuity and a.s. boundedness requirements are certainly satisfied here: '-itb
-ita
'
d
r
dt
lnit
=
= Using
'-itb
-ita
F
-d
e
e
-
-F
21f/
+x
T
-x
-F
d
oo
-x
itx-a)
j/x
e dF@) dt
itx-b)
-e
t dF(x)
lxit
.
(1 1.46)
(11.10.), ' e itx-a)
-.w
-
e
itfx-b)
t
lxit
z'Sj n yx
=
F sjn j(x
.u)
t
'lt
()
-
()
zt
.y)
t,
(1 1.47)
noting that the cosine is an even function, so that the terms containing cosines x (which are also the imaginary terms) vanish in the integral. The limit as F standard fonuula of this expression is obtained from the -..j
jo'='sin
t t
dt
=
zIl,
tx
()
(:
,
-1/2, tx
>
0
.()
<
(j j.4g) 0.
Probability
170
yields the result
(11.46)
Substituting into l.x
e
fftx-a)
il(2'-8
-d
t
lxit
=
0, x
<
a or x > b
l 7:
=
c or x
<
x < b.
x
1, a
b
=
(1 1.49)
Letting F-->x in (11.46) and applying the boundedconvergence theoremnow gives -ita
lim
F-px
T -
-itb
e
-
+x
e
(tldt
lgit
w
-x
(lfc) +
= r1(F(:)
1(s) + ta,bdFx)
F(b-)
+
Fa)
-
-
F(J-)),
(11.50)
which reduces to F(b4 Fa) when a and b are continuity points of F. w -
Lemma 11.11 is the basic ingredient of the following key result, the one that primarily justifies our interest in characteristic functions.
theorem Distributions having the same ch.f. are the same.
11.12 Inversion
Proof We give the proof for the unvariate case only. By (11.42), the c.d.f.s of the two distributions are the same at every point which is a continuity point of both c.d.f.s. Since the set of jump points of each c.d.f. is countable by 8.3 their union is countable, and it follows by 2.10 that the set of continuity points is dense in R. It then follows by 8.4 that the c.d.f.s are the same everywhere. . A simple application of the inversion theorem is to provide a proof of a well known result, that affine functions of Gaussian vectors are also Gaussian.
11.13 Example Let,f Ng,X) m x 1) and F= BX d n x 1) are constants. Then by (11.42),
-d
n
-
#g(f)
=
exptff'#lFtexpt#'sfll
= expfit'Bg If ranktS'xS)
n,
=
(implyingn -(y
.f(J)
expf
=
x 1) where B n x m) and
-
Bg
+d)
-
qt'B'SBtj
(11.51)
.
< m), 11.12 implies that F has p.d.f. -
d) '(#xn')-l(y tz I#IN
(2r)n
,
1/a
I
-Bg
-#))
.
(1 1.52)
< n, (11.51) remains valid although (11.52) is not. But by the same lf rankt#xs') arguments, every linear combination c'F, where c is p x 1, is either scalar Gaussian with variance c'BSB'c, or identically zero, corresponding to the cases B'c 0 respectively. ln this case F is said to have a singular 0 and #'c Gaussian distribution, I::I :#
=
Characteristic
Functions
171
11.5 The Conditional Characteristic Function Let F be a F-measurable r.v., and 1et V c @. The conditional ch.f. of 4)ysg(f), is for each t a random variable having the property
jgtrj
g(/)##
jaeitp,
all G e V.
=
FIV,
(11.53)
The conditional ch.f. shares the properties of the regular ch.f. whenever the theory of conditional expectations parallels that of ordinary expectations according to the results of Chapter 10. Its real and imaginary pms are, respectively, the V-measurable random variables Ftcos /F1 V) and fsin fFl N). It can be expanded as in 11.6, in tenns of the existing conditional moments. lf X is V-measurable, the conditional ch.f. of X+ F is #x+gIv(/) e itxpjeityks; by 10.10. And if i' is Vl-measurable and N and V1 are independent subfields, then =
#i'l5'(J)
l'(/)
=
a.S.
The conditional ch.f. is used to prove a useful inequality due to von Bahr and Essen (1965).We start with a technical lemma which appears obscqre at first sight, but turns out to have useful applications.
11.14 Lemma Suppose
FIZIr
<
0
cx),
<
2. Then
<
r
A'(rlj..+x1 Re(#z(/)) -
Fl W here
K
=
zl
r
=
l+rfJfl-l
(J+*(1cos ?g)/1 u I -
=
-x
(1 1.54)
t
1+r
jyj
a-1r'(r+ j) sin ra/2.
n
The last equality, with F(.) denoting the gamma function, is a standard integral formula for 0 < r < 2. Proof The identity for real z,
lz lr
+oo
=
1 cos zt dt, 1+,
Ar)J.. -
(1 1.55)
jjj
is easily obtained by a change of variable in the integral on the right. The lemma Acos tZ4. . follows on applying 9.32 and noting that Re(9z(/)) =
Tl'lis equality also holds, for ) e C with PC) = 1, if F( 1Zl rl N)((t))and 9zls(J)()) r are substituted for Fl zl and #z(f).In other words, the conditional rth moment and conditional ch.f are almost surely related by the same formula. So consider fv-bounded r.v.s z and X, where Z is T-measurable, and X is N-measurable for N c 5. Suppose that #z!v(/)is a real r.v. almost surely. Then for each e f1,
1
-
Re@x+zIs(f))()
=
1 Retd -
itX
#zIs(f))())
= 1 (cosJX())(zIg(/)()) -
S
(1
-
cos /X())
+
(1
-
9zIs(J)()),
(1 1.56)
Probability
172 thedifference between whichis non-negative
the last two members being (1 cos /X()))(1 for all Hence, for 0 < r < 2, -
#z1g(/)()))
-
.
+x 1 Re@x+zIs(/))
#(r)J.. += f #(r)J.. Ar)J.. -
E IX+
zl
rl
N)
=
t
j+r
jjj
1 cos tx
+=
-
1.r
jyj
l#l
=
'
+
t
1
zl v(/)
-
+
jyj
t
j+,
Izl r IV), a-s.
E
(1 1
.57)
and taking expectations through yields
FIA-+zIr < FIXIr
f'l
+
ZIr.
(11.58)
For the case 0 < r S 1 this inequality holds by the cr inequality for general Z and X, so it is the case 1 < r < 2 that is of special interest here. Generalizing from the remarks following (11.31), the condition that (zIv(/) be real a.s. can be fulfilled by letting Z Y- Y', where F and F' are identically distributed and independent, conditional on V. Note that if R = c(F), then =
F(F'IX)
F(F'IT),
=
(11.59)
a.s.,
by 10.31. Identical conditional distributions means simply that Fv(. 1V) Fg(. IV) a.s., and equivalently that (gIs(/) #glv(/) a.s. Hence
=
=
-fry'
#r-r'1p(/)
=
ffye
Ee
ls)
= Ee itvls )eje-itvj v) IrIs(/)l 2 a.s.,
=
(j j
,
where the right-hand side is a real r.v. Now, for each identity can be verified:
2(1
Re(#y,v(/)())
-
Applying
=
1
-
la + Il
g1s(/)()
-
(t)
-
6o)
e f1, the following
9rIs(f)((.0)
2
I
(11.61)
.
and 11.14, and taking expectations, this yields the inequality
(11.60)
2A'1FIr
r,
Fl Y-
F'I
0
<
<
r
(11.62)
2,
noting that the difference between the two sides here is the non-negative g(/) 2/j J'j 1+rJ/ Of r #(r)J+=F1 1 9y.1 l These arguments lead us to the following conclusion.
function
-
,
.
..x
11.15 Theorem Suppose F(l'IV) = 0 a.s. and X is V-measurable where V i R
c(F), and both variables
FI.Y+ F1r
=i
are Lr-bounded. Then
r s FIXI
+
2FIFIr,
0
r
(11.63)
2.
Proof Let F' be independent and identical with F, conditional on V. Applying (11.59), these conditions jointly imply f(F'I R) &F'I V) F(FI9') 0. Noting =
=
=
Characteristic
Functions
173
that X+ F is A-measurable, it follows by 10.19 (in applying this result be careful to note that R plays the role of the subfield here) that E IX+
r
FI
<
FIX+ (i'- r) I
'.
(11.64)
The conclusion for 1 < r < 2 now follows on applying (11.58)for the case Z F- i'', and then (11.62). The inequality holds for 0 < r f 1 by the cr inequality, and for r 2 from elementary considerations since Elk'.,Y'I0. In these latter cases the factor 2 in (11.63) can be omitted. . =
=
=
This result can be iterated, given a sequence of r.v.s measurable on an increasing for sequence of c-tields. An easy application is to independent rov.s A'1,...,aL, 2,...,n. 0 certainly holds for t which the condition E(Xt Ic(-Y1,...,m-1)) ntzzkxt, and 11.15 c(X1,...,Xn) yields Letting Sn c(&) =
=
=
=
E IsnIr < A'l
'n-t
Ir
+
< 2XA'ImI r,
2Fl Xn Ir
0 < r < 2.
(1 1.65)
>1
If the series on the majorant side converges, this inequality remains valid as n --> x. lt may be contrasted for tightness with the cr inequality for general Xt, (9 62). In this case, 2 must be replaced by nr-1 for 1 < r S 2, which is of no use for large n. .
lll THEORY OF
STOCHASTIC PROCESSES
12 Stochastic Processes
12.1 Basic Ideas and Terminology Let (( 5 #) be a probability space, let 'T be any set, and let RV be the product space generated by taking a copy of R for each element of 'F. Then, a stochastic process is a measurable mapping x: f r--yR'T where ,
,
j
xttDl
=
l-V((,)), ':
(12.1)
e
'T is called the index Jtrf, and the r.v. Xs() is called a coordinate of the process. A stochastic process can also be characterized as a mapping from x'T to R. However, the significant feature of the definition given is the requirement of joint measurability of the coordinates. Something more is implied than having Xs(() a measurable nv. for each 1. Here, 'Ir is an arbitrary set which in principle need not even be ordered, although linear ordering characterizes the important cases. A familiar example is T' ( 1,...,k) where x is a random k-vector. Another important case of 7 is an interval of R, such that x()) is a function of a real variable and R 'T the space of random functions. And when 'T is a countable subset of R, x fXv()), G 7-) defines a stochastic sequence. Thus, a stochastic sequence is a stochastic process whose index set is countable and linearly ordered. When the Xz represent random observations equally spaced in time, no relevant information is lost by assigning and a linear ordering through ENor Z, indicated by the notations f-V()))''*-x.The definition does not rule out 1- containing information about distances between the sequence coordinates, as when the observations are irregularly spacd in time with a real number representing elapsed time from a chosen origin, but cases of this kind will not be considered explicitly. Fnmiliarly, a time series is a time-ordered sequence of observations of (say) economic variables, although the term may extend to unobserved or hypothetical variables, such as the errors in a regression model. Time-series coordinates are labelled t. If a sample is defined as a time series of finite length n (or more generally, a collection of #uch series for different variables) it is convenient observato assume that samples are embedded in infinite sequences of tions. Various mathematical functions of sample observations, statistics or estimators, will also be well known to the reader, characteristically involving a summation of terms over the coordinates. The sample moments of a time series, regression coefficients, log-likelihood functions and their derivatives, are standard examples. By letting n take the values 1,2,3,..., these functions of n observationsgeneratewhatwemay call#crfvdtfsequences.The notionof asequence =
,
':
=
(.V((t)JO
':
dpotential'
Theor.y of Stochastic Processes
in this case comes from the idea of analysing samples of progressively increasing size. The mathematical theory often does not distinguish between the types of sequence under consideration, and some of our definitions and results apply generally, but a clue to the usual application will be given by the choice of index symbol, / or n as the case may be.
A leading case which does not fall under the detinition of a sequence is where 'T is partially ordered. When there are two dimensions to the observations, as in a panel data set having both a time dimension and a dimension over agents, x may be called a random .#e/J. Such cases afe not treated explicitly here, although in many applications one dimension is regarded as tixed and the sequence notion is adequate for asymptotic analysis. However, cases where 'T is ither the product set ; x(N, or a subset thereof, are often met below in a different context. A triangular stochastic array is a doubly-indexed collection of random variables,
X1l Xz1 X51 .Yla X22 X51
11l
(12.2)
X3k1
a=l is some increasing integer compactly written as ( (Akmlmil ) n=1, w he r f/cn )'=' sequence. Array notation is called for when the points of a sample are subjected to scale transformations or the like, depending on the complete sample. A standard Z7=1Var(X,), or some where Xnt = Xtlsn, and sn example is ( lA-n,))=1)=n=j, similar function of the sample moments f'rom 1 to n. e'D
=
12.2 Convergence of Stochastic Sequences Consider the functional expression (Ak())
';
for a random sequence on the space L1,5,P). When evaluated at a point (l) e this denotes a realization of the actual collection real is of numbers generated whenthe outcome sequence, the drawn. lt is natural to consider in the spirit of ordinal'y analysis whether this e f2, we sequepce converges to a limit, say Al. Tf this is the case for every would say that Xn Xsurely (orelementwise) where, if Xnis an T/f-measurable nv. for each n, then so is X, by 3.26. But, except by direct construction, it is usually difficult to establish in terms of a given collection of distributional properties that a stochastic sequence converges surely to a limit. A much more useful notion (becausemore easily shown) is almost sure convergence. Let C ? fl be the set of outcomes such --y
Stochastic Processes
179
.Y()) as n that, for every (t) e C, Xn) x. If #(C) 1, the sequence is said almost surely, or equivalently, with probability one. The notations Xn to converge -->
-->
=
-F.&:
X a.s., and a.s.lim-,, X are al1 used to denote almost sure convergence. A similar concept, of convergence almost everywhere (a.e.),was invoked in connection with the properties of integrals in j4.2. For many purposes, almost sure convergence can be thought of as yielding the same implications as sure convergence in probabilistic arguments. However, attaching probabilities to the convergent set is not the only way in which stochastic convergence can be understood. Associated with any stochastic sequence are various non-stochastic sequences of variables and functions describing aspects of its behaviour, moments being the obvious case. Convergence of the stochastic sequence may be defined in tenns of the ordinary convergence of an associated sequence. If the sequence fExn A'')217converges to zero, th ere is a clearly a sense in which Xn X.,this is called convergence in mean square. Or suppose that for any 6: > 0, the probabilities of the events ( ): IX,,() X()) l < :1 e 5 form a real sequence converging to 1. This is another distinct convergence concept, so-called convergence in probability. In neither case is there any obvious way to attach a probability to the convergent set; this can even be zero! These issues are studied in Part lV. Another convergence concept relates to the sequence of marginal p.m.s of the coordinates, fgwl!', or equivalently the marginal c.d.f.s, (F,,l7. Here we can consider conditions for convergence of the real sequences fgwtAlITfor various sets A e or alternatively, of (F,,(x) J7 for various x e R ln the latter case, uniform or pointwise convergence on R is a possibility, but these are relatively strong notions. It is sufficient for a theory of the limiting distribution if convergence is confined just to the continuity points of the limiting function F, or equivalently (as we shall show in Chapter 22) of gwtAl, to sets A having g,tAl = 0. This condition is referred to as the wdck convergence of the distributions, and forms the subject of Part V. X, or Xn
-->
=
-
-->
-
,
.
12.3 The Probability Model Some very important ideas are implicit in the notion of a stochastic sequence.
Given the equipotency of INand 1, it will suffic to consider the random element (A((l)))T, e f1, mapping from a point of to a point in infinite-dimensional Euclidean space, R=. From a probabilistic point of view, the entire infinite sequence corresponds to a single outcome ) of the underlying abstract probability space. ln principle, a sampling exercise in this umework is the random drawing of a point in R=, called a realization or sample path of the random sequence', we tnay actually observe only a tinite segment of this sequence, but the key idea is that a random experiment consists of drawing a complete realization. Repeated sampling means observing the same tinite segment (relativeto the origin of the index set) of different realizations, not different segments of the same
realization. The reason for this characterization of the random experiment will become clear
18O
Theot'y of Stochastic Processes
in the sequel; for the moment we will just concentrate on placing this slightly outlandish notion of an intinite-dimensioned random element into perspective. To show that there is no difficulty in establishing a correspondence between a probability space of a familiar type and a random sequence, we discuss a simple example in some detail. 12.1 Example Consider a repeated game of coin tossing, generating a random sequence of heads and tails; if the game continues for ever, it will generate a sequence of infinite length. Let 1 represent a head and 0 a tail, and we have a random sequence of ls and 0s. Such a sequence corresponds to the binary (base2) representation of a real number; according to equation (1.15) there is a one-toone correspondence between infinite sequences of coin tosses and points on the unit interval. On this basis, the fundamental space (f,T) for the coin tossing experiment can be chosen as ((0,1),S(0,l)).The form of P can be deduced in an Pttails) 0.5 (i.e.the coin elementary way from the stipulation that Ptheads) is fair) and successive tosses are independent. For example, the events ftails on first toss ) and lheads on Erst toss) are the images of the sets (0,0.5)and g0.5,1)respectively, whosemeasures must accordingly be O.seach. More generally, the probability that the first n tosses in a sequence yields a given configuration 1/2N, of heads and tails out of the ln possible ones is equal in every case to so likely'. The that each sequence is (in an appropriate limiting sense) corresponding sets in (0,1)of the binary expansions with the identical pattern of =
=
tequally
Os and ls in the tirst n positions occupy intervals al1 of width precisely $l2n in the unit interval. The conclusion is that the probability measure of any interval is equal to its width. This is nothing but Lebesgue measure on the half-open interval g0,1).n This example can be elaborated from binary sequences to sequences of real variables without too much difficulty. There is an intimate connection between infinite random sequences and continuous probability distributions on the line, and understanding one class of problem is frequently an aid to understanding the other. The question often posed about the probability of some sequence predicted in advance being realized, say an infinite run of heads or a perpetual alternation of heads and tails, is precisely answered. In either the decimal or binary expansions, all the numbers whose digit sequences either tenuinate or, beyond some point, are found to cycle perpetually through a finite sequence belong to the set of rational numbers. Since the rasionals have Lebesgue measure zero in the space of the reals, we have a proof that the probability of any such sequence occurring
is zero.
Another well-known conundrum concerns the troupe of monkeys equipped with typewriters who, it is claimed, will eventually type out the complete works of Shakespeare. We can show that this event will occur with probability 1. For the sake of argument, assume that a single monkey types into a word processor, and his Ascll-encoded output takes the form of a string of bits (binarydigits). Suppose Shakespeare's encoded complete works occupy k bits, equivalent to kl5 characters allowing for a 3z-character keyboard (upper-caseonly, but including som
Stochastic Processes
181
punctuation marks). This stling is one o f the lk p ossible strings of k bits.
Assuming that each such string is equally likely to arise in k/5 random key presses, the probability that the monkey will type Shakespeare without an error from scratch is exactly 2 -k H ow ever the probability that the second string of lk bits it produces is the right one, given that the tirst one is wrong, is (1 2 '-k)2- w h en the strings are independent. In general, the probability that the monkey will type Shakespeare correctly on the m + llth independent attempt, '&2A11 these events given that the first m attempts were failures, is (1 2 -k) and summing k disjoint, probabilities a11 their 0 yields m over are 1. fmonkey types Shakespeare eventually) ,
.
-
-
.
=
In the meantime, of course, the industrious primate has produced much of the rest of world literature, not to mention a good many telephone books. It is also advisable to estimate the length of time we are likely tcj wait for the desired text to appear, which requires a further calculation. The average waiting time, l-k4m expresse d in units of the time taken to type k bits, is 2-E,1=1rrl(1 2k 1 If we scale down our ambitions and decide to be content with just $TO BE OR NOT TO BE' (5x 18 90 bits), and the monkey takes 1 minute over each attempt, we shall wait on average 2.3 x 1021years. s o the Complete Works don't really bear thinking about. What we have shown is that almost every infinite string of bits contains every t'inite string somewhere in its length', but also, that the mathematical concept of talmost surely' has no difficulty in coinciding with an everyday notion The example is frivolous, but it is useful to be indistinguishable from reminded occasionally that limit theory deals in large numbers. A sense of perspective is always desirable in evaluating the claims of the theol'y. =
-
-
.
=
'never'
.
The first technical challenge we face in the theory of stochastic processes is to handle distributions on R=. To construct the Borel field 000of events on R7 we implicitly endow R= with the Tychonoff, or product, topology. It is not essential to have absorbed the theory of j6.5 to make sense of the discussion that follows, but it may help to glance at Example 6.18 to see what this assumption implies. Given a process x f.il7 we shall write jz 7:k(x) (m,...,A): Rx F-> (jyk =
(
=
.
a)
for each k e EN,to denote the k-dimensional coordinate projection. Let C denote the collection of jnite-dimensional cylinder sets of R=, the sets C
=
(x e
In other words, elements
: ;u@) R oo
e E, E e Bk k e ,
s)
(124) .
of C have the fonn C
=
-1 (F) C:k
(1z 5) .
for some E e T k an d some finite k. Although we may wish to consider arbitrary finite dimensional cylinders, there is no loss of generality in considering the projections onto just the first k coordinates. Any tinite dimensional cylinder can ,
182
Theor.y of Stochastic Processes
be embedded in a cylinder of the form lk -1(F), E e f where k is just the largest of the restricted coordinates. The distinguishing feature of an element of C is that at most a tinite number of its coordinates are restricted. ,
C is a tield.
12.2 Theorem
Proof First, the complement in R= of a set C defined by (12.4)is a-1(Ff) C c = fx e R= : (ak(x)e Ec E e fl k =
,
,
(126) .
which is another element of C, i.e. Cc s C. Second, consider the union of sets C a-1(F') -1 k e C for E, E' G Bk C tp C' is given by (12.4) = Jck E) e C an d C' with E replaced by E tp E ' and hence Ck.l C' e C Tlzird if E e Bk and E' e Bmfor m > k, then Ex? m-ke Bm an d so the argument of the second case applies. . =
,
,
.
.
,
,
Fig. 12.1 It is not easy to imagine sets in arbitrary numbers of dimensions, but good visual intuition is provided by thinking about one-dimensional cylinders in R3 Letting @,y,z)denote the coordinate directions, the one-dimensional cylinder generated by an interval of the axis is a region of 3-space bounded by two infinite planes at right angles to the x axis (see Fig. 12. 1 for a cut-away representation). A union of x-cylinders is another x-cylinder, a collection of But the union and intersection of an x-cylinder with parallel y-cylinder are two-dimensional cylinder sets, a cross' and a column' respectively (see Fig. 12.2). These examples show that the collection of cylinder sets in R k for fixed k is in R3 is a not a field,' the intersection of three mutually orthogonal bounded not a cylinder set. The set of tinite-dimensionalcylinders is nbt closed under the operations of union and complementation (andhence intersection) except in an infinite-dimensional space. This fact is critical in considering c(C), the class obtained by adding the countable unions to C. By the lastmentioned property of unions, c(C) includes sets of the form (12.4)with k tending to infinity. Thus, we have the following theorem. .
.x
<walls'
.
twalls'
Kcube',
Stochastic Processes
12.3 Theorem c(r) ogy. I:l
=
183
f=, the Borel field of sets in R= with the Tychonoff topol-
Fig. 12.2 The condition of this result is something we can take for granted in the usual applications. Recalling that the Borel field of a space is the jmallest c-field containing the open sets, 12.3 is true by definition, since C is a sub-base for the product topology (see j6.5) and a11the open sets of R= are generated by unions and finite intersections of C-sets. To avoid explicit topological considerations, the reader may like to think of 12.3 as providing the definition of f=. One straightforward implication, since the coordinate projections are continuous mappings and hence measurable, is that, given a distribution on @=,=), finite collections of sequence coordinates can always be treated as random vectors. But, while this is obviously a condition that will need to be satistied, the real mtypractical method we have of defining distribproblem runs the other way. utions for infinite sequences is to assign probabilities to finite collections of coordinates, after the manner of j8.4. The serious question is whether this can be done in a consistent manner, so that in particular, there is one and only one p.m. on (R7S=) that corresponds to a given set of the finite-dimensional distributions. The affirmative answer to this question is the famous Kolmogorov consis'l'he
tency theorem.
12.4 The Consistency Theorem The goal is to construct a p.m. on @*,f*),and, following the approach of j3.2, the plausible first step in this direction is to assign probabilities to elements Of C Let p denote a p.m. on th space (R,f$, for k 1,2,3,.... We will say that this family of measures satisfies the consistency property if =
.
gt' )
=
g'-t'
X
R
m-k
)
(12.7)
Theory of Stochastic Processes
184
for E G Bk and al1 m > k > 0. ln other words, any k-dimensional distribution can be obtained from an v-dimensional distribution with m > k, by the usual operation of marginalization.
The consistency theorem actually generalizes to stochastic processes with uncountable index sets 1- (see 27.1) but it is sufficient for present purposes to consider the countable case. consistency theorem Suppose there exists a family of p.m.s (gkJwlich satisfy consistency condition (12.7).Then there exists a stochastic sequence (m,t e N J on a probability space @=,f=,g,)such that p is the p.m. the of coordinate functions (.L,...,Xk)'. u vector of tinite
12.4 Kolmogorov's -'r
=
The candidate
measure for
is defined for sets in C by
.x
FIC)
=
(12.8)
11,1(6-),
where C and E are related by (12.4).The problem is to show that g, is a p.m. on C. If this is the case, then, since C is a field and f= = c(C), we may appeal to the extension theorem (3.8+3.13)to establish the existence of a unique measure on (R*,%=) which agrees with g for all C e C. The theorem has a simple but important corollary. C is a determining class for (R7f=).
12.5 Corollary
E1
ln other words, if g and v are two measures on (R=,f=) and jtk finite k, then jt v.
vk for every
=
=
To prove the consistency theorem we require a technical lemma whose proof is beyond us at this stage. It is quite intuitive, however, and will be proved in a more general context as 26.23.
12.6 Lemma For every E G T an d such that pkE A') < n
0 there exists K, a compact subset of E,
>
-
.
In other words, a p.m. on the space (R Bk) has nearly all of its mass confined to a compact set; this implies in particular the proposition asserted in j8.1, that random variables are finite almost surely. ,
Proof of 12.4 We will verify that jt of (12.8)satisfies the probability axioms with respect to elements of C. When E Rk C R= so that the first two probability axioms, 7.1(a) and (b), are certainly satistied. To establish finite additivfor E e Tk E' e Tm and ity, suppose we have C-sets C gk-1(AD, an d C' zr-1(F') m m k k. If C and C' are disjoint, =
,
=
=
p,(O+ g,tC ') g( =
F) + jt m (F')
= gmE x
=
qvl-J7
=
tp E
,
pmEx R'&-$ + p,m(A'') ,
)
=
g,(c
kp
c')
,
(12.9)
where the second equality applies the consistency condition (12.7),and the third m-k and E z are disjoint if C and C z are. one uses the fact that E x R The remaining, relatively tricky, step is to extend tinite additivity to count-
Stochastic Processes
185
able additivity. This is done by proving continuity, which is an equivalent property according to 3.5. lf and only if the measure is continuous, a monotone 1 C or G' C has the property, g,(G) p,IC). sequence tCj e C ) such that CJ Since Cj Cc where g(C9 = 1 jttt7), it is sufficient to C implies consider the decreasing case. And by considering the sequence Cj C there also is no loss of generality in setting C 0, so that continuity implies g,tGl 0. To prove continuity, it is sufticient to show that if g(() 2 6: for some E > 0, for every j, then C is nonempty. lf C e C for some j k 1, then p,(G)= pkqlEj) for some set Ej G Bkct where ? /cU) ls the dimension of the cylinder Cj. By consistency, pkqlEp '1-
.
--
.1e
'l'
-
-
=
--
=
pmEj X R '''-U)
so there is no loss of generality in assuming ) for any m > 17U), kq) < < k41) < k42) We may therefore define sets F1 e LB that lU) < 1,...,./, by setting E) A) and ...
....
=
,
=
1.
=
Ei xR ko-k
is a decreasing sincefc/),t,-1
i
,
1
=
j
,...,
-
1.
(12.j())
euhsets(A1).t,-1,
sequence, so is the sequence of
for each j k 1.
Consider any tixed j. There exists, by 12-6, a compact set Kj bkkqfEj Detine the sets K1
=
E/2 y+1
&')<
-
c
Ej such that
(!g
.
.
! !)
Kix R kil-ki) e lk) by analogy with the A''1,and so define Fj
=
Bkct
(12. 12)
s)s c
( 12. 13)
O#t; e
.
'=1
Fj i E), and hence Dj ? Cj where Dj
=
lk -1
./
.
Applying 1.1(iii) and then 1.1(i), observe that Ej Fj -
=
Ej ch
./
j
K'ic
=
-
i=1
f=1
Ej K*i)
j
c
E*i -
f=1
r*f)
(12.14)
,
is decreasing. Hence where the inclusion is because the sequence tA'tl)'J.1 bkkitEj F./) f -
-
f=1
=
bklilE t, A-t,)
bkEi
-
Kit
< E/2.
(12.15)
f=1
The first inequality here is from (12.14) by finite subadditivity, which follows from finite additivity as a case of 3.3(iii). The equality applies consistency, and the secon (i inequa lity applies the summation of 2-f-1 Since Ej Fj and Fj are = disjoint and got.l g,(G)> : by assumption, it follows from (12.15) that bklh) pkqlFj) > e/2, and accordingly that Dj is nonempty. Now. we construct a Doint of C. Let Ixlij. i e (N1 denote a seauenc of ooints -
.
=
F/let??'yof Stochastic Processes
186
of R=
with
x(J) e
Dj for each j, so that
(X1U),...,Xw)U))lk(j)@U)) =
Note that for m
=
(12. 16)
Fj.
1,...,./,
1 Km, (XlU),...,Akm)U))'&(,?,)(xU))
(12.17)
=
by (12.12), where Kmis compact. Now 1et m be tixed. Our reasoning ensures that (12. 17) holds for each.j k m. A set in Rkm' is compact if and only if each of the coordinate sets in R is compact, so consider the bounded scalar scquences, fxibij, j r?z) for f 1,...,k4-). Each of these has a cluster point A'1, and we can use the diagonal method (2.36)to construct a single subsequence (Al with the = A': for each 1,...,k(-). By the compactness, property that Xijn) e Km i Em. This is true for every m e IN (X1,...,X(m)) Consider the point a7*e R= defined by 7v(,,,)(x*) for each m e N. (X1,...,X(m)) = Since x* e Cm for each m, we have x* G O'>;=ICk C, as required. . =
.-+
.
=
This theorem shows that, if a p.m. satisfying (12.7)can be assigned to the finite-dimensional distributions of a sequence m x is a random element of a We shall often wish to think of @=,f7jt) as derived probability space from an abstract probability space (f,T,#), and then we shall say that x is ble if F1(A>)e 5 for each event E e S*. This statement implies the 5lB coordinates Xt are F/f-measurable r.v.s for each /, but it also implies a great deal more than this, since it is possible to assign measures to events involving countably many sequence coordinates. ('R*,fO,g,).
''O
-measura
12.5 Uniform and Limiting Properties Much the largest part of stochastic process theory has to do with the joint distributions of sets of coordinates, under the general heading of dependence. Before getting into these topics, we shall deal in the rest of this chapter with the various issues relating exclusively to the marginal distributions of the coordinates. Of special interest are conditions that limit the random behaviour of a sequence as the index tends to infinity. The concept of a unform condition on the marginal distributions often plays a key role. Thus, a collection of rov.s tX,r, T e is said to be uniformly bounded in probability if, for any 8 > 0, there exists Bg < x such that -1)
#( IXzI sup ' It is also said to be unjrmly
)<
Lp-bounded for p
supI1ALll?,B <
<
=.
(12. 18)
:. >
0, if
(12.19)
'
For the case p = x, (12.19)reduces to the condition, supmlXzI < x a.s. ln this case we just say that the sequence is uniformly bounded a.s. For the case p = 1, we have sups f'IXz I < x and one might think it conrct to refer to this property as integrability' Unfortunately. this term is already in use for a ,
Suniform
,
Stochastic Processes
different concept (see the next section) and so must be avoided here. Speak of tuniform fal-boundedness' in this context. To interpret these conditions, recall that in mathematics a property is said to hold unifonnly if holds for all members of a class of objects, including the limits of any convergent sequences in the class. Consider the case where the /. Random collection in question is itself a sequence, with 'T = E$ and with probability 1, and for each finite t e IN,#( 1Al Be variables are finite x, > holds for for The < always point of unifonn Bet 0. : bound is some any a <s to ensure that the constants Bet are not having to get larger as t increases. p > 0, by Liapunov's inequality. Also, uniform fv-boundedness for anyp > 0 implies uniform boundedness in probability; the Markov inequality gives 'r
=
tbounded
,
-->
-x).
-->
,
#(IAI
B4
(12.20)
terminology we sometimes speak of fvboundedness in the case of (12.18). A standard shorthand (dueto Mann and Wald 1943b) for the maximum rate of (positive or negative) increase of a stochastic sequence uses the notion ot Oh' notation unifonu boundedness in probability to extend the tBig Oh' and real E lf, for exists ordinary 0, > Be < x such for there sequences (see j2.6). that the stochastic sequence fXn )T satisfies supa't 1XnI > B < E, we write Xn 0p(1). lf (L)T is another sequence, either stochastic or nonstochastic, and XnlYn = 0p(1) we say that Xn Op(L), or in words, &Xnis at most of order Fn in x, 0 as n tN(1); more we say that Xn Probability' If #( IXnI > :) t7p(1), when words, in GXn is of order lss than Xnlk'n generally Xn o (L') or r l'n in probability The main use of these notations is in manipulating small-order terms in an expression, without specifying them explicitly. Usually, Fn is a positive or negative power of n. To say that Xn tV1)is equivalent to saying that Xn converges in probability to zero, following the terminology of j12.2. Sometimes Xn Op(1) is defined by the condition that for each s > 0 there exists Be < x and an integer Ng > 1 such that #( IXnl > B < s for a1l n Nz. But Xn is finite almost surely, and there necessarily exists (this is by 12.6) a constant BL< #E') < E for 1 f n < Ne. For al1 possib1y 1arg er than Be, such that #( IXn l > practical purposes, the formulations are equivalent. Ktaittle
=
=
--)
-->
=
.
=
=
.
=
=
=,
Iheor.y of Stochastic Processes
188
12.6 Uniform lntegrability lf a r.v. X is integrable, the contributions to the integral of extreme X values must be negligible. ln other words, if FIXI f(IX1 1(IxlzM))
-->
0 as M
(12.21)
-->
x.
However, it is possible to construct unifonnly Al-bounded sequences l#nl which fail to satisfy (12.21)in the limit.
12.7 Example Detine a stochastic sequence as follows: for n 1,2,3,... let Xn = 0 with probability 1 1/rI, and Xn = n with probability 1/a. Note that E( lXn l) = nln = 1 for every n, and hence the sequence is uniformly Al-bounded. But to have =
-
limE IXnIlt lxalA/l)
=
0
(12.22)
MM=
uniformly in n requires that for each e > 0 there exists Ms such that E lXnl 1( xajkvl) < e for al1 M > Mo uniformly in a. Clearly, this condition fails, for e: < 1, in view of the cases n > Me. n Something very strange is going on in this example. Although E(Xn4= 1 for any n, x. To be precise, we may show that Xn = 0 with probability approaching 1 as n 0 (see 18.15). The intuitive concept of expectation appears to fail when Xn faced with nv.s taking values approaching intinity with probabilities approaching --+
-613...:
Zero.
The unform integrability condition rules out this type of perverse behaviour in a sequence. The collection (Xz, $ e T'J is said to be uniformly integrable if
sup 1im F( lXx11( xvl
Te
7
M)
M-yx
)
=
0.
(12.23)
ln our applications the collection in question is usually either a sequence or an array. In the latter case, uniform integrability of (Xkf) (say) is defined by taking the supremum with respect to both t and n. The following is a collection of theorems on unifonn integrability which will tind frequent application later on; 12.8 in particular provides insight into why this concept is so important, since the last example shows that the conclusion does not generally hold without uniform integrability. 12.8 Theorem Let E(Xn)
--$
E (A').
be a unifonnly l.YnlT
integrable sequence. If Xn -6J- X, then
Proof Note that E iXn I) = E IXnl 1( !xnI
s A.T+
Ftlxa
+
E lXnl 1( IxaIkM))
l1( jx,,1kM)).
(12.24)
By choosing M large enough the second tenn on the right can be made uniformly small by assgmption, and it follows that FIXnj is uniformly bounded. Fatou's
stochasticProcesses
189
lemmaimplies that FIXl < x, and F(m exists. Define l',, I.L --YI so that i',z 0 a.s. Since Fn < IaLl+ l-Yl by the triangle inequality, 9.29 gives =
f 2F( IXnl 1( lxnlxv/all
F(i'n1(yn>M))
+
2F( lXl 1( IxI>v/2)).
The second right-hand-side term goes to zero as M integrable if (A%) is. We may write f(Fn)
'(l%1(yn<M))
=
+
--
,
.--.)
so
x,
(12.25)
f Fn) is
A'(L1(yn>v)),
uniformly
(12.26)
and by the bounded convergence theorem there exists, for any E > 0, Ne such that F(l%1(ya<>)) < E/2 for n > #:, for M < oo. M can be chosen large enough that F(L1( ys>>)) < :/2 uniformly in n, so that EYn) < E for n > Nv., or, since 6: is 0. But arbitrary, EYn) --.-)
EYnq
by the
lnodulus
F(I-L--YI) k
=
IF(Xn) -F(ml
(12.27)
inequality, and the theorem follows. w
The next theorem gives an alternative form for the condition in often more convenient for establishing uniform integrability.
(12.23)which
is
12.9 Theorem A collection fXs, e 1-) of nv.s on a probability space (f1,1,#)is uniformly integrable iff it is uniformly Al-bounded and satisties the following condition: V E, > 0, 3 8 > 0 such that, for E E 5, ':
sup (F41 Xzl 1s) l < z.
P(E4 S
(12.28)
': e 'T
Proof To show sufficiency, fix : > 0 and 5 1, Markov inequality for p
E
7. By Al-boundedness and the
=
E
#(1-V1k Af) f and for M large enough, (12.28), it follows since
lxzI < M
(12.29)
x,
#(l-YzI Af) S ,1
6, for any is arbitrary that
>
0. Choosing
lA'(lX,rl 1f Ixzlwzlll < E,
Sup
to satisfy
(12.30)
TeT
and (12.23)follows since
E is arbitrary. note that, for any E e 5 and
To show necessity, F(1 Xvl 1s)
=
'r
E
1-,
1s1( Ixz1<MJ) + .E'(I.YZI 1s1( lxvI&z)) l-Y'rI
E
< MPCEI+
F(1 -Ysl1( Ixzlkvl).
(12.31)
Consider the suprema with respect to of each side of this inequality. For e > 0, (12.23) impliej there exists M < t:'o such that 'r
1F( 1AL1 1s)l
sup T
Uniform Al-oundedness
s MPCE) +
r1:.
now follows on setting F
=
f, and
(12.32) (12.28)also follows
Theor. of Stochastic Processes
190 with 8
<
zllM.
*
Another way to express condition (12.28)is to say that the measures vztp = ## must be absolutely continuous with respect to #, uniformly in :. Finally, weprove aresultwhich showswhy theuniformboundednessof moments of a given order may be important.
Jslxzl
12.10 Theorem lf 1+8
FIXzI for 0
>
0, then limv-pxlx'rl
Proof Note that
1( IxzIuv))
E IX,t11+e
(12.33)
< cxn =
0.
1+91
s (jx z j t l-vzlkxf)) S?E (IX1I1( jxvlzwzl) 2
for any 0 > 0. ne result follows on letting M (12.34) is finite by (12.33)..
..-+
x,
(12.34) since the majorant side of
Example 12.7 illustrated the fact that uniform fal-boundedness is not sufficient for unifonn integrability, but 12.10 shows that unifol'm falma-boundedness is sufficient, for any 0 > 0. Adding this result to those of j12.5, we have established the Merarchy of uniform conditions summarized in the following
theorem. 12.11 Theorem Uniform boundedness a.s.
uniform uniform = = uniform = uniform zz:
None of the reverse implications hold.
fw-boundedness, p > 1 integrability fw-boundedness, 0 < p S 1 boundedness in probability. n
13 Dependence
13.1 Shift Transformations We now consider relationships between the different members of a sequence. ln what ways, for example, might the joint distdbution of Xt and Xt-k depend upon k, and upon /? To answer questions such as these it is sometimes helpful to think aboutthe sequencein anew way. Havingintroducedthenotionof arandom sequence as a point in a probability space, there is a useful analogy between studying the relationships within a sequence and comparing different sequences, that is, different sample outcomes (l) e f. F-> copsider a 1-1 measurable mapping, F: ln a probability space (f1,@,#), (onto). This is a rule for pairing each outcome with another outcome of the space, but if each ) e f maps into an infinite sequence, Finduces a mapping from one sequence to another. F is called measure-preserving if PCTEI PE) for a11E e F. The shift transformation for a seqence (X/)JT is defined bylo =
X(F)
=
X,+1().
(13.1)
F takes each outcome ) into the outcome under which the realized value of X occurring in period t now occurs in period t- 1, for every t. In effect, each coordinate of the sequence from t 2 onwards is relabelled with the previous Xt-vk), the relationship period ' s index. More generally we can write XtlTkl between points in the sequence k periods apart becoming a characteristic of the transformation F k Since Xt is a nv. for a1l f, both the shift transformation and its inverse F -1 th e backsh;ft transformation, must be measurable. Taken together, the single r.v. X1(): f r- R and the shift transformation F, can be thoght of as generating a complete description of the sequence (X/))JT. This can be seen as follows. Given #14), apply the transformation F to ), and each ( e ( defines the mapping A%((0):f F-> R, obtain Xz = X1(F(0). Doing this and we are ready to get Xg X2(F(0). ltrating the procedure generates as many points in the sequence as we require. =
=
.
,
tor
=
be a sequence of coin tosses (with1 13.1 Example Consider 12.1, Let (Xf())JT for heads, 0 for tails) beginning 11010010001 1... (say).Somewhere on the intelwal g0,1) of real numbers (in binary representation), there is also a sequence (A()') J7 beginning 10100100011..., identical to the sequence indexed by (l) apart om the dropping of the initial digit and the backshift of the remaindr by one position. Likewise there is another sequence fXt(''4 1T,a backshifted version of ' ) 17b eg inning 0100101 1...; and so foih. If we define the transformation taVt(0
Theory of Stochastic Processes
192
)j'' F by F -1 ) = (l) r-1)' etc. the sequence (Xl(() )T can be constructed as 1-J ) )T,t h at is the sequence of first members of the sequences thesequence lm(F foundby iterating the transformation, in this case beginning 1,1,0,... I:l '
,
=
,
,
'
,
This device reveals, among other things, the complex stnlcture of the probability space we are postulating. To each point (.t) e fl there must correspond a countably infinite set of points F ) e f w hich reproduce the same sequence apart from the The intertemporal properties of a sequence can absolute date associated with then be treated as a comparison of two sequences, the original and the sequence lagged k periods. Econometricians attempt to make inferences about economic behaviour from recorded economic data. In time-series analysis, the sample available is usually a single realization of a random sequence, economic history as it actually occurred. Because we observe only one world, it is easy to make the mistake of looking on it as the whole sample space, whereas it is really only one of the many possible realizations the workings of historical chance might have generated. Indeed, in our probability model the whole economic universe is the counterpm of a single random outcome ); there is an important sense in which the time series analyst is a statistician who draws inferences from single data points ! But although a single it is realization of the sequenc can be treated as a mapping from a single linked to a countably infinite set of s corresponding to the leads and lags of the seqpence. A large part of our subsequent enquiry can be summarized as posing the question: is this set really rich enough to allow us to make inferences about # from a single realization? ,
m.
,
13.2 Independence
nnd
Stationarity
Independence and statioharity are the best-known restrictions on the behaviour of a sequence, but also the most stringent, from the point of view of describing economic time series. But while the emphasis in this book will mainly be on finding ways to relax these conditions, they remain important because of the many classic theorems in probability and limit theory which are founded on them. The degree to wlch random variations of sequnce coordinates are related to those of their neighbours in the time ordering is sometimes called the memory of a sequence; in the context of time-ordered observations, one may think in tenns of the amount of information contained in the cunrnt state of the sequence about its previous states. A sequence with no memory is a rather special kind of object, because the ordering ceases to have signiticance. lt is like the outcome of a collection of independent random experiments conducted in parallel, and indexed
arbitrarily. When a time ordering does nominally exist, we call such a sequence serially independent. Generalizing the theory of j8.6, a pair of sequences ((X,())T, l Fr() JT1e R= xR*, is independent if, for a11F1, E1 e 5*, /:()1.Y,17 e F1, (F,1: e E,4 Accordingly, a sequence
(A())T
=
#(tx,)7
e Ej4pf
r,)7 e
F2).
(13.2)
is serially independent if it is independent of
Dependence
193
fXt(T k(0) IT fOr a 11k ls 0 This is equivalent to saying that every tinite collection of sequence coordinates is totally independent. Serial independence is the simplest possible assumption about memory. Similarly, looking at the distribution of the sequence as a whole, the simplest treatment is to assume that the joint distribution of the coordinates is invariant with respect to the time index. A random sequence is called strictly stationary if Tllis implies that the sequences the shift transformation is measure-preserving. fXf 17 and fAkJ7=1have the same joint distribution, for every k > 0. Subject to the existence of particular moments, less restrictive versions of the condition are also commonly employed. Letting w E(Xt), and Xkt Covxt,xtn), consider those cases in which the sequence (g'rl7 and also the array ( fXmt are well defined. If w g,, al1 /, we say the sequence is mean J';=()l7wl, where (y,,l stationaty. If a mean stationary sequence has 17'is a sequence wt w of constants, it is called covariance stationaty, or wide sense stationary. If the marginal distriution of Xt is the same for any t, the sequence IXCJis said to be identically distributed. This concept is different from stationarity, which also restricts the joint distribution of neighbours in the sequence. However, Fhen a stochastic sequence is both serially independent and identically distributed (or i.i.d.4, this suftices for stationarity. An i.i.d. sequence is likeanarbitrarily indexedrandom sampledrawngomsomeaunderlyingpopulation. The following clutch of examples include both stationary and nonstationary .
.1
=
=
=1,
=
=
CaKS. (52 < 13.2 Example Let the sequence (6>J b e i i d with mean 0 and variance where fA)7, and let (0j17be a square-summable sequence of constants. Then ''O -x
.
.
(x,
.
*
A
=
77bet-j,
(13.3)
jcst
is a covariance stationary sequence, with Ext) t. This is the inhnite-order moving average additional details. n 13.3 Example lf
kt
=
0 and Ext)2 = c J e? ) sr every (MA(=)) process. See j14.3 for 2:.,.0
,0
is i.i.d. with mean 0, and Xt
for a constant a, then Ex
=
=
cos at
+
(13.4)
st
cos at, depending systematically on /.
I:a
13.4 Example Let (xV)be any stationary sequence with autocovariance sequence (ym, m 0J The sequence (.X)+ A%) has autocovariances given by the array .
l
'o
+
and hence it is nonstationary.
'sm
+
Yt+
'tt,vm,
m 2 0, t k 1 l
,
u
13.5 Example Let X be a r.v. which is symmetrically distributed about 0, with variance c 2 If Xt (- 1)W then (X,)Tis a stationary sequence. In particular, .
Ext)
=
=
,
0, and Cov(Xt,Xt+k4 5.2 when k is even and =
-c2
when k is odd, independent
Theory of Stochastic Processes
194 of t.
(:1
Theseexamples show that, contraryto acommon misconception, stationarity does not imply homogeneity in the appearance of a sequence, or the absence of periodic patterns. The essential feature is that any patterns in the sequence do not depend systematically on the time index. It is also important to distnguish between stationarity and limited dependence, although these notions are often closely linked. Example 13.4 is nonstationaty in view of dependence on initial conditions. The square-summability condition in 13.2 allows us to show covariance stationarity, but is actually a limitation on the long-range dependence of the process. Treatments of time series modelling which focus exclusively on models in the linear MA class often fail to distinguish between these propeties, but examples 13.3 and 13.5 demonstrate that there is no necessary connection betwen
them.
'
Stationarity is a strong assumption, particularly for the description of empirical time series, where features like seasonal patterns are commonly found. It is nonstationarity, sometling we might think of useful to distinguish between capable of elimination by local averaging of the coordinates, and global' as nonstationarity, involving features such as persistent trends in the moments. If sequences (Xr)7=1 and (a'+k)7=1have the same distribution for some (not necessarily every) k > 0, it follows that (.+x)7.1 has the same distribution as (X)+l7=1, and the same property extends to every integer multiple of k. Such a sequence accordingly has certain stationary characteristics, if we think in tenns of the distributions of successive blocks of k coordinates. This idea retains x. Consider the limit of a tinite sequence of length n, divided force even as k into (rl j blocks of length gn 1,plus any remainder, for some a between 0 and 1. here denotes the largest integer below x.) The number of blocks as well (Note, E.;r) as their extent is going to infinity, and the stationarity (or otherwise) of the sequence of blocks in the limit is clearly an issue. Important applications of these ideas arise in Parts V and VI below. It is convenient to formulate a definition embodying this concept in terms of moments. Thus, a zero-mean sequence will be said to be globally covariance stationary if the autocovariance sequences (y/uf are Cesro-summable for each )*r.1 0, where is strictly positive the k in the Cesro sum case of the variances m = m 0). The fllowing are a pair of contrasting counter-examples. ilocal'
-->
13 6 Example A sequence with
'p,
/l5
-
@
is globally nonstationary for any
j
#
0. n
13.7 Example Consider the integer sequence beginning 1,2,1,1,2,2,2,2,1,1,1,1,1,1,1,1,...*, i.e. the value changes at points t 2 k k = 1 2 3 The cesro sum of this x. It fluctuates eventually between the points sequence fails to converge as n lk k = k even. A stochastic sequence having a 5/3 at n 2 k o dd and 4/3 at n of form globally nonstationary. (:1 is variance sequence this =
,
-->
,
,
=
,
,
,
,.,.
Dependence
195
13.3 lnvariant Events The amount of dependence in a sequence is the chief factordetermining how informative a realization of given length can be about the distribution that generated it. At one extreme, the i.i.d. sequence is equivalent to a true random sample. The classical theorems of statistics can be applied to this type of distribution. At the other extreme, it ij easy to specify sequences for which a single realization can never reveal the parameters of the distribution to us, even in the limit as its length tends to intinity. This last possibility is what concerns us most, since we want to know whether averaging operations applied to sequences have useful limiting properties', whether, for example, parameters of the generation process can be consistently estimated in this way. Toclarify theseissues, imaginerepeatedsamplingof random sequences (A((t)) JT', in other words, imagine being given a function X1(.) and transfonnation F, making repeated random drawings of (.t) from fl and constructing the corresponding random sequences', 13.1 illustrates the procedure. Let the sample drawings be denoted j, 1,...,N, and imagine constructing the average of the realizations at some i ,=1 X is called an ensemble average, fixed time / e. The average kxto N-1X% o((tk) which may be contrasted with the time average of a realization of length n for n-11J=1Xj()j). Fig. 13.1 illustrates this procedure, some g iven )j e D, Xntj) showing a sample of three realizations of the sequence. The ensemble average is the average of the points falling on the vertical line labelled /c. lt is clear and the ensemble average as n and N that the limits of the time averg respectively go to infinity are not in general the same; we might expect the ensemble average to tend to the marginal expectation Ext ), but the time average 8 will not do so except in special cases. If the sequence ls nonstationary Ext depends upon /o, but even assuming stationarity, it is still possible that different realizations of the sequence depend upon random effects which are common to a1l t. =
=
,
=
O O
O
O O O
O O
O
O
Fig. 13. 1
Theory of Stochastic Processes
196
In a probability space (f1,F,#) the event E e 5 is said to be invariant under a transfonnation Tif PCTEh F) 0. The criterion for invariance is sometimes given as TE F, but allowing the two events to differ by a set of measure zero does not change anything important in the theory. The set of events in 5 that are invariant under the shift transformation is denoted #'. =
=
13.8 Theorem
,@
is a c-tield.
Proof Since F is onto,
is clearly invariant. Since F is also 1-1,
TEc A E C
CTE4cA Ec
=
(13.5)
TE A E
=
by definition. And, given 3En e #, n e IN), PCTE,tA En)
PITEn
=
&) + P(En
-
-
TEn4 = O
(13.6)
for each n, and also F
UEn
A
n
UEn UTEn A UEn,
(13.7)
=
n
l
n
using 1.2(i). By 1.1(i) and then 1.1(iii),
UTEn UEn U TEn OEL ? U(Ff n
and similarly,
f'n
=
-
n
n
-
n
&),
(13.8)
?1
n
UEn UTEn U(En n
-
n
TEn).
-
(13.9)
n
= 0 The conclusion #gT(Un&) A (Un&)1 now follows by . proof. pletingthe
(13.6)and
3.6(ii), com-
An invariant random variable is one that is J/f-measurable. An invariant nv. Z() has the property that Z(F) =Z(), and an J/f-measurable sequence (Z,()) ):' Z1()) a.s. for every t. The invariant events is trivial in the sense that Zt and associated r.v.s constitute those aspects of the probability model that do not alter with the passage of time. =
= Ff((z)) + Z(), where Xt 13.9 Example Consider the sequence (AX()))) ( F,4(0)) random and example of invariant Z((0) An being a event is E an sequence a r.v. f (.0: Z() K z, Ff() e R ). Clearly E and FF are the same event, since Z is the only thing subject to a condition. Fig. 13. 1 illustrates this case'. lf ( Ff()) ) is a zero-mean stationary sequence, the figure illustrates the cases Z((l)1) = Z((.0:) 0, and Z((l)3) > 0. Even if EZ4 out' in the limit, = 0, the influence of Z() in the time average is not as it will be from the ensemble average. I:a =
=
The behaviour of the time average as n fundamental result.
--
cxn
is summarized by the following
be 13.10 Theorem (Doob 1953: th. X.2. 1) Let a stationary sequence (.&((.t)))T defined by a measurable mapping X1()) and measure-preserving shift transform-
Dependence
ation T, such that
m()
J-1
.Y1(F
=
1imSnbln
), an d
I
Z)=1m()).If
=
F( X1j
,7)()),
ECXL
=
1et Sn ()
x,
(13.10)
(:I
a.s.
<
n-+oo
In words, the limiting case of the time average can be identitied with the mean of the distribution conditional on the c-tield of invariant events. The proof of 13.10 requires a technical lemma.
A(I3) ((t):supatkst))
13.11 Lemma Let
=
Jvca(oXl())J#()
-
nI))>
0) Then for any set M e #', .
2 F#(Mf-hA(p)).
(13.11)
Proof We establish this for the case fs 0. To generalize to any real p, consider the sequence fA p) which is stationary if (XCIis stationary. > 0), the set of outcomes Write A for A(0), and let hj f ): maxlskust) for which the partial sum is positive at least once by time j. Note, the sequence x. Also 1et A as j f A.j) is monotone and hj =
-
,
=
'1h
--
N nj
T-ih
=
-j
n
:
=
max Sklf l Kkf n
>
0
(13.12)
.
'-j
Since k
j+k
&(r?(o) XxtT) =
!F71rf(e)) sjmke)SjTb, z-
=
>1
by defining & Nnj
=
(13.13)
-
J=./m 1
0 we may also write ):
=
max j- 1f k f n
sk
-
Sjbh) > 0
,
0 f j Kn
-
1
.
(13.14)
This is the set of outcomes for which the partial sums of the coordinates f'rom j + 1 to n are positive at least once, and we have the inequality (explained below) n-1
XA+1()
lxnytl
2 0, all
)
e D.
(13.15)
Integrating this sum over the invariant set M gives n-1
n-1
r
0 X j=0 J <
u-suxjwtodrb
v
n-1
m
'0
Jn
v
R'
''nn
X y=o
j,.x.y&+1(rVt,))J#()
n
g
=X j u-g jcz0
=
.X1(9d#((9) J-
=
Jv.ayXl(t9V#(), X jc
(13.16)
1
where the tirst equa lity uses the fact that hn-j = (: T-l(kbe Nnjj and the measure-preserving property of F, and the third is by reversing the order of summation. The dominated convergence teorem applied to fm lMraayl, with I I as the
m
Theory of Stochatic
198
Processes
dominating function, yields
(13.17) Jxawxltltf#tt9l. Jsyraayxltllptl -->
limit by 2.26, so that, as required,
This limit is equal to the Cesro
-1
j'ucvs m()##()
=
1im a-,-
The inequality in
n
Xnj
m()##4)) 0. .
(13.18)
ucxxj
s,
not self-evident, but is justified as follows. The having the property expression on the left is the sum containing only those Xt realization that in the partial sums from the point t onwards are positive at least once, otherwise the tth contribution to the sum is 0. The sum includes only Xt lying in segments of the sequence over which Sk increases, so that their net contribution must be positive. It would be zero only in the case Xt S 0 for 1 f t S n. Fig. 13.2 depicts a realization. for k shows values of Sk 1,...,a, vertical shows the separations between successive so the .Yf((0)are the running sum of the terms of (13.15).The coordinates where the Xt are to be omitted from (13.15) are arrowed, the criterion being that there is no to the which exceeds the right current
(13.15)is ,
'o'
=
&o'
<+'
.
to'
'o'.
+
+
+
* *
*
0
O
*
*
Fig. 13.2 Proof of 13.10 The first step is to show that the seqpence (Snlbln ITconverges almost surely to an invariant r.v. Sl). Consider 5'() = limsupa Snlln. RF)
=
lim sup vl--hx nkm
&+l() n+ 1
n+ 1 n
X1() =
-
n
)),
(13.19)
ltl = liminfn Snln. Hence, so that Rl is invariant, the same being true for sttt)) = M(a,p) invariant. Since i s supnsnln (x < < < the event j3 V((t)ll (: A(p) M(a,p) A(p) where defined in the statement of 13.11. Hence, is V1, M(a,j) in the lemma gives putting M =
lDependence
J>(oj)X1(t0V#())2 But now replace
(
):
supnt-xsktl/n)
199
p#(M(?'p)).
(13.20)
13.11 by -AX)), and observe that M(a,p) Hence we get
in AX(t)) -a)
>
c
A(-a)
=
.
Jsy(a,j)X1(t0V#(tt9 S tx#(M(?'p))'
(13.21)
Since the left-hand sides of (13.21)and (13.20)are equal and (x < p, it follows 5'() with probability 1. that #(M(a,p)) = 0; that is, R)) 8)) This completes the first stage of the proof. It is now required to show that S = Exj I#) a.s., that is, according to equation (10.18),that =
juxltdp Sinc
=
jsjdp,
=
each M e
(13.22)
M is invariant, ''t
j
juxLlkdp =
a-
n
.,yt
X >1
vr
j.
Jyrto, ju ,
X/t0)##((t)
=
-s
Snl
:#4tl))
(13.23)
and the issue hinges on the convergence of the right-hand member of (13.24)to
E(S3u4. Since the sequence (-Y/Jis stationary and integrable, it is also uniformly integrable, and the same is true of the sequence ( F,l, where Ff Xtsu and M e #. For : > 0, it is possible by 12.9 to choose an event E e 5 with #(& < 6, such that sup E 1FJ I1e,) < e. For the same E, the triangle inequality gives =
j
E
j) j s )x-(J yn,-, >1
#,)
Ii-,I E
dp
>1
< :
(13.24)
.
By the same argument, also using stationarity and integrability of Ff,
I j
J1 n-
n
y-, dp
-,
s
Hence by 12.9 the sequence n-v I>. If n-l n n n -11 F) 12.8, by =1
=
J
complete. w
n
,.j
jl
yl
),; jap
=
s j yj j
(13.25)
tn-1Z:=1Ff)
-->
i s a 1so uniformly S a.s., it is clear that n-S
-1
Mn
Since n is arbitrary,
n
-1
:#((t))
--y
( Sudp.
2M
n
integrable, where Sku a.s., so
1M
-->
(13.26)
(13.26)and (13.23)together give (13.22),and the proof
is
13.4 Ergodicity and Mixing The property of a stationary sequence which ensures that the time average and the ensemble average have the same limit is ergodicity, which is defined in terms of
I'heory of Stochastic Processes
200
the probability of invariant events. A measure-preserving transformation F is l or #() is the c-field of 0 for a11E e Ds where ergodic if either #(F) under F. A stationary sequence (Xf() )+-7is said to be ergodic if invariant events J-1 =
x/
=
Xl (F ) for every t where F is measure-preserving and ergodic. Some authors, such ag Doob, use the tenn metrically transitive for ergodic. Events that are invariant under ergodic transformations either occur almost surely, or do not occur almost surely. ln the case of 13.9, Z must be a constant almost surely. Intuitively, stationarity and ergodicity together are seen to be sufficient conditions for time averages and ensemble averages to converge to the same limit. is the mean not just of X1 Stationarity implies that, for example, g, f(m()) but of any member of the sequence. The existence of events that are invariant under the shift transformation means that there are regions of the sample space which a particular realization of the sequence will never visit. lf PIFFA E) = yhen the event Ec occurs with probability 0 in a realization where E occurs. However, if invariant events other than the trivial ones are ruled out, we ensure that a sequence will eventually visit a1l parts of the space, with probability 1. In this case time averaging and ensemble averaging are effectively equivalent
X()
=
=
t
(
operations.
)
The following corollary is the main reason for our interest in Theorem 13.10.
13.12 Ergodic theorem Let (.X)4(0)IT be a stationary, ergodic, integrable sequence. Then 1imSnelln
=
(13.27)
F(X1), a.s.
n-4-
Proof This is immediate from 13.10, since by ergodicity, EIXj I#)
F(X1) a.s. w
=
In an ergodic sequence, conditioning on events of probability zero or one is a is trivial operation almost surely, in that the information contained in trivial. The ergodic theorem is an example of a 1aw of large numbers, the first of several such theorems to be studied in later chapters. Unlike most of the subsequent examples this pne is for stationat'y sequences. Its practical applications in econometrics are limited by the fact that the stationarity assumption is often inappropriate, but it is of much theoretical interest, because ergodicity is a very mild constraint on the dependence, as we now show. Atransformation thatis measure-preserving eventually mixes up the outcomes in a non-invariant event A with those in Ac. The measure-preserving property rules out mapping sets into proper subsets of themselves, so we can be sure that FA jAc is nonempty. Repeated iterations of the transformation generate a sequence of sets l T V ) containing different mixtures of the elements of A and Ac. A positive AC; that is, if implies a negative dependence of # on dependence of # on #(A f-h B) > P(A4PB) then P(Ac /'A Bj PB4 #(A f-'h #) < #(#) #(A)#(#) PIAc)PCB). Intuition suggests that the average dependence of B on mixtures of A and Ac should tend to zero as the mixing-up proceeds. ln fact, ergodicity can be characterized in just this kind of way, as the following theorem shows. ,/
:4
=
-
-
=
llependence
201
shift transfonnation Fis ergodic if and only 13.13 Theorem A measure-preserving for pair of if, events A, B e @, any n
1
-xpvkx
lim
rasl
(13.28)
PA)PB).
=
k=1
n-yx
Proof To show ton 1y ito let A be an invariant event and B A. Then PCTPAf'n B) = #(A) for a11 k, and hence the left-hand side of (13.27)is equal to #(A) for all 1' #(A) 2 implying 84) 0 or 17.7 k. This gives #(A) apply the ergodic theorem to the indicator function of the sets To show and ergodic, to give F V w h re T is measure-preserving =
,
=
=
,
iif'
,
,
1
n
lztxt(0l lim X &1
=
-
(13.29)
#(4), a.s.
n-yx
But for any # e 5,
jj x-1,u((o) -)
-
,(.)
k=1
j
dpv
zj I jJst) x,.-tx-
1,u((o)
#IN
-
&1
lztxt(')l
,(x)
!
dp
I
-#(A))dz'((o)
1
n
=
1 -sX#(F
k
A rhB) -PA4PB4
.
(13.30)
k=1
The sequence whose absolute value is the integrand in the left-hand member of cxl by (13.30) converges almost surely to zero as n (13.29);it is bounded uniformly + is clearly integrable. Hence, uniformly in 1 #(A) absolutely by n, so the left-hand member of (13.30)converges to zero by 12.8, and the theorem -->
follows.
.
Following from this result, ergodicity of a stationary sequence is often associated with convergence to zero in Cesro-sum of the autocovariances, and indeed, in a Gaussian sequence the conditions are equivalent.
13.14 Corollary If (Xr4(4)))Tis a stationary, ergodic, square-integrable sequence, then n 1 XCov(XIm)
*n al
--
0 as n
(13.31)
-+
x.
Proof Setting B A and defining a real sequence by the indicators of F A, Xkk equivalent to (13.31).First extend this result to sequencs = 1wu((o), (13.28)is X,'%1Aj(), so that A%((t))= X1(T) Lwlr-ujt). of simple r.v.s. Let X1(() = The main point to be established is tha.tthe differencebetween X1 and a simple r.v. can be ignored in integiation. In other words, the sets F Aj must form a =
=
.j
Theot'y of Stochastic Processes
202
partition of f1, apart possibly from sets of measure 0. Since F is measureAf)) #(UjAj) 1, using 1.2(ii), and hence #(F-1(Uf preserving, #(UF-V 1, additivity of the #(f1 UjF -1Ai) = 0. And since L#(F-V i ) = Zf#(Af) z4jl -1 is a1so disjoint apart f'rom possible measure implies that the collection (F verifying required the property. sets of measure 0, This argument extends by induction to Xk for any k e N. Hence, =
=
=
=
-
7txlxzlE 777rcslxow-u/) j =
i
= 7) 77a,uyrtA
rn T-kAjl,
,w1z'',4
7/... $J
,.-u ' .7
j
-
t.
.
(13.32)
/ (the sum being absolutely convergent by assumption), and by 13.13, n 1 TExk)
lim n -
n--yx
=
X ywm
&1
i
lim
'
'-
j
n-yx
-
n 1 X#(Af
n
-z
rn T Aj)
&1
afW#(Xf)#(Ay) = Xi X j
&XI)2,
=
(13.33)
whereEXj4 2 EX 14EXk) for any k, by stationarity. The theorem extends to generalsequences by the usual application of 3.28 and the monotone convergence theorem.> =
This result might appear to mean that ergodicity implies some fonn of asymptotic independence of the sequence, since one condition under which (13.31)certainly x. But this is not so. The following holds is where Cov(X),XJ 0 as k example illustrates nicely what ergodicity implies and does not imply. -->
-->
13.15 Example Let the probability space (f,T,#) be defined by fl = f0, 1 J, so that 5 = (fO l 0),f 1 l (0,1)) and #() = 0,5 for (.0 0 and (l) = 1. Let F be the transfonnation that sets F0 = 1 and F1 = 0. ln this setup a random sequence and generating the sequence by f Xr(fJ))J may be defined by letting .Yl4(0)= iterating F. These sequences always consist of alternating Os and 1s, but the inial value is randomly chosen with equal probabilities. Now, T is measurepreserving', the invariant events are f and 0, both trivial, so the sequence is ergodic. And it is easily verified that lima-yxn -1Z2 PITkA fa B) P(A)PB) for 0.5 for k every pair A,B e T. For instance, 1et B = f 1 ) and then #(F V f7#) even and 0 for k odd, so that the limit is indeed 0.25 as required. You can verify, equivalently, that the ergodic theorem holds, since the time average of the sequence will always converge to 0.5, which is the same as the ensemble mean (a of XI()). ,f
=
,
,
=
=j
=
,
.
In this example, Xt is perfectly predictable once we know X; for any t. This shows that ergodicity does not imply independence between different parts of the sequence, even as the time separation increases. By contrast, a mixing sequence has this property. A measure-preserving, ergodic shift transformation Wis said to be mixing if, for each A, B e F, ,
l7ependence
lim#(F 4 cjB4
203
(13.34)
PA4PB).
=
k-x
X1(Ff-')) for each T he stationary sequence f.V)7is said to be mixing if A() t where F is a mixing transformation. Compare this condition with (13.28);Cesro convergence of the sequence t#(F V /'7 #) k e EN ) has been replaced by actual convergence. To obtain a sound intuition about mixing transformations, one cannot do better than reflect on the following oft-quoted example, originally due to Halmos (1956). =
,
13.16 Example Consider a dry martini initially poured as a layer of vennouth (10% of the volume) on top of the gin (90%).Let G denote the gin, and F an arbitrary small region of the fluid, so that Fcj G is the gin contained in F. If 0.9 and #(.) denotes the volume of a set as a proportion of the whole, #(G) =
#(Ff-7 G)/#(/), the proportion of gin in F, is initially eithr 0 or 1. Let F denote the operation of stining the martini with a swizzle stick, so that #(F Vcj G)/#4p is the proportion of gin in F after k stirs. Assuming the fluid is incompressible, stirring is a measure-preserving transfonnation in that #(F #) mixes the martini we would expect the proportion = #4#) for all k. If the stirring PITkFP k hich is gin in G)/#(&, to tend to #4G), so that each region F F F, w of of the martini eventually contains 90% gin. la transfonnaThis is precisely condition (13.34).Repeated applications of a lnixing tion to an event A should eventually mix outcomes in A and Ac so thoroughly that for large enough k the composition of F V gives no clues about the original A. Mixing in a real sequence implies that events such as A ( ): A((l)) S tz) and F 1.4 = f(t): Xt-vk) S JJ are becoming independent as k increases. It is immediate, or virtually so, that for stationary mixing sequences the result of 13.14 can be x. strengthened to Cov(X1,-L) 0 as n =
--y
--y
13.5 Subfields and Regularity We now introduce an alternative approach to studying dependence which considers the collection of c-subfields of events generated by a stochastic sequence. This theory is fundamental to nearly everything we do subsequently, particularly because, unlike the ergodic theory of the preceding sections, it generalizes
(stationaty)case. Consider a doubly infinite sequence (-V,t e r) (notnecessarily stationary) and define the family of subfields l X, s f /l, where TJ = nXs,...,Xt) is the smallest c-field on which the sequence coordinates from dates s to t are measurable. The sets of Sf can be visualized as the inverse images of (/ s4dimensional cylinder sets in :8*; compare the discussion of j 12.3, recalling that IN and Z are equipotent. We can let one or other of the bounds tend to intinity, and aparticularly important sub-family is the increasing sequence (T-'x, t e Z ), which information contained in the sequence up to can be thought of as, in effect, /' which The c-field on the sequence as a whole is measurable is the date limitingcase F+-7 Vts-t.. In cases where the underlying probability model beyond the measure-preserving
-
tthe
=
Theory of Stochastic Processes
204 concerns just the sequence
(XZJwe
shall identify 1+-Zwith 5.
OfT-rx. Another interesting object is the remote c-Jcl# (or tail c-field), F-x This c-tield contains events about which we can learn something by observing any coordinate of the sequence, and it might plausibly be supposed that these events the occurred at time past when the initial conditions for the sequence were set. However, note that the set may be generated in other ways, such =
-x,
RS
t Ot 1+=
,
or
Ot 5-tt
tremote'
.
One of the ways to characterize independence in a sequence is to say that any pair of non-overlapping subtields, of the fonn Tfzland 1143where /1 k h > t5 k /4, are independent (see j10.5). One of the most famous early results in the law' for independent theory of stochastic processes is Kolmogorov's usually given for the case of a sequence (Al7, and the sequences. This theorem is is defined in this c-tield remote case as
07,.18/D,
.
13.17 Zero-one Iaw lf the sequence (A-fITis independent, every remote event is trivial, having either probability 0 or probability 1. and let T be the colProof Let A be a remote event, so that A e i= 0*2.1@=f lection of events having the property #(A rn #) #(A)#(#) if B e V. By indepenand @1f dence, are independent subfields, so that @(c V, for every f. We may therefore say that U7=1T( c V. U'7=1T(,then B e T( for some t, and B' e T(' for some t', and if t' 1 lf B,B' and accordingly U7z=lFlis a tield. t (say)ihen B e F1f'so B fa #' e ;1f' c U7 Moreover, if fBj e Nl is a monotone sequence and Bj #, then A f-'h Bj is also monotone and converges toA rn SforanyA l 1'-', adl e V'by continuity of #, so V is a monotone class. By the argument of 1.24, V therefore contains the union of and so V u c(U7=1T'() V*t any countable collection of sets from U7 2 and T.llowevervd e 5, soA e N wemay setfl=A, giving#td) =#(A) s ence p(4) = 0 or 1. . =
,
=
T'K'
=1@1r,
-->
=l97lf
.191,
=
=
.
.
The zero-one 1aw shows us that for an independent sequence there are no events, other than trivial ones, that can be relevant to a11 sequence coordinates. But clearly, not only independent sequences have the zero-one property, and from our point of view the interesting problem is to identify the wider class of sequences that possess it. A sequence fX))+-7 is said to be regular or mixing if every remote event has probability 0 or 1. Regularity is the term adopted by Ibragimov and Linnik (1971), to whom the basics of this theory are due. ln a suitably unified framework,this is essentially equivalent to the mixing concept defined in j13.4. The following theorem says that in a regular sequence, remote events must be independent of a11 events in 5. Note that trivial events are independent of themselves, on the
definition.
13.18 Theorem (lbragimovand Linnik 1971: th. 17.1.1) tall+-Zis regular if and only if, for every B e 5,
lt ( .
Dependence
sup l#(A ra #)
-
I --A
PA)PB4
205 0 as t
(13.35)
.--.) -=.
A e F!-
Proof To prove suppose 3 E E F-x with 0 regular. Then for every t, E e @-'x and so dif'
<
,
S#P I#(A
f''h
F)
I2
#(A)#(A3
-
ziez.f-
#(A)
-
P(E) < 1, so that (X;)+-=xis not PE4
2>
(ja a6)
()
,
.
which contradicts (13.35).
if assume regularity and define random variables ( lx #(A) (F-lx/f-measurable) and q (Wf-measurable), such that ls #(#) F()n). Then, by the Cauchy-schwartz inequality, #(A fa #) P(A)PB) To prove
Eonly
=
,
=
-
-
=
-
lF((n) I
=
IlF(n I@-'-)112, IuE'tsE'tnI@!-)) I < lItII2
(13.37)
where the equality is by the Iaw of iterated expectations because A e F-lx. Note which will complete the < 1. We show lIF(n l@-,-)112 0 as t that 11(112 of T-fx. roof, arbitrary since element is X an P x)()) Consider the sequence (F(ls I?F-f lt2. For any (.t) e f1, -->
-=,
--
F(1sI8/'-x)())
F(1s l@-x)(t0) as t
--
(13.38)
--> -=,
where by equation (10.18) and the zero-one property,
Jw&1sI;-x)()J#()
f-7#)
PE
=
#(#), PE4 =
1
=
#(A-) = 0
0,
,
E e Fux.
(13.39)
#(#) a.s. agrees with the definition, so It is clear that setting F(ls I@-x)() 5-t.) F(lsl P@) a.s., or, equivalently, that we may say that (F(1s IT!x) /'(#)()2 0, a.s. (13.40) Since 1s()) #(#) < 1 for al1 (1) e f, (F(ls @-fx)() /:(#)()2 is similarly =
--
--)
-
1
-
I
I
-
bounded, uniformly in f. Unifonn integrability of the sequence can therefore be assumed, and it follows from 12.8 that IlF(n l9-/-)112=
as required.
IlF(1sI@-'-)(t,))
-/'(:)112
--
0 as t
(13.41)
--+ -,x,
.
In this theorem, it is less the existence of the limit than the passage to the limit, the fact that the supremum in (13.35)can be made small by choosing large, that gives the result practical significance. When the only remote events are trivial, the dependence of Xt-vkon events in @-x, for fixed k must eventually decline as f increases. The zero-one 1aw is an instant corollary of the necessity part of the theorem, since an independent sequence would certainly satisfy -t
,
(13.35).
Theory of Stochastic Processes
206
There is an obvious connection between the properties of invariance and remoteness. lf T is a measure-preserving shift transformation we have the following simple implication.
13.19 Theorem
If FA
=
A, then A e T-x. l
l
t t t Proof lf A e T-x, then Fz4 e @-x and F A e F-x :4 followsimmediately that e (*s=ts-sxj chtO,k +
-
-
1 .
lf TA
-x@!x)
=
-
=
l
A, F A
T-x.
A and it
=
.
t
The last result of this section establishes formally the relatlonship between regularity and ergodicity which has been implicit in the foregoing discussion.
13.20 Theorem (Ibragimov and Linnik 1971: cor. 17. 1.1) If a stationary sequence x((o) (AX(o))!Zis regular, it is also ergodic. =
Proof Every set A e F+-7is contained in a set At e T-ff, with the sequence (A,) non-increasing and Ar A. Thus, A, may be constnlcted as the inverse image under .x of the (2/+ ll-dimensional cylinder set whose base is the product of the coorof x(A) e The inclusion follows by dinate f-sets for coordinates P(At4 #(A). continuity of By P, that 1.2(iv). we can assume Let A be invariant. Using the measure-preserving property of F, w find .1,
=.
-4,...,/
-->
P(At raAl
=
PAt
Since k is arbitzary, regularity Letting f
--
x
yields #(A)
=
fa
F -kA)
=
PITkA t chz4l.
(13.42)
implies by (13.34)that PAt faAl = PAt)PA). P(A) 2 so that P(A4 0 or 1, as required. . =
,
13.6 Strong and Uniform Mixing The defect of mixing (regularity)as an operational concept is that remote events are of less interest than arbitrary events which happen to be widely separated in time. The extra ingredient we need for a workable theory is the concept of dependence between pairs of c-subtields of events. There are several ways to characterize such dependence, but the following are the concepts that have been most commonly exploited in limit theory. Let L1,5,P) be a probability space, and 1et V and R be c-subfields of @;then
(x(V,X) =
sup G c F Hq
.'(
IP(G
f'n
S)
-
#(G)#(&
I
(13.43)
l
(13.44)
,
is known as the strong mixing coeftkient, #(V,A)
=
sup Gq F,f'1eJf; P(G)>0
and
G4 1/,(/./'1
-#(#)
as the uniform mixing coefficient. These are alternative measures of the dependence between the subfields V and R. If the subfields V and R are independent, then (x(V,J) = 0 and #(V,Jf) 0, and the converse is also true in the case of uniform mixing, although not for strong mixing. At first sight there may appear not much to choose between the defini=
,
Dependence
207
tions, but since they are set up in terms of suprema of the dependence measures over the sets of events in question, it is the extreme (and possibly anomalous) cases which define the characteristics of the mixing coeixcients. The strong mixing concept is weaker than the uniform concept. Since
IPG
r-'hff
-
PCG4PC* I S IPCH, G)
I f 9(5',>t)
PH)
-
(13.45)
for all G e V and H e R, it is clear that a(N,J#) f ((T,Jf). However, the following example shows how the two concepts differ more crucially. 13.21 Example Suppose that, for a sequence of subfields (V-IT,and a subfield X, a%m,R4 0 as m x. This condition is compatible with the existence of sets Gm ?. forJ xp(s) PGm) llm, andlGm fo & e Mmandffe R with thejroperties .-+
--
=alm
=
.
But (/497,s,J#)2 IPCH6Gm) #(/f) I Ia PH4 I for every tilds Mmand R are not independent in the limit. n =
-
-
,?z
k 1, showing that sub-
Evidently, the strong mixing characterization of does not rule out ppssibility dependence negligible between the sets. qf () rnixing coefficients and the only not that are can be defined, although they tx have proved the most popular in applications. Others that have appeared in the literature include tindepndence'
13(V,J#) = sup E HR
j#(SIV)
-
#(S)
j
(13.46)
and
p(V,J#)
sup
=
If((n)I
(
2
n
(13.47)
,
2
where the latter supremum is taken with respect to a1l square integrable, zero mean, V-measurable r.v.s (, and S-measurable r.v.s q. To ompare these alter#(Sl T)() #(&, so that natives, first let (() =
f5(V,X)
-
=
sup SGS
JI(1
dP
I( I js sup
dp
GC%D.H6R
sup
G e F, H R
(.LdP
*
=
a(T,3f).
(13.48)
'''
#(G) and Moreover, since for any sets G e V and H e 3f, (((t)) 1c() n() which members the of is defined, p(V,>t) and F((n) #(& set 1zz((l)) are over while l( I < 1 and lqI S 1 for these cases, it is also clear #(G f''h z.fl PGlm notwithstanding (x its designation, is the weakest of mixing, Thus, that p a. o? variants, although it is these four course stronger than ordinary trivial remote events. We also have j S 9, by an regularity characterized by immediatecorollary of the following result. =
-
=
=
-
-
istrong'
Theory of Stochastic Processes
208
13.22Theorem I#(NIV)
If
#(&
-
9(V,>t) a.s., for a1l H e R. a
The main step in the proof of 13.22 is the following lemma. !' 13.23 Lemma Let X be an talmost surely boundedy) V-measurable nv. Then
1
sup GeF,#(G)>0
jaxdp
po).
ess sup
=
x.
(13.49)
:'
r
E
Proof #(G)-1 j < ess sup X, for any set G ils the designated class. For any E > 0, consider the sets .
fccxlrj
..E
.
.
G G-
f (:
=
=
:
X() -A)
k
v
(ess sup.km 2 (ess sup
-
al
-
6:), E).
By definition of ess sup X, both these sets belong to V and at least one of them is nonempty and has positive probability. Define the set G*
G+, #(G>+)k PG-) =
G -, otherwise
and we may conclude that
1 Jc- xdp ,(c-)
(13.49) now follows on letting
E.
z (ess spp approach 0.
m -:.
(13.50)
*
of 13.22 #ut X #(SIV) #(S) in the lemma, noting that this is a Observe that, for any G e V, V-measurable r.v. lying between +1 and E'(G)-1 .' IJ crxl'l = IPCH, G) Ph'j l Hence the lemma together with (13.44) implies that, for any H e R, Proof
=
-
-1.
-
.
9(N,J#)
ess sup
k with probability 1.
.
f#(SIV)
l#(SI V)
-
#4&
l
-
,
#(S))
(13.51)
14 Mixing
14.1 Mixing Sequences of Random Variables For a sequence
T-Jx
(X,())J=-x,let
=
of...,Xt-z,Xt-j,Xt) as in
913.5,and
similarly
The sequence is said to be vmixing (or define5*t+m n(Xt+m,Xt+m+3,.%+m+1,...4. lim,u-Axa,u 0 where strong mixing) if =
=
qm
sup
=
(x(T-Jx,@7+,,,),
(14.1)
l
and a is defined in (13.43).It is limpz-jxtj,u0, where
said to be
=
Sup
=
zrl
(
b-mixingtoruniform
9(?F-f x,T7+r,,),
mixing) if
(14.2)
and (j is detined in (13.44).(-mixing implies a-mixing as noted in j 13.6, while the converse does not hold. Another difference is that g-mixing is not timex) reversible', in other words, it is not necessarily the case that supytF/ +,,I,T-f =
supytT-lx,T''; By contrast, a-mixing is time-reversible. If the sequence (.X))+-7 is a-mixing, so is the sequence ( Ffl+-7 where J'f X-t. 0 where (#,((.t)))=-xis also said to be absolutely regular if limp-xp,,, +,,).
=
=
p,,,
sup
=
t
13(8?;-'-,8?7-,+,,,)
(14.3)
and 13is defined in (13.46).According to the results in j13.6, absolute regularit,yis a condition intenuediate between svong mixing and uniform mixing. On the other hand, if (m)+= is a stationary, fmbounded sequence, and @0-x c(...,X-1,A%) and %m+* c(X,,,X,,+1,...), the sequence is said to be completely O where p is defined in (13.47).In stationary regular if p,,, p( T0 J;+oo m ) regllarity complete is equivalent to strong mixing. KolmoGaussian sequences, gorov and Rozanov (1960)show that in this case =
..x
=
-->
=
-x,
,
qm/
Lzam.
pm <
(14.4)
In a completely regular sequence, the autocovariances T Extxt-j) must tend to x. A sufficient condition for complete regularity can be expressed in 0 as j terms of the spectral density function.When it exists, the spectral density fh) is the Fourier transform of the autocovariance function, that is to say, =
-->
f z)
1 =
t=
77
c'l?
2,s jzz--sn z)
,
,
e
(-a,a1.
(14.5)
Theory of Stochastic Processes The theorem of Kolmogorov and Rozanov leads to the result proved by Ibragimov and Linnik (1971: th. 17.3.3), that a stationary Gaussian sequence is strong mixing when fk) exists and is continuous and strictly positive, everywhere on
(-a,cj.
This topic is something of a tenninological
minefield. KRegularity' is an undescriptive term, and there does not seem to be unanimity among authors with regard to usage, complete regularity and absolute regularity sometimes being used synonymously. Nor is the list of mixing concepts given here by any means exhaustive. Fortunately, we shall be able to avoid this confusion by sticking with the strong and uniform cases. While there are some applications in which absolute regularity provides just the right condition, we shall not encounter any of these. might be thought appropriate as a synonym for Incidentally, the tenn regularity, but should be avoided as there is a risk of confusion with wetz/cdependence, a term used, often somewhat imprecisely, to refer to sequences having summable covariances. Strongly dcpendent sequences may be stationary and mixing, buttheircovariances are non-summable. (dWeak' implies Icswdependencethan <strong' in this instance, not morel) Confining our attention now to the strong and uniform mixing definitions, measures of the dependence in a sequence can be based in various ways on the rate at which the mixing coefficients xmor 4u tend to zero. To avoid repetition we will discuss just strong mixing, but the following remarks apply equally to the ;-x Since the collections uniform mixing case, on substituting 4)for Gthroughout. and 5*t+m are respectively non-decreasing in t and non-increasing in t and m, the sequence f(xp,lis monotone. The rate of convergence is often quantified by a summability criterions that for some number ? > 0, xm 0 sufficiently fast that tweakmixing'
f
-->
X
7711* < r;=1
x.
(14.6)
The term size has been coined to describe the rate of convergence of the mixing numbers, although different definitions have been used by different authors, and the terminology should be used with caution. One possibility is to say that the if the mixing numbers satisfy (14.6).However, the commonsequence is of size White 1984) is to say that a sequence is G-mixing of example for est usage (see *.11 if am Om -#) for some fp > It is clear that such sequences are size when of the 1/:0, raised to summable so that this concept of size is power stronger than the summability concept. One temptation to be avoided is to define the size as < whe re f? is the largest constant such that the a) * are summable' ; for no such number may exist. Since mixing is not so much a property of the sequence (.X)) as of the sequences of c-fields generated by (A), it holds for any random variables measurable on those c-fields, such as measurable transformations of Xt. More generally, we have the following implication: -(2
-(p()
=
-(?,
14.1 Theorem Let Ff = g(Xt,Xt-3,...,Xt-x) be a measurable function, for finite of size then Ff is also. lf Xt is a-mixing (g-mixing) -(?,
T.
Mixing
211
= Proof Let V-fx= c(..., 1$-1,F, and G*twm c(I$+,,,,im+1,...). Since F, is measuri-Jx which each of Xt.%-j,...,Xt-z are measurable, Wlx i able on any c-field on and it follows that ay,,u f (u-s and M*t-bm i T7+,u-s. Let ay,,, supr (N-fx,Vc;+,,,) for m k 1. With tinite, (xm-z O(m-%) if um Om-%) and the conclusion replacing follows. The same argument follows word for word with w =
.1
=
=
d9'
ttx'
.
14.2 Mixing lnequalities Strong and uniform mixing are restrictions on the complete joint distribution of the sequence, and to make practical use of the concepts we must know what they imply about particular measures of dependence. This section establishes a set of fundamental moment inequalities for mixing processes. The main results bound the m-step-ahead predictions, Extwm I5-tx). Mixing implies that, as we try to forecast the future path of a sequence from knowledge of its histol'y to date, looking further and further forward, we will eventually be unable to improve on the predictor based solely on the distribution of the sequence as a whole, EXt+m). The r.v. Extm lT-'x) EXt+m) is tending to zero as m increases. We prove convergence of the fw-norm. -
14.2 Theorem
p k 1 and with um defined in (14.1), lI,< 2(21/: + 1)a,,,1/r-1/rl1m+,,,lI,.(14.7)
(Ibragimov 1962) For r
IlF(A+r,,IT-'-)
-
Exm)
Proof To simplify notation, substitute X for Xtmm,V for s-too, R for T7+,,,,and a for xm.It will be understood thatxis an X-measurable random variable where V, R c 5. The proof is in two stages, tirst to establish the result for IXI < Mx < cxo a.s., and then to extend it to the case where X is G-boundedfor t'inite r. Define
the V-measurable r.v. q
=
sgn(F(Xl V) F(m) -
1, F(.Yl V) k #(m, =
-1, otherwise.
(14.8)
Using 10.8 and 10.10,
9') A'IFIXI
-
F(ml
=
=
V) -F(-Y))1 FEn(F(XI FEtA'tn.YlN) -nF(m1
= Covtq,m
ICovtq,m I
=
.
(14.9)
Let F be any V-measurable r.v., such as q for example. Noting that R) -F(F)) is A-measurable, similar arguments give sgn(F(FI Icovt-Y,F)
R4 -A'(l'))) I I IF(X(&FI < F(IXII(&FIR4 -F(F))I) < Mxf'l F(y1 R) F(F) l =
-
(
=
Theory of Stochastic Processes
1Z
G A&1COv((,F)
l
(14.10)
,
where the tirst inequality is the modulus inequality. ( and n are simple random variables taking only two distinct values each, so define the sets A+ (n = 1 ), =
A-
(n
=
-1
=
together gives
), B*
FlFl.Yl T) F(m -
=
()
=
=
1 ) and B,
l K Mxl Cov(n,k)l
Mxj (#(A+
chB+) + P(A-
=
((
=
=
Mxlfttnl
fa#-)
PA+c
-
(#(A+)#(#+) +PA-)PB-)
l
-1
Putting
.
-
F(()F(n)
B-)
I
PA- rn #+)1
-
PAhP(B-j
-
(14.9)and (14.10)
PA-4PB+)j
-
< 4Mm. Since IF(-Yl N) F(m -
l < IF(#IT) 1+ lF(ml f lMx. V) F(mIIP S 2M,x(2a)1/F. llF(-YI
I
(14.11)
it follows that, for p
1,
(14.12)
-
This completes the first pal't of the proof. The next step is to let X be Lr-bounded. Choose a finite positive Mx, and define Xl 1( IxjsMxlx and .Y2 By the Minkowski inequality and (14.11), X =
-
=
m .
IlF(.YlN)
-
+ T) e'(A)11, < IIF(-LIT) -F(-Yl)lI, A'(mII, 11(721 2A.za'(2a)1/J' + zjjxallp, u -
and the problem is to bound the second right-hand-side llX2ll/?f
for r
member. But
sf-rlr IlA'IIrr/P
(14.14)
p, so we arrive at 11.E'(x1 N) F(#)ll, < -
Fillally,
(14. 13)
Choosing
Mx
1IXl)ra-
=
1Mx2%)31P I/C
zM-l--rll,-Yllr/.
(14.l5)
and simplifying yields
11F(-Y1 N) -.E'(A')11,/
which is the required result.
+
2(21/P+
lltxl/#-l/rjjxjjr,
*
There is an easy corollary bounding the autocovariances of the sequence.
14.3 Corollary For p
1 and r plp 1), (14.16) ICov(x,,.Y,+,,)l < 2(21-1* + 1)a)-1/#-1/rjjx;IIp1Ix,+mI!r. >
-
Proof
ICovxvxmb l
=
IEXtxtwmj
-
F(A)F(-'+,,,)
I
= IExtExt-vm I9,) 1lpllA'(X,+,, I@,) F(.&+,2II,/?-l) f 11.X) 2(21-1/P+ 1)jj#,ljpIlX,+mIl<)-1/P-1/r f EXt+mttj
-
l
-
(14. 17)
Vixing
213
where the second equality is by 10.8 and 10.10, noting that Xt is Fsmeasurable, the tirst inequality is the Hlder inequality, and the second inequality is by 14.2. . 1.4.4 Theorem
(Serfling 1968: th. 2.2) For r k p k 1, IlA'(Xf+??, r. I11x) EXt+m) < 29,,,1-1/r1I.&+,,,II
11p
-
where
4),,,is
defined in
(14.18)
(14.2).
Proof The result is trivial for r p 1, so assume r > 1. The strategy is to prove the result initially for a sequence of simple r.v.s. Let Xt-vm Z#,=1.xj1xj, = nxtmm,xtwm.vj, For some e ( consider the random 4/ e 17+,,,,where s-tmm T-fx)(t')), Ext-vm although for clarity of notation the dependence on ) is element l not indicated. For r > 1 and q r/tr 1), we have =
=
=
...).
=
IExt..m I@-'-)
E(Xt+m)3r
-
'>7
j
-
l@-'-)
k-xipAi
=
-
i
i
1
r
IxflI#(4fI@-'-)-#(4I
S
r
#(4f))
r '>7
= f
ImlI#( i I5t ) z1.
i
77Ixi I I#(X i I5-t X
l/rl
x) -
#(X
ij
i
S
E IXt-mIrl @-,-)+
1/t?
I #(AjI@-r-)
PAi)
-
--
#(Af) I
-
) (77
I 77I#(A I5
X?
-'
-) -
i
r)
I FIxt-vm
i
IPAi l@-'-)
-
#(Af) I
PAi)
X.
)
I
(14.19)
The second inequality here is by 9.25. The sets Aj pmition and T-rx)+#(Af'I5-t=) a.s. and PAiwAk') PAi QJA/' I@-rx) #(AfI PAi) + PAi') for # i'. Letting A) denote the union of all those Aj for which P(Ai I@-rx) PAi) k 0, and AJthe complement of 4) on f1, ,
=
=
-
77I#tAfl @-'-)-#4A/)1
I#(A) I@-'-)
=
#(A1)I+ IPCAII@-'-) #(A7)I (14.20)
-
j
-
.
By 13.22, the inequalities
lPA; I@-'-)
#(41)
IPA; I@-'-)
PA;)
-
-
l < 4u I S 9,,,
hold with probability 1. Substituting int (14.19)gives
lExmf
@-'-) Exm) -
Taking expectations
a.s. (14.21) ge'(IAu,,, Irl @-r-)+ Flx,+,,,l rjtztTl,,,lr&
Ir
and using the law of iterated expectations
t E If(xf+r,,l T-x)
-
rIq Ext,vm) Ir S lElXt+m Ir (29,,,)
and, raising both sides to the power 1/r,
then gives '
,
(14.22)
214
Theory of Stochastic Processes f 2IIA+-II41 F(A7,a)IIr -1/r.
IT-'-) Il'(kt+,,,
-
(14.23)
Inequality (14.18)follows by Liapunov's inequality. The result extends from simple to general r.v,s using the constmction of 3.28.
ke For any nv. X:+mthere exists a monotone sequence of simple r.v.s fxkywm, IxY(z)r+,,,() 0 as k N ) such that Xt-vmlt l for all e f. This converx) @-f Exkymm), k e (N) by transfers the sequences (F(aV)f+,,, l a.s. to gence 10.15. Then, assuming Xtwmis fv-bounded, the inequality in (14.22)holds as k Ir as the x by the dominated convergence theorem applied to each side, with lxfx.m dominating function, thanks to 10.13(ii). Tlis completes the proof. w -->
--)
=,
-
-
-->
The counterpart of 14.3 is obtained similarly.
14.5 Corollary
For r k 1, ICov(-Yr+,,,.x))l
where, if r
=
1, replace
s zgml/rjjxrjjrjjx/+mjlrytr-j),
by 1IX,+,,,IIx ess IlXf+r,,IIr/(r-1) =
(14.24)
sup Xt.vm.
Proof
ICov(x)+,,,,.')I I9-f-) F(-Y,+r,,)lIs(r-l) s lIA-,IlrlI'(.2t+,,, -
s 2#-17rlIx,lIrII=,+-lIr/(r-1), where the first inequality corresponds is by 14.4. .
(14.25)
to the one in (14.17),and the second one
These results tell us a good deal about the behaviour of mixing sequences. A fundamental property is mean reversion. The mean deviation sequence (A Ext) ) must change sign frequently when the rate of mixing is ligh. lf the sequence exhibits persistent behaviour with Xt Ex tending to have the same sign for a large number of successive periods, thep IE(Xt+mI5-t.) Extmm) j would likewise tend to be large for large m. If this quantity is small the sign of the mean deviation m periods hence is unpredictable, indicating that it changes frequently. Butwhile mixing implies mean reversion, mean reversion need notimply mixing. Theorems 14.2 and 14.4 isolate the properties of greatest importance, but not the only ones. A sequence having the property that Ilvar(Ap, I9-',.) Vart-mlllr > 0 is called conditionally heteroscedastic. Mixing also requires this sequence of norms to converge as m x, and similarly for other integrable functions of Xtwm. Comparison of 14.2 and 14.4 also shows that being able to assert uniform mixing can give us considerably greater flexibility in applications with respect to the existence of moments. In (14.18),the rate of convergence of the left-hand side to F-Jx) Ext+m) I zero with m does not depend upon p, and in particular, FIE(Xt+mI whenever exists 0, condition infinitesimally > for a 117,+,,,111+8 converges stronger than uniform integrability. In the corresponding inequality for am in 14.2, p < r is required for the restriction to Likewise, 14.5 for the case = p 2 yields -
-
-
-
--+
-
Kbite'
.
Mixing
215
I 2#,,,1/2jI.;t1I2IIx,+,,,Il2,
lCovxmnx
but to be useful 14.3 requires that either Xt or Xtwmbe fo-bounded, Mere existence of the variances will not suffice.
(14.26) for
>
0.
14.3 Mixing in Linear Processes fmlO-x
which arises very frequently in econometric A type of stochastic sequence modelling applications has the representation
Xt
=
q
jzt
-j
,
0
<-
(14.27)
-<
q
c'o
,
jcut where (Z,1+-7 (calledthe innovations or shoks) is an independent stochastic is a sequence of fixed coefficients. Assume without loss of sequence, and ( %lp=() generality that the Zt have zero means and that % 1. (14.27)is called a moving average process of order q (MA()). lnto this class fall the finite-order autoregressive and autoregressive-moving average (ARMA) processes commonly would clearly like to know when such model We economic series. used to time sequences are mixing, by reference to the properties of the innovations and of the sequence f 0;1. Several authors have investigated this question, including =
Ibragimov andLinniktlg7l), Chanda(1974), Gorodetskii (1977),Withers (1981a), Pham and Tran (1985),and Athreya and Pantula (1986a,1986b). Mixing is an asymptotic property, and when q < x the sequence is mixing infinitely fast. This case is called q-dependence. The difficulties arise with the cases with q x. Formally, we should think of the MA(=) as the weak limit of a sequence of MA() processes; the characteristic function of Xt has the form =
tlk/)
=
Iltllzr-/fz-t) ,
j-j
#t?f(%) ($(D(pointwisein
(1k.28)
cxp where R) as q 9r(l) is a ch.f. and 0, we may invel't the latter according to 11.12, and identify the continuous at 12 T 0Z The existence of the limit corresponding distribution as that of Xt Zy ) t-j. imposes certain conditions on the coefficient sequence f0./17,0.We clearly need oo, and for the variance of Xt to exist, it is further necesl $1 0 as j the sequence be square-summable. Note that the solutions of finitesary that order ARMA processes are characterized by the approach of l0.j1to 0 at an exponential rate, beyond a finite point in the sequence. If (4) is i.i.d. with mean 0 and variance c2, Xt is stationary and has spectral density function
and if
-->
,
--
=
=
=()
-,..:
--
.wl
co
f (1) 2:: Tjeihi jujl '''
=
2 .
(14.29)
The theorem of Ibragimov and Linnik cited in j14.1 yields thexconditionZ7=()l $1 strong-mixing in txl the for sufficient Gausjian another However, case. < as standard result (seeDoob 1953t ch. X!8, or Ibragimov and Linnik 1971: ch. 16.7)
Theoty of Stochastic Processes
states that every wide-sense stationary sequence admitting a spectral density has moving average representation with orthogonal increments and a (doubly-infinite) square summable coefficients.
But allowing more general distributions for the innovations yields surprising results. Contrary to what might be supposed, having the %tend to zero even at an exponential rate is not sufticient by itself for strong mixing. Here is a simple illustration. Recall that the first-order autoregressive process Xt pm-l + Zt, p/, j 0, 1,2,... Ip I < 1 has the MA(x) form with % =
=
,
=
14.6 Example Let (Z,)7' be an independent sequence of Bernoulli r.v.s, with P(Zt Ztl and 0) = zl Let 1) Pzt =
=
ax'c
=
Xt
=
.
=
.zXl t- j
Zt
+
=
t
F'')l-izt-p
t
1,2,3,
=
(14.30)
...
lt is not difficult to see that the term f
X
2-lZt-j
t
=
2-Xlkzk
(14.31)
belongs for each t to the set of dyadic rationals Bs (kl2t, k = 0,1,2,... 2f+1 2f+1 1 ) Each element of Ff corresponds to one of the p ossible drawings =
-
.
(Zo,...,4),
and has equal probability of 2 -fXt e Bt
whereas iff
ztl
=
=
fkllt, k
1
.
Iff Zo
0,2,4,...,242f
=
=
-
0,
1)),
1, Xt e J'F'f Bt -
=
Lkllt ,
k
=
13 5 ,
,
2f+1 I j -
,...,
.
It follows that (.Y()= 1 ) /a fX, e Btj 0, for every finite /. But it is clear that ,1 PIXO = = 0) P(Xt 6 SJ Hence for every finite m, =
=
.
)
q?n X #(
(&
=
1 ) rn (A% e Bm))
= t,
which contradicts
im
.--.h
-
#(V
=
J
1)#(Xm e Bm4
(14.32)
0. n
Since the process starts at t = 0 in this case it is not stationary, but the example is easily generalized to a wider class of processes, as follows, be an independent sequence of Bern14.7 Theorem (Andrews 1984) Let IZtIEMJx If Xt oulli r.v.s, taking values 1 and 0 tixed probabilities p and 1 (A)*-oo + mixing.' is pXf-1 Zt for p e (0,!, not strong n ,with
-#.
=
Note, the condition on p is purely to expedite the argument. The theorem surely holds for other values of p, although this cannot be proved by the present
approach.
Proof Write Xt+s = p'Xf + Xta where
Mixing
217
x-1
Xts
fZt-s-j. 'X7 j=
=
(14.33)
The support of Xzs is finite for finite p, having at most 2, distinct members. Call this set Bi, so that Wr! (0,1), Fa = (O, 1, p, 1 + p), and so on. ln general, 8$+1is obtained from Wrxby adding p' to each of its elements and forming the union of these elements with those of Bi; formally, =
WQ1= F,
QJ
tw+ p&:w
v,J,
s
s
=
(14.34)
2,3,...
For given s denote the distinct elements of B&by wj, ordered by magnitude with wl < < wJ, for J < ls. Now suppose tha t Xt e (0 p) so that ps#t e (0 p'+1) This means that Xtms for some-j. Defining events A = (aYtE assumesa va lue between wj and wy+ p&+1 r'+1) w+ (0,p)l and Bs lXf+, e Ul/=ltwj, ) we have #(#,IA) == l for any s, howeverlarge. To see that #(A) > 0, consider the case Zt Zf-l Zt-z 0 and zr-a 1 and note that ...
,
,
.
,
,
=
,
=
=
=
x
xp?
;jS j=3
for p e (0,1g.So, unless PBs)
3
x
t-j u ppz
=
j=3
j
P
<
p
.
(14.35)
p
1, strong mixing is contradicted. (Xf e (p,1)l has positive The proof is completed by showing that the set D probability, and is disjoint with #,. D occurs when Zt 0 and Zf-1 1, since =
=
=
=
then, for p e (0,,11,
X
oo
ppz p s :-l j= t-j
and hence #(D)
>
y--lp/ j P ) s
u
=
-
(14.36)
1,
/=1
0. Suppose that
min (ws1 ./k1
-
wyl k
p
t <
+
J-1
(14.37)
.
Then, if D occurs, wy+
p
J+1
pSXf B
f uj. +
p
U/ J=1 (w j,
sy
wj
p&-1s wy
(1438)
1,
.
pS+1)
0. or in other words, Bs rn D The assertion ln (14.37)is certainly true when s 1, so consider the following inductive argument. Suppose the distance between two points in J7, is at least cannot be ()'-1 Then by (14.34),the smallest distance between two points of +,+1 rlpJ-l pf-l when which pf S p' But leSS than the smaller of pf and p e (0,r1), s s impliesp'-1 p 2 p It fo llows that (14.37)holds for every s. w hence, Xtn
wj +
=
wy+
=
,
=
.
-
.
,
-
.
These results may appear surprising when one thinks of the rate at which p' but if so, this is because we are unconsciously thinking approaches 0 with about the problem of predicting gross features of the distribution of Xtmsfrom time t, things like Pxt-vs f xlA) for fixyd x, for example. The notable feature of the sets Bs is their inrlevance to such concerns, at least for large s. What we -;
Theory of Stochastic Processes
218
have shown is that from a practical viewpointthe mixingconcept has some undesi able features. The requirement of a decline of dependence is imposed over 6 events, whereas in practice it might serve our pumoses adequately to tolera certain uninteresting events, such as the Bs defined above, remaining dependent c the initial conditions even at long range. ln the next section we will derive some sufficient conditions for strong mixinl and it turns out that certain smoothness conditions on the marginal distribution of the increments will be enough to rule out this kind of counter-example. But no. consider uniform mixing. 14.8 Example 13C ons ider an AR(1) process with i.i.d. increments, Xt
pX,-l
=
+
Zt, 0
<
<
p
1,
in which the marginal distribution of Zt has unbounded support. We show that (,X)) is not uniform mixing. For 8 > 0 choose a positive constant M to satisfy m-l
plzm-j < # X 7=0
-.M
<
(14.39)
.
Then consider the events
yt
=
(L +A/)) e 10 p '-m
(&
-x
fXm < Lk e sm+* where L is large enough that #(#) k 1 8. We show #(AI> 0 for every m. Let pK = #(Zo < A'), for any constant K. Since Zo has unbounded support, either px < 1 for every K > 0 or, at worst, this holds after substituting (-Z,) for (Zf l and hence ( -XtI for (Xf). pk < 1 fOr a1l K implies, by stationarity, #(X-l < 0) = Pxo < 0) < 1. Since (.%< A-Ji (Z0 < X') QJ (fZ() 2 A') f'n IX-I < 01), independence of the (Zf l implies that B
=
-
PIXZ
So PA) that
>
PB
<
X) K pK+
(1-#r)#(X-1
<
0) < 1. PXJA'C+ ZPJ
0, since K is arbitrary. Since Xm =
I4)
=
# p'Vc +
m- 1
X pizm-j < L
by (14.39).Hence 4u k I#(#lA) means ()a, = 1 for every m. n
-
PB4
p'Ve
I> 1
-
2,
L +M
(14.40) .-)
p'Zm-j, it is clear
<
and since
(14.41) is arbitrary, this
Processes with Gaussian increments fall into the category covered by this example, and if (hmixing fails in the first-order AR case it is pretty clear that counteru examplej exist for more generalMAtx) ases too. The conditions for uniform mixing in linear processes are evidently exemely tough, perhaps too tough for this mixing condition to be very useful. ln the applications to be studied in later chapters, most of the results are found to hold in some form for strong mixing processes, but the ability to assert uniform mixing usually allows a relaxation of
Mixing
219
conditions elsewhere in the problem, so it is still desirable to develop the parallel results for the unifonn case. The strong restrictions needed to ensure processes are mixing, which these examples point to (to be explored further in the next section), threaten to limit the usefulness of the mixing concept. However, technical infringements like the ones demonstrated are often innocuous in practice. Only certain aspects of mixing, encapsulated in the concept of a mixingale, are required for many important limit results to hold. These are shared with so-called near-epoch dependenthmctions of mixing sequences, which include cases like 14.7. The theory of these dependence concepts is treated in Chapters 16 and 17. While Chapter 15 contains some necessal'y background material for those chapters, the interested reader might choose to essence, the diftkulty will be skip ahead at this point to find out how,
tn
resolved.
for Strong and Uniform Mixing
14.4 Sufficient Conitions
The problems in thecounter-examples aboveare with theformof themarginal shock distributions or unbounded, as the case may be. For strong mixing, a degree of smoothness of the distributions appears necessary in addition to summability conditions on the coefticients of linear processes. Several sufficient conditions have been derived, both for general MA(x) processes and for autoregressive and ARMA processes. The suftkiency result for strong mixing proved below is based on the theorems of Chanda (1974)and Gorodetskii (1977).These conditions are not the weakest possible in a1l circumstances, but they have the virtues of generality and comparative ease of verification. -discrete
define a random sequence taY/l=-x,where, for 14.9 Theorem Let Xt X*j=eojzt-j either 0 < r f 2 or r an even positive integer, (a) Zt is unifot-mly fv-bounded, independent, continuous with p.d.f. J'rzt, and =
Suy P
+-Ih-zt-va, j..
I
yz,tzldz u Mlzl,
-
for some whenever Ia I f =, < (y.;1&1+r) x gt (b) Zf=0 w jwre
>
,
&l' <
(14.42)
-,
0;
X
I$1 277 j=t
r,
Grtrl
r < 2,
(14.43)
=
gr-l
x
:2j y-'q
r/2 y.
g. ,
j=t
0./.'Ei
0 for a11complex numbers x with jxl Then (.&)is strong mixing with m O(Zx/z.+1 g t (r)1&1+r)) z (C)
0(1)
=
ZJ T =1
#
=
.
Before proceeding to the proof, we must discuss the implications of these three conditions in a bit more detail. Conditton 14.9(a) may be relaxed somewhat, as we
Theory of Stochastic Processes
220
show below, but we begin with this case for simplicity. The following lemma extends the condition to the joint distributions under independence. 14.10 Lemma lnequality (14.42)implies that for
k
k
Jw,
ml-ljfzrtzr''btz'l
Proof
IJrl M
=
1,...,k,
77lJ,l
.
(14.44)
>1
>1
usingFubini-s
6, t
k
dzz...dzk f I1TzJz2
-
<
theorem, k
k jpk mljfz/zr+*l
l1Tz,tzf)dzt...dzk
-
,.-1 f
jpk
+Jl) I-fzltzl
f
I
...dzk
,
, dz,...dzk l-I.fz,(o) I1.fz,(zr+zh)
+jpk +
Jzl
>2
.fzl(zl)
Ml cl
k +J) Tzltzlllllfz/z,
-
-
>2
>2
k
jvk-
mlj/ktz,
k
+
at4
-
I1Jz,(z
dzz...dzk.
>2
The lemma follows on applying the same inequality to the second term on the right, iteratively for / = 2,...,k. w Condition 14.9(b) is satisfied when I0;1 <<j-V for g. > 1 + 2/r when r K 2 and g > 3/2 + 1/r when r k 2. The double definition of Gtr) is motivated by the fact that for cases with r < 2 we use the von BalmEsseen inequality (11.15)to bound a certain sequence in the proof, whereas with r > 2 we rely on Lemma 14.11 below. Since the latter result requires r to be an even integer, the conditions in the theorem are to be applied in practice by taking r as the nearest even integer below the highest existing absolute moment. Gorodetskii (1977)achieves a further weakening of these summability conditions for r > 2 by the use of an inequality due to Nagaev and Fuk (1971). We will forgo this extension, both because proof of the Nagaev-Fuk inequalities represents a rather complicated detour, and because the present version of the theorem permits a generalization (Corollary 14.13) which would otherwise be awkward to implement. tjzt-j, = Detine W'r so that Xt = 11$+ W,and W and J''f ?Zt-j and J// Z7z of Xt, whose contribuare independent. Think of 54 as the Jcx-measurable x. tion to the sum should become negligible as t =
ZJ;0
Stail'
--
14.11 Lemma lf the sequence (Z,l is independent with zero mean, then cr
Evlm) < 22+-1 t
m
)q0?J sup Ezlm)s j=t
(14.45)
s; 0
for each positive integer m such that sup,ucz,
2-
)<
x.
Mixing
221
Proof First consider the case where the r.v.s Zt-j are symmetrically distributed, meaning that -Zt-j and Zt-j have the same distributions. In this case al1 existing odd-order integer moments about 0 are zero, and E
t+k
lm
t+k
jzt-j
t+k
=
0./j...h,nElzt-jq ...Zt-h,n)
...
jkzzzt jzm=t
j=t
t+k =
t+k
1 1 1 Qj2I .0jmE(Zt ...Zt-jm) -./1
...
..
./l=f jmzzt 1,0J:
Aq
s X 02j sup Ezlm).
(14.46)
sf 0
j=o
The second equality holds since EZt-jL...Zt-jzy) vanishes unless the factors form matching pairs, and the inequality follows slnce, for any r.v. F possessing the 0) for j,k > 0. requisite moments, EY i*k) 2 E(YqEYk) (i.e., Covtl'? F$ x. The result for symmetrically distributed Zs follows on letting k For general Zs, 1et Zs' be distributed identically as, and independent of, zx, is independent of J6, and J 7; has symmetfor each s < 0. Then J4 X; rically distributed independent increments Zt-j Z't-j. Hence ,
-->
=
ctjzl-j
-
-
m
X
Evt
lm)
< Evt
-
k';)241<
X 02j sup E(Zt.j Z't-plm -
j=t
x
V
glm- 1
y:=0
m
62 sup Elzltm-jl j
(14.47)
,
/
where the first inequality is by 10.19, the second by (14.45), and the third is the cr inequality. . Lastly, consider condition 14.9(c). This is designed to pin down the properties of the inverse transformation, taking us from the coordinates of (X,) to those of (4J 14It ensures that the function of a complex variable 0@) possesses an anajzzorykxl.for xl S 1. The particular property needed and lytic inverse I(x) l coefficient sequence (':y) is absolutely condition the by is implied the that under 14.9(c) the inverse representation is also summable. If Xt X; TJ 1 if % 1. An effect of 14.9(c) is Note that 'Io detined, as Zt example where 0@) rule for 01(x)41 x) out cases, as to with 01(.)a summable polynomial. The differencing transfonnation does zltp/yield a mixing process in general, the exception being where it reverses the previous integration of a mixing process. For a t'initenumber of terms the transformation is conveniently expressed using matrix notation. Let .
'
=
x
=()0.j4s,
=
=
tover-differenced'
=0pms.
=
=
=
-
Theory of Stochastic Processes
222 1
0
01 An
01
=
n x n),
(14.48)
0a-2 0n-1 0n-2
01
(-10jzt-j, Zy=0 z (z1,...,za)
1 n can be written x A?g where x is also lower triangulat', with elements = @1,...,ak)and z 0,...,a 1. If >' (v1,...,va)the vector &' Aa # has h replacing 0j for J t-1 f for 1 These operations elements Zyzoljvr-j, can in principle be taken x, subject to 14.9(c). to the limit as n so
that the equations xt z
-1
t
=
,.
=
..,
A,T
= .
=
.
=
-1
%
=
-
=
...,n.
=
,
-->
Proof of 14.9 Without loss of generality, the object is to show that the c-fields 0 = c(...,X-1,Xo) and Ji'x' x. The F-x m+2,..., ) are independent as m m+1 c( X-+1 result does not depend on the choice of origin for the indices. This is shown for a sequence f.f t )7+k for finite p and k, and since k and p are arbitrary, it then follows by the consistency theorem (12.4)that there exists a sequence (XfJ'lx whose finite-dimensional distributions possess the property for every k and p. This sequence is strong mixing on the definition. where 2% = Xt-p,...,Xo)' (px 1), #1 Define a p + m + k-vector X X6X1,XL)' = Xm+L,...,Xm+' k x 1), and also vectors W = (X1,...,Ak)' m x 1), and Xz (W(,W)' and F (FJ,FJ)'such that #1 IFI + F1 and Xz +2 + Fa,.(The elements and F are independent of W. of W and F are defined above 14.11.) The vectors Now, use the notation X = GXg,...,Xt) and define the following sets: ,x
=
-->
=1-p
=
=
=
=
=
.%
0 for some C e e C) e @1-p,
G
=
f(t):
#(,()
H
=
(:
A%() e
DJ
E
=
f ):
Fz()
#) e
e
for some D m e J;'7l,
G
TP ,
Bk ,
10 -x,
where # (p2:Ir21 f n) e T k lv21 denotes the vector whose elements are the is a vector of positive constants. absolute values of p2, and z1 = (q,,,+1,...,n,,,+z)' Also define D pa = (w2:.,2 +#2 e D) e T k =
,
-
.
H may be thought of as the random event that has occun-ed when, first F2 = 1/2 is realized, and then W2 e D p2. By independence, the joint c.d.f. of the variables (Wa,F2,A%) factorizes as F = FwzFh,xo (say) and we can write -
#(&
=
#(X2 e D)
JsJss.va#F(+2,V2,20) =
=
jepjekzvltdFvz,xvl'x'
(14.49)
Mixing
223
where
:(1/2)
#(W2 e D
=
-
v2)
jp-vgdFwzw.
=
( 14.50)
These definitions set the scene for the main business of the proof, which is to show that events G and H are tending to independence as m becomes large. Given 5IB P'm'k ility of X this is sufficient for the result, since C and D are arbitrary. By the same reasoning that gave (14.49),we have -measurab
,
#(G Detine k*
E4
/'7 HIA
supyzssztpz) and
=
X.PIGcjE)
jcjg(v2)#Fvz,xa(v2,&).
=
x.
infpzssztva), and
=
(14.51)
(14.51)implies (14.52)
f PIGCHIAEI < z.PGcjE4.
Hence we have the bounds
PCGchP
+#45rn PCGCDHI-?EI
=
z*#(G)
; and similarly, since
-
k k.P(G4 f
(14.53)
+#(Grnz;faFC)
k z.PGcjE)
= z.P(G)
=
PEc),
+
z. K 1,
PGcllj
Choosing G
H(nEcl
(i.e., C x.
Ec) + #(G chHcj Ec4
fa
P(Ec4.
R#) in
=
PCEC4
-
-
k.PG
( 14.54)
(14.53)and (14.54)gives
f #(& S
z*+
PCEC),
in particular
(14.55)
and combining all these inequalities yields 1#(G rn /f)
1 f x* x++lPEc).
#4G)#(&
-
-
(14.56)
Am+kZ,where Z (Z1,...,Z,,,+k)'and Amn is defined by (14.48).Since I4,?,+I 1 and the (z1,...,Z,,,+kl are independent, the change of variable formula from 8.18 yields the result that G is continuously distributed with Write IP
=
=
=
m+k
/(w) Define B '
(r:
=
y.
-z.
A'I
=
0 v2 ,
G
fztz)
=
#1
I
-z(0)
u 2 sup x(v2) vzB S 2 sup v2G#
JsI
G
11fz,(zr).
=
(14.57)
>1
Bm+k Then the following relations hold: .
j
/ctwz + #2) /c(w2) -
I
dw1
Theor.v of Stochastic Processes
224
jpm..k I /vt.'
u 2 sup, B G
&,
+
v)
l
-yv(w)
dw
m+k
m+k
flJz/z,+ J.,a+z
= 2 #sup. B'
t'%
*'
6
E
p e B'
I-lJz/tz,)
-
>1
>1
m+k
< ZM sup
r)
19,1
(14.58)
,
>-+1
where it is understood in the final inequality (whichis by 14.10) that );, l f W here 6 is defined in condition 14.9(a). The tllird equality substitutes 7 = Am-+1kv 0 if vl = 0 by lower triangularity of Amn. For v e and uses the fact that ;'I #' note that =
1
m+k
m+k
77 I I
l
t-m-
77 jszj 77 zjvt-j
v-,
1=-+1
>-+1
t-m- 1
m+k
S
77 jzujj :7
>-+1
m+k
x
XI
I&ln,-; K
':/1
/=0
A'lnr,
(14.59)
t-nm
assuming has been chosen with elements small enough that the terms in parentheses in the penultimate member do not exceed 6. Tlzis is possible by condition 14.9(c). For the final step, choose r to be the largest order of absolute moment if this is does not exceed 2, and the largest even integer moment, otherwise. Then 'rt
PEc)
lF21 >
P
=
'q)
m+k
P
=
U f I141
>-+1
>
n,l
m+k
m+k
77#(I7,I <>-+1
>
n,)K
>7FI7,I
rn-,r,
(14.60)
J=ra+1
by the Markov inequality, and E j 7, Ir < sup E Iz, IrG,(r),
(14.61)
'
where Gt is given by (14.43),applying 11.15 for r 2 (see(11.65) for the required extension) and Lemma 14.11 for r > 2. Substituting inequalities (14.58), (14.59), (14.60),and (14.61)into (14.56)yields m+k
l#(G
chffl
-
PG4PH4
I
X (n,
+
t=m+3
G(r)n;3.
(14.62)
Mixing
225
i 0 by 14.9(b), it is possible to choose m large enough that (14.59)
Since Gt
andhence (14.62)hold l P(G r'3S)
with
n,
Gtr) 1R1+8 G t (rln-r t for each f > m. We obtain
=
=
m+k
*(G)#(S)
-
l
d
t%(r)1/C1+r' 77 >-+1 X
K
1/(1+r)
77G'r) /=-+1
(14.63)
,
where the right-hand sum is finite by 14.9(b), and goes to zero as m completes the proof. w
This
-->
x.
It is worth examining this argument with care to see how violation of the conditions can lead to trouble. According to (14.56),mixing will follow from two O conditions: the obvious one is that the tail component F2, the F-x-measura bje part of A%,becomes negligible, such that is, #(F) gets close to 1 when m is large, even when Tl is allowed to approach 0. But in addition, to have x* x. disappear, 0, for any D, and whatever #(W2 e D r2) must approach a unique limit as 172 the path of convergence. When the distribution has atoms, it is easy to devise examples where this requirement fails. In 14.6, the set Bt becomes Js Bt on being translated a distance of 2-l For such a case these probabilities evidently x. do not converge, in the limiting case as t However, this is a suftkiency result, and it remains unclearjust how much more than the absence of atoms is strictly necessary. Consider an example where the distribution is continuous, having differentiable p.d.f., but condition (14.42) none the less fails. -
-->
-
-
-..-)
g
.;
14.12 Example Let fz) Coz sin (z ), z e R This is non-negative, continuous everywhere, and bounded by C()c-2 an d hence integrable. By choice of Q we can *fztdz 1, so f is a p.d.f. By the mean value theorem, have =
.
=
Ifz
+
ab
f(z) I
-
=
Ia IIf'z
wheref '(z) 8c osint/lcost/k x and hence, =
-
I
+ a(z)J)
,
2c()sin2(/)g-3
(14.64)
a(z) e (0,11, .
But note that
J+OI f'l ldz
=
-x
1
1
:*
Ifz
J
' Ia I
.-
+ J)
f U)ldz
-
'--h
'x'
as
Ia l
-->
0,
(14.65)
which contrdicts (14.42).The problem is that the density is varying too rapidly in the tails of the distribution, and Ifz + a4 fz) l does not diminish rapidly 0. enough in these regions as a The rate of divergence in (14.65)can be estimated. 15 F or fixed (small)a, sin (z+ a) 4 1 (or 0) Ifz + a) fz) I is at a local maximum at points at which tz)4 z4 4.3 + o(u2) and sin z4 0 (or 1), or in other wor ds w here (z+ :+/2. The solutions to these approximate relations can be written as z = +CI lJI -1/3for C 1 > 0 At these points we can write, again approximately (orders of magnitude are all we need here), -
--
=
-
=
-
.
=
=
Theoty of Stochastic Processes
226
Ifz
a4
+
fz) I S
-
-2 1J
13
I (-C1 It7I 21
lfz)
S ICOC 1
.
-1/3
-1/3j
C 1 jt7l by The integra l is bounded within the interval 1/3 2/3 rectangle having height 2C0C7-1 a I Outside the 4CoC -1 th e area of the I la I and h -2 is bounded by t e integral over this region is bounded by interval,J coz ,
,
.
,
+00 2C0Jcj$(,I-v3Z
Adding up the approximations
dz
for M
<
IJl
2C0Cl
=
.
yields
+x
J..I fz
1/3
.j
.p,
+ J)
fzs 1dz f
-
Ml J
l
js
(14.66)
o
x.
The rate of divergence is critical for relaxing the conditions. Suppose instead of
(14.42) that +oo
J-.I
J)
z+
zbIdz f Mtl
-
could be shown sufficient, where h ltzI) 1, 0 as Ia l 0. Since
JI ), IJ1
f y y s (;
(14.67)
,
an arbitrary increasing function with
(.) is
.
+co
J-.l fz
+ J)
ldz S
f
-
g)+x
(z)tyz m
..f
(14.68)
g
for any a, (14.67)effectively holds for any p.d.f., by the dominated convergence 16 theorem. Simple continuity of the distributions would suffice. This particular result does not seem to be available, but it is possible to relax 14.9(a) substantially, at the cost of an additional restriction on the moving average coefficients. 14.13 Corollary Modify the conditions of 14.9 as follows: for 0 that
;$S
<
1, assume
Zt is uniformly fv-bounded, independent, and continuously distributed with p.d.f. /'z,,and zz-va, (+I sup t
whenever
b-)x-
(
(C ')
tzz)
ct
1/0(x)
<
Itzl
S
(r)I5'(I5+r) <
=
T(x)
-
,
etxl
=
Then Xt is strong mixing
hjz)
:, for some
I
dz
u Mlcl', u
<
(14.69)
-.,
0,'
>
c,(r) is defined in (14.43); for 1x1S 1, and Z'r=I E''=l'r/ J (Tjl 15< J x
,
where
x.
with
m
=
t?(X*t
.,,,+1
Gt (r)lVO+8) .
Proof This follows the proof of 14.9 until (14.58),which becomes
Mixing
227
m+k
z.
-x
+
<
zu
applying the obvious extension m+k
m+k
9,1F
X I
=
using (9.63),since 0
<
y B'
f=?n+1
(14.70)
,
'
of Lemma 14.10. Note that ;$
f-m-j
X X
p=+1
p=?n+l
>'')lf;,1;'
sup
./=0
m+k
x
K
'zjvt-j
XI
X n?
'zj
j=0
(14.71)
f=/?1+1
;'J f 1. Applying assumption
14.13(c'),
m+k
I#(G r-bz,r? -#(G)#(zoI f<
and the
result
is obtained
77 (nf)+c,(r)n-,r),
(14.72)
>-+1
as before, but in this case setting
n,
=
G1,7t+* .
.
Condition 14.13(b') is satisfied when l0./1 ./-B for g > 1/p + 2/r when r < 2, and g > 1/2 + 1/r+ 1/j3 when r 2, which shows how the summability restrictions have to be strengthened when ;'$is close to 0. This is none the less a useful extension because there are important cases where 14.13(b') and 14.13(c') are easily A, both I%I and I satisfied. In pmicular, if the process is finite-order vanish beyond some finite j, and (b') and (c') either decline geometrically or both hold. Condition 14.13(a') is a strengthening of continuity since there exist functions h.) which are slowly varying at 0, that is, which approach 0 more slowly than any positive power of the argument. Look again at 14.12, and note that setting 13 la will satisfy condition 14.13(a') according to (14.65).lt is easy to generalize k Csin 2(z)z for k k 4 the earlier argument is easily the example. Putting j'z) modified to show that the integral converges at the rate Ia 1.1/(k-1)an d tus choice of f'sis appropriate. But for J(z) = Csin 2 e z)z th e integral converges F more s lowly than Ia I for al1 13> 0, and condition 14.13(a') fails. To conclude this chapter, we look at the case of uniform mixing. Manipulating inequalitie (14.52)414.55)yields 'ry1
=
-2
=
,
,
-2
IPIH' G) #4/./)1 S -
(1
z* z. PCEY + -
+
plv) ,
(14.73)
1 for all m exceeding a which shows that unifonn mixing can fail unless #(F) tinite value. Otherwise, we can always construct a sequence of events G whose probability is positive but approaching 0 no slower than PEc4. When the support of X-v,...,Xo4 is unbounded this kind of thing can occur, as illustrated by 14.8. =
The essence of this example does not depend on the AR(1) model, and similar cases could be constructed in the general MA(x) fzamework. Suftkient conditions must include a.s. boundedness of the distributions, and the summability conditions are also modified. We will adapt the extended version of the strong mixing condition in 14.13, although it is easy to deduce the relationship between these conditions and 14.9 by setting f$ 1 below. =
228
Theoty of Stochastic Processes
14.14 Theorem Modify the conditions of 14.13 as follows. Let (a') and (c') hold as before, but replace (b') by (b'') f.o I j I)l5< x, add and (d) (Z,l is uniformly bounded a.s. T hen (mJ is uniform mixing with (,,, O; =,,,+1(E7=f I0./1)f5). XO
xt
=
Proof Follow the proof of 14.9 up to (14.55),but replace (14.56)by (14.73).By condition 14.14(d), th'ere exists K < x such that suprl Zt I < K a.s., and hence I-Y,I 1 < K;=o I0.j1a.s. It further follows, recalling the definition of Fa, that PE) +k. Substituting directly into (14.73) 0./1 for t m + 1,...,< A'Z7=,1 when n, of and a nd making choice this from (14.70) (14.71), n, gives (forany G with #(G) > 0) =
=
,
I,(,.,1
c) rm I -
,
-s+(s-yojj)
(14.74)
.
l=?n+1 j=t
The result now follows by the same considerations as before.
.
x in the These summability conditions are tougher than in 14.13. Letting r OU-B) 14.13(b') for for comparability, satistied is when latter case I%I g, > 1/2 + 1/p, while the corresponding implication of 14.14(b'') is jt > 1 + 1/p. -->
=
15 Martingales
15.1 Sequential Conditioning It is trivial to observe that the arrow of time is unidirectional. Even though we can study a sample realization expost, we know that, when a random sequence is member Xt is determined in an environment in which the generated, the previous members, Xt-k for k > 0, are given and conditionally fixed, whereas the membersfollowingremaincontingent. Thepastisknown, butthefutureis unknown. The operation of conditioning sequentially on past events is therefore of central by specifyimportance intime-series modelling. Wecharacterizepmialknowledge ing a c-subtield of events from @, for which it is known whether each of the events belonging to it has occurred or not. The accumulation of infonnation by an observer as time passes is represented by an increasing sequence of c-fields, F.17 1T,1= suc h that 51 i i @-1 i J70 Tl If Xt is a random variable that is Tr-measurable for each /, ( Tfl=-xis said to The pairs (A,@,)=-x are called an adapted be adapted to the sequence ta)l=-x. /) < s < defines the minimal adapted sequence, but sequence. Setting 9t ctxx, 5t typically has the intepretation of an observer's infonnation set, and can contain more information than the history of a single variable. When Xt is integrable, the conditional expectations A'(ml Fr-1) are defined, and can be thought of as the optimal predictors of Xt from the point of view of observers looking one period ahead (compare10.12). Consider an adapted sequence l&,Tnlo-x on a probability space L't,5nP4, where ( kl is an increasing sequence. If the properties dcurrent'
...?
-x,
...
-x
=
FI,L1 < Esn
I
.n-1)
=
(15.1)
x, s%-l,
a.s.,
(15.2)
hold for every n, the sequence is called a martingale. ln old-fashioned gambling parlance, a mmingale was a policy of attempting to recoup a loss by doubling one's stake on the next bet, but the modern usage of the term in probability theory is closer to describing a gambler's worth in the course of a sequence of fair bets. In view of (10.18),an alternative version of condition (15.2)is Sndp
A
=
Sn-)dP, each A e @n-1.
A
Sometimes the sequence has a tinite lnitialindex, and may be written where 5'1 is an arbitrary integrable r.v.
(15.3)
f,L,F,,lT
F/ldt?ry of Stochastic Processes
230
(X,ITbe an i.i.d.
integrable sequence with zero mean. If Sn Z7=l-Y, and 5n c(Xn,X,,-1,...,X1), l5'q,?LlT is a mmingale, also known as a A'lil walk sequence. Note that S Z7=1A'l-Y,l < x. n random
15.1Example
Let
=
=
15.2 Example Let Z be an integrable, T/f-measurable, zero-mean r.v., ( Kl'-x an increasing sequence of c-tields with limn-yxfn = T, and Sn #tzj ;n). Then E(Sn3 9a-1) F(F(zI@n)I @n-1) F(ZIT,,-1) = .L-1, (15.4) =
=
wherethe second o martingale.
=
equality is by 10.26(i). F1 ,L1 <
FIzI <
by 10.27, so Sn is a
x
on the last definition, a martingale d#erence (m.d.) sequence on L't,5,P) satisfying the properties
Following
fm,sl'<'-x is an adapted sequence
Fl#r1 F(Al Tr-l)
(15.5)
< x, =
(15.6)
0, a.s.,
for every t. Evidently, if fSn) is a mmingale and Xt St St-j. then (.1 is a m.d. Conversely, we may define a martingale as the partial sum of a sequence of m.d.s, as in 15.1 (an independent integrable sequence is clearly a m.d.). However, if Xt has positive variance unifonnly in /, condition (15.1) holds for a1l finite n but not unifonnly in n. To define a mmingale by Sn L1=-=Xt can therefore lead to difticulties. Example 15.2 shows how a mmingale can arise without reference to summation of a difference sequence. It is important not to misunderstand the force of the integrability requirement in (15.1).After all, if we observe Sn-b, predicting Sn might seem to be just a matter of knowing something about the distribution of the increment. The problem is that we cannot eat Esn I&-1,...)as a rt7rltftprl variable without integrability of Sn. Conditioning on S-l is not the same proposition as treating it as a constant, which entails restricting the probability space entirely to the set of repeated random draWings of Xn. The latter problem has no connection with the theory of random sequences. A fundamental result is that a m.d. is uncorrelated with any measurable function of its lagged values. =
-
=
15.3 Theorem If (7r,@t)is a m.d., then Cov(X,,9(.X)-1,X,-2,...))
where
(1) is
any Borel-measurable,
Proof By 10.11
(sze also
=
0,
integrable function of the arguments.
the remarks following) noting that 9(A-1,.&-2,...), is
Tf-l-measurable. w
15.4 Corollary lf (.,121 is a m.d., then E(XtXt-k) = 0, for a1l t and all k
#
Proof Put () Xt-k in 15.3. For k < 0, redefine the subscripts, putting t and t' Il7I t, so as to make the two cases equivalent. m
t k
=
-
=
=
0. -
Martingales
231
One might think of the m.d. propel'ty as intennediate between uncorrelatedness and independence in the hierarchy of constraints on dependence. However, note the asymmetry with respect to time. Reversing the time ordering of an independent sequenceyields anotherindependent sequence, andlikewise areversed uncorrelated sequence is uncorrelated; but a reversed m.d. is not a m.d. in general. The Doob decomposition of an integrable sequence (&,?L)7' is Sn
where Ao
=
0, Mo
(15.7)
Mn +An,
=
%, and
=
Mn An
Mn-t
=
=
.4u-1
I k-1),
(15.8)
Esn lTn-1) 5'n-1.
(15.9)
+
+
Sn Esn -
-
component of Sn.Writing Anis an Fn-l-measurable sequence called Lbepredictable and A,4u hsl'n hSn Xn, we find F(Fn l;t-1), and L =
%
=
=
Xn
=
rn -
E(Ynl?h-1).
(15. 10)
Xn is known as a centred sequence, and also as the innovation sequence of Sn. It is adapted if ( l'n,iu J7'is, and since Fl Fn1 < oo by assumption, E IXn I < E Il',yI + E IEYn lJ;,,-1)I
l Fn IITn-l))
E 1Fn I + EE
=
IE
IraI <
(15.11)
x,
Minkowski' s inequality, the conditional modulus inequality, and by (respectively) 0, (.L,@n)7is a m.d. and so the LIE. Since it is evident that E(Xt I?h-1) mmingale. is lMn,Tnl7 a =
Martingales play an indispensable role in modern probability theory, because m.d.sbehave inmany importantrespects likeindependentsequences. Independence limit results, laws of is the simplifying propel'ty which permitted the large numbers and central limit theorems, to be proved. But independence is a constraint on the entire joint distribution of the sequence. The m.d. property is a much milder restriction on the memory and yet, as we shall see in later chapters, most limit theorem: which hold for independnt sequences can also be proved for m.d.s, with few if any additional restrictions on the marginal distributions. For time series applications, it makes sense to go directly to the martingale version of any result of interest, unless of course a still weaker assumption will suftke. We will rarely need a stronger one. Should we prefer to avoid the use of a definition involving c-fields on an abstract probability space, it is possible to represent a martingale difference as, for example, a sequence with the property iclassical'
E(xt IA-1,Xr-2,...)
=
0 a-s.
( 15. 12)
When a random variable appears in a conditioning set it is to be understood as representing the corresponding minimal c-subfield, in this case c(#f-1sXf-2,...).
Theory of Stochastic Processes
232 tj
This is appealing at an elementary level since it captures the notion of information available to an observer, in this case the sequence realization to date. But since, as we have seen, the conditioning information can extend more widely than the llistory of the sequence itself, this type of notation is relatively clumsy. Suppose we have a vector sequence (4XsZf)), and Xr-though not necessarily c(XsZsXr-1,Zf-1,...) in the sense of Zt is a m.d. with respect to 5t (15.6). This case is distinct from (15.12),and shws that that definition is inadequate, although (15.16)implies (15.12).More important, the representation of conditioning information is not unique, and we have seen (10.3(ii))that any measurably isomorphic transformation of the conditioning variables contains the same information as the oliginal variables. lndeed, the information need not even be represented by a variable, but is merely knowledge of the occurrence/nonoccurrenceof certain abstract events. =
-
15.2 Extensions of the Martingale Concept k Jn=1, w here An adapted triangular array ( tAk, @nf)?21 sequence of integers, for which '-'
'1Xnrl
<
(/(n )= n,z 1 is some
(15.13)
oo,
ft-vtl 9$,f-1)
(15.14)
0 a.s.
=
increasing
for each t 1,...,lk and n k 1, is called a ntartingale dterence array. In many applications we would have just kn n. The double subscripting of the subfield 5nt may be superfluous if the infonnation content of the array does not depend on n, with 5nt 5t for each n, but the additional generality given by the definition k Xnran d 5 n is harmless and could be useful. The sequence t&,Tn ITwhere Sn Xf21 5n,knis not a martingale, but the properties of mmingales can be profitably used to analyse its behaviour. Consider the case Sn = n -1/2E)=1 xt w Iwre (m,@;)is m.d. distdbution of Sn has a Such scaling by sample size the that may ensure a non-degenerate limit. Sn is not a martingale since =
=
=
=
=
ITn-1)
Esn
1/2
E(/l 1)/n1
=
-
5'n-l,
(15.15)
but each column of the m.d. array X1 2 -
-1/2
1/2
X) 3
2 -1/2X2 3
-
X1 4-1/2x 1 X1 4-1/2x 2
.
.
.
.
.
.
.
.
1/2
3 -1/2Xa
(15. 16)
-1/2
4
X5
4 -1/2.X4
is a m.d. sequence, and S,l is the sum of column n. It is a tenn in a mmingale sequence even though this is not the sequence (5'n1. An adapted sequence
of fal-bounded (&,@s)=-x
variables
satisfying
Martingales
IFa)
ESn+L
233
k Sn a.s.
(15.17)
is called a submartingale, in which case Xn Sn &-l is a submalingale difference, having the property E(Xn+3lJ;n) 2 0 a.s. In the Doob decomposition of a submartingale, the predictable sequence An is non-decreasing. Reversing the although, since -Sn is a supermmingale inequality defines a supermartingale, extension. A supermartingale might submartingale, this is minor whenever Snis a a represent a gambler's worth when a sequence of bets is unfair because of a house percentage. The generic tenn semimartingale covers al1 the possibilities. =
-
15.5 Theorem Let 94.):R i.-,b R be continuous and convex. If (&,TnJis a martinthen (9(&),Fn) is a submmingale. If () is also nongale and Fl ((Sn) l < decreasing, (Y(&),8L ) is a submartingale if (&,8Ll a submartingale. =,
Proof For the martingale case,
ITn) f(#(.%+1)
#(F(&+1 I@n))=
a.s. #(u%)
(15.18)
by the conditional Jensen inequality (10.18).For the submartingale case, in (15.18)when xl S x2 = (1)(x1) < 9(.n).x becomes
4='
E2'
If (.X),TfITis a (sublmmingaledifference, (Zr,@rl7' any adapted sequence, and
sn Xx,z,-1, ,.1
(15.19)
=
then
(&,L )1' is Esn..t
a
since (sublmartingale
I@n)
=
XX,z,-l >1
+
ZnEXn+3I?L)
=
Sn
(
(15.20)
'n).
We might think of Xt as the random return on a stake of 1 unit in a sequence of bets, and the sequence (4) as representing a betting system, a rule based on information available at time t 1 for deciding how many units to bet in the next game. The implication of (15.20)is that, if the basic game (in which the same stake is bet every time) is fair, there is no betting system (basedon no more than information about past play) that can turn it into a game favouring the player or for that matter, a game favouring the house into a fair game. ,5,P), a stopping time For an increasing sequence (1,1 of c-subfields of is a random integer having the property f : t ) e 5t. The classic example is gambling which entails withdrawing policy from the game whenever a certain a conditiop depending only on the outcomes to date (suchas one's losses exceeding some limit, or a certain number of successive wins) is realized. If is the random variable defined as the first time the said condition is met in a sequence of bets, it is a stopping time. be a stopping time of f@n Let ), and consider -
-
..:(r4))
'r()
=
'r
'r
Snxz
=
Sn, n ;
%, n
':
> T
(15.21)
Theory of Stochastic Processes
234
where n zx T stands for minf
l,'rl.
fSnoysn
l'X'n.l
is called a stopped process.
then t,Sv,T,, 15.6 Theorem If fSn,5n 1Tis a martingale (submartingale), martingale (submartingale).
JTis
a
Proof Since f 5n 1Tis increasing, f ): k T()) ) e 5n for k < rl, and hence also -1 1(t=z) + n j (a
'r()))
=
n-1
FlSnvl
f TE
151171(1....,:) l +e'lx%1(n
&1
l
n-1
+FxL l < < Xflxsj =1
n k 1.
=,
(15.22)
lf (,L,Tn JTis a martingale then for A e 5n, applying (15.3),
jghnwbjxxdp =
=
jgcsqncgljdp Jxc,jsszlx'lzd' +
Sndp +
Xfafn<ecl
S4P
drnfnll
Snszdp,
=
A
(15.23)
showing that (&+1,u,@n)T is a martingale. The submmingale case follows easily on replacing the second equality by the required inequality in (15.23).* The general conclusion is that a gambler cannot alter the basic fainzess characteristics of a game, whatever gambling policy (bettingsystem plus stopping rulel he or she selects. Al1 these concepts have a natural extension to random vectors. An adapted sequence (A),@r)=-xis detined to be a vector rnzzrffrlgtzfe dterence if and only if (A'X,F,J*-xis a scalar m.d. sequence for a11conformable tixedvectors A # 0. It has the property
f(X+1I5tj
=
(15.24)
0.
The one thing to remember is that a vector martingale difference is not the same thing as a vector of martingale differences. A simple counter-example is the twois an element vector Xt = XoXt-q)'s where Xt is a m.d.; fLX, + V2.Y;-1,@,J adapted sequence, but EhLxt
+
L4-j J?h-1)
=
l2A-j
#
0,
so it is not a m.d.. On the other hand,
f(l-V+l + klxt ITr-l) but (:14+1+ hzxt, ?&) is not adapted, since
=
ml
0, is not Tsmeasurable.
Martingales
235
15.3 Martingale Convergence Applying 15.5 to the case ()4.) l IP and taking unconditional that every martingale or submartingale has the property =
FI5'n+11: 2
.
F1 snlP, p
expectations shows
1.
(15.25)
x, either to a By 2.11 the sequence of pth absolute moments converges as n uniformly where +x. the limit In bounded, Ll-norms finite the are or to case (sublmartingales also exhibit a substantially stronger property; they converge, almost surely, to some point which is random in the sense of having a distribution over realizations, but does not change from one time period t to the next. The intuition is reasonably transparent. (Tnl is an increasing sequence of c-fields which converges to a limit Tx c ;, the c-field that contains nnfor every n. Since Esn ITn) Sn, the convergence of the sequence (Tn) implies that of a uniformly bounded sequence with the property ESn+L I1n) k Sn, so long as these expectations remain well-defined in the limit. Thus, we have the following. --
=
15.7 Theorem If (&,;n)Tis a submartingale sequence and supnfl Sn l f M < x, then Sn S a.s. where S is a i-measurable random variable with FI5'1 S M. Ia --
The proof of 15.7, due to Doob, makes use of a result called the upcrossing inequality, which is proved as a preliminary lemma. Considering the path of a submartingale tluough time, an upcrossing of an interval (a,p1 is a succession of steps starting at or below G and terminating at or above p.To complete more than one upcrossing, there must be one and only one intervening downcrossing, so downcrossings do not require separate consideration. Fig. 15.1 shows two upcrossings of ga,pq, spanning the periods marked by dots on the abscissa.
O O
O O
O O
O O
Fig. 15.1 Let the r.v. Fk be the indicator of gn upcrossing. and then, for k = 2,3,...,n,
To be precise, set
i'I
=
0,
236
Theory of Stochastic Processes
0 if either
1%
Fz-I
=
0, Sk-k
=
0, Sk-k K
a, or l'-1
>
=
1 if either Fz-j
=
1, Sk-j k
or Fk-1 = 1, Sk-j <
(x,
p, p.
(15.26)
The values Of l'k appear at the bottom of Fig. 15.1. Observe that an upcrossing begins the period after Sk falls to or below a, and ends at the tirst step thereafter where ;$is reached or exceeded. 1%is a function of Sk-k and an K-l-measurable random variable. The number of upcrossings of (a,j$)up to time n of the sequence (,s'a((l))17, to be variable. The denoted &a()), is an Fa-measurable random sequence ( 1&4)1Tis condition. satisfies but it following the monotone, by a submartin15.8 Upcrossing inequality The number of upcrossings of g(x,f$J gale (&,@aJTsatisfies Eun)
E lsn1+ Ia
f
l
j) tx
(15.27)
.
-
Proof Define Sn' maxlxa,Gl, a continuous, convex, non-decreasing function of Sn, such that fSn',snj is an adapted sequence and also a submartingale. Un is the set of upcrossings up to n for (&') as well as for (&). Write =
n
Sn 5''l -
=
NXYj X Fkyj X (1 a2
n
X (1 *=2
+
-
J%)xL/
(15.28)
,
&2
&2
where l'k is f'rom (15.26),and E
11
& =
XLis
a submmingale
difference. Then
l
-
i'1)xI
=
X jyg-oixLap k=2
=X a2 jl
ExLbsk-bldp. e,
1*k=01
(15.29)
using the definition of a conditional expectation in the second equality (recalling that i'k is g-j-measurable), and the submartingale property, to give the inequality. We have therefore shown that Esn'
,$() -
l'kAk' X &2
k E
(15.30)
.
X2=21%Xis the sum of the steps made during upcrossings, by definition of J%. Since the sum of the XLover an upcrossing equals at least ;$ G by detinition, we must have -
M
X l%-YI(p
-
a)&s,
(15.31)
&2
where Un is the number of upcrossings completed by time n. Taking the expectation of (15.31)and substituting (15.30),we obtain, as required,
Martingales
(;3 -
a)F(&n) f Eu
5') f Esn'
-
-
237
a)
-
Ifs-ozjsdp
-
E 1Sn I + Ia
< FIsn a 1 < -
(15.32)
I .
w
The upcrossing inequality contains the implication that, if the sequence is uniformly bounded in f.l, the expected number of upcrossings is finite, even as n .,..4x. This is the heart of the convergence proof, for it means that the sequence has to be settling down somewhere beyond a certain point.
f3>
Proof of 15.7 Fix (x and
a. By 15.8,
e'1snI + laI
EUn) <
j3 a
M+ l(x I
S
f3 a
<
(15.33)
x.
-
-
is a positive, non-decreasing sequence and either diverges For ) e (, (!.&((l))J:' x. +c,o Divergence for (t) e C with to or converges to a finite limit U(@ as n whichcontradicts #(C) > 0 would imply EUn) (15.33),so Un-.,-qUa.s., where --
--x,
fl&)
<
=.
and s((1)) R(t)l limsupn-oxxS
liminfu-yxs'n. lf s()) < (x < I < R)), the x, interval (a,Iqis crossed an infinite number of times as n so it must be the < < < pair is for (x a,p. Hence considef any I X) 0. This tnle case that #(j' Define
=
=
-,..:
=
lt0;srtl
<
iltllll
Uls' f
=
tx
;$f X1,
<
a,fs where the union on the right is taken over rational values of (x and #(j' < X) 0 by 3.6(ii), which is the same as j' S S a.s., where of fSn ) Finally, note that =
=
=
(15.34)
0. Evidently,
,5'
is the limit
.
FI5:1 S
liminf E Ivn
If
sup
'l-/oo
FIsnI <
(15.35)
M,
n
where the first inequality is from Fatou's lemma and the last is by assumption. This completes the proof. . Of the examples quoted eaflier, 15.1 does not satisfy the conditions of 15.7. A random walk does not converge, but wanders forever with a variance that is an increasing function of time. But in 15.2, Xt is of course converging to Z. 15.9 Corollary Let fSu,5n l=-xbe a doubly infinite mmingale. where S-x is an Al-bounded nv. as n
Then Sn
--->
S-= a.s.
-->
-x,
Proof
Let U-n denote ihe umber of upcrossings of (a,j31performed by the The argument of 15.8 shows that k j 2 f'j, -1
-nl.
sequence
EU-i,4 ;
FIsj I + Ia I pra
,
all n
1.
Arguments precisely ana 1og ous to those of 15.7 show that
(15.36)
Theoty of Stochastic Processes
238
# liminf Sn < limsup Sn
0,
=
(15.37)
n'-i-co
N-)-x
1
- is non-nega tive nonsequence (FI5k1 1-x so that the limit 5'-x exists a.s. cxo by a nd decreases < <Sdefinition of a martinby increasingas n (15.25), FI l gale. Hence FI5'-.I < x. . If a martingale does not converge, it must not be thought of as converging in F, 'rhe
,
of heading off to +x or never to return. This is an event that occurs only probability the increments having a suitably bounded distribuwith 0. Subject to tion, a nonconvergent mmingale eventually visits al1 regions of the real line, almost surely. -x,
suptlxf
15.10 Theorem Let (Xf,Ffl be a m.d. sequence with
1J=1kt.lf
C
(:
=
E
then P(E
-
C)
=
Snk
((:
=
converges
either
), and
infnxst)
>
or supnxst)
-x
I) <
<
cxa,
and 1etSn =
xl,
0.
Proof For a constant M > 0, define the stopping time Twz((o) as the smallest integer otherwise. The stopped process n such that Sn > M, if one exists, and IA(t) tx,vhn-tlxxu Tnwwl-i-rl is a martingale (15.6),and S M for al1 n. Letting fsnxzu, = S'Lzu maxf 5kZ.,OJ, =
s+nh'w
<
s+( n- 1),xw+
X'bnxzu< M+ sup
I-LI
(15.38)
.
n
= Since Ezp 0, Fj Swwl = zft5Xwl,and hence supne'l skwj < x, and Snvsxuconverges a.s., by 15.7. And since Snvsxufk Snk on the set f : supnxst) S MJ, converges a.s. on the same set. Letting M---k x, and then applying the we obtain the conclusion that Snlk converges a.s. on the same argument to that is, #(Cn E; set AD= #(A-), from which the theorem follows. . =
k%()
-%,
+x, infniat) = Since Note that Ec = ( :supnlnt) P(Cc t.p Ec), a direct consequence of the theorem is that Cc = #(m = 0, which is the claim made above. -x1
#(FC)
=
#((Cf'7F)6) ECDN where =
.
15.4 Convergence and the Conditional Variances lf (&)
is a square-integrable mmingale with diferences fAkl, E(Sln-1 + Xl + J,x S j Tu-j) k &2-j Esn 2j 5 n-1
)
=
-j
n
n n
,
1 is a submartingale.The Doob decomposition of the sequence of squares has and Su and Aa = E(X2n j5 n The the form Sn2 = Mn +A n where hhl'n X1n E(X1n j @n sequence fAn) is called the quadratic variation of f&) The following thorem reveal an intimatelinkbetween mmingaleconvergence and the summability of the conditional variances; the latter propel'ty implies the former almost surely, and AQj ?=1 E in particular, if '=' ( t 5 t-1 ) < x a.s. then Sn S a.s. -j),
=
-1).
-
.
-..4
Martingales
15.11Theorem
Let
C
then #(D
C)
=
a m.d. sequence, and Sn X:=l-V.lf xl : :7.1F(xl e @, I@,-j)() <
(X,,F,)Tbe D
-
239
=
=
0.
( f
:
=
sn
convergesl
e F,
Proof Fix M > 0, and define the stopping time ,1A#(l)) as the smallest value of n having the property Xe'(A
I@,-l)(to)
>1
(15.39)
M.
If there is no finite integer with this property then ,:A#)) = x. If DM ( : 'rA/.((l)) x), D limv-yxfk. The r.v. 1(wka)() is Fs-l-measurable, since it is known at = time n 1 whether the inequality in (15.39)is true. Detine the stopped process =
=
-
N
A7A1(zvvf)
Snxxu.
=
(15.40)
,.=1 is a snvsxu
martingale by 15.6. The increments are orthogonal, and Esnhxu)
sup n
N
=
sup E n
XV ltwkul >1
n
=
supf n
77ltwknlftl
@r-1)
<
M,
(15.41)
>1
where the final inequality holds for the expectation since it holds for each by definition of 1M((t)). By Liapunov's inequality,
supFIskwl N
S sup n
(.t) e
f
< Ml Il&.wII2
and hence Snxzu converges a.s., by 15.7. If ) e DM, xskss/)) xS4))for every n e EN,and hence Sn1 converges, except for ) tna set of zero measure. That is, PDu rn C) PDM4. The theorem follows on taking complements, and then letling =
=
M
-->
x.
.
15.12 Example To get an idea of what convergence entails, consider the case of
(Xf ) an i.i.d. sequence (compare15.1). Then Lxtlatjis a m.d. sequence for any Exlt @f-1) F(aYl) = c2, constant sequence (t7,)o f p ositive constants. Since I a which we assume finite, Sn Xlcjxlat is an a.s. convergent martingale whenever x jlat 2 < For example, at t would satisfy the requirement. a =
=
t
.1
(x,.
=
In the almost sure case of Theorem 15.11 (when#(C) #4D) 1), the summability of the conditional variances transfers to that of the ordinary varipnces, 2 Ftsupfxl) < x, the summability of the conditional variances cf2 EXt). Also w h en is almost equivalent to the summability of the Xt2 themse lves. These are consequences of the following pair of useful results. =
=
=
Theory of Stochastic Processes
240
15.13 Theorem Let lZ?) be a any non-negative stochastic sequence. (i) X*t=jEZt) < x if and only if :7 =1f(Zt1 7t-1) < cxa a.s. (ii) lf Asupzf) < x thn PD A E) 0, where =
D
=
E
=
( ): Z=f=qE(Zt197,-1)4) ( ): Z'';=lZr() < ) x
<
cxo
)
,
.
Proof (i) The first of the sums is the expected value of the second, so the if' part is immediate. Since F(ztl F,-1) is undetined unless Ezt) < x, we may assume &1=qE(Zt)< x for each finite n. These partial sums form a monotone series which either converges to a finite limit or diverges to +x. Suppose Z:=1F(Zrl Tf-1) IYf-1) 0 a.s. converges a.s, implying (bythe Cauchy criterion) that L1+=mn+3EZt &I*=mn+jEZt) Then by the monotoneconvergence theorem, so that as m A n by the same criterion k1=jEZt) k*t=jEZt) < x, as required. = (ii) Define the m.d. sequence Xt Zt F(Z,I1f-l), and 1et Sn = Z:=1m.Clearly supnskt) f k*t=1Z,((t)), and if the majorant side of this inequality is finite, Sn1l)) converges in almost every case, by 15.10. Given the definition of Xt, this 9f-1)()) < x. ln other words, PE D) 0. Now implies in turn that Z7=IF(Z/I apply the same argument to -Xt = E(Zt 111-1) Zt to show that the reverse implication holds almost surely, and PD F) = 0 also. . Konly
-->
-.+0
--)x.
--y
-
-
=
-
-
15.5 Martingale lnequalities Of the many interesting results that can be proved for mmingales, certain inequalities are essential tools of limit theory. Of particular importance are maximal inequalities, which place bounds on the extreme behaviour a sequence is capable of over a succession of steps. We prove two related results of this type. The tirst, a sophisticated cousin of the Markov inequality, was originally proved by Kolmogorov for the case where (Xr) is an independent sequence rather than a m.d., and in this fonn is known as Kolmogorov's inequality. 15.14 Theorem
Let
(&,Tn)Tbe
a martingale. For any p 2
P max lSkI > E I
E lsnj#
1Sk
Proof Define the events A1 A1 =
):
=
f
max #Sj
1u./<1
:
I>
l
kh'14)
IS
-
EP
E,
(15.42)
.
:1,
and for k
1skl) l >
e
=
2,...,n,
e 5k.
The collection A1,...,An is disjoint, and UWz =l
=
max l51
lsfn
>
e,
.
(15.43)
Martingales
c l I51
Since A1
241
el, the Markov inequality (9.10)gives
>
I IP1z:).
-pe'
PA
(15.44)
'k
<
P
By 15.5, ISn I for p 1 is a submartingale, so k f n. Since At e 5k, it follows that '%lP1Ak)
f(1
I&IP
ISnIP I1k)1xJ F( ISn
I EE
=
f
IP1zk),
lz4k where the equality applies. (10.18).Noting X2=1 required, (15.43)415.45), as
=
n
P max 1
I
.%1
>
s
for 1 <
(15.45)
1U2-lxk, we obtain from
n
X#(All
=
F(I&IPI5k4 a.s.,
<
=1
IsnIP1z:) Z-PTEC k=l
N
=
lzu PT k=1
Z-PE ISn I
1sL1#).w Z-PEC
f
(15.46)
The second result converts the probability bound of 15.14 into a moment inequal-
ity.
(&,FnJTbe
15.15 Doob's inequality Let
e'
p
Isklp s
max lffn
-
a martingale. For p p
1
s Isn I
p
lus;n
1
,%l
>
f E-1F( ISnIltmaxlsksalslo
s
(15.47)
.
(15.46)for the
Proof Consider the penultimate member of P max
>
case p
c)),
=
1, that is,
(15.48)
and apply the following ingenioud lemma involving integration by parts. 15.16 Lemma Let X and F be non-negative nv.s. If #(X > 1)1PF(F#), for p > 1. all E > 0, then F(XP) f lp
E)
S E-1F(F1(x>c)) for
-
Proof Letting Fx denote the marginal c.d.f. of X, and integrating by parts, using #(1 Fx) -#Fx and RP(1 Fx@))17 0, -
=
=
-
Exp
j,
-jhpdk
'jpdFxv
=
=
-
l
Xp(#-1(1
=
Fx(())
,
-
Fx6))X =
j*pLP-$PX o
>
j)#j.
(15.49)
> 1 when Define the function 1fx>j)@) (, and 0 otherwise. Letting Fxy denote the joint c.d.f. of X and F and substituting the assumption of the lemma into (15.49),we have =
Ex
upj-r-zstrlfxxklld
p)
0
.7
Theory of Stochastic Processes
242
(J(scsF1tx>()(I)JFx.r(I'F))X
pjo-ie-l
=
-
pj
- 11 =
jkp-zm
y
(R2)+
P
dpkvx,y)
0
jpz,-yx'--'dFkYx-ys
styxp-l;
(15.50)
.
-1
Here (R2)+ denotes the non-negative orthant of R2, or g0,x)x g0,x).The second equality is permitted by Tonelli's theorem, noting that the function Fxrt defines (R3)+. By Hlder's inequality, a c-finite product measure on ECYXP
-j ) K E ly (josl-lytxp;
Substituting into the majorant side of
(15.50)and
.
simplifying
To complete the proof of 15.15, apply the lemma to and i' putting x maxlsjss I I x .%1
,%1
=
=
gives the result. w
(15.48)to
yield
(15.47),
.
Because of the orthogonality of the differences, we have the interesting property of a mmingale l5'nlr that EzSll
=
E
XV
(15.51)
,
>1
where, with s'tl for the case p with the variance property extended
0, Xt St St-k. This lets us extend the last two inequalities 2, to link Ptmaxlsksa lSnl > :) and Ftmaxlxlsa) directly of the increments. It would be most useful if this type of to other values of p, in particular for p e (0,2). One approach to this problem is th von Bahr-Essen inequality of j11.5. Obviously, 11.15 has a direct application to martingales. =
=
-
=
15.17 Theorem
lf
is a (aYf,7f1T
m.d. sequenc
and Sn
=
1=3Xt,
E lsnIP < 2:7 E lx,IP
(15.52)
>1
for 0
<
p f 2.
Proof This is by iterating 11.15 with F Xn, V Tn-la and X = as in the m.d. sequences just as argument leading to (11.65);note that the latter holds for kv-j,
=
for lndependentsequences.
=
.
Another route to this type of result is Burkholder's
inequality (Burkholder
1973). 15.18 Theorem
Let (&,Fn
ITbe a martingale
with increments Xt = St St-j, and -
Martingales 0. For 0 p, such that So
p f 1 there exist positive constants cp and Cp, depending only on
<
=
243
n
p
g
c,l X.xt
n
K E Isn1y <
>1
p A-,) g
c?,f Xj
.
(15.53)
n
/=
side, this extends by the cr inequality to
On the majorant
n
IP
FISn I
p
XF t
< Cp E
n
<
>1
lP,
c XFI Xt 1 p
0
<
p S 1,
(15.54)
>1
which differs from (15.52) only in the specitied constant. 18 I n fact
the
,
Burkholder inequality holds for p > 1 also, although we shall not need to use this becomes result and extending the proof is fairly laborious. Concavity of applied in Readers the that be convexity, so reverse. arguments have to may like to attempt this as an exercise. lemma. The proof employs the following non-probabilistic .?
mbers with yl > 0, and of non-negative < S 1, Then, for 0 p
15.19Lemma Let fyfJ be a sequence 1etFn )7=1y?for t 1 and J'tj 0. =
=
&
F$ < y
+
F--ly, PX >2
<
(1+BP4YL,
(15.55)
where Bp k 0 is a t'inite constant depending only on p. 1 this is trivial, with Bp Proof For p in a Taylor series of tirst order to get =
1'$ where 0n
G
F$-1 +P(l%-1
=
&
=
y;
+
0. Otherwise, expand F$
px
=
ot'nf-1yn,
(J'f-1 +
(L-1 A-yn? (15 56) .
difference equation in
(0,11.Solving the YP
+
=
(15.56)yields
tytf-yt.
(15.57)
>2
Defining Kt
we obtain
-1
#-1
F,-1
=
(F,-1 +
-
%y,)P
(15.58)
,
the result by showing that n
0
< pxKtyt >2
<
BpYL.
(15.59)
r r f The left-hand inequality is immediate. For the right-hand one, note that y fbr y > x > 0 and 0 < r < 1 (see(9.63)).Hence, (15.58)implies that @ -x
-x)r,
1 -.p
Kt K
1
Ff-1
It follows that
1 -
Fl-1 +
a
tyt
=
(Ff-1(Ff-1 + 0ly)
p-1
1.v
1-p
0f yt
.
(15.60)
Theory of Stochastic Processes
244
2,-2 l-p 0 S Ktyt < F,-1 yt
(15.61)
,
and hence n F-P#XA-rzJ n
n
< pv@tlYt-tlls-ps(.y
t /l'n)''
>2
>2
(A''71''-1)X1
-p'
= PT'''/ ?
'P
yt
r-a
E Bpn),
(15.62)
where y; = ytlj'n for t 1,...,n is a collection of non-negative numbers summing to 1, F; = Yxilyr,and Bpn) denotes the supremum of the indicated sum over all such collections, given p and n. =
The terms y;/F;-1 yt /1$-1 for f k 2 are finite since yl > 0. lf at most a number of finite the yt are positive the majorant side of (15.62)is certainly finite, so assume otherwise. Without loss of generality, by reinterpreting n if terms, we can also assume yt > 0 for every t. necessary as the number of nonzero 04?T-1) Ot-1) and applying 2.27 yields the result Bp n) Then, ytlYt-t and y; = ' /41) for a1l p e (0,1). Putting Bp supnfptn) < x completes the proof. w = =
'
=
,
=
Proof of 15.18 Put An Fn
: +
=
2 an d for E Z:=1X?,
=
>
0 and 8
0, set
>
n
Sln +
(: +An)
(1 6)(: +4u) +
=
+
2Xx%-1.V,
(15.63)
>1
2 + 25'f-1 Xt for t k 2, so that in the notation of 15.19, yt = 1', F,-1 (1+ 6).Yf 2 > 0 Then by the left-hand inequality of (15.55), with yl = (1+ 8)(e+X1) =
-
.
is
S
(1 6/18 +
+
?1
X21)P
+
(1
+
t
F(:lV 2pXJ't--ljx%-j.Y/. 6)#X >2
(15.64)
+
N2
we may allow it to approach 0. Taking expectations through, using the law of iterated expectations, and the facts that -1 i s decreasing in its argument, we obtain F(.&l 7/-1) 0 and that However, : is arbitrary in
(15.64),and
.?
=
E(%l
+
6:4.)# < E (1+ S E
(1+
DjPXIP
&fXILP
+
(1+
(:2,-1+ 6x4,-j)P-1V )pX >2 N
+
P XA(--j1F t (1+ 6)45/'-1
(15.65)
.
>2
2 an d yf Xlt for t 2 the right: + 4n, with yl = e:+ But if we put now l'n hand inequality of (15.55)yields (again,as the limiting case as E 1 0)
m
=
Xlp 1
=
,
Fl
+
#XA--lV< (j + Bg)E(AL). >2
(15.66)
Martingales
?
and since (1+ (1 +
6)8P-1(1
+
)P-1
(1+
f
245
this combines with
(15.65)to
give
B )F(A$) k E(Sln+ 8z4ay
2 2:-11(.E'1 l
lp
us-
#F(4j)j,
+
(15.67) .?
where the second inequality is by the concavity of the function
for p S 1.
yields
Rearrangement
2:
FIs I
6J'g21-P(1
(15.68) )/ 1jF(4j). which is the right-hand inequality in (15.54),where Cp is given by choosing 8 to minimize the expression on the majorant side of (15.68). In a similar manner, combining the right-hand inequality of (15.55)with L : + Sn2 w ith (15.65)and (15.67),and using concavity, yields n
S
+.Bp )(1+
-
=
lp
(1+ )(1+l,)FI 5'n1
k
(1+ &)E xll' +
zE
((1
Pxip
)
+
pTn5'l.%-llxj
+
,1
U'-
(1+
Dlpxn
(./-1 +
A,-1)P-1A-l
N2
N
Esn 2 + 6.4n )p
> which rearranges
2#-1(s1
lp
snI
#A'(z4j))
+
(15.69)
as lp
FIS 1
8#(21-P(1
n
+
6)41+#p
-
11-1F(Aj),
which is the left hand inequality of (15.54),with maximize the expression on the majorant side K .
o
1, Bp 0 identically in (15.55)and reproducing the known orthogonality property. Our final result is a so-called exponential inequality. bound fof martingale processes whose increments are accordipgly related directly to the bounding constants, For the case p
=
=
(15.70)
given by choosing cl
=
C1
=
to
1 for any
,
This gives a probability a.s. bounded, which is rather than to absolute
moments.
If (aYf,?;r)';'is a m.d. sequence with 15.20 of positive constants, and Sn Z)=lA, sequence 'heorem
IAI K Bt a.s.,
where
(#fl is a
=
#( IsnI >
:)
S
2exp(-:2/2(J=1#2f) j. u
(15.71)
This is due, in a slightly different form, to Azuma (1967),although the corresponding result for independent sequences is Hoeffding's inequality, (Hoeffding 1963). The chief interest of these results is the fact that the tail probabilities decline exponentially as 6: increases. To fix ideas, consider the case Bt B for This a 11t so that the probability bound becomes #( ISn I > e) f 2exp (-elllnBlt inequalitv is trivial when n is small. since of course P( ISn I > nBj = 0 bv con=
,
.
Theory of Stochastic Processes
246
struction. However, choosing e On 1?2 ) allows us to estimate the tail probabili-1/2 with quantity the associated Sn. The fact that these are becoming ties n exponential suggests an interesting connection with the central limit results to be studied in Chapter 24. =
(-#,,#,) satisfies
Proof of 15.20 By convexity, evel'y x e etM
for any a
>
<
(Bt +
ctfr+ (Bt
.x)d
-xle-l?t
(15.72)
lBt
0. Hence by the m.d. property,
ELemt I1,-1)
qlelt+ e-Bt)
f
S exptzlazsz,
j a.s.,
(15.73)
where the second inequality can be verified using the series expansion of the exponential function. Now employ a neat recursion of 10.10: Ee'nk
@n-I)
=
=
Ee-b-xn
I@n-1)
etxk-lFtetnj
fe osn
(.1
-lexp
(15.74)
@s-1) 2s2 n
)
as .
.
Generalizing this idea yields Een)
=
@,,-1)1,,,-2)... l11))
Eeq...Eefen,
S expfz1(#u2jF(F(...F(c%-l
s Combining
(15.75)with the
e
>
)
(@u-y...( @j))
.
generalized Markov inequality 9-11 gives
2E2.1#2, f exp ( -(y,E + yx1 j 0, which for the choice a = elX1=LBlt4 becomes Psn
for
2x:-1s2,
exp fac,l
Psn
> E)
> E)
<
(15.75)
zexpt-k?!l&1=jB1t) j.
The result follows on repeating the argument of and sumrning the two inequalities. *
(15.75)q15.76)in respect
(15.76) (15.77) of -Sn
A practical application of this sort of result is to team it with a truncation or uniform integrability argument, under which the probabilities of the bound B being j exceeded can also be suitably controll: ,
16 Mixingales
16. 1 Definition and Examples Martingale differences are sequences of a rather special kind. One-step-ahead unp redictability is not a feature we can always expect to encounter in observed time series. In this chapter we generalize to a concept of asymptotic unpredict-
ability.
16.1 Dellnition On a probability space L1,5,P), the sequ e of pairs (A,Tt)=-x, where (Tf) is an increasing sequence of c-subfields of 5 and th .Xt are integrable nv.s, is called an Lp-mixingale if, for p 2 1, there exist sequences of nonx, and negative constants (cfl*-x and ((,,,)7such that (,,, 0 as m -->
-->
IleAl 1,-,211, K c'l,,,
(16.1)
F(A1 T,+,2Il, < c,(,,,+1 117,-
(16.2)
hold for a11 t, and m k 0.
In
A martingale differepce is a mixingale having (,,, 0 for all m > 0. Indeed, tmixingale differences' might appear the more logical terminology, but for the fact that the counterpm of the martingale (i.e. the cumulation of a mixingale sequence) does not play any direct role in this theory. The present terminology, due to Donald McLeish who invented the concept, is standard. Many of the results of this chapter are basically due to McLeish, although his theorems are for the 2. case p mixingales form a very general class of stochastic Unlike martingales, processes; many of the processes for which limit theorems are known to hold can be characterized as mixingles, although supplementary conditions are generally needed. Note that mixingales re not adapted sequences, in general. Xt is not assumed to be Fsmeasurable, although if it is, (16.2)holds kivially for every m k 0. The mixingale property captures the idea that the sequence (Tsl contains progressively more information about Xt as s increases; in the remote past nothing is known according to (16.1),whreas in the remote future everything will eventually be known accordig to (16.2). The constants ct are scaling fatorj to make the choice of (,uscale-independent, and multiples of 11A11pwill often fulfil this role. As for mixing processes (see if (m O(m-%) for (? > (>. j14.1), we usually say that the sequence is of size However, the discussion following (14.6)also applies to this case. =
=
-(>
=
Theory of Stochastic Processes
248
16.2 Example Consider a linear process X
x jc-,77%i-./,
(16.3)
=
where ( I7,)+-Zis a fw-bounded mmingale difference sequence, with p k 1. Also 1et 5t ct&s, s S f). Then =
X
F(Al 5t-m4
=
77 jtb-j,
(16.4)
a.s.
j=m
Xt F(.Y,l 5t+m4
=
-
77e-jUt+js a.s.
(16.5)
j=m+L
w-bounded, the Minkowski inequality shows Assuming ( Us*-ooto be unformly that (16.1)and (16.2)are satisfied with ct sup,ll U,llp for every t, and (,,,= TJ l0j1 + Ie-j!). f-Y,,F,l is therefore a fz-mixingale if X; I0.j1+ l0-./1) 0 are absolutely summable. The as m x, and hence if the coefficients (%J=-x arises sided' process in which % 0 foj < 0 more commonly in the econometric modelling context. In this case Xt is gr-measurable and Xt Ext l5t+m4= 0 a.s., which may increase with t, and does not have to but we may set ct supysfll t&11p satisfy the definition. To prove Xt integrable, given be bounded in the limit to integrbility of the &,, requires the absolute summability of the coefficients, and in this sense, integrability is effectively sufficient for a linear process to be an f-l-mixingale a =
=r,,(
.-.-9
=m
fone-
--
=
-
=
We could say that mixingales are to mixing processes as martingale differences are to independent processes', in each case, a restriction on arbitrary dependence is replaced by a restriction on a simple type of dependence, predictability of the level of the process. Just as martingale differences need not be independent, so mixingales need not be mixing. However, application of 14-2 shows that a mixing zero-mean process is an adapted Ap-mixingale for some p k 1 with respect to the subfields 5t = c(A,Xf-t,...), provided it is bounded in the relevant norm. To be precise, themeandeviations of any fv-bounded sequence whichis a-mixing for r > 1, form an p-mixingale of size 1/r) for p satisfying of size application of 14.4 1 S p < r. If the process is also (-mixingof size tightens up the mixingale size. The mean deviations of a (-mixingfv-bounded 1/r) for 1 < p S r. The is an Lxmixingale of size sequence of size reader can supply suitable definitions of ct in each case. lt is interesting that the indicated mixingale size is lower (absolutely)than the mixing size, except only in the (-mixingsequence having finite sup-nonn (fv-boundedfor all r). Although these relative sizes could be an mefact of the inequalities wlzich can be proved, rather than the sharpest available, this is not an unreasonable result. If a sequence has so many outliers that it fails to possess higher-order moments, it would not be surprising to find that it can be predicted further into the future than a sequence with the same dependence structure but more restricted -:(1//7
-(?,
-
-(?,
-:(1
-(p
-
variations.
Mixingales
249
show the type of case arising in the sequel.
The next examples
16.3 Example An Ar-bounded, zero-mean adapted sequence is an Q-mixingaleof if either r > 2 and the sequence is a-mixing of size 2), or r k 2 size 1). u and it is (-mixingof size -r/(r-
-
-r/2(r
-
16.4 Example Consider for any j k 0 the adapted zero-mean fXtXt+.i c,,?+./,5twjl
sequence
-
,
and (A) is defined as in 16.3. By 14.1 this is a-mixing where ojt-vj Extxt-, (p-mixing) of the same size as Xt for finite j, and is fv/z-bounded, since =
Il.X)-)+./ rn 11
rII.X)+./ IIAII Ilr.
:<
by the Cauchy-schwal'tz inequality. Assuming r > 2 and applying 14.2, this is an in the a-mixing case. To get this result under g-mixing fal-mixingale of size 2), by 14.4, but such a sequence is also a-mixing also requires a size of 2) so there is no separate result for the (-m' of size case. I:a -1
-r/(r-
-r/(r-
'ng
Mixingales generalize naturally
from sequences to arrays.
is an L 16.5 Definitipn The integrable array ( (Xnt, @nfJ'7=-x)';=1 J for p k 1, there exists an anuy of non-negative constants (c,:,) negative sequence ((,,,)7such that (,,, 0 as m ..-A x, and -x,
-mixingale
if, and a non-
--+
IlFtAkrl@n,,-,,,)lIp < c,,,(,?,
(16.6)
IlAk A'(Akrl5n,t+m4 11,f cn,(,u+1
(16.7)
-
hold for
all f, n, and m
0.
(:1
The other details of the definition are as in 16.1. A1l the relevant results for mixingales can be proved for either the sequence or the array case, and the proofs generally differ by no more than the inclusion or exclusion of the extra subscript. Unless the changes are more fundamental than this, we generally discuss the sequence case, and leave the details of the array case to the reader. One word of caution. This is a low-level property adapted to the easy proof of convergence theorems, but it is not a useful constnlct at the level of time-series modelling. Although examples such as 16.4 can be exhibited, the mixingale property is not generally preserved under transfonnations, in the manner of 14.1 for example. Mixingales have to little structure to permit results of that sort. The mixingale concept is mainly useful in conjunction with either mixing assumptions, or approximation results of the kind to be studied in Chapter 17. There we will find that the mxingale property holds for processes for which quite general results on transformations are available.
16.2 Telescoping Sum Representations Mixingale theory is useful mainly because of an ingenious approximation method. involving a remainder which A sum of mixingales is nearly' a martingaleprocess,
Theory of Stochastic Processes
250
canbeneglectedasymptoticallyundervriousassumptionslimiungthedependence. For the sake of brevity, 1et Es stand for Ext ITx). Then note the simple identity, for any integrable random variable Xt and any m 1, Xt
X (f
=
t+kxt
Etn-bxtj
-
+
Et-m-lxt
Xt E ?+p1X).
+
W-m
Verify that each tenn on the right-hand side of signs, except for Xt. For any k, the sequence
(16.8)
-
(16.8)appears twice with
opposite
fEtwkxt F,+1-1X,, 5t+k17=1 -
is a mmingale difference, since Et-vk-kEt.vkxt Et+k-3Xt) 0 by the LIE. When l.&,S) is a mixingale, the remainder terms can be made negligible by taking m large enough. Observe that fEt+nXt, S+,,,I';=-xis a mmingale, and since supmfl Et.vnv'tkS FI.&l< x by 10.14, it converges a.s. both as m c.a and as m 11Fr-,,j.X)IIJ, 0 and -x, by 15.7 and 15.9, respectively. In view of the fact that )1aYfEt+mxtIIp 0, the respective a.s. limits must be 0 and Xt, and hence we are able to assert that =
-
--y
--y
.-+
--
-
X
Xt
(F,+kX,- Et+k-kxtlh,a.s.
=
(16.9)
P=-x
Letting Sn
:=1Xs we similarly have the decomposition
=
m
Sn
=
l
X k=-m
Fn
+
n
XEt-m-jxt
+
>1
Xt X r=1
-
(16.10)
Et+mxt)
where
lu
=
';--Iekxt
F,+,-1A),
-
( 16. 11)
,.-1
and the processes ( Fns a+k) are martingales for each k. By taking m large enough, for fixed n, the remainders can again be made as small as desired. The avantage of this approach is that martingale properties can be exploited in studying the convergence characteristics of sequences of the type Sn. Results of this type are elaborated in j16.3 and j16.4. If the sequence (A1 is stational'y, the constants lcfl can be set to 1 with no loss of generality. ln this case, a modified form of telescoping sum actually yields a representation of a partial sum of mixingales as a single martingale process, plus a remainder whose behaviour can be suitably controlled by limiting the dependence. 16.6 Theorem (afterHall and Heyde 1980: th. 5.4) Let (Xt,S) be a stationary Al-mixingale of size There exists the decomposition -1.
Xt
where
FlZjI <
x
and
=
B
( W$,T,) is
+
Zt
-
(16.12)
Zt+I,
a stationary m.d. sequence.
(:1
Mixingales
Sn
( Fn,gnl
251
(16.13)
Fn + Z1 Zn+1
=
-
is a mmingale.
Proof Start with the identify Xt
F,,s
=
+
Zmt -
zr,,,f+1,
(16.14)
where, for m k 1, m
X
lp,,,f =
Ext..s
Et-3Xt+s4+
-
EtXt+m+L
+A-m-1
-
(16. 15)
Et-tXt-m-3
J=-m
m
Zmt
=
%-Et-$Xt+s -A-,-1 +Et-3Xt-s-t4.
(16.16)
.=o As in (16.8),every term appears twice with different sign in (16.1 Xt. Consider the limiting cases of these random variables as m designated J'Pfand Zt respectively. By stationarity, E Iff-lA+sl
=
), except -->
x,
for to be
FIEt-s-txt I
and Fl Xt-s-j
Et-jxt-s-k
-
1
E IXt Fm-Y,l ;
=
-
hence, applying the triangle inequality, X
to
E Iz,l
S TE M
IEt-s-txt I +
77E IXt r.m
-
Etnxt
I
*
< 2X(,<
(16.17)
=.
M
Writing Jpr
=
Xt Zt + Zf+1, note that -
E IJFfI S
FI-Y,I+ 2F Iz,I <
(16.18)
=,
and it remains to show that J'P'fis a m.d. sequence. Applying 10.26(i) to (16.15), Et-jb'mt
(16.19)
Et-3Xt+m+3a.s.,
=
and stationarity and (16.1) imply that
FIEt-$Xt+m+3l FIEt-m-zxt I --A 0 0 also. Anticipating a result so that FI&-1B%l
(16.20)
=
x, from the theory of as m k e IN1 stochastic convergence (18.6),this means that every subsequence tr?h, e-lW,,,w),rl subsequence IN such 0 a.s. that I contains a further j e ) fmkq), x. for such subsequence, it is possible Since Jpf Wmkqj,t to every as j conclude that A'(WfITf-1) 0 a.s. This completes the proof. w .--h
-->
-->
.-+
-->
=
Theory of Stochastic Processes
252
The technical argument in the final paragraph of this proof can be better appreciated after studying Chapter 18. It is neither possible nor necessary in this 0 a.s. approach to assert that F(G,,vl @r-l) Note how taking conditional expectations of (16.12) yields -->
F(ml 72-1) = Zt 4+1 a.s.
(16.21)
-
It follows that <, is almost surely equal to the centred r.v. Xt Ext -
IF,-l).
16.7 Example Consider the linear process from 16.2, with ( Utj a stationary integrable sequence. Then Xt is stationary, and X
FI !71I 77 I%I <
Fl.Yl l f If the coefficients 00
*.
j=-oo
satisfy a stronger summability
X
X
condition, i.e.
X
< 77 X(( %( 10-/1) 77-10-1 X?nlo-,?,l
+
+
=
za=l
rn=1 j=m
then Xt is an Al-mixingale of size decomposition of (16.12)with
(16.22)
x,
?l=1
of terms we obtain the
By a rearrangement
-1.
X
B,,
77uj &,
=
(16.23)
jcz-x
and oo
Zt
=
o
x
X j=m X 0y Ut-t,t X 0-,. Utwm-q
zrl=l
where
E'IZ,I <
cxa
(16.24)
-
,
j=m
by (16.22).u
16.3 Maximal Inequalities As with martingales, maximal inequalities are central to applications of the mixingale concept in limit theory. The basic idea of these results is to extend Doob's inequality (15.15)by exploiting the representation as a telescoping sum of martingale differences. MacLeish's idea is to 1et m go to x in (16.10),and
accordingly write
X
Sn
(16.25)
Fnz, a.s.
=
#=-x
16.8 Lemma Suppose (xaly has the representation in (16.25).Let be a = collection real with of non-negative if l'ak 0 numbers, summable ak 0 a.s., and ak > 0 otherwise. For any p > 1,
tJklX-x
=
E max Lsjs n
IsjI# u
# -
#
1
x
77a k
b=-x
p-1
>-cf-1Fl
ak>o
ynzl#.
(16.26)
Mixingales
taloJx and positive real sequence
Proof For a real sequence X7=-xlkand note that OQ
X xk
p
x =
X
A''
P=-x
253
p
xklaaklp
=
x
X tz,'-#I.xkI#,
K A'''-'
k=-x
1et K llkloo-x,
(16.27)
#=-x
where the weights aklK sum to unity, and the inequality follows by the convexity of the power transformation (Jensen's inequality). Clearly, (16.27)remains true if the terms corresponding to zero xk are omitted from the sums, and for these cases set ak 0 without loss of generality. Put xk Fn, take the max over 1 S j < n, and then take expectations, to give =
=
E max L<j
To get
Isjbl'
j
<
x
77ak
p- I
tt.---
j
ak y-')
max j yg j
p
.gy
Doob's inequality on the right-hand
(16.26),apply
.
lsjsn
akg
side. w
This lemma yields the key step in the proof of the next theorem, a maximal inequality for Q-mixingales. This may not appear a very appealing result at first sight, but of course the interesting applications arise by judicious choice of the
sequence
lJkl
.
16.9 Theorem = E7.1-Y,, and
be an fw-mixingale, 1et Sn (Macleish 1975a: th. 1.6) Let (Xf,J;2l=-x let ltul7 be any summable sequence of positive reals. Then co
E
maX
ls/fn
f
f 8
co
($2+
F7
Jk
1
(2j)J
+
t-.=l
uk.1j)
(2gu-,1 2F-4 -
=
SJ 1<)
K4
(16.29)
.
put a-k ttzkl=-x,
=
ak for k
>
0. Then,
oo
Cr
E max
N c2j >1
k=l
Proof To get a doubly intinite sequence 2, applying 16.8 for the case p
n
X ak X tz-k1F(L2J .
(16.30)
V-x Since the termsmaking up lk are mmingale differences and pairwise uncorrelated, we have F(Fn2k) Ek-vkxtEt-vk-kxt) whichit follows that NOW,
-
Also let
%)EEt+kXt -Et+k-jXt)l.
,.-1
EEk-tk-vkxtk-vk-titqj
=
E Et-vkxt
=
l=-x
=
E(Elt+kXt Et+k-lxtL$l =
ztk x,- Et-vkxpand =
-
E(Elt+k-jXt) by the LIE, from
Elt+k-;Xt).
(16.32)
it is similarly easy to verify that
EEt+kXt Et+k-LXt41 F(z,,1-1 -
(16.31)
=
=
Ezlk-j 1,
=
z,1)2
-z2,J.
(16.33)
Theory of Stochastic Processes
254
Now apply Abel's partial summation <x)
X
a-x
n
c-k1A'(F2aJX
X a-kE(Et+kXt
=
>1
(2.25),to
get
Etwk-jXtll
-
k=-x
#j
*
<>3
=X X ,-1 t.-o
6l-kE(Elt-kXt Elt-k-jxt)
+
-
n
1F(z2,,:.j z2f:) Xt7-: k=l -
<x)
)q >1
=
formula
<x)
&0-1#(F2X t
t
)
+
TECEI
X)(J-k1
t-k
-
t
tz-k-lj)
k=1 X
J1 -3Ez1 10
+
)
+
y7F(z2J(u1 j t
-
u-11)
al
(16.g4)
where the second equality follows by substituting (16.32)for the cases k f 0, and (16.33) for the cases k > 0. (16.29)now follows, noting from (16.1) that EZlk) 2 X, ) < c2(2 S c2(2 E(Et-k , t k+1 x t k and from (16.2)that .
Pptting x
K
x
-1((2 (2) 1 Tak t'af
8
=
() +
tztl
u-jy.1jl 2y-q (2:(t7-:1
+
(16.35)
-
,
k=1
this result poses the question, whether there exists a summable sequence ftzkl such that K < x. There is no loss of generality in letting the sequence ((:); be monotone. If (,u 0 for m < x, then (v,+y0 for a1lj > 0, and in this case one 2 0,...,- + 1, and K reduces to m + 1)(%2 + (1). 1, k A jtem may choose ak consider where and the case natively, (k> 0 for evel'y k. If we put at ((), then define the recursion =
=
=
=
=
(z g((1+4Jk-1) z ak ztu-j
.(j
a
=
ak is real and positive if this is true of -j
ak
=
8
x
Xtzk
=
> -2
(, so that
(1 ak 2
(1)f
(16.37)
2J(),
we have
x
(2j)+2Xtu
< 16
k=1
1=0
ln this case, for k
and the relation
-2
-c:-1
J-o1((+
(16.36)
,
tu-1
-1
l 2 is satisfied for each k. Since aj- ((c +
K
j/2
x
2
Xak
.
(16.38)
k=o
0 we find -1
=
ak
-1
-z-1)Jk
-1
-2
f ak
-2
-J1-l
(16.39)
Mixingales m
m
X (-,2G (0-2+ X k=o
-
,-z-21
=1
(16.38),we
Substituting into
ak-l
255
)
am-l.
=
(16.40)
get
( j
X*X*(
K f 16
-
2
-
' /2 2
(16.41)
This result links the maximal inequality directly with the issue of the summability of the mixingale coefficients. In particular, we have the following
corollary. 16.10 Corollary
Let
(.X),S)be
an Q-mixingale of size
E maxu/J
K
Then
#Xd,
(16.42)
>1
1f.jf n
where K
-.
< x. -1/2-
-2
Proof If (1 Ok > 0, as the theorem imposes, then Z'l=1(k ) for1(-k2)-1/2 tp(--1-8 ) an d hence is summable Over m. ( and b 2 27 Om 2+2: q ) y The theorem follows by (16.41).w =
.
=
.
=
=
However, it should be noted that the condition m
x
.j/p,
X 77(-,2 is weaker than Ok -1/21/2+ for E > 0, so that k
(k
=
(z
--
X( -2 X(k..v2)(log =
(16.43)
< c.o
). Consider tjj e x
case for evel'y g > ()
k.- 2)2+2e<
(k qk+ g)-1/2(jog =
k + 2)-1-:
ejxen
.
(??;+2)2(log
-+2)2+26
(16.44)
,
and (16.43)follows by 2.31. One may therefore prefer to define the notion of in tenns of the summability condition (16.43),rather than by orders isize of magnitude in m. However, in apractical context assigning an order of magnitud to (,,,is a convenient way to bound the dependence, and we shall find in the sequel that these summability arguments are greatly simplified when the order-ofmagnitude calculus can be routinely applied. -'
=
Theorem 16.9 has noobvious generalizationgomtheo-mixingalecasetogeneral of the Lp for p > 1, as in 15.15, because (16.31)hinges on the uncorrelatedness exist in under consideration, second not the But because moments may terms. cases < p < 2 would be valuable. This is attainable by a result for 1 comparable a slightly different approach, although at the cost of raising the mixingale size in other words, the mixingale numbers will need to be summable. from to ..ul
-1;
16.11 Theorem Let tXr,?hl'l..,obe an fv-mixingale, 1
<
2, of size
-1,
and 1et Sn
Theory of Stochastic Processes
256 E max 1sjn
p
lsjI s
tx
F
P
4p c,
P
N
y7(z 77d,
j
t-.al
(16.45)
>1
where Cp is a positive constant.
Proof Let F,,kbe defined as in (16.11), and apply Burkholder's inequality and then Love's cr inequality with r = pI2 e (, 1) to obtain
FIFazlr f CpE
n
Et-vkxt &+k-1X,)2
X >1
(15.18)
p/2
-
n
S
CAXFI(Et+kXt
&+1-1.41F.
-
(16.46)
>1
Now we have the mixingale inequalities, lIF,+zX,
f IIF,+1AIl,+ < 2c,(z .&,+z--1.Y,I1?? lI.E',+z-1-'tlI,
-
fork
0 and
<
11F,+kA -
for k
(16.47)
z'kll, A',+,-1AlI,I4z,,z-1 S 2c,(1 Iz,k.II, 11p+ < IIz,,k-1 =
-
0, where Ztk is defined above
>
(16.48)
(16.33).Hence, n
e'Irnzl' s
Ec
2,cp(f
(16.49)
>1
(put (0 1), and substitution in (16.26),with (t7z)7'a positive sequence and = ak, gives =
E
mx
lijin
p
#
IS)Ip < y+1c?,
p-j
cx>
Efz t..o k
- 1
Both ak and ak1-#(( can be summa b1efor p conclusion follows. .
>
-ak
n
x
y'-qtzj-ptj N c;.
(16.50)
,...1
t-azo
1 only in the case
(z
=
Oa,
and the
A case of special importance is the linear process of 16.2. Here we can specialize 16.11 as follows:
16.12 Corollary (i) if Xt
Z7=-oo%Ut-),then
=
E mx Lsjsn
(ii) if Xt
For 1 < p S 2,
=
IsjI# S Cp Z7
p
x
IOeI +
j
77( I01( + al
l0-zI)
p
n sup F# (7,1#., s
then
zejh-p
E max I5'./i lsjs n
P
p
s c,
P
p
!
o
p
n
I0zl 77sup FI &,IP. >'') k.'.'zo >1
st
Mixingales Proof In this case, Et-kxt
Et-k-kxt
-
and a-k
constant sequence
l57
kut-k.
=
tJkl7
Letting
be any non-negative
ak,
=
7-7JI-PFI1u1,
I0:I#7'7cf, CvTakz-p
<
ap*
JS/
supsxtll rxllpin case the results. .
wherect supxll&x11pin case (i), and c, = I0kIand substituting in (16.26)yields =
(16.51)
>1
=
ak (ii). choosing
T) Recall that the mixingale coefficients in this case are (,, I0./1 + I0-./1), so linearity yields a dramatic relaxation of the conditions for the inequalities to be satisfied. Absolute summability of the % is sufficient. This corresponds simply to (,,, 0. A mixingale size of zero suffices. Moreover, there is no 2 yields a resuli separate result for faa-bounded linear processes. Putting # that is correspondingly superior in terms of mixingale size restrictions to 16.11. =m(
=
--)
=
16.4 Uniform Square-integrability One of the most important of McLeish's mixingale theorems is a further consequence of 16.9. lt is not a maximal inequality, but belongs to the same family of results and has a related application. The question at issue is the uniform integrability of the sequence of squared partial sums.
16.13 Theorem (fromMacLeish 1975b: lemma6.5,' 1977: lemma3.5) Let taYsTfl 1m, and :,2 7=!clwhere ct is detined in be an Q-mixingaleof size 1 Sn l n l l lf the sequence (Xtlct ) ,=1 is un iformly integrable, then so is the (16.1)M16.2). =
-z,
=
.
oo
sequence(
maxlxjspl/vz,,
jooawl .
Proof A preliminary step is to decompose Xt into three components. Choose posiB 1( jx,Issc,), an d then tive numbers B and m (to be specified below), 1et 1 t =
define
Ut Ft Zt
=
(16.52)
Xt Etwmxt+ Et-mxt
=
-
B
Et-vmxt1 t
=
-
E t-m
a
lf
(1653)
t t
.
B Et-mxt 1 1 t ) Et-mxt 1 1#) t -
-
(16.54)
-
,
such that Xt Ut + Ff+ Zt. This decomposition allows us to exploit the fotlowing collection of properties. (To verify these, use various results from Chgpter 10 on conditional expectations, and consider yhe cases k 2 m and k < m separately.) =
First,
E 2Et-kut Eut
-
Et-vkut)a
for k k 0, where kv m
=
1 E Et-wmjvk S
Ext
-
maxfkml.
=
EE 2 t-k F,
=
=
EEl
t-ksm
2 2
ctLkvm,
(jy
a a Et-vkwmlxa f c/((t,v??;)+1 ,
.
55;
(jy 56; .
Second, El X/ 1#) )# , 1* , , t-m -
,
(16.57)
Theory of Stochtic
258 E
Et+kYt)2
-
Processes
F(F2/+?,, X 1*f -F2 faxkz.?n )# t 1W) , t
=
(16.58)
,
=mintk,rrll.
Theterms areboth zero if kkm and are otherwise bounded Bct. Third,
whereko.1P,m by Exlt ) ;
F#
t-k
zt
EEl
=
E(z t F,+zz,)2 -
=
1 1()
:.)m4
t-
EEl
t+
-
,aftl
1t)
-
-
-
e'lt-ttvj
lt
-
A+(k=)m(1
)), 11/)),
-
where the tenns are zero for k 2 m and bounded by f(#l(1 F(AQf(1 11 that ))/U= ElxtlctjlLf jxycgjlsl) 0 as B the assumption of uniform integrabillty.
x
-
(16.60)
1Pf))
-
-->
-.-.:
(16.59) otherwise. Note uniformly in /, by
The inequality
(x'
2
x,)
>1
s
3
s,) 2 +
z,) 2j
g(x? (x? (x' ,-,) 2
+
>1
>1
(16.61)
>1
for 1 S j S n follows from substituting Xt = Ut + F, + Zt, multiplying applying the Cauchy-schwartz inequality. For brevity. write xj
S)?/v2 n,
=
uj = (Sr=1 '
A'./=
kn
=
Ullgnl,
?= 1
zj = (Yfn Then
out, and
YttllMnl,
lZf)2/v2. s
(16.61)is
equivalent to xj f 3(w+yj +p), for each j = 1,...,n. Also 1et similarly; then clearly, maxlssaay, and define n, y'k, and
.
f 3n
+
yh +
k).
(16.62)
For any r.v. Xk 0 and constant M > 0, introduce the notation &>(A') F(1(x>>).X), x. 0 as M As a so that the object of the proof is to show that supn&ion) of 9.29, and consequence (16.62) =
-.-/
'k
f 3&v/3(
s 6eqn)
+
yh +
-->
k)
+ &wK@-n)
+En)).
(16.63)
We now show that for any er> 0, each of the expectations on the right-hand side of (16.63) can be bounded by e by choosing M laree enough. First consider &;n); =L1/2giveia (16.55)and (16.56),and assuming (,a= Om ), we can app j y j: nto this k-1-9 -1-8 = for k > m. Applying (16.29) for k S m and ak case, setting ak m ' with >1 U t substituted for S) in that expression produces .
,
E(
n
)f
8
(r?z+1)/1-1-8
= O(m ),
+
X k-1-0
&-+1
=
(2m1++2 m X (2,/ 1=>a+1
(16.64)
Mixingales
259
where the order of magnitude in m follows from 2.27(iii). Evidently we can choose m large enough that ELn) < :. Henceforth, let m be fixed at this value.
but in view of (16.59)and (16.60)we A similar argument is applied to E(L4, 2 otherwise. Write, fonnally, EEt-kzt and < choose 1, 0,.. 0 k ak ak may2 y.m, ct(2:, Etnztjl 2 an d Ezt where < t7?(k =
=
=
-
maxlu,u,,F((X,/c,)21( Ixgc,I>s)),
(2
k
<
m,
kk
m,
(16.65)
=
and then application of (16.29)leads to
ELn)f
16(-
+
1) max Elxtlctll
1( 1xgoI>s)).
1< t < n
(16.66)
x, so let B be fixed at a value large enough that This term goes to zero as B Eqnj < E. where For the remaining term, notice that l'f Z'l'=-,u+1(x -->
=
Et-vkxtf B, Et+k-3y , jl ,.
k,1 =
(j6 6y)
-
.
For each k, (efl, @f+:)is a m.d. sequence. If 16.8 is applied for the case p = 4 and ak 1 for IFcl < m, 0 otherwise, we obtain (not forgetting that for yj > 0, =
2 (maxyyy) =
..2 Eyn,
maxytyy?
-
where Fn1
=
j
1
wf
v
J)
max l sj s n
'n
77r, 1
=
F(F4
<
given l',,z
k) + n- j
4F(F3.) n
,
?
1)
w v (2-+ V =
yjup ,
4
1 4 'n
r=
Z:=1(a.Now,
EYnk 4)
4
-'
Fa-j,1+
77E
n
+ 4F(F n
nk), ),4
kzz-m
(,,k,we
6F(F2.j
+
m
(16.68)
have the recursion
2up ,
-1
,
zt3,u)+
F(j4uJ.
(16.69)
(fkare bounded
absolutely by lBct; hence consider the terms on the right-hand side of (16.69).The second one vanishes, by the m.d. property. For the third one, we have The
2 2 f f'(l%-l,t4n2 and for
2
E-n-,lBcn)
2
f
4 2
(2f) Vn-lcn,
2
the fourth one, note that by the Cauchy-schwartz
3 j k (2#)4vn FI L-l,klnk
(16.70) inequality,
-1c3u.
Making these substitutions recursivelyyields Elvkl f n
into
(16.69) and
(16.71) solving
the implied inequality
n
&
N
>1
>2
>1
+ 4Xv,-1c) + (2#)4 6Xvl-1U Xd
4 f 11(2#) 4vn.
(16.72)
Theory of Stochastic Processes
260 Plugging this bound into .Y.k 0 and a > 0, yields
(16.68),and tinally 6
an ) f &Ml6 (.y
.2
uE- (.yn ) f
applying the inequality a &u(m 4642-
(s j
+
1)411(2s)4
s,
< EX2) for
(16.73)
.
By choice of M, this quantity can be made smaller than e. < 18E for large enough M, Thus, according to (16.63)we have shown that &AQ) or, equivalently,
b-)
-->
0 as M
x.
By assumption, the foregoing argument applies uniformly
complete. w
(16.74)
--)
in n, so the proof is
The array version of this result, which is effectively identical, is quoted for the record. and let Sn be an fu-mixingale array of size 16.14Corollary Let (X ,,@nrJ 2 X1=jXnt and vn by if 2 E'Jzzlcnt, is given where cnt (16.6)V16.7), fAkfyc2n, j is is uniformly integrable. uniformlyintegrable, (maxlsyunial/vnz1-=a.! =
-,
'1
=
Proof As for 16.13, after inserting the subscript n as required.
.
.
17 Near-Epoch Dependence
17.1 Definitions and Examples As noted in j14.3, the mixing concept has a serious drawbackfrom the viewpoint of applications in time-series modelling, in that a function of a mixing sequence (even an independent sequence) that depends on an intinite number of lags and/or leads of the sequence is not generally mixing. Let X
=
#f(...,16-l,1$,'P)+1,...),
(17.1)
where Js is a vector of mixing processes. The idea to be developed in tllis chapter is that although Xt may not be mixing, if it depends almost entirely on the epoch' of f F,) it will often have properties permitting the application of limit theorems, of which the mixingale property is the most important. This idea goes back to lbragimov (1962),and had been formalized in different ways by Billingsley (1968),McLeish (1975a),Bierens (1983),Gallant and White (1988), Andrews (1988),and Ptscher and Prucha (1991a),among others. The following definitions encompass and extend most existing ones. Consider first a definition for sequences. tnear
17.1 Dennition For a stochastic sequence l Ffl+-Z,possibly vector-valued, on a is an probability space (f1,@,#),let lj+-: c(J$-,j,...,J$+,,,), such that ( 8??f+-'''''');=m lf, for of integrable 0, of > c-tields. increasing sequence r.v.s a sequence p =
lA
lt7
satisfies
IIX, -
S vp,, F(.X)I ?7-'''''')II,,
(17.2)
' where vm -- 0, and flrl+-2is a sequence of positive constants, Xt will be near-epoch dependent in fv-/ztp?'?zl(JV-NED) on (FfJ+-7.I:l
be said to
Many results in this literature are proved for the case p 2 (Gallant and White, 1988,for example) and the term near-epoch dependence, withoutqualitkation, may be used in tlzis case. As for mixingales, thre is an extension to the array case. =
A'
possibly vector-valued, on 17.2 Definition For a stochastic array f( Fnfl17-x)';=1, a probability space (f1,F,#), let snt-mt c(Fn,,-,,,,...,F,,,r+,,,).lf an integrable -,,,
=
satisfies arrayt t.krll---xl7-l, Ilxn'F(-Yn,lT,$+7-,,,)11,K dntvm, -
0, and fV-NED bn ( Fnr) u
where
Mm
-->
.
fdritt
(17.3)
is an array of positive constants, it is said to be
Theory of Stochastic Processes
262
We discuss the sequence case below with the extensions to the array case being easily supplied when needed. The size terminology wlch has been defined foi mixing processes and mixingales is also applicable here. We will say that the O(m-% for (p > (pf). when Mm sequence or array is LP-NED of size According to the Minkowski and conditional modulus inequalities, -(p0
=
117,e-'L117'''''')11,S 11*-,wll,+ Il.E'(-'t w! /7:)11, <2IIX,- 81,,11,, -
-
-
(17.4)
wherew EXt). The role of the sequence (4) in (17.2)is usually to account g,,ll, is uniformly bounded, forthe possibility of tending moments, and when 11.1, should equal finite dt constant for all f. However, a expect to set to a we drawbackwith the definition is that (l can always be chosen in such a way that =
-
11-Y,Ext -
inf t
IF,'+-:)11,
dt
=
0,
for every m, so that the near-epoch dependence property can break down in the limitwithout violating (17.2).lndeed, (17.2)might not hold except with such a choiceof constants. In many applications this would represent an undesirable weakepingof the condition, which can be avoided by imposing the requirementdrs 2jlm- gll??, or for the array case, dnt S 2II-L,- p.rllp. Under tls restriction we can set v,a < 1 with no loss of generality.
Near-epoch dependence is not an alternative to a mixing assumption', it is a propertyof the mapping from (FfJto (,&J,not of the random variables themselves. The concept acquires importance when (Ffl is a mixing process, because then (m) inherits certain useful characteristics. Note that f;txfl 1j+-:) is a finite-lag, function of amixing process and hence is also mixing, by 14.1. Aorl/s-measurable mixing in the sense of Near-epoch dependence implies that (.') is a pproximatedby weshowbelow, amixingprocess. Andas anear-epochbeingwell dependentfunction of a mixing process, subject to suitable restrictions on the moments,can be a mixingale, so that the various inequalities of j16.2 can be exploitedin this case. From the point of view of applications, near-epoch dependence captures nicely the characteristics of a stable dynamic econometric model in which a dependent variableXt depends mainly on the recent histories of a collection of explanatory variablesor shock processes Fs which might be assumed to be mixing. The symmetricdependence on past and future embodied in the definition of a G-NED functionhas no obvious relevance to this case, but it is at worst a harmless generalization. ln fact, such cases do arise in various practical contexts, such as the application of two-sided seasonal adjustment procedures, or similar smoothing filters; since most published seasonally adjusted time series are the output of a two-sided tilter, none of these variables is strictly measurable without reference to future events. tapproximately'
17.3 Example Let (Jl+-7 be a zero-meqn, As-bounded scalar sequence. and define
Near Sp/ch Dependence
xt
263
X ejvt-j.
=
(17.5)
jc-x
Then, by the Minkowski inequality,
Il, 77 111,Ext I8/',+-''',?,) j=m+3
jvt-j
=
-
e,+-,?,'''Ff-./l +
-
e-jvt-v.ie7,,,':'14+./)) P -
f dtvm,
(17.6)
Jillp, all /. Clearly, v,,z 0 if where v,u 17,.+1(I%I + I0-jI), and dt 2sup,I1 if I0./1+ l 1= the sequence (%) is absolutely summable, and v,,i is of size /4-1-:) for (? > (p. ln the one-sided case with oy 0 foj < 0, we may put #f supysfll7,11p, which may be an increasing function of t; compare 16.2. In -->
=
=
-(p0
-j
=
=
The secondexample, suggestedby GallantandWhite (1988),illustrates hownearepoch dependence generalizes to a wide class of 1ag functions subject to a dynamic stability condition, analogous to the summability condition in the linear example.
17.4 Example Let t %'t)be a Zp-bounded stochastic sequence for p sequence (m) be generated by the nonlinear difference equation Xt
hvt, Xf-1),
=
2 and 1et a
(17.7)
where ((.,.) l is a sequence of differentiable functions satisfying
:/)(r,X sup a.x FA
b
<
1.
<
(17.8)
As a function of h is called a contraction mapping. Abstracting from the of problem, write vl as the dummy first argument of ft. By the stochastic aspect repeated substitution, we have .x,
h
=
=
J,(v,,,-1(v,-1.,-2(v,-2,...))) (17.9)
',(vf,v,-1,vf-2,...),
and, by the chain rule for differentiation of composite functions,
gt
h-j
' -j
f b
vt-j
(17. 10)
.
vt-j
Detine a Fj-m/f-measurable approximation 1ag exceeding m by zeros:
to gt by replacing the arguments with
:,(14,...,v,-r,,,0,0,...). p';'(v,,...,v,-,,,)
(17.11)
=
By a Taylor expansion about 0 with respect to vt-j for j
8t # t -
cw
,:1
m,
+
=
l=m.z.l
>
vt-g.
vt-j,
(17.12)
Theory of Stochastic Processes
264
denotes evaluation of the derivatives at points in the intervals (0,vr-j1. Now define the stochastic sequence (X,) by evaluating gt at (1$,J6-1,...). Note
where
*
that IIX, -
F(X,I1',+-:)112f
#'J'(7,,...,F'-,,,)II2
IIX, -
(17.13)
by 10.12. The Minkowski inequality, (17.12),and thep (17.10) further imply that X
lIA7 -
#:'IIa 77 Gt-.ivt-j =
.j=-+1
l
<jcm- jlGt-jvt-j j)2 X
b.i- llF/-jvt-./llc
<jzm1
s
1-#
bm sup IlF,-./vf-./lia,
(17.14)
j>m *
with where Gt-j is the random variable detined by evaluating Llvt-pgtj and Ft-j bears the corresponding respect to the random point vt-j,vt-j-L with constants relationship with lvt-pft-j. Xt is therefore fU-NED of size <: supssf lIF,F,Ia, Hlder's i f particular, inequality exists. In this nonn tf exist for r > 1, and (IF,IIar/r-1) allows us to make this derivation whenever 11 7,lI2r 14112 < x and Ft is a.s. bounded. n and also if 11 ,...),
-=,
17.2 Near-Epoch Dependence and Mixingales The usefulness of the near-epoch depencence concept is due largely to the next
theorem.
17.5 Theorem Let (X,)*-x be an fv-bounded zero-mean sequence, for r > 1. lf Xt is AP-NED of size (i) faet ( Ff) be a-mixing of size on ( F1) for 1 S p fXl,T-Jx) with size fv-mixingale is of constants dtl an f (>, < r aslp 1/r)) with constants ct t< maxl 11Xf ) llr,#f of size -J. If Xt is fV-NED of size (ii) Let ( $) be (hmixing, on ( Fr) for 1 S p f r with constants (J, (X,,F-'x) is an fw-mixingale of size -.b
-a.
-min
,
-
.
-.b
.-lnint//,
J(1
-
1/r)), with constants ct <( maxf llXfIlr,4).
F,J) where %,' c(Fx,...,F). Also, for m K Proof For brevity we write Est.) = F(. l = 1, let k (-/2), the largest integer not exceeding ml2. By the Minkowski =
inequality,
Ile'-f ljp< oo-mxt
and we bound
jlJ2x-''t.Yf -
Fj+-),V) Ilp+ IIF-'x-mEtkkxtj11,,
each of the right-hand-side
jlE-t
..-'''(.4 -
Etkkxt)
K tL'-t 11p
c: '''
terms. First,
)Xt
-
P)11/P Frf+-l.& I
(17. 15)
Near Fp/c
DepeWence
265
II-Y, VtkkAllp -
< dtvk,
(17.16)
using the conditional Jensen inequality and law of iterated expectations. Second, E7PX function of F,-1,...,Ff+s and hence mixing of -.k t is a tinite-lag measurable the same size as (FfJ for finite k. Hence for part (i), we have, from 14.2, 6al/P-l/rjjxf r < 1I, 6al/P-1/rIIA:+-lA-,II
11F-'--'''(F71k-Y,) <
combining(17.16)
(17.17)into (17.15)
and
r. 11
(17.17)
yields
f maxt II-Y,IIr,4l(,,,, llF-'ol'''XIlp
where(,u
=
(17.18)
+ vk. Also applying 10.28 gives 6a11/#-1/r ,
< ldts'm < 2 lI-, Fl+-'''''W,IIp 111,F-lx+'''-Yfllp -
-
K 2(,,,.
(17. 19)
atlp 1/r) ), part (i) of the theorem holds with ct <( since (,,,is of size T he proof of pal't (ii) is identical except that in place of (17.17) maxflI#,1lr,l. we must substitute, by 14.4, E-tqn-mEftkkx < 2911 A'j+-lkxflj< 2911-l/rjjx, jj . 11 (17.20) r I1p -minth,
-
-l''rjj
,.
Let us also state the following corollary for future reference. be an fv-bounded zero-mean array, r > 1. 17.6 Corollary Let f fxnrl+f for 1 f p < r with constants (t&,)on an an-ay (i) If Xnt is AP-NED of size f Fnrl which is a-mixing of size -J, then tXpv,Fj,-xl is an fw-mixingale of alqlp 1/r) J, with respect to constants cnt (< maxt IlAk,Ilr,t&,l size i If AP-NED of size for 1 f p < r with constants (J,SJon an array Xnt (ii) which is of size -J, then (-Lr,ij,-xl is an fv-mixingale of ( FSCJ t-mixing J(1 1/r)) with respect to constants cnt < maxl llAkfIIr,&,l size '==-xl';xl
-.b
-minty,
-
.
-.b
-minlh,
-
,
.
Proof Immediate on inserting n before the t subscript wherever required in the last proof. w The replacement of Ff by F,v and ;sl by 5ntsis basically a formality here, since none of our applications will make use of it. The role of the array notation will always be to indicate a transfnnation by a function of sample size, typically the normalization of the partial sums to zero mean and unit variance, and in these @fyfor al1 a,. cases 5js Reconsider the AR process of 14.7. As a special case of 17.3, it is clear that in that example Xt is CP-NED of size on Zt, an independent process, and hence of size for is a wmixingale every p > 0. There is no need to impose smoothness assumptions on the marginal distributions to obtain these properties, which will usually be a11 we need to apply limit theoryto the process. These results allow us to tine-tuneassumptions on the rates of mixing and nearepoch dependence to ensure specitic low-levelproperties whichare needed to prove convergence theorems. Among the most important of these is summability of the sequences of autocovariances. If for example we have a sum of terms, Sn Z:=1Xr, =
-x
-x,
=
Theory of Stochastic Processes
266
we should often like to know at what rate the variance of this sum grows with n. 0 with no loss of generality, Assuming Extj =
F()
n- 1
n
XA'(Ff)+
=
2X >1
>1
n
-t
TElxtupwml
,
m=1
a sum of n 2 terms. If the sequence (zY,l is uncorrelated only the n variances appear and. assuming unifonnly bounded moments, Esn)2 = On). s or genera j dependent processes, summability of the sequences of autocovariances (EXtXt-j), j e N ) implies that on a global scale the sequence behaves like an uncorrelated 2 = o (n). sequence, in the sense that, again, E(Sn) For future reference, here is the basic result on absolute summability. To fulfil subsequent requirements, this incorporates two easy generalizations. First we consider a pair Xt,Yt), which effectively permit.s a generalization to random vectors since any element of an autocovariance matrix can be considered. To deal with the leading case discussed above we simply set Ff = Xt. Second, we frame the imposingfv-boundedness resultinsuchawayas toaccommodateendingmoments, uniform fv-boundedness. It is also noted that, like previous results, this but not extends trivially to the array case. 17.7 Theorem Let fXf,)',) be a pair of sequences, each &&r-l)-NED of size W ith respect to constants fdxtvd't) for r > 2, on either (i) an a-mixing process of -1,
size and -s(r-2),
(r,
<<
or
(ii) a g-mixingprocess
of size
-rlr-
1), where
11r,11,. vlaenthe sequences
IExtYt+ms I Xl
J't+m
r
<<
IIx,II,
(17.21)
m e (N
'
r
t,v,
of are summable for each t. Also, if arrays lAk,l%,l are similarly f,s(r-1)-NED dnYt << 11 and rnrjlr, size with respect to constants ftfr,JJllwith dnt rt IIAk/IIr the sequences -1
IE-ktl'rn,t.vmjI Xnt
yn,l+pl
r
are summable for each n and t.
(17.22)
'
r
EI
Since r > 2, the constants appearing in (17.21) and (17.22) are smaller (absolutely) than the autocorrelations, and the latter need not converge at the same rate. But notice too that r/(r- 1) < 2, so it is always sufficient for the result if the functions are JU-NED. Proof As before, let Ests)
=
E.
ity,
lE-vmx
I f 1E-vmxt
The modulus and Hlder
1TJ),and -
let k
Y+-zAll l +
inequalities give
=
(
the triangle inequal-
If(1$+,x1+-1#,)l
.
(17.23)
Near Fp/c
IEb-vmxt X where v,,, is of size 1, and b
Dependence
e+-CA)) < I
267
IlrllA V+-l-'tIIr/(,-I) 1--,+,,, 11 l+,,, Ilgzlcv-l', f 11
-
Also, applying
-1.
(17.24)
-
17.5(i) with p
rllr
=
-
1), a
=
r/tr
-
2),
=
IE
i' t+mE7kx t) I -.k
=
1$+,?,F7kXJ 1 1Ftekt+-1 IIzE)f+-.',1$+,,, r/(r-
IIA1+-',A)r
11 r/tr- 1) 111,11 r 11
11 < E-t-+kYtwm
1)
< 11 :; II-')IIrcl'+,,,(It and (; is of size Combining
F,IIr,Jl'l where cr < maxt 11 and (17.24) (17.25)in (17.23)gives
-1.
(17.25) the inequalities in
ltv( + (() Il.f/llrcr+,,, 11 F,+,,,11ryJ, 1$+,,,IIrt,,,, f
IExtYtmmlI s
maxl
(17.26)
This completes the proof of (i). The proof of where (,,, (vJ+ (() is of size using with r/tr 1) and b 17.5(ii) is similar 1. (ii) a p -1.
=
=
=
=
-
For the array generalization, simply insert the n subscript after every random variable and scaling constant. The argument is identical except that 17.6 is applied in place of 17.5. .
17.3 Near-Epoch Dependence and Transformations Suppose that (Xjt,...,Xvt4' Xt g(...,Ft-1,FsFf+1 is a v-vector of LP-NED functions, and interest focuses on the scalar sequence ((aV)J, where (t: 1- F- R, RV, is a fv/f-measurable function. We may presutne that, uner certain condi1-q tions on the function, ($(#f) will be near-epoch dependent on fJSJif the elements of Xt are. This setup subsumes the important case v 1, in which the question at issue is the effect of nonlinear transformations on the NED property. The dependence of the functional form 9f(.)on t is only occasionally needed, but is worth ,...)
=
=
=
making explicit.
The first cases we look at are the sums and products of pairs of sequences, for which specialized results exist. Let Xt and Ft be LP-NED on F, is AP-NED of size
17.8 Theorem Then Xt +
-mintfpx,tpyl
Proof Minkowski's Xt + F,) 11
-
(J/IJof respective
sizes
-(px
and
-f?y.
.
inequality gives E'7'''''(m + F,) II,<
<
+ 11 112,F7''''''A)Il, FJ
-
dhx tm
+
d% tm
''
< dvmt
-
A7''''''1',11,
(17.27)
Theory of Stochtic
268
wheredt
maxf
=
+vp,F and v,n = vxm
/vt JFlr ,
Processes
tnt,?l-'nint'x,'rll
=
.
.
A variable that is Q-NED is AP-NED for 1 S p S q, by the Liapunov inequality, so there is no loss of generality in equating the orders of norm in this result. The same consideration applies to the next theorem.
17.9 Theorem Let Xt and l be Q-NED on (FrJof respective sizes
-x
and
-tpj..
-.rninttpsfpxl
Then, XtYt is AI-NED of size
.
Proof By rearrangement, and applications of the triangle and Cauchy-schwartz
inequalities, we have E IXtn
-
F7':(mF,)
j
j
+ (me1+-,1F, (F7;m)(M+-''''''J',)) = E xtyt x,l+-rlyfl -
-
-
Fj+-,1((-Y,eX,1m)( 1', A'7': F,)) -
-
IIe'7''',''l',II2II-Y,e'7lx)llz
+ l', 17,11lI2 < 111,11211 -
F, , 11 Ilxre7,,,'''.&II + f'',v,,,Ft/'/v,l -
-
zdkivmY +
ztl%- + ir,11 11 I
11 < 11.1,
dt
=
maxt
j
-
el+-,,,'''Frll ,
(17.28)
F,1128,,dtidxtl IIx,II2t//, 11 Vm
=
X F X 1'+ Mm Mm + MmTm
and gtyyj-rninltpxrli
=
.
g
ln both of the last results we would like to be qble to set 1$ = Xtwjfor some finite j. A slight moditication of the argument is required here. 17.10 Theorem If Xt is AP-NED on ( Wl so is Xtwjfor 0 ,
<
j
<
x.
Proof lf Xt is fP-NED, then Il-Y,w Ext-q -
I@f'+Xy+-'''''') 11, 11,< 2 11.X',+./EXt+) IT',++'t.,+-:) -
S dt
using 10.28,
where
#;X
2JXf
(17.29)
vm,
we can write Xt-vj E t--Yrm./ f d'txvm' 11 l S7,1) 11p =
+j.
(17.30)
-
,
where #'
Vr?j
and
Mm' is
of size
-(p
=
if vm is of size
v0
,
Tm-j, -..
m <
j
>
J
m
.
.
Putting the last two results together gives the following corollary.
17.11 Corollary If Xt and Ff are Q-NED of size
-gx
and
-(ps
XtYtwkis AI-NED of
Near Fp/c
size
DepeWence
269
-minttps:xl
n
.
By considering Zt
the AI-NED m-gwjlk-gwj,
=
/
V?u
numbers can be given here as
Vo,
m f
+ 1 W/21
Vr,l-(1/21-1,
m
>
(W21+ 1
=
assuming wherev,a dYtvm+v,u +vruvp,, an d the constants are 4#f-wmY+k-g1a, that dxt and are not smaller than the corresponding Lz nonns. =
X
F
F
X
Al1 these results extend to the array case as before, by simply including the extra subscript throughout. Corollary 17.11 should be compared with 17.7, and care taken not to confuse the two. ln the former we have k fixed and finite, whereas x. The two theorems naturally the latter result deals with the case as m each other in applying truncation complement arguments to infinite sums of products. Applications will arise in subsequent chapters. More general classes of function can be treated under an assumption of continuity, but in this case we can deal only with cases where 9f(A) is Q-NED. Let --+
RV
4)(x):-I' F-> R, 'T i be a function of
real variables,
1/
ptxl
,.'r
2
and use the taxicab metric on V
)
IA'f1
=
2
-xi
I
(17.31)
,
izu1
RV,
2 to measure the distance between points x 1 an d x We consider a set of results that impose restrictions of differing sevelity on the types of function allowed, but offer a trade-off with the severity of the moment restrictions. To begin with, impose the uniform Lipschitz condition, .
l(1),(.1 1) 9/xl ) I < -
B , pt.tl
xll a.s.,
,
(17.32)
where Bt is a tinite constant. 17.12 Theorem Let Xitbe AZ-NED of size on f F2) for i = 1,...,v, with constants with constants dit. If (17.32)holds, ((aV)) is also AZ-NED on f F2) of size multiple maxftljll. of a finite Proof Let 4),denote a Fj+-l-measurable approximation to /ml. Then -a
-c,
II4h(A)
-
Ak
< 119,(2,) 9,112 FJ-,,''>,(.X))Il2
+
(17.33)
-
by 10.12. Since ((j+-;;1A) is an Tjlrl-measurable random for this choice of ,, and by Minkowski's inequality,
variable,
F7''''''9,(A)112< #,Ilp(.2',,.E7;7,-Y,) 112 119,(1,) -
V
u Bt
2 11-'t,e,!,,,',,-'t,II -
i=1 V
< Bt
ditvim f=1
(17.33)holds
Theoty of Stochastic Processes
270
f dtvm, and v,,,
where dt vffmaxjltitl by assumption. x =
=
(17.34)
Y the latter sequence v -1Zjazlvfm,
being of size
-.a
lf we can assume only that the Xit are TV-NED on F, for some p e (1,2),this argument fails. There is a way to get the result, however, if the functions (t are bounded almost surely. 17.13 Theorem Let Xitbe JV-NED of size dit, i = 1,...,v. Suppose that, for each t,
I9,(.1l)
-
Proof
l#1, -
For brevity, write ( i 9, I / zMmintz, 1 ). Then
=
,7(9)
on
%).
2
-4,:2
,
)) is Q-NED
x.
9r(X
I
on ( Ff ), for 1 f p f 2, with constants (A) j f M < x a.s., and also that
(#l xl)
2
#f(-Y ) l f m inl Blp
Then ((# finite multiple of maxjttf)
where Bt <
-a
,
(Ftl
of size
and let
,
2M) a.s.,
z
=
(17.35)
-apl2,
with constants a
#,p(A'l .Y2)/2M so that ,
,
Jtzs,)(9l f'(z>1,(9l <(2.2 Jtzs,lzz' Jtzoll'l -tbldr
-tg)2'
+
-
<
+
f (234)2s(z,)
= where Bjt
BitEpx,xl)pl,
(17.36)
Bq/2(gAy)1-p/2 combining (17.33)with
=
.
(17.36),we
can write
F7:,(#2Il2s #1rlIp(--F7''''''.f,) 119/2 114,,().;,) -
< #l, < B3t
v =1
v
plz
Ilx .#7j+-,,j&'.lI?) -
p11
ditpim
f=1
(17.37)
< dtvm, W
hel'e
dt
=
#1,vP/2maxj(J(/,2j and vm
assumption.
=
(v-l t=
kvimlpll ,
which is of size
-aplz
by
.
An important ekample of this case (withv = 1) is the tnmcation of Xt, although this must be defined as a continuous transformation.
17.14 Example For M
>
0 let
Near Fp/c
Dependence
.x, IxI f @)
=
M
M, x
>
M
-M, x
<
-M
(17.38)
xl x1( Ixls.v) +M(x/I
)1(I.vI>v) In this 2 l9(-Y1) #(-Y ) I < jxl x2j 1, and 17.13 can be used to show that I()IXJJis The more conventional truncation,
or, equivalently,
4,(x) =
.
case
-
-
,
so set B
G-NED.
=
Xflf IxlsM)
Q-NED if (X,) is
Xt, IXt I S M
=
(17.39)
otherwise,
0,
cannot be shown to be near-epoch dependent by this approach, because of the lack of continuity. n rther variation on 17.12 is to relax the Lipschitz condition (17.32)by letting the scale factor B be a possibly unbounded function of the random vari-
A
ables. Assume
I9,(.Y1) 9r(xl ) j -
where, for each
< B, (x1 x2)p(x1x2) a s., ,
,
(17.40)
.
f,
Btx
1xl ):
'!r
,
x'r
(17.41)
A/s-measurable function. To deal with this case requires a is a non-negative, B lemma due to Gallant and White (1988).
17.15Lemma Let B and p be non-negative < x, and Ilfpllr< x, for q k 1, and r >
Illpll2S
r.v.s and assume 2. Then
< Ilpllq 11#11#(t?-1) c'o,
2(IIpIIJ-2II#II-/1-l)Il#pII;)'/2Cr-1).
(17.42)
Proof Detine
c (IIpIlII#II/(-l)II#pII-rr)''f1-r), -
and let #1
=
Then by the Minkowski inequality,
ltapscls.
< II#1pII2+ -#1)pII2. II#pII2 11(.:
The right-hand-side
(17.43) ( 17.44)
ter'ms are bounded by the same quantity. First, 1/2
II#lpII2 =
Jspsctfplof'
jBp < c1?2
1/2
dp
/ c''2(IIpIIII#ll/f:-1))''2,
(17.45)
F/ldtpry of Stochastic Processes
272 applying the Hlder
inequality. Second,
11(/
Allplla
-
1/2
jsvwcBtldp
=
cl
<
1/2
j
-rll
(splup
Boc
Ij;/2
< t)2
'-r/zljlp
(17.46)
where the first inequality follows from r > 2 and #p/C > l Substituting for C in (17.45) and (17.46)and applying (17.44)yields the result. . .
The general result is then as follows.
17.16 Theorem Let l#f ) be a v-dimensioned random sequence, of which each element is Q-NED of size on (FCJ,and suppose thatY/m)is fvbounded. Suppose further that for 1 s q S 2, -a
< x, I1p(.X',,F7;Jl.fJllt?
< x, lI#,(-Y,,e-7:A',)IIc&c-1)
and for
>
r
2, < oo. Il#?(#/,%+--M#,)p(#f,Fj+->,)IIr
Thn
((f(A)
l
is JU-NED on
( Ff)
of size
-alr
-
2)/2(r
-
1),
Proof For ease of notation, write p for p(.X',,F7:#Jand B for BtxtnE7-zXtl. As in the previous two theorems, the basic inequality (17.33)is applied, but now we have
11(-)) e'7Jr9(A-f)l12< -
114,(.X',)#(F77X))Il2 -
llfpll2 (17.47) s 2IlpII:Cr-2)/2(r-1)Ij#jj4(;-(:21(2)(r-1)jjspjjrr/2(r-1) S
where the last step is by 17.15. For q f 2, V
Ilpllq<
< lIpIl2
V
77 f=1
dtvm7-ltrvf,,, f=1
-.E1+-,,,',,.X112 11.1/, s
=
and Mm v -1E( v im, which is of size Hence, under the stated assumptions,
where dt
=
v maxjltl)
=
.1
< div,?,tr-:D/zr-ll lIf(-'t)-.E:+-.p,''',(-Y,)lI2
7
-a
(17.48) by assumption.
(17.49)
jlrr/ztr-ll(y #J II#llJ)-(22(2)(r-1)jl#p s where =
t.
Observe the important role of 17.15 in tightening this result. Without it, the best we could do in (17.47)would be to apply Hlder's inequality directly, to obtain
Near fp/ch
273
Depeeence
< IIpIIuII#II2w(?-l), 1. lI#pII2 q
(17.50)
The minimum requirement for this inequality to be useful is that B is bounded almost surely pennitting the choice q = 1, which is merely the case covered by 17.12 with the constant scale factors set to ess sup Bt(X 1 ). The following application of this theorem may be conasted with 17.9. The moment conditions have to be strengthened by a factor of at least 2 to ensure that the product of Q-NED functions is also Q-NED, rather thanjust LI-NED. There is also a penalty in terms of the AZ-NED size which does not occur in the other case. .v2
,
Letav < x and 11 XtYt.Assume that lI#,II2r Xt,Yt)and ()tffl 1$II2r 2, and that Xt and Ff are ZZ-NED on ( Fll of size -J. Then
17.17Example <
x
for r
>
-'1F)
l
-
=
-Yl1'l K I
=
-)
I II l'l I + I K ( Ixl I + IFl I)(IFl
.11
.1)
-
-
-
IlFl l'1 + I-Y) l -
A
I)
#(A1,A)p(#l,.f),
(17.51)
=
defining B and p. For any q in the range (4/3,41,the assumptions imply K 1111 < II#(#),-Yl)Ilw(?-1) FllIw(?-1) II/(-1)+ 11 x,
11.X1 Il-Ylll < < IIl'l1I?+ Ilp(.Cl,-Yl)Il? IIl'lIl?+ Il?+
=,
(17.52) (17.53)
and 1') :K 11.11112r11 II#(-Yl,-Yl)p(-Y),-Yl)IIr Il2r+II.X1II2rIlFlll2r+II.X1lIlr lIFlII2rII FlI12r+ IIFlIllr +IIx)lI2rIlA:II2r+ IIFlII2rIIAIl2r +IIFlII2rII.X1II2r+
(17.54)
Putting -Y1
Xt and .Y1 E7-;,Xt, the conditions of 17.16 are satisfied for q in the range (4/3,21and Xtj is Q-NED of size -J(r 2)/2(r- 1). n =
=
-
17.4 Approximability ln j17.2 we showed that N-NED functions of mixing processes were mixingales, and most of the subsequent applications will exploit this fact. Another way to look at the JV-NED property is in tenns of the existence of a finite lag approximation to the process. The conditional mean E(Xt IJ;j+-/l)can be thought of as a function of the variables 6-m,...,Vt+m, and if (Ff) is a mixing sequence so is 1F(A! Fj+-'QI, by 14.1. One approach to proving limit theorems is to team a limit theorem for mixing processes with a proof that the difference between the actual sequence and its approximating sequence can be neglected. This is an alternative way to overcome the problem that 1ag funtions of mixing processes need not be mixing.
Iheory of Stochastic Processes But once this idea occurs to us, it is clear that the conditional mean might not be tlle only function to possess the desired approximability property. More generally, we might introduce a definition of the following sort. Letting 1$ be l x 1, l(2m+1) = F-y R as a gyr-yvlys-measurable function, where st*m t-m t-m we shall think of /C:R c(F,-?,,,...,F,+,,;). Dennition The sequenee fm) will be called (p 0) on the u-approximable 17.18 there seq'u of exists if for each Jncef/li'l s?i/fx'-'/'/-measurable m s s a f v,) sequence variables, and random >
II-Y,-/ClI,S dtvm,
(17.55)
c'o. (X,) will where fdtI is a non-negative constant sequence, and vm--A 0 as m approximable in probability (or mapproximable) on fFf ) if also be said to be > 0, there exist (T), (t$), and (vpr,las above such that, for every --)
#(IA
'71
-
>
dtn) f vm. u
(17.56)
The usual size tenninology can be applied here. There is also the usual extension to arrays, by inclusion of the additional subscript wherever appropriate. If a sequence is fv-approximable for p > 0, then by the Markov inequality
#( lXt 8-J'vj,;
where s'm'=
-
h'l I > dtn) < ()-PlI-Y,
hence an
-
< vm' /1'J'11p#
(17.57)
,
p-approximable
An process is also Q-approximable. although only in the casep 2 are we able to
AP-NED sequence is fp-approximable, is the best w-approximator in the sense that claim (from10.12) that Ext 11j+-'''m) the p-nonns in (17.55) are smaller than for any alternative choice of hl'. =
17.19 Example Consider the linear process of 17.3. The function
h';l X =
(17.58)
jvt-j
j=-m
unless @,'+-':) isdifferent from A'(#,I for xt since zw-approximator
112,hmt11, 77 jvt-j + jzzm1 + joy1 l l) and dt 7=,u+1( -
where vm
=
(J/',) is an independent process, but is also an -jb-vjt
=
-j
> =
< dts'm,
(17.59)
sup,ll 7,IIp. o
17.20 Example In 17.4, the functions g are fv-approximators for Xt, of intinite < x. n size, whenever sup,sf IIF,F,IIp One reason why approximability might have advantages over the AP-NED property is the ease of handling transfonnations. As we found above, transferring the JV-NED property to transformations of the original functions can present difficulties, and impose undesirable moment restrictions. With approximability, these difficulties can be largely overcome. The first step is to show that, subject to Zr-boundedness, a sequence that is approximable in probability is also fw-approximable forr < r, and moreover, that the approximator functions can be bounded for
Near Epoch Depcndence
each finite m. The following is adapted from Ptscher
and Prucha
(1991a).
17.21Theorem Suppose fXf) isfv-bounded, r> 1, andfvapproximablebyT. Then for 0 < p < r it is fv-approximable by J?;' /CI ( Ihmtj
.
Proof Since hmtis an fm-approximator of Xt, we may choose a positive sequence ,u) x. Also such that 8,u 0 and yet #( IXt h'l I > dtnm) f Mm --+ 0 as m f butlfmv,u choose a sequence of numbers (A&) having theproperties Mm -1/2 = For example, Mm vm would serve. T here is no loss of generality in assuming supm'/k-1 ; 1 B y Minkowski's inequality we are able to write -->
--
-
.-+0.
--:x,
.
IlA-/7IIpf
where
(17.60)
41,,,,II(A- 1:,)1(Ix,-pI>.,.u) 11p, =
A1t
=
:7''')1f Ix,-r;'I:.f,A<,,,) t 11(x,-
43
=
(I(xt
m
t l11
Hlder's
A)r,,+z4lm+Al,?,,
llp, 11p.
-''')1(
-
f
Ix,-'krllxtfj:m,shrylstjAfm)
inequality implies that 11A'1'11, S
> 1. II#II,qIIl'II?v&-l),
chooseq r/# and apply (17.61)to Amitfor i dtMm,again by Minkowski's inequality, IImIIr+ thefollowing inequalities. First, =
=
(17.61)
1,2,3. Noting that 112,/')'IIr< lsllfx= #(F), we obtain and that 11 -
/ Il-Y, l'7IIr#(I-Y, /'71 > dtnm) 41,,,, t4(IIA/IIrM,,,-1 + 1)MmMm. f -
-
(17.62)
Second, observe that
l IXt
'l
II';'I >
lS
dtnm.
and hence, when
slm>
&m,
Ixt- /';'I <
dtnm,
I/T1 >
P
-
dtsl)
t IXt I >
dt Mml i S #(
11,1>
dtMm
f IIXIII
rrdirum
by the
dtMm
-
-
,,,)
l
,,,)) -
nml-r
(17.63)
Markov inequality, so that Al,,, <
11.2,-:'IIr#(11,-/T1 < dtnm, I/')'I > J,M,?,)II-Y,IIrrtrfr(Az,, < (IIAlIr+ ,,,)-r
-
S
dtsl)
+ II.X',/f,II3ArJ',,,(M,,, (IIA/4Ilrr+' nm4-r. -
(17.64)
-1 b y 1 ) And lastly, (The final inequality is f'rom replacing Mm .
Atm 3 / dn, m
in view
of the fact that
7;T =
h'l on the set
(1765) .
/';I
fI I S
dtMml. We have therefore
established that Xt 11
A-''' t p S d'vm' t 11
-
(17.66)
,
where
v and
%'m'
'
M v,n+MmMm
=
0 by assumption, since r
--)
d;
=
m)-r+
-
(17.67)
1, and
>
.'
Il.Xr/ttllrr+l,IlA/t4ll rr, 1 ) ( lI.Y,/#,IIr, ztf/rnax
.
(17.68)
.
If o-approximability is satistied with dt
-approximability
17.22 Theorem Let 'J' htt...shmvt) be the fo-approximator of Xt = Xjt,...,Xvt)' RV EBtXt,h'1)E) < x for : > 0, F-> R satisfies (17.40),and If #t: of size then (j,4.Y,)is fo-approximable of size =
-(p.
-(?.
Proof Fix 6
>
0 and M
#( I#r(m) 9,('7)I > -
>
0, and define dt
Elxllff.The
=
dt) S PBt-knhllqpxphl't
+
>
#(#f(A'r,h'7)p(m,:')
F(#/(#s')')E)/3#
+
Markov inequality gives #f(m,:')
s, >
>
dtn, Btxt,h',
Plpxt,hll
>
dt&M).
M4 S M)
(17.69)
Since M is arbitrary the first term on the majorant side can be made as small as desired. The proof is completed by noting that V
Ppxt,h'l)
>
J,&M)
P
Im, X =1
KP
Ut Iff
=
-
'rri
>
dtnlM
>
ditnlM
ll
-
jzz1 %'
#( jXit
<
-
't',
t
l>
dit&lM)
V
f
Xvjm 0 as m -+
--y
x.
w
(17.70)
=l
It might seem as if teaming 17.22 with 17.21 would allow us to show that, given an and fv-bounded, Q-NED sequence, r > 2, which is accordingly Q-approximable hence Q-approximable, any transfonnation satisfying the conditions of 17.22 is and therefore also Q-NED, by 10.12. The catch with circumQ-approximable,
Near Fp/c
Dependence
277
venting the moment restrictions of 17.16 in this manner is that it is not possible to specify the Q-NED size of the transfonned sequence. In (17.67),one cannot put a bound on the rate at which thesequence (,,, ) may converge without specifying the distributions of the Xit in greater detail. However, if it is possible to do this in a given application, we have here an alternative route to dealing with transformations. Ptscher and Prucha (1991a),to whom the concepts of this section are due, define approximability in a slightly different way, in terms of the convergence of the Cesro-sums of the p-nonns or probabilities. These authors say that Xt is fv-approximable p > 0) if
limsup1 -
n
7) 111,-
'yllp
0 as m
--
11 >1
n-x
and is Q-approximable if, for every
(17.71)
.-->
x,
0,
>
n
1
limsupn-X#(lx,- :,1 > )
O as m
--->
(17.72)
--+
x.
>1
n-yx
lt is clear that we might choose to define near-epoch dependence and the mixingale property in an analogous manner, leading to a whole class of alternative convergence results. Comparing these alternatives, it turns out that neither definition dominates, eachpermitting a form of behaviourby the sequences which is ruled out by the other. If (17.55)holds, we may write n
1
limsupn Y 112,--
'/lln
'''
n-yx
''r
-
A
>1
1
n
Tdt n
< limsup
v,a
-->
0
(17.73)
>1
n--hx
so long as the limsup on the majorant side is bounded. On the other hand, if (17.71) holds we may detine n
v,,,
=
limsup
-
17) 111,hmt11,,
(17.74)
>1
n-x
will satisfy 17.18 so long as it is t'initefor and then dt = supp,t 111, Tllp/vml finite t. Evidently, (17.71)permits the existence of a set of sequence coordinates for which the p-norms fail to converge to 0 with m, so long as these are ultimately negligible, accumulating at a rate strictly less than n as n increases. On the other hand, (17.55)permits trending moments, with for example #f 0(/), -
=
,
0, which would contradict (17.71). Similarly, for ,u > 0, and vm > 0, define dtm by the relation >
#( IXt
-
h'II >
dtmm)
Mm,
=
(17.75)
0, define dt = suvmdtm. (17.56)is satisfied and then, allowing v,,i 0 and m if dt < x for each tinite t; this latter condition need not hold under (17.72).On the other hand, (17.72)could fail in cases where, for jed 8 and every m, -->
--
P IXt
-
';l
I > ) is
tending to unity as /
-->
x.
1A? THE LAW OF LARGE NUMBERS
18 Stochastic Convergence
18. 1 Almost Sure Convergence in 512.2.Sometimestheconditionis
wasdefinedformally
Almostsureconvergence stated in the form
# limsup IXn 11
> 6:
-
0, for a11 E
=
>
0.
(18.1)
n->x
1 where, for each (t) Yet another way to express the same idea is to say that #(C) > number E > finite of times as we pass and at most a any s 0, I-L() Xt)l I e C down the sequence. This is also written as =
-
#( IAk Xl -
where i.o. stands for
i.o.)
> E,
0, all E, > 0,
=
(18.2)
often'. probability in (18.2)is assigned to an attribute of the whole Note that the often' idea is sequence, not to a pmicular n. One way to grasp the words, consider the event U';=,,,l event that has occurred to IXn *-1 > erl; in 11 > for least beyond Xn el occurs whenever ( I at one n a given point m in the sequence' lf this event occurs for every m, no matter how large, ( IXn 11 > el occurs infinitely often. ln other words, tinfinitely
'infinitely
'the
-
-
-
.
Cr
f IXn 71 -
>
s, i.o.)
r
O U( IXn
=
-
#1
> El
vl=1 n=m
=
limsupt
IXn
-
11
>
el
.
(18.3)
n-x
Useful facts about this set and its complement
are contained in the following
j emma.
18.1 Lemma Let fEn
TlT be
G
an arbitrary sequence. Then X
(i) P
limsup En
=
1im P n--x
n--x
UEm
.
m=n X
(ii)#
liminf En n-'hoo
=
1im #
OEm
.
m=n
is decreasing monotonically to limsuw Fn. Part Proof The sequence (U7=,aFa1'Q1 (i) therefore follows by 3.4. Part (ii) follows in exactly the same way, since the
The frw of f-zzrg: Numbers
282
sequence f*n=mbJo;=lincreases monotonically to liminf En.
.
A fundamental tool in proofs of a.s. convergence is the Borel-cantelli /'r/;zz?.t7. This has two parts, the part and the part. The former is the most useful, since it yields a very general sufficient condition for convergence, whereas the second parq which generates a necessary condition for convergence, requires independence of the sequence. tconvergence'
Kdivergence'
18.2 Borel-cantelli lemma (i) For an arbitrary sequence of events fEn e
??7)T,
X
XpEnb
PEn, 1
0.
.0.)
<
-
=
(18.4)
n=1
(ii) For a sequence (En e 5 )7 of independent events, X
#(&)
i.o.)
Pk
=
x
=
1.
=
(18.5)
n=1
Proof By countable subadditivity, e
#
'
x
w
U
< XPEn4.
En
(18.6)
n=m
n=m
The premise in (18.4)is that the majorant side of (18.6)is tinite for m x (by 2.25), which further implies This implies X*n=mPEn) 0 as m
1.
=
-->
-->
X
1im P pl-hl
UEn
0.
=
Part (i) now follows by part (i) of 18.1. To prove (ii), note by 7.5 that the collection for any m > 0, and m' > m, /'
#
(18.7)
n=m
(AN,(
e
;)7
is independent; hence
;
;
(1 g)Ecn I-IPYn) 1-I =
h=n1
=
n=m
-
P(En44
n=m
< exp
-XPEn)
--
O as m'
( 18 8)
-+
.x,
.
n=m
by hypothesis, since e-x > 1 x. (18.8)holds for a1l m, so -
X
rtliminfEcnl
=
OYn
lim P m--koo
=
0,
(18.9)
n=m
by 18.1(ii). Hence, PEn i.o.)
=
'tlimsup
En4 = 1 Ptliminf E -
=
1.
.
(18.10)
To appreciate the role of tls result (the convergence partl in showing a.s. convergence, consider the particular case
Stochastic Convergence En
f
=
IXnlk
):
I
X())
-
283
*n=3PEn)< x, the condition PEn) > 0 can hold for at most a finite number of n. The lemma shows that P(En i.o.) has to be zero to avoid a contradiction. Yet another way to characterize a.s. convergence is suggested by the following
If
theorem. (Xn) converges a.s. to X if and only if for all 6: > 0
18.3 Theorem
lim # sup lXn
.:-1
S
-
s
nkm
pl-x
=
1.
(18.11)
e1 e @,
(18.12)
Proof Let X
A,,,(E)
=
(')1:
IS
IAk(t,))
-#()
n=m
and then (18.11) can be written in the fonn lim,,,-+x#(Am(e)) 1. The sequence U,,7=lAm(E), fA,,,(E) 17 is non-decreasing, so A,,,(E) = UXlA/:); letting A(E) 1. (18. 11) can be stated as #(A(:)) Define the set C by the property that, for each ) s C, (Xn((t))lT converges. That is, for (t) e C, =
=
=
3
lA%((J))
such that sup
??z()
-
X(l
numl)
1<
e, for all
Evidently,
e C= ) e Am(E) for some (l, so that CiA(E). tonly ito 1, proving #(A(E)) 1 for al1 E > 0. Set E To show if' assume k, and define #(A(E))
=
i) >
(18.13)
0.
HencePto
1implies
=
.
=
,
X
A*
=
f)
=
t=1
$Ik for positive integer
C
<x)
Ajlk)
=
tj
Atlkjc
(18.14)
.
&1
1. The second equality here is 1.1(iv). By 3.6(ii), #(A*) 1 #(U7=1A(1/k)9 z4* is a convergent outcome in the sense of (18.13), hence A* But every element of c C, and the conclusion follows.w =
=
-
The last theorem characterizes a.s. convergence in tenns of the unform proximity of the tail sequences ( IXnll) X()) l 1';.,,,to zero, on a setAm whose measure x. A related, but distinct, result establishes a direct link approaches 1 as m between a.s. convergence and uniform convergence on subsets of (. -
--y
-6:t-:
X, there exists for every > 0 a 18.4 Egoroff's Theorem lf and only if Xn X() uniformly on C(&). set C(8) with #(C48)) k 1 8, such that Ak() .-+
-
Proof To show if', suppose aL()) converges uniformly on sets 1 2 3 The sequence ( C(1/k) k e (N) can be chosen as non-decreasing tonicityof #, and #(U7=1C(1/k)) 1 by continuity of #. To show Eonly
1
,
;***
,
=
C(I/Q,k
=
by mono1et
tif',
The fuzw of fzzrgd Numbers
284
A,,()
O (t,):lXn(t,))-X()I
=
<
llml,
( l 8 15) .
n=klmj
-2-&'.
In view of a.s. km) being chosen to satisfy the condition #(A,,,(6)) 1 convergence and 18.3, the existence of finite klm) is assured for each m. Then if *
c(8)
nAa,tl,
=
(18.16)
rrI=1
convergence is uniform on C(l by construction; that is, for ever.y ) e C(8), IXnt)l X()) I < Im for n k km), for each m > 0. Applying 1.1(iii) and subadditivity, we tind, as required, -
#(c())
1
=
UAm()f
-#
m=1 *
1 -
E (1 -
#(z4m()))
m=1
k 1 8.
(18.17)
.
-
18,2 Convergence in Probability In spite of its conceptual simplicity, the theol'y of almost sure convergence cannot easily be appreciated without a grasp of probability fundamentals, and traditionally, an alternative convergence concept has been preferred in econometric theory. If, for any E > 0,
1im#(IXn X1 -
> f:)
=
0,
(18.18)
n'M
Xn is said to converge in probability (inpr.) to X. Here the convergent sequences are specified to be, not random elements (X,,()) 1T,but the nonstochastic sequences (#( IXn Xl > E) )T.The probability of the convergent subset of ('1 is left unspecified. However, the following relation is immediate from 18.3, since (18.11) -
implies (18.18).
18.5 Theorem If Xn
-66.9
X then Xn -E'.% X.
(a
The converse does not holdkConvergence in probability imposes a limiting condix. The tion on the marginal distribution of the pth member of the sequence as n negligible deviation of is approaches Xn X 1 probability that the from as we move down the sequence. Almost sure convergence, on the other hand, requires that byond a certain point in the sequence the probability that deviations are negligible from there on approaches 1. While it may not be intuitively obvious that a sequence can convergein pr. but not a.s., in 18.16 below we show thatconvergence in pr. is compatible with a.s. n/nconvergence. However, convergence in probability is equivalent to a.s. convergence on a ..-.h
Stochastic Convergence
285
subsequence', given a sequence that converges in pr., it is always possible, by throwing away some of the members of the sequence, to be left with an a.s. convergent sequence. 18.6 Theorem Xn -EI-> X if and only if every subsequence fAk, k e IN) contains a further subsequence fxnkcj, j e (N) which converges a.s. to X. 0 for any e > 0. This means if : suppose P( IXn XI> E) Proof To prove #( that, for any sequence of integers fns k e INl IXnk XI > e) 0. Hence for each j G LNthere exists an integer kU)such that 'only
-
--
-->
-
,
#( 1Xnk 71
>
-
1/.j) < 2-/, a11k k k(j).
(18.19)
Since this sequence of probabilities is summable over j, we conclude from the first Borel-cantelli lemma that
#( IXnkv
XI >
-
$lj i.o.)
=
0.
(18.20)
It follows, by consideration of the intinite subsequences Lj k J) 0 for every 6: > 0, and hence the that #( IXnkv XI > e i.o.) required. fxnw)J converges a.s. as : if (Ak) does not convergence in probability, there To prove > s)l subsequence (rl1)such that inflf#tlAk-rl s, for some 6: > which nlles out subsequence of in out convergence (?u), pr. on any a.s. on the same subsequence, by 18.5. w =
-
Kif'
18.3 Transformations
for J > 1/:, subsequenc: must exist a 0. This rules convergence
and Convergence
The following set tools of asymptotic even though most k-vector Xn is said
of results on convergence, a.s. and in pr., are lndamental theory. For completeness they are given for the vector case, of our own applications are to scalar sequences. A random to converge a.s. (in pr.) to a vector X if each element of Xn converges a.s. (in pr.) to the corresponding element of X. 18.7 Lemma Xn X a.s. (in pr.) if and Only if IlAk #II 0 a.s. (in pr.).19 -->
-->
-
Proof Take first the case of a.s. convergence. The relation
expressedas
k
P 1im n-yoo
for any
0. But
6: >
xni Xill X f=1
<
-
(18.21)implies
# li m 1Xni Xi I < -
:2
Il#n #ll -
-E't-y 0
may be
(18.21)
that E,
=
1,
.
..,k
=
1,
(18.22)
n'-oo
ito observe that if (18.22)holds, f'tlimp,--hxllAk #1I To prove IS 1 for any : > 0. To get the proof for convergence in pr., replace < k E) #(limn-+x...) everywhere by limn-+x#(...), and the arguments are identical. w proving
Konly
'if'.
-
,
=
,
The fww of f-zzr'd Numbers
286
There are three different approaches, established in the following theorems, to the problem of preserving convergence (a.s. or in pr.) under transformations.
18.8 Theorem Let Rk R be a Borel function, 1etCg R be the set of continuity points of g, and assume PX e G) = 1. #(m. X then g(Xn4 (i) If Xn -Z'.:o .-.E'.4 X then gxn) (ii) If Xn -->
':
-6.+
-6-+
'(m.
t
is by hypothesis a set D e @, with #(D) = 1, such that A%() #4(z)),each (t G D. Continuity and 18.7 together imply that glxnll for each (.t e X-1(Cg ) ch D. This set has probability 1 by 3.6(iii). g(#()) Toprove (ii),analogous reasoning shows that, for each r > 0, 3 > O such that Proof For case
(i), there
--
--
1t0:IlX()
-A)lI
Note that if #(#)
) c'-kr,sj
<
c (: l'(A())
1 then for any A
=
PA rn #)
1 P(Ac
=
-
LJ
G
< :).
-p(#())l
(18.23)
@,
Bc) k PA)
-
PBc)
PA4
=
(18.24)
by de Morgan's law and subadditivity of #. ln paYcular, when PX e Cg) = 1, (18.23) and monotonicity imply
#((I#n-fIl
YS
<
#(I'(Ak)
-
#(mI <
E).
(18.25)
Taking the limit of each side of the inequality, the minorant side tends to 1 by hypothesis. w We may also have cases where only the difference of two sequences is convergent.
18.9 Theorem Let t#a) and (zn) be sequences of randomk-vectors (notnecessarily converging) and g the function detined in 18.8, and let P(Xn e Cg) = P(Zn e Cg) = 1 for evel'y n. 0. (i) lf IlA% 11 O then 1:(.Yn) #(za)I -E',ly 0 then I#(.2L)#(zn)1 0. (ii)If llAk -61..>
-f..>
-zn
-
-znll
-r...y
-
OQazzjvn EZa PCEX) = Z-a1(c#), EX = ()xnmjl, and FZ = J'ut E'n = Xn-lc ), PEZ) ! and 3.6tiii). Also let D be the set on which Il#nZn 11 = 1, by assumptlon The proof is now a straightforward variant of te preceding one, with converges. X EZ r1(C the set E c g ). . playing the role of
Proof
=
.
-
The third result specifies convergence to a constant limit, but relaxes the continuity requirements. R be a Borel function, continuous at a. 18.10 Theorem Let g: R If Xn gx ga). (i) a then (ii) If X. -E(- c then g(Xn4 -E(-> ga). --,9
-6-,.-+
-6-y
Proof By hypothesis there is a set D G F, with #(D) = 1, such that Xn c, ga) for (1)e D, proving (i).Likewise, each (l) e D. Continuity implies #(A%()) --
-->
1:
llA%()
-cll
<
l c lt,):l#(Ak()) -g(c)I
and (ii) follows much as in the preceding theorems.
*
< El,
(18.26)
Stochastic Convergence
287
Theorem 18.10(ii) is commonly known as Slutsky's theorem (Slutsky 1925). These results have a vast range of applications, and represent one of the chief reasons why limit theory is useful. Having established the convergence of one set of statistics, such as the first few empirical moments of a distribution, one can then deduce the convergence of any continuous function of these. Many commonly used estimators fall into this category.
18.11 Example LetA,y be a random matrix whose elements converge a.s. (inpr.) to a limit A. Since the matrix inversion mapping is continuous everywhere, the A ) follow on app j y jyjg j.: : ejement by results a.s.lim An-1 A (plimA,y element. E! -1
-1
-1
=
=
.
The following is a useful supplementary result, for a case not covered Slutsky theorem because Fn is not required to converge in any sense.
by the
18.12 Theorem Let a sequence (Fn)Tbe bounded in probability (i.e.,Op(1) as n x); if Xn -EI-> 0, then XnYn.-CL>0. FPn
Proof For a constant # > 0, define L1 ( Iyklus). The event ( lXaF,,I 6: > 0 is expressible as a disjoint union: IXnYnI sl l IA%IIF$ l k e:1 tp f IXnIIi'n F$ l El
t
For any :
>
0,
-
.
( IXn IIF,, I k #(l-Yn lIi
Ik
c l IXn I 2 e/#l,
:1
el for
=
=
s) < #(l-Yn 1
zlB4
-->
(18.27)
and 0.
-->
(18.28)
> 0, B& < cxa such that By the 0:(1) assumption there exists, for each YB YB #( IL Fn%I > 0) < for n e ENSince ( IXnIIF,, n l sl c l Il'n n I > 01, (18.27)and additivity imply, putting B = Bs in (8.28),that -
-
-
.
lim P( lXnk
lk
e)
<
(18.29)
.
n.->x
The theorem follows since both 6: and 6 are arbitrary.
.
18.4 Convergence in L Norm #
Recall that when F(l Xn IP) < x, we have said that Xn is fw-bounded. Consider, for lf F(llxnllp < :cn all n, and limn-jxllAk --Y11p 0, p > 0, the sequence ( II-L-XIIpl7. When p = 2 we speak of Xn is said to converge in N zm?wzto x (writeXn l convergence in mean square (m.s.). Convergence in probability is sometimes called fvconvergence, terminology which can be explained by the factthatfv-convergence implies fx-convergence for 0 < q < p by Liapunov's inequality, together with the following relationship, which is immediate from the Markov inequality. =
---+
m.
-1f,+
18.1 Theorem lf Xn The converse
X for any p
>
0, then Xn .T.LA X. n
does not follow in general, but see the following theorem.
18.14 Theorem If Xn
.-#-(..>
X, and
( IL IPITis uniformly
integrable, then Xn -f:E.-)X.
288
The fzzw of zlr':
Numbers
Proof For e > 0, e-lAk
-11:
=
F(1( Ix,,-xl''>:) lXn
K e'(1(Ixn-xj''>EJ
-11#)
+
F(1f
--YlP) laY?,
Ixa-xtzkEl
+ s. l-Yn-XI#)
(18.30)
x. Uniform integrab0 as n Convergence in pr. means that #( lXn Xl > e) ility therefore implies, by 12.9, that the expectation on the majorant side of (18.30) converges to zero. The theorem follows since e: is arbitrary. K .-+
.-.+
-
We proved the a.s. counterpart of this result, in effect, as 12.8, whose conclusion can be written as: jXn X1 -E.. 0 implies A'lXn XI 0. The extension f'rom the fw1case to the Lp case is easily obtained by applying 18.8(i) to the case #(.) -,-h
-
-
lP. =I .
One of the useful features of Lp convergence is that the Lp norms of Xn X define a sequence of constants whose order of magnitude in n may be determined, providing a measure of the rate of approach to the limit. We will say for example that Xn converges to X in mean square at the rate nk if Ijxf-.Y112= On-h, but not on -k). This is useful in that the scaled random variable nkxt A') may be non-degenerate in the limit, in the sense of having positive but finite limiting variance. Determining this rate of convergence is often the tirst step in the analysis of limiting distributions, as discussed in Part V below. -
-
18.5 Exnmples Convergence in pr. is a weak mode of convergence in that without side conditions it does not imply, yet is implied by, a.s. convergence and Lp convergence. However, there is no implication from a.s. convergence to Lp convergence, or vice versa. A good way to appreciate the distinctions is to consider cases where one or other mode of convergence fails to hold. dpathological'
18.15 Example Look again at 12.7, in whichAk 0 withprobability 1 1/n, and Xn = = n with probability 1/n, for n 1,2,3,.... A convenient model for this sequence where m is Lebesgue is to let ( be a drawing from the space (r0,1),m,1j,pz) variable and random define the measure, =
Xnlj
=
n,
)
-
e (0,1/a),
(18.31)
0, otherwise.
The set f : limuAkt) # 0) consists of the point (0J, and has p.m. zero, so that Xn -E't-A 0 according to (18.1).But Fl XnlP = 0 (1 1/rl) + t'l'ln = n#-1 lt will be recalled that this sequence is not uniformly integrable. lt fails to converge in 1 for every n. The Lp for any p > 1, but for the case p = 1 we obtain Exn) limiting expectation of Xn is therefore different from its almost sure limit. n -
.
.
=
The same device can be used to define a.s. convergent sequences wltich do not converge in Lp for any p > 0. It is left to the reader to construct examples.
Stochastic Convergence
289
18.16 Example Let a sequence be generated as follows: .Y1 1 with probability 1; xztxzl are either (0,1) or (1,0) with equal probability; (X4,X5,X are chosen from (1,0,0),(0,1,0),(0,0,1) with equal probability; and so forth. For k 1,2,3,... the next k members of the sequence are randomly selected such that one of them is unity, the others zero. Hence, for n in the range (kt/c 1) + 1, jkk + 1)J,P(Xn 1) 1/k, as well as FIXnlP 1/k for p > 0. Since k .-.A x as n .-+ x, it is clear that Xn converges to zero both in pr. and in Lp norm. But since, for any n, Xnwj 1 a.s. for infinitely many j, =
=
-
=
=
=
=
#(I-L I <
i.o.)
E,
=
(18.32)
0
for 0 f 6: f 1. The sequence not only fails to converge a.s., but actually converges with probability 0. Consider also the sequence fkl/rxn J whose members are either 0 or kllr in the ,
lk(k klrx I#) PVr-1 an d b suitable n y range (r'kk 1) + 1 z + 1)j. Note that E( I choice of r we can produce a sequence that does not converge in Lv fop > r. With 1 for al1 n, but as in 18.15, the sequence is not uniformly r 1 we have Ekxn) integrable. The limiting expectation of the sequence exists, but is different from the probability limit. I:a =
-
,
,
=
=
ln these non-uniformly integrable cases in which the sequence converges in Lj but not in f.,l+e for any 0 > 0, one can see the expectation remaining formally well-defined in the limit, but breaking down in the sense of losing its intuitive intepretation as the limit of a sample average. Example 18.15 is a version of the well-known St Petersburg Paradox. Consider a game of chance in which the player announces a number n e N, and bets that a succession of coin tosses will produce n heads before tails comes up, the pay-off for a correct predidion being E2 l+1 T h e probability of winning is 2 -/-1 so th e e xp ected winnings are f 1', that is to say, game' if the stake is fixed at f 1. The sequence of random winnings it is a generated by choosing n 1,2,3,... is exactly the process specified in 18.15. ao Xn .
,
Efair
=
If n is chosen to be a vel'y large number, a moment's reflection shows that the probability limit is a much better guide to one's prospective winnings in a t'inite number of plays than the expectation. The paradox that with large n no one would be willing to bet on this apparently fair game has been explained by appeal to psychological notions such as risk aversion, but it would appear to be an adequate explanation that, for large enough n, the expectation is simply not a practical predictor of the outcome.
18.6 Laws of Large Numbers n-1Z7=1Xf.Suppose that E(Xt) = Let (m)Tbe a stochastic sequence and definex g,?and n -1Z7=1p g, w ith lp,1 < pa,' this is trivial in the mean-stationary case in which gf g, for al1 t. In this simple setting, the sequence is said to obey the weak law of large numbers (WLLN) when kn -CC..Ag,, and the strong law of large numbers (SLLN) when kn p,. These statements of the LLNS are standard and familiar, but as characterizations =
--)
=
-.62-.4
Ihe fzzw of fzzr'd Numbers
290
of a class of convergence results they are rather restrictive. We can set jt 0 with no loss of generality, by simply considering the centred sequence (A) g,flq; centring is generally a good idea, because then it is no longer necessary for the time average of the means to converge in the manner specified. We can quite easily have n -1E7=1h4 cxa at the same time that n-1E: 1(AX)) w) 0. In such cases the 1aw of large numbers requires a modified interpretation, since it ceases to make sense to speak of convergence of the sequence of sample means. =
-
--)
-
--
=
More general modes of convergence also exist. It is possible that kn does not that there exists a converge in the manner specified, even after centring, but -1 (Jn ):' such that an 1 cxa and an X1=jXt-...+0. Results sequence of positive constants will and others too, in a fully general array subsume possibilities, below these ,21 k l n=1 i s a triangular stochastic array with formulation of the problem. lf f(AICJ (1k1':.1 an increasing integer sequence, we will discuss conditions for =
Sn
kn
=
Txnt--EC->0. >1
(18.33)
A result in this form can be specialized to the familiar case with Xnt kn n, bu t there are important applications where the an-1 Xt g' and an greater generality is essential. We have already encountered two cases where the strong law of large numbers applies. According to 13.12, kn g' F(X1) when (m) is a stationary ergodic sequence and FI I < x. We can illustrate the application of this type of result by an example in which the sequence is independent, which is sufficient for =
=
-
=
-810
=
.11
ergodicity.
18.17 Example Consider a sequence of independent Bernoulli variables Xt with
1) P(Xt 0) :1.,that is, of coin tosses expressed in binary form (see 12.1). The conditions of the ergodic theorem are clearly satisfied, and we can E(Xt) j. This is called Borel's normal number conc lu de that n-1E) jA theorem, a normal number being defined as one in which Os and ls occur in its binary expansion with equal frequency, in the limit. The normal number theorem therefore states that almost every point of the unit interval is a normal number; that is, the set of normal numbers has Lebesgue measure 1. Any numberwith aterminating expansion is clearly non-normal and we know that a1l such numbers are rationals; however, rationals can be normal, as for example ,t, which has the binary expansion 0.01010101010101... This is a different result from the well-known zero measure of the rationals, and is much stronger, because the non-normal numbers include inutionals, and fonn an uncountable set. For eyample, anynumberwithabinary expansionof theformo.l 1:1 11:21 1&g11...where the bi are arbitrary digits is non-normal; yet this set can be put into 1-1 corin other words, with the points of respondence with the expansions the whole intelwal. The set of non-normal numbers is equipotent with the reals, but it none the less has Lebesgue measure 0. (a Pxt
=
=
=
=
--f..y
=
=
X
.bjbz%,...,
A useful fact to remember is that the stationary ergodic propery is preserved under measurable transformations', that is, if (A) is stationary and ergodic, so
Stochastic Convergence
291
is the sequence (#(.YJ)whenever g: R F.-+R is a measurable function. For example, 2 < x to be able to assert that n 11J=laY,2 we only need to know that E(X34 2 T he ergo dic theorem serves to establish the strong law for most stationary EX34. sequences we are likely to encounter', recall from j13.5 that ergodicity is a weaker property than regularity or mixing. The interesting problems in stochastic convergence arise when the distributions of sequence coordinates are heterogeneous, so that it is not trivial to assume that averaging of coordinates is a stable procedure in the limit. Another result we know of which yields a strong law is the mmingale conver-1 gence theorem (15.7),which has the interpretation that an Z7=1.V () wjwnever f Z7.1A)) is a submartingale with FI17=1A I < cxl uniformly in n, and an oo. This particular strong 1aw needs to be combined with additional results to give it a broad application, but this is readily done, as we shall show in j20.3. But, lest the law of large numbers appeyr an altogether trivial problem, it might also be a good idea to exhibit some cases where convergence fails to occur. --t.!..y
-
-s...,
--->
18.18 Example Let (m) denote a sequence of independent Cauchy random variables w ith characteristic function 9xf(X) e- lXI for each t (11.9).It is easy to I.I verify using formulae (11.30) and (11.33) that #:a(D= e -aI,l/,, e xxord. ing to the inversion theorem, the average of n independent Cauchy variables is also a Cauchy variable. This result holds for any n, contradicting the possibility that Xncould converge to a constant. l:a =
-
=
.
18.19 Example Consider a process l
Xt
lgsz.
=
Xt- I +
=
jthzt,
t
1,2,3,
=
(18.34)
...
J=1
with Ab 0, where IZ/J7is an independent stationary sequence with mean 0 and variance c2, and (% 17is a sequence of constant coefficients. Notice, these are indexed with the absolute date rather than the lag relative to time /, as in the linear processes considered in j14.3. For m > 0, =
COv(X/,Xtwm)
=
VartXJ c21
t
=
x=l
Fs. l
(18.35)
< x, i n this case the effect fw-bounded requires Zyzzlvs * with of the innovations declines to zero t and Xt approaches a limiting random Without the square-summability assumption, Vartl An X, variable say. example of the latter case is the random walk process, in which vs 1, all s. Since Cov(xY1,m) for every /, these processes are not mixing. Xnhas zero mean, but
For
(X,)Tto be uniformly
.
-->
=.
=
=
W1c2
n
1 7'vartx,l+
vara) nw =
>1
n
277 >2
J-l
yvartxp
(18.36)
.
j=1
c27,=1vl; otherwise Vartxa) x. In either then limn-oxvartk) fixed limit, being either stochastic fails the fYa to converge to a ) sequence case
If
17
J=
lvy?
<
-->
c,o,
=
292
The fww of zzr'd Numbers
asymptotically, or divergent. n These counter-examples illustrate the fact that, to obey the law of large numbers, a sequence must satisfy regularity conditions relating to two distinct factors: the probability of outliers (limitedby bounding absolute moments) and the degree of dependence between coordinates. ln 18.18 we have a case where the mean fails to exist, and in 18.19 an example of long-range dependence. ln neither case can Xn be thought of as a sample statistic which is estimatinj a parameter of the underlying distribution in any meaningf'ul fashion. ln Chapters 19 and 20 we devise sets of regularity conditions sufficient for weak and strong laws to operate, constraining both characteristics in different configurations. The necessity of a set of regularity conditions is usually hard to prove (the exception is when the sequences are independent), but various configurations of mixing and fw-boundedness conditions can be shown to be sufficient. These results usually exhibit a trade-off between the two dimensions of regularity; the stronger the moment restrictions are, the weaker dependence restrictions can be, and vice Versa.
One word of caution before we proceed to the theorems. In j9. 1 we sought to motivate the idea of an expectation by viewing it as the limit of the empirical average. There is a temptation to attempt to defne an expectation as such a limit', but to do so would inevitably involve us in circular reasoning, since the arguments establishing convergence are couched in the laguage of probability. The aim of the theory is to establish convergence in particular sampling schemes. lt cannot, for example, be used to validate the frequentist interpretation of probability. However, it does show that axiomatic probability yields predictions that accord with the frequentist model, and in this sense the laws of large numbers are among the most fundamental results in probability theory.
19 Convergence in Lp Norm
19.1 Weak Laws by Mean-square
Convergence
This chapter surveys a range of techniques for proving (mainly)weak laws of large numbers, ranging from classical results to recent additions to the literature. The common theme in these results is that they depend on showing convergence ih fv-norm, where in general p lies in the interval (1,21.Initially we consider the 2. The regularity conditions for these results relate directly to the case p variances and covariances of the process. While for subsequent results these moments will not need to exist, the faz case is of interest both because the conditions are familiar and intuitive, and because in certain respects the results available are more powedl. and variConsider a stochastic sequence (m)T,with sequence of means (g.r1T, 2 ances (cf )7. There is no loss of generality in setting g,f 0 by simply considering the case of faV w)7,but to focus the discussion on a familiar case, 1et us ---:: p, question, what initially assume Fn n -11J=1p< tfinite) and so consider therelation g)2 for 0:7 elementary conditions E(Xn An is are su fticient =
=
-
=
,
-->
-
Ek
g,)2
-
n
=
vartfnl+ EXn)
g)2.,
-
(19.1)
where the second term on the right-hand side converges to zero by detinition of p.. Thus the question becomes: when does Var(Xn) 0? We have -->
n
E ?z-1X(.2Vg,
Var(Xa) =
-
2
n
n
t- l
N(U 2XXcs /2
?z-2
=
+
>1
>1
(19.2)
,
ml
where (U Vartx and nts Cov(Xt,Xs). Suppose, to make life simple, we assume that the sequence is uncorrelated, with nts 0 for f # s in (19.2).Then we have the following well-known rsult. =
=
=
19.1 Theorem If
(.41Tis
uncorrelated sequence and *
77t
-2g2
,
< x
(19.3)
,
>1
-i+
then Xn
g,.
Proof
This is an application of Kronecker's 0. . implies VartYn) =
?z-21(U
lemma
(2.35), by
which
(19.3)
--
This result yields a weak 1aw of large numbers by application of 18.13, known conditton for (19.3)is that the as Chebyshev's theorem. An (amply)suftkient >nG
The zzw of fxzr'c Numbers
294
variances are uniformly bounded with, say, suprczff B < x. Wide-sense stationary 0(n-1). But since Vartxa) sequences fall into this class. ln such cases we have f1-8 /41), cl x is evidently permissable. If cl for Var(Xn)= a11we need is -lGl has O(f-1 and therefore converges by 2.27. terms of > 0, Z:=1f t is an unnecLooking at (19.2)again, it is also clear that uncorrelatedness essarily tough condition. It will suffice if the magnitude of the covariances can be suitably controlled. Imposing unifonn fu-boundebness to allow the maximum relaxation of constraints on dependence, the Cauchy-schwmz inequality tells us that lcsl S B for a11 t and s. Rearranging the formula in (19.2), =
-.->
-
-8)
,
Var(x-
n
1 n) n
+
-
n
xn x
p.2
c2, + g
<
+
n-1
+
2X
'2cn1
t,t-z
+
...
+
>3
n
x x Jc,.,-- )
u rnczl
n
>1
#
cf ,-I ,
,..1
w'
-
n
n
Xcl 2Xn
-
=
f=m+1
n-1
2
uE
n
-
(19.4)
mlBm,
-=1
where Bm supl ctz-ml and Bm ; B, all m k 1. This suggests the following variant on 19.1. =
,
< x 19.2 Theorem If (a))Tis a uniformly Q-boundedsequence, and Zm''Q1r?:-1#,,j where Bm = suprl nzt-ml then kn -V2-+g. ,
Proof Since n mjln < 1, it is suftkient by (19.4)to show the convergence of llnlkmcjBm n to zero. Tlzis fo llows immediately from the stated condition and . Kronecker's lemma. . -
-1
A sufficient condition, in view of 2.30, is Bm Ottlog m) -1- ), g s (). a very mild restriction on the autocovariances. There are two observations that we might make about these results. The first is to point to the trade-off between the dimensions of dependence and the growth of the variances. Theorems 19.1 and 19.2 are easily combined, and it is found that by tightening the rate at wlzich the covariances diminish the variances can grow faster, and vice versa. The reader can explore these possibilities using the rather simple techniques of the abote proofs, although remember that the Int,t-mI will need to be treated as growing with t as well as diminishing with m. Analogous trade-offs are derived in a different context below. which depends on these factors, can be The order of magnitude in n of Var((Xk), thought of as a measure of the rate of convergence. With no correlation and bounded variances, convergence is at the rate n-1/2 in the sense that Vartx'nl 0(n -1). but from (19.4),Bm Om-&) implies that Var(Xn) 0(n-%. lf convergence rates are thought of as indicating the number of sample observations required to get kn close to g, with high contidence, the weakest sufficient conditions evidently yield convergence only in a notional sense. lt is less easy in some of reqllltq below to link exolicitlv the rate of tlafamoro oonernl convemence with the =
,
=
,
=
=
.
Convergence in Lp Ntpn'rl
295
degree of dependence and/or nonstationarity; this is always an issue to keep in mind. Mixing sequences have the property that the covariances tend to zero, and the mixing inequalities of j14.2 gives the following corollary to 19.2. 19.3 Corollarylf
f.&l'7is either (i)uniformly Q-boundedand uniform
mixing with
X
l/2 y-lrpz-ljl
<
(19.5)
x
rrl=1
or (ii) unifonnly
for 8
fo-bounded Tm rn=1
1
W
>
/(2+)
0, and song
mixing with
(19.6)
< x,
then Xn -il.> p.. /2 Proof For part (i), 14.5 for the case r 2 yields the inequality Bm ; lB(m1 y.or /(z+y a r = 2 + 8 yields Bm S 6 IImII2+8a,,, part (ii), 14.3 for the case p xoting lIAll22+a, the conditions of 19.2 are satisfied in either case. . that B S =
.
=
.
O((1og ra)-2-f) for any 6: > 0. For A sufticient condition for 19 3(i) is (),,, -(1+V)(l+e)) Ottlog m) 19.3(ii), m sr : > () is suftkient. In the size terminology of j14.1, mixing of any size will ensure these conditiops. The most significant cost of using the strong mixing condition is that simple existence of the variances is not sufticient. This is not of course to say that no weak 1aw exists for Q-bounded strong mixing processes, but more subtle arguments, such as those of j19.4, are needed for the proof. =
=
19.2 Almost Sure Convergence by the Method of Subsequences Almost sure convergence does not follow from convergence in mean square (a counter-example is 18.16), but a clever adaptation of the above techniqueg yields a result. The proof of the following theorems makes use of the method of subsequences, exploiting the relation between convergence in pr. and convergence a.s. demonstrated in 18.6. Mainly for the sake of clarity, we first prve the result for the uncorrelated case. Notice how the condittons have to be strengthened, relative to 19.1.
19.4 Theorem lf fXf lTis uniformly fmbounded and uncorrelated, Xn -1.-y
g,. n
A natural place to start in a sufticiency proof of the strong 1aw is with the convergence pal't of the Borel-cantelli lemma. The Chebyshev inequality yields, under the stated conditions,
#(IXn FnI > -
:)
S
vart/n) s S a
z
(19.7)
for B < x, Fith the probability on the left-hand @'side going to zero with the . x. One approach to the probtem of bounding the quantity right-hand side as n -->
The fuzw of fzzr'c Numbers
296
#( IXn Fnl > E, i.o.) would be to add up the inequalities in (19.7)over n. Since the partial sums of 1/n form a divergent sequence, a direct attack on these lines 1 64 and we can add up the Juhsequence of '=' does not succeed. However, Za.1a = for 1,4,9, 16,..., probabilities in (19.7), n the as follows. -
-2
=
Proof of 19.4 By
,
.
(19.7), XP
IXn2Fn2I > e)
n2
1.64/<
f
-
(19.8)
x.
Ea
Now 18.2(i) yields the result that the subsequence (X2,n e N ) converges a.s. The proof is completed by showing that the maximum deviation of the omitted terms fkomthe nearest member of (Xn 2) also converges in mean square. For each n define
on2
lkk knzI
max
=
(19.9)
-
nl Sk<(rl+1)2
and consider the variance of vartx ) (1/n2)).1@ tends =
Of
arfangement
Dnz. Given the assumptions, the sequence of the monotonically to zero. For nl < k < (n + 1)2 rethe terms produces /
'k knz
2
n
(-,
=
-
l
j
knz+
1
-
k
Xt, X p=n2+1
(19. 10)
and when the sequence is uncorrelated the two terms on the right are also uncorrelated. Hence
vart#k Xn c)
l
=
-
n
-
2 2vart#nz)+
=B
-
-u
1
n2 B 1
-
Vartoazl cannot exceed the last tenn in
7) (rl-2-(n+
1)-2)
<
n2
-
< B
z&
nk
(k
+
k u n
-
.s.
1
n
-
nk
k
) X o'zj
v
2
< 1
2
1
>n2+1
2
)-s t,z 1
(a +
1)z
.
(19.11)
(19.11),and
F''l@-2(n+ 1)-2) -
=
1,
(19.12)
n
so the Chebyshev inequality giyes XpDnl n2
>
#
o Eu <
(19.13)
< x,
and the subsequence fDnz, n e INJ also converges a.s. Dn2k l(V-knl1for any k between n l an d (n+ 1)2 and hence, by the triangle inequality, ,
lkk g I s lknz Fn2I + Ikk knzl + lFl Fn2I -
-
-
-
.'. .:.J yyj;y''s ) qjg 5'?tj(j ?..Ey. jjjjy . . 1.
.:y '(''b
L'77;:'.. .'
.
, .
'.
x
.
Jl .'
.
. .
ggg
Convergence in Lp Notm
I+ u Iknz Fn2
(19.14)
Igk Fa2I
Dnz +
-
=
.
The sequences on the majorant side are positive and converge hence a.s. to zero, nl X But holds 8 for 1)2 does < k < (a + their sum. 1, SO (19.14) (p2,n so sr that k ranges over every integer value. We must conclude that kn l1' K --632''
We can generalize the same technique to allow autocorrelation. If (A)1' is uniformly Q-bounded, and
19.5 Corollary
B*
XBm
=
<
(19.15)
x,
m=1
where Bm
=
sup Int,t-mI then X- -E-> g.. u ,
Note how much tougher these conditions are than those of 19.2. lt will suffice -)-1-) for > 0 Instead of, in effect, having the here for Bm = Om -1(lo g merely decline to zero, we now require their summability. autocovariances .
Proof of 19.5 By (19.4),Var(Yu) < CB+ lB*)ln and hence equation themodified form, XP n2
Instead of Vartxk
(19.11) knz)
=
-
Iknl ik I >
1.64(,+zB.)
e) f
-
1 k
X j. p=n2+1
Xt
1 n
-
-
2
--
2 n Vart-Ynzl + 1 1 v = & P 2
k
1-n2-1
k
cl
Xzx.l
2
+
X X
Gt,t-m
,s=1
panaz
Nn
l-1
k
2 2 n - k 1 k -
j6)
Xnz
-
-
.
in
out and tnking expectations,
-
-
(lq
< x.
E2
we have, on multiplying
Var
(19.7)holds
X X wt-m f=nN.l
-
(19.17)
.
m=l-n2
The first term on the right-hand side is bounded by (1 n 2/k)2(#+ 2#*)/r the 2s+ 2 + 2 2 second by k n )(# + IB 4Ik and the third (absolutely)by 2(1 n lk) Adding together these latter tenns and simplifying yields -
,
-
-
.
,
Vartxk Xnz)< -
--j
1
1 1 w n+ ni Note, (1 n 2ln + 1)2)2 (19.13)we cgn write
1
-
y
-
1)-L
-
=
-
B
B
+
+
On-l)
1
2
2 ,
1
u y
+
1
1
=
Z
-
-
2 1 n -
-
n
+
1)d
+
2
B
--
1
+
n2
-
n
+
1)z
2
B
+ .
(19.18)
so the term in #* is summable. ln place of
The ww of fzzrgc Numbers
298
XpDnz
>
s) <
A-I#*
B+
<
Ez
n2
(19.19)
x,
where K3 is a finite constant. From here on the proof follows that of 19.4. w Again there is a straightforward extension to mixing sequences by direct analogy with 19.3.
19.6 Corollary If
(X/ITis eitherti)
unifonnly
Q-boundedand unifonn
mixing with
X
y')j1/2 <
(19.20)
x
tl=1
or (ii) uniformly
Qu-bounded for 8
>
0, and strong mixing with
X
/(2+) Xcu -=1
then kn
<
(19.21)
x,
-.6.'3..-h
0. n
Let it be emphasized that these results have no pretensions to being sharp ! They are given here as an illustration of technique, and also to define the limits of this approach to strong convergence. In Chapter 20 we will see how they can be
improved upon.
19.3 A Mmingale
Weak Law
We now wantto relax the requirement of tinitevariances,andprove fv-convergence for p < 2. The basic idea underlying these results is a truncation argument. Given a sequence tXf )Twhich we assume to have mean 0, define Ff 1( Ix,1ss)Xr, which equals Xt when IA l f B < c'o, and 0 otherwise. Letting Zt Xt Ff, the -EYt4 by construction, and kn Ys+ Zn. component' of Xt, notice that Ezt) Since F, is a.s. bounded and possesses al1 its moments, arguments of the type used in j19.1 might be brought to bear to show that X g,y (say).Some other An obvious technique approach must then be used to show that Zn-Z'..y g,z is to assume uniform integrability of ( IXrIPI In this case, supffl zrlr can be made as small as desired by choosing B large enough, leadin-y(vfcthe Minkowski inequality, for example) to an w-convergence result for Zn. A different approach to limiting dependence is called for here. We cannot assume that J'f is serially uncorrelated just because Xt is. The serial independence assumption would serve, but is rather strong. However, if we let Xt be a martingale difference, a mild strehgthening of uncorrelatedness, this property can also be passed on to l'r, after a centring adjustment. This is the clever idea behind the next result, based on atheorem of Y. S. Chow (1971).Subsequently (seej19.4) the m.d. assumption can be relaxed to a mixingale assumption. We will take this opportunity to switch to an array formulation. The theorems are easily specialized to the case of ordinary sample averages (see j18.6), but in subsequent chapters, array results will be indispensable. =
'tail
=
-
=
=
-i2=
-g,y.
=
.
Convergence in Lp Ntprrrl
299
19.7 Theorem Let t-L,,?L,)be a m.d. array, tcnfla positive constant array, and t/cnl an increasing integer sequence with kn x. 1f, for l S p f 2, '1'
(a) t lXntlcnt6l'l is
uniformly integrable,
kn
(b)limsup Xcn,< n-yx
kn
cn2f
(c) lim77 n-boo
0,
=
>1
--f.-
thenEylAk, t
and
x,
>1
0. o
of this result is where Xnt Xtlan, where fA,F,) with m.d. 5nt 5t and ftzr,lis a positive constant sequence. This is a sequence deserves stating as a corollary, since the formulation can be made slightly more The leading specialization
=
=
transparent. is a m.d. sequence, and fbt) lJnl, and (/cnlare 19.8 Corollary Suppose (.X',,?;,J7' constant positive sequences with an cxa and kn x, and satisfying IPl is uniformly integrable, 1 < p !-; 2, (a) ( kxtlbt kn O(Jn), and (b) ,
'1'
'1'
'/bt
=
>1
kn
b2 (c) 77 >1 ,
-1jg#g
then an
t 1J(t
=
t?(t);
.(eo g.
Proof Immediate from 19.7, defining Xnt
Xtlan and cnt
=
=
btlan. w
Be careful to distinguish the constants an and kn. Although both are equal to n in the sample-average case, more generally their roles are quite different. The case with kn different from n typically arises in arguments, where the array coordinates are generated from successive blocks of underlying sequence coorfor a e (0,1) (g.:jdenoting the largest integer dinates. We might have kn Erltxl 1-a x) where block does not exceed grl j. For an application of of length the below a this sort see j24.4. Conditions 19.8(b) and (c) together imply an 1 x, so this does not need to be separately asserted. To form a clear idea of the role of the assumptions, it is helpful to suppose that bt and an are regularly varying functions of their arguthat the conditions are observed if bt t 15 ments. It is easily verified by 2.271+F for f'J> and an log n fpr f' by choosing an n In for any ;$k a11 f, kn yields for an n particular,setting bt 1 tblocking'
=
-
-1,
-1,
-1.
-
-
=
=
IIXkII, 0. -->
=
=
(19.22)
will automatically satisfy condition (a), and condition (b) choosingan /,z1/7, willalso hold when bt Ot ). On the other hnd, a case where the conditions =
=
The Izzw of wr'd Numbers
300
l-1
t
1 and, for t > 1, bt Zy=l)y= 2 In this case condition (a) fail is where :1 2 imposes the requirement bn = Oan), so that bn2 O (Ju), contra dicting condition (b). The growth rate of bt exceeds that of t 5 for every j3> 0. =
=
.
=
Proof of 19.7 Uniform integrability implies that
supftlA%,/cn,lP1f n
Ix,,,/o,I>>))
0 as M
---)
x.
---
,1
One may therefore
tind, for
tr
SuP(1IXn,1( n
0, a constant Bs
>
<
such that
x
Ilp/cn/j f s.
(19.23)
lxa,I>s:cn,)
,f
zn, Xnt Fn,. Then since Exnt I@n,f-1) Fn,/-1)+ Znt F(Zn,I @n,,-l). F(l%,I
Define Ynt = Ak,1f Ixarlsseo/), and Xnt
=
Fn,
-
=
=
-
0,
-
By the Minkowski inequality,
177'-x-, j s jj(,--,-s(r--1v-,,-1)) j y-;k-
,
>1
+
,
,...1
Consider each of these right-hand-side
,,,,--1)) j:--7.)-j ti,,,/-str--rl
kn
'A''7 znt-fznrl
.
P
>1
(19.24)
terms. First,
t,(
))a
>--.'-j
u
9n,,-1))
(r-r-,(r,--lv,-,,-1.))
1/2
k
N
-F(I&fI@,,,,-1))2 = 77e'(l',,r ,-1 p TEYI
<
1/2
< #s
nt
1/2
k
X o2,
(19.25)
>1
>1
The tirst inequality in (19.25)is Liapunov's inequality, and the equality follows because ( Ynt ElYnt ITn,f-l)) is a m.d., and hence orthogonal. Second, -
kn
77 znt >1
kn
kn -
Ftznfl @n,,-l))
< P
+ 77Ilzsfslp 77lIF(z,,I@,,,,-1)11, >1
>1
kn
<
277 >1
kn
f leTcnt. lIz,,,1l,
(19.26)
>1
The second inequality here follows because f EE Izn,lP l@n,,-l)) F,,,,-l) 137 FIF(zn?I
=
Fl
zn,lP,
from, respectively, the conditional Jensen inequality and the law of iterated expectations. The last is by (19.3). lt follows by (c) that for ir > 0 there exists Ne k 1 such that, for n k Ns,
Convergence in Lp Ntprm
301
kn
2 < s-2:2 A-fnr f:
(19.27)
.
>1
Putting together
(19.24)with (19.25)and (19.23)shows kn
7'IXU;
,..a1 k 1 + 2,21c,,r
for n k Ne, where B this completes the proof. =
K Bz
(19.28)
p
<
x,
by con dition
(b). Since r is
arbitrary,
*
The weak law for martingale
19.9 Corollary
that
differences follows directly, on applying 18.13.
-/-% O k Under the conditions of 19.7 or 19.8, Z/21X,a u .
1 and set cnt 1/n and kn n as above, we get the result If we take the case p of faYt)is sufticient for convergence in probability of unifolnn integability that the sample mean Xn. This cannot be significantly weakened even if the martingale difference assumption is replaced by independence. If we assume identically distributed coordinates, the explicit requirement of uniform integrability can be dropped and Al-boundedness is enough; but of course, this is only because the uniform property is subsumed under the stationarity. You may have observed that (b) in 19.7 can be replaced by =
(b')
limsup-lxd,
=
=
kn <
x.
/=1
n--x
It suftices for the two tenns on the majorant side of (19.24)to converge in Lp, and the cr inequality can be used instead of the Minkowsk inequality in (19.26)
to obtain
kn
E
77(z,,,-F(z,,,I@n,,-l)
kn
p
K
c,(2k,,)P-1>7d,.
/=l
(19.29)
>1
However, the gain in generality here is notional. Condition (b') requires that limsuptr,-ox/cjc'i< x, and if this is true the same property obviously extends to /7 and an nY, flkcuf). For concreteness, put cnt = btlan as in 19.8 with bt ?7t' for x > 0, note that the where j3and y can be any real constants. With kn majorant side of (19.29)is bounded if a(1 + I$) < 0, independent of the value of p. This condition is automatically satisfied as an equality by setting an = k b t, but note how the choice of an can accommodate different choices of kn. Xt2l None the less, in some situations condition (b)is stronger than what we kmow to be sufficient. For the case p 2 it can be omitted, in addition to weakening the martingale difference assumption to uncorrelatedness, and uniform integrability to simple fu-boundedness. Here is the an'ay version of 19.1, with the conditions cast in the amework of 19.7 for comparability, although all they do is to ensure that the variance of the partial sums goes to zero. -
-
-
=
'y
-
The fww of fwr'c Numbers
302
19.10 Corollary lf (-L,) is a zero-mean stochastic an'ay with Exntxns) y: s, and (a) (Xntlcntl is uniformly Q-bounded, and
=
0 for t
kn
(b) 1im n--yx
thenEtzlxn,
7)c2a,
=
0,
>1
-1J-+
0. c!
19.4 A Mixingale Weak Law To generalize the last results from martingale differences to mixingales is not too difficult. The basic tool is the telescoping series' argument developed in j16.2. The array element Xnt can be decomposed into a tinite sum of martingale differences, to which 19.7 can be applied, and two residual components wlch can be treated as negligible. The following result, from Davidson (1993a),is an extension to the heterogeneous case of a theorem due to Andrews (1988).
19.11 Theorem Let the al'ra fAk,,V)=-xbe a Al-mixingale with respect to a constant array (cnf) lf (a) Lxntlcnttis uniformly integrable, .
kn
(b) limsup n--yoo
X cnt <
kn
(c) lim
x,
and
/=1
1 cnt
=
0,
>1
n-yx
where kn is an increasing integer-valued function of n and kn
-#.
'1'
x,
then
0. a
k xnt X/:1
There is no restriction on the size here. It suffices simply for the mixingale coefficients to tend to zero. The remarks following 19.7 apply here in just the same way. In particular, if Xt is a Al-mixingale sequence and fxtlbtj is uniformly integrable for positive constants (/7f) the theorem holds for Xnt = Xtlan and cnt btlan where an l=jbt.Theorems 14.2 and 14.4 give us the corresponding results for mixing sequences, and 17,5 and 17.6 for NED processes. It is sufficient for, say, Xnt to be fv-bounded for r > 1, and V-NED, for p 2 1, on a a-mixing process. Again, no size restrictions need to be specified. Uniform integrability of fxntlcntkwill obtain in those cases where ll-kllris finite for r > 1 and each /, and the XED constants likewise satisfy dnt >) II-L,IIr. A simple lemma is required for the proof: ,
=
=
integrable for # 2 1, so is the
19.12 Lemma lf the array fxntlcntis uniformly atray fEt-jxntlcnt J for j > 0. Proof By the necessity
part of 12.9, for any ir
>
0 3 8
>
0 such that
Convergence in Ap Norm
sup n
sup
,t
jyl.L,/o,l
dP
303
(19.30)
< :,
where the inner supremum is taken over all E e 5 satisfying #(F) < 8. Since 5n,t-j Satisfying #(F) g 5, (19.30)also holds when the supremum is taken overF e %not-j For any such F, < .
JsI
XntlcntIdP
l jyk-j
jyI
Et-jxntlcnt IJ#,
XntlcntIdp
=
(19.31)
by definition of Et-j.), and the conditional Jensen inequality (10.18).We may accordingly say that, for s > 0 H 8 > 0 such that
sup n.t
jsI
Et-jxntlcnt6dP
sup
(19.32)
< :,
taking the inner supremum over E e %n,t-jsatisfying #(F) < 8. Since Et-jxnt is Fa,f-measurable, uniform integrability holds by the sufticiency part of 12.9. . Proof of 19.11 Fix an integer j and let kn
77 k-qxnt Et-vj-kxnt). ,--1 is a martingale, for each j.
Fn./
=
The sequence
-
l Ynj, @n,n+jJ';=1
l kjxnt is uniformly
integrable by
Et-vj-txntjlcntl
-
19.12, it follows by
(a) and
-f?.t-A
k Xnf Zf21
(19.33) kn
kn
X
19.7 that
as a telescoping sum. For any M k 1,
M- 1
Ynj
(b) and (c) and
0.
vnj We now express
Since the array
X
=
j=3-M
Et+M-tXnt- X Et-uxnt,
>1
(19.34)
>1
and hence M- 1
kn
Xxn, X =
kn
Ynj +
j=1-M
>1
kn
X xnt -
Etmu-txnt)
>1
+
Xk-uxnt.
(1.35)
>1
The triangle inequality and the fal-mixingale property now give M- 1
kn
E
kn
s 77 FIFnyl + X-tn, >1
kn
XF1
xnt -
>1
j=-M
M- 1
< 77 j=-M
.#'*8N
-
az'N z'
* .
.
.
.
.
Q'
..
XFI Et-uxnt'
+
>1
kn
2(::Xcnr.
E I1k/1 +
According to the assumptions, !
A',+v-lAkll
n
- .-
the secondmemberon .3
-
(19.36)
>1
!
- - - --
.-
..
FY
the right-hand side of (19.36) ,
*
*'
%
,
'
P'
UIT-
The fww of zzr'd Numbers
304
/ for M k Mv. By choosing n large enough, the sum of 2M- 1 terms on the righthand side of (19.36)can be made smaller than z1:for any finite M, by (19.33).So, <e when n is large enough. The theorem is by choosingMk Me we have 171Vzlxkjl proved since is arbitrary. : w now A comparison with the results of j19. 1 is instructive. In an fu-bounded process, the Q-mixingaleproperty would be a stronger fonn o'f dependence restriction than specified in 19.2, just as the martingale property the limiting uncorrelatedness simple uncorrelatedness. The value of the present result is the is stronger than substantial weakening of the moment conditions.
19.5 ytpproxinAable
Processes
There remains the possibility of cases in which the mixingale property is not because of a nonlinear transformation of a AP-NED easily established process which cannotbe shown to preserve the requisite moments for application of the results in j17.3. In such cases the theory of j 17.4 may yield a result. On the assumption that the approximator sequence is mixing, so that its mean deviations converge in probability by 19.11, it will be sufficient to show that this implies the convergence of the approximable sequence. This is the object of the following -perhaps
theorem.
(Wn/l
is a stochastic array and the 19.13Theorem Suppose that, for each m e !N, centredarray fhmnt Ehmnl satisfies the conditions of 19.11. If the array l-Lrl with respect to a constant array (t&,) and limsupu-,x is Al-approximable by fJln;.'11 .-P-f-+ 0. < hen k B f Xtltdnt Ztjaa, t n t -
,
=,
Establishng the conditions of the theorem will typically be achieved using 17.21, by showing that Xnt is far-bounded for r > 1, and approximable in probability on hmntfor each m, the latter being pz-order lag functions of a mixing array of any
size.
Proof Since kn
kn
XXn,
f
>1
kn
kn
X xnt -
X
hmnt, +
>1
hmnt-F('J)
>1
by the triangle inequality, we have for
XF('lf)
(19.37)
>1
0
kn
kn
#
>
+
( j j (j j ) + (jX.j j !a) (!X.j I (a) X.jxn ,
X.jxn
uP
>
,
-
,) h:7
>
kn
kn
P
la
ht
-
F(I))
>
+
P
F(7,)
>
(19.38)
by subadditivity, since the event whose probability is on the minorant side implies at least one of those tnn the majorant. By the Markov inequality,
Convernence in fg Norm kn
P
X tXnr
-
hmntj
>
f
j'
>1
3E
g
kn
X xnt Jln >1
kn
f
3
-,
m
-
< 3 XFlkr F
305
u,I
m
-
>1
kn
X dnt v
m
(19.39)
.
>1
#( IXktzkEhmnt) I > &3) is equal to either 0 or 1, according to whether the non0 and stochasticinequality holds or does not hold, By the fact that Exnt) Al-approximability, =
IEhmntt I
=
IExnt)
-
E
(Mnf)
I < FIXnt
-
hmnt I < dmvm,
(19.40)
and hence kn
kn
kn
XF(''',,rl
<
>1
< Bvm. 77IEhmn 1,< Tdntvm
(19.41)
>1
>1
We therefore find that for each m e EN kn
limsup# n-+x
S
Txnt
>
>1
3B + limsup # 8 vm n--)x
=
3#
v,u
+
1(svs,>/a),
kn
X >1
m m Ehntjj hnt -
>
8
y
+
1(svm>&3)
(19.42)
by the assumption that hmntsatisfies the WLLN for each m e EN.The proof is x. completed by letting m w --
20 The Strong Law of Large Numbers
20.1 Technical Tricks for Proving
LLNS
In this chapter we explore the strong 1aw under a range of different assumptions, omindependentsequencestoner-epochdependentfunctionsof mixingprocesses. Many of the proofs are based on one or more of a collection of ingenious technical lemmas, and we begin by studying these results. The reader has the option of skpping ahead to j20.2, and referring back as necessary, but there is something to be said for forming an impression of the method of attack at the outset. These theorems are found in several different versions in the literature, usually in a form adapted to the pmicular problem in hand. Here we will take note of the minimal conditions needed to make each trick work. We start with the basic convergence result that shows wy maximal inequalities (for example, 15.14, 15.15, 16.9, and 16.11) are important.
20.1 Convergence Iemma Let (A)T be a stochastic sequence on a probability space (f,T,#), and 1et Sn Z:=IX, and So 0. For (t) e f, let =
=
Mll)
inf sup ISjb
=
I
Smfk
=
m j>m
lf P(M
>
e)
=
0 for all E
0, then Sn
>
-6-->
(20.1)
.
S.
Proof By the Cauchy criterion for convergence, the realization (,%())) converges u%1 ; e:for all j > m, for a11 : > 0; in other if we can find an m such that ISj words, it converges if M4(t)) < c, for al1 : > 0. . -
This result is usually applied in the following way.
20.2 Corollary Let lc,)Tbe a sequence of constants, and suppose there exists p 0 such that, for every m 0 and n > m, and every e > 0, P max lSj 5k1 -
>
e f
mmusn
wjct
where K is a t'inite constant. If X7
-
K
E
< x,
x-xn
X
c(,
(20.2)
>,1+1
then Sn
tcf) is summable
-16+
lim---jxzOf
S. =m+1c(
it follows by 2.25 that be the r.v. in (20.1). By definition, M ; supom jSj Sm1for any m
Proof Since
-
P(M
>
e)
lim m,-nt
sup ISj SmI > er -
j>m
>
0. Let M 0, and hence =
>
aw of fxzr'c Numbes
The Strong
<
r
--
307
=
Xc
lim
erpm--
pt
0
=
(20.3)
,
/=,n+1
where the final inequality is the limiting case of (20.2).20.1 completes proof. w
the
Notice how this proof does not make a direct appeal to the Borel-cantelli lemma to get a.s. convergence. The method is closer to that of 18.3. The essential trick with a maximal inequality is to put a bound on the probpbility of all occurrences of a certain type of event as we move down the sequence, by specifying a probability for the most extreme of them. Since S is tinitealmost surely, Xn 0 is an instant corollary of 20.2. Howalso used result the in a more subtle way in conjunction with can be ever, Kronecker's lemma. lf :=1 F, converges a.s., where f 1',1 (Xtlat and (th) is a 0 This sequence of positive constants wiyh an x, it follows that an-1:7 xt much of weaker L1=jXt condition itself. the of Most than is course a convergence applications feature at t, but the more general formulation also has uses. There is a standard device for extending a.s. convergence to a wider class of seqtlences, once it has been proved for a given class: the method of equivalent sequences. Sequences (A)';' and f F,):' are said to be equivalent if .-61-:
=
-j-.
,1-
*
.1
.
=
X
X Pxt
y,
J',)
<
(20.4)
x.
>1
0. By the first Borel-cantelli lemma (18.2(i)),(20.4)implies P(Xt l, i.o.) of only probability measure zero are there more than a ln other words, on a set finite number of f for which X,((J)) # F,4)).:y:
=
L1=j(Xt Fr) converges a.s.
20.3 Theorem lf Xt and Fr are equivalent,
-
Proof By definition of equivalence and 18.2(i) there exists a subset C of f1, with #(f C) 0, and with the following property: for all (J) e C, there is a finite nn((l))such that X,()) iX(9)for t > ,7(,(). Hence -
=
=
n0(tl9
n
X (m(t,))r/t,))) X (-Y/)) =
-
-
F,(t,))), V n k
/70(t,)),
>1
>1
and the sum converges, for a11 (l) e C.
.
The equivalent sequences concept is often put to use by means of the following
thorem.
20.4 Theorem Let (Al7 be a zero-mean
random
sequence satisfying
X
XE >1
Ix,IPla;
<
(20.5)
x
for some p k 1, and a sequence of positive constants the indicator function 1( Ix,Isn)()),
ttzf) Then, .
putting 17 for
lhe f-aw of fwr'd Numbers
308
X#(I#,l
>
<
a
(20.6)
x,
>1
lF(Al7ll 77 >1
lat <
(20.7)
=,
and for any r k p, X
77F(1A1
rliyaL<
>1
(20.8)
a
x.
result may be apparent. The indicator function is used to truncate a sequence, replacing a member by 0 if it exceeds a given absolute bound. The ratio of the tnlncated sequence to the bound cannot exceed 1 and possesses a1l its absolute moments, while inequality (20.6)tells us that the tnmcated sequence is equivalent to the original under condition (20.5).Proving a strong 1aw under (20.5) can therefore be accomplished by proving a strong 1aw for a truncated sequence, subject to (20.7)and (20.8). The idea behind tls
Proof of Theorem
20.4 We prove the following three inequalities:
#( IXt l
> J,)
F(1
-
11) 1/))/tz(
/ E I.Y,1#(1 < Ekxt6pllal;. -
Here the inequalities are because lXt is non-negative, respectively. F(l.Y,If71J)
IExtsll IIat
=
IF(.&tl
s
E
lx,lP(1
IP/Y>
l for
(.f) G
( IXt I >
tz,l,
and becaus
1/) Ilat
-
< F(1 Al (1
(20.9)
1/))/J,
-
-
j.;))Ia;
< E6xt3p4lal;.
(20.l0)
1/)). The -F(m(l 0, hence EXt$1) The equality in (20.10) is because Ex first inequality is the modulus inequality, and the second is because on the event lat? l 1Xt l > J,l, ( 11,1 IAIlat for p 1. Finally, by similar arguments to the above, =
=
F(lAl
for p rsLllalS F(I.)tI7'1tl)/t4 < E Ix,I#)/t7f.
The theorem follows on summing over t.
-
< r
(20.11)
.
There are a number of variations on this basic result. The tirst is a version for martingale differences in terms of thseone-step-ahead conditional moments, where
The Strong fzzw of fwtzr'd Numbers
309
the weight sequence is also allowed to be stochastic. The style of this result is appropriate to the class of martingale limit theorems we shall examine in j20.4, in which we establish almost-sure equivalence between sets on which certain conditions obtain and on which sequences converge. 20.5 Corollary Let f.X),Tf)be a m.d. sequence, let ( B$) be a sequence of positive T/-l-measurable r.v.s, and for some p k 1 let o
D
=
(t):
< XF(I -Y,IP l@,-1)((,))/M(t,))
>.
e
x
(20. 12)
>1
Also define o
D1
=
):
X#(Iml>
<,1 @,-1)())
<
e ?/
x
(20. 13)
>1 r
Dz
=
):
X IF(m11I1,-1)()
< x I/WXt,))
(20.14)
e 5
>1
D?
=
:
x XF(IA)1 >1
and 1et D' #(D')
=
=
rll
IT,-1)())/M() <
D1 r'n D1 ch D3. Then P(D
-
D')
=
e @,
x
0. ln particular, If #(D)
(20. 15) =
1 then
1.
Proof It suffices to prove the three inequalities (20.9),(20.10), and (20.11) for the case of conditional expectations. Noting that Ext ITf-l) 0 a.s. and using the fact that Wf is Ff-l-measurable, a11 of these go through unchanged, except that the conditional modulus inequality 10.14 is used to get (20.14). lt follows E D is in D'. . that almost every =
Another version of this theorem uses a different truncation, with the truncated variable chosen to be a continuous function of Xt; see 17.13 to appreciate why this variation might be useful.
20.6 Corollary Let t.Yf)Tbe a zero-mean random sequence satisfying (20.5)for p k 1. Define Xtlat,
l'f
=
Xtlz/tzf+
(.X)/I.X)I )(1 -
17)
=
IXt I f
at
1,
Xt > at
-1,
Xt
<
(20.16)
-at.
Then, fr
IEj'tb I 7-7 >1
(20.17)
The zzw of fwtzr'd Numbers X
XF1
'
1,,1
<
(20. 18)
p.
r
=,
,--1 Proof Write hat to denote atxtlk Xt I Inequalities are adapted as follows. .
(20.10)and (20.1 1)
of 20.4
f
IEtt l
lEXt1.1 + (1 llat = lExt (+tz,))(1 1t) Ilat < E IXt 1(1 Skllat + E l 1 1/ j 17))/J( + #( lX/I > < E( I.)1P(1 S 2F( IXt l#)/J(. =
-
-
12)(+J) -
-
-
at4
-
(20.19)
The second equality in (20.19) is again because Ext) = 0. The first inequality is an application of the modulus inequality and triangle inequalities in succession, and the last one uses (20.9).By similar arguments, except that here the cr inequality is used in the second line, we have F41 1'Jl
r)
< F!.Y,1t
+
(1 1tJ)(+tz 1rlaL -
< 2r-1(sj xgl/j S
r)
sj (1 1) j 2r-1(F( + P IXt I > IXt3P3L)lal; zre' jxt1pjtai. rfa;
+
-
J,))
The theorem follows on summing over t as before.
for p ; r
(20.20) .
Clearly, 20.5 could be adapted to this case if desired, but that extension will not be needed for our results. The last extension is relatively modest, but permits summability conditions for nonns to be applied.
20.7 Corollary (20.6),(20.7),(20.8),(20.17),and (20.18)a1l continue to hold if (20.5)is replaced by < 77F(lAI#)1/t#z7A
(20.21)
cxa
>1
for any q 2 1. Proof The modified
#(I#,l
>
forms of a
(20.9),and
of
(20.19)and (20.20)(say) are
l#)/Jf)1& K P IA1 > h)1& S E l-Yf
pyaqlllq lE(Yt4I S lE(Yt4I3Iq < J,1&(.s(jxtj
(20.22) ,
(20.23)
The Strong
zzw of r)l&
< (Fl yf 1
e'Ii',1r
311
zzr'c Numbebs
< zr/qE
pyalklq
Ix,1
(20.24)
1
where in each case the first inequality is because the left-hand-side member does not exceed 1. w For example, by choosing p = q the condition that the sequence summable is seen to be suftkient for 20.4 and 20.6.
( llxf/tzf lIp1is
20.2 The Case of lndependence The classic
results on strong convergence are for the case of independent sequences. The following is the three series theorem' of Kolmogorov:
20.8 Three series theorem Let (aV)be an independent sequence, and Sn = k1=3Xt. Sn S if and only if the following conditions hold for some fixed a > 0: -61-:
oo
X#(l-,I
a) <
>
(20.25)
=,
>1
77A'(1(Ix,Isc)-Y,)
<
(20.26)
x,
>1 X
XVar(1lIx,I
<
=.
(20.27)
n
,-1 5') is the same as the event (5'n+1 S'J convergence is Since the event (& invariant to shift transformations. lt is a remote event by 13.19 and hence in independent sequences occurs with probability either 0 or 1, according to 13.17. 20.8 gives the conditions under which the probability is 1, rather than 0. The theorem has the immediate corollary that Snlan -- 0, whenever an x. The basic idea of these proofs is to prove the convergence result for the truncated variables 1( lxrlsicj.X),and then use the equivalent sequences theorem to extend it to Xt itself. In view of 20.4, the condition .--)
--)
,
'1'
7
XFlx,IP
<
x,
(20.28)
1 < p < 2,
>1
is sufficient for convergence, although not necessary. Another point to notice about the proof is that the necessity part does not assign a value to a. Convergence implies that (20.25)420.27)hold for cv:ry a > 0. Proof of 20.8 Write Fr
lf Ix/lsktzl.Yf, so that the summands in (20.26)and (20.27) variances of Fr. The sequence ( Yt- F(Ff)1 is indethe means and are pendent and hence a martingale difference, so that Sn' Sm'= Z:=,,,+1(Ff EYt)4 is a martingale for tixed m k 0, and Z7=m+1Var(i'r)= Varts' Sm'4.Theorem 15.14 combined with 20.2, setting p 2 in each case and putting cf2 v ar (r)t and K 1, S' when (20.27)holds. lf (20.26)holds, together yield the result that Sn' this further implies that E7=1Fr converges. And then if (20.25) holds the =
respectively
-
-
-
=
=
-6...y
=
The fzzw of fxzr': Numbers -.F...y S, by 20.3. This proves sequences f#fl and f Ffl are equivalent, and so Sn suftkiency of the three conditions. Conversely, suppose Sn S. By 2.25 applied to xL()) for each (l) e f1, it lim--oxz/wruxf 0 a.s. This means that P(1 AI > a, i.o.) 0, for any follows that a > 0, and so (20.25)must follow by the divergence part of the Borel-cantelli lemma (18.2(ii)).20.3 then assures us that Z7.1Fr also converges a.s. Z')=I V ar( cB, ).!(Ff E(Yt)4Isn fails to x Write sn2 yt ) If sln as n converge, but is asymptotically distributed as a standard Gaussian r.v. (This is the central limit theorem see 23.6.) This fact contradicts the possibility of 1J=IF? converging, so we conclude that sn2 is bounded in the limit, which is equivalent to (20.27). Finally, consider the sequence ( I', EYt) l This has mean zero, the same variance as 1$, and #(I%- F(F,) I > 1a4 0 for a11 t. Hence, it satisfies the conditions (20.25)-420.27) (in respect of the constant 2a) and the sufficiency part of theorem implies l'/ converges, the that Z2=1(Fr F(Ff)) converges. And since Z2=1 (20.26) must hold. This completes the proof of necessity. w -'1-.y
=
=
.-+
=
--y
-
.
-
-
.
=
-
The sufficiency part of this result is subsumed under the weaker conditions of 20.10 below, and is now mainly of historical interest; it is the necessity proof that is interesting, since it has no counterpart in the LLNS for dependent sequences. In these cases we cannot use the divergence part of the Borel-cantelli lemma, and it appears difficult to rule out special cases in which convergence is achieved with arbitrary moment conditions. Incidentally, Kolmogorov originally proved the maximal inequality of 15.14, cited in the proof, for the independent case; but again, his result can now be subsumed under the case of martingale differences, and does not need to be quoted separately. Another reason why the independent case is of interest is because of the following very elegant result due to Lvy. This shows that, when we are dealing with partial sums of independent sequences, the concepts of weak and strong convergence coincide.
20.9 Theorem When fA-r)is an independent sequence and Sn = Z)=1Xt, Sn -P- S if S. and only if Sn -i1.-:
Proof Sufficiency is by 18.5. It is the necessity that is unique to the particular 1=m+jXt,and for some 6: > 0 consider the various ways in case cited. Let Smn which the event ( lSmnl > :) can occur. ln particular, consider the disjoint =
collection
max Is'xj
m V
; 2e,
1
For each k, this is the event that the sum from m onwards exceeds 2: absolutely for the hrst time at time k, and thus n
U
kmm-j
.
max #l<./K1-1
l
vmj
l<
2:,
Iskzl >
2:
=
max mjfn
1kj1 >
2:
,
(20.29)
The Strong fww of fatzr'e Numbers
313
where the sets of the union are disjoint. lt is also the case that
U
IsmjI s 2e, I5-,,,tl> 2:
max
k=pl+1 rrlKw/sk-l
ra
l Isknl K :1
i
l 1smnI> sl
,
(20.30)
where the inclusion is ensured by imposing the extra condition for each k. The events in this union are still disjoint, and by the assumption of an independent sequence they are the intersections of independent pairs of events. On applying (20.29), we can conclude from (20.30)that P
max m S./ S n
IsmjI >
rnin #(
2:
m<
I I< vkn
k fn
e) 2j#(
n
77# < k.=?n+1
IsmjI < Jz, I5-pol >
max rnf.jf-l
< #( IsmnI
IsknI <
:)
(20.31)
lf Sn .T-(-) S, there exists by definition m k 1 such that #( ISmnl
(20.32)
for all n > m. According to (20.32),the second factor on the rninorant side of (20.31) is at least as great as 1 :, so for 0 < : < 1, -
# max 1skyl> 2: < (20.33) 1 : mzjsn and then m x, the theorem now follows by 18.3. w .
-
Letting n
---
x
--
This equivalence of weak and strong results is one of the chief benefits stemming from the indepehdence assumption. Since the three-series theorem is equivalent to aweaklaw according to 20.9, we alsohave necessary conditions forconvergencein probability. As far as sufticiency results go, however, practically nothing is lost by passing from the independent to the martingale case, and since showing convergence is usually of disproportionately greater importance than showing nonconvergence, the absenc of necessary cpnditions may be regarded asmall price to pay. However, a feature of the three series theorem that is common to all the strong law results of this chapter is that it is not an array result. Being based on the convergence lemma, all these proofs depend on teaming a convergent stochastic sequence with an increasing constant sequence, such that their ratio goes to zero. Although the results cah be written down in array fonn, there is no counterpart of the weak 1aw of 19.7, more general than its specialization in 19.8.
20.3 Martingale Strong Laws Martingale limit results are remarkably powerful. So long as a sequence is a martnnd the nn ;tq aonenaencenre renllired lnaul,xalt-s-r/-nc.s- nn oarth-r roetrlr-tiranq
The fzzw of fzzr'd Numbers
moment assumptions called for are scarcely tougher than those imposed in the independent case. Moreover, while the m.d. property is stronger than the unconrlatedness assumed in j19.2, the distinction is vel'y largely technical. Given the nature of econometric time-series models, we are usually able to assert that a sequence is uncorrelated because it is a m.d., basically a sequence which is not forecastable in mean one step ahead. The case when it is uncorrelated with its own past values but not with some other function of lajged information could arise, but would be in the nature of a special case.
The results in this section and the next one are drawn or adapted chiefly f'rom Stout (1974)and Hall and Heyde (1980),althoughmany of theideasgo back toDoob (1953). We begin with a standard SLLN for f'z-bounded sequences.
o and 20.10 Theorem Let (.Yf,t.r)7' be a m.d. sequence with variance sequence (c1), 0 if (th) a positive constant sequence with at cxa. Snlan -.1%
'lh
X
77c2//2 t <
x.
t
(20.34)
u
>1
two ways to prove this result. The first is to use the martingale convergence theorem (15.7)directly, and the second is to combine the maxima1 inequality of 15.14 with the convergence lemma 20.2. ln effect, the second line of argument provides an alternative proof of martingale convergence for the square-integrable cse, providing an interesting comparison of techniques. There are
(at least)
First proof Detine Tn l=jxtlat,so that (Tn,5nIis a square-integrable martinusing the norm inequality and orthogonality of (xY)), gale. We can say, =
supFl TnI S supern) n
2 1/2
n
x
=
2
2
Xcr/l,
1/2
<
=,
(20.35)
>1
leading directly to the conclusion Tn F a.s., by 15.7. Now apply the Kronecker lemma to the sequences f L() ) for ) e f1, to show that Snlan -.Q1->0. w .-->
Second proof For m k 0, f Tn Tm, ?L) is a martingale with -
E (F
-
F
)2
=
mwjyltlalt.
(20.36) 2/ 2
Apply 15.14 fop 2, and then 20.2 with ct2 Gt at. Finally, apply the Kronecker before. w lemmaas =
=
(U
S B < oo, say, then setting at = Compare this result with 19.4. If VartX/l x 2 tzzjntlt 2 2 < #Z,=11// 1.64# < x, and the condition of the theorem t, we have Z = is satisfied, hence Xn Snln -F-.-h 0, the same conclusion as before. But the conditions on the variances are now a 1otweaker, and in effect we have converted the weak law of 19.1 into a strong law, at the small cost f substituting the m.d. assumption for orthogonality. As an example of the general formulation, suppose the sequence satisfies =
'
x
=
The Strong fzzw of fzzr.d Number X
X Elxl)l?' t
<
(20.37)
x.
>1
We cannot then rell upon knconverging to zero, but (puttingat tl) we can show that n -2E7=1X, Xnj n w jjj (jo so. The limitation of 20.10 is that it calls for square integrability. The next step is to use 20.4 to extend it to the class of cases that satisfy =
=
Co
XFI xtI#/Jf
<
(20.38)
x
>1
'1'
x. It is important to appreciate that (20.38)for for 1 < p K 2, and some (Jfl p < 2 is not a weaker condition than for p 2, and the latter does not imply the former. For contrast, consider p 1. The Kronecker lemma applied to (20.38) implies that =
=
Fl
an-1y')F1x t >1
1
(20.39)
0.
---/
For an rl, such a sequence has got to be zero or very close to it most of the time. In fact, there is a trivially direct proof of convergence. Applying the -
theorem (4.9),
monotone convergence tzn-l
E lim
Isn I
n--yx
N
< E lim c-ln
:7 I-Y,I >1
n-x
1im = n--yx
c-n1>7e'lx,l
.
>1
(20.40)
For any random variable X, E I.YI 0 if and only if X = 0 a.s.. Notlzing more is needed to show that Snlan converges, regardless of other conditions. Thus, having latitude in te value of p for which the theorem may hold is really a matter of being able to trade off the existence of absolute moments against the rateofdampingnecessrytomakethemsummable. Wemaymeetinterestingcasesin which (20.38)holds for p < 2 only rarely, but jince this extension is available at small extra cost in complexity, it makes sense to take advantage of it. =
20.11 Theorem If l.#),X)7is a m.d. sequence satisfying (20.38)for 1 f p ; 2,
Snlan
--1t.:
0.
Proof Let l'f 1( Ix,I>,)Xs and note that (aYfl and f Ff) are equivalent under by 20.4. Ff is also Tf-measurable, and hence the centred sequence (20.38), f4,;,), where Zt Ff A'(l'rl S-1), is a m.d. Now, =
=
-
Ezi)
F(e'(zlI@,-1)) F(rfl = F(F(I'lI@,-1)
=
-
1,-1)2)
The f-aw of fzzr'c Numbers
316
= s(r2,)
F(F(y,j y,-j)2).
-
(20.38)implies
According to 20.4 with r 2, Ezt) 2 < EY2), by (20.41), =
X Ezlyal t
<
,
(20.41) EYlllal /.1 t '=
that
t < x
,
and so, since
(20.42)
x.
>1
By 20.10, this is sufficient for variable. But n
n
l
'Xz,/z, 7-)
Ytlat
=
By
-.6.1..2/
Llzzjztlat
>1 15.13(i), (20.38)is
-
,..-1
>'-A'(i',j
<%t,
where 5'l is some random
5t-L)lat.
(20.43)
a.s.
(20.44)
,-z1
equivalent to
X
T,-l)/4' A7el.&lpl >1
< x,
< x, a.s. Absolute According to 20.5, (20.44)implies that Z7.11F(l'/ITr-l) may say that convergence of a series implies convergence by 2.24, so wet7n-l7=l -6YA st-Lllat -E1-.:/ '/=1F(I$l + Sz and so l'l Sz. Hence, E7=1 Ytlat 0 follows by 20.3 and equivalence of and Xt Ff the by the Kronecker lemma. lt 0. w implied by (20.38)that Snlan
tlat
-6-.3.-:
,5'1
-61.+
Notice that in this proof there are no short cuts through the martingale converis a martingale, the problem is to gence theorem. While we know that Llzubxtlat establish that it is unifonuly fal-bounded, given only information about the joint distribution of (A)), in the fonn of (20.38).We have to go by way of a result for p 2 to exploit orthogonality, wlch is where the truncation arguments come in =
handy.
20.4 Conditional Variances and Random Weighting A feature of martingale theory exploited in the last theorem is the possibility of relating convergence to the behaviour of the sequences of one-step-ahead conditional moments; we now extend this principle to the conditional variances E (Xt215 t-1 ) The elegant results of this section contain those such as 20.10 and .
20.11.
The conditional variance of a centred coordinate is the variance of the innovation, that is, of Xt Ext IA-1), and in some circumstances it may be more natural to place restrictions on the behaviour of the innovations than on the original setuence. ln regression models, for example, the innovations may coaespond to the regression disturbances. Moreover, the fact that the conditional moments are Tr-l-measurable random variables, so that any constraint upon them is probabilistic, pennits a generalization of the concept of convergence, following the results of j15.4; our confidence in the summability of the weighted conditional variances translates into a probability that the sequence converges, in the -
Fe
Strong fzzw of zzrge Numbers
317
manner of the following theorem. A nice refinement is that the constant weight can be replaced by a sequence of Tf-l-measurable random weights. sequence tt7fl zo-lzrfheoreml-et (Xf,Trlrbeam.d. sequence, (B$Janon-decreasingsequenceof 7=1Arf.Then positive, Tr-j-measurable r.v.s, and Sn =
#X
-' F(xll
@,-1)/+1 <
rntm,
x
0j
jsnlwn --
xj-
'1'
>1
=
0. u
(20.45)
The last statement is perhaps a little opaque, but roughly translated it says that 0), is not less than that the probability of convergence, of the event l5',,/Gn of the intersection of the two other events in (20.45).In particular, when one probability is 1, so is the other. --
Proof If (A) is a m.d. sequence so is (xY,/J7f), since W$is Ff-l-measurable, and 1=jXtIWtis a martingale. For ) e f, if L((l)) F()) and W%() 1 x then Tn 0 by Kronecker's lemma. Applying 15.11 completes the proof. . &()/lL() -->
=
---
See how this result contains 20.10, corresponding to the case of a fixed, divergent weight sequence and a.s. summability. As before, we now weaken the summability conditions from conditional variances to pth absolute moments for 1 f p < 2. However, to exploit 20.5 outside the almost sure case requires a moditkation to the equivalent sequences argument (20.3),as follows. of Tf-measurable r.v.s,
20.13 Theorem If (-YfJand ( Ff) are sequences XPCXt y. l'f 11,-1)
#
<
c,o
A
(A # Ff) e 5t, so that P(Xt y: F, l8h-1)
=
15.13(ii), ):
XF(1s,I877,-1)()
<
c,o
A
(t):
.1
,
=
F(1s,l 1,-1). According to
X 1s,()
<
x
=
0.
(20.47)
>1
>1
But E7 1s,((l))< t'in ite and hence
(20.46)
Cr
Cr
#
F,) converges
-
>1
>1
Proof Let Et
X Xt
cxl
Ff()) is that the numberof coordinates for which Xt x. F,())) implies < 1CX (20.47)therefore (20.46).w t () :#:
means ''o
t
-
=
Now we are able to prove the following extension of 20.11.
20.14 Theorem For 1 f p f 2, let F1 (X7=1F( lXt 1:lT/-1)/Ji$' ( J'IS XJ Under the conditions of 20.12, =
'1*
<
x
l and
Ez
=
.
#((F1
f'''h F2)
-
(&/W%
-->
0J)
=
0.
(20.48)
Proof The basic line of argument follows closely that of 20.11. As before, 1et 1$ that Zt J',- F(F,I1,-1) is a m.d. and 1( = Ix,Isu,)X,, so =
F(z2I@f-1) t
=
F(Fl 1@f-1) (F4r,1 #t-1))2 -
The fzzw of fzzr': E't 2,I
f
,-1),
Numbers
(20.49)
a-s.
Applying 20.5 and the last inequality,
#
E3
XF(z2,
-
>1
I;/.,)/J.y2t
lt follows by 15.11 and the fact that Ej P A'1
Xzr/B'r
-
< x
C
-
(Fl
5'1
--
0.
=
=
(20.50)
D)
-
QJ
(D
-
C) that
(20.51)
0,
>1
where & is some a.s. finite random variable. A second application of 20.5 gives X
P Fj
IF(F,I@r-1)l/B :-.2 >1
-
<
c.o
0,
=
(20.52)
which is equivalent (by 2.24) to
p sl
x- s(y,Iy,-,)/u,,
-
s,
--
-1
1 o, -
(20.53)
)
where S2 is another a.s. finite r.v. And a third application of 20.5 together with 20.13 gives X
P A'1
*
Xx,-X r,
-
>1
s?
-+
N1
=
0,
(20.54)
for some a.s. finite r.v. S?. Now (20.51),(20.53),(20.54),the detinition of Zt, the Kronecker lemma and some more set algebra yield, as required, *
0
=
P
=# =
e'1
-
X F,/u$ >1
(F1 r7 E1)
PEj
5'1+
-+
-
i',
.-+
>1 f'7eil
fn
E1
o
t,5k/en OJ). . -->
-
sz
(20.55)
20.5 Two Strong Laws for Mixingales The mmingale difference assumption is specialized, and the last results are not sufficient to support a general treatment of dependent processes, although they are the central prop. The key to extending them, as in the weak law case, is the mixingale concept. In this section we contrast two approaches to proving mixingale strong convergence. The tirst applies a straightforward generalization of the methods introduced by McLeish (1975a);see also Hansen (1991,1992a) for related
The Strong f-zzw of z7rg' Numbers
319
results. We have two versions of the theorem to choose from, a milder constraint on the dependence being available in return for the existence of second moments. be a fv-mixingale with respect to 20.15 Theorem Let the sequence (m,@f)=-x cfl either for constants ( with mixingale size 2, (i) p or (ii) 1 < p < 2, with mixingale size If X*t < cxo then Sn S. ,
-z1,
=
-1.
czlc'f
-6:4
Proof We have the maximal
inequality, l
E max 1 j
ISIJ'?<
KTcq', 1
(20.56)
/=
where K is a tinite constant. This is by 16.10 in the case of (i) and 16.11 in case (ii). By relabelling coordinates i't can be expressed in the fonn E max
lsj smIP
< K
-
1 j n
X cpt
(20.57)
1=rrl+1
for any choice of m and n. Moreover, P
max m
<j
I
uh
-
SmI
>
e
=
#
max m<jn
Isj smIJ' -
>
e
1 p < ,--E max ISj Sm1
(20.58)
-
m<jn
by the Markov inequality. Inequalities (20.57)and (20.58)combine to yield (20.2), and the convergence lemma 20.2 now yields the result. w We can add the usual corollary
from Kronecker's lemma.
satisfy either (i),or (ii) of 20.15 with respect 20.16 Corollary Let fxtlat,stloloo to constants fctlatt, for a positive sequence fat ) with at x. lf '1h
X
'lcnlant <
(20.59)
x,
f
>1
then Snlan
-1->
0. u
The second resuli exploits a novel and remarkably R. M. de Jong (1992).
powerful argument due to
20.17 Theorem Let (Xf,@,) be an fv-bounded, Al-mixingale with respect to constants f ctj for r 1, and let (Jfl, (#f) be positive constant sequences, and fMt J a positive integer sequence, with an x. lf ,1'
n
co '-zlanll
TM n=1
n exp
3lMlTBl n
1.=1
t
<
x,
(20.60)
The facw of f-zzrpd Numbers
320 X
:7 sl-rss x,Irlat
(20.61)
< x,
t
>1
X
< X Lutctlat ,-1
x where ((m),,,=o are the mixingale
(20.62)
x,
coefficients, then Snlan
-1'..-j
0. u
(#,J and
fA1)) are chosen freely to satisfy the conditions, given (tztJand fcrl, which suggests a considerable amount of flexibility in application. The 1, the role which was sequence (#,1 will be used to define a truncation of played by (tz,)in 20.11. The most interesting of the conditions is (.20.62),wlaich explicitly trades off the rate of decrease of the mixingale numbers with that of the sequence fctlatj. This approach is in contrast with the McLeish method of defining separate summability conditions for themomnts and mixingale numbers, as detailed in j16.3. Here
t#f
Proof Writing 1Bt for 1( 1x,lxs,), start by noting that E t+jXt = Etsj 1t,Xt + Et+j 1 11)Xt. -
Hence, we have the identity Xt
=
(Et+Mt-j
llXr
Et-M:$BtXt)+
-
(f+M,-1(1
li/lA
-
Et-Mtll
-
-
+ xt e',+,r,-1.;)+ Et-u7, -
and, by the usual
ttelescoping
1()Xf)
(t0.63)
sum' argument, Mt- 1
Bx E f+M,-l 1 t t
-
$Bx ;,-.A,z, t t
:- z,,j
=
(20.64)
./=l-Mt
where Zjt 14,I f lBt
=
is a m.d. sequence. Note that Et.t BX E ?+y-1 IW-Y t t t t, and tzys?h+yl a.s., by a double application of 10.13(ii). Summing yields
p X-Y,
-
n
=
zp X X l -Mt J=1
>1
Mt- 1
n
+
77F,+>,-1(1
n
+
=
ubn
+
ThEt-utj
-
>1
x>1
szn
-
1Bt%t
-1
jzz
-
-
1Of
)Xt
l
'+
F7(Xt
-
Et.vut-xtj
,...1
Et-uy <%n
+
s4n+ ssn.
(20.65)
The object will be to show that Sknlan -.6-.A 0 for k = 1,...,5. Stming with Sjn, the main task is to reorganize the double sum. lt can be verified by inspection that
The Strong fxzw of f-zzrge Numbers n
Mt- 1
Mn- 1
321
n
77 zp 77 Yzp 77 j=1 r=l -
j=-Mt
-Ma t-nqj
Mn- 1
Mn- 1 qj- 1
-M2
n
- X Xzy? 77 X Xzyf. '=1
N:1
j=3 -Mn
(20.66)
+
-
j=M2
-Mn
>1
1 for -M1 < j < M1, and qj t for -Mt < j f -Mt-j and Mt-j f j < Mt, / 2,...,,,. Note that, for arbitrary numbers x1,...,xk, l l I > EJ c UiI 1 ( lxi 1 > tlkk. Hence by subadditivity, and Azuma's inequality (15.20),
where qj
for
=
=
t-lxf
=
=
Mn 1 -
P
I.S3nI > an
f
n
X
XZy,
P
j=1 -Mn
anzlnMn
>
>1
Mn- 1
-M;
7) + X =Af2 '=1 -Mn
+
P
Xz,
>
anelnun
>1
,z
n
Blt '-'NX (32152
-rzlalnl
f zfaexp
>1
Mn
-Af2
:j
+
qj- 1
)j
+
'=1-Mn
f 4Mnexp
-:
1
-
exp
252/32A/2
-e
n
; #2 ,
>1
.j=Jf2
lan
n
qj- l
32M2 TBl n
t
(20.67)
.
>1 Under (20.60),these probabilities are summable over n and so Sjnlan first Borel-cantelli lemma. Now let ( l'r) be ay integrable sequence and detine
--1-4
0 by the
n
,7,1 -
X Ytlat.
(20.68)
,-1
By the Markov inequality, P max IS1 m<jxn
<%m' -
I>
1 a er S ,-E max Ifj SmI -
s !
<E -
n
7-)E II', Ilat.
rcal
XA'IYtllat >1
(20.69)
rrl<w/Kn
<
=,
(207U)
The fww of fzzr'd Numbers
322
0 by Kronecker' s then 5',l -6J-> f by an application of 20.2, and hence Sllan remaining result to each of the lemma. We apply tltis terms. For Szn. put l', --t!-
=
BJ),
an d note that
E r+M,-lX(1 1 -
n
?;
l
< A7#1,-rF1 x,$r/cr,
F7FIF,1lat < F7A'1 (1 114Xtl/J/ -
(20.71)
>1
>1
>1
B
r. S5n is dealt with in exusing the fact that IXf(1 1 t) IBt I f lXt (1 1)/#/l t s%n and %n,put successively Ff Xt Et-t-jxt and Ff actly the same way. For -
-
=
=
-
assumption,
Et-utxt, and by the mixingale
n
n
77FIF,1lat
f
77 ctlatjLut.
(20.72)
>1
>1
The proof is completed by noting that the majorant terms of are bounded in the limit by assumption. .
(20.71)and (20.72)
The conditions of 20.17 are rather difticult to apply and interpret. We will restrict them vel'y slightly, to derive a simple summability condition which can be compared directly with 20.15. with 20.18 Corollary Let (A,Tf ) be an Lr-bounded, fyl-mixingale of size respec! to constants (cf). If fcf) and ftz/l are positive, regularly varying x, and << c, and an sequences of constants with Ilmllr -(?()
'1-
n
X (c,/c,)1 <
(20.73)
,,o
Nl
where
'0 then Snlan
-6->
=
1) 2r(pc.#2(r-
(1+?.)(?c
(20.74)
1)'
+2(r-
0.
Proof Define ot (log -1- fo r > 0 This is slowly varying at infinity by 2.28, and the sequence fotltl is summable by 2.31. Apply the conditions of 20.17 with the added stipulation that fBtI and f#4l are regularly varying, increasing sequences, and so consider the conditions for summability of a series of the form Uzln) ), for q > 0. Since Lnonlnjconverges, summability follows Esitnlexpl 0. Taking logarithms, this is equivalent to f'rom (nlonllhlnlexpt U1(n4) =
.
-q
-->
-q
1ogn logtu) -
Since U(n)
nvn)
=
1og&1(n) q&a(n)
where Lnj
log n logton) -
+
+
-
(20.75)
.-.-)
-x.
is slowly varying, tls condition has the form
pllog n + 1og(fy1(a)) 1)?2Q(?7) -
(20.76)
-x,
--
where pl and pa are non-negative constants and LL(n) and Lzln) are slowly yarying. The tenns logt/n) and log(fz1(n)) can be neglected here. Put p2 0 and Lzn) Llon (logn) 1+8 an d the condition reduces to =
=
,
=
The Strong
(1 + pl
-
zzw of fzzr'd Numbers
ntlognlllogn
323
(20.77)
-x, ---
which holds for all pl for any q > 0 and 6 > 0. Condition (20.60)is therefore satisfied (recallingthat (#r) is monotone) if ,
2 2 nMnBnlan < on. 2
Similarly, conditions
(2.61)and (2.62)are B t1-rFl Xt Irja
B1
t ((
(20.78) satisfied if, respectively,
'-rclla
t
t
(20.79)
<( otlt,
and M-t%ctlat < otlt,
(20.80)
(p > (p0.
We can identify the bounding cases of Bt and Mt by replacing the second order-ofmagnitude inequality sign in (20.79),and that in (20.80),by equalities, leaving the required scaling constants implicit. Solving for Mt and Bt in this way, substituting into (20.78),and simplifying yields the condition
(c,lat /
?,/f,
'z
(20.81)
where (, (2r(p+ 2(r 1)1/(41+ rl(g + 2(r 1)j. This is sufficient for (20.60), (20.61), and (20.62)to hold. Since ct and at are specified to be regularly varying, there exist non-negative constants p3, p4, and slowly varying functions Lz and L4 such that ct tP3L5t) /P4fa is summable implies that and at 4 (/). The assumption that (lctlauj which in But (p > (?o implies ( > %, so that (p3 p4)( < (p3 p4)(o K . implies This completes the proof. 1). turn (20.8 =
-
-
=
=
-1.
-1,
-
-
Noting that 1 f Q f 2, the condition in (20.73)may be compared with (20.59).Put whereas with (pn and r 1, we get (() 2 and we obtain (e (pn 242r 1)/(3r 1) which does not exceed r in the relevant range, taling values between 1 when r 1 and 55when r 2. Square-summability of ctlat is sufficient x x. Thus, this theorem does not contain and r only in the limit as both (20 20.16. On jhe other hand, in the cases where (cfJis uniformly bounded and at f, we need only (o> 1, so that any r > 1 and (fb > 0 will serve. These dependence restrictions are on a par with those of the Lj convergence law of 19.10, and a striking improvement on 20.16. The case r 1 is not permitted for sample averftlog f) 1+ for 6 > 0. In other words, the ages, but is compatible with at theorem shows that
J
=
#,
=
=
-
=
=
-
=
=
-->
-->
=
=
=
(rltlogn) l4$-ly-qa:r
...,
t
() a.s.
(20.82)
>1
This amounts to saying that the sequence of sample means is almost surely slowly x; it could diverge, but no faster than a power of 1og n. varying as n --
20.6 Near-Epoch Dependent and Mixing Processes In
vinw
nf
fhe
lnqf
reKllltK.
there are two
Dossible
aooroaches
to the NED case. It
The frw of fxzrgd Numbers
324
turns out that neither approach dominates the other in tel'ms of pennissable conditions. We begin with the simplest of the arguments, the straightforward extension of 20.18. for z0-lgTheorem Letasequence lml=-xwith means lwl=-xbefv-NEDof size g,,llp,on a (possiblyvector-valued) sequence 1 < p S 2, with constants dt f< 11*-, which is FrlJx a-mixing (f-mixing)of size -J. lf ( -b,
-
*
11(.:)g,rl/hllti < 77 >1 for q
>
p in the a-rnixing case q
(20.83)
x
-
p in the
1) lqa lqb + lq +qlb jqxs + lq 1) ( (1 -
( t
Sxt n t= hen z-1E''
=
min
,
-
w)-fJ.-.
-
Nvhere
case) g-nzixing + +
1) gc
(20.84)
,
0.
Proof
J(1 1/)) with By 17.5, (Xt- jttl is a fal-mixingale of size respect to constants (c,), with ct
-
-
This penuits arbitrary mixing and NED sizes and arbitrary moment restrictions, so long as (20.83)holds with ( arbitrarily close to 1. By letting b one obtains result for mixing sequences, and by letting a a result for sequences that a underlying independent AP-NED Interestingly, in each of these on an process. are special cases ( ranges over the interval (1, lqlq + 1)) as the mixing/fa p -NED size is allowed to range from zero to By contrast, a result based on 20.15 would be needed if we could claim only (X, g,,l/thllpl for finite p; this rules out square-summability of the sequence ( 11 for choices of The and first of these rsults comes directly by b. (20.83) any a applying 17.5. -->
(x)
-->
(x)
-=.
-
20.20 Yheorem For real numbers b, p and r, let a squence f-vl'!x with means be JO-NED of size with constants dt << 111,wllp. f on a sequence (J$J=-x, x, For a positive constant sequence (th) let f Xt kttllatl be uniformly fv-bounded, and 1et g,,)D-.x,
-.b
-
'1'
-
X
11(-,w)/hII;< 77 >1 -
n Then an-1Xr=1( Xt ;1, b (i) p 1, 1 (ii) b (iii) b p (iv) b = 1, 1 =
=
=
,
-
=
< =
<
w)
--F--.->0
2, r > 2, p < 2, r 2, r k 2, p < 2, r
Proof By 17.5, conditions
(20.85)
=.
in each of the following cases'. 2)., (Ji) is (x-mixing of size > p, ( F,) is a-mixing of size 1),. f Ff J is (-tnixingof size 1). 2 p, f F,) is (-mixingof size
(i)-(iv)are
-r/(r
-
-#r/(r
-p);
-r/2(r
-
-r/(r-
all sufficient for
l Xt
-
P,,)/Jr,
?h1 to be
The Strong f-aw of fzzr'd Numbers
325
where 5t c(F,, s < f). The mixingale constants are of size II-Y, p.fllrl/c, 11*-, wIIr/c,.The theorem follows by 20.16. w
an fv-mixingale ctlat (t maxtlf,
=
-,
=
-
-
-z1,
As an example, 1etXt possess moments of all orders and be AZ-NED of size on x). of size close Summability of the a-mixing r,,,-.k terms to process (letting an Yaxtlat) is sufficient by (20.85).The same numbers yield ) 8,/on putting q 2 and a 1 in (20.84),which is not far from requiring summability of the Q-norms. However, this theorem requires fv-boundedness, which if r is small constrains the permitted mixing size, as well as offerig poor NED size characteristics for cases w'ith p < 2. lt can be improved upon in these situations by introducing a truncation argument. The third of our strong laws is the following. -1
=
=
=
20.21Theorem Let a sequence (A)=-x with mans lwlm-x be AP-NED of size with (( dt 1 2, < f for constants p g,II,,on a sequence ( F,)--- which is 111,-1/p
either
(i) a-mixing of size (ii) (-mixingof size
2) for r > 2 or 1) for r > 1, and r k p; and for q with p S q S r and a constant positive sequence -r/(r-
-r/2(r-
(th)
'1<
1et
x,
X
Fl 11(-4p.fl/tll'''t -
ntp,k/rl
<
(20.86)
x;
>1
-1
then an 17=1(m w)
-tl.u
g u
-
.
Note the different roles of the three constants specified in the conditions. p controls the size of the NED numbers, q is the minimum order of moment required to exist, and r controls the mixing of (Ff l The distribution of Xt does not otherwise depend on r. .
P roo f The strategy is to shbw that thete is a sequence equivalent f Xt ptllatl, and satisfying the conditions of 20.15(i). As in 20.6, let
to
-
F,
Xt
=
-
ttt3llat
:l:
(1
11),
-
(20.87)
if Xt > go otherwise. Note that denotes where 1t/ 1( Ix,I
<+'
i-'
=
-
-
-1
.
=
-
-;1
F,IIrl ( dtlapll, 11 rrllrK 2( 111,g.rll/crlWr q k p, and 11
ct << max
(20.88)
.
for any q < Here, dt < 2 IImwll for any r by the second inequality of (20.24).Condition (20.86)is therefore sufficient fr the sequence lctzJ to be summable, and ( Ft F(J)) satisfies the conditions of 20.15(i).We can conclude that Z')=1(l',EYt44 -E6-: S3, where l is some random -
-
-
-
variable.
According to 20.6, condition t/ari/xt N';
-
r'/I'
4 tla/xrefnrsx
(20.86)is
rxnnxr,xraee
sufticient for
tn n Gnlte
llmlt
Z'7.1IE)
hx, @ 94
enxr
I<
V=
x. .
r'/v
The 4
-
The zzw of fxzr'c Numbers
326 C1, and
n
X
-E->
I'J
.%
(20.89)
C1.
+
,-1
(20.22)and (20.6)further imply that F2 and Xt bktllatare equivalent sequences, and hence Inequalities
-
n
77((1, vlat >1 -
1$)
-
--6-:
(20.90)
sz,
where 51 is another random variable, by 20.3. We conclude that
X xt vlat ,-1 -
-EI-
5.1+
s, + ti'l s?,
(20.91)
=
J-1J n
say. lt follows by Kronecker ' s lemma that conclusion. w
=
-f.-..y
1(4
w)
-
0, the required
more specialized, result. The linear function of martingale differences with summable coefficients is a case of particular interest since it unifies our two approaches to the strong law. Here is a fill,
20.22Theorem sequence with
where ( U(k is a uniformly Ip-bounded m.d. Let Xt Lcej=-xjut-j 0;1 < n. Then p > 1, and lrv-xl j =
... .
1.19
lk.
((r!ji;;7''') drkr
.kigikr-r
................45))-
.
....... .11 1=1
.
s
b..i
'
-).
4:4()4 .
.. .
4..:-..)-
....
....
$tt),.'..
:..
.
t
.-y
.
.,.
....
.-).
.....-
,
u.. r
r.'t' #t-. ''.'(.
l'.
..
--?---...
.'
h.
.
k' ) '.)
. u'y.,.
.
%.
i..,..
y...y. ... ...
,. .
.
.,
with ct t< 1// and arbitrary size. It was shown Proof Ff Xtlt is a L ! in 16.12 that the maxlmal inequality (20.56)holds for this case. Application of the convergence lemma and Kronecker's lemma lead directly to the result. Alternatively, apply 20.18 to Xt with at /. w -mixingale
=
=
In these results, four features summarize the relevant characteristics of the stochastic process: the order of existing moments, the summability characteristics of themoments, andthe sizes of themixing andnear-epochdependence numbers.The way in which the currently available theorems trade off these features suggests that some unitkation should be possible. The Mct-eish-style argument is revealed by de Jong's approach to be excessively restrictive with respect to the dependence conditions it imposes, whereas the tough summability conditions the latter's theorem requires may also be an artefact of the method adopted. The repertoire of dependent strong laws is currently being extended (deJong, 1994) in work as yet too recent for incorporation in this book.
21 Uniform Stochastic Convergence
21.1 Stochastic Functions on a Parameter Space The setting for this chapter is the class of functions
2f : x 0wllere (f1,F,g.) is a measure space, and (O- is a metri space. We write to ai value which variable the real point random assumed is denote by f the a (,0), is not a random varifor fixed 0. But /(,.), alternatively written just able, but a random element of a space of functions. Ecnometric analysis is very frequently concerned with this type of object. Loglikelihoods, sums of squares, and other criterion functions for the estimation of econometric models, and also the first and second derivatives of these criterion functions, are a1l the subject of important convergence theorems on which proofs of consistency and the derivation of limiting distributions are based. Except in a restricted class of linear models, a11 of these are typically functions both of the model parameters and of random data. --+
,
.f4,0)
,p)
.f(),
To deal with convergence on a function space, it is necessary to have a criterion by which to judge when two functions are close to one another. In this chapter we examine the questions posed by stochastic convergence (almostsure or in probability) when the relevant space of functions is endowed with the uniform metric. A class of set functions that are therefore going to be central to our 2- where discussion have the fonn f*1fl --+
f*k
.f((0,0).
(21.1)
sup 6c e
=
For examjle, if g and h are two stochastic functions whose unifonn proximity is at issue, we would be interestd in the supremum of
.f(,0)
=
I#(,0)
-
(,0) I .
An important technical problem arises here which ought to be confronted at the outset. We have not so far given any results that would justify treating f* as a random variable, when (O- may be an arbitrary metric space. We can write ,p)
((,):fs
> xl
=
U f ): /'(0,)
> xl,
(21.2)
0e O
andthe results of 3.26 show that f: /*4) > xl e 5 when (: O is a subset . each0, when e is a countable set. But typically of the kind, and is uncountable. someihing .f(0,)>x)
e T for of (Rk or ,#s)
The ww of f-wr'd Numbers
328
This is one of a class of measurability problems having ramifications far beyond the uniform convergence issue, and to handle it properly requires a mathematical apparatus going beyond what is covered in Chapter 3. We shall not attempt to deal with this question in depth, and will offer no proofs in this instance. We will merely outline the main features of the theory required for its solution. The essential step is to recognize that the set on the left-hqnd side of (21.2) can be expressed as a projection. Let Ot.l denote the Borel tield of subsets of 0- that is, the smallest c-tield containing the sets of 0- that are open with respect to p. Then 1et (f1x 0, 9 @ B denote the product space endowed with the product c-tield (thec-tield generated from the measurable rectangles of 5 and fe), and suppose that f(.,.) is 5 (& t.lfp-measurable. Observe that, if ,
Ax
=
f(),0): f4,0)
>
x)
9 @ Be,
l
(21.3)
the projection of Ax into f is Ex
f
f (,0) = ((t)..f'u =
:
> >
x, 0 e x)
0-) (2 1.4)
.
In view of 3.24, measurability of f* is equivalnt to the condition that E'iI l 5 for rational x. Projections are not as a rule measurable transformations, but 5P L'j,5P,1b where is the under certain conditions it can be shown that Ex e completion of the probability space. The key notion is that of an analytic set. A standard reference on this topic is DellacherieandMeyertlg78); see alsoDudley (1989:ch.13), and Stinchcombeand White (1992).The latter authors provide the following definition. Letting (f1,T) be a measurable space, a set E c f is called F-analytic if there exists a compact metric space (O,p) such that E is the projection onto of a set A l 5 (&0e. The Also, a function J: f F-) F is collection of T-analytic sets is written for each called T-analytic if (: J()) f xl e e P Since every E e 5 is the projection of E x 0- e 5 @ e, 5 454. A measurable is not in general a c-field, set (or function) is therefore also analytic. although it can be shown to be closed under countable unions and countable intersections. Tiie conditions under which an image underprojection is known to be analytic are somewhat Weaker than the detinition might suggest, and it will actually suftice to let (O,e) be a Souslin space, that is, a space that is measurably isomorphic to an analytic subset of a compact metric space. A sufficient condition, whose proof can be extracted from the results in Stinchcombe and White (1992),is the following: ,
.4(54.
.45)
.x
.45)
,fo)
21.1 Theorem Let (f1,F) be a measurable space and (Ois in M5 (&Se), the projection of B onto n .45).
a Souslin space. If B e
define T&=() t TB where (f1,;B,F) Now , given themeasurable space (f1,@), is the completion of the probability space (f1,@,g,) (see 3.7) and Uthe intersection is taken over al1 p.m.s g defined on the space. The elements of ; are ca lled univer,
Uniform Stochastic Convergence
329
sets. The key conclusion, from Dellacherie and Mayer III.33(a)), is the following.
sally measurable
21.2 Theorem
(1978:
space (f1,T),
For a measurable
.454 c 5 & n
(21.5)
.
S ince by definition
5U
FB
for any choice of jt, it follows that the analytic sets for any choice of g. In other of 5 are measurable under the completion of (f1,@,jt) if analytic there exist is that E A, # such A c E ? # and g(A) jt(#). words, e 5 ln this sense we say that analytic sets are nearly' measurable. Al1 the standard probabilistic arguments, and in particular the values of integrals, will be and we can ignore it. We can unaffected by this technical non-measurability, legitimately treat J*((0)as a random variable, provided the conditions on 0- are observed and we can assume J(.,.) to be tnear-l5 f9 Oe/s-measurable. An analytic subset of a compact space need not be compact but must be totally bounded. It is convenient that we do not have to insist on compactness of the parameter space, since the latter is often required to be open, thanks to strict inequality constraints (thinkof variances, stable roots of polynomials and the like). In the convergence results below, we find that 0- will in any case have to be totally bounded for completely different reasons: to ensure equicontinuity; to ensure that the stochastic functions have bounded moments; and that when a stochastic criterion function is being optimized with respect to 0, the optimum is usually required to lie almost surely in the interior of a compact set. Hence, total boundedness is not an extra restriction in practice. The measurability condition on fl(.,) might be verifiable using an argument from simple functions. lt is certainly necessary by 4.19 that the cross-section functions A.,0): F-> P-and J(),.): 0- 2- be, respectively, T/plmeasurable for ,n) each 0 e 0- and s/s-measurable for each ) e f. For a finite pmition (O-1,...,0of 0- by e-sets, consider the functions c
=
-->
J(??,)(t0,0)J(,$), =
where
j
is a point of O-j. If E) Ax
=
l(,0):
.f(,,z)(tt,0)
=
0 e O-j,j
(): f,%.l
f xl
=
UFJTX
=
(21.6)
1,...,-, f a7l e 5 for each
h
then
O-je 5 (8)f(.),
(21.7)
j being a finite union of measurable rectangles. Since this is true for any fm) is 5 (8)fe/s-measurable. The question to be addressed in any particular case is whether a sequence of such partitions can be constructed such that hm) f as .;r,
--)
m
--
x.
Henceforth we shall assume without further comment that suprema of stohastic functions are random variables. The following result should be carefully noted, not least because of its deceptive similarity to the monotone convergence theprem, although this inequality goes the oppositeway.The monotoneconvergencetheorem concerns the expectation of the supremum of a class of functions (/',,4)1, whereas the present one is more precisely concerned with the envelope of a class of
The fzzw of frr'e
330 functions, the function of f.
J*()) which
assumes the value supeseltttl,ol
sup F(.f(0)) f E sup
21.3 Theorem
0e O
Numbers
0e O
J(0)
at each point
.
Proof Appealing to 3.28, it will suffice to prove this inequality for simple functions. A simple function depending on 0 has the' fonu m
((,0)
=
Xw(0)1s/(tl))w(0), ) =
(21.8)
e Ei.
f=1
Defining
G*j
=
supesowto),
sup ee e
(p4,0)
=
a*j, (t) e Ei.
(2 1.9)
Hence
supF@(0))
E sup (/0) ee O
-
e(-)
X (w(0) u14PEi)
sup
=
-
86O
< 0,
(2 1.10)
f=l
where the final inequality is by definition of a). w
21.2 Pointwise and Uniform Stochastic Convergence Consider the convergence (a.s.,in pr., in Lp, etc.) of the sequence (Q,,(0)Jto a limit function :(0), Typically this is a law-of-large-numbers-type problem, with
pn(0) X4n,(0) >1
(2 1.11)
=
(we use array notation for generality. but the case qnt qtln may usually be assumed), and :(0) 1imn-+xF(:s(0)). Alternatively, we may want to consider the 0 where case Gnj =
=
-->
G,,(0)
=
.Q
7
.(n,(0)
-
F(ts/0))).
(2 1. 12)
>1
By considering (21.12)we divide the problem into two parts, the stochastic convergence of the sum of the mean deviations to zero, and the nonstochastic onvergence assumed in the definition of Q(0).This raises the separate question of whether the latter convergence is uniform, which is a matter for the problem at hand and will not concern us here. As we have seen in previous chapters, obedience to a law of large numbers calls for both the boundedness and the dependence of the sequence to be controlled. In the case of a function on 0- the dependence question presents no extra difficulty; for example, if wr(01)is a mixing or near-epoch dependent array of a given for any 01, 02 l 0. But class, the property will generally be sharid by qnt' the existence of pmicular moments is clearly not independent of 8. If there ,
Unijrm
Stochastic Convergence
331
< exists a positive array (D,,f) such that Iqnte) I S Dnt for a1l 0 G 0- and IIDn,lIr uniformly in t and n, qnt) is said to be Lr-dominated. To ensure pointwise x, convergence on 0- we need to postulate the existence of a dominating array. There is no problem if the qnt) are bounded functions of 0. More generally it is necessary to bound 0- but since 0- will often have to be bounded for a different set of reasons, tlzis does not necessarily present an additional restriction. ,
,
,
on the dependence plus suitable domination conditions, pointwise stochastic convergence follows by considering (Gn(0)) as an ordinary stochastic sequence, for each 0 e 0- However, this line of argument does not juarantee that there is a minimum rate of convergence which applies for a1l 0, the condition of uniform convergence. If pointwise convergenceof (Gn(0)) to the limit G(0) is defined by Given restrictions
.
Gn(0)
Lp, or in pr.), each 0 e 0-
(a.s., in
0
..-0
(21.13)
,
a sequence of stochastic functions (G,,(0) ) is said to converge
lznt/rpz/y
Lp, or in pr.) on 0- if
sup lG,,(0) I 9e (-) To appreciate
0
.--y
(a.s.,in
L;,, or in pr.).
(a.s.,in (21.14)
the difference, consider the following example.
21.4 Example Let 0gnt)
ht =
define a zero-mean
g0,x),and
=
-
n
(%,(0)l
where
0 f 0 f 1/2?z
Z0, Z(3In
+
an'ay
-
0), 1/2n $In
0,
0 f $In
<
<
0
<
(21.15)
cxa
where (/ltJis a zero-mean stochastic sequence, and Z is a binary r.v. with #(Z 1 1) PZ Z:=1%(0) Hn + #n(0), where Hn ?1-1E' t 1h tn J. Then Ga(0) and =
-1)
=
=
=
=
=
Z(1 rl0), Slln
=
=
0 f 0 f 1/2s
Zn, Kn)
=
1/n
0
0 f Sln.
<
-
<
0
(21.16)
< x -61-.:
Hn for 0 > 1/n as well as for 0 0, Gn(0) We assume Hn -E1-> 0. Since Gn) 0 for each fixed 0 e 0. ln other words, Gn) converges pointwise to zero, a.s. However, supese IKn) I Izlzl = ,1 for every n 1. Because Hn converges a.s. there will exist N such that lHn I < :1 for all n k N, with probability 1. You can veri that when IHn I < :1the supremum on e of 1Hn + Kn) I is always attained at 1/2.n. Hence, Fith probability 1, the point 0 =
=
=
=
sup IGn(0) I es e
=
IHn +z1zIfor
n k N,
The fzzw of f-zzr'd Numbers
332
as n
--
It follows that the unifonn Similarly, for n k N,
=.
a.s. limit of Gn)
P sup IGn(0) I k e:
=
(4G O
(21. 17)
,.-3
is not zero.
#( IHn +rlzl
2 E).
--+#( 1z I 2 e)
=
(21.18)
1
,
so that the uniform probability limit is not zero either, although the pointwise probability limit must equal the pointwise a.s. limit. n on uniform a.s. convergence is a classic of the probability literature, the Glivenko-cantelli theorem. This is also of interest as being a case outside the class of f'unctions we shall subsequently consider. For a collection of identically distributed r.v.s (.)((l)),...,A7a((t))) on the probability space (D,F,#), the empirical distribution functionis defined as Our
tirst result
Fn(x,tn)
n
1771(-xx(,X()). n ,-1
--
=
(21.19)
ln other words, the random variable Fnlx,k is the relative frequency of the variables in the set not exceeding x. A natural question to pose is whether (and in what sense) Fn converges to F, the true marginal c.d.f. for the distribution. For fixed x, (&(x,(l)) )Tis a stochastic sequence, the sample mean of n Bernrandom variables which take the value 1 with probability F(x) oulli-distributed and 0 otherwise. lf these form a stationary ergodic sequence, for example, we know F@) a.s. for each x e R. We may say that the strong law of large that Fnlxbl numbers holds pointwise on R in such a case. Convergence is achieved at x for a1l ) e G, where #(G) 1. The problem is that to say that tbefunctions Fn converge a.s. requires that a.s. convergence is achieved at each of an uncountable set of points. We cannot appeal to 3.6(iii) to claim that pt;)xs IRCx) 1, and hence the F@) with probability 1 at a point not specsed assertion that Fnlx, beforehand cannot be proved in this manner. This is a problem for a.s. convergence additional to the possibility of convergence breaking down at certain points of the parameter space, illustrated by 21.4. However, uniform convergence is the condition that suffices to rule out either difficulty. ln this case, thanks to the special form of the c.d.f. which as we know is bounded, monotone, and right-continuous, uniform continuity can be proved by establishing a.s. convergence just at a countable collection of points of R. -->
=
=
.x
--
21.5 Glivenko-cantelli
theorem If Fnx,k
lFnlx,k sup A
-
F@) l
Proof First define, in parallel with Fn,
--
.-->
F(x) a.s. pointwise, for x e R, then 0 a.s.
I:a
(21.20)
Uniform Stochastic Convergence
333
n
F'(x,(,)) n
=
1771(--,x)(x,())),
-
(21.21)
f=1
and note that Fa'@,() integer m > 1 1et
F@-) for a1l
---2,
xjm inf (.xe R: F(x) k
)
in a set Cx', where #(Cx')
.j//n),
j
=
and also let x()r,,
and xmm
-x
=
Fxjm-)
-
1,
-
1. For an
(21.22)
so that, by construction,
+=,
=
1,...,-
=
=
Fxj-j,m) < Slm, j
=
(21.23)
1,...,-.
Lastly let M=((,))
1Fnxjm,k
maxmaxl
=
Sjnm
Then, for j F(x)
-
--
1
=
1,...,- and x
G
-
Fxjm)
I lFn'xjm.k
Ilj.
Fxjm-)
-
,
(21.24)
(xy-1,m,.xs),
Mmnk f Fxj-jvm) Mmnt -
S Fnxj-k,m.q S F%m-)
+
f Fn@.)
Mmnl
S Fn'xjm.)
f F(x)
j +
-+
Mmnlk.
(21.25)
F@) I f 3lm + Mmn That is to say, IFa@,) for every x e R By pointwise strong convergence we may say that limn-yxl/knt) 0 for finite m, (J) Cm* where hence that limn-yxsupx for al1 F(x) 1/-, Fnx, and l l; -
.
=
-
,
c*n1
=
Otcx mj
j=1
But #tlimm-+xCk')
=
fn
cx' ).
(21.26)
m
1 by 3.6(iii), and this completes the proof.
.
Another, quite separate problem calling for unifonn convergence is when a sample statistic is not merely a stochastic function of parameters, but is to be evaluated at a random point in the parameter space. Estimates of covariance matrices of estimators generally have this character, for example. One way such estimates are obtained is as the inverted negative Hessian matrix of the associated sample log-likelihood function, evaluated at estimated parameter values. The problem of provingconsistencyinvolvestwodistinctstochasticconvergencephenomena, andit does not suffice to appeal to an ordina/ 1aw of large numbers to establish convrgence to the true function evaluated at the true point. The following theorem gives suftkient conditions for the double convergence to hold.
21.6 Theorem Let (L,5,P4 be a probability space and (O- a metric space, and let Qn10- x F-+ R be T/f-measurable for each 8 e 0. If (a) 0n*-E'2 %, and (b) :s(0) -.EI.+:(0) unifonnly on an open set Bj containing %, where :(0) is a nonstochastic function continuous at %, ,p)
The zzw of
334
Qn(;. n)
then
..s.o
Numbers
-wrgc
rj()(j;.
Proof Uniform convergence in probability of Qnon Bo implies that, for any E > 0 and 8 > 0, there exists Nj k 1 large enough that, for n k Nj ,
P
Sup
0
E5
-
Bz
-,1.
r1E
I:,,(0) Q(0)I <
(2 1.27)
2 1
Aldo, since 0n*-T-r-> 0o, there exists N2 such that, for zl 2 N1,
#(0
e #0) 2 1 14.
(21.28)
-
To consider the joint occurrence of these two events, use the elementary relation P(A
#) k PA) +PB4
/'7
1.
-
(21.29)
Since
(0n*e for n
#ol
f''h
sup I:a(0) t)E5BQ
:(0()1 <
-
Y
l l:,,(0) Q(0n*)l< Y) (21.30)
c
-
,
k max(N1,N2),
#1 lQn(0n*)Q(0,1) l<
/1
-
Using continuity at
/$r3,
% and
2(1
t)
-
-/.
1
-
(21.31)
1
=
18.10(ii), there exists Nz large enough that, for n k
#( I:(0n*) Q(0())I < /) -
2 1
(21.32)
.
By the triangle inequality,
IQn(M,,*):(%*)l + l:(0,,*) Q(0c)l -
-
l:,/0,1) :()0()281 -
(21.33)
and hence
l I:,;(0n*):(0n*)l <
el
-
fa
( l:(%*) :(00) l < El < c ( I:a(0n*)p(0t))l : l -
-
Applying
(21.29)again
.
(2 l
.34)
gives, for n k max(NI,N2,Na),
#(I :n(0n*):(041 -
< E)
1
-
.
The theorem follows since 8 and e: are arbitrary.
(21.35)
.
Notice why we need unifonn convergence here. Pointwise convergence wold not allow us to assert (21.27)for a single 11 which works for a1I 0 s B). There would be the lisk of a sequence of points existing in Bo on which Nj is diverging, Suppose % = 0 and Gn) = :s(0) Q(0)in 21.4. A sequence approaching %, say ( 1/m, m e IN), has this property; we should have -
#( IQnlm) for arbitrary
6: >
0 and
45
>
-
:(1/,/7)j
<
/)
k 1 1:4 -
(21.36)
0, only for n > m. Therefore we would not be able to
Unrm
Stochastic Convergence
335
claim the existence of a finite n for which (21.31) holds, and the proof collapses. In this example, the sequence of functions (G,,(0) ) is continuous'for each n, but the continuity breaks down in the limit. This points to a link between uniform convergence alzd continuity. We had no need of continuity to prove the GlivenkoCantelli theorem, but the c.d.f. is rather a special type of function, with its behaviour at discontinuities (andelsewhere) subject to tight limitations. In the wider class of functions, not necessarily bounded and monotone, continuity is the condition that has generally been exploited to get uniform convergence results.
21.3 Stochastic Equicontinuity Example 21.4 is characterized by the breakdown of continuity in the limit of the sequence of continuous functions. We may conjecture that to impose continuity unifonnly over the sequence would suffice to eliminate failures of uniform convergence. A natural comparison to draw is with the uniform integrability property of sequepces, but we have to be careful with our terminology because, of course, uniform continuity is a well-established tenu for sometlling completely different. The concept we require is equicontinuity, or, to be more precise, asymptotic Ikplt/b/-/?/ equicontinuity', see (5.47).Our results will be based on the following of the Arzel-Ascoli theorem (5.28). version functions on a 21.7 Theorem Let (A(0),n q (N) be sequence of (nonstochastic) 0 if and only if (O,p). Then, bounded totally parameter space 1 supeee l 0 for a11 0 e 0-e, where 0-t) is a dense subset of 0, and (/ l is fno) asymptotically unifonnly equicontinuous. u .fn(0)
-->
--->
The set F t/'n,n e IN) QJ (01,endowed with the uniform metric, is a subspace of (Co,Jg), and by definition, convergence of fn to 0 in the unifonn metlic is the same thing as unifonn convergence on 0- According to 5.12, compactness of F is equivalent to the property that every sequence in F has a cluster point. ln view of the pointwise convergence, the cluster point must be unique and equal to 0, so that the conclusion of this theorem is really identical with the Arzel-Ascoli theorem, although the method of proof will be adapted to the present case. Where convenient, we shall use the notation =
.
w(/k,)
=
sup 0 (5 O
0/G
sup
510,8)
IJn(0') fn) 1
(21.37)
-
.
The function w(A,.):R+ F-+ R+ is called the modulus ofcontinuity of fn. Asympof the sequence (Jn) is the property that totic unifonn equicntinuity limsupnwt/k,8) 1 0 as 0. .1'
Proof of 21.7 To prove
iif'
:
satisfy
given
e: > 0, there exists by assumption
limsup w(/,) n--x
<
e.
:
>
0 to
(21.38)
The fww of fzzr'c Numbers
336
since 0- is totally bounded, it has a cover f5(f,&2), i 1,...,mJ.For each i, choosep e 0-n such that p(0sp) < /2 (possiblebecause 0-() is dense in 0-) and i 1,...,mJis also a cover for 0- Every 0 0--is contained note that (5'(0f,8), =
ad
=
.
N
in 5'(0/,) for some i, and for this i,
lJn(0)1f
1Jn(0')l
sup
e's
S
-s('4'j,)
Ifne'q .f(pf)I+ IJ(1) I
sup
-
.
.s'(;J,)
e'<
(2 1.39)
We can therefore write
supIfnet I <
()ce
J(pf)I+
.&(e')
max 0'e
liir?z
s w(.&-)
+
sup
-
.('j,)
max l<
max
I.f(4f)I
IJ(pf)I (21.40)
.
l Ki < m
Sufficiency follows on taking the limsup of both sides of this inequality. donly ito follows simply from the facts that unifonn convergence entails pointwise convergence, and that
w(.&,)S
2 sup 0e e
1.f(0)1.
(21.41)
.
To apply this result to the stochastic convergence problem. we must define concepts of stochastic equicontinuity. Several such definitions can be devised, of which we shall give only two: respectively, a weak convergence (in pr.) and a strong convergence (a.s.) variant. Let (O,p) be a metric space and L'j,5,P4 a probability space, and 1et (Gu(0,)), n e IN) be a sequence of stochastic functions Gn O x f F-> R, F/f-measurable for each 0 e 0. The sequence is said to be asympstochastically equicontinuous (in pr.) if for a1l e > 0 H 8 > totically I/nt/brl/y 0 such that
limsup PwGnbn)
2 :)
<
s.
(21.42)
n-x
And it is said to be strongly asymptotically uous if for all 6: > 0 R > 0 such that P limsup w(tL,) 2 :
rfnt/br/nfy stochastically equicontin-
=
0.
(21.43)
n-@x
Clearly, there is a bit of a terminology problem here! The qualifiers asymptotic' and unifonn' will be adopted in al1 the applications in this chapter, so 1et these be understood, and let us speak simply of stochastic equicontinuity and strong stochastic equicontinuity. The abbreviations s.e. and s.s.e. will sometimes
be used.
Unrm
Stochastic Convergence
337
21.4 Generic Uniform Convergence Uniform convergence results and their application in econometrics have been researched by several authors including Hoadley (1971),Bierens (1989),Andrews (1987a, 1992),Newey (lggll,andptscherandpruchatlg8g,lgg4l.Thematerialin the remainder of this chapter is drawn mainly from the work of Andrews and Ptscher and Prucha, who have pioneered alternative approaches to deriving tgeneric' uniform convergence theorems, applicable in a variety of modelling
situations.
These methods rely on establishing a stochastic equicontinuity condition. Thus, once we have 21.7, the proof of unifonn almost sure convergence is direct.
21.8 Theorem Let (Gn(0), n e EN ) be a sequence of stochastic real-valued functions metric bounded totally space (O,p). Then on a -f.-
suplG,,(0)I
0
(21.44)
()EEO
if and only if 0 for each 0 q 0-0, where 0-t) is a dense subset of 0(a) Gn) (b) (Gn) is strongly stochastically equicontinuous. --f.1..:
,
Proof Because (O- is totally bounded it is separable (5.7)and (% can be chosen to be a countable set, say 0-() (01,k G EN ) Condition (a) means that for k with is 0 for (.t) e Ck. 1 such that Gnk, Pc 1,2,... there a set Ck Condition (b)means that the sequences ( Gn J are asymptotically equicontinuous 1. By the suftkiency part of 21.7, supaso IGn(0,tt)) I for all ) e C, with PC') 0 for (l) e C* O';=lC fa C. PC*) 1 by 3.6(iii), proving if' .-
=
=
.
-->
=
=
=
c*
=
.
.
The corresponding probability' result follows very similar lines. The proof 21.7 quite so directly, but the family resemblance in the arguments cannot exploit will be noted. tin
21.9 Theorem Let tGn(0), n e ENl be asequence of stochastic real-valued functions on a totally bounded metric space (O,p). Then
IGn(e) I sup ()s O
.-J'-'.'->
0
(21.45)
if and only if (a) Gn) .-C1.+ 0 for each 0 l 0-(), where 0-() is a dense subset of 0(b) fGnl is stochastically equicontinuous. Proof To show let (5'(4f,8), 1,...,mJwith 4f G 0-() be a finite cover for ().This exists by the assumption of total boundedness and the argument used in the prpof of 21.7. Then, ,
'if'
,
=
338
P
Law of fzlrge Numbers
Fe
tsus polGnt
l
2
2E)
sup ( lG,t@') Gni) ezs.('4f,)
; P max
I + !Gn(0f) I) k
-
Lsism
< #(w(Gn,) < P(wGn,&)
E) +
P max lGnlui) I
e
1S K m
k
< #(w(Gn,)
e) + P
2:
U(iG,/VI
e.)
f=1
:) +
tA(4f)l X#41 izz,1
:),
(21.46)
where we used the fact that fx-y k 2EJ c (x k e)
l..p
(y 2
:)
(21.47)
for real numbers x and y, to get the third inequality. Taking the limsup of both sides of (21.46),(a) and (b) imply that limsup # sup IGn(0) j k 2: n-kcn
(21.48)
0e O
if', pointwise convergence follows immediately from uniform convergence, so it remains to show that s.e. holds; but this follows easily in view of the fact (see (21.41))that To prove
tonly
#(w(Gn,)
k :) S P sup 8e O
lGnl) 1 2
:/2
(21.49)
w
.
There is no loss of generality in considering the case Gn 0 in these theorems. We canjust as easily apply them to the case where Gn(0) = @n(0) :n(0) and Qnis a nonstoehastie function which may really depend on n, or just be a limit funcIn the fonner case there is no need for Qnto converge, as tion so that '-Q n long as Qn Qndoes. Applying the triangle inequality and taking complements in (21.47), we obtain -->
-
-:
=
5
-
-
lwton, ) < eJ fa (w(-p ) < 2e). (21.50) n, n, ) < EJ c fwtpu This means that (Qn -Qn ) is s.e., or s.s.e. as the case may be, provided that and f-:n) is asymptotically equicontinuous in the ordinary (:?:) is s.e., or s.s.e., senseof j5.5. This extesion of 21.8 is obvious, and in 21.9 we can insert the step 2 2E) S #(w(:,,,8) 2 E) + 1(w(7n,)E) (21.51) #(w(Qn Ua,8)
--
-
-
into (21.46),where the second tenu on the right is 0 or 1 depending on whether the indicated nonstochastic condition holds, and this tenn will vanish when n 2 N
Stochastic Convergence
Unbrm
for some N k 1, by assumption.
The s.e. and s.s.e. conditions may not be particularly easy to verify directly, and the existence of Lipschitz-type sufficient conditions could then be very convenient. Andrews (1992)suggests conditions of the following sort.
21.10 Theorem
Suppose there exists N 2 1 such that
IQn(6') Qn(0)l K #n(p(0,0')), -
(21.52)
a.s.
holds for all 0,0' e 0- and n k N, where h is nonstochastic and hx) 1 0 as x 1 0, and (Bn l is a stochastic sequence not depending on 0. Then (i) (Qn) is s.e. if Bn 0r,(1). (ii) (:,,) is s.s.e. if limsupsla < x, a.s. =
Proof The definitions imply that w(Qn,6) < Bnhln) a.s. for n 2 N. To prove > > that, for 0 and 0, 8 note any s
limsup #(w(:n,6)
k
:)
< limsup PBn k
n--)x
E//l(8)).
(i),
(21.53)
n-ex
By definition of Op(1), the right-hand side can be made arbitrarily small by choosing Elh&) large enough. ln particular, fix e > 0, and then by definition of < :. h we may take 8 small enough that limsupn--pxf'tf,l k :/48)) For (ii), we have in the same way that, for small enough 8, P limsup
w(Qn,8)2
is
f P limsup Bn k
:/(6)
< E. .
(21.54)
N'-/*
n--oo
A sufticient condition for Bn Op(1) is to have Bn unifonnly bounded in f,1 norm, i.e., supnElBn) < x (see 12.11), and it is sufticient for limsupnfn to be a.s. 0. bounded if, in addition to this, Bn E(Bn) The conditions of 21.10 offer a striking contrast in restrictiveness. Think of (21.52) as a continuity condition, which says that Qn(0')must be close to Qn(0) when 0' is close to 0. When Qnis stochastic these conditions are vel'y hard to satisfy fojxed Bn, because random changes of scale may lead the condition to be is a continuous function for a1l (l) and violated from time to time even if Q,;(6,(t)) n. The purpose of the factor Bn is to allow for such random scale variations. Under s.e., we require tat the probability of large variations declines as their magnitude increases', this is what Op(l) means. But in the s.s.e. case, the requirement that (#n) be bounded a.s. except for at most a finite number of terms implies that fQn) must satisfy the same condition. This is very restrictive. lt means for example that Q,,(0)cannot be Gaussian, nor have any other distribution with infinite support. ln such a case, no matter what (#,,) and h were chosen, the conditiop in (21.52)would be violated eventually. It does not matter that the probability of large deviations might be extremely small, because over an infinite number of sequence coordinates they will still arise with probability 1. Thus, strong uniform convergence is a phenomenon confined, as far as we are able to show, to a.s. bounded sequences. Although (21.52)is only a sufficient condition, it can be verified that this feature of s.s.e. is implicit in the =
.-61-.:
-
The frw of fzlr'e
340
Numbers
definition. This fact puts the relative merits of working with strong and weak laws of large numbers in a new light. The former are simply not available in many important cases. Fortunately, in probability' results are often sufficient for the purpose at hand, for example, detennining the limits in distribution of estimators and sample statistics', see j25. 1 for more details. Supposing (O- c ('R ,dE), suppose further that Qv(M)is differentiable a.s. at each point of 0-; to be precise, we must specify differentiability a.s. at each point of an open convex set 0-*conta i n in g 0.. (A set B c R is said to be convex if The mean value theorem x e B and y e B imply kr + (1 h)y e B for e (0,1).) * 22 yields the result that, at a pair of points 0,0 z e o,,p)
.
-
,
Qn (O)
-
Qn(O')
k Qn
77 y@y
=
f=1
(0 0r,) a.s., -
jsj;v
(21.55)
where e* e 0- is a point on the line segment joining 8 and O', which exists by *. convexity of 0- Applying the Cauchy-schwartz inequality, we get *
l Qa(0) -
k
j ljjo: jaQn ef
l K Y-% j
Qn(O')
.e.
.
.
i=1
< fnlle
.,r
j ().(j
-
e'll
.
a.s.,
j (2 1
.56)
where Bn
()
*
G
O
j' jj
jj;E) jk;pr ?:
sup
=
*
PO
.
e=e.
(2 1.57)
41jldenotes the Euclidean
length, and onlB is the gradient vector whose elements are the partials of Qn with respect to the 0j. Clearly, (21.52) is satisfied by taking h as the identity function, and Bn defined in (21.57) is a random variable for all rl. Subject to this condition, and Bn satisfying the conditions specified in 21.10, a.s. differentiability emerges as a sufficient condition for s.e.. Here
.
21.5 Uniform Laws of Large Numbers In the last section it was shown that stochastic equicontinuity (strongor in pr.) is a necessary and sufficient condition to go from pointwise to uniform convergence (strongor in pr.). The next task is to find sufficient conditions for stochastic equicontinuity when (:n(0)l is a sequence of partial sums, and hence to derive uniform laws of large numbers. There are several possible approaches to this problem, of which perhaps the simplest is to establish the Lipschitz condition of 21.10.
21.11 Theorem Let ( (tsX(.t),0)l:=1 )''J.1denote a triangular array of real stochastic satisfying, for N k 1, functions with domain (O,p),
Unform Stochastic Convergence
1qntl
?n,(0)
-
lS
#n/(p(0,0')),
341
(21.58)
a.s.,
for all 0,0' e O and n k N, where h is nonstochastic and hx) (#n,l is a stochastic array not depending on 0 with 1=kEBnt) ZVlnf(0)'then (i) Qnis s.e.; 0. (ii) Qnis s.s.e. if lzzkBnt ElBntjj
.1,
0 as 1 0, and /(1). If :n(0) .x
=
=
-61--;/
-
is only necessary by 21.10(i) and the triangle inequality to 0,(1). This follows from the stated condition by the establish that XlctBnt Markov inequality. Likewise, (ii) follows directly from 21.10(ii). w
Proof For
(i) it
=
A second class of conditions is obtained by applying a form of s.e. to the summands. For these results we need to specify Gn to be an unweighted average of n functions, since the conditions to be imposed take the form of Cesro summability of certain related sequences. It is convenient to confine attention to the case 1
Gn(,0)
=
-
n
F7(tp(X,(),0)
-
F(/X,,0))),
(21.59)
>1
where Xt e X is a randpm element drawn from the probability space (X,T,gf). Typically, though not necessarily, Xt is a vector of real r.v.s with X a subset of R''', m k 1, t' being the restriction of Bm to X. The point here is not to restrict but to specify the existence the fonn of the functional relation between qt and 6 P(Xt A) for A e T. The usual of marginal derived measures g,f,with gXA) context will have Gn the sample average of functions that are stochastic through their dependence on some kind of data set, indexed on t. The functions themselves, not just their arguments, can be different for different t. We must find conditions on both the functions qt(.,.) and the p.m.s j which yield the s.e. condition on Gn. The first stage of the argument is to establish conditions on the stochastic functions tp(0) which have to be satisfied for s.e. to hold. Andrews (1992)gives the following result. ,
=
21.12 Theorem lf (a) there exists a positive stochastic
supIqtt I S
sequence
fdt)
satisfying
(21.60)
dt, all t
: s (.) and
limsup n--yx
(b) for every
: >
n 1 TEdt n >1
-
1(>v))
0, there exists
-->
>
0 as M
-->
n
n-yx
>1
(2 1.6 1)
0 such that
> :) limsupn1 77#(w(tp,6) -
x;
<
e',
(21.62)
The fxzw of fxzrgc Numbers
342 then Gn is s.e.
l:n
Condition (21.61)is an interesting Cesro-sum variation on unifonn integrability, and actual uniform integrability of (4) is sufticient, although not necessary. Condition (a) is a domination condition, while condition (b)is called by Andrews
termwise stochastic
equicontinuity.
Proof Given : > 0, choose Msuch that limsupn-oxn -1' then 6 such that
limsup 1 --
n
n-+x
?=I F(2J t 1(u ,
>>))
< :1:2, and
-n
> 77#(w(t.p,) >1
E2) <
jM-1:2.
(21.63)
The first thing to note is that
w((t.n
'(f.n)),)
-
w(F(tp),6)
S w(,,)
+
S w(,,)
+F(w(,,)),
(2 1.64)
where the last inequality is an application of 21.3. Applying (21.64)and using
Markov's
inequality,
#(w(tA,)
>
e) S #
1 .-Q
.-wttp
-
Eqt),nj
> 6r
>1
s
,(!,y) &
2
>1
(w(o,)+s(w(.)))
>
j
n
s -Xs(w(,,)) nc >1 n
o =ns77e-(w(.p,)(1fw(t,,,)xc2s) >1 -
+ 1fs2/6<w(s8)&/')).1,
(21.65)
where the indicator functions in the last member add up to 1. Using the fact that w(t.?s8)S ldt, and hence fwttsl > AfI (24 > AJ'),and taking the limsup, we now obtain
limsup#(w(Ga,) n-+x
> E)
<
2 e2 +
-
-j-
Mlimsup
-
1
n
X# w(t?f,8)
>
p
-j-
>1
n-x
n
+
1 limsup -XF(2J,1(>,>>j) n-yco
n
/=1
(21.66) in viesv of the values chosen for 3T and
.
.
Unrm
Stochastic Convergence
21.12(a) is satisfied depends on both the distribution of Xt and functional form of qt(.4. But something relatively general 21.12(b)). Assume, following Ptscher can be said about tenuwise s.e. (condition and Prucha (1989),that Clearly,
whether
condition
p .?,(x-e)>7rk,(x),1,@,Q), =1
(21.67)
=
R, and X R for fixed 0, are T/f-measurable functions. The idea here is that we can be more liberal in the behaviour allowed to the factors rkt as functions of Xt than to the factors skt; discontinuities are permitted, for example. To be exact, we shall be content to have the rkt uniformly fvl-bounded in Cesro mean:
where rktk X
.k,4.,0):
-->
--
1
sup n -
n
n
FIrktxt) I s 77 >1
B
<
k
cs
1,...,p.
=
(21.68)
As to the factors Jkr@,0), we need these to be asymptotically equicontinuous for a sufficiently large set of x values. Assume there is a sequence of sets (Km 6 T, m 1,2,...J,such that =
n
limsup1X p,rtA1) n >1 n-+x --
and that for each m 2 1 and s
>
--+
n-koo
k
< :,
(21.69)
--.>
x,
0, there exists 8
limsup sup w(sjz(x,.),8) A G
0 as m
=
>
0 such that
1,...,p.
(2 1.70)
Km
(21.70)is a nnstochastic equicontinuity condition, but under condition (21.69)it holds (as one might say) almost surely, on average' when the r.v.
Notice that
Xt is substituted into the formula. These conditions suffice to give tennwise s.e., and hence can be used to prove s.e. of Gn by application of 21.12.
21.13 Theorem If qt(Xt,Q)is defined by (21.67),and (21.68),(21.69),and (21.70) hold, then for every : > 0 there exists 6 > 0 such that n
1 limsup XP(w(t.p,8) -
>1
n-yx
Proof Fix :
>
> E) <
e.
0, and first note that
#(wtr-l>
:) S
P
(F-Iq,l
wtsal
>
&1
,(
s
&1
fIr,-Iw(,,,,)>
j s/,I)
(21.71)
The fzzw of fxzr'c Numbers p
rkrl w(.k,,) < 77#(1 &1
>
zlp4
g,(Ir,,1w(,.,,)1,u x'ss s >
&1
J
+P
(1
)
qll w(.%,,)1rj,
>
lp
.
(21.72)
Consider any one of these p tenns. Choose m large enough that n
: 1 < limsup-Xw(#kc) l-p n-x
n
(21.73)
,
>1
and for this m choose 8 small enough that
limsupsup w(x@,.),) nKm x xs
<
2 4Bp 2
(21.74)
.
Then, by the Markov inequality,
n limsup1 X P lrkt 1w(.q,,)1Au n->x n ,= 1 -
>
K limsup s-.I'Z--+ n
s f
n
j
:
'*
1 -
qJl
n
EFI
q,l
>1
n--yx
4Bp z
,=
x
limsup
vz
1 X# 1
--
>
:
rg
:
zpp (21.75)
-,
2#
and by (21.73), limsup n-/x
-
1
n
X# >1
Irpl wt-qsllrj,
>
:
yp
f limsup
1
n
XPCXt
n-+x
<-;*
Substituting these bounds into
-a
(21.76)
#
(21.72)yields the
Km4
>1
result. w
V THE CENTRAL LIMIT THEOREM
22 Weak Convergence of Distributions
22. 1 Basic Concepts The objects we examine in this pal't of the book are not sequences of random variables, but sequences of marginal distribution functions. There will of course be associated sequences of nv.s generated from these distributions, but the concept of convergence arising here is quite distinct. Formally, if fFn JTis a sequence of c.d.f.s, we say that the sequence converges weakly to a limit F if Fn@) .-.A F@) pointwise for each x e C, where C ? R is the set of points at which F is continuous. Then, if Xn has c.d.f. Fn and X has c.d.f. F, we say that Xn converges in distribution to X. These terms are in practice used more or less interchangeably for the distributions and associated l'.v.s. Equivalent notations for weak convergence are Fn = F, and Xn .-C->X. Although the latter notation is customary, it is also slightly irregular, since to say a sequence of nv.s converges in distribution means only that the limiting r.v. has the given distribution. If both X and F have the distribution specitied by F, then Xn -P-> X and Xn -P-> F are equivalent statements. Moreover, we write things like Xn -P- N(0,1) to indicate that the limiting distribution is standard Gaussian, although #(0,1)' is shorthand for r.v. having the standard Gaussian distribution' ; it does not denote a particular r.v.. Also used by some authors is the standing for convergence in probability law', but we avoid this notation form because of possible confusion with convergence in fp-norm. Pointwise convergence of the distribution functions is all that is needed, rememberig that F is non-decreasing, bounded by 0 and 1, and that every point is either a cpntinuity point or a jump point. lt is possible that F could possess > jump at a point zb which is a continuity point of Fn for all finite n, and in these cases Fnxo) dpes not have a unique limit since any point between F(.zb-) and F(ab) is a candidate. But the jump points of F are at most countable in number, and according to 8.4 the true F can be constructed by assigning the value Fxo) at every jump point xo; hence, the above definition is adequate. If g. represents the corresponding probability measure such that F@) g,((-x, x1) for each e R, we know (seej8.2) that g. and F am equivalent representations of the same measure, and similarly for g. and Fn. Hence, the statement gn cuu) g,is equivalent to Fn F. The corresponding notion of weak convergence for the sequence of measures (jtnl is given by the following theorem. Ea
i-1-+'
=
.x
::uuh
22.1 Theorem g.
=
p, iff gn(A)
-->
p,(A) for every A e B for which j@A)
=
0. rl
The proof of this theorem is postponed to a later point in the development. Note
The Central Limit Theorem
348
meanwhile thatthe exclusion of events whose boundary points have positive probability corresponds to the exclusion of jump points of F, where the events in question have the form ((-x,x11. Just as the theol'y of the expectation is an application of the general theory of integrals, so the theory of weak convergence is a general theory for sequences of finite measures. The results below do not generally depend upon the condition gz(El = 1 for their validity, provided definitions are adjusted appropriately. However, a serious concern of the theory is whether a sequence of distribution functions has a distribution function as its limit; more specifically, should it follow because gw(R) 1 for evel'y n that g,@) = 1:7This is a question that is taken up in j22.5. Meanwhile, the reader should not be distracted by the use of the convenient notations f(.) and #(.) from appreciating the generality of the theory. ,
=
22.2 Example Consider the sequence of binomial distributions fBn,onls n 1,2,3,...), where the probability of x successes in n Bernoulli trials is given by =
Pxn
S
x)
=
(bn)x (1 Vn4n-x
=
x
-
x
,
=
0,...,n
(22.j )
(see 8.7). Here, lv is a constant parameter, so that the probability of a success for every n. falls linearly as the number of trials increases. Note that Exn) (x'')n-A x, and taking the binomial expansion of For fixed 1/x! as n e-Z whereas (1 1/rI)-* 1. We (1 bn) n sh ows that (1 hlnln as n may therefore conclude that =
--)
.x,
--
-->
-->
-
-
=,
--
!'
Pxn
x)
=
-.->
V
-k
-e
.X
x
,
-
0, 1,2,...,
=
(22.2)
.
and accordingly, Fu(c)
:j2P:ln
=
x)
=
-+
,-l
;2
06x<J
-.
x
(22.3)
1'.
0KAfJ
at all points a < x. Thus the limit (and hence the weak limit) of the sequence fBn,hln) l is the Poisson distribution with parameter E1 .
22.3 Example A sequence of discrete distributions on g0,1) is defined by Pxn
=
x)
1/?z, x
=
=
iln ,
0,
i
=
1,...,n.
otherwise
(22.4)
Tls sequence actually converges weakly to Lebesguemeasure m on (0,1), although this fact may be less than obvious; it will be demonstrated below. For any x e where denotes the largest integer (0,1), g,n((0,xq) (nxl/n x = pl(E0,.t)), There for which less than nx. are sets convergence fails, notably the set :(0,1) 1), in view of the fact that gw(Qp,1)) = 1 for every n, of all rationals in and rrl(Qp,1)) 0. But Q((),1j 0,1J and ??l(:(Q(n,1y)) 1, thus the definition of weak convergence in 22.1 is not violated. D
r/lxl
-->
=
=
r0,
=
=
Nctzk Convergence of Distributions
349
Although convergence in disibution is fundamentally different from convergcan ence a.s. and in pr., the latter imply the former. In the next result, be substituted for *-P-%% by 18.5. :-.14'
,
22.4 Theorem If Xn -T-r-->X, then Xn -P-> X. Proof For E
0, we have
>
+ < PX < x+
Plfxn < x)
-xl
f-h
( I.'L
> :))
#(1Ak 71
:) +
(22.5)
-
where the events whose probabilities appear on the right-hand side of the inequality contain (and hence are at least as probable as) the corresponding events on 0 by hypothesis, and hence the left. #( IXn -Yl > E) -.-/
-
limsup Pxn S x) S PX f x + 0.
(22.6)
N.->=
Similarly, #(X f x E) -
#( (X f x
=
:)
-
f-'h
( lXn XI S el) -
+ #((.Y f Pxn
< x)
+
< x- 6:) f-h
l 1-L--Yl
> El)
#( 4Xn -Yl > E),
(22.7)
liminf Pxn
(22.8)
-
and so PX f x
-
:) S
S x).
n-+x
f x) Since 6: is arbitrary, it follows that limn--joofax'a PX < x) at evel'y point x = = E) #(X S x). This condition is for which PX x) 0, such that limozoptx f x weak equivalent to convergence. w =
-
=
The converse of 22.4 is not true in general, but the two conditions are equivalent when the probability limit in question is a constant. A degenerate distribution has the form
F(X
0, x
<
a
(22.9)
=
1, x 2 a
If a random variable is converging to a constant, its c.d.f. converges to the step function (22.94,through a sequence of the sort illustrated in Fig. 22.1.
22.5 Theorem Xn converges in probability to a constant a iff its c.d.f. converges to a step function with jump at a. Proof For any E
0
>
#( jXn a l -
< E)
=
Pa
-
:
S Xn f a
+ E)
Central Limit Theorem
Ihe
350
= Fna
+
e) Fna -
-
e)-).
Convergence to a step function withjump
(22.10)
+ :) limpi-.-yx/kttz
Fa + :) at a implies E)-) 0 for a1l e > 0. The suffi1, and similarly limn-yxFnttc :)-) = Fa ciency pal't follows from (22.10)and the definition of convergence in probability. x, For the necessity, let the left-hand side of (22.10)have a limit of 1 as n for all er > 0. This implies =
=
-
-
=
-->
,
1im LFna
+ :)
-
Fnla
-
e)-)j
=
1.
(22.1 1)
n'M
Since 0 f F S 1, (22.11) will be satistied for all er > 0 only if Fa) F(J-) 0, which detines the function in (22.9).K
=
1 and
=
Fn n=x
1
i
I
N
l@
n = nj
E
n=nz
X
l1 <
E
!
INxs I
0
=
nl
<
<
nz
...
n,
'
-x
J
X
Fig. 22.1.
22.2 The Skorokhod Representation Theorem Notwithstanding the fact that Xn -P-> X does not imply Xn -S1..y X, whenever a sequence of distributions fFn ) converges weakly to Fone can construct a sequence of r.v.s with distributions Fn, which converges almost surely to a limit having distribution F. Shown by Skorokhod (1956)in a more general context (seej26.6), this is an immensely useful fact for proving results about weak convergence. Consider the sequence LFnl converging to F. Each of these functions is a monotone mapping from V to the interval (0,1J.The idea is to invert this mapping. Let a random variable (l) be defined on the probability space ((0,1),fp,1J,??z), where Sp,1J is the Borel field on the unit interval and m is the Lebesgue measure. Define for (t e (0,11
L()
=
inftx:
(l)
S Fn@)).
(22.12)
In words, Fa is the random variable obtained by using the inverse distribution function to map from the uniform distribution on (0,1Jonto 2-,taking care of any discontinuities in % 1() (correspon ding to intervals with zero probability mass intimum Fn) by the of under taking the eligible values. l'n is therefore a nondecreasing, lzft-continuous function. Fig. 22.2 illustrates the construction,
k
'''
W'ctzkConvergence of Distributions
.
.)r .:..
'
t
,
C
essentially the same as used in the proof of 8.5 (compareFig. 8.2). When Fn has that Fn(Fn())) iscontinuities it is only possible to assert (by light-continuity)
.
tlj.... )
351
Fn(Fn(x)) K x, by left-continuity of Fn. The tirst important feature of the Skorokhod construction constant a e R,
. j ..
.
.
:
j
)
.j
whereas
is that, for any
-
PYnu)
< a)
=
#(
S Fnaj)
=
Fna),
(22. 13)
where the last equality follows from the fact that ) is uniformly distributed on (0, 1q. Thus, Fn is the c.d.f. of L. 23 Letting F be a c. d f and F the r.v. corresponding to F according to (22.12), the second important feature of the construction is contained in the following result. .
.
1
Fig. 22.2
22.6 Theorem If Fn
=
F then Fu ---) F a.s.g/? as n
--
x.
(:1
ln working through the proof, it may be helpful to check each assertion about the functions F and i' against the example in Fig. 22.3. This represents the extreme case where F, and hence also F, is a step function; of course, if F is everywhere continuouj and increasing, the mappings are 1-1 nd the problem becomes trivial. Proof Let ) be any continuity point of 1', excluding the end points 0 and 1. For E < x < F()). any e > 0, choose x as a continuity point of F satisfying l'4l such of the countability discontinuities point will always Given the of F, a exist, and according to the definition of F, it must have the property F(x) < (t). F(x), there will be n large enough that Fn(x) < ), and hence x < lf Fnlx) L(), by definition. We therefore have -
-->
F())
-
6: < x <
L().
(22.14)
exists, since e is arbitrary (22.14) allows us Without presuming that limn-yxlv) F4(l)). to conclude that liminfn-xFnt) Next, choose y as a continuity point of F satisfying F() < y < F()) + E. The properties of F give f F(F()) f FV).For lalge enough rl we must also have <
The Central Limit Theorem
352
&@), and hence, again by definition of l%, 1%() f A' < i'()l
(22.15)
+ C.
In the same way as before, we may conclude that limsupa--jxFat) S F(). i'(). superior and inferior limits are therefore equal, and limn-oxikt))
The
=
12.* jrp/
gj(j)
F
yt
' --------.---.------.--.------.---.--.-------.----.----..--
,
(j)
jsrtx )
lj!j! 1
'j
j
..-.-------...---.-.-.-----.-...-----.-------....-----------
-.-...-----
j
I
#
I
! l
-1-:! 1'4)
-
ir
!
:
X
j j
I l i
! l:
I l
j
l i
! 1-
1. I A' F4))
1,4),
i I ! I
'
I
j
j
l,
!
24
.1'u()')
+ ir
1Xa7) Fig. 22.3 This result only holds for continuity points of K However, there is a 1-1 correspondence between the discontinuity points of F and intervals having zero probability under g, in R. A collection of disjoint intervals on the line is at most countable (1.11),and hence the discontinuities of i' (plusthe points 0 and F w.p. 1 grzlj,as 1) are countable, and have Lebesgue measure zero. Hence, L ,--.k
asserted. w
In Fig. 22.3, notice how both functions take their values at the discontinuities lnequality (22.15) holds for > at the points marked A and B. Thus, F(F((0)) ), but need not hold for (Y, a discontinuity point. A counter-example is the sequence of functions Fa obtained by vertical translations of the fixed graph f'rom below, as illustrated. In this case L()') > F(') + : for every n. '
=
.
22.7 Corollary Detine random variables Fs', so that Fn'() 1%()) at each where 0 and 0 at discontinuity points and at the function is continuous, and Fj() F/()) for every (t) e 0,1), and 1. Define Y' similarly. If Fx = F then L'() Fa' F'. and the functions of and Fn F are distribution =
=
=
-->
Proof The convergence for every (l) is immediate. The equivalence of the distributions follows from 8.4, since the discontinuity points are countable and their complement is dense in g0,1j,by 2.10. w ln the form given, 22.6 does not generalize very easily to distributions in R k fo r k > 1, although a generalization does exist. This can be deduced as a special case of 26.25, which derives the Skorokhod representation for distributions on general metric spaces of suitable type. A tinal pbitlt to observe about Skorokhod's representation is its generalization
Gdtzk Convergence of Distributions
353
to any finite measure. If Fn is a non-decreasing right-continuous function with codomain gt7,:j,(22.12) defines a function 1%() on a measure space ((tz,:1,
gu,x,m), where m is Lebesgue measure as before. With appropriate modifications, all the foregoing remarks continue to apply in this case.
The following application of the Skorokhod representation yields a different,
but equivalent, characterization of weak convergence. 22.8 Theorem Xn -P-->X iff
limElfxnll
F(J(m)
=
(22.16)
n-jx
for every bounded, continuous real function f. n The necessity
half of this result is known as the Helly-Bray theorem.
Proof To prove sufficiency, constnlct an example. For a
1,
x < a -x)/8,
fx)
a
=
a
8
-
-
G
R and
>
0, let
6
< x f
(22.17)
c
0, indicator' of the set We call this the continuous function with the properties tsmoothed
Fna
)
-
Fa
By hypothesis,
S
jfdl
S Fna),
8) <
-
JJJF
a11 z7,
(22. 18)
S Fa).
-->
-
8) S
jfdF
(22. 19)
(22.20)
S liminf Fna).
n-A= ---
(See Fig. 22.4.) It is a
fdF, and hence
fdt-n
limsup Fna Letting
(-x, t7l.
n-A*
0, combining
(22.19)and (22.20)yields
limjup Fna-)
K
(22.21)
F(J),
n-/x
F(J-)
f liminf
(22.22)
Fn(J).
These inequalities show that limnFntll exists and is equal to F(J) whenever F(a-4 = Fa), that is, Fn = F. To prove necessity, let f be a bounded function whose points of discontinuity are contained in a set Dy, where gtDyl 0, g, being the p.m. such that F@) g,((-x,x1). When Fn = F (& being the c.d.f. of Xn and F that of Ar)1%/((t) F/()) for every (l) s (0,1J,where Fn'((t))and F'((1))are the Skorolhod variables defined in f(Y'4 a.s.lg) by 18.8(i). 22.7. Since r?z(): F'((l)) 6 Dy) g.(Dy) 0, f(Yn') AU(F')), or The bounded convergence theorem then implies Efl) =
=
.-+
-->
=
=
-->
The Central Limit Theorem
354
jflgdvmLb')
jfyldbk,
(22.23)
jydbn-yb jxdbnf-'
(22.24)
--
where g.a is the p.m. corresponding to Fn. But 9.6 allows us to write
jfll'tdbtnll't =
=
=
Efxntt'
.i
with a similar equality for ElfX)l. (The trivial change of dummy argument from y to is just to emphasize the equivalence of the two fonnulations.) Hence we have E(f%44 it/ AU(m).The result certainly holds for the case Dy 0, so is proved. . .x
-->
=
Konly
' )' 1
-
.
-
1
t l l 1 ! 7 t-
tz-
x
a
Fig. 22.4 for the fundamental Notice how the proof cleverly substitutes (E0, 1),Sp,11,>?1) probability space (L1,5,P4 generating f-L), exploiting the fact that the derived distributions are the same. This result does not say that the expectations converge only for bounded continuous functions; it is simply that convergence is implied at least for al1 members of this large class of functions. The theorem also holds if we substitute any subclass of the class of bounded continuous functions which contains at least the smoothed indicator functions of half-lines, for example the bounded unifonnly continuous functions.
22.9 Example We now give the promised proof of weak convergence for 22.3. Clearly, in that example, 1
n
.fdg.n Tfiln). =
(22.25)
-
i= 1
The limit of the expression on the right of (22.25)as n c,o is by definition the Riemann integral of f on the unit interval. Since this agrees with the Lebesgue integral, we have a proof of weak convergence in this case. In -->
We shall subsequently require the generalization of Theorem 22.8 to general finite measures. This will be stated as a corollary, the modifications to the proof being left to the reader to supply; it is mainly a matter pf modifying the notation to
suit.
Gctzk Convergence of Distributions
355
22.10 Corollary Let (&) be a sequence of bounded, non-decreasing, continuous functions. Fn = F if and only if jl#Fn
right-
JJJF
(22.26)
--
for every bounded, continuous real function J.
E1
Another proof which was deferred earlier can now be given. Proof of 22.1To shtw sufficiency, considerA (-c,o, for which JA (xJ. Weak Aj p,(((-x,xj J) whenever convergence is detined by the condition gw(((-x, )) g,((xJ) 0. To show necessity, consider in the necessity pal't of 22.8 the case J(x) 1x@) for any A e :B. The discontinuity points of this function are 0, we have jtntAl g,(A) as a case of contained in the set A, and if jt(A) when Fn = F. . (22.16), .x1,
=
=
--
=
=
---)
=
22.3 Weak Convergence and Transformations The next result might be thought of as the weak convergence
counterpart of 18.8.
22.11 Continuous mapping theorem Let : R F- R be Borel-measurable with -1 0. If gz = g,, then gw/l discontinuity points confined to a set Dl. where blDR) -1 z:qhyt =
.
hY') Proof By the argument used to prove the Helly-Bray theorem, hYn') a.s.ljLl. mll h-n') (F'). Fa/() Since It follows from 22.4 that -G e A) h1n(X), -->
=
jt,/-1(4)
-1(4))
' rrlt: /1(1%()) e A)
=
zrl((l):F/4) n
e
=
(22.27)
-1
p.
for each A e B, using 3.21. Simtlarly, mlhY ) e A) jt/z *(A). According to the definition of weak convergence, hYn') -P-+ hY') is equivalent to g,,;/l-l = gW-l . -
.-
.
.
-
=
22.12 Corollary If h is the function of 22.11 and Xn -P-> X, then hxn) -P-+ 4m. Proof Imfnediate from the theorem, given that Xn
2213 @
If Xn -P- X(0,l), then
Example
-
Fn and X
-
F. w
X .-P-+x2(1).u
Our second result on transformations is from Cramr called Cramr's theorem:
(1946),and
22.14 Cramr's theorem If Xn -P-> X and L (i) (Xn + i' ) -P-> X+ J). (ii) Ynxn ; aX. (iii) XnlYn -P-> Xla, for a 0.
(constant),then
-E(-> a
is sometimes
---
:#:
Proof This is by an extension of the type of argument used in 22.4. Pxn
+
L f
.%
=
P(Xn +
L
f x,
IL
-
a
I
The Central Limit Theorem
356
S Pxn
S x
a
-
l'n <
IL
+ Pxn
+
+ :)
#( IFn a
+
.x,
-
lk
-
a
Ik
e)
E).
(22.28)
Similarly,
(22.29) and putting these inequalities together, we have
< Pxn
<
a
.x -
+ :)
#(Irn tzI
+
-
:).
(22.30)
Let Fxn and Fxn+yndenote the c.d.f.s of Xn and Xn + F,yrespectively, and let Fx be the c.d.f. of X, such that Fx limn--jxFktx) at a1l continuity points of Fx. Since limn-yxPt IL a I 2 E) 0 for a1l 6: > 0 by assumption, (22.30)implies =
=
-
Fxtx
a
-
; liminf Fxn+ynxj
:)
-
N.-e
; limsup Fxn.vvnlx)f Fxtx a -
+
(22.31)
0.
K.->co
Taking 6: arbitrarily
close to zero shows that
1im Fxn+rntx)
=
Fxtx
-
J)
(22.32)
Fxnxj
=
n.->x
point of Fx. This proves (i). first that To prove (ii), suppose a > 0. By taking 6: > 0 small enough we can ensure a e > 0, and applying the type of argument used in (i) with obvious variations, we obtain the inequalities
whenever x a is a continuity -
-
Plxna
+ E)
S x)
#(
-
IL
-
al
f P(Xn(a
E) S -
:)
Pxn'n
; x)
S a)
#( IFs a I
+
-
e).
(22.33)
Tnking limits gives Fxtx/tl
+ E))
f liminf Fxnynx) n->x
f limsup Fxnynxj f
Fx@/(J
-
E)),
(22.34)
n--hx
and thus 1im Fxnynx)
=
Fx(x/J)
=
Fa/x).
(22.35)
n-)x
0, replace i'n by -Fn and a by -J, repeat the preceding argument, apply 22.12. And if a 0, (22.33)becomes If a
<
=
Pxnz
S x)
-
#( IFnl k e) S Pxn'n
f x)
and then
W'ctz/cConvergence of Distributions S P-
f x)
+
357
#( Ii'n I
(22.36)
:).
0, this yields in the limit Fxayatx) 1, and for x < 0, Fxnvnlx) 0, which defines the degenerate distribution with the mass concentrated at 0. ln this -CC-> 0 in view of 22.5. (Alternatively, see 18.12.) case XnYn l/J if a # 0. To prove (iii) it suftkes to note by 18.10(ii) that plim 1/L Replacing L by 1/L in (ii) yields the result directly. . For x
>
=
=
=
22.4 Convergence of Moments and Characteristic Functions Paralleling the sequence of distribution functions, there may be sequences of JItF@), where it moments. If Xn -P-> X where the c.d.f. of X is F, then F(m exists, is sometimes called the asymptotic expectation of Xn. There is a temptadoes not exist tion to write A'(m limn-oxxn), but there are cases where Ex wlle F(m where for every n exists exists, and also finite E(Xn) for any n cases but F(m does not. This usage is therefore best avoided except in specific circmstances when the convergence is known to obtain. Theorem 22.8 assures us that expectations of boundedrandom variables converge under weak convergence of the corresponding measures. The following theorems indicate how far this result can be extended to more general cases. Recall that FIaYl is defined for every X, although it may take the value +x. =
=
If Xn .-P-.I X then FI.Yl < liminfn-yxflAkl
22.15 Theorem
.
Proof The function a(x) Ixl 1(Ixlua) is real and bounded. If #( IXI a) 0, it follows by 22.11 that hqxn) -P-> atm, and from 22.8 (lettingf be the identity function which is bounded in this case) that =
F(Jla(m)
=
=
lim Ehaxnl)
f liminf FlXn
distribution.
(22.37)
.
n->x
n--x
The result follows on letting a approach
I
=
+x
through continuity points of the
.
The following theorem gives a sufficient condition for F(m to exist, given that E (Xn ) exists for each n. is uniformly integrable, then F1A-1< 22.16 Theorem If Xn -P-) X and I-YnI
E (Xn)
-->
c.
and
E (A9.
--61..:
Proof Let L and J' be the Skorokhod variables of (22.12),so that Fa K Since Xn and Fn have the same distribution, unifonn integrability of (aL) implies that of ( Fn1.Hence we can invoke 12.8 to show that E(Yn) F(l'), F being integrable. F(m as required. x Reverjing the argument then gives FIXI < x and Exn) ---
.-.4
Uniform integrability is a sufticient condition, and although where it fails the existence of EX4 may not be ruled out, 12.7 showed that its interpretation is questionable in these circumstances. it):p A sequence of complex r.v.s which is always uniformlv intecrable is f e 1. for
The Central Limit Theorem
358
any sequence (-Yn). since le in l 1 Given the sequence tions f#xa(f)J,we therefore know that if Fn = F, then #xa(/) 9x(f) =
.
of characteristic func-
(22.38)
-->
(pointwise on R), where the indicated limit should be the characteristic function associated with F. In view of the inversion theorem (11.12),we could then say that Xn -P-> X only if (22.38)holds, where (),(fl is tlte ch.f. of X. However, it is rather than the that is the point of interest here. If a the of characteristic functions converges pointwise to a limit, under what sequence circumstances can we be sure that the limit is a ch.f., in the sense that inverting yields a c.d.f.? A sufficient condition for this is provided by the so-called Lvy continuity theorem: ifX
'only
Eif'
22.17 Lvy continuity theorem Suppose that fFn ) is a sequence of c.d.f.s and Fn non-negativerbounded,
whereFisany tion. If =F,
+x
9s(/)
J
=
non-decreasing,Hght-continuouslnc-
eitxdb
and (j(/) is continuous at the point t (1)is its ch.f. u
9(f),
-->
(22.39)
0, then F is a c.d.f.
=
(i.e., JJF
1) and
=
The fct that the conditions imposed on the limit F in this theorem are not unreasonable will be established by the Helly selection theorem, to be discussed in the next section. Proof Note that 4)s(0) dl c-d.f. For v > 0, =
+x
v
-1 VJgtntvt Jf =
()
1 for any n, by
=
j --
v
eitxdt
+x
(w.n
g J
=
VJ ()
..x
(22.39)and e
fvx
!
sx-
dn,
the fact that Fn is a
(22.40)
..x
the change in order of integration being permitted by 9.32. By 22.10, wlaich extends to complex-valued functions by linearity of the integral, we have, as n
'-
x,
+x e fv.x 1
+x
-
fvx
e
dFn .--A
fv.x -
1
fvx
dF
=
1
v
v
()
-
(dt,
(22.41)
where the equality is by (22.40)and the definition of #. Since () is continuous at t 0, limv--yxv-llxt/llf ((0) and since (n (h we must have ((0) 1. It follows from (22.41)that -->
=
=
=
+x
1
=
ivx 1 e JF li m Jvx
-=
+x
-
.
V-40
dF.
=
(22.42)
-=
In view of the other conditions imposed, this means F is a c.d.f. It follows by tX 22.8 that (/) = Ee ) w h ere X is a random variable having c.d.f. F. This completesthe proof. w
W'ctzkConvergence of Distributions
359
The continuity theorem provides the basic justificationfor investigating limiting distributions by evaluating the limits of sequences of ch.f.s, and then using the inversion theorem of jl 1.5. The next two chapters are devoted to developing these methods. Here we will take the opportunity to mention one useful application, a result similar to 22.4 which may also be proved as a corollal'y.
22.18 Theorem If lXn Zn I -E+ 0 and lXn J converges in distribution, then fZn) converges in distribution to the same limit. -
le in
Proof
0 by 18.9(ii). Since Ierxl = l these functions are itzn the sequence ( Ie in e I) is uniformly integrable. So by i'Zn .--I--:Z 0 However, the complex modulus inequality (11.3) e l i'Zn
e and fa-bounded,
18.14, Ie in gives
-
-
I
-'.%
-
.
FIe itxn e -
itzn
ls e ) yeitzn ) j l9xa(/)Yz.(/)I 0 as
itxn
I
(a,z43)
-
,
.
pointwise on R. Given the assumption of weak convergence, the conclusion now follows from the inversion theorem. .
so that a further consequence is
To get the alternative
-..h
-
proof of 22.4, set Zn
=
n
-->
oo,
X for each n.
22.5 Criteria for Weak Convergence Not every sequence of c.d.f.s has a c.d.f. as its limit. Counter-examples are easy to construct.
22.19 Example Consider the uniform distribution on the interval (-n,n1,such that
Fnla)
=
1a(1 + c/a),
-n
S a S n. Then Fa(c) --A
z1for all a
G
R n .
22.20 Example Consider the degenerate r.v., Xn = n w.p.1. The c.d.f. is a step 0, all a e R n function with jump at n. Fna) --)
.
Although Fn is a c.d.f. for all n, in neither of these cases is the limit F a cxa 1 (0) as a c.d.f., in the sense that Fa) (-=). Nor does intuition suggest to us that the limiting distributions are well defined. The difficulty in the first example is that the probability mass is getting smeared out evenly over al infinite support, so that the density is tending everywhere to zero. It does not make sense to detine a random variable which can take any value in R with equal probability, any more than it does to make a random variable infinite almost surely, which is the limiting case of the second example. In view of these pathological cases, it is important to establish the conditions under whichasequenceof measures canbe expectedtoconvergeweakly. The condition that ensures the limit is well-defined is called uniform tightness. A sequence lgzl of p.m.s on R is uniformly tight if there exists a tinite interval (tz,:J such that, for any : > 0, sup,,pzttl,hll > 1 :. Equivalently, if (Fn J is the sequence of c.d.f.s corresponding to lg.nl,uniform tightness is the condition that for 6: > 0 3 a,b with b a < x and -....:
--
-
-
The Central Limit Theorem
360
sup (Fb) l
Fnaj
-
)>
1
-
(22.44)
E.
It is easy to see that examples 22.19 and 22.20 both fail to satisfy the unifonn tightness condition. However, we can show that, provided a sequence of p,m.s tgzJ is uniformly tight, it does converge to a limit jt which is a p.m. This terminology property that for every derives from the designation tight for a measure with E > 0 there is a compact set A-esuch that g(A1) S e. Every p.m. on (E,f) is tight, although this is not necessarily the case in more general probability spaces. See 926.5 for details on this. An essential ingredient in this argument is a classic result in analysis, Helly's selection theorem. jthe
22.21 Helly's selection theorem If (Fn) is any sequence of c.d.f.s, there exists a subsequence fnk,k 1,2,...1 such that Fnk = F, where F is bounded, non-decreasing and right-continuous, and 0 K F K 1. =
Proof Consider the bounded array ( fFnxij, n G ENJ, i e N 1,where (a),i e EN) is an enumeration of the rationals. By 2.36, this array converges on a subsequence, so that limk-yx/ktxj) = F*x for every Note that F'xik) S F'txfa) whenever < Aa, since this property is satisfied by Fn for every n. Hence consider the nondecreasing function on R, .:/l
.
F(xl
inf F*(xj).
=
(22.45)
Xi >.T
Clearly 0 K F*xij f 1 for a11 i, since the Fnjxij have this property for every k. By definition of F, for x e R 3 xi > x such that F(x4 S F'xi) < Flxj + 8 for Fxi). Further, for any e: > 0, showing that F is right-continuous since F'xi) A7 > such that points of there exist and < F, continuity xiz x xfj x =
F(x)
-
: <
F*(xjj) f F'xiz)
<
Fx)
+
(22.46)
t.
The following inequalities hold in respect of these points for every kl F'bxiL)
lim Fnklxik)f liminf Fnkx)
=
k-yx
k-yx
f limsup Fnkx) < lim Fnklxiz) = F'xi. k-4x
Combining
(22,47)
k-x
(22.46)with (22.47), F@) s < liminf Fnkx) f limsup Fnjxj -
k-x
<
F(x)
+ :.
(22.48)
k->x
Since s is arbitrary, limk-yx/ktx)
=
F@) at all continuity points of F.
.
The only problem here is that F need not be a c.d.f., as in 22.19 and 22.20. We 1 (0) as x .--A x (-x), and tightness is the required need to ensure that F(x) property. -->
22.22 Theorem Let (Fn ) be a sequence of c.d.f.s. If
JFetzk Convergence of Distributions
361
(a) Fnk u:zh F for every convergent subsequence (rlk),and (b) the sequence is uniformly tight, then Fn
=
F, where F is a c.d.f. Condition
(b) is
also necessary.
I:n
Helly' s theorem tells us that (Fn ) has a cluster point F. Condition (a) requires that this F be the unique cluster point, regardless of the subsequence chosen, and the argument of 2.13 applied pointwise to (Fn ) implies that F is the actual limit of the sequence. Uniform tightness is necessary and sufficient for this limit F to be a c.d.f. Proof of 22.22 Let x be a continuity point of F, and suppose Fn(x) zz-y F@). Then lFn@) F(x) I : > 0 for an infinite subsequence of integers, say (rls k e (Nl Define a sequence of c.d.f.s by FJ Fnk, k = 1,2,... According to Helly's theorem, this sequence contains a convergent subsequence, lk, i G (N), say, such that FLi= F. But by (a),F' F, and we have a contradiction, given how the subsequence fkfl was constructed. Hence, Fn = F. Since Fn is a c.d.f. for every n, Fnb) Fn(J) > l for some b a < x, for F at continuity points, increase b and reduce a as necesany t: > 0. Since Fn sary to make them continuity points of F. Assuming unifonn tightness, we have by 1 and (22.44) that Fb) Fa) > 1 :, as required. lt follows that limx-jxfx) limx-o-xfx) 0. Given the monotonicity and right continuity of F established by Helly' s theorem, this means that F is a c.d.f. Fa) < 1 s On the other hand, if the sequence is not uniformly tight, Fb) and +cxn for some : > 0, and every b > a. Letting b a we have F(+=) F(-x) f 1 : < 1. Hence, either F(+x) < 1 or F(-x) > 0 or both, and F is not a c.d.f. . -
=
.
=
-
-
-
.--
-
=
-
=
-
-
-x,
--
-
--
-
The role of the continuity theorem (22.17)should now be apparent. Helly's theorem ensures that the limit F of a sequence of c.d.f.s has all the properties of a 1. Uniform tightness ensures this property, c.d.f. except possibly that of JJF and the continuity of the limiting ch.f. at the origin can now be interpreted as a sufficient condition for tightness of the sequence. lt is of interest to note what happens in the case of our counter-examples. The ch.f. corresponding to example 22.19 is =
G(V)
=
(2n)-1
n (cOs ..n
Mx+ j
siu vxlts
=
sinvrl .
Mn
We may show (usel'Hpital's rulel that (n(0) 1 for every n, whereas for a1l v # 0. ln the case of 22.20 we get =
#u(v)
=
cos s'n + i sinvn,
which fails to converge except at the point v
(22.49) n(v)
--)
0
(22.50) =
0.
22.6 Convergence of Random Sums Most of the important weak convergence results concern sequences of partial sums of a random array fxnt, t 1,...,n, n e (NJ. Let =
The Central Limit Theorem
362
Sn
=
Xnp X -1
(22.51)
and consider the distributions of the sequence (&J as n --A cxn. The array notation (double indexing) permits a normalization depending on n to be introduced. Central limit theorems, the cases in which typically Xnt n -1/2X(, an (j s n converges to the Gaussian distibution, are to be examined in detail in the following chapters, but these are not the only possibility. =
22.23 Example The Blnshln) distribution is the distribution of the sum of n inde1) hln and Plxnt = 0) pendent Bernoulli random variables Xnt, where Pxnt 1 hln. From 22.2 we know that in this case =
=
=
-
Sn
=
XXnf
---/D
Poisson with mean
,.
>1
(22.52)
n
From 11.1 we know that the distribution of a sum of independent r.v.s is given by the convolution of the distributions of the summands. The weak limits of independent sum distributions therefore have to be expressible as infinite convolutions. The class of distributions that have such a representation is necessari1y fairly limited. A distribution F is called infinitely divisible if for every n e INthere exists a distribution Fn such that F has a representation as the n-fold
convolution
F
=
Fn*Fu*...*Fn.
(22.53)
ln view of (11.33), infinite divisibility implies a corresponding multiplicative rule for the ch.f.s. 22.24 Example For the Poisson distribution, and
#x(l;D =
expl/f
9x(?;) (expf(=n)(:'-1)J)n @x(f;/n))n. =
=
-
1) from ,
(11.34), (22.54)
The sum of n independent Poisson variates having parameter hln is therefore a Poisson variate with parameter :. n In certain intinitely divisible distributions, Fn and F have a special relationship, expressed through their characteristic functions. A distribution with ch.f. #x is called stable, with index p, if for each n,
n (9x(p) .
etbn)
tx(sl/p/; () < p x z, ,
(gg.55)
where bln) is some function of n. According to (11.30),the right-hand side of (22.55) is the ch.f. of the nv. bln) + n 3IPX; that is, the sum of n independent drawings from the distribution is a drawing from the same distribution apart from a change of scale and origin. If a stable distribution is also symmetric about zero, it can be shown that the ch.f. must belong to the family of real-valued functions having the fonn
W'ctzkConvergence of Distributions
91/)
=
expl
I/1,)
-.a
363 (22.56)
a k 0.
,
0, having 22.25 Example The Cauchy distribution is stable with p 1 and bnj C(V,6) fOr exptf/v v,8) 1,...,n, 61 ch.f. #x(/; t fI), from (11.38).lf Xt then kn C(v,6). This result retlects the fact already noted (see 18.18) that Cauchy variates fail to observe the 1aw of large numbers. (:1 =
=
=
-
=
-
-
-zlt/zj
by (11.37).This 22.26Example For Gaussian X, (xff',g,,c2) expt g./ n1/2)g, with about index p 2 and bn n iSS table with2 g, 0 we have symmetry c2) and N(0 for obtain the formula in if X Thus, with t t 0, (22.56) 1 a zc 0.24. j,4 N(O, u 1 n then n-ll; =
=
=
=
-
.
=
-
.
,
=
-
It turns out that a stable distribution with index p < l possesses absolute moments of order at most r < p. Thus, the Gaussian is the only stable 1aw for which a variance exists. This last fact is most important in motivating the central limit theorem. Each of the foregoing examples illustrates the possibility of a limit 1aw for sums. A stable distribution naturally has the potential to act as an attractor for rbitrarily distributed sums. While stable convergence laws do operate in cases where the summands possess absolute moments only up to order r < 2, the key result, to be studied in detail in the following chapters, is that the Gaussian acts as the unique attractor for the distributions of scaled sums of arbitra r.v.s having zero mean and finite variances.
23 The Classical Central Linzit Theorem
23. 1 The i.i.d. Case law of error' is justly the most famous result in statistics, and to susceptible mind has an almost mystical fascination. If a sequence of random the variables (#/)Thave means of zero, and the partial sums Z:=1m,n = 1,2,3,... have variances sn2 tendin g to infinity with n although finite for each finite n, then, subject to rather mild additional conditions on the distributions and the The
tnormal
sampling process,
sn
=
-
1
Sn
n
Tx
-P-+ #(0, 1).
(23.1)
>1
This is the central limit theorem (CLT). Establishing sets of suftkient conditions is the main business of this chapter and the next, but before getting into the fofmal results it might be of interest to illustrate the operation of the CLT as an approximation theorem. Particularly if the distribution of the Xt is symmetric, the approach to the limit can be very rapid.
23.1 Example In 11.2 we derived the distribution of the sum of two independent (40, 12 drawings. Similarly, the sum of three such drawings has densty /k+I'+w)
=
f-l(.1(lltw-c-l,w-zldAdz, JoJ
(23.2)
which is plotted in Fig. 23.1. This function is actually piecewise quadratic (the tllree segments are on (0,1j, g1,21and (2,31respectively), but lies remarkably close to the density of the Gaussian nv. having the same mean and variance as
The sum of 10 or 12 independent uniform r.v.s is almost indistinguishable from a Gaussian variate; indeed, the formula S Xl=21#j6, which has mean 0 and variance 1 when Xi ULO,11 and independent, provides a simple and perfectly adequate device for simulating a standard Gaussian variate in computer modelling exercises. n
X+ F+ Z
(alsoplotted).
=
-
-
23.2 Example For a contrast in the manner of convergence consider the Bn,p) variates for fixed p e (0,1).The probadistribution, the sum of n Bernoulli) 1a and n 20 are plotted in Fig. 23.2, together with the Gaussian bilities for p with matching density mean and variance. These distributions are of course discrete for every finite n, and continuous only in the limit. The correspondence of the ordinatej is remarkably close, although remember that for p # ,1 the binomial distributiop is not symmetric and the convergence is correspondingly slower. This =
=
The Classical Central Limit Iheorem
365
example should be compared with 22.2, the non-Gaussian limit in the latter case being obtained by having p decline as a function of h. n
sumof
Nq, 14)
3 &g0,1js N
z
0.4 N
z
0
N
0
l.5
3
Fig. 23.1
0. 2
I Bin(20,
0.5)
x
x(1(),5)
?' / 0.
$
3.
1
?' ?' ?' & J'
O
x
--
O
.
10
20
Fig. 23.2 Proofs of the CLT, like proofs of stochastic convergence, depend on establishing properties for certain ncnstochastic sequences. Previously we considered sample 1, probabilities #( IxL #1 > E), and X()) I for e C with PC4 points lXnk XIP, moments FIXn as different sequences to be shown to converge to 0 to establish the convergence of Xn to X, respectively a.s., in pr., or in Lv. In the present case we consider the expectations of certain functions of the Sn; th key result is Theorem 22.8. The practical trick is to find functions that will fingerprint the limiting distribution conclusively. The characteristic function is by =
-
-
-
'he Central Limit Theorem
366
common consent the convenient choice, since we can exploit the multiplicative property for independent sums. This is not the only possible method though, and the reader can tind an alternative approach in Pollard (1984:H1.4), for example. The simplest case is where the sequence lA) is both stationary and independently drawn. 23.3 Lindeberg-Lvy variance c 2
theorem If
is an i.i.d. sequence (XTIT
having zero mean and
,
l
s
n-llTx /c
=
rl
>1
#x(1)of
Proof The ch.f.s (1 1.33),
Applying 11.6 with k
1t/x(,c whichmakes
-
n
)
-
((VVC-'X-1Y))S =
and
(23.4)
.
j
ljgyt
-
j x s mij,j
3 h-k)2 Izxt j j
(23.5)
j
( o'V 6c an w
it possible to write, for tixed 1,
pxtlc-1n
-1/2
)
=
o(,,-a/2 ).
1 yzjz,t + -
Y:=/:)(J/n)/ Applying the binomial expansion, (1+ aln? Il2 + ()(n-1/2), we j'ind x, and setting a n =
--
(11.30)
2 yields the expansion
=
1/2
(23.3)
Xt are identical for al1 /, 24 so from
p-a(X)
-1
-.P-+x(o, 1). o
t
=
(23.6) --+
;
=tg/./!
=
ea as
-,
lim
9.sa(D =
e- 2/2
(23.7)
.
n-+x
Comparing this formula with (11.36), the limiting ch.f. is revealed as that of the N(0, 1) variable. We then appeal to the inversion theorem (11.12)to establish that the limiting distribution is necessarily Gaussian. . Be careful to note how the existence of E IXl 3 is no t necessary for the expansion function whose expectation appears on the majorant in (23.6)to hold. The side of (23.5)is unquestionably of On -3/2), but also clearly j n teggable for each n on the assumption of finite variance. The Lindeberg-l-vy theorem imposes strong assumptions, but offers the benetit of a simple and transparent proof. Al1 the key features of the central limit 1/2 property are discernible. In (23.6), the expansion of the ch.f. of n - Xt cnsists either of terms common to every centred distibution with tinite variance, or of tenns that can be neglected asymptotically, a fact that ensures that the ltmiting sum distribution is invariant to the component distributions. The imaginary part of xt,c -1n ) is of smaller order than the rea j pa y.t wjch would appear to require a symmetric limit by the remarks following (11.31). The coincidence of these facts with the fact that the centred Gaussian is the only stable symmetric distribution having a second moment appears to r'ule out any Kmin'
-1/2
,
The Classical Central Limit Theorem
alternative to the central limit property under the specified conditions, that is, zero mean and finite variance. The earlier remark that symmetry of the distribution of n -1/2-Y)improves the rate of convergence to the limit can be also in (23.5)can be taken appreciated here. If we can assume Ext)3 0 the expansion -2 =
,
to third order, and the remainder in (23.6)is of On ). On the other hand, if the variance does not exist the expansion of (23.6)fails. Indeed, in the specific case we know of, in which the Akare centred Cauchy. ()(nl/2); the (n3IlX j is not tight, and n= n 1/22 sequence of distributions of n there is no weak convergence. The -1limit 1aw for the sum would itself be Cauchy under the appropriate scaling of n Thedistinction between convergenceindistribution andconvergenceinprobability, and in particular the fact that the former does not imply the latter, can be demonstrated here by means of a counter-exnmple. Consider the sequence (A)!' defined in the statement of the Lindeberg-Lvy theorem, and the corresponding Sn .
in (23.3). in probability.
23.4 Theorem Sn does not converge Proof If it was true that plims-oxs'n = Z, implying
Z, it would also be the case that plims--yxu&a,,
=
plim hn Sn) -
=
0.
(23.8)
n-4x
We will show that
false. We have
(23.8)is
ln
n
t /c + X x%u (2rl)-1/2F'').;&r
kj /c
=
-
=
n
l/2
(Sn +
(23.9)
sp,
p=l+1
>1
whereSn'
=
2n Z,=+1A7c, j.jence
Szn Sn -
=
x%41//1' -
1) +
Sn'IV.
(23.10) 1 2 2
expt and 4x() According to the Lindeberg-l-vy theorem, 4x() x,Sn'c ) contained and is in both Xt Sn sums, are independent, expf sinceno n1. Noting g1/A- 1)J2+(1//jj2 2 /j' and eachwith mean zero and variance applying the properties of ch.f.s, .-.:
-->
-r1,2c2j.
=
-r12(2 -snh)
(x2n
-->
exp
(
-
Wlcz j
.
-
(23.11)
In other words,
v%zn sn -P-> N(0, c2(2 /j')), -
-
whichis the
required
contradiction of
(23.12)
(23.8)..
with, say, (xY4))+ Ykln JT, ) G f1, whereXand F Compate the sequence (&((z))JT the latter sequence converges to a t'ixedlimit, are random variables. For given X(). On the other hand, each new contribution to Sn has equal weight Qth the others, ensured by re-scaling the sum as the sample size increases. For given (, 1/2 'nn) = n - Z)=1m())is not a convergent sequence, for as 23.4 showj Szn is ,
Ihe Central Limit Theorem
368
not neessarily close to Sn no matter how large n becomes. Weak convergence of the distribution functions does not imply convergence of the random sequence. Characteristic function-based arguments can also be used to show convergence in distribution to a degenerate limit. The following is a well-known proof of the weak law of large numbers fori.i.d. sequences, which circumvents theneed to show f,! convetgence. is an identically and independently distributed 23.5 Khinchine's theorem If (XIJT rI-17=I.Y) -ZL> g,. sequence' with tinite mean jt, then kn =
kn has the
Proof The characteristic function of
kt)
form
(xlvnqjn.
=
(23.13)
wherye by application of the argument used for 23.3, (xvnj x we find, by analogy with (23.7), + 0h ln 2). Letting n
=
1 + ihpln
-->
lim4,,:()
=
/1
eikk
(23.14)
*
n-,>oo
fB But Ee 'f) e on ly for the case where X = jt with probability 1. The distribution is degenerate, and convergence in probability follows from 22.5. . =
23.2 lndependent Heterogeneous Sequences The Lindeberg-Lvy theorem imposes conditions which are too strong for theresult to have wide practical applications in econometrics. In the remainder of this chapter we retain the assumption of independence (to be relaxed in Chapter 24), but allow the summands to have different distributions. In this theory it is convenient to work with normalized variables, such that the partial sums always have unit variance. Tlis entails a double indexing scheme. Define the triangular array
fxnt, t
=
1,...,n, n e EN ),
2 suc h that if the elements having zero mean and variances nnt, Sn
XXnC,
=
(23.15)
>1 then (underindependence) Esnh
57cjf
=
->
=
(23.16)
1;
1
Typically we would have Xnt (Fr gfl/ln where (F,Jis the sequence under 2 with study, means (w), and ss l EY f w)2.In this case c2nt 2 E (F, g. 2Isn, S0 th a t these variances sum to unity by constnlction. It is also possible to have Xnt = (F,a bknlsn, the double indexing of the mean arising in situations where the sequence depends on a parameter whose value in turn depends on n. This case arises, for example, in the study of the limiting distributions of =
Kraw'
-
=
-
-
.1
-
=
The Classical Central Limit Theorem
369
deviations from the null hypothesis, test statistics under a sequence of the device known as Pitman drift. The existence of each variance cjf is going to be a necessary baseline condition in all the theorems, just as the existence of the common variance o.2was required in the Lindeberg-Lvy theorem. However, with heterogeneity, not even unifonnly bounded variances are sufficient to get a central limit result. If the Ff are identically distributed, we do not have to worry about a small (i.e. finite) number of members of the sequence exhibiting such extreme behaviour as to influence the distribution of the sum as a whole, even in the limit. But in a heterogeneous sequence this is possible, and could interfere with convergence to the of eachindividual memberof the normal, which usually depends onthecontribution sequence being negligible. The standard result for independent, non-identically distributed sequences is the Lindeberg-Feller theorem, which establishes that a certain condition on the distributions of the summands is sufficient, and in some circumstances also necessal'y. Lindeberg is credited with the sufficiency pal't, and Feller the necessity', we look at the latter in the next section. 'local'
23.6 Lindeberg theorem Let the array Lxnt ) be independent with zero mean and variance sequence (cjr) satisfying (23.16).Then, Sn -P-+ N(0, 1) if 1im
n
X >
nOx
1
j
Xlt dp
0, for all e
=
n
l IXnt I> e.l
0.
>
(23.17)
EI
Equation (23.17)is known as the Lindeberg condition. The proof of the Lindeberg theorem requires a couple of purely mechanical
lemmas.
23.7 Lemma lf x1,...,.v and y1,...,yn are collections of complex numbers with la1 f 1 and lyrl K 1 for t 1,...,n, then =
n
l
1-l 1-1 >1 xt
y,
-
n
K
>1
Proof For n
=
77Ixt
-y,1
(23.18)
.
>1
2,
I.xl.x2yly2 l -
+ (x2 y2)y1I l(.z'1 K Im 11x2I+ 1.x2 -y2I1y1 I < 1x1 I + Ix2-y2I -z1)av
=
-
-y1
(23.19)
-y1
.
The general case follows easily by induction.
.
23.8 Lemma If z is a complex number and Iz I f ;,1 th en
I
CZ
1 -zI <
-
2.
IzI
Proof Using the triangle inequality,
Iez
-
1 zI -
=
x zj
s. j=z J.
=
2
z
x .
j=0
z +
j
2) !
S
2
Iz I
x .
j'
jz j.j +
.
a)!
(23.20)
The Central Limit Theorem
37O Since
T 2-7 Zy=.f)
2 the infinite series on the
=
4x()
Proof of 23.6 We have to show that bounded by
I
9.s-(1)
j
2/2
e-
-
n
#xa,(v) 1-1
=
-
>1
n
side cannot exceed 1.
light-hand
,
e- :2/2as n
--
2
e-'
1-1
=.
--
.
The difference is
o'l
>1
;!
.
f >1119xa,(D 11(1xzc2ntl -
-
>1
n
11
+
e '-
n
2 @
%t12
jj(j j,zgzup
-
-
,
>1
>1
gg
.g
j)
2 = 1, and the where the equality is by definition, using the fact that Z)=1cnr inequality is the triangle inequality. The proof will be complete if we can show that each of the right-hand side terms converges to zero. 2 an d The integrals in (23.17) may be expressed in the form F(1 Xnt), ( IXnt I> e:l
+ F(1( 1xng1>:)X2at) 0-2 F(1( jxa,!<:)Xn2;J Ix.I>:)xn2,) s + F(1f M f2 as n a11 1 s t < n, 2 nt
=
(23.22)
--)
x,
since the Lindeberg condition implies that the second term on the right-hand side of the inequality (whichis positive) goes to zero; since ir can be chosen arbitrarily small, this shows that max ot2
()
-->
(aa aa)
.
.
1K t f n
In the t'irst of the two terms on the majorant side of (23.21),the ch.f.s are al1 2 less than 1 in modulus, and by taking n large enough we can make l 1 g1z2cn/ j < j for any tixed value of 1. Hence by 23.7, -
n
n
I-IIVnJXI gJ(11z,2cu2p -
-
>1
>1
l
<
X jjxa/) >1
To break down the terms on the majorant side of case k 2, combined with (11.29),yields
-
(1
-
xzcznp j .
(23.24)
(23.24),note that 11.6 for
the
=
)
Ak,()
j
S FtminftkYnJz, (1 -1.2X2c2,jJ j ILALI
-
3j)
K Hence, recalling
2 Z:=1ou
=
(Vr,,(X) Yl >1 j -
nt
I>:)X2?:,) +
3:c2?g. jj j
1
,
(1 X2cl) -
( 1x 2f(1
!f
2'XA'(1( IAk,I>s)X,$)+ 161/,13: >1
(23.25)
Thc Classical Central Limit Theorem
11113: as n
0:
371 (23.26)
-0
x,
since the first majorant-side term vanishes by the Lindeberg condition. Since arbitrary, this limit can be made as small as desired. Similarly, for the second tenn of n
j'j
n
g a
-).
c#2
e
jj
.
>1
>1
(1
n
j:2c2uj) <
-
(23.21),take j
y''q je 1
j
.
j
lr/czu,
-
(23.27)
.
/=
2 12 Gnt Setting z (a rea 1 num ber actually) in 23.8 and applying of the majorant side (23.27)gives =
-;
,
X
X
22
-).
cn#2
j-j o >1
.
j'j(j
1yzo.jy;
.
u
>1
is
n large enough so that
aoV2
-
:
1414
Y
y
the result to
g4us
(23.28)
>1
But, N
n
4 (maxd/)Xc2u,max Xcn, <
=
/=l
c2u,
0 as n
--y
(23.29)
--y
x,
tzz1
by (23.23).The
proof is therefore complete.
.
The Lindeberg condition is subtle, and its implications for the behaviour of easier if we look randomsequences can for careful interpretation. Tis will be c2, variance and 17.102,. at the case xnt xtlsn,where xt has mean O and Then the Lindeberg condition becomes .2u
=
=
n
lim-i'1 X
N-x
Sn
xltdp
=
t lx t jso:)
>1
0, for a11 :
>
(23.30)
0.
One point easily verified is that, when the summands are identically distributed, Sn 'Zc7 and (23.30)reduces to limn-yx sn-lEx; 1( jxj j>o.G:)) 0. The Lindeberg condition then holds if and only if X1 has tinite variance, so that the Lindeberg theorem contains the Lindeberg-l-vy as a special case. The problematical cases the Lindeberg condition is designed to exclude are those where the behaviour of a finite subset of sequence elements dominates a1l the others, even in the limit. This can occur either by the sequence becoming excessively disorderly in the limit, or (the other side of the same coin, really) by its being not disorderly enough, beyond a certain point. Thus, the condition clearly fails if the variance sequence (c2f ) is tending to 2 2 = zero in such a way that sn )7=lcr is bounded in n. On the ot jaer jaand if n x for any fixed positive :, and the Lindeberg condition resembles a =, then sne The sum of the terms condition of ' average un iform integrability of (-Y2). t 2 f (1(lx,!>,ns)X,) must grow less fast than no matter how close E is to zero. The following is a counter-example (compare12.7). =
=
,
.2
.-+
,
--
'
,n2
,
-2 23.9 Example Let Xt F, F(Ft) where 1$ 0 with probability 1 t and t with probability f -2 Thus E(Yt) = f-1 and Xt is converging to a degenerate r.v., equ/l =
.
-
=
-
,
The Central Limit Theorem
372
1 t -2 for every t The Lindeberg to 0 with probability 1, although Vartr condition fails here. sn2 n 7.1/ an d for 0 < r f 1 we certainly have t > 1/2 Therefore, sne whenever t > n =
-
.
-2
=
-
,
.
-1
n
1 'n
n
X26IP
X
t lxrl>'nf1
>1
t
2
-2
(/ t ) t -
,GJ+l n
r-r
-
ZJ=1 n
-2
t
(23.31)
01
x. And indeed, we can show rather easily that the CLT fails here. For any as n )721J, then if no 2 1, we put Bo --
=
n
<x
o
P 77F, sup n
>
U
Bo < #
,...1
l Ii',1
f=x+l
>
0)
K
f-2, X r=na+l
(23.32)
where the majorant side can be made as small as desired by choosing no large enough. lt follows that Z:.1J'f 07,(1) and hence, since we also have L1=jEYt) .TJ.-> 0, confinning that the CLT does not operate in this Otlog n), that =
=
CaSe.
xqxtlsn
El
Uniform square-integrability is neither sufficient nor necessary for the Lindeberg condition, soparallels mustbe drawn withcaution. However, thefollowingtheorem gives a simple sufficient condition.
2) is uniformly integrable, and slln 23.10 Theorem If (.;V n (23.30) holds. Proof For any n and e n
1 -E 77e1( kn
>
0, the latter assumption
1 (x,$>&,,eJ.Y2f) < maxFtlt
>
0 for a11 n, then
implies
1x,l>,ao)Xr)j. 2
y l:g//n
>1
B
(23.33)
Hence n
1 F7F(1( 1im a--)x a an
Ix,I>os)Xl) K 1
maxtftlt
ylimsup
>1
n--yx
s
-sup
1
B
Ixjl>xuclxrlj a
1S/Kn
z lim F(1( Ix,I>.so)-Y,) t
N--co
= 0, where the last equality follows by uniform integrability.
(23.34) .
There is no assumption here that the sequence is independent. The conditions involve only the sequence of marginal distributions of the variables. None the less, the conditions are stronger than necessary, and a counter-example as well as further discussion of the conditions appears in j23.4. The following is a popular version of the CLT for independent processes.
23.11 Liapunov's
theorem A sufficient condition for (23.17)is
The Classical Central Limit Theorem N
2+8 1imXFIAk,I n-yx
Proof For 8
>
0 for some 6
=
>
(23.35)
0.
>1
0 and
0,
6r >
2+
2+
2 F( 1f Ixa,I> i;) IXntI
E IXntI
(23.36)
)
.x2
j
2 e: E
( Ixa,I>:)
nt
) .
The theorem follows since, if s olim ,,-,x k1 lF(1f Ixa,I>:)Xl) then the same holds with : rep 1ace d by 1 . =
=
0 for fixed tr > 0,
.
Condition (23.35)is called the Liapunov condition, although this tenn is also used to refer to Liapunov's original result, in which the condition was cast in terms of integer moments, i.e. n
7-)A'l xk,I3
lim
a-+x
=
().
(23.37)
>1
Although stronger than necessary,
the Liapunov condition has the advantage of
being more easily checkable, at least in principle, than the Lindeberg condition, as the following example illustrates.
23.12Theorem < cxo unifonnly
Liapunov ' s con dition holds if snhn > O uniformly in n and in /, 6 > 0.
2+8
A'IX/I
Proof Under the stated conditions, n
1
uX Sn
for all
>1
-
2+ E I.Y,I <
n, where #1 k supe'l
lim Sn2+
follows immediately.
Snc
AI2+8an d 1
n-yx
nBz
n
y7Flxfl -1
<
/1 < x
T-,
(23.38)
B2 < inf n snlln. Then 2+
=
o
(23.39)
.
Note that these conditions imply those of 23.10, by 12.10. It is sufficient to condition in which variances exist but no moments eveh avoid the fractionally higher, provided the sum of those variances is also On4. tknife-edge'
23.3 Feller's Theorem and Asymptotic Negligibility We said above that the Lindeberg condition is sufficient and sometimes also necessary. The following result specifies the side condition which implies ecessity.
23.13 Feller's theorem Let fxnt) be an indepndent sequence with zero mean and Var iance ). If Sn -P-> N, 1) and sequence (cn2f
The Central Limit Theorem
374 max #( IXarl
> E)
0 as n
-->
(23.40)
any e: > 0,
-->
x,
1f t S n
the Lindeberg condition must hold. n The proof of Feller' s theorem is rather fiddly and mechanical, but necessary and it is worth a conditions are rare and difficult to obtain in this theory, little study for that reason alone. Several of the arguments are of the type used
already in the sufticiency part. Proof Since c2nf 0 for evel'y /, the series expansion of the ch.f. suggests that 19xn,(%) 1 l converges to zero for each t. ln fact, we can show that the sum of the squares of these terms converges, and this is the first step in the proof. -->
-
Applying 11.6 for k
l9xa,() 1 I < -
0 and k
=
Ftminlz,
1 respectively,
=
I ,;L,I1) <
we can assert that
2#( 1Ak,1 > E) +
eI11
(23.41)
and
I(l)x (l)
1 I S ftmintz
-
nt
l2A$j) < hlzln'* N
IlAk l
,
(23.42)
In each case the second inequality is by (11.29), setting E = 0 for (23.42). Squaring l9xa,() 1 I adding up over t, and substituting from the inequalities 2 = 1 we o btain remembering E:=1cnf -
,
,
n
:71
j
2
I#x,,,(l).-1 l < max
1f; t < n
J=
<
n
j
jpxa,() j 77l9xa,() -
1
-
/=
2max #( lXntI
>
1S t S n
-->1:13:as
n
1
e) + e I
1I
2
,1 '
(23.43)
-->
x,
using (23.40)and (23.41).
This result is used to show that 9Ja(:) = 117=19xu,() by can be approximated 'rcose-l cz-' (X:=1#xn,(X) if d that 1 1 then I I exp x1 l Note z re an r s using (11.12). Lemma 23.7 can therefore be applied, and for some n large enough, .
n
expX(9xn,(l) >1 -
1)
ie
=
-
n
n
-
-
,
77lexplyxrtl) I-lqlxnrt'l <
-
>1
>1
1)
4x,(l)I
-
M
S
2, 1ttx /1,) 1 l 77 >1
(23.44)
-
n
where the second inequality is an application of 23.8 with z 9xa,(D 1. The condition of the lemma can be satisfied for large enough n according to (23.42) and (23.23).By hypothesis (
-->
,
---)
-
.
.
-
The Classical Central Limit Theorem
exptA-l--, (x--(l)
l)j
log
-l2,
= Xcos ,-1
-
375
hxnt 1) --
(23.45)
-
using (11.12)and (11.13)to get the equality.
Taking the real part of the expansion in (11.24) up to k 2 gives cos x 2 l 2 + k S S for for that 0 0 1, 1 1 zx cos ty-x cos so any x. Fix 6: > 0 and a choose > 2/:, so that the contents of the parentheses on the minorant side below is positive. Then we have =
/
-
2 H.so X-j-
n
?.
XI
E=' tczj J ( IAk/ I>e:)
n
X J((Ik
Xn2dp f t
:1hlxnlt -
I> c)
p.j
n
j
jzyxsj
KX /=
X E tlt,z.;Lz,1 -
+
2 dp
.j
r
Xnt > E) 1 lI l
n
S
=
.x
-
+
cos
cos hxnt)
--
tjt
dp
0,
(23.46)
>1
the last inequality holds since the integrand is positive by construction fOr evel'y Xnt, and the convergence is from (23.45)after substituting Efczn, 1. Since E is arbitrary, the Lindeberg condition must hold according to (23.46).w where
=
negligibility', under which no a condition of whole. summand The chief single may be so influential as to dominate the sum as a ()sst could unless why ---) have the Lindeberg condition, ithout we w reason ,) e (23.40) holds, is that a finite number of summands dominating all the others could happen to be individually Gaussian. The following example illustrates.
Condition
(23.40)is
Easymptotic
-:2/2
23.14 Example Let Xt NnGt) 2 w here c2t 2/ Note that sln E).12' 2?JE)-12-' 2n+1 2 and Xnn Xnlsn N, lnllln+l 2)). Clearly Sn N(0, 1) for every n, by the linearity property of the Gaussian, but condition (23.40)fails. The =
-
=
-
7
-
=
.
=
=()
=
-
-
Lindeberg condition also fails, since n
X j
( 1Xnt I> E l
Xln, dp k --A
Xl dp nn
( IXnnI> E l
F(l Izl>4Is)Z2) > 0 t
(23.47)
where Z is a standard Gaussian variate. u A condition related to P max 1f t f n
(23.40)is
I.L,I >
:
--)
0 as n
-->
x,
any
which says that the largest Xnt converges in probability to zero. 23.15 Theorem
(23.48)implies (23.40).
(23.48)
376
The Central Limit Theorem
(23.48)is
Proof
#
lA',;/IS
the same as Ptmaxlx/un max 1Aul <
:
;)f Ix,,,1;
z'
=
ls/fn
:) :)
>1
min #( j-L,l <
s
1. But
--+
E)
lslfn
=1
max #(
-
1K t < n
I#n,I>
e),
(23.49)
where the inequality is by monotonicity of #. If the tirst member of (23.49) converges to 1, so does the last. . Also, interestingly enough, we have the following.
23.16 Theorem The Lindeberg condition Proof Another way to write
integration) is
implies
(23.48). order of summation and
the (23.17)(interchanging
n
1 N-% x2 > nt .e. ((-r,,l e:)
--->
f,l
() all :
0.
>
,
-
(23 50) .
>1
Accprding to 18.13 this implies n
P
for any
E >
1t Ix X >1
nt
I>c)Xl > :2
.-#-r->
0 as n
-->
0 or equivalently, ,
,
(23.51)
x,
--
0. But notice that
7) 1( lA'af1
0:
Z7.11( jxarjosla'n,2
> ::1
>1
((t))-Y2 nt
((t))> el
=
(o:
max 1
laLf())I>
:
,
(23.52)
so (23.51)is equivalent to (23.48).w Note that the last two results hold generally, and do not impose independence on the sequence. The foregoing theorems establish a network of implications which it may be helpful to summarize symbolically. Let the Lindeberg condition; L independence of the sequence; I AG asymptotic Gaussianity @k( ) e ); , = AN asymptotic negligibility (condition(23.40));and PM max 1XntI -.E(.-y0 (condition(23.48)). Then we have the implications =
=
-2/2
=
--
=
L +1
=
AG +PM
+1
=
AG +AN +1 =
L + 1,
(23.53)
where the tirst implication is the Lindeberg theorem and 23.16, the second is by 23.15, and the third is by the Feller theorem. Under independence, conditions L, AG + PM, and AG + AN are therefore equivalent to one another.
The Classical Central Limit Theorem
377
However, this is not quite the end of the story. The following example shows the possibility of a true CLT operating under asymptotic negligibility, without the Lindeberg condition. with probabilities ,1(1 /-2) each and t and 23.17 Examp le Let Xt z1 and Jn2 ,1/-2 each, so that Ext) 0 and Var(XJ jand W ith probabilities jrl + 0(1). This case has similar characteristics to 23.9. Since l-Y,,rl tsn-1 w jtja 1 probability t -2 an d gsn o therwise, we have for any E > 0 that, whenever n is large enough that Esn > -.t
=
-
-
,
t,
=
=
=
=
-1
,
#( IX// I >
; ns-2
E)
(23.54)
where ne is the smallest integer such that nzsn-' > E. Since nz O(rl1/2) (23.40) holds in this case. However, the argument used in 23.9 shows that the Lindeberg condition is not satisfied. F(1( Ixa,I>:).Y$) sn-l for t k ng, and hence =
&
X
r= 1 l Ivn/I
xltdp z n
l
> E,
n
-
nsn-l
.-+
4 > 5
,
0.
(23.55)
However, consider the random sequence ( W'f), where Gr Xt when ImI ,1 and W'r= 0 otherwise. As t increases, Gf tends to a centred Bernoulli variate with p 1, and defining ll'n, Wtlsn, it is certainly the case that =
=
=
=
n
1#'n/-PX >1
NQ, 15).
(23.56)
is distributed like I'r in 23.9, and applying (23.32)shows that However, lX, W',I lP,1 Op(1), and hence l 7=1Xn,- Z'/=1W'n,l f E7=1IXq:-1IXnt W%,1 -P- 1(0, j), according to 22.18. n O# (n-1/e)tt follows that -
=
*
-
=
xtxnt
A CLT therefore does operate in this case. Feller's theorem is not contradicted because the limit is not the standard Gaussian. The clue to this apparent paradox is that the sequence is not unifonnly square-integrable, having a component which contributes to the variance asymptotically in spite of vanishing in probability. of 1 for every n despite the fact ln these circumstances Sn can have a 15! variance of that its limiting distribution has a tvariance'
23.4 The Case of Trending Variances The Lindeerg-Feller theorems do not imposeuniform integrability or Lr-boundedness conditions, for any r. A trending vatiance sequence, with no uniform bound, is compatible with the Lindeberg condition. lt would be sufficient if, for example, o't2 < c,o for each finite t and the unit-variance sequence tXf IGtj is uniformly square-integrable, provided that the variances do not grow so fast that the largest of them dominates the Cesro sum of the sequence. The following is an extension of the sufficient condition of 23.10. 2
2/ct ) is uniformly integrable, where (c,) is a sequence 23.18 Theorem Suppose (aYf of positive constants. Then fA) satisties the Lindeberg condition; (23.30),if
The Central Limit Theorem
378
2 nMnlsn sup n 2
where Mn
=
C
=
(23.57)
< x,
maxlsfxacr. u
One way to construct the ct might be as maxf 1,c/) The variances of the transfonned sequence are ten bounded by 1, but czf 0 is not rtlled out for some f. .
=
Proof of 23.18 The inequality of &
Ix,I>.c)x:l) 177e-(1( --i
<
u''max
max 1f t f n
of
xictb>sncctbxlcljj
czf
e'(1(
sn 1x,x,,
Sn f= l
lim sup (
(23.33)extends to
te'(
xtlctthj.
1( Ixgctlx,wctl
(23.34)then
(23.58)
gives
= .E'(1(txtlctt:t-snv-lctt.xctql) 0.
(23.59)
.
n-+txz
Notice how (23.57)restricts the growth of the variances whether this be positive 1 > 0 un jsrmjy or negative. Regardless of the choice of (cr),it requires that snln in rl. It permits the ct to grow without limit so long as they are finite for all t, so the variances can do the same; but the rate of increase must not be so rapid as to have a single coordinate dominate the whole sequence. If we 1et ct = ct maxt 1,cf) as above, (23.57)is satisfied (according to 2.27) when c,2 t for any gzf lt. G k 0 but not when ln fact, the conditions of 23.18 are stronger than necessazy in the case of decreasing variances. The variance sequence may actually decline to zero without violating the Lindeberg condition, but in this case it is not possible to state a < a < 0, we would genera1 sufficient condition on the sequence. If cl t with have to replace (23.33)by the condition' -
-
-1
-
n 177F(1( sn -i' 1
Ix,I>xa:)Xl) S
=
1 ymsyasxsFtltjxtjxsasl-vlj,a
(23.60)
egy
2 1+a 2 infnsnln ) > 0 by assumption (note,sn n under independence). cxo is not luled Convergence of the majorant side of (23.60)to zero as n out, depends distribution of the Xt. but on the The following example illustrates both possibilities.
where B
1+G
=
-
-->
23.19 Example Let (X,) be a zero-mean independent sequence with Xt Ilt-ftx,fal lG either growing with f (a > 0) or declining for some rea 1 (x such that c2t = !. with t (a < 0). However, Xt is Ax-bounded for tinite f (see 8.13). The integrals in (23.30)each take the fonn -
,
,
1 zta
tz -t t,,
1(jtI>,ac)( 2tt,
(23.61)
The Classical Central Limit Theorem 2
2a
2a
2a+1
379
1 1 Now, Zz=l'r On where sn !.Zz=1T ) for a > 1 and Otlog n) for a (2.27). Condition (23.57)is satisfied when (x 0. Note that (23.61)is zero if (Yw1T gCOWS jaster than n for all G n 2a)1n E, > ta. ( nwl 0, and hence (23.61) vanishes in the limit for every t in these cases, and the Lindeberg &(- g t1, p,n grows at tjae same ra te as condition is satisfied. But if Xt n 2231/2 the above argument does nOt apply, and the Lindeberg condition (Zw1 fails. Note how condition (23.57)is violated in this case. However, the fact that condition (23.57)is not necessal'y is evident from the ltX) fact that the variance sum diverges at a positive rate when Xt ug-ftx for any even though the variance sequence itself goes to zero. lt can be verified (x that (23.61)vanishes in the limit, and accordingly the Lindeberg conditiop holds, 2 for these cases too. On the other hand, if (x < 1 sn i s bounded in the limit and (23.17) becomes n
=
n
=
.
=
-r
-z
,r2a)1/2
,
,2
-
'
-
1
-
-,,
:3
''
lim:7 a/a -n-yx >1
tc'
1
-.t
1( ItI>4s,x)t2d,
(23.62)
(,
wheres = limn-yxzkl/ 2 < x, an db y choice of small enoughy, (23.62)canbe made arbitrarily close to 1. This is the other extreme at which the Lindeberg condition fails. n
CLTS
24 for Dependent Processes
24. 1 A General Convergence Theorem The results of this chapter are derived from the following fundamental theorem, due to Mctaeish (1974).
24.1 Theorem Let (Z,,j, i = 1,...,rn, n e IN) denote a zero-mean stochastic array, where rn is a positive, increasing integer-valued function of n, and 1et25 rn
Trn
=
11(1i
i),
+
>
(24. 1)
0.
f=1
,--OD Z%1Z,,f /40, 1) if the following conditions hold: I uniformly integrable, (a) Fra is =, 1 as n (b) E lhj
Then ' S ra
=
-->
-->
ra
ni z2 (c) X =1
.TI.A
1 as n
(d)maxlxjsrnlzsjl
-->
.-E(-> 0
x,
as n
-->
c.,.
n
There are a number of features requiring explanation here, regarding both the theorem and the way it has been expressed. This is a generic result in which the elements of the array need not be data points in the conventional way, so that theirnumberrn does notalways correspond withthenumberof sampleobservations, n. rn = n is a leading case, but see 24.6 for another possibility. lt is interesting to note that the Lindeberg condition is not imposed in 24.1, nor is anything specific assumed about the dependence of the sequence. Condition 24.1(d) is condition PM defined in (23.48),and by 23.16 it follows from the Lindeberg condition. We noted in (23.53)that under independence the condition is equivalent to the Lindeberg condition in eases whete (aswe shall prove here) the central lirnit theorem holds. But without idependence, conditions 24.1(a)-(d) need not imply the Lindeberg condition. Proof of 24.1 Consider the series expansion of the logarithmic function, defined for 1x1< 1,
lOg (1 + x )
=
x
-
/
2
3 + 1x
A
-
...
Although a complex number does not possess a unique logarithm, the arithmetic identity obtained by taking the exponential of both sides of this equation is well-detined whep x is complex. The formula yields
for Dependent
CLTS
1 + ihzni
381
,zzazj
Qjlexp (
expt
=
Processes +
where the remainder satisfies I6x) I f Ixl 3 for for i 1,...,rn yields
rihzni)
1x1<
j,
(24.2)
1. Multiplying up the terms
=
exp (ih'rn )
TrnUrn,
=
where Fra is detined in (24.1) and rn
rn
Ur
exp
=
n
1k2
-
r
z2 ni
rtjz
-
i= 1
n
(24.3)
j) .
f=1
Taking expectations produces
(srnk)
=
ETrnurn)
-:2/2
e- klnEl-rn)
=
-
so given condit ion (b) of the theorem, t1).rn (D
I
lim E Trnurn n-jx
e-khl)
-
)),
F(Frn(&rn e
+
-->
(24.4)
e-hlll if
j
(24.5)
The sequence Trnurn
-
e-
-:2/2
jhl)
=
exp fihsrn l
(24.6)
Tne
-
is unifonnly integrable in view of condition (a), the first term on the right-hand side having unit modulus. So in view of 18.14, it suffices to show that plim Fr
urn -
n
n.-/co
e
-1
2/2
)
0.
=
(24.7)
0p(1), the problem reduces, by 18.12, to showing e -klll an d for this in turn it suffices, by condition (c), if
Since Trn is clearly
plimn-pxt/o
=
that
,
rn
rihzn
plim
=
0.
(24.8)
l->:xa f=1
To show that this convergence rn
rn
77
rihzn
zz:1
obtains, we have by the triangle inequality
f
377
I l
1 izzz
3
IzrlfI s
The result now follows from conditions
rn
/
max lzniI 77zlni. l ,I $. 3
'
1f i f n
(c) and (d), and
(24.9)
f=1
18.12. w
lt is instructive to compare this proof with that of the Lindeberg theorem. A different series approximation of the ch.f. is used, and the assumption from independence, that
(b- flyzn, =
#n
is avoided. Of course, we have yet to show that conditions 24.1(a) and 24.1(b) hold under convenient and plausible assumptions about the sequence. The rest of
382
The Central Limit Theorem
this chapter is devoted to this question. M.1(b) will turn out to result from suitable restrictions on the dependence. M.1(a) can be shown to follow from a more primitive moment condition by an argument bsed on the sequences' idea. Kequivalent
24.2 Theorem For an array fZni), 1et
2i
=
znjltzkf--lznzl s 2).
Tra
=
+ iLZ-nf ) is H:1(1 ,
n
(i)
The sequence
2 E max Zni sup n j
<
(24.10)
unifonnly integrable if
(:,4
x.
j < rs
.
j j)
-.E(..y 1, then And if Etnlzzf f n G
,-2
.. (11)
Znf
pr
1;
f=1 rn
*
(iii) Sr.
N
=
.zi
.
-
has the same limiting distribution as Srn.
i'=1
Proof Let Jn
=
min
:
lzn2j E./f-
2)
>
,
0, if at all, from the point i
=
- I2 ITrn The terms
,
rn
=
1zn2j
E(I2
2
>
Jn + 1 onwards. Note that
=
Jn
j + :2z2 ) 11( szz- 1-1( j
2 ni4
+
=
ni
(24. 13)
.
f=1
f=1
2z2 nf are rea 1 and positive. The inequality 1 +x I'LeAf
that 17j(1+xij
for xi
S
(24.12)
otherwise
rn, such that Zni
if
>
< ex
for x > 0 implies
0. Hence,
Jn 1 -
2
ITrn I
=
f
(1 + :2z2ni )() + hlz j''j =1
n
)
exp (VV2-1
j (1 + klz.,h) 1Xj)
< e 22(1+
,
2Z,,Jnl, 2
(24.14)
where the last inequality is by definition of Jn. Then by (24.11), sup FI TrnI2 f e
22
(1+
2
2
sup Ezunjj
<
x.
(24.15)
Uniform boundedness of Sl /raj2 is suftkient for unifonn integrability of proving (i). Since by construction E;21 ni ,
lk,
CLTS
l
# Xi Z-ni
X Z2ni
#
for Dependent
Xi Zn2j>
#
=
i
j
by assumption,
which proves
-
Pzni y,
kni,some
--->
(ii). In
1 < i < rn)
383
2
z2aj1 < P 7) i --) 0 as n
Processes
j
> :
,
for 0 < s < 1,
(24. 16)
x,
addition,
.
Pklr Izn2: > 2)
=
--)
0 as n
--->
x,
-.E(-> 0, and by 22.18, j'n and Sn have the same limiting distribution, so Ilk Sn I proving (iii). . -
24.2 The Mmingale
Case
Although it pennits a law of large numbers, uncorrelatedness is not a strong enough assumption to yield a central limit result. But the martingale difference assumption is similar to uncorrelatedness for practical purposes, and is attractive in other ways too. The next theorem shows how 24.1 applies to this case. be a martingale difference array with finite uncondi24.3 Theorem Let tXnf,8Lrl 1. If tional variances (c$) and XT=IIU, =
,
77-Y,$
...r...y
1, and
(a)>1
(b)maxlsrxnlAkl thensn
Xk,
=
.---:D
-r..+
0,
x(0,1).
>1
Proof We use 24.1 and 24.2, setting rn n, i t, and Zni = Xnt. Conditions (a) and (b) are the same as (c) and (d) of 24.1, so it remains to show that the other conditions of 24.1 are satisfied; not actually by Xnt, but by an equivalent sequence in the sense of M.2(iii). If Tn 17:.1(1+ ihxnt), we show that limn-pxErn) 1 when (Xnf J is a m.d. multiplying out, an-ay. By repeated =
=
=
=
n
Tn
=
ihxnt) 1j41 >1 +
=
ZZ
Fn-l
Ft-
1
H'-141 ikx ?t .=1
+
) is
an
ihTn-qxnn
...
=1 =
+
+
ay-lzk-lxsf. ,.l
Tf-l-measurable r.v., so by the LIE,
(24. 17)
384
'he Central Limit lheorem
/1
Evn)
=
1+
ikTEvt-kxnt) >1 n
=
1 + fl,y7F(z',-IF(xn,l
@s,,-l))
>1
1.
=
)
(24. 18)
This is an exact result for any n, so certainly holds in the limit. If Xnt is a m.d., so is knt Xnt1(Zl=-lX2k f 2), and this satisfies 24.1(b) as above, and certainly also 24.1(d) according to condition (b)of the theorem. Since l 3EXlnt) 1, condition (24.11)holds for Xnt. Hence, knt satisfies 24.1(a) and 24.1(c) according to 24.2(1) and (ii), and so obeys the CLT. The theorem now follows by 24.2(iii). . =
=
=
This theorem holds for independent sequences as a special case of m.d. sequences, but the conditions are slightly stronger than those of the Lindeberg theorem. Underindependence,we know by (23.53)thatM.3tb) is equivalenttothe Lindeberg condition when the CLT holds. However, M.3(a) is notAconsequence of the Lindeberg condition. For the purpose of discussion, assume that Xnt = Xlsn, where sn2 2 an d cf2 EXl)t Under independence, a sufficient extra condition for X7=1cf 2/Gt2 ) be uniformly integrable. In this case, inde24.3(a) is that the sequence (-Y, pendence of (Xr1 implies in dependence of (.Y2,), (-Y2, g2,, @,j is a m.d., and 19.8 o2f)..T-(.y0. (put an = sn2 an d b t = s2) t gives sufficient conditions for This is equivalent to 24.3(a). But of course, it is not the case that (Xl czf @,) is a m.d. merely because tatsFf l is a m.d. lf 24.3(a) cannot be imposed in any other manner, we should have to require Exlt j@r-1) = o'zfa significant strengthening of the assumptions. On the other hand, the theorem does not rule out trending variances. Following the approach of j23.5, we can obtain such a case as follows. =
=
.
-
Js-2E).j(V -
-
,
,
24 4 Theorem lf (.&,?h)is a square-integrable m.d. sequence and E(XltI@,-j) = 0'2, 2 a.s., and there exists a sequence of positive constants fcf) such that fXf2ycf ) js unifonnly integrable and @
2 sup nMnlsn n 2
where Mn2
=
<
(:4
x
2 maxlxruncr, conditions 24.34a) and 24.3*)
hold for Xnt
=
.
1q)
Xtlsn.
Proof By 23.18 the sequence (.L,) satisfies the Lindeberg condition, and hence, 24.3(b) holds by 23.16. Note that neither of thes results imposes restrictions on the dependence of the sequence. To get 24.3(a), apply 19.8 to the m.d. sequence cl, and t4, c2)/c2j (Xt2 o'2 The sequence is 1, bt pu tting p t t an 2 2 assumptions, and note that kl=kbt; nMn Osn) and uniformly integrable on the t?(u4), nsfj S hat both lcsblt t as consequences of (24.19).The conditions of 19.8 are therefore satisfied, and the required result follows by 19.9. w =
-
=
=
((V -
.
=
=
CLTS
for Dependent
Processes
385
24.3 Stationary Ergodic Sequences It is easy to see that any stationary ergodic mmingale difference having finite variance satisfies the conditions of 24.3. Under stationarity, finite variance is sufficient for the Lindeberg condition, which ensures 24.3*) by 23.16, and 24.3(a) follows from the ergodicity by 13.12. The interest of this case stems from the following result, atibuted by Hall and Heyde (1980: 137) to unpublished work of M. 1. Gordin.
24.5 Theorem Let (A,@r) be a stationary ergodic Al-mixingale of size Sn Z:=IX, assume that
and if
-1,
=
limsup n -1/2FIs n j
(g4.z())
<
x
>
0 n-1IlSn .-2-.I #(0 ;1zG2) u
.
N.->oo
Then n -1/2 E lSn 1
2. 0 f 2. <
-->
,
oo.
If
,
,
,
.
Notice that the assumptions for this CLT do not include EXj)2 XBt
YBt
X'Bt
<
x.
ECXBt
.Y,1(Ix,Ixs). The centred sequence I1,-1) isJ'Pa n-llll 2#2 and hence t stationaryergodic m.d. with bounded variance cs2 < -P-->N cs2), by 24.3. Further, the m.d. orthogonality property implies Proof Let
=
=
-
.1
,
1
n
F s-1/2y-q
YB t
l =
cs2
(24.21)
.
>1
E'/=1F( l is therefore uniformly integrable (12.11),and by The sequence (n-1/21 J the continuous mapping theorem it converges in distribution to the half-Gaussian limit(see 8.16)., hence, by 9.8, n
-1/2E
B
X Ft
GB
-->
wa)1/2
(24.22)
.
>1
F/-1), corresponding to 'Bt for B x, and apply the Xt F(.&I Ff+ Zt 4+1, where F, is a stationary decomposition of 16.6 to write Xt ergodic m.d. and Zt is stationary with FIZ1 I < oo. Hence Sn Z:=1l'r + zl zn+1, and by (24.20)there exists A < x such that Now define Ff
=
=
-
=
-
=
limsup n -1/2E
X l't
=
>1
n--yx
jjmsup nIlE
j
s%
-
n
-
j zj + zu4.j
n--x
s
limsup
,,-''2(e'1
-sr,1
+
N-)oo
2FIz1I) <
4.
(24.23)
(cs2,B 1,2,...) is a monotone sequence convergNoting that cs2 J(Ixjluslxzjlr ing either to a finite limit c (say) or to 1/2+=. In the latter case, in view of > A for N n, which (24.22)there would exist N such that n - FI17=1FrI x. x < 2 conclude that king in T B we can a contradicts(24.23),so (24.22) c 1/2 0, yields, in view of the fact that n - FIz1 Zn+1I =
=
=
-
--
The Central Limit Theorem
386
FISn 1 n-1/2
=
?1-1/2F
l
y-')J' + zj zux.j
-.-y
-
,
c(2/a)1/2
(24.24)
.
,--1
1/2 Since I is now known to be a stationary ergodic m.d. with c(2/a) c2, variance and c2 > 0, 24.3 and 22.18 imply that n-1/25'n --P--yN(O,c2), finite whichcompletes the proof. w
Hence, l
=
.
This result can be thought of as the counterpart for dependent sequences of the Lindeberg-l-evy theorem, but unlike that case, we do not have to assume explicitly 2 < x. I n de p endence of (Xf) enforces the condition Zt 0 for al1 t, and that '(X1) nl. It might appear that the then the conditions of the theorem imply required for the CLT, but existence of dependence weakens the momentrestrictions this gain is more technical than real, for it is not obvious how to construct a stationary sequence such that Xl is not square-integrable but l'1 is. 'I'he most useful implication is that the independence assumption can be replaced by arbithemixingale assumption) without weakening trary localdependence (controlledby any of the conclusions of the Lindeberg-Levy theorem. =
EXI) =
24.4 The CLT for NED Functions of Strong Mixing Processes The traditional approach to the problem of general dependence is the so-called method of tBernstein sums' (Bernstein 1927). That is, break up Sn into blocks (partial sums), and consider the sequence of blocks. Each block must be so large, relative to the rate at which the memory of the sequence decays, that the degree to which the next block can be predicted from current information is negligible; but at the same time, the number of blocks must increase with n so that a CLT argument can be applied to this derived sequence. It would suffice to require the sequence of blocks to approach independence in the limit, but a result can also be obtained if it behaves asymptotically like a martingale difference. This is the approach we adopt. The theorem weprove (fromDavidson 1992, 1993b) is given in two versions. The first, 24.6, is fully general but the conditions are complicated and not very intuitive. The second, 24.7, is a special case, whose conditions are simpler but cover almost al1 the possibilities. The excptional cases for which 24.6 is essential are those in which the variances of the process tend to 0 as t increases.
24.6 Theorem Let tXnf, t 1,...,n, n 1 ) be a triangular stochastic array, let < t < x, n k 1 be a (possiblyvector-valued) stochastic array, and let ( XFt, ) 'l-m c(F,u, t m S s S t + m). Also, let Sn Xlcjxnt.Suppose the following 5nt, assumptions hold: t kf-measurable, with Exnt) 0 and ESn1) 1. (a) Xnt is Fn,-x (b) There exists a positive constant array (curl such that supngllAk/cnrllr < cxa for r > 2. (c) Xnt is Q-NED of size on (J&fl with respect to the constants lcnf)speciand + 20)r/(r 2), 0 S 0 < 1a. fied in (b), (Fpgl is a-mixing of size =
-c.,
=
=
-
,
=
=
-1
-(1
-
for Dependent
CLTS
(d) Letting bn
=
1-a
grl 1 and
rn
Processes
for Lnlbnj
=
cnrl for maxi-llbn
some
(x
1,...,rn and
=
1f
obn-
=
1/2
rn+ 1
rn+ 1
Mni
(0,11,and
defining Mni
sin,rn..j
(24.25)
),
()-.1/2
Olbn
=
(24.26)
),
izz1
=
maxrnu-crsnfcn/),
=
the following conditions hold: Mni max K
e
387
where 0 is given in (c), and rn+l 2
Mni
-1
=
Obn
f=l
Then,
sn--P-
(24.27)
).
N(0,1). n
24.7 Corollary The conclusion of 24.6 holds if assumptions (c) and (d) are replaced by of size (c') Xnt is AZ-NED of size 2). on ( J&,), which is G-mixing caf), (d') Letting Mn maxlxrxa ( -r/(r
-1
-
=
nMn2 < sup N
x.
(:,4 g8)
I:a
.
Note that (c) ' i s just 24 6(c) with 0 0. If (24.28)holds, Mni 0(n-1/2) for each i. Since rnbn n, (24.25)and (24.27)hold in this case for any (x in (0,12. While (24.26)holds only for a strictly positive choice of 0, with 0 < a S 20/(20 + 1), 24.7(c') entails satisfaction of 24.6(c) for some 0 > 0. Hence, 24.6 contains 24.7 as a special case. The adaptation assumption in 24.6(a) will be needed because of the asymptotic m.d. property the Bernstein blocks must possess', see 24.19 below for the application. This assumption says that Xntmust not depend onfuture values of the underlying mixing process, F,u for s > t. In econometric applications at least, such an assumption would typically be innocuous. The remaining parts of condition (a) specify the assumed normalization. The roles of the remaining conditions are not particularly transparent in either version of the result, and in reviewing these it will be helpful to keep in mind the leading case with Xnt (1$ plsn where n F(J=1(Ff w))2,although more general interpretations are possible, as noted in j23.2. ln this case it would often be legitimate to choose =
.
=
-
=
=
-
cnt = maxtcsl
)/Jn,
-
(24.29)
whereo't2 is the variance of 1$. The cnt have to be thought of as tending to zero or withn, although possibly growing or shrinking with t also, subject to 24.6(d) J.2 ruled is of not B is tocorrelation out, the sequence '). ecause au no 24.74d variances, partial of but the the defined is just as sum longer
The Central Limit Theorem
388
n
where okt-k
c2,+ 2:7
=X ,=1
=
f
n
-
l
X wt-k,
,-2
(24.30)
t--l
Cov(J',,l',-1).
Assumptions 24.6(c) or 24.7(c') imply, by the norlp inequality, that IIXn, Exnt6 -
for 0 < p < 2, where v,u is consequenceof 17.6.
S cnfvm, ?L'+, T-,,,)lIp
of size
(24.31)
The following lemma is an immediate
-1.
isanfw-mixingale z4.8l-zemmaunderassumptions 24.6(a), (b),and (c),(Xnr,?Lf,-x) + size 20)(r-#)/p(rwith S 1, 2)) for 1 2, Kp of constants (cnfl.n (1 -mint
-(
r/(r 1) the size is + 0), and when p 2 the size is In particular, when p apply, conclusions the assumptions 24.7 Under of the same -1. except that 0 0. There is also a more subtle implication which happeps to be very convenient for the present result. This is the following. =
=
-
=
24.9 Lemma Under 24.6(a) and (b), plus either 24.6(c) or 24.7(c'), F-'--''') cnt/ul S cntcn,t-vklm Fl E(XntXn,t+k3
(24.32)
-
and
FIX nt Xn,f+ k Exntxntnb F-'-++''')l S cntcn,t-vm..k
(24.33)
-
for each k e !N, where nnts
=
Exntxnsj, and
(??: =
Om -1-
)
:
j Or
>
() u .
These inequalities are convenient for the subsequent analysis, but in effect the lemma asserts that for each fixed k the products Xntxnvtwk,after centring, form 2 2 with constants given by maxtcnl,cn,-ll One of these fal-mixipgales of size Unt,snt,-xt, where might be written as, say, ( -1
.
Unt
=
Xn-t-fklzxn,t-vk-tuzjc,,,,-(w2j,,+k.-Ep?2j.
The mixingale coefficients here are
for m
>
(/c/21.
-
(p,' V for =
m
=
0,...,k/2), and
(,,,' (p,-w2J =
by 17.11. The Proof of 24.9 The array (Xntxnvtwklis AI-NED on ( Fnfl of size conclusion then follows by 17.6(i), noting that any constant factors generated in forming the inequalities have been absorbed into (,, in (24.32)and (24.33).. -1
Npw consider M.6(d) and 24.7(d'). These assumptions permit global nonstationarity (see j13.2). This is a fact worthy of emphasis, because in the mctional CLT (seeChapters 27 and 29 for details) global stationarity is a requirement. ln this respect the ordinary CLT is notably unrestrictive, provided we normalize by sp as we do here. The following cases illustrate what is allowed.
24.10 Example Let (F,Jbe an independent sequence with variances (U t for any 13k 0 (compare13.6). It is straightforward to verify that assumption 24.7(d') is -
for Dependent
CLTS
389
Processes
It is Z)=1(U. satisfied for Xnt i'f lsn where cnt cf lsn, and in this case however violated when (U lt, a case that is incompatible with the asymptotic negligibility of individual summands (compare23.19). lt is also violated when j3< 0 (see below). (n .,:2
=
=
=
-
24.11 Example Let ( i',) be an independent sequence with variance sequence generated by the scheme described in 13.7. Putting Xnt l', lsn, 24.7(d') is satistied with cnt 1/Jn for a11 t. u =
=
Among the cases that 24.7(d') rules out are asymptotically degenerate sequences, x, x. but 041) as n 0 as t In these cases, maxlslsacl having cl snlln certain will of 24.7/ it 0. It is usually ' ), given these be the case that assumption allow. is what going that 24.6(d) To is designed to on here, see cases regularly with it is easiest to think in terms of the array (cnf) as varying n and t, within certain limits to be determined. We have the following lemma, whose conditions are somewhat more general than we require, but are easily specialized. ..-y
-->
-->
=
--+
2 24.12 Lemma Suppose cnt
-
fG-Y-1
j3y
for
,
R Then, 24.6(d) holds iff
G
.
13f Y
and
;$ < 2g0 +y(1
13and
Notice that
(24.34) 0). ca
+
(24.35)
y can be of either sign, subject to the lndicated constraints.
Proof We establish that an a e (0, 11exists such that each of conditions (24.25)2 (ibn )V-Y-1 for j k 0, or Mnli (24.27)are satisfied. We have either Mni for I < 0, but in both cases i 1):n) fsn -..
-
-:-1
-
rn
l
Mni
-
1+pyf$
rn
1 (1+;$)1.(1-(y,)IJ-'y-
-m1
nn
n
-
.
(24.36)
izu1
Simplifying shows that condition (24.34)is necessary and sufticient for independently of the value of a, note. Next, (24.25)is equivalent to
2 max cnt
a-1
on
=
lfrsn
which (sincethe maximum is at t requirement
maxlp,ol In view of
(24.34)this
-
'/
<
<
rn
f=1
-
rl+;V2yfV2 n
n
n
t
=
1 otherwise) imposes the
(24.38) '/
<
0, and is equivalent to just
(24.39)
a.
Also, M ni
.
a.
constraint only binds if
-y
(:,4 Ly)
1)2 0, and
n for
=
),
(24.27),
-('y+1)/2
The Central Limit Theorem
390
G(1+j/2)+((l
-N
-u)k-y-
1)/2
(24.40)
,
and (24.26)reduces to
2: +y- jl
a f
1+20
(24.41)
.
l
The existence of (x satisfying (24.41)and (24.39)requires two strict inequalities and 0, but since 0 is positive the first of these, wlich to be satistied by by > (fJ is 0 $, holds (24.34).The second is -F1 + 20) < 20 + y- p, which is the same as (24.35). lt follows that (24.34)and (24.36) are equivalent to 24.6(d), as asserted. . 'y,
,
-
ln terms of the leading case with Xnj (l'f bjlsn, with sn2 given j)y (:4 ao consider first the case cl /, j k 0. To have cnt monotone in t (not essential, but analytically convenient), we could often set =
-
.
,
-
(24.42) max (cx klsn. ix st l under 24.6*) and 24.7(c'), the conditions of 17.7 are satisfied. In this case, Note however that 11 l'r wllr c, is required by 24.6(b). Since ((1)is of size we have, substituting into (24.30), cnt
=
-1
-
-
n
t 1
n
-
+ 2X//2 2 7'2fl5 7) (f
Sn <<
k)F/2(
-
>1
=
()(n1+@).
(24.43)
k=1
>1
This only provides an upper bound, and condition 24.7(c') alone does not exclude a1+Y sl for some < p.However, compliance with condition the poss ibility that n 2 2 (24.34), which follows in tur from M.7(d ' ) (note, Mn cnn in this case), This condition says that the variance of the sum must grow no more enforces X= f'1. slowly than the sum of the vmiances. Now consider the case cf2 t' with 13< 0. Here, we might be able to set 'y
-
=
-
cnt
=
suptcxl/-n.
(24.44)
st
c2,
d (24 34) we would again have n under24.7(c ') an f* < for hence Mnl a-I5-1 and 24.7(d') c2./5.2 some n .
t
=;
-
y
with ('J = y, condition (24.35)reduces to
-
t G-p-1but here Mln = ceases to hold. However, ,
f5> 1-2e +ae,
(24.45)
and it is possible for the conditions of 24.12 to be satisfied, although only with 0 > 0. As 0 is increased, limiting the dependence, a can be increased according to (24.36) and this allows smaller p, increasing the permitted rate of degeneration. As 0 approaches z1,so that the mixing size approaches 2), ;$may approach -17, with (x also approaching These conclusions are summarized in the following corollary. Pal't (i) is a case -2r/(r
-
.
CLTS
of 24.7,
(ii) a
and pal't
for Depettdent
Processes
391
case of 24.6.
rI1+ Let Xnt (l'f ptllsn where cl If either t 15and u (i) 0 ; I < x, and 24.6(b) and 24.7(c') hold with cnt detined in (24.42);or (ii) there exists 0 such that (24.45),24.6(b) and 24.6(c) hold with cnt defined by (24.44); then Sn .-P->N0, 1). (:I
24 13 Corollary
=
@
-
-
-
By an apparent artefact of the proof, t -1/2 represents a jjmit on the permitted rate of degeneration of the variances. We may conjecture that with mixing sizes but a different method of exceeding 2r/(r 2), the CLT can hold with 13f attack would be necessary to obtain this result. Also, both the above cases appear to require that (x e (0,r1),and it is not clear whether larger values of (x might be possible generally. The plausibility of both these conjectures is strengthened by the existence of the following special case. -,1,
-
24.14 Corollary If conditions 24.6(a) and (b) hold, and also (c'') (aYn t,5nt ) is a martingale difference array, and (d'') conditions (24.25)and (24.27)hold with bn 1, then Sn -P- N(0,1). n -.x
,
=
Jn2 E7=jo'2f the case Xnt (Ff (Qlsn,where by the m.d. assumption note that (24.27)is satisfied with bn 1 by constnlction, and (24.25)requires only / is permitted for any 1$ under 2.14(d''). This that sln x, so that result may be compared with 24.4, whose conditions it extends rather as 24.6 extends 24.7. The proof will be given below after the intermediate results for the proof of 24.6 have been established. As an example, 1et Fr g,rbe a m.d. with nt2 nllt where c2 is constant, so that Jn2 log n. Corollary 24.14 establishes that = ln
=
=
-
,
=
(U
'1-
-1
-
-
-
(log n4 -1/2)
(F t
.1
w)-P-+ N,
-
c2).
The limit on the permissable rate of degeneration here is set by the requirement
x. I f the variances are summable, the central limit theorem surely sn2 cxa as n fails. Here is a well-known case where the non-summability condition is violated. --y
-->
24.15 Example Consider the sequence ( Ff) of first differences, with F1 Z1 and =
Ff
=
Zt Zt- 1 t -
,
=
2,3,
...,
where (4) is azero-mean, unifonnly Lr-bounded, independentsequence, with r> 2. Here f Ff) satisfies 24.6(a)-(c), but E:=1F, Zn and sn2 V ar (Z n ) /41). n =
=
=
24.5 Proving the CLT by the Bernstein Blocling Method We shall prove only 24.6, the arguments being at worst a mild extension of those required for 24.7. We show that with a suitable Bernstein-type blocking scheme the blocks will satisfy the conditions of 24.1. In effect, we show thay the blocks behave like martingale differences. In most applications of the Bernstein approch alternating big and small blocks are defined, with the small blocks containing of
The Central Limit Theorem
392
the order of (/l15tl-t* 1summan ds for some ;'$e (0,1),small enough that their omission is negligible in the limit but increasing with n so that the big blpcks are asymptotically independent. Our martingale difference approximation method has the advantage that the small blocks can be dispensed with. Define bn and rn aj in condition 24.6(d) for some Zni
ibn Xntni
77
=
e
(0,1),and
(24.46)
1....,rn,
=
,-(f-l)!u+1
such that Sn
n =
let
rn
Xnt
Zn
=
f=1
,..1
+
+ + Xnn). xn,rnbn-vb
(24.47)
...
The final fragment has fewer than bn tenns, and is asymptotically negligible in 1. the sense that bnrnln Our method is to show the existence of (x > 0 for which an array lZnj) can be constructed according to (24.46),such that 24.1(c) is satisfied, and such that the truncated sequence (Ql defined in (24.10)satisfies the other conditions of 24.1. This will be sufficient to prove the theorem, since then E;212nf-T-r.->1 according to 24.2(ii), and Zni and 2nfare equivalent sequences when 24.1(c) holds, by 24.2(iii). according to M.8, we may apply Since lA%rl is a fmmixingale array of size 16.14 to establish that the sequences lmaxy(/-1):n+1Xn/)2*2uj, j n e EN (24.48) -->
-,1
l ib 2 At least this follows Zf=yf-lllpa+lcnl. are unifonnly integrable, where vnf 1, and it is also clear that the result generalizes to directly in the case i for, although the starting point i 1):n + 1 is increasing with n, the any of the sequence in (24.48)can be embedded in a sequence with fixed coordinate nth starting point which is uniformly integrable by 16.14. This holds uniformly in n l lvlni, 1 S i S r ,,, ra k 1 ) and f, and it follows in particular that the array (Znj is uniformly integrable. This result leads to the following theorem. =
,
=
-
,
24.16 Theorem condition.
Under assumptions M.6(a)-(d), (Znf) satisfies 2
the Lindeberg
1 S bnMni cxl by 0 as n Proof For any i, vnj (24.25)and hence, for : > 0, F(1( tzmifvnit>gvniizlni) E (1(1z,,j1>s)Z2,jj) f max max 2 dhfn2 Lsisrn lsifura vnf -->
--
--> 0 as n by unifonn integrability. The conclusion,
--)
(>a
(24.49)
for Dependent
CLTS
ftlt Iz.fI>i:)Z1)
rn
lzxI>s)Zn$) f
f(1t j=1
393
Processes
max
bnsl'ni 2
lsxra
-- 0 as n
ra
77bnulnj ./.l
(24.50)
-->
x,
now follows since the sum of rn terms on the majorant side is 0(1) from (24.27).. This theorem leads in turn to two further crucial results. First, by way of 23.16 we know that condition 24.1(d) is satistied (and if this holds for Zni it is certainly true for Zj);and second, note that for any e: > 0 rn
2 f max Zni
2
2
1/ i K rn
(24.51)
1( Izwl>i:)Znj.
E + f=1
Taking expectations of both sides of this inequality, and then the limit as n x, shpws that (24.11) holds, and therefore that %nis uniformly integrable (that is, condition 24.1(a) holds) by 24.2(i). This leaves two results yet to be proven, Ehnj to 24.1(b) for the truncated array), and 24.1(c). By -- 1 (corresponding result latter shall establish the pal'ts (ii) and (iii) of 24.2, and then in we view of 24.2(iii) the proof will be complete. We tackle 24.1(c) first. Consider -..-h
rn
ibn
rn
2 1 Zni f=1
2
Xnt
=
-
=1
1
-
=
(24.52)
Au Bn, -
r=(f-1)&+1
say, where rn
An
znli Eznlil) 77 f=l
=
-
ibn
rn = and B
=
ibn-t
2 (Xnt
-
f=1 #'
2 Gntj +
2
r=(f-l)>+1 +
(Xntxn,t.k.kGnt,t-
,
(24.53)
k=l, t
#'' where ibn
ra
B'n
=
f=1 n-1
B'n'
=
n-t
2
X t=rnbn-k
f=t-
1)!u+1 kzzzibn-t-j
c2nr+ 2
n-t
c nr, 1+k
cn,,,+1 + X &1
,
cznn.
(24.54)
(24.55)
+ c2su Here, Gnts Exntxns), and recall that ESnl4 )=-l(c2r & + zlzg-vjototy) 1. It may be helpful to visualize these expressions as made up of elements from outer product and covariance matrices divided into n blocks of dimension bn x bn, with a border corresponding to the final n rnbn terms, if any; see Fig. 24.1. =
=
-
=
Ihe
394
Central Limit Theorem
The terms in An correspond to the rn diagonal blocks, and Bn' and B'n'contain the remaining covariances, those from the off-diagonal and final blocks.
1
2
3
.
.
rn
.
b l //z (!) n
OZZ
2
z'
/?,,
Fz
3
!
:
l
:
:
(
l
l
!
y yy /,,,'''z. '-jy-'t
r
Bn :
2
:
)
I
I I
.
g y Fig. 24.1
An is stochastic and so must be shown to converge in probability, whereas the components of Bn are nonstochastic. The nonstochastic part of the problem is the more straightforward, so we start with this.
24.17 Theorem Under assumptions 24.6(a)Vd),Bn Proof Since r
>
2, r/tr
1) < 2 and conditions
-
InntsI f where ((,u) is a constant
(c) imply
by 17.7 that
(24.56)
,
-1.
Hence,
bn
77 7777
fw+1
=1
bn
24.6(b) and
cngyxljf-.sl
sequence of size
rn ra+ l
lBn'I f 277
0.
-->
n.i-jlbn-vj,q-tlbn-vk
1=l
./=1
rn rn+ 1
bn
sinisl'nl << l=f+l j=$ =1
bn
al
j j t-ijbn-vk-jI
(24.57)
.
To detennine the order of magnitude of the majorant expression, verify that the + ) and some tenns in the parentheses are Obn1- l ) ) j or j j + j order of summation and putting k l i allows us to write 5 > 0. Changing the -1-
...yyk
=
-
,
=
rn+ 1
j
-
-.k
1-
lBnz I << bn
rn
-1-
MniMn,i-vkk
al
But for every k in this sum, the Cauchy-schwartz ra+ 1
-.k
rn+ 1
77l MniMn,i..kK XMS izz1 =
;r
.
(24.58)
=1
inequality and
&(:-n'),
(24.27)give (24.59)
for Dependent
CLTS
'1 I#n
Ob-h
=
n
395
that
as required,
so that (24.58)implies,
Processes
O(rl-Y1-*
=
(24.60)
).
To complete the proof we can also show, by a similar kind of argumentbut applying (24.25), that /(1). OMl a,ra+1 b . Bn'' (24.61) a) =
=
2 Ezni)2 in To solve the stochastic part of the problem, decompose the terms Zni (24.53) into individual mixingale components, each indexed by i 1,...,rn. For a pair of positive integers j and k, 1et -
=
1PnjU,k)Xn,(-Ipa+./X,,,(f-I)!u+.j+k =
It is an immediate consequence of 24.9, specifically of for fixed j and k the triangular array
( Yij,kl,
FafZx;
=
=
Substituting from
=
=
k) is an Al-mixingale of size (?-l)u+jfor p k 1, and constants
bn
-
-
+
,
cn, (sl)?u+./c,,,
bn 1 =
with
(24.64)
i-tlbn-qn.
have the sum of bn2 suc h tnixingales,
-.j
X JLj(#,0)+ 277 j=1
(24.63) -1,
(24.62)into (24.53),we
Zlni Ezlni)
(24.32)and (24.33),that
1 S i f rn, n k N(j.kq 1,
where NU,k) minln'. rn 1, bn k j mixing coefficients tftl (0 and vp
anij,k)
(24.62)
cn,(-1)u+./,(f-1pn+./+t.
-
Wnijnkj
+
Wuj(:u,0).
(24.65)
k=1
Although this definition entails considering k of order bn as n .-A x, note that the inter-block dependence of the summands does not depend on k. The designation is convenient here, but it need not be taken more literally than of the inequalities (24.32)and (24.33)require. The crucial conclusion is that a weak 1aw of large numbers can be applied to these terms. tmixingale'
24.18 Theorem Under assumptions
24.6(a)-(d), Au -E(-> 0.
The object here is to show that the array EZlni)), Fjnx', 1 f i K rn, rn 2 1 ( )
(Xj -
(:1
(24.66)
which satisfy conditions (a),(b), and (c) is an f-j-mixingale with constants ttzajl of 19.11. The proof could be simplified in the present case by using the fact that ble (by 24.6(a)) so that the minorant side of (24.68)below Zni is 5ib n, Cx-measura actually disappears identically for p k 0. But since the result could find other applications, we establish the mixingale property formally, without appealing to this assumption.
2 an d app 1y ing 24-9 term by term, we obtain Proof By multiplying out Zni
)
E Ezlni
Ftzazjlj@-( .f
-
-*bn).
j
The Central Limit Theorem
396
bn- 1
f
j
77 jzc1
l'-tx
E ElWnii,ot
I+ + l
-*bn)
bn
-j
j
Ewniqvk) ITf-f
lTE k=1
lF.-f
E EWnibn'ot
-*bn)
j
j
''pbbnt
bn- 1
Mnli
rx
y) (1+ lbn j=3
-./))j(p-jpu+j
+
jpx
(24.67)
,
and similarly,
l
j
+*bnj E Znli Eznli lFxl' -
Mnli
bn 1
bn
-
-.j
+ Lgn 77 )(;,+1)s-y+ 2F7((p+1)u-j-k k=1
t<
,
j=1
(24.68)
where (j 0j -1-8) for 45 > 0 Write formally, tznjv; to denote the larger of the two majorant expressions in (24.67)and (24.68),such that vp* 0 and ani is fixed by setting v; 1. Evaluating (24.68)at p 1 and (24.68)at p 0 respectively gives =
,
.
-->
=
=
=
bn 1 -
1
E E (Z?lj Ezln j Txf
j
-1)9n)
-
<<Mytj
y'q(1 + j=1
-8
-y))./-1
zlbt
+
y-u1-8
lb n, <<sl'ni and also, putting j'
=
bn
and k'
-j
=
(24.69)
j' k -
bn- 1
E Zni 2 F(Z2ni j5
i bn)
-
--
<<M1 ni
-')
j'- 1
.4-1-8
+
27)/c'-1-
+
1
(24.70)
k'=.*
.j'=l 1 <<sfnibn .
Hence, ani
=
2b n for some tinite constant B. Since BMni rn
rn
XM4fn f f=1
max
MnliTMnli
lKK rn+ 1
=
olbn-lj
(24.71)
f=1
in view of (24.27)and (24.25),these constants satisfy conditions 19.11(b) and /v2ni where Z2f/w2jis unifonnly integrable, they 2Ibn Mlni < z2 ni (c). And since Zni n follows that An f' 0 and the proof is satisfy condition It 19.11(a). also complete.. ---1..+
,
This brings us to the final step in the argument, establishing the asymptotic m.d. property of the Bernstein blocks.
24.19 Theorem Under 24.6(a)-(d), limn-x/ra) Proof Applying
(24.17),
=
1..
for Dependent
CLTS
Processes
rn
'-rrn
1-I
=
=1
+ I7f-l(1 whereFf-1 sl =
(1 +
,
rn
z-n i)
l
=
+
i
T-i
z-
l ni
-
(24.72)
,
i= 1
lxn-measurable sin,--
ihk nj is an )
397
by 24.6(a), and hence
r.v.
rn
ily7Ftwf-lznj)
e'twrn)
1+
=
f=1 rn
= By the Cauchy-schwartz
1 +i,
E f=1
f=1
(24.73)
'bn) ?-l lI2IIF(2nfI@,;t,f--1 11 112,
(24.74)
-
.
inequality, rn
ra
EI
tf 1' bn )) '/f-1 E (kni 15 n,-x
'i-tEkni I@nt,':1 tbn)l /
=1
where11 h-1lI2is uniformly bounded by (24.15),which follows in turn by (24.11), the result hinges on the rate of approach to zero of IlF(znf Isni---bbn) This 112. so
cannot be less than that for Zni, so consider, more conveniently, the latter case. l
ibn
'bn)
IIF(z,,f lsy'ntf---' 2 11
E
=
1
77
1/2
'bn) Exnt I1,$'---1 '$
t-ni- 1)!u+1
il',t 'rltf--''t
Yl
f
t--zi- 1)u+1
E CE x
ibn- 1
+
Xn))2
I
1/2
ibn-t
:--) 7) IEExnt3 2f=(f-1)!u+1
Tnt--l'bnjexnt-vk3
'n)) @nt'--1
k=l
l
(24.71)
Applying 24.8, E(E(Xnt3 Fjf--l 'bnljl < c2uf((,-(j.j)s)2
(24.76)
and
!
E EExnt
*n)) 'bn4Exnt..kI@,$,f--1 IFntf,---1
j
Xn) 1 1I.E'(.kr+,, sni---'bn, 2 IIF(-Yn,I Isni--11 11 A
1
S cn?(,-(f-1)&ncn,,+1(,-(f-l):n+z,
where
(j
=
O(j-1/2-B
) for g, > 0
for the 0 defined in 24.6(c). Hence, bn
IIA'(Znf << Mni ITntf--xlxnlllz A
(24.77)
bn- 1
bn
1/2
(J?
7-7 + 277 Lj77(k
/=1
/=1
(24.78)
P=7+1
Applying (24.26), where the sum of squares is 041) and the double sum is O(:1-2pl) n 24.6(d) implies assumption .
The Central Limit Theorem
398
rn
'X l f=1
A'(/
-1
Ek
n
/1
l *n44
sni,-x
rn
I
A7A'/',,tn(max(!?,,t'-2'z)'21 1)
-
,
=1
tl-g jy-l/z = Otmaxtha n
))
,
This ensures that
71F(h-1Z--
0 as n
ol j;
=
(g4yo)
.
.
which is the desired result.
-->
x,
.
Proof of 24.6 We have established that Erilznf lbjnxnt-P-> N(0, 1). There remains the formality of extending the same conclusion to Sn, but this is easy =
since
lSn E;t1bnXnt l -
IXn'rn b
=
n
+1
+
...
+
XnnI
=
Op(rn-1/2)
(24 80)
,
.
and Sn has the same limiting distribution by 22.18. The proof of 24.6 is therefore complete. * It remains just to modify the proof for the martingale difference case, as was promised above. Proof of 24.14 It is easily seen that 24.16 holds for rn n and bn 1. ln 24.17, Bn 0 identically since (k 0 for k > 0 in (24.57).In 24.18 one may put Zni Xnt, and the conditions for 19.10 follow directly from the assumptions. Lastly, 24.19 holds since the sum in (24.79)vanishes identically under the martingale difference assumption. The proof is completed just as for 24.6. . =
=
=
=
=
25 Some Extensions
The CLT with Estimated Normalization The results of the last two chapters, applied to the case Xnt Xtlsn where EXt) E(X3Xt)l, would not be particularly useful if it were necessary to = 0 and 1 in order to apply them. know the sequences (cf2) and (cf,t-k)f or k Obviously, the relevant normalizing constants must be estimated in practice. Consider the independent case initially, and let Sn Xlcjxtlsnwhere sn2 l E)=I.Yl, 2 Also 1et n and we may write E:-1c,. =
.2n
=
,
,
=
=
=
sn
=
1
n
Sn
>1
-
xt
=
dnsn,
(25.1) -%
snln. If dn -T-+ 1, we could appeal to 22.14 to show that n #(0,1) whenever Sn -P-> N(0,1). The interesting question is whether the minimal conditions sufficient for the CLT are also sufficient for the relevant convergence W
here dn
=
in probability. If the sequence is stationary s well as independent, existence of the variance ..C..A c2 (applying23.5 c 2 is sufficient for both the CLT (23.3)and for rl-1E')=1Xl 2 to Xt4. I n the heterogeneous case, we do not have a weak law of large numbers for lXf2l b ase d solely on the Lindeberg condition. However, the various sufticient conditions for the Lindeberg condition given in Chapter 23, based on unifonn integrability, are sufficient for a WLLN. Without loss of generality, take the case of possibly trending variances. the conditions of
25.1 Theorem lf (XfJis an independent sequence satisfying 23.18, theh
(25.2) this has zero mean, is Proof Consider the sequence (Xt2 c2)/c2 t t. By assumption independent (andhence an m.d.), and unifonnly integrable. The conditions of 19.8, 2 1, bt ct,2 and an with p sn, are satisfied since, by (ga 5y) -
=
=
=
N
2S Xcr >1 where Mn
=
maxlxfxnf ctj
.
Hence
.
nM2 n
=
O(sl) n
,
,
(25.3)
The Central Limit Theorem
400
n Xl ),-1
-
,
E
c2) , =
2 sn
which is sufticient
E
En,- IF,
1
-
2
0,
--
(25.4)
sn
for convergence in probability.
.
(.V)is a martingale
When the sequence
needed for (X,2) to o bey for the mmingale CLTS precisely to the required
difference, spplementary conditions are but WLLN these turn out to be the same as are needed a of 24.3 and 24.4. In fact, condition 24.3(a) corresponds result. We have, immediately, the following theorem. ,
25.2 Theorem Let (m,@t)be a m.d., and let the conditions of 24.3 or 24.4 be satisfied; then (25.2)holds. I:l Although we have spoken of estimating the variance, there is clearly no neces2 sity for (snlnj, or any ot h er sequence ap art from fdn ), to converge. Example 24.11 is a case in point. ln those globally covariance stationary cases (see j13.2) W here snn converges variance' to a finite positive constant, say -c2,the nlln as a consistent estimator of d2. of the sequence, we conventionally refer to l But more generally, the same terminology can always be applied to n with respect to sn, 2 i n the sense of (25.2). Alternative variance estimators can sometimes be defined which exploit the particular structure of the random sequence. ln regression analysis, we typically apply the CLT to sequences of the form Xt Wtut where (Ut,5tIis assumed to be a m.d. with fixed variance c2 (the disturbance), and where Bs (a regressor) is In this case, (A),Tr) is a m.d. with variances czf c2F(Wl), 5t jr2)z).j<2j, 5k2. W for This is the usual hichsuggests the estimator :,,2 (rI-1E) t approach in regression analysis, but of course the mthod is not robust to the Z)=1
=
-l-measurable.
=
=
=
=
scedasticity-consistent. 26
Now consider the case of general dependence. The complicating factor here is 5.2 l that n contains covariances as well as variances, and n is no longer a suitable estimator. A sample analogue of sn2 must i nc lude terms of the fonn Xtxj for qjI k 0, but the problem is to know how many of these to include. 1 as well as foj If we include all of them, in other words for j 1 J,...,n t, the resulting Jn2 is converging, not and the ratio of this quantity to sum is equal to (IJ=1m)2, l 1) consistent estimation we probability 1, but in For in distribution to k ( to must make use of the knowledge that a11but a finite number of the covariances are arbitrarily close to 0, and omit the corresponding sample products from the sum. Similarly to the m.d. case, the conditions of 24.6 contain the required convergence result. Consider (24.46)and (24.47)where Xnt Xtlsn, but now write =
=
-
-
.
=
Zr:j snzni =
ibn
=
F7
Xt, i
>(f-1)!u+1
k=
1,...,r,,.
(25.5)
Some Extensions
401
In the proof of the CLT the construction of the Bernstein blocks was purely conceptual, but we might considerthe option of actually computing them. The sum Ehjzipj is consistent for in the of squares of the unweighted blocks, sense that .2n1
.%2
=
-2
,
r 'N
Jn1
2
Sn
=
2 Zni
---+
pr
1
(25.6)
,
j=1
according to 24.17 and 24.18. An important rider to this proposal is that 24.17 and 24.18 were proved using only (24.25)and (24.27),so that, as noted previously, any e (0,1) will serve to construct the blocks. In the context of the conditions of 24.12 at least, the only constraint imposed by 24.6(d) is represented by (24.39),which puts a possible lower bound on (x in the case of decreasing variances ('y> 0), but no upper bound strictly less than 1. It is sufficient for consistency if bn goes to infinity at a positive rate, and we are not bound to that satisfies the conditions of the CLT to construct the blocks in use the (25.6). But although consistent, J1j is not the obvious choice of estimator. It would be andGallantandWhite more naturaltofollow authors suchasNeweyandWesttlg87) who consider estimators alia, based all the cross-products Xtxt-k (1988), inter on for t = k + 1,...,n and k = 0,...,:,,. In terms of the array represention of Fig. 24.1, these are the elements in the diagonal band of width lbn, rather than the diagonal blocks only. (In this context, bn is referred to as the band width.) The simplest such estimator is
-2
Snl
bn
n
&1 r=kml
of 24.6, applied to Xtlsn,
Under the conditions
*
(25.7)
t +
>1
25 3 Theorem
n
Xx2 zy-l y-qxtxt-k.
=
nhlsnl
-CJ-y 1.
2 Prtmf Let Xnt Xlsn in (24.53),so that sJn deno teS the same sum constructed lz Elnhlljsnl from the Xt in place Of the Xnt. The difference between An and is the quantity =
-
ra 1 -
4* n
=
-
1 2
Sn
ibn
ibn-t- 1
2
xtxt-k okt-
f=1
/=(/-1),n+1
t=l n
77
t-nrnbn-j
bn- 1
(x2, c2,) + -
277 k=1
xtxt-k-c,,,-,)
(25.8)
The components of this sum correspond to the rn 1 triangles' which separate the 1) tenns, plus the terms diagonal blocks in Fig. 24.1, each containing closely analogous to the proof of from the lower-right corner blocks. Reasoning -iL+ 0. The sums of the corresponding covariancesconverge 24 18 shows that An' absolutely to 0 by 24.17, since they are components of Bn' in (24.54),and it folloyvs that -
qbnbn -
402
The Central Limit Theorem l
I n2
l
j
nl
-
-EL>
2
0.
Sn
The theorem therefore follows from 24.18.
.
21 Since this estimator uses sample data which are ( discarded in n there are informal reasons for preferring it in small samples. But there is a problem in 9,,22 is not a square, and nOt always that (the c