ADVANCED TEXTS IN ECONOMETYCS General Editors
C. W. J. Granger
G. E. Mizon
STOCHASTIC LIMIT THEORY An lntroduction for Econometricians
JAMES DAVIDSON
(lxford llniversity Press 1994
Oxjord University Press, Walton Street Oxjord oxa Oxford New York Athens Auckland Banghok Bombay Calcutta Cape Tozvn Dar es Salaam Delhi Florence Hong Kong Istanbul Karachi Kuala Lumpur Madras Madrid Melbourne Mexico City Nairobi Jtzrl Singapore Ttzzc Tohyo Toronto and assoclted companies in Berlin Ibadan ,
6Dp
Published in the Unitei States by Oxford University Pyess l'nc., Nec York
(0 JamesDavidson, 1994 All rlk/lfl reserved. No part of this publication may be repvoduced, stored in a rdfrfcrtz/ system, or transmitted, in any or .',Vtz'rly means, without the yrfor permsion 2'A; zvriting t)./ Oxford Ublbcrlf.y Press. Within the UK, exceptions are allouled p respect of any fair dealing .Jorthe Azlr/t?ycoj research or private study, or crl'li'cl'lpl or reviezv, as permitted under the Co/wrr/, Desns and Patents Act, zq'l', or fr the case V veprographic reproduction in accordance zcff/l the terms V the licences issued by the Copyright Licensing Agency. A'r/tzpzrc: concerntg rcyrotucfpn outside these terms and in other countries should be sent to the Rights Depavtment, Oxjord University Press, at the address above ./brv
F/7l'.book f.ssold subject to the condition that it shall not, by zt?tz.'p oj trade or otherwe, be lent, re-sold, Itived out or otheyulise circulated without the publisher's prior consent s any formof binding (?r covey other than that l'A; which 2'J is published and xithout a similar condition i'ac/z/kg this condhion being imposed on the subsequent purchaser srzk./z Librar.y Ctzftz/twlfag in Publication Data Data available Library
of Congress Cataloging in Publication Data
Data available ISBN 0-19-877402-8 ISBN 0-19-877403-6 ('Pbk) 1 3 5 7 9 10 8 6 4 2 Printed in Great Britain on acid-jkee yl/cr by Biddles Ltd., Guildjord and King
's
Lynn
For Lynette, Julia, and Nicola.
what in me is dark Illumine, what is 1ow raise and support, That, to the height of this great argument, 1 may assert Eternal Providence, And justify the ways of God to men.'
& .
.
.
Paradise f.,t?l/, Book 1, 16-20
Conients Preface
xiii
Mathematical
Symbols and Abbreviations
AXA
Part 1: Mathematics 1. Sets 1.1 1.2 1.3 1.4 1.5 1.6
and Numbers Basic Set Theory Countable Sets The Real Continuum Sequences of Sets Classes of Subsets Sigma Fields
3 8 10 12 13 15
2. Limits and Continuity 2.1 The Topology of the Real Line 2.2 Sequences and Limits 2.3 Functions and Continuity 2.4 Vector Sequences and Functions 2.5 Sequences of Functions 2.6 Summability and Order Relations 2.7 Arrays
20 23 27 29 30 31 33
3. Measure 3.1 Measure Spaces 3.2 The Extension Theorem 3.3 Non-measurability 3.4 Ptoduct Spaces 3.5 Measurable Transfcymations 3.6 Borel Functions
36 40 46 48 50 55
4. lntegration 4.1 Construction of the Integral 4.2 Properties of the lntegral 4.3 Product Measure and Multipl 4 4 The Radon-Nikodym
57 61 64 69
lntepals
'theorem
.''' -' y'
-'
7*
(qd qd 5. Metric
Spaces and Metris Distances 1 5. ) xjj 5.2 Separability and tjmptttt t(((...'.! j y.) .. . r), 5.3 Exampls ' ..)
.....
'
.
...
...
E. .
.........y
.. y. . . . . . E)rt)yy),,:yy't,y:r(,q)'yt 2 .,
i.; ! E E. ... . . . .. ... . ... . q.
. r
. j.
. .. q .
.
...
75 78 82
Contents
5.4 Mappings on Metric Spaces 5.5 Function Spaces
84 87
6. Topology 6.1 Topological Spaces 6.2 Countability and Compactness 6.3 Separation Properties 6.4 Weak Topologies 6.5 The Topolpgy of Product Spaces 6.6 Embedding and Metrization
93 94 97 101 102 105
Part 11: Probability 7. Probability Spaces 7. 1 Probability Measures 7.2 Conditional Probability 7.3 Independence 7.4 Product Spaces
111 113 114 115
8. Random Variables 8. 1 Measures on the Line 8.2 Distribution Functions 8.3 Examples 8.4 Multivariate Distributions 8.5 Independent Random Variables
117 117 122 124 126
9. Expectations 9. 1 Averages and Integrals 9.2 Expectations of Functions of X 9.3 Theorems for the Probabilist's Toolbox 9.4 Multivariate Distributions 9.5 More Theorems for the Toolbox 9.6 Random Variables Depending on a Parameter
128 130 132 135 137 140
10. Conditioning 10.1 Conditioning in Product Measures 10.2 Conditioning on a Sigma Field 10.3 Conditional Expectations 10.4 Some Theorems on Conditional Expectations 10.5 Relationships between Subfields 10.6 Conditional Distributions
143 145 147 149 154 157
11. Characteristic Functions 11.1 The Distribution of Sums of Random Variables 11.2 Complex Numbers
161 162
Contents
11.3 The Theory of Characteristic Functions 11.4 The Inversion Theorem 11.5 The Conditional Characteristic Function
f;r
164 168 171
Part 111:Theory of Stochastic Processes 12. Stochastic Processes 12.1 Basic Ideas and Terminology 12.2 Convergence of Stochastic Sequences 12.3 The Probability Model 12.4 The Consistency Theorem 12.5 Uniform and Limiting Properties 12.6 Unifonn lntegrability
177 178 179 183 186 188
13. Dependence 13.1 Shift Transformations 13.2 Independence and Stationarity 13.3 Invariant Events 13.4 Ergodicity and Mixing 13.5 Subfields and Regularity 13.6 Strong and Uniform Mixing
191 192 195 199 203 206
14. Mixing 14.1 Mixing Sequences of Random Variables 14.2 Mixing Inequalities 14.3 Mixing in Linear Processes 14.4 Sufficient Conditions for Strong and Uniform Mixing
209 211 215 219
15. Martingales 15.1 Sequential Conditioning 15.2 Extensions of the Martingale Concept 15.3 Martingale Convergence 15.4 Convergence and the Conditional Variances 15.5 Martingale Inequalities
229 232 235 238 240
16. Mixingales 16.1 Definition and Examples 16.2 Telescoping Sum Representations 16.3 Maximal Inequalities 16.4 Uniform Square-integrability
247 249 252 257
17. Near-Epoch Dependence 17.1 Definition and zxamples 17.2 Near-Epoc Dependence and Mixingales
261 264
Contents
17.3 Near-Epoch Dependence and Transformations 17.4 Approximability
267 273
Part lV: The Law of Large Numbers 18. Stochastic Convergence 18.1 Almost Sure Convergence 18.2 Convergence in Probability 18.3 Transformations and Convergence 18.4 Convergence in Lp Norm 18.5 Examples 18.6 Laws of Large Numbers
281 284 285 287 288 289
19. Convergence in fp-Norm 19.1 Weak Laws by Mean-square Convergence 19.2 Almost Sure Convergence by the Method of Subsequences 19.3 A Martingale Weak Law 19.4 A Mixingale Weak Law 19.5 Approximable Processes
293 295 298 302 304
20. The 20. 1 20.2 20.3 20.4 20.5 20.6
306 311 313 316 318 323
Strong Law of Large Numbers Technical Tricks for Proving LLNS The Case of Independence Martingale Strong Laws Conditional Variances and Random Weighting Two Strong Laws for Mixingales Near-epoch Dependent and Mixing Processes
21. Uniform Stochastic Convergence 21.1 Stochasti Functions on a Parameter Space 21.2 Pointwise and Unifonn Stochastic Convergence 21.3 Stochastic Equicontinuity 21.4 Generic Uniform Convergence 21.5 Unifonu Laws of Large Numbers
Part V: The Central Lint
327 330 335 337 340
Theorem
22. Weak Convergence of Distributions 22.1 Basic Concepts 22.2 The Skorokhod Representation Theorem 22.3 Weak Convergence and Transformations 22.4 Convergence of Moments and Characteristic Functions 22.5 Criteria for Weak Convergence 22.6 Convergence of Random Sums
347 350 355 357 359 361
Contents
23. The 23. 1 23.2 23.3 23.4 24.
CLTS
24.1 24.2 24.3 24.4 24.5
11
Classical Central Limit Theorem The i.i.d. Case lndependent Heterogeneous Sequences Feller's Theorem and Asymptotic Negligibility The Case of Trending Variances
364 368 373 377
for Dependent Processes A General Convergence Theorem The Martingale Case Stationary Ergodic Sequences The CLT for NED Functions of Mixing Processes Proving the CLT by the Bernstein Blocking Method
380 383 385 386 391
25. Some Extensions 25.1 The CLT with Estimated Normalization 25.2 The CLT with Random Norming 25.3 The Multivariate CLT 25.4 En'or Estimation
399 403 405 407
Pa14 V1: The Functional Central Limit Theorem 26. Weak Convergence in Metric Spaces 26.1 Probability Measures on a Metric Space 26.2 Measures and Expectations 26.3 Weak Convergence 26.4 Metrizing the Space of Measures 26.5 Tightness and Convergence 26.6 Skorokhod's Representation
413 416 418 422 427 431
27. Weak Convergence in a Function Space 27.1 Measures on Function Spaces 27.2 The Space C 11.3 Measures on C 27.4 Brownian Motion 27.5 Weak Convergence on C 27.6 The Functional Central Limit Theorem 27.7 The Multivariate Case
434 437 440 442 447 449 453
28. Cadlag Functions 28.1 The Space D 28.2 Metrizing D 28.3 Billingsley's Vetric 28.4 Measures on D
456 459 461 465
Contents
Xlt
28.5 Prokhorov's Metric 28.6 Compactness and Tightness in D 29.
FCLTS
29.1 29.2 29.3 29.4 29.5
for Dependent Variables The Distribution of Continuous Functions on D Asymptotic Independence The FCLT for NED Functions of Mixing Processes Transfonned Brownian Motion The Multivariate Case
467 469 474 479 481 485 490
30. Weak Convergence to Stochastic Integrals 30.1 Weak Limit Results for Random Functionals 30.2 Stochastic Processes in Continuous Time 30.3 Stochastic Integrals 30.4 Convergence to Stochastic Integrals
496 500 503 509
Notes
517
References
519
lndex
Preface
Recent years have seen a marked increase in the mathematical sophistication of econometricresearch. While thetheory of linearparametric models which fonns the backbone of the subject makes an extensive and clever use of matrix algebra, the statistical prerequisites of this theory are comparatively simple. But now that these models are pretty thoroughly understood, research is concentrated increasingly on the less tractable questions, such as nonlinear and nonparametric estimation and nonstationary data generation processes. The standard econometlics texts are no longer an adequate guide to this new technical literature, and a sound understanding of the probabilistic foundations of the subject is becoming less and less of a luxury. The asymptotic theory traditionally taught to students of econometrics is founded on a small body of classical limit theorems, such as Khinchine's weak law of large numbers and the Lindeberg-Lvy central limit theorem, relevant to the stationary and independent data case. To deal with linear stochastic difference equations, appeal can be made to the results of Mann and Wald (1943a),but even these are rooted in the assumption of independent and identically distributed disturbances. This foundation has become increasingly inadequate to sustain the expanding editke of econometric inference techniques, and recent years have seen a systematic attempt to construct a less restrictive limit theory. Hall and Heyde' s Martingale Limit Fdt??'.y and its Application (1980)is an important landmark, as are a series of papers by econometricians including among others Halbert White, Ronald Gallant, Donald Andrews, and Herman Bierens. This work introduced to the econometrics profession pioneering research into limit theot'y under dependence, done in the preceding decades by probabilists such as J. L. Doob, 1. A. Ibragimov, Patrick Billingsley, Robert Serfling, Murray Rosenblatt, and Donald McLeish. These ltter authors devised various concepts of limited dependence for general nonstationary time series. The concept of a martingale has a long history in probability, but it was primarily Doob's Stochastic Processes (1953)that brought it to prominence as a tool of limit theory. Martingale processes behave like the wealth of a gambler who undertakes a succession of fair bets; the differences of a martingale (the net winnings at each step) are unpredictable from lagged information. Powerful limit theorems are available for martingale difference sequences involving no further restrictions on the dependence of the process. Ibragimov and Rosenblatt respectively defined strong mixing and unbrm mixing as charactermemory' or independence at long range. McLeish defined the izations of notion of a mixingale, the asymptotic counterpart of a martingale difference, becomingunpredictable m steps aheadaspz becomeslarge. This is aweakerproperty than mixing because it involves only low-order moments of the distribution, but mixingales oossess most of those attributes of mixine orocesses needed to make
,
Preface
limit theorems work. Very important from the econometrician's point of view is the property dubbed by Gallant and White (1988)near-epoch dependence from a phrase in one of McLeish's papers, although the idea itself goes back to
Billingsley (1968)andlbragimovtlg6z). Themixingproperty may notbepreserved by transformations of sequences involving an intinite number of lags, but nearepoch dependence is acondition under whichthe outputs of adynamic econometric model can be shown, given some further conditions, to be mixingales when the inputs are mixing. Applications of these results are increasingly in evidence in the econometric literature', Gallant and White's monograph provides an excellent survey of the possibilities. Limit theorems impose restzictions on the amount of dependence between sequence coordinates, and on their marginal distributions. Typically, the probability of outliers must be controlled by requiring the existence of higher-order moments, but there are almost always trade-offs between dependence and moment restrictions, allowing one to buy more of one at the price of less of the other. The fun of proving limit theorems has been to see how far out the envelope of sufficient conditions can be stretched, in one direction or another. To complicate matters, one can get results both by putting limits on the rate of approach to independence (the rate of mixing), and by limiting the type of dependence (the martingale approach), as well as by combining both types of constraint (the mixingale approach). Theresults now available areremarkably powerful,judged by the yardstick of the classical theory. Proofs of necessity are elusive and the limits to the envelope are not yet known with certainty, but they probably lie not too far beyond the currently charted points. Perhaps the major development in time-series econometrics in the 1980s has been the theory of cointegration, and dealing with the distributions of estimators when time series are generated by unit root processes also requires a new type of limit theory. The essential extra ingredient of this theory is the functionalcentral limit theorem (FCLT). The proof of these weak convergence results calls for a limit theory for the space of functions, which throws up some interesting problems which have no connterpart in ordinary probability. These ideas were pioneered by Russian probabilists in the 1950s, notably A. V. Skorokhod and Yu. V. Prokhorov. lt turns out that FCLTS hold under properties generally similar to those for the ordinary CLT (thoughwith a crucial difference), and they can be analysed with the same kind of tools, imposing limitations on dependence and outliers. The probabilistic literature wlch deals with issues of this kind has been seen as accessible to practising econometricians only with difticulty. Few concessions are made to the nonspecialist, and the concerns of probabilists, statisticians, and econometlicians are frequently different. Textbooks on stochastic processes (Cox and Miller 1965 is a distinguished example) often give prominence to topics would regard as fairly specialized (e.g. Markov chains, that econometricians processes in continuous time), while the treatment of important issues like nonstationarity gets tucked away under the heading of advanced or optional material if not omitted altogether. Probability texts are written for students of
mathemtics
and ag&ume
n fnmilinritv
wifh
fln/R
fnllrlnr-
nf
tlao
t.lxlor':
'lqne
Preface
econometricians may lack. The intellectual investmnt required is one that students and practitioners are often, quite reasonably, disinclined to make. It is with issues of this sort in mind that the present book has been written.
The t'irstobjective has been to provide a coherent and unified account of modern asymptotic theory, which can function as both a course text, and as a work of reference. The second has been to provide a grounding in the requisite mathematics and probability theory, making the treatment sufticiently self-contained that even readers with limited mathematical training might make use of it. This is not to say that the material is elementary. Even when the mathematics is mastered, the reasoning can be intricate and demand a degree of patience to absorb. Profs for nearly all the results are provided, but readers should never hesitate to pass over these when they impede progress. The book is also intended to be useful as a reference for students and researchers who only wish to know basic things, like the meaning of technical tenns, and the variety of limit results available. But, that said, it will not have succeeded in its aim unless the reader is sometimes if for no better stimulated to gain a deeper understanding of the material reason, because this is a theory abounding in mathematical elegance, and technical ingenuity which is often dazzling. -
Outline of the Work Part I covers almost a11the mathematics used subsequently. Calculus and matrix algebra are not treated, but in any case there is little of either in the book. Most readers should probably begin by reading Chapters 1 and 2, and perhaps the first sections only of Chapters 3 and 4, noting the definitions and examples but skipping a11 but the briefest proofs initially. These chapters contain some difticult material, which does not a11need to be mastered immediately. Chapters 5 and 6 are strictly required only for Chapter 21 and Part VI, and should be passed over on tirst reading. Nearly everything needed to read the probability literature is covered in these chapters, with perhaps one notable exception the theory of normed spaces. Some treatments in probability use a Hilbel't space framework, but it can be pvoided. The number of pplications exploiting this approach seemed currently too small to justify the added technical overhead, although future developments may require this judgement to be revised. Pm 11 covers what for many readers will be more familiar territory. Chapters 7, 8, and 9 contain essential background t be skimmed or studied in more depth, as appropriate. lt is the collections of inequalities in j9.3 and j9.5 that we will have the most call to refer to subsequently. The content of Chapter 10 is probably less familiar, and is very important. Most readers will want to study this chapter carefully sooner or later. Chapter 11 can be passed over initially, but is needed in conjunction with Part V. In Part 11I the main business of the work begins. Chapter 12 gives an introduction to the main concepts arising in the study of stochastic sequences. Chapters 13 and 14 continue the discussion by reviewing concepts of dependence, and Chapters 15, 16, and 17 deal with s/ecializedclasses of sequence whoseproperties -
7771
Preface
make them amenable to the application of limit theorems. Nearly all readers will want to study Chapters 12, 13, and the earlier sections of 14 and 15 before going further, whereas Chapters 16 and 17 are rather technical and should probably be avoided until the context of these results is understood. In Parts IV and V we arrive at the study of the limit theorems themselves. The aim has been to contrast alternative ways of approaching these problems, and to present a general collection of results ranging from the elementary to the very general. Chapter 18 is devoted to fundamentals, and everyone should read this before going further. Chapter 19 compares classical techniques for proving laws of large nmbers, depending on the existence of second moments, with more modern methods. Although the concept of convergence in probability is adequate in many econometric applications, proofs of strong consistency of estimators are increasingly popular in the econometrics literature, and techniques for dependent processes are considered in Chapter 20. Uniform stochastic convergence is an essential cohcept in the study of econometric estimators, although it has only recently been systematically researched. Chapter 21 contains a synthesis of results that have appeared in print in the last year or two. Part V contrasts the classical central limit theorems for independent processes with the modern results for martingale differences and general dependent processes. Chapter 22 contains the essentials of weak convergence theory for random variables. The treatment is re%onably complete, although one neglected topic, to which much space is devoted in the probability literature, is convergence to stable laws for sequences with infinite variance. This material has found few applications in econometlics to date, although its omission is anothet judgement that may need to be revised in the future. Chapter 23 describes the classic CLTS for independent processes, and Chapter 24 treats modern techniques for dependent, heterogeneous processes. Pal't V1 deals with the functional central limit theorem and related convergence results, including convergence to limits that can be identified with stochastic integrals. A number of new mathematical challenges are presented by this theory, and readers who wish to tackle it seriously will probably want to go back and apply themselves to Chapters 5 and 6 first. Chapter 26 is both the hardest going and the least essential to subsequent developments. It deals with the theory of weak convergence on metric spaces at a greater level of generality than we strictly need, apd is the one section where topological arguments seriously intrude. Almost certa in ly one shoul go first to Chapter 27 ? referring back as needed for definitions and statements of the prerequisite theorems, and pursue the rationale for these results further only as interest dictates. Chapter 28 is likewise a technical prologue to Chapers 29 and 30, and might be skipped over at first reading. The meat of this pal't of the book is in these last two chapters. Results are given on the multivariate invariance principle for heterogeneous dependent processes, paralleling the central limit theorems of Chapter 24. A number of the results in the text are, to the author's knowledge, new. These include 14.13/14, 19.11, 20.18/19, 20.21, 24.6/7/14, 29.14/29.18, and 30.13/14, although some have now appeared in print elsewhere.
Preface
A37II
Further Reading There is a huge range of texts in print covering the relevant mathematics and probability, but the following titles were, for one reason or another, the most frequently consulted in the course of writing this book. T. M. Apostol's Mathematical Analysis (2nd edition) hits just the right note for the basic bread-andbutter results. For more advanced material, Dieudonn's Foundations ofModern Analysis and Royden's Real Analysis are well-known references, the latter being the more user-friendly although the treatment is often fairly concise. Halmos's classic Measure Fdt?ry and Kingman and Taylor's Introduction to Measure and Probability are worth having access to. Willard's General Topology is a clear and well-organized text to put alongside Kelley's classic of the same name. Halmos's Naive Set Fct??'y is a slim volume whose main content falls outside our sphere of interest, but is a good read in its own right. Strongly recommended is Borowski and Borwein's Collins Reference Dictionaty ofMathematics; one can learn more about mathematics in less time by browsing in this little book, and following up the cross references, than by any other method 1 can think of. For a stimulating introduction to metric spaces see MichaelBarnsley's popularFmc/l/. Everywhere. For further reading on probability, one might begin by browsing the slim volume that stmed the whole thing off, Kolmogorov's Foundations of the F/ldtp?-yof Probability. Then, Billingsley's Probability and Measure is an inspiration, both authoritative and highly readable. Breiman's Probability has a refreshingly informal style, and just the right emphasis. Chung's A Course in Probability The value of the Fct??'y is idiosyncratic in parts, but strongly recommended. Probability and Fwr.y Love's edition) Feller's An Introdclassic texts, (4th uction to Probability Fct??-y and its Applications (3rd edition) is self-evident, although these are dense and detailed books that can take a little time and patience to get into, and are chiefly for reference. Cramr's Mathetnatical Methods of Statistics is now old-fashioned, but still useful. Two more recent titles are Shiryayev's Probability, and R. M. Dudley's tough but stimulating Real Analysis and Probqbility. Of the more specialized monographs on stochastic convergence, the following titles (in order of publiation date) are a11 important: Doob, Stochastic The fzzw- of fzzrpd Numbers', Parthasarathy, Probability Processes; Rvsz, Measures on Metric Spaces', Billingsley, Convergence of Probability Measures', losifescu and Theodorescu, Random Processes and Learning; Ibragimov and Linnik, Independent and Stationaly Sequences of Random Variables', Stout, Almost Sure Convergence; Lukacs, Stochastic Convergence; Hall and Heyde, Martingale Limit F/lct??'yand its Application; Pollard, Convetgence ofstochastic Processes; Eberlein and Taqqu (eds.),Dependence in Probability and Statistics. Doob is the founding father of the subject, and his book its O1d Testament. Of the rest, Billingsley's is the most original and influential. Ibragimov and Linnik's essential monograph is now, alas, hard to obtain. The importance of Hall and Heyde was mentioned above. Pollard's book takes up the weak convergence
Preface
story more or less where Billingsley leaves off, and much of the material complements the coverage of the present volume. The Eberlein-rraqqu collection contains up-to-date accounts of mixing theory, covering some related topics outside the range of the present work. The literature on Brownian motion and stochastic integration is extensive, butKaratzas and Shreve's BrownianMotion and Stochastic Calculus is arecent and comprehensive source forreference, and Kopp's Martingales and Stochastic Integrals was found useful at several points. These items receive an individual mention by virtue of being between hard covers. References to the journal literature will be given in context, but it is worth mentioningthatthe fourpapers by DonaldMctaeish, appearing between 1974 and 1977, form possibly the most influential single contribution to our subject. Finally, titles dealing with applications and related contributions include Serfling, Approximation Theorems ofMathentatical Statistics', White, Asymptotic Theotyfor Econometricians', Gallant, Nonlinear Statistical Methods; Gallant and White, A Un6ed Fwr.y of Estimation and Inference for Nonlinear Dynamic Models, Amemiya, Advanced Econometrics. Al1of these are highly recommended for forming a view of what stochastic limit theory is for, and why it matters.
Acknowledgements The idea for this book originated in 1987, in the course of writing a chapter of mathmatical and statistical prerequisites for a projected textbook of econometric theory. The initial, vel'y tentative draft was completed during a stay at the University of California (San Diego) Depmment of Economics in 1988, whose hospitality is gratefully acknowledged. lt has grown a great deal since then, and competing academic commitments as getting itnishedhasinvolvedas%gglewith well as the demands of family life. My family deserve special thanks for their
forbearance.
My colleaguepeterRobinson
has been agreat source of encouragement and help,
and has commented on various drafts. Other people who have read portions of the manuscript and provided invaluable feedback, not least in pointing out my errors, include Getullio Silveira, Robertde Jong, and especially Elizabeth Boardman, who took immense trouble to help me lick the chapters on mathematics into shape. l am also most grateful to Don Andrews, G raham Brightwell, Sgren Johansen, Donald
McLeish, Peter Phillips, Hal White, and a number of anonymous referees for helpful conversations, comments and advice. None of these people is responsible for the various flaws that doubtless remain. The book was written using the ChiWriter 4 technical word processor, and after conversion to Postscript format was produced as camera-ready copy on a HewlettPackard Laserlet 451 printer, direct from the original iiles.I must particularly thank Cay Horstmann, of Horstmann Software Design Corp., for his technical assistance with this task. Lrndon, June 1994
Mathematical Symbols and Abbreviations In the text, the symbol (:a is used to terminate examples and definitions, and also theorems and lemmas unless the proof follows directly. The symbol w tenninates proofs. References to numbered expressions are enclosed in parentheses. References to numbered theorems, examples etc. are given in bold face. References to chapter sections are preceded by j. In statements of theorems, roman numbers (i), (ii), (iii),...are used to indicate the parts of a multi-part result. Lower case letters (a), (b), (c),... are used to itemize the assumptions or conditions specified in a theorem, and also the components of a detinition. The page numbers below refer to fuller definitions or examples of use, as aPpropriate.
absolute value fw-norm Euclidean norm;
1,i -+ D
&
-, / S, 2 <, <(
>
i 1x(.) a.e.
AR
ARMA
a.s., a.s.gpt1
also fineness (of a q/rtition) weak convergence (of measures); also implication monotone convergence convergence almost sure convergence convergence in distribution convergence in Lp nol'm convergence in probability mapping, function composition of mappings equivalence in order of magnitude (of sequencesl; also equiyalence in distribution (of r.v.s) addition modulo 1 set difference partial ordering, inequality strict ordering, strict inequality order of magnitude inequality (sequencesl; also absolutely continuous (measures) mutually singular (of measures) indicator function almost everywhere autoregressive process autoregressive moving average process almost surely, (withresp. to p.m. g)
Symbols AC
I (A)40, Gm
dO V Bnbp) CLT
ch-f. c-d.f. C'p,1j C, Q c D
i Z (n4 dxny) ogc,jy D A :A 6
StlP E.4 E. Ix) E. 1V) eSS
H j+
J-
1
FCLT F(.)
9x(.) 9,,, iff inf
i-i-d. i-o. in pr.
LIE LII
,
lim limsup, lim liminf, 1im Ln) AP-NED MA
m.4
complement of A closure of A intedor of A strong mixing coefficient aleph-nought (cardinalityof (N) dfor every' Binomial distribution with parameters n and p Borel field central limit theorem characteristic function cumulative distribution function continuous functions on the unit interval
set containment strict containment chi-squared distribution with n degreeg of freedom distance between x and y cadlag functions on the unit interval dyadic rationals symmetric difference boundary of A set membership essential supremum expectation conditional expectation (on variable x) conditional expectation (on c-field 5')
Lebesgue measure
3
Symbols m.d. m.g.f. m.s.
martingale difference moment-generating function mean square space of measures
N(jt,c2)
Gaussian distribution with mean jt and variance
s(N()
natural numbers
rh, O
m h #!
0(.) t?(.) Op.) ov.4 p-d.f. P.m.
#(.)
#(.IA)
#(. IN)
H,
1-1
7rj(.)
(D
F.V.
/?(0,1j .f R y R R+
F F+
R& S.e. S.S.e.
SLLN suP Sxvz)
!
Jn
c(C)
c(m fdx
K QJ
(0l
intersection minimum of m and n fBig Oh' order of magnitude relation ,
Oh', strict order of magnitude relation stochastic order relation strict stochastic order relation null set probability density function Ktaittle
probability measure probability conditional probability (on event A) conditional probability (on c-field N) product of numbers', also partition of an interval product measure coordinate projection rational numbers random variable real valued functions on g0,11 relation real line non-negative reals extended real line, R t..p (-x,+x1
R+kJ
f+x)
n-dimensional Euclidean space stochastic equicontinuity strong stochastic equicontinuity strong law of large numbers suptemum e-neighbourhood, sphere sum of random variables variance of Sn c-tield generated by collection C c-field generated by r.v. X
J/Wpo fdF
Lebesgue integral Lebesgue-stieltjes integral
fdp z, E
expected value (integralwith resp. to p.m.) sum
ui
(y2
Symbols
Mlt
shift transformation union unifonn distribution on interval (tz,:4 union of c-fields maximum of m and n weak 1aw of lm'ge numbers with probability 1 sample mean of sequence (XrJZ
F
U
kJ,
Uka,bj
w, V
m V
n
MXLN
sv.p.1
Xn
x, X
Cartesian product
Z
ll l 1T,( ) .
'>O
.
.
ll ll (x1 I,:(l a,b) .
(f1,T)
(f,',p,)
(f1,F,F) (f,T;#) (S,* (X,*
c-field of product sets integers set designation', also sequence, array infinite sequences array largest integer S closed interval bounded by a,b open interval bounded by a,b measurable space measure space complete measure space probability space metric space topological space .t
Common usages A,#,C,D,... X,F,Z,... A',F.Z,...
J,#,,... :,,q
B,5'.I
2,C,O,F',.. 9,V,J't,-.. S,5-,X,... 1,,Y,-
-
d.p T
-
sets random variables random vectors functions positive constants bounding constants collections of subsets c-fields spaces measures metrics topology
1
MATHEMATICS
Sets and Numbers
1. 1 Basic Set Theory A set is any specified collection of objects. ln this book the objects in question are often numbers, but they may also be functions, or other sets, or indeed wholly arbitrary, to be determined by the context in which the theory is applied. ln any analysis there is a set which defines the universe of discourse, containing al1 the objects under consideration, and in what follows, sets denoted A, B etc., are subsets of a set X, with generic element x. q A meaning Set membership is denoted by the symbol belongs to the set writes and A #. The usual way B have the same elements, one A' To show sets A descriptive enclosed define is by in braces, so that for statement a to a set example A tx:x e #) defines membership of A in terms of membership of B, and is an alternative way of writingA #. Another way to denote set membership is by labels. If a set has n elements one can write A fxj, i 1,...,n 1, but any set of labels will do. The statement A lau, (x e C) says that A is the set of elements bearing labels a contained in another set C, called the index set for A. The labels (indices)need not be numbers, and can be any convenient objects at tends to be preferred in all. Sets whose elements are sets (theword this context) are denoted by upper-case script characters. A e C denotes that the set A is in the collection C, or using indices one could write C (Aa: a e Cl B is called a subset of A, written B c A, if al1 the elements of B are also elements of A. If B is a proper subset of A, ruling out B A, the relation is written B c A. The union of A and B is the set whose elements belong to either or both sets, written A k.p #. The union of a collection C, the set of elements belongr ing to one or more A e C, is denoted UA es,A, or, alternatively, one can write UaecAz for the union of the collection fAa: (x l C). The intersection of A and B is the set of elements belonging to both, written A rn #. The intersection of a collection C is the set of elements commop to a1l the sets in C, written f-tlsc/t or OascAa. ln pmicular, the union and intersection of (Aj ,A2, Au ) are written U7=1Ajand O7=1Aj.When the index set is implicit or unspecified we may write just U(xAa, OfAf or similar. The dterence of sets A and #, written A B or by some authors A #, is the of two sets set of elements belonging to A but not to B. The symmetric tft//rcrcc (A B) k.p (B -A). X-A is the complement of A in X, also denoted Ac is A A# when X is understood, and we have the general result that A B 4 rn Bc. The null set (or empty set) is O M, the set with no elements. Sets with no elements in common (havingempty intersection) are called disjoint. A partition of a set is a 61
'
<x
.x
,
=
.
=
=
=
=
=
dcollection'
=
.
=
...,
-
=
-
-
=
=
Mathematics
4
collection of disjoint subsets whose union is the set, such that each of its elements belongs to one and only one member of the collection. Here are the basic rules of set algebra. Unions and intersections obey commutative, associative and distributive laws: A t.p B
=
B QJ A,
A f'n B
=
B rn A,
(A t..p B4 tp C
=
A t.p (B
(A fa #) rn C
=
A f-h CBf'A C),
(1.4)
=
(A fa B4 tp (A rn C),
(1.5)
A fa
(# tp C)
?
C),
(A tp #) r'h(A kp C. There are also rules relating to complements known as de Morgan's laws: A tp
(# fa
C)
=
(A tp B4c
=
(A rhB)c
=
AC
fa
Bc
Ac t.p Bc.
(1.8)
Venn diagrams, illustrated in Fig. 1.1, are a useful device for clarifying relationships between subsets.
Fig. The distributive and de Morgan laws extend to general collections,
1.1 Theorem Let C be a collection of sets, and B a set. Then
(i) (ii)
UA
rn B
OA
tp B
=
AGC
U (A
=
AGC
f-lzt
AeC
#),
r)
(A kp B),
dgC C
(iii)
f-h
AGN
zc
(-Jic,
AEC
as follows.
Sets and Numbers C
(iv)
U
A
OAc. u
=
AeC
AeC
The Cartesian product of two sets A and #, written A x B, is the set of a1l possible ordered pairs of elements, the first taken from A and the second from #; we write A x B (@,y): e A, y G #) For a collection of n sets the Cartesian product is the set of al1 the n-tuples (orderedsets of n elements, with the th element drawn from A/), and is written .:'
=
.
n
Af
.(
=
(x1x2, ,
/=1
...,
xi e Af
Ak):
,
1-
1,...,n )
=
(1.9)
.
If one of the factor sets Af is empty, X2=1Xfis also empty. Product sets are important in a variety of different contexts in mathematics. Some of these are readily appreciated', for example, sets whose elements are s-vectors of real numbers are products of copies of the real line (see j1.3). But prodct sets are also central to the mathematical formalization of the notion of relationship between set elements. Thus: a relation R on a set A is any subset of A xA. If @,y)e #, we usually write x#y. R is said to be reflexive iff xRx, symmetric iff xR y implies yRx, antisymmetric iff xRy and y Rx implies x y, transitive iff xRy and yRz implies xR z, each in where case the indicated condition holds for every x, y, and z e A, as the and only if means case may be. (Note: relation relation equivalence that is reflexive, symmetric, and transiis An a tive. Given an equivalence relation R on A, the equivalence class of an element x e A is the set Ex (y e A: xRy If Ex and Ey are the equivalence classes of elements x and y, then either Ex fa Ey 0, or Ex Fy. The equality relation y is the obvious example of an equivalence relation, but by no means the only one. A partial ordering is any relation that is reflexive, antisymmetric, and transitive. Partial orderings are usually denoted by the symbols f or k, with the understanding that k y is the same as y f x. To every partial ordering there corresponds a strict ordering, defined by the omission of the elements @,x)for a11 x e A. Strict orderings, usually denoted b: < or >, are not reflexive or antisymmetric, but thy are transitive. A set A is said to be linearly ordered by a partial ordering f if one of the relations x < y, x > y, and x y hold forr every pair @,y)e A xA. If there exist elements a e A and b G A such that a < x for al1 x e A, or x S b for all x e A, a and b are called respectively the smallest and largest elements of A. A linearly ordered set A is called well-orderd if every subset of A contains a smallest element. It is of course in sets whose elements are numbers thai the ordering concept is most familiar. Copsider two sets X and F, which can be thought of as representing the universal sets fr a pqir of related problems. The following bundle of definitions contains =
Kiff'
=
.)
.
=
.x
=
.x
=
=
Mathematcs
b
the basic ideas about relationships between the elements of such sets. A (or transformation or function) T: X
F->
tnapping
F
is a rule that associates each element of X with a unique element of F; in other words, for each a7 e X there exists a specified element y e F, denoted F@). X is called the domain of the mapping, and F the codomain. The set Gw
(@,y):
e X, y
.z'
=
is called the graph of T. For T(4)
is called the image of
,4
:4
e /t
Jq
under F, and for B
F -1(#)
=
(1.10)
Xx F
the set
? X,
(F@): x
=
W@))
=
fx: F(x)
e
(1 11)
F
.
1', the set
c
#J c
X
(1.12)
is called the inverse rrlzwe of B under F. The set T(m is called the range of F, and if F(m 1,the mapping is said to be from X onto F, and otherwise, into K If each y is the image of one and only one x e X, so that F(x1) Tx if and only the mapping is said to be one-to-one, or 1-1. if xl =
=
=
.n,
The notions of mapping and graph are really interchangeable, and it is permissable to say that the graph is the mapping, but it is convenient to keep a distinction in mind between the rule and the subset of Xx F which it generates. The term function is usually reserved for cases when the codomain is the set of real numbers (seej1.3). The term correspondence is used for a nlle connecting -1 elements of X to elements of F where the latter are not necessarily unique. T is a con'espondence, but not a mapping unless F is one-to-one. However, the term one-toone correspondence is often used specitically, in certain contexts that will arise below, to refer to a mapping that is both 1. 1 and onto. lf partial orderings are defined on both X and F, a mapping is called order-preserving if F(x1) G Tx iff On the other hand, if X is partially ordered by f, a 1- 1 mapping induces .;q S
.
1.2 Theorem (i) For a collection (Aa c aY), F UAa (:
(ii) for a collection (#a i F), T-' (iii) for #
i', F -1(sc) .
UF(Xa);
=
'
Ufa
=
UF-1(#a);
w-1(s)c. ,
(iv) for A i X, A g F -1(z(u)). #. u (v) for B g F, F(F -1(#)) ,
Here, T -1(B)c means X- F-1(#). Using de Morgan's laws, properties
(ii) and
Sets and Numbers
(iii) are easily extended to the inverse images of intersections and differences', for example, we may show that the inverse images of disjoint sets are also F(A)C F(AC), # in general. Pal'ts (iv) and (v) are disjoint. However, F- F(A) illustrated in Fig. 1.2, where X and i' both correspond to the real line, A and B are intervals of the line, and F is a function of a real variable. =
Fig. 1.2
When F is a 1-1 con-espon dene (1-1 and onto) so is F-1 These properties then hold symmetrically, and the inclusion relations of parts (iv)and (v) also become X and B c K lf Z is a third set, and equalities for a1l A .
U1 F
F--> Z
is a further mapping, UoF: X
F--
the composite
mapping
Z
takes each x e X to the element U(F(x)) e Z. UoT operates as a simple transform-1 from X to Z, and 1.2 applies to this case. For C c Z, (&J07) c' ation -1 =
-1
F
(&
(C)).
0- x E .
j
i
.
.
.
.
.
'
.
' .
.
' .
' .
' .
.
j
F(z4c )
1
+
'
.
)Y
X1Z
Fig. 1.3
1.3 Example lf X 0- x E is a product space, having as elements the ordered pairs .x = (0,(), the mapping F: 0- x E i- E =
,
Mathematics
8
defined by F(0,() (, is called the projection tnapping onto E. The projection of 0- x E onto E (respectively,0-) is the set consisting of the second a set A (resp., first) members of each pair in A. On the other hand, for a set B e E, 0- x # It is a useful exercise to verify 1.2 for this case, and also to F -1 B4 T(AC) in general. In Fig. 1.3, 0- and E are line segments and check that F(A # F(A)C is the union of the indicated line 0- x E is a rectangle in the plane. Here, whereas T(A9 E. u segments, =
=
.
=
The number of elements contained in a set is called the cardinality or cardinal in this context is not a primitive one, number of the set. The notion of but can be reduced to fundamentals by what is called the principle. A set A is said to be equipotent with a set B if there exists a 1-1 correspondence connectingA and #. Think in terms of taking an element from each set and placing the pair in a pigeon-hole. Equipotency means that such a procedure can never exhust one set before the other. Now, think of the number 0 as being just a name for the null set, 0. Let the number 1 be the name for the set that has a single element, the number 0. Let 2 denote the set whoseelements are the numbers 0 and 1.And proceeding recursively, 1et n be the name for the set (0,...,n 1 1. Then, the statement that a set A has n elements, or has cardinal number n, can be interpreted to mean that A is equipotent with the set n. The set of natural numbers, denoted N, is the collection (n: n = 1,2,3,... 1. This collection is well ordered by the relation usually denoted <, where n S m actually means the same as n m under this definition of a number. %number'
dpigeon-hole'
-
1.2 Countable Sets Set theory is trivial when th: number of elements in the set is finite, but formalization becomes indispensable for dealing with sets having an intinite number of elements. The set of natural numbers (Nis a case in point. If n is a member so is n + 1, and this is true for every n. None the less, a cardinal number is formally assigned to iN, and is represented by the symbol Mtl When the elements of an infinite set can be put into a one-to-one correspondence with the natural numbers, the set is said to have cardinal number d(), but, more commonly, to be countable, or equivalently, denumerable. Countability of a set requires that a scheme can be devised for labelling each element with a unique element of (N.This imposes a well-ordering on the elements, such that there is a 'first' element labelled 1, and so on, although this ordering may have significance or be arbitrary, depending on the circumstances. It is the pigeon-hole principle that matters here, that each element has its own unique label. With infinite sets, everyday notions of size and quantity tend to break down. Augmenting the natural numbers by the number 0 defines the set (Ntl (0,1,2,3,... 1. The commonplace observation that (Ntlhas more' element than ENis contradicted by the fact that INand (Nnare equipotent (labeln 1 e Ntlby n e IN).Still more suprisingly, the set of even numbers, E = (2n, n e IN) also has an obvious labelling scheme demonstrating equipotency with EN.The nave idea that there are (Kaleph-nought').
=
-
,
df.xxzlr.o
eac
meanxz'
xlomontc
ln
N
re
1n IF- ;t
Axzithnllf
lfatyinnl fnllnclntlnn
Sets and Numbers
9
infinite subset A of ENhas a natural well-ordering and is equipotent with EN itself, the label of an element x e A being the cardinal number of the set (y e A:
A' V A l than EN consider the integers, I Turning to sets apparently 1,2,... ), the set containing the signed whole numbers and zero. These ( are linearly ordered although not well ordered. They can, however, be paired with scheme: the natural numbers using the .
Elarger'
=
,
...,-1,0,
Ezig-zag'
(1,0),(2,1),(3,-1), (4,2),...,(n, (n/21(-1)''),...,
where gxJdenotes the largest whole number below x. Thus, N and I are equipotent.
Then there are the rational (I)
=
f.x: .x
numbers.
=
alb, a e 1, b e 1, b # 0)
(1.13)
.
We can also show the following.
1.4 Theorem Q is a countable set. Proof We construct a 1-l correspondence between I x J and N. A 1-l correspondence between I x I and I x: is obtained by the methodjust used to show I countable, and one between I x: and ENx: is got by the same method. Then note that the number 2 3b e INis uniquely associated with each pair a,b4 e INxN ; the rule for recovering a and b from 2'3b is g et a as the number of divisions by 2 required to get an odd number, and the number so obtained is 3 ' The collection f 2 '3b : a e ENb G EN) c ENis equipotent with ENitself as shown in the preceding paragraph. The composition of a1l these mappings is the desired correspondence. w <
.
,
,
Generalizing this type of argument leads to the following fundamental result.
1.5 Theorem The union of a countable collection of countable sets is a countable set. n The concept of a sequence is fundamental to a11 the topics in this book. A sequence can be thought of as a mapping whose domain is a wellrordered countable set, the index set. Sinc there is always an order-preserving 1-1 mapping from IN to the index set, there is usually no loss of generality in considering the composite mapping and thinking of N itself as the domain. Another way to characterize a sequence is as the graph of the mapping, that is, a countable collection of pairs having the ordering conferred on it by the elements of the domain. The ranges of the sequences we consider below typically contain either sets or real numbers', the associated theory for these cases is to be found respectively in j1.4 and j2.2. The term sequence may also be applied to mappings having I or another linearly ordered set as index set. This usage broadens the notion, since while such sets can be re-indexed by N (see above) this cannot be done while preserving the original ordering.
Mathematics
10 1.3 The Real Continuum
The real-number continuum R is such a complex object that no single statement of definition can do itjustice. One can emphasize the ordinal and arithmetic properties of the reals, or their geometrical interpretation as the distances of points on a line from the origin (the point zero). But from a set-theoretic point of view, the essentials are captured by defining R as the set of countably infinite sequences of decimal digits, having a decimal point inserted at exactly one position in the sequence, and possibly preceded by a minus sign. Thus, the real number x can be written in the form X
'x
,,,(x)1(/)7-)j(.x)10-f
=
(1. 14)
i=1
where the sequence (J1(.x), J2@),...) consists of decimal digits (elementsof the set (0,1,...,9 J), px) e (Ntldenotes the position of the decimal point in the +1 if k 0, and otherwise (the string (the decimal exponent), and mlx) all of the decimal expansion sign). When dix) 0 for but a finite number terms, of x is said to terminate and the final Os are conventionally omitted from the representation. -1
.x
=
=
by (1.14) is not always unique, and there exists a 1-1 correspondence between elements of R and sequences (>,#,#1,#2,#a,... l only after eliminate arbitrary of leading zeros we must the latter are excluded. To certain 0. And since for example, 0.49999... (the stipulate that dj y: 0 unless p sequence of 9s not terminating) is the same number as 0.5, we always take the terminating representation of a number and exclude sequences having di 9 in al1 but a finite number of places. R is of course linearly ordered, and in terms of (1. 14) the ordering corresponds to the lexicographic ordering of the sequences The representation
of
.x
=
=
fmsmpnmdk,mdznmdL...l.
The choice of base 10 in the definition is of course merely conventional, and of the alternative possibilities, the most important is the binary (base2) represn-
tation,
x
=
>(x)
2d*J>7n(x)2-i f=l
,
(1.15)
where the bi are binary digits, either 0 or 1, and qx) is the binary exponent.
The integers have the representation in (1.14) with the strings terminating elements rationals of R, being those which either after px) digits. The are also terminate after a finite number of places, or else cycle repeatedly through a finite sequence of digits beyond a certain tinite point. The real numbers that are not rational are called irrational. The irrational numbers are overwhelmingly more numerous than the rationals, representing a higher order of infinity. The following is the famous argument of Cantor. ddiagonal'
1.6 Theorem The jet R is uncountable. Proof Assume a 1-1 correspondence between R and ENexists. Now construct a real
Sets and Numbers
number in the following way. Let the first digit be different from that of the real number labelled 1, the second digit be different from that of the real number labelled 2, and in general the nth digit be different from that of the real number labelled n, for every n. This number is different from every member of the labelled collection, and hence it has no label. Since this construction can be performed for any labelling scheme, the assumption is contradicted. w c, where c is the cardinal
number
of R The linear ordering on R is of interest to us chiefly for providing the basis for constnlcting the fundamental subsets of R, the intervals. The set A (.x:a < x < &) is called an open interval, since it does not contain the end points, whereas the interval # = (x: a f x S /71is said to be closed. Common notations are gJ,pj, asb), (tz,:j and Lanb) to denote closed, open and half-open intervals. A set containing just a single point a is called a singleton, written ftz) Unbounded defined by a single boundary point, are written intervals such as C = (x: a < (J,+x), (-x,:), and (J,+x), (-x,:j for the open and closed cases respectively, where the infinities +x and are the fictional (notelements of R) with > +x < for al1 R respective and An important example properties x the xe is the positive half-line (0,+x), which we will denote subsequently by R+.
We say that d ()
<
.
=
.
.x1,
Kpoints'
-=
.r
-x,
.
1.7 Theorem Every open intelwal is uncountable. Proof Let the interval in question be anb). lf a < b, there exists n k 0 such that the n + llth term of the sequence msmpntndjsmdzn...) in the expansion of (1.14) defining b exceeds that in the corresponding sequence for a, whereas the t'irst n digits of each sequence are the same. The elements of a,b4 are those reals whose expansions generate the same initial sequence, with the (n+ llth terms not exceeding that of b nor being exceeded by that of a. If a and b are distinct, n is finite. The result follows on applying the diagonal argument in 1.6 tb these expansions, beginning at position n + 2. w Other useful results concerning
R and its intervals include the follpwing.
1.8 Theorem The points of any open interval are equipotent
with R
.
Proof This might be proved by elaborating the argument of 1.7, but it is simpler just to exhibit a 1-1 mapping from R onto asb). For example, the function
y
=
a+ b
2
for x e R fulfils the requirement. 1 9 Theorem The reai plane R2 *
Proof In
+
(b
2(1
-
+
tzlx
(1.16)
IxI) .+
. =
R x R is equipotent with R
,
view of the last theorem, it suffices to show that the unit interval o Givep points x, y e g0,1J, define 1q (0, is equipotent with the unit square (0,11. the point z e (0,1j according to the decimal expansion in (1.14),by the rule
Mathematics
diz)
di-vblllx),i odd =
Jf/2@),
i
1,2,3,...
=
(1.17)
i even
In words, construct z by taking a digit from x and y alternately. Such a z exists for every pair x and y, and, given z, x and y can be uniquely recovered by setting
(1.18) This defines a 1-1 mapping from (0,112 onto gO 1j as required. w ,
,
This argument
can be extended from R 2 to Rn for any n e N.
1.10 Theorem
Every open intelwal contains a rational number.
,
Proof This is equivalent to the proposition that if x < y, there exists with x < r < y. First suppose x 0. Choose q as the smallest integer such that qy > qx + 1, and choose p as the smallest integer 1/(v qy. Then x < (,p 1)/ < y. For the case < 0 choose an integer n > < r < n +y, .x < r- n < y, where r is the rational satisfying n -x),
-x,
.x
-
.-x
above.
rational r
exceeding
exceeding and then found as
.
1.11 Corollary
Every collection
of disjoint open intervals is countable.
Proof Since each open interval contains a rational appearing in no other interval disjoint with it, a set of disjoint open intervals can be placed in 1-1 correspondence with a subset of the rationals. w The supremum of a set A c R, when it exists, is the smallest number y such that x S y for every x e A, written supA. The inpmum of A, when it exists, is the largest number y such that k y for every x e A, written inf A. These may or may a, and supgl,') not be elements of A. In particular, infgtz,hj inftc,#) suptc,:) b. Open intervals do notpossess largest or smallest elements. However, every subset of R which is bounded above (resp.below) has a supremum (resp. intimum). While unbounded sets in R lack suprema and/or infima, itis customary to called the extended real line. In V, every set has define the set V R t.p ( number or +x, and similarly, every set has an either real a tinite a supremum, infimum. The notation 2-+will also be used subsequently to denote R+ k.p (+x). .x
=
=
=
=
-x,+x),
=
l.4 Sequences of Sets will be written, variously, as (An'. n e IN), (A,,)7, Set sequences fA1,A2,A3, or just fAnl when the context is clear. with each member of A monotone sequence is one that is either non-decreasing, n), being contained in its (An the sequence successor c An+l, V or non-increasing, with each member containing its successor (An+1i An, V n4. We also speak of increasing (resp. decreasing) sequences when the inclusion is strict, with c (resp. D) replacing c (resp.Q). For a non-decreasing sequence, define the set A ...)
=
U';=1A,,and for a non-increasing
sequence the set A
=
fV=IA,;
=
(U';=1Aj)f. These
Sets and Numbers
13 '1'
sets are called the limits of the respective sequences. We may write An A and d,, A. 1 A, and also, in general, limrl-yx,As zt and A,, --->
=
1.12 Example The sequence ((0, 1/n1, n e (Nl is decreasing and has as limit the singleton f0). In fact, limn-+x(0,1/n) = (0) also, whereas 1im,,-4x(0,1/n) 0. The decreasing sequence of open intervals, ((tz 1/n, b + 1/n), n e EN), has as its (J, :1. On the other hand, the sequence ((tz+ 1/n, limit the closed interval its limit is b4. a, a b 1/nj, n e IN) is increasing, and =
-
-
is non-increasConsider an arbitrary sequence (zt,,1.The sequence Bn U,a'='=,#4,, = exists. superior is called limit of the This set the ing, so that B limn-oon (Anl, written limsupnAn, and also as limnAa. Similarly, the limit of sequence the non-decreasing sequence Cn F1e;=,,A,,,is called the inferior limit of the written liminfndn, or limnAr,. Formally: for a sequence (A,,, n e EN ), =
=
sequence,
oo
limsupz
O
=
n=1
n
liminfAn
U n=1
=
Cr
UA,s
(1.19)
m=n
OA,u1 .
m=n
/
(1.20)
The limsup is the set of De Morgan's laws imply that liminfnAn = (limsuwzvlf. elementscontained in infinitely rrltwz of the An, while the liminf is the set belongingto all but a Jl/c number of the An, that is, to every member of the sequence from some point onwards. These concepts provide us with a criterion for convergence of set sequences in genergl. LiminfnAs ? limsupnAn, and if these two sets differ, there are elements that belong to infinitely many of the An, but also do not belong to infinitely many of them. Such a sequence is not convergent. On the other hand, if liminfuA,l = limsupnAn A, the elements of A belong to infinitely many of the An and do not belong to at most a finite number of them. Then the sequence (An) is said to converge to A, and A is called the limit of the sequence. =
1.5 Classes of Subsets The set of a11 the subsets of X is called the power set of X, dnoted 2X The power set of a set with n elements has 2 elements, which accounts for its name and representation. ln the case of a countable set, the power set is thought of fprmally as having 2 20 e le ments. One of the fundamental facts of set theory is tht the number of subsets of a given set strictly exceeds the number of its elments. For finite sets this is obvious, but when extended to countable sets it M to the claim that 2 g > do. ?mntj .
n
1 13 Theorem
.'
')'. ()' td
'' -'
.
Prof '
. . .
.. .
q
E
..'
..'
.
.(
;
))t () it)(lit'(. . . . . '.. .:
'
.
c.
.
.
.
' '
' .
,'
i
(..
=
The proposition is proved if we can show that 2 is equipotent with R or, qivalently (in view of 1.8), with the unit interval (0,1j. For a set A G 2 x (.2.1liitsftqt . th sequence of binat'y digits (:1,hz,,... ) according to the rule,
E . ..
.
2B0
:
. . . ... .. .. . .
-.
.
,
Lbn
Mathematics
14
= 1 if n e 4, bn 0 otherwise'. Using formula (1.15)with m 1 ad q 0, let this sequence detine an element XA of (0,1)(the case where bn 1 for a1l n defines 1). On the other hand, for any element x e (0,1q, construct the set Ax G 2N according to the rule, n in Ax if and only if the nth digit in the binal'y expansion of is a 1'. These constructions define a 1-1 correspondence between 2 ENan d (0 11 w =
=
=
=
'include
.x
,
.
When studying the subsets of a given set, particularly their measure-theoretic properties, the power set is often too big for anything very interesting or useful to be said about it. The idea behind the following detinitions is to specify subX sets of 2 t h a t are large enough to be interesting, but whose characteristics may be more tractable. We typically do this by choosing a base collection of sets with known properties, and then specifying certain operations for creating new sets from existing ones. These operations permit an interesting diversity of class members to be generated, but important properties of the sets may be deduced from those of the base collection, as the following examples show.
1.14 Definition A ring S is a nonempty class of subsets of X satisfying (a) O q S. (b) lf A and B e R then A wB e S, A (AB G R and A B e X. n -
One generates a ring by specifying an arbitrary basic collection C, which must include 0, and then declaring that any sets that can be generated by the specified operations also belong to the class. A ring is said to be closed under the operations of union, intersection and difference. Rings lack a crucial piece of structure, for there is no requirement fclr the set X itself to be a member. lf X is included, a ring becomes aheld, or synonymously AC, this amounts to including all complements, and, in an algebra. Since X-A view of the de Morgan laws, specifying the inclusion of intersections and differences becomes redundant. =
1.15 Dennition A tield 5 is a class of subsets of X satisfying (a) X e 5. (b) lf A 6 5 then Ac 6 T. (c) If A and B G 5 then A t..p # e @. n A field is said to be closed under complementation and finite union, and hence under intersections and differences too; none of these operations can take one outside the class. These classes can be very complex, and also very trivial. The simplest case of a ring is (OJ. The smallest possible t'ield is tX,O). Scarcely less trivial is the AC, O ) where A is any subset of X. What makes any class of sets field (-Y,A, interesting, or not, is the collection C of sets it is declared to contain, which for the class. We speak of the smallest tield we can think of as the generated by C'. field containing C as Rings and fields are natural classes in the sense of being defined in terms of the simple set operations, but their structure is rather restrictive for some of '
,
tseed'
tthe
Sets and Numbers
15
the applications in probability. More inclusive definitions, carefully tailored to include some important cases, are as follows. ,/
is a non-empty class of subsets of X satisfying 1.16 Definition A semi-ring (a) O e $. then A /'n B e (b) If A, B e where (c) If A, B e and A i #, H n < cxa such that B A U7=1G', e = f''h for each j'. O j, n and .f.
,/
,/
./
=
-
.
.'
.
More succinctly, condition partition into '-sets.
that the difference of two '-sets has a finite
(c) says ,/
is a class of subsets 1.17 Dennition A semi-algebra (a) X e (b) If A, B e then A chB e (c) If A e $, 3 n < x such that Ac Uyzlcj,where for each j, j'. a
of X satisfying
.f.
.l'.
,/
=
.
e
.t/?
and C) ch Cj'
O
=
A semi-ring containing X is a semi-algebra. R, and consider the class of all the half-open intervals and Iz < together with the empty set. If h (J1,:1) for I (tz,!?J (J2,:1q, f-'h and 0. And if fl = (tza,:2),then fl h is. one of h, h, (tz1,:2), c 11 (tz2,J11, (:1,:2j, so that a) f tz1 and bj f b1, then h h is one of 3, semi-ritzg conditions defining therefore and The sat(az,atj t.p (:1,&2J, h. a are isfied, although not those detining a ring. If we now let R be a member of the class and follow 1.17, we find that the halfand (tz,+x),plus open iniervals, plus the unbounded intervals of the form In O artd R, constitute a semi-algebra.
1-18 Example
Let X
=
=
a S b
<
-n
+C,o,
=
-
-xtbl
1.6 ji gTa Fields .
. .. .
.
.
Whn we' say tht a field contains the complements ;;!,' t!)'--,$@. ,;' y' .j' r' ;'(k,-':' ..,' iip'.l!:d @'. j,' jr'r' (@.-)--81'* :)'. ):'. '''#' r', jjli'. yrd jd r'. ;d 11,;::2:r11.-*. :-'. i'-' t:'.-'-ll)ll'-' qrrrdkiy--i---.d-,jtls'ltdkdi.lhlikirrstipp.:fl.rd pFq'. yy.;'. ,;r'j-. y' yjd ()..' jt'. ijd k' j'yy.'-r)'.' 133*.* j'i'.i (,'. ;);.;-' ;'@@' r' (y'. j.' )' i@'. j..' t'' )'()' y'; g' q'd ('. (';' kt'qi'. ''Ikiii!r:-'. ((' t,,(,t'' ?' L' ..' yjjjjjyryjjj.,,d :::*-2)1-.* Ft@-'. !!)p'.(' py'i'l'g )tI-'. ytd yy..,';.-: y;j-yy;',. (';' .d 4j' ('. .LLLq'. .:!1211-----*:11... :I.EIII.-.CEII-.'. 'Iii!'.k.d ''Ikii:rr:-d q!IkE7'r!'k..'. .-'Iki:t:i;'.. 'q'IlkEjIIk,'. (.'. ,''-r.-'II::7!IIk... I'EEEII.'. ijyipl,.dtsil----'k-ljptjs..!rrl ,-jb'L)j-..L'tjLkL33,?.,.q-',i-..-j,qk.. t(' .y)'. tjd t'j,'t)' .'')'''';:E:!!.,-. ''k::E:!:.' k.-'-'''Ill::iEl1I-..t.''-.tt'$'12ii!I.-.'. rpr'y r' .-'. r'.' .-' '1I(ii''' (!11:::1,-* 4111E::::,..*..':!1L.*. it:ipd rjqj-dltdllt.-l'ttd desyrves explanation. It is clear that z4l
and finite unions, the qualifier A,, e 5 implies that U& j=3 A) .#hit lteratlon of slmple by n-fold 5 But, the constructlv: palrwlse unlon. e glven t t: (j jm. tj (jj'I Out W 1* legiti 1 k. tju further 1 mate ttlr th t ti t no e e s on q':j).: .) . ' a s P ul ati O n to r o t? 41@.;: )' .(ir.. ytt,rtlh be taken llmlt. a.,g, the By maklng n.operatlon can thls addltlonal to ))(, j).jt (..... .. . . . 'lk::Ei!lk.. @ E..t!1IIp:,I.i.. 'r '. E. !!I1L., :111:7:!1.-. ' ' ' :..-t.'E!IIk..' . .-'.EEE:!!:. ..,.jy-...'- ..:..'.. '. ' .. . :111:7:!1.-.. r,,;-. Jl'... ' ' ' '..-.. .' ' ''. . rr ... '.r.-'...(. . . .'.'i.7tr. i'.' ..luju,-..t9'.j:;:,.j)..jr, r.... f.tltz... jyy;;.jjy..... jj.lyytjr .. .. ...... .. . ;.g,..yt:jj,j.. . ..*1.: j., A .'.j'tp:('.()) r yjj''.'gt). (tsajgebra)5 ls a class Of subsets Of X satlsfylng ('. f l ...;...: uljjjtl .......,..y 't;. r;J) .j . j ?)...j!.y. -y..F..:yqqy t)?.p:. (y .kly(..yy.yq..:)...q ):;1211E11.... .!kjjj;;r;;)., .. ' ..
.' .'.' ':jls')d .'. '-' -j--!,r,!tk'. -:'. --y!j;;'-j-.' -q' -,.'tp..-.(t. -'k' -(r)'. -8)* y' :')' 24jg* -'. -' j;d ('. !111:(q*;LL.' ;..(' i.'. k' .-'. -'-' --'' -,' j'y' -:tr!)i...ti:,jp':F'--)?;t--''' E' ,-.'itt-,ttd ,'-'.k'(.E ;E-;r'.' ?' y' lrrq))d :':11,*. :L';'' ''. g')j;-',' j';'y' y't':. i--d r' s.d ,'g'' E'. )-''E -;.' ''IItEL.FX -' -,'.-y' -'. .)' .q' ''. k' .:' ,''(''jy ;'))'. ,'..(.. (t' ?.' :y'-' ;;' yyd ((' ;'. :('. F'-j. !': q' (j.,' F' C--qy;'E :'-'E' :)')' L3--'; :;' ''-.' ;,'-'L'' ,-'' )'
,-,' -' .','.:'.. -' .','.-tq'-' -,-' .)r,' ,--))'-' ,,,' 'jjjj:yr:jjj,'; .'.'. ''.-'' .!112::*. .'. ,'.'-'. 4:11::.* '''bii'''bi''-''
'.'. .''''--d .'.' .' . .'r:).. ... ''.' '..'. '. ..' . .' . .
'.
:
...
.
.
....
. E- .
(. '
... .
.
.. ' g' '
.
q
(
.'' . .' '( . . EE
...
.
. .. . . .. . . (j;) :.. . . . :
.
.
E
t
,np
E
.
:
.'
.
'!;'j .L'
'. 7 .!L(.
...;
j, ....j .; ,4k .:f .(a z ( .
.
.
. -
-.
..
....'
j.(
...:;.. .
...
...
-
.
(
.
....
'
.'
.. .
g
...
.
7
.
..
.. .
'
.
.
..
..i
..k
. .
.
.:.
.
..
. . ' .
.
.
.
..
dp
4)
..
-11::::::,11;-.
.11:2:::,...
,11::2::,1.-
.
:.. .
.
:
,
. .
y.
.
.
......'
'lk:!r:-.
,y
..
-
. .. .,k;)jjjtyy........
(lzzllp'l. ' (jj:jjj.. (j;jj..,...
.
y.
.
.
.
.
,,
.
,,
,,
7.
u.....r.
:..,. . .: . ... . ;. . ..;.,k y . (. .;..LtEL. :. . , : ry q..;..y.... jjggggyy;: . .. .-'E . .. .y E E. CE.; . . ;.... : .)Ey. -... . r :. . .E . ---i .. . ,. q .. ..E . . .q ,. jy. ..g .y.. EitttliyitztlrE . . rtjyEiljzqttigryyf ,.. . q - yy q . j . .. . . q . y . . .. j, . . . . koyyyyyyjgy . y. . . . . . yyyjj yy. . yju r . ' i EE yu y . E !' '(' :. q: i E.'rit:-. g l.E; :);y.. g yy g ().. y . .. g y y: .. jy q; ; L y. . . r Lj.,L..LjL; I.t: jL'L'q(jLL'; y. '.t;tEqq;;' gE.y))gy .. y j, . . .... .. ... , . . . . . . ....yy .q :.q( .;).(,LL.;..;.. . . . y y. r.@'ry'.'y..r?i..E.!:..... kjjjyj .;. rykt',.E..yi. ....q .. y.; . y.p; y. . . .'. .)L;jt..).q(y,jj,d!!1i-,; :.( ..E.. -y ....) ( ?.jj,.. y-1... y ' ' E i i. E - rFti... t E .j;. ;!.. E (! . !- E . jygy. . - ...;:.. E..- . E. . (E .. E ! .... . : . .;.. .kyk. . EE t'F-i''' - q (. . -. . . ...rj.. . -: -. -, . kL. . qE. i . -; -y.,.-.g.1.j..-..d..,)....y.;,..y...r.-y.-.st$y-. .. y . .;.. -. .y '. EE.. ; . t; . . . ,.f33f. .E E . . .. .- y . -. ':$l8t,E-.k;#)),pl-(i'.-.'j,b.slb . .....p.-.. .'. --..'-... -.pt -y. ,.)ki-f- --.--; #..... .. . ,,,gt;:t!jj:jjj.j .
dp
:1
.
.
.
..
.
.
q . ..
..
..y).
...
-
41
...
..:..:.
.i'.
1p
...t..
.
...;;.;,,
.
.
. .
...y;'y'y...y(iy(y)(yy:jj..yy..:y(j....
.:jy.yj...
2
.
.
:
.
.'
<.
.
.
.2.
.
...'
....
..r
:';'
'
.
.
'
.
......
:
...
.
..
.
,
-
.q.(
'rt,ywpy
..
. .
...
-
EyEi yt($.?E.r(r . .. ,E 'E. i.E k'JE C4jIIE. ..L 3.k() LLj;L.L .(:.g).-. E ;i.EE-)!'(; . ttjii;. )....y.2..2...,..J): :7.:..
E..y.(.
.
: :E..
LjL.L..kL /. ..
..
.
.
-:--
-. - . ..j .. .j(;E ;.; . (y.jjj.(q.. sl/tq E .
..
. .
'
.E.. . .. j q. ; E E/E.Lq ..L.. . . ... . . ) .. --(j ( . . --. ' . i . -.' --- . -.. . . - . E. .-.. . . -. -:--'. -. ..' ' ... g.. t):;.... ..q/.:I.jLL . .. : g (! .... j ...:k..j..g . . LLL.. ;.r .y y.. . y. gy y . .:yyy ... ' bs? qoujyyjd . ;
...:
.
,
o
.
.
E
.:
...
...
'
'
,.
.
:!)
.:
.
'
; . .j . .. . . ... . g . .. 4 . '-'112::..
iE: ':'E: . -.' ..;E.) '.'.E
...;.
.
. : f . !..p ' ... L. ' '. . L...'...2.r .' ' ;''. ' . ( . (.' ,u..km,,,,o,k,,,.,r+y,,,.,,a, .
.
-
E.'.p.:
)Eq
.
.
: .
.'
.. .
...,
..
.
. .
.
..
.
.
.
. .
.
.r.. . g ... j . ., . ...... . . ... .4: j y. .yyjaj . ts. )j; . j .. jjjj:jjj'. !!!' i.$. jys,lt. . k.. i.. . .ty jyy )- jjy . yyrtyj g '#:'l:'j':.j'.jr:!.k::: .. .yy . yyy gc y i: . . l. .r.)(k!r/.:i..tt'.j:r. .tr(g.(..). r. i)'i;r... yt .tk-. . ttr;: .;... ryki j yy y y. t. r..r) ti).;7..i-tl ... j;... t). . ryli l.t:.. ,:,.,:,, ;)).y;..)-k.t8.l9)-., gy'tt..kq.j4:yy.-.-..,.it::!:j-.,.. L,tjtLL r,.. . 442!2:r ,-L..).;;;.)-L66LLj6q?''' jj:rljjs 41222:; jjzkr:i 44:22,), .(::.-tgy..tr..;y(jtty . ...E .-..(jl. 'l.s' 4(222:, 4(22::; jjjj;, rjrr, ,...EEEEEE. 4k22::; ..j'tj'L. .1k.k,j. (jlr:jl. jljrhhhjj :jjjgCIE1j,. :jjji rjjjq- :jjj;, (jrr:lk. :.1-:.9. (jrrjl. :;:;::::;' ; '. ).. i-.ql..?-' 4(4E ,, IILLL,.:LL..).L (jj j(.@i. .-. .:y. i.-. t; t. p ;. )i. . , E. .. t. El(;. jr. .. r ; ;. tgg j)rpi j-. :18-. (..?- rjq. jyy.i . j-y . t-. ;. .-. j.. . ))-. . . '.i . r. . . .. r,r .' ty. (y;j--.;.kr. . . . ;. .. . . . .. . (. yj).r'!jz'.,..,jp'.'..-''.. . . -,y.,4tE:1'. :lir4:::: 4t::. 1. qlL y.-y1,y-, 4:::,. ....,....,.)s,j.,, 112). 4:2:: 4::2.::;7. ;p';!r 4:::). 1ki2: :5'k 11. llrlu 4(22). 1L )y.j. .l,'.k' .::2). .. L6:-. 1ty!1.. 1:EIL llrqlrrkp IIF:I::IL llrl. :5,k .. .... ..-!1-2Ik. h..dki:r (1. qlrl.IEEIk . .:E:j.. 1t) 1l: IIFI. Jlr!l. ql. Ilrl,) jll:)p .11:)'L.. tdk4(22). !rdtgr11.,1. ki-b.' u. Jlrck!q;l.. . .'... . ' L..;L.-.?-..... k,' qtthk. t.. .r)k.-...y . tl.#-''-it--.. ..----. --L(-q...L.t..-. tl),,:... . . . ytyth, t. til. u Cit-'.' . laws, of countble 1ntersecti on also. A . t.g cqqy 7. tp,y E. E ::.rgn ' .E ..r.;E-.:. ng can EF. jC.y E. . g.. .--.. t.. . ykkE (j,.t, . . y 2 ,t ...E.: j....y t( q.,y...y.. .,p... .(),. 'tj. , . .. .. ,ar. . J?'1'l ,. ;w.. tykk,yyal. @(r,t t7l7)li)'yE--jp:ytq-l'h@i'-.--.j#i.tt)r:-.)k2!l$,j);.. ;,; , , .. t..Lti-.(yy....hk-...'i'..-.-i'.---j;.rls tlot , t ,-,Ljb;LLb6.. j, ) ln the sequel )j.rq( shall t;need -)ykyy.tjj.. t:-.t.-)-t...?t-(..--.-..;-yy..,F.rconcept . -?,--y)k-ttg.-tjE '...-..-.--..:.--r-. we g ..yqE a E ;..-...y..-.:..)...;..y..--.y.y....-........... ),,r.tg2s;-...:,:.5-r):(t..---r@t()kttr;jpki-):.y.t2r).-. ,-.)j.Lb1,,.q--. (;. .y......-.. . .. .. .. . . ,,.tzk r:kjt. 1)4 ? E y,pqy),jj 33:13333,..6,31)3 qtp. 2L2t1jjk1,(1.. yo,i, . .. q l,. ;,?,. q?. y. . j,yj(. lnt:rsectlon (E. . . . .qk .. : :6. :;)..y.,@, of al1 the jyjj,, .( jy .yj ls j.j..j. ,,,. yjyg.. yyt;,y( tygyj;. contmnlng y.y,rty..y..j, ,. . : '! ;(y.tjrj. . (j(:,.tjyyy. .. . y . ty.j .. .. . . . . sj jyjjy.t.yjgy, LILLIL) jjjjjj,yyyyyjy.. yyyyy T. .r.. . .. .)g-jjjy)j.).)4jj..j)j.:.,;j.);.j,j;.jj;.)j,.j.L..,.;..j ..., y. .........j3b;,jbbb;$LLL,).jtfjj;6;3$j(. ..g ...s.sy . ... .. .. .. . . .. . .- . . :7',t.,..;r,,#.-. )),.jjj ). .. ..... . . .jj,.., ...yj.,.. . .'''.. . .j.q,jj. . . . . ' q$3t... ,;-. . . ..' r-,-iiq.-r),-iyr,!!.. j. .=-. $.-<..., . . . );,-':'!'tk. '
. '
.
-
..
.:.
-
.
-.
.
''.
-
-
.
..
.
. . .
-
.;
.
,.);.-')
-
,j,q2..
'
.. . .
tju
..
-
..
,
- ,.
,.-
- ..y;)
-
.
..;.
-,...
..
,..
..
.-.
.. ..
y
:.
.
..
. . .
.
(gjj.. -0. ' s.t.).....,-.y....ytt,..c...y.-....1. ?t..'tlt.t.,t.iy.'t?.'rtA)'.i.' .d.e.. '':.('. e..,4(i!I... ''-. . . %)' 4.) 4)d..,'',,M ttytt/p ''4kk,,.,)'#',,,..)4i yt t 14thls iis 7>1'h: j x,,soy s,: xe.)ts..T?th: ,, .
. . .
.
-.
-
.
-.
,.:r
,.
-
.
.
.-.
-
.
.
.-
.
.
.
.
.
.-
. .
.
.
.
. .-
.
. -
.
.
.j(rtrp1k.
.jry,
,,...
.
.
.
. .q;,..
-iji
.
-.
-..
.-
'
.
-.
. .
-
1k-,,,,-Jj
rjjj;;!;;q
.
.-t;,
. .. .
..
.
.
. .
. ..
.
,,,
-,,
.
.....';-.7.
..:.,2,,
'
..
.
,-,.
c-n'
',,
.;
''
,;j),
''j-.k,
-
. . .
,..
.
.
.
'.
--k...
-. .
.
-
. ..
-
.
.
--,.
..
.
be
,,,
.
.
.
..
-
.,.
,,
't(l!jt)t2),.
..
-@jjjgjjk.:
.,i(-$;ydjy-.. .....
. :
.
'
.
.
:
.
..
.
jlj..
. yyjjggk., ?jjy-.$,,;gjjjr..tyy(,.k;.y;jy;j.-
:jjjy...
.
y
.
..
y.s,.y,.y
.
.
.
, .
.
. ..
..
..
'..-,,ibn
y,.;.y
... ,
.....,.--
.
k
s..
,
...
.
.
,,
.
...)...j....j.,..
,.yyy....y,y.y
..
.
.
.,.)......),),.....,).j
.:
.
-..!it,..
....
.
,q
kf,.
,.
.
.
,,
.yy
,4j,j)),,;jjj,-j)))jj)..,;;,)j,,j...
..
-),-.t..
.
.
.
.)
-,,
,.
.
..
,,
.
....
,
.
.
.
,
..
c-flelds
--
.
-,
C
--
.
16
Mathematics
generated by C, customarily denoted c(C). The following theorem establishes a basic fact about c-fields.
calledthe c-field
1.20 Theorem If C is a finite collection c(C) is tinite, otherwise c(C) is always uncountable. Proof Define the relation R between elements of X by Ry iff x and y are elements of the same sets of C' R is an equivalence relation, and hence defines an equivalence class & of disjoint subsets. Each set of & is the intersection of a11the C-sets containing its elements and the complements of the remainder. (For example, see Fig. 1.1. For this collection of regions of R2 & is the partition defined by the complete network of set boundaries.) lf C contains n sets, & contains at most ln sets and c(C), in this case the collection of all unions of &-sets, contains at most lln sets. This proves the first pm of the theorem. Let C be infinite. If it is uncountable then so is c(C) and there is nothing more to show, so assume C is countable. ln this case every set in & is a countable intersection of C-sets or the complements of C-sets, hence & g c(C), and hence is the collection of a11the countable unions of also U4&) c(C), where flX&) t/(&) uncountable, is &-sets. If we show the same will be true of c(C). We may assume that & is countable, since otherwise there is nothing more to show. So 1et the sets of & be indexed by N. Then every union of &-sets corresponds uniquely with a subset of N, and every subset of ENcorresponds uniquely to a union of 2N, &-sets. ln other words, the elements of U4&) are equipotent with those of which are uncountable by 1.13. This completes the proof. . .
,
R, and let C ((-x,rj, r G (E1 ), the collection of closed with rational called Jltilt/'-ff/r6y endpoints. c(C) is the Borel jield of R, generally denotd B. A number of different base collections generate T. Since countable unions of open intervals can be closed intervals, and vice versa, (compare1.12), the set of open half-lines, ((-x,r), r G (EI), will also serve. Or, letting (rnJ b: a decreasing sequence of rational numbers with rn 2, x,
1.21 Example Let X
=
=
(1.21) Such a sequence exists for any G ER(see2.15), and hence the same c-field is generated by the (uncountable)collection of half-lines with real endpoints, ((-x,x), x e R ) lt easily follows that various other collections generate S, including th open intervals of R, the closed intervals, and the half-open .x
.
intervals. n
1 22 Example Let X given. It is
T
=
F, the extended
real line. The Borel
field of F is easily
# e Sl, where B is the Borel field of R. You can verify that S is a c-field, and is generated by the collection F' of 1.21 augmented by the sets (-x) and V. n =
(#, B tp (+xJ, B t.p (-x4, B QJ (+x) t-p (-x):
1.23 Example Givn an interval f of the line, the class
BI
=
LBf-h f: B e B ) is
Sets and Numbers
called the restriction of T to 1, or the Borel field on 1. ln fact, T; is the c-field generated from the collection C ((-x,r1f-h f: r e () ) n =
.
Notice how c(r) has been defined from the outside' It might be thought that c(r) the inside', in terms of a specified sequence of the operacould be defined tions of complementation and countable union applied to the elements of r. But, despite the constructive nature of the definitions, 1.20 suggests how this may be impossible. Suppose we define .Wlas the set that contains r, together with the complement of every set in T' and a11 the finite and countable unions of the sets is not c(C) because it does not contain the complements of the of r. Of course, be the set containing W1together with all the complements and unions. So let in the same finite and countable unions of the sets in W1.Defining 45, manner, itmight be thoughtthat themonotone sequence (W,,) would approach c(C) x; but in fact this is not so. ln the case of the class fm,l!, for as n it example, can be shown that Wx is strictly smaller than c(C) (see Billingsley IX This fact is 1986: 261. On the other hand, c(C) may be sm aller than for in again fm,1), j3.4. demonstrated, The union of two c-fields (the set of elements contained in either or both of them) is not generally a c-field, for the unions of the sets from one tield with those from the other are not guaranteed to belong to it. The concept of union for c-fields is therefore extended by adding in these sets. Given c-fields ; and N, the smallest c-field containing a11the elements of 5 and all the elements of Wis denoted 5 v V, called the union of 5 and V. On the other hand, 5 r'A W (A:A G . and A e V') is a c-field, although for uniformity the notation 5 ZN V may be used for such intersections. Fonnally, 5 zq N denotes the largest of the c-fields whose elements belong to both 5 and V. Both of these operations generalize to the count1,2,3,... we may define able case, so that for a sequence of c-tields 5n, n l;a. V= n n 15 n and 0* Without going prematurely into too many details, it can be said that a large part of the intellectual labour in probability and measure theory is devoted to proving that particular classes of sets are c-tields. Problems of this kind will arige throughout this book. It is usually not too hard to show that Ac e 5 whenever W e 5, but the requirement to show that a class contains the countable unions can be tough to fulfil. The following material can be helpful in this connection, A monotone class A is a class of sets such that, if tAn) is a monotone sequence withL limit 4, and An e A for a1l n, then A G A. If (Aa) is non-decreasing, then zl U7=1An.lf it is non-increasing, then A O0J=Iz4,,. The next theorem shows that, to dtenine whether or not we ar: dalinf with a c-field, it is sufficient to ,'krd r' y' jr';ty 't'ld r' yyy'.yyjjjjyjjjjjjiy.d k' t'(' y' );' jyyd lr'yyy p' )' 'y,d trytd td ttd t'. ild ctmsl'detwhdherthe limits of monotone sequences belong to it, which should often ' b j esiet to establish than the general case. .
tfrom
.d1
.4z
.44,...
--->
.
=
=
=
=
=
C' y' .'E
-'. .' -' :t' '
.'''
.' '''.'' ..'.'.'....'. ..'.' '. ' . . .'... . .. .. ..(E.
' . .. '
E
' g. ...
' . .'
r . .. : :
1);,4
.
. .lj ' .. . y. .. . . Flrqh:oykm
l q)
l g...(
.. . , . . .
l .
.
'
.
.
. .
.
.
?;; is a c-field iff it is both a field and a monotone class.
.t:)2
E
.
.
:.y
:
.
. . .. . . .r ..Eq(
,,k))
y.
t j' ' .
. . The..ftjjjjy, p tjtjjy .
''
of the
ito
is immediate. For
theorem patt the part y(t.: j. . ;) y y. j.r r ,'tlt:' t.ri.jjryry.yi.,yuit y 1:2j1. :jkq-.j(jjji. :g;pC;: ;(.):t..,.(.,)i'.:..'..y.;tt.t,jiyA......,y '.1t jkj:r qlj?ii. 4, )1:41.. .j1r:lg. 1:;(jj.. jpp:p-;,r 4:;:: ...... jlrrjs 4:2:: 422:: j,qbqf. . 4EEEE! jjr-iylr jjy-plrqjj .. jrrqls ,,krIg. :;; igjl. ji. ji. :. ,(EE L.,;.,jfL:..; ... j . I j!..I t;i:;r ni; (1q ... .g. )......)...... tjjii:;s ,b, 4:22EE jj tlij;r (yj,?)'' yj e. (. . k:j?,. .t.y(). r .., j; (jry-;;;(tj yy ). .s.u.. y... r(j-(.. (.; .i.qLL);,;j . y. . ... y... .;jtytytyt, . . . . tg -L$1t.. ttttt . . .;j )')r'--t)-tzjj i-.-tki -,.-(jkyr))-;-..---,r. y is . rj limit with =1A,, t monotone ;, );. by t sequence ) a ulyytjt U.-----.- e y-):-),::::r,:--.. --..L3LLLi..-. .,.-yt.-..,7r-...(-.......). r ;-)y)-. j/l-rjt--j,-....;.LL;L;.,.;j;;-),.,;b;qj, '''-?'''t''-' )jjt.-. (t)-.)rrtt.,..,)))-y)#)---.r),-.-. t;)--?--.--,r...r-....-.,... ...... ... . . .--. ... ;.,-(.$;.,q;-.,-LLLL1L--t;))plt:'-i yryjyti... E #,,.. .. kr. t).y-j-t,yr ....-..--...- ....-.... ttl'i?.-.-;r-..t...)k.-(--..-t..-.ti..-)-.,)---)'.-.-.t.....-..t-., .t'y4?.i@''. ... . ..Li q-?..'..I3;......;-;t.-b. .!.#,..-t,) llt))k...-.... tiy,-3$bqL-;--....').L ..::?)i..-..$kr'-'t)L' .. .)yyyiyyyi 14441111@11..... i
r
,.tE'jt.ty( : ;,
i E kE'
(
(
.
.
-.yy)j,ytj,.i...y.yE.y;;r.,...myt;y.U,
; -. . g . .
; ;
(
.
E
i..E -
-
q
,.
..
.
.,...
,,
-. .
-
-. .
- . . .
:.
-
.
-.
.E-...
,
q .
. . .
.. -.
.
.
:
E .
.
-
k.tlll.
..
-.
..
..
...-....
.....
..
,.
.
.
..
- ...-.-
.
.
. . ,
...
,,
-;::,---::--
.
-
........
-j
..-.
.,
.
.
..
...
'''--'f'':tt--
.....
define
,
-k,(
..
.....,.
x ' '--------8-'-:'''*''''% . -))
'if'
-jlqilii;,;j;r
.
....
..
-......-....
..
.
.
. .
.
.
.
.
.
..-
..
... ..... -
-p
. .
..
..
-
.
18
Mathetnatics Ur1=lAk, so the theorem follows.
assumption.Uo;=1An =
.
Another useful trick is Dynkin ' s zr- theorem.l To develop this result, we define two new classes of subsets of X.
1.25 Definition A class T is a z-system if A and B e T implies A rn B e T. A class .f is a h-system if (a) X e 1. and B (b) If A and B e c A, then A B e 1. is a non-decreasing sequence and Au A, then A e (c) If fAn e n ,f
-
'1'
,f)
.
Conditions (a)and (b)imply that a l-system is closed under complementation (put .f
m.Moreover, since (b) implies that Bn A,,+1 A,i e for each n, (c) implies that a countable union of disjoint d-sets is in 1. In fact, these implica-
X
=
=
-
tions hold in both directions, and we have the following.
is a l-system if and only if 1.26 Theorem A class (a) X e 1. then Bc e 1. () If B e is a disjoint sequence, then UnAn e (c) If (Aa e ,f
.f
,f)
.
Ia
ln particular, a c-field is a X-system, and moreover, a class that is both a is is a c-tield. This follows by 1.24, because a a-system and a class closed under unions by 1.25(c), and by de Morgan's laws is if a monotone closed under both intersections and complemntation. The following result makes these dehnitions useful. -system
-system
1.27 Dynkin's x-A theorem (Billingsley 1979, 1986: th. 3.2) If T is a a-system, ctfis a l-system, and P c 2, then c(T) c 1. containing P (theintersection of a11 Proof Let kTj denote the smallest containing P), so that in particular, (T) the c M.We show that 3.(:P)is remarks l-system. follow the above, it will then that (T) is a c-field, and By a 2, as required. hence that c(T) c &T) For a set A e 1(T), let Vztdenote the class of sets B such that A rn B e (T). We Clearly, X e Vz, so that condition 1.25(a) is shall show that gk is a satisfied. Let #1, Bz e 5k, and Bj c #2; then A rnfl e 14T) and A fa #2 e (T), and (A r''h#1) c (A fa #2), which implies that -system
-systems
-system.
(A rn #2)
(A t'''h#1)
-
A f'n
=
(#2 -
#1) e 14T).
(1.22)
But this means that Bz #1 e Vx, and condition 1.25(b) is satisfied. Lastly, suppose A f''h Bi e ,4T) for each i 1,2,... and Bi #. Then zt f-h # e (T) by 1.25(c), which means that 1.25(c) holds for Vx, and sk is a as asserted. Suppose A e P. Then B e P implies A PB e T p is a a-system) and since P i and '4T) (T), this further implies B c 5k. Hence P c 5k. Since Nx is a containing have in this smallest also (T) gk P, is the we case. So, c V',A and hence A fa B e (T). whn A e T, # e VT) implies B e We can summarize the last conclusion as: -
'1'
=
-system
-system,
-system
Sets and Numbers (A e P, B e 1(T)) Now defining Vs by analogy
(# e
(T),
yt
with
PB e
=
19
(A OB e
(T)).
(1.23)
9%, so that (T))
=
(A e Vs),
(1.24)
we see that (1.23)and (1.24)together yield P c T. Since Vs is also a by the same argument as held for Vx, and contains T, 1(T) c Ns by definition of (T). Thus, suppose B e (T) and C E (T). Then Ce Vs, which means that B f''h Cq (T). So 1(T) is a n-system as required. . -system
+.
2 Limits and Continuity
2.1 The Topology of the Real Line The purpose of this section is to treat rigorously the idea of as it applies to points of the line. The key ingredient of the theory is the distance between a pair of points, x, y e R, defined as the non-negative real number Ix y l what is formally called the Euclidean distance. ln Chapters 5 and 6 we examine the generalization of this theory to non-Euclidean spaces, and find not only that most aspects of the theory have a natural generalization, but that the concept of distance itself can be dispensed with in their development. We are really studying a special case of a very powerful general theory. This fact may be helpful in making sense of certain ideas, the definition of compactness for example, which can otherwise appear a little puzzling at first sight. An e-neighbourhood of a point e R is a set Sx, fy: Ix-yI < :1, for some s > 0. An open set is a set d c R such that for each e A, there exists for some e > 0 an e-neighbourhood which is a subset of d. The open intervals defined in j1.3 are open sets since if a < < b, : minl 1b l > 0 satisfies the la definition. R and O are also open sets on the definition. The concept of an open set is subtle and often gives beginners some diftkulty. Naive intuition strongly favours the notion that in any bounded set of points to' a point outside the set. But open sets are there ought to be one that is sets that do not have this property, and there is no shortage of them in R. For a complete understanding of the issues involved we need the additional concepts of Cauchy sequence and limit, to appear in j2.2 below. Doubters are invited to suspend their disbelief for the moment and just take the definition at face value. The collection of a1l the open sets of k is known as the topology of R. More precisely, we ought to call this the usual topology on R, since other ways of defining open sets of R can be devised, although these will not concern us. (See Chapter 6 for more information on these matters,) More generally, we can discuss subsets of (Rfrom a topological standpoint, although we would tend to use the term subspace rather than subset in this context. If A c S ? R, we say that A is open in S if for each x G 4 there exists Sxtzj, : > 0, such that .(x,E) f''AS is a subset of A. Thus, the interval (0,) is not open in R, but it is open in (0,1). These sets define the relative topology on S, that is, the topology on S relative to R. The following result is an immediate consequence of the definition. inearness',
-
,
.z7
=
.'r
-x1
.7
=
-x1
,
2.1 Theorem lf 4 is open in R, A r''hS is open in the relative topology on S.
l:n
A closure point of a set A is a point x e R such that, for every E > 0, the set
Limits and Continuity
21
is not empty. The closure points of A are not necessalily elements of A /'7 A, open sets being a case in point. The set of closure points of A is called the c losure of A, and will be denoted X-or sometimes (A)- if the set is defined by an expression. On the other hand, an accumulation point of A is a point x e R wich is a closure point of the set A f.x) An accumulation point has other points of A A, it must also arbitrarily close to it, and if x is a closure point of A and accumulation closure is point. A point that not an accumulation point (the be an of contains x former definition being satisfied because each E-neighbourhood itselfl is an isolated point of A. AC /-7 Sxnz) is A boundar.v point of a set A is a point e X such that the set points of A is denoted ?A, and X not empty for any e:> 0 The set of boundary AO A A. A closed set is one containing A t.p :z4. The interior of A is the set all its closure points, i.e. a set A such that A A. For an open interval A a,b) c R, X' (tz,1. Every point of anb) is a closure point, and a and b are also closure points, not belonging to asbj. They are the boundat'y points of both a,b) and gJ,?7). ,&(x,sl
-
.
.x
.x
.x
=
.
=
-
=
=
=
2.2 Theorem The complement of an open set in R is a closed set.
(:I
This gives an alternative definition of a closed set. According to the definitions, O (theempty set) and R are both open and closed. The half-line (-x,xq is the complement of the open set @,+x)and is hence closed. Extending this result to relative topologies, we have the following.
2.3 Theorem lf
:4
is open in S c R, then S -A is closed in S. cl
In particular, a corollary to 2.1 is that if B is closed in R then S r-hB is closed in S. But, for example, the interval l, 1) is not closed in R, although it is is open in (0,1). closed in the set (0,1),since its complement (0,r1) Some additional properties of open sets are given in the following theorems.
2.4 Theorem
(i) The
union of a collection of open sets is open.
(ii) If A and B are open, then A /-7 B is open. n
This result is be proved in a more general context below, as 5.4. Arbitrary intersections of open sets need not be open. See 1.12 for a counter-example.
2.5 Theorem Every open set A q R is the union of a countable collectin joint open intervals.
of dis-
Proof Consider a collection (&x,:x), x e A h where for each x, Ex > 0 is chosen enough 5'@,:x) Then that A. small i Uxex5'tx,:xlA, but, since necessarily A UxeA5'@,sxl UxeA.@,Exl, it follows that A. This shows that A is a union of open intervals. Now define a relation R for elements of A, such that xRy if there exists an open interval f q A with x e f and y E 1. Every x e A is contained in some interval by the preceding argument, so that xRx for all x e A. The symmetry of R is obvious. Lastly, if x,y e I A and y,z e 1' ? A, I /-7 1' is nonempty and hence I t.p 1' is also an open interval, so R is transitive. Hence R is an equivalence =
Mathematics
22
relation, and the intervals I are an equivalence class partitioning A. Thus, A is a union of disjoint open intervals. The theorem now follows from 1.11. w Recall from 1.21 that f, the Borel field of R, is the c-field of sets generated by both the open and the closed half-lines. Since every interval is the intersection of a half-line (openor closed) with the complement of another half-line, 2.2 and 2.5 yield directly the following important fact.
2.6 Theorem
B contains the open sets and the closed sets of R n .
A collection r is called a covering for a set A i R if A an open set, it is called an open covering.
UsswB. If each
B is
covering theorem If C is any collection of open subsets of R, 2.7 Lindelf's is countable subcollection (Bi e C, i e N ) such that there a X
UB U#. =
f=1
Be N
Proof Consider the collection (Sk Srkbs, rk e Q, sk e Q+) ; that is the collection of a11 neighbourhoods of rational points of R, having rational radii. xQ+ is countable by 1.5, and hence is countable; in other words. The set (E1 show exhausts by We k the set. that, for any open set B c R and point indexing e N a' e B, there is a set Sk G such that e Sk c #. Since has a E-neighbourhood inside B by definition, the desired Sk is found by setting sk to any rational from for s > 0 sufticiently small, and then choosing rk e the open interval (0,1as), 5'(a',/) as is possible by 1.10. Now for each x e UsewB choose a member of say Skp, satisfying x e Skp i B for any B e C. Letting k(x) be the smallest index which satisfies the requirement gives an unambiguous choice. The distinct members of this collection form a set that covers Ussc#, but is a subset of and hence countable. Labelling the indices of this set as k1,k2,..., choose Bi as any member of C containing Ski. and hence also for Usecs. w Clearly, U:'xlffis a countable covering for U7=15k, ,/
=
=
,/
.$
.z'
.x
.$,
,/
lt follows that, if C is a covering for a set in R, it contains a countable subcovering. This is sometimes called the Lindelf property. The concept of a covering leads on to the crucial notion of compactness. A set A is said to be compact if every open covering of A contains afinite subcovering. The words that matter in this definition are and Any open covering that has R as a member obviously contains a finite subcovering. But for a set to be compact, there must be no way to construct an inrducible, infinite open covering. Moreover, every interval has an irreducible infinite cover, consisting of the singleton sets of its individual points; but these sets are not open. Gevery'
Eopen'.
2.8 Example Consider the half-open interval (0,11.An open covering is the eountable collection (41/n,1j, n e NJ lt is easy to see that there is no finite subcollection covering (0,11 in this case, so (0,1j is not compact. EI .
A set A is bounded if zt i 5'(x,s) for some x e A and ir > 0. The idea here is that
Limits and Continuity
23
: is a possibly large but tinite number. In other words, a bounded set must be containable within a finite interval. 2.9 Theorem A set in R is compact iff it is closed and bounded. u This can be proved as a case of 5.12 below, and provides an alternative definition of compactness in R. The sufticiency part is known as the Heine-Borel theorem. A subset B of A is said to be dense in A if B c A q X. Readers may think they know what is implied here after studying the following theorem, but denseness is a slightly tricky notion. See also 2.15 and the remarks following before coming to any premature conclusions.
2.10 Theorem Let A be an interval of R, and C c A be a countable set. Then A C is dense in A. -
Proof By 1.7, each neighbourhood of apoint inA contains an uncountable number of points. Hence for each e A (whetheror not e C), the set (A C) f-h 5'(x,:) is not empty for every e > 0, so that is a closure point of A C. Thus, A C c (A C) QJ C A ? (A C) K .x
.v
-
.x
-
=
-
-
-
.
The k-fold Cartesian product of R with copies of itself generates what is called Euclidean k-space, R The points of Rk have the interpretation of k-vectors, or ordered k-tuples of real numbers, x (x1,.n,...,.q)'.All the concepts defined above for sets in R generalize directly to R The only modification required is to replace the scalars x and y by vectors x and y, and define an e-neighbourhood be the Euclidean distance betweenx andy, where 1Icll in a new way. Let Ilx 11 1/2 IEt,-1 , q is the length of the vector a (J1,...,JJ' and then define S(x,z) < el, for som s > 0. An open set A of R2 is one in which every point (y: Ilx 11 x e A can be contained in an open disk with positive radius centred on x. In R3 the open disk becomes an open sphere, and so on. .
=
.
-y
=
=
=
-y
2.2 Sequences and Limits A real sequence is a mapping from E$into R. The elements of the domain are called the indice- and those of the range variously the terms, members, or coordinates of the sequence. We will denote a sequence either by tak,n e EN), or more briefly by IakI:', or just by (xnJ when the context is clear. (xnl'kois said to converge to a limit x, if for every : > 0 there is an integer Nv for which
Ixn
-
xI
<
e for all n > Nv.
Write xn it ij often x, or = limu-jxak. When a sequence is tendinygo +x or said to diverge, but it may also be said to converge in R, to distinguish those cases when it is does not approach any fixed value, but is always wandering. increasing, non-increasing, or decreasA sequence is monotone (non-decreasing, ing) if one of the inequalities ak f xn+1, xn < xn+1, xn k ak+1, or xn > xnx.l holds for every n. To indicate that a monotone sequence is converging, one may write for emphasis either xn a7 will or xn t, x, as appropriate, although xn -.->
.x
-x
'1'
.x
--
24
Mathematics
also do in both cases. The following result does not require elaboration. 2.11 Theorem Evel'y monotone sequence in a compact set converges.
n
A sequence that does not converge may none the less visit the same point an infinite number of times, so exhibiting a kind of convergent behaviour. If fak,n e N l is real sequence, a subsequence is fxnk,k e INl where fnk, k e N ) is any increasing sequence of positive integers. If there exists a subsequence taksk e (h1 ) and a constant c such that xnk (-1)N, c, c is called a cluster point of the n 1,2,3,...) does not converge, but sequence. For example, the sequence f subsequence obtained by only taking the even values of n converges trivially. c is usually a finite constant, but +x and may be cluster points of a sequence if we If the of notion allow convergence in F. a subsequence. is convergent, then so is ) where fmk3 is an any subsequence of the subsequence, defined as (x,,u,k e EN whose members of fnkl members are also increasing sequence The concept of a subsequence is often useful in arguments concerning convergence. A typical line of reasoning employs a two-pronged attack', first one identities a convergent subsequence (a monotone sequence, perhapsl; then one uses other characteristics of the sequence to show that the cluster point is actually a limit. Especially useful in this connection is the knowledge that the members of the sequence are points in a compact set. Such sequences cannot diverge to infinity, since the set is bounded; and because the set is closed, any limit points or cluster points that exist must be in the set. Specifically, we have two uscful --
=
-x
.
results.
2.12 Theorem Every sequence in a compact set of R has at least one cluster point. Proof A monotone sequence converges in a compact set by 2.11. We show that ) hasamonotonesubsequence.Defineasubsequence (xnk) evel'y sequence (ak,ne EN follows. for 1,2,3,... let if S'et 1, and k there nl xnk..! sup,on:ak as exists a finite ?7z+1 satisfying this condition', otherwise let the subsequence terminate at nk. This subsequence is non-increasing. If it terminates, the sub?u) must contain a non-decreasing subsequence. A monotone sequence (.v, n k subsequence therefore exists in every case. w =
=
=
2.13 Theorem A sequence in a compact set either has two or more cluster points, or it converges. Proof Suppose that c is the unique cluster point of the sequence fxn) but that xn zd-yc. Then there is an infinite set of integers (nz, L e EN) such that Iau c I s for some : > 0. Define a sequence fyk)by setting yz xnk. Since (yz)is also a sequence on a compact set, it has a cluster point c' which by construction is different from c. But c' is also a cluster point of fx,,1, of which (ykl is a subsequence, which is a contradiction. Hence, xn c. w ,
-
=
-->
3 ,...,x
n 2.14 Example Consider the sequence ( 1,x,x2 or more j orma jj y ta.gl where is a real number. In the case IxI< 1, this sequence converges to n e EN()1, The condition specified zero,( I.x''lJ being monotone on the compactinterval g0,11. ,a:
.'t7
,...),
,
Limits and Continuity
25
in (2.2) is satisfied for Nc = logtel/log IxI in this case. If x = 1 it converges it to 1, trivially. lf > 1 it diverges in R, but converges in 2-to +x. If x +1 diverges, but oscillates between cluster points and neither converges nor -1
.x
=
-1.
Finally, if x < 1 the sequence diverges in R, but does not converge in Ea mately, it oscillates between the cluster points +x and -
F. Ulti-
-x.
We may discuss the asymptotic behaviour of a real sequence even when it has no limit. The superior limit of a sequence fxn) is
limsup xn n
=
inf sup xm. n
(2.3)
m kn
(Alternative notation'. limn xn.) The limsup is the eventual upper bound of a sequence. Think of (sup-znau, n 1,2,... J as the sequence of the largest values the sequence takes beyond the point n. This may be +x for every n, but in all cases it must be a non-increasing sequene having a limit, either +x or a finite real number; this limit is the limsup of the sequence. A link with the corresponding concept for set sequences is that if xn supA,, for some sequence of sets (4n c R ) then limsup xn supA, where A limsupsAn. The inferior limit is defined likewise, as the eventual lower bound: =
=
=
,
liminf xn
=
=
-
limsupt-ak)
=
sup inf xm, n
n
(2.4)
mkn
also written limnak. Necessarily, liminfaak ; limsupnak. When the limsup and liminf of a sequence are equal the sequence is convergent, and the limit is equal the sequence converges in V. to their common value. If both equal +x, or The usual application of these concepts is in arguments to establish the value of a limit. lt may not be pennissible to assume the existence of the limit, but the limsup and liminf always exist. The trick is to derive these and show them to be equal. For this purpose, it is sufticient in view of the above inequality to show liminfn xn limsupn xn. We often use this type of argument in the sequel. To determine whether a sequence converges it is not necessary to know what the limit is; 4he relationship between sequence coordinates the tail' (as tl criterion for converbecomes large) is sufficient for this purpose. The Cauchy > iff for real 0 of that Ne such ('E'i states every e: fakl converges sequence gence a that Ixn xm I < : whenever n > Nc and m > Ne. A sequence satisfying this criterion is called a Cauchy sequence. Any sequence satisfying (2.2)is a Cauchy sequence, and conversely, a real Cauchy seqqence must possess a limit in R. The two definitions are therefore equivalent (in R, at least), but the Cauchy condition may be easier to verify in practice. The limit of a Cauchy sequence whose members a1lbelong to a setA is by definition a closure point of A, though it need not itself belong to A. Conversely, for every accumulation point of a set A there must exist a Cauchy sequence in the set whose limit is x. Construct such a sequence by taking one point from each of the sequence of sets, -x,
-
.7
(A rn S(x.$In4, n
=
1,2.3,..1,
26
Mathetnatics
none of which are empty by definition. The term limit point is sometimes used synonymously with accumulation point. The following is a fundamental property of the reals.
2.15 Theorem Every real number is the limit of a Cauchy sequence of rationals. Proof For finite n 1et xn be a number whose decimal expansion consists only of zeros beyond the nth place in the sequence. lf the decimal point appears at position m, with m > n, then xn is an integer. If m f n, removing the decimal /10'J-'N point produces a tinite integer a, and xn a so ak is rational. Given any real x, a sequence of rationals (ak) is obtained by replacing with a zero every beyond the ath, for n = 1,2,... Since digit in the decimal expansion of jxa..l l < 10-/, lak) is a Cauchy sequence and xn x as n x. w =
.v
-->
.--->
-xn
The sequence exhibited is increasing, but a decreasing sequence can also be conIf x is structed, as f-yn) where (yn) is an increasing sequence tending to itself rational, this construction works by putting ak = x for every n, which trivially defines a Cauchy sequence, but certain arguments such as in 2.16 below depend on having xn # x for every n. To satisfy this requirement, choose the terminating' representation of the number', for example, instead of 1 take This does not 0.9999999..., and consider the sequence (0.9, 0.99, 0.999, work for the point 0, but then one can choose f0.1, 0.01, 0.001,...1. of a real One lnterestingcorollary of 2.15 is that, since every E-neighbourhood number must contain a rational, (E)is dense in R. We also showed in 2.10 that R (D is dense in R since Q is countable. We must be careful not to jump to the concluAnother version sion that because a set is dense, its complement must be of this proof, at least for points of the intelwal E0,1), is got by using the binary expansion of a real number. The dyadic rationals are the set -x.
<non-
...)
.
-
,
tsparse'
.
D
=
(ilzn,
i
=
1,...,2* 1, n c EN )
(2.5)
-
.
The dyadic rationals corresponding to a finite n define a covering of E0, 11 by 1/2N, which are bisected each time n is incremented. For any x intervals of width e g0,1J, a point of the set f illn' i = 1,,..,22 1 ) is contained in 5'(x,eJfor : < 2/21 so the dyadic rationals are dense in (0,11. D is a convenient analytic tool when we need to detine a sequence of partitions of an interval that is becoming dense in the limit, and will often appear in the sequel. Another set of useful applications concern set limits in R -
.
2.16 Theorem Every open interval is the limit of a sequence of closed subintervals wiih rational endpoints. Proof If a,b) is the interval, with a < &, choose Cauchy sequences of' rationals an a and bn b, with tzI < l)j (alwayspossible by 1.10). By definition, for for all n k N, and hence every x e anbj there exists N k 1 such that x gtzr,,lul > anb) i liminfnlln,lul. On the other hand, since an a and b > bn, abbjc (Jn,:,,1f for all n 2 1, so that a,bjc i liminfagtzn,hnlc. This is equivalent to limsupagtzn,:nj atb). Hence limnltzn,:nl exists and is equal to a,b). . .1e
''
Limits and Continuit.y
27
This shows that the limits of sequences of open sets need not be open, nor the limits of sequences of closed sets closed (takecomplements above). The only hard and fast rules we may 1ay down are the following corollaries of 2.4(i): the limit of a non-decreasing sequence of open sets is open, and (bycomplements) the limit of a non-increasing sequence of closed sets is closed.
2.3 Functions and Continuity hfunction of a real variable is a mapping J:S F-->T, where S c R, and 'F c R. By specifying a subset of R as the codomain, we imply without loss of generality that /S) = T', such that the mapping is onto 7. Consider the image in F, under f, of a Cauchy sequence fxnlin S converging to .x. lf the image of every such sequence converging to x e S is a Cauchy sequence the function is said to be continuous at Continuity is in 'F converging to formally defined, without invoking sequences explicitly, using the 6r approach. < # is continuous at the point x e S if for any e > 0 D > 0 such that Iy whenever < The choice here depend S. of implies lfl may on x. ye fx) I e, lf f is continuous at every point of S, it is simply said to be continuous on S. Perhaps the chief reason why continuity matters is the following result. .f@),
.x.
-
-xl
-
2.17 Theorem lf J: S F-> 'T is continuous at a1l points of S, f -1(A) j s open ju whenever A is open in 7J, and f -1(4) is closed in S whenever 4 is closed in F.
s
(:a
This important result has several generalizations, of which one, the extension to vector functions, is given in the next section. A proof will be given in a still more general context below; see 5.19. Continuity does not ensure that J(A)is open whenA is open. A mapping with this fAC) J(X)C in general, we # property is called an open mapping, although, since cannot assume that an open mapping is also a closed mapping, taking closed sets to clpsed sets. However, a homeomorphism is a function which is 1-1 onto, continuous, and has a continuous inverse. If f is a homeomorphism so is f and hence by 2.17 it is both an open mapping and a closed mapping. It therefore preserves the structum of neighbourhoods, so that, if two points are close in the domain, their images are always close in the range. Such a transformation amounts to a relabelling of axes. If fx + ) has a limit as h i 0, this is denoted f@+4.Likewise, J@-) denotes the limit of f(x ). It is not necessary to have x e S for these limits to exist, but if fx) exists, there is a weaker lkotion of cpntinuity at x. f is said at the point x e S if, for any s > 0, R 6 > 0 such that to be right-continuous and < 8 whenever 0 f x- h e S, .j
,
-
Iflx-h) fx) I < -
lt is said to be ever 0 < h <
(2.6)
E.
left-continuous at x if, for any e: > 0, 3 8 > 0 and x e S,
wch that
when-
-
If@) J@ ) I < -
-
z.
(2.7)
Mathematics
28
Right continuity at x implies J@) J@+)and left continuity at x implies fx) = the function is continuous at x. fx-). If fx) f(x+) Continuity is the property of a point x, not of the function f as a whole. Despite continuity holding pointwise on S, the property may none the less break down as certain points are approached. =
.f@-),
=
=
2.18 Example Conjider fx)
1/x, with S
=
=
T'
=
(0, x).
iff
<
For 6: > 0, 1
Iflx- ) fx) I -
=
x(x+o
and hence the choice of depends on both 0. l:l 0, but not in the limit as that
J: S
F->
sx
-
flx) is continuous
continuous if for every E
'F is unformly
Ix-yl
6: and x.
j
for all x
>
-->
.x
The function
< E
<
=
Ifx,
>
0 H
>
0 such
-.f(A') 1< e:
(2.8)
for every x,y e S. ln 2.18 the function is not uniformly continuous, for whichever is chosen, we can pick x small enough to invalidate the detinition. The problem arises because the set on which the function is defined is open and the boundary point is a discontinuity. Another class of cases that gives difticulty is the one x. where the dofnain is nbounded, and continuity at x is breaking down as x -->
However, we have the following result.
2.19 Theorem lf a function is continuous everywhere on a compact sets, then it is bounded and uniformly continuous on S. In (For proof, see 5.20 and 5.21.) Continuity is the weakest concept of smoothness of a function. So-called Lipschitz conditions provide a whole class of smoothness properties. A function f is said to satisfy a Lipschitz condition at a point x if, for any y e 5'@,) fot some 8 > 0, H M > 0 such that
If@l fx, I K -
ftlx
-yI )
(2.9)
R+ F-> R+ satisfies hld) 1 0 as d 1, 0. f is said to satisfy a uniform Lschitz condition if condition (2.9)holds, with fixed M, for a1l x,y e S. The Continuity (resp.unifonn type of smoothness imposed depends on the function uniform Lipschitz) property for any continuity) follows from the Lipschitz (resp. choice of h. Implicit in continuity is the idea that some function (.): R+ F-> R+ exists satisfying 6(E) 1 0 as 6: 1 0. This is equivalent to the Lipschitz condition holding for some (.),the case h = 8-1 B y imposig some degree of smoothness on h making it a psitive power of the argument for example we impose a degree of smoothness on the function, forbidding sharp corners' The next smoothness concept is undoubtedly well known to the reader, although differential calculus will play a fairly minor role here. Let a function J: S F-> 'F be continuous at x e S. lf
here
W
:
.
.
-
-
.
29
Limits and Continuity
fx-h) fx) h -
/:(x)
=
lim i0
(2.10)
exists, A'@)is called the left-hand derivative of f at x. The right-hand derivative, /1(x), is detined correspondingly for the case h 0. lf J+'(x) f-'(x),the common value is called the derivative of f at x, denoted f'x) or df/dx, and f is said to be dterentiable at x. lf J': S F-.> R is a continuous function, f is said to be continuously dterentiable on S. A function f is said to be non-decreasing (resp. increasing) if f@) flx) (resp. f@)> Ax))whenever y > x. lt is non-increasing (resp.decreasing) if is non-decreasing (resp.increasing). A monotone function is either non-decreasing or non-increasing. 'When the domain is an interval we have yet another smoothness condition. A function f.. (J,:1 F-+ R is of bounded variation if 3 M < x such that for every < xn b, by tinite collections of points a partition of gt7,/71 xo < x1 < '1'
=
-f
=
...
=
n
< M. lfxi) .f(.xf-.1)I 77 k=1
(2.1 1)
-
2.20 Theorem If and only if f is of bounded variation, there exist non-decreasing functions h and h such that f n .42
.f1.
=
-
(For proof see Apostol 1974: Ch. 6.) A function that satisfies the uniform Lipschitz condition on gJ,hl with ( I.x y I) Ix y I is of bounded variation on =
-
-
(J,:1.
2.4 Vector Sequences and Functions A sequence (xs) of real k-vectors is said to converge to a limit x if for every E > 0 there is an integer Nc for which
-x1I< Ilxn
E
for all H > Ne.
The sequence is called a Cauchy sequence in R iff Ne and m > Nv. A function
fl S
i--y
(2.12)
-auIl< e Ilxn
whenever
n
>
1-,
where S c Rk an d 7 R associates each point of S with a unique point of 7. 1ts graph is the subset of S x'r consisting of the k + ll-vectors (x, fx) ) for each x e S. f is continuous at x l S if for any E > 0 H > 0 such that ,
,
#
IIII <
=
I/.(x+)
-
fx) I < e
(2.13)
whenever x + b e S. The choice of may here depend on x. On the other hand, f is uniformly continuous on S if for any E > 0, 3 > 0 such that
IIII<
c:':>
sup x e S,x+: c S
l/'tx+h)
-.f@)
I<
e.
(2.14)
Mathematics
f (h,...,fm)'of functions of x is called, simply enough, a vector function.2 c on tinuity concepts apply element-wise to f in the obvious way. The A vector
=
function
f. . s
F-.
s sc ,
R -1
is said to be one-to-one if there exists a vector function f : S F-.-:hS, such that f -1(/.1)) x fOr each x e S An example of a 1-1 continuous function is the affine transformation 3 ./12) X2 + b = for constants b k x 1) and A (kx k) with IA l 0, having inverse function/-l -1 does not possess a closed form, A (y b4 ln most other cases the but there is a generalization of 2.17, as follows. =
.
=
./-1/)
:y:
-
.
2.21 Theorem Iff / S F- 1- is continuous, where S i Rk an d 1- i R'/ f-1A) is open in S when A is open in I-, and.f -1(A) is closed in S when A is closed in -T. (:1 ,
2.5 Sequences of Functions 7, 'T c R, be a function, where in tb-is case fl may be an arbitrary set, not necessarily a subset of R. Let f/'n,n e (N) be a sequence of such functions. If there exists f such that, for each e f, and E > 0, R Nvjl such that 1Jnt)l J((J)) l < e:when n > Asro,then fn is said to converge to f,pointwiseon f1. As for real sequences, we use the notations fn f, fn f, or fn 1 J, as appropwhere in the latter case the monoriate, for general or monotone convergence, tonicity must apply for eveyy (.t) e f2. This is a relatively weak notion of convergence, for it does not rule out the possibility that the convergence is breaking down at certain points of f1. The following example is related to 2.18 above. Let
fn1f
F-
-
'1'
-->
2.22 Example Let fnx) (0,x) is 1/x. But
=
nllnx
+
1),
fnlxt -
-
.x
1
e
=
x
(0,x). The
pointwise limit of
fnx) on
1 xtrl-x + 1)
,
and llxNcxx + 1)) < e: only for Nu > (1/a 1)(1/x). Thus for given E, Nvx cxo 0 and it is not possible to put an upper bound on Nu such that as x lfnx) 1/xl < s, n 2 Ncx, for every x > 0. n -->
-
--
-
To rule out cases of this type, we define the stronger notion of unifrm convergence. If there exists a function f juch that, for each : > 0, there exists N such th a t
lfnkbt J((,))I < sup eD -
fn is said to converge
E when
to J unifonnly on f1.
n
>
N,
31
Limits and Continuity
2.6 Summability and Order Relations The sum of the tenns of a real sequence faklTis called a series, written X'izzlxa (or just EAk). The terms of the real sequence (X'=lau,n e (N) are called the partial sums of the series. We say that the series converges if the partial sums converge to a tinite limit. A series is said to converge absolutely if the monotone sequence (Z'=1IxmI n e Ehl converges. ,
T x? This 2.23 Example Consider the geometric series, Zy converges to 1/(1 x) when 1x1< 1, and also converges absolutely. It oscillates between cluster points and for other values of x it diverges. u 0 and 1 for x .1
-
.
-1,
=
2.24 Theorem lf a series converges absolutely, then it converges. Proof The sequence (X;.1lxm I n e EN)is monotone, and either diverges to +x or converges to a finite limit. ln the latter case the Cauchy criterion implies that + Ixn..mI + Ixn..mI 0 as m and n tend to infinity. since Ixn I + lxnl + inequality,4 +xn.vm I by the triangle lak + convergence of (X)=1au,n E EN ) follows by the same criterion. w ,
-->
....
....
....
An alternative tenninology speaks of summability. A real sequence taklT is said to be summable if the series Xzkconverges, and absolutely summable if ( Ixnll7is summable. Any absolutely summablesequenceis summableby 2.24,and any summable sequence must b converging to zero. Convergence to zero does not imply summability (see2.27 below, fof example), but convergence of the tail sums to zero is necessary and suftkient.
2.25 Theorem 1ff (xn)7is summable, Zrlzznx- 0 as n -->
n-1
-->
x.
Proof For necessity, write IZ,,=l.zu x ! f IZ,a=1au I + I m-nxm j sincefor any : > 0 < k there exists N such that IE,7ox,ul e: for n N, it follows that lZp1=laulS ,,=,14% Then n I +: < x.n Conversely, assume summability and let A Eklxn. IE#-1 Z,n=n-Ym X Y,,=1.<,a () as n .-.y x s
sx
.
=
-1
-->
=
.
-
.
A sequence fakJTis Cesro-summable if the sequence This is weaker than ordinary convergence.
i .26 Theorem lf (akl:' cbnverges to x, its Cesro
-1
n converges. f?z Eru=laulq
sum also converges to x. n
But a sequence can be Cesro-summable in spite of not converging. The sequence f (-1)/)7 converges in Cesro sum to zero, whereas the partial sum sequence (r=()(-1)''' J7converges in Cesro sum ttl (compare2.14). Various notations are used to indicate the relationships between rates of diverfJnl!' gence or convergence of different sequences. If (xnlTis any real sequence, is a sequence of positive real numbers, and there exists a constant B < x such that IakIIan ; B for a1l n, we say that xn is (at most) of the order of magnitude of an, and write xn Oa. If takIanl converges to zero, we write xn oan), 4nd s>y that xn is of smaller order of magnitude than an. an can be incfeasing or decr:aing, so this notation can be used to express an upper bound either on the fte of eropth of a divereent seauence. or on the rate of convergence of a =
=
32
Mathematics
sequenceto zero. followsfrom the
Here are some rules for manipulation of 0(.), whose proof definition. lf xn On a) and y,j Onq), then =
.;tn +ya x 'n =
=
=
t?tzynwelxtfts; )
(2. 15)
(?(Itz+f5)
(2.16)
,
xls onsq), whenever =
a'fz is defined.
(2.17)
An alternative notation for the case xn 0 is xn f< an, which means that there is a constant, 0 < B < x, such that xn f Ban for al1 n. This may be more convenient in algebraic manipulations. The notation ak an will be used to indicate that there exist N k 0, and t'inite constants A > 0 and B k A, such that infapvak lan) 2 A and supazxtak Ian) S B. This says that l and (Jnl grow ultimately at the same rate, and is different 0. Some f'rom the relation xn Oan), since the latter does not exclude xn Ian 1. authors use xn an in the stronger sense of xn lan -
txn
.-.-)
=
.-->
-
2.27 Theorem If (ak) is a real positive sequence, and xn 1+a. n (i) if a > then Zpual.u n then Erlxlau log n; (ii) if ('x ()(sl+a) (j mzmxm < then )Lxlau x (iii) if (x < an xx
n,
-
-1
-
,
-1
=
-
-1
(x
.
.
Proot By assumption there exist N 2 1 and constants A > 0 and B k A such that An < xn K Bnz for n k #, and hence Amnzmt < mnzxm S Bxmnzm. The limit x for different values of a defines the Riemann zetahmction of 1x.l?'?ztz as n and its rates of divergence for a k for a < are standard results', see e.g. of Apostol (1974:Sects. 8. 12-8.13). Since the sum terms from 1 to N-1 is tinite, their omission cannot change the conclusions. . ..4
-1
-1,
lt is common pratice to express the rate of convergence to zero of a positive real sequence in tenns of the summability of the coordinates raised to a given power. The following device allows some f'urtherrefinement of summability condiV as v x (0) for tions. Let U(v) be a positive function of v. If &(vx)/&(v) > 0 and < p < +=, U is said to be regularly vatying at inhnity zero). lf a .x x (0), 1 for x > 0 as v positive function Lv) has the property L(%'x)lLM) frl/W/y varying said Evidently, is slowly regularly it at zero). to be any varying function can be expressed in the form &(v) = vpfXvl,where L(v) is slowly varying. While the definition allows v to be a real variable, in the cases of interest we will have v n for n e N, with U and L having the interpretation of positive sequences. --
--
-x
-->
-->
=
2.28 Example (logvltxis slowly varying at infinity, for any
G.
u
On the theol'y of regular vqriation see Feller (1971),or Love (1977).The important property is the following. 2.29 Theorem If L is slowly varying at infinity, then for any k 1 such that
>
0 there exists N
33
Limits and Continuity
v -8
Atvl
<
<
v
8
a 11v
,
N
>
.
(2.18)
(a
Hence we have the following corollary of 2.27, which shows how the notion of a convergent power series can be refined by allowing for the presence of a slowly varying function. 'ixlak
2.30Corollary
< If xn OnLnl) then Ln) which are slowly varying at infinity. u =
x
for a11(x
and al1 functions
-1
<
On the other hand, the presence of a slowly varying component can affect the summability of a sequence. The following result can be proved using the integral test for series convergence (Apostol 1974: Sect. 8.12).
2.31 Theorem If ak
X& m=
l.x,,,
l/gntlog n) 1+8 j w ith
-
>
0 then ,
log 1og n. n
-
E=nzz lak
<
If 6
x.
=
0, then
2.32 Theorem (Feller 1971: 275) lf a positive monotone function &(v) satisfies &(v.x) f7(v) -where D is dense in R+, and 0
v(.x),al1 a7 e <
v(x)<
then
x,
(2.19)
D,
v@) =
.x
P
for
<
-x
p
<
x.
To the extent that (2.19) is a fairly general property, we can conclude monotone functions are as a rule regularly varying.
u that
2.33 Theorem The derivative of a monotone regularly varying function is regularly varying at cra. Proof Given &(v) U' (v) If f7(v)
-->
=
=
vPf-(v),
write
pvP-1f-(v)
+vPfJ(v)
=
vP-1(p1,(v)
+vr(v)).
0 there is no more to show, so assume liminfvfxv)
jfzotjvxxilj
Kdv
which implies f7(vx)/f7(v) U'Mx)
U,(v)
Ly'vl
=
x
...0
-
(v)
--
L
(v) fXv)
>
0. Then
(),
(2.21)
w
(2.22)
1. Thus,
p-lpfatvx) =
f-tv-lj
jf7,(vAl
(2.20)
+
vxc'tvxl
pAtvl +vA
,
(v)
--)
p
a7
.
#
2.7 Arrays Arguments concerning stochastic convergence often involve a double-indexing of elements. An array is a mapping whose domain is the Cartesian product of countable, linearly ordered sets, such as ENx ENor r x (N, or a subst thereof. A real double array, in particular, is a double-indexed collection of numbers, or, alternatively, a sequence whose members are real sequences. We will use notation juch as (fakt,t E z J, n e (N), or just fau) when the context is clear.
Mathematics
34
-?'
A collection of tinite sequences (fau, f 1,...,k ), n e IN) where kn x as n --+ x, is called a triangular array. As an example, consider array elements of the 1,...,n) is a real sequence. The question of form xnt ytln, where tys t whether the series ( :=1.x,,,, n e N l converges is equivalent to that of the Cesro convergence of the original sequence', however, the array formulation is frequently =
,
=
=
the more convenient.
2.34 Toeplitz's lemma Suppose (y,;) is a real sequence and yn .-.-: y. lf ((.'xkf, t= such that 1,...,lk ), n e ENl is a triangula.r array (a) xnt
0 as n
--
for each fixed /,
x
--
kn
(b)
lim n-yoo
X Iak,lS
C <
x,
>1
kn lim A'-lau1, =
n-+x
>1
k then E/i,lxn,y, --- y. Proof
Iyn -y
F or y
=
(c) can be
O ,
fyn) for any s > 0 5 Ne k 1 such that for (c), and then (b) and the triangle inequality,
By assumption on I < 8C. Hence by
,
kn
n
>
Nvs
kn
lim Xxna, -y
n-yx
omitted.
X-v/t-'w
lim
=
>1
-y)
>1
n-yx
Nz
Txntyt y)
< lim
-
+
s
wj
n--kx
in view of (a). This completes the proof, since
E
(2.23)
:,
=
is arbitrary.
x
A particular case of an array fxn/)satisfying the conditions of the lemma is xnt where S = (Z''=1y,)-1y/, ty/lis a positive sequence and Z'lsxlys A leading application of this result is to prove the following theorem, a fundamental tool of limit theory. -->
=.
2.35 Kronecker's Iemma Consider sequences (tz,)';'and x, If Ll=jxtlat C < cxa as n numbers, with at -?'
tx/lTof
positive real
-->
=.
--
1
n
-Xx,
Jn
0.
--)
(2.24)
>1
Proof Detining co 0 and cn Xntcjxtlat for n G N, note that xt atct cr-1), 1,...,n. Also define tzo 1,...,n, so that an 0 and bt at fh-l for t t ntzzkbt. Now apply the identity for arbitrary sequences ao,...,an and co,...,cn, =
=
=
77(c,,.-1 Abel's known
c,-1)
as
=
=
-
-
=
n
n
(This is
=
=
=
77(tz,-1 zflcr-l -
+
ancn
-
zoco.
,-1
partial
summation formula.) We obtain
(2.25)
Limits and Continuit.v
1 n -- X xt an >1
n
1
X atlc t an
=
-
>1
= cn
35
1
-
-
ct- 1)
w-xn
--Lbtct-j an
-->
C- C
=
(2.26)
0
>1
is by the Toeplitz lemma, setting xnt
where the convergence
=
bt lan.
.
The notion of array convergence extends the familiar sequenceconcept. Consider subsequences, collection of full generality k (N for a (fxmnk,e l m e EN), an array where fnk, k e INl is an increasing sequence of positive integers. If the limit xm limz--ycxaAun: exists for each m e IN,we would say that the array is convergent; and = its limit is the infinite sequence (x,,,,m e N 1.Whether this sequence converges is a separate question from whether it exists at all. Suppose the array is bounded, in the sense that supkmlxmpulf # < x. We know by 2.12 that for each m there exists at least one cluster point, say xm, of the inner sequence (xaw,k e IN1. An important question in several contexts is this: is it valid to say that the array as a whole has a cluster point?
2.36 Theorem Corresponding to any bounded array f(xrm,k e ENJ, m e N ) there k e N J m e EN exists a sequece (x,,l, the limit of the array ( tx,ug, J as k x, subsequence n:) of f for each m. where (rl:) is the same ,
-->
,
Proof This is by construction of the required subsequence. Begin with a conver1 be a su bsequence o f (nz) such that sj gent subsequence for m 1', let (nkJ x1. Next, consider the sequence (.n,n)J. Like f-n,s:J,this is on the bounded interval (-#,#), and so contains a convergent subsequence. Let the indices of this 1 be denoted f42:) and note that latter subsequence, drawn from the members of (?1k), Proceeding in the same way for each m generxl,nj xl as well as 12,4 ates an array (fn, k e IN1, m e INJ, having the property that lxjw, k e EN) is a convergent sequence for 1 S i S m. k k e ENJ in other wotds, take the first member of Now consider the sequence fns 1 th e secon d member of (rl2k), and so on, For each m, this sequence is a subtlkl, pzth point of the sequence onwards, and hence the sequence of (n'2Jfrom the k e EN ) sequence f.x,,,4,k k PZJis cnkergent. This means that the sequence (xmmj, k = setting rl11 requirement isfies of the theorem. w is convergent, so the f fnzl sat ...-)
.n
=
,
-->
.n.
--
'
,
is called the
,
.
3 s4easure
3. 1 Measure Spaces A measure is a set function, a mapping which associates a (possiblyextended) real number with a set. Commonplace examples of measures include the lengths, areas, in and volumes of geometrical tigures,but wholly abstract sets can be an analogous way. Formally, we have the following definition. <measured'
3.1 Definition Given a class 5 of subsets of a set f, a measure
jt:
; F->
R-
is a function having the following properties: (a) g,(A) k 0, al1 A e %. 0. (b) g,(0) (c) For a countable collection (Aj e F, j e ENl with Aj DA./' and UjAj e ;, =
p. UXy
=
)'
=
O for j
j'
#
X g,(Xy). C1 j
The particular cases at issue in this book are of course the probabilities of random events in a sample space f; more of this in Chapter 7. Condition (a) is optional and set functions taking either sign may be referred to as measures (see e.g. j4.4), but non-negativity is desirable for present purposes. A measurable space is a pair (f1,T) where f is any collection of objects, and 5 is a c-field of subsets of f. When (,T) is a measurable space, the triple (,T,g) is called a measure space. More than one measure can be associated with the measurable space (f1,F), hence the distinction between measure space and measurable space is important. Condition 3.1(c) is called countable additivity. lf a set function has the property
p,tA kJ#)
=
(3.2)
p.(z4)+p,(f)
for each disjoint pair A,#, a property that extends by iteration to finite collections A1,..., An, it is said to be jnitely additive. In 3.1 5 could be a field, but the possibility of extending the properties of g, to the corresponding c-tield, by allowing additivity over countable collections, is an essential feature of a measure.
If g,(f1) < cxn the measure is said to be hnite.And if Q UjLh where lfl is a countable collection of Y-sets, and g.tfl < x for each j, g. is said to be ; G-jinite. In particular, if there is c(./) and Lh l a collection .Y'such that =
,$
=
sieasure ./
(A fa #: # e for each j, g is said to be c-finite on (ratherthan on ;). If is a measurable space and (A,?L,g) is a measure space 11 for some A q @,(A,8L) g,(AC) restriction If in of (f,T,g,) to A. 0 (equivalentto this case called the g(A) gtf1l when p.tfl < cx)l A is called a supportof the measure. When A supports 2, the sets of ?L have the same measures as the corresponding ones of 5. A point e f with the property g,(( )) > 0 is called an atom of the measure. =
=
=
3.2 Example The case closest to everyday intuition is Lebesgue measure, m, on the measurable space (R,O), where tB is the Borel field on R Generalizing the b a to an notion of length in geometry, Lebesgue measure assigns r?z((J,:1) (J,!71. plausible Additivity is an intuitively interval property if we think of measuring the total length of a collection of disjoint intervals. .
=
-
Lebesgue measure is atomless (see3.15 below), every point of the line taking ((J,:1, O(u,,j,m) denote the restriction of measure 0, but ??z(R) x. Letting (R,f,pz) to a finite interval, m is a finite measure on (t7,:1.Since IR can be partitioned into a countable collection of finite intervals, m is c-finite. n =
additional properties may be deduced from the definition:
Sole
3.3 Theorem For arbitrary T-sets A, #, and (Ay,j e N J (i) A c # = g,(A) S g,(#) (monotonicity). (ii) gtA k.p #) + g,(A ch B4 g.(A)+ g(#). (iii) p,(UyAj) f Xg,(Ay) (countablesubadditivity).
,
=
Proof To show (i) note that A and B -A are disjoint sets whose union is #, by hypothesis, and use 3.1(a) and 3.1(c). To show (ii),use A wB A QJ (B -A) and B = (A chB) k.pCB A), where again the sets in each union are disjoint. The result Al and Bn follows on application of 3.1(c). To show (iii), detine #1 UP-14 An j. Note that the sets Bn are disjoint, that Bn ? An, and that U==lfj j UT=IAy. Hence, J =
-
=
-
./
=
=
=1
X
G)
U p,/=1
=
p,
X
oo
Xp,(l U#, /=1 =
j=1
< Xp,(A?).
(3.3)
.
j=1
This proof illustrates a standard technique of measure theory, converting a sequence of sets into a disjoint sequence having the same union by taking differences. This trick will become familiar in numerous later applications. The idea behind 3.3(ii) can be extended to give an expression for the measure of any tinite union. This is the inclusion-exclusion formula'. n
UA/
p,j=
#
n
Xp,(A)
-
1
j= 1
-
77p,(+rnAzl k+j
+
p.tAycAAkchA,l 77 k:gy:y;l + jttx
f'-'h 1 r''hA:2,
...
-
...
r''hXrj),
(3.4)
where the sign of the last term is negative if n is even and positive if n is odd, and there are ln 1 tenns in the sulp in total. The proof of (3.4)is by induction from 3.3(ii), substituting for the second term on the right-hand side of -
Mathematics
38
(,U.-j+) p.(-.-)
g, repeatedly, for n
-
+
p,
-j,'-s
-U.-,'.,) p.
(,
-.-)
(
-
(3.5)
f..h
,
1. n 2,..., 1. IN be n e ) a monotone sequence of T-sets with limit A e T. A set function on g, is said to be continuous if g,(z4,,) g.(A). -
-
(AU,
Let
.--.-:
3.4 Theorem A finite measure is continuous. Proof First 1et (An) be increasing, with A,,-I cAn, and A U*a=IA,,.The sequence (Bj, j G ) where #1 Aj, and Bj Aj -Ay-I for j > 1 is disjoint by eonstrucand tion, with Bj e @, Aa U7=1#j, =
=
,
=
=
p,(An)
=
Xp,(). j=1
The real sequence (g,(A,,)) is therefore monotone, and converges since it is bounded g,(A). above by g,4Q)< x. Countable additivity implies LT) g,(U7 O';=1A,,. Consider Alternatively, 1et (An) be decreasing, with z4n-l Q An and the increasing sequence lA)) determine g,(49 by the same argument, and use finite additivity to conclude that g(A) = gtf1l g,(A9 is the limit of g(An) =l#y.)
=1g,(#j)
=
:4
=
=
,
=
-
P.(fX -
P.(XnC).
K
The finiteness of the measure is needed for the second part of the argument, but the result that g,tAul g(A) when ytu A actually holds generally, not excluding the case g,(A) x. This theorem has a partial converse: '
-->
=
3.5 Theorem A non-negative uous is countably additive.
set function g, which is finitely additive and contin-
Proof Let (#n) be a countable, disjoint sequence. If A,, U7=l, the sequence (An ) is increasing, Bn f'nAn-l 0, and so gtAnl g,(#n)+ g,(An-I) for every n, by finite additivity. Given non-negativity, it follows by induction that (g,(An)Jis ZTJ =Ip,(), whereas continuity implies that g,(A) monotone. lf A U7=tBj, g(A) LLIUTJ * lB =
=
=
=
=
=
=
.
Arguments in the theory of integution often turn on the notion of a set. In a measure space (f1,T,g), a set ofmeasure zero is (simplyenough) a set M e 5 with g.(M) 0. A condition or restriction on the elements of Q is said to occur almost everywhere (a.e.)if it holds on a set E and f E has measure zero. lf more than one measure is assigned to the same space, it may be necessary to indicate which measure the statement applies to, by writing a.e.lgl or a.e.gvq as the case may be. fnegligible'
=
-
3.6 Theorem (i) If M and N are T-sets, M has measure 0 and N i M, then N has measure 0. (ii) lf (A&) is a countable sequence with g(Mj) 0, V j, then g(UjMj4 0. (iii) If fe)) is a countable sequence with g,(r.'pl 0, V j, then g,((UjA))9 = 0. .
=
=
=
Measure Proof
39
(i) is
an application of monotonicity', (ii) is a consequence of countable additivity', and (iii) follows likewise, using the second de Morgan law. .
In j3.2 and j3.3 we will be concerned about the measurability of the sets in a given space. We show that, if the sets of a given collection are measurable, the sets of the c-field generated by that collection are also measurable (the Extension Theorem). For many purposes this fact is sufficient, but there may be sets outside the c-field which can be shown in other ways to be measurable, and it might be desirable to include these in the measure space. In particular, if g(A) g(A) wheneverd c E c #. This is g,(#) it would seem reasonable to assign jtt'l equivalent to assigning measure 0 to any subset of a set of measure 0. The measure space (f1,T,jt) is said to be complete if, for any set E e 5 with g.(F) = 0, a11 subsets of E are also in T. According to the following result, every measure space can be completed without changing any of our conclusions except in respect of these negligible sets. =
=
3.7 Theorem Given any measure space (f,;,jt), there exists a complete measure TB, (f,TB,g), called the completion of (f1,F,p), such that 5 and g(F) space c F(F) for al1 E e %. In =
Notice that the completion of a space is defined with respect to a particular has a different completion for each measure measure. The measurable space (f1,@) that can be defined on it. Proof Let
AB J;B
denote the collection of a11 subsets of i-sets of pmmeasure0, and LF
=
c
D: FA F e
AB
for some E e
.41
.
0, any set F c E satisfies the criterion of (3.7)and so is in FB as the definition requires. For F e @B, let F(#) g,(F), where E is any F-set satisfying E F e AB. To show that the choice of E is immaterial, let Ej and E1 be two such If jt(F)
=
=
sets, and note that
p,(A'lA&) Since jttFl t-p F2)
=
g,((FAF1)A (FAF2))
=
=
0.
g(F1 A F2), we must conclude
p,(F1 r'hE
+
g,tflr-hE
2 p.tfl k p,(F1 r'hF2)
(3.8) that
for i 1 and 2, or, g,(F1) p.(&). Hence, the measure is unique. When F e T, we AB, confirming that the measures agree on T. can choose E F, since FAF= O e TB is a c-tield colgaining ?F.Choosing E F in (3.7) It remains to show that AFC FB. 8/ shows If F e ;B, thenFA F e ABfor E e 5 and hence Ec for Fe 5 c TB. And finally AB where Ec if Fj e J;B foj e (N,there EF e 5, and so Fc e e exist Ej e 5 for j e EN,such that Ejh Fj e AB. Hence =
=
=
=
=
e U1 A U/) c UIF/AFJ ?
? by 3.6(ii). This means that
AB,
j
Ojb 6
TB,
and completes the proof. w
(3.10)
Mathematics
40 3.2 The Extension Theorem
You may wonderwhy, in thedefinition of ameasurable space, Fcould not simply be the set of all subsets; the power set of D. The problem is to tind a consistent method of assigning a measure to every set. This is straightforward when the space has a tinite number of elements, but not in an infinite space where there is no way, even conceptually, to assign a specitic measure to each set. It is necessary to specify a rule which generates a measure for any designated set. The problem of measurability is basically the problem of going beyond constructive methods without running into inconsistencies. We now show how this problem can be solved for c-fields. These are a sufticiently general class of sets to cope with most situations arising in probability. One must begin by assigning a measure, to be denotedgo, to the members of some basic collection C for which this can feasibly be done. For example, to construct the measure b a. Lebesgue measure we started by assigning to each interval (tz,&1 We then reason from the properties of go to extend it from this basic coltection to a11 the sets of interest. C must be rich enough to allow go to be uniquely defined by it. A collection C c 5 is called a determning class for (f,F) if, whenever jt and v are measures on 5, jt(A) v(A) for all A e C implies that g v. Given C, we must also know how to assign gmvalues to any sets derived from rby operations such as union, intersection, complementation, and difference. For disjoint sets A and B w have g)tz4QJ #) go(A) + go(#) by finite additivity, and when B c A, gotA B) go(A) 04#). We also need to be able to determine gotA fa #), which will require specitic knowledge of the relationship between the sets. When such assignments are possible for any pair of sets whose measures are themselves known, the measure is thereby extended to a wider class of sets, to be is closed and C are the same collection, but in any event denoted 'Y'.Often under various finite set operations, and must at least be a semi-ring. In the is typically either a field (algebra)or a semi-algebra. Example applications 1.18 is a god case to keep in mind. However, cannot be a c-field since at most a finite number of operations are permitted to determine go(A) for any A e .Y'.At this point we might pose the opposite question to the one we started with, and ask why might not be a rich enough collection for our needs. ln fact, events of interest frequently arise which cannot contain. 3.15 below illustrates the necessity of being able to go to the limit, and consider events that are expressible only as countably infinite unions or intersections of C-sets. Extending to the events 5 c(#) proves indispensable. We have two results, establishing existence and uniqueness respectively. -
=
=
=
-
=
-
./
-t/'
,/
'f
,/
,$
=
3.8 Extension ,/.
theorem (existence)Let $ be a semi-ring, and let go:
lf 5 = c(,$). there exists a measure jt on measure on ,YI. jL0t'l for each E e n
./
F--y
F+be
(D,?D,such that jt(F)
a =
Although the proof of the theorem is rather lengthy and some of the details are fiddly, the basic idea is simple. Take an event A c f to which we wish to assign a
Measure ./,
,/,
consider choosing a finite we have g,(A) go(A). If A measure g,(z4).If A G that is, a selection of sets Ej e or countable covering for A from members of OjEj. The object is to tind as 1,2,3,... such that A c ./, j a covering as possible, in the sense that Xgo(f;)is as small as possible. The outer measure of A is =
,/,.
teconomical'
=
P,*(X)
inf X.to(F./),
=
(3.11)
where the infimum is taken over a11tinite and countable coverings of zl by y-sets.
If no such covering exists, set g,*(A) x. Clearly, g7(A) gotAl for each A E g,* is called the outer measure because, for any eligible definition of g,(A), g,*(A) 2
Xg,(l
2
p,
j
Uf j
.9.
=
=
2 p,(A), for Ej
,Y'.
E
(3.12) ,/
The first inequality here is by the stipulation that p(Ej) goIFJfor Ej e in side majorant where exists, is infinite. The the case a covering or else the second and third follow by countable subadditivity and monotonicity respectively, because p, is a measure. We could also construct a minimal covering for Ac and, at least if the relevant outer measures are tinite, define the inner measure of 4 as jt+ (4) jt(AC) p,*(z4C). and g,*(A9 k jttAfl by (3.12), Note that since p,(z4) g,tfl g,*(f) =
=
=
-
-
p,(A) 2 g,+(A).
(3.13)
lf g,*(4) g+44), it would make sense to call this common value the measure of A, and say that A is measurable. ln fact, we employ a more stringent criterion. A set A g f is said to be measurable if, for any B c f, =
g,*(Af-'h B4
+
g,*tz4f chB)
=
(3.14)
jt*(#).
This yields g,*(A) p,+(A) as a special case on putting B f1, but remains valid even if g,tfll x. Let A denote the collection of all measurable sets, those subsets of f) satisfying (3.14).Since g,*(A) go(A) forA e and /1.:40) 0, putting 4 O in (3.14) gives the trivial equality g,*(#) g,*(#). Hence O e A, and since the definition implies that Ac E A if 4 e A, f E A too. The next steps are to detennine what properties the set function jt*: A !-+ F shares with a measure. Clearly, =
=
=
,/
=
=
=
=
jt*(A)
0 for all 4
C f1.
(3.15)
Another property which follows directly from the definition of p,*is monotonicity: X1 I X2 =
F*(X1) f >*(X2), fOr A1, 42 i
.
(3.16)
Our goal is to show that countable additivity also holds for g,* in respect of A-sets, but it proves convenient to begin by establishing countable Juhadditivity. 3.9 Lemma If fAy,j
E
EN J is any sequence of subsets of f, then
42
Mathematics
g,* UX./ f
/7 >*(X./).
(3.17)
Proof Assume g*(Ay) < x for each j. (If not, the result is trivial.) For each j, 1et fEjkq denote a countable covering of Aj by y-sets, which satisfies p.*(A>+ 2-:
< Xpo(,) k
: > 0. Such a collection always exists, by the definition of g*. Since LkkEjk, it follows by definition that U#.j ?
for any
p.*UA/ noting Z7=12S ity is strict. .
=
1.
f XpotA>l
<
j,k
j
(3.17)
Xp,*()+ e,
(3.18)
./
now follows since : is arbitrary and the last inequal-
The following is an immediate consequence of the theorem, since subadditivity supplies the reverse inequality to give (3.14).
3.10 Corollary A is measurable if, for any B c D, g,*(AC r''h#) K g*(#). n t *(A r''h ) +
(3.19)
The following lemma is central to the proof of the extension theorem. lt yields countable additivity as a corollary, but also has a wider purpose.
3.11 Lemma A is a monotone
class.
Proof Letting fAj, j e INJ be an increasing sequence of A-sets converging to 4 U#.j, we show A e A. For n > 1 and E e f1, the definition of an A-set gives =
g7(An fa E)
=
g*(4n-1 ra (An rh A=))+ g*(4k-1 f-'h (Anf''h F))
= g*(An-1 rhEI+ g7(& rn E). where Bn g,*(A()rn E)
= =
An -An-1, and the sequence 0', then by induction, ke(An chF)
=
(3.20)
(BjI is disjoint.
X pz*( rn E)
Put
ztttl
=
O so that (3.21)
y=1
holds for every n. The right-hand side of (3.21)for n e INis a monotone real x. Now, since An e A, g,*(4rh A>)as n sequence, and g,*(An(hE) .--)
-->
jt*(F)
r1*(Anrnm
+
h1*(Xrn)
k jt*(An(nE)
+
g*tAc rnfl,
=
using the monotonicity of g,*and the fact that from the foregoing argument that
AC
(3.22) Anf.Taking the limit, we have
sleasure jt*(A7 k g7(A r'A F)
+
43
g.*(AC
f'-h A'),
(3.23)
so that 4 e A by 3.10. For the case of a decreasing sequence, simply move to the complements and argue as above. . Since () is a disjoint sequence, countable additivity emerges as a by-product of the lemma, as the following corollary shows.
3.12 Corollary
If
()
is a disjoint sequence of A-sets,
g,*U j Proof Immediate on putting E
-
Xp,*().
(3.24)
j
in (3.21) and letting n
=
--> x,
noting VjBj
A. w
=
Notice how we needed 3.10 in the proof of 3.11, which is why additivity has been derived from subadditivity rather than the other way about. Proof of 3.8 We have established in (3.15)and (3.24)that g* is a measure for the elements of A. If it can be shown that 5 q A, setting g,(A) g,*(A)for al1 A e 5 will satisfy the existence criteria of the theorem. implies The t'irst step is to show that c A or, by 3.10, that 4 e =
.t/7
,/
g,*(A'r7A) + g,*(A'r-hAf)< g*(F)
(3.25)
,$1
denote a finite or countable covering of E such that Lj gotAyl < p,*(F) + E, for s > 0. If no such covering exists, g,*(A-) cxl by definis a ition and (3.25)holds trivially. Note that E r'A A c U/A; r-7A), and since /''AAC f-7WC), and by simple semi-ring the sets Aj ('''AAare in ,Y'.Similarly, E c Uyt>j algbra definition of and the set a semi-ring,
for any E i (. Let lAy e
=
,/
Aj r'Aytf
=
Aj
Aj chAl
-
=
UG:
(3.26)
k
where the Cjk are a finite collection of yksets, disjoint with each other and also with Aj r'hA. Now, applying 3.9 and the fact that g,*(#) k)(#) for B e we find .$,
=
g,*(A'faA) +
g*(Ff-$AC)
f
X jtot/j chdl X XgotGl +
j
j
k
p,*(eD =Xw)(z4y) ? <
+ :,
(3.27)
is tinitely additive, and Aj DA where the equality follows from (3.26)becau and the Cjk are mutually disjoint. Since s is arbitrary, (3.25)follows. Next, we show that A is a c-field. We have only to show that A is a field, because 3.11 implies it is also a c-field, by 1.24. We lready know that e A and A is closed under complementation, so it remains to show that unions of A-sets are in M. Suppose that Al and A2 are A-sets and E c f. Then .to
:,*(13 =
>*(A1 nA') + F*(Af rhEI
Mathematics
=
jt*(z42 c7z4jra e) +
g*(z4qrndl chE) + g,*(A2/'nAk ra A')+ jt*(A rnAf
f-h
E)
p,*(A2f-7Al ra A')
+ g*((Aq r7Al ra F)
f-h F)) rn E) k.p (Aj/-AAC
k.p (A2 chAf
+p7((42ra4j)cfaF), = jt*((A2rn41) cjE)
(3.28)
where the inequality is by subadditivity, and the rest is set algebra. By 3.10 this is sufficient for A1 f'742 e A, and hence also for A1kJA2 G A, using closure under complementation. It follows that A is a c-tield containing $, and since 5 is the smallest such c-field, we have that 5 i X, as required. . Notice that (3.28)was got by using (3.14)as the relation defining measurability. The proof does not go through using p.*(A) gx(A) as the definition. The style of this argument tells us some important things about the role of .Y'. Any set that has no covering by '-sets is assigned the measure x, so for finite measures it is a requisite that f q LbEj for a tinite or countable collection fEj o? e #') The measure of a union of y-sets must be able to approximate the measure any F-set arbitrarily well, and the bagic content of the theorem is to establish that a semi-ring has this property. To complete the demonstraiion of the extension, there remains the question of uniqueness. To get this result we need to impose c-finiteness, which was not needed for existence. =
.
3.13 Extension theorem (uniqueness)Let jt and g,' denote measures on a space (f1,T), where 5 c(.$), and is a semi-ring. If the measures are c-finite on and g,(A') jt'(F) for a1l E e .T, then g(An) g'(F) for a1l E e %. .$
,/
=
=
=
Proof We
tirst prove
the theorem for the case of finite measures, by an applitheorem. Define .J= fE e F: p.(F) = p,'(AD ) Then $z ,J by hypothBy 1.27, the proof is completed esis. If is a semi-ring, it is also a l-system. contains c(#). show hence if Fe can that .J is a l-system, and and condition 1.26(a) holds. Additivity When the measure is finite, f e implies that, for A e
cation of
the 7:-
.
./
.4
.4,
F(X3
so that 1.26(b) additivity,
=
F(f)
-
XX)
=
LUIf) -
$(A)
=
>'(A3,
(3.29)
holds. Lastly, 1et fA.j) be a disjoint sequence in co
p,
UAp
j=3
X
oo -
77p,(A> j=
-
.4.
By countable
G)
Xp,-+) -
j=1
p,' UAk
(3.30)
,
y=1
and 1.26(c) holds. It follows by 1.26 and the 7:- theorem that 5 c(./) ? W. VjBj where Bj e and pBp Now consider the c-finite case, Let ( g,'(#j) < co. % fBj rnA: A e 5 J is a c-field, so that the (Bj,5) are measurable spaces, =
./
=
=
=
Measure
45 ./
fe'h Bj. The preceding argument on which jt and $ are tinite measures agreeing on showed that, for A e 5, g(#j rnz4) bt'Bj rnA) only if g and g,' are the same =
measllre.
Consider the following recursion. By 3.3(ii) we have p.(Ara (#1t..p#2)) Letting Cn
=
f''h #2) g.(X f''h #1 f''h #2). g,(z4r'7#1) + h1(z4
=
U7=l the
(3.31)
-
same relation yields
(3.32)
rn cn) p.tz4rn Bn) + p,tA ra cn-j) p,tz4rn Bn rn cn-1). g,ta4 =
-
The terms involving Cn-l on the right-hand side can be solved backwards to yield f-'h an expression for g,(A G), as a sum of terms having the general form g,(Ara Bjj rn Bjzra Bj, ra
for some j, say j that gtD ra Bj) that in (3.32) =
=
j, in which ease D
jt'(D ra B? for all D g.(Afa Cn)
olds for any n. Since Cn
This
=
G
=
=
=
g,to ra Bj)
(3.33)
< x
A fn Bh fe'h Bh fa e F. Since we know preceding by 5 the argument, it follows ...
g,'(4 fa Cn4. f as n
--)
F(A)
)
...
-->
x,
(3.34) we obtain in the
F'(A),
limit
(3.35)
the two sides of the equality being either finite and equal, or both equal to This completes th proof, since A is arbitrary. w
+x.
3.14 Example Let A denote the subsets of R which are measurable according to (3.14) when g,* is the outer measure defined on the half-open intervals, whose measures go are taken equal to their lengths. This defines Lebesgue measure m. 'hese sets fol'm a semi-ring by 1.18, a countable collectioh of them covers R, and the extension theorem shows that, given m is a c-finite measure, A contains the Borel field on R (see 1.21), so (R,f,?n) is a measure space. It can be shown (we wonpt) that a11 the Lebesgue-measurable sets not in B are subsets of T-sets of includes a1l measure 0. For any measure g, on (R,f), the complete space @,SB,F) of the Lebzsgue-measurable sets. In The following is a basic property of Lebesgue measure. Notice the need to deal with a countable intersection of intervals to determine so simple a thing as the measure of a point. 3.15 Theorem Any countable set from R hs Lebesgue measure 0. Proof The measure
of a point
(.x) is
zero, since for x e R,
@Q
fx) ppd :
.
'
'
??z((<))q
'
: ''
' '
=
limn-jxl/n
=
=
r)
n=1
(x
-
1/n, xj e B
0. The result follows by 3.6(ii). w
(3.36)
Mathematics
46 3.3 Non-measurability
To give the ideas of the last section their true force. it needs to be shown that D X c 2 i s poss ible in other words, that ( can contain non-measurable subsets. In this section we construct such a set in the half-open unit interval (0,11,a standard counter-example from Lebesgue theory. For m y e (0,11, define the operator ,
.#4.x
A'+
1,
(3.37)
=
y+x- 1,
This is addition modulo 1. Imagine the unit interval mapped onto a circle, like a clock face with 0 at the top. y x is the point obtained by moving a hand clockwise through an angle of 2= from an initial point y on the circumference. For each set A (0,11, and x e (0,1J, detine the set .i.
A J.x
=
(y 4.x:
AJ.
l
y
(3.38)
3.16 Theorem If A is Lebesgue-measurable so is A %x, and m(A 4.a') any
=
mA), for
X.
for any real x such b +x1) b c lna,bj), unions > A-x + of intervals extends < 0 and b 1. The that a x to finite property translated by x. If A is any Lebesgue-measurable subset of (0,11,and A A-x (0,11 where A + x (y+x: y e A J, the construction of the extension similarly is measurable and mA) mA +x). implies that A and A2 A f-h (1 Now let A1 A /'7 (0, 1 11. Then pztdl +x) -(A1) and mAl + x 1) mA1), where the sets on the left-hand sides of these equalities are in each case contained in (0,11. A1 +x and A2 +x 1 are disjoint sets whose union is A %x', and hence Proof For
(J,!71
(0,11,ma
i
+x,
=
=
-
=
..x
=
-x),
-
-x,
=
=
=
=
-
mA 4.x)
rntz4l +x)
=
+
mAz
= rn(A1) + mA
+
=
x
-
1)
m(A). w
(3.39) .i.
Define a relation for points of (0,1q, letting xRy if y = x r for r e :. That is, xR y if y is separated from x by a rational distance along the cifcle. R is an equivalence relation. Defining the equivalence classes Ex
=
fy: y
.i.
=
.x
r, r e
t1),
(3.40)
(fI,
the sets of the collection x e (0,11 ) are either identical, or disjoint. distance is rational from some other point of the interval, these Since every x a sets cover (0,11.A collection formed by choosing just one of each of the identical sets, and discarding the duplicates, is therefore a partition of (0,11. (FA, Write this as x 6 C), where C denotes the residual set of indices. Another example may help the reader to visualize these sets. In the set of integers, the set of even integers is an equivalence class and can be defined as 0 the Ket of inteeers which differ from 0 an even integer. Of course F0 El = =
sfeasure
47
Eln, for any n l Z. The set of odd integers can be detined similarly as F2n+1 E 1 the set o f integers differing by an even integer from 1. F1 = F3 (A'A, for any n e Z. Discarding the redundant members of the collection e 11 F1 0 collection LE partition of the Z. d leaves just J to e tine a Now constnlct a set H by taking an element from Ex for each x e
#
=
=
...
=
,
...
= .x
,
3.17 Theorem H is not Lebesgue-measurable. Proof Consider the countable collection (H # r, r l (I)) We show that this collection is a partition of (0,11. To show disjointness, argue by contradiction. Suppose z e H rl and z e H r2, for rl # r2. This means there are points 1, hl e H, such that .
'i.
.i.
.i.
l rl
.i.
=
z
=
(3.41)
hz r2.
If rl # r1, we cannot have Jll = 2; but if Jll # hl then Jll and hz belong to different equivalence classes by construction of H, and cannot be a rational distance lrl r2 I apart; hence no z satisfying (3.41)exists. On the other hand, LjrH r), and consider any point let H e (0,11. belongs to one of the rational distance of some element of H; equivalence classes, and hence is within a but H* contains al1 the points that are a rational distance r from a point of H, for some r, and hence x e H., and it follows that (0,11c S*. mlH4 for all r. Since mH*) k Suppose mllt exists. Then by 3.16, mH% r?z((0,1J) 1, we must have m+ > 0 by 3.6(ii), but countable additivity then rnzbul'-l r) which is impossible. It follows that m+ does not gives mH.) -
.i.
.&
.x
=
=
=
=
=
=,
exist. w The definition of H involves a slightly controversial area of mathematics, since the set o f equ ivalence classes is uncountable. It is not possible to devise, even in principle, constructive rules for selecting the set C, and elements from Ex for each e C. The proposition that sets like H exist cannot be deduced from the axioms of set theory but must be asserted as an additional axiom, the so-called axiom of choice. lf one chooses to reject the axiom of choice, this counterexample fails. We have made no attempt here to treat set theol'y from the axiomatic standpoint, and the theory in Chapter 1 has been what is technically called nat've is). For us, the problem of (i.e. based on the intuitive notion of what a the axiom of choice reduces to the question: should we admit the existence of a mathematical object that cannot be constructed, even in imagination? The decision is ultimately personal, but suffice it to say that most mathematicians are willing do SO Sets like H do not belong to (0,1j = f# r'h(0,1j, B e B ). It is not difficult to show that a11the sets of (o,1j are Lebesgue-measurable', see 3.14 and restrict m to (0,1) as in 3.2. By sticking with Borel setf we shall not run into measurability difficulties on the line, but this example should serve to make us careful. ln less familiar situations (suchas will arise in Part VI) measurability cah fail in superficially plausible cases. However, if measurability is in doubt one might remember that outer measure g,*is well defined for a1l subsets of f, and coincides vk
tset'
X
to
.
Mathematics
48
with z whenever the latter is defined. Sometimes measurability problems are dealt with by working explicitly with outer measure, and forgetting about them.
3.4 Product Spaces If (f1,T) and (E,T) are two measurable fxE
((,():
=
(t) l
spaces, 1et
f,
(
G
El
(3.42)
be the Cartesian product of f and E, and define ?F@V Spg
=
((Fx G), F e @, G
G
=
c(Spg), where
V).
(3.43.)
a product space, and (f,F) and (E,W) are the coordinate factor spaces, or spaces, of the product. The elements of the collection Sys are called the measurable rectangles. The rectangles of the Euclidean The space
(f1x E, 5 * V) is called
plane R x R
=
R2
(produc ts
of intervals) are a familiar case.
3.18 Example Rather trivially, consider the two-element setsA B f(1,(21e N. The corresponding rectangle is
=
(1,(1
e @,and
=
A XB
=
1), (t,)1,t2), (*2,(1),()2,t2)l. 1(t,)1,('
The sets f(1,(1), ((*,,(2))and ((1,(2), (2,(1)) are not rectangles, unions of rectangles and so are elements of 5 @ T. n
but are
Two important pieces of terminology. If E c f x E, the set an(F) (: (,() e F) is called the projection of E onto f1. And if A D, the inverse projection of A is the set =
o-1(A)
A xE
=
=
(4),4):
)
e A,
(
EJ
e
(3.44)
.
A x E is also called a cylinder set in f x E, with base A. The latter terminology is natural if you think about the case = R2 an d E R Cylinder sets with bases in 5 and T are elements of R,e. One might tlnk that if E e 5 @ N, antA-) should be an F-set, but this is not necessarily the case. :tE'f # ln(M) in general (see is not closed 1.3) so that the collection C of projections of 5 (&N-sets onto = notice under complementation. However, that A ao(A x Z) so that 5 i C. The main task of this section is to establish a pair of results required in the construction of measures on product spaces. =
.
3.19 Theorem If C and b are semi-rings of subsets of and E, respectively, then Sws = (CxD: C e C, D e 01 is a semi-ring of
x E.
Proof There are three conditions from 1.16 to be established. First, Ses clerly contains 0. Second, consider C1, Czq C, andDlyDz G 0. Q rn Czq CandDl rhDZ e D, and as a matter of definition, (C1 xD1) rn (Q xD
=
()
e
Q, (
c Dj
) fa (
e
Q,
(
e Dal
Measure
49
= ((o e cl c2, ( e ol = (c1ch ca)x (D1 cjD1) e Rwe. fazhl
fa
Third. assume that C1 xo2)
xDl i Cz xD2, and by a similar argument,
CI
(c1 xD1)
-
(3.45)
((
=
(
e Cz,
= ((C2
-
e D2): either
cl or (
)
D1l
C1) XD1) tp (C1 X (D2 -D1))
tp ((Q
C1) X (D2 -D1)).
-
(3.46)
where the sets in the union on the right-hand side are disjoint. By hypothesis, the sets Cz C1 and D1 DI are finite disjoint unions of C-sets and O-sets respectively, say (47,...,Cn'), and (D(,...,D,,,'). The product of a finite disjoint union of sets is a disjoint union of products; for example, -
-
n
n
U
/=1
C; x D1
(tl),():(t
=
Uts (
e
,
e D1
=
UICJx D1).
(3.47)
./=1
j=1
Extending the same type of argument, we may also write m
n
(ca cllxtoa-ol) -
-
Utclxol) U(c1xoi) k-p
kp
k=1
.j=1
U(cJxo;) (3.48) .
j,k
All of the product seys in this union are disjoint (i.e., a pair ((t),))can appear in at most one of them) and al1 are in Scs. This completes the proof. . The second theorem leads to the useful result that, to extend a measure on a productspace, it suffices to assign measures to the elements of Scs, where C and D are suitable classes of the factor spaces.
3.20 Theorem If 5 = c(C) and T c(O) where C and D are semi-rings of subsets of f and E respectively, then 5 @ V = C @ 0. =
Proof It ij clear that Rwo i Sps, and hence that C (8)D converse, consider the collection of inverse projections, .t/a (J:o-1(#), F e Fj pwg.
5 (8)W.To show the
c
=
,/p
is a c-field of fl x E, and is in fact the smallIt can easily be verified that is a (;o'(C), C e C J i C (>))0. est c-field containing the colletion and since it follows (& is and hence D c-field by the T l-system, a a theorem that c(#c) ? C @)0. Exactly same conclusion holds for #s, the orresponding collection for V. Evel'y element of Sss is the intersection of an element from #v and one from #'g, and it follows that Spg c C @D. But Sys is a l-system by 3.19 and hence a further application of the 7:- theorem gives 5 (8)N ? u gs; , .1
vt/c
,t/c
=
-system,
xt/p
'
=
irlik pztjlptlof > product extends beyond pairs to triples and general n-tuples, and gd jjjajj be jnterested in the nrooerties of Euclidean a-qnnne ffRlh '
y:
,;r-.
..
.. .
. .
.
.
jjtyj)jjjujf
50
Mathetnatics
For tinite n at least, a separate theory is not needed because results can be obtained by recursion. If (Ylf,X)is a third measurable space, then trivially, fxE
x'P
((,(,v):
=
(
e f,
(,)
= f((,(),v): (,() E) xT. = (f x
e E,
ve
e DXE,
v
t1:l
e .P1
(3.49)
Either or both of (f),T) and (E,V) can be product spaces, and the last two theorems extend to product spaces of any finite dimension.
3.5 Measurable Transformations
F: D
F->
(E,N) in a different context, as domain
and
Consider measurable spaces (f,T) and codomain of a mapping E.
if F -1(#) l 5 for a11# l N. The idea is that a to be sl%-measurable measure p, defined on (,T) can be mapped into (E,V), every event B e V being assigned a measure v(#) g,(F -1(.#)) We havejust encountered one example, the projection mapping, whose inverse defined in (3.44)takes each T-set A into a measurable rectangle. Corresponding to a mesurable transformation there is always a transformed in following the sense. measure, F is said
=
.
3.21 Theorem Let jt be a measure on (f,F) and F: f ation. Then P.F -1 i s a measure on (E V) where ,
jLF -1(#)
E a measurable
F-
transfonu-
,
-1
=
(#)), each
g,(F
B e g,
(gj5o
.
.
Proof we check conditions 3.1(a)-(c). Clearly g,F -1(4 ) 1 0 a1l A e Bv. Since holds by definition, T-1(O) T -1(E) O by 1.2(iii) and so g,F-1(3) p,(T -1(O)) g,(O) 0. For countable additivity we must show ,
=
=
=
=
=
g,F-1
for a disjoint collection that the B; are disjoint becomes
U#y j
=
X/tF-1()
(3.51)
j -1
F Bj4, 1.2 shows both Bj,B1,... e E. Letting B) and that F-1(Uy#y) Equation (3.51) therefore Uy#J. =
=
p,Ur j
=
X g,(#J)
(3.52)
./
for disjoint sets Bgq,which holds because g, is a measure. The main result o
general transformations
.
is the following.
3.22 Theorem Suppose F -1(#) l 5 for each B e D where D is an arbitrary class of sets, and N c4D). Then the transfonnation F is T/v-measurable. ,
=
51
Measure
(
$.,
.
.
.
,
.
,
=
,
j'
T-3LjBj)
T-3Bj)
q @,j q s, then proof By 1.2(ii) and (iii), if l then @ @ It follows d if z'-1(#r) z'-1(#)c z'-1(,) E e ?yan
,
k
=
UjF-1()
that the class of
CtS
a is a c-field.
Since D
t,: w-1(s)s s)
.
N c 4 by definition.
.4,
c
.
This result is easily iterated. If (t1#,J't)is another measurable
is a V/x-measurable transformation, then &oF: (
F-+
space and U: E
F->
VIJ
VP
is F/s-measurable, since for C -1
R, W 1(C)
E
-1
(&0D (0
F
=
N an d hence
E
,
-1
(&
(C))
9.
1
(3.53)
E is called a measurable isomorphism if it is An important special case: F: f' 1-1 oflto, and both F and F -1 are measura ble The measurable spaces (2,1) and (E,N) are said to be isomorphic if such a mapping between them exists. The implication is that measure-theoretic discussions can be conducted equivalently in either (f,T) or (E,V). This might appear related to the homeomorphic property of realfunctions, and ahomeomorphismis indeed measurablyisomorphic. Butthereis no implication the other way. -..:
.
r0,11 F--> (0,11,defined
3.23 Example Consider g: #(A)
x+
,1
0 f x f 1 2
x
1
l <
=
by
(3.54)
.
-
2,
1)
x < 1
Note that g is discontinuous, but is 1-1 onto, of bounded variation, -j Sgo,lj/fgo,lj-measurableby 3.32 below, and g g. I::I
and hence
=
The class of measurable transformations most often encountered is where the codomain is (R,O), B being the linear Borel tield. In this case we speak of a function, and generally use the notation f instead of F. A function may also have the extended real line (RlW)as codomain. The measurability criteria are aj
follows.
3.24 Theorem (i) A function /': f
R for which (): J(() f x) e ?Ffor each x E (Dis So is a function foz which (): fk < x) e 5 for each
F-+
T/f-measurable. x e (D (ii) A function fl f.1 F-> 2- for which f(t): Q tp (+x) t.p (-x) is T/f-measurable. .
.f4))
f xl e ?F for each x e -1(#)
Proof For case (i), the sets f(t): flk f xl are of the form f B e T' where C is defined in 1.21. Since B c(C), the theorem follows by 3.22. The other cllection indicated also genertes :B, and the same argument applies. The extension to case (ii) is equally straightforward. . ,
=
52
Mathentatics
The basic properties of measurable functions follow directly.
3.25 Theorem (i) If f is measurable, so are c + f and c#, where c is any constant. (ii) If f and g are measurable, so is f + g. Proof If .X
e
jq
f
K x, then
f+
c S x + c, so
that f
c is measurable
+
by 3.24. Also, for
,
(: (:
c/)
XJ
S
/*()
((t): /)
=
< xIc ),
c
>
0
< xlclc,
c
<
0
c
=
c
=
f1, 0,
where for each of the cases on the right-hand side and each x/c in 5, proving part (i).
If and only if lt follows that
(:
J())
f+g
+.g((t))
<
there exist r e ) such that
.x,
U (2
.z')
<
=
f(l
<
r)
f-n
(:
(3.55)
0 and x k 0 0 and x < 0
f
<
r
<
R the sets are
6
x g -
:(t0) < x- r1.
(see 1.10). (3.56)
rq Q
The countable union of T-sets on the right-hand side is an T-set, and since this holds for every x, pal't (ii) also follows by 3.24(i), where in this case it is convenient to generate tB from the open half-lines. >
Combining parts (i) and (ii) shows that if h,...,fn are measurable functions so is Xhjcjfj, where the cj are constant coefficients. The measurability of suprema, infima, and limits of sequences of measurable functions is important in many applications, especially the derivation of integrals in Chapter 4. These are the main cases involving the extended line, because of the possibility that sequences in R are diverging. Such limits lying in V are called extended functions. 3.26 Theorem Let f/l be a sequence of F/s-measurable functions. Then infn/n, supn/k, liminfln, and limsuw/ are F/W-measurable. Proof For any x e
t:
F (: fnlk
f xl s 5 for each n by assumption. Hence
,
supnlnt)
<
.z')
=
Of):
n=1
fnk
so that supn/ is measurable by 3.24(ii). Since infn/'n obtain (:
infn/t)
.x)
<
(: sup,,(-Jn())> -xJ = (: supn(-Jn())S -xJf =
@,
S xl e
-supnt-ln),
=
(3.57) we also
Measure
53 C
00
O
n=1
la''. -Tn(a')K
-xl
X
=U1e):/()
<
5.
xl e
(3.58)
n=1
To extend this result from strong to weak inequalities, write X
f xl
infn/kt)
(:
O(:
=
infn/kt)
<
x+
?n=1
Similarly to
(3.59)
5,
(3.60)
may show
(3.57),we
1:
1/rlJ e 5.
supknfkt)
f
Ot:
.x1
=
.fn()
S
.x1
e
kkn
and applying (3.59)to the sequence of functions gn (:
limsupn/t)
sums/k
=
yields
K xl e F.
(3.61)
In much the same way, we can also show f x) e @.
( (t): liminfn/t)
(3.62)
condition of 3.24 is therefore satisfied in each case.
The measurability
.
We could add that limn/kt) exists and is measurable whenever limsupn/t) liminfs/kt). This equality may hold only on a subset of f1, but we say fn converges a.e. when the complement of this set has measure zero. The indicatorfunction 1s()) of a set E e 5 takes the value 1s() 1 when (.t)E characteristichmction otherwise. and 1s((l)) 0 Some authors call ls E, of F. the It may also be written as fs or as zE. We now give some useful facts about indicator functions. =
=
=
3.27 Theorem (i) 1s() is 5IB measurable 1 1s(). (ii) 1sc()) (iii)1Ujsj((l)) sup 1sf(). =
if and only if E
G
F.
-
=
i
(iv) lnjsjt)
inf 1s/)
=
i
=
I-l1sj((l)). i
Proof To show (i) note that, for each B e .S,
1F-1
(#)
'Fhse
sts
ithediyte
=
if 0 G B and 1 G B E if 1 e #, 0 # Ec if 0 e B, 1 , B 0, otherwise
(3.63)
are in 5 if and only if E e @. The other parts of the theorem fwm tlw dfinition. .
re
Mathematics
54
Indicator functions are the building blocks for more elaborate functions, constructed so as to ensure measurability. A simplefunction is a T/o-measurable function J: fl F...: R having finite range', that is, it has the fonn
ftl
af, Xaflsjtt9l =1
=
=
tt e Ei,
(3.64)
where the G1,...,(xa are constants and the collection of T-sets F1,...,Fa is a tinite partition of f1. T/f-measurability holds because, for any # 6 B,
J-1(,)
U Ei s
=
ai
@.
(3.65)
B
EE
Simple functions are ubiquitous devices in measure andprobability theory, because many problems can be solved for such functions rather easily, and then generalized to arbitrary functions by a limiting approximation argument such as the following.
l
Fig. 3.1
3.28 Theorem If f is T/o-measurable and non-negative, there exists a monotone n e EN I such thatnlt) f sequence of T/f-measurable simple functions f/'(n), for evel'y (l) e f1. '
1,...,n2'l, consider the sets Ei = f ): i 1)/2' < il2n ) Proof For i f Augment these with the set Enzn.. f : J() k nl This collection corresponds to a n2n + l-fold partition of (0,x) into S-sets, and since f is a function, each and hence belongs to one and only one Ei. The Ff maps into one and only one therefore constitute a partition of f1. Since J is measurable, Ei s 5 for each i. Define a simple function hn)on the Ei by letting i i 1)/2/, for i = 1,..., nln + 1. Then fn) f f, but A+1())k /4)) for every ); incrementing n bisects n+1)()) is equal to either i 1)/2*, each interval, and if n)()) n)(), 1)/2'+1 &+1 > follows that the sequence is 1)/2 2/ It li or ( J. (). =
-
.
=
.
.f4)),
=
=
-
=
-
-
-
sieasure
55
J(). This holds for each ) e f. To extend from monotone, and lims-yxnlt) non-negative to general functions, one takes the positive and negative parts. Define J+ maxlf,ol and ff'b f, so that both f- and J- are non-negative functions. Then if f'Injand j'-n) are the non-negative simple approximations to it is clear that f+ and f- defined in 3.28, and fnj f =
=
=
-
.f+(n)
-(n),
=
-
l/-J(n)I < lJ+-.f+(n)l+ 1.f---(n)I Fig. 3. 1 illustrates the construction for n is a function on the real line.
=
0. w
--.h
2 and the case
(3.66) =
R, so that
.f()
3.6 Borel Fqnctions lf J is a measurable function, and 5'; S g R, 'T g R is a function of a real variable, is the composite function gof measurable? The answerto this question is yes if and only if g is a Borel hmction.Let Ss (# ros: B e S), where B is the Borel field of R. fs is a c-field of subsets of S, and B DS is open (closed)in the relative topology on S whenever B is open (closed).in R (see 2.1 and 2.3). fs is called the Borel field on S. Define fv similarlywith respect to 7. Then g is called a Borel function (i.e., is Borelmeasurable)if g -1 B) e B s for all sets B l f.r.
gl S
F-+
=
3.29 Example Consider gx) Ixl g -1 ta kes e ach point of R+ into the points x is the set For any B E B + (th e res triction of B to R+) the image under and for each a7 e #, which is an element of f. E1 containing the points x and =
.
.g-1
-x.
-x
1 if is rational, 0 otherwise. Note that (D e B (see 3.30 Example Let g(x) -1 3.15), and g i s defined according to (3.63)with E Q, so g is Borel-measurable. I:a .:
=
=
non-measurable ln fact, to construct a following. but the obvious case is the
4plausible'
function is quite difticult,
3.31 Example Take a set A f; for example, let A be the set H defined in 3.17. l Now construct the indicator function 1x@): R F-> (0,1 ) Since 1x - (f 1 1) A gp Shis function is not measurable. Ia .
=
,
Necessary conditions for Borel measurability are hard to pin down, but the following suftkient conditions are convenient.
3.32 Theorem lf g'. S it is Borel-measurable. Pmof
(i) follows
F-+
'T is either
immediately j cotinuity implies that h B) in T, by 2.17. To prove (ii), consider first prperty h@l S @)when y < ..
(i) continuous
or
(ii) of
botlnded variation,
from 3.22 and the definition of a Borel field, since is open (closed)in S whenever B is open (closed) a non-decreasing function h: R F.-y R, having the x; if A (y: h@l S (x)), sup A = x and A is one =
Mathematics
of (-x,x) and (-=A, so the condition of 3.24 is satisfied. So suppose g is nondecreasing on S; applying the last result to any non-decreasing h with the property hx) = x e S, we have also shown that g is Borel-measurable because = -1 h-'B4 5') f''ASe Ss, for each Brh'T e O.r. Since a function of bounded g CBt''h variation is the difference of two non-decreasing functions by 2.20, the theorem now follows easily by 3.25. . '(x):
This result lets us add a further case to those of 3.25.
3.33 Theorem If J and g are measurable, Proof
jg
=
with 3.25(ii).
z1((J+:)2
-
fl
gl)
-
so is
fg.
and the result follows on combining 3.32(i)
.
The concept of a Borel function extends naturally to Euclidean a-spaces, and indeed, to mappings between spaces of different dimepsion. A vector function 7,' S i R 1- R'/ gl S --./
,
is Borel-measurable if g -1(#) e B s for a11B e Sv, where Bs and fv (# r-75': B e f'''J.
=
(# taS:
# e
l
=
3.34 Theorem
lf g is continuous. it is Borel-measurable.
Proof By 2.21. w Finally, note the application of 3.21 to these cases.
3.35 Theorem If jt is a measure on (R Bk) an d g : S s.y is Borel-measurable where S c Rk an d -Erc R'/ gg is a measure on (T,Sv) where 'r
,
-1
,
gg-1(B)
t-1(s))
=
(3.67)
,
for each B e f.r. n R'N
for m < k. If X is A simp le example is where g is the projection of R onto k x 1 with partition X' = (21,7+'.),where X. is m x 1 and X.. is k m) x 1, 1et gl R k F-> R*' be defined by -
r(A') -1
In this case, g,g B)
=
X..
-1
=
P.W(#))
=
jt(#
xR
(3.68) k-m
)
or p s qrrl .
4 Integration
4. 1 Construction of the lntegral The reader may be familiar with the Riemann integral of a bounded non-negative function f on a bounded interval of the line gJ,:J, usually written fdx. The objects to be studied in this chapter represent a heroic generalization of the same idea. lnstead of intervals of the line, the integral is defined on an arb-
itral'y measure space. Suppose
(f1,@,g)is :
l'1
O
a measure space and
2-6
is a T/s-measurable function into the non-negative, extended real line. The integral of f is defined to be the real valued functional
jfdb
Sup
=
tXj
fte)ljpfj
tinsfo
(4.1)
where the supremum is taken over a11finite pmitions of into sets Ei e T, and the supremum exists. lf no supremum exists, the integral is assigned the value +x. 5 T he integral of the function IAJ,where 1x() is the indicator of the set 4 e 5, is called the integral of f over A, and written fdbk. The expression in (4.1) is sometimes called the lower integral. and denoted sfdp. Likewise defining the upper integral of f,
j'ydb -
inf
E i
(supsy/j bte'i)
(4.2)
,
G
f from below and from we should like these two constnzctions, approximating .fdbk J'/W/whenever above, to agree. And indeed, ttis possible to show that f *fdbk oo if either the set > 0) has is bounded and g(f1) < x. However, (: infinite measure, or f is unbounded on sets of positive measure. Definition (4.1) is preferred because it can yield a finite value in these cases. =
.f()
=
4.1 Example A familiar case is the measure space ('R,f,-), where m is Lebesgue where f is a Borel function is the Lebesgue integral of measure. The integral Ifdm f. This is customarily written j'fdx,reflecting the fact that mx, x + #x1) tfx, even though the sets (A'j) in (4.1) need not be intervals. n =
4.2 Example Consider a measure space (R,f,g,) where g, differs from m. The where f is a Borel function, is the Lebesgue-stieltjes integral. integral Ifdbk'
Mathematics
58 The monotone function Fxt
=
p,((-x,x1)
(4.3)
F(b) F(c), and the measure of the interval has the property (x, x + #< can be written JF@). The notation IfdF means exactly the same as J/Wg.,the choice between the p. and F representations being a matter of taste. See j8.2 and j9.1 for details. n g,((J,:1)
=
-
For a contrast with these cases, consider the Riemann-stieltjes integral. For an 1et a partition into subintervals be defined by a set of points H interval Etz,!71, Akl, < < xn with a = xo < b. Another set H' is called a = (x1,..., refinement of Fl if IR c H'. Given functions f and a: R 1-+ R, let .x1
=
...
5lRvas.fl
=
77f(f)(G(A)
-
a(zf-I)),
(4.4)
i=1 bafd%,
here ti e (xf-l,xf). If there exists a number there is a partition IQ with W
(&l-I.a,.f)
'-jajd
such that for every : > 0
b < 6:
l
for all FI Q IQ and every choice of fff), this is called the Riemann-stieltjes integtal of J with respect to a. Recall in this connection the well-known formula for integration by parts, which states thqt when both integrals exist,
fbtub)
=
/J)a(J)
+
jafd ja@f' +
When (x = x and #is bounded this definition yields the ordinary Riemann integral, and when it exists, this always agrees with the Lebesgue integral of f over gtz,:). Moreover, if a is an increasing function of the form in (4.3),this integral is equal to the Lebesgue-stieltjes integral whenever it is defined. There do exist bounded, measurable functions which are not Riemann-integrable (consider3.30 for example) so that even for bounded intervals the Lebesgue integral is the more inclusive concept. However, the Riemann-stieltjes integral is defined for more general classes of integrator function. In particular, if f is continuous it exists for a of bounded not necessarily monotone. These integrals therefore fall variation on gtz,&), the class defined by (4.1), although note that when a is of bounded variaoutside tion, having a representation as the difference of two increasing functions, the Reimann-stieltjes integral is the difference between a pair of Lebesgue-stieltjes
integrals on (J,:1. The best way to understand the general integral is not to study a particular measure space, such as the line, but to restrict attention initially to particular classes of function. The simplest possible case is the indicator of a set. Then, every partition fEiI yields the same value for the sum of terms in (4.1),which is
jgdn
Integration
59
jxdp
(4.6)
BIX),
=
=
5, the integral is undefined.
for any A e 5. Note that if A
Another case of much importance is the following.
4.3 Theorem lf J
0 a.e.lgl, then
=
fdp
Proof The theorem says there exists C e C. For any partition (Fl,...,&J 1et A'; ity of g,,
77 i
inf (l) e
Ei
/.() g,(e)
=
Xi
inf (l) e
F;
=
0.
1, such that /4)) 0 for ) Ei rn C, and A': Ei F;. By additiv-
with
gtc)
=
=
=
=
/.()l g,(e':) +
Xi
inf f e e:
-
g,(rJ)
= 0,
(4.7)
the first sum of terms disappearing because /*4) 0, and the second disappearing =
by
.6(i)
since pE'i) f g,(C')
0 for each i.
=
.
A class of functions for which evaluation of the integral is simple, as their name suggests,is the non-negative simple functions.
4.4 Theorem Let (?() 5 is a partition of f.
=
where aj k 0 for X:=1%1sj()),
i
=
1,...,n, gnd EL,...,En E
Then
tFp, 77
(4. 8 )
ipEi).
=
=1
Proof Consider an arbitrary finite partition of f1, A1,...,4-, and define inftxxyst). Then, using additivity of g, Ip,(Ay) X SX 77 f=1 .j=1 j=3 =
<
i=1
xi
I
=
p,(AyraEi4
jtt/tjc
Ei4
j=L
(4.9)
ipEi),
= f=1
where the inequality uses the fact that I assumes the smallest value of Gj such that Aj ra Ei # 0, by definition. The theorem follows, given (4.1), since (4.9) holds as an equality for the case m n and Ai = Ei, i 1,...,n. w =
=
So for functions with finite range, the integral is the sum of the possible values of f, weighted by the measures of the sets on which those values hold. Look at Fig. 3.1. The Lebesgue integral of the approximating function fz) in the figure is the sum of the areas of the rectangular regions. To compute the Lebesgue-stielties inteeral with resoect to some measure u.. one renlnces the width of the qetq
62
Mathematics
= yaj
a simple function. Hence,
ja-bdb
+ tzj,
s Af ra Bj,
(t)
(4. 17)
X Xtla, +bbbviAil-bB?
-
j
i
=
zxGf7r/ttArnl
=
'7rafgtA
=
i
,xfs>7/ttAf
+
j
j
rn B?
i
bTfbbkB)
+
j
i
ajdp
+
bjydp,
(4.18)
showing that linearity applies to simple functions. Now applying 4.6,
jaf
+
bgldp
ja jygjtj jt?lj1jsyusp, (suspy +
sup
=
bz4dbk
(pf/f'
=a =
+
b
#
ajfdp
+
bjgdp.
(4. 19)
To extend the result to general functions, note that
la.f
1 If I + Ib I Ig l
bg l < a I
+
.
.
(4.20)
,
so (4.19) shows that af + bg is integrable so long f and g are integrable, and a and b are finite. The identity af
+
bg
=
afl'
afl-
-
bg4'b (bg)-
+
(4.21)
-
implies, applying (4.19), that
jaf
bntdb
+
If a k 0, then
ab+dk -
=
jaldb
afl+dbk
aft-dbk
=
J(&T)-*+ Jt/WI+dB
-
-
afl-d
Ia I f-db
=
-
Jtf/'+lt
f+db
-
=
ment applies to the terms in b and g. So
JJ-Jg,)
IJI
-fd
'-jb8t-db.
=
(4.22) ajfdbk, whereas if a < 0, alfdb. The
=
(4.16) holds as required.
same argu.
is a very useful property. The first application is to show the the integral to the behaviour of functions on sets of measure 0, of invariance extending the basic result of 4.3. Linearity
4.8 Lemma Let f and g be integrable functions. (i) If f f g a.e.lg1, then fdkt; hdp. dp. (ii) If f = g a.e.gg,l, then J/Wg,= Proof For (i), consider first the case J 0. lf g 2 0 ekerywhere, directly from (4.1). So suppose # k 0 a,e.rg,l and define =
dbk 2 0
63
Integration
Jl4)) Then h
0 a.e.gp,) but g
=
jg
0 S
fhdbk0andby
since
J('
To prove
JJJF
2 0
=,
#()
<
.
0 4.7,
jgdp jhdpjgdp, +
=
(4.23)
=
0,
(ii), let
h
J(' + hjdbk
=
:()
h k 0 everywhere, and, applying
hjdp
+
0,
4.3. Now replace g by g J in the last argument to show hence dbkk J/Wptby 4.7.
=
-pdp
+
=
-
=
dp
=
f
-
+
0 by 4.3. Then g so that h 0 a.e.gg,l, and hdp hdbk J#JF, Where the second equality is by 4.7. w =
=
=
These results permit the extension to the more commonly quoted version of the monotone convergence theorem.
4.9 Corollary lf fn k 0 and fn
'1'
limn-oxl/nv
J a.e.g.l,
=
jdp. n
T
Another implication of linearity is the following.
j
Proof
j lf 1 jf'
inequality
4.10 Modulus
lJ lJz
Jjt 2
+#-)tfp,
=
j jjj'dp jf jf'bdp .
-#p,
=
+
Jy-#>l jJ/-d: )Jy::1. .
-
-
In the form of 4.9, the monotone convergence
theorem has several other useful
corollaries. 4.11 Fatou's
lemma If
Jn k
0 a.e.Lg1,then
liminf fndp k nMx
j
liminf
Jn Jg,.
n->oo
Proof Let gn infkn/'k, so that (,n) is a non-decreasing sequence, and gn 1% g sides oo 2 Igndp. Letting n both of liminfn/n. Since Jn k gn, Jfntkyt the on inequality gives =
=
--+
liminf fndp 2 lim gndp n.-+txz
=
gdp
liminf
=
n-M
4.12 Dominated convergence theorem If fn
--
that IfnI
fn #p..
(4.24)
w
n--x
S g a.erlp1 for al1 n and
gdbk <
x,
f a.e.lgj, an there exists g such then JlnIg, .-A JJJP,.
Proof According to 4.8(i), dp < x implies JlfnI#g, < =. Let hn Ifn f I such that 0 f hn K 2g a.e.lg,j and hn 0 a.e.ggj. Applying 4.3 to liminfns, linear=
-->
jw.
.t.......:.')..:..
t.............( .) ..... .:
.......
Fatu' s lymma, attl .).( ,
.'). '
..
.(..
.
...
E .y. ... . .
..
.
.. .
j.... ..
.
-
,
64
Mathematics
ljgdp
J
=
=
liminftzg
-
j (lg jhndp
hn) Jjt f liminf
-
hnjdp
N->x
n-/co
jgdp
limsup
-
(4.25)
,
N-+=
where the last equality uses (2.4). Clearly, limsupn-yxf/lnzv k 0 the modulus inequality implies 1im n-e
=
0, and since
jfndp j/'ljt jhndp < 1im
-
=
0.
j'hndb (4.26)
.
n'M
Taking the case where the g is replaced by a tinite constant produces the following version, often more convenient:
4.13 Bounded convergence theorem If fn then limn--jxlfntfg, Ifdt < x. E1
--y
f a.e.gg1 and IA! f
B<
x
for all n,
-->
Theorem 4.7 extends by recursion from pairs to arbitrary finite sums of funcPut fn tions, and in pmicular we may assert that J(Z:=1'i)Jg, = Xl=tgiand fndbk l=tlgid, where the gi are pon-negative functions. Then, if fn -1xf Z7=l'j < x a.e., 4.9 also permits us to assert the following.
Z:=1J##g..
=
=
=
4.14 Corollary
If
f'fl is
a sequence of non-negative functions,
J Ng'
,
-
# jg,
.
(4.27)
o
i= 1
fcz1
By implication, the two sides of this equation are either both infinite, or finite and equal. This has a particular application to results involving c-finite measures. Suppose we wish to evaluate an integral gdkkusing a method that works for finite measures. To extend to the c-finite case, choose a countable partition igi, lffl of f1, such that g(fk) < x for each Letting gi loy, note that g iidp by gdp (4.27). and =
.
=
=
4.3 Product Measure and Multiple lntegrals Let (f,@,g) and (E,V,v) be measure spaces. ln general, tfx E, 5 @ N, 1) might also be a measure space, with 7t a measure o'n the sets of 5 @ V. In this case respectivlly, are measures g andv, defined by g,(#) 7:(Fx E) andvtG) a(f1 x G4 called the marginal measures conrsponding to T:. Alternatively, suppose that g and v are given, and define the set function 1: Sys F-> 2-+, =
=
where Sss denotes the measurable rectangles 7:(Fx G)
=
g,(&v(G).
of the space f x E, by
(4.28)
We will show that 1: is a measure on Syg, called the product measure. >nd has an extension to 5 * T, so that tfx E, 5 (8) V, a) is indeed a measure space. The tirst
Integration
65
step in this demonstration is to define the mapping F(o: E F-> x E by Fo(() (,(), so that, for G e V, F(o(G) f )) x G. For E e 5 @ V, let =
=
Eo
F;1(F)
=
((: (,()
=
e F)
c
E.
(4.29)
The set Eoj can be thought of as the cross-section through E at the element ), any countable collection of 5 @ V-sets (Ej, j e EN
U
((:(t,),t)
=
j
For future reference,
U)
e
Ul(:(,()
=
e
./
j
EjI
=
U().
the following.
note
e 5 (& V. If F=Fx
Proof We must show thate'to e N wheneverf G e N, it is obvious that Eo, =
c(Sps),
(4.30)
j
4.15 Lemma Toj is a W5 @ Vl-measurable mapping for each
Since 5 (&N
For
(t.
G,
E
F
e
F
(l)
e f1.
G for Fq 5 and
e N.
=
3,
(l)
(4.31)
the lemma follows by 3.22. w
The second step is to show the following.
4.16 Theorem
';:
on Sps.
is a measure
0, recalling that Fx O O x G O Proof Clearly JE is non-negative, and <0) for any F e 5 or G e V, and applying (4.28).It remains to show countable additivity. Let fE. e Sps, j e (N) be a disjoint collection, such that there exist sets Fj e 5 and Gj E V with Ej F) x Gj; and also suppose E VjEj e Spg, such that there exist sets F and G with E Fx G. Any point (,() e Fx G belongs to one and only one of the sets Fj x Gj, so that for any (t) e F, the sets of the subcollection fGjI for which ) e Fj. must constitute a partition of G. Hence, applying (4.30)and (4.31), =
=
=
=
=
=
vE.)
=
v
U j
U 3, j
U()(o j
Gj,
=v
v
=
)
e Fj =
(t)
e Fj
Xj 1,())v(%),
(4.32)
where the additivity of v can be applied since the sets Gj appearing in this dcomposition are disjoint. Since we can also write vtfo) v(G)1A)), we find =
l(F) .f....j....L..r
. ... . .
....
.
:
.
..
.
). .
....
.
=
g,tFlvtGl
=
JY(FV>())
=
J
X 1z)(t0)V(f) 1
tf/t*l
66
Mathentatics
= as required,
Xp.(F?)v(t7;)
Xz:(e;.) j
=
j
(4.33)
the penultimate equality is by 4.14. w
where
It is now straightforward
4.17 Theorem ( x E, 5
to extend the measure from R,g to 5
e N, J:)
(& V.
is a measure space. hence Svs, is a semi-ring by
5 and V are c-fields and hence semi-rings; 3.19. The theorem follows from 4.16 and 3.8. . Proof
Iterating the preceding arguments (i.e. letting (,T) and/or (E,V) be product spaces) allows the concept to be extended to products of higher order. In later chapters, product probability measures will embody the intuitive notion of statistical independence, although this is by no means the only application we shall meet. The following case has a familiar geometrical intemretation. Lebesgue measure in the plane, R2 R x R is detined for intervals
4.18Example by
=
-((Jl,:11
x
(J2,&21)
=
b1
-
J1)(:2
,
-
(4.34)
J2).
Here the measurable rectangles include the actual geometrical rectangles (products of intervals), and o the Borel sets of the plane, is generated from these as a 2 is a measure space in consequence of 3.20. By the foregoing reasoning, (R2,B which the measure of a set is given by its area. n 0A,
,m4
We now construct integrals of functions /*((t),() on the product space. The following lemma is a natural extension of 4.15, for it considers what we might think of yielding a function with as a cross-section through the mapping at a point (J) e domain E. ,
4.19 Lemma Let f:D x E F-> R be 5 @)Ws-measurable. Define Then f(0:E F- R is V/o-measurable. fixed ) e
.f(o(j)
=
A,tlfor
.
Proof We can write
f(()
=
f(eD,()
=
f(Fo(())
=
/oF(().
By 4.15 and the remarks following 3.22, the composite function measurable.
(4.35) is MIB-
fo
K
Suppose we are able to integrate f with respect io v over E. There are two quesJE/'IWV tions of interest that arise here. First, is the resulting function gl F/plmeasurable? And second, if g is now integrated over f1, what is the relationship between this integral and the integral Inxzfd:Over ( x E? The affirmative answer to the first of these questions, and the fact that the integral where these exist, are fiterated' integral is identical with the the most important results for product spaces, known jointly as the Fubini theorem. Since iterated integration is an operation we tend to take for g'ranted xxrltla mllltlnlo piemnnn nerhnns the main Doint needin: to be stressed infnyrnlq =
tdouble'
Integration
67
here is that this convenient property of product measures
(and multivariate
Lebesgue measure in particular) does not generalize to arbitrary measures on product spaces. The first step is to let f be the indicator of a set E e 5 (& V. ln this case f is the indicator of the set Efo defined in (4.29),and
J/'olv
v(F(o)
=
+(e)),
=
(4.36)
2-+is well-defined, say. ln view of 4.15, Efo e V and the function #s: fl values unless in the extended although, v is a finite measure, it may take its shown. line, half as .-.:
(& V,
4.20 Lemma Let jt and v be c-finite. For all E e 5 and Jo#FdB TCZ')'
g; is T/p-measurable
(4.37)
=
By implication, the two sides of the equality in or finite and equal.
(4.37)are
either both infinite,
Proof Assume first that the measures are finite. The theorem is proved for this theorem. Let denote the collection of sets E such that gz case using the satisfies (4.37).Spv i ,W,since if E Fx G then, by (4.31), .4
';:-
=
+() and gzdbk fl x E e 1sz- lsj,
=
(4.38)
Clearly g(#)v(G) l(F) as required. We now show is a = and since holds. lsz-sj 1.25(a) lf F1 c Ez, then, Ej,E2 e so -system.
=
.4
Js1/,()JV(()
gE,-EL4
an 5lT measurable
Js1e'1(t0,()#V(()
=
=
-
'sztt,)l
(4.39)
-%(),
function by 3.25, and so, by additivity of
'n,
.
jgjgh-zbdbk
.'
Y,
F e
.4
.4,
' r' 1*
v(G)1F(),
=
l(F2)
=
-
r(F1)
=
zE2
-
F1),
.VL.
t
(4.k0)
''
showing that .Wsatisfies 1.25(b). Finally, If 41 and A2 are disjoint so are (A1)(o + gy). and (A2)(o,and ujwxct)) u1((l)) To establish 1.25(c), let (. e W, with be IN Ej j e J a monotone sequence, F. Define the disjoint collection (Aj) = With A1 q= E$ and Aj Epj A),j > 1, so that E U7=3Ajand Aj e Wby (4.39). additlvity of v, y ountable
(
=
't
=
-
x ','(l
=
Xuytl.
yj -.t. .E )r-. )..' L.k,.j-L);(b. ..r.),).).t - ; )--'.ti't(.. (yyyE (;()t. .(.yy')-....qy . r. tf''Ett..t.q'? ;t-yjjy:jj(j .yjjjy.;gjjyjjjjjt.. .y yy. q jtyjjj yyyj,;( jqqj)q;((. .!k jyy .ryLLI. gjyj)j. jjrjjk yy. ... jj;gjr j.yy.j.jjjyrr jjygj. yjjjyj $kj)4 yryt;y.)...-i.r..t.E)). .E 7..y jj ... y..q-().r-(.t(. ...-.-!;y..:. ,.-. .. gyjj,. . r) r.q.r t. #t-.-y 1l(!(l:l.)(.. ..((r-. )-)t.. . t.-... .. .. ' ;.1 . . . . . . ... . . .. . . . ((
')'
..t.-)
-'.t.t
-..(q..)--r;:,)j);.tr-ry
.
..
..q
..
y..jt..j. . ;fj,jL)qq;... )) jtjj(j)q..... ..y,r(.jk
-.-
.li.
:
--''
-
- - .
-..)-.-.))t:..,.
-..
.
...
.
tj!jj jj.(gjrj
(4.41)
68
Mathematics x
x
Jo
X'At)
tfBtt0l
g
Xj j=1
=
j=j
v
x
o'Atttltfp,tl
X5(X/)
=
=
1(13,
(4.42)
j=j
n
.d
Since Sys is a where the first equality is by 4.14. This shows that is semi-ring it is also a a-system, and 5 @ N c(Sys) c by 1.27. This completes the proof for finite measures. To extend to the c-finite case, 1et J and (Ey) be countable partitions of f tff and E with finite g-measure and v-measure respectively', then the collection tff x Ej e Sys ) forms a countable pmition of x E having finite measures, attf x Ej) = g.tfvtWl.For a set E l 5 @V, write Eq = E r7 tf x Ej). Then by the last argument, -system.
.4
=
ghdbk
=
where gzql fk disjoint and
F-> 's((l))
nEq),
(4.43)
R+ is defined by +,y((l)) v((F(j)(J, (t) e fk. The sets Eq are vjLjjl whep (l) e ff, or =
=
gE
=
X 1f4()X+/).
(4.44)
j
i
The sum on the right need not converge, and in that case g/ F/s-measurability holds by 3.25/3.26, and
jcysdb -
-
Jo
Xjl XyS's
XXJO:S/B i
j
*'
-
'*'l
+x.
=
However,
Jp, XXn(A) i
-
(4.45)
nk,
j
using 4.14 and countable additivity. This completes the proof. Now extend from indicator functions to non-hegative
.
functions:
4.21 Tonelli's theorem Let 7: be a product measure with c-finite marginal measures g and v, and 1et fl x E F-> R+ be (@(&Vl/f-measurable. Detine functions .f:
f1
= R+ f(,(), is T/f-measurable, (i) g JsfA
E
F->
(ii)
b-V(()
jtstcafdx Jo =
and let gk
=
hfuA.
Then
tfB. E1
ln part (ii) it is again understood that the two sides of the equation are either finite and equal, or both infinite. Like the other results of this section, the and (E,N,v), and the complementary results given theorem is symmetric in (f1,@,g) roles interchanging of the marginal spaces do not require a sepapte stateby the ment. The theorem holds even for measures that are not c-tinite, but thij further complicates the proof. Proof This is on the lines of 4.6. For a partition (F1,...,Fn J of 5 @)T let f Xwlsj, and then f Xii ltsjl(sand g = Xiqis'lEil by 4.4. g is F/p-measurable
=
=
Integration
69
by 3.25, and 4.20 gives vedb gdb Xm fl i
Xaine'i)
=
=
fl
=
j
AE
j'dn,
(4.46)
so that the theorem holds for simple functions. For general non-negative f, choose a monotone sequence of simple functions converging to f as in 3.28, show measurability of g in the limit using 3.26, and apply the monotone convergence theorem. w Extending to general
f
requires the additional
assumption of integrability.
4.22 Fubini's theorem Let a be a product measure with c-finite measuresg, and v; 1et #: fl x E >-> R be (; @ Vl/f-measurable with
marginal
Jzxsl.ftt,)'tllllt,ll <
(4.47)
=;
Then and let g define f)1 E >-> R by Jsffzyfv. (i) f is N/f-measurable and integrable for (z) e A f2, with g,tfl A) (ii) g is F/f-measurable, and integrable on A; .f4,();
.406)
=
=
=
-
JsTto'cldvt'l joxsft,clflttt,'l Jo
(iii)
0',
dpt*'
=
Proof Apart from the integrability, 4.19 shows (i) and Tonelli's Theorem shows (ii) and (iii) for the functions f' = maxl,o) and f- = f'b f, where I f+ + f-. But under (4.47),I I < on a set of a-measure 1. With A defined (i), (ii) and (iii) hold for f'b and as the projection of this set onto finite of the equation in sides with both (iii). Since f f'b f (i) extends to f by 3.25, and (ii) and (iii) extend to f by 4.7. . .41
=
-
.f(),()
'x
-,
,
-,
=
-
4.4 The Radon-Nikodym Theorem Consider c-finite measures g. and v on a measurable space (f2,1).g, is said to be absolutely continuous with respect to v if v(A) 0, for E e 5, implies g(A-) 0. This relationship is written as g <( v. If g tt v and v <( go the measures are said to be equivalent. If there exists a partition (A,A ) of f1, such that g(A) 0 and v(AC) 0, then g and v are said to be mutually singular, written g, v. Mutual singularity is symmetric, such that p. v means the same as v g,. The following result defines the Lebesgue decomposition of g, with respect to v. =
=
c
=
.1.
=
.1.
.
4.23 Theorem If p, and v are c-tinite measures, there exist measures g,1and p.2 such that g = gl + ga, g,l v, and p.2
lf there is a function #: fl 1..-> F+ it follows fairly directly (chooseE such that v(F) 0) such that g(F) Js/tfv, that g, << v, and f might be thought of as the derivative of one measure with =
=
70
Mathematics
respect to the other; we could even write f dbjldv. The result that absolute continuity of g, with respect to v implies the existence of such a function is the =
theorem.
Radon-Nikodym
4.24 Radon-Nikodym theorem Let v and g,2be c-finite measures and 1et p,2t< v. There exists a T/p-measupble function fl f F-y 2-+such that g2(F) fdv for all E e 5. u =
f is called the Radon-Nikodym derivative of g, with respect to v. If g is another for all E e 5, then v(.f #: g) 0, otherwise at such function and g,2(A-) Jsgtfv and E1 (: least one of the sets Ej (): ()< gl ) must contrathe detinition. dict =
=
.j'((l))1
.f4)>
=
=
Proof of these results requires the concept of a signed measure.
4.25 Definition A signed measure on (f,F) is a set function z: 5 F-> R-
satisfying 0. (a) :(0) j) (b) z(U#. XgAj) (c) Either z < x or =
=
disjoint collection (Ay e 5
for any countable, u z>
For example, let g, and v be non-negative measures on a space least one of them finite. For a non-negative constant r, define Z(X)
for any
)
.
-x.
=
((, @), with
p,(A) rvtAl
at
(4.48)
-
A e 5. For disjoint (Ay)
,
zj+l (-s) (x/) -
j=3
p,
-
y=1
= Xtg,tll
rv
/=1
-
rvtAjll,
(4.49)
i=1
so that countable additivity holds. If A is a F-set with the property that # e 5 with B c A, A is z(#)k 0 for every complementary negative defined in the called apositive set, a set being manner. A null negative both is positive and is called careful Be that set. to distinset a guish between positive (negative,null) sets, and sets of positive measure (negative measure, measure zero). A set has measure zero if g,(A) rvtz4l in (4.48), but it is not a null set. By the definition, any subset of a positive set is positive. The following theorem defines the Hahn decomposition. :4
=
4.26 Theorem Letx be a signed measure on a measurable space (f,1), having the property x(A) < cxo for a1l 4 e 5. There exists a partition of f into a positive set
A+ and a negative
set A-
Proof Let sup x(A), where the supremum is taken over the positive sets of x. A+ Choose a sequence of positive sets (Anl such that lima-ox z(Aa) o and 1et z4+ A+. measurable consider E Undn.To show that is also a positive set, any Iaetting B, A1 and B- A- -A--1. n > 1. the seuuence $Bnl is disioint, positive ,
=
=
=
=
=
Integration
71
since Sa c A,i for each n, and UnSn = A+. Likewise, if En
Ecj Bn the sequence fEn ) is disjoint, positive since En c #n, and Unk E. Hence zlE') XnzEn) 0, and since E was arbitrary, A+ is shown to be positive. A+ -An being therefore =
=
=
positive, Z(X+)
Z(X2
=
Z(Z+ -A
+
k Z(Xu), a1l n,
(4.50)
z(A+) k 1, implying z(A+) 1. Now let A- f -A+. We show, by contradiction, that A- has no subset E with positive measure. Suppose there exists E c A- with zE4 > 0. By construction E and A+ are disjoint. Every subset of A+kJ E is the disjoint union of a subset of A+ with a subset of F, so if E is a positive set, so is A+ t? E. By definition of
and hence
=
=
X,
,
2
b-)E4 z(z4+
:+z(A),
=
(4.51)
which requires x(A>) 0, so E cannot be a positive set. If F is a subset of F, it is also a subset of A-, and if positive it must have zero measure, by the argument just applied to E. The desired contradiction is obtained by showing that if zE) > 0, E must have a subset F which is both positive and has positive measure. The technique is to successively remove subsets of negative measure from E until what is left has to be a positive set, and then to show that this remainder has positive measure. Let ?z1 be the smallest integer such that there is a subset F1 c and detine F1 E F1. Then let nl be the smallest integer E with z(F1) < In general, for k = 2,3,..., such that there exists E1 i F1 with z(F2) < 1et nk be the smallest positive integer such that Fk-j has a subset Ek satisfying Z(f) < -$Ink, and let =
-1/n1,
=
-
-1/n2.
Fk
E
=
-
U). j=
(4.52)
lf no such set exists for finite nk, let nk +x and Ek = 0. The sequence (Fk) is x. non-increasing. and so must converge to a limit F as k =
---
We may therefore write E Fb..l (U7=1A), where the sets on the right-hand are mutually disjoint, and hence, by countable additivity, =
oo
z(f)
:4#')
=
+
side
Cr
Xz(A)
<
k=1
llnk. zb7 X k=1
(4.53)
-
Since x(F) > 0 it must be the case that x(#)> 0, but since zF < oo by assumptoo as k x. < x, and hence nk This ion, it is also the case that Z7=1(1/ak) with and negative is therefore a positive measure, means that F contains no subset . having positive measure. set -->
--
For any set B e 5, define Z(#) z(A+ rn B4 and z-B) -z(A- fa #), such that x+(S) z-B). It is easy to verify that zB4 z+and x- are mutually singular, non-negative measures on (f1,F). f z- is called the Jordan decomposition of x a s ig ned measure. z+and z- are called the upper variation and lower variation of =
=
=
-
=
a,
wxvxa
+1-.zk
maooxa...x
l .,
I
l,+
-
..u
%,-
lo
-
zxollxcxzl
tlao
tntnl
a/mrizyimn
nf
q;
Tho
Tnrdnn
Mathematics
72
decomposition shows that all signed measures can be represented in the form of (4.48). Signed measures therefore introduce no new technical difticulties. We can integrate with respect to z by taking the difference of the integrals with respect V to x an (j y We are now able to prove the Radon-Nikodym theorem. It is actually most convenient to derive the Lebesgue decomposition (4.23)in such a way that the Radon-Nikodym theorem emerges as a fairly trivial corollary. It is also easiest to begin with tinite measures, and then extend the results to the c-tinite case. -
.
4.27 Theorem Finite, non-negative measures v and g have a Lebesgue decomposition g /1,1+ g2 where gl v and p,2 << v, and there exists an F-measurable R+ F-+ f such that g,2(F) function /': fdv for a11 E e 5. .1.
=
=
Proof 6 L e t G denote the class of al1 F/f-measurable functions gl f F-> R+ for which IEgS S p,('), a11 E e @. 6 is not empty since 0 is a member. Let (x sup#esf'tfv, so that (x f g,tfM< x. We show there is an element of G at which the supremum is attained. Either this element exists, and there is nothing further to show, or it is possible by detinition of a to choose an element gn of G satisfying f a, for each n G IN.Generate a monotone sequence (/k,n e IN) in G tx 1/a f Igndv follows. Put so that fn 2 max(A-1((z)), J1 :1 and define fn by fn as > %()) for n 2,3,..., and then if fn-l A-l. Define the sets Aa (: e G, =
-
,w()),
=
=
.fn-1((l))
=
Xtfv
=
fn-tdv +
=
F
Erun
Fcxq
gndv
g,(FrnAn) + p.tf f'7Ak)
=
g,(f)
(4.54)
so that fn e G. Since fn 1 f, it follows by the monotone convergence theorem that IEf* S h1,(0, and hence f e G. And since fn gn so that fndv --4
a we must conclude that
-
Lln S
J.ftfv a, =
g,2tD
=
gndv S
fndv S a, Now detine g,2 by
as was required. dv, E
C
F'
(4.55)
Evidently g,2 is a non-negative measure (for countable additivity consider the and use 4.14), and also g,2 q v. Define g,1(e')= g,(f) 41,24f), functions which is non-negative by construction of f, and also a measure. It remains to show .
that p.j
=
IV
-
,
.1.
v.
Let (A+n,A-a)be a Hahn decomposition for the measure 1,2,3,.... Then for E e T, g,(Fr7A+) n
=
p.1(Ff-74)
+
-1v(l
>n
ra x+n) +
g.l v/n, for n -
=
g2(F raA+sl
j
sck.:
y:v
=
j
srn,j
-)
.f
+
w,
(4.56)
Integration
73
and hence p.(A')
p,(Ff'3A1) +p.(FrnAa-) 2 p.(Ff7A1)
=
k
J
El'x<
f
1 tv
+
n
Note from this inequality that
(x k
Jf
+
v
+
fdv J Fzn-a
J + n-11Aj -s
1v(A+n) =
=
+
p.2(Fr-7Aa-)
fdv + JE
-)v(F
r-'hA).
(4.57)
e 6 so that ,
-1v(4+n),
a
+
(4.58)
u
implying vAhjlln 0. This holds for each n G N, so if A U';=1A, v(A) 0. Note g.1(AC) v(AC)/l An-for every n, and so for every n. Hence ; that Ac 0* N= 1Aa-i g,1(Af) 0, and-so jt1 v. w =
=
=
=
.1.
=
It remains
to extend this result to the c-finite case.
such Proof of 4.23 By c-tiniteness there exists a countable partition f l of collection with finite for finite each j. If (Ay) is any that vtf#) and p,tf#l are union and whose is letting f11 Lh -Ay-1 for A1 1 > Aj defines j measures such a partition. If different collections with tinite measures are known for v and p,, say lAjzjl and fAvyl the collection containing all the Ax rnAvk for j,k e N, is countable and of finite measure with respect to both v and g,, and after reindexing this collection can generate (f). Consider the restrictions of g, and v to the measurable spaces (Lh,%4, for j e IN,where % (F t''h Lh, E e 5 J By countable additivity, g,(F) XjttA' rn L%lwith similar equalities for g,l, g,z, and v; by implication, the two sides are in each case either finite and equal, or both +x. If v(F rn Lhl 0 implies g,2(F chLhl 0 for each j, then v(F) 0 implies g2(F) 0 for E e 9, and g,2< v. Similarly, let Aj, AJ) define partitions of the Lh such that g,1(Aj) v(A,) 0', then A U#. j, v(AC) = Qv(A,) = Ac unions, 0. p,1(A) Qg4(AJ 0, and and Ugdlare disjoint Hence g,1 v. K ,
=
,
=
,
=
=
.
=
=
=
=
=
=
=
=
=
=
.
The proof of the Radon-Nikodym theorem is now achieved by extending the other conclusion of 4.27 to the c-finite case. Proof of 4.24 In the countable partition of the last proof, 4.27 implies the such that non-negative X/f-measurable
existence of
g,(F rn Lh)
=
p,ltf
f''h
L1jl + g,2(F r-hLhl
where P,2(A't''h f#) Define
J:
F-
Jsyoo *,
=
R+ by .fttl)l
=
al1 E e
X lf/ttl'tt9l.
5.
(4.59) (4.60) -1
This is a function since the Lh are disjoint, and is g/s-measurable since f B) y-1(#) LtEj e 5 where Ej e 5j, for each # e S. Apply 4.14 to give = Uy/' =
Mathematics
74 :2(A3
=
Xp,ztFr7
Lhl
=
j
Js
1tzjfldv. - X
Xj
Js
Jsljdv
fdv
-
.
(4.6 1)
*
Consider the case where g, is absolutely continuous with respect to another gq, v(A) measure v. If the Lebesgue decomposition with respect to v is g, g,l + g,1(AC) since 0 implies 0 which in implies g,l(A) 0. But g,(A) 0 turn g,l v, too, Thus, g,1(f2) 0 and g, ga. The absolute continuity of a measure implies the existence of a Radon-Nikodym derivative f as an equivalent representation of the fOr measure, given v, in the sense that g,(F) Js/'tfv any E e @. An important application of these results is to measures on the line. =
=
=
=
=
.
=
=
=
4.28 Example Let v in the last result be Lebesgue measure, m, and 1et g, be any other measure on the line. Clearly, g,1 m requires that g,1(A'') 0 except when E is of Lebesgue measure 0. On the other hand, absolute continuity of g,2 with length', any countable collection of respect to m implies that any set of isolated points for example, must have zero measure under g2. If g, is absolutely continuous with respect to m, we may write the integral of a measurable function g as =
.
Ezero
j
+x
'(AVP'@)
=
-X
p+x
1 -
gxtfxtdx'
-X
(4.62)
so that all integrals reduce to Lebesgue integrals. Here, f is known as the density functionof the measure g, and is an equivalent representation of g,, with the relation g,(f)
=
jefxtdx
(the Lebesgue integral of f over E4 holding for each E e f. n
(4.63)
5 Metric Spaces
5. 1 Distances and Metrics Central to the properties of R studied in Chapter 2 was the concept of distance. For any real numbers x and y, the Euclidean distance between them is the number Jstx,y) Ix y I e R+. Generalizing this idea, a set (otherwisearbitrary) having a distance measure, or metric, defined for each pair of elements is called a metric space. Let S denote such a set. =
-
A metric is a mapping J: S xs F-> R+ having the properties dx,y), 0 iff if y, 2 dx, dy, (triangleinequality). metl'ic (S,#) is S A space a set paired with metric #, such that conditions (a)-(c) hold for each pair of elements of S. u
5.1 Dqfinition (a) dy.x) (b) dx,yt (c) dx,yt +
=
.'t7
=
=
0, but dx,y) 0 is possible when x # y, we If 5.1(a) and (c) hold, and dxtx) would call d a pseudo-metric. A fundamental fact is that if (A,#) is a metric space and # c A, (#,#) is also a metric space. If (E1is the set of rational numbers, Q c R and (Q,#s) is a metric space', another example is (g0,11,#s). While the Euclidean metric on R is the familiar case, and the proof that dz satisfies 5.1(a)-(c) is elementary, dz is not the only possible metric on R =
=
.
5.2 Example For x,y
E
R 1et
4@,y)
=
!
Ix +
yI
-
jx
..y
j
(5.1)
.
It is immediate that 5.1(a) :nd (b) hold. To show (c), note that Ix l doxtyjlk dx,yj). The inequality tz/(1 a) + /7/(1 b4 2 c/(1 c) simplifies c). We obtain 5.1(c) on putting a doxty), b doy,z), to a + b 2 c + abl c 4)@,z),and using the fact that 0 f do S 1. Unlike the Euclidean metric, Jo is defined for or y = ix. (V,4) is a metric spae on the definition, while Rwith the Euclidean metric is not. n -y
-
-
-
=
-
=
-
=
=
.x
lh the space R2 a 1arger va riety of metrics is found. 5.3 Example The Euclidean distance on R2 is -y1)2
defx-ytIIx-yII E(m =
d (q2r 4) p,lp ,
.
''
,
.
=
-y,)211'2
+(.n
(5.2)
'
.
is the Eglidean '
'
'
plane. An alternative is the
Etaxicab'
metric,
76
Mathematics
dvx,y)
=
Ixl
-
yl I +
1A'2
-
y2 1
(5.3)
.
dz is the shortest distance between two addresses in Manhattan as the crow flies, but dv is the shortest distance by taxi (see Fig. 5. 1). The reader will note that dv and Js are actually the cases for p 1 and p 2 of a sequence of metrics on R 2 He/she is invited to supply the definition for the case p = 3, and so for any cxo is the maximum metric, p. The limiting case as p =
=
.
-..-h
JA,g,y)
=
maxl
-y2l
1.x1yl I 1x2 -
,
1.
(5.4) R'T
Al1 these distance measures can be shown to satisfy 5.1(a)-(c). Letting R x ERx x R for any finite n, they can be generalized in the obvious fasltions (RR,#r), (R'l,#x) and so forth. n to define metric spaces (R&,Js), =
...
t/yg (-,:,)2 )
------------
t2-(-);,)7
)
Fig. 5.1 Metrics dk and dz on a space S are said to be equivalent > 0, there is a > 0 such that
dblx,yq < 8
=
dlx,yq < E
dzlxty) < 8
=
[email protected]') < E
if, for each
-v
e S and :
for each y e S. The idea here is that the two metrics confer essentially tlte same properties on the space, apart from a possible relabelling of points and axes. A metric that is a continuous, increasing function of another metric is equivalent to it; thus, if d is any metric on S, it is equivalent to the bounded metric dll + d4. Js and do of 5.2 are equivalent in R, as are are Js and du in R2 o n the other hand, consider for any S the discrete metric Jo, where for x,y e S, dE dnxby) 0 if are not y, and 1 otherwise. do is a metric, but Jo and equivalent in R ln metric space theory, the properties of R outlined in j2.1 are revealed as a special case. Many definitions are the same, word for word, although other concepts are novel. In a metric space (S,#) the concept of an open neighbourhood in R generalizes to the sphere or ball, a set Sdxn (y: y e S, J(x,y) < :) where x e S and : > 0. We write simply Sx,z) when the context makes clear which .
=
.x
=
.
=
,
Metric Spaces
77
metric is being adopted. ln (R2 d E ) S@ :) is a circle with centre at x and it is a radius E. In (R2 (rotatedsquare) centred on x with e the distance from x to the vertices. In (R2 du ) it is a regular square centred on x, with sides of 2e. For (R3 d s) we 11 think about it! An open set of (S,#) is a set A S such that, for each x e A, (; > 0 such that 5'@,) is a subset of 4. If metrics dj and dz are equivalent, a set is open in (S,#1) iff it is open in (S,J2). The theory of open sets of R generalizes straightforwardly. For example, the Borel field of S is a well-defined notion, the smallest c-field containing the open sets of (S,J). Here is the general version of 2.4. ,
,
,
tdiamond'
,#w)
,
,
,
...
5.4 Theorem (i) If C is any collection of open sets of (S,#), then
c
UA
=
Ae N
is open.
(ii) lf A and B are open in (S,#), then A ch# is open. Sx, c A and A e C, then S.z,z) c C. Since such a ball exists by a11 definition for x E A, a11 A e C, it follows that one exists for all x e C. (ii) If 5'(x,/) and S(x,k) are two spheres centred on x, then
Proof
(i) If
5'@,6k)f''h 5'@,Es)
(5.8)
5'@,E),
=
If x e A, H 5'(x,/) c A with Ex > 0, and if e #, (5S(x,ze4 E > 0. If x e A chB, 5'@,E) I A rn B, with 6: > 0. x
where e minlerx,Es) c B similarly, with
.x
=
.
The important thing to bear in mind is that openness is not preserved under
arbitrary intersections. A closure point of a set A is a point x e S (not necessarily belonging to A) The set of closure points of A, such that for all 6 > 0 ('!y e A with #(x,y) < . of Closure called the closure A. points is denoted A, are also called adherent points, sticking to' a set though nt necessarily belonging to it. If for some 8 x, so that > 0 the definition of a closure point is satisfied only for y &x,) f'-AA (xJ, x is said to be an isolated point of A. A boundary point of A is a point x e X, such that for all > 0 H z e Ac with The set of boundary points of A is denoted :A, and W A Q.J :4. The J(x,z) < interior of A is A0 A PA. A losed set is one containing al1 its closure Points, such that X A. An Qpen st dyjes not contain a1l of its closure points, since the boundary points do not belng to the set. The empty set 3 and the space S are both open and closed. A Subset pf A is said to be dense in A if B c A c X. A collection of sets T' is calld a. bvering for A if 4 Usssf. If each B is =
=
=
.
=
-
=
.
.
...:.;
.
''
.
'
.
.
;
'
.,>
covtink. A jt A is called compact if every open open, it is called an open kqtog. x is said to be relatively compact if covering of A contains a tinite S itsel? 'A,t,tk'.$,j,'..,k4) is said to be a compact space. The cott, is compact. If s is rk.,i tl compaltlktlyj, ,. ttk y))(y,y;(jty)):Vyty,yjkj4g eqqally relevant to the general case. remarks in j2. 1 about jkyyjytyyyjjy yjj yyy, jj 4::11. d!;p dr:;li 4k220 154!r 11::). tjyod jyyyyyktty.jtyz,y,g,jyjyyyyyjxyyyjyjyy,,yyyyyyytyy ii.1b;1b;Citii. 1l:4(E:1. rr.t).:y...;....,. ..,.4!k. ;yjjy .yj < x, sucj.j tjaatx s sxtrj; and also .
x
. .
.p,
,1.)
.))
'
t
.
rjg)
....r
.
.F ..--
.
Mathematics
78
totally bounded (or precompact) if for every e: > O there exists a finite collection of points x1,...,xm (calledan E-net) such that the spheres Sxz), i z4. The Sxi, can be replaced in this definition by = 1,...,- form a covering for noting that Sxi,z) is contained in Sxi, : + ) for all 4'5 their closures Sxinz), z4. > 0. The points of the E-net need not be elements of An attractive mental image is a region of R 2 covered w ith little cocktail umbrellas of radius : (Fig. 5.2). Any set that is totally bounded is also bounded. In certain cases such as ?n,dE) the converse is also true, but this is not true in general. . .k . .
.
.
... . . .
.
.
.
.
4
.
.
.
.
,
.
.
.
k...
.
.
.
.
.
.
.
.k
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
y .
.
.
.
.
'
j
.
.
.
.
.
.
.
.
.
.
.
' .
.'.
'
.. .
.
.
.
. .. '
.
.
.
.
. .
.
.
. '.
...
.
.
...
.
i
.
.
.
.
.
....
.
.
'.'k. .
.
. .
.
'.
.
.
.i; ..
.
...'
. .
.
...
.
.
4 .
.
...
. .
...'.
kj'..'
Fig. 5.2
5.5 Theorem lf a set is relatively
compact, it is totally bounded.
Proof Let A be relatively compact, and eonsider the covering of X'consisting of the e-balls Sxnz) for a1l x e X'.By the detinition this contains a finite subis an e-net for cover Sxitz), i = 1,...,-, which also covers A. Then (x1,...,au) d, and the theorem follows since e: is arbitrary. K The converse is true only when the space is complete', see 5.13.
5.2 Separability and Completeness In thinking about metric spaces, it is sometimes helpful to visualize the analogue R'J with n f 3, and use one's intuitive knowledge of problem for R, or at most for those cases. But this trick can be misleading if the space in question is too alien to geometrical intuition. A metric space is said to be separable if it contains a countable, dense subset. Separability is one of the properties that might be considered to characterize an 4R-like' space. The rational numbers (I) are countable and dense in R, so R is separable, as is R&. An aliernative definition of a separable metric space is a metric spaee for which the Lindelf property holds (see2-7). This result can be given in the following fonn.
5.6 Theorem In a metric space S the following three properties are equivalent: (a) S is separable. (b) Every open set A S has the representation
Metric Spaces
4
UBi, Bi e
=
79
'P,
izu1
where V is a countable collection of open spheres in %. Every (c) open cover of a set in S has a countable subcover.
I:a
A collection V with property (b) is called a base of S, so that separability is equated in this theorem with the existence of a countable base for the space. In topology this property is called second-countabilit.y (see j6.2). (c) is the Lindelf propely. Proof We first show that (a) implies (b). Let V be the countable collection of spheres (5'(1,r):a7e D, r e t1+), where D is a countable, dense subset of S, and Q+ is the set of positive rationals. If A is an open subset of S, then for each x e A, 3 > 0 such that 5'(x,) g A. For any such x, choose xi e D such that dxie < /2 (possiblesince D is dense) and then choose rational ri to satisfy dxip < ri < &2. Define Bi Sxitr e V, and observe that =
S(A7,8)
Bi I
(5.10)
i X.
Since V as a whole is countable, the subcollection fBiI of al1 the sets that satisfy this condition for at least one x e A is also countable, and clearly A c Uff I X, SO X = U#f. Next we show that (b) implies (c). Since V is countable we may index its elements as ( Jj, j E EN). If C is any collection of open sets covering A, cboose a subcollection ( Cp j e LNJ where Cj is a set from C which contains 5. if such by F-sets, as just exists, otherwise let Cj 0. There exists a covering of shown, and each 5. can itself be covered by other elements of V with smaller radii, so that by taking small enough spheres we may always Gnd an element of C to contain them. Thus 4 c UjG,and the Lindelf property holds. Finally, to show that (c) implies (a),consider the open cover of S by the sets (5'(x,1/n), x e S l If there exists for each n a countable subcover (Sxnktllnj, k e ENJ, for ach k there must be one br more indices k' such that dxnkpxnk')< lln. Since this must be true for every n, the countable set fak, k e (N n e EN ) must be . completes ptoof. the This in S. dense ,
:4
=
.
,
The theorem has a useful corollary, lc'tg C' k' ;''j' y' q.rljljjrjtrj'jk. k:5j';;'jll.' j(j('(jy';;)(;(jjjr-:';E' 'i'. 11(F*))7(j)1))*5i'-!tt)@'.tjjj: rjd t;(.',' y' (jd y.'r' )' :'''jt'ty'k). (y'r;. )' jy.yjy-'tyyd yj'(' (' )jjj'Lqjj.j;Lj))(L:'.. ;';)' j;'16'. illj.Fklfll'r!id j()$.tj!y' !l-i@;yy'. jj.'lq. j'td .'.'k(t(jjj.'. --' ,.' -' .'.,' ,'.r:,ggjj4;4* ..t-d '-7*
'''jd 7* ::.tky,(y)jjj4jj44.2* yjyyjj...d ..-tj).'.' ,.t,' jy'. E'. 5* k(j)jt)':. ;'-'kE''. ,':'.#y' ?'
)'jt. jyd (' jjryjjgj;rrjjj'. yy'' jjjjj..d jjj')j(jk)'. r)'k(!@'' )' h' tl'tjjg t'jkd
''. .' 57 @
Corollary A totally bounded space is separable. n .
Another important property is that gbspaces which we show as follows. y
sR
Tuoorem
If 4:.* is a sp' ..
.
.
..
of separable
spaces are separable.
ab ll,!.'pae and A c S, then A,d) is separable. l l '
.
.'
;
'.;
. .y
y.
:
5:.'
:
')'q q.ty ;..' j,.);.j.;Lj. jy yk y. ..) .. a,,#..z'inl.s.
....y.((
. ..
.'
)
;y..
:
.'
.
g
g
...
.
. d'tll(td. Constnlct the countable set E by ) qp;jt,. jytgr yy a, ( '' supposeD is countbl g.;..j y (. y .). . .... y .. ... .. . . yjtliltjt, . q F tnul iyl!q t ' 'y takingone point from each;.$.',tj,.A,, ' (qi r..i q. y ( .( .. jy- jy.j(. ..jjtk(,... rk ;,-j,bj,.,. ...;,;-j-tyjj-yyy.trr,pjrrj--t:. jq j rk.,yjj ' r...'. .(. .,.-bj',::,;;;LL?3b.,L, y ..g.. .. ;f;;,,,;j:;-f);;,-'LL?33'-.,-j;)--Ljj;::-),'b--jj.)L,,t,;b'f?;));j-.
Ivoof
.
y
E
.
:
(
E
(E $2 .'
EE .
.
E ' ly . .
jE. .
?.
.
: gy. q
.ptllko'tyq)
E. q: . .
:
.E
.
,
.
.
l'.r
-i
y..y)..
E' . .. .
.' ' ::EE. . . . . E .-' (Ej.jijjjjjyryjgggggg, .. . . g: E . y y . y, g . . yy . . . y .jjyy... ..j... . y.-.... . ,!j;pjj42)$g,-.,,y,..(y.kpjr)(.y(y(,yyjjyjjyj;jy.-..,...-y.tyjky,
E
. E .. yj' .E. !. - q. . (-. .(j. ; (E r
.
.
.
.
.
-
. -:;..,-:3626,3.-.; -...633333:,31,'.. . . . .
.
j .....-.-,...yjj..... . .-,,.... .....
#.,..- ytlyy.jitr.. )r,y,-.yyjyjjtjj)y.,,. .--..!1k.. ijijLjlijjjiiljl-. . ... .. ... ..
);.ryj-rkji-. 1,:-::,..-,k. .... lttjj?, . . .;(;;yyit..ykrj.-.yy-j-kjtjrqy-yj. .,jjt;-.. ...y'ptj,;,.-p!h!!!qqj--..,.
...-....
,
( !jr r... ..... . . .. :..
: : : . ..
,
-k,yt.yytyyy-y,......--...-
.
,....y
-.
.
;k.
.
..
.
,.
. ,.
.
),
.
..
.
,
..
,;.j,:-.,,j;;)j.yty,r.t y .(.. - .. .... -
.
.
..
.
4::). jjjjj.s ,-
tjggjjj
Mathematics
80
For any x e A and > 0, we may choose y E D such that J(x,y) < 6/2. For every such y, R z e E satisfying z e A fn S@, for r < 6/2, so that d@,z) < &2. Thus
dxa)
f dx,y)
+
J(.y,z)
<
8,
(5.12)
and since x and 8 are arbitrary it follows that E is dense in A. w This argument does not rule out the possibility that A and D are disjoint. The separability of the irrational numbers, R Q, is a case in point. On the other hand, certain conditions are incompatible with separability. A subset A of a metric space (S,J) is discrete if for each x e A, R 6 > 0 such that (5'(x,6) (xJ)DA is empty. In other words, each element is an isolated point. The integers ; are a discrete set of (R,#s), for example. If S is itself discrete, the discrete metric do is equivalent to J. -
-
5.9 Theorem If a metric space contains an uncountable discrete subset, it is not separable. Proof This is immediate from 5.6. Let A be discrete, and consider the open set where Ex is chosen small enough that the specified spheres folnn a Uxez5'@,exl, disjoint collection. This is an open cover of A, and if A is uncountable it has no countable subcover. . The separability question arises when we come to define measures on metric spaces (see Chapter 26). Unless a space is separable, we cannot be sure that a1l of its Borel sets are measurable. The space Dgtz,,jdiscussed below (5.27)is an important example of this difficulty. The concepts of sequence, limit, subsequence, and cluster point all extend from R to general metric spaces. A sequence (x,,)of points in (S,J) is said to converge to a limit x if for al1 e > 0 there exists Ne k 1 such that
dxng)
< :
for a1l n
>
(5.13)
Ne.
Theorems 2.12 and 2.13 extend in an obvious way, as follows.
5.10 Theorem Every sequence on a compact subset of S has one or more cluster points. n 5.11 Theorem If a sequence on a compact subset of S has a unique cluster point, then it converges. Ia The notion of a Cauchy sequence also remains fundamental. A sequence (xnlof points in a metric space (S,tf) is a Cauchy sequence if for a11 e, > 0, 3 Nz such that dxnbxmj < e,whenever n > Nz and m > Nv.The novelty is that Cauchy sequences in a metric space do not always possess limits. It is possible that the point on which the sequence is converging lies outside the space. Consider the space (Q,#s). The sequence fxn), where xn 1 + 1/2 + 1/6 + + 1/n! e Q, is a Cauchy 0', but of course, xn e (the base of sequence since Iak+1 l = 1/4n + 1)! the natural logarithms), an irrational number. A metric space (S,#) is said to be complete if it contains the limits of all Cauchy sequences defined on it. (?,dE) =
...
-->
-xn
--
Metric Spaces
is a complete space, while
81
Q,dE4
is not. Although compactness is aprimitive notion whichdoes notrequirethe concept of a Cauchy sequence, we can nevertheless detine it, following the idea in 2.12, in terms of the properties of sequences. This is often convenient from a practical point of view.
5.13 Theorem The following statements about a metric space (S,(f) are equivalent: (a) S is compact. (b) Every sequence in S has a cluster point in S. (c) S is totally bounded and complete. Ia Notice the distinction between completeness and compactness. In a complete space all Cauchy sequences converge, which says nothing about the behaviour of nonCauchy sequences. But in a compact space, which is also totally bounded, all sequences contain Cauchy subsequences which converge in the space. Proof We show in turn that (a) implies (b),(b) implies (c), and (c) implies (a). Suppose S is compact. Let fxn,n e IN) be a sequence in S, and detine a decreasing sequence of subsets of S by Bn fxk:k k n,J The sets Tn are closed, and the cluster points of the sequence, if any, compose the set C O';=1X,; IICIC. U';=I#,C,, If C 3, S (U= so that the open sets T'nc are a cover for S, and by assumption these contain a t'initesubcover. This means that, for some m < oo, S (O'2=1))C This leads to the contradiction Tm 0, so that C must LKLc n S U'''ol) be nonempty. Hence, (a) implies (b). Now suppose that every sequence has a cluster point in S. Considering the case of Cauchy sequences, it is clear that the space is complete; it remains to show that it is totally bounded. Suppose not: then there must exist an : > 0 for which such that no e-net exists', in other words, no tinite n and points (x1,...,AkJ x k. But letting n in this case, we have found a dxjtx S : for all j sequence with no cluster point, which is again a contradiction. Hence, (b)implies =
.
=
=
=
n=
=
=
=
=
.
:g
-->
(C).
Finally, 1et C be an arbitrary open covr of S. We assume that C contains no finite subcover of S, and obtain a contradiction. Since S is totally bounded it must possss for each n k 1 a tinite cover of the fonu B ni
sxni, 1/2'')i
=
y'
jyjjjjjyyjjjy'
1
=
...,kn.
(5.14)
,
Fixing ?z, choose an 1 for whlch Bni hs' or finite cover by C-sets (at least one such exists by hypothesis) and all thls st Dn. For n > 1, jlBni))=n 1 is also a jl tjjt X h & X n-1 jja.s Sujpcgyeyy COVCf1ng jw p n-1 ap wg ggyj gjyjpjjj Xyr,(,.)!y;yy;jy fojyjj* njj tg F yyjj . kyry,j;. :,, jyzjojjsea sequence of points (xn e D IS nonemp tyi yj.g.(..)t;(( C-sets, and accordlngly n, y .(-(j,jj(. . )y.( . y.kj. s l tontalns ak, and Ds+1 1s of radi us ) since Dn 1s a ball Of radm# .,.. yy,,.tyjjj;,,,,gy; .yr;yy, jj-j jygyrgjs jkgyy jjygj.. . jtyjjy yggjj jkgyy ;yy gjg gjygjs y....jjygjyyj jjyy;yy (j. yj.gj. ikyj. gj. ;jy yyyjy yjs gjygjyyyg jjygjs $j44,, gjs lln- gj and contalns Ak+1 dxngxq-j) .iJ (j. gyjj,, gjjgjj y 0'1: '''''j . !.E ',..j';;jjc''''j;.)',.*.,.,,;.;,)k,q..qL,.j'q,)j-,Ls))jj)q,;t)).L'j,.j !1..2,'... ......1812.....,.... 1t!12$!E;k..
w
,E
,
.
y
y
,..
..f.kj)
L;..
.
E
.
y y t .y..;.. .. y.... zt'i')>t7)Eitu . . .,2:jj,g.
,1
.
-
-
..
,-
...,,q..yyy-
..,,.j.)
.
...
.
y
..
,
....
..ksysj-sssssyyyssy,j.yr
,..syyyy
.,
.
',','t.)y;;r,))',tyyyiy),.,),y!(jjj,y)g.,,yiityy;j,t,y.,,yy,,,jyjj,
y
,
''''
..j.,
,,
..)
=
j,;.jr
..
d,
jj:rr;jj
''
-'::::g
.iltrr;,:r
..
',-
q.g...j-: .E , q..
.(...j!;r';-r:).,..-jj)...
jjggygi
.....
d.
...t.
,
:
ktrriiir'dt:l'.k;,d::r '-,.,,
'-::::::'
...,
g
..,.fj),)),)jj)j)),,
..
,
...
.
..
,
.rqrrty.r. .
.
.g .jjjj. j,.
y..ygyjjjy2jjj;;;.yyjjyygjjg ;,,.q.;u,..,..,,,,r,,..
jjjjjjya.y.
.jjjgjrj...
,4..
..,..
.,r,...r,r,,,.,....)))))jj,jj.j:)jj(j,;..(qjj.
yjyyyjyy.yyyyj;yyj.. ..,...yyyyyj;..;.
..g yt -t-;t4i!)7-.ygjjtt' -
.
yjjjgj.y,-pk@$1r....-,... . yjya
--
.'.
'' ' .' t.ylii. Ji. .;..
:..... ,-..
..
yjjjjjjjjyyjj.g. yy,,yjjjyyyj,... g.yy.. yyy ...yy.,yyyy;
....yyjgyyyyy..
..
.
--$t'-.
.
.yyjjyyjy.
jsggggggggy
,,..).:,.j
...,..y .
,y
..
yyyjy.yygyyjy,
..yjyyyy. a.aayyayay..
y.
..
y,
-;jj-t.yy.
.
-...
--jjy;;-
..
.
-zi
.
....
..
.. ....
.. yjyjy;.;yjjry,;;jy..j(.. .g...j.jgy,...;.g..g........y.y. ..... . rysg..;
jjryrrjyjtr
.
.')' yj..-T!rt)j-ji.. .. . '-ty,--i)... ,,- EEE; ;.,-. ,qjy)j)y(. . E ,;,'.-);,.. ;' E..:-...'.-..?.-.., ..k:, ;.hF(.f.
...
-..
,..y......
,
.g.yyy..
ysy. (gylrqE..jj
.
).
.
jjjjjj,jj),,,j r.:g; .,yjy,4E ..ypj.'q).k(jyj.,j:;II,..: ),j)jj,..t jpkzjlk.g))jjjjjj):...k -(...
.:;,,::7(22,-
bif'
-.;(
)F?jjjgjjj:... r.()
j .. y,y.q.yt. .E ;..... )i):y:.
j!jjjjj).
jtkkgry
jtg
,,'..
.-;p7.
...
.jrr,
jjyyg;jj jjygg;jjjggygi
.
.'r'y)'-'.
,444,.4
.j(r,
!j,
,jjy,
''
.....
.j;,k:rr
.!jjj;,.
.
''.'''''12.12
.
jl,.
-.
.E.'.:.'.y..E.E......:... ),1lpl-t''t,-;-)@--
.,
.
...
.
.tlih'tjjrt.;2).-:.lqt.1r):.
!,-
77?:?,-' .... ;,..-.. b3-,Lt;j,,1'3$;t311L;;,;(;L)3$L;-t..i;'.).j)). .'.ti;;jit)()(.. . ... ;-
1t;;$(yr';)))(;...---.j$)yyk?Ey-..,it). -. . '-'. -(jj(-,-.'. '.
.-..
yjgy;yyjygyyyyy jygyyyyy
.
.
.
..
,. -
.
.
.
. .
.
y;jjjjyy
82
Mathematics
< 9Iln ensures that Dn c Sxn. But this means Dn c A, which is a contradiction since Dn has no finite cover by C-sets, Hence C contains a finite subcover, and (c) implies (a). K In complete spaces, the set properties of relative compactness and precompactness are identical. The following is the converse of 5.5.
5.13 Corollary In a complete metric space, a totally bounded set A is relatively compact. Proof lf S is complete, every Cauchy sequence in A has a limit in S, and a1l such points are closure points of A. The subspace (A,#)is therefore a complete space. It follows from 5.12 that if A is totally bounded, X is compact. .
5.3 Examples The following cases are somewhat more remote from ordinary geometric intuition than the ones we looked at above.
5.14 Example In j12.3 and subsequently we shall encounter R=, that is, inhnitedipensional Euclidean space. lf x (x1,.n,...)e R=, and y @1,y2,...)e R= similarly, a metric for R* is given by =
=
X
d-x-y) >72-Q()(x,,y,), al
(5. 15)
=
where do is detined in (5.1).Like do, t/x is a bounded metric with #xtx,yl f 1 for all x and y. n (RX,Jx) is separable and complete.
5.15 Theorem
Proof To show separability, consider the collection
4
=
xk rational (x (x1,x2,...):
if k S m, xk
=
R=
=
0 otherwise
)
(5.16)
.
(A,u, m Am is countable, and by 1.5 the collection A and e: > 0, H x e Am such that able. For any y e =
R''O
m
=
1,2,... J is also count-
co
d-x,y)S y2-$ -al
+
-t=
5-' 2-4(0,yk)
S
m+ 1
e+ 2--.
(5.17)
Since the right-hand side can be made as small as desired by choice of e:and m, y is a closure point of z4. Hence, A is dense in R=. To show completeness, suppose (xn @1n,x2n,...), n e N ) is a Cauchy sequence in R ' Since dtxknvxkm)S 2J x @n, xm4 for any k, fxa, n e EN ) must be a Cauchy sequence in R. Since =
.
d-xx)
/
57l-kdxkakn) 1 -t=
+
2-'n
(5.18)
Metric Spaces
83
for all m, we can say that xn x @1,x2,...) xk for each k e R= iff xkn completeness of R implies that lxn) has a limit in R=. w 1,2,...., the -->
--)
=
=
5.16 Example Consider the infinite-dimensional cube', g0,1J=;the Cartesians product of an infinite collection of unit intervals. The space (g0,11=,#x) is separable by 5.8. We can also endow (0,11= with the equivalent and in this case bounded metric, X
772-11x: -ytl
Px(x,y) =
(5.19)
u
.
a1
In a metric space (S,*, where Jcan be assumed bounded without loss of generality, define the distance between a point x e S and a subset A ? S as dxl) infysxJtx,y). Then for a pair of subsets h,# of (S,#) define the function du 2 s x zs j..o R+ =
,
W here
2S
is the power set of S, by du(A,B4
max sup d@A), sup d(y,B)
=
xB
(5.20)
.
yGA
du(A,B) is called the Hausdorff distance between sets 4 and #.
5.17 Theorem Letting J'fs denote the compact, nonempty subsets of S, ls,du) is a metric space. satisfies 5.1(b) since the sets of Rs are 0, so that du is only a pseudo-metric for for any x E 4 and any z E C we have, by d is a metric,
Proof Clearly du satisfies 5.1(a). It closed, although note that duA, W) general subsets of S. To show 5.1(c), definition of dx.B) and the fact that =
sup dx,B) K supttftmz)
)
dz,B)
+
Ac#
xcA
(5.21)
.
Since C is compact, the infimum oyer C of the expression in braces on the righthand side above is attained at a point z e C. We can therefore write + dz,B)) < snp tnf(4(x,z) supltms) A ze xed
.xG
supltz,l). sup (;:,6'D z
(5.22)
.#
y', :'y' rtjy'E (;,'. :'. E' j::yt',.'Lli'y.jjy:yjj'lt-.kt4j?iyrj)j('.' y' j'yyjyy'. k'. ,', jjt!rrg;ji!r'd 111:::2;11!!'* :'i'.j,,g.' sgEy-yzgyj',d ;',rjjy't. (' !yj'. !,,' r' ;'y'g y' .L'. ;'. :@),'.,.,,;y.jjjj;g'j;g'kjrt.ilyjrg-7ldrjjklypysjlljpt'jjq'lrsr'd k' j't'yyjy,,, y'y .r' 44::7* jtd iE-':jd )' j'y;d qtd .y' jgy'1!. jydt.yy.rd j'gi i-(' litjjjltsiytj'y'. j);';' 4* r' yyd t':g'33t'. yjjy'. y;il'. y'jy,ytd yrd ytyy'.' jyy'. yrgj'.;yjyjd .'qL);)'. (''.. y' y-'( jy-'). (i'yd ygy';6!r;.'.jygjyj.yry!;(r!. jj',;L?'. .',jy',rg..jjjj. -'y -'
)k',.;jr!j'(;;'. .-' yy.j.-d .;j,' 33666666636663*, r)'... (tg,' ,,,,,,,,,,,,,'
''. .': .'.'.
sl'milarly,supyssltxz)
supzs
..??
...
-<
,-
.
.
:l::)Er!t. ax
s
(x
dr:s
ll.,l. (14:,.,, ..441k.
dEE.
.;)t:r
...1..-.
.
'..
.
..
.
'
:
.
,
.
'
,
and hence,
,
.
E ..',' .j .k.. 2..' . . . . . '...,-...,.'...g...,.....-,. ..-..:!:.......... ..,..k.., -.s. '..';
(2), drr;ir.n
---.11..
(z ) + supyss:
-<
'
.i(:).-, 'i.ii!Eil'.....,...... ..,1:11.. s..,...12.. tii''fL;.:, (;,..qI...sll.,l.jll:), .!147,.. ..(14;,.. :;.
......(2,...... . .
....
..
.
..
i
. . . ' ..
:
:
c: )y)j
'
.
'.'
. .
'
.
'
,,.-,.11..
.
.
..,..
..
:. .i. E
:
C
: E: E j) E ) ,)L.q/;j-i...L,j;...3L;.....L.. :.E.; )...E... ..'q''. . . tlLi: ..;..-..1qs? .... E..E:(:; . ifr: .q. LE . y(..!. . .: q ( q..;..sy . ;LL;.1Lj. y:.Ey.tjyy.tpyit;kr . . . .. .... ' . . E. .. . . LL..L......L. . :Lj.LL..:....j.L.. Ljf,. . .: . . E. . .. .g k!:..(#k:gk....jtt .... L).;):,.;.)..).;.:L.....j;. ....4:j.. :.. j . .: . :....:(. i. .E. .jg (.... .. ..,. .... .. j.; , :. g. . j y: .;....L.. ; . .kj. jr.). yr . j y. .. ;'. ' (yj ( jjyE. k:, y . ..j E :.' E .y . y! :. . .. .y-.L- . :. g.( q j yy.. j.:; . ..r . . r yg . .y . . . .. .. '. yy. ' y;. . . y y yy . : .,),.;.tj..$ : E i: . . . .. . . :yjytrr...,:jygj ;.,y..... y.y... ! gs.tjjtyyyjygjgytjrjry 4)f,Ljj;.L;;;j).);).f..j;q))..)):k.. L...j),)j)):)..,..,.))q,..L4)q);;q;. .. .L.(. j..-.. y y. yy . .yj r j:.j).,.;t,), yyrjyjjjjjj., y.y. y .j.. .g yy. y...
E
.1:7
-
j
.1,.!;
.
..:..jj..y!.)y.g;...::.).j
....(
..
....;)..
dr:;ir-
.
. .
t)t,.g..y t:
..
.. .
.
.
.
)5.jk; qL., ... .j.y.. (. ..tjtr .. ( y.)(.) .. j.jjyt.yjt... (j) )y(jjy,..t;.yy: y.. . . . . jj;, (yg j . .yt;jg yq . (..;.j j,(jj . t' ):b,. qi(gy-. q.. ...:::::::::( ..... yyj . j .: . . . .. ry .yt,. . . . .j.. y. . jy...yry.jj,s .. t)$:.,;jjg:. . i.. yy,y . ItIL. t.y;y. ... y kk))r.' kjl.yy ; (-. .'. jj;.,. ' ...g. Lqjjjf. rjy . yy4tjj,r.. fjjjj.. .. . ; 11k2. dtr:;.!r-'l)p--.--?-j:,,-(-,-----. ----.-b--b-,i-------;-jbv.... di:22:). ...t .-... dk:r:..'...i...---..-).)--..-.---...-..-........---...;...i..f..t--...-;:., dl::::: 4.:2::,. dliz!,.-.. 1lE5'..lI::!'... :llr7.:':1L. 4:7-.. . .-tk-. ,,- :.1. ,--L-.-----?,,--. II. :!i4;. .. t,!!i9;...qlr.... ....-.... .-........-k;!!iiIr... a.)-....-.--...-.,y.,-.;,:.-... 11:::,,. ..
ti!iiE;k ,(:!:; -,-
..
:
..
...)
. ...
.,.,..;yyj:
:(.ygjq
..
.
.
.
.-jji!!!ijiij''
-.-..11......
,j.!!.!!!!Ii;... ' .
....
.
,
.
.
.
. ..
,
. .
.
.
.
.
.
:
. .
.
....
-
.tg,
,
..
.
.
.
.;
,
.,y...
.
-
.
...
j)
..((y.- -.
- '
- . .
tjkj;j ..(.
..
.
,
.
,),))))j.
.
.
. . .
.
.
,,
(:3?'.i!EiI? dE!!!E
. -
.
(5.23)
.
. .
...
.,,,:
,
.
.r
ll.
'1r:.
.
- .
.
,,.
. .
..
.
.
.
...
.
.
.
.yjj.,.
.
-.
Mathematics
84
5.18 Example Let S = R. The compact intervals with the Hausdorff metric define a complete metric space. Thus, ((0, 1 1/n!, n l EN) is a Cauchy sequence which converges in the Hausdolf metric to (0,11. This is the closure of the set g0,1) which we usually regard as the limit of this sequence (compare2.16), but although (0.1) Cts, JVE0,1),(0,1q) 0. u -
=
Another case is where S @2 J s ) and J'fs contains the closed and bounded subsets of the Euclidean plane. To cultivate intuition about metric spaces, a useful exercise is to draw some figures on a sheet of paper and measure the Hausdorff distances between them, as in Fig. 5.3. For compact yt and B, duA,B) = 0 if and B; compare tlzis with another intuitive concept of the only if A between two sets', infxsx,yss#stmy), which is zero if the sets touch or =
,
tdistance
=
intersect.
'' . .. .
. .
. .
'
.
. .
*1'1 .
.
.
.. .
.
.. .
.. .
. .
'
'
.
.
'
' .
.
' .
. '
.
.
.
.. .
'
.:::
.
. .
.
' .
.
D
: : 727::!:::::12::!; :;:: :::7::t:::!:!J1:::: ::'' :: : : : 7t : : : : : : :: :: :: )(72:2 : 7 7 ! : : 2. :: : l : :: L : l : L: : : ' 2: : : 1 ) . t :2:::::::::r2:2E:1:::2r:2:J:;::;::::t:
. .
' .
'
.. : . . . ..
'
.
..
.'
' .
' .
'
. .
'
'
' .
. '
.
'
. '
. '
.
' .
' .
. '
.
' .
' .
.::::.:.:.:.:.:.:.:.::.:::.::l.:::.:::;;:::i::.:.:.::::.:,.. . : .
.
dr;lr
igrjr
(:) a//qik
aj!jjl ,
()h
Fig. 5.3
5.4 Mappings on metric spaces We have defined a function as a mapping which takes set elements to unique points of R, but the tenn is also used where the codomain is a general metric space. Where the domain is another metric space, the results of j2.3 arise as special cases of the theory. Sme of the following properties are generalizations of those given previously, while others are new. The terms mapping, transformation, etc., are again synonyms for function, but an extra usage ksfunctional, which refers to the case where the domain is a space whose elements are themselves functions, with (usually) R as co-domain. An example is the integral defined in j4.1. The function J: (S,#) F-> (T,p) is said to be continuous at x if for all e > 0 H > 0 such that ys
p(.f@),f(x)) < z. sup sdx,n)
(5.24)
Here, may depend on x. Another way to state the condition is that for e > 0 3 > 0 such that
f(&(x,))i
5'p(.f@),e).
(5.25)
Melrfc Spaces
85
where Sd and s'pare respectively balls in (S,J) and (7,p).Similarly, f is said to be unbrmly continuous on a set A i S if for al1 : > 0, 3 8 > 0 such that < :. sup p(.f(.y),J(x)) supy e 5'J(x,)t''4x
(5.26)
xGz4
Theorem 2.17 was a special case of the following important result. 5.19 Theorem ForA I 1-, f -1 (A) is open (closed)in S wheneverA is open in 1-, iff f is continuous at al1 points of S.
(closed)
.f-144).
Proof Assume A is open, and 1et fx) e A for x q for some E > 0. By 1.2(iv) and continuity at x,
f-'fsdx,nbLtj c
Sdxnn)
Wz have Spflxln
? A
J-1(Sp(f@),e)) i f-'(A).
(5.27)
A is closed and f -1(V A) s If A is open then by 1.2(iii), which -1 is closed if f (A) is open. This proves sufficiency. To prove necessity, suppose J -1(A) is open in S whenever A is open in 1-, and in for E > 0 is open in S. Since particu lar J-1(' p (/x),E)) e f-(S p (.j(x),:)), there is a 8 > 0 such that (5.25)holds. Use complements again for the case of closed sets. . -y-1(4)
''
-
=
-
.x
,
This property of inverse images under J provides an alternative characterization of continuity, and in topological spaces provides the primary definition of continuity. The notion of Borel measurability discussed in j3.6 extends naturally to mappings between pairs of metric spaces, and the theorem establishes that continuous transformations are Borel-measurable. The properties of functions on compact sets are of interest in a number of contexts. The essential results are as follows.
5.20 Theorem
The continuous image of a compact set is compact.
Proof We show that, if A c S is compact and Jis continuous, then /'tAlis compact. -1 Let C be an open covering of fA). Continuity of f means that the sets f @),B s C are open by 5.19, and their union covers A by 1.2(ii). Since A is compact, these sets contain a finite subcover, say, J (#1),...,J Bm4.lt follows that .A
.j
y(A)
c /'
.j
c UBj, U.f-'(#y)U.f(.f-'(>) y=1
(5.28)
-
jv1
y=1
is by 1.2(i) and ihe seond inclusion by 1.2(v). Hence, Bj,...,Bm is a finite subcover of fA) by Cij4ts. sinceT' is arbitrary, it follows .jyjyd -),'jLiii'''L. k' 11:222:::;* y'. ;'y'. jjj;jy!r;'. y' yyd !)' -i-E.(E' y'y y'ty' ry'j y;t'11I::2t!):.!q'-?q#').i. q' j-q,,,4fb.'. ..' jy'. g'g'. j';;;r2qpjjy.'. (' jj122222,,r*. y'(!jjt:r!''-, ;'(j1Ii(r:j11:qr:1I,;.. diIt!j:rjt)'. (jii).d (r:ljill;k.d .y' y' rlll:rjjd ..-,;;p9i;j:r'.-' .tjjjrrjjq:!)gry:yjj(,.'. gjjjjjjjjj'.' j';' jyj''' (.(;-'j4' :!jEi;,-' 1;:Ej5)11,,*. j'y .r.j;4'?i (' LjjkLbftjt' 11*. y-' L-t--..,,blfbqk:t''L.' rjjlg)d rljjj;;-d ..' y' yy'.'y yyy'...-y'yjj'y,(;jjry.qjjjyy.rjj(.. (yjjj.'. ty'-tj.ytjjj't. -.-,'-qi;'fqbI;;'.' (111:r::114.*. (jjlyrrr!ll.d .,' .g' ty.-y.jjy'..' t)j)-',. that fA) is compact. w )')66fLLLj. .jjqn,qjv...'k::kf'qktLLLl''
where the equality
''.'.' .'. .'rs .'.'jy,.
-(j(y'-y' -'. --((2:jp,'.' ,:pj(,jg.' -j;yjjjjj,jyjjj,,'. '-jjj::jjrjj;'. ;'F:' ..:,' ';t';--,::;;;jg!!r;y.-j)-' ---')::' -.' ,'ygjjjj)jjjj(yyrjj(y,'yyy -'.' -' ,':' :-'jj(jgj.y.' -'jyj' ;',',' E'jyyyy j'.-kk;'. )jyyjy-.'' t-,'. y'yd jjr:;rr:r;d jy'y'((r)))I''C' .i-'',,j(;)' y' t'.-jjjyjy:jjj,.' yrd .t't(;yy!ryryjjjgr;rryy.k' ('. jr:'-'.:.,,-j,jj;jjjjjjk,:.,y)--.)t;'i.'. ;,,*j3.*313413;*. ---'ibbfLi',,j,'jg'y.yyj-,-l t'yjytt'js j'yd .y' g' (y;'. 41kk2:::;* jjkykzrr:;d 11;j2r!:;'. .it')E y' :',jjjkjg,r;j'. ;'yyllllirrrjlk.'gj .-;'.jjjjjjrjjjjj.'. t.' :''jjj;:rjjjjj...' )' .y' j'yj,d
-,' .--'.'-.;k1ijj(2!t$;'y ..' .11;:2)* '',*4111::)*. .111:2k::1,,*. -' .'. ','.'i;'i---'jkg;jjpjr.. -'j);,4,j)tf,'3t',' jjryjpjiij'j.. .y'yyjjprrtjjj-;' ,'y.jy'-yj -'. ,k-'-
5.21Theorem If the set
Jis
. .
continuous on
.
,
.
.5.(' .
.)
..'
.
. '.. . .
'.
; y.' . j . '. C
.'
.
.
.
fj....
'
'
.
.
.
( (.
.
..
njpd set, it is uniformly continuous on .
.
' '. ' . . lrEi ''.
'.'
.
.
.
..
.
,.
F y. jjl kj.g::( .)()('; k...
E .r:
..j
.
....
.. .
.
.
j
.
.
ryyy,y',.'yt;9y'jr,tty:j(,.i.yyy,rr,,y; tt'tkttyt, d fr eacil x e A, continity '' a.t x S be compact. C'..h,ptt ...!)E;1il):il).l!;)7tj!: ;..r' .'ij.tE itJ))j#15! .. ... ' 4).k6.). -. 7(; (.q-( ... .. . tjq. .)j k;jy'jiI-k::IIL.. y-;:!iii;y-. 5. .. ' :,!iij;;.. y. li .;;,.:ijj,. ..f- lil-. .IIkrIT).. (,11:::,14,.. :!ij;;--:1EE5111,( ...jj...y....-....C4;CIIi:jyjj,yy.. j;j!jrr:jjy. (qjj::--. ,11I::2jII),. jjjjyrr;jj:. iljj;;: rjjygq 11(2rrjjIIl,. ' . .... . ..yy.. y...y(IIE:r:11l,.. y .. jjjjgl:::)).. . (j11(r:)l1... t!jjI(4(:jpt.. ' gg.. ... .. j . jL:jji,,'''. .. : . .. E j . y . . . . '.. i... . j.j y..... .... . Jiii .).. . .(. . . .r....yyj .. y. y jg.. j. . j. ..yy.))j,. .y . . r jjjyy:t.. ,jjjjj)f:63L... ' ' . ., rrrjj;!lL . . .d1EE!i1. .. . .q-.'.rllqq-ll:E),s jj!jjj;j,. 4422::, d;::::: . y j-. . t. j .....j . .44:)( 4(:4):2,j-:, 11E1!2, . .... (Iir11.. . . tlEsll, (Ir:'' IIEE:!'' ;,jt-.,...jr))-...jj;tj.yjy gj .yy;y-(...-.,#.,.yyj,y,k.-.-.-.yy..y?yy)......t-,,..-.yt;y-.. jky.y..,y(y.yy(jyy!..-jyy-.. . j.yy.y ..y .j.... , .) .. y yyy.yjjyt.. jyy.., j.y;jj.. yk,-..-jy.)..;(.,... ... ,yyy.jjj. . .. ... jj-.. )j.,...;,tyy; 44:)( ((;), ((;), ((;), .; ..... jy.. ;y...yjtj;.....,jy..-jj#,,-.j;L... . ,.y.... . .. . . . . . ...j..... . . .
Proof Let A
.1jlgq,
-:rjjgijr,.
,gjj!r:r
,,.
,gyy..t: ,;....
. .
y..
y
.
E
-(:!i-l'
'
--. ..!tr:())yry-
.
.
...
...
-...
. .
.
:.,..
.,-
.
-
.
... ..
.-y. -.
-.74.
;' ):: l;!. ?!-):;)!t27tE.''''. . ,b:3b... :.'.'.,.Ekikkir:....-.. .;
--k::::::
.j;1d:r7
' .
.
.
.
''.
.
.
.
-
.
..''.
..
.
'.
-
-, . -
.,
.
qE-y q. -
E $it)kyy.)jtt.i..rjjjy)y -.-y j. .- ,, . .
,
.
....
. -
-...
..
.
,-jjj,!r!,jjj(j-.
. .
.
y
.
.
.
.
.
..yy)....r.-kt..j). q. ..
....
.
.
.
,
y .:
.
.
;.;(gt;-). .-.
..
,j.
t;y
E
' '-'..
.. .
.
..
.'Erjj,?.
.
!.
.
.. .
:
.
.
, -
''
,g;j/r:'
.
.,
E'7
. .
y
.
jjjyyygyjjj. ...
.
-,yy -.,
y
.
.
y
...
tyy . ,. y..y..;jj,-, r,jj:j:r:;;. . y-jjrrjqyjjjjj-t., - ,)..t,y.-.y,.yj.t,,yt.,yy,(.t, .y . . .
,
,
.
,
..
,-,.
-
..
.
.
y.
.
j
j . j..yy,.-y.y?.-y......-E.-y-.q.-..-.(q.y . ......-j;.;.;..-,:.,j-).j.;k.q.;jr-,y,..y,.yyy.-.t.y4.
...r.
.
.
-
...(j.....
....
.
. .
.
.
.
.
-
y-. y. .y yyy.
.
-
y. .
. . . . yy. g y-. . y-. yj y. -gyy. . . . . . y.. .. . .
y. .
,
-
-.
Mathematics
86
1,...,-. compact they contain a finite subcover, say Sdlxknr, k consider of points x,y e S such that dlx,y) < minlckcmrk, and a pair < zlE, and also e Sdxk,rk) for some kn so that pfx,fyjj dxkvx)S dlxk,yt + dlx,y) f rk + 6 S lrk,
using the triangle
inequality. Hence plflxnfxl)
S
f
.
(5.29)
r1e,and
< e. S p(J(x),f (x1))+ p(Axk),J(y)) p(f@),J@))
Since, 8 independent of x and y,
Let 8 Now, y =
=
(5.30)
is unifonnly continuous on A. w
T' is 1-1 onto, and f and f -1 are cont inuous, J is called a homeomorphism, and S and are said to be homeomorphic if such a function exists. lf S is homeomorphic with a subset of T', it is said to be embedded in 'T by f. If f also preserves distances so that p(f(x4,f@)) #@,y) for each x,y e S, it is called an isometry. Metrics J1 and #2 in a space S are equivalent if and only if the identity mapping from (S,J1) to (S,J2) (themapping which takes each point of S into itselg is an homeomorphism. If
.f:
S
F..+
'r
=
5.22 Example If Jx and poo are the metrics defined in (5.15) and (5.19) respectively, the mapping gl (R=,#x) ((0,1)=,px),where g = (:1,:2,...)and --)
is an homeomorphism.
Xi
1
'(x) 2 -+
=
2(1
dxnyt<
t.tkJ is
Ixfl)
,
i
=
(5.31)
1,2,...
EI
Right and left continuity are spaces, but there is a concept of the range of the function. A semicontinuous at x if for each
If
+
not well defined notions for general metric continuity which is with respect to function f: (S,#) >-> R is said to be upper E > 0 3 5 > 0 such that, for y e S,
=
/'(J)
<
fx)
(5.32)
+:.
0, upper semicontinuity a sequence of points in S and #@n,x) implies limsupa/xn) f fx). The level sets of the form (x: f(x) < aJ are open for a1l (x e R iff f is upper semicontinuous everywhere on S. f is lower semiis upper semicontinuous, and f is continuous at x iff it is both continuous iff upper and lower semicontinuous at x. A function of a real variable is upper semicontinuous at x if it jumps at x with fx) > max(/'(x-),.f(x+)J;isolated discontinuities such as point h in Fig. 5.4 are not ruled out if this inequality is satisfied, On the other hand, upper semicontinuity fails at point #. Semicontinuity is not the same thing as right/left continuity except in the case of monotone functions', if f is increasing, right tlef't) continuity is equivalent to upper (lower)semicontinuity, and the reverse holds for decreasing functions. The concept of a Lipschitz condition generalizes to metric spaces. A function f on s,dj satisfies a Lipschitz conditionat x e S if for > 0 3 M > 0 such that, for any y e Sdx,), -->
-f
Metric Spaces P(.f(A'),.f@)) f Mhdx,yq)
(5.33)
where (.): R+ F-> ER+satisfies hd) 4, 0 as d 0. It satisfies a unrm Lipschitz conditionif condition (5.33)holds uniformly, with tixed M, for all x e S. The remarks following (2.9) apply equally here. Continuity is enforced by this conditionwith arbitrary and stronger smoothness conditions are obtained for specialcases of h. .1,
,
Fig. 5.4
5.5 Function Spaces The non-Euclidean metric spaces met in later chapters are mostly spaces of real functionson an interval of R. The elements of such spaces are graphs, subsets of R2. However, most of the relevant theory holds for functions whose domain is any metric space (S,#), and accordingly, it is this more general case that we will study.Let Cs denote the set of all bounded continuous functions S F-+ R, and .f:
de/ne
duf
g)
spl
=
J(x)
I
gx)
-
(5.34)
.
xe S
dv is a metric.
5.23 Thorem
Proof Conditions 5 1(a) and (b) re immediate. To prove the triangle inequality CN, write, given functions J, g and h .
ttd .yy.(' t'yyty'
t';'j' ;'E y' t'(' )' t''r' y;t'. j':'jyjy' j'. y' (' .);' j'g' .-j;'. -' -'.jt'
.'.'
dfbhj
=
SLIPj
...
Ies
:
'.
'
.
'
..
...
-
hlx)
.'t ' '.
''
.
.
...4(:). 11.,1.. 4:yx) <- 154, jj(:),. ...j),b:. ). ;.3:''.' .x.....j. (.'.(. ).t . 1......#..... . ).I..jI..j .........4(:4.. .......+.,.... .j.. .
j!jj,r
S
IE
<-d
'
.E .E j : ( .. . . . .'' .. .. . ... .g . j.j . .!#.,p
..
q' . . ; g.: . . ... g.. jy . , .. . '
g.. .
,g.)
.....:.(
'
. .' g.. . .
.
.
.
. bhli ..
,
(5.35)
.
;yyr.,
,, .;y:ry.,:tj..t.yjyEy.yj..-.y i.).. ;.jLE:.y. ... ... . j.y j q.. -
( .. E E riy; . E .. ;)(-. E :..
(y)j
4:yx)j
E :'
(
'
. .
. ..
rt).yjy r ; .y( y. .q.. .): .. . r:l).))-.-'jjyyjjgjj(jgj(yjj(ytk.:i'q).;'f'l. . (2yjy y.. . j.3fk.i.. .yy ..,)L-) t!y(j-(..y. .ytj) (j-EJII?). ..' ' @.q .; )jq)j,gL li.ygry j)..)j) j,yy .;yjgjjy;j(jjtj;;y yyjjt yj;( jkjgr jjyi jjjjyj. gjypj:g;yjjyj y;ry..jy.jir.,,.ggyyj.;, .;.g....y....y,j.t-(y.jy.yy. gj. .
.
.
E- .
. ..)...:t;!..k...
:
.
) ;
j
j/ri!r
--....
..
...jt
-i
. . .. - y jEjqy t..t(Ehi.( ..
..
y
r
-y,.. .. ...-
, -,:j;
..
:.r,
-
,
q:,);jj..-L.L,.j;;.,,fjfL.. . . . k; , ....
-
-,
,,
,jfjj),.
jy....(:.j)-ri..(..y,.q jkg;y jjgyr $, gjygjggjj gkyjryyt,..44:. j)yljjk.jj. . y. . gjjg;j .jj. yjjyjyyjjjyj..,.. .. .j ;q f't;jtj.. vy,, t(; , yy t-.-tjj,..--tj:.?-.,gjykjjj,y--,(11). ,1:2:11Lqf ....y)j.jyjJ;;.-,g;yyr; . dlzz!ri ,1:4::,1 (11.-y -:3)q.jL:'6t3. jl:r-.lk:. ,L...tj,j,(.L)L;));;;--):-:fi. yj jII2:bi:.I2:bi:. 66113L32:'......,. jyjy yryj,jjj-.;jjs---, llk.zll. ..--tE!i!t.r.' :i4; Jjiki11:::),. jlr,:lk.ylrrrllr.rlk.tjjgrr)l fyurjyjjy (; ou yj uou s z.lttrit/lltq $:E1,1, ,..).L......j .,,,)#t)-t!s,i(.. yjjy. j.... rlrr:)ks .. . t..-,,.. ,r,.,y.-,1,). ....,,....s....-r......-..k....y,.z..-. tt. .-,,.,,. -,t!)1y),ty,,j;-.,-..... '.y?.-.-. .. . (-v...;;,.--k.,,,. iills. dk,r jj-,......-,,,,,,,,,-,-,,.;2,... ;;,,,,-#.s---,...,.,...,..,.,...=
.-d,d!!qjj-.
--
IS Compact,
.
'1rr.
.
..,j
.k E'ktliikt..
.1rr.
r
..yyj
.E#.(. .'.j)
t
.,,
...
-
-
. .
?
.
y
4:(11. . 14k1(:2: 4(:;1!' 112. 4kj2:ir;lj!'i!r 4k22: Io i!'-l!hl!'i!. llk(1. (1. llr!l. jry;?-21. ma.j
S
(x) (x)+ glx)
.jj.x,
.
,
,
..,;,.yy.:
Cs
k!r)-;,. y.. & 11::,-.
y ut. .#.. . ..g
.
<,
,..
.
.,
. .
,
.-,..1.
.
.
.
)
.
,
,.-y-.
:(q)
s..,..l
....., ,..
.
,
.,
.
functi o ns lf tj. !lr:.L d:::r d:::r Ik,r (1k. l. dlE:,dt:EE::'t!!,; atl (:j,,. m(14:,, a t ev. 'EEE!E as . -,
.
-,k-,.
-,k-,.
,-
.
-r,,--
Mathematics
88
a uniformly continuous restriction to S, and evel'y f e Us has a continuous extension to V, say J, constructed by setting Jx) = J(x) for a' e S and limu/xu) for a sequence fx:.G S l converging to x, for each x e V-S. Note that duf,f). so that the spaces Ck and Us are for any pair fbf' G S, dvf'l but these isometric. There are functions that are continuous on S and not on cannot be uniformly continuous. The following is a basic property of Cs which holds independently of the nature of the domain S. 'x)
=
=
,
5.24 Theorem
(Cs,#g) is complete.
Proof Let (Jnl be a Cauchy sequence in C's ; in other words, for 6: > 0 3 Ns 1 such that dulfnnfm) f E for n,m > Nz. Then for each x e S, the sequences (/@) ) satisfy Ifnx) fmx)l s Jtz(/,Al; these are Cauchy sequences in R, and so have limits fx). In view of the definition of dv, we may say that jn J unifonnly in S. For any x,y 6 S, the triangle inequality gives -
-->
lfx)
-.f(A') l f
1/'(x) fnx) l + l/@) -
fnlL
-
I + Ifn
Fix e > 0. Since fn e Cs, ('i 6 > 0 such that Ifnlx) Also, by unifonn convergence (; n large enough that -
maxfIfx) fnlx)l Ifaly) f@lI l < -
-
,
so that Ifx) J(y)1< z. Hence f e -
!
-f(.y)
(5.36)
.
I< Jn(.y)
Y if dx,y)
13s,
<
.
(5.37)
C's, which establishes that Cs is complete. w
Notice hpw this property holds by virtue of the unifonn metric. It is easy to devise sequencej of continuous functions converging to discontinuous limits, but none of these are Cauchy sequences. lt is not possible for a continuous function to be arbitrarily close to a discontinuous function at every point of the domain. A number of the results to follow call for us to exhibit a continuous function which lies. uniformly close to a function in Us, but is fully specified by a finite collection of numbers. This is possible when the domain is totally bounded.
5.25 Theorem Let (S,J) be a totally bounded metric space. For any f G &s, there existj for any : > 0 a function g e Us, completely specified by points of the such that dvjqj < E. D and rational numbers J1,...,Jm, domain .x1,...,au
We specify rational numbers here, because this will allow ud to assert in applications that the set of a11possible g. is countable.
Proof n By total boundedness of S, R for 8 > 0 a finite ,1) xi, 1et Af = fx: dx,xi) k 26) and Bi (x: dxnxi) K gi: S r- 0, 1J by '
(x1,...,au)For
-net
=
gix)
dxli) =
dxAij
+ dlX,
s i ),
each and detine functions .
,
(5.38)
where dxA) = infysxltmy). dxl) is a uniformly continuous function of by constmction, and gilx) is also uniformly continuous, for the denominator is never .:
Metric Spaces
less than
).
89
Then define
'bl '4x) '7 -lp/xl7f
=
(5.39)
.
-1:/x)
Being a weighted average of the numbers (JfJ,gx) is bounded. Also, since (xflis for S, there exists for every x e S some i such that dxli) k as well a ZT=Ip(x) and < dx,Bi) dx,xi) Therefore, f k hence that gix) such as and unifonn continuity extends from the gi to g. For arbitrary f e &s, fix E > 0 and choose small enough that Ifx) fyl l < ls when #(x,y) < 26, for any x,y e G. Then tix m large enough and choose xf and ai Jfl for each i. 1,...,-, so that the Sxinn) cover S, and lfx < for i Note that if dxtx k 28 then e Aj and gix) = 0, so that in a1l cases -net
,
,
.
-
=
Y
-
-x
gixtlfx) fx I f
.ftxls.
(5.40)
-
Hence
gix)Ifxt
-
ai
I<
lfx) f
gix)
x
-
I+
Ifx
gix)
-
ai I
< gixtz for each x
6
(5.41)
S and each i. We may conclude that
duf,gl
=
sup Ifx)
-:(x)1
x6 5
S sup
ZT-1p(x)I fx)
Jfl
-
<
Z:'-1'/x)
xss
z.
(5.42)
.
The next result makes use of this approximation theorem, and is fundamental. It tells us (recallingthe earlier discussion of separability) that spaces of continuous functions are not such alien objects from an analytic point of view as they might at first appear, at least when the domain is totally bounded.
5.26 Theorem (i) If (S,#) is totally bounded then (&s,Jg) is separable. (ii) If (S,#) is compact then cn,dulis separable. Proof We need only prove part (i), since for part (ii), Cs Us by 5.21 and the same conclusion follows. Fix m and suitable points (xl,...,aulof S so as to define a countable family of functionsAm = fgmk,k e IN), whUre the g.k are defined as in 5.25, and the index k of rationals. For enumerates the countable collectibn of lrvectofs (J1,...,tu) > 0, there exists m large enppgh yhqt,for each f e &s, duf, gm < : for : each 5 ij, tits'ftibi, apd there exists gk e A such that somek. By 1. A limp-xA?nTh@'! 'd.ttlh,gtA 0. : is dense in &s. . < ,, dvy, gk) : for every > yytlyyttttt. yyty d1:::,i-....t).',t)'tt ..jj3i,1.. ,).() !j,, yj,jsy, EIl..jjojoug uyytsy mox geueraj jyy ..... ., To show that we canno :lrr-.d.pir,... j));;.,j);)q.;j...;);)... ;;)))..;....-..; .,.434... yy...;jygr-.j-y--.. .. )j-#).;-y--t..r.-,)g. y)-... yty. ...t-y.y-.)(tyy.y,yy. =
'
.
,
'.
.
=
'
..
.
.
..y,,y.
.)
'ril,.
'
.
' 'ttyljk
,
,,
'tij.'yjy,,yt.#,y,
--.i1friil-.
.1r:. .)ky,yy..yyy-. ..... r;-.)t);t@ -jyyt-r),t-.y.ytyj-,.-.
,typtl-l-y-y-)..
.
... . .
........... ..
-..y
.
-.
.
..
Mathematics
90
function space.
circumstances, we exhibit a nonseparable
5.27 Example For S (tz,:J,an interval of the real line, consider the metric space Dka,bndu) of real, bounded cadlag functions of a real variable. Cadlag is gauche) to describe droite, limites a colourful French acronym continue functions of a real variable which may have discontinuities, but are right continuous at every point, with the image of every decreasing sequence in Ltz,:)containing its limit point; in other words, there is a limit point to the left of every point. Of course, Cp,s) c Dp,x. Functions with completely arbitrary discontinuities form a larger class still, but one that for most purposes is too unstructured to permit a useful theory. =
To show that Dfa,bjsdu) is not separable, conslder the subset with elements
J/f)
0, t
<
0
=
1, t k 0
,
0
(5.43)
EJ,1.
6
This set is uncountable, containing as many elements as there are points in gtz,hJ. But duh,h 1 for 0 # 0', so it is also discrete. Hence Dta,bj,du) is not separable by 5.9. (n =
Let A denote a collection of functions f : (S,#) F-y (T,p). A is said to be equicontinuous at x G S if Y 6: > 0 3 > 0 such that
sup .feA y
sup pfyt,fxj) G
Sx,&)
,4 is also said to be unformly
supa:ssup
feA
s
(5.44)
< :.
if Y 6: > 0 3 8
equicontinuous
sup pfyl,fxl)
>
0 such that
(5.45)
< :.
yc Saxnnl
Equicontinuity is the property of a set of continuous functions (or uniformly continuous functions, as the case may be) which forbids limit points of the set to Cs (&s) but A is not be not (uniformly)continuous. ln the case when (uniformly) equicontinuous, we cannot rule out the possibility that W Cs (Us)k An important class of applications is to countable A, and if we restrict attention to the case A = ffn, n e IN) A c Cs torUs) may not be essential. lf we are willing to tolerate discontinuity in at most a finite number of the cases, the following concept is the relevant one. A sequence of functions f/, n e IN) will be said to be asymptotically equicontinuous at if Y E > 0 3 > 0 such that :4
,
.x
limsup l-yx
and asymptotically
sup
ye Sdxszj
?zzif/brzlfy
p(A(y),Jn(x)) < e,,
equicontinuous if V e, > 0 R
(5 46) >
0 such that
Metric Spaces
limsup sup n--:x
;r
e
sup
91
p(Jn@),n(x)) <
(5.47)
E.
Jx,&)
ye
If the functions fn are continuous for a11n, limsupn-yx can be replaced by supn in (5.46) and similarly for (5.47)when all the fn are uniformly continuous. ln these circumstances, the qualifier asymptotic can be dropped. The main result on equicontinuous sets is the Arzel-Ascoli theorem. This designation covers a number of closely related results, but the following version, which is the one appropriate to our subsequent needs, identifies equicontinuity as the property of a set of bounded real-valued functions on a totally bounded domain which converts boundedness into total boundedness.
5.28 Arzel-Ascoli theorem Let (S,#) be a totally bounded metric space. A set A c Cs is relatively compact under dv iff it is bounded and uniformly equicontinuous. Proof Since Cs is complete, total boundedness of A is equivalent to relative compactness by 5.13. So to prove if' we assume boundedness and equicontinuity, and construct a tinite E-net for A. It is convenient to define the modulus of continuity of f, that is, the function w: C xR+ h- R+ where ,
s
w(T,) Fix s
>
0, and choose 8
=
sup a.s s
(as is
If(.y4 J@)I
sup
.
ysdx,)
equicontinuity)
possible by unifonn
sup w(.f,)
(5.48)
-
such that
(5.49)
< :.
ysA
Boundedness of A under the unifonn metric means that there exist finite real numbers U and L such that inf
L f
/'lA.xe
be (x1,...,xm)
Let
S
#4x)f
sup DA,XC
U.
(5.50)
5
for S, and construct the finite family
-net
a Dm
fx) S
=
l'k
A, k
q
=
1,...,(v + 1)-)
(5.51)
according to the recipe of 5.25, with the constants ai taken from the finite set
where u and v are integers with v exceeding CU L4Ie and u 0,...,v. This set contains v + 1 real values between U and L which are less than 6: apal't, so that Dm has (v+ 1)= members, as indicated. Since the assumptions imply
fL
+
(&- Ljulvj
-
,
=
4 c &s, it follows by 5.25 that for every f E A there exists gk e Dm with < :. This shows that Dm is a e-net for A, and A is totally bounded. dvf,g ito suppose A is relatively compact, and hence totally bounded. To prove Trivially, total boundedness implies boundedness, and it remains to show unifonn equ icontinuity. Consider for E > 0 the set ionly
,
'
Mathematics
#k(:)
(/': w(/',1/k)
=
< :).
(5.52)
Uniform equicontinuity of A is the condition that, for any
large enough that
X'c
Bkz). lt is easily veritied
lw(.f,)
-
w(',*
6: >
0, there exists k
that
I f 2JrXJ,'),
(5.53)
R*,dE)
F--> is continuous. Bkz) is the inso that the function w(.,): (Cs,d verse image under w(.,8) of the half-line (0,:) which is open in R+, and hence
x 0 as k for each f e Bkzj is open by 5.19. By definition of Cs, w(/', 1/k) Cs In other words, w converges to 0 pointwise on Cs which implies that the Cs, and hence forW.But b.collection (#k(:),k e IN) must be an open coverinAfor covering is such of has A X hypothesis compact, every a finite subcover, and so A q Bke) for finite k, as required. . -->
--
.
,
6 Topology
6. 1 Topological Spaces Metric spaces fonn a subclass of a larger class of mathematical objects called topological spaces. These do not have a distance defined upon them, but the concepts of open set, neighbourhood, and continuous mapping are still well defined. Even though only metric spaces are encountered in the sequel (Part VI), much of the reasoning is essentially topological in character. An appreciation of the topological underpinnings is essential for getting to grips with the theory of
weak convergence. 6.1 Definition A topological space (X,I) is a set X on which is defined a topology, a class of subsets called open sets having the following properties: (a) X e T, O e 1. (b) If C i T, then U()scoe T. Oa G T, then 01 f-'h 02 q T. n (c) If 01 G 'r
'r,
These three conditions dehne an open set, so that openness becomes a primitive concept of which the notion of e-spheres around points is only one characterization. A metric induces a topology on a space because it is one way (thoughnot the only way) of detining what an open set is, and a1l metlic spaces are also topological spaces. On the other hand, some topological spaces may be made into metric spaces by defining a metric on them under which sets of are open in the sense defined in j5.1. Such spaces are called metrizable. A subset of a topological space (X,I) has a topology naturally induced on it by (A f''h 0: O G 'IJ is called the the parent space. If A c X, the collection relative topology for A. (A,n) would normally be referred to as a subspace of X. and If two topologies are defined on a space and :1 c T2, then T1 is said to be coarser, or weaker, than T2, whereas 12 is hner stronger) than I1. In particular, the power set of X is a topology, called the discrete topology, whereas (?,x J is called the trivial topology. Two metrics define the same topology on a space if and only if they are equivalent. If two points are close in one space, their images in the other space must be correspondingly close. If a set O is open, its complement Oc on X is said to be closed. The closure X' of an arbitrary set A ? X is the intersection of a1l the closed sets containing A. As for metric spaces, a set A i #, for B c X, is said to be dense in B if B c X. ':
'u
':1
=
.:2
6.2 Theorem The intersection of any collection of closed sets is closed. X and O are both open and closed. n
Mathematics
94
However, an arbitral'y union of closed sets need not be closed, just as an arbitrary intersection of open sets need not be open. For given e X, a collection Vx of open sets is called a base for the point x if for every open O containing x there is a set B e Vxsuch that x e B and B c 0. This is the generalization to topological spaces of the idea of a system of neighbourhoods or spheres in a metric space. A basefor the topology T on X is a and every x 6 0, there exists B e collection V of sets such that, for every O 6 that V such x e B c 0. The definition implies that any open set can be expressed union the of sets from the base of the topology; a topology may be defined for as specifying a base collection, and letting the open sets be defined as a space by the unions and finite intersections of the base sets. ln the case of R, for example, the open intervals form a base. .x
,r,
6.3 Theorem A collection V is a base for a topology I on X iff (a) Ulevf = X. (b) V #1,#2 e F and x e #1 chB1. 3 B5 e V such that a' e #3 c #1 /n %. Proof Necessity of these conditions follows from the definitions of base and open in terms of the base F, as follows: set. For sufficiency, define a collection 'r
O e
'r
iff, for each
.'&'
e 0, H B e F such that x
q
B c 0.
O s4tisfies the condition in (6.1), and X satisfies it given condition (a) of the theorem. lf C is a collection of T-sets, UtxwO e T since (6.1) holds in this case in respect of a base set B corresponding to any set in C which contains x. And if 01,02 G 1, and x e 01 fa O1. then, letting #1 and B1 be the base sets specified in (6.1) in respect of x and 01 and 01 respectively, condition (b) implies that x e B5 c 01 f''h O1, w ltic h shows that I is closed under finite intersections. Hence, is a topology for X. . ':
The concept of base sets allows us to generalize two further notions familiar points) of a set A in a from metric spaces. The closure points (accumulation topological space (X,'r) are the points x e X such that every set in yhe base of a7 contains a point of A (a point of A other than x). An important exercise is to show that x is a closure point of A if and only if is in the closure of z4. We have generalizations of two other familiar concepts. A sequence tak) of points in a topological space is said to converge to x if, for every open set O containing x, H N k 1 such that ak e O for all n k N. And is called a cluster point of (.:kl if, for every open O containing and every N k 1, xn e O for some n k N. ln general topological spaces the notion of a convergent sequence is inadequate for characterizing basic properties such as the continuity of mappings, and is augmented by the concepts of net and///er. Because we deal mainly with metric spaces, we do not require these extensions (seee.g. Willard 1970: Ch. 4). .x
.x
.x
6.2 Countability and Compactness The countability tutppu provide one classitkation of topological spaces according, roughly speaking, to their degree of structure and amenability to the methods
Topology
95
of analysis. A topological space is said to satisfy the first axiom of countability (to be jrst-countable) if every point of the space has a countable base. It if the space as a satisfies the second axiom of countability (is second-countable) whole has a countable base. Every metric space is first-countable in view of the existence of the countable base composed of open spheres, Sx, 1/n) for each x.
More generally, sequences in first-countable spaces tend to behave in a similar manner to those in metric spaces, as the following theorem illustrates.
6.4 Theorem ln a first-countable space, x is a cluster point of a sequence (.v, n e IN) iff there is a subsequence (xa#,k s IN) converging to x. Proof Sufficiency is immediate. For necessity, the definition of a cluster point implies that 3 n k N such that xn e 0, for every open 0 containing x and every N k 1. Let the countable base of be the collection Vx fBi, i G (11), and choose a monotone sequence of base sets (Ak, k G INl containing x (andhence nonempty) with A1 #l, and Ak c Ak-l f''h Bk for k 2,3,...; this is always possible by 6.3. Since x is a cluster point, we may construct an infinite subsequence by taking xnk as th: next member of the sequence contained in 4s for k = 1,2,... For every open set O containing x, R N k 1 such that xnk E Ak 0, for a1l k k N, and hence xnk x, as required. w --> x as k .x
=
=
=
.-..:
The point of quoting a result such as this has less to do with demonstrating a new property than with reminding us of the need for caution in assuming properties we take for granted in metric spaces. While the intuition derived from R-like situations might lead us to suppose that the existence of a cluster point and a convergent subsequence amount to the same thing, this need not be true unless we can establish first-countability. A topological space is said to be separable if it contains a countable dense subset. Second-countable spaces are separable. This fact follows directly on taking a point from each set in a countable base, and verifying that these points are dense in the space. The converse is not generally true, but it is true for metric spaces, where separability, second countability and the Lindelf property (that every open cover of X has a countable subcover) are all equivalent to one another. This is just what we showed in 5.6. More generally, we can say the
following.
6.5 Theorem A
jecond-countable
space is both separable
and Lindelf.
Proof The proof of separability is in the text above. To prove the Lindelf property, let C be an open cover of X, such that UzeCA X.For eachA e C and e A, we can find a base set Bi such thgt x 6 Sj c A. Since UT=1#fX, we may choose a countable subcollection Af, i 1,2,... such that Bi c 4f for each i, and hence U7=1Xf X. K .x
=
=
=
=
A topological space is said to be compact if every covering of the space by open countably compact if each countable sets has a finite subcover. It is said to e coveringhas a finite subcovering. And it is said to be sequentially compact if subsequpce. Somesimes, compacteach sequence on the space has a convrgnt
96
Mathematics
ness is more conveniently characterized in tenns of the complements. The complements of an open cover of X are a collection of closed sets whose intersection is empty', if and only if X is compact, every such collection must have a finite subcollection with empty intersection. An equivalent way to state this proposition is in terms of the converse implication. A collection of closed sets is said to have the Tnite intersection property if no finite subcollection has an empty intersection. Thus: 6.6 Theorem X is compact (countablycompact) if and only if no collection (countable collection) of closed sets having the finite intersection property has an empty intersection. a The following pair of theorems summarize important relationships different varieties of compactness.
between the
6.7 Theorem A first-countable space X is countably compact iff it is sequentially compact. Proof Let the space be countably compact. Let fxn,n G EN ) be a sequence in X, and detine the sets Bn fxn,ak+1,... ) n 1,2,... The collection of closed sets (Tn, n e (N) clearly possesses the finite intersection property, and hence (X'n is nonempty by 6.6, which is another way of saying that fak) has a cluster point. Since the sequence is arbitrary, sequential compactness follows by 6.4. This proves necessity. For sufficiency, 6.4 implies that under sequential compactness, al1 sequences in X have a cluster point. Let ( Ci, i e INl be a countable collection of closed sets having the finite intersection property such that A,y 0Z=IC/ # 0, for every finite n. Consider a sequence fxn) chosen such that xn e A,,, and note since (A,, ) is monotone that xn e Am for all n 2 m; or in other words, Am contains the /?z). Since (ak) has a cluster point x and Amis closed, x e Am. sequence fak,n k This is true for every m e N so that OT=1Q is nonempty, and X is countably 6.6. by compact w =
,
=
=
,
6.8 Theorem A metl'ic space (S,#) is countably Proof
compact iff it is compact.
Sufficiency is immediate. For necessity, we show first that if S is countably compact, it is separable. A metric space is first-countable, hence countable compactness implies sequential compactness (6.7),which in turn implies that every sequence in S has a cluster point (6.4).This must mean that for any 6: > 0 there exists a finite E-net (m,...,auJ such that, for a1l x e S, dx,xk) < E, for some k E ( 1,...,-) ; for otherwise, we can construct an infinite sequence fxn) with dxn,xn') k 6: for n # n', contradicting the existence of a cluster point. Thus, for each n e IN there is a finite collection of points 4,, such that, for is countable and every x e S, #@,y) < 2-n for some y e An. The set D = U*n=1An dense in S, and S is separable. Separability in a metric space is equivalent by 5.6 to the Lindelf property, that every open cover of S has a countable subcover', but countable compactness implies that this countable subcover has a finite subcover in its turn, so that
Topology
compactness is proved.
97
.
Like separability and compactness, the notion of a continuous mapping may be defined in tenns of a distance measure, but is really topological in character. In the mapping /': X F--y f is said to be contina pair of topological spaces X and uous if J -1(#) is open in X when B is open in and closed in X when B is closed in That in metric spaces this definition is equivalent to the more familiar one in terms of s- and 8-neighbourhoodsfollows from 5.19. The concepts of homeomorphism and embedding, as mappings that are respectively onto or into, and 1-1 continuous with continuous inverse, remain well defined. The following theorem gives two important properties of continuous maps. ,
,
.
6.9 Theorem Suppose there exists a continuous space X onto another space (i) If X is separable, T is separable. is compact. (ii) If X is compact,
mapping
f
from a topological
.
problem is to exhibit a countable, dense subset of Consider .f(D4 where D is dense in X. lf J(D) is the closure of J(D), the inverse image f-JfD is closed by continuity of f, and contains f -1(/'(D)), and hence also coptains D by 1.2(iv). Since X'is the smallest closed set containing D, and X c 5, /X) it follows that X c J-1C(D)). But since the mapping is onto, f ff-1(fD))4 c J(D), where the last inclusion is by 1.2(v). J(D) is therefore ' in dense as required. J(D) is countable if D is countable, and the conclusion follows directly. (ii) Let C be an open cover of Then (J -1(#): B e C) must be an open cover of X by the definition. The compactness of X means that it contains a finite subf-1B n ) such that cover, say f -1(B l ) Proof
(i) The
.
=
.
,...,
y'
=
y(x) y =
Uy-'tsp= J y-1 tjsy c /=1 Usy,
y=1
(6.2)
j=L
where the third equality uses 1.2(ii) and the inclusion, 1.2(v). Hence C contains a Gnite svbcover. . in both these results. The extension Note the importance of the stijulation and f is obvious, and can be supplied of (ii) to the case of compact subsets of X reader. the by Completeness, unlike separability, compactness, and continuity, is not a topological property. To define a Cauchy sequence it is necessary to have the concept of a distance between points. One of the advantages of defining a metric on a space is that the relatively weak notion of completeness provides some of the essential features of compactness in a wider class than the compact spaces.
6.3 Separation Properties Another classification of topological spaces is provided by the separation tztppzu, whichin one sense are more primitive than the countabilityaxioms. They are
Mathematics
98
indicators of the richness of a topology, in the sense of our ability to distinguish between different points of the space. From one point of view, they could be said to define a hierarchy of resemblances between topological spaces and metric spaces. Don't confuse separation with separability, which is a different concept altogether. A topological space X is said to be: with x # y 3 an open set containing x but not y and - a Fl-space, iff V x,y e X also an open set containing y but not Fa-) space, iff V x,y e X with x # y 3 disjoint open sets 01 and - a Hausdotffo 01 in X with e 01 and y E O2; regular iff C Hdisjoint open sets 01 and for each closed set C and space - a 02 with x e 01 and C c O2; - a normal space iff, given disjoint closed sets C1 and Cz, 3 disjoint open sets 01 and 01 such that C1 c 01 and (72 c O2. .x;
.x
.x
A regular Fl-space is called a Fg-space, and a nonnal Fl-space is called a F4-space. In a Fl-space, the singleton sets (.:) are always closed. In this case y e (xJf (xJC is the complement of a closed set, and hence open. whenever y # x, where Conversely, if the F1 property holds, evel'y y # x is contained in an open set not containing x, and the union of all these sets, alsp open by 6.1(b), is (x)f. It is easy to see that F4 implies Tz implies Fa implies T1, although the reverse implications do not hold, and without the F1 property, nonnality need not imply regularity. Metric spaces are always F4, for there is no difficulty in constructing the sets specified in the definition out of unions of E-spheres. We have the following important links btween separation, compactness, countability, and metrizability. The first two results are of general interest but will not be exploited directly in this book, so we forgo the proofs. The proof of 6.12 needs some as yet undefined concepts, and is postponed to j6.6 below.
6.10 Theorem A regular Lindelf
space is normal. a
6.11 Theorem A compact Hausdorff space is F4. 6.12 Urysohn's able. u
metrizatlon
(a
theorem A second-countable
Fg-space is metriz-
In fact, the conditions of the last theorem can be weakened, with F4 replaced by Fa in view of 6.10, since we have already shown that a second-countable space is Lindelf (6.5). The properties of functions from X to the real line play an important role in defining the separation properties of a space. The key to these results is the remarkable Urysohn's lemma.
6.13 Urysohn's Iemma A topological space X is nonnal iff for any pair A and B of disjoint closed subsets there exists a continuous function f: X (0,1j such = 0 and J(#) 1. :l that jA) .-+
=
The function
J is
called a separating
function.
Topology
99
Proof This is by construction of the required function. Recall that the dyadic rationals D are dense in (0,1j. We demonstrate the existence of a system of open sets f Ur, r e D l with the properties A c &r;
Vrfa
#
'X c
0',
=
(6.4)
Ur for r
>
(6.5)
s.
Nonnality implies the existence of an open set 171/2(say)such that 171/2contains :4 an d (&1/2)ccontains B. The same story can be told with &f/2 replacing B in the role of C1 to define 171/4,and then again with 171/2replacing A in the role of C1 to define Um. The argument extends by induction to generate sets ( Umnn, m 1,...,2* 1 ) for any n e EN,and the collection ( Ur, r e D ) is obtained on letting n --A x. lt is easy to verify conditions (6.3)-(6.5)for this collection. Fig. 6.l illustrates the construction for n 3 when A and B are regions of the plane. One of the onion' in the limiting case. must imagine countably many more =
-
=
)1
!/3/8
315ls
:/1/2
!/3/4 Llzls
:/1/4
U 1/8
B
,4
Fig. 6.1 Now define
f:
X
(0,11by
--)
inffr
e D: x e &r) x e ,
Urss Ur
x Urso Ur
(6.6)
-
1 for x e #. Because of the monotone property (6.5), where, in particular, we have f:r any a e (0,1) .f(x)
=
(x: fx)
<
al
(x:inf fr e D: x e = (x:H r < a such that =
&r, = r
<
a)
x e &r)
(6.7)
which is open. On the other hand, because D is dense in g0,1qwe can deduce that,
Mathematics
100
for any k$e
0, 1),
(x: flx)
<
I31 (x:inf tr e =
D: x e &rl <
= (x: x e Ur V r
>
jlJ
))
Ur O rn.js
=
=
r)
(6.8)
r
r>f$
which is closed. Here, the final equality must hold to reconcile the following two and second that, for al1 r > I3,there exists (sinceD facts: first that Ur i and Us Ur by (6.5).We have therefore shown that, > > dense) D with is se r s k$ c for 0 S ;$< a ; 1, ,
(x: j
<
fx)
<
a)
(.x:flx) <
=
a)
fa
fx: flx) K
;'$)C
(6.9)
is open, being the intersection of open sets. Since every open set of (0,11is a union of open intervals (see2.5), it follows that f -1(A) is open in X whenever A is open in g0,11,and accordingly f is continuous. It is immediate that f(A4 0 and f(#) 1 as required, and necessity is proved. Sufficiency is simply a matter, given the existence of f with the indicated and J-1((11j), whose images are PrOPerties of citing the two sets J-1(g0,1)) a a, and respectively, in X, disjoint, and contain B so that X is normal. w open =
=
,
,4
It is delightful the way this theorem conjures a continuous function out of thin air! It shows that the properties of real-valued functions provide a legitimate means of classifying the separation properties of the space. In metric spaces, separating functions are obtained by a simple direct construction. If 4 and B are closed and disjoint subsets of a metric space (S,#), the nonnality property implies the existence of > 0 such that infxsz,yswJtmy) k 6. The required function is
fx)
=
Jtx,Al dxv)
+ ty(x,s)
(6. 10)
where dxl) = infysxttx,y), and J(x,S) is defined similarly. The continuity of f follows since dxl) and #(x,#) are continuous in x, and the denominator in (6.10) A similar construction was used in the proof of 5.25. is bounded below by The regularity property can be strengthened by reqtliring the existence of separating functions for closed sets C and points x. A topological space X is said to be completely regular if, for a1l closed C c X and points x C, 3 a continuous 0 and flx) = 1. A completely regular Fl-space function fl X 1- E0,1)with fc' is called a Tychonoff space or Fgj-space. As the tongue-in-cheek terminology suggests, a L-space is F3j (this is immediate from Urysohn's lemma) and a F3j-space is clearly F3, although the reverse implications do not hold. Being F4, metric spaces are always F?j. .
=
Topology
101
6.4 Weak Topologies Now let's go the other way and, instead of using a topology to define a class of real functions, use a class of functions to define a topology. Let X be a space and F a class of functions f : X i-- y, where the codomains y are topological under which spaces. The weak topology induced by F on X is the weakest topology -1 is continuous. Recall, continuity F (A) is means that J open in X every f e whenever A is open in y. We can also call the weak topology the topology generated by the base of sets V consisting of the inverse images of the open sets of the y, under f e F, together with the finite intersections of these sets. The inverse images themselves are called a sub-base for the topology, meaning that the sets of the topology can be generated from them by operations of union and finite
intersection.
increase the number of sets in this base and If we enlarge F we (potentially) get a stronger topology, and if we contract F we likewise get a weaker topology. With a given F, any topology stronger than the weak topology contains a richer collection of open sets, so the elements of F must retain their continuity in this case, but weakening the topology further must by definition force some fq F to be
discontinuous.
The class of cases in which y R for each f suggests using the concept of a weak topology to investigate the structure of a space. One way to represent the richness of a given topology on X is to ask whether contains, or is contained in, the weak topology generated by a particular collection of bounded real-valued functions on X. For example, complete regularity is the minimal condition which makes the sol4 of construction in 6.13 feasible. According to the next result, this is sufficient to allow the topology to be completely characterized in terms of bounded, continuous real-valued functions on the space. =
':
'r
6.14 Theorem lf a topological space (X,':) is completely regular, the topology is the weak topology induced by the set F of the separating functions. ':
Proof Let F denote the collection of inverse images of open sets under the funcr tions of F. And 1et F denote the weak topology induced by F, such that the F-sets, together with their finite intersections, fonn a base for F. We show that F T. For any x e X, let O e be an open set containing x. Then Oc is closed, and by l and complete regularity there exists f e F taking values in (0,1j with J(I) = J(O9 J-1((, 1q) is therefore an is open in g0,1j, and B 0. The set (1z,1) with Oc containing disjoint and that B c 0. Since this holds for x so open set, every such 0, x has a base Vx consisting of inverse images of open sets under functions from F Since x is arbitrary the collection V = ( Vx,x e XJ forms a base for T. lt follows that T ? F. On the other hand, F is by definition the weakest topology under which every f E F is continuous. Since f E F is a separating function and continuous under it also follows that F c 1. . =
'r
=
=
.
'r,
Mathematics
6.5 The Topology of Product Spaces Let X and f be a pair of topological spaces, and consider the product space X x The plane R x R and subsets thereof are the natural examples for appreciating the properties of product spaces, although it is a useful exercise to think up more exotic cases. An example always given in textbooks of topology is C XC where C is the unit circle; this space has the topology of the torus (doughnut). Let the ordered pair (x,y)be the generic element of X x f. The coordinate defined by projections are the mappings 7:x: X x F-) X and z:v: X x f F-> .
,
lx(.Y,')
=
A'
(6.11)
1:(-Y,)/)
=
y.
(6.12)
are topological spaces, the coordinate projections can be used to generate a new topology on the product space. The product topology on X x is the weak topology induced by the coordinate projections. The underlying idea here is very simple. lf A i X and B q f are open sets, the 1 1(#) will be lk - (z4)an d X x B = 1:yset A x B (A x f) rn (X x #), where A x f rectangle called of X The product regarded as open in X x and is x f. an open topology on X x is the one having the open rectangles as a base. This means that and @2,y2) two points @1,y1) are close in X x provided xl is close to x2 in X, antl yl to y2 in f. Equivalently, it is the weakest topology under which the coordinate projections are continuous. If the factors are metric spaces (X,#x) and f,h), several metrics can be constructed to induce the product topology on X x including If X and
=
=
,
,
,
@2,y2)) maxllxtxl,xzl, p((x1,y1), =
Jr(A'1,y2)l
(6.13)
and p'(@1,A'1), @2,y2)) =
#x@1sx2)+#:/1,y2).
An open sphere in the space (X x p), where p is the metric in happens to be an open rectangle, for ,
5'p(@,A'),)
=
Sdsxnnt
X Sdj,vnt',
(6.14) (6.13),
also
(6.15)
but of course, this is not true for every metric. Since either X or may be a product space, the generalization of these results from two to any finite number of factors is straightforward. The generic element of the space X2=1Xfis the n-tuple (x1,...,ak: xi e Xf), and so on. But to deal with infinite collections of factor spaces, as we shall wish to do below, it is necessary to approach the product from a slightly different viewpoint. Let 4 denote an arbitrary index se and fXa, (x l A ) a collection of spaces indexed by is the collection of a1l the mappings x: 4 4. The Cartesian product X Xasxxa F-y Uaszxa such that x(a) e Xa for each (x e A. This definition contains that given in j1.1 as a special case, but is fundamentally more general in character. The coordinate projections are the mappings 1a: Xn, F-> Xa with ,t
=
Topology
103
l(x(.%) Xq),
(6.16)
=
but can also be defined as the images under x of the points a e A. Thus, a point in the product space is a mapping, the one which generates the coordinate projections when it is evaluated at points of the domain z4. In the case of a finite product, A can be the integers 1,...,l. In a countable case A EN,or a set equipotent with EN,and we should call an infinite sequence, an element of =
.x
X'='(say).A familiar uncountable example is provided by a class of real-valued functions x: R F-> R, so that A R. In this case, x associates each point a e R R with a real number x(a), and defines an element of the product R The product topology is now generalized as follows. Let (Xa, a e A ) be an arbitrary collection of topological spaces. The Tychonoff topology (product X topology) on the space X h as as base the finite-dimensional open rectangles, sets where the Oa ? Xa are open sets and Os Xa except for at of the form Xas.4otx, most a finite number of coordinates. These basic sets can be written as the interkections of tinite collections of cylinders, say =
.
=
B
-1 aajtoaj)
=
-1
f-h
chzramto oo ;
...
,
o j'y; .
for indices a1,...,a,a e A. Let be a topology under which the coordinate projections are continuous. If Oz and hence contains the Tychonoff topology. Since is open in Xa, aa-1(O(: ) e this is true for any such 1, we can characterize the Tychonoff topology as the weak topology generated by the coordinate projections. The sets C:a-1(Oa) fonn the sub-base for the topology, whose hnite intersections yield the base sets. Something to keep in mind in these intinite product spaces is that, if any of X the sets Xct are empty, X i s emp ty. Some of our results are true only for nonempty spaces, so for full rigour the stipulation that elements exist is desirable. 'r
'r
':
,
6.15 Example The space hdv) examined in j5.5 is an uncountable product space having the Tychonoff topology; the unifonn metric is the generalization of the maximum metric p of (6.13).Continuous functions are regarded as close to one another upder dv only if they are close at every point of the domain. The subsequent usefulness of this characterization of C,du) stems mainly from the fact that the coordinate projections are known to be continuous. n Thetwoessentialtheorems onproductspacesextendseparability andcompactness from the factor spaces to the product. The following theorem has a generalization to uncountable products, which we shall not pursue since this is harder to prove, and the countable case is sufticient for our purposes.
6.16 Theorem Finite or countable product spaces are separable under the product topology iff the factor spaces are separable. Proof The proof for tinite products is an easy implication of the countable case, hence consider X* XT=fXf.Let Df fdikndizn... l c Xf be a countable dense set by D each and construct a set i, detining for c =
=
X''O
104
Mathematics
Fm
=
Xojx
i=m+ 1
f=l
fl l
(6. 18)
for m = 1,2,..., and then letting D U;=IF-. Fm is equipotent with the set of m-tuples formed from the elements of the countable DI ,...,Dm, and is countable by induction from 1.4. Hence D is countable, as a countable union of countable sets. We will show that D is dense in X=. Let B = X7=IO/ be a non-empty basic set, with Oi open in niand Of niexcept for a finite number of coordinates. Choose m such that Oi Xf fOr i > m, and then =
=
=
m
B ra Fm
=
=
x X ftl X((?/rno/) f=rn+1
=1
)
y,
?.
(6.19)
l,...,m. Since recalling that the dense property implies Oi fa Di # 0, for B fa Fm B ?'7 D, it follows that B contains a point of D; and since B is an arbitrary basic set, D is dense in X7 as required. w =
One of the most powerful and important results in topology is Tychonoff's theorem, which states that arbitrary products of compact topological spaces are also compact, under the product topology. It will suftice here to prove the result for countable products of metric spaces, and this case can be dealt with using a more elementary and familiar line of argument. lt is not necessary to specify the metrics involved, for we need the spaces to be metric solely to exploit the equivalence of compactness and sequential compactness.
6.17 Theorem A finite or countable product of separable metric spaces (Xf,) is compact under the product topology iff the factor spaces are compact. Proof As before, the finite case follows easily from the countable case, so assume X= = X:=1Xj, where the Xf are separable spaces. In a metric space, which is first countable, compactness implies seprability and is equivalent to sequential compactness by 6.8 and 6.7. Since Di is sequentially compact and tirst-countable, every sequence (xfa,n e INl on Xf has a cluster point xi on the space (6.4)k Applying the diagonal argument of 2.36, there exists a single subsequence of integers, (nk,k E l ), such that xink xi, for every i. Consider the subsequence xw,...). In the product topology, xy in X*, lau, k s N ) where xng @1n:, which proves that X* is sequentlally xo...) iff for i, every xink xi .x (.x1, compact. X= can be endowed with the metric px Z7wl4/zf,which induces the product topology. X= is separable by 6.16, and sequential compactness is equivalent to compactness by 6.8 and 6.7, as above. This proves sufficiency. Necessity follows fzom 6.9(ii), by continuity of the projections as before. w -..-9
=
=
--
--
=
6.18 Example The space R= (see5.14) is endowed with the Tychonoff topology if we take as the base sets of a point the collection .x
N(x.,k,
(y: la)
-yjl
=
< :,
=
1,...,k1., k e N, er > 0.
(6.20)
A point in R* is close to a7 in this topology if many of its coorclinates are close
Topology
to those of x; another point is closer if either more coordinates are within 6: of each other, or the same coordinates are closer than e, or both. The metric Jx defined in (5.15)induces the topology of (6.20).If (xn) is a sequence in X, dxlxnvx) 0 iff V z,k 3 N 1 such that xn e N(x;k,e) for a1l n k N. We already know that R= is separable under Jx (5.15),but now we can deduce this as a purely topological property, since R* inherits separability from R by 6.16. (:1 -->
The intinite cube (0,1j=shares the topology (6.20)with R= and is a compact space by 6.17; to show this we can assign the Euclidean metric to the factor spaces (0,1j. The trick of metrizing a space to establish a topological property is frequently useful, and is one we shall exploit again below.
6.6 Embedding and Metrization Let X be a topological space, and F a class of functions F-y tion p'lzw :: X/rfyis the mapping defined by
exjf
y. The evalua-
J: X F->
xt.
=
(6.21)
The class F may be quite general, but if it were finite we would think of d(x) as the vector whose elements are the J(x),J e F (6.21)could also be written z:yoe = f where ay is the coordinate projection. A minor complication arises because f need not be onto y, and :(X) c Swy is possible. If 4 is a set of points in ly -1(4) may contain points not in e(X). We therefore y, the inverse projection need to express the inverse image of A under f, in tenns of c. as .
:-1(ay-1(4)
f -144) (xyo:)-1(4) =
=
f...j c(x))
c
y.
(6.2:)
The importance of this concept stems from the fact that, under the right conditions, the evaluation map embeds X in the product space generayed by it. It would be holeomorphic to it in the case c(X) Skyy. =
6.19 Theorem Suppose the class F sparates points of X, meaning that fx) # J@) for some f G F whenever x an d y are distinctpoints of X. If X is endowed with the weak topology induced by F, the evaluation map defines an embedding of X into ly J .
.' ,'--f,,' -' ' :.);' .'...' .'-.' .,'
k' r' ::'i;t' lkd ;'y' j' f'.f..
tjd f'
Proof lt has to be shown that e is a 1-1 mapping from X onto a subset of Xro which is continuous with continuou: inverse. Since F separates points of X, e is 1-1, since c@) # :(.y) w he never flx) #(.f@). for some f e F. To show continuity of -1 X rwheneker A is open in y under the weak e, note first that f (A) is open in topology, and sets of the form ly*(A) i open in X/ywith the product Bltt e (z:y(A)) (ayoc) (A) topology, since the projecons are y yyj ; j lh, under e of sets of the form gs J- (A), so we can conclude that. thte '-' .. ).' .T.'1.). (qgt. .y. .g j(. ..t. ),q:.., yj.).yt) ). tyi,j:lt. . .. . y'yt,jgyy,kytt,jyyyryltysy invrfs unions and intersections t; ay-1(4), 4 c :y, are open. since #.-;y.;.L.L.y ( .r (..eryt yrve y (t. (.'( ). ty t . . se sets of Xfy,which are (see 1.2) the same property extrh.d, kygtk..qjlkyrltlqplttit,tt,i ltjj tjju illwl invfj tuj , product topology, and tjo ) r))t.()),qy:,. jjy y Ljjqj.jlqjtjlllq-kjLb. f-iite intersections of these .. . ..; p cztt' yyyt( j))j;)).jjjy)(4tr.)jk;)r...'...))' yyyjyy, y y ,y...'y#)tyv-y..y,' ,.;;.bitiL.) !!F:1. L.)iLj;;Lb'-j)'3';'bb'' 1t1!4,1(E:,. ,y'').%' ,)'..; .y'' ftt. ,:... jjj7td-jjj,1!2421,:'(E:,..ki:r t::' )8,<-..,s.. ..''.y)1I)k-jyjj)'. &.-,. llr.. 154, 421,: d:::: 1ki::1l; :1Er' .kE,: l.rlijjif..' :El2 #:#*'f,ty.-')j.A.yy .rt;...'.7...,tj;-rtr 1lr1. d:E:,h 11. 11. llrl. .'. (.).,j.y,tty. gllr)p %.. ( g .. . jyyy. 1l; 1 :55;4:2::4:::,,llr11l2417:1. tjjb1'3+. )'t*..111!111k!!1-.. k(!r7qp).... $44)14. 4::2ii.'.......11#)i)... #I!2l-y.. jiii 154) ll:ll.i.1Ir71L 1ki!: 4k12: 4:::,, 11.21. .... 1l: ql::)p );.r'#..'y)yy: . ...k' 1bd;i.lq- 4:y# . p.. L3ffL'. ;EE1g (1(2..-, j:''yt,.;Ayz,t., ii. 11.,1. #.:, ()iiL' - ii. . . ''. ; 'lri
Lkwise
qtjt
,$
'tihuoj.
,
,
y
q
yk
y
,
g
,,k;
.'.
:
..
...'...'j;.ji(...
:.i:t
,, '
.
..)
=
,
tyrtj., .;i
..
=
;
,ytl,ii1.
-.)..
.:.
E...
.
.
..
.
..
.,p,y,
..
..
.'
( jy)t,,.E.j
(II'E .E!' '.j2.t(). .:E. .. . :. ,,/'::t Eyyyhtttyyjt.ttjlyyyyyyyyjyyjrjyl,y.yyyt'jtb,t,,.,) ..
..
.
(..))t
.
.
,
,))
,yttytyytyy;y
,
,y(
.:/E.:,tytlr.-)()jtEj4jjkjp.. .Ei...E -jy ;;y.yjE(j..)y.....y''E,#.
''.-.';'.7ti1ir7.
'''1.j.
..
.
4:
..'
... . ..
,
jj riEjptl-l,
'-('
,-t.k.
....
yja
.ij'
.
-
.
..
...t..
.'....TiTi.tj;)))1t-? -yy.'':;)'r.y(..y)j-.,$y.,',.y,yr.;..,-yj:..,.-p.. .
-.7
-:
...:
-,
.
.
.
E : -:-
-:
. '..
.
-
.: !.
;.
-ki. . . . . . . . .. ..
-.
,.
-,.
E .?).!-
,. ..
. .
.. -
.
-
.
.
Mathematics
106
set of the fomf -1(A) where A is open in y. Since F defines the topology on x we know this set to be open, and the finite intersections of such sets form a base for X by assumption. Since e is 1-1 and e -1 a mapp ing, it will suffice to verify that their images under e are open in e(X). Noting that B is a set of the type shown in (6.22),eB) ay-1(A) r''hd(x), but since ly-1(A) is open, eB) is open in '(x) as required. . ,
=
The following is the (forus)mostimportant case of Urysohn
's
embedding theorem.
6.20 Theorem A second-countable F4-space (X,5) can be embedded in (0,1)*.(:I The proof requires
the sufticiency
part of the following lemma.
6.21 Lemma Let x e X, and 1et O X be any open set containing x. 1ff X is a regular space, there exists an open set U with e c 0. -t7
.x
Proof Let X be regular. If O is open nd x e 0, there exist disjoint open sets U OC CC CC 0. Since & by the disand C such that e U and c C, and hence CC closed, Cc is jo intness and we have V ? OC g O and sufficiency is proved. To necessity, is a closed set not containing x, and suppose x e U and Vc 0. prove OC where U and 'P are disjoint open sets. Hence X is regular. w c .x
,
-t7'
Since the space is T4 it is F? and Proof of 6.20 Let V be a countable base for hence reglar. For any x e X and B e V containing x, we have by 6.21 a U e such that G U c #, and also by detinition of a base 3 A e V with e A c U c V, and hence x e X'c V c B. (Xis the smallest closed set containing A, note.) Since V is countable, the collection of all such pairs, say .
'1
.z7
.z'
.4
=
((A,#): A e F, B e F;
X'c BI,
(6.23)
is countable, and so we can label its elements A,B4i (Aj,Sj), i = 1,2,... Every x e X lies in Aj fr some i e IN. Since the space is normal, we have by Urysohn's lemma a separating function : X F-+ (0,IJ for each element of .W,such that fixi) 1 and hB 0. For each a' e X and closed set C such that # C, choose (Aj,#Jsuch that x e Xic Bi c Cc, 0. These separating functions form a countable and then h.@) 1 and ficj class F, a subset of U(X). Since the space is F1, C can be a point (yl so that F separates points. And since the space is F3j and hence completely regular, is the weak topology induced by F, by 6.14. It follows by 6.19 that the evaluation map for F embeds X into (0,1)*.. =
=
=
.x
=
=
':
IJ= endowed with the metric px defined in (5.19)is a compact r0, metric space. lt follows that :(X), which is homeomorphic to X under the evalRecall that
uation mapping by F, is a totally bounded metric space. lt further follows that is metrizable, since, among other possibilities, it can be endowed with the and y of X is taken to be metric under which the distance between points have metrization theorem, 6.12. We Urysohn px(c@),c@)). therefore proved the The topology induced by this metric on g0,11= is the Tychonoff topology. A base for a point p e g0,1)*in this topology is provided by sets of the (p1,p2,...) (X,I)
.x
=
Topology
107
fonn Nlp-nkte.t
=
14./e l0,11*: I#/ -
qi I
i
< E,
=
(6.24)
1,...,/c),
for some finite k, and 0 < 6: < 1, which is the same as (6.20).The topology induced on X by the embedding is accordingly generated by the base sets Nx,.kte)
=
ly e
I @) fily)I <
X:
-
:,
i
=
1,...,/c1,
(6.25)
which can be recognized as finite intersections of the inverse images, under functions from F, of e-neighbourhoods of R; this is indeed the weak topology induced by F. This further serves to remind us of the close link between product topologies and weak topologies. Since metric spaces are F4, separable metric spaces can be embedded in (0,lj= by 6.20. In this case the motivation is not metrization, but usually compactitication that is, to show that separable spaces can be topologized as totally bounded spaces. Both metrization and compactification are techniques with important applications in the theory of weak convergence, which we study in Chapter 26. Although the following theorem is a straightforward corollary of 6.20, the result is of suftkient interest to deserve its own proof; the main interest is to see how in metric spaces there always exists a ready-made collection of functions to define the weak topology. -
A separable metric space (S,#) is homeomorphic to a subset of
6.22 Theorem
(0,11*.
Proof Let Jo J/(1 + #), which satisfies 0 f Jo f 1 and is equivalent to d, so that (X,A)is homeomorphic to (X,#). By separability there exists a countable set of points (zf,i e EN) which is dense in X. Let a countable family of functions be 1,2,..., and define an evaluation map : X F-, defined by fix) dx,zi), i =
=
=
x jj y g0,11
hx)
(%(mz1),J0(x,z2),...). where px(,#) that h is an embedding in (g0,1J=,px)
(6.26)
=
We show 7=1Ihk #kl llk. lf (x,,) is a squence in X converging to x, then for each k, dolxntz 4(x,zk)k Accordingly, V k,z H N 2 1 such that xn e Nxik,e) for a11n k N, poohxnlvhxjj --> 0, and h is continuous at x. On th other hand, if xn zz...hx, there exists : > 0 such that Y N k 1, djxntx) 2 6: for some n k N. Since (z:) is dense in X, there is and J0(mzk) < doxvz I 2 so that 1doxnnz a k for which doxnnz k 1: and hence =
-
--
Y
px((x,,),(x)) 2
/,
/2
1+1
-
(6.27)
k
Since this holds for some n k N for every N k 1, it holds for infinltely many n, and hxn) zz-y (x). We have therefore shown that hxn) (x) if and only (Jxn --).x. This is the property of a 1-1 contingous function with cntinuous inverse. . --
But note too the alternative approach of transforming th distance functions into ting functions as in (6t10),@nd applying6.20.
separa
11 PROBABILITY
7 Probability Spaces
Probability Measures A random experiment is an action or observation whose outcome is uncertain in advance of its occurrence. Tosses of a coin, spins of a roulette wheel, and observations of the price of a stock are familiar examples. A probability space, the triple (D,1,#), is to be thought of as a mathematical model of a random experiment. f is the sample space, the set of all the possible outcomes of the The collection 5 experiment, called the random elements, individually denoted of random events is a c-field of subsets of f1, the event A e 5 being said to have occurred if the outcome of the experiment is an element of A. A measure # is assigned to the elements of 5, PA) being the probability of A. Fonually, we have the following. .
7.1 Definition A probability measure (p.m.)on a measurable space (f1,1)is a set function #: 5 F--> (0,1j satisfying the axioms of probability (a) #(A) k 0, for a11:4 e T. 1. (b) #(D) (c) Countable additivity: for a disjoint collection tz4j e @,j e IN) =
,
#
Uz4yTPAj).
n
=
Under the frequentist intepretation of probability, #(A) is the limiting case of theproportion of a long run of repeated experimnts in which the outcome is in 4. Alternatively, probability may be viewed as a subjective notion with #(A) said to represent n observer's degree of belief that A will occur. For present purposes, the intrpretation given to the probabilities has no relevance. The theory stands or falls by its mathematical consisteny alone, although it is then up to us to decidewhether the results acord with ur intuition and are useful in the analysis of real-world problems. Additional properties of P follow i om the axioms.
7.2 Theorem lf A, #, and (, j e Eq) am afbitrary T-sets, then (i) #(X) f 1. #(AC) 1 #(A). (ii) t E 0. (iii) PZ) ytiitity). (iv) A c # = #(A) f #() tpij lty, yy, y Ir$r7.y7,#kk. yyE ');.(.y.-- ..ty .... ... ...;. .. 4(:(y.-t,d$p1k-2. .)-.(21,q.y.. .. 44:4)..j!!E111, L-i4qii.. jit' ..ty..-. (v)(A w B) = pA) + .ji!!:::,' y )j)t yq j y,, . . . yy ,y jg tr j, ju j yy, jyky y kjy (vi) PojAj) f L)PAj) (ctllitAb. tyt,tt j j,( y y rytyrjjtyry =
-
'
=
i
,
-t.(..-...y...ty-.t
(2)jI.
..))
.jj!.::;i'
-. .
..-...k.
....
..
-'
...tE.y..-
...
.
-
.
.
,jj
.
$jjy
,tjjyyytyyyylyyy,yyqyjyj.ytlyyttyyyyyy,ly
...),
.. .
..
y...
,y
,
y,y ,
..
112
Probability
(vii) Aj
'r'
t, A
A or Aj
PAj) .-A #(A)
(continuity).cl
Most of these are properties of measures in general. The complementation property (ii) is special to #, although an analogous condition holds for any finite measure, with #(X replacing 1 in the formula. (iii) confinns P is a measure, on the definition. Proof Applying 7.1(a), P(A)
+
(b), and (c), PAc)
PA t.JAc)
=
#(f)
=
1,
=
from which follow (i) and (ii), and also (iii) on setting A by 3.3, and (vii) by 3.4. w
=
(.
(iv)-(vi)follow
To create a prbability space, probabilities are assigned to a basic class of according C, events to a hypothesis about the mechanisms underlying the random outcome. For example, in coin or die tossing experiments we have the usual hypothesis of a fair coin or die, and hence of equally likely outcomes. Then, provided C is rich enough to be a determining class for the space, (f,@,#) exists by 3.8
(extension theorem) where ?F = c(C).
7.3 Example
Let Sp,1)
=
(# fn g0,1), #
e f), where T is the Borel field on R.
Then ((0,1), fm,1J, m), where m is Lebesgue measure, is a probability space, 1. The random elements of this space are real numbers between 0 since ,7:(:0,1)) and 1, an a drawing from the distribution is called a random variable. It is said to be distributed rfnt/r/zl/y on the unit interval. The inclusion or exclusion of the endpoints is optional, remembering that /1((0, 1J) r?2440,1)) = 1. n =
=
The atoms of a p.m. are the outcomes (singletonsets of ) that have positive probability. The following is true for tinite measures generally but has special importance in the theory of distributions.
7.4 Theorem The atoms of a p.m. are at most countable. Proof Let l be an atom satisfying P((1 J) #(( J)for al1 (0 6 f1, 1et (t'/2 satisfy #(f tl)2)) k #4(1) for all e f(t)1 ), and so forth, to generate a sequence with -
#(l1l)
2 #(tt%l)
2 #(t2l)
...
(7.3)
The partial sums Z7=1#(fj)) form a monotone sequence which cannot exceed #(D) 1, and therefore converges by 2.11, implying by 2.25 that limn-hxftf.t)a 1) 0. All points with positive probability are therefore in the countable set f(, i e Nl . =
=
.
Suppose a random experimentrepresented by the space (fk9,#) is modified so as to confine the possible outcomes to a subset of the sample space, say A c f1. For example, suppose we switch from playing roulette with a wheel having a zero slot to one without. The restricted probability space is derived as follows. Let 5% denote the collection fEcj A, E e @) 5%is a c-tield (compare1.23) called 5h and #(m/#(A) for E e Fa, Ph can be is called the trace of % on A. Defining #a() (A,Fa,#A) is called the trace of (f1,F,#) on A. triple verified to be a p.m. .
=
'lhe
Probability Spaces This is similar to the restriction of a measure space to a subspace, except that the measure is renormalized so that it remains a p.m. In everyday language, we are inclined to say that events may be or certain' lf such events are none the less elements of @, and hence technically by random, we convey the idea that they will occur or not occur with here assigning them probabilities of zero or one. The usage of the tenn marks is deliberately loose, as the quotation suggest. To say an event cannot it has probability different from saying it cannot occur because is zero occur because the outcomes it contains are not elements of f1. Similarly, to say an event has probability 1 is different from saying it is the event f1. In technical discussion we therefore make the nice distinction between sure, which means the latter, and almost sure, which means the former. An event E is said to occur almostsurely fl E has prob(a.s.), or equivalently, with probability one (w.p.1) if M abilitymeasurezero.This terminologyis synonymouswithlfv/l/eveqwercta.e.) in the measure-theoretic context. When there is ambiguity about the p.m. being considered, the notation may be uded. Aimpossible'
.
tcertainty' tcertain'
=
-
4a.s.(#j'
7.2 Conditional Probability A central issue of probability is the treatment of relationships. When random experiments generate a multi-dimensioned outcome (e.g.a poker deal generates several different hands) questions always arise about relationships between the different aspects of the experiment. The natural way to pose such questions is:
0. #(. IA) satisfies the event B as #(#IA) probability axioms as long as P does and #(A) > 0. In particular, #(A IA) 1, and P(Bc IA) 1 PB lA), since B /-74 and Bc DA are disjoint, and their union is 4. The space (A,@x,fk), the trace of the set A on (D,T,#), models the random experiment from the point of view of an observer who knows that (l) e A. Events A and # are said to be dependent when PB IA) # #(#). In certain respects the conditioning concept seems a little improper. A context in which the components of the random outcome are revealed sequentially to an observer might appear relevant only to a subjective interpretation of prpbability, and lead a sceptical reader to call the neutrality of the mathematical theory ipto question. We might also protest that a random event is random, and hak no business Gthe
=
=
=
-
Probability
114
defining a probability space. ln practice, the applications of conditional probability in limit theory are usually quite remote from any considerations of subjectivity, but there is a serious point here, which is the difficulty of constructing a rigorous theory once we depart from the restricted goal of predicting random outcomes a priori. The way we can overcome improprieties of this kind, and obtain a much more powerful theol'y into the bargain, is to condition on a class of events, a c-subfield of 9. Given an event B e F, let the set function F-> (0,11 /74#1 ): '
represent the contingent probability to be assigned to B after drawing an event A from N, where V i @.We can think of V'as an infonuation set in the sense that, for each A G V, an observer knows whether or not the outcome is in A. Since the elements of the domain are random events, we must think of #(# lN) as itself a random outcome (arandom variable, in the terminology of Chapter 8) derived from the restricted probability space (f1,V,#). We may think of this space as a model of the action of an observer possessing information V, who assigns the conditional probability #(# lA) to B when be observes the occurrence of A, viewed from the standpoint of another observer who has no prior infonnation. N is a c-tield, because if we know an outcome is in A we also know it is not inAc, and if we know whether or not it is in Aj for each j 1,2,3,..., we know whether or not it is in U#.j. The more sets there are in V the larger the volume of information, all the way from the trivial set 5 (f,0) (completeignorance, with 8#1 T) #(#) a.s.) to the set 5 itself, which orresponds to almost sure knowledge of the outcome. In the latter case, P@ 11) 1 a.s. if ( e B, and 0 otherwise. lf you know whether or not (J) G A for every A e @, you eftctively know (J). 8 =
=
=
=
7.3 Independence A pair of events A, B e 5 is said to be independent if #(A ch#) equivalently, if PB jA)
=
#(#).
=
#(A)#(#), or,
(7.4)
lf, in a collection of events C, (7.4)holds for every pair of distinct sets A and B from the cotlection, C is said to be pairwise indepenttent. In addition, C is said to be totally independent if (.D-,A)
#
-
J,'tA'
(7.5)
containing two or more events. This is a stronger condition for every subset # i thanpairwise independence. Suppose Cconsists of setsA, #, and C. Knowing thatS has occurred may not influence the probability we attach to A, and similarly for C) but thejoint occunrnce of B and C may none the less imply something about A. Pairwiseindependence implies thatfA fa B) =PIAIPCBI,PCA rn C) =#(A)#(C), and PB rn C' PB)PC4, but total independence would also require P(A f'n B rn C) = =
P(A)PB)PC).
Probability Spaces
115
Here are two useful facts about independent events. In each theorem let C be a totally independent collection, satisfying ptf)xs .gA) l7ze.y#(A) for each subset =
#' I C. 7.5 Theorem The collection C' which contains independent.
yt
and Ac for each A e C is totally
Proof It is sufficient to prove that the independence of A and B implies that of Ac and B, for B can denote any arbitrary intersection of sets from the collection AC. This is certainly true, since if and (7.5) will be satisfied, for either A or #(A)#(#), then #(A f''h B) =
PCAC
r'A #)
=
PAc ra #)
= #(#)
-
+
P(A fa #)
P(A4PB4
-
#(A)#(#)
PAPB4.
=
(7.6)
w
7.6 Theorem Let () be a countable disjoint collection, and let t e collections consisting of Bj and the sets of C be totally independent for each-j. Then, if B LjBj, the collection consisting of B and C is als independent.
=
Proof Let #' be any subset of C. Using the disjointness of the sr-ts of B, and
countable additivity, P
j
rn
)
xOs ,A
-
P
sA)
zOs ) (#?
(Uy Bj ra ,A xOs XyP -
=Xy
PCB?P
7.4 Product Spaces
sA)
tj
=
Plhj
ra
,PA4.
.
and independence arise when multiple random eyperiments run in parallel, and product spaces play a natural role in the analysis of these issues. Let (f x E, 5 @ V, #) be a probability space where 5 (*)V is the
Questionsof dependence
c-fieldgenerated by the measurable rectangles of (-1x E, and PL x E) 1. The randomoutcome is a pair (,4). This is no more than a case of the general theory of j7.1 (wherethe nature of (l) is unspecified) except that it becomes possible to askquestions about the part of the outcome represented by ) or ( alone. #04/7 =
=
#(Fx E) for F e F, and #E(G) Pl x G4 for G e N, are called the marginal probabilities.(1,1,#0) and (E,T,#E) are probability spaces representing an incompletelyobserved random experiment, with ) or k, respectively, being the only things observed in an experiment generating (,(). On the other hand, suppose we observe ( and subsequently consider the experiment'of observing (t). Knowingtmeans thatforeachf x Gweknowwhetherornot (,() is in it. The conditional probabilities generated by tlzis two-stage experimentcan be wr itten by a slight abuse of notation as #(Fl V), although strictly Fx E, and the elemts of the speakingthe re 1evan t events are the cylinders uzrln enHzw+lxlv.lllv,qia 'n uzn n'zala' rmfiold nre 0./ G for G G W. Cnn ditinnine =
Probability
116
#(Fx E lfl x T). ln this context, product measure assumes a special role as the model of independence. In (f x E, 5 (& V, #), the coordinate spaces are said to be independent when PFx
G)
=
#0(#)#s(G)
for each F e 5 and G e T.
Unity of the notation is preserved since Fx G (Fx E) fa (( x G). We can also write #(Fx EIf x G) /704/-), or with a further slight abuse of notation #(FIG) #n(/'), for any pair Fq 5 and G e N. Independence means that knowing ( does not affect the probabilities assigned to sets of ;. Since the measurable rectangles are a determining class for the space, the p.m. P is entirely detennined by the =
=
marginal measures.
=
8 Random Variables
8. 1 Measures on the Line a probability space. A real random variable (r.v.) is an 5lBf'unction X: ( F.-: R 9 That is to say, aY4)) induces an inverse mapping measurable from B to 5 such that X -1(#) l 5 for every B e B where B is the linear Borel field. The term T-measurable r.v.' may be used when the role of B is understood. The symbol g. will be generally used to denote a p.m. on the line, reserving P for the p.m. on the underlying space. Random variables therefore live in the space (R,S,g,), where g. is the derived measure such that g(#) PX -1(#)) = #( X e B) The term distribution is synonymous with measure in this context. The properties of nv.s are special cases of the results in Chapter 3; in particular, the contents of j3.6 should be reviewed = in conjunction with this chapter. If g: R F-> R is-1a Borel function, then gox is also a r.v., having derived p.m. g,: according to 3.21. p(A)) If there is a set S e B having the property g(& 1, the trace of (R,S,g) on S is equivalent to the original space in the sense that the same measure is assigned to B and to B r''hS, for each # e S. Which space to work with is basically a matter of technical convenience. If X is a r.v., it may be more satisfactory to say that the Borel function X 2 is a r v. distributed on R+ than that it is distributed on R but takes values in R+ almost surely. One could substitute for (M,S,g,) the extended space (F,S,g) (see 1.22), but note that assigning a positive probability to infinity does not lead to meaningf'ul results. Random variables must be finite with probability 1. Thus (R,f,g), the trace of (F,S,g) on R, is equivalent to it for nearly a1l purposes. However, while it is always finite a.s., a r.v. is not necessarily bounded a.s.; there may exist no constant # such that 1X(l I f B for a1l (l) e C, with #(D C4 0. The essential supremum of X is
Let
(f1,1,#) be
.
,
=
.
=
,
.
=
-
ess sup X
=
inf (x: #41XI > x)
and this may be either a finite number, or
0),
=
+x.
8.2 Distribution Functions The cumulative distribution
@.b.f.)Of
f'Ifrlcft??z
X is the function F:
where
2- F-> g0,1J,
< xj, x s F. (8.2) l ( ( fty/, j'q..(( gn the values 0 and 1 to )( jj:t.:.' tk..j.,.((y. we take the domain to be R-since it' iq.(... (('..t'jr(t.t:.....t( .. .. . y)':)j)t..: .i-) ... (j).jL?, f;. ji,j.. . F(x)
=
g((-x, x1)
x
=
.
.
..
.
.
:
'
:
:.)'
.-
..'.''
....
.
..)
.... .
.
'
'
.t.
.
.:$j.i.
(Ej
.
.
....j
... y.. q .))(-.3E,l'( y j.y)y(y)jgE.t..-'(.:,.t.yy.. .t
...yr.j(,?:jyj.jy.
,,;.j..j;...(..j.
..
.
-(.y.'.,
.
. .
..
.
.
.
.
c
..
.
r)().... .. .. . y.....) ,.E.
. .
.
Probability
118
F(-x) and F(+x) respectively. No other values are possible so there is no contradiction in contining attention to just the points of R. To specify a distribution for X it is sufticient to assign a functional form for F; p, and F are equivalent representations of the distribution, each useful for different purposes. To represent g,(A) in terms of F for a set A much more complicated than an interval would be cumbersome, but on the other hand, the graph of Fis an appealing way to display the characteristics of the distribution. To see how probabilities are assigned to sets using F, start with the half-open interval (x,y) for x < y. This is the intersection of the half-lines (x,y) and (-x, (.x,+x).Let A (-=, x) and B = (-x,y), so that g(A) F@) and g(#) F(y); then -xzf
=
=
BIAC
g,(@,)/1) =
=
ra #)
= 1 (g(A) -
1 p,tA
=
+
-
1 g(#))
tJ#C)
g(#)
=
-
=
-
g(z4)
=
F@) F@), -
A and Bc being disjoint. The half-open intervals form a semi-ring (see1.18), and from the results of j3.2 the measure extends uniquely to the sets of f. a) for x c R As an example of the extension, we detennine g,((x)) PX will (compare 3.15). Putting a7 y in (8.3) not yield this result, since A cjAc = 0, not (xl We could obtain fxl as the intersection of (-x, x) and (-x, xlc, but then there is no obvious way to tind the probability for the open interval (-x, x) (-x, xl f.xl The solution to the problem is to consider the monotone sequence of half-lines (-x, x 1/n) for n e N. Since @ 1/p, (-x, aj, 1/n), according have 1/n,<) F(x) lln Fx ra g,(@ to we (-x, x = and where 1/n, x), Fx) F@-), Since T g,((x)) (8.3). (a7Je fx) 0':nltx F@-) is the left limit of F at x. Fx) exceeds Fx-j (i.e. F jumps) at the atoms of the distribution, points with g,((xJ)> 0. We can deduce by the same kind of F(y-4 Fx), g((x,y)) = F@-) F(x-), and that, reasoning that p,(@,y)) generally, measures of open intervals are the same as those of closed intervals unless the endpoints are atoms of the distribution. Certain characteristics imposed on the c.d.f. by its definition in tenns of a measure were implicit in the above conclusions. The next three theorems establish these properties. =
=
=
=
rx,+x)
.
=
-
.
.:71
-
=
-
=
=
-
-
-
-
-
-
.z:
=
-
-
1, 8.1 Theorem F is non-negative and non-decreasing, with F(-x) 0 and F(+x) and is increasing at e R iff every open neighbourhood of x has positive measure. =
=
.x
Proof These are all direct consequences of the definition. Non-negativity is from (8r2), and monotonicity from 7.2(iv). F is increasing at x if Fx + :) > F@ E) for each E > 0. To show the asserted sufficiency, we have for each such E, -
F@ + :)
-
F@ :) k F((x
+ 8)-)
-
For the necessity, suppose g,(5'(x,E))
g,(5'(x,e))= F(@ + e,)-) F@ -
-
=
E)
-
F(x
-
6,)
=
g,(5'@,y)).
(8.4)
0 and note that, by monotonicity of F, k Fx
+ :/2)
-
F@- :/2).
w
The collection of points on which F increases is known as the sunnort of
(8.5) 11
IfA
Random
Variables
119
complement in R, the largest set of zero measure, consists of points that must a11 liein open neighbourhoods of zero measure, and hencemustbe open.-rhe supportof g, is accordingly a closed set. F is right-continuous
8.2 Theorem Proof For x
l
everywhere.
R and n 2 1, additivity of the p.m. implies
g((-x,
.v
+
1/nJ) g,((-x, =
x1)
+
g,(@,x+ 1/n1).
(8.6)
g,(4-x, x+ 1/aJ)1, g.((-x, x1) by continuity of the measure, and hence limn-yxjtttx, + 1/n1) 0. lt follows that for er > 0 there exists Ne such that g,(@,x + 1/nJ) < E, and, accordingly, As n
.-.+
x,
.v
P,((-=,
=
.Y1)
f t((-=,
for n k Ne. Hence F(x+)
=
x+ 1/n1)
<
p,tt-x, x1) +
(8.7)
E,
F(x), proving the theorem since x was arbitrary.
.
If F@) had been defined as g,((-x, x)), similar arguments would show that it was left ontinuous in that case.
'W
Fig. 8.1
8.3 Theorem F has the decomposition F(x)
whereF'@) jumps, and
=
F'x)
+
F'x)
(8.8)
is a right-continuous step function with at most a countable number of r'(x) is everywhere continuous.
Proof By 7.4, the jump points of F are at most countable. Letting
denote these
points,
f-x1,x2, ...)
Probability
120 F'@)
=
X xisx
Fxi,
Fxi-j)
-
(8.9)
is a step function with jumps at the points xi, and F'x)
F(xi-)
=
Flxi) at each xi and is continuous everywhere.
Fx)
=
-
F'x)
has
*
Fig. 8. 1 illustrates the decomposition. This is not the only decomposition of F. The Lebesgue decomposition of g with respect to Lebesgue measure on R (see4.28) is g = p,1+ g.2where g,l is singular with respect to m (is positive only on a set of Lebesgue measure 0) and g2 is absolutely continuous with respect to Lebesgue measure. Recall that g2(A) = fxjdxo A e B where Tis the associated Radon-Nikodym derivative (density function). If we decompose F in the same way, such that Ff(x) w((-x,x1) for i = JQ.f(()J), implying that f@) dFz /#( I(=x.This = 1 and 2, we may write Flx) must hold for almost all x (Lebesgue measure), and we call F2 an absolutely continuous f'unction,meaning it is differentiable almost everywhere on its domain. F' S F1 since F1 may increase on a set of Lebesgue measure 0, and such sets can be uncountable, and hence larger than the set of atoms. It is customary to summarize these relations by decomposing F' into two additive components, the absolutely continuous part &, and a component F3 = F'- F2 which is continuous and also singular, constantexcept on a setof zero Lebesgue measure. This componentcan in most cases be neglected. The collection of half-lines with rational endpoints generates B (1.21) and should be a determining class for measures on @,f). The following theorem establishes the fact that a c.d.f. defined on a dense subset of R is a unique representation of g,. ,
=
=
8.4 Theorem Let z be a tinite measure on (R,f) and D a dense subset of R. The function G defined by 81)
Gx)
=
IX(-=,-Y1),
e D
.<
=
x e R D
F(x+),
-
(8.10)
is identical with F. Proof By definition, R q X'and the points of R D are a11closure points of D. For each x e R, not excluding points in R D, there is a sequence of points in D converging to x (e.g. choose a point from Sx,lln) r'AD for n g N). Since F is F(x+) for each x e R -D. w right-continuous everywhere on R, p,((-x,x)) -
-
=
Finally, we show that every F corresponds
to some g,, as well as every g to an F.
8.5 Theorem Let F: V --A E0,1) be a non-negative, non-decreasing, right=0 and F(+x) 1. There exists a uniquep.m. g on continuous function, with F(-x) (R,S) sqch that Fx) g((-x,x1) for a11a7 G R. u =
=
Right continuity, as noted above, correspopds to the convention of defining F by (8.2). If instead we defined F@) = g((-x,x)), a left-continuous non-decreasing F
Random Variables
121
would represent a p.m.
#: (0,11F- 2', defined
Proof Consider the function (4If)
=
inf (x: u < F(x) )
by
(8. 11)
.
4)can be thought of as the inverse of F; (40) (41) +x, and since F is nonIight-continuous, decreasing and # is non-decreasing and left-continuous; # is Borel-measurable by According to 3.21, we may define a meas3.32(ii). therefore ?n4 -1 (#) for each B e f, where m is Lebesgue measure on theBorel ure on (R,f) by of sets (0,11. ln particular, consider the class C of the half-open intervals (J,:1 for a11a,b e R with a < b. This is a semi-ring by 1.18, and c(C) B by 1.21. Note that t)-1((J, 4) (/g:inf (x: u K F(x)) e (tz,h1 ) (F((7),Fblj. (8.12) =
-x,
=
=
=
=
For each of these sets define the measure P,((J,#1)
=
/19
-1((J, :j))
=
Fb)
-
(8.13)
Fa).
The fact that this is a measure follows from the argument of the preceding paragraph. C is a determining class for (R,O), and the measure has an extension by 3.8. It is a p.m. since g.(R) 1, and is unique by 3.13. x =
The neat construction used in this proof has othef applications in the theory of random variables, and will reappear in more elaborate form in 922.2.The g'raph of () is found by rotating and reflecting the graph of F, sketched in Fig. 8.2; to see the fonner with the usual coordinates, turn the page on its side and view in a
mirror.
Fig. 8.2 If F has a discontinuity at x, then 4) x on the interval (F@-), F(x)1, and -1(lxl) = (F(x-), F(x)j. Thus, g,((x)) -((F(x-), F@)1) F(x) F(x-), as 41 required.On the other hand, if an interval (J,!4 has measure 0 under F, F is =
=
=
-
Probability
122
F(b) = c (say).4) constant on this interval and # has a discontinuity at Fa) -1 takes the value a at this point, by left continuity. Note that () (c) = (J,>1, so that g,((J,:J) mc) = 0, as require. =
=
8.3 Examples Most of the distributions met with in practice are either discrete or continuous. A discrete distribution assigns zero probability to a11 but a countable set of points, with F' = 0 in te decomposition of 8.3.
8.6 Example The Bemoulli (orbinal'y) nv. takes values 1 and 0 with fixed probaThink of it as a mapping from any probability space containbilities p and 1 elements. and tFailure'
isuccess'
;
1
8.7 Example The binomial distribution with parameters n and p (denotedBn,pl) is the disibution of the number of ls obtained in n independent drawings from the Bernoulli distribution, having the probability function P(X
=
x)
4
=
x
p
x
(1
n-x
-p)
x
,
=
8.8 Example The limiting case of (8.14) with p distribution, having probability function P(X
=
x)
-w
cp
1
0,...,p.
hln. as
=
(8.14)
Ia
n
-..-y
x,
is the Poisson
-h,
.
X'.
x ,
x
=
() j ,
,
z
(8.15)
,...
This is a discrete distribution with a countably infinite set of outcomes.
D
In a continuous distribution, F is absolutely continuous with FI = 0 in the Lebesgue decomposition of the c.d.f. The derivative f #F/#x exists a.e.(>1 on R and is called the probability densit.v function(p.d.f.)of th p.m. According to the Radon-Nikodym theorem, the p.d.f. has the property that for each E e f, =
g,(f)
=
jyflvdx.
,
(S' 16)
8.9 Example For the unkf'orm distribution on g0,11 (see 7.3), 0, x F@)
<
x, 0 f
=
1, x
>
0 .x
S 1.
(8.17)
1
The p.d.f is constant at 1 on the interval, but is undefined at 0 and 1.
E1
8.10 Example The standard normal or Gaussian distribution has p.d.f. f(x)
=
1
/-2a
-x2/2
e
-x
'
<
x
< +.=
,
(8.18)
Random Variables
whose graph is the well-known
123
bell-shaped curve with mode at zero. n
8.11 Example The Cauchy distribution has p.d.f.
fxq
1
=
<
-cx,
,
+x2)
zr(1
which, like the Gaussian, is symmetric
x
(8.19)
< +x,
with mode at 0. n
When it exists, the p.d.f. is the usual means of characterizing a distribution. A particularly useful trick is to be able to derive the distribution of gzk' from that of X, when g is a function of a suitable type.
8.12 Theorem Let #: S F--> 1- be a 1-1 function onto 7, where S and 1- are open subsets of R, and let h g -1 be con tinuously differentiable with dhldy # 0 for all y e 7. lf X is continuously distributed with p.d.f. fx, and F #(.X), then F is continuously distributed with p.d.f. =
=
),I
.fx(,-1(,))
s(,) -
.
(8.20)
u
The proof is an easy exercise in differential calculus. This result illustl-ates 3.21, but in most other cases it is a great deal harder than this to derive a closed form for a transformed distribution.
8.13 Example Generalize the uniform distribution (8.9)from (0,1) to an arbitrary interval gtz,:1.The transformation is linear, y so that fyy)
=
b a) -1 on
a
=
(J,:),
-
F@)
b
+
by
(.y
=
-
(8.21)
Jlx,
-
(8.20).The
allb
c.d.f. is d e jjxd
on
a).
-
gc,)j by (8.22)
Membership of the uniform family is denoted by X
-
&(c,:1. n
8.14 Exapple Linear transformations of the standard Gaussian r.v., X
=
jt + cZ,
c
>
0,
(8.23)
generate the Gaussian family of distributions, with p.d.f.s
1 e fx'' g.,0-) /--2ac
-tx-jtlz/acz
'-=
=
,
<
.7
< +=
.
(:I
(8.24)
have better-known designatiops as The location parameter ytand scaleparameter and 9.7 9.4 below. distribution; Membership of the Gaussian of see moments the 2 family is denoted by X #(g,,c ). -
A family of Cauchy distributions is generated from the standard Cauchy by linear transformations X v + Z, > 0. ne family of p.d.f.s with locationparameter v and scale parnmeter take the form
8.15 Example
=
Probability
k=N
J@;v,)
1
1
=
k''g1+
r(x -
<
-=
,
v)/) z
< +x.
.z7
(8.25)
n
8.16 Example Consider the square of a standard Gaussian nv. wit p.= 0 and c = 1. Since the transfonnation is not monotone we cannot use 8.12 to detennine the density, density, strictly speaking. But consider the Khalf-normal'
zlt&l =
2J(ff), u
k 0
0,
<
u
(8.26)
0
where f is given by (8.18).This is the p.d.f. of the absolute value of a Gaussian variable. The transformation g Iu I) = u 2 is 1-1 so the p.d.f. of Z2 is ,
1 e fz24&) W2V
-1/2
-un
u
=
applying(8.20).This is the ch-squared or z241).It is a member (withp (x =
G(?z; ',pj W
herej-+4
=
f!
() < u
,
=
and 1R?zl (n llF(n
-
(8.27)
,
-1 -ea
((xI/)P
vpe
-
.
distribution with one degree of freedom, ) of the gnmma family,
,
0
<
u
< x,
is the gammafunction, J7(P-1e-Q(
=
=
<
.
a
>
0, p
>
0,
(8.28)
having the properties F()
=
W2'V,
1). c!
8.4 Multivariate Distributions In Euclidean k-space R k the k-dimensional Borel field Tk is c(S$ where Rk denotes the measurable rectangles of Rk th e se ts of the form #1 x B1 x x Bk where Sf e B for f = 1,...,k. In a space (D,@,#), a random vector (msXi,...,.V)' = X is a measurable mapping X1 fl Rk ,
,
,
...
.-+
.
lf t is the derived measure such that g,(A) = PE) for A G T k an d E s 5 the mu ltivariate c.d.f., F: Fk (0,1), is defined for (x1,...,x' = x by ,
..-+
F@)
=
g,((-=,x11 x
...
x (-x, aJ).
(8.29)
The extension proceeds much like the scalar case.
8.17 Example Consider the random pair (X,D.Let F(x,y) = g((-x, XJx (-x, y1). The measure of the half-open rectangle (x,x+ Ax1x (z,y +Ay) is M7x, y)
=
F@ + Ax, y + Ay)
-
Fx
+
Ax,y)
-
F@, y + Ay) + Fx, y4.
(8.30)
To show this, consider the four disjoint sets of R 2 illustrated in Fig. 8.3:
Random Variables
125
:4
=
@,x + AXJx @,y + AyJ,
B
=
(-x, x) x (.y,y +Ay),
C
=
(x,x+AXJx (-x, yq,
D
=
(-x,x) x (-x,yj.
A is the set whose probability is sought. Since #(A wB tp Ck.)D) Fx + Ax,y), and #(D) Fx+ Ay + Ay), #(# QJ D) = Fxby + Ay), PCCwD) result probability is immediate from axioms. the Fx, y), the n =
= =
Fig. 8.3 Extending the approach of 8.17 inductively, the measure of the k-dimensional , xi + AQ can be shown to be rectang le Xil@f, AF(I1,...,
A'k)
=
77(:f..6/),
(8.31)
j
where the sum on the right has 2k terms, an d the Fj are the values of F at each of the verices of the k-dimensional rectangle extending from @1,....q)'with sides 1,...,k. The sign pattern depends on k', if k is odd, the Fj of length Mj, i having as arguments even numbers of upper vertices (pointsof the form xi + A'&)) take negative signs, and the others positive; wilile if k is even, the Fj with odd numbers of upper vertices as arguments are negative. Generalizing the monotonicbe ity of the univmiate c.d.f., F must satisfy the condition that AF@1,..., (R)+ non-negative for every chice of (x1,..., ' e R an d (A.x1,..., Axk)' s Applying 3.19 inductively shows that the class of half-open rectangles is a semi-ring, so that the measure defined by F extends to the sets of T k. hence (R Bk is a probability space derived from L1,5,P). If the distribution is continuous with p.d.f. flx), Fubini's iheoremgives =
.q)
.q)
.
-dimensional
,
,g)
,
Fx)
.f((1,...,(z)t:61
.dlk
=
..
(-x,xl1x,.,x(-x,xk)
=
xl
xk ...
J(k1
.,4:2)t2)1
...#(k.
,..
Theorem 8.12 has the following generalization. A dteomorphism
(8.32)
(also,coor-
Probability
126
dinate transformationj is a function g: S F- -T(S and 7 open subsets of Rk) w lli c h # 0 for all x e S, is 1-1 onto and continuously differentiable with etglx') whose element glx' Jacobian matrix is is the gllxi for i,j where (f,.j)th 1,...,1. The inverse of a diffeomorphism is also continuously differentiable. =
8.18 Theorem If F
where g is a diffeomorphism,
g
=
the p.d.f. of F is
h%l flg-1(.:))1yj
(g ag)
=
whereJ
detl:
=
-1
)j
y
'j
.
u
.
This is a standard result in the theory of multiple Lebesgue integrals Apostol 1974: 15.10-15.12).
8.19 Example Letting
denote the standard Gaussian p.d.f.
.f
k
lzi) 1-1
9(z) =
(zzo-Wexpf
=
(see8.10),
(see e.g. consider
-z1(z'z)).
(8.34)
i=1
This is a k-dimensional p.d.f., and the corresponding random vector (z1,...,Zk)', is the standard Gaussian vector. The affine transformation X
AZ+ !z,
=
Z
=
(8.35)
whereA (kx k nonsingular) and M kx 1) are constants, is 1-1 continuous with inversez A -1(X p.) having J IA-1 l 1/lA I Detine E 4A' such that (A -1) x-1 and j j4 j j jEI the positive square root being ( nderstood.Applying 8.18 produces =
=
ix,
,
=
,
-
jx I
lnl-m 14-1I =
(21)-1/2 jx j
,
1
9(A -1@ p.))
j
=
.
-1/2
-1
=
=
=
=
-
,4:4..)-1
.:4.-1
j
expf
-12
-yx
-
Jz)'(A-l)'A-1(x
-
Izlj.
.s;,x-1(.x
t
exp
-j(x .s)
j.
(8.36)
This is the multinormal p.d.f., depending on parametets p. and X. Every such distribution is generated by an affine transform applied to Z. Membership of the multinormal family is denoted X N#,X). n -
8.5 lndependent Random Variables Suppose that, out of a pair of r.v.s (.X,1')on @2T1 we are interested only in situation of interest X. In this the events are the cylinder sets in predicting R 2 having the form B xR, B e S. The marginal distribution of X is detined by (R,S,g,x) where ,g),
,
,
p,x(X)
=
t(X X R)
for W e T. The associated marginal c.d.f. is Fx
(8.37)
F@? +x). The notion of independence defined in 97.4specializes in the following way.xand =
Rattdom Variables F are called independent r.v.s iff pt(XX #)
>x(A)g.l'(S)
=
(8.38)
for al1 pairs of events A,B e f, where g.x is detined by (8.37)and g.y is analogous. Equivalently, g, is the product measure generated by g,x and yty. 8.20 Theorem X and J' are independent iff for each x,y Fxnyq
G
R
(8.39)
Fx@)Fy(X.
=
lf the distribution is continuous the p.d.f. factorizes as x,L
fxxjhb.
=
(8.40)
Proof Obviously, (8.39)is true only if g, satisties (8.38).The problem is to show that the former condition is also suftkient. Consider the half-open rectangles,
C
=
((tz,:)x (c,#1,a.c
e
R-,b,d
e R ).
lf ahd only if (8.39)holds, p,((J,/8
X (c,#1)=
= =
Fb,dl
F(:,c)
-
Fxbq
-
-
Fa,dl +Fa,c4
Fxabttdj
-
Fv)
(8.41)
g,x((J,:1)p,i'((c,tf1),
where the first equality is by 8.17 C is a determining class for (R2 S2) and g is detined by the extension of the measure satisfying (8.41)for ordinary rectangles or, equivalently, satisfying (8.39).The Extension Theorem (uniqueness partl shows that this is identical with the product measure satisfying (8.38).The extension to p.d.f.s follows directly from the detinition. . ,
.
,
With more than two variables there are alternative independence concepts k (compare j7.3). Variables m,...,Xk distributed on the space (R- pk are sa itj to be totally independent if ,g.)
,
1
XAf
p,f=1
k
=
l-lp,xJAd)
(8.42)
f=1
for all k-tuples of events A1,...,Akl 0. By contrast, pairwise independence can hold between each pair Xi&. without implying total independence of the set. Another way to think of total independence is in terms of a partitioning of a vector X = X3....,X' into subvectors .Y1 (jx 1) and Xz k x 1) for 0 < j < k. Under total independence, the measure of X is always expressible as the product measure of the two subvectors, under a11orderings and partitionings of the e lements. -j4
9 Expectations
9. 1 Averages and lntegrals or mean, of a r,v. X()l in a probability space
When it exists, the expectation, L1,5,P) is the integral EX4
Joxtovptl'
=
(9.1)
F(m measures the central tendency of the disibution of X. It is sometimes identified with the limiting value of the sample average of realized values xt drawn in n identical random experiments, xn
=
1
n
(9.2)
.t,
-
>1
as n becomes large. However, the validity of this hypothesis depends on the method of repeating the experiment. See Part IV for the details, but suffice it to say at this point that the equivalence certainly holds if F(A') exists and the random experiments are independent of one another. The connection is most evident for simple random variables. lf X = jxjjej where the ()) are a partition of f1, then by 4.4, osllj). /?(r) !F7 j cr
When the probabilities are interpreted as relative frequencies of the events Ej (: X'4(0) z)) in a large number of drawings from the disibution, (9.2) wlth large n should approximate (9.3). The values xj will appear in the sum in a proportion roughly equal to their probability of occurrence. F(m has a dual characterization, as an abstract integral on the parent probability space and as a Lebcsgue-stieltjes integral on the line, under the derived distribution. lt is equally conrct to write either (9.1) or =
=
+x
EX4
=
xdFxx).
(9.4)
Which of these representations is adopted is mainly a matter of convenience. lf 1x() is the indicator function of a set A G T, then
'(1,4m =
jgxldlk =
Jx(x).x#F@),
(9.5)
Expectations
129
where X(A) e B is the image of A under X. Here the abstract integral is obviously the more direct and simple representation, but by the same token, the Stieltjes fonn is the natural way to represent integration over a set in B.
If the distribution is discrete, X is a simple function and the formula in applies directly. Under the derived distribution,
(9.3)
'k-xj
F(m
where xj, j
=
(9.6)
p,(ta)l),
=
1,2,..., are the atoms of the distribution.
9.1 Example If X is a Bernoulli variable (8.6),EX) = 1.# + 0.(1
-#)
=
p. n
9.2 Example If X is Poisson (8.8), F(m
=
x
e-
ax x Wx
x
e-,E (x
=
vl
a
z-1
'A'
'A'
.
l ) 1.
-
vl
=
,.
u
(9.7)
For a, continuous distribution, the Lebesgue-stieltjes integral of a' coincides with the integral in ordinary Lebesgue measure of the function xfx).
9.3 Example For the uniform distribution on the interval gtz,)1(8.13), F(m
b
1
=
b a
xdx
=
1a(c +
-
a
b). n
(9.8)
9.4 Example For the Gaussian family (8.19), F(A3
1
=
-(x-jt)2/2o.2
#?
xe
Wfic
=
g.
(9.9)
This can be shown by integration by parts, but for a neater proof see 11.8.
I:a
ln a mixed continuous-discrete distzibution with atoms x1,x2,..., we can use the decomposition F F1 + F2 where F1(x) = Zouxgltlayl)and F1(x) is absolutely continuous with derivative hx). Then =
EX)
77.)
p,1(tA)l)
=
+
jxhxtdx.
(9.10)
The set of atoms has Lebesgue measure zero in R, so there is no need to exclude these from the integral on the right-hand side of (9.10). Some random variables do not have an expectation.
9.5 Example Recall the condition for integrability in (4.15),and note that for the Cauchy distribution (8.11),
1
+u
jxj
J-a(
-a
1+
x)2
dx
--
'>9
as
f'
-->
=.
El
(9.11)
130
Probability
9.2 Expectations of Functions of X lf X is a r.v. on the probability space @,f,g), and gl R F-- R is a Borel func-1), as note d in j8.1. This leads is a nv. on the space t'R,f tion, goX to the following dual characterization of the expectation of a function. =
'(m
,g.'
9.6 Theorem If g is a Borel function, F('(A'))
jglxtdbkxb /dB#-1t3')'
=
=
&n):R+
Proof Define a sequence of simple functions m i
z(a)(x)77I =
-
1
(9. 12) F-
R+ by
(9. 13)
1s,.@),
2n
1,...,-. Then, Z(n)(x) 1 and Bi = (2-&( 1), 2-'') for for x k 0, by arguments paralleling 3.28. According to 3.21, (R,S,pw -1 ) j s a g(#-1(#)) for B e S, and so by the monotone measurespace where g' -1(B)
where m
=
nln
't
+
.x
=
-
=
convergencetheorem,
-l(y) y;m
JZ(n)(A')t*'
1
-
.
f=1
zn
jydvg-t).
-1(sj)
.-.y
>'
(9. 14)
Consider first the case of non-negative g. Let 1s@) be the indicator of the set B G 0, and then if g is Borel, so is the composite function
1, glx)
(1so')@)
=
6
B
0, glx) e
B
=
1#-l(s)(x).
(9. 15)
Hence, consider the simple function (Z(,,)o#)@) =
m
77
i
1
-
2n
=1
m
(1sfo#)(x) =
By the same arguments as before, Znjog However, m
Eznjog)
=
77
j
j=1 m i
.
j
2n -
1
gn =X =1
'1'
77 f=I
i
-
2n
1
1#-1(sp@).
g, and Elznlog)
=
Eg)
=
JWg.
.j
g,(# Bi))
-
j
g,# Bi)
=
ztalllfp.'
-
1
(A),
and (9.12)follows from (9.14). To extend the result to general g, consider the non-negative g'b g separately. lt is immediate that maxf ',0J and g-
-->
(9.16)
(9. 17)
functions g'
=
Expectations E(Zn)og+4 Eznjog-) -
m
j.j
m
j-1
X =1
=
Eg-)
-
Eg),
=
(9. 18)
..j
>7 =1 2/
=
Eg')
--
of this limit separately.
so consider each component Eznlog-)
131
p,((#+) Bi)4 (9.19) jooycyg-jo),
-1
ptt# (wi ;;
2*
-..y
0
-1
-1
second equality holds because
where the g +) (Bi) g Bi) for i k g sjnce the 1 the tel'm elements of Bi are a11 positive for these cases, whereas for i disappears. Similarly, -Zn(x) 1 x for < 0, and =
=
.z'
?rj
-7r
-znjog-)
j.j
-
f=1
2/
m
i- 1
-:7
-
zn
j=1
.j
p.((#-)
Bi4)
0
j--ydv.g
-1
p,t, (#7))
--->
..j
(T),
(9.20)
1)1, and in this case the second equality holds where B; (-2-&, because (g- ) (#j) g (-Bi) j or j u z uence -2-''(f
=
-
-1
-1
=
.
Ezn,og
+) -
Ezn,og-)
j-
--
ydjw-lol,
(9.21)
and the theorem follows in view of (9.18).w The quantities F(X), for integer k 2 1, are called the moments of the distribution of X, and for k > 1 the central moments are defined by EX
-
Exlk
=
k
p
X
-
j
Eujh-kjxllk-i
(9.22)
-
A familiar case is the variance, Vartm E(X- F(m) 2 EX ) F(A') the usua j measure of dispersion about the mean. When the distribution is symmetric, with PX- F(A') e A) #(F(m X e 4) for each A e tB,the odd-order central moments are all zero. =
=
9.7 Exampl.
=
2
-
2
,
-
For the Gaussian case
(8.14),the
k'sk EX
-
g) k
=
2
kll
'
(kll)!
central
k even,
moments are
(9.23)
k odd. This formlamay be derived after some manipulation from equation (11.22) below. Vartm c 2 an d a11 the finite-order moments exist although the sequence increases monotoniclly. n =
,
Probability
132
The existence of a moment of given orderrequires the existence of the corresponding absolute moment. If FlXlP < =, for any real p > 0, X is sometimes said to belong to the set Lp (of functions Lebesgue-integrable to order p), or otherwise, to be Lp-bounded.
9.8 Example For X
N(0,c2), we have, by
-
FlXl
2(2a
=
(8.26),
y)-1/2j*.x e-xllloldx ()
(9.24)
l?2 = (2/1) c. n 'raking the corresponding root of the absolute moment is convenient forpumoses of comparison (see 9.23) and for X e Lp, the Lp-ttorm of X is defined as
(FI#I')'* 11111 p
(9.25)
=
The Gaussian disibution possesses a11finite-order moments according to (9.23), but its support is none the less the whole of R, and its p-norms are not unlrmly x, it coincides with the essential bounded. If IIXII, has a finite limit as p supremum of X, so that arandom variable belonging tofax is bounded almost surely. --
9.3 Theorems for the Probabilist's Toolbox The following inequalities for expected values are exploited in the proof of innumerable theorems in probability. The first is better known as Chebyshev inequalf@ for the special case p = 2. 's
9.9 Markov's
For er > 0 and # > 0,
inequality
#( l11 k Proof
tpp
l.Y1
Epjj
:)
=
Al
f:)
Fj aYjf
(9.26)
.
:
yvdFx) K
P
Jjzjzalxl PdFx) f E
1115.*
This inequality does not bind unless F1X1PIZP < 1, but it shows that if Fj -Yjr < x. The order of the tail probabilities converg to zero at the rate z-P as f: fv-boundedness measures the tendency of a distribution to generate outliers. The Markov inequality is a special case of (at least) two more general inequalities. =,
...-4
9.10 Corollary For any event A e F, E
Equivalently,
#((:
j
Arnt
IX()) I
Proof Obvious from 9.9.
lllkcl :1
dp fa
u
j IxI,g,. A
A) f F(1z
(9.27)
lXI#)/e'.
.
9.11 Corollary Let g: R F-> R be a function with the property that x ga) > 0, for a given constant a. Then gxj
a implies
Expectations
PX k a) <
Proof gajpx
k a4
&'(m) .
.
(9.28)
.
ga)
gajxyadFx)
=
133
jxyggxldFx)
S
K
EgT).
w
An increasing function has the requisite property for all a > 0. Let 1 ERbe any interval. A function (: I F-.: R is said to be convex on 1 if
9((1 )x +
,)?)
-
for a11 x, y e I and
.
(0,11. lf
e
(1 )(x)
f
+
-
-9
is convex
*(.y) on t # is
(9.29) on 1.
concave
9.12 Jensen's inequality lf a Borel function (1)is convex on an interval 1 containing the support of an integrable r.v. X, where ((m is also integrable,
(9.30)
9('(A')) f f@(A')). For a concave function the reverse inequality holds. n
The intuition here is easily grasped by thinking about a binary r.v. taking values with probability 1 A convex 4)is illustrated in xl with probability p and whereas F@(A')) p#(x1) + (1 This Fig. 9. 1. #(m px3 + (1 vertical chord the axis by point is mapped from F(m onto the joining and on 9, while 9(F(m) is mapped from the same point by (j itself. -p.
,:2
,-pjxz,
-p)9(x2).
=
=
-:2
.x1
Fig. 9.1 A proof of the inequality is obtaied interior of 1.
9.13 Lemma If and y e 1,
4)is convex
from the following lemma. Let 10 denote the
there exists a function A@) such that, for a11 e 10
X(-:)@
.x
-x4
f #(A') -
9(*.
(9.31)
Proof A convex function possesses right and left derivatives at all points of I0.
Probability
134
(9.29)implies
This follows because (x +
kh)
9(x)s
-
kh
(j(.7 +
for h h) h
-
>
0 that
#(x)
N s ((),jj.
'
(9.32)
+ 1/n) The sequence n e N ) is decreasing, and has a limit 9;(x). (T)(x)), In the case < 0 the inequality in (9.32)is reversed, showing the existence of 4,-/(x)as the limit of an increasing sequence. Note that ()-'@) < (J(x).Tnking the + h gives limit as 2. 4. 0 with > 0 tixed in (9.32)and y
tn(#tx
-
.z7
=
t)-'@)tJ'
S K(-:')(J'
-x4
-xt
S t)(A')
9(.1)
-
<
0 gives, for y
;(.z7)@ S #-'@)1
; th(A') t)@)
whereas the parallel argument with h -x)
-x)
F(m, y
=
X in
=
A('(m)(X-f(m) Taking expectations side has expectation
x,
(9.34)
-
lnequality (9.31)is therefore satisfied with (say)A@) Proof of 9.12 Set x
<
(9.33)
=
#+'(x)..
give
(9.31)to
#(m #(-)).
f
(9.35)
-
of both sides gives inequality
(9.30),since the
left-hand
zero. w
Next, we have an alternative approach to bounding tail probabilities which yields the Markov inequality as a corollary.
9.14 Theorem If X is a non-negative r.v. and r EX
rj-xr-zpx
r)
-
(9.36)
xvx.
>
0
Proof lntegration by parts gives, for some b joxrdFx)
0,
>
=
-
bryb) rjo
= r
-
>
0,
rjoxr-ypdx
Fpldx V--l(s(,) -
x 0
r-lp
dx.
(.x< x u
The theorem follows on letting b tend to infinity.
.
If the left-hand side of (9.37)diverges, so does the theorem is true whether or not EX) is finite.
9.15 Corollary
lf X is non-negative
J-
-xdF
-
spx k
(9.37)
light,
and in this sense the
and integrable,
o
+
j-px
> xvx.
-
(9.38)
Expectations Proof Apply 9.14 Fith r
135
1 to the r.v. 1(xk:)X. This gives
=
J- J-tlfxkslx = Jo 'dF
=
>
xvx
>
xltfx
o
$(1(xk:)X
+
j*PX
>
xldx
:
=
#(X k
E ejodx
jvP(X c
+
>
xjdx.
.
(9.39)
Not only does (9.38)give the Markov inequality on replacing non-negative X by IXI; for p > 0 and arbitral'y X, but the error in the Markov estimate of the tail probability is neatly quantified. Noting that P( IXI E) = 81.11#2 zP),
s ##( I-Yl k e)
j-
=
e
#
IXIJWF
= FIXl# where both the subtracted
j
-
j-
-
gP
P
IA-I'>
#
IXIPJF
px
-
0
I J
xldx
#(IXIP
eP
terms on the right-hand
>
xqdx,
(9.40)
side are non-negative.
9.4 Multivariate Distributions From one point of view, the integral of a function of two or more random variables presents no special problems. For example, if
gl R2
F-y
IR
is Borel-measurable, meaning in this case that g -1 B4 e Bl for every B e B, then #(#((l)),F()) is just a F/f-measurable r.v., and () =
F('(X,F))
=
Jjatldptl
(9.41)
is its expectation, which involves no new ideas apart from the particular way in which the t.v. hlk hppens to be defined. Alternatively,
the Lebesgue-stieltjes Eg-)
=
form is
jpzgx'ybdFx'L'
(9.42)
where dFxny) is to be thought of as the limiting case of &F@,yl defined in (8.30) as the rectangle tends the differential of area. When the distribution is with respect to continuous, the integral is the ordinary integral of '(x,y)J@,y) Lebesgue product measure. According to Fubini's theorem, it is equivalent to an iterated integral, and may be written +x
&'(X,l?))
=
+x
j-yqx,ytx'ybdydx. J.x
But caution must be exercised with formula
(9.43)
(9.42)because
this is not a double
Probability
136
integral in general. lt might seem more appropriate to write d lFx y) instead of JF(x,y), but except in the continuous case this would not be correct. The abstract notation of (9.41)is often preferable, because it avoids these ambiguities. In spite of these caveats, the expectation of a function of (say)X alone can in every case be constructed with respect to either the marginal distribution or te joint distribution. ,
EglxL
9.16 Theorem
jvz
#(X#F(mA')
=
jpgxjdklxj.
=
Proof Define a function +: R2 b..A (R
g
by setting g *@,y)= glx), a 11y e R :*-14#) is a cylinder in R2 with base :-1(#) e B for B e S, and g * i s Sz/s-measurable. For non-negative g, let .
m
* #(a) where m nl n+ 1 an d E i = = l-nli .j xR where Af (x: m
X
=
f=1
i
2
n
(9.44)
ls i
ln
=1
2-''(f 1) S #*(x,y) < 2-NI') e Bl Since Ei 1) K #(x) < 2-% l and tx(Af) = g,(&), -
,
-
.
=
,
j
-
j
.
(@y):
=
F(#*(,'))
j.
=
m
g,t&l
X
=
i
=1
j
-
2
n
p,x(A)
=
Elgnlj.
(9.45)
By the monotone convergence theorem the left and right-hand membrs of (9.45) *x,yldFx,y) and Egj xjdFxxj respectively. Extend converge to E(g.) non-negative general complete the from proof. * to g to =
=
The means and variances of X and F are the leading cases of this result. We also have cross moments, and in particular, the covariance of X and 1' is Cov(X,F)
=
E((X-
&m)(1'- F(F)))
=
Fubini's theorem suggests a characterization
ECXT F(mF(F).
(9.46)
-
of pairwise independence:
9.17 Theorem lf X and F are independent nv.s, Cov@(m,v(F)) of integrable Borel functions ( and
=
0 for al1 pairs
v.
Proof Fubini's theorem gives F(#(mv(F))
=
=
jp,
#@)v(y)#F@,y)
jpxtdlxjvlsbdFkl-bq
= (m)F(v(F)).
w
(9.47)
The condition is actually suffictent as well as necessary for independence, although this cannot be shown using the present approach; see 10-25 below.
Expectations
137
Extending from the bivariate to the general k-dimensional case adds nothing of substance to the above, and is mainly a matter of appropriate notation. lf X is a random k-vector, F(m
=
jxdFx)
(9.48)
denotes the k-vector of expectations, EXi) for 1,...,k. The variance of a scalar r.v. generalizes to the covariance matrix of a random vector. The k x k matrix, =
2 X1X2 X1
XX'
.
.
.
X3Xk
2 X2X1 X2
(9.49)
=
2
Xkxj
Xk
is called the outer product of #, and ECXX') is the k x k positive semi-detinite matl'ix whose elements are the expectations of the elements of XX'. The covariance matrix of X is Vartm
ELCX F(m(#
=
-
-
F(m)'q
=
ECXX') ExTEX4'.
(9.50)
-
is positive semi-detinite, generalizing the non-negative property of a scalar variance. lt is of full rank (notwithstanding that XX' has rank 1) unless an element of X is an exact linear function of the remainder. The following generalizes 4.7, the proof being essentially an exercise in interpreting the matl'ix formulae. Vartm
9.18 Theorem lf F= BX+c where X is an k-vector with EX4 g and Vartm = E, and B and c are respectively an m x k constant matrix and a constant v-vector, then (i) ElY) B# +r. BSB'. a (ii) Vartp =
=
=
Note that it m
>
k Vartl'l is singular, having rank k.
9.19 Example If a random vector Z (Z1,...,Zk)' is standard Gaussian (8.19),it Ik. Applying 9.18 to 0 and EZZ') is easy to verify, applying 9.17, that Ezj = the transformation in (8.35)produces F(A') g and =
=
Vartm
=
EX
-
p)(#-
g4'
=
F(AZZ'A')
=
=
AF(ZZ')A'
=
M'
=
E. n
9.5 More Theorems for the Toolbox Thefollowingcollection of theorems, together with thelensen and Markovinequalities of j9.3, constitute the basic toolbox for the proof of results in proba-
Probability
bility. The student will tind that it will suffice to have his/her thumb in these pages to be able to follow a gratifyingly large number of the arguments to be encountered in subsequent chapters. 9.20 Cauchy-schwartz
inequality
2<
Ex
with equality
attained when F
EXl)Eyl)
(9.51)
,
cX, c a constant.
=
Proof By linearity of the integral, ELaX+ F)2)
a
=
lEXl)
+ E(Yl) ZXIEIXLI
+
for any constant a. (9.51)follows on setting a equality if and only if aX+ F 0. .
=
O
-EIXYIIECXl ), an d holds as an
=
The correlation
c/elcdaf,
rxy
=
1/2 accor dingjy Cov(X,F)/(Var(mVar(D)
the interval (-1,+11. The Cauchy-schwartz inequality is a special case (forp 9.21 Hlder's p/(p
=
-
=
2) of the following.
For any p 2 1,
inequalil
11-Y11,111'11z?.
FI-YFIK where q
jies in
,
1) if p > 1, and q
Proof The proof for the case p
>
if p
x
=
1 requires
=
(9.52)
1.
a small lemma.
9.22 Lemma For any pair of non-negative numbers a,b, ab S
tzp bq
-
P
+
(9.53)
-.
q
If either a or b are zero this is trivial. lf both are oositive, let s = etlq ay = & and t = q log b. Inverting these relations gives a es y rlogtz S/PTf# d an d (9 53) follows from the fact that ex is a convex function of x, noting 3Iq 1 1/p and applying (9.29).* Proof
=
,
,
.
=
Choose a
.
,
=
-
)XI/II-Yllp,b
=
F11t?.For lr1/11
FIXFI
XIp
F
q
= Eab)
these choices, Eap)
S 1/# +
$Iq
=
=
Ebq)
1.
=
1, and
(9.54)
S Fl Xl ess sup F, which holds For the case p = 1, the inequality reduces to FI-YFI since F < ess sup i' a.s., by definition. w
The
lder inequality spawns a range of useful corollaries and special cases, includingthe following.
9.23 Liapunov's
inequality
Proof Let Z F:zlXIF, F = 1, s
(norm inequality) =
r/p. Then,
If r > p > 0, then
(9.52)gives FIZFIf
jlxllr I1-YIlp.
or IIzII,II FIIs(,-1),
Expectations E ( 1X 1#) < E 111
pslls
=
rfr
I11
E
(9.55)
.
.
This result is also obtainable as a corollary of the Jensen inequality.
9.24 Corollary
For each A e
jg IXI'IdP Proof ln
and 1/p + jIq
.
1/p
Jx
p
l71 PdP
<
1,
=
Jx
1Iq
11'IqdP
X by Xlx and i' by i'1x. x
(9.52),replace
probabilistic in character.
Alternative variants of the result are not explicitly
9.25 Corollary n
Let x1,...,x,, and y1,...,yn be any sequences n
77 izz1
1/,
77 1 izzz
lxafl S
771 IyfI
Then
for 1/p + Iq
=
1. n
fzz
be Lebesgue-integrable
functions of a real
'''-(J-
1/,+
(j-
I?f)I,z,)
'-ftkbgtkbkdt s
of numbers.
1/<
n
1x/1J'
and gl
9.26 Corollary Let fl variable. Then
J-
.
I,,(,)l-?,)''
or
lq
-
1. a
Proofs are left as an exercise. The sequences in 9.25 and the functions in 9.26 can be either real or complex-valued (see jl 1.2).
9.27 Minkowski's Proof For r
=
inequality
For r k 1, IIX+ i'llr <
i'llr. + 11 IIXIIr
1 this follows direct from the triangle inequality,
IX+ i'I < I-YI+ I1'l on taking expectations. For r
>
(9.56)
,
1, note that r-1)
FIx+ FIr F(I#+ FII-Y+ FI S E IXI+ IFI)IX+ l'I ) r-1). r-1) + E Irl I x+ I'I = E IxIIX+ l'I =
r-1
inequality to the two right-hand-side
Applying the Hlder
'
E IX+
FI
<
r)1-1/r.
+ 11 X+ l'I FIIr)(FI (II#IIr
Cancelling A'IaY+FI and rearranging r
application to the sum of m variables, generalizes directly to K r
terms yields
(9.58)
gives the result. w
By recursive
Yxi f=1
(9.57)
77Ilmllr =1
for r k 1. For an intinite series pne can write
the Minkowski inequality
(9.59)
Probability
140
77m =1
IImlIr+77xi X i=?n+1 f=1
:
r
If
0 as IIZ7=,,,+1XjIIr --
m
it is permissable to conclude that
tx.
--
(9.60) r
o/
X
i= 1
xi s
f=1
r
IIx)Ilr,
(9.6 1)
not ruling out the possibility that the right-hand side is infinite. 9.28 Love's
cr inequality
For r Fq
e' where cr
=
Txi
>
r
m
s
f=1
1 when r S 1, and cr
0,
=
r, cr XA'1 Xi I
(9.62)
i=1
m r-l wj)en r
j
.
Proof This goes by proving the inequality m
r
m
<-
ai /=1
cr f=1
Iai Ir
(9.63)
for real numbers JI,...,tza, then substituting random variables and taking expectations. Since 1Z0,zzkai lr S (E't', Iai I)r, it will suffice to 1et the ai be non-nega1, tive. For the case 0 < r S 1, 0 f zi f 1 implies z; k zi and hence if Z'?, ctajj. XTt k 1. (9.63)follows on putting zi = ai /(Z0./ For r k 1, on the other hand, convexity implies directly that .1
xlzf
=
=lz(,
j -
m
X icz:l
r
ai
K
-
1
m
XX.
*
izz,1
9.29 Theorem If X, 1', and Z are non-negative r.v.s satisfying X f tz(F+Z) a.s. for a constant a > 0, then, for any constant M > 0, FtltxyvlA')
f
zJtftlf
y>M/2c)F)+F(1(z>ArJw)Z)).
(9.64)
Proof lf we can prove the almost sure inequality 1(x>M)X S 2J(1ty>&z/2u)F+1(z>M/2u)Z),a.s.,
(9.65)
the theorem will follow on taking expectations. 1(x>M)Xis the r.v. that is equal to X if X > M and 0 otherwise. If X K M (9.65)is immediate. At least one of the inequalities F k X/M, Z k Xlla must hold, and if X > M, (9.65) is no less obviously true. w
9.6 Random Variables Depending on a Parameter Let G(,0): f x 0- F-> R, O i R, denote a random function of a real variable 0, or n other words, a family of random variables indexed on points of the real line.
141
Expectations The following results, due to Cramr nated convergence theorem.
easy consequences of the domi-
(1946),are
9.30 Theorem Suppose that for each (l) e C, with #4C) 1, G4,0) is continuous at apoint %,and IG((t),0) I < F((t))for each 0 in an open neighbourhood No of % where =
F(i')
<
Then
x.
lim EG(44
(9.66)
F(G(0o)).
=
8-+%
Proof Passage to a limit 00 through a continuum of points in 0- as indicated in (9.66), is implied by the convergence of a countable sequence in 0- Let (0v,v e NJ be such a sequence, in No, converging to 0o. Putting Gv((t)) G(,0v) defines a countable sequence of r.v.s. and limsupvGvt) and liminfvGvt) are r.v.s by 3.26. By continuity, they are equal to each other and to G(,%) for (l) e Q in other G(%) a.s. The result follows from the dominated convergence words, G(0v) ,
.
=
-->
theorem.
.
9.31 Theorem 1f, for each
)
e C with #(C)
G(),%
+
h4
1, (JG/J0)((t)) exists at a point
=
G(,%)
-
n((o)
< for 0 f
< Jl1, where F(l'1)
KdoEa)
<
j
(x,
and
is independent of
l
ttjG()..)
-
(.,
I
e.
% and
),
then
(9.67)
.
Proof The argument goes like the preceding one, by considering a real sequence (v ) tending to zero through positive values and hence the sequence of r.v.s (SVJ where Hv gG(0o+/lv) G(%)1/v, whose limit H S(0()) exists by assumption. . =
=
-
The same sort of results hold for integrals. Fubini's Theorem provides the extension to general double integrals. The following result for Riemann integrals on intervals of the line is no more than a special case of Fubini, but it is useful to note the requisite assumptions in common notation with the above.
9.32 Theorem Suppose thatforeach (l) E C, withf'to a t'initeopen interval atb), and IG(,0) l < 1'2() for
=
1, G(,0) is continuous on 0 < b, where F(F2) < x.
q <
Then
s(js(e)ts)
J-s(c(e)):e -
lf J7'IG(,0) lJ0 < 1'3(t,))for (t) E C and Ek) and b +=. of a =
-x
(9.68)
.
-
<
cx),
(9.68)holds
for either or both
=
Proof For the case of t'initea and b, consider Hlsb,tl JJG(,0)#0. This has the I lG(),f) I < F2(), for each propertiesIH% I < b J)F2(t,)), and I dHld EH) exists and by /, each 9.31, for t e a, b4 Hence, =
-
.
=
Probability
l42
d EH) E(Gt))
is continuous
=
E
dH
=
dt
EGtj).
on a,b4 by the a.s continuity
a(,) -
jtEol4d
(9.69) of G and 9.30, and hence
-s(s(?))
(9.70)
J
is differentiable on (a,b4,and dhldt = 0 at each point by (9.69).But by defi0, and hence hb) 0 for e C, so that A(J) 0 which is equivnition, H,a) alent to (9.68). =
=
=
JX
G(,0)#0 exists and is Under the stated integrability condition on G(,0), = J!xG(t0,0)#0 is well defined and has an expectation for finite on C. Hence Htt) and and/or b = x. > th argument above goes through with a = all t e R, -X
-.x
10 Conditioning
10.1 Conditioning in Product Measures It is difficult to do justice to conditioning at an elementary level. Without resort to some measure-theoretic insights, one can get only so far with the theory before nznning into problems. There are none the less some relatively simple results which apply to a resicted (albeitimportant) class of distributions. We approach, and so demonstrate the diftiintroduce the topic by way of this arise, before going culties that on to see how they can be resolved. In the bivariate context, the natural question to pose is usually: we know X what of lX,l') F?' For a random real pair is the best predictor on (,T, #) we = x, class conditional evidently define of functions for distribution j7.2) a (see can K For any A e B such that PX e A) > 0, let 'naive'
dif
F@ lX
G
A)
PX e A, F < y) PX s A)
=
( 10. 1)
.
This corresponds to the idea of working in the trace of (f1,T,#) with respect to A, once A is known to have occurred. Proceeding in this way, we can attempt to construct a theory of conditioning for random variables based on the c.d.f. We may tentatively define the conditional distribution function,F(.y Ix),when it exists, as a mapping from R2 to R w hich for tixedx q R is a non-decreasing, right-continuous function of y with F(-x lx) 0 and F(+x lx)= 1, and which for fixed y e R satisties the equation =
Px
e
,4,
J' < y)
jgFl.y
=
Ix)#Fx(x)
(10.2)
in y-space for any A e B (compareRao 1965: j2a.8). Think of the graph of F(.yIx) through the sudace of the joint distribution funcas the protile of a tion, parallel to the y-axis, at the point x. However, much care is needed in interpreting this construction. Unlike the ordinm'y c.d.f., it does not represent a probability in general. If we try to interpret it as P(Y S y IX x), we face the possibility that P(X x) 0, as in a continuous distribution. Since the integral of F@ lx) over a set in the marginal distribution of X yields a probability, as in (10.2),it might even be treated as shows that we would need a type of density function. Taking A (.Y f islice'
=
=
=
.x)
=
,(x,y)j-,'(,I()#,k(t) J-j' g,'(vI():,'x(t) -
-
ex
ex
ex
(10.3)
Probability
144
to hold. Since F(x,y) is an integral over R 2 F u bini's theorem implies that F@ 1x) is well detined only when the integrals in (10.3)are with respect to a product ,
meastlre.
If Xand Fare independent we can say unambiguously (butnot very usefully) that FLyjx) Fy%). F(? j.x)is also well defined for continuous distributions. Let Sx denote the support of X (the set on which fx > 0); the conditional p.d.f. is =
J# Ix)
fx,yj .fx@)
=
,
.x
e
sx,
(10.4)
where jxx) is the marginal density of X. We may validly write, for A e B
px
e
x, r s
jgj'
.ftvlo/'axltxo
y)
-
-
f''h
Sx,
j z-twylxlyxt.xlts (10.5) #
-x
where F(..lxl
j' Avlxlfo.
-
(10.6)
-K
The second equality of (10.5)follows by Fubini's theorem, since the function fxny) is integrated with respect to Lebesgue product measure. However, (10.6) appears to exist by a trick, rather than to have a firm relationship with our intuition. The problem is that we cannot work with the trace (A,Ta,#a) when A = (: X()) xl and #(A) 0, because then #a = #/#(A) is undefined. lt is not clear the case when (,Y = x) has occurred' when this event what it means to fails to occur almost surely. Except in special cases such as the above, the factorization dF(y,xl = dF@ Ix)#Fx@) is not legitimate, but with this very important caveat we can define the mean and other moments of the conditional distribution. The conditional expectation of a measurable function #(-Y,i'), given X = x, can be defined as =
=
Ex,n
Ix)
+*
J
=
glx-ybdFytxq,
(10.7)
-X
also written as Elgx,Y) lX = #. The simplest case is where g(X,j') is just F. f(l'Ix) is to be understood in terms of the attempt of an observer to predict F after the realization of Xhas been observed. When Xand Fare independent, &F1x) where EY4 is the ordinary expectation of F, also called the marginal or = E, unconditional expectation. ln this case, the knowledge that X x is no help in predicting K =
10.1 Example These concepts apply to the bivariate Gaussian distribution. From (8.36), the density is -1
fx,yj
1
=
2a
C11 C12
c12c22
j/2
exp
,r
-
a
611 C12
z't.x g.1,y g,21 C12 -
-
C22
/1
X-
F
-
/2
Conditioning
145
(A' g,2) -
-
1
= 2J(cl
1c22
2 cl2 g,1) (.:7 c1l (x -
-
-
c212)
exp
jzl) 2
-
-
l/2
2 c22
zcjj
c212
,
(10.8)
-
c11 the square in the exponent.
where the last equality is got by completing Evidently, fxky) fy Ixjfxx) where =
(A,
p,2) -
-
1I)
fl.y
1
=
2 WIVc2a c12 c11
exp
ja
c12 c11
2
'--fx
-
-
2 c22 -
-
c21 a
M,1)
(10.9)
c11
and fxxj
l exp 2zrc11
=
-
@- B1)1
zo.11
.
(10.10)
Thus, F(Flx)
(T12
----t.r
=
g,2 +
-
c22
p,1),
(10.1 1)
and VartFlxl If c1a
of the
=
0,
fy lx) reduces
marginals,
=
622
-
c212 .
c11
(10.12)
so that the joint density is the product and x and y are independent. n to
fvy),
10.2 Conditioning on a Sigma Field In view of the limitations of working directly with the distribution of (aY,l'), we pursue the approach inoduced in j7.2, to represent partial knowledge of the distribution of F by specifying a c-field of events N g 5 such that. for each G e N, an observer knows whether or not the realized outcome belongs to G. The idea of knowing the value of a random variable is captured by the concept of subfield measurability. A random variable .X()): fl F-> R is said to be measurable with respect to a c-field V c 5 if x-2B)
=
(:
.Y4(0)G #) e V, for al1 B e f.
(10.13)
The implication of the condition V c 5 is that the r.v. X is not a complete
Probabilit.y
representation of the random outcome ). We denote by c(m the intersection of all c-fields with respect to which X is measurable, called the c-fild generated by X.
Jf, on being confronted with the distribution of a random pair (X()),1'())), we learn that X we shall know whether or not each of the events G e ctm has by determining whether X(G) contains x. The image of each G e c(A3 occurred under the mapping .x,
=
(X()), F()))2 D
F--> R
2
is a cylinder set in R 2 an d the p.m. defined on c(A') is the marginal distribution of X. .
10.2 Example The knowledge that xl fk X ; can be represented R). R c(f(-x, x1: x < xl ), ((x, x): x > .x2
by
.x2),
=
Satisfy yourself that for every element of this c-field we know whether or not X belongs to the set; also that it contains all sets about which we possess this knowledge. The closer together a'1 and are, the more sets there are in R. Wben +x, lO,R 1. n J4 = 5 ctm, and when xl .v1 x2, R .x2
=
=
=
-x,
.:2
=
=
The relationships between transformations and subfield measurability are summarized in the next theorem, of which the first part is an easy consequence of the definitions but the second is trickier. lf two random variables are measurable with respect to the same subfield, the implication is that they contain the same information; knowledge of one is equivalent to knowledge of the other. This means tha t evety Borel set is the image of a Borel set under g- This is a stronger condition than measurability, and requires that g is an isomorphism. It suffices for g to be a homeomorphism, although this is not necessary as was shown in 3.23. .
10.3 Theorem Let X be a r.v. on the space (S,fs,p), and let i' gl S F-+ 'T is a Borel function, with S i R and 'T i R. (i) ctl') c ctm. (ii) c(1?) ctm iff g is a Borel-measurable isomorphism.
=
glX) where
=
Proof Each B e f.r has an image in fs under g -1 w hich in turn has an image in c(m under X -1 This proves (i). To prove (ii), define a class of subsets of S, C (.g-1(#): B e B v.). To every -1 A S there corresponds (sinceg is a mapping) a set B 'T such that A = g B), and making this substitution gives ,
.
=
fs
lz4:
'(z4)
e
.r 1
fp-j (#): gg
=
-1
(#)) e s v l
=
g
jj(;
,
.
j4;
where the inclusion is by measurability of g -1 an d the second equality is because -' = gg (#)) B for any B c 'T, since g is 1-1 onto. It follows from (10.14)that ,
fx -1( s -4)
j:
z4
e ls
)
g
(x-1(:-1(,)) c
f: B s f.y).
(10.15)
If F is V-measurable for some c-field V 5 (suchthat V contains the sets of the right-hand member of (10.15)), then X is also V-measurable. ln particular, ctm ctl?). Part (i) then implies c(m = c(F), proving sufficiency of the conditions. = To show the necessity, suppose first that g is not 1-1 and glxL) = y '(.n)
Conditioning
(x1) and (x2)are elements of Ss but not of C, which contains only g -1(fy)) fxl) k.p (x2) Hence : Bs C, and 3 a Os-set X for which G there is no Or-set B having the property g -1(B) A This implies that (say)for xl
y:
The sets
.n.
=
.
.:-1(4)
=
.
c(F), so that c(F) c ctm. We may therefore assume that g is 1- 1 If g is not Borel-measurable, then by C; and again, definition 34 = g -1(B) G B s suc h thatgtA) =B e tB.r,and henced Ss C, so that c(l') c ctm by the same argument. This completes the proof of
ctm but
-1
.
necessity.
.
We should briefly note the generalization of these results to the vector case. A random vector #4): f F-> Rk i s measura b1e with respect to 97 c 5 if X -1 B4
=
f
:
A)l
e #)
G
s vs ,
s pk
(1().j6)
.
If ctAr) is the c-tield generated by X, we have the following result.
10.4 Theorem Letfbe $ c R an d Bks fl =
F
k arandom vector on the probability space (S,fs,p) Sk) and consider a Borel function s: B e
f-h
,
g(m: S
=
w jaere
F--
1', T' g R2'.
(10.17)
(i) G'(1') l c(A'). (ii) If m k and g is 1-1 with Borel inverse, then c(1') =
=
ctm.
Proof This follows the proof of 10.3 almost word for word, with the substitutions of fs k an d Bkr for Ss and S.r, X and F for X and F, and so forth. .
10.3 Conditional Expectations Let i' be an integrable r.v. on (f,T,#) and N a c-tield contained in F. The term conditional expectation, and symbol F(FIN), can be used to refer to any integrable, N-measurable r.v. having the property jgFtrl
U)dP
=
F(FI joj'dp
G)#(G), all G e N.
=
(10.18)
Intuitively, F(FIV)()) represents the prediction of F() made by an observer having information V, when the outcome (.t) is realized. The second equality of (10. 18) supplies the definition of the constant S(l'l G), although this need not exist unless #4G) > 0. The two extreme cases are F(FI@) i' a.s., and F(Jrl 5) EY) a.s., where T denotes the trivial c-field with elements (f,0 ). Note that f s 5', so integrability of F is necessal'y for the existence of F(FIV). The conditional expectation is a slightly bizarre construction, not only a r.v., but evidently not even an integral. To demonstrate that an object satisfying (10.18) actually does exist, consider initially the case F 0, and define =
v(G)
=
JgF##.
10.5 Theorem v is a measure, and is absolutely
=
(10.19) continuouj with respect to #.
Probability
148
0. lt remains to show Clearly, v(G) 2 0, and #(G) = 0 implies v(G) is a disjoint sequence, then countable additivity. If ll Proof
=
vUb
v,o,Ydp
= *'
j
J
=
J
X j
o,Ydp *'
-
'
Xvt(l,
(10.20)
./
where the second equality holds under disjointness.
.
So the implication of (10.18)for non-negative F turns out to be that F(FIT) is the Radon-Nikodym derivative of v with respect to #. The extension from nonnegative to general F is easy, since we can write F F+ F- where F+ and Yare non-negative, and from (10.18),F(Fl V) F(F+IV) E(Y- IN), where both of the right-hand r.v.s are Radon-Nikodym derivatives. The Radon-Nikodym theorem therefore establishes the existence of F(FI5:); at any rate, it establishes the existence of at least one nv. satisfying (10.18).It does not guarantee that there is only one such r.v., and in the event of nonuniquenes, we speak of the different versions of F(l'IV). The possibility of multiple versions is rarely of practical concern since 10.5 assures us that they are al1 equal to one another a.s.g#j, but it does make it necessary to qualify any statement we make about conditional expectations with the tag a.s.', to indicate that there may be sets of measure zero on which our assertions do not apply. ln the bivariate context, F(Fl c(m), which we can write as FtFlm when the context is clear, is interpreted as the prediction of F made by observers who observe X. This notion is related to (10.7)by thinking of F(l'l x) as a drawing from the distribution of F(FIm. =
=
-
-
10.6 Example ln place of (10.11)we write &Flm(t,))
61 2
=
---(X() p,2 + C1 1
-
p.1),
(10.21)
which is a function of X(l, and hence a r.v. defined on the marginal distribution of X. F(FIm is Gaussian with mean g,2 and variance of c2la/cll. E! Making
F(FIx)a point
in a probability space on R circumvents the difficulty
previously with conditioning on events of probability 0, and our is valid for a11distributions. lt is possible to define F(FIG4 when
encountered construction #(G) 0. What is required is to exhibit a decreasing sequence fGn e N) with #(Gn) > 0 for every n, and Gn 1 G, such that the real sequence (F(1'1 Gn)) converges. This is why (10.7)works for continuous distributions. Take Gn gx,x + 1/p1XR s =
=
c(A'), so that G
=
Using
(x)xR.
(10.4)in (10.18),
+co f*x+1/n
f(rl Gn)
J..J
yfl,ytdldy
+x
x
=
--
+.
ujtn
uJ-?'(''.''vjvy
j-giy'xsdy =
&Fl*,
(10.22)
Conditioning x. Fubini's theorem allows us to evaluate these double integrals one as n dimension at a time, and to take the limits with respect to n inside the integrals with respect to y. --->
Conditional probability can sometimes generate paradoxical results, following case demonstrates.
as the
10.7 Example Let X be a drawing from the space (g0,11, p,IJ, m), where m is Lebesgue measure. Let N c fp,1) denote the c-field generated from the singletons fxl, x e (0,1). A1l countable unions of singletons have measure 0, while all complements have measure 1. Since either #(G) 0 or #4G) = 1 for each G e V, it is clear from (10.18)that F(XIN) F(m a.s. However, consider the following weknowwhetheror notxc Gforeach Ge V, weknow argument. 'Since fx) e N, if x. ln particular, V contains knowledge of the outcome. It ought to be the case that F(XIV) X a.s.' n =
=
=
,
=
The mathematics are unambiguous, but there is evidently some difficulty with the idea that V should always represent partial knowledge. lt must be accepted that the mathematical model may sometimes part company with intuition, and generate paradoxical results. Whether it is the model or the intuition that fails is a nice point for debate.
10.4 Some Theorems on Conditional Expectations 10.8 Law of iterated expectations
(LlE)
FgF(J'IN)j Proof lmmediate from
(10.23)
Ek.
=
(10.18),setting
G
=
D.
.
if constant' under The intuitive idea that conditioning variables can be held the conditional distribution is confirmed by the following pair of results. 'as
10.9 Theorem lf X is integrable and V-measurable, then Proof Since X is N-measurable, E
+
then
jyydp
-
=
1): X()
>
F(XIN)()1
F(.YI jg-yxf jyjxN)J#
This contradicts (10.18),so PE*) = (tt):X((t7) < #(xl V)(@) E V.
N))J#
=
=
F(#IN)
>
=
X, a.s.
e N. lf #(F+)
0.
0. By the same argument, P(E-)
>
0,
(10.24) =
0 where E-
*
10.10 Theorem lf Fis F-measurable and intepable, X is V-measurable for V g @, N) a.s. < x, ten &FaY1T) and E IA-FI -YSIFI =
Proof By definition, the theorem follows if %)dP xYFIFI
Let
xn)
=
o
'z-lajlsj
=
J s
XYdP a.s., for al1 G e T.
(10.25)
be a V-measurable simple f.v., with F1,...,Fn a pmition of
Probability
150
f and Ei e V for each i. (10.25)holds for X
X(n) since, for all G e V,
=
jgxtnwtyl cldp nwj ac, s(I,Itvp x -
i= 1
=X-
.ij
yw,
-
GIAEt'
/=1
j x-,vdp,
(10.26)
G
Ei e N when G e V and Ei e V. Let X k 0 a.s., and let (X(,,)J be a monotone sequence of simple N-measurable converging XY a.s. and IX(n)F( K lXFI functions to X as in 3.28. Then XnlY N) where F1 XFI < x by assumption. Similarly, X(n)F(FI #FIl'IV) a.s., and
noting G
fa
,--+
,
--
e-l-Ytnlftrl N) I
=
1-Y(n)1'l I9-))
F1 F(A)n)FI N) I S EE
= FlaYtall'l
e-IA'l'I<
<
(10.27)
x,
where the first inequality is the conditional modulus inequality, shown in 10.14 below, and the second equality is the LlE. It follows by the dominated convergence JcxftFl N)##, and so (10.25)holds for non%)dP theorem that Jc#(n)F(rI negative X. The extension to general N-measurable X is got by putting -->
x where X*
=
maxfmo)
eFXl 9')
=
=
X+ -X-
(10.28)
2 0 and A'- 2 0, and noting
EIYX'P
= X*
-
-
FX- IN)
X-)F(l'IN)
=
ECYX' IN) EYX- lN) -
XFIFIV) a.s.,
=
using (10.33)below and the result for non-negative X.
(10.29) .
X does not need to be integrable for this result, but the following is an important application to integrable X.
10.11 Theorem lf F is F-measurable and integrable and F(rl V) El') for N c @, then Covtmp = 0 for integrable, V-measurable X. =
Proof From 10.8 and 10.10,
ECXYI #(A'(A-FlN)) =
If
F(FIV)
=
EY) a.s.
(a constant),
=
Ft-tftrl
V)J.
= then ECXYI EkEr).
(10.30) w
Note that in general Covt#,F) is defined only for square integrable r.v.s X and K But Cov(X,Y) 0, or ECXYI= FtmFtF), is a property which an integrable pair can =
satisfy.
The following is the result that justifies the characterization of the conditional mean as the optimal predictor of J' given partial information. optimal'is sen to have the specific connotation of minirnizing the mean of the squared
prediction errors.
Conditioning 10.12 Theorem
.:
t $... .(y:,.. y... ).)... ../.
g
Let
denote any V-measurable approximation to F. Then < (1 Y- f'112. jlY- F(rI N) 112
.j;
-
...
151
(10.31)
-
. .. .. .-.
Proo
f (F-
hence
f)2 (F-F(FIN)j2+2(y-F(Fj f)2IT)
F((F-
=
Fg(F-F(Fj
a,q-
V)jgF(J'l T)
=
N))2jNj + (F(yl N)
-
a..v
Fj
-
+
gF(FIN) F)2, and
f12a.s.,
-
(10.32)
noting that the conditional expectation of the cross-product disappears by definition of F(1'1 V), and 10.10. The proof is completed by taking unconditional expectations tluough (10.32)and using the LIE. w The foregoing are results that have no counterpart in ordinary integration theory, but we can often exploit the fact that the conditional expectation behaves expectation, apart from the standard caveat we are dealing with like a nv.s so that diferent behaviour is possible on sets of measure zero. Linearity holds, for example, since %real'
hFIN)
EaX+
=
aExt T) +
:FIFIT),
(10.33)
a.s.,
is a direct consequence of the detinition in (10.18).The followipg are conditional versions of various results in Chapters 4 and 9. The first extends 4.5 and 4.12. 10.13 Lemma V) 0 a.s. (i) If X 0 a.s., then F(aYI (ii) If X f F a.s., then F(.Yl T) S F(Fl V) a.s. (iii) If X F a.s., then F(aYlN) F(FIV) a.s. =
=
=
=
Proof (i) follows diretly from (10.18).To prove (10. 18) and 4.8(i) together imply jaExk
for a11G e V. SinceA
N)##
=
(ii), note that the hypothesis,
jaxdp hk'dp S
=
JgF(Fl
G4dP
T)((t))> 6(FIT)((t))1e V, itfollows f: F(xYI
that #(A) = 0. The proof of (iii) uses 4.8(ii), and is otherwise identical to that of (ii). w
10.14 Conditional
=
IF(FIT) I f
modulus inequality
E Il'l IT) a.s.
Proof Note that IFI F+ + Y-. where F+ and Y- are defined in (10.28).These are non-negative nv.s so that F(F+l N) > 0 a.s. and EY- j N) 0 a.s. by 10-13(i) 1, and (ii). For ) e C with PC4 =
=
lF(F+
-
Y- lN)()1
=
IF(F+l T)()
-
< F(l'+IT)((o) + =
Ev+w y-
/(F-1 97)())1 EY-
I )(),
where both the equalities are by linearity.
.
lN)() (10.34)
Probability
10.15 Conditional monotone convergence theorem lf L f F and L 1- 1' a.s., then F(FnI5') 1' F(1'1 N) a.s. Proof Consider the monotone sequence Zn L K Since Zn f 0 and Zn S Zn+I, 10.13 implies that 4he sequence Ezn lN) ) is negative and non-decreasing a.s., and hence converges a.s. By Fatou's Lernma, =
-
ja limsup Ezn N) dP limsup jaE(Zn U4dP k l I n'-/oo
n'->
= limsup
Zndp
=
G
N-->=
(10.35)
0
for G e V, the tirst equality being by (10.18), and the second by regular monotone convergence. Choose G = f : limsupaAzn lT)() < 0J which is in N by 3.26, and (10.35) implies that #4G) 0. It follows that ,
=
V) lim F(ZnI
0, a.s.
=
(10.36)
.
n'-co
10-16 Conditional
Fatou's
liminf
lemma If Fn k 0 a.s. then
&FnIV) k
E liminf L IN a.s.
influnrk so that l%' is non-decreasing, and converges to F = EY' IW) by 10.15. F,, k Fr, and hence E liminfarr,. Then E'n' 5') l5>)1 E'n' lT) a.s. by 10.13(ii). The theorem follows on letting n . Proof Put
L'
(10.37)
n-'+oo
n-oo
=
--+
--)
,.
Extending the various other corollaries, such as the dominated convergence theorem, follows the pattern of the last results, and is left to the reader.
10.17 Conditional
Markov
inequality
#(1IFl
P E.l IT) S
F(l
rlpl T) EP
,
a.s.
Proof By Corollary 9.10 we have
s'jolfIrlkcll' By definition, for G e N,
jopf I1'l
f
Jgl
Y'PdP, G e @.
k El I*4dP
and jaEf
FIPIUtdp
=
=
Jclt
ll'lk.:)tf#,
Jcl
YPdP.
Substituting (10.39)and (10.40)into (10.38),it follows that
(10.38)
(10.39)
(10.40)
Conditioning JcEEf#(t
l1'I 2
El
153
I5') -F(I F1#1U4dp
(10.41)
f 0.
The contents of the square brackets in (10.41)is a V-measurable r.v. Let G e N 0. . denote the set on which it is positive, and it is clear that #(G) =
10.18 Conditional Jensen's inequality Let a Borel function () be convex on an interval 1 containing the support of a i-measurable r.v. F where F and (/(J') are integrable. Then
4'tftrlT))
< F@tl'')l
Proof The proof applies 9.13. Setting x
N), a.s.
(10.42)
F(l'IV) and
=
y
J' in
=
T)) < :(F) -#(F(1'IN)).
A(F(1'l V))(F-F(Fl
(9.31), (10.43)
V)) is a random variable. It is not certain that However, unlike A(F(D),A(F(FI the left-hand side of (10.43)is integrable, so the proof cannot proceed exactly like that of 9.12. The extra t'rick is to replace F by 1sF, where E ftl): F(FIN)() f #1 for B < x. F(Fl V) and hence also ls are V-measurable random variables, so F(1sFIT) = 1sF(l'IV) by 10.10, and =
N) e'@(1sl')I
=
= Thus, instead of
F(9(F)ls+ lsfttF)l
#(0)1rIT) V)
(1
+
-
1s)#(0).
(10.44)
(10.43),consider
A(F(ls1'IT))(1sF-
lsF(l'l V)) k #(1sF) -#(1sF(l'l
T)).
(10.45)
The majorant side of (10.45)is integrable given that (/(i') is integrable, and hence so is the minorant side. Application of 10.9 and 10.10 establishes that the conditional expectation of the latter term is zero almost surely, so with (10.44) we get
#(1sF(l'IV)) Finally, 1et #
--
x
< 1sF@(1?)l V)+
so that ls
-->
(1
-
(10.46)
1s)9(0), a.s.
1 to complete the proof.
.
The following is a simple application of the last result which will have use
subsequently.
10.19 Theorem Let X be V-measurable and fv-bounded for r k 1. If F is F-measurable, X+ F is also Lr-bounded, and F(Fl V) 0 a.s., then =
FIxY+ FIr k
FI-YIr.
(10.47)
Proof Take expectations and apply the LlE to the inequality F(1-Y+ FIr! N) 1
IEX+
irj N)Ir
=
lx1r
a.s.
.
(10.48)
Finally, we can generalize the results of j9.6. It will suffice to illustrate with the case of differentiation under the conditional expectation.
154
Probability
10.20 Theorem Let a function G(,0)
e'(-J I I
r)
dedl 9'' -
:-o
E
-
G(%)
hv
T
I
8=00
(v, v e
Proof Take a countable sequence of the conditional xpectation, G(0() + h
satisfy the conditions of 9.31. Then
F(G(%
(Nl with hv +
v)
=
lV)
(10.49)
-.s.
,
-
--
0 as v
x.
--
(G(%) IT)
By linearity
(10.50)
a.s.
hv
If Cv 6 V is the set on which the equality in (10.50)holds, with #(Cv) 1, the #(OvCv) a nd limit 1 by 3.6. The in the two sequences agree on the set Ovcv, left-hand side of (10.50)converges a.s. to the left-hand side of (10.49)by assumption, applying the conditional version of the dominated convergence theorem. Since whenever it exists the a.s. limit of the right-hand side of (10.50) is the right-hand side of (10.49)by definition, the theorem follows. . =
=
10.5 Relationships between Subfields TI
5 and Mzc 5 are independent subfields if, for every pair of events Gj e V1 and Gz s N2, #(Gj r7 G
=
#(G1)#(Gc).
(10.51)
Note that if F is measurable on Vl it is also measurable on any collection containing V1,and on ; in particular. Theorems 10.10 and lo-llcover cases where F as well as X is measurable on a subfietd.
10.21 Theorem Random variables X and F are independent iff ctm and ctF) are independent. Proof Under the inverse mapping in (10.13),G1 e c(m if and only if #1 X(G1) i T with a corresponding condition for c(F). It follows that (10.51)holds for each G1 e c(m, G1 G c(F) iff PX* Bj, i'e Bz4 PX e #1)#(Fe Bz) for eachfl X(G1), Bz F(G2). The only ifo of the theorem then follows directly from the follows, given (8.38),from the fact that every Bi e definition of ctm. The B has an inverse image in any subfield on which a r.v. is measurable. w =
=
=
=
'if'
The
only ifo in the first line of this proof is essential. Independence of the subtields always implies independence of X and F, but the converse holds only for the infimal cases, c(m and f5(1').
10.22 Theorem Let F be integrable and measurable on V1. Then 5'I a.s. for a11 V independent of
F(FIV)
=
EY)
.
Proof Define the simple Vl-measurable r.v.s l'(n) G11,...,G1s of ( where G1j e V1, each i, with l%)
On a partition = Z'lzzlyflclf 1 F as in 3.28. Then
Conditioning
jynlb
5')J#
n
X'faFtltyl/l
=
f=1
=
*'
vl,
%''
#(G)XX#(G1f)
-
155
ynyjacjf
c)
ra
=1
V,
all G e
fOr
PG4EYn)4
=
(10.52)
f=1
F(i') by the monotone convergence theorem. F(F(n) IN) is not a simple function, but EYn) IV) F(Fl V) a.s. by 10.15, and
Ejnj)
--
'1'
joEn)
IVV# --
VV#' jcfk''
(10.53)
convergence. Hence for Vl-measurable F,
by regular monotone
JgFIFIM4dP
=
for a1l G e V.
PGjEY)
(10.54)
Frop the second equality of (10.18) it follows that F(FIG) which proves the theorem. w
10.23 Corollary
If X and F are independent, the
Proof Direct from 10.21 and 10.22, putting V
=
&FI
EY4 for a11 G e V,
=
.)
=
E').
ctm and V1
=
c(F).
.
10.24 Theorem Apairof c-fields V1cF and Vac 5 areindependentiff Cov(X,D 0 for every pair of integrable r.v.s X and F such that X is measurable on V1 and F is measurable on N2. =
Proof By 10.22, independence implies the condition of 10.11 is satisfied for N tif' consider X ito To prove lcj, G1 e V1, and F 1cz for V1, proving G2 e N2. X is Nl-measurable and F is Na-measurable. For this case, =
Cov(X,D
=
Cov(X,D =
=
,
.
#(GI
fa
=
G2) #(Gl)#(G2).
(10.55)
-
0 for every such pair implies V1and V2 are independent by (10.51).w
10.25 Corollary Random variables Xand F are independent iff Cov@(m,v(F)) 0 for every pair of integrable Borel functions 4)and v.
=
Proof By 10.3(i), 9(A') is measurable with respect to c(m for al1 and v(F) is ctpimeasurable for a1l v. If and only if al1 these pairs are uncorrelated, it follows by 10.24 that c(A3. and cti') are independent subfields. The result then follows by 10.21. w ,
An alternative proof of the necessity part is given in 9.17. The next result generalizes the law of iterated expectations to subfields. We say that c-fields V1 and V2 are nested if V1 c V2.
10.26 Theorem lf N1 i Va @, then for T-measurable F, N2)1N11 F(l'l N1) a.s. (i) EEF(l'I N1)1T21 F(l'IT1) a.s. (ii)FEe-(1'l =
=
Probability
156 Proof By definition, jLELECY'
V2)IUjjdp
But, since G e T1 implies G jcyE''
J(JIFI 5'2)## for
=
6
V2)1T11##
al1 G e N1.
(10.56)
N2, (10.18)and (10.56)imply that =
jo-dp
all G e T1,
fOr
(10.57)
T2) IV1 is a version of F(Fl N1), proving (i). Part (ii) is by so that A'(F(Fl NI) is a Vc-measurable r.v. w sinceF(FI
10.9,
lf A simple application of the theorem is to a three-variable disibution. 3 (X(),l'(),Z())) is a random point in R measura ble on T, 1etc(Z) and c(F,@ be infimal c-fields on which Z and (F,@ respectively are measurable, and c(/ the c(F,@ 5. Unifying notation by writing F(l'Iz) = F(FIc(z)) and F(rlmz) = F(FIc4-Y,/), 10.26 implies that ,
F(e'(i'I-Y,z)IZ1
=
F(A'(FIz)I-X-,Z)4
=
F(1'1z).
(10.58)
Our final results derive from the conditional Jensen inequality.
10.27 Theorem Let F be a F-measurable r.v. and T1 i Uz Proof
T. lf
9(.)is convex,
F(t(F(FI5-2))) 2 F(9(e'(rI971))(). Applying 10.18 to the Tz-measurable r.v. F(FIW2)gives F@(f(FIT2))1V1) k #(F(F(l'I91)1N1)) #(F(Fl V1)) a.s.
(10.59) (10.60)
=
where the a.s. equality is by 10.26(i). The theorem follows on taking unconditional expectations and using the LlE. . The application of interest here is the comparison of absolute moments. Since lxlP 1, the absolute moments of F(FIT2)exceed thoseof F(FIT1) when is convexforr N1 i N2. ln particular,
(10.61) FgF(rIT2)21 k FgF(rIN1)2j. SinceFti'l NllandFtFl Vzlbothhavemean ofE, T2))k (10.61)impliesVar(F(FI EYl) for i 1 or 2, (the T1)). Also, F(F(FIVj)2)+F((F- F(FIVj))2) Var(F(FI =
=
expected cross-product vanishes by 10.10), so that an equivalent inequality is Fgtr-e'(J'l N2))21 J Fg(r-F(J'j T1))2j. (10.62) The intepretation is simple. V1represents a smaller infonnation set than V2, and if one predictor is based on more information than another, it exhibits more variation and the prediction error accordingly less variation. The exeme cases ;) are &FI = Fand F(FIT) = F(1'), with variances of Vartp and zero respectively, This generalizes a fundamental inequality, that a variance is non-negative, to the partial infonnation case. While (10.61)generalizes from the square to any convex function, (10.62)does
Conditioning
157
not. However, there is the following nonn inequality for prediction errors. 10.28 Theorem If F is T-measurable and N1 i N2 i ;,
IIF-FIFIN2)11,K 2I1F-F(F1Tl)IIp, p k 1. Proof Let
n
n -
F-
=
F(FIV1). Then
F(n lN2)
=
(10.63)
by 10.26(ii),
F- F(r1 N1)-&F1 T2) +&&FIV1)1T2)
= J'-s(rI va).
(10.64)
The theorem now follows, since
IIq F(q IT2)l1p< IIq11,+11F(nIT:2,)11,f 2IIq11p
(10.65)
-
by, respectively, L1E.
the (Minkowsk.i and conditional Jensen inequalities, and the
.
10.6 Conditional Distributions The conditional probability of an eventA e # can evidently be defined as #(4 IN) F(lx 1N), where 1x() is the indicator function of A. But is it therefore meaningful to speak of a conditional distribution on (D,@), which assigns probabilities PA 1N) to each A e @? nere are two ways to approach this question. First, we can observe straightforwardly that conditional probabilities satisfy the axioms of probability except on sets of probability 0 and, in this sense, satisfactorily mimic the properties of true probabilities, just as was found for the expectations. Thus, we have the following. =
10.29 Theorem (i) P(A IN) k 0, al1 A e F. (ii) Pf IN) 1 a.s. (iii) For a countable collection of disjoint sets Aj e @, =
#
UAjIT
=
X#(4jI 9') a.s.
(i),suppose 3 G e N with
Proof To prove
(10.18),
jaudp
=
PA IV)())
Jg#(4 I@4dP <
(10.66) 0 for a11(t) e G. Then, by
<
(10.67)
0,
which is a contradiction, since the left-hand member is a probability. To prove (ii), note that #(f1 lV) is N-measurable and 1et G+ e N denote the set of (l) such G+, that #(fl V)4)) > 1. Suppose #46*) > 0. Then since G+ rn =
#(&*)
=
josdp <
jo.#(QIT)##
=
jawezdp =
#(G+),
(10.68)
Probability
158
which is a contradiction. Hence, #(G+) 0. Repeating the argument for a set G- on 0. For (iii), (10.18)gives, for any which #4f1 I5')() < 1 shows that #(G-) G 6 V, =
=
J(,#(U,A'
jIU4dP
(10.69) Xyjocvbdp' jocvjpdp jvjacpdp =
=
=
since the sets G /'7 Aj are disjoint if this is tl'ue of the Aj. By definition there exists a version of #(l N) such that V G 6 5',
jocujdp Js#(X/IV) dP,
(10.70)
=
and hence
jop-v' /1 T)#/'
=
XJSPIA/IU4dP
=
./
V)) (Xy #(X/I dP.
jo
(10.71)
The left- and right-hand members of (10.71)define the snme measure on W(see 10.5)and hence #(U.?A.j lN) = Q#(AjIT) a.s. by the Radon-Nikodym theorem. w But there is also a more exacting criterion which we should consider. That is, does there exist, for fixed (z), a p.m. g,r,lon (f,F) which satisfies yu(A)
P(A l534)), each A e 5
=
(10.72)
E G where #(C) 1? lf this condition holds, the fact that conditional for all expectations and probabilities behave like regular expectations and probabilities requires no separate proof, since the properties hold for jt(o. lf a family of p.m.s lg,(o, e fJ satisfying (10.72)does exist, it is said to detine a regular conditional probability on V. However, the existence of regular conditioning is not guaranteed in every case, and counter-examples have been constructed (see e.g. Doob 19532 623-4). The = problem is this. ln (10.66),there is allowed to exist for a given collection (A e TJ an exceptional set, say Ca with PCa) 0, on which the equality fails. This in itself does not violate (10.72),but the set Ca is specific to and uncountable countable subsets number of there typically 5, we since are an Poaj would required both be for 0, be that guarantee to a p.m. cannot as g,fo and to satisfy (10.72). =
.4
=
.d,
.4
=
This is not a particularly serious problem because the existence of the family lju) has not been critical to our development of conditioning theory, but for certain purposes it is useful to know, as the next theorem shows, that p.m.s on the line do admit regular condittonal distributions. *
10.30 Theorem Given a space L1,5,P) and a subfield N c @,a random variable F has a regular conditional distribution defined by Fr/l
for
e C with #(C)
N)() =
=
j ,
lT)(), Fg(. IV)()
#((-x,y1
where
y e R, is a c.d.f. for all
(10.73) e f1.
Conditioning
159
Proof Write F(o*(y)to denote a version of #((-x,y1 I9')(). Let Mq denote the set of ) such that F' ri) > F(o*(rj)for ri, rj e :) with ri < (/ Similarly, 1et Ri denote the set of (l) on which limn-yx/trj + 1/n) # FLlril,ri e e. And tinally, 1et L denote the set of those ) for which F(l(+x) # 1 and F(o*(-x) y: 0. Then C (Uj#C is monotone and right-continuous at rn Lc is the set of on which FfLvl rn F(l(+x) with 1 and Ff)(-x) 0. For y e R let a11rational points of the line, .
-&lc
=
=
Fg'@) y e ()
=
,
Fr(. I5')()
=
F.(.y+), y e R
(,)e c,
-0
(10.74)
Otherwise,
G(X, where G is an arbitrary
c.d.f. ln view of 10.29, #(Mj) 0 for each pair i,j, PRi4 0 for each i and PL) 0. (1fneed be, work ln the completion of the space to define these probabilities.) Since this collection is countable, PC) 1, and in view of 8.4, Fy(. IV)() is a c.d.f. which satisfies (10.73),as it was required to show. . =
=
=
=
lt is straightforward, at least in principle, to generalize this argument to multivariate distributions. For # e B it is possible to write F(1s I5')(t,))
=
V)() jedkyf
(10.75)
a.s.,
and the standard lgument by way of simple functions and monotone convergence leads us full circle, to the representation
F(FI9')()
+x
J
=
-X
y#&@l
syy(y,a.,.
(10.76)
ctm, we have constructions to parallel those of j10.1. Since no restriction had to be placed on the distribution to obtain this result, we have evidently found a way around the difficulties associated with the earlier detinitions. However, Fy(. IV)() is something of a novelty, a c.d.f. that is a random element f'rom a probability space. lntuitively, we must attempt to understand this as representing the subjective distribution of F(@ in the mind of the observer who knows whether or not ) e Gfor each G e V. The particular case Fy(. IN)4) is the one of interest to the statistical modeller wheh the outcome ) is realized. Manyrandom variables may begeneratedfromtheelements of (D,1,#), notonly the but also variables outcome itself in the bivariate case the pair F(),X() and A11 have the quantiles of Fv@ these such as F(FIm(), to be thought I of as different aspects of the same random experiment. Let X and F be r.v.s, and N a subfield with V c Rx c(A') and V ? Rv c(F). We say that X 4nd F are independent conditional on V if If V
=
-
-
m().
=
Fxrtmyl 5')
=
Fx@l S'IFI-I.yI V) ms.
=
(10.77)
Probability
160
This condition implies, for example, that f(Xl'IV) g(.,(o) be the conditional measure such that g,(o(lX 6
(--,A1,
i'
(--,A'1 1)
6
=
=
F(XIV)F(FIV) a.s.
Fxvx'y
Let ju
=
1V)().
with (t) fixed this is a regular p.m. by (the bivariate generalization ofl 10.30, and gwotAfa #) g(o(A)g(o(#) for eachA e Rx and B e Xy, by 10.21. ln this sense, the subtields Rx and Ry can be called conditionally independent. =
10.31 Theorem.
lf X and F are independent conditional on V, then
f(l'1 Rx4
F(1'l N) a.s.
=
(10.78)
Proof By independence of Rx and Rv under JxftFl
XaYll/t
=
JxFtfltto
=
u we can write BtX)JFtV' X e Rx.
(10.79)
This is equivalent to
&1z&F1Jx)l 9')(t,)) F(lxl'l 5')() =
T) IT)() = F(1xF(1'I
were tlae tirst equality also with respect to #, noting f
JxF(FIRdp
=
a.s.(#1,
(10.80)
follows from 10.26(i) and 10.10. lntegrating over f e V, using 4.8(ii) and the LIE, we arrive at JxF(Fl U4dP. A e Rx.
jgk'dp =
This shows F(Fl Rx) is a version of
F(FIN), competing
(10.81)
the proof.
F(FIRx) is in principle
.
Xvmeasurable, it is in fact almost surely I#J equal to a V-measurable r.v. Needless to say, the whole argumentis symmetric in X and Rp The idea we are capturing here is that, to an observer who possesses the information in V (knowswhether (t) e G for each G e V), observing Xdoes not yield any additional information that improves his prediction of F, and vice versa. This need not be true for an observer who does not possess prior information. Equation (10.77) shows that the predictors of F based on the smaller and larger information EY4 a.s., so sets are the same a.s.(#), although this does not imply F(Fl R that X apd F are not independent in the ordinary sense. Thus, while
=
11 Charqcteristic Functions
11.1 The Distribution of Sums of Random Variables Let a pair of independent r.v.s X and F have marginal c.d.f.s Fx@) and Fy@). The c.d.f. of the sum J'P X+ F is given by the convolution of Fx and Fs the =
function
Fx*Fy(w)
+x J..Fx(w
=
(11.1)
-ytdFv(9.
11.1 Theorem If r.v.s X and F are independent, then Fx*Fy(w)
#(X+ F f w)
=
FI'*Fx(w).
=
(1 1.2)
Proof Let 1w(my) be the indicator function of the set fmy:x K w y), so that F(1w(X,F)). By independence Fxty) FxxqFvy), so this is #(X+ F < w) -
=
=
Js21w@,A')F@,A')
+x
P+x
J..J
dFyIyl
.x1w@,.')tFx(I)
=
+x
w-y J..JFx@)
J..
=
dFyvl
+x
Fxtw
=
-
y)#Fy(y),
(1 1.3)
-X
where the first equality is by Fubini's theorem. This establishes the first equality in (11.2). Reversing the roles of X and i' in (11.3) establishes the second. w For continuous distributions, the convolution
f
=
fx*fyof
p.d.f.s
fx and h
is
+x
j-.fxw -').fy(')*,
f(w)
=
such that
Jlx/'ttlx
=
(1 1.4)
F(w).
11.2 Example Let X and Fbe independent drawings from the uniform distribu1m,1j@).Applying (11.4)gives tion on 0,11, so that fxx) =
/k+r(w)
=
1
llw-l,wj#y.
()
that the graph of this function forms an isosceles triangle lt is with base (0,21and height 1. n easily veritied
Probability
162
This is the most direct result on the distribution of sums, but the formulae generated by applying the rule recursively are not easy to handle, and other approaches are preferred. The moment generatinghlnction (m.g.f.)of X, when it exists, is jedFx), Eeth Mxtf) t e R, (11.6) =
=
where e denotes the base of natural logarithms. (Integrals are taken over (-x,+x) unless otherwise indicated.) If X and i' are independent, Mx-vvt)
jjetx*'gdt-xx4dll-y,
=
=
jetxdnxjetydFyy)
= Mx(f)Mg(/).
(11.7)
This suggests a simple approach to analysing the distribution of independent sums. The difficulty is that the method is not universal, since the m.g.f. is not defined for every distribution. Considering the series expansion of e all the moments of X must evidently exist. The solution to this problem is to replace the variable f by it, where i is the imaginary number, VX. The characteristic function (ch.f.)of X is defined as ,
9x(f) =
1l
.2
Ee
itX
)
itx
(1 1.8)
e dFx).
=
Complex Numbers
A complex number is z a + ib, where a and b are real numbers and i V-X.a and b are called the real and imaginary parts of the number, denoted a = Re(z) and b lm(z). The complex conjugate of z is the number i' a ib. Complex adthmetic is mainly a matter of carrying i as an algebraic unknown, and replacing 2 j)y j /3 i 4 by 1 e tc wherever these appear in an expression. by One can represent z as a point in the plane with Cartesian coordinates a and b. The modulus or absolute value of z is its Euclidean distance from the origin, =
=
=
=
-
-
,
-,
,
.,
lz l
=
(zD 1/2
=
a
2
+
jp
1/2
)
(1j
.
.
9)
Polar coordinaies can also be used. Let the complex exponential be defined by
efo = cos 0
+ jsin
8
(11.10)
for real 0. All the usual properties of the exponential function, such as multiplication by summing exponents (according to the nzles of complex arithmetic) go through under this definition, and
Ie i l
=
+ sin2e)1/2 (coszo
=
1
(11.11)
Characteristic
Functions
163
for any 0, by a standard trigonometric identity. We may therefore write z Iz Iei where Retz) Iz Icos 0 and 1m(z) Iz Isin 0. Also note, by (11.11),that =
=
=
.Re(z)+Im(z) j Ie z 1 j =
@Re(z)
jef1m(z)j
=
=
@Re(z)
(j.j. jg;
.
.
random If X and i' are real random variables, Z X+ iY is a complex-valued variable. lts distribution is defined in the obvious way, in terms of a bivariate c.d.f., F@,y). In particular, =
EZ)
(11.13)
F(A') + iElY).
=
Whereas EZ4 is a complex variable, FIZI is of course real, and since 1zl < IXl + Il'I by the triangle inequality, integrability of X and F is suftkient for the integrability of Z. Many of the standard properties of expectations extend to the complex case in a straightforward way. One result needing proof, however, is the generalization of the modulus inequality.
11.3 Theorem
If Z is a complex random variable,
Proot Consider a complex-valued
4F(Z) I S FI I .
.
simple nv.
(a./+ lllsy, zn) 7'j=L -
where the xj and I are real non-negative constants and the Ej e 5 for j constitute a partition of f1. Write Pj F(1sy). Then
(11. 14) =
1,...,p
=
2
IEznl) 1
-
2+
77xjpj
=
2
77fjjpj
+ f5,5/,7? 77(a??+ 7777(%a,+ fbbrjn,
( 11.15)
j*k
j
whereas CEIz(n)I)2
=
=
1/2
>7(a,?+ p,?)Pj
2
./ + + jk2)1/2# j pk. X (( + f5?)#? X X (%?+ Y?)'/2(ak2 J
J
J
j*k
./
(11.16)
The modulus inequality holds for Zn) if
0S =
(E IZ(n) I)2
-
2
IF(z(n)) I
p2,)1/2(%a,+ 1%)1#./#,. 7777E(a?+p,?)''2(a,2+ -
J
(1 1.17)
The coefficients of Pjpk in this expression are the differences of pairs of nonnegative terms, and these differences are non-negative if and only if the differences of the squares are non-negative. But as required,
Probability
164
+ p2,) (%a.+ Ip.)2 (a,?+ If)(a2k -
,
a,?p2,+ a%?
=
=
zwakjf'lk
-
(ay;'J. a.I$)2z O
(11 18)
-
.
.
This result extends to any complex r.v. having non-negative real and imaginary Z X+ iY, using 3.28, and invoking the parts by letting Z(n) = Xn) + in) monotone convergence theorem. To extend to general integrable nv.s, splitXand F X* + iY* into positive and negative parts, so that Z = Z+ r, where Z+ X+ 1'+ A?'-+ iT, with X- k 0 and lr- k 0. Noting that 2 0 and k 0, and Zwith '1k
=
=
-
=
IEZ) I < IE(Z+ + r) l < e'lz++ r I FIzl
(11.19)
=
completes the proof.
.
11.3 The Theory of Characteristic Functions We are now equipped to study some of the properties of the characteristic function x(/). The fact that it is detined for any diskibution follows from the fact 1 and Eeth is finite regardless of the 1 for all x; E( Icfl'fl ) that Ie x I distribution of X. The real and imaginary parts of #x(J) are respectively Ftcos t.k and fsin tX4. =
=
k< 11.4 Theorem If E IA-I
x,
then
k
d px(/)
d
=
Eiuk
k itx
e
).
(1 1.20)
Proof
t-bht #x(J) = h
eixt'ht
+=
-
eitx
-
#(x),
h
-x
(1 1.21)
where, using (11.10), e
t-h)
h
-
e
itx
=
cOs
xt
+
h4 h
-
cOs
tx
+
j is n
.7
(/ + 4 -
siu tx .
h
The limits of the real and imaginary terms in this expression as h sin tx and i(x cos /x), so the limit if the integrand in respectively -->
-x
is
xi cos tx
-
sin fx)
=
fccos tx
+
i sin tx)
=
ie
0 are
(11.21)
it: .
since1tfxlefr-lIxl
the integral exists if A'C-Y'I This proves (11.20) for complete 1. To the case k the proof, the same argument can be applied itx inductively to the integrands (ix)k-1e j or k z a s =
,
=
.
,
,...
It follows that the integer moments of the distribution repeated differentiation with respect to t.
can be obtained by
Characteristic lf Fl#! k <
11.5 Corollary
Functions
165
th en
x,
24x(f)
k k = i EX ). n
dtkx
(1 1.22)
An alternative way to approach the last result is to construct a series expansion of the ch.f. with remainder, using Taylor's theorem. Tlzis gives rise to a very useful approximation theorem.
11-6 Theorem lf
FI.Y1k <
then
x,
k
k 77tffytil (1)x(/)j=0 j! Proof A function
,
(1 1.23)
.
differentiable k times has the expansion
which is
f
+1
l I/XI IfXl k! (k+ 1)!
f E min
-
./*(a/)
1 Jz''(0), 3 z'(0) , +
f(f) f(0) + f '(o), + =
where 0 f
; 1. The expansion
(x
of
#
e itx where yk
=
iksgntfx)
k
jwx
(e
ft)
e
=
+
...
:!
p,
)
it?
glves
#
'
itxf
y'l y!
=
+
6
z
+
Itx I yk,
(1 1.24)
k!
j; ayjtj
-
1
sgnt/x
tx k 0
,
=
.
- 1 tx
<
,
0
Applying (11 10) and (11 11), we can show that Iyk I (2 2cos a/x)1/2 < 2. However, by extending the expansion to tenn k + 1, we also have =
.
.
k
e itx
k+1
'
itx/
=
g! .
jzzo
+
ltx I
j or () < (g s j and where zk ik+1sgnt/x) e (1 1.24) and (11.25) hold, we may conclude that ,
k
d
k
'
itxf
itx y.l ./! -
c
min
(1 1.25)
k + 1)! zk,
k+1 ift.T
=
-
2 Itx I k!
jzkj
1. Gjven that both of
=
1+l
,
Itx I
(k+
1)!
The theorem now follows on replacing x with the r.v. #in ations and using the modulus inequality:
(1 1.26)
.
(11.26),taking
expect-
Probability
166 j4e'tx./l k 9x(/)77 !
t ./
-
s
,
e
k
i
-
(/m./ j--,j y!
:7
+1
k
f E min
l I/.X'1 I/-YI k! (k+ 1)! ,
.
(11.27)
w
There is no need for FIXl k+1to ex i st for this theorem to hold, and we can think of it as giving the best approximation regardless of whether I/XI is large or small. To interpret the expectation on the right-hand side of (11.27), note that, for any pair of non-negative, measurable functions g and ,
Ftminf#tm,tml)
inf F(#(m1,4
=
+
Ae 3
tmlxc),
(11.28)
the infimal set being the one containing those points x on which particular,for any s 2 0, the set A ( IXI> EJbelongs to the class the intimum in (11.28)is taken, and we get the further inequality,
< hlx). In
.(x)
=
.
E m1n
2 Itx
k'!
Ik Itxl ,
k+1
k
k+1
2 I/aYI f E 1( jxjlsj k!
k + 1)!
over which
+
2/ E A'1k1f IxI> .:) ) k! I
E
I?-YI lt jxjssj
(k+
1) !
k+1E,k+1
+
It I
k + 1) !
(1 1.29)
S
2/ I/'I F1x'1 kt F(lxI '1. ( IxI>e) ) + 1)! (.+ k! k+1
The second alternative on the light is obtained in view of the fact that k k k+11( IxI
kax+b
=
ibt E efr(cx+h) ) = e #x(J/).
The second is the counterpart variables X and F,
of
(11.7).For
a pair of independent
(1 j ao .
random
jjeitx*'tdlkxtdFy'
#x+y(/) =
=
jeitxdFxxljeivdFvlp
=$x(f)#,'(/).
(11.31)
An interesting case of the last result is F -X' where X' is an independent drawing from the distribution of X. The disibution of X- X' is the snme as that of =
Characteristic X'-X,
Functions
167
symmetric about 0. The ch.f. of X X' is real,
and hence this r.v. is
-
since
#x(f)9x(-f) I4)xtfl I2 =
(11 32) .
,
Ee-ith in v iew o f the fact that bx#aV).It can be verified from the with real ch. f., a1l the existing odd-order moments expansion in (11.23) that a be the trademark distribution. of symmetric zero, must a =
=
:=1Xj where (X1,...,aL Considering more generally a sum S 1 are a totally recursive yields collection, of application independent (11.31) =
(1),(/)f=1 11(l)x(/).
( 11.33)
=
To investigate the distzibution of S, one need only establish the fonnulae linking the ch.f.s with the relevant c.d.f.s (or where appropriate p.d.f.s) which are known for the standard sampling distributions.
11.7 Example For the Poisson distribution (8.8),
9x(f',)
Ee
=
itX
)u
)
x
(j.
77w
--
e
=
;:
X!
e
i,
e
=
)k(eyy..j j .
(1 1.34)
n
11.8 Example In the standard Gaussian case, (8.10), 9z(J)
=
Ee
itLl
)
1 =
Completing the square yields itz
z(f)
=
e-
/2
+oo
/-2l
-
z2/2
-
1
+=
27:
x
e
itz
2
/2
-z
jf)2/2
-(z
=
dz.
-
(1 1.35) /2/2
-
and hence
-(z+ff)2 -/2
dz
e
-x
=
e
.
(1 1.36)
(The integral in the middle member has the value ulV for any choice of t, note.) Accordingly, consider X = cZ+ po whose p.d.f. is given by (8.24).Using (1 1.30), we obtain
(xlt, . poc) Equation With g,
e/w-c2p/z
.
(11.22) can be used to verify the moment =
(11.37)
.
0 the ch.f. is real, reflecting
formulae given in 9.4 and 9.7.
the symmetry of the distribution. n
11.9 Example The Cauchy distribution (8.11)has no integer moments. The ch.f. turns out to be e -lfI w hich is not differentiable at t = 0, as (11.22) would lead us to expect. The ch.f. for the Cauchy family (8.15)is #x(J;
V, : )
=
e
frv-lfl .
E1
The ch.f. is also defined for multivariate disibutions. (??zx 1) the ch.f. is
(j j ag; .
For a random vector X
Probability
168
expfif'fl),
#(f)
=
(11.39)
where t is a v-vector of arguments. This case will be especially important, not least because of the ease with which, by the generalization of (11.30), the ch.f. +.d (kx 1), can be derived for an affine transformation of the vector. Let F BX where B (kx m) and d k x 1) are constants, and then we have =
g(/)
F(exp(ff'F))
=
exp(I'/'#)F(exp(//'#.Y))
=
=
expfit'dklxlB't).
(11.40)
11.10 Example Let X m x 1) be multinormal with p.d.f. as in (8.36).
ch.f. is
rrhe
#x(f;M,E)
1
=
+=
rn/z
(2a) 1x j = exp l it'p
-
+= ...
1/2
-.
expl/'.t
g)'E-1(x
@
-
-
..
-
M)l#-4r
rlfzy.z )
(1 1.41)
.
The second equality is obtained as before by completing the square: 1 g(x it 'x j/zD ,1(x g j&)I-1(x g j&) itp g,)'E-l (x p) -
where it can be shown mI1
(2,n) Ix j
1/2
=
-
-
-
-
-
-
-
-
,
of the last term integrates over R to
that the exponential
n
.
1 1.4 The lnversion Theorem Paired with (11.8) is a unique inverse transformation hom #4/)to F@), so that the ch.f. and c.d.f. are fully equivalent representations of the distribution. The clef step in the proof of this proposition is the construction of the inverse transfonnation, as follows.
11.11 Lemma If
9(f)is defined
Fb)
-
F(J)
by
1 1im 2a F->x
=
(11.8), then yn
--
e
ittl
--
e
-
itli
(dt
it
..w
for any pair a and b of continuity points of F,' with a generalization of this formula is AF(xl
,...,.q)
=
1 m
za
1im T-+x
z'
z' ...
-w
-z
k
-jx./
e
J''I
jzzj
-
e
(11.42) <
b. The multivariate
-j.%+Ax.j)
jjy
(1 1.43) xtxj...xktt,---ntktdtb---dtk. wher AF(x1,...,.q) is defined in (8.31)and the vertices of the rectangle based at the point x1,...,ak, with sides Lxj > 0, are a11continuity points of F. n --
1.,
wxza..1
xrolon:
f.n
/1 1 q0'
Tt onn
lw
xzorifiofl
Ilqin
e (11
10
thnt
Characteristic
169
-itb
-ita
d
Functions
-d
--: b a as t
0.
-->
-
it
The integrals in (11.42)and (11.43) are therefore well defined in spite of including the point t 0. Despite this, it is necessary to avoid writing (11.42) =
as '-ita
Fb)
F(J)
-
'-itb
+x e
=
e
-
(11.44)
(f)Jf,
lnit
because the Lebesgue integral on the right may not exist. Forexample, suppose the a'it.j j random variable is degenerate at the point 0; this means that (/) e and =
F
1 21
-
e
ita
,
-itb
e
-
1 r j. 7: 1 t
log F,
--dt
dt k
it
-w
=
-
-
( 11.45)
so that the criterion for I rbesgue integrability over (-x,+x) fails. However, the limits in (11.42)and (11.43) do exist, as the proof reveals. Proof of 11.11 Only the univariate case will be proved, the multivariate extension being identical in principle. After substituting for () in (11.43) we can interchange the order of integration by 9.32, whose continuity and a.s. boundedness requirements are certainly satisfied here: '-itb
-ita
'
d
r
dt
lnit
=
= Using
'-itb
-ita
F
-d
e
e
-
-F
21f/
+x
T
-x
-F
d
oo
-x
itx-a)
j/x
e dF@) dt
itx-b)
-e
t dF(x)
lxit
.
(1 1.46)
(11.10.), ' e itx-a)
-.w
-
e
itfx-b)
t
lxit
z'Sj n yx
=
F sjn j(x
.u)
t
'lt
()
-
()
zt
.y)
t,
(1 1.47)
noting that the cosine is an even function, so that the terms containing cosines x (which are also the imaginary terms) vanish in the integral. The limit as F standard fonuula of this expression is obtained from the -..j
jo'='sin
t t
dt
=
zIl,
tx
()
(:
,
-1/2, tx
>
0
.()
<
(j j.4g) 0.
Probability
170
yields the result
(11.46)
Substituting into l.x
e
fftx-a)
il(2'-8
-d
t
lxit
=
0, x
<
a or x > b
l 7:
=
c or x
<
x < b.
x
1, a
b
=
(1 1.49)
Letting F-->x in (11.46) and applying the boundedconvergence theoremnow gives -ita
lim
F-px
T -
-itb
e
-
+x
e
(tldt
lgit
w
-x
(lfc) +
= r1(F(:)
1(s) + ta,bdFx)
F(b-)
+
Fa)
-
-
F(J-)),
(11.50)
which reduces to F(b4 Fa) when a and b are continuity points of F. w -
Lemma 11.11 is the basic ingredient of the following key result, the one that primarily justifies our interest in characteristic functions.
theorem Distributions having the same ch.f. are the same.
11.12 Inversion
Proof We give the proof for the unvariate case only. By (11.42), the c.d.f.s of the two distributions are the same at every point which is a continuity point of both c.d.f.s. Since the set of jump points of each c.d.f. is countable by 8.3 their union is countable, and it follows by 2.10 that the set of continuity points is dense in R. It then follows by 8.4 that the c.d.f.s are the same everywhere. . A simple application of the inversion theorem is to provide a proof of a well known result, that affine functions of Gaussian vectors are also Gaussian.
11.13 Example Let,f Ng,X) m x 1) and F= BX d n x 1) are constants. Then by (11.42),
-d
n
-
#g(f)
=
exptff'#lFtexpt#'sfll
= expfit'Bg If ranktS'xS)
n,
=
(implyingn -(y
.f(J)
expf
=
x 1) where B n x m) and
-
Bg
+d)
-
qt'B'SBtj
(11.51)
.
< m), 11.12 implies that F has p.d.f. -
d) '(#xn')-l(y tz I#IN
(2r)n
,
1/a
I
-Bg
-#))
.
(1 1.52)
< n, (11.51) remains valid although (11.52) is not. But by the same lf rankt#xs') arguments, every linear combination c'F, where c is p x 1, is either scalar Gaussian with variance c'BSB'c, or identically zero, corresponding to the cases B'c 0 respectively. ln this case F is said to have a singular 0 and #'c Gaussian distribution, I::I :#
=
Characteristic
Functions
171
11.5 The Conditional Characteristic Function Let F be a F-measurable r.v., and 1et V c @. The conditional ch.f. of 4)ysg(f), is for each t a random variable having the property
jgtrj
g(/)##
jaeitp,
all G e V.
=
FIV,
(11.53)
The conditional ch.f. shares the properties of the regular ch.f. whenever the theory of conditional expectations parallels that of ordinary expectations according to the results of Chapter 10. Its real and imaginary pms are, respectively, the V-measurable random variables Ftcos /F1 V) and fsin fFl N). It can be expanded as in 11.6, in tenns of the existing conditional moments. lf X is V-measurable, the conditional ch.f. of X+ F is #x+gIv(/) e itxpjeityks; by 10.10. And if i' is Vl-measurable and N and V1 are independent subfields, then =
#i'l5'(J)
l'(/)
=
a.S.
The conditional ch.f. is used to prove a useful inequality due to von Bahr and Essen (1965).We start with a technical lemma which appears obscqre at first sight, but turns out to have useful applications.
11.14 Lemma Suppose
FIZIr
<
0
cx),
<
2. Then
<
r
A'(rlj..+x1 Re(#z(/)) -
Fl W here
K
=
zl
r
=
l+rfJfl-l
(J+*(1cos ?g)/1 u I -
=
-x
(1 1.54)
t
1+r
jyj
a-1r'(r+ j) sin ra/2.
n
The last equality, with F(.) denoting the gamma function, is a standard integral formula for 0 < r < 2. Proof The identity for real z,
lz lr
+oo
=
1 cos zt dt, 1+,
Ar)J.. -
(1 1.55)
jjj
is easily obtained by a change of variable in the integral on the right. The lemma Acos tZ4. . follows on applying 9.32 and noting that Re(9z(/)) =
Tl'lis equality also holds, for ) e C with PC) = 1, if F( 1Zl rl N)((t))and 9zls(J)()) r are substituted for Fl zl and #z(f).In other words, the conditional rth moment and conditional ch.f are almost surely related by the same formula. So consider fv-bounded r.v.s z and X, where Z is T-measurable, and X is N-measurable for N c 5. Suppose that #z!v(/)is a real r.v. almost surely. Then for each e f1,
1
-
Re@x+zIs(f))()
=
1 Retd -
itX
#zIs(f))())
= 1 (cosJX())(zIg(/)()) -
S
(1
-
cos /X())
+
(1
-
9zIs(J)()),
(1 1.56)
Probability
172 thedifference between whichis non-negative
the last two members being (1 cos /X()))(1 for all Hence, for 0 < r < 2, -
#z1g(/)()))
-
.
+x 1 Re@x+zIs(/))
#(r)J.. += f #(r)J.. Ar)J.. -
E IX+
zl
rl
N)
=
t
j+r
jjj
1 cos tx
+=
-
1.r
jyj
l#l
=
'
+
t
1
zl v(/)
-
+
jyj
t
j+,
Izl r IV), a-s.
E
(1 1
.57)
and taking expectations through yields
FIA-+zIr < FIXIr
f'l
+
ZIr.
(11.58)
For the case 0 < r S 1 this inequality holds by the cr inequality for general Z and X, so it is the case 1 < r < 2 that is of special interest here. Generalizing from the remarks following (11.31), the condition that (zIv(/) be real a.s. can be fulfilled by letting Z Y- Y', where F and F' are identically distributed and independent, conditional on V. Note that if R = c(F), then =
F(F'IX)
F(F'IT),
=
(11.59)
a.s.,
by 10.31. Identical conditional distributions means simply that Fv(. 1V) Fg(. IV) a.s., and equivalently that (gIs(/) #glv(/) a.s. Hence
=
=
-fry'
#r-r'1p(/)
=
ffye
Ee
ls)
= Ee itvls )eje-itvj v) IrIs(/)l 2 a.s.,
=
(j j
,
where the right-hand side is a real r.v. Now, for each identity can be verified:
2(1
Re(#y,v(/)())
-
Applying
=
1
-
la + Il
g1s(/)()
-
(t)
-
6o)
e f1, the following
9rIs(f)((.0)
2
I
(11.61)
.
and 11.14, and taking expectations, this yields the inequality
(11.60)
2A'1FIr
r,
Fl Y-
F'I
0
<
<
r
(11.62)
2,
noting that the difference between the two sides here is the non-negative g(/) 2/j J'j 1+rJ/ Of r #(r)J+=F1 1 9y.1 l These arguments lead us to the following conclusion.
function
-
,
.
..x
11.15 Theorem Suppose F(l'IV) = 0 a.s. and X is V-measurable where V i R
c(F), and both variables
FI.Y+ F1r
=i
are Lr-bounded. Then
r s FIXI
+
2FIFIr,
0
r
(11.63)
2.
Proof Let F' be independent and identical with F, conditional on V. Applying (11.59), these conditions jointly imply f(F'I R) &F'I V) F(FI9') 0. Noting =
=
=
Characteristic
Functions
173
that X+ F is A-measurable, it follows by 10.19 (in applying this result be careful to note that R plays the role of the subfield here) that E IX+
r
FI
<
FIX+ (i'- r) I
'.
(11.64)
The conclusion for 1 < r < 2 now follows on applying (11.58)for the case Z F- i'', and then (11.62). The inequality holds for 0 < r f 1 by the cr inequality, and for r 2 from elementary considerations since Elk'.,Y'I0. In these latter cases the factor 2 in (11.63) can be omitted. . =
=
=
This result can be iterated, given a sequence of r.v.s measurable on an increasing for sequence of c-tields. An easy application is to independent rov.s A'1,...,aL, 2,...,n. 0 certainly holds for t which the condition E(Xt Ic(-Y1,...,m-1)) ntzzkxt, and 11.15 c(X1,...,Xn) yields Letting Sn c(&) =
=
=
=
E IsnIr < A'l
'n-t
Ir
+
< 2XA'ImI r,
2Fl Xn Ir
0 < r < 2.
(1 1.65)
>1
If the series on the majorant side converges, this inequality remains valid as n --> x. lt may be contrasted for tightness with the cr inequality for general Xt, (9 62). In this case, 2 must be replaced by nr-1 for 1 < r S 2, which is of no use for large n. .
lll THEORY OF
STOCHASTIC PROCESSES
12 Stochastic Processes
12.1 Basic Ideas and Terminology Let (( 5 #) be a probability space, let 'T be any set, and let RV be the product space generated by taking a copy of R for each element of 'F. Then, a stochastic process is a measurable mapping x: f r--yR'T where ,
,
j
xttDl
=
l-V((,)), ':
(12.1)
e
'T is called the index Jtrf, and the r.v. Xs() is called a coordinate of the process. A stochastic process can also be characterized as a mapping from x'T to R. However, the significant feature of the definition given is the requirement of joint measurability of the coordinates. Something more is implied than having Xs(() a measurable nv. for each 1. Here, 'Ir is an arbitrary set which in principle need not even be ordered, although linear ordering characterizes the important cases. A familiar example is T' ( 1,...,k) where x is a random k-vector. Another important case of 7 is an interval of R, such that x()) is a function of a real variable and R 'T the space of random functions. And when 'T is a countable subset of R, x fXv()), G 7-) defines a stochastic sequence. Thus, a stochastic sequence is a stochastic process whose index set is countable and linearly ordered. When the Xz represent random observations equally spaced in time, no relevant information is lost by assigning and a linear ordering through ENor Z, indicated by the notations f-V()))''*-x.The definition does not rule out 1- containing information about distances between the sequence coordinates, as when the observations are irregularly spacd in time with a real number representing elapsed time from a chosen origin, but cases of this kind will not be considered explicitly. Fnmiliarly, a time series is a time-ordered sequence of observations of (say) economic variables, although the term may extend to unobserved or hypothetical variables, such as the errors in a regression model. Time-series coordinates are labelled t. If a sample is defined as a time series of finite length n (or more generally, a collection of #uch series for different variables) it is convenient observato assume that samples are embedded in infinite sequences of tions. Various mathematical functions of sample observations, statistics or estimators, will also be well known to the reader, characteristically involving a summation of terms over the coordinates. The sample moments of a time series, regression coefficients, log-likelihood functions and their derivatives, are standard examples. By letting n take the values 1,2,3,..., these functions of n observationsgeneratewhatwemay call#crfvdtfsequences.The notionof asequence =
,
':
=
(.V((t)JO
':
dpotential'
Theor.y of Stochastic Processes
in this case comes from the idea of analysing samples of progressively increasing size. The mathematical theory often does not distinguish between the types of sequence under consideration, and some of our definitions and results apply generally, but a clue to the usual application will be given by the choice of index symbol, / or n as the case may be.
A leading case which does not fall under the detinition of a sequence is where 'T is partially ordered. When there are two dimensions to the observations, as in a panel data set having both a time dimension and a dimension over agents, x may be called a random .#e/J. Such cases afe not treated explicitly here, although in many applications one dimension is regarded as tixed and the sequence notion is adequate for asymptotic analysis. However, cases where 'T is ither the product set ; x(N, or a subset thereof, are often met below in a different context. A triangular stochastic array is a doubly-indexed collection of random variables,
X1l Xz1 X51 .Yla X22 X51
11l
(12.2)
X3k1
a=l is some increasing integer compactly written as ( (Akmlmil ) n=1, w he r f/cn )'=' sequence. Array notation is called for when the points of a sample are subjected to scale transformations or the like, depending on the complete sample. A standard Z7=1Var(X,), or some where Xnt = Xtlsn, and sn example is ( lA-n,))=1)=n=j, similar function of the sample moments f'rom 1 to n. e'D
=
12.2 Convergence of Stochastic Sequences Consider the functional expression (Ak())
';
for a random sequence on the space L1,5,P). When evaluated at a point (l) e this denotes a realization of the actual collection real is of numbers generated whenthe outcome sequence, the drawn. lt is natural to consider in the spirit of ordinal'y analysis whether this e f2, we sequepce converges to a limit, say Al. Tf this is the case for every would say that Xn Xsurely (orelementwise) where, if Xnis an T/f-measurable nv. for each n, then so is X, by 3.26. But, except by direct construction, it is usually difficult to establish in terms of a given collection of distributional properties that a stochastic sequence converges surely to a limit. A much more useful notion (becausemore easily shown) is almost sure convergence. Let C ? fl be the set of outcomes such --y
Stochastic Processes
179
.Y()) as n that, for every (t) e C, Xn) x. If #(C) 1, the sequence is said almost surely, or equivalently, with probability one. The notations Xn to converge -->
-->
=
-F.&:
X a.s., and a.s.lim-,, X are al1 used to denote almost sure convergence. A similar concept, of convergence almost everywhere (a.e.),was invoked in connection with the properties of integrals in j4.2. For many purposes, almost sure convergence can be thought of as yielding the same implications as sure convergence in probabilistic arguments. However, attaching probabilities to the convergent set is not the only way in which stochastic convergence can be understood. Associated with any stochastic sequence are various non-stochastic sequences of variables and functions describing aspects of its behaviour, moments being the obvious case. Convergence of the stochastic sequence may be defined in tenns of the ordinary convergence of an associated sequence. If the sequence fExn A'')217converges to zero, th ere is a clearly a sense in which Xn X.,this is called convergence in mean square. Or suppose that for any 6: > 0, the probabilities of the events ( ): IX,,() X()) l < :1 e 5 form a real sequence converging to 1. This is another distinct convergence concept, so-called convergence in probability. In neither case is there any obvious way to attach a probability to the convergent set; this can even be zero! These issues are studied in Part lV. Another convergence concept relates to the sequence of marginal p.m.s of the coordinates, fgwl!', or equivalently the marginal c.d.f.s, (F,,l7. Here we can consider conditions for convergence of the real sequences fgwtAlITfor various sets A e or alternatively, of (F,,(x) J7 for various x e R ln the latter case, uniform or pointwise convergence on R is a possibility, but these are relatively strong notions. It is sufficient for a theory of the limiting distribution if convergence is confined just to the continuity points of the limiting function F, or equivalently (as we shall show in Chapter 22) of gwtAl, to sets A having g,tAl = 0. This condition is referred to as the wdck convergence of the distributions, and forms the subject of Part V. X, or Xn
-->
=
-
-->
-
,
.
12.3 The Probability Model Some very important ideas are implicit in the notion of a stochastic sequence.
Given the equipotency of INand 1, it will suffic to consider the random element (A((l)))T, e f1, mapping from a point of to a point in infinite-dimensional Euclidean space, R=. From a probabilistic point of view, the entire infinite sequence corresponds to a single outcome ) of the underlying abstract probability space. ln principle, a sampling exercise in this umework is the random drawing of a point in R=, called a realization or sample path of the random sequence', we tnay actually observe only a tinite segment of this sequence, but the key idea is that a random experiment consists of drawing a complete realization. Repeated sampling means observing the same tinite segment (relativeto the origin of the index set) of different realizations, not different segments of the same
realization. The reason for this characterization of the random experiment will become clear
18O
Theot'y of Stochastic Processes
in the sequel; for the moment we will just concentrate on placing this slightly outlandish notion of an intinite-dimensioned random element into perspective. To show that there is no difficulty in establishing a correspondence between a probability space of a familiar type and a random sequence, we discuss a simple example in some detail. 12.1 Example Consider a repeated game of coin tossing, generating a random sequence of heads and tails; if the game continues for ever, it will generate a sequence of infinite length. Let 1 represent a head and 0 a tail, and we have a random sequence of ls and 0s. Such a sequence corresponds to the binary (base2) representation of a real number; according to equation (1.15) there is a one-toone correspondence between infinite sequences of coin tosses and points on the unit interval. On this basis, the fundamental space (f,T) for the coin tossing experiment can be chosen as ((0,1),S(0,l)).The form of P can be deduced in an Pttails) 0.5 (i.e.the coin elementary way from the stipulation that Ptheads) is fair) and successive tosses are independent. For example, the events ftails on first toss ) and lheads on Erst toss) are the images of the sets (0,0.5)and g0.5,1)respectively, whosemeasures must accordingly be O.seach. More generally, the probability that the first n tosses in a sequence yields a given configuration 1/2N, of heads and tails out of the ln possible ones is equal in every case to so likely'. The that each sequence is (in an appropriate limiting sense) corresponding sets in (0,1)of the binary expansions with the identical pattern of =
=
tequally
Os and ls in the tirst n positions occupy intervals al1 of width precisely $l2n in the unit interval. The conclusion is that the probability measure of any interval is equal to its width. This is nothing but Lebesgue measure on the half-open interval g0,1).n This example can be elaborated from binary sequences to sequences of real variables without too much difficulty. There is an intimate connection between infinite random sequences and continuous probability distributions on the line, and understanding one class of problem is frequently an aid to understanding the other. The question often posed about the probability of some sequence predicted in advance being realized, say an infinite run of heads or a perpetual alternation of heads and tails, is precisely answered. In either the decimal or binary expansions, all the numbers whose digit sequences either tenuinate or, beyond some point, are found to cycle perpetually through a finite sequence belong to the set of rational numbers. Since the rasionals have Lebesgue measure zero in the space of the reals, we have a proof that the probability of any such sequence occurring
is zero.
Another well-known conundrum concerns the troupe of monkeys equipped with typewriters who, it is claimed, will eventually type out the complete works of Shakespeare. We can show that this event will occur with probability 1. For the sake of argument, assume that a single monkey types into a word processor, and his Ascll-encoded output takes the form of a string of bits (binarydigits). Suppose Shakespeare's encoded complete works occupy k bits, equivalent to kl5 characters allowing for a 3z-character keyboard (upper-caseonly, but including som
Stochastic Processes
181
punctuation marks). This stling is one o f the lk p ossible strings of k bits.
Assuming that each such string is equally likely to arise in k/5 random key presses, the probability that the monkey will type Shakespeare without an error from scratch is exactly 2 -k H ow ever the probability that the second string of lk bits it produces is the right one, given that the tirst one is wrong, is (1 2 '-k)2- w h en the strings are independent. In general, the probability that the monkey will type Shakespeare correctly on the m + llth independent attempt, '&2A11 these events given that the first m attempts were failures, is (1 2 -k) and summing k disjoint, probabilities a11 their 0 yields m over are 1. fmonkey types Shakespeare eventually) ,
.
-
-
.
=
In the meantime, of course, the industrious primate has produced much of the rest of world literature, not to mention a good many telephone books. It is also advisable to estimate the length of time we are likely tcj wait for the desired text to appear, which requires a further calculation. The average waiting time, l-k4m expresse d in units of the time taken to type k bits, is 2-E,1=1rrl(1 2k 1 If we scale down our ambitions and decide to be content with just $TO BE OR NOT TO BE' (5x 18 90 bits), and the monkey takes 1 minute over each attempt, we shall wait on average 2.3 x 1021years. s o the Complete Works don't really bear thinking about. What we have shown is that almost every infinite string of bits contains every t'inite string somewhere in its length', but also, that the mathematical concept of talmost surely' has no difficulty in coinciding with an everyday notion The example is frivolous, but it is useful to be indistinguishable from reminded occasionally that limit theory deals in large numbers. A sense of perspective is always desirable in evaluating the claims of the theol'y. =
-
-
.
=
'never'
.
The first technical challenge we face in the theory of stochastic processes is to handle distributions on R=. To construct the Borel field 000of events on R7 we implicitly endow R= with the Tychonoff, or product, topology. It is not essential to have absorbed the theory of j6.5 to make sense of the discussion that follows, but it may help to glance at Example 6.18 to see what this assumption implies. Given a process x f.il7 we shall write jz 7:k(x) (m,...,A): Rx F-> (jyk =
(
=
.
a)
for each k e EN,to denote the k-dimensional coordinate projection. Let C denote the collection of jnite-dimensional cylinder sets of R=, the sets C
=
(x e
In other words, elements
: ;u@) R oo
e E, E e Bk k e ,
s)
(124) .
of C have the fonn C
=
-1 (F) C:k
(1z 5) .
for some E e T k an d some finite k. Although we may wish to consider arbitrary finite dimensional cylinders, there is no loss of generality in considering the projections onto just the first k coordinates. Any tinite dimensional cylinder can ,
182
Theor.y of Stochastic Processes
be embedded in a cylinder of the form lk -1(F), E e f where k is just the largest of the restricted coordinates. The distinguishing feature of an element of C is that at most a tinite number of its coordinates are restricted. ,
C is a tield.
12.2 Theorem
Proof First, the complement in R= of a set C defined by (12.4)is a-1(Ff) C c = fx e R= : (ak(x)e Ec E e fl k =
,
,
(126) .
which is another element of C, i.e. Cc s C. Second, consider the union of sets C a-1(F') -1 k e C for E, E' G Bk C tp C' is given by (12.4) = Jck E) e C an d C' with E replaced by E tp E ' and hence Ck.l C' e C Tlzird if E e Bk and E' e Bmfor m > k, then Ex? m-ke Bm an d so the argument of the second case applies. . =
,
,
.
.
,
,
Fig. 12.1 It is not easy to imagine sets in arbitrary numbers of dimensions, but good visual intuition is provided by thinking about one-dimensional cylinders in R3 Letting @,y,z)denote the coordinate directions, the one-dimensional cylinder generated by an interval of the axis is a region of 3-space bounded by two infinite planes at right angles to the x axis (see Fig. 12. 1 for a cut-away representation). A union of x-cylinders is another x-cylinder, a collection of But the union and intersection of an x-cylinder with parallel y-cylinder are two-dimensional cylinder sets, a cross' and a column' respectively (see Fig. 12.2). These examples show that the collection of cylinder sets in R k for fixed k is in R3 is a not a field,' the intersection of three mutually orthogonal bounded not a cylinder set. The set of tinite-dimensionalcylinders is nbt closed under the operations of union and complementation (andhence intersection) except in an infinite-dimensional space. This fact is critical in considering c(C), the class obtained by adding the countable unions to C. By the lastmentioned property of unions, c(C) includes sets of the form (12.4)with k tending to infinity. Thus, we have the following theorem. .
.x
<walls'
.
twalls'
Kcube',
Stochastic Processes
12.3 Theorem c(r) ogy. I:l
=
183
f=, the Borel field of sets in R= with the Tychonoff topol-
Fig. 12.2 The condition of this result is something we can take for granted in the usual applications. Recalling that the Borel field of a space is the jmallest c-field containing the open sets, 12.3 is true by definition, since C is a sub-base for the product topology (see j6.5) and a11the open sets of R= are generated by unions and finite intersections of C-sets. To avoid explicit topological considerations, the reader may like to think of 12.3 as providing the definition of f=. One straightforward implication, since the coordinate projections are continuous mappings and hence measurable, is that, given a distribution on @=,=), finite collections of sequence coordinates can always be treated as random vectors. But, while this is obviously a condition that will need to be satistied, the real mtypractical method we have of defining distribproblem runs the other way. utions for infinite sequences is to assign probabilities to finite collections of coordinates, after the manner of j8.4. The serious question is whether this can be done in a consistent manner, so that in particular, there is one and only one p.m. on (R7S=) that corresponds to a given set of the finite-dimensional distributions. The affirmative answer to this question is the famous Kolmogorov consis'l'he
tency theorem.
12.4 The Consistency Theorem The goal is to construct a p.m. on @*,f*),and, following the approach of j3.2, the plausible first step in this direction is to assign probabilities to elements Of C Let p denote a p.m. on th space (R,f$, for k 1,2,3,.... We will say that this family of measures satisfies the consistency property if =
.
gt' )
=
g'-t'
X
R
m-k
)
(12.7)
Theory of Stochastic Processes
184
for E G Bk and al1 m > k > 0. ln other words, any k-dimensional distribution can be obtained from an v-dimensional distribution with m > k, by the usual operation of marginalization.
The consistency theorem actually generalizes to stochastic processes with uncountable index sets 1- (see 27.1) but it is sufficient for present purposes to consider the countable case. consistency theorem Suppose there exists a family of p.m.s (gkJwlich satisfy consistency condition (12.7).Then there exists a stochastic sequence (m,t e N J on a probability space @=,f=,g,)such that p is the p.m. the of coordinate functions (.L,...,Xk)'. u vector of tinite
12.4 Kolmogorov's -'r
=
The candidate
measure for
is defined for sets in C by
.x
FIC)
=
(12.8)
11,1(6-),
where C and E are related by (12.4).The problem is to show that g, is a p.m. on C. If this is the case, then, since C is a field and f= = c(C), we may appeal to the extension theorem (3.8+3.13)to establish the existence of a unique measure on (R*,%=) which agrees with g for all C e C. The theorem has a simple but important corollary. C is a determining class for (R7f=).
12.5 Corollary
E1
ln other words, if g and v are two measures on (R=,f=) and jtk finite k, then jt v.
vk for every
=
=
To prove the consistency theorem we require a technical lemma whose proof is beyond us at this stage. It is quite intuitive, however, and will be proved in a more general context as 26.23.
12.6 Lemma For every E G T an d such that pkE A') < n
0 there exists K, a compact subset of E,
>
-
.
In other words, a p.m. on the space (R Bk) has nearly all of its mass confined to a compact set; this implies in particular the proposition asserted in j8.1, that random variables are finite almost surely. ,
Proof of 12.4 We will verify that jt of (12.8)satisfies the probability axioms with respect to elements of C. When E Rk C R= so that the first two probability axioms, 7.1(a) and (b), are certainly satistied. To establish finite additivfor E e Tk E' e Tm and ity, suppose we have C-sets C gk-1(AD, an d C' zr-1(F') m m k k. If C and C' are disjoint, =
,
=
=
p,(O+ g,tC ') g( =
F) + jt m (F')
= gmE x
=
qvl-J7
=
tp E
,
pmEx R'&-$ + p,m(A'') ,
)
=
g,(c
kp
c')
,
(12.9)
where the second equality applies the consistency condition (12.7),and the third m-k and E z are disjoint if C and C z are. one uses the fact that E x R The remaining, relatively tricky, step is to extend tinite additivity to count-
Stochastic Processes
185
able additivity. This is done by proving continuity, which is an equivalent property according to 3.5. lf and only if the measure is continuous, a monotone 1 C or G' C has the property, g,(G) p,IC). sequence tCj e C ) such that CJ Since Cj Cc where g(C9 = 1 jttt7), it is sufficient to C implies consider the decreasing case. And by considering the sequence Cj C there also is no loss of generality in setting C 0, so that continuity implies g,tGl 0. To prove continuity, it is sufticient to show that if g(() 2 6: for some E > 0, for every j, then C is nonempty. lf C e C for some j k 1, then p,(G)= pkqlEj) for some set Ej G Bkct where ? /cU) ls the dimension of the cylinder Cj. By consistency, pkqlEp '1-
.
--
.1e
'l'
-
-
=
--
=
pmEj X R '''-U)
so there is no loss of generality in assuming ) for any m > 17U), kq) < < k41) < k42) We may therefore define sets F1 e LB that lU) < 1,...,./, by setting E) A) and ...
....
=
,
=
1.
=
Ei xR ko-k
is a decreasing sincefc/),t,-1
i
,
1
=
j
,...,
-
1.
(12.j())
euhsets(A1).t,-1,
sequence, so is the sequence of
for each j k 1.
Consider any tixed j. There exists, by 12-6, a compact set Kj bkkqfEj Detine the sets K1
=
E/2 y+1
&')<
-
c
Ej such that
(!g
.
.
! !)
Kix R kil-ki) e lk) by analogy with the A''1,and so define Fj
=
Bkct
(12. 12)
s)s c
( 12. 13)
O#t; e
.
'=1
Fj i E), and hence Dj ? Cj where Dj
=
lk -1
./
.
Applying 1.1(iii) and then 1.1(i), observe that Ej Fj -
=
Ej ch
./
j
K'ic
=
-
i=1
f=1
Ej K*i)
j
c
E*i -
f=1
r*f)
(12.14)
,
is decreasing. Hence where the inclusion is because the sequence tA'tl)'J.1 bkkitEj F./) f -
-
f=1
=
bklilE t, A-t,)
bkEi
-
Kit
< E/2.
(12.15)
f=1
The first inequality here is from (12.14) by finite subadditivity, which follows from finite additivity as a case of 3.3(iii). The equality applies consistency, and the secon (i inequa lity applies the summation of 2-f-1 Since Ej Fj and Fj are = disjoint and got.l g,(G)> : by assumption, it follows from (12.15) that bklh) pkqlFj) > e/2, and accordingly that Dj is nonempty. Now. we construct a Doint of C. Let Ixlij. i e (N1 denote a seauenc of ooints -
.
=
F/let??'yof Stochastic Processes
186
of R=
with
x(J) e
Dj for each j, so that
(X1U),...,Xw)U))lk(j)@U)) =
Note that for m
=
(12. 16)
Fj.
1,...,./,
1 Km, (XlU),...,Akm)U))'&(,?,)(xU))
(12.17)
=
by (12.12), where Kmis compact. Now 1et m be tixed. Our reasoning ensures that (12. 17) holds for each.j k m. A set in Rkm' is compact if and only if each of the coordinate sets in R is compact, so consider the bounded scalar scquences, fxibij, j r?z) for f 1,...,k4-). Each of these has a cluster point A'1, and we can use the diagonal method (2.36)to construct a single subsequence (Al with the = A': for each 1,...,k(-). By the compactness, property that Xijn) e Km i Em. This is true for every m e IN (X1,...,X(m)) Consider the point a7*e R= defined by 7v(,,,)(x*) for each m e N. (X1,...,X(m)) = Since x* e Cm for each m, we have x* G O'>;=ICk C, as required. . =
.-+
.
=
This theorem shows that, if a p.m. satisfying (12.7)can be assigned to the finite-dimensional distributions of a sequence m x is a random element of a We shall often wish to think of @=,f7jt) as derived probability space from an abstract probability space (f,T,#), and then we shall say that x is ble if F1(A>)e 5 for each event E e S*. This statement implies the 5lB coordinates Xt are F/f-measurable r.v.s for each /, but it also implies a great deal more than this, since it is possible to assign measures to events involving countably many sequence coordinates. ('R*,fO,g,).
''O
-measura
12.5 Uniform and Limiting Properties Much the largest part of stochastic process theory has to do with the joint distributions of sets of coordinates, under the general heading of dependence. Before getting into these topics, we shall deal in the rest of this chapter with the various issues relating exclusively to the marginal distributions of the coordinates. Of special interest are conditions that limit the random behaviour of a sequence as the index tends to infinity. The concept of a unform condition on the marginal distributions often plays a key role. Thus, a collection of rov.s tX,r, T e is said to be uniformly bounded in probability if, for any 8 > 0, there exists Bg < x such that -1)
#( IXzI sup ' It is also said to be unjrmly
)<
Lp-bounded for p
supI1ALll?,B <
<
=.
(12. 18)
:. >
0, if
(12.19)
'
For the case p = x, (12.19)reduces to the condition, supmlXzI < x a.s. ln this case we just say that the sequence is uniformly bounded a.s. For the case p = 1, we have sups f'IXz I < x and one might think it conrct to refer to this property as integrability' Unfortunately. this term is already in use for a ,
Suniform
,
Stochastic Processes
different concept (see the next section) and so must be avoided here. Speak of tuniform fal-boundedness' in this context. To interpret these conditions, recall that in mathematics a property is said to hold unifonnly if holds for all members of a class of objects, including the limits of any convergent sequences in the class. Consider the case where the /. Random collection in question is itself a sequence, with 'T = E$ and with probability 1, and for each finite t e IN,#( 1Al Be variables are finite x, > holds for for The < always point of unifonn Bet 0. : bound is some any a <s to ensure that the constants Bet are not having to get larger as t increases. p > 0, by Liapunov's inequality. Also, uniform fv-boundedness for anyp > 0 implies uniform boundedness in probability; the Markov inequality gives 'r
=
tbounded
,
-->
-x).
-->
,
#(IAI
B4
(12.20)
terminology we sometimes speak of fvboundedness in the case of (12.18). A standard shorthand (dueto Mann and Wald 1943b) for the maximum rate of (positive or negative) increase of a stochastic sequence uses the notion ot Oh' notation unifonu boundedness in probability to extend the tBig Oh' and real E lf, for exists ordinary 0, > Be < x such for there sequences (see j2.6). that the stochastic sequence fXn )T satisfies supa't 1XnI > B < E, we write Xn 0p(1). lf (L)T is another sequence, either stochastic or nonstochastic, and XnlYn = 0p(1) we say that Xn Op(L), or in words, &Xnis at most of order Fn in x, 0 as n tN(1); more we say that Xn Probability' If #( IXnI > :) t7p(1), when words, in GXn is of order lss than Xnlk'n generally Xn o (L') or r l'n in probability The main use of these notations is in manipulating small-order terms in an expression, without specifying them explicitly. Usually, Fn is a positive or negative power of n. To say that Xn tV1)is equivalent to saying that Xn converges in probability to zero, following the terminology of j12.2. Sometimes Xn Op(1) is defined by the condition that for each s > 0 there exists Be < x and an integer Ng > 1 such that #( IXnl > B < s for a1l n Nz. But Xn is finite almost surely, and there necessarily exists (this is by 12.6) a constant BL< #E') < E for 1 f n < Ne. For al1 possib1y 1arg er than Be, such that #( IXn l > practical purposes, the formulations are equivalent. Ktaittle
=
=
--)
-->
=
.
=
=
.
=
=
=,
Iheor.y of Stochastic Processes
188
12.6 Uniform lntegrability lf a r.v. X is integrable, the contributions to the integral of extreme X values must be negligible. ln other words, if FIXI f(IX1 1(IxlzM))
-->
0 as M
(12.21)
-->
x.
However, it is possible to construct unifonnly Al-bounded sequences l#nl which fail to satisfy (12.21)in the limit.
12.7 Example Detine a stochastic sequence as follows: for n 1,2,3,... let Xn = 0 with probability 1 1/rI, and Xn = n with probability 1/a. Note that E( lXn l) = nln = 1 for every n, and hence the sequence is uniformly Al-bounded. But to have =
-
limE IXnIlt lxalA/l)
=
0
(12.22)
MM=
uniformly in n requires that for each e > 0 there exists Ms such that E lXnl 1( xajkvl) < e for al1 M > Mo uniformly in a. Clearly, this condition fails, for e: < 1, in view of the cases n > Me. n Something very strange is going on in this example. Although E(Xn4= 1 for any n, x. To be precise, we may show that Xn = 0 with probability approaching 1 as n 0 (see 18.15). The intuitive concept of expectation appears to fail when Xn faced with nv.s taking values approaching intinity with probabilities approaching --+
-613...:
Zero.
The unform integrability condition rules out this type of perverse behaviour in a sequence. The collection (Xz, $ e T'J is said to be uniformly integrable if
sup 1im F( lXx11( xvl
Te
7
M)
M-yx
)
=
0.
(12.23)
ln our applications the collection in question is usually either a sequence or an array. In the latter case, uniform integrability of (Xkf) (say) is defined by taking the supremum with respect to both t and n. The following is a collection of theorems on unifonn integrability which will tind frequent application later on; 12.8 in particular provides insight into why this concept is so important, since the last example shows that the conclusion does not generally hold without uniform integrability. 12.8 Theorem Let E(Xn)
--$
E (A').
be a unifonnly l.YnlT
integrable sequence. If Xn -6J- X, then
Proof Note that E iXn I) = E IXnl 1( !xnI
s A.T+
Ftlxa
+
E lXnl 1( IxaIkM))
l1( jx,,1kM)).
(12.24)
By choosing M large enough the second tenn on the right can be made uniformly small by assgmption, and it follows that FIXnj is uniformly bounded. Fatou's
stochasticProcesses
189
lemmaimplies that FIXl < x, and F(m exists. Define l',, I.L --YI so that i',z 0 a.s. Since Fn < IaLl+ l-Yl by the triangle inequality, 9.29 gives =
f 2F( IXnl 1( lxnlxv/all
F(i'n1(yn>M))
+
2F( lXl 1( IxI>v/2)).
The second right-hand-side term goes to zero as M integrable if (A%) is. We may write f(Fn)
'(l%1(yn<M))
=
+
--
,
.--.)
so
x,
(12.25)
f Fn) is
A'(L1(yn>v)),
uniformly
(12.26)
and by the bounded convergence theorem there exists, for any E > 0, Ne such that F(l%1(ya<>)) < E/2 for n > #:, for M < oo. M can be chosen large enough that F(L1( ys>>)) < :/2 uniformly in n, so that EYn) < E for n > Nv., or, since 6: is 0. But arbitrary, EYn) --.-)
EYnq
by the
lnodulus
F(I-L--YI) k
=
IF(Xn) -F(ml
(12.27)
inequality, and the theorem follows. w
The next theorem gives an alternative form for the condition in often more convenient for establishing uniform integrability.
(12.23)which
is
12.9 Theorem A collection fXs, e 1-) of nv.s on a probability space (f1,1,#)is uniformly integrable iff it is uniformly Al-bounded and satisties the following condition: V E, > 0, 3 8 > 0 such that, for E E 5, ':
sup (F41 Xzl 1s) l < z.
P(E4 S
(12.28)
': e 'T
Proof To show sufficiency, fix : > 0 and 5 1, Markov inequality for p
E
7. By Al-boundedness and the
=
E
#(1-V1k Af) f and for M large enough, (12.28), it follows since
lxzI < M
(12.29)
x,
#(l-YzI Af) S ,1
6, for any is arbitrary that
>
0. Choosing
lA'(lX,rl 1f Ixzlwzlll < E,
Sup
to satisfy
(12.30)
TeT
and (12.23)follows since
E is arbitrary. note that, for any E e 5 and
To show necessity, F(1 Xvl 1s)
=
'r
E
1-,
1s1( Ixz1<MJ) + .E'(I.YZI 1s1( lxvI&z)) l-Y'rI
E
< MPCEI+
F(1 -Ysl1( Ixzlkvl).
(12.31)
Consider the suprema with respect to of each side of this inequality. For e > 0, (12.23) impliej there exists M < t:'o such that 'r
1F( 1AL1 1s)l
sup T
Uniform Al-oundedness
s MPCE) +
r1:.
now follows on setting F
=
f, and
(12.32) (12.28)also follows
Theor. of Stochastic Processes
190 with 8
<
zllM.
*
Another way to express condition (12.28)is to say that the measures vztp = ## must be absolutely continuous with respect to #, uniformly in :. Finally, weprove aresultwhich showswhy theuniformboundednessof moments of a given order may be important.
Jslxzl
12.10 Theorem lf 1+8
FIXzI for 0
>
0, then limv-pxlx'rl
Proof Note that
1( IxzIuv))
E IX,t11+e
(12.33)
< cxn =
0.
1+91
s (jx z j t l-vzlkxf)) S?E (IX1I1( jxvlzwzl) 2
for any 0 > 0. ne result follows on letting M (12.34) is finite by (12.33)..
..-+
x,
(12.34) since the majorant side of
Example 12.7 illustrated the fact that uniform fal-boundedness is not sufficient for unifonn integrability, but 12.10 shows that unifol'm falma-boundedness is sufficient, for any 0 > 0. Adding this result to those of j12.5, we have established the Merarchy of uniform conditions summarized in the following
theorem. 12.11 Theorem Uniform boundedness a.s.
uniform uniform = = uniform = uniform zz:
None of the reverse implications hold.
fw-boundedness, p > 1 integrability fw-boundedness, 0 < p S 1 boundedness in probability. n
13 Dependence
13.1 Shift Transformations We now consider relationships between the different members of a sequence. ln what ways, for example, might the joint distdbution of Xt and Xt-k depend upon k, and upon /? To answer questions such as these it is sometimes helpful to think aboutthe sequencein anew way. Havingintroducedthenotionof arandom sequence as a point in a probability space, there is a useful analogy between studying the relationships within a sequence and comparing different sequences, that is, different sample outcomes (l) e f. F-> copsider a 1-1 measurable mapping, F: ln a probability space (f1,@,#), (onto). This is a rule for pairing each outcome with another outcome of the space, but if each ) e f maps into an infinite sequence, Finduces a mapping from one sequence to another. F is called measure-preserving if PCTEI PE) for a11E e F. The shift transformation for a seqence (X/)JT is defined bylo =
X(F)
=
X,+1().
(13.1)
F takes each outcome ) into the outcome under which the realized value of X occurring in period t now occurs in period t- 1, for every t. In effect, each coordinate of the sequence from t 2 onwards is relabelled with the previous Xt-vk), the relationship period ' s index. More generally we can write XtlTkl between points in the sequence k periods apart becoming a characteristic of the transformation F k Since Xt is a nv. for a1l f, both the shift transformation and its inverse F -1 th e backsh;ft transformation, must be measurable. Taken together, the single r.v. X1(): f r- R and the shift transformation F, can be thoght of as generating a complete description of the sequence (X/))JT. This can be seen as follows. Given #14), apply the transformation F to ), and each ( e ( defines the mapping A%((0):f F-> R, obtain Xz = X1(F(0). Doing this and we are ready to get Xg X2(F(0). ltrating the procedure generates as many points in the sequence as we require. =
=
.
,
tor
=
be a sequence of coin tosses (with1 13.1 Example Consider 12.1, Let (Xf())JT for heads, 0 for tails) beginning 11010010001 1... (say).Somewhere on the intelwal g0,1) of real numbers (in binary representation), there is also a sequence (A()') J7 beginning 10100100011..., identical to the sequence indexed by (l) apart om the dropping of the initial digit and the backshift of the remaindr by one position. Likewise there is another sequence fXt(''4 1T,a backshifted version of ' ) 17b eg inning 0100101 1...; and so foih. If we define the transformation taVt(0
Theory of Stochastic Processes
192
)j'' F by F -1 ) = (l) r-1)' etc. the sequence (Xl(() )T can be constructed as 1-J ) )T,t h at is the sequence of first members of the sequences thesequence lm(F foundby iterating the transformation, in this case beginning 1,1,0,... I:l '
,
=
,
,
'
,
This device reveals, among other things, the complex stnlcture of the probability space we are postulating. To each point (.t) e fl there must correspond a countably infinite set of points F ) e f w hich reproduce the same sequence apart from the The intertemporal properties of a sequence can absolute date associated with then be treated as a comparison of two sequences, the original and the sequence lagged k periods. Econometricians attempt to make inferences about economic behaviour from recorded economic data. In time-series analysis, the sample available is usually a single realization of a random sequence, economic history as it actually occurred. Because we observe only one world, it is easy to make the mistake of looking on it as the whole sample space, whereas it is really only one of the many possible realizations the workings of historical chance might have generated. Indeed, in our probability model the whole economic universe is the counterpm of a single random outcome ); there is an important sense in which the time series analyst is a statistician who draws inferences from single data points ! But although a single it is realization of the sequenc can be treated as a mapping from a single linked to a countably infinite set of s corresponding to the leads and lags of the seqpence. A large part of our subsequent enquiry can be summarized as posing the question: is this set really rich enough to allow us to make inferences about # from a single realization? ,
m.
,
13.2 Independence
nnd
Stationarity
Independence and statioharity are the best-known restrictions on the behaviour of a sequence, but also the most stringent, from the point of view of describing economic time series. But while the emphasis in this book will mainly be on finding ways to relax these conditions, they remain important because of the many classic theorems in probability and limit theory which are founded on them. The degree to wlch random variations of sequnce coordinates are related to those of their neighbours in the time ordering is sometimes called the memory of a sequence; in the context of time-ordered observations, one may think in tenns of the amount of information contained in the cunrnt state of the sequence about its previous states. A sequence with no memory is a rather special kind of object, because the ordering ceases to have signiticance. lt is like the outcome of a collection of independent random experiments conducted in parallel, and indexed
arbitrarily. When a time ordering does nominally exist, we call such a sequence serially independent. Generalizing the theory of j8.6, a pair of sequences ((X,())T, l Fr() JT1e R= xR*, is independent if, for a11F1, E1 e 5*, /:()1.Y,17 e F1, (F,1: e E,4 Accordingly, a sequence
(A())T
=
#(tx,)7
e Ej4pf
r,)7 e
F2).
(13.2)
is serially independent if it is independent of
Dependence
193
fXt(T k(0) IT fOr a 11k ls 0 This is equivalent to saying that every tinite collection of sequence coordinates is totally independent. Serial independence is the simplest possible assumption about memory. Similarly, looking at the distribution of the sequence as a whole, the simplest treatment is to assume that the joint distribution of the coordinates is invariant with respect to the time index. A random sequence is called strictly stationary if Tllis implies that the sequences the shift transformation is measure-preserving. fXf 17 and fAkJ7=1have the same joint distribution, for every k > 0. Subject to the existence of particular moments, less restrictive versions of the condition are also commonly employed. Letting w E(Xt), and Xkt Covxt,xtn), consider those cases in which the sequence (g'rl7 and also the array ( fXmt are well defined. If w g,, al1 /, we say the sequence is mean J';=()l7wl, where (y,,l stationaty. If a mean stationary sequence has 17'is a sequence wt w of constants, it is called covariance stationaty, or wide sense stationary. If the marginal distriution of Xt is the same for any t, the sequence IXCJis said to be identically distributed. This concept is different from stationarity, which also restricts the joint distribution of neighbours in the sequence. However, Fhen a stochastic sequence is both serially independent and identically distributed (or i.i.d.4, this suftices for stationarity. An i.i.d. sequence is likeanarbitrarily indexedrandom sampledrawngomsomeaunderlyingpopulation. The following clutch of examples include both stationary and nonstationary .
.1
=
=
=1,
=
=
CaKS. (52 < 13.2 Example Let the sequence (6>J b e i i d with mean 0 and variance where fA)7, and let (0j17be a square-summable sequence of constants. Then ''O -x
.
.
(x,
.
*
A
=
77bet-j,
(13.3)
jcst
is a covariance stationary sequence, with Ext) t. This is the inhnite-order moving average additional details. n 13.3 Example lf
kt
=
0 and Ext)2 = c J e? ) sr every (MA(=)) process. See j14.3 for 2:.,.0
,0
is i.i.d. with mean 0, and Xt
for a constant a, then Ex
=
=
cos at
+
(13.4)
st
cos at, depending systematically on /.
I:a
13.4 Example Let (xV)be any stationary sequence with autocovariance sequence (ym, m 0J The sequence (.X)+ A%) has autocovariances given by the array .
l
'o
+
and hence it is nonstationary.
'sm
+
Yt+
'tt,vm,
m 2 0, t k 1 l
,
u
13.5 Example Let X be a r.v. which is symmetrically distributed about 0, with variance c 2 If Xt (- 1)W then (X,)Tis a stationary sequence. In particular, .
Ext)
=
=
,
0, and Cov(Xt,Xt+k4 5.2 when k is even and =
-c2
when k is odd, independent
Theory of Stochastic Processes
194 of t.
(:1
Theseexamples show that, contraryto acommon misconception, stationarity does not imply homogeneity in the appearance of a sequence, or the absence of periodic patterns. The essential feature is that any patterns in the sequence do not depend systematically on the time index. It is also important to distnguish between stationarity and limited dependence, although these notions are often closely linked. Example 13.4 is nonstationaty in view of dependence on initial conditions. The square-summability condition in 13.2 allows us to show covariance stationarity, but is actually a limitation on the long-range dependence of the process. Treatments of time series modelling which focus exclusively on models in the linear MA class often fail to distinguish between these propeties, but examples 13.3 and 13.5 demonstrate that there is no necessary connection betwen
them.
'
Stationarity is a strong assumption, particularly for the description of empirical time series, where features like seasonal patterns are commonly found. It is nonstationarity, sometling we might think of useful to distinguish between capable of elimination by local averaging of the coordinates, and global' as nonstationarity, involving features such as persistent trends in the moments. If sequences (Xr)7=1 and (a'+k)7=1have the same distribution for some (not necessarily every) k > 0, it follows that (.+x)7.1 has the same distribution as (X)+l7=1, and the same property extends to every integer multiple of k. Such a sequence accordingly has certain stationary characteristics, if we think in tenns of the distributions of successive blocks of k coordinates. This idea retains x. Consider the limit of a tinite sequence of length n, divided force even as k into (rl j blocks of length gn 1,plus any remainder, for some a between 0 and 1. here denotes the largest integer below x.) The number of blocks as well (Note, E.;r) as their extent is going to infinity, and the stationarity (or otherwise) of the sequence of blocks in the limit is clearly an issue. Important applications of these ideas arise in Parts V and VI below. It is convenient to formulate a definition embodying this concept in terms of moments. Thus, a zero-mean sequence will be said to be globally covariance stationary if the autocovariance sequences (y/uf are Cesro-summable for each )*r.1 0, where is strictly positive the k in the Cesro sum case of the variances m = m 0). The fllowing are a pair of contrasting counter-examples. ilocal'
-->
13 6 Example A sequence with
'p,
/l5
-
@
is globally nonstationary for any
j
#
0. n
13.7 Example Consider the integer sequence beginning 1,2,1,1,2,2,2,2,1,1,1,1,1,1,1,1,...*, i.e. the value changes at points t 2 k k = 1 2 3 The cesro sum of this x. It fluctuates eventually between the points sequence fails to converge as n lk k = k even. A stochastic sequence having a 5/3 at n 2 k o dd and 4/3 at n of form globally nonstationary. (:1 is variance sequence this =
,
-->
,
,
=
,
,
,
,.,.
Dependence
195
13.3 lnvariant Events The amount of dependence in a sequence is the chief factordetermining how informative a realization of given length can be about the distribution that generated it. At one extreme, the i.i.d. sequence is equivalent to a true random sample. The classical theorems of statistics can be applied to this type of distribution. At the other extreme, it ij easy to specify sequences for which a single realization can never reveal the parameters of the distribution to us, even in the limit as its length tends to intinity. This last possibility is what concerns us most, since we want to know whether averaging operations applied to sequences have useful limiting properties', whether, for example, parameters of the generation process can be consistently estimated in this way. Toclarify theseissues, imaginerepeatedsamplingof random sequences (A((t)) JT', in other words, imagine being given a function X1(.) and transfonnation F, making repeated random drawings of (.t) from fl and constructing the corresponding random sequences', 13.1 illustrates the procedure. Let the sample drawings be denoted j, 1,...,N, and imagine constructing the average of the realizations at some i ,=1 X is called an ensemble average, fixed time / e. The average kxto N-1X% o((tk) which may be contrasted with the time average of a realization of length n for n-11J=1Xj()j). Fig. 13.1 illustrates this procedure, some g iven )j e D, Xntj) showing a sample of three realizations of the sequence. The ensemble average is the average of the points falling on the vertical line labelled /c. lt is clear and the ensemble average as n and N that the limits of the time averg respectively go to infinity are not in general the same; we might expect the ensemble average to tend to the marginal expectation Ext ), but the time average 8 will not do so except in special cases. If the sequence ls nonstationary Ext depends upon /o, but even assuming stationarity, it is still possible that different realizations of the sequence depend upon random effects which are common to a1l t. =
=
,
=
O O
O
O O O
O O
O
O
Fig. 13. 1
Theory of Stochastic Processes
196
In a probability space (f1,F,#) the event E e 5 is said to be invariant under a transfonnation Tif PCTEh F) 0. The criterion for invariance is sometimes given as TE F, but allowing the two events to differ by a set of measure zero does not change anything important in the theory. The set of events in 5 that are invariant under the shift transformation is denoted #'. =
=
13.8 Theorem
,@
is a c-tield.
Proof Since F is onto,
is clearly invariant. Since F is also 1-1,
TEc A E C
CTE4cA Ec
=
(13.5)
TE A E
=
by definition. And, given 3En e #, n e IN), PCTE,tA En)
PITEn
=
&) + P(En
-
-
TEn4 = O
(13.6)
for each n, and also F
UEn
A
n
UEn UTEn A UEn,
(13.7)
=
n
l
n
using 1.2(i). By 1.1(i) and then 1.1(iii),
UTEn UEn U TEn OEL ? U(Ff n
and similarly,
f'n
=
-
n
n
-
n
&),
(13.8)
?1
n
UEn UTEn U(En n
-
n
TEn).
-
(13.9)
n
= 0 The conclusion #gT(Un&) A (Un&)1 now follows by . proof. pletingthe
(13.6)and
3.6(ii), com-
An invariant random variable is one that is J/f-measurable. An invariant nv. Z() has the property that Z(F) =Z(), and an J/f-measurable sequence (Z,()) ):' Z1()) a.s. for every t. The invariant events is trivial in the sense that Zt and associated r.v.s constitute those aspects of the probability model that do not alter with the passage of time. =
= Ff((z)) + Z(), where Xt 13.9 Example Consider the sequence (AX()))) ( F,4(0)) random and example of invariant Z((0) An being a event is E an sequence a r.v. f (.0: Z() K z, Ff() e R ). Clearly E and FF are the same event, since Z is the only thing subject to a condition. Fig. 13. 1 illustrates this case'. lf ( Ff()) ) is a zero-mean stationary sequence, the figure illustrates the cases Z((l)1) = Z((.0:) 0, and Z((l)3) > 0. Even if EZ4 out' in the limit, = 0, the influence of Z() in the time average is not as it will be from the ensemble average. I:a =
=
The behaviour of the time average as n fundamental result.
--
cxn
is summarized by the following
be 13.10 Theorem (Doob 1953: th. X.2. 1) Let a stationary sequence (.&((.t)))T defined by a measurable mapping X1()) and measure-preserving shift transform-
Dependence
ation T, such that
m()
J-1
.Y1(F
=
1imSnbln
), an d
I
Z)=1m()).If
=
F( X1j
,7)()),
ECXL
=
1et Sn ()
x,
(13.10)
(:I
a.s.
<
n-+oo
In words, the limiting case of the time average can be identitied with the mean of the distribution conditional on the c-tield of invariant events. The proof of 13.10 requires a technical lemma.
A(I3) ((t):supatkst))
13.11 Lemma Let
=
Jvca(oXl())J#()
-
nI))>
0) Then for any set M e #', .
2 F#(Mf-hA(p)).
(13.11)
Proof We establish this for the case fs 0. To generalize to any real p, consider the sequence fA p) which is stationary if (XCIis stationary. > 0), the set of outcomes Write A for A(0), and let hj f ): maxlskust) for which the partial sum is positive at least once by time j. Note, the sequence x. Also 1et A as j f A.j) is monotone and hj =
-
,
=
'1h
--
N nj
T-ih
=
-j
n
:
=
max Sklf l Kkf n
>
0
(13.12)
.
'-j
Since k
j+k
&(r?(o) XxtT) =
!F71rf(e)) sjmke)SjTb, z-
=
>1
by defining & Nnj
=
(13.13)
-
J=./m 1
0 we may also write ):
=
max j- 1f k f n
sk
-
Sjbh) > 0
,
0 f j Kn
-
1
.
(13.14)
This is the set of outcomes for which the partial sums of the coordinates f'rom j + 1 to n are positive at least once, and we have the inequality (explained below) n-1
XA+1()
lxnytl
2 0, all
)
e D.
(13.15)
Integrating this sum over the invariant set M gives n-1
n-1
r
0 X j=0 J <
u-suxjwtodrb
v
n-1
m
'0
Jn
v
R'
''nn
X y=o
j,.x.y&+1(rVt,))J#()
n
g
=X j u-g jcz0
=
.X1(9d#((9) J-
=
Jv.ayXl(t9V#(), X jc
(13.16)
1
where the tirst equa lity uses the fact that hn-j = (: T-l(kbe Nnjj and the measure-preserving property of F, and the third is by reversing the order of summation. The dominated convergence teorem applied to fm lMraayl, with I I as the
m
Theory of Stochatic
198
Processes
dominating function, yields
(13.17) Jxawxltltf#tt9l. Jsyraayxltllptl -->
limit by 2.26, so that, as required,
This limit is equal to the Cesro
-1
j'ucvs m()##()
=
1im a-,-
The inequality in
n
Xnj
m()##4)) 0. .
(13.18)
ucxxj
s,
not self-evident, but is justified as follows. The having the property expression on the left is the sum containing only those Xt realization that in the partial sums from the point t onwards are positive at least once, otherwise the tth contribution to the sum is 0. The sum includes only Xt lying in segments of the sequence over which Sk increases, so that their net contribution must be positive. It would be zero only in the case Xt S 0 for 1 f t S n. Fig. 13.2 depicts a realization. for k shows values of Sk 1,...,a, vertical shows the separations between successive so the .Yf((0)are the running sum of the terms of (13.15).The coordinates where the Xt are to be omitted from (13.15) are arrowed, the criterion being that there is no to the which exceeds the right current
(13.15)is ,
'o'
=
&o'
<+'
.
to'
'o'.
+
+
+
* *
*
0
O
*
*
Fig. 13.2 Proof of 13.10 The first step is to show that the seqpence (Snlbln ITconverges almost surely to an invariant r.v. Sl). Consider 5'() = limsupa Snlln. RF)
=
lim sup vl--hx nkm
&+l() n+ 1
n+ 1 n
X1() =
-
n
)),
(13.19)
ltl = liminfn Snln. Hence, so that Rl is invariant, the same being true for sttt)) = M(a,p) invariant. Since i s supnsnln (x < < < the event j3 V((t)ll (: A(p) M(a,p) A(p) where defined in the statement of 13.11. Hence, is V1, M(a,j) in the lemma gives putting M =
lDependence
J>(oj)X1(t0V#())2 But now replace
(
):
supnt-xsktl/n)
199
p#(M(?'p)).
(13.20)
13.11 by -AX)), and observe that M(a,p) Hence we get
in AX(t)) -a)
>
c
A(-a)
=
.
Jsy(a,j)X1(t0V#(tt9 S tx#(M(?'p))'
(13.21)
Since the left-hand sides of (13.21)and (13.20)are equal and (x < p, it follows 5'() with probability 1. that #(M(a,p)) = 0; that is, R)) 8)) This completes the first stage of the proof. It is now required to show that S = Exj I#) a.s., that is, according to equation (10.18),that =
juxltdp Sinc
=
jsjdp,
=
each M e
(13.22)
M is invariant, ''t
j
juxLlkdp =
a-
n
.,yt
X >1
vr
j.
Jyrto, ju ,
X/t0)##((t)
=
-s
Snl
:#4tl))
(13.23)
and the issue hinges on the convergence of the right-hand member of (13.24)to
E(S3u4. Since the sequence (-Y/Jis stationary and integrable, it is also uniformly integrable, and the same is true of the sequence ( F,l, where Ff Xtsu and M e #. For : > 0, it is possible by 12.9 to choose an event E e 5 with #(& < 6, such that sup E 1FJ I1e,) < e. For the same E, the triangle inequality gives =
j
E
j) j s )x-(J yn,-, >1
#,)
Ii-,I E
dp
>1
< :
(13.24)
.
By the same argument, also using stationarity and integrability of Ff,
I j
J1 n-
n
y-, dp
-,
s
Hence by 12.9 the sequence n-v I>. If n-l n n n -11 F) 12.8, by =1
=
J
complete. w
n
,.j
jl
yl
),; jap
=
s j yj j
(13.25)
tn-1Z:=1Ff)
-->
i s a 1so uniformly S a.s., it is clear that n-S
-1
Mn
Since n is arbitrary,
n
-1
:#((t))
--y
( Sudp.
2M
n
integrable, where Sku a.s., so
1M
-->
(13.26)
(13.26)and (13.23)together give (13.22),and the proof
is
13.4 Ergodicity and Mixing The property of a stationary sequence which ensures that the time average and the ensemble average have the same limit is ergodicity, which is defined in terms of
I'heory of Stochastic Processes
200
the probability of invariant events. A measure-preserving transformation F is l or #() is the c-field of 0 for a11E e Ds where ergodic if either #(F) under F. A stationary sequence (Xf() )+-7is said to be ergodic if invariant events J-1 =
x/
=
Xl (F ) for every t where F is measure-preserving and ergodic. Some authors, such ag Doob, use the tenn metrically transitive for ergodic. Events that are invariant under ergodic transformations either occur almost surely, or do not occur almost surely. ln the case of 13.9, Z must be a constant almost surely. Intuitively, stationarity and ergodicity together are seen to be sufficient conditions for time averages and ensemble averages to converge to the same limit. is the mean not just of X1 Stationarity implies that, for example, g, f(m()) but of any member of the sequence. The existence of events that are invariant under the shift transformation means that there are regions of the sample space which a particular realization of the sequence will never visit. lf PIFFA E) = yhen the event Ec occurs with probability 0 in a realization where E occurs. However, if invariant events other than the trivial ones are ruled out, we ensure that a sequence will eventually visit a1l parts of the space, with probability 1. In this case time averaging and ensemble averaging are effectively equivalent
X()
=
=
t
(
operations.
)
The following corollary is the main reason for our interest in Theorem 13.10.
13.12 Ergodic theorem Let (.X)4(0)IT be a stationary, ergodic, integrable sequence. Then 1imSnelln
=
(13.27)
F(X1), a.s.
n-4-
Proof This is immediate from 13.10, since by ergodicity, EIXj I#)
F(X1) a.s. w
=
In an ergodic sequence, conditioning on events of probability zero or one is a is trivial operation almost surely, in that the information contained in trivial. The ergodic theorem is an example of a 1aw of large numbers, the first of several such theorems to be studied in later chapters. Unlike most of the subsequent examples this pne is for stationat'y sequences. Its practical applications in econometrics are limited by the fact that the stationarity assumption is often inappropriate, but it is of much theoretical interest, because ergodicity is a very mild constraint on the dependence, as we now show. Atransformation thatis measure-preserving eventually mixes up the outcomes in a non-invariant event A with those in Ac. The measure-preserving property rules out mapping sets into proper subsets of themselves, so we can be sure that FA jAc is nonempty. Repeated iterations of the transformation generate a sequence of sets l T V ) containing different mixtures of the elements of A and Ac. A positive AC; that is, if implies a negative dependence of # on dependence of # on #(A f-h B) > P(A4PB) then P(Ac /'A Bj PB4 #(A f-'h #) < #(#) #(A)#(#) PIAc)PCB). Intuition suggests that the average dependence of B on mixtures of A and Ac should tend to zero as the mixing-up proceeds. ln fact, ergodicity can be characterized in just this kind of way, as the following theorem shows. ,/
:4
=
-
-
=
llependence
201
shift transfonnation Fis ergodic if and only 13.13 Theorem A measure-preserving for pair of if, events A, B e @, any n
1
-xpvkx
lim
rasl
(13.28)
PA)PB).
=
k=1
n-yx
Proof To show ton 1y ito let A be an invariant event and B A. Then PCTPAf'n B) = #(A) for a11 k, and hence the left-hand side of (13.27)is equal to #(A) for all 1' #(A) 2 implying 84) 0 or 17.7 k. This gives #(A) apply the ergodic theorem to the indicator function of the sets To show and ergodic, to give F V w h re T is measure-preserving =
,
=
=
,
iif'
,
,
1
n
lztxt(0l lim X &1
=
-
(13.29)
#(4), a.s.
n-yx
But for any # e 5,
jj x-1,u((o) -)
-
,(.)
k=1
j
dpv
zj I jJst) x,.-tx-
1,u((o)
#IN
-
&1
lztxt(')l
,(x)
!
dp
I
-#(A))dz'((o)
1
n
=
1 -sX#(F
k
A rhB) -PA4PB4
.
(13.30)
k=1
The sequence whose absolute value is the integrand in the left-hand member of cxl by (13.30) converges almost surely to zero as n (13.29);it is bounded uniformly + is clearly integrable. Hence, uniformly in 1 #(A) absolutely by n, so the left-hand member of (13.30)converges to zero by 12.8, and the theorem -->
follows.
.
Following from this result, ergodicity of a stationary sequence is often associated with convergence to zero in Cesro-sum of the autocovariances, and indeed, in a Gaussian sequence the conditions are equivalent.
13.14 Corollary If (Xr4(4)))Tis a stationary, ergodic, square-integrable sequence, then n 1 XCov(XIm)
*n al
--
0 as n
(13.31)
-+
x.
Proof Setting B A and defining a real sequence by the indicators of F A, Xkk equivalent to (13.31).First extend this result to sequencs = 1wu((o), (13.28)is X,'%1Aj(), so that A%((t))= X1(T) Lwlr-ujt). of simple r.v.s. Let X1(() = The main point to be established is tha.tthe differencebetween X1 and a simple r.v. can be ignored in integiation. In other words, the sets F Aj must form a =
=
.j
Theot'y of Stochastic Processes
202
partition of f1, apart possibly from sets of measure 0. Since F is measureAf)) #(UjAj) 1, using 1.2(ii), and hence #(F-1(Uf preserving, #(UF-V 1, additivity of the #(f1 UjF -1Ai) = 0. And since L#(F-V i ) = Zf#(Af) z4jl -1 is a1so disjoint apart f'rom possible measure implies that the collection (F verifying required the property. sets of measure 0, This argument extends by induction to Xk for any k e N. Hence, =
=
=
=
-
7txlxzlE 777rcslxow-u/) j =
i
= 7) 77a,uyrtA
rn T-kAjl,
,w1z'',4
7/... $J
,.-u ' .7
j
-
t.
.
(13.32)
/ (the sum being absolutely convergent by assumption), and by 13.13, n 1 TExk)
lim n -
n--yx
=
X ywm
&1
i
lim
'
'-
j
n-yx
-
n 1 X#(Af
n
-z
rn T Aj)
&1
afW#(Xf)#(Ay) = Xi X j
&XI)2,
=
(13.33)
whereEXj4 2 EX 14EXk) for any k, by stationarity. The theorem extends to generalsequences by the usual application of 3.28 and the monotone convergence theorem.> =
This result might appear to mean that ergodicity implies some fonn of asymptotic independence of the sequence, since one condition under which (13.31)certainly x. But this is not so. The following holds is where Cov(X),XJ 0 as k example illustrates nicely what ergodicity implies and does not imply. -->
-->
13.15 Example Let the probability space (f,T,#) be defined by fl = f0, 1 J, so that 5 = (fO l 0),f 1 l (0,1)) and #() = 0,5 for (.0 0 and (l) = 1. Let F be the transfonnation that sets F0 = 1 and F1 = 0. ln this setup a random sequence and generating the sequence by f Xr(fJ))J may be defined by letting .Yl4(0)= iterating F. These sequences always consist of alternating Os and 1s, but the inial value is randomly chosen with equal probabilities. Now, T is measurepreserving', the invariant events are f and 0, both trivial, so the sequence is ergodic. And it is easily verified that lima-yxn -1Z2 PITkA fa B) P(A)PB) for 0.5 for k every pair A,B e T. For instance, 1et B = f 1 ) and then #(F V f7#) even and 0 for k odd, so that the limit is indeed 0.25 as required. You can verify, equivalently, that the ergodic theorem holds, since the time average of the sequence will always converge to 0.5, which is the same as the ensemble mean (a of XI()). ,f
=
,
,
=
=j
=
,
.
In this example, Xt is perfectly predictable once we know X; for any t. This shows that ergodicity does not imply independence between different parts of the sequence, even as the time separation increases. By contrast, a mixing sequence has this property. A measure-preserving, ergodic shift transformation Wis said to be mixing if, for each A, B e F, ,
l7ependence
lim#(F 4 cjB4
203
(13.34)
PA4PB).
=
k-x
X1(Ff-')) for each T he stationary sequence f.V)7is said to be mixing if A() t where F is a mixing transformation. Compare this condition with (13.28);Cesro convergence of the sequence t#(F V /'7 #) k e EN ) has been replaced by actual convergence. To obtain a sound intuition about mixing transformations, one cannot do better than reflect on the following oft-quoted example, originally due to Halmos (1956). =
,
13.16 Example Consider a dry martini initially poured as a layer of vennouth (10% of the volume) on top of the gin (90%).Let G denote the gin, and F an arbitrary small region of the fluid, so that Fcj G is the gin contained in F. If 0.9 and #(.) denotes the volume of a set as a proportion of the whole, #(G) =
#(Ff-7 G)/#(/), the proportion of gin in F, is initially eithr 0 or 1. Let F denote the operation of stining the martini with a swizzle stick, so that #(F Vcj G)/#4p is the proportion of gin in F after k stirs. Assuming the fluid is incompressible, stirring is a measure-preserving transfonnation in that #(F #) mixes the martini we would expect the proportion = #4#) for all k. If the stirring PITkFP k hich is gin in G)/#(&, to tend to #4G), so that each region F F F, w of of the martini eventually contains 90% gin. la transfonnaThis is precisely condition (13.34).Repeated applications of a lnixing tion to an event A should eventually mix outcomes in A and Ac so thoroughly that for large enough k the composition of F V gives no clues about the original A. Mixing in a real sequence implies that events such as A ( ): A((l)) S tz) and F 1.4 = f(t): Xt-vk) S JJ are becoming independent as k increases. It is immediate, or virtually so, that for stationary mixing sequences the result of 13.14 can be x. strengthened to Cov(X1,-L) 0 as n =
--y
--y
13.5 Subfields and Regularity We now introduce an alternative approach to studying dependence which considers the collection of c-subfields of events generated by a stochastic sequence. This theory is fundamental to nearly everything we do subsequently, particularly because, unlike the ergodic theory of the preceding sections, it generalizes
(stationaty)case. Consider a doubly infinite sequence (-V,t e r) (notnecessarily stationary) and define the family of subfields l X, s f /l, where TJ = nXs,...,Xt) is the smallest c-field on which the sequence coordinates from dates s to t are measurable. The sets of Sf can be visualized as the inverse images of (/ s4dimensional cylinder sets in :8*; compare the discussion of j 12.3, recalling that IN and Z are equipotent. We can let one or other of the bounds tend to intinity, and aparticularly important sub-family is the increasing sequence (T-'x, t e Z ), which information contained in the sequence up to can be thought of as, in effect, /' which The c-field on the sequence as a whole is measurable is the date limitingcase F+-7 Vts-t.. In cases where the underlying probability model beyond the measure-preserving
-
tthe
=
Theory of Stochastic Processes
204 concerns just the sequence
(XZJwe
shall identify 1+-Zwith 5.
OfT-rx. Another interesting object is the remote c-Jcl# (or tail c-field), F-x This c-tield contains events about which we can learn something by observing any coordinate of the sequence, and it might plausibly be supposed that these events the occurred at time past when the initial conditions for the sequence were set. However, note that the set may be generated in other ways, such =
-x,
RS
t Ot 1+=
,
or
Ot 5-tt
tremote'
.
One of the ways to characterize independence in a sequence is to say that any pair of non-overlapping subtields, of the fonn Tfzland 1143where /1 k h > t5 k /4, are independent (see j10.5). One of the most famous early results in the law' for independent theory of stochastic processes is Kolmogorov's usually given for the case of a sequence (Al7, and the sequences. This theorem is is defined in this c-tield remote case as
07,.18/D,
.
13.17 Zero-one Iaw lf the sequence (A-fITis independent, every remote event is trivial, having either probability 0 or probability 1. and let T be the colProof Let A be a remote event, so that A e i= 0*2.1@=f lection of events having the property #(A rn #) #(A)#(#) if B e V. By indepenand @1f dence, are independent subfields, so that @(c V, for every f. We may therefore say that U7=1T( c V. U'7=1T(,then B e T( for some t, and B' e T(' for some t', and if t' 1 lf B,B' and accordingly U7z=lFlis a tield. t (say)ihen B e F1f'so B fa #' e ;1f' c U7 Moreover, if fBj e Nl is a monotone sequence and Bj #, then A f-'h Bj is also monotone and converges toA rn SforanyA l 1'-', adl e V'by continuity of #, so V is a monotone class. By the argument of 1.24, V therefore contains the union of and so V u c(U7=1T'() V*t any countable collection of sets from U7 2 and T.llowevervd e 5, soA e N wemay setfl=A, giving#td) =#(A) s ence p(4) = 0 or 1. . =
,
=
T'K'
=1@1r,
-->
=l97lf
.191,
=
=
.
.
The zero-one 1aw shows us that for an independent sequence there are no events, other than trivial ones, that can be relevant to a11 sequence coordinates. But clearly, not only independent sequences have the zero-one property, and from our point of view the interesting problem is to identify the wider class of sequences that possess it. A sequence fX))+-7 is said to be regular or mixing if every remote event has probability 0 or 1. Regularity is the term adopted by Ibragimov and Linnik (1971), to whom the basics of this theory are due. ln a suitably unified framework,this is essentially equivalent to the mixing concept defined in j13.4. The following theorem says that in a regular sequence, remote events must be independent of a11 events in 5. Note that trivial events are independent of themselves, on the
definition.
13.18 Theorem (lbragimovand Linnik 1971: th. 17.1.1) tall+-Zis regular if and only if, for every B e 5,
lt ( .
Dependence
sup l#(A ra #)
-
I --A
PA)PB4
205 0 as t
(13.35)
.--.) -=.
A e F!-
Proof To prove suppose 3 E E F-x with 0 regular. Then for every t, E e @-'x and so dif'
<
,
S#P I#(A
f''h
F)
I2
#(A)#(A3
-
ziez.f-
#(A)
-
P(E) < 1, so that (X;)+-=xis not PE4
2>
(ja a6)
()
,
.
which contradicts (13.35).
if assume regularity and define random variables ( lx #(A) (F-lx/f-measurable) and q (Wf-measurable), such that ls #(#) F()n). Then, by the Cauchy-schwartz inequality, #(A fa #) P(A)PB) To prove
Eonly
=
,
=
-
-
=
-
lF((n) I
=
IlF(n I@-'-)112, IuE'tsE'tnI@!-)) I < lItII2
(13.37)
where the equality is by the Iaw of iterated expectations because A e F-lx. Note which will complete the < 1. We show lIF(n l@-,-)112 0 as t that 11(112 of T-fx. roof, arbitrary since element is X an P x)()) Consider the sequence (F(ls I?F-f lt2. For any (.t) e f1, -->
-=,
--
F(1sI8/'-x)())
F(1s l@-x)(t0) as t
--
(13.38)
--> -=,
where by equation (10.18) and the zero-one property,
Jw&1sI;-x)()J#()
f-7#)
PE
=
#(#), PE4 =
1
=
#(A-) = 0
0,
,
E e Fux.
(13.39)
#(#) a.s. agrees with the definition, so It is clear that setting F(ls I@-x)() 5-t.) F(lsl P@) a.s., or, equivalently, that we may say that (F(1s IT!x) /'(#)()2 0, a.s. (13.40) Since 1s()) #(#) < 1 for al1 (1) e f, (F(ls @-fx)() /:(#)()2 is similarly =
--
--)
-
1
-
I
I
-
bounded, uniformly in f. Unifonn integrability of the sequence can therefore be assumed, and it follows from 12.8 that IlF(n l9-/-)112=
as required.
IlF(1sI@-'-)(t,))
-/'(:)112
--
0 as t
(13.41)
--+ -,x,
.
In this theorem, it is less the existence of the limit than the passage to the limit, the fact that the supremum in (13.35)can be made small by choosing large, that gives the result practical significance. When the only remote events are trivial, the dependence of Xt-vkon events in @-x, for fixed k must eventually decline as f increases. The zero-one 1aw is an instant corollary of the necessity part of the theorem, since an independent sequence would certainly satisfy -t
,
(13.35).
Theory of Stochastic Processes
206
There is an obvious connection between the properties of invariance and remoteness. lf T is a measure-preserving shift transformation we have the following simple implication.
13.19 Theorem
If FA
=
A, then A e T-x. l
l
t t t Proof lf A e T-x, then Fz4 e @-x and F A e F-x :4 followsimmediately that e (*s=ts-sxj chtO,k +
-
-
1 .
lf TA
-x@!x)
=
-
=
l
A, F A
T-x.
A and it
=
.
t
The last result of this section establishes formally the relatlonship between regularity and ergodicity which has been implicit in the foregoing discussion.
13.20 Theorem (Ibragimov and Linnik 1971: cor. 17. 1.1) If a stationary sequence x((o) (AX(o))!Zis regular, it is also ergodic. =
Proof Every set A e F+-7is contained in a set At e T-ff, with the sequence (A,) non-increasing and Ar A. Thus, A, may be constnlcted as the inverse image under .x of the (2/+ ll-dimensional cylinder set whose base is the product of the coorof x(A) e The inclusion follows by dinate f-sets for coordinates P(At4 #(A). continuity of By P, that 1.2(iv). we can assume Let A be invariant. Using the measure-preserving property of F, w find .1,
=.
-4,...,/
-->
P(At raAl
=
PAt
Since k is arbitzary, regularity Letting f
--
x
yields #(A)
=
fa
F -kA)
=
PITkA t chz4l.
(13.42)
implies by (13.34)that PAt faAl = PAt)PA). P(A) 2 so that P(A4 0 or 1, as required. . =
,
13.6 Strong and Uniform Mixing The defect of mixing (regularity)as an operational concept is that remote events are of less interest than arbitrary events which happen to be widely separated in time. The extra ingredient we need for a workable theory is the concept of dependence between pairs of c-subtields of events. There are several ways to characterize such dependence, but the following are the concepts that have been most commonly exploited in limit theory. Let L1,5,P) be a probability space, and 1et V and R be c-subfields of @;then
(x(V,X) =
sup G c F Hq
.'(
IP(G
f'n
S)
-
#(G)#(&
I
(13.43)
l
(13.44)
,
is known as the strong mixing coeftkient, #(V,A)
=
sup Gq F,f'1eJf; P(G)>0
and
G4 1/,(/./'1
-#(#)
as the uniform mixing coefficient. These are alternative measures of the dependence between the subfields V and R. If the subfields V and R are independent, then (x(V,J) = 0 and #(V,Jf) 0, and the converse is also true in the case of uniform mixing, although not for strong mixing. At first sight there may appear not much to choose between the defini=
,
Dependence
207
tions, but since they are set up in terms of suprema of the dependence measures over the sets of events in question, it is the extreme (and possibly anomalous) cases which define the characteristics of the mixing coeixcients. The strong mixing concept is weaker than the uniform concept. Since
IPG
r-'hff
-
PCG4PC* I S IPCH, G)
I f 9(5',>t)
PH)
-
(13.45)
for all G e V and H e R, it is clear that a(N,J#) f ((T,Jf). However, the following example shows how the two concepts differ more crucially. 13.21 Example Suppose that, for a sequence of subfields (V-IT,and a subfield X, a%m,R4 0 as m x. This condition is compatible with the existence of sets Gm ?. forJ xp(s) PGm) llm, andlGm fo & e Mmandffe R with thejroperties .-+
--
=alm
=
.
But (/497,s,J#)2 IPCH6Gm) #(/f) I Ia PH4 I for every tilds Mmand R are not independent in the limit. n =
-
-
,?z
k 1, showing that sub-
Evidently, the strong mixing characterization of does not rule out ppssibility dependence negligible between the sets. qf () rnixing coefficients and the only not that are can be defined, although they tx have proved the most popular in applications. Others that have appeared in the literature include tindepndence'
13(V,J#) = sup E HR
j#(SIV)
-
#(S)
j
(13.46)
and
p(V,J#)
sup
=
If((n)I
(
2
n
(13.47)
,
2
where the latter supremum is taken with respect to a1l square integrable, zero mean, V-measurable r.v.s (, and S-measurable r.v.s q. To ompare these alter#(Sl T)() #(&, so that natives, first let (() =
f5(V,X)
-
=
sup SGS
JI(1
dP
I( I js sup
dp
GC%D.H6R
sup
G e F, H R
(.LdP
*
=
a(T,3f).
(13.48)
'''
#(G) and Moreover, since for any sets G e V and H e 3f, (((t)) 1c() n() which members the of is defined, p(V,>t) and F((n) #(& set 1zz((l)) are over while l( I < 1 and lqI S 1 for these cases, it is also clear #(G f''h z.fl PGlm notwithstanding (x its designation, is the weakest of mixing, Thus, that p a. o? variants, although it is these four course stronger than ordinary trivial remote events. We also have j S 9, by an regularity characterized by immediatecorollary of the following result. =
-
=
=
-
-
istrong'
Theory of Stochastic Processes
208
13.22Theorem I#(NIV)
If
#(&
-
9(V,>t) a.s., for a1l H e R. a
The main step in the proof of 13.22 is the following lemma. !' 13.23 Lemma Let X be an talmost surely boundedy) V-measurable nv. Then
1
sup GeF,#(G)>0
jaxdp
po).
ess sup
=
x.
(13.49)
:'
r
E
Proof #(G)-1 j < ess sup X, for any set G ils the designated class. For any E > 0, consider the sets .
fccxlrj
..E
.
.
G G-
f (:
=
=
:
X() -A)
k
v
(ess sup.km 2 (ess sup
-
al
-
6:), E).
By definition of ess sup X, both these sets belong to V and at least one of them is nonempty and has positive probability. Define the set G*
G+, #(G>+)k PG-) =
G -, otherwise
and we may conclude that
1 Jc- xdp ,(c-)
(13.49) now follows on letting
E.
z (ess spp approach 0.
m -:.
(13.50)
*
of 13.22 #ut X #(SIV) #(S) in the lemma, noting that this is a Observe that, for any G e V, V-measurable r.v. lying between +1 and E'(G)-1 .' IJ crxl'l = IPCH, G) Ph'j l Hence the lemma together with (13.44) implies that, for any H e R, Proof
=
-
-1.
-
.
9(N,J#)
ess sup
k with probability 1.
.
f#(SIV)
l#(SI V)
-
#4&
l
-
,
#(S))
(13.51)
14 Mixing
14.1 Mixing Sequences of Random Variables For a sequence
T-Jx
(X,())J=-x,let
=
of...,Xt-z,Xt-j,Xt) as in
913.5,and
similarly
The sequence is said to be vmixing (or define5*t+m n(Xt+m,Xt+m+3,.%+m+1,...4. lim,u-Axa,u 0 where strong mixing) if =
=
qm
sup
=
(x(T-Jx,@7+,,,),
(14.1)
l
and a is defined in (13.43).It is limpz-jxtj,u0, where
said to be
=
Sup
=
zrl
(
b-mixingtoruniform
9(?F-f x,T7+r,,),
mixing) if
(14.2)
and (j is detined in (13.44).(-mixing implies a-mixing as noted in j 13.6, while the converse does not hold. Another difference is that g-mixing is not timex) reversible', in other words, it is not necessarily the case that supytF/ +,,I,T-f =
supytT-lx,T''; By contrast, a-mixing is time-reversible. If the sequence (.X))+-7 is a-mixing, so is the sequence ( Ffl+-7 where J'f X-t. 0 where (#,((.t)))=-xis also said to be absolutely regular if limp-xp,,, +,,).
=
=
p,,,
sup
=
t
13(8?;-'-,8?7-,+,,,)
(14.3)
and 13is defined in (13.46).According to the results in j13.6, absolute regularit,yis a condition intenuediate between svong mixing and uniform mixing. On the other hand, if (m)+= is a stationary, fmbounded sequence, and @0-x c(...,X-1,A%) and %m+* c(X,,,X,,+1,...), the sequence is said to be completely O where p is defined in (13.47).In stationary regular if p,,, p( T0 J;+oo m ) regllarity complete is equivalent to strong mixing. KolmoGaussian sequences, gorov and Rozanov (1960)show that in this case =
..x
=
-->
=
-x,
,
qm/
Lzam.
pm <
(14.4)
In a completely regular sequence, the autocovariances T Extxt-j) must tend to x. A sufficient condition for complete regularity can be expressed in 0 as j terms of the spectral density function.When it exists, the spectral density fh) is the Fourier transform of the autocovariance function, that is to say, =
-->
f z)
1 =
t=
77
c'l?
2,s jzz--sn z)
,
,
e
(-a,a1.
(14.5)
Theory of Stochastic Processes The theorem of Kolmogorov and Rozanov leads to the result proved by Ibragimov and Linnik (1971: th. 17.3.3), that a stationary Gaussian sequence is strong mixing when fk) exists and is continuous and strictly positive, everywhere on
(-a,cj.
This topic is something of a tenninological
minefield. KRegularity' is an undescriptive term, and there does not seem to be unanimity among authors with regard to usage, complete regularity and absolute regularity sometimes being used synonymously. Nor is the list of mixing concepts given here by any means exhaustive. Fortunately, we shall be able to avoid this confusion by sticking with the strong and uniform cases. While there are some applications in which absolute regularity provides just the right condition, we shall not encounter any of these. might be thought appropriate as a synonym for Incidentally, the tenn regularity, but should be avoided as there is a risk of confusion with wetz/cdependence, a term used, often somewhat imprecisely, to refer to sequences having summable covariances. Strongly dcpendent sequences may be stationary and mixing, buttheircovariances are non-summable. (dWeak' implies Icswdependencethan <strong' in this instance, not morel) Confining our attention now to the strong and uniform mixing definitions, measures of the dependence in a sequence can be based in various ways on the rate at which the mixing coefficients xmor 4u tend to zero. To avoid repetition we will discuss just strong mixing, but the following remarks apply equally to the ;-x Since the collections uniform mixing case, on substituting 4)for Gthroughout. and 5*t+m are respectively non-decreasing in t and non-increasing in t and m, the sequence f(xp,lis monotone. The rate of convergence is often quantified by a summability criterions that for some number ? > 0, xm 0 sufficiently fast that tweakmixing'
f
-->
X
7711* < r;=1
x.
(14.6)
The term size has been coined to describe the rate of convergence of the mixing numbers, although different definitions have been used by different authors, and the terminology should be used with caution. One possibility is to say that the if the mixing numbers satisfy (14.6).However, the commonsequence is of size White 1984) is to say that a sequence is G-mixing of example for est usage (see *.11 if am Om -#) for some fp > It is clear that such sequences are size when of the 1/:0, raised to summable so that this concept of size is power stronger than the summability concept. One temptation to be avoided is to define the size as < whe re f? is the largest constant such that the a) * are summable' ; for no such number may exist. Since mixing is not so much a property of the sequence (.X)) as of the sequences of c-fields generated by (A), it holds for any random variables measurable on those c-fields, such as measurable transformations of Xt. More generally, we have the following implication: -(2
-(p()
=
-(?,
14.1 Theorem Let Ff = g(Xt,Xt-3,...,Xt-x) be a measurable function, for finite of size then Ff is also. lf Xt is a-mixing (g-mixing) -(?,
T.
Mixing
211
= Proof Let V-fx= c(..., 1$-1,F, and G*twm c(I$+,,,,im+1,...). Since F, is measuri-Jx which each of Xt.%-j,...,Xt-z are measurable, Wlx i able on any c-field on and it follows that ay,,u f (u-s and M*t-bm i T7+,u-s. Let ay,,, supr (N-fx,Vc;+,,,) for m k 1. With tinite, (xm-z O(m-%) if um Om-%) and the conclusion replacing follows. The same argument follows word for word with w =
.1
=
=
d9'
ttx'
.
14.2 Mixing lnequalities Strong and uniform mixing are restrictions on the complete joint distribution of the sequence, and to make practical use of the concepts we must know what they imply about particular measures of dependence. This section establishes a set of fundamental moment inequalities for mixing processes. The main results bound the m-step-ahead predictions, Extwm I5-tx). Mixing implies that, as we try to forecast the future path of a sequence from knowledge of its histol'y to date, looking further and further forward, we will eventually be unable to improve on the predictor based solely on the distribution of the sequence as a whole, EXt+m). The r.v. Extm lT-'x) EXt+m) is tending to zero as m increases. We prove convergence of the fw-norm. -
14.2 Theorem
p k 1 and with um defined in (14.1), lI,< 2(21/: + 1)a,,,1/r-1/rl1m+,,,lI,.(14.7)
(Ibragimov 1962) For r
IlF(A+r,,IT-'-)
-
Exm)
Proof To simplify notation, substitute X for Xtmm,V for s-too, R for T7+,,,,and a for xm.It will be understood thatxis an X-measurable random variable where V, R c 5. The proof is in two stages, tirst to establish the result for IXI < Mx < cxo a.s., and then to extend it to the case where X is G-boundedfor t'inite r. Define
the V-measurable r.v. q
=
sgn(F(Xl V) F(m) -
1, F(.Yl V) k #(m, =
-1, otherwise.
(14.8)
Using 10.8 and 10.10,
9') A'IFIXI
-
F(ml
=
=
V) -F(-Y))1 FEn(F(XI FEtA'tn.YlN) -nF(m1
= Covtq,m
ICovtq,m I
=
.
(14.9)
Let F be any V-measurable r.v., such as q for example. Noting that R) -F(F)) is A-measurable, similar arguments give sgn(F(FI Icovt-Y,F)
R4 -A'(l'))) I I IF(X(&FI < F(IXII(&FIR4 -F(F))I) < Mxf'l F(y1 R) F(F) l =
-
(
=
Theory of Stochastic Processes
1Z
G A&1COv((,F)
l
(14.10)
,
where the tirst inequality is the modulus inequality. ( and n are simple random variables taking only two distinct values each, so define the sets A+ (n = 1 ), =
A-
(n
=
-1
=
together gives
), B*
FlFl.Yl T) F(m -
=
()
=
=
1 ) and B,
l K Mxl Cov(n,k)l
Mxj (#(A+
chB+) + P(A-
=
((
=
=
Mxlfttnl
fa#-)
PA+c
-
(#(A+)#(#+) +PA-)PB-)
l
-1
Putting
.
-
F(()F(n)
B-)
I
PA- rn #+)1
-
PAhP(B-j
-
(14.9)and (14.10)
PA-4PB+)j
-
< 4Mm. Since IF(-Yl N) F(m -
l < IF(#IT) 1+ lF(ml f lMx. V) F(mIIP S 2M,x(2a)1/F. llF(-YI
I
(14.11)
it follows that, for p
1,
(14.12)
-
This completes the first pal't of the proof. The next step is to let X be Lr-bounded. Choose a finite positive Mx, and define Xl 1( IxjsMxlx and .Y2 By the Minkowski inequality and (14.11), X =
-
=
m .
IlF(.YlN)
-
+ T) e'(A)11, < IIF(-LIT) -F(-Yl)lI, A'(mII, 11(721 2A.za'(2a)1/J' + zjjxallp, u -
and the problem is to bound the second right-hand-side llX2ll/?f
for r
member. But
sf-rlr IlA'IIrr/P
(14.14)
p, so we arrive at 11.E'(x1 N) F(#)ll, < -
Fillally,
(14. 13)
Choosing
Mx
1IXl)ra-
=
1Mx2%)31P I/C
zM-l--rll,-Yllr/.
(14.l5)
and simplifying yields
11F(-Y1 N) -.E'(A')11,/
which is the required result.
+
2(21/P+
lltxl/#-l/rjjxjjr,
*
There is an easy corollary bounding the autocovariances of the sequence.
14.3 Corollary For p
1 and r plp 1), (14.16) ICov(x,,.Y,+,,)l < 2(21-1* + 1)a)-1/#-1/rjjx;IIp1Ix,+mI!r. >
-
Proof
ICovxvxmb l
=
IEXtxtwmj
-
F(A)F(-'+,,,)
I
= IExtExt-vm I9,) 1lpllA'(X,+,, I@,) F(.&+,2II,/?-l) f 11.X) 2(21-1/P+ 1)jj#,ljpIlX,+mIl<)-1/P-1/r f EXt+mttj
-
l
-
(14. 17)
Vixing
213
where the second equality is by 10.8 and 10.10, noting that Xt is Fsmeasurable, the tirst inequality is the Hlder inequality, and the second inequality is by 14.2. . 1.4.4 Theorem
(Serfling 1968: th. 2.2) For r k p k 1, IlA'(Xf+??, r. I11x) EXt+m) < 29,,,1-1/r1I.&+,,,II
11p
-
where
4),,,is
defined in
(14.18)
(14.2).
Proof The result is trivial for r p 1, so assume r > 1. The strategy is to prove the result initially for a sequence of simple r.v.s. Let Xt-vm Z#,=1.xj1xj, = nxtmm,xtwm.vj, For some e ( consider the random 4/ e 17+,,,,where s-tmm T-fx)(t')), Ext-vm although for clarity of notation the dependence on ) is element l not indicated. For r > 1 and q r/tr 1), we have =
=
=
...).
=
IExt..m I@-'-)
E(Xt+m)3r
-
'>7
j
-
l@-'-)
k-xipAi
=
-
i
i
1
r
IxflI#(4fI@-'-)-#(4I
S
r
#(4f))
r '>7
= f
ImlI#( i I5t ) z1.
i
77Ixi I I#(X i I5-t X
l/rl
x) -
#(X
ij
i
S
E IXt-mIrl @-,-)+
1/t?
I #(AjI@-r-)
PAi)
-
--
#(Af) I
-
) (77
I 77I#(A I5
X?
-'
-) -
i
r)
I FIxt-vm
i
IPAi l@-'-)
-
#(Af) I
PAi)
X.
)
I
(14.19)
The second inequality here is by 9.25. The sets Aj pmition and T-rx)+#(Af'I5-t=) a.s. and PAiwAk') PAi QJA/' I@-rx) #(AfI PAi) + PAi') for # i'. Letting A) denote the union of all those Aj for which P(Ai I@-rx) PAi) k 0, and AJthe complement of 4) on f1, ,
=
=
-
77I#tAfl @-'-)-#4A/)1
I#(A) I@-'-)
=
#(A1)I+ IPCAII@-'-) #(A7)I (14.20)
-
j
-
.
By 13.22, the inequalities
lPA; I@-'-)
#(41)
IPA; I@-'-)
PA;)
-
-
l < 4u I S 9,,,
hold with probability 1. Substituting int (14.19)gives
lExmf
@-'-) Exm) -
Taking expectations
a.s. (14.21) ge'(IAu,,, Irl @-r-)+ Flx,+,,,l rjtztTl,,,lr&
Ir
and using the law of iterated expectations
t E If(xf+r,,l T-x)
-
rIq Ext,vm) Ir S lElXt+m Ir (29,,,)
and, raising both sides to the power 1/r,
then gives '
,
(14.22)
214
Theory of Stochastic Processes f 2IIA+-II41 F(A7,a)IIr -1/r.
IT-'-) Il'(kt+,,,
-
(14.23)
Inequality (14.18)follows by Liapunov's inequality. The result extends from simple to general r.v,s using the constmction of 3.28.
ke For any nv. X:+mthere exists a monotone sequence of simple r.v.s fxkywm, IxY(z)r+,,,() 0 as k N ) such that Xt-vmlt l for all e f. This converx) @-f Exkymm), k e (N) by transfers the sequences (F(aV)f+,,, l a.s. to gence 10.15. Then, assuming Xtwmis fv-bounded, the inequality in (14.22)holds as k Ir as the x by the dominated convergence theorem applied to each side, with lxfx.m dominating function, thanks to 10.13(ii). Tlis completes the proof. w -->
--)
=,
-
-
-->
The counterpart of 14.3 is obtained similarly.
14.5 Corollary
For r k 1, ICov(-Yr+,,,.x))l
where, if r
=
1, replace
s zgml/rjjxrjjrjjx/+mjlrytr-j),
by 1IX,+,,,IIx ess IlXf+r,,IIr/(r-1) =
(14.24)
sup Xt.vm.
Proof
ICov(x)+,,,,.')I I9-f-) F(-Y,+r,,)lIs(r-l) s lIA-,IlrlI'(.2t+,,, -
s 2#-17rlIx,lIrII=,+-lIr/(r-1), where the first inequality corresponds is by 14.4. .
(14.25)
to the one in (14.17),and the second one
These results tell us a good deal about the behaviour of mixing sequences. A fundamental property is mean reversion. The mean deviation sequence (A Ext) ) must change sign frequently when the rate of mixing is ligh. lf the sequence exhibits persistent behaviour with Xt Ex tending to have the same sign for a large number of successive periods, thep IE(Xt+mI5-t.) Extmm) j would likewise tend to be large for large m. If this quantity is small the sign of the mean deviation m periods hence is unpredictable, indicating that it changes frequently. Butwhile mixing implies mean reversion, mean reversion need notimply mixing. Theorems 14.2 and 14.4 isolate the properties of greatest importance, but not the only ones. A sequence having the property that Ilvar(Ap, I9-',.) Vart-mlllr > 0 is called conditionally heteroscedastic. Mixing also requires this sequence of norms to converge as m x, and similarly for other integrable functions of Xtwm. Comparison of 14.2 and 14.4 also shows that being able to assert uniform mixing can give us considerably greater flexibility in applications with respect to the existence of moments. In (14.18),the rate of convergence of the left-hand side to F-Jx) Ext+m) I zero with m does not depend upon p, and in particular, FIE(Xt+mI whenever exists 0, condition infinitesimally > for a 117,+,,,111+8 converges stronger than uniform integrability. In the corresponding inequality for am in 14.2, p < r is required for the restriction to Likewise, 14.5 for the case = p 2 yields -
-
-
-
--+
-
Kbite'
.
Mixing
215
I 2#,,,1/2jI.;t1I2IIx,+,,,Il2,
lCovxmnx
but to be useful 14.3 requires that either Xt or Xtwmbe fo-bounded, Mere existence of the variances will not suffice.
(14.26) for
>
0.
14.3 Mixing in Linear Processes fmlO-x
which arises very frequently in econometric A type of stochastic sequence modelling applications has the representation
Xt
=
q
jzt
-j
,
0
<-
(14.27)
-<
q
c'o
,
jcut where (Z,1+-7 (calledthe innovations or shoks) is an independent stochastic is a sequence of fixed coefficients. Assume without loss of sequence, and ( %lp=() generality that the Zt have zero means and that % 1. (14.27)is called a moving average process of order q (MA()). lnto this class fall the finite-order autoregressive and autoregressive-moving average (ARMA) processes commonly would clearly like to know when such model We economic series. used to time sequences are mixing, by reference to the properties of the innovations and of the sequence f 0;1. Several authors have investigated this question, including =
Ibragimov andLinniktlg7l), Chanda(1974), Gorodetskii (1977),Withers (1981a), Pham and Tran (1985),and Athreya and Pantula (1986a,1986b). Mixing is an asymptotic property, and when q < x the sequence is mixing infinitely fast. This case is called q-dependence. The difficulties arise with the cases with q x. Formally, we should think of the MA(=) as the weak limit of a sequence of MA() processes; the characteristic function of Xt has the form =
tlk/)
=
Iltllzr-/fz-t) ,
j-j
#t?f(%) ($(D(pointwisein
(1k.28)
cxp where R) as q 9r(l) is a ch.f. and 0, we may invel't the latter according to 11.12, and identify the continuous at 12 T 0Z The existence of the limit corresponding distribution as that of Xt Zy ) t-j. imposes certain conditions on the coefficient sequence f0./17,0.We clearly need oo, and for the variance of Xt to exist, it is further necesl $1 0 as j the sequence be square-summable. Note that the solutions of finitesary that order ARMA processes are characterized by the approach of l0.j1to 0 at an exponential rate, beyond a finite point in the sequence. If (4) is i.i.d. with mean 0 and variance c2, Xt is stationary and has spectral density function
and if
-->
,
--
=
=
=()
-,..:
--
.wl
co
f (1) 2:: Tjeihi jujl '''
=
2 .
(14.29)
The theorem of Ibragimov and Linnik cited in j14.1 yields thexconditionZ7=()l $1 strong-mixing in txl the for sufficient Gausjian another However, case. < as standard result (seeDoob 1953t ch. X!8, or Ibragimov and Linnik 1971: ch. 16.7)
Theoty of Stochastic Processes
states that every wide-sense stationary sequence admitting a spectral density has moving average representation with orthogonal increments and a (doubly-infinite) square summable coefficients.
But allowing more general distributions for the innovations yields surprising results. Contrary to what might be supposed, having the %tend to zero even at an exponential rate is not sufticient by itself for strong mixing. Here is a simple illustration. Recall that the first-order autoregressive process Xt pm-l + Zt, p/, j 0, 1,2,... Ip I < 1 has the MA(x) form with % =
=
,
=
14.6 Example Let (Z,)7' be an independent sequence of Bernoulli r.v.s, with P(Zt Ztl and 0) = zl Let 1) Pzt =
=
ax'c
=
Xt
=
.
=
.zXl t- j
Zt
+
=
t
F'')l-izt-p
t
1,2,3,
=
(14.30)
...
lt is not difficult to see that the term f
X
2-lZt-j
t
=
2-Xlkzk
(14.31)
belongs for each t to the set of dyadic rationals Bs (kl2t, k = 0,1,2,... 2f+1 2f+1 1 ) Each element of Ff corresponds to one of the p ossible drawings =
-
.
(Zo,...,4),
and has equal probability of 2 -fXt e Bt
whereas iff
ztl
=
=
fkllt, k
1
.
Iff Zo
0,2,4,...,242f
=
=
-
0,
1)),
1, Xt e J'F'f Bt -
=
Lkllt ,
k
=
13 5 ,
,
2f+1 I j -
,...,
.
It follows that (.Y()= 1 ) /a fX, e Btj 0, for every finite /. But it is clear that ,1 PIXO = = 0) P(Xt 6 SJ Hence for every finite m, =
=
.
)
q?n X #(
(&
=
1 ) rn (A% e Bm))
= t,
which contradicts
im
.--.h
-
#(V
=
J
1)#(Xm e Bm4
(14.32)
0. n
Since the process starts at t = 0 in this case it is not stationary, but the example is easily generalized to a wider class of processes, as follows, be an independent sequence of Bern14.7 Theorem (Andrews 1984) Let IZtIEMJx If Xt oulli r.v.s, taking values 1 and 0 tixed probabilities p and 1 (A)*-oo + mixing.' is pXf-1 Zt for p e (0,!, not strong n ,with
-#.
=
Note, the condition on p is purely to expedite the argument. The theorem surely holds for other values of p, although this cannot be proved by the present
approach.
Proof Write Xt+s = p'Xf + Xta where
Mixing
217
x-1
Xts
fZt-s-j. 'X7 j=
=
(14.33)
The support of Xzs is finite for finite p, having at most 2, distinct members. Call this set Bi, so that Wr! (0,1), Fa = (O, 1, p, 1 + p), and so on. ln general, 8$+1is obtained from Wrxby adding p' to each of its elements and forming the union of these elements with those of Bi; formally, =
WQ1= F,
QJ
tw+ p&:w
v,J,
s
s
=
(14.34)
2,3,...
For given s denote the distinct elements of B&by wj, ordered by magnitude with wl < < wJ, for J < ls. Now suppose tha t Xt e (0 p) so that ps#t e (0 p'+1) This means that Xtms for some-j. Defining events A = (aYtE assumesa va lue between wj and wy+ p&+1 r'+1) w+ (0,p)l and Bs lXf+, e Ul/=ltwj, ) we have #(#,IA) == l for any s, howeverlarge. To see that #(A) > 0, consider the case Zt Zf-l Zt-z 0 and zr-a 1 and note that ...
,
,
.
,
,
=
,
=
=
=
x
xp?
;jS j=3
for p e (0,1g.So, unless PBs)
3
x
t-j u ppz
=
j=3
j
P
<
p
.
(14.35)
p
1, strong mixing is contradicted. (Xf e (p,1)l has positive The proof is completed by showing that the set D probability, and is disjoint with #,. D occurs when Zt 0 and Zf-1 1, since =
=
=
=
then, for p e (0,,11,
X
oo
ppz p s :-l j= t-j
and hence #(D)
>
y--lp/ j P ) s
u
=
-
(14.36)
1,
/=1
0. Suppose that
min (ws1 ./k1
-
wyl k
p
t <
+
J-1
(14.37)
.
Then, if D occurs, wy+
p
J+1
pSXf B
f uj. +
p
U/ J=1 (w j,
sy
wj
p&-1s wy
(1438)
1,
.
pS+1)
0. or in other words, Bs rn D The assertion ln (14.37)is certainly true when s 1, so consider the following inductive argument. Suppose the distance between two points in J7, is at least cannot be ()'-1 Then by (14.34),the smallest distance between two points of +,+1 rlpJ-l pf-l when which pf S p' But leSS than the smaller of pf and p e (0,r1), s s impliesp'-1 p 2 p It fo llows that (14.37)holds for every s. w hence, Xtn
wj +
=
wy+
=
,
=
.
-
.
,
-
.
These results may appear surprising when one thinks of the rate at which p' but if so, this is because we are unconsciously thinking approaches 0 with about the problem of predicting gross features of the distribution of Xtmsfrom time t, things like Pxt-vs f xlA) for fixyd x, for example. The notable feature of the sets Bs is their inrlevance to such concerns, at least for large s. What we -;
Theory of Stochastic Processes
218
have shown is that from a practical viewpointthe mixingconcept has some undesi able features. The requirement of a decline of dependence is imposed over 6 events, whereas in practice it might serve our pumoses adequately to tolera certain uninteresting events, such as the Bs defined above, remaining dependent c the initial conditions even at long range. ln the next section we will derive some sufficient conditions for strong mixinl and it turns out that certain smoothness conditions on the marginal distribution of the increments will be enough to rule out this kind of counter-example. But no. consider uniform mixing. 14.8 Example 13C ons ider an AR(1) process with i.i.d. increments, Xt
pX,-l
=
+
Zt, 0
<
<
p
1,
in which the marginal distribution of Zt has unbounded support. We show that (,X)) is not uniform mixing. For 8 > 0 choose a positive constant M to satisfy m-l
plzm-j < # X 7=0
-.M
<
(14.39)
.
Then consider the events
yt
=
(L +A/)) e 10 p '-m
(&
-x
fXm < Lk e sm+* where L is large enough that #(#) k 1 8. We show #(AI> 0 for every m. Let pK = #(Zo < A'), for any constant K. Since Zo has unbounded support, either px < 1 for every K > 0 or, at worst, this holds after substituting (-Z,) for (Zf l and hence ( -XtI for (Xf). pk < 1 fOr a1l K implies, by stationarity, #(X-l < 0) = Pxo < 0) < 1. Since (.%< A-Ji (Z0 < X') QJ (fZ() 2 A') f'n IX-I < 01), independence of the (Zf l implies that B
=
-
PIXZ
So PA) that
>
PB
<
X) K pK+
(1-#r)#(X-1
<
0) < 1. PXJA'C+ ZPJ
0, since K is arbitrary. Since Xm =
I4)
=
# p'Vc +
m- 1
X pizm-j < L
by (14.39).Hence 4u k I#(#lA) means ()a, = 1 for every m. n
-
PB4
p'Ve
I> 1
-
2,
L +M
(14.40) .-)
p'Zm-j, it is clear
<
and since
(14.41) is arbitrary, this
Processes with Gaussian increments fall into the category covered by this example, and if (hmixing fails in the first-order AR case it is pretty clear that counteru examplej exist for more generalMAtx) ases too. The conditions for uniform mixing in linear processes are evidently exemely tough, perhaps too tough for this mixing condition to be very useful. ln the applications to be studied in later chapters, most of the results are found to hold in some form for strong mixing processes, but the ability to assert uniform mixing usually allows a relaxation of
Mixing
219
conditions elsewhere in the problem, so it is still desirable to develop the parallel results for the unifonn case. The strong restrictions needed to ensure processes are mixing, which these examples point to (to be explored further in the next section), threaten to limit the usefulness of the mixing concept. However, technical infringements like the ones demonstrated are often innocuous in practice. Only certain aspects of mixing, encapsulated in the concept of a mixingale, are required for many important limit results to hold. These are shared with so-called near-epoch dependenthmctions of mixing sequences, which include cases like 14.7. The theory of these dependence concepts is treated in Chapters 16 and 17. While Chapter 15 contains some necessal'y background material for those chapters, the interested reader might choose to essence, the diftkulty will be skip ahead at this point to find out how,
tn
resolved.
for Strong and Uniform Mixing
14.4 Sufficient Conitions
The problems in thecounter-examples aboveare with theformof themarginal shock distributions or unbounded, as the case may be. For strong mixing, a degree of smoothness of the distributions appears necessary in addition to summability conditions on the coefticients of linear processes. Several sufficient conditions have been derived, both for general MA(x) processes and for autoregressive and ARMA processes. The suftkiency result for strong mixing proved below is based on the theorems of Chanda (1974)and Gorodetskii (1977).These conditions are not the weakest possible in a1l circumstances, but they have the virtues of generality and comparative ease of verification. -discrete
define a random sequence taY/l=-x,where, for 14.9 Theorem Let Xt X*j=eojzt-j either 0 < r f 2 or r an even positive integer, (a) Zt is unifot-mly fv-bounded, independent, continuous with p.d.f. J'rzt, and =
Suy P
+-Ih-zt-va, j..
I
yz,tzldz u Mlzl,
-
for some whenever Ia I f =, < (y.;1&1+r) x gt (b) Zf=0 w jwre
>
,
&l' <
(14.42)
-,
0;
X
I$1 277 j=t
r,
Grtrl
r < 2,
(14.43)
=
gr-l
x
:2j y-'q
r/2 y.
g. ,
j=t
0./.'Ei
0 for a11complex numbers x with jxl Then (.&)is strong mixing with m O(Zx/z.+1 g t (r)1&1+r)) z (C)
0(1)
=
ZJ T =1
#
=
.
Before proceeding to the proof, we must discuss the implications of these three conditions in a bit more detail. Conditton 14.9(a) may be relaxed somewhat, as we
Theory of Stochastic Processes
220
show below, but we begin with this case for simplicity. The following lemma extends the condition to the joint distributions under independence. 14.10 Lemma lnequality (14.42)implies that for
k
k
Jw,
ml-ljfzrtzr''btz'l
Proof
IJrl M
=
1,...,k,
77lJ,l
.
(14.44)
>1
>1
usingFubini-s
6, t
k
dzz...dzk f I1TzJz2
-
<
theorem, k
k jpk mljfz/zr+*l
l1Tz,tzf)dzt...dzk
-
,.-1 f
jpk
+Jl) I-fzltzl
f
I
...dzk
,
, dz,...dzk l-I.fz,(o) I1.fz,(zr+zh)
+jpk +
Jzl
>2
.fzl(zl)
Ml cl
k +J) Tzltzlllllfz/z,
-
-
>2
>2
k
jvk-
mlj/ktz,
k
+
at4
-
I1Jz,(z
dzz...dzk.
>2
The lemma follows on applying the same inequality to the second term on the right, iteratively for / = 2,...,k. w Condition 14.9(b) is satisfied when I0;1 <<j-V for g. > 1 + 2/r when r K 2 and g > 3/2 + 1/r when r k 2. The double definition of Gtr) is motivated by the fact that for cases with r < 2 we use the von BalmEsseen inequality (11.15)to bound a certain sequence in the proof, whereas with r > 2 we rely on Lemma 14.11 below. Since the latter result requires r to be an even integer, the conditions in the theorem are to be applied in practice by taking r as the nearest even integer below the highest existing absolute moment. Gorodetskii (1977)achieves a further weakening of these summability conditions for r > 2 by the use of an inequality due to Nagaev and Fuk (1971). We will forgo this extension, both because proof of the Nagaev-Fuk inequalities represents a rather complicated detour, and because the present version of the theorem permits a generalization (Corollary 14.13) which would otherwise be awkward to implement. tjzt-j, = Detine W'r so that Xt = 11$+ W,and W and J''f ?Zt-j and J// Z7z of Xt, whose contribuare independent. Think of 54 as the Jcx-measurable x. tion to the sum should become negligible as t =
ZJ;0
Stail'
--
14.11 Lemma lf the sequence (Z,l is independent with zero mean, then cr
Evlm) < 22+-1 t
m
)q0?J sup Ezlm)s j=t
(14.45)
s; 0
for each positive integer m such that sup,ucz,
2-
)<
x.
Mixing
221
Proof First consider the case where the r.v.s Zt-j are symmetrically distributed, meaning that -Zt-j and Zt-j have the same distributions. In this case al1 existing odd-order integer moments about 0 are zero, and E
t+k
lm
t+k
jzt-j
t+k
=
0./j...h,nElzt-jq ...Zt-h,n)
...
jkzzzt jzm=t
j=t
t+k =
t+k
1 1 1 Qj2I .0jmE(Zt ...Zt-jm) -./1
...
..
./l=f jmzzt 1,0J:
Aq
s X 02j sup Ezlm).
(14.46)
sf 0
j=o
The second equality holds since EZt-jL...Zt-jzy) vanishes unless the factors form matching pairs, and the inequality follows slnce, for any r.v. F possessing the 0) for j,k > 0. requisite moments, EY i*k) 2 E(YqEYk) (i.e., Covtl'? F$ x. The result for symmetrically distributed Zs follows on letting k For general Zs, 1et Zs' be distributed identically as, and independent of, zx, is independent of J6, and J 7; has symmetfor each s < 0. Then J4 X; rically distributed independent increments Zt-j Z't-j. Hence ,
-->
=
ctjzl-j
-
-
m
X
Evt
lm)
< Evt
-
k';)241<
X 02j sup E(Zt.j Z't-plm -
j=t
x
V
glm- 1
y:=0
m
62 sup Elzltm-jl j
(14.47)
,
/
where the first inequality is by 10.19, the second by (14.45), and the third is the cr inequality. . Lastly, consider condition 14.9(c). This is designed to pin down the properties of the inverse transformation, taking us from the coordinates of (X,) to those of (4J 14It ensures that the function of a complex variable 0@) possesses an anajzzorykxl.for xl S 1. The particular property needed and lytic inverse I(x) l coefficient sequence (':y) is absolutely condition the by is implied the that under 14.9(c) the inverse representation is also summable. If Xt X; TJ 1 if % 1. An effect of 14.9(c) is Note that 'Io detined, as Zt example where 0@) rule for 01(x)41 x) out cases, as to with 01(.)a summable polynomial. The differencing transfonnation does zltp/yield a mixing process in general, the exception being where it reverses the previous integration of a mixing process. For a t'initenumber of terms the transformation is conveniently expressed using matrix notation. Let .
'
=
x
=()0.j4s,
=
=
tover-differenced'
=0pms.
=
=
=
-
Theory of Stochastic Processes
222 1
0
01 An
01
=
n x n),
(14.48)
0a-2 0n-1 0n-2
01
(-10jzt-j, Zy=0 z (z1,...,za)
1 n can be written x A?g where x is also lower triangulat', with elements = @1,...,ak)and z 0,...,a 1. If >' (v1,...,va)the vector &' Aa # has h replacing 0j for J t-1 f for 1 These operations elements Zyzoljvr-j, can in principle be taken x, subject to 14.9(c). to the limit as n so
that the equations xt z
-1
t
=
,.
=
..,
A,T
= .
=
.
=
-1
%
=
-
=
...,n.
=
,
-->
Proof of 14.9 Without loss of generality, the object is to show that the c-fields 0 = c(...,X-1,Xo) and Ji'x' x. The F-x m+2,..., ) are independent as m m+1 c( X-+1 result does not depend on the choice of origin for the indices. This is shown for a sequence f.f t )7+k for finite p and k, and since k and p are arbitrary, it then follows by the consistency theorem (12.4)that there exists a sequence (XfJ'lx whose finite-dimensional distributions possess the property for every k and p. This sequence is strong mixing on the definition. where 2% = Xt-p,...,Xo)' (px 1), #1 Define a p + m + k-vector X X6X1,XL)' = Xm+L,...,Xm+' k x 1), and also vectors W = (X1,...,Ak)' m x 1), and Xz (W(,W)' and F (FJ,FJ)'such that #1 IFI + F1 and Xz +2 + Fa,.(The elements and F are independent of W. of W and F are defined above 14.11.) The vectors Now, use the notation X = GXg,...,Xt) and define the following sets: ,x
=
-->
=1-p
=
=
=
=
=
.%
0 for some C e e C) e @1-p,
G
=
f(t):
#(,()
H
=
(:
A%() e
DJ
E
=
f ):
Fz()
#) e
e
for some D m e J;'7l,
G
TP ,
Bk ,
10 -x,
where # (p2:Ir21 f n) e T k lv21 denotes the vector whose elements are the is a vector of positive constants. absolute values of p2, and z1 = (q,,,+1,...,n,,,+z)' Also define D pa = (w2:.,2 +#2 e D) e T k =
,
-
.
H may be thought of as the random event that has occun-ed when, first F2 = 1/2 is realized, and then W2 e D p2. By independence, the joint c.d.f. of the variables (Wa,F2,A%) factorizes as F = FwzFh,xo (say) and we can write -
#(&
=
#(X2 e D)
JsJss.va#F(+2,V2,20) =
=
jepjekzvltdFvz,xvl'x'
(14.49)
Mixing
223
where
:(1/2)
#(W2 e D
=
-
v2)
jp-vgdFwzw.
=
( 14.50)
These definitions set the scene for the main business of the proof, which is to show that events G and H are tending to independence as m becomes large. Given 5IB P'm'k ility of X this is sufficient for the result, since C and D are arbitrary. By the same reasoning that gave (14.49),we have -measurab
,
#(G Detine k*
E4
/'7 HIA
supyzssztpz) and
=
X.PIGcjE)
jcjg(v2)#Fvz,xa(v2,&).
=
x.
infpzssztva), and
=
(14.51)
(14.51)implies (14.52)
f PIGCHIAEI < z.PGcjE4.
Hence we have the bounds
PCGchP
+#45rn PCGCDHI-?EI
=
z*#(G)
; and similarly, since
-
k k.P(G4 f
(14.53)
+#(Grnz;faFC)
k z.PGcjE)
= z.P(G)
=
PEc),
+
z. K 1,
PGcllj
Choosing G
H(nEcl
(i.e., C x.
Ec) + #(G chHcj Ec4
fa
P(Ec4.
R#) in
=
PCEC4
-
-
k.PG
( 14.54)
(14.53)and (14.54)gives
f #(& S
z*+
PCEC),
in particular
(14.55)
and combining all these inequalities yields 1#(G rn /f)
1 f x* x++lPEc).
#4G)#(&
-
-
(14.56)
Am+kZ,where Z (Z1,...,Z,,,+k)'and Amn is defined by (14.48).Since I4,?,+I 1 and the (z1,...,Z,,,+kl are independent, the change of variable formula from 8.18 yields the result that G is continuously distributed with Write IP
=
=
=
m+k
/(w) Define B '
(r:
=
y.
-z.
A'I
=
0 v2 ,
G
fztz)
=
#1
I
-z(0)
u 2 sup x(v2) vzB S 2 sup v2G#
JsI
G
11fz,(zr).
=
(14.57)
>1
Bm+k Then the following relations hold: .
j
/ctwz + #2) /c(w2) -
I
dw1
Theor.v of Stochastic Processes
224
jpm..k I /vt.'
u 2 sup, B G
&,
+
v)
l
-yv(w)
dw
m+k
m+k
flJz/z,+ J.,a+z
= 2 #sup. B'
t'%
*'
6
E
p e B'
I-lJz/tz,)
-
>1
>1
m+k
< ZM sup
r)
19,1
(14.58)
,
>-+1
where it is understood in the final inequality (whichis by 14.10) that );, l f W here 6 is defined in condition 14.9(a). The tllird equality substitutes 7 = Am-+1kv 0 if vl = 0 by lower triangularity of Amn. For v e and uses the fact that ;'I #' note that =
1
m+k
m+k
77 I I
l
t-m-
77 jszj 77 zjvt-j
v-,
1=-+1
>-+1
t-m- 1
m+k
S
77 jzujj :7
>-+1
m+k
x
XI
I&ln,-; K
':/1
/=0
A'lnr,
(14.59)
t-nm
assuming has been chosen with elements small enough that the terms in parentheses in the penultimate member do not exceed 6. Tlzis is possible by condition 14.9(c). For the final step, choose r to be the largest order of absolute moment if this is does not exceed 2, and the largest even integer moment, otherwise. Then 'rt
PEc)
lF21 >
P
=
'q)
m+k
P
=
U f I141
>-+1
>
n,l
m+k
m+k
77#(I7,I <>-+1
>
n,)K
>7FI7,I
rn-,r,
(14.60)
J=ra+1
by the Markov inequality, and E j 7, Ir < sup E Iz, IrG,(r),
(14.61)
'
where Gt is given by (14.43),applying 11.15 for r 2 (see(11.65) for the required extension) and Lemma 14.11 for r > 2. Substituting inequalities (14.58), (14.59), (14.60),and (14.61)into (14.56)yields m+k
l#(G
chffl
-
PG4PH4
I
X (n,
+
t=m+3
G(r)n;3.
(14.62)
Mixing
225
i 0 by 14.9(b), it is possible to choose m large enough that (14.59)
Since Gt
andhence (14.62)hold l P(G r'3S)
with
n,
Gtr) 1R1+8 G t (rln-r t for each f > m. We obtain
=
=
m+k
*(G)#(S)
-
l
d
t%(r)1/C1+r' 77 >-+1 X
K
1/(1+r)
77G'r) /=-+1
(14.63)
,
where the right-hand sum is finite by 14.9(b), and goes to zero as m completes the proof. w
This
-->
x.
It is worth examining this argument with care to see how violation of the conditions can lead to trouble. According to (14.56),mixing will follow from two O conditions: the obvious one is that the tail component F2, the F-x-measura bje part of A%,becomes negligible, such that is, #(F) gets close to 1 when m is large, even when Tl is allowed to approach 0. But in addition, to have x* x. disappear, 0, for any D, and whatever #(W2 e D r2) must approach a unique limit as 172 the path of convergence. When the distribution has atoms, it is easy to devise examples where this requirement fails. In 14.6, the set Bt becomes Js Bt on being translated a distance of 2-l For such a case these probabilities evidently x. do not converge, in the limiting case as t However, this is a suftkiency result, and it remains unclearjust how much more than the absence of atoms is strictly necessary. Consider an example where the distribution is continuous, having differentiable p.d.f., but condition (14.42) none the less fails. -
-->
-
-
-..-)
g
.;
14.12 Example Let fz) Coz sin (z ), z e R This is non-negative, continuous everywhere, and bounded by C()c-2 an d hence integrable. By choice of Q we can *fztdz 1, so f is a p.d.f. By the mean value theorem, have =
.
=
Ifz
+
ab
f(z) I
-
=
Ia IIf'z
wheref '(z) 8c osint/lcost/k x and hence, =
-
I
+ a(z)J)
,
2c()sin2(/)g-3
(14.64)
a(z) e (0,11, .
But note that
J+OI f'l ldz
=
-x
1
1
:*
Ifz
J
' Ia I
.-
+ J)
f U)ldz
-
'--h
'x'
as
Ia l
-->
0,
(14.65)
which contrdicts (14.42).The problem is that the density is varying too rapidly in the tails of the distribution, and Ifz + a4 fz) l does not diminish rapidly 0. enough in these regions as a The rate of divergence in (14.65)can be estimated. 15 F or fixed (small)a, sin (z+ a) 4 1 (or 0) Ifz + a) fz) I is at a local maximum at points at which tz)4 z4 4.3 + o(u2) and sin z4 0 (or 1), or in other wor ds w here (z+ :+/2. The solutions to these approximate relations can be written as z = +CI lJI -1/3for C 1 > 0 At these points we can write, again approximately (orders of magnitude are all we need here), -
--
=
-
=
-
.
=
=
Theoty of Stochastic Processes
226
Ifz
a4
+
fz) I S
-
-2 1J
13
I (-C1 It7I 21
lfz)
S ICOC 1
.
-1/3
-1/3j
C 1 jt7l by The integra l is bounded within the interval 1/3 2/3 rectangle having height 2C0C7-1 a I Outside the 4CoC -1 th e area of the I la I and h -2 is bounded by t e integral over this region is bounded by interval,J coz ,
,
.
,
+00 2C0Jcj$(,I-v3Z
Adding up the approximations
dz
for M
<
IJl
2C0Cl
=
.
yields
+x
J..I fz
1/3
.j
.p,
+ J)
fzs 1dz f
-
Ml J
l
js
(14.66)
o
x.
The rate of divergence is critical for relaxing the conditions. Suppose instead of
(14.42) that +oo
J-.I
J)
z+
zbIdz f Mtl
-
could be shown sufficient, where h ltzI) 1, 0 as Ia l 0. Since
JI ), IJ1
f y y s (;
(14.67)
,
an arbitrary increasing function with
(.) is
.
+co
J-.l fz
+ J)
ldz S
f
-
g)+x
(z)tyz m
..f
(14.68)
g
for any a, (14.67)effectively holds for any p.d.f., by the dominated convergence 16 theorem. Simple continuity of the distributions would suffice. This particular result does not seem to be available, but it is possible to relax 14.9(a) substantially, at the cost of an additional restriction on the moving average coefficients. 14.13 Corollary Modify the conditions of 14.9 as follows: for 0 that
;$S
<
1, assume
Zt is uniformly fv-bounded, independent, and continuously distributed with p.d.f. /'z,,and zz-va, (+I sup t
whenever
b-)x-
(
(C ')
tzz)
ct
1/0(x)
<
Itzl
S
(r)I5'(I5+r) <
=
T(x)
-
,
etxl
=
Then Xt is strong mixing
hjz)
:, for some
I
dz
u Mlcl', u
<
(14.69)
-.,
0,'
>
c,(r) is defined in (14.43); for 1x1S 1, and Z'r=I E''=l'r/ J (Tjl 15< J x
,
where
x.
with
m
=
t?(X*t
.,,,+1
Gt (r)lVO+8) .
Proof This follows the proof of 14.9 until (14.58),which becomes
Mixing
227
m+k
z.
-x
+
<
zu
applying the obvious extension m+k
m+k
9,1F
X I
=
using (9.63),since 0
<
y B'
f=?n+1
(14.70)
,
'
of Lemma 14.10. Note that ;$
f-m-j
X X
p=+1
p=?n+l
>'')lf;,1;'
sup
./=0
m+k
x
K
'zjvt-j
XI
X n?
'zj
j=0
(14.71)
f=/?1+1
;'J f 1. Applying assumption
14.13(c'),
m+k
I#(G r-bz,r? -#(G)#(zoI f<
and the
result
is obtained
77 (nf)+c,(r)n-,r),
(14.72)
>-+1
as before, but in this case setting
n,
=
G1,7t+* .
.
Condition 14.13(b') is satisfied when l0./1 ./-B for g > 1/p + 2/r when r < 2, and g > 1/2 + 1/r+ 1/j3 when r 2, which shows how the summability restrictions have to be strengthened when ;'$is close to 0. This is none the less a useful extension because there are important cases where 14.13(b') and 14.13(c') are easily A, both I%I and I satisfied. In pmicular, if the process is finite-order vanish beyond some finite j, and (b') and (c') either decline geometrically or both hold. Condition 14.13(a') is a strengthening of continuity since there exist functions h.) which are slowly varying at 0, that is, which approach 0 more slowly than any positive power of the argument. Look again at 14.12, and note that setting 13 la will satisfy condition 14.13(a') according to (14.65).lt is easy to generalize k Csin 2(z)z for k k 4 the earlier argument is easily the example. Putting j'z) modified to show that the integral converges at the rate Ia 1.1/(k-1)an d tus choice of f'sis appropriate. But for J(z) = Csin 2 e z)z th e integral converges F more s lowly than Ia I for al1 13> 0, and condition 14.13(a') fails. To conclude this chapter, we look at the case of uniform mixing. Manipulating inequalitie (14.52)414.55)yields 'ry1
=
-2
=
,
,
-2
IPIH' G) #4/./)1 S -
(1
z* z. PCEY + -
+
plv) ,
(14.73)
1 for all m exceeding a which shows that unifonn mixing can fail unless #(F) tinite value. Otherwise, we can always construct a sequence of events G whose probability is positive but approaching 0 no slower than PEc4. When the support of X-v,...,Xo4 is unbounded this kind of thing can occur, as illustrated by 14.8. =
The essence of this example does not depend on the AR(1) model, and similar cases could be constructed in the general MA(x) fzamework. Suftkient conditions must include a.s. boundedness of the distributions, and the summability conditions are also modified. We will adapt the extended version of the strong mixing condition in 14.13, although it is easy to deduce the relationship between these conditions and 14.9 by setting f$ 1 below. =
228
Theoty of Stochastic Processes
14.14 Theorem Modify the conditions of 14.13 as follows. Let (a') and (c') hold as before, but replace (b') by (b'') f.o I j I)l5< x, add and (d) (Z,l is uniformly bounded a.s. T hen (mJ is uniform mixing with (,,, O; =,,,+1(E7=f I0./1)f5). XO
xt
=
Proof Follow the proof of 14.9 up to (14.55),but replace (14.56)by (14.73).By condition 14.14(d), th'ere exists K < x such that suprl Zt I < K a.s., and hence I-Y,I 1 < K;=o I0.j1a.s. It further follows, recalling the definition of Fa, that PE) +k. Substituting directly into (14.73) 0./1 for t m + 1,...,< A'Z7=,1 when n, of and a nd making choice this from (14.70) (14.71), n, gives (forany G with #(G) > 0) =
=
,
I,(,.,1
c) rm I -
,
-s+(s-yojj)
(14.74)
.
l=?n+1 j=t
The result now follows by the same considerations as before.
.
x in the These summability conditions are tougher than in 14.13. Letting r OU-B) 14.13(b') for for comparability, satistied is when latter case I%I g, > 1/2 + 1/p, while the corresponding implication of 14.14(b'') is jt > 1 + 1/p. -->
=
15 Martingales
15.1 Sequential Conditioning It is trivial to observe that the arrow of time is unidirectional. Even though we can study a sample realization expost, we know that, when a random sequence is member Xt is determined in an environment in which the generated, the previous members, Xt-k for k > 0, are given and conditionally fixed, whereas the membersfollowingremaincontingent. Thepastisknown, butthefutureis unknown. The operation of conditioning sequentially on past events is therefore of central by specifyimportance intime-series modelling. Wecharacterizepmialknowledge ing a c-subtield of events from @, for which it is known whether each of the events belonging to it has occurred or not. The accumulation of infonnation by an observer as time passes is represented by an increasing sequence of c-fields, F.17 1T,1= suc h that 51 i i @-1 i J70 Tl If Xt is a random variable that is Tr-measurable for each /, ( Tfl=-xis said to The pairs (A,@,)=-x are called an adapted be adapted to the sequence ta)l=-x. /) < s < defines the minimal adapted sequence, but sequence. Setting 9t ctxx, 5t typically has the intepretation of an observer's infonnation set, and can contain more information than the history of a single variable. When Xt is integrable, the conditional expectations A'(ml Fr-1) are defined, and can be thought of as the optimal predictors of Xt from the point of view of observers looking one period ahead (compare10.12). Consider an adapted sequence l&,Tnlo-x on a probability space L't,5nP4, where ( kl is an increasing sequence. If the properties dcurrent'
...?
-x,
...
-x
=
FI,L1 < Esn
I
.n-1)
=
(15.1)
x, s%-l,
a.s.,
(15.2)
hold for every n, the sequence is called a martingale. ln old-fashioned gambling parlance, a mmingale was a policy of attempting to recoup a loss by doubling one's stake on the next bet, but the modern usage of the term in probability theory is closer to describing a gambler's worth in the course of a sequence of fair bets. In view of (10.18),an alternative version of condition (15.2)is Sndp
A
=
Sn-)dP, each A e @n-1.
A
Sometimes the sequence has a tinite lnitialindex, and may be written where 5'1 is an arbitrary integrable r.v.
(15.3)
f,L,F,,lT
F/ldt?ry of Stochastic Processes
230
(X,ITbe an i.i.d.
integrable sequence with zero mean. If Sn Z7=l-Y, and 5n c(Xn,X,,-1,...,X1), l5'q,?LlT is a mmingale, also known as a A'lil walk sequence. Note that S Z7=1A'l-Y,l < x. n random
15.1Example
Let
=
=
15.2 Example Let Z be an integrable, T/f-measurable, zero-mean r.v., ( Kl'-x an increasing sequence of c-tields with limn-yxfn = T, and Sn #tzj ;n). Then E(Sn3 9a-1) F(F(zI@n)I @n-1) F(ZIT,,-1) = .L-1, (15.4) =
=
wherethe second o martingale.
=
equality is by 10.26(i). F1 ,L1 <
FIzI <
by 10.27, so Sn is a
x
on the last definition, a martingale d#erence (m.d.) sequence on L't,5,P) satisfying the properties
Following
fm,sl'<'-x is an adapted sequence
Fl#r1 F(Al Tr-l)
(15.5)
< x, =
(15.6)
0, a.s.,
for every t. Evidently, if fSn) is a mmingale and Xt St St-j. then (.1 is a m.d. Conversely, we may define a martingale as the partial sum of a sequence of m.d.s, as in 15.1 (an independent integrable sequence is clearly a m.d.). However, if Xt has positive variance unifonnly in /, condition (15.1) holds for a1l finite n but not unifonnly in n. To define a mmingale by Sn L1=-=Xt can therefore lead to difticulties. Example 15.2 shows how a mmingale can arise without reference to summation of a difference sequence. It is important not to misunderstand the force of the integrability requirement in (15.1).After all, if we observe Sn-b, predicting Sn might seem to be just a matter of knowing something about the distribution of the increment. The problem is that we cannot eat Esn I&-1,...)as a rt7rltftprl variable without integrability of Sn. Conditioning on S-l is not the same proposition as treating it as a constant, which entails restricting the probability space entirely to the set of repeated random draWings of Xn. The latter problem has no connection with the theory of random sequences. A fundamental result is that a m.d. is uncorrelated with any measurable function of its lagged values. =
-
=
15.3 Theorem If (7r,@t)is a m.d., then Cov(X,,9(.X)-1,X,-2,...))
where
(1) is
any Borel-measurable,
Proof By 10.11
(sze also
=
0,
integrable function of the arguments.
the remarks following) noting that 9(A-1,.&-2,...), is
Tf-l-measurable. w
15.4 Corollary lf (.,121 is a m.d., then E(XtXt-k) = 0, for a1l t and all k
#
Proof Put () Xt-k in 15.3. For k < 0, redefine the subscripts, putting t and t' Il7I t, so as to make the two cases equivalent. m
t k
=
-
=
=
0. -
Martingales
231
One might think of the m.d. propel'ty as intennediate between uncorrelatedness and independence in the hierarchy of constraints on dependence. However, note the asymmetry with respect to time. Reversing the time ordering of an independent sequenceyields anotherindependent sequence, andlikewise areversed uncorrelated sequence is uncorrelated; but a reversed m.d. is not a m.d. in general. The Doob decomposition of an integrable sequence (&,?L)7' is Sn
where Ao
=
0, Mo
(15.7)
Mn +An,
=
%, and
=
Mn An
Mn-t
=
=
.4u-1
I k-1),
(15.8)
Esn lTn-1) 5'n-1.
(15.9)
+
+
Sn Esn -
-
component of Sn.Writing Anis an Fn-l-measurable sequence called Lbepredictable and A,4u hsl'n hSn Xn, we find F(Fn l;t-1), and L =
%
=
=
Xn
=
rn -
E(Ynl?h-1).
(15. 10)
Xn is known as a centred sequence, and also as the innovation sequence of Sn. It is adapted if ( l'n,iu J7'is, and since Fl Fn1 < oo by assumption, E IXn I < E Il',yI + E IEYn lJ;,,-1)I
l Fn IITn-l))
E 1Fn I + EE
=
IE
IraI <
(15.11)
x,
Minkowski' s inequality, the conditional modulus inequality, and by (respectively) 0, (.L,@n)7is a m.d. and so the LIE. Since it is evident that E(Xt I?h-1) mmingale. is lMn,Tnl7 a =
Martingales play an indispensable role in modern probability theory, because m.d.sbehave inmany importantrespects likeindependentsequences. Independence limit results, laws of is the simplifying propel'ty which permitted the large numbers and central limit theorems, to be proved. But independence is a constraint on the entire joint distribution of the sequence. The m.d. property is a much milder restriction on the memory and yet, as we shall see in later chapters, most limit theorem: which hold for independnt sequences can also be proved for m.d.s, with few if any additional restrictions on the marginal distributions. For time series applications, it makes sense to go directly to the martingale version of any result of interest, unless of course a still weaker assumption will suftke. We will rarely need a stronger one. Should we prefer to avoid the use of a definition involving c-fields on an abstract probability space, it is possible to represent a martingale difference as, for example, a sequence with the property iclassical'
E(xt IA-1,Xr-2,...)
=
0 a-s.
( 15. 12)
When a random variable appears in a conditioning set it is to be understood as representing the corresponding minimal c-subfield, in this case c(#f-1sXf-2,...).
Theory of Stochastic Processes
232 tj
This is appealing at an elementary level since it captures the notion of information available to an observer, in this case the sequence realization to date. But since, as we have seen, the conditioning information can extend more widely than the llistory of the sequence itself, this type of notation is relatively clumsy. Suppose we have a vector sequence (4XsZf)), and Xr-though not necessarily c(XsZsXr-1,Zf-1,...) in the sense of Zt is a m.d. with respect to 5t (15.6). This case is distinct from (15.12),and shws that that definition is inadequate, although (15.16)implies (15.12).More important, the representation of conditioning information is not unique, and we have seen (10.3(ii))that any measurably isomorphic transformation of the conditioning variables contains the same information as the oliginal variables. lndeed, the information need not even be represented by a variable, but is merely knowledge of the occurrence/nonoccurrenceof certain abstract events. =
-
15.2 Extensions of the Martingale Concept k Jn=1, w here An adapted triangular array ( tAk, @nf)?21 sequence of integers, for which '-'
'1Xnrl
<
(/(n )= n,z 1 is some
(15.13)
oo,
ft-vtl 9$,f-1)
(15.14)
0 a.s.
=
increasing
for each t 1,...,lk and n k 1, is called a ntartingale dterence array. In many applications we would have just kn n. The double subscripting of the subfield 5nt may be superfluous if the infonnation content of the array does not depend on n, with 5nt 5t for each n, but the additional generality given by the definition k Xnran d 5 n is harmless and could be useful. The sequence t&,Tn ITwhere Sn Xf21 5n,knis not a martingale, but the properties of mmingales can be profitably used to analyse its behaviour. Consider the case Sn = n -1/2E)=1 xt w Iwre (m,@;)is m.d. distdbution of Sn has a Such scaling by sample size the that may ensure a non-degenerate limit. Sn is not a martingale since =
=
=
=
=
ITn-1)
Esn
1/2
E(/l 1)/n1
=
-
5'n-l,
(15.15)
but each column of the m.d. array X1 2 -
-1/2
1/2
X) 3
2 -1/2X2 3
-
X1 4-1/2x 1 X1 4-1/2x 2
.
.
.
.
.
.
.
.
1/2
3 -1/2Xa
(15. 16)
-1/2
4
X5
4 -1/2.X4
is a m.d. sequence, and S,l is the sum of column n. It is a tenn in a mmingale sequence even though this is not the sequence (5'n1. An adapted sequence
of fal-bounded (&,@s)=-x
variables
satisfying
Martingales
IFa)
ESn+L
233
k Sn a.s.
(15.17)
is called a submartingale, in which case Xn Sn &-l is a submalingale difference, having the property E(Xn+3lJ;n) 2 0 a.s. In the Doob decomposition of a submartingale, the predictable sequence An is non-decreasing. Reversing the although, since -Sn is a supermmingale inequality defines a supermartingale, extension. A supermartingale might submartingale, this is minor whenever Snis a a represent a gambler's worth when a sequence of bets is unfair because of a house percentage. The generic tenn semimartingale covers al1 the possibilities. =
-
15.5 Theorem Let 94.):R i.-,b R be continuous and convex. If (&,TnJis a martinthen (9(&),Fn) is a submmingale. If () is also nongale and Fl ((Sn) l < decreasing, (Y(&),8L ) is a submartingale if (&,8Ll a submartingale. =,
Proof For the martingale case,
ITn) f(#(.%+1)
#(F(&+1 I@n))=
a.s. #(u%)
(15.18)
by the conditional Jensen inequality (10.18).For the submartingale case, in (15.18)when xl S x2 = (1)(x1) < 9(.n).x becomes
4='
E2'
If (.X),TfITis a (sublmmingaledifference, (Zr,@rl7' any adapted sequence, and
sn Xx,z,-1, ,.1
(15.19)
=
then
(&,L )1' is Esn..t
a
since (sublmartingale
I@n)
=
XX,z,-l >1
+
ZnEXn+3I?L)
=
Sn
(
(15.20)
'n).
We might think of Xt as the random return on a stake of 1 unit in a sequence of bets, and the sequence (4) as representing a betting system, a rule based on information available at time t 1 for deciding how many units to bet in the next game. The implication of (15.20)is that, if the basic game (in which the same stake is bet every time) is fair, there is no betting system (basedon no more than information about past play) that can turn it into a game favouring the player or for that matter, a game favouring the house into a fair game. ,5,P), a stopping time For an increasing sequence (1,1 of c-subfields of is a random integer having the property f : t ) e 5t. The classic example is gambling which entails withdrawing policy from the game whenever a certain a conditiop depending only on the outcomes to date (suchas one's losses exceeding some limit, or a certain number of successive wins) is realized. If is the random variable defined as the first time the said condition is met in a sequence of bets, it is a stopping time. be a stopping time of f@n Let ), and consider -
-
..:(r4))
'r()
=
'r
'r
Snxz
=
Sn, n ;
%, n
':
> T
(15.21)
Theory of Stochastic Processes
234
where n zx T stands for minf
l,'rl.
fSnoysn
l'X'n.l
is called a stopped process.
then t,Sv,T,, 15.6 Theorem If fSn,5n 1Tis a martingale (submartingale), martingale (submartingale).
JTis
a
Proof Since f 5n 1Tis increasing, f ): k T()) ) e 5n for k < rl, and hence also -1 1(t=z) + n j (a
'r()))
=
n-1
FlSnvl
f TE
151171(1....,:) l +e'lx%1(n
&1
l
n-1
+FxL l < < Xflxsj =1
n k 1.
=,
(15.22)
lf (,L,Tn JTis a martingale then for A e 5n, applying (15.3),
jghnwbjxxdp =
=
jgcsqncgljdp Jxc,jsszlx'lzd' +
Sndp +
Xfafn<ecl
S4P
drnfnll
Snszdp,
=
A
(15.23)
showing that (&+1,u,@n)T is a martingale. The submmingale case follows easily on replacing the second equality by the required inequality in (15.23).* The general conclusion is that a gambler cannot alter the basic fainzess characteristics of a game, whatever gambling policy (bettingsystem plus stopping rulel he or she selects. Al1 these concepts have a natural extension to random vectors. An adapted sequence (A),@r)=-xis detined to be a vector rnzzrffrlgtzfe dterence if and only if (A'X,F,J*-xis a scalar m.d. sequence for a11conformable tixedvectors A # 0. It has the property
f(X+1I5tj
=
(15.24)
0.
The one thing to remember is that a vector martingale difference is not the same thing as a vector of martingale differences. A simple counter-example is the twois an element vector Xt = XoXt-q)'s where Xt is a m.d.; fLX, + V2.Y;-1,@,J adapted sequence, but EhLxt
+
L4-j J?h-1)
=
l2A-j
#
0,
so it is not a m.d.. On the other hand,
f(l-V+l + klxt ITr-l) but (:14+1+ hzxt, ?&) is not adapted, since
=
ml
0, is not Tsmeasurable.
Martingales
235
15.3 Martingale Convergence Applying 15.5 to the case ()4.) l IP and taking unconditional that every martingale or submartingale has the property =
FI5'n+11: 2
.
F1 snlP, p
expectations shows
1.
(15.25)
x, either to a By 2.11 the sequence of pth absolute moments converges as n uniformly where +x. the limit In bounded, Ll-norms finite the are or to case (sublmartingales also exhibit a substantially stronger property; they converge, almost surely, to some point which is random in the sense of having a distribution over realizations, but does not change from one time period t to the next. The intuition is reasonably transparent. (Tnl is an increasing sequence of c-fields which converges to a limit Tx c ;, the c-field that contains nnfor every n. Since Esn ITn) Sn, the convergence of the sequence (Tn) implies that of a uniformly bounded sequence with the property ESn+L I1n) k Sn, so long as these expectations remain well-defined in the limit. Thus, we have the following. --
=
15.7 Theorem If (&,;n)Tis a submartingale sequence and supnfl Sn l f M < x, then Sn S a.s. where S is a i-measurable random variable with FI5'1 S M. Ia --
The proof of 15.7, due to Doob, makes use of a result called the upcrossing inequality, which is proved as a preliminary lemma. Considering the path of a submartingale tluough time, an upcrossing of an interval (a,p1 is a succession of steps starting at or below G and terminating at or above p.To complete more than one upcrossing, there must be one and only one intervening downcrossing, so downcrossings do not require separate consideration. Fig. 15.1 shows two upcrossings of ga,pq, spanning the periods marked by dots on the abscissa.
O O
O O
O O
O O
Fig. 15.1 Let the r.v. Fk be the indicator of gn upcrossing. and then, for k = 2,3,...,n,
To be precise, set
i'I
=
0,
236
Theory of Stochastic Processes
0 if either
1%
Fz-I
=
0, Sk-k
=
0, Sk-k K
a, or l'-1
>
=
1 if either Fz-j
=
1, Sk-j k
or Fk-1 = 1, Sk-j <
(x,
p, p.
(15.26)
The values Of l'k appear at the bottom of Fig. 15.1. Observe that an upcrossing begins the period after Sk falls to or below a, and ends at the tirst step thereafter where ;$is reached or exceeded. 1%is a function of Sk-k and an K-l-measurable random variable. The number of upcrossings of (a,j$)up to time n of the sequence (,s'a((l))17, to be variable. The denoted &a()), is an Fa-measurable random sequence ( 1&4)1Tis condition. satisfies but it following the monotone, by a submartin15.8 Upcrossing inequality The number of upcrossings of g(x,f$J gale (&,@aJTsatisfies Eun)
E lsn1+ Ia
f
l
j) tx
(15.27)
.
-
Proof Define Sn' maxlxa,Gl, a continuous, convex, non-decreasing function of Sn, such that fSn',snj is an adapted sequence and also a submartingale. Un is the set of upcrossings up to n for (&') as well as for (&). Write =
n
Sn 5''l -
=
NXYj X Fkyj X (1 a2
n
X (1 *=2
+
-
J%)xL/
(15.28)
,
&2
&2
where l'k is f'rom (15.26),and E
11
& =
XLis
a submmingale
difference. Then
l
-
i'1)xI
=
X jyg-oixLap k=2
=X a2 jl
ExLbsk-bldp. e,
1*k=01
(15.29)
using the definition of a conditional expectation in the second equality (recalling that i'k is g-j-measurable), and the submartingale property, to give the inequality. We have therefore shown that Esn'
,$() -
l'kAk' X &2
k E
(15.30)
.
X2=21%Xis the sum of the steps made during upcrossings, by definition of J%. Since the sum of the XLover an upcrossing equals at least ;$ G by detinition, we must have -
M
X l%-YI(p
-
a)&s,
(15.31)
&2
where Un is the number of upcrossings completed by time n. Taking the expectation of (15.31)and substituting (15.30),we obtain, as required,
Martingales
(;3 -
a)F(&n) f Eu
5') f Esn'
-
-
237
a)
-
Ifs-ozjsdp
-
E 1Sn I + Ia
< FIsn a 1 < -
(15.32)
I .
w
The upcrossing inequality contains the implication that, if the sequence is uniformly bounded in f.l, the expected number of upcrossings is finite, even as n .,..4x. This is the heart of the convergence proof, for it means that the sequence has to be settling down somewhere beyond a certain point.
f3>
Proof of 15.7 Fix (x and
a. By 15.8,
e'1snI + laI
EUn) <
j3 a
M+ l(x I
S
f3 a
<
(15.33)
x.
-
-
is a positive, non-decreasing sequence and either diverges For ) e (, (!.&((l))J:' x. +c,o Divergence for (t) e C with to or converges to a finite limit U(@ as n whichcontradicts #(C) > 0 would imply EUn) (15.33),so Un-.,-qUa.s., where --
--x,
fl&)
<
=.
and s((1)) R(t)l limsupn-oxxS
liminfu-yxs'n. lf s()) < (x < I < R)), the x, interval (a,Iqis crossed an infinite number of times as n so it must be the < < < pair is for (x a,p. Hence considef any I X) 0. This tnle case that #(j' Define
=
=
-,..:
=
lt0;srtl
<
iltllll
Uls' f
=
tx
;$f X1,
<
a,fs where the union on the right is taken over rational values of (x and #(j' < X) 0 by 3.6(ii), which is the same as j' S S a.s., where of fSn ) Finally, note that =
=
=
(15.34)
0. Evidently,
,5'
is the limit
.
FI5:1 S
liminf E Ivn
If
sup
'l-/oo
FIsnI <
(15.35)
M,
n
where the first inequality is from Fatou's lemma and the last is by assumption. This completes the proof. . Of the examples quoted eaflier, 15.1 does not satisfy the conditions of 15.7. A random walk does not converge, but wanders forever with a variance that is an increasing function of time. But in 15.2, Xt is of course converging to Z. 15.9 Corollary Let fSu,5n l=-xbe a doubly infinite mmingale. where S-x is an Al-bounded nv. as n
Then Sn
--->
S-= a.s.
-->
-x,
Proof
Let U-n denote ihe umber of upcrossings of (a,j31performed by the The argument of 15.8 shows that k j 2 f'j, -1
-nl.
sequence
EU-i,4 ;
FIsj I + Ia I pra
,
all n
1.
Arguments precisely ana 1og ous to those of 15.7 show that
(15.36)
Theoty of Stochastic Processes
238
# liminf Sn < limsup Sn
0,
=
(15.37)
n'-i-co
N-)-x
1
- is non-nega tive nonsequence (FI5k1 1-x so that the limit 5'-x exists a.s. cxo by a nd decreases < <Sdefinition of a martinby increasingas n (15.25), FI l gale. Hence FI5'-.I < x. . If a martingale does not converge, it must not be thought of as converging in F, 'rhe
,
of heading off to +x or never to return. This is an event that occurs only probability the increments having a suitably bounded distribuwith 0. Subject to tion, a nonconvergent mmingale eventually visits al1 regions of the real line, almost surely. -x,
suptlxf
15.10 Theorem Let (Xf,Ffl be a m.d. sequence with
1J=1kt.lf
C
(:
=
E
then P(E
-
C)
=
Snk
((:
=
converges
either
), and
infnxst)
>
or supnxst)
-x
I) <
<
cxa,
and 1etSn =
xl,
0.
Proof For a constant M > 0, define the stopping time Twz((o) as the smallest integer otherwise. The stopped process n such that Sn > M, if one exists, and IA(t) tx,vhn-tlxxu Tnwwl-i-rl is a martingale (15.6),and S M for al1 n. Letting fsnxzu, = S'Lzu maxf 5kZ.,OJ, =
s+nh'w
<
s+( n- 1),xw+
X'bnxzu< M+ sup
I-LI
(15.38)
.
n
= Since Ezp 0, Fj Swwl = zft5Xwl,and hence supne'l skwj < x, and Snvsxuconverges a.s., by 15.7. And since Snvsxufk Snk on the set f : supnxst) S MJ, converges a.s. on the same set. Letting M---k x, and then applying the we obtain the conclusion that Snlk converges a.s. on the same argument to that is, #(Cn E; set AD= #(A-), from which the theorem follows. . =
k%()
-%,
+x, infniat) = Since Note that Ec = ( :supnlnt) P(Cc t.p Ec), a direct consequence of the theorem is that Cc = #(m = 0, which is the claim made above. -x1
#(FC)
=
#((Cf'7F)6) ECDN where =
.
15.4 Convergence and the Conditional Variances lf (&)
is a square-integrable mmingale with diferences fAkl, E(Sln-1 + Xl + J,x S j Tu-j) k &2-j Esn 2j 5 n-1
)
=
-j
n
n n
,
1 is a submartingale.The Doob decomposition of the sequence of squares has and Su and Aa = E(X2n j5 n The the form Sn2 = Mn +A n where hhl'n X1n E(X1n j @n sequence fAn) is called the quadratic variation of f&) The following thorem reveal an intimatelinkbetween mmingaleconvergence and the summability of the conditional variances; the latter propel'ty implies the former almost surely, and AQj ?=1 E in particular, if '=' ( t 5 t-1 ) < x a.s. then Sn S a.s. -j),
=
-1).
-
.
-..4
Martingales
15.11Theorem
Let
C
then #(D
C)
=
a m.d. sequence, and Sn X:=l-V.lf xl : :7.1F(xl e @, I@,-j)() <
(X,,F,)Tbe D
-
239
=
=
0.
( f
:
=
sn
convergesl
e F,
Proof Fix M > 0, and define the stopping time ,1A#(l)) as the smallest value of n having the property Xe'(A
I@,-l)(to)
>1
(15.39)
M.
If there is no finite integer with this property then ,:A#)) = x. If DM ( : 'rA/.((l)) x), D limv-yxfk. The r.v. 1(wka)() is Fs-l-measurable, since it is known at = time n 1 whether the inequality in (15.39)is true. Detine the stopped process =
=
-
N
A7A1(zvvf)
Snxxu.
=
(15.40)
,.=1 is a snvsxu
martingale by 15.6. The increments are orthogonal, and Esnhxu)
sup n
N
=
sup E n
XV ltwkul >1
n
=
supf n
77ltwknlftl
@r-1)
<
M,
(15.41)
>1
where the final inequality holds for the expectation since it holds for each by definition of 1M((t)). By Liapunov's inequality,
supFIskwl N
S sup n
(.t) e
f
< Ml Il&.wII2
and hence Snxzu converges a.s., by 15.7. If ) e DM, xskss/)) xS4))for every n e EN,and hence Sn1 converges, except for ) tna set of zero measure. That is, PDu rn C) PDM4. The theorem follows on taking complements, and then letling =
=
M
-->
x.
.
15.12 Example To get an idea of what convergence entails, consider the case of
(Xf ) an i.i.d. sequence (compare15.1). Then Lxtlatjis a m.d. sequence for any Exlt @f-1) F(aYl) = c2, constant sequence (t7,)o f p ositive constants. Since I a which we assume finite, Sn Xlcjxlat is an a.s. convergent martingale whenever x jlat 2 < For example, at t would satisfy the requirement. a =
=
t
.1
(x,.
=
In the almost sure case of Theorem 15.11 (when#(C) #4D) 1), the summability of the conditional variances transfers to that of the ordinary varipnces, 2 Ftsupfxl) < x, the summability of the conditional variances cf2 EXt). Also w h en is almost equivalent to the summability of the Xt2 themse lves. These are consequences of the following pair of useful results. =
=
=
Theory of Stochastic Processes
240
15.13 Theorem Let lZ?) be a any non-negative stochastic sequence. (i) X*t=jEZt) < x if and only if :7 =1f(Zt1 7t-1) < cxa a.s. (ii) lf Asupzf) < x thn PD A E) 0, where =
D
=
E
=
( ): Z=f=qE(Zt197,-1)4) ( ): Z'';=lZr() < ) x
<
cxo
)
,
.
Proof (i) The first of the sums is the expected value of the second, so the if' part is immediate. Since F(ztl F,-1) is undetined unless Ezt) < x, we may assume &1=qE(Zt)< x for each finite n. These partial sums form a monotone series which either converges to a finite limit or diverges to +x. Suppose Z:=1F(Zrl Tf-1) IYf-1) 0 a.s. converges a.s, implying (bythe Cauchy criterion) that L1+=mn+3EZt &I*=mn+jEZt) Then by the monotoneconvergence theorem, so that as m A n by the same criterion k1=jEZt) k*t=jEZt) < x, as required. = (ii) Define the m.d. sequence Xt Zt F(Z,I1f-l), and 1et Sn = Z:=1m.Clearly supnskt) f k*t=1Z,((t)), and if the majorant side of this inequality is finite, Sn1l)) converges in almost every case, by 15.10. Given the definition of Xt, this 9f-1)()) < x. ln other words, PE D) 0. Now implies in turn that Z7=IF(Z/I apply the same argument to -Xt = E(Zt 111-1) Zt to show that the reverse implication holds almost surely, and PD F) = 0 also. . Konly
-->
-.+0
--)x.
--y
-
-
=
-
-
15.5 Martingale lnequalities Of the many interesting results that can be proved for mmingales, certain inequalities are essential tools of limit theory. Of particular importance are maximal inequalities, which place bounds on the extreme behaviour a sequence is capable of over a succession of steps. We prove two related results of this type. The tirst, a sophisticated cousin of the Markov inequality, was originally proved by Kolmogorov for the case where (Xr) is an independent sequence rather than a m.d., and in this fonn is known as Kolmogorov's inequality. 15.14 Theorem
Let
(&,Tn)Tbe
a martingale. For any p 2
P max lSkI > E I
E lsnj#
1Sk
Proof Define the events A1 A1 =
):
=
f
max #Sj
1u./<1
:
I>
l
kh'14)
IS
-
EP
E,
(15.42)
.
:1,
and for k
1skl) l >
e
=
2,...,n,
e 5k.
The collection A1,...,An is disjoint, and UWz =l
=
max l51
lsfn
>
e,
.
(15.43)
Martingales
c l I51
Since A1
241
el, the Markov inequality (9.10)gives
>
I IP1z:).
-pe'
PA
(15.44)
'k
<
P
By 15.5, ISn I for p 1 is a submartingale, so k f n. Since At e 5k, it follows that '%lP1Ak)
f(1
I&IP
ISnIP I1k)1xJ F( ISn
I EE
=
f
IP1zk),
lz4k where the equality applies. (10.18).Noting X2=1 required, (15.43)415.45), as
=
n
P max 1
I
.%1
>
s
for 1 <
(15.45)
1U2-lxk, we obtain from
n
X#(All
=
F(I&IPI5k4 a.s.,
<
=1
IsnIP1z:) Z-PTEC k=l
N
=
lzu PT k=1
Z-PE ISn I
1sL1#).w Z-PEC
f
(15.46)
The second result converts the probability bound of 15.14 into a moment inequal-
ity.
(&,FnJTbe
15.15 Doob's inequality Let
e'
p
Isklp s
max lffn
-
a martingale. For p p
1
s Isn I
p
lus;n
1
,%l
>
f E-1F( ISnIltmaxlsksalslo
s
(15.47)
.
(15.46)for the
Proof Consider the penultimate member of P max
>
case p
c)),
=
1, that is,
(15.48)
and apply the following ingenioud lemma involving integration by parts. 15.16 Lemma Let X and F be non-negative nv.s. If #(X > 1)1PF(F#), for p > 1. all E > 0, then F(XP) f lp
E)
S E-1F(F1(x>c)) for
-
Proof Letting Fx denote the marginal c.d.f. of X, and integrating by parts, using #(1 Fx) -#Fx and RP(1 Fx@))17 0, -
=
=
-
Exp
j,
-jhpdk
'jpdFxv
=
=
-
l
Xp(#-1(1
=
Fx(())
,
-
Fx6))X =
j*pLP-$PX o
>
j)#j.
(15.49)
> 1 when Define the function 1fx>j)@) (, and 0 otherwise. Letting Fxy denote the joint c.d.f. of X and F and substituting the assumption of the lemma into (15.49),we have =
Ex
upj-r-zstrlfxxklld
p)
0
.7
Theory of Stochastic Processes
242
(J(scsF1tx>()(I)JFx.r(I'F))X
pjo-ie-l
=
-
pj
- 11 =
jkp-zm
y
(R2)+
P
dpkvx,y)
0
jpz,-yx'--'dFkYx-ys
styxp-l;
(15.50)
.
-1
Here (R2)+ denotes the non-negative orthant of R2, or g0,x)x g0,x).The second equality is permitted by Tonelli's theorem, noting that the function Fxrt defines (R3)+. By Hlder's inequality, a c-finite product measure on ECYXP
-j ) K E ly (josl-lytxp;
Substituting into the majorant side of
(15.50)and
.
simplifying
To complete the proof of 15.15, apply the lemma to and i' putting x maxlsjss I I x .%1
,%1
=
=
gives the result. w
(15.48)to
yield
(15.47),
.
Because of the orthogonality of the differences, we have the interesting property of a mmingale l5'nlr that EzSll
=
E
XV
(15.51)
,
>1
where, with s'tl for the case p with the variance property extended
0, Xt St St-k. This lets us extend the last two inequalities 2, to link Ptmaxlsksa lSnl > :) and Ftmaxlxlsa) directly of the increments. It would be most useful if this type of to other values of p, in particular for p e (0,2). One approach to this problem is th von Bahr-Essen inequality of j11.5. Obviously, 11.15 has a direct application to martingales. =
=
-
=
15.17 Theorem
lf
is a (aYf,7f1T
m.d. sequenc
and Sn
=
1=3Xt,
E lsnIP < 2:7 E lx,IP
(15.52)
>1
for 0
<
p f 2.
Proof This is by iterating 11.15 with F Xn, V Tn-la and X = as in the m.d. sequences just as argument leading to (11.65);note that the latter holds for kv-j,
=
for lndependentsequences.
=
.
Another route to this type of result is Burkholder's
inequality (Burkholder
1973). 15.18 Theorem
Let (&,Fn
ITbe a martingale
with increments Xt = St St-j, and -
Martingales 0. For 0 p, such that So
p f 1 there exist positive constants cp and Cp, depending only on
<
=
243
n
p
g
c,l X.xt
n
K E Isn1y <
>1
p A-,) g
c?,f Xj
.
(15.53)
n
/=
side, this extends by the cr inequality to
On the majorant
n
IP
FISn I
p
XF t
< Cp E
n
<
>1
lP,
c XFI Xt 1 p
0
<
p S 1,
(15.54)
>1
which differs from (15.52) only in the specitied constant. 18 I n fact
the
,
Burkholder inequality holds for p > 1 also, although we shall not need to use this becomes result and extending the proof is fairly laborious. Concavity of applied in Readers the that be convexity, so reverse. arguments have to may like to attempt this as an exercise. lemma. The proof employs the following non-probabilistic .?
mbers with yl > 0, and of non-negative < S 1, Then, for 0 p
15.19Lemma Let fyfJ be a sequence 1etFn )7=1y?for t 1 and J'tj 0. =
=
&
F$ < y
+
F--ly, PX >2
<
(1+BP4YL,
(15.55)
where Bp k 0 is a t'inite constant depending only on p. 1 this is trivial, with Bp Proof For p in a Taylor series of tirst order to get =
1'$ where 0n
G
F$-1 +P(l%-1
=
&
=
y;
+
0. Otherwise, expand F$
px
=
ot'nf-1yn,
(J'f-1 +
(L-1 A-yn? (15 56) .
difference equation in
(0,11.Solving the YP
+
=
(15.56)yields
tytf-yt.
(15.57)
>2
Defining Kt
we obtain
-1
#-1
F,-1
=
(F,-1 +
-
%y,)P
(15.58)
,
the result by showing that n
0
< pxKtyt >2
<
BpYL.
(15.59)
r r f The left-hand inequality is immediate. For the right-hand one, note that y fbr y > x > 0 and 0 < r < 1 (see(9.63)).Hence, (15.58)implies that @ -x
-x)r,
1 -.p
Kt K
1
Ff-1
It follows that
1 -
Fl-1 +
a
tyt
=
(Ff-1(Ff-1 + 0ly)
p-1
1.v
1-p
0f yt
.
(15.60)
Theory of Stochastic Processes
244
2,-2 l-p 0 S Ktyt < F,-1 yt
(15.61)
,
and hence n F-P#XA-rzJ n
n
< pv@tlYt-tlls-ps(.y
t /l'n)''
>2
>2
(A''71''-1)X1
-p'
= PT'''/ ?
'P
yt
r-a
E Bpn),
(15.62)
where y; = ytlj'n for t 1,...,n is a collection of non-negative numbers summing to 1, F; = Yxilyr,and Bpn) denotes the supremum of the indicated sum over all such collections, given p and n. =
The terms y;/F;-1 yt /1$-1 for f k 2 are finite since yl > 0. lf at most a number of finite the yt are positive the majorant side of (15.62)is certainly finite, so assume otherwise. Without loss of generality, by reinterpreting n if terms, we can also assume yt > 0 for every t. necessary as the number of nonzero 04?T-1) Ot-1) and applying 2.27 yields the result Bp n) Then, ytlYt-t and y; = ' /41) for a1l p e (0,1). Putting Bp supnfptn) < x completes the proof. w = =
'
=
,
=
Proof of 15.18 Put An Fn
: +
=
2 an d for E Z:=1X?,
=
>
0 and 8
0, set
>
n
Sln +
(: +An)
(1 6)(: +4u) +
=
+
2Xx%-1.V,
(15.63)
>1
2 + 25'f-1 Xt for t k 2, so that in the notation of 15.19, yt = 1', F,-1 (1+ 6).Yf 2 > 0 Then by the left-hand inequality of (15.55), with yl = (1+ 8)(e+X1) =
-
.
is
S
(1 6/18 +
+
?1
X21)P
+
(1
+
t
F(:lV 2pXJ't--ljx%-j.Y/. 6)#X >2
(15.64)
+
N2
we may allow it to approach 0. Taking expectations through, using the law of iterated expectations, and the facts that -1 i s decreasing in its argument, we obtain F(.&l 7/-1) 0 and that However, : is arbitrary in
(15.64),and
.?
=
E(%l
+
6:4.)# < E (1+ S E
(1+
DjPXIP
&fXILP
+
(1+
(:2,-1+ 6x4,-j)P-1V )pX >2 N
+
P XA(--j1F t (1+ 6)45/'-1
(15.65)
.
>2
2 an d yf Xlt for t 2 the right: + 4n, with yl = e:+ But if we put now l'n hand inequality of (15.55)yields (again,as the limiting case as E 1 0)
m
=
Xlp 1
=
,
Fl
+
#XA--lV< (j + Bg)E(AL). >2
(15.66)
Martingales
?
and since (1+ (1 +
6)8P-1(1
+
)P-1
(1+
f
245
this combines with
(15.65)to
give
B )F(A$) k E(Sln+ 8z4ay
2 2:-11(.E'1 l
lp
us-
#F(4j)j,
+
(15.67) .?
where the second inequality is by the concavity of the function
for p S 1.
yields
Rearrangement
2:
FIs I
6J'g21-P(1
(15.68) )/ 1jF(4j). which is the right-hand inequality in (15.54),where Cp is given by choosing 8 to minimize the expression on the majorant side of (15.68). In a similar manner, combining the right-hand inequality of (15.55)with L : + Sn2 w ith (15.65)and (15.67),and using concavity, yields n
S
+.Bp )(1+
-
=
lp
(1+ )(1+l,)FI 5'n1
k
(1+ &)E xll' +
zE
((1
Pxip
)
+
pTn5'l.%-llxj
+
,1
U'-
(1+
Dlpxn
(./-1 +
A,-1)P-1A-l
N2
N
Esn 2 + 6.4n )p
> which rearranges
2#-1(s1
lp
snI
#A'(z4j))
+
(15.69)
as lp
FIS 1
8#(21-P(1
n
+
6)41+#p
-
11-1F(Aj),
which is the left hand inequality of (15.54),with maximize the expression on the majorant side K .
o
1, Bp 0 identically in (15.55)and reproducing the known orthogonality property. Our final result is a so-called exponential inequality. bound fof martingale processes whose increments are accordipgly related directly to the bounding constants, For the case p
=
=
(15.70)
given by choosing cl
=
C1
=
to
1 for any
,
This gives a probability a.s. bounded, which is rather than to absolute
moments.
If (aYf,?;r)';'is a m.d. sequence with 15.20 of positive constants, and Sn Z)=lA, sequence 'heorem
IAI K Bt a.s.,
where
(#fl is a
=
#( IsnI >
:)
S
2exp(-:2/2(J=1#2f) j. u
(15.71)
This is due, in a slightly different form, to Azuma (1967),although the corresponding result for independent sequences is Hoeffding's inequality, (Hoeffding 1963). The chief interest of these results is the fact that the tail probabilities decline exponentially as 6: increases. To fix ideas, consider the case Bt B for This a 11t so that the probability bound becomes #( ISn I > e) f 2exp (-elllnBlt inequalitv is trivial when n is small. since of course P( ISn I > nBj = 0 bv con=
,
.
Theory of Stochastic Processes
246
struction. However, choosing e On 1?2 ) allows us to estimate the tail probabili-1/2 with quantity the associated Sn. The fact that these are becoming ties n exponential suggests an interesting connection with the central limit results to be studied in Chapter 24. =
(-#,,#,) satisfies
Proof of 15.20 By convexity, evel'y x e etM
for any a
>
<
(Bt +
ctfr+ (Bt
.x)d
-xle-l?t
(15.72)
lBt
0. Hence by the m.d. property,
ELemt I1,-1)
qlelt+ e-Bt)
f
S exptzlazsz,
j a.s.,
(15.73)
where the second inequality can be verified using the series expansion of the exponential function. Now employ a neat recursion of 10.10: Ee'nk
@n-I)
=
=
Ee-b-xn
I@n-1)
etxk-lFtetnj
fe osn
(.1
-lexp
(15.74)
@s-1) 2s2 n
)
as .
.
Generalizing this idea yields Een)
=
@,,-1)1,,,-2)... l11))
Eeq...Eefen,
S expfz1(#u2jF(F(...F(c%-l
s Combining
(15.75)with the
e
>
)
(@u-y...( @j))
.
generalized Markov inequality 9-11 gives
2E2.1#2, f exp ( -(y,E + yx1 j 0, which for the choice a = elX1=LBlt4 becomes Psn
for
2x:-1s2,
exp fac,l
Psn
> E)
> E)
<
(15.75)
zexpt-k?!l&1=jB1t) j.
The result follows on repeating the argument of and sumrning the two inequalities. *
(15.75)q15.76)in respect
(15.76) (15.77) of -Sn
A practical application of this sort of result is to team it with a truncation or uniform integrability argument, under which the probabilities of the bound B being j exceeded can also be suitably controll: ,
16 Mixingales
16. 1 Definition and Examples Martingale differences are sequences of a rather special kind. One-step-ahead unp redictability is not a feature we can always expect to encounter in observed time series. In this chapter we generalize to a concept of asymptotic unpredict-
ability.
16.1 Dellnition On a probability space L1,5,P), the sequ e of pairs (A,Tt)=-x, where (Tf) is an increasing sequence of c-subfields of 5 and th .Xt are integrable nv.s, is called an Lp-mixingale if, for p 2 1, there exist sequences of nonx, and negative constants (cfl*-x and ((,,,)7such that (,,, 0 as m -->
-->
IleAl 1,-,211, K c'l,,,
(16.1)
F(A1 T,+,2Il, < c,(,,,+1 117,-
(16.2)
hold for a11 t, and m k 0.
In
A martingale differepce is a mixingale having (,,, 0 for all m > 0. Indeed, tmixingale differences' might appear the more logical terminology, but for the fact that the counterpm of the martingale (i.e. the cumulation of a mixingale sequence) does not play any direct role in this theory. The present terminology, due to Donald McLeish who invented the concept, is standard. Many of the results of this chapter are basically due to McLeish, although his theorems are for the 2. case p mixingales form a very general class of stochastic Unlike martingales, processes; many of the processes for which limit theorems are known to hold can be characterized as mixingles, although supplementary conditions are generally needed. Note that mixingales re not adapted sequences, in general. Xt is not assumed to be Fsmeasurable, although if it is, (16.2)holds kivially for every m k 0. The mixingale property captures the idea that the sequence (Tsl contains progressively more information about Xt as s increases; in the remote past nothing is known according to (16.1),whreas in the remote future everything will eventually be known accordig to (16.2). The constants ct are scaling fatorj to make the choice of (,uscale-independent, and multiples of 11A11pwill often fulfil this role. As for mixing processes (see if (m O(m-%) for (? > (>. j14.1), we usually say that the sequence is of size However, the discussion following (14.6)also applies to this case. =
=
-(>
=
Theory of Stochastic Processes
248
16.2 Example Consider a linear process X
x jc-,77%i-./,
(16.3)
=
where ( I7,)+-Zis a fw-bounded mmingale difference sequence, with p k 1. Also 1et 5t ct&s, s S f). Then =
X
F(Al 5t-m4
=
77 jtb-j,
(16.4)
a.s.
j=m
Xt F(.Y,l 5t+m4
=
-
77e-jUt+js a.s.
(16.5)
j=m+L
w-bounded, the Minkowski inequality shows Assuming ( Us*-ooto be unformly that (16.1)and (16.2)are satisfied with ct sup,ll U,llp for every t, and (,,,= TJ l0j1 + Ie-j!). f-Y,,F,l is therefore a fz-mixingale if X; I0.j1+ l0-./1) 0 are absolutely summable. The as m x, and hence if the coefficients (%J=-x arises sided' process in which % 0 foj < 0 more commonly in the econometric modelling context. In this case Xt is gr-measurable and Xt Ext l5t+m4= 0 a.s., which may increase with t, and does not have to but we may set ct supysfll t&11p satisfy the definition. To prove Xt integrable, given be bounded in the limit to integrbility of the &,, requires the absolute summability of the coefficients, and in this sense, integrability is effectively sufficient for a linear process to be an f-l-mixingale a =
=r,,(
.-.-9
=m
fone-
--
=
-
=
We could say that mixingales are to mixing processes as martingale differences are to independent processes', in each case, a restriction on arbitrary dependence is replaced by a restriction on a simple type of dependence, predictability of the level of the process. Just as martingale differences need not be independent, so mixingales need not be mixing. However, application of 14-2 shows that a mixing zero-mean process is an adapted Ap-mixingale for some p k 1 with respect to the subfields 5t = c(A,Xf-t,...), provided it is bounded in the relevant norm. To be precise, themeandeviations of any fv-bounded sequence whichis a-mixing for r > 1, form an p-mixingale of size 1/r) for p satisfying of size application of 14.4 1 S p < r. If the process is also (-mixingof size tightens up the mixingale size. The mean deviations of a (-mixingfv-bounded 1/r) for 1 < p S r. The is an Lxmixingale of size sequence of size reader can supply suitable definitions of ct in each case. lt is interesting that the indicated mixingale size is lower (absolutely)than the mixing size, except only in the (-mixingsequence having finite sup-nonn (fv-boundedfor all r). Although these relative sizes could be an mefact of the inequalities wlzich can be proved, rather than the sharpest available, this is not an unreasonable result. If a sequence has so many outliers that it fails to possess higher-order moments, it would not be surprising to find that it can be predicted further into the future than a sequence with the same dependence structure but more restricted -:(1//7
-(?,
-
-(?,
-:(1
-(p
-
variations.
Mixingales
249
show the type of case arising in the sequel.
The next examples
16.3 Example An Ar-bounded, zero-mean adapted sequence is an Q-mixingaleof if either r > 2 and the sequence is a-mixing of size 2), or r k 2 size 1). u and it is (-mixingof size -r/(r-
-
-r/2(r
-
16.4 Example Consider for any j k 0 the adapted zero-mean fXtXt+.i c,,?+./,5twjl
sequence
-
,
and (A) is defined as in 16.3. By 14.1 this is a-mixing where ojt-vj Extxt-, (p-mixing) of the same size as Xt for finite j, and is fv/z-bounded, since =
Il.X)-)+./ rn 11
rII.X)+./ IIAII Ilr.
:<
by the Cauchy-schwal'tz inequality. Assuming r > 2 and applying 14.2, this is an in the a-mixing case. To get this result under g-mixing fal-mixingale of size 2), by 14.4, but such a sequence is also a-mixing also requires a size of 2) so there is no separate result for the (-m' of size case. I:a -1
-r/(r-
-r/(r-
'ng
Mixingales generalize naturally
from sequences to arrays.
is an L 16.5 Definitipn The integrable array ( (Xnt, @nfJ'7=-x)';=1 J for p k 1, there exists an anuy of non-negative constants (c,:,) negative sequence ((,,,)7such that (,,, 0 as m ..-A x, and -x,
-mixingale
if, and a non-
--+
IlFtAkrl@n,,-,,,)lIp < c,,,(,?,
(16.6)
IlAk A'(Akrl5n,t+m4 11,f cn,(,u+1
(16.7)
-
hold for
all f, n, and m
0.
(:1
The other details of the definition are as in 16.1. A1l the relevant results for mixingales can be proved for either the sequence or the array case, and the proofs generally differ by no more than the inclusion or exclusion of the extra subscript. Unless the changes are more fundamental than this, we generally discuss the sequence case, and leave the details of the array case to the reader. One word of caution. This is a low-level property adapted to the easy proof of convergence theorems, but it is not a useful constnlct at the level of time-series modelling. Although examples such as 16.4 can be exhibited, the mixingale property is not generally preserved under transfonnations, in the manner of 14.1 for example. Mixingales have to little structure to permit results of that sort. The mixingale concept is mainly useful in conjunction with either mixing assumptions, or approximation results of the kind to be studied in Chapter 17. There we will find that the mxingale property holds for processes for which quite general results on transformations are available.
16.2 Telescoping Sum Representations Mixingale theory is useful mainly because of an ingenious approximation method. involving a remainder which A sum of mixingales is nearly' a martingaleprocess,
Theory of Stochastic Processes
250
canbeneglectedasymptoticallyundervriousassumptionslimiungthedependence. For the sake of brevity, 1et Es stand for Ext ITx). Then note the simple identity, for any integrable random variable Xt and any m 1, Xt
X (f
=
t+kxt
Etn-bxtj
-
+
Et-m-lxt
Xt E ?+p1X).
+
W-m
Verify that each tenn on the right-hand side of signs, except for Xt. For any k, the sequence
(16.8)
-
(16.8)appears twice with
opposite
fEtwkxt F,+1-1X,, 5t+k17=1 -
is a mmingale difference, since Et-vk-kEt.vkxt Et+k-3Xt) 0 by the LIE. When l.&,S) is a mixingale, the remainder terms can be made negligible by taking m large enough. Observe that fEt+nXt, S+,,,I';=-xis a mmingale, and since supmfl Et.vnv'tkS FI.&l< x by 10.14, it converges a.s. both as m c.a and as m 11Fr-,,j.X)IIJ, 0 and -x, by 15.7 and 15.9, respectively. In view of the fact that )1aYfEt+mxtIIp 0, the respective a.s. limits must be 0 and Xt, and hence we are able to assert that =
-
--y
--y
.-+
--
-
X
Xt
(F,+kX,- Et+k-kxtlh,a.s.
=
(16.9)
P=-x
Letting Sn
:=1Xs we similarly have the decomposition
=
m
Sn
=
l
X k=-m
Fn
+
n
XEt-m-jxt
+
>1
Xt X r=1
-
(16.10)
Et+mxt)
where
lu
=
';--Iekxt
F,+,-1A),
-
( 16. 11)
,.-1
and the processes ( Fns a+k) are martingales for each k. By taking m large enough, for fixed n, the remainders can again be made as small as desired. The avantage of this approach is that martingale properties can be exploited in studying the convergence characteristics of sequences of the type Sn. Results of this type are elaborated in j16.3 and j16.4. If the sequence (A1 is stational'y, the constants lcfl can be set to 1 with no loss of generality. ln this case, a modified form of telescoping sum actually yields a representation of a partial sum of mixingales as a single martingale process, plus a remainder whose behaviour can be suitably controlled by limiting the dependence. 16.6 Theorem (afterHall and Heyde 1980: th. 5.4) Let (Xt,S) be a stationary Al-mixingale of size There exists the decomposition -1.
Xt
where
FlZjI <
x
and
=
B
( W$,T,) is
+
Zt
-
(16.12)
Zt+I,
a stationary m.d. sequence.
(:1
Mixingales
Sn
( Fn,gnl
251
(16.13)
Fn + Z1 Zn+1
=
-
is a mmingale.
Proof Start with the identify Xt
F,,s
=
+
Zmt -
zr,,,f+1,
(16.14)
where, for m k 1, m
X
lp,,,f =
Ext..s
Et-3Xt+s4+
-
EtXt+m+L
+A-m-1
-
(16. 15)
Et-tXt-m-3
J=-m
m
Zmt
=
%-Et-$Xt+s -A-,-1 +Et-3Xt-s-t4.
(16.16)
.=o As in (16.8),every term appears twice with different sign in (16.1 Xt. Consider the limiting cases of these random variables as m designated J'Pfand Zt respectively. By stationarity, E Iff-lA+sl
=
), except -->
x,
for to be
FIEt-s-txt I
and Fl Xt-s-j
Et-jxt-s-k
-
1
E IXt Fm-Y,l ;
=
-
hence, applying the triangle inequality, X
to
E Iz,l
S TE M
IEt-s-txt I +
77E IXt r.m
-
Etnxt
I
*
< 2X(,<
(16.17)
=.
M
Writing Jpr
=
Xt Zt + Zf+1, note that -
E IJFfI S
FI-Y,I+ 2F Iz,I <
(16.18)
=,
and it remains to show that J'P'fis a m.d. sequence. Applying 10.26(i) to (16.15), Et-jb'mt
(16.19)
Et-3Xt+m+3a.s.,
=
and stationarity and (16.1) imply that
FIEt-$Xt+m+3l FIEt-m-zxt I --A 0 0 also. Anticipating a result so that FI&-1B%l
(16.20)
=
x, from the theory of as m k e IN1 stochastic convergence (18.6),this means that every subsequence tr?h, e-lW,,,w),rl subsequence IN such 0 a.s. that I contains a further j e ) fmkq), x. for such subsequence, it is possible Since Jpf Wmkqj,t to every as j conclude that A'(WfITf-1) 0 a.s. This completes the proof. w .--h
-->
-->
.-+
-->
=
Theory of Stochastic Processes
252
The technical argument in the final paragraph of this proof can be better appreciated after studying Chapter 18. It is neither possible nor necessary in this 0 a.s. approach to assert that F(G,,vl @r-l) Note how taking conditional expectations of (16.12) yields -->
F(ml 72-1) = Zt 4+1 a.s.
(16.21)
-
It follows that <, is almost surely equal to the centred r.v. Xt Ext -
IF,-l).
16.7 Example Consider the linear process from 16.2, with ( Utj a stationary integrable sequence. Then Xt is stationary, and X
FI !71I 77 I%I <
Fl.Yl l f If the coefficients 00
*.
j=-oo
satisfy a stronger summability
X
X
condition, i.e.
X
< 77 X(( %( 10-/1) 77-10-1 X?nlo-,?,l
+
+
=
za=l
rn=1 j=m
then Xt is an Al-mixingale of size decomposition of (16.12)with
(16.22)
x,
?l=1
of terms we obtain the
By a rearrangement
-1.
X
B,,
77uj &,
=
(16.23)
jcz-x
and oo
Zt
=
o
x
X j=m X 0y Ut-t,t X 0-,. Utwm-q
zrl=l
where
E'IZ,I <
cxa
(16.24)
-
,
j=m
by (16.22).u
16.3 Maximal Inequalities As with martingales, maximal inequalities are central to applications of the mixingale concept in limit theory. The basic idea of these results is to extend Doob's inequality (15.15)by exploiting the representation as a telescoping sum of martingale differences. MacLeish's idea is to 1et m go to x in (16.10),and
accordingly write
X
Sn
(16.25)
Fnz, a.s.
=
#=-x
16.8 Lemma Suppose (xaly has the representation in (16.25).Let be a = collection real with of non-negative if l'ak 0 numbers, summable ak 0 a.s., and ak > 0 otherwise. For any p > 1,
tJklX-x
=
E max Lsjs n
IsjI# u
# -
#
1
x
77a k
b=-x
p-1
>-cf-1Fl
ak>o
ynzl#.
(16.26)
Mixingales
taloJx and positive real sequence
Proof For a real sequence X7=-xlkand note that OQ
X xk
p
x =
X
A''
P=-x
253
p
xklaaklp
=
x
X tz,'-#I.xkI#,
K A'''-'
k=-x
1et K llkloo-x,
(16.27)
#=-x
where the weights aklK sum to unity, and the inequality follows by the convexity of the power transformation (Jensen's inequality). Clearly, (16.27)remains true if the terms corresponding to zero xk are omitted from the sums, and for these cases set ak 0 without loss of generality. Put xk Fn, take the max over 1 S j < n, and then take expectations, to give =
=
E max L<j
To get
Isjbl'
j
<
x
77ak
p- I
tt.---
j
ak y-')
max j yg j
p
.gy
Doob's inequality on the right-hand
(16.26),apply
.
lsjsn
akg
side. w
This lemma yields the key step in the proof of the next theorem, a maximal inequality for Q-mixingales. This may not appear a very appealing result at first sight, but of course the interesting applications arise by judicious choice of the
sequence
lJkl
.
16.9 Theorem = E7.1-Y,, and
be an fw-mixingale, 1et Sn (Macleish 1975a: th. 1.6) Let (Xf,J;2l=-x let ltul7 be any summable sequence of positive reals. Then co
E
maX
ls/fn
f
f 8
co
($2+
F7
Jk
1
(2j)J
+
t-.=l
uk.1j)
(2gu-,1 2F-4 -
=
SJ 1<)
K4
(16.29)
.
put a-k ttzkl=-x,
=
ak for k
>
0. Then,
oo
Cr
E max
N c2j >1
k=l
Proof To get a doubly intinite sequence 2, applying 16.8 for the case p
n
X ak X tz-k1F(L2J .
(16.30)
V-x Since the termsmaking up lk are mmingale differences and pairwise uncorrelated, we have F(Fn2k) Ek-vkxtEt-vk-kxt) whichit follows that NOW,
-
Also let
%)EEt+kXt -Et+k-jXt)l.
,.-1
EEk-tk-vkxtk-vk-titqj
=
E Et-vkxt
=
l=-x
=
E(Elt+kXt Et+k-lxtL$l =
ztk x,- Et-vkxpand =
-
E(Elt+k-jXt) by the LIE, from
Elt+k-;Xt).
(16.32)
it is similarly easy to verify that
EEt+kXt Et+k-LXt41 F(z,,1-1 -
(16.31)
=
=
Ezlk-j 1,
=
z,1)2
-z2,J.
(16.33)
Theory of Stochastic Processes
254
Now apply Abel's partial summation <x)
X
a-x
n
c-k1A'(F2aJX
X a-kE(Et+kXt
=
>1
(2.25),to
get
Etwk-jXtll
-
k=-x
#j
*
<>3
=X X ,-1 t.-o
6l-kE(Elt-kXt Elt-k-jxt)
+
-
n
1F(z2,,:.j z2f:) Xt7-: k=l -
<x)
)q >1
=
formula
<x)
&0-1#(F2X t
t
)
+
TECEI
X)(J-k1
t-k
-
t
tz-k-lj)
k=1 X
J1 -3Ez1 10
+
)
+
y7F(z2J(u1 j t
-
u-11)
al
(16.g4)
where the second equality follows by substituting (16.32)for the cases k f 0, and (16.33) for the cases k > 0. (16.29)now follows, noting from (16.1) that EZlk) 2 X, ) < c2(2 S c2(2 E(Et-k , t k+1 x t k and from (16.2)that .
Pptting x
K
x
-1((2 (2) 1 Tak t'af
8
=
() +
tztl
u-jy.1jl 2y-q (2:(t7-:1
+
(16.35)
-
,
k=1
this result poses the question, whether there exists a summable sequence ftzkl such that K < x. There is no loss of generality in letting the sequence ((:); be monotone. If (,u 0 for m < x, then (v,+y0 for a1lj > 0, and in this case one 2 0,...,- + 1, and K reduces to m + 1)(%2 + (1). 1, k A jtem may choose ak consider where and the case natively, (k> 0 for evel'y k. If we put at ((), then define the recursion =
=
=
=
=
(z g((1+4Jk-1) z ak ztu-j
.(j
a
=
ak is real and positive if this is true of -j
ak
=
8
x
Xtzk
=
> -2
(, so that
(1 ak 2
(1)f
(16.37)
2J(),
we have
x
(2j)+2Xtu
< 16
k=1
1=0
ln this case, for k
and the relation
-2
-c:-1
J-o1((+
(16.36)
,
tu-1
-1
l 2 is satisfied for each k. Since aj- ((c +
K
j/2
x
2
Xak
.
(16.38)
k=o
0 we find -1
=
ak
-1
-z-1)Jk
-1
-2
f ak
-2
-J1-l
(16.39)
Mixingales m
m
X (-,2G (0-2+ X k=o
-
,-z-21
=1
(16.38),we
Substituting into
ak-l
255
)
am-l.
=
(16.40)
get
( j
X*X*(
K f 16
-
2
-
' /2 2
(16.41)
This result links the maximal inequality directly with the issue of the summability of the mixingale coefficients. In particular, we have the following
corollary. 16.10 Corollary
Let
(.X),S)be
an Q-mixingale of size
E maxu/J
K
Then
#Xd,
(16.42)
>1
1f.jf n
where K
-.
< x. -1/2-
-2
Proof If (1 Ok > 0, as the theorem imposes, then Z'l=1(k ) for1(-k2)-1/2 tp(--1-8 ) an d hence is summable Over m. ( and b 2 27 Om 2+2: q ) y The theorem follows by (16.41).w =
.
=
.
=
=
However, it should be noted that the condition m
x
.j/p,
X 77(-,2 is weaker than Ok -1/21/2+ for E > 0, so that k
(k
=
(z
--
X( -2 X(k..v2)(log =
(16.43)
< c.o
). Consider tjj e x
case for evel'y g > ()
k.- 2)2+2e<
(k qk+ g)-1/2(jog =
k + 2)-1-:
ejxen
.
(??;+2)2(log
-+2)2+26
(16.44)
,
and (16.43)follows by 2.31. One may therefore prefer to define the notion of in tenns of the summability condition (16.43),rather than by orders isize of magnitude in m. However, in apractical context assigning an order of magnitud to (,,,is a convenient way to bound the dependence, and we shall find in the sequel that these summability arguments are greatly simplified when the order-ofmagnitude calculus can be routinely applied. -'
=
Theorem 16.9 has noobvious generalizationgomtheo-mixingalecasetogeneral of the Lp for p > 1, as in 15.15, because (16.31)hinges on the uncorrelatedness exist in under consideration, second not the But because moments may terms. cases < p < 2 would be valuable. This is attainable by a result for 1 comparable a slightly different approach, although at the cost of raising the mixingale size in other words, the mixingale numbers will need to be summable. from to ..ul
-1;
16.11 Theorem Let tXr,?hl'l..,obe an fv-mixingale, 1
<
2, of size
-1,
and 1et Sn
Theory of Stochastic Processes
256 E max 1sjn
p
lsjI s
tx
F
P
4p c,
P
N
y7(z 77d,
j
t-.al
(16.45)
>1
where Cp is a positive constant.
Proof Let F,,kbe defined as in (16.11), and apply Burkholder's inequality and then Love's cr inequality with r = pI2 e (, 1) to obtain
FIFazlr f CpE
n
Et-vkxt &+k-1X,)2
X >1
(15.18)
p/2
-
n
S
CAXFI(Et+kXt
&+1-1.41F.
-
(16.46)
>1
Now we have the mixingale inequalities, lIF,+zX,
f IIF,+1AIl,+ < 2c,(z .&,+z--1.Y,I1?? lI.E',+z-1-'tlI,
-
fork
0 and
<
11F,+kA -
for k
(16.47)
z'kll, A',+,-1AlI,I4z,,z-1 S 2c,(1 Iz,k.II, 11p+ < IIz,,k-1 =
-
0, where Ztk is defined above
>
(16.48)
(16.33).Hence, n
e'Irnzl' s
Ec
2,cp(f
(16.49)
>1
(put (0 1), and substitution in (16.26),with (t7z)7'a positive sequence and = ak, gives =
E
mx
lijin
p
#
IS)Ip < y+1c?,
p-j
cx>
Efz t..o k
- 1
Both ak and ak1-#(( can be summa b1efor p conclusion follows. .
>
-ak
n
x
y'-qtzj-ptj N c;.
(16.50)
,...1
t-azo
1 only in the case
(z
=
Oa,
and the
A case of special importance is the linear process of 16.2. Here we can specialize 16.11 as follows:
16.12 Corollary (i) if Xt
Z7=-oo%Ut-),then
=
E mx Lsjsn
(ii) if Xt
For 1 < p S 2,
=
IsjI# S Cp Z7
p
x
IOeI +
j
77( I01( + al
l0-zI)
p
n sup F# (7,1#., s
then
zejh-p
E max I5'./i lsjs n
P
p
s c,
P
p
!
o
p
n
I0zl 77sup FI &,IP. >'') k.'.'zo >1
st
Mixingales Proof In this case, Et-kxt
Et-k-kxt
-
and a-k
constant sequence
l57
kut-k.
=
tJkl7
Letting
be any non-negative
ak,
=
7-7JI-PFI1u1,
I0:I#7'7cf, CvTakz-p
<
ap*
JS/
supsxtll rxllpin case the results. .
wherect supxll&x11pin case (i), and c, = I0kIand substituting in (16.26)yields =
(16.51)
>1
=
ak (ii). choosing
T) Recall that the mixingale coefficients in this case are (,, I0./1 + I0-./1), so linearity yields a dramatic relaxation of the conditions for the inequalities to be satisfied. Absolute summability of the % is sufficient. This corresponds simply to (,,, 0. A mixingale size of zero suffices. Moreover, there is no 2 yields a resuli separate result for faa-bounded linear processes. Putting # that is correspondingly superior in terms of mixingale size restrictions to 16.11. =m(
=
--)
=
16.4 Uniform Square-integrability One of the most important of McLeish's mixingale theorems is a further consequence of 16.9. lt is not a maximal inequality, but belongs to the same family of results and has a related application. The question at issue is the uniform integrability of the sequence of squared partial sums.
16.13 Theorem (fromMacLeish 1975b: lemma6.5,' 1977: lemma3.5) Let taYsTfl 1m, and :,2 7=!clwhere ct is detined in be an Q-mixingaleof size 1 Sn l n l l lf the sequence (Xtlct ) ,=1 is un iformly integrable, then so is the (16.1)M16.2). =
-z,
=
.
oo
sequence(
maxlxjspl/vz,,
jooawl .
Proof A preliminary step is to decompose Xt into three components. Choose posiB 1( jx,Issc,), an d then tive numbers B and m (to be specified below), 1et 1 t =
define
Ut Ft Zt
=
(16.52)
Xt Etwmxt+ Et-mxt
=
-
B
Et-vmxt1 t
=
-
E t-m
a
lf
(1653)
t t
.
B Et-mxt 1 1 t ) Et-mxt 1 1#) t -
-
(16.54)
-
,
such that Xt Ut + Ff+ Zt. This decomposition allows us to exploit the fotlowing collection of properties. (To verify these, use various results from Chgpter 10 on conditional expectations, and consider yhe cases k 2 m and k < m separately.) =
First,
E 2Et-kut Eut
-
Et-vkut)a
for k k 0, where kv m
=
1 E Et-wmjvk S
Ext
-
maxfkml.
=
EE 2 t-k F,
=
=
EEl
t-ksm
2 2
ctLkvm,
(jy
a a Et-vkwmlxa f c/((t,v??;)+1 ,
.
55;
(jy 56; .
Second, El X/ 1#) )# , 1* , , t-m -
,
(16.57)
Theory of Stochtic
258 E
Et+kYt)2
-
Processes
F(F2/+?,, X 1*f -F2 faxkz.?n )# t 1W) , t
=
(16.58)
,
=mintk,rrll.
Theterms areboth zero if kkm and are otherwise bounded Bct. Third,
whereko.1P,m by Exlt ) ;
F#
t-k
zt
EEl
=
E(z t F,+zz,)2 -
=
1 1()
:.)m4
t-
EEl
t+
-
,aftl
1t)
-
-
-
e'lt-ttvj
lt
-
A+(k=)m(1
)), 11/)),
-
where the tenns are zero for k 2 m and bounded by f(#l(1 F(AQf(1 11 that ))/U= ElxtlctjlLf jxycgjlsl) 0 as B the assumption of uniform integrabillty.
x
-
(16.60)
1Pf))
-
-->
-.-.:
(16.59) otherwise. Note uniformly in /, by
The inequality
(x'
2
x,)
>1
s
3
s,) 2 +
z,) 2j
g(x? (x? (x' ,-,) 2
+
>1
>1
(16.61)
>1
for 1 S j S n follows from substituting Xt = Ut + F, + Zt, multiplying applying the Cauchy-schwartz inequality. For brevity. write xj
S)?/v2 n,
=
uj = (Sr=1 '
A'./=
kn
=
Ullgnl,
?= 1
zj = (Yfn Then
out, and
YttllMnl,
lZf)2/v2. s
(16.61)is
equivalent to xj f 3(w+yj +p), for each j = 1,...,n. Also 1et similarly; then clearly, maxlssaay, and define n, y'k, and
.
f 3n
+
yh +
k).
(16.62)
For any r.v. Xk 0 and constant M > 0, introduce the notation &>(A') F(1(x>>).X), x. 0 as M As a so that the object of the proof is to show that supn&ion) of 9.29, and consequence (16.62) =
-.-/
'k
f 3&v/3(
s 6eqn)
+
yh +
-->
k)
+ &wK@-n)
+En)).
(16.63)
We now show that for any er> 0, each of the expectations on the right-hand side of (16.63) can be bounded by e by choosing M laree enough. First consider &;n); =L1/2giveia (16.55)and (16.56),and assuming (,a= Om ), we can app j y j: nto this k-1-9 -1-8 = for k > m. Applying (16.29) for k S m and ak case, setting ak m ' with >1 U t substituted for S) in that expression produces .
,
E(
n
)f
8
(r?z+1)/1-1-8
= O(m ),
+
X k-1-0
&-+1
=
(2m1++2 m X (2,/ 1=>a+1
(16.64)
Mixingales
259
where the order of magnitude in m follows from 2.27(iii). Evidently we can choose m large enough that ELn) < :. Henceforth, let m be fixed at this value.
but in view of (16.59)and (16.60)we A similar argument is applied to E(L4, 2 otherwise. Write, fonnally, EEt-kzt and < choose 1, 0,.. 0 k ak ak may2 y.m, ct(2:, Etnztjl 2 an d Ezt where < t7?(k =
=
=
-
maxlu,u,,F((X,/c,)21( Ixgc,I>s)),
(2
k
<
m,
kk
m,
(16.65)
=
and then application of (16.29)leads to
ELn)f
16(-
+
1) max Elxtlctll
1( 1xgoI>s)).
1< t < n
(16.66)
x, so let B be fixed at a value large enough that This term goes to zero as B Eqnj < E. where For the remaining term, notice that l'f Z'l'=-,u+1(x -->
=
Et-vkxtf B, Et+k-3y , jl ,.
k,1 =
(j6 6y)
-
.
For each k, (efl, @f+:)is a m.d. sequence. If 16.8 is applied for the case p = 4 and ak 1 for IFcl < m, 0 otherwise, we obtain (not forgetting that for yj > 0, =
2 (maxyyy) =
..2 Eyn,
maxytyy?
-
where Fn1
=
j
1
wf
v
J)
max l sj s n
'n
77r, 1
=
F(F4
<
given l',,z
k) + n- j
4F(F3.) n
,
?
1)
w v (2-+ V =
yjup ,
4
1 4 'n
r=
Z:=1(a.Now,
EYnk 4)
4
-'
Fa-j,1+
77E
n
+ 4F(F n
nk), ),4
kzz-m
(,,k,we
6F(F2.j
+
m
(16.68)
have the recursion
2up ,
-1
,
zt3,u)+
F(j4uJ.
(16.69)
(fkare bounded
absolutely by lBct; hence consider the terms on the right-hand side of (16.69).The second one vanishes, by the m.d. property. For the third one, we have The
2 2 f f'(l%-l,t4n2 and for
2
E-n-,lBcn)
2
f
4 2
(2f) Vn-lcn,
2
the fourth one, note that by the Cauchy-schwartz
3 j k (2#)4vn FI L-l,klnk
(16.70) inequality,
-1c3u.
Making these substitutions recursivelyyields Elvkl f n
into
(16.69) and
(16.71) solving
the implied inequality
n
&
N
>1
>2
>1
+ 4Xv,-1c) + (2#)4 6Xvl-1U Xd
4 f 11(2#) 4vn.
(16.72)
Theory of Stochastic Processes
260 Plugging this bound into .Y.k 0 and a > 0, yields
(16.68),and tinally 6
an ) f &Ml6 (.y
.2
uE- (.yn ) f
applying the inequality a &u(m 4642-
(s j
+
1)411(2s)4
s,
< EX2) for
(16.73)
.
By choice of M, this quantity can be made smaller than e. < 18E for large enough M, Thus, according to (16.63)we have shown that &AQ) or, equivalently,
b-)
-->
0 as M
x.
By assumption, the foregoing argument applies uniformly
complete. w
(16.74)
--)
in n, so the proof is
The array version of this result, which is effectively identical, is quoted for the record. and let Sn be an fu-mixingale array of size 16.14Corollary Let (X ,,@nrJ 2 X1=jXnt and vn by if 2 E'Jzzlcnt, is given where cnt (16.6)V16.7), fAkfyc2n, j is is uniformly integrable. uniformlyintegrable, (maxlsyunial/vnz1-=a.! =
-,
'1
=
Proof As for 16.13, after inserting the subscript n as required.
.
.
17 Near-Epoch Dependence
17.1 Definitions and Examples As noted in j14.3, the mixing concept has a serious drawbackfrom the viewpoint of applications in time-series modelling, in that a function of a mixing sequence (even an independent sequence) that depends on an intinite number of lags and/or leads of the sequence is not generally mixing. Let X
=
#f(...,16-l,1$,'P)+1,...),
(17.1)
where Js is a vector of mixing processes. The idea to be developed in tllis chapter is that although Xt may not be mixing, if it depends almost entirely on the epoch' of f F,) it will often have properties permitting the application of limit theorems, of which the mixingale property is the most important. This idea goes back to lbragimov (1962),and had been formalized in different ways by Billingsley (1968),McLeish (1975a),Bierens (1983),Gallant and White (1988), Andrews (1988),and Ptscher and Prucha (1991a),among others. The following definitions encompass and extend most existing ones. Consider first a definition for sequences. tnear
17.1 Dennition For a stochastic sequence l Ffl+-Z,possibly vector-valued, on a is an probability space (f1,@,#),let lj+-: c(J$-,j,...,J$+,,,), such that ( 8??f+-'''''');=m lf, for of integrable 0, of > c-tields. increasing sequence r.v.s a sequence p =
lA
lt7
satisfies
IIX, -
S vp,, F(.X)I ?7-'''''')II,,
(17.2)
' where vm -- 0, and flrl+-2is a sequence of positive constants, Xt will be near-epoch dependent in fv-/ztp?'?zl(JV-NED) on (FfJ+-7.I:l
be said to
Many results in this literature are proved for the case p 2 (Gallant and White, 1988,for example) and the term near-epoch dependence, withoutqualitkation, may be used in tlzis case. As for mixingales, thre is an extension to the array case. =
A'
possibly vector-valued, on 17.2 Definition For a stochastic array f( Fnfl17-x)';=1, a probability space (f1,F,#), let snt-mt c(Fn,,-,,,,...,F,,,r+,,,).lf an integrable -,,,
=
satisfies arrayt t.krll---xl7-l, Ilxn'F(-Yn,lT,$+7-,,,)11,K dntvm, -
0, and fV-NED bn ( Fnr) u
where
Mm
-->
.
fdritt
(17.3)
is an array of positive constants, it is said to be
Theory of Stochastic Processes
262
We discuss the sequence case below with the extensions to the array case being easily supplied when needed. The size terminology wlch has been defined foi mixing processes and mixingales is also applicable here. We will say that the O(m-% for (p > (pf). when Mm sequence or array is LP-NED of size According to the Minkowski and conditional modulus inequalities, -(p0
=
117,e-'L117'''''')11,S 11*-,wll,+ Il.E'(-'t w! /7:)11, <2IIX,- 81,,11,, -
-
-
(17.4)
wherew EXt). The role of the sequence (4) in (17.2)is usually to account g,,ll, is uniformly bounded, forthe possibility of tending moments, and when 11.1, should equal finite dt constant for all f. However, a expect to set to a we drawbackwith the definition is that (l can always be chosen in such a way that =
-
11-Y,Ext -
inf t
IF,'+-:)11,
dt
=
0,
for every m, so that the near-epoch dependence property can break down in the limitwithout violating (17.2).lndeed, (17.2)might not hold except with such a choiceof constants. In many applications this would represent an undesirable weakepingof the condition, which can be avoided by imposing the requirementdrs 2jlm- gll??, or for the array case, dnt S 2II-L,- p.rllp. Under tls restriction we can set v,a < 1 with no loss of generality.
Near-epoch dependence is not an alternative to a mixing assumption', it is a propertyof the mapping from (FfJto (,&J,not of the random variables themselves. The concept acquires importance when (Ffl is a mixing process, because then (m) inherits certain useful characteristics. Note that f;txfl 1j+-:) is a finite-lag, function of amixing process and hence is also mixing, by 14.1. Aorl/s-measurable mixing in the sense of Near-epoch dependence implies that (.') is a pproximatedby weshowbelow, amixingprocess. Andas anear-epochbeingwell dependentfunction of a mixing process, subject to suitable restrictions on the moments,can be a mixingale, so that the various inequalities of j16.2 can be exploitedin this case. From the point of view of applications, near-epoch dependence captures nicely the characteristics of a stable dynamic econometric model in which a dependent variableXt depends mainly on the recent histories of a collection of explanatory variablesor shock processes Fs which might be assumed to be mixing. The symmetricdependence on past and future embodied in the definition of a G-NED functionhas no obvious relevance to this case, but it is at worst a harmless generalization. ln fact, such cases do arise in various practical contexts, such as the application of two-sided seasonal adjustment procedures, or similar smoothing filters; since most published seasonally adjusted time series are the output of a two-sided tilter, none of these variables is strictly measurable without reference to future events. tapproximately'
17.3 Example Let (Jl+-7 be a zero-meqn, As-bounded scalar sequence. and define
Near Sp/ch Dependence
xt
263
X ejvt-j.
=
(17.5)
jc-x
Then, by the Minkowski inequality,
Il, 77 111,Ext I8/',+-''',?,) j=m+3
jvt-j
=
-
e,+-,?,'''Ff-./l +
-
e-jvt-v.ie7,,,':'14+./)) P -
f dtvm,
(17.6)
Jillp, all /. Clearly, v,,z 0 if where v,u 17,.+1(I%I + I0-jI), and dt 2sup,I1 if I0./1+ l 1= the sequence (%) is absolutely summable, and v,,i is of size /4-1-:) for (? > (p. ln the one-sided case with oy 0 foj < 0, we may put #f supysfll7,11p, which may be an increasing function of t; compare 16.2. In -->
=
=
-(p0
-j
=
=
The secondexample, suggestedby GallantandWhite (1988),illustrates hownearepoch dependence generalizes to a wide class of 1ag functions subject to a dynamic stability condition, analogous to the summability condition in the linear example.
17.4 Example Let t %'t)be a Zp-bounded stochastic sequence for p sequence (m) be generated by the nonlinear difference equation Xt
hvt, Xf-1),
=
2 and 1et a
(17.7)
where ((.,.) l is a sequence of differentiable functions satisfying
:/)(r,X sup a.x FA
b
<
1.
<
(17.8)
As a function of h is called a contraction mapping. Abstracting from the of problem, write vl as the dummy first argument of ft. By the stochastic aspect repeated substitution, we have .x,
h
=
=
J,(v,,,-1(v,-1.,-2(v,-2,...))) (17.9)
',(vf,v,-1,vf-2,...),
and, by the chain rule for differentiation of composite functions,
gt
h-j
' -j
f b
vt-j
(17. 10)
.
vt-j
Detine a Fj-m/f-measurable approximation 1ag exceeding m by zeros:
to gt by replacing the arguments with
:,(14,...,v,-r,,,0,0,...). p';'(v,,...,v,-,,,)
(17.11)
=
By a Taylor expansion about 0 with respect to vt-j for j
8t # t -
cw
,:1
m,
+
=
l=m.z.l
>
vt-g.
vt-j,
(17.12)
Theory of Stochastic Processes
264
denotes evaluation of the derivatives at points in the intervals (0,vr-j1. Now define the stochastic sequence (X,) by evaluating gt at (1$,J6-1,...). Note
where
*
that IIX, -
F(X,I1',+-:)112f
#'J'(7,,...,F'-,,,)II2
IIX, -
(17.13)
by 10.12. The Minkowski inequality, (17.12),and thep (17.10) further imply that X
lIA7 -
#:'IIa 77 Gt-.ivt-j =
.j=-+1
l
<jcm- jlGt-jvt-j j)2 X
b.i- llF/-jvt-./llc
<jzm1
s
1-#
bm sup IlF,-./vf-./lia,
(17.14)
j>m *
with where Gt-j is the random variable detined by evaluating Llvt-pgtj and Ft-j bears the corresponding respect to the random point vt-j,vt-j-L with constants relationship with lvt-pft-j. Xt is therefore fU-NED of size <: supssf lIF,F,Ia, Hlder's i f particular, inequality exists. In this nonn tf exist for r > 1, and (IF,IIar/r-1) allows us to make this derivation whenever 11 7,lI2r 14112 < x and Ft is a.s. bounded. n and also if 11 ,...),
-=,
17.2 Near-Epoch Dependence and Mixingales The usefulness of the near-epoch depencence concept is due largely to the next
theorem.
17.5 Theorem Let (X,)*-x be an fv-bounded zero-mean sequence, for r > 1. lf Xt is AP-NED of size (i) faet ( Ff) be a-mixing of size on ( F1) for 1 S p fXl,T-Jx) with size fv-mixingale is of constants dtl an f (>, < r aslp 1/r)) with constants ct t< maxl 11Xf ) llr,#f of size -J. If Xt is fV-NED of size (ii) Let ( $) be (hmixing, on ( Fr) for 1 S p f r with constants (J, (X,,F-'x) is an fw-mixingale of size -.b
-a.
-min
,
-
.
-.b
.-lnint//,
J(1
-
1/r)), with constants ct <( maxf llXfIlr,4).
F,J) where %,' c(Fx,...,F). Also, for m K Proof For brevity we write Est.) = F(. l = 1, let k (-/2), the largest integer not exceeding ml2. By the Minkowski =
inequality,
Ile'-f ljp< oo-mxt
and we bound
jlJ2x-''t.Yf -
Fj+-),V) Ilp+ IIF-'x-mEtkkxtj11,,
each of the right-hand-side
jlE-t
..-'''(.4 -
Etkkxt)
K tL'-t 11p
c: '''
terms. First,
)Xt
-
P)11/P Frf+-l.& I
(17. 15)
Near Fp/c
DepeWence
265
II-Y, VtkkAllp -
< dtvk,
(17.16)
using the conditional Jensen inequality and law of iterated expectations. Second, E7PX function of F,-1,...,Ff+s and hence mixing of -.k t is a tinite-lag measurable the same size as (FfJ for finite k. Hence for part (i), we have, from 14.2, 6al/P-l/rjjxf r < 1I, 6al/P-1/rIIA:+-lA-,II
11F-'--'''(F71k-Y,) <
combining(17.16)
(17.17)into (17.15)
and
r. 11
(17.17)
yields
f maxt II-Y,IIr,4l(,,,, llF-'ol'''XIlp
where(,u
=
(17.18)
+ vk. Also applying 10.28 gives 6a11/#-1/r ,
< ldts'm < 2 lI-, Fl+-'''''W,IIp 111,F-lx+'''-Yfllp -
-
K 2(,,,.
(17. 19)
atlp 1/r) ), part (i) of the theorem holds with ct <( since (,,,is of size T he proof of pal't (ii) is identical except that in place of (17.17) maxflI#,1lr,l. we must substitute, by 14.4, E-tqn-mEftkkx < 2911 A'j+-lkxflj< 2911-l/rjjx, jj . 11 (17.20) r I1p -minth,
-
-l''rjj
,.
Let us also state the following corollary for future reference. be an fv-bounded zero-mean array, r > 1. 17.6 Corollary Let f fxnrl+f for 1 f p < r with constants (t&,)on an an-ay (i) If Xnt is AP-NED of size f Fnrl which is a-mixing of size -J, then tXpv,Fj,-xl is an fw-mixingale of alqlp 1/r) J, with respect to constants cnt (< maxt IlAk,Ilr,t&,l size i If AP-NED of size for 1 f p < r with constants (J,SJon an array Xnt (ii) which is of size -J, then (-Lr,ij,-xl is an fv-mixingale of ( FSCJ t-mixing J(1 1/r)) with respect to constants cnt < maxl llAkfIIr,&,l size '==-xl';xl
-.b
-minty,
-
.
-.b
-minlh,
-
,
.
Proof Immediate on inserting n before the t subscript wherever required in the last proof. w The replacement of Ff by F,v and ;sl by 5ntsis basically a formality here, since none of our applications will make use of it. The role of the array notation will always be to indicate a transfnnation by a function of sample size, typically the normalization of the partial sums to zero mean and unit variance, and in these @fyfor al1 a,. cases 5js Reconsider the AR process of 14.7. As a special case of 17.3, it is clear that in that example Xt is CP-NED of size on Zt, an independent process, and hence of size for is a wmixingale every p > 0. There is no need to impose smoothness assumptions on the marginal distributions to obtain these properties, which will usually be a11 we need to apply limit theoryto the process. These results allow us to tine-tuneassumptions on the rates of mixing and nearepoch dependence to ensure specitic low-levelproperties whichare needed to prove convergence theorems. Among the most important of these is summability of the sequences of autocovariances. If for example we have a sum of terms, Sn Z:=1Xr, =
-x
-x,
=
Theory of Stochastic Processes
266
we should often like to know at what rate the variance of this sum grows with n. 0 with no loss of generality, Assuming Extj =
F()
n- 1
n
XA'(Ff)+
=
2X >1
>1
n
-t
TElxtupwml
,
m=1
a sum of n 2 terms. If the sequence (zY,l is uncorrelated only the n variances appear and. assuming unifonnly bounded moments, Esn)2 = On). s or genera j dependent processes, summability of the sequences of autocovariances (EXtXt-j), j e N ) implies that on a global scale the sequence behaves like an uncorrelated 2 = o (n). sequence, in the sense that, again, E(Sn) For future reference, here is the basic result on absolute summability. To fulfil subsequent requirements, this incorporates two easy generalizations. First we consider a pair Xt,Yt), which effectively permit.s a generalization to random vectors since any element of an autocovariance matrix can be considered. To deal with the leading case discussed above we simply set Ff = Xt. Second, we frame the imposingfv-boundedness resultinsuchawayas toaccommodateendingmoments, uniform fv-boundedness. It is also noted that, like previous results, this but not extends trivially to the array case. 17.7 Theorem Let fXf,)',) be a pair of sequences, each &&r-l)-NED of size W ith respect to constants fdxtvd't) for r > 2, on either (i) an a-mixing process of -1,
size and -s(r-2),
(r,
<<
or
(ii) a g-mixingprocess
of size
-rlr-
1), where
11r,11,. vlaenthe sequences
IExtYt+ms I Xl
J't+m
r
<<
IIx,II,
(17.21)
m e (N
'
r
t,v,
of are summable for each t. Also, if arrays lAk,l%,l are similarly f,s(r-1)-NED dnYt << 11 and rnrjlr, size with respect to constants ftfr,JJllwith dnt rt IIAk/IIr the sequences -1
IE-ktl'rn,t.vmjI Xnt
yn,l+pl
r
are summable for each n and t.
(17.22)
'
r
EI
Since r > 2, the constants appearing in (17.21) and (17.22) are smaller (absolutely) than the autocorrelations, and the latter need not converge at the same rate. But notice too that r/(r- 1) < 2, so it is always sufficient for the result if the functions are JU-NED. Proof As before, let Ests)
=
E.
ity,
lE-vmx
I f 1E-vmxt
The modulus and Hlder
1TJ),and -
let k
Y+-zAll l +
inequalities give
=
(2J.By
the triangle inequal-
If(1$+,x1+-1#,)l
.
(17.23)
Near Fp/c
IEb-vmxt X where v,,, is of size 1, and b
Dependence
e+-CA)) < I
267
IlrllA V+-l-'tIIr/(,-I) 1--,+,,, 11 l+,,, Ilgzlcv-l', f 11
-
Also, applying
-1.
(17.24)
-
17.5(i) with p
rllr
=
-
1), a
=
r/tr
-
2),
=
IE
i' t+mE7kx t) I -.k
=
1$+,?,F7kXJ 1 1Ftekt+-1 IIzE)f+-.',1$+,,, r/(r-
IIA1+-',A)r
11 r/tr- 1) 111,11 r 11
11 < E-t-+kYtwm
1)
< 11 :; II-')IIrcl'+,,,(It and (; is of size Combining
F,IIr,Jl'l where cr < maxt 11 and (17.24) (17.25)in (17.23)gives
-1.
(17.25) the inequalities in
ltv( + (() Il.f/llrcr+,,, 11 F,+,,,11ryJ, 1$+,,,IIrt,,,, f
IExtYtmmlI s
maxl
(17.26)
This completes the proof of (i). The proof of where (,,, (vJ+ (() is of size using with r/tr 1) and b 17.5(ii) is similar 1. (ii) a p -1.
=
=
=
=
-
For the array generalization, simply insert the n subscript after every random variable and scaling constant. The argument is identical except that 17.6 is applied in place of 17.5. .
17.3 Near-Epoch Dependence and Transformations Suppose that (Xjt,...,Xvt4' Xt g(...,Ft-1,FsFf+1 is a v-vector of LP-NED functions, and interest focuses on the scalar sequence ((aV)J, where (t: 1- F- R, RV, is a fv/f-measurable function. We may presutne that, uner certain condi1-q tions on the function, ($(#f) will be near-epoch dependent on fJSJif the elements of Xt are. This setup subsumes the important case v 1, in which the question at issue is the effect of nonlinear transformations on the NED property. The dependence of the functional form 9f(.)on t is only occasionally needed, but is worth ,...)
=
=
=
making explicit.
The first cases we look at are the sums and products of pairs of sequences, for which specialized results exist. Let Xt and Ft be LP-NED on F, is AP-NED of size
17.8 Theorem Then Xt +
-mintfpx,tpyl
Proof Minkowski's Xt + F,) 11
-
(J/IJof respective
sizes
-(px
and
-f?y.
.
inequality gives E'7'''''(m + F,) II,<
<
+ 11 112,F7''''''A)Il, FJ
-
dhx tm
+
d% tm
''
< dvmt
-
A7''''''1',11,
(17.27)
Theory of Stochtic
268
wheredt
maxf
=
+vp,F and v,n = vxm
/vt JFlr ,
Processes
tnt,?l-'nint'x,'rll
=
.
.
A variable that is Q-NED is AP-NED for 1 S p S q, by the Liapunov inequality, so there is no loss of generality in equating the orders of norm in this result. The same consideration applies to the next theorem.
17.9 Theorem Let Xt and l be Q-NED on (FrJof respective sizes
-x
and
-tpj..
-.rninttpsfpxl
Then, XtYt is AI-NED of size
.
Proof By rearrangement, and applications of the triangle and Cauchy-schwartz
inequalities, we have E IXtn
-
F7':(mF,)
j
j
+ (me1+-,1F, (F7;m)(M+-''''''J',)) = E xtyt x,l+-rlyfl -
-
-
Fj+-,1((-Y,eX,1m)( 1', A'7': F,)) -
-
IIe'7''',''l',II2II-Y,e'7lx)llz
+ l', 17,11lI2 < 111,11211 -
F, , 11 Ilxre7,,,'''.&II + f'',v,,,Ft/'/v,l -
-
zdkivmY +
ztl%- + ir,11 11 I
11 < 11.1,
dt
=
maxt
j
-
el+-,,,'''Frll ,
(17.28)
F,1128,,dtidxtl IIx,II2t//, 11 Vm
=
X F X 1'+ Mm Mm + MmTm
and gtyyj-rninltpxrli
=
.
g
ln both of the last results we would like to be qble to set 1$ = Xtwjfor some finite j. A slight moditication of the argument is required here. 17.10 Theorem If Xt is AP-NED on ( Wl so is Xtwjfor 0 ,
<
j
<
x.
Proof lf Xt is fP-NED, then Il-Y,w Ext-q -
I@f'+Xy+-'''''') 11, 11,< 2 11.X',+./EXt+) IT',++'t.,+-:) -
S dt
using 10.28,
where
#;X
2JXf
(17.29)
vm,
we can write Xt-vj E t--Yrm./ f d'txvm' 11 l S7,1) 11p =
+j.
(17.30)
-
,
where #'
Vr?j
and
Mm' is
of size
-(p
=
if vm is of size
v0
,
Tm-j, -..
m <
j
>
J
m
.
.
Putting the last two results together gives the following corollary.
17.11 Corollary If Xt and Ff are Q-NED of size
-gx
and
-(ps
XtYtwkis AI-NED of
Near Fp/c
size
DepeWence
269
-minttps:xl
n
.
By considering Zt
the AI-NED m-gwjlk-gwj,
=
/
V?u
numbers can be given here as
Vo,
m f
+ 1 W/21
Vr,l-(1/21-1,
m
>
(W21+ 1
=
assuming wherev,a dYtvm+v,u +vruvp,, an d the constants are 4#f-wmY+k-g1a, that dxt and are not smaller than the corresponding Lz nonns. =
X
F
F
X
Al1 these results extend to the array case as before, by simply including the extra subscript throughout. Corollary 17.11 should be compared with 17.7, and care taken not to confuse the two. ln the former we have k fixed and finite, whereas x. The two theorems naturally the latter result deals with the case as m each other in applying truncation complement arguments to infinite sums of products. Applications will arise in subsequent chapters. More general classes of function can be treated under an assumption of continuity, but in this case we can deal only with cases where 9f(A) is Q-NED. Let --+
RV
4)(x):-I' F-> R, 'T i be a function of
real variables,
1/
ptxl
,.'r
2
and use the taxicab metric on V
)
IA'f1
=
2
-xi
I
(17.31)
,
izu1
RV,
2 to measure the distance between points x 1 an d x We consider a set of results that impose restrictions of differing sevelity on the types of function allowed, but offer a trade-off with the severity of the moment restrictions. To begin with, impose the uniform Lipschitz condition, .
l(1),(.1 1) 9/xl ) I < -
B , pt.tl
xll a.s.,
,
(17.32)
where Bt is a tinite constant. 17.12 Theorem Let Xitbe AZ-NED of size on f F2) for i = 1,...,v, with constants with constants dit. If (17.32)holds, ((aV)) is also AZ-NED on f F2) of size multiple maxftljll. of a finite Proof Let 4),denote a Fj+-l-measurable approximation to /ml. Then -a
-c,
II4h(A)
-
Ak
< 119,(2,) 9,112 FJ-,,''>,(.X))Il2
+
(17.33)
-
by 10.12. Since ((j+-;;1A) is an Tjlrl-measurable random for this choice of ,, and by Minkowski's inequality,
variable,
F7''''''9,(A)112< #,Ilp(.2',,.E7;7,-Y,) 112 119,(1,) -
V
u Bt
2 11-'t,e,!,,,',,-'t,II -
i=1 V
< Bt
ditvim f=1
(17.33)holds
Theoty of Stochastic Processes
270
f dtvm, and v,,,
where dt vffmaxjltitl by assumption. x =
=
(17.34)
Y the latter sequence v -1Zjazlvfm,
being of size
-.a
lf we can assume only that the Xit are TV-NED on F, for some p e (1,2),this argument fails. There is a way to get the result, however, if the functions (t are bounded almost surely. 17.13 Theorem Let Xitbe JV-NED of size dit, i = 1,...,v. Suppose that, for each t,
I9,(.1l)
-
Proof
l#1, -
For brevity, write ( i 9, I / zMmintz, 1 ). Then
=
,7(9)
on
%).
2
-4,:2
,
)) is Q-NED
x.
9r(X
I
on ( Ff ), for 1 f p f 2, with constants (A) j f M < x a.s., and also that
(#l xl)
2
#f(-Y ) l f m inl Blp
Then ((# finite multiple of maxjttf)
where Bt <
-a
,
(Ftl
of size
and let
,
2M) a.s.,
z
=
(17.35)
-apl2,
with constants a
#,p(A'l .Y2)/2M so that ,
,
Jtzs,)(9l f'(z>1,(9l <(2.2 Jtzs,lzz' Jtzoll'l -tbldr
-tg)2'
+
-
<
+
f (234)2s(z,)
= where Bjt
BitEpx,xl)pl,
(17.36)
Bq/2(gAy)1-p/2 combining (17.33)with
=
.
(17.36),we
can write
F7:,(#2Il2s #1rlIp(--F7''''''.f,) 119/2 114,,().;,) -
< #l, < B3t
v =1
v
plz
Ilx .#7j+-,,j&'.lI?) -
p11
ditpim
f=1
(17.37)
< dtvm, W
hel'e
dt
=
#1,vP/2maxj(J(/,2j and vm
assumption.
=
(v-l t=
kvimlpll ,
which is of size
-aplz
by
.
An important ekample of this case (withv = 1) is the tnmcation of Xt, although this must be defined as a continuous transformation.
17.14 Example For M
>
0 let
Near Fp/c
Dependence
.x, IxI f @)
=
M
M, x
>
M
-M, x
<
-M
(17.38)
xl x1( Ixls.v) +M(x/I
)1(I.vI>v) In this 2 l9(-Y1) #(-Y ) I < jxl x2j 1, and 17.13 can be used to show that I()IXJJis The more conventional truncation,
or, equivalently,
4,(x) =
.
case
-
-
,
so set B
G-NED.
=
Xflf IxlsM)
Q-NED if (X,) is
Xt, IXt I S M
=
(17.39)
otherwise,
0,
cannot be shown to be near-epoch dependent by this approach, because of the lack of continuity. n rther variation on 17.12 is to relax the Lipschitz condition (17.32)by letting the scale factor B be a possibly unbounded function of the random vari-
A
ables. Assume
I9,(.Y1) 9r(xl ) j -
where, for each
< B, (x1 x2)p(x1x2) a s., ,
,
(17.40)
.
f,
Btx
1xl ):
'!r
,
x'r
(17.41)
A/s-measurable function. To deal with this case requires a is a non-negative, B lemma due to Gallant and White (1988).
17.15Lemma Let B and p be non-negative < x, and Ilfpllr< x, for q k 1, and r >
Illpll2S
r.v.s and assume 2. Then
< Ilpllq 11#11#(t?-1) c'o,
2(IIpIIJ-2II#II-/1-l)Il#pII;)'/2Cr-1).
(17.42)
Proof Detine
c (IIpIlII#II/(-l)II#pII-rr)''f1-r), -
and let #1
=
Then by the Minkowski inequality,
ltapscls.
< II#1pII2+ -#1)pII2. II#pII2 11(.:
The right-hand-side
(17.43) ( 17.44)
ter'ms are bounded by the same quantity. First, 1/2
II#lpII2 =
Jspsctfplof'
jBp < c1?2
1/2
dp
/ c''2(IIpIIII#ll/f:-1))''2,
(17.45)
F/ldtpry of Stochastic Processes
272 applying the Hlder
inequality. Second,
11(/
Allplla
-
1/2
jsvwcBtldp
=
cl
<
1/2
j
-rll
(splup
Boc
Ij;/2
< t)2
'-r/zljlp
(17.46)
where the first inequality follows from r > 2 and #p/C > l Substituting for C in (17.45) and (17.46)and applying (17.44)yields the result. . .
The general result is then as follows.
17.16 Theorem Let l#f ) be a v-dimensioned random sequence, of which each element is Q-NED of size on (FCJ,and suppose thatY/m)is fvbounded. Suppose further that for 1 s q S 2, -a
< x, I1p(.X',,F7;Jl.fJllt?
< x, lI#,(-Y,,e-7:A',)IIc&c-1)
and for
>
r
2, < oo. Il#?(#/,%+--M#,)p(#f,Fj+->,)IIr
Thn
((f(A)
l
is JU-NED on
( Ff)
of size
-alr
-
2)/2(r
-
1),
Proof For ease of notation, write p for p(.X',,F7:#Jand B for BtxtnE7-zXtl. As in the previous two theorems, the basic inequality (17.33)is applied, but now we have
11(-)) e'7Jr9(A-f)l12< -
114,(.X',)#(F77X))Il2 -
llfpll2 (17.47) s 2IlpII:Cr-2)/2(r-1)Ij#jj4(;-(:21(2)(r-1)jjspjjrr/2(r-1) S
where the last step is by 17.15. For q f 2, V
Ilpllq<
< lIpIl2
V
77 f=1
dtvm7-ltrvf,,, f=1
-.E1+-,,,',,.X112 11.1/, s
=
and Mm v -1E( v im, which is of size Hence, under the stated assumptions,
where dt
=
v maxjltl)
=
.1
< div,?,tr-:D/zr-ll lIf(-'t)-.E:+-.p,''',(-Y,)lI2
7
-a
(17.48) by assumption.
(17.49)
jlrr/ztr-ll(y #J II#llJ)-(22(2)(r-1)jl#p s where =
t.
Observe the important role of 17.15 in tightening this result. Without it, the best we could do in (17.47)would be to apply Hlder's inequality directly, to obtain
Near fp/ch
273
Depeeence
< IIpIIuII#II2w(?-l), 1. lI#pII2 q
(17.50)
The minimum requirement for this inequality to be useful is that B is bounded almost surely pennitting the choice q = 1, which is merely the case covered by 17.12 with the constant scale factors set to ess sup Bt(X 1 ). The following application of this theorem may be conasted with 17.9. The moment conditions have to be strengthened by a factor of at least 2 to ensure that the product of Q-NED functions is also Q-NED, rather thanjust LI-NED. There is also a penalty in terms of the AZ-NED size which does not occur in the other case. .v2
,
Letav < x and 11 XtYt.Assume that lI#,II2r Xt,Yt)and ()tffl 1$II2r 2, and that Xt and Ff are ZZ-NED on ( Fll of size -J. Then
17.17Example <
x
for r
>
-'1F)
l
-
=
-Yl1'l K I
=
-)
I II l'l I + I K ( Ixl I + IFl I)(IFl
.11
.1)
-
-
-
IlFl l'1 + I-Y) l -
A
I)
#(A1,A)p(#l,.f),
(17.51)
=
defining B and p. For any q in the range (4/3,41,the assumptions imply K 1111 < II#(#),-Yl)Ilw(?-1) FllIw(?-1) II/(-1)+ 11 x,
11.X1 Il-Ylll < < IIl'l1I?+ Ilp(.Cl,-Yl)Il? IIl'lIl?+ Il?+
=,
(17.52) (17.53)
and 1') :K 11.11112r11 II#(-Yl,-Yl)p(-Y),-Yl)IIr Il2r+II.X1II2rIlFlll2r+II.X1lIlr lIFlII2rII FlI12r+ IIFlIllr +IIx)lI2rIlA:II2r+ IIFlII2rIIAIl2r +IIFlII2rII.X1II2r+
(17.54)
Putting -Y1
Xt and .Y1 E7-;,Xt, the conditions of 17.16 are satisfied for q in the range (4/3,21and Xtj is Q-NED of size -J(r 2)/2(r- 1). n =
=
-
17.4 Approximability ln j17.2 we showed that N-NED functions of mixing processes were mixingales, and most of the subsequent applications will exploit this fact. Another way to look at the JV-NED property is in tenns of the existence of a finite lag approximation to the process. The conditional mean E(Xt IJ;j+-/l)can be thought of as a function of the variables 6-m,...,Vt+m, and if (Ff) is a mixing sequence so is 1F(A! Fj+-'QI, by 14.1. One approach to proving limit theorems is to team a limit theorem for mixing processes with a proof that the difference between the actual sequence and its approximating sequence can be neglected. This is an alternative way to overcome the problem that 1ag funtions of mixing processes need not be mixing.
Iheory of Stochastic Processes But once this idea occurs to us, it is clear that the conditional mean might not be tlle only function to possess the desired approximability property. More generally, we might introduce a definition of the following sort. Letting 1$ be l x 1, l(2m+1) = F-y R as a gyr-yvlys-measurable function, where st*m t-m t-m we shall think of /C:R c(F,-?,,,...,F,+,,;). Dennition The sequenee fm) will be called (p 0) on the u-approximable 17.18 there seq'u of exists if for each Jncef/li'l s?i/fx'-'/'/-measurable m s s a f v,) sequence variables, and random >
II-Y,-/ClI,S dtvm,
(17.55)
c'o. (X,) will where fdtI is a non-negative constant sequence, and vm--A 0 as m approximable in probability (or mapproximable) on fFf ) if also be said to be > 0, there exist (T), (t$), and (vpr,las above such that, for every --)
#(IA
'71
-
>
dtn) f vm. u
(17.56)
The usual size tenninology can be applied here. There is also the usual extension to arrays, by inclusion of the additional subscript wherever appropriate. If a sequence is fv-approximable for p > 0, then by the Markov inequality
#( lXt 8-J'vj,;
where s'm'=
-
h'l I > dtn) < ()-PlI-Y,
hence an
-
< vm' /1'J'11p#
(17.57)
,
p-approximable
An process is also Q-approximable. although only in the casep 2 are we able to
AP-NED sequence is fp-approximable, is the best w-approximator in the sense that claim (from10.12) that Ext 11j+-'''m) the p-nonns in (17.55) are smaller than for any alternative choice of hl'. =
17.19 Example Consider the linear process of 17.3. The function
h';l X =
(17.58)
jvt-j
j=-m
unless @,'+-':) isdifferent from A'(#,I for xt since zw-approximator
112,hmt11, 77 jvt-j + jzzm1 + joy1 l l) and dt 7=,u+1( -
where vm
=
(J/',) is an independent process, but is also an -jb-vjt
=
-j
> =
< dts'm,
(17.59)
sup,ll 7,IIp. o
17.20 Example In 17.4, the functions g are fv-approximators for Xt, of intinite < x. n size, whenever sup,sf IIF,F,IIp One reason why approximability might have advantages over the AP-NED property is the ease of handling transfonnations. As we found above, transferring the JV-NED property to transformations of the original functions can present difficulties, and impose undesirable moment restrictions. With approximability, these difficulties can be largely overcome. The first step is to show that, subject to Zr-boundedness, a sequence that is approximable in probability is also fw-approximable forr < r, and moreover, that the approximator functions can be bounded for
Near Epoch Depcndence
each finite m. The following is adapted from Ptscher
and Prucha
(1991a).
17.21Theorem Suppose fXf) isfv-bounded, r> 1, andfvapproximablebyT. Then for 0 < p < r it is fv-approximable by J?;' /CI ( Ihmtj
.
Proof Since hmtis an fm-approximator of Xt, we may choose a positive sequence ,u) x. Also such that 8,u 0 and yet #( IXt h'l I > dtnm) f Mm --+ 0 as m f butlfmv,u choose a sequence of numbers (A&) having theproperties Mm -1/2 = For example, Mm vm would serve. T here is no loss of generality in assuming supm'/k-1 ; 1 B y Minkowski's inequality we are able to write -->
--
-
.-+0.
--:x,
.
IlA-/7IIpf
where
(17.60)
41,,,,II(A- 1:,)1(Ix,-pI>.,.u) 11p, =
A1t
=
:7''')1f Ix,-r;'I:.f,A<,,,) t 11(x,-
43
=
(I(xt
m
t l11
Hlder's
A)r,,+z4lm+Al,?,,
llp, 11p.
-''')1(
-
f
Ix,-'krllxtfj:m,shrylstjAfm)
inequality implies that 11A'1'11, S
> 1. II#II,qIIl'II?v&-l),
chooseq r/# and apply (17.61)to Amitfor i dtMm,again by Minkowski's inequality, IImIIr+ thefollowing inequalities. First, =
=
(17.61)
1,2,3. Noting that 112,/')'IIr< lsllfx= #(F), we obtain and that 11 -
/ Il-Y, l'7IIr#(I-Y, /'71 > dtnm) 41,,,, t4(IIA/IIrM,,,-1 + 1)MmMm. f -
-
(17.62)
Second, observe that
l IXt
'l
II';'I >
lS
dtnm.
and hence, when
slm>
&m,
Ixt- /';'I <
dtnm,
I/T1 >
P
-
dtsl)
t IXt I >
dt Mml i S #(
11,1>
dtMm
f IIXIII
rrdirum
by the
dtMm
-
-
,,,)
l
,,,)) -
nml-r
(17.63)
Markov inequality, so that Al,,, <
11.2,-:'IIr#(11,-/T1 < dtnm, I/')'I > J,M,?,)II-Y,IIrrtrfr(Az,, < (IIAlIr+ ,,,)-r
-
S
dtsl)
+ II.X',/f,II3ArJ',,,(M,,, (IIA/4Ilrr+' nm4-r. -
(17.64)
-1 b y 1 ) And lastly, (The final inequality is f'rom replacing Mm .
Atm 3 / dn, m
in view
of the fact that
7;T =
h'l on the set
(1765) .
/';I
fI I S
dtMml. We have therefore
established that Xt 11
A-''' t p S d'vm' t 11
-
(17.66)
,
where
v and
%'m'
'
M v,n+MmMm
=
0 by assumption, since r
--)
d;
=
m)-r+
-
(17.67)
1, and
>
.'
Il.Xr/ttllrr+l,IlA/t4ll rr, 1 ) ( lI.Y,/#,IIr, ztf/rnax
.
(17.68)
.
If o-approximability is satistied with dt
-approximability
17.22 Theorem Let 'J' htt...shmvt) be the fo-approximator of Xt = Xjt,...,Xvt)' RV EBtXt,h'1)E) < x for : > 0, F-> R satisfies (17.40),and If #t: of size then (j,4.Y,)is fo-approximable of size =
-(p.
-(?.
Proof Fix 6
>
0 and M
#( I#r(m) 9,('7)I > -
>
0, and define dt
Elxllff.The
=
dt) S PBt-knhllqpxphl't
+
>
#(#f(A'r,h'7)p(m,:')
F(#/(#s')')E)/3#
+
Markov inequality gives #f(m,:')
s, >
>
dtn, Btxt,h',
Plpxt,hll
>
dt&M).
M4 S M)
(17.69)
Since M is arbitrary the first term on the majorant side can be made as small as desired. The proof is completed by noting that V
Ppxt,h'l)
>
J,&M)
P
Im, X =1
KP
Ut Iff
=
-
'rri
>
dtnlM
>
ditnlM
ll
-
jzz1 %'
#( jXit
<
-
't',
t
l>
dit&lM)
V
f
Xvjm 0 as m -+
--y
x.
w
(17.70)
=l
It might seem as if teaming 17.22 with 17.21 would allow us to show that, given an and fv-bounded, Q-NED sequence, r > 2, which is accordingly Q-approximable hence Q-approximable, any transfonnation satisfying the conditions of 17.22 is and therefore also Q-NED, by 10.12. The catch with circumQ-approximable,
Near Fp/c
Dependence
277
venting the moment restrictions of 17.16 in this manner is that it is not possible to specify the Q-NED size of the transfonned sequence. In (17.67),one cannot put a bound on the rate at which thesequence (,,, ) may converge without specifying the distributions of the Xit in greater detail. However, if it is possible to do this in a given application, we have here an alternative route to dealing with transformations. Ptscher and Prucha (1991a),to whom the concepts of this section are due, define approximability in a slightly different way, in terms of the convergence of the Cesro-sums of the p-nonns or probabilities. These authors say that Xt is fv-approximable p > 0) if
limsup1 -
n
7) 111,-
'yllp
0 as m
--
11 >1
n-x
and is Q-approximable if, for every
(17.71)
.-->
x,
0,
>
n
1
limsupn-X#(lx,- :,1 > )
O as m
--->
(17.72)
--+
x.
>1
n-yx
lt is clear that we might choose to define near-epoch dependence and the mixingale property in an analogous manner, leading to a whole class of alternative convergence results. Comparing these alternatives, it turns out that neither definition dominates, eachpermitting a form of behaviourby the sequences which is ruled out by the other. If (17.55)holds, we may write n
1
limsupn Y 112,--
'/lln
'''
n-yx
''r
-
A
>1
1
n
Tdt n
< limsup
v,a
-->
0
(17.73)
>1
n--hx
so long as the limsup on the majorant side is bounded. On the other hand, if (17.71) holds we may detine n
v,,,
=
limsup
-
17) 111,hmt11,,
(17.74)
>1
n-x
will satisfy 17.18 so long as it is t'initefor and then dt = supp,t 111, Tllp/vml finite t. Evidently, (17.71)permits the existence of a set of sequence coordinates for which the p-norms fail to converge to 0 with m, so long as these are ultimately negligible, accumulating at a rate strictly less than n as n increases. On the other hand, (17.55)permits trending moments, with for example #f 0(/), -
=
,
0, which would contradict (17.71). Similarly, for ,u > 0, and vm > 0, define dtm by the relation >
#( IXt
-
h'II >
dtmm)
Mm,
=
(17.75)
0, define dt = suvmdtm. (17.56)is satisfied and then, allowing v,,i 0 and m if dt < x for each tinite t; this latter condition need not hold under (17.72).On the other hand, (17.72)could fail in cases where, for jed 8 and every m, -->
--
P IXt
-
';l
I > ) is
tending to unity as /
-->
x.
1A? THE LAW OF LARGE NUMBERS
18 Stochastic Convergence
18. 1 Almost Sure Convergence in 512.2.Sometimestheconditionis
wasdefinedformally
Almostsureconvergence stated in the form
# limsup IXn 11
> 6:
-
0, for a11 E
=
>
0.
(18.1)
n->x
1 where, for each (t) Yet another way to express the same idea is to say that #(C) > number E > finite of times as we pass and at most a any s 0, I-L() Xt)l I e C down the sequence. This is also written as =
-
#( IAk Xl -
where i.o. stands for
i.o.)
> E,
0, all E, > 0,
=
(18.2)
often'. probability in (18.2)is assigned to an attribute of the whole Note that the often' idea is sequence, not to a pmicular n. One way to grasp the words, consider the event U';=,,,l event that has occurred to IXn *-1 > erl; in 11 > for least beyond Xn el occurs whenever ( I at one n a given point m in the sequence' lf this event occurs for every m, no matter how large, ( IXn 11 > el occurs infinitely often. ln other words, tinfinitely
'infinitely
'the
-
-
-
.
Cr
f IXn 71 -
>
s, i.o.)
r
O U( IXn
=
-
#1
> El
vl=1 n=m
=
limsupt
IXn
-
11
>
el
.
(18.3)
n-x
Useful facts about this set and its complement
are contained in the following
j emma.
18.1 Lemma Let fEn
TlT be
G
an arbitrary sequence. Then X
(i) P
limsup En
=
1im P n--x
n--x
UEm
.
m=n X
(ii)#
liminf En n-'hoo
=
1im #
OEm
.
m=n
is decreasing monotonically to limsuw Fn. Part Proof The sequence (U7=,aFa1'Q1 (i) therefore follows by 3.4. Part (ii) follows in exactly the same way, since the
The frw of f-zzrg: Numbers
282
sequence f*n=mbJo;=lincreases monotonically to liminf En.
.
A fundamental tool in proofs of a.s. convergence is the Borel-cantelli /'r/;zz?.t7. This has two parts, the part and the part. The former is the most useful, since it yields a very general sufficient condition for convergence, whereas the second parq which generates a necessary condition for convergence, requires independence of the sequence. tconvergence'
Kdivergence'
18.2 Borel-cantelli lemma (i) For an arbitrary sequence of events fEn e
??7)T,
X
XpEnb
PEn, 1
0.
.0.)
<
-
=
(18.4)
n=1
(ii) For a sequence (En e 5 )7 of independent events, X
#(&)
i.o.)
Pk
=
x
=
1.
=
(18.5)
n=1
Proof By countable subadditivity, e
#
'
x
w
U
< XPEn4.
En
(18.6)
n=m
n=m
The premise in (18.4)is that the majorant side of (18.6)is tinite for m x (by 2.25), which further implies This implies X*n=mPEn) 0 as m
1.
=
-->
-->
X
1im P pl-hl
UEn
0.
=
Part (i) now follows by part (i) of 18.1. To prove (ii), note by 7.5 that the collection for any m > 0, and m' > m, /'
#
(18.7)
n=m
(AN,(
e
;)7
is independent; hence
;
;
(1 g)Ecn I-IPYn) 1-I =
h=n1
=
n=m
-
P(En44
n=m
< exp
-XPEn)
--
O as m'
( 18 8)
-+
.x,
.
n=m
by hypothesis, since e-x > 1 x. (18.8)holds for a1l m, so -
X
rtliminfEcnl
=
OYn
lim P m--koo
=
0,
(18.9)
n=m
by 18.1(ii). Hence, PEn i.o.)
=
'tlimsup
En4 = 1 Ptliminf E -
=
1.
.
(18.10)
To appreciate the role of tls result (the convergence partl in showing a.s. convergence, consider the particular case
Stochastic Convergence En
f
=
IXnlk
):
I
X())
-
283
*n=3PEn)< x, the condition PEn) > 0 can hold for at most a finite number of n. The lemma shows that P(En i.o.) has to be zero to avoid a contradiction. Yet another way to characterize a.s. convergence is suggested by the following
If
theorem. (Xn) converges a.s. to X if and only if for all 6: > 0
18.3 Theorem
lim # sup lXn
.:-1
S
-
s
nkm
pl-x
=
1.
(18.11)
e1 e @,
(18.12)
Proof Let X
A,,,(E)
=
(')1:
IS
IAk(t,))
-#()
n=m
and then (18.11) can be written in the fonn lim,,,-+x#(Am(e)) 1. The sequence U,,7=lAm(E), fA,,,(E) 17 is non-decreasing, so A,,,(E) = UXlA/:); letting A(E) 1. (18. 11) can be stated as #(A(:)) Define the set C by the property that, for each ) s C, (Xn((t))lT converges. That is, for (t) e C, =
=
=
3
lA%((J))
such that sup
??z()
-
X(l
numl)
1<
e, for all
Evidently,
e C= ) e Am(E) for some (l, so that CiA(E). tonly ito 1, proving #(A(E)) 1 for al1 E > 0. Set E To show if' assume k, and define #(A(E))
=
i) >
(18.13)
0.
HencePto
1implies
=
.
=
,
X
A*
=
f)
=
t=1
$Ik for positive integer
C
<x)
Ajlk)
=
tj
Atlkjc
(18.14)
.
&1
1. The second equality here is 1.1(iv). By 3.6(ii), #(A*) 1 #(U7=1A(1/k)9 z4* is a convergent outcome in the sense of (18.13), hence A* But every element of c C, and the conclusion follows.w =
=
-
The last theorem characterizes a.s. convergence in tenns of the unform proximity of the tail sequences ( IXnll) X()) l 1';.,,,to zero, on a setAm whose measure x. A related, but distinct, result establishes a direct link approaches 1 as m between a.s. convergence and uniform convergence on subsets of (. -
--y
-6:t-:
X, there exists for every > 0 a 18.4 Egoroff's Theorem lf and only if Xn X() uniformly on C(&). set C(8) with #(C48)) k 1 8, such that Ak() .-+
-
Proof To show if', suppose aL()) converges uniformly on sets 1 2 3 The sequence ( C(1/k) k e (N) can be chosen as non-decreasing tonicityof #, and #(U7=1C(1/k)) 1 by continuity of #. To show Eonly
1
,
;***
,
=
C(I/Q,k
=
by mono1et
tif',
The fuzw of fzzrgd Numbers
284
A,,()
O (t,):lXn(t,))-X()I
=
<
llml,
( l 8 15) .
n=klmj
-2-&'.
In view of a.s. km) being chosen to satisfy the condition #(A,,,(6)) 1 convergence and 18.3, the existence of finite klm) is assured for each m. Then if *
c(8)
nAa,tl,
=
(18.16)
rrI=1
convergence is uniform on C(l by construction; that is, for ever.y ) e C(8), IXnt)l X()) I < Im for n k km), for each m > 0. Applying 1.1(iii) and subadditivity, we tind, as required, -
#(c())
1
=
UAm()f
-#
m=1 *
1 -
E (1 -
#(z4m()))
m=1
k 1 8.
(18.17)
.
-
18,2 Convergence in Probability In spite of its conceptual simplicity, the theol'y of almost sure convergence cannot easily be appreciated without a grasp of probability fundamentals, and traditionally, an alternative convergence concept has been preferred in econometric theory. If, for any E > 0,
1im#(IXn X1 -
> f:)
=
0,
(18.18)
n'M
Xn is said to converge in probability (inpr.) to X. Here the convergent sequences are specified to be, not random elements (X,,()) 1T,but the nonstochastic sequences (#( IXn Xl > E) )T.The probability of the convergent subset of ('1 is left unspecified. However, the following relation is immediate from 18.3, since (18.11) -
implies (18.18).
18.5 Theorem If Xn
-66.9
X then Xn -E'.% X.
(a
The converse does not holdkConvergence in probability imposes a limiting condix. The tion on the marginal distribution of the pth member of the sequence as n negligible deviation of is approaches Xn X 1 probability that the from as we move down the sequence. Almost sure convergence, on the other hand, requires that byond a certain point in the sequence the probability that deviations are negligible from there on approaches 1. While it may not be intuitively obvious that a sequence can convergein pr. but not a.s., in 18.16 below we show thatconvergence in pr. is compatible with a.s. n/nconvergence. However, convergence in probability is equivalent to a.s. convergence on a ..-.h
Stochastic Convergence
285
subsequence', given a sequence that converges in pr., it is always possible, by throwing away some of the members of the sequence, to be left with an a.s. convergent sequence. 18.6 Theorem Xn -EI-> X if and only if every subsequence fAk, k e IN) contains a further subsequence fxnkcj, j e (N) which converges a.s. to X. 0 for any e > 0. This means if : suppose P( IXn XI> E) Proof To prove #( that, for any sequence of integers fns k e INl IXnk XI > e) 0. Hence for each j G LNthere exists an integer kU)such that 'only
-
--
-->
-
,
#( 1Xnk 71
>
-
1/.j) < 2-/, a11k k k(j).
(18.19)
Since this sequence of probabilities is summable over j, we conclude from the first Borel-cantelli lemma that
#( IXnkv
XI >
-
$lj i.o.)
=
0.
(18.20)
It follows, by consideration of the intinite subsequences Lj k J) 0 for every 6: > 0, and hence the that #( IXnkv XI > e i.o.) required. fxnw)J converges a.s. as : if (Ak) does not convergence in probability, there To prove > s)l subsequence (rl1)such that inflf#tlAk-rl s, for some 6: > which nlles out subsequence of in out convergence (?u), pr. on any a.s. on the same subsequence, by 18.5. w =
-
Kif'
18.3 Transformations
for J > 1/:, subsequenc: must exist a 0. This rules convergence
and Convergence
The following set tools of asymptotic even though most k-vector Xn is said
of results on convergence, a.s. and in pr., are lndamental theory. For completeness they are given for the vector case, of our own applications are to scalar sequences. A random to converge a.s. (in pr.) to a vector X if each element of Xn converges a.s. (in pr.) to the corresponding element of X. 18.7 Lemma Xn X a.s. (in pr.) if and Only if IlAk #II 0 a.s. (in pr.).19 -->
-->
-
Proof Take first the case of a.s. convergence. The relation
expressedas
k
P 1im n-yoo
for any
0. But
6: >
xni Xill X f=1
<
-
(18.21)implies
# li m 1Xni Xi I < -
:2
Il#n #ll -
-E't-y 0
may be
(18.21)
that E,
=
1,
.
..,k
=
1,
(18.22)
n'-oo
ito observe that if (18.22)holds, f'tlimp,--hxllAk #1I To prove IS 1 for any : > 0. To get the proof for convergence in pr., replace < k E) #(limn-+x...) everywhere by limn-+x#(...), and the arguments are identical. w proving
Konly
'if'.
-
,
=
,
The fww of f-zzr'd Numbers
286
There are three different approaches, established in the following theorems, to the problem of preserving convergence (a.s. or in pr.) under transformations.
18.8 Theorem Let Rk R be a Borel function, 1etCg R be the set of continuity points of g, and assume PX e G) = 1. #(m. X then g(Xn4 (i) If Xn -Z'.:o .-.E'.4 X then gxn) (ii) If Xn -->
':
-6.+
-6-+
'(m.
t
is by hypothesis a set D e @, with #(D) = 1, such that A%() #4(z)),each (t G D. Continuity and 18.7 together imply that glxnll for each (.t e X-1(Cg ) ch D. This set has probability 1 by 3.6(iii). g(#()) Toprove (ii),analogous reasoning shows that, for each r > 0, 3 > O such that Proof For case
(i), there
--
--
1t0:IlX()
-A)lI
Note that if #(#)
) c'-kr,sj
<
c (: l'(A())
1 then for any A
=
PA rn #)
1 P(Ac
=
-
LJ
G
< :).
-p(#())l
(18.23)
@,
Bc) k PA)
-
PBc)
PA4
=
(18.24)
by de Morgan's law and subadditivity of #. ln paYcular, when PX e Cg) = 1, (18.23) and monotonicity imply
#((I#n-fIl
YS
<
#(I'(Ak)
-
#(mI <
E).
(18.25)
Taking the limit of each side of the inequality, the minorant side tends to 1 by hypothesis. w We may also have cases where only the difference of two sequences is convergent.
18.9 Theorem Let t#a) and (zn) be sequences of randomk-vectors (notnecessarily converging) and g the function detined in 18.8, and let P(Xn e Cg) = P(Zn e Cg) = 1 for evel'y n. 0. (i) lf IlA% 11 O then 1:(.Yn) #(za)I -E',ly 0 then I#(.2L)#(zn)1 0. (ii)If llAk -61..>
-f..>
-zn
-
-znll
-r...y
-
OQazzjvn EZa PCEX) = Z-a1(c#), EX = ()xnmjl, and FZ = J'ut E'n = Xn-lc ), PEZ) ! and 3.6tiii). Also let D be the set on which Il#nZn 11 = 1, by assumptlon The proof is now a straightforward variant of te preceding one, with converges. X EZ r1(C the set E c g ). . playing the role of
Proof
=
.
-
The third result specifies convergence to a constant limit, but relaxes the continuity requirements. R be a Borel function, continuous at a. 18.10 Theorem Let g: R If Xn gx ga). (i) a then (ii) If X. -E(- c then g(Xn4 -E(-> ga). --,9
-6-,.-+
-6-y
Proof By hypothesis there is a set D G F, with #(D) = 1, such that Xn c, ga) for (1)e D, proving (i).Likewise, each (l) e D. Continuity implies #(A%()) --
-->
1:
llA%()
-cll
<
l c lt,):l#(Ak()) -g(c)I
and (ii) follows much as in the preceding theorems.
*
< El,
(18.26)
Stochastic Convergence
287
Theorem 18.10(ii) is commonly known as Slutsky's theorem (Slutsky 1925). These results have a vast range of applications, and represent one of the chief reasons why limit theory is useful. Having established the convergence of one set of statistics, such as the first few empirical moments of a distribution, one can then deduce the convergence of any continuous function of these. Many commonly used estimators fall into this category.
18.11 Example LetA,y be a random matrix whose elements converge a.s. (inpr.) to a limit A. Since the matrix inversion mapping is continuous everywhere, the A ) follow on app j y jyjg j.: : ejement by results a.s.lim An-1 A (plimA,y element. E! -1
-1
-1
=
=
.
The following is a useful supplementary result, for a case not covered Slutsky theorem because Fn is not required to converge in any sense.
by the
18.12 Theorem Let a sequence (Fn)Tbe bounded in probability (i.e.,Op(1) as n x); if Xn -EI-> 0, then XnYn.-CL>0. FPn
Proof For a constant # > 0, define L1 ( Iyklus). The event ( lXaF,,I 6: > 0 is expressible as a disjoint union: IXnYnI sl l IA%IIF$ l k e:1 tp f IXnIIi'n F$ l El
t
For any :
>
0,
-
.
( IXn IIF,, I k #(l-Yn lIi
Ik
c l IXn I 2 e/#l,
:1
el for
=
=
s) < #(l-Yn 1
zlB4
-->
(18.27)
and 0.
-->
(18.28)
> 0, B& < cxa such that By the 0:(1) assumption there exists, for each YB YB #( IL Fn%I > 0) < for n e ENSince ( IXnIIF,, n l sl c l Il'n n I > 01, (18.27)and additivity imply, putting B = Bs in (8.28),that -
-
-
.
lim P( lXnk
lk
e)
<
(18.29)
.
n.->x
The theorem follows since both 6: and 6 are arbitrary.
.
18.4 Convergence in L Norm #
Recall that when F(l Xn IP) < x, we have said that Xn is fw-bounded. Consider, for lf F(llxnllp < :cn all n, and limn-jxllAk --Y11p 0, p > 0, the sequence ( II-L-XIIpl7. When p = 2 we speak of Xn is said to converge in N zm?wzto x (writeXn l convergence in mean square (m.s.). Convergence in probability is sometimes called fvconvergence, terminology which can be explained by the factthatfv-convergence implies fx-convergence for 0 < q < p by Liapunov's inequality, together with the following relationship, which is immediate from the Markov inequality. =
---+
m.
-1f,+
18.1 Theorem lf Xn The converse
X for any p
>
0, then Xn .T.LA X. n
does not follow in general, but see the following theorem.
18.14 Theorem If Xn
.-#-(..>
X, and
( IL IPITis uniformly
integrable, then Xn -f:E.-)X.
288
The fzzw of zlr':
Numbers
Proof For e > 0, e-lAk
-11:
=
F(1( Ix,,-xl''>:) lXn
K e'(1(Ixn-xj''>EJ
-11#)
+
F(1f
--YlP) laY?,
Ixa-xtzkEl
+ s. l-Yn-XI#)
(18.30)
x. Uniform integrab0 as n Convergence in pr. means that #( lXn Xl > e) ility therefore implies, by 12.9, that the expectation on the majorant side of (18.30) converges to zero. The theorem follows since e: is arbitrary. K .-+
.-.+
-
We proved the a.s. counterpart of this result, in effect, as 12.8, whose conclusion can be written as: jXn X1 -E.. 0 implies A'lXn XI 0. The extension f'rom the fw1case to the Lp case is easily obtained by applying 18.8(i) to the case #(.) -,-h
-
-
lP. =I .
One of the useful features of Lp convergence is that the Lp norms of Xn X define a sequence of constants whose order of magnitude in n may be determined, providing a measure of the rate of approach to the limit. We will say for example that Xn converges to X in mean square at the rate nk if Ijxf-.Y112= On-h, but not on -k). This is useful in that the scaled random variable nkxt A') may be non-degenerate in the limit, in the sense of having positive but finite limiting variance. Determining this rate of convergence is often the tirst step in the analysis of limiting distributions, as discussed in Part V below. -
-
18.5 Exnmples Convergence in pr. is a weak mode of convergence in that without side conditions it does not imply, yet is implied by, a.s. convergence and Lp convergence. However, there is no implication from a.s. convergence to Lp convergence, or vice versa. A good way to appreciate the distinctions is to consider cases where one or other mode of convergence fails to hold. dpathological'
18.15 Example Look again at 12.7, in whichAk 0 withprobability 1 1/n, and Xn = = n with probability 1/n, for n 1,2,3,.... A convenient model for this sequence where m is Lebesgue is to let ( be a drawing from the space (r0,1),m,1j,pz) variable and random define the measure, =
Xnlj
=
n,
)
-
e (0,1/a),
(18.31)
0, otherwise.
The set f : limuAkt) # 0) consists of the point (0J, and has p.m. zero, so that Xn -E't-A 0 according to (18.1).But Fl XnlP = 0 (1 1/rl) + t'l'ln = n#-1 lt will be recalled that this sequence is not uniformly integrable. lt fails to converge in 1 for every n. The Lp for any p > 1, but for the case p = 1 we obtain Exn) limiting expectation of Xn is therefore different from its almost sure limit. n -
.
.
=
The same device can be used to define a.s. convergent sequences wltich do not converge in Lp for any p > 0. It is left to the reader to construct examples.
Stochastic Convergence
289
18.16 Example Let a sequence be generated as follows: .Y1 1 with probability 1; xztxzl are either (0,1) or (1,0) with equal probability; (X4,X5,X are chosen from (1,0,0),(0,1,0),(0,0,1) with equal probability; and so forth. For k 1,2,3,... the next k members of the sequence are randomly selected such that one of them is unity, the others zero. Hence, for n in the range (kt/c 1) + 1, jkk + 1)J,P(Xn 1) 1/k, as well as FIXnlP 1/k for p > 0. Since k .-.A x as n .-+ x, it is clear that Xn converges to zero both in pr. and in Lp norm. But since, for any n, Xnwj 1 a.s. for infinitely many j, =
=
-
=
=
=
=
#(I-L I <
i.o.)
E,
=
(18.32)
0
for 0 f 6: f 1. The sequence not only fails to converge a.s., but actually converges with probability 0. Consider also the sequence fkl/rxn J whose members are either 0 or kllr in the ,
lk(k klrx I#) PVr-1 an d b suitable n y range (r'kk 1) + 1 z + 1)j. Note that E( I choice of r we can produce a sequence that does not converge in Lv fop > r. With 1 for al1 n, but as in 18.15, the sequence is not uniformly r 1 we have Ekxn) integrable. The limiting expectation of the sequence exists, but is different from the probability limit. I:a =
-
,
,
=
=
ln these non-uniformly integrable cases in which the sequence converges in Lj but not in f.,l+e for any 0 > 0, one can see the expectation remaining formally well-defined in the limit, but breaking down in the sense of losing its intuitive intepretation as the limit of a sample average. Example 18.15 is a version of the well-known St Petersburg Paradox. Consider a game of chance in which the player announces a number n e N, and bets that a succession of coin tosses will produce n heads before tails comes up, the pay-off for a correct predidion being E2 l+1 T h e probability of winning is 2 -/-1 so th e e xp ected winnings are f 1', that is to say, game' if the stake is fixed at f 1. The sequence of random winnings it is a generated by choosing n 1,2,3,... is exactly the process specified in 18.15. ao Xn .
,
Efair
=
If n is chosen to be a vel'y large number, a moment's reflection shows that the probability limit is a much better guide to one's prospective winnings in a t'inite number of plays than the expectation. The paradox that with large n no one would be willing to bet on this apparently fair game has been explained by appeal to psychological notions such as risk aversion, but it would appear to be an adequate explanation that, for large enough n, the expectation is simply not a practical predictor of the outcome.
18.6 Laws of Large Numbers n-1Z7=1Xf.Suppose that E(Xt) = Let (m)Tbe a stochastic sequence and definex g,?and n -1Z7=1p g, w ith lp,1 < pa,' this is trivial in the mean-stationary case in which gf g, for al1 t. In this simple setting, the sequence is said to obey the weak law of large numbers (WLLN) when kn -CC..Ag,, and the strong law of large numbers (SLLN) when kn p,. These statements of the LLNS are standard and familiar, but as characterizations =
--)
=
-.62-.4
Ihe fzzw of fzzr'd Numbers
290
of a class of convergence results they are rather restrictive. We can set jt 0 with no loss of generality, by simply considering the centred sequence (A) g,flq; centring is generally a good idea, because then it is no longer necessary for the time average of the means to converge in the manner specified. We can quite easily have n -1E7=1h4 cxa at the same time that n-1E: 1(AX)) w) 0. In such cases the 1aw of large numbers requires a modified interpretation, since it ceases to make sense to speak of convergence of the sequence of sample means. =
-
--)
-
--
=
More general modes of convergence also exist. It is possible that kn does not that there exists a converge in the manner specified, even after centring, but -1 (Jn ):' such that an 1 cxa and an X1=jXt-...+0. Results sequence of positive constants will and others too, in a fully general array subsume possibilities, below these ,21 k l n=1 i s a triangular stochastic array with formulation of the problem. lf f(AICJ (1k1':.1 an increasing integer sequence, we will discuss conditions for =
Sn
kn
=
Txnt--EC->0. >1
(18.33)
A result in this form can be specialized to the familiar case with Xnt kn n, bu t there are important applications where the an-1 Xt g' and an greater generality is essential. We have already encountered two cases where the strong law of large numbers applies. According to 13.12, kn g' F(X1) when (m) is a stationary ergodic sequence and FI I < x. We can illustrate the application of this type of result by an example in which the sequence is independent, which is sufficient for =
=
-
=
-810
=
.11
ergodicity.
18.17 Example Consider a sequence of independent Bernoulli variables Xt with
1) P(Xt 0) :1.,that is, of coin tosses expressed in binary form (see 12.1). The conditions of the ergodic theorem are clearly satisfied, and we can E(Xt) j. This is called Borel's normal number conc lu de that n-1E) jA theorem, a normal number being defined as one in which Os and ls occur in its binary expansion with equal frequency, in the limit. The normal number theorem therefore states that almost every point of the unit interval is a normal number; that is, the set of normal numbers has Lebesgue measure 1. Any numberwith aterminating expansion is clearly non-normal and we know that a1l such numbers are rationals; however, rationals can be normal, as for example ,t, which has the binary expansion 0.01010101010101... This is a different result from the well-known zero measure of the rationals, and is much stronger, because the non-normal numbers include inutionals, and fonn an uncountable set. For eyample, anynumberwithabinary expansionof theformo.l 1:1 11:21 1&g11...where the bi are arbitrary digits is non-normal; yet this set can be put into 1-1 corin other words, with the points of respondence with the expansions the