ADVANCED TEXTS IN ECONOMETYCS General Editors
C. W. J. Granger
G. E. Mizon
STOCHASTIC LIMIT THEORY An lntroduction for Econometricians
JAMES DAVIDSON
(lxford llniversity Press 1994
Oxjord University Press, Walton Street Oxjord oxa Oxford New York Athens Auckland Banghok Bombay Calcutta Cape Tozvn Dar es Salaam Delhi Florence Hong Kong Istanbul Karachi Kuala Lumpur Madras Madrid Melbourne Mexico City Nairobi Jtzrl Singapore Ttzzc Tohyo Toronto and assoclted companies in Berlin Ibadan ,
6Dp
Published in the Unitei States by Oxford University Pyess l'nc., Nec York
(0 JamesDavidson, 1994 All rlk/lfl reserved. No part of this publication may be repvoduced, stored in a rdfrfcrtz/ system, or transmitted, in any or .',Vtz'rly means, without the yrfor permsion 2'A; zvriting t)./ Oxford Ublbcrlf.y Press. Within the UK, exceptions are allouled p respect of any fair dealing .Jorthe Azlr/t?ycoj research or private study, or crl'li'cl'lpl or reviezv, as permitted under the Co/wrr/, Desns and Patents Act, zq'l', or fr the case V veprographic reproduction in accordance zcff/l the terms V the licences issued by the Copyright Licensing Agency. A'r/tzpzrc: concerntg rcyrotucfpn outside these terms and in other countries should be sent to the Rights Depavtment, Oxjord University Press, at the address above ./brv
F/7l'.book f.ssold subject to the condition that it shall not, by zt?tz.'p oj trade or otherwe, be lent, re-sold, Itived out or otheyulise circulated without the publisher's prior consent s any formof binding (?r covey other than that l'A; which 2'J is published and xithout a similar condition i'ac/z/kg this condhion being imposed on the subsequent purchaser srzk./z Librar.y Ctzftz/twlfag in Publication Data Data available Library
of Congress Cataloging in Publication Data
Data available ISBN 0-19-877402-8 ISBN 0-19-877403-6 ('Pbk) 1 3 5 7 9 10 8 6 4 2 Printed in Great Britain on acid-jkee yl/cr by Biddles Ltd., Guildjord and King
's
Lynn
For Lynette, Julia, and Nicola.
what in me is dark Illumine, what is 1ow raise and support, That, to the height of this great argument, 1 may assert Eternal Providence, And justify the ways of God to men.'
& .
.
.
Paradise f.,t?l/, Book 1, 16-20
Conients Preface
xiii
Mathematical
Symbols and Abbreviations
AXA
Part 1: Mathematics 1. Sets 1.1 1.2 1.3 1.4 1.5 1.6
and Numbers Basic Set Theory Countable Sets The Real Continuum Sequences of Sets Classes of Subsets Sigma Fields
3 8 10 12 13 15
2. Limits and Continuity 2.1 The Topology of the Real Line 2.2 Sequences and Limits 2.3 Functions and Continuity 2.4 Vector Sequences and Functions 2.5 Sequences of Functions 2.6 Summability and Order Relations 2.7 Arrays
20 23 27 29 30 31 33
3. Measure 3.1 Measure Spaces 3.2 The Extension Theorem 3.3 Non-measurability 3.4 Ptoduct Spaces 3.5 Measurable Transfcymations 3.6 Borel Functions
36 40 46 48 50 55
4. lntegration 4.1 Construction of the Integral 4.2 Properties of the lntegral 4.3 Product Measure and Multipl 4 4 The Radon-Nikodym
57 61 64 69
lntepals
'theorem
.''' -' y'
-'
7*
(qd qd 5. Metric
Spaces and Metris Distances 1 5. ) xjj 5.2 Separability and tjmptttt t(((...'.! j y.) .. . r), 5.3 Exampls ' ..)
.....
'
.
...
...
E. .
.........y
.. y. . . . . . E)rt)yy),,:yy't,y:r(,q)'yt 2 .,
i.; ! E E. ... . . . .. ... . ... . q.
. r
. j.
. .. q .
.
...
75 78 82
Contents
5.4 Mappings on Metric Spaces 5.5 Function Spaces
84 87
6. Topology 6.1 Topological Spaces 6.2 Countability and Compactness 6.3 Separation Properties 6.4 Weak Topologies 6.5 The Topolpgy of Product Spaces 6.6 Embedding and Metrization
93 94 97 101 102 105
Part 11: Probability 7. Probability Spaces 7. 1 Probability Measures 7.2 Conditional Probability 7.3 Independence 7.4 Product Spaces
111 113 114 115
8. Random Variables 8. 1 Measures on the Line 8.2 Distribution Functions 8.3 Examples 8.4 Multivariate Distributions 8.5 Independent Random Variables
117 117 122 124 126
9. Expectations 9. 1 Averages and Integrals 9.2 Expectations of Functions of X 9.3 Theorems for the Probabilist's Toolbox 9.4 Multivariate Distributions 9.5 More Theorems for the Toolbox 9.6 Random Variables Depending on a Parameter
128 130 132 135 137 140
10. Conditioning 10.1 Conditioning in Product Measures 10.2 Conditioning on a Sigma Field 10.3 Conditional Expectations 10.4 Some Theorems on Conditional Expectations 10.5 Relationships between Subfields 10.6 Conditional Distributions
143 145 147 149 154 157
11. Characteristic Functions 11.1 The Distribution of Sums of Random Variables 11.2 Complex Numbers
161 162
Contents
11.3 The Theory of Characteristic Functions 11.4 The Inversion Theorem 11.5 The Conditional Characteristic Function
f;r
164 168 171
Part 111:Theory of Stochastic Processes 12. Stochastic Processes 12.1 Basic Ideas and Terminology 12.2 Convergence of Stochastic Sequences 12.3 The Probability Model 12.4 The Consistency Theorem 12.5 Uniform and Limiting Properties 12.6 Unifonn lntegrability
177 178 179 183 186 188
13. Dependence 13.1 Shift Transformations 13.2 Independence and Stationarity 13.3 Invariant Events 13.4 Ergodicity and Mixing 13.5 Subfields and Regularity 13.6 Strong and Uniform Mixing
191 192 195 199 203 206
14. Mixing 14.1 Mixing Sequences of Random Variables 14.2 Mixing Inequalities 14.3 Mixing in Linear Processes 14.4 Sufficient Conditions for Strong and Uniform Mixing
209 211 215 219
15. Martingales 15.1 Sequential Conditioning 15.2 Extensions of the Martingale Concept 15.3 Martingale Convergence 15.4 Convergence and the Conditional Variances 15.5 Martingale Inequalities
229 232 235 238 240
16. Mixingales 16.1 Definition and Examples 16.2 Telescoping Sum Representations 16.3 Maximal Inequalities 16.4 Uniform Square-integrability
247 249 252 257
17. Near-Epoch Dependence 17.1 Definition and zxamples 17.2 Near-Epoc Dependence and Mixingales
261 264
Contents
17.3 Near-Epoch Dependence and Transformations 17.4 Approximability
267 273
Part lV: The Law of Large Numbers 18. Stochastic Convergence 18.1 Almost Sure Convergence 18.2 Convergence in Probability 18.3 Transformations and Convergence 18.4 Convergence in Lp Norm 18.5 Examples 18.6 Laws of Large Numbers
281 284 285 287 288 289
19. Convergence in fp-Norm 19.1 Weak Laws by Mean-square Convergence 19.2 Almost Sure Convergence by the Method of Subsequences 19.3 A Martingale Weak Law 19.4 A Mixingale Weak Law 19.5 Approximable Processes
293 295 298 302 304
20. The 20. 1 20.2 20.3 20.4 20.5 20.6
306 311 313 316 318 323
Strong Law of Large Numbers Technical Tricks for Proving LLNS The Case of Independence Martingale Strong Laws Conditional Variances and Random Weighting Two Strong Laws for Mixingales Near-epoch Dependent and Mixing Processes
21. Uniform Stochastic Convergence 21.1 Stochasti Functions on a Parameter Space 21.2 Pointwise and Unifonn Stochastic Convergence 21.3 Stochastic Equicontinuity 21.4 Generic Uniform Convergence 21.5 Unifonu Laws of Large Numbers
Part V: The Central Lint
327 330 335 337 340
Theorem
22. Weak Convergence of Distributions 22.1 Basic Concepts 22.2 The Skorokhod Representation Theorem 22.3 Weak Convergence and Transformations 22.4 Convergence of Moments and Characteristic Functions 22.5 Criteria for Weak Convergence 22.6 Convergence of Random Sums
347 350 355 357 359 361
Contents
23. The 23. 1 23.2 23.3 23.4 24.
CLTS
24.1 24.2 24.3 24.4 24.5
11
Classical Central Limit Theorem The i.i.d. Case lndependent Heterogeneous Sequences Feller's Theorem and Asymptotic Negligibility The Case of Trending Variances
364 368 373 377
for Dependent Processes A General Convergence Theorem The Martingale Case Stationary Ergodic Sequences The CLT for NED Functions of Mixing Processes Proving the CLT by the Bernstein Blocking Method
380 383 385 386 391
25. Some Extensions 25.1 The CLT with Estimated Normalization 25.2 The CLT with Random Norming 25.3 The Multivariate CLT 25.4 En'or Estimation
399 403 405 407
Pa14 V1: The Functional Central Limit Theorem 26. Weak Convergence in Metric Spaces 26.1 Probability Measures on a Metric Space 26.2 Measures and Expectations 26.3 Weak Convergence 26.4 Metrizing the Space of Measures 26.5 Tightness and Convergence 26.6 Skorokhod's Representation
413 416 418 422 427 431
27. Weak Convergence in a Function Space 27.1 Measures on Function Spaces 27.2 The Space C 11.3 Measures on C 27.4 Brownian Motion 27.5 Weak Convergence on C 27.6 The Functional Central Limit Theorem 27.7 The Multivariate Case
434 437 440 442 447 449 453
28. Cadlag Functions 28.1 The Space D 28.2 Metrizing D 28.3 Billingsley's Vetric 28.4 Measures on D
456 459 461 465
Contents
Xlt
28.5 Prokhorov's Metric 28.6 Compactness and Tightness in D 29.
FCLTS
29.1 29.2 29.3 29.4 29.5
for Dependent Variables The Distribution of Continuous Functions on D Asymptotic Independence The FCLT for NED Functions of Mixing Processes Transfonned Brownian Motion The Multivariate Case
467 469 474 479 481 485 490
30. Weak Convergence to Stochastic Integrals 30.1 Weak Limit Results for Random Functionals 30.2 Stochastic Processes in Continuous Time 30.3 Stochastic Integrals 30.4 Convergence to Stochastic Integrals
496 500 503 509
Notes
517
References
519
lndex
Preface
Recent years have seen a marked increase in the mathematical sophistication of econometricresearch. While thetheory of linearparametric models which fonns the backbone of the subject makes an extensive and clever use of matrix algebra, the statistical prerequisites of this theory are comparatively simple. But now that these models are pretty thoroughly understood, research is concentrated increasingly on the less tractable questions, such as nonlinear and nonparametric estimation and nonstationary data generation processes. The standard econometlics texts are no longer an adequate guide to this new technical literature, and a sound understanding of the probabilistic foundations of the subject is becoming less and less of a luxury. The asymptotic theory traditionally taught to students of econometrics is founded on a small body of classical limit theorems, such as Khinchine's weak law of large numbers and the Lindeberg-Lvy central limit theorem, relevant to the stationary and independent data case. To deal with linear stochastic difference equations, appeal can be made to the results of Mann and Wald (1943a),but even these are rooted in the assumption of independent and identically distributed disturbances. This foundation has become increasingly inadequate to sustain the expanding editke of econometric inference techniques, and recent years have seen a systematic attempt to construct a less restrictive limit theory. Hall and Heyde' s Martingale Limit Fdt??'.y and its Application (1980)is an important landmark, as are a series of papers by econometricians including among others Halbert White, Ronald Gallant, Donald Andrews, and Herman Bierens. This work introduced to the econometrics profession pioneering research into limit theot'y under dependence, done in the preceding decades by probabilists such as J. L. Doob, 1. A. Ibragimov, Patrick Billingsley, Robert Serfling, Murray Rosenblatt, and Donald McLeish. These ltter authors devised various concepts of limited dependence for general nonstationary time series. The concept of a martingale has a long history in probability, but it was primarily Doob's Stochastic Processes (1953)that brought it to prominence as a tool of limit theory. Martingale processes behave like the wealth of a gambler who undertakes a succession of fair bets; the differences of a martingale (the net winnings at each step) are unpredictable from lagged information. Powerful limit theorems are available for martingale difference sequences involving no further restrictions on the dependence of the process. Ibragimov and Rosenblatt respectively defined strong mixing and unbrm mixing as charactermemory' or independence at long range. McLeish defined the izations of notion of a mixingale, the asymptotic counterpart of a martingale difference, becomingunpredictable m steps aheadaspz becomeslarge. This is aweakerproperty than mixing because it involves only low-order moments of the distribution, but mixingales oossess most of those attributes of mixine orocesses needed to make
,
Preface
limit theorems work. Very important from the econometrician's point of view is the property dubbed by Gallant and White (1988)near-epoch dependence from a phrase in one of McLeish's papers, although the idea itself goes back to
Billingsley (1968)andlbragimovtlg6z). Themixingproperty may notbepreserved by transformations of sequences involving an intinite number of lags, but nearepoch dependence is acondition under whichthe outputs of adynamic econometric model can be shown, given some further conditions, to be mixingales when the inputs are mixing. Applications of these results are increasingly in evidence in the econometric literature', Gallant and White's monograph provides an excellent survey of the possibilities. Limit theorems impose restzictions on the amount of dependence between sequence coordinates, and on their marginal distributions. Typically, the probability of outliers must be controlled by requiring the existence of higher-order moments, but there are almost always trade-offs between dependence and moment restrictions, allowing one to buy more of one at the price of less of the other. The fun of proving limit theorems has been to see how far out the envelope of sufficient conditions can be stretched, in one direction or another. To complicate matters, one can get results both by putting limits on the rate of approach to independence (the rate of mixing), and by limiting the type of dependence (the martingale approach), as well as by combining both types of constraint (the mixingale approach). Theresults now available areremarkably powerful,judged by the yardstick of the classical theory. Proofs of necessity are elusive and the limits to the envelope are not yet known with certainty, but they probably lie not too far beyond the currently charted points. Perhaps the major development in time-series econometrics in the 1980s has been the theory of cointegration, and dealing with the distributions of estimators when time series are generated by unit root processes also requires a new type of limit theory. The essential extra ingredient of this theory is the functionalcentral limit theorem (FCLT). The proof of these weak convergence results calls for a limit theory for the space of functions, which throws up some interesting problems which have no connterpart in ordinary probability. These ideas were pioneered by Russian probabilists in the 1950s, notably A. V. Skorokhod and Yu. V. Prokhorov. lt turns out that FCLTS hold under properties generally similar to those for the ordinary CLT (thoughwith a crucial difference), and they can be analysed with the same kind of tools, imposing limitations on dependence and outliers. The probabilistic literature wlch deals with issues of this kind has been seen as accessible to practising econometricians only with difticulty. Few concessions are made to the nonspecialist, and the concerns of probabilists, statisticians, and econometlicians are frequently different. Textbooks on stochastic processes (Cox and Miller 1965 is a distinguished example) often give prominence to topics would regard as fairly specialized (e.g. Markov chains, that econometricians processes in continuous time), while the treatment of important issues like nonstationarity gets tucked away under the heading of advanced or optional material if not omitted altogether. Probability texts are written for students of
mathemtics
and ag&ume
n fnmilinritv
wifh
fln/R
fnllrlnr-
nf
tlao
t.lxlor':
'lqne
Preface
econometricians may lack. The intellectual investmnt required is one that students and practitioners are often, quite reasonably, disinclined to make. It is with issues of this sort in mind that the present book has been written.
The t'irstobjective has been to provide a coherent and unified account of modern asymptotic theory, which can function as both a course text, and as a work of reference. The second has been to provide a grounding in the requisite mathematics and probability theory, making the treatment sufticiently self-contained that even readers with limited mathematical training might make use of it. This is not to say that the material is elementary. Even when the mathematics is mastered, the reasoning can be intricate and demand a degree of patience to absorb. Profs for nearly all the results are provided, but readers should never hesitate to pass over these when they impede progress. The book is also intended to be useful as a reference for students and researchers who only wish to know basic things, like the meaning of technical tenns, and the variety of limit results available. But, that said, it will not have succeeded in its aim unless the reader is sometimes if for no better stimulated to gain a deeper understanding of the material reason, because this is a theory abounding in mathematical elegance, and technical ingenuity which is often dazzling. -
Outline of the Work Part I covers almost a11the mathematics used subsequently. Calculus and matrix algebra are not treated, but in any case there is little of either in the book. Most readers should probably begin by reading Chapters 1 and 2, and perhaps the first sections only of Chapters 3 and 4, noting the definitions and examples but skipping a11 but the briefest proofs initially. These chapters contain some difticult material, which does not a11need to be mastered immediately. Chapters 5 and 6 are strictly required only for Chapter 21 and Part VI, and should be passed over on tirst reading. Nearly everything needed to read the probability literature is covered in these chapters, with perhaps one notable exception the theory of normed spaces. Some treatments in probability use a Hilbel't space framework, but it can be pvoided. The number of pplications exploiting this approach seemed currently too small to justify the added technical overhead, although future developments may require this judgement to be revised. Pm 11 covers what for many readers will be more familiar territory. Chapters 7, 8, and 9 contain essential background t be skimmed or studied in more depth, as appropriate. lt is the collections of inequalities in j9.3 and j9.5 that we will have the most call to refer to subsequently. The content of Chapter 10 is probably less familiar, and is very important. Most readers will want to study this chapter carefully sooner or later. Chapter 11 can be passed over initially, but is needed in conjunction with Part V. In Part 11I the main business of the work begins. Chapter 12 gives an introduction to the main concepts arising in the study of stochastic sequences. Chapters 13 and 14 continue the discussion by reviewing concepts of dependence, and Chapters 15, 16, and 17 deal with s/ecializedclasses of sequence whoseproperties -
7771
Preface
make them amenable to the application of limit theorems. Nearly all readers will want to study Chapters 12, 13, and the earlier sections of 14 and 15 before going further, whereas Chapters 16 and 17 are rather technical and should probably be avoided until the context of these results is understood. In Parts IV and V we arrive at the study of the limit theorems themselves. The aim has been to contrast alternative ways of approaching these problems, and to present a general collection of results ranging from the elementary to the very general. Chapter 18 is devoted to fundamentals, and everyone should read this before going further. Chapter 19 compares classical techniques for proving laws of large nmbers, depending on the existence of second moments, with more modern methods. Although the concept of convergence in probability is adequate in many econometric applications, proofs of strong consistency of estimators are increasingly popular in the econometrics literature, and techniques for dependent processes are considered in Chapter 20. Uniform stochastic convergence is an essential cohcept in the study of econometric estimators, although it has only recently been systematically researched. Chapter 21 contains a synthesis of results that have appeared in print in the last year or two. Part V contrasts the classical central limit theorems for independent processes with the modern results for martingale differences and general dependent processes. Chapter 22 contains the essentials of weak convergence theory for random variables. The treatment is re%onably complete, although one neglected topic, to which much space is devoted in the probability literature, is convergence to stable laws for sequences with infinite variance. This material has found few applications in econometlics to date, although its omission is anothet judgement that may need to be revised in the future. Chapter 23 describes the classic CLTS for independent processes, and Chapter 24 treats modern techniques for dependent, heterogeneous processes. Pal't V1 deals with the functional central limit theorem and related convergence results, including convergence to limits that can be identified with stochastic integrals. A number of new mathematical challenges are presented by this theory, and readers who wish to tackle it seriously will probably want to go back and apply themselves to Chapters 5 and 6 first. Chapter 26 is both the hardest going and the least essential to subsequent developments. It deals with the theory of weak convergence on metric spaces at a greater level of generality than we strictly need, apd is the one section where topological arguments seriously intrude. Almost certa in ly one shoul go first to Chapter 27 ? referring back as needed for definitions and statements of the prerequisite theorems, and pursue the rationale for these results further only as interest dictates. Chapter 28 is likewise a technical prologue to Chapers 29 and 30, and might be skipped over at first reading. The meat of this pal't of the book is in these last two chapters. Results are given on the multivariate invariance principle for heterogeneous dependent processes, paralleling the central limit theorems of Chapter 24. A number of the results in the text are, to the author's knowledge, new. These include 14.13/14, 19.11, 20.18/19, 20.21, 24.6/7/14, 29.14/29.18, and 30.13/14, although some have now appeared in print elsewhere.
Preface
A37II
Further Reading There is a huge range of texts in print covering the relevant mathematics and probability, but the following titles were, for one reason or another, the most frequently consulted in the course of writing this book. T. M. Apostol's Mathematical Analysis (2nd edition) hits just the right note for the basic bread-andbutter results. For more advanced material, Dieudonn's Foundations ofModern Analysis and Royden's Real Analysis are well-known references, the latter being the more user-friendly although the treatment is often fairly concise. Halmos's classic Measure Fdt?ry and Kingman and Taylor's Introduction to Measure and Probability are worth having access to. Willard's General Topology is a clear and well-organized text to put alongside Kelley's classic of the same name. Halmos's Naive Set Fct??'y is a slim volume whose main content falls outside our sphere of interest, but is a good read in its own right. Strongly recommended is Borowski and Borwein's Collins Reference Dictionaty ofMathematics; one can learn more about mathematics in less time by browsing in this little book, and following up the cross references, than by any other method 1 can think of. For a stimulating introduction to metric spaces see MichaelBarnsley's popularFmc/l/. Everywhere. For further reading on probability, one might begin by browsing the slim volume that stmed the whole thing off, Kolmogorov's Foundations of the F/ldtp?-yof Probability. Then, Billingsley's Probability and Measure is an inspiration, both authoritative and highly readable. Breiman's Probability has a refreshingly informal style, and just the right emphasis. Chung's A Course in Probability The value of the Fct??'y is idiosyncratic in parts, but strongly recommended. Probability and Fwr.y Love's edition) Feller's An Introdclassic texts, (4th uction to Probability Fct??-y and its Applications (3rd edition) is self-evident, although these are dense and detailed books that can take a little time and patience to get into, and are chiefly for reference. Cramr's Mathetnatical Methods of Statistics is now old-fashioned, but still useful. Two more recent titles are Shiryayev's Probability, and R. M. Dudley's tough but stimulating Real Analysis and Probqbility. Of the more specialized monographs on stochastic convergence, the following titles (in order of publiation date) are a11 important: Doob, Stochastic The fzzw- of fzzrpd Numbers', Parthasarathy, Probability Processes; Rvsz, Measures on Metric Spaces', Billingsley, Convergence of Probability Measures', losifescu and Theodorescu, Random Processes and Learning; Ibragimov and Linnik, Independent and Stationaly Sequences of Random Variables', Stout, Almost Sure Convergence; Lukacs, Stochastic Convergence; Hall and Heyde, Martingale Limit F/lct??'yand its Application; Pollard, Convetgence ofstochastic Processes; Eberlein and Taqqu (eds.),Dependence in Probability and Statistics. Doob is the founding father of the subject, and his book its O1d Testament. Of the rest, Billingsley's is the most original and influential. Ibragimov and Linnik's essential monograph is now, alas, hard to obtain. The importance of Hall and Heyde was mentioned above. Pollard's book takes up the weak convergence
Preface
story more or less where Billingsley leaves off, and much of the material complements the coverage of the present volume. The Eberlein-rraqqu collection contains up-to-date accounts of mixing theory, covering some related topics outside the range of the present work. The literature on Brownian motion and stochastic integration is extensive, butKaratzas and Shreve's BrownianMotion and Stochastic Calculus is arecent and comprehensive source forreference, and Kopp's Martingales and Stochastic Integrals was found useful at several points. These items receive an individual mention by virtue of being between hard covers. References to the journal literature will be given in context, but it is worth mentioningthatthe fourpapers by DonaldMctaeish, appearing between 1974 and 1977, form possibly the most influential single contribution to our subject. Finally, titles dealing with applications and related contributions include Serfling, Approximation Theorems ofMathentatical Statistics', White, Asymptotic Theotyfor Econometricians', Gallant, Nonlinear Statistical Methods; Gallant and White, A Un6ed Fwr.y of Estimation and Inference for Nonlinear Dynamic Models, Amemiya, Advanced Econometrics. Al1of these are highly recommended for forming a view of what stochastic limit theory is for, and why it matters.
Acknowledgements The idea for this book originated in 1987, in the course of writing a chapter of mathmatical and statistical prerequisites for a projected textbook of econometric theory. The initial, vel'y tentative draft was completed during a stay at the University of California (San Diego) Depmment of Economics in 1988, whose hospitality is gratefully acknowledged. lt has grown a great deal since then, and competing academic commitments as getting itnishedhasinvolvedas%gglewith well as the demands of family life. My family deserve special thanks for their
forbearance.
My colleaguepeterRobinson
has been agreat source of encouragement and help,
and has commented on various drafts. Other people who have read portions of the manuscript and provided invaluable feedback, not least in pointing out my errors, include Getullio Silveira, Robertde Jong, and especially Elizabeth Boardman, who took immense trouble to help me lick the chapters on mathematics into shape. l am also most grateful to Don Andrews, G raham Brightwell, Sgren Johansen, Donald
McLeish, Peter Phillips, Hal White, and a number of anonymous referees for helpful conversations, comments and advice. None of these people is responsible for the various flaws that doubtless remain. The book was written using the ChiWriter 4 technical word processor, and after conversion to Postscript format was produced as camera-ready copy on a HewlettPackard Laserlet 451 printer, direct from the original iiles.I must particularly thank Cay Horstmann, of Horstmann Software Design Corp., for his technical assistance with this task. Lrndon, June 1994
Mathematical Symbols and Abbreviations In the text, the symbol (:a is used to terminate examples and definitions, and also theorems and lemmas unless the proof follows directly. The symbol w tenninates proofs. References to numbered expressions are enclosed in parentheses. References to numbered theorems, examples etc. are given in bold face. References to chapter sections are preceded by j. In statements of theorems, roman numbers (i), (ii), (iii),...are used to indicate the parts of a multi-part result. Lower case letters (a), (b), (c),... are used to itemize the assumptions or conditions specified in a theorem, and also the components of a detinition. The page numbers below refer to fuller definitions or examples of use, as aPpropriate.
absolute value fw-norm Euclidean norm;
1,i -+ D
&
-, / S, 2 <, <(
>
i 1x(.) a.e.
AR
ARMA
a.s., a.s.gpt1
also fineness (of a q/rtition) weak convergence (of measures); also implication monotone convergence convergence almost sure convergence convergence in distribution convergence in Lp nol'm convergence in probability mapping, function composition of mappings equivalence in order of magnitude (of sequencesl; also equiyalence in distribution (of r.v.s) addition modulo 1 set difference partial ordering, inequality strict ordering, strict inequality order of magnitude inequality (sequencesl; also absolutely continuous (measures) mutually singular (of measures) indicator function almost everywhere autoregressive process autoregressive moving average process almost surely, (withresp. to p.m. g)
Symbols AC
I (A)40, Gm
dO V Bnbp) CLT
ch-f. c-d.f. C'p,1j C, Q c D
i Z (n4 dxny) ogc,jy D A :A 6
StlP E.4 E. Ix) E. 1V) eSS
H j+
J-
1
FCLT F(.)
9x(.) 9,,, iff inf
i-i-d. i-o. in pr.
LIE LII
,
lim limsup, lim liminf, 1im Ln) AP-NED MA
m.4
complement of A closure of A intedor of A strong mixing coefficient aleph-nought (cardinalityof (N) dfor every' Binomial distribution with parameters n and p Borel field central limit theorem characteristic function cumulative distribution function continuous functions on the unit interval
set containment strict containment chi-squared distribution with n degreeg of freedom distance between x and y cadlag functions on the unit interval dyadic rationals symmetric difference boundary of A set membership essential supremum expectation conditional expectation (on variable x) conditional expectation (on c-field 5')
Lebesgue measure
3
Symbols m.d. m.g.f. m.s.
martingale difference moment-generating function mean square space of measures
N(jt,c2)
Gaussian distribution with mean jt and variance
s(N()
natural numbers
rh, O
m h #!
0(.) t?(.) Op.) ov.4 p-d.f. P.m.
#(.)
#(.IA)
#(. IN)
H,
1-1
7rj(.)
(D
F.V.
/?(0,1j .f R y R R+
F F+
R& S.e. S.S.e.
SLLN suP Sxvz)
!
Jn
c(C)
c(m fdx
K QJ
(0l
intersection minimum of m and n fBig Oh' order of magnitude relation ,
Oh', strict order of magnitude relation stochastic order relation strict stochastic order relation null set probability density function Ktaittle
probability measure probability conditional probability (on event A) conditional probability (on c-field N) product of numbers', also partition of an interval product measure coordinate projection rational numbers random variable real valued functions on g0,11 relation real line non-negative reals extended real line, R t..p (-x,+x1
R+kJ
f+x)
n-dimensional Euclidean space stochastic equicontinuity strong stochastic equicontinuity strong law of large numbers suptemum e-neighbourhood, sphere sum of random variables variance of Sn c-tield generated by collection C c-field generated by r.v. X
J/Wpo fdF
Lebesgue integral Lebesgue-stieltjes integral
fdp z, E
expected value (integralwith resp. to p.m.) sum
ui
(y2
Symbols
Mlt
shift transformation union unifonn distribution on interval (tz,:4 union of c-fields maximum of m and n weak 1aw of lm'ge numbers with probability 1 sample mean of sequence (XrJZ
F
U
kJ,
Uka,bj
w, V
m V
n
MXLN
sv.p.1
Xn
x, X
Cartesian product
Z
ll l 1T,( ) .
'>O
.
.
ll ll (x1 I,:(l a,b) .
(f1,T)
(f,',p,)
(f1,F,F) (f,T;#) (S,* (X,*
c-field of product sets integers set designation', also sequence, array infinite sequences array largest integer S closed interval bounded by a,b open interval bounded by a,b measurable space measure space complete measure space probability space metric space topological space .t
Common usages A,#,C,D,... X,F,Z,... A',F.Z,...
J,#,,... :,,q
B,5'.I
2,C,O,F',.. 9,V,J't,-.. S,5-,X,... 1,,Y,-
-
d.p T
-
sets random variables random vectors functions positive constants bounding constants collections of subsets c-fields spaces measures metrics topology
1
MATHEMATICS
Sets and Numbers
1. 1 Basic Set Theory A set is any specified collection of objects. ln this book the objects in question are often numbers, but they may also be functions, or other sets, or indeed wholly arbitrary, to be determined by the context in which the theory is applied. ln any analysis there is a set which defines the universe of discourse, containing al1 the objects under consideration, and in what follows, sets denoted A, B etc., are subsets of a set X, with generic element x. q A meaning Set membership is denoted by the symbol belongs to the set writes and A #. The usual way B have the same elements, one A' To show sets A descriptive enclosed define is by in braces, so that for statement a to a set example A tx:x e #) defines membership of A in terms of membership of B, and is an alternative way of writingA #. Another way to denote set membership is by labels. If a set has n elements one can write A fxj, i 1,...,n 1, but any set of labels will do. The statement A lau, (x e C) says that A is the set of elements bearing labels a contained in another set C, called the index set for A. The labels (indices)need not be numbers, and can be any convenient objects at tends to be preferred in all. Sets whose elements are sets (theword this context) are denoted by upper-case script characters. A e C denotes that the set A is in the collection C, or using indices one could write C (Aa: a e Cl B is called a subset of A, written B c A, if al1 the elements of B are also elements of A. If B is a proper subset of A, ruling out B A, the relation is written B c A. The union of A and B is the set whose elements belong to either or both sets, written A k.p #. The union of a collection C, the set of elements belongr ing to one or more A e C, is denoted UA es,A, or, alternatively, one can write UaecAz for the union of the collection fAa: (x l C). The intersection of A and B is the set of elements belonging to both, written A rn #. The intersection of a collection C is the set of elements commop to a1l the sets in C, written f-tlsc/t or OascAa. ln pmicular, the union and intersection of (Aj ,A2, Au ) are written U7=1Ajand O7=1Aj.When the index set is implicit or unspecified we may write just U(xAa, OfAf or similar. The dterence of sets A and #, written A B or by some authors A #, is the of two sets set of elements belonging to A but not to B. The symmetric tft//rcrcc (A B) k.p (B -A). X-A is the complement of A in X, also denoted Ac is A A# when X is understood, and we have the general result that A B 4 rn Bc. The null set (or empty set) is O M, the set with no elements. Sets with no elements in common (havingempty intersection) are called disjoint. A partition of a set is a 61
'
<x
.x
,
=
.
=
=
=
=
=
dcollection'
=
.
=
...,
-
=
-
-
=
=
Mathematics
4
collection of disjoint subsets whose union is the set, such that each of its elements belongs to one and only one member of the collection. Here are the basic rules of set algebra. Unions and intersections obey commutative, associative and distributive laws: A t.p B
=
B QJ A,
A f'n B
=
B rn A,
(A t..p B4 tp C
=
A t.p (B
(A fa #) rn C
=
A f-h CBf'A C),
(1.4)
=
(A fa B4 tp (A rn C),
(1.5)
A fa
(# tp C)
?
C),
(A tp #) r'h(A kp C. There are also rules relating to complements known as de Morgan's laws: A tp
(# fa
C)
=
(A tp B4c
=
(A rhB)c
=
AC
fa
Bc
Ac t.p Bc.
(1.8)
Venn diagrams, illustrated in Fig. 1.1, are a useful device for clarifying relationships between subsets.
Fig. The distributive and de Morgan laws extend to general collections,
1.1 Theorem Let C be a collection of sets, and B a set. Then
(i) (ii)
UA
rn B
OA
tp B
=
AGC
U (A
=
AGC
f-lzt
AeC
#),
r)
(A kp B),
dgC C
(iii)
f-h
AGN
zc
(-Jic,
AEC
as follows.
Sets and Numbers C
(iv)
U
A
OAc. u
=
AeC
AeC
The Cartesian product of two sets A and #, written A x B, is the set of a1l possible ordered pairs of elements, the first taken from A and the second from #; we write A x B (@,y): e A, y G #) For a collection of n sets the Cartesian product is the set of al1 the n-tuples (orderedsets of n elements, with the th element drawn from A/), and is written .:'
=
.
n
Af
.(
=
(x1x2, ,
/=1
...,
xi e Af
Ak):
,
1-
1,...,n )
=
(1.9)
.
If one of the factor sets Af is empty, X2=1Xfis also empty. Product sets are important in a variety of different contexts in mathematics. Some of these are readily appreciated', for example, sets whose elements are s-vectors of real numbers are products of copies of the real line (see j1.3). But prodct sets are also central to the mathematical formalization of the notion of relationship between set elements. Thus: a relation R on a set A is any subset of A xA. If @,y)e #, we usually write x#y. R is said to be reflexive iff xRx, symmetric iff xR y implies yRx, antisymmetric iff xRy and y Rx implies x y, transitive iff xRy and yRz implies xR z, each in where case the indicated condition holds for every x, y, and z e A, as the and only if means case may be. (Note: relation relation equivalence that is reflexive, symmetric, and transiis An a tive. Given an equivalence relation R on A, the equivalence class of an element x e A is the set Ex (y e A: xRy If Ex and Ey are the equivalence classes of elements x and y, then either Ex fa Ey 0, or Ex Fy. The equality relation y is the obvious example of an equivalence relation, but by no means the only one. A partial ordering is any relation that is reflexive, antisymmetric, and transitive. Partial orderings are usually denoted by the symbols f or k, with the understanding that k y is the same as y f x. To every partial ordering there corresponds a strict ordering, defined by the omission of the elements @,x)for a11 x e A. Strict orderings, usually denoted b: < or >, are not reflexive or antisymmetric, but thy are transitive. A set A is said to be linearly ordered by a partial ordering f if one of the relations x < y, x > y, and x y hold forr every pair @,y)e A xA. If there exist elements a e A and b G A such that a < x for al1 x e A, or x S b for all x e A, a and b are called respectively the smallest and largest elements of A. A linearly ordered set A is called well-orderd if every subset of A contains a smallest element. It is of course in sets whose elements are numbers thai the ordering concept is most familiar. Copsider two sets X and F, which can be thought of as representing the universal sets fr a pqir of related problems. The following bundle of definitions contains =
Kiff'
=
.)
.
=
.x
=
.x
=
=
Mathematcs
b
the basic ideas about relationships between the elements of such sets. A (or transformation or function) T: X
F->
tnapping
F
is a rule that associates each element of X with a unique element of F; in other words, for each a7 e X there exists a specified element y e F, denoted F@). X is called the domain of the mapping, and F the codomain. The set Gw
(@,y):
e X, y
.z'
=
is called the graph of T. For T(4)
is called the image of
,4
:4
e /t
Jq
under F, and for B
F -1(#)
=
(1.10)
Xx F
the set
? X,
(F@): x
=
W@))
=
fx: F(x)
e
(1 11)
F
.
1', the set
c
#J c
X
(1.12)
is called the inverse rrlzwe of B under F. The set T(m is called the range of F, and if F(m 1,the mapping is said to be from X onto F, and otherwise, into K If each y is the image of one and only one x e X, so that F(x1) Tx if and only the mapping is said to be one-to-one, or 1-1. if xl =
=
=
.n,
The notions of mapping and graph are really interchangeable, and it is permissable to say that the graph is the mapping, but it is convenient to keep a distinction in mind between the rule and the subset of Xx F which it generates. The term function is usually reserved for cases when the codomain is the set of real numbers (seej1.3). The term correspondence is used for a nlle connecting -1 elements of X to elements of F where the latter are not necessarily unique. T is a con'espondence, but not a mapping unless F is one-to-one. However, the term one-toone correspondence is often used specitically, in certain contexts that will arise below, to refer to a mapping that is both 1. 1 and onto. lf partial orderings are defined on both X and F, a mapping is called order-preserving if F(x1) G Tx iff On the other hand, if X is partially ordered by f, a 1- 1 mapping induces .;q S
.
1.2 Theorem (i) For a collection (Aa c aY), F UAa (:
(ii) for a collection (#a i F), T-' (iii) for #
i', F -1(sc) .
UF(Xa);
=
'
Ufa
=
UF-1(#a);
w-1(s)c. ,
(iv) for A i X, A g F -1(z(u)). #. u (v) for B g F, F(F -1(#)) ,
Here, T -1(B)c means X- F-1(#). Using de Morgan's laws, properties
(ii) and
Sets and Numbers
(iii) are easily extended to the inverse images of intersections and differences', for example, we may show that the inverse images of disjoint sets are also F(A)C F(AC), # in general. Pal'ts (iv) and (v) are disjoint. However, F- F(A) illustrated in Fig. 1.2, where X and i' both correspond to the real line, A and B are intervals of the line, and F is a function of a real variable. =
Fig. 1.2
When F is a 1-1 con-espon dene (1-1 and onto) so is F-1 These properties then hold symmetrically, and the inclusion relations of parts (iv)and (v) also become X and B c K lf Z is a third set, and equalities for a1l A .
U1 F
F--> Z
is a further mapping, UoF: X
F--
the composite
mapping
Z
takes each x e X to the element U(F(x)) e Z. UoT operates as a simple transform-1 from X to Z, and 1.2 applies to this case. For C c Z, (&J07) c' ation -1 =
-1
F
(&
(C)).
0- x E .
j
i
.
.
.
.
.
'
.
' .
.
' .
' .
' .
.
j
F(z4c )
1
+
'
.
)Y
X1Z
Fig. 1.3
1.3 Example lf X 0- x E is a product space, having as elements the ordered pairs .x = (0,(), the mapping F: 0- x E i- E =
,
Mathematics
8
defined by F(0,() (, is called the projection tnapping onto E. The projection of 0- x E onto E (respectively,0-) is the set consisting of the second a set A (resp., first) members of each pair in A. On the other hand, for a set B e E, 0- x # It is a useful exercise to verify 1.2 for this case, and also to F -1 B4 T(AC) in general. In Fig. 1.3, 0- and E are line segments and check that F(A # F(A)C is the union of the indicated line 0- x E is a rectangle in the plane. Here, whereas T(A9 E. u segments, =
=
.
=
The number of elements contained in a set is called the cardinality or cardinal in this context is not a primitive one, number of the set. The notion of but can be reduced to fundamentals by what is called the principle. A set A is said to be equipotent with a set B if there exists a 1-1 correspondence connectingA and #. Think in terms of taking an element from each set and placing the pair in a pigeon-hole. Equipotency means that such a procedure can never exhust one set before the other. Now, think of the number 0 as being just a name for the null set, 0. Let the number 1 be the name for the set that has a single element, the number 0. Let 2 denote the set whoseelements are the numbers 0 and 1.And proceeding recursively, 1et n be the name for the set (0,...,n 1 1. Then, the statement that a set A has n elements, or has cardinal number n, can be interpreted to mean that A is equipotent with the set n. The set of natural numbers, denoted N, is the collection (n: n = 1,2,3,... 1. This collection is well ordered by the relation usually denoted <, where n S m actually means the same as n m under this definition of a number. %number'
dpigeon-hole'
-
1.2 Countable Sets Set theory is trivial when th: number of elements in the set is finite, but formalization becomes indispensable for dealing with sets having an intinite number of elements. The set of natural numbers (Nis a case in point. If n is a member so is n + 1, and this is true for every n. None the less, a cardinal number is formally assigned to iN, and is represented by the symbol Mtl When the elements of an infinite set can be put into a one-to-one correspondence with the natural numbers, the set is said to have cardinal number d(), but, more commonly, to be countable, or equivalently, denumerable. Countability of a set requires that a scheme can be devised for labelling each element with a unique element of (N.This imposes a well-ordering on the elements, such that there is a 'first' element labelled 1, and so on, although this ordering may have significance or be arbitrary, depending on the circumstances. It is the pigeon-hole principle that matters here, that each element has its own unique label. With infinite sets, everyday notions of size and quantity tend to break down. Augmenting the natural numbers by the number 0 defines the set (Ntl (0,1,2,3,... 1. The commonplace observation that (Ntlhas more' element than ENis contradicted by the fact that INand (Nnare equipotent (labeln 1 e Ntlby n e IN).Still more suprisingly, the set of even numbers, E = (2n, n e IN) also has an obvious labelling scheme demonstrating equipotency with EN.The nave idea that there are (Kaleph-nought').
=
-
,
df.xxzlr.o
eac
meanxz'
xlomontc
ln
N
re
1n IF- ;t
Axzithnllf
lfatyinnl fnllnclntlnn
Sets and Numbers
9
infinite subset A of ENhas a natural well-ordering and is equipotent with EN itself, the label of an element x e A being the cardinal number of the set (y e A:
A' V A l than EN consider the integers, I Turning to sets apparently 1,2,... ), the set containing the signed whole numbers and zero. These ( are linearly ordered although not well ordered. They can, however, be paired with scheme: the natural numbers using the .
Elarger'
=
,
...,-1,0,
Ezig-zag'
(1,0),(2,1),(3,-1), (4,2),...,(n, (n/21(-1)''),...,
where gxJdenotes the largest whole number below x. Thus, N and I are equipotent.
Then there are the rational (I)
=
f.x: .x
numbers.
=
alb, a e 1, b e 1, b # 0)
(1.13)
.
We can also show the following.
1.4 Theorem Q is a countable set. Proof We construct a 1-l correspondence between I x J and N. A 1-l correspondence between I x I and I x: is obtained by the methodjust used to show I countable, and one between I x: and ENx: is got by the same method. Then note that the number 2 3b e INis uniquely associated with each pair a,b4 e INxN ; the rule for recovering a and b from 2'3b is g et a as the number of divisions by 2 required to get an odd number, and the number so obtained is 3 ' The collection f 2 '3b : a e ENb G EN) c ENis equipotent with ENitself as shown in the preceding paragraph. The composition of a1l these mappings is the desired correspondence. w <
.
,
,
Generalizing this type of argument leads to the following fundamental result.
1.5 Theorem The union of a countable collection of countable sets is a countable set. n The concept of a sequence is fundamental to a11 the topics in this book. A sequence can be thought of as a mapping whose domain is a wellrordered countable set, the index set. Sinc there is always an order-preserving 1-1 mapping from IN to the index set, there is usually no loss of generality in considering the composite mapping and thinking of N itself as the domain. Another way to characterize a sequence is as the graph of the mapping, that is, a countable collection of pairs having the ordering conferred on it by the elements of the domain. The ranges of the sequences we consider below typically contain either sets or real numbers', the associated theory for these cases is to be found respectively in j1.4 and j2.2. The term sequence may also be applied to mappings having I or another linearly ordered set as index set. This usage broadens the notion, since while such sets can be re-indexed by N (see above) this cannot be done while preserving the original ordering.
Mathematics
10 1.3 The Real Continuum
The real-number continuum R is such a complex object that no single statement of definition can do itjustice. One can emphasize the ordinal and arithmetic properties of the reals, or their geometrical interpretation as the distances of points on a line from the origin (the point zero). But from a set-theoretic point of view, the essentials are captured by defining R as the set of countably infinite sequences of decimal digits, having a decimal point inserted at exactly one position in the sequence, and possibly preceded by a minus sign. Thus, the real number x can be written in the form X
'x
,,,(x)1(/)7-)j(.x)10-f
=
(1. 14)
i=1
where the sequence (J1(.x), J2@),...) consists of decimal digits (elementsof the set (0,1,...,9 J), px) e (Ntldenotes the position of the decimal point in the +1 if k 0, and otherwise (the string (the decimal exponent), and mlx) all of the decimal expansion sign). When dix) 0 for but a finite number terms, of x is said to terminate and the final Os are conventionally omitted from the representation. -1
.x
=
=
by (1.14) is not always unique, and there exists a 1-1 correspondence between elements of R and sequences (>,#,#1,#2,#a,... l only after eliminate arbitrary of leading zeros we must the latter are excluded. To certain 0. And since for example, 0.49999... (the stipulate that dj y: 0 unless p sequence of 9s not terminating) is the same number as 0.5, we always take the terminating representation of a number and exclude sequences having di 9 in al1 but a finite number of places. R is of course linearly ordered, and in terms of (1. 14) the ordering corresponds to the lexicographic ordering of the sequences The representation
of
.x
=
=
fmsmpnmdk,mdznmdL...l.
The choice of base 10 in the definition is of course merely conventional, and of the alternative possibilities, the most important is the binary (base2) represn-
tation,
x
=
>(x)
2d*J>7n(x)2-i f=l
,
(1.15)
where the bi are binary digits, either 0 or 1, and qx) is the binary exponent.
The integers have the representation in (1.14) with the strings terminating elements rationals of R, being those which either after px) digits. The are also terminate after a finite number of places, or else cycle repeatedly through a finite sequence of digits beyond a certain tinite point. The real numbers that are not rational are called irrational. The irrational numbers are overwhelmingly more numerous than the rationals, representing a higher order of infinity. The following is the famous argument of Cantor. ddiagonal'
1.6 Theorem The jet R is uncountable. Proof Assume a 1-1 correspondence between R and ENexists. Now construct a real
Sets and Numbers
number in the following way. Let the first digit be different from that of the real number labelled 1, the second digit be different from that of the real number labelled 2, and in general the nth digit be different from that of the real number labelled n, for every n. This number is different from every member of the labelled collection, and hence it has no label. Since this construction can be performed for any labelling scheme, the assumption is contradicted. w c, where c is the cardinal
number
of R The linear ordering on R is of interest to us chiefly for providing the basis for constnlcting the fundamental subsets of R, the intervals. The set A (.x:a < x < &) is called an open interval, since it does not contain the end points, whereas the interval # = (x: a f x S /71is said to be closed. Common notations are gJ,pj, asb), (tz,:j and Lanb) to denote closed, open and half-open intervals. A set containing just a single point a is called a singleton, written ftz) Unbounded defined by a single boundary point, are written intervals such as C = (x: a < (J,+x), (-x,:), and (J,+x), (-x,:j for the open and closed cases respectively, where the infinities +x and are the fictional (notelements of R) with > +x < for al1 R respective and An important example properties x the xe is the positive half-line (0,+x), which we will denote subsequently by R+.
We say that d ()
<
.
=
.
.x1,
Kpoints'
-=
.r
-x,
.
1.7 Theorem Every open intelwal is uncountable. Proof Let the interval in question be anb). lf a < b, there exists n k 0 such that the n + llth term of the sequence msmpntndjsmdzn...) in the expansion of (1.14) defining b exceeds that in the corresponding sequence for a, whereas the t'irst n digits of each sequence are the same. The elements of a,b4 are those reals whose expansions generate the same initial sequence, with the (n+ llth terms not exceeding that of b nor being exceeded by that of a. If a and b are distinct, n is finite. The result follows on applying the diagonal argument in 1.6 tb these expansions, beginning at position n + 2. w Other useful results concerning
R and its intervals include the follpwing.
1.8 Theorem The points of any open interval are equipotent
with R
.
Proof This might be proved by elaborating the argument of 1.7, but it is simpler just to exhibit a 1-1 mapping from R onto asb). For example, the function
y
=
a+ b
2
for x e R fulfils the requirement. 1 9 Theorem The reai plane R2 *
Proof In
+
(b
2(1
-
+
tzlx
(1.16)
IxI) .+
. =
R x R is equipotent with R
,
view of the last theorem, it suffices to show that the unit interval o Givep points x, y e g0,1J, define 1q (0, is equipotent with the unit square (0,11. the point z e (0,1j according to the decimal expansion in (1.14),by the rule
Mathematics
diz)
di-vblllx),i odd =
Jf/2@),
i
1,2,3,...
=
(1.17)
i even
In words, construct z by taking a digit from x and y alternately. Such a z exists for every pair x and y, and, given z, x and y can be uniquely recovered by setting
(1.18) This defines a 1-1 mapping from (0,112 onto gO 1j as required. w ,
,
This argument
can be extended from R 2 to Rn for any n e N.
1.10 Theorem
Every open intelwal contains a rational number.
,
Proof This is equivalent to the proposition that if x < y, there exists with x < r < y. First suppose x 0. Choose q as the smallest integer such that qy > qx + 1, and choose p as the smallest integer 1/(v qy. Then x < (,p 1)/ < y. For the case < 0 choose an integer n > < r < n +y, .x < r- n < y, where r is the rational satisfying n -x),
-x,
.x
-
.-x
above.
rational r
exceeding
exceeding and then found as
.
1.11 Corollary
Every collection
of disjoint open intervals is countable.
Proof Since each open interval contains a rational appearing in no other interval disjoint with it, a set of disjoint open intervals can be placed in 1-1 correspondence with a subset of the rationals. w The supremum of a set A c R, when it exists, is the smallest number y such that x S y for every x e A, written supA. The inpmum of A, when it exists, is the largest number y such that k y for every x e A, written inf A. These may or may a, and supgl,') not be elements of A. In particular, infgtz,hj inftc,#) suptc,:) b. Open intervals do notpossess largest or smallest elements. However, every subset of R which is bounded above (resp.below) has a supremum (resp. intimum). While unbounded sets in R lack suprema and/or infima, itis customary to called the extended real line. In V, every set has define the set V R t.p ( number or +x, and similarly, every set has an either real a tinite a supremum, infimum. The notation 2-+will also be used subsequently to denote R+ k.p (+x). .x
=
=
=
=
-x,+x),
=
l.4 Sequences of Sets will be written, variously, as (An'. n e IN), (A,,)7, Set sequences fA1,A2,A3, or just fAnl when the context is clear. with each member of A monotone sequence is one that is either non-decreasing, n), being contained in its (An the sequence successor c An+l, V or non-increasing, with each member containing its successor (An+1i An, V n4. We also speak of increasing (resp. decreasing) sequences when the inclusion is strict, with c (resp. D) replacing c (resp.Q). For a non-decreasing sequence, define the set A ...)
=
U';=1A,,and for a non-increasing
sequence the set A
=
fV=IA,;
=
(U';=1Aj)f. These
Sets and Numbers
13 '1'
sets are called the limits of the respective sequences. We may write An A and d,, A. 1 A, and also, in general, limrl-yx,As zt and A,, --->
=
1.12 Example The sequence ((0, 1/n1, n e (Nl is decreasing and has as limit the singleton f0). In fact, limn-+x(0,1/n) = (0) also, whereas 1im,,-4x(0,1/n) 0. The decreasing sequence of open intervals, ((tz 1/n, b + 1/n), n e EN), has as its (J, :1. On the other hand, the sequence ((tz+ 1/n, limit the closed interval its limit is b4. a, a b 1/nj, n e IN) is increasing, and =
-
-
is non-increasConsider an arbitrary sequence (zt,,1.The sequence Bn U,a'='=,#4,, = exists. superior is called limit of the This set the ing, so that B limn-oon (Anl, written limsupnAn, and also as limnAa. Similarly, the limit of sequence the non-decreasing sequence Cn F1e;=,,A,,,is called the inferior limit of the written liminfndn, or limnAr,. Formally: for a sequence (A,,, n e EN ), =
=
sequence,
oo
limsupz
O
=
n=1
n
liminfAn
U n=1
=
Cr
UA,s
(1.19)
m=n
OA,u1 .
m=n
/
(1.20)
The limsup is the set of De Morgan's laws imply that liminfnAn = (limsuwzvlf. elementscontained in infinitely rrltwz of the An, while the liminf is the set belongingto all but a Jl/c number of the An, that is, to every member of the sequence from some point onwards. These concepts provide us with a criterion for convergence of set sequences in genergl. LiminfnAs ? limsupnAn, and if these two sets differ, there are elements that belong to infinitely many of the An, but also do not belong to infinitely many of them. Such a sequence is not convergent. On the other hand, if liminfuA,l = limsupnAn A, the elements of A belong to infinitely many of the An and do not belong to at most a finite number of them. Then the sequence (An) is said to converge to A, and A is called the limit of the sequence. =
1.5 Classes of Subsets The set of a11 the subsets of X is called the power set of X, dnoted 2X The power set of a set with n elements has 2 elements, which accounts for its name and representation. ln the case of a countable set, the power set is thought of fprmally as having 2 20 e le ments. One of the fundamental facts of set theory is tht the number of subsets of a given set strictly exceeds the number of its elments. For finite sets this is obvious, but when extended to countable sets it M to the claim that 2 g > do. ?mntj .
n
1 13 Theorem
.'
')'. ()' td
'' -'
.
Prof '
. . .
.. .
q
E
..'
..'
.
.(
;
))t () it)(lit'(. . . . . '.. .:
'
.
c.
.
.
.
' '
' .
,'
i
(..
=
The proposition is proved if we can show that 2 is equipotent with R or, qivalently (in view of 1.8), with the unit interval (0,1j. For a set A G 2 x (.2.1liitsftqt . th sequence of binat'y digits (:1,hz,,... ) according to the rule,
E . ..
.
2B0
:
. . . ... .. .. . .
-.
.
,
Lbn
Mathematics
14
= 1 if n e 4, bn 0 otherwise'. Using formula (1.15)with m 1 ad q 0, let this sequence detine an element XA of (0,1)(the case where bn 1 for a1l n defines 1). On the other hand, for any element x e (0,1q, construct the set Ax G 2N according to the rule, n in Ax if and only if the nth digit in the binal'y expansion of is a 1'. These constructions define a 1-1 correspondence between 2 ENan d (0 11 w =
=
=
=
'include
.x
,
.
When studying the subsets of a given set, particularly their measure-theoretic properties, the power set is often too big for anything very interesting or useful to be said about it. The idea behind the following detinitions is to specify subX sets of 2 t h a t are large enough to be interesting, but whose characteristics may be more tractable. We typically do this by choosing a base collection of sets with known properties, and then specifying certain operations for creating new sets from existing ones. These operations permit an interesting diversity of class members to be generated, but important properties of the sets may be deduced from those of the base collection, as the following examples show.
1.14 Definition A ring S is a nonempty class of subsets of X satisfying (a) O q S. (b) lf A and B e R then A wB e S, A (AB G R and A B e X. n -
One generates a ring by specifying an arbitrary basic collection C, which must include 0, and then declaring that any sets that can be generated by the specified operations also belong to the class. A ring is said to be closed under the operations of union, intersection and difference. Rings lack a crucial piece of structure, for there is no requirement fclr the set X itself to be a member. lf X is included, a ring becomes aheld, or synonymously AC, this amounts to including all complements, and, in an algebra. Since X-A view of the de Morgan laws, specifying the inclusion of intersections and differences becomes redundant. =
1.15 Dennition A tield 5 is a class of subsets of X satisfying (a) X e 5. (b) lf A 6 5 then Ac 6 T. (c) If A and B G 5 then A t..p # e @. n A field is said to be closed under complementation and finite union, and hence under intersections and differences too; none of these operations can take one outside the class. These classes can be very complex, and also very trivial. The simplest case of a ring is (OJ. The smallest possible t'ield is tX,O). Scarcely less trivial is the AC, O ) where A is any subset of X. What makes any class of sets field (-Y,A, interesting, or not, is the collection C of sets it is declared to contain, which for the class. We speak of the smallest tield we can think of as the generated by C'. field containing C as Rings and fields are natural classes in the sense of being defined in terms of the simple set operations, but their structure is rather restrictive for some of '
,
tseed'
tthe
Sets and Numbers
15
the applications in probability. More inclusive definitions, carefully tailored to include some important cases, are as follows. ,/
is a non-empty class of subsets of X satisfying 1.16 Definition A semi-ring (a) O e $. then A /'n B e (b) If A, B e where (c) If A, B e and A i #, H n < cxa such that B A U7=1G', e = f''h for each j'. O j, n and .f.
,/
,/
./
=
-
.
.'
.
More succinctly, condition partition into '-sets.
that the difference of two '-sets has a finite
(c) says ,/
is a class of subsets 1.17 Dennition A semi-algebra (a) X e (b) If A, B e then A chB e (c) If A e $, 3 n < x such that Ac Uyzlcj,where for each j, j'. a
of X satisfying
.f.
.l'.
,/
=
.
e
.t/?
and C) ch Cj'
O
=
A semi-ring containing X is a semi-algebra. R, and consider the class of all the half-open intervals and Iz < together with the empty set. If h (J1,:1) for I (tz,!?J (J2,:1q, f-'h and 0. And if fl = (tza,:2),then fl h is. one of h, h, (tz1,:2), c 11 (tz2,J11, (:1,:2j, so that a) f tz1 and bj f b1, then h h is one of 3, semi-ritzg conditions defining therefore and The sat(az,atj t.p (:1,&2J, h. a are isfied, although not those detining a ring. If we now let R be a member of the class and follow 1.17, we find that the halfand (tz,+x),plus open iniervals, plus the unbounded intervals of the form In O artd R, constitute a semi-algebra.
1-18 Example
Let X
=
=
a S b
<
-n
+C,o,
=
-
-xtbl
1.6 ji gTa Fields .
. .. .
.
.
Whn we' say tht a field contains the complements ;;!,' t!)'--,$@. ,;' y' .j' r' ;'(k,-':' ..,' iip'.l!:d @'. j,' jr'r' (@.-)--81'* :)'. ):'. '''#' r', jjli'. yrd jd r'. ;d 11,;::2:r11.-*. :-'. i'-' t:'.-'-ll)ll'-' qrrrdkiy--i---.d-,jtls'ltdkdi.lhlikirrstipp.:fl.rd pFq'. yy.;'. ,;r'j-. y' yjd ()..' jt'. ijd k' j'yy.'-r)'.' 133*.* j'i'.i (,'. ;);.;-' ;'@@' r' (y'. j.' )' i@'. j..' t'' )'()' y'; g' q'd ('. (';' kt'qi'. ''Ikiii!r:-'. ((' t,,(,t'' ?' L' ..' yjjjjjyryjjj.,,d :::*-2)1-.* Ft@-'. !!)p'.(' py'i'l'g )tI-'. ytd yy..,';.-: y;j-yy;',. (';' .d 4j' ('. .LLLq'. .:!1211-----*:11... :I.EIII.-.CEII-.'. 'Iii!'.k.d ''Ikii:rr:-d q!IkE7'r!'k..'. .-'Iki:t:i;'.. 'q'IlkEjIIk,'. (.'. ,''-r.-'II::7!IIk... I'EEEII.'. ijyipl,.dtsil----'k-ljptjs..!rrl ,-jb'L)j-..L'tjLkL33,?.,.q-',i-..-j,qk.. t(' .y)'. tjd t'j,'t)' .'')'''';:E:!!.,-. ''k::E:!:.' k.-'-'''Ill::iEl1I-..t.''-.tt'$'12ii!I.-.'. rpr'y r' .-'. r'.' .-' '1I(ii''' (!11:::1,-* 4111E::::,..*..':!1L.*. it:ipd rjqj-dltdllt.-l'ttd desyrves explanation. It is clear that z4l
and finite unions, the qualifier A,, e 5 implies that U& j=3 A) .#hit lteratlon of slmple by n-fold 5 But, the constructlv: palrwlse unlon. e glven t t: (j jm. tj (jj'I Out W 1* legiti 1 k. tju further 1 mate ttlr th t ti t no e e s on q':j).: .) . ' a s P ul ati O n to r o t? 41@.;: )' .(ir.. ytt,rtlh be taken llmlt. a.,g, the By maklng n.operatlon can thls addltlonal to ))(, j).jt (..... .. . . . 'lk::Ei!lk.. @ E..t!1IIp:,I.i.. 'r '. E. !!I1L., :111:7:!1.-. ' ' ' :..-t.'E!IIk..' . .-'.EEE:!!:. ..,.jy-...'- ..:..'.. '. ' .. . :111:7:!1.-.. r,,;-. Jl'... ' ' ' '..-.. .' ' ''. . rr ... '.r.-'...(. . . .'.'i.7tr. i'.' ..luju,-..t9'.j:;:,.j)..jr, r.... f.tltz... jyy;;.jjy..... jj.lyytjr .. .. ...... .. . ;.g,..yt:jj,j.. . ..*1.: j., A .'.j'tp:('.()) r yjj''.'gt). (tsajgebra)5 ls a class Of subsets Of X satlsfylng ('. f l ...;...: uljjjtl .......,..y 't;. r;J) .j . j ?)...j!.y. -y..F..:yqqy t)?.p:. (y .kly(..yy.yq..:)...q ):;1211E11.... .!kjjj;;r;;)., .. ' ..
.' .'.' ':jls')d .'. '-' -j--!,r,!tk'. -:'. --y!j;;'-j-.' -q' -,.'tp..-.(t. -'k' -(r)'. -8)* y' :')' 24jg* -'. -' j;d ('. !111:(q*;LL.' ;..(' i.'. k' .-'. -'-' --'' -,' j'y' -:tr!)i...ti:,jp':F'--)?;t--''' E' ,-.'itt-,ttd ,'-'.k'(.E ;E-;r'.' ?' y' lrrq))d :':11,*. :L';'' ''. g')j;-',' j';'y' y't':. i--d r' s.d ,'g'' E'. )-''E -;.' ''IItEL.FX -' -,'.-y' -'. .)' .q' ''. k' .:' ,''(''jy ;'))'. ,'..(.. (t' ?.' :y'-' ;;' yyd ((' ;'. :('. F'-j. !': q' (j.,' F' C--qy;'E :'-'E' :)')' L3--'; :;' ''-.' ;,'-'L'' ,-'' )'
,-,' -' .','.:'.. -' .','.-tq'-' -,-' .)r,' ,--))'-' ,,,' 'jjjj:yr:jjj,'; .'.'. ''.-'' .!112::*. .'. ,'.'-'. 4:11::.* '''bii'''bi''-''
'.'. .''''--d .'.' .' . .'r:).. ... ''.' '..'. '. ..' . .' . .
'.
:
...
.
.
....
. E- .
(. '
... .
.
.. ' g' '
.
q
(
.'' . .' '( . . EE
...
.
. .. . . .. . . (j;) :.. . . . :
.
.
E
t
,np
E
.
:
.'
.
'!;'j .L'
'. 7 .!L(.
...;
j, ....j .; ,4k .:f .(a z ( .
.
.
. -
-.
..
....'
j.(
...:;.. .
...
...
-
.
(
.
....
'
.'
.. .
g
...
.
7
.
..
.. .
'
.
.
..
..i
..k
. .
.
.:.
.
..
. . ' .
.
.
.
..
dp
4)
..
-11::::::,11;-.
.11:2:::,...
,11::2::,1.-
.
:.. .
.
:
,
. .
y.
.
.
......'
'lk:!r:-.
,y
..
-
. .. .,k;)jjjtyy........
(lzzllp'l. ' (jj:jjj.. (j;jj..,...
.
y.
.
.
.
.
,,
.
,,
,,
7.
u.....r.
:..,. . .: . ... . ;. . ..;.,k y . (. .;..LtEL. :. . , : ry q..;..y.... jjggggyy;: . .. .-'E . .. .y E E. CE.; . . ;.... : .)Ey. -... . r :. . .E . ---i .. . ,. q .. ..E . . .q ,. jy. ..g .y.. EitttliyitztlrE . . rtjyEiljzqttigryyf ,.. . q - yy q . j . .. . . q . y . . .. j, . . . . koyyyyyyjgy . y. . . . . . yyyjj yy. . yju r . ' i EE yu y . E !' '(' :. q: i E.'rit:-. g l.E; :);y.. g yy g ().. y . .. g y y: .. jy q; ; L y. . . r Lj.,L..LjL; I.t: jL'L'q(jLL'; y. '.t;tEqq;;' gE.y))gy .. y j, . . .... .. ... , . . . . . . ....yy .q :.q( .;).(,LL.;..;.. . . . y y. r.@'ry'.'y..r?i..E.!:..... kjjjyj .;. rykt',.E..yi. ....q .. y.; . y.p; y. . . .'. .)L;jt..).q(y,jj,d!!1i-,; :.( ..E.. -y ....) ( ?.jj,.. y-1... y ' ' E i i. E - rFti... t E .j;. ;!.. E (! . !- E . jygy. . - ...;:.. E..- . E. . (E .. E ! .... . : . .;.. .kyk. . EE t'F-i''' - q (. . -. . . ...rj.. . -: -. -, . kL. . qE. i . -; -y.,.-.g.1.j..-..d..,)....y.;,..y...r.-y.-.st$y-. .. y . .;.. -. .y '. EE.. ; . t; . . . ,.f33f. .E E . . .. .- y . -. ':$l8t,E-.k;#)),pl-(i'.-.'j,b.slb . .....p.-.. .'. --..'-... -.pt -y. ,.)ki-f- --.--; #..... .. . ,,,gt;:t!jj:jjj.j .
dp
:1
.
.
.
..
.
.
q . ..
..
..y).
...
-
41
...
..:..:.
.i'.
1p
...t..
.
...;;.;,,
.
.
. .
...y;'y'y...y(iy(y)(yy:jj..yy..:y(j....
.:jy.yj...
2
.
.
:
.
.'
<.
.
.
.2.
.
...'
....
..r
:';'
'
.
.
'
.
......
:
...
.
..
.
,
-
.q.(
'rt,ywpy
..
. .
...
-
EyEi yt($.?E.r(r . .. ,E 'E. i.E k'JE C4jIIE. ..L 3.k() LLj;L.L .(:.g).-. E ;i.EE-)!'(; . ttjii;. )....y.2..2...,..J): :7.:..
E..y.(.
.
: :E..
LjL.L..kL /. ..
..
.
.
-:--
-. - . ..j .. .j(;E ;.; . (y.jjj.(q.. sl/tq E .
..
. .
'
.E.. . .. j q. ; E E/E.Lq ..L.. . . ... . . ) .. --(j ( . . --. ' . i . -.' --- . -.. . . - . E. .-.. . . -. -:--'. -. ..' ' ... g.. t):;.... ..q/.:I.jLL . .. : g (! .... j ...:k..j..g . . LLL.. ;.r .y y.. . y. gy y . .:yyy ... ' bs? qoujyyjd . ;
...:
.
,
o
.
.
E
.:
...
...
'
'
,.
.
:!)
.:
.
'
; . .j . .. . . ... . g . .. 4 . '-'112::..
iE: ':'E: . -.' ..;E.) '.'.E
...;.
.
. : f . !..p ' ... L. ' '. . L...'...2.r .' ' ;''. ' . ( . (.' ,u..km,,,,o,k,,,.,r+y,,,.,,a, .
.
-
E.'.p.:
)Eq
.
.
: .
.'
.. .
...,
..
.
. .
.
..
.
.
.
. .
.
.r.. . g ... j . ., . ...... . . ... .4: j y. .yyjaj . ts. )j; . j .. jjjj:jjj'. !!!' i.$. jys,lt. . k.. i.. . .ty jyy )- jjy . yyrtyj g '#:'l:'j':.j'.jr:!.k::: .. .yy . yyy gc y i: . . l. .r.)(k!r/.:i..tt'.j:r. .tr(g.(..). r. i)'i;r... yt .tk-. . ttr;: .;... ryki j yy y y. t. r..r) ti).;7..i-tl ... j;... t). . ryli l.t:.. ,:,.,:,, ;)).y;..)-k.t8.l9)-., gy'tt..kq.j4:yy.-.-..,.it::!:j-.,.. L,tjtLL r,.. . 442!2:r ,-L..).;;;.)-L66LLj6q?''' jj:rljjs 41222:; jjzkr:i 44:22,), .(::.-tgy..tr..;y(jtty . ...E .-..(jl. 'l.s' 4(222:, 4(22::; jjjj;, rjrr, ,...EEEEEE. 4k22::; ..j'tj'L. .1k.k,j. (jlr:jl. jljrhhhjj :jjjgCIE1j,. :jjji rjjjq- :jjj;, (jrr:lk. :.1-:.9. (jrrjl. :;:;::::;' ; '. ).. i-.ql..?-' 4(4E ,, IILLL,.:LL..).L (jj j(.@i. .-. .:y. i.-. t; t. p ;. )i. . , E. .. t. El(;. jr. .. r ; ;. tgg j)rpi j-. :18-. (..?- rjq. jyy.i . j-y . t-. ;. .-. j.. . ))-. . . '.i . r. . . .. r,r .' ty. (y;j--.;.kr. . . . ;. .. . . . .. . (. yj).r'!jz'.,..,jp'.'..-''.. . . -,y.,4tE:1'. :lir4:::: 4t::. 1. qlL y.-y1,y-, 4:::,. ....,....,.)s,j.,, 112). 4:2:: 4::2.::;7. ;p';!r 4:::). 1ki2: :5'k 11. llrlu 4(22). 1L )y.j. .l,'.k' .::2). .. L6:-. 1ty!1.. 1:EIL llrqlrrkp IIF:I::IL llrl. :5,k .. .... ..-!1-2Ik. h..dki:r (1. qlrl.IEEIk . .:E:j.. 1t) 1l: IIFI. Jlr!l. ql. Ilrl,) jll:)p .11:)'L.. tdk4(22). !rdtgr11.,1. ki-b.' u. Jlrck!q;l.. . .'... . ' L..;L.-.?-..... k,' qtthk. t.. .r)k.-...y . tl.#-''-it--.. ..----. --L(-q...L.t..-. tl),,:... . . . ytyth, t. til. u Cit-'.' . laws, of countble 1ntersecti on also. A . t.g cqqy 7. tp,y E. E ::.rgn ' .E ..r.;E-.:. ng can EF. jC.y E. . g.. .--.. t.. . ykkE (j,.t, . . y 2 ,t ...E.: j....y t( q.,y...y.. .,p... .(),. 'tj. , . .. .. ,ar. . J?'1'l ,. ;w.. tykk,yyal. @(r,t t7l7)li)'yE--jp:ytq-l'h@i'-.--.j#i.tt)r:-.)k2!l$,j);.. ;,; , , .. t..Lti-.(yy....hk-...'i'..-.-i'.---j;.rls tlot , t ,-,Ljb;LLb6.. j, ) ln the sequel )j.rq( shall t;need -)ykyy.tjj.. t:-.t.-)-t...?t-(..--.-..;-yy..,F.rconcept . -?,--y)k-ttg.-tjE '...-..-.--..:.--r-. we g ..yqE a E ;..-...y..-.:..)...;..y..--.y.y....-........... ),,r.tg2s;-...:,:.5-r):(t..---r@t()kttr;jpki-):.y.t2r).-. ,-.)j.Lb1,,.q--. (;. .y......-.. . .. .. .. . . ,,.tzk r:kjt. 1)4 ? E y,pqy),jj 33:13333,..6,31)3 qtp. 2L2t1jjk1,(1.. yo,i, . .. q l,. ;,?,. q?. y. . j,yj(. lnt:rsectlon (E. . . . .qk .. : :6. :;)..y.,@, of al1 the jyjj,, .( jy .yj ls j.j..j. ,,,. yjyg.. yyt;,y( tygyj;. contmnlng y.y,rty..y..j, ,. . : '! ;(y.tjrj. . (j(:,.tjyyy. .. . y . ty.j .. .. . . . . sj jyjjy.t.yjgy, LILLIL) jjjjjj,yyyyyjy.. yyyyy T. .r.. . .. .)g-jjjy)j.).)4jj..j)j.:.,;j.);.j,j;.jj;.)j,.j.L..,.;..j ..., y. .........j3b;,jbbb;$LLL,).jtfjj;6;3$j(. ..g ...s.sy . ... .. .. .. . . .. . .- . . :7',t.,..;r,,#.-. )),.jjj ). .. ..... . . .jj,.., ...yj.,.. . .'''.. . .j.q,jj. . . . . ' q$3t... ,;-. . . ..' r-,-iiq.-r),-iyr,!!.. j. .=-. $.-<..., . . . );,-':'!'tk. '
. '
.
-
..
.:.
-
.
-.
.
''.
-
-
.
..
.
. . .
-
.;
.
,.);.-')
-
,j,q2..
'
.. . .
tju
..
-
..
,
- ,.
,.-
- ..y;)
-
.
..;.
-,...
..
,..
..
.-.
.. ..
y
:.
.
..
. . .
.
(gjj.. -0. ' s.t.).....,-.y....ytt,..c...y.-....1. ?t..'tlt.t.,t.iy.'t?.'rtA)'.i.' .d.e.. '':.('. e..,4(i!I... ''-. . . %)' 4.) 4)d..,'',,M ttytt/p ''4kk,,.,)'#',,,..)4i yt t 14thls iis 7>1'h: j x,,soy s,: xe.)ts..T?th: ,, .
. . .
.
-.
-
.
-.
,.:r
,.
-
.
.
.-.
-
.
.
.-
.
.
.
.
.
.-
. .
.
.
.
. .-
.
. -
.
.
.j(rtrp1k.
.jry,
,,...
.
.
.
. .q;,..
-iji
.
-.
-..
.-
'
.
-.
. .
-
1k-,,,,-Jj
rjjj;;!;;q
.
.-t;,
. .. .
..
.
.
. .
. ..
.
,,,
-,,
.
.....';-.7.
..:.,2,,
'
..
.
,-,.
c-n'
',,
.;
''
,;j),
''j-.k,
-
. . .
,..
.
.
.
'.
--k...
-. .
.
-
. ..
-
.
.
--,.
..
.
be
,,,
.
.
.
..
-
.,.
,,
't(l!jt)t2),.
..
-@jjjgjjk.:
.,i(-$;ydjy-.. .....
. :
.
'
.
.
:
.
..
.
jlj..
. yyjjggk., ?jjy-.$,,;gjjjr..tyy(,.k;.y;jy;j.-
:jjjy...
.
y
.
..
y.s,.y,.y
.
.
.
, .
.
. ..
..
..
'..-,,ibn
y,.;.y
... ,
.....,.--
.
k
s..
,
...
.
.
,,
.
...)...j....j.,..
,.yyy....y,y.y
..
.
.
.,.)......),),.....,).j
.:
.
-..!it,..
....
.
,q
kf,.
,.
.
.
,,
.yy
,4j,j)),,;jjj,-j)))jj)..,;;,)j,,j...
..
-),-.t..
.
.
.
.)
-,,
,.
.
..
,,
.
....
,
.
.
.
,
..
c-flelds
--
.
-,
C
--
.
16
Mathematics
generated by C, customarily denoted c(C). The following theorem establishes a basic fact about c-fields.
calledthe c-field
1.20 Theorem If C is a finite collection c(C) is tinite, otherwise c(C) is always uncountable. Proof Define the relation R between elements of X by Ry iff x and y are elements of the same sets of C' R is an equivalence relation, and hence defines an equivalence class & of disjoint subsets. Each set of & is the intersection of a11the C-sets containing its elements and the complements of the remainder. (For example, see Fig. 1.1. For this collection of regions of R2 & is the partition defined by the complete network of set boundaries.) lf C contains n sets, & contains at most ln sets and c(C), in this case the collection of all unions of &-sets, contains at most lln sets. This proves the first pm of the theorem. Let C be infinite. If it is uncountable then so is c(C) and there is nothing more to show, so assume C is countable. ln this case every set in & is a countable intersection of C-sets or the complements of C-sets, hence & g c(C), and hence is the collection of a11the countable unions of also U4&) c(C), where flX&) t/(&) uncountable, is &-sets. If we show the same will be true of c(C). We may assume that & is countable, since otherwise there is nothing more to show. So 1et the sets of & be indexed by N. Then every union of &-sets corresponds uniquely with a subset of N, and every subset of ENcorresponds uniquely to a union of 2N, &-sets. ln other words, the elements of U4&) are equipotent with those of which are uncountable by 1.13. This completes the proof. . .
,
R, and let C ((-x,rj, r G (E1 ), the collection of closed with rational called Jltilt/'-ff/r6y endpoints. c(C) is the Borel jield of R, generally denotd B. A number of different base collections generate T. Since countable unions of open intervals can be closed intervals, and vice versa, (compare1.12), the set of open half-lines, ((-x,r), r G (EI), will also serve. Or, letting (rnJ b: a decreasing sequence of rational numbers with rn 2, x,
1.21 Example Let X
=
=
(1.21) Such a sequence exists for any G ER(see2.15), and hence the same c-field is generated by the (uncountable)collection of half-lines with real endpoints, ((-x,x), x e R ) lt easily follows that various other collections generate S, including th open intervals of R, the closed intervals, and the half-open .x
.
intervals. n
1 22 Example Let X given. It is
T
=
F, the extended
real line. The Borel
field of F is easily
# e Sl, where B is the Borel field of R. You can verify that S is a c-field, and is generated by the collection F' of 1.21 augmented by the sets (-x) and V. n =
(#, B tp (+xJ, B t.p (-x4, B QJ (+x) t-p (-x):
1.23 Example Givn an interval f of the line, the class
BI
=
LBf-h f: B e B ) is
Sets and Numbers
called the restriction of T to 1, or the Borel field on 1. ln fact, T; is the c-field generated from the collection C ((-x,r1f-h f: r e () ) n =
.
Notice how c(r) has been defined from the outside' It might be thought that c(r) the inside', in terms of a specified sequence of the operacould be defined tions of complementation and countable union applied to the elements of r. But, despite the constructive nature of the definitions, 1.20 suggests how this may be impossible. Suppose we define .Wlas the set that contains r, together with the complement of every set in T' and a11 the finite and countable unions of the sets is not c(C) because it does not contain the complements of the of r. Of course, be the set containing W1together with all the complements and unions. So let in the same finite and countable unions of the sets in W1.Defining 45, manner, itmight be thoughtthat themonotone sequence (W,,) would approach c(C) x; but in fact this is not so. ln the case of the class fm,l!, for as n it example, can be shown that Wx is strictly smaller than c(C) (see Billingsley IX This fact is 1986: 261. On the other hand, c(C) may be sm aller than for in again fm,1), j3.4. demonstrated, The union of two c-fields (the set of elements contained in either or both of them) is not generally a c-field, for the unions of the sets from one tield with those from the other are not guaranteed to belong to it. The concept of union for c-fields is therefore extended by adding in these sets. Given c-fields ; and N, the smallest c-field containing a11the elements of 5 and all the elements of Wis denoted 5 v V, called the union of 5 and V. On the other hand, 5 r'A W (A:A G . and A e V') is a c-field, although for uniformity the notation 5 ZN V may be used for such intersections. Fonnally, 5 zq N denotes the largest of the c-fields whose elements belong to both 5 and V. Both of these operations generalize to the count1,2,3,... we may define able case, so that for a sequence of c-tields 5n, n l;a. V= n n 15 n and 0* Without going prematurely into too many details, it can be said that a large part of the intellectual labour in probability and measure theory is devoted to proving that particular classes of sets are c-tields. Problems of this kind will arige throughout this book. It is usually not too hard to show that Ac e 5 whenever W e 5, but the requirement to show that a class contains the countable unions can be tough to fulfil. The following material can be helpful in this connection, A monotone class A is a class of sets such that, if tAn) is a monotone sequence withL limit 4, and An e A for a1l n, then A G A. If (Aa) is non-decreasing, then zl U7=1An.lf it is non-increasing, then A O0J=Iz4,,. The next theorem shows that, to dtenine whether or not we ar: dalinf with a c-field, it is sufficient to ,'krd r' y' jr';ty 't'ld r' yyy'.yyjjjjyjjjjjjiy.d k' t'(' y' );' jyyd lr'yyy p' )' 'y,d trytd td ttd t'. ild ctmsl'detwhdherthe limits of monotone sequences belong to it, which should often ' b j esiet to establish than the general case. .
tfrom
.d1
.4z
.44,...
--->
.
=
=
=
=
=
C' y' .'E
-'. .' -' :t' '
.'''
.' '''.'' ..'.'.'....'. ..'.' '. ' . . .'... . .. .. ..(E.
' . .. '
E
' g. ...
' . .'
r . .. : :
1);,4
.
. .lj ' .. . y. .. . . Flrqh:oykm
l q)
l g...(
.. . , . . .
l .
.
'
.
.
. .
.
.
?;; is a c-field iff it is both a field and a monotone class.
.t:)2
E
.
.
:.y
:
.
. . .. . . .r ..Eq(
,,k))
y.
t j' ' .
. . The..ftjjjjy, p tjtjjy .
''
of the
ito
is immediate. For
theorem patt the part y(t.: j. . ;) y y. j.r r ,'tlt:' t.ri.jjryry.yi.,yuit y 1:2j1. :jkq-.j(jjji. :g;pC;: ;(.):t..,.(.,)i'.:..'..y.;tt.t,jiyA......,y '.1t jkj:r qlj?ii. 4, )1:41.. .j1r:lg. 1:;(jj.. jpp:p-;,r 4:;:: ...... jlrrjs 4:2:: 422:: j,qbqf. . 4EEEE! jjr-iylr jjy-plrqjj .. jrrqls ,,krIg. :;; igjl. ji. ji. :. ,(EE L.,;.,jfL:..; ... j . I j!..I t;i:;r ni; (1q ... .g. )......)...... tjjii:;s ,b, 4:22EE jj tlij;r (yj,?)'' yj e. (. . k:j?,. .t.y(). r .., j; (jry-;;;(tj yy ). .s.u.. y... r(j-(.. (.; .i.qLL);,;j . y. . ... y... .;jtytytyt, . . . . tg -L$1t.. ttttt . . .;j )')r'--t)-tzjj i-.-tki -,.-(jkyr))-;-..---,r. y is . rj limit with =1A,, t monotone ;, );. by t sequence ) a ulyytjt U.-----.- e y-):-),::::r,:--.. --..L3LLLi..-. .,.-yt.-..,7r-...(-.......). r ;-)y)-. j/l-rjt--j,-....;.LL;L;.,.;j;;-),.,;b;qj, '''-?'''t''-' )jjt.-. (t)-.)rrtt.,..,)))-y)#)---.r),-.-. t;)--?--.--,r...r-....-.,... ...... ... . . .--. ... ;.,-(.$;.,q;-.,-LLLL1L--t;))plt:'-i yryjyti... E #,,.. .. kr. t).y-j-t,yr ....-..--...- ....-.... ttl'i?.-.-;r-..t...)k.-(--..-t..-.ti..-)-.,)---)'.-.-.t.....-..t-., .t'y4?.i@''. ... . ..Li q-?..'..I3;......;-;t.-b. .!.#,..-t,) llt))k...-.... tiy,-3$bqL-;--....').L ..::?)i..-..$kr'-'t)L' .. .)yyyiyyyi 14441111@11..... i
r
,.tE'jt.ty( : ;,
i E kE'
(
(
.
.
-.yy)j,ytj,.i...y.yE.y;;r.,...myt;y.U,
; -. . g . .
; ;
(
.
E
i..E -
-
q
,.
..
.
.,...
,,
-. .
-
-. .
- . . .
:.
-
.
-.
.E-...
,
q .
. . .
.. -.
.
.
:
E .
.
-
k.tlll.
..
-.
..
..
...-....
.....
..
,.
.
.
..
- ...-.-
.
.
. . ,
...
,,
-;::,---::--
.
-
........
-j
..-.
.,
.
.
..
...
'''--'f'':tt--
.....
define
,
-k,(
..
.....,.
x ' '--------8-'-:'''*''''% . -))
'if'
-jlqilii;,;j;r
.
....
..
-......-....
..
.
.
. .
.
.
.
.
.
..-
..
... ..... -
-p
. .
..
..
-
.
18
Mathetnatics Ur1=lAk, so the theorem follows.
assumption.Uo;=1An =
.
Another useful trick is Dynkin ' s zr- theorem.l To develop this result, we define two new classes of subsets of X.
1.25 Definition A class T is a z-system if A and B e T implies A rn B e T. A class .f is a h-system if (a) X e 1. and B (b) If A and B e c A, then A B e 1. is a non-decreasing sequence and Au A, then A e (c) If fAn e n ,f
-
'1'
,f)
.
Conditions (a)and (b)imply that a l-system is closed under complementation (put .f
m.Moreover, since (b) implies that Bn A,,+1 A,i e for each n, (c) implies that a countable union of disjoint d-sets is in 1. In fact, these implica-
X
=
=
-
tions hold in both directions, and we have the following.
is a l-system if and only if 1.26 Theorem A class (a) X e 1. then Bc e 1. () If B e is a disjoint sequence, then UnAn e (c) If (Aa e ,f
.f
,f)
.
Ia
ln particular, a c-field is a X-system, and moreover, a class that is both a is is a c-tield. This follows by 1.24, because a a-system and a class closed under unions by 1.25(c), and by de Morgan's laws is if a monotone closed under both intersections and complemntation. The following result makes these dehnitions useful. -system
-system
1.27 Dynkin's x-A theorem (Billingsley 1979, 1986: th. 3.2) If T is a a-system, ctfis a l-system, and P c 2, then c(T) c 1. containing P (theintersection of a11 Proof Let kTj denote the smallest containing P), so that in particular, (T) the c M.We show that 3.(:P)is remarks l-system. follow the above, it will then that (T) is a c-field, and By a 2, as required. hence that c(T) c &T) For a set A e 1(T), let Vztdenote the class of sets B such that A rn B e (T). We Clearly, X e Vz, so that condition 1.25(a) is shall show that gk is a satisfied. Let #1, Bz e 5k, and Bj c #2; then A rnfl e 14T) and A fa #2 e (T), and (A r''h#1) c (A fa #2), which implies that -system
-systems
-system.
(A rn #2)
(A t'''h#1)
-
A f'n
=
(#2 -
#1) e 14T).
(1.22)
But this means that Bz #1 e Vx, and condition 1.25(b) is satisfied. Lastly, suppose A f''h Bi e ,4T) for each i 1,2,... and Bi #. Then zt f-h # e (T) by 1.25(c), which means that 1.25(c) holds for Vx, and sk is a as asserted. Suppose A e P. Then B e P implies A PB e T p is a a-system) and since P i and '4T) (T), this further implies B c 5k. Hence P c 5k. Since Nx is a containing have in this smallest also (T) gk P, is the we case. So, c V',A and hence A fa B e (T). whn A e T, # e VT) implies B e We can summarize the last conclusion as: -
'1'
=
-system
-system,
-system
Sets and Numbers (A e P, B e 1(T)) Now defining Vs by analogy
(# e
(T),
yt
with
PB e
=
19
(A OB e
(T)).
(1.23)
9%, so that (T))
=
(A e Vs),
(1.24)
we see that (1.23)and (1.24)together yield P c T. Since Vs is also a by the same argument as held for Vx, and contains T, 1(T) c Ns by definition of (T). Thus, suppose B e (T) and C E (T). Then Ce Vs, which means that B f''h Cq (T). So 1(T) is a n-system as required. . -system
+.
2 Limits and Continuity
2.1 The Topology of the Real Line The purpose of this section is to treat rigorously the idea of as it applies to points of the line. The key ingredient of the theory is the distance between a pair of points, x, y e R, defined as the non-negative real number Ix y l what is formally called the Euclidean distance. ln Chapters 5 and 6 we examine the generalization of this theory to non-Euclidean spaces, and find not only that most aspects of the theory have a natural generalization, but that the concept of distance itself can be dispensed with in their development. We are really studying a special case of a very powerful general theory. This fact may be helpful in making sense of certain ideas, the definition of compactness for example, which can otherwise appear a little puzzling at first sight. An e-neighbourhood of a point e R is a set Sx, fy: Ix-yI < :1, for some s > 0. An open set is a set d c R such that for each e A, there exists for some e > 0 an e-neighbourhood which is a subset of d. The open intervals defined in j1.3 are open sets since if a < < b, : minl 1b l > 0 satisfies the la definition. R and O are also open sets on the definition. The concept of an open set is subtle and often gives beginners some diftkulty. Naive intuition strongly favours the notion that in any bounded set of points to' a point outside the set. But open sets are there ought to be one that is sets that do not have this property, and there is no shortage of them in R. For a complete understanding of the issues involved we need the additional concepts of Cauchy sequence and limit, to appear in j2.2 below. Doubters are invited to suspend their disbelief for the moment and just take the definition at face value. The collection of a1l the open sets of k is known as the topology of R. More precisely, we ought to call this the usual topology on R, since other ways of defining open sets of R can be devised, although these will not concern us. (See Chapter 6 for more information on these matters,) More generally, we can discuss subsets of (Rfrom a topological standpoint, although we would tend to use the term subspace rather than subset in this context. If A c S ? R, we say that A is open in S if for each x G 4 there exists Sxtzj, : > 0, such that .(x,E) f''AS is a subset of A. Thus, the interval (0,) is not open in R, but it is open in (0,1). These sets define the relative topology on S, that is, the topology on S relative to R. The following result is an immediate consequence of the definition. inearness',
-
,
.z7
=
.'r
-x1
.7
=
-x1
,
2.1 Theorem lf 4 is open in R, A r''hS is open in the relative topology on S.
l:n
A closure point of a set A is a point x e R such that, for every E > 0, the set
Limits and Continuity
21
is not empty. The closure points of A are not necessalily elements of A /'7 A, open sets being a case in point. The set of closure points of A is called the c losure of A, and will be denoted X-or sometimes (A)- if the set is defined by an expression. On the other hand, an accumulation point of A is a point x e R wich is a closure point of the set A f.x) An accumulation point has other points of A A, it must also arbitrarily close to it, and if x is a closure point of A and accumulation closure is point. A point that not an accumulation point (the be an of contains x former definition being satisfied because each E-neighbourhood itselfl is an isolated point of A. AC /-7 Sxnz) is A boundar.v point of a set A is a point e X such that the set points of A is denoted ?A, and X not empty for any e:> 0 The set of boundary AO A A. A closed set is one containing A t.p :z4. The interior of A is the set all its closure points, i.e. a set A such that A A. For an open interval A a,b) c R, X' (tz,1. Every point of anb) is a closure point, and a and b are also closure points, not belonging to asbj. They are the boundat'y points of both a,b) and gJ,?7). ,&(x,sl
-
.
.x
.x
.x
=
.
=
-
=
=
=
2.2 Theorem The complement of an open set in R is a closed set.
(:I
This gives an alternative definition of a closed set. According to the definitions, O (theempty set) and R are both open and closed. The half-line (-x,xq is the complement of the open set @,+x)and is hence closed. Extending this result to relative topologies, we have the following.
2.3 Theorem lf
:4
is open in S c R, then S -A is closed in S. cl
In particular, a corollary to 2.1 is that if B is closed in R then S r-hB is closed in S. But, for example, the interval l, 1) is not closed in R, although it is is open in (0,1). closed in the set (0,1),since its complement (0,r1) Some additional properties of open sets are given in the following theorems.
2.4 Theorem
(i) The
union of a collection of open sets is open.
(ii) If A and B are open, then A /-7 B is open. n
This result is be proved in a more general context below, as 5.4. Arbitrary intersections of open sets need not be open. See 1.12 for a counter-example.
2.5 Theorem Every open set A q R is the union of a countable collectin joint open intervals.
of dis-
Proof Consider a collection (&x,:x), x e A h where for each x, Ex > 0 is chosen enough 5'@,:x) Then that A. small i Uxex5'tx,:xlA, but, since necessarily A UxeA5'@,sxl UxeA.@,Exl, it follows that A. This shows that A is a union of open intervals. Now define a relation R for elements of A, such that xRy if there exists an open interval f q A with x e f and y E 1. Every x e A is contained in some interval by the preceding argument, so that xRx for all x e A. The symmetry of R is obvious. Lastly, if x,y e I A and y,z e 1' ? A, I /-7 1' is nonempty and hence I t.p 1' is also an open interval, so R is transitive. Hence R is an equivalence =
Mathematics
22
relation, and the intervals I are an equivalence class partitioning A. Thus, A is a union of disjoint open intervals. The theorem now follows from 1.11. w Recall from 1.21 that f, the Borel field of R, is the c-field of sets generated by both the open and the closed half-lines. Since every interval is the intersection of a half-line (openor closed) with the complement of another half-line, 2.2 and 2.5 yield directly the following important fact.
2.6 Theorem
B contains the open sets and the closed sets of R n .
A collection r is called a covering for a set A i R if A an open set, it is called an open covering.
UsswB. If each
B is
covering theorem If C is any collection of open subsets of R, 2.7 Lindelf's is countable subcollection (Bi e C, i e N ) such that there a X
UB U#. =
f=1
Be N
Proof Consider the collection (Sk Srkbs, rk e Q, sk e Q+) ; that is the collection of a11 neighbourhoods of rational points of R, having rational radii. xQ+ is countable by 1.5, and hence is countable; in other words. The set (E1 show exhausts by We k the set. that, for any open set B c R and point indexing e N a' e B, there is a set Sk G such that e Sk c #. Since has a E-neighbourhood inside B by definition, the desired Sk is found by setting sk to any rational from for s > 0 sufticiently small, and then choosing rk e the open interval (0,1as), 5'(a',/) as is possible by 1.10. Now for each x e UsewB choose a member of say Skp, satisfying x e Skp i B for any B e C. Letting k(x) be the smallest index which satisfies the requirement gives an unambiguous choice. The distinct members of this collection form a set that covers Ussc#, but is a subset of and hence countable. Labelling the indices of this set as k1,k2,..., choose Bi as any member of C containing Ski. and hence also for Usecs. w Clearly, U:'xlffis a countable covering for U7=15k, ,/
=
=
,/
.$
.z'
.x
.$,
,/
lt follows that, if C is a covering for a set in R, it contains a countable subcovering. This is sometimes called the Lindelf property. The concept of a covering leads on to the crucial notion of compactness. A set A is said to be compact if every open covering of A contains afinite subcovering. The words that matter in this definition are and Any open covering that has R as a member obviously contains a finite subcovering. But for a set to be compact, there must be no way to construct an inrducible, infinite open covering. Moreover, every interval has an irreducible infinite cover, consisting of the singleton sets of its individual points; but these sets are not open. Gevery'
Eopen'.
2.8 Example Consider the half-open interval (0,11.An open covering is the eountable collection (41/n,1j, n e NJ lt is easy to see that there is no finite subcollection covering (0,11 in this case, so (0,1j is not compact. EI .
A set A is bounded if zt i 5'(x,s) for some x e A and ir > 0. The idea here is that
Limits and Continuity
23
: is a possibly large but tinite number. In other words, a bounded set must be containable within a finite interval. 2.9 Theorem A set in R is compact iff it is closed and bounded. u This can be proved as a case of 5.12 below, and provides an alternative definition of compactness in R. The sufticiency part is known as the Heine-Borel theorem. A subset B of A is said to be dense in A if B c A q X. Readers may think they know what is implied here after studying the following theorem, but denseness is a slightly tricky notion. See also 2.15 and the remarks following before coming to any premature conclusions.
2.10 Theorem Let A be an interval of R, and C c A be a countable set. Then A C is dense in A. -
Proof By 1.7, each neighbourhood of apoint inA contains an uncountable number of points. Hence for each e A (whetheror not e C), the set (A C) f-h 5'(x,:) is not empty for every e > 0, so that is a closure point of A C. Thus, A C c (A C) QJ C A ? (A C) K .x
.v
-
.x
-
=
-
-
-
.
The k-fold Cartesian product of R with copies of itself generates what is called Euclidean k-space, R The points of Rk have the interpretation of k-vectors, or ordered k-tuples of real numbers, x (x1,.n,...,.q)'.All the concepts defined above for sets in R generalize directly to R The only modification required is to replace the scalars x and y by vectors x and y, and define an e-neighbourhood be the Euclidean distance betweenx andy, where 1Icll in a new way. Let Ilx 11 1/2 IEt,-1 , q is the length of the vector a (J1,...,JJ' and then define S(x,z) < el, for som s > 0. An open set A of R2 is one in which every point (y: Ilx 11 x e A can be contained in an open disk with positive radius centred on x. In R3 the open disk becomes an open sphere, and so on. .
=
.
-y
=
=
=
-y
2.2 Sequences and Limits A real sequence is a mapping from E$into R. The elements of the domain are called the indice- and those of the range variously the terms, members, or coordinates of the sequence. We will denote a sequence either by tak,n e EN), or more briefly by IakI:', or just by (xnJ when the context is clear. (xnl'kois said to converge to a limit x, if for every : > 0 there is an integer Nv for which
Ixn
-
xI
<
e for all n > Nv.
Write xn it ij often x, or = limu-jxak. When a sequence is tendinygo +x or said to diverge, but it may also be said to converge in R, to distinguish those cases when it is does not approach any fixed value, but is always wandering. increasing, non-increasing, or decreasA sequence is monotone (non-decreasing, ing) if one of the inequalities ak f xn+1, xn < xn+1, xn k ak+1, or xn > xnx.l holds for every n. To indicate that a monotone sequence is converging, one may write for emphasis either xn a7 will or xn t, x, as appropriate, although xn -.->
.x
-x
'1'
.x
--
24
Mathematics
also do in both cases. The following result does not require elaboration. 2.11 Theorem Evel'y monotone sequence in a compact set converges.
n
A sequence that does not converge may none the less visit the same point an infinite number of times, so exhibiting a kind of convergent behaviour. If fak,n e N l is real sequence, a subsequence is fxnk,k e INl where fnk, k e N ) is any increasing sequence of positive integers. If there exists a subsequence taksk e (h1 ) and a constant c such that xnk (-1)N, c, c is called a cluster point of the n 1,2,3,...) does not converge, but sequence. For example, the sequence f subsequence obtained by only taking the even values of n converges trivially. c is usually a finite constant, but +x and may be cluster points of a sequence if we If the of notion allow convergence in F. a subsequence. is convergent, then so is ) where fmk3 is an any subsequence of the subsequence, defined as (x,,u,k e EN whose members of fnkl members are also increasing sequence The concept of a subsequence is often useful in arguments concerning convergence. A typical line of reasoning employs a two-pronged attack', first one identities a convergent subsequence (a monotone sequence, perhapsl; then one uses other characteristics of the sequence to show that the cluster point is actually a limit. Especially useful in this connection is the knowledge that the members of the sequence are points in a compact set. Such sequences cannot diverge to infinity, since the set is bounded; and because the set is closed, any limit points or cluster points that exist must be in the set. Specifically, we have two uscful --
=
-x
.
results.
2.12 Theorem Every sequence in a compact set of R has at least one cluster point. Proof A monotone sequence converges in a compact set by 2.11. We show that ) hasamonotonesubsequence.Defineasubsequence (xnk) evel'y sequence (ak,ne EN follows. for 1,2,3,... let if S'et 1, and k there nl xnk..! sup,on:ak as exists a finite ?7z+1 satisfying this condition', otherwise let the subsequence terminate at nk. This subsequence is non-increasing. If it terminates, the sub?u) must contain a non-decreasing subsequence. A monotone sequence (.v, n k subsequence therefore exists in every case. w =
=
=
2.13 Theorem A sequence in a compact set either has two or more cluster points, or it converges. Proof Suppose that c is the unique cluster point of the sequence fxn) but that xn zd-yc. Then there is an infinite set of integers (nz, L e EN) such that Iau c I s for some : > 0. Define a sequence fyk)by setting yz xnk. Since (yz)is also a sequence on a compact set, it has a cluster point c' which by construction is different from c. But c' is also a cluster point of fx,,1, of which (ykl is a subsequence, which is a contradiction. Hence, xn c. w ,
-
=
-->
3 ,...,x
n 2.14 Example Consider the sequence ( 1,x,x2 or more j orma jj y ta.gl where is a real number. In the case IxI< 1, this sequence converges to n e EN()1, The condition specified zero,( I.x''lJ being monotone on the compactinterval g0,11. ,a:
.'t7
,...),
,
Limits and Continuity
25
in (2.2) is satisfied for Nc = logtel/log IxI in this case. If x = 1 it converges it to 1, trivially. lf > 1 it diverges in R, but converges in 2-to +x. If x +1 diverges, but oscillates between cluster points and neither converges nor -1
.x
=
-1.
Finally, if x < 1 the sequence diverges in R, but does not converge in Ea mately, it oscillates between the cluster points +x and -
F. Ulti-
-x.
We may discuss the asymptotic behaviour of a real sequence even when it has no limit. The superior limit of a sequence fxn) is
limsup xn n
=
inf sup xm. n
(2.3)
m kn
(Alternative notation'. limn xn.) The limsup is the eventual upper bound of a sequence. Think of (sup-znau, n 1,2,... J as the sequence of the largest values the sequence takes beyond the point n. This may be +x for every n, but in all cases it must be a non-increasing sequene having a limit, either +x or a finite real number; this limit is the limsup of the sequence. A link with the corresponding concept for set sequences is that if xn supA,, for some sequence of sets (4n c R ) then limsup xn supA, where A limsupsAn. The inferior limit is defined likewise, as the eventual lower bound: =
=
=
,
liminf xn
=
=
-
limsupt-ak)
=
sup inf xm, n
n
(2.4)
mkn
also written limnak. Necessarily, liminfaak ; limsupnak. When the limsup and liminf of a sequence are equal the sequence is convergent, and the limit is equal the sequence converges in V. to their common value. If both equal +x, or The usual application of these concepts is in arguments to establish the value of a limit. lt may not be pennissible to assume the existence of the limit, but the limsup and liminf always exist. The trick is to derive these and show them to be equal. For this purpose, it is sufticient in view of the above inequality to show liminfn xn limsupn xn. We often use this type of argument in the sequel. To determine whether a sequence converges it is not necessary to know what the limit is; 4he relationship between sequence coordinates the tail' (as tl criterion for converbecomes large) is sufficient for this purpose. The Cauchy > iff for real 0 of that Ne such ('E'i states every e: fakl converges sequence gence a that Ixn xm I < : whenever n > Nc and m > Ne. A sequence satisfying this criterion is called a Cauchy sequence. Any sequence satisfying (2.2)is a Cauchy sequence, and conversely, a real Cauchy seqqence must possess a limit in R. The two definitions are therefore equivalent (in R, at least), but the Cauchy condition may be easier to verify in practice. The limit of a Cauchy sequence whose members a1lbelong to a setA is by definition a closure point of A, though it need not itself belong to A. Conversely, for every accumulation point of a set A there must exist a Cauchy sequence in the set whose limit is x. Construct such a sequence by taking one point from each of the sequence of sets, -x,
-
.7
(A rn S(x.$In4, n
=
1,2.3,..1,
26
Mathetnatics
none of which are empty by definition. The term limit point is sometimes used synonymously with accumulation point. The following is a fundamental property of the reals.
2.15 Theorem Every real number is the limit of a Cauchy sequence of rationals. Proof For finite n 1et xn be a number whose decimal expansion consists only of zeros beyond the nth place in the sequence. lf the decimal point appears at position m, with m > n, then xn is an integer. If m f n, removing the decimal /10'J-'N point produces a tinite integer a, and xn a so ak is rational. Given any real x, a sequence of rationals (ak) is obtained by replacing with a zero every beyond the ath, for n = 1,2,... Since digit in the decimal expansion of jxa..l l < 10-/, lak) is a Cauchy sequence and xn x as n x. w =
.v
-->
.--->
-xn
The sequence exhibited is increasing, but a decreasing sequence can also be conIf x is structed, as f-yn) where (yn) is an increasing sequence tending to itself rational, this construction works by putting ak = x for every n, which trivially defines a Cauchy sequence, but certain arguments such as in 2.16 below depend on having xn # x for every n. To satisfy this requirement, choose the terminating' representation of the number', for example, instead of 1 take This does not 0.9999999..., and consider the sequence (0.9, 0.99, 0.999, work for the point 0, but then one can choose f0.1, 0.01, 0.001,...1. of a real One lnterestingcorollary of 2.15 is that, since every E-neighbourhood number must contain a rational, (E)is dense in R. We also showed in 2.10 that R (D is dense in R since Q is countable. We must be careful not to jump to the concluAnother version sion that because a set is dense, its complement must be of this proof, at least for points of the intelwal E0,1), is got by using the binary expansion of a real number. The dyadic rationals are the set -x.
<non-
...)
.
-
,
tsparse'
.
D
=
(ilzn,
i
=
1,...,2* 1, n c EN )
(2.5)
-
.
The dyadic rationals corresponding to a finite n define a covering of E0, 11 by 1/2N, which are bisected each time n is incremented. For any x intervals of width e g0,1J, a point of the set f illn' i = 1,,..,22 1 ) is contained in 5'(x,eJfor : < 2/21 so the dyadic rationals are dense in (0,11. D is a convenient analytic tool when we need to detine a sequence of partitions of an interval that is becoming dense in the limit, and will often appear in the sequel. Another set of useful applications concern set limits in R -
.
2.16 Theorem Every open interval is the limit of a sequence of closed subintervals wiih rational endpoints. Proof If a,b) is the interval, with a < &, choose Cauchy sequences of' rationals an a and bn b, with tzI < l)j (alwayspossible by 1.10). By definition, for for all n k N, and hence every x e anbj there exists N k 1 such that x gtzr,,lul > anb) i liminfnlln,lul. On the other hand, since an a and b > bn, abbjc (Jn,:,,1f for all n 2 1, so that a,bjc i liminfagtzn,hnlc. This is equivalent to limsupagtzn,:nj atb). Hence limnltzn,:nl exists and is equal to a,b). . .1e
''
Limits and Continuit.y
27
This shows that the limits of sequences of open sets need not be open, nor the limits of sequences of closed sets closed (takecomplements above). The only hard and fast rules we may 1ay down are the following corollaries of 2.4(i): the limit of a non-decreasing sequence of open sets is open, and (bycomplements) the limit of a non-increasing sequence of closed sets is closed.
2.3 Functions and Continuity hfunction of a real variable is a mapping J:S F-->T, where S c R, and 'F c R. By specifying a subset of R as the codomain, we imply without loss of generality that /S) = T', such that the mapping is onto 7. Consider the image in F, under f, of a Cauchy sequence fxnlin S converging to .x. lf the image of every such sequence converging to x e S is a Cauchy sequence the function is said to be continuous at Continuity is in 'F converging to formally defined, without invoking sequences explicitly, using the 6r approach. < # is continuous at the point x e S if for any e > 0 D > 0 such that Iy whenever < The choice here depend S. of implies lfl may on x. ye fx) I e, lf f is continuous at every point of S, it is simply said to be continuous on S. Perhaps the chief reason why continuity matters is the following result. .f@),
.x.
-
-xl
-
2.17 Theorem lf J: S F-> 'T is continuous at a1l points of S, f -1(A) j s open ju whenever A is open in 7J, and f -1(4) is closed in S whenever 4 is closed in F.
s
(:a
This important result has several generalizations, of which one, the extension to vector functions, is given in the next section. A proof will be given in a still more general context below; see 5.19. Continuity does not ensure that J(A)is open whenA is open. A mapping with this fAC) J(X)C in general, we # property is called an open mapping, although, since cannot assume that an open mapping is also a closed mapping, taking closed sets to clpsed sets. However, a homeomorphism is a function which is 1-1 onto, continuous, and has a continuous inverse. If f is a homeomorphism so is f and hence by 2.17 it is both an open mapping and a closed mapping. It therefore preserves the structum of neighbourhoods, so that, if two points are close in the domain, their images are always close in the range. Such a transformation amounts to a relabelling of axes. If fx + ) has a limit as h i 0, this is denoted f@+4.Likewise, J@-) denotes the limit of f(x ). It is not necessary to have x e S for these limits to exist, but if fx) exists, there is a weaker lkotion of cpntinuity at x. f is said at the point x e S if, for any s > 0, R 6 > 0 such that to be right-continuous and < 8 whenever 0 f x- h e S, .j
,
-
Iflx-h) fx) I < -
lt is said to be ever 0 < h <
(2.6)
E.
left-continuous at x if, for any e: > 0, 3 8 > 0 and x e S,
wch that
when-
-
If@) J@ ) I < -
-
z.
(2.7)
Mathematics
28
Right continuity at x implies J@) J@+)and left continuity at x implies fx) = the function is continuous at x. fx-). If fx) f(x+) Continuity is the property of a point x, not of the function f as a whole. Despite continuity holding pointwise on S, the property may none the less break down as certain points are approached. =
.f@-),
=
=
2.18 Example Conjider fx)
1/x, with S
=
=
T'
=
(0, x).
iff
<
For 6: > 0, 1
Iflx- ) fx) I -
=
x(x+o
and hence the choice of depends on both 0. l:l 0, but not in the limit as that
J: S
F->
sx
-
flx) is continuous
continuous if for every E
'F is unformly
Ix-yl
6: and x.
j
for all x
>
-->
.x
The function
< E
<
=
Ifx,
>
0 H
>
0 such
-.f(A') 1< e:
(2.8)
for every x,y e S. ln 2.18 the function is not uniformly continuous, for whichever is chosen, we can pick x small enough to invalidate the detinition. The problem arises because the set on which the function is defined is open and the boundary point is a discontinuity. Another class of cases that gives difticulty is the one x. where the dofnain is nbounded, and continuity at x is breaking down as x -->
However, we have the following result.
2.19 Theorem lf a function is continuous everywhere on a compact sets, then it is bounded and uniformly continuous on S. In (For proof, see 5.20 and 5.21.) Continuity is the weakest concept of smoothness of a function. So-called Lipschitz conditions provide a whole class of smoothness properties. A function f is said to satisfy a Lipschitz condition at a point x if, for any y e 5'@,) fot some 8 > 0, H M > 0 such that
If@l fx, I K -
ftlx
-yI )
(2.9)
R+ F-> R+ satisfies hld) 1 0 as d 1, 0. f is said to satisfy a uniform Lschitz condition if condition (2.9)holds, with fixed M, for a1l x,y e S. The Continuity (resp.unifonn type of smoothness imposed depends on the function uniform Lipschitz) property for any continuity) follows from the Lipschitz (resp. choice of h. Implicit in continuity is the idea that some function (.): R+ F-> R+ exists satisfying 6(E) 1 0 as 6: 1 0. This is equivalent to the Lipschitz condition holding for some (.),the case h = 8-1 B y imposig some degree of smoothness on h making it a psitive power of the argument for example we impose a degree of smoothness on the function, forbidding sharp corners' The next smoothness concept is undoubtedly well known to the reader, although differential calculus will play a fairly minor role here. Let a function J: S F-> 'F be continuous at x e S. lf
here
W
:
.
.
-
-
.
29
Limits and Continuity
fx-h) fx) h -
/:(x)
=
lim i0
(2.10)
exists, A'@)is called the left-hand derivative of f at x. The right-hand derivative, /1(x), is detined correspondingly for the case h 0. lf J+'(x) f-'(x),the common value is called the derivative of f at x, denoted f'x) or df/dx, and f is said to be dterentiable at x. lf J': S F-.> R is a continuous function, f is said to be continuously dterentiable on S. A function f is said to be non-decreasing (resp. increasing) if f@) flx) (resp. f@)> Ax))whenever y > x. lt is non-increasing (resp.decreasing) if is non-decreasing (resp.increasing). A monotone function is either non-decreasing or non-increasing. 'When the domain is an interval we have yet another smoothness condition. A function f.. (J,:1 F-+ R is of bounded variation if 3 M < x such that for every < xn b, by tinite collections of points a partition of gt7,/71 xo < x1 < '1'
=
-f
=
...
=
n
< M. lfxi) .f(.xf-.1)I 77 k=1
(2.1 1)
-
2.20 Theorem If and only if f is of bounded variation, there exist non-decreasing functions h and h such that f n .42
.f1.
=
-
(For proof see Apostol 1974: Ch. 6.) A function that satisfies the uniform Lipschitz condition on gJ,hl with ( I.x y I) Ix y I is of bounded variation on =
-
-
(J,:1.
2.4 Vector Sequences and Functions A sequence (xs) of real k-vectors is said to converge to a limit x if for every E > 0 there is an integer Nc for which
-x1I< Ilxn
E
for all H > Ne.
The sequence is called a Cauchy sequence in R iff Ne and m > Nv. A function
fl S
i--y
(2.12)
-auIl< e Ilxn
whenever
n
>
1-,
where S c Rk an d 7 R associates each point of S with a unique point of 7. 1ts graph is the subset of S x'r consisting of the k + ll-vectors (x, fx) ) for each x e S. f is continuous at x l S if for any E > 0 H > 0 such that ,
,
#
IIII <
=
I/.(x+)
-
fx) I < e
(2.13)
whenever x + b e S. The choice of may here depend on x. On the other hand, f is uniformly continuous on S if for any E > 0, 3 > 0 such that
IIII<
c:':>
sup x e S,x+: c S
l/'tx+h)
-.f@)
I<
e.
(2.14)
Mathematics
f (h,...,fm)'of functions of x is called, simply enough, a vector function.2 c on tinuity concepts apply element-wise to f in the obvious way. The A vector
=
function
f. . s
F-.
s sc ,
R -1
is said to be one-to-one if there exists a vector function f : S F-.-:hS, such that f -1(/.1)) x fOr each x e S An example of a 1-1 continuous function is the affine transformation 3 ./12) X2 + b = for constants b k x 1) and A (kx k) with IA l 0, having inverse function/-l -1 does not possess a closed form, A (y b4 ln most other cases the but there is a generalization of 2.17, as follows. =
.
=
./-1/)
:y:
-
.
2.21 Theorem Iff / S F- 1- is continuous, where S i Rk an d 1- i R'/ f-1A) is open in S when A is open in I-, and.f -1(A) is closed in S when A is closed in -T. (:1 ,
2.5 Sequences of Functions 7, 'T c R, be a function, where in tb-is case fl may be an arbitrary set, not necessarily a subset of R. Let f/'n,n e (N) be a sequence of such functions. If there exists f such that, for each e f, and E > 0, R Nvjl such that 1Jnt)l J((J)) l < e:when n > Asro,then fn is said to converge to f,pointwiseon f1. As for real sequences, we use the notations fn f, fn f, or fn 1 J, as appropwhere in the latter case the monoriate, for general or monotone convergence, tonicity must apply for eveyy (.t) e f2. This is a relatively weak notion of convergence, for it does not rule out the possibility that the convergence is breaking down at certain points of f1. The following example is related to 2.18 above. Let
fn1f
F-
-
'1'
-->
2.22 Example Let fnx) (0,x) is 1/x. But
=
nllnx
+
1),
fnlxt -
-
.x
1
e
=
x
(0,x). The
pointwise limit of
fnx) on
1 xtrl-x + 1)
,
and llxNcxx + 1)) < e: only for Nu > (1/a 1)(1/x). Thus for given E, Nvx cxo 0 and it is not possible to put an upper bound on Nu such that as x lfnx) 1/xl < s, n 2 Ncx, for every x > 0. n -->
-
--
-
To rule out cases of this type, we define the stronger notion of unifrm convergence. If there exists a function f juch that, for each : > 0, there exists N such th a t
lfnkbt J((,))I < sup eD -
fn is said to converge
E when
to J unifonnly on f1.
n
>
N,
31
Limits and Continuity
2.6 Summability and Order Relations The sum of the tenns of a real sequence faklTis called a series, written X'izzlxa (or just EAk). The terms of the real sequence (X'=lau,n e (N) are called the partial sums of the series. We say that the series converges if the partial sums converge to a tinite limit. A series is said to converge absolutely if the monotone sequence (Z'=1IxmI n e Ehl converges. ,
T x? This 2.23 Example Consider the geometric series, Zy converges to 1/(1 x) when 1x1< 1, and also converges absolutely. It oscillates between cluster points and for other values of x it diverges. u 0 and 1 for x .1
-
.
-1,
=
2.24 Theorem lf a series converges absolutely, then it converges. Proof The sequence (X;.1lxm I n e EN)is monotone, and either diverges to +x or converges to a finite limit. ln the latter case the Cauchy criterion implies that + Ixn..mI + Ixn..mI 0 as m and n tend to infinity. since Ixn I + lxnl + inequality,4 +xn.vm I by the triangle lak + convergence of (X)=1au,n E EN ) follows by the same criterion. w ,
-->
....
....
....
An alternative tenninology speaks of summability. A real sequence taklT is said to be summable if the series Xzkconverges, and absolutely summable if ( Ixnll7is summable. Any absolutely summablesequenceis summableby 2.24,and any summable sequence must b converging to zero. Convergence to zero does not imply summability (see2.27 below, fof example), but convergence of the tail sums to zero is necessary and suftkient.
2.25 Theorem 1ff (xn)7is summable, Zrlzznx- 0 as n -->
n-1
-->
x.
Proof For necessity, write IZ,,=l.zu x ! f IZ,a=1au I + I m-nxm j sincefor any : > 0 < k there exists N such that IE,7ox,ul e: for n N, it follows that lZp1=laulS ,,=,14% Then n I +: < x.n Conversely, assume summability and let A Eklxn. IE#-1 Z,n=n-Ym X Y,,=1.<,a () as n .-.y x s
sx
.
=
-1
-->
=
.
-
.
A sequence fakJTis Cesro-summable if the sequence This is weaker than ordinary convergence.
i .26 Theorem lf (akl:' cbnverges to x, its Cesro
-1
n converges. f?z Eru=laulq
sum also converges to x. n
But a sequence can be Cesro-summable in spite of not converging. The sequence f (-1)/)7 converges in Cesro sum to zero, whereas the partial sum sequence (r=()(-1)''' J7converges in Cesro sum ttl (compare2.14). Various notations are used to indicate the relationships between rates of diverfJnl!' gence or convergence of different sequences. If (xnlTis any real sequence, is a sequence of positive real numbers, and there exists a constant B < x such that IakIIan ; B for a1l n, we say that xn is (at most) of the order of magnitude of an, and write xn Oa. If takIanl converges to zero, we write xn oan), 4nd s>y that xn is of smaller order of magnitude than an. an can be incfeasing or decr:aing, so this notation can be used to express an upper bound either on the fte of eropth of a divereent seauence. or on the rate of convergence of a =
=
32
Mathematics
sequenceto zero. followsfrom the
Here are some rules for manipulation of 0(.), whose proof definition. lf xn On a) and y,j Onq), then =
.;tn +ya x 'n =
=
=
t?tzynwelxtfts; )
(2. 15)
(?(Itz+f5)
(2.16)
,
xls onsq), whenever =
a'fz is defined.
(2.17)
An alternative notation for the case xn 0 is xn f< an, which means that there is a constant, 0 < B < x, such that xn f Ban for al1 n. This may be more convenient in algebraic manipulations. The notation ak an will be used to indicate that there exist N k 0, and t'inite constants A > 0 and B k A, such that infapvak lan) 2 A and supazxtak Ian) S B. This says that l and (Jnl grow ultimately at the same rate, and is different 0. Some f'rom the relation xn Oan), since the latter does not exclude xn Ian 1. authors use xn an in the stronger sense of xn lan -
txn
.-.-)
=
.-->
-
2.27 Theorem If (ak) is a real positive sequence, and xn 1+a. n (i) if a > then Zpual.u n then Erlxlau log n; (ii) if ('x ()(sl+a) (j mzmxm < then )Lxlau x (iii) if (x < an xx
n,
-
-1
-
,
-1
=
-
-1
(x
.
.
Proot By assumption there exist N 2 1 and constants A > 0 and B k A such that An < xn K Bnz for n k #, and hence Amnzmt < mnzxm S Bxmnzm. The limit x for different values of a defines the Riemann zetahmction of 1x.l?'?ztz as n and its rates of divergence for a k for a < are standard results', see e.g. of Apostol (1974:Sects. 8. 12-8.13). Since the sum terms from 1 to N-1 is tinite, their omission cannot change the conclusions. . ..4
-1
-1,
lt is common pratice to express the rate of convergence to zero of a positive real sequence in tenns of the summability of the coordinates raised to a given power. The following device allows some f'urtherrefinement of summability condiV as v x (0) for tions. Let U(v) be a positive function of v. If &(vx)/&(v) > 0 and < p < +=, U is said to be regularly vatying at inhnity zero). lf a .x x (0), 1 for x > 0 as v positive function Lv) has the property L(%'x)lLM) frl/W/y varying said Evidently, is slowly regularly it at zero). to be any varying function can be expressed in the form &(v) = vpfXvl,where L(v) is slowly varying. While the definition allows v to be a real variable, in the cases of interest we will have v n for n e N, with U and L having the interpretation of positive sequences. --
--
-x
-->
-->
=
2.28 Example (logvltxis slowly varying at infinity, for any
G.
u
On the theol'y of regular vqriation see Feller (1971),or Love (1977).The important property is the following. 2.29 Theorem If L is slowly varying at infinity, then for any k 1 such that
>
0 there exists N
33
Limits and Continuity
v -8
Atvl
<
<
v
8
a 11v
,
N
>
.
(2.18)
(a
Hence we have the following corollary of 2.27, which shows how the notion of a convergent power series can be refined by allowing for the presence of a slowly varying function. 'ixlak
2.30Corollary
< If xn OnLnl) then Ln) which are slowly varying at infinity. u =
x
for a11(x
and al1 functions
-1
<
On the other hand, the presence of a slowly varying component can affect the summability of a sequence. The following result can be proved using the integral test for series convergence (Apostol 1974: Sect. 8.12).
2.31 Theorem If ak
X& m=
l.x,,,
l/gntlog n) 1+8 j w ith
-
>
0 then ,
log 1og n. n
-
E=nzz lak
<
If 6
x.
=
0, then
2.32 Theorem (Feller 1971: 275) lf a positive monotone function &(v) satisfies &(v.x) f7(v) -where D is dense in R+, and 0
v(.x),al1 a7 e <
v(x)<
then
x,
(2.19)
D,
v@) =
.x
P
for
<
-x
p
<
x.
To the extent that (2.19) is a fairly general property, we can conclude monotone functions are as a rule regularly varying.
u that
2.33 Theorem The derivative of a monotone regularly varying function is regularly varying at cra. Proof Given &(v) U' (v) If f7(v)
-->
=
=
vPf-(v),
write
pvP-1f-(v)
+vPfJ(v)
=
vP-1(p1,(v)
+vr(v)).
0 there is no more to show, so assume liminfvfxv)
jfzotjvxxilj
Kdv
which implies f7(vx)/f7(v) U'Mx)
U,(v)
Ly'vl
=
x
...0
-
(v)
--
L
(v) fXv)
>
0. Then
(),
(2.21)
w
(2.22)
1. Thus,
p-lpfatvx) =
f-tv-lj
jf7,(vAl
(2.20)
+
vxc'tvxl
pAtvl +vA
,
(v)
--)
p
a7
.
#
2.7 Arrays Arguments concerning stochastic convergence often involve a double-indexing of elements. An array is a mapping whose domain is the Cartesian product of countable, linearly ordered sets, such as ENx ENor r x (N, or a subst thereof. A real double array, in particular, is a double-indexed collection of numbers, or, alternatively, a sequence whose members are real sequences. We will use notation juch as (fakt,t E z J, n e (N), or just fau) when the context is clear.
Mathematics
34
-?'
A collection of tinite sequences (fau, f 1,...,k ), n e IN) where kn x as n --+ x, is called a triangular array. As an example, consider array elements of the 1,...,n) is a real sequence. The question of form xnt ytln, where tys t whether the series ( :=1.x,,,, n e N l converges is equivalent to that of the Cesro convergence of the original sequence', however, the array formulation is frequently =
,
=
=
the more convenient.
2.34 Toeplitz's lemma Suppose (y,;) is a real sequence and yn .-.-: y. lf ((.'xkf, t= such that 1,...,lk ), n e ENl is a triangula.r array (a) xnt
0 as n
--
for each fixed /,
x
--
kn
(b)
lim n-yoo
X Iak,lS
C <
x,
>1
kn lim A'-lau1, =
n-+x
>1
k then E/i,lxn,y, --- y. Proof
Iyn -y
F or y
=
(c) can be
O ,
fyn) for any s > 0 5 Ne k 1 such that for (c), and then (b) and the triangle inequality,
By assumption on I < 8C. Hence by
,
kn
n
>
Nvs
kn
lim Xxna, -y
n-yx
omitted.
X-v/t-'w
lim
=
>1
-y)
>1
n-yx
Nz
Txntyt y)
< lim
-
+
s
wj
n--kx
in view of (a). This completes the proof, since
E
(2.23)
:,
=
is arbitrary.
x
A particular case of an array fxn/)satisfying the conditions of the lemma is xnt where S = (Z''=1y,)-1y/, ty/lis a positive sequence and Z'lsxlys A leading application of this result is to prove the following theorem, a fundamental tool of limit theory. -->
=.
2.35 Kronecker's Iemma Consider sequences (tz,)';'and x, If Ll=jxtlat C < cxa as n numbers, with at -?'
tx/lTof
positive real
-->
=.
--
1
n
-Xx,
Jn
0.
--)
(2.24)
>1
Proof Detining co 0 and cn Xntcjxtlat for n G N, note that xt atct cr-1), 1,...,n. Also define tzo 1,...,n, so that an 0 and bt at fh-l for t t ntzzkbt. Now apply the identity for arbitrary sequences ao,...,an and co,...,cn, =
=
=
77(c,,.-1 Abel's known
c,-1)
as
=
=
-
-
=
n
n
(This is
=
=
=
77(tz,-1 zflcr-l -
+
ancn
-
zoco.
,-1
partial
summation formula.) We obtain
(2.25)
Limits and Continuit.v
1 n -- X xt an >1
n
1
X atlc t an
=
-
>1
= cn
35
1
-
-
ct- 1)
w-xn
--Lbtct-j an
-->
C- C
=
(2.26)
0
>1
is by the Toeplitz lemma, setting xnt
where the convergence
=
bt lan.
.
The notion of array convergence extends the familiar sequenceconcept. Consider subsequences, collection of full generality k (N for a (fxmnk,e l m e EN), an array where fnk, k e INl is an increasing sequence of positive integers. If the limit xm limz--ycxaAun: exists for each m e IN,we would say that the array is convergent; and = its limit is the infinite sequence (x,,,,m e N 1.Whether this sequence converges is a separate question from whether it exists at all. Suppose the array is bounded, in the sense that supkmlxmpulf # < x. We know by 2.12 that for each m there exists at least one cluster point, say xm, of the inner sequence (xaw,k e IN1. An important question in several contexts is this: is it valid to say that the array as a whole has a cluster point?
2.36 Theorem Corresponding to any bounded array f(xrm,k e ENJ, m e N ) there k e N J m e EN exists a sequece (x,,l, the limit of the array ( tx,ug, J as k x, subsequence n:) of f for each m. where (rl:) is the same ,
-->
,
Proof This is by construction of the required subsequence. Begin with a conver1 be a su bsequence o f (nz) such that sj gent subsequence for m 1', let (nkJ x1. Next, consider the sequence (.n,n)J. Like f-n,s:J,this is on the bounded interval (-#,#), and so contains a convergent subsequence. Let the indices of this 1 be denoted f42:) and note that latter subsequence, drawn from the members of (?1k), Proceeding in the same way for each m generxl,nj xl as well as 12,4 ates an array (fn, k e IN1, m e INJ, having the property that lxjw, k e EN) is a convergent sequence for 1 S i S m. k k e ENJ in other wotds, take the first member of Now consider the sequence fns 1 th e secon d member of (rl2k), and so on, For each m, this sequence is a subtlkl, pzth point of the sequence onwards, and hence the sequence of (n'2Jfrom the k e EN ) sequence f.x,,,4,k k PZJis cnkergent. This means that the sequence (xmmj, k = setting rl11 requirement isfies of the theorem. w is convergent, so the f fnzl sat ...-)
.n
=
,
-->
.n.
--
'
,
is called the
,
.
3 s4easure
3. 1 Measure Spaces A measure is a set function, a mapping which associates a (possiblyextended) real number with a set. Commonplace examples of measures include the lengths, areas, in and volumes of geometrical tigures,but wholly abstract sets can be an analogous way. Formally, we have the following definition. <measured'
3.1 Definition Given a class 5 of subsets of a set f, a measure
jt:
; F->
R-
is a function having the following properties: (a) g,(A) k 0, al1 A e %. 0. (b) g,(0) (c) For a countable collection (Aj e F, j e ENl with Aj DA./' and UjAj e ;, =
p. UXy
=
)'
=
O for j
j'
#
X g,(Xy). C1 j
The particular cases at issue in this book are of course the probabilities of random events in a sample space f; more of this in Chapter 7. Condition (a) is optional and set functions taking either sign may be referred to as measures (see e.g. j4.4), but non-negativity is desirable for present purposes. A measurable space is a pair (f1,T) where f is any collection of objects, and 5 is a c-field of subsets of f. When (,T) is a measurable space, the triple (,T,g) is called a measure space. More than one measure can be associated with the measurable space (f1,F), hence the distinction between measure space and measurable space is important. Condition 3.1(c) is called countable additivity. lf a set function has the property
p,tA kJ#)
=
(3.2)
p.(z4)+p,(f)
for each disjoint pair A,#, a property that extends by iteration to finite collections A1,..., An, it is said to be jnitely additive. In 3.1 5 could be a field, but the possibility of extending the properties of g, to the corresponding c-tield, by allowing additivity over countable collections, is an essential feature of a measure.
If g,(f1) < cxn the measure is said to be hnite.And if Q UjLh where lfl is a countable collection of Y-sets, and g.tfl < x for each j, g. is said to be ; G-jinite. In particular, if there is c(./) and Lh l a collection .Y'such that =
,$
=
sieasure ./
(A fa #: # e for each j, g is said to be c-finite on (ratherthan on ;). If is a measurable space and (A,?L,g) is a measure space 11 for some A q @,(A,8L) g,(AC) restriction If in of (f,T,g,) to A. 0 (equivalentto this case called the g(A) gtf1l when p.tfl < cx)l A is called a supportof the measure. When A supports 2, the sets of ?L have the same measures as the corresponding ones of 5. A point e f with the property g,(( )) > 0 is called an atom of the measure. =
=
=
3.2 Example The case closest to everyday intuition is Lebesgue measure, m, on the measurable space (R,O), where tB is the Borel field on R Generalizing the b a to an notion of length in geometry, Lebesgue measure assigns r?z((J,:1) (J,!71. plausible Additivity is an intuitively interval property if we think of measuring the total length of a collection of disjoint intervals. .
=
-
Lebesgue measure is atomless (see3.15 below), every point of the line taking ((J,:1, O(u,,j,m) denote the restriction of measure 0, but ??z(R) x. Letting (R,f,pz) to a finite interval, m is a finite measure on (t7,:1.Since IR can be partitioned into a countable collection of finite intervals, m is c-finite. n =
additional properties may be deduced from the definition:
Sole
3.3 Theorem For arbitrary T-sets A, #, and (Ay,j e N J (i) A c # = g,(A) S g,(#) (monotonicity). (ii) gtA k.p #) + g,(A ch B4 g.(A)+ g(#). (iii) p,(UyAj) f Xg,(Ay) (countablesubadditivity).
,
=
Proof To show (i) note that A and B -A are disjoint sets whose union is #, by hypothesis, and use 3.1(a) and 3.1(c). To show (ii),use A wB A QJ (B -A) and B = (A chB) k.pCB A), where again the sets in each union are disjoint. The result Al and Bn follows on application of 3.1(c). To show (iii), detine #1 UP-14 An j. Note that the sets Bn are disjoint, that Bn ? An, and that U==lfj j UT=IAy. Hence, J =
-
=
-
./
=
=
=1
X
G)
U p,/=1
=
p,
X
oo
Xp,(l U#, /=1 =
j=1
< Xp,(A?).
(3.3)
.
j=1
This proof illustrates a standard technique of measure theory, converting a sequence of sets into a disjoint sequence having the same union by taking differences. This trick will become familiar in numerous later applications. The idea behind 3.3(ii) can be extended to give an expression for the measure of any tinite union. This is the inclusion-exclusion formula'. n
UA/
p,j=
#
n
Xp,(A)
-
1
j= 1
-
77p,(+rnAzl k+j
+
p.tAycAAkchA,l 77 k:gy:y;l + jttx
f'-'h 1 r''hA:2,
...
-
...
r''hXrj),
(3.4)
where the sign of the last term is negative if n is even and positive if n is odd, and there are ln 1 tenns in the sulp in total. The proof of (3.4)is by induction from 3.3(ii), substituting for the second term on the right-hand side of -
Mathematics
38
(,U.-j+) p.(-.-)
g, repeatedly, for n
-
+
p,
-j,'-s
-U.-,'.,) p.
(,
-.-)
(
-
(3.5)
f..h
,
1. n 2,..., 1. IN be n e ) a monotone sequence of T-sets with limit A e T. A set function on g, is said to be continuous if g,(z4,,) g.(A). -
-
(AU,
Let
.--.-:
3.4 Theorem A finite measure is continuous. Proof First 1et (An) be increasing, with A,,-I cAn, and A U*a=IA,,.The sequence (Bj, j G ) where #1 Aj, and Bj Aj -Ay-I for j > 1 is disjoint by eonstrucand tion, with Bj e @, Aa U7=1#j, =
=
,
=
=
p,(An)
=
Xp,(). j=1
The real sequence (g,(A,,)) is therefore monotone, and converges since it is bounded g,(A). above by g,4Q)< x. Countable additivity implies LT) g,(U7 O';=1A,,. Consider Alternatively, 1et (An) be decreasing, with z4n-l Q An and the increasing sequence lA)) determine g,(49 by the same argument, and use finite additivity to conclude that g(A) = gtf1l g,(A9 is the limit of g(An) =l#y.)
=1g,(#j)
=
:4
=
=
,
=
-
P.(fX -
P.(XnC).
K
The finiteness of the measure is needed for the second part of the argument, but the result that g,tAul g(A) when ytu A actually holds generally, not excluding the case g,(A) x. This theorem has a partial converse: '
-->
=
3.5 Theorem A non-negative uous is countably additive.
set function g, which is finitely additive and contin-
Proof Let (#n) be a countable, disjoint sequence. If A,, U7=l, the sequence (An ) is increasing, Bn f'nAn-l 0, and so gtAnl g,(#n)+ g,(An-I) for every n, by finite additivity. Given non-negativity, it follows by induction that (g,(An)Jis ZTJ =Ip,(), whereas continuity implies that g,(A) monotone. lf A U7=tBj, g(A) LLIUTJ * lB =
=
=
=
=
=
=
.
Arguments in the theory of integution often turn on the notion of a set. In a measure space (f1,T,g), a set ofmeasure zero is (simplyenough) a set M e 5 with g.(M) 0. A condition or restriction on the elements of Q is said to occur almost everywhere (a.e.)if it holds on a set E and f E has measure zero. lf more than one measure is assigned to the same space, it may be necessary to indicate which measure the statement applies to, by writing a.e.lgl or a.e.gvq as the case may be. fnegligible'
=
-
3.6 Theorem (i) If M and N are T-sets, M has measure 0 and N i M, then N has measure 0. (ii) lf (A&) is a countable sequence with g(Mj) 0, V j, then g(UjMj4 0. (iii) If fe)) is a countable sequence with g,(r.'pl 0, V j, then g,((UjA))9 = 0. .
=
=
=
Measure Proof
39
(i) is
an application of monotonicity', (ii) is a consequence of countable additivity', and (iii) follows likewise, using the second de Morgan law. .
In j3.2 and j3.3 we will be concerned about the measurability of the sets in a given space. We show that, if the sets of a given collection are measurable, the sets of the c-field generated by that collection are also measurable (the Extension Theorem). For many purposes this fact is sufficient, but there may be sets outside the c-field which can be shown in other ways to be measurable, and it might be desirable to include these in the measure space. In particular, if g(A) g(A) wheneverd c E c #. This is g,(#) it would seem reasonable to assign jtt'l equivalent to assigning measure 0 to any subset of a set of measure 0. The measure space (f1,T,jt) is said to be complete if, for any set E e 5 with g.(F) = 0, a11 subsets of E are also in T. According to the following result, every measure space can be completed without changing any of our conclusions except in respect of these negligible sets. =
=
3.7 Theorem Given any measure space (f,;,jt), there exists a complete measure TB, (f,TB,g), called the completion of (f1,F,p), such that 5 and g(F) space c F(F) for al1 E e %. In =
Notice that the completion of a space is defined with respect to a particular has a different completion for each measure measure. The measurable space (f1,@) that can be defined on it. Proof Let
AB J;B
denote the collection of a11 subsets of i-sets of pmmeasure0, and LF
=
c
D: FA F e
AB
for some E e
.41
.
0, any set F c E satisfies the criterion of (3.7)and so is in FB as the definition requires. For F e @B, let F(#) g,(F), where E is any F-set satisfying E F e AB. To show that the choice of E is immaterial, let Ej and E1 be two such If jt(F)
=
=
sets, and note that
p,(A'lA&) Since jttFl t-p F2)
=
g,((FAF1)A (FAF2))
=
=
0.
g(F1 A F2), we must conclude
p,(F1 r'hE
+
g,tflr-hE
2 p.tfl k p,(F1 r'hF2)
(3.8) that
for i 1 and 2, or, g,(F1) p.(&). Hence, the measure is unique. When F e T, we AB, confirming that the measures agree on T. can choose E F, since FAF= O e TB is a c-tield colgaining ?F.Choosing E F in (3.7) It remains to show that AFC FB. 8/ shows If F e ;B, thenFA F e ABfor E e 5 and hence Ec for Fe 5 c TB. And finally AB where Ec if Fj e J;B foj e (N,there EF e 5, and so Fc e e exist Ej e 5 for j e EN,such that Ejh Fj e AB. Hence =
=
=
=
=
e U1 A U/) c UIF/AFJ ?
? by 3.6(ii). This means that
AB,
j
Ojb 6
TB,
and completes the proof. w
(3.10)
Mathematics
40 3.2 The Extension Theorem
You may wonderwhy, in thedefinition of ameasurable space, Fcould not simply be the set of all subsets; the power set of D. The problem is to tind a consistent method of assigning a measure to every set. This is straightforward when the space has a tinite number of elements, but not in an infinite space where there is no way, even conceptually, to assign a specitic measure to each set. It is necessary to specify a rule which generates a measure for any designated set. The problem of measurability is basically the problem of going beyond constructive methods without running into inconsistencies. We now show how this problem can be solved for c-fields. These are a sufticiently general class of sets to cope with most situations arising in probability. One must begin by assigning a measure, to be denotedgo, to the members of some basic collection C for which this can feasibly be done. For example, to construct the measure b a. Lebesgue measure we started by assigning to each interval (tz,&1 We then reason from the properties of go to extend it from this basic coltection to a11 the sets of interest. C must be rich enough to allow go to be uniquely defined by it. A collection C c 5 is called a determning class for (f,F) if, whenever jt and v are measures on 5, jt(A) v(A) for all A e C implies that g v. Given C, we must also know how to assign gmvalues to any sets derived from rby operations such as union, intersection, complementation, and difference. For disjoint sets A and B w have g)tz4QJ #) go(A) + go(#) by finite additivity, and when B c A, gotA B) go(A) 04#). We also need to be able to determine gotA fa #), which will require specitic knowledge of the relationship between the sets. When such assignments are possible for any pair of sets whose measures are themselves known, the measure is thereby extended to a wider class of sets, to be is closed and C are the same collection, but in any event denoted 'Y'.Often under various finite set operations, and must at least be a semi-ring. In the is typically either a field (algebra)or a semi-algebra. Example applications 1.18 is a god case to keep in mind. However, cannot be a c-field since at most a finite number of operations are permitted to determine go(A) for any A e .Y'.At this point we might pose the opposite question to the one we started with, and ask why might not be a rich enough collection for our needs. ln fact, events of interest frequently arise which cannot contain. 3.15 below illustrates the necessity of being able to go to the limit, and consider events that are expressible only as countably infinite unions or intersections of C-sets. Extending to the events 5 c(#) proves indispensable. We have two results, establishing existence and uniqueness respectively. -
=
=
=
-
=
-
./
-t/'
,/
'f
,/
,$
=
3.8 Extension ,/.
theorem (existence)Let $ be a semi-ring, and let go:
lf 5 = c(,$). there exists a measure jt on measure on ,YI. jL0t'l for each E e n
./
F--y
F+be
(D,?D,such that jt(F)
a =
Although the proof of the theorem is rather lengthy and some of the details are fiddly, the basic idea is simple. Take an event A c f to which we wish to assign a
Measure ./,
,/,
consider choosing a finite we have g,(A) go(A). If A measure g,(z4).If A G that is, a selection of sets Ej e or countable covering for A from members of OjEj. The object is to tind as 1,2,3,... such that A c ./, j a covering as possible, in the sense that Xgo(f;)is as small as possible. The outer measure of A is =
,/,.
teconomical'
=
P,*(X)
inf X.to(F./),
=
(3.11)
where the infimum is taken over a11tinite and countable coverings of zl by y-sets.
If no such covering exists, set g,*(A) x. Clearly, g7(A) gotAl for each A E g,* is called the outer measure because, for any eligible definition of g,(A), g,*(A) 2
Xg,(l
2
p,
j
Uf j
.9.
=
=
2 p,(A), for Ej
,Y'.
E
(3.12) ,/
The first inequality here is by the stipulation that p(Ej) goIFJfor Ej e in side majorant where exists, is infinite. The the case a covering or else the second and third follow by countable subadditivity and monotonicity respectively, because p, is a measure. We could also construct a minimal covering for Ac and, at least if the relevant outer measures are tinite, define the inner measure of 4 as jt+ (4) jt(AC) p,*(z4C). and g,*(A9 k jttAfl by (3.12), Note that since p,(z4) g,tfl g,*(f) =
=
=
-
-
p,(A) 2 g,+(A).
(3.13)
lf g,*(4) g+44), it would make sense to call this common value the measure of A, and say that A is measurable. ln fact, we employ a more stringent criterion. A set A g f is said to be measurable if, for any B c f, =
g,*(Af-'h B4
+
g,*tz4f chB)
=
(3.14)
jt*(#).
This yields g,*(A) p,+(A) as a special case on putting B f1, but remains valid even if g,tfll x. Let A denote the collection of all measurable sets, those subsets of f) satisfying (3.14).Since g,*(A) go(A) forA e and /1.:40) 0, putting 4 O in (3.14) gives the trivial equality g,*(#) g,*(#). Hence O e A, and since the definition implies that Ac E A if 4 e A, f E A too. The next steps are to detennine what properties the set function jt*: A !-+ F shares with a measure. Clearly, =
=
=
,/
=
=
=
=
jt*(A)
0 for all 4
C f1.
(3.15)
Another property which follows directly from the definition of p,*is monotonicity: X1 I X2 =
F*(X1) f >*(X2), fOr A1, 42 i
.
(3.16)
Our goal is to show that countable additivity also holds for g,* in respect of A-sets, but it proves convenient to begin by establishing countable Juhadditivity. 3.9 Lemma If fAy,j
E
EN J is any sequence of subsets of f, then
42
Mathematics
g,* UX./ f
/7 >*(X./).
(3.17)
Proof Assume g*(Ay) < x for each j. (If not, the result is trivial.) For each j, 1et fEjkq denote a countable covering of Aj by y-sets, which satisfies p.*(A>+ 2-:
< Xpo(,) k
: > 0. Such a collection always exists, by the definition of g*. Since LkkEjk, it follows by definition that U#.j ?
for any
p.*UA/ noting Z7=12S ity is strict. .
=
1.
f XpotA>l
<
j,k
j
(3.17)
Xp,*()+ e,
(3.18)
./
now follows since : is arbitrary and the last inequal-
The following is an immediate consequence of the theorem, since subadditivity supplies the reverse inequality to give (3.14).
3.10 Corollary A is measurable if, for any B c D, g,*(AC r''h#) K g*(#). n t *(A r''h ) +
(3.19)
The following lemma is central to the proof of the extension theorem. lt yields countable additivity as a corollary, but also has a wider purpose.
3.11 Lemma A is a monotone
class.
Proof Letting fAj, j e INJ be an increasing sequence of A-sets converging to 4 U#.j, we show A e A. For n > 1 and E e f1, the definition of an A-set gives =
g7(An fa E)
=
g*(4n-1 ra (An rh A=))+ g*(4k-1 f-'h (Anf''h F))
= g*(An-1 rhEI+ g7(& rn E). where Bn g,*(A()rn E)
= =
An -An-1, and the sequence 0', then by induction, ke(An chF)
=
(3.20)
(BjI is disjoint.
X pz*( rn E)
Put
ztttl
=
O so that (3.21)
y=1
holds for every n. The right-hand side of (3.21)for n e INis a monotone real x. Now, since An e A, g,*(4rh A>)as n sequence, and g,*(An(hE) .--)
-->
jt*(F)
r1*(Anrnm
+
h1*(Xrn)
k jt*(An(nE)
+
g*tAc rnfl,
=
using the monotonicity of g,*and the fact that from the foregoing argument that
AC
(3.22) Anf.Taking the limit, we have
sleasure jt*(A7 k g7(A r'A F)
+
43
g.*(AC
f'-h A'),
(3.23)
so that 4 e A by 3.10. For the case of a decreasing sequence, simply move to the complements and argue as above. . Since () is a disjoint sequence, countable additivity emerges as a by-product of the lemma, as the following corollary shows.
3.12 Corollary
If
()
is a disjoint sequence of A-sets,
g,*U j Proof Immediate on putting E
-
Xp,*().
(3.24)
j
in (3.21) and letting n
=
--> x,
noting VjBj
A. w
=
Notice how we needed 3.10 in the proof of 3.11, which is why additivity has been derived from subadditivity rather than the other way about. Proof of 3.8 We have established in (3.15)and (3.24)that g* is a measure for the elements of A. If it can be shown that 5 q A, setting g,(A) g,*(A)for al1 A e 5 will satisfy the existence criteria of the theorem. implies The t'irst step is to show that c A or, by 3.10, that 4 e =
.t/7
,/
g,*(A'r7A) + g,*(A'r-hAf)< g*(F)
(3.25)
,$1
denote a finite or countable covering of E such that Lj gotAyl < p,*(F) + E, for s > 0. If no such covering exists, g,*(A-) cxl by definis a ition and (3.25)holds trivially. Note that E r'A A c U/A; r-7A), and since /''AAC f-7WC), and by simple semi-ring the sets Aj ('''AAare in ,Y'.Similarly, E c Uyt>j algbra definition of and the set a semi-ring,
for any E i (. Let lAy e
=
,/
Aj r'Aytf
=
Aj
Aj chAl
-
=
UG:
(3.26)
k
where the Cjk are a finite collection of yksets, disjoint with each other and also with Aj r'hA. Now, applying 3.9 and the fact that g,*(#) k)(#) for B e we find .$,
=
g,*(A'faA) +
g*(Ff-$AC)
f
X jtot/j chdl X XgotGl +
j
j
k
p,*(eD =Xw)(z4y) ? <
+ :,
(3.27)
is tinitely additive, and Aj DA where the equality follows from (3.26)becau and the Cjk are mutually disjoint. Since s is arbitrary, (3.25)follows. Next, we show that A is a c-field. We have only to show that A is a field, because 3.11 implies it is also a c-field, by 1.24. We lready know that e A and A is closed under complementation, so it remains to show that unions of A-sets are in M. Suppose that Al and A2 are A-sets and E c f. Then .to
:,*(13 =
>*(A1 nA') + F*(Af rhEI
Mathematics
=
jt*(z42 c7z4jra e) +
g*(z4qrndl chE) + g,*(A2/'nAk ra A')+ jt*(A rnAf
f-h
E)
p,*(A2f-7Al ra A')
+ g*((Aq r7Al ra F)
f-h F)) rn E) k.p (Aj/-AAC
k.p (A2 chAf
+p7((42ra4j)cfaF), = jt*((A2rn41) cjE)
(3.28)
where the inequality is by subadditivity, and the rest is set algebra. By 3.10 this is sufficient for A1 f'742 e A, and hence also for A1kJA2 G A, using closure under complementation. It follows that A is a c-tield containing $, and since 5 is the smallest such c-field, we have that 5 i X, as required. . Notice that (3.28)was got by using (3.14)as the relation defining measurability. The proof does not go through using p.*(A) gx(A) as the definition. The style of this argument tells us some important things about the role of .Y'. Any set that has no covering by '-sets is assigned the measure x, so for finite measures it is a requisite that f q LbEj for a tinite or countable collection fEj o? e #') The measure of a union of y-sets must be able to approximate the measure any F-set arbitrarily well, and the bagic content of the theorem is to establish that a semi-ring has this property. To complete the demonstraiion of the extension, there remains the question of uniqueness. To get this result we need to impose c-finiteness, which was not needed for existence. =
.
3.13 Extension theorem (uniqueness)Let jt and g,' denote measures on a space (f1,T), where 5 c(.$), and is a semi-ring. If the measures are c-finite on and g,(A') jt'(F) for a1l E e .T, then g(An) g'(F) for a1l E e %. .$
,/
=
=
=
Proof We
tirst prove
the theorem for the case of finite measures, by an applitheorem. Define .J= fE e F: p.(F) = p,'(AD ) Then $z ,J by hypothBy 1.27, the proof is completed esis. If is a semi-ring, it is also a l-system. contains c(#). show hence if Fe can that .J is a l-system, and and condition 1.26(a) holds. Additivity When the measure is finite, f e implies that, for A e
cation of
the 7:-
.
./
.4
.4,
F(X3
so that 1.26(b) additivity,
=
F(f)
-
XX)
=
LUIf) -
$(A)
=
>'(A3,
(3.29)
holds. Lastly, 1et fA.j) be a disjoint sequence in co
p,
UAp
j=3
X
oo -
77p,(A> j=
-
.4.
By countable
G)
Xp,-+) -
j=1
p,' UAk
(3.30)
,
y=1
and 1.26(c) holds. It follows by 1.26 and the 7:- theorem that 5 c(./) ? W. VjBj where Bj e and pBp Now consider the c-finite case, Let ( g,'(#j) < co. % fBj rnA: A e 5 J is a c-field, so that the (Bj,5) are measurable spaces, =
./
=
=
=
Measure
45 ./
fe'h Bj. The preceding argument on which jt and $ are tinite measures agreeing on showed that, for A e 5, g(#j rnz4) bt'Bj rnA) only if g and g,' are the same =
measllre.
Consider the following recursion. By 3.3(ii) we have p.(Ara (#1t..p#2)) Letting Cn
=
f''h #2) g.(X f''h #1 f''h #2). g,(z4r'7#1) + h1(z4
=
U7=l the
(3.31)
-
same relation yields
(3.32)
rn cn) p.tz4rn Bn) + p,tA ra cn-j) p,tz4rn Bn rn cn-1). g,ta4 =
-
The terms involving Cn-l on the right-hand side can be solved backwards to yield f-'h an expression for g,(A G), as a sum of terms having the general form g,(Ara Bjj rn Bjzra Bj, ra
for some j, say j that gtD ra Bj) that in (3.32) =
=
j, in which ease D
jt'(D ra B? for all D g.(Afa Cn)
olds for any n. Since Cn
This
=
G
=
=
=
g,to ra Bj)
(3.33)
< x
A fn Bh fe'h Bh fa e F. Since we know preceding by 5 the argument, it follows ...
g,'(4 fa Cn4. f as n
--)
F(A)
)
...
-->
x,
(3.34) we obtain in the
F'(A),
limit
(3.35)
the two sides of the equality being either finite and equal, or both equal to This completes th proof, since A is arbitrary. w
+x.
3.14 Example Let A denote the subsets of R which are measurable according to (3.14) when g,* is the outer measure defined on the half-open intervals, whose measures go are taken equal to their lengths. This defines Lebesgue measure m. 'hese sets fol'm a semi-ring by 1.18, a countable collectioh of them covers R, and the extension theorem shows that, given m is a c-finite measure, A contains the Borel field on R (see 1.21), so (R,f,?n) is a measure space. It can be shown (we wonpt) that a11 the Lebesgue-measurable sets not in B are subsets of T-sets of includes a1l measure 0. For any measure g, on (R,f), the complete space @,SB,F) of the Lebzsgue-measurable sets. In The following is a basic property of Lebesgue measure. Notice the need to deal with a countable intersection of intervals to determine so simple a thing as the measure of a point. 3.15 Theorem Any countable set from R hs Lebesgue measure 0. Proof The measure
of a point
(.x) is
zero, since for x e R,
@Q
fx) ppd :
.
'
'
??z((<))q
'
: ''
' '
=
limn-jxl/n
=
=
r)
n=1
(x
-
1/n, xj e B
0. The result follows by 3.6(ii). w
(3.36)
Mathematics
46 3.3 Non-measurability
To give the ideas of the last section their true force. it needs to be shown that D X c 2 i s poss ible in other words, that ( can contain non-measurable subsets. In this section we construct such a set in the half-open unit interval (0,11,a standard counter-example from Lebesgue theory. For m y e (0,11, define the operator ,
.#4.x
A'+
1,
(3.37)
=
y+x- 1,
This is addition modulo 1. Imagine the unit interval mapped onto a circle, like a clock face with 0 at the top. y x is the point obtained by moving a hand clockwise through an angle of 2= from an initial point y on the circumference. For each set A (0,11, and x e (0,1J, detine the set .i.
A J.x
=
(y 4.x:
AJ.
l
y
(3.38)
3.16 Theorem If A is Lebesgue-measurable so is A %x, and m(A 4.a') any
=
mA), for
X.
for any real x such b +x1) b c lna,bj), unions > A-x + of intervals extends < 0 and b 1. The that a x to finite property translated by x. If A is any Lebesgue-measurable subset of (0,11,and A A-x (0,11 where A + x (y+x: y e A J, the construction of the extension similarly is measurable and mA) mA +x). implies that A and A2 A f-h (1 Now let A1 A /'7 (0, 1 11. Then pztdl +x) -(A1) and mAl + x 1) mA1), where the sets on the left-hand sides of these equalities are in each case contained in (0,11. A1 +x and A2 +x 1 are disjoint sets whose union is A %x', and hence Proof For
(J,!71
(0,11,ma
i
+x,
=
=
-
=
..x
=
-x),
-
-x,
=
=
=
=
-
mA 4.x)
rntz4l +x)
=
+
mAz
= rn(A1) + mA
+
=
x
-
1)
m(A). w
(3.39) .i.
Define a relation for points of (0,1q, letting xRy if y = x r for r e :. That is, xR y if y is separated from x by a rational distance along the cifcle. R is an equivalence relation. Defining the equivalence classes Ex
=
fy: y
.i.
=
.x
r, r e
t1),
(3.40)
(fI,
the sets of the collection x e (0,11 ) are either identical, or disjoint. distance is rational from some other point of the interval, these Since every x a sets cover (0,11.A collection formed by choosing just one of each of the identical sets, and discarding the duplicates, is therefore a partition of (0,11. (FA, Write this as x 6 C), where C denotes the residual set of indices. Another example may help the reader to visualize these sets. In the set of integers, the set of even integers is an equivalence class and can be defined as 0 the Ket of inteeers which differ from 0 an even integer. Of course F0 El = =
sfeasure
47
Eln, for any n l Z. The set of odd integers can be detined similarly as F2n+1 E 1 the set o f integers differing by an even integer from 1. F1 = F3 (A'A, for any n e Z. Discarding the redundant members of the collection e 11 F1 0 collection LE partition of the Z. d leaves just J to e tine a Now constnlct a set H by taking an element from Ex for each x e
#
=
=
...
=
,
...
= .x
,
3.17 Theorem H is not Lebesgue-measurable. Proof Consider the countable collection (H # r, r l (I)) We show that this collection is a partition of (0,11. To show disjointness, argue by contradiction. Suppose z e H rl and z e H r2, for rl # r2. This means there are points 1, hl e H, such that .
'i.
.i.
.i.
l rl
.i.
=
z
=
(3.41)
hz r2.
If rl # r1, we cannot have Jll = 2; but if Jll # hl then Jll and hz belong to different equivalence classes by construction of H, and cannot be a rational distance lrl r2 I apart; hence no z satisfying (3.41)exists. On the other hand, LjrH r), and consider any point let H e (0,11. belongs to one of the rational distance of some element of H; equivalence classes, and hence is within a but H* contains al1 the points that are a rational distance r from a point of H, for some r, and hence x e H., and it follows that (0,11c S*. mlH4 for all r. Since mH*) k Suppose mllt exists. Then by 3.16, mH% r?z((0,1J) 1, we must have m+ > 0 by 3.6(ii), but countable additivity then rnzbul'-l r) which is impossible. It follows that m+ does not gives mH.) -
.i.
.&
.x
=
=
=
=
=
=,
exist. w The definition of H involves a slightly controversial area of mathematics, since the set o f equ ivalence classes is uncountable. It is not possible to devise, even in principle, constructive rules for selecting the set C, and elements from Ex for each e C. The proposition that sets like H exist cannot be deduced from the axioms of set theory but must be asserted as an additional axiom, the so-called axiom of choice. lf one chooses to reject the axiom of choice, this counterexample fails. We have made no attempt here to treat set theol'y from the axiomatic standpoint, and the theory in Chapter 1 has been what is technically called nat've is). For us, the problem of (i.e. based on the intuitive notion of what a the axiom of choice reduces to the question: should we admit the existence of a mathematical object that cannot be constructed, even in imagination? The decision is ultimately personal, but suffice it to say that most mathematicians are willing do SO Sets like H do not belong to (0,1j = f# r'h(0,1j, B e B ). It is not difficult to show that a11the sets of (o,1j are Lebesgue-measurable', see 3.14 and restrict m to (0,1) as in 3.2. By sticking with Borel setf we shall not run into measurability difficulties on the line, but this example should serve to make us careful. ln less familiar situations (suchas will arise in Part VI) measurability cah fail in superficially plausible cases. However, if measurability is in doubt one might remember that outer measure g,*is well defined for a1l subsets of f, and coincides vk
tset'
X
to
.
Mathematics
48
with z whenever the latter is defined. Sometimes measurability problems are dealt with by working explicitly with outer measure, and forgetting about them.
3.4 Product Spaces If (f1,T) and (E,T) are two measurable fxE
((,():
=
(t) l
spaces, 1et
f,
(
G
El
(3.42)
be the Cartesian product of f and E, and define ?F@V Spg
=
((Fx G), F e @, G
G
=
c(Spg), where
V).
(3.43.)
a product space, and (f,F) and (E,W) are the coordinate factor spaces, or spaces, of the product. The elements of the collection Sys are called the measurable rectangles. The rectangles of the Euclidean The space
(f1x E, 5 * V) is called
plane R x R
=
R2
(produc ts
of intervals) are a familiar case.
3.18 Example Rather trivially, consider the two-element setsA B f(1,(21e N. The corresponding rectangle is
=
(1,(1
e @,and
=
A XB
=
1), (t,)1,t2), (*2,(1),()2,t2)l. 1(t,)1,('
The sets f(1,(1), ((*,,(2))and ((1,(2), (2,(1)) are not rectangles, unions of rectangles and so are elements of 5 @ T. n
but are
Two important pieces of terminology. If E c f x E, the set an(F) (: (,() e F) is called the projection of E onto f1. And if A D, the inverse projection of A is the set =
o-1(A)
A xE
=
=
(4),4):
)
e A,
(
EJ
e
(3.44)
.
A x E is also called a cylinder set in f x E, with base A. The latter terminology is natural if you think about the case = R2 an d E R Cylinder sets with bases in 5 and T are elements of R,e. One might tlnk that if E e 5 @ N, antA-) should be an F-set, but this is not necessarily the case. :tE'f # ln(M) in general (see is not closed 1.3) so that the collection C of projections of 5 (&N-sets onto = notice under complementation. However, that A ao(A x Z) so that 5 i C. The main task of this section is to establish a pair of results required in the construction of measures on product spaces. =
.
3.19 Theorem If C and b are semi-rings of subsets of and E, respectively, then Sws = (CxD: C e C, D e 01 is a semi-ring of
x E.
Proof There are three conditions from 1.16 to be established. First, Ses clerly contains 0. Second, consider C1, Czq C, andDlyDz G 0. Q rn Czq CandDl rhDZ e D, and as a matter of definition, (C1 xD1) rn (Q xD
=
()
e
Q, (
c Dj
) fa (
e
Q,
(
e Dal
Measure
49
= ((o e cl c2, ( e ol = (c1ch ca)x (D1 cjD1) e Rwe. fazhl
fa
Third. assume that C1 xo2)
xDl i Cz xD2, and by a similar argument,
CI
(c1 xD1)
-
(3.45)
((
=
(
e Cz,
= ((C2
-
e D2): either
cl or (
)
D1l
C1) XD1) tp (C1 X (D2 -D1))
tp ((Q
C1) X (D2 -D1)).
-
(3.46)
where the sets in the union on the right-hand side are disjoint. By hypothesis, the sets Cz C1 and D1 DI are finite disjoint unions of C-sets and O-sets respectively, say (47,...,Cn'), and (D(,...,D,,,'). The product of a finite disjoint union of sets is a disjoint union of products; for example, -
-
n
n
U
/=1
C; x D1
(tl),():(t
=
Uts (
e
,
e D1
=
UICJx D1).
(3.47)
./=1
j=1
Extending the same type of argument, we may also write m
n
(ca cllxtoa-ol) -
-
Utclxol) U(c1xoi) k-p
kp
k=1
.j=1
U(cJxo;) (3.48) .
j,k
All of the product seys in this union are disjoint (i.e., a pair ((t),))can appear in at most one of them) and al1 are in Scs. This completes the proof. . The second theorem leads to the useful result that, to extend a measure on a productspace, it suffices to assign measures to the elements of Scs, where C and D are suitable classes of the factor spaces.
3.20 Theorem If 5 = c(C) and T c(O) where C and D are semi-rings of subsets of f and E respectively, then 5 @ V = C @ 0. =
Proof It ij clear that Rwo i Sps, and hence that C (8)D converse, consider the collection of inverse projections, .t/a (J:o-1(#), F e Fj pwg.
5 (8)W.To show the
c
=
,/p
is a c-field of fl x E, and is in fact the smallIt can easily be verified that is a (;o'(C), C e C J i C (>))0. est c-field containing the colletion and since it follows (& is and hence D c-field by the T l-system, a a theorem that c(#c) ? C @)0. Exactly same conclusion holds for #s, the orresponding collection for V. Evel'y element of Sss is the intersection of an element from #v and one from #'g, and it follows that Spg c C @D. But Sys is a l-system by 3.19 and hence a further application of the 7:- theorem gives 5 (8)N ? u gs; , .1
vt/c
,t/c
=
-system,
xt/p
'
=
irlik pztjlptlof > product extends beyond pairs to triples and general n-tuples, and gd jjjajj be jnterested in the nrooerties of Euclidean a-qnnne ffRlh '
y:
,;r-.
..
.. .
. .
.
.
jjtyj)jjjujf
50
Mathetnatics
For tinite n at least, a separate theory is not needed because results can be obtained by recursion. If (Ylf,X)is a third measurable space, then trivially, fxE
x'P
((,(,v):
=
(
e f,
(,)
= f((,(),v): (,() E) xT. = (f x
e E,
ve
e DXE,
v
t1:l
e .P1
(3.49)
Either or both of (f),T) and (E,V) can be product spaces, and the last two theorems extend to product spaces of any finite dimension.
3.5 Measurable Transformations
F: D
F->
(E,N) in a different context, as domain
and
Consider measurable spaces (f,T) and codomain of a mapping E.
if F -1(#) l 5 for a11# l N. The idea is that a to be sl%-measurable measure p, defined on (,T) can be mapped into (E,V), every event B e V being assigned a measure v(#) g,(F -1(.#)) We havejust encountered one example, the projection mapping, whose inverse defined in (3.44)takes each T-set A into a measurable rectangle. Corresponding to a mesurable transformation there is always a transformed in following the sense. measure, F is said
=
.
3.21 Theorem Let jt be a measure on (f,F) and F: f ation. Then P.F -1 i s a measure on (E V) where ,
jLF -1(#)
E a measurable
F-
transfonu-
,
-1
=
(#)), each
g,(F
B e g,
(gj5o
.
.
Proof we check conditions 3.1(a)-(c). Clearly g,F -1(4 ) 1 0 a1l A e Bv. Since holds by definition, T-1(O) T -1(E) O by 1.2(iii) and so g,F-1(3) p,(T -1(O)) g,(O) 0. For countable additivity we must show ,
=
=
=
=
=
g,F-1
for a disjoint collection that the B; are disjoint becomes
U#y j
=
X/tF-1()
(3.51)
j -1
F Bj4, 1.2 shows both Bj,B1,... e E. Letting B) and that F-1(Uy#y) Equation (3.51) therefore Uy#J. =
=
p,Ur j
=
X g,(#J)
(3.52)
./
for disjoint sets Bgq,which holds because g, is a measure. The main result o
general transformations
.
is the following.
3.22 Theorem Suppose F -1(#) l 5 for each B e D where D is an arbitrary class of sets, and N c4D). Then the transfonnation F is T/v-measurable. ,
=
51
Measure
(
$.,
.
.
.
,
.
,
=
,
j'
T-3LjBj)
T-3Bj)
q @,j q s, then proof By 1.2(ii) and (iii), if l then @ @ It follows d if z'-1(#r) z'-1(#)c z'-1(,) E e ?yan
,
k
=
UjF-1()
that the class of
CtS
a is a c-field.
Since D
t,: w-1(s)s s)
.
N c 4 by definition.
.4,
c
.
This result is easily iterated. If (t1#,J't)is another measurable
is a V/x-measurable transformation, then &oF: (
F-+
space and U: E
F->
VIJ
VP
is F/s-measurable, since for C -1
R, W 1(C)
E
-1
(&0D (0
F
=
N an d hence
E
,
-1
(&
(C))
9.
1
(3.53)
E is called a measurable isomorphism if it is An important special case: F: f' 1-1 oflto, and both F and F -1 are measura ble The measurable spaces (2,1) and (E,N) are said to be isomorphic if such a mapping between them exists. The implication is that measure-theoretic discussions can be conducted equivalently in either (f,T) or (E,V). This might appear related to the homeomorphic property of realfunctions, and ahomeomorphismis indeed measurablyisomorphic. Butthereis no implication the other way. -..:
.
r0,11 F--> (0,11,defined
3.23 Example Consider g: #(A)
x+
,1
0 f x f 1 2
x
1
l <
=
by
(3.54)
.
-
2,
1)
x < 1
Note that g is discontinuous, but is 1-1 onto, of bounded variation, -j Sgo,lj/fgo,lj-measurableby 3.32 below, and g g. I::I
and hence
=
The class of measurable transformations most often encountered is where the codomain is (R,O), B being the linear Borel tield. In this case we speak of a function, and generally use the notation f instead of F. A function may also have the extended real line (RlW)as codomain. The measurability criteria are aj
follows.
3.24 Theorem (i) A function /': f
R for which (): J(() f x) e ?Ffor each x E (Dis So is a function foz which (): fk < x) e 5 for each
F-+
T/f-measurable. x e (D (ii) A function fl f.1 F-> 2- for which f(t): Q tp (+x) t.p (-x) is T/f-measurable. .
.f4))
f xl e ?F for each x e -1(#)
Proof For case (i), the sets f(t): flk f xl are of the form f B e T' where C is defined in 1.21. Since B c(C), the theorem follows by 3.22. The other cllection indicated also genertes :B, and the same argument applies. The extension to case (ii) is equally straightforward. . ,
=
52
Mathentatics
The basic properties of measurable functions follow directly.
3.25 Theorem (i) If f is measurable, so are c + f and c#, where c is any constant. (ii) If f and g are measurable, so is f + g. Proof If .X
e
jq
f
K x, then
f+
c S x + c, so
that f
c is measurable
+
by 3.24. Also, for
,
(: (:
c/)
XJ
S
/*()
((t): /)
=
< xIc ),
c
>
0
< xlclc,
c
<
0
c
=
c
=
f1, 0,
where for each of the cases on the right-hand side and each x/c in 5, proving part (i).
If and only if lt follows that
(:
J())
f+g
+.g((t))
<
there exist r e ) such that
.x,
U (2
.z')
<
=
f(l
<
r)
f-n
(:
(3.55)
0 and x k 0 0 and x < 0
f
<
r
<
R the sets are
6
x g -
:(t0) < x- r1.
(see 1.10). (3.56)
rq Q
The countable union of T-sets on the right-hand side is an T-set, and since this holds for every x, pal't (ii) also follows by 3.24(i), where in this case it is convenient to generate tB from the open half-lines. >
Combining parts (i) and (ii) shows that if h,...,fn are measurable functions so is Xhjcjfj, where the cj are constant coefficients. The measurability of suprema, infima, and limits of sequences of measurable functions is important in many applications, especially the derivation of integrals in Chapter 4. These are the main cases involving the extended line, because of the possibility that sequences in R are diverging. Such limits lying in V are called extended functions. 3.26 Theorem Let f/l be a sequence of F/s-measurable functions. Then infn/n, supn/k, liminfln, and limsuw/ are F/W-measurable. Proof For any x e
t:
F (: fnlk
f xl s 5 for each n by assumption. Hence
,
supnlnt)
<
.z')
=
Of):
n=1
fnk
so that supn/ is measurable by 3.24(ii). Since infn/'n obtain (:
infn/t)
.x)
<
(: sup,,(-Jn())> -xJ = (: supn(-Jn())S -xJf =
@,
S xl e
-supnt-ln),
=
(3.57) we also
Measure
53 C
00
O
n=1
la''. -Tn(a')K
-xl
X
=U1e):/()
<
5.
xl e
(3.58)
n=1
To extend this result from strong to weak inequalities, write X
f xl
infn/kt)
(:
O(:
=
infn/kt)
<
x+
?n=1
Similarly to
(3.59)
5,
(3.60)
may show
(3.57),we
1:
1/rlJ e 5.
supknfkt)
f
Ot:
.x1
=
.fn()
S
.x1
e
kkn
and applying (3.59)to the sequence of functions gn (:
limsupn/t)
sums/k
=
yields
K xl e F.
(3.61)
In much the same way, we can also show f x) e @.
( (t): liminfn/t)
(3.62)
condition of 3.24 is therefore satisfied in each case.
The measurability
.
We could add that limn/kt) exists and is measurable whenever limsupn/t) liminfs/kt). This equality may hold only on a subset of f1, but we say fn converges a.e. when the complement of this set has measure zero. The indicatorfunction 1s()) of a set E e 5 takes the value 1s() 1 when (.t)E characteristichmction otherwise. and 1s((l)) 0 Some authors call ls E, of F. the It may also be written as fs or as zE. We now give some useful facts about indicator functions. =
=
=
3.27 Theorem (i) 1s() is 5IB measurable 1 1s(). (ii) 1sc()) (iii)1Ujsj((l)) sup 1sf(). =
if and only if E
G
F.
-
=
i
(iv) lnjsjt)
inf 1s/)
=
i
=
I-l1sj((l)). i
Proof To show (i) note that, for each B e .S,
1F-1
(#)
'Fhse
sts
ithediyte
=
if 0 G B and 1 G B E if 1 e #, 0 # Ec if 0 e B, 1 , B 0, otherwise
(3.63)
are in 5 if and only if E e @. The other parts of the theorem fwm tlw dfinition. .
re
Mathematics
54
Indicator functions are the building blocks for more elaborate functions, constructed so as to ensure measurability. A simplefunction is a T/o-measurable function J: fl F...: R having finite range', that is, it has the fonn
ftl
af, Xaflsjtt9l =1
=
=
tt e Ei,
(3.64)
where the G1,...,(xa are constants and the collection of T-sets F1,...,Fa is a tinite partition of f1. T/f-measurability holds because, for any # 6 B,
J-1(,)
U Ei s
=
ai
@.
(3.65)
B
EE
Simple functions are ubiquitous devices in measure andprobability theory, because many problems can be solved for such functions rather easily, and then generalized to arbitrary functions by a limiting approximation argument such as the following.
l
Fig. 3.1
3.28 Theorem If f is T/o-measurable and non-negative, there exists a monotone n e EN I such thatnlt) f sequence of T/f-measurable simple functions f/'(n), for evel'y (l) e f1. '
1,...,n2'l, consider the sets Ei = f ): i 1)/2' < il2n ) Proof For i f Augment these with the set Enzn.. f : J() k nl This collection corresponds to a n2n + l-fold partition of (0,x) into S-sets, and since f is a function, each and hence belongs to one and only one Ei. The Ff maps into one and only one therefore constitute a partition of f1. Since J is measurable, Ei s 5 for each i. Define a simple function hn)on the Ei by letting i i 1)/2/, for i = 1,..., nln + 1. Then fn) f f, but A+1())k /4)) for every ); incrementing n bisects n+1)()) is equal to either i 1)/2*, each interval, and if n)()) n)(), 1)/2'+1 &+1 > follows that the sequence is 1)/2 2/ It li or ( J. (). =
-
.
=
.
.f4)),
=
=
-
=
-
-
-
sieasure
55
J(). This holds for each ) e f. To extend from monotone, and lims-yxnlt) non-negative to general functions, one takes the positive and negative parts. Define J+ maxlf,ol and ff'b f, so that both f- and J- are non-negative functions. Then if f'Injand j'-n) are the non-negative simple approximations to it is clear that f+ and f- defined in 3.28, and fnj f =
=
=
-
.f+(n)
-(n),
=
-
l/-J(n)I < lJ+-.f+(n)l+ 1.f---(n)I Fig. 3. 1 illustrates the construction for n is a function on the real line.
=
0. w
--.h
2 and the case
(3.66) =
R, so that
.f()
3.6 Borel Fqnctions lf J is a measurable function, and 5'; S g R, 'T g R is a function of a real variable, is the composite function gof measurable? The answerto this question is yes if and only if g is a Borel hmction.Let Ss (# ros: B e S), where B is the Borel field of R. fs is a c-field of subsets of S, and B DS is open (closed)in the relative topology on S whenever B is open (closed).in R (see 2.1 and 2.3). fs is called the Borel field on S. Define fv similarlywith respect to 7. Then g is called a Borel function (i.e., is Borelmeasurable)if g -1 B) e B s for all sets B l f.r.
gl S
F-+
=
3.29 Example Consider gx) Ixl g -1 ta kes e ach point of R+ into the points x is the set For any B E B + (th e res triction of B to R+) the image under and for each a7 e #, which is an element of f. E1 containing the points x and =
.
.g-1
-x.
-x
1 if is rational, 0 otherwise. Note that (D e B (see 3.30 Example Let g(x) -1 3.15), and g i s defined according to (3.63)with E Q, so g is Borel-measurable. I:a .:
=
=
non-measurable ln fact, to construct a following. but the obvious case is the
4plausible'
function is quite difticult,
3.31 Example Take a set A f; for example, let A be the set H defined in 3.17. l Now construct the indicator function 1x@): R F-> (0,1 ) Since 1x - (f 1 1) A gp Shis function is not measurable. Ia .
=
,
Necessary conditions for Borel measurability are hard to pin down, but the following suftkient conditions are convenient.
3.32 Theorem lf g'. S it is Borel-measurable. Pmof
(i) follows
F-+
'T is either
immediately j cotinuity implies that h B) in T, by 2.17. To prove (ii), consider first prperty h@l S @)when y < ..
(i) continuous
or
(ii) of
botlnded variation,
from 3.22 and the definition of a Borel field, since is open (closed)in S whenever B is open (closed) a non-decreasing function h: R F.-y R, having the x; if A (y: h@l S (x)), sup A = x and A is one =
Mathematics
of (-x,x) and (-=A, so the condition of 3.24 is satisfied. So suppose g is nondecreasing on S; applying the last result to any non-decreasing h with the property hx) = x e S, we have also shown that g is Borel-measurable because = -1 h-'B4 5') f''ASe Ss, for each Brh'T e O.r. Since a function of bounded g CBt''h variation is the difference of two non-decreasing functions by 2.20, the theorem now follows easily by 3.25. . '(x):
This result lets us add a further case to those of 3.25.
3.33 Theorem If J and g are measurable, Proof
jg
=
with 3.25(ii).
z1((J+:)2
-
fl
gl)
-
so is
fg.
and the result follows on combining 3.32(i)
.
The concept of a Borel function extends naturally to Euclidean a-spaces, and indeed, to mappings between spaces of different dimepsion. A vector function 7,' S i R 1- R'/ gl S --./
,
is Borel-measurable if g -1(#) e B s for a11B e Sv, where Bs and fv (# r-75': B e f'''J.
=
(# taS:
# e
l
=
3.34 Theorem
lf g is continuous. it is Borel-measurable.
Proof By 2.21. w Finally, note the application of 3.21 to these cases.
3.35 Theorem If jt is a measure on (R Bk) an d g : S s.y is Borel-measurable where S c Rk an d -Erc R'/ gg is a measure on (T,Sv) where 'r
,
-1
,
gg-1(B)
t-1(s))
=
(3.67)
,
for each B e f.r. n R'N
for m < k. If X is A simp le example is where g is the projection of R onto k x 1 with partition X' = (21,7+'.),where X. is m x 1 and X.. is k m) x 1, 1et gl R k F-> R*' be defined by -
r(A') -1
In this case, g,g B)
=
X..
-1
=
P.W(#))
=
jt(#
xR
(3.68) k-m
)
or p s qrrl .
4 Integration
4. 1 Construction of the lntegral The reader may be familiar with the Riemann integral of a bounded non-negative function f on a bounded interval of the line gJ,:J, usually written fdx. The objects to be studied in this chapter represent a heroic generalization of the same idea. lnstead of intervals of the line, the integral is defined on an arb-
itral'y measure space. Suppose
(f1,@,g)is :
l'1
O
a measure space and
2-6
is a T/s-measurable function into the non-negative, extended real line. The integral of f is defined to be the real valued functional
jfdb
Sup
=
tXj
fte)ljpfj
tinsfo
(4.1)
where the supremum is taken over a11finite pmitions of into sets Ei e T, and the supremum exists. lf no supremum exists, the integral is assigned the value +x. 5 T he integral of the function IAJ,where 1x() is the indicator of the set 4 e 5, is called the integral of f over A, and written fdbk. The expression in (4.1) is sometimes called the lower integral. and denoted sfdp. Likewise defining the upper integral of f,
j'ydb -
inf
E i
(supsy/j bte'i)
(4.2)
,
G
f from below and from we should like these two constnzctions, approximating .fdbk J'/W/whenever above, to agree. And indeed, ttis possible to show that f *fdbk oo if either the set > 0) has is bounded and g(f1) < x. However, (: infinite measure, or f is unbounded on sets of positive measure. Definition (4.1) is preferred because it can yield a finite value in these cases. =
.f()
=
4.1 Example A familiar case is the measure space ('R,f,-), where m is Lebesgue where f is a Borel function is the Lebesgue integral of measure. The integral Ifdm f. This is customarily written j'fdx,reflecting the fact that mx, x + #x1) tfx, even though the sets (A'j) in (4.1) need not be intervals. n =
4.2 Example Consider a measure space (R,f,g,) where g, differs from m. The where f is a Borel function, is the Lebesgue-stieltjes integral. integral Ifdbk'
Mathematics
58 The monotone function Fxt
=
p,((-x,x1)
(4.3)
F(b) F(c), and the measure of the interval has the property (x, x + #< can be written JF@). The notation IfdF means exactly the same as J/Wg.,the choice between the p. and F representations being a matter of taste. See j8.2 and j9.1 for details. n g,((J,:1)
=
-
For a contrast with these cases, consider the Riemann-stieltjes integral. For an 1et a partition into subintervals be defined by a set of points H interval Etz,!71, Akl, < < xn with a = xo < b. Another set H' is called a = (x1,..., refinement of Fl if IR c H'. Given functions f and a: R 1-+ R, let .x1
=
...
5lRvas.fl
=
77f(f)(G(A)
-
a(zf-I)),
(4.4)
i=1 bafd%,
here ti e (xf-l,xf). If there exists a number there is a partition IQ with W
(&l-I.a,.f)
'-jajd
such that for every : > 0
b < 6:
l
for all FI Q IQ and every choice of fff), this is called the Riemann-stieltjes integtal of J with respect to a. Recall in this connection the well-known formula for integration by parts, which states thqt when both integrals exist,
fbtub)
=
/J)a(J)
+
jafd ja@f' +
When (x = x and #is bounded this definition yields the ordinary Riemann integral, and when it exists, this always agrees with the Lebesgue integral of f over gtz,:). Moreover, if a is an increasing function of the form in (4.3),this integral is equal to the Lebesgue-stieltjes integral whenever it is defined. There do exist bounded, measurable functions which are not Riemann-integrable (consider3.30 for example) so that even for bounded intervals the Lebesgue integral is the more inclusive concept. However, the Riemann-stieltjes integral is defined for more general classes of integrator function. In particular, if f is continuous it exists for a of bounded not necessarily monotone. These integrals therefore fall variation on gtz,&), the class defined by (4.1), although note that when a is of bounded variaoutside tion, having a representation as the difference of two increasing functions, the Reimann-stieltjes integral is the difference between a pair of Lebesgue-stieltjes
integrals on (J,:1. The best way to understand the general integral is not to study a particular measure space, such as the line, but to restrict attention initially to particular classes of function. The simplest possible case is the indicator of a set. Then, every partition fEiI yields the same value for the sum of terms in (4.1),which is
jgdn
Integration
59
jxdp
(4.6)
BIX),
=
=
5, the integral is undefined.
for any A e 5. Note that if A
Another case of much importance is the following.
4.3 Theorem lf J
0 a.e.lgl, then
=
fdp
Proof The theorem says there exists C e C. For any partition (Fl,...,&J 1et A'; ity of g,,
77 i
inf (l) e
Ei
/.() g,(e)
=
Xi
inf (l) e
F;
=
0.
1, such that /4)) 0 for ) Ei rn C, and A': Ei F;. By additiv-
with
gtc)
=
=
=
=
/.()l g,(e':) +
Xi
inf f e e:
-
g,(rJ)
= 0,
(4.7)
the first sum of terms disappearing because /*4) 0, and the second disappearing =
by
.6(i)
since pE'i) f g,(C')
0 for each i.
=
.
A class of functions for which evaluation of the integral is simple, as their name suggests,is the non-negative simple functions.
4.4 Theorem Let (?() 5 is a partition of f.
=
where aj k 0 for X:=1%1sj()),
i
=
1,...,n, gnd EL,...,En E
Then
tFp, 77
(4. 8 )
ipEi).
=
=1
Proof Consider an arbitrary finite partition of f1, A1,...,4-, and define inftxxyst). Then, using additivity of g, Ip,(Ay) X SX 77 f=1 .j=1 j=3 =
<
i=1
xi
I
=
p,(AyraEi4
jtt/tjc
Ei4
j=L
(4.9)
ipEi),
= f=1
where the inequality uses the fact that I assumes the smallest value of Gj such that Aj ra Ei # 0, by definition. The theorem follows, given (4.1), since (4.9) holds as an equality for the case m n and Ai = Ei, i 1,...,n. w =
=
So for functions with finite range, the integral is the sum of the possible values of f, weighted by the measures of the sets on which those values hold. Look at Fig. 3.1. The Lebesgue integral of the approximating function fz) in the figure is the sum of the areas of the rectangular regions. To compute the Lebesgue-stielties inteeral with resoect to some measure u.. one renlnces the width of the qetq
62
Mathematics
= yaj
a simple function. Hence,
ja-bdb
+ tzj,
s Af ra Bj,
(t)
(4. 17)
X Xtla, +bbbviAil-bB?
-
j
i
=
zxGf7r/ttArnl
=
'7rafgtA
=
i
,xfs>7/ttAf
+
j
j
rn B?
i
bTfbbkB)
+
j
i
ajdp
+
bjydp,
(4.18)
showing that linearity applies to simple functions. Now applying 4.6,
jaf
+
bgldp
ja jygjtj jt?lj1jsyusp, (suspy +
sup
=
bz4dbk
(pf/f'
=a =
+
b
#
ajfdp
+
bjgdp.
(4. 19)
To extend the result to general functions, note that
la.f
1 If I + Ib I Ig l
bg l < a I
+
.
.
(4.20)
,
so (4.19) shows that af + bg is integrable so long f and g are integrable, and a and b are finite. The identity af
+
bg
=
afl'
afl-
-
bg4'b (bg)-
+
(4.21)
-
implies, applying (4.19), that
jaf
bntdb
+
If a k 0, then
ab+dk -
=
jaldb
afl+dbk
aft-dbk
=
J(&T)-*+ Jt/WI+dB
-
-
afl-d
Ia I f-db
=
-
Jtf/'+lt
f+db
-
=
ment applies to the terms in b and g. So
JJ-Jg,)
IJI
-fd
'-jb8t-db.
=
(4.22) ajfdbk, whereas if a < 0, alfdb. The
=
(4.16) holds as required.
same argu.
is a very useful property. The first application is to show the the integral to the behaviour of functions on sets of measure 0, of invariance extending the basic result of 4.3. Linearity
4.8 Lemma Let f and g be integrable functions. (i) If f f g a.e.lg1, then fdkt; hdp. dp. (ii) If f = g a.e.gg,l, then J/Wg,= Proof For (i), consider first the case J 0. lf g 2 0 ekerywhere, directly from (4.1). So suppose # k 0 a,e.rg,l and define =
dbk 2 0
63
Integration
Jl4)) Then h
0 a.e.gp,) but g
=
jg
0 S
fhdbk0andby
since
J('
To prove
JJJF
2 0
=,
#()
<
.
0 4.7,
jgdp jhdpjgdp, +
=
(4.23)
=
0,
(ii), let
h
J(' + hjdbk
=
:()
h k 0 everywhere, and, applying
hjdp
+
0,
4.3. Now replace g by g J in the last argument to show hence dbkk J/Wptby 4.7.
=
-pdp
+
=
-
=
dp
=
f
-
+
0 by 4.3. Then g so that h 0 a.e.gg,l, and hdp hdbk J#JF, Where the second equality is by 4.7. w =
=
=
These results permit the extension to the more commonly quoted version of the monotone convergence theorem.
4.9 Corollary lf fn k 0 and fn
'1'
limn-oxl/nv
J a.e.g.l,
=
jdp. n
T
Another implication of linearity is the following.
j
Proof
j lf 1 jf'
inequality
4.10 Modulus
lJ lJz
Jjt 2
+#-)tfp,
=
j jjj'dp jf jf'bdp .
-#p,
=
+
Jy-#>l jJ/-d: )Jy::1. .
-
-
In the form of 4.9, the monotone convergence
theorem has several other useful
corollaries. 4.11 Fatou's
lemma If
Jn k
0 a.e.Lg1,then
liminf fndp k nMx
j
liminf
Jn Jg,.
n->oo
Proof Let gn infkn/'k, so that (,n) is a non-decreasing sequence, and gn 1% g sides oo 2 Igndp. Letting n both of liminfn/n. Since Jn k gn, Jfntkyt the on inequality gives =
=
--+
liminf fndp 2 lim gndp n.-+txz
=
gdp
liminf
=
n-M
4.12 Dominated convergence theorem If fn
--
that IfnI
fn #p..
(4.24)
w
n--x
S g a.erlp1 for al1 n and
gdbk <
x,
f a.e.lgj, an there exists g such then JlnIg, .-A JJJP,.
Proof According to 4.8(i), dp < x implies JlfnI#g, < =. Let hn Ifn f I such that 0 f hn K 2g a.e.lg,j and hn 0 a.e.ggj. Applying 4.3 to liminfns, linear=
-->
jw.
.t.......:.')..:..
t.............( .) ..... .:
.......
Fatu' s lymma, attl .).( ,
.'). '
..
.(..
.
...
E .y. ... . .
..
.
.. .
j.... ..
.
-
,
64
Mathematics
ljgdp
J
=
=
liminftzg
-
j (lg jhndp
hn) Jjt f liminf
-
hnjdp
N->x
n-/co
jgdp
limsup
-
(4.25)
,
N-+=
where the last equality uses (2.4). Clearly, limsupn-yxf/lnzv k 0 the modulus inequality implies 1im n-e
=
0, and since
jfndp j/'ljt jhndp < 1im
-
=
0.
j'hndb (4.26)
.
n'M
Taking the case where the g is replaced by a tinite constant produces the following version, often more convenient:
4.13 Bounded convergence theorem If fn then limn--jxlfntfg, Ifdt < x. E1
--y
f a.e.gg1 and IA! f
B<
x
for all n,
-->
Theorem 4.7 extends by recursion from pairs to arbitrary finite sums of funcPut fn tions, and in pmicular we may assert that J(Z:=1'i)Jg, = Xl=tgiand fndbk l=tlgid, where the gi are pon-negative functions. Then, if fn -1xf Z7=l'j < x a.e., 4.9 also permits us to assert the following.
Z:=1J##g..
=
=
=
4.14 Corollary
If
f'fl is
a sequence of non-negative functions,
J Ng'
,
-
# jg,
.
(4.27)
o
i= 1
fcz1
By implication, the two sides of this equation are either both infinite, or finite and equal. This has a particular application to results involving c-finite measures. Suppose we wish to evaluate an integral gdkkusing a method that works for finite measures. To extend to the c-finite case, choose a countable partition igi, lffl of f1, such that g(fk) < x for each Letting gi loy, note that g iidp by gdp (4.27). and =
.
=
=
4.3 Product Measure and Multiple lntegrals Let (f,@,g) and (E,V,v) be measure spaces. ln general, tfx E, 5 @ N, 1) might also be a measure space, with 7t a measure o'n the sets of 5 @ V. In this case respectivlly, are measures g andv, defined by g,(#) 7:(Fx E) andvtG) a(f1 x G4 called the marginal measures conrsponding to T:. Alternatively, suppose that g and v are given, and define the set function 1: Sys F-> 2-+, =
=
where Sss denotes the measurable rectangles 7:(Fx G)
=
g,(&v(G).
of the space f x E, by
(4.28)
We will show that 1: is a measure on Syg, called the product measure. >nd has an extension to 5 * T, so that tfx E, 5 (8) V, a) is indeed a measure space. The tirst
Integration
65
step in this demonstration is to define the mapping F(o: E F-> x E by Fo(() (,(), so that, for G e V, F(o(G) f )) x G. For E e 5 @ V, let =
=
Eo
F;1(F)
=
((: (,()
=
e F)
c
E.
(4.29)
The set Eoj can be thought of as the cross-section through E at the element ), any countable collection of 5 @ V-sets (Ej, j e EN
U
((:(t,),t)
=
j
For future reference,
U)
e
Ul(:(,()
=
e
./
j
EjI
=
U().
the following.
note
e 5 (& V. If F=Fx
Proof We must show thate'to e N wheneverf G e N, it is obvious that Eo, =
c(Sps),
(4.30)
j
4.15 Lemma Toj is a W5 @ Vl-measurable mapping for each
Since 5 (&N
For
(t.
G,
E
F
e
F
(l)
e f1.
G for Fq 5 and
e N.
=
3,
(l)
(4.31)
the lemma follows by 3.22. w
The second step is to show the following.
4.16 Theorem
';:
on Sps.
is a measure
0, recalling that Fx O O x G O Proof Clearly JE is non-negative, and <0) for any F e 5 or G e V, and applying (4.28).It remains to show countable additivity. Let fE. e Sps, j e (N) be a disjoint collection, such that there exist sets Fj e 5 and Gj E V with Ej F) x Gj; and also suppose E VjEj e Spg, such that there exist sets F and G with E Fx G. Any point (,() e Fx G belongs to one and only one of the sets Fj x Gj, so that for any (t) e F, the sets of the subcollection fGjI for which ) e Fj. must constitute a partition of G. Hence, applying (4.30)and (4.31), =
=
=
=
=
=
vE.)
=
v
U j
U 3, j
U()(o j
Gj,
=v
v
=
)
e Fj =
(t)
e Fj
Xj 1,())v(%),
(4.32)
where the additivity of v can be applied since the sets Gj appearing in this dcomposition are disjoint. Since we can also write vtfo) v(G)1A)), we find =
l(F) .f....j....L..r
. ... . .
....
.
:
.
..
.
). .
....
.
=
g,tFlvtGl
=
JY(FV>())
=
J
X 1z)(t0)V(f) 1
tf/t*l
66
Mathentatics
= as required,
Xp.(F?)v(t7;)
Xz:(e;.) j
=
j
(4.33)
the penultimate equality is by 4.14. w
where
It is now straightforward
4.17 Theorem ( x E, 5
to extend the measure from R,g to 5
e N, J:)
(& V.
is a measure space. hence Svs, is a semi-ring by
5 and V are c-fields and hence semi-rings; 3.19. The theorem follows from 4.16 and 3.8. . Proof
Iterating the preceding arguments (i.e. letting (,T) and/or (E,V) be product spaces) allows the concept to be extended to products of higher order. In later chapters, product probability measures will embody the intuitive notion of statistical independence, although this is by no means the only application we shall meet. The following case has a familiar geometrical intemretation. Lebesgue measure in the plane, R2 R x R is detined for intervals
4.18Example by
=
-((Jl,:11
x
(J2,&21)
=
b1
-
J1)(:2
,
-
(4.34)
J2).
Here the measurable rectangles include the actual geometrical rectangles (products of intervals), and o the Borel sets of the plane, is generated from these as a 2 is a measure space in consequence of 3.20. By the foregoing reasoning, (R2,B which the measure of a set is given by its area. n 0A,
,m4
We now construct integrals of functions /*((t),() on the product space. The following lemma is a natural extension of 4.15, for it considers what we might think of yielding a function with as a cross-section through the mapping at a point (J) e domain E. ,
4.19 Lemma Let f:D x E F-> R be 5 @)Ws-measurable. Define Then f(0:E F- R is V/o-measurable. fixed ) e
.f(o(j)
=
A,tlfor
.
Proof We can write
f(()
=
f(eD,()
=
f(Fo(())
=
/oF(().
By 4.15 and the remarks following 3.22, the composite function measurable.
(4.35) is MIB-
fo
K
Suppose we are able to integrate f with respect io v over E. There are two quesJE/'IWV tions of interest that arise here. First, is the resulting function gl F/plmeasurable? And second, if g is now integrated over f1, what is the relationship between this integral and the integral Inxzfd:Over ( x E? The affirmative answer to the first of these questions, and the fact that the integral where these exist, are fiterated' integral is identical with the the most important results for product spaces, known jointly as the Fubini theorem. Since iterated integration is an operation we tend to take for g'ranted xxrltla mllltlnlo piemnnn nerhnns the main Doint needin: to be stressed infnyrnlq =
tdouble'
Integration
67
here is that this convenient property of product measures
(and multivariate
Lebesgue measure in particular) does not generalize to arbitrary measures on product spaces. The first step is to let f be the indicator of a set E e 5 (& V. ln this case f is the indicator of the set Efo defined in (4.29),and
J/'olv
v(F(o)
=
+(e)),
=
(4.36)
2-+is well-defined, say. ln view of 4.15, Efo e V and the function #s: fl values unless in the extended although, v is a finite measure, it may take its shown. line, half as .-.:
(& V,
4.20 Lemma Let jt and v be c-finite. For all E e 5 and Jo#FdB TCZ')'
g; is T/p-measurable
(4.37)
=
By implication, the two sides of the equality in or finite and equal.
(4.37)are
either both infinite,
Proof Assume first that the measures are finite. The theorem is proved for this theorem. Let denote the collection of sets E such that gz case using the satisfies (4.37).Spv i ,W,since if E Fx G then, by (4.31), .4
';:-
=
+() and gzdbk fl x E e 1sz- lsj,
=
(4.38)
Clearly g(#)v(G) l(F) as required. We now show is a = and since holds. lsz-sj 1.25(a) lf F1 c Ez, then, Ej,E2 e so -system.
=
.4
Js1/,()JV(()
gE,-EL4
an 5lT measurable
Js1e'1(t0,()#V(()
=
=
-
'sztt,)l
(4.39)
-%(),
function by 3.25, and so, by additivity of
'n,
.
jgjgh-zbdbk
.'
Y,
F e
.4
.4,
' r' 1*
v(G)1F(),
=
l(F2)
=
-
r(F1)
=
zE2
-
F1),
.VL.
t
(4.k0)
''
showing that .Wsatisfies 1.25(b). Finally, If 41 and A2 are disjoint so are (A1)(o + gy). and (A2)(o,and ujwxct)) u1((l)) To establish 1.25(c), let (. e W, with be IN Ej j e J a monotone sequence, F. Define the disjoint collection (Aj) = With A1 q= E$ and Aj Epj A),j > 1, so that E U7=3Ajand Aj e Wby (4.39). additlvity of v, y ountable
(
=
't
=
-
x ','(l
=
Xuytl.
yj -.t. .E )r-. )..' L.k,.j-L);(b. ..r.),).).t - ; )--'.ti't(.. (yyyE (;()t. .(.yy')-....qy . r. tf''Ett..t.q'? ;t-yjjy:jj(j .yjjjy.;gjjyjjjjjt.. .y yy. q jtyjjj yyyj,;( jqqj)q;((. .!k jyy .ryLLI. gjyj)j. jjrjjk yy. ... jj;gjr j.yy.j.jjjyrr jjygj. yjjjyj $kj)4 yryt;y.)...-i.r..t.E)). .E 7..y jj ... y..q-().r-(.t(. ...-.-!;y..:. ,.-. .. gyjj,. . r) r.q.r t. #t-.-y 1l(!(l:l.)(.. ..((r-. )-)t.. . t.-... .. .. ' ;.1 . . . . . . ... . . .. . . . ((
')'
..t.-)
-'.t.t
-..(q..)--r;:,)j);.tr-ry
.
..
..q
..
y..jt..j. . ;fj,jL)qq;... )) jtjj(j)q..... ..y,r(.jk
-.-
.li.
:
--''
-
- - .
-..)-.-.))t:..,.
-..
.
...
.
tj!jj jj.(gjrj
(4.41)
68
Mathematics x
x
Jo
X'At)
tfBtt0l
g
Xj j=1
=
j=j
v
x
o'Atttltfp,tl
X5(X/)
=
=
1(13,
(4.42)
j=j
n
.d
Since Sys is a where the first equality is by 4.14. This shows that is semi-ring it is also a a-system, and 5 @ N c(Sys) c by 1.27. This completes the proof for finite measures. To extend to the c-finite case, 1et J and (Ey) be countable partitions of f tff and E with finite g-measure and v-measure respectively', then the collection tff x Ej e Sys ) forms a countable pmition of x E having finite measures, attf x Ej) = g.tfvtWl.For a set E l 5 @V, write Eq = E r7 tf x Ej). Then by the last argument, -system.
.4
=
ghdbk
=
where gzql fk disjoint and
F-> 's((l))
nEq),
(4.43)
R+ is defined by +,y((l)) v((F(j)(J, (t) e fk. The sets Eq are vjLjjl whep (l) e ff, or =
=
gE
=
X 1f4()X+/).
(4.44)
j
i
The sum on the right need not converge, and in that case g/ F/s-measurability holds by 3.25/3.26, and
jcysdb -
-
Jo
Xjl XyS's
XXJO:S/B i
j
*'
-
'*'l
+x.
=
However,
Jp, XXn(A) i
-
(4.45)
nk,
j
using 4.14 and countable additivity. This completes the proof. Now extend from indicator functions to non-hegative
.
functions:
4.21 Tonelli's theorem Let 7: be a product measure with c-finite marginal measures g and v, and 1et fl x E F-> R+ be (@(&Vl/f-measurable. Detine functions .f:
f1
= R+ f(,(), is T/f-measurable, (i) g JsfA
E
F->
(ii)
b-V(()
jtstcafdx Jo =
and let gk
=
hfuA.
Then
tfB. E1
ln part (ii) it is again understood that the two sides of the equation are either finite and equal, or both infinite. Like the other results of this section, the and (E,N,v), and the complementary results given theorem is symmetric in (f1,@,g) roles interchanging of the marginal spaces do not require a sepapte stateby the ment. The theorem holds even for measures that are not c-tinite, but thij further complicates the proof. Proof This is on the lines of 4.6. For a partition (F1,...,Fn J of 5 @)T let f Xwlsj, and then f Xii ltsjl(sand g = Xiqis'lEil by 4.4. g is F/p-measurable
=
=
Integration
69
by 3.25, and 4.20 gives vedb gdb Xm fl i
Xaine'i)
=
=
fl
=
j
AE
j'dn,
(4.46)
so that the theorem holds for simple functions. For general non-negative f, choose a monotone sequence of simple functions converging to f as in 3.28, show measurability of g in the limit using 3.26, and apply the monotone convergence theorem. w Extending to general
f
requires the additional
assumption of integrability.
4.22 Fubini's theorem Let a be a product measure with c-finite measuresg, and v; 1et #: fl x E >-> R be (; @ Vl/f-measurable with
marginal
Jzxsl.ftt,)'tllllt,ll <
(4.47)
=;
Then and let g define f)1 E >-> R by Jsffzyfv. (i) f is N/f-measurable and integrable for (z) e A f2, with g,tfl A) (ii) g is F/f-measurable, and integrable on A; .f4,();
.406)
=
=
=
-
JsTto'cldvt'l joxsft,clflttt,'l Jo
(iii)
0',
dpt*'
=
Proof Apart from the integrability, 4.19 shows (i) and Tonelli's Theorem shows (ii) and (iii) for the functions f' = maxl,o) and f- = f'b f, where I f+ + f-. But under (4.47),I I < on a set of a-measure 1. With A defined (i), (ii) and (iii) hold for f'b and as the projection of this set onto finite of the equation in sides with both (iii). Since f f'b f (i) extends to f by 3.25, and (ii) and (iii) extend to f by 4.7. . .41
=
-
.f(),()
'x
-,
,
-,
=
-
4.4 The Radon-Nikodym Theorem Consider c-finite measures g. and v on a measurable space (f2,1).g, is said to be absolutely continuous with respect to v if v(A) 0, for E e 5, implies g(A-) 0. This relationship is written as g <( v. If g tt v and v <( go the measures are said to be equivalent. If there exists a partition (A,A ) of f1, such that g(A) 0 and v(AC) 0, then g and v are said to be mutually singular, written g, v. Mutual singularity is symmetric, such that p. v means the same as v g,. The following result defines the Lebesgue decomposition of g, with respect to v. =
=
c
=
.1.
=
.1.
.
4.23 Theorem If p, and v are c-tinite measures, there exist measures g,1and p.2 such that g = gl + ga, g,l v, and p.2
lf there is a function #: fl 1..-> F+ it follows fairly directly (chooseE such that v(F) 0) such that g(F) Js/tfv, that g, << v, and f might be thought of as the derivative of one measure with =
=
70
Mathematics
respect to the other; we could even write f dbjldv. The result that absolute continuity of g, with respect to v implies the existence of such a function is the =
theorem.
Radon-Nikodym
4.24 Radon-Nikodym theorem Let v and g,2be c-finite measures and 1et p,2t< v. There exists a T/p-measupble function fl f F-y 2-+such that g2(F) fdv for all E e 5. u =
f is called the Radon-Nikodym derivative of g, with respect to v. If g is another for all E e 5, then v(.f #: g) 0, otherwise at such function and g,2(A-) Jsgtfv and E1 (: least one of the sets Ej (): ()< gl ) must contrathe detinition. dict =
=
.j'((l))1
.f4)>
=
=
Proof of these results requires the concept of a signed measure.
4.25 Definition A signed measure on (f,F) is a set function z: 5 F-> R-
satisfying 0. (a) :(0) j) (b) z(U#. XgAj) (c) Either z < x or =
=
disjoint collection (Ay e 5
for any countable, u z>
For example, let g, and v be non-negative measures on a space least one of them finite. For a non-negative constant r, define Z(X)
for any
)
.
-x.
=
((, @), with
p,(A) rvtAl
at
(4.48)
-
A e 5. For disjoint (Ay)
,
zj+l (-s) (x/) -
j=3
p,
-
y=1
= Xtg,tll
rv
/=1
-
rvtAjll,
(4.49)
i=1
so that countable additivity holds. If A is a F-set with the property that # e 5 with B c A, A is z(#)k 0 for every complementary negative defined in the called apositive set, a set being manner. A null negative both is positive and is called careful Be that set. to distinset a guish between positive (negative,null) sets, and sets of positive measure (negative measure, measure zero). A set has measure zero if g,(A) rvtz4l in (4.48), but it is not a null set. By the definition, any subset of a positive set is positive. The following theorem defines the Hahn decomposition. :4
=
4.26 Theorem Letx be a signed measure on a measurable space (f,1), having the property x(A) < cxo for a1l 4 e 5. There exists a partition of f into a positive set
A+ and a negative
set A-
Proof Let sup x(A), where the supremum is taken over the positive sets of x. A+ Choose a sequence of positive sets (Anl such that lima-ox z(Aa) o and 1et z4+ A+. measurable consider E Undn.To show that is also a positive set, any Iaetting B, A1 and B- A- -A--1. n > 1. the seuuence $Bnl is disioint, positive ,
=
=
=
=
=
Integration
71
since Sa c A,i for each n, and UnSn = A+. Likewise, if En
Ecj Bn the sequence fEn ) is disjoint, positive since En c #n, and Unk E. Hence zlE') XnzEn) 0, and since E was arbitrary, A+ is shown to be positive. A+ -An being therefore =
=
=
positive, Z(X+)
Z(X2
=
Z(Z+ -A
+
k Z(Xu), a1l n,
(4.50)
z(A+) k 1, implying z(A+) 1. Now let A- f -A+. We show, by contradiction, that A- has no subset E with positive measure. Suppose there exists E c A- with zE4 > 0. By construction E and A+ are disjoint. Every subset of A+kJ E is the disjoint union of a subset of A+ with a subset of F, so if E is a positive set, so is A+ t? E. By definition of
and hence
=
=
X,
,
2
b-)E4 z(z4+
:+z(A),
=
(4.51)
which requires x(A>) 0, so E cannot be a positive set. If F is a subset of F, it is also a subset of A-, and if positive it must have zero measure, by the argument just applied to E. The desired contradiction is obtained by showing that if zE) > 0, E must have a subset F which is both positive and has positive measure. The technique is to successively remove subsets of negative measure from E until what is left has to be a positive set, and then to show that this remainder has positive measure. Let ?z1 be the smallest integer such that there is a subset F1 c and detine F1 E F1. Then let nl be the smallest integer E with z(F1) < In general, for k = 2,3,..., such that there exists E1 i F1 with z(F2) < 1et nk be the smallest positive integer such that Fk-j has a subset Ek satisfying Z(f) < -$Ink, and let =
-1/n1,
=
-
-1/n2.
Fk
E
=
-
U). j=
(4.52)
lf no such set exists for finite nk, let nk +x and Ek = 0. The sequence (Fk) is x. non-increasing. and so must converge to a limit F as k =
---
We may therefore write E Fb..l (U7=1A), where the sets on the right-hand are mutually disjoint, and hence, by countable additivity, =
oo
z(f)
:4#')
=
+
side
Cr
Xz(A)
<
k=1
llnk. zb7 X k=1
(4.53)
-
Since x(F) > 0 it must be the case that x(#)> 0, but since zF < oo by assumptoo as k x. < x, and hence nk This ion, it is also the case that Z7=1(1/ak) with and negative is therefore a positive measure, means that F contains no subset . having positive measure. set -->
--
For any set B e 5, define Z(#) z(A+ rn B4 and z-B) -z(A- fa #), such that x+(S) z-B). It is easy to verify that zB4 z+and x- are mutually singular, non-negative measures on (f1,F). f z- is called the Jordan decomposition of x a s ig ned measure. z+and z- are called the upper variation and lower variation of =
=
=
-
=
a,
wxvxa
+1-.zk
maooxa...x
l .,
I
l,+
-
..u
%,-
lo
-
zxollxcxzl
tlao
tntnl
a/mrizyimn
nf
q;
Tho
Tnrdnn
Mathematics
72
decomposition shows that all signed measures can be represented in the form of (4.48). Signed measures therefore introduce no new technical difticulties. We can integrate with respect to z by taking the difference of the integrals with respect V to x an (j y We are now able to prove the Radon-Nikodym theorem. It is actually most convenient to derive the Lebesgue decomposition (4.23)in such a way that the Radon-Nikodym theorem emerges as a fairly trivial corollary. It is also easiest to begin with tinite measures, and then extend the results to the c-tinite case. -
.
4.27 Theorem Finite, non-negative measures v and g have a Lebesgue decomposition g /1,1+ g2 where gl v and p,2 << v, and there exists an F-measurable R+ F-+ f such that g,2(F) function /': fdv for a11 E e 5. .1.
=
=
Proof 6 L e t G denote the class of al1 F/f-measurable functions gl f F-> R+ for which IEgS S p,('), a11 E e @. 6 is not empty since 0 is a member. Let (x sup#esf'tfv, so that (x f g,tfM< x. We show there is an element of G at which the supremum is attained. Either this element exists, and there is nothing further to show, or it is possible by detinition of a to choose an element gn of G satisfying f a, for each n G IN.Generate a monotone sequence (/k,n e IN) in G tx 1/a f Igndv follows. Put so that fn 2 max(A-1((z)), J1 :1 and define fn by fn as > %()) for n 2,3,..., and then if fn-l A-l. Define the sets Aa (: e G, =
-
,w()),
=
=
.fn-1((l))
=
Xtfv
=
fn-tdv +
=
F
Erun
Fcxq
gndv
g,(FrnAn) + p.tf f'7Ak)
=
g,(f)
(4.54)
so that fn e G. Since fn 1 f, it follows by the monotone convergence theorem that IEf* S h1,(0, and hence f e G. And since fn gn so that fndv --4
a we must conclude that
-
Lln S
J.ftfv a, =
g,2tD
=
gndv S
fndv S a, Now detine g,2 by
as was required. dv, E
C
F'
(4.55)
Evidently g,2 is a non-negative measure (for countable additivity consider the and use 4.14), and also g,2 q v. Define g,1(e')= g,(f) 41,24f), functions which is non-negative by construction of f, and also a measure. It remains to show .
that p.j
=
IV
-
,
.1.
v.
Let (A+n,A-a)be a Hahn decomposition for the measure 1,2,3,.... Then for E e T, g,(Fr7A+) n
=
p.1(Ff-74)
+
-1v(l
>n
ra x+n) +
g.l v/n, for n -
=
g2(F raA+sl
j
sck.:
y:v
=
j
srn,j
-)
.f
+
w,
(4.56)
Integration
73
and hence p.(A')
p,(Ff'3A1) +p.(FrnAa-) 2 p.(Ff7A1)
=
k
J
El'x<
f
1 tv
+
n
Note from this inequality that
(x k
Jf
+
v
+
fdv J Fzn-a
J + n-11Aj -s
1v(A+n) =
=
+
p.2(Fr-7Aa-)
fdv + JE
-)v(F
r-'hA).
(4.57)
e 6 so that ,
-1v(4+n),
a
+
(4.58)
u
implying vAhjlln 0. This holds for each n G N, so if A U';=1A, v(A) 0. Note g.1(AC) v(AC)/l An-for every n, and so for every n. Hence ; that Ac 0* N= 1Aa-i g,1(Af) 0, and-so jt1 v. w =
=
=
=
.1.
=
It remains
to extend this result to the c-finite case.
such Proof of 4.23 By c-tiniteness there exists a countable partition f l of collection with finite for finite each j. If (Ay) is any that vtf#) and p,tf#l are union and whose is letting f11 Lh -Ay-1 for A1 1 > Aj defines j measures such a partition. If different collections with tinite measures are known for v and p,, say lAjzjl and fAvyl the collection containing all the Ax rnAvk for j,k e N, is countable and of finite measure with respect to both v and g,, and after reindexing this collection can generate (f). Consider the restrictions of g, and v to the measurable spaces (Lh,%4, for j e IN,where % (F t''h Lh, E e 5 J By countable additivity, g,(F) XjttA' rn L%lwith similar equalities for g,l, g,z, and v; by implication, the two sides are in each case either finite and equal, or both +x. If v(F rn Lhl 0 implies g,2(F chLhl 0 for each j, then v(F) 0 implies g2(F) 0 for E e 9, and g,2< v. Similarly, let Aj, AJ) define partitions of the Lh such that g,1(Aj) v(A,) 0', then A U#. j, v(AC) = Qv(A,) = Ac unions, 0. p,1(A) Qg4(AJ 0, and and Ugdlare disjoint Hence g,1 v. K ,
=
,
=
,
=
=
.
=
=
=
=
=
=
=
=
=
=
.
The proof of the Radon-Nikodym theorem is now achieved by extending the other conclusion of 4.27 to the c-finite case. Proof of 4.24 In the countable partition of the last proof, 4.27 implies the such that non-negative X/f-measurable
existence of
g,(F rn Lh)
=
p,ltf
f''h
L1jl + g,2(F r-hLhl
where P,2(A't''h f#) Define
J:
F-
Jsyoo *,
=
R+ by .fttl)l
=
al1 E e
X lf/ttl'tt9l.
5.
(4.59) (4.60) -1
This is a function since the Lh are disjoint, and is g/s-measurable since f B) y-1(#) LtEj e 5 where Ej e 5j, for each # e S. Apply 4.14 to give = Uy/' =
Mathematics
74 :2(A3
=
Xp,ztFr7
Lhl
=
j
Js
1tzjfldv. - X
Xj
Js
Jsljdv
fdv
-
.
(4.6 1)
*
Consider the case where g, is absolutely continuous with respect to another gq, v(A) measure v. If the Lebesgue decomposition with respect to v is g, g,l + g,1(AC) since 0 implies 0 which in implies g,l(A) 0. But g,(A) 0 turn g,l v, too, Thus, g,1(f2) 0 and g, ga. The absolute continuity of a measure implies the existence of a Radon-Nikodym derivative f as an equivalent representation of the fOr measure, given v, in the sense that g,(F) Js/'tfv any E e @. An important application of these results is to measures on the line. =
=
=
=
=
.
=
=
=
4.28 Example Let v in the last result be Lebesgue measure, m, and 1et g, be any other measure on the line. Clearly, g,1 m requires that g,1(A'') 0 except when E is of Lebesgue measure 0. On the other hand, absolute continuity of g,2 with length', any countable collection of respect to m implies that any set of isolated points for example, must have zero measure under g2. If g, is absolutely continuous with respect to m, we may write the integral of a measurable function g as =
.
Ezero
j
+x
'(AVP'@)
=
-X
p+x
1 -
gxtfxtdx'
-X
(4.62)
so that all integrals reduce to Lebesgue integrals. Here, f is known as the density functionof the measure g, and is an equivalent representation of g,, with the relation g,(f)
=
jefxtdx
(the Lebesgue integral of f over E4 holding for each E e f. n
(4.63)
5 Metric Spaces
5. 1 Distances and Metrics Central to the properties of R studied in Chapter 2 was the concept of distance. For any real numbers x and y, the Euclidean distance between them is the number Jstx,y) Ix y I e R+. Generalizing this idea, a set (otherwisearbitrary) having a distance measure, or metric, defined for each pair of elements is called a metric space. Let S denote such a set. =
-
A metric is a mapping J: S xs F-> R+ having the properties dx,y), 0 iff if y, 2 dx, dy, (triangleinequality). metl'ic (S,#) is S A space a set paired with metric #, such that conditions (a)-(c) hold for each pair of elements of S. u
5.1 Dqfinition (a) dy.x) (b) dx,yt (c) dx,yt +
=
.'t7
=
=
0, but dx,y) 0 is possible when x # y, we If 5.1(a) and (c) hold, and dxtx) would call d a pseudo-metric. A fundamental fact is that if (A,#) is a metric space and # c A, (#,#) is also a metric space. If (E1is the set of rational numbers, Q c R and (Q,#s) is a metric space', another example is (g0,11,#s). While the Euclidean metric on R is the familiar case, and the proof that dz satisfies 5.1(a)-(c) is elementary, dz is not the only possible metric on R =
=
.
5.2 Example For x,y
E
R 1et
4@,y)
=
!
Ix +
yI
-
jx
..y
j
(5.1)
.
It is immediate that 5.1(a) :nd (b) hold. To show (c), note that Ix l doxtyjlk dx,yj). The inequality tz/(1 a) + /7/(1 b4 2 c/(1 c) simplifies c). We obtain 5.1(c) on putting a doxty), b doy,z), to a + b 2 c + abl c 4)@,z),and using the fact that 0 f do S 1. Unlike the Euclidean metric, Jo is defined for or y = ix. (V,4) is a metric spae on the definition, while Rwith the Euclidean metric is not. n -y
-
-
-
=
-
=
-
=
=
.x
lh the space R2 a 1arger va riety of metrics is found. 5.3 Example The Euclidean distance on R2 is -y1)2
defx-ytIIx-yII E(m =
d (q2r 4) p,lp ,
.
''
,
.
=
-y,)211'2
+(.n
(5.2)
'
.
is the Eglidean '
'
'
plane. An alternative is the
Etaxicab'
metric,
76
Mathematics
dvx,y)
=
Ixl
-
yl I +
1A'2
-
y2 1
(5.3)
.
dz is the shortest distance between two addresses in Manhattan as the crow flies, but dv is the shortest distance by taxi (see Fig. 5. 1). The reader will note that dv and Js are actually the cases for p 1 and p 2 of a sequence of metrics on R 2 He/she is invited to supply the definition for the case p = 3, and so for any cxo is the maximum metric, p. The limiting case as p =
=
.
-..-h
JA,g,y)
=
maxl
-y2l
1.x1yl I 1x2 -
,
1.
(5.4) R'T
Al1 these distance measures can be shown to satisfy 5.1(a)-(c). Letting R x ERx x R for any finite n, they can be generalized in the obvious fasltions (RR,#r), (R'l,#x) and so forth. n to define metric spaces (R&,Js), =
...
t/yg (-,:,)2 )
------------
t2-(-);,)7
)
Fig. 5.1 Metrics dk and dz on a space S are said to be equivalent > 0, there is a > 0 such that
dblx,yq < 8
=
dlx,yq < E
dzlxty) < 8
=
[email protected]') < E
if, for each
-v
e S and :
for each y e S. The idea here is that the two metrics confer essentially tlte same properties on the space, apart from a possible relabelling of points and axes. A metric that is a continuous, increasing function of another metric is equivalent to it; thus, if d is any metric on S, it is equivalent to the bounded metric dll + d4. Js and do of 5.2 are equivalent in R, as are are Js and du in R2 o n the other hand, consider for any S the discrete metric Jo, where for x,y e S, dE dnxby) 0 if are not y, and 1 otherwise. do is a metric, but Jo and equivalent in R ln metric space theory, the properties of R outlined in j2.1 are revealed as a special case. Many definitions are the same, word for word, although other concepts are novel. In a metric space (S,#) the concept of an open neighbourhood in R generalizes to the sphere or ball, a set Sdxn (y: y e S, J(x,y) < :) where x e S and : > 0. We write simply Sx,z) when the context makes clear which .
=
.x
=
.
=
,
Metric Spaces
77
metric is being adopted. ln (R2 d E ) S@ :) is a circle with centre at x and it is a radius E. In (R2 (rotatedsquare) centred on x with e the distance from x to the vertices. In (R2 du ) it is a regular square centred on x, with sides of 2e. For (R3 d s) we 11 think about it! An open set of (S,#) is a set A S such that, for each x e A, (; > 0 such that 5'@,) is a subset of 4. If metrics dj and dz are equivalent, a set is open in (S,#1) iff it is open in (S,J2). The theory of open sets of R generalizes straightforwardly. For example, the Borel field of S is a well-defined notion, the smallest c-field containing the open sets of (S,J). Here is the general version of 2.4. ,
,
,
tdiamond'
,#w)
,
,
,
...
5.4 Theorem (i) If C is any collection of open sets of (S,#), then
c
UA
=
Ae N
is open.
(ii) lf A and B are open in (S,#), then A ch# is open. Sx, c A and A e C, then S.z,z) c C. Since such a ball exists by a11 definition for x E A, a11 A e C, it follows that one exists for all x e C. (ii) If 5'(x,/) and S(x,k) are two spheres centred on x, then
Proof
(i) If
5'@,6k)f''h 5'@,Es)
(5.8)
5'@,E),
=
If x e A, H 5'(x,/) c A with Ex > 0, and if e #, (5S(x,ze4 E > 0. If x e A chB, 5'@,E) I A rn B, with 6: > 0. x
where e minlerx,Es) c B similarly, with
.x
=
.
The important thing to bear in mind is that openness is not preserved under
arbitrary intersections. A closure point of a set A is a point x e S (not necessarily belonging to A) The set of closure points of A, such that for all 6 > 0 ('!y e A with #(x,y) < . of Closure called the closure A. points is denoted A, are also called adherent points, sticking to' a set though nt necessarily belonging to it. If for some 8 x, so that > 0 the definition of a closure point is satisfied only for y &x,) f'-AA (xJ, x is said to be an isolated point of A. A boundary point of A is a point x e X, such that for all > 0 H z e Ac with The set of boundary points of A is denoted :A, and W A Q.J :4. The J(x,z) < interior of A is A0 A PA. A losed set is one containing al1 its closure Points, such that X A. An Qpen st dyjes not contain a1l of its closure points, since the boundary points do not belng to the set. The empty set 3 and the space S are both open and closed. A Subset pf A is said to be dense in A if B c A c X. A collection of sets T' is calld a. bvering for A if 4 Usssf. If each B is =
=
=
.
=
-
=
.
.
...:.;
.
''
.
'
.
.
;
'
.,>
covtink. A jt A is called compact if every open open, it is called an open kqtog. x is said to be relatively compact if covering of A contains a tinite S itsel? 'A,t,tk'.$,j,'..,k4) is said to be a compact space. The cott, is compact. If s is rk.,i tl compaltlktlyj, ,. ttk y))(y,y;(jty)):Vyty,yjkj4g eqqally relevant to the general case. remarks in j2. 1 about jkyyjytyyyjjy yjj yyy, jj 4::11. d!;p dr:;li 4k220 154!r 11::). tjyod jyyyyyktty.jtyz,y,g,jyjyyyyyjxyyyjyjyy,,yyyyyyytyy ii.1b;1b;Citii. 1l:4(E:1. rr.t).:y...;....,. ..,.4!k. ;yjjy .yj < x, sucj.j tjaatx s sxtrj; and also .
x
. .
.p,
,1.)
.))
'
t
.
rjg)
....r
.
.F ..--
.
Mathematics
78
totally bounded (or precompact) if for every e: > O there exists a finite collection of points x1,...,xm (calledan E-net) such that the spheres Sxz), i z4. The Sxi, can be replaced in this definition by = 1,...,- form a covering for noting that Sxi,z) is contained in Sxi, : + ) for all 4'5 their closures Sxinz), z4. > 0. The points of the E-net need not be elements of An attractive mental image is a region of R 2 covered w ith little cocktail umbrellas of radius : (Fig. 5.2). Any set that is totally bounded is also bounded. In certain cases such as ?n,dE) the converse is also true, but this is not true in general. . .k . .
.
.
... . . .
.
.
.
.
4
.
.
.
.
,
.
.
.
k...
.
.
.
.
.
.
.
.k
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
y .
.
.
.
.
'
j
.
.
.
.
.
.
.
.
.
.
.
' .
.'.
'
.. .
.
.
.
. .. '
.
.
.
.
. .
.
.
. '.
...
.
.
...
.
i
.
.
.
.
.
....
.
.
'.'k. .
.
. .
.
'.
.
.
.i; ..
.
...'
. .
.
...
.
.
4 .
.
...
. .
...'.
kj'..'
Fig. 5.2
5.5 Theorem lf a set is relatively
compact, it is totally bounded.
Proof Let A be relatively compact, and eonsider the covering of X'consisting of the e-balls Sxnz) for a1l x e X'.By the detinition this contains a finite subis an e-net for cover Sxitz), i = 1,...,-, which also covers A. Then (x1,...,au) d, and the theorem follows since e: is arbitrary. K The converse is true only when the space is complete', see 5.13.
5.2 Separability and Completeness In thinking about metric spaces, it is sometimes helpful to visualize the analogue R'J with n f 3, and use one's intuitive knowledge of problem for R, or at most for those cases. But this trick can be misleading if the space in question is too alien to geometrical intuition. A metric space is said to be separable if it contains a countable, dense subset. Separability is one of the properties that might be considered to characterize an 4R-like' space. The rational numbers (I) are countable and dense in R, so R is separable, as is R&. An aliernative definition of a separable metric space is a metric spaee for which the Lindelf property holds (see2-7). This result can be given in the following fonn.
5.6 Theorem In a metric space S the following three properties are equivalent: (a) S is separable. (b) Every open set A S has the representation
Metric Spaces
4
UBi, Bi e
=
79
'P,
izu1
where V is a countable collection of open spheres in %. Every (c) open cover of a set in S has a countable subcover.
I:a
A collection V with property (b) is called a base of S, so that separability is equated in this theorem with the existence of a countable base for the space. In topology this property is called second-countabilit.y (see j6.2). (c) is the Lindelf propely. Proof We first show that (a) implies (b). Let V be the countable collection of spheres (5'(1,r):a7e D, r e t1+), where D is a countable, dense subset of S, and Q+ is the set of positive rationals. If A is an open subset of S, then for each x e A, 3 > 0 such that 5'(x,) g A. For any such x, choose xi e D such that dxie < /2 (possiblesince D is dense) and then choose rational ri to satisfy dxip < ri < &2. Define Bi Sxitr e V, and observe that =
S(A7,8)
Bi I
(5.10)
i X.
Since V as a whole is countable, the subcollection fBiI of al1 the sets that satisfy this condition for at least one x e A is also countable, and clearly A c Uff I X, SO X = U#f. Next we show that (b) implies (c). Since V is countable we may index its elements as ( Jj, j E EN). If C is any collection of open sets covering A, cboose a subcollection ( Cp j e LNJ where Cj is a set from C which contains 5. if such by F-sets, as just exists, otherwise let Cj 0. There exists a covering of shown, and each 5. can itself be covered by other elements of V with smaller radii, so that by taking small enough spheres we may always Gnd an element of C to contain them. Thus 4 c UjG,and the Lindelf property holds. Finally, to show that (c) implies (a),consider the open cover of S by the sets (5'(x,1/n), x e S l If there exists for each n a countable subcover (Sxnktllnj, k e ENJ, for ach k there must be one br more indices k' such that dxnkpxnk')< lln. Since this must be true for every n, the countable set fak, k e (N n e EN ) must be . completes ptoof. the This in S. dense ,
:4
=
.
,
The theorem has a useful corollary, lc'tg C' k' ;''j' y' q.rljljjrjtrj'jk. k:5j';;'jll.' j(j('(jy';;)(;(jjjr-:';E' 'i'. 11(F*))7(j)1))*5i'-!tt)@'.tjjj: rjd t;(.',' y' (jd y.'r' )' :'''jt'ty'k). (y'r;. )' jy.yjy-'tyyd yj'(' (' )jjj'Lqjj.j;Lj))(L:'.. ;';)' j;'16'. illj.Fklfll'r!id j()$.tj!y' !l-i@;yy'. jj.'lq. j'td .'.'k(t(jjj.'. --' ,.' -' .'.,' ,'.r:,ggjj4;4* ..t-d '-7*
'''jd 7* ::.tky,(y)jjj4jj44.2* yjyyjj...d ..-tj).'.' ,.t,' jy'. E'. 5* k(j)jt)':. ;'-'kE''. ,':'.#y' ?'
)'jt. jyd (' jjryjjgj;rrjjj'. yy'' jjjjj..d jjj')j(jk)'. r)'k(!@'' )' h' tl'tjjg t'jkd
''. .' 57 @
Corollary A totally bounded space is separable. n .
Another important property is that gbspaces which we show as follows. y
sR
Tuoorem
If 4:.* is a sp' ..
.
.
..
of separable
spaces are separable.
ab ll,!.'pae and A c S, then A,d) is separable. l l '
.
.'
;
'.;
. .y
y.
:
5:.'
:
')'q q.ty ;..' j,.);.j.;Lj. jy yk y. ..) .. a,,#..z'inl.s.
....y.((
. ..
.'
)
;y..
:
.'
.
g
g
...
.
. d'tll(td. Constnlct the countable set E by ) qp;jt,. jytgr yy a, ( '' supposeD is countbl g.;..j y (. y .). . .... y .. ... .. . . yjtliltjt, . q F tnul iyl!q t ' 'y takingone point from each;.$.',tj,.A,, ' (qi r..i q. y ( .( .. jy- jy.j(. ..jjtk(,... rk ;,-j,bj,.,. ...;,;-j-tyjj-yyy.trr,pjrrj--t:. jq j rk.,yjj ' r...'. .(. .,.-bj',::,;;;LL?3b.,L, y ..g.. .. ;f;;,,,;j:;-f);;,-'LL?33'-.,-j;)--Ljj;::-),'b--jj.)L,,t,;b'f?;));j-.
Ivoof
.
y
E
.
:
(
E
(E $2 .'
EE .
.
E ' ly . .
jE. .
?.
.
: gy. q
.ptllko'tyq)
E. q: . .
:
.E
.
,
.
.
l'.r
-i
y..y)..
E' . .. .
.' ' ::EE. . . . . E .-' (Ej.jijjjjjyryjgggggg, .. . . g: E . y y . y, g . . yy . . . y .jjyy... ..j... . y.-.... . ,!j;pjj42)$g,-.,,y,..(y.kpjr)(.y(y(,yyjjyjjyj;jy.-..,...-y.tyjky,
E
. E .. yj' .E. !. - q. . (-. .(j. ; (E r
.
.
.
.
.
-
. -:;..,-:3626,3.-.; -...633333:,31,'.. . . . .
.
j .....-.-,...yjj..... . .-,,.... .....
#.,..- ytlyy.jitr.. )r,y,-.yyjyjjtjj)y.,,. .--..!1k.. ijijLjlijjjiiljl-. . ... .. ... ..
);.ryj-rkji-. 1,:-::,..-,k. .... lttjj?, . . .;(;;yyit..ykrj.-.yy-j-kjtjrqy-yj. .,jjt;-.. ...y'ptj,;,.-p!h!!!qqj--..,.
...-....
,
( !jr r... ..... . . .. :..
: : : . ..
,
-k,yt.yytyyy-y,......--...-
.
,....y
-.
.
;k.
.
..
.
,.
. ,.
.
),
.
..
.
,
..
,;.j,:-.,,j;;)j.yty,r.t y .(.. - .. .... -
.
.
..
.
4::). jjjjj.s ,-
tjggjjj
Mathematics
80
For any x e A and > 0, we may choose y E D such that J(x,y) < 6/2. For every such y, R z e E satisfying z e A fn S@, for r < 6/2, so that d@,z) < &2. Thus
dxa)
f dx,y)
+
J(.y,z)
<
8,
(5.12)
and since x and 8 are arbitrary it follows that E is dense in A. w This argument does not rule out the possibility that A and D are disjoint. The separability of the irrational numbers, R Q, is a case in point. On the other hand, certain conditions are incompatible with separability. A subset A of a metric space (S,J) is discrete if for each x e A, R 6 > 0 such that (5'(x,6) (xJ)DA is empty. In other words, each element is an isolated point. The integers ; are a discrete set of (R,#s), for example. If S is itself discrete, the discrete metric do is equivalent to J. -
-
5.9 Theorem If a metric space contains an uncountable discrete subset, it is not separable. Proof This is immediate from 5.6. Let A be discrete, and consider the open set where Ex is chosen small enough that the specified spheres folnn a Uxez5'@,exl, disjoint collection. This is an open cover of A, and if A is uncountable it has no countable subcover. . The separability question arises when we come to define measures on metric spaces (see Chapter 26). Unless a space is separable, we cannot be sure that a1l of its Borel sets are measurable. The space Dgtz,,jdiscussed below (5.27)is an important example of this difficulty. The concepts of sequence, limit, subsequence, and cluster point all extend from R to general metric spaces. A sequence (x,,)of points in (S,J) is said to converge to a limit x if for al1 e > 0 there exists Ne k 1 such that
dxng)
< :
for a1l n
>
(5.13)
Ne.
Theorems 2.12 and 2.13 extend in an obvious way, as follows.
5.10 Theorem Every sequence on a compact subset of S has one or more cluster points. n 5.11 Theorem If a sequence on a compact subset of S has a unique cluster point, then it converges. Ia The notion of a Cauchy sequence also remains fundamental. A sequence (xnlof points in a metric space (S,tf) is a Cauchy sequence if for a11 e, > 0, 3 Nz such that dxnbxmj < e,whenever n > Nz and m > Nv.The novelty is that Cauchy sequences in a metric space do not always possess limits. It is possible that the point on which the sequence is converging lies outside the space. Consider the space (Q,#s). The sequence fxn), where xn 1 + 1/2 + 1/6 + + 1/n! e Q, is a Cauchy 0', but of course, xn e (the base of sequence since Iak+1 l = 1/4n + 1)! the natural logarithms), an irrational number. A metric space (S,#) is said to be complete if it contains the limits of all Cauchy sequences defined on it. (?,dE) =
...
-->
-xn
--
Metric Spaces
is a complete space, while
81
Q,dE4
is not. Although compactness is aprimitive notion whichdoes notrequirethe concept of a Cauchy sequence, we can nevertheless detine it, following the idea in 2.12, in terms of the properties of sequences. This is often convenient from a practical point of view.
5.13 Theorem The following statements about a metric space (S,(f) are equivalent: (a) S is compact. (b) Every sequence in S has a cluster point in S. (c) S is totally bounded and complete. Ia Notice the distinction between completeness and compactness. In a complete space all Cauchy sequences converge, which says nothing about the behaviour of nonCauchy sequences. But in a compact space, which is also totally bounded, all sequences contain Cauchy subsequences which converge in the space. Proof We show in turn that (a) implies (b),(b) implies (c), and (c) implies (a). Suppose S is compact. Let fxn,n e IN) be a sequence in S, and detine a decreasing sequence of subsets of S by Bn fxk:k k n,J The sets Tn are closed, and the cluster points of the sequence, if any, compose the set C O';=1X,; IICIC. U';=I#,C,, If C 3, S (U= so that the open sets T'nc are a cover for S, and by assumption these contain a t'initesubcover. This means that, for some m < oo, S (O'2=1))C This leads to the contradiction Tm 0, so that C must LKLc n S U'''ol) be nonempty. Hence, (a) implies (b). Now suppose that every sequence has a cluster point in S. Considering the case of Cauchy sequences, it is clear that the space is complete; it remains to show that it is totally bounded. Suppose not: then there must exist an : > 0 for which such that no e-net exists', in other words, no tinite n and points (x1,...,AkJ x k. But letting n in this case, we have found a dxjtx S : for all j sequence with no cluster point, which is again a contradiction. Hence, (b)implies =
.
=
=
=
n=
=
=
=
=
.
:g
-->
(C).
Finally, 1et C be an arbitrary open covr of S. We assume that C contains no finite subcover of S, and obtain a contradiction. Since S is totally bounded it must possss for each n k 1 a tinite cover of the fonu B ni
sxni, 1/2'')i
=
y'
jyjjjjjyyjjjy'
1
=
...,kn.
(5.14)
,
Fixing ?z, choose an 1 for whlch Bni hs' or finite cover by C-sets (at least one such exists by hypothesis) and all thls st Dn. For n > 1, jlBni))=n 1 is also a jl tjjt X h & X n-1 jja.s Sujpcgyeyy COVCf1ng jw p n-1 ap wg ggyj gjyjpjjj Xyr,(,.)!y;yy;jy fojyjj* njj tg F yyjj . kyry,j;. :,, jyzjojjsea sequence of points (xn e D IS nonemp tyi yj.g.(..)t;(( C-sets, and accordlngly n, y .(-(j,jj(. . )y.( . y.kj. s l tontalns ak, and Ds+1 1s of radi us ) since Dn 1s a ball Of radm# .,.. yy,,.tyjjj;,,,,gy; .yr;yy, jj-j jygyrgjs jkgyy jjygj.. . jtyjjy yggjj jkgyy ;yy gjg gjygjs y....jjygjyyj jjyy;yy (j. yj.gj. ikyj. gj. ;jy yyyjy yjs gjygjyyyg jjygjs $j44,, gjs lln- gj and contalns Ak+1 dxngxq-j) .iJ (j. gyjj,, gjjgjj y 0'1: '''''j . !.E ',..j';;jjc''''j;.)',.*.,.,,;.;,)k,q..qL,.j'q,)j-,Ls))jj)q,;t)).L'j,.j !1..2,'... ......1812.....,.... 1t!12$!E;k..
w
,E
,
.
y
y
,..
..f.kj)
L;..
.
E
.
y y t .y..;.. .. y.... zt'i')>t7)Eitu . . .,2:jj,g.
,1
.
-
-
..
,-
...,,q..yyy-
..,,.j.)
.
...
.
y
..
,
....
..ksysj-sssssyyyssy,j.yr
,..syyyy
.,
.
',','t.)y;;r,))',tyyyiy),.,),y!(jjj,y)g.,,yiityy;j,t,y.,,yy,,,jyjj,
y
,
''''
..j.,
,,
..)
=
j,;.jr
..
d,
jj:rr;jj
''
-'::::g
.iltrr;,:r
..
',-
q.g...j-: .E , q..
.(...j!;r';-r:).,..-jj)...
jjggygi
.....
d.
...t.
,
:
ktrriiir'dt:l'.k;,d::r '-,.,,
'-::::::'
...,
g
..,.fj),)),)jj)j)),,
..
,
...
.
..
,
.rqrrty.r. .
.
.g .jjjj. j,.
y..ygyjjjy2jjj;;;.yyjjyygjjg ;,,.q.;u,..,..,,,,r,,..
jjjjjjya.y.
.jjjgjrj...
,4..
..,..
.,r,...r,r,,,.,....)))))jj,jj.j:)jj(j,;..(qjj.
yjyyyjyy.yyyyj;yyj.. ..,...yyyyyj;..;.
..g yt -t-;t4i!)7-.ygjjtt' -
.
yjjjgj.y,-pk@$1r....-,... . yjya
--
.'.
'' ' .' t.ylii. Ji. .;..
:..... ,-..
..
yjjjjjjjjyyjj.g. yy,,yjjjyyyj,... g.yy.. yyy ...yy.,yyyy;
....yyjgyyyyy..
..
.
--$t'-.
.
.yyjjyyjy.
jsggggggggy
,,..).:,.j
...,..y .
,y
..
yyyjy.yygyyjy,
..yjyyyy. a.aayyayay..
y.
..
y,
-;jj-t.yy.
.
-...
--jjy;;-
..
.
-zi
.
....
..
.. ....
.. yjyjy;.;yjjry,;;jy..j(.. .g...j.jgy,...;.g..g........y.y. ..... . rysg..;
jjryrrjyjtr
.
.')' yj..-T!rt)j-ji.. .. . '-ty,--i)... ,,- EEE; ;.,-. ,qjy)j)y(. . E ,;,'.-);,.. ;' E..:-...'.-..?.-.., ..k:, ;.hF(.f.
...
-..
,..y......
,
.g.yyy..
ysy. (gylrqE..jj
.
).
.
jjjjjj,jj),,,j r.:g; .,yjy,4E ..ypj.'q).k(jyj.,j:;II,..: ),j)jj,..t jpkzjlk.g))jjjjjj):...k -(...
.:;,,::7(22,-
bif'
-.;(
)F?jjjgjjj:... r.()
j .. y,y.q.yt. .E ;..... )i):y:.
j!jjjjj).
jtkkgry
jtg
,,'..
.-;p7.
...
.jrr,
jjyyg;jj jjygg;jjjggygi
.
.'r'y)'-'.
,444,.4
.j(r,
!j,
,jjy,
''
.....
.j;,k:rr
.!jjj;,.
.
''.'''''12.12
.
jl,.
-.
.E.'.:.'.y..E.E......:... ),1lpl-t''t,-;-)@--
.,
.
...
.
.tlih'tjjrt.;2).-:.lqt.1r):.
!,-
77?:?,-' .... ;,..-.. b3-,Lt;j,,1'3$;t311L;;,;(;L)3$L;-t..i;'.).j)). .'.ti;;jit)()(.. . ... ;-
1t;;$(yr';)))(;...---.j$)yyk?Ey-..,it). -. . '-'. -(jj(-,-.'. '.
.-..
yjgy;yyjygyyyyy jygyyyyy
.
.
.
..
,. -
.
.
.
. .
.
y;jjjjyy
82
Mathematics
< 9Iln ensures that Dn c Sxn. But this means Dn c A, which is a contradiction since Dn has no finite cover by C-sets, Hence C contains a finite subcover, and (c) implies (a). K In complete spaces, the set properties of relative compactness and precompactness are identical. The following is the converse of 5.5.
5.13 Corollary In a complete metric space, a totally bounded set A is relatively compact. Proof lf S is complete, every Cauchy sequence in A has a limit in S, and a1l such points are closure points of A. The subspace (A,#)is therefore a complete space. It follows from 5.12 that if A is totally bounded, X is compact. .
5.3 Examples The following cases are somewhat more remote from ordinary geometric intuition than the ones we looked at above.
5.14 Example In j12.3 and subsequently we shall encounter R=, that is, inhnitedipensional Euclidean space. lf x (x1,.n,...)e R=, and y @1,y2,...)e R= similarly, a metric for R* is given by =
=
X
d-x-y) >72-Q()(x,,y,), al
(5. 15)
=
where do is detined in (5.1).Like do, t/x is a bounded metric with #xtx,yl f 1 for all x and y. n (RX,Jx) is separable and complete.
5.15 Theorem
Proof To show separability, consider the collection
4
=
xk rational (x (x1,x2,...):
if k S m, xk
=
R=
=
0 otherwise
)
(5.16)
.
(A,u, m Am is countable, and by 1.5 the collection A and e: > 0, H x e Am such that able. For any y e =
R''O
m
=
1,2,... J is also count-
co
d-x,y)S y2-$ -al
+
-t=
5-' 2-4(0,yk)
S
m+ 1
e+ 2--.
(5.17)
Since the right-hand side can be made as small as desired by choice of e:and m, y is a closure point of z4. Hence, A is dense in R=. To show completeness, suppose (xn @1n,x2n,...), n e N ) is a Cauchy sequence in R ' Since dtxknvxkm)S 2J x @n, xm4 for any k, fxa, n e EN ) must be a Cauchy sequence in R. Since =
.
d-xx)
/
57l-kdxkakn) 1 -t=
+
2-'n
(5.18)
Metric Spaces
83
for all m, we can say that xn x @1,x2,...) xk for each k e R= iff xkn completeness of R implies that lxn) has a limit in R=. w 1,2,...., the -->
--)
=
=
5.16 Example Consider the infinite-dimensional cube', g0,1J=;the Cartesians product of an infinite collection of unit intervals. The space (g0,11=,#x) is separable by 5.8. We can also endow (0,11= with the equivalent and in this case bounded metric, X
772-11x: -ytl
Px(x,y) =
(5.19)
u
.
a1
In a metric space (S,*, where Jcan be assumed bounded without loss of generality, define the distance between a point x e S and a subset A ? S as dxl) infysxJtx,y). Then for a pair of subsets h,# of (S,#) define the function du 2 s x zs j..o R+ =
,
W here
2S
is the power set of S, by du(A,B4
max sup d@A), sup d(y,B)
=
xB
(5.20)
.
yGA
du(A,B) is called the Hausdorff distance between sets 4 and #.
5.17 Theorem Letting J'fs denote the compact, nonempty subsets of S, ls,du) is a metric space. satisfies 5.1(b) since the sets of Rs are 0, so that du is only a pseudo-metric for for any x E 4 and any z E C we have, by d is a metric,
Proof Clearly du satisfies 5.1(a). It closed, although note that duA, W) general subsets of S. To show 5.1(c), definition of dx.B) and the fact that =
sup dx,B) K supttftmz)
)
dz,B)
+
Ac#
xcA
(5.21)
.
Since C is compact, the infimum oyer C of the expression in braces on the righthand side above is attained at a point z e C. We can therefore write + dz,B)) < snp tnf(4(x,z) supltms) A ze xed
.xG
supltz,l). sup (;:,6'D z
(5.22)
.#
y', :'y' rtjy'E (;,'. :'. E' j::yt',.'Lli'y.jjy:yjj'lt-.kt4j?iyrj)j('.' y' j'yyjyy'. k'. ,', jjt!rrg;ji!r'd 111:::2;11!!'* :'i'.j,,g.' sgEy-yzgyj',d ;',rjjy't. (' !yj'. !,,' r' ;'y'g y' .L'. ;'. :@),'.,.,,;y.jjjj;g'j;g'kjrt.ilyjrg-7ldrjjklypysjlljpt'jjq'lrsr'd k' j't'yyjy,,, y'y .r' 44::7* jtd iE-':jd )' j'y;d qtd .y' jgy'1!. jydt.yy.rd j'gi i-(' litjjjltsiytj'y'. j);';' 4* r' yyd t':g'33t'. yjjy'. y;il'. y'jy,ytd yrd ytyy'.' jyy'. yrgj'.;yjyjd .'qL);)'. (''.. y' y-'( jy-'). (i'yd ygy';6!r;.'.jygjyj.yry!;(r!. jj',;L?'. .',jy',rg..jjjj. -'y -'
)k',.;jr!j'(;;'. .-' yy.j.-d .;j,' 33666666636663*, r)'... (tg,' ,,,,,,,,,,,,,'
''. .': .'.'.
sl'milarly,supyssltxz)
supzs
..??
...
-<
,-
.
.
:l::)Er!t. ax
s
(x
dr:s
ll.,l. (14:,.,, ..441k.
dEE.
.;)t:r
...1..-.
.
'..
.
..
.
'
:
.
,
.
'
,
and hence,
,
.
E ..',' .j .k.. 2..' . . . . . '...,-...,.'...g...,.....-,. ..-..:!:.......... ..,..k.., -.s. '..';
(2), drr;ir.n
---.11..
(z ) + supyss:
-<
'
.i(:).-, 'i.ii!Eil'.....,...... ..,1:11.. s..,...12.. tii''fL;.:, (;,..qI...sll.,l.jll:), .!147,.. ..(14;,.. :;.
......(2,...... . .
....
..
.
..
i
. . . ' ..
:
:
c: )y)j
'
.
'.'
. .
'
.
'
,,.-,.11..
.
.
..,..
..
:. .i. E
:
C
: E: E j) E ) ,)L.q/;j-i...L,j;...3L;.....L.. :.E.; )...E... ..'q''. . . tlLi: ..;..-..1qs? .... E..E:(:; . ifr: .q. LE . y(..!. . .: q ( q..;..sy . ;LL;.1Lj. y:.Ey.tjyy.tpyit;kr . . . .. .... ' . . E. .. . . LL..L......L. . :Lj.LL..:....j.L.. Ljf,. . .: . . E. . .. .g k!:..(#k:gk....jtt .... L).;):,.;.)..).;.:L.....j;. ....4:j.. :.. j . .: . :....:(. i. .E. .jg (.... .. ..,. .... .. j.; , :. g. . j y: .;....L.. ; . .kj. jr.). yr . j y. .. ;'. ' (yj ( jjyE. k:, y . ..j E :.' E .y . y! :. . .. .y-.L- . :. g.( q j yy.. j.:; . ..r . . r yg . .y . . . .. .. '. yy. ' y;. . . y y yy . : .,),.;.tj..$ : E i: . . . .. . . :yjytrr...,:jygj ;.,y..... y.y... ! gs.tjjtyyyjygjgytjrjry 4)f,Ljj;.L;;;j).);).f..j;q))..)):k.. L...j),)j)):)..,..,.))q,..L4)q);;q;. .. .L.(. j..-.. y y. yy . .yj r j:.j).,.;t,), yyrjyjjjjjj., y.y. y .j.. .g yy. y...
E
.1:7
-
j
.1,.!;
.
..:..jj..y!.)y.g;...::.).j
....(
..
....;)..
dr:;ir-
.
. .
t)t,.g..y t:
..
.. .
.
.
.
)5.jk; qL., ... .j.y.. (. ..tjtr .. ( y.)(.) .. j.jjyt.yjt... (j) )y(jjy,..t;.yy: y.. . . . . jj;, (yg j . .yt;jg yq . (..;.j j,(jj . t' ):b,. qi(gy-. q.. ...:::::::::( ..... yyj . j .: . . . .. ry .yt,. . . . .j.. y. . jy...yry.jj,s .. t)$:.,;jjg:. . i.. yy,y . ItIL. t.y;y. ... y kk))r.' kjl.yy ; (-. .'. jj;.,. ' ...g. Lqjjjf. rjy . yy4tjj,r.. fjjjj.. .. . ; 11k2. dtr:;.!r-'l)p--.--?-j:,,-(-,-----. ----.-b--b-,i-------;-jbv.... di:22:). ...t .-... dk:r:..'...i...---..-).)--..-.---...-..-........---...;...i..f..t--...-;:., dl::::: 4.:2::,. dliz!,.-.. 1lE5'..lI::!'... :llr7.:':1L. 4:7-.. . .-tk-. ,,- :.1. ,--L-.-----?,,--. II. :!i4;. .. t,!!i9;...qlr.... ....-.... .-........-k;!!iiIr... a.)-....-.--...-.,y.,-.;,:.-... 11:::,,. ..
ti!iiE;k ,(:!:; -,-
..
:
..
...)
. ...
.,.,..;yyj:
:(.ygjq
..
.
.
.
.-jji!!!ijiij''
-.-..11......
,j.!!.!!!!Ii;... ' .
....
.
,
.
.
.
. ..
,
. .
.
.
.
.
.
:
. .
.
....
-
.tg,
,
..
.
.
.
.;
,
.,y...
.
-
.
...
j)
..((y.- -.
- '
- . .
tjkj;j ..(.
..
.
,
.
,),))))j.
.
.
. . .
.
.
,,
(:3?'.i!EiI? dE!!!E
. -
.
(5.23)
.
. .
...
.,,,:
,
.
.r
ll.
'1r:.
.
- .
.
,,.
. .
..
.
.
.
...
.
.
.
.yjj.,.
.
-.
Mathematics
84
5.18 Example Let S = R. The compact intervals with the Hausdorff metric define a complete metric space. Thus, ((0, 1 1/n!, n l EN) is a Cauchy sequence which converges in the Hausdolf metric to (0,11. This is the closure of the set g0,1) which we usually regard as the limit of this sequence (compare2.16), but although (0.1) Cts, JVE0,1),(0,1q) 0. u -
=
Another case is where S @2 J s ) and J'fs contains the closed and bounded subsets of the Euclidean plane. To cultivate intuition about metric spaces, a useful exercise is to draw some figures on a sheet of paper and measure the Hausdorff distances between them, as in Fig. 5.3. For compact yt and B, duA,B) = 0 if and B; compare tlzis with another intuitive concept of the only if A between two sets', infxsx,yss#stmy), which is zero if the sets touch or =
,
tdistance
=
intersect.
'' . .. .
. .
. .
'
.
. .
*1'1 .
.
.
.. .
.
.. .
.. .
. .
'
'
.
.
'
' .
.
' .
. '
.
.
.
.. .
'
.:::
.
. .
.
' .
.
D
: : 727::!:::::12::!; :;:: :::7::t:::!:!J1:::: ::'' :: : : : 7t : : : : : : :: :: :: )(72:2 : 7 7 ! : : 2. :: : l : :: L : l : L: : : ' 2: : : 1 ) . t :2:::::::::r2:2E:1:::2r:2:J:;::;::::t:
. .
' .
'
.. : . . . ..
'
.
..
.'
' .
' .
'
. .
'
'
' .
. '
.
'
. '
. '
.
' .
' .
. '
.
' .
' .
.::::.:.:.:.:.:.:.:.::.:::.::l.:::.:::;;:::i::.:.:.::::.:,.. . : .
.
dr;lr
igrjr
(:) a//qik
aj!jjl ,
()h
Fig. 5.3
5.4 Mappings on metric spaces We have defined a function as a mapping which takes set elements to unique points of R, but the tenn is also used where the codomain is a general metric space. Where the domain is another metric space, the results of j2.3 arise as special cases of the theory. Sme of the following properties are generalizations of those given previously, while others are new. The terms mapping, transformation, etc., are again synonyms for function, but an extra usage ksfunctional, which refers to the case where the domain is a space whose elements are themselves functions, with (usually) R as co-domain. An example is the integral defined in j4.1. The function J: (S,#) F-> (T,p) is said to be continuous at x if for all e > 0 H > 0 such that ys
p(.f@),f(x)) < z. sup sdx,n)
(5.24)
Here, may depend on x. Another way to state the condition is that for e > 0 3 > 0 such that
f(&(x,))i
5'p(.f@),e).
(5.25)
Melrfc Spaces
85
where Sd and s'pare respectively balls in (S,J) and (7,p).Similarly, f is said to be unbrmly continuous on a set A i S if for al1 : > 0, 3 8 > 0 such that < :. sup p(.f(.y),J(x)) supy e 5'J(x,)t''4x
(5.26)
xGz4
Theorem 2.17 was a special case of the following important result. 5.19 Theorem ForA I 1-, f -1 (A) is open (closed)in S wheneverA is open in 1-, iff f is continuous at al1 points of S.
(closed)
.f-144).
Proof Assume A is open, and 1et fx) e A for x q for some E > 0. By 1.2(iv) and continuity at x,
f-'fsdx,nbLtj c
Sdxnn)
Wz have Spflxln
? A
J-1(Sp(f@),e)) i f-'(A).
(5.27)
A is closed and f -1(V A) s If A is open then by 1.2(iii), which -1 is closed if f (A) is open. This proves sufficiency. To prove necessity, suppose J -1(A) is open in S whenever A is open in 1-, and in for E > 0 is open in S. Since particu lar J-1(' p (/x),E)) e f-(S p (.j(x),:)), there is a 8 > 0 such that (5.25)holds. Use complements again for the case of closed sets. . -y-1(4)
''
-
=
-
.x
,
This property of inverse images under J provides an alternative characterization of continuity, and in topological spaces provides the primary definition of continuity. The notion of Borel measurability discussed in j3.6 extends naturally to mappings between pairs of metric spaces, and the theorem establishes that continuous transformations are Borel-measurable. The properties of functions on compact sets are of interest in a number of contexts. The essential results are as follows.
5.20 Theorem
The continuous image of a compact set is compact.
Proof We show that, if A c S is compact and Jis continuous, then /'tAlis compact. -1 Let C be an open covering of fA). Continuity of f means that the sets f @),B s C are open by 5.19, and their union covers A by 1.2(ii). Since A is compact, these sets contain a finite subcover, say, J (#1),...,J Bm4.lt follows that .A
.j
y(A)
c /'
.j
c UBj, U.f-'(#y)U.f(.f-'(>) y=1
(5.28)
-
jv1
y=1
is by 1.2(i) and ihe seond inclusion by 1.2(v). Hence, Bj,...,Bm is a finite subcover of fA) by Cij4ts. sinceT' is arbitrary, it follows .jyjyd -),'jLiii'''L. k' 11:222:::;* y'. ;'y'. jjj;jy!r;'. y' yyd !)' -i-E.(E' y'y y'ty' ry'j y;t'11I::2t!):.!q'-?q#').i. q' j-q,,,4fb.'. ..' jy'. g'g'. j';;;r2qpjjy.'. (' jj122222,,r*. y'(!jjt:r!''-, ;'(j1Ii(r:j11:qr:1I,;.. diIt!j:rjt)'. (jii).d (r:ljill;k.d .y' y' rlll:rjjd ..-,;;p9i;j:r'.-' .tjjjrrjjq:!)gry:yjj(,.'. gjjjjjjjjj'.' j';' jyj''' (.(;-'j4' :!jEi;,-' 1;:Ej5)11,,*. j'y .r.j;4'?i (' LjjkLbftjt' 11*. y-' L-t--..,,blfbqk:t''L.' rjjlg)d rljjj;;-d ..' y' yy'.'y yyy'...-y'yjj'y,(;jjry.qjjjyy.rjj(.. (yjjj.'. ty'-tj.ytjjj't. -.-,'-qi;'fqbI;;'.' (111:r::114.*. (jjlyrrr!ll.d .,' .g' ty.-y.jjy'..' t)j)-',. that fA) is compact. w )')66fLLLj. .jjqn,qjv...'k::kf'qktLLLl''
where the equality
''.'.' .'. .'rs .'.'jy,.
-(j(y'-y' -'. --((2:jp,'.' ,:pj(,jg.' -j;yjjjjj,jyjjj,,'. '-jjj::jjrjj;'. ;'F:' ..:,' ';t';--,::;;;jg!!r;y.-j)-' ---')::' -.' ,'ygjjjj)jjjj(yyrjj(y,'yyy -'.' -' ,':' :-'jj(jgj.y.' -'jyj' ;',',' E'jyyyy j'.-kk;'. )jyyjy-.'' t-,'. y'yd jjr:;rr:r;d jy'y'((r)))I''C' .i-'',,j(;)' y' t'.-jjjyjy:jjj,.' yrd .t't(;yy!ryryjjjgr;rryy.k' ('. jr:'-'.:.,,-j,jj;jjjjjjk,:.,y)--.)t;'i.'. ;,,*j3.*313413;*. ---'ibbfLi',,j,'jg'y.yyj-,-l t'yjytt'js j'yd .y' g' (y;'. 41kk2:::;* jjkykzrr:;d 11;j2r!:;'. .it')E y' :',jjjkjg,r;j'. ;'yyllllirrrjlk.'gj .-;'.jjjjjjrjjjjj.'. t.' :''jjj;:rjjjjj...' )' .y' j'yj,d
-,' .--'.'-.;k1ijj(2!t$;'y ..' .11;:2)* '',*4111::)*. .111:2k::1,,*. -' .'. ','.'i;'i---'jkg;jjpjr.. -'j);,4,j)tf,'3t',' jjryjpjiij'j.. .y'yyjjprrtjjj-;' ,'y.jy'-yj -'. ,k-'-
5.21Theorem If the set
Jis
. .
continuous on
.
,
.
.5.(' .
.)
..'
.
. '.. . .
'.
; y.' . j . '. C
.'
.
.
.
fj....
'
'
.
.
.
( (.
.
..
njpd set, it is uniformly continuous on .
.
' '. ' . . lrEi ''.
'.'
.
.
.
..
.
,.
F y. jjl kj.g::( .)()('; k...
E .r:
..j
.
....
.. .
.
.
j
.
.
ryyy,y',.'yt;9y'jr,tty:j(,.i.yyy,rr,,y; tt'tkttyt, d fr eacil x e A, continity '' a.t x S be compact. C'..h,ptt ...!)E;1il):il).l!;)7tj!: ;..r' .'ij.tE itJ))j#15! .. ... ' 4).k6.). -. 7(; (.q-( ... .. . tjq. .)j k;jy'jiI-k::IIL.. y-;:!iii;y-. 5. .. ' :,!iij;;.. y. li .;;,.:ijj,. ..f- lil-. .IIkrIT).. (,11:::,14,.. :!ij;;--:1EE5111,( ...jj...y....-....C4;CIIi:jyjj,yy.. j;j!jrr:jjy. (qjj::--. ,11I::2jII),. jjjjyrr;jj:. iljj;;: rjjygq 11(2rrjjIIl,. ' . .... . ..yy.. y...y(IIE:r:11l,.. y .. jjjjgl:::)).. . (j11(r:)l1... t!jjI(4(:jpt.. ' gg.. ... .. j . jL:jji,,'''. .. : . .. E j . y . . . . '.. i... . j.j y..... .... . Jiii .).. . .(. . . .r....yyj .. y. y jg.. j. . j. ..yy.))j,. .y . . r jjjyy:t.. ,jjjjj)f:63L... ' ' . ., rrrjj;!lL . . .d1EE!i1. .. . .q-.'.rllqq-ll:E),s jj!jjj;j,. 4422::, d;::::: . y j-. . t. j .....j . .44:)( 4(:4):2,j-:, 11E1!2, . .... (Iir11.. . . tlEsll, (Ir:'' IIEE:!'' ;,jt-.,...jr))-...jj;tj.yjy gj .yy;y-(...-.,#.,.yyj,y,k.-.-.-.yy..y?yy)......t-,,..-.yt;y-.. jky.y..,y(y.yy(jyy!..-jyy-.. . j.yy.y ..y .j.... , .) .. y yyy.yjjyt.. jyy.., j.y;jj.. yk,-..-jy.)..;(.,... ... ,yyy.jjj. . .. ... jj-.. )j.,...;,tyy; 44:)( ((;), ((;), ((;), .; ..... jy.. ;y...yjtj;.....,jy..-jj#,,-.j;L... . ,.y.... . .. . . . . . ...j..... . . .
Proof Let A
.1jlgq,
-:rjjgijr,.
,gjj!r:r
,,.
,gyy..t: ,;....
. .
y..
y
.
E
-(:!i-l'
'
--. ..!tr:())yry-
.
.
...
...
-...
. .
.
:.,..
.,-
.
-
.
... ..
.-y. -.
-.74.
;' ):: l;!. ?!-):;)!t27tE.''''. . ,b:3b... :.'.'.,.Ekikkir:....-.. .;
--k::::::
.j;1d:r7
' .
.
.
.
''.
.
.
.
-
.
..''.
..
.
'.
-
-, . -
.,
.
qE-y q. -
E $it)kyy.)jtt.i..rjjjy)y -.-y j. .- ,, . .
,
.
....
. -
-...
..
.
,-jjj,!r!,jjj(j-.
. .
.
y
.
.
.
.
.
..yy)....r.-kt..j). q. ..
....
.
.
.
,
y .:
.
.
;.;(gt;-). .-.
..
,j.
t;y
E
' '-'..
.. .
.
..
.'Erjj,?.
.
!.
.
.. .
:
.
.
, -
''
,g;j/r:'
.
.,
E'7
. .
y
.
jjjyyygyjjj. ...
.
-,yy -.,
y
.
.
y
...
tyy . ,. y..y..;jj,-, r,jj:j:r:;;. . y-jjrrjqyjjjjj-t., - ,)..t,y.-.y,.yj.t,,yt.,yy,(.t, .y . . .
,
,
.
,
..
,-,.
-
..
.
.
y.
.
j
j . j..yy,.-y.y?.-y......-E.-y-.q.-..-.(q.y . ......-j;.;.;..-,:.,j-).j.;k.q.;jr-,y,..y,.yyy.-.t.y4.
...r.
.
.
-
...(j.....
....
.
. .
.
.
.
.
-
y-. y. .y yyy.
.
-
y. .
. . . . yy. g y-. . y-. yj y. -gyy. . . . . . y.. .. . .
y. .
,
-
-.
Mathematics
86
1,...,-. compact they contain a finite subcover, say Sdlxknr, k consider of points x,y e S such that dlx,y) < minlckcmrk, and a pair < zlE, and also e Sdxk,rk) for some kn so that pfx,fyjj dxkvx)S dlxk,yt + dlx,y) f rk + 6 S lrk,
using the triangle
inequality. Hence plflxnfxl)
S
f
.
(5.29)
r1e,and
< e. S p(J(x),f (x1))+ p(Axk),J(y)) p(f@),J@))
Since, 8 independent of x and y,
Let 8 Now, y =
=
(5.30)
is unifonnly continuous on A. w
T' is 1-1 onto, and f and f -1 are cont inuous, J is called a homeomorphism, and S and are said to be homeomorphic if such a function exists. lf S is homeomorphic with a subset of T', it is said to be embedded in 'T by f. If f also preserves distances so that p(f(x4,f@)) #@,y) for each x,y e S, it is called an isometry. Metrics J1 and #2 in a space S are equivalent if and only if the identity mapping from (S,J1) to (S,J2) (themapping which takes each point of S into itselg is an homeomorphism. If
.f:
S
F..+
'r
=
5.22 Example If Jx and poo are the metrics defined in (5.15) and (5.19) respectively, the mapping gl (R=,#x) ((0,1)=,px),where g = (:1,:2,...)and --)
is an homeomorphism.
Xi
1
'(x) 2 -+
=
2(1
dxnyt<
t.tkJ is
Ixfl)
,
i
=
(5.31)
1,2,...
EI
Right and left continuity are spaces, but there is a concept of the range of the function. A semicontinuous at x if for each
If
+
not well defined notions for general metric continuity which is with respect to function f: (S,#) >-> R is said to be upper E > 0 3 5 > 0 such that, for y e S,
=
/'(J)
<
fx)
(5.32)
+:.
0, upper semicontinuity a sequence of points in S and #@n,x) implies limsupa/xn) f fx). The level sets of the form (x: f(x) < aJ are open for a1l (x e R iff f is upper semicontinuous everywhere on S. f is lower semiis upper semicontinuous, and f is continuous at x iff it is both continuous iff upper and lower semicontinuous at x. A function of a real variable is upper semicontinuous at x if it jumps at x with fx) > max(/'(x-),.f(x+)J;isolated discontinuities such as point h in Fig. 5.4 are not ruled out if this inequality is satisfied, On the other hand, upper semicontinuity fails at point #. Semicontinuity is not the same thing as right/left continuity except in the case of monotone functions', if f is increasing, right tlef't) continuity is equivalent to upper (lower)semicontinuity, and the reverse holds for decreasing functions. The concept of a Lipschitz condition generalizes to metric spaces. A function f on s,dj satisfies a Lipschitz conditionat x e S if for > 0 3 M > 0 such that, for any y e Sdx,), -->
-f
Metric Spaces P(.f(A'),.f@)) f Mhdx,yq)
(5.33)
where (.): R+ F-> ER+satisfies hd) 4, 0 as d 0. It satisfies a unrm Lipschitz conditionif condition (5.33)holds uniformly, with tixed M, for all x e S. The remarks following (2.9) apply equally here. Continuity is enforced by this conditionwith arbitrary and stronger smoothness conditions are obtained for specialcases of h. .1,
,
Fig. 5.4
5.5 Function Spaces The non-Euclidean metric spaces met in later chapters are mostly spaces of real functionson an interval of R. The elements of such spaces are graphs, subsets of R2. However, most of the relevant theory holds for functions whose domain is any metric space (S,#), and accordingly, it is this more general case that we will study.Let Cs denote the set of all bounded continuous functions S F-+ R, and .f:
de/ne
duf
g)
spl
=
J(x)
I
gx)
-
(5.34)
.
xe S
dv is a metric.
5.23 Thorem
Proof Conditions 5 1(a) and (b) re immediate. To prove the triangle inequality CN, write, given functions J, g and h .
ttd .yy.(' t'yyty'
t';'j' ;'E y' t'(' )' t''r' y;t'. j':'jyjy' j'. y' (' .);' j'g' .-j;'. -' -'.jt'
.'.'
dfbhj
=
SLIPj
...
Ies
:
'.
'
.
'
..
...
-
hlx)
.'t ' '.
''
.
.
...4(:). 11.,1.. 4:yx) <- 154, jj(:),. ...j),b:. ). ;.3:''.' .x.....j. (.'.(. ).t . 1......#..... . ).I..jI..j .........4(:4.. .......+.,.... .j.. .
j!jj,r
S
IE
<-d
'
.E .E j : ( .. . . . .'' .. .. . ... .g . j.j . .!#.,p
..
q' . . ; g.: . . ... g.. jy . , .. . '
g.. .
,g.)
.....:.(
'
. .' g.. . .
.
.
.
. bhli ..
,
(5.35)
.
;yyr.,
,, .;y:ry.,:tj..t.yjyEy.yj..-.y i.).. ;.jLE:.y. ... ... . j.y j q.. -
( .. E E riy; . E .. ;)(-. E :..
(y)j
4:yx)j
E :'
(
'
. .
. ..
rt).yjy r ; .y( y. .q.. .): .. . r:l).))-.-'jjyyjjgjj(jgj(yjj(ytk.:i'q).;'f'l. . (2yjy y.. . j.3fk.i.. .yy ..,)L-) t!y(j-(..y. .ytj) (j-EJII?). ..' ' @.q .; )jq)j,gL li.ygry j)..)j) j,yy .;yjgjjy;j(jjtj;;y yyjjt yj;( jkjgr jjyi jjjjyj. gjypj:g;yjjyj y;ry..jy.jir.,,.ggyyj.;, .;.g....y....y,j.t-(y.jy.yy. gj. .
.
.
E- .
. ..)...:t;!..k...
:
.
) ;
j
j/ri!r
--....
..
...jt
-i
. . .. - y jEjqy t..t(Ehi.( ..
..
y
r
-y,.. .. ...-
, -,:j;
..
:.r,
-
,
q:,);jj..-L.L,.j;;.,,fjfL.. . . . k; , ....
-
-,
,,
,jfjj),.
jy....(:.j)-ri..(..y,.q jkg;y jjgyr $, gjygjggjj gkyjryyt,..44:. j)yljjk.jj. . y. . gjjg;j .jj. yjjyjyyjjjyj..,.. .. .j ;q f't;jtj.. vy,, t(; , yy t-.-tjj,..--tj:.?-.,gjykjjj,y--,(11). ,1:2:11Lqf ....y)j.jyjJ;;.-,g;yyr; . dlzz!ri ,1:4::,1 (11.-y -:3)q.jL:'6t3. jl:r-.lk:. ,L...tj,j,(.L)L;));;;--):-:fi. yj jII2:bi:.I2:bi:. 66113L32:'......,. jyjy yryj,jjj-.;jjs---, llk.zll. ..--tE!i!t.r.' :i4; Jjiki11:::),. jlr,:lk.ylrrrllr.rlk.tjjgrr)l fyurjyjjy (; ou yj uou s z.lttrit/lltq $:E1,1, ,..).L......j .,,,)#t)-t!s,i(.. yjjy. j.... rlrr:)ks .. . t..-,,.. ,r,.,y.-,1,). ....,,....s....-r......-..k....y,.z..-. tt. .-,,.,,. -,t!)1y),ty,,j;-.,-..... '.y?.-.-. .. . (-v...;;,.--k.,,,. iills. dk,r jj-,......-,,,,,,,,,-,-,,.;2,... ;;,,,,-#.s---,...,.,...,..,.,...=
.-d,d!!qjj-.
--
IS Compact,
.
'1rr.
.
..,j
.k E'ktliikt..
.1rr.
r
..yyj
.E#.(. .'.j)
t
.,,
...
-
-
. .
?
.
y
4:(11. . 14k1(:2: 4(:;1!' 112. 4kj2:ir;lj!'i!r 4k22: Io i!'-l!hl!'i!. llk(1. (1. llr!l. jry;?-21. ma.j
S
(x) (x)+ glx)
.jj.x,
.
,
,
..,;,.yy.:
Cs
k!r)-;,. y.. & 11::,-.
y ut. .#.. . ..g
.
<,
,..
.
.,
. .
,
.-,..1.
.
.
.
)
.
,
,.-y-.
:(q)
s..,..l
....., ,..
.
,
.,
.
functi o ns lf tj. !lr:.L d:::r d:::r Ik,r (1k. l. dlE:,dt:EE::'t!!,; atl (:j,,. m(14:,, a t ev. 'EEE!E as . -,
.
-,k-,.
-,k-,.
,-
.
-r,,--
Mathematics
88
a uniformly continuous restriction to S, and evel'y f e Us has a continuous extension to V, say J, constructed by setting Jx) = J(x) for a' e S and limu/xu) for a sequence fx:.G S l converging to x, for each x e V-S. Note that duf,f). so that the spaces Ck and Us are for any pair fbf' G S, dvf'l but these isometric. There are functions that are continuous on S and not on cannot be uniformly continuous. The following is a basic property of Cs which holds independently of the nature of the domain S. 'x)
=
=
,
5.24 Theorem
(Cs,#g) is complete.
Proof Let (Jnl be a Cauchy sequence in C's ; in other words, for 6: > 0 3 Ns 1 such that dulfnnfm) f E for n,m > Nz. Then for each x e S, the sequences (/@) ) satisfy Ifnx) fmx)l s Jtz(/,Al; these are Cauchy sequences in R, and so have limits fx). In view of the definition of dv, we may say that jn J unifonnly in S. For any x,y 6 S, the triangle inequality gives -
-->
lfx)
-.f(A') l f
1/'(x) fnx) l + l/@) -
fnlL
-
I + Ifn
Fix e > 0. Since fn e Cs, ('i 6 > 0 such that Ifnlx) Also, by unifonn convergence (; n large enough that -
maxfIfx) fnlx)l Ifaly) f@lI l < -
-
,
so that Ifx) J(y)1< z. Hence f e -
!
-f(.y)
(5.36)
.
I< Jn(.y)
Y if dx,y)
13s,
<
.
(5.37)
C's, which establishes that Cs is complete. w
Notice hpw this property holds by virtue of the unifonn metric. It is easy to devise sequencej of continuous functions converging to discontinuous limits, but none of these are Cauchy sequences. lt is not possible for a continuous function to be arbitrarily close to a discontinuous function at every point of the domain. A number of the results to follow call for us to exhibit a continuous function which lies. uniformly close to a function in Us, but is fully specified by a finite collection of numbers. This is possible when the domain is totally bounded.
5.25 Theorem Let (S,J) be a totally bounded metric space. For any f G &s, there existj for any : > 0 a function g e Us, completely specified by points of the such that dvjqj < E. D and rational numbers J1,...,Jm, domain .x1,...,au
We specify rational numbers here, because this will allow ud to assert in applications that the set of a11possible g. is countable.
Proof n By total boundedness of S, R for 8 > 0 a finite ,1) xi, 1et Af = fx: dx,xi) k 26) and Bi (x: dxnxi) K gi: S r- 0, 1J by '
(x1,...,au)For
-net
=
gix)
dxli) =
dxAij
+ dlX,
s i ),
each and detine functions .
,
(5.38)
where dxA) = infysxltmy). dxl) is a uniformly continuous function of by constmction, and gilx) is also uniformly continuous, for the denominator is never .:
Metric Spaces
less than
).
89
Then define
'bl '4x) '7 -lp/xl7f
=
(5.39)
.
-1:/x)
Being a weighted average of the numbers (JfJ,gx) is bounded. Also, since (xflis for S, there exists for every x e S some i such that dxli) k as well a ZT=Ip(x) and < dx,Bi) dx,xi) Therefore, f k hence that gix) such as and unifonn continuity extends from the gi to g. For arbitrary f e &s, fix E > 0 and choose small enough that Ifx) fyl l < ls when #(x,y) < 26, for any x,y e G. Then tix m large enough and choose xf and ai Jfl for each i. 1,...,-, so that the Sxinn) cover S, and lfx < for i Note that if dxtx k 28 then e Aj and gix) = 0, so that in a1l cases -net
,
,
.
-
=
Y
-
-x
gixtlfx) fx I f
.ftxls.
(5.40)
-
Hence
gix)Ifxt
-
ai
I<
lfx) f
gix)
x
-
I+
Ifx
gix)
-
ai I
< gixtz for each x
6
(5.41)
S and each i. We may conclude that
duf,gl
=
sup Ifx)
-:(x)1
x6 5
S sup
ZT-1p(x)I fx)
Jfl
-
<
Z:'-1'/x)
xss
z.
(5.42)
.
The next result makes use of this approximation theorem, and is fundamental. It tells us (recallingthe earlier discussion of separability) that spaces of continuous functions are not such alien objects from an analytic point of view as they might at first appear, at least when the domain is totally bounded.
5.26 Theorem (i) If (S,#) is totally bounded then (&s,Jg) is separable. (ii) If (S,#) is compact then cn,dulis separable. Proof We need only prove part (i), since for part (ii), Cs Us by 5.21 and the same conclusion follows. Fix m and suitable points (xl,...,aulof S so as to define a countable family of functionsAm = fgmk,k e IN), whUre the g.k are defined as in 5.25, and the index k of rationals. For enumerates the countable collectibn of lrvectofs (J1,...,tu) > 0, there exists m large enppgh yhqt,for each f e &s, duf, gm < : for : each 5 ij, tits'ftibi, apd there exists gk e A such that somek. By 1. A limp-xA?nTh@'! 'd.ttlh,gtA 0. : is dense in &s. . < ,, dvy, gk) : for every > yytlyyttttt. yyty d1:::,i-....t).',t)'tt ..jj3i,1.. ,).() !j,, yj,jsy, EIl..jjojoug uyytsy mox geueraj jyy ..... ., To show that we canno :lrr-.d.pir,... j));;.,j);)q.;j...;);)... ;;)))..;....-..; .,.434... yy...;jygr-.j-y--.. .. )j-#).;-y--t..r.-,)g. y)-... yty. ...t-y.y-.)(tyy.y,yy. =
'
.
,
'.
.
=
'
..
.
.
..y,,y.
.)
'ril,.
'
.
' 'ttyljk
,
,,
'tij.'yjy,,yt.#,y,
--.i1friil-.
.1r:. .)ky,yy..yyy-. ..... r;-.)t);t@ -jyyt-r),t-.y.ytyj-,.-.
,typtl-l-y-y-)..
.
... . .
........... ..
-..y
.
-.
.
..
Mathematics
90
function space.
circumstances, we exhibit a nonseparable
5.27 Example For S (tz,:J,an interval of the real line, consider the metric space Dka,bndu) of real, bounded cadlag functions of a real variable. Cadlag is gauche) to describe droite, limites a colourful French acronym continue functions of a real variable which may have discontinuities, but are right continuous at every point, with the image of every decreasing sequence in Ltz,:)containing its limit point; in other words, there is a limit point to the left of every point. Of course, Cp,s) c Dp,x. Functions with completely arbitrary discontinuities form a larger class still, but one that for most purposes is too unstructured to permit a useful theory. =
To show that Dfa,bjsdu) is not separable, conslder the subset with elements
J/f)
0, t
<
0
=
1, t k 0
,
0
(5.43)
EJ,1.
6
This set is uncountable, containing as many elements as there are points in gtz,hJ. But duh,h 1 for 0 # 0', so it is also discrete. Hence Dta,bj,du) is not separable by 5.9. (n =
Let A denote a collection of functions f : (S,#) F-y (T,p). A is said to be equicontinuous at x G S if Y 6: > 0 3 > 0 such that
sup .feA y
sup pfyt,fxj) G
Sx,&)
,4 is also said to be unformly
supa:ssup
feA
s
(5.44)
< :.
if Y 6: > 0 3 8
equicontinuous
sup pfyl,fxl)
>
0 such that
(5.45)
< :.
yc Saxnnl
Equicontinuity is the property of a set of continuous functions (or uniformly continuous functions, as the case may be) which forbids limit points of the set to Cs (&s) but A is not be not (uniformly)continuous. ln the case when (uniformly) equicontinuous, we cannot rule out the possibility that W Cs (Us)k An important class of applications is to countable A, and if we restrict attention to the case A = ffn, n e IN) A c Cs torUs) may not be essential. lf we are willing to tolerate discontinuity in at most a finite number of the cases, the following concept is the relevant one. A sequence of functions f/, n e IN) will be said to be asymptotically equicontinuous at if Y E > 0 3 > 0 such that :4
,
.x
limsup l-yx
and asymptotically
sup
ye Sdxszj
?zzif/brzlfy
p(A(y),Jn(x)) < e,,
equicontinuous if V e, > 0 R
(5 46) >
0 such that
Metric Spaces
limsup sup n--:x
;r
e
sup
91
p(Jn@),n(x)) <
(5.47)
E.
Jx,&)
ye
If the functions fn are continuous for a11n, limsupn-yx can be replaced by supn in (5.46) and similarly for (5.47)when all the fn are uniformly continuous. ln these circumstances, the qualifier asymptotic can be dropped. The main result on equicontinuous sets is the Arzel-Ascoli theorem. This designation covers a number of closely related results, but the following version, which is the one appropriate to our subsequent needs, identifies equicontinuity as the property of a set of bounded real-valued functions on a totally bounded domain which converts boundedness into total boundedness.
5.28 Arzel-Ascoli theorem Let (S,#) be a totally bounded metric space. A set A c Cs is relatively compact under dv iff it is bounded and uniformly equicontinuous. Proof Since Cs is complete, total boundedness of A is equivalent to relative compactness by 5.13. So to prove if' we assume boundedness and equicontinuity, and construct a tinite E-net for A. It is convenient to define the modulus of continuity of f, that is, the function w: C xR+ h- R+ where ,
s
w(T,) Fix s
>
0, and choose 8
=
sup a.s s
(as is
If(.y4 J@)I
sup
.
ysdx,)
equicontinuity)
possible by unifonn
sup w(.f,)
(5.48)
-
such that
(5.49)
< :.
ysA
Boundedness of A under the unifonn metric means that there exist finite real numbers U and L such that inf
L f
/'lA.xe
be (x1,...,xm)
Let
S
#4x)f
sup DA,XC
U.
(5.50)
5
for S, and construct the finite family
-net
a Dm
fx) S
=
l'k
A, k
q
=
1,...,(v + 1)-)
(5.51)
according to the recipe of 5.25, with the constants ai taken from the finite set
where u and v are integers with v exceeding CU L4Ie and u 0,...,v. This set contains v + 1 real values between U and L which are less than 6: apal't, so that Dm has (v+ 1)= members, as indicated. Since the assumptions imply
fL
+
(&- Ljulvj
-
,
=
4 c &s, it follows by 5.25 that for every f E A there exists gk e Dm with < :. This shows that Dm is a e-net for A, and A is totally bounded. dvf,g ito suppose A is relatively compact, and hence totally bounded. To prove Trivially, total boundedness implies boundedness, and it remains to show unifonn equ icontinuity. Consider for E > 0 the set ionly
,
'
Mathematics
#k(:)
(/': w(/',1/k)
=
< :).
(5.52)
Uniform equicontinuity of A is the condition that, for any
large enough that
X'c
Bkz). lt is easily veritied
lw(.f,)
-
w(',*
6: >
0, there exists k
that
I f 2JrXJ,'),
(5.53)
R*,dE)
F--> is continuous. Bkz) is the inso that the function w(.,): (Cs,d verse image under w(.,8) of the half-line (0,:) which is open in R+, and hence
x 0 as k for each f e Bkzj is open by 5.19. By definition of Cs, w(/', 1/k) Cs In other words, w converges to 0 pointwise on Cs which implies that the Cs, and hence forW.But b.collection (#k(:),k e IN) must be an open coverinAfor covering is such of has A X hypothesis compact, every a finite subcover, and so A q Bke) for finite k, as required. . -->
--
.
,
6 Topology
6. 1 Topological Spaces Metric spaces fonn a subclass of a larger class of mathematical objects called topological spaces. These do not have a distance defined upon them, but the concepts of open set, neighbourhood, and continuous mapping are still well defined. Even though only metric spaces are encountered in the sequel (Part VI), much of the reasoning is essentially topological in character. An appreciation of the topological underpinnings is essential for getting to grips with the theory of
weak convergence. 6.1 Definition A topological space (X,I) is a set X on which is defined a topology, a class of subsets called open sets having the following properties: (a) X e T, O e 1. (b) If C i T, then U()scoe T. Oa G T, then 01 f-'h 02 q T. n (c) If 01 G 'r
'r,
These three conditions dehne an open set, so that openness becomes a primitive concept of which the notion of e-spheres around points is only one characterization. A metric induces a topology on a space because it is one way (thoughnot the only way) of detining what an open set is, and a1l metlic spaces are also topological spaces. On the other hand, some topological spaces may be made into metric spaces by defining a metric on them under which sets of are open in the sense defined in j5.1. Such spaces are called metrizable. A subset of a topological space (X,I) has a topology naturally induced on it by (A f''h 0: O G 'IJ is called the the parent space. If A c X, the collection relative topology for A. (A,n) would normally be referred to as a subspace of X. and If two topologies are defined on a space and :1 c T2, then T1 is said to be coarser, or weaker, than T2, whereas 12 is hner stronger) than I1. In particular, the power set of X is a topology, called the discrete topology, whereas (?,x J is called the trivial topology. Two metrics define the same topology on a space if and only if they are equivalent. If two points are close in one space, their images in the other space must be correspondingly close. If a set O is open, its complement Oc on X is said to be closed. The closure X' of an arbitrary set A ? X is the intersection of a1l the closed sets containing A. As for metric spaces, a set A i #, for B c X, is said to be dense in B if B c X. ':
'u
':1
=
.:2
6.2 Theorem The intersection of any collection of closed sets is closed. X and O are both open and closed. n
Mathematics
94
However, an arbitral'y union of closed sets need not be closed, just as an arbitrary intersection of open sets need not be open. For given e X, a collection Vx of open sets is called a base for the point x if for every open O containing x there is a set B e Vxsuch that x e B and B c 0. This is the generalization to topological spaces of the idea of a system of neighbourhoods or spheres in a metric space. A basefor the topology T on X is a and every x 6 0, there exists B e collection V of sets such that, for every O 6 that V such x e B c 0. The definition implies that any open set can be expressed union the of sets from the base of the topology; a topology may be defined for as specifying a base collection, and letting the open sets be defined as a space by the unions and finite intersections of the base sets. ln the case of R, for example, the open intervals form a base. .x
,r,
6.3 Theorem A collection V is a base for a topology I on X iff (a) Ulevf = X. (b) V #1,#2 e F and x e #1 chB1. 3 B5 e V such that a' e #3 c #1 /n %. Proof Necessity of these conditions follows from the definitions of base and open in terms of the base F, as follows: set. For sufficiency, define a collection 'r
O e
'r
iff, for each
.'&'
e 0, H B e F such that x
q
B c 0.
O s4tisfies the condition in (6.1), and X satisfies it given condition (a) of the theorem. lf C is a collection of T-sets, UtxwO e T since (6.1) holds in this case in respect of a base set B corresponding to any set in C which contains x. And if 01,02 G 1, and x e 01 fa O1. then, letting #1 and B1 be the base sets specified in (6.1) in respect of x and 01 and 01 respectively, condition (b) implies that x e B5 c 01 f''h O1, w ltic h shows that I is closed under finite intersections. Hence, is a topology for X. . ':
The concept of base sets allows us to generalize two further notions familiar points) of a set A in a from metric spaces. The closure points (accumulation topological space (X,'r) are the points x e X such that every set in yhe base of a7 contains a point of A (a point of A other than x). An important exercise is to show that x is a closure point of A if and only if is in the closure of z4. We have generalizations of two other familiar concepts. A sequence tak) of points in a topological space is said to converge to x if, for every open set O containing x, H N k 1 such that ak e O for all n k N. And is called a cluster point of (.:kl if, for every open O containing and every N k 1, xn e O for some n k N. ln general topological spaces the notion of a convergent sequence is inadequate for characterizing basic properties such as the continuity of mappings, and is augmented by the concepts of net and///er. Because we deal mainly with metric spaces, we do not require these extensions (seee.g. Willard 1970: Ch. 4). .x
.x
.x
6.2 Countability and Compactness The countability tutppu provide one classitkation of topological spaces according, roughly speaking, to their degree of structure and amenability to the methods
Topology
95
of analysis. A topological space is said to satisfy the first axiom of countability (to be jrst-countable) if every point of the space has a countable base. It if the space as a satisfies the second axiom of countability (is second-countable) whole has a countable base. Every metric space is first-countable in view of the existence of the countable base composed of open spheres, Sx, 1/n) for each x.
More generally, sequences in first-countable spaces tend to behave in a similar manner to those in metric spaces, as the following theorem illustrates.
6.4 Theorem ln a first-countable space, x is a cluster point of a sequence (.v, n e IN) iff there is a subsequence (xa#,k s IN) converging to x. Proof Sufficiency is immediate. For necessity, the definition of a cluster point implies that 3 n k N such that xn e 0, for every open 0 containing x and every N k 1. Let the countable base of be the collection Vx fBi, i G (11), and choose a monotone sequence of base sets (Ak, k G INl containing x (andhence nonempty) with A1 #l, and Ak c Ak-l f''h Bk for k 2,3,...; this is always possible by 6.3. Since x is a cluster point, we may construct an infinite subsequence by taking xnk as th: next member of the sequence contained in 4s for k = 1,2,... For every open set O containing x, R N k 1 such that xnk E Ak 0, for a1l k k N, and hence xnk x, as required. w --> x as k .x
=
=
=
.-..:
The point of quoting a result such as this has less to do with demonstrating a new property than with reminding us of the need for caution in assuming properties we take for granted in metric spaces. While the intuition derived from R-like situations might lead us to suppose that the existence of a cluster point and a convergent subsequence amount to the same thing, this need not be true unless we can establish first-countability. A topological space is said to be separable if it contains a countable dense subset. Second-countable spaces are separable. This fact follows directly on taking a point from each set in a countable base, and verifying that these points are dense in the space. The converse is not generally true, but it is true for metric spaces, where separability, second countability and the Lindelf property (that every open cover of X has a countable subcover) are all equivalent to one another. This is just what we showed in 5.6. More generally, we can say the
following.
6.5 Theorem A
jecond-countable
space is both separable
and Lindelf.
Proof The proof of separability is in the text above. To prove the Lindelf property, let C be an open cover of X, such that UzeCA X.For eachA e C and e A, we can find a base set Bi such thgt x 6 Sj c A. Since UT=1#fX, we may choose a countable subcollection Af, i 1,2,... such that Bi c 4f for each i, and hence U7=1Xf X. K .x
=
=
=
=
A topological space is said to be compact if every covering of the space by open countably compact if each countable sets has a finite subcover. It is said to e coveringhas a finite subcovering. And it is said to be sequentially compact if subsequpce. Somesimes, compacteach sequence on the space has a convrgnt
96
Mathematics
ness is more conveniently characterized in tenns of the complements. The complements of an open cover of X are a collection of closed sets whose intersection is empty', if and only if X is compact, every such collection must have a finite subcollection with empty intersection. An equivalent way to state this proposition is in terms of the converse implication. A collection of closed sets is said to have the Tnite intersection property if no finite subcollection has an empty intersection. Thus: 6.6 Theorem X is compact (countablycompact) if and only if no collection (countable collection) of closed sets having the finite intersection property has an empty intersection. a The following pair of theorems summarize important relationships different varieties of compactness.
between the
6.7 Theorem A first-countable space X is countably compact iff it is sequentially compact. Proof Let the space be countably compact. Let fxn,n G EN ) be a sequence in X, and detine the sets Bn fxn,ak+1,... ) n 1,2,... The collection of closed sets (Tn, n e (N) clearly possesses the finite intersection property, and hence (X'n is nonempty by 6.6, which is another way of saying that fak) has a cluster point. Since the sequence is arbitrary, sequential compactness follows by 6.4. This proves necessity. For sufficiency, 6.4 implies that under sequential compactness, al1 sequences in X have a cluster point. Let ( Ci, i e INl be a countable collection of closed sets having the finite intersection property such that A,y 0Z=IC/ # 0, for every finite n. Consider a sequence fxn) chosen such that xn e A,,, and note since (A,, ) is monotone that xn e Am for all n 2 m; or in other words, Am contains the /?z). Since (ak) has a cluster point x and Amis closed, x e Am. sequence fak,n k This is true for every m e N so that OT=1Q is nonempty, and X is countably 6.6. by compact w =
,
=
=
,
6.8 Theorem A metl'ic space (S,#) is countably Proof
compact iff it is compact.
Sufficiency is immediate. For necessity, we show first that if S is countably compact, it is separable. A metric space is first-countable, hence countable compactness implies sequential compactness (6.7),which in turn implies that every sequence in S has a cluster point (6.4).This must mean that for any 6: > 0 there exists a finite E-net (m,...,auJ such that, for a1l x e S, dx,xk) < E, for some k E ( 1,...,-) ; for otherwise, we can construct an infinite sequence fxn) with dxn,xn') k 6: for n # n', contradicting the existence of a cluster point. Thus, for each n e IN there is a finite collection of points 4,, such that, for is countable and every x e S, #@,y) < 2-n for some y e An. The set D = U*n=1An dense in S, and S is separable. Separability in a metric space is equivalent by 5.6 to the Lindelf property, that every open cover of S has a countable subcover', but countable compactness implies that this countable subcover has a finite subcover in its turn, so that
Topology
compactness is proved.
97
.
Like separability and compactness, the notion of a continuous mapping may be defined in tenns of a distance measure, but is really topological in character. In the mapping /': X F--y f is said to be contina pair of topological spaces X and uous if J -1(#) is open in X when B is open in and closed in X when B is closed in That in metric spaces this definition is equivalent to the more familiar one in terms of s- and 8-neighbourhoodsfollows from 5.19. The concepts of homeomorphism and embedding, as mappings that are respectively onto or into, and 1-1 continuous with continuous inverse, remain well defined. The following theorem gives two important properties of continuous maps. ,
,
.
6.9 Theorem Suppose there exists a continuous space X onto another space (i) If X is separable, T is separable. is compact. (ii) If X is compact,
mapping
f
from a topological
.
problem is to exhibit a countable, dense subset of Consider .f(D4 where D is dense in X. lf J(D) is the closure of J(D), the inverse image f-JfD is closed by continuity of f, and contains f -1(/'(D)), and hence also coptains D by 1.2(iv). Since X'is the smallest closed set containing D, and X c 5, /X) it follows that X c J-1C(D)). But since the mapping is onto, f ff-1(fD))4 c J(D), where the last inclusion is by 1.2(v). J(D) is therefore ' in dense as required. J(D) is countable if D is countable, and the conclusion follows directly. (ii) Let C be an open cover of Then (J -1(#): B e C) must be an open cover of X by the definition. The compactness of X means that it contains a finite subf-1B n ) such that cover, say f -1(B l ) Proof
(i) The
.
=
.
,...,
y'
=
y(x) y =
Uy-'tsp= J y-1 tjsy c /=1 Usy,
y=1
(6.2)
j=L
where the third equality uses 1.2(ii) and the inclusion, 1.2(v). Hence C contains a Gnite svbcover. . in both these results. The extension Note the importance of the stijulation and f is obvious, and can be supplied of (ii) to the case of compact subsets of X reader. the by Completeness, unlike separability, compactness, and continuity, is not a topological property. To define a Cauchy sequence it is necessary to have the concept of a distance between points. One of the advantages of defining a metric on a space is that the relatively weak notion of completeness provides some of the essential features of compactness in a wider class than the compact spaces.
6.3 Separation Properties Another classification of topological spaces is provided by the separation tztppzu, whichin one sense are more primitive than the countabilityaxioms. They are
Mathematics
98
indicators of the richness of a topology, in the sense of our ability to distinguish between different points of the space. From one point of view, they could be said to define a hierarchy of resemblances between topological spaces and metric spaces. Don't confuse separation with separability, which is a different concept altogether. A topological space X is said to be: with x # y 3 an open set containing x but not y and - a Fl-space, iff V x,y e X also an open set containing y but not Fa-) space, iff V x,y e X with x # y 3 disjoint open sets 01 and - a Hausdotffo 01 in X with e 01 and y E O2; regular iff C Hdisjoint open sets 01 and for each closed set C and space - a 02 with x e 01 and C c O2; - a normal space iff, given disjoint closed sets C1 and Cz, 3 disjoint open sets 01 and 01 such that C1 c 01 and (72 c O2. .x;
.x
.x
A regular Fl-space is called a Fg-space, and a nonnal Fl-space is called a F4-space. In a Fl-space, the singleton sets (.:) are always closed. In this case y e (xJf (xJC is the complement of a closed set, and hence open. whenever y # x, where Conversely, if the F1 property holds, evel'y y # x is contained in an open set not containing x, and the union of all these sets, alsp open by 6.1(b), is (x)f. It is easy to see that F4 implies Tz implies Fa implies T1, although the reverse implications do not hold, and without the F1 property, nonnality need not imply regularity. Metric spaces are always F4, for there is no difficulty in constructing the sets specified in the definition out of unions of E-spheres. We have the following important links btween separation, compactness, countability, and metrizability. The first two results are of general interest but will not be exploited directly in this book, so we forgo the proofs. The proof of 6.12 needs some as yet undefined concepts, and is postponed to j6.6 below.
6.10 Theorem A regular Lindelf
space is normal. a
6.11 Theorem A compact Hausdorff space is F4. 6.12 Urysohn's able. u
metrizatlon
(a
theorem A second-countable
Fg-space is metriz-
In fact, the conditions of the last theorem can be weakened, with F4 replaced by Fa in view of 6.10, since we have already shown that a second-countable space is Lindelf (6.5). The properties of functions from X to the real line play an important role in defining the separation properties of a space. The key to these results is the remarkable Urysohn's lemma.
6.13 Urysohn's Iemma A topological space X is nonnal iff for any pair A and B of disjoint closed subsets there exists a continuous function f: X (0,1j such = 0 and J(#) 1. :l that jA) .-+
=
The function
J is
called a separating
function.
Topology
99
Proof This is by construction of the required function. Recall that the dyadic rationals D are dense in (0,1j. We demonstrate the existence of a system of open sets f Ur, r e D l with the properties A c &r;
Vrfa
#
'X c
0',
=
(6.4)
Ur for r
>
(6.5)
s.
Nonnality implies the existence of an open set 171/2(say)such that 171/2contains :4 an d (&1/2)ccontains B. The same story can be told with &f/2 replacing B in the role of C1 to define 171/4,and then again with 171/2replacing A in the role of C1 to define Um. The argument extends by induction to generate sets ( Umnn, m 1,...,2* 1 ) for any n e EN,and the collection ( Ur, r e D ) is obtained on letting n --A x. lt is easy to verify conditions (6.3)-(6.5)for this collection. Fig. 6.l illustrates the construction for n 3 when A and B are regions of the plane. One of the onion' in the limiting case. must imagine countably many more =
-
=
)1
!/3/8
315ls
:/1/2
!/3/4 Llzls
:/1/4
U 1/8
B
,4
Fig. 6.1 Now define
f:
X
(0,11by
--)
inffr
e D: x e &r) x e ,
Urss Ur
x Urso Ur
(6.6)
-
1 for x e #. Because of the monotone property (6.5), where, in particular, we have f:r any a e (0,1) .f(x)
=
(x: fx)
<
al
(x:inf fr e D: x e = (x:H r < a such that =
&r, = r
<
a)
x e &r)
(6.7)
which is open. On the other hand, because D is dense in g0,1qwe can deduce that,
Mathematics
100
for any k$e
0, 1),
(x: flx)
<
I31 (x:inf tr e =
D: x e &rl <
= (x: x e Ur V r
>
jlJ
))
Ur O rn.js
=
=
r)
(6.8)
r
r>f$
which is closed. Here, the final equality must hold to reconcile the following two and second that, for al1 r > I3,there exists (sinceD facts: first that Ur i and Us Ur by (6.5).We have therefore shown that, > > dense) D with is se r s k$ c for 0 S ;$< a ; 1, ,
(x: j
<
fx)
<
a)
(.x:flx) <
=
a)
fa
fx: flx) K
;'$)C
(6.9)
is open, being the intersection of open sets. Since every open set of (0,11is a union of open intervals (see2.5), it follows that f -1(A) is open in X whenever A is open in g0,11,and accordingly f is continuous. It is immediate that f(A4 0 and f(#) 1 as required, and necessity is proved. Sufficiency is simply a matter, given the existence of f with the indicated and J-1((11j), whose images are PrOPerties of citing the two sets J-1(g0,1)) a a, and respectively, in X, disjoint, and contain B so that X is normal. w open =
=
,
,4
It is delightful the way this theorem conjures a continuous function out of thin air! It shows that the properties of real-valued functions provide a legitimate means of classifying the separation properties of the space. In metric spaces, separating functions are obtained by a simple direct construction. If 4 and B are closed and disjoint subsets of a metric space (S,#), the nonnality property implies the existence of > 0 such that infxsz,yswJtmy) k 6. The required function is
fx)
=
Jtx,Al dxv)
+ ty(x,s)
(6. 10)
where dxl) = infysxttx,y), and J(x,S) is defined similarly. The continuity of f follows since dxl) and #(x,#) are continuous in x, and the denominator in (6.10) A similar construction was used in the proof of 5.25. is bounded below by The regularity property can be strengthened by reqtliring the existence of separating functions for closed sets C and points x. A topological space X is said to be completely regular if, for a1l closed C c X and points x C, 3 a continuous 0 and flx) = 1. A completely regular Fl-space function fl X 1- E0,1)with fc' is called a Tychonoff space or Fgj-space. As the tongue-in-cheek terminology suggests, a L-space is F3j (this is immediate from Urysohn's lemma) and a F3j-space is clearly F3, although the reverse implications do not hold. Being F4, metric spaces are always F?j. .
=
Topology
101
6.4 Weak Topologies Now let's go the other way and, instead of using a topology to define a class of real functions, use a class of functions to define a topology. Let X be a space and F a class of functions f : X i-- y, where the codomains y are topological under which spaces. The weak topology induced by F on X is the weakest topology -1 is continuous. Recall, continuity F (A) is means that J open in X every f e whenever A is open in y. We can also call the weak topology the topology generated by the base of sets V consisting of the inverse images of the open sets of the y, under f e F, together with the finite intersections of these sets. The inverse images themselves are called a sub-base for the topology, meaning that the sets of the topology can be generated from them by operations of union and finite
intersection.
increase the number of sets in this base and If we enlarge F we (potentially) get a stronger topology, and if we contract F we likewise get a weaker topology. With a given F, any topology stronger than the weak topology contains a richer collection of open sets, so the elements of F must retain their continuity in this case, but weakening the topology further must by definition force some fq F to be
discontinuous.
The class of cases in which y R for each f suggests using the concept of a weak topology to investigate the structure of a space. One way to represent the richness of a given topology on X is to ask whether contains, or is contained in, the weak topology generated by a particular collection of bounded real-valued functions on X. For example, complete regularity is the minimal condition which makes the sol4 of construction in 6.13 feasible. According to the next result, this is sufficient to allow the topology to be completely characterized in terms of bounded, continuous real-valued functions on the space. =
':
'r
6.14 Theorem lf a topological space (X,':) is completely regular, the topology is the weak topology induced by the set F of the separating functions. ':
Proof Let F denote the collection of inverse images of open sets under the funcr tions of F. And 1et F denote the weak topology induced by F, such that the F-sets, together with their finite intersections, fonn a base for F. We show that F T. For any x e X, let O e be an open set containing x. Then Oc is closed, and by l and complete regularity there exists f e F taking values in (0,1j with J(I) = J(O9 J-1((, 1q) is therefore an is open in g0,1j, and B 0. The set (1z,1) with Oc containing disjoint and that B c 0. Since this holds for x so open set, every such 0, x has a base Vx consisting of inverse images of open sets under functions from F Since x is arbitrary the collection V = ( Vx,x e XJ forms a base for T. lt follows that T ? F. On the other hand, F is by definition the weakest topology under which every f E F is continuous. Since f E F is a separating function and continuous under it also follows that F c 1. . =
'r
=
=
.
'r,
Mathematics
6.5 The Topology of Product Spaces Let X and f be a pair of topological spaces, and consider the product space X x The plane R x R and subsets thereof are the natural examples for appreciating the properties of product spaces, although it is a useful exercise to think up more exotic cases. An example always given in textbooks of topology is C XC where C is the unit circle; this space has the topology of the torus (doughnut). Let the ordered pair (x,y)be the generic element of X x f. The coordinate defined by projections are the mappings 7:x: X x F-) X and z:v: X x f F-> .
,
lx(.Y,')
=
A'
(6.11)
1:(-Y,)/)
=
y.
(6.12)
are topological spaces, the coordinate projections can be used to generate a new topology on the product space. The product topology on X x is the weak topology induced by the coordinate projections. The underlying idea here is very simple. lf A i X and B q f are open sets, the 1 1(#) will be lk - (z4)an d X x B = 1:yset A x B (A x f) rn (X x #), where A x f rectangle called of X The product regarded as open in X x and is x f. an open topology on X x is the one having the open rectangles as a base. This means that and @2,y2) two points @1,y1) are close in X x provided xl is close to x2 in X, antl yl to y2 in f. Equivalently, it is the weakest topology under which the coordinate projections are continuous. If the factors are metric spaces (X,#x) and f,h), several metrics can be constructed to induce the product topology on X x including If X and
=
=
,
,
,
@2,y2)) maxllxtxl,xzl, p((x1,y1), =
Jr(A'1,y2)l
(6.13)
and p'(@1,A'1), @2,y2)) =
#x@1sx2)+#:/1,y2).
An open sphere in the space (X x p), where p is the metric in happens to be an open rectangle, for ,
5'p(@,A'),)
=
Sdsxnnt
X Sdj,vnt',
(6.14) (6.13),
also
(6.15)
but of course, this is not true for every metric. Since either X or may be a product space, the generalization of these results from two to any finite number of factors is straightforward. The generic element of the space X2=1Xfis the n-tuple (x1,...,ak: xi e Xf), and so on. But to deal with infinite collections of factor spaces, as we shall wish to do below, it is necessary to approach the product from a slightly different viewpoint. Let 4 denote an arbitrary index se and fXa, (x l A ) a collection of spaces indexed by is the collection of a1l the mappings x: 4 4. The Cartesian product X Xasxxa F-y Uaszxa such that x(a) e Xa for each (x e A. This definition contains that given in j1.1 as a special case, but is fundamentally more general in character. The coordinate projections are the mappings 1a: Xn, F-> Xa with ,t
=
Topology
103
l(x(.%) Xq),
(6.16)
=
but can also be defined as the images under x of the points a e A. Thus, a point in the product space is a mapping, the one which generates the coordinate projections when it is evaluated at points of the domain z4. In the case of a finite product, A can be the integers 1,...,l. In a countable case A EN,or a set equipotent with EN,and we should call an infinite sequence, an element of =
.x
X'='(say).A familiar uncountable example is provided by a class of real-valued functions x: R F-> R, so that A R. In this case, x associates each point a e R R with a real number x(a), and defines an element of the product R The product topology is now generalized as follows. Let (Xa, a e A ) be an arbitrary collection of topological spaces. The Tychonoff topology (product X topology) on the space X h as as base the finite-dimensional open rectangles, sets where the Oa ? Xa are open sets and Os Xa except for at of the form Xas.4otx, most a finite number of coordinates. These basic sets can be written as the interkections of tinite collections of cylinders, say =
.
=
B
-1 aajtoaj)
=
-1
f-h
chzramto oo ;
...
,
o j'y; .
for indices a1,...,a,a e A. Let be a topology under which the coordinate projections are continuous. If Oz and hence contains the Tychonoff topology. Since is open in Xa, aa-1(O(: ) e this is true for any such 1, we can characterize the Tychonoff topology as the weak topology generated by the coordinate projections. The sets C:a-1(Oa) fonn the sub-base for the topology, whose hnite intersections yield the base sets. Something to keep in mind in these intinite product spaces is that, if any of X the sets Xct are empty, X i s emp ty. Some of our results are true only for nonempty spaces, so for full rigour the stipulation that elements exist is desirable. 'r
'r
':
,
6.15 Example The space hdv) examined in j5.5 is an uncountable product space having the Tychonoff topology; the unifonn metric is the generalization of the maximum metric p of (6.13).Continuous functions are regarded as close to one another upder dv only if they are close at every point of the domain. The subsequent usefulness of this characterization of C,du) stems mainly from the fact that the coordinate projections are known to be continuous. n Thetwoessentialtheorems onproductspacesextendseparability andcompactness from the factor spaces to the product. The following theorem has a generalization to uncountable products, which we shall not pursue since this is harder to prove, and the countable case is sufticient for our purposes.
6.16 Theorem Finite or countable product spaces are separable under the product topology iff the factor spaces are separable. Proof The proof for tinite products is an easy implication of the countable case, hence consider X* XT=fXf.Let Df fdikndizn... l c Xf be a countable dense set by D each and construct a set i, detining for c =
=
X''O
104
Mathematics
Fm
=
Xojx
i=m+ 1
f=l
fl l
(6. 18)
for m = 1,2,..., and then letting D U;=IF-. Fm is equipotent with the set of m-tuples formed from the elements of the countable DI ,...,Dm, and is countable by induction from 1.4. Hence D is countable, as a countable union of countable sets. We will show that D is dense in X=. Let B = X7=IO/ be a non-empty basic set, with Oi open in niand Of niexcept for a finite number of coordinates. Choose m such that Oi Xf fOr i > m, and then =
=
=
m
B ra Fm
=
=
x X ftl X((?/rno/) f=rn+1
=1
)
y,
?.
(6.19)
l,...,m. Since recalling that the dense property implies Oi fa Di # 0, for B fa Fm B ?'7 D, it follows that B contains a point of D; and since B is an arbitrary basic set, D is dense in X7 as required. w =
One of the most powerful and important results in topology is Tychonoff's theorem, which states that arbitrary products of compact topological spaces are also compact, under the product topology. It will suftice here to prove the result for countable products of metric spaces, and this case can be dealt with using a more elementary and familiar line of argument. lt is not necessary to specify the metrics involved, for we need the spaces to be metric solely to exploit the equivalence of compactness and sequential compactness.
6.17 Theorem A finite or countable product of separable metric spaces (Xf,) is compact under the product topology iff the factor spaces are compact. Proof As before, the finite case follows easily from the countable case, so assume X= = X:=1Xj, where the Xf are separable spaces. In a metric space, which is first countable, compactness implies seprability and is equivalent to sequential compactness by 6.8 and 6.7. Since Di is sequentially compact and tirst-countable, every sequence (xfa,n e INl on Xf has a cluster point xi on the space (6.4)k Applying the diagonal argument of 2.36, there exists a single subsequence of integers, (nk,k E l ), such that xink xi, for every i. Consider the subsequence xw,...). In the product topology, xy in X*, lau, k s N ) where xng @1n:, which proves that X* is sequentlally xo...) iff for i, every xink xi .x (.x1, compact. X= can be endowed with the metric px Z7wl4/zf,which induces the product topology. X= is separable by 6.16, and sequential compactness is equivalent to compactness by 6.8 and 6.7, as above. This proves sufficiency. Necessity follows fzom 6.9(ii), by continuity of the projections as before. w -..-9
=
=
--
--
=
6.18 Example The space R= (see5.14) is endowed with the Tychonoff topology if we take as the base sets of a point the collection .x
N(x.,k,
(y: la)
-yjl
=
< :,
=
1,...,k1., k e N, er > 0.
(6.20)
A point in R* is close to a7 in this topology if many of its coorclinates are close
Topology
to those of x; another point is closer if either more coordinates are within 6: of each other, or the same coordinates are closer than e, or both. The metric Jx defined in (5.15)induces the topology of (6.20).If (xn) is a sequence in X, dxlxnvx) 0 iff V z,k 3 N 1 such that xn e N(x;k,e) for a1l n k N. We already know that R= is separable under Jx (5.15),but now we can deduce this as a purely topological property, since R* inherits separability from R by 6.16. (:1 -->
The intinite cube (0,1j=shares the topology (6.20)with R= and is a compact space by 6.17; to show this we can assign the Euclidean metric to the factor spaces (0,1j. The trick of metrizing a space to establish a topological property is frequently useful, and is one we shall exploit again below.
6.6 Embedding and Metrization Let X be a topological space, and F a class of functions F-y tion p'lzw :: X/rfyis the mapping defined by
exjf
y. The evalua-
J: X F->
xt.
=
(6.21)
The class F may be quite general, but if it were finite we would think of d(x) as the vector whose elements are the J(x),J e F (6.21)could also be written z:yoe = f where ay is the coordinate projection. A minor complication arises because f need not be onto y, and :(X) c Swy is possible. If 4 is a set of points in ly -1(4) may contain points not in e(X). We therefore y, the inverse projection need to express the inverse image of A under f, in tenns of c. as .
:-1(ay-1(4)
f -144) (xyo:)-1(4) =
=
f...j c(x))
c
y.
(6.2:)
The importance of this concept stems from the fact that, under the right conditions, the evaluation map embeds X in the product space generayed by it. It would be holeomorphic to it in the case c(X) Skyy. =
6.19 Theorem Suppose the class F sparates points of X, meaning that fx) # J@) for some f G F whenever x an d y are distinctpoints of X. If X is endowed with the weak topology induced by F, the evaluation map defines an embedding of X into ly J .
.' ,'--f,,' -' ' :.);' .'...' .'-.' .,'
k' r' ::'i;t' lkd ;'y' j' f'.f..
tjd f'
Proof lt has to be shown that e is a 1-1 mapping from X onto a subset of Xro which is continuous with continuou: inverse. Since F separates points of X, e is 1-1, since c@) # :(.y) w he never flx) #(.f@). for some f e F. To show continuity of -1 X rwheneker A is open in y under the weak e, note first that f (A) is open in topology, and sets of the form ly*(A) i open in X/ywith the product Bltt e (z:y(A)) (ayoc) (A) topology, since the projecons are y yyj ; j lh, under e of sets of the form gs J- (A), so we can conclude that. thte '-' .. ).' .T.'1.). (qgt. .y. .g j(. ..t. ),q:.., yj.).yt) ). tyi,j:lt. . .. . y'yt,jgyy,kytt,jyyyryltysy invrfs unions and intersections t; ay-1(4), 4 c :y, are open. since #.-;y.;.L.L.y ( .r (..eryt yrve y (t. (.'( ). ty t . . se sets of Xfy,which are (see 1.2) the same property extrh.d, kygtk..qjlkyrltlqplttit,tt,i ltjj tjju illwl invfj tuj , product topology, and tjo ) r))t.()),qy:,. jjy y Ljjqj.jlqjtjlllq-kjLb. f-iite intersections of these .. . ..; p cztt' yyyt( j))j;)).jjjy)(4tr.)jk;)r...'...))' yyyjyy, y y ,y...'y#)tyv-y..y,' ,.;;.bitiL.) !!F:1. L.)iLj;;Lb'-j)'3';'bb'' 1t1!4,1(E:,. ,y'').%' ,)'..; .y'' ftt. ,:... jjj7td-jjj,1!2421,:'(E:,..ki:r t::' )8,<-..,s.. ..''.y)1I)k-jyjj)'. &.-,. llr.. 154, 421,: d:::: 1ki::1l; :1Er' .kE,: l.rlijjif..' :El2 #:#*'f,ty.-')j.A.yy .rt;...'.7...,tj;-rtr 1lr1. d:E:,h 11. 11. llrl. .'. (.).,j.y,tty. gllr)p %.. ( g .. . jyyy. 1l; 1 :55;4:2::4:::,,llr11l2417:1. tjjb1'3+. )'t*..111!111k!!1-.. k(!r7qp).... $44)14. 4::2ii.'.......11#)i)... #I!2l-y.. jiii 154) ll:ll.i.1Ir71L 1ki!: 4k12: 4:::,, 11.21. .... 1l: ql::)p );.r'#..'y)yy: . ...k' 1bd;i.lq- 4:y# . p.. L3ffL'. ;EE1g (1(2..-, j:''yt,.;Ayz,t., ii. 11.,1. #.:, ()iiL' - ii. . . ''. ; 'lri
Lkwise
qtjt
,$
'tihuoj.
,
,
y
q
yk
y
,
g
,,k;
.'.
:
..
...'...'j;.ji(...
:.i:t
,, '
.
..)
=
,
tyrtj., .;i
..
=
;
,ytl,ii1.
-.)..
.:.
E...
.
.
..
.
..
.,p,y,
..
..
.'
( jy)t,,.E.j
(II'E .E!' '.j2.t(). .:E. .. . :. ,,/'::t Eyyyhtttyyjt.ttjlyyyyyyyyjyyjrjyl,y.yyyt'jtb,t,,.,) ..
..
.
(..))t
.
.
,
,))
,yttytyytyy;y
,
,y(
.:/E.:,tytlr.-)()jtEj4jjkjp.. .Ei...E -jy ;;y.yjE(j..)y.....y''E,#.
''.-.';'.7ti1ir7.
'''1.j.
..
.
4:
..'
... . ..
,
jj riEjptl-l,
'-('
,-t.k.
....
yja
.ij'
.
-
.
..
...t..
.'....TiTi.tj;)))1t-? -yy.'':;)'r.y(..y)j-.,$y.,',.y,yr.;..,-yj:..,.-p.. .
-.7
-:
...:
-,
.
.
.
E : -:-
-:
. '..
.
-
.: !.
;.
-ki. . . . . . . . .. ..
-.
,.
-,.
E .?).!-
,. ..
. .
.. -
.
-
.
.
Mathematics
106
set of the fomf -1(A) where A is open in y. Since F defines the topology on x we know this set to be open, and the finite intersections of such sets form a base for X by assumption. Since e is 1-1 and e -1 a mapp ing, it will suffice to verify that their images under e are open in e(X). Noting that B is a set of the type shown in (6.22),eB) ay-1(A) r''hd(x), but since ly-1(A) is open, eB) is open in '(x) as required. . ,
=
The following is the (forus)mostimportant case of Urysohn
's
embedding theorem.
6.20 Theorem A second-countable F4-space (X,5) can be embedded in (0,1)*.(:I The proof requires
the sufticiency
part of the following lemma.
6.21 Lemma Let x e X, and 1et O X be any open set containing x. 1ff X is a regular space, there exists an open set U with e c 0. -t7
.x
Proof Let X be regular. If O is open nd x e 0, there exist disjoint open sets U OC CC CC 0. Since & by the disand C such that e U and c C, and hence CC closed, Cc is jo intness and we have V ? OC g O and sufficiency is proved. To necessity, is a closed set not containing x, and suppose x e U and Vc 0. prove OC where U and 'P are disjoint open sets. Hence X is regular. w c .x
,
-t7'
Since the space is T4 it is F? and Proof of 6.20 Let V be a countable base for hence reglar. For any x e X and B e V containing x, we have by 6.21 a U e such that G U c #, and also by detinition of a base 3 A e V with e A c U c V, and hence x e X'c V c B. (Xis the smallest closed set containing A, note.) Since V is countable, the collection of all such pairs, say .
'1
.z7
.z'
.4
=
((A,#): A e F, B e F;
X'c BI,
(6.23)
is countable, and so we can label its elements A,B4i (Aj,Sj), i = 1,2,... Every x e X lies in Aj fr some i e IN. Since the space is normal, we have by Urysohn's lemma a separating function : X F-+ (0,IJ for each element of .W,such that fixi) 1 and hB 0. For each a' e X and closed set C such that # C, choose (Aj,#Jsuch that x e Xic Bi c Cc, 0. These separating functions form a countable and then h.@) 1 and ficj class F, a subset of U(X). Since the space is F1, C can be a point (yl so that F separates points. And since the space is F3j and hence completely regular, is the weak topology induced by F, by 6.14. It follows by 6.19 that the evaluation map for F embeds X into (0,1)*.. =
=
=
.x
=
=
':
IJ= endowed with the metric px defined in (5.19)is a compact r0, metric space. lt follows that :(X), which is homeomorphic to X under the evalRecall that
uation mapping by F, is a totally bounded metric space. lt further follows that is metrizable, since, among other possibilities, it can be endowed with the and y of X is taken to be metric under which the distance between points have metrization theorem, 6.12. We Urysohn px(c@),c@)). therefore proved the The topology induced by this metric on g0,11= is the Tychonoff topology. A base for a point p e g0,1)*in this topology is provided by sets of the (p1,p2,...) (X,I)
.x
=
Topology
107
fonn Nlp-nkte.t
=
14./e l0,11*: I#/ -
qi I
i
< E,
=
(6.24)
1,...,/c),
for some finite k, and 0 < 6: < 1, which is the same as (6.20).The topology induced on X by the embedding is accordingly generated by the base sets Nx,.kte)
=
ly e
I @) fily)I <
X:
-
:,
i
=
1,...,/c1,
(6.25)
which can be recognized as finite intersections of the inverse images, under functions from F, of e-neighbourhoods of R; this is indeed the weak topology induced by F. This further serves to remind us of the close link between product topologies and weak topologies. Since metric spaces are F4, separable metric spaces can be embedded in (0,lj= by 6.20. In this case the motivation is not metrization, but usually compactitication that is, to show that separable spaces can be topologized as totally bounded spaces. Both metrization and compactification are techniques with important applications in the theory of weak convergence, which we study in Chapter 26. Although the following theorem is a straightforward corollary of 6.20, the result is of suftkient interest to deserve its own proof; the main interest is to see how in metric spaces there always exists a ready-made collection of functions to define the weak topology. -
A separable metric space (S,#) is homeomorphic to a subset of
6.22 Theorem
(0,11*.
Proof Let Jo J/(1 + #), which satisfies 0 f Jo f 1 and is equivalent to d, so that (X,A)is homeomorphic to (X,#). By separability there exists a countable set of points (zf,i e EN) which is dense in X. Let a countable family of functions be 1,2,..., and define an evaluation map : X F-, defined by fix) dx,zi), i =
=
=
x jj y g0,11
hx)
(%(mz1),J0(x,z2),...). where px(,#) that h is an embedding in (g0,1J=,px)
(6.26)
=
We show 7=1Ihk #kl llk. lf (x,,) is a squence in X converging to x, then for each k, dolxntz 4(x,zk)k Accordingly, V k,z H N 2 1 such that xn e Nxik,e) for a11n k N, poohxnlvhxjj --> 0, and h is continuous at x. On th other hand, if xn zz...hx, there exists : > 0 such that Y N k 1, djxntx) 2 6: for some n k N. Since (z:) is dense in X, there is and J0(mzk) < doxvz I 2 so that 1doxnnz a k for which doxnnz k 1: and hence =
-
--
Y
px((x,,),(x)) 2
/,
/2
1+1
-
(6.27)
k
Since this holds for some n k N for every N k 1, it holds for infinltely many n, and hxn) zz-y (x). We have therefore shown that hxn) (x) if and only (Jxn --).x. This is the property of a 1-1 contingous function with cntinuous inverse. . --
But note too the alternative approach of transforming th distance functions into ting functions as in (6t10),@nd applying6.20.
separa
11 PROBABILITY
7 Probability Spaces
Probability Measures A random experiment is an action or observation whose outcome is uncertain in advance of its occurrence. Tosses of a coin, spins of a roulette wheel, and observations of the price of a stock are familiar examples. A probability space, the triple (D,1,#), is to be thought of as a mathematical model of a random experiment. f is the sample space, the set of all the possible outcomes of the The collection 5 experiment, called the random elements, individually denoted of random events is a c-field of subsets of f1, the event A e 5 being said to have occurred if the outcome of the experiment is an element of A. A measure # is assigned to the elements of 5, PA) being the probability of A. Fonually, we have the following. .
7.1 Definition A probability measure (p.m.)on a measurable space (f1,1)is a set function #: 5 F--> (0,1j satisfying the axioms of probability (a) #(A) k 0, for a11:4 e T. 1. (b) #(D) (c) Countable additivity: for a disjoint collection tz4j e @,j e IN) =
,
#
Uz4yTPAj).
n
=
Under the frequentist intepretation of probability, #(A) is the limiting case of theproportion of a long run of repeated experimnts in which the outcome is in 4. Alternatively, probability may be viewed as a subjective notion with #(A) said to represent n observer's degree of belief that A will occur. For present purposes, the intrpretation given to the probabilities has no relevance. The theory stands or falls by its mathematical consisteny alone, although it is then up to us to decidewhether the results acord with ur intuition and are useful in the analysis of real-world problems. Additional properties of P follow i om the axioms.
7.2 Theorem lf A, #, and (, j e Eq) am afbitrary T-sets, then (i) #(X) f 1. #(AC) 1 #(A). (ii) t E 0. (iii) PZ) ytiitity). (iv) A c # = #(A) f #() tpij lty, yy, y Ir$r7.y7,#kk. yyE ');.(.y.-- ..ty .... ... ...;. .. 4(:(y.-t,d$p1k-2. .)-.(21,q.y.. .. 44:4)..j!!E111, L-i4qii.. jit' ..ty..-. (v)(A w B) = pA) + .ji!!:::,' y )j)t yq j y,, . . . yy ,y jg tr j, ju j yy, jyky y kjy (vi) PojAj) f L)PAj) (ctllitAb. tyt,tt j j,( y y rytyrjjtyry =
-
'
=
i
,
-t.(..-...y...ty-.t
(2)jI.
..))
.jj!.::;i'
-. .
..-...k.
....
..
-'
...tE.y..-
...
.
-
.
.
,jj
.
$jjy
,tjjyyytyyyylyyy,yyqyjyj.ytlyyttyyyyyy,ly
...),
.. .
..
y...
,y
,
y,y ,
..
112
Probability
(vii) Aj
'r'
t, A
A or Aj
PAj) .-A #(A)
(continuity).cl
Most of these are properties of measures in general. The complementation property (ii) is special to #, although an analogous condition holds for any finite measure, with #(X replacing 1 in the formula. (iii) confinns P is a measure, on the definition. Proof Applying 7.1(a), P(A)
+
(b), and (c), PAc)
PA t.JAc)
=
#(f)
=
1,
=
from which follow (i) and (ii), and also (iii) on setting A by 3.3, and (vii) by 3.4. w
=
(.
(iv)-(vi)follow
To create a prbability space, probabilities are assigned to a basic class of according C, events to a hypothesis about the mechanisms underlying the random outcome. For example, in coin or die tossing experiments we have the usual hypothesis of a fair coin or die, and hence of equally likely outcomes. Then, provided C is rich enough to be a determining class for the space, (f,@,#) exists by 3.8
(extension theorem) where ?F = c(C).
7.3 Example
Let Sp,1)
=
(# fn g0,1), #
e f), where T is the Borel field on R.
Then ((0,1), fm,1J, m), where m is Lebesgue measure, is a probability space, 1. The random elements of this space are real numbers between 0 since ,7:(:0,1)) and 1, an a drawing from the distribution is called a random variable. It is said to be distributed rfnt/r/zl/y on the unit interval. The inclusion or exclusion of the endpoints is optional, remembering that /1((0, 1J) r?2440,1)) = 1. n =
=
The atoms of a p.m. are the outcomes (singletonsets of ) that have positive probability. The following is true for tinite measures generally but has special importance in the theory of distributions.
7.4 Theorem The atoms of a p.m. are at most countable. Proof Let l be an atom satisfying P((1 J) #(( J)for al1 (0 6 f1, 1et (t'/2 satisfy #(f tl)2)) k #4(1) for all e f(t)1 ), and so forth, to generate a sequence with -
#(l1l)
2 #(tt%l)
2 #(t2l)
...
(7.3)
The partial sums Z7=1#(fj)) form a monotone sequence which cannot exceed #(D) 1, and therefore converges by 2.11, implying by 2.25 that limn-hxftf.t)a 1) 0. All points with positive probability are therefore in the countable set f(, i e Nl . =
=
.
Suppose a random experimentrepresented by the space (fk9,#) is modified so as to confine the possible outcomes to a subset of the sample space, say A c f1. For example, suppose we switch from playing roulette with a wheel having a zero slot to one without. The restricted probability space is derived as follows. Let 5% denote the collection fEcj A, E e @) 5%is a c-tield (compare1.23) called 5h and #(m/#(A) for E e Fa, Ph can be is called the trace of % on A. Defining #a() (A,Fa,#A) is called the trace of (f1,F,#) on A. triple verified to be a p.m. .
=
'lhe
Probability Spaces This is similar to the restriction of a measure space to a subspace, except that the measure is renormalized so that it remains a p.m. In everyday language, we are inclined to say that events may be or certain' lf such events are none the less elements of @, and hence technically by random, we convey the idea that they will occur or not occur with here assigning them probabilities of zero or one. The usage of the tenn marks is deliberately loose, as the quotation suggest. To say an event cannot it has probability different from saying it cannot occur because is zero occur because the outcomes it contains are not elements of f1. Similarly, to say an event has probability 1 is different from saying it is the event f1. In technical discussion we therefore make the nice distinction between sure, which means the latter, and almost sure, which means the former. An event E is said to occur almostsurely fl E has prob(a.s.), or equivalently, with probability one (w.p.1) if M abilitymeasurezero.This terminologyis synonymouswithlfv/l/eveqwercta.e.) in the measure-theoretic context. When there is ambiguity about the p.m. being considered, the notation may be uded. Aimpossible'
.
tcertainty' tcertain'
=
-
4a.s.(#j'
7.2 Conditional Probability A central issue of probability is the treatment of relationships. When random experiments generate a multi-dimensioned outcome (e.g.a poker deal generates several different hands) questions always arise about relationships between the different aspects of the experiment. The natural way to pose such questions is:
0. #(. IA) satisfies the event B as #(#IA) probability axioms as long as P does and #(A) > 0. In particular, #(A IA) 1, and P(Bc IA) 1 PB lA), since B /-74 and Bc DA are disjoint, and their union is 4. The space (A,@x,fk), the trace of the set A on (D,T,#), models the random experiment from the point of view of an observer who knows that (l) e A. Events A and # are said to be dependent when PB IA) # #(#). In certain respects the conditioning concept seems a little improper. A context in which the components of the random outcome are revealed sequentially to an observer might appear relevant only to a subjective interpretation of prpbability, and lead a sceptical reader to call the neutrality of the mathematical theory ipto question. We might also protest that a random event is random, and hak no business Gthe
=
=
=
-
Probability
114
defining a probability space. ln practice, the applications of conditional probability in limit theory are usually quite remote from any considerations of subjectivity, but there is a serious point here, which is the difficulty of constructing a rigorous theory once we depart from the restricted goal of predicting random outcomes a priori. The way we can overcome improprieties of this kind, and obtain a much more powerful theol'y into the bargain, is to condition on a class of events, a c-subfield of 9. Given an event B e F, let the set function F-> (0,11 /74#1 ): '
represent the contingent probability to be assigned to B after drawing an event A from N, where V i @.We can think of V'as an infonuation set in the sense that, for each A G V, an observer knows whether or not the outcome is in A. Since the elements of the domain are random events, we must think of #(# lN) as itself a random outcome (arandom variable, in the terminology of Chapter 8) derived from the restricted probability space (f1,V,#). We may think of this space as a model of the action of an observer possessing information V, who assigns the conditional probability #(# lA) to B when be observes the occurrence of A, viewed from the standpoint of another observer who has no prior infonnation. N is a c-tield, because if we know an outcome is in A we also know it is not inAc, and if we know whether or not it is in Aj for each j 1,2,3,..., we know whether or not it is in U#.j. The more sets there are in V the larger the volume of information, all the way from the trivial set 5 (f,0) (completeignorance, with 8#1 T) #(#) a.s.) to the set 5 itself, which orresponds to almost sure knowledge of the outcome. In the latter case, P@ 11) 1 a.s. if ( e B, and 0 otherwise. lf you know whether or not (J) G A for every A e @, you eftctively know (J). 8 =
=
=
=
7.3 Independence A pair of events A, B e 5 is said to be independent if #(A ch#) equivalently, if PB jA)
=
#(#).
=
#(A)#(#), or,
(7.4)
lf, in a collection of events C, (7.4)holds for every pair of distinct sets A and B from the cotlection, C is said to be pairwise indepenttent. In addition, C is said to be totally independent if (.D-,A)
#
-
J,'tA'
(7.5)
containing two or more events. This is a stronger condition for every subset # i thanpairwise independence. Suppose Cconsists of setsA, #, and C. Knowing thatS has occurred may not influence the probability we attach to A, and similarly for C) but thejoint occunrnce of B and C may none the less imply something about A. Pairwiseindependence implies thatfA fa B) =PIAIPCBI,PCA rn C) =#(A)#(C), and PB rn C' PB)PC4, but total independence would also require P(A f'n B rn C) = =
P(A)PB)PC).
Probability Spaces
115
Here are two useful facts about independent events. In each theorem let C be a totally independent collection, satisfying ptf)xs .gA) l7ze.y#(A) for each subset =
#' I C. 7.5 Theorem The collection C' which contains independent.
yt
and Ac for each A e C is totally
Proof It is sufficient to prove that the independence of A and B implies that of Ac and B, for B can denote any arbitrary intersection of sets from the collection AC. This is certainly true, since if and (7.5) will be satisfied, for either A or #(A)#(#), then #(A f''h B) =
PCAC
r'A #)
=
PAc ra #)
= #(#)
-
+
P(A fa #)
P(A4PB4
-
#(A)#(#)
PAPB4.
=
(7.6)
w
7.6 Theorem Let () be a countable disjoint collection, and let t e collections consisting of Bj and the sets of C be totally independent for each-j. Then, if B LjBj, the collection consisting of B and C is als independent.
=
Proof Let #' be any subset of C. Using the disjointness of the sr-ts of B, and
countable additivity, P
j
rn
)
xOs ,A
-
P
sA)
zOs ) (#?
(Uy Bj ra ,A xOs XyP -
=Xy
PCB?P
7.4 Product Spaces
sA)
tj
=
Plhj
ra
,PA4.
.
and independence arise when multiple random eyperiments run in parallel, and product spaces play a natural role in the analysis of these issues. Let (f x E, 5 @ V, #) be a probability space where 5 (*)V is the
Questionsof dependence
c-fieldgenerated by the measurable rectangles of (-1x E, and PL x E) 1. The randomoutcome is a pair (,4). This is no more than a case of the general theory of j7.1 (wherethe nature of (l) is unspecified) except that it becomes possible to askquestions about the part of the outcome represented by ) or ( alone. #04/7 =
=
#(Fx E) for F e F, and #E(G) Pl x G4 for G e N, are called the marginal probabilities.(1,1,#0) and (E,T,#E) are probability spaces representing an incompletelyobserved random experiment, with ) or k, respectively, being the only things observed in an experiment generating (,(). On the other hand, suppose we observe ( and subsequently consider the experiment'of observing (t). Knowingtmeans thatforeachf x Gweknowwhetherornot (,() is in it. The conditional probabilities generated by tlzis two-stage experimentcan be wr itten by a slight abuse of notation as #(Fl V), although strictly Fx E, and the elemts of the speakingthe re 1evan t events are the cylinders uzrln enHzw+lxlv.lllv,qia 'n uzn n'zala' rmfiold nre 0./ G for G G W. Cnn ditinnine =
Probability
116
#(Fx E lfl x T). ln this context, product measure assumes a special role as the model of independence. In (f x E, 5 (& V, #), the coordinate spaces are said to be independent when PFx
G)
=
#0(#)#s(G)
for each F e 5 and G e T.
Unity of the notation is preserved since Fx G (Fx E) fa (( x G). We can also write #(Fx EIf x G) /704/-), or with a further slight abuse of notation #(FIG) #n(/'), for any pair Fq 5 and G e N. Independence means that knowing ( does not affect the probabilities assigned to sets of ;. Since the measurable rectangles are a determining class for the space, the p.m. P is entirely detennined by the =
=
marginal measures.
=
8 Random Variables
8. 1 Measures on the Line a probability space. A real random variable (r.v.) is an 5lBf'unction X: ( F.-: R 9 That is to say, aY4)) induces an inverse mapping measurable from B to 5 such that X -1(#) l 5 for every B e B where B is the linear Borel field. The term T-measurable r.v.' may be used when the role of B is understood. The symbol g. will be generally used to denote a p.m. on the line, reserving P for the p.m. on the underlying space. Random variables therefore live in the space (R,S,g,), where g. is the derived measure such that g(#) PX -1(#)) = #( X e B) The term distribution is synonymous with measure in this context. The properties of nv.s are special cases of the results in Chapter 3; in particular, the contents of j3.6 should be reviewed = in conjunction with this chapter. If g: R F-> R is-1a Borel function, then gox is also a r.v., having derived p.m. g,: according to 3.21. p(A)) If there is a set S e B having the property g(& 1, the trace of (R,S,g) on S is equivalent to the original space in the sense that the same measure is assigned to B and to B r''hS, for each # e S. Which space to work with is basically a matter of technical convenience. If X is a r.v., it may be more satisfactory to say that the Borel function X 2 is a r v. distributed on R+ than that it is distributed on R but takes values in R+ almost surely. One could substitute for (M,S,g,) the extended space (F,S,g) (see 1.22), but note that assigning a positive probability to infinity does not lead to meaningf'ul results. Random variables must be finite with probability 1. Thus (R,f,g), the trace of (F,S,g) on R, is equivalent to it for nearly a1l purposes. However, while it is always finite a.s., a r.v. is not necessarily bounded a.s.; there may exist no constant # such that 1X(l I f B for a1l (l) e C, with #(D C4 0. The essential supremum of X is
Let
(f1,1,#) be
.
,
=
.
=
,
.
=
-
ess sup X
=
inf (x: #41XI > x)
and this may be either a finite number, or
0),
=
+x.
8.2 Distribution Functions The cumulative distribution
@.b.f.)Of
f'Ifrlcft??z
X is the function F:
where
2- F-> g0,1J,
< xj, x s F. (8.2) l ( ( fty/, j'q..(( gn the values 0 and 1 to )( jj:t.:.' tk..j.,.((y. we take the domain to be R-since it' iq.(... (('..t'jr(t.t:.....t( .. .. . y)':)j)t..: .i-) ... (j).jL?, f;. ji,j.. . F(x)
=
g((-x, x1)
x
=
.
.
..
.
.
:
'
:
:.)'
.-
..'.''
....
.
..)
.... .
.
'
'
.t.
.
.:$j.i.
(Ej
.
.
....j
... y.. q .))(-.3E,l'( y j.y)y(y)jgE.t..-'(.:,.t.yy.. .t
...yr.j(,?:jyj.jy.
,,;.j..j;...(..j.
..
.
-(.y.'.,
.
. .
..
.
.
.
.
c
..
.
r)().... .. .. . y.....) ,.E.
. .
.
Probability
118
F(-x) and F(+x) respectively. No other values are possible so there is no contradiction in contining attention to just the points of R. To specify a distribution for X it is sufticient to assign a functional form for F; p, and F are equivalent representations of the distribution, each useful for different purposes. To represent g,(A) in terms of F for a set A much more complicated than an interval would be cumbersome, but on the other hand, the graph of Fis an appealing way to display the characteristics of the distribution. To see how probabilities are assigned to sets using F, start with the half-open interval (x,y) for x < y. This is the intersection of the half-lines (x,y) and (-x, (.x,+x).Let A (-=, x) and B = (-x,y), so that g(A) F@) and g(#) F(y); then -xzf
=
=
BIAC
g,(@,)/1) =
=
ra #)
= 1 (g(A) -
1 p,tA
=
+
-
1 g(#))
tJ#C)
g(#)
=
-
=
-
g(z4)
=
F@) F@), -
A and Bc being disjoint. The half-open intervals form a semi-ring (see1.18), and from the results of j3.2 the measure extends uniquely to the sets of f. a) for x c R As an example of the extension, we detennine g,((x)) PX will (compare 3.15). Putting a7 y in (8.3) not yield this result, since A cjAc = 0, not (xl We could obtain fxl as the intersection of (-x, x) and (-x, xlc, but then there is no obvious way to tind the probability for the open interval (-x, x) (-x, xl f.xl The solution to the problem is to consider the monotone sequence of half-lines (-x, x 1/n) for n e N. Since @ 1/p, (-x, aj, 1/n), according have 1/n,<) F(x) lln Fx ra g,(@ to we (-x, x = and where 1/n, x), Fx) F@-), Since T g,((x)) (8.3). (a7Je fx) 0':nltx F@-) is the left limit of F at x. Fx) exceeds Fx-j (i.e. F jumps) at the atoms of the distribution, points with g,((xJ)> 0. We can deduce by the same kind of F(y-4 Fx), g((x,y)) = F@-) F(x-), and that, reasoning that p,(@,y)) generally, measures of open intervals are the same as those of closed intervals unless the endpoints are atoms of the distribution. Certain characteristics imposed on the c.d.f. by its definition in tenns of a measure were implicit in the above conclusions. The next three theorems establish these properties. =
=
=
=
rx,+x)
.
=
-
.
.:71
-
=
-
=
=
-
-
-
-
-
-
.z:
=
-
-
1, 8.1 Theorem F is non-negative and non-decreasing, with F(-x) 0 and F(+x) and is increasing at e R iff every open neighbourhood of x has positive measure. =
=
.x
Proof These are all direct consequences of the definition. Non-negativity is from (8r2), and monotonicity from 7.2(iv). F is increasing at x if Fx + :) > F@ E) for each E > 0. To show the asserted sufficiency, we have for each such E, -
F@ + :)
-
F@ :) k F((x
+ 8)-)
-
For the necessity, suppose g,(5'(x,E))
g,(5'(x,e))= F(@ + e,)-) F@ -
-
=
E)
-
F(x
-
6,)
=
g,(5'@,y)).
(8.4)
0 and note that, by monotonicity of F, k Fx
+ :/2)
-
F@- :/2).
w
The collection of points on which F increases is known as the sunnort of
(8.5) 11
IfA
Random
Variables
119
complement in R, the largest set of zero measure, consists of points that must a11 liein open neighbourhoods of zero measure, and hencemustbe open.-rhe supportof g, is accordingly a closed set. F is right-continuous
8.2 Theorem Proof For x
l
everywhere.
R and n 2 1, additivity of the p.m. implies
g((-x,
.v
+
1/nJ) g,((-x, =
x1)
+
g,(@,x+ 1/n1).
(8.6)
g,(4-x, x+ 1/aJ)1, g.((-x, x1) by continuity of the measure, and hence limn-yxjtttx, + 1/n1) 0. lt follows that for er > 0 there exists Ne such that g,(@,x + 1/nJ) < E, and, accordingly, As n
.-.+
x,
.v
P,((-=,
=
.Y1)
f t((-=,
for n k Ne. Hence F(x+)
=
x+ 1/n1)
<
p,tt-x, x1) +
(8.7)
E,
F(x), proving the theorem since x was arbitrary.
.
If F@) had been defined as g,((-x, x)), similar arguments would show that it was left ontinuous in that case.
'W
Fig. 8.1
8.3 Theorem F has the decomposition F(x)
whereF'@) jumps, and
=
F'x)
+
F'x)
(8.8)
is a right-continuous step function with at most a countable number of r'(x) is everywhere continuous.
Proof By 7.4, the jump points of F are at most countable. Letting
denote these
points,
f-x1,x2, ...)
Probability
120 F'@)
=
X xisx
Fxi,
Fxi-j)
-
(8.9)
is a step function with jumps at the points xi, and F'x)
F(xi-)
=
Flxi) at each xi and is continuous everywhere.
Fx)
=
-
F'x)
has
*
Fig. 8. 1 illustrates the decomposition. This is not the only decomposition of F. The Lebesgue decomposition of g with respect to Lebesgue measure on R (see4.28) is g = p,1+ g.2where g,l is singular with respect to m (is positive only on a set of Lebesgue measure 0) and g2 is absolutely continuous with respect to Lebesgue measure. Recall that g2(A) = fxjdxo A e B where Tis the associated Radon-Nikodym derivative (density function). If we decompose F in the same way, such that Ff(x) w((-x,x1) for i = JQ.f(()J), implying that f@) dFz /#( I(=x.This = 1 and 2, we may write Flx) must hold for almost all x (Lebesgue measure), and we call F2 an absolutely continuous f'unction,meaning it is differentiable almost everywhere on its domain. F' S F1 since F1 may increase on a set of Lebesgue measure 0, and such sets can be uncountable, and hence larger than the set of atoms. It is customary to summarize these relations by decomposing F' into two additive components, the absolutely continuous part &, and a component F3 = F'- F2 which is continuous and also singular, constantexcept on a setof zero Lebesgue measure. This componentcan in most cases be neglected. The collection of half-lines with rational endpoints generates B (1.21) and should be a determining class for measures on @,f). The following theorem establishes the fact that a c.d.f. defined on a dense subset of R is a unique representation of g,. ,
=
=
8.4 Theorem Let z be a tinite measure on (R,f) and D a dense subset of R. The function G defined by 81)
Gx)
=
IX(-=,-Y1),
e D
.<
=
x e R D
F(x+),
-
(8.10)
is identical with F. Proof By definition, R q X'and the points of R D are a11closure points of D. For each x e R, not excluding points in R D, there is a sequence of points in D converging to x (e.g. choose a point from Sx,lln) r'AD for n g N). Since F is F(x+) for each x e R -D. w right-continuous everywhere on R, p,((-x,x)) -
-
=
Finally, we show that every F corresponds
to some g,, as well as every g to an F.
8.5 Theorem Let F: V --A E0,1) be a non-negative, non-decreasing, right=0 and F(+x) 1. There exists a uniquep.m. g on continuous function, with F(-x) (R,S) sqch that Fx) g((-x,x1) for a11a7 G R. u =
=
Right continuity, as noted above, correspopds to the convention of defining F by (8.2). If instead we defined F@) = g((-x,x)), a left-continuous non-decreasing F
Random Variables
121
would represent a p.m.
#: (0,11F- 2', defined
Proof Consider the function (4If)
=
inf (x: u < F(x) )
by
(8. 11)
.
4)can be thought of as the inverse of F; (40) (41) +x, and since F is nonIight-continuous, decreasing and # is non-decreasing and left-continuous; # is Borel-measurable by According to 3.21, we may define a meas3.32(ii). therefore ?n4 -1 (#) for each B e f, where m is Lebesgue measure on theBorel ure on (R,f) by of sets (0,11. ln particular, consider the class C of the half-open intervals (J,:1 for a11a,b e R with a < b. This is a semi-ring by 1.18, and c(C) B by 1.21. Note that t)-1((J, 4) (/g:inf (x: u K F(x)) e (tz,h1 ) (F((7),Fblj. (8.12) =
-x,
=
=
=
=
For each of these sets define the measure P,((J,#1)
=
/19
-1((J, :j))
=
Fb)
-
(8.13)
Fa).
The fact that this is a measure follows from the argument of the preceding paragraph. C is a determining class for (R,O), and the measure has an extension by 3.8. It is a p.m. since g.(R) 1, and is unique by 3.13. x =
The neat construction used in this proof has othef applications in the theory of random variables, and will reappear in more elaborate form in 922.2.The g'raph of () is found by rotating and reflecting the graph of F, sketched in Fig. 8.2; to see the fonner with the usual coordinates, turn the page on its side and view in a
mirror.
Fig. 8.2 If F has a discontinuity at x, then 4) x on the interval (F@-), F(x)1, and -1(lxl) = (F(x-), F(x)j. Thus, g,((x)) -((F(x-), F@)1) F(x) F(x-), as 41 required.On the other hand, if an interval (J,!4 has measure 0 under F, F is =
=
=
-
Probability
122
F(b) = c (say).4) constant on this interval and # has a discontinuity at Fa) -1 takes the value a at this point, by left continuity. Note that () (c) = (J,>1, so that g,((J,:J) mc) = 0, as require. =
=
8.3 Examples Most of the distributions met with in practice are either discrete or continuous. A discrete distribution assigns zero probability to a11 but a countable set of points, with F' = 0 in te decomposition of 8.3.
8.6 Example The Bemoulli (orbinal'y) nv. takes values 1 and 0 with fixed probaThink of it as a mapping from any probability space containbilities p and 1 elements. and tFailure'
isuccess'
;
1
8.7 Example The binomial distribution with parameters n and p (denotedBn,pl) is the disibution of the number of ls obtained in n independent drawings from the Bernoulli distribution, having the probability function P(X
=
x)
4
=
x
p
x
(1
n-x
-p)
x
,
=
8.8 Example The limiting case of (8.14) with p distribution, having probability function P(X
=
x)
-w
cp
1
0,...,p.
hln. as
=
(8.14)
Ia
n
-..-y
x,
is the Poisson
-h,
.
X'.
x ,
x
=
() j ,
,
z
(8.15)
,...
This is a discrete distribution with a countably infinite set of outcomes.
D
In a continuous distribution, F is absolutely continuous with FI = 0 in the Lebesgue decomposition of the c.d.f. The derivative f #F/#x exists a.e.(>1 on R and is called the probability densit.v function(p.d.f.)of th p.m. According to the Radon-Nikodym theorem, the p.d.f. has the property that for each E e f, =
g,(f)
=
jyflvdx.
,
(S' 16)
8.9 Example For the unkf'orm distribution on g0,11 (see 7.3), 0, x F@)
<
x, 0 f
=
1, x
>
0 .x
S 1.
(8.17)
1
The p.d.f is constant at 1 on the interval, but is undefined at 0 and 1.
E1
8.10 Example The standard normal or Gaussian distribution has p.d.f. f(x)
=
1
/-2a
-x2/2
e
-x
'
<
x
< +.=
,
(8.18)
Random Variables
whose graph is the well-known
123
bell-shaped curve with mode at zero. n
8.11 Example The Cauchy distribution has p.d.f.
fxq
1
=
<
-cx,
,
+x2)
zr(1
which, like the Gaussian, is symmetric
x
(8.19)
< +x,
with mode at 0. n
When it exists, the p.d.f. is the usual means of characterizing a distribution. A particularly useful trick is to be able to derive the distribution of gzk' from that of X, when g is a function of a suitable type.
8.12 Theorem Let #: S F--> 1- be a 1-1 function onto 7, where S and 1- are open subsets of R, and let h g -1 be con tinuously differentiable with dhldy # 0 for all y e 7. lf X is continuously distributed with p.d.f. fx, and F #(.X), then F is continuously distributed with p.d.f. =
=
),I
.fx(,-1(,))
s(,) -
.
(8.20)
u
The proof is an easy exercise in differential calculus. This result illustl-ates 3.21, but in most other cases it is a great deal harder than this to derive a closed form for a transformed distribution.
8.13 Example Generalize the uniform distribution (8.9)from (0,1) to an arbitrary interval gtz,:1.The transformation is linear, y so that fyy)
=
b a) -1 on
a
=
(J,:),
-
F@)
b
+
by
(.y
=
-
(8.21)
Jlx,
-
(8.20).The
allb
c.d.f. is d e jjxd
on
a).
-
gc,)j by (8.22)
Membership of the uniform family is denoted by X
-
&(c,:1. n
8.14 Exapple Linear transformations of the standard Gaussian r.v., X
=
jt + cZ,
c
>
0,
(8.23)
generate the Gaussian family of distributions, with p.d.f.s
1 e fx'' g.,0-) /--2ac
-tx-jtlz/acz
'-=
=
,
<
.7
< +=
.
(:I
(8.24)
have better-known designatiops as The location parameter ytand scaleparameter and 9.7 9.4 below. distribution; Membership of the Gaussian of see moments the 2 family is denoted by X #(g,,c ). -
A family of Cauchy distributions is generated from the standard Cauchy by linear transformations X v + Z, > 0. ne family of p.d.f.s with locationparameter v and scale parnmeter take the form
8.15 Example
=
Probability
k=N
J@;v,)
1
1
=
k''g1+
r(x -
<
-=
,
v)/) z
< +x.
.z7
(8.25)
n
8.16 Example Consider the square of a standard Gaussian nv. wit p.= 0 and c = 1. Since the transfonnation is not monotone we cannot use 8.12 to detennine the density, density, strictly speaking. But consider the Khalf-normal'
zlt&l =
2J(ff), u
k 0
0,
<
u
(8.26)
0
where f is given by (8.18).This is the p.d.f. of the absolute value of a Gaussian variable. The transformation g Iu I) = u 2 is 1-1 so the p.d.f. of Z2 is ,
1 e fz24&) W2V
-1/2
-un
u
=
applying(8.20).This is the ch-squared or z241).It is a member (withp (x =
G(?z; ',pj W
herej-+4
=
f!
() < u
,
=
and 1R?zl (n llF(n
-
(8.27)
,
-1 -ea
((xI/)P
vpe
-
.
distribution with one degree of freedom, ) of the gnmma family,
,
0
<
u
< x,
is the gammafunction, J7(P-1e-Q(
=
=
<
.
a
>
0, p
>
0,
(8.28)
having the properties F()
=
W2'V,
1). c!
8.4 Multivariate Distributions In Euclidean k-space R k the k-dimensional Borel field Tk is c(S$ where Rk denotes the measurable rectangles of Rk th e se ts of the form #1 x B1 x x Bk where Sf e B for f = 1,...,k. In a space (D,@,#), a random vector (msXi,...,.V)' = X is a measurable mapping X1 fl Rk ,
,
,
...
.-+
.
lf t is the derived measure such that g,(A) = PE) for A G T k an d E s 5 the mu ltivariate c.d.f., F: Fk (0,1), is defined for (x1,...,x' = x by ,
..-+
F@)
=
g,((-=,x11 x
...
x (-x, aJ).
(8.29)
The extension proceeds much like the scalar case.
8.17 Example Consider the random pair (X,D.Let F(x,y) = g((-x, XJx (-x, y1). The measure of the half-open rectangle (x,x+ Ax1x (z,y +Ay) is M7x, y)
=
F@ + Ax, y + Ay)
-
Fx
+
Ax,y)
-
F@, y + Ay) + Fx, y4.
(8.30)
To show this, consider the four disjoint sets of R 2 illustrated in Fig. 8.3:
Random Variables
125
:4
=
@,x + AXJx @,y + AyJ,
B
=
(-x, x) x (.y,y +Ay),
C
=
(x,x+AXJx (-x, yq,
D
=
(-x,x) x (-x,yj.
A is the set whose probability is sought. Since #(A wB tp Ck.)D) Fx + Ax,y), and #(D) Fx+ Ay + Ay), #(# QJ D) = Fxby + Ay), PCCwD) result probability is immediate from axioms. the Fx, y), the n =
= =
Fig. 8.3 Extending the approach of 8.17 inductively, the measure of the k-dimensional , xi + AQ can be shown to be rectang le Xil@f, AF(I1,...,
A'k)
=
77(:f..6/),
(8.31)
j
where the sum on the right has 2k terms, an d the Fj are the values of F at each of the verices of the k-dimensional rectangle extending from @1,....q)'with sides 1,...,k. The sign pattern depends on k', if k is odd, the Fj of length Mj, i having as arguments even numbers of upper vertices (pointsof the form xi + A'&)) take negative signs, and the others positive; wilile if k is even, the Fj with odd numbers of upper vertices as arguments are negative. Generalizing the monotonicbe ity of the univmiate c.d.f., F must satisfy the condition that AF@1,..., (R)+ non-negative for every chice of (x1,..., ' e R an d (A.x1,..., Axk)' s Applying 3.19 inductively shows that the class of half-open rectangles is a semi-ring, so that the measure defined by F extends to the sets of T k. hence (R Bk is a probability space derived from L1,5,P). If the distribution is continuous with p.d.f. flx), Fubini's iheoremgives =
.q)
.q)
.
-dimensional
,
,g)
,
Fx)
.f((1,...,(z)t:61
.dlk
=
..
(-x,xl1x,.,x(-x,xk)
=
xl
xk ...
J(k1
.,4:2)t2)1
...#(k.
,..
Theorem 8.12 has the following generalization. A dteomorphism
(8.32)
(also,coor-
Probability
126
dinate transformationj is a function g: S F- -T(S and 7 open subsets of Rk) w lli c h # 0 for all x e S, is 1-1 onto and continuously differentiable with etglx') whose element glx' Jacobian matrix is is the gllxi for i,j where (f,.j)th 1,...,1. The inverse of a diffeomorphism is also continuously differentiable. =
8.18 Theorem If F
where g is a diffeomorphism,
g
=
the p.d.f. of F is
h%l flg-1(.:))1yj
(g ag)
=
whereJ
detl:
=
-1
)j
y
'j
.
u
.
This is a standard result in the theory of multiple Lebesgue integrals Apostol 1974: 15.10-15.12).
8.19 Example Letting
denote the standard Gaussian p.d.f.
.f
k
lzi) 1-1
9(z) =
(zzo-Wexpf
=
(see8.10),
(see e.g. consider
-z1(z'z)).
(8.34)
i=1
This is a k-dimensional p.d.f., and the corresponding random vector (z1,...,Zk)', is the standard Gaussian vector. The affine transformation X
AZ+ !z,
=
Z
=
(8.35)
whereA (kx k nonsingular) and M kx 1) are constants, is 1-1 continuous with inversez A -1(X p.) having J IA-1 l 1/lA I Detine E 4A' such that (A -1) x-1 and j j4 j j jEI the positive square root being ( nderstood.Applying 8.18 produces =
=
ix,
,
=
,
-
jx I
lnl-m 14-1I =
(21)-1/2 jx j
,
1
9(A -1@ p.))
j
=
.
-1/2
-1
=
=
=
=
-
,4:4..)-1
.:4.-1
j
expf
-12
-yx
-
Jz)'(A-l)'A-1(x
-
Izlj.
.s;,x-1(.x
t
exp
-j(x .s)
j.
(8.36)
This is the multinormal p.d.f., depending on parametets p. and X. Every such distribution is generated by an affine transform applied to Z. Membership of the multinormal family is denoted X N#,X). n -
8.5 lndependent Random Variables Suppose that, out of a pair of r.v.s (.X,1')on @2T1 we are interested only in situation of interest X. In this the events are the cylinder sets in predicting R 2 having the form B xR, B e S. The marginal distribution of X is detined by (R,S,g,x) where ,g),
,
,
p,x(X)
=
t(X X R)
for W e T. The associated marginal c.d.f. is Fx
(8.37)
F@? +x). The notion of independence defined in 97.4specializes in the following way.xand =
Rattdom Variables F are called independent r.v.s iff pt(XX #)
>x(A)g.l'(S)
=
(8.38)
for al1 pairs of events A,B e f, where g.x is detined by (8.37)and g.y is analogous. Equivalently, g, is the product measure generated by g,x and yty. 8.20 Theorem X and J' are independent iff for each x,y Fxnyq
G
R
(8.39)
Fx@)Fy(X.
=
lf the distribution is continuous the p.d.f. factorizes as x,L
fxxjhb.
=
(8.40)
Proof Obviously, (8.39)is true only if g, satisties (8.38).The problem is to show that the former condition is also suftkient. Consider the half-open rectangles,
C
=
((tz,:)x (c,#1,a.c
e
R-,b,d
e R ).
lf ahd only if (8.39)holds, p,((J,/8
X (c,#1)=
= =
Fb,dl
F(:,c)
-
Fxbq
-
-
Fa,dl +Fa,c4
Fxabttdj
-
Fv)
(8.41)
g,x((J,:1)p,i'((c,tf1),
where the first equality is by 8.17 C is a determining class for (R2 S2) and g is detined by the extension of the measure satisfying (8.41)for ordinary rectangles or, equivalently, satisfying (8.39).The Extension Theorem (uniqueness partl shows that this is identical with the product measure satisfying (8.38).The extension to p.d.f.s follows directly from the detinition. . ,
.
,
With more than two variables there are alternative independence concepts k (compare j7.3). Variables m,...,Xk distributed on the space (R- pk are sa itj to be totally independent if ,g.)
,
1
XAf
p,f=1
k
=
l-lp,xJAd)
(8.42)
f=1
for all k-tuples of events A1,...,Akl 0. By contrast, pairwise independence can hold between each pair Xi&. without implying total independence of the set. Another way to think of total independence is in terms of a partitioning of a vector X = X3....,X' into subvectors .Y1 (jx 1) and Xz k x 1) for 0 < j < k. Under total independence, the measure of X is always expressible as the product measure of the two subvectors, under a11orderings and partitionings of the e lements. -j4
9 Expectations
9. 1 Averages and lntegrals or mean, of a r,v. X()l in a probability space
When it exists, the expectation, L1,5,P) is the integral EX4
Joxtovptl'
=
(9.1)
F(m measures the central tendency of the disibution of X. It is sometimes identified with the limiting value of the sample average of realized values xt drawn in n identical random experiments, xn
=
1
n
(9.2)
.t,
-
>1
as n becomes large. However, the validity of this hypothesis depends on the method of repeating the experiment. See Part IV for the details, but suffice it to say at this point that the equivalence certainly holds if F(A') exists and the random experiments are independent of one another. The connection is most evident for simple random variables. lf X = jxjjej where the ()) are a partition of f1, then by 4.4, osllj). /?(r) !F7 j cr
When the probabilities are interpreted as relative frequencies of the events Ej (: X'4(0) z)) in a large number of drawings from the disibution, (9.2) wlth large n should approximate (9.3). The values xj will appear in the sum in a proportion roughly equal to their probability of occurrence. F(m has a dual characterization, as an abstract integral on the parent probability space and as a Lebcsgue-stieltjes integral on the line, under the derived distribution. lt is equally conrct to write either (9.1) or =
=
+x
EX4
=
xdFxx).
(9.4)
Which of these representations is adopted is mainly a matter of convenience. lf 1x() is the indicator function of a set A G T, then
'(1,4m =
jgxldlk =
Jx(x).x#F@),
(9.5)
Expectations
129
where X(A) e B is the image of A under X. Here the abstract integral is obviously the more direct and simple representation, but by the same token, the Stieltjes fonn is the natural way to represent integration over a set in B.
If the distribution is discrete, X is a simple function and the formula in applies directly. Under the derived distribution,
(9.3)
'k-xj
F(m
where xj, j
=
(9.6)
p,(ta)l),
=
1,2,..., are the atoms of the distribution.
9.1 Example If X is a Bernoulli variable (8.6),EX) = 1.# + 0.(1
-#)
=
p. n
9.2 Example If X is Poisson (8.8), F(m
=
x
e-
ax x Wx
x
e-,E (x
=
vl
a
z-1
'A'
'A'
.
l ) 1.
-
vl
=
,.
u
(9.7)
For a, continuous distribution, the Lebesgue-stieltjes integral of a' coincides with the integral in ordinary Lebesgue measure of the function xfx).
9.3 Example For the uniform distribution on the interval gtz,)1(8.13), F(m
b
1
=
b a
xdx
=
1a(c +
-
a
b). n
(9.8)
9.4 Example For the Gaussian family (8.19), F(A3
1
=
-(x-jt)2/2o.2
#?
xe
Wfic
=
g.
(9.9)
This can be shown by integration by parts, but for a neater proof see 11.8.
I:a
ln a mixed continuous-discrete distzibution with atoms x1,x2,..., we can use the decomposition F F1 + F2 where F1(x) = Zouxgltlayl)and F1(x) is absolutely continuous with derivative hx). Then =
EX)
77.)
p,1(tA)l)
=
+
jxhxtdx.
(9.10)
The set of atoms has Lebesgue measure zero in R, so there is no need to exclude these from the integral on the right-hand side of (9.10). Some random variables do not have an expectation.
9.5 Example Recall the condition for integrability in (4.15),and note that for the Cauchy distribution (8.11),
1
+u
jxj
J-a(
-a
1+
x)2
dx
--
'>9
as
f'
-->
=.
El
(9.11)
130
Probability
9.2 Expectations of Functions of X lf X is a r.v. on the probability space @,f,g), and gl R F-- R is a Borel func-1), as note d in j8.1. This leads is a nv. on the space t'R,f tion, goX to the following dual characterization of the expectation of a function. =
'(m
,g.'
9.6 Theorem If g is a Borel function, F('(A'))
jglxtdbkxb /dB#-1t3')'
=
=
&n):R+
Proof Define a sequence of simple functions m i
z(a)(x)77I =
-
1
(9. 12) F-
R+ by
(9. 13)
1s,.@),
2n
1,...,-. Then, Z(n)(x) 1 and Bi = (2-&( 1), 2-'') for for x k 0, by arguments paralleling 3.28. According to 3.21, (R,S,pw -1 ) j s a g(#-1(#)) for B e S, and so by the monotone measurespace where g' -1(B)
where m
=
nln
't
+
.x
=
-
=
convergencetheorem,
-l(y) y;m
JZ(n)(A')t*'
1
-
.
f=1
zn
jydvg-t).
-1(sj)
.-.y
>'
(9. 14)
Consider first the case of non-negative g. Let 1s@) be the indicator of the set B G 0, and then if g is Borel, so is the composite function
1, glx)
(1so')@)
=
6
B
0, glx) e
B
=
1#-l(s)(x).
(9. 15)
Hence, consider the simple function (Z(,,)o#)@) =
m
77
i
1
-
2n
=1
m
(1sfo#)(x) =
By the same arguments as before, Znjog However, m
Eznjog)
=
77
j
j=1 m i
.
j
2n -
1
gn =X =1
'1'
77 f=I
i
-
2n
1
1#-1(sp@).
g, and Elznlog)
=
Eg)
=
JWg.
.j
g,(# Bi))
-
j
g,# Bi)
=
ztalllfp.'
-
1
(A),
and (9.12)follows from (9.14). To extend the result to general g, consider the non-negative g'b g separately. lt is immediate that maxf ',0J and g-
-->
(9.16)
(9. 17)
functions g'
=
Expectations E(Zn)og+4 Eznjog-) -
m
j.j
m
j-1
X =1
=
Eg-)
-
Eg),
=
(9. 18)
..j
>7 =1 2/
=
Eg')
--
of this limit separately.
so consider each component Eznlog-)
131
p,((#+) Bi)4 (9.19) jooycyg-jo),
-1
ptt# (wi ;;
2*
-..y
0
-1
-1
second equality holds because
where the g +) (Bi) g Bi) for i k g sjnce the 1 the tel'm elements of Bi are a11 positive for these cases, whereas for i disappears. Similarly, -Zn(x) 1 x for < 0, and =
=
.z'
?rj
-7r
-znjog-)
j.j
-
f=1
2/
m
i- 1
-:7
-
zn
j=1
.j
p.((#-)
Bi4)
0
j--ydv.g
-1
p,t, (#7))
--->
..j
(T),
(9.20)
1)1, and in this case the second equality holds where B; (-2-&, because (g- ) (#j) g (-Bi) j or j u z uence -2-''(f
=
-
-1
-1
=
.
Ezn,og
+) -
Ezn,og-)
j-
--
ydjw-lol,
(9.21)
and the theorem follows in view of (9.18).w The quantities F(X), for integer k 2 1, are called the moments of the distribution of X, and for k > 1 the central moments are defined by EX
-
Exlk
=
k
p
X
-
j
Eujh-kjxllk-i
(9.22)
-
A familiar case is the variance, Vartm E(X- F(m) 2 EX ) F(A') the usua j measure of dispersion about the mean. When the distribution is symmetric, with PX- F(A') e A) #(F(m X e 4) for each A e tB,the odd-order central moments are all zero. =
=
9.7 Exampl.
=
2
-
2
,
-
For the Gaussian case
(8.14),the
k'sk EX
-
g) k
=
2
kll
'
(kll)!
central
k even,
moments are
(9.23)
k odd. This formlamay be derived after some manipulation from equation (11.22) below. Vartm c 2 an d a11 the finite-order moments exist although the sequence increases monotoniclly. n =
,
Probability
132
The existence of a moment of given orderrequires the existence of the corresponding absolute moment. If FlXlP < =, for any real p > 0, X is sometimes said to belong to the set Lp (of functions Lebesgue-integrable to order p), or otherwise, to be Lp-bounded.
9.8 Example For X
N(0,c2), we have, by
-
FlXl
2(2a
=
(8.26),
y)-1/2j*.x e-xllloldx ()
(9.24)
l?2 = (2/1) c. n 'raking the corresponding root of the absolute moment is convenient forpumoses of comparison (see 9.23) and for X e Lp, the Lp-ttorm of X is defined as
(FI#I')'* 11111 p
(9.25)
=
The Gaussian disibution possesses a11finite-order moments according to (9.23), but its support is none the less the whole of R, and its p-norms are not unlrmly x, it coincides with the essential bounded. If IIXII, has a finite limit as p supremum of X, so that arandom variable belonging tofax is bounded almost surely. --
9.3 Theorems for the Probabilist's Toolbox The following inequalities for expected values are exploited in the proof of innumerable theorems in probability. The first is better known as Chebyshev inequalf@ for the special case p = 2. 's
9.9 Markov's
For er > 0 and # > 0,
inequality
#( l11 k Proof
tpp
l.Y1
Epjj
:)
=
Al
f:)
Fj aYjf
(9.26)
.
:
yvdFx) K
P
Jjzjzalxl PdFx) f E
1115.*
This inequality does not bind unless F1X1PIZP < 1, but it shows that if Fj -Yjr < x. The order of the tail probabilities converg to zero at the rate z-P as f: fv-boundedness measures the tendency of a distribution to generate outliers. The Markov inequality is a special case of (at least) two more general inequalities. =,
...-4
9.10 Corollary For any event A e F, E
Equivalently,
#((:
j
Arnt
IX()) I
Proof Obvious from 9.9.
lllkcl :1
dp fa
u
j IxI,g,. A
A) f F(1z
(9.27)
lXI#)/e'.
.
9.11 Corollary Let g: R F-> R be a function with the property that x ga) > 0, for a given constant a. Then gxj
a implies
Expectations
PX k a) <
Proof gajpx
k a4
&'(m) .
.
(9.28)
.
ga)
gajxyadFx)
=
133
jxyggxldFx)
S
K
EgT).
w
An increasing function has the requisite property for all a > 0. Let 1 ERbe any interval. A function (: I F-.: R is said to be convex on 1 if
9((1 )x +
,)?)
-
for a11 x, y e I and
.
(0,11. lf
e
(1 )(x)
f
+
-
-9
is convex
*(.y) on t # is
(9.29) on 1.
concave
9.12 Jensen's inequality lf a Borel function (1)is convex on an interval 1 containing the support of an integrable r.v. X, where ((m is also integrable,
(9.30)
9('(A')) f f@(A')). For a concave function the reverse inequality holds. n
The intuition here is easily grasped by thinking about a binary r.v. taking values with probability 1 A convex 4)is illustrated in xl with probability p and whereas F@(A')) p#(x1) + (1 This Fig. 9. 1. #(m px3 + (1 vertical chord the axis by point is mapped from F(m onto the joining and on 9, while 9(F(m) is mapped from the same point by (j itself. -p.
,:2
,-pjxz,
-p)9(x2).
=
=
-:2
.x1
Fig. 9.1 A proof of the inequality is obtaied interior of 1.
9.13 Lemma If and y e 1,
4)is convex
from the following lemma. Let 10 denote the
there exists a function A@) such that, for a11 e 10
X(-:)@
.x
-x4
f #(A') -
9(*.
(9.31)
Proof A convex function possesses right and left derivatives at all points of I0.
Probability
134
(9.29)implies
This follows because (x +
kh)
9(x)s
-
kh
(j(.7 +
for h h) h
-
>
0 that
#(x)
N s ((),jj.
'
(9.32)
+ 1/n) The sequence n e N ) is decreasing, and has a limit 9;(x). (T)(x)), In the case < 0 the inequality in (9.32)is reversed, showing the existence of 4,-/(x)as the limit of an increasing sequence. Note that ()-'@) < (J(x).Tnking the + h gives limit as 2. 4. 0 with > 0 tixed in (9.32)and y
tn(#tx
-
.z7
=
t)-'@)tJ'
S K(-:')(J'
-x4
-xt
S t)(A')
9(.1)
-
<
0 gives, for y
;(.z7)@ S #-'@)1
; th(A') t)@)
whereas the parallel argument with h -x)
-x)
F(m, y
=
X in
=
A('(m)(X-f(m) Taking expectations side has expectation
x,
(9.34)
-
lnequality (9.31)is therefore satisfied with (say)A@) Proof of 9.12 Set x
<
(9.33)
=
#+'(x)..
give
(9.31)to
#(m #(-)).
f
(9.35)
-
of both sides gives inequality
(9.30),since the
left-hand
zero. w
Next, we have an alternative approach to bounding tail probabilities which yields the Markov inequality as a corollary.
9.14 Theorem If X is a non-negative r.v. and r EX
rj-xr-zpx
r)
-
(9.36)
xvx.
>
0
Proof lntegration by parts gives, for some b joxrdFx)
0,
>
=
-
bryb) rjo
= r
-
>
0,
rjoxr-ypdx
Fpldx V--l(s(,) -
x 0
r-lp
dx.
(.x< x u
The theorem follows on letting b tend to infinity.
.
If the left-hand side of (9.37)diverges, so does the theorem is true whether or not EX) is finite.
9.15 Corollary
lf X is non-negative
J-
-xdF
-
spx k
(9.37)
light,
and in this sense the
and integrable,
o
+
j-px
> xvx.
-
(9.38)
Expectations Proof Apply 9.14 Fith r
135
1 to the r.v. 1(xk:)X. This gives
=
J- J-tlfxkslx = Jo 'dF
=
>
xvx
>
xltfx
o
$(1(xk:)X
+
j*PX
>
xldx
:
=
#(X k
E ejodx
jvP(X c
+
>
xjdx.
.
(9.39)
Not only does (9.38)give the Markov inequality on replacing non-negative X by IXI; for p > 0 and arbitral'y X, but the error in the Markov estimate of the tail probability is neatly quantified. Noting that P( IXI E) = 81.11#2 zP),
s ##( I-Yl k e)
j-
=
e
#
IXIJWF
= FIXl# where both the subtracted
j
-
j-
-
gP
P
IA-I'>
#
IXIPJF
px
-
0
I J
xldx
#(IXIP
eP
terms on the right-hand
>
xqdx,
(9.40)
side are non-negative.
9.4 Multivariate Distributions From one point of view, the integral of a function of two or more random variables presents no special problems. For example, if
gl R2
F-y
IR
is Borel-measurable, meaning in this case that g -1 B4 e Bl for every B e B, then #(#((l)),F()) is just a F/f-measurable r.v., and () =
F('(X,F))
=
Jjatldptl
(9.41)
is its expectation, which involves no new ideas apart from the particular way in which the t.v. hlk hppens to be defined. Alternatively,
the Lebesgue-stieltjes Eg-)
=
form is
jpzgx'ybdFx'L'
(9.42)
where dFxny) is to be thought of as the limiting case of &F@,yl defined in (8.30) as the rectangle tends the differential of area. When the distribution is with respect to continuous, the integral is the ordinary integral of '(x,y)J@,y) Lebesgue product measure. According to Fubini's theorem, it is equivalent to an iterated integral, and may be written +x
&'(X,l?))
=
+x
j-yqx,ytx'ybdydx. J.x
But caution must be exercised with formula
(9.43)
(9.42)because
this is not a double
Probability
136
integral in general. lt might seem more appropriate to write d lFx y) instead of JF(x,y), but except in the continuous case this would not be correct. The abstract notation of (9.41)is often preferable, because it avoids these ambiguities. In spite of these caveats, the expectation of a function of (say)X alone can in every case be constructed with respect to either the marginal distribution or te joint distribution. ,
EglxL
9.16 Theorem
jvz
#(X#F(mA')
=
jpgxjdklxj.
=
Proof Define a function +: R2 b..A (R
g
by setting g *@,y)= glx), a 11y e R :*-14#) is a cylinder in R2 with base :-1(#) e B for B e S, and g * i s Sz/s-measurable. For non-negative g, let .
m
* #(a) where m nl n+ 1 an d E i = = l-nli .j xR where Af (x: m
X
=
f=1
i
2
n
(9.44)
ls i
ln
=1
2-''(f 1) S #*(x,y) < 2-NI') e Bl Since Ei 1) K #(x) < 2-% l and tx(Af) = g,(&), -
,
-
.
=
,
j
-
j
.
(@y):
=
F(#*(,'))
j.
=
m
g,t&l
X
=
i
=1
j
-
2
n
p,x(A)
=
Elgnlj.
(9.45)
By the monotone convergence theorem the left and right-hand membrs of (9.45) *x,yldFx,y) and Egj xjdFxxj respectively. Extend converge to E(g.) non-negative general complete the from proof. * to g to =
=
The means and variances of X and F are the leading cases of this result. We also have cross moments, and in particular, the covariance of X and 1' is Cov(X,F)
=
E((X-
&m)(1'- F(F)))
=
Fubini's theorem suggests a characterization
ECXT F(mF(F).
(9.46)
-
of pairwise independence:
9.17 Theorem lf X and F are independent nv.s, Cov@(m,v(F)) of integrable Borel functions ( and
=
0 for al1 pairs
v.
Proof Fubini's theorem gives F(#(mv(F))
=
=
jp,
#@)v(y)#F@,y)
jpxtdlxjvlsbdFkl-bq
= &#(m)F(v(F)).
w
(9.47)
The condition is actually suffictent as well as necessary for independence, although this cannot be shown using the present approach; see 10-25 below.
Expectations
137
Extending from the bivariate to the general k-dimensional case adds nothing of substance to the above, and is mainly a matter of appropriate notation. lf X is a random k-vector, F(m
=
jxdFx)
(9.48)
denotes the k-vector of expectations, EXi) for 1,...,k. The variance of a scalar r.v. generalizes to the covariance matrix of a random vector. The k x k matrix, =
2 X1X2 X1
XX'
.
.
.
X3Xk
2 X2X1 X2
(9.49)
=
2
Xkxj
Xk
is called the outer product of #, and ECXX') is the k x k positive semi-detinite matl'ix whose elements are the expectations of the elements of XX'. The covariance matrix of X is Vartm
ELCX F(m(#
=
-
-
F(m)'q
=
ECXX') ExTEX4'.
(9.50)
-
is positive semi-detinite, generalizing the non-negative property of a scalar variance. lt is of full rank (notwithstanding that XX' has rank 1) unless an element of X is an exact linear function of the remainder. The following generalizes 4.7, the proof being essentially an exercise in interpreting the matl'ix formulae. Vartm
9.18 Theorem lf F= BX+c where X is an k-vector with EX4 g and Vartm = E, and B and c are respectively an m x k constant matrix and a constant v-vector, then (i) ElY) B# +r. BSB'. a (ii) Vartp =
=
=
Note that it m
>
k Vartl'l is singular, having rank k.
9.19 Example If a random vector Z (Z1,...,Zk)' is standard Gaussian (8.19),it Ik. Applying 9.18 to 0 and EZZ') is easy to verify, applying 9.17, that Ezj = the transformation in (8.35)produces F(A') g and =
=
Vartm
=
EX
-
p)(#-
g4'
=
F(AZZ'A')
=
=
AF(ZZ')A'
=
M'
=
E. n
9.5 More Theorems for the Toolbox Thefollowingcollection of theorems, together with thelensen and Markovinequalities of j9.3, constitute the basic toolbox for the proof of results in proba-
Probability
bility. The student will tind that it will suffice to have his/her thumb in these pages to be able to follow a gratifyingly large number of the arguments to be encountered in subsequent chapters. 9.20 Cauchy-schwartz
inequality
2<
Ex
with equality
attained when F
EXl)Eyl)
(9.51)
,
cX, c a constant.
=
Proof By linearity of the integral, ELaX+ F)2)
a
=
lEXl)
+ E(Yl) ZXIEIXLI
+
for any constant a. (9.51)follows on setting a equality if and only if aX+ F 0. .
=
O
-EIXYIIECXl ), an d holds as an
=
The correlation
c/elcdaf,
rxy
=
1/2 accor dingjy Cov(X,F)/(Var(mVar(D)
the interval (-1,+11. The Cauchy-schwartz inequality is a special case (forp 9.21 Hlder's p/(p
=
-
=
2) of the following.
For any p 2 1,
inequalil
11-Y11,111'11z?.
FI-YFIK where q
jies in
,
1) if p > 1, and q
Proof The proof for the case p
>
if p
x
=
1 requires
=
(9.52)
1.
a small lemma.
9.22 Lemma For any pair of non-negative numbers a,b, ab S
tzp bq
-
P
+
(9.53)
-.
q
If either a or b are zero this is trivial. lf both are oositive, let s = etlq ay = & and t = q log b. Inverting these relations gives a es y rlogtz S/PTf# d an d (9 53) follows from the fact that ex is a convex function of x, noting 3Iq 1 1/p and applying (9.29).* Proof
=
,
,
.
=
Choose a
.
,
=
-
)XI/II-Yllp,b
=
F11t?.For lr1/11
FIXFI
XIp
F
q
= Eab)
these choices, Eap)
S 1/# +
$Iq
=
=
Ebq)
1.
=
1, and
(9.54)
S Fl Xl ess sup F, which holds For the case p = 1, the inequality reduces to FI-YFI since F < ess sup i' a.s., by definition. w
The
lder inequality spawns a range of useful corollaries and special cases, includingthe following.
9.23 Liapunov's
inequality
Proof Let Z F:zlXIF, F = 1, s
(norm inequality) =
r/p. Then,
If r > p > 0, then
(9.52)gives FIZFIf
jlxllr I1-YIlp.
or IIzII,II FIIs(,-1),
Expectations E ( 1X 1#) < E 111
pslls
=
rfr
I11
E
(9.55)
.
.
This result is also obtainable as a corollary of the Jensen inequality.
9.24 Corollary
For each A e
jg IXI'IdP Proof ln
and 1/p + jIq
.
1/p
Jx
p
l71 PdP
<
1,
=
Jx
1Iq
11'IqdP
X by Xlx and i' by i'1x. x
(9.52),replace
probabilistic in character.
Alternative variants of the result are not explicitly
9.25 Corollary n
Let x1,...,x,, and y1,...,yn be any sequences n
77 izz1
1/,
77 1 izzz
lxafl S
771 IyfI
Then
for 1/p + Iq
=
1. n
fzz
be Lebesgue-integrable
functions of a real
'''-(J-
1/,+
(j-
I?f)I,z,)
'-ftkbgtkbkdt s
of numbers.
1/<
n
1x/1J'
and gl
9.26 Corollary Let fl variable. Then
J-
.
I,,(,)l-?,)''
or
lq
-
1. a
Proofs are left as an exercise. The sequences in 9.25 and the functions in 9.26 can be either real or complex-valued (see jl 1.2).
9.27 Minkowski's Proof For r
=
inequality
For r k 1, IIX+ i'llr <
i'llr. + 11 IIXIIr
1 this follows direct from the triangle inequality,
IX+ i'I < I-YI+ I1'l on taking expectations. For r
>
(9.56)
,
1, note that r-1)
FIx+ FIr F(I#+ FII-Y+ FI S E IXI+ IFI)IX+ l'I ) r-1). r-1) + E Irl I x+ I'I = E IxIIX+ l'I =
r-1
inequality to the two right-hand-side
Applying the Hlder
'
E IX+
FI
<
r)1-1/r.
+ 11 X+ l'I FIIr)(FI (II#IIr
Cancelling A'IaY+FI and rearranging r
application to the sum of m variables, generalizes directly to K r
terms yields
(9.58)
gives the result. w
By recursive
Yxi f=1
(9.57)
77Ilmllr =1
for r k 1. For an intinite series pne can write
the Minkowski inequality
(9.59)
Probability
140
77m =1
IImlIr+77xi X i=?n+1 f=1
:
r
If
0 as IIZ7=,,,+1XjIIr --
m
it is permissable to conclude that
tx.
--
(9.60) r
o/
X
i= 1
xi s
f=1
r
IIx)Ilr,
(9.6 1)
not ruling out the possibility that the right-hand side is infinite. 9.28 Love's
cr inequality
For r Fq
e' where cr
=
Txi
>
r
m
s
f=1
1 when r S 1, and cr
0,
=
r, cr XA'1 Xi I
(9.62)
i=1
m r-l wj)en r
j
.
Proof This goes by proving the inequality m
r
m
<-
ai /=1
cr f=1
Iai Ir
(9.63)
for real numbers JI,...,tza, then substituting random variables and taking expectations. Since 1Z0,zzkai lr S (E't', Iai I)r, it will suffice to 1et the ai be non-nega1, tive. For the case 0 < r S 1, 0 f zi f 1 implies z; k zi and hence if Z'?, ctajj. XTt k 1. (9.63)follows on putting zi = ai /(Z0./ For r k 1, on the other hand, convexity implies directly that .1
xlzf
=
=lz(,
j -
m
X icz:l
r
ai
K
-
1
m
XX.
*
izz,1
9.29 Theorem If X, 1', and Z are non-negative r.v.s satisfying X f tz(F+Z) a.s. for a constant a > 0, then, for any constant M > 0, FtltxyvlA')
f
zJtftlf
y>M/2c)F)+F(1(z>ArJw)Z)).
(9.64)
Proof lf we can prove the almost sure inequality 1(x>M)X S 2J(1ty>&z/2u)F+1(z>M/2u)Z),a.s.,
(9.65)
the theorem will follow on taking expectations. 1(x>M)Xis the r.v. that is equal to X if X > M and 0 otherwise. If X K M (9.65)is immediate. At least one of the inequalities F k X/M, Z k Xlla must hold, and if X > M, (9.65) is no less obviously true. w
9.6 Random Variables Depending on a Parameter Let G(,0): f x 0- F-> R, O i R, denote a random function of a real variable 0, or n other words, a family of random variables indexed on points of the real line.
141
Expectations The following results, due to Cramr nated convergence theorem.
easy consequences of the domi-
(1946),are
9.30 Theorem Suppose that for each (l) e C, with #4C) 1, G4,0) is continuous at apoint %,and IG((t),0) I < F((t))for each 0 in an open neighbourhood No of % where =
F(i')
<
Then
x.
lim EG(44
(9.66)
F(G(0o)).
=
8-+%
Proof Passage to a limit 00 through a continuum of points in 0- as indicated in (9.66), is implied by the convergence of a countable sequence in 0- Let (0v,v e NJ be such a sequence, in No, converging to 0o. Putting Gv((t)) G(,0v) defines a countable sequence of r.v.s. and limsupvGvt) and liminfvGvt) are r.v.s by 3.26. By continuity, they are equal to each other and to G(,%) for (l) e Q in other G(%) a.s. The result follows from the dominated convergence words, G(0v) ,
.
=
-->
theorem.
.
9.31 Theorem 1f, for each
)
e C with #(C)
G(),%
+
h4
1, (JG/J0)((t)) exists at a point
=
G(,%)
-
n((o)
< for 0 f
< Jl1, where F(l'1)
KdoEa)
<
j
(x,
and
is independent of
l
ttjG()..)
-
(.,
I
e.
% and
),
then
(9.67)
.
Proof The argument goes like the preceding one, by considering a real sequence (v ) tending to zero through positive values and hence the sequence of r.v.s (SVJ where Hv gG(0o+/lv) G(%)1/v, whose limit H S(0()) exists by assumption. . =
=
-
The same sort of results hold for integrals. Fubini's Theorem provides the extension to general double integrals. The following result for Riemann integrals on intervals of the line is no more than a special case of Fubini, but it is useful to note the requisite assumptions in common notation with the above.
9.32 Theorem Suppose thatforeach (l) E C, withf'to a t'initeopen interval atb), and IG(,0) l < 1'2() for
=
1, G(,0) is continuous on 0 < b, where F(F2) < x.
q <
Then
s(js(e)ts)
J-s(c(e)):e -
lf J7'IG(,0) lJ0 < 1'3(t,))for (t) E C and Ek) and b +=. of a =
-x
(9.68)
.
-
<
cx),
(9.68)holds
for either or both
=
Proof For the case of t'initea and b, consider Hlsb,tl JJG(,0)#0. This has the I lG(),f) I < F2(), for each propertiesIH% I < b J)F2(t,)), and I dHld EH) exists and by /, each 9.31, for t e a, b4 Hence, =
-
.
=
Probability
l42
d EH) E(Gt))
is continuous
=
E
dH
=
dt
EGtj).
on a,b4 by the a.s continuity
a(,) -
jtEol4d
(9.69) of G and 9.30, and hence
-s(s(?))
(9.70)
J
is differentiable on (a,b4,and dhldt = 0 at each point by (9.69).But by defi0, and hence hb) 0 for e C, so that A(J) 0 which is equivnition, H,a) alent to (9.68). =
=
=
JX
G(,0)#0 exists and is Under the stated integrability condition on G(,0), = J!xG(t0,0)#0 is well defined and has an expectation for finite on C. Hence Htt) and and/or b = x. > th argument above goes through with a = all t e R, -X
-.x
10 Conditioning
10.1 Conditioning in Product Measures It is difficult to do justice to conditioning at an elementary level. Without resort to some measure-theoretic insights, one can get only so far with the theory before nznning into problems. There are none the less some relatively simple results which apply to a resicted (albeitimportant) class of distributions. We approach, and so demonstrate the diftiintroduce the topic by way of this arise, before going culties that on to see how they can be resolved. In the bivariate context, the natural question to pose is usually: we know X what of lX,l') F?' For a random real pair is the best predictor on (,T, #) we = x, class conditional evidently define of functions for distribution j7.2) a (see can K For any A e B such that PX e A) > 0, let 'naive'
dif
F@ lX
G
A)
PX e A, F < y) PX s A)
=
( 10. 1)
.
This corresponds to the idea of working in the trace of (f1,T,#) with respect to A, once A is known to have occurred. Proceeding in this way, we can attempt to construct a theory of conditioning for random variables based on the c.d.f. We may tentatively define the conditional distribution function,F(.y Ix),when it exists, as a mapping from R2 to R w hich for tixedx q R is a non-decreasing, right-continuous function of y with F(-x lx) 0 and F(+x lx)= 1, and which for fixed y e R satisties the equation =
Px
e
,4,
J' < y)
jgFl.y
=
Ix)#Fx(x)
(10.2)
in y-space for any A e B (compareRao 1965: j2a.8). Think of the graph of F(.yIx) through the sudace of the joint distribution funcas the protile of a tion, parallel to the y-axis, at the point x. However, much care is needed in interpreting this construction. Unlike the ordinm'y c.d.f., it does not represent a probability in general. If we try to interpret it as P(Y S y IX x), we face the possibility that P(X x) 0, as in a continuous distribution. Since the integral of F@ lx) over a set in the marginal distribution of X yields a probability, as in (10.2),it might even be treated as shows that we would need a type of density function. Taking A (.Y f islice'
=
=
=
.x)
=
,(x,y)j-,'(,I()#,k(t) J-j' g,'(vI():,'x(t) -
-
ex
ex
ex
(10.3)
Probability
144
to hold. Since F(x,y) is an integral over R 2 F u bini's theorem implies that F@ 1x) is well detined only when the integrals in (10.3)are with respect to a product ,
meastlre.
If Xand Fare independent we can say unambiguously (butnot very usefully) that FLyjx) Fy%). F(? j.x)is also well defined for continuous distributions. Let Sx denote the support of X (the set on which fx > 0); the conditional p.d.f. is =
J# Ix)
fx,yj .fx@)
=
,
.x
e
sx,
(10.4)
where jxx) is the marginal density of X. We may validly write, for A e B
px
e
x, r s
jgj'
.ftvlo/'axltxo
y)
-
-
f''h
Sx,
j z-twylxlyxt.xlts (10.5) #
-x
where F(..lxl
j' Avlxlfo.
-
(10.6)
-K
The second equality of (10.5)follows by Fubini's theorem, since the function fxny) is integrated with respect to Lebesgue product measure. However, (10.6) appears to exist by a trick, rather than to have a firm relationship with our intuition. The problem is that we cannot work with the trace (A,Ta,#a) when A = (: X()) xl and #(A) 0, because then #a = #/#(A) is undefined. lt is not clear the case when (,Y = x) has occurred' when this event what it means to fails to occur almost surely. Except in special cases such as the above, the factorization dF(y,xl = dF@ Ix)#Fx@) is not legitimate, but with this very important caveat we can define the mean and other moments of the conditional distribution. The conditional expectation of a measurable function #(-Y,i'), given X = x, can be defined as =
=
Ex,n
Ix)
+*
J
=
glx-ybdFytxq,
(10.7)
-X
also written as Elgx,Y) lX = #. The simplest case is where g(X,j') is just F. f(l'Ix) is to be understood in terms of the attempt of an observer to predict F after the realization of Xhas been observed. When Xand Fare independent, &F1x) where EY4 is the ordinary expectation of F, also called the marginal or = E, unconditional expectation. ln this case, the knowledge that X x is no help in predicting K =
10.1 Example These concepts apply to the bivariate Gaussian distribution. From (8.36), the density is -1
fx,yj
1
=
2a
C11 C12
c12c22
j/2
exp
,r
-
a
611 C12
z't.x g.1,y g,21 C12 -
-
C22
/1
X-
F
-
/2
Conditioning
145
(A' g,2) -
-
1
= 2J(cl
1c22
2 cl2 g,1) (.:7 c1l (x -
-
-
c212)
exp
jzl) 2
-
-
l/2
2 c22
zcjj
c212
,
(10.8)
-
c11 the square in the exponent.
where the last equality is got by completing Evidently, fxky) fy Ixjfxx) where =
(A,
p,2) -
-
1I)
fl.y
1
=
2 WIVc2a c12 c11
exp
ja
c12 c11
2
'--fx
-
-
2 c22 -
-
c21 a
M,1)
(10.9)
c11
and fxxj
l exp 2zrc11
=
-
@- B1)1
zo.11
.
(10.10)
Thus, F(Flx)
(T12
----t.r
=
g,2 +
-
c22
p,1),
(10.1 1)
and VartFlxl If c1a
of the
=
0,
fy lx) reduces
marginals,
=
622
-
c212 .
c11
(10.12)
so that the joint density is the product and x and y are independent. n to
fvy),
10.2 Conditioning on a Sigma Field In view of the limitations of working directly with the distribution of (aY,l'), we pursue the approach inoduced in j7.2, to represent partial knowledge of the distribution of F by specifying a c-field of events N g 5 such that. for each G e N, an observer knows whether or not the realized outcome belongs to G. The idea of knowing the value of a random variable is captured by the concept of subfield measurability. A random variable .X()): fl F-> R is said to be measurable with respect to a c-field V c 5 if x-2B)
=
(:
.Y4(0)G #) e V, for al1 B e f.
(10.13)
The implication of the condition V c 5 is that the r.v. X is not a complete
Probabilit.y
representation of the random outcome ). We denote by c(m the intersection of all c-fields with respect to which X is measurable, called the c-fild generated by X.
Jf, on being confronted with the distribution of a random pair (X()),1'())), we learn that X we shall know whether or not each of the events G e ctm has by determining whether X(G) contains x. The image of each G e c(A3 occurred under the mapping .x,
=
(X()), F()))2 D
F--> R
2
is a cylinder set in R 2 an d the p.m. defined on c(A') is the marginal distribution of X. .
10.2 Example The knowledge that xl fk X ; can be represented R). R c(f(-x, x1: x < xl ), ((x, x): x > .x2
by
.x2),
=
Satisfy yourself that for every element of this c-field we know whether or not X belongs to the set; also that it contains all sets about which we possess this knowledge. The closer together a'1 and are, the more sets there are in R. Wben +x, lO,R 1. n J4 = 5 ctm, and when xl .v1 x2, R .x2
=
=
=
-x,
.:2
=
=
The relationships between transformations and subfield measurability are summarized in the next theorem, of which the first part is an easy consequence of the definitions but the second is trickier. lf two random variables are measurable with respect to the same subfield, the implication is that they contain the same information; knowledge of one is equivalent to knowledge of the other. This means tha t evety Borel set is the image of a Borel set under g- This is a stronger condition than measurability, and requires that g is an isomorphism. It suffices for g to be a homeomorphism, although this is not necessary as was shown in 3.23. .
10.3 Theorem Let X be a r.v. on the space (S,fs,p), and let i' gl S F-+ 'T is a Borel function, with S i R and 'T i R. (i) ctl') c ctm. (ii) c(1?) ctm iff g is a Borel-measurable isomorphism.
=
glX) where
=
Proof Each B e f.r has an image in fs under g -1 w hich in turn has an image in c(m under X -1 This proves (i). To prove (ii), define a class of subsets of S, C (.g-1(#): B e B v.). To every -1 A S there corresponds (sinceg is a mapping) a set B 'T such that A = g B), and making this substitution gives ,
.
=
fs
lz4:
'(z4)
e
.r 1
fp-j (#): gg
=
-1
(#)) e s v l
=
g
jj(;
,
.
j4;
where the inclusion is by measurability of g -1 an d the second equality is because -' = gg (#)) B for any B c 'T, since g is 1-1 onto. It follows from (10.14)that ,
fx -1( s -4)
j:
z4
e ls
)
g
(x-1(:-1(,)) c
f: B s f.y).
(10.15)
If F is V-measurable for some c-field V 5 (suchthat V contains the sets of the right-hand member of (10.15)), then X is also V-measurable. ln particular, ctm ctl?). Part (i) then implies c(m = c(F), proving sufficiency of the conditions. = To show the necessity, suppose first that g is not 1-1 and glxL) = y '(.n)
Conditioning
(x1) and (x2)are elements of Ss but not of C, which contains only g -1(fy)) fxl) k.p (x2) Hence : Bs C, and 3 a Os-set X for which G there is no Or-set B having the property g -1(B) A This implies that (say)for xl
y:
The sets
.n.
=
.
.:-1(4)
=
.
c(F), so that c(F) c ctm. We may therefore assume that g is 1- 1 If g is not Borel-measurable, then by C; and again, definition 34 = g -1(B) G B s suc h thatgtA) =B e tB.r,and henced Ss C, so that c(l') c ctm by the same argument. This completes the proof of
ctm but
-1
.
necessity.
.
We should briefly note the generalization of these results to the vector case. A random vector #4): f F-> Rk i s measura b1e with respect to 97 c 5 if X -1 B4
=
f
:
A)l
e #)
G
s vs ,
s pk
(1().j6)
.
If ctAr) is the c-tield generated by X, we have the following result.
10.4 Theorem Letfbe $ c R an d Bks fl =
F
k arandom vector on the probability space (S,fs,p) Sk) and consider a Borel function s: B e
f-h
,
g(m: S
=
w jaere
F--
1', T' g R2'.
(10.17)
(i) G'(1') l c(A'). (ii) If m k and g is 1-1 with Borel inverse, then c(1') =
=
ctm.
Proof This follows the proof of 10.3 almost word for word, with the substitutions of fs k an d Bkr for Ss and S.r, X and F for X and F, and so forth. .
10.3 Conditional Expectations Let i' be an integrable r.v. on (f,T,#) and N a c-tield contained in F. The term conditional expectation, and symbol F(FIN), can be used to refer to any integrable, N-measurable r.v. having the property jgFtrl
U)dP
=
F(FI joj'dp
G)#(G), all G e N.
=
(10.18)
Intuitively, F(FIV)()) represents the prediction of F() made by an observer having information V, when the outcome (.t) is realized. The second equality of (10. 18) supplies the definition of the constant S(l'l G), although this need not exist unless #4G) > 0. The two extreme cases are F(FI@) i' a.s., and F(Jrl 5) EY) a.s., where T denotes the trivial c-field with elements (f,0 ). Note that f s 5', so integrability of F is necessal'y for the existence of F(FIV). The conditional expectation is a slightly bizarre construction, not only a r.v., but evidently not even an integral. To demonstrate that an object satisfying (10.18) actually does exist, consider initially the case F 0, and define =
v(G)
=
JgF##.
10.5 Theorem v is a measure, and is absolutely
=
(10.19) continuouj with respect to #.
Probability
148
0. lt remains to show Clearly, v(G) 2 0, and #(G) = 0 implies v(G) is a disjoint sequence, then countable additivity. If ll Proof
=
vUb
v,o,Ydp
= *'
j
J
=
J
X j
o,Ydp *'
-
'
Xvt(l,
(10.20)
./
where the second equality holds under disjointness.
.
So the implication of (10.18)for non-negative F turns out to be that F(FIT) is the Radon-Nikodym derivative of v with respect to #. The extension from nonnegative to general F is easy, since we can write F F+ F- where F+ and Yare non-negative, and from (10.18),F(Fl V) F(F+IV) E(Y- IN), where both of the right-hand r.v.s are Radon-Nikodym derivatives. The Radon-Nikodym theorem therefore establishes the existence of F(FI5:); at any rate, it establishes the existence of at least one nv. satisfying (10.18).It does not guarantee that there is only one such r.v., and in the event of nonuniquenes, we speak of the different versions of F(l'IV). The possibility of multiple versions is rarely of practical concern since 10.5 assures us that they are al1 equal to one another a.s.g#j, but it does make it necessary to qualify any statement we make about conditional expectations with the tag a.s.', to indicate that there may be sets of measure zero on which our assertions do not apply. ln the bivariate context, F(Fl c(m), which we can write as FtFlm when the context is clear, is interpreted as the prediction of F made by observers who observe X. This notion is related to (10.7)by thinking of F(l'l x) as a drawing from the distribution of F(FIm. =
=
-
-
10.6 Example ln place of (10.11)we write &Flm(t,))
61 2
=
---(X() p,2 + C1 1
-
p.1),
(10.21)
which is a function of X(l, and hence a r.v. defined on the marginal distribution of X. F(FIm is Gaussian with mean g,2 and variance of c2la/cll. E! Making
F(FIx)a point
in a probability space on R circumvents the difficulty
previously with conditioning on events of probability 0, and our is valid for a11distributions. lt is possible to define F(FIG4 when
encountered construction #(G) 0. What is required is to exhibit a decreasing sequence fGn e N) with #(Gn) > 0 for every n, and Gn 1 G, such that the real sequence (F(1'1 Gn)) converges. This is why (10.7)works for continuous distributions. Take Gn gx,x + 1/p1XR s =
=
c(A'), so that G
=
Using
(x)xR.
(10.4)in (10.18),
+co f*x+1/n
f(rl Gn)
J..J
yfl,ytdldy
+x
x
=
--
+.
ujtn
uJ-?'(''.''vjvy
j-giy'xsdy =
&Fl*,
(10.22)
Conditioning x. Fubini's theorem allows us to evaluate these double integrals one as n dimension at a time, and to take the limits with respect to n inside the integrals with respect to y. --->
Conditional probability can sometimes generate paradoxical results, following case demonstrates.
as the
10.7 Example Let X be a drawing from the space (g0,11, p,IJ, m), where m is Lebesgue measure. Let N c fp,1) denote the c-field generated from the singletons fxl, x e (0,1). A1l countable unions of singletons have measure 0, while all complements have measure 1. Since either #(G) 0 or #4G) = 1 for each G e V, it is clear from (10.18)that F(XIN) F(m a.s. However, consider the following weknowwhetheror notxc Gforeach Ge V, weknow argument. 'Since fx) e N, if x. ln particular, V contains knowledge of the outcome. It ought to be the case that F(XIV) X a.s.' n =
=
=
,
=
The mathematics are unambiguous, but there is evidently some difficulty with the idea that V should always represent partial knowledge. lt must be accepted that the mathematical model may sometimes part company with intuition, and generate paradoxical results. Whether it is the model or the intuition that fails is a nice point for debate.
10.4 Some Theorems on Conditional Expectations 10.8 Law of iterated expectations
(LlE)
FgF(J'IN)j Proof lmmediate from
(10.23)
Ek.
=
(10.18),setting
G
=
D.
.
if constant' under The intuitive idea that conditioning variables can be held the conditional distribution is confirmed by the following pair of results. 'as
10.9 Theorem lf X is integrable and V-measurable, then Proof Since X is N-measurable, E
+
then
jyydp
-
=
1): X()
>
F(XIN)()1
F(.YI jg-yxf jyjxN)J#
This contradicts (10.18),so PE*) = (tt):X((t7) < #(xl V)(@) E V.
N))J#
=
=
F(#IN)
>
=
X, a.s.
e N. lf #(F+)
0.
0. By the same argument, P(E-)
>
0,
(10.24) =
0 where E-
*
10.10 Theorem lf Fis F-measurable and intepable, X is V-measurable for V g @, N) a.s. < x, ten &FaY1T) and E IA-FI -YSIFI =
Proof By definition, the theorem follows if %)dP xYFIFI
Let
xn)
=
o
'z-lajlsj
=
J s
XYdP a.s., for al1 G e T.
(10.25)
be a V-measurable simple f.v., with F1,...,Fn a pmition of
Probability
150
f and Ei e V for each i. (10.25)holds for X
X(n) since, for all G e V,
=
jgxtnwtyl cldp nwj ac, s(I,Itvp x -
i= 1
=X-
.ij
yw,
-
GIAEt'
/=1
j x-,vdp,
(10.26)
G
Ei e N when G e V and Ei e V. Let X k 0 a.s., and let (X(,,)J be a monotone sequence of simple N-measurable converging XY a.s. and IX(n)F( K lXFI functions to X as in 3.28. Then XnlY N) where F1 XFI < x by assumption. Similarly, X(n)F(FI #FIl'IV) a.s., and
noting G
fa
,--+
,
--
e-l-Ytnlftrl N) I
=
1-Y(n)1'l I9-))
F1 F(A)n)FI N) I S EE
= FlaYtall'l
e-IA'l'I<
<
(10.27)
x,
where the first inequality is the conditional modulus inequality, shown in 10.14 below, and the second equality is the LlE. It follows by the dominated convergence JcxftFl N)##, and so (10.25)holds for non%)dP theorem that Jc#(n)F(rI negative X. The extension to general N-measurable X is got by putting -->
x where X*
=
maxfmo)
eFXl 9')
=
=
X+ -X-
(10.28)
2 0 and A'- 2 0, and noting
EIYX'P
= X*
-
-
FX- IN)
X-)F(l'IN)
=
ECYX' IN) EYX- lN) -
XFIFIV) a.s.,
=
using (10.33)below and the result for non-negative X.
(10.29) .
X does not need to be integrable for this result, but the following is an important application to integrable X.
10.11 Theorem lf F is F-measurable and integrable and F(rl V) El') for N c @, then Covtmp = 0 for integrable, V-measurable X. =
Proof From 10.8 and 10.10,
ECXYI #(A'(A-FlN)) =
If
F(FIV)
=
EY) a.s.
(a constant),
=
Ft-tftrl
V)J.
= then ECXYI EkEr).
(10.30) w
Note that in general Covt#,F) is defined only for square integrable r.v.s X and K But Cov(X,Y) 0, or ECXYI= FtmFtF), is a property which an integrable pair can =
satisfy.
The following is the result that justifies the characterization of the conditional mean as the optimal predictor of J' given partial information. optimal'is sen to have the specific connotation of minirnizing the mean of the squared
prediction errors.
Conditioning 10.12 Theorem
.:
t $... .(y:,.. y... ).)... ../.
g
Let
denote any V-measurable approximation to F. Then < (1 Y- f'112. jlY- F(rI N) 112
.j;
-
...
151
(10.31)
-
. .. .. .-.
Proo
f (F-
hence
f)2 (F-F(FIN)j2+2(y-F(Fj f)2IT)
F((F-
=
Fg(F-F(Fj
a,q-
V)jgF(J'l T)
=
N))2jNj + (F(yl N)
-
a..v
Fj
-
+
gF(FIN) F)2, and
f12a.s.,
-
(10.32)
noting that the conditional expectation of the cross-product disappears by definition of F(1'1 V), and 10.10. The proof is completed by taking unconditional expectations tluough (10.32)and using the LIE. w The foregoing are results that have no counterpart in ordinary integration theory, but we can often exploit the fact that the conditional expectation behaves expectation, apart from the standard caveat we are dealing with like a nv.s so that diferent behaviour is possible on sets of measure zero. Linearity holds, for example, since %real'
hFIN)
EaX+
=
aExt T) +
:FIFIT),
(10.33)
a.s.,
is a direct consequence of the detinition in (10.18).The followipg are conditional versions of various results in Chapters 4 and 9. The first extends 4.5 and 4.12. 10.13 Lemma V) 0 a.s. (i) If X 0 a.s., then F(aYI (ii) If X f F a.s., then F(.Yl T) S F(Fl V) a.s. (iii) If X F a.s., then F(aYlN) F(FIV) a.s. =
=
=
=
Proof (i) follows diretly from (10.18).To prove (10. 18) and 4.8(i) together imply jaExk
for a11G e V. SinceA
N)##
=
(ii), note that the hypothesis,
jaxdp hk'dp S
=
JgF(Fl
G4dP
T)((t))> 6(FIT)((t))1e V, itfollows f: F(xYI
that #(A) = 0. The proof of (iii) uses 4.8(ii), and is otherwise identical to that of (ii). w
10.14 Conditional
=
IF(FIT) I f
modulus inequality
E Il'l IT) a.s.
Proof Note that IFI F+ + Y-. where F+ and Y- are defined in (10.28).These are non-negative nv.s so that F(F+l N) > 0 a.s. and EY- j N) 0 a.s. by 10-13(i) 1, and (ii). For ) e C with PC4 =
=
lF(F+
-
Y- lN)()1
=
IF(F+l T)()
-
< F(l'+IT)((o) + =
Ev+w y-
/(F-1 97)())1 EY-
I )(),
where both the equalities are by linearity.
.
lN)() (10.34)
Probability
10.15 Conditional monotone convergence theorem lf L f F and L 1- 1' a.s., then F(FnI5') 1' F(1'1 N) a.s. Proof Consider the monotone sequence Zn L K Since Zn f 0 and Zn S Zn+I, 10.13 implies that 4he sequence Ezn lN) ) is negative and non-decreasing a.s., and hence converges a.s. By Fatou's Lernma, =
-
ja limsup Ezn N) dP limsup jaE(Zn U4dP k l I n'-/oo
n'->
= limsup
Zndp
=
G
N-->=
(10.35)
0
for G e V, the tirst equality being by (10.18), and the second by regular monotone convergence. Choose G = f : limsupaAzn lT)() < 0J which is in N by 3.26, and (10.35) implies that #4G) 0. It follows that ,
=
V) lim F(ZnI
0, a.s.
=
(10.36)
.
n'-co
10-16 Conditional
Fatou's
liminf
lemma If Fn k 0 a.s. then
&FnIV) k
E liminf L IN a.s.
influnrk so that l%' is non-decreasing, and converges to F = EY' IW) by 10.15. F,, k Fr, and hence E liminfarr,. Then E'n' 5') l5>)1 E'n' lT) a.s. by 10.13(ii). The theorem follows on letting n . Proof Put
L'
(10.37)
n-'+oo
n-oo
=
--+
--)
,.
Extending the various other corollaries, such as the dominated convergence theorem, follows the pattern of the last results, and is left to the reader.
10.17 Conditional
Markov
inequality
#(1IFl
P E.l IT) S
F(l
rlpl T) EP
,
a.s.
Proof By Corollary 9.10 we have
s'jolfIrlkcll' By definition, for G e N,
jopf I1'l
f
Jgl
Y'PdP, G e @.
k El I*4dP
and jaEf
FIPIUtdp
=
=
Jclt
ll'lk.:)tf#,
Jcl
YPdP.
Substituting (10.39)and (10.40)into (10.38),it follows that
(10.38)
(10.39)
(10.40)
Conditioning JcEEf#(t
l1'I 2
El
153
I5') -F(I F1#1U4dp
(10.41)
f 0.
The contents of the square brackets in (10.41)is a V-measurable r.v. Let G e N 0. . denote the set on which it is positive, and it is clear that #(G) =
10.18 Conditional Jensen's inequality Let a Borel function () be convex on an interval 1 containing the support of a i-measurable r.v. F where F and (/(J') are integrable. Then
4'tftrlT))
< F@tl'')l
Proof The proof applies 9.13. Setting x
N), a.s.
(10.42)
F(l'IV) and
=
y
J' in
=
T)) < :(F) -#(F(1'IN)).
A(F(1'l V))(F-F(Fl
(9.31), (10.43)
V)) is a random variable. It is not certain that However, unlike A(F(D),A(F(FI the left-hand side of (10.43)is integrable, so the proof cannot proceed exactly like that of 9.12. The extra t'rick is to replace F by 1sF, where E ftl): F(FIN)() f #1 for B < x. F(Fl V) and hence also ls are V-measurable random variables, so F(1sFIT) = 1sF(l'IV) by 10.10, and =
N) e'@(1sl')I
=
= Thus, instead of
F(9(F)ls+ lsfttF)l
#(0)1rIT) V)
(1
+
-
1s)#(0).
(10.44)
(10.43),consider
A(F(ls1'IT))(1sF-
lsF(l'l V)) k #(1sF) -#(1sF(l'l
T)).
(10.45)
The majorant side of (10.45)is integrable given that (/(i') is integrable, and hence so is the minorant side. Application of 10.9 and 10.10 establishes that the conditional expectation of the latter term is zero almost surely, so with (10.44) we get
#(1sF(l'IV)) Finally, 1et #
--
x
< 1sF@(1?)l V)+
so that ls
-->
(1
-
(10.46)
1s)9(0), a.s.
1 to complete the proof.
.
The following is a simple application of the last result which will have use
subsequently.
10.19 Theorem Let X be V-measurable and fv-bounded for r k 1. If F is F-measurable, X+ F is also Lr-bounded, and F(Fl V) 0 a.s., then =
FIxY+ FIr k
FI-YIr.
(10.47)
Proof Take expectations and apply the LlE to the inequality F(1-Y+ FIr! N) 1
IEX+
irj N)Ir
=
lx1r
a.s.
.
(10.48)
Finally, we can generalize the results of j9.6. It will suffice to illustrate with the case of differentiation under the conditional expectation.
154
Probability
10.20 Theorem Let a function G(,0)
e'(-J I I
r)
dedl 9'' -
:-o
E
-
G(%)
hv
T
I
8=00
(v, v e
Proof Take a countable sequence of the conditional xpectation, G(0() + h
satisfy the conditions of 9.31. Then
F(G(%
(Nl with hv +
v)
=
lV)
(10.49)
-.s.
,
-
--
0 as v
x.
--
(G(%) IT)
By linearity
(10.50)
a.s.
hv
If Cv 6 V is the set on which the equality in (10.50)holds, with #(Cv) 1, the #(OvCv) a nd limit 1 by 3.6. The in the two sequences agree on the set Ovcv, left-hand side of (10.50)converges a.s. to the left-hand side of (10.49)by assumption, applying the conditional version of the dominated convergence theorem. Since whenever it exists the a.s. limit of the right-hand side of (10.50) is the right-hand side of (10.49)by definition, the theorem follows. . =
=
10.5 Relationships between Subfields TI
5 and Mzc 5 are independent subfields if, for every pair of events Gj e V1 and Gz s N2, #(Gj r7 G
=
#(G1)#(Gc).
(10.51)
Note that if F is measurable on Vl it is also measurable on any collection containing V1,and on ; in particular. Theorems 10.10 and lo-llcover cases where F as well as X is measurable on a subfietd.
10.21 Theorem Random variables X and F are independent iff ctm and ctF) are independent. Proof Under the inverse mapping in (10.13),G1 e c(m if and only if #1 X(G1) i T with a corresponding condition for c(F). It follows that (10.51)holds for each G1 e c(m, G1 G c(F) iff PX* Bj, i'e Bz4 PX e #1)#(Fe Bz) for eachfl X(G1), Bz F(G2). The only ifo of the theorem then follows directly from the follows, given (8.38),from the fact that every Bi e definition of ctm. The B has an inverse image in any subfield on which a r.v. is measurable. w =
=
=
=
'if'
The
only ifo in the first line of this proof is essential. Independence of the subtields always implies independence of X and F, but the converse holds only for the infimal cases, c(m and f5(1').
10.22 Theorem Let F be integrable and measurable on V1. Then 5'I a.s. for a11 V independent of
F(FIV)
=
EY)
.
Proof Define the simple Vl-measurable r.v.s l'(n) G11,...,G1s of ( where G1j e V1, each i, with l%)
On a partition = Z'lzzlyflclf 1 F as in 3.28. Then
Conditioning
jynlb
5')J#
n
X'faFtltyl/l
=
f=1
=
*'
vl,
%''
#(G)XX#(G1f)
-
155
ynyjacjf
c)
ra
=1
V,
all G e
fOr
PG4EYn)4
=
(10.52)
f=1
F(i') by the monotone convergence theorem. F(F(n) IN) is not a simple function, but EYn) IV) F(Fl V) a.s. by 10.15, and
Ejnj)
--
'1'
joEn)
IVV# --
VV#' jcfk''
(10.53)
convergence. Hence for Vl-measurable F,
by regular monotone
JgFIFIM4dP
=
for a1l G e V.
PGjEY)
(10.54)
Frop the second equality of (10.18) it follows that F(FIG) which proves the theorem. w
10.23 Corollary
If X and F are independent, the
Proof Direct from 10.21 and 10.22, putting V
=
&FI
EY4 for a11 G e V,
=
.)
=
E').
ctm and V1
=
c(F).
.
10.24 Theorem Apairof c-fields V1cF and Vac 5 areindependentiff Cov(X,D 0 for every pair of integrable r.v.s X and F such that X is measurable on V1 and F is measurable on N2. =
Proof By 10.22, independence implies the condition of 10.11 is satisfied for N tif' consider X ito To prove lcj, G1 e V1, and F 1cz for V1, proving G2 e N2. X is Nl-measurable and F is Na-measurable. For this case, =
Cov(X,D
=
Cov(X,D =
=
,
.
#(GI
fa
=
G2) #(Gl)#(G2).
(10.55)
-
0 for every such pair implies V1and V2 are independent by (10.51).w
10.25 Corollary Random variables Xand F are independent iff Cov@(m,v(F)) 0 for every pair of integrable Borel functions 4)and v.
=
Proof By 10.3(i), 9(A') is measurable with respect to c(m for al1 and v(F) is ctpimeasurable for a1l v. If and only if al1 these pairs are uncorrelated, it follows by 10.24 that c(A3. and cti') are independent subfields. The result then follows by 10.21. w ,
An alternative proof of the necessity part is given in 9.17. The next result generalizes the law of iterated expectations to subfields. We say that c-fields V1 and V2 are nested if V1 c V2.
10.26 Theorem lf N1 i Va @, then for T-measurable F, N2)1N11 F(l'l N1) a.s. (i) EEF(l'I N1)1T21 F(l'IT1) a.s. (ii)FEe-(1'l =
=
Probability
156 Proof By definition, jLELECY'
V2)IUjjdp
But, since G e T1 implies G jcyE''
J(JIFI 5'2)## for
=
6
V2)1T11##
al1 G e N1.
(10.56)
N2, (10.18)and (10.56)imply that =
jo-dp
all G e T1,
fOr
(10.57)
T2) IV1 is a version of F(Fl N1), proving (i). Part (ii) is by so that A'(F(Fl NI) is a Vc-measurable r.v. w sinceF(FI
10.9,
lf A simple application of the theorem is to a three-variable disibution. 3 (X(),l'(),Z())) is a random point in R measura ble on T, 1etc(Z) and c(F,@ be infimal c-fields on which Z and (F,@ respectively are measurable, and c(/ the c(F,@ 5. Unifying notation by writing F(l'Iz) = F(FIc(z)) and F(rlmz) = F(FIc4-Y,/), 10.26 implies that ,
F(e'(i'I-Y,z)IZ1
=
F(A'(FIz)I-X-,Z)4
=
F(1'1z).
(10.58)
Our final results derive from the conditional Jensen inequality.
10.27 Theorem Let F be a F-measurable r.v. and T1 i Uz Proof
T. lf
9(.)is convex,
F(t(F(FI5-2))) 2 F(9(e'(rI971))(). Applying 10.18 to the Tz-measurable r.v. F(FIW2)gives F@(f(FIT2))1V1) k #(F(F(l'I91)1N1)) #(F(Fl V1)) a.s.
(10.59) (10.60)
=
where the a.s. equality is by 10.26(i). The theorem follows on taking unconditional expectations and using the LlE. . The application of interest here is the comparison of absolute moments. Since lxlP 1, the absolute moments of F(FIT2)exceed thoseof F(FIT1) when is convexforr N1 i N2. ln particular,
(10.61) FgF(rIT2)21 k FgF(rIN1)2j. SinceFti'l NllandFtFl Vzlbothhavemean ofE, T2))k (10.61)impliesVar(F(FI EYl) for i 1 or 2, (the T1)). Also, F(F(FIVj)2)+F((F- F(FIVj))2) Var(F(FI =
=
expected cross-product vanishes by 10.10), so that an equivalent inequality is Fgtr-e'(J'l N2))21 J Fg(r-F(J'j T1))2j. (10.62) The intepretation is simple. V1represents a smaller infonnation set than V2, and if one predictor is based on more information than another, it exhibits more variation and the prediction error accordingly less variation. The exeme cases ;) are &FI = Fand F(FIT) = F(1'), with variances of Vartp and zero respectively, This generalizes a fundamental inequality, that a variance is non-negative, to the partial infonnation case. While (10.61)generalizes from the square to any convex function, (10.62)does
Conditioning
157
not. However, there is the following nonn inequality for prediction errors. 10.28 Theorem If F is T-measurable and N1 i N2 i ;,
IIF-FIFIN2)11,K 2I1F-F(F1Tl)IIp, p k 1. Proof Let
n
n -
F-
=
F(FIV1). Then
F(n lN2)
=
(10.63)
by 10.26(ii),
F- F(r1 N1)-&F1 T2) +&&FIV1)1T2)
= J'-s(rI va).
(10.64)
The theorem now follows, since
IIq F(q IT2)l1p< IIq11,+11F(nIT:2,)11,f 2IIq11p
(10.65)
-
by, respectively, L1E.
the (Minkowsk.i and conditional Jensen inequalities, and the
.
10.6 Conditional Distributions The conditional probability of an eventA e # can evidently be defined as #(4 IN) F(lx 1N), where 1x() is the indicator function of A. But is it therefore meaningful to speak of a conditional distribution on (D,@), which assigns probabilities PA 1N) to each A e @? nere are two ways to approach this question. First, we can observe straightforwardly that conditional probabilities satisfy the axioms of probability except on sets of probability 0 and, in this sense, satisfactorily mimic the properties of true probabilities, just as was found for the expectations. Thus, we have the following. =
10.29 Theorem (i) P(A IN) k 0, al1 A e F. (ii) Pf IN) 1 a.s. (iii) For a countable collection of disjoint sets Aj e @, =
#
UAjIT
=
X#(4jI 9') a.s.
(i),suppose 3 G e N with
Proof To prove
(10.18),
jaudp
=
PA IV)())
Jg#(4 I@4dP <
(10.66) 0 for a11(t) e G. Then, by
<
(10.67)
0,
which is a contradiction, since the left-hand member is a probability. To prove (ii), note that #(f1 lV) is N-measurable and 1et G+ e N denote the set of (l) such G+, that #(fl V)4)) > 1. Suppose #46*) > 0. Then since G+ rn =
#(&*)
=
josdp <
jo.#(QIT)##
=
jawezdp =
#(G+),
(10.68)
Probability
158
which is a contradiction. Hence, #(G+) 0. Repeating the argument for a set G- on 0. For (iii), (10.18)gives, for any which #4f1 I5')() < 1 shows that #(G-) G 6 V, =
=
J(,#(U,A'
jIU4dP
(10.69) Xyjocvbdp' jocvjpdp jvjacpdp =
=
=
since the sets G /'7 Aj are disjoint if this is tl'ue of the Aj. By definition there exists a version of #(l N) such that V G 6 5',
jocujdp Js#(X/IV) dP,
(10.70)
=
and hence
jop-v' /1 T)#/'
=
XJSPIA/IU4dP
=
./
V)) (Xy #(X/I dP.
jo
(10.71)
The left- and right-hand members of (10.71)define the snme measure on W(see 10.5)and hence #(U.?A.j lN) = Q#(AjIT) a.s. by the Radon-Nikodym theorem. w But there is also a more exacting criterion which we should consider. That is, does there exist, for fixed (z), a p.m. g,r,lon (f,F) which satisfies yu(A)
P(A l534)), each A e 5
=
(10.72)
E G where #(C) 1? lf this condition holds, the fact that conditional for all expectations and probabilities behave like regular expectations and probabilities requires no separate proof, since the properties hold for jt(o. lf a family of p.m.s lg,(o, e fJ satisfying (10.72)does exist, it is said to detine a regular conditional probability on V. However, the existence of regular conditioning is not guaranteed in every case, and counter-examples have been constructed (see e.g. Doob 19532 623-4). The = problem is this. ln (10.66),there is allowed to exist for a given collection (A e TJ an exceptional set, say Ca with PCa) 0, on which the equality fails. This in itself does not violate (10.72),but the set Ca is specific to and uncountable countable subsets number of there typically 5, we since are an Poaj would required both be for 0, be that guarantee to a p.m. cannot as g,fo and to satisfy (10.72). =
.4
=
.d,
.4
=
This is not a particularly serious problem because the existence of the family lju) has not been critical to our development of conditioning theory, but for certain purposes it is useful to know, as the next theorem shows, that p.m.s on the line do admit regular condittonal distributions. *
10.30 Theorem Given a space L1,5,P) and a subfield N c @,a random variable F has a regular conditional distribution defined by Fr/l
for
e C with #(C)
N)() =
=
j ,
lT)(), Fg(. IV)()
#((-x,y1
where
y e R, is a c.d.f. for all
(10.73) e f1.
Conditioning
159
Proof Write F(o*(y)to denote a version of #((-x,y1 I9')(). Let Mq denote the set of ) such that F' ri) > F(o*(rj)for ri, rj e :) with ri < (/ Similarly, 1et Ri denote the set of (l) on which limn-yx/trj + 1/n) # FLlril,ri e e. And tinally, 1et L denote the set of those ) for which F(l(+x) # 1 and F(o*(-x) y: 0. Then C (Uj#C is monotone and right-continuous at rn Lc is the set of on which FfLvl rn F(l(+x) with 1 and Ff)(-x) 0. For y e R let a11rational points of the line, .
-&lc
=
=
Fg'@) y e ()
=
,
Fr(. I5')()
=
F.(.y+), y e R
(,)e c,
-0
(10.74)
Otherwise,
G(X, where G is an arbitrary
c.d.f. ln view of 10.29, #(Mj) 0 for each pair i,j, PRi4 0 for each i and PL) 0. (1fneed be, work ln the completion of the space to define these probabilities.) Since this collection is countable, PC) 1, and in view of 8.4, Fy(. IV)() is a c.d.f. which satisfies (10.73),as it was required to show. . =
=
=
=
lt is straightforward, at least in principle, to generalize this argument to multivariate distributions. For # e B it is possible to write F(1s I5')(t,))
=
V)() jedkyf
(10.75)
a.s.,
and the standard lgument by way of simple functions and monotone convergence leads us full circle, to the representation
F(FI9')()
+x
J
=
-X
y#&@l
syy(y,a.,.
(10.76)
ctm, we have constructions to parallel those of j10.1. Since no restriction had to be placed on the distribution to obtain this result, we have evidently found a way around the difficulties associated with the earlier detinitions. However, Fy(. IV)() is something of a novelty, a c.d.f. that is a random element f'rom a probability space. lntuitively, we must attempt to understand this as representing the subjective distribution of F(@ in the mind of the observer who knows whether or not ) e Gfor each G e V. The particular case Fy(. IN)4) is the one of interest to the statistical modeller wheh the outcome ) is realized. Manyrandom variables may begeneratedfromtheelements of (D,1,#), notonly the but also variables outcome itself in the bivariate case the pair F(),X() and A11 have the quantiles of Fv@ these such as F(FIm(), to be thought I of as different aspects of the same random experiment. Let X and F be r.v.s, and N a subfield with V c Rx c(A') and V ? Rv c(F). We say that X 4nd F are independent conditional on V if If V
=
-
-
m().
=
Fxrtmyl 5')
=
Fx@l S'IFI-I.yI V) ms.
=
(10.77)
Probability
160
This condition implies, for example, that f(Xl'IV) g(.,(o) be the conditional measure such that g,(o(lX 6
(--,A1,
i'
(--,A'1 1)
6
=
=
F(XIV)F(FIV) a.s.
Fxvx'y
Let ju
=
1V)().
with (t) fixed this is a regular p.m. by (the bivariate generalization ofl 10.30, and gwotAfa #) g(o(A)g(o(#) for eachA e Rx and B e Xy, by 10.21. ln this sense, the subtields Rx and Ry can be called conditionally independent. =
10.31 Theorem.
lf X and F are independent conditional on V, then
f(l'1 Rx4
F(1'l N) a.s.
=
(10.78)
Proof By independence of Rx and Rv under JxftFl
XaYll/t
=
JxFtfltto
=
u we can write BtX)JFtV' X e Rx.
(10.79)
This is equivalent to
&1z&F1Jx)l 9')(t,)) F(lxl'l 5')() =
T) IT)() = F(1xF(1'I
were tlae tirst equality also with respect to #, noting f
JxF(FIRdp
=
a.s.(#1,
(10.80)
follows from 10.26(i) and 10.10. lntegrating over f e V, using 4.8(ii) and the LIE, we arrive at JxF(Fl U4dP. A e Rx.
jgk'dp =
This shows F(Fl Rx) is a version of
F(FIN), competing
(10.81)
the proof.
F(FIRx) is in principle
.
Xvmeasurable, it is in fact almost surely I#J equal to a V-measurable r.v. Needless to say, the whole argumentis symmetric in X and Rp The idea we are capturing here is that, to an observer who possesses the information in V (knowswhether (t) e G for each G e V), observing Xdoes not yield any additional information that improves his prediction of F, and vice versa. This need not be true for an observer who does not possess prior information. Equation (10.77) shows that the predictors of F based on the smaller and larger information EY4 a.s., so sets are the same a.s.(#), although this does not imply F(Fl R that X apd F are not independent in the ordinary sense. Thus, while
=
11 Charqcteristic Functions
11.1 The Distribution of Sums of Random Variables Let a pair of independent r.v.s X and F have marginal c.d.f.s Fx@) and Fy@). The c.d.f. of the sum J'P X+ F is given by the convolution of Fx and Fs the =
function
Fx*Fy(w)
+x J..Fx(w
=
(11.1)
-ytdFv(9.
11.1 Theorem If r.v.s X and F are independent, then Fx*Fy(w)
#(X+ F f w)
=
FI'*Fx(w).
=
(1 1.2)
Proof Let 1w(my) be the indicator function of the set fmy:x K w y), so that F(1w(X,F)). By independence Fxty) FxxqFvy), so this is #(X+ F < w) -
=
=
Js21w@,A')F@,A')
+x
P+x
J..J
dFyIyl
.x1w@,.')tFx(I)
=
+x
w-y J..JFx@)
J..
=
dFyvl
+x
Fxtw
=
-
y)#Fy(y),
(1 1.3)
-X
where the first equality is by Fubini's theorem. This establishes the first equality in (11.2). Reversing the roles of X and i' in (11.3) establishes the second. w For continuous distributions, the convolution
f
=
fx*fyof
p.d.f.s
fx and h
is
+x
j-.fxw -').fy(')*,
f(w)
=
such that
Jlx/'ttlx
=
(1 1.4)
F(w).
11.2 Example Let X and Fbe independent drawings from the uniform distribu1m,1j@).Applying (11.4)gives tion on 0,11, so that fxx) =
/k+r(w)
=
1
llw-l,wj#y.
()
that the graph of this function forms an isosceles triangle lt is with base (0,21and height 1. n easily veritied
Probability
162
This is the most direct result on the distribution of sums, but the formulae generated by applying the rule recursively are not easy to handle, and other approaches are preferred. The moment generatinghlnction (m.g.f.)of X, when it exists, is jedFx), Eeth Mxtf) t e R, (11.6) =
=
where e denotes the base of natural logarithms. (Integrals are taken over (-x,+x) unless otherwise indicated.) If X and i' are independent, Mx-vvt)
jjetx*'gdt-xx4dll-y,
=
=
jetxdnxjetydFyy)
= Mx(f)Mg(/).
(11.7)
This suggests a simple approach to analysing the distribution of independent sums. The difficulty is that the method is not universal, since the m.g.f. is not defined for every distribution. Considering the series expansion of e all the moments of X must evidently exist. The solution to this problem is to replace the variable f by it, where i is the imaginary number, VX. The characteristic function (ch.f.)of X is defined as ,
9x(f) =
1l
.2
Ee
itX
)
itx
(1 1.8)
e dFx).
=
Complex Numbers
A complex number is z a + ib, where a and b are real numbers and i V-X.a and b are called the real and imaginary parts of the number, denoted a = Re(z) and b lm(z). The complex conjugate of z is the number i' a ib. Complex adthmetic is mainly a matter of carrying i as an algebraic unknown, and replacing 2 j)y j /3 i 4 by 1 e tc wherever these appear in an expression. by One can represent z as a point in the plane with Cartesian coordinates a and b. The modulus or absolute value of z is its Euclidean distance from the origin, =
=
=
=
-
-
,
-,
,
.,
lz l
=
(zD 1/2
=
a
2
+
jp
1/2
)
(1j
.
.
9)
Polar coordinaies can also be used. Let the complex exponential be defined by
efo = cos 0
+ jsin
8
(11.10)
for real 0. All the usual properties of the exponential function, such as multiplication by summing exponents (according to the nzles of complex arithmetic) go through under this definition, and
Ie i l
=
+ sin2e)1/2 (coszo
=
1
(11.11)
Characteristic
Functions
163
for any 0, by a standard trigonometric identity. We may therefore write z Iz Iei where Retz) Iz Icos 0 and 1m(z) Iz Isin 0. Also note, by (11.11),that =
=
=
.Re(z)+Im(z) j Ie z 1 j =
@Re(z)
jef1m(z)j
=
=
@Re(z)
(j.j. jg;
.
.
random If X and i' are real random variables, Z X+ iY is a complex-valued variable. lts distribution is defined in the obvious way, in terms of a bivariate c.d.f., F@,y). In particular, =
EZ)
(11.13)
F(A') + iElY).
=
Whereas EZ4 is a complex variable, FIZI is of course real, and since 1zl < IXl + Il'I by the triangle inequality, integrability of X and F is suftkient for the integrability of Z. Many of the standard properties of expectations extend to the complex case in a straightforward way. One result needing proof, however, is the generalization of the modulus inequality.
11.3 Theorem
If Z is a complex random variable,
Proot Consider a complex-valued
4F(Z) I S FI I .
.
simple nv.
(a./+ lllsy, zn) 7'j=L -
where the xj and I are real non-negative constants and the Ej e 5 for j constitute a partition of f1. Write Pj F(1sy). Then
(11. 14) =
1,...,p
=
2
IEznl) 1
-
2+
77xjpj
=
2
77fjjpj
+ f5,5/,7? 77(a??+ 7777(%a,+ fbbrjn,
( 11.15)
j*k
j
whereas CEIz(n)I)2
=
=
1/2
>7(a,?+ p,?)Pj
2
./ + + jk2)1/2# j pk. X (( + f5?)#? X X (%?+ Y?)'/2(ak2 J
J
J
j*k
./
(11.16)
The modulus inequality holds for Zn) if
0S =
(E IZ(n) I)2
-
2
IF(z(n)) I
p2,)1/2(%a,+ 1%)1#./#,. 7777E(a?+p,?)''2(a,2+ -
J
(1 1.17)
The coefficients of Pjpk in this expression are the differences of pairs of nonnegative terms, and these differences are non-negative if and only if the differences of the squares are non-negative. But as required,
Probability
164
+ p2,) (%a.+ Ip.)2 (a,?+ If)(a2k -
,
a,?p2,+ a%?
=
=
zwakjf'lk
-
(ay;'J. a.I$)2z O
(11 18)
-
.
.
This result extends to any complex r.v. having non-negative real and imaginary Z X+ iY, using 3.28, and invoking the parts by letting Z(n) = Xn) + in) monotone convergence theorem. To extend to general integrable nv.s, splitXand F X* + iY* into positive and negative parts, so that Z = Z+ r, where Z+ X+ 1'+ A?'-+ iT, with X- k 0 and lr- k 0. Noting that 2 0 and k 0, and Zwith '1k
=
=
-
=
IEZ) I < IE(Z+ + r) l < e'lz++ r I FIzl
(11.19)
=
completes the proof.
.
11.3 The Theory of Characteristic Functions We are now equipped to study some of the properties of the characteristic function x(/). The fact that it is detined for any diskibution follows from the fact 1 and Eeth is finite regardless of the 1 for all x; E( Icfl'fl ) that Ie x I distribution of X. The real and imaginary parts of #x(J) are respectively Ftcos t.k and fsin tX4. =
=
k< 11.4 Theorem If E IA-I
x,
then
k
d px(/)
d
=
Eiuk
k itx
e
).
(1 1.20)
Proof
t-bht #x(J) = h
eixt'ht
+=
-
eitx
-
#(x),
h
-x
(1 1.21)
where, using (11.10), e
t-h)
h
-
e
itx
=
cOs
xt
+
h4 h
-
cOs
tx
+
j is n
.7
(/ + 4 -
siu tx .
h
The limits of the real and imaginary terms in this expression as h sin tx and i(x cos /x), so the limit if the integrand in respectively -->
-x
is
xi cos tx
-
sin fx)
=
fccos tx
+
i sin tx)
=
ie
0 are
(11.21)
it: .
since1tfxlefr-lIxl
the integral exists if A'C-Y'I This proves (11.20) for complete 1. To the case k the proof, the same argument can be applied itx inductively to the integrands (ix)k-1e j or k z a s =
,
=
.
,
,...
It follows that the integer moments of the distribution repeated differentiation with respect to t.
can be obtained by
Characteristic lf Fl#! k <
11.5 Corollary
Functions
165
th en
x,
24x(f)
k k = i EX ). n
dtkx
(1 1.22)
An alternative way to approach the last result is to construct a series expansion of the ch.f. with remainder, using Taylor's theorem. Tlzis gives rise to a very useful approximation theorem.
11-6 Theorem lf
FI.Y1k <
then
x,
k
k 77tffytil (1)x(/)j=0 j! Proof A function
,
(1 1.23)
.
differentiable k times has the expansion
which is
f
+1
l I/XI IfXl k! (k+ 1)!
f E min
-
./*(a/)
1 Jz''(0), 3 z'(0) , +
f(f) f(0) + f '(o), + =
where 0 f
; 1. The expansion
(x
of
#
e itx where yk
=
iksgntfx)
k
jwx
(e
ft)
e
=
+
...
:!
p,
)
it?
glves
#
'
itxf
y'l y!
=
+
6
z
+
Itx I yk,
(1 1.24)
k!
j; ayjtj
-
1
sgnt/x
tx k 0
,
=
.
- 1 tx
<
,
0
Applying (11 10) and (11 11), we can show that Iyk I (2 2cos a/x)1/2 < 2. However, by extending the expansion to tenn k + 1, we also have =
.
.
k
e itx
k+1
'
itx/
=
g! .
jzzo
+
ltx I
j or () < (g s j and where zk ik+1sgnt/x) e (1 1.24) and (11.25) hold, we may conclude that ,
k
d
k
'
itxf
itx y.l ./! -
c
min
(1 1.25)
k + 1)! zk,
k+1 ift.T
=
-
2 Itx I k!
jzkj
1. Gjven that both of
=
1+l
,
Itx I
(k+
1)!
The theorem now follows on replacing x with the r.v. #in ations and using the modulus inequality:
(1 1.26)
.
(11.26),taking
expect-
Probability
166 j4e'tx./l k 9x(/)77 !
t ./
-
s
,
e
k
i
-
(/m./ j--,j y!
:7
+1
k
f E min
l I/.X'1 I/-YI k! (k+ 1)! ,
.
(11.27)
w
There is no need for FIXl k+1to ex i st for this theorem to hold, and we can think of it as giving the best approximation regardless of whether I/XI is large or small. To interpret the expectation on the right-hand side of (11.27), note that, for any pair of non-negative, measurable functions g and ,
Ftminf#tm,tml)
inf F(#(m1,4
=
+
Ae 3
tmlxc),
(11.28)
the infimal set being the one containing those points x on which particular,for any s 2 0, the set A ( IXI> EJbelongs to the class the intimum in (11.28)is taken, and we get the further inequality,
< hlx). In
.(x)
=
.
E m1n
2 Itx
k'!
Ik Itxl ,
k+1
k
k+1
2 I/aYI f E 1( jxjlsj k!
k + 1)!
over which
+
2/ E A'1k1f IxI> .:) ) k! I
E
I?-YI lt jxjssj
(k+
1) !
k+1E,k+1
+
It I
k + 1) !
(1 1.29)
S
2/ I/'I F1x'1 kt F(lxI '1. ( IxI>e) ) + 1)! (.+ k! k+1
The second alternative on the light is obtained in view of the fact that k k k+11( IxI
kax+b
=
ibt E efr(cx+h) ) = e #x(J/).
The second is the counterpart variables X and F,
of
(11.7).For
a pair of independent
(1 j ao .
random
jjeitx*'tdlkxtdFy'
#x+y(/) =
=
jeitxdFxxljeivdFvlp
=$x(f)#,'(/).
(11.31)
An interesting case of the last result is F -X' where X' is an independent drawing from the distribution of X. The disibution of X- X' is the snme as that of =
Characteristic X'-X,
Functions
167
symmetric about 0. The ch.f. of X X' is real,
and hence this r.v. is
-
since
#x(f)9x(-f) I4)xtfl I2 =
(11 32) .
,
Ee-ith in v iew o f the fact that bx#aV).It can be verified from the with real ch. f., a1l the existing odd-order moments expansion in (11.23) that a be the trademark distribution. of symmetric zero, must a =
=
:=1Xj where (X1,...,aL Considering more generally a sum S 1 are a totally recursive yields collection, of application independent (11.31) =
(1),(/)f=1 11(l)x(/).
( 11.33)
=
To investigate the distzibution of S, one need only establish the fonnulae linking the ch.f.s with the relevant c.d.f.s (or where appropriate p.d.f.s) which are known for the standard sampling distributions.
11.7 Example For the Poisson distribution (8.8),
9x(f',)
Ee
=
itX
)u
)
x
(j.
77w
--
e
=
;:
X!
e
i,
e
=
)k(eyy..j j .
(1 1.34)
n
11.8 Example In the standard Gaussian case, (8.10), 9z(J)
=
Ee
itLl
)
1 =
Completing the square yields itz
z(f)
=
e-
/2
+oo
/-2l
-
z2/2
-
1
+=
27:
x
e
itz
2
/2
-z
jf)2/2
-(z
=
dz.
-
(1 1.35) /2/2
-
and hence
-(z+ff)2 -/2
dz
e
-x
=
e
.
(1 1.36)
(The integral in the middle member has the value ulV for any choice of t, note.) Accordingly, consider X = cZ+ po whose p.d.f. is given by (8.24).Using (1 1.30), we obtain
(xlt, . poc) Equation With g,
e/w-c2p/z
.
(11.22) can be used to verify the moment =
(11.37)
.
0 the ch.f. is real, reflecting
formulae given in 9.4 and 9.7.
the symmetry of the distribution. n
11.9 Example The Cauchy distribution (8.11)has no integer moments. The ch.f. turns out to be e -lfI w hich is not differentiable at t = 0, as (11.22) would lead us to expect. The ch.f. for the Cauchy family (8.15)is #x(J;
V, : )
=
e
frv-lfl .
E1
The ch.f. is also defined for multivariate disibutions. (??zx 1) the ch.f. is
(j j ag; .
For a random vector X
Probability
168
expfif'fl),
#(f)
=
(11.39)
where t is a v-vector of arguments. This case will be especially important, not least because of the ease with which, by the generalization of (11.30), the ch.f. +.d (kx 1), can be derived for an affine transformation of the vector. Let F BX where B (kx m) and d k x 1) are constants, and then we have =
g(/)
F(exp(ff'F))
=
exp(I'/'#)F(exp(//'#.Y))
=
=
expfit'dklxlB't).
(11.40)
11.10 Example Let X m x 1) be multinormal with p.d.f. as in (8.36).
ch.f. is
rrhe
#x(f;M,E)
1
=
+=
rn/z
(2a) 1x j = exp l it'p
-
+= ...
1/2
-.
expl/'.t
g)'E-1(x
@
-
-
..
-
M)l#-4r
rlfzy.z )
(1 1.41)
.
The second equality is obtained as before by completing the square: 1 g(x it 'x j/zD ,1(x g j&)I-1(x g j&) itp g,)'E-l (x p) -
where it can be shown mI1
(2,n) Ix j
1/2
=
-
-
-
-
-
-
-
-
,
of the last term integrates over R to
that the exponential
n
.
1 1.4 The lnversion Theorem Paired with (11.8) is a unique inverse transformation hom #4/)to F@), so that the ch.f. and c.d.f. are fully equivalent representations of the distribution. The clef step in the proof of this proposition is the construction of the inverse transfonnation, as follows.
11.11 Lemma If
9(f)is defined
Fb)
-
F(J)
by
1 1im 2a F->x
=
(11.8), then yn
--
e
ittl
--
e
-
itli
(dt
it
..w
for any pair a and b of continuity points of F,' with a generalization of this formula is AF(xl
,...,.q)
=
1 m
za
1im T-+x
z'
z' ...
-w
-z
k
-jx./
e
J''I
jzzj
-
e
(11.42) <
b. The multivariate
-j.%+Ax.j)
jjy
(1 1.43) xtxj...xktt,---ntktdtb---dtk. wher AF(x1,...,.q) is defined in (8.31)and the vertices of the rectangle based at the point x1,...,ak, with sides Lxj > 0, are a11continuity points of F. n --
1.,
wxza..1
xrolon:
f.n
/1 1 q0'
Tt onn
lw
xzorifiofl
Ilqin
e (11
10
thnt
Characteristic
169
-itb
-ita
d
Functions
-d
--: b a as t
0.
-->
-
it
The integrals in (11.42)and (11.43) are therefore well defined in spite of including the point t 0. Despite this, it is necessary to avoid writing (11.42) =
as '-ita
Fb)
F(J)
-
'-itb
+x e
=
e
-
(11.44)
(f)Jf,
lnit
because the Lebesgue integral on the right may not exist. Forexample, suppose the a'it.j j random variable is degenerate at the point 0; this means that (/) e and =
F
1 21
-
e
ita
,
-itb
e
-
1 r j. 7: 1 t
log F,
--dt
dt k
it
-w
=
-
-
( 11.45)
so that the criterion for I rbesgue integrability over (-x,+x) fails. However, the limits in (11.42)and (11.43) do exist, as the proof reveals. Proof of 11.11 Only the univariate case will be proved, the multivariate extension being identical in principle. After substituting for () in (11.43) we can interchange the order of integration by 9.32, whose continuity and a.s. boundedness requirements are certainly satisfied here: '-itb
-ita
'
d
r
dt
lnit
=
= Using
'-itb
-ita
F
-d
e
e
-
-F
21f/
+x
T
-x
-F
d
oo
-x
itx-a)
j/x
e dF@) dt
itx-b)
-e
t dF(x)
lxit
.
(1 1.46)
(11.10.), ' e itx-a)
-.w
-
e
itfx-b)
t
lxit
z'Sj n yx
=
F sjn j(x
.u)
t
'lt
()
-
()
zt
.y)
t,
(1 1.47)
noting that the cosine is an even function, so that the terms containing cosines x (which are also the imaginary terms) vanish in the integral. The limit as F standard fonuula of this expression is obtained from the -..j
jo'='sin
t t
dt
=
zIl,
tx
()
(:
,
-1/2, tx
>
0
.()
<
(j j.4g) 0.
Probability
170
yields the result
(11.46)
Substituting into l.x
e
fftx-a)
il(2'-8
-d
t
lxit
=
0, x
<
a or x > b
l 7:
=
c or x
<
x < b.
x
1, a
b
=
(1 1.49)
Letting F-->x in (11.46) and applying the boundedconvergence theoremnow gives -ita
lim
F-px
T -
-itb
e
-
+x
e
(tldt
lgit
w
-x
(lfc) +
= r1(F(:)
1(s) + ta,bdFx)
F(b-)
+
Fa)
-
-
F(J-)),
(11.50)
which reduces to F(b4 Fa) when a and b are continuity points of F. w -
Lemma 11.11 is the basic ingredient of the following key result, the one that primarily justifies our interest in characteristic functions.
theorem Distributions having the same ch.f. are the same.
11.12 Inversion
Proof We give the proof for the unvariate case only. By (11.42), the c.d.f.s of the two distributions are the same at every point which is a continuity point of both c.d.f.s. Since the set of jump points of each c.d.f. is countable by 8.3 their union is countable, and it follows by 2.10 that the set of continuity points is dense in R. It then follows by 8.4 that the c.d.f.s are the same everywhere. . A simple application of the inversion theorem is to provide a proof of a well known result, that affine functions of Gaussian vectors are also Gaussian.
11.13 Example Let,f Ng,X) m x 1) and F= BX d n x 1) are constants. Then by (11.42),
-d
n
-
#g(f)
=
exptff'#lFtexpt#'sfll
= expfit'Bg If ranktS'xS)
n,
=
(implyingn -(y
.f(J)
expf
=
x 1) where B n x m) and
-
Bg
+d)
-
qt'B'SBtj
(11.51)
.
< m), 11.12 implies that F has p.d.f. -
d) '(#xn')-l(y tz I#IN
(2r)n
,
1/a
I
-Bg
-#))
.
(1 1.52)
< n, (11.51) remains valid although (11.52) is not. But by the same lf rankt#xs') arguments, every linear combination c'F, where c is p x 1, is either scalar Gaussian with variance c'BSB'c, or identically zero, corresponding to the cases B'c 0 respectively. ln this case F is said to have a singular 0 and #'c Gaussian distribution, I::I :#
=
Characteristic
Functions
171
11.5 The Conditional Characteristic Function Let F be a F-measurable r.v., and 1et V c @. The conditional ch.f. of 4)ysg(f), is for each t a random variable having the property
jgtrj
g(/)##
jaeitp,
all G e V.
=
FIV,
(11.53)
The conditional ch.f. shares the properties of the regular ch.f. whenever the theory of conditional expectations parallels that of ordinary expectations according to the results of Chapter 10. Its real and imaginary pms are, respectively, the V-measurable random variables Ftcos /F1 V) and fsin fFl N). It can be expanded as in 11.6, in tenns of the existing conditional moments. lf X is V-measurable, the conditional ch.f. of X+ F is #x+gIv(/) e itxpjeityks; by 10.10. And if i' is Vl-measurable and N and V1 are independent subfields, then =
#i'l5'(J)
l'(/)
=
a.S.
The conditional ch.f. is used to prove a useful inequality due to von Bahr and Essen (1965).We start with a technical lemma which appears obscqre at first sight, but turns out to have useful applications.
11.14 Lemma Suppose
FIZIr
<
0
cx),
<
2. Then
<
r
A'(rlj..+x1 Re(#z(/)) -
Fl W here
K
=
zl
r
=
l+rfJfl-l
(J+*(1cos ?g)/1 u I -
=
-x
(1 1.54)
t
1+r
jyj
a-1r'(r+ j) sin ra/2.
n
The last equality, with F(.) denoting the gamma function, is a standard integral formula for 0 < r < 2. Proof The identity for real z,
lz lr
+oo
=
1 cos zt dt, 1+,
Ar)J.. -
(1 1.55)
jjj
is easily obtained by a change of variable in the integral on the right. The lemma Acos tZ4. . follows on applying 9.32 and noting that Re(9z(/)) =
Tl'lis equality also holds, for ) e C with PC) = 1, if F( 1Zl rl N)((t))and 9zls(J)()) r are substituted for Fl zl and #z(f).In other words, the conditional rth moment and conditional ch.f are almost surely related by the same formula. So consider fv-bounded r.v.s z and X, where Z is T-measurable, and X is N-measurable for N c 5. Suppose that #z!v(/)is a real r.v. almost surely. Then for each e f1,
1
-
Re@x+zIs(f))()
=
1 Retd -
itX
#zIs(f))())
= 1 (cosJX())(zIg(/)()) -
S
(1
-
cos /X())
+
(1
-
9zIs(J)()),
(1 1.56)
Probability
172 thedifference between whichis non-negative
the last two members being (1 cos /X()))(1 for all Hence, for 0 < r < 2, -
#z1g(/)()))
-
.
+x 1 Re@x+zIs(/))
#(r)J.. += f #(r)J.. Ar)J.. -
E IX+
zl
rl
N)
=
t
j+r
jjj
1 cos tx
+=
-
1.r
jyj
l#l
=
'
+
t
1
zl v(/)
-
+
jyj
t
j+,
Izl r IV), a-s.
E
(1 1
.57)
and taking expectations through yields
FIA-+zIr < FIXIr
f'l
+
ZIr.
(11.58)
For the case 0 < r S 1 this inequality holds by the cr inequality for general Z and X, so it is the case 1 < r < 2 that is of special interest here. Generalizing from the remarks following (11.31), the condition that (zIv(/) be real a.s. can be fulfilled by letting Z Y- Y', where F and F' are identically distributed and independent, conditional on V. Note that if R = c(F), then =
F(F'IX)
F(F'IT),
=
(11.59)
a.s.,
by 10.31. Identical conditional distributions means simply that Fv(. 1V) Fg(. IV) a.s., and equivalently that (gIs(/) #glv(/) a.s. Hence
=
=
-fry'
#r-r'1p(/)
=
ffye
Ee
ls)
= Ee itvls )eje-itvj v) IrIs(/)l 2 a.s.,
=
(j j
,
where the right-hand side is a real r.v. Now, for each identity can be verified:
2(1
Re(#y,v(/)())
-
Applying
=
1
-
la + Il
g1s(/)()
-
(t)
-
6o)
e f1, the following
9rIs(f)((.0)
2
I
(11.61)
.
and 11.14, and taking expectations, this yields the inequality
(11.60)
2A'1FIr
r,
Fl Y-
F'I
0
<
<
r
(11.62)
2,
noting that the difference between the two sides here is the non-negative g(/) 2/j J'j 1+rJ/ Of r #(r)J+=F1 1 9y.1 l These arguments lead us to the following conclusion.
function
-
,
.
..x
11.15 Theorem Suppose F(l'IV) = 0 a.s. and X is V-measurable where V i R
c(F), and both variables
FI.Y+ F1r
=i
are Lr-bounded. Then
r s FIXI
+
2FIFIr,
0
r
(11.63)
2.
Proof Let F' be independent and identical with F, conditional on V. Applying (11.59), these conditions jointly imply f(F'I R) &F'I V) F(FI9') 0. Noting =
=
=
Characteristic
Functions
173
that X+ F is A-measurable, it follows by 10.19 (in applying this result be careful to note that R plays the role of the subfield here) that E IX+
r
FI
<
FIX+ (i'- r) I
'.
(11.64)
The conclusion for 1 < r < 2 now follows on applying (11.58)for the case Z F- i'', and then (11.62). The inequality holds for 0 < r f 1 by the cr inequality, and for r 2 from elementary considerations since Elk'.,Y'I0. In these latter cases the factor 2 in (11.63) can be omitted. . =
=
=
This result can be iterated, given a sequence of r.v.s measurable on an increasing for sequence of c-tields. An easy application is to independent rov.s A'1,...,aL, 2,...,n. 0 certainly holds for t which the condition E(Xt Ic(-Y1,...,m-1)) ntzzkxt, and 11.15 c(X1,...,Xn) yields Letting Sn c(&) =
=
=
=
E IsnIr < A'l
'n-t
Ir
+
< 2XA'ImI r,
2Fl Xn Ir
0 < r < 2.
(1 1.65)
>1
If the series on the majorant side converges, this inequality remains valid as n --> x. lt may be contrasted for tightness with the cr inequality for general Xt, (9 62). In this case, 2 must be replaced by nr-1 for 1 < r S 2, which is of no use for large n. .
lll THEORY OF
STOCHASTIC PROCESSES
12 Stochastic Processes
12.1 Basic Ideas and Terminology Let (( 5 #) be a probability space, let 'T be any set, and let RV be the product space generated by taking a copy of R for each element of 'F. Then, a stochastic process is a measurable mapping x: f r--yR'T where ,
,
j
xttDl
=
l-V((,)), ':
(12.1)
e
'T is called the index Jtrf, and the r.v. Xs() is called a coordinate of the process. A stochastic process can also be characterized as a mapping from x'T to R. However, the significant feature of the definition given is the requirement of joint measurability of the coordinates. Something more is implied than having Xs(() a measurable nv. for each 1. Here, 'Ir is an arbitrary set which in principle need not even be ordered, although linear ordering characterizes the important cases. A familiar example is T' ( 1,...,k) where x is a random k-vector. Another important case of 7 is an interval of R, such that x()) is a function of a real variable and R 'T the space of random functions. And when 'T is a countable subset of R, x fXv()), G 7-) defines a stochastic sequence. Thus, a stochastic sequence is a stochastic process whose index set is countable and linearly ordered. When the Xz represent random observations equally spaced in time, no relevant information is lost by assigning and a linear ordering through ENor Z, indicated by the notations f-V()))''*-x.The definition does not rule out 1- containing information about distances between the sequence coordinates, as when the observations are irregularly spacd in time with a real number representing elapsed time from a chosen origin, but cases of this kind will not be considered explicitly. Fnmiliarly, a time series is a time-ordered sequence of observations of (say) economic variables, although the term may extend to unobserved or hypothetical variables, such as the errors in a regression model. Time-series coordinates are labelled t. If a sample is defined as a time series of finite length n (or more generally, a collection of #uch series for different variables) it is convenient observato assume that samples are embedded in infinite sequences of tions. Various mathematical functions of sample observations, statistics or estimators, will also be well known to the reader, characteristically involving a summation of terms over the coordinates. The sample moments of a time series, regression coefficients, log-likelihood functions and their derivatives, are standard examples. By letting n take the values 1,2,3,..., these functions of n observationsgeneratewhatwemay call#crfvdtfsequences.The notionof asequence =
,
':
=
(.V((t)JO
':
dpotential'
Theor.y of Stochastic Processes
in this case comes from the idea of analysing samples of progressively increasing size. The mathematical theory often does not distinguish between the types of sequence under consideration, and some of our definitions and results apply generally, but a clue to the usual application will be given by the choice of index symbol, / or n as the case may be.
A leading case which does not fall under the detinition of a sequence is where 'T is partially ordered. When there are two dimensions to the observations, as in a panel data set having both a time dimension and a dimension over agents, x may be called a random .#e/J. Such cases afe not treated explicitly here, although in many applications one dimension is regarded as tixed and the sequence notion is adequate for asymptotic analysis. However, cases where 'T is ither the product set ; x(N, or a subset thereof, are often met below in a different context. A triangular stochastic array is a doubly-indexed collection of random variables,
X1l Xz1 X51 .Yla X22 X51
11l
(12.2)
X3k1
a=l is some increasing integer compactly written as ( (Akmlmil ) n=1, w he r f/cn )'=' sequence. Array notation is called for when the points of a sample are subjected to scale transformations or the like, depending on the complete sample. A standard Z7=1Var(X,), or some where Xnt = Xtlsn, and sn example is ( lA-n,))=1)=n=j, similar function of the sample moments f'rom 1 to n. e'D
=
12.2 Convergence of Stochastic Sequences Consider the functional expression (Ak())
';
for a random sequence on the space L1,5,P). When evaluated at a point (l) e this denotes a realization of the actual collection real is of numbers generated whenthe outcome sequence, the drawn. lt is natural to consider in the spirit of ordinal'y analysis whether this e f2, we sequepce converges to a limit, say Al. Tf this is the case for every would say that Xn Xsurely (orelementwise) where, if Xnis an T/f-measurable nv. for each n, then so is X, by 3.26. But, except by direct construction, it is usually difficult to establish in terms of a given collection of distributional properties that a stochastic sequence converges surely to a limit. A much more useful notion (becausemore easily shown) is almost sure convergence. Let C ? fl be the set of outcomes such --y
Stochastic Processes
179
.Y()) as n that, for every (t) e C, Xn) x. If #(C) 1, the sequence is said almost surely, or equivalently, with probability one. The notations Xn to converge -->
-->
=
-F.&:
X a.s., and a.s.lim-,, X are al1 used to denote almost sure convergence. A similar concept, of convergence almost everywhere (a.e.),was invoked in connection with the properties of integrals in j4.2. For many purposes, almost sure convergence can be thought of as yielding the same implications as sure convergence in probabilistic arguments. However, attaching probabilities to the convergent set is not the only way in which stochastic convergence can be understood. Associated with any stochastic sequence are various non-stochastic sequences of variables and functions describing aspects of its behaviour, moments being the obvious case. Convergence of the stochastic sequence may be defined in tenns of the ordinary convergence of an associated sequence. If the sequence fExn A'')217converges to zero, th ere is a clearly a sense in which Xn X.,this is called convergence in mean square. Or suppose that for any 6: > 0, the probabilities of the events ( ): IX,,() X()) l < :1 e 5 form a real sequence converging to 1. This is another distinct convergence concept, so-called convergence in probability. In neither case is there any obvious way to attach a probability to the convergent set; this can even be zero! These issues are studied in Part lV. Another convergence concept relates to the sequence of marginal p.m.s of the coordinates, fgwl!', or equivalently the marginal c.d.f.s, (F,,l7. Here we can consider conditions for convergence of the real sequences fgwtAlITfor various sets A e or alternatively, of (F,,(x) J7 for various x e R ln the latter case, uniform or pointwise convergence on R is a possibility, but these are relatively strong notions. It is sufficient for a theory of the limiting distribution if convergence is confined just to the continuity points of the limiting function F, or equivalently (as we shall show in Chapter 22) of gwtAl, to sets A having g,tAl = 0. This condition is referred to as the wdck convergence of the distributions, and forms the subject of Part V. X, or Xn
-->
=
-
-->
-
,
.
12.3 The Probability Model Some very important ideas are implicit in the notion of a stochastic sequence.
Given the equipotency of INand 1, it will suffic to consider the random element (A((l)))T, e f1, mapping from a point of to a point in infinite-dimensional Euclidean space, R=. From a probabilistic point of view, the entire infinite sequence corresponds to a single outcome ) of the underlying abstract probability space. ln principle, a sampling exercise in this umework is the random drawing of a point in R=, called a realization or sample path of the random sequence', we tnay actually observe only a tinite segment of this sequence, but the key idea is that a random experiment consists of drawing a complete realization. Repeated sampling means observing the same tinite segment (relativeto the origin of the index set) of different realizations, not different segments of the same
realization. The reason for this characterization of the random experiment will become clear
18O
Theot'y of Stochastic Processes
in the sequel; for the moment we will just concentrate on placing this slightly outlandish notion of an intinite-dimensioned random element into perspective. To show that there is no difficulty in establishing a correspondence between a probability space of a familiar type and a random sequence, we discuss a simple example in some detail. 12.1 Example Consider a repeated game of coin tossing, generating a random sequence of heads and tails; if the game continues for ever, it will generate a sequence of infinite length. Let 1 represent a head and 0 a tail, and we have a random sequence of ls and 0s. Such a sequence corresponds to the binary (base2) representation of a real number; according to equation (1.15) there is a one-toone correspondence between infinite sequences of coin tosses and points on the unit interval. On this basis, the fundamental space (f,T) for the coin tossing experiment can be chosen as ((0,1),S(0,l)).The form of P can be deduced in an Pttails) 0.5 (i.e.the coin elementary way from the stipulation that Ptheads) is fair) and successive tosses are independent. For example, the events ftails on first toss ) and lheads on Erst toss) are the images of the sets (0,0.5)and g0.5,1)respectively, whosemeasures must accordingly be O.seach. More generally, the probability that the first n tosses in a sequence yields a given configuration 1/2N, of heads and tails out of the ln possible ones is equal in every case to so likely'. The that each sequence is (in an appropriate limiting sense) corresponding sets in (0,1)of the binary expansions with the identical pattern of =
=
tequally
Os and ls in the tirst n positions occupy intervals al1 of width precisely $l2n in the unit interval. The conclusion is that the probability measure of any interval is equal to its width. This is nothing but Lebesgue measure on the half-open interval g0,1).n This example can be elaborated from binary sequences to sequences of real variables without too much difficulty. There is an intimate connection between infinite random sequences and continuous probability distributions on the line, and understanding one class of problem is frequently an aid to understanding the other. The question often posed about the probability of some sequence predicted in advance being realized, say an infinite run of heads or a perpetual alternation of heads and tails, is precisely answered. In either the decimal or binary expansions, all the numbers whose digit sequences either tenuinate or, beyond some point, are found to cycle perpetually through a finite sequence belong to the set of rational numbers. Since the rasionals have Lebesgue measure zero in the space of the reals, we have a proof that the probability of any such sequence occurring
is zero.
Another well-known conundrum concerns the troupe of monkeys equipped with typewriters who, it is claimed, will eventually type out the complete works of Shakespeare. We can show that this event will occur with probability 1. For the sake of argument, assume that a single monkey types into a word processor, and his Ascll-encoded output takes the form of a string of bits (binarydigits). Suppose Shakespeare's encoded complete works occupy k bits, equivalent to kl5 characters allowing for a 3z-character keyboard (upper-caseonly, but including som
Stochastic Processes
181
punctuation marks). This stling is one o f the lk p ossible strings of k bits.
Assuming that each such string is equally likely to arise in k/5 random key presses, the probability that the monkey will type Shakespeare without an error from scratch is exactly 2 -k H ow ever the probability that the second string of lk bits it produces is the right one, given that the tirst one is wrong, is (1 2 '-k)2- w h en the strings are independent. In general, the probability that the monkey will type Shakespeare correctly on the m + llth independent attempt, '&2A11 these events given that the first m attempts were failures, is (1 2 -k) and summing k disjoint, probabilities a11 their 0 yields m over are 1. fmonkey types Shakespeare eventually) ,
.
-
-
.
=
In the meantime, of course, the industrious primate has produced much of the rest of world literature, not to mention a good many telephone books. It is also advisable to estimate the length of time we are likely tcj wait for the desired text to appear, which requires a further calculation. The average waiting time, l-k4m expresse d in units of the time taken to type k bits, is 2-E,1=1rrl(1 2k 1 If we scale down our ambitions and decide to be content with just $TO BE OR NOT TO BE' (5x 18 90 bits), and the monkey takes 1 minute over each attempt, we shall wait on average 2.3 x 1021years. s o the Complete Works don't really bear thinking about. What we have shown is that almost every infinite string of bits contains every t'inite string somewhere in its length', but also, that the mathematical concept of talmost surely' has no difficulty in coinciding with an everyday notion The example is frivolous, but it is useful to be indistinguishable from reminded occasionally that limit theory deals in large numbers. A sense of perspective is always desirable in evaluating the claims of the theol'y. =
-
-
.
=
'never'
.
The first technical challenge we face in the theory of stochastic processes is to handle distributions on R=. To construct the Borel field 000of events on R7 we implicitly endow R= with the Tychonoff, or product, topology. It is not essential to have absorbed the theory of j6.5 to make sense of the discussion that follows, but it may help to glance at Example 6.18 to see what this assumption implies. Given a process x f.il7 we shall write jz 7:k(x) (m,...,A): Rx F-> (jyk =
(
=
.
a)
for each k e EN,to denote the k-dimensional coordinate projection. Let C denote the collection of jnite-dimensional cylinder sets of R=, the sets C
=
(x e
In other words, elements
: ;u@) R oo
e E, E e Bk k e ,
s)
(124) .
of C have the fonn C
=
-1 (F) C:k
(1z 5) .
for some E e T k an d some finite k. Although we may wish to consider arbitrary finite dimensional cylinders, there is no loss of generality in considering the projections onto just the first k coordinates. Any tinite dimensional cylinder can ,
182
Theor.y of Stochastic Processes
be embedded in a cylinder of the form lk -1(F), E e f where k is just the largest of the restricted coordinates. The distinguishing feature of an element of C is that at most a tinite number of its coordinates are restricted. ,
C is a tield.
12.2 Theorem
Proof First, the complement in R= of a set C defined by (12.4)is a-1(Ff) C c = fx e R= : (ak(x)e Ec E e fl k =
,
,
(126) .
which is another element of C, i.e. Cc s C. Second, consider the union of sets C a-1(F') -1 k e C for E, E' G Bk C tp C' is given by (12.4) = Jck E) e C an d C' with E replaced by E tp E ' and hence Ck.l C' e C Tlzird if E e Bk and E' e Bmfor m > k, then Ex? m-ke Bm an d so the argument of the second case applies. . =
,
,
.
.
,
,
Fig. 12.1 It is not easy to imagine sets in arbitrary numbers of dimensions, but good visual intuition is provided by thinking about one-dimensional cylinders in R3 Letting @,y,z)denote the coordinate directions, the one-dimensional cylinder generated by an interval of the axis is a region of 3-space bounded by two infinite planes at right angles to the x axis (see Fig. 12. 1 for a cut-away representation). A union of x-cylinders is another x-cylinder, a collection of But the union and intersection of an x-cylinder with parallel y-cylinder are two-dimensional cylinder sets, a cross' and a column' respectively (see Fig. 12.2). These examples show that the collection of cylinder sets in R k for fixed k is in R3 is a not a field,' the intersection of three mutually orthogonal bounded not a cylinder set. The set of tinite-dimensionalcylinders is nbt closed under the operations of union and complementation (andhence intersection) except in an infinite-dimensional space. This fact is critical in considering c(C), the class obtained by adding the countable unions to C. By the lastmentioned property of unions, c(C) includes sets of the form (12.4)with k tending to infinity. Thus, we have the following theorem. .
.x
<walls'
.
twalls'
Kcube',
Stochastic Processes
12.3 Theorem c(r) ogy. I:l
=
183
f=, the Borel field of sets in R= with the Tychonoff topol-
Fig. 12.2 The condition of this result is something we can take for granted in the usual applications. Recalling that the Borel field of a space is the jmallest c-field containing the open sets, 12.3 is true by definition, since C is a sub-base for the product topology (see j6.5) and a11the open sets of R= are generated by unions and finite intersections of C-sets. To avoid explicit topological considerations, the reader may like to think of 12.3 as providing the definition of f=. One straightforward implication, since the coordinate projections are continuous mappings and hence measurable, is that, given a distribution on @=,=), finite collections of sequence coordinates can always be treated as random vectors. But, while this is obviously a condition that will need to be satistied, the real mtypractical method we have of defining distribproblem runs the other way. utions for infinite sequences is to assign probabilities to finite collections of coordinates, after the manner of j8.4. The serious question is whether this can be done in a consistent manner, so that in particular, there is one and only one p.m. on (R7S=) that corresponds to a given set of the finite-dimensional distributions. The affirmative answer to this question is the famous Kolmogorov consis'l'he
tency theorem.
12.4 The Consistency Theorem The goal is to construct a p.m. on @*,f*),and, following the approach of j3.2, the plausible first step in this direction is to assign probabilities to elements Of C Let p denote a p.m. on th space (R,f$, for k 1,2,3,.... We will say that this family of measures satisfies the consistency property if =
.
gt' )
=
g'-t'
X
R
m-k
)
(12.7)
Theory of Stochastic Processes
184
for E G Bk and al1 m > k > 0. ln other words, any k-dimensional distribution can be obtained from an v-dimensional distribution with m > k, by the usual operation of marginalization.
The consistency theorem actually generalizes to stochastic processes with uncountable index sets 1- (see 27.1) but it is sufficient for present purposes to consider the countable case. consistency theorem Suppose there exists a family of p.m.s (gkJwlich satisfy consistency condition (12.7).Then there exists a stochastic sequence (m,t e N J on a probability space @=,f=,g,)such that p is the p.m. the of coordinate functions (.L,...,Xk)'. u vector of tinite
12.4 Kolmogorov's -'r
=
The candidate
measure for
is defined for sets in C by
.x
FIC)
=
(12.8)
11,1(6-),
where C and E are related by (12.4).The problem is to show that g, is a p.m. on C. If this is the case, then, since C is a field and f= = c(C), we may appeal to the extension theorem (3.8+3.13)to establish the existence of a unique measure on (R*,%=) which agrees with g for all C e C. The theorem has a simple but important corollary. C is a determining class for (R7f=).
12.5 Corollary
E1
ln other words, if g and v are two measures on (R=,f=) and jtk finite k, then jt v.
vk for every
=
=
To prove the consistency theorem we require a technical lemma whose proof is beyond us at this stage. It is quite intuitive, however, and will be proved in a more general context as 26.23.
12.6 Lemma For every E G T an d such that pkE A') < n
0 there exists K, a compact subset of E,
>
-
.
In other words, a p.m. on the space (R Bk) has nearly all of its mass confined to a compact set; this implies in particular the proposition asserted in j8.1, that random variables are finite almost surely. ,
Proof of 12.4 We will verify that jt of (12.8)satisfies the probability axioms with respect to elements of C. When E Rk C R= so that the first two probability axioms, 7.1(a) and (b), are certainly satistied. To establish finite additivfor E e Tk E' e Tm and ity, suppose we have C-sets C gk-1(AD, an d C' zr-1(F') m m k k. If C and C' are disjoint, =
,
=
=
p,(O+ g,tC ') g( =
F) + jt m (F')
= gmE x
=
qvl-J7
=
tp E
,
pmEx R'&-$ + p,m(A'') ,
)
=
g,(c
kp
c')
,
(12.9)
where the second equality applies the consistency condition (12.7),and the third m-k and E z are disjoint if C and C z are. one uses the fact that E x R The remaining, relatively tricky, step is to extend tinite additivity to count-
Stochastic Processes
185
able additivity. This is done by proving continuity, which is an equivalent property according to 3.5. lf and only if the measure is continuous, a monotone 1 C or G' C has the property, g,(G) p,IC). sequence tCj e C ) such that CJ Since Cj Cc where g(C9 = 1 jttt7), it is sufficient to C implies consider the decreasing case. And by considering the sequence Cj C there also is no loss of generality in setting C 0, so that continuity implies g,tGl 0. To prove continuity, it is sufticient to show that if g(() 2 6: for some E > 0, for every j, then C is nonempty. lf C e C for some j k 1, then p,(G)= pkqlEj) for some set Ej G Bkct where ? /cU) ls the dimension of the cylinder Cj. By consistency, pkqlEp '1-
.
--
.1e
'l'
-
-
=
--
=
pmEj X R '''-U)
so there is no loss of generality in assuming ) for any m > 17U), kq) < < k41) < k42) We may therefore define sets F1 e LB that lU) < 1,...,./, by setting E) A) and ...
....
=
,
=
1.
=
Ei xR ko-k
is a decreasing sincefc/),t,-1
i
,
1
=
j
,...,
-
1.
(12.j())
euhsets(A1).t,-1,
sequence, so is the sequence of
for each j k 1.
Consider any tixed j. There exists, by 12-6, a compact set Kj bkkqfEj Detine the sets K1
=
E/2 y+1
&')<
-
c
Ej such that
(!g
.
.
! !)
Kix R kil-ki) e lk) by analogy with the A''1,and so define Fj
=
Bkct
(12. 12)
s)s c
( 12. 13)
O#t; e
.
'=1
Fj i E), and hence Dj ? Cj where Dj
=
lk -1
./
.
Applying 1.1(iii) and then 1.1(i), observe that Ej Fj -
=
Ej ch
./
j
K'ic
=
-
i=1
f=1
Ej K*i)
j
c
E*i -
f=1
r*f)
(12.14)
,
is decreasing. Hence where the inclusion is because the sequence tA'tl)'J.1 bkkitEj F./) f -
-
f=1
=
bklilE t, A-t,)
bkEi
-
Kit
< E/2.
(12.15)
f=1
The first inequality here is from (12.14) by finite subadditivity, which follows from finite additivity as a case of 3.3(iii). The equality applies consistency, and the secon (i inequa lity applies the summation of 2-f-1 Since Ej Fj and Fj are = disjoint and got.l g,(G)> : by assumption, it follows from (12.15) that bklh) pkqlFj) > e/2, and accordingly that Dj is nonempty. Now. we construct a Doint of C. Let Ixlij. i e (N1 denote a seauenc of ooints -
.
=
F/let??'yof Stochastic Processes
186
of R=
with
x(J) e
Dj for each j, so that
(X1U),...,Xw)U))lk(j)@U)) =
Note that for m
=
(12. 16)
Fj.
1,...,./,
1 Km, (XlU),...,Akm)U))'&(,?,)(xU))
(12.17)
=
by (12.12), where Kmis compact. Now 1et m be tixed. Our reasoning ensures that (12. 17) holds for each.j k m. A set in Rkm' is compact if and only if each of the coordinate sets in R is compact, so consider the bounded scalar scquences, fxibij, j r?z) for f 1,...,k4-). Each of these has a cluster point A'1, and we can use the diagonal method (2.36)to construct a single subsequence (Al with the = A': for each 1,...,k(-). By the compactness, property that Xijn) e Km i Em. This is true for every m e IN (X1,...,X(m)) Consider the point a7*e R= defined by 7v(,,,)(x*) for each m e N. (X1,...,X(m)) = Since x* e Cm for each m, we have x* G O'>;=ICk C, as required. . =
.-+
.
=
This theorem shows that, if a p.m. satisfying (12.7)can be assigned to the finite-dimensional distributions of a sequence m x is a random element of a We shall often wish to think of @=,f7jt) as derived probability space from an abstract probability space (f,T,#), and then we shall say that x is ble if F1(A>)e 5 for each event E e S*. This statement implies the 5lB coordinates Xt are F/f-measurable r.v.s for each /, but it also implies a great deal more than this, since it is possible to assign measures to events involving countably many sequence coordinates. ('R*,fO,g,).
''O
-measura
12.5 Uniform and Limiting Properties Much the largest part of stochastic process theory has to do with the joint distributions of sets of coordinates, under the general heading of dependence. Before getting into these topics, we shall deal in the rest of this chapter with the various issues relating exclusively to the marginal distributions of the coordinates. Of special interest are conditions that limit the random behaviour of a sequence as the index tends to infinity. The concept of a unform condition on the marginal distributions often plays a key role. Thus, a collection of rov.s tX,r, T e is said to be uniformly bounded in probability if, for any 8 > 0, there exists Bg < x such that -1)
#( IXzI sup ' It is also said to be unjrmly
)<
Lp-bounded for p
supI1ALll?,B <
<
=.
(12. 18)
:. >
0, if
(12.19)
'
For the case p = x, (12.19)reduces to the condition, supmlXzI < x a.s. ln this case we just say that the sequence is uniformly bounded a.s. For the case p = 1, we have sups f'IXz I < x and one might think it conrct to refer to this property as integrability' Unfortunately. this term is already in use for a ,
Suniform
,
Stochastic Processes
different concept (see the next section) and so must be avoided here. Speak of tuniform fal-boundedness' in this context. To interpret these conditions, recall that in mathematics a property is said to hold unifonnly if holds for all members of a class of objects, including the limits of any convergent sequences in the class. Consider the case where the /. Random collection in question is itself a sequence, with 'T = E$ and with probability 1, and for each finite t e IN,#( 1Al Be variables are finite x, > holds for for The < always point of unifonn Bet 0. : bound is some any a <s to ensure that the constants Bet are not having to get larger as t increases. p > 0, by Liapunov's inequality. Also, uniform fv-boundedness for anyp > 0 implies uniform boundedness in probability; the Markov inequality gives 'r
=
tbounded
,
-->
-x).
-->
,
#(IAI
B4
(12.20)
terminology we sometimes speak of fvboundedness in the case of (12.18). A standard shorthand (dueto Mann and Wald 1943b) for the maximum rate of (positive or negative) increase of a stochastic sequence uses the notion ot Oh' notation unifonu boundedness in probability to extend the tBig Oh' and real E lf, for exists ordinary 0, > Be < x such for there sequences (see j2.6). that the stochastic sequence fXn )T satisfies supa't 1XnI > B < E, we write Xn 0p(1). lf (L)T is another sequence, either stochastic or nonstochastic, and XnlYn = 0p(1) we say that Xn Op(L), or in words, &Xnis at most of order Fn in x, 0 as n tN(1); more we say that Xn Probability' If #( IXnI > :) t7p(1), when words, in GXn is of order lss than Xnlk'n generally Xn o (L') or r l'n in probability The main use of these notations is in manipulating small-order terms in an expression, without specifying them explicitly. Usually, Fn is a positive or negative power of n. To say that Xn tV1)is equivalent to saying that Xn converges in probability to zero, following the terminology of j12.2. Sometimes Xn Op(1) is defined by the condition that for each s > 0 there exists Be < x and an integer Ng > 1 such that #( IXnl > B < s for a1l n Nz. But Xn is finite almost surely, and there necessarily exists (this is by 12.6) a constant BL< #E') < E for 1 f n < Ne. For al1 possib1y 1arg er than Be, such that #( IXn l > practical purposes, the formulations are equivalent. Ktaittle
=
=
--)
-->
=
.
=
=
.
=
=
=,
Iheor.y of Stochastic Processes
188
12.6 Uniform lntegrability lf a r.v. X is integrable, the contributions to the integral of extreme X values must be negligible. ln other words, if FIXI f(IX1 1(IxlzM))
-->
0 as M
(12.21)
-->
x.
However, it is possible to construct unifonnly Al-bounded sequences l#nl which fail to satisfy (12.21)in the limit.
12.7 Example Detine a stochastic sequence as follows: for n 1,2,3,... let Xn = 0 with probability 1 1/rI, and Xn = n with probability 1/a. Note that E( lXn l) = nln = 1 for every n, and hence the sequence is uniformly Al-bounded. But to have =
-
limE IXnIlt lxalA/l)
=
0
(12.22)
MM=
uniformly in n requires that for each e > 0 there exists Ms such that E lXnl 1( xajkvl) < e for al1 M > Mo uniformly in a. Clearly, this condition fails, for e: < 1, in view of the cases n > Me. n Something very strange is going on in this example. Although E(Xn4= 1 for any n, x. To be precise, we may show that Xn = 0 with probability approaching 1 as n 0 (see 18.15). The intuitive concept of expectation appears to fail when Xn faced with nv.s taking values approaching intinity with probabilities approaching --+
-613...:
Zero.
The unform integrability condition rules out this type of perverse behaviour in a sequence. The collection (Xz, $ e T'J is said to be uniformly integrable if
sup 1im F( lXx11( xvl
Te
7
M)
M-yx
)
=
0.
(12.23)
ln our applications the collection in question is usually either a sequence or an array. In the latter case, uniform integrability of (Xkf) (say) is defined by taking the supremum with respect to both t and n. The following is a collection of theorems on unifonn integrability which will tind frequent application later on; 12.8 in particular provides insight into why this concept is so important, since the last example shows that the conclusion does not generally hold without uniform integrability. 12.8 Theorem Let E(Xn)
--$
E (A').
be a unifonnly l.YnlT
integrable sequence. If Xn -6J- X, then
Proof Note that E iXn I) = E IXnl 1( !xnI
s A.T+
Ftlxa
+
E lXnl 1( IxaIkM))
l1( jx,,1kM)).
(12.24)
By choosing M large enough the second tenn on the right can be made uniformly small by assgmption, and it follows that FIXnj is uniformly bounded. Fatou's
stochasticProcesses
189
lemmaimplies that FIXl < x, and F(m exists. Define l',, I.L --YI so that i',z 0 a.s. Since Fn < IaLl+ l-Yl by the triangle inequality, 9.29 gives =
f 2F( IXnl 1( lxnlxv/all
F(i'n1(yn>M))
+
2F( lXl 1( IxI>v/2)).
The second right-hand-side term goes to zero as M integrable if (A%) is. We may write f(Fn)
'(l%1(yn<M))
=
+
--
,
.--.)
so
x,
(12.25)
f Fn) is
A'(L1(yn>v)),
uniformly
(12.26)
and by the bounded convergence theorem there exists, for any E > 0, Ne such that F(l%1(ya<>)) < E/2 for n > #:, for M < oo. M can be chosen large enough that F(L1( ys>>)) < :/2 uniformly in n, so that EYn) < E for n > Nv., or, since 6: is 0. But arbitrary, EYn) --.-)
EYnq
by the
lnodulus
F(I-L--YI) k
=
IF(Xn) -F(ml
(12.27)
inequality, and the theorem follows. w
The next theorem gives an alternative form for the condition in often more convenient for establishing uniform integrability.
(12.23)which
is
12.9 Theorem A collection fXs, e 1-) of nv.s on a probability space (f1,1,#)is uniformly integrable iff it is uniformly Al-bounded and satisties the following condition: V E, > 0, 3 8 > 0 such that, for E E 5, ':
sup (F41 Xzl 1s) l < z.
P(E4 S
(12.28)
': e 'T
Proof To show sufficiency, fix : > 0 and 5 1, Markov inequality for p
E
7. By Al-boundedness and the
=
E
#(1-V1k Af) f and for M large enough, (12.28), it follows since
lxzI < M
(12.29)
x,
#(l-YzI Af) S ,1
6, for any is arbitrary that
>
0. Choosing
lA'(lX,rl 1f Ixzlwzlll < E,
Sup
to satisfy
(12.30)
TeT
and (12.23)follows since
E is arbitrary. note that, for any E e 5 and
To show necessity, F(1 Xvl 1s)
=
'r
E
1-,
1s1( Ixz1<MJ) + .E'(I.YZI 1s1( lxvI&z)) l-Y'rI
E
< MPCEI+
F(1 -Ysl1( Ixzlkvl).
(12.31)
Consider the suprema with respect to of each side of this inequality. For e > 0, (12.23) impliej there exists M < t:'o such that 'r
1F( 1AL1 1s)l
sup T
Uniform Al-oundedness
s MPCE) +
r1:.
now follows on setting F
=
f, and
(12.32) (12.28)also follows
Theor. of Stochastic Processes
190 with 8
<
zllM.
*
Another way to express condition (12.28)is to say that the measures vztp = ## must be absolutely continuous with respect to #, uniformly in :. Finally, weprove aresultwhich showswhy theuniformboundednessof moments of a given order may be important.
Jslxzl
12.10 Theorem lf 1+8
FIXzI for 0
>
0, then limv-pxlx'rl
Proof Note that
1( IxzIuv))
E IX,t11+e
(12.33)
< cxn =
0.
1+91
s (jx z j t l-vzlkxf)) S?E (IX1I1( jxvlzwzl) 2
for any 0 > 0. ne result follows on letting M (12.34) is finite by (12.33)..
..-+
x,
(12.34) since the majorant side of
Example 12.7 illustrated the fact that uniform fal-boundedness is not sufficient for unifonn integrability, but 12.10 shows that unifol'm falma-boundedness is sufficient, for any 0 > 0. Adding this result to those of j12.5, we have established the Merarchy of uniform conditions summarized in the following
theorem. 12.11 Theorem Uniform boundedness a.s.
uniform uniform = = uniform = uniform zz:
None of the reverse implications hold.
fw-boundedness, p > 1 integrability fw-boundedness, 0 < p S 1 boundedness in probability. n
13 Dependence
13.1 Shift Transformations We now consider relationships between the different members of a sequence. ln what ways, for example, might the joint distdbution of Xt and Xt-k depend upon k, and upon /? To answer questions such as these it is sometimes helpful to think aboutthe sequencein anew way. Havingintroducedthenotionof arandom sequence as a point in a probability space, there is a useful analogy between studying the relationships within a sequence and comparing different sequences, that is, different sample outcomes (l) e f. F-> copsider a 1-1 measurable mapping, F: ln a probability space (f1,@,#), (onto). This is a rule for pairing each outcome with another outcome of the space, but if each ) e f maps into an infinite sequence, Finduces a mapping from one sequence to another. F is called measure-preserving if PCTEI PE) for a11E e F. The shift transformation for a seqence (X/)JT is defined bylo =
X(F)
=
X,+1().
(13.1)
F takes each outcome ) into the outcome under which the realized value of X occurring in period t now occurs in period t- 1, for every t. In effect, each coordinate of the sequence from t 2 onwards is relabelled with the previous Xt-vk), the relationship period ' s index. More generally we can write XtlTkl between points in the sequence k periods apart becoming a characteristic of the transformation F k Since Xt is a nv. for a1l f, both the shift transformation and its inverse F -1 th e backsh;ft transformation, must be measurable. Taken together, the single r.v. X1(): f r- R and the shift transformation F, can be thoght of as generating a complete description of the sequence (X/))JT. This can be seen as follows. Given #14), apply the transformation F to ), and each ( e ( defines the mapping A%((0):f F-> R, obtain Xz = X1(F(0). Doing this and we are ready to get Xg X2(F(0). ltrating the procedure generates as many points in the sequence as we require. =
=
.
,
tor
=
be a sequence of coin tosses (with1 13.1 Example Consider 12.1, Let (Xf())JT for heads, 0 for tails) beginning 11010010001 1... (say).Somewhere on the intelwal g0,1) of real numbers (in binary representation), there is also a sequence (A()') J7 beginning 10100100011..., identical to the sequence indexed by (l) apart om the dropping of the initial digit and the backshift of the remaindr by one position. Likewise there is another sequence fXt(''4 1T,a backshifted version of ' ) 17b eg inning 0100101 1...; and so foih. If we define the transformation taVt(0
Theory of Stochastic Processes
192
)j'' F by F -1 ) = (l) r-1)' etc. the sequence (Xl(() )T can be constructed as 1-J ) )T,t h at is the sequence of first members of the sequences thesequence lm(F foundby iterating the transformation, in this case beginning 1,1,0,... I:l '
,
=
,
,
'
,
This device reveals, among other things, the complex stnlcture of the probability space we are postulating. To each point (.t) e fl there must correspond a countably infinite set of points F ) e f w hich reproduce the same sequence apart from the The intertemporal properties of a sequence can absolute date associated with then be treated as a comparison of two sequences, the original and the sequence lagged k periods. Econometricians attempt to make inferences about economic behaviour from recorded economic data. In time-series analysis, the sample available is usually a single realization of a random sequence, economic history as it actually occurred. Because we observe only one world, it is easy to make the mistake of looking on it as the whole sample space, whereas it is really only one of the many possible realizations the workings of historical chance might have generated. Indeed, in our probability model the whole economic universe is the counterpm of a single random outcome ); there is an important sense in which the time series analyst is a statistician who draws inferences from single data points ! But although a single it is realization of the sequenc can be treated as a mapping from a single linked to a countably infinite set of s corresponding to the leads and lags of the seqpence. A large part of our subsequent enquiry can be summarized as posing the question: is this set really rich enough to allow us to make inferences about # from a single realization? ,
m.
,
13.2 Independence
nnd
Stationarity
Independence and statioharity are the best-known restrictions on the behaviour of a sequence, but also the most stringent, from the point of view of describing economic time series. But while the emphasis in this book will mainly be on finding ways to relax these conditions, they remain important because of the many classic theorems in probability and limit theory which are founded on them. The degree to wlch random variations of sequnce coordinates are related to those of their neighbours in the time ordering is sometimes called the memory of a sequence; in the context of time-ordered observations, one may think in tenns of the amount of information contained in the cunrnt state of the sequence about its previous states. A sequence with no memory is a rather special kind of object, because the ordering ceases to have signiticance. lt is like the outcome of a collection of independent random experiments conducted in parallel, and indexed
arbitrarily. When a time ordering does nominally exist, we call such a sequence serially independent. Generalizing the theory of j8.6, a pair of sequences ((X,())T, l Fr() JT1e R= xR*, is independent if, for a11F1, E1 e 5*, /:()1.Y,17 e F1, (F,1: e E,4 Accordingly, a sequence
(A())T
=
#(tx,)7
e Ej4pf
r,)7 e
F2).
(13.2)
is serially independent if it is independent of
Dependence
193
fXt(T k(0) IT fOr a 11k ls 0 This is equivalent to saying that every tinite collection of sequence coordinates is totally independent. Serial independence is the simplest possible assumption about memory. Similarly, looking at the distribution of the sequence as a whole, the simplest treatment is to assume that the joint distribution of the coordinates is invariant with respect to the time index. A random sequence is called strictly stationary if Tllis implies that the sequences the shift transformation is measure-preserving. fXf 17 and fAkJ7=1have the same joint distribution, for every k > 0. Subject to the existence of particular moments, less restrictive versions of the condition are also commonly employed. Letting w E(Xt), and Xkt Covxt,xtn), consider those cases in which the sequence (g'rl7 and also the array ( fXmt are well defined. If w g,, al1 /, we say the sequence is mean J';=()l7wl, where (y,,l stationaty. If a mean stationary sequence has 17'is a sequence wt w of constants, it is called covariance stationaty, or wide sense stationary. If the marginal distriution of Xt is the same for any t, the sequence IXCJis said to be identically distributed. This concept is different from stationarity, which also restricts the joint distribution of neighbours in the sequence. However, Fhen a stochastic sequence is both serially independent and identically distributed (or i.i.d.4, this suftices for stationarity. An i.i.d. sequence is likeanarbitrarily indexedrandom sampledrawngomsomeaunderlyingpopulation. The following clutch of examples include both stationary and nonstationary .
.1
=
=
=1,
=
=
CaKS. (52 < 13.2 Example Let the sequence (6>J b e i i d with mean 0 and variance where fA)7, and let (0j17be a square-summable sequence of constants. Then ''O -x
.
.
(x,
.
*
A
=
77bet-j,
(13.3)
jcst
is a covariance stationary sequence, with Ext) t. This is the inhnite-order moving average additional details. n 13.3 Example lf
kt
=
0 and Ext)2 = c J e? ) sr every (MA(=)) process. See j14.3 for 2:.,.0
,0
is i.i.d. with mean 0, and Xt
for a constant a, then Ex
=
=
cos at
+
(13.4)
st
cos at, depending systematically on /.
I:a
13.4 Example Let (xV)be any stationary sequence with autocovariance sequence (ym, m 0J The sequence (.X)+ A%) has autocovariances given by the array .
l
'o
+
and hence it is nonstationary.
'sm
+
Yt+
'tt,vm,
m 2 0, t k 1 l
,
u
13.5 Example Let X be a r.v. which is symmetrically distributed about 0, with variance c 2 If Xt (- 1)W then (X,)Tis a stationary sequence. In particular, .
Ext)
=
=
,
0, and Cov(Xt,Xt+k4 5.2 when k is even and =
-c2
when k is odd, independent
Theory of Stochastic Processes
194 of t.
(:1
Theseexamples show that, contraryto acommon misconception, stationarity does not imply homogeneity in the appearance of a sequence, or the absence of periodic patterns. The essential feature is that any patterns in the sequence do not depend systematically on the time index. It is also important to distnguish between stationarity and limited dependence, although these notions are often closely linked. Example 13.4 is nonstationaty in view of dependence on initial conditions. The square-summability condition in 13.2 allows us to show covariance stationarity, but is actually a limitation on the long-range dependence of the process. Treatments of time series modelling which focus exclusively on models in the linear MA class often fail to distinguish between these propeties, but examples 13.3 and 13.5 demonstrate that there is no necessary connection betwen
them.
'
Stationarity is a strong assumption, particularly for the description of empirical time series, where features like seasonal patterns are commonly found. It is nonstationarity, sometling we might think of useful to distinguish between capable of elimination by local averaging of the coordinates, and global' as nonstationarity, involving features such as persistent trends in the moments. If sequences (Xr)7=1 and (a'+k)7=1have the same distribution for some (not necessarily every) k > 0, it follows that (.+x)7.1 has the same distribution as (X)+l7=1, and the same property extends to every integer multiple of k. Such a sequence accordingly has certain stationary characteristics, if we think in tenns of the distributions of successive blocks of k coordinates. This idea retains x. Consider the limit of a tinite sequence of length n, divided force even as k into (rl j blocks of length gn 1,plus any remainder, for some a between 0 and 1. here denotes the largest integer below x.) The number of blocks as well (Note, E.;r) as their extent is going to infinity, and the stationarity (or otherwise) of the sequence of blocks in the limit is clearly an issue. Important applications of these ideas arise in Parts V and VI below. It is convenient to formulate a definition embodying this concept in terms of moments. Thus, a zero-mean sequence will be said to be globally covariance stationary if the autocovariance sequences (y/uf are Cesro-summable for each )*r.1 0, where is strictly positive the k in the Cesro sum case of the variances m = m 0). The fllowing are a pair of contrasting counter-examples. ilocal'
-->
13 6 Example A sequence with
'p,
/l5
-
@
is globally nonstationary for any
j
#
0. n
13.7 Example Consider the integer sequence beginning 1,2,1,1,2,2,2,2,1,1,1,1,1,1,1,1,...*, i.e. the value changes at points t 2 k k = 1 2 3 The cesro sum of this x. It fluctuates eventually between the points sequence fails to converge as n lk k = k even. A stochastic sequence having a 5/3 at n 2 k o dd and 4/3 at n of form globally nonstationary. (:1 is variance sequence this =
,
-->
,
,
=
,
,
,
,.,.
Dependence
195
13.3 lnvariant Events The amount of dependence in a sequence is the chief factordetermining how informative a realization of given length can be about the distribution that generated it. At one extreme, the i.i.d. sequence is equivalent to a true random sample. The classical theorems of statistics can be applied to this type of distribution. At the other extreme, it ij easy to specify sequences for which a single realization can never reveal the parameters of the distribution to us, even in the limit as its length tends to intinity. This last possibility is what concerns us most, since we want to know whether averaging operations applied to sequences have useful limiting properties', whether, for example, parameters of the generation process can be consistently estimated in this way. Toclarify theseissues, imaginerepeatedsamplingof random sequences (A((t)) JT', in other words, imagine being given a function X1(.) and transfonnation F, making repeated random drawings of (.t) from fl and constructing the corresponding random sequences', 13.1 illustrates the procedure. Let the sample drawings be denoted j, 1,...,N, and imagine constructing the average of the realizations at some i ,=1 X is called an ensemble average, fixed time / e. The average kxto N-1X% o((tk) which may be contrasted with the time average of a realization of length n for n-11J=1Xj()j). Fig. 13.1 illustrates this procedure, some g iven )j e D, Xntj) showing a sample of three realizations of the sequence. The ensemble average is the average of the points falling on the vertical line labelled /c. lt is clear and the ensemble average as n and N that the limits of the time averg respectively go to infinity are not in general the same; we might expect the ensemble average to tend to the marginal expectation Ext ), but the time average 8 will not do so except in special cases. If the sequence ls nonstationary Ext depends upon /o, but even assuming stationarity, it is still possible that different realizations of the sequence depend upon random effects which are common to a1l t. =
=
,
=
O O
O
O O O
O O
O
O
Fig. 13. 1
Theory of Stochastic Processes
196
In a probability space (f1,F,#) the event E e 5 is said to be invariant under a transfonnation Tif PCTEh F) 0. The criterion for invariance is sometimes given as TE F, but allowing the two events to differ by a set of measure zero does not change anything important in the theory. The set of events in 5 that are invariant under the shift transformation is denoted #'. =
=
13.8 Theorem
,@
is a c-tield.
Proof Since F is onto,
is clearly invariant. Since F is also 1-1,
TEc A E C
CTE4cA Ec
=
(13.5)
TE A E
=
by definition. And, given 3En e #, n e IN), PCTE,tA En)
PITEn
=
&) + P(En
-
-
TEn4 = O
(13.6)
for each n, and also F
UEn
A
n
UEn UTEn A UEn,
(13.7)
=
n
l
n
using 1.2(i). By 1.1(i) and then 1.1(iii),
UTEn UEn U TEn OEL ? U(Ff n
and similarly,
f'n
=
-
n
n
-
n
&),
(13.8)
?1
n
UEn UTEn U(En n
-
n
TEn).
-
(13.9)
n
= 0 The conclusion #gT(Un&) A (Un&)1 now follows by . proof. pletingthe
(13.6)and
3.6(ii), com-
An invariant random variable is one that is J/f-measurable. An invariant nv. Z() has the property that Z(F) =Z(), and an J/f-measurable sequence (Z,()) ):' Z1()) a.s. for every t. The invariant events is trivial in the sense that Zt and associated r.v.s constitute those aspects of the probability model that do not alter with the passage of time. =
= Ff((z)) + Z(), where Xt 13.9 Example Consider the sequence (AX()))) ( F,4(0)) random and example of invariant Z((0) An being a event is E an sequence a r.v. f (.0: Z() K z, Ff() e R ). Clearly E and FF are the same event, since Z is the only thing subject to a condition. Fig. 13. 1 illustrates this case'. lf ( Ff()) ) is a zero-mean stationary sequence, the figure illustrates the cases Z((l)1) = Z((.0:) 0, and Z((l)3) > 0. Even if EZ4 out' in the limit, = 0, the influence of Z() in the time average is not as it will be from the ensemble average. I:a =
=
The behaviour of the time average as n fundamental result.
--
cxn
is summarized by the following
be 13.10 Theorem (Doob 1953: th. X.2. 1) Let a stationary sequence (.&((.t)))T defined by a measurable mapping X1()) and measure-preserving shift transform-
Dependence
ation T, such that
m()
J-1
.Y1(F
=
1imSnbln
), an d
I
Z)=1m()).If
=
F( X1j
,7)()),
ECXL
=
1et Sn ()
x,
(13.10)
(:I
a.s.
<
n-+oo
In words, the limiting case of the time average can be identitied with the mean of the distribution conditional on the c-tield of invariant events. The proof of 13.10 requires a technical lemma.
A(I3) ((t):supatkst))
13.11 Lemma Let
=
Jvca(oXl())J#()
-
nI))>
0) Then for any set M e #', .
2 F#(Mf-hA(p)).
(13.11)
Proof We establish this for the case fs 0. To generalize to any real p, consider the sequence fA p) which is stationary if (XCIis stationary. > 0), the set of outcomes Write A for A(0), and let hj f ): maxlskust) for which the partial sum is positive at least once by time j. Note, the sequence x. Also 1et A as j f A.j) is monotone and hj =
-
,
=
'1h
--
N nj
T-ih
=
-j
n
:
=
max Sklf l Kkf n
>
0
(13.12)
.
'-j
Since k
j+k
&(r?(o) XxtT) =
!F71rf(e)) sjmke)SjTb, z-
=
>1
by defining & Nnj
=
(13.13)
-
J=./m 1
0 we may also write ):
=
max j- 1f k f n
sk
-
Sjbh) > 0
,
0 f j Kn
-
1
.
(13.14)
This is the set of outcomes for which the partial sums of the coordinates f'rom j + 1 to n are positive at least once, and we have the inequality (explained below) n-1
XA+1()
lxnytl
2 0, all
)
e D.
(13.15)
Integrating this sum over the invariant set M gives n-1
n-1
r
0 X j=0 J <
u-suxjwtodrb
v
n-1
m
'0
Jn
v
R'
''nn
X y=o
j,.x.y&+1(rVt,))J#()
n
g
=X j u-g jcz0
=
.X1(9d#((9) J-
=
Jv.ayXl(t9V#(), X jc
(13.16)
1
where the tirst equa lity uses the fact that hn-j = (: T-l(kbe Nnjj and the measure-preserving property of F, and the third is by reversing the order of summation. The dominated convergence teorem applied to fm lMraayl, with I I as the
m
Theory of Stochatic
198
Processes
dominating function, yields
(13.17) Jxawxltltf#tt9l. Jsyraayxltllptl -->
limit by 2.26, so that, as required,
This limit is equal to the Cesro
-1
j'ucvs m()##()
=
1im a-,-
The inequality in
n
Xnj
m()##4)) 0. .
(13.18)
ucxxj
s,
not self-evident, but is justified as follows. The having the property expression on the left is the sum containing only those Xt realization that in the partial sums from the point t onwards are positive at least once, otherwise the tth contribution to the sum is 0. The sum includes only Xt lying in segments of the sequence over which Sk increases, so that their net contribution must be positive. It would be zero only in the case Xt S 0 for 1 f t S n. Fig. 13.2 depicts a realization. for k shows values of Sk 1,...,a, vertical shows the separations between successive so the .Yf((0)are the running sum of the terms of (13.15).The coordinates where the Xt are to be omitted from (13.15) are arrowed, the criterion being that there is no to the which exceeds the right current
(13.15)is ,
'o'
=
&o'
<+'
.
to'
'o'.
+
+
+
* *
*
0
O
*
*
Fig. 13.2 Proof of 13.10 The first step is to show that the seqpence (Snlbln ITconverges almost surely to an invariant r.v. Sl). Consider 5'() = limsupa Snlln. RF)
=
lim sup vl--hx nkm
&+l() n+ 1
n+ 1 n
X1() =
-
n
)),
(13.19)
ltl = liminfn Snln. Hence, so that Rl is invariant, the same being true for sttt)) = M(a,p) invariant. Since i s supnsnln (x < < < the event j3 V((t)ll (: A(p) M(a,p) A(p) where defined in the statement of 13.11. Hence, is V1, M(a,j) in the lemma gives putting M =
lDependence
J>(oj)X1(t0V#())2 But now replace
(
):
supnt-xsktl/n)
199
p#(M(?'p)).
(13.20)
13.11 by -AX)), and observe that M(a,p) Hence we get
in AX(t)) -a)
>
c
A(-a)
=
.
Jsy(a,j)X1(t0V#(tt9 S tx#(M(?'p))'
(13.21)
Since the left-hand sides of (13.21)and (13.20)are equal and (x < p, it follows 5'() with probability 1. that #(M(a,p)) = 0; that is, R)) 8)) This completes the first stage of the proof. It is now required to show that S = Exj I#) a.s., that is, according to equation (10.18),that =
juxltdp Sinc
=
jsjdp,
=
each M e
(13.22)
M is invariant, ''t
j
juxLlkdp =
a-
n
.,yt
X >1
vr
j.
Jyrto, ju ,
X/t0)##((t)
=
-s
Snl
:#4tl))
(13.23)
and the issue hinges on the convergence of the right-hand member of (13.24)to
E(S3u4. Since the sequence (-Y/Jis stationary and integrable, it is also uniformly integrable, and the same is true of the sequence ( F,l, where Ff Xtsu and M e #. For : > 0, it is possible by 12.9 to choose an event E e 5 with #(& < 6, such that sup E 1FJ I1e,) < e. For the same E, the triangle inequality gives =
j
E
j) j s )x-(J yn,-, >1
#,)
Ii-,I E
dp
>1
< :
(13.24)
.
By the same argument, also using stationarity and integrability of Ff,
I j
J1 n-
n
y-, dp
-,
s
Hence by 12.9 the sequence n-v I>. If n-l n n n -11 F) 12.8, by =1
=
J
complete. w
n
,.j
jl
yl
),; jap
=
s j yj j
(13.25)
tn-1Z:=1Ff)
-->
i s a 1so uniformly S a.s., it is clear that n-S
-1
Mn
Since n is arbitrary,
n
-1
:#((t))
--y
( Sudp.
2M
n
integrable, where Sku a.s., so
1M
-->
(13.26)
(13.26)and (13.23)together give (13.22),and the proof
is
13.4 Ergodicity and Mixing The property of a stationary sequence which ensures that the time average and the ensemble average have the same limit is ergodicity, which is defined in terms of
I'heory of Stochastic Processes
200
the probability of invariant events. A measure-preserving transformation F is l or #() is the c-field of 0 for a11E e Ds where ergodic if either #(F) under F. A stationary sequence (Xf() )+-7is said to be ergodic if invariant events J-1 =
x/
=
Xl (F ) for every t where F is measure-preserving and ergodic. Some authors, such ag Doob, use the tenn metrically transitive for ergodic. Events that are invariant under ergodic transformations either occur almost surely, or do not occur almost surely. ln the case of 13.9, Z must be a constant almost surely. Intuitively, stationarity and ergodicity together are seen to be sufficient conditions for time averages and ensemble averages to converge to the same limit. is the mean not just of X1 Stationarity implies that, for example, g, f(m()) but of any member of the sequence. The existence of events that are invariant under the shift transformation means that there are regions of the sample space which a particular realization of the sequence will never visit. lf PIFFA E) = yhen the event Ec occurs with probability 0 in a realization where E occurs. However, if invariant events other than the trivial ones are ruled out, we ensure that a sequence will eventually visit a1l parts of the space, with probability 1. In this case time averaging and ensemble averaging are effectively equivalent
X()
=
=
t
(
operations.
)
The following corollary is the main reason for our interest in Theorem 13.10.
13.12 Ergodic theorem Let (.X)4(0)IT be a stationary, ergodic, integrable sequence. Then 1imSnelln
=
(13.27)
F(X1), a.s.
n-4-
Proof This is immediate from 13.10, since by ergodicity, EIXj I#)
F(X1) a.s. w
=
In an ergodic sequence, conditioning on events of probability zero or one is a is trivial operation almost surely, in that the information contained in trivial. The ergodic theorem is an example of a 1aw of large numbers, the first of several such theorems to be studied in later chapters. Unlike most of the subsequent examples this pne is for stationat'y sequences. Its practical applications in econometrics are limited by the fact that the stationarity assumption is often inappropriate, but it is of much theoretical interest, because ergodicity is a very mild constraint on the dependence, as we now show. Atransformation thatis measure-preserving eventually mixes up the outcomes in a non-invariant event A with those in Ac. The measure-preserving property rules out mapping sets into proper subsets of themselves, so we can be sure that FA jAc is nonempty. Repeated iterations of the transformation generate a sequence of sets l T V ) containing different mixtures of the elements of A and Ac. A positive AC; that is, if implies a negative dependence of # on dependence of # on #(A f-h B) > P(A4PB) then P(Ac /'A Bj PB4 #(A f-'h #) < #(#) #(A)#(#) PIAc)PCB). Intuition suggests that the average dependence of B on mixtures of A and Ac should tend to zero as the mixing-up proceeds. ln fact, ergodicity can be characterized in just this kind of way, as the following theorem shows. ,/
:4
=
-
-
=
llependence
201
shift transfonnation Fis ergodic if and only 13.13 Theorem A measure-preserving for pair of if, events A, B e @, any n
1
-xpvkx
lim
rasl
(13.28)
PA)PB).
=
k=1
n-yx
Proof To show ton 1y ito let A be an invariant event and B A. Then PCTPAf'n B) = #(A) for a11 k, and hence the left-hand side of (13.27)is equal to #(A) for all 1' #(A) 2 implying 84) 0 or 17.7 k. This gives #(A) apply the ergodic theorem to the indicator function of the se