Probability, Random Processes, and Ergodic Properties, Second Edition

Probability, Random Processes, and Ergodic Properties

“This page left intentionally blank.”

Robert M. Gray

Probability, Random Processes, and Ergodic Properties Second Edition

Robert M. Gray Department of Electrical Engineering Stanford University Stanford, CA 94305-4075 USA [email protected]

ISBN 978-1-4419-1089-9 e-ISBN 978-1-4419-1090-5 DOI 10.1007/978-1-4419-1090-5 Springer Dordrecht Heidelberg London New York Library of Congress Control Number: 2009932896 © Springer Science+Business Media, LLC 2009 All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identifie as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.

Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

To my brother, Peter R. Gray April 1940 – May 2009

“This page left intentionally blank.”

Preface

This book has a long history. It began over two decades ago as the first half of a book then in progress on information and ergodic theory. The intent was and remains to provide a reasonably self-contained advanced (at least for engineers) treatment of measure theory, probability theory, and random processes, with an emphasis on general alphabets and on ergodic and stationary properties of random processes that might be neither ergodic nor stationary. The intended audience was mathematically inclined engineering graduate students and visiting scholars who had not had formal courses in measure theoretic probability or ergodic theory. Much of the material is familiar stuff for mathematicians, but many of the topics and results had not then previously appeared in books. Several topics still find little mention in the more applied literature, including descriptions of random processes as dynamical systems or flows; general alphabet models including standard spaces, Polish spaces, and Lebesgue spaces; nonstationary and nonergodic processes which have stationary and ergodic properties; metrics on random processes which quantify how different they are in ways useful for coding and signal processing; and stationary or sliding-block mappings – coding or filtering – of random processes. The original project grew too large and the first part contained much that would likely bore mathematicians and discourage them from the second part. Hence I followed a suggestion to separate the material and split the project in two. The original justification for the present manuscript was the pragmatic one that it would be a shame to waste all the effort thus far expended. A more idealistic motivation was that the presentation had merit as filling a unique, albeit small, hole in the literature. Personal experience indicates that the intended audience rarely has the time to take a complete course in measure and probability theory in a mathematics or statistics department, at least not before they need some of the material in their research.

vii

viii

Preface

Many of the existing mathematical texts on the subject are hard for the intended audience to follow, and the emphasis is not well matched to engineering applications. A notable exception is Ash’s excellent text [1], which was likely influenced by his original training as an electrical engineer. Still, even that text devotes little effort to ergodic theorems, perhaps the most fundamentally important family of results for applying probability theory to real problems. In addition, there are many other special topics that are given little space (or none at all) in most texts on advanced probability and random processes. Examples of topics developed in more depth here than in most existing texts are the following.

Random processes with standard alphabets The theory of standard spaces as a model of quite general process alphabets is developed in detail. Although not as general (or abstract) as examples sometimes considered by probability theorists, standard spaces have useful structural properties that simplify the proofs of some general results and yield additional results that may not hold in the more general abstract case. Three important examples of results holding for standard alphabets that have not been proved in the general abstract case are the Kolmogorov extension theorem, the ergodic decomposition, and the existence of regular conditional probabilities. All three results play fundamental roles in the development and understanding of mathematical information theory and signal processing. Standard spaces include the common models of finite alphabets (digital processes), countable alphabets, and real alphabets as well as more general complete separable metric spaces (Polish spaces). Thus they include many function spaces, Euclidean vector spaces, two-dimensional image intensity rasters, etc. The basic theory of standard Borel spaces may be found in the elegant text of Parthasarathy [82], and treatments of standard spaces and the related Lusin and Suslin spaces may be found in Blackwell [7], Christensen [12], Schwartz [94], Bourbaki [8], and Cohn [15]. The development of the basic results here is more coding-oriented and attempts to separate clearly the properties of standard spaces, which are useful and easy to manipulate, from the demonstrations that certain spaces are standard, which are more complicated and can be skipped. Unlike the traditional treatments, we define and study standard spaces first from a purely probability theory point of view and postpone the topological metric space details until later. In a nutshell, a standard measurable space is one for which any nonnegative set function that is finitely additive (usually easy to prove) on a field has a unique extension to a probability measure on the σ -field generated by the field. Spaces with this property ease the construction of probability measures from set functions that arise naturally in real systems, such as sample aver-

Preface

ix

ages and relative frequencies. The fact that well behaved (separable and complete) metric spaces with the usual (Borel) σ -field are standard also provides a natural measure of approximation (or distortion) between random variables, vectors, and processes that will be seen to be quite useful in quantifying the performance of codes and filters. This book and its sequel have numerous overlaps with ergodic theory, and most of the ergodic theory literature focuses on another class of probability spaces and the associated models of random processes and dynamical spaces — Lebesgue spaces. Standard spaces and Lebesgue spaces are intimately connected, and both are discussed here and comparisons made. The full connection between the two is stated, but not proved because doing so would require significant additional material not part of the primary goals of this book.

Nonstationary and nonergodic processes The theory of asymptotically mean stationary processes and the ergodic decomposition are treated in depth in order to model many physical processes better than can traditional stationary and ergodic processes. Both topics are virtually absent in all books on random processes, yet they are fundamental to understanding the limiting behavior of nonergodic and nonstationary processes. Both topics are considered in Krengel’s excellent book on ergodic theorems [62], but the treatment here is in greater depth. The treatment considers both the common two-sided or bilateral random processes, which are considered to have been producing outputs forever, and the often more difficult one-sided or unilateral processes, which better model processes that are “turned on” at some specific time and which exhibit transient behavior,.

Ergodic properties and theorems The notions of time averages and probabilistic averages are developed together in order to emphasize their similarity and to demonstrate many of the implications of the existence of limiting sample averages. The ergodic theorem is proved for the general case of asymptotically mean stationary processes. In fact, it is shown that asymptotic mean stationarity is both sufficient and necessary for the classical pointwise or almost everywhere ergodic theorem to hold for all bounded measurements. The subadditive ergodic theorem of Kingman [60] is included as it is useful for studying the limiting behavior of certain measurements on random processes that are not simple arithmetic averages. The proofs are based on simple proofs of the ergodic theorem developed by Ornstein and Weiss [79], Katznelson and Weiss [59], Jones [55], and Shields [98]. These

x

Preface

proofs use coding arguments reminiscent of information and communication theory rather than the traditional (and somewhat tricky) maximal ergodic theorem. The interrelations of stationary and ergodic properties of processes that are stationary or ergodic with respect to block shifts are derived. Such processes arise naturally in coding and signal processing systems that operate sequentially on disjoint blocks of values produced by stationary and ergodic processes. The topic largely originated with Nedoma [73] and it plays an important role in the general versions of Shannon channel and source coding theorems in the sequel and elsewhere.

Process distance measures Measures of “distance” between random processes which quantify how “close” one process is to another are developed for two applications. The distributional distance is introduced as a means of considering spaces of random processes, which in turn provide the means of proving the ergodic decomposition of certain functionals of random processes. The optimal coupling or Kantorovich/Wasserstein/Ornstein or generalized d-distance provides a means of characterizing how close or different the long term behavior of distinct random processes can be expected to be. The distributional distance, quantifies how well the probabilities on a generating field match, and the optimal coupling distance characterizes the smallest achievable distortion between two processes with given distributions. The optimal coupling distance has cropped up in many areas under a variety of names as a distance between distributions, random variables, or random objects, but here the emphasis is on extensions of Ornstein’s distance between random processes. The process distance measures are related to other metrics on measures and random variables, including the Prohorov, variation, and distribution distance.

B-processes Several classes of random processes are introduced as examples of the general ideas, including the class of B-processes introduced by Ornstein, arguably the most important class for virtually all engineering applications, yet a class virtually invisible in the engineering literature, except for the special case of the outputs of time-invariant stable linear filters driven by independent, identically distributed (IID) processes. A discretetime B-process is the result of a time-invariant or stationary coding or filtering of an IID process; a continuous time B-process is one for which there is some sampling period T for which the discrete-time process formed by sampling every T seconds is a discrete-time B-process. This

Preface

xi

class is of fundamental importance in ergodic theory because it is the largest class of stationary and ergodic processes for which the Ornstein isomorphism theorem holds, that is, the largest class for which equal entropy rate implies isomorphic processes. It is also of importance in applications, because it is exactly the class of processes which can be modeled as stationary coding or filtering of coin flips, roulette wheel spins, or Gaussian or uniform IID sequences. In a sense, B-processes are the most random possible since they have at their core IID sequences.

What’s new in the Second Edition Much of the material has been rearranged and revised for pedagogical reasons to improve the flow of ideas and to introduce several basic topics earlier in order to reduce the clutter in the later, more difficult, ideas. The original overlong first chapter has been split in order to allow a more thorough treatment of basic probability before tackling random processes and dynamical systems. This permits moving material on many of the basic spaces treated in the book — metric spaces, Borel spaces, Polish spaces – into the discussion of sample spaces, providing a smoother flow of ideas both in the fundamentals of probability and in the later development of Polish Borel spaces. The final chapter in the original printed version has similarly been broken into two pieces to provide separate emphasis on process metrics and the ergodic decomposition of affine functionals. Completion of event spaces and probability measures is treated in more detail in order to allow a more careful discussion of the metric space based Borel spaces emphasized here and the Lebesgue spaces ubiquitous in ergodic theory. I have long found existing treatments of the relative merits of these two important probability spaces confusing and insufficient in describing why one or the other of the two models should be preferred. The emphasis remains on the former model, but I have made every effort to clarify why Borel spaces are the more natural and useful for many of the results developed here and in the sequel. Much more material is now given on the history and ideas underly¯ ing the the Ornstein process distance and its extensions. The d-distance and its extensions have their roots in the classic problem of “optimal transportation” or “optimal transport” introduced by Monge in 1781 [71] and extended by Kantorovich in the mid twentieth century [56, 57, 58]. A discussion of the optimal transport problem has been included as it provides insight into origins and properties of a key tool used in this book and its sequel as well as a liaison with a currently active area of research which has found application in a variety of branches of pure and applied mathematics. The discussion of the distance has been con-

xii

Preface

siderably expanded and reorganized to facilitate comparison with the related Monge/Kantorovich history, while still emphasizing Ornstein’s extension of these ideas to processes. Examples have been added where the distance can actually be computed, including independent identically distributed random processes and Gaussian random processes. Previously unpublished results extend the Gaussian results to a family of of linear process models driven by a common memoryless process. More specific examples of random processes have been introduced, most notably B-processes and K-processes, in order to better describe a hierarchy of examples of stationary and ergodic processes in terms of increasingly strong mixing properties. In addition, I believe strongly that the idea of B-processes is insufficiently known in information theory and signal processing, and that many results in those fields can be strengthened and simplified by focusing on B-processes. Numerous minor additions have been made of material I have found useful through over four decades of work in the field. Many classic inequalities (such as those of Jensen, Young, Hölder, and Minkowski) are now incorporated into the text along with proofs and many citations have been added.

Missing topics Having described the topics treated here that are lacking in most texts, I admit to the omission of many topics usually contained in advanced texts on random processes or second books on random processes for engineers. First, the emphasis is on discrete time processes and the development for continuous time processes is not nearly as thorough. It is more complete, however, than in earlier versions of this book, where it was ignored. The emphasis on discrete time is somewhat justified by a variety of excuses: The advent of digital systems and sampled-data systems has made discrete time processes at least equally important as continuous time processes in modeling real world phenomena. The shift in emphasis from continuous time to discrete time in texts on electrical engineering systems can be verified by simply perusing modern texts. The theory of continuous time processes is inherently more difficult than that of discrete time processes. It is harder to construct the models precisely and much harder to demonstrate the existence of measurements on the models, e.g., it is usually harder to prove that limiting integrals exist than limiting sums. One can, however, approach continuous time models via discrete time models by letting the outputs be pieces of waveforms. Thus, in a sense, discrete time systems can be used as a building block for continuous time systems.

Preface

xiii

Another topic clearly absent is that of spectral theory and its applications to estimation and prediction. This omission is a matter of taste and there are many books on the subject. Power spectral densities are considered briefly in the discussion of process metrics. A further topic not given the traditional emphasis is the detailed theory of the most popular particular examples of random processes: Gaussian and Poisson processes. The emphasis of this book is on general properties of random processes rather than the specific properties of special cases. Again, I do consider some specific examples, including the Gaussian case, as a sideline in the development. The final noticeably absent topic is martingale theory. Martingales are only briefly discussed in the treatment of conditional expectation. This powerful theory is not required in this book or its sequel on information and ergodic theory. If I had had the time, however, I would have included the convergence of conditional probability given increasing or decreasing σ -fields, results following from Doob’s Martingale convergence theorem.

Goals The book’s original goal of providing the needed machinery for a subsequent book on information and ergodic theory remains. That book rests heavily on this book and only quotes the needed material, freeing it to focus on the information measures and their ergodic theorems and on source and channel coding theorems and their geometric interpretations. In hindsight, this manuscript also serves an alternative purpose. I have been approached by engineering students who have taken a master’s level course in random processes using my book with Lee Davisson [36, 37] and who are interested in exploring more deeply into the underlying mathematics that is often referred to, but rarely exposed. This manuscript provides such a sequel and fills in many details only hinted at in the lower level text. As a final, and perhaps less idealistic, goal, I intended in this book to provide a catalogue of many results that I have found need of in my own research together with proofs that I could follow. I also intended to clarify various connections that I had found confusing or insufficiently treated in my own reading. These are goals for which I can judge the success — I have often found myself consulting this manuscript to find the conditions for some convergence result, the reasons for some required assumption, the generality of the existence of some limit, or the relative merits of different models for the various spaces arising in probability and random processes. If the book provides similar service for others, it will have succeeded in a more global sense. Each time the book has been

xiv

Preface

revised, an effort has been made to expand the index to help searching by topic.

Assumed Background The book is aimed at graduate engineers and hence does not assume even an undergraduate mathematical background in functional analysis or measure theory. Therefore topics from these areas are developed from scratch, although the developments and discussions often diverge from traditional treatments in mathematics texts. Some mathematical sophistication is assumed for the frequent manipulation of deltas and epsilons, and hence some background in elementary real analysis or a strong calculus knowledge is required. Familiarity with basic set theory is also assumed.

Exercises A few exercises are strewn throughout the book. In general these are intended to make handy results which do not merit the effort and space of a formal proof, but which provide additional useful examples or properties or otherwise reinforce the text.

Errors As I age my frequency of typographical and other errors seems to grow along with my ability to see through them. I apologize for any that remain in the book. I will keep a list of all errors found by me or sent to me at [email protected] and I will post the list at my Web site, http://ee.stanford.edu/∼gray/.

Acknowledgments The research in information theory that yielded many of the results and some of the new proofs for old results in this book was supported by the National Science Foundation. Portions of the research and much of the early writing were supported by a fellowship from the John Simon

Preface

xv

Guggenheim Memorial Foundation. Later revisions were supported by gifts from Hewlett Packard, Inc. The book benefited greatly from comments from numerous students and colleagues through many years: most notably Paul Shields, Lee Davisson, John Kieffer, Dave Neuhoff, Don Ornstein, Bob Fontana, Jim Dunham, Farivar Saadat, Mari Ostendorf, Michael Sabin, Paul Algoet, Wu Chou, Phil Chou, Tom Lookabaugh, and, in recent years, Támas Linder, John Gill, Stephen Boyd, and Abbas El Gamal. They should not be blamed, however, for any mistakes I have made in implementing their suggestions. I would like to thank those who have sent me typos for correction through the years, including Hossein Kakavand, Ye Wang, and Zhengdao Wang. I would also like to acknowledge my debt to Al Drake for introducing me to elementary probability theory, to Wilbur Davenport for introducing me to random processes, and to Tom Pitcher for introducing me to measure theory. All three were extraordinary teachers. Finally and most importantly, I would like to thank my family, especially Lolly, Tim, and Lori, for all their love and support through the years spent writing and rewriting. I am dedicating this second edition to my brother Pete, who taught me how to sail, helped me to survive my undergraduate and early graduate years, was a big fan of my Web books about our grandmother Amy Heard Gray, and set a high standard for living a good life — even during his final battle with Alzheimer’s and Parkinson’s diseases. Sailing with Pete as his heavy-weather crew when MIT won the 1961 Henry A. Morss Memorial Trophy at Annapolis remains an unforgettable high point of my life.

La Honda, California

Robert M. Gray June 2009

“This page left intentionally blank.”

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi 1

Probability Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Sample Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Metric Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Measurable Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Borel Measurable Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Polish Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Probability Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Complete Probability Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8 Extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 1 2 10 14 16 19 25 26

2

Random Processes and Dynamical Systems . . . . . . . . . . . . . . . . . . . 2.1 Measurable Functions and Random Variables . . . . . . . . . . . . . 2.2 Approximation of Random Variables and Distributions . . . . 2.3 Random Processes and Dynamical Systems . . . . . . . . . . . . . . . . 2.4 Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Equivalent Random Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Codes, Filters, and Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Isomorphism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35 35 38 40 45 51 54 56

3

Standard Alphabets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Extension of Probability Measures . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Standard Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Some Properties of Standard Spaces . . . . . . . . . . . . . . . . . . . . . . . 3.4 Simple Standard Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Characterization of Standard Spaces . . . . . . . . . . . . . . . . . . . . . . 3.6 Extension in Standard Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 The Kolmogorov Extension Theorem . . . . . . . . . . . . . . . . . . . . . . 3.8 Bernoulli Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

61 61 63 68 72 74 76 77 78 xvii

xviii

Contents

3.9 Discrete B-Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.10 Extension Without a Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.11 Lebesgue Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.12 Lebesgue Measure on the Real Line . . . . . . . . . . . . . . . . . . . . . . . .

80 82 90 91

4

Standard Borel Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.1 Products of Polish Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.2 Subspaces of Polish Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.3 Polish Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4.4 Product Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 4.5 IID Random Processes and B-processes . . . . . . . . . . . . . . . . . . . . 110 4.6 Standard Spaces vs. Lebesgue Spaces . . . . . . . . . . . . . . . . . . . . . . 112

5

Averages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.1 Discrete Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.2 Quantization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 5.3 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5.4 Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 5.5 Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 5.6 Integrating to the Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 5.7 Time Averages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 5.8 Convergence of Random Variables . . . . . . . . . . . . . . . . . . . . . . . . 142 5.9 Stationary Random Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 5.10 Block and Asymptotic Stationarity . . . . . . . . . . . . . . . . . . . . . . . . 154

6

Conditional Probability and Expectation . . . . . . . . . . . . . . . . . . . . . . 157 6.1 Measurements and Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 6.2 Restrictions of Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 6.3 Elementary Conditional Probability . . . . . . . . . . . . . . . . . . . . . . . 163 6.4 Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 6.5 The Radon-Nikodym Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 6.6 Probability Densities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 6.7 Conditional Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 6.8 Regular Conditional Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 6.9 Conditional Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 6.10 Independence and Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . 190

7

Ergodic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 7.1 Ergodic Properties of Dynamical Systems . . . . . . . . . . . . . . . . . 195 7.2 Implications of Ergodic Properties . . . . . . . . . . . . . . . . . . . . . . . . 200 7.3 Asymptotically Mean Stationary Processes . . . . . . . . . . . . . . . . 206 7.4 Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 7.5 Asymptotic Mean Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 7.6 Limiting Sample Averages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 7.7 Ergodicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 7.8 Block Ergodic and Totally Ergodic Processes . . . . . . . . . . . . . . . 230

Contents

xix

7.9 The Ergodic Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 8

Ergodic Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 8.1 The Pointwise Ergodic Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 8.2 Mixing Random Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 8.3 Block AMS Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 8.4 The Ergodic Decomposition of AMS Systems . . . . . . . . . . . . . . 252 8.5 The Subadditive Ergodic Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 253

9

Process Approximation and Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 263 9.1 Distributional Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 9.2 Optimal Coupling Distortion/Transportation Cost . . . . . . . . . 269 9.3 Coupling Discrete Spaces with the Hamming Distance . . . . . 274 9.4 Fidelity Criteria and Process Optimal Coupling Distortion . 277 9.5 The Prohorov Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 9.6 The Variation/Distribution Distance for Discrete Alphabets 288 9.7 Evaluating dp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 9.8 Measures on Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293

10 The Ergodic Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 10.1 The Ergodic Decomposition Revisited . . . . . . . . . . . . . . . . . . . . . 295 10.2 The Ergodic Decomposition of Markov Processes . . . . . . . . . . 299 10.3 Barycenters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 10.4 Affine Functions of Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 10.5 The Ergodic Decomposition of Affine Functionals . . . . . . . . . 309 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317

“This page left intentionally blank.”

Introduction

Abstract Random process are introduced with their traditional basic definition as a family of random variables and two additional models are described and motivated. The Kolmogorov or directly-given representation describes a random process as a probability distribution on the space of output sequences or waveforms producible by the random process. A dynamical system representation yields a random process by defining a single random variable or measurement on a probability space with a transformation or family of transformations on the underlying space. The content of the book is summarized as filling in the details of the many components of these models and developing their ergodic properties — behavior exhibited by limiting sample averages — and ergodic theorems — results relating sample averages to probabilistic averages. The organization of the book is described by a tour of the chapters, including introductory comments and overviews. Random processes provide the mathematical structure for the study of random or stochastic phenomena in the real world, such as the overwhelming variety of signals, measurements, and data communicated through wires, cables, the atmosphere, and fiber; stored in hard disks, compact discs, flash cards, and volatile memory; and processed by seemingly uncountable computers and other devices to produce useful information for human or machine users. Most of the theory underlying the analysis of such systems for communications, transmission, storage, interpretation, recognition, classification, estimation, detection, signal processing, and so on is based on the theory of random processes, especially on results relating computable and tractable predictions of behavior based on mathematical models to the likely actual behavior resulting when algorithms based on the theory are applied to “real-world” signals.

xxi

xxii

Introduction

Mathematically, a random process is simply an indexed family of random variables {Xt ; t ∈ T} defined on a common probability space (Ω, B, P ), which consists of a sample space Ω, event space or σ -field B, and a probability measure P . These objects, along with all of the others mentioned in this introductory chapter, are developed in depth in the remaining chapters. Typically the index set T corresponds to time, which might be discrete, say the integers or the nonnegative integers, or continuous, say the real line or the nonnegative portion of the real line. In this book the emphasis is on the discrete index or discrete time case and we typically write {Xn ; n ∈ T} to represent a discrete time random process or abbreviate it to {Xn } or even X when clear from context. The random variables Xn take values in some set or alphabet A, which might be as simple as the binary numbers {0, 1}, the unit interval [0, 1), or the real line R, or as complicated as a vector or function space. There are two other representations of random processes of particular interest here. One is the Kolmogorov or directly given representation where a new probability space is constructed having the form ∞ (A∞ , B∞ is the sample space of output sequences of the A , µ), where A ∞ random process, BA is a suitable event space, and µ is a probability measure on the sequence space called the process distribution of the random process. This construction deals directly with probabilities of sequences produced by the random process and not on abstract random variables defined on an underlying space. Both constructions have their uses, and they are clearly related — the Kolmogorov construction gives us a model of the first type by simply defining the Xn as the coordinate or sampling functions; that is, given a sequence x = (· · · , xn−1 , xn , xn+1 , · · · ) ∈ A∞ , define Xn (x) = xn . Probability theory allows to go the other way, we can begin with the basic definition of an infinite family of random variables and then construct the Kolmogorov model by deriving the necessary distributions for the output process from the original probability space and the definitions of the random variables, the sequence space inherits its measure from the original space. Both the original model and the Kolmogorov model produce the same results so far as the random process outputs are concerned. The original model, however, might also have other random processes of interest defined on it, not all of which are determinable from the Kolmogorov model for X. The third representation of a random process derives from the idea of a dynamical system, the principal object of interest in ergodic theory, a branch of probability theory that deals with measure-preserving transformations. This book and its sequel, Entropy and Information Theory [30], draw heavily on the ideas and methods from ergodic theory in the exposition and application of random processes. The early incorporation of the basic relevant structures eases the development. To approach this idea from the random process point of view, consider the Kolmogorov model for a two-sided random process {Xn ; n ∈

Introduction

xxiii

Z}, where Z is the set of all integers. If A is the alphabet of the Xn , then the action of “time” is captured by a (left) shift transformation T : A∞ → A∞ defined by T (· · · , xn−1 , xn , xn+1 , · · · ) = (· · · , xn , xn−1 , xn , · · · ). In words, T takes a sequence and moves every symbol in the sequence one slot to the left. The time n value of the random process can now be written as a function of the sequence as Xn (x) = X0 (T n x); that is, instead of viewing the random process as an infinite family of random variables defined on a common probability space, we can view it as a single random variable, X0 combined with a shift transformation T . The collection (A∞ , B∞ A , µ, T ) consisting of a probability space and a transformation on that space is a special case of a dynamical system. The general case is obtained by not requiring that the underlying probability space be a sequence space or Kolmogorov representation of a process. A dynamical system (Ω, B, P , T ) is a probability space together with a transformation defined on the space (to be sure there are additional technical assumptions to make this precise, but this is an introduction). A dynamical system combined with a random variable, say f , produces a random process in the same way Xn was constructed above: fn = f T n , by which we mean fn (ω) = f (T n ω) for every ω ∈ Ω. Given the underlying probability space, (Ω, B, P ) and the transformation, we can (in theory, at least) find out everything we wish to know about a process. The transformation T is measure preserving if the probability of any event (any set in B) is unchanged by shifting. In mathematical terms this means µ(F ) = µ({x : x ∈ F }) = µ({x : T x ∈ F }) = µ(T −1 F ) for all events F . The fact that shifting or transforming by T preserves the measure µ can also be interpreted as stating that given the transformation T , the measure µ is invariant or stationary with respect to the transformation T . Thus the theory of measure-preserving transformations in ergodic theory is simply the theory of stationary processes. In this book, however, we do not confine interest to measure-preserving transformations or stationary processes. The intent is to capture the most general possible class of processes or transformations which possess the same useful properties of stationary processes or measure-preserving transformations, but which do not satisfy the strict requirements of stationary processes. This leads to a class of processes that asymptotically have similar properties, but which can depart from strict stationarity in a variety of ways that arise in applications, including the possession of tran-

xxiv

Introduction

sients that die out with time, artifacts due to blocking, and anomolies that get averaged out over time. Each of the three models — a family of random variables, a distribution on a sequence space, and a dynamical system plus a measurement — says something about the others, and each will prove the most convenient representation at times, depending on what we are trying to prove. This book develops the details of many aspects of random processes, beginning with the fundamental components: a sample space Ω, a σ field or event space B, a measurable space (Ω, B), a probability measure P , a probability space (Ω, B, P ), a process alphabet A, measurable functions or random variables, random vectors, and random processes, and dynamical systems. In addition to studying the outputs of random processes, we study what happens when the processes are themselves coded or filtered to form new processes. In fact we have already built this generality into the infrastructure: we might begin with one random process X and define a random variable f : A∞ → B mapping a sequence of the original random process into a symbol in a new alphabet. The dynamical system implied by the Kolmogorov representation of the first process plus the random variable together yield a new random process, say Yn = f T n , that is, Yn (x) = f (T n x). This is an example of a time-invariant or stationary code or filter, and it combines ideas from random processes and dynamical systems to model a common construction in communications and signal processing — mapping one process into another. A primary goal of the theory (and of ergodic theory in particular) is to quantify the properties and interrelationships between two kinds of averages: probabilistic averages or expectations and time/sample/empirical averages. Probabilistic averages are found by calculus (integration) and look like Z Z EP f = f dP = f (ω)dP (ω), where P tells us the probability measure and f is a measurement or random variable, usually assumed to be real-valued or possibly a realvalued vector. An important example of a measurement f is an indicator functions of an event F taking the form ( f (x) = 1F (x) =

1 x∈F 0

x 6∈ F

.

In this case case the expected value is the probability of F is simply the probability of the event: Z EP 1F = dP = P (F ). F

Often these simple averages are not enough to quantify the behavior in sufficient detail, so we deal with conditional expectations which take

Introduction

xxv

the form EP (f | M), where we are given some information about what happened, usually the value of some other measurements on the system or the occurrence or nonoccurrence of relevant events. Time-averages look at an actual output of the process and make a sequence of measurements on the process and form a simple sum of the successive values and divide by the numer of terms, e.g., given a sequence x ∈ A∞ and a measurement f : A∞ → R, how does the sample average < f >n (x) =

n−1 n−1 1 X 1 X f (T i x) or, simply, < f >n = fTi n i=0 n i=0

behave in the limit as n → ∞? In the special case where f = X0 , the zero-time output of a random process {Xn }, the sample average is the time-average mean n−1 1 X Xi . n i=0 In the special case where the measurement is an indicator function, the sample average becomes a relative frequency or empirical probability . < 1F >n (x) =

1 { number of i, 0 ≤ i < n : T i x ∈ F }. n

Results characterizing the behavior of such limiting sample averages when they exist are known as ergodic properties, and results relating such limits to expectations are known as ergodic theorems or laws of large numbers. Together such results allow us to use the calculus of probability to predict the behavior of the actual outputs of random systems, at least if our models are good. An important related topic is how we measure how good our models are. This raises the issue of how well one process approximates another (one process might be the model we construct, the other process the unknown true model). Having a notion of the distortion or distance between two processes can be useful in quantifying the mismatch resulting from doing theory with one model and applying the results to another. This topic is touched upon in the consideration of distortion and distance measures (different from probability measures) and fidelity criteria considered in some detail in this book. The organization of the book is most easily described by an introduction and overview to each chapter. The remainder of this introduction presents such a tour of the topics and the motivation behind them.

xxvi

Introduction

Chapter 1: Probability Spaces The basic tool for describing and quantifying random phenomena is probability theory, which is measure theory for finite measures normalized to one. The history of probability theory is long, fascinating, and rich (see, for example, Maistrov [69]). Its modern origins lie primarily in the axiomatic development of Kolmogorov in the 1930s [61]. Notable landmarks in the subsequent development of the theory (and still good reading) are the books by Cramér [16], Loève [66], and Halmos [45]. More recent treatments that I have found useful for background and reference are Ash [1], Breiman [10], Chung [14], the treatment of probability theory in Billingsley [3], and Doob’s gem of a book on measure theory [21]. There are a variety of types of general “spaces” considered in this book, including sample spaces, metric spaces, measurable spaces, measure spaces and the special case of probability spaces. All of these are introduced in this chapter, along with several special cases such as Borel spaces and Polish spaces — complete, separable, metric spaces. Additional spaces are encountered later, including standard spaces and Lebesgue spaces. All of these spaces form models of various aspects of random phenomena and often arise in multiple settings. For example, metric spaces can be used to construct probability spaces on which random variables or processes are defined, and then new metric spaces can be constructed quantifying the quality of approximation of random variables and processes.

Chapter 2: Random Processes and Dynamical Systems The basic ideas of mathematical models of random processes are introduced in this chapter. The emphasis is on discrete time or discrete parameter processes, but continuous time processes are also considered. Such processes are also called stochastic processes or information sources, and, in the discrete time case, time series. Intuitively a random process is something that produces a succession of symbols called “outputs” in a random or nondeterministic manner. The symbols produced may be real numbers such as produced by voltage measurements from a transducer, binary numbers as in computer data, two-dimensional intensity fields as in a sequence of images, continuous or discontinuous waveforms, and so on. The space containing all of the possible output symbols at any particular time is called the alphabet of the random process, and the directly given model of a random process is an assignment of a probability measure to events consisting of sequences or collections of samples drawn from the alphabet. In this chapter three models are developed for random processes: the traditional model of a family of

Introduction

xxvii

random variables defined on a common probability space, the directly given model of a distribution on a sequence or waveform space of outputs of the random process, and the dynamical system model of a transformation on a probability space combined with a random variable or measurement. A variety of means of producing new random processes from a given random process are described, including coding and filtering one process to obtain another. The very special case of invertible mappings and isomorphism are described. In ergodic theory two processes are considered to be the “same” if one can be coded or filtered into another in an invertible manner. This book does not go into isomorphism theory in any detail, but the ideas are required both here and in the sequel for describing relevant results from the literature and interpreting several results developed here. My favorite classic reference for random processes is Doob [20] and for classical dynamical systems and ergodic theory Halmos [46] remains a wonderful introduction.

Chapter 3: Standard Alphabets It is desirable to develop a theory under the most general possible assumptions. Random process models with very general alphabets are useful because they include all conceivable cases of practical importance. On the other hand, considering only the abstract spaces of Chapter 2 can result in both weaker properties and more complicated proofs. Restricting the alphabets to possess some structure is necessary for some results and convenient for others. Ideally, however, we can focus on a class of alphabets that possesses useful structure and still is sufficiently general to well model all examples likely to be encountered in the real world. Standard spaces are a candidate for this goal and are the topic of this chapter and the next. In this chapter the focus is on the definitions and properties of standard spaces, leaving the more complicated demonstration that specific spaces are standard to the next chapter. The reader in a hurry can skip the next chapter. The theory of standard spaces is usually somewhat hidden in theories of topological measure spaces. Standard spaces are related to or include as special cases standard Borel spaces, analytic spaces, Lusin spaces, Suslin spaces, and Radon spaces. Such spaces are usually defined by their relation via a mapping to a complete separable metric space. Good supporting texts are Parthasarathy [82], Christensen [12], Schwartz [94], Bourbaki [8], and Cohn [15], and the papers by Mackey [68] and Bjornsson [6]. Standard spaces are also intimately connected with Lebesgue spaces, as developed by Rohlin [90, 89], as will be discussed.

xxviii

Introduction

The presentation here differs from the traditional one in that we focus on the fundamental properties of standard spaces purely from a probability theory viewpoint and postpone application of the topological and metric space ideas until later. As a result, we can define standard spaces by the properties that are needed and not by an isomorphism to a particular kind of probability space with a metric space alphabet. This provides a shorter route to a description of such a space and to its basic properties. In particular, it is demonstrated that standard measurable spaces are the only measurable spaces for which all finitely additive candidate probability measures are also countably additive and that this in turn implies many other useful properties. A few simple, but important, examples are provided: sequence spaces drawn from countable alphabets and certain subspaces of such sequence spaces. The next chapter shows that more general topological probability spaces satisfy the required properties.

Chapter 4: Standard Borel Spaces In this chapter we develop the most important (and, in a sense, the most general) class of standard spaces– Borel spaces formed from complete separable metric spaces. In particular, we show that if 1. (A, B(A)) is a Polish Borel measurable space; that is, A is a complete separable metric space and B(A) is the σ -field generated by the open subsets of A, and 2. Ω ∈ B(A); that is, Ω is a Borel set of A, then (Ω, Ω ∩ B(A)) is a standard measurable space. The event Ω itself need not be Polish. We accomplish this by showing that such spaces are isomorphic to a standard subspace of a countable alphabet sequence space and hence are themselves standard. The proof involves a form of coding, this time corresponding to quantization — mapping continuous spaces into discrete spaces. The classic text for this material is Parthasarathy [82], but we prove a simpler, if weaker, result.

Chapter 5: Averages The basic focus of classical ergodic theory was the development of conditions under which sample or time averages consisting of arithmetic means of a sequence of measurements on a random process converged to a probabilistic or ensemble average of the measurement as expressed by an integral of the measurement with respect to a probability mea-

Introduction

xxix

sure. Theorems relating these two kinds of averages are called ergodic theorems or laws of large numbers. In this chapter these two types of averages are defined, and several useful properties are gathered and developed. The treatment of ensemble averages or expectations is fairly standard, except for the emphasis on quantization and the fact that a general integral can be considered as a natural limit of simple integrals (sums, in fact) of quantized versions of the function being integrated. In this light, general (Lebesgue) integration is actually simpler to define and deal with than is the integral usually taught to most engineers, the ill-behaved and complicated Riemann integral. The only advantage of the Riemann integral is in computation, actually evaluating the integrals of specific functions. Although an important attribute for applications, it is not as important as convergence and continuity properties in developing theory. The theory of expectation is simply a special case of the theory of integration. Excellent treatments may be found in Halmos [45], Ash [1], and Chung [14]. Many proofs in this chapter follow those in these books. The treatment of expectation and sample averages together is uncommon, but it is useful to compare the definitions and properties of the two types of average in preparation for the ergodic theorems to come. As we wish to form arithmetic sums and integrals of measurements, in this chapter we emphasize real-valued random variables.

Chapter 6: Conditional Probability and Expectation We begin the chapter by exploring some relations between measurements, that is, measurable functions, and events, that is, members of a σ -field. In particular, we explore the relation between knowledge of the value of a particular measurement or class of measurements and knowledge of an outcome of a particular event or class of events. Mathematically these are relations between classes of functions and σ -fields. Such relations are useful for developing properties of certain special functions such as limiting sample averages arising in the study of ergodic properties of information sources. In addition, they are fundamental to the development and interpretation of conditional probability and conditional expectation, that is, probabilities and expectations when we are given partial knowledge about the outcome of an experiment. Although conditional probability and conditional expectation are both common topics in an advanced probability course, the fact that we are living in standard spaces results in additional properties not present in the most general situation. In particular, we find that the conditional probabilities and expectations have more structure that enables them to be interpreted and often treated much like ordinary unconditional

xxx

Introduction

probabilities. In technical terms, there will always be regular versions of conditional probability and we will be able to define conditional expectations constructively as expectations with respect to conditional probability measures.

Chapter 7: Ergodic Properties In this chapter we formally define ergodic properties as the existence of limiting sample averages, and we study the implications of such properties. We shall see that if sample averages converge for a sufficiently large class of measurements, e.g., the indicator functions of all events, then the random process must have a property called asymptotic mean stationarity or be asymptotically mean stationary (AMS) and that there is a stationary measure, called the stationary mean of the process, that has the same sample averages. In addition, it is shown that the limiting sample averages can be interpreted as conditional probabilities or conditional expectations and that under certain conditions convergence of sample averages implies convergence of the corresponding expectations to a single expectation with respect to the stationary mean. A key role is played by invariant measurements and invariant events, measurements and events which are not affected by shifting. A random process is said to be ergodic if all invariant events are trivial, that is, have probability 0 or 1. Ergodicity is shown to be a necessary condition for limiting sample averages to be constants instead of to general random variables. Finally, we show that any dynamical system or random process possessing ergodic processes has an ergodic decomposition in that its ergodic properties are the same as those produced by a mixture of stationary and ergodic processes. In other words, if a process has ergodic properties but it is not ergodic, then the process can be considered as a random selection from a collection of ergodic processes. Although there are several forms of convergence of random variables and hence we could consider several types of ergodic properties, we focus on almost everywhere convergence and later consider implications for other forms of convergence. The primary goal of this chapter is the development of necessary conditions for a random process to possess ergodic properties and an understanding of the implications of such properties. In Chapter 8 sufficient conditions are developed.

Introduction

xxxi

Chapter 8: Ergodic Theorems At the heart of ergodic theory are the ergodic theorems, results providing sufficient conditions for dynamical systems or random processes to possess ergodic properties, that is, for sample averages of the form < f >n =

n−1 1 X fTi n i=0

to converge to an invariant limit. Traditional developments of the pointwise ergodic theorem focus on stationary systems and use a subsidiary result known as the maximal ergodic lemma (or theorem) to prove the ergodic theorem. The general result for AMS systems then follows since an AMS source inherits ergodic properties from its stationary mean; that is, since the set {x :< f >n (x) converges} is invariant and since a system and its stationary mean place equal probability on all invariant events, one measure possesses almost everywhere ergodic properties with respect to a class of measurements if and only if the other measure does and the limiting sample averages are the same. The maximal ergodic lemma is due to Hoph [52], and Garsia’s simple and elegant proof [28] is almost universally used in published developments of the ergodic theorem. Unfortunately, however, the maximal ergodic lemma is not very intuitive and the proof is somewhat tricky and adds little insight into the ergodic theorem itself. Furthermore, we also develop limit theorems for sequences of measurements that are not additive, but are instead subadditive or superadditive, in which case the maximal lemma is insufficient to obtain the desired results without additional techniques. During the early 1980s several new and simpler proofs of the ergodic theorem for stationary systems were developed and subsequently published by Ornstein and Weiss [79], Jones [55], Katznelson and Weiss [59], and Shields [98]. These developments dealt directly with the behavior of long run sample averages, and the proofs involved no mathematical trickery. We here largely follow the approach of Katznelson and Weiss. My favorite modern texts for ergodic theorems, which includes many topics not touched here, is Krengle [62].

Chapter 9: Process Approximation and Metrics Given two probability measures on a common probability space, how different or distant from each other are they? Similarly, given two random processes with known distributions, how different are the processes

xxxii

Introduction

from each other and what impact does such a difference have on their respective ergodic properties? There are a wealth of notions of distance between measures. For example, more than seventy examples of probability metrics can be found in the encyclopedic treatise by Rachev [86]. A metric on spaces of probability measures yields a metric space of such measures, and thereby yields useful notions of convergence and continuity of measures and distributions. Of the many notions, however, we focus on two which are particularly relevant for the purposes of this book and its sequel. A few other relevant metrics are considered for reasons of comparison and as useful tools in their own right. More generally, however, we are interested in quantifying the difference between two random objects in a way that need not satisfy the properties of a metric or even a pseudometric. Such generalizations of metrics are called distortion measures and fidelity criteria in information theory, but they are also known as cost functions in other fields. As the sequel to this book is concerned with information theory, we use the term distortion measure and the corresponding intuition that one random object can be considered as a distorted version of another and that the distortion measure quantifies the degree of distortion. Even within information theory, however, “cost function” has been used as a description for distortion measures, especially when describing the constrained optimization problems arising in communications systems and the associated Shannon coding theorems. Distortion measures and fidelity criteria — a family of distortion measures — need not be symmetric in their arguments and need not satisfy the triangle inequality. The classic example of an important distortion measure that is not a metric is the squared-error ubiquitous in information and communications theory and signal processing, as well as in the practical world of signal quality measurements based on signal-to-noise ratios. True, a sum of squares can be converted into a metric by taking the square root to produce the Euclidean distance, but the square root operation changes the mathematics and does not reflect the actual practice. On the other hand, the fact that a distortion measure can be a simple function of a metric, such as a positive power of the metric, allows many results for metrics to be extended to the distortion measure. In particular, the triangle inequality provides a useful measure of mismatch and robustness in information theory — it leads to useful bounds on the loss of performance when designing a system for one random object but applying the system to another. The richest collection of results follows for the case where distortion measures are powers of metrics, which turns out to be the primary case of interest in the parallel theory of optimal transportation. In this chapter we develop two quite distinct notions of the distortion or distance between measures or processes and use these ideas to delve

Introduction

xxxiii

further into the ergodic properties of processes. The first notion considered, the distributional distance, measures how well the probabilities of certain important events match up for the two probability measures, and hence this metric need not have relate to any underlying metric on the original sample space. The second measure of distortion between processes is constructed from a more basic measure of distortion on the underlying process alphabets, which might be a metric on a common alphabet. The distributional distance is primarily useful for the derivation of the final result of this book in the next chapter: the ergodic decomposition of affine functionals. This result plays an important role in the sequel where certain information theoretic quantities are shown to be affine functionals of the underlying processes, notably the Shannon entropy rate and the Shannon distortion-rate function. The second notion of distortion has arisen in multiple fields and has many names, including the Monge/Kantorovich/Wasserstein/Ornstein distance, transportation distance, d-distance, optimal coupling distance, earth mover’s distance, and Mallow’s distance. We indulge in more of an historical discussion then is generally the case in this book because past books and papers in ergodic theory and information theory have scarcely mentioned the older history of the Monge/Kantorovich optimal transportation theory and the fundamentally similar ideas between the distance measures used, and because the modern rich literature of op¯ timal transportation theory makes virtually no mention of Ornstein’s ddistance in ergodic theory and its extensions in information theory. Similar ideas have cropped up under other names in a variety of branches of mathematics, probability, economics, and physics. The process distortion will be seen to quantify the difference between generic or frequencytypical sequences from distinct stationary ergodic sources, which provides a geometric interpretation of ergodic properties of random processes. In the sequel, the distortion is be used extensively as a tool in the development of Shannon source coding theory. It quantifies the goodness or badness of an approximation to or reproduction of an original process or “source” and it provides a measure of the mismatch resulting when a communications system optimized for a particular random process is applied to another. These coding applications are not pursued in this volume, but they provide motivation for a careful look at such process metrics and their properties and evaluation. There are several excellent and encyclopedic texts on the coupling distortion for random variables and vectors, including [86, 87, 109, 110, 111]. Fewer references exist for the extension of most interest here: process distortion measures and metrics, but the classics are Ornstein’s survey paper and book: [77, 78]. See also [97, 42, 99].

xxxiv

Introduction

Chapter 10 The Ergodic Decomposition The ergodic decomposition as developed in Chapters 7 and 8 demonstrates that all AMS measures have a stationary mean that can be considered as a mixture of stationary and ergodic components. Mathematically, any stationary measure m is a mixture of stationary and ergodic measures px and takes the form Z m = px dm(x), (0.1) which is an abbreviation for Z m(F ) = px (F ) dm(x), all F ∈ B. In addition, any m-integrable f can be integrated in two steps as Z Z Z f (x) dm(x) = ( dpx (y)f (y)) dm(x).

(0.2)

Thus stationary measures can be viewed as the output of mixtures of stationary and ergodic measures or, equivalently, a stationary and ergodic measure is randomly selected from some collection and then made visible to an observer. We effectively have a probability measure on the space of all stationary and ergodic probability distributions and the first measure is used to select one of the distributions of the given class. The preceding relations show that both probabilities and expectations of measurements over the resulting mixture can be computed as integrals of the probabilities and expectations over the component measures. In applications of the theory of random processes such as information and communication theory, one is often concerned not only with ordinary functions or measurements, but also with functionals of probability measures, that is, mappings of the space of probability measures into the real line. Examples include the entropy rate of a dynamical system, the distortion-rate function of a random process with respect to a fidelity criterion, the information capacity of a noisy channel, the rate required to code a given process with nearly zero reproduction error, the rate required to code a given process for transmission with a specified fidelity, and the maximum rate of coded information that can be transmitted over a noisy channel with nearly zero error probability. (See, for example, the papers and commentary in [38].) All of these quantities are functionals of measures: a given process distribution or measure yields a real number as a value of the functional. Given such a functional of measures, say D(m), it is natural to inquire under what conditions the analog of (0.2) for functions might hold, that is, conditions under which

Introduction

xxxv

Z D(m) =

D(px ) dm(x).

(0.3)

This is in general a much more difficult issue than before since unlike an ordinary measurement f , the functional depends on the underlying probability measure. In this chapter conditions are developed under which (0.3) holds. The classic result along this line is Jacob’s ergodic decomposition of entropy rate [54] and this material was originally developed to extend his results to more general affine functionals, which in particular need not be bounded (but must be nonnegative). This chapter’s primary use is in the sequel [30], but it belongs in this book because it is an application of ergodic properties and ergodic theorems.

Chapter 1

Probability Spaces

Abstract The fundamentals of probability spaces are developed: sample spaces, event spaces or σ -fields, and probability measures. The emphasis is on sample spaces that are Polish spaces — complete, separable metric spaces — and on Borel σ -fields — event spaces generated by the open sets of the underlying metric space. Complete probability spaces are considered in order to facilitate later comparison between standard spaces and Lebesgue spaces. Outer measures are used to prove the Carathéodory extension theorem, providing conditions under which a candidate probability measure on a field extends to a unique probability measure on a σ -field.

1.1 Sample Spaces A sample space Ω is an abstract space, a nonempty collection of points or members or elements called sample points (or elementary events or elementary outcomes). The abstract space intuitively corresponds to the “finest grain” outcomes of an experiment, but mathematically it is simply a set containing points. Important common examples are the binary space {0, 1}, a finite collection of consecutive integers ZM = {0, 1, . . . , M −1}, the collection of all integers Z = {. . . , −2, −1, 0, 1, 2, . . .}, and the collection of all nonnegative integers Z+ = {0, 1, 2, . . .}. These are examples of discrete sample spaces. Common continuous sample spaces include the unit interval [0, 1) = {r : 0 ≤ r < 1}, the real line R = (−∞, ∞) and the nonnegative real line R+ = [0, ∞),. Ordinary set theory is the language for manipulating points and sets in sample spaces. Points will usually be denoted by lower case Greek or Roman letters such as ω, a, x, r with point inclusion in a set F denoted by ω ∈ F and set inclusion of F = {ω : ω ∈ F } in another set G by F ⊂ G. The usual set theoretic operations of union ∪, intersection ∩, R.M. Gray, Probability, Random Processes, and Ergodic Properties, DOI: 10.1007/978-1-4419-1090-5_1, © Springer Science + Business Media, LLC 2009

1

2

1 Probability Spaces

complementation c , set difference −, and set symmetric difference ∆ will be introduced and defined as needed. All of these spaces can be used to construct more complicated spaces. For example, given a space A and a positive integer n, a simple product space or Cartesian product space Ω = An is defined as the collection of all n-tuples ω = an = (a0 , a1 , . . . , an−1 ), ai ∈ A for i ∈ Zn . For example, Rn is n-dimensional Euclidean space and {0, 1}n is the space of all binary n-tuples. Along with these finite-dimensional product spaces we can consider infinite dimensional product spaces such as M = {0, 1}Z+ = ×∞ i=0 {0, 1}, the space of all one-sided binary sequences (u0 , u1 , . . .), ui ∈ {0, 1} for i ∈ Z+ . More general product spaces will be introduced later, but the simple special cases of the binary space {0, 1} and the one-sided binary sequence space M+ will prove to be of fundamental importance throughout the book, a fact which can be interpreted as implying that flipping coins is the most fundamental of all random phenomena. ∆

1.2 Metric Spaces A metric space generalizes the cited examples of sample spaces in a way that provides an extremely useful notion of the quality of approximation of points in the sample space, how well or badly one point approximates another. Most of this book will focus on random processes with outputs taking values in metric spaces. A set A with elements called points is called a metric space if for every pair of points a, b in A there is an associated nonnegative real number d(a, b) such that d(a, b) = 0 if and only if a 6= b,

(1.1)

d(a, b) = d(b, a) (symmetry),

(1.2)

d(a, b) ≤ d(a, c) + d(c, b) all c ∈ A (triangle inequality) .

(1.3)

The function d : A×A → [0, ∞) is called a metric or distance or a distance function. A metric space will be denoted by (A, d) or, simply, A if the metric d is clear from context. If d(a, a) = 0 for all a ∈ A, but d(a, b) = 0 does not necessarily imply that a = b, then d is called a pseudometric and A is a pseudometric space. Pseudo-metric spaces can be converted into metric spaces by replacing the original space A by the collection of equivalence classes of points having zero distance from each other, but following Doob [21] we will usually just stick with the pseudometric space. Most definitions and results stated for metric spaces will also hold for pseudometric spaces.

1.2 Metric Spaces

3

In information theory, signal processing, communications, and other applications it is often useful to consider more general notions of approximation that might be neither symmetric nor satisfy the triangle inequality. A function d : A × B → [0, ∞) will be called a distortion measure or cost function. A common example of a distortion measure is a nonnegative power of a metric. Metric spaces provide mathematical models of many interesting alphabets. Several common examples are listed below. Example 1.1. Hamming distance A is a discrete (finite or countable) set with metric d(a, b) = dH (a, b) = 1 − δa,b , where δa,b is the Kronecker delta function ( δa,b =

1

a=b

0

a≠b

Example 1.2. Lee distance A = ZM with metric d(a, b) = dL (a, b) = |a − b| mod M, where k mod M = r in the unique representation k = aM + r , a an integer, 0 ≤ r ≤ M − 1. dL is called the Lee metric or circular distance. Example 1.3. The unit interval A = [0, 1) or the real line A = R with d(a, b) =| a − b |, the absolute magnitude of the difference. These familiar examples can be used as sample spaces by defining Ω = A. They can also be used to construct more interesting examples such as product spaces. Example 1.4. Average Hamming distance A is a discrete (finite or countable) set and Ω = An with metric d(x n , y n ) = n−1

n−1 X

dH (xi , yi ),

i=0

the arithmetic average of the Hamming distances in the coordinates. This distance is also called the Hamming distance, but the name is ambiguous since the Hamming distance in the sense of Example 1.1 applied to A can take only values of 0 and 1 while the distance of this example can take on values of i/n; i = 0, 1, . . . , n − 1. Better names for this example are the average Hamming distance or mean Hamming distance. This distance is useful in error correcting and error detecting coding examples as a measure of the change in digital codewords caused by noisy communication media.

4

1 Probability Spaces

Example 1.5. Levenshtein distance Ω = An is as in the preceding example. Given two elements x n and y n of Ω, an ordered set of integers K = {(ki , ji ); i = 1, . . . , r } is called a match of x n and y n if xki = yji for i = 1, 2, . . . , r . The number r = r (K) is called the size of the match K. Let K(x n , y n ) denote the collection of all matches of x n and y n . The best match size mn (x n , y n ) is the largest r such that there is a match of size r , that is, mn (x n , y n ) = max r (K). K∈K(x n ,y n )

The Levenshtein distance λn on An is defined by λn (x n , y n ) = n − mn (x n , y n ) =

min

(n − r (K)),

K∈K(x n ,y n )

that is, it is the minimum number of insertions, deletions, and substitutions required to map one of the n-tuples into the other [64] [105]. The Levenshtein distance is useful in correcting and detecting errors in communication systems where the communication medium may not only cause symbol errors: it can also drop symbols or insert erroneous symbols. Note that unlike the previous example, the distance between vectors is not the sum of the component distances, that is, it is not additive. The Levenshtein distance is also called the edit distance. Example 1.6. Euclidean distance For n = 1, 2, . . . A = [0, 1)n , the unit cube, or A = Rn , n-dimensional Euclidean space, with 1/2

 n

n

n−1 X

d(a , b ) = 

|ai − bi |

2

,

i=0

the Euclidean distance. Example 1.7. A = Rn with d(an , bn ) =

max

i=0,...,n−1

|ai − bi |

The previous two examples are also special cases of normed linear vector spaces . Such spaces are described next. Example 1.8. Normed linear vector space A with norm |.| and distance d(a, b) = |a − b|. A vector space or linear space A is a space consisting of points called vectors and two operations — one called addition which associates with each pair of vectors a, b a new vector a + b ∈ A and one called multiplication which associates with each vector a and each number r ∈ R (called a scalar) a vector r a ∈ A — for which the usual rules of arithmetic hold, that is, if a, b, c ∈ A and r , s ∈ R, then

1.2 Metric Spaces

5

a + b = b + a (commutative law) (a + b) + c = a + (b + c) (associative law) r (a + b) = r a + r b (left distributive law) (r + s)a = r a + sa (right distributive law) (r s)a = r (sa) (associative law for multiplication 1a = a. In addition, it is assumed that there is a zero vector, also denoted 0, for which a+0=a 0a = 0. We also define the vector −a = (−1)a. A normed linear space is a linear vector space A together with a function kak called a norm defined for each a such that for all a, b ∈ A and r ∈ R, kak is nonnegative and kak = 0 if and only if a = 0,

(1.4)

||a + bk ≤ ka|| + kb|| (triangle inequality) ,

(1.5)

kr ak = |r |kak.

(1.6)

If kak = 0 does not imply that a = 0, then k · || is called a seminorm. For example, the Euclidean space A = Rn of Example 1.6 is a normed linear space with norm 1/2



n−1 X

kak = 

a2i 

,

i=0

which gives the Euclidean distance of Example 1.6. The space A = Rn is also a normed linear space with norm kak =

n−1 X

| ai |,

i=0

which yields the distance of Example 1.7. Both of these norms and the corresponding metrics are special case of the following. Example 1.9. `p norm For any p ≥ 1, Euclidean space of Example 1.6 is a normed linear space with norm 1/p



n−1 X

kakp = 

i=0

p

ai 

.

6

1 Probability Spaces

The fact that the triangle inequality holds follows from Minkowski’s inequality for sums, which is later proved in Corollary 5.7. Example 1.10. Inner product space A with inner product (·, ·) and distance d(a, b) = ka − bk, where kak = (a, a)1/2 is a norm. An inner product space (or pre-Hilbert space) is a linear vector space A such that for each pair of vectors a, b ∈ A there is a real number (a, b) called an inner product such that for a, b, c ∈ A, r ∈ R (a, b) = (b, a) (a + b, c) = (a, c) + (b, c) (r a, b) = r (a, b) (a, a) ≥ 0 and (a, a) = 0 if and only if a = 0. Inner product spaces and normed linear vector spaces include a variety of general alphabets and function spaces, many of which will be encountered in this book. Example 1.11. Fractional powers of metrics Given any metric space (A, d) and 0 < p < 1, then (A, dp ) is also a metric space. Clearly dp (a, b) is nonnegative and symmetric if d is. To show that dp satisfies the triangle inequality, we use the following simple inequality. Lemma 1.1. Given a, b ≥ 0 and 0 < p < 1, (a + b)p ≤ ap + bp .

(1.7)

Proof. If 0 < p < 1 and 0 < x ≤ 1, then x p ≥ x. Thus ap + b p ap bp = + p p (a + b) (a + b) (a + b)p  p  p a b = + a+b a+b b a + =1 ≥ a+b a+b

2 Given any a, b, c ∈ A, since d is a metric we have that d(a, c) ≤ d(a, b) + d(b, c), which with Lemma 1.1 yields dp (a, c) ≤ (d(a, b) + d(b, c))p ≤ dp (a, b) + dp (b, c), proving that dp satisfies the triangle inequality. Example 1.12. Product Spaces Given a metric space (A, d) and an index set I that is some subset of the integers (e.g., Z+ = {0, 1, 2, . . .}), the Cartesian product

1.2 Metric Spaces

7

AI = ×i∈I Ai , where the Ai are all replicas of A, is a metric space with metric dI (a, b) =

X

2−|i|

i∈I

d(ai , bi ) . 1 + d(ai , bi )

The somewhat strange form above ensures that the metric on the sequences in AI is finite–simply adding up the component metrics would likely yield a sum that blows up. Dividing by 1 plus the metric ensures that each term is bounded and the 2−|i| ensure convergence of the infinite sum. In the special case where the index set I is finite, then the product space is a metric space with the simpler metric X dI (a, b) = d(ai , bi ). i∈I

If I = Zn , then the following shorthand is common: dn (a, b) = dZn (a, b) =

n−1 X

d(ai , bi ).

(1.8)

i=0

The triangle inequality follows from that of the component metrics. Example 1.13. Product Spaces and Metric Powers A more general construction of a metric dn on an n-fold product spaces is based on positive powers of the component metrics. Given a metric space (A, d) and p > 0, define  1/p  Pn−1 p 1≤p (p) i=0 d (ai , bi ) dn (a, b) = Pn−1 . (1.9)  p d (a , b ) 0
i

i

If 0 < p ≤ 1, the triangle inequality is immediate as in (1.8) since dp is a metric. For p ≥ 1, the triangle inequality follows from the Minkowski sum inequality (Corollary 5.7).

Open Sets and Topology Let A denote a metric space with metric d. Define for each a in A and each real r > 0 the open sphere (or open ball) with center a and radius r by Sr (a) = {b : b ∈ A, d(a, b) < r }. For example, an open sphere in the real line R with respect to the Euclidean metric is simply an open interval (a, b) = {x : a < x < b}. A subset F of A is called an open set if given any point a ∈ F there is an r > 0 such that Sr (a) is completely contained in F , that is, there are no points in F on the edge of F . Let OA denote the class of all open sets in A. The following lemma collects

8

1 Probability Spaces

several useful properties of open sets. The proofs are left as an exercise as they are easy and may be found in any text on topology or metric spaces; e.g., [100]. Lemma 1.2. All spheres are open sets. A and ∅ are open sets. A set is open if and only if it can be written as a union of spheres; in fact, [ [ F= Sr (a), (1.10) a∈F r :r
where d(a, G) = inf d(a, b). b∈G

; t ∈ T } is any (countable or uncountable) collection of open sets, If {FtS then t∈T Ft is open. If Fi , i ∈ N, is any finite collection of open sets, then ∩i∈N Fi is open. When we have a class of open sets V such that every open set F in OA can be written as a union of elements in V , e.g., the preceding open spheres, then we say that V is a base for the open sets and write OA = OPEN(V ) = {all unions of sets in V }. In general, a set A together with a class of subsets OA called open sets such that all unions and finite intersections of open sets yield open sets is called a topology on A. If V is a base for the open sets, then it is called a base for the topology. Observe that many different metrics may yield the same open sets and hence the same topology. For example, the metrics d(a, b) and d(a, b)/(1 + d(a, b)) yield the same open sets and hence the same topology. Two metrics will be said to be equivalent on A if they yield the same topology on A.

Convergence in Metric Spaces Given a metric space (A, d), a sequence an , n = 1, 2, . . . of points in A is said to converge to a point a if lim d(an , a) = 0,

n→∞

that is, if given  > 0 there is an N such that n ≥ N implies that d(an , a) ≤ . If an converges to a, then we write a = limn→∞ an or an →n→∞ a. A metric space A is said to be sequentially compact if every sequence has a convergent subsequence, that is, if an ∈ A, n = 1, 2, . . ., then there is a subsequence n(k) of increasing integers and a point a in A such that limk→∞ an(k) = a. The notion of sequential compactness is

1.2 Metric Spaces

9

considered somewhat old-fashioned in some modern analysis texts, but occasionally it will prove to be exactly what we need. We will need the following two easy results on convergence in product spaces. Lemma 1.3. Given a metric space A with metric d and a countable index set I, let AI denote the Cartesian product with the metric of Example 1.12. A sequence {xn }I = {xn,i ; i ∈ I} converges to a sequence x I = {xi , i ∈ I} if and only if limn→∞ xn,i = xi for all i ∈ I. Proof. The metric on the product space of Example 1.12 goes to zero if 2 and only if all of the individual terms of the sum go to zero. Corollary 1.1. Let A and AI be as in the lemma. If A is sequentially compact, then so is AI . Proof. Let xn , n = 1, 2, . . . be a sequence in the product space A. Since the first coordinate alphabet is sequentially compact, we can choose a 1 subsequence, say xn , that converges in the first coordinate. Since the second coordinate alphabet is sequentially compact, we can choose a 2 1 further subsequence, say xn , of xn that converges in the second coordinate. We continue in this manner, each time taking a subsequence of the previous subsequence that converges in one additional coordinate. n The so-called diagonal sequence xn will then converge in all coordinates as n → ∞ and hence by the lemma will converge with respect to the given product space metric. This method of taking subsequences of subsequences with desirable properties is referred to as a standard diagonalization. 2

Limit Points and Closed Sets If A is a metric space with metric d, a point a in A is said to be a limit point of a set F if every open sphere Sr (a) contains a point b 6= a such that b is in F . A set F is closed if it contains all of its limit points. The set F together with its limit points is called the closure of F and is denoted by F . Given a set F , define its diameter diam(F ) = sup d(a, b). a,b∈F

Lemma 1.4. A set F is closed if and only if for every sequence {an , n = 1, 2, . . .} of elements of F such that an → a we have that also a is in F . If d and m are two metrics on A that yield equivalent notions of convergence, that is, d(an , a) → 0 if and only if m(an , a) → 0 for an , a in A, then d and m are equivalent metrics on A, that is, they yield the same open sets and generate the same topology.

10

1 Probability Spaces

Proof. If a is a limit point of F , then one can construct a sequence of elements an in F such that a = limn→∞ an . For example, choose an as any point in S1/n (a) (there must be such an an since a is a limit point). Thus if a set F contains all limits of sequences of its points, it must contain its limit points and hence be closed. Conversely, let F be closed and an a sequence of elements of F that converge to a point a. If for every  > 0 we can find a point an distinct from a in S (a), then a is a limit point of F and hence must be in F since it is closed. If we can find no such distinct point, then all of the an must equal a for n greater than some value and hence a must be in F . If two metrics d and m are such that d-convergence and m-convergence are equivalent, then they must yield the same closed sets and hence by the previous lemma they must yield the same open sets. 2

Exercises 1. Prove Lemma 1.2. 2. Show that in inner product space with norm k · k satisfies the parallelogram law ||a + b||2 + ||a − b||2 = 2||a||2 + 2||b||2 3. Show that dI and d of Example 1.12 are metrics. 4. Show that if di ; i = 0, 1, . . . , n−1 are metrics on A, then so is d(x, y) = maxi di (x, y). 5. Suppose that d is a metric on A and r is a fixed positive real number. Define ρ(x, y) to be 0 if d(x, y) ≤ r and K if d(x, y) > r . Is ρ a metric? a pseudometric?

1.3 Measurable Spaces A measurable space (Ω, B) is a pair consisting of a sample space Ω together with a σ -field B of subsets of Ω (also called the event space). A σ -field or σ -algebra B is a collection of subsets of Ω with the following properties: Ω ∈ B. (1.11) If F ∈ B, then F c = {ω : ω 6∈ F } ∈ B. If Fi ∈ B; i = 1, 2, . . . , then

∞ [

(1.12)

Fi = {ω : ω ∈ Fi for some i} ∈ B. (1.13)

i=1

The definition is in terms of unions, but from basic set theory this also implies that countable sequences of other set theoretic operations

1.3 Measurable Spaces

11

on events also yield events. For example, from de Morgan’s “law” of elementary set theory it follows that countable intersections are also in B: ∞ ∞ \ [ Fi = {ω : ω ∈ Fi for all i} = ( Fic )c ∈ B. i=1

i=1

Other set theoretic operations of interest are set difference defined by F − G = F ∩ Gc and symmetric difference is defined by F ∆G = (F ∩ Gc ) ∪ (F c ∩ G) which implies that F ∆G = (F ∪ G) − (F ∩ G) = (F − G) ∪ (G − F ).

(1.14)

Set difference is often denoted by a backslash, F \G. Since both set difference and set symmetric difference can be expressed in terms of unions and complements and intersections, these operations on events yield other events. Set theoretic difference will quite often be used to express arbitrary unions of events as unions of disjoint events, a construction summarized in the following lemma for reference. The proof is omitted as it is elementary set theory. Lemma 1.5. An arbitrary sequence of events Fn , n = 1, 2, . . . can be converted into a sequence of disjoint events Gn with the same finite unions as follows: for n = 1, 2, . . . Gn = Fn −

n−1 [ i=1

n [ i=1

Gi =

n [

Fi .

Fi = Fn −

n−1 [

Gi

(1.15)

i=1

(1.16)

i=1

Thus an event space is a collection of subsets of a sample space (called events by virtue of belonging to the event space) such that any countable sequence of set theoretic operations (union, intersection, complementation, difference, symmetric difference) on events produces other events. Note that there are two extreme σ -fields: the largest possible σ -field of Ω is the collection of all subsets of Ω (sometimes called the power set), and the smallest possible σ -field is {Ω, ∅}, the entire space together with the null set ∅ = Ωc (called the trivial space). If instead of the closure under countable unions required by (1.13), we only require that the collection of subsets be closed under finite unions, then we say that the collection of subsets is a field or algebra. Fields will usually be simpler to work with, but unfortunately they do not have sufficient structure to develop required convergence properties. In par-

12

1 Probability Spaces

ticular, σ -fields possess the additional important property that it contains all of the limits of sequences of sets in the collection. In particular, if Fn , n = 1, 2, . . . is an S∞increasing sequence of sets in a σ -field, i.e., Fn−1 ⊂ Fn , and if F = n=1 Fn (in which case we write Fn ↑ F or limn→∞ Fn = F ), then also F is contained in the σ -field. This property may not hold true for fields, that is, fields need not contain the limits of sequences of field elements. If a field has the property that it contains all increasing sequences of its members, then it is also a σ -field. In a similar fashion we can define decreasing sets. If Fn decreases to F in the sense T∞ that Fn+1 ⊂ Fn and F = n=1 Fn , then we write Fn ↓ F . If Fn ∈ B for all n, then F ∈ B. Because of the importance of the notion of converging sequences of sets, it is useful to generalize the definition of a σ -field to emphasize such limits. A collection M of subsets of Ω is called a monotone class if it has the property that if Fn ∈ M for n = 1, 2, . . . and either Fn ↑ F or Fn ↓ F , then also F ∈ M. Clearly a σ -field is a monotone class, but the reverse need not be true. If a field is also a monotone class, however, then it must be a σ -field. A σ -field is sometimes referred to as a Borel field in the literature and the resulting measurable space called a Borel space. We will reserve this nomenclature for the more common use of these terms as the special case of a σ -field having a certain topological structure described in the Section 1.4.

Generating σ -Fields and Fields When constructing probability measures it will often be natural to define a candidate probability measure on a particular collection of important events, such as intervals in the real line, and then to extend the candidate measure to a measure on a σ -field that contains the important collection. It is convenient to do this in two steps, first extending the candidate measure to a field containing the important events (the field may have a simple structure easing such an extension), and then extending from the field to the full σ -field. This approach is captured in the idea of generating a field or σ -field from a collection of important sets. Given any collection G of subsets of a space Ω, then σ (G) will denote the smallest σ -field of subsets of Ω that contains G and it will be called the σ -field generated by G. By smallest we mean that any σ -field containing G must also contain σ (G). The σ -field is well defined since there must exist at least one σ -field containing G, the collection of all subsets of Ω. The intersection of all σ -fields that contain G must itself be a σ field, it must contain G, and it must in turn be contained by all σ -fields that contain G. Thus there is a smallest σ -field containing G.

1.3 Measurable Spaces

13

Similarly, define the field generated by G, F (G), to be the smallest field containing G. Clearly G ⊂ F (G) ⊂ σ (G) and the field will often provide a useful intermediate step. In particular, σ (G) = σ (F (G)) so that if it is convenient or necessary to have a generating class be a field, F (G) is the natural class to consider. A σ -field B will be said to be countably generated if there is a countable class G which generates B, that is, for which B = σ (G). It would be a good idea to understand the structure of a σ -field generated from some class of events. Unfortunately, however, there is no constructive means of describing the σ -field so generated. That is, we cannot give a prescription of adding all countable unions, then all complements, and so on, and be ensured of thereby giving an algorithm for obtaining all σ -field members as sequences of set theoretic operations on members of the original collection. We can, however, provide some insight into the structure of such generated σ -fields when the original collection is a field. This structure will prove useful when considering extensions of measures. Recall that a collection M of subsets of Ω is a monotone class if whenever Fn ∈ M, n = 1, 2, . . ., and Fn ↑ F or Fn ↓ F , then also F ∈ M. Lemma 1.6. Given a field F , then σ (F ) is the smallest monotone class containing F . Proof. Let M be the smallest monotone class containing F and let F ∈ M. Define MF as the collection of all sets G ∈ M for which F ∩ G, F ∩ Gc , and F c ∩ G are all in M. Then MF is a monotone class. If F ∈ F , then all members of F must also be in MF since they are in M and since F is a field. Since both classes contain F and M is the minimal monotone class, M ⊂ MF . Since the members of MF are all chosen from M, M = MF . This implies in turn that for any G ∈ M, then all sets of the form G ∩ F , G ∩ F c , and Gc ∩ F for any F ∈ F are in M. Thus for this G, all F ∈ F are members of MG or F ⊂ MG for any G ∈ M. By the minimality of M, this means that MG = M. We have now shown that for F , G ∈ M = MF , then F ∩ G, F ∩ Gc , and F c ∩ G are also in M. Thus M is a field. Since it also contains increasing limits of its members, it must be a σ -field and hence it must contain σ (F ) since it contains F . Since σ (F ) is a monotone class containing F , it must contain M; hence the two classes are identical.

Exercises 1. Prove (1.14). Hint: When proving two sets F and G are equal, the straightforward approach is to show first that if ω ∈ F , then also

14

1 Probability Spaces

ω ∈ G and hence F ⊂ G. Reversing the procedure proves the sets equal. 2. Show that F ∪ G = F ∪ (F − G).

3. 4. 5.

6.

7.

and hence the union of two events can always be written as the union of two disjoint events. Prove deMorgan’s laws. Show that the intersection of a collection of σ -fields is itself a a σ field. Let Ω be an arbitrary space. Suppose that Fi , i = T 1, 2, . . . are all σ fields of subsets of Ω. Define the collection F = i Fi ; that is, the collection of all sets that are in all of the Fi . Show that F is a σ -field. Given a measurable space (Ω, F ), a collection of sets G is called a subσ -field of F if it is a σ -field and if all of its elements belong to F , in which case we write G ⊂ F . Show that G is the intersection of all σ -fields of subsets of Ω of which it is a sub-σ -field. Prove Lemma 1.5.

1.4 Borel Measurable Spaces Given a metric space A together with a class of open sets OA of A, the corresponding Borel field (actually a Borel σ -field) B(A) is defined as σ (OA ), the σ -field generated by the open sets. The members of B(A) are called the Borel sets of A. A metric space A together with a Borel σ -field B(A) is called a Borel measurable space (A, B(A)) or, simply, a Borel space when it is clear from context that the space is a measurable space. Thus a Borel space is a measurable space where the σ -field is generated by the open sets of the alphabet with respect to some metric. Observe that equivalent metrics on a space A will yield the same Borel measurable space since they yield the same open sets. The structure of a Borel measurable space leads to several properties. Lemma 1.7. A set is closed if and only if its complement is open and hence all closed sets are in the Borel σ -field. Arbitrary intersections and finite unions of closed sets are closed. B(A) = σ (all closed sets). The closure of a set is closed. If F is the closure of F , then diam(F ) = diam(F ). Proof. If F is closed and we choose x in F c , then x is not a limit point of F and hence there is an open sphere Sr (x) that is disjoint with F and hence contained in F c . Thus F c must be open. Conversely, suppose that F c is open and x is a limit point of F and hence every sphere Sr (x) contains a point of F . Since F c is open this means that x cannot be in F c and hence must be in F . Hence F is closed. This fact, DeMorgan’s laws,

1.4 Borel Measurable Spaces

15

and Lemma 1.2 imply that finite unions and arbitrary intersections of closed sets are closed. Since the complement of every closed set is open and vice versa, the σ -fields generated by the two classes are the same. To prove that the closure of a set is closed, first observe that if x is in c F and hence neither a point nor a limit point of F , then there is an open sphere S centered at x which has no points in F . Since every point y in S therefore has an open sphere containing it which contains no points in c F , no such y can be a limit point of F . Thus F contains S and hence is open. Thus its complement, the closure of F , must be closed. Any points in the closure of a set must have points in the set within an arbitrarily close distance. Hence the diameter of a set and that of its closure are the same. 2 In addition to open and closed sets, two other kinds of sets arise often T∞ in the study of Borel spaces: Any set of the form i=1 Fi with the Fi open is called a Gδ set or simply a Gδ . Similarly, any countable union of closed sets is called an Fσ . Observe that Gδ and Fσ sets are not necessarily either closed or open. We have, however, the following relation. Lemma 1.8. Any closed set is a Gδ , any open set is an Fσ . Proof. If F is a closed subset of A, then ∞ \

F=

{x : d(x, F ) < 1/n},

n=1

and hence F is a Gδ . Taking complements completes the proof since any closed set is the complement of an open set and the complement of a Gδ is an Fσ . 2 The added structure of Borel measurable spaces and Lemma 1.8 permit a different characterization of σ -fields: Lemma 1.9. Given a metric space A, then B(A) is the smallest class of subsets of A that contains all the open (or closed) subsets of A, and which is closed under countable disjoint unions and countable intersections. Proof. Let G be the given class. Clearly G ⊂ B(A). Since G contains the open (or closed) sets (which generate the σ -field B(A)), we need only show that it contains complements to complete the proof. Let H be the class of all sets F such that F ∈ G and F c ∈ G. Clearly H ⊂ G. Since G contains the open sets and countable intersections, it also contains the Gδ ’s and hence also the closed sets from the previous lemma. Thus both the open sets and closed sets all belong to H . Suppose now that Fi ∈ H , i = 1, 2, . . . and hence both Fi and their complements belong to G. Since G contains countable disjoint unions and countable intersections ∞ [ i=1

Fi =

∞ [

c (Fi ∩ F1c ∩ F2c ∩ · · · ∩ Fi−1 )∈G

i=1

16

1 Probability Spaces

and (

∞ [

Fi )c =

i=1

∞ \

Fic ∈ G.

i=1

Thus the countable unions of members of H and the complements of such sets are in G. This implies that H is closed under countable unions. If F ∈ H , then F and F c are in G, hence also F c ∈ H . Thus H is closed under complementation and countable unions and hence H contains both countable disjoint unions and countable intersections. Since H also contains the open sets, it must therefore contain G. Since we have already argued the reverse inclusion, H = G. This implies that G is closed under complementation since H is, thus completing the proof. 2

1.5 Polish Spaces Separable Metric Spaces Let (A, d) be a metric space. A set F is said to be dense in A if every point in A is a point in F or a limit point of F . A metric space A is said to be separable if it has a countable dense subset, that is, if there is a discrete set, say B, such that all points in A can be well approximated by points in B in the sense that all points in A are points in B or limits of points in B. For example, the rational numbers are dense in the real line R and are countable, and hence R of Example 1.3 is separable. Similarly, n-dimensional vectors with rational coordinates are countable and dense in Rn , and hence Example 1.6 also provides a separable metric space. The discrete metric spaces of Examples 1.1 through 1.5 are trivially separable since A is countable and is dense in itself. An example of a nonseparable space is the binary sequence space M = {0, 1}Z+ with the sup-norm metric d(x, y) = sup |xi − yi |. i∈Z+

The principal use of separability of metric spaces is the following lemma. Lemma 1.10. Let A be a metric space with Borel σ -field B(A). If A is separable, then B(A) is countably generated and it contains all of the points of the set A. Proof. Let {ai ; i = 1, 2, . . .} be a countable dense subset of A and define the class of sets VA = {S1/n (ai ); i = 1, 2, . . . ; n = 1, 2, . . .}.

(1.17)

1.5 Polish Spaces

17

Any open set F can be written as a countable union of sets in VA analogous to (1.10) as [ [ F= S1/n (ai ). (1.18) i:ai ∈F n:n−1
Thus VA is a countable class of open sets and σ (VA ) contains the class of open sets OA which generates B(A). Hence B(A) is countably generated. To see that the points of A belong to B(A), index the sets in VA as {Vi ; i = 0, 1, 2, . . .} and define the sets Vi (a) by Vi if a is in Vi and Vic if a is not in Vi . Then ∞ \ {a} = Vi (a) ∈ B(A). i=1

2 For example, the class of all open intervals (a, b) = {x : a < x < b} with rational endpoints b > a is a countable generating class of open sets for the Borel σ -field of the real line with respect to the usual Euclidean distance. Similar subsets of [0, 1) form a similar countable generating class for the Borel sets of the unit interval.

Complete Metric Spaces A sequence {an ; = 1, 2, . . .} in A is called a Cauchy sequence if for every  > 0 there is an integer N such that d(an , am ) <  if n ≥ N and m ≥ N. Alternatively, a sequence is Cauchy if and only if lim diam(an , an+1 , . . .) = 0.

n→∞

A metric space is complete if every Cauchy sequence converges, that is, if {an } is a Cauchy sequence, then there is an a in A for which a = limn→∞ an . In the words of Simmons [100], p. 71, a complete metric space is one wherein “every sequence that tries to converge is successful.” A standard result of elementary real analysis is that Examples 1.3 and 1.6, [0, 1)k and Rk with the Euclidean distance, are complete (see Rudin [93], p. 46). A complete inner product space (Example 1.10) is called a Hilbert space. A complete normed space (Example 1.8) is called a Banach space. A fundamental property of complete metric spaces is given in the following lemma, which is known as Cantor’s intersection theorem. The corollary is a slight generalization. Lemma 1.11. Given a complete metric space A, let Fn , n = 1, 2, . . . be a decreasing sequence of nonempty closed subsets of A for which diam(Fn ) → 0. Then the intersection of all of the Fn contains exactly one point.

18

1 Probability Spaces

Proof. Since the diameter of the sets is decreasing to zero, The intersection cannot contain more than one point. If xn is a sequence of points in Fn , then it is a Cauchy sequence and hence must converge to some point x since A is complete. If we show that x is in the intersection, the proof will be complete. Since the Fn are decreasing, for each n the sequence {xk , k ≥ n} is in Fn and converges to x. Since Fn is closed, however, it must contain x. Thus x is in all of the Fn and hence in the intersection.

2

The similarity of the preceding condition to the finite intersection property of atoms of a basis of a standard space is genuine and will be exploited soon. Corollary 1.2. Given a complete metric space A, let Fn , n = 1, 2, . . . be a decreasing sequence of nonempty subsets of A such that diam(Fn ) → 0 and such that the closure of each set is contained in the previous set, that is, F n ⊂ Fn−1 , all n, where we define F0 = A. Then the intersection of all of the Fn contains exactly one point. Proof. Since Fn ⊂ F n ⊂ Fn−1 ∞ \

Fn ⊂

n=1

∞ \

Fn ⊂

n=1

∞ \

Fn .

n=0

Since the Fn are decreasing, the leftmost and rightmost terms above are equal. Thus ∞ ∞ \ \ Fn = F n. n=1

n=1

Since the Fn are decreasing, so are the F n . Since these sets are closed, the right-hand term contains a single point from the lemma. 2 The following technical result will be useful toward the end of the book, but its logical place is here and it provides a little practice. Lemma 1.12. The space [0, 1] with the Euclidean metric |x − y| is sequentially compact. The space [0, 1]I with I countable and the metric of Example 1.12 is sequentially compact. Proof. The second statement follows from the first and Corollary 1.1. To prove the first statement, let Sin , n = 1, 2, . . . , n denote the closed spheres [(i − 1)/n, i/n] having diameter 1/n and observe that for each n = 1, 2, . . . n [ [0, 1] ⊂ Sin . i=1

1.6 Probability Spaces

19

Let {xi } be an infinite sequence of distinct points in [0,1]. For n = 2 there is an integer k1 such that K1 = Sk21 contains infinitely many xi . S Since Skn1 ⊂ i Si3 we can find an integer k2 such that K2 = Sk21 ∩ Sk32 contains infinitely many xi . Continuing in this manner we construct a sequence of sets Kn such that Kn ⊂ Kn−1 and the diameter of Kn is less than 1/n. Since [0, 1] is complete, it follows from the finite intersection T property of Lemma 1.11 that n Kn consists of a single point x ∈ [0, 1]. Since every neighborhood of x contains infinitely many of the xi , x must be a limit point of the xi . 2

Polish Spaces A complete separable metric space is called a Polish space because of the pioneering work by Polish mathematicians on such spaces. Polish spaces play a central role in functional analysis and provide models for most alphabets of scalars, vectors, sequences, and functions that arise in communications applications. The best known examples are the Euclidean spaces, but we shall encounter others such as the space of square integrable functions. A Borel space (A, B(A)) with A a Polish metric space will be called a Polish Borel space.

1.6 Probability Spaces A probability space (Ω, B, P ) is a triple consisting of a sample space Ω , a σ -field B of subsets of Ω, and a probability measure P defined on the σ -field; that is, P (F ) assigns a real number to every member F of B so that the following conditions are satisfied: Nonnegativity: P (F ) ≥ 0, all F ∈ B,

(1.19)

P (Ω) = 1.

(1.20)

Normalization: Countable Additivity: If Fi ∈ B, i = 1, 2, . . . are disjoint, then P(

∞ [

i=1

Fi ) =

∞ X

P (Fi ).

(1.21)

i=1

Since the probability measure is defined on a σ -field, such countable unions of subsets of Ω in the σ -field are also events in the σ -field.

20

1 Probability Spaces

Alternatively, a probability space is a measurable space together with a probability measure on the σ -field of the measurable space. A set function satisfying (1.21) only for finite sequences of disjoint events is said to be additive or finitely additive. A set function P satisfying only (1.19) and (1.21) but not necessarily (1.20) is called a measure, and the triple (Ω, B, P ) is called a measure space. The emphasis here will be almost entirely on probability spaces rather than the more general case of measure spaces. There will, however, be occasional uses for the more general idea as a means of constructing specific probability measures of interest. A measure space (Ω, B, P ) is said to be finite if P (Ω) < ∞. Any finite measure space can be converted into a probability space by suitable normalization. A measure space is said to be sigma-finite if Ω is the union of a collection of measurable sets ofSfinite measure, that is, if there is a collection of events {Fi } such that i Fi = Ω and P (Fi ) < ∞ for all i. A straightforward exercise provides an alternative characterization of a probability measure involving only finite additivity, but requiring the addition of a continuity requirement: a set function P defined on events in the σ -field of a measurable space (Ω, B) is a probability measure if (1.19) and (1.20) hold, if the following conditions are met: Finite Additivity:

If Fi ∈ B, i = 1, 2, . . . , n are disjoint, then P(

n [

Fi ) =

i=1

n X

P (Fi ).

(1.22)

i=1

Continuity at ∅: If Gn ↓ ∅ (the empty or null set); that is, if Gn+1 ⊂ Gn , T∞ all n, and n=1 Gn = ∅, then lim P (Gn ) = 0.

n→∞

(1.23)

The equivalence of continuity and countable additivity is obvious from Lemma 1.5. Countable additivity for the Fn will hold if and only if the continuity relation holds for the Gn . It is also easy to see that condition (1.23) is equivalent to two other forms of continuity: Continuity from Below: If Fn ↑ F , then lim P (Fn ) = P (F ). n→∞

(1.24)

Continuity from Above: If Fn ↓ F , then lim P (Fn ) = P (F ). n→∞

(1.25)

Thus a probability measure is an additive, nonnegative, normalized set function on a σ -field or event space with the additional property that

1.6 Probability Spaces

21

if a sequence of sets converges to a limit set, then the corresponding probabilities must also converge. If we wish to demonstrate that a set function P is indeed a valid probability measure, then we must show that it satisfies the preceding properties (1.19), (1.20), and either (1.21) or (1.22) and one of (1.23), (1.24), or (1.25). Observe that if a set function satisfies (1.19), (1.20), and (1.22), then for any disjoint sequence of events {Fi } and any n P(

∞ [

Fi ) = P (

i=0

n [

i=0 n [

≥ P(

∞ [

Fi ) + P (

Fi )

i=n+1

Fi ) =

i=0

n X

P (Fi ),

i=0

and hence we have taking the limit as n → ∞ that P(

∞ [

Fi ) ≥

i=0

∞ X

P (Fi ).

(1.26)

i=0

Thus to prove that P is a probability measure one must show that the preceding inequality is in fact an equality. Example 1.14. Discrete Probability Spaces As a special case that is both simple and important, consider the case where Ω is discrete, either a finite or a countably infinite set. In this case the most common σ -field is the power set and any probability measure P has a simple description in terms of the probabilities it assigns to one point sets. Given a probability measure P , define the corresponding probability mass function (PMF) p by p(ω) = P ({ω}). (1.27) From countable additivity, the probability of any event F is easily found by summing the PDF: X P (F ) = p(ω). (1.28) ω∈F

Conversely, given any function p : Ω → [0, 1) that is nonnegative and normalized in the sense that X p(ω) = 1, ω∈Ω

then the properties of summation imply that the set function defined by (1.28) is a probability measure. This is obvious for a finite discrete set and it follows from the properties of summation for a generally countably infinite set.

22

1 Probability Spaces

Elementary Properties of Probability The next lemma collects a variety of elementary probability relations for later reference. Lemma 1.13. Suppose that (Ω, B, P ) is a probability space. All sets below are assumed to be members of B. 1. If G ⊂ F , then P (F − G) = P (F ) − P (G). 2. Monotonicity If G ⊂ F , then P (G) ≤ P (F ). 3. Countable Subadditivity Given a sequence of events Fi ; i = 1, 2, . . ., then ∞ ∞ [ X P( Fi ) ≤ P (Fi ) (1.29) i=1

i=1

This is also referred to as the Bonferroni or union bound. 4. Given two events F and G, two notions of how dissimilar the sets are in a probabilistic sense will be encountered shortly and throughout the book: the probability of the symmetric difference of the two sets and the difference of the probabilities of the two sets. The following inequality shows that the latter is always smaller than the former. | P (F ) − P (G) |≤ P (F ∆G).

(1.30)

5. P (F ∆G) = P (F ∪ G) − P (F ∩ G)

(1.31)

= P (F − G) + P (G − F )

(1.32)

= P (F ) + P (G) − 2P (F ∩ G)

(1.33)

6. A triangle inequality P (F ∆G) ≤ P (F ∆H) + P (H∆G)

(1.34)

Proof. 1. Since G ⊂ F , P (F ) = P (F ∩ G) + P (F ∩ Gc ) = P (G) + P (F − G). 2. Follows from the previous part and nonnegativity of probability. 3. Use Lemma 1.5, additivity, and monotonicity to write for all n P(

n [ i=1

Fi ) = P (

n [

i=1

Gi ) =

n X i=1

P (Gi ) ≤

n X

P (Fi ).

(1.35)

i=1

Taking the limit as n → ∞ the result follows from the continuity of probability and the fact that the right hand side is monotonically nondecreasing and hence has a limit. 4. Since F ∩ G ⊂ F ∪ G, P (F ∆G) = P (F ∪ G − F ∩ G) = P (F ∪ G) − P (F ∩ G) and P (F ∪ G) = P (F ) + P (G) − P (F ∩ G), we have

1.6 Probability Spaces

23

P (F ∆G) = P (F ) + P (G) − 2P (F ∩ G) ≥ P (F ) + P (G) − 2 min(P (F )P (G)) = | P (F ) − P (G) | 5. Eq. (1.31) follows from (1.14) and the first part of the lemma. Eq. (1.32) follows from (1.14) and the third axiom of probability. Eq. (1.31) follows from (1.31) and (1.37). 6. F ∆G = (F ∩ Gc ) ∪ (F c ∩ G) = (F ∩ Gc ∩ H) ∪ (F ∩ Gc ∩ H c ) ∪ (F c ∩ G ∩ H) ∪ (F c ∩ G ∩ H c ) = (F ∩ Gc ∩ H c ) ∪ (F c ∩ G ∩ H) ∪ (F ∩ Gc ∩ H) ∪ (F c ∩ G ∩ H c ) ⊂ (F ∩ H c ) ∪ (F c ∩ H) ∪ (Gc ∩ H) ∪ (G ∩ H c ) = F ∆H ∪ H∆G and therefore from the union bound P (F ∆G) = P (F ∆H ∪ H∆G) ≤ P (F ∆H) + P (H∆G).

2

Approximation of Events and Probabilities Lemma 1.13 yields some useful notions of approximation for events and probability measures. First consider the triangle inequality for the probability of symmetric differences. Example 1.15. Given a probability space (Ω, B, P ) define the function dP : B × B → [0, 1] by dP (F , G) = P (F ∆G). (1.36) Then dP is a pseudometric and (B, dP ) is a pseudometric space. The function dP is only a pseudometric because the distance is 0 if the two events differ by a null set. We could construct an ordinary metric space by replacing B by the space of equivalence classes of its members, that is, by identifying any to events which differ by a null set. The pseudometric space (B, dP ) will be referred to as the pseudometric space associated with the probability space (Ω, B, P ) or simply as the associated pseudometric space. A similar metric space based on probabilities is the partition distance between two measurable partitions of a probability space. Given a measurable space (Ω, B) and an index set I (such as Z(n)), P = {Pi ; i ∈ I} is a

24

1 Probability Spaces

S (measurable) partition of Ω if the Pi are disjoint, i∈I Pi = Ω, and Pi ∈ B for all i ∈ I. The sets Pi are called the atoms of the partition. Example 1.16. Suppose that (Ω, B, µ) is a probability space. Let P denote the class of all measurable partitions of Ω with K atoms, that is, a member of P has the form P = {Pk , i = 1, 2, . . . , K}, where the Pi ∈ B are disjoint. A metric on the space is d(P , Q) =

K X

µ(Pi ∆Qi ).

i=1

This metric is called the partition distance. As previously, strictly speaking d is a pseudometric.

Exercises 1. Prove that if P satisfies (1.19), (1.20), and (1.22), then (1.23)-(1.25) are equivalent, that is, any one holds if and only if the other two also hold. 2. Prove the following elementary properties of probability (all sets are assumed to be events). a. P (F

[

G) = P (F ) + P (G) − P (F ∩ G).

(1.37)

b. P (F c ) = 1 − P (F ). c. For all events F , P (F ) ≤ 1. d. If Fi , i = 1,T2, . . . is a sequence of events that all have probability 1, show that i Fi also has probability 1. 3. Suppose that Pi , i = 1, 2, . . . is a countable family of probability measures on a space (Ω, B) and that ai , i = 1, 2, . . . is a sequence of positive real numbers that sums to one. Show that the set function m defined by ∞ X m(F ) = ai Pi (F ) i=1

is also a probability measure on (Ω, B). This is an example of a mixture probability measure. 4. Show that the partition distance in Example 1.16 satisfies X X d(P , Q) = 2 m(Pi ∩ Qj ) = 2(1 − m(Pi ∩ Qi )). i6=j

i

1.7 Complete Probability Spaces

25

1.7 Complete Probability Spaces Completion is a technical detail which we will ignore most of the time, but which needs to be considered occasionally because it is required to explain differences among the most important probability spaces, especially the standard Borel spaces emphasized here and the Lebesgue spaces that are ubiquitous in ergodic theory. Completion is literally a topic of much ado about nothing — how to treat possibly nonmeasurable subsets of sets of measure zero. The issue will take up space here and in the development of the extension theorem in order to have a reference point for avoiding the issue later, save for occasional comparisons. Doubtless mathematicians may find the discussion belaboring the obvious and engineers will wonder why it is raised at all, but I have found the subject sufficiently annoying at times to try to clarify it in context and thereby to avoid the appearance in being too cavalier where the subject does not crop up. By way of motivation, completion yields a bigger collection of events when one has a specific measure or candidate measure, but not considering completion allows consideration of important properties of a measurable space common to all measures. Usually our emphasis is on the latter, but there are a two named specific measures, the Borel measure and the Lebesgue measure, which deserve special consideration, and that requires the notion of complete probability spaces. Given a probability space (Ω, B, P ), an event F ∈ B is called a null set (with respect to P ) or a P -null set if P (F ) = 0. A probability space is said to be complete if all subsets of null sets are themselves in B. Sometimes it is the σ -field B that is said to be complete, but completeness depends on the measure and not only on the σ -field. Any probability space that is not complete can be made complete (completed) by simply incorporating all subsets of null sets in a suitable manner, as described in the following lemma. Lemma 1.14. Given a probability space (Ω, B, P ), define BP as the collection of all sets of the form F ∪N, where F ∈ B and N is a subset of a P -null set. Then BP is complete, contains B, and is the smallest σ -field with these properties. The probability measure P extends to a probability measure P on the completed space (Ω, BP ) so that P (F ) = P (F ) for all F in the original σ -field B. Proof. Since ∅ is a subset of all sets including null sets, B ⊂ BP . By construction BP is closed under countable unions. To see that BP is also closed under complementation and hence is a σ -field, first observe that if F ∈ B and N is a subset of a null set G, then F ∪ N = (F ∪ G) − [G ∩ (F ∪ N)c ] = F 0 − G0 = F ∩ (G0 )c where F 0 = F ∪ G ∈ B and G0 = G ∩ (F ∪ N)c is a subset of a null set. Thus (F ∪ N)c = (F 0 )c ∪ G0 ∈ BP .

26

1 Probability Spaces

¯ is a complete σ -field that contains B, then it must contain all If B subsets of P -null sets and hence all sets of the form F ∪ N for F ∈ B and ¯ must contain BP . Therefore BP is the N a subset of a P -null set, thus B smallest σ -field with the desired properties. Define the set function P on BP by P (F ∪ N) = P (F ). To see that the set function is well defined, suppose that G = F ∪ N = F 0 ∪ N 0 with F , F 0 ∈ B, N, N 0 subsets of P null sets, say M and M 0 , respectively. Then the symmetric difference F ∆F 0 ∈ B since F , F 0 ∈ B, and F ∆F 0 = (F ∩ N 0 ) ∪ (F 0 ∩ N) ⊂ (F ∩ M 0 ) ∪ (F 0 ∩ M) ⊂ M ∪ M 0, which is a null set. Hence P (F ∆F 0 ) = 0. From (1.30) this implies that P (F ) = P (F 0 ), and hence the set function P¯ is well-defined. By construction P is countably additive and normalized and hence is a probability measure Also by construction, P is complete and for every set F ∈ B, P (F ) = P (F ) so that the two probability measures agree on sets for which both are defined. 2 The complete probability space (Ω, BP , P ) is called the completion of (Ω, B, P ). It is common practice to drop the overbar and superscript P and simply use the same notation for the completed space as for the original space unless it is useful to distinguish the two spaces. There is no effect of completion on the computation of probabilities of interest, and in a similar vein completion will not effect the computation of expectations encountered later. It does effect the measurability of sets and, as described later, of functions. As mentioned earlier, the primary inconvenience of dealing with complete probability spaces is that the details depend on the specific measure. We will often be concerned with constructing measures and hence will be dealing with with measurable spaces initially that do not yet have a measure with respect to which completion is defined. Hence the issue of completion is usually ignored until or unless it is needed, at which point we can complete the probability space with respect to a specific measure.

1.8 Extension Event spaces or σ -fields can be generating from a class of sets, for example a simple field of subsets of Ω. In this section a similar method is used to find a probability measure on an event space by extending a nonnegative countably additive normalized set function on a field. This solves the first and more traditional step of constructing a probability

1.8 Extension

27

measure. It leaves the more difficult problem for later — how to demonstrate that a nonnegative countably additive normalized set function on a field (usually easy to demonstrate) extends to a probability measure on the σ -field generated by the field. This first step is accomplished by the Carathéodory extension theorem. The theorem states that if we have a probability measure on a field, then there exists a unique probability measure on the σ -field that agrees with the given probability measure on events in the field. We shall develop the result in a series of steps. The development is patterned on that of Halmos [45]. Suppose that F is a field of subsets of a space Ω and that P is a probability measure on a field F ; that is, P is a nonnegative, normalized, countably additive set function when confined to sets in F . We wish to obtain a probability measure, say λ, on σ (F ) with the property that for all F ∈ F , λ(F ) = P (F ). Eventually we will also wish to demonstrate that there is only one such λ. Toward this end define the set function X λ(F ) = inf S P (Fi ). (1.38) {Fi }∈F ,F ⊂

i

Fi

i

The infimum is over all countable collections of field elements whose unions contain the set F . Such a collection of field members whose union contains F is called a cover of F . From Lemma 1.5 we could confine interest to covers whose members are all disjoint. Observe that this set function is defined for all subsets of Ω. From the definition, given any set F and any  > 0, there exists a cover {Fi } such that X X P (Fi ) −  ≤ λ(F ) ≤ P (Fi ). (1.39) i

i

A cover satisfying (1.39) will be called an -cover for F . The goal is to show that λ is in fact a probability measure on σ (F ). Obviously λ is nonnegative, so it must be shown to be normalized and countably additive. This is accomplished in a series of steps, beginning with the simplest: Lemma 1.15. The set function λ of (1.38) satisfies (a) (b) (c)

λ(∅) = 0. Monotonicity: If F ⊂ G, then λ(F ) ≤ λ(G). Subadditivity: For any two sets F , G, λ(F ∪ G) ≤ λ(F ) + λ(G).

(d)

(1.40)

Countable Subadditivity: Given sets {Fi }, λ(∪i Fi ) ≤

∞ X i=1

λ(Fi ).

(1.41)

28

1 Probability Spaces

Proof. Property (a) follows immediately from the definition since ∅ ∈ F and contains itself, hence λ(∅) ≤ P (∅) = 0. From (1.39) given G and  we can choose a cover {Gn } for G such that X P (Gi ) ≤ λ(G) + . i

Since F ⊂ G, a cover for G is also a cover for F and hence X P (Gi ) ≤ λ(G) + . λ(F ) ≤ i

Since  is arbitrary, (b) is proved. To prove (c) let S {Fi } and {Gi } be /2 S covers for F and G. Then {Fi Gi } is a cover for F G and hence X X X P (Fi ∪ Gi ) ≤ P (Fi ) + P (Gi ) ≤ λ(F ) + λ(G) + . λ(F ∪ G) ≤ i

i

i

Since  is arbitrary, this proves (c). −i To prove (d), for each Fi let {Fik ; k = 1, 2, . . .} be S an 2 cover for Fi . Then {Fik ; i = 1, 2, . . . ; j = 1, 2, . . .} is a cover for i Fi and hence λ(∪i Fi ) ≤

∞ ∞ X X

P (Fik ) ≤

i=1 k=1

∞ X

(λ(Fi ) + 2−i ) =

i=1

∞ X

λ(Fi ) + ,

i=1

which completes the proof since  is arbitrary. 2 We note in passing that a set function λ on a collection of sets having properties (a)-(d) of the lemma is called an outer measure on the collection of sets. The simple properties have an immediate corollary: the set function λ agrees with P on field events and hence it makes sense to call λ an extension of P . Corollary 1.3. If F ∈ F , then λ(F ) = P (F ). Thus, for example, λ(Ω) = 1. Proof. Since a set covers itself, we have immediately that λ(F ) ≤ P (F ) for all field events F . Suppose that {Fi } is an  cover for F . Then X X λ(F ) ≥ P (Fi ) −  ≥ P (F ∩ Fi ) − . i

S

i

S

S Since F ∩ Fi ∈ F and F ⊂ i Fi , i (F ∩ Fi ) = F ∩ i Fi = F ∈ F and hence, invoking the countable additivity of P on F , X P (F ∩ Fi ) = P (∪i F ∩ Fi ) = P (F ) i

and hence λ(F ) ≥ P (F ) − .

1.8 Extension

29

2 Thus far the properties of λ have depended primarily on manipulating the definition. In order to show that λ is countably additive (or even only finitely additive) over σ (F ), a new concept is required which implies new collection of sets that we will later see contains σ (F ). The definitions may seem artificial, but some similar form of tool seems to be necessary to get to the desired goal. By way of motivation, we are trying to show that a set function is additive on some class of sets. Perhaps the simplest form of additivity looks like λ(F ) = λ(F ∩ R) + λ(F ∩ R c ). Hence it should not seem too strange to build at least this form into the class of sets considered. To do this, define a set R ∈ σ (F ) to be λ-measurable if λ(F ) = λ(F ∩ R) + λ(F ∩ R c ), all F ∈ σ (F ). In words, a set R is λ-measurable if it splits all events in σ (F ) in an additive fashion. This property is not determined by the σ -field alone, it depends on how the outer measure λ behaves on members of σ (F ). Let H denote the collection of all λ-measurable sets. We shall see that indeed λ is countably additive on the collection H and that H contains σ (F ). Observe for later use that since λ is subadditive, to prove that R ∈ H requires only that we prove λ(F ) ≥ λ(F ∩ R) + λ(F ∩ R c ), all F ∈ σ (F ). Lemma 1.16. H is a field. Proof. Clearly Ω ∈ H since λ(∅) = 0 and Ω ∩ F = F . Equally clearly, F ∈ H implies that S F c ∈ H . The only work required is show that if F , G ∈ H , then also F G ∈ H ç. To accomplish this begin by recalling that for F , G ∈ H and for any H ∈ σ (F ) we have that λ(H) = λ(H ∩ F ) + λ(H ∩ F c ) and hence since both arguments on the right are members of σ (F ) λ(H) = λ(H ∩ F ∩ G) + λ(H ∩ F ∩ Gc ) + λ(H ∩ F c ∩ G) + λ(H ∩ F c ∩ Gc ).

(1.42)

As S(1.42) is valid for any event H ∈ σ (F ), we can replace H by H ∩ (F G) to obtain

30

1 Probability Spaces

λ(H ∩ (F ∪ G)) = λ(H ∩ (F ∩ G)) + λ(H ∩ (F ∩ Gc )) + λ(H ∩ (F c ∩ G)),

(1.43)

where the fourth term has disappeared since the argument is null. Plugging (1.43) into (1.42) yields λ(H) = λ(H ∩ (F ∪ G)) + λ(H ∩ (F c ∩ Gc )) which implies that F

S

G ∈ H since F c ∩ Gc = (F

S

G)c .

2

Lemma 1.17. H is a σ -field. Proof. Suppose that Fn ∈ H , n = 1, 2, . . . and ∞ [

F=

Fi .

i=1

It is convenient to work with disjoint sets, so use the decomposition of Sk−1 Lemma 1.5 and define as in (1.16) Gk = Fk − i=1 Fi , k = 1, 2, . . . Since H is a field, clearly the Gi and all finite unions of Fi or Gi are also in H . First apply (1.43) with G1 and G2 replacing F and G. Using the fact that G1 and G2 are disjoint, λ(H ∩ (G1 ∩ G2 )) = λ(H ∩ G1 ) + λ(H ∩ G2 ). Using this equation and mathematical induction yields λ(H ∩

n [

Gi ) =

i=1

n X

λ(H ∩ Gi ).

(1.44)

i=1

For convenience define the set F (n) =

n [ i=1

Fi =

n [

Gi .

i=1

Since F (n) ∈ H and since H is a field, F (n)c ∈ H and hence for any event H ∈ H λ(H) = λ(H ∩ F (n)) + λ(H ∩ F (n)c ). Using the preceding formula, (1.44), the fact that F (n) ⊂ F and hence F c ⊂ F (n)c , the monotonicity of λ, and the countable subadditivity of λ, λ(H) ≥

∞ X

λ(H ∩ Gi ) + λ(H ∩ F c ) ≥ λ(H ∩ F ) + λ(H ∩ F c ).

(1.45)

i=1

Since H is an arbitrary event in H , this proves that the countable union F is in H and hence that H is a σ -field. 2

1.8 Extension

31

In fact much more was proved than the stated lemma; the implicit conclusion is made explicit in the following corollary. Corollary 1.4. λ is countably additive on H Proof. Take Gn to be an arbitrary sequence of disjoint events in H in the preceding proof (instead of obtaining them as differences of another arbitrary sequence) and let F denote the union of the Gn . Then from the lemma F ∈ H and (1.45) with H = F implies that X λ(F ) ≥ λ(Gi ). i=1

Since λ is countably subadditive, the inequality must be an equality, proving the corollary. 2 The strange class H is next shown to be exactly the σ -field generated by F . Lemma 1.18. H = σ (F ). Proof. Since the members of H were all chosen from σ (F ), H ⊂ σ (F ). Let F ∈ F and G ∈ σ (F ) and let {Gn } be an -cover for G. Then λ(G) +  ≥

∞ X

P (Gn ) =

i=1

∞ X

(P (Gn ∩ F ) + P (Gn ∩ F c ))

i=1

≥ λ(G ∩ F ) + λ(G ∩ F c ) which implies that for F ∈ F λ(G) ≥ λ(G ∩ F ) + λ(G ∩ F c ) for all events G ∈ σ (F ) and hence that F ∈ H . This implies that H contains F . Since H is a σ -field, however, it therefore must contain σ (F ). 2 It has now been proved that λ is a probability measure on σ (F ) which agrees with P on F . The only remaining question is whether it is unique. The following lemma resolves this issue. Lemma 1.19. If two probability measures λ and µ on σ (F ) satisfy λ(F ) = µ(F ) for all F in the field F , then the measures are identical; that is, λ(F ) = µ(F ) for all F ∈ σ (F ). Proof. Let M denote the collection of all sets F such that λ(F ) = µ(F ). From the continuity of probability, M is a monotone class. Since it contains F , it must contain the smallest monotone class containing F . That class is σ (F ) from Lemma 1.6 and hence M contains the entire σ -field.

2 Combining all of the pieces of this section yields the principal result.

32

1 Probability Spaces

Theorem 1.1. The Carathéodory Extension Theorem. If a set function P satisfies the properties of a probability measure (nonnegativity, normalization, and countable additivity or finite additivity plus continuity) for all sets in a field F of subsets of a space Ω, then there exists a unique measure λ on the measurable space (Ω, σ (F )) that agrees with P on F . When no confusion is possible, we will usually denote the extension of a measure P by P rather than by a different symbol. This section is closed with an important corollary to the extension theorem that shows that an arbitrary event can be approximated closely by a field event in a probabilistic way. Corollary 1.5. (Approximation Theorem) Given a probability space (Ω, B, P ) and a generating field F , that is, B = σ (F ), then given F ∈ B and  > 0, there exists an F0 ∈ F such that P (F ∆F0 ) ≤ . Proof. From the definition of λ in (1.38) (which yielded the extension of the original measure P ) and the ensuing discussion one can find S a countable collection of disjoint field events {Fn } such that F ⊂ n Fn and ∞ ∞ X X P (Fi ) − /2 ≤ λ(F ) ≤ P (Fi ). i=1

i=1

Renaming λ as P P (F ∆

∞ [ i=1

Fn ) = P (

∞ [

Fn − F ) = P (

i=1

∞ [

Fn ) − P (F ) ≤

i=1

 . 2

Sn

Choose a finite union F˜ = i=1 Fi with n chosen large enough to ensure that ∞ ∞ ∞ [ X [  P( Fi ∆F˜) = P ( Fi − F˜) = P (Fi ) ≤ . 2 i=1 i=1 i=n+1 From the previous lemma we see that P (F ∆F˜) ≤ P (F ∆

∞ [ i=1

Fi ) + P (

∞ [

Fi ∆F˜) ≤ .

i=1

2 If a field F is used to generate the σ -field, then the approximation theorem says that for a probability space (Ω, σ (F ), P ), any event can be approximated arbitrarily closely in the sense of probability of symmetric difference and hence in the pseudometric dP of the associated pseudometric space (σ (F ), P ) (see Example 1.15) by one of a countable collection of “points” in σ (F ), that is, by a member of F . If this were a metric space, this would mean that the metric space is separable. Extending the

1.8 Extension

33

definition of separability to pseudometric spaces (or replacing the pseudometric space by a metric space of equivalence classes) show that if a σ -field is countably generated, then it is separable with respect to dP .

Exercise 1. Show that if m and p are two probability measures on (Ω, σ (F )), where F is a field, then given an arbitrary event F and  > 0 there is a field event F0 ∈ F such that m(F ∆F0 ) ≤  and p(F ∆F0 ) ≤ .

“This page left intentionally blank.”

Chapter 2

Random Processes and Dynamical Systems

Abstract The theoretical fundamentals random variables, random processes, and dynamical systems are developed and elaborated. The emphasis is on discrete time, where the general notion of a dynamical system — a probability space along with a transformation — yields a random process when combined with a random variable. The transformation might, but need not, correspond to a time shift operation. Continuous time processes are discussed briefly by considering a family of transformations or flow. Probability distributions describing the variables and processes are related using product σ -fields. The basic ideas of codes, filters, factors, and isomorphism are introduced.

2.1 Measurable Functions and Random Variables Given a measurable space (Ω, B), let (A, BA ) denote another measurable space. The first space can be thought of as an input space and the second as an output space. A random variable or measurable function defined on (Ω, B) and taking values in (A, BA ) is a mapping or function f : Ω → A (or f : (Ω, B) → (A, BA ) when it helps to make the σ -fields explicit) with the property that if F ∈ BA , then f −1 (F ) = {ω : f (ω) ∈ F } ∈ B.

(2.1)

The name random variable is commonly associated with the case where A is the real line R and B = B(R), the Borel field of subsets of the real line. Occasionally a more general sounding name such as random object is used for a measurable function to include implicitly real-valued random variables (A = R, the real line), real-valued random vectors (A = Rk , k-dimensional Euclidean space), and real-valued random processes (A a sequence or waveform space), as well as non necessarily real-valued ran-

R.M. Gray, Probability, Random Processes, and Ergodic Properties, DOI: 10.1007/978-1-4419-1090-5_2, © Springer Science + Business Media, LLC 2009

35

36

2 Random Processes and Dynamical Systems

dom variables, vectors, and processes. We will usually use the term random variable in the general sense of including both scalar- and vectorvalued functions, providing clarification when needed or helpful. When it is not clear from context, random variables etc. will be said to be realvalued if the space A is either a subset of the real line or a product space of such subsets. A random variable is just a function or mapping with the property that inverse images of output events determined by the random variable are events in the original measurable space (input events). This simple property ensures that the output of the random variable will inherit its own probability measure. For example, with the probability measure Pf defined by Pf (B) = P (f −1 (B)) = P ({ω : f (ω) ∈ B}); B ∈ BA , (A, BA , Pf ) becomes a probability space since measurability of f and elementary set theory ensure that Pf is indeed a probability measure. The induced probability measure Pf is called the distribution of the random variable f . The measurable space (A, BA ) or, simply, the sample space A is called the alphabet of the random variable f . We shall occasionally also use the abbreviated notation P f −1 for P f −1 (F ) = P (f −1 (F )). The shorter notation is less awkward when f itself is a function with a complicated name, e.g., ΠT→M . If the alphabet A of a random variable f is not clear from context, then we shall refer to f as an A-valued random variable. If f is a measurable function from (Ω, B) to (A, BA ), we will say that f is B/BA -measurable. The names of the σ -fields may be omitted if they are clear from context. If the alphabet A of a random variable f is a discrete set, then the random variable is called a discrete-alphabet or discrete random variable and its distribution Pf is completely described by a probability mass function.

Composite Mappings Given a probability space (Ω, B), a measurable function f defined on that space, and the induced output probability space (A, BA ) for the measurable function, a new random variable g can be defined on the output probability space (A, BA ) with its own output measurable space (B, BB ). The composite or cascade mapping g ◦ f : (Ω, B) → (B, BB ) defined by g ◦ f (ω) = g(f (ω)) is then a measurable function on the original space since for F ∈ BB we have that (g ◦ f )−1 (F ) = g −1 (f −1 (F )) ∈ B since the individual functions are measurable. When it is clear from context that

2.1 Measurable Functions and Random Variables

37

composite functions and not ordinary products are being considered, the notation will be abbreviated to gf = g ◦ f .

Continuous Mappings Suppose that we have two Borel spaces (A, B(A)) and (B, B(B)) with metrics d and ρ, respectively. A function f : A → B is continuous at a in A if for each  > 0 there exists a δ > 0 such that d(x, a) < δ implies that ρ(f (x), f (a)) < . It is an easy exercise to show that if a is a limit point, then f is continuous at a if and only if an → a implies that f (an ) → f (a). A function is continuous if it is continuous at every point in A. The following lemma collects the main properties of continuous functions: a definition in terms of open sets and their measurability. Lemma 2.1. A function f is continuous if and only if the inverse image f −1 (S) of every open set S in B is open in A. A continuous function is Borel measurable. Proof. Assume that f is continuous and S is an open set in B. To prove f −1 (S) is open we must show that about each point x in f −1 (S) there is an open sphere Sr (x) contained in f −1 (S). Since S is open we can find an s such that ρ(f (x), y) < s implies that y is in S. By continuity we can find an r such that d(x, b) < r implies that d(f (x), f (b)) < s. If f (x) and f (b) are in S, however, x and b are in the inverse image and hence Sr (x) so obtained is also in the inverse image. Conversely, suppose that f −1 (S) is open for every open set S in B. Given x in A and  > 0 define the open sphere S = S (f (x)) in B. By assumption f −1 (S) is then also open, hence there is a δ > 0 such that d(x, y) < δ implies that y ∈ f −1 (S) and hence f (y) ∈ S and hence ρ(f (x), f (y)) < , completing the proof. 2 Since inverse images of open sets are open and hence Borel sets and since open sets generate the Borel sets, f is measurable from Lemma 2.4.

Random Variables and Completion How is measurability of a function affected by completing a probability space. Happily things behave as they should. Suppose that f : (Ω, B) → (A, BA ) is a random variable and that P is a probability measure on (Ω, B) and Pf = P f −1 is the induced probability measure on (A, BA ). Suppose that B is replaced by the larger σ -field BP , the completion of B with respect to P , and let P¯ denote the completed measure. Since B ⊂ BP ,

38

2 Random Processes and Dynamical Systems

the function f is automatically BP /BA -measurable. If the output event space is similarly completed with respect to P¯f to form BPA , then f is also BP /BPA -measurable. This follows since if F ∈ BPA , it must have the form F = F 0 ∪ N for F 0 ∈ BA and N a subset of a Pf null set, say M ∈ BA , Pf (M) = 0. Then f −1 (F ) = f −1 (F 0 ∪ N) = f −1 (F 0 ) ∪ f −1 (N) since inverse images preserve set theoretic operations. Preservation of set theoretic operations also means that f −1 (N) ⊂ f −1 (M). But P (f −1 (M)) = Pf (M) = 0, and hence f −1 (N) is a subset of a P null set and hence f −1 (F 0 ) ∪ f −1 (N) ∈ BP .

Exercises 1. Let f : Ω → A be a random variable. Prove the following properties of inverse images:  c f −1 (Gc ) = f −1 (G) f −1 (

∞ [ k=1

Gk ) =

∞ [

f −1 (Gk )

k=1

If G ∩ F = ∅, then f −1 (G) ∩ f −1 (F ) = ∅ If G ⊂ F , then f −1 (G) ⊂ f −1 (F ). Thus inverse images preserve set theoretic operations and relations. 2. If F ∈ B, show that the indicator function 1F defined by 1F (x) = 1 if x ∈ F and 0 otherwise is a random variable. Describe its distribution. Is the product of indicator functions measurable? 3. Show that for two events F and G, |1F (x) − 1G (x)| = 1F ∆G (x).

2.2 Approximation of Random Variables and Distributions Two measurements or random variables defined on a common probability space are considered to be the same if their distributions are the same. Suppose the distributions are not the same, then how does one quantify how different the two random variables are? Several approaches are possible, but only one of which will extend to random processes in a useful way. Example 2.1. Suppose that X and Y are two A-valued random variables defined on a common probability space (Ω, B, m), where A is discrete.

2.2 Approximation of Random Variables and Distributions

39

Define the distance between X and Y as the probability that they are different, that is, d(X, Y ) = Pr(X ≠ Y ) = m({ω : X(ω) ≠ Y (ω)}). This is a pseudometric, but it can be turned into a metric in the usual way by identifying random variables that are equal with probability one. This is just the partition metric of Example 1.16 in disguise. Define the partitions P = {Pa ; a ∈ A} and Q = {Qa ; a ∈ A} by Pa = {ω : X(ω) = a} = X −1 (a)} Qa = {ω : Y (ω) = a} = Y −1 (a)}. (2.2) The partition distance is given by X X c m(Pa ∆Qa ) = m((Pa ∩ Qa ) ∪ (Pac ∩ Qa )) dpartition (P , Q) = a∈A

=

X

a∈A

m(Pa ∩

c Qa )

+ m(Pac ∩ Qa ))




a∈A

=

X

c m(Pa ∩ Qa )+

a∈A

X

m(Pac ∩ Qa )

a∈A

c ) + m(∪a∈A Pac ∩ Qa ) = m(∪a∈A Pa ∩ Qa

and therefore d(X, Y ) = m({ω : X(ω) ≠ Y (ω)}) X m({ω : X(ω) = a, Y (ω) = b}) = a∈A,b∈A:a≠b

X

=

X

m({ω : X(ω) = a, Y (ω) = b})

a∈A b∈A:a≠b

X

=

m(Pa ∩ ∪b∈A:a≠b Qa ) =

a∈A

X

c m(Pa ∩ Qa )

a∈A

c ) = m(∪a∈A Pa ∩ Qa

and a similar line of equalities shows that d(X, Y ) = m(∪a∈A Pac ∩ Qa ), thus X d(X, Y ) = m({ω : X(ω) = a, Y (ω) = b}) a∈A,b∈A:a≠b

=

1 X 1 dpartition (P , Q) = m(Pa ∆Qa ). 2 2 a∈A

(2.3)

The previous example provides a metric on two random variables defined on a common probability space. We will also be interested in a distance between two random variables or their distributions when they are not defined on a common space. Example 2.2. Suppose that X and Y are two random variables with a common discrete alphabet A with event space BA and distributions PX and

40

2 Random Processes and Dynamical Systems

PY , respectively. Suppose one has a choice of all possible joint distributions PX,Y having PX and PY as marginals, that is, PX,Y (F × A) = PX (F ) and PX,Y (A × F ) = PY (F ) for all F ∈ BA . What choice minimizes the metric d(X, Y ) of the previous example? This distance between two distributions of random variables is formalized as ¯ X , PY ) = d(P

inf

PX,Y ⇒PX ,PY

PX,Y (X ≠ Y ).

(2.4)

This type of distance has a long history and many names and it will be treated in some depth in later chapters. Example 2.3. Another distance on the space of probability measure on the discrete measurable space (A, BA ) of Example 2.2 is the `1 norm of the difference of the two distributions, X | PX − PY |= | pX (a) − pY (a) |, (2.5) a∈A

where lower case denotes the probability mass function for the discrete distribution, that is, pX (a) = PX ({a}). This distance is called the distribution distance.. As we shall later see in Lemma 9.5, it is closely related to the variation distance or total variation distance. ¯ X , PY ) = |PX − PY |/2. It will later be seen in Lemma 9.5 that d(P

2.3 Random Processes and Dynamical Systems We now consider two mathematical models for a random process. The first is the familiar one in elementary and advanced courses: a random process is just an indexed family {Xt ; t ∈ T} of random variables, which we will abbreviate to X when context makes it clear that we are considering random processes. The second model is less familiar in the engineering literature: a random process can be constructed from a single random variable together with an abstract dynamical system consisting of a probability space and a family of transformations on the space. The two models are connected by considering transformations as time shifts, but examples will show that other transformations can be useful. The dynamical system framework will be seen to provide a general description of random processes that provides additional insight into their structure and a natural structure for stating and proving ergodic theorems.

2.3 Random Processes and Dynamical Systems

41

Random Processes A random process is a family of random variables {Xt }t∈T or {Xt ; t ∈ T} defined on a common probability space (Ω, B, P ), where T is an index set. We assume that all of the random variables Xt share a common alphabet A. If T is a discrete set such as Z or Z+ , the process is said to be a discrete time or discrete parameter random process. If T is a continuous set such as R or R+ , the process is said to be a continuous time or continuous parameter random process . If the index set T = Z or R, the process is said to be two-sided (or double-sided or bilateral. If the index set T = Z+ or R+ , the process is said to be one-sided (or singlesided or unilateral. One-sided random processes will usually prove to be more difficult in theory, but they provide better models for physical random processes that must be “turned on” at some time or that have transient behavior. For a fixed ω ∈ Ω, the {Xt (ω); t ∈ T} is called a sample sequence if T is discrete and a sample function or sample waveform if T is continuous These are by far the most common and important examples of index sets, and usually the index corresponds to time as measured in some units. Most of the theory will remain valid, however, if we simply require that T be closed under addition, that is, if t and s are in T, then so is t + s (where the “+” denotes a suitably defined addition in the index set). Mathematically, we assume that T is a semi-group with respect to addition, that is, T is closed with respect to addition and addition is associative. In the two-sided case, T becomes a group since there is an identity with respect to addition (0) and an inverse. We henceforth make this assumption. In general T need not correspond to time or even be one-dimensional. For example T = R2 or Z2 can be used to define random fields for use in modeling random images. We abbreviate a random process to {Xt } or even Xt or X, when the context makes clear the index set and that a process rather than a single random variable is being considered. In the discrete time case, we will usually write Xn . In the continuous time case the notation X(t) is common, and in the discrete time case X[n] is often encountered in the engineering literature. Since the alphabet A is general, a continuous time random processes could also be modeled as a discrete time random process by letting A consist of a family of waveforms defined on an interval, e.g., each random variable Xn could be in fact a continuous time waveform X(t) for t ∈ [nT , (n + 1)T ), where T is some fixed positive real number. In the special case where T be a finite set, random vector is a better name than random process. As a final technical point, a random process can be viewed as a function of two arguments, Xt (ω) or X : Ω × T → A. We will assume that the process is measurable as a function of two arguments with respect to

42

2 Random Processes and Dynamical Systems

suitably defined σ -fields (except possibly for a set of total probability 1, as usual we ignore null sets). This assumption is unnecessary in the discrete time case as simply having a family of random variables guarantees measurability. But it is needed in the continuous time case, primarily in RT order to make sense of time integrals such as 0 Xt dt. We will assume all continuous time random processes encountered herein are measurable, leaving proofs to the literature.

Discrete-Time Dynamical Systems: Transformations A discrete-time abstract dynamical system consists of a probability space (Ω, B, P ) together with a measurable transformation T : Ω → Ω of Ω into itself. Measurability means that if F ∈ B, then also T −1 F = {ω : T ω ∈ F } ∈ B. The quadruple (Ω, B, P , T ) is called a (discrete-time) dynamical system in ergodic theory. The interested reader can find excellent introductions to classical ergodic theory and dynamical system theory in the books of Halmos [46] and Sinai [101]. More complete treatments may be found in [3] [97] [84] [17] [113] [78] [26] [62]. The name dynamical systems comes from the focus of the theory on the long term dynamics or dynamical behavior of repeated applications of the transformation T on the underlying measure space. An alternative to modeling a random process as a sequence or family of random variables defined on a common probability space is to consider a single random variable together with a transformation defined on the underlying probability space. The outputs of the random process will then be values of the random variable taken on transformed points in the original space. The transformation will usually be related to shifting in time, and hence this viewpoint will focus on the action of time itself. Suppose now that T is a measurable mapping of points of the sample space Ω into itself. It is easy to see that the cascade or composition of measurable functions is also measurable. Hence the transformation T n defined as T 2 ω = T (T ω) and so on (T n ω = T (T n−1 ω)) is a measurable function for all positive integers n. The sequence {T n ω; n ∈ T} is called the orbit of ω ∈ Ω. If f is an A-valued random variable defined on (Ω, B), then the functions f T n : Ω → A defined by f T n (ω) = f (T n ω) for ω ∈ Ω will also be random variables for all n in Z+ . Thus a dynamical system together with a random variable or measurable function f defines a single-sided discrete-time random process {Xn }n∈Z+ by Xn (ω) = f (T n ω). Note in particular that setting n = 0 (assuming of course that 0 ∈ T) yields X0 (ω) = f (ω)

2.3 Random Processes and Dynamical Systems

43

and hence f can be thought of as the zero-time time mapping or projection. If it should be true that T is invertible, that is, T is one-to-one and its inverse T −1 is measurable, then one can define a double-sided random process by Xn = f T n , all n in Z. The most common dynamical system for modeling random processes is that consisting of a sequence space Ω containing all one- or twosided A-valued sequences together with the shift transformation T , that is, the transformation that maps a sequence {xn } into the sequence {xn+1 } wherein each coordinate has been shifted to the left by one time unit. Thus, for example, let Ω = AZ+ = {all x = (x0 , x1 , . . .) with xi ∈ A for all i} and define T : Ω → Ω by T (x0 , x1 , x2 , . . .) = (x1 , x2 , x3 , . . .). The transformation T is called the shift or left shift transformation on the one-sided sequence space. The shift for two-sided spaces is defined similarly. We shall treat this sequence-space representation of a random process in more detail later in this chapter when we consider distributions of random processes. Some interesting dynamical systems in communications applications do not, however, have the structure of a shift on sequences. As an example, suppose that a probability space (Ω, B, P ) has sample space Ω = [0, 1), the unit interval. Define a binary random variable f : Ω → {0, 1} by ( 1 u ∈ [0, 1/2) f (u) = 0 u ∈ [1/2, 1) (this is an example of a quantizer) and a transformation T : Ω → Ω defined by ( 2u u ∈ [0, 1/2) T u = 2u mod 1 = 2u − 1 u ∈ [1/2, 1). The resulting binary alphabet random process is now defined by Xn = f T n. As another example, consider a mathematical model of a device called a sigma-delta modulator, that is used for analog-to-digital conversion, that is, encoding a sequence of real numbers into a binary sequence (analog-to-digital conversion), which is then decoded into a reproduction sequence approximating the original sequence (digital-to-analog conversion) [53] [11] [33]. Given an input sequence {xn } and an initial state u0 , the operation of the encoder is described by the difference equations en = xn − q(un ), un = en−1 + un−1 ,

44

2 Random Processes and Dynamical Systems

where q(u) is +b if its argument is nonnegative and −b otherwise (q is a binary quantizer). The decoder is described by the equation ˆn = x

N 1 X q(un−i ). N i=1

The basic idea of the code’s operation is this: An incoming continuous time, continuous amplitude waveform is sampled at a rate that is so high that the incoming waveform stays fairly constant over N sample times (in engineering parlance the original waveform is oversampled or sampled at many times the Nyquist rate). The binary quantizer then produces outputs for which the average over N samples is very near the ˆkN is a good approximation to the input so that the decoder output x ˆn has only a discrete number input at the corresponding times. Since x of possible values (N + 1 to be exact), one has thereby accomplished analog-to-digital conversion. Because the system involves only a binary quantizer used repeatedly, it is a popular one for microcircuit implementation. As an approximation to a very slowly changing input sequence xn , it is of interest to analyze the response of the system to the special case of a constant input xn = x ∈ [−b, b) for all n (called a quiet input). This can be accomplished by recasting the system as a dynamical system as follows: Given a fixed input x, define the transformation T by ( Tu =

u + x − b;

if u ≥ 0

u + x + b;

if u < 0.

Given a constant input xn = x, n = 1, 2, . . . , N, and an initial condition u0 (which may be fixed or random), the resulting Un sequence is given by u n = T n u0 . If the initial condition u0 is selected at random, then the preceding formula defines a dynamical system which can be analyzed. The example is provided simply to emphasize the fact that time shifts are not the only interesting transformation when modeling communication systems. The different models provide equivalent models for a given process: one emphasizing the sequence of outputs and the other emphasising the action of a transformation on the underlying space in producing these outputs.

2.4 Distributions

45

Continuous-Time Dynamical Systems: Flows A continuous-time abstract dynamical system (Ω, B, P , {Tt ; t ∈ T})is described by a family of measurable transformations {Tt ; t ∈ T}, Tt : Ω → Ω, instead of by a single transformation, where it is assumed that Tt+s = Tt Ts , that is, Tt+s (ω) = Tt (Ts (ω)). The collection {Tt ; t ∈ T} is called a flow. Sometimes it is called a semi-flow or a (semi-)group action of T on Ω if T is one-sided. As in the discrete-time case, the waveform {Tt ω; t ∈ T} is called the orbit of ω ∈ Ω. A continuous-time dynamical system combined with a random variable f yields a continuous-time random process Xt (ω) = f (Tt ω). The discrete-time dynamical system fits into this general definition since the entire family {Tn ; n ∈ T} is determined by the single transformation T = T1 through Tn = T n . The name flow is usually reserved for the continuous-time case, however, and we use the simpler notation (Ω, B, P , T ) instead of (Ω, B, P , {Tt ; t ∈ T}) in the discrete time case. As with continuous-time random processes, we will always make an extra technical assumption when dealing with continuous-time dynamical systems: we assume that the mapping Tt (ω) mapping T × Ω into Ω is measurable. Discrete-time dynamical systems can be constructed from continuoustime dynamical systems by sampling. Given a continuous-time dynamical system (Ω, B, P , {Tt ; t ∈ R}) and a fixed time τ > 0, then (Ω, B, P , Tτ ) is a discrete-time dynamical system which can interpreted as a sampling of the continuous-time system at every τ seconds. Given a random variable f , the continuous-time dynamical system yields a random process {Xt = f Tt , t ∈ R} while the discrete-dynamical system yields a random process {Xn = f Tτn , n ∈ Z}.

2.4 Distributions Although in principle all probabilistic properties of a random process can be determined from the underlying probability space, it is often more convenient to deal with the induced probability measures or distributions on the space of possible outputs of the random process. In particular, this allows us to compare different random processes without regard to the underlying probability spaces and thereby permits us to reasonably equate two random processes if their outputs have the same probabilistic structure, even if the underlying probability spaces are quite different. We have already seen that each random variable Xt of the random process {Xt ; t ∈ T} inherits a distribution because it is measurable. To describe a process, however, we need more than simply probability mea-

46

2 Random Processes and Dynamical Systems

sures on output values of separate single random variables: we require probability measures on families or collections of random variables such as vectors and sequences of random variables formed by sampling the random process. In order to place probability measures on collections of outputs of a random process, we first must construct the appropriate measurable spaces. A convenient technique for accomplishing this is to consider product spaces, spaces for sequences or waveforms formed by concatenating spaces for individual outputs as introduced in Section 1.1

Product Spaces Let T be the index set of the random process {Xt ; t ∈ T}. Denote a sample function or sequence of the random process by x T = {xt ; t ∈ T}. if Xt has alphabet At for t ∈ T, then x T will be in the Cartesian product space ×t∈T At = {all x T : xt ∈ At all t ∈ T}. In all cases of interest in this book, all of the At will be replicas of a single alphabet A and the preceding product will be denoted simply by AT . In a similar fashion we can consider a collection of sample values of the random variables Xt taken for t ∈ S ⊂ T. The most important example of such a subset S will be a finite collection of the form I = (t1 , t2 , . . . , tn ) of n ordered sample times t1 < t2 < · · · < tn , ti ∈ T, i = 1, 2, . . . n. In this case we have a vector x I = (xt1 , xt2 , . . . , xtn ) ∈ AI = An , where the final form An follows since it is simply the Cartesian product of n identical copies of A. In the common special case where I = Z(n) = {0, 1, 2, . . . , n − 1} we abbreviate x Z(n) to x n = (x0 , x1 , . . . , xn−1 ). For the moment, however, it is useful to deal with a more general S ⊂ T which need not be finite or even countable if T is continuous. We allow S = T in the definitions below. To obtain a useful σ -fields for a product space AS = ×t∈S At we introduce the idea of a rectangle in a product space.

2.4 Distributions

47

Rectangles in Product Spaces Suppose that Bt are the coordinate σ -fields for the alphabets At , t ∈ T. A set B ⊂ AS is said to be a (finite-dimensional) rectangle in AS if it has the form B = {x S ∈ AS : xt ∈ Bt ; all t ∈ I}, (2.6) where I is a finite subset of S and Bt ∈ Bt for all t ∈ I. The phrase measurable rectangle is also used. A rectangle as in (2.6) can be written as a finite intersection of onedimensional rectangles as \ B= {x S ∈ AS : xt ∈ Bt }. (2.7) t∈I

As rectangles in AS are clearly fundamental events, they should be members of any useful σ -field of subsets of AS . One approach is simply to define the product σ -field BSA as the smallest σ -field containing all of the rectangles, that is, the collection of sets that contains the clearly important class of rectangles and the minimum amount of other stuff required to make the collection a σ -field. This notion will often prove useful.

Product σ -Fields and Fields Given a set S ⊂ T, let rect(Bt , t ∈ S) denote the set of all rectangles in AS taking coordinate values in sets in Bt , t ∈ S . In other words, every element of rect(Bt , t ∈ S) has the form of (2.6) for some finite set I ⊂ T. Define the product σ -field and the product field of AS by BA S = σ (rect(Bt , t ∈ S))

(2.8)

FA = F (rect(Bt , t ∈ S))

(2.9)

S

The appropriate σ -fields for product spaces have been generated from the class of finite-dimensional rectangles.Although generated σ -fields usually lack simple constructions, the field generated by the rectangles does have a simple constructive definition, which will be extremely useful when constructing probability measures for random processes. The construction is given and proved in the following lemma. Lemma 2.2. Let Bt be the coordinate σ -fields for the alphabets At , t ∈ T. Given a set S ⊂ T, let rect(Bt , t ∈ S) denote the set of all rectangles in AS taking coordinate values in sets in Bt , t ∈ S . In other words, every element of rect(Bt , t ∈ S) has the form of (2.6) for some finite set I ⊂ T.

48

2 Random Processes and Dynamical Systems

Then the field generated by the rectangles, FA S = F (rect(Bt , t ∈ S)) is exactly the collection of all finite disjoint unions of rectangles. Before proving Lemma 2.2, we isolate the key technical step in a separate lemma. Lemma 2.3. Given the setup of the previous lemma, any finite union of rectangles can be written as a finite union of disjoint rectangles. Proof of Lemma 2.3. The proof uses the trick of (1.16) and the heart of the proof is the demonstration that taking differences of rectangles yields rectangles. Suppose that we are given a finite union of rectangles Fi ∈ rect(Bt , t ∈ S); i = 1, 2, . . . N with Fi = {x S ∈ AS : xt ∈ Fi,t ; all t ∈ Ii }; Fi,t ∈ Bt , t ∈ Ii . SN We wish to show that i=1 Fi can be expressed as a finite union of disjoint rectangles. It simplifies notationSto combine the finite collection of index N sets into a single index set I = i=1 Ii . This causes no loss of generality since if for a fixed i an index j ∈ I is not in Ii , then we set Fi,j = A and have the same rectangle with the expanded index set. Thus we can assume Fi = {x S ∈ AS : xt ∈ Fi,t ; all t ∈ I}; Fi,t ∈ Bt , t ∈ I. Define as in (1.16) the difference sets Gn = Fn −

n−1 [ i=1

N [ i=1

Gi =

N [

Fi = Fn −

n−1 [

Gi ; n = 2, . . . , N

i=1

Fi

i=1

To prove the lemma requires only a demonstration that the Gn are themselves rectangles. Consider first n = 2. G2 = F2 − F1 = {x S ∈ AS : xt ∈ F2,t ; all t ∈ I} − {x S ∈ AS : xt ∈ F1,t ; all t ∈ I} = {x S ∈ AS : xt ∈ F2,t − F1,t ; all t ∈ I}, which is a rectangle since F2,t − F1,t ∈ Bt , t ∈ I. In a similar manner, for n = 3, . . . , N,

2.4 Distributions

Gn = Fn −

n−1 [

49

Fi

i=1

= {x S ∈ AS : xt ∈ Fn,t ; all t ∈ I} −

n−1 [

{x S ∈ AS : xt ∈ Fi,t ; all t ∈ I}

i=1

= {x S ∈ AS : xt ∈ Fn,t −

n−1 [

Fi,t ; all t ∈ I},

i=1

Sn−1 which is a rectangle since Fn,t − i=1 Fi,t ∈ Bt , t ∈ I. 2 Proof of Lemma 2.2. Let H be the collection of all finite disjoint unions of rectangles and note that H ⊂ FAS . The lemma is proved by showing that H is a field since FAS is by definition the smallest field containing the rectangles. SN First consider complements. Let F ∈ H so that it has the form F = i=1 Fi for disjoint rectangles Fi = {x S ∈ AS : xt ∈ Fi,t ; all t ∈ Ii }; Fi,t ∈ Bt , t ∈ Ii . As in the previous proof, we can assume a common index set Ii = I. The complement is c  N N [ \ c Fi  = Fic F = i=1

=

N \

i=1

c {x S ∈ AS : xt ∈ Fi,t ; all t ∈ I}

i=1

= {x S ∈ AS : xt ∈

N \

c Fi,t ; all t ∈ I},

i=1

which is a rectangle. Finite unions of of finite unions disjoint rectangles are finite unions of possibly nondisjoint rectangles, but from Lemma 2.3 any such union equals a finite union of disjoint rectangles. Hence H is closed under complementation and finite unions and hence is a field. 2

Distributions Given an index set T and an A-valued random process {Xt }t∈T defined on an underlying probability space (Ω, B, P ), at first glance it would appear that for any S ⊂ T the measurable space (AS , BSA ) should inherit a probability measure from the underlying space through the random variables X S = {Xt ; t ∈ S}. The only hitch is that so far we know only that individual random variables Xt for each individual t are measurable (and

50

2 Random Processes and Dynamical Systems

hence inherit a probability measure, called the marginal distribution) . To make sense here we must first show that collections of random variables X S = {Xt ; t ∈ S} such as X n = {X0 , . . . , Xn−1 } are also measurable and hence themselves random variables in the general sense. It is easy to show that inverse images of the mapping X S : Ω → AS yieldTevents in B if we confine attention to rectangles. For a rectangle B = t∈I {x S ∈ AS : xt ∈ Bt } we have that \ Xt −1 (Bt ) (2.10) (X I )−1 (B) = t∈I

and hence the measurability of each individual Xt and the fact that a countable unions of events are events implies that (X I )−1 (B) ∈ B.

(2.11)

We will have the desired measurability if we can show that if (2.11) is satisfied for all rectangles, then it is also satisfied for all events in the σ -field BA S = σ (rect(Bt , t ∈ S)) generated by the rectangles. This result is an application of an approach named the good sets principle by Ash [1], p. 5. We shall occasionally wish to prove that all events possess some particular desirable property that is easy to prove for generating events. The good sets principle consists of the following argument: Let S be the collection of good sets consisting of of all events F ∈ σ (G) possessing the desired property. If • G ⊂ S and hence all the generating events are good, and • S is a σ -field, then σ (G) ⊂ S and hence all of the events F ∈ σ (G) are good. Lemma 2.4. Given measurable spaces (Ω1 ,B) and (Ω2 , σ (G)), then a function f : Ω1 → Ω2 is B-measurable if and only if f −1 (F ) ∈ B for all F ∈ G; that is, measurability can be verified by showing that inverse images of generating events are events. Proof. If f is B-measurable, then f −1 (F ) ∈ B for all F and hence for all F ∈ G. Conversely, if f −1 (F ) ∈ B for all generating events F ∈ G, then define the class of sets S = {G : G ∈ σ (G), f −1 (G) ∈ B}. It is straightforward to verify that S is a σ -field, clearly Ω1 ∈ S since it is the inverse image of Ω2 . The fact that S contains countable unions of its elements follows from the fact that σ (G) is closed to countable unions and inverse images preserve set theoretic operations, that is, f −1 (∪i Gi ) = ∪i f −1 (Gi ).

2.5 Equivalent Random Processes

51

Furthermore, S contains every member of G by assumption. Since S contains G and is a σ -field, σ (G) ⊂ S by the good sets principle. 2 We have shown that given a random process {Xt ; t ∈ T} defined on a probability space (Ω, B, P ) and a set S ∈ T, then the mapping X S : Ω → AS is B/σ (rect(Bt , t ∈ S))-measurable. In other words, the output measurable space (AS , BSA ) inherits a probability measure from the underlying probability space and thereby determines a new probability space (AS , BSA , PX S ), where the induced probability measure is defined by PX S (F ) = P ((X S )−1 (F )) = P ({ω : X S (ω) ∈ F }), F ∈ BA S .

(2.12)

Such probability measures induced on the outputs of random variables are referred to as distributions for the random variables, exactly as in the simpler case first treated. When S = {m, m + 1, . . . , m + n − 1}, e.g., when we are treating X n taking values in An , the distribution is referred to as an n-dimensional or nth order distribution and it describes the behavior of an n-dimensional random variable. If S = T, then the induced probability measure is said to be the distribution of the process or, simply, the process distribution. Thus, for example, a probability space (Ω, B, P ) together with a doubly infinite sequence of random variables {Xn }n∈Z induces a new probability space (AZ , BZA , PX Z ) and PX Z is the distribution of the process. For simplicity, we usually denote a process distribution simply by m (or µ or some other simple symbol).

2.5 Equivalent Random Processes Since the sequence space (AT , BTA , m) of a random process {Xt }t∈T is a probability space, we can define random variables and hence also random processes on this space. One simple and useful such definition is that of a sampling or coordinate or projection function defined as follows: Given a product space AT , define for any t ∈ T the sampling function Πt : AT → A by Πt (x T ) = xt , x T ∈ AT , t ∈ T.

(2.13)

The sampling function is named Π since it is also a projection. Observe that the distribution of the random process {Πt }t∈T defined on the probability space (AT , BTA , m) is exactly the same as the distribution of the random process {Xt }t∈T defined on the probability space (Ω, B, P ). In fact, so far they are the same process since the {Πt } simply read off the values of the {Xt }. What happens, however, if we no longer build the Πt on the Xt , that is, we no longer first select ω from Ω according to P , then form the sequence x T = X T (ω) = {Xt (ω)}t∈T , and then define Πt (x T ) = Xt (ω)?

52

2 Random Processes and Dynamical Systems

Instead we directly choose an x in AT using the probability measure m and then view the sequence of coordinate values. In other words, we are considering two completely separate experiments, one described by the probability space (Ω, B, P ) and the random variables {Xt } and the other described by the probability space (AT , BTA , m) and the random variables {Πt }. In these two separate experiments, the actual sequences selected may be completely different. Yet intuitively the processes should be the same in the sense that their statistical structures are identical, that is, they have the same process distribution. We make this intuition formal by defining two processes to be equivalent if their process distributions are identical, that is, if the probability measures on the output spaces are the same, regardless of the functional form of the random variables and of the underlying probability spaces. Simply put, we consider two random variables to be equivalent if their distributions are identical. We have described two equivalent processes or two equivalent models for the same random process, one defined as a sequence of perhaps very complicated random variables on an underlying probability space, the other defined as a probability measure directly on the measurable space of possible output sequences. The second model will be referred to as a directly given random process or the Kolmogorov representation of the process. Which model is better depends on the application. For example, a directly given model for a random process may focus on the random process itself and not its origin and hence may be simpler to deal with. If the random process is then coded or measurements are taken on the random process, then it may be better to model the encoded random process in terms of random variables defined on the original random process and not as a directly given random process. This model will then focus on the input process and the coding operation. We shall let convenience determine the most appropriate model. Recall that previously we introduced dynamical systems model for a random process in the context of a directly-given random process. Thus we have considered three methods of modeling a random process: • An indexed family of random variables defined on a common probability space. • A directly-given or Kolmogorov representation as a distribution on a sequence or function space. • A dynamical system consisting of a probability space and a family of transformations together with a single random variable. In a sense, the dynamical system model is the most general since it can be used to model any random process defined by the other models, but not every dynamical system can be well modeled by a random process. Each of these models will have its uses, but the dynamical systems ap-

2.5 Equivalent Random Processes

53

proach has the advantage of capturing the notion of evolution in time by a transformation or family of transformations. The shift transformation introduced previously on a sequence space is the most important transformation that we shall encounter. It is not, however, the only important transformation. Hence when dealing with transformations we will usually use the notation T to reflect the fact that it is related to the action of a simple left shift of a sequence, yet we should keep in mind that occasionally other operators will be considered and the theory to be developed will remain valid,; that is, T is not required to be a simple time shift. For example, we will also consider block shifts of vectors instead of samples and variable length shifts. Most texts on ergodic theory focus on the case of an invertible transformation, that is, where T is a one-to-one transformation and the inverse mapping T −1 is measurable. This is the case for the shift on AZ and AR , the two-sided shifts. Here the nonivertible case is also considered in detail, but the simplifications and additional properties in the simpler invertible case are described. Since random processes are considered equivalent if their distributions are the same, we shall adopt the notation [A, m, X] for a random process {Xt ; t ∈ T} with alphabet A and process distribution m, the index set T usually being clear from context. We will occasionally abbreviate this to [A, m], but it is often convenient to note the name of the output random variables as there may be several; e.g., an input X and output Y . By the associated probability space or Kolmogorov representation of a random process [A, m, X] we shall mean the sequence probability space (AT , BTA , m). It will often be convenient to consider the random process as a directly given random process, that is, to view Xt as the coordinate functions Πt on the sequence or function space AT rather than as being defined on some other abstract space. This will not always be the case, however, as processes might be formed by coding or communicating other random processes. Context should render such bookkeeping details clear.

Exercises 1. Given a random process {Xn } with alphabet A, show that the class F0 = rect(Bi ; i ∈ T) of all rectangles is a field. 2. Let F (G) denote the field generated by a class of sets G, that is, F (G) contains the given class and is in turn contained by all other fields containing G. Show that σ (G) = σ (F (G)).

54

2 Random Processes and Dynamical Systems

2.6 Codes, Filters, and Factors In communication systems and signal processing it is common to begin with one random process and to perform an operation on the process to obtain another random process. Such operations are at the heart of many applications, including signal processing, communications, and information theory. The multiple models for random processes provide a natural context for a few basic ideas along this line. Begin with a directly-given model of a random process {Xn ; n ∈ T} and hence a probability space (Ω, B, P ) = (AT , BT , mX ), where mX denotes the process distribution of {Xn ; n ∈ T}. Let TA denote the shift transformation on AT . Let f : AT → B be a random variable with alphabet B. Invoking the dynamical systems model for a random process, the original process plus the mapping yields a new B-valued random process {Yn ; n ∈ T} defined by Yn (x T ) = f (TAn x T ); t ∈ T.

(2.14)

Intuitively, in the discrete-time case the sample sequence y T of the output random process is formed from the sample sequence of the input x T by first mapping the input into a value y0 = f (x T ), then shifting the input sequence by one and again applying f to form y1 = f (T x T ), then shifting again to form y2 = f (T 2 x T ), and so on. As a simple concrete example, suppose that T = Z, A = B = {0, 1}, and that f (x T ) = g(x0 , x−1 , x−2 ) where g is given by the table x0 x−1 x−2 000 001 010 011 100 101 110 111 . g 0 0 0 0 0 0 0 1 The code can be depicted using a shift register and a mapping as

2.6 Codes, Filters, and Factors

{xn }

-

55

xn

xn−1

@

xn−2

shift register

? @ # R @ g

- yn = g(xn , xn−1 , xn−2 )

"!

function, table

and a portion of the input and output sequences is shown below: n -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 xn 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 Yn 000000001 1 0 0 0 The code is called a sliding-block or sliding-window code because it “slides” a block or window across the input sequence, looks up the block in a table, and prints out a single symbol. This is in contrast to a block code which independently maps disjoint input blocks into disjoint output blocks. In general such mapping is called a time-invariant coding if the input alphabet A is discrete, and a time-invariant filtering if it is continuous. A synonym for time-invariant is stationary. The distinction between coding and filtering is historical, and we will generally refer to both operations as sliding-block or stationary or time-invariant coding. The mapping f of a sample function into an output symbol induces a mapping into an output sample function, f¯(x T ) : AT → B T defined by f¯(x T ) = {f (TAn x T ); n ∈ T}.

(2.15)

This mapping has the property that shifting the input results in a corresponding shift of the output, that is, f¯(TA x T ) = TB f¯(x T ); t ∈ T

(2.16)

which is why the coding or filtering is said to be shift-invariant or stationary. There are a variety of extensions of this idea which allow different shifts on the input and output spaces, e.g., in the discrete-time case B might itself be a product space or we might shift the input sequence N times and produce N output symbols at each operation.

56

2 Random Processes and Dynamical Systems

The abstraction of the idea of a stationary coding or filtering from random processes to dynamical systems is called a factor in ergodic theory. Suppose that we have two dynamical systems (Ω, B, P , T ) and (Λ, S, µ, S) and a measurable mapping f : Ω → Λ with the property that f (T ω) = Sf (ω). In the random processes case this corresponds to a stationary code or filter, i.e., shifting the input yields a shifted output. In the coding case T is the shift on AT and S is the shift on B T . In this abstract case, the new dynamical system (Λ, S, µ, S) is said to be a factor of the original system (Ω, B, P , T ).

2.7 Isomorphism We have defined random variables or random processes to be equivalent if they have the same output probability spaces, that is, if they have the same probability measure or distribution on their output value measurable space. This is not the only possible notion of two random processes being the same. For example, suppose we have a random process with a binary output alphabet and hence an output space made up of binary sequences. We form a new directly given random process by taking each successive pair of the binary process and considering the outputs to be a sequence of quaternary symbols, that is, the original random process is coded into a new random process via the mapping 00 → 0 01 → 1 10 → 2 11 → 3 applied to successive pairs of binary symbols. The two random processes are not equivalent since their output sequence measurable spaces are different; yet clearly they are the same since each can be obtained by a simple relabeling or coding of the other. This leads to a more fundamental definition of sameness: isomorphism. The definition of isomorphism is for dynamical systems rather than for random processes since, as previously noted, this is the more fundamental notion and hence the definition applies to random processes. There are, in fact, several notions of isomorphism: isomorphic measurable spaces, isomorphic probability spaces, and isomorphic dynamical systems. We present these definitions together as they are intimately connected.

2.7 Isomorphism

57

Isomorphic Measurable Spaces Two measurable spaces (Ω, B) and (Λ, S) are isomorphic if there exists a measurable function f : Ω → Λ that is one-to-one and has a measurable inverse f −1 . In other words, the inverse image f −1 (λ) of a point λ ∈ Λ consists of exactly one point in Ω and the inverse mapping so defined, say g : Λ → Ω, g(λ) = f −1 (λ), is itself a measurable mapping. The function f (or its inverse g) with these properties is called an isomorphism. An isomorphism between two measurable spaces is thus an invertible mapping between the two sample spaces that is measurable in both directions.

Isomorphic Probability Spaces Two probability spaces (Ω, B, P ) and (Λ, S, Q) are isomorphic if there is an isomorphism f : Ω → Λ between the two measurable spaces (Ω, B) and (Λ, S) with the added property that Q = Pf and P = Qf −1 that is, Q(F ) = P (f −1 (F )); F ∈ S P (G) = Q(f (G)); G ∈ B. Two probability spaces are isomorphic 1. if one can find for each space a random variable defined on that space that has the other as its output space, and 2. the random variables can be chosen to be inverses of each other; that is, if the two random variables are f and g, then f (g(λ)) = λ and g(f (ω)) = ω. Note that if the two probability spaces (Ω, B, P ) and (Λ, S, Q) are isomorphic and f : Ω → Λ is an isomorphism with inverse g, then the random variable f g defined by f g(λ) = f (g(λ)) is equivalent to the identity random variable i : Λ → Λ defined by i(λ) = λ.

Isomorphism Mod 0 A weaker notion of isomorphism between two probability spaces is that of isomorphism mod 0 . Two probability spaces are isomorphic mod 0 or metrically isomorphic if the mappings have the desired properties on

58

2 Random Processes and Dynamical Systems

sets of probability 1, that is, the mappings can be defined except for null sets in the respective spaces. That is, two probability spaces (Ω, B, P ) and (Λ, S, Q) are isomorphic (mod 0) if there are null sets Ω0 ∈ B and λ0 ∈ S and a measurable one-to-one onto map f : Ω − Ω0 → Λ − Λ0 with measurable inverse such that Q(F ) = P (f −1 (F )) for all F ∈ S. This weaker notion is the standard one in ergodic theory.

Isomorphic Dynamical Systems With the ideas of isomorphic measurable spaces and isomorphic probability spaces in hand, we now can define isomorphic dynamical systems. Roughly speaking, two dynamical systems are isomorphic if one can be coded or filtered onto the other in an invertible way so that the coding carries one transformation into the other, that is, one can code from one system into the other and back again and coding and transformations commute. Two dynamical systems (Ω, B, P , S) and (Λ, S, m, T ) are isomorphic if there exists an isomorphism f : Ω → Λ such that T f (ω) = f (Sω); ω ∈ Ω. As with the isomorphism of probability spaces, isomorphic mod 0 means that the properties need hold only on sets of probability 1, but we also require that the null sets in the respective spaces on which the isomorphism is not defined to be invariant with respect to the appropriate transformation. If the probability space (Λ, S, m) is the sequence space of a directly given random process, T is the shift on this space, and Π0 the sampling function on this space, then the random process Π0 T n = Πn defined on (Λ, S, m) is equivalent to the random process Π0 (f S n ) defined on the probability space (Ω, B, P ). More generally, any random process of the form gT n defined on (Λ, S, m) is equivalent to the random process g(f S n ) defined on the probability space (Ω, B, P ). A similar conclusion holds in the opposite direction. Thus, any random process that can be defined on one dynamical system as a function of transformed points possesses an equivalent model in terms of the other dynamical system and its transformation. In addition, not only can one code from one system into the other, one can recover the original sample point by inverting the code (at least with probability 1). The binary example introduced at the beginning of this section is easily seen to meet this definition of sameness. Let B(Z(n)) denote the σ -field of subsets of Z(n) comprising all possible subsets of Z(n) (the power set of Z(n)). The described mapping of binary pairs or

2.7 Isomorphism

59

members of Z(2)2 into Z(4) induces a mapping f : Z(2)Z+ → Z(4)Z+ mapping binary sequences into quaternary sequences. This mapping is easily seen to be invertible (by construction) and measurable (use the good sets principle and focus on rectangles). Let T be the shift on the binary sequence space and S be the shift on the quaternary sequence space; then the dynamical systems (Z(2)Z+ , B(Z(2))Z+ , m, T 2 ) and (Z(4)Z+ , B(Z(4))Z+ , mf , T ) are isomorphic; that is, the quaternary model with an ordinary shift is isomorphic to the binary model with the two-shift that shifts symbols a pair at a time. Isomorphism will often provide a variety of equivalent models for random processes. Unlike the previous notion of equivalence, however, isomorphism will often yield dynamical systems that are decidedly different, but that are isomorphic and hence produce equivalent random processes by invertible coding for discrete-alphabet processes or filtering for continuous-alphabet processes.

“This page left intentionally blank.”

Chapter 3

Standard Alphabets

Abstract Standard spaces provide a model for random process outputs that is arguably as general as needed in almost all applications and has sufficient structure to yield several fundamental results for random processes that do not necessarily hold for more abstract models. The structure eases the construction of probability measures on product spaces and from sample sequences which is essential to the Kolmogorov extension theorem, the existence of regular conditional probability measures, and the ergodic decomposition, all results encountered in this book. In this chapter the basic definitions and properties are developed for standard measurable spaces and random processes with standard alphabets and simple examples are given. The properties are used to prove the Kolmogorov extension theorem for processes with standard alphabets. As an example of the Kolmogorov theorem, simple Bernoulli processes (discrete IID processes) are constructed and used to generate a fairly general class of discrete random processes known as B-processes. In the next chapter more complicated and general examples of standard spaces are treated.

3.1 Extension of Probability Measures It is often desirable to construct probability measures and sources by specifying the values of the probability of certain simple events and then by finding a probability measure on the σ -field containing these events. It was shown in Section 1.8 that if one has a set function meeting the required conditions of a probability measure on a field, then there is a unique extension of that set function to a consistent probability measure on the σ -field generated by the field. Unfortunately, the extension theorem is often difficult to apply since countable additivity or continuity of a set function can be difficult to prove, even on a field. Nonnegativity, R.M. Gray, Probability, Random Processes, and Ergodic Properties, DOI: 10.1007/978-1-4419-1090-5_3, © Springer Science + Business Media, LLC 2009

61

62

3 Standard Alphabets

normalization, and finite additivity are, however, usually quite simple to test. Because several of the results to be developed will require such extensions of finitely additive measures to countably additive measures, we shall develop in this chapter a class of alphabets for which such extension will always be possible. To apply the Carathéodory extension theorem, one must first pin down a candidate probability measure on a generating field. In most such constructions it is at best feasible to force a set function to be nice on a countable collection of sets, so we focus on σ -fields that are countably generated, that is, σ -fields for which there is a countable collection of sets G that generates. Say that we have a σ -field B = σ (G) for some countable class G. Let F (G) denote the field generated by G, that is, the smallest field containing G. Unlike σ -fields, there is a simple constructive definition of the field generated by a class: F (G) consists exactly of all elements of G together with all sets obtainable from finite sequences of set theoretic operations on G. Thus if G is countable, so is F (G). It is easy to show using the good sets principle that if B = σ (G), then also B = σ (F (G)) and hence if B is countably generated, then it is generated by a countable field. Our goal is to find countable generating fields which have the property that every nonnegative, normalized, finitely additive set function on the field is also countably additive on the field (and hence will have a unique extension to a probability measure on the σ -field generated by the field). We formalize this property in a definition: A field F is said to have the countable extension property if it has a countable number of elements and if every set function P satisfying (1.19), (1.20), and (1.22) on F also satisfies (1.23) on F . A measurable space (Ω, B) is said to have the countable extension property if B = σ (F ) for some field F with the countable extension property. Thus a measurable space has the countable extension property if there is a countable generating field such that all normalized, nonnegative, finitely additive set functions on the field extend to a probability measure on the full σ -field. This chapter is devoted to characterizing those fields and measurable spaces possessing the countable extension property and to prove one of the most important results for such spaces– the Kolmogorov extension theorem. The Kolmogorov result provides one of the most useful means for describing specific basic random process models, which can then in turn be used to construct a rich variety of other processes by means of coding or filtering. We also develop some simple properties of standard spaces in preparation for the next chapter, where we develop the most important and most general known class of such spaces.

3.2 Standard Spaces

63

3.2 Standard Spaces As a first step towards characterizing fields with the countable extension property, we consider the special case of fields having only a finite number of elements. Such finite fields trivially possess the countable extension property. We shall then proceed to construct a countable generating field from a sequence of finite fields and we will determine conditions under which the limiting field will inherit the extendibility properties of the finite fields. Let F = {Fi , i = 0, 1, 2, . . . , n − 1} be a finite field of a sample space Ω, that is, F is a finite collection of sets in Ω that is closed under finite set theoretic operations. Note that F is trivially also a σ -field. F itself can be represented as the field generated by a more fundamental class of sets. A set F in F will be called an atom if its only subsets that are also field members are itself and the empty set, that is, it cannot be broken up into smaller pieces that are also in the field. Let A denote the collection of atoms of F . Clearly there are fewer than n atoms. It is easy to show that A consists exactly of all nonempty sets of the form n−1 \

Fi∗

i=0

where Fi∗ is either Fi or Fic . Such sets are called intersection sets and any two intersection sets must be disjoint since for at least one i one intersection set must lie inside Fi and the other within Fic . Thus all intersection sets must be disjoint. Any field element can be written as a finite union of intersection sets — just take the union of all intersection sets contained in the given field element. Let G be an atom of F . Since it is an element of F , it is the union of disjoint intersection sets. There can be only one nonempty intersection set in the union, however, or G would not be an atom. Hence every atom is an intersection set. Conversely, if G is an intersection set, then it must also be an atom since otherwise it would contain more than one atom and hence contain more than one intersection set, contradicting the disjointness of the intersection sets. In summary, given any finite field F of a space Ω we can find a unique collection of atoms A of the field such that the sets in A are disjoint, nonempty, and have the entire space Ω as their union (since Ω is a field element and hence can be written as a union of atoms). Thus A is a partition of Ω. Furthermore, since every field element can be written as a union of atoms, F is generated by A in the sense that it is the smallest field containing A. Hence we write F = F (A). Observe that if we assign nonnegative numbers pi to the atoms Gi in A such that their sum is 1, then this immediately gives a finitely additive, nonnegative, normalized set function on F by the formula

64

3 Standard Alphabets

P (F ) =

X

pi .

i:Gi ⊂F

Furthermore, this set function is trivially countably additive since there are only a finite number of elements in F . The next step is to consider an increasing sequence of finite fields that in the limit will give a countable generating field. A sequence of finite fields Fn , n = 1, 2, . . . is said to be increasing if the elements of each field are also elements of all of the fields with higher indices, that is, if Fn ⊂ Fn+1 , all n. This implies that if An are the corresponding collections of atoms, then the atoms in An+1 are formed by splitting up the atoms in An . Given an increasing sequence of fields Fn , define the limit F as the union of all of the elements in all of the Fn , that is, F=

∞ [

Fi .

i=1

F is easily seen to be itself a field. For example, any F ∈ F must be in Fn for some n, hence the complement F c must also be in Fn and hence also in F . Similarly, any finite collection of sets Fn , n = 1, 2, . . . , m must all be contained in some Fk for sufficiently large k. The latter field must hence contain the union and hence so must F . Thus the increasing sequence Fn can be thought of as increasing to a limit field F , an idea captured by the notation Fn ↑ F if Fn ⊂ Fn+1 , all n, and F is the union of all of the elements of the Fn . Note that F has by construction a countable number of elements. When Fn ↑ F , we shall say that the sequence Fn asymptotically generates F . Lemma 3.1. Given any countable field F of subsets of a space Ω, then there is a sequence of finite fields {Fn ; n = 1, 2, . . .} that asymptotically generates F . In addition, the sequence can be constructed so that the corresponding sets of atoms An of the field Fn can be indexed by a subset of the set of all binary n-tuples, that is, An = {Gun , un ∈ B}, where B is a subset of {0, 1}n , with the property that Gun ⊂ Gun−1 . Thus if un is a prefix of um for m > n, then Gum is contained in Gun . (We shall refer to such a sequence of atoms of finite fields as a binary indexed sequence.) Proof. Let F = {Fi , i = 0, 1, . . .}. Consider the sequence of finite fields defined by Fn = F (Fi , i = 0, 1, . . . , n − 1), that is, the field generated by the first n elements in F . The sequence is increasing since the generating classes are increasing. Any given element in F is eventually in one of the Fn and hence the union of the Fn contains all of the elements in F and is itself a field, hence it must contain F . Any element in the union of the Fn , however, must be in an Fn for some n and hence must be an element of F . Thus the two fields are the same. A similar argument to that used

3.2 Standard Spaces

65

above to construct the atoms of an arbitrary finite field will demonstrate that the atoms in F (F0 , . . . , Fn−1 ) are simply all nonempty sets of the Tn−1 form k=0 Fk∗ , where Fk∗ is either Fk or Fkc . For each such intersection set let un denote the binary n-tuple having a one in each coordinate i for which Fi∗ = Fi and zeros in the remaining coordinates and define Gun as the corresponding intersection set. Then each Gun is either an atom or empty and all atoms are obtained as u varies through the set {0, 1}n of all binary n-tuples. By construction, ω ∈ Gun if and only if ui = 1Fi (ω), i = 0, 1, . . . , n − 1,

(3.1)

where for any set F 1F (ω) is the indicator function of the set F , that is, ( 1F (ω) =

1

ω∈F

0

ω 6∈ F .

Eq. (3.1) implies that for any n and any binary n-tuple un that Gun ⊂ Gun−1 , completing the proof. 2 Before proceeding to further study sequences of finite fields, we note that the construction of the binary indexed sequence of atoms for a countable field presented above provides an easy means of determining which atoms contain a given sample point. For later use we formalize this notion in a definition. Given an enumeration {Fn ; n = 0, 1, . . .} of a countable field F of subsets of a sample space Ω and the single-sided binary sequence space M+ = {0, 1}Z+ = ×∞ i=0 {0, 1}, define the canonical binary sequence function f : Ω → M+ by f (ω) = {1Fi (ω) ; i = 0, 1, . . .}.

(3.2)

Given an enumeration of a countable field and the corresponding binary indexed set of atoms An = {Gun } as above, then for any point ω ∈ Ω we have that ω ∈ Gf (ω)n , n = 1, 2, . . . , (3.3) where f (ω)n denotes the binary n-tuple comprising the first n symbols in f (ω). Thus the sequence of decreasing atoms containing a given point ω can be found as a prefix of the canonical binary sequence function. Observe, however, that f is only an into mapping, that is, some binary sequences may not correspond to points in Ω. In addition, the function may be many to one, that is, different points in Ω may yield the same binary sequences.

66

3 Standard Alphabets

Unfortunately, the sequence of finite fields converging upward to a countable field developed above does not have sufficient structure to guarantee that probability measures on the finite fields will imply a probability measure on the limit field. The missing item that will prove to be the key is specified in the following definitions: A sequence of finite fields Fn , n = 0, 1, . . ., is said to be a basis for a field F if it has the following properties: 1. It asymptotically generates F , that is, Fn ↑ F .

(3.4)

2. If Gn is a sequence of atoms of Fn such that if Gn ∈ An and Gn+1 ⊂ Gn , n = 0, 1, 2, . . ., then ∞ \ Gn 6= ∅. (3.5) n=1

A basis for a field is an asymptotically generating sequence of finite fields with the property that a decreasing sequence of atoms cannot converge to the empty set. The property (3.4) of a sequence of fields is called the finite intersection property. The name comes from the similar use in topology and reflects the fact that if all finite intersections of the sequence of sets Gn are not empty (which is true since the Gn are a decreasing sequence of nonempty sets), then the intersection of all of the Gn cannot be empty. A sequence Fn , n = 1, . . . is said to be a basis for a measurable space (Ω, B) if the {Fn } form a basis for a field that generates B. If the sequence Fn ↑ F and F generates B, that is, if B = σ (F ) = σ (

∞ [

Fn ),

n=1

then the Fn are said to asymptotically generate the σ -field B. A field F is said to be standard if it possesses a basis. A measurable space (Ω, B) is said to be standard if B can be generated by a standard field, that is, if B possesses a basis. The requirement that a σ -field be generated by the limit of a sequence of simple finite fields is a reasonably intuitive one if we hope for the σ field to inherit the extendibility properties of the finite fields. The second condition — that a decreasing sequence of atoms has a nonempty limit — is less intuitive, however, and will prove harder to demonstrate. Although nonintuitive at this point, we shall see that the existence of a basis is a sufficient and necessary condition for extending arbitrary finitely additive measures on countable fields to countably additive measures. The proof that the standard property is sufficient to ensure that any finitely additive set function is also countably additive requires addi-

3.2 Standard Spaces

67

tional machinery that will be developed later. The proof of necessity, however, can be presented now and will perhaps provide some insight into the standard property by showing what can go wrong in its absence. Lemma 3.2. Let F be a field of subsets of a space Ω. A necessary condition for F to have the countable extension property is that it be standard, that is, that it possess a basis. Proof. We assume that F does not possess a basis and we construct a finitely additive set function that is not continuous at ∅ and hence not countably additive. To have the countable extension property, F must be countable. From Lemma 3.1 we can construct a sequence of finite fields Fn such that Fn ↑ F . Since F does not possess a basis, we know that for any such sequence Fn there must exist a decreasing sequence of atoms Gn of Fn such that Gn ↓ ∅. Define set functions Pn on Fn as follows: If Gn ⊂ F , then Pn (F ) = 1, if F ∩ Gn = ∅, then Pn (F ) = 0. Since F ∈ Fn , F either wholly contains the atom Gn or F and Gn are disjoint, hence the Pn are well defined. Next define the set function P on the limit field F in the natural way: If F ∈ F , then F ∈ Fn for some smallest value of n (e.g., if the Fn are constructed as before as the field generated by the first n elements of F , then eventually every element of the countable field F must appear in one of the Fn ). Thus we can set P (F ) = Pn (F ). By construction, if m ≥ n then also Pm (F ) = Pn (F ) and hence P (F ) = Pn (F ) for any n such that F ∈ Fn . P is obviously nonnegative, P (Ω) = 1 (since all of the atoms Gn in the given sequence are in the sample space), and P is finitely additive. To see the latter fact, let Fi , i = 1, . . . , m be a finite collection of disjoint sets in F . By construction, all must lie in some field Fn for sufficiently large n. If Gn lies in one of the sets Fi (it can lie in at most one since the sets are disjoint), then (1.22) holds with both sides equal to one. If none of the sets contains Gn , then (1.22) holds with both sides equal to zero. Thus P satisfies (1.19), (1.20), and (1.22). To prove P to be countably additive, then, we must verify (1.23). By construction P (Gn ) = Pn (Gn ) = 1 for all n and hence lim P (Gn ) = 1 6= 0,

n→∞

and therefore (1.23) is violated since by assumption the Gn decrease to the empty set ∅ which has zero probability. Thus P is not continuous at ∅ and hence not countably additive. If a field is not standard, then finitely additive probability measures that are not countably additive can always be constructed by putting probability on a sequence of atoms that collapses down to nothing. Thus there can always be probability on ever smaller sets, but the limit cannot support the probability since it is empty. Thus the necessity of the standard property for the extension of arbitrary additive probability measures justifies its candidacy for a general, useful alphabet.

68

3 Standard Alphabets

Corollary 3.1. A necessary condition for a countably generated measurable space (Ω, B) to have the countable extension property is that it be a standard space. Proof. To have the countable extension property, a measurable space must have a countable generating field. If the measurable space is not standard, then no such field can possess a basis and hence no such field will possess the countable extension property. In particular, one can always find as in the proof of the lemma a generating field and an additive set function on that field which is not countably additive and hence does not extend. 2

Exercise 1. A class of subsets V of A is said to be separating if given any two points x, y ∈ A, there is a V ∈ V that contains only one of the two points and not the other. Suppose that a σ -field B has a countable generating class V = {Vi ; i = 1, 2, . . .} that is also separating. Describe the intersection sets ∞ \ Vn∗ , n=1

where

Vn∗

is either Vn or

Vnc .

3.3 Some Properties of Standard Spaces The following results provide some useful properties of standard spaces. In particular, they show how certain combinations of or mappings on standard spaces yield other standard spaces. These results will prove useful for demonstrating that certain spaces are indeed standard. The first result shows that if we form a product space from a countable number of standard spaces as in Section 2.4, then the product space is also standard. Thus if the alphabet of a source or random process for one sample time is standard, then the space of all sequences produced by the source is also standard. Lemma 3.3. Let Fi , i ∈ I, be a family of standard fields for some countable index set I. Let F be the product field generated by all rectangles of the form F = {x I : xi ∈ Fi , i ∈ M}, where Fi ∈ Fi all i and M is any finite subset of I. That is, F = F (rect(Fi , i ∈ I)), then F is also standard.

3.3 Some Properties of Standard Spaces

69

Proof. Since I is countable, we may assume that I = {1, 2, . . .}. For each i ∈ I, Fi is standard and hence possesses a basis, say {Fi (n), n = 1, 2, . . .}. Consider the sequence Gn = F (rect(Fi (n), i = 1, 2, . . . , n)),

(3.6)

that is, Gn is the field of subsets formed by taking all rectangles formed from the nth order basis fields Fi (n), i = 1, . . . , n in the first n coordinates. The lemma will be proved by showing that Gn forms a basis for the field F and hence the field is standard. The fields Gn are clearly finite and increasing since the coordinate fields are. The field generated by the union of all the fields Gn will contain all of the rectangles in F since for each i the union in that coordinate contains the full coordinate field Fi . Thus Gn ↑ F . Say we have a sequence Gn of atoms of Gn (Gn ∈ Gn for all n) decreasing to the null set. Each such atom must have the form Gn = {x I : xi ∈ Gn (i); i = 1, 2, . . . , n}. where Gn (i) is an atom of the coordinate field Fi (n). For Gn ↓ ∅, however, this requires that Gn (i) ↓ ∅ at least for one i, violating the definition of a basis for the ith coordinate. Thus Fn must be a basis. 2 Corollary 3.2. Let (Ai , Bi ), i ∈ I, be a family of standard spaces for some countable index set I. Let (A, B) be the product measurable space, that is, (A, B) = ×i∈I (Ai , Bi ) = (×i∈I Ai , ×i∈I Bi ), where ×i∈I Ai = all x I = {(xi , i ∈ I) : xi ∈ Ai ; i ∈ I} is the Cartesian product of the alphabets Ai , and ×i∈I Bi = σ (rect(Bi , i ∈ I)). Then (A, B) is standard. Proof. Since each (Ai , Bi ), i ∈ I, is standard, each possesses a basis, say {Fi (n); n = 1, 2, . . .}, which asymptotically generates a coordinate field Fi which in turn generates Bi . (Note that these are not the same as the Fi (n) of (3.6).) From Lemma 3.3 the product field of the nth order basis fields given by (3.6) is a basis for F . Thus we will be done if we can show that F generates B. It is an easy exercise, however, to show that if Fi generates Bi , all i ∈ I, then B = σ (rect(Bi , i ∈ I)) = σ (rect(Fi , i ∈ I)) = σ (F (rect(Fi , i ∈ I))) = σ (F ).

(3.7)

2

70

3 Standard Alphabets

We now turn from Cartesian products of different standard spaces to certain subspaces of standard spaces. First, however, we need to define subspaces of measurable spaces. Let (A, B) be a measurable space (not necessarily standard). For any F ∈ B and any class of sets G define the new class G ∩ F = {all sets of the form G ∩ F for some G ∈ G}. Given a set F ∈ B we can define a measurable space (F , B ∩ F ) called a subspace of (A, B). It is easy to verify that B ∩ F is a σ -field of subsets of F . Intuitively, it is the full σ -field B cut down to the set F . We cannot show in general that arbitrary subspaces of standard spaces are standard. One case where they are is described next. Lemma 3.4. Suppose that (A, B) is standard with a basis Fn , n = 1, 2, . . .. If F is any countable union of members of the Fn , that is, if F=

∞ [ i=1

Fi ; Fi ∈

∞ [

Fn ,

n=1

then the space (A − F , B ∩ (A − F )) is also standard. Proof. The basic idea is that if you take a standard space with a basis and remove some atoms, the original basis with these atoms removed still works since decreasing sequences of atoms must still be empty for some finite n or have a nonempty intersection. Given a basis Fn for (A, B), form a candidate basis Gn = Fn ∩ F c . This is essentially the original basis with all of the atoms in the sets in F removed. Let us call the countable collection of basis field atoms in F “bad atoms.” From the good sets principle, the Gn asymptotically generate B ∩ F c . Suppose that Gn ∈ Gn is a decreasing sequence of nonempty atoms. By construction Gn = An ∩ F c for some sequence ofTatoms An in Fn . Since the original ∞ space is standard, the intersection n=1 An is nonempty, say some set c A. Suppose that A ∩ F is empty, then A ⊂ F and hence A must be contained in one of the bad atoms, say A ⊂ Bm ∈ Fm . But the atoms of Fm are disjoint and A ⊂ Am with Am ∩ Bm = ∅, a contradiction unless Am is empty in violation of the assumptions. 2 Next we consider spaces isomorphic to standard spaces. Recall that given two measurable spaces (A, B) and (B, S), an isomorphism f is a measurable function f : A → B that is one-to-one and has a measurable inverse f −1 . Recall also that if such an isomorphism exists, the two measurable spaces are said to be isomorphic. We first provide a useful technical lemma and then show that if two spaces are isomorphic, then one is standard if and only if the other one is standard. Lemma 3.5. If (A, B) and (B, S) are isomorphic measurable spaces, let f : A → B denote an isomorphism. Then B = f −1 (S) = { all sets of the form f −1 (G) for G ∈ S}.

3.3 Some Properties of Standard Spaces

71

Proof. Since f is measurable, the class of sets F = f −1 (S) has all its elements in B. Conversely, since f is an isomorphism, its inverse function, say g, is well defined and measurable. Thus if F ∈ B, then g −1 (F ) = f (F ) = D ∈ S and hence F = f −1 (D). Thus every element of B is given by f −1 (D) for some D in S. In addition, F is a σ -field since the inverse image operation preserves all countable set theoretic operations. Suppose that there is an element D in B that is not in F . Let g : B → A denote the inverse operation of f . Since g is measurable and one-toone, g −1 (D) = f (D) is in G and therefore its inverse image under f , f −1 (f (D)) = D, must be in F = f −1 (G) since f is measurable. But this is a contradiction since D was assumed to not be in F and hence F = B.

2 Corollary 3.3. If (A, B) and (B, S) are isomorphic, then (A, B) is standard if and only if (B, S) is standard. Proof. Say that (B, S) is standard and hence has a basis Gn , n = 1, 2, . . . and that G is the generating field formed by the union of all of the Gn . Let f : A → B denote an isomorphism and define Fn = f −1 (Gn ), n = 1, 2, . . . Since inverse images preserve set theoretic operations, the Fn are inS creasing finite fields with Fn = f −1 (G). A straightforward application of the good sets principle show that if G generates S and B = f −1 (S), then f −1 (G) = F also generates B and hence the Fn asymptotically generate B. Given any decreasing set of atoms, say Dn , n = 1, 2, . . ., of Fn , then f (Dn ) = g −1 (Dn ) must also be a decreasing sequence of atoms of Gn (again, since inverse images preserve set theoretic operations). The latter sequence cannot be empty since the Gn form a basis, hence neither can the former sequence since the mappings are one-to-one. Reversing the argument shows that if either space is standard, then so is the other, completing the proof. 2 As a final result of this section we show that if a field is standard, then we can always find a basis that is a binary indexed sequence of fields in the sense of Lemma 3.1. Indexing the atoms by binary vectors will prove a simple and useful representation. Lemma 3.6. If a field F is standard, then there exists a binary indexed basis and a corresponding canonical binary sequence function. Proof. The proof consists simply of relabeling of a basis in binary vectors. Since F is standard, then it possesses a basis, say Fn , n = 0, 1, . . .. Use this basis to enumerate all of the elements of F in order, that is, first list the elements of F0 , then the elements of F1 , and so on. This produces a sequence {Fn , n = 0, 1, . . .} = F . Now construct a corresponding binary indexed asymptotically generating sequence of fields, say Gn , n = 0, 1, . . ., as in Lemma 3.1. To prove Gn is a basis we need only show that it has the finite intersection property of (3.4). First observe

72

3 Standard Alphabets

that for every n there is an m ≥ n such that Fn = Gm . That is, eventually the binary indexed fields Gm match the original fields, they just take longer to get all of the elements since they include them one new one at a time. If Gn is a decreasing sequence of atoms in Gn , then there is a sequence of integers n(k) ≥ k, k = 1, 2, . . ., such that Gn(k) = Bk , where Bk is an atom of Fk . If Gn collapses to the empty set, then so must the subsequence Gn(k) and hence the sequence of atoms Bk of Fk . But this is impossible since the Fk are a basis. Thus the Gn must also possess the finite intersection property and be a basis. The remaining conclusion follows immediately from Lemma 3.1 and the definition that follows it.

2

3.4 Simple Standard Spaces First consider a measurable space (A, B) with A a finite set and B the class of all subsets of A. This space is trivially standard–simply let all fields in the basis be B. Alternatively one can observe that since B is finite, the space trivially possesses the countable extension property and hence must be standard. Next assume that (A, B) has a countable alphabet and that B is again the class of all subsets of A. A natural first guess for a basis is to first enumerate the points of A as {a0 , a1 , . . .} and to define the increasing sequence of finite fields Fn = F ({a0 }, . . . , {an−1 } ). The sequence is indeed asymptotically generating, but it does not have the finite intersection property. The problem is that each field has as an atom the leftover or garbage set Gn = {an , an+1 , . . .} containing all of the points of A not yet included in the generating class. This sequence of atoms is nonempty and decreasing, yet it converges to the empty set and hence Fn is not a basis. In this case there is a simple fix. Simply adjoin the undesirable atoms Gn containing the garbage to a good atom, say a0 , to form a new sequence of finite fields Gn = F (b0 (n), b1 (n), . . . , bn−1 (n)) with b0 (n) = a0 ∪ Gn , bi (n) = ai , i = 1, . . . , n − 1. This sequenceTof fields also gen∞ erates the full σ -field (note for example that a0 = n=0 b0 (n)). It also possesses the finite intersection property and hence is a basis. Thus any countable measurable space is standard. The above example points out that it is easy to find sequences of asymptotically generating finite fields for a standard space that are not bases and hence which cannot be used to extend arbitrary finitely additive set functions. In this case, however, we were able to rig the field so that it did provide a basis.

3.4 Simple Standard Spaces

73

From Corollary 3.2 any countable product of standard spaces must be standard. Thus products of finite or countable alphabets give standard spaces, a conclusion summarized in the following lemma. Lemma 3.7. Given a countable index set I, if Ai are finite or countably infinite alphabets and Bi the σ -fields of all subsets of Ai for i ∈ I, then ×i∈I (Ai , Bi ) is standard. Thus, for example, the following spaces are standard: the integer sequence space Z

Z

(N+ , BN+ ) = (Z++ , BZ++ ),

(3.8)

where Z+ = {0, 1, . . .} and BZ+ is the σ -field of all subsets of Z+ (the power set) of Z+ ), and the binary sequence space Z

+ ). (M+ , BM+ ) = ({0, 1}Z+ , B{0,1}

(3.9)

The proof of Lemma 3.3 coupled with the discussion for the finite and countable cases provide a specific construction for the bases of the above two processes. For the case of the binary sequence space M+ , the fields Fn consist of n dimensional rectangles of the form {x : xi ∈ Bi , i = 0, 1, . . . , n − 1}, Bi ⊂ {0, 1} all i. These rectangles are themselves finite unions of simpler rectangles of the form c(bn ) = {x : xi = bi ; i = 0, 1, . . . , n − 1}

(3.10)

for bn ∈ {0, 1}n . These sets are called thin cylinders and are the atoms of Fn . The thin cylinders generate the full σ -field and any decreasing sequence of such rectangles converges down to a single binary sequence. Hence it is easy to show that this space is standard. Observe that this also implies that individual binary sequences x = {xi ; i = 0, 1, . . .} are themselves events since they are a countable intersection of thin cylinders. For the case of the integer sequence space N+ , the field Fn contains all rectangles of the form {x : xi ∈ Fi , i = 0, 1, . . . , n}, where Fi ∈ Fi (n) for all i, and Fi (n) = {b0 (n), b1 (n), . . . , bn−1 (n)} for all i with b0 (n) = {0, n, n + 1, n + 2, . . .} and bj = {j} for j = 1, 2, . . . , n − 1. As previously mentioned, it is not true in general that an arbitrary subset of a standard space is standard, but the previous construction of a basis combined with Lemma 3.4 does provide a means of showing that certain subspaces of M+ and N+ are standard. Lemma 3.8. Given the binary sequence space M+ or the integer sequence space N+ , if we remove from these spaces any countable collection of rectangles, then the remaining space is standard.

74

3 Standard Alphabets

Proof. The rectangles are eventually in the fields of the basis and the result follows from Lemma 3.4. 2

Exercises 1. Consider the space of (3.9) of all binary sequences and the corresponding event space. Is the class of all finite unions of thin cylinders a field? Show that a decreasing sequence of nonempty atoms of Fn must converge to a point, i.e., a single binary sequence.

3.5 Characterization of Standard Spaces As a step towards demonstrating that a space is standard, the following theorem provides an alternative to the definition. Theorem 3.1. A field F of subsets of Ω is standard if and only if (a) F is countable, and (b) ifSFn , n = 1, 2, . . ., is a sequence of nonempty disjoint elements in F , then n Fn 6∈ F . Given (a), condition (b) is equivalent to the condition (b’) that if Gn , n = 1,T2, . . ., is a sequence of strictly decreasing nonempty elements in F , then n Gn 6= ∅. Proof. We begin by showing that (b) and (b’) are equivalent. We first show that (b) implies (b’). Suppose that Gn satisfy the hypothesis of (b’). Then H=(

∞ \

n=1

Gn )c =

∞ [ n=1

c Gn =

∞ [

c c (Gn − Gn−1 ),

n=1

c c − Gn−1 yields a sequence satisfying the where G0 = Ω. Defining Fn = Gn conditions of (a) and hence H and therefore H c are not in F , showing T c that n Gn = H can not be empty (or it would be in F since a field must contain the empty set). To prove that (b’) implies (b), suppose that S the sequence F satisfies the hypothesis of (b) and define F = F n n and Sn n Gn = F − j=1 Fj . Then Gn is a strictly decreasing sequence of nonempty sets with empty intersection. From (b’), the sequence Gn cannot all be in F , which implies that F 6∈ F . We now show that the equivalent conditions (b) and (b’) hold if and only if the field is standard. First suppose that (b) and (b’) hold and that F = {Fn , n S = 1, 2, . . .}. Define Fn = F (F1 , F2 , . . . , Fn ) for n = 1, 2, . . .. ∞ Clearly F = n=1 Fn since F contains all of the elements of all of the Fn and each element in F is in some Fn . Let An be a nonincreasing sequence of nonempty atoms in Fn as in theThypothesis for a basis. If ∞ the An are all equal for large n, then clearly n=1 An 6= ∅. If this is not

3.5 Characterization of Standard Spaces

75

true, then there is a strictly decreasing subsequence Ank and hence from (b’) ∞ ∞ \ \ An = Ank 6= ∅. n=1

k=1

This means that the Fn form a basis for F and hence F is standard. Lastly, suppose that F is standard with basis Fn , n = 1, 2, . . ., and suppose that Gn is a sequence of strictly decreasing elements in F as in (b’). Define the nonincreasing sequence of nonempty sets Hn ∈ Fn as follows: If G1 ∈ F1 , then set H1 = G1 and set n1 = 1. If this is not the case, then G1 is too “fine” for the field F1 so we set H1 = Ω. Continue looking for an n1 such that G1 ∈ Fn1 . When it is found (as it must be since G1 is a field element), set Hi = Ω for i < n1 and Hn1 = G1 . Next consider G2 . If G2 ∈ Fn1 +1 , then set Hn1 +1 = G2 . Otherwise set Hn1 +1 = G1 and find an n2 such that G2 ∈ Fn2 . Then set Hi = G1 for n1 ≤ i < n2 and Hn2 = G2 . Continue in this way filling in the sequence of Gn with enough repeats to meet the condition. The sequence Hn ∈ Fn is thus a nonempty nonincreasing sequence of field elements. Clearly by construction ∞ ∞ \ \ Hn = Gn n=1

n=1

and hence we will be done if we can prove this intersection nonempty. We accomplish this by assuming that the intersection is empty and show that this leads to a contradiction with the assumption of a standard space. Roughly speaking, a decreasing set of nonempty field elements collapsing to the empty set must contain a decreasing sequence of nonempty basis atoms collapsing to the empty set and that violates the properties of a basis. Pick for each n an ωn ∈ Hn and consider the sequence f (ωn ) in the binary sequence space M+ . The binary alphabet {0, 1} is trivially sequentially compact and hence so is M+ from Corollary 1.1. Thus there is a subsequence, say f (ωn(k) ), which converges to some u ∈ M+ as k → ∞. From Lemma 1.3 this means that f (ωn(k) ) must converge to u in each coordinate as k → ∞. Since there are only two possible values for each coordinate in u, this means that given any positive integer n, there is an N such that for n(k) ≥ N, f (ωn(k) )i = ui for i = 0, 1, . . . , n − 1, and hence from (3.1) that ωn(k) ∈ Aun , the atom of Fn indexed by un . Assuming that we also choose k large enough to ensure that n(k) ≥ n, then the fact that the fields are decreasing implies that ωn(k) ∈ Hn(k) ⊂ Hn . Thus we must have that Gun is contained in Hn . That is, the point ωn(k) is contained in an atom of Fn(k) which is itself contained in an atom of Fn . Since the point is also in the set Hn of Fn , the set must contain the entire atom. Thus we have constructed a sequence of atoms

76

3 Standard Alphabets

contained in a sequence of sets that decreases to the empty set, which violates the standard assumption and hence completes the proof. 2

3.6 Extension in Standard Spaces In this section we combine the ideas of the previous two sections to complete the proofs of the following results. Theorem 3.2. A field has the countable extension property if and only if it is standard. Corollary 3.4. A measurable space has the countable extension property if and only if it is standard. These results will complete the characterization of those spaces having the desirable extension property discussed at the beginning of the section. The “only if” portions of the results are contained in Section 3.2. The corollary follows immediately from the theorem and the definition of standard. Hence we need only now show that a standard field has the countable extension property. Proof of the theorem. Since F is standard, construct as in Lemma 3.1 a basis with a binary indexed sequence of atoms and let f : Ω → M+ denote the corresponding canonical binary sequence function of (3.2). Let P be a nonnegative, normalized, finitely additive set function of F . We wish to show that (1.23) is satisfied. Let Fn be a sequence of field elements decreasing to the null set. If for some finite n = N, FN is empty, then we are done since then lim P (Fn ) = P (FN ) = P (∅) = 0.

n→∞

(3.11)

Thus we need only consider sequences Fn of nonempty decreasing field elements that converge to the empty set. From Theorem 3.1, however, there cannot be such sequences since the field is standard. Thus (3.11) trivially implies (1.23) and the theorem is proved by the Carathéodory extension theorem. 2 The key to proving sufficiency above is the characterization of standard fields in Theorem 3.1. This has the interpretation using part (b) that a field is standard if and only if truly countable unions of nonempty disjoint field elements are not themselves in the field. Thus finitely additive set functions are trivially countably additive on the field because all countable disjoint unions of field elements can be written as a finite union of disjoint nonempty field elements.

3.7 The Kolmogorov Extension Theorem

77

3.7 The Kolmogorov Extension Theorem In Chapter 1 in the section on distributions we saw that given a random process, we can determine the distributions of all finite dimensional random vectors produced by the process. We now possess the machinery to prove the reverse result–the Kolmogorov extension theorem–which states that given a family of distributions of finite dimensional random vectors that is consistent, that is, higher dimensional distributions imply the lower dimensional ones, then there is a process distribution that agrees with the given family. Thus a consistent family of finite dimensional distributions is sufficient to completely specify a random process possessing the distributions. This result is one of the most important in probability theory since it provides realistic requirements for the construction of mathematical models of physical phenomena. Theorem 3.3. Let I be a countable index set, let (Ai , Bi ), i ∈ I, be a family of standard measurable spaces, and let (AI , BI ) = ×i∈I (Ai , Bi ) be the product space. Similarly define for any index set M ⊂ I the product space (AM , BM ). Given index subsets K ⊂ M ⊂ I, define the projections ΠM→K : AM → AK by ΠM→K (x M ) = x K , that is, the projection ΠM→K simply looks at the vector or sequence x M = {xi , i ∈ M} and produces the subsequence x K . A family of probability measures PM on the measurable spaces (AM , BM ) for all finite index subsets M ⊂ I is said to be consistent if whenever K ⊂ M, then K PK (F ) = PM (Π−1 M→K (F )), all F ∈ B ,

(3.12)

that is, probability measures on index subsets agree with those on further subsets. Given any consistent family of probability measures, then there exists a unique process probability measure P on (AI , BI ) which agrees with the given family, that is, such that for all M ⊂ I M PM (F ) = P (Π−1 I→M (F )), all F ∈ B , all finite M ⊂ I.

(3.13)

Proof. Abbreviate by Πi : AI → Ai the one dimensional sampling function defined by Πi (x I ) = xi , i ∈ I. The consistent families of probability measures induces a set function P on all rectangles of the form \ Π−1 B= i (Bi ) i∈M

for any Bi ∈ Bi via the relation

78

3 Standard Alphabets

P (B) = PM (×i∈M Bi ),

(3.14)

that is, we can take (3.13) as a definition of P on rectangles. Since the spaces (Ai , Bi ) are standard, each has a basis {Fi (n), n = 1, 2, . . .} and the corresponding generating field Fi . From Lemma 3.3 and Corollary 3.2 the product fields given by (3.6) form a basis for F = F (rect(Fi , i ∈ I)), which in turn generates BI ; hence P is also countably additive on F . Hence from the Carathéodory extension theorem, P extends to a unique probability measure on (AI , BI ). Lastly observe that for any finite index set M ⊂ I, both PM and P (ΠI→M )−1 (recall that P f −1 (F ) is defined as P (f −1 (F )) for any function f : it is the distribution of the random variable f ) are probability measures on (AM , BM ) that agree on a generating field F (rect(Fi , i ∈ M)) and hence they must be identical. Thus the process distribution indeed agrees with the finite dimensional distributions for all possible index subsets, completing the proof. 2 By simply observing in the above proof that we only required the probability measure on the coordinate generating fields and that these fields are themselves standard, we can weaken the hypotheses of the theorem somewhat to obtain the following corollary. The details of the proof are left as an exercise. Corollary 3.5. Given standard measurable spaces (Ai , Bi ), i ∈ I, for a countable index set I, let Fi be corresponding generating fields possessing a basis. If we are given a family of normalized, nonnegative, finitely additive set functions PM that are defined on all sets F ∈ F (rect(Fi , i ∈ I)) and are consistent in the sense of satisfying (3.13) for all such sets, then the conclusions of the previous theorem hold. In summary, if we have a sequence space composed of standard measurable spaces and a family of finitely additive candidate probability measures that are consistent, then there exists a unique random process or a process distribution that agrees with the given probabilities on the generating events.

3.8 Bernoulli Processes The principal advantage of a standard space is the countable extension property ensuring that any finitely additive candidate probability measure on a standard field is also countably additive. We have not yet, however, actually demonstrated an example of the construction of a probability measure by extension. To accomplish this in the manner indicated we must explicitly construct a basis for a sample space and then provide a finitely additive set function on the associated finite fields. In this section the simplest possible examples of this construction are provided,

3.8 Bernoulli Processes

79

the Bernoulli processes or Bernoulli shifts. Although very simple, they will prove useful in constructing quite general classes of random processes. We have demonstrated an explicit basis for the special case of a discrete space and for the relatively complicated example of a product space with discrete coordinate spaces. Hence in principle we can construct measures on these spaces by providing a finitely additive set function for this basis. As a specific example, consider the binary sequence space (M+ , BM+ ) of (3.9). Define the set function m on thin cylinders c(bn ) = {x : x n = bn } by m(c(bn )) = 2−n ; all bn ∈ {0, 1}n , n = 1, 2, 3, . . . S∞ and extend this to Fn and hence to F = n=1 Fn in a finitely additive way, that is, any G ∈ F is in Fn for some nSand hence can be written N as a disjoint union of thin cylinders, say G = i=1 c(bin ). In this case dePN fine m(G) = i=1 m(c(biN )). This defines m for all events in F and the resulting m is finitely additive by construction. It was shown in Section 3.4 that the Fn , n = 1, 2, . . . form a basis for F , however, and hence m extends uniquely to a measure on BM+ . Thus with a basis we can easily construct probability measures by defining them on simple sets and demonstrating only finite and not countable additivity. Observe that in this case of equal probability on all thin cylinders of a given length, the probability of individual sequences must be 0 from the continuity of probability. For example, given a sequence x = (x0 , x1 , . . .), we have that n \ c(x n ) ↓ x i=1

and hence m({x}) = lim m(c(x n )) = lim 2−n = 0. n→∞

n→∞

The probability space (M+ , BM+ , m, T ) yields an important dynamical system by letting T be the shift on M+ . Similarly, the example yields an important random process by defining X0 as the zero-time coordinate function on M+ and defining the random process {Xn ; n ∈ Z+ } by defining Xn = T n X0 , that is, Xn (ω) = X0 (T n ω), ω ∈ M+ . Alternatively, any ω ∈ M+ has the form ω = (x0 , x1 , · · · ) and hence Xn (ω) = xn . From the definition it is immediate that for any positive integer n and any collection of sample times (t0 , t1 , . . . , tn−1 ) that the distribution for the random vector (Xt0 , Xt1 , . . . , Xtn−1 ) is described by the PMF pXt0 ,Xt2 ,...,Xtn−1 (x0 , x1 , . . . , xn−1 ) =

1 ; 2n n x = (x0 , x1 , . . . , xn−1 ) ∈ {0, 1}n .

80

3 Standard Alphabets

Thus, for example, the marginal PMF’s pXn (x) are all identical and hence equal to a common PMF pX (x). Furthermore, pXt1 ,Xt2 ,...,Xtn (x1 , x2 , . . . , xn ) =

n Y

pX (xi ).

(3.15)

i=1

When a discrete-alphabet random process has distributions of this form, i.e., the PMFs for sample vectors are products of a common PMF, the process is said to be independent and identically distributed (IID). If the alphabet is binary, the process is called a Bernoulli process, although the term is also used for more general discrete-alphabet processes satisfying (3.15). In the general (not necessary binary or equiprobable) discretealphabet case, the corresponding dynamical system is called a Bernoulli shift in ergodic theory. Bernoulli processes provide a classic model for random phenomena like flipping coins and spinning roulette wheels. The binary example with equiprobable 0 and 1 (pX (x) = 1/2 for x = 0, 1) will be referred to as the fair coin flipping process. The IID random process with an alphabet of 37 and a uniform marginal PMF is a roulette process (38 in the USA, where 00 is added to 0). We have constructed one-sided models for these processes, but twosided models with the same distributions follow in the same way by considering two-sided Cartesian products. Such random processes have no memory and may appear to be very limited in their ability to model real-life random phenomena, but they can lead to a useful and more general class of processes by coding as discussed in Section 2.6. This construction is explored in the next section.

3.9 Discrete B-Processes Fix a discrete alphabet A and let p denote a PMF on A and let P denote the corresponding distribution on BA (the power set of A since A is discrete). As in the previous constructions, construct a sequence measurable space (AZ , BZA ) and the Kolmogorov representation of an IID random process (AZ , BZA , µ) where µ is the process distribution from the Kolmogorov extension theorem applied to the product PMFs satisfying (3.15). Let TA denote the (left) shift on AZ . Given any discrete alphabet A and any PMF p on A the previous construction demonstrates the existence of an IID random process X = {Xn } with this alphabet and marginal PMF. As in Section 2.6, suppose that Y is a stationary coding of X; that is, there is a measurable mapping f : AZ → B from (AZ , BZA ) to a space (B, BB ) and the implied stationary code f¯ : AZ → B Z defined by

3.9 Discrete B-Processes

81

f¯(x Z ) = {f (TAn x Z ); n ∈ Z}. For a concrete simple example, see the construction of Section 2.6. Stationary coding yields a new random process {Yn } with a Kolmogorov directly-given representation (B Z , BZB , η) where η = µ f¯−1 . The corresponding dynamical system is (B Z , BZB , η, TB ), where TB is the shift on B Z . As in Section 2.6, f¯(TA x Z ) = TB f¯(x Z ).

(3.16)

The class of all random processes constructed in this fashion, that is, by stationary (or sliding-block) coding an IID process, is called the class of B-processes after Ornstein[77, 78]. While it is not by any means the only class of interesting processes, it is arguably the most important single class of random process models for both theory and practical applications. The dependence of the code f on the entire input sequence is for generality, in practice it may depend on the input sequence through only a few coordinate values. For example, f might view only a “window” of recent inputs, say (x−K+1 , x−K+2 , . . . , x0 ), in order to produce the output y0 = f (x Z ) = fK (x−K+1 , x−K+2 , . . . , x0 ) for a mapping fK : AK → B. As examples, suppose that the input process is binary, A = {0, 1}. fK might produce a 1 if there are an odd number of ones in in (x−K+1 , x−K+2 , . . . , x0 ) and a 0 otherwise. This can be viewed as a code, or as a linear filter using mod 2 arithmetic — y0 is the mod 2 sum of the x−K+1 , x−K+2 , . . . , x0 . As another code, we might divide AK into two classes, say C0 and C1 , that is, {C0 , C1 } for a partition of AK . fK is then defined as 0 if (x−K+1 , x−K+2 , . . . , x0 ) ∈ C0 and 1 otherwise. The output alphabet B need not be the same as A. For example, we could define 0 1 X fK (x−K+1 , x−K+2 , . . . , x0 ) = xi , K i=−K+1 which has a discrete alphabet {0, 1/K, . . . , (K − 1)/K} ⊂ [0, 1). Alternatively, we could form an exponentially weighted sum fK (x−K+1 , x−K+2 , . . . , x0 ) =

0 X

xi 2−i .

i=−K+1

Since the xi are binary, this sum converges as K → ∞ to produce a limiting code 0 X f (x) = xi 2−i−1 ∈ [0, 1]. (3.17) i=−∞

82

3 Standard Alphabets

While the mappings fK are measurable since the alphabet B is discrete, we have a problem now verifying measurability because the output space B is no longer discrete, it is the entire unit interval, and we have not yet constructed a σ -field for this case. We will do so in the next section. Given an IID input, none of these new processes is itself IID. By allowing general mappings f , one can construct quite complicated random processes. Just how general is a question that is fundamental to ergodic theory and will be occasionally considered in this book and its sequel. The immediate purpose, however, was to provide examples of the construction of complete random process models, and the IID processes and B-processes provide both the simplest possible example and an important more general class. Before continuing, we note a simple but important property of Bprocesses. Lemma 3.9. A stationary coding of a B-process is itself a B-process. Proof. A B-process is a stationary coding of an IID process. If the Bprocess is itself coded by a stationary mapping, then the cascade of the two mappings will be measurable and stationary, hence the final process is also a stationary coding of the original IID process. 2

3.10 Extension Without a Basis One goal of this section is to show that in general it is not easy to construct a basis as it was in the product space construction used to describe IID discrete-alphabet random processes. This is true even in an apparently simple case such as the unit interval [0, 1). Thus although a standard space may be useful in theory, it may be less useful in practice if one only knows that a basis exists, but is unable to construct one. A second goal of this section is to show that the inability to construct a basis may not be a problem when constructing certain explicit, but very important, probability measures. A basis is nice and important for theory because all finitely additive candidate probability measures on a basis are also countably additive and hence have an extension. If we are trying to prove a specific candidate probability measure is countably additive, we can sometimes use the structure of a related standard space without having to explicitly describe a basis. To demonstrate the above considerations we focus on a particular fundamentally important example: The Borel and Lebesgue measures on the unit interval. We shall show the difficulties in trying to find a basis and how to circumvent them by instead using the structure of binary sequence space.

3.10 Extension Without a Basis

83

Borel Measure on the Unit Interval Suppose that we wish to construct a probability measure m on the unit interval [0, 1) = {r : 0 ≤ r < 1} in such a way that the probability of an open interval of the form (a, b) = {r : a < r < b} for 0 ≤ a < b < 1 will be simply the length of the interval, b − a, that is, m((a, b)) = b − a.

(3.18)

The question is, is this enough to completely describe a measure on [0, 1)? That is, does this set function extend to a normalized, nonnegative, countably additive set function on a suitably defined σ -field? To begin, a useful σ -field for this alphabet is required. Here it is natural to use the the Borel field for metric spaces of Section 1.4. Let B([0, 1)) = σ (O([0, 1)), the collection of all open intervals of the form {x : a < x < b}, 0 ≤ a < b < 1, in [0, 1)). Note that B([0, 1)) must contain all of the one-point sets {a} for a ∈ [0, 1). To see this, if a > 0, {a} = limn→∞ (a − 1/n, a + 1/n) and for large enough n all of the sets (a−1/n, a+1/n) are in [0, 1) and the σ -field must contain such decreasing limits. If a = 0, the {a} = (0, 1)c and hence {0} is in the σ -field. Thus half open intervals of the form [a, b) = {r : a ≤ r < b} = {a} ∪ (a, b) are in σ (O([0, 1)). Lemma 3.10. B([0, 1)) is countably generated. Proof. Define G([0, 1)) as the collection of all half open intervals of the form [0, k/n) = {r : 0 ≤ r < k/n} for all positive integers 0 ≤ k < n. This collection is clearly countable, and hence the σ -field σ (G([0, 1))) is countably generated. The σ -field σ (G([0, 1))) must contain all half open intervals of the form [0, a) for 0 ≤ a < 1 since we can construct a sequence of decreasing rational an converging downward to a and hence [0, an ) is in the σ -field and hence so is its limit [0, a). Given any 0 ≤ a < b < 1, [a, b) = [0, b) − [0, a) must be in σ (G([0, 1))) since it is a set theoretic difference of two sets in σ (G([0, 1))). Points in [0, 1) must be in the σ field, e.g., since {a} = limn→∞ [a, a + 1/n). This means all open intervals are in σ (G([0, 1))) since (a, b) = [a, b) − {a}. Since O ⊂ σ (G([0, 1))), and the latter is a σ -field, σ (O) ⊂ σ (G([0, 1))). Conversely, every set in G([0, 1)) is a half-open interval and hence is in σ (O([0, 1))) and therefore σ (G([0, 1))) ⊂ σ (O([0, 1))). Thus σ (G([0, 1))) = σ (O([0, 1)). 2 Next consider the field generated by the countable collection G([0, 1)), F (G([0, 1))). It has been seen that this field is formed by including G([0, 1)) together with all finite sequences of set theoretic operations on members of G([0, 1)). It turns out, however, that there is an even simpler description of F (G([0, 1))).

84

3 Standard Alphabets

Lemma 3.11. Define as before G([0, 1)) as the collection of all half open intervals [0, k/n) for positive integers 0 ≤ k < n. Then every set F ∈ F (G([0, 1))) has the form F=

n [

[ai , bi )

i=1

for some finite positive integer n and rational 0 ≤ ai < bi < 1, i = 1, 2, . . . n, where the intervals are all disjoint. Note: The importance of the lemma is that it provides an obvious extension of the definition of m from G([0, 1)) to F (G([0, 1))) using finite additivity: n n X X m(F ) = m([ai , bi )) = (bi − ai ). (3.19) i=1

i=1

It immediately follows that m is finitely additive on F (G([0, 1))) Sn Proof. Consider the collection H of all sets of the form F = i=1 [ai , bi ) where the ai and bi are rational and the sets are disjoint. First consider complements. If F has this form, then Fc =

n \

[ai , bi )c =

i=1

 =

n \

([0, ai ) ∪ [bi , 1))

i=1



n \



[0, ai ) ∪ 

i=1

n \

 [bi , 1)

i=1

= [0, amin ) ∪ [bmax , 1),

(3.20)

where amin = min ai , bmax = max bi . i

i

c

This has the required form and hence F ∈ H . Next consider finite unions. Suppose that we have n sets Fi ; i = 1, 2, . . . , n in H , say Snj Fj = i=1 [aj,i , bj,i ), and we form the union F=

N [ j=1

Fj =

nj N [ [

[aj,i , bj,i ).

j=1 i=1

Sn This at least has the form F = i=1 [ci , di ), but the half-open intervals [ci , di ) need not be disjoint. It will be seen, however, that this collection can be used to obtain an equivalent representation of the desired form. First relabel things if necessary to order the ci so that 0 ≤ c1 ≤ c2 ≤ c3 ≤ · · · ≤ cn . Analogous to the construction based on (1.16) used in the proof of Lemma 1.17, define the sets

3.10 Extension Without a Basis

85

G1 = [c1 , d1 ) G2 = [c2 , d2 ) − G1 2 [

G3 = [c3 , d3 ) −

Gi

i=1

.. . Gn = [cn , dn ) −

n−2 [

Gi .

i=1

By construction the Gi are disjoint and F=

n [

Gi .

i=1

The proof is completed by showing that each of the Gi has the desired form of being a finite union of disjoint half-intervals and hence F also has this form since the Gi are disjoint and F is their union. Consider first G2 : G2 = [c2 , d2 ) − [c1 , d1 ) = [c2 , d2 ) ∩ [c1 , d1 )c = [c2 , d2 ) ∩ ([0, c1 ) ∪ [d1 , 1)) = [c2 , d2 ) ∩ [d1 , 1) since c2 ≥ c1 . Thus   if d2 ≤ d1  ∅ G2 = [c2 , d2 ) ∩ [d1 , 1) = [d1 , d2 ) if c2 ≤ d1 < d2   [c , d ) if d < c 2 2 1 2 and hence S G2 is either S2empty or is a half-open interval disjoint from G1 . 2 Thus also i=1 Gi = i=1 [ci , di ) is a finite union of disjoint intervals. Sn−1 Proceed by S induction. Suppose that i=0 Gi is a finite union of disjoint m intervals, say i=0 [ai , bi ), and use (3.20) to obtain Gn = [cn , dn ) −

n−1 [ i=0

 Gi = [cn , dn ) ∩ 

m [

c [ai , bi )

i=0

= [cn , dn ) ∩ ([0, amin ) ∪ [bmax , 1)) = ([cn , dn ) ∩ [0, amin )) ∪ ([cn , dn ) ∩ [bmax , 1)) which is either empty, a single half-open interval, or the union of two disjoint half-open intervals. Thus Gn is a finiteSunion of disjoint halfn open intervals (possibly empty) and hence so is i=1 Gi . 2

86

3 Standard Alphabets

Thus we have constructed a finitely-additive candidate probability measure on a countable field F (G([0, 1))) that generates B([0, 1)) The question now is whether m is also countably additive on F . This would follow if we new the measurable space ([0, 1), B([0, 1))) were standard, but we have not (yet) shown this to be the case. The discussion of standard spaces suggests that we should construct a basis Fn ↑ F = F (G([0, 1))). Knowing that m is finitely additive on Fn will then suffice to prove that it is countably additive on F and hence uniquely extends to B([0, 1)). So far so good. Unfortunately, however, there is no simple, easily describable basis for F . A natural guess is to have ever larger and finer Fn generated by intervals with rational endpoints. For example, let Fn be all finite unions of intervals of the form Gn,k = [k2−n , (k + 1)2−n ), k = 0, 1, . . . , 2n .

(3.21)

Analogous to the binary sequence example, the measure of the atoms Gn,k of Fn is 2−n and the set function extends to a finitely additive set function on F . Clearly Fn ↑ F , but unfortunately this simple sequence does not form a basis because atoms can collapse to empty sets! As an example, consider the atoms Gn ∈ Fn defined by Gn = [kn 2−n , (kn + 1)2−n ), where we choose kn = 2n−1 − 1 so that Gn = [2−1 − 2−n , 2−1 ). The atoms Gn are nonempty and have nonempty finite intersections, but ∞ \

Gn = ∅,

n=1

which violates the conditions required for Fn to form a basis. Unlike the discrete or product space case, there does not appear to be a simple fix and no simple variation on the above Fn appears to provide a basis. Thus we cannot immediately infer that m is countably additive on F . A second approach is to try to use an isomorphism as in Lemma 3.5 to extend m by actually extending a related measure in a known standard space. Again there is a natural first guess. Let (M+ , BM+ ) be the binary sequence space of (3.9), where BM+ is the σ -field generated by the thin cylinders. The space is standard and there is a natural mapping f : M+ → [0, 1) defined by ∞ X f (u) = uk 2−k−1 mod 1, (3.22) k=0

where u = (u0 , u1 , u2 , · · · ) and where by mod1 we take the value of the sum modulo 1 so that the sequence of all 1s is mapped into 0. This

3.10 Extension Without a Basis

87

technical detail allows us to consider the output alphabet of f to be the half-open unit interval [0, 1) instead of [0, 1]. The mapping f (u) is simply the real number whose binary expansion is .u0 u1 u2 · · · , and we have previously encountered it in (3.17) as a sliding-block code. The function f will be shortly proved to be measurable and hence one might hope that [0,1) would indeed inherit a basis from M+ through the mapping f . Sadly, this is not immediately the case — the problem being that f is a many-to-one mapping and not a one-to-one mapping. Hence it cannot be an isomorphism. For example, both 011111 · · · and 10000 · · · yield the value f (u) = 1/2. We could make f a one-to-one mapping by removing from M+ all sequences with only a finite number of 0’s, that is, form a new space M+ 0 = M+ − H

(3.23)

by removing the countable collection H of all sequences sequences with no 0’s, with a single 0, with two zeros, etc.: 11111 · · · , 01111 · · · , 00111 · · · , 10111 · · · , 000111 · · · and so on and consider f as a mapping from M+ 0 to [0,1). Here again we are stymied — none of the results proved so far permit us to conclude that the reduced space M+ 0 is standard. In fact, since we have removed individual sequences (rather than cylinders as in Lemma 3.8), there will be atoms collapsing to the empty set and the previously constructed basis for M+ does not work for the smaller space M+ 0 . We have reached an apparent impasse in that we can not (yet) either show that F has a basis or that m on F is countably additive. We resolve this dilemma by giving up trying to find a basis for F and instead use the standard space M+ to show directly that m is countably additive. We begin by showing that f : M+ → [0, 1) is measurable. From Lemma 2.4 this will follow if we can show that for any G ∈ F , f −1 (G) ∈ BM+ . Since every G ∈ F can be written as a finite union of disjoint sets of the form (3.21), it suffices to show that f −1 (G) ∈ BM+ for all sets Gn,k of the form (3.21). To find f −1 (Gn,k ) = {u : k2−n ≤ f (u) < (k + 1)2−n }, let bn = (b0 , b1 , · · · , bn−1 ) satisfy n−1 X j=0

bj 2n−j−1 =

n−1 X

bn−j−1 2j = k,

(3.24)

j=0

that is, (bn−1 , · · · , b0 ) is the unique binary representation of the integer k. The thin cylinder c(bn ) will then have the property that if u ∈ c(bn ), then

88

3 Standard Alphabets ∞ X

f (u) =

bj 2−j−1 ≥

j=0

n−1 X

bj 2−j−1 = 2−n

j=0

and f (u) ≤

n−1 X

n−1 X

bj 2n−j−1 = k2−n

j=0

bj 2−j−1 +

j=0

∞ X

2−j−1 = k2−n + 2−n

j=n

with equality on the upper bound if and only if bj = 1 for all j ≥ n. Thus if u ∈ c(bn ) − {bn 111 · · · }, then u ∈ f −1 (Gn,k ) and hence c(bn ) − {bn 111 · · · } ⊂ f −1 (Gn,k ). Conversely, if u ∈ f −1 (Gn,k ), then f (u) ∈ Gn,k and hence ∞ X k2−n ≤ uj 2−j−1 < (k + 1)2−n . (3.25) j=0

The left-hand inequality of (3.25) implies k≤

n−1 X

∞ X

uj 2n−j−1 + 2n

j=0

uj 2−j−1 ≤

j=n

n−1 X

uj 2n−j−1 + 1,

(3.26)

j=0

with equality on the right if and only if uj = 1 for all j > n. The righthand inequality of (3.25) implies that n−1 X

uj 2n−j−1 ≤

n−1 X

j=0

2n

∞ X

uj 2n−j−1 + 2n

j=0

∞ X

uj 2−j−1

j=n

uj 2n−j−1 < k + 1.

(3.27)

j=0

Combining (3.26) and (3.27) yields for u ∈ f −1 (Gn,k ) k−1≤

n−1 X

uj 2n−j−1 < k + 1,

(3.28)

j=0

with equality on the left if and only if uj = 1 for all j ≥ n. Since the sum in (3.28) must be an integer, it can only be k or k − 1. It can only be k − 1, however, if uj = 1 for all j ≥ n. Thus either n−1 X

uj 2n−j−1 = k,

j=0

in which case u ∈ c(bn ) with bn defined as in (3.24), or

3.10 Extension Without a Basis n−1 X

89

uj 2n−j−1 = k − 1 and uj = 1, all j ≥ n.

j=0

The second relation can only occur if for bn as in (3.24) we have that bn−1 = 1 and uj = bj , j = 0, . . . , n − 2, un−1 = 0, and uj = 1, j ≥ n. Letting w = (bn−1 0111 · · · ) denote this sequence, we therefore have that f −1 (Gn,k ) ⊂ c(bn ) ∪ w. In summary, every sequence in c(bn ) is contained in f −1 (Gn,k ) except the single sequence (bn 111 · · · ) and every sequence in f −1 (Gn,k ) is contained in c(bn ) except the sequence (bn−1 0111 · · · ) if bn = 1 in (3.24). Thus given Gn,k , if bn = 0 f −1 (Gn,k ) = c(bn ) − (bn 111 · · · )

(3.29)

(c(bn ) − (bn 111 · · · )) ∪ (bn−1 0111 · · · ).

(3.30)

and if bn = 1

In both cases, f −1 (Gn,k ) ∈ BM+ , proving that f is measurable. We note in passing that this shows that the sliding-block code of (3.17) is measurable and hence well-defined. Measurability of f almost immediately provides the desired conclusions: Simply specify a measure P on the standard space (M+ , BM+ ) via P (c(bn )) = 2−n ; all bn ∈ {0, 1}n and extend this to the field generated by the cylinders in an additive fashion as before. Since this field is standard, this provides a measure on (M+ , BM+ ). Now define a measure m on ([0, 1), B([0, 1))) by m(G) = P (f −1 (G)), all G ∈ B[0,1) . As in Section 3.2, f is a random variable and m is its distribution. In particular, m is a measure. From (3.28) and the fact that the individual sequences have 0 measure, we have that m([k2−n , (k + 1)2−n )) = P (c(bn )) = 2−n , k = 1, 2, . . . , n − 1. From the continuity of probability, this implies that for any interval [a, b), 1 > b > a ≥ 0, m([a, b)) = b − a. Also from the continuity of property, m({a}) = 0 for any point a ∈ [0, 1), and hence m((a, b)) = b−a as originally desired. Since this agrees with the original description of the measure on intervals, it must also agree on the field generated by intervals. Thus there exists a probability measure agreeing with (3.18) and hence with the additive extension of (3.18) to the field

90

3 Standard Alphabets

generated by the intervals. By the uniqueness of extension, the measure we have constructed must be the unique extension of (3.18). The probability measure m constructed in this way is called the Borel measure on [0, 1) and the space ([0, 1), B([0, 1)), m) is an example of a Borel measure space. The corresponding completed probability space of Lemma 1.14 ([0, 1), B, λ), with B = B(([0, 1))m , the completion of B(([0, 1)) with respect to m, and λ the extension of m to B, is called the Lebesgue measure space on [0, 1) and λ is the Lebesgue measure on [0, 1). The name “Lebesgue measure” is sometimes used in the literature to denote the uncompleted probability space as well as the completed measure. The dominant usage, however, is to use “Lebesgue” for the completed measure. Unfortunately “Borel measure” also has a more general meaning as any measure on a Borel measurable space. The meaning should be clear from context, and we will use the articles “a” and “the” to distinguish between the general and particular cases. We have not shown that the measurable space ([0, 1), B([0, 1))) we have constructed based on the unit interval is standard, all we have shown is that for this measurable space we could extend a specific finitely additive set function (candidate measure) to a measure — the Borel measure — on this measurable space. (We will show that this measurable space is standard in the next chapter, but this will require additional work.) Thus even without a basis or an isomorphism, we can construct probability measures on a given space if we have a measurable mapping from a standard space to the given space. The example of the Borel measure space may seem artificially simple, but it will turn out to lead to a surprisingly general class of probability spaces, which is defined in the next and final section of this chapter. The same construction can be used to obtain a Borel measure space (and the corresponding Lebesgue measure space by completion) on any finite interval [a, b) for −∞ < a < b < ∞. Specifically, there is a measure m on [a, b) and a σ -field of subsets generated by G[a,b) , the collection all sets of the form [a, a + k/n) for all positive integers k, n such that k/n ≤ b−a, such that for any a ≤ c < d < b m([c, d)) = d−c, the length of the interval. The measure can be made into a probability measure by normalizing, that is, by defining m([c, d)) = (d − c)/(b − a). The name “Lebesgue measure” is reserved for the completed unnormalized case, so that the measure of an interval is the length of the interval.

3.11 Lebesgue Spaces The mapping f that took binary sequences into numbers in the unit interval was almost an isomorphism, and it was the failure to actually be an isomorphism that caused all of the technical difficulties, since other-

3.12 Lebesgue Measure on the Real Line

91

wise it would have been immediate that ([0, 1), B([0, 1))) was standard. Suppose that f is cut down to the reduced space M+ 0 = M+ − H of 3.23. Then the mapping f : M+ 0 → [0, 1) is still measurable since we have removed a countable number of sequences from its domain of definition and the mapping f is now one-to-one. Furthermore its inverse f −1 is well defined, it is the unique binary expansion (without having only a finite number of 0s) of [0, 1). Since M+ 0 was formed by removing a countable set from, f is an isomorphism except for a null set in the binary sequence space and hence is an isomorphism mod 0 as introduced in Section 2.7. This sets the stage for a general class of probability spaces. A complete probability space (Ω, B, P ) is called a Lebesgue space if P is a mixture of the form P = λP1 + (1 − λ)P2 for λ ∈ [0, 1], where (Ω, B, P1 ) is isomorphic (mod 0) to a Lebesgue measure space and where P2 is discrete probability measure, that is, there are a countable number of points xi ∈ Ω, i = 0, 1, 2, . . ., with probabilities pi = P1 ({xi }) and X pi ; F ∈ Ω. P2 (F ) = i:xi ∈F

The theory of Lebesgue spaces is generally credited to Rohlin [90] and Halmos and von Neuman [48]. Lebesgue spaces consist of a continuous portion isomorphic to and hence essentially the same as a uniform distribution on the unit interval plus a purely discrete component, with both extremes being possible. Thus, for example, the Kolmogorov model for fair coin flips constructed earlier in this chapter is a Lebesgue space. Lebesgue spaces are the most common probability space used in ergodic theory because of the useful properties they possess (including those developed in this chapter) and their generality (which will be discussed later). They are sometimes referred to as “standard Lebesgue spaces” and we shall see that they are essentially equivalent (if we ignore null sets) to the probability spaces of most interest here — probability spaces built from Polish spaces. Our focus will provide more structure based on the underlying metric, and the structure will prove useful in the sequel book on information theory [30]. We shall see in the next chapter that the measurable space constructed for a Lebesgue space is, in fact, a standard measurable space.

3.12 Lebesgue Measure on the Real Line This section is the only section of this book to focus on a measure that is not a probability measure and cannot be normalized to make a probability measure — the Lebesgue measure on the real line. The case is important in the context of this book because it provides a means of de-

92

3 Standard Alphabets

scribing the probability distributions of continuous random variables in terms of integrals of probability density functions. For any positive integer n let ([−n, n), B([−n, n)), λn ) denote the Lebesgue measure space on [−n, n) as developed in the previous section. Given the real line R, define the collection of subsets BR = {all F : F ∩ [−n, n) ∈ B([−n, n)) for all n = 1, 2, . . .}. It is easy to verify that BR is a σ -field. First consider complements. If F ∈ BR , then for every n F ∩[−n, n) ∈ B([−n, n)) and hence the complement of F ∩[−n, n) in [−n, n), [−n, n)−F −[−n, n)∩F c is in B([−n, n)) for all n, which implies that F c ∈ BR . Next consider countable unions. Suppose that for i = 1, 2, . . . Fi ∈ BR and hence Fi ∩ [−n, n) ∈ S B([−n, n)) for all S n. Since B([−n, n)) is a σS -field, i (Fi ∩ [−n, n)) = ( i Fi ) ∩ [−n, n) ∈ B([−n, n)) and therefore i Fi ∈ BR . Thus BR is indeed a σ -field. To define a measure on the measurable space, (R, BR ), let F ∈ BR and consider the measures λn (F ∩ [−n, n)). First observe that if F ∈ B[−k,k) for a fixed k, then λn (F ) = λk (F ) for all n ≥ k. This is obvious for intervals and hence also for the field generated by the intervals. Since the two measures agree on a generating field, they must be the same. This implies that for a fixed F , λn (F ∩ [−n, n)) is an non-decreasing sequence in n since λn+1 (F ∩ [−(n + 1), n + 1)) = λn+1 (F ∩ [−n, n)) + λn+1 (F ∩ ([−(n + 1), n + 1) − [−n, n)) ≥ λn+1 (F ∩ [−n, n)) = λn (F ∩ [−n, n)). Since λn (F ∩[−n, n)) is a nondecreasing and bounded by 1, it has a limit, say λ(F ) = lim λn (F ∩ [−n, n)). (3.31) n→∞

The set function λ is obviously nonnegative, so to demonstrate that it is a measure only countable additivity need be demonstrated. Suppose that Fk ∈ BR for k = 1, 2, . . . are disjoint. Then   ∞ ∞ [ [ Fk ) = lim λn ( Fk  ∩ [−n, n)) λ( k=1

n→∞

k=1

= lim λn ( n→∞

= lim

n→∞

∞ [

(Fk ∩ [−n, n)))

k=1 ∞ X

λn (Fk ∩ [−n, n)).

k=1

Since λn (Fk ∩ [−n, n)) is nondecreasing, from Corollary 5.8 the limit can be taken term-by-term to obtain

3.12 Lebesgue Measure on the Real Line

λ(

∞ [

k=1

Fk ) =

∞ X k=1

lim λn (Fk ∩ [−n, n)) =

n→∞

93 ∞ X

λ(Fk ),

k=1

and hence λ is a measure on (R, BR ), which is called the Lebesgue measure on the real line. S∞ The Lebesgue measure on the real line is sigmafinite since R = n=1 [−n, n) and λ([−n, n)) = λn ([−n, n)) = 2n is finite for all n. Although the Lebesgue measure on the real line is not itself a probability measure, it will later be seen to be quite useful in constructing probability measures.

“This page left intentionally blank.”

Chapter 4

Standard Borel Spaces

Abstract The most important class of standard spaces is developed, standard Borel spaces formed using separate, complete metric spaces or Polish spaces. The spaces are shown to be standard by constructing an isomorphism with a standard subset of a discrete sequence space.

4.1 Products of Polish Spaces This section and the next present properties of Polish spaces that resemble those of standard spaces. We show that Cartesian products of Polish spaces are Polish spaces, and that certain subsets of Polish spaces are also Polish. Given a Polish space A and an index set I, define a metric dI on the product space AI as in Example 1.12. The following lemma shows that the product space is also Polish. Lemma 4.1. (a) If A is a Polish space with metric d and I is a countable index set, then the product space AI is a Polish space with respect to the product space metric dI . (b) In addition, if B(A) is the Borel σ -field of subsets of A with respect to the metric d, then the product σ -field of Section 2.4, B(A)I = σ (rect(Bi , i ∈ N)), where the Bi are all replicas of B(A), is exactly the Borel σ -field of AI with respect to dI . That is, B(A)I = B(AI ). Proof. (a) If A has a dense subset, say B, then a dense subset for AI can be obtained as the class of all sequences taking values in B on a finite number of coordinates and taking some fixed reference value, say b, on the remaining coordinates. Hence AI is separable. If a sequence I I I xn = {xn,i , i ∈ I} is Cauchy, then dI (xn , xm ) → 0 as n, m → ∞. From Lemma 1.3 this implies that for each coordinate d(xn,i , xm,i ) → 0 as R.M. Gray, Probability, Random Processes, and Ergodic Properties, DOI: 10.1007/978-1-4419-1090-5_4, © Springer Science + Business Media, LLC 2009

95

96

4 Standard Borel Spaces

n, m → ∞ and hence the sequence xn,i is a Cauchy sequence in A for each i. Since A is complete, the sequence has a limit, say yi . From Lemma I 1.3 again, this implies that xn →n→∞ y I and hence AI is complete. (b) We next show that the product σ -field of a countably generated Borel σ -field is the Borel σ -field with respect to the product space metric; that is, both means of constructing σ -fields yield the same result. (This result need not be true for products of Borel σ -fields that are not countably generated.) For each integer n let Πn : AI → A be the coordinate or sampling function Πn (x I ) = xn . It is easy to show that Πn is a continuous function I and hence is also measurable from Lemma 2.1. Thus Π−1 n (F ) ∈ B(A ) for all F ∈ B(A) and n ∈ N. This implies that all finite intersections of such sets and hence all rectangles with coordinates in B(A) must be in B(AI ). Since these rectangles generate B(A)I , we have shown that B(A)I ⊂ B(AI ). Conversely, let Sr (x) be an open sphere in AI : Sr (x) = {y : y ∈ AI , dI (x, y) < r }. Observe that for any n dI (x, y) ≤

n X k=1

2−k

d(xk , yk ) + 2−n 1 + d(xk , yk )

and hence we can write Sr (x) =

[

[ Pn

n:2−n
k=1

n \

dk
I Π−1 k ({a : d (xk , a)

dk }), 1 − dk

where the union over the {dk } is restricted to rational dk . Every piece of the union on the right-hand side is contained in the sphere on the left, and every point in the sphere on the left is eventually included in one of the terms in the union on the right for sufficiently large n. Thus all spheres in AI under dI can be written as countable unions of of rectangles with coordinates in B(A), and hence these spheres are in B(A)I . Since AI is separable under dI all open sets in AN can be written as a countable union of spheres in AI using (1.18), and hence all open sets in AI are also in B(A)I . Since these open sets generate B(AI ), B(AI ) ⊂ B(A)I , completing the proof. 2 Example 4.1. Integer Sequence Spaces. As a simple example of some of the ideas under consideration, consider the space of integer sequences introduced in Lemma 3.7. Let B = Z+ and A = N = B B , the space of sequences with nonnegative integer values. A discrete space such as A0 is a Polish space with respect to the Hamming metric dH of Example 1.1. This is trivial since the countable set of

4.2 Subspaces of Polish Spaces

97

symbols is dense in itself and if a sequence {an } is Cauchy, then choosing  < 1 implies that there is an integer N such that d(an , am ) < 1 and hence an = am for all n, m ≥ N. Note that an open sphere S (a) in B with  < 1 is just a point and that any subset of B is a countable union of such open spheres. Thus in this case the Borel σ -field B(Z+ ) is just the power set, the collection of all subsets of B. The sequence space A is a metric space with the metric of Example 1.12, which with a coordinate Hamming distance becomes dZ+ (x, y) =

∞ 1 X −i 2 dH (xi , yi ). 2 i=0

(4.1)

Applying Lemma 4.1 (b) and (4.1) we have Z

B(N) = B(Z++ ) = B(Z+ )Z+ .

(4.2)

It is interesting to compare the structure used in Lemma 3.7 with the Borel or topological structure currently under consideration. Recall that in Lemma 3.7 the σ -field B(Z+ )Z+ was obtained as the σ -field generated by the thin cylinders, the rectangles or thin cylinders having the form {x : x n = an } for some positive integer n and some n-tuple an ∈ B n . For convenience abbreviate dB to d. Suppose we know that d(x, y) < 1/2. Then necessarily x0 = y0 and hence both sequences must lie in a common thin cylinder, the thin cylinder {u : u0 = x0 }. Similarly, if d(x, y) < 1/4, then necessarily x 2 = y 2 and both sequences must again lie in a common thin cylinder, {u : u2 = x 2 }. In general, if d(x, y) <  and  ≤ 2−(n+1) , then x n = y n . Thus if x ∈ A and  ≤ 2−(n+1) , then S (x) = {y : y n = x n }.

(4.3)

This means that the thin cylinders are simply the open spheres in A! Thus the generating class used in Lemma 3.7 is really the same as that used to obtain a Borel σ -field for A. In other words, not only are the σ -fields of (4.2) all equal, the collections of sets used to generate these fields are the same.

4.2 Subspaces of Polish Spaces The following result shows that certain subsets of Polish spaces are themselves Polish, although in some cases we need to consider an equivalent metric rather than the original metric. Lemma 4.2. Let A be a complete, separable metric space with respect to a metric d. If F = C, C a closed subset of A, then F is itself a complete,

98

4 Standard Borel Spaces

separable metric space with respect to d. If F = O, where O is an open subset of A, then F is itself a complete separable metric space with respect to a metric d0 such that d0 -convergence and d-convergence are equivalent on F and hence d and d0 are equivalent metrics on F . In fact, the following provides such a metric: d0 (x, y) = d(x, y) + |

1 1 − |. c d(x, O ) d(y, O c )

(4.4)

If F is the intersection of a closed set C with an open set O of A, then F is a complete separable metric space with respect to a metric d0 such that d0 -convergence and d-convergence are equivalent on F , and hence d and d0 are equivalent metrics on F . Here, too, d0 of (4.4) provides such a metric. Proof. Suppose that F is an arbitrary subset of a separable metric space A with metric d having the countable base VA = {Vi ; i = 1, 2, . . .} of (1.18). Define the class of sets VF = {F ∩ Vi ; i = 1, 2, . . .}, where we eliminate all empty sets. Let yi denote an arbitrary point in F ∩ Vi . We claim that the set of all yi is dense in F and hence F is separable under d. To see this let x ∈ F . Given T  there is a Vi of radius less than  which contains x and hence F Vi is not empty (since it at least contains x ∈ F ) and d(x, yi ) ≤ sup d(x, y) ≤ sup d(x, y) ≤ 2. y∈F ∩Vi

y∈Vi

Thus for any x in F we can find a yi ∈ VF that is arbitrarily close to it. Thus VF is dense in F and hence F is separable with respect to d. The resulting metric space is not in general complete, however, because completeness of A only ensures that Cauchy sequences converge to some point in A and hence a Cauchy sequence of points in F may converge to a point not in F . If F is closed, however, it contains all of its limit points and hence F is complete with respect to the original metric. This completes the proof of the lemma for the case of a closed subset. If F is an open set O, then we can find a metric that makes O complete by modifying the original metric to blow up near the boundary and hence prevent Cauchy sequences from converging to something not in O. In other words, we modify the metric so that if a sequence an ∈ O is Cauchy with respect to d but converges to a point outside O, then the sequence will not be Cauchy with respect to d0 . Define the metric d0 on O as in (4.4). Observe that by construction d ≤ d0 , and hence convergence or Cauchy convergence under d0 implies the same under d. From the triangle inequality, |d(an , O c ) − d(a, O c )| ≤ d(an , a). Thus provided the an and a are in O, d-convergence implies d0 -convergence. Thus dconvergence and d0 -convergence are equivalent on O and hence yield the same open sets from Lemma 1.4. In addition, points in O are limit points

4.2 Subspaces of Polish Spaces

99

under d if and only if they are limit points under d0 and hence O is separable with respect to d0 since it is separable with respect to d. Since a sequence Cauchy under d0 is also Cauchy under d, a d0 -Cauchy sequence must have a limit since A is complete with respect to d. This point must also be in O, however, since otherwise 1/d(an , O c ) → 1/d(a, O c ) = ∞ and hence d0 (an , am ) could not be converging to zero and hence {an } could not be Cauchy. (Recall that for such a double limit to tend to zero, it must so tend regardless of the manner in which m and n go to infinity.) Thus O is complete. Since O is complete and separable, it is Polish. If F is the intersection of a closed set C and an open set O, then the preceding metric d0 yields a Polish space with the desired properties for the same reasons. In particular, d-convergence and d0 -convergence are equivalent for the same reasons and a d0 -Cauchy sequence must converge to a point that is in the intersection of C and O: it must be in C since C is closed and it must be in O or d0 would blow up. 2 It is important to point out that d-convergence and d0 -convergence are only claimed to be equivalent within F ; that is, a sequence an in F converges to a point a in F under d if and only if it converges under d0 . It is possible, however, for a sequence an to converge to a point a not in F under d and to not converge at all under d0 (the d0 distance is not even defined for points not in F ). In other words, F may not be a closed subset of A with respect to d. Example 4.2. Integer Sequence Spaces Revisited We return to the integer sequence space of Example 4.1 to consider a simple but important special case where a subset of a Polish space is Polish. The following manipulation will resemble that of Lemma 3.4. Let A and d be as in that lemma and recall that A is Polish and has a Borel σ -field B(A) = B(B)B . Suppose that we have a S countable collection Fn , n = 1, 2, . . . of thin cylinders with union F = n Fn . Consider the sequence space ∞ ∞ [ \ ˜=A− A Fn = Fnc n=1

n=1

formed by removing the countable collection of thin cylinders from A. Since the thin cylinders are open sets, their complements are closed. ˜ is a closed subSince an arbitrary intersection of closed sets is closed, A ˜ set of A. Thus A is itself Polish with respect to the same metric, d. ˜ can be easily related by The Borel σ -fields of the spaces A and A observing as in Example 4.1 that an open sphere about a sequence in either sequence space is simply a thin cylinder. Every thin cylinder in A is either contained in or disjoint from A−F . Thus B(A) is generated by all ˜ is generated by the thin cylinders in A ˜ of the thin cylinders in A and B(A) and hence by all sets of the form {x : x n = an }∩(A−F ), that is, by all the ˜ is also thin cylinders except those removed. Thus every open sphere in A

100

4 Standard Borel Spaces

˜ ⊂ B(A) ∩ (A − F ). an open sphere in A and is contained in F c ; thus B(A) Similarly, every open sphere in A that is contained in F is also an open ˜ and hence the reverse inclusion holds. In summary, sphere in A, ˜ = B(A) ∩ (A − F ). B(A)

(4.5)

Carving We now turn to a technique for carving up complete, separable metric spaces into pieces with a useful structure. From the previous lemma, we will then also be able to carve up certain other sets in a similar manner. Lemma 4.3. (The carving lemma.) Let A be a complete, separable metric space. Given any real r > 0, there exists a countably infinite family of Borel sets (sets in B(A)) {Gn , n = 1, 2, . . .} with the following properties: (a) The {Gn } partition A; that is, they are disjoint and their union is A. (b) Each {Gn } is the intersection of the closure of an open sphere and an open set. (c) For all n, diam(Gn ) ≤ 2r . Proof. Let {ai , 1 = 0, 1, 2, . . .} be a countable dense set in A. Since A is separable, the family {Vi , i = 0, 1, 2, . . .} of the closures of the open spheres Si = {x : d(x, ai ) < r } covers A; that is, its union is A. In addition, diam(Vi ) = diam(Si ) ≤ r . The proof uses the construction of (3.11) in a manner similar to that used in the proof of Lemma 1.17. Construct the set Gi (1) as the set Vi with all sets of lower index removed and all empty sets omitted; that is, G0 (1) = V0 G1 (1) = V1 − V0 .. . [ Gn (1) = Vn − Vi i
= Vn −

[

Gi (1)

i
.. . where we remove all empty sets from the sequence. The sets are clearly disjoint and partition B. Furthermore, each set Gn (1) has the form \ Vic ); Gn (1) = Vn ∩ ( i
4.3 Polish Schemes

101

that is, it is the intersection of the closure of an open sphere with an open set. Since Gn (1) is a subset of Vn , it has diameter less than r . This provides the sequence with properties (a)–(c). 2 We are now ready to iterate on the previous lemmas to show that we can carve up Polish spaces into shrinking pieces that are also Polish spaces that can then be themselves carved up. This representation of a Polish space is called a scheme, and it is detailed in the next section.

Exercises 1. Prove the claims made in the first paragraph of this section. 2. Let (Ω, B, P ) be a probability space and let A = B, the collection of all events. Two events are considered to be the same if the probability of the symmetric difference is 0. Define a metric d on A as in Example 1.15. Is the corresponding Borel field a countably generated σ -field? Is A a separable metric space under d? 3. Prove that in the Example 4.2 that all finite dimensional rectangles in A are open sets with respect to d. Since the complements of rectangles are other rectangles, this means that all rectangles (including the thin cylinders) are also closed. Thus the thin cylinders are both open and closed sets. (These unusual properties occur because the coordinate alphabets are discrete.)

4.3 Polish Schemes In this section we develop a representation of a Polish space that resembles an infinite version of the basis used to define standard spaces; the principal difference is that at each level we can now have an infinite rather than finite number of sets, but now the diameter of the sets must be decreasing as we descend through the levels. It will provide a canonical representation of Polish spaces and will provide the key element for proving that Polish spaces yield standard Borel spaces. Recall that Z+ ={0, 1, 2, . . .}, Zn + is the corresponding finite dimenZ sional Cartesian product, and N = Z++ is the one-sided infinite Cartesian product. Given a metric space A, a scheme is defined as a family of sets A(un ), n = 1, 2, . . ., indexed by un in Zn + with the following properties: 1. A(un ) ⊂ A(un−1 ) (the sets are decreasing),

(4.6)

102

4 Standard Borel Spaces

A(un ) ∩ A(v n ) = ∅ if un 6= v n

(4.7)

(the sets are disjoint), 2. lim diam(A(un )) = 0, all u = {u0 , u1 , . . .} ∈ N.

n→∞

(4.8)

3. If a decreasing sequence of sets A(un ), n = 1, 2, . . . are all nonempty, then the limit is not empty; that is, A(un ) 6= ∅ all n implies Au 6= ∅,

(4.9)

where Au is defined by Au =

∞ \

A(un ).

(4.10)

n=1

The scheme is said to be measurable if all of the sets A(un ) are Borel sets. The collection of all nonempty sets Au , u ∈ N, of the form (4.10) will be called the kernel of the scheme. We shall let A(u0 ) denote the original space A. Theorem 4.1. If A is a complete, separable metric space, then it is the kernel of a measurable scheme. Proof. Construct the sequence Gn (1) as in the first part of the carving lemma to satisfy properties (a)–(c) of that lemma. Choose a radius r of, say, 1. This produces the first level sets A(u1 ), u1 ∈ Z+ . Each set thus produced is the intersection of the d closure C(u1 ) of a d-open sphere and a d−open set O(u1 ) in A, and hence by Lemma 4.2 each set A(u1 ) is itself a Polish space with respect to a metric du1 that within A(u1 ) is equivalent to d. We shall also make use of the property of the specific equivalent metric of (4.4) of that lemma that du1 (a, b) ≤ d(a, b). Henceforth let du0 denote the original metric d. The closure of A(u1 ) with respect to the original metric d may not be in A(u1 ) itself, but it is in the set C(u1 ). We then repeat the procedure on these sets by applying Lemma 4.2 to each of the new Polish spaces with an r of 1/2 to obtain sets A(u2 ) that are intersections of a du1 closure C(u2 ) of an open sphere and a du1 -open set O(u2 ). Each of these sets is then itself a Polish space with respect to a metric du2 (x, y) which is equivalent to the parent metric du1 (x, y) within A(u2 ). In addition, du2 (x, y) ≥ du1 (x, y). Since A(u2 ) ⊂ A(u1 ) and the parent metric du1 is equivalent to its parent metric du0 within A(u1 ), du2 is equivalent to the original metric d within A(u2 ). A final crucial property is that the closure of A(u2 ) with respect to its parent metric du1 is contained within its parent set A(u1 ). This is true since A(u2 ) = C(u2 ) ∩ O(u2 ) and C(u2 ) is a du1 -closed subset of

4.3 Polish Schemes

103

A(u1 ) and therefore the du1 - closure of A(u2 ) lies within C(u2 ) and hence within A(u1 ). We now continue in this way to construct succeeding levels of the scheme so that the preceding properties are retained at each level. At level n, n = 1, 2, . . ., we have sets A(un ) = C(un ) ∩ O(un ),

(4.11)

where C(un ) is a dun−1 -closed subset of A(un−1 ) and O(un ) is a dun−1 open subset of A(un−1 ). The set C(un ) is the dun−1 -closure of an open ball of the form S(un−1 , i) = S1/n (ci ; un−1 ) = {x : x ∈ A(un−1 ), dun−1 (x, ci ) < 1/n}, i = 0, 1, 2, . . . .

(4.12)

Each set A(un ) is then itself a Polish space with metric dun (x, y) = 1 1 . dun−1 (x, y) + − dun−1 (x, O(un )c ) dun−1 (y, O(un )c )

(4.13)

We have by construction that for any un in Zn + d ≤ du1 ≤ du2 ≤ . . . ≤ dun

(4.14)

and hence since A(un ) is in the closure of an open sphere of the form (4.12) having dun−1 radius 1/n, we have also that the d-diameter of the sets satisfies diam(A(un )) ≤ 2/n. (4.15) Furthermore, on A(un ), dun (the complete metric) and dun−1 yield equivalent convergence and are equivalent metrics. Since each A(un ) is contained in all of the Aui for i = 0, . . . , n − 1, this implies that given ak ∈ A(un ), k = 1, 2, . . . , and a ∈ A(un ), then dun (ak , a) → 0 iff duj (ak , a) → 0 for all j = 0, 1, 2, . . . , n − 1. (4.16) k→∞

k→∞

This sequence of sets satisfies conditions (4.6)–(4.8). To complete the proof of the theorem we need only show that (4.9) is satisfied. If we can demonstrate that the d-closure in A of the set A(un ) lies completely in its parent set A(un−1 ), then (4.9) will follow immediately from Corollary 1.2. We next provide this demonstration. Say we have a sequence ak ; k = 1, 2, . . . such that ak ∈ A(un ), all k, and a point a ∈ A such that d(ak , a) → 0 as k → ∞. We will be done if we can show that necessarily a ∈ A(un−1 ). We accomplish this by

104

4 Standard Borel Spaces

assuming the contrary and developing a contradiction. Hence we assume that ak → a in d, but a 6∈ A(un−1 ). First observe from (4.14) and (4.12) that since all of the ak are in the closure of a sphere of dun−1 -radius 1/n centered at, say, c, then we must have that duj (ak , c) ≤ 1/n, j = 0, 1, 2, . . . , n − 1; all k;

(4.17)

that is, the ak must all be close to the center c ∈ A(un−1 ) of the sphere used to construct C(un ) with respect to all of the previous metrics. By assumption a 6∈ A(un−1 ). Suppose that l is the smallest index such that a 6∈ A(u1 ). Since the A(ui ) are nested, we will then have that a ∈ A(ui ) for i < l. In addition, l ≥ 1 since A(u0 ) is the entire space and hence clearly a ∈ A(u0 ). Since a ∈ A(ul−1 ) and ak ∈ A(u1 ) ⊂ A(ul−1 ), from (4.16) applied to A(ul−1 ) d-convergence of ak to a is equivalent to dul−1 -convergence. Since the set C(u1 ) ⊂ A(ul−1 ) used to form A(u1 ) via A(u1 ) = C(u1 ) ∩ O(u1 ) is closed under dul−1 , we must have that a ∈ C(u1 ). Thus since a 6∈ A(ul ) = C(ul ) ∩ O(ul ) and a ∈ C(ul ), then necessarily a ∈ O(u1 )c . This, however, contradicts (4.17) for j = l since 1 1 dul (ak , c) = dul−1 (ak , c) + − l c l c dul−1 (ak , O(u ) ) dul−1 (c, O(u ) ) 1 1 → ∞ ≥ − dul−1 (ak , O(ul )c ) dul−1 (c, O(ul )c ) k→∞ since dul−1 (ak , O(ul )c ) =

inf

b∈O(ul )c )

dul−1 (ak , b) ≤ dul−1 (ak , a) → 0, k→∞

where the final inequality follows since a ∈ O(u1 )c . Thus a cannot be in A(ul−1 ). But this contradicts the assumption that l is the smallest index for which a 6∈ A(ul ) and thereby completes the proof that the given sequence of sets is indeed a scheme. Since at each stage every point in A must be in one of the sets A(un ), if we define A(un )(x) as the set containing x, then {x} =

∞ \ n=1

A(un )(x)

(4.18)

4.3 Polish Schemes

105

and hence every singleton set {x} is in the kernel of the scheme. Observe that many of the sets in the scheme may be empty and not all sequences Z u in N = Z++ will produce a point via (4.18). Since every point in A is in the kernel of a measurable scheme, (4.18) gives a map f : A → N where f (x) is the u for which (4.18) is satisfied. This scheme can be thought of as a quantizer mapping a continuous space A into a discrete representation in the following sense: Suppose that we view a point a and we encode it into a sequence of integers un ; n = 1, 2, . . . which are then viewed by a receiver. The receiver at time n uses the n integers to produce an estimate or reproduction of the original point, say gn (un ). The badness or distortion of the reproduction can be measured by d(a, gn (un )). The goal is to decrease this distortion to 0 as n → ∞. The scheme provides such a code: Let the encoder at time n map x into the un for which x ∈ A(un ) and let the decoder gn (un ) simply produce an arbitrary member of A(un ) (if it is not empty). This distortion will be bound above by diam(A(un )) ≤ 2/n. This produces the sequence encoder mapping f : A → N. Unfortunately the mapping f is in general into and not onto. For exn ample, if at level n for some n-tuple bn ∈ Zn + A(b ) is empty, then the encoding will produce no sequences that begin with bn since there can never be an x in A(bn ). In other words, any sequence u with un = bn cannot be of the form u = f (x) for any x ∈ A. Roughly speaking, not all of the integers are useful at each level since some of the cells may be empty. Alternatively, some integer sequences do not correspond to (they cannot be decoded into) any point in A. We pause to tackle the problems of empty cells. Let N1 denote all integers u1 for which A(u1 ) is empty. Let N2 denote the collection of all integer pairs u2 such that A(u2 ) is empty but A(u1 ) is not. That is, once we throw out an empty cell we no longer consider its descendants. Continue in this way to form the sets Nk consisting of all uk for which A(uk ) is empty but A(uk−1 ) is not. These sets are clearly all countable. Define next the space N0 = N −

∞ [

[

n=1 bn ∈Nn

{u : un = bn } =

∞ \

\

{u : un = bn }c .

(4.19)

n=1 bn ∈Nn

Thus N0 is formed by removing all thin cylinders of the form {u : un = bn } for some bn ∈ Nn , that is, all thin cylinders for which the first part of the sequence corresponds to an empty cell. This space has several convenient properties: 1. For all x ∈ A, f (x) ∈ N0 . This follows since if x ∈ A, then x ∈ A(un ) for some un for every n; that is, x can never lie in an empty cell, and hence f (x) cannot be in the portion of N that was removed.

106

4 Standard Borel Spaces

2. The set N0 is a Borel set since it is formed by removing a countable number of thin cylinders from the entire space. 3. If we form a σ -field of N0 in the usual way as B(N) ∩ N0 , then from Lemma 3.4 (or Lemma 3.8) the resulting measurable space (N0 , B(N)∩ N0 ) is standard. 4. From Examples 3.2.1 and 3.2.2 and (4.5), the Borel field B(N0 ) (using dZ+ ) is the same as B(N) ∩ N0 . Thus with Property 3, (N0 , B(N0 )) is standard. 5. In the reduced space N0 , all of the sequences correspond to sequences of nonempty cells in A. Because of the properties of a scheme, any descending sequence of nonempty cells must contain a point, and if the integer sequences ever differ, the points must differ. Define the mapping g : N0 → A by ∞ \

g(u) =

A(un ).

n=1

Since every point in A can be encoded uniquely into some integer sequence in N0 , the mapping g must be onto. 6. N0 = f (A); that is, N0 is the range space of N. To see this, observe first that Property 1 implies that f (A) ⊂ N0 . Property 5 implies that every point in N0 must be of the form f (x) for some x ∈ A, and hence N0 ⊂ f (A), thereby proving the claim. Note that with Property 2 this provides an example wherein the forward image f (A) of a Borel set is also a Borel set. By construction we have for any x ∈ A that g(f (x)) = x and that for any u ∈ N0 f (g(u)) = u. Thus if we define fˆ : A → N0 as the mapping f considered as having a range N0 , then fˆ is onto, the mapping g is just the inverse of the mapping fˆ, and the mappings are one-to-one. If these mappings are measurable, then the spaces (A, B(A)) and (N0 , B(N0 )) are isomorphic. Thus if we can show that the mappings fˆ and g are measurable, then (A, B(A)) will be proved standard by Lemma 3.5 and Property 4 because it is isomorphic to a standard space. We now prove this measurability and complete the proof. First consider the mapping g : N0 → A. N and hence the subspace N0 are metric spaces under the product space metric dZ+ using the Hamming metric of Example 1.1; that is, dZ+ (u, v) =

∞ X

2−i dH (ui , vi ),

i=1

Given an  > 0, find an n such that 2/n <  and then choose a δ < 2−n . If dZ+ (u, v) < δ, then necessarily un = v n which means that both g(u)

4.3 Polish Schemes

107

and g(v) must be in the same A(un ) and hence d(g(u), g(v)) < 2/n < . Thus g is continuous and hence B(N0 )-measurable. Next consider f : A → N0 . Let G = c(v n ) denote the thin cylinder {u : u ∈ N, un = v n } in N0 . Then by construction f −1 (G) = g(G) = A(v n ), which is a measurable set. From Example 3.2.2, however, such thin cylinders generate B(N0 ). Thus f is measurable on a generating class and hence, by Lemma 2.4, f is measurable. We have now proved the following result. Theorem 4.2. If A is a Polish space, then the Borel space (A, B(A)) is standard. Thus the real line, the unit interval, Euclidean space, and separable Hilbert spaces and Banach spaces are standard. We shall encounter several specific examples in the sequel. The development of the theorem and some simple observations provide some subsidiary results that we now collect. We saw that a Polish space A is the kernel of a measurable scheme and that the Borel space (A, B(A)) is isomorphic to (N0 , B(N0 )), a subspace of the space of integer sequences, using the metric dZ+ . In addition, the inverse mapping in the construction is continuous. The isomorphism and its properties depended only on the fact that that the set was the kernel of a measurable scheme. Thus if any set F in a metric space is the kernel of a measurable scheme, then it is isomorphic to some (N0 , B(N0 )), N0 ⊂ N, with the metric dZ+ on N0 , and the inverse mapping is continuous. We next show that this subspace, say N0 , is closed with respect to dZ+ . If ak → a and ak ∈ N0 , then the first n coordinates of ak must match those of a for sufficiently large k and hence be contained in the thin cylinder c(an ) = {u : un = an }. But these sets are all nonempty for all n (or some ak would not be in N0 ) and hence a 6∈ N0c and a ∈ N0 . Thus N0 is closed. Since it is a closed subset of a Polish space, it is also Polish with respect to the same metric. Thus any kernel of a measurable scheme is isomorphic to a Polish space (N0 in particular) with a continuous inverse. Conversely, suppose that F is a subset of a Polish space and that (F , B(F )) is itself isomorphic to another Polish Borel space (B, B(B)) with metric m and suppose the inverse mapping is continuous. Suppose that {B(un )} forms a measurable scheme for B. If f : A → B is the isomorphism and g : B → A the continuous inverse, then A(un ) defined by f −1 (B(un )) = g(B(un )) satisfy the conditions (4.6) and (4.7) because inverse images preserve set theoretic operations. Since g is continuous, diam(B(un )) → 0 implies that also diam(A(un )) → 0. If the A(un ) are nonempty for all n, then so are the B(un ). Since B is Polish, they must have a nonempty intersection of exactly one point, say b, but then the intersection of the A(un ) will be g(b) and hence nonempty. Thus F is

108

4 Standard Borel Spaces

the kernel of a measurable scheme. We have thus proved the following lemma. Lemma 4.4. Let A be a Polish space and F a Borel set in A. Then F is the kernel of a measurable scheme if and only if there is an isomorphism with continuous inverse from (F , B(F )) to some Polish Borel space. Corollary 4.1. Let (A, B(A)) be the Borel space of a Polish space A. Let G be the class of all Borel sets F with the property that F is the kernel of a measurable scheme. Then G is closed under countable disjoint unions and countable intersections. Proof. If Fi , i = 1,2, . . . are disjoint S kernels of measurable schemes, then it is easy to say that the union i Fi will also be the kernel of a measurable scheme. The cells of the first level are all of the first level cells for all of the Fi . They can be reindexed to have the required form. The descending levels follow exactly as in the individual Fi . Since the Fi are disjoint, so are the cells, and the remaining properties follow from those of the individual sets Fi . T∞ Define the intersection F = i=1 Fi . From the lemma there are isomorphisms fi : Fi → Ni with Ni being Polish subspaces of N which have continuous inverses gi : Ni → Fi . Define the space N ∞ = ×∞ i=1 Ni and a subset H of N ∞ containing all of those (z1 , z2 , . . .), zi ∈ Ni , such that g1 (z1 ) = g2 (z2 ) = . . . and define a mapping g : H → F by g(z1 , z2 , . . .) = g1 (z1 ) (which equals gn (zn ) for all n). Since the gn are continuous, H is a closed subset of N ∞ . Since H is a closed subset of N ∞ and since N ∞ is Polish from Lemma 4.1, H is also Polish. g is continuous since the gn are, hence it is also measurable. It is also one-to-one, its inverse being f : F → H being defined by f (a) = (f1 (a), f2 (a), . . .). Since the mappings fn are measurable, so is f . Thus F is isomorphic to a Polish space and has a continuous inverse and hence is the kernel of a measurable scheme. 2 Theorem 4.3. If F is a Borel set of a Polish space A, then the Borel space (F , B(F )) is standard. Comment It should be emphasized that the set F might not itself be Polish. Proof. From the previous corollary the collection of all Borel sets that are kernels of measurable schemes contains countable disjoint unions and countable intersections. First, we know that each such set is isomorphic to a Polish Borel space by Lemma 4.4 and that each of these Polish spaces is standard by Theorem 4.2. Hence each set in this collection is

4.4 Product Measures

109

isomorphic to a standard space and so by Lemma 3.5 is itself standard. To complete the proof we need to show that our collection includes all the Borel sets. Consider the closed subsets of A. Lemma 4.2 implies that each such subset is itself a Polish space and hence, from Theorem 4.1, is the kernel of a measurable scheme. Hence our collection contains all closed subsets of A. Since it also contains all countable disjoint unions and countable intersections, by Lemma 1.2 it contains all the Borel sets. Since our collection is itself a subset of the Borel sets, the collection must be exactly the Borel σ -field. 2 This theorem gives the most general known class of standard Borel spaces. The development here departs from the traditional and elegant development of Parthasarathy [82] in a variety of ways, although the structure and use of schemes strongly resembles his use of analytic sets. The use of schemes allowed us to map a Polish space directly onto a subset of a standard space that was easily seen to be both a measurable set and standard itself. By showing that all Borel sets had such schemes, the general result was obtained. The traditional method is instead to map the original space into a subset of a known standard space that is then shown to be a Borel set and to possess a subset isomorphic to the full standard space. By showing that a set sandwiched between two isomorphic spaces is itself isomorphic, the result is proved. As a side result it is shown that all Polish spaces are actually isomorphic to N or, equivalently, to M. This development perhaps adds insight, but it requires the additional and difficult proof that the one-to-one image of a Borel set is itself a Borel set: a result that we avoided by mapping onto a subset of N formed by simply removing a countable number of thin cylinders from it. While the approach adopted here does not yield the rich harvest of related results of Parthasarathy, it is a simpler and more direct route to the desired result. Alternative but often similar developments of the properties of standard Borel spaces may be found in Schwartz [94], Christensen [12], Bourbaki [8], Cohn [15], Mackey [68], and Bjornsson [6].

4.4 Product Measures Product measures on Borel subsets of Rk provide an example of the machinery developed thus far. We could consider more general Polish spaces, but R provides a concrete example. Let (R, B(R)) be the Borel measurable space of the real line, which has been shown to a standard measurable space. Let A ∈ B(R). The measurable space (A, B(A)), where B(A) = {all F ∩ A, F ∈ B(R)}, has also been shown to be a standard measurable space. Let P 1 be a probability measure on the space. Next consider the product measurable space (Ak , B(A)k ), where B(A)k is the σ -field generated by the rectangles. Recall from Lemma 4.1(b) that

110

4 Standard Borel Spaces

since A is Polish B(A)k = B(Ak ), that is, the σ -field generated by the rectangles is the same as the Borel σ -field generated by the open sets with respect to the Euclidean metric. Since (Ak , B(A)k ) is the product of a standard measurable space, it is standard. Define a set function P k on rectangles F = ×ki=1 Fi ), Fi ∈ B(A) by Pn (×ki=1 Fi ) =

k Y

P1 (Fi ).

i=1

This set function immediately extends to the field FAn generated by the rectangles using Lemma 2.2 since every F ∈ FAn has the form SN F = n=1 Fn where the Fn = ×ki=1 Fn,i : P k (F ) =

N X n=1

P k (Fn ) =

N Y k X

P 1 (Fn,i ).

(4.20)

n=1 i=1

This set function is obviously nonnegative and it is finitely additive by construction since the union of unions of disjoint rectangles is itself a union of disjoint rectangles. Hence since the measurable space is standard, the measure on the field extends to the σ -field generated by the field and hence there is a unique probability measure P n on (Ak , B(A)k ) agreeing with its value on cylinders. This probability measure is called a product measure and is a generalization of the discretealphabet Bernoulli processes already considered. As an example, let (A, B(A), P 1 ) = ([0, 1), B([0, 1)), λ), with λ([a, b) = b − a for all 0 ≤ a < b < 1, that is, the Borel (or uncompleted Lebesgue) measure on [0, 1)). Then the product measure λk on ([0, 1)k , B([0, 1)k ) is well defined. The product measure λn extended to the product measurable space is called the Borel measure on [0, 1)k and the completion is called the Lebesgue measure on [0, 1)k . (As mentioned before, the name “Lebesgue measure” in the literature sometimes refers to the uncompleted measure.) As in the case with k = 1, essentially the same arguments can be used to derive the Borel or Lebesgue measure λk on ([a, b)k , B([a, b)k ) for any finite a, b and the Borel or Lebesgue measure on (Rk , B(Rk )).

4.5 IID Random Processes and B-processes The construction of the previous section together with the Kolmogorov extension theorem provide the continuous-alphabet analog of the discretetime IID processes. Again suppose that (R, B(R)) is the Borel measurable space of the real line and A ∈ B(R) and let µ 1 be a probability measure

4.5 IID Random Processes and B-processes

111

on the measurable space (A, B(A)). Let T be Z or Z+ and consider the the product measurable space (AT , B(A)T ). The goal is a probability measure µ on this product space of sequences which will be a process distribution for a Kolmogorov directly given random process {Xn ; n ∈ T} where Xn is the coordinate function on AT . For any finite index set I ⊂ T, define the product measure µ n on the product measurable space (AI , B(A)I ) exactly as in the previous section, that is, define the set function on rectangles, extend it to the field generated by rectangles, and then extend it to (AI , B(A)I ). This provides a distribution for any collection of n samples {Xn ; n ∈ I} from the random process {Xn ; n ∈ T}.These distributions are all consistent since rectangles of smaller dimension can be formed by simply substituting the alphabet A for one or more of the coordinate events of the higher dimensional rectangles. Thus the conditions of the Kolmogorov extension theorem are met and there exists a unique process distribution agreeing with the distributions for finite-dimensional sample vectors. A random process having identical product distributions for all possible n-dimensional sample vectors is said to be independent, identically distributed (IID), generalizing the discrete-alphabet Bernoulli processes already considered in Section 3.8. For example, ([0, 1), B([0, 1)), λ1 ) with λ1 the Borel measure on [0, 1) could be the scalar probability space used to produce an IID process with uniformly distributed outputs Xn . Alternatively, the process could be constructed using the real line as an alphabet using the scalar space (R, B(R), µ 1 ) with µ 1 (F ) = λ1 (F ∩ [0, 1)) for all F ∈ B(R). As with IID discrete-alphabet processes, these memoryless processes can be used to construct new processes by stationary mapping, either coding (with discrete outputs) or filtering (with continuous outputs). The discrete alphabet B-process definition immediately extends to the realvalued alphabet case: a B-process is defined as any random process that is equivalent to a process formed by a stationary mapping of an IID process. As an important example, suppose that {Xn ; n ∈ I} is an IID uniform process and that hk ; k = 0, . . . , K are finite real numbers. Define the time-invariant filter mapping an input sequence x into an output symbol y0 = f (x) = g(x−K , x−K+1 , . . . , x−1 , x0 ) =

K X

hk x−k .

k=0

Then the output at time n will be yn = f (T n x) = g(xn−K , xn−K+1 , . . . , xn−1 , x0 ) =

K X k=0

or in terms of the random processes,

hk xn−k ,

112

4 Standard Borel Spaces

Yn =

K X

hk Xn−k .

k=0

This is an example of a linear time-invariant (LTI) filter, sometimes called a moving average filter. We include only a finite number of terms to avoid the technicalities of convergence in the general case, but the basic idea does not change. Driving an LTI filter with an IID continuous alphabet process produces a B-process.

4.6 Standard Spaces vs. Lebesgue Spaces A natural question is how standard spaces and Lebesgue spaces are related. As has been mentioned, Lebesgue spaces are the ubiquitous model in ergodic theory, but standard spaces are emphasized in this book because they capture the key properties of standard Borel spaces for many of the results developed herein. This section highlights similarities and differences between the two types of probability space. Suppose that a probability space (Ω, B, P ) is a Lebesgue space and hence • P can be expressed as a mixture or two measures P = αP1 + (1 − α)P2 where α ∈ [0, 1], • (Ω, B, P1 ) is isomorphic (mod 0) to Lebesgue measure space on [0, 1), and • P2 is purely discrete, that is, there is an at most countable collection of points Ω2 = {ai , i = 1, 2, . . .} with P2 (Ω2 ) = 1. Isomorphism mod 0 means that there are sets Ω1 ∈ B, Λ ∈ B([0, 1) for which P1 (Ω1 ) = λ(Λ) = 1 and there is an isomorphism φ : Ω1 → Λ, that is, φ is one-to-one and both φ and its inverse φ−1 are measurable. Lebesgue spaces are in a sense more general since they are complete while in general standard measure spaces are not, in particular, their σ -fields contain more sets. But the additional sets do not change the probabilities of events with nonzero probabilities, they simply include as events all subsets of events with 0 probability. The Lebesgue measure space ([0, 1), B, λ) is the completion of the Borel measure space ([0, 1), B([0, 1)), λ), where λ of any interval is the length of the interval and the measurable space ([0, 1), B([0, 1))) is standard. Since the Lebesgue measure space of the unit interval is the completion of a particular standard Borel measure space (which is a standard measurable space), the continuous part of any Lebesgue space is isomorphic mod 0 to the completion of a standard Borel measure space. The discrete part of any Lebesgue space is countable and hence standard (and standard Borel). Hence any Lebesgue space is a mixture of a

4.6 Standard Spaces vs. Lebesgue Spaces

113

standard measure space and a measure space isomorphic mod 0 to the completion of a particular standard Borel space. In this sense Lebesgue spaces can be considered as being less general than standard measurable spaces since the are a mixture of spaces isomorphic mod 0 to the completion of a particular standard measure space, the Borel measure space of the unit interval. On the other hand, Borel spaces are not really more general in the following sense, which we state without proof (the result follows, e.g., from results of Parthasarthy [82]). The completion of a standard Borel space is a Lebesgue space. Thus Lebesgue spaces can be viewed as simply completed standard Borel measure spaces. The two spaces are intimately connected and share many properties, but in general Borel measurable spaces have the countable additivity property, while the Lebesgue measurable space does not, but the Lebesgue space is complete and the Borel measure space is not. The key fact for this book is that standard measure spaces have the countable additivity property, that is, any finitely additive set function uniquely extends to a countably additive set function, and hence it will be enough to construct candidate probability measures that are finitely additive. This property is not possessed by Lebesgue spaces in general because null sets with respect to one set function may not be null sets with respect to another, hence including all subsets of a null set with respect to one set function in the σ -field can cause problems for extending another set function which does not assign zero probability to the original null set. Standard measure spaces usually have smaller σ -fields than do Lebesgue spaces (they will be the same if the sample space is discrete), but this eases extension of candidate probability measures to true probability measures.

“This page left intentionally blank.”

Chapter 5

Averages

Abstract Probabilistic averages or expectations are developed along with time-averages and the two types of averages are seen to share many fundamental properties. The underlying connections between the two are explored in detail in the chapters on ergodic properties (Chapter 7) and ergodic theorems (Chapter 8).

5.1 Discrete Measurements We begin by focusing on measurements that take on only a finite number of possible values. This simplifies the definitions and proofs of the basic properties and provides a fundamental first step for the more general results. Let (Ω, B, m) be a probability space, e.g., the probability space (AI , BIA , m) of a directly given random process with alphabet A. A realvalued random variable f : Ω → R will also be called a measurement since it is often formed by taking a mapping or function of some other set of more general random variables, e.g., the outputs of some random process that might not have real-valued outputs. Measurements made on such processes, however, will always be assumed to be real. Suppose next we have a measurement f whose range space or alphabet f (Ω) ⊂ R of possible values is finite. Then f is called a discrete random variable or discrete measurement or digital measurement or, in the common mathematical terminology, a simple function. Such discrete measurements are of fundamental mathematical importance because many basic results are easy to prove for simple functions and, as we shall see, all real random variables can be expressed as limits of such functions. In addition, discrete measurements are good models for many important physical measurements. No human can distinguish all real numbers on a meter, nor is any meter perfectly accurate. Hence, in a sense, all physical measurements have for all practical purposes a R.M. Gray, Probability, Random Processes, and Ergodic Properties, DOI: 10.1007/978-1-4419-1090-5_5, © Springer Science + Business Media, LLC 2009

115

116

5 Averages

finite (albeit possibly quite large) number of possible readings. Finally, many measurements of interest in communications systems are inherently discrete–binary computer data and ASCII codes being ubiquitous examples. Given a discrete measurement f , suppose that its range space is f (Ω) = {bi , i = 1, . . . , N}, where the bi are distinct. Define the sets Fi = f −1 (bi ) = {x : f (x) = bi }, i = 1, . . . , N. Since f is measurable, the Fi are all members of B. Since the bi are distinct, the Fi are disjoint. Since every input point in Ω must map into some bi , the union of the Fi equals Ω. Thus the collection {Fi ; i = 1, 2, . . . , N} forms a partition of Ω. We have therefore shown that any discrete measurement f can be expressed in the form M X f (x) = bi 1Fi (x), (5.1) i=1

where bi ∈ R, the Fi ∈ B form a partition of Ω, and 1Fi is the indicator function of Fi , i = 1, . . . , M. Every simple function has a unique representation in this form except possibly for sets that differ on a set of measure 0. The expectation or ensemble average or probabilistic average or mean of a discrete measurement f : Ω → R as in (5.1) with respect to a probability measure m is defined by Em f =

M X

bi m(Fi ).

(5.2)

i=0

The following simple lemma shows that the expectation of a discrete measurement is well defined. Lemma 5.1. Given a discrete measurement (measurable simple function) f : Ω → R defined on a probability space (Ω, B, m), then expectation is well defined; that is, if f (x) = then Em f =

N X

bi 1Fi (x) =

M X

aj 1Gj (x)

i=1

j=1

N X

M X

bi m(Fi ) =

i=1

aj m(Gj ).

j=1

Proof. Given the two partitions {Fi } and {Gj }, form the new partition (called the join of the two partitions) {Fi ∩ Gj } and observe that f (x) =

M N X X i=1 j=1

cij 1Fi ∩Gj (x)

5.1 Discrete Measurements

117

where cij = ai = bj . Thus N X

bi m(Fi ) =

i=1

N X

bi (

i=1

=

M X

m(Gj ∩ Fi )) =

j=1

N X M X

M X

cij m(Gj ∩ Fi ) =

j=1

aj (

bi m(Gj ∩ Fi )

i=1 j=1

i=1 j=1

=

M N X X

M X N X

aj m(Gj ∩ Fi )

j=1 i=1 N X

m(Gj ∩ Fi )) =

i=1

M X

bj m(Gj ).

j=1

2 An immediate consequence of the definition of expectation is the simple but useful fact that for any event F in the original probability space, Em 1F = m(F );

(5.3)

that is, probabilities can be found from expectations of indicator functions. The next lemma collects the basic properties of expectations of discrete measurements. These properties will be later seen to hold for general expectations and similar ones to hold for time averages. Thus they can be viewed as fundamental properties of averages. Lemma 5.2. (a) If f ≥ 0 with probability 1, then Em f ≥ 0. (b) Em 1 = 1. (c) Expectation is linear; that is, for any real α, β and any discrete measurements f and g, Em (αf + βg) = αEm f + βEm g. (d)

If f is a discrete measurement, then |Em f | ≤ Em |f |.

(e)

Given two discrete measurements f and g for which f ≥ g with probability 1 (m({x : f (x) ≥ g(x)}) = 1), then Em f ≥ Em g.

Comments: The first three statements bear a strong resemblance to the first three axioms of probability and will be seen to follow directly from those axioms: Since probabilities are nonnegative, normalized, and additive, expectations inherit similar properties. A limiting version of additivity will be postponed. Properties (d) and (e) are useful basic integration inequalities. The following proofs are all simple, but they provide some useful practice manipulating integrals of discrete functions.

118

5 Averages

Proof. (a) If f ≥ 0 with probability 1, then bi ≥ 0 in (5.1) and Em f ≥ 0 from (5.2) and the nonnegativity of probability. (b) Since 1 = 1Ω , Em 1 = 1m(Ω) = 1 and the property follows from the normalization of probability. (c) Suppose that X X f = bi 1Fi , g = ai 1Gi . i

j

Again consider the join of the two partitions and write XX bi 1Fi ∩Gj f = i

g=

j

XX i

aj 1Fi ∩Gj .

j

Then αf + βg =

XX i

(αbi + βaj )1Fi ∩Gj

j

is a discrete measurement with the partition {Fi ∩ Gj } and hence Em (αf + βg) =

XX i

=α

(αbi + βaj )m(Fi ∩ Gj )

j

XX i

=α

bi m(Fi ∩ Gj ) + β

XX

j

X

i

bi m(Fi ) + β

i

X

aj m(Fi ∩ Gj )

j

aj m(Gj ),

j

from the additivity of probability. This proves the claim. (d) Let J denote those i for which bi in (5.1) are nonnegative. Then X bi m(Fi ) Em f = i

=

X i∈J

≤

X

|bi |m(Fi ) −

X

|bi |m(Fi )

i6∈J

|bi |m(Fi ) = Em |f |.

i

2 (e) This follows from (a) applied to f − g. In the next section we introduce the notion of quantizing arbitrary real measurements into discrete approximations in order to extend the definition of expectation to such general measurements by limiting arguments.

5.2 Quantization

119

5.2 Quantization Again let (Ω, B, m) be a probability space and f : Ω → R a measurement, that is, a real-valued random variable or measurable real-valued function. The following lemma shows that any measurement can be expressed as a limit of discrete measurements and that if the measurement is nonnegative, then the simple functions converge upward. Lemma 5.3. Given a real random variable f , define the sequence of quantizers qn : R → R, n = 1, 2, . . ., as follows:   n     −n   (k − 1)2 qn (r ) = −(k − 1)2−n     −n     .

r ≥n (k − 1)2−n ≤ r < k2−n , k = 1, 2, . . . , n2n −k2−n ≤ r < −(k − 1)2−n , k = 1, 2, . . . , n2n r < −n

The sequence of quantized measurements have the following properties: (a) f (x) = lim qn (f (x)), all x. n→∞

(b) If f (x) ≥ 0, all x, then f ≥ qn+1 (f ) ≥ qn (f ), all n; that is, qn (x) ↑ f . More generally, |f | ≥ |qn+1 (f )| ≥ |qn (f )|. (c) If f ≥ g ≥ 0, then also qn (f ) ≥ qn (g) ≥ 0 for all n. (d) The magnitude quantization error |qn (f ) − f | satisfies |qn (f ) − f | ≤ |f |

(5.4)

|qn (f ) − f | ↓ 0;

(5.5)

that is, the magnitude error is monotonically nonincreasing and converges to 0. (e) If f is G-measurable for some sub-σ -field G, than so are the fn . Thus if f is G-measurable, then there is a sequence of G-measurable simple functions which converges to f . Proof. Statements (a)-(c) are obvious from the the construction. To prove (d) observe that if x is fixed, then either f is nonnegative or it is negative. If it is nonnegative, then so is qn (f (x)) and |qn (f )−f | = f −qn (f ). This is bound above by f and decreases to 0 since qn (f ) ↑ f from part (b). If f (x) is negative, then so is qn (f ) and |qn (f ) − f | = qn (f ) − f . This is bound above by −f = |f | and decreases to 0 since qn (f ) ↓ f from part (b). To prove (e), suppose that f is G-measurable and hence that inverse images of all Borel sets under f are in G. If qn , n = 1, 2, . . . is

120

5 Averages

the quantizer sequence, then the functions qn (f ) are also G-measurable (and hence also B(R) measurable) since the inverse images satisfy qn (f )−1 (F ) = {ω : qn (f )(ω) ∈ F } = {ω : qn (f (ω)) ∈ F } −1 = {ω : f (ω) ∈ qn (F )} −1 = f −1 (qn (F )) ∈ G

2 since all inverse images under f of events F in B(R) are in G. Note further that if |f | ≤ K for some finite K, then for n > K |f (x) − qn (f (x))| ≤ 2−n .

Measurability As a prerequisite to the study of averages, we pause to consider some special properties of real-valued random variables. We will often be interested in the limiting properties of a sequence fn of integrable functions. Of particular importance is the measurability of limits of measurements and conditions under which we can interchange limits and expectations. In order to state the principal convergence results, we require first a few definitions. Given a sequence an , n = 1, 2, . . . of real numbers, define the limit supremum or upper limit or limit superior lim sup an = inf sup ak n→∞

n k≥n

and the limit infimum or lower limit or limit inferior lim inf an = sup inf ak . n→∞

n

k≥n

In other words, lim supn→∞ an = a if a is the largest number for which an has a subsequence, say an(k) , k = 1, 2, . . . converging to a. Similarly, lim infn→∞ an is the smallest limit attainable by a subsequence. When clear from context, we shall often drop the n → ∞ and simply write lim sup an or lim inf an . The limit of a sequence an , say limn→∞ an = a, will exist if and only if lim sup an = lim inf an = a, that is, if all subsequences converge to the same thing–the limit of the sequence. Lemma 5.4. Given a sequence of measurements fn , n = 1, 2, . . ., then the upper and lower limits, f = lim supn→∞ fn and f = lim infn→∞ fn , are also measurements, that is, are also measurable functions. If limn→∞ fn exists, then it is a measurement. If f and g are measurements, then so are the functions f + g, f − g, |f − g|, min(f , g), and max(f , g); that is, all these functions are measurable.

5.2 Quantization

121

Proof. To prove a real-valued function f : Ω → R measurable, it suffices from Lemma 2.4 to show that f −1 (F ) ∈ B for all F in a generating class. From Chapter 4 R is generated by the countable collection of open spheres of the form {x : |x − ri | < 1/n}; i = 1, 2, . . .; n = 1, 2, . . ., where the ri run through the rational numbers. More simply, we can generate the Borel σ -field of R from the class of all sets of the form (−∞, ri ] = {x : x ≤ ri }, with the same ri . This is because the σ -field containing these half-open intervals also contains the open intervals since it contains the half-open intervals (a, b] = (−∞, b] − (−∞, a] with rational endpoints, the rationals themselves: a=

∞ \ n=1

(a −

1 , a], n

and hence all open intervals with rational endpoints (a, b) = (a, b] − b. Since this includes the spheres described previously, this class generates the σ -field. Consider the limit supremum f . For f ≤ r we must have for any  > 0 that fn ≤ r +  for all but a finite number of n. Define Fn () = {x : fn (x) ≤ r + }. Then the collection of x for which fn (x) ≤ r +  for all but a finite number of n is ∞ \ ∞ [ F () = Fn (). n=1 k=n

This set is called the limit infimum of the Fn () in analogy to the notion for sequences. As this type of set will occasionally be useful, we pause to formalize the definition: Given a sequence of sets Fn , n = 1, 2, . . ., define the limit infimum of the sequence of sets by F = lim inf Fn = n→∞

∞ \ ∞ [

Fn () =

n=1 k=n

{x : x ∈ Fn for all but a finite number of n}.

(5.6)

Since the fn are measurable, Fn () ∈ B for all n and  and hence F () ∈ B. Since we must be in F () for all  for f ≤ r , F = {x : f (x) ≤ r } =

∞ \ n=1

F(

1 ). n

122

5 Averages

Since the F (1/n) are all measurable, so is F . This proves the measurability of f . That for f follows similarly. If the limit exists, then f = f and measurability follows from the preceding argument. The remaining results are easy if the measurements are discrete. The conclusions then follow from this fact and Lemma 5.3. 2

Exercises 1. Fill in the details of the preceding proof; that is, show that sums, differences, minima, and maxima of measurable functions are measurable. 2. Show that if f and g are measurable, then so is the product f g. Under what conditions is the division f /g measurable? 3. Prove the following corollary to the lemma: Corollary 5.1. Let fn be a sequence of measurements defined on (Ω,B,m). Define Ff as the set of x ∈ Ω for which limn→∞ fn exists; then 1F is a measurement and F ∈ B. 4. Is it true that if F is the limit infimum of a sequence Fn of events, then 1F (x) = lim inf 1Fn (x)? n→∞

5.3 Expectation We now define expectation for general measurements in two steps. If f ≥ 0, then choose as in Lemma 5.3 the sequence of asymptotically accurate quantizers qn and define Em f = lim Em (qn (f )). n→∞

(5.7)

Since the qn are discrete measurements on f , the qn (f ) are discrete measurements on Ω (qn (f )(x) = qn (f (x)) is a simple function), and hence the individual expectations are well defined. Since the qn (f ) are nondecreasing, so are the Em (qn (f )) from Lemma 5.2(e). Thus the sequence must either converge to a finite limit or grow without bound, in which case we say it converges to ∞. In both cases the expectation Em f is well defined, although it may be infinite. It is easily seen that if f is a discrete measurement, then (5.7) is consistent with the previous definition (5.1). If f is an arbitrary real random variable, define its positive and negative parts f + (x) = max(f (x), 0) and f − (x) = − min(f (x), 0) so that

5.3 Expectation

123

f (x) = f + (x) − f − (x) and set Em f = Em f + − Em f −

(5.8)

provided this does not have the form +∞ − ∞, in which case the expectation does not exist. Observe that an expectation exists for all nonnegative random variables, but that it may be infinite. The following lemma provides an alternative definition of expectation as well as a limiting theorem that shows that there is nothing magic about the particular sequence of discrete measurements used in the definition. Lemma 5.5. (a)

If f ≥ 0, then Em f =

sup Em g. discrete g:g≤f

(b) Suppose that f ≥ 0 and fn is any nondecreasing sequence of discrete measurements for which limn→∞ fn (x) = f (x), then Em f = lim Em fn . n→∞

Comments: Part (a) states that the expected value of a nonnegative measurement is the supremum of the expected values of discrete measurements that are no greater than f . This property is often used as a definition of expectation. Part (b) states that any nondecreasing sequence of asymptotically accurate discrete approximations to f can be used to evaluate the expectation. This is a special case of the monotone convergence theorem to be considered shortly. Proof. Since the qn (f ) are all discrete measurements less than or equal to f , clearly Em f = lim Em qn (f ) ≤ sup Em g, n→∞

g

where the supremum is over all discrete measurements less than or equal to f . For  > 0 choose a discrete measurement g with g ≤ f and Em g ≥ sup Em g − . g

From Lemmas 5.3 and 5.2 Em f = lim Em qn (f ) ≥ lim Em qn (g) = Em g ≥ sup Em g − . n→∞

n→∞

g

Since  is arbitrary, this completes the proof of (a). P To prove (b), use (a) to choose a discrete function g = i gi 1Gi that nearly gives Em f ; that is, fix  > 0 and choose g ≤ f so that

124

5 Averages

Em g =

X

gi m(Gi ) ≥ Em f −

i

 . 2

Define for all n the set Fn = {x : fn (x) ≥ g(x) −

 }. 2

Since g(x) ≤ f (x) and fn (x) ↑ f (x) by assumption, Fn ↑ Ω, and hence the continuity of probability from below implies that for any event H, lim m(Fn ∩ H) = m(H).

n→∞

(5.9)

Clearly 1 ≥ 1Fn and for each x ∈ Fn fn ≥ g − . Thus fn ≥ 1Fn fn ≥ 1Fn (g − ), and all three measurements are discrete. Application of Lemma 5.2(e) therefore gives Em fn ≥ Em (1Fn fn ) ≥ Em (1Fn (g − )). which, from the linearity of expectation for discrete measurements (Lemma 5.2(c)), implies that   Em fn ≥ Em (1Fn g) − m(Fn ) ≥ Em (1Fn g) − 2 2 X X   = Em (1Fn gi 1Gi ) − = Em ( gi 1Gi ∩Fn ) − 2 2 i i X  gi m(Fn ∩ Gi ) − . = 2 i Taking the limit as n → ∞, using (5.9) for each of the finite number of probabilities in the sum, lim Em fn ≥

n→∞

X i

gi m(Gi ) −

  = Em g − ≥ Em f − , 2 2

2 which completes the proof. With the preceding alternative characterizations of expectation in hand, we can now generalize the fundamental properties of Lemma 5.2 from discrete measurements to more general measurements. Lemma 5.6. (a) If f ≥ 0 with probability 1, then Em f ≥ 0. (b) Em 1 = 1. (c) Expectation is linear; that is, for any real α, β and any measurements f and g, Em (αf + βg) = αEm f + βEm g.

5.3 Expectation

(d)

125

Em f exists and is finite if and only if Em |f | is finite and |Em f | ≤ Em |f |.

(e)

Given two measurements f and g for which f ≥ g with probability 1, then Em f ≥ Em g.

Proof. (a) follows by defining qn (f ) as in Lemma 5.3 and applying Lemma 5.2(a) to conclude Em (qn (f )) ≥ 0, which in the limit proves (a). (b) follows from Lemma 5.2(b). To prove (c), first consider nonnegative functions by using Lemma 5.2(c) to conclude that Em (αqn (f ) + βqn (g)) = αEm (qn (f )) + βEm (qn (G)). The right-hand side converges upward to αEm f + βEm g. By the linearity of limits, the sequence αqn (f ) + βqn (g) converges upward to f + g, and hence, from Lemma 5.5, the left-hand side converges to Em (αf + βg), proving the result for nonnegative functions. Note that it was to prove this piece that we first considered Lemma 5.5 and hence focused on a nontrivial limiting property before proving the fundamental properties of this lemma. The result for general measurements then follows by breaking the functions into positive and negative parts and using the nonnegative measurement result. Part (d) follows from the fact that f = f + − f − and |f | = f + + f − . The expectation Em |f | exists and is finite if and only if both positive and negative parts have finite expectations, this implying that Em f exists and is also finite. Conversely, if Em f exists and is finite, then both positive and negative parts must have finite integrals, this implying that |f | must be finite. If the integrals exist, clearly Em |f | = Em f + + Em f − ≥ Em f + ≥ Em f + − Em f − = Em f . Part (e) follows immediately by application of part (a) to f − g.

2

Convex Functions and Jensen’s Inequality The linearity of expectation has a trivial, but important, special case. Suppose that X is a random variable defined on (Ω, B, m) and that f (x) = ax is a simple linear function of its argument for some scalar a. Then E(f (X)) = E(aX) = aE(X) = f (EX) so that for this very special case we can interchange expectation and the function f . Clearly in general it is not possible to perform this exchange, for example, for nontrivial distributions E(eX ) ≠ eEX — we can make the exchange only for linear functions. Unfortunately most interesting functions are, like the

126

5 Averages

exponential, not linear. There is, however, an important class of functions that generalize linear functions for which a simple and powerful bound relates the expected value of a function to the function of the expected value. We pause to describe the function and prove the inequality since the techniques of the previous lemma are key to the proof. A real-valued random variable f on a linear vector space Ω is said to be convex if f (θx + (1 − θ)y) ≤ θ(f (x) + (1 − θ)f (y); x, y ∈ Ω, θ ∈ (0, 1). (5.10) The random variable is said to be concave if −f is convex.. In some of the literature convex is referred to as convex up or convex ∪ and concave is referred to as convex down or convex ∩. Convexity can be thought of as subadditivity: as θ goes from 0 to 1, θx + (1 − θ)y traces a straight line from x to y. For each point on this line, the function evaluated at the point lies below the corresponding linear combination of the values of the weighted functions at the end points, θf (x) + (1 − θ)f (y). Many interesting functions in information theory and signal processing are convex, and there is a wide literature on the constrained optimization of such functions (see, e.g., [9]. Lemma 5.7. Given a real-valued random variable X defined on a probability space (Ω, B, m) and a convex function f , then Em [f (X)] ≤ f (Em X).

(5.11)

Proof. First consider the case where X is a discrete random variable with only two possible values xi , i = 0, 1 with probabilities p0 = θ, p1 = 1 − θ. In this case the definition of convexity reduces to (5.11). Next suppose that X is discrete with an arbitrary finite alphabet xi , i = 0, 1, . . . , K − 1. Jensen’s inequality then follows by induction from the binary case. The general case then follows by a limiting argument as in the proof of linearity of expectation. 2

Young’s Inequality We consider as an example the one special case needed later in this chapter for the derivation of other inequalities. Suppose that r > 0 and f (x) = r x . To show that f is convex it is necessary to show that θ(f (x) + (1 − θ)f (y) − f (θx + (1 − θ)y) = θr x + (1 − θ)r y − r θx r (1−θ)y

5.3 Expectation

127

is nonnegative for all real x, y and θ ∈ (0, 1). Dividing both sides by r y this is equivalent to proving that θr x−y +(1−θ)−r θ(x−y) is nonnegative for all x, y and θ ∈ (0, 1) or, equivalently, that γ(z) = θz + (1 − θ) − zθ is nonnegative for all z and θ ∈ (0, 1). This, however, follows by a simple calculus minimization, setting the derivative of γ(z) equal to zero shows a stationary point of 0 at z = 1 and the stationary point is a unique minimum since the second derivative of γ(z) is positive for θ ∈ (0, 1) and positive z. Convexity of r z implies that r θx r (1−θ)y ≤ θr x + (1 − θ)r y . If we choose α = r x , β = r y , this yields the inequality αθ β1−θ ≤ θα + (1 − θ)β.

(5.12)

Alternatively, setting a = αθ , b = β1−θ , the inequality becomes for nonnegative real a, b and θ ∈ (0, 1) 1

1

ab ≤ θa θ + (1 − θ)b 1−θ ,

(5.13)

which is known as Young’s inequality

Integration The expectation is also called an integral and is denoted by any of the following: Z Z Z Em f = f dm = f (x)dm(x) = f (x) m(dx). The subscript m denoting the measure with respect to which the expectation is taken will occasionally be omitted if it is clear from context. The development for expectations assumed probability measures, but all of the methods and results remain true in the more general case of an integral where m need not be normalized. We will continue to equate measurable functions with measurements, but we will reserve the name “random variable” for the case of probability spaces with probabilityR measures. A measurement f is said to be integrable or m-integrable if f dm exists and is finite. Define L1 (m) to be the space of all m-integrable functions. Given any m-integrable f and an event B, define

128

5 Averages

Z

Z f dm = B

f (x)1B (x)dm(x).

Two measurable functions f and g are said to be equal m-almosteverywhere or equal m-a.e. if m(f ≠ g) = m({x : f (x) ≠ g(x)}) = 0. If m is a probability measure, then we also say that f = g with mprobability one since m(f ≠ g) = m({x : f (x) = g(x)}) = 1. The m- is dropped if it is clear from context.

Sums Sums fit nicely into the integration framework. Suppose that the measurable space is Z+ and the σ -field is the collection B of all subsets of Z+ . Define the measure of points in the space by m({k}) = ak ≥ 0 and define the measure on any event by X m(F ) = ak . k∈F

P

If m(Ω) = k∈Z+ ak = 1, then m is a (discrete) probability measure. The integral of a measurement f then becomes Z X f dm = f (k)ak . k∈Z+

For example, if ak = 1 for all n, this becomes Z X f dm = f (k)

(5.14)

k∈Z+

5.4 Limits The emphasis now returns to probability measures and expectations, but unless stated otherwise the appropriate extensions to integrals follow in the same manner. The following corollary to the definition of expectation and its linearity characterizes the expectation of an arbitrary integrable random variable as the limit of the expectations of the simple measurements formed by quantizing the random variables. This is an example of an integral limit theorem, a result giving conditions under which the integral and limit can be interchanged. Corollary 5.2. Let qn be the quantizer sequence of Lemma 5.3 If f is mintegrable, then

5.4 Limits

129

Em f = lim Em qn (f ) n→∞

and lim Em |f − qn (f )| = 0.

n→∞

Proof. If f is nonnegative, the two results are equivalent and follow immediately from the definition of expectation and the linearity of Lemma 5.6(c). For general f qn (f ) = qn (f + ) − qn (f − ) and qn (f + ) ↑ f + and qn (f − ) ↑ f − by construction, and hence from the nonnegative f result and the linearity of expectation that E|f − qn | = E(f + − qn (f + )) + E(f − − qn (f − )) → 0 n→∞

and E(f − qn ) = E(f + − qn (f + )) − E(f − − qn (f − )) → 0. n→∞

2 The lemma can be used to prove that expectations have continuity properties with respect to the underlying probability measure: Corollary 5.3. If f is m-integrable, then Z lim f dm = 0; m(F )→0

F

that is, given  > 0 there is a δ > 0 such that if m(F ) < δ, then Z f dm < . F

Proof. If f is a simple function of the form (5.1),then Z F

f dm = E(1F

M X i=1

=

M X i=1

bi 1Fi ) =

M X

bi E1F ∩Fi

i=1

bi m(F ∩ Fi ) ≤

M X

bi m(F )

i=1

→ m(F )→0

0.

If f is integrable, let qn be the preceding quantizer sequence. Given  > 0 use the previous corollary to choose k large enough to ensure that Em |f − qk (f )| < . From the preceding result for simple functions

130

5 Averages

Z

Z f dm| = |

| F

Z

Z

f 1F dm −

qk (f ) 1F dm + qk (f )1F dm| Z ≤ | (f − qk (f ) 1F dm| + | qk (f ) 1F dm| Z Z ≤ |f − qk (f )|1F dm + | qk (f )dm| F Z ≤  + | qk (f )dm| → . Z

m(F )→0

F

Since  is arbitrary, the result is proved. 2 The preceding proof is typical of many results involving expectations or, more generally, integrals: the result is easy for discrete measurements and it is extended to general functions via limiting arguments. This continuity property has two immediate consequences, the proofs of which are left as exercises. Corollary 5.4. If f is nonnegative m-a.e., m-integrable, and Em f 6= 0, then the set function m defined by Z f dm m(F ) = F

is a measure and the set function P defined by R f dm P (F ) = F Em f is a probability measure. Corollary 5.5. If f is m-integrable, then Z f dm = 0. lim r →∞

f ≥r

Corollary 5.6. Suppose that f ≥ 0 m-a.e. Then f = 0 m-a.e.

R

f dm = 0 if and only if

Corollary 5.6 shows that we can consider L1 (m) to be a metric space R with respect to the metric d(f , g) = |f − g| dm if we consider measurements identical if m(f = g) = 0.

5.5 Inequalities Probability theory contains many important inequalities, results which bound probabilities and distortions in situations where evaluations may

5.5 Inequalities

131

be tedious or impossible. In this section we provide the most important and useful of these inequalities. Given a real-valued random variable with distribution m and p ≥ 1, define 1/p kf kp = (Em (f p )) . (5.15) Denote by Lp the collection of all random variables f for which kf kp is finite. If the probability measure m is not clear from context, the space is denoted by Lp (m). Lemma 5.8. Inequalities (a) The Markov Inequality Given a random variable f with f ≥ 0 m-a.e., then for a > 0 Em f . a

m(f ≥ a) ≤ Furthermore, for p > 0 m(f ≥ a) ≤

Em (f p ) . ap

(b) Tchebychev’s Inequality Given a random variable f with expected value Ef , then if a > 0, m(|f − Ef | ≥ a) ≤

Em (|f − Ef |2 ) . a2

(c) If E(f 2 ) = 0, the m(f = 0) = 1. (d) The Cauchy-Schwarz Inequality Given random variables f and g, then |E(f g)| ≤ E(f 2 )1/2 E(g 2 )1/2 . (e)

Hölder’s Inequality (A generalization of the Cauchy-Schwarz inequality.) Suppose that p, q ≥ 1 and that 1 1 + = 1. p q Then kf gk1 ≤ kf kp kgkq or

1


1

E(|f g|) ≤ (E(|f |p ) p E(|g|q ) (f)

Minkowski’s Inequality Suppose that p ≥ 1 and f , g ∈ Lp . Then

q

.

132

5 Averages

kf + gkp ≤ kf kp + kgkp , and hence k · k is a norm on Lp (if one identifies functions that are equal a.e.). Proof. (a) Let 1{f ≥a} denote the indicator function of the event {x : f (x) ≥ a}. We have that Ef = E(f (1{f ≥a} + 1{f 0, f p > ap if and only if f > a. (b) Tchebychev’s inequality follows by replacing f by |f − Ef |2 . (c) From the general Markov inequality, if E(f 2 ) = 0, then for any  > 0, m(f 2 ) ≥ 2 ) ≤

E(f 2 ) =0 2

and hence m(|f | ≤ 1/n) = 1 for all n. Since {ω : |f (ω)| ≤ 1/n} ↓ {ω : f (ω) = 0}, from the continuity of probability, m(f = 0) = 1. (d) To prove the Cauchy-Schwarz inequality set a = E(f 2 )1/2 and b = E(g 2 )1/2 . If a = 0, then f = 0 a.e. from (c) and the inequality is trivial. Hence we can assume that both a and b are nonzero. In this case 0 ≤ E(

f g E(f g) ± )2 = 2 ± 2 . a b ab

(e) Applying Young’s inequality (5.13) with θ = 1/p, 1 − θ = 1/q, a = f /kf kp , b = g/kgkq Z

f g dm ≤ kf kp kgkq

Z" θ

fp

gq

#

p + (1 − θ) q kf kp kgkq = θ+1−θ =1

dm

(f) First we show that if if f , g ∈ Lp , then also f + g ∈ Lp . Using the simple inequality max(|f |, |g|) ≤ |f | + |g| ≤ 2 max(|f |, |g|) it follows for any p > 0 that
p

|f + g|p ≤ (|f | + |g|)p ≤ 2 max(|f |, |g|)

= max(2p |f |, 2p |g|) ≤ 2p (|f |p + |g|p ).

(5.16)

Thus the integrability of the right hand side ensures that of the left. The result is immediate if kf + gkp = 0, so assume it is not. Since

5.5 Inequalities

133

|f (x) + g(x)|p = |f (x) + g(x)| × |f (x) + g(x)|p−1 ≤ |f (x)| × |f (x) + g(x)|p−1 |g(x)| × |f (x) + g(x)|p−1 So that Hölder’s inequality can be applied to each term to obtain Z p kf + gkp = |f + g|p dm ≤ kf kp kf + gk

1−1 p

+ kgkp kf + gk

1− p1

.

p−1

Dividing both sides by kf + gk p completes the proof of Minkowski’s inequality. 2 For ease of reference, we state Minkowski’s inequality for sums. It follows from the fact that sums are special cases of integrals. Corollary 5.7. Given two sequences of nonnegative numbers ai , bi and p ≥ 1,  1 1  1  p X X p p X p p  (ai + bi )p  ≤  a  +  b  (5.17) i i i

i

i

Uniform Integrability A sequence fn is said to be uniformly integrable if the limit of Corollary 5.5 holds uniformly in n; that is, if Z |fn |dm = 0 lim r →∞

|fn |≥r

uniformly in n, i.e., if given  > 0 there exists an R such that Z |fn |dm < , all r ≥ R and all n. |fn |≥r

Alternatively, the sequence fn is uniformly integrable if Z lim sup |fn |dm = 0. r →∞

n

|fn |≥r

The following lemma collects several examples of uniformly integrable sequences. Lemma 5.9. If there is an integrable function g (e.g., a constant) for which |fn | ≤ g, all n, then the fn are uniformly integrable. If a > 0 and supn E(|fn |1+a ) < ∞, then the fn are uniformly integrable. If |fn | ≤ |gn | for all n and the gn are uniformly integrable, then so are the fn . If a sequence fn2 is uniformly integrable, then so is the sequence fn .

134

5 Averages

Proof. The first statement is immediate from the definition and from Corollary 5.4. The second follows from Markov’s inequality. The third example follows directly from the definition. The fourth example follows from the definition and the Cauchy-Schwarz inequality. 2

5.6 Integrating to the Limit The following lemma collects the basic asymptotic integration results on integrating to the limit. Lemma 5.10. (a) Monotone Convergence Theorem If fn , n = 1, 2, . . ., is a sequence of measurements such that fn ↑ f and fn ≥ 0, m-a.e., then Z Z lim fn dm = f dm. (5.18) n→∞

R R In particular, if limn→∞ fn dm < ∞, then (limn→∞ fn )dm exists and is finite. (b) Extended Monotone Convergence Theorem If there is a measurement h such that Z hdm > −∞ (5.19) and if fn , n = 1, 2, . . ., is a sequence of measurements such that fn ↑ f and fn ≥ h, m-a.e., then (5.18) holds. Similarly, if there is a function h such that Z hdm < ∞

(5.20)

and if fn , n = 1, 2, . . ., is a sequence of measurements such that fn ↓ f and fn ≤ h, m-a.e., then (5.18) holds. (c) Fatou’s Lemma If fn is a sequence of measurements and fn ≥ h, where h satisfies (5.19), then Z Z (lim inf fn )dm ≤ lim inf n→∞

n→∞

fn dm.

(5.21)

If fn is a sequence of measurements and fn ≤ h, where h satisfies (5.20), then Z Z lim sup fn dm ≤ (lim sup fn )dm. (5.22) n→∞

n→∞

(d) Extended Fatou’s Lemma If fn , n = 1, 2, . . . is a sequence of uniformly integrable functions, then

5.6 Integrating to the Limit

135

Z

Z (lim inf fn )dm ≤ lim inf n→∞

n→∞

Z fn dm ≤ lim sup n→∞

Z fn dm ≤

(lim sup fn )dm. n→∞

(5.23) Thus, for example, if the limit of a sequence fn of uniformly integrable functions exists a.e., then that limit is integrable and Z Z ( lim fn )dm = lim fn dm. n→∞

(e)

n→∞

Dominated Convergence Theorem If fn , n = 1, 2, . . ., is a sequence of measurable functions that converges almost everywhere to f and if h is an integrable function for which |fn | ≤ h, then f is integrable and (5.18) holds.

Proof. (a) Part (a) follows in exactly the same manner as the proof of Lemma 5.5(b) except that the measurements fn need not be discrete and Lemma R5.6 is invoked instead of Lemma 5.2. R R (b) If hdm is infinite, then so is fn dm and f dm. Hence assume it is finite. Then fn −h ≥ 0 and fn −h ↑ f −h and the result follows from (a). The result for decreasing functions follows by a similar manipulation. (c) Define gn = infk≥n fk and g = lim inf fn . Then gn ≥ h for all n, h satisfies (5.19), and gn ↑ g. Thus from the fact that gn ≤ fn and part (b) Z Z Z Z fn dm ≥ gn dm ↑ gdm = (lim inf fn )dm, n→∞

which proves (5.21). Eq. (5.22) follows in like manner. (d) For any r > 0, Z Z Z fn dm = fn dm + fn dm. fn −r

fn ≥−r

Since the fn are uniformly integrable, given  > 0 r can be chosen large enough to ensure that the first integral on the right has magnitude less than . Thus Z Z fn dm ≥ − +

1fn ≥−r fn dm.

The integrand on the left is bound below by −r , which is integrable, and hence from part (c) Z Z lim inf fn dm ≥ − + (lim inf 1fn ≥−r fn )dm n→∞ n→∞ Z ≥ − + (lim inf fn )dm, n→∞

where the last inequality follows since 1fn ≥−r fn ≥ fn . Since  is arbitrary, this proves the limit infimum part of (5.23). The limit supremum

136

5 Averages

part follows similarly. (e) Follows immediately from (d) and Lemma 5.9.

2 Application of the monotone convergence theorem to the special case of sums using (5.14) yields the following simple corollary, which will be useful later. Corollary 5.8. Suppose that fn is a sequence of measurable functions such that fn ↑ f . Then lim

n→∞

∞ X

fn (k) =

k=1

∞ X k=1

lim fn (k) =

n→∞

∞ X

f (k),

k=1

that is, the infinite sum can be summed term-by-term or, in other words, the two limits involved in the summation and the summand can be interchanged. To complete this section we present several additional properties of integration and expectation that will be used later. The proofs are simple and provide practice with manipulating the preceding fundamental results. The first result provides a characterization of uniformly integrable functions. Lemma 5.11. A sequence of integrable functions fn is uniformly integrableR if and only if the following two conditions are satisfied: (i) The integrals |fn |dm are bounded; that is, there is some finite K that bounds all of these expectations from above. (ii) The continuity in the sense of Corollary 5.5 of the integrals of fn is uniform; that is, given  > 0 there is a δ > 0 such that if m(F ) < δ, then Z |fn |dm < , all n. F

Proof. First assume that (i) and (ii) hold. From Markov’s inequality and (i) it follows that Z K → 0 m(|fn | ≥ r ) ≤ r −1 |fn |dm ≤ r r →∞ uniformly in n. This plus uniform continuity then implies uniform integrability. Conversely, assume that the fn are uniformly integrable. Fix  > 0 and choose r so large that Z |fn |dm < /2, all n. |fn |≥r

Then Z

Z |fn |dm =

|fn |
Z |fn |dm +

|fn |≥r

|fn |dm ≤ /2 + r

5.6 Integrating to the Limit

137

uniformly in n, proving (i). Choosing F so that m(F ) < /(2r ), Z Z Z |fn |dm = |fn |dm + |fn |dm F F ∩kfn |≥r F ∩kfn |
≤ /2 + /2 = ,

2

proving (ii) since  is arbitrary.

Corollary 5.9. If a sequence of random variables fn is uniformly integrable, then so is the sequence n−1

n−1 X

fi .

i=0

Proof. If the fn are uniformly integrable, then there exists a finite K such that E(|fn |) < K, and hence E(|

n−1 n−1 1 X 1 X fi |) ≤ E|fi | < K. n i=0 n i=0

The fn are also uniformly continuous and hence given  there is a δ such that if m(F ) ≤ δ, then Z F

fi dm ≤ , all i,

and hence Z

|n−1

F

n−1 X

fi |dm ≤ n−1

i=0

n−1 XZ i=0

F

|fi |dm ≤ , all n.

Applying the previous lemma then proves the corollary. 2 The final lemma of this section collects a few final journeyman results on expectation. The proofs are left as exercises. Lemma 5.12. (a)

Given two integrable functions f and g, if Z Z f dm ≤ g dm all events F , F

F

then f ≤ g, m-a.e. If the preceding relation holds with equality, then f = g m-a.e. (b) (Change of Variables Formula) Given a measurable mapping f from a standard probability space (Ω, S, P ) to a standard measurable space (Γ , B) and a real-valued random variable g : Γ → R, let P f −1 denote the probability measure on (Γ , B) defined by P f −1 (F ) = P (f −1 F ), all

138

5 Averages

F ∈ B. Then Z g(y) dP f

(c)

−1

Z (y) =

g(f (x)) dP (x);

that is, if either integral exists so does the other and they are equal. Chain Rule Consider a probability measure P defined as in Corollary 5.4 as R f dm P (F ) = F Em f for some m-integrable f with nonzero expectation. If g is q-integrable, then EP g = Em (gf ).

Exercises 1. Prove that (5.7) yields (5.2) as a special case if f is a discrete measurement. 2. Prove Corollary 5.4. 3. Prove Corollary 5.5. 4. Prove Corollary 5.6. 5. Prove Lemma 5.12. Hint for proof of (b) and (c): first consider simple functions and then limits. 6. Show that for a nonnegative random variable X ∞ X

P r (X ≥ n) ≤ EX.

n=1

7. Verify the following relations: m(F ∆G) = Em [(1F − 1G )2 ] m(F ∩ G) = Em (1F 1G ) m(F ∪ G) = Em [max(1F , 1G )]. 8. Verify the following: m(F ∩ G) ≤

q

m(F )m(G) q q m(F ∆G) ≥ | m(F ) − m(G|2 .

5.7 Time Averages

139

9. Given disjoint events Fn , n = 1, 2, . . ., and a measurement f , prove that integrals are countably additive in the following sense: Z ∪∞ n=1 Fn

f dm =

∞ Z X n=1

f dm Fn

R 10. Prove that if ( F f dm)/m(F ) ∈ [a, b] for all events F , then f ∈ [a, b] m-a.e. 11. Show that if f , g ∈ L2 (m), then Z Z Z 1 1 |f g|dm ≤ f 2 dm + g 2 dm. 2 2 12. Show that if f ∈ L2 (m), then Z Z |f |dm ≤ ( f 2 dm)1/2 (m({|f | > r }))1/2 . |f |>r

5.7 Time Averages Given a random process {Xn } we will often wish to apply a given measurement repeatedly at successive times, that is, to form a sequence of outputs of a given measurement. For example, if the random process has a real alphabet corresponding to a voltage reading, we might wish to record the successive outputs of the meter and hence the successive random process values Xn . Similarly, we might wish to record successive values of the energy in a unit resistor: Xn2 . We might wish to estimate the autocorrelation of lag or delay k via successive values of the form Xn Xn+k . Although all such sequences are random and hence themselves random processes, we shall see that under certain circumstances the arithmetic or Cesàro means of such sequences of measurements will converge. Such averages are called time averages or sample averages and this section is devoted to their definition and to a summary of several possible forms of convergence. A useful general model for a sequence of measurements is provided by a dynamical system (AI , BIA , m, T ), that is, a probability space, (AI , BIA , m) and a transformation T : AI → AI (such as the shift), together with a real-valued measurement f : AI → R. Define the corresponding sequence of measurements f T i , i = 0, 1, 2, . . . by f T i (x) = f (T i x); that is, f T i corresponds to making the measurement f at time i. For example, assume that the dynamical system corresponds to a real-alphabet random process with distribution m and that T is the shift on AZ . If we are given a sequence x = (x0 , x1 , . . .) in AZ , where Z = {0, 1, 2, . . .}, if f (x) = Π0 (x) = x0 , the “time-zero” coordinate function, then f (T i x) = xi , the

140

5 Averages

outputs of the directly given process with distribution m. More generally, f could be a filtered or coded version of an underlying process. The key idea is that a measurement f made on a sample sequence at time i corresponds to taking f on the ith shift of the sample sequence. Given a measurement f on a dynamical system (Ω, B, m, T ), the nth order time average or sample average < f >n is defined as the arithmetic mean or Césaro mean of the measurements at times 0, 1, 2, . . . , n − 1; that is, n−1 X < f >n (x) = n−1 f (T i x), (5.24) i=0

or, more briefly, < f >n = n−1

n−1 X

f T i.

i=0

For example, in the case with f being the time-zero coordinate function, < f >n (x) = n−1

n−1 X

xi ,

i=0

the sample mean or time-average mean of the random process. If f (x) = x0 xk , then n−1 X xi xi+k , < f >n (x) = n−1 i=0

the sample autocorrelation of lag k of the process. If k = 0, this is the time average mean of the process since if the xk represent voltages, xk2 is the energy across a resistor at time k. Finite order sample averages are themselves random variables, but the following lemma shows that they share several fundamental properties of expectations. Lemma 5.13. Sample averages have the following properties for all n: (a) (b) (c)

If f T i ≥ 0 with probability 1, i = 0, 1, . . . , n − 1 then < f >n ≥ 0. < 1 >n = 1. For any real α, β and any measurements f and g, < αf + βg >n = α < f >n +β < g >n .

(d) | < f >n | ≤< |f | >n . (e)

Given two measurements f and g for which f ≥ g with probability 1, then < f >n ≥< g >n .

5.7 Time Averages

141

Proof. The properties all follow from the properties of summations. Note that for (a) we must strengthen the requirement that f is nonnegative a.e. to the requirement that all of the f T i are nonnegative a.e. Alternatively, this is a special case of Lemma 5.5 in which for a particular x we define a probability measure on measurements taking values f (T k x) with probability 1/n for k = 0, 1, . . . , n − 1. 2 We will usually be interested not in the finite order sample averages, but in the long term or asymptotic sample averages obtained as n → ∞. Since the finite order sample averages are themselves random variables, we need to develop notions of convergence of random variables (or measurable functions) in order to make such limits precise. There are, in fact, several possible approaches. In the remainder of this section we consider the notion of convergence of random variables closest to the ordinary definition of a limit of a sequence of real numbers. In the next section three other forms of convergence are considered and compared. Given a measurement f on a dynamical system (Ω, B, m, T ), let Ff denote the set of all x ∈ Ω for which the limit n−1 1 X f (T k x) n→∞ n k=0

lim

exists. For all such x we define the time average or limiting time average or sample average < f > by this limit; that is, f = lim < f >n . n→∞

Before developing the properties of < f >, we show that the set Ff is indeed measurable. This follows from the measurability of limits of measurements, but it is of interest to prove it directly. Let fn denote < f >n and define the events Fn () = {x : |fn (x) − f (x)| ≤ }

(5.25)

and define as in (5.6) F () = lim inf Fn () = n→∞

∞ \ ∞ [

Fk ()

n=1 k=n

= {x : x ∈ Fn () for all but a finite number of n}. Then the set Ff =

∞ \

1 F( ) k k=1

(5.26)

(5.27)

is measurable since it is formed by countable set theoretic operations on measurable sets. The sequence of sample averages < f >n will be said

142

5 Averages

to converge to a limit < f > almost everywhere or with probability one if m(Ff ) = 1. The following lemma shows that limiting time averages exhibit behavior similar to the expectations of Lemma 5.6. Lemma 5.14. Time averages have the following properties when they are well defined; that is, when the limits exist: (a) (b) (c)

If f T i ≥ 0 with probability 1 for i = 0, 1, . . ., then < f >≥ 0. < 1 >= 1. For any real α, β and any measurements f and g, < αf + βg >= α < f > +β < g > .

(d) | < f > | ≤< |f | > . (e)

Given two measurements f and g for which f ≥ g with probability 1, then < f >≥< g > .

Proof. The proof follows from Lemma 5.13 and the properties of limits. In particular, (a) is proved as follows: If f T i ≥ 0 a.e. for each i, then the event {f T i ≥ 0, all i} has probability one and hence so does the event {< f >n ≥ 0 all i}. (The countable intersection of a collection of sets with probability one also has probability one.) 2 Note that unlike Lemma 5.13, Lemma 5.14 cannot be viewed as being a special case of Lemma 5.6.

5.8 Convergence of Random Variables We have introduced the notion of almost everywhere convergence of a sequence of measurements. We now further develop the properties of such convergence along with three other notions of convergence. Suppose that {fn ; = 1, 2, . . .} is a sequence of measurements (random variables, measurable functions) defined on a probability space with probability measure m. Definitions: 1. The sequence {fn } is said to converge with probability one or with certainty or m-almost surely (m-a.s.or m-a.e.) to a random variable f if m({x : lim fn (x) = f (x)}) = 1; n→∞

that is, if with probability one the usual limit of the sequence fn (x) exists and equals f (x). In this case we write f = limn→∞ fn , m-a.e.

5.8 Convergence of Random Variables

143

2. The {fn } are said to converge in probability to a random variable f if for every  > 0 lim m({x : |fn (x) − f (x)| > }) = 0,

n→∞

in which case we write fn → f m

or f = m lim fn . n→∞

If for all positive real r lim m({x : fn (x) ≤ r }) = 0,

n→∞

we write fn → ∞ or m lim fn = ∞. m

n→∞

1

functions is a pseudo-normed 3. The space L (m) of all m-integrable R space with pseudo-norm kf k1 = |f |dm, and hence it is a pseudometric space with metric Z d(f , g) = |f − g|dm. We can consider the space to be a normed metric space if we identify or consider equal functions for which d(f , g) is zero. Note that d(f , g) = 0 if and only if f = g m-a.e. and hence the metric space really has as elements equivalence classes of functions rather than functions. A sequence {fn } of measurements is said to converge to a measurement f in L1 (m) if f is integrable and if limn→∞ ||fn − f ||1 = 0, in which case we write fn → f . L1 (m)

Convergence in L1 (m) has a simple but useful consequence. From Lemma 5.2(d) for any event F Z Z Z Z fn dm − f dm |fn − f |dm = (fn − f )dm ≤ F

F

F

F

≤ kfn − f k1 → 0 n→∞

and therefore Z

Z lim

n→∞

F

fn dm =

F

f dm if fn → f ∈ L1 (m).

(5.28)

R R In particular, if F is the entire space, then limn→∞ fn dm = f dm.

144

5 Averages

4. Denote by L2 (m) the space of R all square-integrable functions, that is, the space of all f such that f 2 dmR is finite. This is an inner product space with inner product (f R , g) = f gdm and hence also a normed space with norm kf k2 = ( f 2 dm)1/2 and hence also a metric space with pseudo-metric d(f , g) = (E[(f − g)2 ])1/2 (a metric that we identify functions f and g for which d(f , g) = 0). A sequence of squareintegrable functions {fn } is said to converge in L2 (m) or to converge in mean square or to converge in quadratic mean to a function f if f is square-integrable and limn→∞ ||fn − f ||2 = 0, in which case we write fn → f L2 (m)

or f = l.i.m. fn , n→∞

where l.i.m. stands for limit in the mean Although we shall focus on L1 and L2 , we note in passing that they are special cases of general Lp norms defined as follows: Let Lp denote the space of all measurements f for which |f |p is integrable for some integer p ≥ 1. Then Z 1

kf kp = ( |f |p dm) p

is a pseudo-norm on this space. It is a norm if we consider measurements to be the same if they are equal almost everywhere. We say that a sequence of measurements fn converges to f in Lp if kfn − f kp → 0 as n → ∞. As an example of such convergence, the following lemma shows that the quantizer sequence of Lemma 5.3 converges to the original measurement in Lp provided the f is in Lp . Lemma 5.15. If f ∈ Lp (m) and qn (f ), n = 1, 2, . . ., is the quantizer sequence of Lemma 5.3, then qn (f ) →p f . L

Proof. From Lemma 5.3 |qn (f ) − f | is bound above by |f | and monotonically nonincreasing and goes to 0 everywhere. Hence |qn (f )−f |p also is monotonically nonincreasing, tends to 0 everywhere, and is bound above by the integrable function |f |p . The lemma then follows from the dominated convergence theorem (Lemma 5.10(e)). 2 The following lemmas provide several properties and alternative characterizations of and some relations among the various forms of convergence. Of all of the forms of convergence, only almost everywhere convergence relates immediately to the usual notion of convergence of sequences of real numbers. Hence almost everywhere convergence inherits

5.8 Convergence of Random Variables

145

several standard limit properties of elementary real analysis. The following lemma states several of these properties without proof. (Proofs may be found, e.g., in Rudin [93].) Lemma 5.16. If limn→∞ fn = f , m-a.e. and limn→∞ gn = g, m-a.e. , then limn→∞ (fn + gn ) = f + g and limn→∞ fn gn =f g, m-a.e.. If also g 6= 0 m-a.e. then limn→∞ fn /gn = f /g, m-a.e. If h : R → R is a continuous function, then limn→∞ h(fn ) = h(f ), m-a.e. The next lemma and corollary provide useful tools for proving and interpreting almost everywhere convergence. The lemma is essentially the classic Borel-Cantelli lemma of probability theory. Lemma 5.17. Given a probability space (Ω, B, m) and a sequence of events Gn ∈ B, n = 1, 2, . . . , define the set {Gn i.o.} as the set {ω : ω ∈ Gn for an infinite number of n}; that is, ∞ [ ∞ \

{Gn i.o.} =

Gj = lim sup Gn . n→∞

n=1 j=n

Then m(Gn i.o.) = 0 if and only if lim m(

n→∞

∞ [

Gk ) = 0.

k=n

A sufficient condition for m(Gn i.o.) = 0 is that ∞ X

m(Gn ) < ∞.

(5.29)

n=1

Proof. From the continuity of probability, m(Gn i.o.) = m(

∞ [ ∞ \

Gj ) = lim m( n→∞

n=1 j=n

∞ [

Gj ),

j=n

which proves the first statement. From the union bound, m(

∞ [

j=n

Gn ) ≤

∞ X

m(Gn ),

j=n

which goes to 0 as n → ∞ if (5.29) holds. Corollary 5.10. limn→∞ fn = f m-a.e. if and only if for every  > 0 lim m(

n→∞

∞ [ k=n

{x : |fk (x) − f (x)| < }) = 0.

2

146

5 Averages

If for any  > 0 ∞ X

m(|fn | > ) < ∞,

(5.30)

n=0

then fn → 0 m-a.e. Proof. Define Fn () and F () as in (5.25)-(5.26). Applying the lemma to Gn = Fn ()c shows that the condition of the corollary is equivalent to m(Fn ()c i.o.) = 0 and hence m(F ()) = 1 and therefore ∞ \

1 F ( )) = 1. k k=1

m({x : lim fn (x) = f (x)}) = m( n→∞

Conversely, if 1 m(∩k F ( )) = 1, k then since

T

k≥1

F (1/k) ⊂ F () for all  > 0, m(F ()) = 1 and therefore [ lim m( Fk ()c ) = 0. n→∞

k≥n

Eq. (5.30) is (5.29) for Gn = {x : fn (x) > }, and hence this condition is sufficient from the lemma. 2 Note that almost everywhere convergence requires that the probability that |fk − f | >  for any k ≥ n tend to zero as n → ∞, whereas convergence in probability requires only that the probability that |fn − f | >  tend to zero as n → ∞. Lemma 5.18. : Given random variables fn , n = 1, 2, . . ., and f , then fn → f ⇒ fn → f ⇒ fn → f , L2 (m)

L1 (m)

m

lim fn = f m − a.e. ⇒ fn → f .

n→∞

m

Proof. From the Cauchy-Schwarz inequality kfn − f k1 ≤ kfn − f k2 and hence L2 (m) convergence implies L1 (m) convergence. From Markov’s inequality Em (|fn − f |) m({|fn − f | ≥ }) ≤ ,  and hence L1 (m) convergence implies convergence in m-probability. If fn → f m-a.e., then m(|fn − f | > ) ≤ m(

∞ [

k=n

and hence m − limn→∞ fn = f .

{|fk − f | > }) → 0 n→∞

2

5.8 Convergence of Random Variables

147

Corollary 5.11. If fn → f and fn → g in any of the given forms of convergence, then f = g, m-a.e. Proof. Fix  > 0 and define Fn (δ) = {x : |fn (x)−f (x)| ≤ δ} and Gn (δ) = {x : |fn (x) − g(x)| ≤ δ}, then Fn (/2) ∩ Gn (/2) ⊂ {x : |f (x) − g(x)| ≤ } and hence   m(|f − g| > ) ≤ m( Fn ( )c ∪ Gn ( )c ) 2 2   ≤ m( Fn ( )c ) + m( Gn ( )c ) → 0. n→∞ 2 2 Since this is true for all , m(|f − g| > 0) = 0, so the corollary is proved for convergence in probability. From the lemma, convergence in probability is implied by the other forms, and hence the proof is complete.

2

The following lemma shows that the metric spaces L1 (m) and L2 (m) are each Polish spaces (when we consider two measurements to be the same if they are equal m-a.e.). Lemma 5.19. The spaces Lp (m), k = 1, 2, . . . are Polish spaces (complete, separable metric spaces). Proof. The spaces are separable since from Lemma 5.15 the collection of all simple functions with rational values is dense. Let fn be a Cauchy sequence, that is, kfm − fn k →n,m→∞ 0, where k · k denotes the Lp (m) norm. Choose an increasing sequence of integers nl , l = 1, 2, . . . so that 1 kfm − fn k ≤ ( )l ; for m, n ≥ nl . 4 Let gl = fnl and define the set Gn = {x : |gn (x) − gn+1 (x)| ≥ 2−n }. From the Markov inequality m(Gn ) ≤ 2nk kgn − gn+1 kk ≤ 2−nk and therefore from Lemma 5.18 m(lim sup Gn ) = 0. n→∞

Thus with probability 1 x 6∈ lim supn Gn . If x 6∈ lim supn Gn , however, then |gn (x) − gn+1 (x)| < 2−n for large enough n. This means that for such x gn (x) is a Cauchy sequence of real numbers, which must have a limit since R is complete under the Euclidean norm. Thus gn converges on a set of probability 1 to a function g. The proof is completed by showing that fn → g in Lp . We have that

148

5 Averages

kfn − gkk =

Z

Z

|fn − g|k dm = lim inf |fn − gl |k dm l→∞ Z ≤ lim inf |fn − fnl |k dm = lim inf kfn − fnl kk l→∞

l→∞

using Fatou’s lemma and the definition of gl . The right-most term, however, can be made arbitrarily small by choosing n large, proving the required convergence. 2 Lemma 5.20. If fn → f and the fn are uniformly integrable, then f is m

integrable and fn → f in L1 (m). If fn → f and the fn2 are uniformly m

integrable, then f ∈ L2 (m) and fn → f in L2 (m). Proof. We only prove the L1 result as the other is proved in a similar manner. Fix  > 0. Since the {fn } are uniformly integrable we can choose from Lemma 5.11 a δ > 0 sufficiently small to ensure that if m(F ) < δ, then Z  |fn |dm < , all n. 4 F Since the fn converge to f in probability, we can choose N so large that if n > N, then m({|fn − f | > /4}) < δ/2 for n > N. Then for all n, m > N (using the L1 (m) norm) Z Z kfn − fm k = |f − f | ≤  |fn − fm |dm + |f − f | >  |fn − fm |dm. n

and |fm − f | ≤

n

4

or |fm − f | >

 4

4

 4

If fn and fm are each within /4 of f , then they are within /2 of each other from the triangle inequality. Also using the triangle inequality, the right-most integral is bound by Z Z |fn |dm + |fm |dm F

F

where F = {|fn − f | > /4 or |fm − f | > /4}, which, from the union bound, has probability less than δ/2 + δ/2 = δ, and hence the preceding integrals are each less than /4. Thus ||fn − fm || ≤ /2 + /4 + /4 = , and hence {fn } is a Cauchy sequence in L1 (m), and it therefore must have a limit, say g, since L1 (m) is complete. From Lemma 5.18, however, this means that also the fn converge to g in probability, and hence from Corollary 5.11 f = g, m-a.e., completing the proof. 2 The lemma provides a simple generalization of Corollary 5.2 and an alternative proof of a special case of Lemma 5.15. Corollary 5.12. If qn is the quantizer sequence of Lemma 5.3 and f is square-integrable, then qn (f ) → f in L2 .

5.8 Convergence of Random Variables

149

Proof. From Lemma 5.5, qn (f (x)) → f (x) for all x and hence also in probability. Since for all n |qn (f )| ≤ |f | by construction, |qn (f )|2 ≤ |f |2 , and hence from Lemma 5.9 the qn (f )2 are uniformly integrable and the conclusion follows from the lemma. 2 The next lemma uses uniform integrability ideas to provide a partial converse to the convergence implications of Lemma 5.18. Lemma 5.21. If a sequence of random variables fn converges in probability to a random variable f , then the following statements are equivalent (where p = 1 or 2): (a) fn →p f . L

(b) (c)

The

p fn

are uniformly integrable. lim E(|fn |p ) = E(|f |p ).

n→∞

Proof. Lemma 5.20 implies that (b) => (a). If (a) is true, then the triangle inequality implies that |kfn k − kf k| ≤ kfn − f k → 0, which implies (c). Finally, assume that (c) holds. Consider the integral Z Z Z |fn |k dm = |fn |k dm − fn dm |fn |k ≥r |f |k r Z Z n = |fn |k dm − min(|fn |k , r )dm Z +r dm. (5.31) |fn |k ≥r

R The first term on the right tends to |f |k dm by assumption. We consider the limits of the remaining two terms. Fix δ > 0 small and observe that E(| min(|fn |k , r ) − min(|f |k , r )|) ≤ δm(||fn |k − |f |k | ≤ δ) + r m(||fn |k − |f |k | > δ). Since fn → f in probability, it must also be true that |fn |k → |f |k in probability, and hence the preceding expression is bound above by δ as n → ∞. Since δ is arbitrarily small lim |E(min(|fn |k , r )) − E(min(|f |k , r ))| ≤

n→∞

lim E(| min(|fn |k , r ) − min(|f |k , r )|) = 0.

n→∞

Thus the middle term on the right of (5.31) converges to E(min(|f |k , r )). The remaining term is r m(|fn |k ≥ r ). For any  > 0

150

5 Averages

m(|fn |k ≥ r ) = m(|fn |k ≥ r and ||fn |k − |f |k | ≤ ) + m(|fn |k ≥ r and ||fn |k − |f |k | > ). The right-most term tends to 0 as n → ∞, and the remaining term is bound above by m(|f |k ≥ r − ). From the continuity of probability, this can be made arbitrarily close to m(|f |k ≥ r ) by choosing  small enough, e.g., choosing Fn = {|f |k ∈ [r − 1/n, ∞)}, then Fn decreases to the set F = {|f |k ∈ [r , ∞)} and limn→∞ m(Fn ) = m(F ). Combining all of the preceding facts Z Z k lim |fn | dm = |f |k dm. n→∞

|fn |k ≥r

|f |k ≥r

From Corollary 5.5 given  > 0 we can choose R so large that for r > R the right-hand side is less than /2 and then choose N so large that Z |fn |k dm ≤ , n > N. (5.32) |fn |k >R

Then Z lim sup

r →∞

n

|fn |k dm ≤ Z Z k lim sup |fn | dm + lim sup

|fn |k >r

r →∞ n≤N

|fn |k >r

r →∞ n>N

|fn |k >r

|fn |k dm.

(5.33)

The first term to the right of the inequality is zero since the limit of each of the finite terms in the sequence is zero from Corollary 5.5. To bound the second term, observe that for r > R each of the integrals inside the supremum is bound above by a similar integral over the larger region {|fn |k > R}. From (5.32) each of these integrals is bound above by , and hence so is the entire right-most term. Since the left-hand side of (5.33) is bound above by any  > 0, it must be zero, and hence the |fn |k are uniformly integrable. 2 Note as a consequence of the Lemma that if fn → f in Lp for p = p 1, 2, . . . then necessarily the fn are uniformly integrable.

Exercises 1. Prove Lemma 5.16. 2. Show that if fk ∈ L1 (m) for k = 1, 2, . . ., then there exists a subsequence nk , k = 1, 2, . . ., such that with probability one

5.9 Stationary Random Processes

151

lim

k→∞

fnk =0 nk

Hint: Use the Borel-Cantelli lemma. 3. Suppose that fn converges in probability to f . Show that given , δ > 0 there is an N such that if Fn,m () = {x : |fn (x) − fm (x) > }, then m(Fn , m) ≤ δ for n, m ≥ N; that is, lim m(Fn,m ()) = 0, all  > 0.

n,m→∞

In words, if fn converges in probability, then it is a Cauchy sequence in probability. 4. Show that if fn → f and gn − fn → 0 in probability, then gn → f in probability. Show that if fn → f and gn → g in L1 , then fn +gn → f +g in L1 . Show that if fn → f and gn → g in L2 and supn ||fn ||2 < ∞, then fn gn → f g in L1 .

5.9 Stationary Random Processes We have seen in the discussions of ensemble averages and time averages that uniformly integrable sequences of functions have several nice convergence properties. In particular, we can compare expectations of limits with limits of expectations and we can infer L1 (m) convergence from m-a.e. convergence from such sequences. As time averages of the form < f >n are the most important sequences of random variables for ergodic theory, a natural question is whether or not such sequences are uniformly integrable if f is integrable. Of course in general time averages of integrable functions need not be uniformly integrable, but in this section we shall see that in a special, but very important, case the response is affirmative. Let (Ω, B, m, T ) be a dynamical system, where as usual we keep in mind the example where the system corresponds to a directly given random process {Xn , n ∈ I} with alphabet A and where T is a shift. In general the expectation of the shift f T i of a measurement f can be found from the change of variables formula of Lemma 5.11(b); that is, if mT −i is the measure defined by mT −i (F ) = m(T −i F ), all events F , then i

Em f T =

Z

i

f T dm =

Z

f dmT −i = EmT −i f .

(5.34)

(5.35)

152

5 Averages

We shall find that the sequences of measurements f T n and sample averages < f >n will have special properties if the preceding expectations do not depend on i; that is, the expectations of measurements do not depend on when they are taken. From (5.35), this will be the case if the probability measures mT −n defined by mT −n (F ) = m(T −n F ) for all events F and all n are the same. This motivates the following definition. The dynamical system is said to be stationary (or T -stationary or stationary with respect to T ) or invariant (or T -invariant or invariant with respect to T ) if m(T −1 F ) = m(F ), all events F . (5.36) We shall also say simply that the measure m is stationary with respect to T if (5.36) holds. A random process {Xn } is said to be stationary (with the preceding modifiers) if its distribution m is stationary with respect to T , that is, if the corresponding dynamical system with the shift T is stationary. A stationary random process is one for which the probabilities of an event are the same regardless of when it occurs. Stationary random processes will play a central, but not exclusive, role in the ergodic theorems. Observe that (5.36) implies that m(T −n F ) = m(F ), all events F , all n = 0, 1, 2, . . .

(5.37)

The next lemma follows immediately from (5.34), (5.35), and (5.37). Lemma 5.22. If m is stationary with respect to T and f is integrable with respect to m, then Em f = Em f T n , n = 0, 1, 2, . . . As an example, suppose that f = X0 is the zero coordinate function in a Kolmogorov representation, i.e., f (x) = x0 , and hence the nth coordinate is Xn = f T n . Then for a stationary process the mean function E(Xn ) satisfies E(Xn ) = E(X0 ). Another common moment of interest is the autocorrelation function RX (n, k) = E(Xn Xk ). Application of the lemma implies that RX (k, n) = E(Xn−k X0 ). Since this depends on the two arguments only through their difference, the notation is usually simplified to RX (n − k). Functions depending on two arguments through their difference are known as Toeplitz functions. This shows that the autocorrelation of a stationary process is Toeplitz. Lemma 5.23. If f is in L1 (m) and m is stationary with respect to T , then the sequences {f T n ; n = 0, 1, 2, . . .} and {< f >n ; n = 0, 1, 2, . . .}, where < f >n = n−1

n−1 X i=0

f T i,

5.9 Stationary Random Processes

153

are uniformly integrable. If also f ∈ L2 (m), then in addition the sequence {f f T n ; n = 0, 1, 2, . . .} and its arithmetic mean sequence {wn ; n = 0, 1, . . .} defined by wn (x) = n−1

n−1 X

f (x)f (T i x)

(5.38)

i=0

are uniformly integrable. Proof. First observe that if G = {x : |f (x)| ≥ r }, then T −i G = {x : f (T i x) ≥ r } since x ∈ T −i G if and only if T i x ∈ G if and only if f (T i x) > r . Thus by the change of variables formula and the assumed stationarity Z Z i |f T |dm = |f |dm, independent of i, T −i G

and therefore Z |f T i |r

|f T i |dm =

G

Z |f |dm → 0 uniformly in i. |f |r

r →∞

This proves that the f T i are uniformly integrable. Corollary 5.9 implies that the arithmetic mean sequence is then also uniformly integrable. 2 We close this section with an important attribute of stationary random processes. We have seen that random processes can be one-sided or two sided. In the stationary case the two types of processes are essentially equivalent. Suppose that {Xn ; n ∈ Z} is a two-sided stationary random process with alphabet A and a directly given Kolmogorov representation of (AZ , AZ , mX ). Then mX implies a distribution on all finite dimensional cylinders, and the distribution is unchanged by shifting. Hence we can construct a one-sided process {Xn0 ; n ∈ Z+ } such that for any N and any N-dimensional cylinder set C ∈ BN A we have that 0 0 Pr((Xn , Xn+1 , · · · , Xn+N−1 ) ∈ C) = Pr((Xn0 , Xn+1 , · · · , Xn+N−1 ) ∈ C) for all n ≥ 0. In other words, the two distributions agree on all nonnegative time cylinders, and hence also on BN A . Thus a one-sided process can be extracted from the two-sided process by restricting the probabilities to nonnegative time indexes. Similarly, the cylinder probabilities from the one-sided process define the probabilities for all shifts of cylinders in the two-sided process, so one can embed the stationary one-sided process into a two-sided process. This trick only works for stationary processes, but we will see that most nonstationary processes of interest are related to a stationary process.

154

5 Averages

5.10 Block and Asymptotic Stationarity While stationarity is the most common assumption used to obtain ergodic theorems relating sample averages and expectations, there are generalizations of the idea which lead to similar results. The generalizations model nonstationary behavior that arises in real-world applications such as artifacts due to mapping blocks or vectors when coding or signal processing and initial conditions that die out with time. A dynamical system (Ω, B, m, T ) is said to be N-stationary for a positive integer N if it is stationary with respect to T N ; that is, m(T −N F ) = m(F ), all F ∈ B.

(5.39)

This implies that for any event F m(T −nN F ) = m(F ) and hence that m(T −n F ) is a periodic function in n with period N. An N-stationary process can be “stationarized” by forming a finite mixture of its shifts. Define the measure mN by mN (F ) =

N−1 1 X m(T −i F ) N i=0

for all events F , a definition that is abbreviated by N−1 1 X mN = mT −i . N i=0

If m is N-stationary, then mN is stationary since N−1 1 X m(T −i T −1 F ) N i=0   N−1 1 X = m(T −i F ) + m(T −N F ) N i=1

mN (T −1 F ) =

=

N−1 1 X m(T −i F ) = mN (F ) N i=0

A dynamical system with measure m is said to be asymptotically stationary if there is a stationary measure m for which lim m(T −i F ) = m(F ); all F ∈ B.

n→∞

A dynamical system with measure m is said to be asymptotically mean stationary if there is a stationary measure m for which

5.10 Block and Asymptotic Stationarity

155

n−1 1 X m(T −i F ) = m(F ); all F ∈ B. n→∞ n i=0

lim

The following lemma relates these stationarity properties to the corresponding expectations. The proof is left as an exercise. Lemma 5.24. Let (Ω, B, m, T ) be a dynamical system and that f is a measurement on the system. If m is N-stationary, then EmN f =

N−1 1 X Em f T i . N i=1

If f is a bounded measurement, that is, |f | ≤ K for some finite K with probability one, then n−1 1 X Em (f T i ). n→∞ n i=0

Em f = lim

Exercises 1. Does Lemma 5.23 hold for N-stationary processes? 2. Prove Lemma 5.24. Hint: As usual, first consider discrete measurements and then take limits. 3. Show that stationary, N-stationary, and asymptotically stationary systems are also asymptotically mean stationary.

“This page left intentionally blank.”

Chapter 6

Conditional Probability and Expectation

Abstract The relations between measurements, that is, measurable functions, and events, that is, members of a σ -field, are explored and used to extend the basic ideas of probability to probability conditioned on measurements and events. Such relations are useful for developing properties of certain special functions such as limiting sample averages arising in the study of ergodic properties of information sources. In addition, they are fundamental to the development and interpretation of conditional probability and conditional expectation, that is, probabilities and expectations when we are given partial knowledge about the outcome of an experiment. Although conditional probability and conditional expectation are both common topics in an advanced probability course, the fact that we are living in standard spaces results in additional properties not present in the most general situation. In particular, we find that the conditional probabilities and expectations have more structure that enables them to be interpreted and often treated much like ordinary unconditional probabilities. In technical terms, there are always be regular versions of conditional probability and we will be able to define conditional expectations constructively as expectations with respect to conditional probability measures.

6.1 Measurements and Events Say we have a measurable spaces (Ω, B) and (A, B(A)) and a function f : Ω → A. Recall from Chapter 2 that f is a random variable or measurable function or a measurement if f −1 (F ) ∈ B for all F ∈ B(A). Clearly, being told the outcome of the measurement f provides information about certain events and likely provides no information about others. In particular, if f assumes a value in F ∈ B(A), then we know that f −1 (F ) occurred. We might call such an event in B an f -event since it is R.M. Gray, Probability, Random Processes, and Ergodic Properties, DOI: 10.1007/978-1-4419-1090-5_6, © Springer Science + Business Media, LLC 2009

157

158

6 Conditional Probability and Expectation

specified by f taking a value in B(A). Consider the class G = f −1 (B(A)) = {all sets of the form f −1 (F ), F ∈ B(A)}. Since f is measurable, all sets in G are also in B. Furthermore, it is easy to see that G is a σ -field of subsets of Ω. Since G ⊂ B and it is a σ -field, it is referred to as a sub-σ -field. Intuitively G can be thought of as the σ - field or event space consisting of those events whose outcomes are determinable by observing the output of f . Hence we define the σ -field induced by f as σ (f ) = f −1 (B(A)). Next observe that by construction f −1 (F ) ∈ σ (f ) for all F ∈ B(A); that is, the inverse images of all output events may be found in a sub-σ field. That is, f is measurable with respect to the smaller σ -field σ (f ) as well as with respect to the larger σ -field B. As we will often have more than one σ -field for a given space, we emphasize this notion. Say we are given measurable spaces (Ω, B) and (A, B(A)) and a sub-σ -field G of B (and hence (Ω, G) is also a measurable space). Then a function f : Ω → A is said to be measurable with respect to G or G-measurable or, in more detail, G/B(A)-measurable if f −1 (F ) ∈ G for all F ∈ B(A). Since G ⊂ B a G-measurable function is clearly also B-measurable. Usually we shall refer to B-measurable functions as simply measurable functions or as measurements. We have seen that if σ (f ) is the σ -field induced by a measurement f and that f is measurable with respect to σ (f ). Observe that if G is any other σ -field with respect to which f is measurable, then G must contain all sets of the form f −1 (F ), F ∈ B(A), and hence must contain σ (f ). Thus σ (f ) is the smallest σ -field with respect to which f is measurable. Having begun with a measurement and found the smallest σ -field with respect to which it is measurable, we can next reverse direction and begin with a σ -field and study the class of measurements that are measurable with respect to that σ -field. The short term goal will be to characterize those functions that are measurable with respect to σ (f ). Given a σ -field G let M(G) denote the class of all measurements that are measurable with respect to G. We shall refer to M(G) as the measurement class induced by G. Roughly speaking, this is the class of measurements whose output events can be determined from events in G. Thus, for example, M(σ (f )) is the collection of all measurements whose output events could be determined by events in σ (f ) and hence by output events of f . We shall shortly make this intuition precise, but first a comment is in order. The basic thrust here is that if we are given information about the outcome of a measurement f , in fact we then have information about the outcomes of many other measurements, e.g., f 2 , |f |, and other functions of f . The class of functions M(σ (f )) will prove to be exactly that class containing functions whose outcome is determinable from knowledge of the outcome of the measurement f . The following lemma shows that if we are given a measurement f : Ω → A, then a real-valued function g is in M(σ (f )), that is, is mea-

6.1 Measurements and Events

159

surable with respect to σ (f ), if and only if g(ω) = h(f (ω)) for some measurement h : A → R, that is, if g depends on the underlying sample points only through the value of f . Lemma 6.1. Given a measurable space (Ω, B) and a measurement f : Ω → A, then a measurement g : Ω → R is in M(σ (f )), that is, is measurable with respect to σ (f ), if and only if there is a B(A)/B(R)-measurable function h : A → R such that g(ω) = h(f (ω)), all ω ∈ Ω. Proof. If there exists such an h, then for F ∈ B(R), then g −1 (F ) = {ω : h(f (ω)) ∈ F } = {ω : f (ω) ∈ h−1 (F )} = f −1 (h−1 (F )) ∈ B since h−1 (F ) is in B(A) from the measurability of h and hence its inverse image under f is in σ (f ) by definition. Thus any g of the given form is in M(σ (f )). Conversely, suppose that g is in M(σ (f )). Let qn : A → R be the sequence of quantizers of Lemma 5.3. Then gn defined by gn (ω) = qn (f (ω)) are σ (f )-measurable simple functions that converge to g. Thus we can write gn as gn (ω) =

M X

ri 1F (i) (ω),

i=1 −1 (ri ) = where the F (i) are disjoint, and hence since gn ∈ M(σ (f )) that gn −1 F (i) ∈ σ (f ). Since σ (f ) = f (B(A)), there are disjoint sets Q(i) ∈ B(A) such that F (i) = f −1 (Q(i)). Define the function hn : A → R by

hn (a) =

M X

ri 1Q(i) (a)

i=1

and hn (f (ω)) =

M X i=1

=

M X

ri 1Q(i) (f (ω)) =

M X

ri 1f −1 (Q(i)) (ω)

i=1

ri 1F (i) (ω) = gn (ω).

i=1

This proves the result for simple functions. By construction we have that g(ω) = limn→∞ gn (ω) = limn→∞ hn (f (ω)) where, in particular, the right-most limit exists for all ω ∈ Ω. Define the function h(a) = limn→∞ hn (a) where the limit exists and 0 otherwise. Then g(ω) = limn→∞ hn (f (ω)) = h(f (ω)). 2 Thus far we have developed the properties of σ -fields induced by random variables and of classes of functions measurable with respect to σ -fields. The idea of a σ -field induced by a single random variable is

160

6 Conditional Probability and Expectation

easily generalized to random vectors and sequences. We wish, however, to consider the more general case of a σ -field induced by a possibly uncountable class of measurements. Then we will have associated with each class of measurements a natural σ -field and with each σ -field a natural class of measurements. Toward this end, given a class of measurements M, define σ (M) as the smallest σ -field with respect to which all of the measurements in M are measurable. Since any σ -field satisfying this condition must contain all of the σ (f ) for f ∈ M and hence must contain the σ -field induced by all of these sets and since this latter collection is a σ -field, [ σ (M) = σ ( σ (f )). f ∈M

The following lemma collects some simple relations among σ -fields induced by measurements and classes of measurements induced by σ fields. Lemma 6.2. Given a class of measurements M, then M ⊂ M(σ (M)). Given a collection of events G, then G ⊂ σ (M(σ (G))). If G is also a σ -field, then G = σ (M(G)), that is, G is the smallest σ -field with respect to which all G-measurable functions are measurable. If G is a σ -field and I(G) = {all 1G , G ∈ G} is the collection of all indicator functions of events in G, then G = σ (I(G)), that is, the smallest σ -field induced by indicator functions of sets in G is the same as that induced by all functions measurable with respect to G. Proof. If f ∈ M, then it is σ (f )-measurable and hence in M(σ (M)). If G ∈ G, then its indicator function 1G is σ (G)-measurable and hence 1G ∈ M(σ (G)). Since 1G ∈ M(σ (G)), it must be measurable with respect to σ (M(σ (G))) and hence 1−1 G (1) = G ∈ σ (M(σ (G))), proving the second statement. If G is a σ -field, then since since all functions in M(G) are G-measurable and since σ (M(G)) is the smallest σ -field with respect to which all functions in M(G) are G-measurable, G must contain σ (M(G)). Since I(G) ⊂ M(G),

6.1 Measurements and Events

161

σ (I(G)) ⊂ σ (M(G)) = G. If G ∈ G, then 1G is in I(G) and hence must be measurable with respect to σ (I(G)) and hence 1−1 G (1) = G ∈ σ (I(G)) and hence G ⊂ σ (I(G)). 2 We conclude this section with a reminder of the motivation for considering classes of events and measurements. We shall often consider such classes either because a particular class is important for a particular application or because we are simply given a particular class. In both cases we may wish to study both the given events (measurements) and the related class of measurements (events). For example, given a class of measurements M, σ (M) provides a σ -field of events whose occurrence or nonoccurrence is determinable by the output events of the functions in the class. In turn, M(σ (M)) is the possibly larger class of functions whose output events are determinable from the occurrence or nonoccurrence of events in σ (M) and hence by the output events of M. Thus knowing all output events of measurements in M is effectively equivalent to knowing all output events of measurements in the more structured and possibly larger class M(σ (M)), which is in turn equivalent to knowing the occurrence or nonoccurrence of events in the σ -field σ (M). Hence when a class M is specified, we can instead consider the more structured classes σ (M) or M(σ (M)). From the previous lemma, this is as far as we can go; that is, σ (M(σ (M))) = σ (M).

(6.1)

We have seen that a function g will be in M(σ (f )) if and only if it depends on the underlying sample points through the value of f . Since we did not restrict the structure of f , it could, for example, be a random variable, vector, or sequence, that is, the conclusion is true for countable collections of measurements as well as individual measurements. If instead we have a general class M of measurements, then it is still a useful intuition to think of M(σ (M)) as being the class of all functions that depend on the underlying points only through the values of the functions in M.

Exercises 1. Which of the following relations are true and which are false? f 2 ∈ σ (f ), f ∈ σ (f 2 ) f + g ∈ σ (f , g), f ∈ σ (f + g)

162

6 Conditional Probability and Expectation

2. If f : A → Ω, g : Ω → Ω, and g(f ) : A → Ω is defined by g(f )(x) = g(f (x)), then g(f ) ∈ σ (f ). S 3. Given a class M of measurements, is f ∈M σ (f ) a σ -field? 4. Suppose that (A, B) is a measure space and B is separable with a countable generating class {Vn ; n = 1, 2 . . .}. Describe σ (1Vn ; n = 1, 2, . . .).

6.2 Restrictions of Measures The first application of the classes of measurements or events considered in the previous section is the notion of the restriction of a probability measure to a sub-σ -field. This occasionally provides a shortcut to evaluating expectations of functions that are measurable with respect to sub-σ -fields and in comparing such functions. Given a probability space (Ω, B, m) and a sub-σ -field G of B, define the restriction of m to G, mG , by mG (F ) = m(F ), F ∈ G. Thus (Ω, G, mG ) is a new probability space with a smaller event space. The following lemma shows that if f is a G-measurable real-valued random variable, then its expectation can be computed with respect to either m or mG . Lemma 6.3. Given a G-measurable real-valued measurement f ∈ L1 (m), then also f ∈ L1 (mG ) and Z Z f dm = f dmG , where mG is the restriction of m to G. Proof. If f is a simple function, the result is immediate from the definition of restriction. More generally use Lemma 5.3(e) to infer that qn (f ) is a sequence of simple G-measurable functions converging to f and combine the simple function result with Corollary 5.2 applied to both measures. 2 Corollary 6.1. Given G-measurable functions f , g ∈ L1 (m), if Z Z f dm ≤ g dm, all F ∈ G, F

F

then f ≤ g m-a.e. If the preceding holds with equality, then f = g m-a.e. Proof. From the previous lemma, the integral inequality holds with m replaced by mG , and hence from Lemma 5.12 the conclusion holds mG -

6.3 Elementary Conditional Probability

163

a.e. Thus there is a set, say G, in G with mG probability one and hence also m probability one for which the conclusion is true. 2 The usefulness of the preceding corollary is that it allows the comparison of G-measurable functions by considering only the restricted measures and the corresponding expectations.

6.3 Elementary Conditional Probability Say we have a probability space (Ω, B, m), how is the probability measure m altered if we are told that some event or collection of events occurred? For example, how is it influenced if we are given the outputs of a measurement or collection of measurements? The notion of conditional probability provides a response to this question. In fact there are two notions of conditional probability: elementary and nonelementary. Elementary conditional probabilities cover the case where we are given an event, say F , having nonzero probability: m(F ) > 0. We would like to define a conditional probability measure m(G|F ) for all events G ∈ B. Intuitively, being told an event F occurred will put zero probability on the collection of all points outside F , but it should not effect the relative probabilities of the various events inside F . In addition, the new probability measure must be renormalized so as to assign probability one to the new “certain event” F . This suggests the definition m(G|F ) = km(G ∩ F ), where k is a normalization constant chosen to ensure that m(F |F ) = km(F ∩ F ) = km(F ) = 1. Thus we define for any F such that m(F ) > 0. the conditional probability measure m(G|F ) =

m(G ∩ F ) , all G ∈ B. m(F )

We shall often abbreviate the elementary conditional probability measure m(·|F ) by mF . Given a probability space (Ω, B, m) and an event F ∈ B with m(F ) > 0, then we have a new probability space (F , B ∩ F , mF ), where B ∩ F = {all sets of the form G ∩ F , G ∈ B }. It is easy to see that we can relate expectations with respect to the conditional and unconditional measures by R Z f dm E m 1F f EmF (f ) = f dmF = F = , (6.2) m(F ) m(F ) where the existence of either side ensures that of the other. In particular, f ∈ L1 (mF ) if and only if f 1F ∈ L1 (m). Note further that if G = F c , then Em f = m(F )EmF (f ) + m(G)EmG (f ).

(6.3)

164

6 Conditional Probability and Expectation

Instead of being told that a particular event occurred, we might be told that a random variable or measurement f is discrete and takes on a specific value, say a with nonzero probability. Then the elementary definition immediately yields m(F |f = a) =

m(G ∩ f = a) . m(f = a)

If, however, the measurement is not discrete and takes on a particular value with probability zero, then the preceding elementary definition does not work. One might attempt to replace the previous definition by some limiting form, but this does not yield a useful theory in general and it is clumsy. The standard alternative approach is to replace the preceding constructive definition by a descriptive definition, that is, to define conditional probabilities by the properties that they should possess. The mathematical problem is then to prove that a function possessing the desired properties exists. In order to motivate the descriptive definition, we make several observations on the elementary case. First note that the previous conditional probability depends on the value a assumed by f , that is, it is a function of a. It will prove more useful and more general instead to consider it a function of ω that depends on ω only through f (ω), that is, to consider conditional probability as a function m(G | f )(ω) = m(G|{λ : f (λ) = f (ω)}), or, simply, m(G | f = f (ω)), the probability of an event G given that f assumes a value f (ω). Thus a conditional probability is a function of the points in the underlying sample space and hence is itself a random variable or measurement. Since it depends on ω only through f , from Lemma 5.2.1 the function m(G | f ) is measurable with respect to σ (f ), the σ -field induced by the given measurement. This leads to the first property: For any fixed G, m(G|f ) is σ (f ) − measurable.

(6.4)

Next observe that in the elementary case we can compute the probability m(G) by averaging or integrating the conditional probability m(G | f ) over all possible values of f ; that is, Z X m(G|f ) dm = m(G|f = a)m(f = a) a

=

X

m(G ∩ f = a) = m(G).

(6.5)

a

In fact we can and must say more about such averaging of conditional probabilities. Suppose that F is some event in σ (f ) and hence its occurrence or nonoccurrence is determinable by observation of the value of f , that is, from Lemma 6.1 1F (ω) = h(f (ω)) for some function h. Thus

6.3 Elementary Conditional Probability

165

if we are given the value of f (ω), being told that ω ∈ F should not add any knowledge and hence should not alter the conditional probability of an event G given f . To try to pin down this idea, assume that m(F ) > 0 and let mF denote the elementary conditional probability measure given F , that is, mF (G) = m(G ∩ F )/m(F ). Applying (6.5) to the conditional measure mF yields the formula Z mF (G|f ) dmF = mF (G), where mF (G | f ) is the conditional probability of G given the outcome of the random variable f and given that the event F occurred. But we have argued that this should be the same as m(G | f ). Making this substitution, multiplying both sides of the equation by m(F ), and using (6.2) we derive Z m(G|f ) dm = m(G ∩ F ), all F ∈ σ (f ). (6.6) F

To make this plausibility argument rigorous observe that since f is assumed discrete we can write X f (ω) = a1f −1 (a) (ω) a∈A

and hence 1F (ω) = h(f (ω)) = h(

X

a1f −1 (a) (ω))

a∈A

and therefore F=

[

f −1 (a).

a:h(a)=1

We can then write Z Z Z m(G|f ) dm = 1F m(G|f ) dm = h(f )m(G|f ) dm F

X m(G ∩ {f = a}) m({f = a}) = h(a)m(G ∩ f −1 (a)) m({f = a}) a a [ f −1 (a)) = m(G ∩ =

X

h(a)

a:h(a)=1

= m(G ∩ F ). Although (6.6) was derived for F with m(F ) > 0, the equation is trivially satisfied if m(F ) = 0 since both sides are then 0. Hence the equation is valid for all F in σ (f ) as stated. Eq. (6.6) states that not only must we be able to average m(G | f ) to get m(G), a similar relation must hold when we cut down the average to any σ (f )-measurable event.

166

6 Conditional Probability and Expectation

Equations (6.4) and (6.6) provide the properties needed for a rigorous descriptive definition of the conditional probability of an event G given a random variable or measurement f . In fact, this conditional probability is defined as any function m(G | f )(ω) satisfying these two equations. At this point, however, little effort is saved by confining interest to a single conditioning measurement, and hence we will develop the theory for more general classes of measurements. In order to do this, however, we require a basic result from measure theory—the Radon-Nikodym theorem. The next two sections develop this result.

Exercises 1. Let f and g be discrete measurements on a probability space (Ω, B, m). Suppose in particular that X g(ω) = ai 1Gi (ω). i

Define a new random variable E(g|f ) by X E(g|f )(ω) = Em(.|f (ω)) (g) = ai m(Gi |f (ω)). i

This random variable is called the conditional expectation of g given f . Prove that Em (g) = Em [E(g|f )].

6.4 Projections One of the proofs of the Radon-Nikodym theorem is based on the projection theorem of Hilbert space theory. As this proof is more intuitive than others (at least to me) and as the projection theorem is of interest in its own right, this section is devoted to developing the properties of projections and the projection theorem. Good references for the theory of Hilbert spaces and projections are Halmos [47] and Luenberger [67]. Recall from Chapter 1 that a Hilbert space is a complete inner product space as defined in Example 1.10, which is in turn a normed linear space as Example 1.8. In fact, all of the results considered in this section will be applied in the next section to a very particular Hilbert space, the space L2 (m) of all square integrable functions with respect to a probability measure m. Since this space is Polish, it is a Hilbert space.

6.4 Projections

167

Let L be a Hilbert space with inner product (f , g) and norm ||f || = (f , f )1/2 . For the case where L = L2 (m), the inner product is given by (f , g) = Em (f g). A (measurable) subset M of L will be called a subspace of L if f , g ∈ M implies that af + bg ∈ M for all scalar multipliers a, b. We will be primarily interested in closed subspaces of L. Recall from Lemma 4.2 that a closed subset of a Polish space is also Polish with the same metric, and hence M is a complete space with the same norm. A fundamentally important property of linear spaces with inner products is the parallelogram law kf + gk2 + kf − gk2 = 2kf k2 + 2kgk2 .

(6.7)

This is proved simply by expanding the left-hand side as (f , f ) + 2(f , g) + (g, g) + (f , f ) − 2(f , g) + (g, g). A simple application of this formula and the completeness of a subspace yield the following result: given a fixed element f in a Hilbert space and given a closed subspace of the Hilbert space, we can always find a unique member of the subspace that provides the stpproximation to f in the sense of minimizing the norm of the difference. Lemma 6.4. Fix an element f in a Hilbert space L and let M be a closed subspace of L. Then there is a unique element fˆ ∈ M such that kf − fˆk = inf kf − gk,

(6.8)

g∈M

where f and g are considered identical if ||f −g|| = 0. Thus, in particular, the infimum is actually a minimum. Proof. Let ∆ denote the infimum of (6.8) and choose a sequence of fn ∈ M so that ||f − fn || → ∆. From the parallelogram law kfn − fm k2 = 2kfn − f k2 + 2kfm − f k2 − 4k

f n + fm − f k2 . 2

Since M is a subspace, (fn + fm )/2 ∈ M and hence the latter norm squared is bound below by ∆2 . This means, however, that lim

n→∞,m→∞

kfn − fm k2 ≤ 2∆2 + 2∆2 − 4∆2 = 0,

and hence fn is a Cauchy sequence. Since M is complete, it must have a limit, say fˆ. From the triangle inequality

168

6 Conditional Probability and Expectation

kf − fˆk ≤ kf − fn k + kfn − fˆk → ∆, n→∞

which proves that there is an fˆ in M with the desired property. Suppose next that there are two such functions, e.g., fˆ and g. Again invoking (6.7) fˆ + g kfˆ − gk2 ≤ 2kfˆ − f k2 + 2kg − f k2 − 4k − f k2 . 2 Since (fˆ +g)/2 ∈ M, the latter norm can be no smaller than ∆, and hence kfˆ − gk2 ≤ 2∆2 + 2∆2 − 4∆2 = 0, and therefore kfˆ − gk = 0 and the two are identical. 2 Two elements f and g of L will be called orthogonal and we write f ⊥ g if (f , g) = 0. If M ⊂ L, we say that f ∈ L is orthogonal to M if f is orthogonal to every g ∈ M. The collection of all f ∈ L which are orthogonal to M is called the orthogonal complement of M. Lemma 6.5. Given the assumptions of Lemma 6.4, fˆ will yield the infimum if and only if f − fˆ ⊥ M; that is, fˆ is chosen so that the error f − fˆ is orthogonal to the subspace M. Proof. First we prove necessity. Suppose that f − fˆ is not orthogonal to M. We will show that fˆ cannot then yield the infimum of Lemma 6.4. By assumption there is a g ∈ M such that g is not orthogonal to f − fˆ. We can assume that ||g|| = 1 (otherwise just normalize g by dividing by its norm). Define a = (f − fˆ, g) 6= 0. Now set h = fˆ + ag. Clearly h ∈ M and we now show that it is a better estimate of f than fˆ is. To see this write kf − hk2 = kf − fˆ − agk2 = kf − fˆk2 + a2 kgk2 − 2a(f − fˆ, g) = kf − fˆk2 − a2 < kf − fˆk2 as claimed. Thus orthogonality is necessary for the minimum norm estimate. To see that it is sufficient, suppose that fˆ ⊥ M. Let g be any other element in M. Then kf − gk2 = kf − fˆ + fˆ − gk2 = kf − fˆk2 + kfˆ − gk2 − 2(f − fˆ, fˆ − g). Since fˆ − g ∈ M, the inner product is 0 by assumption, proving that kf − gk is strictly greater than kf − fˆk if kfˆ − gk is not identically 0. 2 Combining the two lemmas we have the following famous result. Theorem 6.1. (The Projection Theorem)

6.5 The Radon-Nikodym Theorem

169

Suppose that M is a closed subspace of a Hilbert space L and that f ∈ L. Then there is a unique fˆ ∈ M with the properties that f − fˆ ⊥ M and

kf − fˆk = inf kf − gk. g∈M

The resulting fˆ is called the projection of f onto M and will be denoted PM (f ).

Exercises 1. Show that projections satisfy the following properties: a. If f ∈ M, then PM (f ) = f . b. For any scalar a PM (af ) = aPM (f ). c. For any f , g ∈ L and scalars a, b: PM (af + bg) = aPM (f ) + bPM (g). 2. Consider the special case of L2 (m) of all square-integrable functions with respect to a probability measure m. Let P be the projection with respect to a closed subspace M. Prove the following: a. If f is a constant, say c, then P(f ) = c. b. If f ≥ 0 with probability one, then P(f ) ≥ 0. 3. Suppose that g is a discrete measurement and hence is in L2 (m). Let M be the space of all square-integrable σ (g)-measurable functions. Assume that M is complete. Given a discrete measurement f , let E(f |g) denote the conditional expectation of exercise 5.4.1. Show that E(f |g) = PM (f ), the projection of f onto M. 4. Suppose that H ⊂ M are closed subspaces of a Hilbert space L. Show that for any f ∈ L, kf − PM (f )k ≤ kf − PH (f )k.

6.5 The Radon-Nikodym Theorem This section is devoted to the statement and proof of the Radon-Nikodym theorem, one of the fundamental results of measure and integration the-

170

6 Conditional Probability and Expectation

ory and a key result in the proof of the existence of conditional probability measures. A measure m is said to be absolutely continuous with respect to another measure P on the same measurable space and we write m  P if P (F ) = 0 implies m(F ) = 0, that is, m inherits all of P ’s zero probability events. Theorem 6.2. The Radon-Nikodym Theorem Given two measures m and P on a measurable space (Ω, B) such that m  P , then there is a measurable function h : Ω → R with the properties that h ≥ 0 P -a.e. (and hence also m-a.e.) and Z h dP , all F ∈ B. (6.9) m(F ) = F

If g and h both satisfy (6.9), then g = h P -a.e. If in addition f : Ω → R is measurable, then Z Z f dm = f h dP , all F ∈ B. (6.10) F

F

The function h is called the Radon-Nikodym derivative of m with respect to P and is written h = dm/dP . Proof. First note that (6.10) follows immediately from (6.9) for simple functions. The general result (6.10) then follows, using the usual approximation techniques. Hence we need prove only (6.9), which will take some work. Begin by defining a mixture measure q = (m + P )/2, that is, q(F ) = m(F )/2 + P (F )/2 for all events F . For any f ∈ L2 (q), necessarily f ∈ L2 (m), and hence Em f is finite from the Cauchy-Schwarz inequality. As a first step in the proof we will show that there is a g ∈ L2 (q) for which Z (6.11) Em f = f gdq, all f ∈ L2 (q). Define the class of measurements M = {f : f ∈ L2 (q), Em f = 0} and observe that the properties of expectation imply that M is a closed linear subspace of L2 (q). Hence the projection theorem implies that any f ∈ L2 (q) can be written as f = fˆ + f 0 , where fˆ ∈ M and {f }0 ⊥ M. Furthermore, this representation is unique q-a.e. If all of the functions in the orthogonal complement of M were identically 0, then f = fˆ ∈ M and (6.11) would be trivially true with f 0 = 0 since Em f = 0 for f ∈ M by definition. Hence we can Rassume that there is some r ⊥ M that is not identically 0. Furthermore, r dm 6= 0 lest we have an r ∈ M such that r ⊥ M, which can only be if r is identically 0. Define for x ∈ Ω

6.5 The Radon-Nikodym Theorem

171

g(x) =

r (x)Em r , kr k2

where here kr k is the L2 (q) norm. Clearly g ∈ L2 (q) and by construction g ⊥ M since r is. We have that Z Z Z Em f Em f f gdq = (f − r )gdq + r g dq. Em r Em r The first term on the right is 0 since g ⊥ M and f − r (Em f )/(Em r ) has zero expectation with respect to m and hence is in M. Thus Z f gdq =

Em f Em r

Z r gdq =

Em f Em r kr k2 = Em f Em r kr k2

which proves (6.11). Observe that if we set f = 1F for any event F , then Z Em 1F = m(F ) = gdq, F

a formula which resembles the desired formula (6.9) except that the integral is with respect to q instead of m. From the definition of q 0 ≤ m(F ) ≤ 2q(F ) and hence Z Z gdq ≤ 2dq, all events F . 0≤ F

F

From Lemma 5.12 this implies that g ∈ [0, 2] q-a.e.. Since m  q and P  q, this statement also holds m-a.e. and P -a.e.. As a next step rewrite (6.11) in terms of P and m as Z Z g fg f (1 − ) dm = (6.12) dP , all f ∈ L2 (q). 2 2 We next argue that (6.12) holds more generally for all measurable f in the sense that each integral exists if and only if the other does, in which case they are equal. To see this, first assume that f ≥ 0 and choose the usual quantizer sequence of Lemma 5.3. Then (6.12) implies that Z Z g qn (f )g qn (f )(1 − ) dm = dP , all f ∈ L2 (q) 2 2 for all n. Since g is between 0 and 2 with both m and P probability 1, both integrands are nonnegative a.e. and converge upward together R to a finite number or diverge together. In either case, the limit is f (1 − R g/2) dm = f g/2dP and (6.12) holds. For general f apply this argument to the pieces of the usual decomposition f = f + −f − , and the conclusion follows, both integrals either existing or not existing together.

172

6 Conditional Probability and Expectation

To complete the proof, define the set B = {x : g(x) = 2}. Application of (6.12) to 1B implies that Z Z Z g g dP = 1B dP = 1B (1 − ) dm = 0, P (B) = 2 2 B and hence P (B c ) = 1. Combining this fact with (6.12) with f = (1 − g/2)−1 implies that for any event F m(F ) = m(F ∩ B) + m(F ∩ B c ) Z g 1− 2 = m(F ∩ B) + g dm F ∩B c 1 − 2 Z g 1 dP . = m(F ∩ B) + g c 2 1 − F ∩B 2

(6.13)

Define the function h by h(x) =

g(x) 1B c (x). 2 − g(x)

(6.14)

and (6.13) becomes Z m(F ) = m(F ∩ B) +

F ∩B c

hdP , all F ∈ B.

(6.15)

Note that (6.15) is true in general; that is, we have not yet used the fact that m  P . Taking this fact into account, m(B) = 0, and hence since P (B c ) = 1, Z m(F ) =

hdP

(6.16)

F

for all events F . This formula combined with (6.14) and the fact that g ∈ [0, 2] with P and m probability 1 completes the proof. 2 In fact we have proved more than just the Radon-Nikodym theorem; we have also almost proved the Lebesgue decomposition theorem. We next complete this task. Theorem 6.3. The Lebesgue Decomposition Theorem Given two probability measures m and P on a measurable space (Ω, B), there are unique probability measures ma and p and a number λ ∈ [0, 1] such that such that m = λma + (1 − λ)p, where ma  P and where p and P are singular in the sense that there is a set G ∈ B with P (G) = 1 and p(G) = 0. ma is called the part of m absolutely continuous with respect to P . Proof. If m  P , then the result is just the Radon-Nikodym theorem with ma = m and λ = 1. If m is not absolutely continuous with respect to P then several cases are possible: If m(B) in (6.16) is 0, then the result

6.6 Probability Densities

173

again follows as in (6.16). If m(B) = 1, then p = m and P are singular and the result holds with λ = 0. If m(B) ∈ (0, 1), then (6.15) yields the theorem by identifying R hdP ma (F ) = R F , Ω hdP p(F ) = mB (F ) = and

Z λ=

m(F ∩ B) , m(B)

hdP = m(B c ),

Ω

where the last equality follows from (6.15) with F = B c . 2 The Lebesgue decomposition theorem is important because it permits any probability measure to be decomposed into an absolutely continuous part and a singular part. The absolutely continuous part can always be computed by integrating a Radon-Nikodym derivative, and hence such derivatives can be thought of as probability density functions.

Exercises 1. Show that if m  P , then P (F ) = 1 implies that m(F ) = 1. Two measures m and P are said to be equivalent if m  P and P  m. How are the Radon-Nikodym derivatives dP /dm and dm/dP related?

6.6 Probability Densities The Radon-Nikodym theorem provides a recipe for constructing many common examples of random variables, vectors, and processes. Suppose that λ is the (uncompleted) Lebesgue measure (i.e., the Borel measure) on some interval in R or on R itself. Completion is not an issue here, but we will refer to the measure as Lebesgue and completing the measure has no effect save to expand the σ -field of the measurable space. Suppose that m is a probability measure on (R, B(R)) and that P  λ. Then the Radon-Nikodym theorem implies that there is a nonnegative function f = dP /dλ such that Z P (F ) =

f dλ,

(6.17)

F

an equation which is commonly written as Z P (F ) = f (x)dx. F

(6.18)

174

6 Conditional Probability and Expectation

From the axioms of probability Z f (x)dx = 1.

(6.19)

R

A nonnegative function which integrates to 1 and whose integral over an event F gives the probability of the event is called a probability density function or PDF . Distributions of continuous real-valued random variables are usually described by their PDFs. A few common examples are listed below. Uniform PDF

Given −∞ < a < b < ∞, ( 1 x ∈ [a, b) f (x) = b−a . 0 otherwise

This is simply a normalized Lebesgue measure. Exponential PDF Given λ > 0, λe−λx

x≥0

0

otherwise

( f (x) =

.

I apologize for using λ to denote a positive constant so soon after having used it to denote Lebesgue measure, but this use of λ is ubiquitous in the engineering literature. Doubly exponential or Laplacian PDF Given λ > 0, f (x) = λe−λ|x| , x ∈ R. Gaussian PDF

Given σ 2 > 0, m ∈ R, 1

2

e− 2 (x−m) , x ∈ R. f (x) = √ 2π σ 2 All of these densities can be used to construct IID random processes as in Sections 3.8 and 4.5, which considered only discrete alphabet and uniformly distributed marginals. In turn, these can be used to construct B-processes, generalizing the special cases of Sections 3.9 and 4.5. In a few examples it is possible to specify consistent multidimensional probability distributions and hence a random process using PDFs. The most important examples of such distributions are the IID distributions and the Gaussian PDF. Multidimensional Gaussian PDF An k-dimensional PDF f (x), x ∈ Rk is Gaussian if it has the form f (x) = (2π )−k/2 (det Λ)−1/2 e−(x−m)

t Λ−1 (x−m)/2

, x ∈ Rk ,

(6.20)

6.6 Probability Densities

175

where t denotes transpose, m ∈ Rn , and where Λ is a symmetric positive definite matrix, that is, at Λa > 0 for all a ∈ Rn . The above definition assumes that Λ is (strictly) positive definite and hence invertible. The definition of a Gaussian random vector (and later process) can be extended to the case where Λ is only assumed to be nonnegative definite and hence not necessarily invertible, but here we stick to the simpler case. It can be shown with some tedious calculus that m is the mean vector in the sense that if X is a random vector with a Gaussian PDF as in (6.20), then EX = m, that Λ is the covariance matrix E[(X − m)(X − m)t ] = Λ, and where det Λ is the determinant of the matrix Λ. Since the matrix Λ is positive definite, the inverse of Λ exists and hence the PDF is well defined. A random vector with a distribution specified by a Gaussian PDF is said to be a Gaussian random vector. Gaussian random process A random process {Xt ; t ∈ T} is Gaussian if there is a mean function {mt ; t ∈ T} and a covariance function {Λ(t, s); t, s ∈ T} and for every positive integer k and collection of times t0 , t1 , . . . , tk−1 the random vector (Xt0 , Xt1 , . . . , Xtn−1 ) has a Gaussian distribution with mean vector (mt0 , mt1 , . . . , mtn−1 ) and covariance matrix Λ(ti , tj ); i, j = 0, 1, . . . , n − 1}. Gaussian random vectors and processes have many nice properties. See, for example, [20, 37]. These are important enough to list (without proof) • Linear operations on Gaussian random vectors (processes) produce new Gaussian random vectors (processes). This implies that if a Gaussian IID process is the input to a time-invariant linear filter as in Section 4.5, then the output is also Gaussian. • If a stationary Gaussian random process {Xn } is such that it is uncorrelated in the sense that for all n RX (n) = E(Xn X0 ) = E(X02 )δn,0 , then the process is also IID. • A Gaussian random process is said to be nondeterministic if the logarithm of the Fourier transform of its autocorrelation function is not negative infinity. Any nondeterministic Gaussian process can be modeled as a time invariant linear filtering of an IID Gaussian process. (This is not to hard to prove for the finite order filters considered earlier, it requires significant extra machinery in the general case.) Roughly speaking, a nondeterministic Gaussian process is one for which one can never perfectly predict the future given the past. • Many real world phenomena are well modeled as Gaussian processes. This includes thermal noise and other processes formed by adding up a large number of independent random components (the central limit theorem).

176

6 Conditional Probability and Expectation

Our main purpose in introducing such processes is that they provide a nice example for process metrics and for information measures in the sequel.

6.7 Conditional Probability Assume that we are given a probability space (Ω, B, m) and class of measurements M and we wish to define a conditional probability m(G | M) for all events G. Intuitively, if we are told the output values of all of the measurements f in the class M (which might contain only a single measurement or an uncountable family), what then is the probability that G will occur? The first observation is that we have not yet specified what the given output values of the class are, only the class itself. We can easily nail down these output values by thinking of the conditional probability as a function of ω ∈ Ω since given ω all of the f (ω), f ∈ M, are fixed. Hence we will consider a conditional probability as a function of sample points of the form m(G | M)(ω) = m(G | f (ω)), all f ∈ M, that is, the probability of G given that f = f (ω) for all measurements f in the given collection M. Analogous to (6.4), since m(G | M) is to depend only on the output values of the f ∈ M, mathematically it should be in M(σ (M)), that is, it should be measurable with respect to σ (M), the σ -field induced by the conditioning class. Analogous to (6.6) the conditional probability of G given M should not be changed if we are also given that an event in σ (M) occurred and hence averaging yields Z m(G ∩ F ) = m(G|M) dm, F ∈ σ (M). F

This leads us to the following definition. Given a class of measurements M and an event G, the conditional probability m(G | M) of G given M is defined as any function such that m(G|M) is σ (M) − measurable, and Z m(G|M) dm, all F ∈ σ (M). m(G ∩ F ) =

(6.21) (6.22)

F

Clearly we have yet to show that in general such a function exists. First, however, observe that the preceding definition really depends on M only through the σ -field that it induces. Hence more generally we can define a conditional probability of an event G given a sub-σ -field G as any function m(G | G) such that m(G|G) is G − measurable, and

(6.23)

6.7 Conditional Probability

177

Z m(G ∩ F ) =

m(G|G) dm, all F ∈ G.

(6.24)

F

Intuitively, the conditional probability given a σ -field is the conditional probability given the knowledge of which events in the σ -field occurred and which did not or, equivalently, given the outcomes of the indicator functions for all events in the σ -field. That is, as stated in Lemma 6.2, if G is a σ -field and M = {all 1F ; F ∈ G}, then G = σ (M) and hence m(G | G) = m(G | M). By construction, m(G|M) = m(G|σ (M)). (6.25) The conditional probability m(G|M(σ (M))) given the induced class of measurements is given similarly by m(G|σ (M(σ (M))). From (6.1), however, the conditioning σ -field is exactly σ (M). Thus m(G|M) = m(G|σ (M)) = m(G|M(σ (M))),

(6.26)

reinforcing the intuition discussed in Section 6.1 that knowing the outcomes of functions in M is equivalent to knowing the occurrence or nonoccurrence of events in σ (M), which is in turn equivalent to knowing the outcomes of functions in M(σ (M)). It will usually be more convenient to develop general results for the more general notion of a conditional probability given a σ -field. We are now ready to demonstrate the existence and essential uniqueness of conditional probability. Theorem 6.4. Given a probability space (Ω, B, m), an event G ∈ B, and a sub-σ -field G, then there exists a version of the conditional probability m(G | G). Furthermore, any two such versions are equal m-a.e. Proof. If m(G) = 0, then (6.24) becomes Z 0= m(G|G) dm, all F ∈ G, F

and hence if m(G | G) is G-measurable, it is 0 m-a.e. from Corollary 6.1, completing the proof for this case. If m(G) 6= 0, then define the probability measure mG on (Ω, G) as the elementary conditional probability mG (F ) = m(F |G) =

m(F ∩ G) , F ∈ G. m(G)

G of m(·|G) to G is absolutely continuous with respect The restriction mG G to mG , the restriction of m to G. Thus mG  mG and hence from the Radon-Nikodym theorem there exists an essentially unique almost everywhere positive G-measurable function h such that

178

6 Conditional Probability and Expectation

Z

G mG (F ) =

F

hdmG

and hence from Lemma 6.3 G mG (F ) =

and hence

m(F ∩ G) = m(G)

Z hdm F

Z h dm, F ∈ G F

and hence m(G)h is the desired conditional probability measure m(G|G). We close this section with a useful technical property: a chain rule for Radon-Nikodym derivatives. Lemma 6.6. Suppose that three measures on a common space satisfy m  λ  P . Then P -a.e. dm dm dλ = . dP dλ dP Proof. From (6.9) and (6.10) Z Z dm dm dλ dλ = ( ) dP , m(F ) = dλ F F dλ dP which defines

dm dλ dλ dP

as a version of

dm dP .

Exercise 1. Given a probability space (Ω, B, m), suppose that G is a finite field with atoms Gi ; i = 1, 2, . . . , N. Is it true that m(F |G)(ω) =

N X

m(F |Gi )1Gi (ω)?

i=1

6.8 Regular Conditional Probability Given an event F , the elementary conditional probability m(G | F ) of G given F is itself a probability measure m(·|F ) when considered as a function of the events G. A natural question is whether or not a similar conclusion holds in the more general case. In other words, we have defined conditional probabilities for a fixed event G and a sub-σ -field G. What if we fix the sample point ω and consider the set function m(·|G)(ω); is

6.8 Regular Conditional Probability

179

it a probability measure on B? In general it may not be, but we shall see two special cases in this section for which a version of the conditional probabilities exists such that with probability one m(F |G)(ω), F ∈ B, is a probability measure. The first case considers the trivial case of a discrete sub-σ -field of an arbitrary σ -field. The second case considers the more important (and more involved) case of an arbitrary sub-σ -field of a standard space. Given a probability space (Ω, B, m) and a sub-σ - field G, then a regular conditional probability given G is a function f : B × Ω → [0, 1] such that f (F , ω); F ∈ B (6.27) is a probability measure for each ω ∈ Ω, and f (F , ω); ω ∈ Ω

(6.28)

is a version of the conditional probability of F given G. Lemma 6.7. Given a probability space (Ω, B, m) and a discrete sub-σ -field G (that is, G has a finite or countable number of members), then there exists a regular conditional probability measure given G. Proof. Let {Fi } denote a countable collection of atoms of the countable sub-σ -field G. Then we can use elementary probability to write a version of the conditional probability for each event G: m(G|G)(ω) = P (G|Fi ) =

m(G ∩ Fi ) ; ω ∈ Fi m(Fi )

for those Fi with nonzero probability. If ω is in a zero probability atom, then the conditional probability can be set to p ∗ (G) for some arbitrary probability measure p ∗ . If we now fix ω, it is easily seen that the given conditional probability measure is regular since for each atom Fi in the countable sub-σ -field the elementary conditional probability P (·|Fi ) is a probability measure. 2 Theorem 6.5. Given a probability space (Ω, B, m) and a sub-σ -field G, then if (Ω, B) is standard there exists a regular conditional probability measure given G. Proof. For each event G let m(G | G)(ω), ω ∈ Ω, be a version of the conditional probability of G given G. We have that Z m(G|G) dm = m(G ∩ F ) ≥ 0, all F ∈ G, F

and hence from Corollary 6.1 m(G|G) ≥ 0 a.e. Thus for each event G there is an event, say H(G), with m(H(G)) = 1 such that if ω ∈ H(G),

180

6 Conditional Probability and Expectation

then m(G|G)(ω) ≥ 0. In addition, H(G) ∈ G since the indicator function of H(G) is 1 on the set of ω on which a G-measurable function is nonnegative, and hence this indicator function must also be G measurable. Since (Ω, B) is standard, it has a basis {Fn } which generates B as in Chapter 2. Let F be the countable union of all of the sets in the basis. Define the set \ H0 = H(G) G∈F

and hence H0 ∈ G, m(H0 ) = 1, and if ω ∈ H0 , then m(G | G)(ω) ≥ 0 for all G ∈ F . We also have that Z Z m(Ω|G) dm = m(Ω ∩ F ) = m(F ) = 1 dm, all F ∈ G, F

F

and hence m(Ω|G) = 1 a.e. Let H1 be the set {ω : m(Ω|G)(ω) = 1} and hence H1 ∈ G and m(H1 ) = 1. Similarly, let F1 , F2 , . . . , Fn be any finite collection of disjoint sets in F . Then Z m( F

n [

Fi |G) dm = m((

i=1

n [

i=1

=

n X i=1

=

F

Hence m(

n [

i=1

Fi ∩ F )

i=1

m(Fi ∩ F ) =

Z X n

n [

Fi ) ∩ F ) = m( n Z X i=1

F

m(Fi |G) dm

m(Fi |G) dm.

i=1

Fi |G) =

n X

m(Fi |G) a.e.

i=1

Thus there is a set of probability one in G on which the preceding relation holds. Since there are a countable number of choices of finite collections of disjoint unions of elements of F , we can find a set, say H2 ∈ G, such that if ω ∈ H2 , then m(·|G)(ω) is finitely additive on F for all ω ∈ H2 and m(H2 ) = 1. Note that we could not use this argument to demonstrate countable additivity on F since there are an uncountable number of choices of countable collections of F . Although this is a barrier in the general case, it does not affect the standard space case. We now have a set H = H0 ∩H1 ∩H2 ∈ G with probability one such that for any ω in this set, m(·|G)(ω) is a normalized, nonnegative, finitely additive set function on F . We now construct the desired regular conditional probability measure. Define f (F , ω) = m(F |G)(ω) for all ω ∈ H, F ∈ F . Fix an arbitrary probability measure P and for all ω 6∈ H define f (F , ω) = P (F ), all F ∈ B. From Theorem 2.6.1 F has the countable extension property,

6.8 Regular Conditional Probability

181

and hence for each ω ∈ H we have that f (·, ω) extends to a unique probability measure on B, which we also denote by f (·, ω). We now have a function f such that f (·, ω) is a probability measure for all ω ∈ Ω, as required. We will be done if we can show that if we fix an event G, then the function we have constructed is a version of the conditional probability for G given B, that is, if we can show that it satisfies (6.23) and (6.24). First observe that for G ∈ F , f (G, ω) = m(G | G)(ω) for ω ∈ H, where H ∈ G, and f (G, ω) = P (G) for ω ∈ H c . Thus f (G, ω) is a Gmeasurable function of ω for fixed G in F . This and the extension formula (1.38)imply that more generally f (G, ω) is a G-measurable function of ω for any G ∈ B, proving (6.23). If G ∈ F , then by construction m(G | G)(ω) = f (G, ω) a.e. and hence (6.24) holds for f (G, .). To prove that it holds for more general G, first observe that (6.24) is trivially true for those F ∈ G with m(F ) = 0. If m(F ) > 0 we can divide both sides of the formula by m(F ) to obtain Z dm(ω) mF (G) = . f (G, ω) m(F ) F We know the preceding equality holds for all G in a generating field F . To prove that it holds for all events G ∈ B, first observe that the lefthand side is clearly a probability measure. We will be done if we can show that the right-hand side is also a probability measure since two probability measures agreeing on a generating field must agree on all events in the generated σ -field. The right-hand side is clearly nonnegative since f (·, ω) is a probability measure for all ω. This also implies that f (Ω, ω) = 1 for all ω, and hence the right-hand side yields 1 for G = Ω. If Gk , k = 1, 2, . . . is a countable sequence of disjoint sets with union G, then since the f (Gi , ω) are nonnegative and the f (·, ω) are probability measures, n X i=1

f (Gi , ω) ↑

∞ X

f (Gi , ω) = f (G, ω)

i=1

and hence from the monotone convergence theorem that

182

6 Conditional Probability and Expectation ∞ Z X i=1

F

f (Gi , ω)

n Z X dm(ω) dm(ω) = lim f (Gi , ω) n→∞ m(F ) m(F ) F i=1 Z X n dm(ω) = lim ( f (Gi , ω)) n→∞ F m(F ) i=1 Z n X dm(ω) = ( lim f (Gi , ω) n→∞ m(F ) F i=1 Z dm(ω) = , f (G, ω) m(F ) F

which proves the countable additivity and completes the proof of the theorem. 2 The existence of a regular conditional probability is one of the most useful aspects of probability measures on standard spaces. We close this section by describing a variation on the regular conditional probability results. This form will be the most common use of the theorem. Let X : Ω → AX and Y : Ω → AY be two random variables defined on a probability space (Ω, B, m). Let σ (Y ) = Y −1 (BAY ) denote the σ -field generated by the random variable Y and for each G ∈ B let m(G|σ (Y ))(ω) denote a version of the conditional probability of F given σ (Y ). For the moment focus on an event of the form G = X −1 (F ) for F ∈ BAX . This function must be measurable with respect to σ (Y ), and hence from Lemma 6.1 there must be a function, say g : AY → [0, 1], such that m(X −1 (F )|σ (Y ))(ω) = g(Y (ω)). Call this function g(y) = PX|Y (F |y). By changing variables we know from the properties of conditional probability that PXY (F × D) = m(X −1 (F ) ∩ Y −1 (D)) Z = m(X −1 (F )|σ (Y ))(ω) dm(ω) Y −1 (D) Z Z PX|Y (F |Y (ω)) dm(ω) = PX|Y (F |y) dPY (y), = Y −1 (D)

D

where PY = mY −1 is the induced distribution for Y . The preceding formulas simply translate the conditional probability statements from the original space and σ -fields to the more concrete example of distributions for random variables conditioned on other random variables. Corollary 6.2. Let X : Ω → AX and Y : Ω → AY be two random variables defined on a probability space (Ω, B, m). Then if any of the following conditions is met, there exists a regular conditional probability measure PX|Y (F |y) for X given Y : (a)

The alphabet AX is discrete.

6.9 Conditional Expectation

(b) (c)

183

The alphabet AY is discrete. Both of the alphabets AX and AY are standard.

Proof. Using the correspondence between conditional probabilities given sub-σ -fields and random variables, two cases are immediate: If the alphabet AY of the conditioning random variable is discrete, then the result follows from Lemma 6.7. If the two alphabets are standard, then so is the product space, and the result follows in the same manner from Theorem 6.5. If AX is discrete, then its σ -field is standard and we can mimic the proof of Theorem 6.5. Let F now denote the union of all sets in a countable basis for BAX and replace m(F |G)(ω) by PX|Y (F |y) and dm by dPY . The argument then goes through virtually unchanged. 2

6.9 Conditional Expectation In the previous section we showed that given a standard probability space (Ω, B, m) and a sub-σ -field G, there is a regular conditional probability measure given G: m(G | G)(ω), G ∈ B, ω ∈ Ω; that is, for each fixed ω m(·|G)(ω) is a probability measure, and for each fixed G, m(G | G)(ω) is a version of the conditional probability of G given G, i.e., it satisfies (6.23) and (6.24). We can use the individual probability measures to define a conditional expectation Z (6.29) E(f |G)(ω) = f (u) dm(u|G)(ω) if the integral exists. In this section we collect a few properties of conditional expectation and relate the preceding constructive definition to the more common and more general descriptive one. Fix an event F ∈ G and let f be the indicator function of an event G ∈ B. Integrating (6.29) over F using (6.24) and the fact that E(1G | G)(ω) = m(G | G)(ω) yields Z Z E(f |G) dm = f dm (6.30) F

F

since the right-hand side is simply m(G ∩F ). From the linearity of expectation (by (6.29) a conditional expectation is simply an ordinary expectation with respect to a measure that is a regular conditional probability measure for a particular sample point ) and integration, (6.30) also holds for simple functions. For a nonnegative measurable f , take the usual quantizer sequence qn (f ) ↑ f of Lemma 5.3, and (6.30) then holds since the two sides are equal for each n and since both sides converge up to the appropriate integral, the right-hand side by the definition of the integral of a nonnegative function and the left-hand side by virtue of the

184

6 Conditional Probability and Expectation

definition and the monotone convergence theorem. For general f , use the usual decomposition f = f + − f − and apply the result to each piece. The preceding relation then holds in the sense that if either integral exists, then so does the other and they are equal. Note in particular that this implies that if f is in L1 (m), then E(f |G)(ω) must be finite m-a.e. In a similar sequence of steps, the fact that for fixed G the conditional probability of G given G is a G-measurable function implies that E(f |G)(ω) is a G-measurable function if f is an indicator function, which implies that it is also G-measurable for simple functions and hence also for limits of simple functions. Thus we have the following properties: If f ∈ L1 (m) and h(ω) = E(f |G)(ω), then h(ω) is G − measurable, (6.31) and

Z

Z h dm =

f dm, all F ∈ G.

F

(6.32)

F

Eq. (6.32) is often referred to as iterated expectation or nested expectation since it shows that the expectation of f can be found in two steps: first find the conditional expectation given a σ -field or class of measurements (in which case G is the induced σ -field) and then integrate the conditional expectation. These properties parallel the defining descriptive properties of conditional probability and reduce to those properties in the case of indicator functions. The following simple lemma shows that the properties provide an alternative definition of conditional expectation that is valid more generally, that is, holds even when the alphabets are not standard. Toward this end we now give the general descriptive definition of conditional expectation: Given a probability space (Ω, B, m), a sub-σ -field G, and a measurement f ∈ L1 (m), then any function h is said to be a version of the conditional expectation E(f |G). From Corollary 6.1, any two versions must be equal a.e. If the underlying space is standard, then the descriptive definition therefore is consistent with the constructive definition. It only remains to show that the conditional expectation exists in the general case, that is, that we can always find a version of the conditional expectation even if the underlyingPspace is not standard. To do this observe that if f is a simple function i ai 1Gi , then h=

X

ai m(Gi |G)

i

is G-measurable and satisfies (6.32) from the linearity of integration and the properties of conditional expectation:

6.9 Conditional Expectation

Z F

185

X X ( ai m(Gi |G)) dm = ai i

Z

i

Z (

= F

X

F

m(Gi |G) =

X i

Z ai

F

1Gi dm

Z ai 1Gi ) dm =

i

f dm. F

We then proceed in the usual way. If f ≥ 0, let qn be the asymptotically accurate quantizer sequence of Lemma 5.3. Then qn (f ) ↑ f and hence E(qn (f )|G) is nondecreasing (Lemma 6.3), and h = limn→∞ E(qn (f )|G) satisfies (6.31) since limits of G-measurable functions are measurable. The dominated convergence theorem implies that h also satisfies (6.32). The result for general integrable f follows from the decomposition f = f + − f − . We have therefore proved the following result. Lemma 6.8. Given a probability space (Ω, B, m), a sub-σ - field G, and a measurement f ∈ L1 (m), then there exists a G-measurable real-valued function h satisfying the formula Z Z h dm = f dm, all F ∈ G; (6.33) F

F

that is, there exists a version of the conditional expectation E(f |G) of f given G. If the underlying space is standard, then also (6.29) holds a.e.; that is, the constructive and descriptive definitions are equivalent on standard spaces. The next result shows that the remark preceding (6.30) holds even if the space is not standard. Corollary 6.3. Given a probability space (Ω, B, m) (not necessarily standard), a sub-σ -field G, and an event G ∈ G, then with probability one m(G|G) = E(1G |G). Proof. From the descriptive definition of conditional expectation Z Z E(1G |G) dm = 1G dm = m(G ∩ F ), for allF ∈ G, F

F

R for any G ∈ B. But, by (6.24), this is just F m(G|G)dm for all F ∈ G. Since both E(1G |G) and m(G|G) are G-measurable, Corollary 6.1 completes the proof. 2 Corollary 6.4. If f ∈ L1 (m) is G-measurable, then E(f |G) = f m-a.e.. Proof. The proof follows immediately from (6.32) and Corollary 6.1. 2 If a conditional expectation is defined on a standard space using the constructive definitions, then it inherits the properties of an ordinary expectation; e.g., Lemma 5.6 can be immediately extended to conditional

186

6 Conditional Probability and Expectation

expectations. The following lemma shows that Lemma 5.6 extends to conditional expectation in the more general case of the descriptive definition. Lemma 6.9. Let (Ω, B, m) be a probability space, G a sub-σ -field, and f and g integrable measurements. (a) If f ≥ 0 with probability 1, then Em (f |G) ≥ 0. (b) Em (1|G) = 1. (c) Conditional expectation is linear; that is, for any real α, β and any measurements f and g, Em (αf + βg|G) = αEm (f |G) + βEm (g|G). (d)

Em (f |G) exists and is finite if and only if Em (|f ||G) is finite and |Em f | ≤ Em |f |.

(e)

Given two measurements f and g for which f ≥ g with probability 1, then Em (f |G) ≥ Em (g|G).

Proof. (a) If f ≥ 0 and Z Z E(f |G) dm = F f dm, all F ∈ G, F

then Lemma 5.6 implies that the right-hand side is nonnegative for all F ∈ G. Since E(f |G) is G-measurable, Corollary 6.1 then implies that E(f |G) ≥ 0. (b) For f = 1, the function h = 1 is both G-measurable and satisfies Z Z hdm = f dm = m(F ), all F ∈ G. F

F

Since (6.31)-(6.32) are satisfied with h = 1, 1 must be a version of E(1|G). (c) Let E(f |G) and E(g|G) be versions of the conditional expectations of f and g given G. Then h = αE(f |G) + βE(g|G) is G-measurable and satisfies (6.32) with f replaced by αf +βg. Hence h is a version of E(αf + βg|G). (d) Let f = f + − f − be the usual decomposition into positive and negative parts f + ≥ 0, f − ≥ 0. From part (c), E(f |G) = E(f + |G) − E(f − |G). From part (a), E(f + |G) ≥ 0 and E(f − |G) ≥ 0. Thus again using part (c) E(f |G) ≤ E(f + |G) + E(f − |G) = E(f + + f − |G) = E(|f kG). (e) The proof follows from parts (a) and (c) by replacing f by f − g.

2

6.9 Conditional Expectation

187

The following result shows that if a measurement f is square-integrable, then its conditional expectation E(f |G) has a special interpretation. It is the projection of the measurement onto the space of all squareintegrable G-measurable functions. Lemma 6.10. Given a probability space (Ω, B, m), a measurement f ∈ L2 (m), and a sub-σ -field G ⊂ B. Let M denote the subset of L2 (m) consisting of all square-integrable G-measurable functions. Then M is a closed subspace of L2 (m) and E(f |G) = PM (f ). Proof. Since sums and limits of G-measurable functions are G-measurable, M is a closed subspace of L2 (m). From the projection theorem (Theorem 6.1) there exists a function fˆ = PM (f ) with the properties that fˆ ∈ M and f − fˆ ⊥ M. This fact implies that Z Z f g dm = fˆgdm (6.34) for all g ∈ M. Choosing g = 1G for any G ∈ G yields Z Z Gf dm = fˆ dm, all G ∈ G. Since fˆ is F -measurable, this defines fˆ as a version of E(f |G) and proves the lemma. 2 Eq. (6.34) and the lemma immediately yield a generalization of the nesting property of conditional expectation. Corollary 6.5. If g is G-measurable and f , g ∈ L2 (m), then E(f g) = E(gE(f |G)). The next result provides some continuity properties of conditional expectation. Lemma 6.11. For i = 1, 2, if f ∈ Li (m), then kE(f |G)ki ≤ kf ki , i = 1, 2, and thus if f ∈ Li (m), then also E(f |G) ∈ L1 (m). If f , g ∈ Li (m), then kE(f |G) − E(g|G)ki ≤ kf − gki . Thus for i = 1, 2, if fn → f in Li (m), then also E(fn |G) → E(f |G). Proof. The first inequality implies the second by replacing f with f − g and using linearity. For i = 1 from the usual decomposition f = f + − f − , f + ≥ 0, f − ≥ 0, the linearity of conditional expectation, and Lemma 6.9(c) that

188

6 Conditional Probability and Expectation

Z kE(f |G)ki = Z =

Z |E(f |G)|dm =

|E(f + − f − |G)|dm

|E(f + |G) − E(f − |G)| dm.

From Lemma 6.9(a), however, E(f + |G) ≥ 0 and E(f − |G) ≥ 0 a.e., and hence the right-hand side is bound above by Z Z (E(f + |G) + E(f − |G)) dm = E(f + + f − |G) dm Z = E(|f ||G) dm = E(|f |) = kf k1 . For i = 2 observe that 0 ≤ E[(f − E(f |G))2 ] = E(f 2 ) + E[E(f |G)2 ] − 2E[f E(f |G)]. Apply iterated expectation and Corollary 6.5 to write E[f E(f |G)] = E(E[f E(f |G)]|G) = E(E(f |G)E(f |G)) = E[E(f |G)2 ]. Combining these two equations yields E(f 2 ) − E[E(f |G)2 ] ≥ 0 or kf k22 ≥ kE(f |G)k22 . 2 A special case where the conditional results simplify occurs when the measurement and the conditioning are independent. Given a probability space (Ω, B, m), two events F and G are said to be independent if m(F ∩ G) = m(F )m(G). Two measurements or random variables f : Ω → Af and g : Ω → Ag are said to be independent if the events f −1 (F ) and g −1 (G) are independent for all events F and G; that is, m(f ∈ F and g ∈ G) = m(f −1 (F ) ∩ g −1 (G)) = m(f −1 (F ))m(g −1 (G)) = m(f ∈ F )m(g ∈ G); all F ∈ BAf ; G ∈ BAg . The following intuitive result for independent measurements is proved easily by first treating simple functions and then applying the usual methods to get the general result. Corollary 6.6. If f and g are independent measurements, then E(f |σ (g)) = E(f ), and E(f g) = E(f )E(g). Thus, for example, if f and g are independent, then m(f ∈ F |g) = m(f ∈ F ) a.e.

6.9 Conditional Expectation

189

Exercises 1. Prove Corollary 6.6. 2. Prove the following generalization of Corollary 6.5. Lemma 6.12. Suppose that G1 ⊂ G2 are two sub-σ -fields of B and that f ∈ L1 (m). Prove that E(E(f |G2 )|G1 ) = E(f |G1 ). 3. Given a probability space (Ω, B, m) and a measurement f , for what sub-σ -field G ⊂ B is it true that E(f |G) = E(f ) a.e.? For what sub-σ field H ⊂ B is it true that E(f |H ) = f a.e.? 4. Suppose that Gn is an increasing sequence of sub-σ -fields: Gn ⊂ Gn+1 and f is an integrable measurement. Define the random variables Xn = E(f |Gn ).

(6.35)

Prove that Xn has the properties that it is measurable with respect to Gn and E(Xn |X1 , X2 , . . . , Xn−1 ) = Xn−1 ; (6.36) that is, the conditional expectation of the next value given all the past values is just the most recent value. A sequence of random variables with this property is called a martingale and there is an extremely rich theory regarding the convergence and uniform integrability properties of such sequences. (See, e.g., the classic reference of Doob [20] or the treatments in Ash [1] or Breiman [10].) We point out in passing the general form of martingales, but we will not develop the properties in any detail. A martingale consists of a sequence of random variables {Xn } defined on a common probability space (Ω, B, m) together with an increasing sequence of sub-σ -fields Bn with the following properties: for n = 1, 2, . . . E|Xn | < ∞,

(6.37)

Xn is Bn − measurable,

(6.38)

E(Xn+1 |Bn ) = Xn .

(6.39)

and If the third relation is replaced by E(Xn+1 |Bn ) ≥ Xn , then {Xn , Bn } is instead a submartingale. If instead E(Xn+1 |Bn ) ≤ Xn , then {Xn , Bn } is a supermartingale. Martingales have their roots in gambling theory, in which a martingale can be considered as a fair game–the capital expected next time is given by the current capital. A submartingale represents an unfavorable game (as in a casino), and a supermartin-

190

6 Conditional Probability and Expectation

gale a favorable game (as was the stock market during the decade preceding the first edition of this book). 5. Show that {Xn , Bn } is a martingale if and only if for all n Z Z Xn+1 dm = Xn dm, all G ∈ Bn . G

G

6.10 Independence and Markov Chains In this section we apply some of the results and interpretations of the previous sections to obtain formulas for probabilities involving random variables that have a particular relationship called the Markov chain property. Let (Ω, B, P ) be a probability space and let X : Ω → AX , Y : Ω → AY , and Z : Ω → AZ be three random variables defined on this space with σ -fields BAX , BAY , and BAZ , respectively. As we wish to focus on the three random variables rather than on the original space, we can consider (Ω, B, P ) to be the space (AX × AY × AZ , BAX × BAY × BAZ , PXY Z ). As a preliminary, let PXY denote the joint distribution of the random variables X and Y ; that is, PXY (F × G) = P (X −1 (F ) ∩ Y −1 (G)); F ∈ BX , G ∈ BY . Translating the definition of independence of measurements of Section 6.9 into distributions, the random variables X and Y are independent if and only if PXY (F × G) = PX (F )PY (F ); F ∈ BX , G ∈ BY , where PX and PY are the marginal distributions of X and Y . We can also state this in terms of joint distributions in a convenient way. Given two distributions PX and PY , define the product distribution PX × PY or PX×Y (both notations are used) as the distribution on (AX × AY , BAX × BAY ) specified by PX×Y (F × G) = PX (F )PY (G); F ∈ BX , G ∈ BY . Thus the product distribution has the same marginals as the original distribution, but it forces the random variables to be independent. Lemma 6.13. Random variables X and Y are independent if and only if PXY = PX×Y . Given measurements f : AX → Af and g : AY → Ag , then if X and Y are independent, so are f (X) and g(Y ) and Z Z Z f gdPX×Y = ( f dPX )( gdPY ).

6.10 Independence and Markov Chains

191

Proof. The first statement follows immediately from the definitions of independence and product distributions. The second statement follows from the fact that Pf (X),g(Y ) (F , G) = PX,Y (f −1 (F ), g −1 (G)) = PX (f −1 (F ))PY (g −1 (G)) = Pf (X) (F )Pg(Y ) (G).

2 The final statement then follows from Corollary 6.6. We next introduce a third random variable and a form of conditional independence among random variables. As considered in Section 6.8, the conditional probability P (F |σ (Z)) can be written as g(Z(ω)) for some function g(z) and we define this function to be P (F |z). Similarly the conditional probability P (F |σ (Z, Y )) can be written as r (Z(ω), Y (ω)) for some function r (z, y) and we denote this function P (F |z, y). We can then define the conditional distributions PX|Z (F |z) = P (X −1 (F )|z) PY |Z (G|z) = P (Y −1 (G)|z) PXY |Z (F × G|z) = P (X −1 (F ) ∩ Y −1 (G)|z) PX|Z,Y (F |z, y) = P (X −1 (F )|z, y). We say that Y → Z → X is a Markov chain if for any event F ∈ BAX with probability one PX|Z,Y (F |z, y) = PX|Z (F |z), or in terms of the original space, with probability one P (X −1 (F )|σ (Z, Y )) = P (X −1 (F )|σ (Z)). The following lemma provides an equivalent form of this property in terms of a conditional independence relation. Lemma 6.14. The random variables Y ,Z, X form a Markov chain Y → Z → X if and only if for any F ∈ BAX and G ∈ BAY we know that with probability one PXY |Z (F × G|z) = PX|Z (F |z)PY |Z (G|z), or, equivalently, with probability one P (X −1 (F ) ∩ Y −1 (G)|σ (Z)) = P (X −1 (F )|σ (Z))P (Y −1 (G)|σ (Z)). If the random variables have the preceding property, we also say that X and Y are conditionally independent given Z.

192

6 Conditional Probability and Expectation

Proof. For any F ∈ BAX , G ∈ BAY , and D ∈ BAZ we know from the properties of conditional probability that Z P (X −1 (F )|σ (Z, Y )) dP P (X −1 (F ) ∩ Y −1 (G) ∩ Z −1 (D)) = Y −1 (G)∩Z −1 (D) Z = P (X −1 (F )|σ (Z)) dP Y −1 (G)∩Z −1 (D)

using the definition of the Markov property. Using first (6.32) and then Corollary 6.3 this integral can be written as Z 1Z −1 (D) 1Y −1 (G) P (X −1 (F )|σ (Z)) dP Z = E[1Z −1 (D) 1Y −1 (G) P (X −1 (F )|σ (Z))|σ (Z)] dP Z = 1Z −1 (D) E[1Y −1 (G) |σ (Z)]P (X −1 (F )|σ (Z))dP , where we have used the fact that both 1Z −1 (D) and P (X −1 (F )|σ (Z)) are measurable with respect to σ (Z) and hence come out of the conditional expectation. From Corollary 6.3, however, E[1Y −1 (G) |σ (Z)] = P (Y −1 (G)|σ (Z)), and hence combining the preceding relations yields Z −1 −1 −1 P (Y −1 (G)|σ (Z))P (X −1 (F )|σ (Z)) dP . P (X F ∩ Y G ∩ Z D) = Z −1 (D)

We also know from the properties of conditional probability that Z −1 −1 −1 P (X F ∩ Y G ∩ Z D) = P (Y −1 (G) ∩ X −1 (F )|σ (Z)) dP Z −1 (D)

and hence P (X −1 F ∩ Y −1 G ∩ Z −1 D) Z = P (Y −1 (G)|σ (Z))P (X −1 (F )|σ (Z)) dP Z −1 (D) Z = P (Y −1 (G) ∩ X −1 (F )|σ (Z)) dP . Z −1 (D)

(6.40)

Since the integrands in the two right-most integrals are measurable with respect to σ (Z), it follows from Corollary 6.1 that they are equal almost everywhere, proving the lemma. 2 As a simple example, if the random variables are discrete with probability mass function pXY Z , then Y , Z, X form a Markov chain if and only

6.10 Independence and Markov Chains

193

if for all x, y, z pX|Y ,Z (x|y, z) = pX|Z (x|y) or, equivalently, pX,Y |Z (x, y|z) = pX|Z (x|z)pY |Z (y|z). We close this section and chapter with an alternative statement of the previous lemma. The result is an immediate consequence of the lemma and the definitions. Corollary 6.7. Given the conditions of the previous lemma, let PXY Z denote the distribution of the three random variables X, Y , and Z. Let PX×Y |Z denote the distribution (if it exists) for the same random variables specified by the formula PX×Y |Z (F × G × D) Z P (Y −1 (G)|σ (Z))P (X −1 (F )|σ (Z)) dP = Z −1 (D) Z PX|Z (F |z)PY |Z (G|z)dPZ (z), F ∈ BAX ; G ∈ BAY ; D ∈ BAZ ; = D

that is, PX×Y |Z is the distribution on X, Y , Z which agrees with the original distribution PXY Z on the conditional distributions of X given Z and Y given Z and with the marginal distribution Z, but which is such that Y → Z → X forms a Markov chain or, equivalently, such that X and Y are conditionally independent given Z. Then Y → Z → X forms a Markov chain if and only if PXY Z = PX×Y |Z . Comment: The corollary contains the technical qualification that the given distribution exists because it need not in general, that is, there is no guarantee that the given set function is indeed countably additive and hence a probability measure. If the alphabets are assumed to be standard, however, then the conditional distributions are regular and it follows that the set function PX×Y |Z is indeed a distribution.

Exercise 1. Suppose that Yn , n = 1, 2, . . . is a sequence of independent random variables with the same marginal distributions PYn = PY . Define the sum n X Yi . Xn = i=1

194

6 Conditional Probability and Expectation

Show that the sequence Xn is a martingale. This points out that martingales can be viewed as one type of memory in a random process, here formed simply by adding up memoryless random variables.

Chapter 7

Ergodic Properties

Abstract Random processes and dynamical systems possess ergodic properties if the time averages of measurements converge. This chapter looks at the implications of ergodic properties and limiting sample averages. The relation between expectations of the averages and averages of the expectations are derived, and it is seen that if a process has sufficient ergodic properties, then the process must be asymptotic mean stationary and that the the properties describe a stationary measure with the same ergodic properties as the asymptotically mean stationary process. A variety of related characterizations of the system transformation are given that relate to the asymptotic behavior, including ergodicity, recurrence, mixing, block stationarity and ergodicity, and total ergodicity. The chapter concludes with the ergodic decomposition which shows that the stationary measure implied by convergent sample averages is equivalent to a mixture of stationary and ergodic processes. The ergodic theorems of the next chapter provide conditions under which a process or system will possess ergodic properties for large classes of measurements. In particular, it is shown that asymptotic mean stationarity is a sufficient condition as well as a necessary condition.

7.1 Ergodic Properties of Dynamical Systems As in the previous chapter we focus on time and ensemble averages of measurements made on a dynamical system. Let (Ω, B, m, T ) be a dynamical system. The case of principal interest will be the dynamical system corresponding to a random process, that is, a dynamical system (AI , B(A)I , m, T ), where I is an index set, usually either the nonnegative integers Z+ or the set Z of all integers, and T is a shift on the sequence space AI . The sequence of coordinate random variables {Πn , n ∈ I} is the corresponding random process and m is its distribution. The dynamical R.M. Gray, Probability, Random Processes, and Ergodic Properties, DOI: 10.1007/978-1-4419-1090-5_7, © Springer Science + Business Media, LLC 2009

195

196

7 Ergodic Properties

system may or may not be that associated with a directly given alphabet A random process {Xn ; n ∈ I} described by a distribution m and having T be the shift on the sequence space AI . Dynamical systems corresponding to a directly given random process will be the most common example, but we shall be interested in others as well. For example, the space may be the same, but we may wish to consider block or variable-length shifts instead of the unit time shift. As previously, given a measurement f (i.e., a measurable mapping of Ω into R ) define the sample averages < f >n = n−1

n−1 X

f (T i x).

i=0

A dynamical system will be said to have the ergodic property with respect to a measurement f if the sample average < f >n converges almost everywhere as n → ∞. A dynamical system will be said to have ergodic properties with respect to a class M of measurements if the sample averages < f >n converge almost everywhere as n → ∞ for all f in the given class M. When unclear from context, the measure m will be explicitly given. We shall also refer to mean ergodic properties (of order i) when the convergence is in Li . If limn→∞ < f >n (x) exists, we will usually denote it by < f > (x) or fˆ(x). For convenience we consider < f > (x) to be 0 if the limit does not exist. From Corollary 5.11 such limits are unique only in an almost everywhere sense; that is, if < f >n →< f > and < f >n → g a.e., then < f >= g a.e. Interesting classes of functions that we will consider include the class of all indicator functions of events, the class of all bounded measurements, and the class of all integrable measurements with respect to a given probability measure. The properties of limits immediately yield properties of limiting sample averages that resemble those of probabilistic averages. Lemma 7.1. Given a dynamical system, let E denote the class of measurements with respect to which the system has ergodic properties. (a) If f ∈ E and f T i ≥ 0 a.e. for all i, then also < f >≥ 0 a.e. (The condition is met, for example, if f (ω) ≥ 0 for all ω ∈ Ω.) (b) The constant functions f (ω) = r , r ∈ R, are in E. In particular, < 1 >= 1. (c) If f , g ∈ E then also that < af + bg >= a < f > +b < g > and hence af + bg ∈ E for any real a, b. Thus E is a linear space. (d) If f ∈ E, then also f T i ∈ E, i = 1, 2, . . . and < f T i >=< f >; that is, if a system has ergodic properties with respect to a measurement, then it also has ergodic properties with respect to all shifts of the measurement and the limit is the same. Proof. Parts (a)-(c) follow from Lemma 5.14. To prove (d) we need only consider the case i = 1 since that implies the result for all i. If f ∈ E then with probability one

7.1 Ergodic Properties of Dynamical Systems

197

n−1 1 X f (T i x) = f (x) n→∞ n i=0

lim

or, equivalently, | < f >n −f | → 0. n→∞

The triangle inequality shows that < f T >n must have the same limit: | < f T >n − < f > | n+1 = |( ) < f >n+1 −n−1 f − < f >| n n+1 ≤ |( ) < f >n+1 − < f >n+1 | +| < f >n+1 −f | +n−1 | f | n 1 1 ≤ | < f >n+1 | +| < f >n+1 −f | + |f | → 00 n→∞ n n where the middle term goes to zero by assumption, implying that the first term must also go to zero, and the right-most term goes to zero since f cannot assume ∞ as a value. 2 The preceding properties provide some interesting comparisons between the space E of all measurements on a dynamical system (Ω, F, m, T ) with limiting sample averages and the space L1 of all measurements with expectations with respect to the same measure. Properties (b) and (c) of limiting sample averages are shared by expectations. Property (a) is similar to the property of expectation that f ≥ 0 a.e. implies that Ef ≥ 0, but here we must require that f T i ≥ 0 a.e. for all nonnegative integers i. Property (d) is not, however, shared by expectations in general since integrability of f does not in general imply integrability of f T i . There is one case, however, where the property is shared: if the measure is invariant, then from Lemma 5.22 f is integrable if and only if f T i is, and, if the integrals exist, they are equal. In addition, if the measure is stationary, then m({ω : f (T n ω) ≥ 0}) = m({T −n ω : f (ω) ≥ 0}) = m(f ≥ 0) and hence in the stationary case the condition for (a) to hold is equivalent to the condition that f ≥ 0 a.e.. Thus for stationary measures, there is an exact parallel among the properties (a)-(d) of measurements in E and those in L1 . These similarities between time and ensemble averages will be useful for both the stationary and nonstationary systems. The proof of the lemma yields a property of limiting sample averages that is very useful. We formalize this fact as a corollary after we give a needed definition: Given a dynamical system (Ω, B, m, T ) a function f : Ω → Λ is said to be invariant (with respect to T ) or stationary (with respect to T ) if f (T ω) = f (ω), all ω ∈ Ω. When it is understood that T is a shift, the name time-invariant is also used.

198

7 Ergodic Properties

Corollary 7.1. If a dynamical system has ergodic properties with respect to a measurement f , then the limiting sample average < f > is an invariant function, that is, < f > (T x) =< f > (x). Proof. If limn→∞ < f >n (x) =< f >, then from the lemma limn→∞ < f >n (T x) also exists and is the same, but the limiting time average of f T (x) = f (T x) is just < f T > (x), proving the corollary. 2 The corollary simply formalizes the observation that shifting x once cannot change a limiting sample average, hence such averages must be invariant. Ideally one would like to consider the most general possible measurements at the outset, e.g., integrable or square-integrable measurements. We initially confine attention to bounded measurements, however, as a compromise between generality and simplicity. The class of bounded measurements is simple because the measurements and their sample averages are always integrable and uniformly integrable and the class of measurements considered does not depend on the underlying measure. All of the limiting results of probability theory immediately and easily apply. This permits us to develop properties that are true of a given class of measurements independent of the actual measure or measures under consideration. Eventually, however, we will wish to demonstrate for particular measures that sample averages converge for more general measurements. Although a somewhat limited class, the class of bounded measurements contains some important examples. In particular, it contains the indicator functions of all events, and hence a system with ergodic properties with respect to all bounded measurements will have limiting values for all relative frequencies, that is, limits of < 1F >n as n → ∞. The quantizer sequence of Lemma 5.3 can be used to prove a form of converse to this statement: if a system possesses ergodic properties with respect to the indicator functions of events, then it also possesses ergodic properties with respect to all bounded measurable functions. Lemma 7.2. A dynamical system has ergodic properties with respect to the class of all bounded measurements if and only if it has ergodic properties with respect to the class of all indicator functions of events, i.e., all measurable indicator functions. Proof. Since indicator functions are a subclass of all bounded functions, one implication is immediate. The converse follows by a standard approximation argument, which we present for completeness. Assume that a system has ergodic properties with respect to all indicator functions of events. Let E be the class of all measurements with respect to which the system has ergodic properties. Then E contains all indicator functions of events, and, from the previous lemma, it is linear. Hence it also contains all measurable simple functions. The general limiting sample

7.1 Ergodic Properties of Dynamical Systems

199

averages can then be constructed from those of the simple functions in the same manner that we used to define ensemble averages, that is, for nonnegative functions form the increasing limit of the averages of simple functions and for general functions form the nonnegative and negative parts. Assume that f is a bounded measurable function with, say, | f |≤ K. Let ql denote the quantizer sequence of Lemma 5.3 and assume that l > K and hence | ql (f (ω)) − f (ω) |≤ 2−l for all l and ω by construction. This also implies that | ql (f (T i ω)) − f (T i ω) | ≤ 2−l for all i, l, ω. It is the fact that this bound is uniform over i (because f is bounded) that makes the following construction work. Begin by assuming that f is nonnegative. Since ql (f ) is a simple function, it has a limiting sample average, say < ql (f ) >, by assumption. We can take these limiting sample averages to be increasing in l since j ≥ l implies that < qj (f ) >n − < ql (f ) >n = < qj (f ) − ql (f ) >n = n−1

n−1 X

(qj (f T i ) − ql (f T i )) ≥ 0,

i=0

since each term in the sum is nonnegative by construction of the quantizer sequence. Hence from Lemma 7.1 we must have that qj (f )−ql (f ) = qj (f ) − ql (f ) ≥ 0 and hence < ql (f ) > (ω) =< ql (f (ω)) > is a nondecreasing function and hence must have a limit, say fˆ. Furthermore, since f and hence ql (f ) and hence < ql (f ) >n and hence < ql (f ) > are all in [0, K], so must fˆ be. We now have from the triangle inequality that for any l |< f >n −fˆ |≤|< f >n − < ql (f ) >n | + |< ql (f ) >n −ql (f ) | + | ql (f ) − fˆ | . The first term on the right-hand side is bound above by 2−l . Since we have assumed that < ql (f ) >n converges with probability one to < ql (f ) >, then with probability one we get an ω such that < ql (f ) >n → < ql (f ) > n→∞

for all l = 1, 2, . . . (The intersection of a countable number of sets with probability one also has probability one.) Assume that there is an ω in this set. Then given  we can choose an l so large that |< ql (f ) > −fˆ |≤ /2 and 2−l ≤ /2, in which case the preceding inequality yields lim sup |< f >n −fˆ |≤ . n→∞

200

7 Ergodic Properties

Since this is true for arbitrary , for the given ω it follows that < f >n (ω) → fˆ(ω), proving the almost everywhere result for nonnegative f . The result for general bounded f is obtained by applying the nonnegative result to f + and f − in the decomposition f = f + − f − and using the fact that for the given quantizer sequence ql (f ) = ql (f + ) − ql (f − ).

2 Because of the importance of these ergodic properties, we shall simplify the phrase “ergodic properties with respect to all bounded functions” to simply “bounded ergodic properties.” Thus a dynamical system (or random process) is said to have bounded ergodic properties if for any bounded measurable function (including any measurable indicator function) it assigns probability 1 to the set of all sequences for which the sample average of the function (or the relative frequency of the event) converges.

7.2 Implications of Ergodic Properties To begin developing the implications of ergodic properties, suppose that a system possesses ergodic properties for a class of bounded measurements. Then corresponding to each bounded measurement f in the class there will be a limit function, say < f >, such that with probability one < f >n →< f > as n → ∞. Convergence almost everywhere and the boundedness of f imply L1 convergence, and hence for any event G | n−1

n−1 XZ i=0

f T i dm − G

Z

Z

(n−1

f dm | = | G

n−1 X

G

Z

|n−1

≤ G

f T i − f )dm|

i=0 n−1 X

f T i − f | dm

i=0

≤ k < f >n −f k1 → 0. n→∞

Thus we have proved the following lemma. Lemma 7.3. If a dynamical system with measure m has the ergodic property with respect to a bounded measurement f , then lim n

n→∞

−1

n−1 XZ i=0

i

Z

f T dm = G

f dm, all G ∈ B. G

Thus, for example, if we take G as the whole space

(7.1)

7.2 Implications of Ergodic Properties

lim n−1

n→∞

n−1 X

201

Em (f T i ) = Em f .

(7.2)

i=0

Suppose that a random process has ergodic properties with respect to the class of all bounded measurements and let f be an indicator function of an event F . Application of (7.1) yields lim n−1

n→∞

n−1 X

m(T −i F ∩ G) = lim n−1 n→∞

i=0

n−1 XZ i=0

G

1F T i dm

= Em (1F 1G ), all F , G ∈ B.

(7.3)

For example, if G is the entire space than (7.3) becomes lim n−1

n→∞

n−1 X

m(T −i F ) = Em (1F ), all events F .

(7.4)

i=0

The limit properties of the preceding lemma are trivial implications of the definition of ergodic properties, yet they will be seen to have farreaching consequences. We note the following generalization for later use. Corollary 7.2. If a dynamical system with measure m has the ergodic property with respect to a measurement f (not necessarily bounded) and if the sequence < f >n is uniformly integrable, then (7.1) and hence also (7.2) hold. Proof. The proof follows immediately from Lemma 5.20 since this ensures the needed L1 convergence. 2 Recall from Corollary 5.9 that if the f T n are uniformly integrable, then so are the < f >n . Thus, for example, if m is stationary and f is mintegrable, then Lemma 5.23 and the preceding lemma imply that (7.1) and (7.2) hold, even if f is not bounded. In general, the preceding lemmas state that if the arithmetic means of a sequence of “nice” measurements converges in any sense, then the corresponding arithmetic means of the expected values of the measurements must also converge. In particular, they imply that the arithmetic mean of the measures mT −i must converge, a fact that we emphasize as a corollary: Corollary 7.3. If a dynamical system with measure m has ergodic properties with respect to the class of all indicator functions of events, then n−1 1 X m(T −i F ) exists, all events F . n→∞ n i=0

lim

(7.5)

202

7 Ergodic Properties

Recall that a measure m is stationary or invariant with respect to T if all of these measures are identical. Clearly (7.5) holds for stationary measures. More generally, say that (7.5) holds for a measure m and transformation T . Define the set functions mn , n = 1, 2, . . . by mn (F ) = n−1

n−1 X

m(T −i F ).

i=0

The mn are easily seen to be probability measures, and hence (7.5) implies that there is a sequence of measures mn such that for every event F the sequence mn (F ) has a limit. If the limit of the arithmetic mean of the transformed measures mT −i exists, we shall denote it by m. The following lemma shows that if a sequence of measures mn is such that mn (F ) converges for every event F , then the limiting set function is itself a measure. Thus if (7.5) holds and we define the limit as m, then m is itself a probability measure. Lemma 7.4. (The Vitali-Hahn-Saks Theorem) Given a sequence of probability measures mn and a set function m such that lim mn (F ) = m(F ), all events F , n→∞

then m is also a probability measure. Proof. It is easy to see that m is nonnegative, normalized, and finitely additive. Hence we need only show that it is countably additive. Observe that it would not help here for the space to be standard as that would only guarantee that m defined on a field would have a countably additive extension, not that the given m that is defined on all events would itself be countably additive. To prove that m is countably additive we assume the contrary and show that this leads to a contradiction. Assume that m is not countably additive and hence from (1.26) there must exist a sequence of disjoint sets Fj with union F such that ∞ X

m(Fj ) = b < a = m(F ).

(7.6)

j=1

We shall construct a sequence of integers M(k), k = 1, 2, . . ., a corresponding grouping of the Fj into sets Bk as Bk =

M(k) [

Fj , k = 1, 2, . . . ,

(7.7)

j=M(k−1)+1

a set B defined as the union of all of the Bk for odd k: B = B 1 ∪ B3 ∪ B 5 ∪ . . . ,

(7.8)

7.2 Implications of Ergodic Properties

203

and a sequence of integers N(k), k = 1, 2, 3, . . ., such that mN(k) (B) oscillates between values of approximately b when k is odd and approximately a when k is even. This will imply that mn (B) cannot converge and hence will yield the desired contradiction. Fix  > 0 much smaller than a − b. Fix M(0) = 0. Step 1: Choose M(1) so that b−

M X

m(Fj ) <

j=1

 if M ≥ M(1). 8

(7.9)

and then choose N(1) so that |

M(1) X

mn (Fj ) − b |= |mn (B1 ) − b |<

j=1

and |mn (F ) − a |≤

 , n ≥ N(1), 4

(7.10)

 , n ≥ N(1). 4

(7.11)

Thus for large n, mn (B1 ) is approximately b and mn (F ) is approximately a. Step 2: We know from (7.10)-(7.11) that for n ≥ N(1) |mn (F − B1 ) − (a − b)| ≤ |mn (F ) − a| + |mn (B1 ) − b| ≤

 , 2

and hence we can choose M(2) sufficiently large to ensure that |mN(1) (

M(2) [

Fj ) − (a − b)| = |mN(1) (B2 ) − (a − b)| ≤

j=M(1)+1

3 . 4

(7.12)

From (7.9) b−

M(2) X j=1

m(Fj ) = b − m(B1 ∪ B2 ) <

 , 8

and hence analogous to (7.10) we can choose N(2) large enough to ensure that  |b − mn (B1 ∪ B2 )| < , n ≥ N(2). (7.13) 4 Observe that mN(1) assigns probability roughly b to B1 and roughly (a − b) to B2 , leaving only a tiny amount for the remainder of F . mN(2) , however, assigns roughly b to B1 and to the union B1 ∪ B2 and hence assigns almost nothing to B2 . Thus the compensation term a − b must be made up at higher indexed Bi ’s. Step 3: From (7.13) of Step 2 and (7.11)

204

7 Ergodic Properties

|mn (F −

2 [

Bi ) − (a − b)| ≤ |mn (F ) − a| + |mn (

i=1

2 [

Bi ) − b|

i=1

≤

   + = , n ≥ N(2). 4 4 2

Hence as in (7.12) we can choose M(3) sufficiently large to ensure that M(3) [

|mN(2) (

Fj ) − (a − b)| = |mN(2) (B3 ) − (a − b)| ≤

j=M(2)+1

3 . 4

(7.14)

Analogous to (7.13) using (7.9) we can choose N(3) large enough to ensure that 3 [  |b − mn ( Bi )| ≤ , n ≥ N(3). (7.15) 4 i=1 .. . Step k: From step k − 1 for sufficiently large M(k) |mN(k−1) (

M(k) [

Fj ) − (a − b)| = |mN(k−1) (Bk ) − (a − b)| ≤

j=M(k−1)+1

3 (7.16) 4

and for sufficiently large N(k) |b − mn (

k [ i=1

Bi )| ≤

 , n ≥ N(k). 4

(7.17)

Intuitively, we have constructed a sequence mN(k) of probability measures such that mN(k) always puts a probability of approximately b on the union of the first k Bi and a probability of about a − b on Bk+1 . In fact, most of the probability weight of the union of the first k Bi lies in B1 from (7.9). Together this gives a probability of about a and hence yields the probability of the entire set F , thus mN(k) puts nearly the total probability of F on the first B1 and on Bk+1 . Observe then that mN(k) keeps pushing the required difference a − b to higher indexed sets Bk+1 as k grows. If we now define the set B of (7.8) as the union of all of the odd Bj , then the probability of B will include b from B1 and it will include the a − b from Bk+1 if and only if k + 1 is odd, that is, if k is even. The remaining sets will contribute a negligible amount. Thus we will have that mN(k) (B) is about b + (a − b) = a for k even and b for k odd. We complete the proof by making this idea precise. If k is even, then from (7.10) and (7.16)

7.2 Implications of Ergodic Properties

mN(k) (B) =

205

X

mN(k) (Bj ) odd j ≥ mN(k) (B1 ) + mN(k) (Bk+1 )  3 ≥ b − + (a − b) − 4 4 = a − , k even .

(7.18)

For odd k the compensation term (a − b) is not present and we have from (7.11) and (7.16) X mN(k) (B) = mN(k) (Bj ) odd j ≤ mN(k) (F ) − mN(k) (Bk+1 ) 3  ) ≤ a + − ((a − b) − 4 4 = b + . (7.19) Eqs. (7.18)-(7.19) demonstrate that mN(k) (B) indeed oscillates as claimed and hence cannot converge, yielding the desired contradiction and completing the proof of the lemma. 2 Say that a probability measure m and transformation T are such that the arithmetic means of the iterates of m with respect to T converge to a set function m; that is, lim n−1

n→∞

n−1 X

m(T −i F ) = m(F ), all events F .

(7.20)

i=0

From the previous lemma, m is a probability measure. Furthermore, it follows immediately from (7.20) that m is stationary with respect to T , that is, m(T −1 F ) = m(F ), all events F . Since m is the limiting arithmetic mean of the iterates of a measure m and since it is stationary, a probability measure m for which (7.5) holds, that is, for which all of the limits of (7.20) exist, is said to be asymptotically mean stationary or AMS. The limiting probability measure m is called the stationary mean of m.. Since we have shown that a random process possessing ergodic properties with respect to the class of all indicator functions of events is AMS we have proved the following result. Theorem 7.1. A necessary condition for a random process to possess ergodic properties with respect to the class of all indicator functions of events (and hence with respect to any larger class such as the class of all bounded measurements) is that it be asymptotically mean stationary.

206

7 Ergodic Properties

We shall see in the next chapter that the preceding condition is also sufficient.

Exercise 1. A dynamical system (Ω, B, m, T ) is N-stationary for a positive integer N if it is stationary with respect to T N , that is, if for all F m(T −i F ) is periodic with period N. A system is said to be block stationary if it is N-stationary for some N. Show that an N-stationary process is AMS with stationary mean m(F ) =

N−1 1 X m(T −i F ). N i=0

Show in this case that m  m.

7.3 Asymptotically Mean Stationary Processes In this section we continue to develop the implications of ergodic properties. The emphasis here is on a class of measurements and events that are intimately connected to ergodic properties and will play a basic role in further necessary conditions for such properties as well as in the demonstration of sufficient conditions. These measurements help develop the relation between an AMS measure m and its stationary mean m. The theory of AMS processes follows the work of Dowker [22] [23], Rechard [88], and Gray and Kieffer [39].

Invariant Events and Measurements As before, we have a dynamical system (Ω, B, m, T ) with ergodic properties with respect to a measurement, f and hence with probability one the sample average n−1 1 X fTi n→∞ n i=0

lim < f >n = lim

n→∞

(7.21)

exists. From Corollary 7.1, the limit, say < f >, must be an invariant function. Define the event

7.3 Asymptotically Mean Stationary Processes

F = {x : lim n−1 n→∞

n−1 X

f (T i x) exists},

207

(7.22)

i=0

and observe that by assumption m(F ) = 1. Consider next the set T −1 F . Then x ∈ T −1 F or T x ∈ F if and only if the limit lim n−1

n→∞

n−1 X

f (T i (T x)) = f (T x)

i=0

exists. Provided that f is a real-valued measurement and hence cannot take on infinity as a value (that is, it is not an extended real-valued measurement), then as in Lemma 7.1 the left-hand limit is exactly the same as the limit n−1 X lim n−1 f (T i x). n→∞

i=0

In particular, either limit exists if and only if the other one does and hence x ∈ T −1 F if and only if x ∈ F . We formalize this property in a definition: If an event F is such that T −1 F = F , then the event is said to be invariant or T -invariant or invariant with respect to T . Thus we have seen that the event that limiting sample averages converge is an invariant event and that the limiting sample averages themselves are invariant functions. Observe that we can write for all x that n−1 1 X f (T i x). n→∞ n n=0

f (x) = 1F (x) lim

We shall see that invariant events and invariant measurements are intimately connected. When T is the shift, invariant measurements can be interpreted as measurements whose output does not depend on when the measurement is taken and invariant events are events that are unaffected by shifting. Limiting sample averages are not the only invariant measurements; other examples are lim sup < f >n (x), n→∞

and lim inf < f >n (x). n→∞

Observe that an event is invariant if and only if its indicator function is an invariant measurement. Observe also that for any event F , the event

208

7 Ergodic Properties

Fi.o. = lim sup T −n F = n→∞

∞ [ ∞ \

T −k F

n=0 k=n

= {x : x ∈ T −n F i.o.} = {x : T n x ∈ F i.o.}

(7.23)

is invariant, where i.o. means infinitely often. To see this, observe that T −1 Fi.o. =

∞ [ ∞ \

T −k F

n=1 k=n

and

∞ [

T −k F ↓ Fi.o. .

k=n

Invariant events are “closed” to transforming, e.g., shifting: If a point begins inside an invariant event, it must remain inside that event for all future transformations or shifts. Invariant measurements are measurements that yield the same value on T n x for any n. Invariant measurements and events yield special relationships between an AMS random process and its asymptotic mean, as described in the following lemma. Lemma 7.5. Given an AMS system with measure m with stationary mean m, then m(F ) = m(F ), all invariant F , and Em f = Em f , all invariant f , where the preceding equation means that if either integral exists, then so does the other and they are equal. Proof. The first equality follows by direct substitution into the formula defining an AMS measure. The second formula then follows from standard approximation arguments, i.e., since it holds for indicator functions, it holds for simple functions by linearity. For positive functions take an increasing quantized sequence and apply the simple function result. The two limits must converge or diverge together. For general f , decompose it into positive and negative parts and apply the positive function result. 2 In some applications we deal with measurements or events that are “almost invariant” in the sense that they equal their shift with probability one instead of being invariant in the strict sense. As there are two measures to consider here, an AMS measure and its stationary mean, a little more care is required to conclude results like Lemma 7.5. For example, suppose that an event F is invariant m-a.e. in the sense that m(F ∆T −1 F ) = 0. It does not then follow that F is also invariant m-a.e.

7.3 Asymptotically Mean Stationary Processes

209

or that m(F ) = m(F ). The following lemma shows that if one strengthens the definitions of almost invariant to include all shifts, then m and m will behave similarly for invariant measurements and events. Lemma 7.6. An event F is said to be totally invariant m-a.e. if m(F ∆T −k F ) = 0, k = 1, 2, . . . . Similarly, a measurement f is said to be totally invariant m-a.e. if m(f = f T k ; k = 1, 2, . . .) = 1. Suppose that m is AMS with stationary mean m. If F is totally invariant m-a.e. then m(F ) = m(F ), and if f is totally invariant m-a.e. Em f = Em f . Furthermore, if f is totally invariant m-a.e., then the event f −1 (G) = {ω : f (ω) ∈ G} is also totally invariant and hence m(f −1 (G)) = m(f −1 (G)). Proof. It follows from the elementary inequality |m(F ) − m(G)| ≤ m(F ∆G) that if F is totally invariant m-a.e., then m(F ) = m(T −k F ) for all positive integers k. This and the definition of m imply the first relation. The second relation then follows from the first as in Lemma 7.5. If m puts probability 1 on the set D of ω for which the f (T k ω) are all equal, then for any k m(f −1 (G)∆T −k f −1 (G)) = m({ω : f (ω) ∈ G}∆{ω : f (T k ω) ∈ G)}) = m({ω : ω ∈ D} ∩ ({ω : f (ω) ∈ G}∆{ω : f (T k ω) ∈ G})) =0 since if ω is in D, f (ω) = f (T k ω) and the given difference set is empty.

2 Lemma 7.5 has the following important implication, Corollary 7.4. Given an AMS system (Ω, B, m, T ) with stationary mean m, then the system has ergodic properties with respect to a measurement f if and only if the stationary system (Ω, B, m, T ) has ergodic properties

210

7 Ergodic Properties

with respect to f . Furthermore, the limiting sample average can be chosen to be the same for both systems. Proof. The set F = {x : limn→∞ < f >n exists} is invariant and hence m(F ) = m(F ) from the lemma. Thus either both measures assign probability one to this set and hence both systems have ergodic properties with respect to f or neither does. Choosing < f > (x) as the given limit for x ∈ F and 0 otherwise completes the proof. 2 Another implication of the lemma is the following relation between an AMS measure and its stationary mean. We first require the definition of an asymptotic form of domination or absolute continuity. Recall that m  η means that η(F ) = 0 implies m(F ) = 0. We say that a measure η asymptotically dominates m (with respect to T ) if η(F ) = 0 implies that lim m(T −n F ) = 0.

n→∞

Roughly speaking, if η asymptotically dominates m, then zero probability events under η will have vanishing probability under m if they are shifted far into the future. This can be considered as a form of steady state behavior in that such events may have nonzero probability as transients, but eventually their probability must tend to 0 as things settle down. Corollary 7.5. If m is asymptotically mean stationary with stationary mean m, then m asymptotically dominates m. If T is invertible (e.g., the two-sided shift), then asymptotic domination implies ordinary domination and hence m  m. Proof. From the lemma m and m place the same probability on invariant sets. Suppose that F is an event such that m(F ) = 0. Then the set G = lim supn→∞ T −n F of (7.23) has m measure 0 since from the continuity of probability and the union bound m(G) = lim m( n→∞

≤

∞ X

∞ [

T −n F ) ≤ m(

k=n

∞ [

T −n F )

k=1

m(T −n F ) = 0.

j=1

In addition G is invariant, and hence from the lemma m(G) = 0. Applying Fatou’s lemma (Lemma 5.10(c)) to indicator functions yields lim sup m(T −n F ) ≤ m(lim sup T −n F ) = 0, n→∞

n→∞

proving the first statement. Suppose that η asymptotically dominates m and that T is invertible. Say η(F ) = 0. Since T is invertible the set

7.3 Asymptotically Mean Stationary Processes

211

S∞

n n=−∞ T F is measurable and as in the preceding argument has m measure 0. Since the set is invariant it also has m measure 0. Since the set contains the set F , we must have m(F ) = 0, completing the proof. 2 The next lemma pins down the connection between invariant events and invariant measurements. Let I denote the class of all invariant events and, as in the previous chapter, let M(I) denote the class of all measurements (B-measurable functions) that are measurable with respect to I. The following lemma shows that M(I) is exactly the class of all invariant measurements.

Lemma 7.7. Let I denote the class of all invariant events. Then I is a σ field and M(I) is the class of all invariant measurements. In addition, I = σ (M(I)) and hence I is the σ -field generated by the class of all invariant measurements. Proof. That I is a σ -field follows immediately since inverse images (under T ) preserve set theoretic operations. For example, if F and G are invariant, then T −1 (F ∪ G) = T −1 F ∪ T −1 G = F ∪ G and hence the union is also invariant. The class M(I) contains all indicator functions of invariant P events, hence all simple invariant functions. For example, if f (ω) = bi 1Fi (ω) is a simple function in this class, then its inverse images must be invariant, and hence f −1 (bi ) = Fi ∈ I is an invariant event. Thus f must be an invariant simple function. Taking limits in the usual way then implies that every function in M(I) is invariant. Conversely, suppose that f is a measurable invariant function. Then for any Borel set G, T −1 f −1 (G) = (f T )−1 (G) = f −1 (G), and hence the inverse images of all Borel sets under f are invariant and hence in I. Thus measurable invariant function must be in M(I). The remainder of the lemma follows from Lemma 6.2. 2

Tail Events and Measurements The sub-σ -field of invariant events plays an important role when studying limiting averages. There are other sub-σ -fields that likewise play an important role in studying asymptotic behavior, especially when dealing with one-sided processes or noninvertible transformations. Given a dynamical system (Ω, B, m, T ), consider a random process {Xn } constructed in the usual way from a random variable X0 and the transformation T as Xn (ω) = X0 (T n ω). Define F0 = σ (X0 , X1 , X2 , . . .),

(7.24)

the sub-σ -field generated by the collection or random variables with nonnegative time indexes. If we are dealing with the Kolmogorov representation of a one-sided random process (noninvertible shift), then this is

212

7 Ergodic Properties

simply the full σ -field for the process {Xn }, F0 = B. Define the sub-σ fields Fn = T −n F0 = σ (Xn , Xn+1 , Xn+2 , . . .); n = 0, 1, . . . ∞ \ Fn . F∞ =

(7.25) (7.26)

n=0

F∞ is called the tail σ -field of the process {Xn } if F0 has the form of (7.24). The tail σ -field F∞ goes by many other names, including the remote σ -field , the right tail sub-σ -field , and the future tail sub-σ -field Intuitively, it contains all of those events that can be determined by looking at future outcomes arbitrarily far into the future; that is, membership in a tail event cannot be determined by any of the individual Xn , only by their aggregate behavior in the distant future. Strictly speaking, we should always refer to the relevant process {Xn } when referring to a tail σ -field, but for the present discussion we will simply assume that we are dealing with a directly-given representation of the process {Xn } and that (7.24) holds. If T is invertible, one can also define a left tail sub-σ -field by replacing T −1 by T — which considers events depending only on the remote past. The left tail sub-σ -field is also called the past tail sub-σ -field. We shall concentrate on the right or future tail. In the case of a directly-given one-sided random process {Xn }, F0 = B and hence Fn = T −n B ∞ \ T −n B. F∞ = n=0

If F ∈ F0 is invariant, then T −n F = F ∈ T −n Fn for all n and hence F ∈ F∞ , so that invariant events are tail events and I ⊂ F∞ . This argument does not hold in the directly-given two-sided case, however, where F0 ≠ B and T −n B = B. In the one-sided case, this fact has an interesting and simple implication. The process {Xn } is ergodic if and only if I, the σ field of invariant events, is trivial, that is, if all events in this σ -field have probability 0 or 1. A process {Xn } is said to satisfy the 0-1 law and to be a Kolmogorov process or K-process if it has the property that all events in F∞ are trivial, that is, if all tail events have probability 0 or 1. See, e.g., [3, 103, 77]. Thus in the two-sided case at least, a K-process must be ergodic. A dynamical system (Ω, B, m, T ) is said to be K or Kolmogorov if all random processes defined on it in the usual way are K-processes; that is, for any random variable X0 defined on (Ω, B, m) the random process {Xn } defined by Xn (ω) = X(T n ω) is a K-process.

7.3 Asymptotically Mean Stationary Processes

213

In the two-sided case, things are more complicated. Invariant sets might not be tail events, but it can be shown that if the measure is stationary, then invariant sets differ from tail events by null sets (see, e.g., [62], p. 27) and hence a stationary K-process is ergodic in general. A measurement f is called a tail function if it is measurable with respect to the tail σ -field. Intuitively, a measurement f is a tail function only if we can determine its value by looking at the outputs Xn , Xn+1 , . . . for arbitrarily large values of n.

Asymptotic Domination by a Stationary Measure We have seen that if T is invertible as in a two-sided directly given random process and if m is AMS with stationary mean m, then m  m, and hence any event having 0 probability under m also has 0 probability under m. In the one-sided case this is not in general true, although we have seen that it is true for invariant events. We shall shortly see that it is also true for tail events. First, however, we need a basic property of asymptotic dominance. If m  m, the Radon-Nikodym theorem implies that R m(F ) = F (dm/dm)dm. The following lemma provides an asymptotic version of this relation for asymptotically dominated measures. Lemma 7.8. Given measures m and η, let (mT −n )a denote the absolutely continuous part of mT −n with respect to η as in the Lebesgue decomposition theorem (Theorem 6.3) and define the Radon-Nikodym derivative fn =

d(mT −n )a ; n = 1, 2, . . . . dη

If η asymptotically dominates m then lim [sup |m(T −n F ) −

n→∞ F ∈B

Z F

fn dη|] = 0.

Proof. From the Lebesgue decomposition theorem and the Radon-Nikodym theorem (Theorems 6.3 and 6.2) for each n = 1, 2, . . . there exists a Bn ∈ B such that Z fn dη; F ∈ B, (7.27) mT −n (F ) = mT −n (F ∩ Bn ) + F

with η(Bn ) = 0. Define B = sumption

S∞

n=0

0 ≤ mT −n (F ) −

Z F

Bn . Then η(B) = 0, and hence by asfn dη = mT −n (F ∩ Bn )

≤ mT −n (F ∩ B) ≤ mT −n (B) → 0. n→∞

214

7 Ergodic Properties

Since the bound is uniform over F , the lemma is proved.

2

Corollary 7.6. Suppose that η is stationary and asymptotically dominates m (e.g., m is AMS and η is the stationary mean), then if F is a tail event (F ∈ F∞ ) and η(F ) = 0, then also m(F ) = 0. Proof. First observe that if T is invertible, the result follows from the fact that asymptotic dominance implies dominance. Hence assume that T is not invertible and that we are dealing with a one-sided process {Xn } and hence F∞ = ∩T −n B. Let F ∈ F∞ have η probability 0. Find {Fn } so that T −n Fn = F , n = 1, 2, . . .. (This is possible since F ∈ F∞ means that also F ∈ T −n B for all n and hence there must be an Fn ∈ B such that R −n F = T F .) From the lemma, m(F ) − f nR Fn n dη → 0. Since η is stationary, R n f dη = f T dη = 0, hence m(F ) = 0. 2 Fn n F n Corollary 7.7. Let µ and η be probability measures on (Ω, B) with η stationary. Then the following are equivalent: (a) (b) (c)

η asymptotically dominates µ. If F ∈ B is invariant and η(F ) = 0, then also µ(F ) = 0. If F ∈ F∞ and η(F ) = 0, then also µ(F ) = 0.

Proof. The previous corollary shows that (a) => (c). (c) => (b) immediately from the fact that invariant events are also tail events. To prove that (b) => (a), assume that η(F ) = 0. The set G = lim supn→∞ T −n F of (7.23) is invariant and has η measure 0. Thus as in the proof of Corollary 7.5 lim sup m(T −n F ) ≤ m(lim sup T −n F ) = 0. n→∞

n→∞

2 Corollary 7.8. Let µ and η be probability measures on (Ω, B). If η is stationary and has ergodic properties with respect to a measurement f and if η asymptotically dominates µ, then also µ has ergodic properties with respect to f . Proof. The set on which {x : limn→∞ < f >n exists} is invariant and hence if its complement (which is also invariant) has measure 0 under η, it also has measure zero under m from the previous corollary. 2

Exercises 1. Is the event lim inf T −n F = n→∞

∞ \ ∞ [

T −k F

n=0 k=n

= {x : x ∈ T −n F for all but a finite number of n}

7.4 Recurrence

215

invariant? Is it a tail event? 2. Show that given an event F and  > 0, the event {x : |

n−1 1 X f (T i x)| ≤  i.o.} n i=0

is invariant. 3. Suppose that m is stationary and let G be an event for which T −1 G ⊂ G. Show that m(G − T −1 G) = 0 and that there is an invariant event G with m(G∆G) = 0. 4. Is an invariant event a tail event? Is a tail event an invariant event? Is Fi.o. of (7.23) a tail event? 5. Suppose that m is the distribution of a stationary process and that G is an event with nonzero probability. Show that if G is invariant, then the conditional distribution mG defined by mG (F ) = m(F |G) is also stationary. Show that even if G is not invariant, mG is AMS. 6. Suppose that T is an invertible transformation (e.g., the shift in a twosided process) and that m is AMS with respect to T . Show that m(F ) > 0 implies that m(T n F ) > 0 i.o. (infinitely often).

7.4 Recurrence We have seen that AMS measures are closely related to stationary measures. In particular, an AMS measure is either dominated by a stationary measure or asymptotically dominated by a stationary measure. Since the simpler and stronger nonasymptotic form of domination is much easier to work with, the question arises as to conditions under which an AMS system will be dominated by a stationary system. We have already seen that this is the case if the transformation T is invertible, e.g., the system is a two-sided random process with shift T . In this section we provide a more general set of conditions that relates to what is sometimes called the most basic ergodic property of a dynamical system, that of recurrence. The original recurrence theorem goes back to Poincare [85]. We briefly describe it as it motivates some related definitions and results. Given a dynamical system (Ω, B, m, T ), let G ∈ B. A point x ∈ G is said to be recurrent with respect to G if there is a finite n = nG (x) such that T nG (x) ∈ G, that is, if the point in G returns to G after some finite number of shifts. An event G having positive probability is said to be recurrent if almost every point in G is recurrent with respect to G. In order to have a useful

216

7 Ergodic Properties

definition for all events (with or without positive probability), we state it in a negative fashion: Define G∗ =

∞ [

T −n G,

(7.28)

n=1

the set of all sequences that land in G after one or more shifts. Then B = G − G∗

(7.29)

is the collection of all points in G that never return to G, the bad or nonrecurrent points. We now define an event G to be recurrent if m(G − G∗ ) = 0. The dynamical system will be called recurrent if every event is a recurrent event. Observe that if the system is recurrent, then for all events G m(G ∩ G∗ ) = m(G) − m(G ∩ G∗c ) = m(G) − m(G − G∗ ) = m(G).

(7.30)

Thus if m(G) > 0, then the conditional probability that the point will eventually return to G is m(G∗ |G) = m(G ∩ G∗ )/m(G) = 1. Before stating and proving Poincaré recurrence theorem, we make two observations about the preceding sets. First observe that if x ∈ B, then we cannot have T k x ∈ G for any k ≥ 1. Since B is a subset of G, this means that we cannot have T k x ∈ B for any k ≥ 1, and hence the events B and T −k B must be pairwise disjoint for k = 1, 2, . . .; that is, B ∩ T −k B = ∅; k = 1, 2, . . . If these sets are empty, then so must be their inverse images with respect to T n for any positive integer n, that is, ∅ = T −n (B ∩ T −k B) = (T −n B) ∩ (T −(n+k) B) for all integers n and k. This in turn implies that T −k B ∩ T −j B = ∅; k, j = 1, 2, . . . , k 6= j;

(7.31)

that is, the sets T −n B are pairwise disjoint. We shall call any set W with the property that T −n W are pairwise disjoint a wandering set. Thus (7.31) states that the collection G − G∗ of nonrecurrent sequences is a wandering set. As a final property, observe that G∗ as defined in (7.28) has a simple property when it is shifted:

7.4 Recurrence

217

T −1 G∗ =

∞ [

∞ [

T −n−1 G =

n=1

T −n G ⊂ G∗ ,

n=2

and hence T −1 G∗ ⊂ G∗ .

(7.32)

∗

Thus G has the property that it is shrunk or compressed by the inverse transformation. Repeating this shows that T −k G∗ =

∞ [

T −n G ⊂ T −(k+1) G∗ .

(7.33)

n=k+1

We are now prepared to state and prove what is often considered the first ergodic theorem. Theorem 7.2. (The Poincaré Recurrence Theorem) If the dynamical system is stationary, then it is recurrent. Proof. We need to show that for any event G, the event B = G − G∗ has 0 measure. We have seen, however, that B is a wandering set and hence its iterates are disjoint. This means from countable additivity of probability that ∞ ∞ [ X m( T −n B) = m(T −n B). n=0

n=0

Since the measure is stationary, all of the terms of the sum are the same. They then must all be 0 or the sum would blow up, violating the fact that 2 the left-hand side is less than 1. Although such a recurrence theorem can be viewed as a primitive ergodic property, unlike the usual ergodic properties, there is no indication of the relative frequency with which the point will return to the target set. We shall shortly see that recurrence at least guarantees infinite recurrence in the sense that the point will return infinitely often, but again there is no notion of the fraction of time spent in the target set. We wish next to explore the recurrence property for AMS systems. We shall make use of several of the ideas developed above. We begin with several definitions regarding the given dynamical system. Recall for comparison that the system is stationary if m(T −1 G) = m(G) for all events G. The system is said to be incompressible if every event G for which T −1 G ⊂ G has the property that m(G − T −1 G) = 0. Note the resemblance to stationarity: For all such G m(G) = m(T −1 G). The system is said to be conservative if all wandering sets have 0 measure. As a final definition, we modify the definition of recurrent. Instead of the requirement that a recurrent system have the property that if an event has nonzero probability, then points in that event must return to that event with probability 1, we can strengthen the requirement to require that the points return to the event infinitely often with probability 1. Define as in (7.23) the set

218

7 Ergodic Properties

Gi.o. =

∞ [ ∞ \

T −k G

n=0 k=n

of all points that are in G infinitely often. The dynamical system will be said to be infinitely recurrent if for all events G m(G − Gi.o. ) = 0. Note from (7.33) that ∞ [

T −k G = T −(n+1) G∗ ,

k=n

and hence since the T −n G∗ are decreasing Gi.o. =

∞ \

T −(n+1) G∗ =

n=0

∞ \

T −n G∗ .

(7.34)

n=0

The following theorem is due to Halmos [44], and the proof largely follows Wright [114], Petersen [84], or Krengel [62]. We do not, however, assume as they do that the system is nonsingular in the sense that mT −1  m. When this assumption is made, T is said to be null preserving because shifts of the measure inherit the zero probability sets of the original measure. Theorem 7.3. Suppose that (Ω, B, m, T ) is a dynamical system. Then the following statements are equivalent: (a) (b) (c) (d)

T T T T

is infinitely recurrent. is recurrent. is conservative. is incompressible.

Proof. (a)⇒ (b): In words: if we must return an infinite number of times, then we must obviously return at least once. Alternatively, application of (7.34) implies that G − Gi.o. = G −

∞ \

T −n G∗

(7.35)

n=0

so that G − G∗ ⊂ G − Gi.o. and hence the system is recurrent if it is infinitely recurrent. (b)⇒ (c): Suppose that W is a wandering set. From the recurrence property W − W ∗ must have 0 measure. Since W is wandering, W ∩ W ∗ = ∅ and hence m(W ) = m(W − W ∗ ) = 0 and the system is conservative. (c)⇒ (d): Consider an event G with T −1 G ⊂ G. Then G∗ = T −1 G since all the remaining terms in the union defining G∗ are further subsets. Thus

7.4 Recurrence

219

G − T −1 G = G − G∗ . We have seen, however, that G − G∗ is a wandering set and hence has 0 measure since the system is conservative. (b) and (d)⇒ a: From (7.35) we need to prove that ∞ \

m(G − Gi.o. ) = m(G −

T −n G∗ ) = 0.

n=0

Toward this end we use the fact that the T −n G∗ are decreasing to decompose G∗ as G∗ = (

∞ \

∞ [

T −n G∗ )

n=1

(T −n G∗ − T −(n+1) G∗ ).

n=0

T∞ This breakup can be thought of as a “core” n=1 T −n G∗ containing the sequences that shift into G∗ for all shifts combined with a union of an infinity of “doughnuts,” where the nth doughnut contains those sequences that when shifted n times land in G∗ , but never arrive in G∗ if shifted more than n times. The core and all of the doughnuts are disjoint and together yield all the sequences in G∗ . Because the core and doughnuts are disjoint, ∞ \

T −n G∗ = G∗ −

n=1

∞ [

(T −n G∗ − T −(n+1) G∗ )

n=0

so that G−

∞ \

T

−n

∗

∗

G = (G − G ) ∪ (G ∩ (

n=1

∞ [

(T −n G∗ − T −(n+1) G∗ )).

(7.36)

n=0

Thus since the system is recurrent we know by (7.36) and the union bound that m(G −

∞ \

T −n G∗ ) ≤ m(G − G∗ ) + m(G ∩ (

n=1

∞ [

(T −n G∗ − T −(n+1) G∗ ))

n=0 ∞ [

≤ m(G − G∗ ) + m(

(T −n G∗ − T −(n+1) G∗ ))

n=0

= m(G − G∗ ) +

∞ X

m(T −n G∗ − T −(n+1) G∗ )),

n=0

using the disjointness of the doughnuts. (Actually, the union bound and an inequality would suffice here.) Since T −1 G∗ ⊂ G∗ , it is also true that T −1 T −n G∗ = T −n T −1 G∗ ⊂ T −n G∗ and hence since the system is incompressible, all the terms in the previous sum are zero.

220

7 Ergodic Properties

(d) ⇒ (b) Given an event G, let F = G ∪ G∗ . Then T −1 F = G∗ ⊂ F and hence m(F − T −1 F ) = 0 by incompressibility. But F − T −1 F = (G ∪ G∗ ) − G∗ = G − G∗ and hence m(G − G∗ ) = 0 implying recurrence. We have now shown that (a)⇒ (b)⇒ (c)⇒ (d), that (b) and (d) together imply (a), and that (d) ⇒ (d) and hence (d)⇒ (a), which completes the proof that any one of the conditions implies the other three. 2 Thus, for example, Theorem 7.2 implies that a stationary system has all the preceding properties. Our goal, however, is to characterize the properties for an AMS system. The following theorem characterizes recurrent AMS processes: An AMS process is recurrent if and only if it is dominated by its stationary mean. Theorem 7.4. Given an AMS dynamical system (Ω, B, m, T ) with stationary mean m, then m  m if and only the system is recurrent. Proof. If m  m, then the system is obviously recurrent since m is from the previous two theorems and m inherits all of the zero probability sets from m. To prove the converse implication, suppose the contrary. Say that there is a set G for which m(G) > 0 and m(G) = 0 and that the system is recurrent. From Theorem 7.3 this implies that the system is also infinitely recurrent, and hence m(G − Gi.o. ) = 0 and therefore m(G) = m(Gi.o. ) > 0. We have seen, however, that Gi.o is an invariant set in Section 7.3 and hence from Lemma 7.5, m(Gi.o. ) > 0. Since m is stationary and gives probability 0 to G, m(T −k G) = 0 for all k, and hence m(Gi.o. ) = m(

∞ [ ∞ \ n=0 k=n

T −n G) ≤ m(

∞ [ k=0

T −n G) ≤

∞ X

m(G) = 0,

k=0

yielding a contradiction and proving the theorem. 2 If the transformation T is invertible; e.g., the two-sided shift, then we have seen that m  m and hence the system is recurrent. When the shift is noninvertible, the theorem proves that domination by a stationary measure is equivalent to recurrence. Thus, in particular, not all AMS processes need be recurrent.

7.5 Asymptotic Mean Expectations

221

Exercises 1. Show that if an AMS dynamical system is nonsingular (mT −1  m), then m and the stationary mean m will be equivalent (have the same probability 0 events) if and only if the system is recurrent. 2. Show that if m is nonsingular (null preserving) with respect to T , then incompressibility of T implies that it is conservative and hence that all four conditions of Theorem 7.3 are equivalent.

7.5 Asymptotic Mean Expectations We have seen that the existence of ergodic properties implies that limits of sampleP means of expectations exist; e.g., if < f >n converges, then so n−1 does n−1 i=0 Em f T i . In this section we relate these limits to expectations with respect to the stationary mean m. Lemma 7.3 showed that the arithmetic mean of the expected values of transformed measurements converged. The following lemma shows that the limit is given by the expected value of the original measurement under the stationary mean. Lemma 7.9. Let m be an AMS random process with stationary mean m. If a measurement f is bounded, then lim Em (< f >n ) = lim n−1

n→∞

n→∞

n−1 X

Em (f T i )

i=0

= lim Emn f = Em (f ). n→∞

(7.37)

Proof. The result is immediately true for indicator functions from the definition of AMS. It then follows for simple functions from the linearity of limits. In general let qk denote the quantizer sequence of Lemma 5.3 and observe that for each k we have |Em (< f >n ) − Em (f )| ≤ |Em (< f >n ) − Em (< qk (f ) >n )| + |Em (< qk >n ) − Em (qk (f ))| + |Em (qk (f )) − Em (f ) | . Choose k larger that the maximum of the bounded function f . The middle term goes to zero from the simple function result. The rightmost term is bound above by 2−k since we are integrating a function that is everywhere less than 2−k . Similarly the first term on the left is bound above by n−1 X n−1 Em |f T i − qk (f T i )| ≤ 2−k , i=0

222

7 Ergodic Properties

where we have used the fact that qm (f )(T i x) = qm (f (T i x)). Since k can be made arbitrarily large and hence 2 × 2−k arbitrarily small, the lemma is proved. 2 Thus the stationary mean provides the limiting average expectation of the transformed measurements. Combining the previous lemma with Lemma 7.3 immediately yields the following corollary. Corollary 7.9. Let m be an AMS measure with stationary mean m. If m has ergodic properties with respect to a bounded measurement f and if the limiting sample average is < f >, then Em < f >= Em f .

(7.38)

Thus the expectation of the measurement under the stationary mean gives the expectation of the limiting sample average under the original measure. If we also have uniform integrability, we can similarly extend Corollary 7.2. Lemma 7.10. Let m be an AMS random process with stationary mean m that has the ergodic property with respect to a measurement f . If the < f >n are uniformly m-integrable, then f is also m integrable and (7.37) and (7.38) hold. Proof. From Corollary 7.2 lim Em (< f >n ) = Em f .

n→∞

In particular, the right-hand integral exists and is finite. From Corollary 7.1 the limit < f > is invariant. If the system is AMS, the invariance of < f > and Lemma 7.5 imply that Em f = Em f . Since m is stationary Lemma 5.23 implies that the sequence < f >n is uniformly integrable with respect to m. Thus from Lemma 5.20 and the convergence of < f >n on a set of m probability one (Corollary 7.4), the right-hand integral is equal to lim Em (n−1

n→∞

n−1 X

f T i ).

i=0

From the linearity of expectation, the stationarity of m, and Lemma 5.22, for each value of n the preceding expectation is simply Em f . 2

7.6 Limiting Sample Averages

223

Exercises 1. Show that if m is AMS, then for any positive integer N and all i = 0, 1, . . . , N − 1, then the following limits exist: n−1 1 X m(T −i−jN F ). n→∞ n j=0

lim

2. Define the preceding limit to be mi . Show that the mi are N-stationary and that if m is the stationary mean of m, then m(F ) =

N−1 1 X mi (F ). N i=0

7.6 Limiting Sample Averages In this section limiting sample averages are related to conditional expectations. The basis for this development is the result of Section 7.1 that limiting sample averages are invariant functions and hence are measurable with respect to I, the σ -field of all invariant events. We begin with limiting relative frequencies, that is, the limiting sample averages of indicator functions. Say we have a dynamical system (Ω, B, m, T ) that has ergodic properties with respect to the class of all indicator functions and which is therefore AMS. Fix an event G and let < 1G > denote the corresponding limiting relative frequency, that is < 1G >n →< 1G > as n → ∞ a.e. This limit is invariant and therefore in M(I) and is measurable with respect to I. Fix an event F ∈ I and observe that from Lemma 7.5 Z Z 1G dm = 1G dm. (7.39) F

F

For invariant F , 1F is an invariant function and hence < 1F f >n =

n−1 n−1 1 X 1 X (1F f )T i = 1F f T i = 1F < f >n n i=0 n i=0

and hence if < f >n →< f > then < 1F f >= 1F < f > . Thus from Corollary 7.9 we have for invariant F that

(7.40)

224

7 Ergodic Properties

Z F

< 1G > dm = Em < 1F 1G >= Em 1F 1G Z = < 1G > dm = m(F ∩ G), F

which with (7.39) implies that Z < 1G > dm = m(F ∩ G), all F ∈ I.

(7.41)

F

Eq. (7.41) together with the I-measurability of the limiting sample average yield (6.23) and (6.24) and thereby show that the limiting sample average is a version of the conditional probability m(G|I) of G given the σ -field I or, equivalently, given the class of all invariant measurements! We formalize this in the following theorem. Theorem 7.5. Given a dynamical system (Ω, B, m, T ) let I denote the class of all T -invariant events. If the system is AMS and if it has ergodic properties with respect to the indicator function 1G of an event G (e.g., if the system has ergodic properties with respect to all indicator functions of events), then with probability one under m and m < 1G >n = n−1

n−1 X i=0

1G T i → m(G|I), n→∞

In other words, < 1G >= m(G|I).

(7.42)

It is important to note that the conditional probability is defined by using the stationary mean measure m and not the original probability measure m. Intuitively, this is because it is the stationary mean that describes asymptotic events. Eq. (7.42) shows that the limiting sample average of the indicator function of an event, that is, the limiting relative frequency of the event, satisfies the descriptive definition of the conditional probability of the event given the σ -field of invariant events. An interpretation of this result is that if we wish to guess the probability of an event G based on the knowledge of the occurrence or nonoccurrence of all invariant events or, equivalently, on the outcomes of all invariant measurements, then our guess is given by one particular invariant measurement: the relative frequency of the event in question. A natural question is whether Theorem 7.5 holds for more general functions than indicator functions; that is, if a general sample average < f >n converges, must the limit be the conditional expectation? The following lemma shows that this is true in the case of bounded measurements.

7.6 Limiting Sample Averages

225

Lemma 7.11. Given the assumptions of Theorem 7.5, if the system has ergodic properties with respect to a bounded measurement f , then with probability one under m and m < f >n = n−1

n−1 X i=0

f T i → Em (f |I). n→∞

In other words, f = Em (f |I). Proof. Say that the limit of the left-hand side is an invariant function < f >. Apply Lemma 7.9 to the bounded measurement g = f 1F with an invariant event F and lim n−1

n→∞

n−1 XZ i=0

f T i dm = F

Z f dm. F

R From Lemma 7.3 the left-hand side is also given by F f dm. The equality of the two integrals for all invariant F and the invariance of the limiting sample average imply from Lemma 7.12 that < f > is a version of the conditional expectation of f given I. 2 Replacing Lemma 7.9 by Lemma 7.10 in the preceding proof gives part of the following generalization. Corollary 7.10. Given a dynamical system (Ω, B, m, T ) let I denote the class of all T -invariant events. If the system is AMS with stationary mean m and if it has ergodic properties with respect to a measurement f for which either (i) the < f >n are uniformly integrable with respect to m or (ii) f is m-integrable, then with probability one under m and m < f >n = n−1

n−1 X i=0

f T i → Em (f |I). n→∞

In other words, f = Em (f |I). Proof. Part (i) follows from the comment before the corollary. From Corollary 8.5, since m has ergodic properties with respect to f , then so does m and the limiting sample average < f > is the same (up to an almost everywhere equivalence under both measures). Since f is by assumption m-integrable and since m is stationary, then < f >n is uniformly m-integrable and hence application of (i) to m then proves that the given equality holds m-a.e.. The set on which the the equality holds is invariant, however, since both < f > and Em (f |I) are invariant. Hence the given equality also holds m-a.e. from Lemma 7.5. 2

226

7 Ergodic Properties

Exercises 1. Let m be AMS with stationary mean m. Let M denote the space of all square-integrable invariant measurements. Show that M is a closed linear subspace of L2 (m). If m has ergodic properties with respect to a bounded measurement f , show that f = PM (f ); that is, the limiting sample average is the projection of the original measurement onto the subspace of invariant measurements. From the projection theorem, this means that < f > is the minimum meansquared estimate of f given the outcome of all invariant measurements. 2. Suppose that f ∈ L2 (m) is a measurement and fn ∈ L2 (m) is a sequence of measurements for which ||fn − f || →n→∞ 0. Suppose also that m is stationary. Prove that n−1 1 X fi T i → < f >, n i=0 L2 (m)

where < f > is the limiting sample average of f .

7.7 Ergodicity A special case of ergodic properties of particular interest occurs when the limiting sample averages are constants instead of random variables, that is, when a sample average converges with probability one to a number known a priori rather than to a random variable. This will be the case, for example, if a system is such that all invariant measurements equal constants with probability one. If this is true for all measurements, then it is true for indicator functions of invariant events. Since such functions can assume only values of 0 or 1, for such a system m(F ) = 0 or 1 for all F such that T −1 F = F .

(7.43)

Conversely, if (7.43) holds every indicator function of an invariant event is either 1 or 0 with probability 1, and hence every simple invariant function is equal to a constant with probability one. Since every invariant measurement is the pointwise limit of invariant simple measurements (combine Lemma 5.3 and Lemma 7.7), every invariant measurement is also a constant with probability one (the ordinary limit of the constants

7.7 Ergodicity

227

equaling the invariant simple functions). Thus every invariant measurement will be constant with probability one if and only if (7.43) holds. Suppose next that a system possesses ergodic properties with respect to all indicator functions or, equivalently, with respect to all bounded measurements. Since the limiting sample averages of all indicator functions of invariant events (in particular) are constants with probability one, the indicator functions themselves are constant with probability one. Then, as previously, (7.43) follows. Thus we have shown that if a dynamical system possesses ergodic properties with respect to a class of functions containing the indicator functions, then the limiting sample averages are constant a.e. if and only if (7.43) holds. This motivates the following definition: A dynamical system (or the associated random process) is said to be ergodic with respect to a transformation T or T -ergodic or, if T is understood, simply ergodic, if every invariant event has probability 1 or 0. Another name for ergodic that is occasionally found in the mathematical literature is metrically transitive. Given the definition, we have proved the following result. Lemma 7.12. A dynamical system is ergodic if and only if all invariant measurements are constants with probability one. A necessary condition for a system to have ergodic properties with limiting sample averages being a.e. constant is that the system be ergodic. Lemma 7.13. If a dynamical system (Ω, B, m, T ) is AMS with stationary mean m, then (Ω, B, m, T ) is ergodic if and only if (Ω, B, m, T ) is.

2 Proof. The proof follows from the definition and Lemma 7.5. Coupling the definition and properties of ergodic systems with the previously developed ergodic properties of dynamical systems yields the following results. Lemma 7.14. If a dynamical system has ergodic properties with respect to a bounded measurement f and if the system is ergodic, then with probability one n−1 X lim < f >n = Em f = lim n−1 Em f T i . n→∞

n→∞

i=0

If more generally the < f >n are uniformly integrable but not necessarily bounded (and hence the convergence is also in L1 (m)), then the preceding still holds. If the system is AMS and either the < f >n are uniformly integrable with respect to m or f is m-integrable, then also with probability one lim < f >n = Em f . n→∞

In particular, the limiting relative frequencies are given by

228

7 Ergodic Properties

lim n−1

n→∞

n−1 X

1F T i = m(F ), all F ∈ B.

i=0

In addition, lim n−1

n−1 XZ

n→∞

i=0

G

f T i dm = (Em f )m(G), all G ∈ B,

where again the result holds for measurements for which the < f >n are uniformly integrable or f is m integrable. Letting f be the indicator function of an event F , then lim n−1

n→∞

n−1 X

m(T −i F ∩ G) = m(F )m(G) all F , G ∈ B.

(7.44)

i=0

Proof. If the limiting sample average is constant, then it must equal its own expectation. The remaining results then follow from the results of this chapter. 2 The lemma shows that (7.44) is necessary for a process with ergodic properties to be ergodic. It is also sufficient since if F is invariant, then from Lemma 7.5 m(F ) = m(F ) and hence (7.44) becomes with G = F m(T −i F ∩ G) = m(F ) = m(F ) = m(F ) = m(F )2 and hence m(F ) = 0 or 1. Thus we have proved the following corollary. Corollary 7.11. A necessary and sufficient condition for a system with ergodic properties for all bounded measurements to be ergodic is that (7.44) hold for all events F and G. Thus (7.44) thus provides a test for ergodicity. It can be viewed as a weak form of asymptotic independence. Happily we shall see shortly that the test need not be applied to all events, just to a generating field. Observe that if m is actually stationary and hence its own stationary mean, then (7.44) becomes lim n−1

n→∞

n−1 X

m(T −i F ∩ G) = m(F )m(G)

(7.45)

i=0

for all events G. The final result of the previous lemma yields a useful test for determining whether a system with ergodic properties is ergodic. Lemma 7.15. Suppose that a dynamical system (Ω, B, m, T ) has ergodic properties with respect to the indicator functions and hence is AMS with stationary mean m and suppose that B is generated by a field F , then m is ergodic if and only if

7.7 Ergodicity

229

lim n−1

n→∞

n−1 X

m(T −i F ∩ F ) = m(F )m(F ), all F ∈ F .

(7.46)

i=0

Proof. Necessity follows from the previous result. To show that the preceding formula is sufficient for ergodicity, first assume that the formula holds for all events F and let F be invariant. Then the left-hand side is simply m(F ) = m(F ) and the right-hand side is m(F )m(F ) = m(F )2 . But these two quantities can be equal only if m(F ) is 0 or 1, i.e., if it (and hence also m) is ergodic. We will be done if we can show that the preceding relation holds for all events given that it holds on a generating field. Toward this end fix  > 0. From Corollary 1.5 we can choose a field event F0 such that m(F ∆F0 ) ≤  and m(F ∆F0 ) ≤ . To see that we can choose a field event F0 that provides a good approximation to F simultaneously for both measures m and m, apply Corollary 1.5 to the mixture measure p = (m + m)/2 to obtain an F0 for which p(F ∆F0 ) ≤ /2. This implies that both m(F ∆F0 ) and m(F ∆F0 )) must be less than . From the triangle inequality

|

n−1 1 X m(T −i F ∩ F ) − m(F )m(F ) |≤ n i=0

|

|

n−1 n−1 1 X 1 X m(T −i F ∩ F ) − m(T −i F0 ∩ F0 ) | + n i=0 n i=0

n−1 1 X m(T −i F0 ∩F0 )−m(F0 )m(F0 ) | + | m(F0 )m(F0 )−m(F )m(F ) | . n i=0

The middle term on the right goes to 0 as n → ∞ by assumption. The rightmost term is bound above by 2. The leftmost term is bound above by n−1 1 X |m(T −i F ∩ F ) − m(T −i F0 ∩ F0 ) | . n i=0 Since for any events D, C |m(D) − m(C)| ≤ m(D∆C)

(7.47)

each term in the preceding sum is bound above by m((T −i F ∩F )∆(T −i F0 ∩ F0 )). Since for any events D, C, H m(D∆C) ≤ m(D∆H) + m(H∆C), each of the terms in the sum is bound above by m((T −i F ∩ F )∆(T −i F0 ∩ F )) + m((T −i F0 ∩ F )∆(T −i F0 ∩ F0 )) ≤ m(T −i (F ∆F0 )) + m(F ∆F0 ).

230

7 Ergodic Properties

Thus the remaining sum term is bound above by n−1 1 X m(T −i (F ∆F0 )) + m(F ∆F0 ) → m(F ∆F0 ) + m(F ∆F0 ) ≤ 2, n→∞ n i=0

2

which completes the proof.

7.8 Block Ergodic and Totally Ergodic Processes A random process is N-ergodic if it is ergodic with respect to T N , that is, if T −N F = F . An N-ergodic process is also ergodic since a T -invariant event is also T N -invariant and hence has probability zero or one. The converse is not true, however, an ergodic process need not be N-ergodic. This can cause difficulties when looking at properties of processes produced by block encoding an ergodic process, that is, processes resulting when successive N-tuples of the original process are operated on in an independent fashion to produce N-tuples of a new process. A process is said to be totally ergodic if it is N-ergodic.

Exercises 1. Suppose that system is N-ergodic and N-stationary. Show that the mixture N−1 1 X m(F ) = m(T −i F ) N i=0 is stationary and ergodic. 2. Suppose that a process is N-ergodic and N-stationary and has the ergodic property with respect to f . Show that Ef =

N−1 1 X E(f T i ). N i=0

3. If a process is stationary and ergodic and T is invertible, show that m(F ) > 0 implies that m(

∞ [

T i F ) = 1.

i=−∞

Show also that if m(F ) > 0 and m(G) > 0, then for some i m(F ∩ T i G) > 0. Show that, in fact, m(F ∩ T i G) > 0 infinitely often.

7.8 Block Ergodic and Totally Ergodic Processes

231

4. Suppose that (Ω, B, m) is a directly given stationary and ergodic source with shift T and that f : Ω → Λ is a measurable mapping of sequences in Ω into another sequence space Λ with shift S with the property that f (T x) = Sf (x). Such a mapping is said to be stationary or time invariant since shifting the input simply results in shifting the output. Intuitively, the mapping does not change with time. Such sequence mappings are induced by sliding-block codes and timeinvariant filters as considered in Section 2.6. Show that the induced process (with distribution mf −1 ) is also stationary and ergodic. 5. Show that if m has ergodic properties with respect to all indicator functions and it is AMS with stationary mean m, then n−1 n−1 n−1 1 X 1 X X m(T −i F ∩ F ) = lim 2 m(T −i F ∩ T −j F ). n→∞ n n→∞ n i=0 i=0 j=0

lim

6. Show that a process m is stationary and ergodic if and only if for all f , g ∈ L2 (m) we have n−1 1 X E(f gT i ) = (Ef )(Eg). n→∞ n i=0

lim

7. Show that the process is AMS with stationary mean m and ergodic if and only if for all bounded measurements f and g n−1 1 X Em (f gT i ) = (Em f )(Em g). n→∞ n i=0

lim

8. Suppose that m is a process and f ∈ L2 (m). Assume that the following hold for all integers k and j:

Em

Em f T k = Em f    (f T k )(f T j ) = Em (f )(f T |k−j| )



lim Em (f f T |k| ) = (Em f )2 .

k→∞

9. 10. 11. 12.

Prove that < f >n → Em f in L2 (m).P N−1 Consider the measures m and N −1 i=0 mT −i . Does ergodicity of either imply that of the other? Suppose that m is N-stationary and N-ergodic. Show that the measures mT −n have the same properties. Suppose an AMS process is not stationary but it is ergodic. Is it recurrent? Infinitely recurrent? Is a mixture of recurrent processes also recurrent?

232

7 Ergodic Properties

7.9 The Ergodic Decomposition In this section we find a characterization of the stationary mean of a process with bounded ergodic properties as a mixture of stationary and ergodic processes. Roughly speaking, if we view the outputs of a system with ergodic properties, then the relative frequencies and sample averages will be the same as if we were viewing an ergodic system, but we do not know a priori which one. If we view the limiting sample frequencies, however, we can determine the ergodic source being seen. We have seen that under the given assumptions, the limiting relative frequencies (the sample averages of the indicator functions) are conditional probabilities, < 1F >= m(F |I), where I is the sigma-field of invariant events. In addition, if the alphabet is standard, one can select a regular version of the conditional probability and hence using iterated expectation one can consider the stationary mean distribution m as a mixture of measures {m(·|I)(x); x ∈ Ω}. In this section we show that almost all of these measures are both stationary and ergodic, and hence µ can be written as a mixture of stationary and ergodic distributions. Intuitively, observing the asymptotic sample averages of system with ergodic properties is equivalent to observing the same measurements for a stationary and ergodic system that is an ergodic component of the stationary mean, but we may not know in advance which ergodic component is being observed. This result is known as the ergodic decomposition and it is due in various forms to Von Neumann [112], Rohlin [89], Kryloff and Bogoliouboff [63], and Oxtoby [80]. The development here most closely parallels that of Oxtoby. (See also Gray and Davisson [34] for the simple special case of discrete alphabet processes.) We here take the somewhat unusual step of proving the theorem first under the more general assumption that a process has bounded ergodic properties rather than that the process is stationary. The usual theorem will then follow for AMS systems from the ergodic theorem of the next chapter, which includes stationary systems as a special case. Our point is that the key parts of the ergodic decomposition are a direct result of the ergodic properties of sequences with respect to indicator functions rather than of the ergodic theorem. Although we could begin with Theorem 7.5, which characterizes the limiting relative frequencies as conditional probabilities, and then use Theorem 6.4 to choose a regular version of the conditional probability, it is more instructive to proceed in a direct fashion by assuming bounded ergodic properties. In addition, this yields a stronger result wherein the decomposition is determined by the properties of the points themselves and not the underlying measure; that is, the same decomposition works for all measures with bounded ergodic properties.

7.9 The Ergodic Decomposition

233

We begin by gathering together the necessary definitions. Let (Ω, B, m, T ) be a dynamical system and assume that (Ω, B) is a standard measurable space with a countable generating set F . For each event F define the invariant set G(F ) = {x : limn→∞ < 1F >n (x) exists} and denote the invariant limit by < 1F >. Define the set \ G(F ) = G(F ), F ∈F

the set of points x for which the limiting relative frequencies of all generating events exist. We shall call G(F ) the set of generic points or frequency-typical points Thus for any x ∈ G(F ) lim

n→∞

n−1 X

1F (T i x) = 1F (x); all F ∈ F .

i=0

Observe that membership in G(F ) is determined strictly by properties of the points and not by an underlying measure. If m has ergodic properties with respect to the class of all indicator functions (and is therefore necessarily AMS) then clearly m(G(F )) = 1 for any event F and hence we also have for the countable intersection that m(G(F )) = 1. If x ∈ G(F ), then the set function Px defined by Px (F ) = 1F (x) ; F ∈ F is nonnegative, normalized, and finitely additive on F from the properties of sample averages. Since the space is standard, this set function extends to a probability measure, which we shall also denote by Px , on (Ω, B). Thus we easily obtain that every generic point induces a probability measure Px and hence also a dynamical system (Ω, B, Px , T ) via its convergent relative frequencies. For x not in G(F ) we can simply fix Px as some fixed arbitrary stationary and ergodic measure p ∗ . We now confine interest to systems with bounded ergodic properties so that the set of generic sequences will have probability one. Then Px (F ) =< 1F > (x) almost everywhere and Px (F ) = m(F |I)(x) for all x in a set of m measure one. Since this is true for any event F , we can define the countable intersection H = {x : Px (F ) = m(F |I) all F ∈ F } and infer that m(H) = 1. Thus with probability one under m the two measures Px and m(·|I)(x) agree on F and hence are the same. Thus from the properties of conditional probability and conditional expectation

234

7 Ergodic Properties

Z m(F ∩ D) = D

Px (F )dm, all D ∈ I,

and if f is in L1 (m), then Z Z Z f dm = ( f (y) dPx (y))dm, all D ∈ I. D

(7.48)

(7.49)

D

Thus choosing D = Ω we can interpret (7.48) as stating that the stationary mean m is a mixture of the probability measures Px and (7.49) as stating that the expectation of measurements with respect to the stationary mean can be computed (using iterated expectation) in two steps: first find the expectations with respect to the components Px for all x, and second take the average. Such mixture and average representations will be particularly useful because, as we shall show, the component measures are both stationary and ergodic with probability one. For focus and motivation, we next state the principal result of this section — the ergodic decomposition theorem — which makes this notion rigorous in a useful form. The remainder of the section is then devoted to the proof of this theorem via a sequence of lemmas that are of some interest in their own right. Theorem 7.6. The Ergodic Decomposition of Systems with Bounded Ergodic Properties Given a standard measurable space (Ω, B), let F denote a countable generating field. Given a transformation T : Ω → Ω define the set of generic sequences G(F ) = {x : lim < 1F >n (x) exists all F ∈ F }. n→∞

For each x ∈ G(F ) let Px denote the measure induced by the limiting ˆ ⊂ G(F ) by E ˆ = {x : Px is stationary relative frequencies. Define the set E and ergodic}. Define the stationary and ergodic measures px by px = Px ˆ and otherwise px = p ∗ , some fixed stationary and ergodic for x ∈ E measure. (Note that so far all definitions involve properties of the points only and not of any underlying measure.) We shall refer to the collection of measures {px ; x ∈ Ω} as the ergodic decomposition of the triple (Ω, B, T ). We have then that ˆ is an invariant measurable set, (a) E (b) px (F ) = pT x (F ) for all points x ∈ Ω and events F ∈ B, (c) If m has bounded ergodic properties and m is its stationary mean, then ˆ = m(E) ˆ = 1, m(E) (7.50) (d)

for any F ∈ B Z m(F ) =

Z px (F )dm(x) =

px (F )dm(x),

(7.51)

7.9 The Ergodic Decomposition

(e)

(g)

235

and if f ∈ L1 (m), then so is f dpx and Z Z Z  f dm = f dpx dm(x). R

(7.52)

In addition, given any f ∈ L1 (m), then lim n−1

n→∞

n−1 X

f (T i x) = Epx f , m − a.e. and m − a.e.

(7.53)

i=0

Comment: The final result is extremely important for both mathematics and intuition. Say that a system (Ω, B, m, T ) has bounded ergodic properties and that f is integrable with respect to the stationary mean m; e.g., f is measurable and bounded and hence integrable with respect to any measure. Then with probability one a point will be produced such that the limiting sample average exists and equals the expectation of the measurement with respect to the stationary and ergodic measure induced by the point. Thus if a system has bounded ergodic properties, then the limiting sample averages will behave as if they were in fact produced by a stationary and ergodic system. These ergodic components are determined by the points themselves and not the underlying measure; that is, the same collection works for all measures with bounded ergodic properties. Proof. We begin the proof by observing that the definition of Px implies that PT x (F ) = 1F (T x) = 1F (x) = Px (F ); all F ∈ F ; all x ∈ G(F )

(7.54)

since limiting relative frequencies are invariant. Since PT x and Px agree on generating field events, however, they must be identical. Since we will ˆ select the px to be the Px on a subset of G(F ), this proves (b) for x ∈ E. Next observe that for any x ∈ G(F ) and F ∈ F Px (F ) = 1F (x) = lim

n→∞

= lim

n→∞

n−1 X

n−1 X

1F (T i x)

i=0

1T −1 F (T i x) = 1T −1 F (x) = Px (T −1 F ).

i=0

Thus for x ∈ G(F ) the measure Px looks stationary, at least on generating events. The following lemma shows that this is enough to infer that the Px are indeed stationary.

236

7 Ergodic Properties

Lemma 7.16. A measure m is stationary on (Ω, σ (F )) if and only if m(T −1 F ) = m(F ) for all F ∈ F . Proof. Both m and mT −1 are measures on the same space, and hence they are equal if and only if they agree on a generating field. 2 Note in passing that no similar countable description of an AMS meaPn−1 sure is known, that is, if limn→∞ n−1 i=0 m(T −i F ) converges for all F in a generating field, it does not follow that the measure is AMS. ˆ denote the subset of G(F ) consisting of all of the As before let E points x for which Px is ergodic. From Lemma 7.15, the definition of Px , ˆ as and the stationarity of the Px we can write E ˆ = {x : x ∈ G(F ); lim n−1 E n→∞

n−1 X

Px (T −i F ∩ F )

i=0

2

= Px (F ) , all F ∈ F }. ˆ where Since the probabilities Px are measurable functions, the set E the given limit exists and has the given value is also measurable. Since Px (F ) = PT x (F ), the set is also invariant. This proves (a) of the theorem. ˆ For x 6∈ E ˆ we have that px = p ∗ . We have already proved (b) for x ∈ E. ˆ is invariant, we also know that T x 6∈ E ˆ and hence also pT x = p ∗ Since E so that pT x = px . We now consider a measure m with bounded ergodic properties so that the system is AMS. Lemma 7.15 provided a countable description of ergodicity that was useful to prove that the collection of all points inducing ergodic measures is an event, the following lemma provides another countable description of ergodicity that is useful for computing ˆ the probability of the set E. Lemma 7.17. Let (Ω, σ (F ), m, T ) be an AMS dynamical system with stationary mean m. Define the sets n−1 1 X 1F (T i x) = m(F )}; F ∈ F . n→∞ n i=0

G(F , m) = {x : lim

Thus G(F , m) is the set of sequences for which the limiting relative frequency exists for all events F in the generating field and equals the probability of the event under the stationary mean. Define also \ G(F , m) = G(F , m). F ∈F

A necessary and sufficient condition for m to be ergodic is that m(G(F, m)) = 1.

7.9 The Ergodic Decomposition

237

Comment: The lemma states that an AMS measure m is ergodic if and only if it places all of its probability on the set of generic points that induce m through their relative frequencies. Proof. If m is ergodic then < 1F >= m(F ) m-a.e. for all F and the result follows immediately. Conversely, if m(G(F , m)) = 1 then also m(G(F , m)) = 1 since the set is invariant. Thus integrating to the limit using the dominated convergence theorem yields lim n−1

n→∞

n−1 X

m(T −i F ∩ F ) =

Z G(F ,m)

i=0

1F ( lim n−1 n→∞

n−1 X

1F T i )dm

i=0

2

= m(F ) , all F ∈ F , which with Lemma 7.15 proves that m is ergodic, and hence so is m from Lemma 7.13. 2 ˆ = 1, then the remainder Proof of the Theorem. If we can show that m(E) of the theorem will follow from (7.48) and (7.49) and the fact that px = m(·|I)(x) m−a.e. From the previous lemma, this will be the case if with px probability one 1F = px (F ), all F ∈ F , or, equivalently, if for all F ∈ F 1F = px (F ), px − a.e. This last equation will hold if for each F ∈ F Z dpx (y)|1F (y) − px (F )|2 = 0. This relation will hold on a set of x of m probability one if Z Z dm(x) dpx (y) |1F (y) − px (F )|2 = 0. Thus proving the preceding equality will complete the proof of the ergodic decomposition theorem. Expanding the square and then using the fact that G(F ) has m probability one yields Z

Z dm(x) G(F )

G(F )

Z 2 G(F )

dpx (y)1F (y)2 − Z dm(x)px (F )

G(F )

dpx (y)1F (y) + Z dm(x)px (F )2 . G(F )

238

7 Ergodic Properties

Since < 1F > (y) = py (F ), the first and third terms on the right are equal from (7.49). (Set f (x) =< 1F (x) >= px (F ) in (7.49).) Since px is stationary, Z dpx < 1F >= px (F ). Thus the middle term cancels the remaining terms and the preceding expression is zero, as was to be shown. This completes the proof of the ergodic decomposition theorem for a dynamical system (or the corresponding directly given random process) which has bounded ergodic properties. 2

Chapter 8

Ergodic Theorems

Abstract The basic ergodic theorems are proved showing that asymptotic mean stationarity is a sufficient as well as necessary condition for a random process or dynamical system to have convergent sample averages for all bounded measurements. Conditions are found for the convergence of unbounded sample averages in both the almost everywhere and mean sense. The proof follows the coding-flavored proofs developed in the ergodic theory literature in the 1970s rather than the classical route of the maximal ergodic theorem.

8.1 The Pointwise Ergodic Theorem This section will be devoted to proving the following result, known as the pointwise or almost everywhere ergodic theorem or as the Birkhoff ergodic theorem after George Birkhoff, who first proved the result for a stationary transformations. Theorem 8.1. A sufficient condition for a dynamical system (Ω, B, m, T ) to possess ergodic properties with respect to all bounded measurements f is that it be AMS. In addition, if the stationary mean of m is m, then the system possesses ergodic properties with respect to all m-integrable measurements, and if f ∈ L1 (m), then with probability one under m and m n−1 1 X f T i = Em (f |I), lim n→∞ n i=0 where I is the sub-σ -field of invariant events. Observe that the final statement follows immediately from the existence of the ergodic property from Corollary 7.10. In this section we will assume that we are dealing with processes to aid interpretations; that R.M. Gray, Probability, Random Processes, and Ergodic Properties, DOI: 10.1007/978-1-4419-1090-5_8, © Springer Science + Business Media, LLC 2009

239

240

8 Ergodic Theorems

is, we will consider points to be sequences and T to be the shift. This is simply for intuition, however, and the mathematics remain valid for the more general dynamical systems. We can assume without loss of generality that the function f is nonnegative (since if it is not it can be decomposed into f = f + − f − and the result for the nonnegative f + and f − and linearity then yield the result for f ). Define f = lim inf n→∞

and f = lim sup n→∞

n−1 1 X fTi n i=0

n−1 1 X f T i. n i=0

To prove the first part of the theorem we must prove that f = f with probability one under m. Since both f and f are invariant, the event F = {x : f (x) = f (x)} = {x: limn→∞ < f >n exists} is also invariant and hence has the same probability under m and m. Thus we must show that m(f = f ) = 1. Thus we need only prove the theorem for stationary measures and the result will follow for AMS measures. As a result, we can henceforth assume that m is itself stationary. Since f ≤ f everywhere by definition, the desired result will follow if we can show that Z Z f (x) dm(x) ≤ f (x) dm(x), R In other words, if f − f ≥ 0 and (f − f )dm ≤ 0, then we must have f − f = 0, m-a.e. We accomplish this by proving the following: Z

Z f (x) dm(x) ≤

Z f (x) dm(x) ≤

f (x) dm(x).

(8.1)

By assumption f ∈ L1 (m) since in the theorem statement either f is bounded or it is integrable with respect to m, which is m since we have now assumed m to be stationary. Thus (8.1) implies not only that lim < f >n exists, but that it is finite. We simplify things at first by temporarily assuming that f is bounded: Suppose for the moment that the supremum of f is M < ∞. Fix  > 0. Since f is the limit supremum of < f >n , for each x there must exist a finite n for which n−1 1 X f (T i x) ≥ f (x) − . (8.2) n i=0

8.1 The Pointwise Ergodic Theorem

241

Define n(x) to be the smallest integer for which (8.2) is true. Since f is an invariant function, we can rewrite (8.2) using the definition of n(x) as n(x)−1 n(x)−1 X X f (T i x) ≤ f (T i x) + n(x). (8.3) i=0

i=0

Since n(x) is finite for all x (although it is not bounded), there must be an N for which ∞ X

m(x : n(x) > N) =

m({x : n(x) = k}) ≤

k=N+1

 M

since the sum converges to 0 as N → ∞.. Choose such an N and define B = {x : n(x) > N}. B has small probability and can be thought of as the “bad” sequences, where the sample average does not get close to the limit supremum within a reasonable time. Observe that if x ∈ B c , then also T i x ∈ B c for i = 1, 2, . . . , n(x) − 1; that is, if x is not bad, then the next n(x) − 1 shifts of n(x) are also not bad. To see this assume the contrary, that is, that there exists a p < n(x) − 1 such that T i x ∈ B c for i = 1, 2, . . . , p − 1, but T p x ∈ B. The fact that p < n(x) implies that p−1

X

f (T i x) < p(f (x) − )

i=0

and the fact that T p x ∈ B implies that n(x)−1 X

f (T i x) < (n(x) − p)(f (x) − ).

i=p

These two facts together, however, imply that n(x)−1 X

f (T i x) < n(x)(f (x) − ),

i=0

contradicting the definition of n(x). We now modify f (x) and n(x) for bad sequences so as to avoid the problem caused by the fact that n(x) is not bounded. Define ( f˜(x) =

f (x)

x 6∈ B

M

x∈B

and ( ˜ n(x) =

n(x)

x 6∈ B

1

x ∈ B.

242

8 Ergodic Theorems

Analogous to (8.3) the modified functions satisfy ˜ X n(x)−1

f (T i x) ≤

i=0

˜ X n(x)−1

˜ f˜(T i x) + n(x)

(8.4)

i=0

˜ and n(x) is bound above by N. To see that (8.4) is valid, note that it is trivial if x ∈ B. If x 6∈ B, as argued previously T i x 6∈ B for i = 1, 2, . . . , n(x) − 1 and hence f˜(T i x) = f (T i x) and (8.4) follows from (8.3). Observe for later reference that Z Z Z f˜(x) dm(x) = f˜(x) dm(x) + f˜(x) dm(x) c ZB ZB f (x) dm(x) + M dm(x) ≤ c B ZB ≤ f (x) dm(x) + Mm(B) Z ≤ f (x) dm(x) + , (8.5) where we have used the fact that f was assumed positive in the second inequality. We now break up the sequence into nonoverlapping blocks such that within each block the sample average of f˜ is close to the limit supremum. Since the overall sample average is a weighted sum of the sample averages for each block, this will give a long term sample average near the limit supremum. To be precise, choose an L so large that NM/L <  and inductively define nk (x) by n0 (x) = 0 and ˜ nk−1 (x) x). nk (x) = nk−1 (x) + n(T Let k(x) be the largest k for which nk (x) ≤ L − 1, that is, the number of the blocks in the length L sequence. Since L−1 X

f (T i x) =

i=0

k(x) X nk (x)−1 X k=1 i=nk−1 (x)

f (T i x) +

L−1 X

f (T i x),

i=nk(x) (x)

we can apply the bound of (8.4) to each of the k(x) blocks, that is, each of the previous k(x) inner sums, to obtain L−1 X i=0

f (T i x) ≤

L−1 X

f˜(T i x) + L + (N − 1)M,

i=0

where the final term overbounds the values in the final block by M and where the nonnegativity of f˜ allows the sum to be extended to L − 1. We

8.1 The Pointwise Ergodic Theorem

243

can now integrate both sides of the preceding equation, divide by L, use the stationarity of m, and apply (8.5) to obtain Z Z Z (N − 1)M f dm ≤ f˜ dm +  + ≤ f dm + 3, (8.6) L which proves the left-hand inequality of (8.1) for the case where f is nonnegative and f is bounded since  is arbitrary. Next suppose that f is not bounded. Define for any M the “clipped” function f M (x) = min(f (x), M) and replace f by f M from (8.2) in the preceding argument. Since f M is also invariant, the identical steps then lead to Z Z f M dm ≤ f dm. Taking the limit as M → ∞ then yields the desired left-hand inequality of (8.1) from the monotone convergence theorem. To prove the right-hand inequality of (8.1) we proceed in a similar manner. Fix  > 0 and define n(x) to be the least positive integer for which n−1 1 X f (T i x) ≤ f (x) +  n i=0 and define B = {x : n(x) > N}, where N is chosen large enough to ensure that Z f (x) dm(x) < . B

This time define ( f˜(x) =

f (x)

x 6∈ B

0

x ∈ B.

and ( ˜ n(x) =

n(x)

x 6∈ B

1

x∈B

and proceed as before. This proves (8.1) and thereby shows that the sample averages of all m-integrable functions converge. From Corollary 7.10, the limit must be the conditional expectations. (The fact that the limits must be conditional expectations can also be deduced from the preceding arguments by restricting the integrations to invariant sets.) 2 Combining the theorem with the results of previous chapter immediately yields the following corollaries. Corollary 8.1. If a dynamical system (Ω, B, m, T ) is AMS with stationary Pn−1 mean m and if the sequence n−1 i=0 f T i is uniformly integrable with respect to m, then the following limit is true m-a.e., m-a.e., and in L1 (m):

244

8 Ergodic Theorems n−1 1 X f T i =< f >= Em (f |I), n→∞ n i=0

lim

where I is the sub-σ -field of invariant events, and Em < f >= E < f >= Ef . If in addition the system is ergodic, then n−1 1 X f T i = Em (f ) n→∞ n i=0

lim

in both senses. Corollary 8.2. A necessary and sufficient condition for a dynamical system to possess bounded ergodic properties is that it be AMS. Corollary 8.3. A necessary and sufficient condition for a an AMS process to be ergodic is that (7.44) hold; that is, lim n−1

n→∞

n−1 X

m(T −i F ∩ G) = m(F )m(G) all F , G ∈ B.

(8.7)

i=0

Corollary 8.4. A necessary and sufficient condition for a an AMS process to be ergodic is that (7.46) hold; that is, lim n−1

n→∞

n−1 X

m(T −i F ∩ F ) = m(F )m(F ) all F ∈ F ,

(8.8)

i=0

where F generates B

Exercises 1. Suppose that m is stationary and that g is a measurement that has the form g = f −f T for some measurement f ∈ L2 (m). Show that < g >n has an L2 (m) limit as n → ∞. Show that any such g is orthogonal to I 2 , the subspace of all square-integrable invariant measurements; that is, E(gh) = 0 for all h ∈ I 2 . Suppose that gn is a sequence of functions of this form that converges to a measurement r in L2 (m). Show that < r >n converges in L2 (m). 2. Suppose that < f >n converges in L2 (m) for a stationary measure m. Show that the sample averages look increasingly like invariant functions in the sense that

8.2 Mixing Random Processes

245

|| < f >n − < f >n T || →n→∞ 0. 3. Show that if p and m are two stationary and ergodic measures, then they are either identical or singular in the sense that there is an event F for which p(F ) = 1 and m(F ) = 0. 4. Show that if p and m are distinct stationary and ergodic processes and λ ∈ (0, 1), then the mixture process λp + (1 − λ)m is not ergodic.

8.2 Mixing Random Processes We can harvest several important examples of ergodic processes from Corollary 8.4. Suppose that we have a random process {Xn } with a distribution m and alphabet A. Recall from Section 4.5 that a process is said to be independent and identically distributed (IID) if there is a measure q on (A, BA ) such that for any rectangle G = ×i∈J Gi , Gi ∈ BA , j ∈ J, J ⊂ I a finite collection of distinct indices, Y q(Gj ); m(×j∈J Gj ) = j∈J

that is, the random variables Xn are mutually independent. Clearly an IID process has the property that if G and F are two finite-dimensional rectangles, then there is an integer M such that for N ≥ M m(F ∩ T −n G) = m(F )m(G); that is, shifting rectangles sufficiently far apart ensures their independence. From Corollary 8.4, IID processes are ergodic. A generalization of this idea is obtained by having a process satisfy this behavior asymptotically as n → ∞. A measure m is said to be mixing or strongly mixing with respect to a transformation T if for all F , G ∈ B lim |m(F ∩ T −n G) − m(F )m(T −n G)| = 0

n→∞

(8.9)

and weakly mixing if for all F , G ∈ B n−1 1 X |m(F ∩ T −i G) − m(F )m(T −i G)| = 0. n→∞ n j=0

lim

(8.10)

Mixing is a form of asymptotic independence, the probability of an event and a large shift of another event approach the product of the probabilities. The definitions of mixing are usually restricted to stationary processes, in which case they simplify. A stationary measure m is said to

246

8 Ergodic Theorems

be mixing or strongly mixing with respect to a transformation T if for all F, G ∈ B lim |m(F ∩ T −n G) − m(F )m(G)| = 0 (8.11) n→∞

and weakly mixing if for all F , G ∈ B n−1 1 X lim |m(F ∩ T −i G) − m(F )m(G)| = 0. n→∞ n j=0

(8.12)

We have shown that IID processes are ergodic, and stated that a natural generalization of IID process is strongly mixing, but we have not actually proved that an IID process is mixing, since the mixing conditions must be proved for all events, not just the rectangles considered in the discussion of the IID process. The following lemma shows that proving that (8.9) or (8.10) holds on a generating field is sufficient in certain cases. Lemma 8.1. If a measure m is AMS and if (8.10) holds for all sets in a generating field, then m is weakly mixing. If a measure m is asymptotically stationary in the sense that limn→∞ m(T −n F ) exists for all events F and if (8.9) holds for all sets in a generating field, then m is strongly mixing. Proof. The first result is proved by using the same approximation techniques of the proof of Lemma 7.15, that is, approximate arbitrary events F and G, by events in the generating field for both m and its stationary mean m. In the second case, as in the theory of asymptotically mean stationary processes, the Vitali-Hahn-Saks theorem implies that there must exist a measure m such that lim m(T −n F ) = m(F ),

n→∞

and hence arbitrary events can again be approximated by field events under both measures and the result follows. 2 An immediate corollary of the lemma is the fact that IID processes are strongly mixing since (8.9) is satisfied for all rectangles and rectangles generate the entire event space. If the measure m is AMS, then either of the preceding mixing properties together with the ergodic theorem ensuring ergodic properties for AMS processes implies that the conditions of Lemma 7.15 are satisfied since

8.2 Mixing Random Processes

|n−1

n−1 X

247

m(T −i F ∩ F ) − m(F )m(F )|

i=0

≤ |n−1

n−1 X

(m(T −i F ∩ F ) − m(T −i F )m(F )| +

i=0

|n−1

n−1 X

m(T −i F )m(F )) − m(F )m(F )|

i=0

≤ n−1

n−1 X

|m(T −i F ∩ F ) − m(T −i F )m(F )| +

i=0

m(F ) | n−1

n−1 X i=0

m(T −i F ) − m(F ) | → 0. n→∞

Thus AMS weakly mixing systems are ergodic, and hence AMS strongly mixing systems and IID processes are also ergodic. In fact, Lemma 7.15 can be (and often is) interpreted as a very weak mixing condition. In the case of m stationary, it is obvious that weak mixing implies ergodicity since the magnitude of a sum is bound above by the sum of the magnitudes. A simpler proof of the ergodicity of weakly and strongly mixing measures follows from the observation that if F is invariant, then m(F ∩ T −n F ) = m(F ) and hence weak or strong mixing implies that m(F ∩F ) = m(F )2 , which in turn implies that m(F ) must be 0 or 1 and hence that m is ergodic.

Kolmogorov Mixing A final form of mixing which is even stronger than strong mixing is more difficult to describe, but we take a little time and space for it because the condition plays a significant role in ergodic theory. For the current discussion we consider only stationary measures. A random process {Xn } with process distribution m is said to be Kolmogorov mixing or K-mixing if for every event F lim

sup

n→∞ G∈F =σ (X ,X +1,··· ) n n n

|m(F ∩ G) − m(F )m(G)| = 0

(8.13)

See, e.g., Smorodinsky [103] or Petersen [84]. A dynamical system (Ω,B, m,T ) is said to be K-mixing if every random process defined on the system (using T ) is K-mixing, but we focus on a single process here. In the case where the process is two-sided and the shift is invertible, a K-mixing process is called a K-automorphism in ergodic theory. It can

248

8 Ergodic Theorems

be shown (with some effort) that for stationary processes, K-mixing processes are equivalent to K-processes, that is, a process is K-mixing if and only if it has trivial tail. One way is easy. Suppose a process is K-mixing. Choose an event F = G in the tail σ -field F∞ = ∩∞ n=0 Fn . (See Section 7.3.) Then F ∈ Fn for all n and the condition reduces to m(F ) = m2 (F ), which implies that m(F ) = 0 or 1. Thus a K-mixing process must be a Kprocess. The other way is more difficult and requires the convergence of conditional probability measures given a decreasing set of sigma-fields, a result which follows from Doob’s martingale convergence theorem. Details can be found, e.g., in [3, 103]. It can further be shown that stationary K-mixing processes are strong mixing. Suppose that a stationary process is is K-mixing. Since stationarity implies asymptotic stationary, Lemma 8.1 implies that we need only show the defining condition for strong mixing holds on a generating field, so focus on cylinder sets. Let F and G be any two finite dimensional cylinders and consider the probabilities m(F ∩ T −n G). By stationarity we can assume without loss of generality that both F and G are nonnegative time cylinders, that is, both are in F0 = σ (X0 , X1 , X2 , · · · ). Thus T −n G ∈ Fn = σ (Xn , Xn+1 , Xn+2 , · · · ). Thus from the definition of K-mixing, |m(F ∩ T −n G) − m(F )m(G)| ≤ sup |m(F ∩ G) − m(F )m(G)| → 0, G∈Fn

n→∞

and hence the process is strong mixing. The original example of a K-mixing process is an IID process, and Kolmogorov’s proof that IID processes have trivial tail is the classic Kolmogorov 0-1 law. I have chosen not to include further details primarily because of the extra machinery required to prove some of these results, in particular the necessity of using the Doob martingale convergence theorem showing the convergence of conditional probabilities giving nested σ -fields. B-processes are K-processes (see, e.g., Ornstein [77]), and hence K-processes sit between strongly mixing processes and B-processes in terms of generality. It is quite difficult, however, to construct K-processes that are not B-processes (see, e.g., [78]) and Kolmogorov himself conjectured earlier that all K-processes were also B-processes.

Exercises 1. Show that if m and p are two stationary and ergodic processes with ergodic properties with respect to all indicator functions, then either they are identical or they are singular in the sense that there is an event G such that p(G) = 1 and m(G) = 0. 2. Suppose that mi are distinct stationary and ergodic sources with ergodic properties with respect to all indicator functions. Show that the

8.3 Block AMS Processes

249

mixture m(F ) =

X

λi mi (F ),

i

where X

λi = 1

i

is stationary but not in general ergodic. Show that more generally if the mi are AMS, then so is m. 3. Prove that if a stationary process m is weakly mixing, then there is a subsequence J = {j1 < j2 < . . .} of the positive integers that has density 0 in the sense that n−1 1 X 1J (i) = 0 n→∞ n i=0

lim

and that has the property that lim

n→∞,n6∈J

|m(T −n F ∩ G) − m(F )m(G)| = 0

for all appropriate events F and G. 4. Are mixtures of mixing processes mixing? 5. Show that IID processes and strongly mixing processes are totally ergodic.

8.3 Block AMS Processes Given a dynamical system (Ω, B, m, T ) or a corresponding source {Xn }, one may be interested in the ergodic properties of successive blocks or vectors N XnN = (XnN , XnN+1 , . . . , X(n+1)N−1 ) or, equivalently, in the N-shift T N . For example, in communication systems one may use block codes that parse the input source into N-tuples and then map these successive nonoverlapping blocks into codewords for transmission. A natural question is whether or not a source being AMS with respect to T (T -AMS) implies that it is also T N -AMS or vice versa. This section is devoted to answering this question. If a source is T -stationary, then it is clearly N-stationary (stationary with respect to T N ) since for any event F m(T −N F ) = m(T −N−1 F ) = · · · = m(F ). The converse is not true, however, since, for example, a process that produces a single sequence that is periodic with period N is N-stationary but not stationary. The following lemma, which is a corol-

250

8 Ergodic Theorems

lary to the ergodic theorem, provides a needed step for treating AMS systems. Lemma 8.2. If µ and η are probability measures on (Ω, B) and if η is T stationary and asymptotically dominates µ (with respect to T ), then µ is T -AMS. Proof. If η is stationary, then from the ergodic theorem it possesses ergodic properties with respect to all bounded measurements. Since it asymptotically dominates µ, µ must also possesses ergodic properties with respect to all bounded measurements from Corollary 7.8. From Theorem 7.1, µ must therefore be AMS. 2 The principal result of this section is the following: Theorem 8.2. If a system (Ω, B, m, T ) is T N -AMS for any positive integer N, then it is T N -AMS for all integers N. Proof. Suppose first that the system is T N -AMS and let mN denote its N-stationary mean. Then for any event F n−1 1 X m(T −iN T −j F ) = mN (T −j F ); j = 0, 1, . . . , N − 1. n→∞ n i=0

lim

(8.14)

We then have N−1 n−1 N−1 n−1 1 X 1 X 1 X X lim m(T −iN T −j F ) = lim m(T −iN T −j F ) n→∞ nN N j=0 n→∞ n i=0 j=0 i=0 n−1 1 X m(T −i F ), n→∞ n i=0

= lim

(8.15)

where, in particular, the last limit must exist for all F and hence m must be AMS Next assume only that m is AMS with stationary mean m. This implies from Corollary 7.5 that m T -asymptotically dominates m, that is, if m(F ) = 0, then it is also true that lim supn→∞ m(T −n F ) = 0. But this also means that for any integer K lim sup m(T −n KF ) = 0 n→∞

and hence that m also T K -asymptotically dominates m. But this means from Lemma 8.2 applied to T K that m is T K -AMS. 2 Corollary 8.5. Suppose that a process µ is AMS and has a stationary mean n−1 1 X m(T −i F ) n→∞ n i=0

m(F ) = lim

8.3 Block AMS Processes

251

and an N-stationary mean n−1 1 X m(T −iN F ). n→∞ n i=0

mN (F ) = lim

Then the two means are related by N−1 1 X mN (T −i F ) = m(F ). N i=0

Proof. The proof follows immediately from (8.14)-(8.15). 2 As a final observation, if an event is invariant with respect to T , then it is also N-invariant, that is, invariant with respect to T N . Thus if a system is N-ergodic or ergodic with respect to T N , then all N-invariant events must have probability of 0 or 1 and hence so must all invariant events. Thus the system must also be ergodic. The converse, however, is not true: An ergodic system need not be N-ergodic. A system that is Nergodic for all N is said to be totally ergodic. A strongly mixing system can easily be seen to be totally ergodic.

Exercises 1. Prove that a strongly mixing source is totally ergodic. 2. Suppose that m is N-ergodic and N-stationary. Is the mixture m=

N−1 1 X mT −i N i=0

stationary and ergodic? Are the limits of the form n−1 1 X fTi n i=0

the same under both m and m? What about limits of the form n−1 1 X f T iN ? n i=0

3. Suppose that you observe a sequence from the process mT −i where you know m but not i. Under what circumstances could you determine i correctly with probability one by observing the infinite output sequence?

252

8 Ergodic Theorems

8.4 The Ergodic Decomposition of AMS Systems Combining the ergodic decomposition theorem for systems with bounded ergodic properties, Theorem 7.6, with the equivalence of bounded ergodic properties and AMS, Corollary 8.2, yields immediately the ergodic decomposition of AMS systems, which is largely a repeat of the statement of Theorem 7.6. Theorem 8.3. The Ergodic Decomposition of AMS Dynamical Systems Given a standard measurable space (Ω, B), let F denote a countable generating field. Given a transformation T : Ω → Ω define the set of frequency-typical or generic sequences by G(F ) = {x : lim < 1F >n (x) exists all F ∈ F }. n→∞

For each x ∈ G(F ) let Px denote the measure induced by the limiting ˆ ⊂ G(F ) by E ˆ = {x : Px is stationary relative frequencies. Define the set E and ergodic}. Define the stationary and ergodic measures px by px = Px ˆ and otherwise px = p ∗ , some fixed stationary and ergodic for x ∈ E measure. (Note that so far all definitions involve properties of the points only and not of any underlying measure.) We shall refer to the collection of measures {px ; x ∈ Ω} as the ergodic decomposition. We have then that (a) (b) (c)

ˆ is an invariant measurable set, E px (F ) = pT x (F ) for all points x ∈ Ω and events F ∈ B, If m is an AMS measure with stationary mean m, then ˆ = 1, ˆ = m(E) m(E)

(d)

for any F ∈ B Z m(F ) =

(e)

(g)

(8.16)

Z px (F )dm(x) =

px (F )dm(x),

R and if f ∈ L1 (m), then so is f dpx and Z Z Z  f dm = f dpx dm(x).

(8.17)

(8.18)

In addition, given any f ∈ L1 (m), then lim n−1

n→∞

n−1 X

f (T i x) = Epx f , m − a.e. and m − a.e.

(8.19)

i=0

Comment: The following comments are almost a repeat of those following Theorem 7.6, but for the benefit of the reader who does not recall

8.5 The Subadditive Ergodic Theorem

253

those comments or has not read the earlier result they are included. The ergodic decomposition is extremely important for both mathematics and intuition. Say that a system s AMS and that f is integrable with respect to the stationary mean; e.g., f is measurable and bounded and hence integrable with respect to any measure. Then with probability one a point will be produced such that the limiting sample average exists and equals the expectation of the measurement with respect to the stationary and ergodic measure induced by the point. Thus if a system is AMS, then the limiting sample averages will behave as if they were in fact produced by a stationary and ergodic system. These ergodic components are determined by the points themselves and not the underlying measure; that is, the same collection works for all AMS measures.

The Nedoma N-Ergodic Decomposition Although an ergodic process need not be N-ergodic, an application of the ergodic decomposition theorem to T N yields the following result showing that a stationary ergodic process can be written as a mixture of N-ergodic processes. The proof is left as an exercise. Corollary 8.6. Suppose that a measure m is stationary and ergodic and N is a fixed positive integer. Show that there is a measure η that is Nstationary and N-ergodic that has the property that m=

N−1 1 X ηT −i . N i=0

This result is due to Nedoma [73]. A direct proof of the result for discrete random processes can be found in Gallager [27], where the result plays a key role in the proof of the Shannon theorem on block source coding with a fidelity criterion.

8.5 The Subadditive Ergodic Theorem Sample averages are not the only measurements of systems that depend on time and converge. In this section we consider another class of measurements that have an ergodic theorem: subadditive measurements. An example of such a measurement is the Levenshtein distance between sequences (see Example 1.5). The subadditive ergodic theorem was first developed by Kingman [60], but the proof again follows that of Katznelson and Weiss [59]. A se-

254

8 Ergodic Theorems

quence of functions {fn ; = 1, 2, . . .} satisfying the relation fn+m (x) ≤ fn (x) + fm (T n x), all n, m = 1, 2, . . . for all x is said to be subadditive. Note that the unnormalized sample averages fn = n < f >n of a measurement f are additive and satisfy the relation with equality and hence are a special case of subadditive functions. The following result is for stationary systems. Generalizations to AMS systems are considered later. Theorem 8.4. Let (Ω, B, m, T ) be a stationary dynamical system. Suppose that {fn ; n = 1, 2, . . .} is a subadditive sequence of m-integrable measurements. Then there is an invariant function φ(x) such that m-a.e. lim

n→∞

1 fn (x) = φ(x). n

The function φ(x) is given by φ(x) = inf n

1 1 Em (fn |I) = lim Em (fn |I). n→∞ n n

Most of the remainder of this section is devoted to the proof of the theorem. On the way some lemmas giving properties of subadditive functions are developed. Observe that since the conditional expectations with respect to the σ -field of invariant events are themselves invariant functions, φ(x) is invariant as claimed. Since the functions are subadditive, Em (fn+m |I) ≤ Em (fn |I) + Em (fm T n |I). From the ergodic decomposition theorem, however, the third term is EPx (fm T n ) = EPx (fm ) since Px is a stationary measure with probability one. Thus we have with probability one that Em (fn+m |I)(x) ≤ Em (fn |I)(x) + Em (fm |I)(x). For a fixed x, the preceding is just a sequence of numbers, say ak , with the property that an+k ≤ an + ak . A sequence satisfying such an inequality is said to be subadditive. The following lemma (due to Hammersley and Welch [49]) provides a key property of subadditive sequences. Lemma 8.3. If {an ; n = 1, 2, . . .} is a subadditive sequence, then lim

n→∞

an an = inf , n≥1 n n

8.5 The Subadditive Ergodic Theorem

255

where if the right-hand side is −∞, the preceding means that an /n diverges to −∞. Proof. First observe that if an is subadditive, for any k, n akn ≤ an + a(k−1)n ≤ an + an + a(k−2)n ≤ · · · ≤ kan .

(8.20)

Call the infimum γ and first consider the case where it is finite. Fix  > 0 and choose a positive integer N such that N −1 aN < γ + . Define b = max1≤k≤3N−1 ak . For each n ≥ 3N there is a unique integer k(n) for which (k(n) + 2)N ≤ n < (k(n) + 3)N. Since the sequence is subadditive, an ≤ ak(n)N + a(n−k(n)N) . Since n − k(n)N < 3N by construction, an−k(n)N ≤ b and therefore γ≤

ak(n)N + b an ≤ . n n

Using (8.20) this yields γ≤

an k(n)aN b k(n)N aN b k(n)N b ≤ + ≤ + ≤ (γ + ) + . n n n n N n n n

By construction k(n)N ≤ n − 2N ≤ n and hence γ≤

b an ≤γ++ . n n

Thus given an , choosing n0 large enough yields γ ≤ n−1 an ≤ γ + 2 for all n > n0 , proving the lemma. If γ is −∞, then for any positive M we can choose n so large that n−1 an ≤ −M. Proceeding as before for any  there is an n0 such that if n ≥ n0 , then n−1 an ≤ −M + . Since M is arbitrarily big and  arbitrarily small, n−1 an goes to −∞. 2 Returning to the proof of the theorem, the preceding lemma implies that with probability one φ(x) = inf

n≥1

1 1 Em (fn |I)(x) = lim Em (fn |I)(x). n→∞ n n

As in the proof of the ergodic theorem, define f (x) = lim sup n→∞

and f (x) = lim inf n→∞

1 fn (x) n

1 fn (x). n

Lemma 8.4. Given a subadditive function fn (x); = 1, 2, . . ., f and f as defined previously are invariant functions m-a.e.

256

8 Ergodic Theorems

Proof. From the definition of subadditive functions f1+n (x) ≤ f1 (x) + fn (T x) and hence

fn (T x) 1 + n f1+n (x) f1 (x) ≥ − . n n 1+n n

Taking the limit supremum yields f (T x) ≥ f (x). Since m is stationary, Em f T = Em f and hence f (x) = f (T x) m-a.e. The result for f follows similarly. 2 We now tackle the proof of the theorem. It is performed in two steps. The first uses the usual ergodic theorem and subadditivity to prove that f ≤ φ a.e. Next a construction similar to that of the proof of the ergodic theorem is used to prove that f = φ a.e., which will complete the proof. As an introduction to the first step, observe that since the measurements are subadditive n−1 1 1 X f1 (T i x). fn (x) ≤ n n i=0

Since f1 ∈ L1 (m) for the stationary measure m, we know immediately from the ergodic theorem that f (x) ≤ f1 (x) = Em (f1 |I).

(8.21)

We can obtain a similar bound for any of the fn by breaking it up into fixed size blocks of length, say N and applying the ergodic theorem to the sum of the fN . Fix N and choose n ≥ N. For each i = 0, 1, 2, . . . , N − 1 there are integers j, k such that n = i + jN + k and 1 ≤ k < N. Hence using subadditivity j−1

fn (x) ≤ fi (x) +

X

fN (T lN+i x) + fk (T jN+i x).

l=0

Summing over i yields Nfn (x) ≤

N−1 X i=0

jN−1

fi (x) +

X l=0

fN (T l x) +

N−1 X

fn−i−jN (T jN+i x),

i=0

and hence jN−1 N−1 N−1 1 1 X 1 X 1 X fn (x) ≤ fN (T l x) + fi (x) + fn−i−jN (T jN+i x). n nN l=0 nN i=0 nN i=0

8.5 The Subadditive Ergodic Theorem

257

As n → ∞ the middle term on the right goes to 0. Since fN is assumed integrable with respect to the stationary measure m, the ergodic theorem implies that the leftmost term on the right tends to N −1 < fN >. Intuitively the rightmost term should also go to zero, but the proof is a bit more involved. The rightmost term can be bound above as N−1 N−1 N 1 X 1 X X |fn−i−jN (T jN+i x)| ≤ |fl (T jN+i x)| nN i=0 nN i=0 l=1

where j depends on n (the last term includes all possible combinations of k in fk and shifts). Define this bound to be wn /n. We shall prove that wn /n tends to 0 by using Corollary 5.10, that is, by proving that ∞ X

m(

n=0

wn > ) < ∞. n

(8.22)

First note that since m is stationary, wn =

N−1 N 1 X X |fl (T jN+i x)| N i=0 l=1

has the same distribution as w=

N−1 N 1 X X |fl (T i x)|. N i=0 l=1

Thus we will have proved (8.22) if we can show that ∞ X

m(w > n) < ∞.

(8.23)

n=0

To accomplish this we need a side result, which we present as a lemma. Lemma 8.5. Given a nonnegative random variable X defined on a probability space (Ω, B, m), ∞ X m(X ≥ n) ≤ EX. n=1

Proof.

258

8 Ergodic Theorems

EX = ≥

=

∞ X n=1 ∞ X n=1 ∞ X

E(X|X ∈ [n − 1, n))m(X ∈ [n − 1, n)) (n − 1)m(X ∈ [n − 1, n)) nm(X ∈ [n − 1, n)) − 1.

n=1

By rearranging terms the last sum can be written as ∞ X

m(X ≥ n),

n=0

2

which proves the lemma. Applying the lemma to the left side of (8.23) yields ∞ X n=0

m(

∞ X 1 w w > n) ≤ ≥ n) ≤ Em (w) < ∞, m(    n=0

since m-integrability of the fn implies m-integrability of their absolute magnitudes and hence of a finite sum of absolute magnitudes. This proves (8.23) and hence (8.22). Thus we have shown that f (x) ≤

1 1 fN (x) = Em (fN |I) N N

for all N and hence that f (x) ≤ φ(x), m − a.e.

(8.24)

This completes the first step of the proof. If φ(x) = −∞, then (8.24) completes the proof since the desired limit then exists and is −∞. Hence we can focus on the invariant set G = {x : φ(x) > −∞}. Define the event GM = {x : φ(x) ≥ −M}. We shall show that for all M Z Z f dm ≥ φ dm, (8.25) GM

GM

which with (8.24) proves that f = φ on GM . We have, however, that G=

∞ [

GM .

M=1

Thus proving that f = φ on all GM also will prove that f = φ on G and complete the proof of the theorem.

8.5 The Subadditive Ergodic Theorem

259

We now proceed to prove (8.25) and hence focus on x in GM and hence only x for which φ(x) ≥ −M. We proceed as in the proof of the ergodic theorem. Fix an  > 0 and define f M = max(f , −M − 1). Observe that this implies that φ(x) ≥ f M (x), all x. (8.26) Define n(x) as the smallest integer n ≥ 1 for which n−1 fn (x) ≤ f M (x)+ . Define B = {x : n(x) > N} and choose N large enough so that Z (|f1 (x)| + M + 1) dm(x) < . (8.27) B

Define the modified functions as before: ( f M (x) ˜ f M (x) = f1 (x)

x 6∈ B x∈B

and ( ˜ n(x) =

n(x)

x 6∈ B

1

x∈B

.

From (8.27) Z Z Z Z Z ˜ dm = ˜ dm + ˜ dm = f f dm + f1 dm f f M M M M c c B ZB ZB Z ZB f M dm + |f1 | dm ≤ f M dm +  − (M + 1) dm ≤ c B Bc B Z Z ZB f M dm +  + f M dm ≤ f M dm + . (8.28) ≤ Bc

B

Since f M is invariant, fn(x) (x) ≤

n(x)−1 X

f M (T i x) + n(x)

i=0

and hence also fn(x) (x) ≤ ˜

˜ X n(x)−1

˜ f˜M (T i x) + n(x)

i=0

Thus for L > N fL (x) ≤

L−1 X

f˜M (T i x) + L + N(M − 1) +

i=0

Integrating and dividing by L results in

L−1 X i=L−N

|f1 (T i x)|.

260

Z

8 Ergodic Theorems

1 fL (x) dm(x) L Z 1 = fL (x) dm(x) L Z Z N(M + 1) N ≤ f˜M (x) dm(x) +  + + |f1 (x)| dm(x). L L Z

φ(x) dm(x) ≤

Letting L → ∞ and using (8.28) yields Z Z φ(x) dm(x) ≤ f M (x) dm(x). From (8.26), however, this can only hold if in fact f M (x) = φ(x) a.e. This implies, however, that f (x) = φ(x) a.e. since the event f (x) 6= f M (x) implies that f M (x) = −M − 1 which can only happen with probability 0 from (8.26) and the fact that f M (x) = φ(x) a.e. This completes the proof of the subadditive ergodic theorem for stationary measures. 2 Unlike the ergodic theorem, the subadditive ergodic theorem does not easily generalize to AMS measures since if m is not stationary, it does not follow that f or f are invariant measurements or that the event {f = f } is either an invariant event or a tail event. The following generalization is a slight extension of unpublished work of Wu Chou. [13] Theorem 8.5. Let (Ω, B, m, T ) be an AMS dynamical system with stationary mean m. Suppose that {fn ; n = 1, 2, . . .} is a subadditive sequence of m-integrable measurements. There is an invariant function φ(x) such that m-a.e. 1 lim fn (x) = φ(x) n→∞ n if and only if m << m, in which case the function φ(x) is given by φ(x) = inf n

1 1 Em (fn |I) = lim Em (fn |I). n→∞ n n

Comment: Recall from Theorem 7.4 that the condition that the AMS measure is dominated by its stationary mean is equivalent to the condition that the system be recurrent. Note that the theorem immediately implies that the Kingman ergodic theorem holds for two-sided AMS processes. What is new here is the result for one-sided AMS processes. Proof. If m << m, then the conclusions follow immediately from the ergodic theorem for stationary sources since m inherits the probability 1 and 0 sets from m. Suppose that m is not dominated by m. Then from Theorem 7.4, the system is not conservative and hence there exists a wandering set W with nonzero probability, that is, the sets T −n W are pairwise disjoint and m(W ) > 0. We now construct a counterexample to the subadditive ergodic theorem, that is, a subadditive function that does not converge. Define

8.5 The Subadditive Ergodic Theorem

 kn   −2 fn (x) = fn (T j x) − 2n   0

261

x ∈ W and 2kn ≤ n ≤ 2kn +1 , kn = 0, 1, 2, . . . x ∈ T −j W otherwise.

We first prove that fn is subadditive, that is, that fn+m (x) ≤ fn (x) + fm (T n x) for S∞all x ∈ Ω. To do this we consider three cases: (1) x ∈ W , (2) x ∈ W ∗ = i=1 T −i W , and (3) x 6∈ W ∪ W ∗ . In case (1), fn (x) = −2kn and fn+m (x) = −2kn+m . To evaluate fm (T n x) observe that x ∈ W implies that T n x 6∈ W ∪ W ∗ since otherwise ∞ ∞ [ [ T −k W = T −k W , x ∈ T −n k=0

k=n

which is impossible because x ∈ W and W is a wandering set. Thus fn (T m x) = 0. Thus for x ∈ W fn (x) + fm (T n x) − fn+m (x) = −2kn + 2kn+m . Since by construction kn + m ≥ kn , this is nonnegative and f is subadditive in case (1). For case (3) the result is trivial: If x 6∈ W ∪ W ∗ , then no shift of x is ever in W , and all the terms are 0. This leaves case (2). Suppose now that x ∈ T −j W for some k. Then fn+m (x) = fn+m (T j x) − 2n+m = −2kn+m − 2n+m , and fn (x) = fn (T j x) − 2n = −2kn − 2n . For the final term there are two cases. If n > j, then analogously to case (1) x ∈ T −j W implies that T n x 6∈ W ∪ W ∗ since otherwise x ∈ T −n

∞ [ k=0

T −k W =

∞ [

T −k W ,

k=n

which is impossibleSsince W is a wandering set and hence x cannot be in ∞ both T −j W and in k=n T −k W since n > j. If n ≤ j, then fm (T n x) = fm (T j x) − 2m = −2km − 2m . Therefore if n > j, fn (x) + fm (T n x) − fn+m (x) = −2kn − 2n + 0 + 2kn+m + 2n+m ≥ 0.

262

8 Ergodic Theorems

If n ≤ j then fn (x) + fm (T n x) − fn+m (x) = −2kn − 2n − 2km − 2m + 2kn+m + 2n+m . Suppose that n ≥ m ≥ 1, then 2n+m − 2n − 2m = 2n (2m − 1 − 2−|n−m| ) ≥ 0. Similarly, since then kn ≥ km , then 2kn+m − 2kn − 2km = 2kn (2kn+m −kn − 1 − 2−|kn −km | ≥ 0. Similar inequalities follow when n ≤ m, proving that fn is indeed subadditive on Ω. We now complete the proof. The event W has nonzero probability, but if x ∈ W , then 1 −2n lim n f2n (x) = lim n = −1 n→∞ 2 n→∞ 2 whereas

1 −2n−1 1 n −1 (x) = lim f =− 2 n→∞ 2n − 1 n→∞ 2n − 1 2 lim

and hence n−1 fn does not have a limit on this set of nonzero probability, proving that the subadditive ergodic theorem does not hold in this case.

2

Chapter 9

Process Approximation and Metrics

Abstract The goodness or badness of approximation of one process for another can be quantified using various metrics (or distances) or distortion measures (or cost functions). Two of the most useful such notions of approximation are introduced and studied in this chapter — the distributional distance and the dp distortion. The former leads to a useful metric space of probability measures. The second combines ideas from the Monge/Kantorovich optimal transportation cost and Ornstein’s d (dbar) distance on random variables, vectors, and processes and is useful for quantifying the different time-average behavior of frequency-typical or frequency-typical sequences produced by distinct random processes.

9.1 Distributional Distance We begin with probability metric that is not very strong, but suffices for developing the ergodic decomposition of affine functionals in the next chapter. Given any class G = {Fi ; i = 1, 2, . . .} consisting of a countable collection of events we can define for any measures p, m ∈ P(Ω, B) the function ∞ X dG (p, m) = 2−i |p(Fi ) − m(Fi )|. (9.1) i=1

If G contains a generating field, then dG is a metric on P(Ω, B) since from Lemma 1.19 two measures defined on a common σ -field generated by a field are identical if and only if they agree on the generating field. We shall always assume that G contains such a field. We shall call such a distance a distributional distance. Usually we will require that G be a standard field and hence that the underlying measurable space (Ω, B) is standard. Occasionally, however, we will wish a larger class G, but R.M. Gray, Probability, Random Processes, and Ergodic Properties, DOI: 10.1007/978-1-4419-1090-5_9, © Springer Science + Business Media, LLC 2009

263

264

9 Process Approximation and Metrics

will not require that it be standard. The different requirements will be required for different results. When the class G is understood from context, the subscript will be dropped. Let (P(Ω, B), dG ) denote the metric space of P(Ω, B) with the metric dG . A key property of spaces of probability measures on a fixed measurable space is given in the following lemma. Lemma 9.1. The metric space (P((Ω, B)), dG ) of all probability measures on a measurable space (Ω, B) with a countably generated sigma-field is separable if G contains a countable generating field. If also G is standard (as is possible if the underlying measurable space (Ω, B) is standard), then also (P, dG ) is complete and hence Polish. Proof. For each n let An denote the set of nonempty intersection sets or atoms of {F1 , . . . , Fn }, the first n sets in G. (These are Ω sets, events in the original space.) For each set G ∈ An choose an arbitrary point xG such that xG ∈ G. We will show that the class of all measures of the form X pG 1F (xG ), r (F ) = G∈An

where the pG are nonnegative and rational and satisfy X pG = 1, G∈An

forms a dense set in P(Ω, B). Since this class is countable, P(Ω, B) is separable. Observe that we are approximating all measures by finite sums of point masses. Fix a measure m ∈ (P(Ω, B), dG ) and an  > 0. Choose n so large that 2−n < /2. Thus to match up two measures in d = dG , (9.1) implies that we must match up the probabilities of the first n sets in G since the contribution of the remaining terms is less that 2−n . Define X m(G)1F (xG ) rn (F ) = G∈An

and note that m(F ) =

X

m(G)m(F |G),

G∈An

where m(F |G) = m(F ∩G)/m(G) is the elementary conditional probability of F given G if m(G) > 0 and is arbitrary otherwise. For convenience we now consider the preceding sums to be confined to those G for which m(G) > 0. Since the G are the atoms of the first n sets {Fi }, for any of these Fi either G ⊂ Fi and hence G ∩ Fi = G or G ∩ Fi = ∅. In the first case 1Fi (xG ) = m(Fi |G) = 1, and in the second case 1Fi (xG ) = m(Fi |G) = 0, and hence in both cases

9.1 Distributional Distance

265

rn (Fi ) = m(Fi ); i = 1, 2, . . . , n. This implies that ∞ X

d(rn , m) ≤

2−i = 2−n ≤

i=n+1

 . 2

Enumerate the atoms of An as {Gl ; l = 1, 2, . . . , L, where L ≤ 2n . For all l but the last (l = L) pick a rational number pGl such that m(Gl ) − 2−l

 ≤ pGl ≤ m(Gl ); 4

that is, we choose a rational approximation to m(Gl ) that is slightly less than m(Gl ). We define pGL to force the rational approximations to sum to 1: L−1 X pG L = 1 − pG l . l=1

Clearly pGL is also rational and |pGL − m(GL )| = |

L−1 X

pGl −

l=1

≤

L−1 X

L−1 X

m(Gl )|

l=1

|pGl − m(Gl )| ≤

l=1

L−1  X −l  2 ≤ . 4 l=1 4

Thus X

|pG − m(G)| =

G∈An

L−1 X

|pGl − m(Gl )| + |pGL − m(GL )| ≤

l=1

 . 2

Define the measure tn by tn (F ) =

X

pG 1F (xG )

G∈An

and observe that d(tn , rn ) =

n X

2−i |tn (Fi ) − rn (Fi ) =

i=1

≤

n X i=1

Thus

n X

2−i |

G∈An

i=1

2−i

X G∈An

|pG − m(G)| ≤

X

 . 4

1Fi (pG − m(G))|

266

9 Process Approximation and Metrics

d(tn , m) ≤ d(tn , rn ) + d(rn , m) ≤

  + = , 2 2

which completes the proof of separability. Next assume that mn is a Cauchy sequence of measures. From the definition of d this can only be if also mn (Fi ) is a Cauchy sequence of real numbers for each i. Since the real line is complete, this means that for each i mn (Fi ) converges to something, say α(Fi ). The set function α is defined on the class G and is clearly nonnegative, normalized, and finitely additive. Hence if G is also standard, then α extends to a probability measure on (Ω, B); that is, α ∈ P(Ω, B). By construction d(mn , α) → 0 as n → ∞, and hence in this case P(Ω, B) is complete.

2

Lemma 9.2. Assume as in the previous lemma that (Ω, B) is standard and G is a countable generating field. Then (P((Ω, B)), dG ) is sequentially compact; that is, if {µn } is a sequence of probability measures in P(Ω, B), then it has a subsequence µnk that converges. Proof. Since G is a countable generating field, a probability measure µ in P is specified via the Carathéodory extension theorem by the infinite dimensional vector µ = {µ(F ); F ∈ G} ∈ [0, 1]Z+ . Suppose that µn is a sequence of probability measures and let µn also denote the corresponding sequence of vectors. From Lemma 1.12, [0, 1]Z+ is sequentially compact with respect to the product space metric, and hence there must exist a µ ∈ [0, 1]Z+ and a subsequence µnk such that µnk → µ. The vector µ provides a set function µ assigning a value µ(F ) for all sets F in the generating field G. This function S is nonnegative, normalized, and finitely additive. In particular, if F = i Fi is a union of a finite number of disjoint events in G, then X µ(F ) = lim µnk (F ) = lim µnk (Fi ) k→∞

=

X i

k→∞

lim µnk (Fi ) =

k→∞

i

X

µ(Fi ).

i

Since the alphabet is standard, µ must extend to a probability measure on (Ω, B). By construction µnk (F ) → µ(F ) for all F ∈ G and hence dG (µnk , µ) → 0, completing the proof. 2

An Example As an example of the previous construction, suppose that Ω is itself a Polish space with some metric d and that G is the field generated by the countable collection of spheres VΩ of Lemma 1.10. Thus if mn → µ

9.1 Distributional Distance

267

under dG , mn (F ) → µ(F ) for all spheres in the collection and all finite intersections and unions of such spheres. From (1.18) any open set G S can be written as a union of elements of VΩ , say i Vi . Given , choose an M such that M [ m( Vi ) > m(G) − . i=1

Then m(G) −  < m(

M [

Vi ) = lim mn ( n→∞

i=1

M [

Vi ) ≤ lim inf mn (G).

i=1

n→∞

Thus we have proved that convergence of measures under dG implies that lim inf mn (G) ≥ m(G); all open G. n→∞

Since the complement of a closed set is open, this also implies that lim sup mn (G) ≤ m(G); all closed G. n→∞

Along with the preceding limiting properties for open and closed sets, convergence under dG has strong implications for continuous functions. Recall from Section 1.4 that a function f : Ω → R is continuous if given ω ∈ Ω and  > 0 there is a δ such that d(ω, ω0 ) < δ ensures that |f (ω)− f (ω0 )| < . We consider two such results, one in the form we will later need and one to relate our notion of convergence to that of weak convergence of measures. Lemma 9.3. If f is a nonnegative continuous function and (Ω, B) and dG are as previously, then dG (mn , m) → 0 implies that lim sup Emn f ≤ Em f . n→∞

Proof. For any n we can divide up [0, ∞) into the countable collection of intervals [k/n, (k + 1)/n), k = 0, 1, 2, . . . that partition the nonnegative real line. Define the sets Gk (n) = {r : k/n ≤ f (r ) < (k + 1)/n} and define the closed sets Fk (n) = {r : k/n ≤ f (r )}. Observe that Gk (n) = Fk (n) − Fk+1 (n). Since f is nonnegative for any n, ∞ ∞ ∞ X X X k k+1 k 1 m(Gk (n)) ≤ Em f ≤ m(Gk (n)) = m(Gk (n)) + . n n n n k=0 k=0 k=0

The sum can be written as ∞ ∞ 1 X 1 X k (m(Fk (n)) − m(Fk+1 (n)) = m(Fk (n)), n k=0 n k=0

268

9 Process Approximation and Metrics

and therefore

∞ 1 X m(Fk (n)) ≤ Em f . n k=0

By a similar construction Emn f <

∞ 1 X 1 mn (Fk (n)) + n k=0 n

and hence from the property for closed sets lim sup Emn f ≤ n→∞

≤

∞ 1 X 1 lim sup mn (Fk (n)) + n k=0 n→∞ n ∞ 1 X 1 1 ≤ Em f + . m(Fk (n)) + n k=0 n n

Since this is true for all n, the lemma is proved.

2

Corollary 9.1. Given (Ω, B) and dG as in the lemma, suppose that f is a bounded, continuous function. Then dG (mn , m) → 0 implies that lim Emn f = Em f .

n→∞

Proof. If f is bounded we can find a finite constant c such that g = f + c ≥ 0. As g is clearly continuous, we can apply the lemma to g to obtain lim sup Emn g ≤ Em g = c + Em f . n→∞

Since the left-hand side is c + lim supn→∞ Emn f , lim sup Emn f ≤ Em f , n→∞

as we found with positive functions. Now, however, we can apply the same result to −f (which is also bounded) to obtain lim inf Emn f ≥ Em f , n→∞

which together with the limit supremum result proves the corollary. 2 Given a space of measures on a common metric space (Ω, B), a sequence of measures mn is said to converge weakly to m if lim Emn f = Em f

n→∞

for all bounded continuous functions f . We have thus proved the following:

9.2 Optimal Coupling Distortion/Transportation Cost

269

Corollary 9.2. Given the assumptions of Lemma 9.3, if dG (mn , m) → 0, then mn converges weakly to m. We will not make much use of the notion of weak convergence, but the preceding result is included to relate the convergence that we use to weak convergence for the special case of Polish alphabets. For a more detailed study of weak convergence of measures, the reader is referred to Billingsley [4, 5].

9.2 Optimal Coupling Distortion/Transportation Cost In the previous section we considered a distance function on probability measures. This section treats a very different distance or distortion measure, which can be viewed as a metric or distortion measure between random variables or vectors. In the next section the ideas will be extended to random processes. Unlike the distributional distances which measure how well or badly probabilities match up, this distortion or cost function will measure how small the average distortion between two random variables or vectors can be. The idea is extended to random processes in the next section by means of a fidelity criterion, a family of distortion measures. This is a particularly useful notion when dealing with communications and signal processing systems where the quality of a system is usually measured by how good (or how bad) one process is as an approximation to another.

Distortion Measures A distortion measure is not a “measure” in the sense used so far; it is an assignment of a nonnegative real number which indicates how bad an approximation one symbol or random object is of another — the smaller the distortion, the better the approximation. The idea is a natural generalization of a metric or distance — a distortion measure need not satisfy the symmetry and triangle inequality properties of a metric. If the two random objects correspond to the input and output of a communication system, for example, then the distortion provides a measure of the quality of performance of the system, small (large) distortion means good (bad) quality. As another example, we might try to simulate a particular process by another process from a constrained class. The constraint might be a finite-alphabet constraint, a constraint on the memory, or, as considered in the sequel [30], a constraint on the entropy rate. The question naturally arises as to what the best such constrained approximation

270

9 Process Approximation and Metrics

might be, and a process metric provides a useful means of quantifying this optimization. Given two measurable spaces (A, B(A)) and (B, B(B)), a distortion measure on A×B is a nonnegative measurable mapping ρ : A×B → [0, ∞) which assigns a real number ρ(x, y) to each x ∈ A as y ∈ B which can be thought of as the cost of reproducing x and y. Infinite values can be allowed in the abstract, but most distortion measures of practical interest take on only finite (but possibly unbounded) values. Metrics or distances are a common choice for distortion measures, and for our purposes the most important example of a metric distortion measure arises when the input space A is a Polish space, a complete separable metric space, and B is either A itself or a Borel subset of A. In this case the distortion measure is fundamental to the structure of the alphabet and the alphabets are standard since the space is Polish. Many practically important distortion measures are not metrics. The most important single distortion measure in signal processing, is squared error, which is the square of the Euclidean metric and not itself a metric. The structure of the underlying metric, however, is preserved in this case, and more generally in the case of distortion measures formed as a positive power of a metric. This is the case that will be emphasized here.

Optimal Coupling Distortion and Transportation Distance Suppose that π is a measure on the product measurable space (A × B, B(A) × B(B)). The average distortion is defined in the obvious way as Eπ ρ. The average distortion can be restated in terms of random variables or vectors. Suppose that X and Y are random objects with alphabets AX and AY . Let (X, Y ) denote the identity mapping on the product space AX × AY and let X and Y denote the projections onto the component spaces given by X(x, y) = x, Y (x, y) = y. Then π can be viewed as the joint distribution of X and Y and the average distortion is given by Eπ ρ(X, Y ). Any such measure π on the product space induces marginal measures on the individual component spaces, say µX and µY , where µX (F ) = π (F × B); F ∈ B(A); µY (G) = π (A × G); G ∈ B(B)

(9.2)

A joint measure consistent with two given marginals is called a coupling or joining of the two marginals. The question now naturally arises: given two marginal distributions µX and µY , what joint distribution π minimizes the average distortion? This yields the optimization problem

9.2 Optimal Coupling Distortion/Transportation Cost

ρ(µX , µY ) =

inf

π ∈P(µX ,µY )

Eπ ρ(X, Y ),

271

(9.3)

where P(µX , µY ) is the collection of all joint measures π satisfying (9.2). Note that the set P(µX , µY ) is not empty since it at least contains the product measure. We first encountered this idea in Example 2.2, where we defined a distance measure between distributions PX and PY describing random variables X and Y by d(PX , PY ) =

inf

PX,Y ∈P(PX ,PY )

PX,Y (X ≠ Y ).

This is simply 9.3 for the particular case ρ = dH , the Hamming distance of Example 1.1 where dH (x, y) = 0 if x = y and 1 otherwise, so that PX,Y (X ≠ Y ) = EPX,Y dH (X, Y ).

(9.4)

This definition is more suited to random variables and vectors than to random processes, but before extending the definition to processes in the Section 9.4, some historical remarks are in order. The above formulation dates back to Kantorovich’s minimization of average cost over couplings with prescribed marginals [57]. Kantorovich considered compact metric spaces with metric cost functions. The modern literature considers cost functions that are essentially as general as the distortion measures considered in information theoretic coding theorems. Results exist for general distortion measures/cost functions, but stronger and more useful results exist when the structure of metric spaces is added by concentrating on distortion measures which are positive powers of metrics. The problem for Kantorovich arose in the study of resource allocation in economics, work for which he later shared the 1974 Nobel prize for economics. The problem is an early example of linear programming, and Kantorovich is generally recognized as the inventor of linear programming. Kantorovich later realized [58] that his problem was a generalization of a 1781 problem of Monge [71], which can be described in in the current context as follows. Fix as before the marginal distributions µX , µY . Given a measurable mapping f : A → B, define as usual the induced measure µX f −1 by µX f −1 (G) = µX (f −1 (G)); G ∈ BB .

(9.5)

Define ˜ X , µY ) = ρ(µ

inf

f :µX f −1 =µY

EµX ρ(X, f (X)).

(9.6)

In this formulation the coupling of two random objects by a joint measure is replaced by a deterministic mapping of the first into the second.

272

9 Process Approximation and Metrics

This effectively constrains the optimization to the special case of deterministic mappings and the resulting optimization is no longer one with linear constraints, which results in increased difficulty of analysis and evaluation. The problem of (9.3) is widely known as the Monge/Kantorovich optimal transportation or transport problem, and when the cost function is a metric the resulting minimum average distortion has been called the Monge/Kantorovich distance. There is also a more general meaning of the name “Monge/Kantorovich distance,” as will be discussed. While it is the Kantorovich formulation that is most relevant here, the Monge formulation will provide interesting analogies in the sequel when coding schemes are introduced since the mapping f can be interpreted as a coding of X into Y . A significant portion of the optimal transportation literature deals with conditions under which the two optimizations yield the same values and the optimal coupling is in fact a deterministic mapping. When the idea was rediscovered in 1974 [74, 42] as part of an extension of Ornstein’s d- distance for random processes [76, 77, 78] (more about this later), it was called the ρ or “rho-bar distance.” It was recognized, however, that the finite dimensional case (random vectors) was Vasershtein’s distance [108], which had been popularized by Dobrushin [18], and, in [74], as the transportation distance. In the optimal transportation literature, the German spelling of Vasershtein, Wasserstein, also caught on as a suitable name for the Monge/Kantorovich distance and remains so. The Monge/Kantorovich optimal cost provides a distortion or distance between measures, random variables, and random vectors. It has found a wide variety of applications and there are several thorough modern books detailing the theory and applications. The interested reader is referred to [86, 87, 109, 110, 111] and the enormous number of references therein. The richness of the field is amply demonstrated by the more than 700 references cited in [111] alone! Many examples of evaluation, dual formulations, and methods for constructing the measures and functions involved have appeared in the literature. Names associated with the distance include Gini [29], Fréchet [25], Vasershtein/Wasserstein [108], Vallender [107], Dobrushin [18], Mallows [70], and Bickel and Freedman [2]. It was rediscovered in the computer science literature as the “earth mover’s distance” with an intuitive description close to that of the original transportation distance [92, 65]. Because of the plethora of contributing authors from different fields, the distortion presents a challenge of choosing both notation and a name. Because this book shares the emphasis on random processes with Ornstein and ergodic theory, the notation and nomenclature will generally follow the ergodic theory literature and the derivative information theory literature. Initially, however, we adopt the traditional names for

9.2 Optimal Coupling Distortion/Transportation Cost

273

the traditional setting, and much of the notation in the various fields is similar.

The Monge/Kantorovich Distance Assume that both µX and µY are distributions on random objects X and Y defined on a common Polish space (A, d). Define the distortion measure ρ = dp for p ≥ 0, where the p = 0 case is shorthand for the Hamming distance, that is, d0 (x, y) = dH (x, y) = 1 − δx,y .

(9.7)

Define the collection of measures Pp (A, B(A)) as the collection of all probability measures µ on (A, B(A)) for which there exists a point a∗ ∈ A such that Z dp (x, a∗ )dµ(x) < ∞.

(9.8)

The point a∗ is called a reference letter in information theory [27]. Define ( ρ(µX , µY ) 0≤p≤1 dp (µX , µY ) = ρ(µX , µY )min(1,1/p) = . (9.9) ρ 1/p (µX , µY ) 1 ≤ p The following lemma shows that dp is indeed a metric on Pp (A, B(A)). Lemma 9.4. Given a Polish space (A, d) with a Borel σ -field B(A), let Pp (A, B(A)) denote the collection of all Borel probability measures on the Borel measurable space (A, B(A)) with a reference letter. Then for any p ≥ 0, dp is a metric on Pp (A, B(A)). Proof. First consider the case with 0 ≤ p ≤ 1, in which case ρ is itself a metric. This immediately implies that ρ is nonnegative and symmetric in its arguments. To prove that the triangle inequality holds, suppose that µX , µY , and µZ are three distributions on (A, B(A)). Suppose that π approximately yields ρ(µX , µZ ) and that r approximately yields ρ(µZ , µY ): Eπ ρ(X, Z) ≤ ρ(µX , µZ ) +  Er ρ(Z, Y ) ≤ ρ(µZ , µY ) + . The existence of the reference letter ensures that the expectations are all finite. These two distributions can be used to construct a joint distribution q for the triple (X, Z, Y ) such that X → Z → Y is a Markov chain and such that the coordinate vectors have the correct pairwise and marginal distributions. In particular, r and µZ together imply a conditional distribution PY |Z and π and µX together imply a conditional distribution PZ|X .

274

9 Process Approximation and Metrics

Since the spaces are standard these can be chosen to be regular conditional probabilities and hence the distribution specified by its values on rectangles. If we define νx (F ) = pZ|X (F |x), then Z Z dµX (x) dνx (z)pY |Z (F |z). q(F × G × D) = D

G

For this distribution, since ρ is a metric, ρ(µX , µY ) ≤ Eq ρ(X, Y ) ≤ Eq ρ(X, Z) + Eq ρ(Z, Y ) = Eπ ρ(X, Z) + Er ρ(Z, Y ) ≤ ρ(µX , µZ ) + ρ(µZ , µY ) + 2, which proves the triangle inequality for ρ. Suppose that ρ = 0. This implies that there exists a π ∈ Pp (µX , µY ) such that π (x, y : x 6= y) = 0 since ρ(x, y) is 0 only if x = y. This is only possible, however, if µX (F ) = µY (F ) for all events F , that is, the two marginals are identical. Next assume that p ≥ 1, in which case ρ need not be a metric. Construct the measures π , r and q as before except that now π and r are chosen to satisfy 1/p

≤ ρ 1/p (µX , µZ ) + 

1/p

≤ ρ 1/p (µZ , µY ) + .

(Eπ ρ(X, Z))

(Er ρ(Z, Y ))

Again the existence of the reference letter ensures that the expectations are all finite. Then dp (µX , µY ) = ρ 1/p (µX , µY )  1/p  1/p ≤ Eq ρ(X, Y ) = Eq dp (X, Y ) 1/p  ≤ Eq [(d(X, Z) + d(Z, Y ))p ] 1/p

≤ (Eπ [dp (X, Z)])

1/p

+ (Er [dp (Z, Y )])

using Minkowski’s inequality (Lemma 5.8). Thus dp (µX , µY ) ≤ dp (µX , µZ ) + dp (µZ , µY ) + 2.

2

9.3 Coupling Discrete Spaces with the Hamming Distance Dobrushin [18] developed several interesting properties for the special case of the transportation or Monge/Kantorovich distance, which

9.3 Coupling Discrete Spaces with the Hamming Distance

275

he called the Vasershtein distance [108], with a Hamming distortion. The properties prove useful when developing examples of the process distance, so they are collected here. Although the properties are often quoted in the literature, they are usually given without proof or for more general continuous spaces where the proof is more complicated. For this section we consider probability measures µX and µY on a common discrete space A. We will use µX to denote both the measure and the PMF, e.g., µX (G) is the probability of a subset G ⊂ S and µ(x) = µ({x}) is the probability of the one point set {x}. Specifically, X and Y are random objects with alphabets AX = AY = A. Let (X, Y ) denote the identity mapping on the product space A × A and let X and Y denote the projections onto the component spaces given by X(x, y) = x, Y (x, y) = y. Denote as before the collection of all couplings π satisfying (9.2) by P(µX , µY ). The Hamming transportation distance is given by dH (µX , µY ) =

inf

π ∈P(µX ,µY )

dH (X, Y ) =

inf

π ∈P(µX ,µY )

π (X ≠ Y ).

Two other distances between µX and µY ) are considered here: The variation distance var(µX , µY ) = sup | µX (G) − µY (G) |, G

where the supremum is over all events G, and the distribution distance of Example 2.3 X | µX − µY |= | µX (x) − µY (x) | . x∈A

The variation (or variational or total variation) distance quantifies the notion of the maximum possible difference the two measures can assign to a single event. The distribution distance is simply an `1 norm of the difference of the two measures. Lemma 9.5. dH (µX , µY ) = var(µX , µY ) =

1 | µX − µY | . 2

Proof. Define the set B = {x : µX (x) ≥ µY (x)}. For any event G,

(9.10)

276

9 Process Approximation and Metrics

| µX (G) − µY (G) | = |

X

µX (x) −

x∈G

µY (x) |=|

x∈G

X

=|

X

(µX (x) − µY (x)) |

x∈G

X

(µX (x) − µY (x)) −

(µY (x) − µX (x)) |

x∈G∩B c

x∈G∩B

|

X

{z

|

}

a

{z b

}

The terms a and b are each positive. If for a given event G, a > b, then |a − b| = a − b. In this case µX (G) − µY (G) can be increased by removing all of the x ∈ B c from G and adding to G all of the x ∈ B, which implies that | µX (G) − µY (G) |≤ µX (B) − µY (B). Similarly, if a ≤ b, then | µX (G) − µY (G) | ≤ µY (B c ) − µX (B c = (1 − µY (B)) − (1 − µX (B)) = µX (B) − µY (B). Since one of these conditions must hold | µX (G) − µY (G) |≤ µX (B) − µY (B) for all G, which upper bound can be achieved by choosing G = B. Thus var(µX , µY ) = µX (B) − µY (B). Since X |µX (x) − µY (x)| | µX (G) − µY (G) | = x

=

X

(µX (x) − µY (x)) +

X

(µY (x) − µX (x))

x∈B c c

x∈B

= µX (B) − µY (B) + µY (B ) − µX (B c ) = µX (B) − µY (B) + (1 − µY (B)) − (1 − µX (B)) = 2(µX (B) − µY (B)), whence var(µX , µY ) =

1 | µX − µY | . 2

Suppose that π ∈ P(µX , µY ).Then since π (x, x) ≤ min(µX (x), µY (x)), X π (X ≠ Y ) = 1 − π (X = Y ) = 1 − π (x, x) x

≥ 1−

X

min(µX (x), µY (x))

x

1X (µX (x) + µY (x) − 2 min(µX (x), µY (x))) 2 x 1X 1 = | µX (x) − µY (x) |= | µX − µY | . 2 x 2

=

This lower bound is in fact achievable by defining π in terms of the event B of (9.10) for x = y by

9.4 Fidelity Criteria and Process Optimal Coupling Distortion

277

π (x, x) = min(µX (x), µY (x))

(9.11)

(µX (x) − µY (x))1B (x)(µY (x) − µX (x))1B c (x). . µX (B) − µY (B)

(9.12)

and for x ≠ y by

Thus dH (µX , µY ) =

1 |µX − µY |. 2

2

9.4 Fidelity Criteria and Process Optimal Coupling Distortion In information theory, ergodic theory, and signal processing, it is important to quantify the distortion between processes rather than only between probability distributions, random variables, or random vectors. This means that instead of having a single space of interest, one is interested in a limit of finite-dimensional spaces or a single infinitedimensional space. In this case one needs a family of distortion measures for the finite-dimensional spaces which will yield a notion of distortion and optimal coupling for the entire process. Towards this end, suppose next that we have a sequence of product spaces An and B n (with the associated product σ -fields) for n = 1, 2, · · · . A fidelity criterion ρn , n = 1, 2, · · · is a sequence of distortion measures on An × B n [95, 96]. If one has a pair random process, say {Xn , Yn }, then it will be of interest to find conditions under which there is a limiting per symbol distortion in the sense that 1 ρ∞ (x, y) = lim ρn (x n , y n ) n→∞ n exists. As one might guess, the distortion measures in the sequence need to be interrelated in order to get useful behavior. The simplest and most common example is that of an additive or single-letter fidelity criterion which has the form ρn (x n , y n ) =

n−1 X

ρ1 (xi , yi ).

i=0

Here if the pair process is AMS and ρ1 satisfies suitable integrability assumptions, then the limiting distortion will exist and be invariant from the ergodic theorem. More generally, if ρn is subadditive in the sense that

278

9 Process Approximation and Metrics

ρn (x n , y n ) ≤ ρk (x k , y k ) + ρn−k (xkn−k , ykn−k ), then stationarity of the pair process and integrability of ρ1 will ensure that n−1 ρn converges from the subadditive ergodic theorem. For example, if d is a distortion measure (possibly a metric) on A × B, then 1/p

 n

n−1 X

n

ρn (x , y ) = 

d(xi , yi )

p

i=0

for p > 1 is subadditive from Minkowski’s inequality. By far the bulk of the information theory literature considers only additive fidelity criteria and we will share this emphasis. The most common examples of additive fidelity criteria involve defining the per-letter distortion ρ1 in terms of an underlying metric d. The most commonly occurring examples are to set ρ1 (x, y) = d(x, y), in which case ρn is also a metric for all n, or ρ1 (x, y) = dp (x, y) for some p > 0. If 1 > p > 0, then again ρn is a metric for all n. If p ≥ 1, then ρn is not a metric, but 1/p ρn is a metric. We do not wish to include the 1/p in the definition of the fidelity criterion, however, because what we gain by having a metric distortion we more than lose when we take the expectation. For example, a popular distortion measure is the expected squared error, not the expectation of the square root of the squared error. Thanks to Lemma 9.4 we can still get the benefits of a distance even with the nonmetric distortion measure, however, by taking the appropriate root outside of the expectation. A small fraction of the information theory and communications literature consider the case of ρ1 = f (d) for a nonnegative convex function f , which is more general than ρ1 = dp if p ≥ 1. We will concentrate on the simpler case of a positive power of a distortion. We will henceforth make the following assumptions: • {ρn ; n = 1, 2, . . .} is an additive fidelity criterion defined by n

n

ρn (x , y ) =

n−1 X

dp (xi , yi ),

(9.13)

i=0

where d is a metric and p > 0. We will refer to this as the dp distortion. We will also allow the case p = 0 which is defined by ρ1 (x, y) = d0 (x, y) = dH (x, y), the Hamming distance dH (x, y) = 1 − δx,y . • The random process distributions µ considered will possess a reference letter in the sense of Gallager [27]: given µ there exists an a ∈ A for which

9.4 Fidelity Criteria and Process Optimal Coupling Distortion

Z

dp (x, a∗ )dµ 1 (x) < ∞.

279

(9.14)

This is trivially satisfied if the distortion is bounded. Define P p (A, d) to be the space of all stationary process distributions µ on (A, B(A))T satisfying (9.14), where both T = Z and Z+ will be considered. Note that previously Pp (A, d) referred to a space of distributions of random objects taking values in a metric space A. Now it refers to a space of process distributions with all individual component random variables taking values in a common metric space A, that is, the process distributions are on (A, B(A))T . In the process case we will consider Monge/Kantorovich distances on the spaces of random vectors with alphabets An for all n = 1, 2, . . ., but we shall define a process distortion and metric as the supremum over all possible vector dimensions.

Process Optimal Coupling Distortion Given two process distributions µX and µY describing random processes {Xn } and {Yn }, let the induced n-dimensional distributions be µX n , µY n . For each positive integer n we can define the n-dimensional or n-th order optimal coupling distortion as the distortion between the induced ndimensional distributions: ρ n (µX n , µY n ) =

inf

π ∈Pp (µX n ,µY n )

Eπ ρn (X n , Y n )

(9.15)

The process optimal coupling distortion (or ρ distortion) between µX and µY is defined by ρ(µX , µY ) = sup n

1 ρ (µX n , µY n ). n n

(9.16)

The extension of the optimal coupling distortion or transportation cost to processes was developed in the early 1970s by D.S. Ornstein for the case of ρ1 = dH and the resulting process metric, called the d distance, played a fundamental role in the development of the Ornstein isomorphism theorem of ergodic theory (see [76, 77, 78] and the references therein). The idea was extended to processes with Polish alphabets and metric distortion measures and the square of metric distortion measures in 1975 [42] and applied to problems of quantizer mismatch [35], Shannon information theory [40, 75, 41, 31, 43, 32], and robust statistics [81]. While there is a large literature on the finite-dimensional optimal coupling distance, the literature for the process optimal coupling distance seems limited to the information theory and ergodic theory literature.

280

9 Process Approximation and Metrics

Here the focus is on the process case, but the relevant finite-dimensional results are also treated as needed. The key aspect of the process distortion is that if it is small, then necessarily the distortion between all sample vectors produced by the two processes is also small.

The dp -distance The process distortion is clearly a metric for the important special case of an additive fidelity criterion with a metric per letter distortion. This subsection shows how a process metric can be obtained in the most important special case of an additive fidelity criterion with a per letter distortion given by a positive power of a metric. The result generalizes the special cases of process metrics in [77, 42, 106] and extends the well known finite-dimensional version of optimal transportation theory (e.g., Theorem 7.1 in [110]). Theorem 9.1. Given a Polish space (A, d) and p ≥ 0, define the additive fidelity criterion ρn : An × An → R+ ; n = 1, 2, . . . by ρn (x n , y n ) =

n−1 X

dp (xi , yi ),

n=0

where d0 is shorthand for the Hamming distance dH . Define min(1,1/p)

dp (µX , µY ) = sup n−1 ρ n n

(µX n , µY n ) = ρ min(1,1/p) (µX , µY ). (9.17)

Then dp is a metric on P p (A, d), the space of all stationary random processes with alphabet A. Comments: The theorem together with Lemma 9.4 says that if ρ1 = dp , then dp = ρ 1/p is a metric for both the vector and process case if p ≥ 1, and dp = ρ is a metric if 0 ≤ p ≤ 1. The two cases agree for p = 1 and the p = 0 case is simply shorthand for the p = 1 case with the Hamming metric. In the case of p = 0, the process distance d0 is Ornstein’s d-distance and the notation is usually abbreviated to simply d to match usage in the ergodic theory literature. It merits pointing out that d0 (µX n , µY n ) is not the transportation distance with a Hamming distance on An , it is the transportation distance with respect to the disPn−1 tance d(x n , y n ) = i=0 dH (xi , yi ), the sum of the Hamming distances between symbols. This is also n times the average Hamming distance of Example 1.4. The two metrics on An are related through the simple bounds

9.4 Fidelity Criteria and Process Optimal Coupling Distortion n−1 X

dH (xi , yi ) ≥ dH (x n , y n ) ≥

i=0

n−1 1 X dH (xi , yi ). n i=0

281

(9.18)

min(1,1/p)

Proof. Since the dp (µX n , µY n ) = ρ n (µX n , µY n ) are nonnegative and symmetric, so is their supremum dp (µX , µY ). Lemma 9.4 shows that for each n, dp (µX n , µY n ) satisfies the triangle inequality. Hence so does the supremum since 1 dp (µX n , µY n ) n n  1 ≤ sup dp (µX n , µZ n ) + dp (µZ n , µY n ) n n 1 1 ≤ sup dp (µX n , µZ n ) + sup dp (µZ n , µY n ) n n n n

dp (µX , µY ) = sup

= dp (µX , µZ ) + dp (µZ , µY ). Suppose that dp (µX , µY ) = 0 and hence also dp (µX n , µZ n ) = 0 for all n. As in the proof of Lemma 9.4, this implies that µX n = µY n for each n, and hence that the process distributions µX and µY must also be identical. 2 The next theorem collects several more properties of the dp distance, including the facts that the supremum defining the process distance is a limit, that the distance between IID processes reduces to the Monge/Kantorovich distance between the first order marginals, and a characterization of the process distance as an optimization over processes. Theorem 9.2. Suppose that µX and µY are stationary process distributions with a common standard alphabet A and that ρ1 = dp is a positive power of a metric on A and that ρn is defined on An in an additive fashion as before. Then (a)

limn→∞ n−1 ρ n (µX n , µY n ) exists and equals supn n−1 ρ n (µX n , µY n ). min(1,1/p)

(µX n , µY n ). Thus dp (µX , µY ) = limn→∞ n−1 ρ n (b) If µX and µY are both IID, then ρ(µX , µY ) = ρ 1 (µX0 , µY0 ) and hence min(1,1/p)

dp (µX , µY ) = ρ 1 (µX0 , µY0 ) (c) Let Ps = Ps (µX , µY ) denote the collection of all stationary distributions πXY having µX and µY as marginals, that is, distributions on {Xn , Yn } with coordinate processes {Xn } and {Yn } having the given distributions. Define the process distortion measure ρ 0 ρ 0 (µX , µY ) =

inf EπXY ρ(X0 , Y0 ).

πXY ∈Ps

Then ρ(µX , µY ) = ρ 0 (µX , µY );

282

9 Process Approximation and Metrics

that is, the limit of the finite dimensional minimizations is given by a minimization over stationary processes. (d) Suppose that µX and µY are both stationary and ergodic. Define Pe = Pe (µX , µY ) as the subset of Ps containing only ergodic processes, then ρ(µX , µY ) = inf EπXY ρ(X0 , Y0 ). πXY ∈Pe

(e)

Suppose that µX and µY are both stationary and ergodic. Let GX denote a collection of frequency-typical sequences for µX in the sense of Section 7.9. Recall that by measuring relative frequencies on frequencytypical sequences one can deduce the underlying stationary and ergodic measure that produced the sequence. Similarly let GY denote a set of frequency-typical sequences for µY . Define the process distortion measure ρ 00 (µX , µY ) =

inf

x∈GX ,y∈GY

lim sup n→∞

n−1 1 X ρ1 (x0 , y0 ). n i=0

Then ρ(µX , µY ) = ρ 00 (µX , µY ). that is, the ρ distortion gives the minimum long term time average distance obtainable between frequency-typical sequences from the two processes. (f) The infima defining ρ n and ρ 0 are actually minima. Proof. (a) Suppose that π N is a joint distribution on (AN ×AN , B(A)N × B(A)N ) describing (X N , Y N ) that approximately achieves ρ N , e.g., π N has µX N and µY N as marginals and for  > 0 Eπ N ρ(X N , Y N ) ≤ ρ N (µX N , µY N ) + . For any n < N let π n be the induced distribution for (X n , Y n ) and πnN−n that for (XnN−n , YnN−n ). Then since the processes are stationary π n ∈ Pn and πnN−n ∈ PN−n and hence Eπ N ρ(X N , Y N ) = Eπ n ρ(X n , Y n ) + EπnN−n ρ(XnN−n , YnN−n ) ≥ ρ n (µX n , µY n ) + ρ N−n (µX N−n , µY N−n ). Since  is arbitrary we have shown that if an = ρ n (µX n , µY n ), then for all N > n aN ≥ an + aN−n ; that is, the sequence an is superadditive. It then follows from Lemma 8.4 (since the negative of a superadditive sequence is a subadditive sequence) that

9.4 Fidelity Criteria and Process Optimal Coupling Distortion

lim

n→∞

283

an an = sup , n n≥1 n

which proves (a). (b) Suppose that π 1 approximately yields ρ 1 , that is, has the correct marginals and Eπ 1 ρ1 (X0 , Y0 ) ≤ ρ 1 (µX0 , µY0 ) + . Then for any n the product measure π n defined by 1 π n = ×n−1 i=0 π

has the correct vector marginals for X n and Y n since they are IID, and hence π n ∈ Pn , and therefore ρ n (µX n , µY n ) ≤ Eπ n ρn (X n , Y n ) = nEπ 1 ρ1 (X0 , Y0 ) ≤ n(ρ 1 (µX0 , µY0 ) + ). Since by definition ρ is the supremum of n−1 ρ n , this proves (c). (c) Given  > 0 let π ∈ Ps be such that Eπ ρ1 (X0 , Y0 ) ≤ ρ 0 (µX , µY ) + . The induced distribution on {X n , Y n } is then contained in Pn , and hence using the stationarity of the processes ρ n (µX n , µY n ) ≤ Eρn (X n , Y n ) = nEρ1 (X0 , Y0 ) ≤ nρ 0 (µX , µY ), and therefore ρ 0 ≥ ρ since  is arbitrary. Let π n ∈ Pn , n = 1, 2, . . . be a sequence of measures such that Eπ n [ρn (X n , Y n )] ≤ ρ n + n where n → 0 as n → ∞. Let qn denote the product measure on the sequence space induced by the πn , that is, for any N and N-dimensional rectangle F = ×i∈I Fi with all but a finite number N of the Fi being A × B, Y (j+1)n−1 q(F ) = π n (×i=jn Fi ); j∈I

that is, qn is the pair process distribution obtained by gluing together independent copies of π n . This measure is obviously N-stationary and we form a stationary process distribution πn by defining πn (F ) =

n−1 1 X qn (T −i F ) n i=0

for all events F . If we now consider the metric space of all process distributions on (AI × AI , B(A)I × B(A)I ) of Section 9.1 with a metric dG based on a countable generating field of rectangles (which exists since the spaces are standard), then from Lemma 9.2 the space

284

9 Process Approximation and Metrics

is sequentially compact and hence there is a subsequence pnk that converges in the sense that there is a measure π and πnk (F ) → π (F ) for all F in the generating field. Assuming the generating field to contain all shifts of field members, the stationarity of the πn implies that π (T −1 F ) = π (F ) for all rectangles F in the generating field and hence π is stationary. A straightforward computation shows that for all n π induces marginal distributions on X n and Y n of µX n and µY n and hence π ∈ Ps . In addition, Eπ ρ1 (X0 , Y0 ) = lim Eπnk ρ1 (X0 , Y0 ) k→∞

= lim

k→∞

n−1 k

nX k −1

Eqnk ρ1 (Xi , Yi )

i=0

= lim (ρ nk + nk ) = ρ, k→∞

proving that ρ 0 ≤ ρ and hence that they are equal. (d) If π ∈ Ps nearly achieves ρ 0 , then from the ergodic decomposition it can be expressed as a mixture of stationary and ergodic processes πλ and that Z Eπ ρ1 =

dW (λ)Eπλ ρ1 .

All of the πλ must have µX and µY as marginal process distributions (since they are ergodic) and at least one value of λ must yield an Eπλ ρ1 no greater than the preceding average. Thus the infimum over stationary and ergodic pair processes can be no worse than that over stationary processes. (Obviously it can be no better.) (e) Suppose that x ∈ GX and y ∈ GY . Fix a dimension n and define for each N and each F ∈ B(A)n , G ∈ B(A)n µN (F × G) =

N−1 1 X 1F ×G (T i (x, y)). N i=0

The set function µN extends to a measure on (An ×An , B(A)n ×B(A)n ). Since the sequences are assumed to be frequency-typical, µN (F × AI ) → µX n (F ), N→∞

µN (AI × G) → µY n (G), N→∞

for all events F and G and generating fields for B(A)n . From Lemma 9.2 (as used previously) there is a convergent subsequence of the µN . If µ is such a limit point, then for generating F , G µ(F × AI ) = µX n (F ) and µ(AI ×G) = µY n (G), which imply that the same relation must hold for all F and G (since the measures agree on generating events). Thus

9.4 Fidelity Criteria and Process Optimal Coupling Distortion

285

µ ∈ Pn . Since µN is just a sample distribution, EµN [ρn (X n , Y n )] = Thus

N−1 N−1 n−1 1 X 1 X X ρn (xjn , yjn ) = [ ρ1 (xi+j , yi+j )]. N j=0 N j=0 i=0

N−1 1 X 1 Eµ [ρn (X n , Y n )] ≤ lim sup ρ1 (xi , yi ) n N→∞ N i=0

and hence ρ ≤ ρ 0 . Choose for  > 0 a stationary measure p ∈ Ps such that Eπ ρ1 (X0 , Y0 ) = ρ 0 + . Since π is stationary, n−1 1 X ρ1 (xi , yi ) n→∞ n i=0

ρ∞ (x, y) = lim

exists with probability one. Furthermore, since the coordinate processes are stationary and ergodic, with probability one x and y must be frequency-typical. Thus ρ∞ (x, y) ≥ ρ 00 . Taking the expectation with respect to π and invoking the ergodic theorem, ρ 00 ≤ Eπ ρ∞ = Eπ ρ1 (X0 , Y0 ) ≤ ρ 0 + . Hence ρ 0 ≤ ρ 0 = ρ and hence ρ 0 = ρ. (g) Consider ρ 0 . Choose a sequence πn ∈ Ps such that Eπn ρ1 (X0 , Y0 ) ≤ ρ 0 (µX , µY ) + 1/n. From Lemma 9.2 πn must have a convergent subsequence with limit, say, π . The measure π must be stationary and have the correct marginal processes (since all of the πn do) and from the preceding equation Eπ ρ1 (X0 , Y0 ) ≤ ρ 0 (µX , µY ). A similar argument works for ρ n .

2

286

9 Process Approximation and Metrics

9.5 The Prohorov Distance In this section the focus is on metric distortion measures for simplicity. The section develops relations between the the Prohorov and transportation distances between random vectors. The Prohorov distance measure resembles the Monge/Kantorovich distance for random variables and has many applications in probability theory, ergodic theory, and statistics. (See, for example, Dudley [24], Billingsley [4] [5], Moser, et al. [72], Strassen [104], and Hample [50].) The Prohorov distance unfortunately does not extend in a useful way to provide a distance measure between processes and it does not ensure the frequency-typical sequence closeness properties that the ρ distortion possesses. We present a brief description, however, for completeness and to provide the relation between the two distance measures on random vectors. The development follows [81]. Analogous to the definition of the nth order ρ distance, we define for positive integer n the nth order Prohorov distance Πn = Πn (µX n , µY n ) between two distributions for random vectors X n and Y n with a common standard alphabet An and with a metric ρn defined on An by Πn = inf inf{γ : π ({x n , y n : π ∈P

1 ρn (x n , y n ) > γ}) ≤ γ} n

where P is as in the definition of ρ n , that is, it is the collection of all joint distributions µX n ,Y n having the given marginal distributions µX n and µY n . It is known from Strassen [104] and Dudley [24] that the infimum is actually a minimum. It can be proved in a manner similar to that for the corresponding ρ n result. Lemma 9.6. The ρ and Prohorov distances satisfy the following inequalities: 1 Πn (µX n , µY n )2 ≤ ρ n (µX n , µY n ) ≤ ρ(µX , µY ). (9.19) n If ρ1 is bounded, say by ρmax , then 1 ρ (µX n , µY n ) ≤ Πn (µX n , µY n )(1 + ρmax ). n n

(9.20)

If there exists an a∗ such that EµX0 (ρ1 (X0 , a∗ )2 ) ≤ ρ ∗ < ∞, EµY0 (ρ1 (Y0 , a∗ )2 ) ≤ ρ ∗ < ∞,

(9.21)

then 1 ρ (µX n , µY n ) ≤ Πn (µX n , µY n ) + 2(ρ ∗ Πn (µX n , µY n ))1/2 . n n

(9.22)

9.5 The Prohorov Distance

287

Comment: The Lemma shows that the ρ distance is stronger than the Prohorov in the sense that making it small also makes the Prohorov distance small. If the underlying metric either is bounded or has a reference letter in the sense of (9.21), then the two distances provide the same topology on nth order distributions. Proof. From Theorem 9.2 the infimum defining the ρ n distance is actually a minimum. Suppose that π achieves this minimum; that is, Eπ ρn (X n , Y n ) = ρ n (µX n , µY n ). For simplicity we shall drop the arguments of the distance measures and just refer to Πn and ρ n . From the Markov inequality p({x n , y n :

1 Eπ n ρn (X n , Y n ) 1 ρn (x n , y n ) > }) ≤ = n 

1 n ρn



.

Setting  = (n−1 ρ n )1 /2 then yields p({x n , y n :

q q 1 ρn (x n , y n ) > n−1 ρ n }) ≤ n−1 ρ n , n

proving the first part of the first inequality. The fact that ρ is the supremum of ρ n completes the first inequality. If ρ1 is bounded, let π yield Πn and write 1 1 ρ ≤ Eπ ρn (X n , Y n ) n n n ≤ Πn + π ({x n , y n :

1 ρn (x n , y n ) > Πn })ρmax n

= Πn (1 + ρmax ). Next consider the case where (9.21) holds. Again let π yield Πn , and write, analogously to the previous equations, Z 1 1 ρ n ≤ Πn + dp(x n , y n ) ρn (x n , y n ) n n G where G = {x n , y n : n−1 ρn (x n , y n ) > Πn }. Let 1G be the indicator of G and let E denote expectation with respect to π . Since ρn is a metric, the triangle inequality and the Cauchy-Schwartz inequality yield

288

9 Process Approximation and Metrics

1 1 ρ ≤ Eρn (X n , Y n ) n n n 1 1 ≤ Πn + E(ρn (X n , a∗n )1G ) + E( ρn (Y n , a∗n )1G ) n n 1 ≤ Πn + [E( ρn (X n , a∗n ))2 ]1/2 [E(12G )]1/2 n 1 +[E( ρn (Y n , a∗n ))2 ]1/2 [E(12G )]1/2 . n Applying the Cauchy-Schwartz inequality for sums and invoking (9.21) yield   n−1 X 1 1 ρ1 (Xi , a∗ ))2  E( ρn (X n , a∗n ))2 = E ( n n i=0 ≤ E( Thus

n−1 1 X ρ1 (Xi , a∗ )2 ) ≤ ρ ∗ . n i=0

1 ρ ≤ 2(ρ ∗ )1/2 (p(G))1/2 = Πn + 2(ρ ∗ )1/2 Π1/2 n , n n

2

completing the proof.

9.6 The Variation/Distribution Distance for Discrete Alphabets In the case of discrete alphabets and the average Hamming fidelity criterion, the resulting d0 -distance is simply Ornstein’s d distance and the variation distance provides some useful bounds and comparisons. For convenience we modify notation slightly and and let d = dH denote the symbol distortion ρ1 and consider the additive fidelity criterion ρn (x n , y n ) =

n−1 X

d(xi , yi )

i=0

and following the ergodic theory literature we will denote dn = ρn , dn (µX n , µY n ) = ρ n (µX n , µY n ), and d(µX , µY ) = ρ(µX , µY ). Then Lemma 9.5 and (9.18) immediately yield the following lemma. (See, e.g., Neuhoff [74].) Lemma 9.7. Given discrete-alphabet process distributions, for n = 1, 2, · · · dn (µX n , µY n ) ≥

1 1 | µX n − µY n | = dH (µX n , µY n ) ≥ dn (µX n , µY n ) (9.23) 2 n

9.7 Evaluating dp

289

with equality for n = 1. The lemma shows that for discrete alphabet random processes, the d distance on vectors is equivalent to the variation or distribution distance, if the dimension is fixed forcing either distance to be small makes the other one small as well. The conclusion is not true in the limit, however. Having a very small d distance between processes forces d/n to be small for all n, but it does not force the variation distance to be small. The lemma coupled with Theorem 9.2 provides a means of computing the d distance between two IID processes as d(µX , µY ) = d1 (µX0 , µY0 ) =

1 | µX0 − µY0 | . 2

(9.24)

Thus, for example, the distance between two binary Bernoulli processes with parameters p and q is simply |p − q|.

9.7 Evaluating dp There is a substantial literature on bounding and evaluating the various optimal coupling distances and distortions for random variables, vectors, and processes. Most of it is concerned with one-dimensional real valued random variables, which from Theorem 9.2 also yields the process metric for IID random processes with the given marginals. Lemma 9.7 solves the problem for IID discrete alphabet random processes with an additive fidelity criterion with the Hamming per symbol distance, that is, for the classic Ornstein d distance. Another evaluation example applicable to IID processes follows from a result for one-dimensional real random variables due to Hoeffding [51] and Fréchet [25]. Suppose that X and Y are real-valued random variables with distributions µX and µY and cumulative distribution functions (CDFs) FX (x) = µX ((−∞, x]) = Pr(X ≤ x) and FY (y), respectively, and that the distortion is squared error. Define the inverse CDF FX−1 (u) = infx:FX (x)>u x. Then Z1 ρ(µX , µY ) =

0

| FX−1 (u) − FY−1 (u) |2 du !1

Z1 d2 (µX , µY ) =

2

| 0

(9.25)

FX−1 (u)

−

FY−1 (u) |2

du

(9.26)

It is a standard result of elementary probability theory that if X is a realvalued random variable with CDF FX and if U is a uniformly distributed random variable on [0, 1], then the random variable FX−1 (U) has CDF FX . Hence one coupling for µX and µY is the distribution of the pair of

290

9 Process Approximation and Metrics

random variables (FX−1 (U ), FY−1 (U)). Thus the right hand side of (9.25) immediately gives an upper bound. The proof of equality can be found, e.g., in Chapter 3 of Rachev and Rüschendorf [87] or Theorem 2.18 of Villani [110].

Gaussian Processes Few examples of the exact evaluation of the distance between processes with memory are known. An exception is the ρ distortion with a squarederror distortion and hence also the d2 -distance between Gaussian processes, found by Gray, Neuhoff, and Shields [42]. As pointed out by Tchen [106], the development holds somewhat more generally, but the Gaussian case is the most important. The development here follows [42], but a few details are added and the Gaussian assumption is not made until needed. The development requires some basic spectral theory, but mostly the elementary kind sufficient for engineering developments of random processes. Suppose that {Xn } is a real-valued random process with process distribution µX and constant mean EµX (Xn ) = 0. The assumption of zero mean makes things much simpler and only zero mean processes will be considered for the remainder of this section. Define the autocorrelation function RX (n, k) = EµX (Xn Xk ) for all appropriate n, k. The process {Xn } is said to be weakly stationary or stationary in the wide sense or wide-sense stationary RX (n, k) depends on its arguments only through their difference (it is toeplitz), in which case we write RX (n, k) = RX (n − k). The autocorrelation is a nonnegative definite, symmetric function. Wide-sense stationarity is a generalization of stationarity, and obviously a stationary process is also weakly stationary. We consider the more general notion for the moment since it is sufficient for the basic results we are after. Assume that RX (k) is sufficiently well behaved to have a Fourier transform (e.g., at is absolutely summable or square summable with the appropriate mathematical machinery added). Define the power spectral density of the process as the Fourier transform of the autocorrelation: SX (f ) =

∞ X

1 1 RX (k)e−j2π f k , f ∈ [− , ]. 2 2 k=−∞

(9.27)

The autocorrelation can be recovered from the power spectral density by taking the inverse Fourier transform: Z 1/2 RX (k) =

−1/2

SX (f )ej2π f k .

(9.28)

9.7 Evaluating dp

291

See, e.g., [37] for an engineering-oriented development or [91] for an advanced treatment of the properties and applications of autocorrelation functions and power spectral densities along with the various ideas from linear systems introduced below. Here our interest is solely in its use in finding the d2 -distance between Gaussian processes and a related class of processes. Now assume there are two processes X and Y with process distributions µX and µY and power spectral densities SX and SY , respectively. Let π denote a coupling of the two processes and define the cross correlation function RXY (k, n) = Eπ (Xk Yn ). Assume that the two processes are jointly weakly sationary in the sense that they are both weakly stationary and, in addition, RXY (k, n) depends on its arguments only through their difference, in which case we write RXY (k, n) = RXY (k − n). Joint stationarity of the two processes implies joint wide sense stationarity. Again assume that the function is sufficiently well-behaved to have a Fourier transform, which we call the cross spectral density ∞ X SXY (f ) = RXY (k)e−j2π f k . (9.29) k=−∞

For any such coupling π , Eπ [(X0 − Y0 )2 ] = Eπ [X02 + Y02 − 2X0 Y0 ] = RX (0) + RY (0) − 2RXY (0) Z 1/2 = (SX (f ) + SY (f ) − 2SXY (f )) df . −1/2

An inequality of Rozanov [91] states that |SXY (f )|2 ≤ SX (f )SY (f ).

(9.30)

This implies that for any coupling π , Z 1/2 

2

Eπ [(X0 − Y0 ) ] ≥

−1/2

Z 1/2 = −1/2

q



SX (f ) + SY (f ) − 2 SX (f )SY (f ) df q q | SX (f ) − SY (f ) |2 df ,

(9.31)

the square of the `2 distance between the square roots of the power spectral densities. Since this is true for all couplings, it follows that "Z d2 (µX , µY ) ≥

1/2

−1/2

#1/2 q q 2 | SX (f ) − SY (f ) | df ,

(9.32)

292

9 Process Approximation and Metrics

so that the d2 distance is always bound below by the `2 distance between the square roots of the spectra. This bound can be achieved under certain circumstances, which means that then (9.32) actually yields the d2 distance between processes. Suppose that Xn has the following form: there is a sequence Un with RU (k) = 1 for k = 0 and 0 otherwise and a sequence hn for which X Xn = hn−k Uk for all n k

In words, Xn can be expressed as the output of a linear time invariant filter with an uncorrelated input (such as an IID input). We are being intentionally casual in the mathematics here in order to avoid the details required for appropriate generality. Things are simple, for example, in the two-sided case when hn has only a finite number of nonzero values. The sequence hn is called the Kronecker delta response (or discrete-time impulse response) of the system. If Un is IID, then Xn is a B-process. The time-invariant filter is just a linear time-invariant filter, so such processes might reasonably be dubbed linear B-processes. A fundamental result of the theory of Gaussian random processes is that every purely nondeterministic (or regular) stationary Gaussian random process has this form, that is, can be written as a linear filtering of an IID random process, which is also Gaussian. The Fourier transform of the Kronecker delta response, H(f ), is called the frequency response of the linear system and a classic result of linear systems analysis is that SX (f ) = |H(f )|2 SU (f ), which in our case reduces to SX (f ) = |H(f )|2 . Suppose that both X and Y can be expressed in this way, where X is formed by passing U through a linear filter with frequency response H(f ) and Y is formed by passing the same U through a linear filter with frequency response, say, G(f ). Thus, in particular, SX (f ) = |H(f )|2 and SY (f ) = |G(f )|2 . Standard linear systems analysis shows that the implied coupling by generating X and Y in this way, separate linear filtering of a common uncorrelated source, yields a cross spectral density satisfying q |SXY (f )| = |H(f )| × |G(f )| = SX (f )SY (f ),

(9.33)

which achieves the lower bound of (9.30) and hence shows that for such processes "Z d2 (µX , µY ) =

1/2

−1/2

#1/2 q q 2 | SX (f ) − SY (f ) | df .

(9.34)

This result was derived for stationary Gaussian random processes in [42], but the argument shows that the result holds more generally for any two weakly stationary processes with linear models driven by a common uncorrelated process. For example, this gives the optimal coupling and

9.8 Measures on Measures

293

the d2 -distance between any two “linear B-processes” formed by linear time-invariant filterings of a common IID processes, which could have Gaussian, uniform, or other marginal distributions.

9.8 Measures on Measures Measures can be placed on a suitably defined space of measures. To accomplish this we consider the distributional distance for a particular G and hence on a particular metric for generating the Borel field of measures. We require that (Ω, B) be standard and take G to be a standard generating field. We will denote the resulting metric simply by d. Thus P(Ω, B) will itself be Polish from Lemma 9.1. We will usually wish to assign nonzero probability only to a nice subset of this space, e.g., the ˆ be a closed and consubset of all stationary measures. Henceforth let P ˆ is a closed subset of a Polish space, it vex subset of P(Ω, B). Since P ˆ = B(P, ˆ d) denote the Borel is also Polish under the same metric. Let B ˆ with respect to the metric d. Since P ˆ is Polish, this field of subsets of P ˆ B) ˆ is standard. measurable space (P, ˆ B). ˆ Now consider a probability measure W on the standard space (P, ˆ B) ˆ of all This probability measure thus is a member of the class P(P, ˆ B), ˆ where P ˆ probability measures on the standard measurable space (P, is a closed convex subset of the collection of all probability measures on the original standard measurable space (Ω, B). ˆ via the We now argue that any such W induces a new measure m ∈ P mixture relation Z m = αdW (α). (9.35) To be precise and to make sense of the preceding formula, consider for ˆ the expectation Eα 1F . We can any fixed F ∈ G and any measure α ∈ P consider this as a function (or functional) mapping probability measures ˆ into R, the real line. This function is continuous, however, since in P given  > 0, if F = Fi we can choose d(α, m) < 2−i and find that |Eα 1F − Em 1F | = |α(F ) − m(F )| ≤ 2i

∞ X

2−j |α(Fj ) − m(Fj )| ≤ .

j=1

Since the mapping is continuous, it is measurable from Lemma 2.1. Since it is also bounded, the following integrals all exist: Z m(F ) = Eα 1F dW (α); F ∈ G. (9.36)

294

9 Process Approximation and Metrics

This defines a set function on G that is nonnegative, normalized, and finitely additive from the usual properties of integration. Since the space (Ω, B) is standard, this set function extends to a unique probability measure on (Ω, B), which we also denote by m. This provides a mapping ˆ B) ˆ to a unique measure m ∈ P ˆ ⊂ P(Ω, B). from any measure W ∈ P(P, This mapping is denoted symbolically by (9.35) and is defined by (9.36). We will use the notation W → m to denote it. The induced measure m is called the barycenter of W .

Chapter 10

The Ergodic Decomposition

Abstract In previous chapters it was shown that an asymptotically mean stationary dynamical system or random process has an ergodic decomposition, which means that the sample averages imply a stationary and ergodic process distribution with the same ergodic properties. This followed because AMS systems or processes have ergodic properties — convergent sample averages — for all bounded measurements, and this in turn implied an ergodic decomposition. In this chapter an alternative form for the ergodic decomposition is derived which clarifies its nature as the random selection of a stationary and ergodic system, the ergodic decomposition is applied to Markov processes, and an ergodic decomposition for affine, lower semicontinuous functionals of random processes in terms of the distributional distance is derived.

10.1 The Ergodic Decomposition Revisited Say we have an stationary dynamical system (Ω, B, m, T ) with a standard measurable space. The ergodic decomposition then implies a mapping ψ : Ω → P((Ω, B)) defined by x → px as given in the ergodic decomposition theorem. In fact, this mapping is into the subset of ergodic and stationary measures. In this section we show that this mapping is measurable, relate integrals in the two spaces, and determine the barycenter of the induced measure on measures. In the process we obtain an alternative statement of the ergodic decomposition. Lemma 10.1. Given a stationary system (Ω, B, m, T ) with a standard measurable space, let P = P(Ω, B) be the space of all probability measures on (Ω, B) with the topology of Section 9.8. Then the mapping ψ : Ω → P defined by the mapping x → px of the ergodic decomposition is a measurable mapping. (All generic points x are mapped into the

R.M. Gray, Probability, Random Processes, and Ergodic Properties, DOI: 10.1007/978-1-4419-1090-5_10, © Springer Science + Business Media, LLC 2009

295

296

10 The Ergodic Decomposition

corresponding ergodic component; all remaining points are mapped into a common arbitrary stationary ergodic process.) Proof. To prove ψ measurable, it suffices to show that any set of the form V = {β : d(α, β) < r } has an inverse image ψ−1 (V ) ∈ B. This inverse image is given by {x :

∞ X

2−i |px (Fi ) − α(Fi )| < r }.

i=1

Defining the function γ : Ω → < by γ(x) =

∞ X

2−i |px (Fi ) − α(Fi )|,

i=1

then the preceding set can also be expressed as γ −1 ((−∞, r )). The function γ, however, is a sum of measurable real-valued functions and hence is itself measurable, and hence this set is in B. Given the ergodic decomposition {px } and a stationary measure m, ˆ B). ˆ Given any measurable function define the measure W = mψ−1 on (P, ˆ → <, from the measurability of ψ and the chain rule of integration f :P Z Z f (α)dW (α) = f (α)dmψ−1 (α) Z Z = f (ψx) dm(x) = f (px ) dm(x). Thus the rather abstract looking integrals on the space of measures equal the desired form suggested by the ergodic decomposition, that is, one integral exists if and only if the other does, in which case they are equal. This demonstrates that the existence of certain integrals in the more abstract space implies their existence in the more concrete space. The more abstract space, however, is more convenient for developing the desired results because of its topological structure. In particular, the ergodic decomposition does not provide all possible probability measures on the original probability space, only the subset of ergodic and stationary measures. The more general space is required by some of the approximation arguments. For example, one cannot use the separability lemma of Section 9.8 to approximate arbitrary ergodic processes by point masses on ergodic processes since a mixture of distinct ergodic processes is not ergodic! Letting f (α) = Eα 1F = α(F ) for F ∈ G we see immediately that Z Z Eα 1F dmψ−1 (α) = px (F ) dm(x), and hence since the right side is

10.1 The Ergodic Decomposition Revisited

Z m(F ) =

297

Eα 1F dmψ−1 (α);

that is, m is the barycenter of the measure induced on the space of all measures by the original stationary measure m and the mapping defined by the ergodic decomposition. Since the ergodic decomposition puts probability one on stationary and ergodic measures, we should be able to cut down the measure W to these sets and hence confine interest to functionals of stationary measures. In order to do this we need only show that these sets are measurable, that is, in the Borel field of all probability measures on (Ω, B). This is accomplished next. Lemma 10.2. Given the assumptions of the previous lemma, assume also that G is chosen to include a countable generating field and that G is −1 chosen S so that F ∈ G implies T F ∈ G. (If this is not true, simply expand G to i T −i G. For example, G could be a countable generating family of ˆ consisting respectively cylinders.) Let Ps and Pe denote the subsets of P of all stationary measures and all stationary and ergodic measures. Then Ps is closed (and hence measurable) and convex. Pe is measurable. Proof. Let αn ∈ Ps be a sequence of stationary measures converging to a measure α and hence for every F ∈ G αn (F ) → α(F ). Since T −1 F ∈ G this means that αn (T −1 F ) → α(T −1 F ). Since the αn are stationary, however, this sequence is the same as αn (F ) and hence converges to α(F ). Thus α(T −1 F ) = α(F ) on a generating field and hence α is stationary. Thus Ps is a closed subset under d and hence must be in the Borel field. From Lemma 7.15, an α ∈ Ps is also in Pe if and only if \ α∈ G(F ), F ∈G

where here G(F ) = {m : lim n

−1

n→∞

= {m : lim n−1 n→∞

n−1 X

m(T −i F ∩ F ) = m(F )2 }

i=0 n−1 X

Em 1T −i F ∩F = (Em 1F )2 }.

i=0

The Em 1F terms are all measurable functions of m since the sets are all in the generating field. Hence the collection of m for which the limit exists and has the given value is also measurable. Since the G(F ) are all ˆ so is their countable intersection. in B, ˆ to be Ps and observe that for any AMS measure Thus we can take P W (Ps ) = m(x : px ∈ Ps ) = 1 from the ergodic decomposition. 2

298

10 The Ergodic Decomposition

The following theorem uses the previous results to provide an alternative and intuitive statement of the ergodic decomposition theorem. Theorem 10.1. Fix a standard measurable space (Ω, B) and a transformation T : Ω → Ω. Then there are a standard measurable space (Λ, L), a family of stationary ergodic measures {mλ ; λ ∈ Λ} on (Ω, B), and a mapping ψ : Ω → Λ such that (a) ψ is measurable and invariant, and (b) if m is a stationary measure on (Ω, B) and Wψ is the induced distribution; that is, Wψ (G) = m(ψ−1 (G)) for G ∈ Λ (which is well defined from (a)), then Z Z m(F ) = dm(x)mψ(x) (F ) = dWψ (λ)mλ (F ), all F ∈ B, and if f ∈ L1 (m), then so is

R

f dmλ mψ-a.e. and Z Em f = dWψ (λ)Emλ f .

Finally, for any event F mψ (F ) = m(F |ψ), that is, given the ergodic decomposition and a stationary measure m , the ergodic component λ is a version of the conditional probability under m given ψ = λ. Proof. Let P denote the space of all probability measures described in Lemma 9.2 along with the Borel sets generated by using the given distance and a generating field. Then from Lemma 9.1 the space is Polish. Let Λ = Pe denote the subset of P consisting of all stationary and ergodic measures. From Lemma 9.1 this set is measurable and hence is a Borel set. Theorem 4.3 then implies that Λ together with its Borel sets is standard. Define the mapping ψ : Ω → P as in Lemma 10.1. From Lemma 10.1 the mapping is measurable. Since ψ maps points in Ω into stationary and ergodic processes, it can be considered as a mapping ψ : Ω → Λ that is also measurable since it is the restriction of a measurable mapping to a measurable set. Simply define mλ = λ (since each λ is a stationary ergodic process). The formulas for the probabilities and expectations then follow from the ergodic decomposition theorem and change of variable formulas. The conditional probability m(F |ψ = λ) is defined by the relation Z m(F ∩ {ψ ∈ G}) = m(F |ψ = λ)dWψ (λ), G

for all G ∈ L. Since ψ is invariant, σ (ψ) is contained in the σ -field of all invariant events. Combining the ergodic decomposition theorem and is the σ -field the ergodic theorem we have that Epx f = Em (f |I), where IP of invariant events. (This follows since both equal lim n−1 i f T i .) Thus the left-hand side is

10.2 The Ergodic Decomposition of Markov Processes

m(F ∩ ψ−1 (G)) =

299

Z ψ−1 (G)

px (F ) dm(x)

Z

Z

= ψ−1 (G)

mψ(x) (F )dm(x) =

G

mλ (F )dWψ (λ),

where the first equality follows using the conditional expectation of f = 1F , and the second follows from a change of variables. The theorem can be viewed as simply a reindexing of the ergodic components and a description of a stationary measure as a weighted average of the ergodic components where the weighting is a measure on the indices and not on the original points. In addition, one can view the function ψ as a special random variable defined on the system or process that tells which ergodic component is in effect and hence view the ergodic component as the conditional probability with respect to the stationary measure given the ergodic component in effect. For this reason we will occasionally refer to the ψ of Theorem 10.1 as the ergodic component function.

10.2 The Ergodic Decomposition of Markov Processes The ergodic decomposition has some rather surprising properties for certain special kinds of processes. Suppose we have a process {Xn } with standard alphabet and distribution m. For the moment we allow the process to be either one-sided or two-sided. The process is said to be kth order Markov or k-step Markov if for all n and all F ∈ σ (Xn , . . .) m((Xn , Xn+1 , . . .) ∈ F |Xi ; i ≤ n − 1) = m((Xn , Xn+1 , . . .) ∈ F |Xi ; i = n − k, . . . , n − 1); that is, if the probability of a turevent given the entire past depends only on the k most recent samples. When k is 0 the process is said to be memoryless. The following provides a test for whether or not a process is Markov. Lemma 10.3. A process {Xn } is k-step Markov if and only if the conditional probabilities m((Xn , Xn+1 , . . .) ∈ F |Xi ; i ≤ n − 1) are measurable with respect to σ (Xn−k , · · · , Xn−1 ), i.e., if for all F m((Xn , Xn+1 , . . .) ∈ F |Xi ; i ≤ n − 1) depends on Xi ; i ≤ n − 1 only through Xn−k , . . . , Xn−i . Proof. By iterated conditional probabilities, E(m((Xn , Xn+1 , . . .) ∈ F |Xi ; i ≤ n − 1)|Xn−k , . . . , Xn−1 ) = m((Xn , Xn+1 , . . .) ∈ F |Xi ; i = n − k, . . . , n − 1).

300

10 The Ergodic Decomposition

Since the spaces are standard, we can treat a conditional expectation as an ordinary expectation, that is, an expectation with respect to a regular conditional probability. Thus if the argument ζ = m((Xn , Xn+1 , . . .) ∈ F |Xi ; i ≤ n − 1) is measurable with respect to σ (Xn−k , . . . , Xn−1 ), then from Corollary 6.4 the conditional expectation of ζ is ζ itself and hence the left-hand side equals the right-hand side, which in turn implies that the process is k-step Markov. This conclusion can also be drawn by using Lemma 6.12. 2 If a k-step Markov process is stationary, then it has an ergodic decomposition. What do the ergodic components look like? In particular, must they also be Markov? First note that in the case of k = 0 the answer is trivial: If the process is memoryless and stationary, then it must be IID, and hence it is strongly mixing and hence it is ergodic. Thus the process puts all of its probability on a single ergodic component, the process itself. In other words, a stationary memoryless process produces with probability one an ergodic memoryless process. Markov processes exhibit a similar behavior: We shall show in this section that a stationary k-step Markov process puts probability one on a collection of ergodic k-step Markov processes having the same transition or conditional probabilities. The following lemma will provide an intuitive explanation. Lemma 10.4. Suppose that {Xn } is a two-sided stationary process with distribution m. Let {mλ ; λ ∈ Λ} denote the ergodic decomposition for a standard space Λ of Theorem 10.1 and let ψ be the ergodic component function. Then the mapping ψ of Theorem 10.1 can be taken to be measurable with respect to σ (X−1 , X−2 , . . .). Furthermore, m((X0 , X1 , . . .) ∈ F |X−1 , X−2 , . . .) = m((X0 , X1 , . . .) ∈ F |ψ, X−1 , X−2 , . . .) = mψ ((X0 , X1 , . . .) ∈ F |X−1 , X−2 , . . .), m-a.e. Comment: The lemma states that we can determine the ergodic component of a two-sided process by looking at the past alone. Since the past determines the ergodic component, the conditional probability of the future given the past and the component is the same as the conditional probability given the past alone. Proof. The ergodic component in effect can be determined by the limiting sample averages of a generating set of events. The mapping of Theorem 10.1 produced a stationary measure ψ(x) = px specified by the relative frequencies of a countable number of generating sets computed from x. Since the finite-dimensional rectangles generate the complete σ -field and since the probabilities under stationary measures are not affected by shifting, it is enough to determine the probabilities of all events of the form Xi ∈ Fi ; i = 0, . . . , k − 1 for all one-dimensional generating sets Fi . Since the process is stationary, the shift T is invertible, and

10.2 The Ergodic Decomposition of Markov Processes

301

hence for any event F of this type we can determine its probability from the limiting sample average n−1 1 X 1F (T −i x) n→∞ n i=k

lim

(we require in the definition of generic sets that both these time averages as well as those with respect to T converge). Since these relative frequencies determine the ergodic component and depend only on x−1 , x−2 , . . ., ψ has the properties claimed (and still satisfies the construction that was used to prove Theorem 10.1, we have simply constrained the sets considered and taken averages with respect to T −1 ). Since ψ is measurable with respect to σ (X−1 , X−2 , . . .), the conditional probability of a future event given the infinite past is unchanged by also conditioning on ψ. Since conditioning on ψ specifies the ergodic component in effect, these conditional probabilities are the same as those computed using the ergodic component. (The final statement can be verified by plugging into the defining equations for conditional probabilities.) 2 We are now ready to return to k-step Markov processes. If m is twosided and stationary and k-step Markov, then from the Markov property and the previous lemma m((X0 , . . .) ∈ F |X−1 , . . . , X−k ) = m((X0 , . . .) ∈ F |X−1 , X−2 , . . .) = mψ ((X0 , . . .) ∈ F |X−1 , X−2 , . . .). But this says that mψ ((X0 , . . .) ∈ F |X−1 , X−2 , . . .) is measurable with respect to σ (X−1 , . . . , X−k ) (since it is equal to a function that is), which implies from Lemma 10.3 that it is k-step Markov. We have now proved the following lemma for two-sided processes. Lemma 10.5. If {Xn } is a stationary Markov process and {mλ ; λ ∈ Λ} is the ergodic decomposition, and ψ is the ergodic component function, then mψ ((Xn , . . .) ∈ F |Xi ; i ≤ n) = m((Xn , . . .) ∈ F |Xn−k , . . . , Xn−1 ) almost everywhere. Thus with probability one the ergodic components are also k-step Markov with the same transition probabilities. Proof. For two-sided processes the statement follows from the previous argument and the stationarity of the measures. If the process is onesided, then we can embed the process in a two-sided process with the same behavior. Given a one-sided process m, let p be the stationary twosided process for which p((Xk , . . . , Xk+n−1 ) ∈ F ) = m((X0 , . . . , Xn−1 ) ∈ F ) for all integers k and n dimensional sets F . This uniquely specifies the two-sided process. Similarly, given any stationary two-sided process there is a unique one-sided process assigning the same probabilities to

302

10 The Ergodic Decomposition

finite-dimensional events. It is easily seen that the ergodic decomposition of the one-sided process yields that for the two-sided process. The one-sided process is Markov if and only if the two-sided process is and the transition probabilities are the same. Application of the two-sided result to the induced two-sided process then implies the one-sided result.

2

10.3 Barycenters Barycenters have some useful linearity and continuity properties. These are the subject of this section. To begin recall from Section 9.2 and (9.35) that the barycentric mapˆ B), ˆ the space of probability ping is a mapping of a measure W ∈ P(P, ˆ into a measure m ∈ P, ˆ that is, m is a probameasures on the space P, bility measure on the original space (Ω, B). The mapping is denoted by R W → m or by m = α dW (α). First observe that if W1 → m1 and W2 → m2 and if λ ∈ (0, 1), then λW1 + (1 − λ)W2 → λm1 + (1 − λ)m2 ; that is, the mapping W → m of a measure into its barycenter is affine. The following lemma provides a countable version of this property. Lemma 10.6. If Wi → mi , i = 1, 2, . . ., and if ∞ X

ai = 1

i=1

for a nonnegative sequence {ai }, then ∞ X i=1

ai Wi →

∞ X

ai m i .

i=1

Comment: The given countable mixtures of probability measures are easily seen to be themselves probability measures. As an example of the construction, suppose that W is described by a probability mass function, that is, W places a probability ai on a probability measure αi ∈ Pˆ for each i. We can then take the Wi of the lemma as the point masses placing probability 1 on αi . Then Wi → αi and P the mixture W of these point masses has the corresponding mixture αi as barycenter. Proof. Let W denots the weighted average of the Wi and m the weighted average of the mi . The assumptions imply that for all i

10.3 Barycenters

303

Z mi (F ) =

Eα 1F dWi (α); F ∈ G.

(That Eα 1F is measurable was discussed prior to (9.36).) Thus to prove the lemma we must prove that ∞ X

Z

Z Eα 1F dWi (α) =

ai

Eα 1F dW (α).

i=1

for all F ∈ G. We have by construction that for any indicator function of ˆ that an event D ∈ B ∞ X

Z

Z

ai

1D (α)dWi (α) =

1D dW .

i=1

Via the usual integration arguments this implies the corresponding equality for simple functions and then for bounded nonnegative measurable functions and hence for the Eα 1F . 2 The next lemma shows that if a measure W puts probability 1 on a ˆ then the barycenter must be inside the ball. closed ball in P, Lemma 10.7. Given a closed ball S = {α : d(α, β) ≤ r } in P((Ω, B)) and a measure W such that W (S) = 1, then W → m implies that m ∈ S. Proof. We have d(m, β) =

=

≤

∞ X

2−i |m(Fi ) − β(Fi )|

i=1 ∞ X i=1 ∞ X i=1

2−i |

Z S

2−i

α(Fi )dW (α) − β(Fi )|

Z S

|α(Fi ) − β(Fi )| dW (α)

Z d(α, β) dW (α) ≤ r ,

= S

where the integral is pulled out of the sum via the dominated conver2 gence theorem. This completes the proof. The final result of this section provides two continuity properties of the barycentric mapping. Lemma 10.8. (a) Assume as earlier that the original measurable space ˆ of all measures (Ω, B) is standard. Form a metric space from the subset P on (Ω, B) using a distributional metric d = dW with W a generating ˆ is the resulting Borel field, then the Borel space (P, ˆ B) ˆ field for B. If B is also standard. Given these definitions, the barycentric mapping W →

304

10 The Ergodic Decomposition

m is continuous when considered as a mapping from the metric space ˆ B), ˆ dG ) to the metric space (P, ˆ d), where dG is the metric based (P(P, on the countable collection of sets G = F ∪ S, where F is a countable ˆ of subsets of P ˆ and where generating field of the standard sigma-field B S is the collection of all sets of the form {α : α(F ) = Eα 1F ∈ [

k k+1 , )} n n

for all F in the countable field that generates B and all n = 1, 2, . . ., and ˆ since the functions defining k = 0, 1, . . . , n − 1. (These sets are all in B them are measurable.) (b) Given the preceding and a nonnegative bounded measurable funcˆ → < mapping probability measures into the real line, define tion D : P the class H of subsets P as all sets of the form {α :

D(α) k k+1 )} ∈[ , Dmax n n

where Dmax is the maximum absolute value of D, n = 1, 2, . . ., and k = ˆ B)) ˆ induced by 0, + − 1, . . . , + − n − 1. Let dG denote the metric on P((P, R the class of sets G = F ∪ H , with F and H as before. Then D(α)dW (α) is continuous with respect to dG ; that is, if dG (Wn , W ) →n→∞ 0, then also Z Z lim D(α) dWn (α) = D(α) dW (α). n→∞

Comment: We are simply adding enough sets to the metric to make sure that the various integrals have the required limits. Proof. (a) Define the uniform quantizer qn (r ) =

k k+1 k ; r ∈[ , ). n n n

Assume that dG (WN , W ) → 0 and consider the integral |mN (F ) − m(F )| Z Z = | Eα 1F dWN (α) − Eα 1F dW (α)| Z

Z

Z

Eα 1F dWN (α) − qn (Eα 1F ) dWN (α)| + | qn (Eα 1F ) dWN (α) Z Z Z − qn (Eα 1F ) dW (α)| + | qn (Eα 1F ) dW (α) − Eα 1F dW (α)|.

≤|

The uniform quantizer provides a uniformly good approximation to within 1/n and hence given  we can choose n so large that the leftmost and rightmost terms of the bound are each less than, say, /3 uniformly

10.4 Affine Functions of Measures

305

in N. The center term is the difference of the integrals of a fixed simple function. Since S was included in G, taking N → ∞ forces the probabilities under Wn of these events to match the probabilities under W and hence the finite weighted combinations of these probabilities goes to zero as N → ∞. In particular N can be chosen large enough to ensure that this term is less than /3 for the given n. This proves (a). Part (b) follows in essentially the same way. 2

10.4 Affine Functions of Measures We now are prepared to develop the principal result of this chapter. Let Pˆ ⊂ P((Ω, B)) denote as before a convex closed subset of the space of all probability measures on a standard measurable space (Ω, B). Let d be the metric on this space induced by a standard generating field of ˆ → < is said to subsets of Ω. A real-valued function (or functional) D : P ˆ affine be measurable if it is measurable with respect to the Borel field B, ˆ and λ ∈ (0, 1) we have if for every α1 , α2 ∈ P D(λα1 + (1 − λ)α2 ) = λD(α1 ) + (1 − λ)D(α2 ), and upper semicontinuous if d(αn , α) → 0 ⇒ D(α) ≥ lim sup D(αn ). n→∞

n→∞

If the inequality is reversed and the limit supremum replaced by a limit infimum, then D is said to be lower semicontinuous. An upper or lower semicontinuous function is measurable. We will usually assume that D is nonnegative, that is, that D(m) ≥ 0 for all m. ˆ → < is a nonnegative upper semicontinuous affine Theorem 10.2. If D : P functional on a closed and convex set of measures on a standard space (Ω, B), if D(α) is W -integrable, and if m is the barycenter of W , then Z D(m) = D(α) dW (α). Comment: The import of the theorem is that if a functional of probability measures is affine, then it can be expressed as an integral. This is a sort of converse to the usual implication that a functional expressible as an integral is affine. Proof. We begin by considering countable mixtures. Suppose that W is a measure assigning probability to a countable number of measures e.g., W assigns probability ai to αi for i = 1, 2, , . . .. Since the barycenter of W is

306

10 The Ergodic Decomposition

Z m=

αdW (α) =

∞ X

αi ai ,

i=1

and D(

∞ X

αi ai ) = D(m).

i=1

As in Lemma 10.6, we consider a slightly more general situation. Suppose ˆ B)), ˆ the corresponding that we have a sequence of measures Wi ∈ P((P, barycenters mi , and of real numbers ai that sum P a nonnegative sequence P to one. Let W = i ai Wi and m = i ai mi . From Lemma 10.6 W → m. Define for each n ∞ X bn = ai i=n+1

and the probability measures ∞ X

−1 m(n) = bn

ai m i ,

i=n+1

and hence we have a finite mixture m=

n X

mi ai + bn m(n) .

i=1

From the affine property D(m) = D(

n X

n X

ai mi + bn m(n) ) =

i=1

ai D(mi ) + bn D(m(n) ).

(10.1)

i=1

Since D is nonnegative, this implies that D(m) ≥

∞ X

D(mi )ai .

(10.2)

i=1

Making the Wi point masses as in the comment following Lemma 10.6 yields the fact that ∞ X D(m) ≥ D(αi )ai . (10.3) i=1

Observe that if D were bounded, but not necessarily nonnegative, then (10.1) would imply the countable version of the theorem, that is, that D(m) =

∞ X i=1

D(αi )ai .

10.4 Affine Functions of Measures

307

First fix M large and define DM (α) = min(D(α), M). Note that DM may not be affine (it is if D is bounded and M is larger than the maximum). DM is clearly measurable, however, and hence we can apply the metric of part (b) of the previous lemma together with Lemma 9.1 to get a sequence of measures Wn consisting of finite sums of point masses (as in Lemma 9.1) with the property that if Wn → W and mn is the barycenter of Wn , then also mn → m (in d) and Z Z DM (α)dW (α) DM (α)dWn (α) → n→∞

as promised in the previous lemma. From upper semicontinuity D(m) ≥ lim sup D(mn ), n→∞

and from (10.3) Z D(mn ) ≥

D(α)dWn (α).

Since D(α) ≥ DM (α), therefore Z Z D(m) ≥ lim sup DM (α)dWn (α) = DM (α)dW (α). n→∞

Since DM (α) is monotonically increasing to D(α), from the monotone convergence theorem Z D(m) ≥ D(α)dW (α), which half proves the theorem. ˆ into a For each n = 1, 2, . . . carve up the separable (under d) space P sequence of closed balls S1/n (αi ) = β : d(αi , β) ≤ 1/n; i = 1, 2, , . . . Call the collection of all such sets G = {Gi , i = 1, 2, . . .}. We use this collection to construct a sequence of finite partitions Vn = {Vi (n), i = 0, 1, 2, . . . , Nn }, n=1,2, . . . . Let Vn be the collection of all nonempty intersection sets of the form n \ Gi∗ , i=1

where GTi∗ is either Gi or Gic . For convenience we always take V0 (n) as n the set i=1 Gic of all points in the space not yet included in one of the first n Gi (assuming that the set is not empty). Note that S1/1 (αi ) is the whole set since d is bound above by one. Observe also that Vn has no more than 2n atoms and that each Vn is a refinement of its predecessor.

308

10 The Ergodic Decomposition

Finally note that for any fixed α and any  > 0 there will always be for sufficiently large n an atom in Vn that contains α and that has diameter less than . If we denote by V n (α) the atom of Vn that contains α (and one of them must), this means that ˆ lim diam(V n (α)) = 0; all α ∈ P.

n→∞

(10.4)

For each n, define for each V ∈ Vn for which W (V ) > 0 the elementary conditional probability W V ; that is, W V (F ) = W (V ∩ F )/W (V ). Break up W using these elementary conditional probabilities as X W = W (V )W V . V ∈Vn ,W (V )>0

For each such V let mV denote the barycenter of W V , that is, the mixture of the measures in V with respect to the measure W V that gives probability one to V . Since W has barycenter m and since it has the finite mixture form given earlier, Lemma 10.6 implies that X mV W (V ); m= V ∈Vn ,W (V )>0

that is, for each n we can express W as a finite mixture of measures W V for V ∈ Vn and m will be the corresponding mixture of the barycenters mV . Thus since D is affine X D(m) = W (V )D(mV ). V ∈Vn ,W (V )>0

Define Dn (α) as D(mV ) if α ∈ V for V ∈ Vn . The preceding equation then becomes Z D(m) = Dn (α) dW (α). (10.5) Recall that V n (α) is the set in Vn that contains α. Let G denote those α for which W (V n (α)) > 0 for all n. We have W (G) = 1 since at each stage we can remove any atoms having zero probability. The union of all such atoms is countable and hence also has zero probability. For α ∈ G given  we have as in (10.4) that we can choose N large enough to ensure that for all n > N α is in an atom V ∈ Vn that is contained in a closed ball of diameter less than . Since the measure W V on V can also be considered as a measure on the closed ball containing V , Lemma 10.7 implies that the barycenter of V must be within the closed ball and hence also d(α, mV n (α) ) ≤ ;

n ≥ N,

10.5 The Ergodic Decomposition of Affine Functionals

309

and hence the given sequence of barycenters mV n (α) converges to α. From upper semicontinuity this implies that D(α) ≥ lim sup D(mV n (α) ) = lim sup Dn (α). n→∞

n→∞

Since the Dn (α) are nonnegative and D(α) is W -integrable, this means that the Dn (α) are uniformly integrable. To see this, observe that the preceding equation implies that given  > 0 there is an N such that Dn (α) ≤ D(α) + ;

n ≥ N,

and hence for all n Dn (α) ≤

N X

Di (α) + D(α) + ,

i=1

which from Lemma 5.9 implies the Dn are uniformly integrable since the right-hand side is integrable by (10.5) and the integrability of D(α). Since the sequence is uniformly integrable, we can apply Lemma 5.10 and (10.5) to write Z Z Z D(α)dW(α) ≥ (lim sup Dn (α)) dW (α) ≥ lim sup Dn (α) dW (α)=D(m), n→∞

n→∞

which completes the proof of the theorem.

2

10.5 The Ergodic Decomposition of Affine Functionals Now assume that we have a standard measurable space (Ω, B) and a measurable transformation T : Ω → Ω. Define Pˆ = Ps . From Lemma 10.2 this is a closed and convex subset of P((Ω, B)), and hence Theorem 10.2 applies. Thus we have immediately the following result: Theorem 10.3. Let (Ω, B, m, T ) be a dynamical system with a standard measurable space (Ω, B) and an AMS measure m with stationary mean m. Let {px ; x ∈ Ω, } denote the ergodic decomposition. Let D : Ps → < be a nonnegative functional defined for all stationary measures with the following properties: (a) D(px ) is m-integrable. (b) D is affine. (c) D is an upper semicontinuous function; that is, if Pn is a sequence of stationary measures converging to a stationary measure P in the sense that Pn (F ) → P (F ) for all F in a standard generating field for B, then

310

10 The Ergodic Decomposition

D(P ) ≥ lim sup D(Pn ). n→∞

Then

Z D(m) =

D(px ) dm(x).

Comment: The functional is required to have the given properties only for stationary measures since we only require its value in the preceding expression for the stationary mean (the barycenter) and for the stationary and ergodic components. We cannot use the previous techniques to evaluate D of the AMS process m because we have not shown that it has a representation as a mixture of ergodic components. (It is easy to show when the transformation T is invertible and the measure is dominated by its stationary mean.) We can at least partially relate such functionals for the AMS measure and its stationary mean since n−1

n−1 X

mT −i (F ) → m(F ),

i=0

n→∞

and hence the affine and upper semicontinuity properties imply that lim n−1

n→∞

n−1 X

D(mT −i ) ≤ D(m).

(10.6)

i=0

The preceding development is patterned after that of Jacobs [54] for the case of bounded (but not necessarily nonnegative) D except that here the distributional distance is used and there weak convergence is used.

References

1. R. B. Ash. Real Analysis and Probability. Academic Press, New York, 1972. 2. P. J. Bickel and D. A. Freedman. Some asymptotic theory for the bootstrap. Annals of Statistics, Vol. 9, No. 1 196–1217, 1981. 3. P. Billingsley. Ergodic Theory and Information. Wiley, New York, 1965. 4. P. Billingsley. Convergence of Probability Measures. Wiley Series in Probability and Mathematical Statistics. Wiley, New York, 1968. 5. P. Billingsley. Weak Convergence of Measures: Applications in Probability. SIAM, Philadelphia, 1971. 6. O. J. Bjornsson. A note on the characterization of standard borel spaces. Math. Scand., Vol. 47, 135–136, 1980. 7. D. Blackwell. On a class of probability spaces. In Proc. 3rd Berkeley Symposium on Math. Sci. and Prob., Vol. II, 1–6, Berkeley, 1956. Univ. California Press. 8. N. Bourbaki. Elements de Mathematique,Livre VI,Integration. Hermann, Paris, 1956–1965. 9. S. Boyd and L. Vandenberghe, Convex Optimization Cambridge University Press, Cambridge, 2004. 10. L. Breiman. Probability. Addison-Wesley, Menlo Park,Calif., 1968. 11. J. C. Candy. A use of limit cycle oscillations to obtain robust analog-to-digital converters. IEEE Trans. Comm., Vol. COM-22, 298–305, March 1974. 12. J. P. R. Christensen. Topology and Borel Structure, volume Mathematics Studies 10. North-Holland/American Elsevier, New York, 1974. 13. W. Chou, unpublished. 14. K. L. Chung. A Course in Probability Theory. Academic Press, New York, 1974. 15. D. C. Cohn. Measure Theory. Birkhauser, New York, 1980. 16. H. Cramér. Mathematical Methods of Statistics. Princeton University Press, Princeton,NJ, 1946. 17. M. Denker, C. Grillenberger, and K. Sigmund. Ergodic Theory on Compact Spaces, volume 57 of Lecture Notes in Mathematics. Springer-Verlag, New York, 1970. 18. R.L. Dobrushin Prescribing a system of random variables by conditional distributions. Theor. Prob. Appl. 15:458–486, 1970. 19. R. Dobrushin. private correspondence. 20. J. L. Doob. Measure Theory. Graduate Texts in Mathematics 143, Springer, New York, 1994. 21. J. L. Doob. Stochastic Processes. Wiley, New York, 1953. 22. Y. N. Dowker. Finite and sigma-finite invariant measures. Ann. of Math., Vol. 54, 595–608, November 1951. 23. Y. N. Dowker. On measurable transformations in finite measure spaces. Ann. of Math., Vol. 62, 504–516, November 1955. 311

312

References

24. R. M. Dudley. Distances of probability measures and random variables. Ann. of Math. Statist., Vol. 39, 1563–1572, 1968. 25. M. Fréchet, Sur la distance de deux lois de probabilité. C. R. Acad. Sci. Paris, Vol. ˘ S692. 366. Frisch, 1957 244, 689âA¸ 26. N. A. Friedman. Introduction to Ergodic Theory. Van Nostrand Reinhold Company, New York, 1970. 27. R. G. Gallager. Information Theory and Reliable Communication John Wiley and Sons, New York, 1968. 28. A. Garsia. A simple proof of E. Hoph’s maximal ergodic theorem. J. Math. Mech., Vol. 14, 381–382, 1965. 29. C. Gini Sulla misura della concentrazione e della variabilit‘a del caratteri. Atti del R. Instituto Veneto, 1913–1914, 1914. 30. R. M. Gray. Entropy and Information Theory. Springer-Verlag http:ee.stanford.edu/∼gray/it.html. (Second edition in preparation.) 31. R. M. Gray, “Sliding-block source coding,”IEEE Trans. on Info. Theory, Vol. IT-21, No. 4, 357–368, July 1975. 32. R. M. Gray, “Time-invariant trellis encoding of ergodic discrete-time sources with a fidelity criterion,” IEEE Trans. on Info. Theory, Vol. IT-23, 71–83, Jan. 1977. 33. R. M. Gray. Oversampled sigma-delta modulation. IEEE Trans. Comm., Vol. COM35, 481–489, April 1987. 34. R. M. Gray and L. D. Davisson. Source coding without the ergodic assumption. IEEE Trans. Inform. Theory, Vol. IT-20, 502–516, 1974. 35. R. M. Gray and L. D. Davisson, “Quantizer Mismatch,”IEEE Trans. on Comm., Vol. COM–23, No. 4, 439–442, April 1975 36. R. M. Gray and L. D. Davisson. Random Processes: A Mathematical Approach for Engineers. Prentice-Hall, Englewood Cliffs,New Jersey, 1986. 37. R.M. Gray and L. D. Davisson, Introduction to Statistical Signal Processing, Cambridge University Press, Cambridge, UK, 2004. 38. R. M. Gray, R. M. Gray, and L. D. Davisson, editors. Ergodic and Information Theory, volume 19 of Benchmark Papers in Electrical Engineering and Computer Science. Dowden,Hutchinson,& Ross, Stroudsbug PA, 1977. 39. R. M. Gray and J. C. Kieffer. Asymptotically mean stationary measures. Ann. Probab., Vol. 8, 962–973, 1980. 40. R. M. Gray, D. L. Neuhoff and D. S. Ornstein, “Nonblock source coding with a fidelity criterion,”Annals of Prob., Vol. 3, No. 3, 478–491, June 1975. 41. R. M. Gray, D. L. Neuhoff and J. K. Omura, “Process definitions of distortion rate functions and source coding theorems,”IEEE Trans. on Info. Theory, Vol. IT-21, No. 5, 524–532, Sept. 1975. 42. R. M. Gray, D. L. Neuhoff, and P. C. Shields. A generalization of Ornstein’s d-bar distance with applications to information theory. Ann. Probab., Vol. 3, 315–328, April 1975. 43. R. M. Gray and D. S. Ornstein, “Sliding-block joint source/noisy–channel coding theorems,”IEEE Trans. on Info. Theory, Vol. IT-22, 682–690, Nov. 1976. 44. P. R. Halmos. Invariant measures. Ann. of Math., Vol. 48, 735–754, 1947. 45. P. R. Halmos. Measure Theory. Van Nostrand Reinhold, New York, 1950. 46. P. R. Halmos. Lectures on Ergodic Theory. Chelsea, New York, 1956. 47. P. R. Halmos. Introduction to Hilbert Space. Chelsea, New York, 1957. 48. P.R. Halmos and J. von Neumann. “Operator methods in classical mechanics. II” ˘ S-350. Ann. of Math. , 43 : 2 (1942) 332âA¸ 49. J. M. Hammersley and D. J. A. Welsh. First-passage,percolation,subadditive processes, stochastic networks,and generalized renewal theory. Bernoulli-BayesLaplace Anniversary Volume, 1965. 50. F. R. Hample. A general qualitative definition of robustness. Ann. of Math. Statist., Vol. 42, 1887–1896, 1971.

References

313

51. W. Hoeffding. Maβstabinvariante korrelationstheory Schriften des Mathematischen sInstituts für Angewandte Mathematik der Universit¨ [at Berlin Vol. 5, 181– 233, 1940. 52. E. Hoph. Ergodentheorie. Springer-Verlag, Berlin, 1937. 53. H. Inose and Y. Yasuda. A unity bit coding method by negative feedback. Proc. IEEE, Vol. 51, 1524–1535, November 1963. 54. K. Jacobs. The ergodic decomposition of the Kolmogorov-Sinai invariant. In F. B. Wright and F. B. Wright, editors, Ergodic Theory. Academic Press, New York, 1963. 55. R. Jones. New proof for the maximal ergodic theorem and the hardy-littlewood maximal inequality. Proc. Amer. Math. Soc., Vol. 87, 681–684, 1983. 56. L.V. Kantorovich, “On one effective method of solving certain classes of extremal ˘ S-215, 1940. problems,” Dokl. Akad. Nauk, Vol. 28, 212âA¸ 57. L.V. Kantorovich, “On the transfer of masses,” Uspekhi Mat. Nauk, Vol. 37, 7–8, 1942. 58. L.V. Kantorovich, “On a problem of Monge,” Dokl. Akad. Nauk, Vol. 3, 225–226, 1948. 59. I. Katznelson and B. Weiss. A simple proof of some ergodic theorems. Israel Journal of Mathematics, Vol. 42, 291–296, 1982. 60. J. F. C. Kingman. The ergodic theory of subadditive stochastic processes. Ann. Probab., Vol. 1, 883–909, 1973. 61. A. N. Kolmogorov. Foundations of the Theory of Probability. Chelsea, New York, 1950. 62. U. Krengel. Ergodic Theorems. De Gruyter Series in Mathematics. De Gruyter, New York, 1985. 63. N. Kryloff and N. Bogoliouboff. La théorie générale de la mesure dans son application à l’étude des systèmes de la mecanique non linéaire. Ann. of Math., Vol. 38, 65–113, 1937. 64. V. I. Levenshtein. Binary codes capable of correcting deletions, insertions,and reversals. Sov. Phys.-Dokl., Vol. 10, 707–710, 1966. 65. E. Levina and P. Bickel. The earth mover’s distance is the Mallows distance: some insights from statistics. Proceedings Eighth IEEE International Conference on In Computer Vision (ICCV 2001), Vol.2, 251–256, 2001. 66. M. Loeve. Probability Theory. D. Van Nostrand, Princeton,New Jersey, 1963. Third Edition. 67. D. G. Luenberger. Optimization by Vector Space Methods. Wiley, New York, 1969. 68. G. Mackey. Borel structures in groups and their duals. Trans. Amer. Math. Soc., Vol. 85, 134–165, 1957. 69. L. E. Maistrov. Probability Theory: A Historical Sketch. Academic Press, New York, 1974. Translated by S. Kotz. 70. C.L. Mallows. A note on asymptotic joint normality Ann. Math.l Stat. Vol. 43, ˘ S 515, 1972. 508âA¸ 71. G. Monge. Mémoire sur la théorie des déblais et des remblais, 1781. 72. J. Moser, E. Phillips, and S. Varadhan. Ergodic Theory: A Seminar. Courant Institute of Math. Sciences, New York, 1975. 73. J. Nedoma. On the ergodicity and r-ergodicity of stationary probability measures. Z. Wahrsch. Verw. Gebiete, Vol. 2, 90–97, 1963. 74. David Lee Neuhoff Source coding and distance measures on random processes PhD dissertation, Stanford University August 1974 75. D.L. Neuhoff, R. M. Gray and L. D. Davisson. Fixed rate universal block source coding with a fidelity criterion. IEEE Trans. on Info. Theory, Vol. IT-21, No. 5, 511–523, Sept. 1975. 76. D. S. Ornstein, Bernoulli shifts with the same entropy are isomorphic, Advances in Math., Vol. 4, 337–352, 1970.

314

References

77. D. Ornstein. An application of ergodic theory to probability theory. Ann. Probab., Vol. 1, 43–58, 1973. 78. D. Ornstein. Ergodic Theory,Randomness,and Dynamical Systems. Yale University Press, New Haven, 1975. 79. D. Ornstein and B. Weiss. The Shannon-McMillan-Breiman theorem for a class of amenable groups. Israel J. of Math, Vol. 44, 53–60, 1983. 80. J. C. Oxtoby. Ergodic sets. Bull. Amer. Math. Soc., Vol. 58, 116–136, 1952. 81. P. Papantoni-Kazakos and R. M. Gray. Robustness of estimators on stationary observations. Ann. Probab., Vol. 7, 989–1002, Dec. 1979. 82. K. R. Parthasarathy. Probability Measures on Metric Spaces. Academic Press, New York, 1967. 83. I. Paz. Stochastic Automata Theory. Academic Press, New York, 1971. 84. K. Petersen. Ergodic Theory. Cambridge University Press, Cambridge, 1983. 85. H. Poincaré. Les méthodes nouvelles de la mécanique céleste, volume I,II,III. Gauthiers-Villars, Paris, 1892,1893,1899. Also Dover,New York,1957. 86. S.T. Rachev Probability metrics and the stability of stochastic models. John Wiley & Sons Ltd., Chichester, 1991. 87. S.T. Rachev and L. Rüschendorf Mass Transportation Problems Vol. I: Theory, Vol. II: Applications. Probability and its applications. Springer- Verlag, New York, 1998. 88. O. W. Rechard. Invariant measures for many-one transformations. Duke J. Math., Vol. 23, 477–488, 1956. 89. V. A. Rohlin. Selected topics from the metric theory of dynamical systems. Uspechi Mat. Nauk., Vol. 4, 57–120, 1949. English translation: Amer. Math. Soc. Transl. (2) Vol. 49, 171–240, 1966. 90. V. A. Rohlin. On the fundamental ideas of measure theory, Mat. Sb., Vol. 25, 107–150, 1949. English translation: Amer. Math. Soc. Transl.Amer. Math. Soc. Transl., 91. Y. Rozanov. Stationary Random Processes. Holden Day, San Francisco, 1967. 92. Y. Rubner, C. Tomasi, and L.J. Guibas. A metric for distributions with applications to image databases. In Proceedings of the IEEE International Conference on Computer Vision, 59-66. Bombay, India, Jan. 1998. 93. W. Rudin. Principles of Mathematical Analysis. McGraw-Hill, New York, 1964. 94. L. Schwartz. Radon Measures on Arbitrary Topological Spaces and Cylindrical Measures. Oxford University Press, Oxford, 1973. 95. C. E. Shannon. A mathematical theory of communication. Bell Syst. Tech. J., Vol. 27, 379–423, 623–656, 1948. 96. C. E. Shannon. Coding theorems for a discrete source with a fidelity criterion. In IRE National Convention Record,Part 4, 142–163, 1959. 97. P. C. Shields. The Theory of Bernoulli Shifts. The University of Chicago Press, Chicago,Ill., 1973. 98. P. C. Shields. The ergodic and entropy theorems revisited. IEEE Trans. Inform. Theory, Vol. IT-33, 263–266, 1987. 99. P. C. Shields. The Ergodic Theory of Discrete Sample Paths. Graduate Studies in Mathematics, Vol. 13. American Mathematical Society, Providence, Rhode Island, 1991. 100. G. F. Simmons. Introduction to Topology and Modern Analysis. McGraw-Hill, New York, 1963. 101. Ya. G. Sinai. Introduction to Ergodic Theory. Mathematical Notes,Princeton University Press, Princeton, 1976. 102. M. Smorodinsky. A partition on a Bernoulli shift which is not weak Bernoulli Math. Systems Theory, Vol. 5, No. 3, 201–203, 1971. 103. M. Smorodinsky. Ergodic Theory, Entropy in Lecture Notes in Mathematics, Vol. 214 Springer-Verlag, New York, 1971.

References

315

104. V. Strassen. The existence of probability measures with given marginals. Ann. of Math. Statist., Vol. 36, 423–429, 1965. 105. E. Tanaka and T. Kasai. Synchronization and subsititution error correcting codes for the Levenshtein metric. IEEE Trans. Inform. Theory, Vol. IT-22, 156– 162, 1976. 106. A. H. Tchen. Inequalities for distributions with given marginals. The Annals of Probability, Vol. 8, No. 4, 814–827, August 1980. 107. S. S. Vallender. Computing the Wasserstein distance between probability distributions on the line. Theory Probab. Appl., Vol. 18, 824–827, 1973. 108. L. N. Vasershtein. Markov processes on countable product space describing large systems of automata. Problemy Peredachi Informatsii, Vol. 5, 64–73, 1969. 109. A.M. Vershik. The Kantorovich metric: the initial history and little-known applications. Zap. Nauchn. Sem. S.-Peterburg. Otdel. Mat. Inst. Steklov. (POMI) 312, ˘ S085, 311. Teor. Predst. Din. Sist. Komb. i Algoritm. Metody. 11 (2004), 69âA¸ 110. C. Villani. Topics in Optimal Transportation Vol. 58 of Graduate Studies in Mathematics. American Mathematical Society, Providence, RI, 2003. 111. C. Villani. Optimal Transport, Old and New Spring, 2009. 112. J. von Neumann. Zur operatorenmethode in der klassischen mechanik. Ann. of Math., Vol. 3, 587–642, 1932. 113. P. Walters. Ergodic Theory-Introductory Lectures. Lecture Notes in Mathematics No. 458. Springer-Verlag, New York, 1975. 114. F. B. Wright. The recurrence theorem. Amer. Math. Monthly, Vol. 68, 247–248, 1961.

“This page left intentionally blank.”

Index

Fσ set, 15 Gδ , 15 Gδ set, 15 Lp norm, 144 N-ergodic, 230 Z, 1, 41 ZM , 1 R, 1, 41 σ -field, 10, 158 countably generated, 13 remote, 212 σ -fields product, 47 Z+ , 1, 41 R+ , 1, 41 0-1 law, 212 absolutely continuous, 170, 172 addition, 4, 41 additive, 20 finitely, 20 affine, 302, 305 almost everywhere, 128, 142 almost everywhere ergodic theorem, 239 almost surely, 142 alphabet, xx, xxiv, 36, 115 AMS, 205 block, 249 analytic space, xxv arithmetic mean, 140 asymoptotically dominates, 210 asymptotic mean expectation, 221 asymptotic mean stationary, xxviii asymptotically stationary, 154 asymptotically generate, 66 asymptotically generates, 64, 66

asymptotically mean stationary, 154, 205 atom, 63 atoms, 24 autocorrelation, 140 autocorrelation function, 290 average, xxvii average Hamming distance, 3 B-process, viii B-processes, 80 bad atoms, 70 ball open, 7 Banach space, 17 Banach space, 17 barycenter, 294, 302 base, 8 basis, 66 Bernoulli process, 78, 80 Bernoulli shift, 80 binary indexed, 64 binary sequence function, 65 binary sequence space, 73 Birkhoff ergodic theorem, 239 Birkhoff, George, 239 block AMS, 249 block ergodic, 230 block stationary, 206 Borel field, 14 Borel measurable space, 14 Borel measure, 82 Borel measure on [0, 1), 90 Borel measure on [0, 1)k , 110 Borel measure space, 90 Borel sets, 14 317

318 Borel space, xxv, xxvi, 14 Borel-Cantelli lemma, 145 bounded ergodic properties, 200 Céaro mean, 140 canonical binary sequence function, 65 Cantor’s intersection theorem, 17 Carathéodory, 27, 32, 62, 76, 78, 266 Cartesian product, 6 Cartesian product space, 46 carving, 100 cascade mappings, 36 Cauchy sequence, 17 Cauchy-Schwarz inequality, 131 chain rule, 138 change of variables, 137 Chou, Wu, 260 circular distance, 3 class separating, 68 closure, 9 code, 56 stationary, 55 coding, 55 coin flipping, 80 coin flips, 91 complete metric space, 17 completion, 25, 26 composite mappings, 36 compressed, 217 concave, 126 conditional expectation, 166 conservative, 217 consistent, 77 continuity at ∅, 20 continuity from above, 20 continuity from below, 20 continuous, 267 continuous function, 37 continuous time random process, 41 continuous-alphabet, 110 converge in mean square, 144 in quadratic mean, 144 converge weakly, 268 convergence, 8, 142 L1 (m), 143 L2 (m), 144 with probability one, 142 convex, 126 convex function, 125 coordinate function, 140 cost function, 3 cost functions, xxx

Index countable extension property, 62, 76 countable subadditivity, 27 countably generated, 13, 62 coupling, 270 covariance function, 175 cover, 27 covergence in probability, 143 cylinders thin, 73 de Morgan’s law, 11 dense, 16 diagonal sequence, 9 diagonalization, 9 diameter, 9 digital measurement, 115 directly given random process, 52 distance, 2 circular, 3 distribution, 40, 275 distributional, 263 edit, 4 Hamming, 3 Levenshtein, 4 Prohorov, 286 total variation , 40 variation, 40, 275 distortion measure, 3, 269, 270 distortion measures, xxx distribution, 36, 51 marginal, 50 distribution distance, 275 distribution, process, 51 distributional distance, 263 dominated convergence theorem, 135 dynamical system recurrent, 216 dynamical system, discrete-time, 42, 45 edit distance, 4 elementary events, 1 elementary outcomes, 1 empirical probability, xxiii ensemble average, 116 equivalent, 173 equivalent metrics, 8 equivalent random processes, 52 ergodic, xxviii, 227 totally, 251 ergodic component, 299 ergodic decomposition, 232, 234, 252, 295, 309 ergodic properties, xxiii, 195, 226

Index bounded, 200 ergodic property, 196 ergodic theorem, xxvii subadditive, 253 ergodic theorem, subadditive, 278 ergodic theorems, xxiii ergodic theory, 42 ergodicity, 226 Euclidean space, 4, 19 event elementary, 1 recurrent, 216 event space, xx, 10 events independent, 188 output, 36 expectation, 116 conditional, 166 extended Fatou’s lemma, 134 extended monotone convergence theorem, 134 extension of probability measures, 61 extension theorem, 77 factor, 56 Fatou’s lemma, 134, 148 fidelity criteria, xxx fidelity criterion, 277 fidelity criterion, additive, 277 fidelity criterion, single-letter, 277 fidelity criterion, subadditive, 277 field, 11 standard, 66, 76 fields increasing, 64 product, 47 filtering, 55 finite additivity, 20 finite intersection property, 66 finitely additive, 20 flow, 45 frequency-typical points, 233 frequency-typical sequences, 252 function, 2 continuous, 37 tail, 213 future tail sub-σ -field, 212 Gaussian Processes, 290 Gaussian random vector, 175 generated σ -field, 12 generated field, 13 generic points, 233 generic sequences, 252

319 good sets priciple, 70 good sets principle, 50, 62 Hölder’s inequality, 131 Hamming distance, 3 average, 3 Hilbert Space, 167 Hilbert space, 17, 166 i.o., 208 IID, 80, 110, 245 incompressible, 217 increasing fields, 64 independent and identically distributed, 80, 245 independent events, 188 independent measurements, 188 independent random variables, 188 independent, identically distributed, 110 indicator function, 38, 65 inequality Young’s, 127 inequality, Jensen’s, 125 infinitely recurrent, 218 information source, xxiv inner product, 144 inner product space, 6, 144, 166 integer sequence space, 73, 96, 99 integrable, 127 integral, 127 integrating to the limit, 134 intersection sets, 63 invariant, 152, 207 totally, 209 invariant function, 197 invariant measure, 197 invertible, 43 isomorphic dynamical systems, 58 measurable spaces, 57 probability spaces, 57 isomorphic, metrically, 57 isomorphism, 56, 57 isomorphism mod 0, 57, 91 iterated expectation, 184 Jensen’s Inequality, 125 join, 116 joining, 270 K-automorphism, 247 K-process, 212 kernel, 102

320 Kolmogorov Extension Theorem, 77 Kolmogorov extension theorem, 62 Kolmogorov process, 212 Kolmogorov representation, 53 Kolmogorov, A.N., xxiv Kronecker δ function, 3 l.i.m., 144 lag, 140 las of large numbers, xxvii law of large numbers, xxiii Lebesgue decomposition theorem, 172 Lebesgue measure, 82 on the real line, 93 Lebesgue measure on [0, 1)k , 110 Lebesgue measure space, 90 Lebesgue space, 90, 91 Lee metric, 3 left shift, 43 left tail sub-σ -field, 212 Levenshtein distance, 4 limit in the mean, 144 limit inferior, 120 limit infimum, 120, 121 limit point, 9 limit superior, 120 limit supremum, 120 limiting time average, 141 lower limit, 120 Lusin space, xxv marginal distribution, 50 Markov k-step, 299 kth order, 299 Markov chain, 190, 191, 273 Markov inequality, 131 Markov property, 190 martingale, 189 match, 4 maximal ergodic lemma or theorem, xxix mean, 116 mean function, 175, 290 mean vector, 175 measurability, 42 measurable, 29 measurable function, 35, 157, 158 measurable scheme, 102 measurable space, 10, 157 standard, 66, 76 measure, 20 Borel, 82 Lebesgue, 82

Index measure space, 20 finite, 20 sigma-finite, 20 measurement, 115 digital, 115 discrete, 115 measurements independent, 188 metric, 2 Hamming, 3 Lee, 3 metric space, 2 complete, separable, 97 metrically isomorphic, 57 metrically transitive, 227 metrics equivalent, 8 Minkowski sum inequality, 7 Minkowski’s inequality, 6, 131, 278 mixing, 245, 246 mixture, xxxii, 24, 232 monotone class, 12, 13, 31 monotone convergence theorem, 134 monotonicity, 27 multiplication, 4 nested expectation, 184 nonsingular, 218 norm, 144 normed linear space, 4, 5 normed linear vector space, 4 null preserving, 218 null set, 25 one-sided, 2 one-sided random process, 41 open ball, 7 open set, 7 open sphere, 7 orbit, 42, 45 orthogonal, 168 outer measure, 28 output events, 36 parallelogram law, 167 partition, 24, 63, 116 partition distance, 24 past tail sub-σ -field, 212 PDF, 174 perpindicular complement, 168 point, 2 points frequency-typical, 233 generic, 233

Index pointwise ergodic theorem, 239 Polish Borel space, 19 Polish schemes, 101 Polish space, 19, 147, 270 power set, 11, 58, 73 power spectral density, 290 pre-Hilbert space, 6 probabilistic average, 116 probability density function, 174 probability measure, xx probability space, 19 associated, 53 discrete, 21 probability space, complete, 25 process asymptotically mean stationary, 206 process distance, viii process distribution, xx, 51 product cartesion, 6 product σ -field, 47 product distribution, 190 product field, 47 product space, 2, 6, 46 Prohorov distance, 286 projection, 166, 169 projection theorem, 166, 168 pseudometric, 2 pseudometric space associated with the probability space, 23 quantization, 119 quantizer, 43, 105, 119 Radon space, xxv Radon-Nikodym derivative, 170 Radon-Nikodym theorem, 166, 169 random object, 35 random process, xx, 41 two-sided, 41 continuous parameter, 41 continuous time, 41 discrete parameter, 41 discrete time, 41 bilateral, 41 coin flipping, 80 directly given, 52 double-sided, 41 IID, 80 one-sided, 41 roulette, 80 single-sided, 41 unilateral, 41 random processes

321 equivalent, 52 random variable, 35, 36 discrete, 36, 115 random variables independent, 188 random vector, 41 rectangle, 47 recurrence, 215 recurrent, 215 recurrent dynamical system, 216 recurrent event, 215, 216 reference letter, 273 regular conditional probability, xxviii, 157 relative frequency, xxiii remote σ -field, 212 restriction, 162 right tail sub-σ -field, 212 roullette, 80 sample average, 139–141 sample function, 41 sample mean, 140 sample points, 1 sample sequence, 41 sample space, 1, 10 sample waveform, 41 sampling, 45 scalar, 4 scheme, 101 measurable, 102 schemes, 101 semi-flow, 45 seminorm, 5 separable, 16 separating class, 68 sequence space binary, 73 integer, 73 sequentially compact, 8 set Fσ , 15 Gδ , 15 open, 7 shift, 43, 139 shift transformation, 43 shift-invariant, 55 sigma-field, 10 sigma-finite, 20 simple function, 115 singular, 172 size, 4 smallest σ -field, 12 space

322 Euclidean, 4 Hilbert, 17 inner product, 6, 144 metric, 2 normed linear, 4 normed linear vector, 4 pre-Hilbert, 6 standard, 63 standard measurable, 76 sphere open, 7 standard diagonalization, 9 standard field, 66, 76 standard measurable space, 66, 76 standard probability space, 137 standard space, 63 standard spaces, xxv state, 43 stationary, 55, 152 block, 206 weakly, 290 wide-sense, 290 stationary function, 197 stationary mean, xxviii, 205 stochastic process, xxiv strongly mixing, 245, 246 sub-σ -field, 158 subadditive, 126, 254 subadditive ergodic theorem, 253 subadditivity, 27 submartingale, 189 subspace, 70 sums, 128 superadditive, 282 supermartingale, 189 Suslin space, xxv tail σ -field, 212 tail function, 213

Index tail sub-σ -field future, 212 past, 212 right, 212 Tchebychev’s inequality, 131 term-by-term, 136 theorem:processmetric, 280 thin cylinders, 73 time average, 139, 140 time averge, 141 time invariant, 197 time series, xxiv time-averge mean, 140 toeplitz, 290 topology, 8 total variation distance, 40 totally ergodic, 230, 251 totally invariant, 209 transport, 272 transportation, 272 trivial space, 11 uniformly integrable, 133 upper limit, 120 variation distance, 40, 275 version of the conditional expectation, 184 Vitali-Hahn-Saks Theorem, 202 wandering set, 216 weak convergence, 268 weakly mixing, 245, 246 weakly stationary, 290 Young’s inequality, 127 zero-time mapping, 43