Information ThPory ThP Centrval Limit Thtorem and
This page intentionally left blank
Information Theory The Central...
41 downloads
895 Views
10MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
Information ThPory ThP Centrval Limit Thtorem and
This page intentionally left blank
Information Theory The Central Limit Theorem and
Oliver Johnson University of Cambridge, UK
Imperial College Press
Published by Imperial College Press 57 Shelton Street Covent Garden London WC2H 9HE
Distributed by World Scientific Publishing Co. Pte. Ltd.
5 Toh Tuck Link, Singapore 596224 USA ofice: Suite 202, 1060 Main Street, River Edge, NJ 07661 UK ofice: 57 Shelton Street, Covent Garden, London WC2H 9HE
British Library Cataloguing-in-PublicationData A catalogue record for this book is available from the British Library.
INFORMATION THEORY AND THE CENTRAL LIMIT THEOREM Copyright 0 2004 by Imperial College Press
All rights reserved. This book, or parts thereoj may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any informution storuge and retrieval system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.
ISBN 1-86094-473-6
Printed in Singapore by World scientific Printers ( S ) Pte Ltd
To Maria, Thanks for everything.
This page intentionally left blank
Preface
“Information theory must precede probability theory and not be based on it.” A.N.Kolmogorov, in [Kolmogorov, 19831.
This book applies ideas from Shannon’s theory of communication [Shannon and Weaver, 19491 to the field of limit theorems in probability. Since the normal distribution maximises entropy subject to a variance constraint, we reformulate the Central Limit Theorem as saying that the entropy of convolutions of independent identically distributed real-valued random variables converges to its unique maximum. This is called convergence in relative entropy or convergence in Kullback-Leibler distance. Understanding the Central Limit Theorem as a statement about the maximisation of entropy provides insights into why the result is true, and why the normal distribution is the limit. It provides natural links with the Second Law of Thermodynamics and, since other limit theorems can be viewed as entropy maximisation results, it motivates us to give a unified view of these results. The paper [Linnik, 19591 was the first to use entropy-theoretic methods to prove the Central Limit Theorem, though Linnik only proved weak convergence. Barron published a complete proof of the Central Limit Theorem in [Barron, 19861, using ideas from entropy theory. Convergence in relative entropy is a strong result, yet Barron achieves it in a natural way and under optimally weak conditions. Chapter 1 introduces the concepts of entropy and Fisher information that are central to the rest of the book. We describe the link between
vii
...
Vlll
Informatzon Theory and the Central Limzt Theorem
information-theoretic and thermodynamic entropy and introduce the analogy with the Second Law of Thermodynamics. Chapter 2 partly discusses the ideas of [Johnson and Barron, 20031. It considers the case of independent identically distributed (IID) variables and develops a theory based on projections and Poincar6 constants. The main results of the chapter are Theorems 2.3 and 2.4, which establish that Fisher information and relative entropy converge at the optimal rate of O(l/n). In Chapter 3 we discuss how these ideas extend to independent nonidentically distributed random variables. The Lindeberg-Feller theorem provides an analogue of the Central Limit Theorem in such a case, under the so-called Lindeberg condition. This chapter extends the Lindeberg-Feller theorem, under similar conditions, to obtain Theorem 3.2, a result proving convergence of Fisher information of non-identically distributed variables. Later in Chapter 3 we develop techniques from the case of one-dimensional random variables to multi-dimensional random vectors, to establish Theorem 3.3, proving the convergence of Fisher information in this case. Chapter 4 shows how some of these methods extend to dependent families of random variables. In particular, the case of variables under Rosenblatt-style a-mixing conditions is considered. The principal results of this chapter are Theorems 4.1 and 4.2, which prove the convergence of Fisher information and entropy respectively. Theorem 4.1 holds under only very mild conditions, namely the existence of the (2 S)th moment, and the right rate of decay of correlations. In Chapter 5, we discuss the problem of establishing convergence to other non-Gaussian stable distributions. Whilst not providing a complete solution, we offer some suggestions as to how the entropy-theoretic method may apply in this case. Chapter 6 considers convergence to Haar measure on compact groups. Once again, this is a setting where the limiting distribution maximises the entropy, and we can give an explicit rate of convergence in Theorem 6.9. Chapter 7 discusses the related question of the ‘Law of Small Numbers’. That is, on summing small binomial random variables, we expect the sum to be close in distribution to a Poisson. Although this result may seem different in character to the Central Limit Theorem, we show how it can be considered in a very similar way. In this case, subadditivity is enough to prove Theorem 7.4, which establishes convergence to the Poisson distribution. Theorem 7.5 gives tighter bounds on how fast this occurs. In Chapter 8 we consider Voiculescu’s theory of free (non-commutative) random variables. We show that the entropy-theoretic framework extends
+
Preface
ix
to this case too, and thus gives a proof of the information-theoretic form of the Central Limit Theorem, Theorem 8.3, bypassing the conceptual difficulties associated with the R-transform. In the Appendices we describe analytical facts that are used throughout the book. In Appendix A we show how to calculate the entropy of some common distributions. Appendix B discusses the theory of Poincark inequalities. Appendices C and D review the proofs of the de Bruijn identity and Entropy Power inequality respectively. Finally, in Appendix E we compare the strength of different forms of convergence. Chapter 4 is based on the paper [Johnson, 20011, that is “Information inequalities and a dependent Central Limit Theorem” by O.T. Johnson, first published in Markov Processes and Related Fields, 2001, volume 7 , pages 627-645. Chapter 6 is based on the paper [Johnson and Suhov, 20001, that is “Entropy and convergence on compact groups” by 0.T.Johnson and Y.M.Suhov, first published in the Journal of Theoretical Probability, 2000, volume 13, pages 843-857. This book is an extension of my PhD thesis, which was supervised by Yuri Suhov. I am extremely grateful to him for the time and advice he was able to give me, both during and afterwards. This book was written whilst I was employed by Christ’s College Cambridge, and hosted by the Statistical Laboratory. Both have always made me extremely welcome. I would like to thank my collaborators, Andrew Barron, Ioannis Kontoyiannis and Peter Harremoes. Many people have offered useful advice during my research career, but I would particularly like to single out HansOtto Georgii, Nick Bingham and Dan Voiculescu for their patience and suggestions. Other people who have helped with the preparation of this book are Christina Goldschmidt, Ander Holroyd and Rich Samworth. Finally, my family and friends have given great support throughout this project.
OLIVERJOHNSON CAMBRIDGE, OCTOBER2003 EMAIL:0 . T . Johnson. 92Qcantab.net
This page intentionally left blank
Contents
Preface
vii
1. Introduction t o Information Theory
1
1.1 Entropy and relative entropy . . . . . . . . . . 1.1.1 Discrete entropy . . . . . . . . . . . . . . . 1.1.2 Differential entropy . . . . . . . . . . . . 1.1.3 Relative entropy . . . . . . . . . . . . . . 1.1.4 Other entropy-like quantities . . . . . . 1.1.5 Axiomatic definition of entropy . . . . . 1.2 Link to thermodynamic entropy . . . . . . . . . 1.2.1 Definition of thermodynamic entropy . . 1.2.2 Maximum entropy and the Second Law . 1.3 Fisher information . . . . . . . . . . . . . . . . . 1.3.1 Definition and properties . . . . . . . . . 1.3.2 Behaviour on convolution . . . . . . . . 1.4 Previous information-theoretic proofs . . . . . . 1.4.1 Rknyi’s method . . . . . . . . . . . . . . . 1.4.2 Convergence of Fisher information . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
. . .
. . . . . . .
2 . Convergence in Relative Entropy 2.1 Motivation . . . . . . . . . . . . . . . . . . . . 2.1.1 Sandwich inequality . . . . . . . . . . 2.1.2 Projections and adjoints . . . . . . . . 2.1.3 Normal case . . . . . . . . . . . . . . . . 2.1.4 Results of Brown and Barron . . . . . 2.2 Generalised bounds on projection eigenvalues xi
1 1 5 8 13 16 17 17 19 21 21 25 27 27 30
33 . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . .
33 33 36 38 41 43
Information Theory and the Central Limit Theorem
xii
2.2.1 2.2.2 2.2.3 2.3 Rates 2.3.1 2.3.2 2.3.3
Projection of functions in L2 . . . . . . . . . . . . . . Restricted Poincark constants . . . . . . . . . . . . . Convergence of restricted Poincari: constants . . . . . of convergence . . . . . . . . . . . . . . . . . . . . . . Proof of O ( l / n ) rate of convergence . . . . . . . . . Comparison with other forms of convergence . . . . . Extending the Cramkr-Rao lower bound . . . . . . .
55
3. Non-Identical Variables and Random Vectors
3.1 Non-identical random variables . . . . . . 3.1.1 Previous results . . . . . . . . . . . . 3.1.2 Improved projection inequalities . . 3.2 Random vectors . . . . . . . . . . . . . . . . 3.2.1 Definitions . . . . . . . . . . . . . . . 3.2.2 Behaviour on convolution . . . . . 3.2.3 Projection inequalities . . . . . . .
43 44 46 47 47 50 51
. . . . .
. . . . .
. . . . .
. . . . .
. . . . . . . . . . . . . . . . . . . . . . .....
. . . . . . . . . .
. . . . . . . . . .
4 . Dependent Random Variables 4.1 Introduction and notation . . . . . . . . . . . . . . . . . . . 4.1.1 Mixing coefficients . . . . . . . . . . . . . . . . . . . 4.1.2 Main results . . . . . . . . . . . . . . . . . . . . . . . 4.2 Fisher information and convolution . . . . . . . . . . . . . . 4.3 Proof of subadditive relations . . . . . . . . . . . . . . . . . 4.3.1 Notation and definitions . . . . . . . . . . . . . . . . 4.3.2 Bounds on densities . . . . . . . . . . . . . . . . . . . 4.3.3 Bounds on tails . . . . . . . . . . . . . . . . . . . . . 4.3.4 Control of the mixing coefficients . . . . . . . . . . . 5 . Convergence to Stable Laws 5.1 Introduction to stable laws . . . . . . . . . . . . . . . . . . 5.1.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . 5.1.2 Domains of attraction . . . . . . . . . . . . . . . . . 5.1.3 Entropy of stable laws . . . . . . . . . . . . . . . . . 5.2 Parameter estimation for stable distributions . . . . . . . . 5.2.1 Minimising relative entropy . . . . . . . . . . . . . . 5.2.2 Minimising Fisher information distance . . . . . . . . 5.2.3 Matching logarithm of density . . . . . . . . . . . . . 5.3 Extending de Bruijn’s identity . . . . . . . . . . . . . . . .
55 55 57 64 64 65 66 69 69 69 72 74 77 77 79 82 83 87 87 87 89 91 92 92 94 95 96
...
Contents
Xlll
5.3.1 Partial differential equations . . . . . 5.3.2 Derivatives of relative entropy . . . . 5.3.3 Integral form of the identities . . . . 5.4 Relationship between forms of convergence . 5.5 Steps towards a Brown inequality . . . . . .
. . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6 . Convergence on Compact Groups 6.1 Probability on compact groups . . . . . . 6.1.1 Introduction to topological groups 6.1.2 Convergence of convolutions . . . . 6.1.3 Conditions for uniform convergence 6.2 Convergence in relative entropy . . . . . . 6.2.1 Introduction and results . . . . . . 6.2.2 Entropy on compact groups . . . . 6.3 Comparison of forms of convergence . . . 6.4 Proof of convergence in relative entropy . 6.4.1 Explicit rate of convergence . . . . 6.4.2 No explicit rate of convergence . .
109
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
109 109 111 114 118 118 119 121 125 125 . . . . . . . . . . 126
7. Convergence to the Poisson Distribution 7.1 Entropy and the Poisson distribution . . . . 7.1.1 The law of small numbers . . . . . . 7.1.2 Simplest bounds on relative entropy 7.2 Fisher information . . . . . . . . . . . . . . . 7.2.1 Standard Fisher information . . . . . 7.2.2 Scaled Fisher information . . . . . . 7.2.3 Dependent variables . . . . . . . . . 7.3 Strength of bounds . . . . . . . . . . . . . . . 7.4 De Bruijn identity . . . . . . . . . . . . . . . 7.5 L 2 bounds on Poisson distance . . . . . . . 7.5.1 L2 definitions . . . . . . . . . . . . . . 7.5.2 Sums of Bernoulli variables . . . . . 7.5.3 Normal convergence . . . . . . . . . 8 . Free Random Variables
96 97 100 102 105
129 . . . . . . . . . 129 . . . . . . . . . 129 . . . . . . . . . 132 . . . . . . . . 136 . . . . . . . . . 136 . . . . . . . . . 138 . . . . . . . . . 140 . . . . . . . . 142 . . . . . . . . 144 . . . . . . . . . 146 . . . . . . . . 146 . . . . . . . . . 147 . . . . . . . . . 150
153
8.1 Introduction to free variables . . . . . . . . . . . . . . . . . 153 8.1.1 Operators and algebras . . . . . . . . . . . . . . . . . 153 8.1.2 Expectations and Cauchy transforms . . . . . . . . . 154
Information Theory and the Central Lamat Theorem
xiv
8.1.3 Free interaction . . . . . . . . . . . . 8.2 Derivations and conjugate functions . . . 8.2.1 Derivations . . . . . . . . . . . . . . 8.2.2 Fisher information and entropy . . 8.3 Projection inequalities . . . . . . . . . . . . Appendix A
. . . . . . . . .
. . . .
. . . .
. . . . . . . . ....... . . . . . . . . . . . . . . .
171
Calculating Entropies
A.l Gamma distribution . . . . . . . . . . . . . . . . . . . . . . A.2 Stable distributions . . . . . . . . . . . . . . . . . . . . . . . Appendix B Poincark Inequalities B.l Standard Poincark inequalities B.2 Weighted Poincark inequalities
158 161 161 163 166
171 173 177
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
177 179
Appendix C
de Bruijn Identity
183
Appendix D
Entropy Power Inequality
187
Appendix E
Relationships Between Different Fornis of Convergence 191
E . l Convergence in relative entropy t o the Gaussian . . . . . . . 191 E.2 Convergence to other variables . . . . . . . . . . . . . . . . 194 E.3 Convergence in Fisher information . . . . . . . . . . . . . . 195
Bibliography
199
Index
207
Chapter 1
Introduction to Information Theory
Summary This chapter contains a review of some results from information theory, and defines fundamental quantities such as Kullback-Leibler distance and Fisher information, as well as giving the relationship between them. We review previous work in the proof of the Central Limit Theorem (CLT) using information-theoretic methods. It is possible to view the CLT as an analogue of the Second Law of Thermodynamics, in that convergence t o the normal distribution will be seen as an entropy maximisation result. 1.1 1.1.1
Entropy and relative entropy
Discrete entropy
This book is based on information-theoretic entropy, developed by Shannon in the landmark paper [Shannon, 19481. This quantified the idea that ‘some random variables are more random than others’. Entropy gives a numerical measure of how far from deterministic a random variable is.
Definition 1.1 If event A occurs with probability P(A), define the ‘information’ I ( A ) gained by knowing that A has occurred to be
I ( A ) = - log, P(A).
(1.1)
The intuitive idea is that the rarer an event A , the more information we gain if we know it has occurred. For example, since it happens with very high probability, our world view changes very little when the sun rises each morning. However, in the very unlikely event of the sun failing t o rise, our 1
2
Informatzon Theory and the Centro.1 rJ7,m2t Theorem
model of physics would require significant updating. Of course, we could use any decreasing function of P(A) in the definition of information. An axiomatic approach (see Section 1.1.5) gives a post hoc justification of the choice of log,, a choice which dates back t o [Hartley, 19281.
Definition 1.2 Given a discrete random variable X taking values in the finite set X = {zl,2 2 , . . . , z,} with probabilities p = ( p 1 , p 2 , . . . ,p,), we define the (Shannon) entropy of X t o be the expected amount of information gained on learning the value of X : n
H ( X ) =EI({X
= x,}) =
-CpLIOgZpa.
(14
a=1
Considerations of continuity lead us to adopt the convention that 0 log 0 = 0. Since H depends on X only though its probability vector p and not through the actual values x, we will refer t o H ( X ) and H(p) interchangably, for the sake of brevity.
Example 1.1 For X a random variable with Bernoulli(p) distribution, that is taking the value 0 with probability 1 - p and 1 with probability p ,
H ( X ) = -PlO&P-
(1 - p ) l o g z ( l - P I .
(1.3)
Notice that for p = 0 or 1, X is deterministic and H ( X ) = 0. On the other hand, for p = l / Z , X is ‘as random as it can be’ and H ( X ) = 1, the maximum value for random variables taking 2 values, see the graph Figure 1.1. This fits in with our intuition as to how a sensible measure of uncertainty should behave. Further, since H”(p) = log e / ( p (1 - p ) ) 2 0, we know that the function is concave. We can extend Definition 1.2 to cover variables taking countably many values. We simply replace the sum from 1 t o n with a sum from 1 t o 00 (and adopt the convention that if this sum diverges, the entropy is infinite):
Example 1.2 For X a random variable with a geometric(p) distribution, that is with P ( X = r ) = (1 - p)p‘ for r = 0 , 1 , . . .
H ( X )= =
since E X = p / ( 1 - p )
C P(X
= r ) (-
-log,(l
-
P) -
log, (1 - p ) - r log, p )
(1.4)
(1.5)
Introductzon t o i n f o r m a t i o n theory
3
Fig. 1.1 T h e entropy of a Bernoulli(p) variable for different p
Shannon entropy is named in analogy with the thermodynamic entropy developed by authors such as Carnot, Boltzmann, Gibbs and Helmholtz (see Section 1.2 for more information). We will see that in the same way that maximum entropy states play a significant role within statistical mechanics, maximum entropy distributions are often the limiting distributions in results such as the Central Limit Theorem. Some simple properties of entropy are as follows:
Lemma 1.1
(1) Although we refer to the entropy of random variable X , the entropy depends only o n the (unordered) probabilities { p i } and not the values {Xi}.
(2) I n particular, this means that discrete entropy is both shift and scale invariant, that is f o r all a
# 0 and for all b:
H(aX
+ b) = H ( X ) .
(1.6)
(3) The entropy of a discrete random variable is always non-negative, and is strictly positive unless X is deterministic (that is unless pi = 1 for some i).
4
(4)
Information Theory and the Central Limit Theorem
The entropy of a random variable taking n values is maximised by the uniform distribution pi = l / n , so that 0 5 H ( X ) 5 logn.
These first three properties are clear: the last is best proved using the positivity of relative entropy defined in Definition 1.5.
Definition 1.3 Given a pair of random variables ( Y , Z ) ,we can define the joint entropy
H ( Y , Z ) = - c P ( ( Y , Z ) = ( r , s ) ) l o g P ( ( Y , Z )= ( r , s ) ) .
(1.7)
7,s
Whilst this appears to be a new definition, it is in fact precisely the same as Definition 1.2, since we can think of (Y,2)as a single variable X and the pair ( T , s) as a single label i (appealing to the first part of Lemma 1.1,that the values themselves are irrelevant, and only the probabilities matter). Notice that if Y and Z are independent then
H ( Y , 2)=
-
c P ( ( Y ,2 ) = ( T , s)) log (P(Y = T)P(Z = s ) )
(1.8)
T3S
=
H ( Y )+ H ( 2 ) .
+
(1.9)
In general H ( Y ,2)5 H ( Y ) H ( 2 ) (see Lemma 1.14), with equality if and only if Y and Z are independent. Shannon’s work was motivated by the mathematical theory of communication. Entropy is seen to play a significant role in bounds in two fundamental problems; that of sending a message through a noisy channel (such as talking down a crackling phone line or trying t o play a scratched CD) and that of data compression (storing a message in as few bits as possible). In data compression we try t o encode independent realisations of the random variable X in the most efficient way possible, representing the outcome x , by a binary string or ‘word’ y,, of length r,. Since we will store a concatenated series of words yal V yZ2V . . ., we require that the codewords should be decipherable (that is, that the concatenated series of codewords can be uniquely parsed back to the original words). One way t o achieve this is to insist that the code be ‘prefix-free’, that is there do not exist codewords y1 and yJ such that y, = yJ V z for some z . It turns out that this constraint implies the Kraft inequality:
Introduction t o znformataon theory
5
Lemma 1.2 Given positive integers r1 . . . , rn, there exists a decipherable code with codeword lengths r1, . . . rn if and only i f 22-T.
5
1.
(1.10)
i=l
Since we want to minimise the expected codeword length we naturally consider an optimisation problem: n
n
minimise: z r , p , such that: z 2 - ' % 5 1 2=1
(1.11)
2=1
Standard techniques of Lagrangian optimisation indicate that relaxing the requirement that r, should be an integer, the optimal choice would be T , = -logp,. However, we can always get close to this, by picking rz = logp,], so that the expected codeword length is less than H ( X ) 1. By an argument based on coding a block of length n from the random variable, we can in fact code arbitrarily close t o the entropy. Books such as [Goldie and Pinch, 19911, [Applebaum, 19961 and [Cover and Thomas, 19911 review this material, along with other famous problems of information theory. This idea of coding a source using a binary alphabet means that the natural units for entropy are 'bits', and this explains why we use the nonstandard logarithm t o base 2. Throughout this book, the symbol log will refer to logarithms taken t o base 2, and log, will represent the natural logarithm. Similarly, we will sometimes use the notation exp2(z) t o denote 2", and exp(z) for ex.
r-
+
Diflerential entropy
1.1.2
We can generalise Definition 1.2 t o give a definition of entropy for continuous random variables Y with density p . One obvious idea is t o rewrite Equation (1.2) by replacing the sum by an integral and the probabilities by densities. A quantisation argument that justifies this is described in pages 228-9 of [Cover and Thomas, 19911, as follows. Consider a random variable Y with a continuous density p . We can produce a quantised version Y'. We split the real line into intervals It = (td, ( t 1)6), and let
+
P(Y' = t ) =
J,,p ( y ) d y = d p ( y t ) , for some yt E ~
t .
(1.12)
Information Theory and the Central Limit Theorem
6
(The existence of such a yt follows by the mean value theorem, since the density f is continuous.) Then
H ( Y 6 ) = - C P ( Y 6 = t )logP(Y6 = t )
(1.13)
t
=
-
c
(1.14)
6 P ( Y t ) log P(!/t) - 1% 6,
t
+
so by Riemann integrability, 1ims-o (H(Ys) log 6) = - s p ( y ) logp(y)dy. We take this limit t o be the definition of differential entropy.
Definition 1.4 The differential entropy of a continuous random variable Y with density p is:
H ( Y ) = H ( P ) = - / P ( Y ) 1ogdy)dy.
(1.15)
Again, 0 log 0 is taken as 0. For a random variable Y without a continuous distribution function (so no density), the H ( Y ) = 03. We know that t o encode a real number to arbitrary precision will require an infinite number of bits. However, using similar arguments as before, if we wish to represent the outcome of a continuous random variable Y t o an accuracy of n binary places, we will require an average of H ( Y ) n bits. Although we use the same symbol, H , t o refer t o both discrete and differential entropy, it will be clear from context which type of random variable we are considering.
+
Example 1.3 (1) If Y has a uniform distribution on [0, c]:
H(Y)= -
ic; (f> log
dx
= logc.
(1.16)
(2) If Y has a normal distribution with mean 0 and variance n 2 :
H ( Y )=
03
~
(
( 1og(27ra2) ~ ) 2
x2 loge +-)dx= 2a2
log( 27rea2) 2 . (1.17)
( 3 ) If Y has an n-dimensional multivariate normal distribution with mean 0 and covariance matrix C: log( ( 2 ~det ) C) ~ -
log( ( 2 ~ det ) ~C) 2
loge ) dx (1.18) + (xTCP1x) 2
+--n log2 e - log( ( 2 ~2 edet) ~C) .
(1.19)
Introduction to information theory
7
These examples indicate that differential entropy does not retain all the useful properties of the discrete entropy.
Lemma 1.3 (1) Differential entropies can be both positive and negative. (2) H ( Y ) can even be m i n u s infinity. (3) Although the entropy again only depends o n the densities and not the values, since the density itself is not scale invariant, for all a and b:
H(aY
+ b) = H ( Y ) + log a
(1.20)
(so shift, but not scale invariance, holds).
Proof. (1) See Example 1.3(1). (2) If p,(x) = C,x-' (- log, x)--(,+l)on 0 5 x 5 e-l, then for 0 < r 5 1, H(p,) = -co. We work with logarithms to the base e rather than base 2 throughout. By making the substitution y = - log, x,we deduce that e-*
.I
1 x)T+1 d x
x( - log,
1
" 1
=
is 1/r if I- > 0 and 00 otherwise. Hence we know that C, using z = log, (- log, x), e-l
dx =
loge(- loge
x(- log, which is 1/r2 if r > 0 and
x)T+1
I"
(1.21)
yr+'dy
z exp( - r z ) d z ,
= r.
Further,
(1.22)
otherwise. Hence: (1.23)
(1.24) - r(1
log, (- log, x ) dx, (1.25) + r ) x- (-log, x),+l
so for 0 < r 5 1, the second term is infinite, and the others are finite. (3) Follows from properties of the density.
0
Information Theory and the Central Limit Theorem
8
1.1.3
Relative entropy
We can recover scale invariance by introducing the relative entropy distance:
Definition 1.5 For two discrete random variables taking values in {XI,.. . x,} with probabilities p = ( p l , . . . p n ) and q = (41,. . . qn) respectively, we define the relative entropy distance from p to q t o be (1.26)
In the case of continuous random variables with densities p and q , define the relative entropy distance from p t o q t o be
(1.27) In either case if the support supp(p)
supp(q), then D(pllq) = 03.
Relative entropy goes by a variety of other names, including KullbackLeibler distance, information divergence and information discrimination.
Lemma 1.4 (1) Although we refer t o it as a distance, note that D is not a metric: it is not symmetric and does n o t obey the triangle rule. (2) However, D is positive semi-definite: for any probability densities p and q , D(p1lq) 2 0 with equality if and only i f p = q almost everywhere. This last fact is known as the Gibbs inequality. It will continue to hold if we replace the logarithm in Definition 1.5 by other functions.
Lemma 1.5 If ~ ( y ): [0,03] R i s a n y function such that ~ ( y )2 c ( l - l / y ) for some c > 0 , with equality if and only if y = 1 then for any probability densities p and q: --f
(1.28) with equality if and only if p = q almost everywhere.
Introduction to information theory
Proof.
Defining B = supp(p) = {x : p(x)
9
> 0): (1.29)
with equality if and only if q ( x ) = p ( x ) almost everywhere.
0
Other properties of x which prove useful are continuity (so a small change in p / q produces a small change in Dx), and convexity (which helps us t o understand the behaviour on convolution). Kullback-Leibler distance was introduced in [Kullback and Leibler, 19511 from a statistical point of view. Suppose we are interested in the density of a random variable X and wish t o test Ho: X has density fo against H I : X has density fi. Within the Neyman-Pearson framework, it is natural t o consider the log-likelihood ratio. That is define as log(fo(x)/fl(z)) as the information in X for discrimination between Ho and H I , since if it is large, X is more likely t o occur when HO holds. D(folIf1) is the expected value under HO of this discrimination information. Kullback and Leibler go on t o define the divergence D(follf1) D(f1 Ilfo), which restores the symmetry between HO and H I . However, we , we wish t o maintain asymmetry between prefer to consider D ( f O [ [ f l )since accepting and rejecting a hypothesis. For example, D appears as the limit in Stein’s lemma (see Theorem 12.8.1 of [Cover and Thomas, 19911):
+
Lemma 1.6 T a k e X I . . . X , independent and identically distributed (IID) w i t h distribution Q . C o n s i d e r t h e hypotheses Ho : Q = QO and H I : Q = Q1, a n acceptance region A, w i t h Type I a n d 11 error probabilities: CY, = Po; rL (A:) and pn = PQ: (A,). Define: .I
pi
= inf n,,<e
p,.
(1.31)
-D(QollQ1).
(1.32)
Then 1 lim lim - logph r-On-cc n
=
That is, the larger the distance D , the smaller the ,B that can be achieved for a given C Y , so the easier it is to tell the distributions apart. This Lemma helps to explain why the relative entropy D also occurs as an exponent in large deviation theory, in CramQ’s theorem and Sanov’s
Injormatzon Theory and the Central Lamat Theorem
10
theorem. For example, combining the statement of CramQ's theorem (see for example Theorem 2.2.3 of [Dembo and Zeitouni, 19981) with Example 2.2.23 of the same book. we deduce that:
Lemma 1.7 For X I , . . . X, a collection of independent identically distributed Bernoulli(p), for a n y 1 > q > p > 0: 2 4 n-w
n
)
= (1
-p)log
(3) + p l o g (;),
(1.33)
1-q
t h e relative entropy distance from a Bernoulli(p) t o a Bernoulli(q). Another motivation for the relative entropy, in the discrete case at least, is to suppose we code optimally in the belief that the distribution is q ( z ) ,and hence use codewords of length 1- logq(z)l. If in fact the true distribution is p ( z ) , our coding will no longer be optimal. The average number of extra bits required, compared with the optimal coding, is about D(pllq). Being able to control the relative entropy is a very strong result, and implies more standard forms of convergence. For example, the result obtained by [Kullback, 19671:
Lemma 1.8
Convergence in D as stronger t h a n convergence in L1, since
for a n y probability densities p and q :
(1.34) Proof. The key is to consider the set A = { p ( z ) 5 q ( z ) } ,since then s I p ( z ) - q(z)ldz = 2 ( q ( A ) - p ( A ) ) = 2 ( p ( A C )- q ( A " ) ) . We will first establish the so-called log-sum inequality (1.37). For any positive functions p and q (not necessarily integrating t o l ) , we can define new probability densities p ( z ) = p ( z ) I ( z E A ) / p ( A ) and q ( z ) = q(z)I(zE A ) / q ( A ) . Then
(1.35) (1.36) (1.37)
Introduction to informatzon theory
11
by the Gibbs inequality, Lemma 1.4, which means that D(pllq) 2 0. By a similar argument for A", we deduce that:
(1.38) Now, for fixed p
= p ( A ) , consider as a function of y (1.39)
Notice that f ( p ) = 0 and f ' ( y ) = ( l O g e ) ( y - p ) / ( y ( 1 - y ) ) so that for any q 2 p : f(4) = f ( P ) + / - q f ' ( Y ) d Y
2 (4loge)
P
2 (4loge)(y-p),
l
(Y-P)dY = ( 2 1 0 g e ) ( q - P ) 2 , (1.40)
0
as required.
Remark 1.1 1% e
(.I'
Whilst [Volkonskii and R o z a n o v , 19591 gives a bound:
IP(Z) -
ii(")ldz)
-
1% (1
+
.I'
IP(Z)
~
V(Z)ldZ) I D(pllq),
(1.41) that appears better, in that i t s e e m s t o be roughly linear in t h e L 1 distance, expanding t h e logarithm shows it i s to be quadratic in I p ( x ) - q(x)ldx and in f a c t a strictly worse bound.
s
Remark 1.2 I n f a c t , the problem of t h e best bounds of this type i s completely solved in [Fedotov et al., 20031. T h e y u s e methods of convex analysis t o determine t h e allowable values of t h e total variation distance f o r a given value of t h e relative entropy distance. Although the relative entropy does not define a metric, we can understand its properties to some extent. In particular using convexity, as the previous results suggest, Theorem 12.6.1 of [Cover and Thomas, 19911 shows that the relative entropy behaves like the square of Euclidean distance.
Lemma 1.9 For a closed convex set E of probability measures, and distribution Q $! E , let P* be t h e distribution s u c h that D(P*IIQ) = minpEED(PIIQ). T h e n
D(PIIQ) 2 D(PIIP*) + D(P*IIQ),
(1.42)
so in particular i f P, i s a sequence in E where l i m n + m D ( P n ~ ~ Q=) D(P*IIQ) t h e n limn+m D(P,IIP*) = 0:
Information Theory and the Central Limit Theorem
12
A similar result from [Topsoe, 19791 requires control of entropy, not relative entropy. Lemma 1.10
For a convex set C and (P,) i s a sequence in C where limH(P,) = sup H ( P ) < co,
(1.43)
PEG
n
t h e n there exists P* s u c h that limn.+ooD(PnIIP*)= 0.
As the name suggests, the relative entropy D generalises entropy, by considering random variables with respect t o a different reference measure. For example, in Definition 1.5, if q2 = 1 / n then
c
P ( X ) 1% P ( X ) + = logn - H(p),
D(Pll9) =
c
P ( Z ) 1%
n
(1.44) (1.45)
so (up t o a linear transformation), H ( p ) corresponds to taking the relative entropy distance from p to a uniform distribution. It seems natural that in considering the Central Limit Theorem, we should take the relative entropy with respect t o a normal, or Gaussian, measure. This leads to a variational principle and provides a maximum entropy result.
Lemma 1.11 If p is the density of a r a n d o m variable w i t h variance a’, and 4uz is t h e density of a N ( 0 , 0 2 )r a n d o m variable t h e n (1.46)
w i t h equality if and only if p i s a G a u s s i a n density. Proof. By shift invariance we may assume p has mean 0. Then, because log q 5 u ~( x ) is a quadratic function in x:
(1.47) = -H(p)
-+ log(27rea2) 2 -
(1.48)
0
This property gave Linnik the initial motivation to consider the Central Limit Theorem, in which the Gaussian distribution plays a special role, in terms of Shannon’s entropy. Other random variables can be seen to maximise entropy, under appropriate conditions:
Introduction to information theory
13
Lemma 1.12 If p is the density of a random variable supported on the positive half-line and with mean p , and q, is the density of an exponential distribution with mean p then:
with equality if and only if p is an exponential density.
Proof.
Expanding in the same way:
0
I D(pllq,)
X
=
/ m p ( x ) (logp(x) t -loge 0
P
+ logp (1.51)
0 Another useful property (in contrast to the stable case of Chapter 5) is that given a random variable X with density p, it is easy to tell which normal distribution comes closest to it: Lemma 1.13
For a random variable X with a density:
W X ):= infi q x l l d p , d )= D(XIIdiEX,Var x).
(1.52)
,>o
Pro0f. (1.53)
Hence, it is clear that the optimal choice of p is EX,leaving us to minimise log(a2) Var X log e / a 2 as a function of a2. Differentiating, we deduce 0 that we should take o2 = Var X.
+
1.1.4
Other entropy-like quantities
The positivity of the relative entropy offers the easiest proof of many properties of entropy. For example the final part of Lemma 1.1 can be proved simply by considering p, our test measure taking n values, and q , uniform on these same values. Then 0 5 D(pllq) = -H(p) f C , p , l o g n , so the result follows. Similarly:
Information Theory and the Central Limit Theorem
14
For any random variables X and Y .
Lemma 1.14
H ( X , Y )5 H ( X )
+H(Y),
(1.56)
with equality if and only if X and Y are independent.
Proof. Consider the distance from the joint distribution t o the product of the marginals: 0 2 D(P(Z1Y)llP(Z)P(Y))
(1.57)
= C P ( G Y ) (lOgP(z1Y) - b P ( Z ) - logP(Y)) X,Y
=
+
-H(X, Y ) H ( X )
+H(Y)
(1.58) (1.59)
as required.
+
This quantity, H ( X ) H ( Y ) - H ( X , Y ) , occurs commonly enough that it is given its own name: mutual information I ( X , Y ) . It represents how easy it is to make inference about random variable X from a knowledge of random variable Y (and vice versa - an interesting symmetry property).
Definition 1.6
For random variables X and Y , define:
+
(1) mutual information I ( X ,Y ) = H ( X ) H ( Y ) - H ( X ,Y ) (2) conditional entropy H ( Y I X ) = H ( X , Y ) - H ( X ) . These different entropies are illustrated schematically in Figure 1.2.
In fact, this schematic diagram suggests an interesting relationship, known as the Hu correspondence, which indicates that our understanding of entropy as ‘the amount of information gained’ really carries through. This correspondence was first proved in [Hu, 19621 and later discussed on page 52 of [Csiszk and Korner, 19811. Specifically, the correspondence works by a 1-1 matching between entropies and the size p of sets. That is (1.60) (1.61) (1.62) (1.63) Then Hu shows that any linear relationship in these H and I is true, if and only if the corresponding relationship in terms of set sizes also holds. Thus
Introduction to information theory
15
H ( X ,Y )
Different types of entropies
Fig. 1.2
-
for example, the statement of Lemma 1.14 is clear since
H ( X 1 Y ) 5 H ( X )+ H ( Y )
P(A u B ) I P(A) + P ( B )
(1.64)
In fact the situation is slightly more complicated for intersections of 3 or more sets. It turns out that 1-1can be negative, so we have to think of it as a signed measure. Conditional entropy gains its name since
(1.66) (1.67) X
=
Y
E p ( x ) H ( Y / X = x)
(1.68)
X
(here treating Y I X = x as a random variable, for each x). The conditional entropy allows a neat proof of the fact that entropy increases on convolution.
Information Theory and the Central Lamat Theorem
16
Lemma 1.15
For X a n d Y independent:
H(X Proof.
+ Y ) 2 max(H(X),H ( Y ) ) .
(1.69)
Since only the probabilities matter
H(X
+ Y ( X )= H ( Y 1 X ) = H ( Y ) ,
(1.70)
so we know that
H ( Y ) = H(X
+ Y l X ) I H(X + Y ) ,
(1.71)
since entropy decreases on conditioning. This is a direct consequence of 0 Lemma 1.14, since H ( U I V ) = H ( U , V ) - H ( V ) 5 H ( U ) .
1.1.5
A x i o m a t i c definition of e n t r o p y
It is reasonable to ask whether Definition 1.2 represents the only possible definition of entropy. [ R h y i , 19611 introduces axioms on how we would expect a measure of information, or ‘randomness’ to behave, and then discusses which functions satisfy them. A somewhat different system of axioms was proposed in [Faddeev, 19561. Rknyi’s method discusses generalised probability distributions, which sum to less than 1, whereas Faddeev deals only with probability distributions. R h y i ’ s axioms are:
Definition 1.7 For any collection of positive numbers P = { p l , . . . p k } , such that 0 < C,p, 5 1: (1) Symmetry: H ( P ) is symmetric in indices p . (2) Continuity: H ( { p } ) is continuous in p , for 0 < p 5 1. (3) Normalisation: H({1/2}) = 1. (4) Independence: for X , Y independent H(X, Y )= H ( X ) ( 5 ) Decomposition:
where
+H(Y)
and g(x) is some fixed function.
Introduction t o information theory
17
He shows that if g(z) 5 1, the only possible function H satisfying these properties is a generalised version of the discrete entropy of Definition 1.2: (1.73) Further, if g(x) = 2(”-l)“, then the only possible H is the so-called R h y i &-entropy defined by:
Definition 1.8
Given a probability distribution p = ( P I , . . . p k ) define the Rknyi a-entropy to be: (1.74) Note that by L’H6pital’s rule, since %aZ
= a”
log, a ,
Similarly:
Definition 1.9 Given probability distributions p and q, define the relative a-entropy by (1.76) Again by L’H6pital’s rule, lim,-l
D,(pJJq)= D(pllq) since
The paper [Hayashi, 20021 discusses the definition and limiting behaviour of this quantity D,. 1.2
1.2.1
Link to thermodynamic entropy Definition of thermodynamic entropy
Information-theoretic entropy is named by analogy with the better-known thermodynamic entropy. Indeed, Tribus in [Levine and Tribus, 19791 reports a conversation where von Neumann suggested to Shannon that he should use the same name:
18
Information Theory and the Central Lzmit Theorem
You should call it ‘entropy’ and for two reasons; first, the function is already in use in thermodynamics under that name; second, and more importantly, most people don’t know what entropy really is, and if you use the word ‘entropy’ in an argument you will win every time. In the study of statistical physics, we contrast macrostates (properties of large numbers of particles, such as temperature and pressure) and microstates (properties of individual molecules, such as position and momentum). The link comes as follows:
Definition 1.10 Suppose there are R microstates corresponding t o a particular macrostate. Then the entropy of the macrostate is
s = klog, R ,
(1.78)
where k is Boltzmann’s constant (we don’t particularly worry about constant factors, they can just pass through our analysis). Suppose each microstate r can occur with probability p,. Consider a very large ensemble of v replicas of the same system, then on average there will be u, replicas in the r t h microstate, where v, is the closest integer to up,. In this case, by Stirling’s formula (1.79) Hence the entropy of the ensemble will be: (1.80) (1.81) (1.82) Since the entropy of a compound system is the sum of the parts of the system,
s = sv/u= -k
c
p,log,p,,
(1.83)
which, up t o a constant, is the formula from Definition 1.2 for informationtheoretic entropy. This is discussed in more detail in [Georgii, 20031. We argue that the relative entropy plays a role analogous t o the Helmholtz free energy described on pages 64-5 of [Mandl, 19711. That
Introduction to information theory
19
is, the free energy is
F=E-TS,
(1.84)
where E is energy, T is temperature and S is entropy. In comparison, if for some potential function h(z), the density g(z) = exp(-,Bh(z)), then (1.85)
where p is the inverse temperature ,B = 1/T 1.2.2
M a x i m u m entropy and the Second Law
Lagrangian methods show that the entropy S is maximised subject t o an energy constraint by the so-called Gibbs states. Lemma 1.16 -
The maximum of C p , . l o g p , such that
Cp, = 1, C p T E T= E
(1.87)
comes at pi = exp(-PEi)/Zp, f o r some /3 determined by E and where the exp(-PEi). partition function Zo =
Xi
We can find
P , given a knowledge of 20,since
The Second Law of Thermodynamics states that the thermodynamic entropy always increases with time, implying some kind of convergence t o the Gibbs state. Conservation of energy means that E remains constant during this time evolution, so we can tell from the start which Gibbs state will be the limit. We will regard the Central Limit Theorem in the same way, by showing that the information-theoretic entropy increases to its maximum as we take convolutions, implying convergence to the Gaussian. Normalising appropriately means that the variance remains constant during convolutions, so we can tell from the start which Gaussian will be the limit. Note, however, that in this book we only use the Second Law as an analogy, not least because of its controversial status. Whilst it might sound surprising to refer t o such a well-known and long-established principle in
20
Information Theory and the Central Limit Theorem
this way, there remains a certain amount of argument about it. Depending on the author, the Second Law appears t o be treated as anything from something so obvious as not t o require discussion, to something that might not even be true. A recent discussion of the history and status of the Second Law is provided by [Uffink, 20011. He states that: Even deliberate attempts at careful forniulation of the Second Law sometimes end up in a paradox. Uffink provides a selection of quotations to illustrate the controversial status of the Second Law, offering Eddington’s view (from page 81 of [Eddington, 19351) that if your theory is found to be against the second law of thermodynamics, I can give you no hope; there is nothing for it but to collapse in deepest humiliation. In contrast, Truesdell complains that (see page 335 of [Truesdell, 19801) Seven times in the past thirty years have I tried to follow the argument Clausius offers [...I and seven times has it blanked and gravelled me [. . .]. I cannot explain what I cannot understand. Work continues to try to set the Second Law on a rigorous footing. For example [Lieb and Yngvason, 19981 uses an axiomatic framework, considering a partial ordering where X + Y denotes that state X can be transformed into state Y via a ‘physical’ process, and going on to deduce the existence of a universal quantity referred to as entropy. From our point of view, all this controversy and discussion will not be a problem; we will never require a rigorous statement of the Second Law. We rather merely draw the reader’s attention t o the striking fact that the Central Limit Theorem (probably the best-known result in probability and statistics) can be seen as similar t o the Second Law of Thermodynamics (one of the best-known results of physics). In general, we prefer to consider decrease of relative entropy, rather than increase of Shannon entropy. In the Central Limit Theorem, thanks t o the way that the normalisation fixes variance, this will be equivalent. However, for other models, the difference is important. For example, consider the nstep distribution P, of a Markov chain converging to a non-uniform equilibrium distribution w . The entropy clearly need not increase; if Po is uniform, then H(P0) is maximal, and so since PI # Po, H(P0) > H(P1). However
Introduction to information theory
21
Theorem 1.1in Section 1.4.1 shows that for all n: D(Pnllw) 5 D ( P n - l l / ~ ) , and indeed limn+,m D(Pnllw) = 0.
1.3 1.3.1
Fisher information
Definition and properties
Another quantity which will be significant for our method is Fisher information.
Definition 1.11 Let U be a random variable with density p(u) which is a differentiable function of parameters (el,&, . . . On). The Fisher information matrix is defined to have ( i , j ) t h entry given as (1.89)
If the derivative does not exist then the Fisher information is defined to be infinite. The idea is that we measure how high the peaks are in the log-likelihood function - that is, how much small changes in the parameters affect the likelihood. In the next few chapters, we will only need to consider the special case where p(x) = h ( z - 8), that is for a location parameter 8. However in Chapter 5, which discusses stable distributions, we shall use the full general definition. In this special case of a location parameter, z 6’) = ( - a / d z ) h ( z - 0) then (with a sign change) we can since ( a / a Q ) h (define:
Definition 1.12 If U is a random variable with continuously differentiable density p ( u ) , define the score function pr/(u) = p ’ ( u ) / p ( u ) = d/du(log, p ( u ) ) and the Fisher information (1.90) We sometimes refer to this as the ‘Fisher information with respect t o location parameter’ to distinguish it from the more general Definition 1.11.
If U is N ( p , a 2 ) , then the score function p ~ / ( u = ) Example 1.4 -(z - p ) / u 2 and J ( U ) = l/a2. Since p ( u ) = p(O)exp(J:pr/(v)dv), the score function is linear if and only if the variable is Gaussian. This is the characterisation of the Gaussian that we shall use.
Information Theory and the Central Limit Theorem
22
Example 1.5 If U has a I’(n,0) distribution, that is if p ( u ) = e-o”Onun-l/I’(n), the score function pu = p ’ ( u ) / p ( u ) = -0 + ( n - l ) / u . Then, using the fact that IEU“ = r(n s ) / ( r ( n ) e “(if ) s > -n):
+
IEpi(U) = (n- 1 ) 2 for n
e2 ( n - 2 ) ( n - 1)
-
0 2 ( n - I)Qn-1
02
+ @ = n- - 2
1
(1.91)
> 2.
Notice the scaling of the Fisher information. Since the density of aU satis= p u ( z / u ) / a and p h c r ( s ) = p’c,(z/a)/a2,then fies pa,-,(.)
Lemma 1.17
One useful property of the score function is the Stein identity:
Lemma 1.18 Given a random variable X with density p and score function p, f o r any smooth function f which is suitably well-behaved at 3 ~ 0 3 :
Proof.
This is simply integration by parts:
(1.96) Hence it is clear that ‘suitably well-behaved’ includes a requirement that 0 limz+hm f ( z ) p ( z )= 0. This observation is the basis of Stein’s method, another way of proving Gaussian convergence (see for example [Reinert, 19981 for a review of the method). In summary, for normal 2,Equation (1.94) implies that for any test function f ,
IEZf(2) - f’(2)= 0.
(1.97)
This means that we can assess the closeness of some W to normality by bounding (1.98)
Introduction to i n f o m a t i o n theory
23
In fact, Stein’s method uses a continuous bounded function h such that (1.99) and bounds
Eh(W) - IEh(Z),
(1.100)
since this, together with (1.99), gives control of (1.98). In our situation, notice that controlling Fisher information will control (1.98) for functions f in L 2 ( W ) not , just uniformly bounded. That is:
n J I E ( P ( W )+ WY. (1,101) We can give an equivalent to the maximum entropy result of Lemma 1.11. That is, the Gaussian minimises the Fisher information, under a variance constraint, a fact known as the Cram&-Rao lower bound.
E W f ( W ) - f’(W) = W ( W ) ( W+ P
W ) ) Ir
Lemma 1.19 Given a random variable U with m e a n p and variance u 2 , the Fisher information is bounded below:
J ( U ) 2 1/a2,
(1.102)
with equality zf and only if U is N ( p , u 2 ) . By the Stein identity, Lemma 1.18, for any a , b: E(aU+b)pu(U) = -a. Hence:
Proof.
Equality holds if and only if p,(U) is linear.
0
Definition 1.13 For random variables U and V with densities f and g respectively, define the Fisher information distance:
Notice that we can make an entirely equivalent definition of Fisher information and Fisher information distance. That is, for random variables U
Information Theory and the Central Limit Theorem
24
and V with densities f and g notice that
(1.107) and
The Cramkr-Rao lower bound, Lemma 1.19, motivates us to give a standardised version of the Fisher information, with the advantages of positive semi-definiteness (with equality only occurring for the Gaussian) and scaleinvariance. Recall that these are properties shared by the relative entropy.
Definition 1.14 If U is a random variable with mean p , variance u2 and score function pu, define the standardised Fisher information: 2IEp;
(U)- 1 =
where 2 is a N ( p ,n 2 ) Example 1.6 By Examples 1.4 and 1.5: if U is N ( 0 ,0 2 )then J,t(U) and if U is r(n,O)then J s t ( U ) = (n/02)(Oz/(n - 2)) - 1 = 2 / ( n - 2).
=
0,
Since, under a variance constraint, the Gaussian maximises entropy and minimises Fisher information, it is perhaps not surprising that a link exists between these two quantities. The link is provided by de Bruijn’s identity Theorem C . l (see Appendix C for more details and a proof) which states that if U is a random variable with density f and variance 1, and 2, is N ( 0 ,T ) , independent of U , then
Notice that by Lemma 1.19, the integrand is non-negative, and is zero if and only if U 2, is Gaussian, which occurs if and only if U is Gaussian, as we would expect. We will prove convergence in relative entropy by proving that Jst(U 2,)converges to 0 for each r , and using a monotone convergence result to extend the result up to convergence in D.The advantage of this approach is that the perturbed random variables J ( U ZT)have densities which are smooth, and other useful properties.
+
+
+
Introduction to information theory
25
The book [F’rieden, 19981 derives many of the fundamental principles of physics, including Maxwell’s equations, the Klein-Gordon equation and the Dirac equations, from a universal principle of trying to minimise Fisher information-like quantities. 1.3.2
Behavaour o n convolution
The advantage of Fisher information over entropy from our point of view is that we can give an exact expression for its behaviour on convolution, and exploit the theory of L2 spaces and projections. Lemma 1.20 If U , V are independent r a n d o m variables and W = U w i t h score f u n c t i o n s p u , p v a n d p w , t h e n
+V
Proof. If U , V have densities p ( u ) ,q ( v ) ,U + V has the convolution density r ( w ) = J p ( u ) q ( w - u ) d u , so that r’(w) =
J
p ( u ) -aq (w-u)du
aw
=-
J 2
p ( ~ ) - ( w - u ) d ~=
J
p’(u)q(W-u)du (1.112)
and hence:
(1.114)
Similarly, we can produce an expression in terms of the score function d(V) 0
/m.
Lemma 1.21
+
If U , V are independent t h e n f o r a n y P E [O, I]:
(1) J ( U V ) 5 P (2) J ( J P U + m
2 W ) + (1 - P)”(V), v )5 P J P ) + (1 - P ) J ( V )
w i t h equality only if U a n d V are Gaussian.
Proof. We can add p times the first expression in Lemma 1.20 to 1 - P times the second one. to obtain:
Information Theory and the Central Limit Theorem
26
Then Jensen’s inequality gives
J ( W )= E p $ ( W )
+ (1 P)Pv(V)IW)~I 5 E [ W ( P P U ( U )+ (1 P)pr.(V))21W)1 = P 2 ~ P 2 , ( U+ ) (1 P)”EP2v(V), = E[E(Ppu(U)
-
-
-
(1.116) (1.117) (1.118)
substituting flu and m V for U and V respectively, we recover the second result. A proof that equality can only occur in the Gaussian case is given in [Blachman, 19651, exploiting the fact that equality holds if and only if: PPr.r(W
-
+
u) (1 - P ) P V ( V )
= PW(W),
for all u , w .
(1.119)
+
Integrating with respect to v , -Plogp(w - u) (1 - P)logq(v) = u ( r ’ ( w ) / r ( w ) ) c ( w ) , Setting w = 0, we deduce that C ( W ) and p w ( w ) are differentiable. Differentiating with respect t o w and setting w = 0, we see that p ’ ( - u ) / p ( - u ) is linear in v , and hence p is a normal density. 0
+
This result is a powerful one: it allows us to prove that the Fisher information decreases ‘on average’ when we take convolutions. In particular, we establish a subadditive relation in the IID case. That is, writing U, = (C,”=, X i ) / & , for independent identically distributed Xi,taking = n / ( n rn) in Lemma 1.21(2) gives that
+
nJ(Un)
+ rnJ(um)2 ( n+ m ) J ( U n + m )
(1.120)
(with a corresponding result holding on replacing the J by the standardised version Jst). Now, as discussed in for example [Grimmett, 19991, if such an expression holds, and if J ( U n ) is finite for some n, then J ( U n ) converges to a limit J . Further, by taking m = n in (1.120), it is clear that this convergence is monotone along the ‘powers-of-2 subsequence’ n k = 2 k . Of course, this in itself does not identify the limit, nor show that the Fisher information converges to the Cram&-Rao bound, but it provides hope that such a result might be found. Two particular choices of P in Lemma 1.21(1) are significant here:
Lemma 1.22 (1) J ( U
If U , V are independent random variables then:
+ V )I J ( U )
1
>-+-1
(2) J ( U + V ) - J ( U )
1
J(V)’ with equality in the second case if and only i f U and V are Gaussian.
Introduction to information theory
27
Proof. Taking /3 = 1 we deduce the first result, and the second follows by taking /3 = J ( V ) / ( J ( U ) J ( V ) ) ,which is the optimal value of /3 for given J ( U ) ,J ( V ) . 0
+
Whilst we might expect t o generalise this result t o the case of weakly dependent random variables, with an error term which measures the degree of dependence, such a result has proved elusive. If found, it would allow an extension of the entropy-theoretic proof of the Central Limit Theorem to the weakly dependent case. The paper [Zamir, 19981 gives an alternative proof of Lemma 1.22(2). The argument involves a chain rule for Fisher information, so that for random variables X and Y
and hence for any function
4,
This implies that by a rescaling argument, for any random vector N and matrix A:
J(AN) 5 (AJ(N)-’AT)-’,
(1.123)
so taking A = (1,l)and N = (U,V ) :
(1.124)
1.4 1.4.1
Previous information-theoretic proofs
Re‘nyi’s method
Section 4 of [Rknyi, 19611 uses Kullback-Leibler distance t o provide a proof of convergence to equilibrium of Markov chains presented below as Theorem 1.1. In this proof, Rknyi introduces a general method, which has applications in several areas - it is also the basis of CsiszBr’s proof [CsiszBr, 19651 of convergence to Haar measure for convolutions of measures on compact groups, and a similar argument is used in Shimizu’s proof of weak convergence in the Central Limit Theorem [Shimizu, 19751. Unfortunately, Rknyi’s method seems t o only apply in ‘IID cases’, where there is some sort
Information Theory and the Central Limit Theorem
28
of homegeneous or identical structure. In these circumstances, define Yn to be the nth convolution power X * X * . . . * X , (defined appropriately).
Definition 1.15
Rknyi’s method has three parts:
(1) Introduce an estimator F and, using convexity, show that F ( Y * X) 5 F ( Y ) , with equality only in a particular ‘special’ case. Since F ( Y n ) is monotonic decreasing and bounded below, we know that F ( Y n ) 4 F for some F . (2) Use compactness arguments to show that for some subsequence R,, and some Y , Y,, + Y . ( 3 ) Then, using continuity of F , since subsequences of convergent subsequences also converge to the same limit:
F
=
lim F ( Y n s )= F ( Y )
(1.125)
S+CC
F‘
= lim 5-03
F ( Y n s + l ) = Slim F(Yn3* X ) = F ( Y * X ) A C 0
(1.126)
Then, equating (1.125) and (1.126) gives F ( Y * X ) = F ( Y ) ,so we can identify Y and hence the limit F .
As an example, consider Rknyi’s proof of convergence to equilibrium for finite-state Markov chains: Theorem 1.1 Consider a Markov chain taking values on a finite state space { 1 , 2 , . . . , N } , with transition matrix P , n-step transition matrix P(,) and invariant distribution w. If Pzl > 0 for all i , j , then writing Ai for the ith row of any matrix A :
lim
n+m
D ( P J ~ ) I I W= )
o f o r any i .
(1.127)
Proof. Step (1): We show that if w is the invariant distribution of P , then for any probability vector x : D(xPllw) 5 D ( x l J w )with , equality if and only if x = w .
Introduction to information theory
Define Q to be the reversed transition matrix: Hence,
29
Qij = WiPij/wj.
Then
(XP)j = c k X k P k j = W j x k XkQkj/Wk.
(1.128)
The inequality comes from Jensen's inequality applied to f ( ~ =) IC log x , with the probability distribution Qk. so that E X log EX 5 E(X log X ) . We also use the fact that C, w3Qk3 = C, W k P k 3 = w k . Since Q k g is non-zero for all k, equality holds if and only if 21, = wk for all k. Hence if we define D , = D(P,(")IIw),D, is decreasing and bounded below, and thus converges to some D 2 0. Step (2): The second part of the argument is trivial - since we have finitely many states, by compactness of [0,1]N Z ,there exists a subsequence ns and a stochastic matrix R , such that P3T 4R 3 k for all 3 , k . Step (3): We conclude by saying that:
D
=
lim
D ( P , ( ~ ' )= / /D ~ ()
R,II~)
(1.131)
S-03
D
=
lim D(P,( n . + l ) llw) S-30
=
D ( ( R* P),llW)
=
lim D((P("\)P),llw)
(1.132)
S'CC
(1.133)
Now as above, equating (1.131) and (1.133), D ( ( R * P ) z l l w ) = D(R,llw) 0 implies that R, = w , and hence D = 0. The paper [Kendall, 19631 extends this argument to countable state spaces, for both discrete and continuous time. Since this method uses several times the fact that the Markov chain has a homogeneous structure, it highlights the challenge to come up with a similar argument in non-IID cases. Notice that although other methods tell us that convergence will occur at an exponential rate, this method does not give an explicit bound on the rate of convergence. Linnik, in two papers [Linnik, 19591 and [Linnik, 19601, produced a proof of the Central Limit Theorem establishing Gaussian convergence for normalised sums of independent random variables. First [Linnik, 19591 considers random variables valued on the real line, satisfying the Lindeberg
30
Information Theory and the Central Limit Theorem
condition, Condition 1 (see Chapter 3). The second [Linnik, 19601 extends the results to the case of real random vectors. See Chapter 3 for more details. Now [Rknyi, 19611 promises that another paper will provide “a simplified version of Linnik’s information-theoretic proof of the Central Limit Theorem”. Commenting on this, in his note after this paper, [CsiszAr, 19761 points out that Rknyi never achieved this. He states that “It seems that the Central Limit Theorem does not admit an informational theoretic proof comparable in simplicity with the familiar ones”. He goes on to state that Rknyi’s method “indicates the type of limit problems to which the information theoretical approach is really suitable”. However later papers use information-theoretic arguments in a natural way to provide such a proof. 1.4.2
Convergence of Fisher information
The Central Limit Theorem is perhaps the best-known result in probability and statistics. It states that: Theorem 1.2 For X I ,Xz, . . . a collection of independent identically distributed random variables with finite variance, the normalised s u m
converges weakly to a normal N ( 0 , l ) .
This theorem is a special case of the Lindeberg-Feller theorem, theorem 3.1, discussed later. The standard proof of the Central Limit Theorem involves characteristic functions. This book offers an alternative approach, using quantities based on information theory. We hope that this will offer improved understanding of why the Gaussian should be the limit, and allow us to view convergence in other regimes (including convergence to the Poisson and convergence to the Wigner law in free probability) where the behaviour of the characteristic function is more obscure, in the same way. Since the Gaussian distribution is the limit in this Central Limit regime, and since it maximises the entropy and minimises the Fisher information subject to a variance constraint (see Lemma 1.11 and Lemma 1.19), it is natural to wonder whether the entropy and Fisher information of the normalised sum converge to this extremal value. The fact that the normalisation ensures that the variance remains constant makes this even more
Introduction to information theory
31
provocative, inviting the parallels with the Second Law of Thermodynamics previously described in Section 1.2. Papers [Shimizu, 19751 and [Brown, 19821 deal with the question of the behaviour of Fisher information on convolution in the independent identically distributed case. Shimizu identifies the limit of the Fisher information, which allows him to deduce that weak convergence occurs. Whilst Brown does not identify the limit, he also manages to prove weak convergence. However, Brown’s methods are the ones that Barron extends in the course of his proof [Barron, 19861 of convergence in Kullback-Leibler distance and the ones that we extend to the non-identical case in Chapter 3. The paper [Shimizu, 19751 manages to identify the limit of the Fisher information using a version of RBnyi’s method.
Theorem 1.3 Consider X I X ,2 , . . . independent identically distributed random variables with zero mean, variance cr2 and define U, = (CzlX i ) / and Y, = U, + 2;) . Here Z,(.)are N ( O , T ) independent of Xi and each other. T h e n for any T > 0 : lim Jst(Yn)= 0,
(1.135)
71-00
and hence Y, converges weakly t o N(O,1 + T ) . Proof. By Lemma 1.21, if U and U’are IID, then J ( ( U + U ’ ) / f i ) 5 J ( U ) , with equality if and only if U is Gaussian. The compactness part of the argument runs as follows: there must be a subsequence n, such that U p , converges weakly to U . If fs(z) is the density of the smoothed variable Yp.., then f s and its derivative d f J d z must converge, so by dominated convergence, J(Y2.+) -+ J ( Y ) , where Y = U+Z,.Further, J ( Y 2 r L s + ~J)( ( Y + Y ’ ) / d ) As . with RBnyi’s method, we , so J ( Y ) = l / ( l + ~Subadditivity ). deduce that Y must be N ( 0 , 1 + ~ )and allows us to fill in the gaps in the sequence. Let QOz denote the N ( 0 , a 2 ) and @ the N ( 0 , l ) distribution function. Then Lemma E.2 shows that if F is the distribution function of X , with EX = 0, then --f
(1.136)
+ +
allowing us to deduce weak convergence of Y, to the N ( 0 , 1 T ) for all T . Equation (4.7) from [Brown, 19821 states that if Y = U Z,,where 2, is N ( O , T )and independent of U , then considering distribution functions,
Information Theory and the Central Limit Theorem
32
for all
E:
@(E/T)FII ( E~)
I F Y ( ~I)@ ( € / ~ ) F u+( zE) + @ ( - E / T ) ,
(1.137)
which allows us to deduce weak convergence of Un to N ( 0 , l ) . Now, since it is based on a RBnyi-type argument, we have problems in generalising Shimizu’s method to the non-identical case. On the other hand, the method of [Brown, 19821 described in the next chapter will generalise. Miclo, in [Miclo, 20031, also considers random variables perturbed by the addition of normal random variables, and even establishes a rate of convergence under moment conditions. Other authors have used information theoretic results to prove results in probability theory. For example Barron, in [Barron, 19911 and [Barron, 20001, gives a proof of the martingale convergence theorem, and [O’Connell, 20001 gives an elementary proof of the 0-1 law.
Chapter 2
Convergence in Relative Entropy
Summary In this chapter we show how a proof of convergence in Fisher information distance and relative entropy distance can be carried out. First we show how our techniques are motivated via the theory of projections, and consider the special case of normal distributions. We then generalise these techniques, using Poincark inequalities t o obtain a sandwich inequality. With care this allows us to show that convergence occurs at a rate of O(l/n), which is seen to be the best possible.
2.1 2.1.1
Motivation
Sandwich inequality
Although we would like to prove results concerning the behaviour of relative entropy on convolution, it proves difficult to do so directly, and easier t o do so by considering Fisher information. Specifically the logarithm term in the definition of entropy behaves in a way that is hard to control directly on convolution. In contrast, the L2 structure of Fisher information makes it much easier to control, using ideas of projection (conditional expectation). This means that we can give an exact expression, not just an inequality, that quantifies the change in Fisher information on convolution: Lemma 2.1 For independent and identically distributed random variables U and V , with score functions pu and pv, writing p* for the score function 33
Information Theory and the Central Limit Theorem
34
J ( U )- J
7)
u+v ( 7 = E (P* ) (u+v
-
Jz1 ( P U ( U ) + PV(V)))
2
. (2.1)
Proof. We assume here and throughout the next four chapters that the random variables have densities. Recall that Lemma 1.20 gives us information on how the score function behaves on convolution. That is, for U ,V independent random variables and W = U V with score functions p ~p~, and pw then
+
so for any
p, by taking linear combinations,
In the case where U and V are independent and identically distributed, it is natural to take p = 1/2. By rescaling using the standard Central Limit Theorem normalisation, we obtain by combining (1.92)and (2.3)that
Hence expanding, we obtain that
Convergence in relative entropy
35
Note that independence means that Epu(U)pv(V)= ( E ~ ~ J ( U ) ) ( E ~ ~=( 0, since the Stein identity (Lemma 1.18) with f(x) = 1 gives us that 0 Epu(U)= Epv(V) = 0. Further, in the IID case, it is simple t o see how the Fisher information distance (see Definition 1.13) behaves. That is, if U and V have variance g 2 , then so does T = (U V ) / & , so that
+
r=
.] ,
(2.10)
and the linear term just passes through. In particular, it follows that
Lemma 2.2 For an IID collection of variables Xi,if U, = (XI + . . . + X n ) / & and UA = (Xn+l . . . X 2 n ) / & , then i f Un has score function
+ +
Pn.
Notice that the RHS of this expression is a perfect square and hence is positive. This means that Jst(Un) is decreasing on the powers-of-two subsequence, and since it is bounded below by zero, it must converge on this subsequence. Hence the sequence of differences Jst(Un)- Jst(UZn) converges to zero. This means that the right-hand side of the identity (2.11) must itself converge to zero. (Of course, the fact that Jst(Un)converges does not mean that it must converge t o zero, but we do hope to identify the limit). We are therefore motivated t o consider the question of when a function of a sum f(x + y) can be close t o a sum of functions g(z) h(y). Clearly, for f linear, we can find functions g and h such that f ( x y) = g(z) h ( y ) , and for non-linear functions this will not be possible. So, it seems that the fact that f (x y) gets closer to g(z) h ( y ) means that the function f must get ‘closer to linear’. Since the score function being close t o linearity implies that we are close t o normal in Fisher information distance, we hope t o deduce that Jst(Un) converges t o zero.
+
+
+
+
+
Information Theory and the Central Limzt Theorem
36
2.1.2
Projections and adjoints
It becomes clear that we need to understand the action of the projection map M from pu t o pw. However, it turns out t o be easier t o consider the adjoint map L.
Lemma 2.3 If U and V are independent r a n d o m variables w i t h densities p and q respectively and W = U V h a s density r , define t h e m a p s M and L by
+
(2.12)
Lk(z)=
J
k(z + y ) q ( y ) d y = Ek(2 + V).
(2.13)
T h e n L i s t h e adjoint of M ( w i t h respect to appropriate i n n e r products). Proof.
For any g and h:
(2.16)
by taking u = 'u
+ z.
0
Figure 2.1 depicts the effect of these projections: the upper line represents the set of functions of ( X Y ) ,the lower line represents the set of sums of functions of X and Y (with the point of intersection representing the linear functions). The effect of L and M to map from one set to the other is shown schematically. This offers us an alternative proof of Lemma 1.20. We use the Stein identity characterisation that p is the only function such that (h,p), = -(h', l), for all h (see also the discussion of conjugate functions in Chapter 8). Notice that the derivative map commutes with the map L , so that
+
Convergence in relative entropy
Fig. 2.1
37
Role of projections
(Lk)’ = Lk‘. Hence for any function f:
so that M p has the corresponding property, and so must be the score with
respect to r . Equipped with this notation, we can see that we would like t o bound
Further, since ( L M ) is self-adjoint, its eigenfunctions with distinct eigenvalues are orthogonal. Hence we would like t o bound from above all the eigenvalues of L M , t o establish the existence of a non-zero spectral gap, or equivalently a finite Poincark constant (see Appendix B for a discussion of these quantities).
Given independent identically distributed random variables U and V, if there exists a constant K such that for all f
Lemma 2.4
Information Theory and the Central Limit Theorem
38
then for all 9:
( M g ,M d P (S>S)P
5-
1
2+K'
(2.23)
Proof. There is a 1-1matching between eigenfunctions of L M and eigenfunctions of M L (that is, if h is a A-eigenfunction of M L : M L h = Ah implies that L M L h = ALh so ( L M ) ( L h )= A ( L h ) ,so Lh is a A-eigenfunction of L M ) . This means that the bound (2.21) is equivalent to bounding (for all functions f ) (2.24) Hence Equation (2.22) implies that for all f and g
(2.26) (2.27) and so the result follows.
0
Trivially, Equation (2.22) always holds with K = 0, which implies that the largest eigenvalue will be less than or equal to l / 2 . The question is, can we improve this? 2.1.3
Normal case
In the special case of Gaussian random variables, we can exploit our knowledge of these maps L and M and their eigenfunctions. In this way Brown [Brown, 19821 establishes the following result: Lemma 2.5
For any functions f and g there exist some a,b such that
when U , V are independent identically distributed normals.
Convergence an relative entropy
39
Proof. In our notation, Brown fixes g and varies f to exploit the fact that for all f and g
lE (f(U+ V ) - g(U) - g(V)I2L E (2Mg(U + V )- g(U) - d V ) Y = 2Eg2(U) - 4EMg(U
Lemma 2.6Lestablishesthat IE(Mg(U
+ V)2.
+ V))’ 5 IEg(U)’/4.
(2.29) (2.30)
0
We prefer to fix f and vary g, since for all f and g
IE ( f ( U + V ) - g ( U ) - g
( q 22
IE ( f ( U + V )- L f ( U ) - L s ( v ) ) 2 (2.31)
= IEf(U+V)2
-2lE(Lf(U))2
(2.32)
2 21E(Lf(u))2,
(2.33)
+
using the equivalent result that E(Lf(U))’ 5 E f ( U V)’/4. In either case, as the linear part passes through on convolution, as shown in Equations (2.9) and (2.10), we can subtract off the best linear approximation to f and g. We will show how to prove these results, in the spirit of the spectral representation already described. For the remainder of the chapter, we denote by 4,,2 the N ( 0 , u 2 )density, and write 4 for the N ( 0 , l )density. We closely related to the M and 15 of Lemma 2.3 can define maps hil and via a scaling:
z,
(2.34) (2.35)
We can identify the eigenfunctions of 2 and as the Hermite polynomials, which form an orthogonal basis in L2 with Gaussian weights (see the book [Szego, 19581 for more details of orthogonal polynomials).
Definition 2.1 The generating function of the Hermite polynomials with respect to the weight 4 u is: ~ G(z, t ) =
tr
--H,(z, r!
u2 ) = exp
(-2 +t ~, ) u2t2
IC E
R.
(2.36)
Using this, H o ( z , a 2 )= 1, H l ( z , u 2 )= IC, H 2 ( x , u 2 )= IC’ - u2 and
( H n ,H,)
= S H , ( z , ~ ’ ) H , ( z , u ’ ) 4 ~ , ( : r )= d b,na2nn!, z
(2.37)
Information Theory and the Central Limit Theorem
40
and {H,} form an orthogonal basis for L2(4nz( x ) d x ) .
Lemma 2.6
z z z
For the % and
defined in (2.34) and (2.35):
and are adjoint to each other. (1) (2) and are the same map. each take H,(x, 1) to 2-'/2H,(2, 1). (3) The maps and
a
Pro0f. (1) See Lemma 5.9 for a proof in a more general case. For any g and h:
as required, by taking y = f i u - x. ( 2 ) Rearranging the special form of the normal density in the integral kernel, we see that in this case,
so that with a change of variables y =
Ax - u , for all u (2.43) (2.44)
-
= Lh(u).
(2.45)
Convergence an relative entropy
41
( 3 ) Considering the action on the generating function, for all u
(2.46) (2.47) (2.48)
(2.49)
and comparing coefficients of t , the result follows.
0
This allows us to prove Lemma 2.5.
Proof. By subtracting the linear part from g, we are concerned with the span of ( H 2 , H s ?H 4 , . . .}. On this span, Lemma 2.6 shows that the maximum eigenvalue of Lhl is 1/4. = L H - I . Hence Now if H p ( u ) = p ( f i u ) , then M = H M and LM = ZHH-lg = and hence the maximum eigenvalue of L M is 0 also 1/4. This implies that Equation (2.22) holds with K = 2.
m,
2.1.4
Results of Brown and Barron
The other main result of [Brown, 19821 is a lower bound on the density of perturbed variables.
Lemma 2.7 There exists a constant Er > 0 such that for any random variable X with mean zero and variance 1, the sum Y, = X Z ( T )(where Z(') is a normal N ( 0 , r ) independent of X ) has density f bounded below by c r 4 r / 2 .
+
Information Theory and the Central Limit Theorem
42
Proof. Since U has variance 1, by Chebyshev's inequality, P(lUl < 2) 2 3/4. Hence, if F u is the distribution function of U , for any s E R
f ( s )=
(2.50)
L
(2.51)
2 2 Lemmas 2.5 and 2.7 can be combined t o give Equation (4.2) of [Brown,
19821: Proposition 2.1 Given IID random variables X I and X 2 , define Y, = X,+Z,"' (where Z,(.) is N ( 0 ,r ) and zndependent of X , ) wzth score function p ( x ) . If H,(z, 1-12> are the Hermite polynomzals spannzng L2(45T/2(x)dx), and p ( x ) = C ,a z H z ( xr/2) , then there exists a constant tTsuch that
This is the sandwich inequality referred t o above, and shows that the nonlinear part of the score function tends to zero, indicating that the variable must converge t o a Gaussian. Using control over the moments, it must converge weakly t o N(O,1 T ) , for all r . By letting T tend to zero, one deduces weak convergence of the original random variables, using Equation (1.137). Note that Proposition 2.1 requires X I and X 2 to be independent and identically distributed. However, our Proposition 2.2 gives a version of this result for arbitrary independent XI and X 2 . Proposition 3.3 extends the result to arbitrary independent n-dimensional vectors. Barron, in [Barron, 19861, uses Brown's work as a starting point t o prove convergence in relative entropy distance. His principal theorem is as follows:
+
Theorem 2.1 Let 4 be the N ( 0 , l ) density. Given IID random variables X l , X z , . . . with densities and variance n2, let gn represent the density of U, = Xi) The relative entropy converges to zero:
(c:=l /m.
(2.55)
Convergence in relative entropy
43
if and only if D(gnI14)is finite for some n.
Proof.
Barron uses Brown’s Proposition 2.1 as a starting point, using
a uniform integrability argument t o show that the Fisher information converges t o l / ( l + ~ Convergence ). in relative entropy follows using de Bruijn’s
identity (Theorem C . l ) and a monotone convergence argument.
0
We will consider how this proof can be both simplified and generalised, t o avoid the technical subleties of uniform integrability arguments. Further, we shall also consider the issue of establishing rates of convergence.
2.2
Generalised bounds on projection eigenvalues Projection of functions in L2
2.2.1
Using the theory of projections and Poincari: inequalities, we can provide a generalisation of Brown’s inequality, Lemma 2.5. Brown’s proof of convergence in Fisher information uses the bounds implied by Lemma 2.7, and hence is only valid for random variables perturbed by the addition of a normal. The proof here, first described in [Johnson and Barron, 20031, works for any independent random variables.
Proposition 2.2 Consider independent random variables Y1 Yz and a function f where IEf (Y1+ Yz) = 0 . There exists a constant p such that f o r any p E [0,I]:
E (f (YI+ Y2)- 91(Yl)- g2(Yz))2
(2.56)
2 (T1 (PE (d(Y1) - P I 2 + (1 - P)E (d(YZ) -d )
I
where
7=
(1 -
EY, f (Yl + v). Proof.
P)J(Yl) + PJ(Yz), and gi(u) = IEy,f(u
(2.57)
+ Yz),g2(v)
=
We shall consider functions
+ YZ)- g1(u)
r1(u) = EY2 [(f(u
).(z.
gz(Yz)) P2(YZ)l = EY, [(f(Yl+ ). - 91(Yl) - gz(w)) PI(Y1)I , -
1
(2.58) (2.59)
and show that we can control their norms. Indeed, by Cauchy-Schwarz applied to (2.58), for any u:
r f ( u )I = EY,
+
( f ( u yz) - gi(u)- gz(yz))2~~:(~z) (2.60)
(f(. + Y2)- 91(.)
-
92(Y2))2 J(Y2)I
(2.61)
Information Theory and the Central Limit Theorem
44
so taking expectations over
Y1,we deduce that
ET?(Yl) I E (f(Y1 + Yz)- Sl(Y1)- g2(Yz))2 J(yz).
(2.62)
Similarly,
Further, we can explicitly identify a relationship between ,rz and 91, gz, using the Stein identity Eh(Yz)pz(Yz) = -Eh’(Yz), with ~ ( Y z = )f(u
+
Yz)- g1(.)
-
gz(Y2):
+
By definition ErI(Y1)= 0, so define p = Egh(Y2)= Ef’(Y1 Yz).An interchange of differentiation and expectation (justified by dominated convergence) means that we can rewrite this as
Thus substituting Equation (2.65) in Equation (2.62) we deduce that
Using the similar expression for
T ~ ( u= )
-(gh(w)
-
p ) , we deduce that
Finally, adding p times Equation (2.66) to (1 - p) times Equation (2.67), 0 we deduce the result. Hence we see that if the function of the sum f(Y1+Yz)is close t o the sum of the functions g(Y1)+g(Yz), then g has a derivative that is close to constant. Now, we expect that this means that g itself is close to linear, which we can formally establish with the use of Poincark constants (see Appendix B). 2.2.2
Restricted Poincare‘ constants
In the spirit of,the Poincare constant defined in Definition B.1, we introduce the idea of a ‘restricted Poincark constant’, where we maximise over a smaller set of functions:
Convergence in relative entropv
Definition 2.2 constant R;:
45
Given a random variable Y , define the restricted Poincar6
(2.68) where H;(Y) is the space of absolutely continuous functions g such that Var g(Y)> 0, Eg(Y)= 0 and Eg2(Y)< co,and also Eg’(Y)= 0.
Lemma 2.8
The following facts are immediate:
(1) For all Y with mean 0 and variance 1: (EY4- 1)/4 5 R; 5 R y . (2) For Z N ( 0 , 0 2 ) ,R; = 0’12, with g(x) = x2 - o2 achieving this.
-
f.
PTOO
(1) The first bound follows by considering g(x) = x 2 - 1, the second since we optimise over a smaller set of functions. (2) By expanding g in the basis of Hermite polynomials. 0
If Y1,Y2 have finite restricted Poincar6 constants RF, R; then we can extend Lemma 2.5 from the case of normal Y1,Yz to more general distributions, providing an explicit exponential rate of convergence of Fisher information. Proposition 2.3 Consider Y1,Y2 IID with Fisher information J and restricted Poincare‘ constant R ” . For any function f (with IEf (Y1 Y2) = 0) there exists p such that
+
1
E (f (Yl + Y2) - Lf (Yl)- Lf (y2))22 JR*IE(Lf (Yl)- pYd2, (2.69) and hence by Lemma 2.4 (2.70)
Remark 2.1 (1) This bound is scale-invariant, since so is J R * , as J ( a X ) R : ,
=
(J(X)/a2)(a2R>). (2) Since for Y1,E N(0,cr2),R* = 0 2 / 2 , J = 1/o2, we recover (2.33) and hence Brown’s Lemma 2.5 i n the normal case. (3) For X discrete-valued, X 2, has a finite Poincare‘ constant, and so writing S, = (CyZlX,)/-, this calculation of an explicit rate of convergence of J,,(S, + Z T ) holds. Via L e m m a E.1 we know that S, Z, converges weakly for any r and hence S, converges weakly to the standard normal. N
+
+
Information Theory and the Central Limit Theorem
46
Convergence of restricted Poincare‘ constants
2.2.3
We obtain Theorem 2.2, the asymptotic result that Fisher information halves as sample size doubles, using a careful analysis of restricted Poincar6 constants, showing that they tend to 112.
Lemma 2.9 Given Yl,Y2 IID with finite Poincare‘ constant R and restricted Poincare‘ constant R*, write Z for (Y1 Y 2 ) / d .
+
RZ
-
112 5
R* - 112 2
+ J2RJ,to.
(2.71)
Consider a test function f such that Ef(Y1 +Y2) = 0 and E f ’ ( Y i + Y2) = 0 , and define g ( x ) = Ef(x Y2),so that g’(x) = Ef’(z Y2),and hence Eg(Y1) = Eg’(Y1) = 0. Now by our projection inequality, Proposition 2.2:
Proof.
+
2Eg’(Y1)2 5 Ef’(Y1
+
+ y2)2.
(2.72)
By the Stein identity, we can expand:
Ef (x
+
Y2)2=
E(f(z
+ Y2)
-
g(x) - s’(x)Y2)2
+ g 2 ( x )+ g’(x)2
+2g’(x)E(p(Y2)+ Y 2 ) f ( z+ Y2)
(2.73)
+
+
By definition, E ( f ( x Y2)- g ( z )- g ’ ( ~ ) Y 2 )I~ R*E( f’(z Y2)- g ’ ( x ) ) 2= R*(IEf’(x+ Y2)2- E g ’ ( ~ c ) ~Writing ). E ( Z ) for the last term,
Ef(z
+ Y z )5~R*Ef’(x + Y z )+~g 2 ( x )+ ( 1
-
R*)g’(x)2+ E(Z)
Now taking expectations of Equation (2.74) with respect to that
Ef(Yi
Y1
(2.74)
we deduce
+ Y2)’ 5 R * E f ’ ( Y i + Yz)’ + g 2 ( Y i ) + ( 1 R * ) g ’ ( y ~+)f(Y1) ~ (2.75) (2.76) 5 R * E f ’ ( Y l + Y2)2+ g’(Yl)2+ E(Y1) (2.77) 5 (R*+ 1 / 2 ) E f ’ ( Y 1 + Yz)’ + ~ ( y i ) , -
where the last line follows by Equation (2.72). By Cauchy-Schwarz, so, that E ( Y ~5) we deduce that ~ ( x )5 2 g ’ ( ~ ) d E f ( x + & ) ~ d m 2 J m J E f ( y l Y 2 ) 2 J J , t o I Ef’(Y1 + y 2 ) 2 d m so , the result follows on rescaling. 0
+
Using this we can prove:
Given X I ,X 2 , . . . IID with finite variance d,define the Theorem 2.2 normalised sum Un = (Cy’l Xi)/@. If Xi has finite Fisher information
47
Convergence in relative entropy
I and Poincare' constant R , then there exists a constant C = C ( I ,R) such that C (2.78) Jst(Un) 5 -, for all n. n
Proof. Write €2; for the restricted Poincark constant of s k . First, since R is finite, then by Proposition 2.3, J s t ( S n ) converges to zero with an m < CQ. Now, summing Lemma 2.9: exponential rate, so that En d
5
1 " (Rk+l - l / 2 ) I 2 (RE - 1/21 n= 1 n=l
+
E5 d n=l
w ,
(2.79)
and hence C(R:
-
1/2) < 00.
(2.80)
n
Now, Proposition 2.3 and Equation (2.80) implies convergence at the correct rate along the powers-of-2 subsequence:
Hence J s t ( S k ) 5 C'/2k, since log,z 5 z - 1, so C1oge(2R:J) I C2R:J 1 < 00. We can fill in the gaps using subadditivity: Jst(un+m) = Jst
and for any N , we can write N
)I ( Jst
= 2k
+
/ X U n ) n+m
-
-
n t m n
-J s t ( U n ) ,
(2.82) m, where m 5 2 k , so N/2k 5 2. (2.83)
0 2.3 2.3.1
Rates of convergence
Proof of O ( l / n ) rate of convergence
In fact, using techniques described in more detail in [Johnson and Barron, 20031, we can obtain a more explicit bound on the rate of convergence of Fisher information, which leads to the required proof of the rate of convergence of relative entropy. Specifically, the problem with Theorem 2.2 from
48
Information Theory and the Central L i m i t Theorem
our point of view is that whilst it gives convergence of Fisher information distance at the rate Cln, the constant C depends on I and R in an obscure way. Our strategy would be t o integrate the bound on Fisher information t o give a bound on relative entropy, that is, since (2.84) by the de Bruijn identity, Theorem C . l . If we could bound (2.85) for C ( X ) a universal constant depending on X only, not t , then this would imply that
as we would hope. However, the bound on Fisher information that Theorem 2.2 gives a constant C = C ( X , t ) also dependent on t (due to a factor of I in the denominator), and which behaves very badly for t close t o zero. This problem is overcome in the paper [Johnson and Barron, 20031 and also in [Ball et al., 20031. In [Johnson and Barron, 20031, Proposition 2.2 is improved in two ways. Firstly, Proposition 2.2 works by analysing the change in Fisher information on summing two variables. In [Johnson and Barron, 20031, by taking successive projections, it becomes better to compare the change in Fisher information on summing n variables, where n may be very large (see Proposition 3.1 for a generalisation to non-identical variables). Secondly, a careful analysis of the geometry of the projection space allows us to lose the troublesome factor of J in the denominator of (2.70) and deduce Theorem 1.5 of [Johnson and Barron, 20031, which states that:
Theorem 2.3 Given X l , X z , . . . IID and with finite variance g 2 , define the normalised sum Un = (C,"=, Xi)/@. If X , have finite restricted Poincare' constant R* then
for all n Using this we can integrate up using de Bruijn's identity t o obtain the second part of Theorem 1.5 of [Johnson and Barron, 20031 which states that:
Convergence zn relatzve entropy
49
Theorem 2.4 Given X I ,X 2 , . . . IID and with finite variance u 2 , define the normalised sum U, = (CZ,Xi)/@. If Xi have finite Poincare' constant R, then writing D(Un) for D(Unl14): 2R
D(un)
2R
D ( X ) 5 -D(x) + (2R n - l)a2 nu2
for all n.
(2.88)
Recent work [Ball et al., 20031 has also considered the rate of convergence of these quantities. Their paper obtains similar results, but by a very different method, involving transportation costs and a variational characterisation of Fisher information. Whilst Theorems 2.3 and 2.4 appear excellent, in the sense that they give a rate of the right order in n , the conditions imposed are stronger than may well be necessary. Specifically, requiring the finiteness of the Poincari: constant restricts us t o too narrow a class of random variables. That is, the Poincari: constant of X being finite implies that the random variable X has moments of every order (see Lemma B . l ) . However, results such as the Berry-Esseen theorem 2.6 and Lemma 2.11 suggest that we may only require finiteness of the 4th moment t o obtain an O(l / n ) rate of convergence. It is therefore natural to wonder what kind of result can be proved in the case where not all moments exist. Using a truncation argument, Theorem 1.7 of [Johnson and Barron, 20031 shows that:
Theorem 2.5 Given X I ,X 2 , . . . IID with finite variance u 2 , define the If Jst(Um) is finite for some m normalised sum U , = (Cr=lX i ) / @ . then (2.89)
Proof. The argument works by splitting the real line into two parts, an interval [-B,, B,] and the rest, and considering the contribution to Fisher information on these two parts separately. On the interval [-B,, B,], the variable has a finite Poincark constant, since the density is bounded away from zero. On the rest of the real line, a uniform integrability argument controls the contribution t o the Fisher information from the tails. On 0 letting B, tend to infinity sufficiently slowly, the result follows.
50
2.3.2
Information Theo6y and the Central Limit Theorem
Comparison with other forms of convergence
Having obtained an O ( l / n ) rate of convergence in relative entropy and Fisher information in Theorems 2.3 and 2.4, we will consider the best possible rate of Convergence. Since we can regard the I'(n,O) distribution as the sum of n exponentials (or indeed the sum of m independent I'(n/m,O)distributions), the behaviour of the r ( n , 0 ) distribution with n gives an indication of the kind of behaviour we might hope for. Firstly, Example 1.6 shows that
2 Jst(r(n,q) = n-2'
(2.90)
Further, we can deduce that (see Lemma A.3 of Appendix A)
(2.91) These two facts lead us to hope that a rate of convergence of O ( l / n ) can be found for both the standardised Fisher information J,t and the relative entropy distance D of the sum of n independent random variables. The other factor suggesting that a O ( l / n ) rate of convergence is the best we can hope for is a comparison with more standard forms of convergence. Under weak conditions, a O ( l / f i ) rate of convergence can be achieved for weak convergence and uniform convergence of densities. For example the Berry-Esseen theorem (see for example Theorem 5.5 of [Petrov, 19951) implies that:
Theorem 2.6 For X, independent identically distributed random variables with m e a n zero, variance cr2 and finite absolute 3rd m o m e n t IEIXI3 the density gn of Un = ( X I . . . X , ) / m satisfies
+
(2.92)
for some absolute constant A. This links to the conjectured O(l / n )rate of convergence for J,t and D , since Lemmas E.l and E.2 imply that if X is a random variable with density f , and 4 is a standard normal, then:
(2.93) (2.94)
Convergence i n relative entropy
2.3.3
51
Extending the Cram&-Rao lower bound
It turns out that under moment conditions similar to those of this classical Berry-Esseen theorem, we can give a lower bound on the possible rate of convergence. In fact, by extending the techniques that prove the CramkrRao lower bound, Lemma 1.19, we can deduce a whole family of lower bounds: Lemma 2.10 For r a n d o m variables U and V with densities f and g respectively, for any test f u n c t i o n k :
(2.95)
Proof. Just as with the Cramkr-Rao lower bound, we use the positivity of a perfect square. That is, (2.96)
+a2 =
1
(2.97)
f(x)k(x)2dx
J(f11g) - 2a (IEk’(U)
+ IEk(U)p,(U)) + a21Ek(U)2.
Choosing the optimal value of a , we obtain the Lemma.
(2.98)
0
Example 2.1 For example, for a random variable with mean zero and variance u 2 , taking 2 N ( 0 ,a’) we obtain N
a strengthening of the Cram&-Rao lower bound. This allows us to give a lower bound on what rate of convergence is possible. We have a free choice of the function k . We can obtain equality by picking k = p f , which obviously gives the tightest possible bounds. However, it is preferable t o pick k in such a way that we can calculate Ek(U,) easily, for U, the normalised sum of variables. The easiest way to do this is clearly to take k ( x ) as a polynomial in x,since we then only need to keep track of the behaviour of the moments on convolution.
52
Information Theory and the Central Limit Theorem
Lemma 2.11 Given X i a collection of IID random variables with variance u2 and EX? finite, then for U, = ( X l + . . . X,) /fi,
+
where s is the skewness, m 3 ( X ) / a 3 (writing m r ( X ) for the centred r t h moment of X ) .
Proof. Without loss of generality, assume E X = 0 so that EU = 0, Var U = a2. Taking k ( u ) = u2 - g 2 ,we obtain that E k ' ( U ) - U k ( U ) / a 2= -EU3/a2 = - m 3 ( U ) / a 2 , and IE/c(U)~ = m 4 ( U ) - a 4 . Then Equation (2.99) implies that (2.101) Now since m3(Un) = m 3 ( X ) / & and m 4 ( U n ) = m 4 ( X ) / n we can rewrite this lower bound as
+ 3a4(n
-
l)/n
(2.102) where y is the excess kurtosis m4(X)/a4 - 3 of X . Hence, if y is finite, the result follows. 0 This implies that the best rate of convergence we can expect for is O ( l / n ) , unless EX3 = 0. This is an example of a more general result, where we match up the first r moments and require 2r moments to be finite.
Lemma 2.12 Given X i a collection of IID random variables with EX:' finite and EX: = EZs (that is the moments of X match those of some normal 2 ) for s = 0 , . . . r . For U , = ( X I . . . X,) I&:
+ +
(2.103)
Proof. Again, without loss assume EX = 0 and take k ( u ) to be the r t h Hermite polynomial H T ( u ) . Now, for this k, k'(u) - u k ( u ) / 0 2 = -HT+1(u)/c2. Hence:
( 2.104)
Convergence in relatzve entropy
53
since HT+l(x)has the coefficient of x'+l equal t o 1, and since all lower moments of U and Z agree. Since the first T moments of X and Z agree, by induction we can show that (2.107) We can combine (2.106) and (2.107) to obtain that
As in Lemma 2.11, we also need t o control IEk(U,)'. We appeal to Theorem 1 of [von Bahr, 19651, which gives that for any t, if IEJXltis finite then t-2
(2.109) for constants c 3 , t ,which depend only on X , not on n. Further, t even. Hence,
~
1 =, 0 for ~
(2.110) for any even t. Thus, when we expand Elc(U,)', for
mial of degree
T,
lc the Hermite polyno-
we will obtain: (2.111)
for some constant D ( r ) . Now, further, we know from (2.37) that Ek(2)' ~ ! ( g so ~ )that ~ , Ek(U,)' 5 r!g2r D ( r ) / n . Then Equation (2.99) and (2.108) imply that
=
+
(2.112)
so letting n. 403 we deduce the result.
0
We can consider how sharp the bounds given by Lemma 2.11 and Theorem 2.4 are, in the case of a r ( n ,1) variable. Firstly, a r ( t , 1) variable U has centred moments m z ( U ) = t , r n s ( U ) = 2t, m 4 ( U ) = 3t2 6t. Hence the lower bound from Equation (2.101) becomes
+
(2.113)
54
Information Theory and the Central Limit Theorem
Recall that by (2.90), Jst(r(nI0))
2 n-2'
(2.114)
=-
so the lower bound is not just asymptotically of the right order, but also has the correct constant. On the other hand, we can provide an upper bound, by considering the r ( n , 1) variable as the sum of m independent r(t,1) variables, for n = mt. Taking H ( z ) = exp(z/(l t ) ) ( l - z/(1 + t ) ) in Theorem 2.1 of [Klaassen, 19851, we deduce that a r(t,1) random variable has R* 5 (1+ t ) 2 / t .Thus Theorem 2.4 implies
+
'(A)
2(1
+ t)2
(A) +
2(1+ t ) 2 2 ( l + t ) 2 ( m- l ) t 2 = 2 4t t2 nt' (2.115) Now, the larger t is, the tighter this bound is. For example, taking t = 5 gives (for n 2 5) Jst(r(n'
+
+ +
(2.116) a bound which is of the right order in n , though not with the best constant.
Chapter 3
Non-Identical Variables and Random Vectors
Summary In this chapter we show how our methods extend to the case of non-identical variables under a Lindeberg-like condition and t o the case of random vectors. Many of the same techniques can be applied here, since the type of projection inequalities previously discussed will also hold here, and we can define higher dimensional versions of Poincari: constants.
3.1 3.1.1
Non-identical random variables Previous results
Whilst papers such as those of [Shimizu, 19751 and [Brown, 19821 have considered the problem of Fisher information convergence in the Central Limit Theorem regime, they have only been concerned with the case of independent identically distributed (IID) variables. Linnik, in two papers [Linnik, 19591 and [Linnik, 19601, uses entropytheoretic methods in the case of non-identical (though still independent) variables. However, he only proves weak convergence, rather than true convergence in relative entropy. Consider a series of independent real-valued random variables XI, X 2 , . . ,, with zero mean and finite variances a:, a;, . . .. Define the variance of their sum 21, = Cr=lg,"and
u,= C;=l xi
6 .
(3.1)
Writing II for the indicator function, the following condition is standard in this context. 55
56
Information Theory and the Central Lzmit Theorem
Condition 1 [Lindeberg] Defining: .
n
(3.3)
t h e Lindeberg condition holds i f f o r a n y
E
> 0 t h e limn+m &(n)= 0.
If the Lindeberg condition holds, then (see for example [Petrov, 19951, pages 124-5) the following condition holds: Condition 2
[Individual Smallness] maxl<,sn ut/wn + 0.
This means that w, must diverge. Furthermore, roughly speaking this allows ut t o be a polynomial in i , but not an exponential. The Lindeberg condition allows us to eliminate two troublesome cases. Firstly, if the sum of variances doesn't diverge, we could have a case where X , became deterministic, so the normalised sum need not tend to a Gaussian. Secondly, if we add random variables which are a long way from Gaussian and have very large variances then the sum could be pulled away from the Gaussian. Rotar discusses in [Rotar, 19821 sufficient conditions to prove convergence when the Individual Smallness condition does not hold. The Lindeberg-Feller theorem (see Theorem 4.7 of [Petrov, 19951) states that:
Theorem 3.1 For X, a collectzon of independent r a n d o m variables, t h e normalised s u m U , = ( X I ... X,)/Jvn converges weakly to a norm a l N ( 0 , l ) and Condition 2 holds if and only i f the Lindeberg condition, C o n d i t i o n 1 holds.
+
+
Whilst this theorem provides necessary and sufficient conditions for weak convergence, we will be concerned with proving convergence in KullbackLeibler distance, which will require separate conditions. In particular, in the rest of this chapter, we will require that the random variables have densities, and write g, for the density of the normalised sum U,. In [Linnik, 19591, Linnik uses entropy-theoretic methods t o prove convergence to the Gaussian. Page 601 of [Rknyi, 19701, suggests that Linnik proves that Condition 1 alone implies convergence of D(gnI14) 0. However this cannot be right - Condition l is not strong enough alone t o imply convergence in relative entropy. This is clear from Barron's Example E . l (see Appendix) which shows that D(gnI14) can always be infinite in the --f
Non-identical variables and random vectors
57
IID case (where Lindeberg Condition 1 holds as a consequence of finite variance). In fact, as both [Barron, 19861 and later Gnedenko and Korolev (page 212 of [Gnedenko and Korolev, 19961) comment, Linnik does not actually prove convergence in relative entropy. First he truncates the random variables, and then smooths them by the addition of small normal distribution random variables. Thus [Linnik, 19591 only establishes weak convergence of the original non-transformed variables. Nonetheless, certain parts of Linnik’s argument are still noteworthy. Firstly, rather than the original Lindeberg condition (which requires that for any E , C, there exists ~ o ( EC), such that n 2 ~ o ( EC),, h,(n) 5 0, in the paper [Linnik, 19591 he introduces: Condition 3 Given E > 0 and C1 > 0 (for example C1 = d ) ,X j meets the Individual Lindeberg condition (with these 6 ,
Equation (3.5) from [Linnik, 19591 states that if Xm+l satisfies the Individual Lindeberg condition then there exists a uniform constant B such that
where a , = of,,/um and E comes from the Individual Lindeberg condition. This indicates a phenomenon common t o many entropy-theoretic arguments. Either the left hand side is large, in which case Urn+, is much closer to the normal than Urn,so we move much closer t o the normal distribution in taking the convolution, or it is small, in which case the Fisher information of J ( U m ) is close to the Cramkr-Rao lower bound. That is, heuristically speaking, either we get closer t o the normal, or we were already close to it. The paper [Johnson, 20001 extended the papers [Brown, 19821 and [Barron, 19861 to the non-identical case. In this section, we show how these results can be improved by using the techniques described in Chapter 2. 3.1.2
Improved projection inequalities
We can consider successive projections using an argument similar t o that of the previous chapter.
Information Theory and the Central Limit Theorem
58
Proposition 3.1 Consider independent random variables X i , with mean 0 and Fisher information Ji. Given any function f with Ef ( X I . .+Xn) = 0 , consider the projection functions fi(z) = Ef ( X I . . . Xi-1 z Xi+l+ . . . + X,). Now there exists a constant p such that:
+. + +
+
(3.6)
+
+ + + +
Proof. Defining successive projections g z ( z ) = Ef(z Xz+l . . . X n ) , the best approximation t o gz as a sum of functions of ( X I . . . X,-1) and X , is gz-l f z . The squared distance between successive projections is
+
r, = IE (9,
+ . . . + x,)- g z - l
(x1
(XI
+ . .. +
~
~
-
1 f) , ( ~ ~ ) ) ~ , (3.7)
and we bound r, from below in Lemma 3.1 to obtain that for z
2 2:
and indeed for i = 1, where both sides equal zero. Now, define
Since s , = Eg,2
Ef:,
-
=I
then
E 2~ , - ~ g ,2- , - IEf;
=
r,.
(3.11)
(A picture of these projections appears in Figure 3.1 - we project from the set of functions of the sum X I + . . . + X , into (a) the sum of functions of X I . . . X,-l and X , (b) the sum of functions of X , ) . Hence summing the telescoping sum in Equation (3.11), s, = Cr=lr, ,
+
and by Lemma 3.1 below we deduce that
(3.12)
Non-zdentical variables and random vectors
Fig. 3.1
59
Role of projections
Now, simply by reversing the order of the X , , we can obtain a complementary bound that:
(3.13) Adding Equation (3.12) and (3.13) together the result follows.
0
We now prove the lower bound on ri required in the previous proof. Lemma 3.1 Consider independent random variables X i , with mean 0 and Fisher information J,. Given any function f with IEf ( X I . . . X,) = 0 , consider the projection functions f i ( z ) = I E f ( X I . . . Xi-1 z Xi+l . . . X n ) and g i ( z ) = IEf ( z Xa+l . . . X,). Defining
+
+
+ + Xi)
ri = E(gi (XI . . .
-
+
+ + + +
+ + gi-1 ( X I + . . . + X i - 1 ) - f ( X i ) ) 2,
+
(3.14)
Information Theory and the Central Limit Theorem
60
then (3.15)
Given constants u1, u2,. . . u,, we evaluate the function
Proof.
+ . ’ . + XI-1 + 2) - gz-1 (Xl + . . . + x,-1)- fi(X,)) (3.16) x (UlPl(X1) + . ’ . + ~,-lP,-l(Xt-l))]
P ( 2 ) = E [(gz (Xl
in two different ways. Firstly, by the Stein identity: (3.17) where p
=
Ef,’-, = Ef’, so that
Secondly, we apply Cauchy-Schwarz to p ( z ) to obtain p ( z )2 5
E (gz (Xl + . . . + xz-1 + 2) - gz-1 (Xl + . . . + X%-l)- f z ( 2 ) ) 2 XE(.ulPl(Xl)
+“‘+~,-lp,~l(~,-l))2.
(3.19)
Since the variables are independent, if z # 3 , then lEp,(X,)p,(X,) = % ~ ~ ( X ~ ) ~ p ,= ( x 0, , ) so that E(ulpl(X1) . . . U ~ - ~ ~ , - I ( X ~ = -I))~ u : J ( X , ) . On taking expected values, we deduce that
c:I:
+ +
(3.20) On combining Equations (3.18) and (3.20) we see that
Now, we can exploit the free choice of the u z ,and substitute in the optimal values. We know that C u: 5, takes its extreme value for a fixed C u, on choosing u3 proportional t o l/JJ. Taking this choice of the u J , we deduce the result. 0
Non-identical variables and random vectors
61
Assuming the variables have finite (restricted) Poincar6 constants, we can simply replace the derivative f,' by f i itself, at the price of a factor of Rf.
Lemma 3.2 Consider independent random variables X , , with means 0 , restricted Poincare' constants Rf and Fisher information Ji. Given any function f with Ef ( X I + . . . + X,) = 0 , consider the projection functions f i ( z ) = E f ( X l + . . . X,-1+ z + X i + ~ + .. . + X n ) . Now there exists a constant p such that:
(3.22) I n the case where X , are identically distributed, this reduces to
Now, we can simply apply Lemma 3.2 to obtain a bound on the growth of Fisher information on convolution:
Proposition 3.2 Consider independent random variables X,,with means 0 , variances a?, restricted Poincare' constants R: and Fisher information J, . Then:
(3.24)
where c, = (C,,, l/JJ)/(2R:), and a, = cP/(Cr=l UP) = aP/v,. In the IID case, thzs reduces to c, = ( n - 1 ) / ( 2 R *J ) , and a , = l / n .
Information Theory and the Central Limit Theorem
62
Proof. Writing projections of p:
for the score function of
c n
CY?(J(Xi)- l / U ? ) - ( J ( X 1
X1
+ ...+ Xn) -
+ . . . + X,,
and
fi
for the
(3.25)
l/?Jn)
i=l
;ij(Xl
+ . . . + X,)
-
n
X1+ . . . X , un
-
x N i ( p i ( X i ) - Xi/u:) i=l
(3.27)
n
n
i=l
i=l
(3.29) (3.30) (3.31) by Lemma 3.2, and since ax2
+ by2 2 (ab/(a+ b ) ) ( z
-
y)’.
0
Whilst Proposition 3.2 represents a good bound, for the purposes of calculation it may well be easiest to eliminate some terms. For example, we can impose an “average boundedness” condition on the Jstand R.
Condition 4 such that
Using the notation above, there exist constants C and D
(3.32) (3.33)
Remark 3.1 Consider the special case where for all i, there exists a J with J,,(X,) 5 J (for example where X , N U2 + ZTg; for 2, N N ( 0 , t ) ) . Then, since l / J z 2 o f / ( J l ) ,Equation (3.32) holds with C = 1 / ( J + 1 ) . Secondly, Equation (3.33) reduces to requiring the existence of an E such
+
Non-identical variables and random vectors
63
that (3.34)
Theorem 3.2 Consider independent random variables X i , with means 0 , variances a;, restricted Poincare‘ constants Rf and Fisher information J,. Then under the Lindeberg Condition 1 and Condition 4: lim J,t
n-m
) =o.
(3.35)
Proof. Recall that the Lindeberg condition implies that the sum of variances diverges. We can exploit the bound that: (3.36)
since 2Rf 2 deduce that
CT:
>_ l / J i . Hence substituting this in Proposition 3.2, we
(3.37)
(3.39)
and the result follows by the divergence of the sum of variances.
0
In terms of the relationship between our Condition 4 and the more standard Lindeberg condition, Condition 1, note that for any variable Y , Lemma 2.8 gives that IEY4 5 4(1 a$R;), and hence by Chebyshev we deduce that for any 6 > 0
+
(3.40)
Hence, if we have control over the R: with (3.34) holding and u, diverging then the Lindeberg condition 1 follows since:
Information Theory and the Central Limit Theorem
64
Remark 3.2 Note that [Johnson, ZOOO], like [Linnik, 19591 produces bounds that rely on unaform control of the variables, rather than control of their average, as for Condition 4 and the Lindeberg condition, Condition 1.
3.2
Random vectors
3.2.1
Definitions
Since the definition of differential entropy and relative entropy given by Definitions 1.4 and 1.5 do not depend on the numerical values of the random variables, but rather on their probability densities, we can repeat them for vector-valued variables t o obtain: Definition 3.1 The differential entropy of a continous vector-valued random variable Y with density f is:
Again, OlogO is taken as 0. The Kullback-Leibler distance from density p to density q is
(3.43) Similarly, we can define the Fisher information matrix as follows. Given a function p , write Vp for the gradient vector ( d p l d z l , . . . , dp/az,) and V2p for the Hessian matrix ( V 2 p ) t 3 = d2p/dz&j. Definition 3.2 For a random vector U with differentiable density f and covariance matrix C > 0, define the score vector-function p u ( x ) = V log f ( x ) = V f ( x ) /f ( x ) . Define the Fisher information matrix J and its standardised version J,t by:
J ( U ) = GI ( P U ( u ) P U ( u ) T ) > J(U1lZ) = J(U) - c-1,
Jst(U)= C ( J ( U )- C - ' ) , where Z
-
N ( 0 ,C ) .
(3.44) (3.45) (3.46)
65
Non-identical variables and random vectors
Example 3.1 The multivariate Gaussian distribution U with covariance C has density const exp(xTC-lx/2), with
and so
J(U) = c(c-l)2= c-l.
(3.48)
A version of the Stein identity, Lemma 1.18, holds for vectors as well. Lemma 3.3 If pi = (a/axi)(logp(x))is the ith component of the score vector function then for a n y smooth function f well-behaved at infinity (3.49)
or
Since by the Stein identity, Lemma 3.3, IE(YP(Y)~)= - I , we know J(YJ]Z) = E(p(Y) + C-'Y)(p(Y) + C-lY)T is positive semi-definite. 3.2.2
Behaviour on convolution
We can provide the equivalent of Lemma 1.20, that shows that the score vectors of sums of independent random vectors are again given by conditional expectations (projections):
Lemma 3.4 If U , V are independent random vectors and W with score functions pu, pv and pw then
Proof. If U , V have densities p(u),q(v) then U density ~ ( w=) sp(u)q(w- u ) d u , so that
-(w) ddWra
=
J
8q ~ ( u ) - ( w - u)du = -
awi
S
=
U
+V
+ V has the convolution
~ ( u )89 -(w aui
-
u ) d u (3.52)
(3.53)
Information Theory and the Central Limit Theorem
66
and hence
=
[(P")i(U)lW = w] .
(3.55)
Similarly, we can produce an expression in terms of the score function of V. 0 Observe that
D(fll4c)= =
1 2
- l o g ( ( 2 ~ edet ) ~ C) - H ( f )
log e + -(tr(C-'B) 2
log e H ( 4 c ) - ~ ( f+ )T ( t r ( C - l B )
-
n).
-
n)(3.56)
(3.57)
Multidimensional Gaussian densities again obey a version of the heat equation, which means that an n-dimensional version of the de Bruijn identity holds, proved later as Theorem C.2. This considers X a random vector with density f and covariance matrix B , and fT the density of Y , = X ZC,, where Z c 7 is independent of X. Then:
+
Note that if B = C then (3.59)
3.2.3
Projection inequalities
We can produce a similar expression to Proposition 2.2 for random ndimensional vectors Y1, Y2. The proof uses a similar method.
Proposition 3.3 Given independent random vectors Y1,Y 2 and a function f, with Ef(Y1 Yz) = 0, there exists a constant p such that for any
+
0:
(f(Y1 + YZ) - g l ( y 1 ) - g 2 ( y 2 ) ) 2
(3.60)
Non-identical variables and random vectors
where 7 = PtrJ(Y1)
+ (1
-
67
P)trJ(YZ) and
9 1 ( 4 = EY,
[f(u + YZ)],
(3.62)
+
g2(v) = EY, [f(Y1 v)l .
(3.63)
Proof. As in Chapter 2, we define the two vector functions:
[(f(U + YZ) - 91(u) - gz(Y2)) PZ(Y2)I , r2(V) = E Y , [(f(yl+ v ) -g1(Y1) - ~ ~ ( v ) ) P ~ ( Y I ) ]
r1(u)
(3.64)
= EY,
(3.65)
1
which we expect to be small. Indeed, by Cauchy-Schwarz, for any u:
+
r ? ( i ~I) EY, ( f ( u Yz) - 91(u) - g2(Y2))’ EyztrpZpif(Yz),
(3.66)
so taking expectations over Y1, we deduce that
Er?(Yi) I E(f(Y1
+ Yz)
gi(Y1) - g2(Y2))2trJ(Yz).
(3.67)
I E (f(Y1 + Yz) - gi(Y1) - g ~ ( Y 2 )trJ(Y1). )~
(3.68)
-
Similarly, E$(Yz)
Further, we can explicitly identify a relationship between r l , 1-2 and gl, g2, using the Stein identity n ( u ) = - (EVf(u
+ Y z )- EVgZ(Y2)).
(3.69)
An interchange of differentiation and expectation (justified by dominated convergence) means that we can rewrite this as r1(u) = - (VL?l(U) - EVSZ(Y2)).
(3.70)
Using the similar expression for r2(v) = -(Vg2(v) - EVg1(Y1)), and adding p times Equation (3.67) to (1- 0)times Equation (3.68), we deduce the result. 0 We define the n-dimensional equivalent of the restricted Poincark constant of Definition 2.2:
Definition 3.3 Given a vector-valued random variable Y, define the restricted Poincark constant R;:
(3.71) where H;(Y) is the space of absolutely continuous functions g such that (as before) Var g(Y) > 0, Eg(Y) = 0 and Eg2(Y) < 03, and also EVg(Y) = 0.
68
Information Theory and the Central Limit Theorem
As in Chapter 2 we deduce that: Theorem 3.3 Given independent identically distributed random vectors Y1 and Y2 with covariance C and finite restricted Poincare' constant R*, for any positive definite matrix D :
where JD= t r ( D J ( Y ) ) and Z Proof.
N
N ( 0 ,C )
Notice that we can factor D = B T B , then t r ( D J ( Y ) ) = tl.(BTBP(Y)PT(Y)) = (Bp,B p ) ,
(3.73)
and our Proposition 3.3 allows us to consider the behaviour on convolution 0 of this function Bp. The proof goes through just as in Chapter 2. Hence if X, are independent identically distributed random vectors with finite Poincark constant R and finite Fisher information, then the standardised Fisher information of their normalised sum tends to zero. By the de Bruijn identity the relative entropy distance to the normal also tends to zero. We can combine the methods of Sections 3.1 and 3.2 t o give a proof of convergence for non-identically distributed random vectors.
Chapter 4
Dependent Random Variables
Summary Having dealt with the independent case, we show how similar ideas can prove convergence in relative entropy in the dependent case, under Rosenblatt-style mixing conditions. The key is to work with random variables perturbed by the addition of a normal random variable, giving us good control of the joint and marginal densities and hence the mixing coefficient. We strengthen results of Takano and of Carlen and Soffer to provide entropytheoretic, not weak convergence. It is important that such results hold, since the independent case is not physical enough to give a complete description of interesting problems.
4.1 4.1.1
Introduction and notation
Mixing coefficients
Whilst the results of Chapters 2 and 3 are encouraging, in that they show an interesting reinterpretation of the Central Limit Theorem, they suffer from a lack of physical applicability, due to the assumption of independence. That is, data from real-life experiments or models of physical systems will inevitably show dependence and correlations, and results of Central Limit Theorem type should be able t o reflect this. This has been achieved in the classical sense of weak convergence, under a wide variety of conditions and measures of dependence - see for example [Ibragimov, 19621. This chapter will show how analogous results will hold for convergence in Fisher information and relative entropy. The chapter is closely based on the paper [Johnson, 20011. We work in the spirit of [Brown, 19821 - we consider random variables perturbed by 69
Information Theory and the Central Limit Theorem
70
the addition of a normal variable. We exploit a lower bound on densities, and use uniform integrability. A naive view might be that it would be enough to control the covariances between random variables. Indeed, this is the case when the variables also have the FKG property (see for example [Newman, 19801 and [Johnson, 2003bl). However, in general we require more delicate control than just control of the correlations. We will express it through so-called mixing coefficients. Bradley, in [Bradley, 19861, discusses the properties and alternative definitions of different types of mixing coefficients. We borrow his notation and definitions here. First, we give alternative ways of measuring the amount of dependence between random variables.
Definition 4.1 Given two random variables S , T , define the following mixing coefficients:
a ( S , T )= SUP p ( ( S E A ) n (T E B ) ) - P(S E A)P(T E B)I ,
(4.1)
A,B
4(S,7')
= SUP
1P(T E B ) S E A ) - P(T E B)1
(44
A,B
IP((S E A ) n (T E B ) ) - P(S E A)P(T E B)I , A,B P(S E A ) p((SE A ) n (T E B ) ) - P(S E A)P(T E B)J . $ J ( S > T=)SUP P(S E A)P(T E B ) A,B =
sup
(4.3)
(4.4)
Next, we consider how these quantities decay along the array of random variables.
Fig. 4.1
Definition 4.2
Separation on the array of random variables
If Cg is the a-field generated by X,, X,+I,.. . , Xb (where
Dependent random variables
71
a or b can be infinite), then for each t , define:
s E Co_,,T 4(t) = sup { $ ( S , T ): s E CO_,,T
€
C?} ,
€
ct"},
$ ( t ) = SUP {$(SIT): S E C!,,T
E CF}.
a(t) = s u p { a ( S , T )
:
Define the process t o be
(I) a-mixing (or strong mixing) if a ( t ) -+ O as t 4 m. (2) +mixing (or uniform mixing) if &(t)-+ 0 as t + 03. (3) $-mixing if $ ( t ) + 0 as t 4 03. Remark 4.1 For a n y S and T , since the numerator of (4.1)) (4.3) and (4.4) is the same for each term, the $ ( S , T ) 2 4 ( S , T )2 a ( S , T ) ,so that $-mixing implies &mixing implies a-mixing. For Markov chains, &mixing is equivalent to the Doeblin condition. All m-dependent processes (that is, those where X , and X j are independent for li - j ( > m ) are a-mixing, as well as a n y strictly stationary, real aperiodic Harris chain (which includes every finite state irreducible aperiodic Markov chain), see Theorem 4.3i) of [Bradley, 19861.
Example 4.1 Consider a collection of independent random variables . . . , Vo,V1,Vz, . . ., and a finite sequence ao, a l , . . . an-l. Then defining n-1
j=O
the sequence ( X k ) is n-dependent, and hence a-mixing. Less obviously, Example 6.1 of [Bradley, 19861 shows that a-mixing holds even if the sequence ( a k ) is infinite, so long as it decays fast enough: Example 4.2 Consider a collection of independent random variables . . . , VO, V1, Vz,. . ., and a sequence ao, a1 . . . , where ak decay exponentially fast. Then the sequence ( X k ) defined by (4.9) j=O
is a-mixing, with a ( k ) also decaying at an exponential rate.
72
Information Theory and the Central Limit Theorem
Takano considers in two papers, [Takano, 19961 and [Takano, 19981, the entropy of convolutions of dependent random variables, though he imposes a strong &-mixing condition (see Definition 4.4). The paper [Carlen and Soffer, 19911 also uses entropy-theoretic methods in the dependent case, though the conditions which they impose are not transparent. Takano, in common with Carlen and Soffer, does not prove convergence in relative entropy of the full sequence of random variables, but rather convergence of the ‘rooms’ (in Bernstein’s terminology), equivalent to weak convergence of the original variables. Our conclusion is stronger. In a previous paper [Johnson, 2003b], we used similar techniques t o establish entropy-theoretic convergence for FKG systems, which whilst providing a natural physical model, restrict us t o the case of positive correlation. 4.1.2
Main results
We will consider a doubly infinite stationary collection of random variables . . . , X - 1 , X O ,XI,X2,. . ., with mean zero and finite variance. We write vn for Var (C:=,X,) and U, = (C:=,X,)/fi. We will consider perturbed random variables VAT’ = (C:=,X, Z:.’)/&L N U, Z ( T ) ,for Z:.) a sequence of N ( 0 ,r) independent of X , and each other. In general, Z(”) will denote a N ( 0 ,s) random variable. If the limit Cov(X0, X,)exists then we denote it by u. Our main theorems concerning strong mixing variables are as follows:
+
+
CE-,
Theorem 4.1 Consider a stationary collection of random variables Xi, with finite (2 + 6 ) t h m o m e n t . If C,”=, ~ ( j ) ’ / ( ~ + ’<) 00, then for any r > 0 (4.10)
Note that the condition on the a ( j ) implies that v,/n 4 u < 0;) (see Lemma 4.1). In the next theorem, we have to distinguish two cases, where v = 0 and where u > 0. For example, if Yj are IID, and Xj = Yj - Yj+lthen v, = 2 and so v = 0, and U, = (Y1 - Y,+l)/&L converges in probability to 0. However, since we make a normal perturbation, we know by Lemma 1.22 that
so the case v = 0 automatically works in Theorem 4.1.
Dependent random variables
73
We can provide a corresponding result for convergence in relative entropy, with some extra conditions:
Theorem 4.2 Consider a stationary collection of random variables X i , with finite (2 6 ) t h moment. Suppose that
+
Dfining
T h e n , writing g, for the density of
we require that for some N,
(cy=~ Xi)/&; (4.12)
Proof. Follows from Theorem 4.1 by a dominated convergence argument using de Bruijn’s identity, Theorem C . l . Note that convergence in relative entropy is a strong result and implies convergence in L1 and hence weak convergence of the original variables (see Appendix E for more details).
Remark 4.2 Convergence of Fisher information, Theorem 4.1, is actually implied by Ibragimov’s classical weak convergence result [Ibragimov, 19621. T h i s follows since the density of Vir’ (and its derivative) can be expressed as expectations of a continuous bounded function of U,. Theorem 1.3, based o n [Shimizu, 19751, discusses this technique which can only work f o r random variables perturbed by a normal. W e hope our method m a y be extended to the general case in the spirit of Chapters 2 and 3, since results such as Proposition 4.2 do not need the random variables t o be in this smoothed f o r m . I n any case, we feel there is independent interest in seeing why the normal distribution i s the limit of convolutions, as the score function becomes closer t o the linear case which characterises the Gaussian.
74
4.2
Information Theory and the Central Limit Theorem
Fisher information and convolution
Definition 4.3 For random variables X, Y with score functions p x , p y , for any p, we define j7 for the score function of f l X m Y and then:
+
A ( X l Y , P )= E ( f i P X ( X )
+r
n P Y ( Y ) - F(&X
+ (4.13)
Firstly, we provide a theorem which tells us how Fisher information changes on the addition of two random variables which are nearly independent. Theorem 4.3 Let S and T be random variables, such that max(Var S,Var T ) 5 K T . Define X = S Z$) and Y = T 2;' (for 2 : ' and Z$) normal N ( 0 , r ) independent of S , T and each other), with score functions px and p y . There exists a constant C = C ( K , T , E such ) that
+
PJ(X)+(l-P)J(Y)-J
+
(fix + -Y)+CQ(S,T)'/~-'
2 A(X,Y,P). (4.14)
+
If S,T have bounded kth moment, we can replace 1/3 by k / ( k 4). The proof requires some involved analysis, and is deferred to Section 4.3. In comparison Takano, in papers [Takano, 19961 and [Takano, 19981, produces bounds which depend on 64(S,T ) ,where: Definition 4.4 For random variables S,T with joint density p ~ , ~ ( s and marginal densities p s ( s ) and p ~ ( t )define , the 6, coefficient to be:
Remark 4.3 I n the case where S,T have a continuous joint density, it i s clear that Takano's condition is more restrictive, and lies between two more standard measures of dependence f r o m Definition 4.2
4 4 s , T ) F 64(S, T ) F S,(S, T ) = $ ( S ,T ) .
(4.16)
Another use of the smoothing of the variables allows us to control the mixing coefficients themselves:
+
+
Theorem 4.4 For S and T , define X = S Z$) and Y = T Z g ) , where max(Var S,Var T ) 5 K r . If Z has variance E , then there exists a function f K such that
4 x + 2,Y ) 5 a ( X ,Y )+ f K ( f ) ,
(4.17)
Dependent random variables
where
~ K ( E4 ) 0 as E
Proof.
4
75
0.
See Section 4.3.4.
0
To complete our analysis, we need lower bounds on the term A ( X ,Y ,p). For independent X , Y it equals zero exactly when px and p y are linear, and if it is small then px and p y are close to linear. Indeed, in [Johnson, 20001 we make two definitions: Definition 4.5 For a function $, define the class of random variables X with variance 0 : such that cqj = { X : EX211(IXI 2 R a x ) I a i $ ( R ) ) .
Further, define a semi-norm
11
(4.18)
on functions via
Combining results from previous papers we obtain: For S and T with max(Var S,Var T ) I K r , define X = S + Z g ) , Y = T + 2;’. For any $, 6 > 0 , there exists a function v = v + , ~ , , y ,with ~ , V ( E )40 as E + 0 , such that i f X , Y E C$, and /3 E ( 6 , l - 6 ) then
Proposition 4.1
Proof. We reproduce the proof of Lemma 3.1 of [Johnson and Suhov, 20011, which implies p(x, y) 2 (exp(-4K)/4)&/2(z)&p(y). This follows since by Chebyshev’s inequality 1(s2 t2 5 4 K r ) d F s , ~ ( ts ), 2 l / 2 , and since (z - s)’ 5 2x2 2s2:
+
+
(4.22)
Information Theory and the Central Limit Theorem
76
Hence writing h ( z ,y ) then:
= flpx
+~
P ( y ) - Y (v%
(x)
+m
y
)I
(4.25)
’ L
1
16
d)T/2(z)d)T/2(!/)h(z,
P(1 - P ) e x p ( - W
32
(IIPXII&
y)2dXd!/
+ IIPYII&)
I
(4’26) (4.27)
by Proposition 3.2 of [Johnson, 20001. The crucial result of [Johnson, 20001 implies that for fixed y!~, if the sequence X , E C+ have score functions p,, 0 then llpnlle + 0 implies that J,t(X,) + 0. We therefore concentrate on random processes such that the sums (XI X2 . . . X,) have uniformly decaying tails:
+
Lemma 4.1 If { X , } are stationary with EIX12f6 < and C,”=, ( ~ ( j ) ~< /03,~ then + ~ (1) ( X I (2) u,/n
+ . . . X,) -+
belong to some class
03
+
for some 6 > 0
C ~ uniformly , in m.
u = C,”=-,Cow(X0, X,)< 03.
We are able to complete the proof of the CLT, under strong mixing conditions, giving a proof of Theorem 4.1.
Proof.
Combining Theorems 4.3 and 4.4, and defining
(C:=,+,X , + Z,(‘1 )/fi, we obtain that for m 2 n,
vJT’ =
where c(m) + 0 as m --+ 00. We show this using the idea of ‘rooms and corridors’ - that the sum can be decomposed into sums over blocks which are large, but separated, and so close to independence. For example, writing w2/2)= (Em+, z=m+l X , ) / , / 6 Z(‘/2), Theorem 4.4 shows that
+
(+),
1 <- Q (V,-,?(‘I2)
W(T/2) n
w y q +f K ( l / J m )
In the notation of Theorem 4.3, c(m) = fk (1/ fi))1/3-c.
=Q(Jm)+fk(l/Jm).
(4.29) C(K,~/2,~)(a(fi)
+
77
Dependent random variables
We first establish convergence along the ‘powers of 2 subsequence’ s
k
=
v,!.),writing i?k for ( c i = 2 k xi + ~!~))/fi, since 2k+l
Jst(sk+l)
5
Jst(Sk)
+ c(k) - A ( S k , 5 k r 1/21
(4.30)
where c(k) -+ 0. Then use an argument structured like Linnik’s proof , all k 2 K . [Linnik, 19591. Given E , we can find K such that c(k) I ~ / 2 for Now (1) either for all k
2 K , 2 c ( k ) 5 A(Sk,Sk, 1/2), and so Jst(Sk)
-
Jst(Sk+l)
2 A(&,
Sk,
w / 2 ,
(4.31)
Ck
so summing the telescoping sum, we deduce that A(Sk, S k , 1/2) is SL,l / 2 ) I E . finite, and hence there exists L such that A(SL, (2) or for some L 2 K , 2c(L) 2 A(SL,g , ~1/2), , then A(SL,SL,l / 2 ) I E .
Thus, in either case, there exists L such that A(SL,SL,l / 2 ) I E , and hence by Proposition 4.1, J s t ( S L ) I u ( E ) . Now, for any k 2 L , either J s t ( S k + l ) I J s t ( S k ) , or A ( S k , ? k , 1/2) I c(k) 5 E . In the second case, J s t ( S k ) 5 u ( E ) so , that J , t ( S k + l ) I v ( t ) E . In either case, we prove by induction that for all k 2 L , that J , t ( S k + i ) 5
+
U(E)
+
E.
We can fill in the gaps to gain control of the whole sequence, adapting the proof of the standard subadditive inequality, using the methods 0 described in Appendix 2 of [Grimmett, 19991.
4.3
4.3.1
Proof of subadditive relations
Notation and definitions
This is the key part of the argument, proving the bounds at the heart of the limit theorems. However, although the analysis is somewhat involved, it is not technically difficult. We introduce notation where it will be clear whether densities and score functions are associated with joint or marginal distributions, by their number of arguments: px(x) will be the score function of X , and p i ( . ) the ~ , ~ y) ( Zwill , derivative of its density. For joint densities px,y(x,y), P (1) be the derivative of the density with respect to the first argument and (1) (1) PX,Y(Z> Y) = P x , y b Y)/PX,Y , (xC1Y), and so on.
Information Theory and the Central Limzt Theorem
78
Note that a similar equation t o the independent case tells us about the behaviour of Fisher information of sums:
Lemma 4.2 If X , Y are random variables, with joint density p(x,y), (2) and score functions and px,y then X + Y has score function j5 given bY
pi?y
[
F ( z ) = E p$ly(X, Y)lx
+Y =
21
=E
[ p?)i(X,Y)Ix + Y =
i].
(4.32)
Proof.
Since
X
+ Y has density r ( z ) = S p x , y ( z - y , y ) d y , then (4.33)
Hence dividing, we obtain that
0
as claimed. For given a,b, define the function M ( z ,y) = Ma,b(x,y) by
M ( X , Y ) = a ( P p Y ( z . Y ) - PX(.))
+b (Pju2iy(4
-
PY(Y))
1
(4.35)
which is zero if X and Y are independent. Using properties of the perturbed density, we will show that if a ( S , T )is small, then M is close to zero.
Proposition 4.2 If X , Y are random variables, with marginal score functions p x , p y , and i f the s u m flX m Y has score function p then
PJ(4
+ (1
-
P ) J ( Y )- J
+ (fix+ m
y
)
+ 2 d m m P X ( X ) P Y ( Y )+ 2EMfi,-(X,W(X
+Y)
Proof. By the two-dimensional version of the Stein identity, for any function f ( z , y) and for i = 1 , 2 : Epj;i,(X, Y )f (X, Y )= -IEf@)(X, Y).
(4.37)
+ y ) , for any a, b: E(apx ( x ) + b p y (Y))ij(X + Y ) = (a+ b)J(x+ Y ) E h f a , b(x,Y)F(x+ Y). Hence, we know that taking f(x,y) = j?(x
-
(4.38)
Dependent random variables
79
+
+
+
By considering s p ( x , y ) ( a p x ( z ) b p y ( y ) - (u b)p(x Y))~dxdy, dealing with the cross term with the expression above, we deduce that:
+
a 2 J ( X ) b 2 J ( Y )- ( a
+ b)2J(X + Y )
+
+ b)IEMa,b(X,Y ) p ( X + Y ) ( a + b)F(X + Y ) y .
+2abEpx(X)py ( Y ) 2(a = E (.px(X)
+bpy(Y)
-
(4.39)
As in the independent case, we can rescale, and consider X ' = f l X , Y' = m y , and take a = P , b = 1 - P. Note that ,@px,(u) = px(u/,@), m P Y W =P Y ( U / r n ) . 0 4.3.2
Bounds o n densities
Next, we require an extension of Lemma 3 of [Barron, 19861 applied to single and bivariate random variables:
+
+
Lemma 4.3 For any S , T , define ( X , Y ) = ( S Z g ) , T Z$') and define p(2T)for the density o f ( S + Z r T ) ,T+Z?')). There em& a constant cT,k = f i ( 2 k / T e ) k ' 2 such that for all x , y:
and hence (4.43)
Proof.
We adapt Barron's proof, using Holder's inequality and the bound cT,k&'T(U)for all u.
(U/r)kdT(U)5
(4.44)
A similar argument gives the other bounds.
0
Informatzon Theory and the Central Limit Theorem
80
Now, the normal perturbation ensures that the density doesn't decrease too large, and so the modulus of the score function can't grow too fast.
Lemma 4.4 Consider X of the form X = S If X has score function p, then for B > 1: BJ;
p(u)2du5
8 B3 ~
J;
+ Zg),where Var S IK r .
(3 + 2 K ) .
(4.47)
Proof. As in Proposition 4.1, p(u) 2 (2exp2K)-'&p(u), so that for u E (-B&, B&), (B&p(u))-' 5 2&exp(B2+2K)/B 5 2&rexp(B2+ 2 K ). Hence for any k 2 1, by Holder's inequality:
Since we have a free choice of k 2 1 to maximise kexp(v/k), choosing ti = w 2 1 means that k exp(v/k) exp(-1) = w. Hence we obtain a bound of BJ;
8B p ( ~ ) ~Id u (B'
J;
8B3 + 2K + l o g ( 2 6 ) ) L (3 + 2 K ) . J;
(4.51)
0
By considering S normal, so that p grows linearly with u,we know that the B3 rate of growth is a sharp bound.
+
+
For random variables S , T , let X = S ' : 2 and Y = Y ZR),define LB = (1x1 IBJ?, lyl 5 BJ?}. Ifmax(Var S,Var T ) 5 K r then there exists a function f l ( K ,r ) such that for B 2 1:
Lemma 4.5
+
EMa,b(x,Y ) F ( ~Y ) I I ( ( XY, ) E
LB)
I Q ( SW , 4 ( a+ b ) f l ( K 7). , (4.52)
Proof. Lemma 1.2 of Ibragimov [Ibragimov, 19621 states that if E , v are random variables measurable with respect to A, B respectively, with 5 C1 and IvI 5 C2 then:
Dependent random variables
Now since 1$,(u)1 5 l/&, deduce that
81
and 1u&(u)/~1i exp(-1/2)/-,
we
Similarly:
(4.56)
By rearranging
hfa,br
we obtain:
By Cauchy-Schwarz
I 2cu(s,
7rr
JEEqG%q(a
+ b)
+ J16B4(3 + 2K) (4.60)
This follows firstly since by Lemma 4.4
(4.61) and by Lemma 4.4 (4.62)
Information Theory and the Central Limit Theorem
82
4.3.3
Bounds o n tails
Now uniform decay of the tails gives us control everywhere else:
Lemma 4.6 For S, T with m e a n zero and variance 5 K r , let X = S Z g ) and Y = T Z$'. There exists a function f 2 ( r , K , E ) such that
+
+
For S,T with k t h m o m e n t (k 2 2 ) bounded above, we can achieve a rate of decay of l / B k - ' . Proof.
By Chebyshev's inequality
+
P (( S + Z p T )T, + Z?") $ L"> 5
~ P ( ~ ' ) ( Zy ), ( x 2 y 2 ) / ( 2 B 2 r ) d z d y5 ( K l / q = 1:
for l / p
+
+ 2 ) / B 2 so by Holder's inequality
(4.67)
(4.68)
By choosing p arbitrarily close to 1, we can obtain the required expression. 0 The other terms work in a similar way. Similarly we bound the remaining product term:
Lemma 4.7 For random variables S , T with m e a n zero and variances satisfying max(Var S , Var T ) 5 K r , let X = S + 2;) and Y = T Z g ) . There exist functions f Z ( r , K ) and f 4 ( r ,K ) such that
+
Proof. Using part of Lemma 4.5, we know that p x , y ( a , y ) p x ( x ) p y ( y ) 5 Zcu(S, T ) / ( m ) . Hence by an argument similar to that of
Dependent random variables
83
Lemmas 4.6, we obtain that
0
as required.
We can now give a proof of Theorem 4.3
Proof. Combining Lemmas 4.5, 4.6 and 4.7, we obtain for given K , 7, E that there exist constants C1, Cz such that
so choosing B = ( 1 / 4 c ~ ( S , T ) ) ~ > / ' 1, we obtain a bound of C ~ I ( S , T ) ~ / ~ - ' By Lemma 4.6, note that if X , Y have bounded lcth moment, then we obtain decay at the rate C1a(S,T)B4 C2/Bk',for any k' < lc. Choosing B = C P ( S , T ) - ' / ( " +we ~ )obtain , a rate of C X ( S , T ) " / ( ~ ' + ~ ) . 0
+
4.3.4
Control of the mixing coeficients
+
To control a ( X 2 , Y ) and t o prove Theorem 4.4, we use truncation, smoothing and triangle inequality arguments similar t o those of the previous section. Write W for X 2,L B = ((2, y) : 1x1 5 B f i , IyI 5 B f i } , and for R n ( - B f i , B f i ) . Note that by Chebyshev's inequality, P((W,Y ) E L L ) 5 IP(lW( 2 B f i ) IP(IY1 2 B f i ) 5 2(K 1 ) / B 2 . Hence by the
+
+
+
84
Information Theory and the Central Limit Theorem
triangle inequality, for any sets S , T :
Here, the first inequality follows on splitting R2 into L B and LC.,, the second by repeated application of the triangle inequality, and the third by expanding out probabilities using the densities. Now the key result is that:
+
+
Proposition 4.3 For S and T , define X = S 2 : ) and Y = T Zg), where max(Var S,Var T ) K r . If Z has variance t , then there exists a constant C = C ( B ,K , r ) such that
Proof.
We can show that for
121
I b2 and 1x1 5 B J ;
(4.79)
(4.80)
Dependent random variables
85
by adapting Lemma 4.4 to cover bivariate random variables. Hence we know that
(4.83)
I (expC6 - I) + 2 P ( l ~ 2l 6’). Thus choosing b =
the result follows.
(4.84)
0
Similar analysis allows us to control (4.85)
This page intentionally left blank
Chapter 5
Convergence to Stable Laws
Summary In this chapter we discuss some aspects of convergence to stable distributions. These are hard problems, not least because explicit expressions for the densities are only known in a few special cases. We review some of the theory of stable distributions, and show how some of the techniques used in earlier chapters will carry forward, with potential to prove convergence to the Cauchy or other stable distributions. In particular, we will show how certain partial differential equations imply a new de Bruijn-like identity for these variables. Whilst the results in this chapter do not give a complete solution to the problem, we will discuss how our techniques have some relevance.
5.1
5.1.1
Introduction to stable laws
Definitions
Although the Central Limit Theorem is the most familiar result of its kind, there are other possible types of convergence on the real line, under different normalisations. Indeed the normal distribution may be viewed as just one of the family of one-dimensional stable distributions. Authors such as [Zolotarev, 19861 and [Samorodnitsky and Taqqu, 19941 review the theory of these stable distributions. It appears that many such distributions have a physical significance, in fields such as economics, biology and modelling the large-scale structure of the universe. For example [Holtsmark, 19191 considered the distribution of atomic spectral lines - by assuming a uniform distribution of sources in 87
I n f o m a l i o n Theory and the Central Limit Theorem
88
three-dimensional space, the inverse square law implies the distribution will have the Holtsmark law, that is, a stable distribution with cy = 3/2. The second part of [Uchaikin and Zolotarev, 19991 and Chapter 1 of [Zolotarev, 19861 review these applications.
Definition 5.1 A random variable Z is stable if it can be expressed as the limit (in the sense of weak convergence) of normalised sums:
where Xi are independent and identically distributed, and a,, b, are sequences of real numbers.
It turns out that essentially one only needs to consider the case a , = 7
~
~
Example 5.1 (1) If a = 1 and E X i is finite, the law of large numbers means that the limit law will be deterministic. (2) For any value of a , if Xi are deterministic then so is the limit. (3) The Central Limit Theorem corresponds t o the case CY = 2. (4) If a = 1 then the Cauchy distribution may be reached as a limit.
There is a large amount of theory concerning the stable laws, dating back to authors such as [Lkvy, 19241, [Khintchine and L&y, 19361 and [Gnedenko and Kolmogorov, 19541. These papers generally use the method of L6vy decomposition, which is concerned with categorising the possible values of the logarithm of the characteristic function. This method leads to a complete categorisation of all stable distributions, including the fact that convergence in Equation (5.1) is only possible if 0 < a 5 2. That is:
Theorem 5.1 A function @ ( t ) is the characteristic function of a stable distribution if and only if: Q ( t ) = exp [zyt - cltla(l
= exp [iyt - cltIa(I
where y E Iw, c
+ ipsgn(t) tan(xa/2))] f o r a # 1, -
2 0, 0 < CY 5 2 and
ipsgn(t)2log ltl/x)] for a
=
1,
(5.2) (5.3)
-1 5 /3 5 1
The paper [Hall, 19811 discusses the history of this representation in great detail, describing how most authors have given an incorrect version of the formula.
Convergence t o stable laws
89
Note that Theorem 5.1 gives a complete description of the characteristic functions of stable random variables, not the densities. [Zolotarev, 19941 discusses the fact that only three families of stable distributions have densities which can be expressed in terms of standard functions. [Zolotarev, 19941 and [Hoffman-Jorgensen, 19931 go on to discuss ways of representing other densities as expansions in the incomplete hypergeometric function. [Gawronski, 19841 shows that all stable densities are bell-shaped - that is, the kth derivative has exactly k simple zeroes. In particular, stable densities are unimodal, though not in general log-concave. However, for now, we shall just list the three families where an explicit expression for the density is possible. Note that in each case the pair ( a ,0) determines the family of density, whilst y is a location parameter (referring to the position of the density), and c is a scaling parameter. In particular, if gr,c gives the density within a particular stable family, then
except in the case a = 1, p
# 0, which we ignore for now.
Definition 5.2 (1) Gaussian density has ( a ,p) = (2,O);
(2) Cauchy density has ( a l p )= ( 1 , O ) ;
(3) L&y (or inverse Gaussian) density has (a,/3)= (1/2, fl); L,Y(X)
=
L,Y(X)
=
5.1.2
C
.(I > y) for p
= 1,
C
I(x < y) for
=
I./I./-
-1.
(5.7) (5.8)
Domains of attraction
Definition 5 . 3 A random variable X is said to belong in the ‘domain of normal attraction’ of the stable random variable 2 if weak convergence in
90
Information Theory and the Central Limit Theorem
Definition 5.1 occurs with a ,
N
nila
We know from the Central Limit Theorem, Theorem 1.2, that this occurs in the case a: = 2 if and only if the variance J- x c 2 d F x(x) is finite. In other cases Theorem 2.6.7 of [Ibragimov and Linnik, 19711 states:
Theorem 5.2 T h e random variable X belongs in the domain of normal attraction of a stable random variable Z with parameter CY with 0 < CY < 2 , if and only if there exist constants Icl and 162 (at least one of which i s nonzero) such that Fx(z)(zJ+ a 161 as z
+
-m,
(1 - FX ( x ) ) x a+ Icz as x
+
co.
(5.9) (5.10)
Hence we can see that f o r stable 2 with parameter 0 < 01
< 2, (5.11) (5.12)
Pages 88-90 of [Ibragimov and Linnik, 19711 relate the constants kl,IC2 to the Lkvy decomposition, 5.1. A rate of convergence in L1 is provided by [Banis, 19751, under conditions on the moments. His Theorem 2 implies: Theorem 5.3 Consider independent random variables X I ,X z , . . ., each with densityp, and consider g a stable density of parameter cy for 1 5 cy 5 2 . Defining the normalised s u m U, = ( X I . . . X,)/n1Ia t o have density p , then if
+ +
1.1
(1) p ( m ) = s x " ( p ( x ) - g ( x ) ) d x = 0 f o r m = 0 , 1 , . . . l + (2) ~ ( m=)J Ixl"(p(z) - g ( z ) ) d x < 03 f o r m = 1 a
+
then:
1
IPn(S)
-
0
g ( x ) (dx = 0 n-lIa .
(5.13)
This suggests via Lemma 1.8 and Proposition 5.1 that we should be aiming for an O(n-'Ia) rate of convergence of Fisher information and relative entropy.
Convergence to stable laws
5.1.3
91
Entropy of stable laws
In [Gnedenko and Korolev, 19961, having summarised Barron's approach to the Gaussian case, the authors suggest that a natural generalisation would be t o use similar techniques t o prove convergence t o other stable laws. They also discuss the 'principle of maximum entropy'. They suggest that distributions which maximise entropy within a certain class often turn out to have favourable properties: (1) The normal maximises entropy, among distributions with a given variance (see Lemma 1.11). It has the useful property that vanishing of covariance implies independence. (2) The exponential maximises entropy among distributions with positive support and fixed mean (see Lemma 1.12). It has the useful property of lack of memory.
Next [Gnedenko and Korolev, 19961 suggests that it would be useful t o understand within what classes the stable distributions maximise the entropy. Since we believe that both stable and maximum entropy distributions have a physical significance, we need t o understand the relationship between them. In the characteristic function method, the fact that convergence in Definition 5.1 is only possible in the non-deterministic case with a, = nl/" for cr 5 2 is not obvious, but rather the consequence of the fact that a certain integral must exist. In our entropy-theoretic framework, we can view it as a consequence of Shannon's Entropy Power inequality, Theorem D . l , which is why the case cy = 2, corresponding to the normal distribution, is seen t o be the extreme case.
Lemma 5.1 If there exist IID random variables X and Y with finite entropy H ( X ) such that ( X Y)/2lla has the same distribution as X , then cr 5 2.
+
+
Proof. By the Entropy Power inequality, Theorem D . l , H ( X Y) 2 H ( X ) + 1 / 2 . Hence as H ( X ) = H ( ( X + Y ) / 2 1 / Q = ) H(X+Y)-log21/a 2 0 H(X) l / 2 - l / a , the result holds.
+
In Appendix A we show how t o calculate the entropies of these stable families. Lemmas A.5 and A.7 give
+
(1) The entropy of a Lkvy distribution H ( l c , 7 ) = log(167rec4)/2 3 ~ . / 2 , where K Y 0.5772 = limn-m(Cr=l l/i - logn) is Euler's constant.
92
Information Theory and the Central Limit Theorem
(2) The entropy of a Cauchy density p c , ? ( z ) = c/7r(c2
5.2
+ x2) is log4m.
Parameter estimation for stable distributions
The first problem of stable approximation is even to decide which stable distribution we should approximate by. That is, given a density p , which we hope is close to a stable family {gc,-,}, how do we decide which values of the parameters y and c provide the best fit? For general stable distributions, this is a hard problem, in contrast with the normal and Poisson distributions, for which it is a simple matter of matching moments (see Lemma 1.13 and Lemma 7.2). Further, in those cases these optimising parameters behave well on convolution, that is if $ X and + y are the closest normals to X and Y , the closest normal to X Y will simply be +X + y (and a similar fact holds for the Poisson). As we might expect, the situation for general stable distributions is more complicated. Indeed we suggest three separate methods of deciding the closest stable distribution, which will give different answers.
+
+
5.2.1
M i n i m i s i n g relative e n t r o p y
In general, given a density p , and a candidate stable family, note that since
differentiating with respect to c and y will yield that for the minimum values:
(5.15) (5.16) introducing a natural role for the score functions pc and pr (with respect to scale and location parameters respectively) of the stable distribution gc,r.
Lemma 5.2 For a random variable X , the closest Cauchy distribution in the sense of relative entropy has y = G , c = C , where
f x ( c i + G) = I 2c ?’
(5.17)
for f x ( z ) the Cauchy transform (see Definition 8.2), f x ( z ) = E(z - X ) - l .
Convergence to stable laws
Proof.
93
If X has density p then
Now, if this is minimised over c and y respectively a t ( C , G ) ,we deduce that
(5.20) (5.21) Now, adding
(-2)
times ( 5 . 2 1 ) t o (5.20), we deduce that
-a G-X-iC dx = 2c = / p ( x ) C 2 + ( G - x)2
/
p ( x ) dz G-x+iC
=
f x ( C i + G). (5.22)
For example, in the case where X is Cauchy with parameters c and y,since fx(z)= 1/(z - y ci), so that f x ( c i y) = l / ( c i y - y ci) = -2/(2c). We can also make progress in the analysis of the Lkvy distribution.
+
+
+
+
Lemma 5 . 3 For a r a n d o m variable X supported on t h e set [a,oo), the closest Le‘vy distribution in t h e sense of relative entropy has y = a , c = 1/JIE(1/(X - a ) ) . Proof. We can exploit the fact that the Lkvy distribution gc,? is only supported on the set { x 2 y}. By definition, if y > a , the relative entropy will be infinite, so we need only consider y 5 a (so in fact the case a = -CC can be ignored).
Now, differentiation shows that this is an decreasing function in y,so the best value is the largest possible value of y,that is y = a.
94
Information Theory and the Central Limit Theorem
Without loss of generality, therefore, we can assume that a = y = 0. In this case, then, we can differentiate with respect to c to deduce that (5.26)
implying that 1/c2 = E(l/(X - a ) ) , or c = l / & ( l / ( X - a ) ) . Hence in this case, at least we can read off the values of c and y,though 0 the behaviour of c on convolution remains obscure.
5.2.2
Minimising Fisher information distance
Of course, another way to choose the parameters c and y would be to minimise the Fisher information distance, rather than relative entropy distance. In the case of the normal, the two estimates coincide.
Lemma 5.4 Consider {goz,,} the family of normal densities with score function (with respect to location parameter) ~ ~ = 2 (dg,z,,/dx)/g,z,,. , ~ Given a density p with score p = (dp/dx)/p,the Fisher information distance
is minimised for p Proof.
= EX,
a’ = Var X
In general we know that for a stable family gc,r:
In the case of the normal; pUz,, = - ( x - p ) / v ’ , so the second term becomes s p ( x ) ( - 2 / u 2 (x - P ) ~ / C Tso ~ )we , can see that the optimising values are p = E X , u2 = Var X. 0
+
For other families, Equation (5.29) can sometimes be controlled. For example, for the L6vy family with y = 0 (fixed as before by consideration of the support), pc,?(z)= - 3 / ( 2 x ) c 2 / ( 2 z 2 ) . In this case Equation (5.29)
+
Convergence to stable laws
95
tells us to minimise: (5.30) Differentiating with respect to c2, we obtain 0 = -7E-
1 x3
+ CZE-,x14
(5.31)
showing that we should take c such that c2 = 7(E1/X3)/(E1/X4).
5.2.3
Matching logarithm of density
Definition 5.4 For given probability densities f,g we can define:
A,(f)
=
/
03
-f (.I
1ogds)dT
(5.32)
-m
setting A g ( f )
= 00
if the support of f is not contained in the support of g.
We offer a natural replacement for the variance constraint:
Condition 5 Given a density h, and a family of stable densitiesp,, define the approximating parameter c to be the solution in c to (5.33)
Lemma 5.5
Equation (5.33) has a unique solution in c.
Proof. We can use the fact that pc(x) = pl(x/cl/*)/cl/*, where pl has a unique maximum a t zero. Hence, Equation (5.33) becomes
where u = l/cl/" and K ( u ) = J f(y)logpl(yu)dy. Since K'(u) = J f(y)(pi(yu)/pl(yu))ydy, the unimodality of p l means that K ' ( u ) < 0. Now, since Jpl(x)xa' < 00 for a' < a , the set such that pl(z)za' > k is finite for any k. We deduce that K ( u ) + --03 as u 400. Hence we know that K ( 0 ) = logpl(0) 2 Jpl(y)logpl(y)dy 2 K ( ~ o ) , 0 so there is a unique solution, as claimed. Notice that in the Gaussian case A$(f) is a linear function of Var f and is constant on convolution, and A$( f) needs to be finite to ensure convergence in the Gaussian case. In the general case, we might hope that convergence
Information Theory and the Central Limit Theorem
96
in relative entropy t o a particular stable density would occur if and only if the sequence Ap(fn)converges to a limit strictly between - logp(0) and infinity. Since - f(x)log g ( z ) d s = [ - F ( x ) logp(z)l: J~~ ~(s)p,(x)dz, Lemma A.6 shows that we can rewrite h,(f) in terms of the distribution function F and score function p s , t o obtain (for any t )
s,”
h , ( f )= - log g ( t )
+
+ loge
t
[L
P(Y)F(Y)dY
+
Lrn
P ( Y ) ( l - F(Y))dY]
.
(5.35) By Theorem 5.2, we know that distribution function F is in the domain of normal attraction of a Cauchy distribution if limy*-m l y J F ( y ) = limy+my(l - F ( y ) ) = k. Since the Cauchy score function p ( y ) = - 2 y / ( l + y2), we deduce that if F is in the normal domain of attraction of a Cauchy distribution, then A p , , o ( f )is finite.
5.3 5.3.1
Extending de Bruijn’s identity Partial differential equations
Recall that the proof of convergence in relative entropy to the normal distribution in [Barron, 1986) is based on expressing the Kullback-Leibler distance as an integral of Fisher informations, the de Bruijn identity (see Appendix C). This in turn is based on the fact that the normal density satisfies a partial differential equation with constant coefficients, see Lemma C . l . However, as Medgyessy points out in a series of papers, including [Medgyessy, 19561 and [Medgyessy, 19581, other stable distributions also satisfy partial differential equations with constant coefficients. Fixing the location parameter y = 0 for simplicity:
Example 5.2 (1) For the L6vy density l c ( x ) = c / s e x p (-c2/22), one has: d21,
81,
ac2
ax
- =2--.
(2) For the Cauchy density p c ( z ) = c/(7r(x2
-+@Pc
d2pc
dC2
3x2
(5.36)
+ c 2 ) ) ,one has:
= 0.
(5.37)
Convergence to stable laws
97
These are examples of a more general class of results, provided by [Medgyessy, 19561:
Theorem 5.4 If a = m / n (where m,n are coprime integers), and k,M are integers such that ,G'tan(~ra/2)= tan(xk/Mn
-
7ra/2),
(5.38)
then the stable density g with characteristic function given b y Theorem 5.1 satisfies the equation:
where if a = 1, p m u s t equal 0, and where B B=Oifcu=l. E x a m p l e 5.3
= fl tan(-rra/2)
zf a
#
1, and
Theorem 5.4 allows us t o deduce that
(1) If a = 2, p = 0 (normal distribution), then m = 2, n = 1,k = 1,M = 1,B = 0, so a Z g / a x 2 = ag/ac. (2) If a = m / n , p = 0 (where m = am'), then k = m', M = 1,B = 0, so
(- l)l+"mg/dxm = a n g / a c n . = 1, p = 0 (Cauchy distribution), then m = l , n = l,k= 1 , M = 2 , B = 0, so a2g/3x2+ d2g/3c2= 0. (4) If Q = m/n, fl = 0 ( m odd), then k = m , M = 2 , B = 0 , so a2mg/ax2m a 2 n g p C 2 n = 0. ( 5 ) If a = l / 2 , /3 = -1 ( L b y distribution), then m = 1,n = 2, k = 0, M = 1,B = 2, so 2dg/dx = d2g/dc? (6) If a = m / n , p = -1, then Ic = 0, M = 1,B = - tan-ira/2 provides a solution, so (1 B 2 ) d m g / a z m = dg"/d"c. ( 3 ) If a
+
+
Theorem 5.4 provides examples where, although we don't have an explicit expression for the density, we can characterise it via a differential equation. More importantly, we know that if gc is a stable density which satisfies a partial differential equation with constant coefficients, then so does h,, the density of f * g,, for any density f .
5.3.2
Derivatives of relative entropy
Using these results, we can develop results strongly reminiscent of the de Bruijn identity, expressing the Kullback-Leibler distance in terms of an integral of a quantity with a n L 2 structure. Again, we would hope to use
98
Informatton Theory and the Central Limit Theorem
the theory of L2 projection spaces to analyse how this integrand behaves on convolution. Recall that in Definition 1.11,we define the Fisher matrix of a random variable U with density g depending on parameters 81, . . .On as follows. Defining the score functions p i ( % ) = g(x)-'(ag/dOi)(x), the Fisher matrix has (2, j ) t h coefficient:
(5.40) Example 5.4 If U is the family of Lkvy random variables with density 1,,7 then p c ( x ) = 1/c - c/x and p y ( x ) = 3/2x - c2/2x2,and
J(l,,7)
=
(3/22 21/2c4) 2/c2
3/2c3
(5.41) '
In analogy with the Fisher information distance with respect to location parameter, we define a corresponding quantity for general parameters:
Definition 5.5 For random variables U and V with densities f , and g, respectively, define the Fisher information distance:
As in the normal case, we consider the behaviour of D and J along the semigroup. Given a density f , define h , t o be the density of f * g c , for g c a scaled family of stable densities and k, for the density of g1 * g,. Then, writing D" = D(h,IIk,) and J" = J"(h,//k,), dD"
(5.43)
IC, ac
d2D" d3Dc
dJ"
h, d2k,
+ J"
(5.44)
Convergence to stable laws
Furthermore:
J 2 log
(2)
dx =-
99
dh, 1 dk, 1 Jhc(zh, - z--) = J -hk ,c dakxc' (5.46) dh,dk, 1
1!% (2) log
dz
=
ax3
=
-
J 13% h, ax a x 2
-12 J $
+
I
d2h, dk, __-ax2
(5.47) 1
ax IC,
d2h, d k , 1 (2)3d~+/=Kk; . (5.48)
Similarly,
J --dx=Tx? :
J zd k(, - -1-d h , k , ax
hc--) dk, 1 dx k,2
(5.49)
Using this we can deduce:
Theorem 5.5 G i v e n a density f , define h , to be t h e density o f f * l C , o , and k, for t h e density of l l , o * l C , o , where lc,7 is t h e L i v y density. (5.50)
Proof.
Note that h, and l l + , both satisfy the partial differential equation (5.51)
Hence, by Equation (5.44): d2h, =J
h, d 2 k ,
+ h,
(2k 2k)
2
-
(2,I,
dx (5.52)
2 2 1 0 g ( 2 ) - h, 2 dkc ~ ~ + h ,
So, on substituting Equation (5.46), the result follows.
d c k , ) 2 dz. (5.53) akc
0
Similarly :
Theorem 5.6 G i v e n a density f , define h, t o be t h e density o f f * p C , o , and k , for the density of p l , o * p c , o , where p,,? i s t h e C a u c h y density.
Information Theory and the Central Limit Theorem
100
Proof. This follows in a similar way on combining Equations (5.44), 0 (5.47) and (5.49). Such expressions can even be found in cases where an explicit expression for the density is not known. For example,
If
Lemma 5.6
gc is the family of densities with ( c Y , ~ = ) (3/2, -l), then
(5.55)
Proof.
Since (5.56)
by Equation (5.44): (5.57) (5.58)
This last formula can be rearranged to give the claimed result.
5.3.3
0
Integral f o r m of the identities
Just as in the normal case, we can convert Theorems 5.5 and 5.6 to an integral form. For example, in the Cauchy case, for any s < t ,
+
D“ = Dt - ( t - s)(Dt)’
I’
(U
- s)(D~)”C/U.
(5.60)
Consider fixing s close t o zero, and increasing t t o infinity. First, we can argue that Dt is an decreasing function of t , by using the conditioning argument from Section 1 of page 34 of [Cover and Thomas, 19911, and therefore it converges t o a limit. Specifically, by the log-sum inequality, Equation (1.37), for any integrable functions g ( x ) ,h ( x ) ,
Convergence t o stable laws
101
normalising t o get probability densities p ( z ) = g(z)/ J g ( v ) d v , q(z) = h ( z ) / h(w)dw, the positivity of D(pllq) implies that
This means that
(5.63) Since the integral (5.60) tends t o a limit as t ca,two of the terms on the RHS converge, and therefore so must the third; that is limt-m -(t s)(Dt)’ = E 2 0. However, if the limit E is non-zero, then D ( t ) asymptotically behaves like --E log t , so does not converge. Hence, we deduce that ---f
Ds
=
+
lim Dt ‘t03
Loo
(u- s)(D”)”du.
(5.64)
Now, since the relative entropy is scale-invariant the limit limt,m Dt is: lim A,, (aU
a-0
+ (1 - a ) Z )
-
H(aU
+ (1 - a ) Z ) .
(5.65)
By the increase of entropy on convolution (Lemma 1.15), H ( a U a ) Z ) 2 H((1 - a ) Z ) ,so if we can show that lim Apl (crU
a-0
+ (1
-
+ (1
-
(5.66)
a ) Z ) = Ap,( Z ) ,
+
we deduce that limt+oo Dt = 0. This follows since (a) aU (1 - a ) Z Z in probability (since a ( U - 2) 0 in probability) (b) A,, (aU (1 - a ) Z ) form a UI family; since - logpl(2) is bounded for small x, and concave for large x. By semi-continuity of the entropy, lims-o D“ = D(fllpl,o),so we deduce that ---f
D(fllP1,o) =
loo
‘LL(DU)”d?
where the form of (D”)” is given by Theorem 5.6.
---f
+
(5.67)
Information Theory and the Central Limit Theorem
102
Relationship between forms of convergence
5.4
Again, convergence in Fisher information (with respect to the location parameter) is a strong result, and implies more familiar forms of convergence. Just as with Lemma E . l and Theorem 7.6, the relationship comes via a bound of the form const
m.
Proposition 5.1 Suppose h is a log-concave density, symmetric about 0, with score Ph = h‘/h. Then there exists a constant K = K ( h ) such that for any random with density f : (1)
s If(.)
-
(2) SUPZ If(.)
Proof.
h(x)ldx I 2 K v T i - m - h(x)l I (1+
WO))drn.
We follow and adapt the argument of [Shimizu, 19751. Write P(X) =
h ( x ) ( f( x ) / h ( x ) ) ’= f ’ ( x ) - f ( x ) m ( x ) .
(5.68)
Then: (5.69) where C
=
f(O)/h(O).Hence, for any set L:
(5.70)
(5.72) Now, picking a value a > 0, for any a 2 x 2 0 , unimodality of h and Equation (5.72) imply that
Alternatively, we can define for x
>a >0 (5.74)
so that
(5.75)
Convergence to stable laws
103
implying by Equation (5.72): (5.76) Similarly
Then, using the Equations (5.76) and (5.78), we deduce that (5.79)
1
m
h ( z ) ( - p ( z ) ) A ( z ) d z= =
+
h(a)A(a)
/
-h'(z)A(z)
(5.80)
W
(5.81)
h(z)A'(z)
a
(5.82) (5.83) by Equation (5.72). Overall then, combining Equations (5.69), (5.73) and (5.83),
5
hm
h(x)
LZ
-dydz
(5.85)
Now, at this stage, we take advantage of the free choice of the parameter a. It can be seen that the optimal value to take is the a* such that
Information Theory and the Central Limit Theorem
104
and we call the constant a t this point K
= 3/(-ph(a*))
+ a*. That is:
This allows us to deduce that
1I f
(x)--h(x)ldx
I
1
If(.)-Ch(~)ld~+l1-CI
1
Ih(x)ldx
I2 K d m m l
(5.89) as claimed. Further, combining Equations (5.69), (5.72) and (5.88), we deduce that for any x,
as required.
0
If h ( z ) = gc(z) = g ( z / c l / " ) / c l / * (part of a stable family) the condition (5.87) implies that the optimising value is QZ = c ' / ~ Q *and the bounds from Proposition 5.1 become scale-invariant and take the farm:
Remark 5.1 This suggests a conjecture concerning standardised versions of the Fisher information distance. Given a stable random variable of parameter Q < 2, f o r any k < a , define
JSt(XllZ) = (Elx1k)2ikJ(X1l.z). W e conjecture that f o r U, = ( X I
(5.93)
+ . . . X , ) / r ~ l / ~where , X , are IID,
if and only if X I is in the domain of normal attraction of 2, and if Js,(XlllZ) is finite. For example, consider Z Cauchy. We can see that for X with too heavy a tail (for example a Lkvy distribution), such that EJXlk= 00, J s t ( U n ) will remain infinite. On the other hand, for X with too light a tail (for example
Convergence t o stable laws
105
if X is normal), U, will be normal, so (IE(Ulk)2’k= co;. Conversely, the extended Cramkr-Rao lower bound, Equation (2.95) with k(x) = z gives
(
J(U((22 ) 1-IE-
1 2u2 +u2
)2
a;’
(5.95)
so the limitlikm
5.5
Steps towards a Brown inequality
In some cases, we can describe the equivalent of Hermite polynomials, giving a basis of functions orthogonal with respect to the L6vy and Cauchy distributions.
Lemma 5.7
If H1 are the Hermite polynomials with respect t o the Gaussian weight d l / c ~then , Kl(x)= c2‘H[(1/&) f o r m a n orthogonal basis with respect to the L4uy density l c ( x ) . Proof.
Using z = l / y 2 we know that
(5.96)
(5.98)
where F ( y ) = f ( l / y 2 ) , G(y) = g ( l / y 2 ) . Hence, if ( f ,f) is finite, then ( F ,F ) is finite, and F so can be expressed as a sum of c k H k ( y ) , where H k are Hermite polynomials. Thus f ( z ) = F ( 1 / & ) = ckHk(l/fi). 0
xk
Lemma 5.8 Let T, and U , be the Chebyshev polynomials of the first and second type defined byT,(cosO) = cosnO and U,(cosO) = sin(n+l)O/sinQ. T h e n P,(x)= T,(2x/(l+x2)) andQ,(z) = (1-x2))/(1+z2))un(2z/(1+z2)) f o r m a n orthogonal basis with respect to Cauchy weight pi(.) = ~ / ( l x+2 )
Information Theory and the Central Limit Theorem
106
Proof. Using the substitution .ir/4-6/2 = tan-' x , we know that cos 6' 2 2 / ( 1 + x 2 ) and that -df3/2 = 1/(1+ x 2 ) d x . Hence:
=
and so on. Hence the result follows from standard facts concerning Fourier series. 0 Note that the maps the stable case.
Lemma 5.9
and
2 described in Chapter 2 can be generalised t o
For a stable density p with parameter a , the maps (5.101) (5.102)
are adjoint to each other.
Proof.
For any g and h:
=
J'p(z)h(x)
(J' 2'lap(2'lau
=
-
1
rc)g(u)du d x
(5.105) (5.106)
0
m,
Now, we'd like to find the eigenfunctions of and calculate the minimum non-zero eigenvalue. Unfortunately, the situation is not as simple as it is for the normal distribution. We always know that:
Lemma 5.10 For a stable density p with parameter a , the function uk ( p - ' ) d k p / d x k i s a (2-'/la)-eigenfunction of %.
=
Convergence t o stable laws
Proof.
107
By definition (5.107)
since p ( u ) = 2 1 / " J p ( x ) p ( 2 1 / a u - x ) d x , we can use the fact that d / d u ( p ( 2 ' l a u - x)) = - 2 l / * d / d x ( p ( 2 l l Q u- x)). 0 Now in the normal case, we can complete the proof since and are equal. However, this will not be true in general, where we can only proceed as follows:
(J'p(u)k
(s) d u ) dx
(5.109)
(5.110)
so we would like an explicit expression as a function of u and w for
J'
2+p(x)p(21/"u
-
x)p(21/%
-
z)
(5.112)
dx.
P('lL)
We can at least find such an expression where p is the Cauchy density; use the fact that for s and t :
for u ( s ) = 2 s / ( l + s2), so that Equation (5.112) becomes:
A
4
((1
+ u2
+ w2)(1+
so if k satisfies
(w- u)2)
+-1+1 + 1 + (ww2
u)2
),
(5.114)
j" k ( w ) p ( w ) d w= 0 , then: -
MLk = ( M
+ L)k/2,
(5.115)
where M and L are the maps defined in (2.12) and (2.13). To continue the method of [Brown, 19821, the next step required would be a uniform bound on the ratio of densities of U 2, and Z,,,, where U belongs t o a certain class (for example, for IEIUlr = 1 for some T ) , an equivalent of Lemma 2.7.
+
Information Theory and the Central Limit Theorem
108
We will use properties of general stable distributions, summarised by [Zolotarev, 19861, page 143. Specifically, we firstly use the facts, that all stable distributions are unimodal, without loss of generality, having mode a t 0. Secondly, if p # f l , then the density is supported everywhere (if ,L? = f l , the support will be the positive or negative half axis).
Lemma 5.11 Let 2, be a scaled family of stable densities such that p # f l , with densities g,, and U be a random variable. Writing p c f o r the density of U Z,, so p , = p * g,, then there exists a constant R > 0 such that f o r all x
+
(5.116)
Proof. Notice that g c l z ( t ) (Lemma 2.7),
=
21/ag,(21/at).
As in the normal case
(5.117)
Now by monotonicity of the density, we need to bound:
(
min(R-, R + ) = min g,(x
-
2)/gc(2'/ax),g,(x
+ 2)/gC(2'/"x)) ,
(5.118)
without loss of generality, we consider the case where x is positive. Notice that if x 2 5 2l/"x then R + 2 1. Otherwise R+ 2 g,(2/(1 2 - 1 / a ) ) / g c ( 0 ) , which is just a constant. Similarly, if z 2 2, g,(x - 2 ) 2 g , ( x Z), so R- 2 R+ and if 0 5 x 5 2, 0 g c ( x - 2) L g c ( - 2 ) , so R- 2 g c ( - 2 ) / g , ( O ) .
+
+
The case where /3 = f l is similar, though we need to restrict to random variables supported on the same set as 2,. Thus, if we consider a normalised sum of random variables U, = (XI . . . X n ) / ( m 1 l a ) we , shall require that IP(IUnl 5 2 ) is uniformly bounded below. One way to achieve this is to obtain an upper bound on the 6th moment of U,, since then Chebyshev's inequality gives us the control we require. Further work is clearly required to give a complete proof of convergence of Fisher information to the Cauchy or Lkvy distribution, let alone a general stable distribution.
+
+
Chapter 6
Convergence on Compact Groups
Summary We investigate the behaviour of the entropy of convolutions of independent random variables on compact groups. We provide an explicit exponential bound on the rate of convergence of entropy to its maximum. Equivalently, this proves convergence in relative entropy to the uniform density. We prove that this convergence lies strictly between uniform convergence of densities (as investigated by Shlosman and Major), and weak convergence (the sense of the classical Ito-Kawada theorem). In fact it lies between convergence in L1+‘ and convergence in L1.
6.1 6.1.1
Probability on compact groups
Introduction t o topological groups
In our understanding of probability on topological groups, we shall generally follow the method of presentation of [Kawakubo, 19911 and [Grenander, 19631. The book [Heyer, 19771 provides a comprehensive summary of many results concerning probability on algebraic structures. We use some arguments from [CsiszAr, 19651, where an adaption of Renyi’s method is used t o prove weak convergence. In Section 6.1.2 we lay the foundations for this, reviewing the compactness arguments that will be necessary. In Section 6.1.3 we review a series of papers by Shlosman ([Shlosman, 19801 arid [Shlosman, 1984]), and by Major and Shlosman [Major and Shlosman, 19791, which refer to uniform convergence. In defining topological groups, we consider a set G in two ways. Firstly, we require that the set G should form the ‘right kind’ of topological space. Secondly, we define an action of composition on it, and require that a group 109
Information Theory and the Central Limit Theorem
110
structure be present.
Definition 6.1
A set G is a topological group if
(1) The members of G form a Hausdorff space (given any two distinct points, there exist non-intersecting open sets containing them). (2) G forms a group under the action of composition ( 3 ) The map y : G x G -+ G defined by ~ ( gh ,) = g * h-' is continuous.
*.
Example 6.1 Any group equipped with the discrete topology, the real numbers under addition, the set of non-singular n x n real-valued matrices G L ( n ,R), the orthogonal group O(n) and the special orthogonal group SO(n) are all examples of topological groups. Some authors require that the space G should be separable (have a countable basis) but this is generally for the sake of simplicity of exposition, rather than a necessary condition. We will be concerned with compact groups, defined in the obvious fashion:
Definition 6.2 pact.
A topological group G is compact if the space G is com-
Example 6.2 Any finite group is compact, when equipped with the discrete topology. Other compact groups are S O ( n ) , the group of rotations . .x,) E [0,1)" under of Rn and the torus T" (the set of vectors x = (51,. component-wise addition mod 1). The groups S O ( n ) and T" are also connected, in an obvious topological sense, which is an important property. The main reason that we focus on compact groups is because of the existence of Haar measure p , the unique probability distribution on G invariant under group actions. A convenient form of the invariance property of Haar measure is:
Definition 6.3 Haar measure p is the unique probability measure such that for any bounded continuous function f on G and for all t E G,
J
f(S)dP(S) =
J f ( s * t)dP(S)
=
1f *
( t s)dl*.(s) =
J
f(s-l)dl.(s). (6.1)
We will often write d s for d p ( s ) , just as in the case of Lebesgue measure. In a manner entirely analogous t o definitions based on Lebesgue measure on the real line, we define densities with respect to Haar measure. This in turn allows us to define entropy and relative entropy distance.
Convergence on compact groups
111
Definition 6.4
Given probability densities f and g with respect to Haar measure, we define the relative entropy (or Kullback-Leibler) distance:
By the Gibbs inequality, Lemma 1.4, this is always non-negative. We usually expect the limit density of convolutions to be uniform with . relative respect to Haar measure on G, a density which we write as 1 ~The entropy distance from such a limit will be minus the entropy: (6.3)
and thus we establish a maximum entropy principle, that H ( f ) 5 0, with equality iff f is uniform. Given probability measures P, Q , we define their convolution as a probability measure R = P * Q such that
Lf
( s ) R ( d s )=
s,L
f ( s* W d s ) Q ( d t ) ,
(6.4)
for all bounded continuous functions f . The uniqueness of R follows from the Riesz theorem.
Lemma 6.1
For any Bore1 set B , the convolution takes on the form
R ( B )=
11 G
Il(s*t E B ) P ( d s ) Q ( d t ) .
G
Note that this definition even holds in the case where only semigroup structure is present. We will consider the case where the probability measures have densities with respect to Haar measure. In this case the convolution takes on a particularly simple form.
Lemma 6.2 Given probability densities f and g with respect t o Haar measure, their convolution (f * g ) is
6.1.2
Convergence of convolutions
Now, having defined convolutions of measures, we can go on to consider whether convolutions will converge. We will consider a sequence of probability measures ui,and write pn for the measure u1 * . . . v,, and gn for the
Information Theory and the Central Limit Theorem
112
density of pn, if it exists. Note that one obvious case where the measures do not converge is where supp(u,) is a coset of some normal subgroup for all i, in which case periodic behaviour is possible, though the random variables in question may not have a density.
Example 6.3 Let G = Z 2 , then if ui is 61,the measure concentrated a t = . . . = So, but p2m+l = ~ 2 ~ + = 3. . . = S 1 , the point 1, then pzm = so convergence does not occur. However, broadly speaking, so long as we avoid this case, the classical theorem of [Stromberg, 19601 (which is a refinement of an earlier result by Ito-Kawada) provides weak convergence of convolution powers.
Theorem 6.1 W h e n u, are identical, u, = u, then if supp(u) is not 1~ contained in a n y non-trivial coset of a normal subgroup then p n weakly. --f
Using elementary methods [Kloss, 19591 shows exponential decay in total variation, that is:
Theorem 6.2 If for each measurable set A , ui(A) 2 c i p ( A ) , then, uniformly over measurable sets B ,
We are interested in methods which extend these results up to uniform convergence and convergence in relative entropy, and in obtaining useful bounds on the rate of such convergence. One problem is that of sequential compactness. Any set of measures has a weakly convergent subsequence. However there exist sequences of probability densities on a compact group with no uniformly convergent subsequence, and even with no L1 convergent subsequence.
Example 6.4 If G is the circle group [0, l), define the density fm(u)= mI(O 5 u 5 l / m ) . Then for n 2 m: llfm
-
fnlll =
1
Ifm(u)- fn(u)ldp(u)= 2 ( 1 - m / n ) .
Thus if n # m then - f27.t in L', so cannot converge. I I f i r k
//I
2 1. Hence
{fn}
(6.8)
is not a Cauchy sequence
Convergence o n compact groups
113
One approach is described in [Csiszbr, 19651, where he proves that sequences of convolutions have uniformly convergent subsequences, allowing the use of Rknyi’s method. Recall the following definitions:
Definition 6.5 Consider F,a collection of continuous functions from a topological space S into a metric space ( X ,d ) . We say that 3 is equicontinuous if for all s E S , E > 0, there exists a neighbourhood U ( s ) such that for all t E U ( s )
d(f(s), f ( t ) ) 5 E for all f E 3.
(6.9)
We define ‘uniform equicontinuity’ in the obvious way, requiring the existence of some neighbourhood V, such that s * V C U ( s ) . In that case, t * s-l E V implies t E s * V C U ( s ) ,so that d( f (s), f ( t ) ) 5 E .
Definition 6.6 A metric space ( X , d ) is totally bounded if for all there exists a finite set G such that for any z E X , d ( z , G ) 5 E .
E
> 0,
Clearly, in a space which is complete and totally bounded, every sequence has a convergent subsequence. We can combine this with the Arzelb-Ascoli theorem, which states that:
Theorem 6.3 Let ( X ,d ) be a compact metric space and 3 be a collection of continuous functions o n X . T h e n 3 is totally bounded an supremum n o r m if and only if it is uniformly equicontinuous. Hence, using Arzelh-Ascoli, we deduce that in a complete space, a uniformly equicontinuous family has a subsequence convergent in supremum norm. This is used by CsiszSr as follows:
Theorem 6.4 Given a sequence of probability measures u1, u 2 , . . . o n a compact group G , where u, has a continuous density for some m, define gn to be the density of u1* v2 A . . . * un for n 2 m. There exists a subsequence n k such that gnk is uniformly convergent. Proof. Since v, has a continuous density, g, is continuous. Since it is continuous on a compact space, it is uniformly continuous, so there exists U such that if s 1 * sY1 E U , then l g m ( s l ) - gm(s2)l 5 E.
Informatzon Theory and the Central Limzt Theorem
114
Now if l g n ( s l ) - gn(s2)1 I E for all gn+l(sz)l 5 E for all s 1 * szl E U , since
s1
* syl
E
U then l g n + l ( s l )
-
(6.12)
*
since (s1* t - l ) ( s 2 *t-')-' = s1* s;' E U . This implies uniform equicontinuity of the set { g n : n 2 m } , and hence by Arzel&-Ascoli, the existence 0 of a uniformly convergent subsequence. We would like t o understand better the geometry of Kullback-Leibler distance. Note that Example 6.4 uses fm such that D ( f m l l l ~= ) logm, so we might conjecture that a set of densities gm such that D(gm /I 1 ~5)D 5 00 has a subsequence convergent in relative entropy. If such a result did hold then, by decrease of distance, if D(gmII 1 ~is )ever finite, there exists a subsequence convergent in relative entropy. Further we could identify the limit using Rknyi's method and results such as Proposition 6.1. 6.1.3
Conditions f o r uniform convergence
In a series of papers in the early 1980s, Shlosman and Major determined more precise criteria for the convergence of convolutions of independent measures. They gave explicit bounds on the uniform distance between convolution densities and the uniform density, in contrast t o theorems of Ito-Kawada and Stromberg (Theorem 6.1), which only establish weak convergence. We are particularly interested in the question of what conditions are necessary to ensure an exponential rate of convergence in the IID case. All their methods use characteristic functions. We present their principal results as stated in the original papers. Firstly, Theorem 1 of Shlosman [Shlosman , 19801:
Theorem 6.5 Let ul, u2,. . . be probability measures o n a compact group G , where u1, uz,. . . have densities p l , p 2 , . . . s u c h that p i 5 ci 5 00 for all i . W r i t i n g gn for the density of v1 * u2 * . . . un, for a n y n 2 3, r n
1-1
(6.13)
Convergence o n compact groups
115
Hence boundedness of densities in the IID case provides decay rates of O(l/n).
Proof. The proof works by first proving the result for the circle group, by producing bounds on the Fourier coefficients, and using Parseval’s inequality to translate this into a bound on the function itself. Next extend up to general Lie groups. An application of the Peter-Weyl theorem shows that the Lie groups form a dense set, so the results follow. 0 Note that since the random variables have densities, we avoid the case where the measure is concentrated on a normal subgroup. Given a signed measure T ,there exist measurable H + , H - such that for any measurable E : T ( En H + ) 2 0, T ( En H - ) _< 0. This is the so-called Hahn decomposition of the measure. Definition 6.7 quantities:
Given a measure u and x,y E
w)l,
M/(.)
= P[H+(U
N,,(y)
= inf{x : M,,(x)
-
R,define the following (6.14)
< y}
(the inverse of M ) ,
(6.15) (6.16)
S(v)=
1=’4
(6.17)
x2Nv(x)dx.
If v has a density p, then M,,(x) is the Haar meaure of the set {u : p(u) 2 x}. Example 6.5 If G is the circle group [0,1) and v, has density fJy) = c 2(1 - c)y then
+
5 c, x-c =I----for c I x 5 2 - c, 2(1- c) = 0 for x 2 2 - c.
M v L ( x )= 1 for
N,c((y)= 2
-
2
c - y(2
= 0 for y
-
2c) for 0 5 y 5 1,
2 1.
+
(6.18) (6.19) (6.20) (6.21)
(6.22)
Hence, we know that s ( v c )= (1 c)/2, and that S(v,) = (7r3/6144)((16 c(37r - 8)). We plot these functions in Figures 6.1 and 6.2.
37r)
+
Using this, Theorem 2 of [Shlosman, 19801 states that:
116
Information Theory and the Central Lzmit Theorem
Fig. 6.1 Muc (I)plotted as a function of I for c = 0.3.
Fig. 6.2
N v , ( z ) plotted as a function of I for c = 0.3
Theorem 6.6 Let v1,u 2 , . . . be probability measures o n a compact group G. If u l , 14,. . . v6 have densities pl,pz,.. . ,p6 E L 2 ( G , d s ) t h e n writing gn for t h e density of v1 * v2 * . . . u,:
This proof works by decomposing the measures into the sum of bounded densities. Again we obtain a decay rate of O(l/n).
Convergence o n compact groups
117
The main Theorem of Major and Shlosman [Major and Shlosman, 19791 actually only requires two of the measures to have densities in L 2 ( G ,ds):
Theorem 6.7 Let u1, U Z , . . . be probability measures on a compact group G. If for some i, j , measures ui, uj have densities p i , p j E L 2 ( G , d s )then, writing gn for the density of u1* u2 . . . v,,
Proof. As with Theorem 6.5, use the Peter-Weyl theorem t o gain bounds based on the n-dimensional unitary representation. The proof considers the sphere SZn-' E covering it with balls. The crucial lemma states that if p ( z , y ) is the angle between the two points, then the balls of pradius r have size less than r (when r 5 7r/4), when measured by any invariant probability measure. Hence any invariant measure is dominated by p . 0 Using these techniques, we can in fact obtain exponential decay in the IID case. Corollary 3 of Shlosman [Shlosman, 19801 states that:
Theorem 6.8 If for all i, ui = u , where u is a measure with density f E L1+'(G,ds) for some E , then there exist a , p such that for n > 2 1 6 : SUP 1gn(S) -
sEG
11 I aexp(-nP).
(6.25)
Proof. This result follows since if f l E LP(G,ds) and f 2 E Lq(G,d s ) then by Holder's inequality, f l * f 2 E L*q(G,ds). We can then apply Theorem 6.7. 0 In [Shlosman, 19841, Shlosman shows that if the group is non-Abelian, then convergence will occur in more general cases. This indicates that lack of commutativity provides better mixing, since there are no low-dimensional representations. In particular, the group SO(n) is a significant example. Theorem 6.5 indicates that a sufficient condition for convergence is the divergence of CZ1cL2. Another result from [Shlosman, 19841 indicates is sufficient. that in the non-Abelian case, divergence of CZ,'c, We hope to come up with different bounds, using entropy-theoretic techniques, in order to see what conditions are sufficient to guarantee convergence in relative entropy and what conditions will ensure an exponential rate of convergence. However, first we need t o understand the relationship of convergence in D to other forms of convergence. The remainder of this chapter is based on the paper [Johnson and Suhov, 20001.
Information Theory and the Central Limit Theorem
118
6.2 6.2.1
Convergence in relative entropy I n t r o d u c t i o n and results
We will consider X i , X 2 , . . ., independent random variables on a compact group with measures u1, u2, . . .. We will write p, for the convolution u1*. . .* u,. We will assume that X , has a density with respect to Haar measure, and we write g, for the density of p, (for n 2 m). This notation holds throughout this paper. We define two quantities based on these measures: Definition 6.8
For each measure u,, define: c,
=
d, =
NV,,Q),
(6.26)
(Lrn
When u, has density f,: c,
-1
Y-2(1 - M&( W Y )
= ess
inf(f,), d;'
=
.
(6.27)
f,(g)-'dp(g)
Our principal results are the following. Firstly, an explicit calculation of rates: Theorem 6.9
For all n 2 m: D(gn111G) 5 D(gn-l/11G)(1 - c n ) .
Hence i f D ( g , l l l ~ )is everfinite, and
CC, = 00,
(6.28)
then D(gnIIIG) + 0.
Although the classical Ito-Kawada theorem is strictly not a corollary of this result, Theorem 6.9 essentially generalises it in several directions. In particular we gain an exponential rate of convergence in the IID case. The question of whether different types of convergence have exponential rates is an important one. Alternatively, we replace c, by the strictly larger d,, but in this case cannot provide an explicit rate. Theorem 6.10 If for some m and and Ed, = 00 then D(g,IllG) -+ 0.
E,
the L1+, norm llgmlll+E is finite,
In Section 6.2.2, we discuss entropy and Kullback-Leibler distance on compact groups, and show that the distance always decreases on convolution. In Section 6.3, we discuss the relationship between convergence in relative entropy and other types of convergence. In Section 6.4, we prove the main theorems of the paper.
Convergence o n compact groups
119
We would like to understand better the geometry of Kullback-Leibler distance. Is it true, for example, that a set of measures bounded in D has a D-convergent subsequence? If so, by decrease of distance, if D(g,([lG) is ever finite, there exists a D-convergent subsequence, and we hope to identify the limit using results such as Proposition 6.1.
6.2.2
Entropy on compact groups
We require our measures to have densities with respect to p , in order for the Kullback-Leibler distance to exist. Notice that if g ( u ) = II(u E G) = ~ G ( u ) , the Kullback-Leibler distance D( f 119) is equal to - H ( f ) , where H ( f ) is the entropy of f . Hence, as with Barron’s proof of the Central Limit Theorem [Barron, 19861, we can view convergence in relative entropy as an entropy maximisation result. A review article by Derriennic [Derriennic, 19851 discusses some of the history of entropy-theoretic proofs of limit theorems. It is easy to show that entropy cannot decrease on convolution. It is worth noting that this fact is much simpler than the case of convolutions on the real line, where subadditivity in the IID case is the best result possible. The latter result was obtained by Barron as a consequence of Shannon’s Entropy Power inequality, Theorem D. 1.
Lemma 6.3 Let X I ,X p ,. . . be independent random variables o n a compact group with measures ulru 2 . . .. If u1 has a density, write gn for the density of u1* . . . * u,. T h e n defining the right shzjl ( R , f ) ( u )= f ( u * v),
and hence, by positivity of D , D(gn(llG) decreases o n convolution.
Information Theory and the Central Limit Theorem
120
Proof. Now since (6.30)
(6.31)
(6.32)
Definition 6.9 Given random variables X , Y with marginal densities f x , fY and joint density f x , Y , define the mutual information
Proposition 6.1 If X , Y are independent and identically distributed with density f x and D ( f y * X J I l G )= D ( f Y l l l G ) then the support of fx is a coset H u of a closed normal subgroup H , and f x is uniform o n H * u.
*
Proof. The support of a measure defined as in [Grenander, 19631 and [Heyer, 19771 is a closed set of v-measure 1, so we can assume that fx(Ic) > 0 on SUPP(VX).
I ( X ;Y * X ) =
//
f x ( u ) f y ( v* u-')log
('7'*
Y*X
d p ( u ) d p ( u ) (6.35)
(6.36) (6.37) Hence equality holds iff (X,Y * X ) are independent, which is in fact a very restrictive condition, since it implies that for any u , u:
f x ( u ) f Y * x ( v= ) fX,Y*X(U,.)
= fx(u)fY('u*u-').
(6.38)
Hence, if f x ( u )> 0, f v ( v * u - l ) = f y * x ( v ) ,so f y ( v * u - l ) is constant for all u E supp(vx). Note this is the condition on page 597 of [CsiszBr, 19651,
121
Convergence o n compact groups
from which he deduces the result required. Pick an element u E supp(vx), then for any u1,u2 E supp(u,y), write hi = ui * u-l.Then, for any w , f y ( ~ (h1h2)-l) * =
~ ~ ( w * u * u ~ ~ * u * u ~ ~ ) (6.39)
* * u y l * u * u - ~ =) f y ( w * h y l ) , (6.40)
= f y ( ~u
Hence for any u,taking w = u* hl *ha, w = v* h2 and w = u* ha* hl* hz, (6.41)
f y ( u )= f y ( u * h i ) = f y ( u * h Y ' ) = f u ( u * h z * h i ) .
Hence, since f y = f x ,we deduce that supp(vx) * u-l forms a group H , and that f x is constant on cosets of it. Since this argument would work for both left and right cosets, we deduce that H is a normal subgroup. 0 6.3
Comparison of forms of convergence
In this section, we show that convergence in relative entropy is in general neither implied by nor implies convergence uniform convergence. However, we go on t o prove that convergence in relative entropy t o the uniform density is strictly weaker than convergence in any L1+', but stronger than convergence in L1.
Example 6.6 For certain densities h, supu Ifn(u) - h(u)l 0 does not imply that D(fnllh) 4 0. For example, in the case G = 2 2 (densities with respect to the uniform measure (1/2,1/2)): --f
h(0) = 0 , h(1) = 2, f n ( 0 ) = l / n , fn(l) =2
-
l/n.
(6.42)
Hence for all n, supulfn(u) - h(u)I = l / n , D(fn\lh) = 03. Whenever h is zero on a set of positive Haar measure, we can construct a similar example. Now, not only does convergence in relative entropy not imply uniform convergence, but for any E > 0, convergence in relative entropy does not imply convergence in L1+'.
Example 6.7 D(fnllh) + 0 does not imply that s I f n ( u ) h(u)ll+'dp(u)-+ 0 , for 1 < E < 2. In the case of the circle group [O, l ) ,
h(u)= 1 f n ( u )= 1
-
(6.43)
+ n2/€
= 1 - n 2 / € / ( 7 2 2 - 1)
(0 521 < l / d )
(6.44)
5 u < 1).
(6.45)
(l/n2
122
Information Theory and the Central Limit Theorem
Clearly D(fnllh) -+ 0, but
Ifn(u) - h(u)Jl+'dp(u) 2 1.
Note that by compactness, uniform convergence of densities implies L1 convergence of densities. By the same argument as in the real case, convergence in relative entropy also implies L1 convergence of densities If(.) - h(u)ldp(u)+ 0 (see [Saloff-Coste, 19971, page 360). By the triangle inequality, L' convergence of densities implies convergence in variation: If ( A )- h(A)I = I f ( u )- h(u))n(uE A)dp(u)l 0 , uniformly over measurable sets A. This is the equivalent of uniform convergence of distribution functions, the classical form of the Central Limit Theorem. This is in turn stronger than weak convergence. However, things are much simpler whenever h(u) is bounded away from zero. Firstly, D ( f l l h ) is dominated by a function of the L2 distance. This is an adaption of a standard result; see for example [Saloff-Coste, 19971.
s(
---f
Lemma 6.4 If h(u) is a probability density function, such that h(u) 2 c > 0 for all u , then for any probability density function f ( u ) ,
Proof.
In this case, since logy 5 (y - 1)loge,
Firstly, we can analyse this term to get Eq.(6.46) since it equals
Secondly, we obtain Eq.(6.47), since it is less than
In fact, we can prove that if the limit density is bounded away from zero then convergence of densities in L1+' implies convergence in relative entropy. This means that convergence to uniformity in D lies strictly between convergence in Life and convergence in L1. Example 6.7 and examples based
Convergence on compact gTOUpS
123
on those in Barron [Barron, 19861 show that the implications are strict. We use a truncation argument, first requiring a technical lemma.
Lemma 6.5
For any
E
> 0 and K > 1, there exists ~ ( K , Esuch ) that (6.51)
Proof. We can write the left hand side as a product of three terms, and deal with them in turn:
(6.53) Finally, calculus shows that on {x 2 K > l}: x-'logz _< (loge)/(cexp 1). Combining these three expressions we see that
Theorem 6.11 If f n is a sequence of probability densities such that for some E and c > 0 : 1 f n ( u )- h(u))'"'dp(u) + 0 , and h(u) 2 c > 0 , then
(6.55) Proof.
For any K > 1, Lemma 6.5 means that (6.56)
r
(6.58) Hence using convergence in L1+', and the fact that K is arbitrary, the proof is complete. 0 We can produce a partial converse, using some of the same techniques t o show that convergence in relative entropy is not much stronger than convergence in L1.
Information Theory and the Central Limit Theorem
124
Lemma 6.6 F o r a n y a bility measure,
2 1, i f f is a probability density, a n d v a proba-
Ilf Proof.
*vIIa
L Ilflla.
(6.59)
Since
(f*v)(g) =
J f (g*h-l)d.(h)
5
( J f ( g* h - l ) " d v ( h y ( J d v o )
l-l/a 1
(6.60) the result follows, in an adaptation of an argument on pages 139-140 of [Major and Shlosman, 19791. 0
Theorem 6.12 L e t gn = v1 * v2 * . . . vn, where v,, h a s a density f o r s o m e ml, and there exists E > 0 s u c h t h a t llgmlll+E< co f o r s o m e m > ml. If t h e sequence of densities gn (n 2 m ) i s s u c h that llgn - 1111 4 0 t h e n
Proof. Lemma 6.6 means that (gn - 1) is uniformly bounded in L1+' for n 2 m. Now, by Holder's inequality, for l / p + l / g = 1,
Taking
t' =
€12, use
l+t
1 + E
E - E'
p = - 1+ p q = ,s= E - E' 1+/
I
+ E ' , b = r + s = 1 + 2E + EE' l+e
> 1, (6.63)
to deduce that
and hence we obtain convergence in Lb which implies convergence in D . 0
Convergence o n compact groups
6.4
125
Proof of convergence in relative entropy
6.4.1
Explicit rate of convergence
One approach providing a proof of D-convergence is the following. The difference D(g,-lII l ~ ) - D ( g , (1( ~is )either large, in which case convergence is rapid, or it is small, in which case by Lemma 6.3, gn-l is close to each of the shifts of g n , and is therefore already close to uniformity. The case we must avoid is when v, has support only on a coset of a subgroup, and gn-l is close to the shifts Rug, only where v,(u) is positive. First we require two technical lemmas: Lemma 6.7
Ifp and q are probability densities, and (R,q)(v) = q(v*u):
Proof. By Jensen'sinequality:
(6.70)
0 Lemma 6.8 If v1 has density 91, and density g2, a n d g2 2 c2 everywhere.
Proof.
Picking z = c2
-
E,
c2 =
Nv2(l),t h e n
u1
* u2 has
a
p ( H + ) = 1, where H + = H+(v2 - z p ) . For
any u:
J 2/
g 2 ( u )2
=Z
gl(u v-l)n(v E ~ + ) d ~ ( v )
g1( U
U ~ (U
* v-l)n(v E ~ + ) x d p ( u )
* H’) = 5 ,
where u E H‘ if and only if u-l E H f .
(6.71) (6.72) (6.73)
Information Theory and the Central Limit Theorem
126
We can now give the proof of Theorem 6.9:
Proof. If c, is zero for all m, this is just the statement of decrease of distance. Otherwise, if cm is non-zero, by Lemma 6.8, D(g,-lIIRug,) is bounded uniformly in n 2 m 1 and u,since it is less than D(g,-lII 1 ~- ) log c, . Since c, = Nv,,(l),for any E , using z = c, - E , Mvn(x) = 1, so writing H + = H f ( u n - x p ) , and H - for the complement of H + , p ( H - ) = 0. Now by Lemma 6.3:
+
~ ( g n - l l l l c )- D(gnIllc)
(6.74)
and the fact that t ness of the uniform boundedness of arbitrarily close to zero the result follows. cchoosing 6.4.2
No explicit rate of convergence
We can provide alternative conditions for convergence (though this does not give an explicit rate). Given a measure u , we can define for each point u E G: z ( u ) = inf{y : u
$ H+(v - yp)}.
(6.78)
(the intuition here is that if v has density g, then z coincides with 9). Lemma 6.9
Proof. Hence
For any bounded function f on the group G,
For any
X<
1, taking y = Xz(u) means that u E H + ( v - yp).
Convergence o n compact groups
127
So, by Cauchy-Schwarz the LHS:
and since X is arbitrary, the result follows. We will therefore be interested in d, = of Shlosman's functions, this is (J,"y-'(l Lemma 6.9 is trivial when Y has density f .
0
(s l/z(u)dp(u))-'. -
MVn(y))dy)-'.
In terms Note that
Theorem 6.13 Let X I Xz, , . . . be independent random variables on a compact group with measures u1, u ~ ,. .. . Suppose X I has a density and write gn f o r the density of u1 * . . . * vn. If D(g,IllG) is ever finite, and Ed, 400 then llgn - 1\11 + 0 . Proof. 19671:
By Lemmas 6.3 and 6.9 and the equality of Kullback [Kullback,
D(gn-lll1G)
-
D(gnIl1G) =
2
1
D(gn-llIRugn)dvn(u)
(1
11gn-l-
(6.82)
~ u g n ~ ~ ; d u n (/2 u ) ) (6.83)
Now using the triangle inequality, we obtain 111Sn-1
-
RugnIIldP(u) (6.85)
(6.87)
So, suppose D = D(gm111G) is finite. llgn - 1111 decreases, so if it doesn't tend to zero, there exists positive E , such that llg, - 1I(1 > E for all n. But if this is so, then (6.88) n=m
n=m
128
Information Theory and the Central Limit Theorem
providing a contradiction. Hence we can conclude the proof of Theorem 6.10:
Proof.
By Theorem 6.12 and Theorem 6.13.
Chapter 7
Convergence to the Poisson Distribution
Summary In this chapter we describe how our methods can solve a different, though related problem, that of the ‘law of small numbers’ convergence to the Poisson. We define two analogues of Fisher information, with finite differences replacing derivatives, such that many of our results will go through. Although our bounds are not optimally sharp, we describe the parallels between Poisson and normal convergence, and see how they can be viewed in a very similar light.
7.1
7.1.1
Entropy and the Poisson distribution
The law of small numbers
Whilst the Central Limit Theorem is the best known result of its kind, there exist other limiting regimes, including those for discrete-valued random variables. In particular, all the random variables that we consider in this chapter will take values in the non-negative integers. A well-studied problem concerns a large number of trials where in each trial a rare event may occur, but has low probability. An informal statement of typical results in such a case is as follows. Let X I , XZ, . . . , X , be Bernoulli(pi) random variables. Their sum
s, = X I + x2 + . . . + x,
(7.1)
has a distribution close t o Po(X), the Poisson distribution with parameter X = C p .%,if: (1) The ratio supi p z / X is small. 129
130
Information Theory and the Central Limit Theorem
(2) The random variables
Xi are not strongly dependent.
Such results are often referred to as 'laws of small numbers', and have been extensively studied. Chapter 1 of [Barbour et al., 19921 gives a history of the problem, and a summary of such results. For example, Theorem 2 of [Le Cam, 19601 gives:
For X i independent Bernoulli(pi) random variables, wrating Sn = Cy=lX i and X = Cr=lp i :
Theorem 7.1
(7.2) For example, if p , = X/n for each i , the total variation distance 5 8Aln. Improvements to the constant have been made since, but this result can be seen to be of the right order. For example, Theorem 1.1of [Deheuvels and Pfeifer, 19861 gives: Theorem 7.2 For X , independent Bernoulli(p,), writing S, = C 2 , X , and 1, = - log(1 - p , ) , i f 1, 5 1 then for all p:
xr=l
Hence for p , = A/n, if X < 1 this bound asymptotically gives d T v ( S n , Po(p)) 2 (Az exp(-X)/2) In. We will argue that these results can be viewed within the entropytheoretic framework developed in earlier chapters of this book. Specifically, just as Lemma 1.11 shows that the Gaussian maximises entropy within a particular class (the random variables with given variance), [Harremoes, 20011 has proved that the Poisson distribution maximises entropy within the class B(X) of Bernoulli sums of mean A. Formally speaking:
Lemma 7.1
Define the class of Bernoulli s u m s
B(A) = { S , : Sn = X I + . . .+X,, f o r X , independent Bernoulli,IES,
= A}.
(7.4) T h e n the entropy of a random variable in B(X) is dominated by that of the Poisson distribution:
sup
H(X)= H(Po(X))
XEB(X)
Note that Po(X) is not in the class B(X).
(7.5)
Convergence to the Poisson distribution
131
One complication compared with the Gaussian case is that we do not have a simple closed form expression for the entropy of a Po(X) random variable. That is, writing PA(.) = e-’Xx/x! for the probability mass function of a Po(X) variable, for X N Po(A) the simplest form is:
H(X) =
c
P’(2)(A
- 2 log
x + log z!)
(7.6)
2
=
(7.7)
A-XlogX+IElogX!.
The best that we can do is t o bound this expression using Stirling’s formula; for large A, the entropy behaves like log(2neX)/2, the entropy of the normal with the same mean (as the Central Limit Theorem might suggest). Although we lack an expression in a closed form for the entropy, as Harremoes shows, we can find the closest Poisson distribution to a particular random variable just by matching the means. Given an integer-valued random variable X , the relative Lemma 7.2 entropy D(XI\Po(X)) is minimised over X at X = E X .
Proof.
Given X , we can simply expand the relative entropy as: D(x/IPo(X)) = C
p ( z )(logp(z) + logz! + Xloge
X
=
-H(X)
+ IElog X! + Xlog e
-
- zlogX)
(EX)log A.
(7.8)
(7.9)
Hence, differentiating with respect to A, we obtain EX = A, as required.0 We can therefore define
D ( X ) := inf D(XIIPo(X)) = D(XIIPo(EX)). x
(7.10)
In this chapter, we will show how the techniques previously described can be adapted t o provide a proof of convergence t o the Poisson distribution in the law of small numbers regime. When p , = X/n for all i , the bound from Theorem 7.1 becomes 8X/n. Combined with Lemma 1.8, this suggests that we should be looking for bounds on Fisher information and relative entropy of order O(l/n2). In Section 7.1.2, we give simple bounds on relative entropy, though not of the right order. In Section 7.2, we define Fisher information, and deduce results concerning the convergence of this quantity. In Section 7.4, we relate Fisher information and relative entropy, and in Section 7.3 we discuss the strength of the bounds obtained.
132
7.1.2
Information Theory and the Central Limit Theorem
Simplest bounds on relative entropy
Let us consider the question of whether relative entropy always decreases on convolution. In the case of relative entropy distance from the Gaussian, the de Bruijn identity (Theorem C.l), combined with Lemma 1.21, ensures that for X and Y independent and identically distributed,
D (X
+ Y)I D ( X ) ,
(7.11)
~ for conwhere D ( X ) = inf,,,z D(XI14p,,*) = D ( X I I ~ E X x, V) .~However, vergence to the Poisson, the situation is more complicated. We can make some progress, though, in the case where the distributions are binomial and thus the entropy is easy to calculate. We know that
+(n-r)log(l - p ) + r l o g e ) .
(7.12)
Example 7.1 Hence D(Bin(1,p))= (1- p ) log(1 - p ) +ploge. Similarly, D(Bin(2,p)) = 2(1 - p ) l o g ( l - p ) +2ploge - p 2 . That means that if we take X and Y independent and both having the Bin(1,p) distribution then D ( X Y ) I D ( X ) if and only if p 2 2 (1 - p ) log(1 - p ) plog e, which occurs iff p 5 p * , where p* 2 0.7035.
+
+
Lemma 7.3 The relative entropy distance from a binomial random varaable to a Poisson random variable satisfies
) n p 2 loge. D(Bin(n,p ) ( ( P o ( n p ) I
(7.13)
Proof. Expand Equation (7.12), since log (n!/((n - r)!n')) 5 log(l(1 1/n)(1- 2 / n ) . . .) 5 0, and C:=,r(:)pr(l - p)"-' = n p . This gives the bound D(Bin(n,p)jlPo(np)) 5 n((1 - p)log(l - p ) ploge), and since 0 log( 1 - p ) 5 - p log e the result follows.
+
Now, this result can be extended to the case of sums of non-identical variables, even those with dependence. For example, Theorem 1 of Kontoyiannis, Harremoes and Johnson [Kontoyiannis et al., 20021 states that:
cYEl
Theorem 7.3 If S, = X i is the sum of n (possibly dependent) Bernoulli(p,) random variables XI, X2, . . . , Xnj with E(Sn) = pi = A, then n
D(S,IIPo(X)) 5 loge
- H ( X l , X z ,. . . , X,)
Convergence to the Poisson distribution
133
(In the original paper, the result appears without the factor of loge, but this is purely due to taking logarithms with respect to different bases in the definition of entropy). Notice that the first term in this bound measures the individual smallness of the X i , and the second term measures their dependence (as suggested at the start of the chapter). The original proof of Theorem 1 of [Kontoyiannis e t al., 20021 is based on an elementary use of a data-processing inequality. We provide an alternative proof, for independent variables and general Rknyi a-entropies, which will appear as Theorem 7.4. Given probabilities Fx(T)= P(X = T ) , where X has mean p , we will consider the ‘relative probability’ ~ x ( T = ) FX (T)/P@(T). We would like to be able to bound f x in La(P,) for 1 5 a 5 2, since the R6nyi relative entropy formula for probability measures (see Definition 1.9) gives
(7.16) Then:
This is the expression used in [Dembo et al., 19911 to establish the Entropy Power inequality, relying in turn on [Beckner, 19751 which establishes a sharp form of the Hausdorff-Young inequality, by using hypercontractivity (see Appendix D). We will use Theorem 1 of [Andersen, 19991, a version of which is reproduced here in the Poisson case: Lemma 7.4
F*(z)=
Consider functions F1 and F2 and define their convolution Fl(y)Fz(x- y ) . Then for any 1 5 (Y < 00:
c,=,
with equality if and only if constants q ,c2, a .
some
Fl(x)= PA(x)clear, P2(y)
=
PP(y)c2eay, for
Information Theory and the Central Limit Theorem
134
For any c, by Holder's inequality, if a' is the conjugate of a (that
Proof. is, if 1/a
+ 1/a' = 1):
(7.20)
Hence picking c = l/a' = ( a - 1)/a,taking both sides t o the a t h power and rearranging,
I F*(Y 1I a IPx+,(y)la-l< -
($
IFl(Z)l" P 2 ( Y - % ) l a IPx(x)l*-l P,(y - 5)-1
)
'
(7.22)
and summing over y, Equation (7.19) holds. Equality holds exactly when equality holds in Holder's inequality. Now, in general, C , If(z)g(z)l = I~(~)I~)'/*(c, Ig(z)Ia')l/a', if and only if 1g(z)1= Iclf(s)l*-',for some constant Ic. That is, in this case we require a constant Ic(y) such that for each x
(c,
Rearranging, for each x we need
Fl(Z)F2(?/ - x) = Ic(dPA(4pJY- XI, so (by the lemma on page 2647 of [Andersen, 19991)
FZ(y)= P,(y)c2eay, for some constants c1,c2, a .
(7.24)
Fl(x)= PX(x)qear, 0
Hence by taking logs, we deduce the bound on the Rknyi a-entropy:
Theorem 7.4 Given X, independent, with mean p,, define S, C;=, X , , and X = C;="=,,. Then for any 1 5 a < CQ, n
Da(SnllPO(X))
n
I ~ ~ a ( x , I I P o ( P ,I) )( 1 0 g e ) a C P : . t=l
=
(7.25)
z=1
Proof. The first inequality follows simply from Lemma 7.4. Note that we can express each summand as
135
Convergence to the Poisson distribution
so that (7.28) Now, we can expand this in a power series as
or alternatively bound it from above, using the fact that logx/loge (z - l),so that
I
(7.30) (7.31) (7.32) where g ( t ) = (1 - P ) ~ .Now, since g " ( t ) = (log,(l - p))'(l maximum of g ' ( t ) occurs a t the end of the range. That is
D"(XiIIPo(P)) log e
Ip +
g'((r)
=p
+ (1 - p)* log,(l
-
-
p)
(7.33) (7.34)
I P - P ( 1 - P)"
Ip(1 - (1 - CUP)) = ap2
P ) ~>_ 0, the
(7.35)
1
0
by the Bernoulli inequality.
Hence if Cr=lp:is small, we can control Da(SnIIPo(A)). In particular, letting a + 1, we recover the bounds from Theorem 7.3 in the independent case. However, note that the bounds of Lemma 7.3 and Theorem 7.4 are not of the right order; in the IID case, the bound on D(Bin(n,p)((Po(np))is O ( l / n ) , rather than the O(l/n') which is achieved in the following lemma. Lemma 7.5 The relative entropy distance f r o m a binomial r a n d o m variable to a P o i s s o n r a n d o m variable satisfies (7.36)
and hence i f p = X/n, the bound is c / n 2 , where c = (A3
+ A')
loge/2.
Information Theory and the Central Limit Theorem
136
Proof. We need to expand Equation (7.12) more carefully. We can use the fact that
log (( n -n!r)!nr
)
= log ((1 -
log e
A)
(1
-
T-1
a)
G))
. . . (1 r(r - 1)
(7.38)
2n Hence, Equation (7.12) becomes, for X
(
D(Bin(n,p)llPo(np))5 loge EX
-
N
(7.37)
Bin(n,p),
EX(X - 1’) 2n
+ (n - EX) log(1 - p), (7.39)
so, since EX = np and EX(X - 1) = n(n
D(Bin(n,p)(lPo(np))I loge (-(n-;)pz
-
l)p2,
+ n ( l -P)log,(l -P) + n P
1
.
(7.40) Since for 0 5 p 5 1; log,(I - p ) 5 ( - p
7.2 7.2.1
-
0
p 2 / 2 ) , the result follows.
Fisher information Standard Fisher information
Just as in the case of continuous random variables, we can also obtain bounds on the behaviour of Fisher information in the discrete case, using the definition given by [Johnstone and MacGibbon, 19871 and the theory of L2 projections. The first change is that we need to replace derivatives by finite differences in our equations.
Definition 7.1
For any function f : Z
+
R, define A f ( z )
=
+
f(z 1) -
f (XI. Definition 7.2 For X a random variable with probability mass function f define the score function p x ( x ) = A f ( x - l ) / f ( z ) = 1 - f ( z - l ) / f ( x ) and Fisher information
J ( X )= Epx(X)2 =
cf f X
(x (x)
-
1.
(7.41)
Convergence to the Poisson distribution
137
Example 7.2 If X is Poisson with distribution f ( x ) = e-'X2/x!, the score p x ( x ) = 1 - z/X, and so J ( X ) = E(1- 2X/X
+ X 2 / X 2 ) = 1/x.
(7.42)
We need t o take care at the lowest and highest point of the support of the random variable.
Lemma 7.6 If a random variable has bounded support, it has infinite Fisher information. Proof. We set f ( - l ) = 0, so that p(0) = 1. Note that if f ( n ) # 0 but f(n 1) = 0, then f(n l ) p ( n 1) = f(n 1) - f ( n ) = - f ( n ) . This implies that f ( n l ) p ( n = 00. 0
+
+ +
+
+
+
However, just as in the case of random variables with densities, where adding a normal perturbation smooths the density, by adding a small Poisson variable, the Fisher information will become finite. The score function and difference are defined in such a way as t o give an analogue of the Stein identity, Lemma 1.18.
Lemma 7.7 For a random variable X with score function p x and f o r any test function g , Epx(X)g(X) = - E A g ( X ) Proof.
= E g ( X ) - Eg(X
+ 1).
(7.43)
Using the conventions described above, the LHS equals
X
X
(7.44)
0
as required.
Using the Stein identity we can define a standardised version of Fisher information. Since Lemma 7.7 implies that E p x ( X ) ( a X 6) = -a then
E(px(X) - ( a x + b))'
+ = J ( X ) + a21EX2 + 2abp + b2 + 2a.
(7.45)
Now, this is minimised by taking a = -l/02, b = p / u 2 , (where p is the mean and oz is the variance) making the RHS equal to J ( X ) - 1/02.
Definition 7.3 For a random variable X with mean p and variance define the standardised Fisher information to be 1
2 0.
02,
(7.46)
Information Theory and the Central Limit Theorem
138
As in the case of normal convergence in Fisher information, we exploit the theory of L2 spaces, and the fact that score functions of sums are conditional expectations (projections) of the original score functions, t o understand the behaviour of the score function on convolution. Lemma 7.8 If X a n d Y are r a n d o m variables w i t h probability m a s s f u n c t i o n s fx a n d f y , t h e n
+
px+y(z)= E[px(X)IX Y Proof.
=
4 = IE[PY(Y)IX+ Y = 4.
(7.47)
For any x, by the definition of the convolution, (7.48)
= E[px(X)IX
+ Y = z ],
(7.50)
and likewise for Y by symmetry. Hence we deduce that: Proposition 7.1 any a:
For independent r a n d o m variables X and Y , a n d for
J(X
+ Y ) 5 a z J ( X )+ (1
-
a)’J(Y),
PX J s t ( X ) + PY J s t ( Y ) , where PX = a $ / ( a $ + o$) and PY = 1 PX = o $ / ( o $ + &). Jst
( X + Y )5
(7.51) (7.52)
-
Proof. The first result follows precisely as in the proof of Lemma 1.21. The second part is proved by choosing a = Px. 0 7.2.2
Scaled Fisher information
The fact that, for example, binomial distributions have infinite Fisher information J leads us to define an alternative expression K with better properties, as described in [Kontoyiannis et al., 20021. Definition 7.4 Given a random variable X with probability mass function P and mean A, define the scaled score function
(7.53)
Convergence t o t h e Poisson distribution
139
and scaled Fisher information
K ( X )= x
1P(x)(p,(x))2
= xEy,(x)2
(7.54)
2
We offer several motivations for this definition. (1) Clearly, by definition,
K(X) 2 0
(7.55)
with equality iff p x ( X ) = 0 with probability 1, that is, when P ( z ) = ( X / x ) P ( x- l),so X is Poisson. (2) Another way to think of it is that the Poisson distribution is characterised via its falling moments E ( x ) k , where ( 5 ) k is the falling factorial z ! / ( x- k ) ! . The Poisson(X) distribution has the property that E ( X ) k = x k . w e can understand this using generating functions, since
where M x ( s ) = Esx is the moment generating function of X . Hence for g(z) = ( z ) k , 1
Epx(x)g(x)= X E ( X ) k + l - E ( x ) k ,
(7.57)
so the closer X is to Poisson, the closer this term will be to zero. (3) In the language of [Reinert, 19981, we can give an equivalent to (7.43), and interpret E p , ( X ) g ( X ) as follows. Define the X-size biased distribution X* to have probabilities
P ( X * = x) =
x P ( X = x) A
.
(7.58)
Then for any test function g,
= Eg(X* -
1) - E g ( X ) .
(7.60)
Stein’s method, described in Chapter 1, extends to establish convergence to the Poisson distribution. The equivalent of (1.97) holds here too, in that E(Xg(2
+ 1) - Z g ( 2 ) ) = 0
(7.61)
140
Information Theory and the Central Limit Theorem
for all g if and only if 2 is Poisson(X). Again, the trick is to express a given function f as Xg(r 1 ) - r g ( r ) ,and to try to show that (7.61) is small for a given random variable. The papers [Barbour et al., 19921 and [Reinert, 19981 give more details. Proposition 3 of [Kontoyiannis et al., 20021 tells us that K is subadditive:
+
Theorem 7.5 If Sn = Cy=,X i is the s u m of n independent integervalued random variables X I ,X a , . . . , X,, with means IE(Xi) = pi and X = C?=,pi then
c n
K(Sn)5
(7.62)
FK(X2).
i=l
Thus, we can deduce convergence of Fisher information for the triangular array of Bernoulli variables. If the X i are independent Bernoulli(pi) random variables with pi = A, then K ( X a )= p : / ( l - p i ) and Theorem 7.5 gives
c:=,
(7.63) In the IID case, this bound is O ( l / n 2 )which , as we shall see in Section 7.3 is the right order. Theorem 7.5 is implied by Lemma 3 of [Kontoyiannis et al., 20021, which gives:
Lemma 7.9 If X and Y are random variables with probability distributions P and Q and means p and q , respectively, then PX+Y(Z) =
where Qx = P / ( P
+ QYPY(Y)I x + y =
E[Qxpx(X)
21,
(7.64)
+ 41, QY = q / ( p + s)
Then, as with Lemma 2.1, this conditional expectation representation will imply the required subadditivity. 7.2.3
Dependent variables
+
We can also consider bounding K ( X Y ) in the case where X and Y are dependent, in the spirit of Chapter 4. Here too a subadditive inequality with error terms will hold (the equivalent of Proposition 4.2) and is proved using a conditional expectation representation (the equivalent of Lemma 4.2).
Convergence to the Poisson distribution
141
As before, for joint probabilities p$!y(z,y) will be the score function with respect to the first argument, that is: (7.65) Lemma 7.10 If X , Y are random variables, with joint probability mass function p ( z ,y), and score functions p$iY and p$iy then X Y has score function given by
+
p x + y ( z )= E
where ax
=p/(p
[ax pyy(x,Y) + UYP$;,(x,Y)I
x +y = z]
(7.66)
1
+ q ) and a y = q / ( p + q )
Proof. Since X + Y has probability mass function F , where F ( z ) Ef=,P ( x ,z - x), then we have 2+1
-
Px+y(z) =
x=o z+l
(z
+ l)P(z,z
-2
+ 1)
-1 +q)F(z) z P ( x ,z z + 1) (2 z + l)P(z,z + -
z+l
+ay
2 x=o
-
z
+ 1)
(P + q ) F ( z )
x=n
-
(7.67)
(p
-
[
x=n
z P ( x ,z - x
+ 1)
P ( z - 1,z
-z
x
-
+ l ) P ( x ,z - z + 1) P ( x ,z q P ( x ,z - ).
[
P ( x ,z - x) (ax (z F(z) +QY
as required.
[
-
+ 1)
F(z) (2 -
=
-
x)
-
1 (7.68)
I' (7.69)
11
F(z)
+ 1)P(x+ 1,z - x)
(2 - z
PP(Z,z
).
-
+ 1)P(z,z q P ( z ,z
-
).
-
-
z
+ 1) -
11)
1
(7.70)
0
Define the function M ( z ,y ) by
which is zero if X and Y are independent. As in Chapter 4, we deduce that:
Information T h e o y and the Central Lamat Theorem
142
Proposition 7.2
If X , Y are dependent random variables then
Hence in order t o prove a Central Limit Theorem, we would need t o bound the cross-terms, that is, the terms of the form:
Note that if the Xi have a Bernoulli ( p i ) distribution, where for all i, pi 5 p* for some p * , then Ipx, I is uniformly bounded for all i, and hence so is Ips,, I for all n.
7.3
Strength of bounds
If we formally expand the logarithm in the integrand of the de Bruijn identity of the next section (see Equation (7.89)) in a Taylor series, then the first term in the expansion (the quadratic term) turns out to be equal to K ( X t ) / Z ( X t ) . Therefore,
+
D(PIIPo(X)) Fz -
Dc)
K(X
+ Po(t))dt
X+t
1
(7.74)
a formula relating scaled Fisher information and entropy. Now, Theorem 7.5 implies that
K(X
X + Po(t)) i --K(X) X+t
t + ----K(Po(t)) X+t
X
= -K(X). X+t
(7.75)
Hence the RHS of Equation (7.74) becomes (7.76) In fact, this intuition can be made rigorous. Proposition 3 of [Kontoyiannis et al., 20021 gives a direct relationship between relative entropy and scaled Fisher information. That is:
Theorem 7.6
For a random variable X with probability mass function
Convergence to the Poisson distribution
143
P and with support that is a n interval (possibly infinite) then
m
Proof. This result is proved using a log-Sobolev inequality, Corollary 4 of [Bobkov and Ledoux, 19981. The key observation is that we can give an expression for K that is very similar to the definition of the Fisher information distance, Definition 1.13. That is, writing f (z) = P(z)/Px(z), then
=
P ( z + 1) Px(z+ 1)
--P ( z )
PX(2)
-
-( Ph(2) 1
P(z
+ l ) ( z + 1) x
-
P(,))
(7.79) (7.80)
This means that
which is precisely the quantity that is considered in [Bobkov and Ledoux, 19981. The second result follows simply by Lemma 1.8. 0 Note the similarity between Equation (7.81) and the Fisher information distance of Definition 1.13. There, given random variables with densities p and q , writing f(z) = p ( z ) / q ( z ) ,
Hence
(7.84)
Information Theory and the Central Limit Theorem
144
7.4
De Bruijn identity
Recall that in the case of the Gaussian, relative entropy can be expressed as an integral of Fisher information, using the de Bruijn identity, Theorem C.l. Hence we can prove convergence in relative entropy by proving convergence in Fisher information. In the discrete case, a de Bruijn-style identity holds, so we can still obtain useful bounds on relative entropy.
Lemma 7.11 For a random variable 2 , write qx for the probability mass function of Z+Po(X), where Po(X) is a Poisson(X) independent of 2. The qx satisfy a differential-difference equation: 1) -
Proof. As in the Gaussian case, first we observe that the Poisson probabilities px satisfy a similar expression: (7.86) and hence by linearity: (7.87) =
-
Y)(PA(Y
-
1) - P A ( ! / ) )
= 4x(x -
1) - 4x(x),
(7.88)
Y
0
as required. This implies that
Proposition 7.3 For any integer-valued random variable X with distribution P and mean A, define Pt to be the distribution of the random variable X t = X + P o ( t ) where Po(t) is an independent Poisson(t) random variable. Further let pt(r) = ( r l ) P ( X t = r l ) / ( X t ) . Then ZfX is the sum of Bernoulli variables
+
+
+
(7.89)
Proof.
Expanding, for any a , by the fundamental theorem of calculus,
D(PIIPo(X))- D(PaJIPo(X+ a ) ) =
-la
-&D(PtlIPo(X
+ t))dt.
(7.90)
Convergence to the Poisson distribution
145
Now, we can use Theorem 7.6, since it implies that
using Theorem 7.5. Since K(X) is finite, the lima+m D(P,IIPo(X+a)) Hence
=
=
(7.94)
1 + -lm it + -
=
" d -D(P,IIPO(X dt - ((A
(log(X
t)
+t)
-
-
= 0.
t))dt
(7.95)
E[Xt bg(X
d -lE[log(Xt!)] at
+ t)] + IE[log(Xt!)]
-
H(Xt))dt (7.96)
"
+ --H(X,) dt
dt.
(7.97)
We can deal with each term separately, using Equation (7.88) and the Stein identity:
d -IElog(Xt)! dt
=
c
=
CMr
8% -(r)logr!
(7.98)
at
-
1) - qt(r))log T !
(7.99)
T
=
Cqt(r)log((r
+ 1)!/r!) = IElog(Xt + 1).
(7.100)
T
Next,
(7.103) Putting it all together, the result follows.
0
Information Theory and the Central Limit Theorem
146
7.5
L2 bounds on Poisson distance
7.5.1
L2 definitions
We can achieve tighter bounds in the L2 case, exploiting the theory of projection spaces and Poisson-Charlier polynomials, which are orthogonal with respect to the Poisson measure (see for example [Szego, 19581):
Definition 7.5
Define the Poisson-Charlier polynomials
(7.104) where ( z ) j is the falling factorial x!/(z- j ) ! . Note that they form an orthogonal set with respect to Poisson weights:
c
P p ( x ) C k ( I I : ; p)cl("; p ) = PL-ICk!6kl,
(7.105)
2
and have generating function:
(7.106)
Lemma 7.12 Given independent random variables X and Y with means p and A, write f x for the relative density of X with respect to the Po(p) distribution, and f y f o r the relative density of Y with respect to the Po(A) distribution. T h e n if a = p/(X p ) ,
+
fX+Y(T)
(7.107)
= La[fx,f Y I ( T ) ,
where L , is a bilinear m a p defined by 5).
Proof.
(7.108)
We can expand the convolution probability:
(7.109)
(7.110)
Convergence to the Poisson distribution
147
and rearranging, the result follows.
0
Now a knowledge of the relative probability of independent X and a simple expression for the relative probability of X Y :
+
Y gives
Proposition 7.4 Consider independent X and Y with means p and A. Assume that C , f i ( x ) P p ( x ) = C x F $ ( x ) / P p ( x ) < cc (and similarly for Y).Expanding fx(x) = u c ) c k ( x ; p )and fy(2) = z k '$)ck(x; A) then
xk
fX+Y(x)= c
k
u F ! + y c k ( ZI-1 ;
+ A),
where k
u$wy
=
C,(J)(k-d .
(7.112)
XUY
j=O
Proof. First we show that the Poisson-Charlier polynomials are wellbehaved with respect t o the map L , with La[Ck(.,
Cl(.,
A)] = C k + l (.i p
+ A).
(7.113)
This can be proved using generating functions,
- ,-t-s -
e-t-s
( l + at p - a ) (1
+
s) Y
xs ) y
(7.116)
'
(7.117)
so the result follows by comparing coefficients. Hence Equation (7.112) follows from Equation (7.113) using the bilinearity of L . 0 7.5.2
Sums of Bernoulli variables
For X a Bernoulli variable, we can read off the coefficients
very easily.
Lemma 7.13 If random variable X is distributed as Bernoulli(p) then we can expand fx(x) = X k u $ ) c k ( x ; p ) , with u$)= (-l)k(l - k ) p k / k ! .
Information Theory and the Central Limit Theorem
148
Note that for any 1:
Proof.
(7.118) An alternative expression for the same term is
Now, since X takes values 0 and 1, we know that 1 for Ic
=0
(7.120)
0 otherwise. Substituting this into the definition of the Poisson-Charlier polynomials, 0 Definition 7.5, and recalling that p = p the result follows. Now combining Proposition 7.4 and Lemma 7.13, we can bound the first few coefficients for the sums of Bernoulli variables. Suppose we have X, independent Bernoulli(p,) random variables and S = X, is their sum. Write X = Cr', p, and = Cr=lp:/X. From Lemma 7.13, we know that
c:=l
We use results based on those described in Chapter 2 of [Macdonald, 19951. First, we define the set of partitions
A,
=
(1 : 11
+ 12 + . . . + 1,
= n}.
(7.122)
We write mi to be the number of parts of the partition that equal i. Let T, be the power sum pi" p$ . . . p:. The key lemma will give us that:
+ +
Lemma 7.14
For any n:
that is, we s u m over partitions of n which have n o part equal t o 1
Proof. By Proposition 7.4 and Lemma 7.13, the generating function of the us is k
2
Convergence to the Poisson distribution
149
for Ej the j t h symmetric polynomial. Now (2.14') of [Macdonald, 19951 shows that we can express (7.125)
We simply adapt the argument of [Macdonald, 19951, by considering the 0 generating function P * ( t )= P ( t ) - At = Cr>2~rtr-1. Thus, for example, the only partition of 3 with no part equal to 1 is the trivial one, so Lemma 7.14 gives
the partitions of 4 are 4
=2
+ 2 and 4 = 4, so (7.127)
Now, by convexity
(C:=lpT)'
5
(C:=lP : ) ~ ,for m 2 2. Hence (7.128)
and
Hence, expanding we obtain (7.130)
(uk)) p = 1 + - p12 + - p2-3 + s3-4 2 3
+...
We need to bound the tail of this expansion
2
24
2
+ x4 (up)) + . . .
(7.131) (7.132)
Information Theory and the Central Limit Theorem
150
7.5.3
Normal convergence
Now, note that Proposition 7.4 gives precisely the same relation that holds on taking convolutions in the normal case. Define a particular scaling of the Hermite polynomials (see Definition 2.1) by K T ( z ; p )= H T ( z ; p ) / p T . Then
Definition 7.6
Let K T ( x p ; ) be given by the generating sequence
(7.133)
Then
In the same way as in the Poisson case, given densities Fx,where X has mean 0 and variance p , we can define fx(z)= F X ( X ) / ~ , ( Z )Then, . the equivalent of Proposition 7.4 is:
Proposition 7.5 Consider independent X and Y with mean 0 and variances p and A. If we can expand f x ( x ) = z k u j u k ) K k ( x ; p ) and fy(z)= C ku f ) K ~ , (A)z ; then fx+y(z)= ujukiyKk(z; p A), where
Ck
+
(7.135)
Proof.
Again, we can consider the definition of fx+y:
(7.136)
Convergence to the Poisson dzstributzon
151
Now, we can follow the effect of L on these Hermite polynomials, once again using generating functions.
The result follows on comparing coefficients.
0
This page intentionally left blank
Chapter 8
Free Random Variables
Summary In this chapter we discuss free probability, a particular model of non-commutative random variables. We show how free analogues of entropy and Fisher information can be defined, and how some of our previous methods and techniques will apply in this case. This offers an alternative view of the free Central Limit Theorem, avoiding the so-called R-transform.
8.1 8.1.1
Introduction to free variables Operators and algebras
Voiculescu has developed the theory of free probability, originally as a means of understanding the properties of operator algebras, but which more recently has been seen as a way to understand the behaviour of large random matrices. At the heart of the theory of free random variables lies the free convolution BJ, which allows a free Central Limit Theorem, Theorem 8.2, to be proved, with the Wigner (semicircular) distribution as the limit. The free convolution is usually understood through the R-transform, an analogue of the logarithm of the classical Fourier transform, in that i t linearises convolution (see Theorem 8.1). We offer a different approach, motivated by the properties of Fisher information described earlier in the book. We give the equivalent of Brown’s Lemma 2.5 for free pairs of semicircular random variables, and (as in Chapter 2 and [Johnson and Barron, 20031) show how similar results hold for more general pairs of free X , Y . This allows us to deduce a rate of convergence in Fisher information and relative entropy. We first describe some results from the theory of free probability. For 153
Informatzon Theory and the Central Limit Theorem
154
more details, see Voiculescu, Dykema and Nica [Voiculescu e t al., 19921. Biane, in two papers [Biane, 19971 and [Biane, 19981,provides an introduction to free probability with a more probabilistic flavour. Free probability is a theory where random variables do not commute with one another. We therefore choose t o associate a random variable X with an operator fx acting on a Hilbert space of countable dimension. Formally, the collection of such operators is known as a von Neumann algebra (a set of bounded operators, closed under addition, multiplication, scaling, taking adjoints and under the action of measurable functions f ) . We think of this von Neumann algebra A as the whole ‘universe’ of possible random variables, and require that it contains the ‘unit’ or identity L. This is the equivalent of the classical case, where if X and Y are random variables, then so are X Y ,X U , aX and f ( X ) . We will often need to consider subalgebras B (which are subsets of A containing the unit), and their extensions B [ X ] (the set of functions of operator fx with coefficients in B , such as fx,7fi and so on). Often, we will think of B as being just @, the constant maps.
+
8.1.2
Expectations and Cauchy transforms
We associate expectations with a particular fixed linear map r in this von Neumann algebra. The map r is normalised such that the identity L has T ( L ) = 1. Definition 8.1 Consider a random variable X with corresponding operator f x . For any function P , define the expectation of P ( X ) to be r ( P (fx)). Further, we can associate the random variable with a measure p x , such that IEP(X) = r ( P ( f x ) )= J P ( t ) d p x ( t ) for , all functions P . If this measure px is absolutely continuous with respect to Lebesgue measure on the reals, then as usual we can refer to the density of the random variable. Example 8.1 A simple (but motivating) case is where the algebra is n-dimensional (and so fx is a n x n matrix) and where the map r(f) = t r (f)/n.Then p will be the empirical distribution of the eigenvalues.
Suppose n = 2, and fx =
(l i’).
Then
that is, there exists a matrix a such that fx
fx is a diagonalisable matrix; =
a d a - l , where d
=
(::).
B e e random variables
155
For P ( t ) = t k , k a positive integer we have 1
E P ( x )= ~ ( P ( f x=)5tr ) ((ada-l)
hence the measure p x is
(62
k
) = -tr 2 1
(adka-l)
(8.1)
+ 63)/2.
It is possible to prove a free analogue of the Central Limit Theorem, Theorem 8.2. Unfortunately, existing proofs require the use of the R-transform defined below in Definition 8.4, which we would like t o avoid. In the free setting, the following random variable plays the role of the Gaussian in the commutative case:
Example 8.2 The Wigner or semi-circular random variable with mean and variance u2 will be denoted as W ( p ,u 2 ) and has density
It is natural to ask which operator this Wigner random variable corresponds to. In fact, it corresponds to c = (u+ u T ) a / 2 , where a is a shift operator and oT is the transpose. This can be understood by seeing the operator in question as the limit of n x n matrices c , = (a, aT)a/2. Then, the characteristic polynomial,
+
is a multiple of the Chebyshev polynomial of the second type U , (see for example [Weisstein, 20031). Now, since Un(x)= sin(n l)#/sin#, where 2 = (20) cos 4, the zeroes occur at 4 k = k.ir/(n 1) for k = 1. . . n,so under the inverse cosine transformation, we can see the eigenvalues will be X k = (20) c0s-l ( k r / ( n +1))and the empirical distribution of the eigenvalues will converge t o the semicircle law.
+
+
156
Information Theory and the Central Limit Theorem
Lemma 8.1 For W a Wigner W ( 0 ,u2)random variable, we can evaluate the m o m e n t s of W .
Proof.
By definition x'2
xTdx =
( 2 0 cos 6')2
(2a sin O)TdO (8.6)
(using x = 2a sin 6') , where
I,,, TI2
IT=
sinTQdO =
(
r;2)
II(r even) 2T
In the same way, we can find a set of orthogonal functions with respect to the Wigner law.
Lemma 8.2 Define the Chebyshev polynomials as the expansion in powers of 2 cos 6' of Pi(2cos Q ) = sin(k + l)8/ sin 6'. N o w further define (8.9)
T h e n the first f e w polynomials are PE'uz(x) = 1, Pf'uz(x) = (x - p ) / a 2 , P['"'(x) = ((x - p)' - .'))/a4 and the P[>uz form a n orthogonal set with respect to the Wigner W ( p ,a') law: (8.10)
Proof.
By changing variables so that x = p
+ 2acos6, we deduce that
dx = -2a sin 0 and 2,
I,,S J 5 Lx + =[
1
4a2 - (x - p)2P,"~"2(x)P~"Z(x)dx
--
=
= &nn,
(sin Q)'
sin(n
+
+
(8.11)
sin(n 1)O sin(m l)6' d6' sin 6' sin 0
(8.12)
+ l)d6'
(8.13)
l)Qsin(m
(8.14)
Free random variables
157
0
proving orthogonality.
The Cauchy transform is an important tool in the study of free random variables.
Definition 8.2 Let @+ denote the open upper half plane of the complex plane. Given a finite positive measure p on R,its Cauchy transform is defined for z E @+ by (8.15) where the integral is taken in the sense of the Cauchy principal value.
Lemma 8.3 (1) For W a deterministic random variable with point mass at a , G ( z ) = 1/(2 - a ) . (2) For W Wagner W ( 0 , u 2 ) with density wO,+, G ( z ) = ( z J 2 3 2 ) / 2 2 .
Proof.
The result for the Wigner variables is built on a formal expansion
in powers of l / z . Note that we require E(z - W ) - l = E(z(1 - W / z ) ) - ' = E CZO W T / z ' +' . Using the expansion in (8.8), this sum becomes
(8.16)
(8.18) (8.19) using the fact that for r
< 1/4, (8.20)
Lemma 8.4
Some properties of the Cauchy transform are as follows:
(1) For any p the Cauchy transform G,(z) is a n analytic function from the upper half-plane @+ to the lower half-plane @- .
Information Theory and the Central Limit Theorem
158
(2) For any set B:
P(B) = im
/
-1 ~
Im G ( x
+ic)I(x E B)ds,
(8.21)
so i f p has density p then -1
p(x) = - lim Im G ( x + zc).
(8.22)
77 c+o+
8.1.3
Free interaction
The property of independence in classical probability theory is replaced by the property of freeness. We first define the property of freeness for algebras, and then define random variables t o be free if they lie in free algebras.
Definition 8.3
A collection of algebras B, is said to be free if T(UIU2..
.a,)
(8.23)
= 0,
whenever r ( u i ) = 0 for all i , and where u3 E Bil with i l # 22, 23 # i 4 . . .. Hence random variables X I , . . . X , are free if the algebras Bi which they generate are free.
22
=
#
23,
B[X,]
Given two free random variables X and Y , we use Definition 8.3 to calculate E X t 1 Y ~ 1 X i 2. . . Y j - for sequences of integers ( i T ,j s ) . We construct new random variables X = X - p x , 7= Y - p y , where p x is the mean EX. The structure proves to be much richer than in the commutative case , where E X i l Y J I X i Z , ,, y j r . = E X c , z-Yc,js = ( E X c , 2 , ) ( E Y C . y J s ) for independent X and Y .
+
Example 8.3 For X and Y free: E ( X Y ) = E(X p x ) ( Y E ( W X p y p x y p x p y ) = p x p y , using Definition 8.3.
+
+
+
+ py)
=
This is the simplest example, and agrees with the classical case, despite the non-commutative behaviour. However, for even slightly more complicated moments, calculations quickly become very complicated.
Example 8.4
For X and Y free:
i E ( X Y X Y ) = E(X
+px)(Y+ P Y ) ( T+P X H Y + PY).
(8.24)
159
Free random variables
Expanding this out, we obtain the sum of: (8.25) (8.26) (8.27) (8.28) (8.29) (8.30) (8.31) (8.32) (8.33) (8.34) (8.35) (8.36)
(8.37) (8.38) (8.39) (8.40)
(x)'
-
since = x2+ (IE(X2)- p $ ) - 2 p x X , where % = X 2 - IE(X2). Overall then, summing Equations (8.25) to (8.40) we obtain E ( X Y X Y ) = E(X2)(lEY)2+ E(Y2)(EX)2- (EX)2(EY)2.
(8.41)
Hence it becomes clear that given free X and Y we can, in theory at least, calculate IE(X + Y)", and that this quantity will be determined by the first n moments of X and Y. Thus, given two measures p~ and p y , we can in theory calculate their free convolution p x H py , the measure corresponding to X and Y .The fact that we can do this shows that freeness is a powerful condition; in general, just knowing the eigenvalues of matrices A and B will not allow us to calculate the eigenvalues of A B. However, Example 8.4 indicates that calculating E ( X + Y)" will be a non-trivial exercise for large n and its dependence on the moments of X
+
Information Theory and the Central Limit Theorem
160
and Y will be via some highly non-linear polynomials. In the theory of free probability, this is usually summarised by the so-called R-transform, which plays the part of the logarithm of the characteristic function in classical commutative probability. (An alternative approach comes in [Speicher, 19941 which characterises these moments via the combinatorics of so-called non-crossing partitions). Having defined the Cauchy transform, we can define the R-transform:
Definition 8.4 If measure p has Cauchy transform G,(z), then define K,(z) for the formal inverse of G, ( z ) . Then, the R-transform of p is defined to be 1 (8.42) R,(z) = K,(z) - -. z Example 8.5 (1) For W a deterministic random variable with point mass at a , G ( z ) = l / ( z - a ) , so K ( z ) = 1/z a and R ( z ) = a . ( 2 ) For W Wigner W ( 0 , u 2 ) with density W O , ~ Z , G ( z ) = ( z d-)/2cr2, so that K ( z ) = u 2 z 1/z and R ( z ) = u 2 z .
+
+
Interest in the R-transform is motivated by the following theorem, first established in [Maassen, 19921 for bounded variables and then extended to the general case in [Bercovici and Voiculescu, 19931.
Theorem 8.1 If measures 4-11 and p2 are free, with R-transforms R1 and R2 respectively, then their s u m has a measure p1 H p2 with R-transform R1m2 given by
R i w z ( 2 ) = Ri(z) + R z ( z ) .
(8.43)
Hence, for example, if p1 is a Wigner (0,a:) measure and pa a Wigner ( 0 , ~ ; )measure, their R-transforms are u : z , and the sum p1 €El pa is a Wigner (0, uf a;) variable. Theorem 8.1 shows that the R-transform is the equivalent of the logarithm of the Fourier transform in commutative probability. Since G, is a Laurent series in z , with the coefficients determined by the moments of p , [Biane, 19981 shows how R can formally be expanded as a power series in z . In particular, for X with mean zero and variance u 2 ,the R-transform R ( z ) N u 2 z O ( z 2 ) .Hence, since the normalised sum of n free copies of X will have R-transform J;ER(z/J;E), as n 4 co, the R-transform of the sum tends to u 2 z . This enables us to deduce that the
+
+
Free random variables
161
Wigner law is the limiting distribution in the free Central Limit Theorem, established in [Voiculescu, 19851.
Theorem 8.2 Let X I , X 2 , . . . be free random variables with mean zero, variance u2 and measure p . Then the scaled free convolution:
x1 + x2 + . . . x , fi
4
W(0,2).
(8.44)
The theory of free probability therefore serves t o explain the observation, dating back t o Wigner at least (see for example [Wigner, 19551, [Wigner, 19581) that large random matrices (for example symmetric real matrices M , with Mij = Mji N ( 0 , a 2 / n )for 1 5 i 5 j 5 n) have the Wigner law as the limiting (as n 4 co)empirical distribution of their eigenvalues. This can be understood since Theorems 2.2 and 2.3 of [Voiculescu, 19911 show that large random matrices are asymtotically free. N
8.2
8.2.1
Derivations and conjugate functions
Derivations
We offer an alternative view of the free Central Limit Theorem, via the conjugate function (which is the free equivalent of the score function). To introduce the idea of a conjugate function, we first need the idea of a derivation (the free equivalent of the derivative), introduced in [Voiculescu, 19981:
Definition 8.5 If B and X are algebraically free, there exists a unique derivation 6 ’ :~B[X] + B[X] @ B[X] such that: (1) ax(b)= 0 if b E B. (2) &(X) = 1 8 1. ( 3 ) dx(m1m2) = a x ( m l ) ( l @m2) Further, define
+ (1 8 m1)dx(m2).
: @[XIH CIX]@‘(”+l) by
[in the case of polynomials over
6’g)= (ax@ I@(n-1))6’
C think of 6’f(s,t ) = ( f ( s )- f ( t ) ) / ( s- t ) ] .
<
Definition 8.6 An element is called the pth order conjugate function of X with respect t o C if for all m E @[XI: T(<m)= T@(p+1)(8?)m).
We denote this element by
JLxl.
(8.45)
Information Theory and the Central Limit Theorem
162
The classical Stein identity, Lemma 1.18, states that for all m
W x ( X ) m ( X )= ) -Em'(X),
(8.46)
so in the classical case, up t o a sign the score function plays the role of the 1st order conjugate variable, so we shall write px for J ,I11. In what follows, we will make use of the following result (proved as the simplest case of Lemma 3.7 of [Voiculescu, 19981). It is precisely the equivalent of the result Lemma 1.20 which holds in the commutative case.
Lemma 8.5 The score function of the free sum can be expressed as a conditional expectation of score functions: (8.47)
PX+Y = IEB[X+Y]PX= IEBIX+Y]PY.
Proof. Just as in the commutative case, where the key observation is that for f a function of x + y, (8.48) in the free case, the derivations 6x and
6x+y
satisfy
+
where a is a function of X Y. This can be proved for a(.) = U" using induction, and the properties of Definition 8.5. That is, for n = 0 and 1, the first and second properties ensure equality. After that, the third property means that by the inductive hypothesis: 8X+Y (
X
+ Y)"
(8.50)
((x+ Y)"-'(x + Y)) = a x + y ( x + y)"-l(iB ( x+ Y))
(8.51)
=
+ (1 (x+ y)"--l)a x + y ( x + Y) = dx(X + Y)"-'(lB ( X + Y ) ) + (1 @ (X + Y)"-') = ax ( ( x+ Y ) " - ~ ( x + Y)) = ax(x+ Y)".
(8.52)
dx(X
+ Y)
Then, the proof concludes since this implies that for any m ( X 7@'(2)(dx+ym =) W ) ( d x m )= 7(pxm),
so taking
px+y = ~
+
(8.53) (8.54)
+Y): (8.55)
( p x l X Y) means we have the required property.
0
Free random variables
163
An alternative approach t o defining a free equivalent of the score function comes via the real part of the Cauchy transform.
Definition 8.7
Define the Hilbert transform: -1
( H p ) ( z )= - lim Re G(z + 2 6 ) . n t+O+ Example 8.6 For W Wigner W ( 0 ,g2) with density W Hp(z) = x/(27r2).
(8.56) O , ~for ~ ,
1x1 5 2a2:
Now Proposition 3.5 of [Voiculescu, 19971 proves that
Lemma 8.6 If B = C and and X has a measure which is absolutely continuous with density p E L3 then JI”’ is 2nHp. Proof.
(We reproduce Voiculescu’s proof for the sake of completeness.)
and the exchange of integral and limit is justified in [Voiculescu, 19971. 0
8.2.2
Fisher information and entropy
We can define the free analogue of the Fisher information (note that some authors define the quantity with a different multiplicative constant at the front).
Definition 8.8 For a random variable X with density p , we define the free Fisher information of X t o be
@(X)=
s
p 3 ( s ) d s= 3
(8.61)
164
Information Theory and the Central Limit Theorem
(The equivalence of the two definitions follows by Lemma 8.4(2): integrating G3(2)around the semicircle of radius R centred on i f , we deduce that
-1 . 0 = - lim Im 7r3
t+O+
L 00
G3(x+ i f ) d x =
s
( ~ ( x- )3 p~ ( x ) ( H p ( ~ ) )dx, ~) (8.62)
see Lemma 3.3 of [Voiculescu, 19931 for details.) The preceding discussion shows that we should consider the Hilbert transform as a free analogue of the score function, firstly because of its role in the definition of the Fisher information, and secondly because Example 8.7 shows that it is linear for the limiting Wigner distribution, giving a Cram&-Rao lower bound (see Lemma 1.19).
Lemma 8.7
For a random variable X with density p , mean 0 and varz-
ance g 2 :
(8.63) with equality i f and only i f ( H p ) ( x )= x/(27rg2)for all x where p ( x ) # 0 .
Proof. Since 27r(Hp) is JiX1, and T ( X J : ” ’ ( X ) ) = T @ ( ~ ) ( = ~ X1,)we know that 27r J p ( x ) z ( H p ( z ) ) d x= 1. Hence: (8.64)
(8.66)
0 In a series of papers [Voiculescu, 19931, [Voiculescu, 19941, [Voiculescu, 19971, [Voiculescu, 19981, Voiculescu develops definitions for a free analogue of the entropy. In [Voiculescu, 19931, he gives a motivation for this particular definition of the entropy, similar to that in Section 1.2, based on random matrix heuristics. That is, approximate variable X by a function of a large Gaussian random matrix X = p ( Y n ) . Since Y, will have eigenvalues drawn from the Wigner law, we can calculate the distribution of p ( Y n ) . Definition 8.9 For a random variable X with density p , we define the free entropy of X to be (8.67)
Free random variables
165
This expression turns out to be non-trivial to evaluate, and will require results from the theory of logarithmic potentials. For example, pages 195197 of [Hiai and Petz, 20001 show how to calculate the free entropy of the semicircular law. The key identity, Equation (5.3.6), in their calculation actually comes from Section IV.5 of [Saff and Totik, 19971. It states that for p a Wigner W ( 0 ,g2)density 22
p ( y ) log ) x - yldy = 4a2
+ -21 loga2 - -21’
for 1x1 I 2 u .
(8.68)
This means that
Example 8.7
For X a Wigner W ( 0 , a 2 )random variable, 1 1 C ( X )= - l o g 2 - 2 4’
(8.69)
Remark 8.1 Further, as with the Gaussian and Poisson variable, the Wigner law satisfies a m a x i m u m entropy property. T h a t is, as page 197 of [Hiai and Petz, 2UUU] states, for all random variables Y with variance c2,
C ( Y ) 5 C(W),
(8.70)
where W is a W ( 0 , a 2 ) variable. ) Thus, motivated by this, we could define the relative entropy distance f r o m a Wigner law as
where Var W
= Var
Y.
It is also encouraging for our purposes that an analogue of the de Bruijn identity (see Appendix C) exists as Lemma 3.2 of [Voiculescu, 19931. Thus, as in the classical case, there is a link between the entropy and Fisher informat ion. There does seem to be some disagreement concerning the correct constant to place in this identity.
Lemma 8.8 Consider Y ,Wt a free pair of random variables such that Y is compactly supported o n Iw with m e a n U and variance 1, and Wt is a m e a n 0, variance t Wigner measure. T h e n C ( Y + WT) - C ( Y ) =
2ri2FI’
+
@(Y W t ) d t .
(8.72)
Information Theory and the Central Limit Theorem
166
Proof. Involves the fact that G satisfies the Burger equation, which is the analogue of the fact that the normal density satisfies the heat equation, Lemma C.l. We can argue that the right constant of proportionality to use is c = 27r210ge/3. This follows since for Y N W(O,l),by (8.69), C ( Y W T )0 C ( Y ) = log(1 T ) / 2 ,and by (8.63), @(Y W t )= 3/(4r2(1 t ) ) .
+
+
+
+
Now, much of the structure of Barron's entropy-theoretic proof of the Central Limit Theorem [Barron, 19861 is in place. Instead of considering the entropy directly, we consider the Fisher information. The characterisation of the Wigner law that we shall use is that it makes the Hilbert transform linear, which corresponds to its minimum value. We need to show that taking convolutions makes the Hilbert transform 'more linear' in some sense.
8.3
Projection inequalities
We can control Fisher information by using projection inequalities in the spirit of those of Chapters 2 and 3. Definition 8.10
Define the derivative V y to be
Note that this is the same as the A introduced in Section 6 of [Dembo et al., 20031. Definition 8.11 constant R if
We say that the random variable Y has free Poincark
for all functions g E L 2 ( Y ) .
Example 8.8 The Wigner W ( 0 ,c 2 )distribution has free Poincark constant c2 (since for each Chebyshev polynomial V P ~ + " ( x=) P:flz(x)/c2, so that we can expand both sides in the orthogonal polynomials). Another property that we shall require comes from Proposition 2.3 of [Voiculescu, 20001:
Free random variables
167
Consider X and Y free random variables, then o n the space
Lemma 8.9 B[X Y]
+
(TY
(8.75)
(23 7 Y ) 0 dX+Y = & F Y .
For example, consider X and Y of mean 0. Then taking f ( u ) = u2:
= (TY @ T Y ) (1@ =
(1@ x
+
Y) ( X + Y ) ( X + Y )@ 1)
(TY @ 7 Y ) 0 d , Y + Y f ( X
+
+ x @ 1)
(8.76) (8.77)
and
d x ~ f y( X
+ Y ) = d x ( X 2+
C)
= (1 @
X
+X
@
1) ,
(8.78)
where constant c
= .(Y2).
Lemma 8.10 p = P[X + YI,
Consider X and Y free random variables, then f o r a n y
.X(PX(WP(X
+ Y ) )=
(7x1
+ y,x2 + Y2)).(8.79)
8 ~X2+Y2)(aY+YP(x1
For any function m, writing U ( Y ) for the LHS,
Proof.
7Y U ( Y ) m ( Y ) = TX,Y( P X
( X ) P ( X+ Y ) m ( Y ) )
(8.80)
+
= 'x";yax (P(X Y ) m ( Y ) ) = (1 8 .Y,) ( ( T X X 1 , Y l8 TX*)dXP(Xl
+
(8.81) Yll x 2
+ YZ))(1 8 m)(Yz), (8.82)
as required.
0
Proposition 8.1 Consider free random variables X , Y and a function h such that IEh(X Y ) = 0 . W e can define the projections g X ( X ) = ~ y h ( X Y ) and g y ( Y ) = Q ~ ( X Y ) . There exists a constant p such that f o r any
+
+
+
IE ( h ( X + Y ) - g x ( X ) - g Y ( y ) ) 2 3
2 5 (m ( V g x ( X )- P I 2 + PIE ( V g Y ( Y )- d where CP
=
(8.83)
) 7
(8.84)
+
(1 - P ) @ ( X ) p @ ( Y ) .
Proof. Exactly as in Chapter 2 and in [Johnson and Barron, 2003], we can define:
Information Theory and the Central Limit Theorem
168
NOW,firstly by Cauchy-Schwarz:
P(Y)’
I .x(h(X
+ Y )- g x ( X ) - g Y ( Y ) ) 2 ( q x ) / 3 ) ,
(8.86)
so taking expectations over Y :
Secondly, by Lemma and
we deduce that
where
by Lemma 8.9. This means that we establish the result that
By symmetry, we can give the corresponding result that
Then, adding p times Equation (8.95) to (1-0)times Equation (8.94), the result follows. 0
Theorem 8.3 Consider identically distributed free random variables X , Y and functions h , f x , f y such that Eh(X + Y ) = 0. Then for some p:
In particular writing 4ff’T2Q,- 3:
aStfor
the standardised Fisher information
QSt
=
(8.97)
Analytical background
169
Proof. Unless and R are finite, this is a triviality. Note that for a , b positive: ax2 by2 2 ( a b / ( a b ) ) ( x - 9)'. By Proposition 8.1, and the definition of the projections,
+
+
lE ( h ( X + Y ) - f X ( X ) - f Y ( y ) ) 2 = IE ( h ( X + W X
+Y )
-
(8.98)
S X ( 4 -S Y ( Y V
(XI - f x ( x ) )+ 2%Y
( Y )- f u (Y)I2
(8.99)
(8,102)
since E V g x ( X ) - p = 0. In particular, in the case where f = 7, hx = h y = p/2 then the result follows. 0 Hence if X , are free identical variables with finite @ and R then lim
. . . + x, ) (xl+fi
=O.
(8.103)
Further, by the free de Bruijn identity, Lemma 8.8, if
Xi have finite R , then
71-00
n-cc lim
D
(xl+h+xnlw
)
= 0,
where W is a Wigner law with the same variance as Xi.
(8.104)
This page intentionally left blank
Appendix A
Calculating Entropies
A.l
Gamma distribution
In this appendix, we describe methods that can be used to calculate the entropy of more complicated distributions than those in Example 1.3. First of all, we shall require a lemma that extends the results in [Purcaru, 19911: Lemma A . l
Two integrals that occur in the calculation of these functions
are
for m > 0 , a > 0 , where the f matches the sign of n and function.
Proof.
r
i s the gamma
Define F ( a , m , n )=
I"
x-mn-l
First, making the substitution y -un/x"+ldx.So for n > 0 F(a,m,n)=
--
1
umn
J'
"
=
exp
(- 5)dx.
a / x n , we deduce that dy =
(2)m-1(-2) exp
y"-'exp(-y)dy 171
2 xn+ld s
=
-.r(n) amn
(A.4)
Information Theory and the Central Limit Theorem
172
Similarly for n
< 0,
1 -1 loo
F ( a , r n , n ) = -l o o( 5 ) m - 1 e x p amn -
amn
(-5)
2, p +dl x
ym-1 exp(-y)dy = --. r(n) amn
(A.6)
(A.7)
Now, we can evaluate the second integral, since
dF -(n, drn
rn,a) = -n /(log, x)x-mn-l exp (-5) dx. Xn
(A.8)
0
Using this, we can evaluate the entropy of a gamma distribution:
Lemma A.2 For Urn a r ( m , 8 ) distribtuion, with density f m ( x ) e-ex8mxm-1/I’(m), the entropy is
H ( U m ) = m - log8 - (m - 1)loge-r’(m) r(m) Here
+ log r ( m ) .
=
(~.9)
r‘/r is often referred to as the digamma function.
Proof.
Taking logarithms and using Equation (A.2) above, with n = -1,
a = 6’:
H(Um) =
s
f m ( x )(- log fm(x)) dx
(A.lO)
(A.13)
0 Note the error on page 486 of [Cover and Thomas, 19911, where the definition should read $ ( z ) = d/dz(lnr(z)).
Lemma A.3
For Urn a r ( m ,0) distribution: (A.14)
Proof.
We know that (see Equation (1.48)) (A.15)
Analytical background
173
Using asymptotic expansions, firstly Stirling’s formula and secondly an expansion from the entry on the digamma function in [Weisstein, 20031, loger(m) 2 (m - 1)-r’(m) r(m)
-
( k)
1 logem-m+--12m 1 1 (m - 1)loge(m - 1) - 2 12(m- 1 ) ’
+
m--
1 (A.16) 360m3
+
(A.17)
so that in Equations (A.15) and (A.13) we deduce that (A.18) =loge
A.2
-+-
(:m
1 6m2
+...).
(A.19)
0
Stable distributions
Next we discuss some ways of calculating the entropy of certain families of stable distributions:
Lemma A.4 G i v e n a f a m i l y of stable densities w i t h parameters ( a , p ) , t h e densities qc corresponding t o different values of c have entropies: H ( q c ) = H(q1)
+ (logc)/cr.
(A.20)
Proof. Since differential entropy is shift invariant, we need only consider the case where y = 0. We first consider the case cr # 1. Consider X with density qc and characteristic function
Q x ( t )= exp(-cltl*(l
+ ipsgn(t) t a n ( ~ a / 2 ) ) ) .
Now a X has characteristic function Qx((at), so taking a characteristic function
= c-’/~,
+ ipsgn(t) t a n ( ~ a / 2 ) ) ) , Then, since H ( a X ) = H ( X ) + loga,
= exp(-ltl*(l
(A.21) a X has (A.22)
H(q1) = and so has density q1. H ( s c ) - (logc)/a. In the case cr = 1, the same argument tells us that X/c has characteristic function
Information Theory and the Central Limit Theorem
174
which is just a shift of the original density, and so has the same entropy.0
Lemma A.5 The entropy of a Lkvy distribution H(lc,-,) = log(16mc4)/2 + 3n/2, where IF. = 0.5772 = l i m n - , ( ~ ~ = l l/i - logn) is Euler's constant. Proof. By Lemma A.4, we know we need only deal with the case c since the others follow by scaling. In this case,
H(ll,y) = log(2n)/2
+
s
Il,O(Y) (loge/2y
+ 3 b Y / 2 ) dy.
=
1,
(A.24)
So set,ting a = 1/2, n = 1 and m = 3/2 in Equation (A.l) gives that the first term in the integral is loge/2, and by Equation (A.2), the second term is -3/2(log2 I"(1/2)/r(l/2) loge). Now [Purcaru, 19911 shows that P ( m ) / r ( m ) = ( - l / m - IF. m C k l / k ( m + k ) ) , SO that I?(1/2)/r(1/2) = (-2-1~.+2(1-1og2)), so the 0 second term is I IF. log2)/2.
+
+
+
Recall that in Definition 5.4 we define 03
Adf)
-f (z) logg(z)dz -,
=
(A.25)
for given probability densities f,9.
Lemma A.6 If density g has score function p, and F is a distribution function with density f then for any t such that 0 < g ( t ) < 00:
A, (f)= - log g ( t )
Proof.
+ log e
t
[L
P(Y)F(Y)dY +
Since logg(z) = (loge) Jt" p(y)dy
1,
P(Y)(l - F(Y))dY] (A.26)
+ logg(t): (A.27)
Analytical background
175
+
The entropy of a Cauchy density P = , ~ ( X=) c/7r(c2 x 2 ) is
Lemma A.7 log 47rc.
Proof. By Lemma A.4, we know we need only deal with the case c = 1, since the others follow by scaling. Taking t = 0, and f = g = p l in Lemma A.6 we deduce that
(A.31)
(A.32) = logr -
4loge
-
log, (cos w)dw,
(A.33)
by using the substitution w = tan-' y in the first integral, and IJ = - tan-l y in the second. Now use the fact that by symmetry of the sine function
1
TI2
2
=
2
log,(cos w)dw
lTI2
log,(sin 2w)dw - 2
1
TI2
=-
log,(sinw)dw
= -7rlog, 2,
+
r2
log,(sin w)dw - 7r log, 2 (A.34)
L2
log,(sinw)dw - rlog, 2
and so the result follows since (log, 2)(log e ) = 1.
(A.35) (A.36)
0
This page intentionally left blank
Appendix B
Poincarh Inequalities
B.l
Standard Poincarh inequalities
Poincark (or spectral gap) inequalities provide a relationship between L2 norms of functions and their derivatives. For example [Borovkov and Utev, 19841 gives: Definition B . l
Given a random variable Y , define the Poincark constant
Ry :
where HI ( Y )is the space of absolutely continuous functions on the real line such that Var g(Y)> 0 and Eg/(Y)2< 00. By considering g ( y ) = y, it is clear that RY 2 Var Y . Our intuition might suggest that integration will act as a smoothing, so if g' is small, then g will be close t o constant, and that therefore RY will tend to be small. However, the next two examples show that this need not be the case. Informally speaking, the Poincar6 constant will be infinite if there is too much mass in the tails of the random variable, or a hole in its support.
Example B . l For any random variable Y with some moment infinite, the Poincark constant is infinite. Consider k such that EY2kis finite but IEY2"' is infinite, and functions which tend to g(y) = y"'. Hence the finiteness of Ry will imply the existence of moments of all orders. Example B.2 For any discrete random variable, indeed any random variable whose support is not an interval, the Poincark constant will be infinite. For example, consider the discrete random variable, where P(X = 1) = P(X = -1) = 1/2. Then, we can choose g such that 177
Information Theory and the Central Lamit Theorem
178
g’(-1) = g’(1) = but lEg’(X)’ = E’.
E,
but g(-1) = -1, g ( l ) = 1, so that V a r g ( X ) = 1,
Hence, Ry will not in general be finite, however it will be finite for the normal and other strongly unimodal distributions (see for example [Klaassen, 19851, [Chernoff, 19811, [Chen, 19821, [Cacoullos, 19821, [Nash, 19581, [Borovkov and Utev, 19841). Theorems 2, 3 and 4 of Borovkov and Utev provide the following results:
Lemma 8 . 1
For the constant Rx defined above
%X+b = a 2 R x (2) If X , Y are independent, then RX+Y 5 Rx RY (3) Rx 2 Var X, with equality i f and only i f X is normal (4) If Rx is finite then lEexp(1X - IEX1/12&) 5 2, so X has moments of all orders. (5) If Rx,,/Var X, + 1, then IEw(X,) + IEw(Z), where Z is normal, for any continuous w with Iw(t)I < exp(cltl), f o r suficiently small c.
(1)
+
These properties are reminiscent of those of l/(Fisher information) - a subadditive relation holds (see Lemma 1.22) and the minimising case characterises the normal distribution (see Lemma 1.19). In analogy with the approach to the Central Limit Theorem described elsewhere in the book, in [Johnson, 2003al this is exploited to show the convergence of Poincark constants. We add an extra term into the subadditive relation RX+Y5 Rx R y , which is sandwiched as convergence occurs. This gives us a n answer t o the question posed by [Chen and Lou, 19901, of identifying the limit of the Poincark constant in the Central Limit Theorem. This was also answered by [Utev, 19921, though without the explicit rate of convergence that we provide.
+
Theorem B . l Consider Xl,Xz,. . . IID, with R = R x , and I = I ( X ) finite. Defining U, = ( X I . . . X n ) / m , then there exists a constant C , depending only on I and R, such that Ru,, - 1 5 Cln.
+
Poincark inequalities are also known as spectral gap inequalities. This comes from knowledge of the adjoint map, since if D+ is the adjoint of map D, then
(91 D + D d = (Dg, D d ,
03.2)
so if we know the lowest eigenvalue of D+D is A, then
( D g ,9)2
9).
03.3)
179
Analytical background
We can calculate the adjoint of the derivative map as follows: Lemma B.2 For Of(.) = f‘(z),t h e adjoint ft(5), so that D’Df(z) = - p ( z ) f ’ ( z ) - f”(z).
Proof.
off(.) = - p ( z ) f ( z )
-
For any g and h:
An alternative way of seeing the significance of these maps D + D is t o calculate directly. Notice that for a given Y , if g is a local maximum of Var g ( Y ) / ( E g ’ ( Y ) 2 ) then for all functions h and small t
so multiplying out, 0 5 t2(RgIEh”- IEh2)+ 2t(RgEg’h’ - E g h ) for all t , which can only hold on both positive and negative sides of zero if
RgEg’h’ = E g h for all h.
(B.8)
+
Integration by parts implies therefore that g = -R,(pyg’ 9”) = R, ( D + D g ) ,so local maxima correspond to eigenfunctions of the Laplacian D + D , and the global maximum to the least strictly negative eigenvalue.
B.2
Weighted Poincarg inequalities
Now, although the Poincark inequalites are a good tool for analysis in the case of convergence t o the normal distribution, we suggest that we will require weighted versions of them to establish convergence to other stable distributions. Recall that the Hermite polynomials form an orthogonal basis with respect to the normal distribution. Then, the map D acts as a ‘ladder operator’ and takes Hk to a scalar multiple of Hk-1. Now, for other, non-normal stable distributions, there will be a different orthogonal basis. We suggest that the right map D to consider may well be the ladder operator there.
Information Theory and the Central Limit Theorem
180
For example, for the Cauchy distribution, as described in Lemma 5.8 functions derived from the Chebyshev polynomials form an orthogonal set. In this case, the map D f ( z ) = ((1 z2)/2)f’(s) acts as a ladder operator. ’ ( zas) a Similarly, for the Lkvy distribution, the map D f ( z ) = 2 ~ ~ / ~ facts ladder operator between orthogonal functions. In general, we can again calculate the adjoint of such a weighted derivative map.
+
For
Lemma Proof.
the adjoint
For any g and h:
+
+
Hence D + D f ( z ) = D + ( m f ’ )= - ( p ( z ) u ( x ) u ’ ( z ) ) f ’ ( z ) u(z)f”(z), for u ( z )= m ( 2 ) 2 . In the Cauchy case, it turns out that the map Df(z) = ((1+z2)/2)f’(z) is self-adjoint (this will be the case whenever m ’ ( z ) / m ( z )= - p ( z ) , that is when m ( z ) = c / p ( z ) ) ,and that D + D f ( z ) = -(2z(l z2)f’(z) (1 52)2f’’(z)) /4. Another useful property that may influence our choice of m ( z ) is that C / ( f ( z ) p ’ ( z ) ) for some c, we ensure that by choosing u ( z ) = -l/p’(z) p ( z ) is an (-1)-eigenfunction. For example, in the normal N(O,1) case, we can take m ( z )= 1. Similarly, in the Cauchy case, taking C = - l / 2 ensures that m ( z )= (z2 1 ) / 2 has this property. In the spirit of [Borovkov and Utev, 19841, we can provide a condition t o satisfy such a weighted Poincari: inequality:
+
+ +
+
+
Lemma B.4
Consider the map Dg(y) = m(y)g’(y), where m(y) =
Analytical background
d m , f o r some p(y).
181
Now if the density h(y) satisfies (B.13)
(B.14)
f o r some C , then (B.15) f o r all functions g .
Proof.
For the function g, by Cauchy-Schwarz, for any x:
Hence
by reordering the limits, so the result follows.
0
This page intentionally left blank
Appendix C
de Bruijn Identity
Since the Gaussian extremises both entropy and Fisher information, it is perhaps unsurprising that a link exists between the two quantities. The result is provided by the de Bruijn identity:
Theorem C . l If U is a random variable with density f and variance 1, and Z, is N ( 0 ,r ) , independent of U , then
This is a rescaling of Lemma 1 of [Barron, 19861, which is an integral form of the original de Bruijn identity, as proved by [Stam, 19591 and also by [Bakry and Emery, 19851. The important observation here is that 4T satisfies the heat equation:
and hence so does j T ,the density of V+ZT,since f T ( x ) =
f(y)&(z-y)dy
so that
Thus using the fact that the smoothed densities decay at infinity,
In fact, we give a proof of the n-dimensional version of this result, as required for proofs in the vector case in Chapter 3. 183
Information Theory and the Central Limit Theorem
184
Definition C . l
For random vectors u, v, define the norm: (Uy =
(C.5)
E(uTu),
and the inner product ( u , v ) = E(u*v). (PU, PV) = EtrP*Pvu*.
Note that for any matrix P ,
Next we will use the natural inner product space for random vectors and the Fisher information matrix. Lemma C . l G i v e n a r a n d o m vector X, consider X Z c r , where ZcT i s independent of X . T h e n
+
fT,
t h e density of Y
=
Proof. f7 is twice continuously differentiable, so we know that V 2 f T exists. Now the Gaussian density satisfies the n-dimensional version of the heat equation:
and so
Hence, we deduce that tr
(cv24CT(z))=
( '," + --
Z
~
y
z
a4cr
) 4 C T ( Z ) = 2-(Z),Z
87.
E R"
(C.9)
and taking expectations with respect to X provides the result, since
so
2-(x) dfr 87as required.
= 2E-(x a4cT
87-
-
X) = CcijiE-d24CT (x - X), dX,dXj
(C.11)
0
Analytacal background
185
When density f has covariance matrix B, 1 log((2re)" det C) - H ( f ) log e(tr(C-lB) - n ) / 2 (C.12) 2 = H ( 4 c ) - H ( f ) log e(tr(C-'B) - n)/2, (C.13)
D ( f 114~)=
+
-
+
where H represents the differential entropy. In other words, again we can view the relative entropy distance from the normal as a linear function of the entropy.
Theorem C.2 Given X a random vector with density f and covariance matrix B , let f T be the density of Y, = X+Zc,, where Z c T i s independent of X. Then
+ * 2i m t r ( ~ ( ( ~ + ~ i ) - l
-
c)) 1+r
dT.
(C.14)
Note that if B = C then (C.15) Proof. This result is an integral form of Lemma C . l . As N -+ co, f N becomes closer to a normal, so limN,,(H(fN) - n l o g d m ) = 1/2log((2re)"detC). As M -+ 0 , f M + f in probability, so H ( f M ) -+ H ( f ) by upper semi-continuity of H , where H ( f ) may be -a. Hence if the integral is finite, by Lemma C.1, (C.16)
= H ( f N ) - n log d
-*s," 2
tr
i D
))
(c (J(YT) '-' -
l+r
d7.
(C.19)
186
Information Theory and the Central Limit Theorem
We obtain the result by taking the limit as N -+ 00. If the integral is then by Fatou H ( f ) = -m. Rearranging, we obtain the first form.
-00,
0
Appendix D
Entropy Power Inequality
Shannon stated the Entropy Power inequality as Theorem 15 of [Shannon and Weaver, 19491, and sketched a proof in Appendix 6. However, this proof was not sufficiently rigorous, and new proofs were offered by Blachman [Blachman, 19651 and later by Dembo, Cover and Thomas [Dembo et al., 19911. Theorem D.1
For X and Y independent n-dimensional random vectors
with equality if and only if X and Y are normally distributed with proportional covariance matrices.
Proof. We will write exp,(z) for 2". We summarise the proof of the 1dimensional case, which is based on two results previously described; firstly the de Bruijn identity Theorem C . l , and secondly the bounds on Fisher information of Lemma 1.22: 1
J(U
+V )
1 >-+-
1
J(U)
-
+
J(V)'
+
+
-
The key is to define X f X Z f ( t ) Yg , Y Z g ( t )and , zh = Xf Yg (X+ Y ) Zh(t),Here Z f ( t )and Z,(t) are normals, independent of each other and X , Y , each with mean zero, and variance f ( t )and g ( t ) respectively, and hence h ( t ) = f ( t ) + g ( t ) . Then, defining the function
+
N
187
188
Information Theory and the Central Limit Theorem
we know that by de Bruijn's identity Theorem
This means that
Hence choosing f’(t) = expz(2H(Xf))and g’(t) = expz(2H(Yg))to ensure that this last term is a square, s’(t) 2 0. Now as t + cu, s(t) -+ 1 (since each of the variables become ‘more normal’), so s(0) 5 1, as required. 0 The proof of the Entropy Power inequality given by Dembo, Cover and Thomas in [Dembo et al., 19911 is based on two observations. Firstly, Shannon entropy is a limiting case of the R h y i a-entropy of Definition 1.8. Secondly, [Beckner, 19751 shows the exact values of the constants in the Hausdorff-Young inequality:
Lemma D.I Writing l l f l l p = ( J ~ f ( x ) l ~ )for l’~ thep-norm o f f , i f ~ / r + 1 = l / p + l / q then
where cp = (p)’/P/(p’)’/p‘, for p’ satisfying l / p
+ l/p’ = 1.
Armed with this, [Dembo et al., 19911 complete the proof, since Hp(f) = -p’log I l f l l p , so taking the right limiting regime for p, q , the Entropy Power inequality follows. A stronger result (in the case where one of the variables is a normal perturbation) is provided by [Costa, 19851. A simpler proof, which we summarise here, is provided in [Dembo, 19891.
Lemma D.2 The entropy power N(Xt) = exp2(2H(Xt))/(27re)is concave in t, where Xt = X &, for Zt a N(0, t) independent of X .
+
189
Analytical background
Proof.
We are required t o prove that
d2 --N(Xt) dt2
50.
By the differential form of the de Bruijn identity, Equation ((2.4) this is equivalent t o proving that d
-J(Xt)+ J(x,)2 50.
(D.9)
dt
Now by Equation (D.2), with U
=X
1
J(X + 2t+T)
+ Zt and V = 2, then 1
J(X + 2,) r.
(D.lO)
+
Rearranging we obtain that
and letting r
----f
0 we deduce Equation (D.9) by continuity.
0
This page intentionally left blank
Appendix E
Relationships Between Different Forms of Convergence
E.l
Convergence in relative entropy to the Gaussian
Since we have established a proof of convergence in relative entropy in various circumstances, we will discuss how strong a result this really is, and how it compares t o more standard forms of convergence. We consider how strong convergence to the normal distribution in relative entropy really is. The paper [Gibbs and Su, 20021 discusses the relationship between these different forms of convergence in more detail. An example provided in Section 5 of [Barron, 19861 shows that convergence in relative entropy is strictly stronger than weak convergence.
Example E.l
for some r
> 0.
Consider a probability density
Then for
Xi independent
and each with density f r , define
Barron shows that for all n, the relative en( X I + . . . X,)/&. tropy distance D(U,ll4) is infinite, but the X have finite variance so weak
U,
=
convergence occurs. Hence, in addition to the intrinsic interest that convergence in relative entropy provides, because of Lemma 1.8 and Example E.l it provides a strong form of convergence. In particular, it is stronger than the convergence in L1 that Prohorov proves using characteristic function techniques in [Prohorov, 19521. Barron also provides an example that shows that convergence in relative entropy is strictly weaker than uniform convergence. 191
Information Theory and the Central Limit Theorem
192
Example E.2
Consider a probability density
for some r > 0. Then for X i independent and each with density f T , define Un = ( X I . . . X n ) / f i . Barron shows that for n > l / r l D(Unll$) is finite, so it must converge t o zero. However, the Un have unbounded densities for all n, and therefore they cannot converge uniformly t o a normal density.
+
Based on an observation from [Takano, 19871, we deduce:
Theorem E . l C o n s i d e r independent identically distributed r a n d o m variables X i w i t h finite variance c2 and densities. W r i t e g, f o r t h e density of U , and 6, = supz Igm(x) - +(z)1. Iflim,+OO 6, = 0 t h e n lim
n-cc
Proof.
%7nII+) =
0.
(E.3)
For a given sequence b,, define the interval B, = [-bnr b,]. We
have that
which is less than (loge)b,/+(b,), which converges to zero if b, + co at such a rate that 6, exp(b2/2) 4 0. Now for n sufficiently large, g, 5 C, so log(g,(x)/$(z)) 5 logC log(Zr)/Z + (log e)x2/2 = P x 2 Q for some P, Q. By Chebyshev's inequality, we need t o bound l x 2 1 ( x $? B n ) g n ( x ) d x . Given E , we can find Ic such that Jx21(1x12 k)+(x)dx 5 E . Since by the Central Limit Theorem gn converges weakly and is continuous and bounded on compact sets we know that
+
+
lim / x 2 1 ( x $2 B,)g,(x)dx
5 lim
11-00
,+00
=
s
1(1x1 2 Ic)g,(x)dx
1 - lim n+cc
J
z21(lzl 5 k ) g , ( z ) d z
(E.6)
Analytical background
We deduce the result since
E
193
is arbitrary.
0
To see how Theorem E.l compares to the results of Barron, recall the following local limit theorem (Theorem 1 of Section 46 of [Gnedenko and Kolmogorov, 19541): Theorem E.2 Writing g m for the density of Urn, if gm E L' for some m and 1 < r 5 2 , then
Remark E . l Note that since the maximum of log, x / x T - l occurs at x = e x p ( l / ( r - l ) ) ,we know that Jgmloggm 5 cJ(g,)', so boundedness in L' is strictly stronger than boundedness in D . Thus Theorem E.l is strictly
weaker than that of Barron.
Boundedness in L', r
\
/
(Theorem E.2)
Convergence in L"
(Theorem E.1)
I
1
Convergence in D
(Lemma 1.8)
I
-
>1
(Theorem 2.1) *
(Remark E.l)
\
Boundedness in D
Convergence in L1 Fig. E.l
Convergence of convolution densities to the normal
In Figure E.l, we present a summary of these relationships between various forms of convergence of densities g m in the IID case.
Information Theory and the Central Limit Theorem
194
E.2
Convergence to other variables
A result from [Takano, 19871 proves convergence in relative entropy for integer valued random variables.
Theorem E.3 Let Xi be IID, integer valued random variables, with finite variance cr2, mean p . Define S, = X I + . . . X,, with probabilities q , ( i ) = P(S, = i ) . Define a lattice version of the Gaussian N ( n p , n a 2 ) , as
+
(E.lO)
If the X , are aperiodic (that is, there exists no r and no h P(X, = hn + r ) = 1) then
#
1 such that
Proof. Takano's bound from above does not use any new techniques to prove convergence t o the Gaussian, but rather is based on another local limit theorem (see [Gnedenko and Kolmogorov, 19541, Section 49, page 233)' which states that if the random variables are aperiodic then (E.12) Notice that 4, is not a probability distribution, but Takano's Lemma 1 states that I & ( i ) - 11 = O ( l / n ) . The log-sum inequality, Equation (1.37)' gives that 4 , log(qnl4n) 2 A71 2 4n) log((C qn)/(C471))2 - loge( 1- C &) = O( l / n ) , so we deduce the result using the method used 0 to prove Theorem E . l above.
xi
c
(c
A similar argument is used in [Vilenkin and Dyachkov, 19981, to extend a local limit theorem into precise asymptotics for the behaviour of entropy on summation. For arbitrary densities it is not the case that convergence in L" implies convergence in D. As hinted in the proof of Theorem E . l , we require the variances to be unbounded. Example E.3
Consider h,(z) = c$(z)gn(z),where
gn(z) = 1 for z < n ,
(E.13)
gn(z)= exp(n2/2)/n for z 2 R.
(E.14)
Analytical background
195
Then defining C, = J-”, hn(z)dx,consider the probability density fn(x) = hn(z)/Cn. Then
Proof. We use the tail estimate (see for example [Grimmett and StirzaFirstly ker, 19921, page 121) that Jncc 4 ( x ) d ~ exp(-n2/2)/(&n). limn+cc Cn = 1, since N
L lw n
C, - 1 = =
4(z)dx+
/
00
4 ( x )exp(n2)/ndx - 1
(E.16)
n
4(z)dz [exp(n’/a)/n
-
11
(E.17) (E.18)
Then 1 lim sup 1hn(x) - ~ ( Z )= I lim -exp(-n2/2) n+cu
n-w
6
(E.19)
and so uniform convergence occurs;
and convergence in relative entropy does not occur; (E.22)
and hence limn-o3 D(fnl14)= 1/(2d%).
E.3
Convergence in Fisher information
The paper [Shimizu, 19751 shows that
Information Theory and the Central Limit Theorem
196
Lemma E . l If X is a random variable with density f , and q5 is a standard normal density then
(E.25) In [Johnson and Barron, 20031, an improvement to Equation (E.25) is offered, that
/ IfI.(
- 4(.)ldx
i 2 d H ( f , 4)I J z m ,
(E.26)
where d ~ ( f4) , is the Hellinger distance (J I m- m 1 2 d x ) result is proved using Poincark inequalities. Indeed:
1/2
. This
Lemma E.2 If random variable 2 has density h and Poincare‘ constant R t h e n for all random variables X with density f:
1
If(.)
-
h(z)ldz I 2 d H ( f , h) 5
Proof. We can consider the function g of the Poincark constant
=
d
m
.
(E.27)
d m . By the existence (E.28)
where p
=
s h(x)g(x)dx
=
Sd
m d x . Hence, this rearranges t o give
1- p 2 5 R ( J ( X \ I Z ) / 4 ) . Now, since (1 - p ) 5 (1 - p)(l
+ p ) = (1
-
(E.29)
P ) ~ we , know that
( 2 d ~ ( hf ,) ) 2= 8(1 - p ) I 8(1 - p)2 5 2 R J ( x l l z ) .
(E.30)
The relationship between the Hellinger and L’ distance completes the argument (see for example page 360 of [Saloff-Coste, 19971). 0 Convergence in Fisher information is in fact stronger than convergence in relative entropy, a fact established using the Entropy Power inequality Theorem D.l.
197
Analytical background
For any random variable X ,
Lemma E.3
(E.31)
Proof.
The Entropy Power inequality, Theorem D.l, gives that (for any
t) exPz(2H(X 27re
+ 2,)) 2 ..Pz(2H(X)) 2ne
+
exP2(2H(Zt)) 2ne
(E.32)
+
Hence, defining s ( t ) = expz(2H(X Zt))/(27-re), this implies that
+
(E.33)
s(t) 2 s(0) t Hence, s’(0) = lim,,o(s(t)-s(O))/t Theorem C . l ,
2 1. However, by the de Bruijn identity (E.34)
so that rearranging,
~ J ( 2x (27rea’) ) exp2(-2H(X))
(E.35)
= expz(2H(Z) - 2 H ( X ) ) = exp2(2D(X)),
and D ( X ) L log(1
+ Jst(X))/2 5 Jst(x)(loge)/2.
(E.36)
0
This result can also be obtained using the celebrated log-Sobolev inequality of [Gross, 19751, which gives that for any function f
1lf(4l2
log, lf(.)I4(.)d.
where
llf11$
=
I
/
l O f ( 4 l 2d z ) d z + 1 fl122log,
Ilfllz,
03.37)
I f ( z ) l z ~ ( x ) d x . Now, in particular, taking f(z)to be
d m ,for a particular probability density g , Equation (E.37) gives
(E.38) Hence, the presence of a log-Sobolev constant will be enough to give a bound similar to Lemma E.3.
This page intentionally left blank
Bibliography
Andersen, K. (1999). Weighted inequalities for iterated convolutions. Proc. Amer. Math. SOC.,127(9):2643-2651. Applehaum, D. (1996). Probability and information: A n integrated approach. Cambridge University Press, Cambridge. Bakry, D. and Emery, M. (1985). Diffusions hypercontractives. In Se'minaire de probabilite's, X I X , 1983/84, volume 1123 of Lecture Notes in Math., pages 177-206. Springer, Berlin. Ball, K., Barthe, F., and Naor, A. (2003). Entropy jumps in the presence of a spectral gap. Duke Math. J., 119(1):41-63. Banis, I. I. (1975). Convergence in the mean for densities in the case of a stable limit law. Litovsk. Mat. Sb., 15(1):71-78. In Russian. Barbour, A . , Holst, L., and Janson, S. (1992). Poisson Approzimation. Clarendon Press, Oxford. Barron, A. (1986). Entropy and the Central Limit Theorem. Ann. Probab., 14(1):336-342. Barron, A. (1991). Information theory and martingales. In Int. Symp. Inform. Theory, Budapest, Hungary. Barron, A. (2000). Limits of information, Markov chains and projection. In Int. Symp. Inform. Theory, Sorrento, Italy. Beckner, W. (1975). Inequalities in Fourier analysis. A n n . of Math. (2), 102(1):159-182. Bercovici, H. and Voiculescu, D. (1993). Free convolution of measures with unbounded support. Indiana Univ. Math. J., 42(3):733-773. Biane, P. (1997). On the free convolution with a semi-circular distribution. Indiana Univ. Math. J., 46(3):705-718. Biane, P. (1998). Free probability for probabilists. math.PR/9809193, MRSI 1998-040. Blachman, N. (1965). The convolution inequality for entropy powers. IEEE Trans. Information Theory, 11:267-271. Bobkov, S. G. and Ledoux, M. (1998). On modified logarithmic Sobolev inequalities for Bernoulli and Poisson measures. J . Funct. Anal., 156(2):347-365. Borovkov, A. and Utev, S. (1984). On an inequality and a related characterisation 199
200
Information Theory and the Central Limit Theorem
of the normal distribution. Theory Probab. Appl., 28(2):219-228. Bradley, R. (1986). Basic properties of strong mixing conditions. In Eberlein, E. and Taqqu, M., editors, Dependence in Probability and Statistics, pages 165-192. Birkhauser, Boston. Brown, L. (1982). A proof of the Central Limit Theorem motivated by the Cram&-Rao inequality. In Kallianpur, G., Krishnaiah, P., and Ghosh, J., editors, Statistics and Probability: Essays in Honour of C . R . Rao, pages 141-148. North-Holland, New York. Cacoullos, T. (1982). On upper and lower bounds for the variance of a function of a random variable. A n n . Probab., 10(3):799-809. Carlen, E. and Soffer, A. (1991). Entropy production by block variable summation and Central Limit Theorems. Comm. Math. Phys., 140(2):339-371. Chen, L. (1982). An inequality for the normal distribution. J . Multivariate Anal., 12(2) :306-315. Chen, L. and Lou, J. (1990). Asymptotic normality and convergence of eigenvalues. Stochastic Process. Appl., 34(2):197. Chernoff, H. (1981). A note on an inequality involving the normal distribution. A n n . Pro bab., 9(3 ):533-535. Costa, M. H. M. (1985). A new entropy power inequality. IEEE Trans. Inform. Theory, 31(6):751-760. Cover, T. and Thomas, J. (1991). Elements of Informatzon Theory. John Wiley, New York. Csiszar, I. (1965). A note on limiting distributions on topological groups. Magyar Tud. Akad. Math. Kutatd Int. Kozl., 9:595-599. Csiszar, I. (1976). Note on paper 180. In Turan, P., editor, Selected Papers of Alfre'd Rknyi, volume 2, page 580. Akadkmiai Kiado. Csiszar, I. and Korner, J. (1981). Information theory: Coding theorems for discrete memoryless systems. Probability and Mathematical Statistics. Academic Press Inc. [Harcourt Brace Jovanovich Publishers], New York. Deheuvels, P. and Pfeifer, D. (1986). A semigroup approach t o Poisson approximation. A n n . Probab., 14(2):663-676. Dembo, A. (1989). Simple proof of the concavity of the entropy power with respect to added Gaussian noise. IEEE Trans. Inform. Theory, 35(4):887-888. Dembo, A , , Cover, T., and Thomas, J. (1991). Information theoretic inequalities. IEEE Trans. Information Theory, 37(6):1501-1518. Dembo, A , , Guionnet, A , , and Zeitouni, 0. (2003). Moderate deviations for the spectral measure of certain random matrices. Ann. Inst. H. Poincare Prob. Stats., 39(6):1013-1 042. Dembo, A. and Zeitouni, 0. (1998). Large Deviations Techniques and Applications. Springer, second edition. Derriennic, Y. (1985). Entropie, thCor6mes limite et marches alkatoires. In Heyer, H., editor, Probability Measures on Groups VIII, Oberwolfach, number 1210 in Lecture Notes in Mathematics, pages 241-284, Berlin. Springer-Verlag. In French. Eddington, A. (1935). The Nature of the Physical World. J.M.Dent and Sons, London.
Bibliography
201
Faddeev, D. (1956). On the concept of entropy of a finite probabilistic scheme. Uspekhi Mat. Nauk, 11(1):227-231. Fedotov, A., Harremoes, P., and Topscle, F. (2003). Refinements of Pinsker's inequality. IEEE Trans. Inform. Theory, 49(6):1491-1498. F'rieden, B. (1998). Physics from Fisher information, A unzfication. Cambridge University Press, Cambridge. Gawronski, W. (1984). On the bell-shape of stable densities. A n n . Probab., 12(1):230-242. Georgii, H.-0. (2003). Probabilistic aspects of entropy. In Greven, A . , Keller, G., and Warnecke, G., editors, Entropy, Princeton and Oxford. Princeton University Press. Presented at International Symposium on Entropy, Dresden, June 25-28, 2000. Gibbs, A. and Su, F. (2002). On choosing and bounding probability metrics. Internat. Statist. Rev., 70(3):419-435. Gnedenko, B. and Kolmogorov, A. (1954). Limit Distributions for sums of independent Random Variables. Addison-Wesley, Cambridge, Mass. Gnedenko, B. and Korolev, V. (1996). Random Summation: Limit Theorems and Applications. CRC Press, Boca Raton, Florida. Goldie, C. and Pinch, R. (1991). Communication theory, volume 20 of London Mathematical Society Student Texts. Cambridge University Press, Cambridge. Grenander, U. (1963). Probabilities on Algebraic Structures. John Wiley, New York. Grimmett, G. (1999). Percolation (Second Editzon). Springer-Verlag, Berlin. Grimmett, G. and Stirzaker, D. (1992). Probabzlity and Random Processes (Second Edition). Oxford Science Publications, Oxford. Gross, L. (1975). Logarithmic Sobolev inequalities. Amer. J . Math., 97(4):10611083. Hall, P. (1981). A comedy of errors: the canonical form for a stable characteristic function. Bull. London Math. SOC.,13(1):23-27. Harremoes, P. (2001). Binomial and Poisson distributions as maximum entropy distributions. IEEE Trans. Information Theory, 47(5):2039-2041. Hartley, R. (1928). Transmission of information. Bell System Tech. J., 7:535-563. Hayashi, M. (2002). Limiting behaviour of relative RBnyi entropy in a non-regular location shift family. math.PR/0212077. Heyer, H. (1977). Probability Measures on Locally Compact Groups. SpringerVerlag, Berlin. Hiai, F. and Petz, D. (2000). The semicircle law, free random variables and entropy, volume 77 of Mathematical Surveys and Monographs. American Mathematical Society, Providence, RI. Hoffman-Jergensen, J. (1993). Stable densities. Theory Probab. Appl., 38(2):350355. Holtsmark, J. (1919). Uber die Verbreiterung von Spektrallinien. A n n . Physik, 58:577-630. Hu, K.-T. (1962). On the amount of information. Theory Probab. Appl., 7:439447.
202
Information Theory and the Central Limit Theorem
Ibragimov, I. (1962). Some limit theorems for stationary processes. Theory Probab. Appl., 7:349-381. Ibragimov, I. and Linnik, Y. (1971). Independent and stationary sequences of random variables. Wolters-Noordhoff, Groningen. Johnson, 0. (2000). Entropy inequalities and the Central Limit Theorem. Stochastic Process. Appl., 88(2):291-304. Johnson, 0 . (2001). Information inequalities and a dependent Central Limit Theorem. Markov Process. Related Fields, 7(4):627-645. Johnson, 0 . (2003a). Convergence of the PoincarC constant. To appear in Theory Probab. Appl., 48(3). Statslab Research Report 2000-18, math.PR/0206227. Johnson, 0 . (2003b). An information-theoretic Central Limit Theorem for finitely susceptible FKG systems. I n submission. Statslab Research Report 200103, math.PR/0109156. Johnson, 0. and Barron, A. (2003). Fisher Information Inequalities and the Central Limit Theorem. I n submisszon. Statslab Research Report 2001-17, math.PR/O111020. Johnson, 0. and Suhov, Y. (2000). Entropy and convergence on compact groups. J . Theoret. Probab., 13(3):843-857. Johnson, 0. and Suhov, Y. (2001). Entropy and random vectors. J . Statist. Phys., 104(1):147-167. Johnstone, I. and MacGibbon, B. (1987). Une mesure d'information caractkrisant la loi de Poisson. In Se'minaire de Probabilite's, X X I , pages 563-573. Springer, Berlin. Kawakubo, K. (1991). The Theory of Transformatzon Groups. Oxford University Press, Oxford. Kendall, D. (1963). Information theory and the limit theorem for Markov Chains and processes with a countable infinity of states. Ann. Inst. Statist. Math., 15:137-143. Khintchine, A. and LCvy, P. (1936). Sur les lois stables. C. R. Acad. Sci. Paris, 202~374-376. Klaassen, C . (1985). On an inequality of Chernoff. Ann. Probab., 13(3):966-974. Kloss, B. (1959). Probability distributions on bicompact topological groups. Theory Probab. Appl., 4:237-270. Kolniogorov, A. N. (1983). On logical foundations of probability theory. In Probability theory and mathematical statistics (Tbilisi, 1982), volume 1021 of Lecture Notes in Math., pages 1-5. Springer, Berlin. Kontoyiannis, I., Harremoes, P., and Johnson, 0. (2002). Entropy and the law of small numbers. I n submission. Statslab Research Report 2002-16, math. PR/0211020. Kullback, S. (1967). A lower bound for discrimination information in terms of variation. IEEE Trans. Information Theory, 13:126-127. Kullback, S . and Leibler, R. (1951). On information and sufficiency. Ann. Math. Statist., 22:79-86. Le Cam, L. (1960). An approximation theorem for the Poisson Binomial distribution. Pacific J . Math., 10:1181-1197. Levine, R. D. and Tribus, M., editors (1979). The maximum entropy formalism:
Bibliography
203
A Conference held at the Massachusetts Institute of Technology, Cambridge, Mass., May 2-4, 1978. MIT Press, Cambridge, Mass. LCvy, P. (1924). Thborie des erreurs. la loi de gauss et les lois exceptionelles. Bull. SOC.Math, 52:49-85. Lieb, E. and Yngvason, J. (1998). A guide to Entropy and the Second Law of Thermodynamics. Notices Amer. Math. Soc., 45(5):571-581. Linnik, Y. (1959). An information-theoretic proof of the Central Limit Theorem with the Lindeberg Condition. Theory Probab. Appl., 4:288-299. Linnik, Y. (1960). On certain connections between the information measures of Shannon and Fisher and the summation of random vectors. In Transactions of the 2nd Prague Conference. Czechoslovak Academy of Sciences. In Russian. Maassen, H. (1992). Addition of freely independent random variables. J. Funct. Anal., 106(2):409-438. Macdonald, I. G. (1995). Symmetric functions and Hall polynomials. Oxford Mathematical Monographs. The Clarendon Press Oxford University Press, New York, second edition. With contributions by A. Zelevinsky, Oxford Science Publicat ions. Major, P. and Shlosman, S. (1979). A local limit theorem for the convolution of probability measures on a compact connected group. Z. Wahrsch. Verw. Gebiete, 50( 2):137-148. Mandl, F. (1971). Statistical Physics. John Wiley, London. Medgyessy, P. (1956). Partial differential equations for stable density functions and their applications. Magyar Tud. Akad. Math. Kutatd Int. Kozl., 1:489518. In Hungarian. Medgyessy, P. (1958). Partial integro-differential equations for stable density functions and their applications. Publ. Math. Debrecen, 5:288-293. Miclo, L. (2003). Notes on the speed of entropic convergence in the Central Limit Theorem. In Proceedings of the conference on ‘Stochastic inequalities and their applications ’, Barcelona June 2002. Nash, J. (1958). Continuity of solutions of parabolic and elliptic equations. Amer. J . Math., 80:931-954. Newman, C. (1980). Normal fluctuations and the FKG inequalities. Comm. Math. Phys., 74(2):129-140. O’Connell, N. (2000). Information-theoretic proof of the Hewitt-Savage zero-one law. Hewlett-Packard technical report. Available via http://www.hpl.hp.com/techreports/2OOO/HPL-BRIMS-2OOO-l8. html. Petrov, V. (1995). Limit Theorems of Probability: Sequences of Independent Random Variables. Oxford Science Publications, Oxford. Prohorov, Y. (1952). On a local limit theorem for densities. Doklady Akad. Nauk S S S R (N.S.), 83:797-800. In Russian. Purcaru, I. (1991). Note sur l’entropie d’une distribution continue. Anal. Nume‘r. The‘or. Approx., 20(1-2):69-75. In French. Reinert, G. (1998). Couplings for normal approximations with Stein’s method. In Microsurveys in discrete probability (Princeton, N J , 1997), volume 41 of DIMACS Ser. Discrete Math. Theoret. Comput. Sci., pages 193-207. Anier.
204
Information Theory and the Central Limit Theorem
Math. SOC.,Providence, RI. Rknyi, A. (1961). On measures of entropy and information. In Neyman, J., editor, Proceedings of the 4th Berkeley Conference on Mathematical Statistics and Probability, pages 547-561, Berkeley. University of California Press. Rknyi, A. (1970). Probability theory. North-Holland Publishing Co., Amsterdam. Rotar, V. (1982). Summation of independent terms in a nonclassical situation. Russian Math. Surveys, 37(6):151-175. Saff, E. B. and Totik, V. (1997). Logarithmic potentials with external fields, volume 316 of Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Springer-Verlag, Berlin. Appendix B by Thomas Bloom. Saloff-Coste, L. (1997). Lectures on finite Markov Chains. In Bernard, P., editor, Lectures on Probability Theory and Statistics, St-Flour 1996, number 1665 in Lecture Notes in Mathematics, pages 301-413. Springer-Verlag. Samorodnitsky, G. and Taqqu, M. (1994). Stable non- Gaussian random processes; stochastic models with infinite variance. Chapman & Hall, New York. Shannon, C. (1948). A mathematical theory of communication. Bell System Tech. J . , 271379-423, 623-656. Shannon, C. and Weaver, W. (1949). A Mathematical Theory of Communication. University of Illinois Press, Urbana, IL. Shimizu, R. (1975). On Fisher's amount of information for location family. In G.P.Pati1 et al, editor, Statistical Distributions in Scientific Work, Volume 3, pages 305-312. Reidel. Shlosman, S. (1980). Limit theorems of probability theory for compact topological groups. Theory Probab. Appl., 25(3):604-609. Shlosman, S . (1984). The influence of non-commutativity on limit theorems. 2. Wahrsch. Verw. Gebiete, 65(4):627-636. Speicher, R. (1994). Multiplicative functions on the lattice of noncrossing partitions and free convolution. Math. Ann., 298(4):611-628. Stam, A. (1959). Some inequalities satisfied by the quantities of information of Fisher and Shannon. Information and Control, 2:lOl-112. Stromberg, K. (1960). Probabilities on a compact group. Trans. Amer. Math. SOC.,94:295-309. Szego, G . (1958). Orthgonal Polynomials. American Mathematical Society, New York, revised edition. Takano, S. (1987). Convergence of entropy in the Central Limit Theorem. Yokohama Math. J., 35(1-2):143-148. Takano, S. (1996). The inequalities of Fisher Information and Entropy Power for dependent variables. In Watanabe, S., Fukushima, M., Prohorov, Y., and Shiryaev, A., editors, Proceedings of the 7th Japan-Russia Symposium on Probability Theory and Mathematical Statistics, Tokyo 26-30 July 1995, pages 460-470, Singapore. World Scientific. Takano, S. (1998). Entropy and a limit theorem for some dependent variables. In Prague Stochastics '98, volume 2 , pages 549-552. Union of Czech Mathematicians and Physicists. Topsoe, F. (1979). Information-theoretical optimization techniques. Kybernetika
Bibliography
205
(Prague), 15(1):8-27. Truesdell, C . (1980). The Tragicomical History of Thermodynamics 1822-1854. Springer-Verlag, New York. Uchaikin, V. and Zolotarev, V. (1999). Chance and stability. Modern Probability and Statistics. VSP, Utrecht. Stable distributions and their applications, With a foreword by V.Y. Korolev and Zolotarev. Uffink, J. (2001). Bluff your way in the Second Law of Thermodynamics. Stud. Hist. Philos. Sci. B Stud. Hist. Philos. Modern Phys., 32(3):305-394. Utev, S . (1992). An application of integrodifferential inequalities in probability theory. Siberian Adw. Math., 2(4):164-199. Vilenkin, P. and Dyachkov, A. (1998). Asymptotics of Shannon and Rknyi entropies for sums of independent random variables. Problems Inform. Transmission, 34(3):2 19-232. Voiculescu, D. (1985). Symmetries of some reduced free product C*-algebras. In Operator algebras and their connectzons with topology and ergodic theory (Bugteni, 19831, volume 1132 of Lecture Notes in Math., pages 556-588. Springer, Berlin. Voiculescu, D. (1991). Limit laws for random matrices and free products. Invent. Math., 104(1):201-220. Voiculescu, D. (1993). The analogues of entropy and of Fisher’s information measure in free probability theory. I. Comm. Math. Phys., 155(1):71-92. Voiculescu, D. (1994). The analogues of entropy and of Fisher’s information measure in free probability theory. 11. Invent. Math., 118(3):411-440. Voiculescu, D. (1997). The analogues of entropy and of Fisher’s information measure in free probability theory. IV. Maximum entropy and freeness. In Free probability theory (Waterloo, ON, 1995), pages 293-302. Amer. Math. Soc., Providence, RI. Voiculescu, D. (1998). The analogues of entropy and of Fisher’s information measure in free probability theory. V. Noncommutative Hilbert transforms. Invent. Math., 132(1):189-227. Voiculescu, D. (2000). The coalgebra of the free difference quotient and free probability. Internat. Math. Res. Notices, ZOOO(2) :79-106. Voiculescu, D., Dykema, K., and Nica, A. (1992). Free random variables. American Mathematical Society, Providence, RI. A noncommutative probability approach t o free products with applications to random matrices, operator algebras and harmonic analysis on free groups. Volkonskii, V. and Rozanov, Y. (1959). Some limit theorems for random functions I. Theory Probab. Appl., 4:178-197. von Bahr, B. (1965). On the convergence of moments in the central limit theorem. A n n . Math. Statist., 36:808-818. Eric Weisstein’s World of Mathematics. See Weisstein, E. (2003). htt p://mat hworld.wolfram.com. Wigner, E. (1955). Characteristic vectors of bordered matrices with infinite dimensions. Ann. of Math. (2), 62:548-564. Wigner, E. (1958). On the distribution of the roots of certain symmetric matrices. Ann. of Math. (2), 67:325-327.
206
Information Theory and the Central Limit Theorem
Zamir, R. (1998). A proof of the Fisher information inequality via a data processing argument. ZEEE Trans. Z n f o m . Theory, 44(3):1246-1250. Zolotarev, V. (1986). One-dimensional Stable Distributions, volume 65 of Translations of Mathematical Monographs. American Mathematical Society, Providence. Zolotarev, V. (1994). On representation of densities of stable laws by special functions. Theory Pro bab. A ppl., 39(2):354-362.
Index
adjoint, 36, 106, 178, 179
rate of, 47-54, 125 uniform, 114 CramCr’s theorem, 9 CramCr-Rao lower bound, 23, 51-54, 57, 105, 164
Barron, 32, 41-43, 166, 191-193 Bernoulli distribution, 2, 10, 129, 130, 132, 140 entropy of, 2 Berry-Esseen theorem, 49-51 bound on density, 41, 79, 107-108 Brown inequality, 38, 105 Burger equation, 166
de Bruijn identity, 24, 48, 66, 73, 96, 97, 132, 144, 165, 183-187, 197 proof, 183 dependent random variables, 69-85, 140-142 derivation, 161- 162 differential entropy definition, 6 properties, 6-7 discrete-valued random variables, 129-151 domain of normal attraction, 89, 96, 104
Cauchy distribution, 88, 89, 96, 99, 104, 105, 180 entropy of, 92, 175 Cauchy transform, 92, 157, 163 Central Limit Theorem, 12, 20, 29-30, 69, 87-90, 153, 161 characteristic function, 30, 88, 91, 97, 114 Chebyshev polynomials, 105, 155, 156, 180 Chebyshev’s inequality, 42, 75, 82, 83, 108, 192 compact groups, see groups conditional expectation, 33-35, 65, 138-140, 162 conjugate functions, 36, 161 convergence in Fisher information, 31, 46-49, 63, 72, 76-77, 140, 169 in relative entropy, 42, 49, 73, 118, 125, 134, 169
eigenfunctions, 38-41, 106, 178-180 entropy axioms, 16 behaviour on convolution, 16 conditional, 14 definition, 2, 164 differential, see differential entropy joint, 4 maximum, see maximum entropy properties, 3 RCnyi a-, 17, 133-135, 188 relative, see relative entropy 207
208
Information Theory and the Central Limit Theorem
thermodynamic, see thermodynamic entropy Entropy Power inequality, 119, 133, 187-189, 196 equicontinuous, 113 excess kurtosis, 52 exponential distribution as maximum entropy distribution, 13. 91 Fisher information behaviour on convolution, 26 convergence in, see convergence in Fisher information definition, 21, 163 distance, 23 lower bound, see CramCr-Rao lower bound matrix, 64 properties, 21-27 standardised, 24 free energy, 18 free probability, 153-169 convolution, 159 freeness, 158
Hermite polynomials, 39, 45, 52, 105, 150, 179 Hilbert transform, 163 IID variables, 33-54 Individual Smallness condition, 56, 63 information, 1 information discrimination, see relative entropy information divergence, see relative entropy Ito-Kawada theorem, 112, 114, 118 Kraft inequality, 5 Kullback-Leibler distance, see relative entropy Lkvy decomposition, 88, 90 Lkvy distribution, 89, 93, 96, 99, 105, 180 entropy of, 91, 174 large deviations, 9 law of large numbers, 88 law of small numbers, 129 Lindeberg condition, 30, 55, 63 Individual, 57 Lindeberg-Feller theorem, 30, 56 Linnik, 29, 55-57, 77 log-Sobolev inequality, 143, 197 log-sum inequality, 10, 100, 194
gamma distribution, 22, 172 entropy of, 172 Fisher information of, 22, 53 Gaussian distribution, 6, 21 as maximum entropy distribution, 12, 91 entropy of, 6 Fisher information of, 21 multivariate, 6, 65 geometric distribution, 2 entropy of, 2 Gibbs inequality, 8, 11, 111 Gibbs states, 19 groups probability on, 109-128
normal distribution, see Gaussian distribution
Haar measure, 110 Hausdorff-Young inequality, 133, 188 heat equation, 66, 166, 183, 184
orthogonal polynomials, see Hermite polynomials, Poisson-Charlier polynomials, Chebyshev
maximum entropy, 4, 12, 13, 111, 119, 130, 165 mixing coefficients, 69-72, 83-85 a , 69, 71, 74 6 4 , 72, 74 4, 71 *, 71 mutual information, 14, 120
Index
polynomials
209
subadditivity, 26, 47, 77, 140, 178
parameter estimation, 13, 92-95, 131 Poincard constant, 37, 43, 44, 57, 166, 177-181 behaviour on convolution, 46 restricted, 44, 67 Poisson distribution, 131 as maximum entropy distribution, 130 convergence to, 129-151 Poisson-Charlier polynomials, 146-1 48 projection inequalities, 43-44, 57-61, 66-67, 166
thermodynamic entropy, 3, 17-20 topological groups, see groups
R-transform, 160-161 Rknyi’s method, 27-29, 31, 109 random vectors, 64-68 relative entropy, 96, 111 convergence in, see convergence in relative entropy definition, 8 properties, 8-13
Wigner distribution, 153, 155-157, 160, 161, 163, 165, 166 as maximum entropy distribution, 165
score function, 21, 34, 36, 65, 78, 138, 140, 141, 162 behaviour on convolution, 34, 65, 78, 138, 140, 141, 162 Second Law of Thermodynamics, 19-20 semicircular distribution, see Wigner distribution size-biased distribution, 139 skewness, 52 source coding, 4-5 spectral gap, see Poincare constant stable distributions, 87-108, 173 entropy of, 91, 173 Stein identity, 22, 23, 35, 36, 44, 46, 65, 67, 78, 137, 162 Stein’s lemma, 9 Stein’s method, 22, 139 Stirling’s formula, 18, 131, 173 strong mixing, see mixing coefficients, Ly
uniform distribution, 4, 6, 111 as maximum entropy distribution, 4,111 entropy of, 6 uniform integrability, 43, 49, 70, 76 uniform mixing, see mixing coefficients, q5 vectors, see random vectors von Neumann algebra, 154
