A of λ, the greater the amount of shrinkage. The quadratic ridge penalty term makes βˆ a linear function of y. Frank
Absolute Penalty Estimation Ejaz S. Ahmed , Enayetur Raheem , Shakhawat Hossain Professor and Department Head of Mathematics and Statistics University of Windsor, Windsor, ON, Canada University of Windsor, Windsor, ON, Canada
and Friedman () introduced bridge regression, a generalized version of penalty (or absolute penalty type) estimation, which includes ridge regression when γ = . For a given penalty function π(⋅) and regularization parameter λ, the general form can be written as
In statistics, the technique of 7least squares is used for estimating the unknown parameters in a linear regression model (see 7Linear Regression Models). This method minimizes the sum of squared distances between the observed responses in a set of data, and the fitted responses from the regression model. Suppose we observe a collection of data {yi , xi }ni= on n units, where yi s are responses and xi = (xi , xi , . . . , xip )T is a vector of predictors. It is convenient to write the model in matrix notation, as,
where the penalty function is of the form
y = Xβ + ε,
ϕ(β) = (y − X β)T (y − X β) + λπ(β),
()
where y is n × vector of responses, X is n × p matrix, known as the design matrix, β = (β , β , . . . , β p )T is the unknown parameter vector and ε is the vector of random errors. In ordinary least squares (OLS) regression, we estimate β by minimizing the residual sum of squares, RSS = (y − Xβ)T (y − Xβ), giving βˆ OLS = (X TX)− X Ty. This estimator is simple and has some good statistical properties. However, the estimator suffers from lack of uniqueness if the design matrix X is less than full rank, and if the columns of X are (nearly) collinear. To achieve better prediction and to alleviate ill conditioning problem of X T X, Hoerl and Kernard () introduced ridge regression (see 7Ridge and Surrogate Ridge Regressions), which minimizes the RSS subject to a constraint, ∑ β j ≤ t, in other words ⎧ ⎫ p p ⎪ ⎪ ridge ⎪N ⎪ βˆ = argmin ⎨∑(yi − β − ∑ xij β j ) + λ ∑ β j ⎬ , ⎪ ⎪ ⎪ ⎪ β j= j= ⎩ i= ⎭ () where λ ≥ is known as the complexity parameter that controls the amount of shrinkage. The larger the value
p
π(β) = ∑ ∣β j ∣γ , γ > .
()
j=
The penalty function in () bounds the Lγ norm of the γ parameters in the given model as ∑m j= ∣β j ∣ ≤ t, where t is the tuning parameter that controls the amount of shrinkage. We see that for γ = , we obtain ridge regression. However, if γ ≠ , the penalty function will not be rotationally invariant. Interestingly, for γ < , it shrinks the coefficient toward zero, and depending on the value of λ, it sets some of them to be exactly zero. Thus, the procedure combines variable selection and shrinkage of coefficients of penalized regression. An important member of the penalized least squares (PLS) family is the L penalized least squares estimator or the lasso [least absolute shrinkage and selection operator, Tibshirani ()]. In other words, the absolute penalty estimator (APE) arises when the absolute value of penalty term is considered, i.e., γ = in (). Similar to the ridge regression, the lasso estimates are obtained as ⎧ ⎫ p p ⎪ ⎪ lasso ⎪ ⎪n βˆ = argmin ⎨∑(yi − β − ∑ xij β j ) + λ ∑ ∣β j ∣⎬ . ⎪ ⎪ ⎪ ⎪ β j= j= ⎩ i= ⎭ () The lasso shrinks the OLS estimator toward zero and depending on the value of λ, it sets some coefficients to exactly zero. Tibshirani () used a quadratic lasso programming method to solve () for βˆ . Later,
Efron et al. () proposed least angle regression (LAR), a type of stepwise regression, with which the
Miodrag Lovric (ed.), International Encyclopedia of Statistical Science, DOI ./----, © Springer-Verlag Berlin Heidelberg
A
Absolute Penalty Estimation
lasso estimates can be obtained at the same computational cost as that of an ordinary least squares estimation Hastie et al. (). Further, the lasso estimator remains numerically feasible for dimensions m that are much higher than the sample size n. Zou and Hastie () introduced a hybrid PLS regression with p the so called elastic net penalty defined as λ ∑j= (αβ j + ( − α) ∣β j ∣). Here the penalty function is a linear combination of the ridge regression penalty function and lasso penalty function. A different type of PLS, called garotte is due to Breiman (). Further, PLS estimation provides a generalization of both nonparametric least squares and weighted projection estimators, and a popular version of the PLS is given by Tikhonov regularization (Tikhonov ). Generally speaking, the ridge regression is highly efficient and stable when there are many small coefficients. The performance of lasso is superior when there are a small-to-medium number of moderatesized coefficients. On the other hand, shrinkage estimators perform well when there are large known zero coefficients. Ahmed et al. () proposed an APE for partially linear models. Further, they reappraised the properties of shrinkage estimators based on Stein-rule estimation. There exists a whole family of estimators that are better than OLS estimators in regression models when the number of predictors is large. A partially linear regression model is defined as yi = xTi β + g(ti ) + ε i , i = , . . . , n,
()
where ti ∈ [, ] are design points, g(⋅) is an unknown real-valued function defined on [, ], and yi , x, β, and ε i ’s are as defined in the context of (). We consider experiments where the vector of coefficients β in the linear part of () can be partitioned as (β T , β T )T , where β is the coefficient vector of order p × for main effects (e.g., treatment effects, genetic effects) and β is a vector of order p × for “nuisance” effects (e.g., age, laboratory). Our relevant hypothesis is H : β = . Let βˆ be a semiparametric least squares estimator of β , and we let β˜ denote the restricted semiparametric least squares estimator of β . Then the semiparametric Stein-type estimator (see 7James-Stein Estimator and Semiparametric RegresS sion Models), βˆ , of β is S − βˆ = β˜ + { − (p − )T }( βˆ − β˜ ),
p ≥
()
where T is an appropriate test statistic for the H . S+ A positive-rule shrinkage estimator (PSE) βˆ is defined as βˆ
S+
= β˜ + { − (p − )T − }+ (βˆ − β˜ ),
p ≥
()
where z+ = max(, z). The PSE is particularly important to S control the over-shrinking inherent in βˆ . The shrinkage estimators can be viewed as a competitor to the APE approach. Ahmed et al. () finds that, when p is relatively small with respect to p, APE performs better than the shrinkage method. On the other hand, the shrinkage method performs better when p is large, which is consistent with the performance of the APE in linear models. Importantly, the shrinkage approach is free from any tuning parameters, easy to compute and calculations are not iterative. The shrinkage estimation strategy can be extended in various directions to more complex problems. It may be worth mentioning that this is one of the two areas Bradley Efron predicted for the early twenty-first century (RSS News, January ). Shrinkage and likelihood-based methods continue to be extremely useful tools for efficient estimation.
About the Author The author S. Ejaz Ahmed is Professor and Head Department of Mathematics and Statistics. For biography, see entry 7Optimal Shrinkage Estimation.
Cross References 7Estimation 7Estimation: An Overview 7James-Stein Estimator 7Linear Regression Models 7Optimal Shrinkage Estimation 7Residuals 7Ridge and Surrogate Ridge Regressions 7Semiparametric Regression Models
References and Further Reading Ahmed SE, Doksum KA, Hossain S, You J () Shrinkage, pretest and absolute penalty estimators in partially linear models. Aust NZ J Stat ():– Breiman L () Better subset selection using the non-negative garotte. Technical report, University of California, Berkeley Efron B, Hastie T, Johnstone I, Tibshirani R () Least angle regression (with discussion). Ann Stat ():– Frank IE, Friedman JH () A statistical view of some chemometrics regression tools. Technometrics :– Hastie T, Tibshirani R, Friedman J () The elements of statistical learning: data mining, inference, and prediction, nd edn. Springer, New York Hoerl AE, Kennard RW () Ridge regression: biased estimation for nonorthogonal problems. Technometrics :– Tibshirani R () Regression shrinkage and selection via the lasso. J R Stat Soc B :–
Accelerated Lifetime Testing
Tikhonov An () Solution of incorrectly formulated problems and the regularization method. Soviet Math Dokl :– , English translation of Dokl Akad Nauk SSSR , , – Zou H, Hastie T () Regularization and variable selction via the elastic net. J R Stat Soc B ():–
Accelerated Lifetime Testing Francisco Louzada-Neto Associate Professor Universidade Federal de São Carlos, Sao Paulo, Brazil
by a lifetime distribution, such as exponential, Weibull, log-normal, log-logistic, among others. The other is a stress-response relationship (SRR), which relates the mean lifetime (or a function of this parameter) with the stress levels. Common SRRs are the power law, Eyring and Arrhenius models (Meeker and Escobar ) or even a general log-linear or log-non-linear SRR which encompass the formers. For sake of illustration, we shall assume an exponential distribution as the lifetime model and a general log-linear SRR. Here, the mean lifetime under the usual working conditions shall represent our device reliability measure of interesting. Let T > be the lifetime random variable with an exponential density f (t, λ i ) = λ i exp {−λ i t} ,
Accelerated life tests (ALT) are efficient industrial experiments for obtaining measures of a device reliability under the usual working conditions. A practical problem for industries of different areas is to obtain measures of a device reliability under its usual working conditions. Typically, the time and cost of such experimentation are long and expensive. The ALT are efficient for handling such situation, since the information on the device performance under the usual working conditions are obtained by considering a time and cost-reduced experimental scheme. The ALT are performed by testing items at higher stress covariate levels than the usual working conditions, such as temperature, pressure and voltage. There is a large literature on ALT and interested readers can refer to Mann et al. (), Nelson (), Meeker and Escobar () which are excellent sources for ALT. Nelson (a, b) provides a brief background on accelerated testing and test plans and surveys the related literature point out more than related references. A simple ALT scenario is characterized by putting k groups of ni items each under constant and fixed stress covariate levels, Xi (hereafter stress level), for i = , . . . , k, where i = generally denotes the usual stress level, that is, the usual working conditions. The experiment ends after a certain pre-fixed number ri < ni of failures, ti , ti , . . . , tiri , at each stress level, characterizing a type II censoring scheme (Lawless ; see also 7Censoring Methodology). Other stress schemes, such as step (see 7Step-Stress Accelerated Life Tests) and progressive ones, are also common in practice but will not be considered here. Examples of those more sophisticated stress schemes can be found in Nelson (). The ALT models are composed by two components. One is a probabilistic component, which is represented
A
()
where λ i > is an unknown parameter representing the constant failure rate for i = , . . . , k (number of stress levels). The mean lifetime is given by θ i = /λ i . The likelihood function for λ i , under the i-th stress level Xi , is given by ⎛ ri ⎞ Li (λ i ) = ∏ f (tij , λ i ) (S(tiri , λ i ))ni −ri = λ ri i exp {−λ i Ai } , ⎝ j= ⎠ where S(tiri , λ i ) is the survival function at tiri and Ai = ri tij + (ni − ri )tiri denotes the total time on test for the ∑j= i-th stress level. Considering data under the k random stress levels, the likelihood function for the parameter vector λ = (λ , λ , . . . , λ k ) is given by k
L(λ) = ∏ λ ri i exp {−λ i Ai } .
()
i=
We consider a general log-linear SRR defined as λ i = exp(−Zi − β − β Xi ),
()
where X is the covariate, Z = g(X) and β and β are unknown parameters such that −∞ < β , β < ∞. The SRR () has several models as particular cases. The Arrhenius model is obtained if Zi = , Xi = /Vi, β =−α and β = α , where Vi denotes a level of the temperature variable. If Zi = , Xi = −log(Vi), β = log(α) and β = α , where Vi denotes a level of the voltage variable we obtain the power model. Following Louzada-Neto and Pardo-Fernandéz (), the Eyring model is obtained if Zi = − log Vi , Xi = /Vi, β = −α and β = α , where Vi denotes a level of the temperature variable. Interested readers can refer to Meeker and Escobar () for more information about the physical models considered here.
A
A
Accelerated Lifetime Testing
From () and (), the likelihood function for β and β is given by
Two types of software for ALT are provided by Meeker and Escobar () and ReliaSoft Corporation ().
k
L(β , β ) = ∏{exp(−Zi − β − β Xi )ri
About the Author
i=
exp(− exp(−Zi − β − β Xi )Ai )}.
()
The maximum likelihood estimates (MLEs) of β and β can be obtained by direct maximization of (), or by solving the system of nonlinear equations, ∂ log L/∂θ = , where θ ′ = (β , β ). Obtaining the score function is conceptually simple and the expressions are not given explicitly. The MLEs of θ i can be obtained, in principle, straightforwardly by considering the invariance property of the MLEs. Large-sample inference for the parameters can be based on the MLEs and their estimated variances, obtained by inverting the expected information matrix (Cox and Hinkley ). For small or moderate-sized samples however we may consider simulation approaches, such as the bootstrap confidence intervals (see 7Bootstrap Methods) that are based on the empirical evidence and are therefore preferred (Davison and Hinkley ). Formal goodnessof-fit tests are also feasible since, from (), we can use the likelihood ratio statistics (LRS) for testing goodness-of-fit of hypotheses such as H : β = . Although we considered only an exponential distribution as our lifetime model, more general lifetime distributions, such as the Weibull (see 7Weibull Distribution and Generalized Weibull Distributions), log-normal, log-logistic, among others, could be considered in principle. However, the degree of difficulty in the calculations increase considerably. Also we considered only one stress covariate, however this is not critical for the overall approach to hold and the multiple covariate case can be handle straightforwardly. A study on the effect of different reparametrizations on the accuracy of inferences for ALT is discussed in LouzadaNeto and Pardo-Fernandéz ). Modeling ALT with a log-non-linear SRR can be found in Perdoná et al. (). Modeling ALT with a threshold stress, below which the lifetime of a product can be considered to be infinity or much higher than that for which it has been developed is proposed by Tojeiro et al. (). We only considered ALT in presence of constant stress loading, however non-constant stress loading, such as step stress and linearly increasing stress are provided by Miller and Nelson () and Bai, Cha and Chung (), respectively. A comparison between constant and step stress tests is provided by Khamis (). A log-logistic step stress model is provided by Srivastava and Shukla ().
Francisco Louzada-Neto is an associate professor of Statistics at Universidade Federal de São Carlos (UFSCar), Brazil. He received his Ph.D in Statistics from University of Oxford (England). He is Director of the Centre for Hazard Studies (–, UFSCar, Brazil) and Editor in Chief of the Brazilian Journal of Statistics (–, Brazil). He is a past-Director for Undergraduate Studies (–, UFSCar, Brazil) and was Director for Graduate Studies in Statistics (–, UFSCar, Brazil). Louzada-Neto is single and joint author of more than publications in statistical peer reviewed journals, books and book chapters, He has supervised more than assistant researches, Ph.Ds, masters and undergraduates.
Cross References 7Degradation Models in Reliability and Survival Analysis 7Modeling Survival Data 7Step-Stress Accelerated Life Tests 7Survival Data
References and Further Reading Bai DS, Cha MS, Chung SW () Optimum simple ramp tests for the Weibull distribution and type-I censoring. IEEE T Reliab :– Cox DR, Hinkley DV () Theoretical statistics. Chapman and Hall, London Davison AC, Hinkley DV () Bootstrap methods and their application. Cambridge University Press, Cambridge Khamis IH () Comparison between constant- and step-stress tests for Weibull models. Int J Qual Reliab Manag :– Lawless JF () Statistical models and methods for lifetime data, nd end. Wiley, New York Louzada-Neto F, Pardo-Fernandéz JC () The effect of reparametrization on the accuracy of inferences for accelerated lifetime tests. J Appl Stat :– Mann NR, Schaffer RE, Singpurwalla ND () Methods for statistical analysis of reliability and life test data. Wiley, New York Meeker WQ, Escobar LA () Statistical methods for reliability data. Wiley, New York Meeker WQ, Escobar LA () SPLIDA (S-PLUS Life Data Analysis) software–graphical user interface. http://www.public. iastate.edu/~splida Miller R, Nelson WB () Optimum simple step-stress plans for accelerated life testing. IEEE T Reliab :– Nelson W () Accelerated testing – statistical models, test plans, and data analyses. Wiley, New York Nelson W (a) A bibliography of accelerated test plans. IEEE T Reliab :– Nelson W (b) A bibliography of accelerated test plans part II – references. IEEE T Reliab :–
Acceptance Sampling
Perdoná GSC, Louzada Neto F, Tojeiro CAV () Bayesian modelling of log-non-linear stress-response relationships in accelerated lifetime tests. J Stat Theory Appl ():– Reliasoft Corporation () Optimum allocations of stress levels and test units in accelerated tests. Reliab EDGE :–. http://www.reliasoft.com Srivastava PW, Shukla R () A log-logistic step-stress model. IEEE T Reliab :– Tojeiro CAV, Louzada Neto F, Bolfarine H () A Bayesian analysis for accelerated lifetime tests under an exponential power law model with threshold stress. J Appl Stat ():–
Acceptance Sampling M. Ivette Gomes Professor Universidade de Lisboa, DEIO and CEAUL, Lisboa, Portugal
Introduction Acceptance sampling (AS) is one of the oldest statistical techniques in the area of 7statistical quality control. It is performed out of the line production, most commonly before it, for deciding on incoming batches, but also after it, for evaluating the final product (see Duncan ; Stephens ; Pandey ; Montgomery ; and Schilling and Neubauer , among others). Accepted batches go into the production line or are sold to consumers; the rejected ones are usually submitted to a rectification process. A sampling plan is defined by the size of the sample (samples) taken from the batch and by the associated acceptance–rejection criterion. The most widely used plans are given by the Military Standard tables, developed during the World War II, and first issued in . We mention MIL STD E () and the civil version ANSI/ASQC Z. () of the American National Standards Institution and the American Society for Quality Control. At the beginning, all items and products were inspected for the identification of nonconformities. At the late s, Dodge and Romig (see Dodge and Romig ), in the Bell Laboratories, developed the area of AS, as an alternative to % inspection. The aim of AS is to lead producers to a decision (acceptance or rejection of a batch) and not to the estimation or improvement of the quality of a batch. Consequently, AS does not provide a direct form of quality control, but its indirect effects in quality are important: if a batch is rejected, either the supplier tries improving its production methods or the consumer (producer) looks for a better supplier, indirectly increasing quality.
A
Regarding the decision on the batches, we distinguish three different approaches: () acceptance without inspection, applied when the supplier is highly reliable; () % inspection, which is expensive and can lead to a sloppy attitude towards quality; () an intermediate decision, i.e., an acceptance sampling program. This increases the interest on quality and leads to the lemma: make things right in the first place. The type of inspection that should be applied depends on the quality of the last batches inspected. At the beginning of inspection, a so-called normal inspection is used, but there are two other types of inspection, a tightened inspection (for a history of low quality), and a reduced inspection (for a history of high quality). There are special and empirical switching rules between the three types of inspection, as well as for discontinuation of inspection.
Factors for Classifications of Sampling Plans Sampling plans by attributes versus sampling plans by variables. If the item inspection leads to a binary result (conforming or nonconforming), we are dealing with sampling by attributes, detailed later on. If the item inspection leads to a continuous measurement X, we are sampling by variables. Then, we generally use sampling plans based on the sample mean and standard deviation, the so-called variable sampling plans. If X is normal, it is easy to compute the number of items to be collected and the criteria that leads to the rejection of the batch, with chosen risks α and β. For different sampling plans by variables, see Duncan (), among others. Incoming versus outgoing inspection. If the batches are inspected before the product is sent to the consumer, it is called outgoing inspection. If the inspection is done by the consumer (producer), after they were received from the supplier, it is called incoming inspection. Rectifying versus non-rectifying sampling plans. All depends on what is done with nonconforming items that were found during the inspection. When the cost of replacing faulty items with new ones, or reworking them is accounted for, the sampling plan is rectifying. Single, double, multiple and sequential sampling plans. ●
Single sampling. This is the most common sampling plan: we draw a random sample of n items from the batch, and count the number of nonconforming items (or the number of nonconformities, if more than one nonconformity is possible on a single item). Such a
A
A
●
●
●
Acceptance Sampling
plan is defined by n and by an associated acceptancerejection criterion, usually a value c, the so-called acceptance number, the number of nonconforming items that cannot be exceeded. If the number of nonconforming items is greater than c, the batch is rejected; otherwise, the batch is accepted. The number r, defined as the minimum number of nonconforming items leading to the rejection of the batch, is the so-called rejection number. In the most simple case, as above, r = c + , but we can have r > c + . Double sampling. A double sampling plan is characterized by four parameters: n << n, the size of the first sample, c the acceptance number for the first sample, n the size of the second sample and c (> c ) the acceptance number for the joint sample. The main advantage of a double sampling plan is the reduction of the total inspection and associated cost, particularly if we proceed to a curtailment in the second sample, i.e. we stop the inspection whenever c is exceeded. Another (psychological) advantage of these plans is the way they give a second opportunity to the batch. Multiple sampling. In the multiple plans a predetermined number of samples are drawn before taking a decision. 7Sequential sampling. The sequential plans are a generalization of multiple plans. The main difference is that the number of samples is not pre-determined. If, at each step, we draw a sample of size one, the plan, based on Wald’s test, is called sequential item-to-item; otherwise, it is sequential by groups. For a full study of multiple and sequential plans see, for instance, Duncan () (see also the entry 7Sequential Sampling).
Special sampling plans. Among the great variety of special plans, we distinguish: ●
●
Chain sampling. When the inspection procedures are destructive or very expensive, a small n is recommendable. We are then led to acceptance numbers equal to zero. This is dangerous for the supplier and if rectifying inspection is used, it is expensive for the consumer. In , Dodge suggested a procedure alternative to this type of plans, which uses also the information of preceding batches, the so-called chain sampling method (see Dogdge and Romig ). Continuous sampling plans (CSP). There are continuous production processes, where the raw material is not naturally provided in batches. For this type of production it is common to alternate sequences of sampling inspection with % inspection – they are in a certain sense rectifying plans. The simplest plan of this type, the CSP-, was suggested in by Dodge. It begins
with a % inspection. When a pre-specified number i of consecutive nonconforming items is achieved, the plan changes into sampling inspection, with the inspection of f items, randomly selected, along the continuous production. If one nonconforming item is detected (the reason for the terminology CSP-), % inspection comes again, and the nonconforming item is replaced. For properties of this plan and its generalizations see Duncan ().
A Few Characteristics of a Sampling Plan OCC. The operational characteristic curve (OCC) is Pa ≡ Pa (p) = P(acceptance of the batch ∣ p), where p is the probability of a nonconforming item in the batch. AQL and LTPD (or RQL). The sampling plans are built taken into account the wishes of both the supplier and the consumer, defining two quality levels for the judgment of the batches: the acceptance quality level (AQL), the worst operating quality of the process which leads to a high probability of acceptance of the batch, usually % – for the protection of the supplier regarding high quality batches, and the lot tolerance percent defective (LTPD) or rejectable quality level (RQL), the quality level below which an item cannot be considered acceptable. This leads to a small acceptance of the batch, usually % – for the protection of the consumer against low quality batches. There exist two types of decision, acceptance or rejection of the batch, and two types of risks, to reject a “good" (high quality) batch, and to accept a “bad" (low quality) batch. The probabilities of occurrence of these risks are the so-called supplier risk and consumer risk, respectively. In a single sampling plan, the supplier risk is α = − Pa (AQL) and the consumer risk is β = Pa (LTPD). The sampling plans should take into account the specifications AQL and LTPD, i.e. we are supposed to find a single plan with an OCC that passes through the points (AQL, -α) and (LTPD, β). The construction of double plans which protect both the supplier and the consumer are much more difficult, and it is no longer sufficient to provide indication on two points of the OCC. There exist the so-called Grubbs’ tables (see Montgomery ) providing (c , c , n , n ), for n = n , as an example, α = ., β = . and several rates RQL/AQL. AOQ, AOQL and ATI. If there is a rectifying inspection program – a corrective program, based on a % inspection and replacement of nonconforming by conforming items, after the rejection of a batch by an AS plan –, the most relevant characteristics are the average outgoing quality (AOQ), AOQ(p) = p ( − n/N) Pa , which attains
Actuarial Methods
a maximum at the so-called average output quality limit (AOQL), the worst average quality of a product after a rectifying inspection program, as well as the average total inspection (ATI), the amount of items subject to inspection, equal to n if there is no rectification, but given by ATI(p) = nPa + N( − Pa ), otherwise.
Acknowledgments Research partially supported by FCT/OE, POCI and PTDC/FEDER.
About the Author For biography of M. Ivette Gomes see the entry 7Statistical Quality Control.
Cross References 7Industrial Statistics 7Sequential Sampling 7Statistical Quality Control 7Statistical Quality Control: Recent Advances
References and Further Reading Dodge HF, Romig HG () Sampling inspection tables, single and double sampling, nd edn. Wiley, New York Duncan AJ () Quality control and industrial statistics, th edn. Irwin, Homehood Montgomery DC () Statistical quality control: a modern introduction, th edn. Wiley, Hoboken, NJ Pandey BN () Statistical techniques in life-testing, reliability, sampling theory and quality control. Narosa, New Delhi Schilling EG, Neubauer DV () Acceptance sampling in quality control, nd edn. Chapman and Hall/CRC, New York Stephens KS () The handbook of applied acceptance sampling: plans, principles, and procedures. ASQ Quality, Milwaukee
Actuarial Methods Vassiliy Simchera Director Rosstat’s Statistical Research Institute, Moscow, Russia A specific (and relatively new) type of financial calculations are actuarial operations, which represent a special (in majority of countries they are usually licensed) sphere of activity related to identifications of risks outcomes and market assessment of future (temporary) borrowed current assets and liabilities costs for their redemption.
A
The broad range of existing and applicable actuarial calculations require use of various methods and inevitably predetermines a necessity of their alteration depending on concrete cases of comparison analysis and selection of most efficient of them. The condition of success is a typology of actuarial calculations methods, based on existing typology fields and objects of their applications, as well as knowledge of rule for selection of most efficient methods, which would provide selection of target results with minimum costs or high accuracy. Regarding the continuous character of financial transactions, the actuarial calculations are carried out permanently. The aim of actuarial calculations in every particular case is probabilistic determination of profit sharing (transaction return) either in the form of financial liabilities (interest, margin, agio, etc.) or as commission charges (such as royalty). The subject of actuarial calculations can be distinguished in the narrow and in the broad senses. The given subject in the broad sense covers financial and actuarial accounts, budgeting, balance, audit, assessment of financial conditions and financial provision for all categories and types of borrowing institutions, basis for their preferential financial decisions and transactions, conditions and results of work for different financial and credit institutions; financial management of cash flows, resources, indicators, mechanisms, instruments, as well as financial analysis and audit of financial activity of companies, countries, nations their groups and unions, including national system of financial account, financial control, engineering, and forecast. In other words, the subject of actuarial calculations is a process of determination of any expenditures and incomes from any type of transactions in the shortest way. In the narrow sense it is a process of determination, in the same way, of future liabilities and their comparison with present assets in order to estimate their sufficiency, deficit of surplus. We can define general and efficient actuarial calculations, the principals of which are given below. Efficient actuarial calculations imply calculations of any derivative indicators, which are carried out through conjugation (comparison) of two or more dissimilar initial indicators, the results of which are presented as different relative numbers (coefficients, norms, percents, shares, indices, rates, tariffs, etc.), characterizing differential (effect) of anticipatory increment of one indicator in comparison with another one. In some cases similar values are called gradients, derivatives (of different orders), elasticity coefficients, or
A
A
Actuarial Methods
anticipatory coefficients and can be determined by reference to more complex statistical and mathematical methods including geometrical, differential, integral, and correlation and regression multivariate calculations. Herewith in case of application of nominal comparison scales for two or more simple values (so called scale of simple interests, which are calculated and represented in terms of current prices) they are determined and operated as it was mentioned by current nominal financial indicators, but in case of real scales application, i.e. scales of so called compound interests, they are calculated and represented in terms of future or current prices, that is real efficient financial indicators. In case of insurance scheme the calculation of efficient financial indicators signify the special type of financial calculations i.e. actuarial calculations, which imply additional profit (discounts) or demanding compensation of loss (loss, damage or loss of profit) in connection with occurrence of contingency and risks (risk of legislation alteration, exchange rates, devaluation or revaluation, inflation or deflation, changes in efficiency coefficients). Actuarial calculations represent special branch of activity (usually licensed activity) dealing with market assessment of compliance of current assets of insurance, joint-stock, investment, pension, credit and other financial companies (i.e. companies engaged in credit relations) with future liabilities to the repayment of credit in order to prevent insolvency of a debtor and to provide efficient protection for investors-creditors. Actuarial calculations assume the comparison of assets (ways of use or allocation of obtained funds) with liabilities (sources of gained funds) for borrowing companies of all types and forms, which are carried out in aggregate by particular items of their expenses under circumstances of mutual risks in order to expose the degree of compliance or incompliance (surplus or deficit) of borrowed assets with future liabilities in term of repayment, in other words to check the solvency of borrowing companies. Borrowing companies – insurance, stock, broker and auditor firms, banks, mutual, pension, and other specialized investment funds whose accounts payable two or more times exceeds their own assets and appear to be a source of high risk, which in turn affects interests of broad groups of business society as well as population – are considered as companies that are subjects to obligatory insurance and actuarial assessment. Actuarial calculations assume the construction of balances for future assets and liabilities, probabilistic assessment of future liabilities repayment (debts) at the expense of disposable assets with regard to risks of changes of their amount on hand and market prices. The procedures
of documentary adoption, which include construction of actuarial balances and preparation of actuarial reports and conclusions, are called actuarial estimation; the organizations that are carrying out such procedures are called actuarial organizations. Hence, there is a necessity to learn the organization and technique of actuarial methods (estimations) in aggregate; as well as to introduce the knowledge of actuarial subjects to any expert who is involved in direct actuarial estimations of future assets and liabilities costs of various funds, credit, insurance, and similarly financial companies. This is true for assets and liabilities of any country. The knowledge of these actuarial assessments and practical use is a significant reserve for increasing not only efficiency but (more important today) legitimate, transparent, and protected futures for both borrowing and lending companies.
Key Terms Actuary (actuarius – Latin) – profession, appraiser of risks, certified expert on assessment of documentary insurance (and wider – financial) risks; in insurance – insurer; in realty agencies – appraiser; in accounting – auditor; in financial markets – broker (or bookmaker); in the past registrar and holder of insurance documents; in England – adjuster or underwriter. Actuarial transactions – special field of activity related to determination of insurance outcomes in circumstances of uncertainty that require knowledge of probability theory and actuarial statistics methods and mathematics, including modern computer programs. Actuarial assessment – type of practical activity, licensed in the majority of countries, related to preparation of actuarial balances, market assessment of current and future costs of assets and liabilities of insurer (in case of pension insurance assets and liabilities of nongovernmental pension funds, insurances companies and specialized mutual trust funds); completed with preparation of actuarial report according to standard methodologies and procedures approved, as a rule in conventional (sometimes in legislative) order. Actuarial estimations – documentary estimations of chance outcomes (betting) of any risk (gambling) actions (games) with participation of two or more parties with fixed (registered) rates of repayment of insurance premium and compensations premium for possible losses. They differ by criteria of complexity – that is elementary (simple or initial) and complex. The most widespread cases of elementary actuarial estimations are bookmaker estimations of profit and loss from different types of gambling including playing cards, lottery, and casinos, as well as risk
Actuarial Methods
taking on modern stock exchange, foreign exchange markets, commodity exchanges, etc. The complex estimations assume determination of profit from second and consequent derived risks (outcomes over outcomes, insurance over insurance, repayment on repayment, transactions with derivatives, etc.). All of these estimations are carried out with the help of various method of high mathematics (first of all, numeric methods of probability theory and mathematical statistics). They are also often represented as methods of high actuarial estimations. Generally due to ignorance about such estimations, current world debt (in approximately trillion USD, including trillion USD in the USA) has drastically exceeded real assets, which account for about trillion USD, which is actually causing the enormous financial crisis everywhere in the world. Usually such estimations are being undertaken towards future insurance operations, profits and losses, and that is why they are classified as strictly approximate and represented in categories of probabilistic expectations. The fundamental methods of actuarial estimations are the following: methods for valuing investments, selecting portfolios, pricing insurance contracts, estimating reserves, valuing portfolios, controlling pension scheme, finances, asset management, time delays and underwriting cycle, stochastic approach to life insurance mathematics, pension funding and feed back, multiple state and disability insurance, and methods of actuarial balances. The most popular range of application for actuarial methods are: ) investments, (actuarial estimations) of investments assets and liabilities, internal and external, real and portfolio types their mathematical methods and models, investments risks and management; ) life insurance (various types and methods, insurance bonuses, insurance companies and risks, role of the actuarial methods in management of insurance companies and reduction of insurance risks); ) general insurance (insurance schemes, premium rating, reinsurance, reserving); ) actuarial provision of pension insurance (pension investments – investment policy, actuarial databases, meeting the cost, actuarial researches). Scientist who have greatly contributed to actuarial practices: William Morgan, Jacob Bernoulli, A. A. Markov, V. Y. Bunyakovsky, M. E. Atkinson, M. H. Amsler, B. Benjamin, G. Clark, C. Haberman, S. M. Hoem, W. F. Scott, and H. R. Watson. World’s famous actuary’s schools and institutes: The Institute of Actuaries in London, Faculty of Actuaries in Edinburgh (on May , following a ballot of Fellows of both institutions, it was announced that the Institute and Faculty would merge to form one body – the “Institute and
A
Faculty of Actuaries”), Charted Insurance Institute, International Association of Actuaries, International Forum of Actuaries Associations, International Congress of Actuaries, and Groupe Consultatif Actuariel Européen.
About the Author Professor Vassiliy M. Simchera received his PhD at the age of and his Doctor’s degree when he was . He has been Vice-president of the Russian Academy of Economical Sciences (RAES), Chairman of the Academic Council and Counsel of PhDs dissertations of RAES, Director of Russian State Scientific and Research Statistical Institute of Rosstat (Moscow, from ). He was also Head of Chair of statistics in the All-Russian Distant Financial and Statistical Institute (–), Director of Computer Statistics Department in the State Committee on statistics and techniques of the USSR (–), and Head of Section of Statistical Researches in the Science Academy of the USSR (–). He has supervised Doctors and over PhD’s. He has (co-) authored over books and articles, including the following books: Encyclopedia of Statistical Publications (, p., in co-authorship), Financial and Actuarial Calculations (), Organization of State Statistics in Russian Federation () and Development of Russia’s Economy for Years, – (). Professor Simchera was founder and executive director (–) of Russian Statistical Association, member of various domestic and foreign academies, as well as scientific councils and societies. He has received numerous honors and awards for his work, including Honored Scientist of Russian Federation () (Decree of the President of the Russian Federation) and Saint Nicolay Chudotvoretz honor of III degree (). He is a full member of the International Statistical Institute (from ).
Cross References 7Careers in Statistics 7Insurance, Statistics in 7Kaplan-Meier Estimator 7Life Table 7Population Projections 7Probability, History of 7Quantitative Risk Management 7Risk Analysis 7Statistical Aspects of Hurricane Modeling and
Forecasting 7Statistical Estimation of Actuarial Risk Measures for Heavy-Tailed Claim Amounts 7Survival Data
A
A
Adaptive Linear Regression
References and Further Reading
7Computational Statistics and 7Statistical Software: An
Benjamin B, Pollard JH () The analysis of mortality and other actuarial statistics, nd edn. Heinemann, London Black K, Skipper HD () Life insurance. Prentice Hall, Englewood Cliffs, New Jersey Booth P, Chadburn R, Cooper D, Haberman S and James D () Modern actuarial theory and practice. Chapman and Hall/CHC, London, New York Simchera VM () Introduction to financial and actuarial calculations. Financy and Statistika Publishing House, Moscow Teugels JL, Sundt B () The encyclopedia of actuarial science, vols. Wiley, Hoboken, NJ Transactions of International Congress of Actuaries, vol. –; J Inst Actuar, vol. –
Overview). On the other hand, this progress has put an applied statistician into a difficult situation: If one needs to fit the data with a regression hyperplane, he (she) is hesitating which procedure to use. If there is more information on the model, then the estimation procedure can be chosen accordingly. If the data are automatically collected by a computer and the statistician is not able to make any diagnostics, then he (she) might use one of the high breakdown-point estimators. However, many decline this idea due to the difficult computation. Then, at the end, the statistician can prefer the simplicity to the optimality and uses either the classical least squares (LS), LAD-method or other reasonably simple method. Instead of to fix ourselves on one fixed method, one can try to combine two convenient estimation methods, and in this way diminish eventual shortages of both. Taylor () suggested to combine the LAD (minimizing the L norm) and the least squares (minimizing the L norm) methods. Arthanari and Dodge () considered a convex combination of LAD- and LS-methods. Simulation study by Dodge and Lindstrom () showed that this procedure is robust to small deviations from the normal distribution (see 7Normal Distribution, Univariate). Dodge () extended this method to a convex combination of LAD and Huber’s M-estimation methods (see 7Robust Statistics and Robust Statistical Methods). Dodge and Jureˇcková () observed that the convex combination of two methods could be adapted in such a way that the resulted estimator has the minimal asymptotic variance in the class of estimators of a similar kind, no matter what is the unknown distribution. The first numerical study of this procedure was made by Dodge et al. (). Dodge and Jureˇcková (, ) then extended the adaptive procedure to the combinations of LAD- with M-estimation and with the trimmed least squares estimation. The results and examples are summarized in monograph of Dodge and Jureˇcková (), where are many references added. Let us describe the general idea, leading to a construction of an adaptive convex combination of two estimation methods: We consider a family of symmetric densities indexed by an suitable measure of scale s :
Adaptive Linear Regression Jana Jureˇcková Professor Charles University in Prague, Prague, Czech Republic Consider a set of data consisting of n observations of a response variable Y and of vector of p explanatory variables X = (X , X , . . . , Xp )⊺ . Their relationship is described by the linear regression model (see 7Linear Regression Models) Y = β X + β X + . . . + β p Xp + e. In terms of the observed data, the model is Yi = β xi + β xi + . . . + β p xip + ei ,
i = , , . . . , n.
The variables e , . . . , en are unobservable model errors, which are assumed being independent and identically distributed random variables with a distribution function F and density f . The density is unknown, we only assume that it is symmetric around . The vector β = (β , β , . . . , β p )⊺ is an unknown parameter, and the problem of interest is to estimate β based on observations Y , . . . , Yn and xi = (xi , . . . , xip )⊺ , i = , . . . , n. Besides the classical 7least squares estimator, there exists a big variety of robust estimators of β. Some are distributionally robust (less sensitive to deviations from the assumed shape of f ), others are resistant to the leverage points in the design matrix and have a high breakdown point [introduced originally by Hampel (), the finite sample version is studied in Donoho and Huber ()]. The last years brought a host of statistical procedures, many of them enjoying excellent properties and being equipped with a computational software (see
F = {f : f (z) = s− f (z/s), s > }. The shape of f is generally unknown; it only satisfies some regularity conditions and the unit element f ∈ F has the scale s = . We take s = /f () when we combine L -estimator with other class of estimators.
Adaptive Linear Regression
The scale characteristic s is estimated by a consistent estimator ˆsn based on Y , . . . , Yn , which is regressioninvariant and scale-equivariant, i.e., p
(a) ˆsn (Y) → s as n → ∞ (b) ˆsn (Y + Xb) = ˆsn (Y) for any b ∈ Rp (regression-invariance) (c) ˆsn (cY) = cˆsn (Y) for c > (scale-equivariance).
Such estimator based on the regression quantiles was constructed e.g., by Dodge and Jureˇcková (). Other estimators are described in the monograph by Koenker (). The adaptive estimator Tn (δ) of β is defined as a solution of the minimization problem
A
If we know all moments in (), we minimize the variance () with respect to δ, under the restriction ≤ δ ≤ . It is minimized for δ = δ where if σ ≤ E < /
⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ σ − E δ = ⎨ ⎪ σ − E + ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩
if E < / and E < σ if / ≤ E < σ .
The estimator Tn (δ ) of β is a solution of the minimization n
⊺ ∑ ρ((Yi − xi t)/ˆsn ) := min,
t ∈ Rp ,
i= n
∑ρ( i=
Yi − x⊺i t ) := min ˆsn
ρ(z) = ( − δ )z + δ ∣z∣, z ∈ R .
with respect to t ∈ R , where p
ρ(z) = δρ (z) + ( − δ)ρ (z)
()
with a suitable fixed δ, ≤ δ ≤ , where ρ (z) and ρ (z) are symmetric (convex) discrepancy functions defining the respective estimators. For instance, ρ (z) = ∣z∣ and ρ (z) = z if we want to combine LAD and √ LS estimators. Then n(Tn (δ) − β) has an asymptotically normal distribution (see 7Asymptotic Normality) Np (, Q− σ (δ, ρ, f )) with the variance dependent on δ, ρ and f , where Q = lim Qn , n→∞
Qn =
n ⊺ ∑ xi xi . n i=
Using δ = δ which minimizes σ (δ, ρ, f ) with respect to δ, ≤ δ ≤ , we get an estimator Tn (δ ) minimizing the asymptotic variance for a fixed distribution shape. Typically, σ (δ, ρ, f ) depends on f only through two moments of f . However, these moments should be estimated on the data. Let us illustrate the procedure on the combination of the least squares and the L procedures. Set σ = ∫ z f (z)dz, E
= ∫ ∣z∣ f (z)dz,
σ = ∫ z f (z)dz
()
But δ is unknown, because the entities in () depend on the unknown distribution f . Hence, we should replace δ by an appropriate estimator based on Y. We shall proceed in the following way: First estimate E = E /s = f () ∫RI ∣z∣ f (z)dz by − ′̂ ̂ E = ˆs− n (n − p) ∑ ∣Yi − xi β n ( )∣ i= n
Adaptive Linear Regression. Table Decision procedure Compute ̂ E as in (). () If ̂ En < /, calculate n
̂ β(/)) σn = ̂s (n−p) ∑ (Yi − x⊺i ̂ n i=
() If ̂ En ≥ ̂ σn , put ̂ δn = . Then Tn is the ordinary LS
() If ̂ En < ̂ σn , calculate
̂ δn =
E = ∫ ∣z∣ f (z)dz = sE
s σ (δ, f , s) = {( − δ) σ + δ( − δ)E + δ }.
n
i=
()
̂ σn −̂ En −̂ ̂ σn En +
and perform the minimization () with the function ρ equal to ( − ̂ δn ) ∑ (
and the corresponding asymptotic variance of Tn (δ) is
and go to (). If not, go to ().
Then σ = ∫ z f (z)dz = s σ ,
()
where ̂ β n (/) is the LAD-estimator of β. The choice of optimal ̂ δ n is then based on the following decision procedure (Table ). p It can be proved that ̂ δ n Ð→ δ as n → ∞ and T n (̂ δ n ) is a consistent estimator of β and is asymptotically normally distributed with the minimum possible variance.
estimator of β. If not, go to ().
E = ∫ ∣z∣ f (z)dz.
()
Yi −x′i t ) ̂ sn
n
+̂ δn ∑ ∣ i=
Yi −x⊺i t ∣. ̂ sn
() Put ̂ δn = ; then Tn is the LAD-estimate of β.
A
A
Adaptive Methods
Many numerical examp les based on real data can be find in the monograph Dodge and Jureˇcková ().
Acknowledgments The research was supported by the Czech Republic Grant // and by Research Projects MSM and LC .
About the Author Jana Jureˇcková was born on September , in Prague, Czechoslovakia. She has her Ph.D. in Statistics from Czechoslovak Academy of Sciences in ; some twenty years later, she was awarded the DrSc from Charles University. Her dissertation, under the able supervision of late Jaroslav Hajek, related to “uniform asymptotic linearity of rank statistics” and this central theme led to significant developments in nonparametrics, robust statistics, time series, and other related fields. She has extensively collaborated with other leading statisticians in Russia, USA, Canada, Australia, Germany, Belgium, and of course, Czech Republic, among other places. A (co-)author of several advanced monographs and texts in Statistics, Jana has earned excellent international reputation for her scholarly work, her professional accomplishment and her devotion to academic teaching and councelling. She has been with the Mathematics and Physics faculty at Charles University, Prague, since , where she earned the Full Professor’s rank in . She has over publications in the leading international journals in statistics and probability, and she has supervised a number of Ph.D. students, some of them have acquired international reputation on their own. (Communicated by P. K. Sen.)
Cross References 7Robust Regression Estimation in Generalized Linear
Models 7Robust Statistical Methods 7Robust Statistics
References and Further Reading Arthanari TS, Dodge Y () Mathematical programming in statistics. Wily, Interscience Division, New York; () Wiley Classic Library Dodge Y () Robust estimation of regression coefficient by minimizing a convex combination of least squares and least absolute deviations. Comp Stat Quart :– Dodge Y, Jureˇcková J () Adaptive combination of least squares and least absolute deviations estimators. In: Dodge Y (ed) Statistical data analysis based on L – norm and related methods. North-Holland, Amsterdam, pp – Dodge Y, Jureˇcková J () Adaptive combination of M-estimator and L – estimator in the linear model. In: Dodge Y, Fedorov VV,
Wynn HP (eds) Optimal design and analysis of experiments. North-Holland, Amsterdam, pp – Dodge Y, Jureˇcková J () Flexible L-estimation in the linear model. Comp Stat Data Anal :– Dodge Y, Jureˇcková J () Estimation of quantile density function based on regression quantiles. Stat Probab Lett : – Dodge Y, Jureˇcková J () Adaptive regression. Springer, New York. ISBN ---X Dodge Y, Lindstrom FT () An alternative to least squares estimations when dealing with contaminated data. Technical report No , Oregon State University, Corvallis Dodge Y, Antoch J, Jureˇcková J () Adaptive combination of least squares and least absolute deviation estimators. Comp State Data Anal :– Donoho DL, Huber PJ () The notion of breakdown point. In: Bickel PJ, Doksum KA, Hodges JL (eds) A festschrift for Erich Lehmann. Wadsworth, Belmont, California Hampel FR () Contributions to the theory of robust estimation. PhD Thesis, University of California, Berkely Koenker R () Quantile regression. Cambridge University Press, Cambridge. ISBN --- Taylor LD () Estimation by minimizing the sum of absolute errors. In: Zarembka P (ed) Frontiers in econometrics. Academic, New York, pp –
Adaptive Methods Saïd El Melhaoui Professor Assistant Université Mohammed Premier, Oujda, Morocco
Introduction Statistical procedures, the efficiencies of which are optimal and invariant with regard to the knowledge or not of certain features of the data, are called adaptive statistical methods. Such procedures should be used when one suspects that the usual inference assumptions, for example, the normality of the error’s distribution, may not be met. Indeed, traditional methods have a serious defect. If the distribution of the error is non-normal, the power of classical tests, as pseudo-Gaussian tests, can be much less than the optimal power. So, the variance of the classical least squares estimator is much bigger than the smallest possible variance.
What Is Adaptivity? The adaptive methods deal with the problem of estimating and testing hypotheses about a parameter of interest θ in the presence of nuisance parameter ν. The fact that ν remains unspecified induces, in general, a loss of efficiency
Adaptive Methods
with the situation where ν is exactly specified. Adaptivity occurs when the loss of efficiency is null, i.e., when we can estimate (testing hypotheses about) θ as when not knowing ν as well as when knowing ν. The method used in this respect is called adaptive. Adaptivity is a property of the model under study, the best known of which is undoubtedly the symmetric location model; see Stone (). However, under a totally unspecified density, possibly non-symmetric, the mean can not be adaptively estimated.
Approaches to Adaptive Inference Approaches to adaptive inference mainly belong to one of two types: either to estimate the unknown parameters ν in some way, or to use the data itself to determine which statistical procedure is the most appropriate to these data. These two approaches are the starting points of two rather distinct strands of the statistical literature. Nonparametric adaptive inference, on one hand, where ν is estimated from the sample, and on the other hand, data-driven methods, where the shape of ν is identified via a selection statistic to distinguish the effective statistical procedure suitable at the current data.
Nonparametric Methods The first approach is often used for the semiparametric model, where θ is a Euclidean parameter and the nuisance parameter is an infinite dimensional parameter f - often, the unspecified density of some white noise underlying the data generating process. Stein () introduced the notion of adaptation and gave a simple necessary condition for adaptation in semiparametric models. A comprehensive account of adaptive inference can be found in the monograph by Bickel et al. () for semiparametric models with independent observations. Adaptive inference for dependent data have been studied in a series of papers, e.g., Kreiss (), Drost et al. (), and Koul and Schick (). The current state of the art is summarized in Grenwood et al. (). The basic idea in this literature is to estimate the underlying f using a portion of the sample, and to reduce locally and asymptotically the semiparametric problem to a simpler parametric one, through the so-called “least favorable parametric submodel” argument. In general, the resulting computations are non-trivial. An alternative technique is the use of adaptive rank based statistics. Hallin and Werker () proposed a sufficient condition for adaptivity; that is, adaptivity occurs if a parametrically efficient method based on rank statistics can be derived. Then, it suffices, to substitute f in the rank statistics by an estimate fˆ measurable on the 7order
A
statistics. Some results in this direction have been obtained by Hájek (), Beran (), and Allal and El Melhaoui (). Finally, these nonparametric adaptive methods, when they exist, are robust in efficiency: they cannot be outperformed by any non-adaptive method. However, these methods have not been widely used in practice, because the estimation of density, typically, requires a large number of observations.
Data-Driven Methods The second strand of literature addresses the same problem of constructing adaptive inference, and consists of the use of the data to determine which statistical procedure should be used and then using the data again to carry out the procedure. The was first proposed by Randles and Hogg (). Hogg et al. () used the measure of symmetry and tailweight as selection statistics in and adaptive two-sample test. If the selection fell into one of the regions defined by the adaptive procedure, then a certain set of rank scores was selected, whereas if the selection statistic fell into a different region, then different rank scores would be used in the test. Hogg and Lenth () proposed an adaptive estimator of the mean of symmetric distribution. They used selection statistics to determine if a mean, a % trimmed mean, or median should be used as an estimate of the mean of population. O’Gorman () proposed an adaptive procedure that performs the commonly used tests of significance, including the two-sample test, a test for a slope in linear regression, and a test for interaction in two-way factorial design. A comprehensive account of this approach can be found in the monograph by O’Gorman (). The advantage of the data-driven methods is that if an adaptive method is properly constructed, it automatically downweight outliers and could easily be applied in practice. However, and contrary to the nonparametric approach, the adaptive data-driven method is the best among the existing procedures, but not the best that can be built. As a consequence, the method so built is not definitively optimal.
Cross References 7Nonparametric Rank Tests 7Nonparametric Statistical Inference 7Robust Inference 7Robust Statistical Methods 7Robust Statistics
A
A
Adaptive Sampling
References and Further Reading Allal J, El Melhaoui S () Tests de rangs adaptatifs pour les modèles de régression linéaires avec erreurs ARMA. Annales des Sciences Mathématiques du Québec :– Beran R () Asymptotically efficient adaptive rank estimates in location models. Annals of Statistics :– Bickel PJ, Klaassen CAJ, Ritov Y, Wellner JA () Efficient and adaptive estimation for semiparametric models. Johns Hopkins University Press, Baltimore, New York Drost FC, Klaassen CAJ, Ritov Y, Werker BJM () Adaptive estimation in time-series models. Ann Math Stat : – Greenwood PE, Muller UU, Wefelmeyer W () An introduction to efficient estimation for semiparametric time series. In: Nikulin MS, Balakrishnan N, Mesbah M, Limnios N (eds) Parametric and semiparametric models with applications to reliability, survival analysis, and quality of life. Statistics for Industry and Technology, Birkhäuser, Boston, pp. – Hájek J () Asymptotically most powerful rank-order tests. Ann Math Stat :– Hallin M, Werker BJM () Semiparametric Efficiency Distribution-Freeness, and Invariance. Bernoulli :– Hogg RV, Fisher DM, Randles RH () A two simple adaptive distribution-free tests. J Am Stat Assoc :– Hogg RV, Lenth RV () A review of some adaptive statistical techniques. Commun Stat – Theory Methods :– Koul HL, Schick A () Efficient estimation in nonlinear autoregressive time-series models. Bernoulli :– Kreiss JP () On adaptive estimation in stationary ARMA processes. Ann Stat :– O’Gorman TW () An adaptive test of significance for a subset of regression coefficients. Stat Med :– O’Gorman TW () Applied adaptive statistical methods: tests of significance and confidence intervals. Society for Industrial and Applied Mathematics, Philadelphia Randles RH, Hogg RV () Adaptive distribution-free tests. Commun Stat :– Stein C () Efficient nonparametric testing and estimation. In: Proceedings of Third Berkeley Symposium on Mathametical Statististics and Probability, University of California Press, Berkeley, vol , pp. – Stone CJ () Adaptive maximum likelihood estimators of a location parameter. Ann Stat :–
Adaptive Sampling George A. F. Seber , Mohammad Salehi M. Emeritus Professor of Statistics Auckland University, Auckland, New Zealand Professor Isfahan University of Technology, Isfahan, Iran Adaptive sampling is particularly useful for sampling populations that are sparse but clustered. For example, fish can form large, widely scattered schools with few fish in
between. Applying standard sampling methods such as simple random sampling (SRS, see 7Simple Random Sample) to get a sample of plots from such a population could yield little information, with most of the plots being empty. The idea can be simply described follows. We go fishing in a lake using a boat and, assuming complete ignorance about the population, we select a location at random and fish. If we don’t catch anything we select another location at random and try again. If we do catch something we fish in a specific neighborhood of that location and keep expanding the neighborhood until we catch no more fish. We then move on to a second location. This process continues until we have, for example, fished at a fixed number of locations or until our total catch has exceeded a certain number of fish. This kind of technique where the sampling is adapted to what turns up at each stage has been applied to a variety of diverse populations such as marine life, birds, mineral deposits, animal habitats, forests, and rare infectious diseases, and to pollution studies. We now break down this process into components and introduce some general notation. Our initial focus will be on adaptive 7cluster sampling, the most popular of the adaptive methods developed by Steven Thompson in the s. Suppose we have a population of N plots and let yi be a variable that we measure on the ith plot (i = , , . . . , N). This variable can be continuous (e.g., level of pollution or biomass), discrete (e.g., number of animals or plants), or even just an indicator variable taking the value for presence and zero for absence. Our aim is to estimate some function of the population y values such as, for example, the population total τ = ∑Ni= yi , the population mean µ = τ/N, or the population density D = τ/A, where A is the population area. The next step is to determine the nature of the neighborhood of each initially chosen plot. For example, we could choose all the adjacent units with a common boundary which, together with unit i, form a “cross” Neighborhoods can be defined to have a variety of patterns and the units in a neighborhood do not have to be contiguous (next to each other). We then specify a condition C such as yi > c which determines when we sample the neighborhood of the ith plot; typically c = if y is a count. If C for the ith plot or unit is satisfied, we sample all the units in the neighborhood and if the rule is satisfied for any of those units we sample their neighborhoods as well, and so on, thus leading to a cluster of units. This cluster has the property that all the units on its “boundary” (called “edge units”) do not satisfy C. Because of a dual role played by the edge units, the underlying theory is based on the concept of a network, which is a cluster minus its edge units. It should be noted that if the initial unit selected is any one of the units in the cluster except an edge unit, then
Adaptive Sampling
all the units in the cluster end up being sampled. Clearly, if the unit is chosen at random, the probability of selecting the cluster will depend on the size of the cluster. For this reason adaptive cluster sampling can be described as unequal probability cluster sampling – a form of biased sampling. The final step is to decide how we choose both the size and the method of selecting the initial sample size. Focusing on the second of these for the moment, one simple approach would be to use SRS to get a sample of size n , say. If a unit selected in the initial sample does not satisfy C, then there is no augmentation and we have a cluster of size one. We note that even if the units in the initial sample are distinct, as in SRS, repeats can occur in the final sample as clusters may overlap on their edge units or even coincide. For example, if two non-edge units in the same cluster are selected in the initial sample, then that whole cluster occurs twice in the final sample. The final sample then consists of n (not necessarily distinct) clusters, one for each unit selected in the initial sample. We finally end up with a total of n units, which is random, and some units may be repeated. There are many modifications of the above scheme depending on the nature of the population and we mention just a few. For example, the initial sample may be selected by sampling with replacement, or by using a form of systematic sampling (with a random start) or by using unequal probability sampling, as in sampling a tree with probability proportional to its basal area. Larger initial sampling units other than single plots can be used, for example a strip transect (primary unit) commonly used in both aerial and ship surveys of animals and marine mammals. Other shaped primary units can also be used and units in the primary unit need not be contiguous. If the population is divided into strata, then adaptive cluster sampling can be applied within each stratum, and the individual estimates combined. How they are combined depends on whether clusters are allowed to cross stratum boundaries or not. If instead of strata, we simply have a number of same-size primary units and choose a sample of primary units at random, and then apply the adaptive sampling within each of the chosen primary units, we have two-stage sampling with its appropriate theory. In some situations, the choice of c in condition C is problematical as, with a wrong choice, we may end up with a feast or famine of plots. Thompson suggested using the data themselves, in fact the 7order statistics for the yi values in the initial sample. Sometimes animals are not always detected and the theory has been modified to allow for incomplete detectability. If we replace yi by a vector, then the scheme can be modified to allow for multivariate data.
A
We now turn our attention to sample sizes. Several ways of controlling sample sizes have been developed. For example, to avoid duplication we can remove a network once it has been selected by sampling networks without replacement. Sequential methods can also be used, such as selecting the initial sample sequentially until n exceeds some value. In fact Salehi, in collaboration with various other authors has developed a number of methods using both inverse and sequential schemes. One critical question remains: How can we use a pilot survey to design an experiment with a given efficiency or expected cost? One solution has been provided using the two-stage sampling method mentioned above (Salehi and Seber ). We have not said anything about actual estimates as this would take several pages. However, a number of estimates associated with the authors Horvitz-Thompson (see 7Horvitz–Thompson Estimator), Hansen-Hurwitz, and Murthy have all been adapted to provide unbiased estimates for virtually all the above schemes and modifications. Salehi () has also used the famous 7RaoBlackwell theorem to provide more efficient unbiased estimates in a number of cases. The mentioned estimators based on small samples under adaptive cluster sampling often have highly skewed distributions. In such situations, confidence intervals (see 7Confidence Interval) based on traditional normal approximation can lead to unsatisfactory results, with poor coverage properties; for another solution see Salehi et al. (a). As you can see, the topic is rich in applications and modifications and we have only told part of the story! For example, there is a related topic called adaptive allocation that has been used in fisheries; for a short review of adaptive allocation designs see Salehi et al. (b). Extensive references to the above are Thompson and Seber () and Seber and Salehi ().
About the Author Professor Seber was appointed to the foundation Chair in Statistics and Head of a newly created Statistics Unit within the Mathematics Department at the University of Auckland in . He was involved in forming a separate Department of Statistics in . He was awarded the Hector Medal by the Royal Society of New Zealand for fundamental contributions to statistical theory, for the development of the statistics profession in New Zealand, and for the advancement of statistics education through his teaching and writing (). He has authored or coauthored ten books as well as several second editions, and numerous research papers. However, despite the breadth of his contribution from linear models, multivariate statistics, linear regression, non-linear models, to adaptive sampling, he is perhaps still best known internationally for his research
A
A
Advantages of Bayesian Structuring: Estimating Ranks and Histograms
on the estimation of animal abundance. He is the author of the internationally recognized text Estimation of Animal Abundance and Related Parameters (Wiley, nd edit., ; paperback reprint, Blackburn, ). The third conference on Statistics in Ecology and Environmental Monitoring was held in Dunedin () “to mark and recapture the contribution of Professor George Seber to Statistical Ecology.”
Cross References 7Cluster Sampling 7Empirical Likelihood Approach to Inference from
Sample Survey Data 7Statistical Ecology
References and Further Reading Salehi MM () Rao-Blackwell versions of the Horvitz-Thompson and Hansen-Hurwitz in adaptive cluster sampling. J Environ Ecol Stat :– Salehi MM, Seber GAF () Two stage adaptive cluster sampling. Biometrics :– Salehi MM, Mohammadi M, Rao JNK, Berger YG (a) Empirical Likelihood confidence intervals for adaptive cluster sampling. J Environ Ecol Stat :– Salehi MM, Moradi M, Brown JA, Smith DR (b) Efficient estimators for adaptive two-stage sequential sampling. J Stat Comput Sim, DOI: ./ Seber GAF, Salehi MM () Adaptive sampling. In: Armitage P, Colton T (eds) Encyclopedia of biostatistics, vol , nd edn. Wiley, New York Thompson SK, Seber GAF () Adaptive sampling. Wiley, New York
Advantages of Bayesian Structuring: Estimating Ranks and Histograms Thomas A. Louis Professsor Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA
Introduction Methods developed using the Bayesian formalism can be very effective in addressing both Bayesian and frequentist goals. These advantages are conferred by full probability modeling are most apparent in the context of 7nonlinear models or in addressing non-standard goals. Once the likelihood and the prior have been specified and data
observed, 7Bayes’ Theorem maps the prior distribution into the posterior. Then, inferences are computed from the posterior, possibly guided by a 7loss function. This last step allows proper processing for complicated, nonintuitive goals. In this context, we show how the Bayesian approach is effective in estimating 7ranks and CDFs (histograms). We give the basic ideas; see Lin et al. (, ); Paddock et al. () and the references thereof for full details and generalizations. Importantly, as Carlin and Louis () and many authors caution, the Bayesian approach is not a panacea. Indeed, the requirements for an effective procedure are more demanding than those for a frequentist approach. However, the benefits are many and generally worth the effort, especially now that 7Markov Chain Monte Carlo (MCMC) and other computing innovations are available.
A Basic Hierarchical Model Consider a basic, compound sampling model with parameters of interest θ = (θ , . . . , θ K ) and data Y = (Y , . . . , YK ). The θ k are iid and conditional on the θs, the Yk are independent. iid
θ k ∼ G(⋅)
()
indep
Yk ∣θ k ∼ fk (Yk ∣θ k ) in practice, the θ k might be the true differential expression of the kth gene, the true standardized mortality ratio for the kth dialysis clinic, or the true, underlying regionspecific disease rate. Generalizations of () include adding a third stage to represent uncertainty in the prior, a regression model in the prior, or a priori association among the θs. Assume that the θ k and η are continuous random variables. Then, their posterior distribution is, K
g(θ ∣ Y) = ∏ g(θ k ∣ Yk )
()
g(θ k ∣ Yk ) =
fk (Yk ∣ θ k )g(θ k ) fk (Yk ∣ θ k )g(θ k ) = f (Y ∣ s)g(s)ds fG (Yk ) ∫ k k
Ranking The ranking goal nicely shows the beauty of Bayesian structuring. Following Shen and Louis (), if the θ k were directly observed, then their ranks (Rk ) and percentiles (Pk ) are: K
Rk (θ) = rank(θ k ) = ∑ I{θ k ≥θ j } ; Pk (θ) = Rk (θ)/(K + ). j=
()
A
Advantages of Bayesian Structuring: Estimating Ranks and Histograms
The smallest θ has rank and the largest has rank K. Note that the ranks are monotone transform invariant (e.g., ranking the logs of parameters produces the original ranks) and estimated ranks should preserve this invariance. In practice, we don’t get to observe the θ k , but can use their posterior distribution () to make inferences. For example, minimizing posterior squared-error loss for the ranks produces,
Advantages of Bayesian Structuring: Estimating Ranks and Histograms. Table Simulated preposterior , × SEL for gmv = . As a baseline for comparison, if the data provided no information on the θ k (gmv = ∞), all entries would be . If the data provided perfect information (gmv = ), all entries would be Percentiles based on Pˆ k
rls K
R¯ k (Y) = Eθ∣Y [Rk (θ) ∣ Y] = ∑ pr(θ k ≥ θ j ∣ Y).
()
θ kpm
exp{θ kpm +
(−Bk )σk }
Yk
j=
The R¯ k are shrunk towards the mid-rank, (K + )/, and generally are not integers. Optimal integer ranks result from ranking the R¯ k , producing, Rˆ k (Y) = rank(R¯ k (Y)); Pˆ k = Rˆ k /(K + ).
()
Unless the posterior distributions of the θ k are stochastically ordered, ranks based on maximum likelihood estimates or those based on hypothesis test statistics perform poorly. For example, if all θ k are equal, MLEs with relatively high variance will tend to be ranked at the extremes; if Z-scores testing the hypothesis that a θ k is equal to the typical value are used, then the units with relatively small variance will tend to be at the extremes. Optimal ranks compromise between these two extremes, a compromise best structured by minimizing posterior expected loss in the Bayesian context.
Example: The basic Gaussian-Gaussian model We specialize () to the model with a Gaussian prior and Gaussian sampling distributions, with possibly different sampling variances. Without loss of generality assume that the prior mean is µ = and the prior variance is τ = . We have, θ k iid N(, ), Yk ∣θ k ∼ N(θ k , σk ) θ k ∣ Yk ind N (θ k , ( − Bk )σk ) pm
θ k = ( − Bk )Yk ; Bk = σk /(σk + ). pm
The σk are an ordered, geometric sequence with ratio of the largest σ to the smallest rls = σK /σ and 7geometric mean gmv = GM(σ , . . . , σK ). When rls = , the σk are all equal. The quantity gmv measures the typical sampling variance and here we consider only gmv = . Table documents SEL performance for Pˆ k (the optipm mal approach), Yk (the MLE), ranked θ k and ranked exp {θ k + pm
(−B k )σ k }
(the posterior mean of eθ k ). We
present this last to assess performance for a monotone,
non-linear transform of the target parameters. For rls = , the posterior distributions are stochastically ordered and the four sets of percentiles are identical, as is their performance. As rls increases, performance of Yk -derived pm percentiles degrades, those based on the θ k are quite ˆ competitive with Pk , but performance for percentiles based on the posterior mean of eθ k degrades as rls increases. Results show that though the posterior mean can perform well, in general it is not competitive with the optimal approach.
Estimating the CDF or Histogram Similar advantages of the Bayesian approach apply to estimating the empirical distribution function (EDF) of the θ k , GK (t ∣ θ) = K − ∑ I{θ k ≤t} . As shown by Shen and Louis (), The optimal SEL estimate is ¯ K (t∣Y) = E[GK (t ∣ θ)∣Y] = K − ∑ Pr(θ k ≤ t∣Y). G The optimal discrete distribution estimate with at most K ˆ K , with mass K − at mass points is G j − ˆj = G ¯ − ∣ Y), U K ( K
j = , . . . , K
The EDF is easy to compute from MCMC sampling. After burn-in, pool all θs, order them and set Uj equal to the (j − )th order statistic. Bayesian structuring to estimate GK pays big dividends. As shown in Fig. , for the basic Gaussian model it produces the correct spread, whereas the histogram pm of the θ k (the posterior means) is under-dispersed and that of the Yk (the MLEs) is over dispersed. More generally, when the true EDF is asymmetric or multi-modal,
A
A
Advantages of Bayesian Structuring: Estimating Ranks and Histograms
PM
ML
GR
0.14
0.14
0.12
0.12
0.12
0.10
0.10
0.10
0.08
0.08
0.08
0.06
Proportion
0.14
Proportion
Proportion
0.06
0.06
0.04
0.04
0.04
0.02
0.02
0.02
0.00
0.00 −4
−2
0
2
4
0.00 −4
−2
0
2
4
−4
−2
0
2
4 −
Advantages of Bayesian Structuring: Estimating Ranks and Histograms. Fig. Histogram estimates using θ pm , ML, and GK for the basic Gaussian/Gaussian model. GM({σk }) = , rls =
the Bayesian approach also produces the correct shape Paddock et al. ().
Discussion The foregoing are but two examples of the effectiveness of Bayesian structuring. Many more are available in the cited references and in other literature. In closing, we reiterate that the Bayesian approach needs to be used with care; there is nothing automatic about realizing its benefits.
Acknowledgments Research supported by NIH/NIDDK Grant RDK.
JASA Applications and Case Studies (–), Currently he is Co-editor, Biometrics (–). He is principal or co-advisor for doctoral students and more than masters students. He has delivered more than invited presentations. Professor Louis has (co-)authored about refereed papers and books, including Bayesian Methods for Data Analysis (with B.P. Carlin, Chapman & Hall/CRC, rd edition, ).
Cross References 7Bayes’ Theorem 7Bayesian Statistics 7Bayesian Versus Frequentist Statistical Reasoning 7Prior Bayes: Rubin’s View of Statistics
About the Author Dr. Thomas Louis is Professor of Biostatistics, Johns Hopkins Bloomberg School of Public Health. He was President, International Biometric Society (IBS), Eastern North American Region () and President, International Biometric Society (–). He is a Fellow of the American Statistical Association (), American Association for the Advancement of Science (), and Elected member, International Statistical Institute (). He was Editor,
References and Further Reading Carlin BP, Louis TA () Bayesian methods for data analysis, rd edn. Chapman and Hall/CRC, Boca Raton Lin R, Louis TA, Paddock SM, Ridgeway G () Loss function based ranking in two-stage, hierarchical models. Bayesian Anal :– Lin R, Louis TA, Paddock SM, Ridgeway G () Ranking of USRDS, provider-specific SMRs from –. Health Serv Out Res Methodol :–
African Population Censuses
Paddock S, Ridgeway G, Lin R, Louis TA () Flexible distributions for triple-goal estimates in two-stage hierarchical models. Comput Stat Data An ():– Shen W, Louis TA () Triple-goal estimates in two-stage, hierarchical models. J Roy Stat Soc B :–
African Population Censuses James P. M. Ntozi Professor of Demographic Statistics Makerere University, Kampala, Uganda
Definition A Population 7census is the total process of collecting, compiling, evaluating, analyzing and disseminating demographic, economic and social data related to a specified time, to all persons in a country or a well defined part of a country.
History of Population Censuses Population censuses are as old as human history. There are records of census enumerations as early as in bc in Babylonia, in bc in China and in bc in Egypt. The Roman Empire conducted population censuses and one of the most remembered censuses was the one held around ad when Jesus Christ was born as his parents had moved from Nazareth to Bethlehem for the purpose of being counted. However, modern censuses did not start taking place until one was held in Quebec, Canada in . This was followed by one in Sweden in , USA in , UK in and India .
African Population Censuses In the absence of complete civil registration systems in Africa, population censuses provide one of the best sources of socioeconomic and demographic information for the continent. Like in other parts of the world, censuses in Africa started as headcounts and assemblies until after the Second World War. The British were the first to introduce modern censuses in their colonial territories in west, east and southern Africa. For example in East Africa, the first modern census was conducted in in what was being referred to as British East Africa consisting of Kenya and Uganda. This was followed by censuses in in Tanzania, in in Uganda and in Kenya to prepare the countries for their political independence in , and , respectively. Other censuses have followed in these three
A
countries. Similarly, the British West African countries of Ghana, Gambia, Nigeria and Sierra Leone were held in s, s and s. In Southern Africa, similar censuses were held in Botswana, Lesotho, Malawi, Swaziland, Zambia and Zimbabwe in s and s, long before the Francophone and Lusophone countries did so. It was not until in s and s that the Francophone and Lusophone African countries started doing censuses instead of sample surveys which they preferred. To help African countries do population censuses, United Nations set up an African census programme in late s. Out of countries, participated in the programme. This programme closed in and was succeeded by the Regional Advisory Services in the demographic statistics set up as a section of Statistics Division at the United Nations Economic Commission for Africa. This section supported many African countries in conducting the and rounds of censuses. The section was superseded by the UNFPA sub-regional country support teams stationed in Addis Ababa, Cairo, Dakar and Harare. Each of these teams had census experts to give advisory services to countries in the round of censuses. These teams have now been reduced to three teams stationed in Pretoria, Cairo and Dakar and are currently supporting the African countries in population censuses. There were working group committees on census on each round of censuses to work on the content of census 7questionnaire. For instance, in the round of censuses the working group recommended that the census questionnaire should have geographic characteristics, demographic characteristics, economic characteristics, community level variables and housing characteristics. In round of censuses, questions on the disabled persons were recommended to be added to the round questions. Later in the round of censuses, questions on economic establishments, agricultural sector and deaths in households were added. In the current round of censuses, the questions on disability were sharpened to capture the data better. New questions being asked include those on child labour, age at first marriage, ownership of mobile phone, ownership of email address, access to internet, distance to police post, access to salt in household, most commonly spoken language in household and cause of death in household. In the and s round of censuses, Post enumeration surveys (PES) to check on the quality of the censuses were attempted in Ghana. However, the experience with and results from PES were not encouraging, which discouraged most of the African countries from conducting them. Recently, the Post enumeration surveys have been revived and conducted in several African
A
A
Aggregation Schemes
countries like South Africa, Tanzania and Uganda. The challenges of PES have included: poor cartographic work, neglecting operational independence, inadequate funding, fatigue after the census, matching alternative names, lack of qualified personnel, useless questions in PES, probability sample design and selection, field reconciliation, lack of unique physical addresses in Africa and neglect of pretest of PES. The achievements of the African censuses include supplying the needed sub-national data to the decentralized units for decision making processes, generating data for monitoring poverty reduction programmes, providing information for measuring indicators of most MDGs, using the data for measuring the achievement of indicators of International Conference on Population and Development (ICP), meeting the demand for data for emerging issues of socioeconomic concerns, accumulating experience in the region of census operations and capacity building at census and national statistical offices. However, there are still several limitations associated with the African censuses. These have included inadequate participation of the population of the region; only % of the African population was counted in the round of censuses, which was much below to what happened in other regions: Oceania – %, Europe and North America – %, Asia – %, South America – % and the world – %. Other shortcomings were weak organizational and managerial skills, inadequate funding, non-conducive political environment, civil conflicts, weak technical expertise at NSOs and lack of data for gender indicators.
About the Author Dr. James P. M. Ntozi is a Professor of demographic statistics at the Institute of Statistics, Makerere University, Kampala, Uganda. He is a founder and Past president of Uganda Statistical Society and Population Association of Uganda. He was a Council member of the International Statistical Institute and Union for African Population Studies, currently a Fellow and Chartered Statistician of the Royal Statistical Society and Council member of the Uganda National Academy of Sciences. He has authored, coauthored, and presented over scientific papers as well as books on fertility and censuses in Africa. He was an Editor of African Population Studies, co-edited books, and is currently on the editorial board of African Statistical Journal and the Journal of African Health Sciences. He has received awards from Population Association of America, Uganda Statistical Society, Makerere University, Bishop Stuart University, Uganda and Ankole Diocese, Church of
Uganda. James has been involved in planning and implementation of past Uganda censuses of population and housing of , , and . He is currently helping the Liberian Statistical office to analyze the census data. Professor Ntozi is a past Director of the Institute of Statistics and Applied Economics, a regional statistical training center based at Makerere University, Uganda, and responsible for training many leaders in statistics and demography in sub-Saharan Africa for over years. His other professional achievements have been research and consultancies in fertility, HIV/AIDS, Human Development Reports, and strategic planning.
Cross References 7Census 7Economic Statistics 7Integrated Statistical Databases 7Population Projections 7Promoting, Fostering and Development of Statistics in
Developing Countries 7Role of Statistics: Developing Country Perspective 7Selection of Appropriate Statistical Methods in Developing Countries
References and Further Reading Onsembe JO () Postenumeration surveys in Africa. Paper presented at the th ISI session, Durban, South Africa Onsembe JO, Ntozi JPM () The round of censuses in Africa: achievements and challenges. Afr Stat J , November
Aggregation Schemes Devendra Chhetry President of the Nepal Statistical Association (NEPSA), Professor and Head Tribhuvan University, Kathmandu, Nepal Given a data vector x = (x , x , . . . , xn ) and a weight vector w = (w , w , . . . , wn ), there exist three aggregation schemes in the area of statistics that, under certain assumptions, generate three well-known measures of location: arithmetic mean (AM), 7geometric mean (GM), and 7harmonic mean (HM), where it is implicitly understood that the data vector x contains values of a single variable. Among all these three measures, AM is more frequently used in statistics for some theoretical reasons. It is well known that AM ≥ GM ≥ HM where equality holds only when all components of x are equal.
A
Aggregation Schemes
In recent years, some of these three and a new aggregation scheme are being practiced in the aggregation of development or deprivation indicators by extending the definition of data vector to a vector of indicators, in the sense that it contains measurements of development or deprivation of several sub-population groups or measurements of several dimensions of development or deprivation. The measurements of development or deprivation are either available in the form of percentages or need to be transformed in the form of unit free indices. Physical Quality of Life Index (Morris ), Human Development Index (UNDP ), Gender-related Development Index (UNDP ), Gender Empowerment Measure (UNDP ), and Human Poverty Index (UNDP ) are some of the aggregated indices of several dimensions of development or deprivation. In developing countries, aggregation of development or deprivation indicators is a challenging task, mainly due to two reasons. First, indicators usually display large variations or inequalities in the achievement of development or in the reduction of deprivation across the sub-populations or across the dimensions of development or deprivation within a region. Second, during the process of aggregation it is desired to incorporate the public aversion to social inequalities or, equivalently, public preference for social equalities. Public aversion to social inequalities is essential for development workers or planners of developing countries for bringing marginalized sub-populations into the mainstream by monitoring and evaluation of the development works. Motivated by this problem, Anand and Sen (UNDP ) introduced the notion of the gender-equality sensitive indicator (GESI). In societies of equal proportion of female and male population, for example, the AM of and percent of male and female literacy rate is the same as that of and percent, showing that AM fails to incorporate the public aversion to gender inequality due to the AM’s built-in problem of perfect substitutability, in the sense that a percentage point decrease in female literacy rate in the former society as compared to the latter one is substituted by the percentage point increase in male literacy rate. The GM or HM, however, incorporates the public aversion to gender inequality because they do not posses the perfect substitutability property. Instead of AM, Anand and Sen used HM in the construction of GESI. In the above example consider that society perceives the social problem from the perspective of deprivation; that is, instead of gender-disaggregated literacy rates society considers gender-disaggregated illiteracy rates. Arguing as before, it immediately follows that AM fails to incorporate the public aversion to gender inequality. It also
follows that neither GM nor HM incorporates the public aversion to gender inequality. A new aggregation scheme is required for aggregating indicators of deprivation. So far, currently practiced aggregation schemes are accommodated within a slightly modified version of the following single mathematical function due to Hardy et al. () under the assumption that components of x and w are positive and the sum of the components of w is unity. /r ⎧ n ⎪ ⎪ r ⎪ ⎪ ( w x ) ∑ i i ⎪ ⎪ ⎪ i= µ(x, w, r) = ⎨ ⎪ n ⎪ ⎪ w ⎪ ⎪ ∏ xi i ⎪ ⎪ ⎩i=
if r ≠ , () if r = .
For fixed x and w, the function () is defined for all real numbers, implying that the function () yields an infinite number of aggregation schemes. In particular, it yields AM when r = , HM when r = −, and obviously GM when r = , and a new aggregation scheme suggested by Anand and Sen in constructing Human Poverty Index when n = , w = w = w = / and r = (UNDP ). It is well known that the values of the function are bounded between x() and x(n) , where x() = min{x , x , . . . , xn } and x(n) = max{x , x , . . . , xn }, and the function is strictly increasing with respect to r if all the components of data vector are not equal (see Fig. when w = w = ., x = % and x = %). The first two partial derivatives of the function with respect to the kth component of the vector x yield the following results where g(x, w) is GM. r− ⎧ ⎪ w ( xk ) ⎪ ∂µ(x, w, r) ⎪ ⎪ k µ(x,w,r) =⎨ ⎪ ∂xk ⎪ − ⎪ ⎪ ⎩wk g(x,w)xk
if r ≠ , if r = . ()
r−
⎧ r xk ⎪ ∑ w i xi ⎪(r − )wk [ µ(x,w,r) ] i≠k ∂ µ(x,w, r) ⎪ ⎪ =⎨ ⎪ ∂xk ⎪ − ⎪ ⎪ ⎩wk (wk − )g(x,w)xk
if r ≠ , if r = . ()
⎛r < ⎞ ⎟ and w, () and () imply that For fixed ⎜ ⎝r > ⎠ ⎛concave⎞ ⎟ with the function () is increasing and ⎜ ⎝ convex ⎠
A
A
Aggregation Schemes
60.0
55.0
50.0
45.0
40.0
35.0 30.0 –15
–10
–5
0 r
5
10
15
Aggregation Schemes. Fig. Nature of the function in a particular case
respect to each xk , implying that the aggregated value ⎛decreasing⎞ ⎟ rate with respect to each comincreases at ⎜ ⎝ increasing ⎠ ponent of x. These properties are desirable for aggregat⎛development⎞ ⎟ indicators, since the aggregated ing the ⎜ ⎝ deprivation ⎠ ⎛rise⎞ ⎛development⎞ ⎟ is expected to ⎜ ⎟ from the value of ⎜ ⎝ fall ⎠ ⎝ deprivation ⎠ ⎛floor to the ceiling value⎞ ⎟ at decreasing rate with respect ⎜ ⎝ceiling to the floor value⎠ to each component of x. For given x and w, the function () ⎛r < ⎞ ⎟, could be used to aggregate the with any value of r, ⎜ ⎝r > ⎠ ⎛development⎞ ⎜ ⎟ indicators if the public aversion to social ⎝ deprivation ⎠ inequalities should be incorporated. What value of r should one use in practice? There is no simple answer to this question, since the answer depends upon the society’s degree of preference for social equality. If a society has no preference for social equality, then one can use r = in aggregating development or deprivation indicators, which is still a common practice in developing countries, even though the public efforts for bringing marginalized sub-populations into the mainstream has become a major agenda of development.
If a society has preference for social equality, then subjective judgment in the choice of r seems to be unavoidable. For the purpose of monitoring and evaluation, such judgment does not seem to be a serious issue as long as a fixed value of r is decided upon. In this context, Anand and Sen suggested using r = − for aggregating the indicators of development when n = (UNDP ), and r = for aggregating the indicators of deprivation when n = (UNDP ). A lot of research work still needs to be done in this area for producing social-equality sensitive indicators of development or deprivation.
Cross References 7Composite Indicators 7Lorenz Curve 7Role of Statistics: Developing Country Perspective
References and Further Reading Hardy GH, Littlewood JE, Polya G () Inequalities. Cambridge University Press, London Morris MD () Measuring the condition of the world’s poor: the physical quality of life index. Frank Case, London UNDP () Human Development Report , Financing Human Development Oxford University Press, New York UNDP () Human Development Report , Gender and Human Development. Oxford University Press, New York UNDP () Human Development Report , Human Development to Eradicate Poverty. Oxford University Press, New York
Agriculture, Statistics in
Agriculture, Statistics in Gavin J. S. Ross Rothamsted Research, Harpenden, UK
The need to collect information on agricultural production has been with us since the dawn of civilization. Agriculture was the main economic activity, supplying both food for growing populations and the basis for taxation. The Sumerians of Mesopotamia before BC developed writing systems in order to record crop yields and livestock numbers. The Ancient Egyptians recorded the extent and productivity of arable land on the banks of the Nile. Later conquerors surveyed their new possessions, as in the Norman conquest of England which resulted in the Domesday Book of , recording the agricultural potential of each district in great detail. The pioneers of scientific agriculture, such as J.B. Lawes and J.H.Gilbert at Rothamsted, England, from onwards, insisted on accurate measurement and recording as the first requirement for a better understanding of the processes of agricultural production. The Royal Statistical Society (RSS) was founded in with its symbol of a sheaf of corn, implying that the duty of statisticians was to gather numerical information, but for others to interpret the data. Lawes published numerous papers on the variability of crop yields from year to year, and later joined the Council of the RSS. By agricultural experiments were conducted in several countries, including Germany, the Netherlands and Ireland, where W.S. Gosset, publishing under the name of “Student,” conducted trials of barley varieties for the brewing industry. In R.A. Fisher was appointed to analyze the accumulated results of years of field experimentation at Rothamsted, initiating a revolution in statistical theory and practice. Fisher had already published the theoretical explanation of Student’s t-distribution and the sampling distribution of the correlation coefficient, and challenged Karl Pearson’s position that statistical analysis was only possible with large samples. His first task was to study the relationship between rainfall and crop yields on the long-term experiments, for which he demanded a powerful mechanical calculator, the “Millionaire.” Introducing orthogonal polynomials to fit the yearly weather patterns and to eliminate the long-term trend in crop yield, he performed multiple regressions on the rainfall components, and developed the variance ratio test (later the F-distribution) to justify which terms to
A
include using what became the 7analysis of variance. If the results were of minor interest to farmers, the methods used were of enormous importance in establishing the new methodology of curve fitting, regression analysis and the analysis of variance. Fisher’s work with agricultural scientists brought him a whole range of statistical challenges. Working with small samples he saw the role of the statistician as one who extracts the information in a sample as efficiently as possible. Working with non-normally distributed data he proposed the concept of likelihood, and the method of maximum likelihood to estimate parameters in a model. The early field experiments at Rothamsted contained the accepted notion of comparison of treatments with controls at the same location, and some plots included factorial combinations of fertilizer sources. Fisher saw that in order to apply statistical methods to assess the significance of observed effects it was necessary to introduce 7randomization and replication. Local control on land of varying fertility could be improved by blocking, and for trends in two directions he introduced Latin Square designs. The analysis of factorial experiments could be expressed in terms of main effects and interaction effects, with the components of interaction between blocks and treatments regarded as the basic residual error variance. Fisher’s ideas rapidly gained attention and his ideas and methods were extended to many fields beyond agricultural science. George Snedecor in Iowa, Mahalanobis and C.R. Rao in India, were early disciples, and his assistants included L.H.C. Tippett, J. Wishart and H. Hotelling. He was visited in by J. Neyman, who was working with agricultural scientists in Poland. In he was joined by Frank Yates who had experience of 7least squares methods as a surveyor in West Africa. Fisher left Rothamsted in to pursue his interests in genetics, but continued to collaborate with Yates. They introduced Balanced Incomplete Blocks and Lattice designs, and Split Plot designs with more than one component of error variance. Their Statistical Tables, first published in , were widely used for many decades later. Yates expanded his department to provide statistical analysis and consulting to agricultural departments and institutes in Britain and the British Empire. Field experimentation spread to South America with W.L Stevens, and his assistants W.G. Cochran, D.J. Finney and O. Kempthorne became well-known statistical innovators in many applications. During World War II Yates persuaded the government of the value of sample surveys to provide information about farm productivity, pests and diseases and fertilizer use. He later advised Indian statisticians on
A
A
Agriculture, Statistics in
the design and analysis of experiments in which small farmers in a particular area might be responsible for one plot each. In Yates saw the potential of the electronic computer in statistical research, and was able to acquire the first computer devoted to civilian research, the Elliott . On this computer the first statistical programs were written for the analysis of field experiments and surveys, for bioassay and 7probit analysis, for multiple regression and multivariate analysis, and for model fitting by maximum likelihood. All the programs were in response to the needs of agricultural scientists, at field or laboratory level, including those working in animal science. Animal experiments typically had unequal numbers of units with different treatments, and iterative methods were needed to fit parameters by least squares or maximum likelihood. Animal breeding data required lengthy computing to obtain components of variance from which to estimate heritabilities and selection indices. The needs of researcher workers in fruit tree research, forestry, glasshouse crops and agricultural engineering all posed different challenges to the statistical profession. In J.A. Nelder came to Rothamsted as head of the Statistics Department, having been previously at the National Vegetable Research Station at Wellesbourne, where he had explored the used of systematic designs for vegetable trials, and had developed the well-used Simplex Algorithm with R. Mead to fit 7nonlinear models. With more powerful computers it was now possible to combine many analyses into one system, and he invited G.N. Wilkinson from Adelaide to include his general algorithm for the analysis of variance in a more comprehensive system that would allow the whole range of nested and crossed experimental designs to be handled, along with facilities for regression and multivariate analysis. The program GENSTAT is now used world-wide in agricultural and other research settings. Nelder worked with R.M. Wedderburn to show how the methodology of Probit Analysis (fitting binomial data to a transformed regression line) could be generalized to a whole class of 7Generalized Linear Models. These methods were particularly useful for the analysis of multiway contingency tables, using logit transformations for binomial data and log transformations for positive data with long-tailed distributions. The applications may have been originally in agriculture but found many uses elsewhere, such as in medical and pharmaceutical research. The needs of soil scientists brought new classes of statistical problems. The classification of soils was complicated by the fact that overlapping horizons with
different properties did not occur at the same depth, although samples were essential similar but displaced. The method of Kriging, first used by South African mining engineers, was found to be useful in describing the spatial variability of agricultural land, with its allowance for differing trends and sharp boundaries. The need to model responses to fertilizer applications, the growth of plants and animals, and the spread of weeds, pests and diseases led to developments in fitting non-linear models. While improvements in the efficiency of numerical optimization algorithms were important, attention to the parameters to be optimized helped to show the relationship between the model and the data, and which observations contributed most to the parameters of interest. The limitations of agricultural data, with many unknown or unmeasurable factors present, makes it necessary to limit the complexity of the models being fitted, or to fit common parameters to several related samples. Interest in spatial statistics, and in the use of models with more than one source of error, has led to developments such as the powerful REML algorithm. The use of intercropping to make better use of productive land has led to appropriate developments in experimental design and analysis. With the increase in power of computers it became possible to construct large, complex models, incorporating where possible known relationships between growing crops and all the natural and artificial influences affecting their growth over the whole cycle from planting to harvest. These models have been valuable in understanding the processes involved, but have not been very useful in predicting final yields. The statistical ideas developed by Fisher and his successors have concentrated on the choices which farmers can make in the light of information available at the time, rather than to provide the best outcomes for speculators in crop futures. Modeling on its own is no substitute for continued experimentation. The challenge for the st century will be to ensure sustainable agriculture for the future, taking account of climate change, resistance to pesticides and herbicides, soil degradation and water and energy shortages. Statistical methods will always be needed to evaluate new techniques of plant and animal breeding, alternative food sources and environmental effects.
About the Author Gavin J.S. Ross has worked in the Statistics Department at Rothamsted Experimental Station since , now as a retired visiting worker. He served under Frank Yates,
Akaike’s Information Criterion
John Nelder and John Gower, advising agricultural workers, and creating statistical software for nonlinear modelling and for cluster analysis and multivariate analysis, contributing to the GENSTAT program as well as producing the specialist programs MLP and CLASP for his major research interests. His textbook Nonlinear Estimation (Springer ) describes the use of stable parameter transformations to fit and interpret nonlinear models. He served as President of the British Classification Society.
Cross References 7Analysis of Multivariate Agricultural Data 7Farmer Participatory Research Designs 7Spatial Statistics 7Statistics and Climate Change
References and Further Reading Cochran WG, Cox GM () Experimental designs, nd edn. Wiley, New York Finney DJ () An introduction to statistical science in agriculture. Edinburgh, Oliver and Boyd Fisher RA () The influence of rainfall on the yield of wheat at Rothamsted. Phil Trans Roy Soc London B :– Mead R, Curnow RM () Statistical methods in agriculture and experimental biology, nd edn. Chapman and Hall, London Patterson HD, Thompson R () Recovery of interblock information when block sizes are unequal. Biometrika (): – Webster R, Oliver MA () Geostatistics for environmental scientists, nd edn. Wiley, New York Yates F () Sampling methods for censuses and surveys, th edn. Griffin, London
A
The following relation illustrates the significant characteristic of the log likelihood: I(g : f ) − I(g : f ) = −E(log f (X) − log f (X)). This formula shows that for an observation x of X the log likelihood log f (x) provides a relative measure of the closeness of the model f to the truth, or the goodness of the model. This measure is useful even when the true structure g is unknown. For a model f (X/a) with unknown parameter a the maximum likelihood estimate a(x) is defined as the value of a that maximizes the likelihood f (x/a) for a given observation x. Due to this process the value of log f (x/a(x)) shows an upward bias as an estimate of log f (X/a). Thus to use log f (x/a(x)) as the measure of the goodness of the model f (X/a), it must be corrected for the expected bias. In typical application of the method of maximum likelihood this expected bias is equal the dimension, or the number of components, of the unknown parameter a. Thus the relative goodness of a model determined by the maximum likelihood estimate is given by AIC = − (log maximum likelihood − (number of parameters)). Here log denotes natural logarithm. The coefficient − is used to make the quantity similar to the familiar chi-square statistic in the test of dimensionality of the parameter. AIC is the abbreviation of An Information Criterion.
About the Author
Akaike’s Information Criterion Hirotugu Akaike† Former Director General of the Institute of Statistical Mathematics and a Kyoto Prize Winner Tokyo, Japan The Information Criterion I(g : f ) that measures the deviation of a model specified by the probability distribution f from the true distribution g is defined by the formula I(g : f ) = E log g(X) − E log f (X). Here E denotes the expectation with respect to the true distribution g of X. The criterion is a measure of the deviation of the model f from the true model g, or the best possible model for the handling of the present problem.
Professor Akaike died of pneumonia in Tokyo on th August , aged . He was the Founding Head of the first Department of Statistical Science in Japan. “Now that he has left us forever, the world has lost one of its most innovative statisticians, the Japanese people have lost the finest statistician in their history and many of us a most noble friend” (Professor Howell Tong, from “The Obituary of Professor Hirotugu Akaike.” Journal of the Royal Statistical Society, Series A, March, ). Professor Akaike had sent his Encyclopedia entry on May , adding the following sentence in his email: “This is all that I could do under the present physical condition.”
Cross References 7Akaike’s Information Criterion: Background, Derivation,
Properties, and Refinements 7Cp Statistic 7Kullback-Leibler Divergence 7Model Selection
A
A
Akaike’s Information Criterion: Background, Derivation, Properties, and Refinements
Akaike’s Information Criterion: Background, Derivation, Properties, and Refinements Joseph E. Cavanaugh , Andrew A. Neath Professor The University of Iowa, Iowa City, IA, USA Professor Southern Illinois University Edwardsville, Edwardsville, IL, USA
Introduction The 7Akaike Information Criterion, AIC, was introduced by Hirotogu Akaike in his seminal paper “Information Theory and an Extension of the Maximum Likelihood Principle.” AIC was the first model selection criterion to gain widespread attention in the statistical community. Today, AIC continues to be the most widely known and used model selection tool among practitioners. The traditional maximum likelihood paradigm, as applied to statistical modeling, provides a mechanism for estimating the unknown parameters of a model having a specified dimension and structure. Akaike extended this paradigm by considering a framework in which the model dimension is also unknown, and must therefore be determined from the data. Thus, Akaike proposed a framework wherein both model estimation and selection could be simultaneously accomplished. For a parametric candidate model of interest, the likelihood function reflects the conformity of the model to the observed data. As the complexity of the model is increased, the model becomes more capable of adapting to the characteristics of the data. Thus, selecting the fitted model that maximizes the empirical likelihood will invariably lead one to choose the most complex model in the candidate collection. 7Model selection based on the likelihood principle, therefore, requires an extension of the traditional likelihood paradigm.
for a certain candidate model, we assume there exists a k-dimensional parametric class of density functions F(k) = { f (y∣ θ k ) ∣ θ k ∈ Θ(k)} , a class in which the parameter space Θ(k) consists of k-dimensional vectors whose components are functionally independent. Let L(θ k ∣ y) denote the likelihood corresponding to the density f (y∣ θ k ), i.e., L(θ k ∣ y) = f (y∣ θ k ). Let θˆk denote a vector of estimates obtained by maximizing L(θ k ∣ y) over Θ(k). Suppose we formulate a collection of candidate models of various dimensions k. These models may be based on different subsets of explanatory variables, different mean and variance/covariance structures, and even different specifications for the type of distribution for the response variable. Our objective is to search among this collection for the fitted model that “best” approximates g(y). In the development of AIC, optimal approximation is defined in terms of a well-known measure that can be used to gauge the similarity between the true model g(y) and a candidate model f (y∣ θ k ): the Kullback–Leibler information (Kullback and Leibler ; Kullback ). The Kullback–Leibler information between g(y) and f (y∣ θ k ) with respect to g(y) is defined as I(θ k ) = E {log
g(y) }, f (y∣ θ k )
where E(⋅) denotes the expectation under g(y). It can be shown that I(θ k ) ≥ with equality if and only if f (y∣ θ k ) is the same density as g(y). I(θ k ) is not a formal metric, yet we view the measure in a similar manner to a distance: i.e., as the disparity between f (y∣ θ k ) and g(y) grows, the magnitude of I(θ k ) will generally increase to reflect this separation. Next, define d(θ k ) = E{− log f (y∣ θ k )}. We can then write
Background To formally introduce AIC, consider the following model selection framework. Suppose we endeavor to find a suitable model to describe a collection of response measurements y. We will assume that y has been generated according to an unknown density g(y). We refer to g(y) as the true or generating model. A model formulated by the investigator to describe the data y is called a candidate or approximating model. We will assume that any candidate model structurally corresponds to a parametric class of distributions. Specifically,
I(θ k ) = d(θ k ) − E{− log g(y)}. Since E{− log g(y)} does not depend on θ k , any ranking of a set of candidate models corresponding to values of I(θ k ) would be identical to a ranking corresponding to values of d(θ k ). Hence, for the purpose of discriminating among various candidate models, d(θ k ) serves as a valid substitute for I(θ k ). We will refer to d(θ k ) as the Kullback discrepancy. To measure the separation between between a fitted candidate model f (y∣ θˆk ) and the generating model
Akaike’s Information Criterion: Background, Derivation, Properties, and Refinements
g(y), we consider the Kullback discrepancy evaluated at θˆk : d(θˆk ) = E{− log f (y∣ θ k )}∣ ˆ . θ k =θ k
Obviously, d(θˆk ) would provide an attractive means for comparing various fitted models for the purpose of discerning which model is closest to the truth. Yet evaluating d(θˆk ) is not possible, since doing so requires knowledge of the true distribution g(⋅). The work of Akaike (, ), however, suggests that − log f (y∣ θˆk ) serves as a biased estimator of d(θˆk ), and that the bias adjustment E{d(θˆk )} − E{− log f (y∣ θˆk )}
AIC = − log f (y∣ θˆk ) + k will asymptotically approach the expected value of d(θˆk ), say ∆(k) = E{d(θˆk )}. Specifically, we will establish that E{AIC} + o() = ∆(k).
The justification of () requires the strong assumption that the true density g(y) is a member of the candidate class F(k). Under this assumption, we may define a parameter vector θ o having the same size as θ, and write g(y) using the parametric form f (y∣ θ o ). The assumption that f (y∣ θ o ) ∈ F(k) implies that the fitted model is either correctly specified or overfit. To justify (), consider writing ∆(k) as indicated: ∆(k) ˆ = E{d(θ)} ˆ = E{− log f (y∣ θ)}
()
can often be asymptotically estimated by twice the dimension of θ k . Since k denotes the dimension of θ k , under appropriate conditions, the expected value of
()
AIC therefore provides an asymptotically unbiased estimator of ∆(k). ∆(k) is often called the expected Kullback discrepancy. In AIC, the empirical log-likelihood term − log f (y∣ θˆk ) is called the goodness-of-fit term. The bias correction k is called the penalty term. Intuitively, models which are too simplistic to adequately accommodate the data at hand will be characterized by large goodness-of-fit terms yet small penalty terms. On the other hand, models that conform well to the data, yet do so at the expense of containing unnecessary parameters, will be characterized by small goodness-of-fit terms yet large penalty terms. Models that provide a desirable balance between fidelity to the data and parsimony should correspond to small AIC values, with the sum of the two AIC components reflecting this balance.
A
ˆ + [E{− log f (y∣ θ o )} − E{− log f (y∣ θ)}] ˆ − E{− log f (y∣ θ o )}]. + [E{d(θ)}
() ()
The following lemma asserts that () and () are both within o() of k. We assume the necessary regularity conditions required to ensure the consistency and 7asymptotic normality of ˆ the maximum likelihood vector θ. Lemma ˆ = k + o(), () E{− log f (y∣ θ o )} − E{− log f (y∣ θ)} ˆ − E{− log f (y∣ θ o )} = k + o(). () E{d(θ)} Proof Define I(θ) = E [−
∂ log f (y∣ θ) ] ∂θ∂θ ′
and I(θ, y) = [−
∂ log f (y∣ θ) ]. ∂θ∂θ ′
I(θ) denotes the expected Fisher information matrix and I(θ, y) denotes the observed Fisher information matrix. First, consider taking a second-order expansion of ˆ and evaluating the expectation of − log f (y∣ θ o ) about θ, ˆ the the result. Since − log f (y∣ θ) is minimized at θ = θ, first-order term disappears, and we obtain ˆ E{− log f (y∣ θ o )} = E{− log f (y∣ θ)} ′ ˆ y)}(θˆ − θ o )} + E {(θˆ − θ o ) {I(θ, + o(). Thus,
Derivation To justify AIC as an asymptotically unbiased estimator of ∆(k), we will focus on a particular candidate class F(k). For notational simplicity, we will suppress the dimension index k on the parameter vector θ k and its estimator θˆk .
ˆ E{− log f (y∣ θ o )} − E{− log f (y∣ θ)} ˆ y)}(θˆ − θ o )} + o(). = E {(θˆ − θ o ) {I(θ, ′
()
Next, consider taking a second-order expansion of ˆ about θ o , again evaluating the expectation of the d(θ)
A
A
Akaike’s Information Criterion: Background, Derivation, Properties, and Refinements
result. Since d(θ) is minimized at θ = θ o , the first-order term disappears, and we obtain ˆ = E{− log f (y∣ θ o )} E{d(θ)} ′ + E {(θˆ − θ o ) {I(θ o )}(θˆ − θ o )} + o(). Thus, ˆ − E{− log f (y∣ θ o )} E{d(θ)} = E {(θˆ − θ o ) {I(θ o )}(θˆ − θ o )} + o(). ′
()
Recall that by assumption, θ o ∈ Θ(k). Therefore, the quadratic forms ˆ y)}(θˆ − θ o ) and (θˆ − θ o ) {I(θ o )}(θˆ − θ o ) (θˆ − θ o ) {I(θ, ′
′
both converge to centrally distributed chi-square random variables with k degrees of freedom. Thus, the expectations of both quadratic forms are within o() of k. This fact along with () and () establishes () and ().
Properties The previous lemma establishes that AIC provides an asymptotically unbiased estimator of ∆(k) for fitted candidate models that are correctly specified or overfit. From a practical perspective, AIC estimates ∆(k) with negligible bias in settings where n is large and k is comparatively small. In settings where n is small and k is comparatively large (e.g., k ≈ n/), k is often much smaller than the bias adjustment, making AIC substantially negatively biased as an estimator of ∆(k). If AIC severely underestimates ∆(k) for higher dimensional fitted models in the candidate collection, the criterion may favor the higher dimensional models even when the expected discrepancy between these models and the generating model is rather large. Examples illustrating this phenomenon appear in Linhart and Zucchini (, –), who comment (p. ) that “in some cases the criterion simply continues to decrease as the number of parameters in the approximating model is increased.” AIC is asymptotically efficient in the sense of Shibata (, ), yet it is not consistent. Suppose that the generating model is of a finite dimension, and that this model is represented in the candidate collection under consideration. A consistent criterion will asymptotically select the fitted candidate model having the correct structure with probability one. On the other hand, suppose that the generating model is of an infinite dimension, and therefore
lies outside of the candidate collection under consideration. An asymptotically efficient criterion will asymptotically select the fitted candidate model which minimizes the mean squared error of prediction. From a theoretical standpoint, asymptotic efficiency is arguably the strongest optimality property of AIC. The property is somewhat surprising, however, since demonstrating the asymptotic unbiasedness of AIC as an estimator of the expected Kullback discrepancy requires the assumption that the candidate model of interest subsumes the true model.
Refinements A number of AIC variants have been developed and proposed since the introduction of the criterion. In general, these variants have been designed to achieve either or both of two objectives: () to relax the assumptions or expand the setting under which the criterion can be applied, () to improve the small-sample performance of the criterion. In the Gaussian linear regression framework, Sugiura () established that the bias adjustment () can be exactly evaluated for correctly specified or overfit models. The resulting criterion, with a refined penalty term, is known as “corrected” AIC, or AICc. Hurvich and Tsai () extended AICc to the frameworks of Gaussian nonlinear regression models and time series autoregressive models. Subsequent work has extended AICc to other modeling frameworks, such as autoregressive moving average models, vector autoregressive models, and certain 7generalized linear models and 7linear mixed models. The Takeuchi () information criterion, TIC, was derived by obtaining a general, large-sample approximation to each of () and () that does not rely on the assumption that the true density g(y) is a member of the candidate class F(k). The resulting approximation is given by the trace of the product of two matrices: an information matrix based on the score vector, and the inverse of an information matrix based on the Hessian of the log likelihood. Under the assumption that g(y) ∈ F(k), the information matrices are equivalent. Thus, the trace reduces to k, and the penalty term of TIC reduces to that of AIC. Bozdogon () proposed a variant of AIC that corrects for its lack of consistency. The variant, called CAIC, has a penalty term that involves the log of the determinant of an information matrix. The contribution of this term leads to an overall complexity penalization that increases with the sample size at a rate sufficient to ensure consistency. Pan () introduced a variant of AIC for applications in the framework of generalized linear models fitted
Algebraic Statistics
using generalized estimating equations. The criterion is called QIC, since the goodness-of-fit term is based on the empirical quasi-likelihood. Konishi and Kitagawa () extended the setting in which AIC has been developed to a general framework where () the method used to fit the candidate model is not necessarily maximum likelihood, and () the true density g(y) is not necessarily a member of the candidate class F(k). Their resulting criterion is called the generalized information criterion, GIC. The penalty term of GIC reduces to that of TIC when the fitting method is maximum likelihood. AIC variants based on computationally intensive methods have also been proposed, including crossvalidation (Stone ; Davies et al. ), bootstrapping (Ishiguro et al. ; Cavanaugh and Shumway ; Shibata ), and Monte Carlo simulation (Hurvich et al. ; Bengtsson and Cavanaugh ). These variants tend to perform well in settings where the sample size is small relative to the complexity of the models in the candidate collection.
About the Authors Joseph E. Cavanaugh is Professor of Biostatistics and Professor of Statistics and Actuarial Science at The University of Iowa. He is an associate editor of the Journal of the American Statistical Association (–present) and the Journal of Forecasting (–present). He has published over refereed articles. Andrew Neath is a Professor of Mathematics and Statistics at Southern Illinois University Edwardsville. He has been recognized for his work in science education. He is an author on numerous papers, merging Bayesian views with model selection ideas. He wishes to thank Professor Miodrag Lovric for the honor of an invitation to contribute to a collection containing the works of so many notable statisticians.
Cross References 7Akaike’s Information Criterion 7Cp Statistic 7Kullback-Leibler Divergence 7Model Selection
References and Further Reading Akaike H () Information theory and an extension of the maximum likelihood principle. In: Petrov BN, Csáki F (eds) Proceedings of the nd International symposium on information theory. Akadémia Kiadó, Budapest, pp – Akaike H () A new look at the statistical model identification. IEEE T Automat Contra AC-:–
A
Bengtsson T, Cavanaugh JE () An improved Akaike information criterion for state-space model selection. Comput Stat Data An :– Bozdogan H () Model selection and Akaike’s information criterion (AIC): the general theory and its analytical extensions. Psychometrika :– Cavanaugh JE, Shumway RH () A bootstrap variant of AIC for state-space model selection. Stat Sinica :– Davies SL, Neath AA, Cavanaugh JE () Cross validation model selection criteria for linear regression based on the KullbackLeibler discrepancy. Stat Methodol :– Hurvich CM, Shumway RH, Tsai CL () Improved estimators of Kullback-Leibler information for autoregressive model selection in small samples. Biometrika :– Hurvich CM, Tsai CL () Regression and time series model selection in small samples. Biometrika :– Ishiguro M, Sakamoto Y, Kitagawa G () Bootstrapping log likelihood and EIC, an extension of AIC. Ann I Stat Math :– Konishi S, Kitagawa G () Generalised information criteria in model selection. Biometrika :– Kullback S () Information Theory and Statistics. Dover, New York Kullback S, Leibler RA () On information and sufficiency. Ann Math Stat :– Linhart H, Zucchini W () Model selection. Wiley, New York Pan W () Akaike’s information criterion in generalized estimating equations. Biometrics :– Shibata R () Asymptotically efficient selection of the order of the model for estimating parameters of a linear process. Ann Stat :– Shibata R () An optimal selection of regression variables. Biometrika :– Shibata R ()Bootstrap estimate of Kullback-Leibler information for model selection. Stat Sinica :– Stone M () An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. J R Stat Soc B :– Sugiura N () Further analysis of the data by Akaike’s information criterion and the finite corrections. Commun Stat A:– Takeuchi K () Distribution of information statistics and criteria for adequacy of models. Mathematical Sciences :–(in Japanese)
Algebraic Statistics Sonja Petrovi´c , Aleksandra B. Slavkovi´c Research Assistant Professor University of Illinois at Chicago, Chicago, IL, USA Associate Professor The Pennsylvania State University, University Park, PA, USA Algebraic statistics applies concepts from algebraic geometry, commutative algebra, and geometric combinatorics to better understand the structure of statistical models, to
A
A
Algebraic Statistics
improve statistical inference, and to explore new classes of models. Modern algebraic geometry was introduced to the field of statistics in the mid s. Pistone and Wynn () used Gröbner bases to address the issue of confounding in design of experiments, and Diaconis and Sturmfels () used them to perform exact conditional tests. The term algebraic statistics was coined in the book by Pistone et al. (), which primarily addresses experimental design. The current algebraic statistics literature includes work on contingency tables, sampling methods, graphical and latent class models, and applications in areas such as statistical disclosure limitation (e.g., Dobra et al. ()), and computational biology and phylogenetics (e.g., Pachter and Sturmfels ()).
Algebraic Geometry of Statistical Models Algebraic geometry is a broad subject that has seen an immense growth over the past century. It is concerned with the study of algebraic varieties, defined to be (closures of) solution sets of systems of polynomial equations. For an introduction to computational algebraic geometry and commutative algebra, see Cox et al. (). Algebraic statistics studies statistical models whose parameter spaces correspond to real positive parts of algebraic varieties. To demonstrate how this correspondence works, consider the following simple example of the independence model of two binary random variables, X and Y, such that joint probabilities are arranged in a × matrix p := [pij ]. The model postulates that the joint probabilities factor as a product of marginal distributions: pij = pi+ p+j , where i, j ∈ {, }. This is referred to as an explicit algebraic statistical model. Equivalently, the matrix p is of rank , that is, its × determinant is zero: p p − p p = . This is referred to as an implicit description of the independence model. In algebraic geometry, the set of rank- matrices, where we allow pij to be arbitrary complex numbers, is a classical object called a Segre variety. Thus, the independence model is the real positive part of the Segre variety. Exponential family models, in general, correspond to toric varieties, whose implicit description is given by a set of binomials. For a broad, general definition of algebraic statistical models, see Drton and Sullivant (). By saying that “we understand the algebraic geometry of a model,” we mean that we understand some basic information about the corresponding variety, such as: degree, dimension and codimension (i.e., degrees of freedom); the defining equations (i.e., the implicit description of the model); the singularities (i.e., degeneracy in the model). The current algebraic statistics literature demonstrates that understanding the geometry of a model can be useful
for statistical inference (e.g., exact conditional inference, goodness-of-fit testing, parameter identifiability, and maximum likelihood estimation). Furthermore, many relevant questions of interest in statistics relate to classical open problems in algebraic geometry.
Algebraic Statistics for Contingency Tables A paper by Diaconis and Sturmfels () on algebraic methods for discrete probability distributions stimulated much of the work in algebraic statistics on contingency tables, and has led to two general classes of problems: () algebraic representation of a statistical model, and () conditional inference. The algebraic representation of the independence model given above generalizes to any k-way table and its corresponding hierarchical log-linear models (e.g., see Dobra et al. ()). A standard reference on log-linear models is Bishop et al. (). Most of the algebraic work for contingency tables has focused on geometric characterizations of log-linear models and estimation of cell probabilities under those models. Algebraic geometry naturally provides an explicit description of the closure of the parameter space. This feature has been utilized, for example, by Eriksson et al. () to describe polyhedral conditions for the nonexistence of the MLE for log-linear models. More recently, Petrovi´c et al. () provide the first study of algebraic geometry of the p random graph model of Holland and Leinhardt (). Conditional inference relies on the fact that datadependent objects are a convex bounded set, Pt = {x : xi ∈ R≥ , t = Ax}, where x is a table, A is a design matrix, and t a vector of constraints, typically margins, that is, sufficient statistics of a log-linear model. The set of all integer points inside Pt is referred to as a fiber, which is the support of the conditional distribution of tables given t, or the so-called exact distribution. Characterization of the fiber is crucial for three statistical tasks: counting, sampling and optimization. Diaconis and Sturmfels () provide one of the fundamental results in algebraic statistics regarding sampling from exact distributions. They define a Markov basis, a set of integer valued vectors in the kernel of A, which is a smallest set of moves needed to perform a 7random walk over the space of tables and to guarantee connectivity of the chain. In Hara et al. (), for example, the authors use Markov bases for exact tests in a multiple logistic regression. The earliest application of Markov bases, counting and optimization was in the area of statistical disclosure limitation for exploring issues of confidentiality with the release of contingency table data; for an overview,
Algebraic Statistics
see Dobra et al. (), and for other related topics, see Chen et al. (), Onn (), and Slavkovi´c and Lee ().
Graphical and Mixture Models Graphical models (e.g., Lauritzen ()) are an active research topic in algebraic statistics. Non-trivial problems, for example, include complete characterization of Markov bases for these models, and counting the number of solutions of their likelihood equations. Geiger et al. () give a remarkable result in this direction: decomposable graphical models are precisely those whose Markov bases consist of squarefree quadrics, or, equivalently, those graphical models whose maximum likelihood degree is . More recently, Feliz et al. () made a contribution to the mathematical finance literature by proposing a new model for analyzing default correlation. 7Mixture models, including latent class models, appear frequently in statistics, however, standard asymptotics theory often does not apply due to the presence of singularities (e.g., see Watanabe ()). Singularities are created by marginalizing (smooth) models; geometrically, this is a projection of the corresponding variety. Algebraically, mixture models correspond to secant varieties. The complexity of such models presents many interesting problems for algebraic statistics; e.g., see Fienberg et al. () for the problems of maximum likelihood estimation and parameter identifiability in latent class models. A further proliferation of algebraic statistics has been supported by studying mixture models in phylogenetics (e.g., see Allman et al. ()), but many questions about the geometry of these models still remain open.
Further Reading There are many facets of algebraic statistics, including generalizations of classes of models discussed above: experimental design, continuous multivariate problems, and new connections between algebraic statistics and information geometry. For more details see Putinar and Sullivant (), Drton et al. (), Gibilisco et al. (), and references given therein. Furthermore, there are many freely available algebraic software packages (e.g., ti (ti team), CoCoA (CoCoATeam)) that can be used for relevant computations alone, or in combination with standard statistical packages.
Acknowledgments Supported in part by National Science Foundation grant SES- to the Department of Statistics, Pennsylvania State University.
A
Cross References 7Categorical Data Analysis 7Confounding and Confounder Control 7Degrees of Freedom 7Design of Experiments: A Pattern of Progress 7Graphical Markov Models 7Logistic Regression 7Mixture Models 7Statistical Design of Experiments (DOE) 7Statistical Inference 7Statistical Inference: An Overview
References and Further Reading ti team. ti – a software package for algebraic, geometric and combinatorial problems on linear spaces. http://WWW.ti.de Allman E, Petrovi´c S, Rhodes J, Sullivant S () Identifiability of two-tree mixtures under group-based models. IEEE/ACM Trans Comput Biol Bioinfor. In press Bishop YM, Fienberg SE, Holland PW () Discrete multivariate analysis: theory and practice. MIT Cambridge, MA (Reprinted by Springer, ) Chen Y, Dinwoodie I, Sullivant S () Sequential importance sampling for multiway tables. Ann Stat ():– CoCoATeam. CoCoA: a system for doing computations in commutative algebra. http://cocoa.dima.unige.it Cox D, Little J, O’Shea D () Ideals, varieties, and algorithms: an introduction to computational algebraic geometry and commutative algebra, rd edn. Springer, New York Diaconis P, Sturmfels B () Algebraic algorithms for sampling from conditional distributions. Ann Stat :– Dobra A, Fienberg SE, Rinaldo A, Slavkovi´c A, Zhou Y() Algebraic statistics and contingency table problems: estimations and disclosure limitation. In: Emerging Applications of Algebraic Geometry: IMA volumes in mathematics and its applications, :– Drton M, Sturmfels B, Sullivant S () Lectures on algebraic statistics, vol. Oberwolfach seminars, Birkhäuser Eriksson N, Fienberg SE, Rinaldo A, Sullivant S () Polyhedral conditions for the nonexistence of the mle for hierarchical logliner models. J Symb Comput ():– Feliz I, Guo X, Morton J, Sturmfels B () Graphical models for correlated default. Math Financ (in press) Fienberg SE, Hersh P, Zhou Y () Maximum likelihood estimation in latent class models for contingency table data. In: Gibilisco P, Riccomagno E, Rogantin M, Wynn H (eds) Algebraic and geometric methods in statistics. Cambridge University Press, London, pp – Geiger D, Meek C, Sturmfels B () On the toric algebra of graphical models. Ann Stat ():– Gibilisco P, Riccomagno E, Rogantin M, Wynn H () Algebraic and geometric methods in statistics, Cambridge University press Hara H, Takemura A, Yoshida R () On connectivity of fibers with positive marginals in multiple logistic regression. J Multivariate Anal ():– Holland PW, Leinhardt S () An exponential family of probability distributions for directed graphs (with discussion). J Am Stat Assoc :–
A
A
Almost Sure Convergence of Random Variables
Lauritzen SL () Graphical models. Clarendon, Oxford Onn S () Entry uniqueness in margined tables. Lect Notes Comput Sci :– Pachter L, Sturmfels B () Algebraic statistics for computational biology. Cambridge University Press, New York, NY Petrovi´c S, Rinaldo A, Fienberg SE () Algebraic statistics for a directed random graph model with reciprocation. In: Viana MAG, Wynn H (eds) Algebraic methods in statistics and probability, II, Contemporary Mathematics. Am Math Soc Pistone G, Wynn H () Generalised confounding with Gröbner bases. Biometrika ():– Pistone G, Riccomagno E, Wynn H () Algebraic statistics: computational commutative algebra in statistics. CRC, Boca Raton Putinar M, Sullivant S () Emerging applications of algebraic geometry. Springer, Berlin Slavkovi´c AB, Lee J () Synthetic two-way contingency tables that preserve conditional frequencies. Stat Methodal (): – Watanabe S () Algebraic geometry and statistical learning theory: Cambridge monographs on applied and computational mathematics, , NewYork, Cambridge University Press
Almost Sure Convergence of Random Variables Herold Dehling Professor Ruhr-Universität Bochum, Bochum, Germany
Definition and Relationship to Other Modes of Convergence Almost sure convergence is one of the most fundamental concepts of convergence in probability and statistics. A sequence of random variables (Xn )n≥ , defined on a common probability space (Ω, F, P), is said to converge almost surely to the random variable X, if P({ω : lim Xn (ω) = X(ω)}) = . n→∞
a.s.
Commonly used notations are Xn Ð→ X or limn→∞ Xn = X (a.s.). Conceptually, almost sure convergence is a very natural and easily understood mode of convergence; we simply require that the sequence of numbers (Xn (ω))n≥ converges to X(ω) for almost all ω ∈ Ω. At the same time, proofs of almost sure convergence are usually quite subtle. There are rich connections of almost sure convergence with other classical modes of convergence, such as convergence in probability, defined by limn→∞ P(∣Xn − X∣ ≥ є) = for all є > , convergence in distribution, defined by limn→∞ Ef (Xn ) = Ef (X) for all real-valued bounded, continuous functions f , and convergence in Lp , defined by limn→∞ E∣Xn − X∣p = . Almost sure convergence implies
convergence in probability, which again implies convergence in distribution, but not vice versa. Almost sure convergence neither implies nor is it implied by convergence in Lp . A standard counterexample, defined on the probability space [, ], equipped with the Borel σ-field and Lebesgue measure, is the sequence Xn (ω) = [ j , j+ ] (ω), if n = k +j, k
k
k ≥ , ≤ j < k . The sequence (Xn )n≥ converges to zero in probability and in Lp , but not almost surely. On the same probability space, the sequence defined by Xn = n/p [, ] n provides an example that converges to zero almost surely, but not in Lp . Although convergence in probability does not imply almost sure convergence, there is a partial result in this direction. If (Xn )n≥ converges in probabilty to X, one can a.s. find a subsequence (nk )k≥ such that Xnk Ð→ X. Skorohod’s almost sure representation theorem is a partial converse to the fact that almost sure convergence implies convergence in distribution. If (Xn )n≥ converges in distribution to X, one can find a sequence of random variables (Yn )n≥ and a random variable Y such that Xn and Yn have the same distribution, for each n, X and Y have the same distribution, and limn→∞ Yn = Y almost surely. Originally proved by Skorohod () for random variables with values in a separable metric space, this representation theorem has been extended by Dudley () to noncomplete spaces and later by Wichura () to nonseparable spaces. By some standard arguments, one can show that almost sure convergence of (Xn )n≥ to X is equivalent to lim P(sup ∣Xk − X∣ ≥ є) = , for all є > .
n→∞
k≥n
Thus almost sure convergence holds, if the series ∑k≥ P(∣Xk − X∣ ≥ є) converges. In this case, the sequence (Xn )n≥ is said to converge completely to X.
Important Almost Sure Convergence Theorems Historically the earliest and also the best known almost sure convergence theorem is the Strong Law of Large Numbers, established originally by Borel (). Given an i.i.d. sequence (Xk )k≥ of random variables that are uniformly distributed on [, ], Borel showed that a.s. Sn Ð→ E(X ), n where Sn := ∑nk= Xk denotes the partial sum. Later, this was generalized to sequences with arbitrary distributions. Finally, Kolmogorov () could show that the existence of first moments is a necessary and sufficient condition for the strong law of large numbers for i.i.d. random variables.
Almost Sure Convergence of Random Variables
Hsu and Robbins () showed complete convergence in the law of large numbers, provided the random variables have finite second moments; Baum and Katz () showed that this condition is also necessary. Birkhoff () proved the Ergodic Theorem, i.e., the validity of the strong law of large numbers for stationary ergodic sequences (Xk )k≥ with finite first moments. Kingman () generalized this to the Subadditive Ergodic Theorem, valid for doubly indexed subadditive process (Xs,t ) satisfying a certain moment condition. Doob () established the Martingale Convergence Theorem, which states that every L -bounded submartingale converges almost surely. The Marcinkiewicz-Zygmund Strong Law of Large Numbers () is a sharpening of the law of large numbers for partial sums of i.i.d. random variables, stating that for ≤ p < we have lim n→∞
n/p
The functional law of the iterated logarithm gives a remarkably sharp information about the behavior of the partial sum process. The Almost Sure Invariance Principle, originally established by Strassen () is an important technical tool in many limit thorems. Strassen’s theorem states that for i.i.d. random variables with finite variance, one can define a standard Brownian motion (see 7Brownian Motion and Diffusions) W(k) satisfying n √ ∑(Xk − E(X )) − σW(n) = o( n log log n), a.s.. k=
Komlos et al. () gave a remarkable sharperning of the error term in the almost sure invariance principle, showing that for p > one can find a standard Brownian motion (Wt )t≥ satisfying n
n
∑(Xk − E(Xk )) = a.s., k=
if and only if the random variables have finite p-th moments. Note that for p = this result is false as it would contradict the central limit theorem (see 7Central Limit Theorems). For i.i.d. random variables with finite variance σ ≠ , Hartman and Wintner () proved the Law of the Iterated Logarithm, stating that n lim sup √ ∑(Xi − E(X )) = a.s., n→∞ σ n log log n k=
and that the corresponding lim inf equals −. In the special case of a symmetric 7random walk, this theorem had been established earlier by Khintchin (). The law of the iterated logarithm gives a very precise information about the behavior of the centered partial sum. Strassen () proved the Functional Law of the Iterated Logarithm, which concerns the normalized partial sum process, defined by k k fn ( ) := √ ∑(Xi − E(Xi )), ≤ k ≤ n, n σ n log log n i=
and linearly interpolated in between. The random sequence of functions (fn )n≥ is almost surely relatively compact and has the following set of limit points
A
∑(Xk − E(X )) − σW(n) = o(n
/p
), a.s..
k=
if and only if the random variables have finite p-th moments. In this way, results that hold for Brownian motion can be carried over to the partial sum process. E.g., many limit theorems in the statistical analysis of change-points are proved by a suitable application of strong approximations. In the s, Brosamler, Fisher and Schatte independently discovered the Almost Sure Central Limit Theorem, stating that for partial sums Sk := ∑ki= Xi of an i.i.d. sequence (Xi )i≥ with mean zero and variance σ n ∑ {Sk /σ √k≤x} = Φ(x), n→∞ log n k= k lim
where Φ(x) = ∫−∞ √π e−t / dt denotes the standard normal distribution function. The remarkable feature of this theorem is that one can observe the central limit theorem, which in principle is a distributional limit theorem, along a single realization of the process. In , Glivenko and Cantelli independently discovered a result that is now known as the Glivenko–Cantelli Theorem (see 7Glivenko-Cantelli Theorems). Given a sequence (Xk )k≥ of i.i.d random variables with distribution function F(x) := P(X ≤ x), we define the empirical distribution function Fn (x) = n ∑nk= {Xk ≤x} . The Glivenko–Cantelli theorem states that x
a.s.
sup ∣Fn (x) − F(x)∣ Ð→ . x∈R
K = {x ∈ C[, ] : x is absolutely continuous and
′ ∫ (x (t)) dt ≤ }.
This theorem is sometimes called the fundamental theorem of statistics, as it shows that it is possible to recover the
A
A
Almost Sure Convergence of Random Variables
distribution of a random variable from a sequence of observations. Almost sure convergence has been established for Ustatistics, a class of sample statistics of great importance in mathematical statistics. Given a symmetric kernel h(x, y), we define the bivariate U-statistic n Un := ( )
−
∑ h(Xi , Xj ).
≤i<j≤n
Hoeffding () proved the U-Statistic Strong Law of Large Numbers, stating that for any integrable kernel and i.i.d. random variables (Xi )i≥ , a.s.
Un Ð→ Eh(X , X ). Aaronson et al. () established the corresponding U-Statistic Ergodic Theorem, albeit under extra conditions. The U-statistic Law of the Iterated Logarithm, in the case of i.i.d. data (Xi ) was established by Sen (). In the case of degenerate kernels, i.e., kernels satisfying Eh(x, X ) = , for all x, this was sharpened by Dehling et al. () and Dehling (). Their Degenerate U-Statistic Law of the Iterated Logarithm states that lim sup n→∞
∑ h(Xi , Xj ) = ch , a.s., n log log n ≤i<j≤n
where ch is the largest eigenvalue (see 7Eigenvalue, Eigenvector and Eigenspace) of the integral operator with kernel h(x, y). A functional version as well as an almost sure invariance principle were established by the same authors.
Proofs of Almost Sure Convergence In most situations, especially in applications in Statistics, almost sure convergence is proved by identifying a given sequence as a a continuous function of a sequence of a type studied in one of the basic theorems on almost sure convergence. The proofs of the basic almost sure convergence theorems are quite subtle and require a variety of technical tools, such as exponential inequalities, maximal inequalities, truncation techniques and the Borel-Cantelli lemma (see 7Borel–Cantelli Lemma and Its Generalizations).
About the Author Herold Dehling (born in Westrhauderfehn, Germany) is Professor of Mathematics at the Ruhr-Universität Bochum, Germany. From to , he was on the faculty of the University of Groningen, The Netherlands. Prior to that he held postdoc positions at Boston University and at the University of Göttingen. Herold Dehling studied Mathematics at the University of Göttingen and the
University of Illinois at Urbana-Champaign. He obtained his Ph.D. in at Göttingen. Herold Dehling is an elected member of the International Statistical Institute. In he was awarded the Prix Gay-Lussac–Humboldt of the Republic of France. Herold Dehling conducts research in the area of asymptotic methods in probability and statistics, with special emphasis on dependent processes. He has published more than research papers in probability and statistics. Herold Dehling is co-author of three books, Kansrekening (Epsilon Uitgaven, Utrecht , with J. N. Kalma), Einführung in die Wahrscheinlichkeitsrechnung und Statistik (Springer, Heidelberg , with B. Haupt) and Stochastic modelling in process technology (Elsevier Amsterdam , with T. Gottschalk and A. C. Hoffmann). Moreover, he is coeditor of the books Empirical Process Techniques for Dependent Data (Birkhäuser, Boston , with T. Mikosch and M. Sorensen) and Weak Dependence in Probability, Analysis and Number Theory (Kendrick Press, Utah , with I. Berkes, R. Bradley, M. Peligrad and R. Tichy).
Cross References 7Brownian Motion and Diffusions 7Convergence of Random Variables 7Ergodic Theorem 7Random Variable 7Weak Convergence of Probability Measures
References and Further Reading Aaronson J, Burton RM, Dehling H, Gilat D, Hill T, Weiss B () Strong laws for L- and U-statistics. Trans Am Math Soc :– Baum LE, Katz M () Convergence rates in the law of large numbers. Trans Am Math Soc :– Birkhoff GD () Proof of the Ergodic theorem. Proc Nat Acad Sci USA :– Borel E () Les probabilités dénombrables et leurs application arithmétique. Rendiconti Circolo Mat Palermo : — Brosamler G () An almost everywhere central limit theorem. Math Proc Cambridge Philos Soc :– Cantelli FP () Sulla determinazione empirica della leggi di probabilita. Gior Ist Ital Attuari :– Csörgo M, Révész P () Strong approximations in probability and statistics. Academic, New York Dehling H, Denker M, Philipp W () Invariance principles for von Mises and U-Statistics. Z Wahrsch verw Geb : – Dehling H () The functional law of the iterated logarithm for von-Mises functionals and multiple Wiener integrals. J Multiv Anal :– Dehling H () Complete convergence of triangular arrays and the law of the iterated logarithm for U-statistics. Stat Prob Lett :– Doob JL () Stochastic processes. Wiley, New York
Analysis of Areal and Spatial Interaction Data
Dudley RM () Distances of probability measures and random variables. Ann Math Stat :– Fisher A () Convex invariant means and a pathwise central limit theorem. Adv Math :– Glivenko VI () Sulla determinazione empirica della leggi di probabilita. Gior Ist Ital Attuari :– Hartmann P, Wintner A () On the law of the iterated logarithm. Am J Math :– Hoeffding W () The strong law of large numbers for U-statistics. University of North Carolina, Institute of Statistics Mimeograph Series Hsu PL, Robbins H () Complete convergence and the law of large numbers. Proc Nat Acad Sci USA :– Khintchin A () Über einen Satz der Wahrscheinlichkeitsrechnung. Fund Math :– Kingman JFC () The ergodic theory of subadditive stochastic processes. J R Stat Soc B :– Kolmogorov AN () Sur la loi forte des grandes nombres. Comptes Rendus Acad Sci Paris :– Komlos J, Major P, Tusnady G () An approximation of partial sums of independent RVs and the sample DF I. Z Wahrsch verw Geb :– Marcinkiewicz J, Zygmund A () Sur les fonctions indépendantes. Fund Math :– Schatte P () On strong versions of the central limit theorem. Math Nachr :– Sen PK () Limiting behavior of regular functionals of empirical distributions for stationary mixing processes. Z Wahrsch verw Geb :– Serfling RJ () Approximation theorems of mathematical statistics. Wiley, New York Skorohod AV () Limit theorems for stochastic processes. Theory Prob Appl :– Stout WF () Almost sure convergence. Academic, New York Strassen V () An invariance principle for the law of the iterated logarithm. Z Wahrsch verw Geb :– Van der Vaart AW () Asymptotic statistics. Cambridge University Press, Cambridge Wichura MJ () On the construction of almost uniformly convergent random variables with given weakly convergent image laws. Ann Math Stat :–
A
to countries, counties, townships, political districts, constituencies or other spatial regions that are featured by more or less natural boundaries. Examples for data yi might be the number of persons having a certain chronic illness, number of enterprises startups, average income, population density, number of working persons, area of cultivated land, air pollution, etc. Like all spatial data, also areal data are marked by the fact that they exert spatial correlation to the data from neighboring areas. Tobler () expresses this in his first law of geography: “everything is related to everything else, but near things are more related than distant things.” It is this spatial correlation which is investigated, modeled and taken into account in the analysis of areal data. Spatial proximity matrix. A mathematical tool that is common to almost all areal analysis methods is the socalled (n × n) spatial proximity matrix W, each of whose elements, wij , represents a measure of spatial proximity of area Ai and area Aj . According to Bailey and Gatrell () some possible criteria might be: ● ● ● ●
wij = if Aj shares a common boundary with Ai and wij = else. wij = if the centroid of Aj is one of the k nearest centroids to that of Ai and wij = else. γ wij = dij if the inter-centroid distance dij < δ (δ > , γ < ); and wij = else. l wij = liji , where lij is the length of common boundary between Ai and Aj and li is the perimeter of Ai .
All diagonal elements wii are set to . The spatial proximity matrix W must not be symmetric. For instance, case and case above lead to asymmetric proximity matrices. For more proximity measures we refer to Bailey and Gatrell () and any other textbook on areal spatial analysis like Anselin ().
Spatial Correlation Measures
Analysis of Areal and Spatial Interaction Data Gunter Spöck , Jürgen Pilz Associate Professor University of Klagenfurt, Klagenfurt, Austria Professor University of Klagenfurt, Klagenfurt, Austria
Areal Data Areal data yi are data that are assigned to spatial regions Ai , i = , , . . . , n. Such data and spatial areas naturally arise at different levels of spatial aggregation, like data assigned
Global measures of spatial correlation. The global Moran index I, first derived by Moran (), is a measure for spatial correlation of areal data having proximity matrix W. Defining S = ∑ni= ∑nj= wij and y¯, the mean of the data yi , i = , , . . . , n, the global Moran index may be written I=
n n n ∑i= ∑j= wij (yi − y¯)(yj − y¯) . S ∑ni= (yi − y¯)
()
Thus the global Moran index may be be interpreted as measuring correlation between y = (y , y , . . . , yn )T and the spatial lag-variable Wy. But the Moran index does not necessarily take values between − and . Its expectation . Values of the for independent data yi is E[I] = − n− Moran index larger than this value thus are an indication of
A
A
Analysis of Areal and Spatial Interaction Data
positive global spatial correlation; values smaller than this value indicate negative spatial correlation. A global correlation measure similar to the variogram known from classical geostatistics is the Geary-index (Geary’s c, Geary ): n
c=
n
n − ∑i= ∑j= wij (yi − yj ) S ∑ni= (yi − y¯)
()
Under the independence assumption for the yi its expectation is E[c] = . Values of c larger than indicate negative correlation and values smaller than positive correlation. The significance for Moran’s I and Geary’s c may be tested by means of building all n! permutations of the yi , i = , , . . . , n, assigning them to the the different areas Aj , j = , , . . . , n, calculating for each permutation Moran’s I or Geary’s c and then considering the distributions of these permuted spatial correlation statistics. True correlation statistics at the lower or upper end of these distributions are an indication of significance of the global correlation measures. A map often useful for detecting spatial clusters of high or low values is the so-called LISA map. It may be shown that Moran’s I is exactly the upward slope of the regression line between the regressors (y − n y¯) and the spatial lag-variables W(y − n y¯) as responses, where the matrix W is here standardized to have rows which sum up to one. The corresponding scatterplot has four quadrants PP, NN, PN and NP, with P and N indicating positive and negative values for the regressors and responses. If one codes these four classes into which the pairs [yi − y¯, ∑nj= wij (yj − y¯)] may fall with colors and visualizes these colors in a map of the areas one can easily detect clusters of areas that are surrounded by low or high neighboring values. Both statistics, the Moran I and Geary’s c make a global assumption of second order stationarity, meaning that the yi , i = , , . . . , n all have the same constant mean and variance. If one doubts that this condition is fully met one has to rely on local measures of spatial correlation, for local versions of Moran’s I and Geary’s c see Anselin ().
Spatial Linear Regression A problem frequently occuring in areal data analysis is the regression problem. Response variables yi and corresponding explanatory vectors xi are observed in spatial areas Ai , i = , , . . . , n and one is interested in the linear regression relationship yi ≈ xiT β, where β is an unknown regression parameter vector to be estimated. Subsuming all row vectors xiT in the (n × p) design matrix X and writing y = (y , y , . . . , yn )T the ordinary 7least squares solution to this regression problem, which does not take account
of spatial correlation, is known to be βˆ = (XT X)− XT y. If the data in y are known to be correlated the above ordinary least squares estimator is known to be inefficient and statistical significance tests in this regression model are known to be misleading. Problems may be resolved by considering the generalized least squares estimator βˆ = (XT Σ− X)− XT Σ− y, where the covariance matrix Σ is measuring the correlation between the data in y. All regression procedures used in areal data analysis deal more or less with the modeling and estimation of this covariance structure Σ and the estimation of β. In all subsequent sections we will assume that the spatial proximity matrix W is standardized such that its rows sum up to one. Simultaneous autoregressive model (SAR). The SAR model is given as follows: y = Xβ + u, u = λWu + є.
()
Here λ is an unknown parameter, − < λ < , measuring spatial correlation; the parameters λ and β are to be estimated. The error vector є has uncorrelated components with constant unknown variances σ , like u it has expectation zero. The two equations may be combined to get y = λWy + Xβ − λWXβ + є Obviously y is modeled as being influenced also by the spatial lag-variables Wy and the spatial lag-regression WXβ. The coefficient λ is measuring the strength of this influence. The covariance matrix of u may be shown to be cov[u] = σ ((In − λW)T (In − λW))− . An estimation procedure for the SAR model is implemented in the R-package spdep, Bivand (). It is based on the Gaussian assumption for y and iteratively calculates maximum (profile) likelihood estimates for σ and λ and generalized least squares estimates for β based on the covariance matrix cov[u] and the estimates for σ and λ calculated a step before. Spatial lag model. The so-called spatial lag model may be written y = λWy + Xβ + є.
()
It is simpler in structure than the SAR model because the lag-regression term −λWXβ is missing. For its estimation, again, an iterative profile likelihood procedure similar to the SAR procedure may be used. Spatial Durbin model. The spatial Durbin model is a generalization of the SAR model and given as y = λWy + Xβ + WXγ + є,
()
with WXγ having its own regression parameter vector γ. By means of the restriction γ = −λβ the Durbin model
Analysis of Areal and Spatial Interaction Data
becomes equivalent to a SAR model. The so-called common factor test (Florax and de Graaf ), a likelihood ratio test, can be used to decide between the two hypotheses, - SAR-model and spatial Durbin model. As an alternative to the above models one may also use a SAR model with a lag-error component y = Xβ + λWє + є.
()
Deciding between models. For the investigation whether a SAR model, a spatial lag model or ordinary least squares give the best fit to the data one may adopt Lagrange multiplier tests as described in Florax and de Graaf (). Interestingly, these tests are based on ordinary least squares residuals and for this reason are easily calculable. Breitenecker () gives a nice overview on all the possibilities related to testing models. Geographically weighted regression. Fotheringham et al. () propose, as an alternative to the above mentioned regression models, geographically weighted regression. The proposed methodology is particularly useful when the assumption of stationarity for the response and explanatory variables is not met and the regression relationship changes spatially. Denoting by (ui , vi ) the centroids of the spatial areas Ai , i = , , . . . , n, where the responses yi and explanatory vectors xi are observed, the model for geographically weighted regression may be written
A
appear in epidemiology, and Bayesian methods are not dealt with here; for those methods the interested reader is referred to Lawson ().
Spatial Interaction Data This is a further category of spatial data which is related to modeling the “flow” of people and/or objects between a set of origins and a set of destinations. In contrast with areal (and geostatistical) data, which are located at points or in areas, spatial interaction data are related to pairs of points, or pairs of areas. Typical examples arise in health services (e.g., flow to hospitals), transport of freight goods, population migration and journeys-to-work. Good introductory material on spatial interaction models can be found in Haynes and Fotheringham (). The primary objective is to model aggregate spatial interaction, i.e. the volume of flows, not the flows at an individual level. Having m origins and n destinations with associated flow data considered as random variables Yij (i = , . . . , m; j = , . . . , n), the general spatial interaction model is of the form Yij = µ ij + ε ij ; i = , . . . , m; j = , . . . , n
()
The regression vector β(ui , vi ) is thus dependent on the spatial location (ui , vi ) and is estimated by means of a weighted least squares estimator that is locally dependent on a diagonal weight matrix Ci :
where E(Yij ) = µ ij and ε ij are error terms with E(ε ij ) = . The goal is then to find suitable models for µ ij involving flow propensity parameters of the origins i, attractiveness parameters of the destinations j, and the effects of the “distances” dij between them. Here, the quantities dij may be real (Euclidean) distances, travel times, costs of travel or any other measure of the separation between origins and destinations. One of the most widely used classes of models for µ ij is the so-called gravity model
ˆ i , vi ) = (XT Ci X)− XT Ci y β(u
µ ij = α i β j exp(γ dij )
yi = xiT β(ui , vi ) + є i , i = , , . . . , n.
The diagonal elements
(i) cjj
()
of Ci are defined by means
(i) cjj
of a kernel function, e.g. = exp(−dij /h). Here dij is a value representing the distance beetween Ai and Aj ; dij may either be Euclidean distance or any other metric measuring distance between areas. Further, h is the bandwidth measuring how related areas are and can be determined by means of crossvalidating the residuals from the regression or based on the 7Akaike’s information criterion (Brunsdon et al. ). Selecting the bandwidth h too large results in oversmoothing of the data. On the other hand a bandwidth too small allows for too less data during estimation. All areal analysis methods discussed so far are implemented in the R-packages spdep and spgwr, (Bivand , ). Methods for counting data, as they frequently
()
involving origin parameters α i , destination parameters β j and a scaling parameter γ. Under the assumption that the Yij are independent Poisson random variables with mean µ ij , this model can be treated simply as a particular case of a generalised linear model with a logarithmic link. Model fitting can then proceed by deriving maximum likelihood estimates of the parameters using iteratively weighted least squares (IRLS) techniques. The above gravity models can be further enhanced when replacing the parameters β j by some function of observed covariates xj = (xj , . . . , xjk )T characterising the attractiveness of each of the destinations j = , . . . , n. Again, this is usually done in a log-linear way, and the model becomes µ ij = α i exp(g(xj , θ) + γ dij )
()
A
A
Analysis of Areal and Spatial Interaction Data
where g is some function (usually linear) of the vector of destination covariates and a vector of associated parameters θ. Contrary to (), which reproduces both the total flows from any origin and the total observed flows to each destination, the new model () is only origin-constrained. The obvious counterpart to () is one which is destinationconstrained: µ ij = β j exp(h(z i , ω) + γ dij ) where h is some function of origin characteristics z i and a vector of associated parameters ω. Finally, when modeling both α i and β j as functions of observed characteristics at origins and destinations, we arrive at the unconstrained model log µ ij = h(z i , ω) + g(xj , θ) + γ dij () In population migration one often uses a particular form of (), where z i and xj are taken to be univariate variables meaning the logarithms of the population Pi and Pj at origin i and destination j, respectively. Adding an overall scaling parameter τ to reflect the general tendency for migration, the following simple model results: Yij = τPiω Pjθ exp(γ dij ) + ε ij
()
Likewise, in all the above models one can introduce more complex distance functions than exp(γ dij ). Also, as mentioned before, dij could be replaced by a general separation term sij embracing travel time, actual distance and costs of overcoming distance. The interaction models considered so far are only models for µ ij , the mean flow from i to j. Thus, they are only first order models, no second order effects are included and the maximum likelihood methods for estimating the parameters of the gravity models rest on the explicit assumption that fluctuations about the mean are independent. Up to now, there has been only little work done on taking account of spatially correlated errors in interaction modeling. To address such problems, pseudolikelihood-methods are in order. Good references for further reading on spatial interaction models are Upton and Fingleton (), Bailey and Gatrell () and Anselin and Rey (). Spatial interaction models have found broad attention among (economic) geographers and within the GIS community, but have received only little attention in the spatial statistics community. The book by Anselin and Rey () forms a bridge between the two different worlds. It contains a reprint of the original paper by Getis (), who first suggested that the family of spatial interaction models is a special case of a general model of spatial autocorrelation. Fischer et al. () present a generalization of the GetisOrd statistic which enables to detect local non-stationarity
and extend the log-additive model of spatial interaction to a general class of spatial econometric origin-destination flow models, with an error structure that reflects origin and/or destination autoregressive spatial dependence. They finally arrive at the general spatial econometric model (), where the design matrix X includes the observed explanatory variables as well as the origin, destination and separation variables, and W is a row-standardized spatial weights matrix.
About the Author For biography of the author Jürgen Pilz see the entry 7Statistical Design of Experiments (DOE).
Cross References 7Data Depth 7Gaussian Processes 7Geostatistics and Kriging Predictors 7Model-Based Geostatistics 7Spatial Point Pattern 7Spatial Statistics 7Statistical Ecology
References and Further Reading Anselin L () Spatial econometrics: methods and models. Kluwer Academic, Dordrecht Anselin L () Local indicators of spatial association – LISA. Geogr Anal :– Anselin L, Rey SJ (eds) () Perspectives on spatial data analysis. Springer, Berlin Bailey T, Gatrell A () Interactive spatial data analysis. Longman Scientific and Technical, New York Breitenecker R () Raeumliche lineare modelle und autokorrelationsstrukturen in der gruendungsstatistik. Ibidem, Stuttgart Bivand R () SPDEP: spatial dependence: weighting schemes, statistics and models. R-package Version .- Bivand R () SPGWR: geographically weighted regression. R-package Version .- Brunsdon C, Fotheringham S, Charlton M () Geographically weighted regression – modelling spatial non-stationary. The Statistician :– Fischer MM, Reismann M, Scherngell Th () Spatial interaction and spatial autocorrelation. In: Rey SJ, Anselin A (eds) perspective on spatial data analysis. Springer, Berlin, pp – Florax R, de Graaf T () The performance of diagnostic tests for spatial dependence in linear regression models: a metaanalysis of simulation studies. In: Anselin L et al (eds) Advances in spatial econometrics: methodology, tools and applications. Springer, Berlin, pp – Fotheringham S, Brunsdon C, Charlton M () Geographically weighted regression: the analysis of spatially varying relationships. Wiley, Chichester
Analysis of Covariance
Geary R () The contiguity ratio and statistical mapping. Inc Stat :– Getis A () Spatial interaction and spatial autocorrelation: a cross-product approach. Environ plann A:– Haynes KF, Fotheringham AS () Gravity and spatial models. Sage, London Lawson A () Bayesian disease mapping: hierarchical modeling in spatial epidemiology. CRC, Chapman and Hall, New York Moran P () Notes on continuous stochastic phenomena. Biometrica :– Tobler W () A computer model simulating urban growth in the Detroit region. Econ Geogr :– Upton GJG, Fingleton B () Spatial data analysis by example, vol. . Wiley, Chichester
A
characteristic that varies across the experimental units. For example, one may test for a difference in the efficacy of a new medical treatment and an existing treatment protocol by randomly assigning the treatments to patients (experimental units) and testing for a difference in the outcomes. However, if the 7randomization results in the placement of a disproportionate number of young patients in the group that receives the new treatment and/or placement of a disproportionate number of elderly patients in the group that receives the existing treatment, the results will be biased if the treatment is more (or less) effective on young patients than it is on elderly patients. Under such conditions one could collect additional information on the patients’ ages and include this variable in the model. The resulting general linear model Yij = µ + τ j + βXij + ε ij , i = , . . . , nj , j = , . . . , m.
Analysis of Covariance James J. Cochran Associate Professor Louisiana Tech University, Ruston, LA, USA
Introduction The Analysis of Covariance (generally known as ANCOVA) is a statistical methodology for incorporating quantitatively measured independent observed (not controlled) variables in a designed experiment. Such a quantitatively measured independent observed variable is generally referred to as a covariate (hence the name of the methodology – analysis of covariance). Covariates are also referred to as concomitant variables or control variables. If we denote the general linear model (GLM) associated with a completely randomized design as Yij = µ + τ j + ε ij , i = , . . . , nj , j = , . . . , m where Yij = the ith observed value of the response variable at the jth treatment level µ = a constant common to all observations τ j = the effect of the jth treatment level ε ij = the random variation attributable to all uncontrolled influences on the ith observed value of the response variable at the jth treatment level For this model the within group variance is considered to be the experimental error, and this implies that the treatments have similar effects on all experimental units. However, in some experiments the effect of the treatments on the experimental units varies systematically with some
where Xij = the ith observed value of the covariate at the jth treatment level, β = the estimated change in the response that corresponds to a one unit increase in the value of the covariate at a fixed level of the treatment is said to be a completely randomized design ANCOVA model and describes an experimental design GLM one factor experiment with a single covariate. Note that the addition of covariate(s) can accompany many treatment and design structures. This article focuses on the simple one way treatment structure in a completely randomized design for the sake of simplicity and brevity.
Purpose of ANCOVA There are three primary purposes for including a covariate in the 7analysis of variance of an experiment: . To increase the precision of estimates of treatment means and inferences on differences in the response between treatment levels by accounting for concomitant variation on quantitative but uncontrollable variables. In this respect covariates are the quantitative analogies to blocks (which are qualitative/categorical) in that they are () not controlled and () used to remove a systematic source of variation from the experimental error. Note that while the inclusion of a covariate will result in a decrease in the experimental error, it will also reduce the degrees of freedom associated with the experimental error, and so inclusion of a covariate in an experimental model will not always result in greater precision and power.
A
A
Analysis of Covariance
. To allow for the assessment of the nature of the relationship between the covariate(s) and the response variable after taking into consideration the treatment effects. In this respect covariates are analogous to independent variables in linear regression, and their associated slopes can provide important insight into the nature of the relationship between the response and the covariate. . To statistically adjust comparisons of the response between groups for imbalances in quantitative but uncontrollable variables. In this respect covariates are analogous to stratification and are of particular importance in situations where stratification on the covariate is impractical or infeasible.
means) to conduct inference on individual LS means and contrasts based on the LS means.
Assumptions of ANCOVA In addition to the standard ANOVA assumptions: ● ● ●
One must also consider the regression assumptions when performing statistical inference with ANCOVA. The regression assumptions include: ●
Applications of ANCOVA Typical applications of analysis of covariance include: ●
●
●
●
Clinical trials in which quantitative but uncontrollable variables such as the weight, height, and age of the patients may influence the effectiveness of a treatment protocol. Marketing research in which quantitative but uncontrollable variables such as the pretest rating of a product given by a respondent may influence the respondent’s posttest rating (i.e., after exposure to the test condition) of the product. Education experiments in which quantitative but uncontrollable variables such as the age, intelligence (if this can be measured), and prior scholastic performance of the students may influence the effectiveness of a pedagogical approach. Agricultural experiments in which quantitative but uncontrollable variables such as rainfall and historical yield of fruit bearing trees may influence the yield during an experiment.
Comparing Treatments in ANCOVA Least squares means (or LS means) are generally used to compare treatment effects in experiments that include one or more covariates. LS means (which are sometimes referred to as marginal means, estimated marginal means, or adjusted treatment means) are the group means when the covariate is set equal to its grand mean X m (mean of the covariate over all observations across all treatments). These are easily calculated by substituting the grand mean of the covariate into the estimated general linear model, i.e., Yˆ j = µ + τ j + βX m , j = , . . . , m Standard errors for LS means are typically calculated and used (in conjunction with the 7asymptotic normality of LS
Independence of error terms Homogeneity of variance of the error terms across treatments Normality of the error terms across treatments
●
A linear relationship exists between the covariate and the response variable. If no relationship exists between the covariate and response, there is no reason to include the covariate in the experiment or resulting model. If the relationship between the covariate and the response variable is nonlinear, the inclusion of a covariate in the model will not remove all variation in the observed values of the response that can potentially be accounted for by the covariate. The nature of the relationship between the covariate and the response can be assessed with scatter plots of these two variables by treatment. If a nonlinear relationship exists between the covariate and the response, one can utilize a polynomial ANCOVA model. Homogeneity of the regression slopes associated with the covariate (i.e., parallel regression lines across treatments). The calculations of the LS means are predicated on the lack of existence of a response by covariate interaction. If this assumption is violated, the adjustment to the response variable for a common value of the covariate is misleading. This assumption can be assessed through either scatter plots of the covariate and the response by treatment or through the inclusion of a treatment-covariate interaction in the model.
If the sample results suggest that any of these assumptions are not satisfied, inference based on the model may not be valid.
Alternatives to ANCOVA Bonate () provides a good discussion of alternatives to ANCOVA in pretest-posttest designs; he considers the relative merits of difference scores, relative change functions, various blocking methods, and repeated-measures analysis. Several authors have suggested more general nonparametric alternatives to ANCOVA based on an analysis
Analysis of Multivariate Agricultural Data
of covariance of the ranks of the response and covariance. Some notable examples of these approaches have been suggested by Quade (, ), Puri and Sen (), McSweeney and Porter (), Burnett and Barr (), Shirley (), Conover and Iman (), Chang (), Lesaffre and Senn (), and Tsangari and Akritas ().
A
Analysis of Multivariate Agricultural Data Asghar Ali Professor and Chairman Bahauddin Zakariya University, Multan, Pakistan
About the Author For biography see the entry 7Role of Statistics in Advancing Quantitative Education.
Cross References 7Analysis of Variance 7General Linear Models 7Multivariate Analysis of Variance (MANOVA) 7Nonparametric Models for ANOVA and ANCOVA
Designs 7Rank Transformations 7Statistical Fallacies: Misconceptions, and Myths 7Statistics: An Overview
References and Further Reading Bonate PL () Analysis of pretest-posttest designs. Chapman and Hall/CRC, Boca Raton Burnett TD, Barr DR () A nonparametric analogy of analysis of covariance. Educ Psychol Meas ():– Chang GH () Nonparametric analysis of covariance in block designs. Dissertation (Texas Tech University) Conover WJ, Iman RL () Analysis of covariance using the rank transformation. Biometrics :– Doncaster CP, Davey AJH () Analysis of variance and covariance: how to choose and construct models for the life sciences. Cambridge University Press, Cambridge Huitema BE () The analysis of covariance and alternatives. Wiley, New York Lesaffre E, Senn S () A note on non-Parametric ANCOVA for covariate adjustment in randomized clinical trials. Stat Med ():– McSweeney M, Porter AC () Small sample properties of nonparametric index of response and rank analysis of covariance. Presented at the Annual Meeting of the American Educational Research Association, New York Milliken GA, Johnson DE () Analysis of messy data vol.: analysis of covariance. Chapman and Hall, New York Puri ML, Sen PK () Analysis of covariance based on general rank scores. Ann Math Stat :– Quade D () Rank analysis of covariance. J Am Stat Assoc :– Quade D () Nonparametric analysis of covariance by matching. Biometrics :– Rutherford A () Introducing ANOVA and ANCOVA: a GLM approach. Sage, Los Angeles Shirley EA () a Distribution-Free Method for Analysis of Covariance Based on Ranked Data. J Appl Stats :– Tsangari H, Akritas MG () Nonparametric ANCOVA with two and three covariates. J Multivariate Anal ():–
Agricultural research is most oftenly based on observational studies and experimentation resulting in multi-response variables. The selection of appropriate variety to grow; amount and types of fertilizers, insecticides and pesticides to apply; the irrigation system to use; the plant sowing technology to apply and to assess the soil fertility through chemical analysis of macro and micro nutrients available in the soil are the major areas of interest for the researcher to work on for the improvement of the agricultural productivity in terms of quality and quantity. The role of Statistics in planning agricultural research, designing experiments, data collection, analysis, modeling and interpretation of agricultural results is very well established. The basic principles and theoretical development of experimental designs pioneered by R. A. Fisher are the result of collaborative work of agricultural scientists and statisticians. In the process of experimentation and observational studies, the researcher is keen to have as many data information as possible so that nothing is left unattended related to the phenomenon under study as there will be no chance to repeat the experiment till the next season of the crop and it will not be less than a miracle if data from one year of the crop is consistent with the results of second year, no matter how much care is taken to keep the experimental conditions identical. Agricultural data obtained through experimentation is initially analyzed using 7analysis of variance technique and then depending on the nature of treatments/factors applied, either the approach of multiple comparisons or 7response surface methodology is used to explore further the hidden features of the data. For example, the experimenter might be interested to compare different varieties of a particular crop such that there are two local varieties (V , V ) in practice; three varieties are imported (V , V , V ) and two new varieties (V , V ) are developed by a local agricultural institute. If results obtained from analysis of variance conclude that performance of the varieties is significantly different from each other then obvious questions arise are to test the difference between the following variety comparisons: [V and V ]; [V and V ]; [(V , V ) and (V , V )]; [(V , V , V , V ) and (V , V , V )]; and if V is a hybrid variety, then one has two more comparisons to test i.e., [V with (V , V )]
A
A
Analysis of Multivariate Agricultural Data
and [V with V ]. These contrasts are orthogonal to each other but it will not always be the case, other techniques of multiple comparisons will have to be used then, which are available in almost all the books on experimental designs. On the other hand, if multifactor experiments are conducted to determine the appropriate levels of the applied factors on which optimum response is achieved. For this purpose data sets are modeled in adequate functional forms and the researcher is intended to fit simple functional form. Ordinary polynomials are the most popular functional forms which are used to model experimental data from many fields of scientific research. If first order polynomial is fitted, the researcher very simply states that the concerned factor has linear effect and the interpretation is made accordingly that with increase of levels of factor will result in increase (or decrease) in the response. The second order polynomials are used with the expectation that it will be possible to identify the levels of the applied factors to get the optimum response. A number of response functions that have been widely used by the agricultural and biological researchers have been discussed by Mead and Pike (). It should not be taken as granted that one response function considered applicable to one sort of situation will also be applicable to other similar situations; it is advisable that graphical approach be used to guess the appropriate functional form of the response under consideration. An extremely useful concept that is revealed by Nelder () is known as Inverse Polynomial Response Functions (IPRF). It emphasizes that in agricultural research the effect of increasing a factor indefinitely is either to produce a saturation effect, in which case the response does not exceed a finite amount, or to produce a toxic effect, in which case the response eventually fall to zero and the response curve has no build-in-symmetry. Nelder () and Pike () advocated these surfaces as giving responses that are nonnegative and bounded if regression coefficients are constrained to be positive and it is further assumed that Var(Y) ∝ [E(Y)] . Extension in the ideas has been developed by Nelder and Wedderburn () and McCullagh and Nelder () for the response variables that may not be normal and that the expected response may be a function of the linear predictors rather than just the linear predictors itself. Ali () and Ali et al. () have objected on placing constrains on the parameters as it will violate all the properties of good estimators and will no longer follow the distributional structure required for valid inferences. Their experience of examining many sets of data leads them not to expect all regression coefficients to be positive. Taking into account the error structure and functional form used for IPRF, Ali () proposed the form of a response function
called as Log Linear Response Functions (LLRF) based on the logarithmic transformation of the response variable and assuming that log Y ∼ N(E(log Y), σ ). The estimation of regression coefficients achieved by carrying out a multiple regression of log Y on the terms required fitting the data adequately; the resulting estimators are therefore Minimum Variance Linear Unbiased Estimators. It is simple to estimate the variance-covariance matrix of these estimators and to test hypotheses concerning parameters by the usual linear regression methods. On the theoretical grounds the LLRF model therefore has much to commend it. The assumption that log Yi follows the normal distribution may not always be true; in such cases it is recommended that Box-Cox family of transformation may be used under the same structure of the response function as has been used for LLRF and IPRF. In order to produce an adequate prediction the researcher is usually uncertain as to which of the large number of terms should be included in the final model. The main point to bear in mind is that it should have as many terms as necessary so that maximum variation of the data is explained and as few terms as possible so that it can easily be interpreted. Ali () argued that for summarizing the data from agricultural experiments the terms in the final model are required to be selected in a conforming order by preferring main effect terms over the interactions and lower order terms over the higher. It is further to remember that the inverse terms describe the rising ridge of the surfaces, the linear terms describe the optimum region and the higher degree terms contribute in explaining the falling portion of the surfaces. It is therefore recommended that for building the appropriate model one should concentrate on selection the inverse and linear terms along with their associated interaction terms. One who is not convinced with such types of model building method has the option to use the approach established by Nelder (). The methods used for selection of final model are mainly based on the Minimum Mean Square Error criterion. It is possible to find more than one models which fulfills this criterion. In such cases one should select the one which has reasonable shape of the response surface, capable to determine the values of quantitative factors at which the response is an optimum, statistically significant regression coefficients and simple functional form. There is no ambiguity to recognize the agricultural research as multifactor and multi-response and that these responses are measured at different stages of the maturity of the crop and that these are interrelated with one another. The univariate analyses of these variables therefore have partial impact on the true findings of research. Multivariate analyses are therefore natural and essential to consider the
Analysis of Multivariate Agricultural Data
data by giving due weight to the interrelationships among the variables under study. One possible approach which is widely used by the researchers is to study the correlation matrix of the variables. This approach only facilitates to assess the relationship among the pairs of variables and it can be extended to triplets of variables by considering the partial correlations and the multiple correlations among those. As a result there would be
p(p + ) pairs and
triplets to consider and it will certainly be confusing if the number p of variables under consideration is large. To overcome this difficulty, Principal Component Analysis (PCA) can be used. It is a multivariate technique that has its aim the explanation of relationships among several difficult-to-interpret, correlated variables in terms of a few conceptually meaningful components which are uncorrelated with each other and are capable of accounting for nearly all the variation present in the observed data. PCA therefore finds a linear transformation of original variables into a new set of variables called as principal components which are uncorrelated with each other; are capable of accounting for the variation of the obtained data and are derived in such a way that the first few of them can often provide enough information about the data and so the dimensionality of the problem can considerably be reduced. The variables with higher component loadings in a particular principal component are considered to be the important ones and it is assumed that the principal component is the representative of these variables; hence it is interpreted only in terms of these variables. This approach of interpretation of principal components is acceptable if principal components are extracted using the correlation matrix R. The variance–covariance matrix Σ is as well used to derive principal components; since the principal component technique is scale dependent, the principal component loadings with this approach will therefore be much influenced by the unit of the measurements of the variables under consideration, hence, the interpretation of principal components just based on the magnitude of the loading may become questionable and misleading. Ali et al. () suggested using the correlation between the original variables and the principal component for selection of representative variables in a particular principal component instead of using principal component loadings. PCA is extremely useful technique when interest lies in investigating the interrelationship within a set of variables; when the relationship of two sets of variables, within and among the sets is of interest, the PCA is not a valid technique. The agricultural researchers always encounter with such types of problems where the assessment of relationship among and within the
A
twosets is essential e.g. the interdependence of nutritional status and vegetative related characteristics with the crop yield related characteristics is pivotal. For such cases, 7Canonical Correlation Analysis (CCA) technique developed by Hotelling () is of great benefit. It has certain maximal properties similar to those of PCA and in a way is an extension of the multiple regression analysis. The object of this approach is to find the linear functions of the variables for each of the sets such that the correlation between these linear functions is as high as possible. After locating such a pair of linear functions which are maximally correlated with each other, we look for other pairs of linear functions which are maximally correlated subject to the restriction that the new pair of linear functions must be uncorrelated with all other previously located functions. For the purpose of interpretation of the results, it is proposed to use correlation between the canonical variates and the original variables instead of canonical weights as has been already proposed for interpretation of PCA results. Details of PCA and CCA may be found in Mardia et al. () and Jolliffe ().
About the Author Asghar Ali holding M.Sc and D.Phil in Statistics from Sussex University and Post Doctorate from University of Kent at Canterbury, UK is Professor of Statistics at Bahauddin Zakariya University (BZU) Multan, Pakistan. Since , he is serving BZU in various capacities: Presently, he is Chairman, Department of Statistics and also for a period of three years, he held the position of chairmanship, Department of Computer Science. Especially commendable have been his period as Principal, College of Agriculture, BZU. He has been publishing regularly related to data analysis in the field of agricultural sciences which is now being cited in related text books.
Cross References 7Agriculture, Statistics in 7Canonical Correlation Analysis 7Farmer Participatory Research Designs 7Multivariate Data Analysis: An Overview 7Multivariate Statistical Analysis 7Principal Component Analysis
References and Further Reading Ali A () Interpretation of multivariate data: Comparison of several methods of interpreting multivariate data from a series of nutritional experiments, University of Sussex, Unpublished PhD thesis Ali A, Clarke GM, Trustrum K () Principal component analysis applied to some data from fruit nutrition experiments. The Statistician :–
A
A
Analysis of Variance
Ali A, Clarke GM, Trustrum K () Log-linear response functions and their use to model data from plant nutrition experiments. J Sci Food & Agric :– Hotelling H () Relation between two sets of variates. Biometrika :– Jolliffe IT () Principal component analysis, nd edn. Springer, USA Mardia KV, Kent JT, Bibi JM () Multivariate analysis. Academic, London McCullagh P, Nelder JA () Generalized linear models, nd edn. Chapman and Hall, London Mead R, Pike DJ () A review of response surface methodology from a biometric viewpoint. Biometrics ():– Nelder JA () Inverse polynomials, a useful group of multifactor response functions. Biometrics :– Nelder JA () A reformation of linear models (with discussion), J R Stat Soc A:– Nelder JA, Wedderburn WM () Generalized linear models. J R Stat Soc (General) A():– Pike DJ () Inverse polynomials: A study of parameter estimation procedures and comparison of the performance of several experimental design criteria. University of Reading, Unpublished PhD thesis
Analysis of Variance Gudmund R. Iversen Professor Emeritus Swarthmore College, Swarthmore, PA, USA Analysis of variance is the name given to a collection of statistical methods originally used to analyze data obtained from experiments. The experiments make us of a quantitative dependent variable, also known as a metric variable or an interval or ratio variable, and one or more qualitative independent variables, also known as categorical or nominal variables. These analysis methods grew out of agricultural experiments in the beginning of the twentieth century, and the great English statistician Sir Ronald Fisher developed many of these methods. As an example, the dependent variable could be the yield in kilos of wheat from different plots of land and the independent variable could by types of fertilizers used on the plots of land.
Experimental Design The way an experiment is run affects the particular analysis of variance method used for the analysis of the data. Experiments are designed according to different plans, and the choice of the design of the experiment affects which analysis of variance method being used. Without going
into details about designs of experiments, an experiment could follow a factorial design, a randomized block design, a Latin square design, etc. There exist too many designs of experiments and accompanying analysis of variance methods for the analysis of the resulting data to cover all of them in this short presentation. But it is possible to present the underlying features of all analysis of variance methods.
Analysis of Variance and Multiple Regression But first it is worth noting that analysis of variance is closely related to regression analysis. Indeed, it is possible to see both analyses as special cases of the so-called general linear model. In particular, using 7dummy variables for the independent variables in analysis of variance, the analysis quickly turns into a regression analysis. The main difference is that when data are collected through a properly designed experiment, it is possible to conclude that there is a causal effect of the independent variable(s) on the dependent variable. When data are collected through observational, studies there may be a causal effect of the independent variable(s) or not.
Statistical Software Much of the early work on analysis of variance consisted of finding efficient ways of making the necessary computations with the use of simple calculators. With the introduction of modern statistical software for electronic computers, this line of work is now less important. Instead, statisticians have worked on showing the similarities of the computations needed for both analysis of variance and multiple regression, and the old distinction between the two approaches to data analysis is no longer of any importance. However, statistical software packages still make a distinction between the two, and the output from the two methods often look very different.
One-Way Analysis of Variance This name is given to the design where there is one independent nominal variable with several categories and a quantitative dependent variable with a unit of measurement and often a meaningful zero. An example of an experiment could be where students are randomly assigned to two different groups and the students in one group were taught using a new method of teaching while the students in the second group, as a control group, were taught using the old method. The random assignment to the different groups means that the effects of all other variables, for
Analysis of Variance
example gender, is canceled out, and any observed difference between the two groups is causally due to the teaching method being used. In this simple case, the statistical method is the same as the t-test for the difference between two groups. From a regression point of view we could use a dummy variable and assign the value to all the students in the control group and the value to all the students in the experiment group. In this case, the intercept of the regression line would equal the mean of the dependent variable for the control group and the slope of the line would equal the difference between the means of the two groups. Thus, the t-test for the null hypothesis that the population regression line has zero slop becomes the same as the t-test for the difference between the two means. The fundamental question in an analysis of variance is whether the population means for different groups are equal or not. But the methods for analysis of variance use variances to answer the question about means, thus the name analysis of variance. The analysis is based on identifying two factors that determine the values of the dependent variable. One such factor is the net effect of all factors except the independent variable, known as the residual variable, and the other factor is the independent variable.
The Residual Sum of Squares If the residual variable had no effect, then all the values of the dependent variable for the control group would be equal to each other, and all the values of the dependent variable for the experimental group would be equal to each other. The best estimates of these two values would be the mean of the dependent variable for the group. To the extent that the values within each group are not equal, is due to the residual variable. Thus, the effect of the residual variable for a single observation can be seen as the difference between the observed value and the group mean. For each observation we now have such a difference. One way to summarize the values of these differences for a group is to square each difference and add all these squares. We then have a sum of squares for each of the two groups, and by adding these two sums we have a measure of the overall effect of the residual variable. If the dependent variable is known as Y and yij is the ith observation in the jth group and yj is the mean in the jth group, then the residual sum of squares RSS can be written RSS = ∑ ∑(yij − yj ) . Note that there are many other ways we could combine these differences. For example, we could have taken the absolute value of each difference and added those
A
differences instead of using squares. Thus, the final conclusion from the analysis should include a statement that the conclusion is based on squares and not some other mathematical operation. Even though nobody does include such a warning, it should be made clear that the analysis is based on squares.
The Treatment Sum of Squares We also need a measure of how different the two groups are from each other. One way to do that is find how different the group means are from the overall mean. If the treatment variable has no effect, then the two group means would be equal and equal to the overall mean. One way to measure how different the group means are from the overall mean is to take each group mean and subtract the overall mean. By squaring each difference and weighing each square by the number of observations in the group nj , then the treatment sum of squares between the groups GSS can be written GSS = ∑ ∑ nj (yj − y) .
The F-Test
The residual sum of squares is also known as the within group sum of squares and the group sum of squares is sometimes known as the between group squares. The final step consists of making a comparison between the two sums of squares. If the residual sum of squares is large in comparison with the group sum of squares, then it seems that the difference between the group means is not statistically significant. For this comparison we take into how many groups we have, here and in general k groups, and how many observations n there are all together. A mathematical development shows that we should compute the ratio GSS/(k − ) F= RSS/(n − k) This is known as the F-ratio and is named in honor of Ronald Fisher. It gives rise to the F-distribution, and the distribution has been extensively tabulated. The two numbers (k−) and (n−k) are the so-called degrees of freedom, and they are used to take into account how many groups there are in the experiment and how many observations there are in the experiment. For example, for a % significance level with k = groups and n = observations, the critical value of F on and degrees of freedom equals .. Thus, for any observed value of F larger than ., we conclude that there is a statistically significant difference between the two groups. In this case, had we done a t-test
A
A
Analysis of Variance Model, Effects of Departures from Assumptions Underlying
for the difference between the special√case of two group means, the critical value of t becomes . = ..
Other Analyses It is possible to generalize to an experiment with more than just two groups. The null hypothesis of equal group means is tested the same way as with two groups, and the computations follow the same plan as above. With two or more independent variables the analysis becomes more extensive. We can still represent the independent variables by dummy variables and do a regression analysis. But that way it is easy to overlook the possible interaction effect of the two independent variables. This means we could have an effect of the independent variables together over and beyond their separate effects. Finally, in analysis of variance we distinguish between using all values of the independent variables (Model I) and only using a sample of possible values (Model II).
About the Author Dr. Gudmund Iversen is Professor Emeritus of Statistics at Swarthmore College, Swarthmore PA . He chaired the Department of Mathematics and Statistics at three different intervals and directed the College’s Center for Social and Policy Studies for several years. Prior to Swarthmore he taught statistics at the University of Michigan and also directed the Summer Training Program, Inter-university Consortium for Political and Social Research. He was a visiting lecturer in the American Statistical Association visiting lecture program. He has been a visiting professor at Zentrum für Umfragen, Methoden und Analysen (ZUMA), Mannheim, West Germany , at Department of Political Science, University of Oslo, Norway (–), at The Graduate School of Social Work and Social Research, Bryn Mawr College, spring , fall , and at School of Social Policy and Practice at the University of Pennsylvania . He was a member of the joint committee of Mathematical Association of America and American Statistical Association on the statistics curriculum (–). He was Associate book review editor, Journal of American Statistical Association (–), and Associate editor, Journal of Statistics Education (–). He has published articles and books on statistics and statistical education.
7Analysis of Variance Model, Effects of Departures from
Assumptions Underlying 7Data Analysis 7Experimental Design: An Introduction 7F Distribution 7Farmer Participatory Research Designs 7General Linear Models 7Graphical Analysis of Variance 7Multiple Comparison 7Multiple Comparisons Testing from a Bayesian Perspective 7Multivariate Analysis of Variance (MANOVA) 7Multivariate Data Analysis: An Overview 7Nonparametric Models for ANOVA and ANCOVA Designs 7Parametric Versus Nonparametric Tests 7Rank Transformations 7Statistical Analysis of Drug Release Data Within the Pharmaceutical Sciences 7Statistical Software: An Overview 7Statistics: An Overview 7Tests for Homogeneity of Variance
References and Further Reading Hinkelmann K, Kempthorne O () Design and analysis of experiments, I and II, nd edn. Wiley, New York Iversen GR, Norpoth H () Analysis of variance, nd edn. Sage Baverly Hills, CA
Analysis of Variance Model, Effects of Departures from Assumptions Underlying Hardeo Sahai , Mohammed I. Ageel , Anwer Khurshid Professor University of Puerto Rico, San Juan, Puerto Rico Founder and President of the Saudi Association of Statistical Sciences, Professor Jazan University, Jazan, Saudi Arabia Professor University of Karachi, Karachi, Pakistan
Introduction Cross References 7Agriculture, Statistics in 7Analysis of Covariance 7Analysis of Multivariate Agricultural Data
Every statistical model has its own underlying “assumptions” that must be verified to validate the results. In some situations, violations of these assumptions will not change substantive research conclusions, while in others,
Analysis of Variance Model, Effects of Departures from Assumptions Underlying
violation of assumptions can be critical to meaningful research. For a meaningful and conclusive data analysis by 7Analysis of Variance (ANOVA), the following assumptions are needed: (a) (b) (c)
Errors be normally distributed Errors have same variances (homogeneity of variances) Errors be independently distributed
However, the question arising is What would be the effects of any departure from the assumptions of the model on the inferences made? The answer is simple: It may either influence the probability of making Type I error (i.e., incorrectly rejecting null hypothesis) or a Type II error (i.e., failing to reject a null hypothesis when it is false). For a thorough discussion of the topic, the reader is referred to Scheffé (), Miller (), Snedecor and Cochran (), Sahai and Ojeda (), and Sahai and Ageel (). Some of the main findings are discussed in the following section.
Effects of Departures from Assumptions Departures from Normality For fixed effects model, due to the central limit theorem (see 7Central Limit Theorems) the lack of normality causes no problems in large samples, as long as the assumptions hold. In general, when true 7randomization occurs the violations of normality is acceptable. Also, heterogeneity of variances can result in nonnormality, so ensuring homogeneity of variances may also result in normality. Only highly skewed distributions would have a marked effect either on the level of significance or the power of the F test. However, it is worth mentioning that kurtosis of the error distribution (either more or less peaked than a normal distribution) is more important than skewness of the distribution in terms of the effects on inferences. Both analytical results (see, e.g., Scheffé :–) and the empirical studies by Pearson (), Geary (), Gayen (), Box and Anderson (), Boneau (, ), Srivastava (), Bradley (), Tiku (, ), and Donaldson () attest to the conclusion that lack of normality would have little effect of F test either in terms of level of significance or power. Hence, the F test is generally robust against departures from normality (in skewness and/or kurtosis) if sample sizes are large or even if moderately large. For instance, the specified level of significance might be ., whereas the actual level for a nonnormal error distribution might vary from . to . depending on the sample size and the magnitude of the kurtosis. Generally, the actual level of significance in the presence of positive kurtosis (platykurtic) is slightly higher than
A
the specified one and the real power of the test for positive kurtosis is slightly higher than the normal one. If the underlying population has negative kurtosis (leptokurtic), the actual power of the test will be slightly lower than the normal one (Glass et al. ). Single interval estimates of the factor level means and contrasts and some of the multiple comparison methods are also not much affected by the lack of normality provided the sample sizes are not too small. The robustness of multiple comparison tests in general has not been as thoroughly studied. Among few studies in this area is that of Brown (). Some other studies have investigated the robustness of several multiple comparison procedures, including Tukey and Scheffé, for exponential and chi-square distributions and found little effect on both α and power (see, e.g., Petrinovich and Hardyck ; Keselman and Rogan ). Dunnett () reported that Tukey is conservative both with respect to α and power for long-tailed distributions and to 7outliers. Similarly, Ringland () found that Scheffé was conservative for distributions with influence to outliers. Lange and Ryan () gave several examples that show that nonnormality of random effects is, indeed, encountered in practice. For random effects model, the lack of normality has more serious implications than fixed effects model. The estimates of the variance components are still unbiased, but the actual confidence coefficients for interval estimates of σe , σα , σα /σe may be substantially different from the specified one (Singhal and Sahai ). Moreover, when testing the null hypothesis, if the variance of a random effect is some specified value different from zero, the test is not robust to the assumption of normality. For some numerical results of this, the reader is referred to Arvesen and Schmitz () and Arvesen and Layard (). However, if one is concerned only with a test of hypothesis σα = , then slight departures from normality have only minor consequences for the conclusions reached when the sample size is reasonably large (see, e.g., Tan and Wong ; Singhal et al. ).
Departures from Equal Variances Both the analytical derivations by Box () and the empirical studies indicate that if the variances are unequal, the F test for the equality of means under fixed effects model is only slightly affected provided there is no remarkable difference in sample sizes and the parent populations are approximately normally distributed. When the variances are unequal, an approximate test similar to the approximate t test when two group variances are unequal may be used (Welch ). Generally, unequal error variances increase the actual level of significance slightly
A
A
Analysis of Variance Model, Effects of Departures from Assumptions Underlying
higher than the specified level and result in a slight elevation of the power function to a degree related to the magnitude of differences among variances. If larger variances are associated with larger sample sizes, the level of significance will be slightly less than the nominal value, and if they are associated with smaller sample sizes, the level of significance will be slightly greater than the nominal value (Horsnell ; Kohr and Games ). Similarly, the Scheffé’s multiple comparison procedure based on the 7F distribution is not affected to any appreciable degree by unequal variances if the sample sizes are approximately equal. Thus, the F test and the related analyses are robust against unequal variances if the sample sizes are nearly equal. On the other hand, when different number of cases appear in various samples, violation of the assumption of homogeneous variances can have serious effects in the validity of the final inference (see, e.g., Scheffé ; Welch ; James ; Box ; Brown and Forsythe ; Bishop and Dudewicz ; Tan and Tabatabai ). Krutchkoff () made an extensive simulation study to determine the size and power of several analysis of variance procedures, including the F test, Kruskal–Wallis test, and a new procedure called the K test. It was found that both the F test and the Kruskal–Wallis test are highly sensitive whereas the K test is relatively insensitive to the heterogeneity of variances. Kruskal–Wallis test, however, is not as sensitive to the unequal error variances as the F test and was found to be more robust to nonnormality (when the error variances are equal) than either the F test or the K test. Thus, whenever possible, the experimenter should try to achieve the same number of cases in each factor level unless the assumption of equal population variances can reasonably be assured in the experimental context. The use of equal sample sizes for all factor levels not only tends to minimize the effects of unequal variances using the F test, but also simplifies the computational procedure. For random effects model, however, the lack of homoscedasticity or unequal error variances can have serious effects on inferences about the variance components, even when all factor levels contain equal sample sizes. Boneau () has shown that when variances are different in the various groups and sample sizes are small and different, ANOVA can produce highly misleading results.
and random effects models. If this assumption is not met, the F ratio may be strongly affected severely in serious errors in inferences (Scheffé ). The direction of the effect depends on the nature of the dependence of the error terms. In most cases encountered in practice, the dependence tends to make the value of the ratio too large and consequently the significance level will be smaller than it should be (although the opposite can also be true). Since the remedy of violation of this assumption is often difficult, every possible effort should be made to obtain independent random samples. The use of randomization in various stages of the study can be most important protection against independence of error terms. In general, great care should be taken to ensure that the data are based on independent observations, both between and within groups, i.e., each observation is in no way related to any of the other observations. Although, dependency among the error terms creates a special problem in any analysis of variance, it is not necessary that the observations themselves must be completely independent for applying the random effects model. In summary, ANOVA is very robust to violations of the assumptions, as long as only one assumption is violated. If two or more assumptions are severely violated the results are not to be trusted. Further if the data are: (a)
(b)
Not normally distributed, but satisfies the homogeneity of variance and independent assumptions, the findings may still be valid. Normally distributed and are independent samples, but does not satisfy the homogeneity of variance assumption, the findings may still be valid.
The above review and discussion are restricted to the oneway analysis of variance. A similar finding for two-way classification without and with interaction can be found in Sahai and Ageel ().
Tests for Departures from Assumptions As we have seen in the preceding section, the analysis of variance procedure is robust and can tolerate certain departures from the specified assumptions. It is, nevertheless, recommended that whenever a departure is suspected it should be checked out. In this section, we shall briefly state the tests for normality and homoscedasticity.
Tests for Normality Departures from Independence of Error Terms Lack of independence can result from biased measurements or possibly from a poor allocation of treatments to experimental units. Nonindependence of the error terms can have important effects on inferences for both fixed
A relatively simple technique to determine the appropriateness of the assumption of normality is to graph the data points on a normal probability paper. If a straight line can be drawn through the plotted points, the assumption of normality is considered to be reasonable. Some formal tests for normality are the chi-square goodness of fit test, and the
Analysis of Variance Model, Effects of Departures from Assumptions Underlying
tests for skewness and kurtosis that are often used as supplements to the chi-square test (see 7Chi-Square Tests). For a detailed discussion of these tests refer to Sahai and Ageel (). The tests mentioned above are some of the classical tests of normality. Over the years, a large number of other techniques have been developed for testing for departures from normality. Some powerful omnibus tests proposed for the problem are Shapiro–Wilk’s test (Shapiro and Wilk ), Shapiro–Francia’s test (Shapiro and Francia ), and D’Agostino’s test (D’Agostino ). For a discussion of tests especially designed for detecting outliers see Barnett and Lewis (). Robust estimation procedures have also been employed in detecting extreme observations. The procedures give less weight to data values that are extreme in comparison to the rest of the data. Robust estimation techniques have been reviewed by Hampel et al. ().
Tests for Homoscedasticity If there are just two populations, the equality of two population variances can be tested by using the usual F test. However, more than two population, rather than making all pairwise F tests, we want a single test that can be used to verify the assumption of equality of population variances. There are several tests available for this purpose. The three most commonly used tests are the Bartlett’s, Hartley’s, and Cochran’s tests. The 7Bartlett’s test (Bartlett a, b) compares the weighted arithmetic and geometric means of the sample variances. The Hartley’s test (Hartley ) compares the ratio of the largest to the smallest variance. The Cochran’s test (Cochran ) compares the largest sample variance with the average of all the sample variances. For a full description of these procedures and illustration of their applications with examples see Sahai and Ageel (). These tests, however, have lower power than is desired for most applications and are adversely affected by nonnormality. Detailed practical comments on Bartlett’s, Hartley’s, and Cochran’s tests are also given by Sahai and Ageel. In recent years, there have appeared a number of tests in the literature that are less sensitive to normality in the data and are found to have a good power for a variety of population distributions see Levene (). Following Levene (), a number of other robust procedures have been proposed, which are essentially based on techniques of applying ANOVA to transformed scores. For example, Brown and Forsythe (a) proposed applying an ANOVA to the absolute deviations from the mean. A somewhat different approach known as 7jackknife was proposed by Miller () where the original scores in each group are replaced by the contribution of that observation to the group variance. O’Brien (, ) proposed a
A
procedure, which is a blend of Levene’s squared deviation scores and the jackknife. In recent years, there have been a number of studies investigating the robustness of these procedures. For a further discussion and details, the reader is referred to Conover et al. (), Olejnik and Algina (), and Ramsey ().
Corrections for Departures from Assumptions of the Model Departure from independence could arise in an experiment in which experimental units or plots are laid out in a field so that adjacent plots give similar yields. Lack of independence can also result from correlation in time rather than in space. If the data set in a given problem violates the assumptions of the analysis of variance model, a choice of possible corrective measures is available. One approach is to modify the model. However, this approach has the disadvantage that more often than not the modified model involves fairly complex analysis. Another approach may be to consider using some nonparametric tests. A third approach to be discussed in this section is to use transformations on the data. Sometimes it is possible to make an algebraic transformation of the data to make them appear more nearly normally distributed, or to make the variances of the error terms constant. Conclusions derived from the statistical analyses performed on the transformed data are also applicable to the original data. In this section, we briefly discuss some commonly used transformations to correct for the lack of normality and homoscedasticity. An extremely thorough and detailed monograph on transformation methodology has been prepared by Thöni (). An excellent and thorough introduction and a bibliography of the topic can be found in a review paper by Hoyle (). For a more recent bibliography of articles on transformations see Draper and Smith (:–).
Transformations to Correct Lack of Normality Some transformations to correct for the departures from normality are logarithmic transformation, square-root transformation, and arcsine transformation.
Transformations to Correct Lack of Homoscedasticity There are several types of data in which the variances of the error terms are not constant. If there is evidence of some systematic relationship between treatment mean and variance, homogeneity of the error variance may be achieved through an appropriate transformation of the data. Bartlett () has given a formula for deriving such transformations provided the relationship between µ i and σe is known. In many cases where the nature of the relationship is not clear, the experimenter can, through trial
A
A
Analysis of Variance Model, Effects of Departures from Assumptions Underlying
and error, find a transformation that will stabilize the variance. We give some commonly employed transformations to stabilize the variance. These are logarithmic transformation, square-root transformation, reciprocal transformation, arcsine transformation, and power transformation. For a detailed discussion of these transformations and their applicability refer to Sahai and Ageel (). These are some of the more commonly used transformations. Still other transformations can be found applicable for various other relationships between the means and the variances. Further, the transformations to stabilize the variance also often make the population distribution nearly normal. For equal sample sizes, however, these transformations may not usually be necessary. Moreover, the use of such transformations may often result in different group means. It is possible that the means of the original scores are equal but the means of the transformed scores are not, and vice versa. Further, the means of transformed scores are often changed in ways that are not intuitively meaningful or are difficult to interpret.
Acknowledgment We are grateful to Professor Miodrag Lovric, editor-in chief, for his nice editorial efforts and for his valuable suggestions and comments that led to the improvement in the contribution.
About the Authors Dr. Hardeo Sahai held Professorial and visiting Professorial positions at the University of Puerto Rico, Mayaguez and San Juan, University of Ceara, Brazil, University of Granada, Spain, University of Veracruzana, Mexico, University of Nacional de Colombia, University of Nacional de Trujillo, Peru. He has received the University of Kentuky Outstanding Alumnus award, Medal of honor University of Granada (Spain), Plaque of honor University of Nacional de Trujillo (Peru). He has published over papers in the statistical (bio)medical and epidemiological literature and is coauthor of the several books which include: Statistics in Epidemiology: Methods, Techniques and Applications (with Anwer Khurshid, CRC Press, Boca Raton, Florida, ), and Analysis of Variance for Random Models, Vol. : Balanced Data and Vol. : Unbalanced Data (with Mario M. Ojeda, Birkhäuser ). Professor Sahai is a Fellow of the American Statistical Association, Royal Statistical Society and Elected Member of the International Statistical Institute. Dr. Mohammed Ibrahim Ali Ageel was a Full Professor and Chairman of Mathematics, Department of Mathematics, King Saud University (KSU), Saudi Arabia, and later the Chairman of Mathematics Department, King Khalid
University (KKU), Saudi Arabia. He was the Dean of Graduate School, King Khalid University, Saudi Arabia. Professor Ageel was also a Full Professor of Mathematics and Dean of Engineering, Najran University (NU), Saudi Arabia. Ageel is currently a Full Professor of Mathematics, Jazan University, Jazan, Saudi Arabia. He is a Founder and President of Saudi Association of Statistical Sciences (SASS). He is an Elected member of the International Statistical Institute, and was elected as a Fellow of the Royal Statistical Society. He has published more than research papers and articles in both theoretical and applied areas. Professor Ageel is a coauthor of the book The Analysis of Variance: Fixed, Random and Mixed Models (with Hardeo Sahai, Birkhäuser, Boston ). Anwer Khurshid is a Professor at the Department of Statistics, University of Karachi, Pakistan. During – he had a faculty position at the Sultan Qaboos University, Oman and University of Nizwa, Oman. He is author or coauthor of more than papers and two books (both with Professor Hardeo Sahai). In recognition of his teaching and research contributions Professor Khurshid was awarded a certificate of appreciation in by the Chancellor, University of Karachi, Pakistan.
Cross References 7Analysis of Variance 7Bartlett’s Test 7Heteroscedasticity 7Normality Tests 7Robust Inference 7Robust Statistics 7Tests for Homogeneity of Variance
References and Further Reading Arvesen JN, Layard MWJ () Asymptotically robust tests in unbalanced variance component models. Ann Stat :– Arvesen JN, Schmitz TH () Robust procedures for variance component problems using the jackknife. Biometrics :– Barnett VD, Lewis T () Outliers in statistical data, rd edn. Wiley, New York Bartlett MS () The square root transformation in the analysis of variance. J R Stat Soc :– Bartlett MS (a) Properties of sufficiency and statistical tests. Proc R Soc Lond Ser A :– Bartlett MS (b) Some examples of statistical methods of research in agriculture and applied biology. J R Stat Soc Suppl : – Boneau CA () The effects of violation of assumptions underlying the t-test. Psychol Bull :– Boneau CA () A comparison of the power of the U and t tests. Psychol Rev :–
Analysis of Variance Model, Effects of Departures from Assumptions Underlying
Box GEP () Some theorems on quadratic forms applied in the study of analysis of variance problems: I. Effect of inequality of variance in the one-way classification. Ann Math Stat :– Box GEP, Anderson SL () Permutation theory in the derivation of robust criteria and the study of departures from assumption. J R Stat Soc Ser B :– Bradley JV () Studies in research methodology, VI. The central limit effect for a variety of populations and the robustness of Z, t, and F. Technical report no. AMRL--. Aerospace Medical Research Laboratories, Wright-Patterson Air Force Base, Dayton, Ohio Brown RA () Robustness of the studentized range statistic. Biometrika :– Brown MB, Forsythe AB (a) Robust tests for the equality of variances. J Am Stat Assoc :– Brown MB, Forsythe AB (b) The small size sample behavior of some statistics which test the equality of several means. Technometrics :– Brown MB, Forsythe AB (c) The ANOVA and multiple comparisons for data with heterogeneous variances. Biometrics :– Cochran WG () The distribution of the largest of a set of estimated variances as a fraction of their total. Ann Eugen :– Conover WJ, Johnson ME, Johnson MM () A comparative study of tests for homogeneity of variances with applications to outer continental shelf bidding data. Technometrics :– (Corrigendum Technometrics :) D’Agostino RB () An omnibus test of normality for moderate and large size samples. Biometrika :– Donaldson TS () Robustness of the F-test to errors of both kinds and the correlation between the numerator and denominator of the F-ratio. J Am Stat Assoc :– Draper NR, Smith H () Applied regression analysis, nd edn. Wiley, New York Dunnett CW () Robust multiple comparisons. Commun Stat Part A: Theory Methods :– Gayen AK () The distribution of the variance ratio in random samples of any size drawn from non-normal universes. Biometrika :– Geary RC () Testing for normality. Biometrika :– Glass GV, Peckham PD, Sanders JR () Consequences of failure to meet assumptions underlying the fixed effects analysis of variance and covariance. Rev Educ Res :– Hampel FR, Rochetti EM, Rousseeuw PJ, Stahel WA () Robust statistics: the approach based on influence functions. Wiley, New York Horsnell G () The effect of unequal group variances on the F-test for the homogeneity of group means. Biometrika : – Hoyle MH () Transformations – an introduction and a bibliography. Int Stat Rev :– James GS () The comparison of several groups of observations when the ratio of population variances are unknown. Biometrika :– Keselman HJ, Rogan JC () A comparison of modified-Tukey and Scheffé methods of multiple comparisons for pairwise contrasts. J Am Stat Assoc :– Kohr RL, Games PA () Robustness of the analysis of variance, the Welch procedure, and a Box procedure to heterogeneous variances. J Exp Educ :–
A
Krutchkoff RG () One way fixed effects analysis of variance when the error variances may be unequal. J Stat Comput Simul :– Lange N, Ryan L () Assessing normality in random effects models. Ann Stat :– Levene H () Robust tests for equality of variances. In: Olkin I, Ghurye SG, Hoeffding W, Madow WG, Mann HB (eds) Contributions to probability and statistics. Stanford University Press, Stanford, pp – Miller RG Jr () Jackknifing variances. Ann Math Stat : – O’Brien RG () An improved ANOVA method for robust tests of additive models for variances. J Am Stat Assoc :– O’Brien RG () A simple test for variance effects in experimental designs. Psychol Bull :– Olejnik SF, Algina J () Type I error rates and power estimates of selected parametric and nonparametric tests of scales. J Educ Stat :– Pearson ES () The analysis of variance in cases of non-normal variation. Biometrika :– Petrinovich LF, Hardyck CD () Error rates for multiple comparison methods. Psychol Bull :– Ramsey PH () Testing variances in psychological and educational research. J Educ Stat :– Ringland JT () Robust multiple comparisons. J Am Stat Assoc :– Sahai H, Ageel MI () The analysis of variance: fixed, random and mixed models. Birkhäuser/Springer, Boston Sahai H, Ojeda M () Analysis of variance for random models: unbalanced data. Birkhauser, USA Scheffé H () The analysis of variance. Wiley, New York Shapiro SS, Francia RS () An approximate analysis of variance test for normality. J Am Stat Assoc :– Shapiro SS, Wilk MB () An analysis of variance test for normality (complete samples). Biometrika :– Singhal RA, Sahai H () Sampling distribution of the ANOVA estimator of between variance component in samples from a non-normal universe. J Stat Comput Simul :– Singhal RA, Tiwari CB, Sahai H () A selected and annotated bibliography on the robustness studies to nonnormality in variance component models. J Jpn Stat Soc : – Snedecor GW, Cochran WG () Statistical methods, th edn. Iowa State University Press, Ames Srivastava ABL () Effects of non-normality on the power of the analysis of variance test. Biometrika :– Tan WY, Tabatabai MA () Some Monte Carlo studies on the comparison of several means under heteroscedasticity and robustness with respect to departure from normality. Biom J :– Tan WY, Wong SP () On approximating the null and nonnull distributions of the F ratio in unbalanced random effects models from non-normal universes. J Am Stat Assoc : – Thöni H () Transformation of variables used in the analysis of experimental and observational data: a review. Technical report no. . Statistical Laboratory, Iowa State University, Ames Tiku ML () Approximating the general non-normal varianceratio sampling distributions. Biometrika :– Tiku ML () Power function of F-test under non-normal situations. J Am Stat Assoc :– Welch BL () On linear combinations of several variances. J Am Stat Assoc :–
A
A
Anderson–Darling Tests of Goodness-of-Fit
If ψ(z) = , the statistic Wn is the Cramér-von Mises statistic, denoted by nω . Anderson and Darling () gave a table of the limiting distribution of nω as n → ∞. For example, the % significance point is . and the % significance point is ..
Anderson–Darling Tests of Goodness-of-Fit Theodore W. Anderson Professor of Statistics and Economics, Emeritus Stanford University, Stanford, CA, USA
The Anderson–Darling Statistic
Introduction A “goodness-of-fit” test is a procedure for determining whether a sample of n observations, x , . . . , xn , can be considered as a sample from a given specified distribution. For example, the distribution might be a normal distribution with mean and variance . More generally, the specified distribution is defined as
For a given x and hypothetical distribution F (⋅), the random variable n Fn (x) has a 7binomial distribution with probability F (x). The expected value of n Fn (x) is nF (x) and the variance is n F (x) [ − F (x)]. The definition of the goodness-of-fit statistic () permits the choice of weight function ψ(⋅). In particular the investigator may want to emphasize the tails of the presumed distribution F (x). In that case the choice is
x
F(x) = ∫ f (y)dy,
−∞ < x < ∞ ,
()
where f (y) is a specified density. This density might be suggested by a theory, or it might be determined by a previous study of similar data. When X is a random variable with distribution function F(x) = Pr {X ≤ x} , then U = F(X) is a random variable with distribution function Pr {U ≤ u} = Pr {F(X) ≤ u} = u,
≤ u ≤ .
()
The model specifies u = F(x ), . . . , un = F(xn ) as a sample from the distribution (), that is, the standard uniform distribution (see 7Uniform Distribution in Statistics) on the unit interval [, ] written U(, ). A test of the hypothesis that x , . . . , xn is a sample from a specified distribution, say F (x), is equivalent to a test that u = F (x ), . . . , un = F (xn ) is a sample from U(, ). Define the empirical distribution function as k , n
()
−∞ < x < ∞,
if k of (x , . . . , xn ) are ≤ x. A goodness-of-fit test is a comparison of Fn (x) with F (x). The hypothesis H : F(x) = F (x), −∞ < x < ∞, is rejected if Fn (x) is very different from F (x). “Very different” is defined here as Wn
∞
= n ∫ [Fn (x) − F (x)] ψ [F (x)] dF (x) −∞ ∞
. u( − u)
()
Then for a specified x
−∞
Fn (x) =
ψ(u) =
√
Fn (x) − F (x) n√ F (x) [ − F (x)]
has mean and variance when the null hypothesis is true. The Anderson–Darling statistic is
−∞
being large; here () holds and ψ(z) is a weight function such that ψ(z) ≥ , and f (x) is the density of F (x).
∞
[Fn (x) − F (x)] d F (x). An = n ∫ F (x) [ − F (x)]
()
−∞
It was shown in Anderson and Darling () that () can be written as n An = −n− ∑ (j − ) [log u(j) + log ( − u(n−j+) )] () n j= where u(j) = F (x(j) ) and x() < x() < . . . < x(n) is the ordered sample. Anderson and Darling found the limiting distribution of An [for weight function ()]. In the next section the development of this distribution is outlined. The % significance point of the limiting distribution is . and the % point is .. The mean of this limiting distribution is and the variance is (π − )/ ∼ ..
Outline of Derivation
Let u = F (x), ui = F () (xi ), i = , . . . , n, and u(i) = F () (x(i) ), i = , . . . , n. Let Gn (u) be the empirical distribution function of u , . . . , un ; that is
= n ∫ [Fn (x) − F (x)] ψ [F (x)] f (x)dx ()
()
Gn (u) =
k , n
≤ u ≤ ,
()
if k of u , . . . , un are ≤ u. Thus Gn [F (x)] = Fn (x),
()
A
Anderson–Darling Tests of Goodness-of-Fit
and
where X , X , . . ., are independent N(, ) variables. Then
Wn
()
= n ∫ [Gn (u) − u] ψ(u)du,
when the null hypothesis F(x) = F () (x) is true. For every u ( ≤ u ≤ ) √ Yn (u) = n [Gn (u) − u] () is a random variable, and the set of these may be considered as a stochastic process with parameter u. Thus
∫ z (u)du = ∑ λ Xj , j j=
E [Yn (u)] = ,
EYn (u)Yn (v) = min(u, v) − uv. ()
The limiting process of {Yn (u)} is a Gaussian process y(u), ≤ u ≤ , and Ey(u) = and Ey(u)y(v) = min (u, v) − uv. Let ⎧ ⎫ ⎪ ⎪ ⎪ ⎪ a(z) = Pr ⎨∫ y (u)ψ(u)du ≤ z⎬ . ⎪ ⎪ ⎪ ⎪ ⎩ ⎭
()
Then An (z) → a(z), ≤ z < ∞. The mathematical problem for the Anderson–Darling statistic is to find the distribution function a(z) when ψ(u) = /u( − u). We briefly sketch the procedure to find the distribution
of ∫ z (u)du, where z(u) is a Gaussian stochastic process
with Ez(u) = and Ez(u)z(v) = k(u, v). When the kernel is continuous and square integrable (as is the case here), it can be written as ∞
k(u, v) = ∑ fj (u) fj (v), λ j= j
()
where λ j is an eigenvalue and fj (u) is the corresponding normalized eigenfunction of the integral equation
()
λ ∫ k(u, v)f (u)du = f (v),
∫ fj (u)du = ,
∫ fi (u)fj (u)du = ,
i =/ j.
()
Then the process can be written ∞ z(u) = ∑ √ Xj fj (u), λj j=
()
()
with characteristic function ⎤ ∞ ⎡ ⎥ ⎢ ⎢ E exp ⎢it ∫ z (u)du⎥ ⎥ = ∏ E (exp itXj /λ j ) ⎥ j= ⎢ ⎦ ⎣ ∞
= ∏ ( − it/λ j )
⎫ ⎧ ⎪ ⎪ ⎪ ⎪ Pr {Wn ≤ z} = Pr ⎨∫ Yn (u)ψ(u)du ≤ z⎬ = An (z), ⎪ ⎪ ⎪ ⎪ ⎩ ⎭ () say. For a fixed set u , . . . , uk the k-variate distribution of Yn (u ) , . . . , Yn (uk ) approaches a multivariate normal distribution (see 7Multivariate Normal Distributions) as n → ∞ with mean and covariance function
∞
−
.
()
j=
√ The process Yn∗ (u) = ψ(u) Yn (u) has covariance function √ √ k(u, v) = ψ(u) ψ(v) [min (u, v) − uv] ; () as n → ∞, the process Yn∗ (u) approaches y∗ (u) = √ ψ(u) y(u) with covariance (). The characteristic function of the limiting distribution of nω is ¿ √ Á it Á À √ () sin it for ψ(u) = , and that of the limiting distribution of An is ¿ Á Á Á À
−πit . π√ cos ( + it)
()
for ψ(u) = /u( − u). The integral equation () can be transformed to a differential equation h′′ (t) + λψ(t) h((t) = .
()
Anderson–Darling Tests with Unknown Parameters When parameters in the tested distribution are not known, but are estimated efficiently, the covariance () is modified, and the subsequent limiting distribution theory for both nω and An follows the same lines as above, with this new covariance. If the parameters are location and/or scale, the limiting distributions do not depend on the true parameter values, but depend on the class of tested distributions. If the parameters are shape parameters, the limiting distribution depends on shape. Limiting distributions have been evaluated and percentage points given for a number of different tested distributions; see Stephens (, ). Tests for three parameter Weibull, and von Mises have been given by Lockhart and Stephens (, ).
A
A
Approximations for Densities of Sufficient Estimators
The percentage points for these tests are much smaller than those given above for the case when parameters are known.
Acknowledgments
Lockhart RA, Stephens MA () Estimation and tests of fit for the three-parameter Weibull distribution. J R Stat Soc B :– Stephens MA () Asymptotic results for goodness-of-fit statistics with unknown parameters. Ann Stat :– Stephens MA () In: D’Agostino R, Stephens MA (eds) Goodnessof-fit techniques, chap. . Marcel Dekker, New York
The assistance of Michael A. Stephens is gratefully acknowledged.
About the Author Professor Anderson was born June , in Minneapolis, Minnesota. He is Past President, Institute of Mathematical Statistics (), Vice President, American Statistical Association (–), Fellow of the American Academy of Arts and Sciences (elected ), Member of the National Academy of Sciences (elected ). Professor Anderson has been awarded the R. A. Fisher Award of Committee of Presidents of Statistical Societies () and Samuel S. Wilks Memorial Medal, American Statistical Association (). He holds four honorary doctorates. Professor Anderson has published over articles in statistical, econometric, and mathematical journals, and seven books, including the internationally recognized text An Introduction to Multivariate Statistical Analysis (, Wiley; rd edition ). The th International Workshop in Matrices and Statistics was held in Tomar (Portugal July ), in honour of Professor Theodore Wilbur Anderson th birthday. The Collected Papers of T. W. Anderson: – (edited by George P. H. Styan) were published by Wiley in , comprising papers and commentaries, in a -volume set covering , pages.
Cross References 7Cramér-Von Mises Statistics for Discrete Distributions 7Jarque-Bera Test 7Kolmogorov-Smirnov Test 7Normality Tests 7Normality Tests: Power Comparison 7Omnibus Test for Departures from Normality 7Tests of Fit Based on The Empirical Distribution
Function
References and Further Reading Anderson TW, Darling DA () Asymptotic theory of certain ‘goodness-of-fit’ criteria based on stochastic processes. Ann Math Stat :– Anderson TW, Darling DA () A test of goodness-of-fit. J Am Stat Assoc :– Lockhart RA, Stephens MA () Tests of fit for the von-Mises distribution. Biometrika :–
Approximations for Densities of Sufficient Estimators Juan Carlos Abril President of the Argentinean Statistical Society, Professor Universidad Nacional de Tucumán and Consejo Nacional de Investigaciones Científicas y Técnicas, San Miguel de Tucumán, Argentina
Introduction Durbin (a) proposed a simple method for obtaining asymptotic expansions for the densities of sufficient estimators. The expansion is a series which is effectively in powers of n− , where n is the sample size, as compare with the 7Edgeworth expansion which is in powers of n−/ . The basic approximation is just the first term of this series. This has an error of order n− compare to the error of n−/ in the usual asymptotic normal approximation (see 7Asymptotic Normality). The order of magnitude of the error can generally be reduced to order n−/ by renormalization. Suppose that the real m-dimensional random vector Sn = (Sn , Sn , . . . , Smn )′ has a density with respect to Lebesgue measure which depends on integer n > N for some positive N and on θ ∈ Θ, where Θ is a subset of Rq for q an arbitrary positive integer. Let Dn (θ) = n− E {Sn − E(Sn )} {Sn − E(Sn )}′
()
which we assume is finite and positive-definite for all n and θ, and which we assume converges to a finite positivedefinite matrix D(θ ) as n → ∞ and θ → θ , where θ is a particular value of θ, usually the true value. ′ Let ϕ n (z, θ) = E(eiz Sn ) be the characteristic function of Sn where z = (z , z , . . . , zm )′ . Whenever the appropriate derivatives exists, let ∂ j log ϕ n (̃ z, θ) ∂zj
Approximations for Densities of Sufficient Estimators
denote the set of jth order derivatives ∂ j log ϕ n (z, θ)/ j j ∂z ⋯∂zmm for all integers j , j ,…, jm ≥ satisfying ∑k jk = j, evaluated at z = ̃ z. The jth cumulant κ nj (θ) of Sn , where it exists, satisfies the relation i j κ nj (θ) =
∂ j log ϕ n (, θ) . ∂zj
()
In what follows, let θ and θ be points in an open subset Θ of Θ, and let r be a specified integer. We use the word limit in the sense of joint limit, and introduce three assumptions. Assumption . If n is large enough ∣ϕ n (z, θ)∣ is integrable over Rm , and if δ is an arbitrary positive constant the limit of r
n
−
∫
√ ∣ϕ n (z/ n, θ)∣ dz,
B δ √n
as n → ∞ and θ → θ is zero, where Bδ √n is the region √ ∥z∥ ≥ δ n and ∥⋅∥denotes the Euclidean norm. Assumption . The rth derivative ∂ r log ϕ n (z, θ)/∂zr exists for z in a neighborhood of the origin and the limit of ∂ r log ϕ n (z, θ) n ∂zr −
as n → ∞, θ → θ and ∥z∥ → exists. Assumption . The cumulant κ nj (θ) = O(n) uniformly for θ in a neighborhood of θ for j = , . . . , r − . Now we present the Edgeworth expansion and the corresponding approximation to the density hn (x, θ) of Xn = n−/ E {Sn − E(Sn )}. Suppose that there is an integer r ≥ such that Assumptions – hold. Then there is a neighborhood ∥θ − θ ∥ < δ of θ such that hn (x, θ) − ̂ hn (x, θ) = o {n−(r/)+ }
()
uniformly in x and in θ for ∥θ − θ ∥ < δ , where ∣Dn (θ)∣−/ ̂ exp {− x′ D− hn (x, θ) = n (θ)x} (π)m/ ⎧ ⎫ r ⎪ ⎪ ⎪ ⎪ ⎨ + ∑ n−(j/)+ Pnj (x, θ)⎬ , () ⎪ ⎪ ⎪ ⎪ ⎩ j= ⎭ and where Pnj (x, θ) is a generalized Edgeworth polynomial of order j the definition of which is given in Durbin (a). The practical construction of Pnj (x, θ) is described by Chambers (, pp. –).
A
Approximations to the Densities of Sufficient Estimators
A
Suppose that y = (y , . . . , yn )′ is a matrix of observations of n continuos or discrete random ℓ× vectors, not necessarily independent or identically distributed, with density f (y, θ) = G(t, θ)H(y),
y ∈ Y, θ ∈ Θ,
()
where t = (t , . . . , tm )′ is the value computed from y of an estimator Tn of the m-dimensional parameter θ, where Y and Θ are observation and parameter spaces and where Y and H do not depend upon θ. We assume that f (y, θ) > for all y ∈ Y and θ ∈ Θ. By the factorization theorem Tn is sufficient for θ. Suppose that a transformation y , . . . , yn → t , . . . , tm , um+ , . . . , unℓ exists such that on substituting for y on the right-hand side of () and integrating or summing out um+ , . . . , unℓ we obtain the marginal density g(t, θ) of Tn in the form g(t, θ) = G(t, θ)H (t) where H does not depend upon θ. We therefore have f (y, θ) = g(t, θ)h(y),
()
where h(y) = H(y)/H (t). The derivation of () from () has been given in this form to avoid measure-theoretic complications. Suppose further that although functions G(t, θ) satisfying () can be deduced immediately from inspection of f (y, θ), the density g(t, θ) is unknown and we want to obtain an approximation to it for a particular value θ of θ. Since () holds for all θ ∈ Θ we have f (y, θ ) = g(t, θ )h(y).
()
On dividing () by () the unknown factor h(y) is eliminated and we obtain immediately g(t, θ ) =
f (y, θ ) g(t, θ). f (y, θ)
()
If we substitute t for θ in (), as is legitimate since we have assumed that t ∈ Θ, we obtain g(t, θ ) =
f (y, θ ) g(t, t). f (y, t)
()
The basic idea is to obtain an approximation ̂ g (t, θ ) for g(t, θ ) by substituting a series approximation ̂ g (t, t) for g(t, t) in (), giving ̂ g (t, θ ) =
f (y, θ ) ̂ g (t, t). f (y, t)
()
In effect, the method rescales the approximation ̂ g (t, t) at θ = t by the likelihood ratio f (y, θ )/f (y, t).
A
Approximations for Densities of Sufficient Estimators
A second idea is to substitute an Edgeworth series approximation ̂ g (t, ̃ θ) for g(t, θ) in (), where ̃ θ is chosen as the value of θ for which the mean of the distribution of Tn coincides with t. The reason for using this indirect approach instead of approximating g(t, θ) directly is that a straightforward Edgeworth approximation of g(t, θ), would normally be in powers of n−/ whereas an Edgeworth approximation of g(t, t) or ̂ g (t, ̃ θ) is normally a − series in powers of n . Suppose that E(Tn ) = θ − β n (θ), where β n (θ) = O(n− ) uniformly for θ in a neighborhood of θ , and that nTn = Sn , where Sn satisfies the Assumptions – given above with r = . Maximum likelihood estimators often satisfy these assumptions. We make the following further assumption: Assumption . Uniformly for θ in a neighborhood of θ , ∣Dn (θ)∣ = ∣D(θ)∣ { + O(n− )} . The assumption is, of course, satisfied when Sn is a sum of independent and identically distributed vectors but it is also satisfied in other cases of interest, notably in some applications in time series analysis. We suppose that we require a single-term approximation which has an error of order n− at most. √ Since Xn = n−/ E {Sn − E(Sn )} = n {Tn −θ+β n (θ)}, √ the value of Xn when Tn = t and θ = t is x = β n (θ) n. With r = , () gives ∣Dn (t)∣−/ ̂ exp {− nβ n (t)′ D− hn (x, t) = n (t)β n (t)} (π)m/ ⎤ ⎡ ⎥ ⎢ √ −(j/)+ Pnj {β n (θ) n, t}⎥ ×⎢ ⎥. ⎢ + ∑ n ⎥ ⎢ j= ⎦ ⎣ () − Now nβ n (t)′ D− n (t)β n (t) = O(n ) and the constant term of Pn is O(). Moreover Pn contains no constant term and √ hence is O(n−/ ) when x = β n (θ) n. We note that these orders of magnitude are uniform for t in a neighborhood of θ under the Assumptions –. Because of Assumption , we have ∣D(t)∣−/ ̂ { + O(n− )} hn (x, t) = (π)m/ uniformly for t in a neighborhood of θ .
Let hn (x, t) be the true density of Xn , then by () hn (x, t) = ̂ hn (x, t) + o(n− ) =
∣D(t)∣−/ { + O(n− )} + o(n− ). (π)m/
()
Since the term o(n− ) is uniform for t in a neighborhood of ∥t − θ ∥ < δ , where δ is a suitably chosen positive constant independent of n, and since ∣D(t)∣ is continuous at θ and hence is bounded away from zero for t in the neighborhood, the term o(n− ) of () can be absorbed inside the curly bracket. We thus have uniformly hn (x, t) =
∣D(t)∣−/ { + O(n− )} . (π)m/
Transforming from x to t we obtain for the density of Tn at Tn = θ = t, g(t, t) = (
n m/ ) ∣D(t)∣−/ { + O(n− )} . π
()
Substituting in () we obtain f (y, θ ) n m/ { + O(n− )} , ) ∣D(t)∣−/ π f (y, t) () uniformly in t for ∥t − θ ∥ < δ . Expression () is the basic approximation for the density of the sufficient estimator Tn . The fact that the error is a proportional error which is uniform over the region ∥t − θ ∥ < δ is important since the limiting probability that Tn falls outside this region is zero. Assuming appropriate regularity conditions to be satisfied, D− (θ) is the limiting mean information matrix I(θ), where g(t, θ ) = (
I(θ) = lim E [−n− n→∞
∂ log f (y, θ) ]. ∂θ∂θ ′
We then have for the basic approximation f (y, θ ) n m/ { + O(n− )} , ) ∣I(t)∣/ π f (y, t) () uniformly in t for ∥t − θ ∥ < δ . The simplicity of the structure of this approximation should be noted. It consists of the normal approximation to the density when θ = t, namely {n/(π)}m/ ∣I(t)∣/ , multiplied by the likelihood ratio f (y, θ )/f (y, t). g(t, θ ) = (
Approximations to Distributions
Durbin (a) proved that when either () or () is integrated over any subset of Rm , the error term remains O(n− ). This result is, in fact, of great importance in practical situations since it demonstrates that the basic approximation can be integrated for inference purposes with an error which is of order n− at most. He proved as well that when the constant term of the approximation (), and consequently also of (), is adjusted to make the integral over the whole space equal to unity, the order of magnitude of the error is often reduced from O(n− ) to Ox (n−/ ), where Ox (n−q ) denotes a quan√ tity which is O(n−q ) for each fixed x = n {t − E(Tn )} −q but which is not O(n ) uniformly for all x. This process of adjusting the constant term is generally called renormalization.
A
Hampel FR () Some small sample asymptotics. Proc Prague Symp Asymptotic Stat :– Loève M () Probability theory, vol I, th edn. Springer, Berlin Phillips PCB () Edgeworth and saddlepoint approximations in a first order autoregression. Biometrika :– Wallace DL () Asymptotic approximations to distributions. Ann Math Stat :–
Approximations to Distributions
About the Author
Juan Carlos Abril President of the Argentinean Statistical Society, Professor Universidad Nacional de Tucumán and Consejo Nacional de Investigaciones Científicas y Técnicas, San Miguel de Tucumán, Argentina
Professor Abril is co-editor of the Revista de la Sociedad Argentina de Estadstica (Journal of the Argentinean Statistical Society).
Introduction
Cross References 7Approximations to Distributions 7Edgeworth Expansion 7Properties of Estimators 7Sufficient Statistics
References and Further Reading Abril JC () Asymptotic expansions for time series problems with applications to moving average models. Ph.D. Thesis, The London School of Economics and Political Science, University of London, England Barndorff-Nielsen O, Cox DR () Edgeworth and saddle-point approximations with statistical applications. J R Stat Soc B :– Bhattacharya RN, Ghosh JK () On the validity of the formal Edgeworth expansion. Ann Statist :– Bhattacharya RN, Rao RR () Normal approximation and asymptotic expansions. Wiley, New York Chambers JM () On Methods of asymptotic approximation for multivariate distributions. Biometrika :– Daniels HE () Saddlepoint approximations in statistics. Ann Math Statist :– Daniels HE () The approximate distribution of serial correlation coefficients. Biometrika :– Durbin J (a) Approximations for the densities of sufficient estimators. Biometrika :– Durbin J (b) The approximate distribution of partial serial correlation coefficient calculated from residual from regression on Fourier series. Biometrika :– Feller W () An introduction to probability theory and its applications, vol , nd edn. Wiley, New York
The exact probability distribution of estimators for finite samples is only available in convenient form for simple functions of the data and when the likelihood function is completely specified. Frequently, these conditions are not satisfied and the inference is based on approximations to the sample distribution. Typically, large sample methods based on the central limit theorem (see 7Central Limit Theorems) are generally used. For example, if Tn is an estimator of the parameter θ based on a sample of size n, it is sometimes possible to obtain functions σ(θ) such that √ the distribution of the random variable n(Tn − θ)/σ(θ) converges to the standard normal distribution as n tends to infinity. In such a case, it is very common to approximate the distribution of Tn by a normal distribution with mean θ and variance σ (θ)/n. These asymptotic approximations can be good even for very small samples. The mean of independent draws from a rectangular distribution has a bell-shaped density for n as small as three. But it is easy to construct examples where the asymptotic approximation is bad even when the sample has hundreds of observations. It is therefore desirable to know the conditions under which the asymptotic approximations are reasonable and to have alternative methods available when these approximations do not work properly. Most of the material discussed here is closely related with the topic Asymptotic, higher order which is presented as well in this Encyclopedia. There is a good literature treating the theory and practice of approximations to distributions, but introductory
A
A
Approximations to Distributions
texts are relatively few. A very brief summary can be seen in Bickel and Doksum (), while some discussion is given in Johnson and Kotz (). The extension to asymptotic expansions can be seen in the excellent paper by Wallace (), although it is outdated. For a good treatment of the subject, an incursion upon the advanced probability and numerical analysis textbooks is needed. For those with enough time and patience, Chaps. and of Feller () are well worth reading.
The Central Limit Theorem The center of a large part of the asymptotic theory is the central limit theorem, initially formulated for sums of independent random variables. Let {Yn } be a sequence of independent random variables. Denote by Hn the distribution function of the standardized sum n
∑ {Yj − E(Yj )} j= , Xn = ¿ Á n Á À{ ∑ V(Yj )} j=
where V(Yj ) is the variance of Yj , and by N (.) the standard normal distribution function. The central limit theorem then states that lim Hn (x) = N (x), as n → ∞, for every fixed x, provided only that the means and variances are finite. If the {Yj } are not identically distributed, an additional condition guaranteeing that the distributions are not too unbalanced is necessary. For time series problems, for example, where in general the variables are not independent, there have been particularized versions of this theorem guaranteeing the asymptotic behavior of statistics used in this area. Good references are the textbook by Anderson (), Brockwell and Davis (), Hannan (), and Priestley () where one can find an excellent treatment of the asymptotic theory applied to time series problems. Some authors have shown that the order of magnitude of the errors in the central limit theorem is O(n−/ ). While the central limit theorem is very useful theoretically and often in practice, it is not always satisfactory since for small or moderate n the errors of the normal approximation may be too large.
Curve Fitting The most simplest form for obtaining an approximation to a distribution is to look for a family of curves with the correct shape and select the member that fits best. If the moments, specially those of low order, of the true distribution are known, they can be used in the fitting process.
Otherwise one can use Monte Carlo simulations or any other information about the true distribution. Durbin and Watson () describe a number of different approximations to the null distribution of the statistic d used for testing serial correlation in regression analysis. One of the most accurate is the beta approximation proposed by Henshaw (). Since d is between zero and four and it seems to have a unimodal density, it is reasonable to think that a linear transformation from a beta distributed variable can be a good approximation to the true distribution. Suppose that Y is a random variable with beta distribution function y
t p− ( − t)q− dt = G(y; p.q), Pr(Y ≤ y) = B(p, q) ∫
where
∞
B(p, q) = ∫ t p− ( − t)q− dt.
Then, for a and b constant, the random variable a + bY has moments depending on p, q, a and b. These moments are easy to express analytically. Moreover, the moments of the Durbin–Watson’s statistic d are simple functions of the matrix of explanatory variables. Equating the first four moments of d with the corresponding moments of a + bY, one obtains four equations with four unknowns. For a given matrix of explanatory variables these equations give a unique solution, p∗ , q∗ , a∗ and b∗ say. So Pr(d ≤ y) can be approximated by G {(y − a∗ )/b∗ ; p∗ , q∗ }. This approximation gives good results in many cases. Theil and Nagar () proposed a similar approximation but using the approximated moments of d instead of the true moments. Since these approximated moments are independent of the matrix of explanatory variables, Theil–Nagar’s approximation does not depend on the data and can be tabulated without any problem. Unfortunately the approximated moments are not always accurate and the resulting approximation to the distribution is less satisfactory than Henshaw’s approximation. If one has enough information over the true density, the curve fitting methods give simple and correct approximations. However these methods are not so attractive when the purpose is not quantitative but qualitative. The comparison of alternative procedures is difficult because the curve fitting methods does not produce, in general, parametric families of curves easily comparable. If two statistics are approximately normal, they can be compared by their means and variances. If one statistic is approximately beta and another is approximately normal, the comparison between them is not easy since the usual parameters that describe one of the distributions are
Approximations to Distributions
not of much interest for obtaining information about the other. The flexibility that makes the curve fitting method so accurate is, as well, an inconvenience for using it in comparisons.
Transformations Suppose that Y is a random variable and b a monotonically increasing function such that b(Y) has a distribution funĉ Since Pr(Y ≤ y) is tion H which can be approximated by H. equal to Pr {b(Y) ≤ b(y)}, the distribution function of Y ̂ {b(y)}. A well known example can be approximated by H of this technique is Fisher’s z transformation. The sample correlation coefficient ̂ ρ based on a random sample from a bivariate normal population is very far from symmetry when the true coefficient ρ is large in absolute value. But, z = b(̂ ρ) = − log {( + ̂ ρ )/( − ̂ ρ)} is almost symmetric and can be approximated by a normally distributed random variable with mean − log {( + ρ)/( − ρ)} and variance n− . Therefore Pr(̂ ρ ≤ y) can be approxi√ √ mated by N { nb(y) − nb(ρ)} for moderate sample size n. The use of transformations for approximating distributions is an art. Sometimes, as in the case of the correlation coefficient, the geometry of the problem can suggest the appropriate transformation b. Since ̂ ρ can be interpreted as the cosine of the angle between two normally distributed random vectors, an inverse trigonometric transformation can be useful. In other cases, arguments based on approximations to the moments are helpful. Suppose that b(Y) can be expanded as a power series about µ = E(Y) ′′ b(Y) = b(µ) + b′ (µ)(Y − µ) + b (µ)(Y − µ) + ⋯, where Y − µ is in some sense small. so we can do E(b) ≈ b(µ) + b′′ (µ)E(Y − µ) ,
to differential equations in b(µ). Note that Fisher’s transformation of ̂ ρ stabilizes the approximated variance of b making it independent of ρ. Jenkins () and Quenouille () apply inverse trigonometric transformations to the case of the autocorrelation coefficient in time series. The use of transformations in econometrics seems, however, to be minimum due mainly to the fact that the method is closely related with univariate distributions.
Asymptotic Expansions Frequently it is possible to decompose the problem of finding the distribution in a sequence of similar problems. If the sequence has a limit which can easily be found, one can obtain an approximation to the solution of the original problem by a solution of the limit problem. The sequence of the problem is indexed by a parameter, which usually is the sample size n. Suppose for instance that we want an approximation to the distribution of an estimator, computed from a sample, of a parameter θ. We define an infinite sequence θ̂n of estimators, one for each sample size n = , , . . ., and we consider the problem of obtaining the distribution of each θ̂n . Of course, it is necessary to have some description of the joint distribution of the observations for each n. Given such a sequence of problems, the asymptotic approach implies three steps: (a) To look for a simple monotonic transformation Xn = b(θ̂n ; θ, n) such that the estimator Xn is not very sen-
sitive to n. Since the majority of estimators are centerd upon the true value of the parameter and they have a dispersion which decreases at the same rate √ ̂ n(θ n − θ) is as n−/ , the transformation Xn = frequently used. ̂ n (x) to the distribution (b) To look for an approximation H function Hn (x) = Pr(Xn ≤ x) such that, when n tends to infinity, the error ̂ n (x) − Hn (x)∣ ∣H
V(b) ≈ {b′ (µ)} V(Y),
E {b − E(b)} ≈ {b′ (µ)} E(Y − µ) +
′ {b (µ)} b′′ (µ)E(Y − µ) ,
and choose b in such a way that these approximates moments are equal to the moments of the approximated distribution. If the approximated distribution is normal, we can require that the variance V(b) be a constant independent of µ; or we can require that the third order moment be zero. If the moments of Y are (almost) known and the above approximation is used, the criterion leads
A
tends to zero. ̂n , (c) The distribution function of θ̂n is approximated by H i.e., Pr(θ̂n ≤ a) = Pr {Xn ≤ bn (a; θ, n)} is approxî n {bn (a; θ, n)}. mated by H ̂ n (x) be an approximation to the distribution Let H function Hn (x). If, for every x, ̂ n (x) − Hn (x)∣ = , lim n(r/)− ∣H
n→∞
r = , , . . . ,
we write ̂ n (x) + o {n(r/)− } , Hn (x) = H
r = , , . . . ,
A
A
Approximations to Distributions
̂ n (x) is an approximation o {n(r/)− } or and we say that H an approximation of order r − . These names are used as well when approximating density functions. The asymptotic distribution is an approximation o(n ) = o() or a first order approximation. These concepts are related with the topic Asymptotic, higher order which is presented as well in this Encyclopedia. The number n measures the velocity at which the error of approximation tends to zero as n tends to infinity. If we ̂ n vary genchoose the transformation b such that Hn and H tly with n, the value of r can give an indication of the error of approximation for moderate values of n. There are two well known methods for obtaining high order approximations to distributions, both based on the Fourier inversion of the characteristic function. Let ϕ n (z, θ) = E {exp(izXn )} be the characteristic function of Xn and let ψ n (z, θ) = log ϕ n (z, θ) be the cumulant generating function. If ϕ n is integrable, the density function hn of Xn can be written as ∞
e−ixz ϕ n (z, θ)dz hn (x; θ) = π ∫ −∞
∞
=
exp {−ixz + ψ n (z, θ)} dz. π ∫
()
−∞
Frequently it is possible to expand ψ n (z, θ) in power series where the successive terms are increasing powers of n−/ . In this case the integrand can be approximated by the first few terms of this series expansion. Integrating term by term, one obtains a series approximation to hn ; afterward integration will give an approximation to the distribution function. The approximation known as Edgeworth approximation or 7Edgeworth expansion consists in expanding ψ n (z, θ) at z = . This method is the most frequently used in practice because of its relative simplicity. It does not require a complete knowledge of ψ n (z, θ). It is enough if one knows the first low order cumulants of Xn . More details about this method is given in this Encyclopedia under the name Edgeworth expansion. The approximation known as saddlepoint approximation is obtained by expanding ψ n (z, θ) at the “saddlepoint” value z∗ where the integrand of () is maximized. This method, introduced by Daniels (), is more complex and requires a deeper knowledge of the function ψ n (z, θ).When this knowledge is available, the method gives accurate approximations specially in the “tail” region of the distribution. Daniels () and Phillips () applied this method to some autocorrelation statistics in time series analysis. More details about
this method is given in this Encyclopedia under the name Saddlepoint approximations. Wallace () gives an excellent introduction to the approximations based on expansions of the characteristic function. An exposition with emphasis on multivariate expansions can be found in Barndorff-Nielsen and Cox (). Durbin () proposed a simple method for obtaining a second order approximation to the density of a large class of statistics. This method is discuss in this Encyclopedia under the name Approximations for densities of sufficient estimators.
Attitudes and Perspectives The theory of approximate distributions, like the theory of exact distributions, depends on the assumptions made about the stochastic process which generates the data. The quality of the approximations will not be better than the quality of the specifications sustaining them. one certainly will not rely upon a theory of distribution unless the conclusions are so robust that they do not vary significantly in front of moderate changes of basic assumptions. Since the majority of the methods of approximation use information about the first four moments at least, while the usual asymptotic theory only need information about the first two moments, some loss of robustness has to be expected. However, if some idea about the degree of skewness and kurtosis is available, this information can be helpful to obtain better approximations to the distribution of statistics. Recently there has been an increasing interest in asymptotic theory. Great efforts have been made in order to demonstrate that some statistics are asymptotically normal and efficient. Of course, the asymptotic theory is important to have an idea of the sample properties of a given statistical procedure. Unfortunately there has been some confusion with the use of the terms “asymptotic” and “approximated.” The fact that a standardized estimator has an asymptotic normal distribution is purely a mathematical proposition about the limit of the probabilities measures under a set of previously specified assumptions. The fact that a given estimator is approximately normal suggests that, for this particular problem, one believes in the possibility of treating the estimator as if it was normal. Sometimes, under certain circumstances, asymptotic arguments lead to good approximations, bet frequently they do not. A careful analyst, with some knowledge of statistical theory, a modest computer and a great amount of common sense can find reasonable approximations for a given inferential problem.
Association Measures for Nominal Categorical Variables
About the Author Professor Abril is co-editor of the Revista de la Sociedad Argentina de Estadstica (Journal of the Argentinean Statistical Society).
A
Theil H, Nagar AL () Testing the independence of regression disturbances. J Am Stat Assoc :– Wallace DL () Asymptotic approximations to distributions. Ann Math Stat :–
Cross References 7Approximations for Densities of Sufficient
Estimators 7Asymptotic Normality 7Asymptotic, Higher Order 7Central Limit Theorems 7Cornish-Fisher Expansions 7Edgeworth Expansion 7Limit Theorems of Probability Theory 7Saddlepoint Approximations 7Strong Approximations in Probability and Statistics
References and Further Reading Abril JC () Asymptotic expansions for time series problems with applications to moving average models. PhD thesis, The London School of Economics and Political Science, University of London, England Anderson TW () The statistical analysis of time series. Wiley, New York Barndorff-Nielsen O, Cox DR () Edgeworth and saddle-point approximations with statistical applications. J R Stat Soc B :– Bickel PJ, Doksum KA () Mathematical statistics. Holden-Day, San Francisco Brockwell PJ, Davis RA () Time series: theory and methods, nd edn. Springer, New York Daniels HE () Saddlepoint approximations in statistics. Ann Math Stat :– Daniels HE () The approximate distribution of serial correlation coefficients, Biometrika :– Durbin J, () Approximations for the densities of sufficient estimates. Biometrika :– Durbin J, Watson GS () Testing for serial correlation in least squares regression, III. Biometrika :– Feller W () An Introduction to probability theory and its applications, vol , nd edn. Wiley, New York Hannan EJ () Multiple time series. Wiley, New York Henshaw RC () Testing single-equation least-squares regression models for autocorrelated disturbances. Econometrica :– Jenkins GM () An angular transformation for the serial correlation coefficient. Biometrika :– Johnson NI, Kotz S () Continuous univariate distributions, vol . Wiley, New York Phillips PCB () Edgeworth and saddlepoint approximations in a first order autoregression. Biometrika :– Priestley MB () Spectral analysis and time series. Academic, London Quenouille MH () Some results in the testing of serial correlation coefficients. Biometrika :–
Association Measures for Nominal Categorical Variables Tarald O. Kvålseth Professor Emeritus University of Minnesota, Minneapolis, MN, USA
As a means of summarizing the potential relationship between two (or more) random categorical variables X and Y, a number of measures of association have been proposed over the years. A historical review of such measures and new proposals have been presented in a series of papers by Goodman and Kruskal () [see also Kendall and Stuart (), Ch. and Liebetrau ()]. Such summary measures depend on whether X and Y are nominal or ordinal as well as on whether X and Y are to be treated symmetrically or asymmetrically. In the symmetric case, X and Y are treated equivalently and no causal relationship is assumed to exist between them. In the asymmetric case, a causal relationship between X and Y is considered to exist so that one variable is treated as the explanatory variable (X) and the other variable treated as the response variable (Y). The focus here will be on the case when both X and Y are nominal categorical variables, i.e., no natural ordering exists for the variables. Association measures for both the symmetric and asymmetric case will be considered.
Symmetric Measures For the variable X with I categories and the variable Y with J categories, their joint and marginal probabilities are defined as Pr(X = i, Y = j) = pij , Pr(X = i) = pi+ , and Pr(Y = j) = p+j for i = , . . . , I and j = , . . . , J where
∑i = I ∑j = Jpij = ∑i = Ipi+ = ∑j = Jp+j = . In terms
of a two-way contingency table with I rows and J columns, the cell entries come from the joint distribution {pij }, with pij being the entry in cell (i, j), and {pi+ } and {p+j } are the marginal distributions (totals) for the rows and columns, respectively. The conditional distribution of Y given X is defined in terms of pj∣i = pij /pi+ for all i and j.
A
A
Association Measures for Nominal Categorical Variables
Several early suggested association measures were based on the (Pearson) coefficient of mean square contingency defined by I
J
Φ = ∑ ∑ i= j=
pij
I J (pij − pi+ p+j ) = ∑∑ − . pi+ p+j i= j= pi+ p+j
()
If the pij represent sample estimates (of population probabilities π ij ) pij = nij /N based on the multinominal frequen
cies nij for all i, j and sample size N = ∑i = I ∑j = Jnij , then it is recognized that Φ = X /N where X is the familiar Pearson chi-square goodness-of-fit statistic for testing the null hypothesis or independence between X and Y, i.e., I
J
X = N(∑ ∑ i= j=
nij − ). ni+ n+j
()
The most popular such association measure based on X appears to be Cramér’s () V defined as √
X , N(M − )
G ≤ GX
I
= ∑ ni+ log ( i=
N ) and G ≤ GY ni+
N = ∑ n+j log ( ). n +j j= Thus, analogously to V in (), one could define the association measure √ G () W= min{GX , GY } with GX and GY as given in (). This new measure W can also be interpreted as the divergence (“distance”) of the distribution {pij } from the independence distribution {pi+ p+j } relative to its maximum [see also Kvålseth ()]. The W has the same type of properties as Cramér’s V in () and can be expected to take on values quite similar to those of V. For instance, for the data n =
n =
n =
This V ranges in value between and , inclusive, for any given I and J, with V = if, and only if, X and Y are independent and V = when there is no more than one non-zero entry in either each row or in each column. The V is invariant with any permutations of the rows or the columns. The estimated standard error of V is given in Bishop et al. (, p. ), but its expression is rather messy. Kendall and Stuart (, p. ), have shown that V is the mean squared canonical correlation. However, it has been argued that values of V are difficult to interpret since V has no obvious probabilistic meaning or interpretation. Nevertheless, V does reflect the divergence (or “distance”) of the distribution {pij } from the independence distribution {pi+ p+j } relative to the maximum divergence. There is some uncertainty in the literature as to whether V or V is the proper measure to use. This issue will be addressed in a section below. It may also be pointed out that a similar association measure can be formulated in terms of the likelihood-ratio statistic G , which has the same asymptotic chi-square distribution as χ under the null hypothesis and is often used instead of χ . For the G under independence, i.e., for
n =
n =
n =
I
J
G = ∑ ∑ nij log ( i= j=
Nnij ) ni+ n+j
()
J
()
V=
M = min {I, J} .
and since nij ≤ ni+ and nij ≤ n+j for all i and j, it follows that
it is found from () and () that V = . and W = ..
Asymmetric Measures Goodman and Kruskal have discussed two different asymmetric association measures (λ Y∣X ) and (τ Y∣X ) for the case when X can be considered to be the explanatory variable and Y the response variable. Such measures are frequently referred to as proportional reduction in error (PRE) measures since they can be interpreted in terms of the relative difference between two error probabilities PY and PY∣X , i.e., PY − PY∣X () PREY∣X = PY where PY is the probability of error when predicting the Y – category of a randomly selected observation or item without knowing its X – category and PY∣X is the corresponding expected (weighted mean) error probability given its X – category. The optimal prediction strategy would clearly be to predict that a randomly selected observation (item) would belong to a maximum-probability (modal) category, so that with p+m = max{p+ , . . . , p+J }; and pim = max{pi , . . . , piJ },
()
i = , . . . , I
Association Measures for Nominal Categorical Variables
the error probabilities PY and PY∣X become I
I
i=
i=
PY = −p+m , PY∣X = ∑ pi+ ( − pim /pi+ ) = − ∑ pim . () From ()–(), the so-called Goodman–Kruskal lambda becomes
∑i = Ipim − p+m
λ Y∣X =
− p+m
()
which is the relative decrease in the error probability when predicting the Y-category as between not knowing and knowing the X-category. Another asymmetric measure is based on a different prediction rule: Predictions are made according to the given probabilities. Thus, a randomly chosen observation (item) is predicted to fall in the jth category of Y with probability p+j (j = , . . . , J) if its X-category is unknown. If, however, the observation is known to belong to the ith category of X, it is predicted to belong to the jth category of Y with the (conditional) probability pij /pi+ (j = , . . . , J). The error probabilities are then given by J
, PY = − ∑ p+j
I
J
i=
j=
PY∣X = ∑ pi+ [ − ∑(pij /pi+ ) ] ()
j=
so that, from () and (), the following so-called Goodman–Kruskal tau results: I
J
J
∑ ∑ pij /pi+ − ∑ p+j
τ Y∣X =
i= j=
j=
−
J
()
∑ p+j j=
which gives the relative reduction in the error probability when predicting an observation’s Y-category as between its X-category not given and given. Both measures in () and (), and whose estimated standard errors are given elsewhere [e.g., Bishop et al. (, pp. –), Goodman and Kruskal (), and Liebetrau ()], can assume values between and , inclusive. Both equal if, and only if, each row of the contingency table contains no more than one non-zero cell entry. Both are invariant under permutations of rows or of columns. However, their zero-value conditions differ. The τ Y∣X = if, and only if, X and Y are independent, whereas λ Y∣X = if () X and Y are independent or () the modal probabilities pim in all rows fall in the same column. This second condition is most likely to occur when the marginal distribution {pi+ } is highly uneven (non-uniform). Thus, in cases of highly uneven {pi+ }, λ Y∣X may be or very small, while other measures such as τ Y∣X may be substantially larger. The high sensitivity of λ Y∣X to {pi+ } is one
A
limitation of this measure that may lead to misleadingly low association values. Symmetric version of lambda and tau can also be formulated in terms of weighted averages (Goodman and Kruskal ). Thus, in terms of the general expression in (), a symmetric PRE could be formulated as the following weighted mean of PREY∣X and PREX∣Y : PRE =
PY − PY∣X + PX − PX∣Y . PY + PX
However, there would seem to be no strong reason for preferring such symmetricized measures over the V or W in () and (). It should be pointed out that asymmetric association measures can also be formulated in terms of relative reduction in variation, somewhat analogously to the coefficient of determination (R ) used in regression analysis. This can be done by basically replacing the prediction error probabilities in () with appropriate measures of categorical variation (Agresti , pp. –).
Concluding Comment For Cramér’s V in (), there is inconsistency in the literature concerning the use of V versus V (and Cramér himself proposed V (Cramér , p. )). Also, concern has been expressed that different measures such as those in () and () can produce quite different results. Such issues are indeed important and are often overlooked. As with any summary measure, and so it is with association measures, it is essential that a measure takes on values throughout its range that are reasonable in that they provide true or valid representations of the attribute being measured. In order to make such an assessment for the above association measures, consider the simple case of a × table with all the marginal probabilities equal to . and with the following cell entries: p = ( − w)/p = ( + w)/ p = ( + w)/p = ( − w)/ with ≤ w ≤ . Each of these probabilities are seen to be the weighted mean of the corresponding probabilities for the case of perfect association and zero association (independence) for the given marginal probabilities, i.e., p = p = w()+(−w)(.),
p = p = w(.)+(−w)(.)
In order for some association measure A ∈ [, ] to take on reasonable values in this ease, the only logical requirement is clearly that A = w(A = ) + ( − w)(A = ) = w, w ∈ [, ]
A
A
Astrostatistics
It is readily seen that the measures in () and () meet this requirement for all w, i.e., V = w (and not V ) and λ Y∣X = w for the above {pij } – distribution. However, it is √ ′ seen that, for (), λ Y∣X = w . This shows that τ Y∣X = τ Y∣X should be used as an association measure rather than τ Y∣X . In the case of W in (), it is apparent that W is only approximately equal to w, but the approximation appears to be sufficiently close for W to be a competitive association measure.
About the Author For biography see the entry 7Entropy.
Cross References 7Categorical Data Analysis 7Scales of Measurement 7Variables 7Variation for Categorical Variables
References and Further Reading Agresti A () Categorical data analysis, nd edn. Wiley, Hoboken, NJ Bishop YMM, Fienberg SE, Holland PW() Discrete multivariate analysis. MIT, Cambridge, MA Cramér H () Mathematical methods for statistics. Princeton University Press, Princeton, NJ Goodman LA, Kruskal WH () Measures of association for crossclassifications. Springer, New York Kendall M, Stuart A () The advanced theory of statistics, vol., th edn. Charles Griffin, London Kvålseth TO () Entropy and correlation: some comments. IEEE Transactions on Systems, Man, and Cybernetics, SMC-, – Liebetrau AM () Measures of Association. Beverly Hills, CA: Sage Publications.
Astrostatistics Eric D. Feigelson Professor, Associate Director Penn State Center for Astrostatistics Pennsylvania State University, University Park, PA, USA
Introduction The term “astronomy” is best understood as short-hand for “astronomy and astrophysics.” Astronomy is the observational study of matter beyond Earth: planets and other bodies in the Solar System, stars in the Milky Way Galaxy, galaxies in the Universe, and diffuse matter between these
concentrations of mass. The perspective is rooted in our viewpoint on or near Earth, typically using telescopes or robotic satellites. Astrophysics is the study of the intrinsic nature of astronomical bodies and the processes by which they interact and evolve. This is an inferential intellectual effort based on the well-confirmed assumption that physical processes established to rule terrestrial phenomena – gravity, thermodynamics, electromagnetism, quantum mechanics, plasma physics, chemistry, and so forth – also apply to distant cosmic phenomena. Statistical techniques play an important role in analyzing astronomical data and at the interface between astronomy and astrophysics. Astronomy encounters a huge range of statistical problems: samples selected with truncation; variables subject to censoring and heteroscedastic measurement errors; parameter estimation of complex models derived from astrophysical theory; anisotropic spatial clustering of galaxies; time series of periodic, stochastic, and explosive phenomena; image processing of both grayscale and Poissonian images; 7data mining of terabytepetabyte datasets; and much more. Thus, astrostatistics is not focused on a narrow suite of methods, but rather brings the insights from many fields of statistics to bear on problems arising in astronomical research.
History As the oldest observational science, astronomy was the driver for statistical innovations over many centuries (Stigler ; Hald ). Hipparchus, Ptolemy, al-Biruni, and Galileo Galilei were among those who discussed methods for averaging discrepant astronomical measurements. The least squares method (see 7Least Squares) and its understanding in the context of the normal error distribution were developed to address problems in Newtonian celestial mechanics during the early nineteenth century by Pierre-Simon Laplace, Adrian Legendre, and Carl Friedrich Gauss. The links between astronomy and statistics considerably weakened during the first decades of the twentieth century as statistics turned its attention to social and biological sciences while astronomy focused on astrophysics. Maximum likelihood methods emerged slowly starting in the s, and Bayesian methods are now gaining considerably popularity. Modern astrostatistics has grown rapidly since the s. Several cross-disciplinary research groups emerged to develop advanced methods and critique common practices (http://hea-www.harvard.edu/AstroStat; http://www. incagroup.org; http://astrostatistics.psu.edu). Monographs were written on astrostatistics (Babu and Feigelson ), galaxy clustering (Martinez and Saar ), image processing (Starck and Murtagh ), Bayesian analysis
Astrostatistics
(Gregory ), and Bayesian cosmology (Hobson et al. ). The Statistical Challenges in Modern Astronomy (Babu and Feigelson ) conferences bring astronomers and statisticians together to discuss methodological issues. The astronomical community is devoting considerable resources to the construction and promulgation of large archival datasets, often based on well-designed surveys of large areas of the sky. These surveys can generate petabytes of images, spectra and time series. Reduced data products include tabular data with approximately ten variables measured for billions of astronomical objects. Major projects include the Sloan Digital Sky Survey, International Virtual Observatory, and planned Large Synoptic Survey Telescope (http://www.sdss.org, http://www.ivoa.net, http://www.lsst.org). Too large for traditional treatments, these datasets are spawning increased interest in computationally efficient data visualization, data mining, and statistical analysis. A nascent field of astroinformatics allied to astrostatistics is emerging.
A
are estimated by minimizing the weighted sum of squared ˆ /σi of the observed data Oi residuals ∑i (Oi − Mi (θ) where σi are the known variances of the measurement errors. More sophisticated methods are being developed, but have not yet entered into common usage. Kelly () treats structural regression as an extension of a normal mixture model, constructing a likelihood which can either be maximized with the EM Algorithm or used in 7Bayes’ theorem. The Bayesian approach is more powerful, as it also can simultaneous incorporate censoring and truncation into the measurement error model. Delaigle and Meister () describe a nonparametric kernel density estimator that takes into account the heteroscedastic errors. More methods (e.g., for multivariate clustering and time series modeling) are needed.
Censoring and Truncation
Topics in Contemporary Astrostatistics Given the vast range of astrostatistics, only a small portion of relevant issues can be outlined here. We outline three topics of contemporary interest (The astronomical research literature can be accessed online through the SAO/NASA Astrophysics Data System, http://adsabs. harvard.edu.).
Heteroscedastic Measurement Errors Astronomical measurements at telescopes are made with carefully designed and calibrated instruments, and “background” levels in dark areas of the sky are examined to quantitatively determine the noise levels. Thus, unlike in social and biological science studies, heteroscedastic measurement error are directly obtained for each astronomical measurement. This produces unusual data structures. For example, a multivariate table of brightness of quasars in six photometric bands will have columns of numbers giving the measured brightness and the associated measurement error in each band. Unfortunately, few statistical techniques are available for this class of non-identically distributed data. Most errors-in-variables methods are designed to treat situations where the heteroscedasticity is not measured, and instead becomes part of the statistical model (Carroll et al. ). Methods are needed for density estimation, regression, multivariate analysis and classification, spatial processes, and time series analysis. Common estimation procedures in the astronomical literature weight each measurement by its associated error. For instance, in a functional regression model, the parameters θˆ in model M
In the telescopic measurement of quasar brightnesses outlined above, some targeted quasars may be too faint to be seen above the background noise level in some photometric bands. These nondetections lead to censored data points. The situation is similar in some ways to censoring treated by standard survival analysis, but differs in other ways: the data are left-censored rather than rightcensored; censoring can occur in any variable, not just a single response variable; and censoring levels are linked to measurement error levels. Survival techniques have come into common usage in astronomy since their introduction (Isobe et al. ). They treat some problems such as density estimation (with the Kaplan-Meier product-limit estimator), two-sample tests (with the Gehan, logrank and Peto-Prentice tests), correlation (using a generalization of Kendall’s τ), and linear regression (using the BuckleyJames line). Consider a survey of quasars at a telescope with limited sensitivity where the quasar sample is not provided in advance, but is derived from the photometric colors of objects in the survey. Now quasars which are too faint for detection are missing entirely from the dataset. Recovery from this form of truncation is more difficult than recovery from censoring with a previously established sample. A major advance was the derivation of the nonparametric estimator for a randomly truncated dataset, analogous to the Kaplan-Meier estimator for censored data, by astrophysicist Lynden-Bell (). This solution was later recovered by statistician Woodroofe (), and bivariate extensions were developed by Efron and Petrosian ().
A
A
Astrostatistics
Periodicity Detection in Difficult Data
Cross References
Stars exhibit a variety of periodic behaviors: binary star or planetary orbits; stellar rotation; and stellar oscillations. While Fourier analysis is often used to find and characterize such periodicities, the data often present problems such as non-sinusoidal repeating patterns, observations of limited duration, and unevenly-spaced observations. Nonsinusoidal periodicities occur in elliptical orbits, eclipses, and rotational modulation of surface features. Unevenlyspaced data arise from bad weather at the telescope, diurnal cycles for ground-based telescopes, Earth orbit cycles for satellite observatories, and inadequate observing time provided by telescope allocation committees. Astronomers have developed a number of statistics to locate periodicities under these conditions. The Lomb-Scargle periodogram (Scargle ) generalizes the Schuster periodogram to treat unevenly-spaced data. Stellingwerf () presents a widely used least-squared technique where the data are folded modulo trial periods, grouped into phase bins, and intra-bin variance is compared to inter-bin variance using χ . The method treats unevenly spaced data, measurement errors, and non-sinusoidal shapes. Dworetsky () gives a similar method without binning suitable for sparse datasets. Gregory and Loredo () develop a Bayesian approach for locating non-sinusoidal periodic behaviors from Poisson distributed event data. Research is now concentrating on methods for computationally efficient discovery of planets orbiting stars as they eclipse a small fraction during repeated transits across the stellar surface. These methods involve matched filters, Bayesian estimation, least-squares box-fitting, maximum likelihood, 7analysis of variance, and other approaches (e.g., Pontopappas et al. ).
7Chaotic Modelling 7False Discovery Rate 7Heteroscedasticity 7Linear Regression Models 7Statistics, History of
About the Author Dr. Eric Feigelson was trained in astronomy at Haverford College and Harvard University during the s, and entered the Astronomy and Astrophysics faculty at Pennsylvania State University in where he received an NSF Presidential Young Investigator Award and is now Professor. In addition to X-ray astronomical studies of star and planet formation, he has a long-standing collaboration with statisticians. Working with G. Jogesh Babu at Penn State’s Center for Astrostatistics, he organizes summer schools, conferences and other resources for advancing statistical methodology in astronomy. He serves as an Associate Editor of the Astrophysical Journal, on the Virtual Astronomical Observatory Science Council, and other organizations relating statistics to astronomy.
References and Further Reading Babu GJ, Feigelson ED () Astrostatistics. Chapman and Hall, London Babu GJ, Feigelson ED () Statistical challenges in modern astronomy IV. Astronomical Society of the Pacific, San Fransisco, California Carroll RJ, Ruppert D, Stefanski LA, Crainiceanu CM () Measurement errors in nonlinear models. Chapman and Hall/CRC, Boca Raton, FL Delaigle A, Meister A () Density estimation with heteroscedastic error. Bernoulli :– Dworetsky MM () A period-finding method for sparse randomly spaced observations of ‘How long is a piece of string?’. Mon Not Royal Astro Soc :– Efron B, Petrosian V () A simple test of independence for truncated data with applications to redshift surveys. Astrophys J :– Gregory PC () Bayesian logical data analysis for the physical sciences. Cambridge University Press, Cambridge, UK Gregory PC, Loredo TJ () A new method for the detection of a periodic signal of unknown shape and period. Astrophys J :– Hald A () A history of mathematical statistics from to . Wiley, New York Hobson MP et al (eds) () Bayesian methods in cosmology. Cambridge University Press, Cambridge Isobe T, Feigelson ED, Nelson PI () Statistical methods for astronomical data with upper limits. II–correlation and regression. Astrophys J :– Kelly BC () Some Aspects of Measurement Error in Linear Regression of Astronomical Data. Astrophys J :– Lynden-Bell D () A method of allowing for known observational selection in small samples applied to CR quasars. Mon Not R Astro Soc :– Martinez VJ, Saar E () Statistics of the galaxy distribution. CRC, Boca Raton, USA Protopapas P, Jimenez R, Alcock C () Fast identification of transits from light-curves. Mon Not R Astro Soc : – Scargle JD () Studies in astronomical time series analysis. II Statistical aspects of spectral analysis of unevenly spaced data. Astrophys J :– Starck J-L, Murtagh F () Astronomical image and data analysis. Springer, New York Stellingwerf RF () Period determination using phase dispersion minimization. Astrophys J :– Stigler SM () The history of Statistics: the measurement of uncertainty before . Harvard University Press, Cambridge, MA Woodroofe MB () Estimating a distribution function with truncated data. Ann Statist :–
Asymptotic Normality
Asymptotic Normality John E. Kolassa Professor Rutgers University, Newark, NJ, USA Consider a sequence of random variables Tn , whose distribution depends on a parameter n that generally represents sample size. The sequence is said to be asymptotically normal if there exists a sequences µ n and σn such that limn→∞ P [(Tn − µ n )/σn ≤ x] = Φ(x) for all x, where Φ(x) is the standard Gaussian distribution function x −/ dy. () ∫ exp(−y /)(π)
●
−∞
One often writes Tn ∼ AN(µ n , σn )
()
to express asymptotic normality. Note that µ n generally depend on n, and furthermore may be data-dependent. Furthermore, in some cases Tn might be a sequence of random vectors; in this case, µ n is a sequence of vectors, σn is a sequence of matrices, and Φ the vector valued counterpart of (). In the scalar case, for fixed n, the quantity σn is called the standard error of Tn . Many frequentist statistical inferential procedures are performed by constructing a Tn so that () holds under a null hypothesis, with a dissimilar distribution under interesting alternative hypotheses, and reporting ( − Φ(∣(Tn − µ n )/σn ∣))
()
as a two-sided p-value; the application for one-sided p-values is similar, and there are also Bayesian applications of a similar flavor. Serfling () provides further information. Consider the following examples of quantities that are asymptotically normal: ●
If Tn is the mean of n independent and identically distributed random variables, each with expectation µ and standard deviation σ, then Tn ∼ AN(µ, σ /n).
()
Furthermore, if sn is the traditional standard deviation of the contributors the the mean, Tn ∼ AN(µ, sn /n);
()
note that the standard error here is data-dependent, √ and it is incorrect to call sn / n a standard deviation of Tn , even approximately. In the present case square root
●
A
of the second argument to the AN operator estimates the standard deviation of Tn , but a further example shows that even this need not be true. In this case, the standard Z-test for a single sample mean follows from using () when σ is known, and when the components of Tn are binary, the standard standard Z-test for a single sample mean follows from using () with σ the standard Bernoulli variance. When σ is unknown, () is often used instead, and for n ≤ , the t distribution function is generally used in place of Φ in () for greater accuracy. Many rank-based statistics are asymptotically normal; for example, if Tn is the Wilcoxon signed-rank statistic (see 7Wilcoxon-Signed-Rank Test) for testing whether the expectation of n independent and identically distributed random variables takes on a particular null value, assuming symmetry and continuity of the underlying distribution. Without loss of generality, take this null mean to be zero. Then Tn is obtained by ranking the absolute values of the observations, and summing the ranks of those observations that are positive. Hettmansperger () √ notes that () holds with µ n = n(n + )/ and σn = n(n + )(n + )/, and the test against the two-sided alternative reports the p-value (). In this case, Tn may be written as the sum of independent but not identically-distributed random variables, or as the sum of identically-distributed but not independent random variables. Many parameter estimates resulting from fitting models with independent observations are asymptotically normal. For example, consider independent Bernoulli observations Yi with P [Yi = ] = exp(β + β xi )/( + exp(β + β xi )). Let n
ℓ(β) = ∑ [Yi β + xi Yi β − log( + exp(β + β xi ))] , i=
() and let βˆ maximize ℓ; here βˆ implicitly depends on n. Then βˆ ∼ AN(β, [−ℓ′′ (β)]− ),
()
as one can see by heuristically expressing ℓ′ (β) + ℓ′′ ˆ = , and solving for βˆ to obtain (β)(βˆ − β) ≈ ℓ′ (β) βˆ ≈ β − [ℓ′′ (β)]− ℓ′ (β), noting that ℓ′′ (β) is nonrandom, and noting that a variant of the Central Limit Theorem proves the asymptotic normality of ℓ′ (β), ˆ This heuristic argument is easily made and hence of β. rigorous once one notes βˆ is consistent (i.e., for any є > , limn→∞ P [∥βˆ − β∥ > є] = ; see Cox and Hinkley ). In this example, the outcome Yi = ∀i
A
A
Asymptotic Relative Efficiency in Estimation
has positive probability, and for such {Y , . . . , Yn }, βˆ is infinite. A similar result holds for Yi = ∀i. Hence the variance of βˆ does not exist.
About the Author John Kolassa is Professor of Statistics and Biostatistics, and Director of the Graduate Programs in Statistics and Biostatistics at Rutgers University. John Kolassa was previously Assistant and Associate Professor in the Department of Biostatistics, and Graduate Student Advisor, at the University of Rochester. John Kolassa is an Elected Fellow of the ASA and IMS, and has received grants from and served on grant review panels for the National Science Foundation and the National Institutes of Health.
Cross References 7Approximations to Distributions 7Asymptotic Relative Efficiency in Testing 7Central Limit Theorems 7Chernoff-Savage Theorem 7Limit Theorems of Probability Theory 7Martingale Central Limit Theorem 7Properties of Estimators 7Sampling Problems for Stochastic Processes 7Statistical Estimation of Actuarial Risk Measures for
Heavy-Tailed Claim Amounts
References and Further Reading Cox DR, Hinkley DV () Theoretical statistics. Chapman and Hall, New York Hettmansperger TP () Statistical inference based on ranks. Krieger, Melbourne, FL Serfling RJ () Approximation theorems of mathematical statistics. Wiley, New York
Asymptotic Relative Efficiency in Estimation Robert Serfling Professor University of Texas at Dallas, Richardson, TX, USA
Asymptotic Relative Efficiency of Two Estimators For statistical estimation problems, it is typical and even desirable that several reasonable estimators can arise for consideration. For example, the mean and median parameters of a symmetric distribution coincide, and so the sample
mean and the sample median become competing estimators of the point of symmetry. Which is preferred? By what criteria shall we make a choice? One natural and time-honored approach is simply to compare the sample sizes at which two competing estimators meet a given standard of performance. This depends upon the chosen measure of performance and upon the particular population distribution F. To make the discussion of sample mean versus sample median more precise, consider a distribution function F with density function f symmetric about an unknown point θ to be estimated. For {X , . . . , Xn } a sample from F, put X n = n− ∑ni= Xi and Medn = median{X , . . . , Xn }. Each of X n and Medn is a consistent estimator of θ in the sense of convergence in probability to θ as the sample size n → ∞. To choose between these estimators we need to use further information about their performance. In this regard, one key aspect is efficiency, which answers: How spread out about θ is the sampling distribution of the estimator? The smaller the variance in its sampling distribution, the more “efficient” is that estimator. Here we consider “large-sample” sampling distributions. For X n , the classical central limit theorem (see 7Central Limit Theorems) tells us: if F has finite variance σF , then the sampling distribution of X n is approximately N (θ, σF /n), i.e., Normal with mean θ and variance σF /n. For Medn , a similar classical result (Serfling ) tells us: if the density f is continuous and positive at θ, then the sampling distribution of Medn is approximately N(θ, /[f (θ)] n). On this basis, we consider X n and Medn to perform equivalently at respective sample sizes n and n if σF = . n [f (θ)] n Keeping in mind that these sampling distributions are only approximations assuming that n and n are “large,” we define the asymptotic relative efficiency (ARE) of Med to X as the large-sample limit of the ratio n /n , i.e., ARE (Med, X, F) = [f (θ)] σF .
()
Definition in the General Case For any parameter η of a distribution F, and for estimators ̂ η() and ̂ η() approximately N(η, V (F)/n) and N(η, V (F)/n), respectively, the ARE of ̂ η() to ̂ η() is given by V (F) . () ARE (̂ η() , ̂ η() , F) = V (F) Interpretation. If ̂ η() is used with a sample of size n, the number of observations needed for ̂ η() to perform equiv() () alently is ARE(̂ η ,̂ η , F) × n.
Asymptotic Relative Efficiency in Estimation
Extension to the case of multidimensional parameter. For a parameter η taking values in Rk , and two estimators ̂ η(i) which are k-variate Normal with mean η and nonsingular covariance matrices Σ i (F)/n, i = , , we use [see Serfling ()] ARE (̂ η() , ̂ η() , F) = (
∣Σ (F)∣ ) ∣Σ (F)∣
/k
,
()
the ratio of generalized variances (determinants of the covariance matrices), raised to the power /k.
Connection with the Maximum Likelihood Estimator Let F have density f (x ∣ η) parameterized by η ∈ R and satisfying some differentiability conditions with respect to η. Suppose also that I(F) =
∂ Eη {[ ∂η
log f (x ∣ η)] } (the
Fisher information) is positive and finite. Then (Lehmann and Casella ) it follows that (a) the maximum likelihood estimator η̂ (ML) of η is approximately N(η, /I(F)n), and (b) for a wide class of estimators ̂ η that are approximately N(η, V(̂ η, F)/n), a lower bound to V(̂ η, F) is /I(F). In this situation, () yields ARE (̂ η, ̂ η(ML) , F) =
≤ , I(F)V(̂ η, F)
()
making ̂ η (ML) (asymptotically) the most efficient among the given class of estimators ̂ η. We note, however, as will be discussed later, that () does not necessarily make ̂ η(ML) the estimator of choice, when certain other considerations are taken into account.
Detailed Discussion of Estimation of Point of Symmetry Let us now discuss in detail the example treated above, with F a distribution with density f symmetric about an unknown point θ and {X , . . . , Xn } a sample from F. For estimation of θ, we will consider not only X n and Medn but also a third important estimator.
Mean versus Median Let us now formally compare X n and Medn and see how the ARE differs with choice of F. Using () with F = N (θ, σF ), it is seen that ARE (Med, X, N (θ, σF )) = /π = .. Thus, for sampling from a Normal distribution, the sample mean performs as efficiently as the sample median using only % as many observations. (Since θ and σF are location and scale parameters of F, and since the estimators
A
X n and Medn are location and scale equivariant, their ARE does not depend upon these parameters.) The superiority of X n here is no surprise since it is the MLE of θ in the model N (θ, σF ). As noted above, asymptotic relative efficiencies pertain to large sample comparisons and need not reliably indicate small sample performance. In particular, for F Normal, the exact relative efficiency of Med to X for sample size n = is a very high %, although this decreases quickly, to % for n = , to % for n = , and to % in the limit. For sampling from a double exponential (or Laplace) distribution with density f (x) = λe−λ∣x−θ∣ /, −∞ < x < ∞ (and thus variance /λ ), the above result favoring X n over Medn is reversed: () yields ARE (Med, X, Laplace) = , so that the sample mean requires % as many observations to perform equivalently to the sample median. Again, this is no surprise because for this model the MLE of θ is Medn .
A Compromise: The Hodges–Lehmann Location Estimator We see from the above that the ARE depends dramatically upon the shape of the density f and thus must be used cautiously as a benchmark. For Normal versus Laplace, X n is either greatly superior or greatly inferior to Medn . This is a rather unsatisfactory situation, since in practice we might not be quite sure whether F is Normal or Laplace or some other type. A very interesting solution to this dilemma is given by an estimator that has excellent overall performance, the so-called Hodges–Lehmann location estimator (Hodges and Lehmann ; see 7Hodges-Lehmann Estimators): Xi + Xj }, HLn = Median { the median of all pairwise averages of the sample observations. (Some authors include the cases i = j, some not.) We have (Lehmann a) that HLn is asymptotically N(θ, /[∫ f (x)dx] n), which yields that ARE (HL, X, N (θ, σF )) = /π = . and ARE(HL, X, Laplace) = .. Also, for the 7Logistic distribution with density f (x) = σ − e(x−θ)/σ /[ + e(x−θ)/σ ] , −∞ < x < ∞, for which HLn is the MLE of θ and thus optimal, we have ARE(HL, X, Logistic) = π / = . [see Lehmann (b)]. Further, for F the class of all distributions symmetric about θ and having finite variance, we have inf F ARE(HL, X, F) = / = . [see Lehmann (a)]. The estimator HLn is highly competitive with X at Normal distributions, can be infinitely more efficient at some other symmetric distributions F, and is never much
A
A
Asymptotic Relative Efficiency in Estimation
less efficient at any distribution F in F. The computation of HLn appears at first glance to require O(n ) steps, but a much more efficient O(n log n) algorithm is available [see Monohan ()].
Efficiency versus Robustness Trade-Off Although the asymptotically most efficient estimator is given by the MLE, the particular MLE depends upon the shape of F and can be drastically inefficient when the actual F departs even a little bit from the nominal F. For example, if the assumed F is N(µ, ) but the actual model differs by a small amount ε of “contamination,” i.e., F = (−ε)N(µ, )+ εN(µ, σ ), then ARE(Med, X, F) =
( − ε + εσ − ) ( − ε + εσ ) , π
which equals /π in the “ideal” case ε = but otherwise → ∞ as σ → ∞. A small perturbation of the assumed model thus can destroy the superiority of the MLE. One way around this issue is to take a nonparametric approach and seek an estimator with ARE satisfying a favorable lower bound. Above we saw how the estimator HLn meets this need. Another criterion by which to evaluate and compare estimators is robustness. Here let us use finite-sample breakdown point (BP): the minimal fraction of sample points which may be taken to a limit L (e.g., ±∞) without the estimator also being taken to L. A robust estimator remains stable and effective when in fact the sample is only partly from the nominal distribution F and contains some non-F observations which might be relatively extreme contaminants. A single observation taken to ∞ (with n fixed) takes X n with it, so X n has BP = . Its optimality at Normal distributions comes at the price of a complete sacrifice of robustness. In comparison, Medn has extremely favorable BP = . but at the price of a considerable loss of efficiency at Normal models. On the other hand, the estimator HLn appeals broadly, possessing both quite high ARE over a wide class of F and relatively high BP = − −/ = .. As another example, consider the problem of estimation of scale. Two classical scale estimators are the sample standard deviation sn and the sample MAD (median absolute deviation about the median) MADn . They estimate scale in different ways but can be regarded as competitors in the problem of estimation of σ in the model F = N(µ, σ ), as follows. With both µ and σ unknown, the estimator sn is (essentially) the MLE of σ and is asymptotically most efficient. Also, for this F, the population MAD is equal to Φ− (/)σ, so that the estimator ̂ σn =
MADn /Φ− (/) = . MADn competes with sn for estimation of σ. (Here Φ denotes the standard normal distribution function, and, for any F, F − (p) denotes the pth quantile, inf{x : F(x) ≥ p}, for < p < .) To compare with respect to robustness, we note that a single observation taken to ∞ (with n fixed) takes sn with it, sn has BP = . On the other hand, MADn and thus ̂ σn have BP = ., like Medn . However, ARE(̂ σn , sn , N(µ, σ )) = ., even worse than the ARE of Medn relative to X. Clearly desired is a more balanced trade-off between efficiency and robustness than provided by either of sn and ̂ σn . Alternative scale estimators having the same . BP as ̂ σn but much higher ARE of . relative to sn are developed in Rousseeuw and Croux (). Also, further competitors offering a range of trade-offs given by (BP, ARE) = (., .) or (., .) or (., .), for example, are developed in Serfling (). In general, efficiency and robustness trade off against each other. Thus ARE should be considered in conjunction with robustness, choosing the balance appropriate to the particular application context. This theme is prominent in the many examples treated in Staudte and Sheather ().
A Few Additional Aspects of ARE Connections with Confidence Intervals In view of the asymptotic normal distribution underlying the above formulation of ARE in estimation, we may also characterize the ARE given by () as the limiting ratio of sample sizes at which the lengths of associated confidence intervals at approximate level ( − α)%, √ Vi (F) α (i) − ̂ , i = , , η ± Φ ( − ) ni converge to at the same rate, when holding fixed the coverage probability − α. (In practice, of course, consistent estimates of Vi (F), i = , , are used in forming the CI.)
Fixed Width Confidence Intervals and ARE One may alternatively consider confidence intervals of fixed length, in which case (under typical conditions) the noncoverage probability depends on n and tends to at an exponential rate, i.e., n− log α n → c > , as n → ∞. For fixed width confidence intervals of the form ̂ η (i) ± d σF , i = , , we thus define the fixed width asymptotic relative efficiency (FWARE) of two estimators as the limiting ratio of sample (i) sizes at which the respective noncoverage probabilities α n , i = , , of the associated fixed width confidence intervals
Asymptotic Relative Efficiency in Estimation
converge to zero at the same exponential rate. In particular, for Med versus X, and letting η = and σF = without loss of generality, we obtain (Serfling and Wackerly ) FWARE(Med, X, F) =
log m(−d) , log[(F(d) − F (d))/ ]
()
where m(−d) is a certain parameter of the 7moment generating function of F. The FWARE is derived using large deviation theory instead of the central limit theorem. As d → , the FWARE converges to the ARE. Indeed, for F a Normal distribution, this convergence (to /π = .) is quite rapid: the expression in () rounds to . for d = , to . for d = , and to . for d ≤ ..
Confidence Ellipsoids and ARE For an estimator ̂ η which is asymptotically k-variate Normal with mean η and covariance matrix Σ/n, as the sample size n → ∞, we may form (see Serfling ) an associated ellipsoidal confidence region of approximate level ( − α)% for the parameter η, En,α = {η : n (̂ η − η)′ Σ− (̂ η − η) ≤ cα }, with P (χ k > cα ) = α and in practice using a consistent estimate of Σ. The volume of the region En,α is π k/ (cα /n)k/ ∣Σ∣/ . Γ((k + )/) Therefore, for two such estimators ̂ η(i) , i = , , the ARE given by () may be characterized as the limiting ratio of sample sizes at which the volumes of associated ellipsoidal confidence regions at approximate level ( − α)% converge to at the same rate, when holding fixed the coverage probability − α. Under regularity conditions on the model, the maximum likelihood estimator ̂ η(ML) has a confidence ellipsoid En,α attaining the smallest possible volume and, moreover, lying wholly within that for any other estimator ̂ η.
Connections with Testing Parallel to ARE in estimation as developed here is the notion of Pitman ARE for comparison of two hypothesis test procedures. Based on a different formulation, although the central limit theorem is used in common, the Pitman ARE agrees with () when the estimator and the hypothesis test statistic are linked, as for example X paired with the t-test, or Medn paired with the 7sign test, or HLn paired with the 7Wilcoxon-signed-rank test. See Lehmann b, Nikitin , Nikitin , and Serfling .
A
Other Notions of ARE As illustrated above with FWARE, several other important approaches to ARE have been developed, typically using either moderate or large deviation theory. For example, instead of asymptotic variance parameters as the criterion, one may compare probability concentrations of the estimators in an ε-neighborhood of the target parameter η: P (∣̂ η(i) − η∣ > ε), i = , . When we have (i)
log P(∣̂ η n − η∣ > ε) → γ (i) (ε, η), i = , , n as is typical, then the ratio of sample sizes n /n at which these concentration probabilities converge to at the same rate is given by γ () (ε, η)/γ () (ε, η), which then represents () another ARE measure for the efficiency of estimator ̂ ηn () relative to ̂ η n . See Serfling (, ..) for discussion and Basu () for illustration that the variance-based and concentration-based measures need not agree on which estimator is better. For general treatments, see Nikitin (), Puhalskii and Spokoiny (), Nikitin (), and Serfling (, Chap. ), as well as the other references cited below. A comprehensive bibliography is beyond the present scope. However, very productive is ad hoc exploration of the literature using a modern search engine.
Acknowledgments Support by NSF Grant DMS- and NSA Grant H--- is gratefully acknowledged.
About the Author Robert Serfling is author of the classic textbook Approximation Theorems of Mathematical Statistics, Wiley, , and has published extensively in statistical journals. He received a Humboldt-Preis, awarded by the Alexander von Humboldt Stiftung, Germany, “in recognition of accomplishments in research and teaching” (). He is a Fellow of the American Statistical Association and of Institute of Mathematical Statistics, and an Elected Member of International Statistical Institute. Professor Serfling was Editor of the IMS Lecture Notes Monograph Series (–) and currently is an Associate Editor for Journal of Multivariate Analysis (–) and for Journal of Nonparametric Statistics (–).
Cross References 7Asymptotic Relative Efficiency in Testing 7Estimation 7Estimation: An Overview 7Mean Median and Mode 7Normality Tests
A
A
Asymptotic Relative Efficiency in Testing
7Properties of Estimators 7Statistical Fallacies: Misconceptions, and Myths
References and Further Reading Basu D () On the concept of asymptotic relative efficiency. Sankhy¯a :– Hodges JL, Lehmann EL () Estimates of location based on rank tests. Ann Math Stat :– Lehmann EL (a) Elements of large-sample theory. Springer, New York Lehmann EL (b) Nonparametrics: statistical methods based on ranks. Prentice-Hall, Upper Saddle River, NJ Lehmann EL, Casella G () Theory of point estimation, nd edn. Springer, New York Monohan JF () Algorithm : fast computation of the Hodges– Lehmann location estimator. ACM T Math Software :– Nikitin Y () Asymptotic efficiency of nonparametric tests. Cambridge University Press, Cambridge Nikitin Y () Asymptotic relative efficiency in testing. International Encyclopedia of Statistical Sciences. Springer, New York Puhalskii A, Spokoiny V () On large-deviation efficiency in statistical inference. Bernoulli :– Rousseeuw PJ, Croux C () Alternatives to the median absolute deviation. J Am Stat Assoc :– Serfling R () Approximation Theorems of Mathematical Statistics. Wiley, New York Serfling R () Efficient and robust fitting of lognormal distributions. N Am Actuarial J :– Serfling R, Wackerly DD () Asymptotic theory of sequential fixed-width confidence interval procedures. J Am Stat Assoc :– Staudte RG, Sheather SJ () Robust estimation and testing. Wiley, New York
Asymptotic Relative Efficiency in Testing Yakov Nikitin Professor, Chair of Probability and Statistics St. Petersburg University, St. Petersburg, Russia
Asymptotic Relative Efficiency of Two Tests Making a substantiated choice of the most efficient statistical test of several ones being at the disposal of the statistician is regarded as one of the basic problems of Statistics. This problem became especially important in the middle of XX century when appeared computationally simple but “inefficient” rank tests. Asymptotic relative efficiency (ARE) is a notion which enables to implement in large samples the quantitative comparison of two different tests used for testing of the same statistical hypothesis. The notion of the asymptotic
efficiency of tests is more complicated than that of asymptotic efficiency of estimates. Various approaches to this notion were identified only in late forties and early fifties, hence, – years later than in the estimation theory. We proceed now to their description. Let {Tn } and {Vn } be two sequences of statistics based on n observations and assigned for testing the nullhypothesis H against the alternative A. We assume that the alternative is characterized by real parameter θ and for θ = θ turns into H. Denote by NT (α, β, θ) the sample size necessary for the sequence {Tn } in order to attain the power β under the level α and the alternative value of parameter θ. The number NV (α, β, θ) is defined in the same way. It is natural to prefer the sequence with smaller N. Therefore the relative efficiency of the sequence {Tn } with respect to the sequence {Vn } is specified as the quantity eT,V (α, β, θ) = NV (α, β, θ)/NT (α, β, θ) , so that it is the reciprocal ratio of sample sizes NT and NV . The merits of the relative efficiency as means for comparing the tests are universally acknowledged. Unfortunately it is extremely difficult to explicitly compute NT (α, β, θ) even for the simplest sequences of statistics {Tn }. At present it is recognized that there is a possibility to avoid this difficulty by calculating the limiting values eT,V (α, β, θ) as θ → θ , as α → and as β → keeping two other parameters fixed. These limiting values ePT,V , eBT,V and eHL T,V are called respectively the Pitman, Bahadur and Hodges–Lehmann asymptotic relative efficiency (ARE), they were proposed correspondingly in Pitman (), Bahadur () and Hodges and Lehmann (). Only close alternatives, high powers and small levels are of the most interest from the practical point of view. It keeps one assured that the knowledge of these ARE types will facilitate comparing concurrent tests, thus producing well-founded application recommendations. The calculation of the mentioned three basic types of efficiency is not easy, see the description of theory and many examples in Serfling (), Nikitin () and Van der Vaart (). We only mention here, that Pitman efficiency is based on the central limit theorem (see 7Central Limit Theorems) for test statistics. On the contrary, Bahadur efficiency requires the large deviation asymptotics of test statistics under the null-hypothesis, while Hodges–Lehmann efficiency is connected with large deviation asymptotics under the alternative. Each type of efficiency has its own merits and drawbacks.
Asymptotic Relative Efficiency in Testing
Pitman Efficiency Pitman efficiency is the classical notion used most often for the asymptotic comparison of various tests. Under some regularity conditions assuming 7asymptotic normality of test statistics under H and A, it is a number which has been gradually calculated for numerous pairs of tests. We quote now as an example one of the first Pitman’s results that stimulated the development of nonparametric statistics. Consider the two-sample problem when under the null-hypothesis both samples have the same continuous distribution and under the alternative differ only in P location. Let eW, t be the Pitman ARE of the two-sample Wilcoxon rank sum test (see 7Wilcoxon–Mann–Whitney Test) with respect to the corresponding Student test (see 7Student’s t-Tests). Pitman proved that for Gaussian samP ples eW, t = /π ≈ . , and it shows that the ARE of the Wilcoxon test in the comparison with the Student test (being optimal in this problem) is unexpectedly high. Later Hodges and Lehmann () proved that . ≤ ePW, t ≤ + ∞ , if one rejects the assumption of normality and, moreover, the lower bound is attained at the density √ ⎧ ⎪ ⎪ ( − x )/ ( ) f (x) = ⎨ ⎪ ⎪ ⎩
if ∣x∣ ≤
√
,
otherwise.
Hence the Wilcoxon rank test can be infinitely better than the parametric test of Student but their ARE never falls below .. See analogous results in Serfling () where the calculation of ARE of related estimators is discussed. Another example is the comparison of independence tests based on Spearman and Pearson correlation coefficients in bivariate normal samples. Then the value of Pitman efficiency is /π ≈ .. In numerical comparisons, the Pitman efficiency appear to be more relevant for moderate sample sizes than other efficiencies Groeneboom and Oosterhoff (). On the other hand, Pitman ARE can be insufficient for the comparison of tests. Suppose, for instance, that we have a normally distributed sample with the mean θ and variance and we are testing H : θ = against A : θ > . Let compare two significance tests based on the sample mean ¯ and the Student ratio t. As the t-test does not use the X information on the known variance, it should be inferior to the optimal test using the sample mean. However, from the point of view of Pitman efficiency, these two tests are equivalent. On the contrary, Bahadur efficiency eBt,X¯ (θ) is strictly less than for any θ > .
A
If the condition of asymptotic normality fails, considerable difficulties arise when calculating the Pitman ARE as the latter may not at all exist or may depend on α and β. Usually one considers limiting Pitman ARE as α → . Wieand () has established the correspondence between this kind of ARE and the limiting approximate Bahadur efficiency which is easy to calculate.
Bahadur Efficiency The Bahadur approach proposed in Bahadur (; ) to measuring the ARE prescribes one to fix the power of tests and to compare the exponential rate of decrease of their sizes for the increasing number of observations and fixed alternative. This exponential rate for a sequence of statistics {Tn } is usually proportional to some non-random function cT (θ) depending on the alternative parameter θ which is called the exact slope of the sequence B (θ) of two sequences {Tn }. The Bahadur ARE eV,T of statistics {Vn } and {Tn } is defined by means of the formula B eV,T (θ) = cV (θ) / cT (θ). It is known that for the calculation of exact slopes it is necessary to determine the large deviation asymptotics of a sequence {Tn } under the null-hypothesis. This problem is always nontrivial, and the calculation of Bahadur efficiency heavily depends on advancements in large deviation theory, see Dembo and Zeitouni () and Deuschel and Stroock (). It is important to note that there exists an upper bound for exact slopes cT (θ) ≤ K(θ) in terms of Kullback–Leibler information number K(θ) which measures the “statistical distance” between the alternative and the null-hypothesis. It is sometimes compared in the literature with the 7Cramér–Rao inequality in the estimation theory. Therefore the absolute (nonrelative) Bahadur efficiency of the sequence {Tn } can be defined as eBT (θ) = cT (θ)/K(θ). It is proved that under some regularity conditions the likelihood ratio statistic is asymptotically optimal in Bahadur sense (Bahadur ; Van der Vaart , Sect. .; Arcones ). Often the exact Bahadur ARE is uncomputable for any alternative θ but it is possible to calculate the limit of Bahadur ARE as θ approaches the null-hypothesis. Then one speaks about the local Bahadur efficiency. The indisputable merit of Bahadur efficiency consists in that it can be calculated for statistics with non-normal asymptotic distribution such as Kolmogorov-Smirnov, omega-square, Watson and many other statistics.
A
A
Asymptotic Relative Efficiency in Testing
Asymptotic Relative Efficiency in Testing. Table Some local Bahadur efficiencies Statistic Distribution Hyperbolic Gauss Logistic Laplace cosine Cauchy Gumbel Dn
.
.
.
.
.
ωn
.
.
.
.
.
Consider, for instance, the sample with the distribution function (df) F and suppose we are testing the goodnessof-fit hypothesis H : F = F for some known continuous df F against the alternative of location. Well-known distribution-free statistics for this hypothesis are the Kolmogorov statistic Dn and omega-square statistic ω n . The following table presents their local absolute efficiency in case of six standard underlying distributions: We see from Table that the integral statistic ω n is in most cases preferable with respect to the supremum-type statistic Dn . However, in the case of Laplace distribution the Kolmogorov statistic is locally optimal, the same happens for the Cramér-von Mises statistic in the case of hyperbolic cosine distribution. This observation can be explained in the framework of Bahadur local optimality, see Nikitin ( Chap. ). See also Nikitin () for the calculation of local Bahadur efficiencies in case of many other statistics.
Hodges–Lehmann efficiency This type of the ARE proposed in Hodges and Lehmann () is in the conformity with the classical NeymanPearson approach. In contrast with Bahadur efficiency, let us fix the level of tests and let compare the exponential rate of decrease of their second-kind errors for the increasing number of observations and fixed alternative. This exponential rate for a sequence of statistics {Tn } is measured by some non-random function dT (θ) which is called the Hodges–Lehmann index of the sequence {Tn }. For two such sequences the Hodges–Lehmann ARE is equal to the ratio of corresponding indices. The computation of Hodges–Lehmann indices is difficult as requires large deviation asymptotics of test statistics under the alternative. There exists an upper bound for the Hodges–Lehmann indices analogous to the upper bound for Bahadur exact slopes. As in the Bahadur theory the sequence of statistics {Tn } is said to be asymptotically optimal in the Hodges– Lehmann sense if this upper bound is attained. The drawback of Hodges–Lehmann efficiency is that most two-sided tests like Kolmogorov and Cramér-von Mises tests are asymptotically optimal, and hence this kind
of efficiency cannot discriminate between them. On the other hand, under some regularity conditions the onesided tests like linear rank tests can be compared on the basis of their indices, and their Hodges–Lehmann efficiency coincides locally with Bahadur efficiency, see details in Nikitin (). Coupled with three “basic” approaches to the ARE calculation described above, intermediate approaches are also possible if the transition to the limit occurs simultaneously for two parameters at a controlled way. Thus emerged the Chernoff ARE introduced by Chernoff (), see also Kallenberg (); the intermediate, or the Kallenberg ARE introduced by Kallenberg (), and the Borovkov– Mogulskii ARE, proposed in Borovkov and Mogulskii (). Large deviation approach to asymptotic efficiency of tests was applied in recent years to more general problems. For instance, the change-point, “signal plus white noise” and regression problems were treated in Puhalskii and Spokoiny (), the tests for spectral density of a stationary process were discussed in Kakizawa (), while Taniguchi () deals with the time series problems, and Otsu () studies the empirical likelihood for testing moment condition models.
About the Author Professor Nikitin is Chair of Probability and Statistics of St. Petersburg University. He is an Associate editor of Statistics and Probability Letters, and member of the editorial Board, Mathematical Methods of Statistics and Metron. He is a Fellow of the Institute of Mathematical Statistics. Professor Nikitin is the author of the text Asymptotic efficiency of nonparametric tests, Cambridge University Press, NY, , and has authored more than papers, in many international journals, in the field of Asymptotic efficiency of statistical tests, large deviations of test statistics and nonparametric Statistics.
Cross References 7Asymptotic Relative Efficiency in Estimation 7Chernoff-Savage Theorem 7Nonparametric Statistical Inference 7Robust Inference
References and Further Reading Arcones M () Bahadur efficiency of the likelihood ratio test. Math Method Stat :– Bahadur RR () Stochastic comparison of tests. Ann Math Stat :– Bahadur RR () Rates of convergence of estimates and test statistics. Ann Math Stat :–
Asymptotic, Higher Order
Borovkov A, Mogulskii A () Large deviations and testing of statistical hypotheses. Siberian Adv Math (, ); (, ) Chernoff H () A measure of asymptotic efficiency for tests of a hypothesis based on sums of observations. Ann Math Stat :– Dembo A, Zeitouni O () Large deviations techniques and applications, nd edn. Springer, New York Deuschel J-D, Stroock D () Large deviations. Academic, Boston Groeneboom P, Oosterhoff J () Bahadur efficiency and small sample efficiency. Int Stat Rev :– Hodges J, Lehmann EL () The efficiency of some nonparametric competitors of the t-test. Ann Math Stat :– Kakizawa Y () Bahadur exact slopes of some tests for spectral densities. J Nonparametric Stat :– Kallenberg WCM () Intermediate efficiency, theory and examples. Ann Stat :– Kallenberg WCM () Chernoff efficiency and deficiency. Ann Stat :– Nikitin Y () Asymptotic efficiency of nonparametric tests. Cambridge University Press, Cambridge Otsu T () On Bahadur efficiency of empirical likelihood. J Econ :– Pitman EJG () Lecture notes on nonparametric statistical inference. Columbia University, Mimeographed Puhalskii A, Spokoiny V () On large-deviation efficiency in statistical inference. Bernoulli :– Serfling R () Approximation theorems of mathematical statistics. Wiley, New York Serfling R () Asymptotic relative efficiency in estimation. In: Lovric M (ed) International encyclopedia of statistical sciences. Springer Taniguchi M () On large deviation asymptotics of some tests in time series. J Stat Plann Inf :– Van der Vaart AW () Asymptotic statistics. Cambridge University Press, Cambridge Wieand HS () A condition under which the Pitman and Bahadur approaches to efficiency coincide. Ann Statist :–
A
truly approximating a target quantity up to the claimed degree of accuracy. Classical asymptotics is based on the notion of asymptotic distribution, often derived from the central limit theorem (see 7Central Limit Theorems), and usually the approximations are correct up to O(n−/ ), where n is the sample size. Higher order asymptotics provides refinements based on asymptotic expansions of the distribution or density function of an estimator of a parameter. They are rooted in the Edgeworth theory, which is itself a refinement of the central limit theorem. The theory of higher order asymptotic is very much related with the corresponding to Approximations to distributions treated as well in this Encyclopedia. When higher order asymptotic is correct up to o(n−/ ), it is second order asymptotic. When further terms are picked up, so that the asymptotic is correct up to o(n− ), it is third order asymptotic. In his pioneering papers, C. R. Rao coined the term second order efficiency for a concept that would now is called third order efficiency. The new terminology is essentially owing to Pfanzagl and Takeuchi.
About the Author Professor Abril is co-editor of the Revista de la Sociedad Argentina de Estadística (Journal of the Argentinean Statistical Society).
Cross References 7Approximations to Distributions 7Edgeworth Expansion
References and Further Reading
Asymptotic, Higher Order Juan Carlos Abril President of the Argentinean Statistical Society, Professor Universidad Nacional de Tucumán and Consejo Nacional de Investigaciones Científicas y Técnicas, San Miguel de Tucumán, Argentina Higher order asymptotic deals with two sorts of closely related things. First, there are questions of approximation. One is concerned with expansions or inequalities for a distribution function. Second, there are inferential issues. These involve, among other things, the application of the ideas of the study of higher order efficiency, admissibility and minimaxity. In the matter of expansions, it is as important to have usable, explicit formulas as a rigorous proof that the expansions are valid in the sense of
Abril JC () Asymptotic expansions for time series problems with applications to moving average models. PhD thesis. The London School of Economics and Political Science, University of London, England Barndorff-Nielsen O, Cox DR () Edgeworth and saddle-point approximations with statistical applications. J R Stat Soc B :– Daniels HE () Saddlepoint approximations in statistics. Ann Math Stat :– Durbin J () Approximations for the densities of sufficient estimates. Biometrika :– Feller W () An introduction to probability theory and its applications, vol , nd edn. Wiley, New York Ghosh JK () Higher order Asymptotic. NSF-CBMS Regional Conference Series in Probability and Statistics, . Hayward and Alexandria: Institute of Mathematical Statistics and American Statistical Association Pfanzagl J () Asymptotic expansions in parametric statistical theory. In: Krishnaiah PR (ed) Developments in statistics, vol. . Academic, New York, pp – Rao CR () Asymptotic efficiency and limiting information. In Proceedings of Fourth Berkeley Symposium on Mathematical
A
A
Autocorrelation in Regression
Statistics and Probability. Berkeley: University of California Press, pp – Rao CR () Efficient estimates and optimum inference procedure in large samples. J R Stat Soc B :– Rao CR () Criteria of estimation in large samples. Sankhya B :– Rao CR () Linear statistical inference and its applications, nd edn. Wiley, New York Wallace DL () Asymptotic approximations to distributions. Ann Math Stat :–
Autocorrelation in Regression Badi H. Baltagi Distinguished Professor of Economics Syracuse University, Syracuse, NY, USA Linear regressions are a useful empirical tool for economists and social scientists and the standard 7least squares estimates are popular because they are the best linear unbiased estimators (BLUE) under some albeit strict assumptions. These assumptions require the regression disturbances not to be correlated with the regressors, also homoskedastic, i.e., with constant variance, and not autocorrelated. Violation of the no autocorrelation assumption on the disturbances, will lead to inefficiency of the least squares estimates, i.e., no longer having the smallest variance among all linear unbiased estimators. It also leads to wrong standard errors for the regression coefficient estimates. This in turn leads to wrong t-statistics on the significance of these regression coefficients and misleading statistical inference based on a wrong estimate of the variance–covariance matrix computed under the assumption of no autocorrelation. This is why standard regression packages have a robust heteroskedasticity and autocorrelation-consistent covariance matrix (HAC) option for these regressions which at least robustifies the standard errors of least squares and shows how sensitive they would be to such violations, see Newey and West (). Autocorrelation is more likely to occur in time-series than in cross-section studies. Consider estimating the consumption function of a random sample of households. An unexpected event, like a visit of family members will increase the consumption of this household. However, this positive disturbance need not be correlated with the disturbances affecting consumption of other randomly drawn households. However, if we were estimating this consumption function using aggregate time-series data for the U.S., then it is very likely that a recession year affecting consumption negatively that year, may have a carry over effect to the next few years. A shock to the economy like an oil
embargo in is likely to affect the economy for several years. A labor strike this year may affect production for the next few years. The simplest work horse for illustrating this autocorrelation in time series on the regression disturbances, say ut is the first-order autoregressive process denoted by AR(): ut = ρut− + є t
t = , , . . . , T
where є t is independent and identically distributed with mean and variance σє . It is autoregressive because ut is related to its lagged value ut− . One can show, see for example Baltagi (), that the correlation coefficient between ut and ut− is ρ. Also, that the correlation coefficient between ut and ut−r is ρ r . When ρ = , there is no autocorrelation and one test for this null hypothesis is the Durbin and Watson () test discussed as a separate entry in this encyclopedia by Krämer. This AR() process is also stationary as long as ∣ ρ ∣< . If ρ = , then this process has a unit root and it is called a 7random walk. See the entry by Dickey on testing for this unit root using the 7Dickey-Fuller tests. Note that if the process is stationary, then ρ is a fraction and the correlation for two disturbances r periods apart is ρ r , i.e., a fraction raised to an integer power. This means that the correlation is decaying between the disturbances the further apart they are. This is reasonable in economics and may be the reason why this AR() form is so popular. One should note that this is not the only form that would correlate the disturbances across time. Other forms like the Moving Average (MA) process, and higher order Autoregressive Moving Average (ARMA) processes are popular, see Box and Jenkins (), but these are beyond the scope of this entry. Since least squares is no longer BLUE under autocorrelation of the disturbances, Cochrane and Orcutt () suggested a simple estimator that corrects for autocorrelation of the AR() type. This method starts with an initial estimate of ρ, the most convenient is , and minimizes the residual sum of squares of the regression. This gives us the least squares estimates of the regression coefficients and the corresponding least squares residuals which we denote by et . In the next step, one regresses et on et− to get an estimate of ρ, say ̂ ρ. The second step of the Cochrane–Orcutt procedure (SCO) is to perform the regression of (Yt − ̂ ρYt− ) on (Xt − ̂ ρ Xt− ) to get estimates of the regression coefficients. One can iterate this procedure (ITCO) until convergence. Both the SCO and the ITCO are asymptotically efficient as the sample size gets large. The argument for iterating must be justified in terms of small sample gains. Other methods of correcting for serial correlation include Prais and Winsten (), Durbin (), as well as maximum likelihood methods, all studied more extensively in Chap. of Baltagi (). The Prais and
Axioms of Probability
Winsten method recaptures the initial observation lost in the Cochrane–Orcutt method. Monte Carlo studies using an autoregressive regressor, and various values of ρ, found that least squares is still a viable estimator as long as ∣ρ∣ < ., but if ∣ρ∣ > ., then it pays to perform procedures that correct for serial correlation based on an estimator of ρ. For trended regressors, which is usually the case with economic data, least squares outperforms SCO, but not the Prais-Winsten procedure that recaptures the initial observation. In fact, Park and Mitchell () who performed an extensive Monte Carlo using trended and untrended regressors recommend that one should not use regressions based on (T − ) observations as in the Cochrane and Orcutt procedure. They also found that test of hypotheses regarding the regression coefficients performed miserably for all estimators based on an estimator of ρ. Correcting for serial correlation is not without its critics. Mizon () argues this point forcefully in his article entitled “A simple message for autocorrelation correctors: Don’t.” The main point being that serial correlation is a symptom of dynamic misspecification which is better represented using a general unrestricted dynamic specification.
About the Author Badi H. Baltagi is distinguished Professor of Economics, and Senior Research Associate at the Center for Policy Research, Syracuse University. He received his Ph.D. in Economics at the University of Pennsylvania in . He served on the faculty at the University of Houston and Texas A&M University. He was a visiting Professor at the University of Arizona and the University of California, San Diego. He is the author of Econometric Analysis of Panel Data (Wiley, th edn. ); Econometrics (Springer, th edn. ), and editor of A Companion to Theoretical Econometrics (Wiley–Blackwell); Recent Developments in the Econometrics of Panel Data, Volumes I and II (Edward-Elgar); Nonstationary Panels, Panel Cointegration, and Dynamic Panels (Elsevier); Panel Data Econometrics: Theoretical Contributions and Empirical Applications (Physica-Verlag, ); Spatial Econometrics: Methods and Applications, (with Giuseppe Arbia), Physica-Verlag, . He is author or co-author of over publications, all in leading economics and statistics journals. Professor Baltagi was the holder of the George Summey, Jr. Professor Chair in Liberal Arts and was awarded the Distinguished Achievement Award in Research at Texas A&M University. He is co-editor of Empirical Economics, and Associate editor of Journal of Econometrics and Econometric Reviews. He is the replication editor of the Journal of Applied Econometrics and the series editor for Contributions to Economic Analysis. He is a fellow of the Journal of Econometrics and
A
a recipient of the Multa and Plura Scripsit Awards from Econometric Theory. He is a founding fellow and member of the Board of Directors of the Spatial Econometrics Association.
Cross References 7Approximations to Distributions 7Box–Jenkins Time Series Models 7Correlation Coefficient 7Dickey-Fuller Tests 7Durbin–Watson Test 7Linear Regression Models 7Structural Time Series Models 7Tests of Independence 7Time Series 7Time Series Regression
References and Further Reading Baltagi BH () Econometrics, th edn., Springer, Berlin Box GEP, Jenkins GM () Time series analysis, forecasting and control. Holden Day, San Francisco Cochrane D, Orcutt G () Application of least squares regression to relationships containing autocorrelated error terms. J Am Stat Assoc :– Durbin J () Estimation of parameters in time-series regression model. J R Stat Soc, B, :– Durbin J, Watson G () Testing for serial correlation in least squares regression-II. Biometrika :– Mizon GE () A simple message for autocorrelation correctors: don’t. J Econometrics :– Newey WK, West KD () A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica :– Park RE, Mitchell BM () Estimating the autocorrelated error model with trended data. J Econometrics :– Prais S, Winsten C () Trend estimation and serial correlation. Discussion Paper , Cowles Commission, Chicago
Axioms of Probability Vincenzo Capasso Professor of Probability and Mathematical Statistics University of Milan, Milan, Italy
Ingredients of Probability Spaces Definition A collection F of subsets of a set Ω is called a ring on Ω if it satisfies the following conditions: . A, B ∈ F ⇒ A ∪ B ∈ F, . A, B ∈ F ⇒ A ∖ B ∈ F. A ring F is called an algebra if Ω ∈ F.
A
A
Axioms of Probability
Definition A ring F on Ω is called a σ-ring if it satisfies the following additional condition:
the axioms of probability according to the Kolmogorov approach [see Kolmogorov ()].
.
Definition Given a set Ω, and a σ-algebra F of subsets of Ω, a probability measure on F is any function P : F → [, ] such that
For every countable family (An )n∈N of subsets of F: ⋃n∈N An ∈ F.
A σ-ring F on Ω is called a σ-algebra (or σ-field) if Ω ∈ F. Proposition The following properties hold: . If F is a σ-algebra of subsets of a set Ω, then it is an algebra. . If F is a σ-algebra of subsets of Ω, then ● For any countable family (En )n∈N∖{} of elements of F: ⋂∞ n= En ∈ F, ● For any finite family (Ei )≤i≤n of elements of F : n ⋂i= Ei ∈ F, ● B ∈ F ⇒ Ω ∖ B ∈ F. Definition Every pair (Ω, F) consisting of a set Ω and a σ-ring F of subsets of Ω is a measurable space. Furthermore, if F is a σ-algebra, then (Ω, F) is a measurable space on which a probability measure can be built. Example . Generated σ-algebra. If A is a family of subsets of a set Ω, then there exists the smallest σ-algebra of subsets of Ω that contains A. This is the σ-algebra generated by A, denoted by σ(A). If, now, G is the set of all σ-algebras of subsets of Ω containing A, then it is not empty because it has the set P(Ω) of all subsets of Ω, among its elements, so that σ(A) = ⋂C∈G C. . Borel σ-algebra. Let Ω be a topological space. Then the Borel σ-algebra on Ω, denoted by BΩ , is the σ-algebra generated by the set of all open subsets of Ω. Its elements are called Borel sets. . Product σ-algebra. Let (Ω i , Fi )≤i≤n be a family of measurable spaces, with all Fi , ≤ i ≤ n, σ-algebras, and let Ω = ∏ni= Ω i . Defining n
R = {E ⊂ Ω∣∀i = , . . . , n ∃Ei ∈ Fi such that E = ∏Ei}, i=
the σ-algebra on Ω generated by R is called the product σ-algebra of the σ-algebras (Fi )≤i≤n . Proposition Let (Ω i )≤i≤n be a family of topological spaces with a countable base and let Ω = ∏ni= Ω i . Then the Borel σ-algebra BΩ is identical to the product σ-algebra of the family of Borel σ-algebras (BΩ i )≤i≤n .
Axioms of Probability We assume that the reader is already familiar with the basic motivations and notions of probability theory. We present
P . P(Ω) = , P . for any countable family A , . . . , An , . . . of elements of F such that Ai ∩ Aj = /, whenever i ≠ j: P (⋃ An ) = ∑ P(An ). n
n
Definition A probability space is an ordered triple (Ω, F, P), where Ω is a set, F is a σ-algebra of subsets of Ω, and P : F → [, ] is a probability measure on F. The set Ω is called the sample space, the elements of F are called events. Definition A probability space (Ω, F, P) is finite if Ω has finitely many elements. Remark If Ω is at most countable, then it is usual to assume that F = P(Ω), the σ-algebra of all subsets of Ω. In this case all sets {ω} reduced to sample points ω ∈ Ω are events; they are called elementary events. Remark If the σ−algebra of events F is finite, then the requirement of countable additivity in the definition of the probability measure P can be reduced to finite additivity. Remark It is worth mentioning that an important alternative approach to probability theory is the so called subjective probability; this approach does not insist on Axiom P ., and rather uses the finite version of it (De Finetti –). Definition A finite probability space (Ω, F, P) with F = P(Ω) is an equiprobable or uniform space, if ∀ω ∈ Ω :
P({ω}) = k (constant);
i.e., its elementary events are equiprobable. Remark Following the axioms of a probability space and the definition of a uniform space, if (Ω, F, P) is equiprobable, then , ∀ω ∈ Ω : P({ω}) = ∣Ω∣ where ∣⋅∣ denotes the cardinal number of elementary events in Ω, and ∣A∣ . ∀A ∈ F ≡ P(Ω) : P(A) = ∣Ω∣ Intuitively, in this case we may say that P(A) is the ratio of the number of favorable outcomes, divided by the number of all possible outcomes.
Axioms of Probability
Example Consider an urn that contains balls, of which are red and are black but that are otherwise identical, from which a player draws a ball. Define the event R: The first drawn ball is red. Then P(R) =
∣R∣ = = .. ∣Ω∣
Definition We shall call any event F ∈ F such that P(F) = , a null event. Elementary consequences of the above definitions are the following ones. Proposition Let (Ω, F, P) be is a probability space. . . . . .
P(Ac ) = − P(A), for any A ∈ F; P(/) = ; If A, B ∈ F, A ⊆ B, then P(B) = P(A) + P(B ∖ A); If A, B ∈ F, A ⊆ B, then P(A) ≤ P(B) (monotonicity); If A, B ∈ F, then P(B ∖ A) = P(B) − P(B ∩ A) P(A ∪ B) = P(A) + P(B) − P(A ∩ B);
. .
If A, B ∈ F, A ⊆ B, then P(B ∖ A) = P(B) − P(A); (Principle of inclusion-exclusion) Let A , . . . , An ∈ F, then P (∪ni= Ai )
= ∑ni= P(Ai ) − ∑i<j P(Ai ∩ Aj ) + . . . + (−)n+ P(A ∩ . . . ∩ An );
.
If A , . . . , An ∈ F, then n
n
P (⋃ Ai ) ≤ ∑ P(Ai ). i=
i=
About the Author Fellow of the European Academy of Sciences (–), President of ECMI (the European Consortium for Mathematics in Industry) (–); Founder () and
A
President of the European Society for Mathematical and Theoretical Biology (–); Founder () and Director of SASIAM: “School for Advanced Studies in Industrial and Applied Mathematics”, Tecnopolis, Bari, Italy (–), Founder and Director of MIRIAM (Milan Research Centre for Industrial and Applied Mathematics) (–) and later of ADAMSS (Research Centre for Advanced Applied Mathematical and Statistical Sciences) of the University of Milano (–), Director of CIMAB (InterUniversity Centre for Mathematics Applied to Biology, Medicine, Environment, etc.) (–).
Cross References 7Foundations of Probability 7Imprecise Probability 7Measure Theory in Probability 7Philosophy of Probability 7Probability Theory: An Outline
References and Further Reading Ash RB () Real analysis and probability. Academic, London Bauer H () Probability theory and elements of measure theory. Academic, London Billingsley P () Probability and measure. Wiley, New York Breiman L () Probability. Addison–Wesley, Reading, MA Chung KL () A course in probability theory, nd edn. Academic, New York De Finetti B (–) Theory of probability, vols and . Wiley, London Dudley RM () Real analysis and probability. Cambridge Studies in Advanced Mathematics , Cambridge University Press, Cambridge Fristedt B, Gray L () A modern approach to probability theory. Birkhäuser, Boston Kolmogorov AN () Foundations of the theory of probability. Chelsea, New York Métivier M () Notions fondamentales de la théorie des probabilités., Dunod, Paris
A
B Balanced Sampling Yves Tillé Professor University of Neuchâtel, Neuchâtel, Switzerland
Balanced sampling is a random method of selection of units from a population that provides a sample such that the Horvitz–Thompson estimators (see 7Horvitz– Thompson Estimator) of the totals are the same or almost the same as the true population totals for a set of control variables. More precisely, let U = {, . . . , k, . . . , N} be a finite population and s ⊂ U a sample or a subset of U. A sampling design p(s) is a probability distribution on the set of all the subsets s ⊂ U, i.e. p(s) ≥ and ∑ p(s) = . s⊂U
The inclusion probability π k = pr(k ∈ s) of a unit k is its probability of being selected in the sample s. Consider a variable of interest y that takes the value yk on unit k. Let also Y be the total of the values taken by y on the units of the population Y = ∑ yk . k∈U
If π k > , for all k ∈ U, the Horvitz–Thompson () estimator ̂ = ∑ yk Y k∈S
πk
is unbiased for Y. Let also x , . . . , xj , . . . , xJ be a sequence of auxiliary variables whose the values xk , . . . , xkj , . . . , xkJ are known for each unit k of the population. According to the definition given in Tillé (), a sampling design is said to be balanced on the x variables if ∑ k∈S
x kj πk
≈ ∑ xkj , for j = , . . . , J. k∈U
Balanced sampling involves sampling with fixed sample size and stratification. Indeed, a stratified design is balanced on
the indicator variables of the strata, because the Horvitz– Thompson estimators of the sizes of the strata are equal to the population sizes of the strata. In the design-based inference, balanced sampling allows a strong improvement of the efficiency of the Horvitz–Thompson estimator when the auxiliary variables are correlated with the interest variable (see Deville and Tillé ). In the model-based inference, balanced samples are advocated to protect under miss-specification of the model (see Valliant et al. ). Balanced sampling must not be confused with a representative sample. Representativity is a vague concept that usually means that some groups have the same proportions in the sample and in the population. This definition is fallacious because some groups can be over or underrepresent in a sample to obtain a more accurate unbiased estimator. Moreover, balanced sampling implies that the sample is randomly selected and that predefined inclusion probabilities, that can be unequal, are satisfied at least approximately. There exists a large family of methods for selecting balanced samples. The first one was probably proposed by Yates () and consists of selecting a sample by a simple random sampling and next eventually changing some units to get a more balanced sample. Other methods, called rejective, consist of selecting several samples with an initial sampling design until obtaining a sample that is well balanced. Rejective methods however have the drawback that the inclusion probabilities of the rejective design are not the same as the ones of the initial design and are generally impossible to compute. The cube method proposed by Deville and Tillé () is a general multivariate algorithm for selecting balanced samples that can use several decades of auxiliary variables with equal of unequal inclusion probabilities. The cube method exactly satisfies the predefined inclusion probabilities and provides a sample that is balanced as well as possible. SAS and R language implementations are available.
About the Author For biography see the entry 7Sampling Algorithms.
Cross References 7Horvitz–Thompson Estimator 7Representative Samples
Miodrag Lovric (ed.), International Encyclopedia of Statistical Science, DOI ./----, © Springer-Verlag Berlin Heidelberg
B
Banking, Statistics in
7Sample Survey Methods 7Sampling Algorithms 7Stratified Sampling
References and Further Reading Deville J-C, Tillé Y () Efficient balanced sampling: the cube method. Biometrika :– Horvitz DG, Thompson DJ () A generalization of sampling without replacement from a finite universe. J Am Stat Assoc :– Tillé Y () Sampling algorithms. Springer, New York Valliant R, Dorfman AH, Royall RM () Finite population sampling and inference: a prediction approach. Wiley, New York Yates F () Sampling methods for censuses and surveys. Griffin, London
Banking, Statistics in Héctor Manuel Zárate Solano Professor, Head of Statistics Section Universidad Nacional de Colombia, Banco de la República Bogotá, Colombia Statistics in banking is becoming increasingly important to the extent that modern banking at all levels would be impossible without the application of statistical methods (Hand , p. ). Particularly the financial crisis has underlined the importance of high quality and timely data to the banking sector. Indeed, “information shortages – directly related to a lack of timely and accurate data – with regard to complex credit derivatives lie at the heart of the current crisis and have complicated the crisis management efforts of central banks around the world.” (Current challenges and the Future of Central Bank Statistics). According to the Monetary and Financial Statistics (International Monetary Fund ), the banking sector consists of all resident corporations mainly engaged in financial intermediation in any given economy. These corporations consist of the central bank which is the national financial institution with the principal responsibility of ensuring monetary stability and financial sector soundness (Bangura ), and other depository corporations including commercial retail banks, merchant banks, savings banks, credit unions, credit cooperatives, rural and agricultural banks, etc. The range of statistical tools and models in banking applications is vast, starting from purely descriptive models, risk assessment, statistics of extreme values, Markov
chain approaches (including mover–stayer models), cluster analysis (see 7Cluster Analysis: An Introduction), statistical methods of fraud detection (see Bolton and Hand ; Hand , ; Sudjianto et al. ), 7logistic regression, classification trees, etc.
Statistics in Banking: The Colombian Perspective The Central Bank has focused significantly on the use of statistics to conduct monetary policy. Statistical methodologies have evolved in quantity, depth, and degree of sophistication as economic and financial systems have become more complex because of the generalized economic growth over time, increased market participants, and all sorts of financial transactions. Moreover, the adoption annual inflation-targeting, based on the consumer price index (CPI), to achieve a long-term GDP growth trend as the monetary policy rule since has set new challenges for our statistics section and practitioners as well. As a result, we have had to adapt to new demands for information. First, the inflation forecasting model of the Banco de la República requires monitoring the inflationary pressures continually, and second, assessing the economic activity. To accomplish these, econometric models based on observational data have been put into place, and the Bank has made efforts in using best practices with respect to surveys as an alternative to full reporting. To elaborate on the above, we need to produce statistics in an efficient way, including a new form of data analysis provided by individual reports coming from financial and nonfinancial institutions mostly through electronic reporting. The degree of sophistication of statistical techniques depends on the data availability. A priority task involves data compilation, which is a statistical activity that requires the implementation of new ideas that combine statistical techniques, view points from economists, and the management of large databases. For instance, nominal short-term interest rates have remained as a main policy tool in this country, which depend on the reliability of the micro data reported daily to the Bank. We have to establish graphical and statistical tools to guarantee the quality and accuracy of the aggregate data. Aggregation also involves the incorporation of new methodologies, such as the index number theory, which explains the behavior of crucial indicators over time in carrying out monetary policy management. Consequently, we have worked on building core inflation indices, forward looking indicators, stock indices, real estate prices, and real exchange rates. Furthermore, one important part of the transmission mechanism is to know how monetary policy affects aggregate demand and it has
Banking, Statistics in
been important to identify shocks of trade terms, fiscal policy, and real world price of commodities. With regard to surveys, we should note that the recent increased responses of financial markets, and others, to economic expectations. Thus, the Bank has made several surveys protecting statistical confidentiality: The Monthly Business Surveys, which provide clues on formation expectations in the private sector and fill the need for more timely indicators that can be used to gain insight into the economic climate before official statistics are published. Furthermore, they indicate signals of turning points in economic activity. The regional aspect of this survey also creates network contacts in the business community. Another survey is the Monthly Expert Inflation Forecasts, which summarize short macroeconomic forecasts from the experts. The statistics derived from the latter provide relevant information on the professional inflation consensus and allow us to assess the credibility of monetary policy through the dispersion of the forecast of the inflation. In addition, we developed Quarterly Market Expectation Surveys to ascertain the entrepreneurship perception of current and expected economic developments over the very short term in the main macroeconomic variables. We are also working on surveys involved with remittance flows and the cost of remitting money. Nevertheless, the realization of surveys has imposed central bank statisticians do serious efforts on issues related to statistical survey methodologies. For instance, introducing good questionnaire design, using probabilistic sampling techniques, and assuring adequate response rates. Also, we have to notice difficulty of survey implementation as there is a lack of incentives for respondent to cooperate, absence of proper legal mandate to collect information, and resources constraints. These factors are considered through the cost-benefit evaluation of the generation of this kind of information. Statistics is decisive to monetary policy decisions by central bank policy makers, who face diverse forms of uncertainty, and it is important to rely on various information sources assessment of economic statistics and judgment. In fact, they need to analyze financial markets, watch carefully national accounts and labor force statistics. This statistical challenge has intensified cooperation with the national statistical institute (DANE) and producers of statistics in the private sector to try to improve data coherence, minimize the response burden to reporting agents, exchange micro data, share methodologies, and ameliorate data quality under international recognized standards. In addition, the experience from other countries has led to coordinate regional tasks in order to homogenize key economic indicators. It is also of paramount
B
importance for the central bank’s credibility to keep the public clearly informed about the actions that have been taken by the monetary authorities. As a result, new ways of presenting statistics have been implemented in different channels of communication such as Inflation Report, Report to Congress, and Press releases, among others. Statistics contribute to understand trends in the economy policy evaluations and help to design future policies. Therefore, the ability to research inflation issues and find recent statistical methodologies to present new economic indicators mainly where information is insufficient is a challenge for central bank statisticians. For example, it is actually required to understand the service sector trends, the behavior of flexible labor markets, and how the population’s quality of life has improved.
Acknowledgment Disclaimer: The views on the above do not necessarily reflect those of the Bank or the University.
Cross References 7Business Statistics 7Copulas in Finance 7Financial Return Distributions 7Quantitative Risk Management 7Risk Analysis 7Semi-Variance in Finance 7Statistical Modeling of Financial Markets
References and Further Reading Bangura SAF () Statistical information and the banking sector. In: Fourth meeting of the committee on development of information. United Nations Conference centre, Addis Ababa, Ethiopia Bolton RJ, Hand DJ () Statistical fraud detection: a review. Stat Sci ():– Current challenges and the future of central bank statistics. http://web.incisive- events.com/rma///central- banking/ rmcb_statistics- course—st.pdf Hand DJ () Statistics in banking. In: Encyclopedia of statistical sciences, nd edn., vol . Wiley, Hoboken, p Hand DJ () Statistical techniques for fraud detection, prevention, and evaluation, video lecture. http://videolectures.net/ mmdsshandstf/ Hand DJ () Fraud detection in telecommunications and banking: discussion of Becker, Volinsky, and Wilks () and Sudjianto et al. Technometrics :– International Monetary Fund () Monetary and financial statistics manual. IMF, Washington Sudjianto A, Nair S, Yuan M, Zhang A, Kern D, Cela-Díaz F () Statistical methods for fighting financial crimes. Technometrics :–
B
B
Bartlett and Bartlett-Type Corrections
Bartlett and Bartlett-Type Corrections Gauss M. Cordeiro Professor Universidade Federal Rural de Pernambuco, Recife, Brazil
Bartlett Correction The log-likelihood ratio (LR) statistic is one of the most commonly used statistics for inference in parametric models. Let w be the LR statistic for testing some composite or simple null hypothesis H against an alternative hypothesis H. In regular problems, the null distribution of w is asymptotically χ q , where q is the difference between the dimensions of the parameter spaces under the two hypotheses tested. However, as the samples sizes decreases, the use of such a statistic becomes less justifiable. One way of improving the χ approximation to the LR statistic is by multiplying w by a correction factor c known as the Bartlett correction (Lawley ; Hayakawa ; Cordeiro ). This idea was pioneered by Bartlett () and later put into a general framework by Lawley (). Bartlett obtained a number of these corrections in the area of multivariate analysis in several papers published between and , and these corrections became widely used for improving the large-sample χ approximation to the null distribution of w. Bartlett () used the following approach to improve the χ approximation to the null distribution of w. Suppose that, under the null hypothesis H , E(w) is calculated up to order n− , say E(w) = q + b + O(n− ), where b is a constant of order n− and n is the number of observations or some related quantity. Specifically, the Bartlett correction is determined by the relation c = (+b/q) and it represents an important tool for improving the χ approximation for w. The corrected statistic w∗ = w/c has an expected value that comes closer to that of χ q than does the expected value of w. Moreover, for continuous data, the distribution of w∗ is, in general, closer to χ q than is the distribution of w. Box () used Bartlett’s approach to investigate in detail the general expression for the moments of the statistic w in the following cases: the test of constancy of variance and covariance of k sets of p-variate samples and Wilk’s test for the independence of k sets of residuals, the ith set having pi variables. He showed, at least for these cases, that the modified statistic w∗ follows a χ q distribution more closely than does the unmodified statistic w. Box’s results are applicable to all tests for which the Laplace transform of the test statistic can be explicitly written in terms of gamma functions and reciprocal gamma functions.
A general method to obtain Bartlett corrections for regular statistical models was developed in full generality by Lawley (), who gave a general formula for the correction c as function of covariant tensors. He derived expressions for the moments of certain derivatives of the log-likelihood function, and, via an exceedingly complicated calculation, obtained a general formula for the null expected value of w. Further, he showed that all cumulants of the corrected statistic w∗ for testing composite hypotheses agree with those of the reference χ q distribution with error of order O(n− ) [see Hayakawa () and Cordeiro ()]. Calculations of the Bartlett corrections via Lawley’s approach are, however, notoriously cumbersome since they involve substantial effort into computing certain joint cumulants of log-likelihood derivatives. See also Eqs. .–. in Barndorff-Nielsen and Cox’s () book. A further step on the improvement of the statistic w was taken by Hayakawa (), who derived an asymptotic expansion for the null distribution of w for testing a composite null hypothesis H against a composite alternative hypothesis H. He showed that to order O(n− ) Pr(w ≤ z) = Fq (z) + (n)− [A Fq+ (z) − (A − A )Fq+ (z) + (A − A )Fq (z)], () where Fs (⋅) is the cumulative distribution function of a χ random variable with s degrees of freedom. Here, A is a function of expected values of the first four log-likelihood derivatives and of the first two derivatives of these expected values with respect to the parameters of the model and its expression holds for both simple and composite hypotheses, thus allowing for nuisance parameters. When nuisance parameters are present, A can be calculated as the difference between two identical functions evaluated under the null and alternative hypotheses. The error in Equation is O(n− ) and not O(n−/ ) as it is sometimes reported. However, the Bartlett correction is given by c = + (nq)− A , which differs from the one obtained from the above expansion, unless A = . This points to a conflict between Hayakawa’s and Lawley’s results. The answer to this puzzle came with papers by Harris () and Cordeiro (). Harris showed that A should not be present in (), whereas Cordeiro showed that the term A is always equal to zero. The main contribution of Equation with A = is that it provides a relatively simple demonstration that w∗ = w/c has a χ q distribution with error O(n− ). In fact, Cordeiro () demonstrated that the simple correction of the first moment of w to order O(n− ), quite generally has the effect of eliminating the term of
Bartlett and Bartlett-Type Corrections
this order in the asymptotic expansion of the corrected statistic w∗ . This result was a starting point for numerous subsequent research efforts in the direction of establishing several expressions for Bartlett corrections in various classes of statistical models. One difficulty encountered with the use of w∗ rather than w is the fact that the required expectation may be very difficult or even impossible to compute. A general matrix formula for c was derived by Cordeiro (a), which has advantages for numerical and analytical purposes. For continuous data the effect of the Bartlett correction is amazingly good even for very small sample sizes. However, for discrete data, the Bartlett correction does not in general yield an improvement in the asymptotic error rate of the chi-squared approximation. Several papers have focused on deriving these corrections for special regression models by using matrix formulae for specific models, bypassing the traditional machinery of calculating these cumulants. We can always obtain these matrix formulae when the joint cumulants of log-likelihood derivatives are invariant under permutation of parameters. The matrix formulae in conjunction with computer algebra systems (Mathematica or Maple for example) and programming languages with support for matrix operations (Gauss, R and Ox) represent a computationally much simpler way of deriving Bartlett corrections in rather general classes of statistical models. Several papers have focused on deriving matrix formulae for Bartlett corrections in general classes of regression models. Cordeiro (, ) described Bartlett’s approach for univariate 7generalized linear models. Corrected LR statistics for exponential family nonlinear models were obtained by Cordeiro and Paula (). They gave general matrix expressions for Bartlett corrections in these models involving an unpleasant looking quantity which may be regarded as a measure of nonlinearity of the systematic component of the model. Attfield () and Cordeiro (b) showed how to correct LR statistics for heteroskedasticity. An algorithm for computing Bartlett corrections was given by Andrews and Stafford (). Cordeiro et al. () proposed matrix formulae for Bartlett corrections in dispersion models. Cordeiro () presented extensive simulation results on the performance of the corrected statistic w∗ in generalized linear models focusing on gamma and log-linear models. For a detailed account of the applicability of Bartlett corrections, see Cribari-Neto and Cordeiro (). Barndorff-Nielsen and Cox () gave an indirect method for computing Bartlett corrections under rather general parametric models by establishing a simple connection between the correction term b and the norming constants of the general expression for the conditional
B
distribution of the maximum likelihood estimator, namely q n , where A and A are the normalized conb = ( AA ) π stants of the general formula for the density of the maximum likelihood estimator conditional on an exact or approximate ancillary statistic when this formula is applied to the full and null models, respectively. It is usually easier to obtain the Bartlett correction for special cases using Lawley’s formula than using Barndorff-Nielsen and Cox’s expression, since the former involves only moments of log-likelihood derivatives whereas the latter requires exact or approximate computation of the conditional distribution of the maximum likelihood estimator. When there are many nuisance parameters, it may not be easy to obtain ancillary statistics for these parameters, and hence the evaluation of Barndorff-Nielsen and Cox’s formula can be quite cumbersome. The constants A and A are usually functions of the maximal ancillary statistic, although to the relevant order of magnitude, w∗ is independent of the ancillary statistic selected. They have also obtained various expressions for these quantities and, in particular, an approximation which does not require integration over the sample space for the one-parameter case. Since the statistic w is invariant under reparametrization, it is possible to obtain a large sample expansion for this statistic and its expectation in terms of invariants. McCullagh and Cox () used this fact to represent the Bartlett correction as a function of invariant combinations of cumulants of the first two log-likelihood derivatives and gave it a geometric interpretation in full generality in terms of the model curvature. It is also noteworthy that McCullagh and Cox’s () general formula coincides with Lawley’s () formula. The advantage of McCullagh and Cox’s formula is its geometric interpretation, whereas the main advantage of Lawley’s result is that it can be more easily implemented to obtain Bartlett corrections for special models.
Bartlett-Type Correction The problem of developing a correction similar to the Bartlett correction to other test statistics, such as the score (S) and Wald (W) statistics, was posed by Cox () and addressed three years later in full generality by Cordeiro and Ferrari (), and by Chandra and Mukerjee () and Taniguchi () for certain special cases. We shall focus on Cordeiro and Ferrari’s results since they are more general in the sense that they allow for nuisance parameters. The score test has been widely used in statistics because it has an advantage over other large sample tests such as the LR and Wald tests. Thus, while these tests involve
B
B
Bartlett and Bartlett-Type Corrections
estimation of the parameters under the alternative hypothesis, the score test requires estimation only under the null hypothesis. The LR, Wald and score statistics have the same chi-squared distribution asymptotically. Harris () obtained an asymptotic expansion for the null distribution of the score statistic S to order O(n− ) as Pr(S ≤ z) = Fq (z) + (n)− [A Fq+ (z) + (A − A )Fq+ (z) + (A − A + A )Fq+ (z) + (A − A + A )Fq (z)], () where A , A and A are functions of some cumulants of log-likelihood derivatives. The general expressions for these coefficients are given in Harris’ paper. He showed that the first three cumulants of the score statistic are (to order n− ) given by κ = q + A /(n), κ = q + (A + A )/(n) and κ = q + (A + A + A )/n. Equation holds for both simple and composite hypotheses. More importantly, this result implies that there exists no scalar transformation based on the score statistic which corrects all cumulants to a certain order of precision, as it is the case with the Bartlett correction to the LR statistic. The coefficients A’s can be used to obtain corrections for models based on independent, but not necessarily identically distributed observations, thus covering a number of linear and nonlinear regression models (see 7Nonlinear Regression). From Eq. , Cordeiro and Ferrari () proposed a Bartlett-type correction to improve the score statistic. They defined a modified score statistic having a chi-squared distribution to order O(n− ) under the null hypothesis ⎛ j− ⎞ , S∗ = S − ∑ γ j S ⎠ ⎝ n j=
()
where γ = (A −A +A )/(q), γ = (A −A )/{q(q+ )} and γ = A /{q(q + )(q + )}. They demonstrated that S∗ is distributed as χ q when terms of order smaller than O(n− ) are neglected. When the A’s involve unknown parameters they should be replaced by their maximum likelihood estimates under the null hypothesis but this does not affect the order of approximation of the correction. The correction factor in () is a function of the unmodified statistic, and hence this correction is not a “Bartlett correction” in the classical sense. Given its similarity with the Bartlett correction, however, it is termed “Bartlett-type correction.” Cordeiro and Ferrari () obtained a more general result to be applied to any test statistic which converges to χ which can be described as follows. Let S be a test statistic
which is asymptotically distributed as χ q . Chandra () showed, under mild regularity conditions, that Pr(S ≤ z) = Fq (z) +
k ∑ ai Fq+i (z) n i=
()
when terms of order O(n− ) or smaller are neglected. Equation implies that the distribution function to O(n− ) of a test statistic asymptotically distributed as chi-squared is, under certain conditions, a linear combination of chisquareds with degrees of freedom q, q + , . . . , q + k. The coefficients a’s are linear functions of cumulants of log-likelihood derivatives for a general test statistic T. For LR and SR , k = and k = , respectively, where the a’s are linear functions of the A’s in Eqs. and . Let µ′i = i Γ(i + q/)/{Γ(q/)} be the ith moment about zero of the χ q distribution, where Γ(⋅) is the gamma function. Cordeiro and Ferrari () demonstrated that the modified test statistic ⎧ ⎫ k ⎛ k ⎪ ⎪ ⎞ ⎪ ⎪ − S∗ = S ⎨ − ∑ ∑ aj (µ ′i ) Si− ⎬ ⎪ ⎪ ⎝ ⎠ ⎪ ⎪ i= j= ⎩ ⎭ − is distributed as χ q to order O(n ). This is a very general result which can be used to improve many important tests in econometrics and statistics. Building upon the result described above, Cordeiro et al. () and Cribari-Neto and Ferrari () obtained Bartlett-type corrections to improve score tests in generalized linear models with known and unknown dispersion parameter, respectively. Cordeiro and Ferrari () demonstrated gave matrix formula for computing Bartletttype corrections to improve score tests in a general statistical model and in exponential family nonlinear models, respectively. Finally, Cordeiro and Ferrari () derived Bartlett-type corrections for chi-squared statistics based on the calculation of their moments.
About the Author Gauss M. Cordeiro was President of the Brazilian Statistical Association (–). He has published more than research articles in international journals. He was one of the founding editors of the Brazilian Journal of Probability and Statistics and the Editor in Chief of this journal (–). He was awarded the “Prêmio ABE” award from the Brazilian Statistical Association for his contributions to statistics.
Cross References 7Bias Correction
References and Further Reading Andrews D, Stafford JE () Tools for the symbolic computation of asymptotic expansions. J R Stat Soc B :–
Bartlett’s Test
Attfield CLF () A Bartlett-adjustment to the likelihood ratio test for homoskedasticity in the linear model. Econ Lett : – Barndorff-Nielsen OE, Cox DR () Bartlett adjustments to the likelihood ratio statistic and the distribution of the maximum likelihood estimator. J R Stat Soc B :– Barndorff-Nielsen OE, Cox DR () Inference and asymptotics. Chapman and Hall, London Bartlett MS () Properties of sufficiency and statistical tests. Proc R Soc A :– Box GEP () A general distribution theory for a class of likelihood criteria. Biometrika :– Chandra TK () Asymptotic expansions of perturbed chi-square variables. Sankhyã A :– Chandra TK, Mukerjee R () Bartlett-type modification for Rao’s efficient score statistic. J Multivariate Anal :– Cordeiro GM () Improved likelihood ratio statistics for generalized linear models. J R Stat Soc B :– Cordeiro GM () On the corrections to the likelihood ratio statistics. Biometrika :– Cordeiro GM (a) General matrix formula for computing Bartlett corrections. Stat Probab Lett :– Cordeiro GM (b) Bartlett corrections and bias correction for two heteroscedastic regression models. Comm Stat Theor :– Cordeiro GM () Performance of a Bartlett-type modification for the deviance. J Stat Comput Sim :– Cordeiro GM, Ferrari SLP () A modified score statistic having chi-squared distribution to order n − . Biometrika :– Cordeiro GM, Ferrari SLP () A note on Bartlett-type corrections for the first few moments of test statistics. J Stat Plan Infer :– Cordeiro GM, Paula GA () Improved likelihood ratio statistics for exponential family nonlinear models. Biometrika :– Cordeiro GM, Ferrari SLP, Paula GA () Improved score tests for generalized linear models. J R Stat Soc B :– Cordeiro GM, Paula GA, Botter DA () Improved likelihood ratio tests for dispersion models. Int Stat Rev :– Cox DR () Some aspects of conditional and asymptotic inference: a review. Sankhyã A :– Cribari-Neto F, Cordeiro GM () On Bartlett and Bartlett-type corrections. Econom Rev :– Cribari-Neto F, Ferrari SLP () Second order asymptotics for score tests in generalized linear models. Biometrika :– Ferrari SLP, Cordeiro GM () Matrix formulae for computing improved score tests. J Stat Comput Sim :– Ferrari SLP, Cordeiro GM () Corrected score tests for exponential family nonlinear models. Stat Probab Lett :– Harris P () An asymptotic expansion for the null distribution of the efficient score statistic. Biometrika :– Harris P () A note on Bartlett adjustments to likelihood ratio tests. Biometrika :– Hayakawa T () The likelihood ratio criterion and the asymptotic expansion of its distribution. Ann Inst Stat Math A :– Lawley DN () A general method for approximating to the distribution of the likelihood ratio criteria. Biometrika :– McCullagh P, Cox DR () Invariants and likelihood ratio statistics. Ann Stat :– Taniguchi M () Third-order asymptotic properties of a class of test statistics under a local alternative. J Multivariate Anal :–
B
Bartlett’s Test Hossein Arsham , Miodrag Lovric Harry Wright Distinguished Research Professor of Statistics and Management Science University of Baltimore, Baltimore, MD, USA Professor University of Kragujevac, Kragujevac, Serbia
Bartlett’s test (introduced in by Maurice Barlett (– )) is an inferential procedure used to assess the equality of variance in different populations (not in samples as sometimes can be found, since there is no point in testing whether the samples have equal variances – we can always easily calculate and compare their values). Some common statistical methods assume that variances of the populations from which different samples are drawn are equal. Bartlett’s test assesses this assumption. It tests the null hypothesis that the population variances are equal. All statistical procedures have underlying assumptions. In some cases, violation of these assumptions will not change substantive research conclusions. In other cases, violation of assumptions is critical to meaningful research. Establishing that one’s data meet the assumptions of the procedure one is using is an expected component of all quantitatively based journal articles, theses, and dissertations. The following are the two general areas where Bartlett’s test is applicable: . In regression analysis and time series analyses, the residuals should have a constant variance (i.e., homoskedastic condition). One may check this condition by dividing the residuals data into two or more groups, then using the Bartlett’s test. . Another area of application is the F-test in ANOVA that requires the assumption that each underlying population has the same variance (i.e., homogeneity condition). Bartlett’s test is derived from the likelihood ratio test under the normal distribution. Therefore, it is dependent on meeting the assumption of normality. Bartlett’s test of homogeneity of variance is based on a chi-square statistic with (k−) degrees of freedom, where k is the number of categories (or groups) in the independent variable. In other words, Bartlett’s test is used to test if k populations have equal variances. We wish to test the null hypothesis: H : σ = σ = . . . = σk
B
B
Bartlett’s Test
against the alternative that at least two population variances are not equal. The following briefly explains the procedure employed by Bartlett’s test. To investigate the significance of the differences between the variances of k normally distributed populations, independent samples are drawn from each of the populations. Let Sj denote the variance of a sample of nj items from the jth population (j = , . . . , k). The test statistic has the following expression: ⎛ k (n − ) s ⎞ ∑ i i ⎟ k ⎜ i= ⎟ − ∑ (ni − ) ln (si ) (N − k) ln ⎜ ⎟ ⎜ N −k ⎟ i= ⎜ ⎠ ⎝ B= +
k [( ∑ )− ] (k − ) i= ni − N −k
where N corresponds to the sum of all sample sizes. It is asymptotically distributed as a χ distribution with (k − ) degrees of freedom. The null hypothesis of equal population variances is rejected if test statistics is larger than the critical value. One may also use online tools to perform this test, under condition that each sample contains at least five observations. Bartlett’s test is known to be powerful if the underlying populations are normal. According to some recent results based on simulation (Legendre and Borcard), Bartlett’s test and Box’s test are the best overall methods to test the homogeneity of variances. With non-normal data, Bartlett’s and Box’s tests can be used if the samples are fairly large. It was shown that Bartlett’s test is unbiased (Pitman ) and consistent (Brown ). One of its major drawbacks, however, is that it is extremely sensitive (that is non-robust) to the departures from normality (Box ). Since in reality heterogeneity and non-normality frequently simultaneously occur, when the null hypothesis is rejected, we cannot know whether this is due to unequal population variances or non-normality, or both. This test is so sensitive to departures from normality that Box commented that it well may be used as a good test of the population normality. Box also remarked that using Bartlett’s test to check whether it is appropriate to apply ANOVA would be rather like “putting a rowing boat to sea to find out whether conditions are sufficiently calm for an ocean liner to leave a port” (Box , p ). Finally, for the unequal sample-sizes case, several approaches have been advised for finding exact critical values (see, for example, Chao and Glaser () and Manoukian et al. ()).
About the Author Dr. Hossein Arsham is The Wright Distinguished Research Professor in Decision Science and Statistics. He has created one of the most comprehensive sites on statistical analysis on the Internet. Currently, he is Associate editor for International Journal of Ecological Economics and Statistics, International Journal of Statistics and Systems, and Journal of Interdisciplinary Mathematics. He is a Fellow of The Royal Statistical Society (), The Operational Research Society (), the Institute of Combinatorics and Its Applications (), and The World Innovation Foundation: Scientific Discovery (). Professor Arsham has received numerous awards for his teaching.
Cross References 7Analysis of Variance Model, Effects of Departures from
Assumptions Underlying 7Tests for Homogeneity of Variance
References and Further Reading Arsham H () The P-values for Chi-square distribution. http:// home.ubalt.edu/ntsbarsh/Business- stat/otherapplets/pvalues. htm#rchdist Arsham H () Homogeneity of Multi-variances: the Bartlett’s test. http://home.ubalt.edu/ntsbarsh/Business- stat/otherapplets/Bar tletTest.htm Box GEP () Non-normality and tests on variances. Biometrika :– Brown GW () On the power of the L test for equality of several variances. Ann Math Stat :– Chao M-T, Glaser RE () The exact distribution of Bartlett’s test statistic for homogeneity of variances with unequal sample sizes. JASA ():– Legendre P, Borcard D. Statistical comparison of univariate tests of homogeneity of variances (Submitted to the Journal of Statistical Computation and Simulation) Lemeshko B, Yu EM () Bartlett test in measurements with probability laws different from normal. Meas Tech (): – Levin I () Relating statistics and experimental design. Sage, Thousand Oaks Manoukian EB, Maurais J, Ouimet R () Exact critical values of Bartlett’s test of homogeneity of variances for unequal sample sizes for two populations and power of the test. Metrika ():– Pitman EJG () Tests of hypotheses concerning location and scale parameters. Biometrika :– Ramsey PH () Testing variances in psychological and educational research. J Educ Stat ():– Shoemaker LH () Fixing the F test for equal variances. Am Stat ():– Wu J, Wong A () A note on determining the p-value of Bartlett’s test of homogeneity of variances. Commun Stat Theory ():–
Bayes’ Theorem
B
a test for the presence of HIV in a patient. If B is the event of a positive result from the test, suppose
Bayes’ Theorem Joseph B. Kadane Leonard J. Savage University Professor of Statistics, Emeritus Carnegie Mellon University, Pittsburg, PA, USA
P{B ∣ A} = . and P{B ∣ A} = .,
The conditional probability P{A ∣ B} of event A given event B, is commonly defined as follows:
which means that if the person has HIV, the test is % likely to find it, and if the person does not have HIV, the test is % likely to report a positive result. Then (.)(.) (.)(.) + (.)(.) . = . + . . = . = .
P{A ∣ B} = P{AB} P{A ∣ B} = P{B}
()
provided P{B} > . Alternatively, () can be reexpressed as ()
P{AB} = P{A ∣ B}P{B}.
The left-hand side of () is symmetric in A and B, while the right-hand side does not appear to be. Therefore we have ()
P{A ∣ B}P{B} = P{B ∣ A}P{A}, or P{A ∣ B} =
P{B ∣ A}P{A} , P{B}
()
which is the first form of Bayes’ Theorem. Note that the event B can be reexpressed as ()
B = AB ∪ AB. Because AB and AB are disjoint, we have
P{B} = P{AB} + P{AB} = P{B ∣ A}P{A} + P{B ∣A}P{A}. () Substituting () into () yields the second form of Bayes’ Theorem: P{A ∣ B} =
P{B ∣ A}P{A} . P{B ∣ A}P{A} + P{B ∣ A}P{A}
()
Finally, let A , A , . . . , Ak be disjoint sets whose union is the whole space S. Then, generalizing (), B = ∪ki= Ai B
()
and the sets Ai B are disjoint. Then k
k
P{B} = ∑ P{Ai B} = ∑ P{B ∣ Ai }P(Ai }. i=
()
Thus a positive test result, on a respectable test, does not imply a high probability that the patient actually has HIV. Bayes’ Theorem has an aura of being controversial. Since everything above is a consequence of the laws of probability and the definition of conditional probability, the correctness of the formulas above is conceded by all the various schools of interpretation of probability. The Bayesian school uses Bayes’ Theorem as a way to understand the extent to which knowing that the event B (test result in the example) has changed the probability of the event A, resulting in a map from P{A} to P{A ∣ B}. In the example, P{A} = . is the probability the person has HIV given that person’s risk factors. P{A ∣ B} = . is the updated probability that takes into account the positive test result B. The controversy comes from the fact that some statisticians (Fisher ) take the view that the only events that have probabilities are those that can be regarded as elements of an infinite (or large) sequence of independent and identically distributed events. Then the probability of A is taken to be the relative frequency of the event A in this sequence. Thus the controversy has to do not with the legitimacy of Bayes’ Theorem itself, but rather with the applications of it.
i=
About the Author
Then generalizing (), we have P{Aj ∣ B} =
P{B ∣ Aj }P{Aj } ∑ki= P{B ∣ Ai }Pi {Ai }
,
()
the third form of Bayes’ Theorem. As an example, let A be the event that a particular person has HIV, and suppose P{A} = .. Suppose there is
Joseph B. Kadane received his Ph.D. in Statistics, Stanford University, . He was Head, Department of Statistics, Carnegie Mellon University (–). He was Associate editor of Journal of the American Statistical Association (–) and Applications and Coordinating Editor (–), Associate Editor of Annals of Statistics
B
B
Bayesian Analysis or Evidence Based Statistics?
(–) and Journal of Business and Economic Statistics (–). He was also Elected Member, Board of Directors, International Society for Bayesian Analysis (–). Professor Kadane was named the Pittsburgh Statistician of the Year (), chosen by the Pittsburgh Chapter of the American Statistical Association. He was Co-winner, Frank Wilcoxon Award for the Best Applied Paper in Technometrics, . He has (co-)authored over papers. ( in Economics and Econometrics; in Sociology and Demography; in Political Science; in Operations Research and Computer Science; in Mathematics and Statistical Theory; in Law; in Management Science; in Psychology, Medicine, and Biology and in Environmental and Physical Sciences.)
promoted as an alternative to the present likelihood based frequentist analysis of data with model, the evidence-based approach of central statistics. And we are led to focus on the bane of Bayes: parameter curvature.
Bayes Bayes () examined the Binomial model f (y; θ) = (nθ ) θ y ( − θ)n−y and proposed the flat prior π(θ) = on [, ]. Then with data y he used a lemma from probability cal culus to derive the posterior π(θ∣y ) = cθ y ( − θ)n−y on [, ]. And then for an interval say (θ, ) he calculated the integral of the posterior,
s(θ) = ∫
θ
Cross References 7Bayesian Statistics 7Bayesian Versus Frequentist Statistical Reasoning 7Conditional Expectation and Probability 7Inversion of Bayes’ Formula for Events 7Philosophy of Probability 7Probability, History of 7Statistical Inference: An Overview 7Statistics, History of 7Statistics: An Overview
References and Further Reading Fisher R () Statistical methods and scientific inference, nd edn. Oliver and Boyd, Edinburgh and London
θ y ( − θ)n−y dθ/ ∫
θ y ( − θ)n−y dθ
and referred to it as probability that the parameter belonged to the interval (θ, ). Many endorsed the proposed calculation and many disputed it. As part of his presentation he used an analogy. A ball was rolled on a level table, perhaps an available billiard table, and was viewed as having equal probability of stopping in any equal sized area. The table was then divided conceptually by a North-South line through the position where the ball stopped, with area θ to the West and ( − θ) to the East. The ball was then rolled n further times and the number y of time that it stopped left of the line observed. In the analogy itself, the posterior probability calculation given data seems entirely appropriate.
The Economist
Bayesian Analysis or Evidence Based Statistics? D. A. S. Fraser Professor University of Toronto, Toronto, ON, Canada
Introduction The original Bayes proposal leads to likelihood and confidence for many simple examples. More generally it gives approximate confidence but to achieve exact confidence reliability it needs refinement of the argument and needs more than just the usual minimum of the likelihood function from observed data. A general Bayes approach provides a flexible and fruitful methodology that has blossomed in contrast to the widely-based long-standing frequentist testing with focus on the % level. We examine some key events in the evolution of the Bayes approach
In an article entitled “In praise of Bayes,” the Economist () speaks of an “increasingly popular approach to statistics (but) not everyone is persuaded of its validity.” The article mentions many areas of recent application of the Bayesian approach, and cites “the essence …is to provide a mathematical rule explaining how you should change your existing beliefs in the light of new evidence.” The indicated areas of application are wide spread and there is emphasis on attaining definitive answers. And this is set in full contrast to “traditional ways of presenting results” indicated to be the mid-twentieth-century decision theoretic view of accepting a null view –% on some departure scale. The article does offer some caution for “when used indiscriminently” in the form of a quotation from Larry Wasserman that it can become “more a religion than a science.” The mathematical rule cited as the essence of the Bayesian approach is a very broad expansion from Bayes original proposal where a statistical model f (y; θ) evaluated at an observed data value y giving f (y ; θ) is combined with a constant mathematical prior π(θ) = and
Bayesian Analysis or Evidence Based Statistics?
treated as a conditional density. The force of the rule is that with new model-data information the new likelihood would be folded with the old. But this is of course standard practice in statistics: use the up-to-date likelihood, and possibly refine such a procedure with metaanalysis. What is different is that the Bayesian method essentially overlooks evidence beyond the observed likelihood function and does so on principle.
Validity or Analogy Bayes considered a uniform prior and a Binomial (n, p) model, and used analogy to justify combining them by a standard lemma from probability calculus. For the analogy involving balls on a billiard table, the calculations seem entirely proper and appropriate. The more generally interpreted Bayes approach has a statistical model f (y; θ) with data y coupled with a mathematical prior π(θ) representing symmetries or other properties of the model or context. Analogies can be great for explaining an argument but not to be the argument itself: there is no billiard table equivalent in the typical binomial or more general context. There is an explicit time line: There is a context with a true value θ ∗ for the parameter θ; there is an investigation f (y; θ) yielding an observed y from the true value model f (y; θ ∗ ); and possibilities for θ are then to be assessed. Thus in order: θ ∗ is realized but unknown; y is observed; then assess θ. The values θ ∗ and y are realized and are in the past. And the issue is what can be said about θ given the model f (y; θ) and data y . If θ is understood in fact to come from an objective source π(θ) with realized value θ ∗ ; then the time line is longer. Accordingly: π(θ) produces θ ∗ ; f (y; θ ∗ ) produces y ; and the issue is to assess θ. In this situation π(θ) is properly an objective prior. And an option is of course is to examine and present the composite model π(θ)f (y; θ) with observed y . But an even more compelling option is to examine and present π(θ) and to separately examine and present f (y; θ) with y . Now consider the model f (y; θ) with data y ; and the mathematical prior π(θ) as proposed by Bayes. The lemma from the probability calculus has two probability inputs say π(x) and f (y∣x) and it has one probability output π(x∣y ); the output records the behavior of x that is associated with the observed value y = y . For the Bayes case π(x) would be π(θ) and π(x∣y ) would be π(θ∣y ). Is the lemma applicable or relevant in the Bayes case? In the Bayes case there is just one probability input f (y; θ); and the other nominal input is π(θ), a mathematical object that refers to symmetry or patterns in the model and has no probability status whatsoever. Thus the assumptions of the lemma do not
B
hold, and consequently the output of the lemma is ... by analogy ... not by derivation. The usage of the lemma in the proposed argument is not proper and can be viewed as fraudulent logic. The standard frequentist would refer to f (y ; θ) as likelihood L(θ; y ) = L (θ). An exploration with weighted likelihood π(θ)L (θ) can be a very natural, obvious and sensible procedure ... for just that, for exploring possibilities for θ. But for obtaining probabilities, perhaps a pipe dream!
Likelihood and Confidence Bayes’ () original approach suggested a density cπ(θ)f (y ; θ) as a description of an unknown θ in the presence of observed data y . As such it records likelihood L (θ) or weighted likelihood. And this was long before the formal introduction (Fisher ) of likelihood. Both viewpoints record the same formal information concerning the parameter; the differences are in the color or flavor associated with the particular argument; and with properties attributed to the output. Bayes () also offered a distribution as a summary of information concerning the parameter θ; the distribution had density cπ(θ)f (y ; θ) = cπ(θ)L (θ). The majority of models at that time had least-squares location structure and for such models the posterior π(θ)L (θ) using a natural prior just reproduces what is now called confidence (Fisher , ). It thus seems appropriate to acknowledge that Bayes introduced the primary concepts of likelihood and confidence long before Fisher (, ) and long before the refinement offered by Neyman (). For likelihood he offered the extra flexibility of the weight function but for confidence he did not have the statistical refinement that later provided the logical extension to non-location models; this latter can be viewed as a matter of reasonable fine tuning of the argument, of intellectual evolution, and of the familiar iterative processes of science.
Laplace and Venn Laplace () seems to have fully endorsed the proposals arising from Bayes (). And Venn () seems equally to have rejected them. Certainly asserting the conclusions of a theorem or lemma when one of the premises does not hold is unacceptable from a mathematical or logical viewpoint. Nonetheless the results were impressive and filled a substantial need, but indeed with downstream risks. And it does have, as is now becoming apparent, the support of approximate confidence (Fraser ). At present Bayes and confidence lead a coexistence, perhaps an uneasy unstable coexistence!
B
B
Bayesian Analysis or Evidence Based Statistics?
Priors and Priors The original Bayes prior was a flat prior π(θ) = for a probability θ that then in sequence becomes the parameter in a Binomial (n, θ) model; the resulting posterior is π(θ)L (θ), which focally uses the observed likelihood from the Binomial context. Some aspects of invariance were invoked to support the particular choice. The possible plausible extensions are immense. For a location model f {y − β(θ)} the natural prior would be π(θ)dθ = dβ(θ) = β′ (θ)dθ. Thus for f (y − Xβ) we would have π(β)dβ = dXβ = cdβ, giving a flat prior for the regression coefficients. Motivation would come by noting that y − β(θ) has a fixed θ-free distribution. Extensions are possible by seeking approximate θ-free distributions. This was initiated by Jeffreys () and then fine-tuned to acknowledge various types of parameters (Jeffreys ). These extensions use expected information i(θ) = E{−ℓ θ θ (θ; y); θ} in the model to calibrate the scale for θ; here −ℓ θ θ (θ) is the negative second derivative of likelihood and the initial Jeffreys prior is π(θ)dθ = ∣i(θ)∣/ dθ, and it is parameterization invariant. For the regression model, where y = Xβ + σz with N(, ) error, the Jeffreys () prior is π(θ)dθ = dβdσ/σ r+ where r is the column rank of X. The second or modified Jeffreys () is π(θ)dθ = dβdσ/σ and gives generally more acceptable results, often in agreement with confidence. The approximate approach can be modified (Fraser et al. ) to work more closely with the location invariance indicated by the initial Bayes () approach. In many regular problems continuity within the model leads to a relationship d θˆ = W(θ)dθ where W(θ) is a p × p matrix; the d θˆ refers to an increment at the data y and the dθ refers to an increment dθ at θ. This immediately indicates the prior π(θ)dθ = ∣W(θ)∣dθ based on simple extension of the translation invariance dy = β ′ (θ)dθ for the model f (y − β(θ))dθ; and it widely agrees with preferred priors in many problems. But the parameter must not have curvature: the bane of Bayes! The approximate approach can also be modified to make use of an asymptotic result that to second order the statistical model can be treated as an exponential model (Reid and Fraser ; Fraser et al. ). This uses continuity to obtain a nominal reparameterization φ(θ) that yields second and third order inference by acting as if the model were just g(s; θ) = exp{ℓ(θ)+φ(θ)s}h(s) with data s = . This allows information to be calculated within the approximating model using the information function jφφ (θ; s) = −ℓ φφ {θ(φ)}; this draws attention to marginalization and to curvature effects that are not usually apparent in the search for default priors (Fraser et al. ).
The preceding can also be viewed as a somewhat natural evolution from the original Bayes proposal with some reference to location invariance. The evolution has been assisted by the fact that many posterior distributions have appealing and sensible properties. It is our view here that these sensible properties are precisely the approximate confidence properties that have become evident quite separately. In any case the priors just described can all be classified as default priors, priors that one might choose to use as a default without strong arguments for something different. The Bayesian approach is committed to using a weight function applied to an observed likelihood and thus to formally omitting other properties of the model. Within this approach the default priors are widely called objective priors. But the term objective means objective reference, and this as a property is specifically absent here; there is a strong flavor of deception. Thus using the term objective for default priors seems highly inappropriate, but could be viewed as just a seeking for a wider area of application. The author was present at the Bayesian convention when the term was being chosen and did not register an objection, being perhaps somewhat of an outsider, not a good defense! We will however explicitly refer to them as default priors, and keep the term objective for contexts where the prior does have an explicit reference in context. A difficulty with the use of default priors is that a posterior probability obtained by marginalization from a full posterior distribution may not be equal the posterior probability calculated directly from the appropriate marginal model; this was given prominence by Dawid et al. () and applies equally to confidence distributions and other attempts to present model-data information as a distribution for the parameter. The complication in any of these cases derives from parameter curvature: for some discussion see Fraser et al. () and Fraser and Sun (). The wealth of possibilities available from a weightfunction combined with likelihood is well documented in the development of the Bayesian methods as just described. Its success can amply be supported as “approximate confidence” but derived by a route that is typically much easier. Approximate confidence provides full support for the acceptable, often meritorious behavior of Bayes posterior probabilities. We address later whether there can be anything beyond approximate confidence in support of the Bayesian approach. Another approach, somewhat different from the original Bayes way of obtaining a weight function is derived from Kullback-Leibler distance on measure spaces (Bernardo ): this chooses a prior to maximize the
Bayesian Analysis or Evidence Based Statistics?
statistical distance from prior to posterior. Modifications of this distance approach have been developed to obtain specialized priors for different component parameters of interest, often parameters that have a statistical curvature. The richness available from using just a likelihood function is clearly evident to Bayesians if not to frequentists; but is not widely acknowledged. Much of recent likelihood theory divides on whether or not to use more than the observed likelihood, specifically sampling properties that are associated with likelihood characteristics but are not widely or extensively available. In many ways central statistics has ignored the extra in going beyond likelihood, and indeed has ignored the wealth available from just likelihood alone. Meanwhile those committed to using just the weighted likelihoods, those associated with the development of the Bayes approach as we have just described, have aggressively sought to use the weighted likelihood approach as a general approach to updating information and to producing decisions. Central to this direction is the subjective approach with a major initiative coming from Savage (). This takes a prior to represent the views, the understanding, the personal probabilities concerning the true value of the parameter; these might come from highly personal thoughts, from detailed elicitation from knowledgeable people, from gut feelings as one approaches a game at a casino; and they can have the benefit of intuition or the merits of a seasoned gambler, with or without insider information. But who should use it? Certainly the chronic gambler will. But from the statistical perspective here there is nothing of substance to say that such prior “information” π(θ) should be combined with likelihood. With due respect it can be presented as π(θ) alongside a presentation of the evidence-based well calculated confidence. If a user would like to combine them, it would certainly be plausible for him to do so but it would not be an imperative despite Bayesian persuasion. Certainly place them both to be seen and available. In wide generality combining them is not a necessary statistical step, although sometimes expedient.
Lindley and Territory Fisher’s (, ) proposal for confidence with effective support from Neyman () offered strong alternatives to a prominent sympathy for the Bayesian approach. Then Jeffreys (, ) with great prominence in geophysics provided reinforcement for the use of the Bayesian approach in the physical sciences. Meanwhile the confidence approach gained strength both in mathematics departments and in scientific applications. Both
B
approaches lead from model and data to a distribution for the parameter, but the results were often in conflict. Both sides clearly felt threatened, and each side in a practical sense had territory to defend. Lindley () focused on the very basic case, a scalar parameter and a scalar variable, say with distribution function F(y; θ). The Bayesian answer with prior π(θ) is given by the posterior distribution cπ(θ)Fy (y; θ)dθ where the subscript y denotes differentiation with respect to the argument y thus giving the density or likelihood function. By contrast the Fisher (, ) approach gives the confidence distribution ∣Fθ (y; θ)∣dθ. Lindley examined when these would be equal and solved for π(θ): π(θ) = c
∂ F;θ (y; θ) = c y(u; θ); Fy (y; θ) ∂θ
the right hand expression records the derivative of the quantile function for fixed p-value u = f (y; θ) as pursued in Fraser et al. (). The equation is actually a differential equation that asserts that the model must be a location model, the form of model actually found in section “7Likelihood and Confidence” to have good Bayesian answers. In Fraser et al. (ibid) the equation is used to determine the data dependent priors that give posterior probabilities having objective validation. Lindley was concerned that the confidence approach did not follow the primal Bayesian concept that a probability statement concerning a parameter should be updated by multiplication by new likelihood and his criticism had a profound effect suggesting that the confidence distribution approach was defective. We now know that the defect is the attempt to use a distribution as the summary, either by Bayes or by confidence. And if there were to be a lesser of two evils then calling confidence probability and calling Bayes approximate confidence would be safer. Another view might be that this was just a territorial dispute as to who had the rights to provide a distributional description of the parameter in the model data context. But the social conflict aspects were not in evidence. Rather there was a wide spread perception that giving a distribution of confidence was wrong. Neyman () of course had provided a route around. But nonetheless, the judgment stuck: a confidence distribution was wrong and a Bayesian analysis was all right. Of course, in Dawid et al. (), there is a clear message that neither approach can handle vector parameters without special fine-tuning. Clearly Lindley had focused on a substantive issue but the arguments invoked had not quite attained the point of acknowledging that an effective prior must in general be data dependent; for some current discussion see Fraser et al. ().
B
B
Bayesian Analysis or Evidence Based Statistics?
Bayesian Analysis and Imperatives Bayesian () analysis has been around for a long time, but alternative views perhaps now identified as frequentist are perhaps older although somewhat less formalized. These approaches have cross dialogued and often been in open conflict. Each has made various appeals to holding the truth. And they have actively sought territorial advantage. In particular Fisher’s initial steps towards confidence were directly to provide an alternative to inverse probability, the name at the time attached to the Bayesian approach. So it is not surprising that there would be a very focal reverse criticism (Lindley ) of the confidence approach. Those favoring the Bayesian approach have frequently felt they were underdogs, often for example having their articles rejected by journals for just being Bayesian. It thus seems rather natural that the Bayesian supporters would seek to broaden their methodology and their community. The subjective approach as strongly initiated by Savage () has led to a powerful following in an area where prior probabilities are extended to include personal feelings, elicited feelings, and betting view points. Certainly such extensions are a guide for gambling and much more. But there is nothing of substance to assert that they should be ... the imperative ... used for the analysis. The prior subjective assessment and the objective evidence-based assessment can be placed side by side for anyone to see and to use as deemed appropriate. And the Bayes combination of these can also be presented for anyone to use if so inclined. Perhaps the Bayesian expansion was ill advised to promote the imperative: that the proper analysis was that of the Bayes paradigm. What is perhaps even more dangerous is the widely promoted hierarchical model where each parameter is given a prior distribution, and then parameters in the prior distributions are themselves given priors, perhaps then multilevel. Often an impressive edifice that seems only equaled by the lack of evidence for the various introduced elements and the impressive resort to McMC. The resort to multilevel Bayes modeling would seemingly be best viewed as one of expediency, to extend the base of Bayes without supporting evidence. And then of course there are model data situations where the true parameter has come from a source with a known frequency distribution. In such cases the obvious name for the prior would be objective prior. But assembled Bayesians as mentioned earlier have adopted that name for the opposite situation, where there is in fact no objective reference, and the prior is purely a mathematical construct. But what about the multitude of cases where
there is an identified source for the true parameter value? These can arise widely when the entity being examined has been obtained by sampling from some identified subpopulation; or they can arise by genetics or by Mendel or perhaps by updated genetic laws. Or much more. In reality this is just a modeling issue: what aspect of the context, of the immediate environment, or the more extended environment should be modeled. It is a modeling issue. It is perhaps only natural that Bayesian promotion should seek to subsume wider and wider contexts as part of the evolution of the approach. Especially when traditional statistics has been widely immersed in technical criteria connected with some global optimization or with decision rules to reject at some % level or accept at some / level, even when it was becoming abundantly apparent that these rules for scientific publication have serious defects. But if there is an objective source π(θ) for a true value in a model-data context, there is nothing that says it should be folded into a combined model for analysis. The prior source π(θ) can be set in parallel with the more directly evidence-based analysis of the model-data combination. And of course even the combined model-data-prior analysis presented. But again there is no substantive precept that says the combined analysis is the statistical inference. Such a step would be purely an assertion of extended territory for Bayesian analysis.
Curvature: The Bane of Bayes Contours of a parameter can have obvious curvature. A simple example can throw light on the effects of such curvature. Consider (y , y ) with a Normal {(θ , θ ); I} distribution on the plane. With data (y , y ) the basic original Bayes approach would say that (θ , θ ) was Normal {(y , y ) ; I}. First we examine an obviously linear parameter ψ = θ and assess say the value ψ = on the basis of data, say (y , y ) = (, ). In an obvious way y measures ψ and has the Normal(ψ; ) distribution. Accordingly the p-value for ψ from the observed data is p(ψ) = Φ {(y − ψ) /} = Φ(−ψ). And for assessing the value ψ = we have p() = %. Now consider the Bayesian assessment of the value ψ. the marginal posterior distribution of ψ is N (y , ) and the corresponding posterior survivor value is s(ψ) = − Φ ((ψ − y ) /) = − Φ(ψ)
Bayesian Analysis or Evidence Based Statistics?
at the observed data. In particular for assessing ψ = we would have s() = %. The Bayesian and frequentist values are equal for the special ψ = and also for general ψ. Now consider a clearly curved parameter, the distance ψ on the parameter space from the point (−, ) to the parameter value (θ , θ ), ψ = {(θ + ) + θ }
/
.
An obvious way to measure this parameter is by using the distance r from the point (−, ); thus r = {(y + ) + /
y } . The distribution of r is noncentral Chi-square with two degrees of freedom and noncentrality δ = ψ . The indicated p-value for assessing ψ is then p(ψ) = H (r ; ψ ) where H is the noncentral Chi-square distribution function with two degrees of freedem and noncentrality δ = ψ . This is readily available in R. In particular for assessing ψ = we would have p() = H (; ) = .% which is substantially less than %. Now consider the Bayesian assessment of the curved parameter ψ. The posterior distribution of ψ from the observed data is noncentral Chi-square with two degrees of freedom and noncentrality δ = . If follows that the posterior survivor value for assessing ψ = is s() = − H (; ) = .% which is substantially larger than %. For this simple example we have seen that the p-value and the survivor value are equal for a linear parameter. This happens generally for linear parameters (Fraser and Reid ). And with the introduction of a curvature change to the parameter, the Bayesian and frequentist values go in opposite directions. This happens widely with curved parameters: as a parameter contour is changed from linear to curved, the Bayesian survivor changes in the opposite direction from the frequentist. Thus the Bayesian can be viewed as correcting negativity, that is making an adjustment opposite to what is appropriate in a context. For some recent discussion see Fraser (). The example above suggests that curvature is precisely the reason that Bayes fails to correctly assess parameters. Consider y with a Normal {θ, σ (θ)} distribution where the variance σ (θ) depends weakly on the mean θ.
B
Precise p-values are available for assessing θ: p(θ) = Φ{(y − θ)/σ(θ)} with a clear frequency interpretations. The confidence inversion is well established (Fisher , ). The Bayesian inversion does not seem to have an obvious prior that targets the parameter θ. How does one assess the merits of a proposed distribution for a parameter? The use of two-sided intervals provides a slippery slope. Strange tradeoffs can be made between two interval bounds; see for example Fraser et al. () on statistics for discovering new particles in High Energy Physics. A more direct approach is to examine a particular quantile of a proposed distribution, say the β-th quantile θˆ β which has posterior probability β to the right and ( − β) to the left. One can certainly simulate or have an oracle and determine what proportion of the time the true value is larger than the particular quantile being considered; and determine whether the true proportion bears a sensible relation to the alleged value β. This has been addressed at length in Fraser (). In particular for the Normal{θ, σ (θ)} example there is no determination of a prior that will give the third order accuracy that is available from the confidence approach unless the prior is directly specific to the observed data value. This result holds in wide generality: the use of a default or Bayesian prior cannot lead to the third order accuracy readily available from the evidence-based procedures of frequentist inference. And parameter curvature is the number one culprit.
Why Bayes? Linear approximations are widely used throughout statistics, mathematics, physics, the sciences generally, and much more. They provide a local replica of something that might be intangible otherwise and when used iteratively can provide exploration of something unknown otherwise. There is substantial evidence that the Bayes procedure provides an excellent first order approximation for the analysis of a statistical model. There are also ample warnings that global acceptance of Bayes results can be extremely hazardous. Use but be cautious! The Bayes calculus asserts that the posterior results are probabilities. And the name itself is assertive. The Bayesian supporters have also been vocal, asserting that confidence results do not have the status of probabilities calculated by the Bayes paradigm; some indication of this is implicit in Lindley (); and further indication is found in the active broadening of the application area for Bayesian analysis. From an evidence-based approach it
B
B
Bayesian Analysis or Evidence Based Statistics?
is clear that the direct use of the likelihood function provides substantial information, first order information. And higher order results are available with the careful choice of prior. But beyond that, the Bayes procedure comes up short, unless the priors become data dependent and the calculations are carefully targeted using an evidence-based formulation. Thus linear approximations can be hugely useful but they can carry substantial risks. The assertion of probability status is directly contradicted by reality! And no indications seem available that a Bayesian calculation could yield more than just approximate confidence. The promotional assertiveness that accompanies current Bayes development is misleading and misleading to the extent of being fraudulent. Of course there can be contexts where there is an objective prior π(θ) that records how the true value was generated. The Bayes paradigm can be applied but it is inappropriate; the direct approach is a matter of modeling, of what aspect of the context is appropriate to include. From this viewpoint the indicated methodology predates Bayes; it is just probability analysis. Even then it allows the prior information and the evidence based information to be presented separately, thus of course allowing the end user to combine if needed or wanted. There is no imperative that says the prior and the evidence-based should be combined. It is an option. And it is an option with risks!
About the Author Donald A.S. Fraser was born in Toronto in . He obtained his Ph.D. in under the supervision of Samuel Wilks at Princeton University. He was the first Chair of the Department of Statistics, University of Toronto (– ). Among many awards, Professor Fraser was the first recipient of the Gold Medal of the Statistical Society of Canada, inaugurated in . He received the R.A. Fisher Award and Prize, American Statistical Association (), and Gold Medal, Islamic Statistical Society (). He was the first statistician to be named a Fellow of the Royal Society of Canada (). He has supervised Ph.D. students. Professor Fraser has (co-)authored over papers and authored five books, including Nonparametric Methods in Statistics (Wiley ), The Structure of Inference (Wiley ) and Inference and Linear Models (McGraw Hill ). In , he was awarded a degree of Doctor of Science, honoris causa, by the University of Toronto. In (April –May ), the third in a series of conferences, titled Data Analysis and Statistical Foundations III, was held to honor the accomplishments of Professor Fraser.
Cross References 7Bayesian Statistics 7Bayesian Versus Frequentist Statistical Reasoning 7Bayesian vs. Classical Point Estimation: A Comparative
Overview 7Foundations of Probability 7Frequentist Hypothesis Testing: A Defense 7Likelihood 7Model Selection 7Philosophical Foundations of Statistics 7Prior Bayes: Rubin’s View of Statistics 7Probability Theory: An Outline 7Probability, History of 7Significance Tests: A Critique 7Statistical Inference 7Statistical Inference: An Overview 7Statistics: An Overview 7Statistics: Nelder’s view
References and Further Reading Bayes T () An essay towards solving a problem in the doctrine of chances. Phil Trans R Soc London :–; :–. Reprinted in () Biometrika :– Bernardo JM () Reference posterior distributions for Bayesian inference (with discussion). J R Stat Soc B : – Dawid AP, Stone M, Zidek JV () Marginalization paradoxes in Bayesian and structural inference. J R Stat Soc B : – Fisher RA () On the mathematical foundations of theoretical statistics. Phil Trans R Soc London A :– Fisher RA () Inverse probability. Proc Camb Phil Soc : – Fisher RA () The fiducial argument in statistical inference. Ann Eugenic B :– Fraser DAS () Is Bayes posterior just quick and dirty confidence? Stat Sci, in review Fraser DAS, Reid N () Strong matching of frequentist and Bayesian parametric inference. J Stat Plann Infer : – Fraser DAS, Reid N, Wong A () Inference for bounded parameters. Physics Rev D : Fraser DAS, Reid N, Marras E, Yi GY () Default prior for Bayesian and frequentist inference. J R Stat Soc B to appear. Fraser DAS, Sun Y () Some corrections for Bayes curvature. Pak J Statist, :– Jeffreys H () Theory of probability, rd edn. Oxford University Press, Oxford Jeffreys H () An invariant form by the prior probabilities in estimation problem. Proc Roy Soc A :– Laplace PS () Théorie analytique des probabilités. Courcier, Paris Lindley DV () Fiducial distribution and Bayes theorem. J R Stat Soc B :– Neyman J () Outline of a theory of statistical estimation based on the classical theory of probability. Phil T R Soc A : –
Bayesian Approach of the Unit Root Test
Reid N, Fraser DAS () Mean likelihood and higher order inference. Biometrica, :– Savage LJ () The Foundations of Statistics. Wiley, New York
hypothesis of a unit root in real exchange rates because nominal and real exchange rates behave almost like random walks (see 7Random Walk). Now, suppose that we have a sample of T consecutive observations on a time series yt generated by yt = ρyt− + µ t
Bayesian Approach of the Unit Root Test Hocine Fellag , Lynda Atil Professor in Statistics University of Tizi-Ouzou, Tizi-Ouzou, Algeria University of Tizi-Ouzou, Tizi-Ouzou, Algeria In statistical inference, two approaches of hypothesis testing can be used. The first one, called classical approach, is based on calculating probabilities of the data in hand under certain conditions (which are encompassed by the null and alternative hypotheses). The second one, named Bayesian approach, looks at probabilities of competing conditions (which are the hypotheses being compared) given the data in hand. This approach integrates prior probabilities associated with competing conditions into the assessment of which condition is the most likely explanation for the data in hand. Also, it allows to evaluate the likelihood of competing conditions by evaluating the change in the odds associated with these conditions, a change produced by assessing the data in hand. If the odds change sufficiently when the data are examined, then the scientist may alter his opinion about which competing condition is the most likely. One of the main papers published on this topic is the paper of Schotman and Van Dijk (a), where a very important problem in Bayesian analysis is tackled, namely, the Bayesian approach for unit root testing. Several economists, Dejong and Whiteman (), Koop (), and in particular, Sims (), and Sims and Uhlig (), have advocated forcefully for Bayesian alternatives over the more traditional classical approach such as the ADF tests (Dickey and Fuller ) in unit root testing. Despite the apparent advantages of the Bayesian approach over the classical approach in unit root testing, a relatively small number of studies have used the Bayesian approach. The reasons may be that the Bayesian approach requires a likelihood function and the use of prior information. The modeling objective of the Bayesian approach is not to reject a hypothesis based on a predetermined level of significance, but to determine how probable a hypothesis is relative to other competing hypotheses. Schotman and Van Dijk (a) propose a posterior odds analysis of the
B ()
where . y is a known constant. . µ t are identically and independently (i.i.d) normally distributed with mean zero and unknown variance σ . . ρ ∈ S ∪ {}, S = {ρ/ − < a ≤ ρ < }. The econometric analysis aims at discriminating between a stationary model (here defined as a ≤ ρ < ) and the nonstationary model with ρ = . The lower bound a in assumption () largely determines the specification of the prior for ρ. Recall that the principal Bayesian tool to compare a sharp null hypothesis with a composite alternative hypothesis is the posterior odds ratio, which is defined as ∞
p(ρ = ∣ y) ∫ p(σ)L(y ∣ ρ = , σ, y )dσ = ∞ ∫S ∫ p(σ)p(ρ)L(y ∣ ρ, σ, y )dσdρ p(ρ ∈ S ∣ y) () K and K are the prior odds and the posterior odds in favor of the hypothesis ρ = , respectively. p(ρ) represents the prior density of ρ ∈ S, p(σ) the prior density of σ. L(y ∣ .) is the likelihood function of the observed data ′ ′ ′ y = (y . . . yT ) and Y = (y , y ) is all observed data. The Bayes factor is defined as the ratio of the marginal posterior density of ρ under the null hypothesis ρ = over a weighted average of the marginal posterior under the alternative using the prior density of ρ as a weight function. Then, one can notice that the posterior odds K is equal to the prior odds K times the Bayes factor. The prior odds express the special weight given to the null hypothesis, the point ρ = is given the discrete prior probability ϑ = K /( + K ). From the posterior odds, one can compute the posterior probability of the null hypothesis as K /( + K ). For the complete specification of the marginal prior of ρ and σ, we assume that
K = K
Pr(ρ = ) = v p(ρ ∣ ρ ∈ S) = p(σ) ∝
σ
−a
() () ()
The prior of ρ is uniform on S but has a discrete probability ϑ that ρ = . The likelihood function for the vector of T observations y is
B
B
Bayesian Approach of the Unit Root Test
L(y ∣ ρ, σ, y ) = (πσ )−T/ exp {−
′ µ µ} σ
()
where µ = y − y− ρ, and y− = (y , . . . , yT− )′ Having computed the relevant integrals in (), the posterior odds ratio becomes K =
σ CT− v ( ) / (T − ) − v σˆ −F (
a − ρˆ )] s ρˆ
−T/
(
− ρˆ −a ) [F ( ) s ρˆ s ρˆ
−
()
where
σ =
(y−y− )′(y−y− ) T−
σˆ =
′ (y− y) ⎞ ⎛′ y y− ′ T− ⎝ (y− y− )⎠
′ y− y Γ((T − )/)Γ(/) CT = ′ y− y− Γ(T/) () The empirical lower bound a∗ is given by a∗ = ρˆ + s ρˆ F − (αF(−τˆ)). F(.) is the cumulative t-distribution with ˆ ρ− (T − ) degrees of freedom and τˆ = s ρˆ is the Dickey– Fuller test statistic. The unit root model is preferred if K > or P(ρ = ∣y, y ) ≥ ., thus treating the null and the alternative in a symmetric way. After fixing numerical values for ϑ and α, the posterior odds is just a function of the data like any other test statistic. Due to a specific way that the lower bound has been constructed, the posterior odds are directly related to the 7Dickey–Fuller test. Setting the prior odds equal to one and for large T, Schotman & Van Dijk approximate F(.) to the cumulative normal distribution. The posterior odds become a function of the Dickey–Fuller statistic τˆ. −
′ y− ) sρˆ = σˆ (y−
ρˆ =
−τˆ − F − (αF(−τˆ)) ) () ln K = − ln(π) − τˆ + ln ( F(−τˆ) Since the posterior odds is a function of the Dickey–Fuller test statistic, its sampling properties correspond exactly to those of the Dickey–Fuller test. In literature, there is a great attention to the nature of suitable noninformative priors for the autoregressive coefficients. For example, Sims () and Sims and Uhlig () advocate the use of flat priors. Phillips (a) proved that flat priors bias the inference toward stationary models, and suggests to use Jeffrey priors derived from conditional likelihood functions. Also, Uhlig (a) determines the Jeffreys priors for an AR() process from the exact likelihoods and justifies the use of flat priors in some specific cases only. Uhlig (b) summarizes the Bayesian contribution to the unit root problem and discusses the sensitivity of the
tails of the predictive densities on the prior treatment of explosive roots. Schotman and Van Dijk (a) stress the sensitivity of the posterior odds to the size of the stationary region and suggest restricting the later’s size. Berger and Yang () consider a reference prior approach for the AR() model. It is particularly interesting to note that Marriott and Newbold () criticized the use of priors, such as the uniform or the Jeffreys prior, for the autoregressive coefficients in this context and advocate the use of sharp informative prior distributions. However, for the simple problem of testing for a unit root in a first-order autoregressive process, they find that the prior distribution for the autoregressive coefficient has a substantial impact on the posterior odds, so that, a very sharp beta prior performs extremely well when the generating process is stationary autoregressive, but the uniform prior is preferable when the true model is nonstationary. Marriott and Newbold () explore the use of the 7beta distribution as a prior specification. They have explained how Bayesian calculations can be carried out, noting the importance of the analyst, giving careful thought to the question of what might be an appropriate prior.
Conclusions Generally, authors agree with the idea that the Bayesian approach offers an alternative and a more useful way than the classical approach in empirical modeling. In unit root testing, Sims (), Sims and Uhlig (), and Koop (, ) have advocated the Bayesian approach over the classical ADF tests. In some papers, various opinions were expressed saying that the Bayesian solution is clear and simple when the classical approach is logically unsound. In a series of empirical applications, using a Bayesian approach, Dejong and Whiteman (, a, b) obtained results rejecting the presence of a unit root in various economic series, which contradicted those of Nelson and Plosser (). Ahking () studied the power of Koop’s “objective” unit root test. In particular, he was interested in whether or not it provides a better alternative to the classical ADF unit root test, and whether or not the use of “objective” priors are appropriate. However, there is no evidence to suggest that the “objective” Bayesian test is better than the classical ADF tests in unit root tests. Moreover, the “objective” priors do not seem to be appropriate since they tend to produce results that are biased in favor of the trend stationary hypothesis. Thus, unfortunately, while there is a need for more objective analysis of Bayesian time series, Koop’s “objective” Bayesian test does not appear to move us closer to that goal. So, one can ask, what is the best use in unit root tests, classical or Bayesian procedure? Intuitively,
Bayesian Nonparametric Statistics
it is very hazardous to discriminate between two competing economic theories on the basis of a univariate model. However, unit root tests may be helpful when they are used in a more complete modeling strategy as a protection against gross errors, as well as misspecification tests.
About the Author Professor Fellag was Vice chancellor of Mouloud Mammeri University of Tizi-Ouzou, Algeria, in charge of international relations from until . He is also a honorary professor of Rey Juan Carlos University of Madrid (Spain) in . He is Director of the national Algerian doctoral school of statistics since . Dr. Fellag has supervised more than doctoral students in statistics and has published around papers in international journals.
Cross References 7Bayesian Statistics 7Box–Jenkins Time Series Models 7Dickey-Fuller Tests 7Seasonal Integration and Cointegration in Economic
Time Series 7Seasonality
References and Further Reading Ahking FW () The power of the ‘Objective’ Bayesian unit root test. Working Paper –, University of Connecticut Berger JO, Yang RY () Noninformative priors and Bayesian testing for the AR() model. Economet Theor :– Dejong DN, Whiteman CH () Trends and cycles as unobserved components in US real GNP: a Bayesian perspective. J Am Stat Assoc Pap proc – Dejong DN, Whiteman CH (a) The temporal stability of dividends and stock prices: Evidence from the likelihood function. Am Econ Rev :– Dejong DN, Whiteman CH (b) Reconsidering’Trends and random walk in macroeconomic time series. J Monetary Econ :– Dickey DA, Fuller WA () Likelihood ratio statistics for autoregressive time series with a unit root. Econometrica : – Koop G () Objective’ Bayesian unit root tests. J Appl Econ : – Koop G () Recent progress in applied Bayesian econometrics. J Econ Surv :– Marriott J, Newbold P () Bayesian comparison of ARIMA and stationary ARMA models. Int Stat Rev ():– Nelson CR, Plosser CI () Trends and random walk in macroeconomic time series: Some evidence and implication. J Monetary Econ :– Phillips PCB (a) To criticize the critics: An objective Bayesian analysis of stochastics trends. J Appl Econ :– Phillips PCB (b) Bayesian routes and unit roots: De rebus priorbus semper est disputandum. J Appl Econ :–
B
Schotman P, Van Dijk HK (a) A Bayesian analysis of the unit root in real exchange rates. J Econ :– Schotman P, Van Dijk HK (b) On Bayesian routes to unit roots. J Appl Econ :– Sims CA () Bayesian skepticism on unit root econometrics. J Econ Dyn Control :– Sims CA, Uhlig H () Understanding unit rooters: A helicopter tour. Econometrica :– Uhlig H (a) On Jeffreys’ prior when using the exact likelihood function. Economet Theor :– Uhlig H (b) What macroeconomists should know about unit roots: A Bayesian perspective. Economet Theor :–
Bayesian Nonparametric Statistics Jaeyong Lee Associate Professor Seoul National University, Seoul, Korea
Bayesian nonparametric statistics covers Bayesian analysis of nonparametric models, statistical models whose parameter space is not finite-dimensional, and allows more flexible modeling than the parameteric alternatives. Bayesian analysis of a statistical model consists of three ingredients: prior, model (or likelihood), and posterior. The prior π(θ) is a probability measure on the parameter space Θ that reflects the analyst’s knowledge about the unknown parameter θ before he or she observes the data. The model describes the random mechanism by which the observation X is generated given the parameter θ, that is, X∣θ ∼ f (x∣θ). The posterior is the conditional distribution of θ given X, π(θ∣X), which reflects the analyst’s knowledge about θ after X is observed. The prior and posterior of nonparametric models are, thus, probability measures of infinite dimensional parameter spaces. Examples of such parameter spaces include the space of all probability measures on the real line, the space of all probability density functions on the real line, the space of all smooth functions, and many more. The most widely used nonparametric prior is the Dirichlet process (Ferguson ), a probability measure on the space of all probability measures. Let X be a measurable space with a σ-field A and α be a nonnull finite measure on X . A random probability measure P on X is said to follow a Dirichlet process with parameter α, denoted by DP(α), if for every measurable partition (A , A , . . . , Ak ) of X , (P(A ), P(A ), . . . , P(Ak )) follows
B
B
Bayesian Nonparametric Statistics
(finite-dimensional) Dirichlet distribution with parameter (α(A ), α(A ), . . . , α(Ak )). The Dirichlet process has many important properties that have theoretical and practical consequences. The class of Dirichlet processes is a conjugate prior class in the following sense. Suppose P ∼ DP(α), X , X , . . . ∣P ∼ P.
()
Then, the posterior P given X , X , . . . , Xn also follows Dirichlet process, that is, P∣X , . . . , Xn ∼ DP (α + ∑ni= δ Xi ), where δ x is a degenerate probability measure at x. Under model (), the sequence X , X , . . . form, marginally, the Pölya urn sequence, that is, X ∼ α/α(X ) and for n ≥ , Xn+ ∣X , . . . , Xn ∼
α + ∑ni= δ Xi . α(X ) + n
()
This property is the key to the posterior computation of the mixtures of Dirichlet process models. Sethuraman () derived an alternative constructive definition of the Dirichlet process. Suppose Y , Y , . . . are iid α/α(X ), θ , θ . . . are iid Beta(, α(X )) independently of Yi′ s. Let n−
p = θ , pn = θ n ∏( − θ i ), n ≥ .
()
i=
Then, P = ∑∞ i= pi δ Y i ∼ DP(α). This property shows P is discrete with probability if P ∼ DP(α), which was initially thought to be a shortcoming of the Dirichlet process. To remedy the discreteness of the Dirichlet process, the mixtures of Dirichlet process model has been proposed, which turns out to be the most successful model with Dirichlet process. It has the following structure: P ∼ DP(α) iid
X , X , . . . , Xn ∣P ∼ ∫ h(x∣θ)dP(θ), where h(x∣θ) is a probability density function with parameter θ, or equivalently, P ∼ DP(α) iid
θ , θ , . . . , θ n ∣P ∼ P ind
Xi ∣θ i ∼ h(x∣θ i ), ≤ i ≤ n. The mixtures of Dirichlet processes have been used for different problems, for example, Bayesian density estimation, cluster analysis (see 7Cluster Analysis: An Introduction), etc. Especially, mixtures of Dirichlet processes have been successful in cluster analysis. Since the random probability measure P is discrete, the random sample θs from P naturally have ties, and clusters of Xs are based on ties in θs.
Since the Dirichlet process was proposed, many other nonparametric priors have appeared. Among them are neutral to the right process (Doksum ), Gaussian process (O’Hagan ), beta process (Hjort ; Lo ), Polya tree process (Lavine ), and species sampling model (Pitman ), all of which are priors for distribution except beta process and gaussian process. The beta process and gaussian process are priors for cumulative hazard function and regression function, respectively. The posterior computation with nonparametric models can be complicated. There are algorithms specialized for specific priors (e.g., MacEachern ; MacEachern and Müller ; Lee ). Recently, DPpackage (Jara ), an R package that automates the posterior computation of some nonparametric models, has been built and lessens computational effort of practical users of nonparametric models. Unlike parametric models, whose posteriors behave asymptotically optimal in the frequentist sense, nonparametric posteriors can behave suboptimally. Diaconis and Freedman () have shown that even an innocentlooking prior may generate inconsistent posterior. This observation spurs the research effort to obtain conditions for posterior consistency, posterior convergence rate, and 7asymptotic normality of the posterior (Bernstein-von Mises theorem). There is now a large body of literature on the asymptotic issue of 7Bayesian statistics (e.g., Ghosal et al. ; Shen and Wasserman ; Kim and Lee ; Freedman ).
Acknowledgment I thank Professor Fernando Quintana for his helpful discussion and comments on the paper. This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology ().
Cross References 7Bayesian Statistics 7Bayesian vs. Classical Point Estimation: A Comparative
Overview 7Nonparametric Statistical Inference 7Parametric Versus Nonparametric Tests 7Posterior Consistency in Bayesian Nonparametrics 7Statistical Inference: An Overview
References and Further Reading Diaconis P, Freedman D () On inconsistent Bayes estimates of location. Ann Stat ():–
Bayesian P-Values
Doksum K () Tailfree and neutral random probabilities and their posterior distributions. Ann Probab :– Ferguson TS () A Bayesian analysis of some nonparametric problems. Ann Stat :– Freedman D () On the Bernstein-von Mises theorem with infinitedimensional parameters. Ann Stat ():– Ghosal S, Ghosh JK, van der Vaart AadW () Convergence rates of posterior distributions. Ann Stat ():– Hjort NL () Nonparametric Bayes estimators based on beta processes in models for life history data. Ann Stat (): – Jara A () Applied Bayesian non- and semi-parametric inference using dppackage. Rnews ():– Kim Y, Lee J () On posterior consistency of survival models. Ann Stat ():– Lavine M () Some aspects of Pólya tree distributions for statistical modelling. Ann Stat ():– Lee J () Sampling methods of neutral to the right processes. J Comput Graph Stat ():– Lo AY () A Bayesian bootstrap for censored data. Ann Stat ():– MacEachern SN () Estimating normal means with a conjugate style Dirichlet process prior. Comm Stat Sim Comput (): – MacEachern SN, Müller P () Estimating mixture of dirichlet process models. J Comput Graph Stat ():– O’Hagan A () Curve fitting and optimal design for prediction. J R Stat Soc Series B ():– Pitman J () Some developments of the Blackwell-MacQueen urn scheme. In: Statistics, probability and game theory, vol of IMS Lecture Notes Monogr. Ser., pp –. Inst. Math. Statist., Hayward, CA Sethuraman J () A constructive definition of Dirichlet priors. Stat Sinica, ():– Shen X, Wasserman L () Rates of convergence of posterior distributions. Ann Stat ():–
Bayesian P-Values Jayanta K. Ghosh , Mohan Delampady Professor of Statistics Purdue University, West Lafayette, IN, USA Professor Indian Statistical Institute, Bangalore, India While Bayesians do not like classical 7P-values and prefer measuring evidence in data through posterior probabilities of parameters or models, some problems like testing or exploration of goodness of fit of a single given model have led to the introduction of P-values. We confine ourselves to this particular context of goodness of fit in the brief discussion of Bayesian P-values. Most of this material is taken from Ghosh, Delampady and Samanta () and Ghosh, Purkayastha and Samanta ().
B
Suppose that we have a single model M that specifies a density f (x∣θ), θ ∈ Θ for the observable X and the Bayesian has a prior π(θ). The Bayesian wishes to examine how well the model M fits the data xobs on the basis of a statistic T(X) which measures the goodness of fit of data and model. Of course, T is also chosen by the Bayesian even though it is not part of the usual paradigm for Bayesian inference. Let mπ (x) = ∫Θ f (x∣θ)π(θ) dθ be the prior predictive density. Box () defines p = ∫{T(x)>T(xobs )} mπ (x) dx as a prior predictive P-value. Of course, this depends on the prior π of the Bayesian. To reduce the dependence on π and also to make it possible to use an improper non-informative prior π with proper posterior π(θ∣xobs ), Gutman (), Rubin (), Meng () and Gelman et al. () propose a posterior predictive P-value p∗ defined as follows. Let m∗ (x∣xobs ) = ∫ f (x∣θ)π(θ∣xobs ) dθ, Θ
∗
m∗ (x∣xobs ) dx.
p =∫ {T(x)>T(x obs )}
However, as pointed out by Bayarri and Berger (), p∗ involves a double use of the data in both the integrand and the tail area of the integrand defining p∗ . In order to remove this undesirable feature, Bayarri and Berger () introduce what they call a conditional predictive P-value which is defined as follows. Identify a statistic U(X) which is not a function of T(X) and let m(t∣u) be the conditional predictive density of T given U. Then the conditional predictive P-value is pc = ∫
m(t∣uobs ) dt.
{T(x)>T(x obs )}
Bayarri and Berger () also define a partial posterior predictive P-value in the same vein. This alternative P-value does not require the choice of the auxiliary statistic U and to that extent is less arbitrary. When the model is simple, meaning that the distribution of the random observable has no unknown parameters, then under this model, all these P-values reduce to the tail area probability under this distribution (which is exactly the classical P-value). In this case, the P-value treated as random (i.e., p = p(X)) has the U(, ) distribution, a desirable property as far as its interpretation is concerned. This property is desirable even when the model is composite and the nuisance parameters are eliminated
B
B
Bayesian P-Values
in some way. However, this is not always likely and what can be expected is that they be asymptotically uniformly distributed. Robins et al. () argue that in the absence of the U(, ) property, one should at least require that these P-values not suffer from serious conservative or anticonservative behavior; p(X) is defined to be conservative (anti-conservative) when P(p(X) < u) is smaller (larger) than u for all u < /, with P denoting the distribution of X under (any θ from) the model. Assuming certain regularity conditions on the model as well as the prior, Robins et al. () prove that the conditional predictive and partial predictive P-values are asymptotically uniformly distributed, whereas the posterior predictive P-value is often conservative. The different P-values are illustrated below with two examples from Bayarri and Berger (, ). Example Suppose X = (X , X , . . . , Xn ) is a random sample from some distribution, and we want to check if it is N(µ , σ ), σ unknown and µ is a specified value for its ¯ ). mean. The natural discrepancy statistic is T(X) = (X−µ Consider the usual non-informative prior π(σ ) ∝ /σ . Then the prior predictive P-value doesn’t exist since this ¯ /n. We obtain prior is improper. Let s = ∑ni= (Xi − X) ) /(σ )), π(σ ∣xobs ) ∝ (σ )−n/− exp (−n (sobs + tobs
and thus the posterior predictive density of T is m(t∣xobs ) ∝ ( +
nt ) n sobs + tobs
−(n+)/
∞
exp (−nv {sobs + t }) vn/
∝∫
∝ ( +
(n − )t ) n − sobs
dv v
−n/
.
This implies, under the conditional predictive distribution, √ T ∼ tn− . n− sobs The conditional predictive P-value, therefore, is pc = { − Tn− (
√ n − tobs )} . sobs
Bayarri and Berger (, ) show that in this example the partial predictive P-value and the conditional predictive P-value coincide. √ ∗ Listed in Table√ are values of p√ , tobs = n − (¯xobs − /(n − ) correµ )/sobs and t ∗ = n/(n − )tobs / + tobs sponding to different values of n when pc is fixed at ., . and ., respectively. Example Suppose, as in the previous example, X , X , . . . , Xn is a random sample from some population. The target model now is Exponential(λ). Consider π(λ) ∝ /λ. Let T = X() be the model checking statistic and let S = n ∑i= Xi . The posterior density of λ given S = sobs is proportional to λ n− exp(−sobs λ) so that the posterior predictive density of T given S = sobs is
.
n snobs (nt + sobs )−(n+) .
therefore the posterior predictive P-value is √ ⎧ ⎫ ⎪ ⎛ ntobs ⎞⎪ ⎪ ⎪ p∗ = ⎨ − T n √ ⎬, ⎠⎪ ⎪ ⎝ sobs + tobs ⎪ ⎪ ⎩ ⎭ with Tν denoting the c.d.f. of Student’s tν . Choose U(X) ≡ s , and note that nU∣σ ∼ σ χ n− . Therefore, π(σ ∣U = s ) ∝ (σ )−(n−)/− exp(−ns /(σ )), and consequently, the conditional predictive density of T given U = sobs is m(t∣sobs ) = ∫
∞ ∞
∝∫
fT (t∣σ )π (σ ∣sobs ) dσ nt −(n−)/ (σ )−/ exp (− ) (σ ) σ
n dσ exp (− sobs ) σ σ
Bayesian P-Values. Table P-values: p∗ versus pc for the normal model n pc = .
tobs
pc = .
∗
∗
pc = .
t
p
pc = . tobs pc = . t
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
∗
.
.
.
.
.
.
.
pc = . p∗
.
.
.
.
.
.
.
pc = .
tobs
.
.
.
.
.
.
.
pc = .
t∗
.
.
.
.
.
.
.
pc = .
∗
.
.
.
.
.
.
.
p
Bayesian P-Values
Bayesian P-Values. Table P-values: p∗ versus pc for the exponential model n
t∗ = .
pc
t∗ = .
p∗
.
.
.
t∗ = .
pc
.
.
.
t∗ = .
p∗
.
.
.
.
t∗ = .
pc
.
.
.
t = .
p
.
.
.
.
t∗ = .
pc
.
.
.
.
t = .
p
.
.
.
.
t∗ = .
pc
.
.
.
.
t = .
p
∗
.
.
.
.
.
t∗ = .
pc
.
.
.
.
.
t = .
p
.
.
.
.
.
∗
∗
∗
∗
∗
∗
∗
This yields the posterior predictive P-value of p∗ = ( +
ntobs −n ) . sobs
A direct calculation gives f (x∣T = t, λ) ∝ λ n− exp(−λ(s − nt)), from which the partial posterior density for λ is seen to be λ n− exp(−λ(sobs − ntobs )) ; Γ(n − )(sobs − ntobs )−(n−) This is called partial posterior because it is obtained from the partial likelihood, f (xobs ∣T = tobs , λ) instead of the full likelihood f (xobs ∣λ). Then the partial posterior predictive density of T is obtained as n(n − )(sobs − ntobs )n− . (nt + sobs − ntobs )n This leads to the following expression for the partial posterior predictive P-value: ( −
ntobs n− ) . sobs
Bayarri and Berger () go on to show that this coincides with the conditional predictive P-value pc upon
B
taking the conditioning (auxiliary) statistic to be the MLE of λ from f (x∣T = t, λ) (which is given by (n − )/(S − nT). In Table , values of p∗ and pc corresponding to some values of t ∗ = ntobs /sobs and n are displayed.
About the Authors Dr Jayanta Kumar Ghosh (born May , ) is Professor of Statistics, Department of Statistics, Purdue University. He graduated from the University of Calcutta. He has spent a substantial part of his career in the Indian Statistical Institute, Calcutta. He served as the Director of ISI and later held the Jawaharlal Nehru Professorship. “Professor Jayanta Kumar Ghosh, or J. K. Ghosh, as he is commonly known, has been a prominent contributor to the discipline of statistics for five decades. The spectrum of his contributions encompasses sequential analysis, the foundations of statistics, Finite populations, Edgeworth expansions, second order efficiency, Bartlett corrections, noninformative, and especially matching priors, semiparametric inference, posterior limit theorems, Bayesian non-parametrics, model selection, Bayesian hypothesis testing and high dimensional data analysis, as well as some applied work in reliability theory, statistical quality control, modeling hydrocarbon discoveries, geological mapping and DNA Fingerprinting.” [B. Clarke and S. Ghosal (). J. K. Ghosh’s contribution to statistics: A brief outline, In: Pushing the Limits of Contemporary Statistics: Contributions in Honor of Jayanta K. Ghosh, IMS Collections, Institute of Mathematical Statistics, Beachwood, Ohio (), p. .] He has (co-)authored over papers and several books, including: Higher Order Asymptotics (published jointly by Institute of Mathematical Statistics and American Statistical Association, ), Bayesian Nonparametrics (with R.V. Ramamoorthi, Springer, ), and An Introduction to Bayesian Analysis, Theory and Methods (with M. Delampady and T. Samanta, Springer, ). He has supervised more than PhD students. He was awarded the Shanti Swarup Bhatnagar Award for Mathematical Science (), Mahalanobis Gold Medal of Indian Science Congress Association () and P.V. Sukhatme Prize for Statistics (). Professor Jayanta Ghosh is a Fellow of the Indian Academy of Sciences, Fellow of the Indian National Science Academy, and Vice-President of the Calcutta Statistical Association. He is Past President, International Statistical Institute (). Mohan Delampady is Professor, Theoretical Statistics and Mathematics Division, Indian Statistical Institute, Bangalore. He was awarded the Professor L.J. Savage prize for excellence in research towards a Ph.D. thesis in Bayesian econometrics and statistics by NBER and NSF in .
B
B
Bayesian Reliability Modeling
Cross References 7Bayesian Statistics 7P-Values 7P-Values, Combining of
References and Further Reading Bayarri MJ, Berger J () Quantifying surprise in the data and model verification. In Bernardo JM et al. (eds) Bayesian statistics , Oxford University Press, Oxford, pp – Bayarri MJ, Berger JO () P values for composite null models. J Am Stat Assoc :–, – (discussion) Box GEP () Sampling and Bayes inference in scientific modeling and robustness. J R Stat Soc (Series A) :– Gelman A, Meng X, Stern H () Posterior predictive assessment of model fitness via realized discrepancies. Stat Sinica :– (with discussion) Ghosh JK, Delampady M, Samanta T () An introduction to Bayesian analysis: theory and methods. Springer, New York Ghosh JK, Purkayastha S, Samanta T () Role of P-values and other measures of evidence in Bayesian analysis. In: Dey DK, Rao CR (eds) Handbook of statistics, , Bayesian thinking: modeling and computation. pp – Guttman I () The use of the concept of a future observation in goodness-of-fit problems. J R Stat Soc (Series B) :– Meng XL () Posterior predictive p-values. Ann Stat :– Robins JM, van der Vaart A, Ventura V () Asymptotic distribution of P Values for composite null models. J Am Stat Assoc :–, – (discussion) Rubin DB () Bayesianly justifiable and relevant frequency calculations for the applied statistician. Ann Stat :–
from proving ground sources, factors in a product operating environment and any partially relevant evidence, say, warranty data, customer research surveys, etc. The basic form of Bayesian reliability modeling can be illustrated by following formulation and example. When evidence of system performance denoted by x, is scarce, a reliability (or quality) index of the system, denoted by s ( ≤ s ≤ ), we may use the soft evidence on s in the form of a prior density ρ(s) in terms of 7Bayes’ theorem to calculate the posterior density of s, f (s∣x) =
l(s∣x)ρ(s)
. ∫ l(s∣x)ρ(s)ds
()
∆
where l(s∣x) = f (x∣s) is called the likelihood function, obtained from the joint density of the sample evidence x. For example, N electronic devices are under testing until a preset time T, by assuming that the failure time of a random individual device follows an exponential distribution with the density: f (t∣λ) = λe−λt , t ≥ , λ > .
()
Suppose that in testing period [, T], x units failed and the failure times are recorded as t , t , ⋯, tx , thus the sample evidence as t = (t , t , ⋯, tx , T, T, ⋯, T), and the likelihood function is then given by x
l(λ∣t) = [∏ (λe
−λt i
)] [ − ( − e−λT )]
n−x
= λ x e−λη , ()
i= x
where η = (n − x) T + ∑ti . Then a gamma prior density i=
Bayesian Reliability Modeling
on the failure rate λ, with priori parameters α and β, may represent the expert knowledge: ρ(λ) =
β α β− −α λ λ e , λ > , α ≥ , β ≥ . Γ (α)
()
Renkuan Guo Professor University of Cape Town, Cape Town, South Africa
The posterior density of the failure rate λ is evaluated in terms of Eq. :
Bayesian reliability modeling is an application of rigorous Bayesian statistical inference theory, one of the frontiers of modern statistics. It is particularly useful in the context of the scarcity of system failure (or quality index) data. Bayesian decision theory can guide reliability engineers to utilize both “soft” and “hard” evidence relevant to the reliability index under investigation. Soft evidence includes expert knowledge, design and performance of similar products, and their mathematical treatment, etc. In contrast, hard evidence includes the direct failure (or quality testing) data, any other transformable evidence having functional relationship with failure rate, such as test data
l(λ∣x)ρ(λ) (α + η) β+x β+x+ −(α+η)λ λ e , = Γ (α + x) ∫ l(λ∣x)ρ(λ)dλ () which is also a gamma density because the prior density takes the form of the conjugate family. Further analysis or inference will be based on the posterior density. It is necessary to emphasize that in reliability engineering reality, the formations and applications of Bayesian reliability modeling are far more complicated and diversified than that shown as the basic form in Eq. because the concrete formation of an individual reliability problem is heavily dependent upon the compositional form of Bayesian decision criterion engaged, the form of distribution of f (λ∣x) =
Bayesian Reliability Modeling
quality index and hence the form of the likelihood function, which is inevitably linked to a host of factors which may have serious impacts. The factors include any sample censoring mechanism, the form of prior and the design or physical structure of the system under study, the manner of collecting evidence (e.g., sequential or otherwise) and even research progress of the relevant mathematical branches. For example, Dai et al. (), demonstrated that graph theory related fault tree analysis is a common classical reliability analysis model, while the Bayesian counterpart, Bayesian networks (abbreviated as BN), is just a combination of fault trees and appropriate arrangement of conditional probability and on probability assessments which can combine the soft and hard evidence under a Bayesian risk criterion. Bayesian networks can model the complicated physical structure as well as failure dependence structures of a complex system and predict the system failure behavior. A dynamic version of BN (DBN) was also developed for large system reliability modeling. Artificial 7Neural Networks form a powerful regression modeling tool, permitting a non-linear analysis. Combination of a Bayesian theoretical frame and Standard Neural Networks structure creates the Bayesian Neural Networks (abbreviated as BNNs), which allow active Bayesian reliability modeling for complex systems with complicated sample evidence. Mathematically, BNNs are nothing but probabilistic networks. For details, see Waszczyszyn et al. (). Markov chain Monte Carlo (abbreviated as MCMC) simulation (see 7Markov Chain Monte Carlo) is another frontier of modern computational statistics. Merging of BNN and MCMC has created an extremely powerful but convenient computational reliability engineering model type. It is well-known that the conventional Bayesian risk criterion is based on quadratic loss function and use of a conjugate family. Maximum Entropy modeling is an important Bayesian inference. The reliability engineering community has made efforts in this direction. However, Maximum entropy (abbreviated as Maxent) modeling software development and application in environmental context may improve from attention to Bayesian reliability modeling efforts. See Phillips et al. (). Therefore, Bayesian reliability modeling is widely applied in business and industries, particularly in complex systems, nano-electronics and software industry, say, accelerating life testing models, reliability growth models, testing models for new product design, etc. See Blischke and Murthy (), Kuo (), and Garg et al. (). As we pointed out at the beginning, Bayesian reliability modeling is a small-sample inference in its mathematical nature. Hence there is no reason to ignore
B
the other developments in small-sample inference. It will be beneficial to combine some elements in small-sample inference with developments in the Bayesian reliability modeling. Jin et al. () illustrated how to use a “grey differential equation” approach to improve BNNs in software reliability modeling, although the “grey” concept is not yet well-accepted in mathematical and statistical societies. Guo et al. () proposed a differential equation associated regression (abbreviated as DEAR) model, which is small-sample based inference with a rigorous mathematical foundations and resolves the problem in “grey differential equations.” Ushakov and Harrison () offered a systematic treatment in Chap. . Singpurwalla’s book () is authoritative in Bayesian reliability modeling, and reflects the author’s state-of-art modeling experiences and judgments.
About the Author Dr Renkuan Guo is a Professor in the Department of Statistical Sciences, University of Cape Town, South Africa. He served as Regional Editor for Economic Quality Control. His research work and publications ( papers and four book chapters) can be divided into three aspects: () from to , he focused on reliability modeling, including Cox PH models, and Kijima age models; () from to , he concentrated on fuzzy reliability and quality modeling, including credibilistic fuzzy reliability models, grey differential equation models, random fuzzy models, particularly, the DEAR (Differential Equation Associated Regression) models; () from up to now, he starts initiating the uncertainty statistics on Professor Baoding Liu’s axiomatic uncertain measure foundation, including uncertain Bayesian decision model, uncertain reliability models and hybrid reliability models with uncertain parameter.
Cross References 7Degradation Models in Reliability and Survival Analysis 7Imprecise Reliability 7Industrial Statistics 7Parametric and Nonparametric Reliability Analysis
References and Further Reading Blischke WR, Murthy DNP () Reliability – modeling, prediction, and optimization. Wiley, New York Dai YS, Xie M, Long Q, Ng SH () Uncertainty analysis in software reliability modeling by Bayesian analysis with maximumentropy principle. IEEE Trans Software Eng ():– Garg RK, Gupta VK, Agrawal VP () Reliability modelling and analysis of a tribo-mechanical system. Int J Model Simulat ():–.
B
B
Bayesian Semiparametric Regression
Guo R, Guo D, Tim Dunne T, Thiart C () DEAR Model – The theoretical foundation. J Uncertain Syst ():– Jin A, Jiang JH, Lou JG, Zhang R () Software reliability modeling based on grey system theory. J Comput Appl ():– Kuo W () Challenges related to reliability in nano electronics. IEEE Trans Reliab ():– Phillips SJ, Anderson RP, Schapire RE () Maximum entropy modeling of species geographic distributions. Ecol Model :– Singpurwalla ND () Reliability and risk - a Bayesian perspective. Wiley, New York Ushakov IA, Harrison RA () Handbook of reliability engineering. Wiley, New York Waszczyszyn Z, Slonski M, Miller B, Piatkowski G () Bayesian neural networks in the regression analysis of structural mechanics problems. In: th World congress on computational mechanics (WCCM), th European congress on computational methods in applied sciences and engineering (ECCOMAS ), Venice, Italy, June– July,
(c) Heterogeneity among individuals or units may be insufficiently described by covariates. Hence, unobserved unit- or cluster specific heterogeneity must be considered appropriately. (d) Interactions between covariates may be of complex, nonlinear form. 7Semiparametric regression models extend models with
linear predictors by incorporating additional non- and semiparametric components. Bayesian semiparametric regression regularizes the resulting high-dimensional inferential problem by imposing appropriate priors.
Observation Models We consider some semiparametric regression models that may be considered as special classes of structured additive regression (STAR) models (Fahrmeir et al. ). Generalized additive models (GAMs) extend the linear predictor of GLMs to η i = x′i β + f (z ) + . . . + fp (zip ),
Bayesian Semiparametric Regression Ludwig Fahrmeir Professor Ludwig-Maximilians-University, Munich, Germany
i = , . . . , n
()
where fj are smooth functions of continuous covariates z , . . . , zp . Most semiparametric regression approaches assume that unknown functions are represented or approximated through a linear combination of basis functions, i.e., K
f (z) = ∑ γ k Bk (z) k=
Linear or 7generalized linear models assume that the (conditional) mean µ = E(y∣x), of the response y, given the covariate vector x, is linked to a linear predictor µ by µ = h(η),
η = x′ β.
Here, h is a known response function and β is an unknown vector of regression parameters. More generally, other characteristics of the response distribution, such as 7variance or 7skewness may be related to covariates in similar manner. Another example is the Cox model for 7survival data, where the hazard rate is assumed to have the form λ(t∣x) = λ (t) exp(x′ β) with λ (t) as an (unspecified) baseline hazard rate. In most practical regression situations, however, we are facing at least one of the following problems. (a) For the continuous covariates in the data set, the assumption of a strictly linear effect on the predictor may not be appropriate. (b) Observations may be spatially correlated.
for a typical function f . The most popular basis function representations are spline functions, with truncated power series or B-spline basis functions Bk (z), a relatively large number of knots, and a correspondingly high-dimensional vector γ of basis function coefficients γ , ..., γ K . Collecting all predictors η i in the predictor vector η, and constructing appropriate design matrix, the predictor () can be rewritten in matrix notation as a highdimensional linear predictor η = Xβ + Zγ + . . . + Zpγ p .
()
GAMs can be extended by additively incorporating e.g., interaction terms f∣ (z , z ) of two continuous covariates, varying coefficient terms g(z)x, where the effect of x varies with z, a spatial effect fspat (s) where s denotes spatial location and individual – or group specific i.i.d. random effects α g , g = , . . . , G. Combining these different types of effects in additive form leads to STAR models with generalized additive mixed models (GAMMs), varying coefficient models and geoadditive models as important subclasses. The Cox model can be extended in quite similar fashion, see Hennerfeind et al. () and Kneib and Fahrmeir (). It turns out that after appropriate definition of
Bayesian Statistics
design matrices and coefficient vectors, the predictor still is of additive structure as in ().
B
Cross References 7Bayesian Statistics 7Semiparametric Regression Models
Priors and Inference For Bayesian inference, flat priors p(β) ∝ const or weakly informative Gaussian priors are usually assumed for linear effects. In a Gaussian smoothness prior approach, it turns out that all priors for parameters γ j representing smooth functions, Gaussian Markov random fields for spatial effects, i.i.d. random effects, etc., have the same generic conditionally Gaussian form ⎞ ⎛ ′ p (γγ j ∣τ j ) ∝ exp − γ j Kjγ j . ⎠ ⎝ τ j
()
The precision matrix Kj depends on the specific effect and acts as a penalty matrix to enforce smoothness, and τ j is an inverse smoothing parameter, controlling the amount of smoothness. In full Bayesian inference, a hyperprior is assigned to τ j , and regression and smoothness parameters are estimated jointly through MCMC techniques (Brezger and Lang ). Another possibility is to look at () and () as a mixed model with correlated random effects, enabling empirical Bayes inference with mixed model technology, see Fahrmeir et al. () and Ruppert et al. (). A recent review on (Bayesian) semiparametric regression is Ruppert et al. (). A forthcoming book (Fahrmeir and Kneib ) provides details on all issues. Approximate full Bayesian inference avoiding MCMC has been recently proposed by Rue et al. (). A somewhat different approach to Bayesian inference in semiparametric regression is based on adaptive selection of knots of B-splines or coefficients of basis functions through concepts of Bayesian variable selection, see for example the book by Denison et al. () or Smith et al. (), Kohn et al. ().
About the Author Ludwig Fahrmeir is a Professor, Department of Statistics, Ludwig-Maximilians-University Munich, Germany. He was Chairman of the Collaborative Research Centre “Statistical Analysis of Discrete Structures with Applications in Econometrics and Biometrics” (–) and is currently coordinator of the project “Analysis and Modelling of Complex Systems in Biology and Medicine” at the University of Munich. He is an Elected Fellow of the International Statistical Institute. He has authored and coauthored more than papers and seven books, including Multivariate statistical modelling based on generalized linear models (with Tutz, G., nd enlarged edition, Springer Series in Statistics, New York, ).
References and Further Reading Brezger A, Lang S () Generalized structured additive regression based on Bayesian P-splines. Comput Stat Data Anal : – Denison DGT, Holmes CC, Mallick BK, Smith AFM () Bayesian methods for nonlinear classification and regression. Wiley, Chichester Fahrmeir L, Kneib T () Bayesian smoothing and regression of longitudinal, spatial and event history data. Oxford University Press, to appear Fahrmeir L, Kneib T, Lang S () Penalized structured additive regression for space-time data: a Bayesian perspective. Stat Sinica :– Hennerfeind A, Brezger A, Fahrmeir L () Geoadditive survival models. J Am Stat Assoc :– Kneib T, Fahrmeir L () Structured additive regression for multi-categorical space-time data: a mixed model approach. Biometrics :– Kohn R, Smith M, Chan D () Nonparametric regression using linear combinations of basis functions. Stat Comput : – Rue H, Martino S, Chopin N () Approximate Bayesian inference for latent gaussian models by using integrated nested laplace approximations. J R Stat Soc B :– Ruppert D, Wand M, Carroll RJ () Semiparametric regression. Cambridge University Press, Cambridge Ruppert D, Wand MP, Carroll RJ () Semiparametric regression during –. Electron J Stat :– Smith M, Kohn R, Yau P () Nonparametric Bayesian bivariate surface estimation. In: Schimek G (ed) Smoothing and regression, Ch . Wiley, New York
Bayesian Statistics José M. Bernardo Professor of Statistics, Facultad de Matemáticas Universitat de València, Burjassot, Spain
Introduction Available observations generally consist of (possibly many) sets of data of the general form D = {x , . . . , xn }, where the xi ’s are somewhat “homogeneous” (possibly multidimensional) observations xi . Statistical methods are then typically used to derive conclusions on both the nature of the process which has produced those observations, and on the expected behavior at future instances of the same process. A central element of any statistical analysis is the specification of a probability model which is assumed to describe
B
B
Bayesian Statistics
the mechanism which has generated the observed data D as a function of a (possibly multidimensional) parameter (vector) ω ∈ Ω, sometimes referred to as the state of nature, about whose value only limited information (if any) is available. All derived statistical conclusions are obviously conditional on the assumed probability model. Unlike most other branches of mathematics, conventional methods of statistical inference suffer from the lack of an axiomatic basis; as a consequence, their proposed desiderata are often mutually incompatible, and the analysis of the same data may well lead to incompatible results when different, apparently intuitive procedures are tried (see Lindley () and Jaynes () for many instructive examples). In marked contrast, the Bayesian approach to statistical inference is firmly based on axiomatic foundations which provide a unifying logical structure, and guarantee the mutual consistency of the methods proposed. Bayesian methods constitute a complete paradigm to statistical inference, a scientific revolution in Kuhn’s sense. Bayesian statistics only require the mathematics of probability theory and the interpretation of probability which most closely corresponds to the standard use of this word in everyday language: it is no accident that some of the more important seminal books on Bayesian statistics, such as the works of Laplace (), Jeffreys () or de Finetti () are actually entitled “Probability Theory.” The practical consequences of adopting the Bayesian paradigm are far reaching. Indeed, Bayesian methods () reduce statistical inference to problems in probability theory, thereby minimizing the need for completely new concepts, and () serve to discriminate among conventional statistical techniques, by either providing a logical justification to some (and making explicit the conditions under which they are valid), or proving the logical inconsistency of others. The main consequence of these foundations is the mathematical need to describe by means of probability distributions all uncertainties present in the problem. In particular, unknown parameters in probability models must have a joint probability distribution which describes the available information about their values; this is often regarded as the characteristic element of a Bayesian approach. Notice that (in sharp contrast to conventional statistics) parameters are treated as random variables within the Bayesian paradigm. This is not a description of their variability (parameters are typically fixed unknown quantities) but a description of the uncertainty about their true values. An important particular case arises when either no relevant prior information is readily available, or that
information is subjective and an “objective” analysis is desired, one that is exclusively based on accepted model assumptions and well-documented data. This is addressed by reference analysis, which uses information-theoretic concepts to derive appropriate reference posterior distributions, defined to encapsulate inferential conclusions on the quantities of interest solely based on the supposed model and the observed data. In this article it is assumed that probability distributions may be described through their probability density functions, and no distinction is made between a random quantity and the particular values that it may take. Bold italic roman fonts are used for observable random vectors (typically data) and bold italic greek fonts are used for unobservable random vectors (typically parameters); lower case is used for variables and upper case for their dominion sets. Moreover, the standard mathematical convention of referring to functions, say f and g of x ∈ χ, respectively by f (x) and g(x), will be used throughout. Thus, p(θ ∣ C) and p(x ∣ C) respectively represent general probability densities of the random vectors θ ∈ Θ and x ∈ χ under conditions C, so that p(θ ∣ C) ≥ , ∫Θ p(θ ∣ C) dθ = , and p(x ∣ C) ≥ , ∫ χ p(x ∣ C) dx = . This admittedly imprecise notation will greatly simplify the exposition. If the random vectors are discrete, these functions naturally become probability mass functions, and integrals over their values become sums. Density functions of specific distributions are denoted by appropriate names. Thus, if x is a random quantity with a normal distribution of mean µ and standard deviation σ, its probability density function will be denoted N(x ∣ µ, σ). Bayesian methods make frequent use of the concept of logarithmic divergence, a very general measure of the goodness of the approximation of a probability density p(x) by another density pˆ (x). The Kullback-Leibler, or logarithmic divergence of a probability density pˆ (x) of the random vector x ∈ χ from its true probability density p(x), is defined as δ{pˆ (x) ∣ p(x)} = ∫ χ p(x) log{p(x)/pˆ (x)} dx. It may be shown that () the logarithmic divergence is nonnegative (and it is zero if, and only if, pˆ (x) = p(x) almost everywhere), and () that δ{pˆ (x) ∣ p(x)} is invariant under one-to-one transformations of x. This article contains a brief summary of the mathematical foundations of Bayesian statistical methods (section “7Foundations”), an overview of the paradigm (section “7The Bayesian Paradigm”), a description of useful inference summaries, including both point an region estimation and hypothesis testing (section “7Inference Summaries”), an explicit discussion of objective Bayesian methods (section “7Reference Analysis”), and a final
Bayesian Statistics
discussion which includes pointers to further issues not addressed here (section “7Discussion”).
Foundations A central element of the Bayesian paradigm is the use of probability distributions to describe all relevant unknown quantities, interpreting the probability of an event as a conditional measure of uncertainty, on a [, ] scale, about the occurrence of the event in some specific conditions. The limiting extreme values and , which are typically inaccessible in applications, respectively describe impossibility and certainty of the occurrence of the event. This interpretation of probability includes and extends all other probability interpretations. There are two independent arguments which prove the mathematical inevitability of the use of probability distributions to describe uncertainties; these are summarized later in this section.
Probability as a Measure of Conditional Uncertainty Bayesian statistics uses the word probability in precisely the same sense in which this word is used in everyday language, as a conditional measure of uncertainty associated with the occurrence of a particular event, given the available information and the accepted assumptions. Thus, Pr(E ∣ C) is a measure of (presumably rational) belief in the occurrence of the event E under conditions C. It is important to stress that probability is always a function of two arguments, the event E whose uncertainty is being measured, and the conditions C under which the measurement takes place; “absolute” probabilities do not exist. In typical applications, one is interested in the probability of some event E given the available data D, the set of assumptions A which one is prepared to make about the mechanism which has generated the data, and the relevant contextual knowledge K which might be available. Thus, Pr(E ∣ D, A, K) is to be interpreted as a measure of (presumably rational) belief in the occurrence of the event E, given data D, assumptions A and any other available knowledge K, as a measure of how “likely” is the occurrence of E in these conditions. Sometimes, but certainly not always, the probability of an event under given conditions may be associated with the relative frequency of “similar” events in “similar” conditions. The following examples are intended to illustrate the use of probability as a conditional measure of uncertainty. Probabilistic diagnosis. A human population is known to contain .% of people infected by a particular virus. A person, randomly selected from that population, is subject to a test which, from laboratory data, is known to yield positive results in % of infected people and in % of non-infected, so that, if V denotes the event that a
B
person carries the virus and + denotes a positive result, Pr(+ ∣ V) = . and Pr(+ ∣ V) = .. Suppose that the result of the test turns out to be positive. Clearly, one is then interested in Pr(V ∣ +, A, K), the probability that the person carries the virus, given the positive result, the assumptions A about the probability mechanism generating the test results, and the available knowledge K of the prevalence of the infection in the population under study (described here by Pr(V ∣ K) = .). An elementary exercise in probability algebra, which involves 7Bayes’ theorem in its simplest form (see section “7The Bayesian Paradigm”), yields Pr(V ∣ +, A, K) = .. Notice that the four probabilities involved in the problem have the same interpretation: they are all conditional measures of uncertainty. Besides, Pr(V ∣ +, A, K) is both a measure of the uncertainty associated with the event that the particular person who tested positive is actually infected, and an estimate of the proportion of people in that population (about .%) that would eventually prove to be infected among those which yielded a positive test. Estimation of a proportion. A survey is conducted to estimate the proportion θ of individuals in a population who share a given property. A random sample of n elements is analyzed, r of which are found to possess that property. One is then typically interested in using the results from the sample to establish regions of [, ] where the unknown value of θ may plausibly be expected to lie; this information is provided by probabilities of the form Pr(a < θ < b ∣ r, n, A, K), a conditional measure of the uncertainty about the event that θ belongs to (a, b) given the information provided by the data (r, n), the assumptions A made on the behavior of the mechanism which has generated the data (a random sample of n Bernoulli trials), and any relevant knowledge K on the values of θ which might be available. For example, after a political survey in which citizens out of a random sample of have declared their support to a particular political measure, one may conclude that Pr(θ < . ∣ , , , A, K) = ., indicating a probability of about % that a referendum of that issue would be lost. Similarly, after a screening test for an infection where people have been tested, none of which has turned out to be infected, one may conclude that Pr(θ < . ∣ , , A, K) = ., or a probability of about % that the proportion of infected people is smaller than %. Measurement of a physical constant. A team of scientists, intending to establish the unknown value of a physical constant µ, obtain data D = {x , . . . , xn } which are considered to be measurements of µ subject to error. The probabilities of interest are then typically of the form
B
B
Bayesian Statistics
Pr(a < µ < b ∣ x , . . . , xn , A, K), the probability that the unknown value of µ (fixed in nature, but unknown to the scientists) lies within an interval (a, b) given the information provided by the data D, the assumptions A made on the behavior of the measurement mechanism, and whatever knowledge K might be available on the value of the constant µ. Again, those probabilities are conditional measures of uncertainty which describe the (necessarily probabilistic) conclusions of the scientists on the true value of µ, given available information and accepted assumptions. For example, after a classroom experiment to measure the gravitational field with a pendulum, a student may report (in m/sec ) something like Pr(. < g < . ∣ D, A, K) = ., meaning that, under accepted knowledge K and assumptions A, the observed data D indicate that the true value of g lies within . and . with probability ., a conditional uncertainty measure on a [,] scale. This is naturally compatible with the fact that the value of the gravitational field at the laboratory may well be known with high precision from available literature or from precise previous experiments, but the student may have been instructed not to use that information as part of the accepted knowledge K. Under some conditions, it is also true that if the same procedure were actually used by many other students with similarly obtained data sets, their reported intervals would actually cover the true value of g in approximately % of the cases, thus providing some form of calibration for the student’s probability statement (see section “7Frequentist Properties”). Prediction. An experiment is made to count the number r of times that an event E takes place in each of n replications of a well defined situation; it is observed that E does take place ri times in replication i, and it is desired to forecast the number of times r that E will take place in a future, similar situation. This is a prediction problem on the value of an observable (discrete) quantity r, given the information provided by data D, accepted assumptions A on the probability mechanism which generates the ri ’s, and any relevant available knowledge K. Hence, simply the computation of the probabilities {Pr(r ∣ r , . . . , rn , A, K)}, for r = , , . . ., is required. For example, the quality assurance engineer of a firm which produces automobile restraint systems may report something like Pr(r = ∣ r = . . . = r = , A, K) = ., after observing that the entire production of airbags in each of n = consecutive months has yielded no complaints from their clients. This should be regarded as a measure, on a [, ] scale, of the conditional uncertainty, given observed data, accepted assumptions and contextual knowledge, associated with the event that no airbag complaint will come from next month’s production and, if conditions remain constant, this is also an
estimate of the proportion of months expected to share this desirable property. A similar problem may naturally be posed with continuous observables. For instance, after measuring some continuous magnitude in each of n randomly chosen elements within a population, it may be desired to forecast the proportion of items in the whole population whose magnitude satisfies some precise specifications. As an example, after measuring the breaking strengths {x , . . . , x } of ten randomly chosen safety belt webbings to verify whether or not they satisfy the requirement of remaining above kN, the quality assurance engineer may report something like Pr(x > ∣ x , . . . , x , A, K) = .. This should be regarded as a measure, on a [, ] scale, of the conditional uncertainty (given observed data, accepted assumptions and contextual knowledge) associated with the event that a randomly chosen safety belt webbing will support no less than kN. If production conditions remain constant, it will also be an estimate of the proportion of safety belts which will conform to this particular specification. Often, additional information of future observations is provided by related covariates. For instance, after observing the outputs {y , . . . , yn } which correspond to a sequence {x , . . . , xn } of different production conditions, it may be desired to forecast the output y which would correspond to a particular set x of production conditions. For instance, the viscosity of commercial condensed milk is required to be within specified values a and b; after measuring the viscosities {y , . . . , yn } which correspond to samples of condensed milk produced under different physical conditions {x , . . . , xn }, production engineers will require probabilities of the form Pr(a < y < b ∣ x, (y , x ), . . . , (yn , xn ), A, K). This is a conditional measure of the uncertainty (always given observed data, accepted assumptions and contextual knowledge) associated with the event that condensed milk produced under conditions x will actually satisfy the required viscosity specifications.
Statistical Inference and Decision Theory Decision theory not only provides a precise methodology to deal with decision problems under uncertainty, but its solid axiomatic basis also provides a powerful reinforcement to the logical power of the Bayesian approach. We now summarize the basic argument. A decision problem exists whenever there are two or more possible courses of action; let A be the class of possible actions. Moreover, for each a ∈ A, let Θa be the set of relevant events which may affect the result of choosing a, and let c(a, θ) ∈ Ca , θ ∈ Θa , be the consequence of having chosen action a when event θ takes place. The class of
Bayesian Statistics
pairs {(Θa , Ca ), a ∈ A} describes the structure of the decision problem. Without loss of generality, it may be assumed that the possible actions are mutually exclusive, for otherwise one would work with the appropriate Cartesian product. Different sets of principles have been proposed to capture a minimum collection of logical rules that could sensibly be required for “rational” decision-making. These all consist of axioms with a strong intuitive appeal; examples include the transitivity of preferences (if a > a given C, and a > a given C, then a > a given C), and the surething principle (if a > a given C and E, and a > a given C and E, then a > a given C). Notice that these rules are not intended as a description of actual human decision-making, but as a normative set of principles to be followed by someone who aspires to achieve coherent decision-making. There are naturally different options for the set of acceptable principles, but all of them lead basically to the same conclusions, namely: . Preferences among consequences should necessarily be measured with a real-valued bounded utility function u(c) = u(a, θ) which specifies, on some numerical scale, their desirability. . The uncertainty of relevant events should be measured with a set of probability distributions {(p(θ ∣ C, a), θ ∈ Θ a ), a ∈ A} describing their plausibility given the conditions C under which the decision must be taken. . The desirability of the available actions is measured by their corresponding expected utility u(a ∣ C) = ∫
Θa
u(a, θ) p(θ ∣ C, a) dθ,
a ∈ A.
It is often convenient to work in terms of the non-negative loss function defined by ℓ(a, θ) = sup{u(a, θ)} − u(a, θ), a∈A
which directly measures, as a function of θ, the “penalty” for choosing a wrong action. The relative undesirability of available actions a ∈ A is then measured by their expected loss ℓ(a ∣ C) = ∫
Θa
ℓ(a, θ) p(θ ∣ C, a) dθ,
a ∈ A.
Notice that, in particular, the argument described above establishes the need to quantify the uncertainty about all relevant unknown quantities (the actual values of the θ’s), and specifies that this quantification must have the mathematical structure of probability distributions. These probabilities are conditional on the circumstances C under which the decision is to be taken, which typically, but
B
not necessarily, include the results D of some relevant experimental or observational data. It has been argued that the development described above (which is not questioned when decisions have to be made) does not apply to problems of statistical inference, where no specific decision making is envisaged. However, there are two powerful counterarguments to this. Indeed, () a problem of statistical inference is typically considered worth analyzing because it may eventually help make sensible decisions (as Ramsey () put it, a lump of arsenic is poisonous because it may kill someone, not because it has actually killed someone), and () it has been shown (Bernardo a) that statistical inference on θ actually has the mathematical structure of a decision problem, where the class of alternatives is the functional space A = {p(θ ∣ D);
p(θ ∣ D) > , ∫ p(θ ∣ D) dθ = } Θ
of the conditional probability distributions of θ given the data, and the utility function is a measure of the amount of information about θ which the data may be expected to provide.
Exchangeability and Representation Theorem Available data often take the form of a set {x , . . . , xn } of “homogeneous” (possibly multidimensional) observations, in the precise sense that only their values matter and not the order in which they appear. Formally, this is captured by the notion of exchangeability. The set of random vectors {x , . . . , xn } is exchangeable if their joint distribution is invariant under permutations. An infinite sequence {xj } of random vectors is exchangeable if all its finite subsequences are exchangeable. Notice that, in particular, any random sample from any model is exchangeable in this sense. The concept of exchangeability, introduced by de Finetti () put it, is central to modern statistical thinking. Indeed, the general representation theorem implies that if a set of observations is assumed to be a subset of an exchangeable sequence, then it constitutes a random sample from some probability model {p(x ∣ ω), ω ∈ Ω}, x ∈ X , described in terms of (labeled by) some parameter vector ω; furthermore this parameter ω is defined as the limit (as n → ∞) of some function of the observations. Available information about the value of ω in prevailing conditions C is necessarily described by some probability distribution p(ω ∣ C). For example, in the case of a sequence {x , x , . . .} of dichotomous exchangeable random quantities xj ∈ {, }, de Finetti’s representation theorem establishes that the
B
B
Bayesian Statistics
joint distribution of (x , . . . , xn ) has an integral representation of the form n
x −x ∏ θ i ( − θ) i p(θ ∣ C) dθ,
p(x , . . . , xn ∣ C) = ∫
θ = lim
i=
n→∞
r , n
where r = ∑ xj is the number of positive trials. This is nothing but the joint distribution of a set of (conditionally) independent Bernoulli trials with parameter θ, over which some probability distribution p(θ ∣ C) is therefore proven to exist. More generally, for sequences of arbitrary random quantities {x , x , . . .}, exchangeability leads to integral representations of the form n
p(x , . . . , xn ∣ C) = ∫ ∏ p(xi ∣ ω) p(ω ∣ C) dω, Ω i=
where {p(x ∣ ω), ω ∈ Ω} denotes some probability model, ω is the limit as n → ∞ of some function f (x , . . . , xn ) of the observations, and p(ω ∣ C) is some probability distribution over Ω. This formulation includes “nonparametric” (distribution free) modeling, where ω may index, for instance, all continuous probability distributions on X . Notice that p(ω ∣ C) does not describe a possible variability of ω (since ω will typically be a fixed unknown vector), but a description on the uncertainty associated with its actual value. Under appropriate conditioning, exchangeability is a very general assumption, a powerful extension of the traditional concept of a random sample. Indeed, many statistical analyses directly assume data (or subsets of the data) to be a random sample of conditionally independent observations from some probability model, so that p(x , . . . , xn ∣ ω) = n ∏i= p(xi ∣ ω); but any random sample is exchangeable, since ∏ni= p(xi ∣ ω) is obviously invariant under permutations. Notice that the observations in a random sample are only independent conditional on the parameter value ω; as nicely put by Lindley, the mantra that the observations {x , . . . , xn } in a random sample are independent is ridiculous when they are used to infer xn+ . Notice also that, under exchangeability, the general representation theorem provides an existence theorem for a probability distribution p(ω ∣ C) on the parameter space Ω, and that this is an argument which only depends on mathematical probability theory. Another important consequence of exchangeability is that it provides a formal definition of the parameter ω which labels the model as the limit, as n → ∞, of some function f (x , . . . , xn ) of the observations; the function f obviously depends both on the assumed model and the chosen parametrization. For instance, in the case of a
sequence of Bernoulli trials, the parameter θ is defined as the limit, as n → ∞, of the relative frequency r/n. It follows that, under exchangeability, the sentence “the true value of ω” has a well-defined meaning, if only asymptotically verifiable. Moreover, if two different models have parameters which are functionally related by their definition, then the corresponding posterior distributions may be meaningfully compared, for they refer to functionally related quantities. For instance, if a finite subset {x , . . . , xn } of an exchangeable sequence of integer observations is assumed to be a random sample from a Poisson distribution Po(x ∣ λ), so that E[x ∣ λ] = λ, then λ is defined as limn→∞ {xn }, where xn = ∑j xj /n; similarly, if for some fixed non-zero integer r, the same data are assumed to be a random sample for a negative binomial NBi(x ∣ r, θ), so that E[x ∣ θ, r] = r( − θ)/θ, then θ is defined as limn→∞ {r/(xn +r)}. It follows that θ ≡ r/(λ+r) and, hence, θ and r/(λ + r) may be treated as the same (unknown) quantity whenever this might be needed as, for example, when comparing the relative merits of these alternative probability models.
The Bayesian Paradigm The statistical analysis of some observed data D typically begins with some informal descriptive evaluation, which is used to suggest a tentative, formal probability model {p(D ∣ ω), ω ∈ Ω} assumed to represent, for some (unknown) value of ω, the probabilistic mechanism which has generated the observed data D. The arguments outlined in section “7Foundations” establish the logical need to assess a prior probability distribution p(ω ∣ K) over the parameter space Ω, describing the available knowledge K about the value of ω prior to the data being observed. It then follows from standard probability theory that, if the probability model is correct, all available information about the value of ω after the data D have been observed is contained in the corresponding posterior distribution whose probability density, p(ω ∣ D, A, K), is immediately obtained from Bayes’ theorem, p(ω ∣ D, A, K) =
p(D ∣ ω) p(ω ∣ K) ∫Ω p(D ∣ ω) p(ω ∣ K) dω
,
where A stands for the assumptions made on the probability model. It is this systematic use of Bayes’ theorem to incorporate the information provided by the data that justifies the adjective Bayesian by which the paradigm is usually known. It is obvious from Bayes’ theorem that any value of ω with zero prior density will have zero posterior density. Thus, it is typically assumed (by appropriate restriction, if necessary, of the parameter space Ω) that prior distributions are strictly positive (as Savage put it,
Bayesian Statistics
keep the mind open, or at least ajar). To simplify the presentation, the accepted assumptions A and the available knowledge K are often omitted from the notation, but the fact that all statements about ω given D are also conditional to A and K should always be kept in mind. Example (Bayesian inference with a finite parameter space). Let p(D ∣ θ), where θ ∈ {θ , . . . , θ m }, be the probability mechanism which is assumed to have generated the observed data D, so that θ may only take a finite number of values. Using the finite form of Bayes’ theorem, and omitting the prevailing conditions from the notation, the posterior probability of θ i after data D have been observed is Pr(θ i ∣ D) =
p(D ∣ θ i ) Pr(θ i ) , p(D ∣ θ j ) Pr(θ j )
∑m j=
i = , . . . , m.
For any prior distribution p(θ) = {Pr(θ ), . . . , Pr(θ m )} describing available knowledge about the value of θ, Pr(θ i ∣ D) measures how likely should θ i be judged, given both the initial knowledge described by the prior distribution, and the information provided by the data D. An important, frequent application of this simple technique is provided by probabilistic diagnosis. For example, consider the simple situation where a particular test designed to detect a virus is known from laboratory research to give a positive result in % of infected people and in % of non-infected. Then, the posterior probability that a person who tested positive is infected is given by Pr(V ∣ +) = (. p)/{. p+. (−p)} as a function of p = Pr(V), the prior probability of a person being infected (the prevalence of the infection in the population under study). Figure shows Pr(V ∣ +) as a function of Pr(V).
1 0.8 0.6 0.4 0.2
0.2
0.4
0.6
0.8
1
Bayesian Statistics. Fig. Posterior probability of infection Pr(V ∣ +) given a positive test, as a function of the prior probability of infection Pr(V)
B
As one would expect, the posterior probability is only zero if the prior probability is zero (so that it is known that the population is free of infection) and it is only one if the prior probability is one (so that it is known that the population is universally infected). Notice that if the infection is rare, then the posterior probability of a randomly chosen person being infected will be relatively low even if the test is positive. Indeed, for say Pr(V) = ., one finds Pr(V ∣ +) = ., so that in a population where only .% of individuals are infected, only .% of those testing positive within a random sample will actually prove to be infected: most positives would actually be false positives. In the rest of this section, we describe in some detail the learning process described by Bayes’ theorem, discuss its implementation in the presence of nuisance parameters, show how it can be used to forecast the value of future observations, and analyze its large sample behavior.
The Learning Process In the Bayesian paradigm, the process of learning from the data is systematically implemented by making use of Bayes’ theorem to combine the available prior information with the information provided by the data to produce the required posterior distribution. Computation of posterior densities is often facilitated by noting that Bayes’ theorem may be simply expressed as p(ω ∣ D) ∝ p(D ∣ ω) p(ω), (where∝ stands for ‘proportional to’ and where, for simplicity, the accepted assumptions A and the available knowledge K have been omitted from the notation), since the missing proportionality constant [∫Ω p(D ∣ ω) p(ω) dω]− may always be deduced from the fact that p(ω ∣ D), a probability density, must integrate to one. Hence, to identify the form of a posterior distribution it suffices to identify a kernel of the corresponding probability density, that is a function k(ω) such that p(ω ∣ D) = c(D) k(ω) for some c(D) which does not involve ω. In the examples which follow, this technique will often be used. An improper prior function is defined as a positive function π(ω) such that ∫Ω π(ω) dω is not finite. The formal expression of Bayes’ theorem, remains technically valid if p(ω) is replaced by an improper prior function π(ω) provided the proportionality constant exists, thus leading to a well defined proper posterior density π(ω ∣ D) ∝ p(D ∣ ω)π(ω). It will later be established (section “7Reference Analysis”) that Bayes’ theorem also remains philosophically valid if p(ω) is replaced by an appropriately chosen reference “noninformative” (typically improper) prior function π(ω).
B
B
Bayesian Statistics
Considered as a function of ω, p(D ∣ ω) is often referred to as the likelihood function. Thus, Bayes’ theorem is simply expressed in words by the statement that the posterior is proportional to the likelihood times the prior. It follows from Bayes’ theorem that, provided the same prior p(ω) is used, two different data sets D and D , with possibly different probability models p (D ∣ ω) and p (D ∣ ω) but yielding proportional likelihood functions, will produce identical posterior distributions for ω. This immediate consequence of Bayes theorem has been proposed as a principle on its own, the likelihood principle, and it is seen by many as an obvious requirement for reasonable statistical inference. In particular, for any given prior p(ω), the posterior distribution does not depend on the set of possible data values, or the outcome space. Notice, however, that the likelihood principle only applies to inferences about the parameter vector ω once the data have been obtained. Consideration of the outcome space is essential, for instance, in model criticism, in the design of experiments, in the derivation of predictive distributions, or (see section “7Reference Analysis”) in the construction of objective Bayesian procedures. Naturally, the terms prior and posterior are only relative to a particular set of data. As one would expect from the coherence induced by probability theory, if data D = {x , . . . , xn } are sequentially presented, the final result will be the same whether data are globally or sequentially processed. Indeed, p(ω ∣ x , . . . , xi+ ) ∝ p(xi+ ∣ ω) p(ω ∣ x , . . . , xi ), for i = , . . . , n − , so that the “posterior” at a given stage becomes the “prior” at the next. In most situations, the posterior distribution is “sharper” than the prior so that, in most cases, the density p(ω ∣ x , . . . , xi+ ) will be more concentrated around the true value of ω than p(ω ∣ x , . . . , xi ). However, this is not always the case: occasionally, a “surprising” observation will increase, rather than decrease, the uncertainty about the value of ω. For instance, in probabilistic diagnosis, a sharp posterior probability distribution (over the possible causes {ω , . . . , ω k } of a syndrome) describing, a “clear” diagnosis of disease ω i (that is, a posterior with a large probability for ω i ) would typically update to a less concentrated posterior probability distribution over {ω , . . . , ω k } if a new clinical analysis yielded data which were unlikely under ω i . For a given probability model, one may find that some particular function of the data t = t(D) is a sufficient statistic in the sense that, given the model, t(D) contains all information about ω which is available in D. Formally, t = t(D) is sufficient if (and only if) there exist nonnegative functions f and g such that the likelihood function may be factorized in the form p(D ∣ ω) = f (ω, t)g(D). A sufficient
statistic always exists, for t(D) = D is obviously sufficient; however, a much simpler sufficient statistic, with a fixed dimensionality which is independent of the sample size, often exists. In fact this is known to be the case whenever the probability model belongs to the generalized exponential family, which includes many of the more frequently used probability models. It is easily established that if t is sufficient, the posterior distribution of ω only depends on the data D through t(D), and may be directly computed in terms of p(t ∣ ω), so that, p(ω ∣ D) = p(ω ∣ t) ∝ p(t ∣ ω) p(ω). Naturally, for fixed data and model assumptions, different priors lead to different posteriors. Indeed, Bayes’ theorem may be described as a data-driven probability transformation machine which maps prior distributions (describing prior knowledge) into posterior distributions (representing combined prior and data knowledge). It is important to analyze whether or not sensible changes in the prior would induce noticeable changes in the posterior. Posterior distributions based on reference “noninformative” priors play a central role in this 7sensitivity analysis context. Investigation of the sensitivity of the posterior to changes in the prior is an important ingredient of the comprehensive analysis of the sensitivity of the final results to all accepted assumptions which any responsible statistical study should contain. Example (Inference on a binomial parameter). If the data D consist of n Bernoulli observations with parameter θ which contain r positive trials, then p(D ∣ θ, n) = θ r ( − θ)n−r , so that t(D) = {r, n} is sufficient. Suppose that prior knowledge about θ may be approximately described by a 7Beta distribution Be(θ ∣ α, β), so that p(θ ∣ α, β) ∝ θ α− ( − θ) β− . Using Bayes’ theorem, the posterior density of θ is p(θ ∣ r, n, α, β) ∝ θ r ( − θ)n−r θ α− ( − θ) β− ∝ θ r+α− ( − θ)n−r+β− , the Beta distribution Be(θ ∣ r + α, n − r + β). Suppose, for example, that in the light of precedent surveys, available information on the proportion θ of citizens who would vote for a particular political measure in a referendum is described by a Beta distribution Be(θ ∣ , ), so that it is judged to be equally likely that the referendum would be won or lost, and it is judged that the probability that either side wins less than % of the vote is .. A random survey of size is then conducted, where only citizens declare to be in favor of the proposed measure. Using the results above, the corresponding posterior distribution is then Be(θ ∣ , ). These prior and posterior densities are plotted in Fig. ; it may be appreciated that, as one would expect, the effect of the data is to drastically reduce the initial uncertainty on the value of θ
Bayesian Statistics
B
and the required posterior of θ is the appropriate marginal density, obtained by integration over the nuisance parameter λ,
30 25 20
p(θ ∣ D) = ∫ p(θ, λ ∣ D) dλ. Λ
15 10 5 0.35
0.4
0.45
0.5
0.55
0.6
0.65
Bayesian Statistics. Fig. Prior and posterior densities of the proportion θ of citizens that would vote in favor of a referendum text
and, hence, on the referendum outcome. More precisely, Pr(θ < . ∣ , , , H, K) = . (shaded region in Fig. ) so that, after the information from the survey has been included, the probability that the referendum will be lost should be judged to be about %. The general situation where the vector of interest is not the whole parameter vector ω, but some function θ = θ(ω) of possibly lower dimension than ω, will now be considered. Let D be some observed data, let {p(D ∣ ω), ω ∈ Ω} be a probability model assumed to describe the probability mechanism which has generated D, let p(ω) be a probability distribution describing any available information on the value of ω, and let θ = θ(ω) ∈ Θ be a function of the original parameters over whose value inferences based on the data D are required. Any valid conclusion on the value of the vector of interest θ will then be contained in its posterior probability distribution p(θ ∣ D) which is conditional on the observed data D and will naturally also depend, although not explicitly shown in the notation, on the assumed model {p(D ∣ ω), ω ∈ Ω}, and on the available prior information encapsulated by p(ω). The required posterior distribution p(θ ∣ D) is found by standard use of probability calculus. Indeed, by Bayes’ theorem, p(ω ∣ D) ∝ p(D ∣ ω) p(ω). Moreover, let λ = λ(ω) ∈ Λ be some other function of the original parameters such that ψ = {θ, λ} is a one-to-one transformation of ω, and let J(ω) = (∂ψ/∂ω) be the corresponding Jacobian matrix. Naturally, the introduction of λ is not necessary if θ(ω) is a one-to-one transformation of ω. Using standard changeof-variable probability techniques, the posterior density of ψ is
Notice that elimination of unwanted nuisance parameters, a simple integration within the Bayesian paradigm is, however, a difficult (often polemic) problem for conventional statistics. Sometimes, the range of possible values of ω is effectively restricted by contextual considerations. If ω is known to belong to Ω c ⊂ Ω, the prior distribution is only positive in Ω c and, using Bayes’ theorem, it is immediately found that the restricted posterior is p(ω ∣ D, ω ∈ Ω c ) =
p(ω ∣ D) , ∫Ω c p(ω ∣ D)
ω ∈ Ωc ,
and obviously vanishes if ω ∉ Ω c . Thus, to incorporate a restriction on the possible values of the parameters, it suffices to renormalize the unrestricted posterior distribution to the set Ω c ⊂ Ω of parameter values which satisfy the required condition. Incorporation of known constraints on the parameter values, a simple renormalization within the Bayesian pardigm, is another very difficult problem for conventional statistics. Example (Inference on normal parameters). Let D = {x , . . . xn } be a random sample from a normal distribution N(x ∣ µ, σ). The corresponding likelihood function is immediately found to be proportional to σ −n exp[−n{s + (x − µ) }/(σ )], with nx = ∑i xi , and ns = ∑i (xi − x) . It may be shown (see section “7Reference Analysis”) that absence of initial information on the value of both µ and σ may formally be described by a joint prior function which is uniform in both µ and log(σ), that is, by the (improper) prior function p(µ, σ) = σ − . Using Bayes’ theorem, the corresponding joint posterior is p(µ, σ ∣ D) ∝ σ −(n+) exp[−n{s + (x − µ) }/(σ )]. Thus, using the Gamma integral in terms of λ = σ − to integrate out σ, ∞
p(µ ∣ D) ∝ ∫
σ
−(n+)
p(ψ ∣ D) = p(θ, λ ∣ D) = [
p(ω ∣ D) ] ∣ J(ω) ∣ ω=ω(ψ)
exp [ −
n [s + (x − µ) ]] dσ ∝ [s + (x − µ) ]−n/ , σ
B
B
Bayesian Statistics
which is recognized as a kernel of the Student density √ St (µ ∣ x, s/ n − , n − ). Similarly, integrating out µ, ∞
p(σ ∣ D) ∝ ∫
σ
−(n+)
requested, the corresponding posterior distribution is found to be p(σ ∣ D) = σ − Ga(σ − ∣ ., .). This has a mean E[σ ∣ D] = . and yields Pr(. < σ < . ∣ D) = ..
−∞
exp [ −
n ns −n [s + (x − µ) ]] dµ ∝ σ exp [− ]. σ σ
Changing variables to the precision λ = σ − results in p(λ ∣ D) ∝ λ(n−)/ ens λ/ , a kernel of the Gamma density Ga(λ ∣ (n − )/, ns /). In terms of the standard deviation σ this becomes p(σ ∣ D) = p(λ ∣ D)∣∂λ/∂σ∣ = σ − Ga(σ − ∣ (n − )/, ns /), a square-root inverted gamma density. A frequent example of this scenario is provided by laboratory measurements made in conditions where central limit conditions apply, so that (assuming no experimental bias) those measurements may be treated as a random sample from a normal distribution centered at the quantity µ which is being measured, and with some (unknown) standard deviation σ. Suppose, for example, that in an elementary physics classroom experiment to measure the gravitational field g with a pendulum, a student has obtained n = measurements of g yielding (in m/s ) a mean x = ., and a standard deviation s = .. Using no other information, the corresponding posterior distribution is p(g ∣ D) = St(g ∣ ., ., ) represented in the upper panel of Fig. . In particular, Pr(. < g < . ∣ D) = ., so that, with the information provided by this experiment, the value of g at the location of the laboratory may be expected to lie between . and . with probability .. Formally, the posterior distribution of g should be restricted to g > ; however, as immediately obvious from Fig. , this would not have any appreciable effect, due to the fact that the likelihood function is actually concentrated on positive g values. Suppose now that the student is further instructed to incorporate into the analysis the fact that the value of the gravitational field g at the laboratory is known to lie between . m/s (average value at the Equator) and . m/s (average value at the poles). The updated posterior distribution will then be √ St (g ∣ m, s/ n − , n) p(g ∣ D, g ∈ Gc ) = , g ∈ Gc , √ ∫g∈Gc St (g ∣ m, s/ n − , n) represented in lower panel of Fig. , where Gc = {g; . < g < .}. Simple 7numerical integration may be used to verify that Pr(g > . ∣ D, g ∈ Gc ) = .. Moreover, if inferences about the standard deviation σ of the measurement procedure are also
Predictive Distributions Let D = {x , . . . , x n }, x i ∈ X , be a set of exchangeable observations, and consider now a situation where it is desired to predict the value of a future observation x ∈ X generated by the same random mechanism that has generated the data D. It follows from the foundations arguments discussed in section “7Foundations” that the solution to this prediction problem is simply encapsulated by the predictive distribution p(x ∣ D) describing the uncertainty on the value that x will take, given the information provided by D and any other available knowledge. Suppose that contextual information suggests the assumption that data D may be considered to be a random sample from a distribution in the family {p(x ∣ ω), ω ∈ Ω}, and let p(ω) be a prior distribution describing available information on the value of ω. Since p(x ∣ ω, D) = p(x ∣ ω), it then
40
30
20
10
9.75
9.8
9.85
9.75
9.8
9.85
9.9
40
30
20
10
9.7
9.9
Bayesian Statistics. Fig. Posterior densities p(g ∣ m, s, n) of the value g of the gravitational field
Bayesian Statistics
follows from standard probability theory that p(x ∣ D) = ∫Ω p(x ∣ ω) p(ω ∣ D) dω, which is an average of the probability distributions of x conditional on the (unknown) value of ω, weighted with the posterior distribution of ω given D. If the assumptions on the probability model are correct, the posterior predictive distribution p(x ∣ D) will converge, as the sample size increases, to the distribution p(x ∣ ω) which has generated the data. Indeed, the best technique to assess the quality of the inferences about ω encapsulated in p(ω ∣ D) is to check against the observed data the predictive distribution p(x ∣ D) generated by p(ω ∣ D). Example (Prediction in a Poisson process). Let D = {r , . . . , rn } be a random sample from a Poisson distribution Po(r ∣ λ) with parameter λ, so that p(D ∣ λ) ∝ λ t e−λn , where t = ∑ ri . It may be shown (see section “7Reference Analysis”) that absence of initial information on the value of λ may be formally described by the (improper) prior function p(λ) = λ−/ . Using Bayes’ theorem, the corresponding posterior is p(λ ∣ D) ∝ λ t e−λn λ−/ ∝ λ t−/ e−λn , the kernel of a Gamma density Ga(λ ∣ , t + /, n), with mean (t + /)/n. The corresponding predictive distribution is the Poisson-Gamma mixture ∞ Po(r ∣ λ) Ga (λ ∣ , t + , n) dλ p(r ∣ D) = ∫ Γ(r + t + /) nt+/ . = Γ(t + /) r! ( + n)r+t+/ Suppose, for example, that in a firm producing automobile restraint systems, the entire production in each of consecutive months has yielded no complaint from their clients. With no additional information on the average number λ of complaints per month, the quality assurance department of the firm may report that the probabilities that r complaints will be received in the next month of production are given by the last equation, with t = and n = . In particular, p(r = ∣ D) = ., p(r = ∣ D) = ., and p(r = ∣ D) = .. Many other situations may be described with the same model. For instance, if meteorological conditions remain similar in a given area, p(r = ∣ D) = . would describe the chances of no flash flood next year, given years without flash floods in the area. Example (Prediction in a Normal process). Consider now prediction of a continuous variable. Let D = {x , . . . , xn } be a random sample from a normal distribution N(x ∣ µ, σ). As mentioned in Example , absence of initial information on the values of both µ and σ is formally
B
described by the improper prior function p(µ, σ) = σ − , and this leads to the joint posterior density describe above. The corresponding (posterior) predictive distribution is ∞
∞
N(x ∣ µ, σ) p(µ, σ ∣ D) dµdσ √ ⎞ ⎛ n+ ,n − . = St x ∣ x, s n − ⎠ ⎝
p(x ∣ D) = ∫
∫
−∞
If µ is known to be positive, the appropriate prior function will be the restricted function p(µ, σ) = σ − if µ > and p(µ, σ) = otherwise. However, the result will still hold, provided the likelihood function p(D ∣ µ, σ) is concentrated on positive µ values. Suppose, for example, that in the firm producing automobile restraint systems, the observed breaking strengths of n = randomly chosen safety belt webbings have mean x = . kN and standard deviation s = . kN, and that the relevant engineering specification requires breaking strengths to be larger than kN. If data may truly be assumed to be a random sample from a normal distribution, the likelihood function is only appreciable for positive µ values, and only the information provided by this small sample is to be used, then the quality engineer may claim that the probability that a safety belt randomly chosen from the same batch as the sample tested would satisfy the required specification is Pr(x > ∣ D) = .. Besides, if conditions remain constant, .% of the safety belt webbings may be expected to have acceptable breaking strengths.
Asymptotic Behavior The behavior of posterior distributions when the sample size is large is now considered. This is important for, at least, two different reasons: () asymptotic results provide useful first-order approximations when actual samples are relatively large, and () objective Bayesian methods typically depend on the asymptotic properties of the assumed model. Let D = {x , . . . , xn }, x ∈ X , be a random sample of size n from {p(x ∣ ω), ω ∈ Ω}. It may be shown that, as n → ∞, the posterior distribution p(ω ∣ D) of a discrete parameter ω typically converges to a degenerate distribution which gives probability one to the true value of ω, and that the posterior distribution of a continuous parameter ω typically converges to a normal distribution centered at ˆ (MLE), with a variance its maximum likelihood estimate ω matrix which decreases with n as /n. Consider first the situation where Ω = {ω , ω , . . .} consists of a countable (possibly infinite) set of values, such that the probability model which corresponds to the true parameter value ω t is distinguishable from
B
B
Bayesian Statistics
the others in the sense that the logarithmic divergence δ{p(x ∣ ω i ) ∣ p(x ∣ ω t )} of each of the p(x ∣ ω i ) from p(x ∣ ω t ) is strictly positive. Taking logarithms in Bayes’ theorem, defining zj = log[p(xj ∣ ω i )/p(xj ∣ ω t )], j = , . . . , n, and using the strong law of large numbers on the n conditionally independent and identically distributed random quantities z , . . . , zn , it may be shown that lim p(ω t ∣ x , . . . , xn ) = ,
n→∞
lim p(ω i ∣ x , . . . , xn ) = ,
n→∞
i ≠ t.
Thus, under appropriate regularity conditions, the posterior probability of the true parameter value converges to one as the sample size grows. Consider now the situation where ω is a k-dimensional continuous parameter. Expressing Bayes’ theorem as p(ω ∣ x , . . . , xn ) ∝ exp {log[p(ω)] + ∑nj= log[p(xj ∣ ω)]}, expanding ∑j log[p(xj ∣ ω)] about its maximum (the MLE ˆ and assuming regularity conditions (to ensure that ω), terms of order higher than quadratic may be ignored and that the sum of the terms from the likelihood will dominate the term from the prior) it is found that the posterior density of ω is the approximate k-variate normal ˆ S(D, ω)}, ˆ p(ω ∣ x , . . . , xn ) ≈ Nk {ω, n
S− (D, ω) = ( − ∑ l=
∂ log[p(xl ∣ ω)] ). ∂ω i ∂ω j
A simpler, but somewhat poorer, approximation may be obtained by using the strong law of large numbers on the ˆ ≈ n F(ω), ˆ where sums above to establish that S− (D, ω) F(ω) is Fisher’s information matrix, with general element F ij (ω) = − ∫ p(x ∣ ω) X
∂ log[p(x ∣ ω)] dx, ∂ω i ∂ω j
so that ˆ n F (ω)). ˆ p(ω ∣ x , . . . , xn ) ≈ Nk (ω ∣ ω, −
−
Thus, under appropriate regularity conditions, the posterior probability density of the parameter vector ω approaches, as the sample size grows, a multivarite norˆ with a variance matrix mal density centered at the MLE ω, which decreases with n as n− . Example (Inference on a binomial parameter, continued). Let D = (x , . . . , xn ) consist of n independent Bernoulli trials with parameter θ, so that p(D ∣ θ, n) = θ r ( − θ)n−r . This likelihood function is maximized at θˆ = r/n, and Fisher’s information function is F(θ) = θ − ( − θ)− . Thus, using the results above, the posterior distribution of θ will be the approximate normal, ˆ s(θ)/ ˆ √n) , s(θ) = {θ( − θ)}/ p(θ ∣ r, n) ≈ N (θ ∣ θ,
ˆ − θ)/n. ˆ with mean θˆ = r/n and variance θ( This will provide a reasonable approximation to the exact posterior if () the prior p(θ) is relatively “flat” in the region where the likelihood function matters, and () both r and n are moderately large. If, say, n = , and r = , this leads to p(θ ∣ D) ≈ N(θ ∣ ., .), and to Pr(θ > . ∣ D) ≈ ., which may be compared with the exact value Pr(θ > . ∣ D) = . obtained from the posterior distribution which corresponds to the prior Be(θ ∣ , ). It follows from the joint posterior asymptotic behavior of ω and from the properties of the multivariate normal distribution (see 7Multivariate Normal Distributions) that, if the parameter vector is decomposed into ω = (θ, λ), and Fisher’s information matrix is correspondingly partitioned, so that ⎛F (θ, λ) θθ F(ω) = F(θ, λ) = ⎜ ⎜ ⎝F λθ (θ, λ)
F θ λ (θ, λ)⎞ ⎟ ⎟ F λ λ (θ, λ)⎠
and ⎛S (θ, λ) θθ S(θ, λ) = F − (θ, λ) = ⎜ ⎜ ⎝S λθ (θ, λ)
Sθ λ (θ, λ)⎞ ⎟, ⎟ S λ λ (θ, λ)⎠
then the marginal posterior distribution of θ will be ˆ n− Sθ θ (θ, ˆ λ)}, ˆ p(θ ∣ D) ≈ N{θ ∣ θ, while the conditional posterior distribution of λ given θ will be ˆ p(λ ∣ θ, D) ≈ N {λ ∣ λˆ − F − λ λ (θ, λ) ˆ θˆ − θ), n− F − ˆ F λθ (θ, λ)( λ λ (θ, λ)} . Notice that F − λ λ = S λ λ if (and only if) F is block diagonal, i.e., if (and only if) θ and λ are asymptotically independent. Example (Inference on normal parameters, continued). Let D = (x , . . . , xn ) be a random sample from a normal distribution N(x ∣ µ, σ). The corresponding likelihood ˆ σˆ ) = (x, s), and function p(D ∣ µ, σ) is maximized at ( µ, Fisher’s information matrix is diagonal, with F µ µ = σ − . Hence, the posterior distribution of µ is approximately √ N (µ ∣ x, s/ n); this may be compared with the exact result √ p(µ ∣ D) = St (µ ∣ x, s/ n − , n − ) obtained previously under the assumption of no prior knowledge.
Inference Summaries From a Bayesian viewpoint, the final outcome of a problem of inference about any unknown quantity is nothing but the corresponding posterior distribution. Thus, given some data D and conditions C, all that can be said about any function ω of the parameters which govern the model is contained in the posterior distribution p(ω ∣ D, C), and
Bayesian Statistics
all that can be said about some function y of future observations from the same model is contained in its posterior predictive distribution p(y ∣ D, C). As mentioned before, Bayesian inference may technically be described as a decision problem where the space of available actions is the class of those posterior probability distributions of the quantity of interest which are compatible with accepted assumptions. However, to make it easier for the user to assimilate the appropriate conclusions, it is often convenient to summarize the information contained in the posterior distribution by () providing values of the quantity of interest which, in the light of the data, are likely to be “close” to its true value and by () measuring the compatibility of the results with hypothetical values of the quantity of interest which might have been suggested in the context of the investigation. In this section, those Bayesian counterparts of traditional estimation and hypothesis testing problems are briefly considered.
Estimation In one or two dimensions, a graph of the posterior probability density of the quantity of interest (or the probability mass function in the discrete case) immediately conveys an intuitive, “impressionist” summary of the main conclusions which may possibly be drawn on its value. Indeed, this is greatly appreciated by users, and may be quoted as an important asset of Bayesian methods. From a plot of its posterior density, the region where (given the data) a univariate quantity of interest is likely to lie is easily distinguished. For instance, all important conclusions about the value of the gravitational field in Example are qualitatively available from Fig. . However, this does not easily extend to more than two dimensions and, besides, quantitative conclusions (in a simpler form than that provided by the mathematical expression of the posterior distribution) are often required. Point Estimation. Let D be the available data, which are assumed to have been generated by a probability model {p(D ∣ ω), ω ∈ Ω}, and let θ = θ(ω) ∈ Θ be the quantity of interest. A point estimator of θ is some function of the data ˜ θ˜ = θ(D) which could be regarded as an appropriate proxy for the actual, unknown value of θ. Formally, to choose a point estimate for θ is a decision problem, where the action space is the class Θ of possible θ values. From a decisiontheoretic perspective, to choose a point estimate θ˜ of some quantity θ is a decision to act as if the true value of θ were ˜ not to assert something about the value of θ (although θ, desire to assert something simple may well be the reason to obtain an estimate). As prescribed by the foundations of decision theory (section “7Foundations”), to solve this decision problem it is necessary to specify a loss function
B
˜ θ) measuring the consequences of acting as if the true ℓ(θ, ˜ when it is actually θ. value of the quantity of interest were θ, ˜ The expected posterior loss if θ were used is ˜ θ) p(θ ∣ D) dθ, ℓ[θ˜ ∣ D] = ∫ ℓ(θ, Θ
and the corresponding Bayes estimator θ ∗ is that function of the data, θ ∗ = θ ∗ (D), which minimizes this expectation. Example (Conventional Bayes estimators). For any given model and data, the Bayes estimator obviously depends on the chosen 7loss function. The loss function is context specific, and should be chosen in terms of the anticipated uses of the estimate; however, a number of conventional loss functions have been suggested for those situations where no particular uses are envisaged. These loss functions produce estimates which may be regarded as simple descriptions of the location of the posterior distribution. For example, if the loss function is quadratic, so ˜ θ) = (θ˜ − θ)t (θ˜ − θ), then the Bayes estimator is that ℓ(θ, the posterior mean θ ∗ = E[θ ∣ D], assuming that the mean exists. Similarly, if the loss function is a zero-one function, ˜ θ) = if θ˜ belongs to a ball or radius є centered so that ℓ(θ, ˜ θ) = otherwise, then the Bayes estimator θ ∗ in θ and ℓ(θ, tends to the posterior mode as the ball radius є tends to zero, assuming that a unique mode exists. If θ is univariate and ˜ θ) = c (θ˜ − θ) if the loss function is linear, so that ℓ(θ, ˜ ˜ ˜ θ ≥ θ, and ℓ(θ, θ) = c (θ − θ) otherwise, then the Bayes estimator is the posterior quantile of order c /(c + c ), so that Pr[θ < θ ∗ ] = c /(c + c ). In particular, if c = c , the Bayes estimator is the posterior median. The results derived for linear loss functions clearly illustrate the fact that any possible parameter value may turn out be the Bayes estimator: it all depends on the loss function describing the consequences of the anticipated uses of the estimate. Example (Intrinsic estimation). Conventional loss functions are typically non-invariant under reparametrization, so that the Bayes estimator ϕ∗ of a one-to-one transformation ϕ = ϕ(θ) of the original parameter θ is not necessarily ϕ(θ ∗ ) (the univariate posterior median, which is invariant, is an interesting exception). Moreover, conventional loss functions focus on the “distance” between the estimate θ˜ and the true value θ, rather than on the “distance” between the probability models they label. Intrinsic losses directly focus on how different the probability model p(D ∣ θ, λ) is from its closest approximation within ˜ λ i ), λ i ∈ Λ}, and typically produce the family {p(D ∣ θ, invariant solutions. An attractive example is the intrinsic ˜ θ) defined as the minimum logarithdiscrepancy, d(θ, mic divergence between a probability model labeled by θ
B
B
Bayesian Statistics
˜ When there are no and a probability model labeled by θ. nuisance parameters, this is given by
function), which is not equal to ϕ(E[θ ∣ D]). The intrinsic loss function in this problem is
˜ θ) = min{δ(θ˜ ∣ θ), δ(θ ∣ θ)}, ˜ d(θ, p(t ∣ θ) dt, δ(θ i ∣ θ) = ∫ p(t ∣ θ) log p(t ∣ θ i ) T
˜ θ) = n min{δ(θ˜ ∣ θ), δ(θ ∣ θ)}, ˜ d(θ, −θ θ , δ(θ i ∣ θ) = θ log + ( − θ) log θi − θi
where t = t(D) ∈ T is any sufficient statistic (which may well be the whole data set D). The definition is easily extended to problems with nuisance parameters; in this case, ˜ θ, λ) = min d(θ, ˜ λ i , θ, λ) d(θ, λ i ∈Λ
measures the logarithmic divergence from p(t ∣ θ, λ) of its ˜ and the loss function closest approximation with θ = θ, now depends on the complete parameter vector (θ, λ). Although not explicitly shown in the notation, the intrinsic discrepancy function typically depends on the sample size n; indeed, when the data consist of a random sample D = {x , . . . , xn } from some model p(x ∣ θ) then δ(θ i ∣ θ) = n ∫ p(x ∣ θ) log χ
p(x ∣ θ) dx, p(x ∣ θ i )
so that the discrepancy associated with the full model is simply n times the discrepancy which corresponds to a single observation. The intrinsic discrepancy is a symmetric, non-negative loss function with a direct interpretation in information-theoretic terms as the minimum amount of information which is expected to be necessary to distinguish between the model p(D ∣ θ, λ) and its closest approx˜ λ i ), λ i ∈ Λ}. Moreover, imation within the class {p(D ∣ θ, it is invariant under one-to-one reparametrization of the parameter of interest θ, and does not depend on the choice of the nuisance parameter λ. The intrinsic estimator is naturally obtained by minimizing the posterior expected intrinsic discrepancy ˜ θ, λ)p(θ, λ ∣ D) dθdλ. d(θ˜ ∣ D) = ∫ ∫ d(θ, Λ
Θ
Since the intrinsic discrepancy is invariant under reparametrization, minimizing its posterior expectation produces invariant estimators. Example (Inference on a binomial parameter, continued). In the estimation of a binomial proportion θ, given data D = (n, r) and a Beta prior Be(θ ∣ α, β), the Bayes estimator associated with the quadratic loss (the corresponding posterior mean) is E[θ ∣ D] = (r + α)/(n+ α + β), while the quadratic loss based estimator of, say, the logodds ϕ(θ) = log[θ/( − θ)], is E[ϕ ∣ D] = ψ(r + α) − ψ(n−r + β) (where ψ(x) = d log[Γ(x)]/dx is the digamma
and the corresponding intrinsic estimator θ ∗ is obtained by minimizing the expected posterior loss d(θ˜ ∣ D) = ˜ θ) p(θ ∣ D) dθ. The exact value of θ ∗ may be ∫ d(θ, obtained by numerical minimization, but a very good approximation is given by θ∗ ≈
r+α eψ(r+α) . + n + α + β eψ(r+α) + eψ(n−r+β)
Since intrinsic estimation is an invariant procedure, the intrinsic estimator of the log-odds will simply be the logodds of the intrinsic estimator of θ. As one would expect, when r + α and n − r + β are both large, all Bayes estimators of any well-behaved function ϕ(θ) will cluster around ϕ(E[θ ∣ D]). Interval Estimation. To describe the inferential content of the posterior distribution of the quantity of interest p(θ ∣ D) it is often convenient to quote regions R ⊂ Θ of given probability under p(θ ∣ D). For example, the identification of regions containing %, %, %, or % of the probability under the posterior may be sufficient to convey the general quantitative messages implicit in p(θ ∣ D); indeed, this is the intuitive basis of graphical representations of univariate distributions like those provided by boxplots. Any region R ⊂ Θ such that ∫R p(θ ∣ D)dθ = q (so that, given data D, the true value of θ belongs to R with probability q), is said to be a posterior q-credible region of θ. Notice that this provides immediately a direct intuitive statement about the unknown quantity of interest θ in probability terms, in marked contrast to the circumlocutory statements provided by frequentist confidence intervals. Clearly, for any given q there are generally infinitely many credible regions. A credible region is invariant under reparametrization; thus, for any q-credible region R of θ, ϕ(R) is a q-credible region of ϕ = ϕ(θ). Sometimes, credible regions are selected to have minimum size (length, area, volume), resulting in highest probability density (HPD) regions, where all points in the region have larger probability density than all points outside. However, HPD regions are not invariant under reparametrization: the image ϕ(R) of an HPD region R will be a credible region for ϕ, but will not generally be HPD; indeed, there is no compelling reason to restrict attention to HPD credible regions. Posterior quantiles are often used to derive credible regions. Thus, if θ q = θ q (D) is the q% posterior quantile of θ, then R = {θ; θ ≤ θ q } is a one-sided, typically unique
Bayesian Statistics
q-credible region, and it is invariant under reparametrization. Indeed, probability centered q-credible regions of the form R = {θ; θ (−q)/ ≤ θ ≤ θ (+q)/ } are easier to compute, and are often quoted in preference to HPD regions.
B
∫Θ ℓ(a , θ) p(θ ∣ D) dθ,islargerthantheexpectedposterior loss of rejecting, ∫Θ ℓ(a , θ) p(θ ∣ D) dθ, that is, if (and only if) ∫ [ℓ(a , θ) − ℓ(a , θ)] p(θ ∣ D) dθ Θ
Example (Inference on normal parameters, continued). In the numerical example about the value of the gravitational field described in the top panel of Fig. , the interval [., .] in the unrestricted posterior density of g is a HPD, %-credible region for g. Similarly, the interval [., .] in the bottom panel of Fig. is also a %-credible region for g, but it is not HPD. The concept of a credible region for a function θ = θ(ω) of the parameter vector is trivially extended to prediction problems. Thus, a posterior q-credible region for x ∈ χ is a subset R of the outcome space X with posterior predictive probability q, so that ∫R p(x ∣ D)dx = q. For a description of the choice of credible regions using the intrinsic loss function, see Bernardo (b).
Hypothesis Testing The posterior distribution p(θ ∣ D) of the quantity of interest θ conveys immediate intuitive information on those values of θ which, given the assumed model, may be taken to be compatible with the observed data D, namely, those with a relatively high probability density. Sometimes, a restriction θ ∈ Θ ⊂ Θ of the possible values of the quantity of interest (where Θ may possibly consists of a single value θ ) is suggested in the course of the investigation as deserving special consideration, either because restricting θ to Θ would greatly simplify the model, or because there are additional, context specific arguments suggesting that θ ∈ Θ . Intuitively, the hypothesis H ≡ {θ ∈ Θ } should be judged to be compatible with the observed data D if there are elements in Θ with a relatively high posterior density. However, a more precise conclusion is often required and, once again, this is made possible by adopting a decision-oriented approach. Formally, testing the hypothesis H ≡ {θ ∈ Θ } is a decision problem where the action space has only two elements, namely to accept (a ) or to reject (a ) the proposed restriction. To solve this decision problem, it is necessary to specify an appropriate loss function, ℓ(ai , θ), measuring the consequences of accepting or rejecting H as a function of the actual value θ of the vector of interest. Notice that this requires the statement of an alternative a to accepting H ; this is only to be expected, for an action is taken not because it is good, but because it is better than anything else that has been imagined. Given data D, the optimal action will be to reject H if (and only if) the expected posterior loss of accepting,
= ∫ ∆ℓ(θ) p(θ ∣ D) dθ > . Θ
Therefore, only the loss difference ∆ℓ(θ) = ℓ(a , θ) − ℓ(a , θ), which measures the advantage of rejecting H as a function of θ, has to be specified. Thus, as common sense dictates, the hypothesis H should be rejected whenever the expected advantage of rejecting H is positive. A crucial element in the specification of the loss function is a description of what is actually meant by rejecting H . By assumption a means to act as if H were true, i.e.,as if θ ∈ Θ , but there are at least two obvious options for the alternative action a . This may either mean () the negation of H , that is to act as if θ ∉ Θ or, alternatively, it may rather mean () to reject the simplification implied by H and to keep the unrestricted model, θ ∈ Θ, which is true by assumption. Both options have been analyzed in the literature, although it may be argued that the problems of scientific data analysis where hypothesis testing procedures are typically used are better described by the second alternative. Indeed, an established model, identified by H ≡ {θ ∈ Θ }, is often embedded into a more general model, {θ ∈ Θ, Θ ⊂ Θ}, constructed to include possibly promising departures from H , and it is required to verify whether presently available data D are still compatible with θ ∈ Θ , or whether the extension to θ ∈ Θ is really required. Example (Conventional hypothesis testing). Let p(θ ∣ D), θ ∈ Θ, be the posterior distribution of the quantity of interest, let a be the decision to work under the restriction θ ∈ Θ and let a be the decision to work under the complementary restriction θ ∉ Θ . Suppose, moreover, that the loss structure has the simple, zero-one form given by {ℓ(a , θ) = , ℓ(a , θ) = } if θ ∈ Θ and, similarly, {ℓ(a , θ) = , ℓ(a , θ) = } if θ ∉ Θ , so that the advantage ∆ℓ(θ) of rejecting H is if θ ∉ Θ and it is − otherwise. With this loss function it is immediately found that the optimal action is to reject H if (and only if) Pr(θ ∉ Θ ∣ D) is larger than Pr(θ ∈ Θ ∣ D). Notice that this formulation requires that Pr(θ ∈ Θ ) > , that is, that the hypothesis H has a strictly positive prior probability. If θ is a continuous parameter and Θ has zero measure (for instance if H consists of a single point θ ), this requires the use of a nonregular “sharp” prior concentrating a positive probability mass on Θ .
B
B
Bayesian Statistics
Example (Intrinsic hypothesis testing). Again, let p(θ ∣ D), θ ∈ Θ, be the posterior distribution of the quantity of interest, and let a be the decision to work under the restriction θ ∈ Θ , but let a now be the decision to keep the general, unrestricted model θ ∈ Θ. In this case, the advantage ∆ℓ(θ) of rejecting H as a function of θ may safely be assumed to have the form ∆ℓ(θ) = d(Θ , θ) − d∗ , for some d∗ > , where () d(Θ , θ) is some measure of the discrepancy between the assumed model p(D ∣ θ) and its closest approximation within the class {p(D ∣ θ ), θ ∈ Θ }, such that d(Θ , θ) = whenever θ ∈ Θ , and () d∗ is a context dependent utility constant which measures the (necessarily positive) advantage of being able to work with the simpler model when it is true. Choices of both d(Θ , θ) and d∗ which may be appropriate for general use will now be described. For reasons similar to those supporting its use in point estimation, an attractive choice for the function d(Θ , θ) is an appropriate extension of the intrinsic discrepancy; when there are no nuisance parameters, this is given by d(Θ , θ) = inf min{δ(θ ∣ θ), δ(θ ∣ θ )} θ ∈Θ
where δ(θ ∣ θ) = ∫T p(t ∣ θ) log{p(t ∣ θ)/p(t ∣ θ )}dt, and t = t(D) ∈ T is any sufficient statistic, which may well be the whole dataset D. As before, if the data D = {x , . . . , xn } consist of a random sample from p(x ∣ θ), then δ(θ ∣ θ) = n ∫ p(x ∣ θ) log χ
p(x ∣ θ) dx. p(x ∣ θ )
Naturally, the loss function d(Θ , θ) reduces to the intrinsic discrepancy d(θ , θ) of Example when Θ contains a single element θ . Besides, as in the case of estimation, the definition is easily extended to problems with nuisance parameters, with d(Θ , θ, λ) =
inf
θ ∈Θ ,λ ∈Λ
d(θ , λ o , θ, λ).
The hypothesis H should be rejected if the posterior expected advantage of rejecting is d(Θ ∣ D) = ∫ ∫ d(Θ , θ, λ) p(θ, λ ∣ D) dθdλ > d∗ , Λ
∗
Θ
for some d > . It is easily verified that the function d(Θ , D) is nonnegative. Morovever, if ϕ = ϕ(θ) is a one-to-one transformation of θ, then d(ϕ(Θ ), D) = d(Θ , D), so that the expected intrinsic loss of rejecting H is invariant under reparametrization. It may be shown that, as the sample size increases, the expected value of d(Θ , D) under sampling tends to
one when H is true, and tends to infinity otherwise; thus d(Θ , D) may be regarded as a continuous, positive measure of how inappropriate (in loss of information units) it would be to simplify the model by accepting H . In traditional language, d(Θ , D) is a test statistic for H and the hypothesis should be rejected if the value of d(Θ , D) exceeds some critical value d∗ . In sharp contrast to conventional hypothesis testing, this critical value d∗ is found to be a context specific, positive utility constant d∗ , which may precisely be described as the number of information units which the decision maker is prepared to lose in order to be able to work with the simpler model H , and does not depend on the sampling properties of the probability model. The procedure may be used with standard, continuous regular priors even in sharp hypothesis testing, when Θ is a zero-measure set (as would be the case if θ is continuous and Θ contains a single point θ ). Naturally, to implement the test, the utility constant d∗ which defines the rejection region must be chosen. All measurements are based on a comparison with a standard; comparison with the “canonical” problem of testing a value µ = µ for the mean of a normal distribution with known variance (see below) makes it possible to calibrate this information scale. Values of d(Θ , D) of about one should be regarded as an indication of no evidence against H , since the expected value of d(Θ , D) under H is exactly equal to one. Values of d(Θ , D) of about ., and should be respectively regarded as an indication of mild evidence against H , and significant evidence against H since, in the canonical normal problem, these values correspond to the observed sample mean x respectively lying two or three posterior standard deviations from the null value µ . Notice that, in sharp contrast to frequentist hypothesis testing, where it is hazily recommended to adjust the significance level for dimensionality and sample size, this provides an absolute scale (in information units) which remains valid for any sample size and any dimensionality. Example (Testing the value of a normal mean). Let the data D = {x , . . . , xn } be a random sample from a normal distribution N(x ∣ µ, σ), where σ is assumed to be known, and consider the “canonical” problem of testing whether these data are or are not compatible with some specific sharp hypothesis H ≡ {µ = µ } on the value of the mean. The conventional approach to this problem requires a non-regular prior which places a probability mass, say p , on the value µ to be tested, with the remaining − p probability continuously distributed over R. If this prior is chosen to be p(µ ∣ µ ≠ µ ) = N(µ ∣ µ , σ ), Bayes theorem
Bayesian Statistics
may be used to obtain the corresponding posterior probability, Pr[µ ∣ D, λ] =
B (D, λ) p , ( − p ) + p B (D, λ)
n / n ) exp [− z ] , λ n+λ √ where z = (x − µ )/ (σ/ n) measures, in standard deviations, the distance between x and µ and λ = σ /σ is the ratio of model to prior variance. The function B (D, λ), a ratio of (integrated) likelihood functions, is called the Bayes factor in favor of H . With a conventional zero-one loss function, H should be rejected if Pr[µ ∣ D, λ] < /. The choices p = / and λ = or λ = /, describing particular forms of sharp prior knowledge, have been suggested in the literature for routine use. The conventional approach to sharp hypothesis testing deals with situations of concentrated prior probability; it assumes important prior knowledge about the value of µ and, hence, should not be used unless this is an appropriate assumption. Moreover, as pointed out by Bartlett (), the resulting posterior probability is extremely sensitive to the specific prior specification. In most applications, H is really a hazily defined small region rather than a point. For moderate sample sizes, the posterior probability Pr[µ ∣ D, λ] is an approximation to the posterior probability Pr[µ − є < µ < µ − є ∣ D, λ] for some small interval around µ which would have been obtained from a regular, continuous prior heavily concentrated around µ ; however, this approximation always breaks down for sufficiently large sample sizes. One consequence (which is immediately apparent from the last two equations) is that for any fixed value of the pertinent statistic z, the posterior probability of the null, Pr[µ ∣ D, λ], tends to one as n → ∞. Far from being specific to this example, this unappealing behavior of posterior probabilities based on sharp, non-regular priors (discovered by Lindley , and generally known as Lindley’s paradox) is always present in the conventional Bayesian approach to sharp hypothesis testing. The intrinsic approach may be used without assuming any sharp prior knowledge. The intrinsic discrepancy is d(µ , µ) = n(µ − µ ) /(σ ), a simple transformation of the standardized distance between µ and µ . As later explained (section “7Reference Analysis”), absence of initial information about the value of µ may formally be described in this problem by the (improper) uniform prior function p(µ) = ; Bayes’ theorem may then be used to obtain the corresponding (proper) posterior distribution, √ p(µ ∣ D) = N (µ ∣ x, σ/ n). The expected value of d(µ , µ) with respect to this posterior is d(µ , D) = ( + z )/, B (D, λ) = ( +
B
√ where z = (x − µ )/ (σ/ n) is the standardized distance between x and µ . As foretold by the general theory, the expected value of d(µ , D) under repeated sampling is one if µ = µ , and increases linearly with n if µ = µ . Moreover, in this canonical example, to reject H whenever ∣z∣ > or ∣z∣ > , that is whenever µ is two or three posterior standard deviations away from x, respectively corresponds to rejecting H whenever d(µ , D) is larger than ., or larger than . But the information scale is independent of the problem, so that rejecting the null whenever its expected discrepancy from the true model is larger than d∗ = units of information is a general rule (and one which corresponds to the conventional “σ” rule in the canonical normal case). If σ is unknown, the intrinsic discrepancy becomes
d(µ , µ, σ) =
µ − µ n log [ + ( ) ]. σ
Moreover, as mentioned before, absence of initial information about both µ and σ may be described by the (improper) prior function p(µ, σ) = σ − . The intrinsic test statistic d(µ , D) is found as the expected value of d(µ , µ, σ) under the corresponding joint posterior distribution; this may be exactly expressed in terms of hypergeometric functions, and is approximated by t n + log ( + ), n √ where t is the traditional statistic t = n − (x − µ )/s, ns = ∑j (xj − x) . For instance, for samples sizes , and , , and using the utility constant d∗ = , the hypothesis H would be rejected whenever ∣t∣ is respectively larger than ., ., and .. d(µ , D) ≈
Reference Analysis Under the Bayesian paradigm, the outcome of any inference problem (the posterior distribution of the quantity of interest) combines the information provided by the data with relevant available prior information. In many situations, however, either the available prior information on the quantity of interest is too vague to warrant the effort required to have it formalized in the form of a probability distribution, or it is too subjective to be useful in scientific communication or public decision making. It is therefore important to be able to identify the mathematical form of a “noninformative” prior, a prior that would have a minimal effect, relative to the data, on the posterior inference. More formally, suppose that the probability mechanism which has generated the available data D is assumed to be p(D ∣ ω), for some ω ∈ Ω,
B
B
Bayesian Statistics
and that the quantity of interest is some real-valued function θ = θ(ω) of the model parameter ω. Without loss of generality, it may be assumed that the probability model is of the form p(D ∣ θ, λ), θ ∈ Θ, λ ∈ Λ, where λ is some appropriately chosen nuisance parameter vector. As described in section “7The Bayesian Paradigm”, to obtain the required posterior distribution of the quantity of interest p(θ ∣ D) it is necessary to specify a joint prior p(θ, λ). It is now required to identify the form of that joint prior π θ (θ, λ), the θ-reference prior, which would have a minimal effect on the corresponding posterior distribution of θ, π(θ ∣ D) ∝ ∫ p(D ∣ θ, λ) π θ (θ, λ) dλ, Λ
a prior which, to use a conventional expression, “would let the data speak for themselves” about the likely value of θ. Properly defined, reference posterior distributions have an important role to play in scientific communication, for they provide the answer to a central question in the sciences: conditional on the assumed model p(D ∣ θ, λ), and on any further assumptions of the value of θ on which there might be universal agreement, the reference posterior π(θ ∣ D) should specify what could be said about θ if the only available information about θ were some well-documented data D. Much work has been done to formulate “reference” priors which would make the idea described above mathematically precise. This section concentrates on an approach that is based on information theory to derive reference distributions which may be argued to provide the most advanced general procedure available. In the formulation described below, far from ignoring prior knowledge, the reference posterior exploits certain well-defined features of a possible prior, namely those describing a situation where relevant knowledge about the quantity of interest (beyond that universally accepted) may be held to be negligible compared to the information about that quantity which repeated experimentation (from a particular data generating mechanism) might possibly provide. Reference analysis is appropriate in contexts where the set of inferences which could be drawn in this possible situation is considered to be pertinent. Any statistical analysis contains a fair number of subjective elements; these include (among others) the data selected, the model assumptions, and the choice of the quantities of interest. Reference analysis may be argued to provide an “objective” Bayesian solution to statistical inference problems in just the same sense that conventional statistical methods claim to be “objective”: in that the solutions only depend on model assumptions and observed
data. The whole topic of objective Bayesian methods is, however, subject to polemic; interested readers will find in Bernardo (a) and references therein some pointers to the relevant literature.
Reference Distributions One parameter. Consider the experiment which consists of the observation of data D, generated by a random mechanism p(D ∣ θ) which only depends on a real-valued parameter θ ∈ Θ, and let t = t(D) ∈ T be any sufficient statistic (which may well be the complete data set D). In Shannon’s general information theory, the amount of information I θ {T , p(θ)} which may be expected to be provided by D, or (equivalently) by t(D), about the value of θ is defined by p(t, θ) dθdt I θ {T , p(θ)} = ∫ ∫ p(t, θ) log p(t)p(θ) T
Θ
= Et [ ∫ p(θ ∣ t) log Θ
p(θ ∣ t) dθ] p(θ)
the expected logarithmic divergence of the prior from the posterior. This is naturally a functional of the prior p(θ): the larger the prior information, the smaller the information which the data may be expected to provide. The functional I θ {T , p(θ)} is concave, non-negative, and invariant under one-to-one transformations of θ. Consider now the amount of information I θ {T k , p(θ)} about θ which may be expected from the experiment which consists of k conditionally independent replications {t , . . . , t k } of the original experiment. As k → ∞, such an experiment would provide any missing information about θ which could possibly be obtained within this framework; thus, as k → ∞, the functional I θ {T k , p(θ)} will approach the missing information about θ associated with the prior p(θ). Intuitively, a θ-“noninformative” prior is one which maximizes the missing information about θ. Formally, if π k (θ) denotes the prior density which maximizes I θ {T k , p(θ)} in the class P of strictly positive prior distributions which are compatible with accepted assumptions on the value of θ (which may well be the class of all strictly positive proper priors) then the θ-reference prior π(θ) is the limit as k → ∞ (in a sense to be made precise) of the sequence of priors {π k (θ), k = , , . . .}. Notice that this limiting procedure is not some kind of asymptotic approximation, but an essential element of the definition of a reference prior. In particular, this definition implies that reference distributions only depend on the asymptotic behavior of the assumed probability model, a feature which greatly simplifies their actual derivation.
Bayesian Statistics
Example (Maximum entropy). If θ may only take a finite number of values, so that the parameter space is Θ = {θ , . . . , θ m } and p(θ) = {p , . . . , pm }, with pi = Pr(θ = θ i ), then the missing information associated to {p , . . . , pm } may be shown to be m
lim I θ {T k , p(θ)} = H(p , . . . , pm ) = − ∑i= pi log(pi ),
k→∞
that is, the entropy of the prior distribution {p , . . . , pm }. Thus, in the finite case, if there is no further structure in the problem (which should then be taken into account), the reference prior is that with maximum entropy in the class P of priors compatible with accepted assumptions. Consequently, the reference prior algorithm contains “maximum entropy” priors as the particular case which obtains when the parameter space is finite, the only case where the original concept of 7entropy (in statistical mechanics, as a measure of uncertainty) is unambiguous and well-behaved. If, in particular, P contains all priors over {θ , . . . , θ m }, then the reference prior is the uniform prior, π(θ) = {/m, . . . , /m}. Formally, the reference prior function π(θ) of a univariate parameter θ is defined to be the limit of the sequence of the proper priors π k (θ) which maximize I θ {T k , p(θ)} in the precise sense that, for any value of the sufficient statistic t = t(D), the reference posterior, the pointwise limit π(θ ∣ t) of the corresponding sequence of posteriors {π k (θ ∣ t)}, may be obtained from π(θ) by formal use of Bayes theorem, so that π(θ ∣ t) ∝ p(t ∣ θ) π(θ). Reference prior functions are often simply called reference priors, even though they are usually not probability distributions. They should not be considered as expressions of belief, but technical devices to obtain (proper) posterior distributions which are a limiting form of the posteriors which could have been obtained from possible prior beliefs which were relatively uninformative with respect to the quantity of interest when compared with the information which data could provide. If () the sufficient statistic t = t(D) is a consistent estimator θ˜ of a continuous parameter θ, and () the class P contains all strictly positive priors, then the reference prior may be shown to have a simple form in terms of any asymptotic approximation to the posterior distribution of θ. Notice that, by construction, an asymptotic approximation to the posterior does not depend on the prior. Specifically, if the posterior density p(θ ∣ D) has an asymp˜ n), the reference totic approximation of the form p(θ ∣ θ, prior is simply ˜ n)∣ π(θ) ∝ p(θ ∣ θ, ˜ θ=θ
B
One-parameter reference priors are shown to be invariant under reparametrization; thus, if ψ = ψ(θ) is a piecewise one-to-one function of θ, then the ψ-reference prior is simply the appropriate probability transformation of the θ-reference prior. Example (Jeffreys’ prior). If θ is univariate and continuous, and the posterior distribution of θ given {x . . . , xn } is asymptotically normal with standard devi˜ √n, then, using the last displayed equation, the ation s(θ)/ reference prior function is π(θ) ∝ s(θ)− . Under regularity conditions (often satisfied in practice, see section “7Asymptopic Behavior”), the posterior distribution of θ ˆ where is asymptotically normal with variance n− F − (θ), F(θ) is Fisher’s information function and θˆ is the MLE of θ. Hence, the reference prior function in these conditions is π(θ) ∝ F(θ)/ , which is known as Jeffreys’ prior. It follows that the reference prior algorithm contains Jeffreys’ priors as the particular case which obtains when the probability model only depends on a single continuous univariate parameter, there are regularity conditions to guarantee 7asymptotic normality, and there is no additional information, so that the class of possible priors P contains all strictly positive priors over Θ. These are precisely the conditions under which there is general agreement on the use of Jeffreys’ prior as a “noninformative” prior. Example (Inference on a binomial parameter, continued). Let data consist of a sequence D = {x , . . . , xn } of n conditionally independent Bernoulli trials, so that p(x ∣ θ) = θ x ( − θ)−x , x ∈ {, }; this is a regular, one-parameter continuous model, whose Fisher’s information function is F(θ) = θ − ( − θ)− . Thus, the reference prior π(θ) is proportional to θ −/ ( − θ)−/ , so that the reference prior is the (proper) Beta distribution Be(θ ∣ /, /). Since the reference algorithm is invariant under reparametrization, the reference prior of ϕ(θ) = √ arc sin θ is π(ϕ) = π(θ)/∣∂ϕ/∂/θ∣ = ; thus, the reference prior is uniform on √ the variance-stabilizing transformation ϕ(θ) = arc sin θ, a feature generally true under regularity conditions. In terms of the original parameter θ, the corresponding reference posterior is Be(θ ∣ r + /, n − r + /), where r = ∑ xj is the number of positive trials. Suppose, for example, that n = randomly selected people have been tested for an infection and that all tested negative, so that r = . The reference posterior distribution of the proportion θ of people infected is then the Beta distribution Be(θ ∣ ., .), represented in Fig. . It may well be known that the infection was rare, leading to the assumption that θ < θ , for some upper bound
B
B
Bayesian Statistics
600
. If π(λ ∣ θ) is proper, this may be used to integrate out the nuisance parameter thus obtaining the oneparameter integrated model p(t ∣ θ) = ∫Λ p(t ∣ θ, λ) π(λ ∣ θ) dλ, to which the one-parameter algorithm may be applied again to obtain π(θ). The θ-reference prior is then π θ (θ, λ) = π(λ ∣ θ) π(θ), and the required reference posterior is π(θ ∣ t) ∝ p(t ∣ θ)π(θ).
500 400 300 200 100 0.01
0.02
0.03
0.04
0.05
Bayesian Statistics. Fig. Posterior distribution of the proportion of infected people in the population, given the results of n = tests, none of which were positive
θ ; the (restricted) reference prior would then be of the form π(θ) ∝ θ −/ (−θ)−/ if θ < θ , and zero otherwise. However, provided the likelihood is concentrated in the region θ < θ , the corresponding posterior would virtually be identical to Be(θ ∣ ., .). Thus, just on the basis of the observed experimental results, one may claim that the proportion of infected people is surely smaller than % (for the reference posterior probability of the event θ > . is .), that θ is smaller than . with probability . (area of the shaded region in Fig. ), that it is equally likely to be over or below .% (for the median, represented by a vertical line, is .), and that the probability that a person randomly chosen from the population is infected is . (the posterior mean, represented in the figure by a black circle), since Pr(x = ∣ r, n) = E[θ ∣ r, n] = .. If a particular point estimate of θ is required (say a number to be quoted in the summary headline) the intrinsic estimator suggests itself; this is found to be θ ∗ = . (represented in the figure with a white circle). Notice that the traditional solution to this problem, based on the asymptotic behavior of the MLE, here θˆ = r/n = for any n, makes absolutely no sense in this scenario. One nuisance parameter. The extension of the reference prior algorithm to the case of two parameters follows the usual mathematical procedure of reducing the problem to a sequential application of the established procedure for the single parameter case. Thus, if the probability model is p(t ∣ θ, λ), θ ∈ Θ, λ ∈ Λ and a θ-reference prior π θ (θ, λ) is required, the reference algorithm proceeds in two steps: . Conditional on θ, p(t ∣ θ, λ) only depends on the nuisance parameter λ and, hence, the one-parameter algorithm may be used to obtain the conditional reference prior π(λ ∣ θ).
If the conditional reference prior is not proper, then the procedure is performed within an increasing sequence {Λ i } of subsets converging to Λ over which π(λ ∣ θ) is integrable. This makes it possible to obtain a corresponding sequence of θ-reference posteriors {π i (θ ∣ t} for the quantity of interest θ, and the required reference posterior is the corresponding pointwise limit π(θ ∣ t) = limi π i (θ ∣ t). A θ-reference prior is then defined as a positive function π θ (θ, λ) which may be formally used in Bayes’ theorem as a prior to obtain the reference posterior, i.e., such that, for any t ∈ T, π(θ ∣ t) ∝ ∫Λ p(t ∣ θ, λ) π θ (θ, λ) dλ. The approximating sequences should be consistently chosen within a given model. Thus, given a probability model {p(x ∣ ω), ω ∈ Ω} an appropriate approximating sequence {Ω i } should be chosen for the whole parameter space Ω. Thus, if the analysis is done in terms of, say, ψ = {ψ , ψ } ∈ Ψ(Ω), the approximating sequence should be chosen such that Ψi = ψ(Ω i ). A natural approximating sequence in location-scale problems is {µ, log σ} ∈ [−i, i] . The θ-reference prior does not depend on the choice of the nuisance parameter λ; thus, for any ψ = ψ(θ, λ) such that (θ, ψ) is a one-to-one function of (θ, λ), the θ-reference prior in terms of (θ, ψ) is simply π θ (θ, ψ) = π θ (θ, λ)/∣∂(θ, ψ)/∂(θ, λ)∣, the appropriate probability transformation of the θ-reference prior in terms of (θ, λ). Notice, however, that the reference prior may depend on the parameter of interest; thus, the θ-reference prior may differ from the ϕ-reference prior unless either ϕ is a piecewise one-to-one transformation of θ, or ϕ is asymptotically independent of θ. This is an expected consequence of the fact that the conditions under which the missing information about θ is maximized are not generally the same as the conditions which maximize the missing information about some function ϕ = ϕ(θ, λ). The non-existence of a unique “noninformative prior” which would be appropriate for any inference problem within a given model was established by Dawid et al. (), when they showed that this is incompatible with consistent marginalization. Indeed, if given the model p(D ∣ θ, λ), the reference posterior of the quantity of interest θ, π(θ ∣ D) = π(θ ∣ t), only depends on the data through a statistic t whose sampling distribution, p(t ∣ θ, λ) = p(t ∣ θ), only depends on θ, one would expect the reference posterior to be of the form π(θ ∣ t) ∝ π(θ) p(t ∣ θ)
B
Bayesian Statistics
for some prior π(θ). However, examples were found where this cannot be the case if a unique joint “noninformative” prior were to be used for all possible quantities of interest. Example (Regular two dimensional continuous reference prior functions). If the joint posterior distribution of (θ, λ) is asymptotically normal, then the θ-reference prior may be derived in terms of the corresponding Fisher’s information matrix, F(θ, λ). Indeed, if ⎛F (θ, λ) θθ F(θ, λ) = ⎜ ⎜ ⎝Fθ λ (θ, λ)
Fθ λ (θ, λ)⎞ ⎟, ⎟ F λ λ (θ, λ)⎠
⎛ F(ϕ, σ) = ⎜ ⎜ − ⎝ϕσ
and
S(θ, λ) = F − (θ, λ), then the θ-reference prior is π θ (θ, λ) = π(λ ∣ θ) π(θ), where / π(λ ∣ θ) ∝ F λ λ (θ, λ), λ ∈ Λ. If π(λ ∣ θ) is proper, −/
π(θ) ∝ exp { ∫ π(λ ∣ θ) log [Sθ θ (θ, λ)] dλ},
θ ∈ Θ.
Λ
If π(λ ∣ θ) is not proper, integrations are performed on an approximating sequence {Λ i } to obtain a sequence {π i (λ ∣ θ) π i (θ)}, (where π i (λ ∣ θ) is the proper renormalization of π(λ ∣ θ) to Λ i ) and the θ-reference prior π θ (θ, λ) is defined as its appropriate limit. Moreover, if () both / −/ F λ λ (θ, λ) and Sθ θ (θ, λ) factorize, so that −/
Sθ θ (θ, λ) ∝ fθ (θ) gθ (λ),
/
F λ λ (θ, λ) ∝ f λ (θ) g λ (λ),
and () the parameters θ and λ are variation independent, so that Λ does not depend on θ, then the θ-reference prior is simply π θ (θ, λ) = fθ (θ) g λ (λ), even if the conditional reference prior π(λ ∣ θ) = π(λ) ∝ g λ (λ) (which will not depend on θ) is actually improper. Example (Inference on normal parameters, continued). The information matrix which corresponds to a normal model N(x ∣ µ, σ) is ⎛σ − F(µ, σ) = ⎜ ⎜ ⎝
the fact that F(µ, σ) is diagonal and also anticipated, it is similarly found that the σ-reference prior is π σ (µ, σ) = σ − , the same as π µ (µ, σ). Suppose, however, that the quantity of interest is not the mean µ or the standard deviation σ, but the standardized mean ϕ = µ/σ. Fisher’s information matrix in terms of the parameters ϕ and σ is F(ϕ, σ) = J t F(µ, σ) J, where J = (∂(µ, σ)/∂(ϕ, σ)) is the Jacobian of the inverse transformation; this yields
⎞ ⎟, ⎟ − σ ⎠
⎛σ ⎞ ⎟; S(µ, σ) = F − (µ, σ) = ⎜ ⎟ ⎜ ⎝ σ ⎠ √ / hence Fσ σ (µ, σ) = σ − = fσ (µ) gσ (σ), with gσ (σ) = −/ σ − , and π(σ ∣ µ) = σ − . Similarly, S µ µ (µ, σ) = σ − = f µ (µ) g µ (σ), with f µ (µ) = , and π(µ) = . Therefore, the µ-reference prior is π µ (µ, σ) = π(σ ∣ µ) π(µ) = σ − , as already anticipated. Moreover, as one would expect from
⎛ + ϕ S(ϕ, σ) = ⎜ ⎜ ⎝ − ϕσ −/
⎞ ⎟, ⎟ σ − ( + ϕ )⎠ ϕσ −
− ϕσ ⎞ ⎟. ⎟ σ ⎠ −/
Thus, Sϕϕ (ϕ, σ) ∝ ( + ϕ )
/
and Fσ σ (ϕ, σ) ∝
σ − ( + ϕ )/ . Hence, using again the results in Exam−/ − ple , π ϕ (ϕ, σ) = ( + ϕ ) σ . In the original −/
parametrization, this is π ϕ (µ, σ) = ( + (µ/σ) ) σ − , which is very different from π µ (µ, σ) = π σ (µ, σ) = σ − . The corresponding reference posterior of ϕ is π(ϕ ∣ x , . . ., / −/ p(t ∣ ϕ) where t = (∑ xj )/ (∑ xj ) , xn ) ∝ ( + ϕ ) a one-dimensional (marginally sufficient) statistic whose sampling distribution, p(t ∣ µ, σ) = p(t ∣ ϕ), only depends on ϕ. Thus, the reference prior algorithm is seen to be consistent under marginalization. Many parameters. The reference algorithm is easily generalized to an arbitrary number of parameters. If the model is p(t ∣ ω , . . . , ω m ), a joint reference prior π(θ m ∣ θ m− , . . . , θ ) × . . . × π(θ ∣ θ ) × π(θ ) may sequentially be obtained for each ordered parametrization {θ (ω), . . . , θ m (ω)} of interest, and these are invariant under reparametrization of any of the θ i (ω)’s. The choice of the ordered parametrization {θ , . . . , θ m } precisely describes the particular prior required, namely that which sequentially maximizes the missing information about each of the θ i ’s, conditional on {θ , . . . , θ i− }, for i = m, m − , . . . , . Example (Stein’s paradox). Let D be a random sample from a m-variate normal distribution with mean µ = {µ , . . . , µ m } and unitary variance matrix. The reference prior which corresponds to any permutation of the µ i ’s is uniform, and this prior leads indeed to appropriate reference posterior distributions for any of the µ i ’s, √ namely π(µ i ∣ D) = N (µ i ∣ xi , / n). Suppose, however, that the quantity of interest is θ = ∑i µ i , the distance of µ to the origin. As showed by Stein in the s, the
B
B
Bayesian Statistics
posterior distribution of θ based on that uniform prior (or in any “flat” proper approximation) has very undesirable properties; this is due to the fact that a uniform (or nearly uniform) prior, although “noninformative” with respect to each of the individual µ i ’s, is actually highly informative on the sum of their squares, introducing a severe positive bias (Stein’s paradox). However, the reference prior which corresponds to a parametrization of the form {θ, λ , . . . , λ m− } produces, for any choice of the nuisance parameters λ i = λ i (µ), the reference posterior π(θ ∣ D) = π(θ ∣ t) ∝ θ −/ χ (nt ∣ m, nθ), where t = ∑i xi , and this posterior is shown to have the appropriate consistency properties. Far from being specific to Stein’s example, the inappropriate behavior in problems with many parameters of specific marginal posterior distributions derived from multivariate “flat” priors (proper or improper) is indeed very frequent. Hence, sloppy, uncontrolled use of “flat” priors (rather than the relevant reference priors), is strongly discouraged. Limited information. Although often used in contexts where no universally agreed prior knowledge about the quantity of interest is available, the reference algorithm may be used to specify a prior which incorporates any acceptable prior knowledge; it suffices to maximize the missing information within the class P of priors which is compatible with such accepted knowledge. Indeed, by progressive incorporation of further restrictions into P, the reference prior algorithm becomes a method of (prior) probability assessment. As described below, the problem has a fairly simple analytical solution when those restrictions take the form of known expected values. The incorporation of other type of restrictions usually involves numerical computations. Example (Univariate restricted reference priors). If the probability mechanism which is assumed to have generated the available data only depends on a univarite continuous parameter θ ∈ Θ ⊂ R, and the class P of acceptable priors is a class of proper priors which satisfies some expected value restrictions, so that P = {p(θ);
p(θ) > , ∫ p(θ) dθ = , Θ
∫ gi (θ) p(θ) dθ = β i , i = , . . . , m} Θ
then the (restricted) reference prior is m
π(θ ∣ P) ∝ π(θ) exp [∑j= γ i gi (θ)] where π(θ) is the unrestricted reference prior and the γ i ’s are constants (the corresponding Lagrange multipliers), to
be determined by the restrictions which define P. Suppose, for instance, that data are considered to be a random sample from a location model centered at θ, and that it is further assumed that E[θ] = µ and that Var[θ] = σ . The unrestricted reference prior for any regular location problem may be shown to be uniform. Thus, the restricted reference prior must be of the form π(θ ∣ P) ∝ exp{γ θ + γ (θ − µ ) }, with ∫Θ θ π(θ ∣ P) dθ = µ and ∫Θ (θ − µ ) π(θ ∣ P) dθ = σ . Hence, π(θ ∣ P) is a normal distribution with the specified mean and variance.
Frequentist Properties Bayesian methods provide a direct solution to the problems typically posed in statistical inference; indeed, posterior distributions precisely state what can be said about unknown quantities of interest given available data and prior knowledge. In particular, unrestricted reference posterior distributions state what could be said if no prior knowledge about the quantities of interest were available. A frequentist analysis of the behavior of Bayesian procedures under repeated sampling may, however, be illuminating, for this provides some interesting connections between frequentist and Bayesian inference. It is found that the frequentist properties of Bayesian reference procedures are typically excellent, and may be used to provide a form of calibration for reference posterior probabilities. Point Estimation. It is generally accepted that, as the sample size increases, a “good” estimator θ˜ of θ ought to get the correct value of θ eventually, that is to be consistent. Under appropriate regularity conditions, any Bayes estimator ϕ∗ of any function ϕ(θ) converges in probability to ϕ(θ), so that sequences of Bayes estimators are typically consistent. Indeed, it is known that if there is a consistent sequence of estimators, then Bayes estimators are consistent. The rate of convergence is often best for reference Bayes estimators. It is also generally accepted that a “good” estimator should be admissible, that is, not dominated by any other estimator in the sense that its expected loss under sampling (conditional to θ) cannot be larger for all θ values than that corresponding to another estimator. Any proper Bayes estimator is admissible; moreover, as established by Wald (), a procedure must be Bayesian (proper or improper) to be admissible. Most published admissibility results refer to quadratic loss functions, but they often extend to more general loss functions. Reference Bayes estimators are typically admissible with respect to intrinsic loss functions. Notice, however, that many other apparently intuitive frequentist ideas on estimation have been proved to be potentially misleading. For example, given a sequence of n
Bayesian Statistics
Bernoulli observations with parameter θ resulting in r positive trials, the best unbiased estimate of θ is found to be r(r − )/{n(n − )}, which yields θ˜ = when r = ; but to estimate the probability of two positive trials as zero, when one positive trial has been observed, is not at all sensible. In marked contrast, any Bayes reference estimator provides a reasonable answer. For example, the intrinsic estimator of θ is simply (θ ∗ ) , where θ ∗ is the intrinsic estimator of θ described in section “7Estimation”. In particular, if r = and n = the intrinsic estimator of θ is (as one would naturally expect) (θ ∗ ) = /. Interval Estimation. As the sample size increases, the frequentist coverage probability of a posterior q-credible region typically converges to q so that, for large samples, Bayesian credible intervals may (under regularity conditions) be interpreted as approximate frequentist confidence regions: under repeated sampling, a Bayesian q-credible region of θ based on a large sample will cover the true value of θ approximately q% of times. Detailed results are readily available for univariate problems. For instance, consider the probability model {p(D ∣ ω), ω ∈ Ω}, let θ = θ(ω) be any univariate quantity of interest, and let t = t(D) ∈ T be any sufficient statistic. If θ q (t) denotes the q% quantile of the posterior distribution of θ which corresponds to some unspecified prior, so that Pr[θ ≤ θ q (t) ∣ t] = ∫
p(θ ∣ t) dθ = q, θ≤θ q (t)
then the coverage probability of the q-credible interval {θ; θ ≤ θ q (t)}, Pr[θ q (t) ≥ θ ∣ ω] = ∫
p(t ∣ ω) dt, θ q (t)≥θ
is such that Pr[θ q (t) ≥ θ ∣ ω] = Pr[θ ≤ θ q (t) ∣ t] + O(n−/ ). This asymptotic approximation is true for all (sufficiently regular) positive priors. However, the approximation is better, actually O(n− ), for a particular class of priors known as (first-order) probability matching priors. Reference priors are typically found to be probability matching priors, so that they provide this improved asymptotic agreement. As a matter of fact, the agreement (in regular problems) is typically quite good even for relatively small samples. Example (Product of normal means). Consider the case where independent random samples {x , . . . , xn } and {y , . . . , ym } have respectively been taken from the normal densities N(x ∣ ω , ) and N(y ∣ ω , ), and suppose that the quantity of interest is the product of their means, ϕ = ω ω (for instance, one may be interested in inferences about the area ϕ of a rectangular piece of land, given measurements {xi } and {yj } of its sides). Notice that this is a simplified version of a problem that it is often encountered in
B
the sciences, where one is interested in the product of several magnitudes, all of which have been measured with error. Using the procedure described in Example , with the natural approximating sequence induced by (ω , ω ) ∈ [−i, i] , the ϕ-reference prior is found to be −/
π ϕ (ω , ω ) ∝ (n ω + m ω )
,
very different from the uniform prior π ω (ω , ω ) = π ω (ω , ω ) = which should be used to make objective inferences about either ω or ω . The prior π ϕ (ω , ω ) may be shown to provide approximate agreement between Bayesian credible regions and frequentist confidence intervals for ϕ; indeed, this prior (with m = n) was originally suggested by Stein in the s to obtain such approximate agreement. The same example was later used by Efron () to stress the fact that, even within a fixed probability model {p(D ∣ ω), ω ∈ Ω}, the prior required to make objective inferences about some function of the parameters ϕ = ϕ(ω) must generally depend on the function ϕ. The numerical agreement between reference Bayesian credible regions and frequentist confidence intervals is actually perfect in special circumstances. Indeed, as Lindley () pointed out, this is the case in those problems of inference which may be transformed to location-scale problems. Example (Inference on normal parameters, continued). Let D = {x , . . . xn } be a random sample from a normal distribution N(x ∣ µ, σ). As mentioned before, the reference posterior of the quantity of interest µ is the Student √ distribution St (µ ∣ x, s/ n − , n − ). Thus, normalizing √ µ, the posterior distribution of t(µ) = n − (x − µ)/s, as a function of µ given D, is the standard Student St(t ∣ , , n − ) with n − degrees of freedom. On the other hand, this function t is recognized to be precisely the conventional t statistic, whose sampling distribution is well known to also be standard Student with n − degrees of freedom. It follows that, for all sample sizes, posterior reference credible intervals for µ given the data will be numerically identical to frequentist confidence intervals based on the sampling distribution of t. A similar result is obtained in inferences about the variance. Thus, the reference posterior distribution of λ = σ − is the 7Gamma distribution Ga(λ ∣ (n − )/, ns /) and, hence, the posterior distribution of r = ns /σ , as a function of σ given D, is a (central) χ with n − degrees of freedom. But the function r is recognized to be a conventional statistic for this problem, whose sampling distribution is well known to also be χ with n − degrees of freedom. It follows that, for all sample sizes, posterior reference credible intervals for σ (or any one-to-one
B
B
Bayesian Statistics
function of σ ) given the data will be numerically identical to frequentist confidence intervals based on the sampling distribution of r. For a recent review or modern objective Bayesian inference, see Bernardo ().
Discussion In writing a broad article it is always hard to decide what to leave out. This article focused on the basic concepts of the Bayesian paradigm; methodological topics which have unwillingly been omitted include design of experiments, sample surveys, linear models and sequential methods. The interested reader is referred to the bibliography for further information. This final section briefly reviews the main arguments for the Bayesian approach, and includes pointers to further issues which have not been discussed in more detail due to space limitations.
Coherence By using probability distributions to characterize all uncertainties in the problem, the Bayesian paradigm reduces statistical inference to applied probability, thereby ensuring the coherence of the proposed solutions. There is no need to investigate, on a case by case basis, whether or not the solution to a particular problem is logically correct: a Bayesian result is only a mathematical consequence of explicitly stated assumptions and hence, unless a logical mistake has been committed in its derivation, it cannot be formally wrong. In marked contrast, conventional statistical methods are plagued with counterexamples. These include, among many others, negative estimators of positive quantities, q-confidence regions (q < ) which consist of the whole parameter space, empty sets of “appropriate” solutions, and incompatible answers from alternative methodologies simultaneously supported by the theory. The Bayesian approach does require, however, the specification of a (prior) probability distribution over the parameter space. The sentence “a prior distribution does not exist for this problem” is often stated to justify the use of non-Bayesian methods. However, the general representation theorem proves the existence of such a distribution whenever the observations are assumed to be exchangeable (and, if they are assumed to be a random sample then, a fortiori, they are assumed to be exchangeable). To ignore this fact, and to proceed as if a prior distribution did not exist, just because it is not easy to specify, is mathematically untenable.
Objectivity It is generally accepted that any statistical analysis is subjective, in the sense that it is always conditional on accepted
assumptions (on the structure of the data, on the probability model, and on the outcome space) and those assumptions, although possibly well founded, are definitely subjective choices. It is, therefore, mandatory to make all assumptions very explicit. Users of conventional statistical methods rarely dispute the mathematical foundations of the Bayesian approach, but claim to be able to produce “objective” answers in contrast to the possibly subjective elements involved in the choice of the prior distribution. Bayesian methods do indeed require the choice of a prior distribution, and critics of the Bayesian approach systematically point out that in many important situations, including scientific reporting and public decision making, the results must exclusively depend on documented data which might be subject to independent scrutiny. This is of course true, but those critics choose to ignore the fact that this particular case is covered within the Bayesian approach by the use of reference prior distributions which () are mathematically derived from the accepted probability model (and, hence, they are “objective” insofar as the choice of that model might be objective) and, () by construction, they produce posterior probability distributions which, given the accepted probability model, only contain the information about their values which data may provide and, optionally, any further contextual information over which there might be universal agreement. An issue related to objectivity is that of the operational meaning of reference posterior probabilities; it is found that the analysis of their behavior under repeated sampling provides a suggestive form of calibration. Indeed, Pr[θ ∈ R ∣ D] = ∫R π(θ ∣ D) dθ, the reference posterior probability that θ ∈ R, is both a measure of the conditional uncertainty (given the assumed model and the observed data D) about the event that the unknown value of θ belongs to R ⊂ Θ, and the limiting proportion of the regions which would cover θ under repeated sampling conditional on data “sufficiently similar” to D. Under broad conditions (to guarantee regular asymptotic behavior), all large data sets from the same model are “sufficiently similar” among themselves in this sense and hence, given those conditions, reference posterior credible regions are approximate unconditional frequentist confidence regions. The conditions for this approximate unconditional equivalence to hold exclude, however, important special cases, like those involving “extreme” or “relevant” observations. In very special situations, when probability models may be transformed to location-scale models, there is an exact unconditional equivalence; in those cases reference posterior credible intervals are, for any sample size, exact unconditional frequentist confidence intervals.
B
Bayesian Statistics
Applicability In sharp contrast to most conventional statistical methods, which may only be exactly applied to a handful of relatively simple stylized situations, Bayesian methods are (in theory) totally general. Indeed, for a given probability model and prior distribution over its parameters, the derivation of posterior distributions is a well-defined mathematical exercise. In particular, Bayesian methods do not require any particular regularity conditions on the probability model, do not depend on the existence of sufficient statistics of finite dimension, do not rely on asymptotic relations, and do not require the derivation of any sampling distribution, nor (a fortiori) the existence of a “pivotal” statistic whose sampling distribution is independent of the parameters. However, when used in complex models with many parameters, Bayesian methods often require the computation of multidimensional definite integrals and, for a long time in the past, this requirement effectively placed practical limits on the complexity of the problems which could be handled. This has dramatically changed in recent years with the general availability of large computing power, and the parallel development of simulation-based numerical integration strategies like importance sampling or 7Markov chain Monte Carlo (MCMC). These methods provide a structure within which many complex models may be analyzed using generic software. MCMC is numerical integration using Markov chains. Monte Carlo integration proceeds by drawing samples from the required distributions, and computing sample averages to approximate expectations. MCMC methods draw the required samples by running appropriately defined 7Markov chains for a long time; specific methods to construct those chains include the Gibbs sampler and the Metropolis algorithm, originated in the s in the literature of statistical physics. The development of improved algorithms and appropriate diagnostic tools to establish their convergence, remains a very active research area. Actual scientific research often requires the use of models that are far too complex for conventional statistical methods. This article concludes with a glimpse at some of them. Hierarchical structures. Consider a situation where a possibly variable number ni of observations, {xij , j = , . . . , ni }, i = , . . . , m, are made on each of m internally homogeneous subsets of some population. For instance, a firm might have chosen m production lines for inspection, and ni items might have been randomly selected among those made by production line i, so that xij is the result of the measurements made on item j of production line i. As another example, animals of some species are captured to
study their metabolism, and a blood sample taken before releasing them again; the procedure is repeated in the same habitat for some time, so that some of the animals are recaptured several times, and xij is the result of the analysis of the j-th blood sample taken from animal i. It those situations, it is often appropriate to assume that the ni observations on subpopulation i are exchangeable, so that they may be treated as a random sample from some model p(x ∣ θ i ) indexed by a parameter θ i which depends on the subpopulation observed, and that the parameters which label the subpopulations may also be assumed to be exchangeable, so that {θ , . . . , θ m } may be treated as a random sample from some distribution p(θ ∣ ω). Thus, the complete hierarchical model which is assumed to have generated the observed data D = {x , . . . , xmnm } is of the form p(D ∣ ω) = ∫
Θm
ni
m
m
j=
i=
i=
[ ∏ p(xij ∣ θ i )][ ∏ p(θ i ∣ ω)] [ ∏ dθ i ].
Hence, under the Bayesian paradigm, a family of conventional probability models, say p(x ∣ θ), θ ∈ Θ, and an appropriate “structural” prior p(θ ∣ ω), may be naturally combined to produce a versatile, complex model {p(D ∣ ω), ω ∈ Ω} whose analysis is often well beyond the scope of conventional statistics. The Bayesian solution only requires the specifiction a prior distribution p(ω), the use Bayes’ theorem to obtain the corresponding posterior p(ω ∣ D) ∝ p(D ∣ ω) p(ω), and the performance of the appropriate probability transformations to derive the posterior distributions of the quantities of interest (which may well be functions of ω, functions of the θ i ’s, or functions of future observations). As in any other Bayesian analysis, the prior distribution p(ω) has to describe available knowledge about ω; if none is available, or if an objective analysis is required, an appropriate reference prior function π(ω) may be used. Contextual information. In many problems of statistical inference, objective and universally agreed contextual information is available on the parameter values. This information is usually very difficult to handle within the framework of conventional statistics, but it is easily incorporated into a Bayesian analysis by simply restricting the prior distribution to the class P of priors which are compatible with such information. As an example, consider the frequent problem in archaeology of trying to establish the occupation period [α, β] of a site by some past culture on the basis of the radiocarbon dating of organic samples taken from the excavation. Radiocarbon dating is not precise, so that each dating xi is typically taken to be a normal observation from a distribution N(x ∣ µ(θ i ), σi ), where θ i is the actual, unknown calendar date of the sample, µ(θ)
B
B
Bayesian Statistics
is an internationally agreed calibration curve, and σi is a known standard error quoted by the laboratory. The actual calendar dates {θ , . . . , θ m } of the samples are typically assumed to be uniformly distributed within the occupation period [α, β]; however, stratigraphic evidence indicates some partial orderings for, if sample i was found on top of sample j in undisturbed layers, then θ i > θ j . Thus, if C denotes the class of values of {θ , . . . , θ m } which satisfy those known restrictions, data may be assumed to have been generated by the hierarchical model m
p(x , . . . , xm ∣ α, β) = ∫ [ ∏ N (xi ∣ µ(θ i ), σi ) ] C −m
(β − α)
i=
dθ . . . dθ m .
Often, contextual information further indicates an absolute lower bound α and an absolute upper bound β for the period investigated, so that α < α < β < β . If no further documented information is available, the corresponding restricted reference prior for the quantities of interest, {α, β} should be used; this is found to be π(α, β) ∝ (β − α)− whenever α < α < β < β and zero otherwise. The corresponding reference posterior π(α, β ∣ x , . . . , xm ) ∝ p(x , . . . , xm ∣ α, β) π(α, β) summarizes all available information on the occupation period. Covariate information. Over the last years, both linear and non-linear regression models have been analyzed from a Bayesian point of view at increasing levels of sophistication. These studies range from the elementary objective Bayesian analysis of simple linear regression structures (which parallel their frequentist counterparts) to the sophisticated analysis of time series involved in dynamic forecasting which often make use of complex hierarchical structures. The field is far too large to be reviewed in this article, but the bibliography contains some relevant pointers. Model Criticism. It has been stressed that any statistical analysis is conditional on the accepted assumptions of the probability model which is presumed to have generated the data. Recent years have shown a huge effort into the development of Bayesian procedures for model criticism and model choice. Most of these are sophisticated elaborations of the procedures described in section “7Hypothesis Testing” under the heading of hypothesis testing. Again, this is too large a topic to be reviewed here, but some key references are included in the bibliography.
Acknowledgments Work has been partially funded with Grant MTM of the MEC, Spain.
About the Author Professor José Bernardo is founder co-President of the International Society for Bayesian Analysis (–). He is the personal organizer and Programme Committee Member of the Valencia International Meetings on Bayesian Statistics, established world forums on Bayesian Methods, held every years in Spain since . Spanish Royal Academy of Science invited Professor Bernardo to become the Editor of the special issue of its journal the Rev. Acad. Cien. Madrid, devoted entirely to Bayesian Methods in the Sciences (). He has co-edited (as the first author) eight proceedings of the Valencia meetings on Bayesian Statistics. Professor Bernardo is a Fellow of the American Statistical Association, Royal Statistical Society and Elected Member of the International Statistical Institute. He was Founding Editor of Test (–), associate editor for the Journal of the Royal Statistical Society (Series B) (–), The Statistician (–), Questio (–); Contributing Editor of the Current Index of Statistics and Statistics Theory and Methods Abstracts (–). Currently he is an Associate editor for the Journal of the Iranian Statistical Society. He was delivering lectures at more than universities worldwide. He is a co-author (with Adrian Smith) of the well known text Bayesian Theory (Wiley, ).
Cross References 7Bayes’ Theorem 7Bayesian Analysis or Evidence Based Statistics? 7Bayesian Nonparametric Statistics 7Bayesian Versus Frequentist Statistical Reasoning 7Bayesian vs. Classical Point Estimation: A Comparative
Overview 7Foundations of Probability 7Likelihood 7Markov Chain Monte Carlo 7Model Selection 7Prior Bayes: Rubin’s View of Statistics 7Statistical Inference 7Statistical Inference: An Overview 7Statistics: An Overview
References and Further Reading Bartlett M () A comment on D.V. Lindley’s statistical paradox. Biometrika :– Berger JO () Statistical decision theory and Bayesian analysis. Springer, Berlin Berger JO () Bayesian analysis: a look at today and thoughts of tomorrow. J Am Stat Assoc :– Berger JO, Bernardo JM () Estimating a product of means: Bayesian analysis with reference priors. J Am Stat Assoc :–
Bayesian Versus Frequentist Statistical Reasoning
Berger JO, Bernardo JM () On the development of reference priors. In: Bernardo JM, Berger JO, Dawid AP, Smith AFM (eds) Bayesian statistics, vol . Oxford University Press, Oxford, pp – (with discussion) Berger J, Bernardo JM, Sun D (a) The formal definition of reference priors. Ann Stat :– Berger JO, Bernardo JM, Sun D (b) Natural induction: an objective Bayesian approach. Rev Acad Sci Madrid A :– (with discussion) Bernardo JM (a) Expected information as expected utility. Ann Stat :– Bernardo JM (b) Reference posterior distributions for Bayesian inference. J R Stat Soc B : – (with discussion). In: Tiao GC, Polson GC (eds) Reprinted in Bayesian Inference . Edward Elgar, Oxford, pp – Bernardo JM () Noninformative priors do not exist. J Stat Plann Infer :– (with discussion) Bernardo JM (a) Reference analysis. In: Dey DK, Rao CR (eds) Handbook of Statistics, vol . Elsevier, Amsterdam, pp – Bernardo JM (b) Intrinsic credible regions: An objective Bayesian approach to interval estimation. Test :– (with discussion) Bernardo JM () Integrated objective Bayesian estimation and hypothesis testing. In: Bernardo JM et al. (eds) Bayesian Statistics . Oxford: Oxford University Press, (to appear, with discussion) Bernardo JM, Ramón JM () An introduction to Bayesian reference analysis: inference on the ratio of multinomial parameters. The Statistician :– Bernardo JM, Rueda R () Bayesian hypothesis testing: a reference approach. Int Stat Rev :– Bernardo JM, Smith AFM () Bayesian theory. Wiley, Chichester Bernardo JM, Bayarri MJ, Berger JO, Dawid AP, Heckerman D, Smith AFM, West M (eds) () Bayesian statistics . Oxford University Press, Oxford Bernardo JM, Bayarri MJ, Berger JO, Dawid AP, Heckerman D, Smith AFM, West M (eds) () Bayesian statistics . Oxford University Press, Oxford Berry DA () Statistics, a Bayesian perspective. Wadsworth, Belmont Box GEP, Tiao GC () Bayesian inference in statistical analysis. Addison-Wesley, Reading Dawid AP, Stone M, Zidek JV () Marginalization paradoxes in Bayesian and structural inference. J R Stat Soc B :– (with discussion) de Finetti B () Teoria delle Probabilità. Einaudi, Turin. English translation: Theory of Probability () Wiley, Chichester DeGroot MH () Optimal statistical decisions. McGraw-Hill, New York de Finetti B () La prévision, ses lois logiques, ser sources sunjecives. Ann Inst Henri Poincaré :– Efron B () Why isn’t everyone a Bayesian? Am Stat :– (with discussion) Geisser S () Predictive inference: an introduction. Chapman and Hall, London Gelfand AE, Smith AFM () Sampling based approaches to calculating marginal densities. J Am Stat Assoc : – Gelman A, Carlin JB, Stern H, Rubin DB () Bayesian data analysis. Chapman and Hall, London
B
Gilks WR, Richardson S, Spiegelhalter DJ () Markov chain Monte Carlo in practice. Chapman and Hall, London Jaynes ET () Confidence intervals vs Bayesian intervals. In: Harper WL, Hooker CA (eds) Foundations of probability theory, statistical inference and statistical theories of science, vol . Reidel, Dordrecht, pp – (with discussion) Jeffreys H () Theory of probability, rd edn. Oxford University Press, Oxford Kass RE, Raftery AE () Bayes factors. J Am Stat Assoc : – Laplace PS () Théorie Analitique des Probabilités. Paris: Gauthier-Villars Lindley DV () A statistical paradox. Biometrika :– Lindley DV () Fiducial distribution and Bayes theorem. J R Stat Soc B :– Lindley DV () Introduction to probability and statistics from a Bayesian viewpoint. Cambridge University Press, Cambridge Lindley DV () Bayesian Statistics, a review. SIAM, Philadelphia Lindley DV () Making Decisions, nd edn. Wiley, Chichester Lindley DV () The philosophy of statistics. The Statistician :– (with discussion) O’Hagan A () Bayesian Inference. Edward Arnold, London Press SJ () Applied multivariate analysis: using Bayesian and frequentist methods of inference. Krieger, Melbourne Ramsey FP () Truth and probability. In: Braithwaite RB (ed) The foundations of mathematics and other logical essays. London: Kegan Paul, pp – Wald A () Statistical decision functions. Wiley, Chichester West M, Harrison PJ () Bayesian forecasting and dynamic models. Springer, Berlin Zellner A () An introduction to Bayesian inference in econometrics. Wiley, New York. Reprinted in , Krieger, Melbourne
Bayesian Versus Frequentist Statistical Reasoning Jordi Vallverdú Universitat Autònoma de Barcelona, Catalonia, Spain We can consider the existence of two main statistical schools: Bayesian and frequentist. Both provide ways to deal with probability, although their methods and theories are mutually exclusive (Vallverdú ).
Bayesian Statistics From a historical perspective, Bayesian appeared first, in , when Richard Price published posthumously the paper of late Rev. Thomas Bayes “An Essay towards solving a Problem in the Doctrine of Chances” (Dale ). In this paper, Bayes presented his ideas about the best way
B
B
Bayesian Versus Frequentist Statistical Reasoning
of dealing with probability (and trying to solve the problem of inverse probability), which can be exemplified today with the classic formula called “Bayes’ Rule” or “Bayes’ Theorem”: P (B ∣A ) P (A) . P (A ∣B ) = P (B) We must look at the notation and terminology involved: ●
● ●
●
P(A∣B) is the conditional probability of A, given B. It is also called the posterior probability because it is derived from or depends upon the specified value of B. P(B∣A) is the conditional probability of B given A. P(A) is the prior probability or marginal probability of A. It is “prior” in the sense that it does not take into account any information about B. P(B) is the prior or marginal probability of B, and acts as a normalizing constant.
We can see, then, that our posterior belief P(A∣B) is calculated by multiplying our prior belief P(A) by the likelihood P(B∣A) that B will occur if A is true. Although Bayes’ method was enthusiastically taken up by Laplace and other leading probabilists of the day, it fell into disrepute in the nineteenth century because they did not yet know how to handle prior probabilities properly. The prior probability of A represents our best estimate of the probability of the fact we are considering, prior to consideration of the new piece of evidence. Therefore, in the Bayesian paradigm, current knowledge about the model parameters is expressed by placing a probability distribution on the parameters, called the “prior distribution.” When new data become available, the information they contain regarding the model parameters is expressed in the “likelihood,” which is proportional to the distribution of the observed data given the model parameters. This information is then combined with the prior to produce an updated probability distribution called the “posterior distribution,” on which all Bayesian inference is based. 7Bayes’ Theorem, an elementary identity in probability theory, states how the update is done mathematically: the posterior is proportional to the prior times the likelihood. There are a large number of types of Bayesians (speaking ironically, Good () spoke of “, kinds of Bayesians”), depending on their attitude toward subjectivity in postulating priors. Some recent Bayesian books are Earman (), Howson and Urbach (), Bernardo and Smith ().
Frequentist Statistics On the other hand, we have the frequentist paradigm. Its followers understand probability as a long-run frequency of a “repeatable” event and developed a notion of confidence intervals. Probability would be, then, a measurable
frequency of events determined from repeated experiments. We can express it as: P (A) = n/N, where n is the number of times event A occurs in N opportunities. From the frequentist viewpoint two closely related methods have been developed. One is the Neyman– Pearson theory of significance tests and the other is based on Fisher’s notion of 7p-values. The researchers who follow this approach, consider frequentism as the only allowed statistical method for achieving sound scientific inferences (Mayo and Cox ).
The Debate: Degrees of Belief Versus Relative Frequencies As early as in , Maurice George Kendall () wrote a paper, “On the Reconciliation of Theories of Probability,” in which he coined the word “frequentist” and stated: “Few branches of scientific method have been subject to so much difference of opinion as the theory of probability.” He tried to attempt mediation between the contestants, but failed. Clearly, one of the recurrent arguments against/in favor of one of the two positions (frequentist or Bayesian) consists in saying that a true scientist is always/never frequentist/Bayesian (you can choose between the two possibilities). As an example of this confrontation see the ideas of Giere (): “Are Scientists Bayesian Agents? (. . .) The overwhelming conclusion is that humans are not bayesian agents,” and of Efron () or Cousins (). The last two do not need to be quoted. It seems to be an epistemological law about statistical practices: “A true scientist never belongs to the opposite statistical school” (Vallverdú ). It could seem that frequentists are realists, when they consider relative frequencies and that Bayesian are subjective, when they defend degrees of belief of prior probabilities but the truth is that in cluster investigations, for example, the frequentist approach is just as subjective as the Bayesian approach, although the Bayesian approach is less ambitious in that it treats the analysis as a synthesis of data and personal judgments (possibly poor ones), rather than objective reality (Coory et al. ).
Why to Become Frequentist/Bayesian? Bland and Altman (, p. ) have their own answer: “Most statisticians have become Bayesians or Frequentists as a result of their choice of university.” And as the epidemiologist Berger () says: “practicing epidemiologists are given little guidance in choosing between these approaches apart from the ideological adherence of mentors, colleagues and editors.”
Bayesian Versus Frequentist Statistical Reasoning
So, the arguments go beyond the ethereal philosophical arena and become more practical ones. Better opportunities to find a good job is an important argument, and the value of a Bayesian academic training is now accepted: “where once graduate students doing Bayesian dissertations were advised to try not to look too Bayesian when they went on the job market, now great numbers of graduate students try to include some Bayesian flavor in their dissertations to increase their marketability” (Wilson , p. ). Therefore, and following Hacking (, p. ): “Euler at once retorted that this advice is metaphysical, not mathematical. Quite so! The choice of primitive concepts for inference is a matter of ‘metaphysics’. The orthodox statistician has made one metaphysical choice and the Bayesian another.” To be honest, there is not a fatally flawed position, but different context-based applications of both main approaches. As Gigerenzer et al. () express “we need statistical thinking, not statistical rituals.” Lilford and Braunholtz (, p. ) go further: “when the situation is less clear cut (. . .) conventional statistics may drive decision makers into a corner and produce sudden, large changes in prescribing. The problem does not lie with any of the individual decision makers, but with the very philosophical basis of scientific inference. We propose that conventional statistics should not be used in such cases and that the Bayesian approach is both epistemologically and practically superior.” There is also a structural aspect: computational facilities; due to recent innovations in scientific computing (faster computer processors) and drastic drops in the cost of computers, the number of statisticians trained in Bayesian methodology has increased (Tan ). Trying to offer a midpoint perspective, Berger () proposes using both models and studying case by case their possibilities: “based on the philosophical foundations of the approaches, Bayesian models are best suited to addressing hypotheses, conjectures, or public-policy goals, while the frequentist approach is best suited to those epidemiological studies which can be considered ‘experiments’, i.e., testing constructed sets of data.” Usually, we find no such equitable position. Considering all these facts, we can conclude that both frequentist and Bayesian statisticians use sound science in their researches and that, in the end, this debate is a deep philosophical one, not a matter of rational argument.
Acknowledgment This research was supported by the Project “El diseño del espacio en entornos de cognición distribuida: plantillas y affordances,” MCI [FFI-/FISO].
About the Author For biography see the entry 7Probability, History of.
B
Cross References 7Bayesian Analysis or Evidence Based Statistics? 7Bayesian Statistics 7Bayesian vs. Classical Point Estimation: A Comparative
Overview 7Foundations of Probability 7Frequentist Hypothesis Testing: A Defense 7Likelihood 7Model Selection 7Philosophical Foundations of Statistics 7Prior Bayes: Rubin’s View of Statistics 7Probability Theory: An Outline 7Probability, History of 7Significance Tests: A Critique 7Statistical Inference 7Statistical Inference: An Overview 7Statistics: An Overview 7Statistics: Nelder’s view
References and Further Reading Berger ZD () Bayesian and frequentist models: legitimate choices for different purposes. AEP (): Berger JA et al () Unified frequentist and Bayesian testing of a precise hypothesis. Stat Sci ():– Bernardo JM, Smith AFM () Bayesian theory. Wiley, USA Bland MJ, Altman DG () Bayesian and frequentists. Br Med J :– Coory MD, Wills RA, Barnett AG () Bayesian versus frequentist statistical inference for investigating a one-off cancer cluster reported to a health department. BMC Med Res Methodol : Cousins RD () Why isn’t every physicist a Bayesian? Am J Phys : Dale AI () Most honourable remembrance. The life and work of Thomas Bayes. Springer, New York Earman J () Bayes or bust? MIT Press, Cambridge, MA Efron B () Why isn’t everyone a Bayesian? Am Stat :– Giere R () Understanding scientific reasoning. University of Chicago, Chicago, p Gigerenzer G et al () The Empire of Chance. How probability changed science and everyday life. Cambridge University Press, Cambridge Good IJ () , kinds of Bayesians. Am Stat :– Hacking I () Likelihood. Brit J Philos Sci :– Howson C, Urbach P () Bayesian reasoning in science. Nature :– Kendall MG () On the Reconciliation of Theories of Probability. Biometrika :– Lilford RJ, Braunholtz D () For debate: the statistical basis of public policy: a paradigm shift is overdue. Br Med J :– Mayo DG, Cox DR () Frequentist statistics as a theory of inductive inference. nd Lehmann symposium - optimality IMS lecture notes - monographs series, – Tan SB () Bayesian methods for medical research. Ann Acad Med ():– Vallverdú J () The false dilemma: Bayesian versus Frequentist. E – L O G O S Electronic Journal for Philosophy – Wilson G () Tides of change: is Bayesianism the new paradigm in statistics? J Stat Plan Infer :–
B
B
Bayesian vs. Classical Point Estimation: A Comparative Overview
Bayesian vs. Classical Point Estimation: A Comparative Overview Francisco J. Samaniego Professor University of California-Davis, Davis, CA, USA The foundations of the classical theory of point estimation are embedded in the work of Frederick Gauss, Karl Pearson and Ronald Fisher, though there have been many other contributors, as documented in Stigler’s () historical masterpiece or, in more technical terms, in Lehmann and Casella (). In the framework of independent, identically distributed (i.i.d.) observations, the theory seeks to obtain good estimators (or “best guesses”) of an unknown scalar or vector-valued parameter θ based on a “random sample” of observations drawn from a distribution indexed by this parameter. The adjective “frequentist” is often used in referring to classical methods, largely because the theory of their performance is based on the premise that the experiment from which data is drawn can be replicated repeatedly and that estimators (and other statistical procedures) may be evaluated and compared on the basis of their expected performance over the intended number of replications or on a hypothesized infinite number of i.i.d. trials. Finite sample methods leading, for example, to 7least-squares, best-unbiased or best-invariant estimators, and estimators based on asymptotic theory, the premier example of which is the maximum likelihood estimators proposed by Fisher, are generally studied separately and are, together, the mainstays of the theory and practice of frequentist estimation. These are discussed in more detail below, as well as elsewhere in the Encyclopedia. Bayesian estimation theory tends to start at the same place outlined above. It begins with a model for the observable data, and assumes the existence of data upon which inference about a target parameter will be based. The important point of departure from classical inference is the position that uncertainty should be treated stochastically. From this, it follows that since the target parameter in a point estimation problem is unknown, one’s uncertainty about its value is appropriately represented by a “prior probability distribution” on that parameter. The Bayesian position is not simply a whim or a matter of convenience; it is in fact motivated and defended through a system of axioms about the comparison of possible uncertain events and the fact that the axiom system leads to this position as a derived result. See De Groot () for further details. The Bayesian paradigm for estimation can be by described as involving three steps: the specification of a
prior distribution (through introspection or the elicitation of expert opinion), the updating of that prior on the basis of available data, leading to the “posterior distribution” on the parameter, and the estimation of the parameter based on characteristics of the posterior distribution. The mean of the posterior distribution is probably the most commonly used Bayesian point estimator. Comparisons between frequentist and Bayesian estimators raise some challenging issues. There are important philosophical differences between the approaches that make it difficult to compare them side by side. For example, the Likelihood Principle (see, for example, Berger and Wolpert ()) stipulates that inference about an unknown parameter should depend of the experiment only through the observed data x = (x , x , . . . , xn ) or, equivalently, through the likelihood function, which for the present purposes, may be thought of as a constant multiple of the joint density L(θ∣x) = f (x∣θ), with x fixed and known. As a consequence, the process of averaging over the sample space, as the frequentist does, for example, in minimizing the variance among unbiased estimators of a parameter, is inappropriate from the Bayesian viewpoint. Only the observed data, rather than what might have been observed in the experiment but wasn’t, is relevant in Bayesian inference about θ. Maximum likelihood estimation, which obeys the likelihood principle in the sense that its calculation relies solely on the maximization of L(θ∣x), might thus seem consistent with the Bayesian approach, but it runs afoul of that paradigm on the basis of the facts that it fails to quantify the uncertainty about θ stochastically and its entire theoretical justification is based on the familiar averaging process over the sample space, albeit in a limiting sense. There are many texts, monographs and research papers which treat Bayesian and frequentist estimators and offer some discussion on how one might compare them. In this brief review, the discussion of this literature is clearly out of the question. We will limit ourselves to arguments and theory that may be found in the books by Lehmann and Casella (), Robert (), Cox () and in the paper by Samaniego and Reneau (). A more comprehensive treatment of the subject of the comparative analysis of Bayesian and frequentist point estimators can be found in the monograph by Samaniego (). The references cited in the sources above are wide in scope and will provide the interested reader with an enormous collection of collateral reading that is highly relevant to the subject of interest here. When one seeks to compare the Bayesian and frequentist approaches to point estimation, one might begin with a
Bayesian vs. Classical Point Estimation: A Comparative Overview
foundational issue, namely, which approach has defensible logical underpinnings. As it happens, this is a rather easy matter to resolve. The classical theory of estimation has no logical underpinnings to speak of. The theories of unbiased estimators, invariant estimators and maximum likelihood estimators (MLEs) are justified by frequentists on intuitive grounds. For example, “unbiasedness” and “invariance” are intuitively appealing ways of restricting the class of all possible estimators, a class which is “too large” and contains no “best estimator” in any nontrivial problem. Maximum likelihood estimators “work well” based on a sample of infinite size, so perhaps they will work well the given finite-sample problem at hand. The frequentist approach to estimation is unabashedly ad hoc, and there is widespread recognition that the particular frequentist tool one might wish to use in different problems might vary, with least squares (or its well-known variants) used in regression and ANOVA problems, UMVUEs used in finite-sample problems in which their derivation is feasible and MLEs used in estimation problems in which the sample size is thought to be suitably large. In contrast with these circumstances, the Bayesian approach to point estimation is based on a system of “plausible” axioms about how one should deal with uncertainty. The fact that the Bayesian approach to statistical inference is built upon a logical basis which one can formally study and scrutinize places it on a higher “logical” plane than frequentist inference. Most readers of the axioms of Bayesian inference will find them “reasonable”; for example, the transitivity of one’s assessments about which events are more likely to occur than others is typical of the assumptions one is expected to make. On logical grounds, the Bayesian approach appears to have a strong advantage. Is that enough to justify the claim that Bayesian inference is the preferred approach? The approach does provide a logically consistent process of prior assessment, updating and posterior assessment, leading to inference that can clearly be defended on logical grounds. A disturbing counterargument is that logic, by itself, is not enough. After all, it is possible to be perfectly logical and yet woefully wrong. Poor prior specification can lead to logically consistent estimation in which the Bayesian estimator is way off the mark. Other considerations must be brought to bear on these comparisons. Bayesian estimation (as well as other modes of Bayesian inference) has a fairly apparent Achilles heal, the fact that the Bayesian interjects subjective prior information into the inference process, potentially infusing errors or biases that would not otherwise be present. This is a legitimate concern, and one that the “orthodox” Bayesian (that is, one who uses a proper probability distribution for his prior) must always be mindful of and careful about. While the usual counterargument does diffuse the
B
criticism somewhat, a general fix does not exist. The argument that helps mitigate the criticism is that it is often the case that the Bayesian approach is essential in making sensible inferences. For example, a frequentist who observes ten heads in ten tosses of a newly minted coin has no choice but to estimate the probability of heads as p = , while a typical Bayesian, using a quite reasonable beta prior, might instead estimate p to be about .. In situations such as this, Bayesian methods may be seen as a reasonable way of averting disaster. There are many other grounds on which comparisons can be made. Regarding asymptotic performance for example, it is known that proper Bayes estimators based on priors that place positive mass on all open subsets of the parameter space enjoy the same asymptotic properties as the best asymptotically normal (BAN) estimators of frequentist estimation theory (most notably, MLEs). Multivariate analysis poses greater challenges to the Bayesian than to the frequentist, largely due to the difficulty of obtaining and quantifying useful prior information on high-dimensional parameters. The frequentists would seem to have a bit of an advantage in this area. Robert () employs decision theoretic arguments quite prominently in his defense of Bayesian methods. One of the centerpieces of that defense is the so-called complete class theorem which, in essence, says that any admissible decision rule (here, estimator) is Bayes (or nearly Bayes) with respect to some prior. This statement begs the question: why ever use anything else? To some, this argument seems to have obvious weaknesses, and the question is rather easy to answer. The fact that an estimator belongs to a class that contains all the “good” estimators hardly justifies its use. After all, the estimator that always estimates the scalar parameter θ to be is admissible but would (essentially) never be recommended for use. In addition to the inconclusive arguments above, there are examples on both sides that show that the other approach gives silly answers to particular statistical questions. Bayesians often use the term “incoherent” when referring to frequentist procedures of this sort. To the frequentist school, the most pressing concern is the very real possibility of obtaining poor answers from a Bayesian analysis due to poor prior input. Samaniego and Reneau () offer a quite different, performance-based comparison of Bayesian and frequentist estimators – the Bayes risk of an estimator relative to the “truth”, with the latter modeled as a (possibly degenerate) “true prior distribution” on the unknown parameter. The threshold problem (i.e., finding the boundary separating Bayes estimators that outperform the frequentist estimator of choice from Bayes estimators that don’t) is introduced. Explicit solutions to the threshold problem are
B
B
Behrens–Fisher Problem
obtained in the context of sampling distributions belonging to one-parameter exponential families, conjugate prior families and squared error loss. In Samaniego (), subsequent extensions of this work to the estimation of high-dimensional parameters and to estimation under asymmetric loss are treated. While the “true state” of the target parameter remains unknown throughout these analyses, it is seen that useful practical guidance can nevertheless be gleaned from them. In many problems, the class of Bayes estimators that are superior to frequentist alternatives is surprisingly broad. Bayesians who are both misguided (with a poorly centered prior distribution) and stubborn (with a tightly concentrated prior distribution) will generally not do well. Interestingly, it is shown that one flaw or the other need not be fatal by itself. But perhaps the most important conclusion of these studies is the simple fact that that neither the Bayesian nor the frequentist approach to point estimation will be uniformly dominant in any well-defined point estimation problem. In all such problems, there will be a threshold separating “good” priors from “bad” ones, and the remaining challenge, one that is by no means trivial, is trying to make a sensible judgment about which side of the threshold one is on, given the prior information one has in hand. This examination may lead one to a Bayes estimator or to a frequentist estimator in the particular problem of interest.
7Philosophical Foundations of Statistics 7Prior Bayes: Rubin’s View of Statistics 7Probability Theory: An Outline 7Probability, History of 7Significance Tests: A Critique 7Statistical Inference 7Statistical Inference: An Overview 7Statistics: An Overview
References and Further Reading Berger JO, Wolpert R () The likelihood principle. IMS Monograph Series, Hayward Cox DR () Principles of statistical inference. Cambridge University Press, Cambridge De Groot MH () Optimal statistical decisions. McGraw-Hill, New York Lehmann E, Casella G () Theory of point estimation, nd edn. Springer Verlag, New York Robert C () The Bayesian choice: a decision theoretic motivation, nd edn. Chapman and Hall, London Samaniego FJ, Reneau DM () Toward a reconciliation of the Bayesian and frequentist approaches to point estimation. J Am Stat Assoc :– Samaniego FJ () A comparison of the Bayesian and frequentist approaches to estimation. Springer, New York Stigler S () The history of statistics: the measurement of uncertainty before . Harvard University Press, Cambridge
About the Author Francisco Samaniego holds the title of Distinguished Professor of Statistics at the University of California, Davis. He has authored two recent Springer monographs, the first on “system signatures,” a notion that he introduced to the engineering literature in a IEEE paper, and the second on “comparative statistical inference” focusing on the relative performance of Bayesian and frequentist point estimators. He is a Fellow of the ASA, IMS, RSS and a member of the ISI. He is a former Editor of The Journal of the American Statistical Association. In , he received the U.S. Army Samuel S. Wilks Award for career contributions to reliability.
Cross References 7Bayesian Analysis or Evidence Based Statistics? 7Bayesian Statistics 7Bayesian Versus Frequentist Statistical Reasoning 7Estimation 7Estimation: An Overview 7Foundations of Probability 7Frequentist Hypothesis Testing: A Defense 7Likelihood 7Model Selection
Behrens–Fisher Problem Allan S. Cohen, Seock-Ho Kim Professors University of Georgia, Athens, GA, USA
Introduction The Behrens–Fisher problem is the problem in statistics of hypothesis testing and interval estimation regarding the difference between the means of two independent normal populations without assuming the variances are equal. The solution of this problem was first offered by Behrens () and reformulated later by Fisher () using t′ =
(¯x − x¯ ) − (µ − µ ) = t sin θ − t cos θ, √ s /n + s /n
where the sample mean x¯ and sample variance s are obtained from the random sample of size n from the normal distribution with mean µ and variance σ , t = (¯x − √ µ )/ s /n has a t distribution with ν = n − degrees of freedom, the respective quantities with subscript are √ √ defined similarly, and tan θ = (s / n )/(s / n ) or
Behrens–Fisher Problem
√ √ θ = tan− [(s / n )/(s / n )]. The distribution of t ′ is the Behrens–Fisher distribution. It is, hence, a mixture of the two t distributions. Under the usual null hypothesis of H : µ = µ , the test statistic t ′ , can be obtained and compared with the percentage points of the Behrens–Fisher distribution. Tables for the Behrens–Fisher distribution are available from Fisher and Yates () and Lindley and Scott (). The table entries are based on the four numbers: ν , ν , θ, and the Type I error rate α. For example, Fisher and Yates () presented significance points of the Behrens–Fisher distribution in two tables, one for ν = ν = , , , ∞, θ = ○ , ○ , ○ , ○ , ○ , ○ , ○ , and α = ., ., and the other for ν that is greater than ν = , , , , , , , θ = ○ , ○ , ○ , ○ , ○ , ○ , ○ , and α = ., ., ., .. Using the Behrens–Fisher distribution, the (−α)% interval that contains µ − µ can be constructed with √ ′ x¯ − x¯ ± tα/ (ν , ν , θ)
s /n + s /n ,
′ where the probability that t ′ > tα/ (ν , ν , θ) is α/ or, ′ ′ equivalently, Pr[t > tα/ (ν , ν , θ)] = α/. The Behrens–Fisher t ′ statistic and the Behrens–Fisher distribution are based on Fisher’s () fiducial approach. The approach is to find a fiducial probability distribution that is a probability distribution of a parameter from observed data. Consequently, the interval that involves ′ tα/ (ν , ν , θ) is referred to as the ( − α)% fiducial interval.
Example Driving times from a person’s house to work were measured for two different routes with n = and n = (see Lehmann , p. ). The ordered data from the first route are ., ., ., ., . yielding x¯ = . and s = ., and the data from the second route are ., ., ., ., ., ., ., ., ., ., . yielding x¯ = . and s = .. It is assumed that the two independent samples were drawn from two normal distributions having means µ and µ and variances σ and σ , respectively. A researcher wants to know whether the average driving times differed for the two routes. The test statistic under the null hypothesis of equal population means is t ′ = . with ν = , ν = , and θ = .. The result, Pr(t ′ > .) = ., indicates the null hypothesis cannot be rejected at α = . when the alternative hypothesis is non-directional, Ha : µ ≠ µ , because p = .. The corresponding % interval for the population mean difference is [−., .].
B
Other Solutions The Student’s t test (see 7Student’s t-Tests) for independent means can be used when the two population variances are assumed to be equal and σ = σ = σ , t=
(¯x − x¯ ) − (µ − µ ) , √ sp /n + sp /n
where the pooled variance that provides the estimate of the common population variance σ is defined as sp = [(n − )s + (n − )s ] /(n + n − ). It has a t distribution with ν = n + n − degrees of freedom. The example data yield the Student’s t = ., ν = , the two-tailed p = ., and the % confidence interval of [., .]. The null hypothesis of equal population means is rejected at the nominal α = ., and the confidence interval does not contain . When the two variances cannot be assumed to be the same, there are several alternative solutions in addition to use the Behrens–Fisher t ′ statistic. One simple way to solve this two means problem, called the smaller degrees of freedom t test, is to use the same t ′ statistic that has a t distribution with different degrees of freedom (e.g., Moore , p. ): t ′ ∼ t[min(ν , ν )], where the degrees of freedom is the smaller value of ν or ν . Note that this method should be used only if no statistical software is available because it yields a conservative test result and a wider confidence interval. The example data yield t ′ = ., ν = , the two-tailed p = ., and the % confidence interval of [−., .]. The null hypothesis of equal population means is not rejected at α = ., and the confidence interval contains . Welch ()’s approximate t test also uses the same t ′ statistic that has a t distribution with the approximate degrees of freedom ν′ (see Moore , p. ): t ′ ∼ t(ν ′ ), where ν ′ = /[c /ν + ( − c) /ν ] with c = (s /n ) / [(s /n ) + (s /n )]. It can be noted that the equivalent of this Welch’s approximate t test was proposed by Smith (). Moore () indicated that the approximation is accurate when both sample sizes are or larger. Although there are other solutions, Welch’s approximate t test currently seems to be the best practical solution to the Behrens–Fisher problem because of its availability from popular statistical software including SPSS and SAS. The example data yield t ′ = ., ν ′ = ., the two-tailed p = ., and the % confidence interval of [−., .]. The null hypothesis of equal population means is
B
B
Behrens–Fisher Problem
not rejected at α = ., and the confidence interval contains . In addition to the previous method, Welch (, ) and Aspin () presented an approximation of the distribution of t ′ by the method of moments (i.e., Welch-Aspin t test; see Kim and Cohen , for the detailed expansion terms for the approximation). The example data yield t ′ = . and the critical value under the Welch-Aspin t test for the two-tailed test is . at α = .. The corresponding % confidence interval is [−., .]. Again, the null hypothesis of equal population means is not rejected at α = ., and the confidence interval contains . The Bayesian solution to the Behrens–Fisher problem was offered by Jeffreys (). When uninformative uniform priors are used for the population parameters, the Bayesian solution to the Behrens–Fisher problem is identical to that of Fisher’s (). The Bayesian highest posterior density interval that contains the population mean difference with the probability of − α is identical to the ( − α)% fiducial interval. There are many solutions to the Behrens–Fisher problem based on the frequentist approach of the Neyman and Pearson () sampling theory. Among the methods, Kim and Cohen () indicated that Welch (, ), Aspin (), and Tsui and Weerahandi () are the most important ones from the frequentist perspective. The critical values and the confidence intervals from various methods under the frequentist approach are in general different from either fiducial or Bayesian approach. For the one-sided alternative hypothesis, however, it is interesting to note that the generalized extreme region to obtain the generalized p by Tsui and Weerahandi () is identical to the extreme area from the Behrens–Fisher t ′ statistic (see also Weerahandi , pp. –). For the example data, the smaller degrees of freedom t test yielded the most conservative result with the largest critical value and the widest confidence interval. The Student’s t test yielded the smallest critical value and the shortest confidence interval. All other intervals lie between these two intervals. Robinson () pointed out that the differences between many solutions to the Behrens–Fisher problem might be less than their differences from the Student’s t test when sample sizes are greater than . The popular statistical software programs SPSS and SAS produce results from the Welch’s approximate t test and the Student’s t test as well as the respective confidence intervals. It is essential to have a table that contains the percentage points of the Behrens–Fisher distribution or computer programs that can calculate the tail areas and percentage values in order to use the Behrens–Fisher t test or to obtain the fiducial interval. Note that the Welch’s
approximate t test may not be as effective as the WelchAspin t test (Wang ). Note also that the sequential testing of the population means based on the result from either Levene’s test of the equal population variances from SPSS or the folded F test from SAS is not recommended in general due to the complicated nature of control of the Type I error in the sequential testing.
About the Authors Dr. Allan S. Cohen is Professor of Research, Evaluation, Measurement, and Statistics Program, Department of Educational Psychology and Instructional Technology, and Director, the Georgia Center for Assessment, both at the University of Georgia. He has authored or co-authored over journal articles, conference papers and technical reports as well as several tests, test manuals, and computer programs. He is an editorial board member of Applied Psychological Measurement and the Journal of Educational Measurement and is a Member of the American Educational Research Association and the National Council on Measurement in Education. Dr. Seock-Ho Kim is Professor of Research, Evaluation, Measurement, and Statistics Program, Department of Educational Psychology and Instructional Technology, University of Georgia. He has co-authored over papers and several computer programs. He has been the Editorial board member of Educational and Psychological Measurement, Journal of Educational Measurement, School Psychology Quarterly, and Measurement in Physical Education and Exercise Science. He has a co-authored book with Frank B. Baker entitled Item Response Theory: Parameter Estimation Techniques (nd edn, Dekker, ). He is a Member of the American Statistical Association, Institute of Mathematical Statistics, American Educational Research Association, American Psychological Association, National Council on Measurement in Education, and Psychometric Society. Professor Seock-Ho Kim has served as Vice-President and President of the Korean-American Educational Researchers Association.
Cross References 7Fiducial Inference 7Heteroscedasticity 7Permutation Tests 7Statistical Software: An Overview 7Student’s t-Tests
References and Further Reading Aspin AA () An examination and further development of a formula arising in the problem of comparing two mean values. Biometrika :–
Best Linear Unbiased Estimation in Linear Models
Behrens WU () Ein Beitrag zur Fehlerberechnung bei wenigen Beobachtungen (A contribution to error estimation with few observations). Landwirtschaftliche Jahrbücher :– Fisher RA () The fiducial argument in statistical inference. Ann Eugenic :– Fisher RA () The comparison of samples with possibly unequal variances. Ann Eugenic :– Fisher RA, Yates F () Statistical tables for biological, agricultural and medical research, th edn. Oliver and Boyd, Edinburgh, England Jeffreys H () Note on the Behrens–Fisher formula. Ann Eugenic :– Kim S-H, Cohen AS () On the Behrens–Fisher problem: a review. J Educ Behav Stat :– Lehmann EL () Nonparametrics: statistical methods based on ranks. Holden-Day, San Francisco Lindley DV, Scott WF () New Cambridge elementary statistical tables, nd edn. Cambridge University Press, Cambridge, England Moore DS () The basic practice of statistics, th edn. W.H. Freeman, New York Neyman J, Pearson ES () On the use and interpretation of certain test criteria for purposes of statistical inference. Biometrika A(–):– Robinson GK () Behrens–Fisher problem. In: Kotz S, Johnson NL, Read CB (eds) Encyclopedia of statistical sciences, vol . Wiley, New York pp – Smith HF () The problem of comparing the results of two experiments with unequal errors. J Coun Sci Ind Res :– Tsui K-H, Weerahandi S () Generalized p-values in significance testing of hypotheses in the presence of nuisance parameters. J Am Stat Assoc :–; Correction : Wang YY () Probability of the type I error of the Welch tests for the Behrens-Fisher problem. J Am Stat Assoc :– Weerahandi S () Exact statistical methods for data analysis. Springer, New York Welch BL () The significance of the difference between two means when the population variances are unequal. Biometrika :– Welch BL () The generalization of ‘Student’s’ problem when several different population variances are involved. Biometrika :–
Best Linear Unbiased Estimation in Linear Models Simo Puntanen , George P. H. Styan University of Tampere, Tampere, Finland McGill University, Montréal, QC, Canada
In this article we consider the general linear model y = Xβ + ε,
where X is a known n × p model matrix, the vector y is an observable n-dimensional random vector, β is a p× vector of unknown parameters, and ε is an unobservable vector of random errors with expectation E(ε) = , and covariance matrix cov(ε) = σ V, where σ > is an unknown constant. The nonnegative definite (possibly singular) matrix V is known. In our considerations σ has no role and hence we may put σ = . As regards the notation, we will use the symbols A′ , − A , A+ , C (A), C (A) , and N (A) to denote, respectively, the transpose, a generalized inverse, the Moore–Penrose inverse, the column space, the orthogonal complement of the column space, and the null space, of the matrix A. By (A : B) we denote the partitioned matrix with A and B as submatrices. By A we denote any matrix satisfying C (A ) = N (A′ ) = C (A) . Furthermore, we will write PA = AA+ = A(A′ A)− A′ to denote the orthogonal projector (with respect to the standard inner product) onto C (A). In particular, we denote H = PX and M = In − H. One choice for X is of course the projector M. Let K′ β be a given vector of parametric functions specified by K′ ∈ IRq×p . Our object is to find a (homogeneous) linear estimator Ay which would provide an unbiased and in some sense “best” estimator for K′ β under the model M . However, not all parametric functions have linear unbiased estimators; those which have are called estimable parametric functions, and then there exists a matrix A such that E(Ay) = AXβ = K′ β for all β ∈ IRp . Hence K′ β is estimable if and only if there exists a matrix A such that K′ = AX, i.e., C (K) ⊂ C (X′ ). The ordinary 7least squares estimator of K′ β is defined ˆ where βˆ is any solution to the noras OLSE(K′ β) = K′ β, ′ ˆ mal equation X X β = X′ y; hence β = βˆ minimizes (y − Xβ)′ (y − Xβ) and it can be expressed as βˆ = (X′ X)− X′ y, while X βˆ = Hy. Now the condition C (K) ⊂ C (X′ ) guarantees that K′ βˆ is unique, even though βˆ may not be unique.
The Best Linear Unbiased Estimator (BLUE) of Xβ The expectation Xβ is trivially estimable and Gy is unbiased for Xβ whenever GX = X. An unbiased linear estimator Gy for Xβ is defined to be the best linear unbiased estimator, BLUE, for Xβ under M if cov(Gy) ≤L cov(Ly)
Introduction or in short M = {y, Xβ, σ V},
B
for all L∶ LX = X,
where “≤L ” refers to the Löwner partial ordering. In other words, Gy has the smallest covariance matrix (in the Löwner sense) among all linear unbiased estimators. We
B
B
Best Linear Unbiased Estimation in Linear Models
˜ If X has full denote the BLUE of Xβ as BLUE(Xβ) = X β. column rank, then β is estimable and an unbiased estimator Ay is the BLUE for β if AVA′ ≤L BVB′ for all B such that BX = Ip . The Löwner ordering is a very strong ordering implying for example var(β˜i ) ≤ var (β∗i ) , i = , . . . , p, ˜ ≤ tracecov(β ∗ ), det cov( β) ˜ ≤ det cov(β ∗ ), tracecov( β) for any linear unbiased estimator β ∗ of β; here “det” denotes determinant. The following theorem gives the “Fundamental BLUE equation”; see, e.g., Rao (), Zyskind () and Puntanen et al. (). Theorem Consider the general linear model M = {y, Xβ, V}. Then the estimator Gy is the BLUE for Xβ if and only if G satisfies the equation G(X : VX ) = (X : ).
()
The corresponding condition for Ay to be the BLUE of an estimable parametric function K′ β is A(X : VX ) = (K′ : ). It is sometimes convenient to express () in the following form, see Rao (). Theorem (Pandora’s Box) Consider the general linear model M = {y, Xβ, V}. Then the estimator Gy is the BLUE for Xβ if and only if there exists a matrix L ∈ IRp×n so that G is a solution to ⎛V ⎜ ⎜ ′ ⎝X
X⎞ ⎛G′ ⎞ ⎛ ⎞ ⎟⎜ ⎟ = ⎜ ⎟. ⎟⎜ ⎟ ⎜ ⎟ ⎠ ⎝ L ⎠ ⎝ X′ ⎠
The equation () has a unique solution for G if and only if C (X : V) = IRn . Notice that under M we assume that the observed value of y belongs to the subspace C (X : V) with probability ; this is the consistency condition of the linear model, see, e.g., Baksalary et al. (). The consistency condition means, for example, that whenever we have some statements which involve the random vector y, these statements need hold only for those values of y that belong to C (X : V). The general solution for G can be expressed, for example, in the following ways: G = X(X′ W− X)− X′ W− + F (I n − WW− ), G = H − HVM(MVM)− M + F [In − MVM(MVM)− ]M, where F and F are arbitrary matrices, W = V + XUX′ and U is any arbitrary conformable matrix such that
C (W) = C (X : V). Notice that even though G may not be unique, the numerical value of Gy is unique because y ∈ C (X : V). If V is positive definite, then BLUE(Xβ) = X(X′ V− X)− X′ V− y. Clearly OLSE(Xβ) = Hy is the BLUE under {y, Xβ, σ I}. It is also worth noting that the matrix G satisfying () can be interpreted as a projector: it is a projector onto C (X) along C (VX ), see Rao ().
OLSE vs. BLUE Characterizing the equality of the Ordinary Least Squares Estimator (OLSE) and the BLUE has received a lot of attention in the literature, but the major breakthroughs were made by Rao () and Zyskind (); for a detailed review, see Puntanen and Styan (). We present below six characterizations for the OLSE and the BLUE to be equal (with probability ). Theorem (OLSE vs. BLUE) Consider the general linear model M = {y, Xβ, V}. Then OLSE(Xβ) = BLUE(Xβ) if and only if any one of the following six equivalent conditions holds. (Note: V may be replaced by its Moore–Penrose inverse V+ and H and M = I − H may be interchanged.) HV = VH, HVM = , C (VH) ⊂ C (H), C (X) has a basis comprising orthonormal eigenvectors of V, . V = HAH + MBM for some A and B, . V = αIn + HKH + MLM for some α ∈ IR, and K and L.
. . . .
Two Linear Models Consider now two linear models M = {y, Xβ, V } and M = {y, Xβ, V }, which differ only in their covariance matrices. For the proof of the following proposition and related discussion, see, e.g., Rao (, Theorem ., Theorem .), and Mitra and Moore (, Theorem ., Theorems .–.). Theorem Consider the linear models M = {y, Xβ, V } and M = {y, Xβ, V }, and let the notation { BLUE(Xβ ∣ M ) } ⊂ { BLUE(Xβ ∣ M ) } mean that every representation of the BLUE for Xβ under M remains the BLUE for Xβ under M . Then the following statements are equivalent: . . . .
{BLUE(Xβ ∣ M ) } ⊂ { BLUE(Xβ ∣ M ) }, C (V X ) ⊂ C (V X ),
V = V + XN X′ + V MN MV , for some N and N , V = XN X′ + V MN MV , for some N and N .
Best Linear Unbiased Estimation in Linear Models
Notice that obviously { BLUE(Xβ ∣ M ) } = { BLUE(Xβ ∣ M ) } ⇐⇒ C (V X ) = C (V X ).
For the equality between the BLUEs of X β under two partitioned models, see Haslett and Puntanen (a).
B
In terms of Pandora’s Box (Theorem ), Ay is the BLUP for yf if and only if there exists a matrix L such that A satisfies the equation ⎛ V X⎞ ⎛A′ ⎞ ⎛V ⎞ ⎟ ⎜ ⎟ = ⎜ ⎟ . ⎜ ⎟⎜ ⎟ ⎜ ′ ⎟ ⎜ ′ ⎝X ⎠ ⎝ L ⎠ ⎝ X f ⎠
The Mixed Model
Model with New Observations: Best Linear Unbiased Predictor (BLUP)
A mixed linear model can be presented as
Consider the model M = {y, Xβ, V}, and let yf denote an m × unobservable random vector containing new observations. The new observations are assumed to follow the linear model yf = Xf β + ε f , where Xf is a known m × p model matrix associated with new observations, β is the same vector of unknown parameters as in M , and ε f is an m × random error vector associated with new observations. Our goal is to predict the random vector yf on the basis of y. The expectation and the covariance matrix are
where X ∈ IRn×p and Z ∈ IRn×q are known matrices, β ∈ IRp is a vector of unknown fixed effects, γ is an unobservable vector (q elements) of random effects with cov(γ, ε) = q×p and
⎛ y ⎞ ⎛ Xβ ⎞ ⎟, ⎟ ⎜ E⎜ ⎜ ⎟=⎜ ⎟ ⎝y f ⎠ ⎝X f β ⎠
⎛ y ⎞ ⎛V = V ⎟ ⎜ cov ⎜ ⎜ ⎟=⎜ ⎝yf ⎠ ⎝ V
V ⎞ ⎟, ⎟ V ⎠
which we may write as ⎧ ⎪ ⎛ ⎞ ⎪ ⎪ ⎪ y⎟ M f = ⎨⎜ ⎜ ⎟, ⎪ ⎪ ⎪ y ⎪ ⎩⎝ f ⎠
⎛ Xβ ⎞ ⎜ ⎟, ⎜ ⎟ ⎝X f β ⎠
⎛V ⎜ ⎜ ⎝V
⎫ ⎪ ⎪ V ⎞⎪ ⎟⎪ . ⎟⎬ ⎪ ⎪ V ⎠⎪ ⎪ ⎭
A linear predictor Ay is said to be unbiased for yf if E(Ay) = E(yf ) = Xf β for all β ∈ IRp . Then the random vector yf is said to be unbiasedly predictable. Now an unbiased linear predictor Ay is the best linear unbiased predictor, BLUP, if the Löwner ordering cov(Ay − yf ) ≤L cov(By − yf ) holds for all B such that By is an unbiased linear predictor for yf . The following theorem characterizes the BLUP; see, e.g., Christensen (, p ), and Isotalo and Puntanen (, p ). Theorem (Fundamental BLUP equation) Consider the linear model Mf , where Xf β is a given estimable parametric function. Then the linear estimator Ay is the best linear unbiased predictor (BLUP) for yf if and only if A satisfies the equation A(X : VX ) = (Xf : V X ).
y = Xβ + Zγ + ε,
or in short
Mmix = {y, Xβ + Zγ, D, R},
E(γ) = q , cov(γ) = Dq×q , E(ε) = n , cov(ε) = Rn×n . This leads directly to Theorem Consider the mixed model Mmix = {y, Xβ + Zγ, D, R}. Then the linear estimator By is the BLUE for Xβ if and only if B(X : ΣX ) = (X : ), where Σ = ZDZ′ + R. Moreover, Ay is the BLUP for γ if and only if A(X : ΣX ) = ( : DZ′ X ). In terms of Pandora’s Box (Theorem ), Ay = BLUP(γ) if and only if there exists a matrix L such that A satisfies the equation ⎛ Σ X⎞ ⎛A′ ⎞ ⎛ZD⎞ ⎟⎜ ⎟ = ⎜ ⎟. ⎜ ⎟⎜ ⎟ ⎜ ⎟ ⎜ ′ X ⎠⎝ L ⎠ ⎝ ⎠ ⎝ For the equality between the BLUPs under two mixed models, see Haslett and Puntanen (b, c).
Acknowledgment The research of the second author was supported in part by the Natural Sciences and Engineering Research Council of Canada.
About the Authors Simo Puntanen is Adjunct Professor in the Department of Mathematics and Statistics, University of Tampere, Tampere, Finland. Currently he is the Book Review Editor of the International Statistical Review, and a member of the Editorial Board of the Annals of the Institute of Statistical Mathematics, Communications in Statistics, Statistical Papers, and Mathematical Inequalities & Applications. He is a founding member (with George P. H. Styan)
B
B
Beta Distribution
of the International Workshop on Matrices and Statistics, held regularly from onwards. He is a co-author (with George P. H. Styan and Jarkko Isotalo) of the book Matrix Tricks for Linear Statistical Models: Our Personal Top Twenty (Springer, ). George P. H. Styan is Professor Emeritus of Mathematics and Statistics at McGill University in Montréal (Québec), Canada, and is currently the Abstracting Editor of Current Index to Statistics. In , Professor Styan was named an Honorary Member of the Statistical Society of Canada “for his deep research at the interface of Matrix Theory and Statistics; for his remarkable editorial work within Canada and beyond, his mentoring of graduate and postdoctoral students; and for innumerable other scholarly and professional contributions to the international statistical community.”
Rao CR () Least squares theory using an estimated dispersion matrix and its application to measurement of signals. In: Le Cam LM, Neyman J (eds) Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability: Berkeley, California, /, vol . University of California Press, Berkeley, pp – Rao CR () Unified theory of linear estimation. Sankhy¯a, Series A :– (Corrigenda (), : and ) Rao CR () Projectors, generalized inverses and the BLUE’s. J R Stat Soc Series B :– Haslett SJ, Puntanen S (b) Equality of BLUEs or BLUPs under two linear models using stochastic restrictions. Statistical Papers : – Haslett SJ, Puntanen S (c) On the equality of the BLUPs under two linear mixed models. Metrika, aailable online, DOI ./s---. Zyskind G () On canonical forms, non-negative covariance matrices and best and simple least squares linear estimators in linear models. Ann Math Stat :–
Cross References 7Autocorrelation in Regression 7Gauss-Markov Theorem 7General Linear Models 7Least Squares 7Linear Regression Models 7Ordered Statistical Data: Recent Developments 7Small Area Estimation 7Statistical Design of Experiments (DOE) 7Unbiased Estimators and Their Applications
References and Further Reading Baksalary JK, Rao CR, Markiewicz A () A study of the influence of the ‘natural restrictions’ on estimation problems in the singular Gauss–Markov model, J Stat Plann Infer :– Christensen R () Plane answers to complex questions: the theory of linear models, rd edn. Springer, New York Haslett SJ, Puntanen S (a) Effect of adding regressors on the equality of the BLUEs under two linear models. J Stat Plann Infer :– Haslett SJ, Puntanen S (b) Equality of BLUEs or BLUPs under two linear models using stochastic restrictions. Statistical Papers :– Haslett SJ, Puntanen S (c) On the equality of the BLUPs under two linear mixed models. Metrika, DOI ./S–– – Isotalo J, Puntanen S () Linear prediction sufficiency for new observations in the general Gauss–Markov model. Commun Stat-Theor : – Mitra SK, Moore BJ () Gauss–Markov estimation with an incorrect dispersion matrix. Sankhy¯a Series A :– Puntanen S, Styan GPH () The equality of the ordinary least squares estimator and the best linear unbiased estimator (with comments by Oscar Kempthorne and by Shayle R. Searle and with “Reply” by the authors). Am Stat :– Puntanen S, Styan GPH, Werner HJ () Two matrix-based proofs that the linear estimator Gy is the best linear unbiased estimator. J Stat Plann Infer :–
Beta Distribution Arjun K. Gupta Distinguished University Professor Bowling Green State University, Bowling Green, OH, USA
A random variable X is said to have the beta distribution with parameters a and b if its probability density function is fX (x) =
xa− ( − x)b− , B(a, b)
< x < ,
a > , b > ()
where
B(a, b) = ∫
ua− ( − u)b− du
denotes the beta function. The beta family, whose origin can be traced to . in a letter from Sir Isaac Newton to Henry Oldenberg, has been utilized extensively in statistical theory and practice. Originally defined on the unit interval, many generalizations of () have been proposed in the literature; see Karian and Dudewicz () for a four parameter generalization defined over a finite interval, McDonald and Richards (a, b) for a generalization obtained by power transformation of X; Libby and Novick () and Armero and Bayarri () for generalizations obtained by dividing () by certain algebraic functions; Gordy () for a generalization obtained by multiplying () by an exponential function; and, Nadarajah and Kotz () for a generalization obtained by multiplying () by a Gauss hypergeometric function. For further details, the reader is referred to
Bias Analysis
Chap. in Johnson et al. () and Gupta and Nadarajah (). Some properties of beta distribution () are listed here. . rth moment about zero µ ′r
B(a + r, b) = B(a, b) Γ(a + r)Γ(a + b) . = Γ(a)Γ(a + b + r)
In particular, E(X) = Var(X) =
a , a+b
ab . (a + b + )(a + b)
. Characteristic function E[eitX ] = F (a; a + b; it) where F is the confluent hypergeometric function defined by ∞ (α)k z k . F (α; β; z) = ∑ k= (β)k k!
B
7Statistical Distributions: An Overview 7Uniform Distribution in Statistics
References and Further Reading Armero C, Bayarri MJ () Prior assessments for prediction in queues. The Statistician :– Gordy MB () Computationally convenient distributional assumptions for commonvalue auctions. Comput Econ : – Gupta AK, Nadarajah S () Handbook of beta distribution and its applications. Marcel Dekker, New York Johnson NL, Kotz S, Balakrishnan N () Continuous univariate distributions, vol , nd edn. Wiley, New York Karian ZA, Dudewicz EJ () Fitting statistical distributions: the generalized lambda distribution and generalized bootstrap methods. CRC, Boca Raton, Florida Libby DL, Novick MR () Multivariate generalized betadistributions with applications to utility assessment. J Educ Stat :– McDonald JB, Richards DO (a) Model selection: some generalized distributions. Commun Stat A:– McDonald JB, Richards DO (b) Some generalized models with application to reliability. J Stat Plann Inf :– Nadarajah S, Kotz S () A generalized beta distribution II. InterStat
. The random variable X −X has the Pearson type VI distribution defined by Y=
fY (y) =
ya− , y > . B(a, b) ( + y)a+b
About the Author Professor Gupta is Distinguished University Professor, Department of Mathematics and Statistics Bowling Green State University. He was Chair of the Department (–). He is Elected Fellow of the Ohio Academy of Science (), ASA, ISI and Royal Statistical Society. He was awarded the All India Mathematical Society Golden Jubilee Award (). Professor Gupta was editor and associate editor of many international journals. Currently, he is Associate editor for Communications in Statistics. He has written or co-authored more than papers and books, including Handbook of Beta Distribution and Its Applications (with Saralees Nadarajah, eds., CRC, ).
Cross References 7Approximations to Distributions 7Bayesian Statistics 7F Distribution 7Location-Scale Distributions 7Multivariate Statistical Distributions 7Relationships Among Univariate Statistical Distributions
Bias Analysis Sander Greenland Professor University of California-Los Angeles, Los Angeles, CA, USA Methodological shortcomings of studies can lead to bias, in the sense of systematic (nonrandom) distortion of estimates from the studies. Well-recognized shortcomings (bias sources) include the following: Units may be selected for observation in a nonrandom fashion; stratifying on additional unmeasured covariates U may be essential for the X-Y association to approximate a target causal effect; inappropriate covariates may be entered into the analysis; and components of X or Y or Z may not be adequately measured. In methodologic modeling or bias analysis, one models these shortcomings. In effect, one attempts to model the design and execution of the study, including features (such as selection biases and measurement errors) beyond investigator control. The process is thus a natural extension to imperfect experiments and observational studies of the design-based paradigm in experimental and survey statistics.
B
B
Bias Analysis
Bias analysis well established in engineering and policy research and are covered in many books, albeit in a wide variety of forms and specialized applications. Little and Rubin () focus on missing-data problems; Eddy et al. () focus on medical and health-risk assessment; and Vose () covers general risk assessment. Models for specific biases have a long if scattered history in epidemiology, going back to Berkson () and Cornfield et al. (); Greenland and Lash () give a review. Nonetheless, methods for statistical inference from bias models have only recently begun to appear in observational health research (Robins et al. ; Graham ; Lash and Fink ; Lash et al. ; Phillips ; Greenland a, , a, b; Gustafson , a, b; Fox et al. ).
Statistical Formulation Many of the parameters in realistic bias models will not be identifiable (estimable) from the data, necessitating inferential approaches well beyond those of conventional statistics. The simplest approach is to fix those parameters at specific values, estimate effects assuming those values are correct, and see how effect estimates change as those values are varied. This process is an example of 7sensitivity analysis. One can also assign the parameters prior probability distributions based on background information, and summarize target estimates over these distributions or over the resulting posterior distribution. Consider the problem of estimating the effect of X on Y, given a collection of antecedent covariates Z, as common in causal inference (see Causation and Causal Inference). Standard approaches estimate the regression of Y on X and Z, E(Y∣x,z), and then taking the fitted (partial) regression of Y on X given Z as the effect of X on Y. Usually a parametric model r(x,z;β) for E(Y∣x,z) is fit and the coefficient for X is taken as the effect (this approach is reflected in common terminology that refers to such coefficients as “main effects”); the logistic model for a binary Y is the most common epidemiologic example. Model fitting is almost always done as if . Within levels of X and Z, the data are a 7simple random sample and any missingness is completely random. . The causal effect of X on Y is accurately reflected by the association of X and Y given Z (i.e., there is no residual confounding – as might be reasonable to assume if X were randomized within levels of Z). . X, Y, and Z are measured without error. In reality, () sampling and missing-data probabilities may jointly depend on X, Y, and Z in an unknown fashion, ()
stratifying or adjusting for certain unmeasured (and possibly unknown) covariates U might be essential for the association of X and Y to correspond to a causal effect of X on Y, and () some of the X, Y and Z components are mismeasured.
Selection Biases Let V = (X,Y,Z). One approach to sampling (selection) biases posits a model s(v;σ) for the probability of selection given v, then uses this model in the analysis along with r(x,z;β), e.g., by incorporating s(v;σ) into the likelihood function (Eddy et al. ; Little and Rubin ; Gelman et al. ; Greenland b) or by using s(v; σ)− as a weighting factor (Robins et al. , ; Copas and Li ; Scharfstein et al. ). The parameters β and σ are usually cannot be completely estimated from the data under analysis, so one must either posit various fixed values for σ and estimate β for each chosen σ (sensitivity analysis), or else give β σ a prior distribution and conduct a Bayesian analysis. A third, somewhat informal approach between sensitivity and Bayesian analysis is Monte-Carlo risk analysis or Monte-Carlo sensitivity analysis (MCSA). This approach repeatedly samples σ from its prior distribution, resamples (bootstraps) the data, and re-estimates β using the sampled σ and data; it then outputs the distribution of results obtained from this repeated samplingestimation cycle. MCSA can closely approximate Bayesian results under certain (though not all) conditions (Greenland , ), most notably when β and σ are a priori independent and there is negligible prior information about β. These selection-modeling methods can be generalized (with many technical considerations) to handle arbitrary missing data (Little and Rubin ; Robins et al. , ).
Confounding Suppose U is a collection of unmeasured (latent) covariates required for identification of the effect of X on Y. One approach to problem () is to model the distribution of U and V with a probability model p(u,v;β, γ) = p(y∣u,x,z;β)p(u,x,z;γ). Again, one can estimate β by likelihood-based or by weighting methods, but because U is unmeasured, the parameter (β, γ) will not be fully estimable from the data and so some sort of sensitivity analysis or prior distribution will be needed (e.g., Cornfield et al. ; Yanagawa ; Robins et al. ; Rosenbaum ; Greenland a, b, , b). Results will depend heavily on the prior specification given U. For example, U may be a specific unmeasured covariate (e.g., smoking status) with well studied relations to X, Y, and Z,
Bias Analysis
which affords straightforward Bayesian and MCSA analyzes (Steenland and Greenland ). On the other hand, U may represent an unspecified aggregation of latent confounders, in which case the priors and hence inferences are more uncertain (Greenland a).
Measurement Error and Misclassification Suppose that the collection of “true” values V = (X,Y,Z) has a corresponding collection of measurements or surrogates W (which might include multiple surrogates for X, Y, or Z). The measurement-error problem (problem ) can then be expressed as follows: For some or all units, at least one of the V components is missing, but the measurements in W that correspond to the missing V components are present. If enough units are observed with both V and W complete, the problem can be handled by standard missing-data methods. For example, given a model for the distribution of V and W one can use likelihood-based methods (Little and Rubin ), or impute V components where absent and then fit the model r(x,z;β) for E(Y∣x,z) to the completed data (Cole et al. ), or fit the model to the complete records using weights derived from all records using a model for missing-data patterns (Robins et al. ). Direct Bayesian approaches to measurement error are also available (Gustafson ; Greenland a, b). Alternatively, there are many measurement-error correction procedures that adjust the “naïve” β estimates obtained by fitting the regression using W as if it were V. This adjustment is usually accomplished with a model relating V to W fitted to the complete records, as in instrumental-variable (regression calibration) corrections and their extensions (Carroll et al. ). Some recent methods are based on assuming various subsamples with information on multiple surrogates are available (so W may be of much higher dimension than V and may have complex missing-data patterns) (e.g., Spiegelman et al. ). All methods assume that missingness in V and W components is random, which is often quite implausible because noncooperation increases with demands on subjects, collection of some components may be demanding (e.g., as when W includes diet diaries or biomarkers), and cooperation may be related to unobserved true values or confounders. Thus selection modeling will be needed along with measurement modeling to account for this nonrandom (“nonignorable”) missingness. Further nonidentified modeling becomes a necessity if a component of V is never observed on any unit (or, more practically, if there are too few complete records to support large-sample missing-data or measurement-error
B
procedures). Latent-variable methods are natural for this situation (Berkane ). For example, one could model the distribution of (V,W) or a sufficient factor from that distribution by a parametric model; the unobserved components of V are then the latent variables in the model. As before, the parameters will not be fully identified, making Bayesian methods a natural choice for summary inferences (Gustafson , a, b; Greenland , a, b). Realistic specification for nonidentified measurement error models can become quite complex, with inferences displaying extreme sensitivity to parameter constraints or prior distributions. Nonetheless, methodologic modeling helps provide an honest accounting for the large uncertainty that can be generated by even modest measurement error.
Acknowledgments Some of the above material is adapted from Greenland ().
About the Author For biography see the entry 7Confounding and Confounder Control.
Cross References 7Bias Correction 7Confounding and Confounder Control 7Identifiability 7Measurement Error Models 7Sensitivity Analysis
References and Further Reading Berkane M (ed) () Latent variable modeling and applications to causality. Lecture Notes in Statistics (), Springer, New York Berkson J () Limitations of the application of fourfold table analysis to hospital data. Biometrics Bull :– Brumback BA, Hernán MA, Haneuse S, Robins JM () Sensitivity analyses for unmeasured confounding assuming a marginal structural model for repeated measures. Stat Med : – Carroll RJ, Ruppert D, Stefanski LA, Crainicenau CM () Measurement error in nonlinear models: a modern perspective, nd edn. Chapman and Hall/CRC Press, Boca Raton, FL Cole SR, Chu H, Greenland S () A simulation study of multipleimputation for measurement error correction. Int J Epidemiol :– Copas JB, Li HG () Inference for non-random samples. J Roy Stat Soc B :– Cornfield J, Haenszel W, Hammond WC, Lilienfeld AM, Shimkin MB, Wynder EL () Smoking and lung cancer: recent evidence and a discussion of some questions. J Natl Cancer Inst :–
B
B
Bias Correction
Eddy DM, Hasselblad V, Schachter R () Meta-analysis by the confidence profile method. Academic, New York Fox MP, Lash TL, Greenland S () A method to automate probabilistic sensitivity analyses of misclassified binary variables. Int J Epidemiol :– Gelman A, Carlin JB, Stern HS, Rubin DB () Bayesian data analysis, nd edn. Chapman and Hall/CRC, New York Goetghebeur E, van Houwelingen HC (eds) () Analyzing noncompliance in clinical trials. Stat Med :– Graham P () Bayesian inference for a generalized population attributable fraction. Stat Med :– Greenland S () Sensitivity analysis, Monte-Carlo risk analysis, and Bayesian uncertainty assessment. Risk Anal :– Greenland S (a) The impact of prior distributions for uncontrolled confounding and response bias. J Am Stat Assoc :– Greenland S (b) Quantifying biases in causal models: classical confounding versus collider-stratification bias. Epidemiology :– Greenland S () An overview of methods for causal inference from observational studies. Ch. . In: Gelman A, Meng XL (eds) Applied Bayesian modeling and causal inference from an incomplete-data perspective. New York, Wiley, pp – Greenland S () Multiple-bias modeling for observational studies (with discussion). J Roy Stat Soc A :– Greenland S (a) Bayesian perspectives for epidemiologic research. III. Bias analysis via missing-data methods. Int J Epidemiol ():– Greenland S (b) Relaxation penalties and priors for plausible modeling of nonidentified bias sources. Stat Sci (): – Greenland S, Lash TL () Bias analysis. Ch. . In: Rothman KJ, Greenland S, Lash TL (eds) Modern epidemiology, rd edn. Philadelphia, Lippincott, pp – Gustafson P () Measurement error and misclassification in statistics and epidemiology. Chapman and Hall, New York Gustafson P (a) On model expansion, model contraction, identifiability, and prior information: two illustrative scenarios involving mismeasured variables (with discussion). Stat Sci :– Gustafson P (b) The utility of prior information and stratification for parameter estimation with two screening tests but no gold standard. Stat Med :– Gustafson P, Le ND, Saskin R () Case-control analysis with partial knowledge of exposure misclassification probabilities. Biometrics :– Joseph L, Gyorkos T, Coupal L () Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. Am J Epidemiol :– Lash TL, Fink AK () Semi-automated sensitivity analysis to assess systematic errors in observational epidemiologic data. Epidemiology :– Lash TL, Fox MP, Fink AK () Applying quantitative bias analysis to epidemiologic data. Springer, New York Little RJA, Rubin DB () Statistical analysis with missing data, nd edn. Wiley, New York Phillips CV () Quantifying and reporting uncertainty from systematic errors. Epidemiology :– Robins JM () Marginal structural models versus structural nested models as tools for causal inference. In: Halloran ME,
Berry DA (eds) Statistical models in epidemiology. Springer, New York, pp – Robins JM, Rotnitzky A, Zhao LP () Estimation of regression coefficients when some regressors are not always observed. J Am Stat Assoc :– Robins JM, Rotnitzky A, Scharfstein DO () Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In: Halloran ME, Berry DA (eds) Statistical models in epidemiology. Springer, New York, pp – Rosenbaum P () Observational studies, nd edn. Springer, New York Scharfstein DO, Rotnitsky A, Robins JM () Adjusting for nonignorable drop-out using semiparametric nonresponse models. J Am Stat Assoc :– Spiegelman D, Zhao B, Kim J () Correlated errors in biased surrogates: study designs and methods for measurement error correction. Stat Med :– Steenland K, Greenland S () Monte-Carlo sensitivity analysis and Bayesian analysis of smoking as an unmeasured confounder in a study of silica and lung cancer. Am J Epidemiol : – Vose D () Risk analysis. Wiley, New York Yanagawa T () Case-control studies: assessing the effect of a confounding factor. Biometrika :–
Bias Correction Gauss M. Cordeiro Professor Universidade Federal Rural de Pernambuco, Recife, Brazil
Introduction A central object in asymptotic likelihood theory is the calculation of the second-order biases of the maximum likelihood estimates (MLEs). To improve the accuracy of these estimates, substantial effort has gone into computing the cumulants of log-likelihood derivatives which are, however, notoriously cumbersome. The MLEs typically have biases of order O(n− ) for large sample size n, which are commonly ignored in practice, the justification being that they are small when compared to the standard errors of the parameter estimates that are of order O(n−/ ). For small samples sizes, however, these biases can be appreciable and of the same magnitude as the corresponding standard errors. In such cases, the biases cannot be neglected, and for turning feasible estimation of their size in practical applications, corresponding formulae for their calculation need to be established for a wide range of probability distributions and regression models.
Bias Correction
Bias correction has been extensively studied in the statistical literature and there has been considerable interest in finding simple matrix expressions for second-order biases of MLEs in some classes of regression models which do not involve cumulants of log-likelihood derivatives. The methodology has been applied to several regression models. We focus on the following models: normal nonlinear models (Cook et al. ), generalized log-gamma regression model (Young and Bakir ), 7generalized linear models (Cordeiro and McCullagh ), ARMA models (Cordeiro and Klein ), multivariate nonlinear regression models (Cordeiro and Vasconcellos ), generalized linear models with dispersion covariates (Botter and Cordeiro ), symmetric nonlinear regression models (Cordeiro et al. ), Student t regression model with unknown degrees of freedom (Vasconcellos and Silva ), beta regression models (Ospina et al. ) and a class of multivariate normal model where the mean vector and the covariance matrix have parameters in common (Patriota and Lemonte ). In general two parameter continuous distributions, Stósic and Cordeiro () showed how to symbolically compute the biases of the MLEs bypassing the traditional computation of joint cumulants of log-likelihood derivatives. The bias approximation may be used to produce a biascorrected estimator by subtracting the bias approximation from the MLE. Alternatively, an examination of the form of the bias may suggest a re-parametrization of the model that results is less biased estimates.
General Formula Consider that the total log-likelihood function ℓ(θ), based on n observations not necessarily independent and identically distributed, is a function of a p× vector θ of unknown parameters. We assume that ℓ = ℓ(θ) is regular (Cox and Hinkley ) with respect to all θ derivatives up to and including those of third order. We introduce the following log-likelihood derivatives in which we reserve lower-case subscripts r, s, t, . . . to denote components of the vector θ: Ur = ∂l/∂θ r , Urs = ∂ l/∂θ r ∂θ s , and so on. The standard notation is adopted for the cumulants of log-likelihood derivatives κ rs = E(Urs ), κ r,s = E(Ur Us ), κ rs,t = E(Urs Ut ), etc, where all κ’s refer to a total over the sample and are, in general, of order n. The elements of the information matrix K − are κ r,s = −κ rs and let κ r,s = −κ rs denote the corresponding elements of the inverse information matrix K − , which is of order O(n− ). The MLE θ̂ of θ can be obtained as a solution of a ̂r = for r = , . . . , p. system of nonlinear equations U A general formula for the O(n− ) bias of θ̂ for a regular statistical model with p unknown parameters was given
B
by Cox and Snell () and Cordeiro and McCullagh (). Assuming standard regularity conditions (Cox and ̂r = to obtain Ur + Hinkley ), we can expand U ̂ ∑s Urs (θ s − θ s ) + Op () = and write in matrix notation U = J(θ̂ − θ) + Op (), where U is the score vector and J is the observed information matrix. Since J = K + Op (n/ ) we have U = K(θ̂ − θ) + Op () and θ̂ − θ = K − U + Op (n− ).
()
Equation is important to provide higher-order moments ̂ If we now expand U ̂r up and cumulants of the estimate θ. to terms of second-order, we obtain Ur + ∑ Urs (θˆs −θ s )+ ∑ Urst (θˆs −θ s )(θˆt −θ t )+op () = s,t s and then by calculating its expected value ∑ κ rs E(θˆs − θ s ) + ∑ Cov(Urs , θˆs − θ s ) s
s
+ ∑ κ rst (−κ st ) + o() = . s,t
()
Up to terms of order O(n− ), we can write Cov(Urs , θ̂s − θ s ) = Cov (Urs , − ∑t κ st Ut ) = − ∑t κ rs,t κ st . Let B(θ̂a ) be the O(n− ) bias of the estimate θ̂a for a = , . . . , p. Inverting Eq. , we can write B(θ̂a ) as (Cox and Snell ) B(θ̂a ) = ∑ κ ar κ st (κ rs,t + κ rst ) r,s,t (t) = ∑ κ ar κ st (κ rs − κ rst ) . r,s,t
()
We can verify that the two alternative forms for B(θ̂a ) are equivalent using Bartlett identity. In general regression models, we can derive matrix expressions for the bias of the ̂ say B(θ), ̂ from Eq. when the cumulants κ’s are estimate θ, invariant under permutations of parameters (see Cordeiro and McCullagh ). ̂ to define bias The estimate θ̂ can be inserted in B(θ) ̂ where ̂ ̂ is the value corrected estimate θ̃ = θ̂ − ̂ B(θ), B(θ) ̂ at θ. ̂ The bias corrected estimate θ̃ is expected to of B(θ) have better sampling properties than the classical estimate ̂ In fact, several simulations presented in the literature θ. (Botter and Cordeiro ; Cordeiro et al. ; Vasconcellos and Silva ; Ospina et al. ; Patriota and Lemonte ) have shown that the corrected estimates θ̃ have smaller biases than their corresponding uncorrected estimates, thus suggesting that these bias corrections have the effect of shrinking the corrected estimates toward to the true parameter values. However, we can not say that the bias corrected estimates offer always some improvement
B
B
Bias Correction
over the MLEs, since they can have larger mean squared errors than the uncorrected estimates. We give a simple example by taking n iid observations from a normal distribution N(µ, σ ), where we are interested to calculate the n− biases of the MLEs of µ and σ. The elements of the information matrix are: κ µ,µ = n/σ , κ µ,σ = and κ σ ,σ = n/σ . The third cumulantes are easily obtained: κ µ µ µ = κ µ,µ µ = κ σ ,µ µ = κ σ ,µσ = κ µ,σ σ = κ µσ σ = , κ µ µσ = −κ µ,µσ = n/σ , κ σ ,σ σ = −n/σ and κ σ σ σ = n/σ . Thus, B(̂ µ) = since ̂ µ = Σyi /n has no bias. Further, after some algebra, B(σˆ ) = −σ/n. This result is in agreement with the exact bias of ̂ σ = {Σ(yi − y) /n}/ √ Γ( n− ) given by E(σˆ ) = σ, which is obtained from the n Γ( n )
χ n− distribution of (n − )σˆ /σ . In fact, using Stirling + O(n− )}. expansion in E(σˆ ) yields E(σˆ ) = σ { − n ) σˆ . The corrected estimate of σ is then σ˜ = ( + n − For a one-parameter model, the n bias of θ̂ follows from Eq. by setting all parameters equal to θ. We obtain a formula first derived by Bartlett ()
to be identical stressing the extremely high level of confidence that may be attributed to analytic manipulations involved. For both Maple and Mathematica, after specifying the form and the domain of the density function f (y; µ, ϕ) as well as the assumptions to be made on µ and ϕ (e.g. µ ∈ IR or µ > ), the program first defines and analytically calculates the cumulants (κ’s), the second-order cumulants are then subsequently inserted into intermediate expression for the information matrix, in order to find the inverse information matrix, which are then used together with the third-order cumulants to produce the final result from Eq. . From now on, we use the notation ψ ′ (p) and ψ ′′ (p) for the derivatives of the digamma function ψ(p) = d log{Γ(p)}/dp, γ = − ψ() for the Euler’s constant and ζ(p) for the Riemann Zeta function. The examples below are obtained using these programs and agree with previously reported results in the literature: . Normal distribution with mean µ and variance ϕ
ˆ = κ θ θ (κ θ θ ,θ + κ θ θ θ ) = κ θ θ (κ (θ) − κ θ θ θ ) . B(θ) θθ
Stósic and Cordeiro () presented simple programs (scripts) that may be used with algebraic manipulation software Maple and Mathematica to calculate closed-form analytic expressions for the bias corrections B µ and Bϕ of the MLEs of the parameters µ and ϕ for arbitrary two-parameter continuous distributions through a straightforward application of Eq. . While these symbolic computation software packages have currently the ability to deal with analytic expressions of formidable size and complexity, limitations still exist, and it turns out that the complexity of the formulae involved in calculating the cumulants of log-likelihood derivatives for some distributions exceed their capacity. It is precisely for this reason that they presented equivalent scripts for calculating the bias correction terms in both frameworks. For some very rare cases neither Maple nor Mathematica were able to produce closed form expressions, but even for these, the current scripts may be expected to produce results on future versions of the softwares (under the assumption that backward compatibility of the scripting language is maintained). It should be pointed out that the above fact does not diminish the usefulness of either of the software packages (in their current versions), as both have produced closed form expressions in a large majority of the tested continuous density functions. Moreover, in all cases where both packages came up with a closed form expression, the results were found
nB µ = , nBϕ = −
ϕ . n
. Inverse Gaussian distribution with mean µ and precision parameter ϕ nB µ = , nBϕ =
ϕ . n
. Gamma distribution with mean µ and shape parameter ϕ B µ = , Bϕ = −
− ϕψ ′ (ϕ) + ϕ ψ ′′ (ϕ) . n (ϕψ ′ (ϕ) − )
. Weibull distribution with scale µ and shape ϕ (here E(y) = µΓ( + /ϕ)) Bµ =
µ {π ( − ϕ) + π ϕ n π [ + γ + ϕ − γ ( + ϕ)] + (γ−) ϕ ζ()} ,
Bϕ =
ϕ (π − ζ()) π n
.
. Logistic distribution with mean µ and variance π ϕ / B µ = , Bϕ = −
ϕ ( π + ) . n (π + )
Bias Correction
B
. Extreme value distribution with mean µ + γ ϕ and variance π ϕ / Bµ =
ϕ ( (− + γ) π + π − (− + γ) ζ())
π − ϕ (π − ζ()) . Bϕ = π
,
µ (π − ζ()) , π n ϕ Bϕ = { (− + γ + γ ) π + π + π log(ϕ) π n + log(ϕ) [( + γ) π − ζ()]
Bµ =
. Beta distribution with parameters µ and ϕ Bµ =
[ψ ′ (µ) (ψ ′ (ϕ) − ψ ′ (µ + ϕ)) − ψ ′ (ϕ) ψ ′ (µ + ϕ)] n × {− [ψ ′ (µ + ϕ) (ψ ′ (µ + ϕ) (ψ ′′ (µ) − ψ ′′ (ϕ))
− (− + γ) ζ()} .
+ψ ′ (µ) ψ ′′ (ϕ))]
. Fisher-Tippett distribution with mode µ and variance π ϕ /
+ ψ ′ (ϕ) [−ψ ′′ (µ) + ψ ′′ (µ + ϕ)] +ψ ′ (ϕ) [ ψ ′ (µ + ϕ) ψ ′′ (µ) + ψ ′ (µ) ψ ′′ (µ + ϕ)]} ,
× {ψ ′ (µ + ϕ) [ψ ′′ (µ) − ψ ′′ (ϕ)] + ψ ′ (µ) ψ ′ (µ + ϕ) ψ ′′ (ϕ) +ψ (µ) [−ψ ′′ (ϕ) + ψ ′′ (µ + ϕ)] ′
+ψ ′ (ϕ) [− (ψ ′ (µ + ϕ) ψ ′′ (µ)) + ψ ′ (µ) ψ ′′ (µ + ϕ)]} .
. Student t distribution with location parameter µ and dispersion parameter ϕ − (− + ν + ν ) ϕ . n ν ( + ν)
When ν tends to infinity, we obtain Bϕ = −ϕ/(n) as is well known in the normal case. . Generalized Rayleigh distribution with mean Γ(ϕ+ ) √ µ Γ(ϕ+)
Bµ =
Bϕ =
and variance
µ
[ + ϕ −
Γ(ϕ+ ) Γ(ϕ+)
]
µ [− ψ ′ ( + ϕ) + ( + ϕ) ψ ′ ( + ϕ) − ( + ϕ) ψ ′′ ( + ϕ)]
[− + ( + ϕ) ψ ′ ( + ϕ)] − + ( + ϕ) ψ ′ ( + ϕ) − ( + ϕ) ψ ′′ ( + ϕ) n [− + ( + ϕ) ψ ′ ( + ϕ)]
variance
,
.
. Type I Gumbel distribution with mean
γ+log(ϕ) µ
and
π µ
µ (π − ζ()) , π n ϕ Bϕ = { (− + γ + γ ) π + π + π log(ϕ) π n + log(ϕ) [( + γ) π − ζ()]
Bµ =
− (− + γ) ζ()} .
ϕ ( (− + γ) π + π − (− + γ) ζ()) , π n − ϕ (π − ζ()) Bϕ = . π n For these twelve distributions tested, Maple fails to yield closed form analytic expressions for bias corrections B µ and Bϕ for the extreme value, Student t, type I Gumbel and Fisher-Tippett distributions, whereas Mathematica fails only for the 7logistic distribution. Comparing the equations obtained with Maple and Mathematica with some results previously reported in the literature, we note that, in most cases, all of the terms fully agree with the previously reported results, and where discrepancies were observed, it was found that the current results are correct, and errors were identified in the previous publications. This fact builds confidence regarding the correctness of the presented scripts, and the ability of these software for analytic formulae manipulation, so that application of the scripts to other density functions may be expected to produce reliable closed form expressions. It is worth emphasizing that there are other methods to obtain bias corrected estimates. In regular parametric problems, Firth () developed the so-called “preventive” method, which also allows for the removal of the second-order bias. His method consists of modifying the original score function to remove the first-order term from the asymptotic bias of these estimates. In exponential families with canonical parameterizations, his correction scheme consists in penalizing the likelihood by the Jeffreys invariant priors. This is a preventive approach to bias adjustment which has its merits, but the connections between our results and his work are not pursued here. We should also stress that it is possible to avoid cumbersome and tedious algebra on cumulant calculations by using Efron’s bootstrap; see Efron and Tibshirani (). We use Bµ =
Bϕ = [ψ ′ (µ) (ψ ′ (ϕ) − ψ ′ (µ + ϕ)) − ψ ′ (ϕ) ψ ′ (µ + ϕ)] n
B µ = , Bϕ =
. Type II Gumbel distribution with mean ϕ µ Γ ( − µ ) and variance ⎡ ⎤ ⎢ ⎥ ϕ µ ⎢Γ ( − ) − Γ ( − ) ⎥ ⎢ µ µ ⎥ ⎣ ⎦
B
B
Binomial Distribution
the analytical approach here since this leads to a nice formula. Moreover, the application of the analytical bias approximation seems to generally be the most feasible procedure to use and it continues to receive attention in the literature.
Vasconcellos KLP, Silva SG () Corrected estimates for student t regression models with unknown degrees of freedom. J Stat Comput Sim :– Young DH, Bakir ST () Bias correction for a generalized loggamma regression model. Technometrics :–
About the Author For biography see the entry 7Bartlett and Bartlett-Type Corrections.
Cross References 7Bartlett and Bartlett-Type Corrections 7Bias Analysis 7Estimation: An Overview 7Jackknife 7Likelihood 7Target Estimation: A New Approach to Parametric
Estimation
References and Further Reading Bartlett MS () Confidence intervals II. Biometrika : – Botter DA, Cordeiro GM () Improved estimators for generalized linear models with dispersion covariates. J Stat Comput Sim :– Cook DR, Tsai CL, Wei BC () Bias in nonlinear regression. Biometrika :– Cordeiro GM, Klein R () Bias correction in ARMA Models. Stat Probab Lett :– Cordeiro GM, McCullagh P () Bias correction in generalized linear models. J R Stat Soc B :– Cordeiro GM, Vasconcellos KLP () Bias correction for a class of multivariate nonlinear regression models. Stat Probab Lett :– Cordeiro GM, Ferrari SLP, Uribe-Opazo MA, Vasconcellos KLP () Corrected maximum-likelihood estimation in a class of symmetric nonlinear regression models. Stat Probab Lett :– Cox DR, Hinkley DV () Theoretical statistics. Chapman and Hall, London Cox DR, Snell EJ () A general definition of residuals (with discussion). J R Stat Soc B :– Efron B, Tibshirani RJ () An introduction to the bootstrap. Chapman and Hall, London Firth D () Bias reduction of maximum likelihood estimates. Biometrika :– Ospina R, Cribari–Neto F, Vasconcellos KLP () Improved point and interval estimation for a beta regression model. Comput Stat Data Anal :– Patriota AG, Lemonte AJ () Bias correction in a multivariate regression model with genereal parameterization. Stat Probab Lett :– Stósic B, Cordeiro GM () Using Maple and Mathematica to derive bias corrections for two parameter distributions. J Stat Comput Sim :–
Binomial Distribution Andreas N. Philippou , Demetrios L. Antzoulakos Professor of Probability and Statistics University of Patras, Patras, Greece Associate Professor University of Piraeus, Piraeus, Greece The binomial distribution is one of the most important distributions in Probability and Statistics and serves as a model for several real life problems. Special cases of it were first derived by Pascal () and Bernoulli (). Definition and genesis. Denote by X the number of successes in a sequence of n (≥ ) independent trials of an experiment, and assume that each trial results in a success (S) or a failure (F) with respective probabilities p ( < p < ) and q = −p. The random variable (rv) X is said to have the binomial distribution with parameters n and p, and it is denoted by B(n, p). The probability mass function (pmf) f (x) of X is given by n f (x) = P(X = x) = ( )px qn−x , x
x = , , , . . . , n,
()
where (nx) = n!/x!(n − x)! for ≤ x ≤ n and otherwise. In fact a typical element of the event {X = x} is a sequence SSFS . . . SF of x S’s and n − x F’s, having probability px qn−x because of P(S) = p and the independence of the trials. Since there are (nx) such distinct allocations of x S’s and n − x F’s, the result follows. The name of the distribution is due to the binomial theorem, which implies that B(n, p) is a proper probability distribution, since f (x) ≥ for x ∈ R and n n n x n−x = (q + p)n = . ∑ f (x) = ∑ ( )p q x= x= x
()
The cumulative distribution function (cdf) of X is related with the incomplete beta function by the formula F(x) = P(X ≤ x) = ∑xi= (ni)pi qn−i q
= (n − x)(nx) ∫ t n−x− ( − t)x dt, x = , , , . . . , n. ()
Binomial Distribution
Its 7moment generating function (mgf) is n n M(t) = E(etX ) = ∑ etx ( )px qn−x = (q + pet )n x x=
()
from which the mean and the variance follow as µ = E(X) = M ′ () = np,
σ = Var(X) = E(X ) − µ
= M ′′ () − (M ′ ()) = npq. () When n = , the binomial distribution B(n, p) is known as the Bernoulli distribution. By substitution, we get from () the recurrence relation (n − x + )p f (x) = f (x − ), xq
x = , , . . . , n,
()
which along with the initial condition f () = qn is very useful for calculating binomial probabilities. It follows from () that if (n + )p is not an integer, then f (x) has a unique mode at x = [(n + )p]. If (n + )p is an integer, then f (x) has two modes, one at x = (n + )p and one at x = (n + )p − . The binomial distribution arises whenever a sample of size n is drawn randomly with replacement from a population containing just two types of elements. Urn models. From an urn containing w white and b black balls, n balls are drawn randomly with replacement. Let X be the number of white balls drawn. Then b n−x n w x ) ( ) , f (x) = P(X = x) = ( ) ( w+b x w+b
x = , , , . . . , n.
g(x) = P(Y = x) =
b ) (wx)(n−x
(w+b ) n
,
x = , , , . . . , n.
()
If, in addition, w/(w + b) → p ( < p < ), as w → ∞ and b → ∞, it follows from () that n lim g(x) = ( )px qn−x , x
w,b→∞
x = , , , . . . , n.
The last equation practically means that g(x) is approximately equal (≃) to the RHS of () for large w and b in comparison to n. The approximation is considered adequate if w + b > n. The binomial distribution converges to the following distribution named after Poisson. Poisson convergence (Poisson ). Let Xn be a sequence of rv’s distributed as B(n, pn ) and assume that as n → ∞, pn → and npn → λ (> ). Then λx n lim ( )pxn ( − pn )n−x = e−λ , n→∞ x x!
()
x = , , , . . . . ()
In practice, () means (np)x n ( )px qn−x ≃ e−np x! x
()
for large n and small p. The approximation is quite accurate if n ≥ and p ≤ . or if n ≥ and p ≤ .. The binomial distribution can be approximated by the normal distribution. Normal approximation. Let X be a rv distributed as B(n, p). Since X can be viewed as a sum of n independent Bernoulli rv’s, a direct application of the Central Limit Theorem yields the following approximation for the cdf of X x − np F(x) = P(X ≤ x) ≃ Φ ( √ ) () npq where Φ(⋅) denotes the cdf of the standard normal distribution. If a 7continuity correction is used we have the following improved approximation
()
In fact considering as a success S the drawing of a white ball, and noting that that the n balls are drawn randomly with replacement, it follows that X is the number of successes in a sequence of n independent trials of an experiment with success probability p = w/(w + b). The result then follows from (). Assume now that the balls are drawn randomly without replacement and denote by Y the number of white balls drawn. The rv Y follows the hypergeometric distribution with pmf
B
F(x) = P(X ≤ x) ≃ Φ (
x + . − np ) √ npq
()
The approximation is fairly good provided n and p are such that npq > . The approximation of binomial probabilities by the normal distribution was proved by de Moivre () for p = / and for arbitrary values of p by Laplace (). We end this note by the following example. Example. The manager of Alpha Airlines knows from past data that % of the passengers who buy Alpha tickets do not show up for travel. Based on this, Alpha Airlines sell tickets when the available seats in their planes are only . Find the probability that each ticket holder showing up will find a seat available and hence there will be no complaints. Solution. Denote by X the number of ticket holders who do not show up for travel. Then the required probability is P(X ≥ ) = − P(X ≤ ). Assuming that each of the ticket holders has the same probability
B
B
Bioinformatics
p = . of not showing up, independently of the others, we have that X is distributed as B(, .). Therefore, the exact probability of no complaints is P(X ≥ ) = x −x −∑ =.. The Poisson approxx= ( x )(.) (.) − x imation by () gives P(X ≥ )≃ − ∑ /x! = x= e .. The normal approximation by () is P(X ≥ ) ≃ √ − Φ ((−)/ .) =Φ(.)=.. Finally, the normal approximation with continuity correction by () is √ P(X≥)≃ − Φ ((+.−)/ .)=Φ(.)=..
About the Author For biography see the entry 7Distributions of order K.
Cross References 7Bayesian Analysis or Evidence Based Statistics? 7Bayesian Statistics 7Distributions of Order K 7Generalized Linear Models 7Geometric and Negative Binomial Distributions 7Inverse Sampling 7Modeling Count Data 7Multivariate Statistical Distributions 7Proportions, Inferences, and Comparisons 7Relationships Among Univariate Statistical Distributions 7Statistical Distributions: An Overview 7Statistical Methods in Epidemiology 7Univariate Discrete Distributions: An Overview
References and Further Reading Bernoulli J () Ars conjectandi. Thurnisius, Basilea de Moivre A () The doctrine of chances, nd edn. Woodfall, London Laplace PS () Théorie Analytique des Probabilités, rd edn. , Courcier Imprimeur, Paris. Reprinted by EJ Gabay, , Paris Pascal B () Varia opera Mathematica D. Petri de Fermat, Tolossae Poisson SD () Récherches sur la probabilité des jugements en matiere criminelle et en matiere civile, precedees des regles generales du calcul des probabilites, Paris: Bachelier, ImprimeurLibraire pour les Mathematiques, la Physique, etc.
Bioinformatics Susan R. Wilson Professor, Faculty of Medicine, Faculty of Science University of New South Wales, Sydney, NSW, Australia Bioinformatics is a relatively young, cross-disciplinary research area at the intersection of the biological sciences with the mathematical, statistical, and physical sciences
and chemistry and information technology. In the past decade or so, there has been phenomenal growth of life science databases. For example, the most widely used nucleotide sequence database is Genbank that is maintained by the National Center for Biotechnology Information (NCBI) of the US National Library of Medicine; as of February it contained approximately billion nucleotides from million sequences. Its size continues to grow exponentially as more genomes are being sequenced. However, there is a very large gap (that will take a long time to fill) between our knowledge of the functioning of the genome and the generation (and storing) of raw genomic data. This overview touches briefly on those aspects of bioinformatics that will be of interest to statisticians. The stated goal for many researchers is for developments in Bioinformatics to be focused at finding the fundamental laws that govern biological systems, as in physics. However, if such laws exist, they are a long way from being determined for biological systems. Instead the current aim is to find insightful ways to model limited components of biological systems and to create tools which biologists can use to analyze data. Examples include tools for statistical assessment of the similarity between two or more DNA sequences or protein sequences, for finding genes in genomic DNA, for quantitative analysis of functional genomics data, and for estimating differences in how genes are expressed in say different tissues, for analysis and comparison of genomes from different species, for phylogenetic analysis, and for DNA sequence analysis and assembly. Tools such as these involve statistical modeling of biological systems. Although the most reliable way to determine a biological molecule’s structure or function is by direct experimentation, there is much that can be achieved in vitro, i.e., by obtaining the DNA sequence of the gene corresponding to an RNA or protein and analyzing it, rather than the more laborious finding of its structure or function by direct experimentation. Much biological data arise from mechanisms that have a substantial probabilistic component, the most significant being the many random processes inherent in biological evolution, and also from randomness in the sampling process used to collect the data. Another source of variability or randomness is introduced by the biotechnological procedures and experiments used to generate the data. So the basic goal is to distinguish the biological “signal” from the “noise”. Today, as experimental techniques are being developed for studying genome wide patterns, such as expression arrays, the need to appropriately deal with the inherent variability has been multiplied astronomically. For example, we have progressed from studying one or
Bioinformatics
a few genes in comparative isolation to being able to evaluate simultaneously thousands of genes. Not only must methodologies be developed which scale up to handle the enormous data sets generated in the post-genomic era, they need to become more sensitive to the underlying biological knowledge and understanding of the mechanisms that generate the data. For statisticians, research has reached an exciting and challenging stage at the interface of computational statistics and biology. The need for novel approaches to handle the new genome-wide data has coincided with a period of dramatic change in approaches to statistical methods and thinking. This “quantum” change has been brought about, or even has been driven by, the potential of ever more increasing computing power. What was thought to be intractable in the past is now feasible, and so new methodologies need to be developed and applied. Unfortunately too many of the current practices in the biological sciences rely on methods developed when computational resources were very limiting and are often either (a) simple extensions of methods for working with one or a few outcome measures, and do not work well when there are thousands of outcome measures, or (b) ad-hoc methods (that are commonly referred to as “statistical” or “computational”, or more recently “data mining”! methods (see 7Data Mining)) that make many assumptions for which there is often no (biological) justification. The challenge now is to creatively combine the power of the computer with relevant biological and stochastic process knowledge to derive novel approaches and models, using minimal assumptions, and which can be applied at genomic wide scales. Such techniques comprise the foundation of bioinformatic methods in the future. Hidden Markov model (HMMs) is a tool that is popular in bioinformatics. Here, applications of HMMs are related to prediction of protein-coding regions in genome sequences, modeling families of related DNA or protein sequences or prediction of secondary structure elements from protein primary sequences. Biological data are the result of evolution, that is, the result of an incredibly complex, and currently unknown, stochastic process. Very simplified models of this process are often used, particularly for the construction of phylogenetic trees. Such evolutionary models have been developed as both discrete time and continuous time processes. A key requirement of these processes is that they be reversible, since statistical comparisons of, say, two contemporary species, can require one tracing up the tree of evolution to a common ancestor and then down the tree to the other species, and so the stochastic process for tracing must be the same in each direction.
B
Of recent years, gene expression data have become of increasing interest to statistical scientists. The technology has evolved too. Such data are being generated using technologies like microarrays, and very recently, next-generation sequencing. These data can be thought of as lying in very high dimensional space, and the resultant challenges are at the forefront of modern statistical science. In the past, biologists looked at genes and proteins one at a time. Now we have the technology to start to look at all the elements – genes, mRNA, proteins, interactions, and so on - in a biological system, and to explore their relationships as the system functions in response to biological, and environmental, perturbations. This is known as systems biology. The long-term goal is to have (mathematical) models to describe the behavior of the system given any kind of perturbation, and then to be able to redesign systems by modifications, such as drugs, to have completely new properties. The key question is whether it is possible to take a systematic approach to mapping pathways. The preliminary step involves building a comprehensive “scaffold” of molecular interactions that broadly covers many aspects of cellular function and physiological responses. At the broad, overall level, statistical data mining approaches are used to search for patterns and relationships between variables, and Bayesian networks model conditional dependencies. At a more detailed level, Markov chains model predictions, loss and interconversion among molecular species and states. Today, systems biology is very much in its infancy, but with a challenging future that will also involve expansion to include many more levels of complexity. An excellent starting point reference for statisticians is Ewens and Grant (). Bioconductor (http://www. bioconductor.org) is an open source and open development software project for the analysis of genomic data.
About the Author For biography see the entry 7Biostatistics.
Cross References 7Biostatistics 7Forensic DNA: Statistics in 7Statistical Genetics
References and Further Reading Ewens WJ, Grant G () Statistical methods in bioinformatics, nd edn. Springer, New York
B
B
Biopharmaceutical Research, Statistics in
Biopharmaceutical Research, Statistics in Christy Chuang-Stein Vice President of the American Statistical Association (–), Head Pfizer Inc., Kalamazoo, MI, USA
“Statistics in Biopharmaceutical Research” is the title of the on-line journal launched by the American Statistical Association in . There are at least two other international peer-reviewed journals completely dedicated to the use of statistics to support the development of pharmaceutical products. They are Journal of Biopharmaceutical Statistics (Taylor and Francis Group) and Pharmaceutical Statistics (John Wiley and Sons). There are many books devoted to this area of statistical applications also, e.g., Senn (), Dmitrienko et al. () and Dmitrienko et al. (). In the United States, pharmaceutical and biotech industries employ thousands of statisticians, either directly or indirectly. These statisticians support the discovery, development and commercialization of valuable medicines, medicines that have made substantial contributions to a longer life expectancy and a better quality of life in the past years. The development of pharmaceutical products is a long and high risk proposition. It takes an average of years for a new compound to be discovered and eventually submitted to regulators for approval. As of , the cost of developing a new drug is estimated to be between $ million and $ billion US dollars (Masia ). Biopharmaceutical research begins in the laboratory where chemists synthesize compounds and biologists screen the compounds for activities. Because of the large number of compounds, it is essential to develop an efficient algorithm-based process to conduct high-throughput screening for the maximum yield of promising compounds. Once a compound is judged to meet the level of required potency, it needs to go through formulation development so that the active ingredient could be delivered to the target site of actions in test subjects. The first test subjects are laboratory animals used to evaluate the effect of the compound on cardiovascular function, reproductive function, tumor development and the general wellbeing of offspring born to animals exposed to the compound. Most of the animal experiments are conducted according to the International Conference on Harmonisation (ICH) guidance M(R) () on nonclinical safety studies. The need to use the smallest number of animals at this preclinical testing stage has led to the use of efficient experimental designs with repeated measures
on each animal. In addition, data mining techniques (see 7Data Mining) are used widely to search for chemical and physical properties that tend to associate with compounds that turn out to be successful. The majority of statistical support in biopharmaceutical research takes place during the clinical testing in humans. In the United States, this support has grown substantially since the Kefauver-Harris Amendment (Krantz ) that required drug sponsors to prove a product’s safety and efficacy in controlled clinical trials before receiving marketing authorization. It is usually thought that the first properly randomized control trial in the twentieth century that was recorded involved the use of streptomycin for the treatment of pulmonary tuberculosis (MRC, ). Since that time, the number of clinical trials (both randomized and non-randomized) has skyrocketed as evidenced by the number of trials registered at the www.clinicaltrials.gov site in the United States. The clinical development of a new treatment is often divided into three phases. All clinical trials need to follow ICH E guidance on good clinical practice () and the Declaration of Helsinki. In Phase trials, healthy volunteers are randomized to receive a single dose or multiple doses of the compound or a placebo to study the tolerance and pharmacokinetics of the new compound. The objective is to decide an acceptable dose range. For cytotoxic agents, Phase trials are conducted in cancer patients with the objective to estimate the maximum tolerated dose. Because of the small number of subjects (e.g., –) at this stage, safety evaluation focuses on identifying common side effects of the new treatment. If the tolerance and pharmacokinetic profiles based on the limited data are judged to be acceptable, testing will proceed to Phase . In Phase , treatment using the new compound will be compared against a concurrent comparator (placebo, an approved product or the standard of care) in patients with the target disease. The primary objective is to gather safety and efficacy data in patients. Different dose strengths are typically used in these trials to help estimate the dose– response relationship. The latter often involves fitting an Emax model or logistic model. For oncology trials, Phase studies can be a single arm study using the maximum tolerated dose estimated from Phase . Some researchers further divide Phase into Phase a proof-of-concept and Phase b dose-ranging studies. The former often includes a single dose strength to verify the hypothesized mechanism while the latter uses different dose strengths and clinically relevant endpoints. Phase and Phase trials are designed to help a sponsor learn about the new treatment (Sheiner ). They are exploratory in nature and the analysis will focus on estimation instead of hypothesis testing.
Biopharmaceutical Research, Statistics in
This is a critical phase of product development. It is during this stage that important information on dose(s), dosing schedule(s), endpoints and the target population will be evaluated and decided upon. Statistics is heavily used to design trials, to analyze the results and to make Go or No-Go decisions. If data from the Phase development offer good reasons to believe that the new treatment has a positive benefit to risk balance and can bring value to patients, development will move into the confirmatory stage, or Phase . In general, Phase trials are double-blind randomized trials if blinding is at all possible. These trials are typically large (hundreds to thousands of patients) with a longer duration. The primary objective is to confirm the presence of a treatment effect and to collect additional safety information in a more diverse population. In terms of efficacy assessment, it could be either superiority over a comparator or non-inferiority to an active control (ICH E ). For life-threatening conditions, interim analyses are often conducted to stop a trial early for efficacy (positive outcome) or futility (negative outcome) for ethical reasons. Interim analyses are typically conducted by individuals independent of the study and reviewed by an Independent Data Monitoring Committee (FDA DMC guidance ). Except for those pre-specified in the protocol as possible mid-trial adaptations, changes are strongly discouraged at this stage. When multiple comparisons (multiple doses, multiple endpoints, multiple subgroups, interim efficacy analysis etc.) are conducted for inferential purposes, the significance levels for comparisons need to be properly adjusted so that the family-wide Type I error rate is strongly controlled. The adjustment method needs to be pre-specified and can’t be changed once the trial results become known. In short, the design and analysis of Phase trials need to be carefully planned and rigorously executed. Statistical principles, as articulated in ICH E (), should be followed with very few deviations. Deviations, when occurring, need to be justified and sensitivity analyses should be conducted to evaluate the impact of the deviations on conclusions. Statistics is the basis for inferential conclusions in these trials. After a product receives a marketing authorization, testing typically continues for additional uses of the product. A new product could also be included in a headto-head comparison against another marketed product for differentiation or comparative effectiveness research. Much of the work on comparative effectiveness is to support the valuation of a new product, particularly in regions where a government board decides if a new product is eligible for reimbursement and the price of the product under a national healthcare system. These efforts often
B
involve pooling data from multiple studies and can rely on endpoints different from those used to make marketing authorization decision. The work requires statisticians to collaborate closely with health economists, health care providers, third party payers and patients. Systematic review (including 7meta-analysis) is often an integral part of such efforts. Following several highly visible product withdrawals, the safety of pharmaceutical products has been a major source of public attention in recent years. Data from clinical trials, spontaneous reports of adverse reactions collected in pharmacovigilance databases and longitudinal patient information from healthcare or claims databases are increasingly used to explore possible product-induced injuries. Statistical techniques, based on the concept of proportionality (Almenoff et al. ), have been developed and applied extensively to look for possible safety signals. The decrease in the overall productivity measured by the number of approved new molecular entities each year has led industry and regulators to look for better ways to conduct biopharmaceutical research. Examples include FDA’s Critical Path Initiative (, http://www.fda.gov/ ScienceResearch/SpecialTopics/CriticalPathInitiative / ) and European Union’s Innovative Medicines Initiative () (http://www.imi.europa.eu/index_en.html). One outcome from this emphasis is the extensive research on adaptive trial designs over the last years (Gaydos et al. ; Bornkamp et al. ; Bretz et al. ; Gallo et al. etc.). Research on adaptive design includes designs for dose-ranging studies and confirmatory studies. Central to the concept of adaptive designs is a more efficient use of data and a more agile response to accumulated evidence on the effect of a new treatment. In biopharmaceutical research, statisticians are at the heart of evidence collection, synthesis and communication. Statisticians have enormous opportunities and face probably an equal number of challenges. We can expect both opportunities and challenges to increase in the twenty-first century. Statisticians need to be in tune with the dynamic environment, to help meet the needs of multiple customers, to cash in on the opportunities and rise to the challenges (Chuang-Stein et al. )!
About the Author Dr. Christy Chuang-Stein is Head of the Statistical Research and Consulting Center, Pfizer Inc, United States. She is a Vice President of the American Statistical Association (–) and a past Chair of the Biostatistics and Data Management Technical Group of the Pharmaceutical Research and Manufacturers of America (–).
B
B
Biostatistics
She is a Founding Editor (with two others) of the journal Pharmaceutical Statistics (–). She is a Fellow of the American Statistical Association (elected in ). She has authored and co-authored more than papers including a book on Analysis of Clinical Trials Using SAS: A Practical Guide (). She coedited a book on Pharmaceutical Statistics Using SAS (). Dr. Chuang-Stein received the Donald E Francke Award for Overall Excellence in Journal Publishing (, , ) and the Thomas Teal Award for Excellence in Statistics Publishing (, ) from the Drug Information Association. She also received the Excellence in Continuing Education Award from the American Statistical Association (). She has served as an Associate Editor for The American Statistician (–), Journal of Biopharmaceutical Statistics (–), Wiley Encyclopedia of Clinical Trials (–) and Drug Information Journal (–present).
Cross References 7Clinical Trials: An Overview 7Clinical Trials: Some Aspects of Public Interest 7Medical Research, Statistics in 7Statistical Analysis of Drug Release Data Within the
Pharmaceutical Sciences 7Statistical Analysis of Drug Release Data Within the Pharmaceutical Sciences
Gaydos B, Anderson K, Berry D, Burnham N, Chuang-Stein C, Dudinak J, Fardipour P, Gallo P, Givens S, Lewis R, Maca J, Pinheiro J, Pritchett Y, Krams M () Good practices for adaptive clinical trials in pharmaceutical product development. Drug Inform J ():– International Conference on Harmonisation E(R), Guidance for Good Clinical Practice, Step , May . Available at http://www.ich.org/LOB/media/MEDIA.pdf International Conference on Harmonisation E, Statistical Principles for Clinical Trials, Step , Feb . Available http://www.ich.org/LOB/media/MEDIA.pdf International Conference on Harmonisation E, The Choice of Control and Relatede Issues in Clinical Trials, Step , July . Available at http://www.ich.org/LOB/media/MEDIA.pdf International Conference on Harmonisation M(R): Maintenance of the ICH Guideline on Non-Clinical Safety Studies for the Conduct of Human Clinical Trials for Pharmaceuticals, Step , June . Available at http://www.ich.org/LOB/ media/MEDIA.pdf Krantz JC Jr () New drugs and the Kefauver-Harris amendment. J New Drugs :– Masia N () The cost of developing a new drug. In Focus on intellectual property rights. A US Department of State Publication, April . Available at http://www.america.gov/st/econEnglish//April/myleen..html Medical Research Council () Streptomycin treatment of pulmonary tuberculosis. Br Med J :– Senn S () Statistical issues in drug development (statistics in practice), nd edn. Wiley, New York Sheiner LB () Learning versus confirming in clinical drug development. Clin Pharm Ther :–
References and Further Reading Almenoff JS, Pattishall EN, Gibbs TG, DuMouchel W, Evans SJ, Yuen N () Novel statistical tools for monitoring the safety of marketed drugs. Clin Pharm Ther :– Bornkamp B, Bretz F, Dmitrienko A, Enas G, Gaydos B, Hsu C-H, Konig F, Liu Q, Neuenschwande B, Parke T, Pinheiro J, Roy A, Sax R, Shen F () Innovative approaches for designing and analyzing adaptive dose-ranging Trials. J Biopharm Stat : – Bretz F, Schmidli H, Racine A, Maurer W () Confirmatory seamless phase II/III clinical trials with hypotheses selection at interim: general concepts. Biometrical J ():– Chuang-Stein C, Bain R, Branson M, Burton C, Hoseyni C, Rockhold F, Ruberg S, Zhang J () Statisticians in the pharmaceutical industry: the st century. J Biopharm Stat ():– Declaration of Helsinki, History. Available at http://cme.cancer. gov/c/a_.htm Dmitrienko A, Molenberghs G, Chuang-Stein C, Offen W () Analysis of clinical trials using SAS: a practical guide. Cary NC, SAS Dmitrienko A, Chuang-Stein C, D’Agostino R () Pharmaceutical statistics using SAS. SAS, Cary NC, SAS FDA Guide for Clinical Trial Sponsors – Establishment and Operation of Clinical Trial Data Monitoring Committees, March . Available at http://www.fda.gov/downloads/ RegulatoryInformation/Guidances/ucm.pdf. Gallo P, Chuang-Stein C, Dragalin V, Gaydos B, Krams M, Pinheiro J () Adaptive designs in clinical drug development. J Biopharm Stat :–
Biostatistics Susan R. Wilson Professor, Faculty of Medicine, Faculty of Science University of New South Wales, Sydney, NSW, Australia The term Biostatistics is formed from the words biology and statistics, but these days more commonly refers to a somewhat narrower coverage of statistical methods needed in medicine and public health. In its broadest sense, biostatistics is the science of collecting and analyzing biological data to create knowledge about biological processes. The field of biostatistics (even as the term is generally used today) is wide and in its scope includes applications in clinical medicine, public health, epidemiology, genetics (genomics and all the other ’omics), health services, 7demography, and laboratory research. An essential part of the practice of biostatistics is collaboration between the statistician and the medical scientist or health professional. To give some insight into the very many facets encompassed by biostatistics we use the five central principles
Biostatistics
of applied statistics (of which Biostatistics arguably can be regarded as the largest branch) proposed by Cox (). .
.
.
.
Formulation of Objectives: This can vary very widely, dependent on the biological, health and related environmental issues being studied. Such issues range over, for example, calculations of birth and death rates, and life expectancy, finding the genomic and environmental basis of complex disease, evaluating the efficacy of a new treatment/drug, determination of efficacious dose for reliable drug production in pharmaceutical research, improving the delivery of health care. It can be useful to distinguish between decision-making and inference. A Bayesian approach is usually to be preferred for decision making in a narrow sense. Inference of causality is often a major objective and is widely studied, particularly in epidemiology. In the early stage of an investigation, biostatisticians can be important in helping to focus on the specific formulation of concrete objectives. Study design: This ranges from observational studies and sample surveys, to secondary analysis of collected data, through to experimental design. Major study types include the ecologic study, the cross-sectional study, the case-control study, the cohort study and clinical trials. Measurement: The actual entities being measured depend on the objectives, and specific situation, and range from instrument measurements to quality of life measures. Basic criteria include relevance, precision and minimal (preferably no) bias. Care may be needed to ensure that data quality does not drop if too many measurements are being taken, particularly when dealing with human subjects. One also needs to avoid any confounding effect on the system due to the measurement process itself; a much-studied example is the placebo effect in human studies. Analysis of Biostatistical Data: The three basic, common phases in any statistical data analysis are data editing and quality control, preliminary, often exploratory, analysis and graphics, and more detailed analysis. Of recent years there has been an explosion in statistical reasoning and methods for analysis of studies of human health. Examples include developments in epidemiological methods, clinical trials, survival analysis, statistical inference based on likelihood methods, 7statistical genetics. Although much analysis is based on probabilistic models, increasingly purely algorithmic approaches, such as cluster analysis (see 7Cluster Analysis: An Introduction), machine learning and 7bioinformatics, are being used. This is particularly the
.
B
case in the analyses of extremely large data sets where the current challenge is dealing with the extremely large number of measurements that is many times the number of observations. Interpretation: The borderline with analysis is not clearcut, but it is fundamentally important that biostatisticians present the conclusions in a way that is readily interpretable by the relevant members of the health community.
The human context of biostatistics differentiates it from other areas of applied statistics. In particular, ethical issues may arise, patients often do not make ideal “experimental units” and establishment of causation (as in all areas of biology) can be problematic. To achieve evidence-based medical practice, the Cochrane Collaboration was established in with the aims of facilitating meta-analyses of randomized clinical trials, to disseminate results effectively, and update these regularly. Many parts of the global field of biostatistics have become disciplines in their own right, such as statistical genetics, demography, actuarial science, methods for clinical trials, as well as bioassay. Often there is overlap with a part of another scientific area; one such example is the recently emerged discipline of bioinformatics. Many journals are either entirely devoted to biostatistics (including Biostatistics, Statistics in Medicine, Statistical Methods for Medical Research) or have a substantial part devoted (including Biometrics, Biometrical Journal). There are specialist journals for specific subareas, (such as Pharmaceutical Statistics, Journal of the Society for Clinical Trials). Besides specialist societies dealing with a subarea, many of the major statistical societies have a Biostatistics (or 7Medical Statistics) section. Biostatistical software has been expanding rapidly to handle developments of new methodology, as well as changing to meet the ongoing improvements in computer power and capability. There is an enormous, exponentially growing, number of biostatistics reference and textbooks, aimed at the great variety of backgrounds of those interested in biostatistics. The Encyclopedia of Biostatistics () offers a relatively definitive reference on the development and use of statistical methods for addressing the problems and critical questions that confront scientists, practitioners and policy makers involved in the medical and life sciences. Therein, broad sections of the subject were identified, covering specific biostatistical work (clinical trials, epidemiologic studies, clinical epidemiology, vital and health statistics, health services research, laboratory studies, biological models, health surveys and biomedical experiments), as well as particular branches of statistical methodology of special
B
B
Bivariate Distributions
biostatistical interest (7categorical data analysis, statistical models, longitudinal data analysis, multivariate analysis, survival analysis, sampling and experimental design, statistical computing), medical specialities with statistical applications, human genetics and genetic epidemiology. The growing importance and application of 7biostatistics is reflected in the increasing number of statisticians employed in the healthcare sector, pharmaceutical industry and medical schools. The boundaries are vague as the discipline of biostatistics is broad, linking a theoretical discipline, namely mathematical statistics, to a diversity of applied sciences relevant to medicine and human health.
About the Author Susan Wilson is Professor in the Faculties of Science and of Medicine at the University of New South Wales. She previously was Director, Centre for Bioinformation Science, Mathematical Sciences Institute, Australian National University. Professor Wilson is an elected Fellow of the Institute of Mathematical Statistics and of the American Statistical Association. She is a Past-President of the International Biometric Society.
Cross References 7Bioinformatics 7Biopharmaceutical Research, Statistics in 7Clinical Trials: An Overview 7Clinical Trials: Some Aspects of Public Interest 7Forensic DNA: Statistics in 7Medical Research, Statistics in 7Medical Statistics 7Statistical Genetics 7Statistical Methods in Epidemiology
areas such as dependence, correlation and conditional distributions. We consider the continuous and discrete cases, separately.
Continuous Bivariate Distributions Let (X, Y) denote two random variables defined on a domain of support Λ ⊂ IR , where we assume Λ is an open set in IR . Then a function f : Λ → IR+ is a joint bivariate pdf (probability density function) if it has the following properties: f (x, y) > , for (x, y) ∈ Λ ∫ ∫ f (x, y)dxdy = P((X, Y) ∈ S) = ∫ ∫ f (x, y)dxdy, for any S ⊂ Λ S
The joint cdf (cumulative distribution function) is given by:
Colin Rose Director Theoretical Research Institute, Sydney, NSW, Australia Bivariate distributions allow one to model the relationship between two random variables, and thus they raise subject
f (v, w)dvdw ()
∫ −∞
−∞
where ≤ F(x, y) ≤ . The probability content of a rectangular region S = {(x, y) : a < x < b, c < y < d} can be expressed in terms of the cdf F(x, y) as: P(a < X < b, c < Y < d) = F(a, c) − F(a, d) − F(b, c) + F(b, d)
()
The marginal pdf of X, denoted fx (x), is: fx (x) = ∫ f (x, y)dy
()
y
and similarly, the marginal pdf of Y, denoted fy (y), is: fy (y) = ∫ f (x, y)dx
()
x
The conditional pdf of X given Y = y is denoted by f (x∣Y = y) or, for short, f (x∣y). It is defined by f (x∣y) =
Bivariate Distributions
x
y
F(x, y) = P(X ≤ x, Y ≤ y) = ∫
References and Further Reading Cox DR () Applied statistics: a review. Ann Appl Stat :– Armitage P, Colton T (eds) () Encyclopedia of biostatistics, Wiley, Chicheste
()
Λ
f (x, y) , fy (y)
provided fy (y) >
()
Table lists some well-known bivariate pdf ’s. Whereas, in a univariate world, one speaks of the 7Gamma distribution (one single functional form), by contrast, in a bivariate world, there are a multitude of different bivariate Gamma distributions. This is because the term bivariate Gamma, or bivariate Exponential, or bivariate Uniform etc. is applied to essentially any bivariate distribution whose marginal pdf ’s are univariate Gamma, Exponential, or Uniform, respectively.
x > , y >
x − ρxy + y ) ( + √ v( − ρ ) π − ρ ( + x + y )−/ π
f (x, y) =
f (x, y) = f (x, y) = e−x−y−xyθ (( + θx)( + θy) − θ)
Bivariate T (standardised)
Bivariate Cauchy
Bivariate Exponential (Gumbel Type I)
√
<x
(x, y) ∈ IR
ca+b x a− (y − x)b− e−cy Γ[a]Γ[b] e−x−y ( + e−x + e−y )
f (x, y) =
f (x, y) = f (x, y) = + α(x − )(y − )
Bivariate Gamma (McKay)
Bivariate Logistic (Gumbel)
Bivariate Uniform (Morgenstern)
≤ x ≤ , ≤ y≤
x > , y >
ex+y + α(ex − )(ey − ) e(x+y)
f (x, y) =
(x, y) ∈ IR
(x, y) ∈ IR
(x, y) ∈ IR
Bivariate Exponential (Gumbel Type II)
−− v
− ρ y − µy y − µy ⎞ x − µx ⎛ x − µx ⎜ ( σx ) − ρ ( σx ) ( σy ) + ( σy ) ⎟ ⎟ ⎜ ⎟ exp ⎜− ) ⎟ ⎜ ( − ρ ⎟ ⎜ ⎠ ⎝ πσx σy
f (x, y) =
(x, y) ∈ IR
Bivariate Normal (general)
x − ρxy + y exp (− ) √ ( − ρ ) π − ρ
Domain
f (x, y) =
Bivariate Normal (standardised)
pdf
Distribution
Bivariate Distributions. Table Some bivariate continuous pdf’s
− ≤ α ≤
a, b, c >
− < α <
− ≤ θ ≤
− < ρ < , v >
σx > , σy >
− < ρ < ,
− < ρ <
Parameters
Bivariate Distributions
B
B
B
Bivariate Distributions
0.4 f
1.0 0.2 0.0 0.0
0.5 y
0.8 0.6 F 0.4 0.2 0.0 0
10
5y
0.5 x
x 1.0 0.0
pdf
5
cdf
10 0
Bivariate Distributions. Fig. The joint pdf f (x, y) = exp[(− − x)/y]x/y of Example (left) and the cdf F(x, y) (right)
2.0
1.5 1 3 4
y 1.0 5
29
2
29
29
29
29
0.5
0.0 0
1
2 x
3
4
Bivariate Distributions. Fig. Contours of the joint pdf f (x, y) = exp[(− − x)/y]x/y of Example , illustrated when f (x, y) = /, /, /, /, /
Example Joint pdf −−x Consider the function f (x, y) = e y x /y with domain of support Λ = {(x, y) : < x < ∞, < y < ∞}. Clearly, f is positive over its domain, and it integrates to unity over the domain. Thus, f (x, y) can represent the joint pdf of a pair
of random variables. Figure plots f (x, y) over part of its support, and the cdf over a somewhat wider region of its support … A contour plot allows one to pick out specific contours along which z = f (x, y) is constant. That is, each
Bivariate Distributions
contour joins points on the surface that have the same height z. Figure plots all combinations of x and y such that f (x, y) =
y
and similarly, the marginal pmf of Y, denoted fy (y), is:
Constructing Continuous Bivariate Distributions: Copulae
x
()
The conditional pmf of Y given X = x is denoted by f (y∣X = x) or, for short, f (y∣x). It is defined by f (y∣x) =
Let (X, Y) denote two random variables defined on a domain of support Λ ⊂ IR . Then a function f : Λ → IR+ is a joint pmf (probability mass function) if it has the following properties:
⇀
Π ( t ) = E [tX tY ]
()
⇀
P(X = r, Y = s) = r!s!
()
Λ
P((X, Y) ∈ S) = ∑ ∑ f (x, y), for any S ⊂ Λ S
The joint cdf is: ()
∂ r+s Π ( t ) ∂ tr ∂ ts
∣⇀
⇀
t =
()
Example Joint pmf Let random variables X and Y have joint pmf f (x, y) =
x+−y with domain of support Λ = {(x, y) : x ∈ {, , },
y ∈ {, , , }}, as per Table . This is a well-defined pmf since all the probabilities are positive, and they sum to . Figure plots the joint pmf and the joint cdf. For computational details, see Rose and Smith (). Example A bivariate Poisson Let Z , Z and Z be mutually stochastically independent univariate Poisson random variables, with non-negative parameters λ , λ and λ , respectively, and pmf ’s gi (zi ) for i ∈ {, , }: gi (zi ) =
f (x, y) = P(X = x, Y = y) > , for (x, y) ∈ Λ
v≤ x w≤ y
()
The pgf provides a way to determine the probabilities:
Discrete Bivariate Distributions
F(x, y) = P(X ≤ x, Y ≤ y) = ∑ ∑ f (v, w)
provided fx (x) >
()
Table lists some examples of copulae. With the exception of the independent case, each copula in Table includes parameter α. This term induces a new parameter into the joint bivariate distribution h(x, y), which gives added flexibility. In each case, setting parameter α = (or taking the limit α → , in the Frank case) yields the independent copula C = ΦG as a special case. For more detail on copulae, see, for instance, Nelson (). For many alternative ways to construct continuous bivariate distributions, see Balakrishnan and Lai ().
∑ ∑ f (x, y) =
f (x, y) , fx (x)
For extensive detail on discrete bivariate distributions, see Kocherlakota and Kocherlakota (). Given discrete random variables defined on subsets of ⇀ the non-negative integers {, , , . . .}, and t = (t , t ) ∈ IR , the bivariate probability generating function (pgf) is:
where C denotes the copula function. Then, the joint pdf h(x, y) is given by ∂ H(x, y) ∂x ∂y
()
fy (y) = ∑ f (x, y)
Copulae (see 7Copulas) provide a method for constructing bivariate distributions from known marginal distributions. Let the continuous random variable X have cdf Φ(x); similarly, let the continuous random variable Y have cdf G(y). We wish to create a bivariate distribution H(x, y) from these known marginals. The joint distribution function H(x, y) is given by
h(x, y) =
()
fx (x) = ∑ f (x, y)
For extensive detail on continuous bivariate distributions, see Balakrishnan and Lai ().
The marginal pmf of X, denoted fx (x), is:
, , , and .
H(x, y) = C(Φ, G)
B
e−λ i λ zi i defined on zi ∈ {, , , . . .} zi !
()
Due to independence, the joint pmf of (Z , Z , Z ) is g (z )g (z )g (z ). Then, a non-trivial bivariate Poisson distribution is obtained as the joint distribution of X and Y where: X = Z + Z and Y = Z + Z () The joint pmf of X and Y can be found via the method of transformations (see, for instance, Rose and Smith (,
B
B
Bivariate Distributions
Bivariate Distributions. Table Some examples of copulae Copula
Formula
Independent
C = ΦG
Morgenstern
C = ΦG( + α( − Φ)( − G))
− ≤ α ≤
ΦG C= − α( − Φ)( − G) ⎡ (e−αΦ − ) (e−αG − ) ⎤ ⎢ ⎥ ⎥ C = − log ⎢ ⎢ + ⎥ α e−α − ⎢ ⎥ ⎣ ⎦
− ≤ α <
Ali−Mikhail−Haq
Frank
Restrictions
Bivariate Distributions. Table Joint pmf of x+−y h(x, y) = Y= X= X= X=
Y=
The covariance of X and Y, denoted Cov(X, Y), is defined as µ , , namely:
Y=
α≠
Y=
Cov(X, Y) = E[(X − E[X])(Y − E[Y])]
()
The correlation between X and Y is defined as ρ = √
Cov(X, Y)
()
Var(X)Var(Y) where it can be shown, by the Cauchy–Schwarz inequality, that − ≤ ρ ≤ . ⇀
Let t = (t , t ) ∈ IR denote two dummy variables. Then the bivariate 7moment generating function (mgf) ⇀
p. )). Doing so yields the bivariate Poisson pmf as: f (x, y) = P(X = x, Y = y) x
= e(−λ −λ −λ ) ∑
⇀
MX,Y ( t ) is a function of t , defined by ⇀
i= i!(x − i)!(y − i)!
y = e(−λ −λ −λ ) λ x (−λ ) −x λ
()
M ( t ) = E[exp(t X + t Y)]
y−i λ i λ x−i λ
provided the expectation exists for all ti ∈ (−c, c), for some
U (−x, − x
+ y, − λλ λ )
x!y!
()
⇀
constant c > , i = , . If it exists, the mgf M ( t ) can be used to generate the product raw moments µ´ r,s = E [X r Y s ] as follows: ⇀
with domain of support {(x, y) : x ∈ {, , , . . .}, y ∈ {, , , . . .}}, and where U(a, b, c) denotes the confluent hypergeometric function.
Product Moments
∂ r+s M ( t ) µ´ r,s = E [X r Y s ] =
∂ tr ∂ ts
∣⇀
⇀
t =
()
The cumulant generating function is the natural logarithm of the mgf. The bivariate characteristic function is similar to () and given by
The bivariate raw moment µ´ r,s is defined as: ⇀ r
s
µ´ r,s = E [X Y ].
C ( t ) = E[exp(i (t X + t Y))]
()
() where i denotes the unit imaginary number.
With s = , µ´ r, denotes the rth raw moment of X. Similarly, with r = , µ´ ,s denotes the sth raw moment of Y. More generally, µ´ r,s is known as a product raw moment or joint raw moment. The bivariate central moment µ r,s is defined as µ r,s = E[(X − E[X])r (Y − E[Y])s ]
()
Dependence Let random variables X and Y have joint pdf f (x, y), with marginal density functions fx (x) and fy (y). Then X and Y are said to be mutually stochastically independent if and only if f (x, y) = fx (x)fy (y) ()
0.00 3
0.05
4 x
5 6 7 0
1
y
2
3 0.0
F(x,y) 0.5
1.0
0
3
1
y
x+−y of Example (left) and the joint cdf (right)
pmf
Bivariate Distributions. Fig. Joint pmf of h(x, y) =
f(x,y)
0.10
0.15
2
3
x
5 7 cdf
Bivariate Distributions
B
B
B 2
Bootstrap Asymptotics r =–0.98
2
0 –2 –2 2
2
2
0
0
2
r = –0.25
–2 –2 2
0 –2 –2
r =–0.74
0
0
2
r=0
2
r = 0.49
–2 –2 2
–2 –2 2
0
0
r =–0.49
0
2
r = 0.25
0
0
2
r = 0.73
–2 –2 2
0
0
0
–2 –2
–2 –2
–2 –2
of the Springer text Mathematical Statistics with Mathematica. His current area of research is exact computational methods in mathematical statistics, in particular, with application to the mathStatica project (winner of the Best Software Contribution at CompStat Berlin). He is an Editor of the Journal of Statistical Software, and a recent guest editor of the Mathematica Journal.
Cross References 0
2
r = 0.98
7Copulas 7Multivariate Statistical Distributions 7Statistical Distributions: An Overview 7Univariate Discrete Distributions: An Overview
References and Further Reading 0
2
0
2
0
2
Bivariate Distributions. Fig. Contour plots of the bivariate Normal pdf, for different values of ρ
i.e., if and only if the joint pdf is equal to the product of the marginal pdf ’s. If X and Y are mutually stochastically independent, then, amongst other properties, Cov(X, Y) = . Independence implies zero covariance, but the converse is not true: that is, zero covariance does not imply that X and Y are independent. Figure illustrates contour plots for a bivariate Normal pdf with zero means and variance–covariance matrix ⎛ ρ⎞ ⎜ ⎟. Here, ρ denotes the correlation coefficient between ⎝ρ ⎠ X and Y. Each plot corresponds to a specific value of ρ. In the top left corner, ρ = −. (almost perfect negative correlation), whereas in the bottom right corner, ρ = . (almost perfect positive correlation). The middle plot corresponds to the case of zero correlation. In any given plot, the edge of each shaded region represents the contour line, and each contour is a two-dimensional ellipse along which the bivariate Normal pdf f (x, y) is constant. Balakrishnan and Lai (, Chaps. and ) provide detail on alternative and more sophisticated concepts of dependence.
About the Author Dr. Colin Rose is director of the Theoretical Research Institute (Sydney). He is the co-author (with M.D. Smith)
Balakrishnan N, Lai CD () Continuous bivariate distributions, nd edn. Springer, New York Johnson NL, Kotz S, Balakrishnan N () Discrete multivariate distributions. Marcel Dekker, New York Kocherlakota S, Kocherlakota K () Bivariate discrete distributions. Wiley, New York Nelsen RB () An introduction to copulas, nd edn. Springer, New York Rose C, Smith MD () Mathematical statistics with Mathematica. Springer, New York
Bootstrap Asymptotics Rudolf Beran Distinguished Professor Emeritus University of California-Davis, Davis, CA, USA
The bootstrap (see 7Bootstrap Methods), introduced by Efron (), merges simulation with formal model-based statistical inference. A statistical model for a sample Xn of size n is a family of distributions {Pθ,n ∶ θ ∈ Θ}. The parameter space Θ is typically metric, possibly infinitedimensional. The value of θ that identifies the true distribution from which Xn is drawn is unknown. Suppose that θˆn = θˆn (Xn ) is a consistent estimator of θ. The bootstrap idea is (a) Create an artificial bootstrap world in which the true parameter value is θˆn and the sample Xn∗ is generated from the fitted model Pθˆn ,n . That is, the conditional distribution of Xn∗ , given the data Xn , is Pθˆn ,n .
Bootstrap Asymptotics
(b) Act as if a sampling distribution computed in the fully known bootstrap world is a trustworthy approximation to the corresponding, but unknown, sampling distribution in the model world.
B
distribution converges in probability to the value of that functional at the limit distribution. Such equicontinuity reasoning, in various formulations, is widespread in the literature on bootstrap convergence. For statistical models of practical interest, considerable insight may be needed to devise a metric on Θ such that the template sufficient conditions both hold. Some early papers on bootstrap convergence after Efron () are Bickel and Freedman (), Hall (), Beran (). Broader references are the books and monographs by Hall (), Mammen (), Efron and Tibshirani (), Davison and Hinkley () and the review articles in the bootstrap issue of Statistical Science (). These references leave the impression that bootstrap methods often work, in the sense of correct pointwise asymptotic convergence or pointwise second-order accuracy, at every θ in the parameter space Θ. Counterexamples to this impression have prompted further investigations. One line of research has established necessary and sufficient conditions for correct pointwise convergence of bootstrap distributions as n tends to infinity [cf. Beran (), van Zwet and Zwet ()]. In another direction, Putter () showed: Suppose that the parameter space Θ is complete metric and that (a) Hn (θ) ⇒ H(θ) for every θ ∈ Θ as n → ∞; (b) Hn (θ) is continuous in θ, in the topology of weak convergence, for every n ≥ ; (c) θˆn → θ in Pθ ,n -probability for every θ ∈ Θ as n → ∞. Then Hn (θˆn ) ⇒ H(θ) in Pθ ,n -probability for “almost all” θ ∈ Θ. The technical definition of “almost all” is a set of Baire category II. While “almost all” θ may sound harmless, the failure of bootstrap convergence on a tiny set in the parameter space typically stems from non-uniform convergence of bootstrap distributions over neighborhoods of that set. When that is the case, pointwise limits are highly deceptive. To see this concretely, let θˆn,S denote the 7James-Stein estimator for an unknown p-dimensional mean vector θ on which we have n i.i.d. observations, each having a N(, Ip ) error. Let Hn (θ) be the sampling distribution of the root n/ (θˆn,S − θ) under this model. As n tends to infinity with p ≥ fixed, we find (cf. Beran ()):
For example, consider constructing a confidence set for a parametric function τ(θ), whose range is the set T. As in the classical pivotal method, let Rn (Xn , τ(θ)) be a specified root, a real-valued function of the sample and τ(θ). Let Hn (θ) be the sampling distribution of the root under the model. The bootstrap distribution of the root is Hn (θˆn ), a random probability measure that can also be viewed as the conditional distribution of Rn (Xn∗ , τ(θˆn )) given the sample Xn . An associated bootstrap confidence set for τ(θ), of nominal coverage probability β, is then Cn,B = {t ∈ T∶ Rn (Xn , t) ≤ Hn− (β, θˆn )}. The quantile on the right can be approximated, for instance, by Monte Carlo techniques. The intuitive expectation is that the coverage probability of Cn,B will be close to β whenever θˆn is close to θ. When does the bootstrap approach work? Bootstrap samples are perturbations of the data from which they are generated. If the goal is to probe how a statistical procedure performs on data sets similar to the one at hand, then repeating the statistical procedure on bootstrap samples stands to be instructive. An exploratory rationale for the bootstrap appeals intellectually when empirically supported probability models for the data are lacking. Indeed, the literature on “statistical inference” continues to struggle with an uncritical tendency to view data as a random sample from a statistical model known to the statistician apart from parameter values. In discussing the history of probability theory, Doob () described the mysterious interplay between probability models and physical phenomena: “But deeper and subtler investigations had to await until the blessing and curse of direct physical significance had been replaced by the bleak reliability of abstract mathematics.” Efron () and most of the subsequent bootstrap literature postulate that the statistical model {Pθ ,n ∶ θ ∈ Θ} for the data is credible. “The bootstrap works” is taken to ¯ n ), where X ¯n mean that bootstrap distributions, and interesting func- (a) The natural bootstrap distribution Hn (X tionals thereof, converge in probability to the correct limits is the sample mean vector, converges correctly almost as sample size n increases. The convergence is typically everywhere on the parameter space, except at θ = . established pointwise for each value of θ in the parameter A similar failure occurs for the bootstrap distribution space Θ. A template argument: Suppose that Θ is metric Hn (θˆn,S ). and that (a) θˆn → θ in Pθ ,n -probability as n → ∞; (b) for (b) The weak convergences of the sampling distribution any sequence {θ n ∈ Θ} that converges to θ, Hn (θ n ) ⇒ Hn (θ) and of the two bootstrap distributions just H(θ). Then Hn (θˆn ) ⇒ H(θ) in Pθ ,n -probability. Moredescribed are not uniform over neighborhoods of the over, any weakly continuous functional of the bootstrap point of bootstrap failure, θ = .
B
B
Bootstrap Asymptotics
(c) The exact quadratic risk of the James-Stein estimator ¯ n at every θ, especially at strictly dominates that of X θ = . If the dimension p is held fixed, the region of substantial dominance in risk shrinks towards θ = as n increases. The asymptotic risk of the James-Stein estimator dominates that of the sample mean only at θ = . That the dominance is strict for every finite n ≥ is missed by the non-uniform limit. Apt in describing non-uniform limits is George Berkeley’s celebrated comment on infinitesimals: “ghosts of departed quantities.” In the James-Stein example, correct pointwise convergence of bootstrap distributions as n tends to infinity is an inadequate “bootstrap works” concept, doomed by lack of uniform convergence. The example provides a leading instance of an estimator that dominates classical counterparts in risk and fails to bootstrap naively. The message extends farther. Stein (, first section) already noted that multiple shrinkage estimators, which apply different shrinkage factors to the summands in a projective decomposition of the mean vector, are “better for most practical purposes.” Stein () developed multiple shrinkage estimators in detail. In recent years, low risk multiple shrinkage estimators have been constructed implicitly through regularization techniques, among them adaptive penalized least squares with quadratic penalties, adaptive submodel selection, or adaptive symmetric linear estimators. Naive bootstrapping of such modern estimators fails as it does in the James-Stein case. Research into these difficulties has taken two paths: (a) devising bootstrap patches that fix pointwise convergence of bootstrap distributions as the number of replications n tends to infinity [cf. Beran () for examples and references to the literature]; (b) studying bootstrap procedures under asymptotics in which the dimension p of the parameter space increases while n is held fixed or increases. Large p bootstrap asymptotics turn out to be uniform over usefully large subsets of the parameter space and yield effective bootstrap confidence sets around the 7James-Stein estimator and other regularization estimators [cf. Beran (), Beran and Dümbgen ()]. The first section of Stein () foreshadowed the role of large p asymptotics in studies of modern estimators.
About the Author Rudolf Beran was Department Chair at UC Davis (– ) and at UC Berkeley (–). He received in the Memorial Medal of the Faculty of Mathematics and
Physics, Charles University, Prague, in recognition of “distinguished and wide-ranging achievements in mathematical statistics, . . . devoted service to the international statistical community, and a long-lasting collaboration with Czech statisticians.” During – he held an Alexander von Humboldt U.S. Senior Scientist Award at Heidelberg University. He has authored or co-authored over papers in international journals and published lecture notes (with G. R. Ducharme) on Asymptotic Theory for Bootstrap Methods in Statistics (Publications CRM, Université de Montréal, ).
Cross References 7Bootstrap Methods 7Functional Derivatives in Statistics: Asymptotics and
Robustness
References and Further Reading Beran R () Prepivoting to reduce level error of confidence sets. Biometrika :– Beran R () Stein confidence sets and the bootstrap. Statistica Sinica :– Beran R () Diagnosing bootstrap success. Ann Inst Stat Math :– Beran R, Dümbgen L () Modulation of estimators and confidence sets. Ann Stat :– Bickel PJ, Freedman DA () Some asymptotic theory for the bootstrap. Ann Stat :– Davison AC, Hinkley DV () Bootstrap methods and their application. Cambridge University Press, Cambridge Doob JL () William Feller and twentieth century probability. In: Le Cam LM, Neyman J, Scott EL (eds) Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability. II, University of California Press, Berkeley and Los Angeles, pp xv–xx Efron B () Bootstrap methods: another look at the jackknife. Ann Stat :– Efron B, Tibshirani R () An introduction to the bootstrap. Chapman and Hall, New York Hall P () On the bootstrap and confidence intervals. Ann Stat :– Hall P () The bootstrap and Edgeworth expansion. Springer, New York Mammen E () When does bootstrap work? Lecture Notes in Statistics . Springer, New York Putter H () Consistency of resampling methods. PhD dissertation, Leiden University, Leiden Stein C () Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In: Neyman J (ed) Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability I, University of California Press, Berkeley and Los Angeles, pp – Stein C () An approach to the recovery of inter-block information in balanced incomplete block designs. In: David FN (ed) Festschrift for Jerzy Neyman. Wiley, New York, pp – van Zwet EW, Zwet WR () A remark on consistent estimation. Math Method Stat :–
Bootstrap Methods
Bootstrap Methods
Michael R. Chernick , Wenceslao González-Manteiga , Rosa M. Crujeiras , Erniel B. Barrios Lankenau Institute for Medical Research, Wynnewood, PA, USA University of Santiago de Compostela, Santiago de Compostela, Spain Professor and Dean University of the Philippines, Quezon City, Philippines
Introduction Use of the bootstrap idea goes back at least to Simon () who used it as a tool to teach statistics. But the properties of the bootstrap and its connection to the 7jackknife and other resampling methods, was not realized until Efron (). Similar resampling methods such as the jackknife and subsampling go back to the late s and s respectively (Quenouille () for the jackknife and Hartigan () and McCarthy () for subsampling). In the impact that the bootstrap would have was not really appreciated and the motivation for Efron’s paper was to better understand the jackknife and its properties. But over the past years it has had a major impact on both theoretical and applied statistics with the applications sometimes leading the theory and vice versa. The impact of Efron’s work has been so great that he was awarded with the President’s Medal of Science by former President George W. Bush and Kotz and Johnson () included the Annals of Statistics paper in their three volume work on breakthroughs in statistics. After the publication of Efron’s paper, Simon’s interest in bootstrapping was revitalized and he and Peter Bruce formed the company Resampling Stats which publicized the methodology and provided elementary software for teaching and basic data analysis (see Simon and Bruce ()). The bootstrap is not simply another statistical technique but is rather a general approach to statistical inference with very broad applicability and very mild modeling assumptions. There are now a number of excellent books that specialize in bootstrap or resampling in general. Included in this list are Efron (), Efron and Tibshirani (), Davison and Hinkley (), Chernick (, ), Hall (), Manly (), Lunneborg (), Politis et al. (), Shao and Tu () and Good (, ). Many other texts devote chapters to the bootstrap including a few introductory statistics texts. There are even a couple of books that cover subcategories of bootstrapping (Westfall and Young () and Lahiri (b)).
B
The Basic Idea Formally, denote by X = (X , . . . , Xn ) a random sample from X with unknown distribution F and consider a random variable T = T(X , . . . , Xn ; F) which may be as simple as T = X − µ, with µ = ∫ xdF(x), or a more complicated one such as a nonparametric kernel density estimator of x − Xi n ), the density f given by T = fˆh (x) = ∑K ( nh i= h where h denotes the bandwidth parameter and K is a kernel function. The main goal in statistical inference is to determine the sampling distribution of T, namely IPF (T(X, F) ≤ x). If Fn denotes the empirical distribution of X, from the sample X, then the bootstrap version of T is given by T ∗ = T (X∗ , . . . , Xn∗ ; Fn ) where X∗ = (X∗ , . . . , Xn∗ ) is a random sample from Fn . This is known as the naive bootstrap. We may also replace F by a smoothed version x F˜ n (x) = ∫−∞ fˆh (t)dt (called the smooth or generalized bootstrap) or by a parametric estimate of F say Fθˆn (parametric bootstrap). See Efron (, ) as introductory references, and Silverman and Young () and Dudewicz () for detailed coverage on the smooth bootstrap. As it has been said, the main goal is to estimate the sampling distribution of T. The bootstrap estimator of IPF (T(X, F) ≤ x) is given by IPFn (T(X∗ , Fn ) ≤ x) = IP∗ (T(X∗ , Fn ) ≤ x), where IP∗ is the associated probability in the bootstrap world (the distribution associated with sampling with replacement from Fn in the naive bootstrap case). Since in most cases this probability cannot be computed, Monte Carlo methods are used in order to obtain an approximation, based on B bootstrap replicates X∗j , with j = , . . . B. The estimate of IP∗ (T(X∗ , Fn ) ≤ x) is just ∗j ̂∗ (T(X∗ , Fn ) ≤ x) = #{j; T(X , Fn ) ≤ x} , IP B
where # denotes the cardinal of the set. Once we have approximated the statistic’s distribution, other inference problems such as bias and variance estimation, confidence interval construction or hypothesis testing, etc. can be tackled. With confidence intervals, the variable of interest is usually given by T = θˆ − θ, where θ is an unknown parameter of the distribution and θˆ is the corresponding estimator. The simplest case is based on the direct approximation of the distribution of T, known as the percentile method. Several refinements such as bias-corrected percentile, percentile-t (also called bootstrap-t) or other corrections based on Edgeworth expansions (see 7Edgeworth Expansion), have been proposed. For more complete coverage of bootstrap confidence intervals, see Hall (, ), Efron and Tibshirani (), Chernick () or Davison and Hinkley ().
B
B
Bootstrap Methods
Generally speaking, the bootstrap methodology aims to reproduce from the sample the mechanism generating the data which may be a probability distribution, a regression model, a time series, etc. Nowadays, bootstrap methods have been applied to solve different inference problems, including bandwidth selection in curve estimation (Cao ), distribution calibration in 7empirical processes or empirical regression processes (Stute et al. ) or inference for incomplete data detailed below, among others.
Extension to Dependent Data For the sake of simplicity, we may distinguish two perspectives. First, data involved in the statistic may be directly observed, where the basic bootstrap resampling procedures introduced above can be applied. Secondly, data may exhibit a complex generating mechanism. As a special case, consider a parametric regression model yi = mθ (xi ) + ε i ,
i = , . . . , n,
where mθ (⋅) is the regression function and ε i denotes the associated ith error. For fixed design and parametric regression function, we may proceed by resampling the residuals ei = yi − mθˆ (xi ), where θˆ is a parameter estimator. Naive bootstrap samples ˜e∗i drawn from the empirical distribution of the centered residuals {ei − e} are used to get the bootstrap regression model y∗i = mθˆ (xi ) + ˜e∗i . This approach is called model-based bootstrapping. Efron also introduced the bootstrap in this context. Each of the bootstrap samples can provide an estimate of the regression parameter (possibly a vector of parameters) following the same estimation procedure that was used with the original fitted model (e.g., ordinary 7least squares). From all the bootstrap replicates, we get a Monte Carlo approximation to the bootstrap distribution of the regression parameter and this is then used to make inferences about the parameter based on this approximation to the sampling distribution for the parameter estimate. In model-based inference, Paparoditis and Politis () underscored the importance of the choice of residuals. For example, to maximize power in bootstrap-based hypothesis testing, residuals are obtained using a sequence of parameter estimators that converge to the true parameter value under both the null and alternative hypotheses. Different modifications of this simple idea allow for adapting to random design, heterocedastic models or situations where the regression function is not totally specified or is unknown, such as in nonparametric regression. From the first references, specially in parametric regression, by Bickel and Freedman () and Freedman (), several
advances have been introduced in this context. For the nonparametric case, see Chap. in Schimek (). Although bootstrap originally started with independent sample observations, just as in the regression models described above, there are extensions to dependent data: 7time series, 7spatial statistics, 7point processes, spatiotemporal models and more. For example, similar to the ideas of bootstrap in regression models, given an explicit dependence structure such an autoregressive model, we may write: yi = m(yi− , . . . , yi−p ) + ε i and proceed by resampling from the residuals. When an explicit parametric equation is not available, an alternative is block bootstrap, which consists of resampling blocks of subsamples, trying to capture the dependence in the data. Bootstrap replicates obtained by these methods may not be stationary even though the original data came from a stationary process. In order to solve this problem, Politis and Romano (a) propose the stationary bootstrap. These bootstrap procedures can be adapted for predicting the future values of the process. An overview of bootstrap methods for estimation and prediction in time series can be found in Cao (). The idea of block bootstrap has also been extended to the spatial setting (see Lahiri b) There is theoretical justification for bootstrapping time series. As an example, Politis and Romano (b) established convergence of certain sums of stationary time series that can facilitate bootstrap resampling. The block bootstrap was among the early proposals for time series data. While the method is quite straightforward, there are associated problems like getting independence between blocks while maintaining the dependence structure within the block. The size of the block is a crucial quantity that should be determined to assure success in block bootstrapping. The AR-sieve method was also introduced as a residual-based method similar to the model-based approach. Local bootstrap was also introduced but in the context of a local regression framework (nonparametric) and to account for the nonparametric model, resampling allows the empirical distribution to vary locally in the time series. Bühlman () compared different methods for time series bootstrapping. The block bootstrap is recognized as the most general and simplest generalization of the original independent resamples but is sometimes criticized for the possible artifacts it may exhibit when blocks are linked together. Blocking can potentially introduce some dependence structure in addition to those naturally existing in the data. The AR-sieve is less sensitive to selection of a
Bootstrap Methods
model than the block to the block length. The local bootstrap for 7nonparametric estimation is observed to yield slower rates of convergence. Generally, the AR-sieve is relatively advantageous among the bootstrap approaches for time series data. Recently, the bootstrap has been introduced in more complex and complicated models. In modeling nonstationary volatility, Xu () used an autoregression around a polynomial trend with stable autoregressive roots to illustrate how non-stationary volatility affects the consistency, convergence rates and asymptotic distributions of the estimators. Westerlund and Edgerton () proposed a bootstrap test for the null hypothesis of cointegration in 7panel data. Dumanjug et al. () developed a block bootstrap method in a spatial-temporal model.
Diversity of Applications The use of bootstrap as a tool for calibrating the distribution of a statistic has been extended to most topics in statistical inference. Not trying to be exhaustive, it is worth it considering the immersion of bootstrap in the following fields of study: .
.
.
.
.
Incomplete data. When dealing with censored data, the empirical estimator of the distribution is replaced by the 7Kaplan-Meier estimator (Efron ). Missing information. If some observations are missing or imputed, bootstrap estimators must be suitably adapted (Efron ). Hypothesis testing in regression models. When the goal is to check whether a parametric regression model mθ fits the data, the distribution of a test statistic ˆ h , mθˆ ), where m ˆ h is a nonparametric estimaT = D(m tor of the regression function m and D is a distance measure, must be calibrated. In this context, a broad literature can be cited, such as Härdle and Mammen () or Cao and González-Manteiga (). For a recent review on the topic, see González-Manteiga and Crujeiras (). Small area inference. Bootstrap has also shown a great development in finite populations (Shao and Tu ), specially in recent years with the appearance of small area models. See Hall and Maiti () and Lahiri (a). Bootstrap has also recently been used in learning theory and high dimensional data. As an example, see the application of bootstrap in the regression model with functional data (Ferraty et al. ) or the bootstrap for variable choice in regression or classification models (Hall et al. ).
B
The continuing development of bootstrap methods has been motivated by the increasing progress in computational efficiency in recent years. Other computer-intensive methods are the 7Markov Chain Monte Carlo, usually known as MCMC [see Smith and Roberts (), for instance] and subsampling (Politis et al. ).
Some Historical Development The bootstrap’s popularity rests in its relaxation of distribution assumptions that can be restrictive and its wide variety of applications as described above. We see that the development of the bootstrap evolved as follows. Efron introduced it in () with some theoretical and heuristic development. Theoretical activity followed quickly with Athreya, Bickel, Freedman, Singh, Beran and Hall providing notable contributions in the early s. Efron realized its practical value early on and efforts to make the scientific community aware of its potential were the Diaconis and Efron () article in Scientific American and the article by Efron and Tibshirani (). So by the early s enough theory and successful applications had developed to lead to an explosion of papers, mostly applied and some extending the theory. The literature was so large that Chernick () contains more than , references. An excellent and nearly up-to-date survey article on the bootstrap is Lahiri ().
When and Why Bootstrap Can Fail to Work For the bootstrap to work, the bootstrap estimates must be consistent. But even when the first results on consistency of the bootstrap estimate of a mean were derived, Bickel, Freedman and others realized that there were cases where the bootstrap is inconsistent. Two notable examples are () the sample mean when the population distribution has an infinite variance but the ordinary sample mean appropriately normalized still converges to a stable law and () the maximum of a sample when the population distribution is in the domain of attraction of an extreme value distribution. These examples are covered in Chap. of Chernick (). These results on bootstrap inconsistency were already published in the s and led to a concern about what the real limitations of the bootstrap are. The volume edited by LePage and Billard () and the monograph by Mammen () address these concerns. Consistency is one requirement but what about the small sample properties? This was addressed beautifully using simulation as illustrated in Shao and Tu () and Efron (). The work of Efron
B
B
Bootstrap Methods
and others on small sample accuracy of bootstrap estimates of error rates in classification is summarized in Chap. of Chernick ().
Remedies for Failure Efron’s bootstrap principle states that the nonparametric bootstrap mimics sampling from a population by letting the empiric distribution Fn play the role of the unknown population distribution F and letting the bootstrap distribution Fn∗ play the role of Fn . This is to say in words what was described using formal mathematics earlier in this article. Efron thought it was natural to take the size of a bootstrap sample to be n but others saw no reason why a value m < n could not be used. For many problems where consistency was shown m = n works fine. Bickel and Ren () introduced a bootstrap approach called the m-out-of-n bootstrap which takes m < n and works well in some problems. A recent advance in bootstrapping was the proof that for the two examples previously described, where the bootstrap is inconsistent when m = n, the m-out-of-n bootstrap is consistent provided m tends to infinity at a slower rate than n and slow enough for m/n → . Prior to the formal introduction of the m-out-of-n bootstrap, Athreya () showed that for heavy-tailed distributions a trimmed mean could converge to the population mean with the advantage that the sampling distribution of the trimmed mean has second moments. Using this idea he proved consistency of the m-out-of-n bootstrap for the mean in the infinite variance case. Fukuchi () did the same for extremes. Both results require m and n to approach infinity at a rate that would have m/n → . The m-out-of-n bootstrap has been further studied by Bickel et al. () and Politis et al. (). Zelterman () found a different way to modify the bootstrap to make it consistent for the extreme values. This is all summarized in Chap. of Chernick (). The theoretical developments from to the present have been in the area of () modifying the bootstrap to fix inconsistency in order to widen its applicability and () extending the theory to dependent situations (as previously mentioned). Lahiri (a) is the ideal reference for a detailed account of these developments with dependent data.
Problems and Refinements for Bootstrap Confidence Intervals Bootstrap confidence intervals have been a concern and Efron recognized early on that getting the asymptotic coverage nearly correct in small samples required more sophistication than his simple percentile method bootstrap. So the bias corrected bootstrap was developed to
do that. However, in Schenker () the example of variance estimation for a particular chi square population distribution showed that even the BC method had coverage problems in small samples. Efron () introduced the BCa method which remedied the problem discovered by Schenker. However, in recent years variance estimation for other examples with skewed or heavy-tailed distributions has shown all bootstrap confidence interval methods to be problematic in small samples. A large Monte Carlo investigation, Chernick and LaBudde (), compares the coverage of various bootstrap confidence intervals for the variance estimate from a variety of population distributions when sample sizes are small. They also provide an idea of rates of convergence by showing how the coverage improves as the sample size gets large. An interesting surprise is that in some situations for small sample sizes the lower order bootstrap work better than the higher order ones. This is because they are simpler and do not involve estimating biases and acceleration constants which depend on third order moments of the distribution. For the lognormal population they show that at the small sample sizes (–) the coverage error is shockingly high for all methods.
About the Authors Dr. Chernick is the author of Bootstrap Methods: A Practitioner’s Guide (Wiley ), with the second edition title Bootstrap Methods: A Guide for Practitioners and Researchers (Wiley ). He is also the coauthor of Introductory Biostatistics for the Health Sciences: Modern Methods including Bootstrap (Wiley ). He is currently Manager of Biostatistical Services at the Lankenau Institute for Medical Research. He is the winner of the Jacob Wolfowitz Prize in and is a past President of the Southern California Chapter of the American Statistical Association. He has taught at California State University and the University of Southern California. He is elected Fellow of the American Statistical Association (). Wenceslao González-Manteiga is a full-time professor and main researcher at the Department of Statistics and Operations Research (University of Santiago de Compostela, Spain). He received his Ph.D. degree in Mathematics at this University. He was Editor-in-Chief of TEST for years and the International Journal of the Spanish Statistical Society. Over the years, he has collaborated with researchers in many fields, visiting several universities and has published about papers in journals such as Technometrics, Chemometrics, Journal of the American Statistical Association, Annals of Statistics and Computational
Bootstrap Methods
B
Statistics and Data Analysis. He has also collaborated in several projects in applied statistics. Rosa M. Crujeiras is an associate professor and the Department of Statistics and Operations Research (University of Santiago de Compostela, Spain). With a dissertation thesis on spectral methods for spatial data, she obtained her Ph.D. degree in Mathematics at this University in January . She has been a post-doctoral researcher at the Université catholique de Louvain (Belgium). She has published several papers in international journals, such as Environmetrics, Theory of Probability and its Applications or Computational Statistics and Data Analysis. Dr. Erniel Barrios is a Professor and Dean, School of Statistics, University of the Philippines Diliman He was a board member of the Philippine Statistical Association. He served as visiting researcher at The Ohio State University (USA) and the Asian Development Bank Institute (Japan). He has authored and co-authored over papers and a forthcoming book on small area estimation. He served as Associate Editor of Social Science Diliman, a journal published by University of the Philippines Diliman (– ) and is incoming Editor of The Philippine Statistician published by the Philippine Statistical Association.
(eds) Robust statistics, data analysis, and computer-intensive methods. Lecture Notes in statistics, vol. . Springer–verlag, New York, pp – Bühlman P () Sieve bootstrap for time series. Bernoulli :– Bühlman P () Bootstrap for time series. Stat Sci :– Cao R () Bootstrapping the mean integrated squared error. J Multivariate Anal :– Cao R () An overview of bootstrap methods for estimating and predicting in time series. Test :– Cao R, Gonzälez-Manteiga W () Bootstrap methods in ion smoothing. J Nonparametric Stat :– Chernick MR () Bootstrap methods: a practitioners guide. Wiley, New York Chernick MR () Bootstrap methods: a guide for practitioners and researchers, nd edn. Wiley, Hoboken Chernick MR, LaBudde R () Revisiting qualms about bootstrap confidence intervals. Am J Math Manage Sci, To appear Davison AC, Hinkley DV () Bootstrap methods and their applications. Cambridge University Press, Cambridge Diaconis P, Efron B () Computer-intensive methods in statistics. Sci Am :– Dudewicz EJ () The generalized bootstrap. In: Jockel K-H, Rothe G, Sendler W (eds) Bootstrapping and related techniques, Proceedings Trier FRG. Lecture Notes in Economics and Mathematical Systems, vol. . Springer, Berlin, pp –
Cross References
Efron B () Bootstrap methods. Another look at the jackniffe. Ann Stat :–
7Bootstrap Asymptotics 7Exact Goodness-of-Fit Tests Based on Sufficiency 7Functional Derivatives in Statistics: Asymptotics and
Robustness 7Jackknife 7Markov Chain Monte Carlo 7Monte Carlo Methods in Statistics 7Multiple Imputation 7Statistical Inference for Stochastic Processes 7Statistics: An Overview 7Target Estimation: A New Approach to Parametric Estimation
References and Further Reading Andrews D () Inconsistency of the bootstrap when a parameter is on the boundary of the parameter space. Econometrica :– Athreya KB () Bootstrap estimation of the mean in the infinite variance case. Annals of Statistics :– Beran R () Diagnosing bootstrap success. Ann Inst Stat Math :– Bickel PJ, Freedman DA () Some asymptotic theory for the bootstrap. Ann Stat :– Bickel PJ, Gotze F, van Zwet WR () Resampling fewer than n observations: gains, loses, and remedies for losses. Statistica Sinica :– Bickel PJ, Ren JJ () The m out of n bootstrap and goodness of fit tests with doubly censored data. In: Huber PJ, Rieter H
Dumanjug C, Barrios E, Lansangan J () Bootstrap procedures in a spatial-temporal model. J Stat Comput Sim :–
Efron B () Censored data and the Bootstrap. J Am Stat Assoc :– Efron B () The jacknife, the bootstrap and other resampling plans. SIAM, Philadelphia Efron B () Estimating the error rate of a prediction rule: improvements on cross-validation. J Am Stat Assoc :– Efron B () Better bootstrap confidence intervals (with discussion). J Am StatAssoc :– Efron B () Missing data, imputation and the bootstrap. J Am Stat Assoc :– Efron B () The bootstrap and modern statistics. J Am Stat Assoc :– Efron B, Tibshirani R () Bootstrap methods for standard errors, confidence intervals and other measures of statistical accuracy. Stat Sci :– Efron B, Tibshirani RJ () An introduction to the bootstrap. Chapman and Hall, London Ferraty F, Van Keilegom I, Vieu P () On the validity of the bootstrap in nonparametric functional regression. Scand J Stat :– Freedman DA () Bootstrapping regression models. Ann Stat :– Fukuchi JI () Bootstrapping extremes of random variables. PhD dissertation. Iowa State University, Ames González-Manteiga W, Cao R () Testing the hypothesis of a general linear model using nonparametric regression estimation. Test :– González-Manteiga W, Crujeiras RM () A review on goodnessof-fit tests for regression models. Pyrenees Int Workshop Stat Probab Oper Res: SPO :–
B
B
Borel–Cantelli Lemma and Its Generalizations
Good P () Resampling methods: a practical guide to data analysis. Birkhauser, Boston Good P () Permutation, parametric, and bootstrap tests of hypotheses, rd edn. Springer, New York Hall P () Theoretical comparison of bootstrap confidence intervals. Ann Stat :– Hall P () The bootstrap and Edgeworth expansion. Springer, Berlin Hall P, Maiti T () On parametric bootstrap methods for small area prediction. J R Stat Soc, Series B Stat Method :– Hall P, Lee ER, Park BU () Bootstrap-based penalty choice for the Lasso achieving oracle performance. Statistica Sinica :– Härdle W, Mammen E () Comparing nonparametric versus parametric regression fits. Ann Stat :– Hartigan JA () Using subsample values as typical values. J Am Stat Assoc :– Kotz S, Johnson NL () Breakthroughs in statistics vol. II: methodology and distribution. Springer, New York Lahiri P (a) On the impact of bootstrap in survey sampling and small-area estimation. Stat Sci – Rev J Inst Math Stat : – Lahiri SN (b) Resampling methods for dependent data. Springer Series in Statistics, Springer, New York Lahiri SN () Bootstrap methods: a review. In: Fan J, Koul HL (eds) Frontiers in statistics. Imperial College Press, London, pp – LePage R, Billard L (eds) () Exploring the limits of bootstrap. Wiley, New York Lunneborg CE () Data analysis by resampling: concepts and applications. Brooks/Cole, Pacific Grove Mammen E () When does the bootstrap work? Asymptotic results and simulations. Springer, Heidelberg Manly BFJ () Randomization, bootstrap and Monte Carlo methods in biology, nd edn. Chapman and Hall, London McCarthy PJ () Pseudo-replication: half-samples. Int Stat Rev :– Paparoditis E, Politis D () Bootstrap hypothesis testing in regression models. Stat Probab Lett :– Politis DN, Romano JP (a) The stationary bootstrap. J Am Stat Assoc :– Politis DN, Romano J (b) Limit theorems for weakly dependent Hilbert Space valued random variables with application to the stationary bootstrap. Statistica Sinica :– Politis DN, Romano JP, Wolf M () Subsampling. Springer, New York Quenouille MH () Appraximate tests of correlation in time series. J R Stat Soc B :– Schenker N () Qualms about bootstrap confidence intervals. J Am Stat Assoc :– Schimek MG () Smoothing and regression. Wiley series in probability and statistics: applied probability and statistics, Wiley, New York Shao J, Tu DS () The jackknife and bootstrap. Springer Series in Statistics, Springer, New York Silverman BW, Young GA () The bootstrap: smooth or not to smooth? Biometrika :– Simon JL () Basic research methods in social science. Random House, New York Simon JL, Bruce P () Resampling: a tool for everyday statistical work. Chance :–
Smith AFM, Roberts GO () Bayesian computation via the Gibbs sampler and related Markov Chain Monte Carlo methods. J R Stat Soc B Methodol :– Stute W, Gonzlez-Manteiga W, Presedo-Quindimil MA () Bootstrap approximations in model checks for regression. J Am Stat Asoc :– Tukey JW () Bias and confidence in not quite large samples (abstract). Ann Math Stat : Westerland J, Edgerton D () A panel bootstrap cointegration test. Econ Lett :– Westfall P, Young SS () Resampling-based multiple testing: examples and methods for p-value adjustment. Wiley, New York Xu K () Bootstrapping autoregression under nonstationary volatility. Econ J :– Zelterman D () A semiparametric bootstrap technique for simulating extreme order statistics. J Am Stat Assoc :–
Borel–Cantelli Lemma and Its Generalizations Tapas Kumar Chandra , Fejzi Kolaneci Professor Indian Statistical Institute, Calcutta, India Professor University of New York, Tirana, Albania The celebrated Borel–Cantelli Lemma is important and useful for proving the 7laws of large numbers in the strong form. Consider a sequence of random events {An } on a probability space (Ω, F, P), and we are interested in the question of whether infinitely many random events occur or if possibly only a finite number of them occur. The upper limit of the sequence {An } is the random event defined by ∞ ∞
{An i.o.} = lim sup An = ⋂ ⋃ Ak , n→∞
n= k=n
which occurs if and only if an infinite number of events An occur. This i.o. stands for “infinitely often.” Below we shall use the fact that if {An } is a sequence of random events, then ∞
∞
P ( ⋃ An ) ≤ ∑ P(An ). n=
(∗)
n=
The Borel–Cantelli Lemma Lemma If ∑∞ n= P(An) < ∞, then P(lim supn→∞ An) = . If the random events A , A , ⋯, An , ⋯ are independent and ∞ ∑n= P(An ) = ∞, then P(lim sup An ) = . n→∞
Borel–Cantelli Lemma and Its Generalizations
Intuitively, P(lim sup An ) is the probability that the n→∞
random events An occur “infinitely often” and will be denoted by P(An i.o.).
The first part of the Borel-Cantelli Lemma was generalized in Barndorff-Nielsen ().
∞
First part
lim P(An ) =
∞
n→∞
Note that lim sup An ⊂ ⋃ Ak for each m ≥ . So for n→∞
′
∑ P(An An+ ) < ∞. n=
Then P(lim sup An ) = .
∞
∞
P (lim sup An ) ≤ P ( ⋃ Ak ) ≤ ∑ P(Ak ) n→∞
k=m
k=m
We show that ′
− P(lim sup An ) = P((lim sup An ) ) = , n→∞
n→∞
by (∗).
∞ Since ∑∞ n= P(An ) is convergent, the tails ∑k=m P(Ak ) → as m → ∞. Letting m → ∞, we get P(lim supn→∞ An ) = . Second part
n→∞
Lemma holds true if the random events A′n An+ are substituted with An A′n+ . It should be noted that the hypothesis in Lemma is weaker than the hypothesis in Lemma . A further generalization of Lemma was obtained in Stepanov (). Lemma Let {An } be a sequence of random events satisfying the condition limn→∞ P(An ) = . Assume that there exists m ≥ such that ∞
where A′ denotes the complement of A. To this end, it is enough to show that
′ ′ ′ ∑ P(An An+ ⋯An+m− An+m ) < ∞. n=
Then
∞
P ( ⋂ A′k ) = for each m ≥ ,
P(lim sup An ) = .
k=m
n→∞
k=m
Observe that the hypothesis in Lemma when m ≥ is weaker than the hypothesis in Lemma . Many attempts were made in order to weaken the independence condition in the second part of the BorelCantelli Lemma. This condition means mutual independence of random events A , A , ⋯, An for every n. Erdös and Rényi () discovered that the independence condition can be replaced by the weaker condition of pairwise independence of the random events A , A , ⋯, An for every n. Indeed they also proved the result which is the special case ‘l = ’ in the following lemma due to Kochen and Stone ():
⎛ m+j ⎞ ≤ exp − ∑ P(Ak ) . ⎝ k=m ⎠
Lemma If {An } is a sequence of random events satisfying the conditions
since then by De Morgan’s Rule ∞ ∞
∞
∞
P ((lim sup An )′ ) = P ( ⋃ ⋂ A′k ) ≤ ∑ P ( ⋂ A′k ) n→∞
n= k=n
n=
k=n
= by (∗). Fix such an m. Since − x ≤ exp(−x) for each real x, we have for each j ≥ m+j
m+j
P ( ⋂ A′k ) ≤ P ( ⋂ A′k ) = ∏ ( − P(Ak )) k=m
and
k=m
each m ≥ .
k=m
∞ As ∑∞ n= P(An ) diverges, so does ∑k=m P(Ak ) which m+j implies that ∑k=m P(Ak ) → ∞ as j → ∞. As limx→∞ exp(−x) = , we get upon letting j → ∞ that ∞
P ( ⋂ A′k ) = . k=m
Generalizations
Lemma Let {An } be a sequence of random events satisfying the conditions
Proofaaaaa
∞
B
∞
∑ P(An ) = ∞ n=
and
lim inf n→∞
n ∑i,k= P(Ai ⋂ Ak ) = l, [∑nk= P(Ak )]
then P(lim sup An ) ≥ . l n→∞ A further extension due to Chandra () is given below.
B
B
Box–Cox Transformation
Lemma Let {An } be a sequence of random events such ∞
that ∑ P(An ) = ∞. Let n=
lim inf n→∞
∑≤i≤j≤n (P(Ai ⋂ Aj ) − aij ) =L (∑≤k≤n P(Ak ))
where aij = (c P(Ai ) + c P(Aj ))P(Aj−i ) + c P(Ai )P(Aj ) for ≤ i < j, c ≥ , c ≥ , c ∈ R being constants (L may depend on c , c , c ). Assume that L is finite. Then c + L ≥ and P (lim sup An ) ≥ (c + L)− n→∞
Statistics, Mathematical Analysis, and Differential Equations. Professor Kolaneci is past Secretary of Natural and Technical Sciences of Albanian Academy of Sciences, and past Dean of the Teaching Faculty of University “Fan S. Noli.” He is member of London Mathematical Society, and member of Society for Industrial and Applied Mathematics.
Cross References 7Almost Sure Convergence of Random Variables 7Stochastic Global Optimization 7Strong Approximations in Probability and Statistics
where c = (c + c ) + c . As a special case of Lemma , we have the following result. Lemma
Let {An } be a sequence of random events such
∞
that ∑ P(An ) = ∞. If for some constants c ≥ , c ≥ , and n=
c ∈ R there exists an integer N ≥ such the P(Ai ⋂ Aj ) ≤ aij whenever N ≤ i < j where the aij are as in Lemma , then c ≥ and P(lim supn→∞ An ) ≥ /c, c being as in Lemma . Petrov () found conditions that are necessary and sufficient for the equality P(lim sup An ) = p ,
where ≤ p ≤ ,
n→∞
as well as for the inequality P(lim sup An ) ≥ p ,
where < p ≤ .
n→∞
For a different type of extensions of the Borel-Cantelli lemma, see Chen () and Serfling ().
About the Authors Professor Chandra’s area of specialization spans Statistical Inference, Asymptotic Theory of Statistics, Limit Theorems and Large Deviations. He earned Ph.D. degree in Statistics in from The Indian Statistical Institute. He has published two books and about papers in international journals. One important contribution is the demonstration of the superiority of the Score Test over a large family of tests including the Likelihood Ratio test and Wald’s test under local alternatives, thereby settling an old conjecture of Professor C.R. Rao. Dr. Fejzi Kolaneci is Professor and Chair of the Department of Mathematics, University of New York Tirana, Albania. He is Past Editor-in-Chief of AJNTS, Albanian Journal of Natural and Technical Sciences. He has written several textbooks, including Probability Theory And
References and Further Reading Balakrishnan N, Stepanov A () Generalization of BorelCantelli lemma. Math Sci (), http://www.appliedprobability. org/content.aspx?Group=tms&Page=tmsabstracts Barndorff-Nielsen O () On the rate of growth of the partial maxima of a sequence of independent identically distributed random variables. Math Scan :– Chandra TK () The Borel–Cantelli lemma under dependence conditions. Stat Probabil Lett :– Chen LHY () A short note on the conditional Borel-Cantelli lemma. Ann Probab :– Erdös P, Rényi A () On Cantor’s series with convergent ∑ /q n . Ann Univ Sci Budapest Sec Math :– Kochen SB, Stone CJ () A note on the Borel-Cantelli lemma, Illinois. J Math ():– Petrov VV () Limit theorems of probability theory. Oxford University Press, Oxford Serfling RJ () A general Poisson approximation theorem. Ann Prob :– Stepanov A () Generalization of Borel-Cantelli lemma. eprint: arXiv:math/v
Box–Cox Transformation Takashi Daimon Hyogo College of Medicine, Hyogo, Japan Osaka University Hospital, Osaka, Japan
Box and Cox () proposed a family of power transformations in order to improve additivity, normality, and homoscedasticity of observations. The Box–Cox transformation, which was a modification of a family of power transformations introduced by Tukey (), was named for their work. For each value of a real or vector valued transformation parameter λ, let ψ(y, λ) be a strictly monotone increasing transformation for a positive y in
Box–Cox Transformation
some interval. Tukey’s power transformations takes the following form: ⎧ ⎪ λ ⎪ ⎪ ⎪ y , ψ(y, λ) = ⎨ ⎪ ⎪ ⎪ log y, ⎪ ⎩
λ =/ , λ = .
To take account of the discontinuity at λ = in the above equation, the original form of the Box–Cox transformation takes the following form: ⎧ ⎪ yλ − ⎪ ⎪ , ⎪ λ ψ BC (y, λ) = ⎨ ⎪ ⎪ ⎪ log y, ⎪ ⎩
λ =/ , λ = .
They also proposed an extended form of the Box–Cox transformation, “shifted” power transformation which could deal with situations where y is negative but bounded below: ⎧ ⎪ (y + λ ) λ − ⎪ ⎪ , λ =/ , ⎪ λ ψ(y, λ) = ⎨ ⎪ ⎪ ⎪ log(y + λ ), λ = , ⎪ ⎩ where λ = (λ , λ )T . However, since the range of the distribution is determined by the unknown shift parameter λ , the asymptotic results of maximum likelihood theory may not apply. Consequently, there have existed some alternative versions of the Box–Cox transformation which could handle a negative y. For example, Manly () proposed the exponential transformation: ⎧ exp(λy) − ⎪ ⎪ ⎪ , ⎪ λ ψ(y, λ) = ⎨ ⎪ ⎪ ⎪ y, ⎪ ⎩
λ =/ , λ = .
John and Draper () presented the so-called modulus transformation: ⎧ ⎪ (∣y∣ + ) λ − ⎪ ⎪ , λ =/ , ⎪ sign(y) λ ψ(y, λ) = ⎨ ⎪ ⎪ ⎪ sign(y) log(∣y∣ + ), λ = , ⎪ ⎩ where
⎧ ⎪ ⎪ ⎪ y ≥ , ⎪ , sign(y) = ⎨ ⎪ ⎪ ⎪ −, y < . ⎪ ⎩ Bickel and Doksum () suggested another modification: λ ψ(y, λ) = ∣y∣ sign(y) − , λ
λ =/ .
Yeo and Johnson () proposed another power transformation family motivated by the above modified modulus transformation:
⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ψ(y, λ) = ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩
(y + ) λ − , λ log(y + ),
B
y ≥ , λ =/ , y ≥ , λ = ,
( − y)−λ − , y < , λ =/ , − −λ − log( − y), y < , λ = .
The main objective in the analysis using Box–Cox transformation is to make inference on the transformation parameter λ. Box and Cox () applied the maximum likelihood as well as Bayesian methods for estimating the transformation parameter, but there have been many approaches to other inferences including hypothesis testing on the transformation parameter (see Sakia for details, which gave a comprehensive review on the Box– Cox transformation). The Box–Cox transformation can be applied to a regressor, a combination of regressors, and/or to the response variable in a linear or nonlinear regression. For example let us consider the following linear functional form: q
ψ BC (y, λ ) = β + ∑ β j ψ BC (xj , λ j ) + ε, j=
where ψ(y, λ ) and ψ(xj , λ j ) represent the transformed response variable and explanatory variables, respectively, where λ j (j = , , . . . , p) are the transformation parameters, and ε represents the errors. When using such a functional form with the Box–Cox transformation, it is helpful to explore the underlying relationship in cases where the determination of the functional form need not be based on a priori rationale in any research field. In addition, for example, for nonlinear regressions (See 7Nonlinear Regression) we can consider the following form (see Carroll and Ruppert ): ψ BC (y, λ) = ψ BC (f (x, β), λ) + ε, where f (x, β) is a functional form (possibly, corresponding to a theoretical model) that has explanatory variables x and is nonlinear with respect to a real or vector valued parameter β. It is noted that the both sides in the above equation have the same Box–Cox transformation. Thus the objective here is to reduce 7heteroscedasticity and autocorrelation of the error structure as well as non-normality of the error (or residual) itself, rather than the determination of the functional form as above mentioned. The Box–Cox transformation has been widely utilized. However, when using it we note that this transformation seldom fulfills the basic assumptions required for statistical
B
B
Box–Jenkins Time Series Models
inference such as linearity, normality and homoscedasticity simultaneously as originally suggested by Box and Cox ().
About the Author Dr. Takashi Daimon was an Assistant Professor of the Division of Drug Evaluation and Informatics, School of Pharmaceutical Sciences, University of Shizuoka. Currently he is the Chief of Data Center and a Specially Appointed Lecturer of the Medical Center for Translational Research at Osaka University Hospital. He is an Elected Member of the International Statistical Institute and an Associate Editor of the Japanese Journal of Biometrics.
Cross References 7Preprocessing in Data Mining 7Skewness 7Statistical Fallacies: Misconceptions, and Myths 7Statistical Quality Control
References and Further Reading Bickel PJ, Doksum KA () An analysis of transformations revisited. J Am Stat Assoc :– Box GEP, Cox DR () An analysis of transformations. J Roy Stat Soc, Ser B :– Carroll RJ, Ruppert D () Transformations and weighting in regression. Chapman & Hall, London John JA, Draper NR () An alternative family of transformations. Appl Stat :– Manly BF () Exponential data transformation. Statistician :– Sakia RM () The Box–Cox transformation technique: a review. Statistician :– Tukey JW () The comparative anatomy of transformations. Ann Math Stat :– Yeo I-K, Johnson RA () A new family of power transformations to improve normality or symmetry. Biometrika ():–
Box–Jenkins Time Series Models John Boland Associate Professor University of South Australia, Adelaide, SA, Australia
that we will undertake is the Box–Jenkins (BJ) Methodology, which describes an iterative process for identifying a model and then using that model for forecasting. The Box–Jenkins methodology comprises four steps: ● ● ● ●
Identification of process Estimation of parameters Verification of model Forecasting
Identification of Process Assume we have a (at least weakly) stationary time series, i.e., no trend, seasonality, and it is homoscedastic (constant variance). Stationarity will be discussed further in section Stationarity. The general form of an ARMA model is Xt − ϕ Xt− − ⋯ − ϕ p Xt−p = Zt + θ Zt− + ⋯ + θ q Zt−q , () where {Xt } are identically distributed random variables ∼(, σX ) and {Zt } are white noise, i.e., independent and identically distributed (iid) ∼(, σZ ). ϕ i and θ j are the coefficients of polynomials satisfying ϕ(y) = − ϕ y − ⋯ − ϕ p y p
()
q
θ(y) = + θ y + ⋯ + θ q y , where ϕ(y), θ(y) are the autoregressive and moving average polynomials respectively. Define the backward shift operator Bj Xt = Xt−j , j = , , , . . . and we may then write () in the form ϕ(B)Xt = θ(B)Zt () defining an ARMA(p, q) model. If ϕ(B) = , we then have a moving average model of order q, designated MA(q). Alternatively, if we have θ(B) = , we have an autoregressive model of order p, designated AR(p). The question is, how do we identify whether we have an MA(q), AR(p) or ARMA(p, q)? To do so, we can examine the behavior of the autocorrelation and partial autocorrelation functions.
Autocorrelation and Partial Autocorrelation Functions We need some definitions to begin with. Suppose two variables X and Y have means µ X , µ Y respectively. Then the covariance of X and Y is defined to be Cov(X, Y) = E{(X − µ X )(Y − µ Y )}.
()
If X and Y are independent, then
Introduction We are going to examine the Autoregressive Moving Average (ARMA) process for identifying the serial correlation attributes of a stationary time series (see Boland ; Box and Jenkins ). Another name for the processes
Cov(X, Y) = E{(X − µ X )(Y − µ Y )} = E(X − µ X )E(Y − µ Y ) = .
()
If X and Y are not independent, then the covariance may be positive or negative, depending on whether high
Box–Jenkins Time Series Models
values of X tend to happen coincidentally with high or low values of Y. It is usual to standardise the covariance by dividing by the product of their respective standard deviations, creating the correlation coefficient. If X and Y are random variables for the same stochastic process at different times, then the covariance coefficient is called the autocovariance coefficient, and the correlation coefficient is called the autocorrelation coefficient. If the process is stationary, then the standard deviations of X and Y will be the same, and their product will be the variance of either. Let {Xt } be a stationary time series. The autocovariance function (ACVF) of {Xt } is γ X (h) = Cov(Xt+h , Xt ), and the autocorrelation function (ACF) of {Xt } is
ρ X (h) =
γ X (h) = Corr(Xt+h , Xt ). γ X ()
()
The autocovariance and autocorrelation functions can be estimated from observations of X , X , . . . , Xn to give the sample autocovariance function (SAF) and the sample autocorrelation function (SACF), the latter defined by
⎡ ⎢ ⎢ ⎢ ⎢ ⎢ r ⎢ ⎢ ⎢ ⎢ ⎢ . ⎢ ⎢ ⎢ ⎢ . ⎢ ⎢ ⎢ ⎢ ⎢rk− ⎣
rk =
t= n
.
()
∑(xt − x) t=
Thus the SACF is a measure of the linear relationship between time series separated by some time period, denoted by the lag k. Similar to the correlation coefficient of linear regression, rk will take a value between + and −, and the closer to ±, the stronger the relationship. What relationship are we talking about? Consider a lag value close to + as an example. This means that there is a strong relationship between Xt and Xt− , Xt− and Xt− , . . . , Xt−k+ and Xt−k , and so on. The interesting thing is that what can happen in practice is that because of this serial correlation, it can appear that Xt has a strong relationship with Xt−k , k time units away from Xt , when in fact it is only because of this interaction. To sort out this potential problem, one estimates the partial autocorrelation function (PACF). The partial autocorrelation between Xt and Xt−k is the correlation between them after their mutual linear dependency on the intervening variables Xt− , . . . , Xt−k+ has been removed. The sample PACF (SPACF) is given by the Yule–Walker equations,
⋯
rk−
r
⋯
rk−
.
.
⋯
.
.
.
⋯
.
rk−
rk−
⋯
r
⎤⎡ ˆ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ rk− ⎥ ⎥ ⎢ ϕ k ⎥ ⎢ r ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ rk− ⎥ ⎥ ⎢ϕˆk ⎥ ⎢r ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥ = ⎢ ⎥ . () . ⎥ ⎥⎢ . ⎥ ⎢ . ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ . ⎥ ⎥⎢ . ⎥ ⎢ . ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎥⎢ˆ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎥ ⎦ ⎣ϕ kk ⎦ ⎣rk ⎦
Once we have calculated these estimates for a stationary time series, we can use them to give an indication whether we should fit an AR(p), MA(q), or ARMA(p, q) model. The criteria are in general: ●
●
∑ (xt − x)(xt+k − x)
r
The value of ϕˆkk gives the estimate of the PACF at lag k. These equations can be solved using Cramer’s Rule to obtain: ˆ rm − ∑m− j= ϕ m−,j rm−j . () ϕˆmm = ˆ − ∑m− j= ϕ m−,j rj
● n−k
r
B
When the SACF dies down gradually and the SPACF has insignificant spikes at lags greater than p we should fit an AR(p). When the SACF has a significant spike at lag q and the SPACF dies down gradually, we should fit an MA(q). If both die down gradually, we fit an ARMA(p, q). In this case, we will have to progressively increase p, q until we get a suitable model.
The last point brings up an interesting question; how do we decide between competing models? In fact, the situation is often not as simple as these criteria make it seem. Sometimes it is difficult to decide between for instance, an AR() and an ARMA(, ) model. An aid in identifying the appropriate model comes from the principle of parsimony, using criteria from Information Theory. The 7Akaike’s Information Criterion (AIC) is one such measure (Akaike ). The goal is to pick the model that minimises AIC = − { ln(likelihood) + l}. T
()
Here, l is the number of parameters fitted and T the number of data values. There is a competing criterion, that penalises the number of parameters fitted even more, called the (Schwarz) Bayesian Information Criterion (BIC) (Schwarz ), BIC = −
l ln(T) ln(likelihood) + . T T
()
B
B
Box–Jenkins Time Series Models
Moving Average Process In a moving average process MA(q), the present value of the series is written as the weighted sum of past shocks: Xt = Zt + θ Zt− + θ Zt− + ⋯ + θ q Zt−q , where Zt ∼
()
WN(, σZ ).
We find immediately that E(Xt ) = , Var(Xt ) =
If the characteristic equation has a unit root of multiplicity r then the process is integrated of order r, denoted I(r). The time series differenced r times will be stationary. The process defined by Xt = Xt− + Zt has a unit root and this process defines a 7random walk. The process is customarily started at zero when t = , so X = Z , and Xt = ∑ti= Zi , so we obtain Var(Xt ) = tσZ , dependent on t.
Conditional Heteroscedastic Modelling
q σZ ( + ∑ θ ). i=
()
Autoregressive Process The general form of an AR(p) process is: ϕ(B)Xt = Zt ( − ϕ B − ϕ B − ⋯ − ϕ p Bp )Xt = Zt Xt = ϕ Xt− + ϕ Xt− + ⋯ + ϕ p Xt−p + Zt . A first order autoregressive process AR() is referred to as a Markov Chain (see 7Markov Chains). It can, through successive substitutions, be written as: Xt = ϕXt− + Zt
There is one particular type of non-stationarity that is receiving increasing attention. In financial markets, and other applications such as modelling wind farm output (Boland et al. ), a phenomenon has been identified wherein the SACF of Zt shows no residual autocorrelation, but the SACF of Zt does. This property means that the noise is uncorrelated but not independent. This is reflective of a process that retains conditional heteroscedasticity, wherein there is what is termed as volatility clustering. Periods of high volatility can be followed by periods of low volatility. The volatility is generally modelled by Autoregressive Conditional Heteroscedastic (ARCH) or Generalised ARCH (GARCH) models, or utilising a concept called realised volatility (see Tsay and references therein).
= ϕ(ϕXt− + Zt− ) + Zt = Zt + ϕZt− + ϕ Zt− + ϕ Zt− + ⋯. i From this we write Var(Xt ) = σZ (+∑∞ i= ϕ ) =
About the Author σZ
−
ϕ
.
Any AR(p) process can be rewritten as an infinite order moving average process.
Stationarity If neither the mean µ t nor the autocovariances γ X (h) are dependent on t, then the process is said to be weakly stationary. This means that the autocovariances depend only on the length of time separating the observations h. A process is strictly stationary if for any values of h , h , . . . , hn , the joint distribution of (Xt , Xt+h , Xt+h , . . . , Xt+hn ) depends not on t, but only on the time intervals between the variables. A sufficient condition for negating weak stationarity (and thus strict stationarity) is failure of the unit root test. A stochastic process has a unit root if its characteristic equation has as a root. For Xt = ϕ Xt− + ϕ Xt− + ⋯ + ϕ p Xt−p + Zt , the characteristic equation is given by: mp − ϕ mp− − ϕ mp− − ⋯ − ϕ p− m − ϕ p = . If m = is a root of this characteristic equation, the process has a unit root or is termed integrated of order , denoted I(). A first difference of the time series will be stationary.
Associate Professor John Boland has extensive expertise in environmental modelling, specialising in time series and statistical modelling of climate variables and modelling systems under uncertainty. He has published papers in many areas of environmental modelling and is Associate Editor for two international journals: Renewable Energy, and Case Studies in Business, Industry and Government Statistics. He, along with colleagues, is present or past holder of three Australian Research Council (ARC) Linkage and three Discovery grants. One Linkage and two Discovery Grants have been in the area of water cycle management. The present one in this area is focusing on using advanced mathematical techniques to better understand the structure of the Murray Darling Basin and its management options. Presently, he holds a Linkage and a Discovery grant in the energy area, focusing on the future of the electricity supply network with significant increases in the penetration of renewable energy into the grid. All these projects focus on stochastic modelling and risk management, particularly utilising conditional value at risk. His expertise in modelling systems under uncertainty has led him to being selected to run a one day Masterclass in Managing Data for Energy Storage and Renewable Energy Supply at the Energy Storage Forum, Beijing, March .
Brownian Motion and Diffusions
Cross References 7Business Forecasting Methods 7Forecasting with ARIMA Processes 7Forecasting: An Overview 7Intervention Analysis in Time Series 7Mathematical and Statistical Modeling of Global
Warming 7Moving Averages 7Seasonality 7Structural Time Series Models 7Time Series
References and Further Reading Akaike H () Information theory and an extension of the maximum likelihood principle. In Petrov BN, Csaki F (eds) Second international symposium on information theory. Akademia Kiado, Budapest, pp – Boland J, Gilbert K, Korolkowicz M (– December ) Modelling wind farm output variability. MODSIM, Christchurch, New Zealand Boland J () Time series and statistical modeling of solar radiation. In Badescu V (ed) Recent advances in solar radiation modeling. Springer, Berlin, pp – Box G, Jenkins G () Time series analysis: forecasting and control. Holden-Day, San Francisco, CA Schwarz G () Estimating the dimension of a model. Ann Stat :– Tsay RS () Analysis of financial time series, nd edn. Wiley, New York
B
the assumption that the three components determining its motion in space are independent. Definition A real-valued process (Wt )t∈IR+ is a Wiener process if it satisfies the following conditions: . W = almost surely. . (Wt )t∈IR+ is a process with independent increments. . Wt − Ws is normally distributed with N(, t − s), ( ≤ s < t). Remark From point of Definition it becomes obvious that every Wiener process is time homogeneous. Proposition
If (Wt )t∈IR+ is a Wiener process, then
. E[Wt ] = for all t ∈ IR+ , . K(s, t) = Cov[Wt , Ws ] = min{s, t},
s, t ∈ IR+ .
Proof . Fixing t ∈ IR, we observe that Wt = W + (Wt − W ) and thus E[Wt ] = E[W ] + E[Wt − W ] = . The latter is given by the fact that E[W ] = (by of Definition ) and E[Wt − W ] = (by of Definition ). . Let s, t ∈ IR+ and Cov[Wt , Ws ] = E[Wt Ws ] − E[Wt ]E[Ws ], which (by point ) gives Cov[Wt , Ws ] = E[Wt Ws ]. For simplicity, if we suppose that s < t, then E[Wt Ws ] = E[Ws (Ws + (Wt − Ws ))] = E [Ws ] +E[Ws (Wt − Ws )]. Since (Wt )t∈IR+ has independent increments, we obtain E[Ws (Wt − Ws )] = E[Ws ]E[Wt − Ws ]
Brownian Motion and Diffusions Vincenzo Capasso Professor of Probability and Mathematical Statistics University of Milan, Milan, Italy
Brownian Motion and the Wiener Process A small particle (e.g., a pollen corn) suspended in a liquid is subject to infinitely many collisions with atoms, and therefore it is impossible to observe its exact trajectory. With the help of a microscope it is only possible to confirm that the movement of the particle is entirely chaotic. This type of movement, discovered under similar circumstances by the botanist Robert Brown, is called Brownian motion. As its mathematical inventor Einstein already observed, it is necessary to make approximations, in order to describe the process. The formalized mathematical model defined on the basis of these is called a Wiener process. Henceforth, we will limit ourselves to the study of the one-dimensional Wiener process in IR, under
and by of Definition it follows that this is equal to zero, thus Cov[Wt , Ws ] = E [Ws ] = Var[Ws ]. If we now observe that Ws = W + (Ws − W ) and hence Var[Ws ] = Var[W + (Ws − W )], then, by the independence of the increments of the process, we get Var[W + (Ws − W )] = Var[W ] + Var[Ws − W ]. Therefore, by points and of Definition it follows that Var[Ws ] = s = inf{s, t}, which completes the proof. Remark By of Definition , it follows, for all t ∈ IR+ , Wt = Wt − W almost surely and by of the same definition, that Wt is distributed as N(, t). Thus b −x P(a ≤ Wt ≤ b) = √ ∫ e t dx, πt a
a ≤ b.
Remark The Wiener process is a Gaussian process. In fact, if n ∈ IN∗ , (t , . . . , tn ) ∈ IR+n with = t < t < . . . < tn
B
B
Brownian Motion and Diffusions
and (a , . . . , an ) ∈ IRn , (b , . . . , bn ) ∈ IRn , such that ai ≤ bi , i = , , . . . , n, it can be shown that P(a ≤ Wt ≤ b , . . . , an ≤ Wtn ≤ bn ) bn
b
g(∣x , t )g(x ∣x , t − t )
⋯∫
=∫
()
an
a
⋯g(xn− ∣xn , tn − tn− )dxn ⋯dx , where
Proposition If (Wt )t∈IR+ is a Wiener process, then it is also a martingale (see 7Martingales). Proof The proposition follows from the fact that (Wt )t∈IR+ is a zero mean process with independent increments.
∀h > ,
P (max Ws > ) = P ( min Ws < ) = . ≤s≤h
≤s≤h
Moreover, for almost every ω ∈ Ω the process (Wt )t∈IR+ has a zero (i.e., crosses the spatial axis) in [, h], for all h > .
Proposition (scaling property). Let (Wt )t∈IR+ be a ˜ t )t∈IR+ Wiener process. Then the time-scaled process (W defined by ˜ t = tW/t , t > , W
˜= W
is also a Wiener process. Every Wiener process (Wt )t∈IR+ is a Markov
Proof The theorem follows directly from the fact that (Wt )t∈IR+ is a process with independent increments. Remark Since Brownian motion is continuous in probability, it admits a separable and progressively measurable modification. Theorem (Kolmogorov’s continuity theorem). Let (Xt )t∈IR+ be a separable real-valued stochastic process. If there exist positive real numbers r, c, є, δ such that ∀h < δ, ∀t ∈ IR+ ,
E[∣Xt+h − Xt ∣r ] ≤ ch+є ,
()
then, for almost every ω ∈ Ω, the trajectories are continuous in IR+ . Theorem If (Wt )t∈IR+ is a real-valued Wiener process, then it has continuous trajectories almost surely. Proof Let t ∈ IR+ and h > . Because Wt+h − Wt is nor−W t √ , Zt,h mally distributed as N(, h), putting Zt,h = Wt+h h has standard normal distribution. Therefore, it is clear that there exists an r > such that E[∣Zt,h ∣r ] > , and thus r E[∣Wt+h − Wt ∣r ] = E[∣Zt,h ∣r ]h . If we write r = ( + є), we obtain E[∣Wt+h − Wt ∣r ] = ch+є , with c = E[∣Zt,h ∣r ]. The assertion then follows by Kolmogorov’s continuity theorem. Theorem then
If (Wt )t∈IR+ is a real-valued Wiener process,
Theorem Almost every trajectory of the Wiener process (Wt )t∈IR+ is differentiable almost nowhere.
∣x−y∣
e− t . g(x∣y, t) = √ πt
Theorem process.
Theorem then,
If (Wt )t∈IR+ is a real-valued Wiener process,
. P(supt∈IR+ Wt = +∞) = . P(inf t∈IR+ Wt = −∞) =
Proposition (Strong law of large numbers). Let (Wt )t∈IR+ be a Wiener process. Then Wt → , t
as
t → +∞,
a.s.
Proposition (Law of iterated logarithms). Let (Wt )t∈IR+ be a Wiener process. Then lim sup √
= ,
t ln ln t Wt
t→+∞
lim inf √ t→+∞
Wt
= −,
a.s., a.s.
t ln ln t
As a consequence, for any є > , there exists a t > , such that for any t > t we have √ √ −( + є) t ln ln t ≤ Wt ≤ ( + є) t ln ln t,
a.s.
Existence of the Wiener process is guaranteed by the following fundamental theorem (see, e.g., Billingsley ). Theorem (Donsker). Let ξ , ξ , . . . , ξ n , . . . be a sequence of independent identically distributed random variables defined on a common probability space (Ω, F, P), with mean and finite, positive variance σ : E[ξ n ] = ,
E [ξ n ] = σ .
Let S = and, for any n ∈ IN∖{}, let Sn = ξ + ξ +⋯+ ξ n . Then the sequence of random processes defined by Xn (t, ω) = √ S[nt] (ω) + (nt − [nt]) √ ξ[nt]+ (ω) σ n σ n for any t ∈ IR+ , ω ∈ Ω, n ∈ IN, weakly converges to a Wiener process.
Brownian Motion and Diffusions
Diffusion Processes Markov Processes
Definition Let (Xt )t∈IR+ be a stochastic process on a probability space, valued in (E, B) and adapted to the increasing family (Ft )t∈IR+ of σ-algebras of subsets of F. (Xt )t∈IR+ is a Markov process with respect to (Ft )t∈IR+ if the following condition is satisfied: ∀B ∈ B, ∀(s, t) ∈ IR+ × IR+ , s < t :
P(Xt ∈ B∣Fs ) = P(Xt ∈ B∣Xs ) a.s. ()
Remark If, for all t ∈ IR+ , Ft = σ(Xr , ≤ r ≤ t), then condition () becomes P(Xt ∈ B∣Xr , ≤ r ≤ s) = P(Xt ∈ B∣Xs ) a.s. for all B ∈ B, for all (s, t) ∈ IR+ × IR+ , and s < t. Proposition Under the assumptions of Definition , the following two statements are equivalent: . For all B ∈ B and all (s, t) ∈ IR+ × IR+ , s < t : P(Xt ∈ B∣Fs ) = P(Xt ∈ B∣Xs ) almost surely. . For all g : E → IR, B-BIR -measurable such that g(Xt ) ∈ L (P) for all t, for all (s, t) ∈ IR+ , s < t : E[g(Xt )∣Fs ] = E[g(Xt )∣Xs ] almost surely.
(s, t) ∈ IR+ , s < t, for all A ∈ BIR , and for all x ∈ IR, we obtain p(s, x, t, A) = p(, x, t − s, A) a.s. We may then define a one-parameter transition function p(¯t , x, A) := p(, x, t − s, A) with ¯t = (t − s), x ∈ IR, A ∈ BIR .
Semigroups Associated with Markov Transition Probability Functions Let BC(IR) be the space of all continuous and bounded functions on IR, endowed with the norm ∥ f ∥ = supx∈IR ∣ f (x)∣ (< ∞), and let p(s, x, t, A) be a transition probability function ( ≤ s < t ≤ T, x ∈ IR, A ∈ BIR ). We consider the operator Ts,t : BC(IR) → BC(IR),
p(s, x, t, A) = P(Xt ∈ A∣Xs = x),
≤ s < t < ∞, x ∈ IR, A ∈ BIR .
p is a Markov transition probability function, i.e., it is a nonnegative function defined for ≤ s < t < ∞, x ∈ IR, A ∈ BIR , which satisfies the following properties . For all ≤ s < t < ∞, for all A ∈ BIR , p(s, ⋅, t, A) is BIR -measurable. . For all ≤ s < t < ∞, for all x ∈ IR, p(s, x, t, ⋅) is a probability measure on BIR . . p satisfies the Chapman–Kolmogorov equation:
(Ts,t f )(x) = ∫ f (y)p(s, x, t, dy). IR
If s = t, then ⎧ ⎪ ⎪ ⎪ if x ∈ A, p(s, x, s, A) = ⎨ ⎪ ⎪ if x ∉ A. ⎪ ⎩ Therefore, Tt,t = I (identity).
IR
Ts,t Tt,u = Ts,u ,
≤ s < t < u.
()
In fact, if f ∈ BC(IR) and x ∈ IR, (Ts,t (Tt,u f ))(x) = ∫ (Tt,u f )(y)p(s, x, t, dy) IR
=∫ ∫
f (z)p(t, y, u, dz)p(s, x, t, dy)
= ∫ f (z) ∫ p(t, y, u, dz)p(s, x, t, dy) IR
Definition A Markov process (Xt )t∈[t ,T] is said to be homogeneous if the transition probability functions p(s, x, t, A) depend on t and s only through their difference t − s. Therefore, for all (s, t) ∈ [t , T] , s < t, for all u ∈ [, T − t], for all A ∈ BIR , and for all x ∈ IR:
()
Moreover, we have that
IR
p(s, x, t, A) = ∫ p(s, x, r, dy)p(r, y, t, A) ∀x ∈ IR, s < r < t.
≤ s < t ≤ T,
defined by assigning, for all f ∈ BC(IR),
Theorem Every real stochastic process (Xt )t∈IR+ with independent increments is a Markov process. Proposition Consider a real valued Markov process (Xt )t∈IR+ , and let
B
IR
(by Fubini’s theorem) = ∫ f (z)p(s, x, u, dz) IR
(by the Chapman–Kolmogorov equation) = (Ts,u f )(x).
p(s, x, t, A) = p(s + u, x, t + u, A)
a.s.
Remark If (Xt )t∈IR+ is a homogeneous Markov process with transition probability function p, then, for all
Definition The family {Ts,t }≤s≤t≤T is a semigroup associated with the transition probability function p(s, x, t, A) (or with its corresponding Markov process).
B
B
Brownian Motion and Diffusions
Definition If (Xt )t∈IR+ is a Markov process with transition probability function p and associated semigroup {Ts,t }, then the operator Ts,s+h f − f , As f = lim h↓ h
s ≥ , f ∈ BC(IR)
is called the infinitesimal generator of the Markov process (Xt )t≥ . Its domain DAs consists of all f ∈ BC(IR) for which the above limit exists uniformly (and therefore in the norm of BC(IR)) (see, e.g., Feller ). Remark From the preceding definition we observe that (As f )(x) = lim h↓
[f (y) − f (x)]p(s, x, s + h, dy). h ∫IR
Definition Let (Xt )t∈IR+ be a Markov process with transition probability function p(s, x, t, A), and {Ts,t } (s, t ∈ IR+ , s ≤ t) its associated semigroup. If, for all f ∈ BC(IR), the function (t, x) ∈ IR+ ×IR → (Tt,t+λ f )(x) =∫ p(t, x, t+λ, dy)f (y) ∈ IR
for any g ∈ BC(IR). The infinitesimal generator will be time independent. It is defined as Ag = lim (T(t)g − g) t→+ t for g ∈ D(A), the subset of BC(IR) for which the above limit exists, in BC(IR), with respect to the sup norm. Given the above definitions, it is obvious that for all g ∈ D(A), Ag(x) = lim E[g(Xt )∣X = x], t→+ t
If (T(t), t ∈ IR+ ) is the contraction semigroup associated with a time-homogeneous Markov process, it is not difficult to show that the mapping t → T(t)g is rightcontinuous in t ∈ IR+ provided that g ∈ BC(IR) is such that the mapping t → T(t)g is right continuous in t = . Then, for all g ∈ D(A) and t ∈ IR+ , t
T(s)gds ∈ D(A)
∫
and
IR
t
t
is continuous for all λ > , then we say that the process satisfies the Feller property. Theorem If (Xt )t∈IR+ is a Markov process with rightcontinuous trajectories satisfying the Feller property, then, for all t ∈ IR+ , Ft = Ft+ , where Ft+ = ⋂t′ >t σ(X(s), ≤ s ≤ t ′ ), and the filtration (Ft )t∈IR+ is right-continuous. Remark It can be shown that Ft+ is a σ-algebra. Example Wiener processes are processes with the Feller property, or simply Feller processes. If we consider the time-homogeneous case, a Markov process (Xt )t∈IR+ on (IR, BIR ), will be defined in terms of a transition kernel p(t, x, B) for t ∈ IR+ , x ∈ IR, B ∈ BIR , such that
x ∈ IR.
T(s)gds = ∫
T(t)g − g = A ∫
AT(s)gds
t
T(s)Agds
=∫
by considering Riemann integrals. The following, so-called Dynkin’s formula, establishes a fundamental link between 7Markov processes and 7martingales (see Rogers and Williams , p ). Theorem Assume (Xt )t∈IR+ is a time-homogeneous Markov process on (IR, BIR ), with transition kernel p(t, x, B), t ∈ IR+ , x ∈ IR, B ∈ BIR . Let (T(t), t ∈ IR+ ) denote its transition semigroup and A its infinitesimal generator. Then, for any g ∈ D(A), the stochastic process t
M(t) := g(Xt ) − g(X ) − ∫
Ag(Xs )ds
p(h, Xt , B) = P(Xt+h ∈ B∣Ft )
∀t, h ∈ IR+ , B ∈ BIR ,
is an Ft -martingale.
given that (Ft )t∈IR+ is the natural filtration of the process. Equivalently, if we denote by BC(IR) the space of all continuous and bounded functions on IR, endowed with the sup norm,
The next proposition shows that a Markov process is indeed characterized by its infinitesimal generator via a martingale problem (see, e.g., Rogers and Williams , p ).
E[g(Xt+h )∣Ft ] =∫ g(y)p(h, Xt , dy) ∀t, h ∈ IR+ , g ∈ BC(IR).
Theorem (Martingale problem for Markov processes). If an RCLL (right continuous with left limits) Markov process (Xt )t∈IR+ is such that
E
In this case the transition semigroup of the process is a one-parameter contraction semigroup (T(t), t ∈ IR+ ) on BC(IR) defined by T(t)g(x) := ∫ g(y)p(t, x, dy) = E[g(Xt )∣X = x], x ∈ IR, E
t
g(Xt ) − g(X ) − ∫
Ag(Xs )ds
is an Ft -martingale for any function g ∈ D(A), where A is the infinitesimal generator of a contraction semigroup on
Business Forecasting Methods
E, then Xt is equivalent to a Markov process having A as its infinitesimal generator.
Markov Diffusion Processes Definition A Markov process on IR with transition probability function p(s, x, t, A) is called a diffusion process if . For all є > , for all t ≥ , and for all x ∈ IR : limh↓ h ∫∣x−y∣>є p(t, x, t + h, dy) = . . There exist a(t, x) and b(t, x) such that, for all є > , for all t ≥ , and for all x ∈ IR, (y − x)p(t, x, t + h, dy) = a(t, x), h ∫∣x−y∣<є (y − x) p(t, x, t + h, dy) = b(t, x). lim ∫ h↓ h ∣x−y∣<є lim h↓
a(t, x) is the drift coefficient and b(t, x) the diffusion coefficient of the process. Proposition If (Xt )t∈IR+ is a diffusion process with transition probability function p and drift and diffusion coefficients a(x, t) and b(x, t), respectively, and if As is the infinitesimal generator associated with p, then we have that ∂f ∂ f b(s, x), () a(s, x) + ∂x ∂x provided that f is bounded and twice continuously differentiable. (As f )(x) =
Proposition A Wiener process is a time homogeneous diffusion process, with drift zero and diffusion coefficient equal to . Its infinitesimal generator is then (Af )(x) =
∂ f , ∂x
About the Author For biography see the entry 7Axioms of Probability.
Cross References 7Almost Sure Convergence of Random Variables 7Central Limit Theorems 7First Exit Time Problem 7Itô Integral 7Khmaladze Transformation 7Lévy Processes 7Markov Processes 7Non-Uniform Random Variate Generations 7Random Walk 7Statistical Inference for Stochastic Processes
B
7Statistical Modeling of Financial Markets 7Stochastic Difference Equations and Applications 7Stochastic Processes 7Stochastic Processes: Applications in Finance and
Insurance 7Stochastic Processes: Classification 7Strong Approximations in Probability and Statistics 7Testing Exponentiality of Distribution
References and Further Reading Ash RB, Gardner MF () Topics in stochastic processes. Academic, London Billingsley P () Convergence of probability measures. Wiley, New York Breiman L () Probability. Addison-Wesley, Reading, MA Capasso V, Bakstein D () An introduction to continuous-time stochastic processes theory, models, and applications to finance, biology, and medicine, Birkhäuser, Boston, MA Doob JL () Stochastic processes. Wiley, New York Dynkin EB () Markov processes, vols –. Springer, Berlin Feller W () An introduction to probability theory and its applications. nd edn. vol . Wiley, New York Friedman A () Stochastic differential equations and applications. Academic, London Gihman II, Skorohod AV (–) The theory of stochastic processes, vols –. Springer, Berlin Ikeda N, Watanabe S () Stochastic differential equations and diffusion processes. North-Holland, Kodansha Karatzas I, Shreve SE () Brownian motion and stochastic calculus. Springer, New York Karlin S, Taylor HM () A first course in stochastic processes. Academic, New York Karlin S, Taylor HM () A second course in stochastic processes. Academic, New York Rogers LCG, Williams D () Diffusions, Markov processes and martingales, vol . Wiley, New York Taira K () Diffusion processes and partial differential equations. Academic, New York
()
Business Forecasting Methods Rob J. Hyndman Professor of Statistics Monash University, Melbourne, VIC, Australia
Forecasting, Planning, and Goals Forecasting is a common statistical task in business where it helps inform decisions about scheduling of production, transportation and personnel, and provides a guide to long-term strategic planning. However, business forecasting is often done poorly and is frequently confused with planning and goals. They are three different things.
B
B
Business Forecasting Methods
Forecasting is about predicting the future as accurately as possible, given all the information available including historical data and knowledge of any future events that might impact the forecasts. Goals are what you would like to happen. Goals should be linked to forecasts and plans, but this does not always occur. Too often, goals are set without any plan for how to achieve them, and no forecasts for whether they are realistic. Planning is a response to forecasts and goals. Planning involves determining the appropriate actions that are required to make your forecasts match your goals.
●
Forecasting should be an integral part of the decision-making activities of management, as it can play an important role in many areas of a company. Modern organizations require short-, medium-, and long-term forecasts, depending on the specific application. Short-term forecasts are needed for scheduling of personnel, production, and transportation. As part of the scheduling process, forecasts of demand are often also required. Medium-term forecasts are needed to determine future resource requirements in order to purchase raw materials, hire personnel, or buy machinery and equipment. Long-term forecasts are used in strategic planning. Such decisions must take account of market opportunities, environmental factors, and internal resources. An organization needs to develop a forecasting system involving several approaches to predicting uncertain events. Such forecasting systems require the development of expertise in identifying forecasting problems, applying a range of forecasting methods, selecting appropriate methods for each problem, and evaluating and refining forecasting methods over time. It is also important to have strong organizational support for the use of formal forecasting methods if they are to be used successfully.
observed value: yˆ n+h∣n = yn , h = , , . . . . If the data follow a random walk process (yt = yt− + et where et is white noise – a series of iid random variables with zero mean), then this is the optimal method of forecasting. Consequently, it is popular for stock price and stock index forecasting, and for other time series that measure the behavior of a market that can be assumed to be efficient. Simple exponential smoothing was developed in the s (Brown ) and has been widely used ever since. Forecasts can be computed recursively as each new data point is observed: yˆ t+∣t = αyt + ( − α)ˆyt∣t− ,
●
where < α < . (Longer-term forecasts are constant: yˆ t+h∣t = yˆ t+∣t , h ≥ .) Consequently, only the most recent data point and most recent forecast need to be stored. This was an attractive feature of the method when computer storage was expensive. The method has proved remarkably robust to a wide range of time series, and is optimal for several processes including the ARIMA(,,) process (Chatfield et al. ). Holt’s linear method (Holt ) is an extension of simple exponential forecasting that allows a locally linear trend to be extrapolated. Forecasts are given by yˆ t+h∣t = ℓ t + hbt where ℓ t = αyt + ( − α)(ℓ t− + bt− ), bt = β(ℓ t − ℓ t− ) + ( − β)bt− ,
●
and the two parameters α and β must lie in [, ]. Here ℓ t denotes the level of the series and bt the slope of the trend at time t. For seasonal data, a popular method is the Holt–Winters’ method, also introduced in Holt (), which extends Holt’s method to include seasonal terms. Then yˆ t+h∣t = ℓ t + hbt + st−m+h+m where
Commonly Used Methods Typically, businesses use relatively simple forecasting methods that are often not based on statistical modelling. However, the use of statistical forecasting is growing and some of the most commonly used methods are listed below.
Time Series Methods Let the historical time series data be denoted by y , . . . , yn , and the forecast of yn+h be given by yˆ n+h∣n , h > . ●
Naïve forecasting is where the forecasts of all future values of a time series are set to be equal to the last
ℓ t = α(yt − st−m ) + ( − α)(ℓ t− + bt− ), bt = β(ℓ t − ℓ t− ) + ( − β)bt− , st = γ(yt − ℓ t ) + ( − γ)st−m , h+m = [(h − ) mod m] + , and the three parameters α, β and γ all lie in [,]. There is also a multiplicative version of the Holt–Winters’ method, and damped trend versions of both Holt’s linear method and Holt–Winters’ method (Makridakis et al. ). None of these methods are explicitly based on
Business Forecasting Methods
underlying time series models, and as a result the estimation of parameters and the computation of prediction intervals is often not done. However, all the above methods have recently been shown to be optimal for some state space models (Hyndman et al. ), and maximum likelihood estimation of parameters, statistical model selection and computation of prediction intervals is now becoming more widespread. Other time series models sometimes used in business forecasting include ARIMA models, GARCH models (especially in finance), structural models and 7neural networks.
Explanatory Models for Forecasting The use of explanatory models in business forecasting does not have such a long history as the use of time series methods. ●
Linear regression modeling (see 7Linear Regression Models) is now widely used (e.g., Pardoe ) where a variable to be forecast is modeled as a linear combination of potential input variables: J
yt = ∑ cj xj,t + et , j=
●
where et denotes an iid error term with zero mean. An interesting application of regression models to forecasting is given by Byron and Ashenfelter () who use a simple regression models to predict the quality of a Grange wine using simple weather variables. However, it is far more common for regression modelling to be used to explain historical variation than for it to be used for forecasting purposes. In some domains, the use of nonparametric additive models for forecasting is growing (e.g., Hyndman and Fan ). Here, the model is often of the form J
yt = ∑ fj (xj,t ) + et , j=
●
where fj is a smooth nonlinear function to be estimated nonparametrically. In advertising, there is a well-developed culture of using distributed lag regression models (e.g., Hanssens et al. ) such as J
yt = ∑ α λ j xt−j + et , j=
where xt denotes advertising expenditure in month t, < λ < and α > .
B
Data Mining Methods for Business Forecasting Outside of traditional statistical modelling, an enormous amount of forecasting is done using data mining methods (see 7Data Mining). Most of these methods have no formal statistical model, prediction intervals are not computed, and there is limited model checking. But some of the data-mining methods have proven powerful predictors in some contexts, especially when there is a vast quantity of available data. Predictive methods include neural networks, support vector machines, and regression trees. Many of the best-known business predictive algorithms are based on data-mining methods including the prediction of Netflix ratings (see 7Data Mining Time Series Data, Forecasting Principles).
About the Author For biography see the entry 7Forecasting: An Overview.
Cross References 7Business Statistics 7Data Mining Time Series Data 7Exponential and Holt-Winters Smoothing 7Forecasting Principles 7Forecasting: An Overview 7Time Series
References and Further Reading Brown RG () Statistical forecasting for inventory control. McGraw-Hill, New York Byron RP, Ashenfelter O () Predicting the quality of an unborn Grange. Econ Rec ():– Chatfield C, Koehler AB, Ord JK, Snyder RD () A new look at models for exponential smoothing. J Roy Stat Soc, Ser D: Statistician ():– Hanssens DM, Parsons LJ, Schultz RL () Market response models: econometric and time series analysis, nd edn. Kluwer Academic, Boston, MA Holt CC () Forecasting trends and seasonals by exponentially weighted averages, O.N.R. Memorandum /, Carnegie Institute of Technology Hyndman RJ, Fan S () Density forecasting for long-term peak electricity demand, IEEE Transactions on Power Systems, ():– Hyndman RJ, Koehler AB, Ord JK, Snyder RD () Forecasting with exponential smoothing: the state space approach. Springer, Berlin Makridakis S, Wheelwright SC, Hyndman RJ () Forecasting: methods and applications, rd edn. Wiley, New York Pardoe I () Applied regression modeling: a business approach, Wiley, Hoboken, NJ
B
B
Business Intelligence
Business Intelligence Jasna Soldic-Aleksic, Rade Stankic Professors, Faculty of Economics Belgrade University, Belgrade, Serbia There is no unique definition of the term and concept of Business Intelligence (BI) adopted both in the academic community and commercial business circles. However, there are various short and broad definitions that emphasize some specific aspects of this complex concept. In general, the concept of Business Intelligence refers to in-depth analysis of company data for better decision-making. However, more specifically, the concept may be explained as follows: BI is an umbrella term combining architectures, tools, databases, analytical tools, applications, and methodologies for gathering, storing, analyzing, and providing access to data for improving business performance and helping enterprise users make better business and strategic decisions (Turban et al. ). From a historical point of view, the term business intelligence appeared in the late s, but many years later its usage has been widened to cover the sense it is associated with nowadays. The history of BI concept development is closely connected to the evolution of information systems for enterprise decision support. This is the field in which the roots of the BI concept can be identified, i.e., the Management Information System – MIS reporting systems of the s. These systems were characterized by static reporting features without analytical capabilities. The next generation of information systems – the Executive Information System (EIS) that emerged in the early s provided some additional capabilities, such as: adhoc reporting, on-demand reporting, forecasting and prediction, trend analysis, drill-down capability, and critical success factors. These capabilities were essential for computerized support to top-level managers and executives and they were all integrated in the BI system. While the widespread usage of the term and the concept had been recorded in the late s, the rapid growth of BI tools and technologies became evident in s, when many sophisticated data analysis tools were being included in BI enterprise information systems. Considering the broader scene of the main business and technological trends that had an impact on the development of the BI concept, the following are particularly worth noting: globalization, rapid business growth, strong market competition, data explosion and information overload in all spheres of business and ordinary life, user dissatisfaction with fragmented information
systems capabilities, client/server architecture, Enterprise Resource Planning (ERP), Data warehouse (DW) technology, artificial intelligence computing, and web-based technologies and applications. Among all of these technologies, one can be distinguished as the biggest technological catalyst for BI rapid development and success – the DW technology. DW is a repository of subject-oriented, consistent, integrated, historic, and non-volatile collection of data, organized in such way to allow the end-user easy access to data in a form acceptable for analytical processing activities. One of the key features of the DW refers to specific data organization. In contrast to classical data organization approach, with data stored in operational systems (legacy, inventory, shipping, or ERP) and organized according to a concrete business process (purchasing, shipping, billing. . .), data in DW are organized by subject. Special software, called ETL (Extract, Transform and Load) is responsible for extracting data from different sources, cleaning them up and loading them into a data warehouse. Thanks to this feature, DW provides fast retrieval of historical data (allowing users to analyze time series, trends, and historical patterns) and allows users to analyze broader sets of data. In addition to the data warehouse, the architecture of a BI system comprises three other components (Turban et al. ): business analytics as a collection of tools for data query and reporting, online analytical processing, statistical analysis, prediction, data visualization, 7data mining, text and web mining; business performance management (BPM) used for monitoring and analyzing performance through business performance indicators; and a user interface that aims to adequate user communication with the BI system (e.g., dashboard, alerts, and notifications). For analytical purposes, the BI concept assumes a broad usage of statistical software products and machine learning techniques. The combination of statistics and predictive analytics is recognized as one of the crucial trends in business intelligence. So, it is not unusual to observe some of the most famous statistical software products being incorporated in the integrated predictive analytics technology suits or portfolios. The most impressive examples in this sense are the transformation of two statistical software leaders – SPSS (“Statistical Package for the Social Sciences”) and SAS (“Statistical Analysis System”) into remarkable providers of predictive analytics software and services. By using this combined technology, organizations can address their vital BI needs, be it reporting, querying, business visualization, statistics, survey analysis, 7data mining, text and Web analysis, decision optimization, or, very often, a combination of previous capabilities.
Business Statistics
The market of BI products and tools has been on the rise in the last few years and shows a strong tendency for further expansion in the future. According to the Business Intelligence Tools Survey (published in October ), the most renowned providers of BI products are: Oracle, SAP, SAS Institute, IBM, EFM Software, Information Builders, Microsoft, QlikTech, Microstrategy, and Actuate.
Cross References 7Business Statistics 7Statistical Software: An Overview
References and Further Reading Inmon WH () Building the data warehouse, th edn. Wiley, New York Turban E, Arinson JE, Liang T, Sharda R () Decision support and business intelligence systems, th edn. Pearson Prentice Hall, Upper Saddle River, NJ Turban E, Ledner D, McLean E, Wetherbe J () Information technology for management: transforming organizations in the digital economy, th edn. Wiley, New York
Business Statistics Mark L.Berenson Professor Montclair State University, Montclair, NJ, USA
An Overview and Definitions Business statistics can be viewed from two perspectives. One focuses on the use of the statistics themselves. The other sees business statistics as a practitioner-based discipline. To the user, business statistics are intended to be helpful information pertaining to the efficacy of either a company (e.g., financial statements and financial ratios at a particular point in time over several time periods), an industry (e.g., a series of data or an index constructed over time), or the overall economy. The information gleaned is often intended to help the user in making decisions regarding planning, monitoring or investing. To the practitioner, however, business statistics is a particular academic branch of study, similar to other applicationsbased branches of statistics such as agricultural statistics, 7astrostatistics, 7biostatistics, educational statistics, 7medical statistics, psychological statistics and sociological statistics. The operational definition of business statistics adopted herein is the latter.
B
Business statistics is an applied branch of mathematics that transforms data into useful information for decision-making purposes (Berenson et al. a). The applications transcend the various functional areas of business – accounting (e.g., auditing), finance (e.g., portfolio development, 7risk analysis, forecasting, econometric modeling), information systems (e.g., database management, e-commerce analysis), management (e.g., project management, quality and productivity management) and marketing (e.g., consumer behavior, sales forecasting, market segmentation, conjoint analysis). Owing to the wide variety of applications within these areas, business statistics is sometimes referred to as industrial statistics or economic statistics. However, the scope of such nomenclature is too narrow and the more encompassing definition of business statistics is preferred. Industrial statistics typically pertains to quality and productivity whereas economic statistics usually pertains to econometric modeling. The key feature which distinguishes business statistics as a discipline distinct from other subject area applications of statistics is that in business two types of studies can be conducted, enumerative or analytic, and each has its own methodology. To distinguish between these two types of studies, consider a photograph in a frame versus a motion picture film played on a DVD. Whereas an enumerative study is a “snapshot at a single moment in time,” an analytic study is a “motion picture taken over time.” Enumerative studies are common to other applicationbased branches of statistics and various methodologies employed transcend these statistics disciplines. Enumerative studies involve decision making regarding a population and/or its characteristics at a particular point in time. The sampled data collected can be analyzed descriptively and inferentially provided that appropriate probability sampling methods are used in the data collection stage. Inferences drawn through confidence interval estimation and regression modeling would be considered appropriate for that population at that moment in time. For example, a % confidence interval estimate based on a survey conducted today might show that adult males are between % and % more confident than are adult females that there will be improvements in global economic conditions next year, but who would believe that such opinions would remain static over time as events which led to such opinions change? Decisions made based on inferences drawn from enumerative studies hold true only as long as the population frames from which the samples were drawn remain unchanged. On the other hand, analytic studies are specifically applicable to industrial or business situations (Deming
B
B
Business Statistics
). Analytic studies involve taking some action on a process to improve future performance (Berenson et al. a; Cryer and Miller ; Hoerl and Snee ). Data from a process are monitored over time and the focus of the study is to understand process behavior in order to make improvements to the process. Data collected typically constitute samples (i.e., subgroups) from some ongoing production process or service process obtained at regular intervals of time. The data can be analyzed descriptively but inferential analysis at this stage is usually limited to studying whether there are recognizable trends or other patterns in some critical-to-quality (CTQ) characteristic over time. By monitoring and understanding process variation, once a process is deemed “in-control,” the objective is to improve the process either by enhancing the critical-to-quality characteristic’s average and/or reducing process variation. This is done through inference – designing appropriate experiments based on the principle of 7randomization (Box et al. ). The fields of quality and productivity management, CQI/TQM, and Six-Sigma management have evolved from analytic studies. A parallel to the analytic study described in managing a manufacturing or service process is time-series analysis (Roberts ) which, for many years, was used for either decomposing a series of data into its trend, seasonal or cyclical components in order to compare the series with others in the same industry, across industries or against movements in the overall economy or for forecasting the series into the future (Croxton and Cowden ). The fields of econometric modeling and forecasting have evolved from time-series analysis.
Evolution of Business Statistics Education Through the s teaching emphasis was mainly on descriptive statistical methods along with time series decomposition, forecasting, and index number construction (Croxton and Cowden ). Numbers-crunching was typically achieved through the use of desktop calculators. However, in the s, as growth in computer technology permitted the storing of large data sets along with more rapid numbers-crunching capability, research using multivariate analysis methods that were developed in the s through s was finally plausible. Statistical software packages such as SPSS, SAS and BMDP, designed for research in the social sciences, psychology and medicine, were also adopted for use in business, particularly in marketing research and in finance. Thus, from the mid s through the mid s, business statistics education mainly focused on probability (Schlaifer ), sampling and, in particular, statistical inference.
In the mid s, sparked by world-wide recognition of the achievements of W. Edwards Deming, the need to distinguish between enumerative and analytic studies in a business environment in which collected data are often longitudinal and not a random sample led to the development of annual MSMESB (Making Statistics More Effective in Schools and Business) conferences and resulted in the infusion of TQM into the business statistics curriculum (Tiao et al. ). Probability and inference were deemphasized. Following Deming’s death in , the TQM approach, which so successfully developed in Japan, was marred by financial failures and corrupt business practices in that country. Unfortunately, MSMESB was unable to sustain its earlier momentum regarding the importance of TQM in the business statistics course curriculum, particularly after TQM evolved into a Six-Sigma approach (see 7Six Sigma) which B-school faculty considered “too much” to handle in a -year, let alone one-semester introductory course. The late s and the first decade of the new millennium have witnessed an even more rapid expansion of information technology capability. Owing to increased computer literacy and the ability to use computer software to solve larger-scale problems, there has been a reduction in numbers-crunching and more emphasis is now placed on analyzing computer-obtained results. In addition, advances in information technology have led to further advances in the data mining of enormous data bases and emphasis on regression model-building and other analytical tools for evaluating such large data sets is emerging in business statistics education.
The Future of Business Statistics Education Approximately % of all undergraduate students in the US who currently take an introductory statistics course are business majors. Thus, as recognized at the very first MSMESB conference (Tiao et al. ) in , it is essential that the course be practical and relevant, demonstrate applications to the functional areas of business, use real data, employ computer information technology rather than “numbers crunching” to get results, emphasize analysis and interpretation of results, and enhance critical thinking and problem-solving skills through written articulation of findings (Berenson et al. b). In addition, attention must be given to proper tabular and graphic presentation that assist in descriptive analysis and decision making. The importance of probability sampling and randomization in making inferences from sample statistics to population parameters in enumerative studies must be emphasized. And the course must provide an understanding of models useful for prediction and for
Business Surveys
forecasting as well as an understanding of process management in analytic studies through control chart monitoring and the use of other quality tools (Berenson et al. a). Owing to the need for B-schools to produce graduates who will be able to make useful contributions to business in an ever dynamic global environment, the field of business statistics will continue to make important contributions by providing the tools and methods along with enhancing the quantitative and critical thinking skills of the future workforce. With continued advances in IT, and continued developments in data mining and methodology for handling and analyzing enormously large sets of data collected over time, emerging functional area disciplines in business such as business analytics and business informatics will likely subsume business statistics along with the fields of operations research, management information systems and supply chain management. Future students preparing for work in such emerging fields will have a tremendous opportunity to learn and contribute to the world at large.
About the Author Dr. Mark L. Berenson is Professor, Department of Management and Information Systems, Montclair State University, USA, and Professor Emeritus, Statistics, Zicklin School of Business, Baruch College CUNY, USA. He is also a former President of the New York Metropolitan Area Chapter of the American Statistical Association. Over his academic career Professor Berenson has received several awards for excellence in teaching. He has authored and/or co-authored dozens papers and books, including Basic Business Statistics: Concepts and Applications with David M. Levine and Timothy C. Krehbiel (th edition, Prentice Hall, ), Business Statistics: A First Course with David M. Levine and Timothy C. Krehbiel (th edition, Prentice Hall, ), and Statistics for Managers Using Microsoft Excel with David M. Levine, David F. Stephan and Timothy C. Krehbiel (th edition, Prentice Hall, ).
B
7SIPOC and COPIS: Business Flow–Business Optimiza-
tion Connection in a Six Sigma Context 7Statistics Education 7Statistics: An Overview
References and Further Reading Berenson ML, Levine DM, Krehbiel TC (a) Basic business statistics: concepts and applications, th edn. Prentice Hall, Upper Saddle River, NJ Berenson ML, McKenzie J, Ord JK (b) Statistics in business schools: the future? In: Proceedings of the joint statistical meetings, Washington, DC, Box GEP, Hunter WG, Hunter JS () Statistics for experimenters. Wiley, New York Croxton FE, Cowden DJ () Applied General Statistics, nd edn. Prentice Hall, Englewood Cliffs, NJ Cryer J, Miller R () Statistics for business: data analysis and modeling. PWS-Kent, Boston Deming WE () Out of the crisis. MIT Center for Advanced Engineering Study, Cambridge, MA Hoerl R, Snee RD () Statistical thinking: improving business performance. Duxbury, Pacific Grove, CA Roberts HV () Data analysis for managers with minitab. Scientific Press, Redwood City, CA Schlaifer R () Introduction to statistics for business decisions. McGraw-Hill, New York Tiao G, Roberts HV, Easton G, organizers () First annual conference on making statistics more effective in schools and business (MSMESB). The University of Chicago, June –,
Business Surveys Ger Snijkers, , Mojca Bavdaž Professor in Business Survey Methodology at Utrecht University, Utrecht, The Netherlands Senior Researcher in Business Survey Data Collection Methodology at Statistics, The Netherlands Assistant Professor at the Faculty of Economics University of Ljubljana, Ljubljana, Slovenia
Cross References 7Banking, Statistics in 7Business Forecasting Methods 7Business Intelligence 7Business Surveys 7Data Analysis 7Detection of Turning Points in Business Cycles 7Economic Statistics 7Index Numbers 7Industrial Statistics 7Insurance, Statistics in 7Multivariate Data Analysis: An Overview 7National Account Statistics
The label business survey is typically attached to surveys that concern organizations involved in business activities such as manufacturing, commerce, finance, and other services. Business surveys may be considered a subgroup of establishment surveys. Establishment surveys refer to any formal organization engaged in any kind of productive activities, which also includes schools, hospitals, prisons, and other types of institutions that always or prevalently lack a lucrative purpose. Surveying businesses serves several aims. Governments use business surveys to collect data for economic indicators within the system of national accounts
B
B
Business Surveys
(e.g., for gross domestic product) and other areas of economic and 7business statistics. The collected data are quantitative rather than categorical data, and among the quantitative data the continuous type tends to be more frequent than discrete (Cox and Chinnappa ). Some examples are revenues, costs, value of imported and exported goods, number of employees, amount of produced goods, and energy consumption. The most important statistical parameters include totals, structures, and averages, for example, total production or total employment, the structure of businesses by size, and average wages. Changes in these statistics may even be more interesting for the conduct of economic policies (Rivière ). This is why panel or longitudinal sampling designs and recurring surveys are often used in such business surveys. In addition, academic researchers use business surveys to collect data for theoretical models that often describe complex structures and relationships, for example, organizational structures, the relationship between business innovativeness and revenue growth, the relationship between atmosphere at work and working at home, etc. Chambers of commerce and industry, boards of trade, commodity and industrial boards, and similar organizations representing businesses also use business surveys when they are interested in getting data from their members. Finally, there are business research methods (e.g., Blumberg et al. ) that are used by businesses to support their internal decision making related to business-to-business relations and their market position. The nature of participation in business surveys depends on their aims. In Europe, business surveys conducted by governmental institutions are typically mandatory, which means that sanctions may be used in case of noncompliance. In the US, this also applies to many but not all business surveys and censuses carried out by the Census Bureau. Mandatory business surveys usually achieve a response rate of % or more, often after a series of reminders. Voluntary business surveys not conducted by governmental statistical agencies achieve much lower response rates. These may be slightly higher for academic surveys (e.g., Baruch and Holtom () calculated an average around %) compared to commercial surveys. Business surveys have many unique characteristics given specific characteristics of the business population. The business population is volatile (this is particularly true of small enterprises) and difficult to track due to mergers, acquisitions, numerous forms of linkages, and cooperation. Various business units beyond administrative ones may constitute the most appropriate and meaningful unit of observation depending on the survey objectives: an
enterprise, a local unit, a kind-of-activity unit etc., but they may not be simple to determine in practice. The choice, definition, and identification of the unit of observation are of paramount importance as they may have a considerable impact on estimates, in particular on breakdowns by activity and region. A public business register can support this complex task and help to construct the survey frame, particularly if it builds on standard statistical units, provides common background variables (at least main economic activity, size indicator, and location), uses standard classifications for these variables (see, for instance, the UN Classifications Registry 7unstats.un.org/unsd/cr/registry/ and the latest version of the International Standard Industrial Classification of all Economic Activities (ISIC)), and is regularly updated. This reduces frame errors like omissions (e.g., start-ups), erroneous inclusions (e.g., bankrupt companies), duplications (e.g., change of the business name), and misclassifications (e.g., change of the main economic activity). This is why the construction, maintaining, and updating of business registers has traditionally been given a lot of attention in business survey methodology (see, for instance, Cox et al. ). Businesses are also heterogeneous in terms of their activity, behavior, and size. Many variables in business surveys have skewed distributions. Larger businesses in the economy (or in an activity or a region) may therefore be selected in the sample at all times and receive special attention in statistics production. Availability of background variables and a larger weight of some businesses make stratified sampling and probability-proportional-tosize typical sampling methods for business surveys (see 7Sample survey methods). Questionnaires in business surveys often resemble forms with item labels. Item labels substitute survey questions; they often consist of technical terms, are accompanied by detailed instructions, and request data kept in business records, which all make the response task quite burdensome and time-consuming, and call for selfadministered modes of data collection. The trend in governmental business surveys is towards mixed-mode designs in which paper questionnaires are complemented with electronic ones that can either be completed on-line or downloaded to be completed off-line (see 7Internet survey methodology – recent trends and developments). Both paper and electronic questionnaires should be designed in such a way that it serves the response process within businesses (see e.g., Snijkers et al. ). In business surveys, more than one person may be necessary to reach the decision on survey participation and provide survey answers because authority, capacity, and motivation to respond rarely reside in
Business Surveys
one single person within the organization. Organizations have several characteristics that influence their decision on survey participation such as centralization of decision making, fragmentation of knowledge through specialization and founding of subsidiaries and branch plants, boundary-spanning roles, environmental dependence, etc. (Tomaskovic-Devey et al. ; Snijkers et al. ). Some factors influencing this decision are under researchers’ control while others are not. The former, for instance, include the selection of the mode, the questionnaire design, and the contact strategies (e.g., introducing the survey, motivating respondents to participate, and respondent chasing); the latter include the selection of the actual respondent(s), internal business factors (e.g., policy not to participate in surveys), and factors in external environment such as survey-taking climate, political and economic climate, and legal and regulatory requirements (Willimack et al. ; Snijkers ). Paying attention to these factors, even taking the factors that are out of control into account in the survey design, may help prevent or reduce non-response in business surveys. As in other surveys, the measurement process culminates in the process of responding to survey questions. Tourangeau () pointed to four cognitive components of this process: understanding of the survey question, retrieval of information from memory, judgment of retrieved pieces of information, and communication of the response (see 7Sample survey methods). While the essence of the response process stays the same also in business surveys, many specifics shape this process considerably. Several people are often involved in the process: in addition to the respondent, also referred to as a reporter or an informant that indeed acts as a business representative or spokesperson, there may be other business participants such as gatekeepers (e.g., boundary-spanning units, receptionists), response coordinators, data providers, and authorities who also may serve as gatekeepers. When the required data and/or knowledge to extract them are dispersed across the business, response coordinators organize the survey task. Respondents retrieve the required data themselves or have data providers to retrieve data for them. Authorities have an important role in providing a mandate to work on the 7questionnaire, delegating and scheduling the survey response task and in authorizing the survey response. Another specific of business surveys is heavy reliance on data stored in business records. When the requested data are not (readily) available in business records, the response burden increases. If the survey design is tailored to internal business processes, the response burden is likely
B
to decrease, which affects the occurrence of non-response and measurement errors (Dale and Haraldsen ; Snijkers ). Complexity of data collection in business surveys gave rise to several models of response processes. The latest contributions are the hybrid model by Willimack and Nichols (), the model introducing the idea of socially distributed cognition by Lorenc (), the response model focusing on motivating people and influencing their response behavior by Snijkers (), and the multidimensional integral business survey response (MIBSR) model by Bavdaž (b). Quality assessment of business surveys may be based on quality frameworks developed by statistical organizations (e.g., IMF, Eurostat, Statistics Canada, Australian Bureau of Statistics, etc.), especially if these surveys provide data for official statistics. None of these frameworks can do without the accuracy of statistical data. Accuracy can be assessed using the concept of the total survey error (see 7Total survey error) even though some specifics may apply especially to non-sampling errors (see 7Non-sampling errors in surveys). Specification errors, for instance, may be quite large because business surveys attempt to measure complex economic phenomena. Frame errors depend on the existence and quality of a business register; these errors do not only refer to under coverage and over coverage but also to duplications and misclassifications. When several units within a business constitute units of observation, businesses may not report data for all of them, which results in withinbusiness non-response; when units of observation differ from existing organisational units, businesses may report data for wrong units, which results in measurement errors. Given the specifics and complexity of the response process in business surveys, sources of measurement errors include all business participants and survey staff involved in the response process, the business environment, the survey instrument, and various survey characteristics and procedures beyond the mode of data collection, in particular the recurrence of the response process (Bavdaž a). For many items in business survey questionnaires, it is difficult to distinguish between item non-response and a valid zero value. Non-response and measurement errors may be reduced in the data editing process. Given the usual abundance of auxiliary information taken from previous data reporting in the same survey, other surveys, or administrative sources, data editing has traditionally been given a lot of attention in business surveys. This is why data processing errors may be relatively small. Since the s, business survey methodology has been increasingly given a systematic approach in order to solve business surveys’ unique problems (Cox et al. ). Up
B
B
Business Surveys
to now, three International Conferences on Establishments Surveys (ICES) have been organized in , and , and the fourth one is coming up (). Other initiatives also serve this goal, for example, the International Workshop on Business Data Collection Methodology (www.ssb.no/bdcmethods) and the European Establishment Statistics Workshop (www.enbes.org). In Europe, Eurostat has started a major program on Modernisation of European Enterprise and Trade Statistics (MEETS; Behrens ). These initiatives try to provide a platform for business survey methodologists to discuss their problems, promote exchange of ideas and international collaboration, foster research in this field, and come to current best practices.
Cross References 7Business Statistics 7Internet Survey Methodology: Recent Trends and
Developments 7Sample Survey Methods
References and Further Reading Baruch Y, Holtom BC () Survey response rate levels and trends in organizational research. Hum Relat ():– Bavdaž M (a) Sources of measurement errors in business surveys. J Official Stat ():– Bavdaž M (b) The multidimensional integral business survey response model. Surv Methodol ():– Behrens A () Modernisation of European Enterprise and Trade Statistics (MEETS). Paper presented at the international conference on new techniques and technologies for statistics, Brussels, Belgium, – February (epp.eurostat.ec.europa.eu/ portal/page/portal/research methodology/NTTS ) Blumberg B, Cooper D, Schindler PS () Business research methods: second European edition. McGraw Hill, London
Cox BG, Chinnappa BN () Unique features of business surveys. In: Cox BG et al (eds) Business survey methods. Wiley, New York, pp – Cox BG, Binder DA, Chinnappa BN, Christianson A, Colledge MJ, Kott PS (eds) () Business survey methods. Wiley, New York Dale T, Haraldsen G (eds) () Handbook for monitoring and evaluating business survey response burden. Eurostat, Luxembourg Lorenc B () Two topics in survey methodology: modelling the response process in establishment surveys; inference from nonprobability samples using the double samples setup. Doctoral dissertation, Stockholm University Rivière P () What makes business statistics special? Int Stat Rev ():– Snijkers G () Getting data for business statistics: a response model. In: Proceeding of Q; European conference on quality in official statistics, Rome, – July Snijkers G, Berkenbosch B, Luppes M () Understanding the decision to participate in a business survey. In: Proceedings of ICES-III (rd international conference on establishment surveys), Montreal, Canada, – June , American Statistical Association, Alexandria, pp – Snijkers G, Onat E, Vis-Visschers R () The annual structural business survey: developing and testing an electronic form. In: Proceedings of ICES-III (rd international conference on establishment surveys), Montreal, Canada, – June , American Statistical Association, Alexandria, pp – Tomaskovic-Devey D, Leiter J, Thompson S () Organizational survey nonresponse. Adm Sci Q ():– Tourangeau R () Cognitive science and survey methods. In: Jabine TB, Straf M, Tanur JM, Tourangeau R (eds) Cognitive aspects of survey methodology: building a bridge between disciplines. National Academy Press, Washington, pp – Willimack D, Nichols E () A hybrid response process model for business surveys. J Official Stat ():– Willimack D, Nichols E, Sudman S () Understanding unit and item nonresponse in business surveys. In: Groves RM, Dillman DA, Eltinge JL, Little RJA (eds) Survey nonresponse. Wiley, New York, pp –
C Calibration Christos P. Kitsos Professor and Head Technological Educational Institute of Athens, Athens, Greece
where U is the design space, which can always be transformed to [−, ]. Moreover, the involved error is assumed to be from the normal distribution with mean zero and variance σ > . The aim is to estimate the value of u = u given n = C, i.e., u = (C − θ ) /θ
Various methods and different (linear or not, simple linear, or multivariate) models have been adopted in industry to address the calibration problem. In practice, most of the models attempt to deal with the simple linear calibration technique, mostly applied in chemical applications, especially when some instruments are to be calibrated (examples include pH meters, NIR instruments, and establishing calibration graphs in chromatography). The early work of Shukla () put forward the problem on the real statistical dimensions, and even early on it was realized that when a non-linear model describes the phenomenon (Schwartz ), a linear approximation is eventually adopted. But even so, in the end we come to a nonlinear function to be estimated as best as possible (Kitsos and Muller ). When the variance of the measurement is due to many sources of variability, different techniques are used. Statistical calibration has been reviewed by Osborn (), who provides a list of pertinent references; when a robust approach might be appropriate, see Kitsos and Muller (). Certainly, to consider the variance constant and to follow a statistical quality control method (see 7Statistical Quality Control), Hochberg and Marom () might be helpful, but not in all cases. For the multivariate case, see the compact book of Brown (), Brereton (), and for an application Oman and Wax (). Moreover, different methods have been used on the development of the calibration problem like crossvalidation (see Clark ). Next we briefly introduce the statistical problem and the optimal design approach is adopted in the sequence to tackle the problem. Consider the simple regression model with n = E(y∣u) = θ + θ u
u ∈ U = [−, ]
which is a nonlinear function of the involved linear parameters, as we have already emphasized above. The most well-known competitive estimators of u when y is provided are the so-called “classical predictor” C (u ) = x¯ +
Sxx (˙y − y¯) Sxy
and the “inverse predictor” I (u ) = u¯ +
Sxy (˙y − y¯) Syy
with: Str = ∑(ti − ¯t )(ri − ¯r) were by y˙ we mean the average of the possible k observations taken at the prediction stage (or experimental condition) and y¯ as usually the average of the collected values. The comparison of C(u ) and I(u ) is based on the values of the sample size n and the proportion ∣σ/θ ∣ under the assumption that x belongs to the experimenter area. One of the merits of C(u ) is that when the usual normal assumption for the errors is imposed, the classical predictor is the maximum likelihood estimator. Moreover, C(u ) is a consistent estimator while I(u ) is inconsistent. The I(u ) estimation is criticized as it provides a confidence interval that might be the whole real line or, in the best case, two disjoint semi-infinite intervals. When ∣σ/θ ∣ is small the asymptotic mse (mean square error) of C(u ) is smaller than with the use of I(u ), when x does not lie in the neighborhood of u¯ . The main difficulty is the construction of confidence intervals, as the variance of u does not exist. This provides food for thought for an optimal design approach for the calibration problem. To face these difficulties the
Miodrag Lovric (ed.), International Encyclopedia of Statistical Science, DOI ./----, © Springer-Verlag Berlin Heidelberg
C
Canonical Analysis and Measures of Association
optimal experimental approach is adopted (see Optimum Experimental Designs, also see Kitsos ). For the one-stage design we might use of the criterion function Φ, either D-optimality for (θ , θ ) or c-optimality for estimating u . The D-optimal design is of interest because its effectiveness can be investigated, as measured by the c-optimality criterion. Under c-optimality, thanks to Elfving’s theorem, locally optimal two-point design can be constructed geometrically. The criterion that experimenters like to use is min Var(ˆu ). Different approaches have been adopted for this crucial problem: Bayesian, see Chaloner and Verdinelli (), Hunter and Lamboy (); structural inference, see Kalotay (). There is a criticism that structural inference is eventually Bayesian, but this is beyond the scope of this discussion. When suitable priors for u are chosen the calibrative density functions come from the non-central Student with mean Ba(u ) as Ba(u ) = u¯ +
Syy (˙y − y¯) r
where r = Syy + ∑kj (yj − y¯) . When k = the Bayesian estimator coincides with the inverse, namely Ba(u ) = I(u ). The structural approach forms the simple linear model as a “structural model” and obtains a rather complicated model, which, again, with k = , coincides with the inverse regression. The nonlinear calibration has attracted classical and Bayesian approaches, both based on the Taylor expansion of the nonlinear model. Therefore, calibration is based on the linear approach of the nonlinear model.
About the Author Dr. Christos Kitsos is a Professor and Head, Department of Mathematics, of the Technological Educational Institute of Athens, Greece. His first degree is in mathematics, Athens University, his MA degree from the Math and Stat of University of New Brunswick, Canada, while his PhD is from the Glasgow University, UK. He is a member of ISI (and ISI-Committee of Risk Analysis), IEEE and was elected member of IASC, were he is also a member. He has organized a number of statistical conferences in Greece, and has founded the series of International Conference on Cancer Risk Assessment (ICCRA). He has published papers in journals and proceedings volumes, participated in conferences, and published books in Greek ( are textbooks), and is a coauthor of international books as editor. Professor Kitsos has been the national representative at EUROSTAT and OECD for educational statistics.
Cross References 7Chemometrics 7Measurement Error Models 7Optimal Regression Design 7Optimum Experimental Design
References and Further Reading Brereton GR () Introduction to multivariate calibration in analytical chemistry. Analyst ():– Brown JP () Measurement, regression and calibration. Oxford Science Publication, Oxford Chaloner K, Verdinelli I () Bayesian experimental design: a review. Stat Sci :– Clark RM () Calibration, cross-validation and carbon-. II. J Roy Stat Soc, Ser A :– Frank IE, Friedman JH () A Statistical view of some chemometrics regression tools. Technometrics :–. With discussion Hochberg Y, Marom I () On improved calibrations of unknowns in a system of quality-controlled assays. Biometrics :– Hunter WG, Lamboy WFA () Bayesian analysis of the linear calibration problem. Technometrics :– Kalotay AJ () Structural solution to the linear calibration problem. Technometrics :– Kanatani K () Statistical analysis of focal-length calibration using vanishing points. IEEE Trans Robotic Autom :– Kitsos CP () Quasi-sequential procedures for the calibration problem. In Dodge Y, Whittaker J (eds) COMPSTAT , vol . Physica-Verlag, Heidelberg, pp – Kitsos CP () The simple linear calibration problem as an optimal experimental design. Commun Stat – Theory Meth :– Kitsos CP, Muller Ch () Robust linear calibration. Statistics :– Oman SD, Wax Y () Estimating fetal age by ultrasound measurements: an example of multivariate calibration. Biometrics :– Osborne C () Statistical calibration: a review. Int Stat Rev :– Schwartz LM () Statistical uncertainties of analyses by calibration of counting measurements. Anal Chem :– Shukla GK () On the problem of calibration. Technometrics :–
Canonical Analysis and Measures of Association Jacques Dauxois , Guy Martial Nkiet Professor Institut de Mathématiques de Toulouse, Toulouse, France Professor Université des Sciences et Techniques de Masuku, Franceville, Gabon
Introduction The Bravais–Pearson linear correlation coefficient and the Sarmanov maximal coefficient are well known statistical
Canonical Analysis and Measures of Association
C
Proposition Consider a compact CA ({ρ i }i∈I , { f }i∈I , {g}i∈I ), of H and H , where the ρ i ’s are arranged in nonincreasing order. Then:
Remark . The ρ i ’s are termed the canonical coefficients. They permit to study the relative postions of each of the preceding subspace with respect to the other. For instance, the nullity of all these coefficients is equivalent to the orthogonality of H and H , and if one of these subspaces is included into the other these coefficients are all equal to . Note that, in this later case there does not exist a compact CA when the considered subspaces are infinite-dimensional ones. Nevertheless, it is possible to find a triple having the same properties than a compact CA. Such a triple can be given by (I, ( fi )i∈N∗ , (gi )i∈N∗), where I is the numerical sequence with all terms equal to , ( fi )i∈N∗ is a orthonormal basis of H and (gi )i∈N∗ is the previous system possibly completed with an orthonormal basis of ker T = H ∩ H⊥ so as to obtain an orthonormal basis of H . . From the previous notion of CA it is possible to define a canonical analysis of two subspaces H and H relatively to a third one H . It is just the CA of the subspaces H⋅ := (H ⊕ H ) ∩ H⊥ and H⋅ := (H ⊕ H ) ∩ H⊥ . This CA leads to interesting properties given in Dauxois et al. (a), and is useful in statistics for studying conditional independence between random vectors (see, e.g., Dauxois et al. [b]). p T p T . When X = (X , ⋯, X ) and X = (X , ⋯, X ) are j two random vectors such that any Xi belongs to L (P), their Linear Canonical Analysis (LCA) is the CA of H p and H where Hi = span (Xi , ⋯, Xi i ). The spectral analysis of T is equivalent to that of V− V V− V , where Vi (resp. V ; resp. V ) denotes the covariance (resp. cross-covariance) operator of Xi (resp. X and X ; resp. X and X ). So, it is just the CA of random vectors introduced by Hotelling (). The first canonical coefficient is the Bravais–Pearson linear correlation coefficient. . When X and X are arbitrary random variables, their Nonlinear Canonical Analysis (NLCA) is the CA of H and H where Hi is the closed subspace of L (P) consisting in random variables of the form φ(Xi ), where φ is a measurable function valued into R. In this case, the first canonical coefficient just is the Sarmanov maximal coefficient.
. {ρ i }i∈I is the noincreasing sequence of eigenvalues of T and T and, for any i ∈ I , one has ≤ ρ i ≤ . . { f }i∈I (resp. {g}i∈I ) is an orthonormal basis of H (resp. H ) such that, for any i ∈ I , fi (resp. gi ) is an eigenvector of T (resp. T ) associated with ρ i . . ∀(i, j) ∈ (I ) , ⟨ fi , gj ⟩ = δ ij ρ i , Π H fi = ρ i gi , Π H gi = ρ i fi . . { f }i∈I −I (resp. {g}i∈I −I ) is an orthonormal basis of ker(T ) = H ∩ H⊥ (resp. ker(T ) = H ∩ H⊥ ).
Let C be the set of pairs (H , H ) of closed subspaces of a Hilbert space, having a compact CA or being infinite-dimensional and such that H ⊂ H or H ⊂ H . We consider an equivalence relation ≃ defined on C, such that (H , H ) ≃ (E , E ) if there exists a pair (I , I ) of isometries satisfying: I (H ) = E , I (H ) = E and ∀(x, y) ∈ H × H , < I (x), I (y) >E =< x, y >H , where
tools that permit to measure, respectively, correlation (also called linear dependence) and stochastic dependence of two suitable random variables X and X defined on a probability space (Ω, A, P). Since these coefficients just are the first canonical coefficients obtained from linear and nonlinear canonical analysis, respectively, it is relevant to improve them by using all the canonical coefficients. In order to give an unified framework for these notions, we introduce the canonical analysis (CA) of two closed subspaces H and H of a Hilbert space H. Then, a class of measures of association that admits the aforementioned coefficients as particular cases can be constructed.
Canonical Analysis Let H be a separable real Hilbert space with inner product and related norm denoted by ⟨⋅, ⋅⟩ and ∥⋅∥ respectively, and H and H be two closed subspaces of H. Then we have the following definition that comes from Dauxois and Pousse (). Definition any triple
The canonical analysis (CA) of H and H is ({ρ i }i∈I , { f }i∈I , {g}i∈I ),
∗
with Iℓ ⊂ N for ℓ ∈ {, , }, that satisfies: . The system { f }i∈I (resp. {g}i∈I ) is an orthonormal basis of H (resp. H ) . ρ = ⟨ f , g ⟩ = sup {⟨ f , g⟩; ( f , g) ∈ H ×H , ∥ f ∥ = ∥g∥ =} . For any i ∈ I such that i ≥ , one has: ρ i = ⟨ fi , gi ⟩ = sup{⟨ f , g⟩; ( f , g) ∈ Fi⊥ × G⊥i , ∥ f ∥ = ∥g∥ = } where Fi = span { f , ⋯, fi− } and Gi = span {g , ⋯, gi− }. Conditions for existence of canonical analysis have been investigated in the aforementioned work. More precisely, denoting by Π E the orthogonal projector onto the closed subspace E of H, a sufficient condition is the compacity of T = Π H Π H ∣H or, equivalently, that of T = Π H Π H ∣H . In this case, we say that we have a compact CA, and the following proposition holds:
Measures of Association
C
C
Canonical Analysis and Measures of Association
H (resp. E) denotes the separable real Hilbert space which contains H and H (resp. E and E ). We also consider a preordering relation ⪯ on C, such that (H , H ) ⪯ (E , E ) if there exists a pair (E′ , E′ ) of closed subspaces satisfying: E′ ⊂ E , E′ ⊂ E and (H , H ) ≃ (E′ , E′ ). Definition A measure of association r between Hilbertian subspaces is any map from a subset Cr of C into [, ] such that the following conditions are satisfied:
H ⊥ H ⇔ r(H , H ) = ; or
H ⊂ H ⇒ r(H , H ) = ;
(H , H ) ≃ (E , E ) ⇒ r(H , H ) = r(E , E ); (H , H ) ⪯ (E , E ) ⇒ r(H , H ) ≤ r(E , E ).
Remark
p
. When X = (X , ⋯, X )
For all x ∈ cΦ and σ ∈ P(N∗ ), one has xσ ∈ cΦ and Φ (∣xσ ∣) = Φ (x).
.
For all (x, y) ∈ (cΦ ) , if ∀n, ∣xn ∣ ≤ ∣yn ∣, then Φ (x) ≤ Φ (y).
.
There exists a nondecreasing function fΦ from R to R such that : fΦ () = ; ∀u ∈ R, (u, , ⋯) ∈ cΦ and Φ (u, , ⋯) = fΦ (∣u∣).
We denote by Ψ the map from C to c ∪ {I} such that Ψ(H , H ) is the noincreasing sequence of canonical coefficients of H and H . Then we have:
r(H , H ) = r(H , H );
H ⊂ H
.
T
and X =
T
p (X , ⋯, X ) j Xi belongs to
are two random vectors such that any L (P), we obtain a measure of linear dependence between X and X by putting r(X , X ) := p r(H , H ) with Hi = span (Xi , ⋯, Xi i ). Indeed, from second axiom given above, r(X , X ) = if and only if X and X are uncorrelated, that is V = . From the third one, it is seen that if there exists a linear map A such that X = AX then r(X , X ) = . . When X and X are arbitrary random variables, considering Hi = {φ(Xi ) / E(φ(Xi ) ) < +∞}, a measure of stochastic dependence of X and X is obtained by putting r(X , X ) := r(H , H ). In this case, the above axioms are closed to the conditions proposed by Rényi () for good measures of dependence. In particular, the second axiom gives the equivalence between the independence of X and X and the nullity of r(X , X ), and from the third axiom it is seen that for any one to one and bimeasurable functions f and g, one has r( f (X ), g(X )) = r(X , X ). A class of measures of association can be built by using symmetric non decreasing functions of canonical coefficients. In what follows, P(N∗ ) denotes the set of permutations of N∗ . For σ ∈ P(N∗ ) and x = (xn )n ∈ c , we put xσ = (xσ(n) )n and ∣x∣ = (∣xn ∣)n . We denote by c the space of numerical sequences x = (xn )n such that limn→∞ xn = . Definition A symmetric nondecreasing function (sndf) is a map Φ from a subset cΦ of c to R+ satisfying:
Proposition Let Φ be a sndf with definition domain cΦ , and such that Φ (I) = . Then, the map rΦ = Φ○Ψ is a measure of association defined on the subset CΦ = {(H , H ) ∈ C; Ψ(H , H ) ∈ cΦ ∪ {I}}. This proposition means that a measure of association between two subspaces is obtained as a function of the related nonincreasing sequence of canonical coefficients through a sndf. Some examples of such measures are:
+∞
r (H , H ) = − exp (− ∑ ρ i , r,p (H , H )) i=
¿ Á ∑+∞ ρ p Á ∗ i= i À =Á p (p ∈ N ), r (H , H ) + ∑+∞ ρ i= i = max ∣ρ i ∣ = ρ . i≥
On the one hand, this class of measures of association contains all the measures built by using LCA of random vectors (see Cramer and Nicewander (), Lin (), Dauxois and Nkiet (b)). On the other hand, when H and H are the subspaces considered in the second assertion of Remark , r just is the Sarmanov maximal coefficient. In this case, estimation of the coefficients from NLCA and, therefore, the related measures of associations can be obtained from approximation based on step functions or B-spline functions, and from sampling. Using this approach, a class of independence tests that admits the chisquared test of independence as particular case, have been be proposed (see Dauxois and Nkiet ()).
Cross References 7Canonical Correlation Analysis 7Multivariate Data Analysis: An Overview
Canonical Correlation Analysis
References and Further Reading Cramer EM, Nicewander WA () Some symmetric invariant measures of multivariate association. Psychometrika :– Dauxois J, Nkiet GM (a) Canonical analysis of Euclidean subspaces and its applications. Linear Algebra Appl :– Dauxois J, Nkiet GM (b) Testing for the lack of a linear relationship. Statistics :– Dauxois J, Nkiet GM () Nonlinear canonical analysis and independence tests. Ann Stat :– Dauxois J, Pousse A () Une extension de l’analyse canonique. Quelques Applications. Ann Inst Henri Poincaré XI:– Dauxois J, Nkiet GM, Romain Y (a) Canonical analysis relative to a closed subspace. Linear Algebra Appl :– Dauxois J, Nkiet GM, Romain Y (b) Linear relative canonical analysis, asymptotic study and some applications. Ann Inst Stat Math :– Hotelling H () Relations between two sets of variables. Biometrika :– Lin PE () Measures of association between vectors. Commun Stat Theory Meth :– Rényi A () On measures of dependence. Acta Math Acad Sci Hung :–
Canonical Correlation Analysis Tenko Raykov Professor of Measurement and Quantitative Methods Michigan State University, East Lansing, MI, USA
Introduction Canonical correlation analysis (CCA) is one of the most general multivariate statistical analysis methods (see 7Multivariate Statistical Analysis). To introduce CCA, consider two sets of variables, denoted A and B for ease of reference (e.g., Raykov and Marcoulides ). Let A consist of p members collected in the vector x, and let B consist of q members placed in the vector y (p > , q > ). In an application setting, the variables in either set may or may not be considered response variables (dependent or outcome measures) or alternatively independent variables (predictors, explanatory variables). As an example, A may consist of variables that have to do with socioeconomic status (e.g., income, education, job prestige, etc.), while B may comprise cognitive functioning related variables (e.g., verbal ability, spatial ability, intelligence, etc.). Consider the correlation matrix R of all variables in A and B taken together, which has ( p + q) ⋅ ( p + q − )/ non-duplicated (non-redundant) correlations. Obviously, even for relatively small p and q, there are many nonduplicated elements of R. CCA deals with reducing this potentially quite large number of correlations to a more
C
manageable group of interrelationship indices that represent the ways in which variables in A covary with variables in B, i.e., the interrelationships among these two sets of variables. More specifically, the purpose of CCA is to obtain a small number of derived variables (measures) from those in A on the one hand, and from those in B on the other, which show high correlations across the two sets (e.g., Johnson and Wichern ). That is, a main goal of CCA is to “summarize” the correlations between variables in set A and those in set B into a much smaller number of corresponding linear combinations of them, which in a sense are representative of those correlations. With this feature, CCA can be used as a method for () examining independence of two sets of variables (viz. A and B), () data reduction, and () preliminary analyses for a series of subsequent statistical applications. Achieving this goal is made feasible through the following steps (cf. Raykov and Marcoulides ). First, a linear combination Z of the variables x in A is sought, as is a linear combination W of the variables y in B, such that their correlation ρ , = Corr(Z , W ) is the highest possible across all choices of combination weights for W and Z (see next section for further details). Call Z and W the first pair of canonical variates, and ρ , the first canonical correlation. In the next step, another linear combination Z of variables in A is found and a linear combination W of variables in B, with the following property: their correlation ρ , = Corr(Z , W ) is the highest possible under the assumption of Z and W being uncorrelated with the variables in the first combination pair, Z and W . Z and W are referred to as the second pair of canonical variates, and ρ , as the second canonical correlation. This process can be continued until t pairs of canonical variates are obtained, where t = min(p, q) being the smaller of p and q. While in many applications t may be fairly large, it is oftentimes the case that only up to the first two or three pairs of canonical variates are really informative (see following section). If all canonical correlations are then uniformly weak and close to zero, A and B can be considered largely (linearly) unrelated. Otherwise, one could claim that there is some (linear) interrelationship between variables in A with those in B. Individual scores on the canonical variates can next be computed and used as values on new variables in subsequent analyses. These scores may be attractive then, since they capture the essence of the cross-set variable interrelationships.
Procedure
To begin a CCA, two linear combinations Z = a′ x and W = b′ y are correspondingly sought from the variables
C
C
Canonical Correlation Analysis
in A and in B, such that ρ , = Corr(Z , W ) is at its maximal possible value across all possible choices of a and b . Consider the covariance matrix S of the entire set of p + q variables in A and B: ⎡ ⎢ S ⎢ S=⎢ ⎢ ⎢ S ⎣
⎤ S ⎥ ⎥ ⎥, ⎥ S ⎥ ⎦
where S is the covariance matrix of the p variables in A, S that of the q variables in B, S that of the q variables in B with the p in A, and S denotes the covariance matrix of the p variables in A with the q measures in B. It can be shown (e.g., Johnson and Wichern ) that this maximum correlation ρ , will be achieved if the following holds: . a is taken as the (generalized) eigenvector pertaining to the largest solution ρ of the equation ∣S S− S − ρ S ∣ = , where ∣.∣ denotes determinant, that is, a fulfils the equation (S S− S − ρ S ) a = , with ρ being the largest solution of the former equation. . b is the (generalized) eigenvector pertaining to the largest root of the equation ∣S S− S − π S ∣ = , that is, b fulfils the equation (S S− S − π S ) b = , with the largest π satisfying the former equation. The solutions of the two involved determinantal equations are identical, that is, ρ = π , and the positive square root of the largest of them equals ρ() = π() = ρ , = Corr(Z , W ), the maximal possible correlation between a linear combination of variables in A with a linear combination of those in B. Then Z = a′ x and W = b′ y represent the first canonical variate pair, with this maximal correlation, Corr(Z , W ), being the first canonical correlation. As a next step, the second canonical variate pair is furnished as a linear combination of the variables in A, using the eigenvector pertaining to the second largest solution of ∣S S− S − ρ S ∣ = on the one hand, and a linear combination of the B variables using the second largest solution of ∣S S− S − π S ∣ = on the other hand; then their correlation is the second canonical correlation. One continues in the same manner until t = min(p, q) canonical variate pairs are obtained; the corresponding canonical correlations are calculated as their interrelationship indices (correlations). From the construction of the canonical variates follows that they are uncorrelated with one another: Cov(Zi , Zj ) = Cov(Wi , Wj ) = Cov(Zi , Wj ) = (for all i ≠ j; i, j = , . . . , t).
Interpretation Even though there are t = min(p, q) canonical variate pairs and canonical correlations, oftentimes in applications not all are important for understanding the relationships among variables in A and B. Statistical tests are available which help evaluate the importance of canonical variate pairs and aid a researcher in finding out how many pairs could be retained for further analysis. The tests assume multivariate normality and examine hypotheses of canonical correlations being in a given population. The first test evaluates the null hypothesis that all canonical correlations are . If this hypothesis is rejected, at least the first canonical variate pair is of relevance when trying to understand the interrelationship between the variables in A and B; more specifically, at least the first canonical correlation is not zero in the population. Then the second test examines the null hypothesis that apart from the first canonical correlation, all remaining ones are ; and so on. If the first tested hypothesis is not rejected, it can be concluded that A and B are (linearly) unrelated to one another. After completing these tests, and in case at least the first canonical correlation is significant, the next question may well be how to interpret the canonical variates. To this end, one can use the correlations of each canonical variate with variables within its pertinent set. That is, when trying to interpret Z , one can look at its correlations with the variables in A. Similarly, when trying to interpret W , one can examine its correlations with the variables B; and so on for the subsequent canonical variate pairs and their members. The principle to follow thereby, is to interpret each canonical variate as representing the common features of initial variables correlated highly with that variate. Furthermore, for a given canonical correlation ρ i = π i , its square ρ i can be interpreted as a squared multiple correlation coefficient for the regression relating the ith canonical variate for any of the sets A or B, with the variables of the other set (B or A, respectively; i = , . . . , t). With this in mind, ρ i can be viewed as proportion shared variance between A and B, as captured by the ith canonical variate pair (i = , . . . , t); the square of the first canonical correlation is interpretable as a measure of “set overlap.” Similarly to principal components and factors, canonical variates can be used to obtain individual subject scores on them. They can be used in subsequent analyses, e.g., as scores on explanatory variables. Like principal components, the units of a canonical variate may not be meaningful. It is stressed that canonical variates are not latent variables, but instead share the same observed status as manifest (recorded) variables, since they are linear combinations of the latter.
Careers in Statistics
Relationship to Discriminant Function Analysis It can be shown (Tatsuoka ) that with k > groups discriminant function analysis (DFA) is identical to CCA using additionally defined variables D , D , . . . , Dk− as comprising set A, while the original explanatory (predictor) variables, say x = (x , x , . . . , xp )′ , are treated as set B. These 7dummy variables D , D , . . . , Dk− are defined in exactly the same way they would be for purposes of regression analysis with categorical predictors. If one then performs a CCA with these sets A and B, the results will be identical to those obtained with a DFA on the original variables x. Specifically, the first canonical variate for B will equal the first discriminant function; the second canonical variate for B will equal the second discriminant function, etc. The test for significance of the canonical correlations is then a test for significance of discriminant functions, and the number of significant such functions and of canonical correlations is the same. Further, each consecutive eigenvalue for the discriminant criterion, vi , is related to a corresponding generalized eigenvalue (determinantal equation root) ρ i : vi =
ρi (i = , , . . . , r; Johnson − ρi
and Wichern ). Testing the significance of discriminant functions is thus equivalent to testing significance of canonical correlations.
C
analysis method, which subsumes a number of others that are widely used in statistical applications.
About the Author Tenko Raykov is a Professor of Measurement and Quantitative Methods at Michigan State University. He received his Ph.D. in Mathematical Psychology from Humbolt University in Berlin. He is an editorial board member of the Structural Equation Modeling, Multivariate Behavioral Research, Psychological Methods and the British Journal of Mathematical and Statistical Psychology. He is a coauthor (with G.A. Marcoulides) of the text A First Course in Structural Equation Modeling (Lawrence Erlbaum Associates, ), An Introduction to Applied Multivariate Analysis (Routledge ), and Introhhduction to Psychometric Theory (Routledge ).
Cross References 7Analysis of Multivariate Agricultural Data 7Canonical Analysis and Measures of Association 7Discriminant Analysis: An Overview 7Eigenvalue, Eigenvector and Eigenspace 7Multivariate Data Analysis: An Overview 7Multivariate Statistical Analysis
References and Further Reading Generality of Canonical Correlation Analysis CCA is a very general multivariate statistical method that unifies a number of analytic approaches. The canonical correlation concept generalizes the notion of bivariate correlation that is a special case of the former for p = q = variables. The multiple correlation coefficient of main relevance in regression analysis is also a special case of canonical correlation, which is obtained when the set A consists of p = variable – the response measure – and the set B consists of q variables that are the predictors (explanatory variables) in the pertinent regression model. The multiple correlation coefficient is then identical to the first canonical correlation. Third, since various uni- and multivariate ANOVA designs can be obtained as appropriate special cases of regression analysis, these designs and corresponding ANOVAs can be seen as special cases of canonical correlation analysis. Also, as indicated in the preceding section, discriminant function analysis is a special case of CCA as well. (Since DFA is a “reverse” MANOVA – e.g., Raykov and Marcoulides – one can alternatively see the latter also as a special case of CCA.) Hence, canonical correlation analysis is a very general multivariate
Johnson RA, Wichern DW () Applied multivariate statistical analysis. Prentice Hall, Upper Saddle River Raykov T, Marcoulides GA () An introduction to applied multivariate analysis. Taylor & Francis, New York Tatsuoka MM () Multivariate analysis: techniques for educational and psychological research. Wiley, New York
Careers in Statistics Daniel R. Jeske , Janet Myhre Professor and Chair University of California-Riverside, Riverside, CA, USA MARC and Professor Emeritus Claremont McKenna College, Claremont, CA, USA Statistics has changed over the last decades from being a discipline that primarily studied ways to characterize randomness and variation to a discipline that emphasizes the importance of data in the explanation of phenomenon and in problem solving. While statisticians routinely use mathematics and computer programming languages as key
C
C
Careers in Statistics
tools in their work, they usually also function as an important data-driven decision maker within their application domain. Consequently, a statistician must have a genuine curiosity about the subject domain they work within, and furthermore, must have strong collaborative and communication skills in order successfully interact with the many colleagues they will encounter and rely on for information. As the world becomes more quantitative through the data revolution, more professions and businesses depend on data and on the understanding and analyses of these data. Data are not simply numbers. Data contain information that needs to be understood and interpreted. As a result, statisticians are much more than bean counters or number crunchers. They possess skills to find needles in haystacks and to separate noise from signal. They are able to translate a problem or question into a framework that enables data collection and data analysis to provide meaningful insights that can lead to practical conclusions. Loosely speaking there is a spectrum of statisticians that ranges from very applied on one end to very theoretical on the other end. Applied statisticians skillfully select and implement an appropriate statistical methodology to solve a problem. They are a statistician who has a problem and is looking for a solution. Theoretical statisticians are interested in trying to expand the toolkit of applied statisticians by generalizing or creating new methods that are capable of solving new problems or solving existing problems more efficiently. They are statisticians who might get motivated by a problem someone else encountered in practice, but who then abstract the problem as much as possible so that their solution has as broad an impact as possible. Most statisticians are not planted firmly on either end of this spectrum, but instead find themselves moving around and adapting to the particular challenge they are facing. Another way to loosely categorize statisticians is in terms of industrial (or government) versus academic statisticians. Academic statisticians are primarily involved with innovative research and the teaching of statistics classes. Aside from Statistics departments, there are many alternative departments for academic statisticians including Mathematics, Economics, Business, Sociology and Psychology. Research goals for an academic statistician vary with their interests, and also depend on their emphasis toward either applied or theoretical research. In addition, the University at which they work can emphasize either a teaching or research mission that further dictates the quantity and type of research they engage in. In any case, it is a primary responsibility of an academic statistician to publish papers in leading statistics journals to advance the field. Teaching responsibilities can include introductory Statistics for undergraduate non-majors, core statistical
theory and methods classes for Statistics majors and in many cases advanced graduate-level Statistics classes for students pursuing an MS and/or PhD degree in Statistics. Industrial statisticians are frequently focused on problems that have some bearing on the company’s business. In some large companies there may be a fundamental research group that operates more like an academic environment, but in recent years the number and size of these groups are diminishing as corporations are more squarely focused on their bottom lines. Industrial statisticians are expected to assimilate the company culture and add value to the projects they work on that goes well beyond the contributions that their statistical skills alone enable. They might, for example, become project managers and even technical managers where their organizational, motivational, and leadership skills become important assets to the company. Many statisticians engage in statistical consulting, either as their primary vocation or as a part-time endeavor. Academic statisticians, for example, often have opportunities to lend their data analysis and quantitative problem solving skills to government and industry clients, and can contribute to litigation cases as an expert consultant or even an expert witness. Consultants must have exceptionally strong communication skills to be able to translate the interpretation of their findings into the language of the client. In the same way, they have to be able to elicit information from their clients that will ensure the efficacy of their data analyses. Industrial statisticians often function as internal consultants to the company they work for. This is particularly true in large companies where there can be a group of statisticians that serve as a shared central resource for the entire company. The following alphabetical list is meant to provide an appreciation of the diversity of fields where statisticians are gainfully employed: Agriculture, Archaeology, Astronomy, Biology, Chemistry, Computer Science, Demography, Economics, Ecology, Education, Engineering, Epidemiology, Finance, Forestry, Genetics, Health Sciences, Insurance, Law, Manufacturing, Medicine, National Defense, Pharmacology, Physics, Psychology, Public Health, Safety, Sociology, Sports, Telecommunications, and Zoology. To be more specific, consider the following brief descriptions of work and employment of statisticians in the following fields:
Medicine Florence Nightingale was not only a historic figure because of what she brought to the profession of nursing, but she was also a pioneering figure in the use of statistics. Statistical work in medicine involves designing studies
Careers in Statistics
and analyzing their data to determine if new (or existing) drugs, medical procedures and medical devices are safe and effective. Statisticians find careers at pharmaceutical companies, medical research centers and governmental agencies concerned with drugs, public health and medicine.
Ecology Research laboratories, commercial firms and government environmental agencies employ statisticians to evaluate the environmental impact of air, water and soil pollutants. Statisticians also work with government lawyers to analyze the impact (false positive or false negative) of proposed federal or state pollution tests and regulations.
Market Research Statisticians analyze consumer demand for products and services, analyze the effectiveness of various types of advertising, and analyze the economic risk of satisfying consumer demand for products and services.
C
Nuclear power plants, national defense agencies and airlines are just a few of the businesses that statistically analyze safety risks.
Telecommunications The reliability of voice and data networks is paramount to a telecommunication company’s revenue and their brand name image. Statisticians work collaboratively with engineers to model alternative design architectures and choose the most cost-effective design that minimizes customer-perceived downtime. Statisticians working in telecommunication companies frequently shift into new technology areas to keep up with the vastly changing landscape of high-tech companies. The authors have found that their careers in statistics involve work that is usually very interesting, often involves new ideas and learning experiences, and can definitely bring value to problem solving. For more information on careers in statistics consult www.amstat.org or e-mail
[email protected].
Manufacturing The success of manufacturing industries such as aerospace, electronics, automobile, chemical or other product producing industries depends, at least in part, on the efficiency of production and the quality and reliability of their products. Statistical techniques and models are used for predicting inventory needs, improving production flow, quality control, reliability prediction and improvement, and development of product warranty plans. The Deming Prize, named after the prolific statistician W. Edwards Deming, was established in and is annually awarded to companies that make major advances in quality improvement. The Malcolm Baldridge National Quality Award, named after Malcolm Baldridge who served as the United States Secretary of Commerce under President Regan, was established in and is annually awarded to U.S. organizations for performance excellence.
Actuarial Sciences Actuarial statisticians use Mathematics and Statistics to assess the risk of insurance and financial portfolios. Statistical methods are used, for example, to determine a wide variety of appropriate insurance premiums (e.g., homeowner, life, automobile, flood, etc.) and to manage investment and pension funds.
Safety Statisticians are employed by many businesses and agencies to model safety concerns and to estimate and predict the probability of occurrence of these safety concerns.
About the Authors Dr. Daniel R. Jeske is a Professor and Chair, Department of Statistics, University of California – Riverside, CA, and is the first Director of the Statistical Consulting Collaboratory at UCR. He has published over journal articles and over refereed conference papers. Prior to his academic career, he was a Distinguished Member of Technical Staff and a Technical Manager at AT&T Bell Laboratories. Concurrent with that, he was a part-time Visiting Lecturer in the Department of Statistics at Rutgers University. Currently, he is an Associate Editor for The American Statistician. Dr. Janet Myhre is President and Founder of Mathematical Analysis Research Corporation of Claremont, California. She is Professor Emeritus, Honorary Alumna, and Founder of the Reed Institute of Applied Statistics at Claremont McKenna College. She is a Fellow of the American Statistical Association and and has served as an Associate Editor of Technometrics and as Chair of the Committee on Careers in Statistics of the American Statistical Association.
Cross References 7Online Statistics Education 7Rise of Statistics in the Twenty First Century 7Role of Statistics 7Role of Statistics in Advancing Quantitative Education 7Statistics Education
C
C
Case-Control Studies
interest, is approximated well by the odds ratio if the disease is rare. If the overall probability of a case can be estimated from other sources, then this can be combined with the relative risk to give estimates of the absolute risk of disease for exposed (x = ) and non-exposed (x = ) groups. All this extends immediately to general β , all of whose components represent individual log odds ratios.
Case-Control Studies Alastair Scott, Chris Wild Professors The University of Auckland, Auckland, New Zealand
Introduction The basic aim of a case-control study is to investigate the association between a disease (or some other condition of interest) and potential risk factors by drawing separate samples of “cases” (people with the disease, say) and “controls” (people at risk of developing the disease). Let Y denote a binary response variable which can take values Y = , corresponding to a case, or Y = , corresponding to a control, and let x be a vector of explanatory variables or covariates. Our aim is to fit a binary regression model to explain the probabilistic behavior of Y as a function of the observed values of the explanatory variables recorded in x. We focus particularly on the logistic regression model (see 7Logistic Regression), logit{pr(Y ∣ x; β)} = log {
pr(Y = ∣ x; β) } pr(Y = ∣ x; β)
= β + xT β ,
()
since this makes the analysis of case-control data particularly simple and is the model of choice in most applications. In principle, the most straightforward way of obtaining data from which to build regression models for pr(Y ∣ x) would be to use a prospective sampling design. Here covariate information is ascertained for a cohort of individuals who are then tracked through time until the end of the study when whether they have contracted the disease (Y = ) or not (Y = ) is recorded. With prospective sampling designs, observation proceeds from covariates (explanatory variables) to response, corresponding to the logic underlying the modelling. With case-control sampling, the order is reversed, with data collection proceeding from response to covariates. The parameter β in Model () can still be estimated, however. Consider the simplest situation of a single binary covariate taking values x = or x = . Using Bayes Theorem, Cornfield () showed that the prospective pr(Y=∣x=) pr(Y=∣x=) / pr(Y=∣x=) , can be expressed as odds ratio, pr(Y=∣x=) pr(x=∣Y=) pr(x=∣Y=) / , which only involves quantities that pr(x=∣Y=) pr(x=∣Y=)
can be estimated directly from case-control data. Cornfield also pointed out that the relative risk, pr(Y = ∣ x = )/pr(Y = ∣ x = ), which is usually of more
Types of Case-Control Studies There are two broad types of case-control study, populationbased and matched, corresponding to two different ways of controlling for confounding variables. In the simplest form of population-based sampling sampling, random samples are drawn independently from the case- and control-subpopulations of a real, finite target population or cohort. Covariate information, x, is then ascertained for sampled individuals. Fitting logistic model () is particularly simple here. Following earlier work for discrete covariates, Prentice and Pyke () showed that we can get valid inferences about β by running the case-control data through a standard logistic regression program designed for prospective data. The intercept β , which is needed if we want to estimate the absolute risk for given values of the covariates, is completely confounded with the relative sampling rates of cases and controls but can be recovered using additional information such as the finite population totals of cases and controls. Prentice and Pyke extended this to stratified casecontrol sampling, where the target population is first split into strata on the basis of variables known for the whole population and separate case-control samples are drawn from each stratum. Again we get valid inferences about all the other coefficients by running the data through a prospective logistic regression program, provided that we introduce a separate intercept for each stratum into our model. Otherwise standard logistic programs need to be modified slightly to produce valid inferences (Scott andh Wild ). In designing a population-based study, it is important to make sure that the controls really are drawn from the same population, using the same protocols, as the cases. Increasingly, controls are selected using modern sample survey techniques, involving multi-stage sampling and varying selection probabilities, to help ensure this. The modifications needed to handle these complications are surveyed in Scott and Wild (). In a matched case-control study, each case is individually matched with one or more controls. This could be regarded as an limiting case of a stratified study with the
Case-Control Studies
strata so finely defined that each stratum includes only a single case. If we introduce an extra intercept for each matched set, then we can no longer use a simple logistic program since the plethora of parameters will lead to inconsistent parameter estimates. Instead we need to carry out a conditional analysis. More specifically, suppose that there are M controls in the jth matched set and model () is replaced by logit{pr(Y ∣ x; β)} = β j + xT β for these observations. Then the conditional probability that the covariates xj are those of the case and (xj , . . . , xjM ) are those of the M controls, given the set of M+ covariates can T be expressed in the form exp (xTj β ) / ∑M+ m= exp (x jm β ), which does not involve the intercept terms. Inferences about β can then be made from the conditional likelihood obtained when we combine these terms over all matched sets. With pair matching (M = ), this likelihood is identical to a simple logistic regression on the difference between the paired covariates. More sophisticated designs, including incidence density sampling, nested case-control studies and casecohort studies, that can handle complications such as time-varying covariates and 7survival data are discussed in other chapters in this volume.
Discussion Case-control sampling is a cost-reduction device. If we could afford to collect data on the whole finite population or cohort, then we would do so. There many practical difficulties that need to be overcome to run a successful case-control study; a good account of these is given in Breslow (). Despite this, the case-control study in its various forms is one of the most common designs in health research. In fact, Breslow and Day () described such studies as “perhaps the dominant form of analytical research in epidemiology” and since that time the rate of appearance of papers reporting on case-control studies has gone up by a factor of more than . These designs are also used in other fields, sometimes under other names. In econometrics, for example, the descriptor “choice-based” is used (see Manski and McFadden ()). There are several reasons for the popularity of casecontrol studies. The first is the simplicity of the logistic analysis outlined above. The other two reasons concern efficiency: time efficiency and statistical efficiency. The former comes from being able to use historical information immediately rather than having to follow individuals through time and then wait to observe an outcome as in a prospective study. The first chapter of Breslow and Day () has a good discussion of the attendant risks. The gain in statistical efficiency can be huge. For example,
C
suppose that we have a condition that affects only individual in on average and we wish to investigate the effect of an exposure that affects % of people. In this situation a case-control study with equal numbers of cases and controls has the same power for detecting a small increase in risk as a prospective study with approximately five times as many subjects. If the condition affects only one individual in then the prospective study would need times as many subjects!
About the Authors Professor Scott is Past President of the New Zealand Statistical Society (–). He is of one of New Zealand’s foremost mathematical scientists. He was founding head of the University of Auckland’s Department of Statistics (and previously Head of the Department of Mathematics and Statistics). He is an elected member of the International Statistical Institute, and one of honorary life members of the New Zealand Statistical Association. His paper with JNK Rao, published in the Journal of American Statistical Association, was selected as one of landmark papers in the history of survey sampling for the centenary volume of the International Association of Survey Statisticians. Professor Wild is, with Professor Scott, the only statistician in New Zealand that has been elected a Fellow of the American Statistical Association. He is a Past President of the International Association for Statistics Education (–). He is currently Editor of the International Statistical Review.
Cross References 7Medical Statistics 7Statistical Methods in Epidemiology
References and Further Reading Breslow NE () Statistics in epidemiology: the case-control study. J Am Stat Assoc :– Breslow NE () Case-control studies. In: Aherns W, Pigeot I (eds) Handbook of epidemiology. Springer, New York, pp – Breslow NE, Day NE () The analysis of case-control studies. International Agency for Research on Cancer, Lyon Cornfield J () A method of estimating comparitive rates from clinical data. J Natl Cancer Inst :– Manski CF, McFadden D (eds) () Structural analysis of discrete data with econometric applications. Wiley, New York. Models and case-control studies. Biometrika :– Scott AJ, Wild CJ () Fitting regression models to case-control data by maximum likelihood. Biometrika :– Scott AJ, Wild CJ () Population-based case control studies. In: Pfefferman D, Rao CR (eds) Ch in Handbook of statistics : sample surveys. Elsevier, Amsterdam, pp –
C
C
Categorical Data Analysis
Categorical Data Analysis Alan Agresti , Maria Kateri Distinguished Professor Emeritus University of Florida, Gainesville, FL, USA Associate Professor University of Ioannina, Ionnina, Greece
Introduction A categorical variable consists of a set of non-overlapping categories. Categorical data are counts for those categories. The measurement scale is ordinal if the categories exhibit a natural ordering, such as opinion variables with categories from “strongly disagree” to “strongly agree.” The measurement scale is nominal if there is no ordering. The types of possible analysis depend on the measurement scale. When the subjects measured are cross-classified on two or more categorical variables, the table of counts for the various combinations of categories is a contingency table. The information in a contingency table can be summarized and further analyzed through appropriate measures of association and models. A standard reference on association measures is Goodman and Kruskal (). Most studies distinguish between one or more response variables and a set of explanatory variables. When the main focus is on the association and interaction structure among a set of response variables, such as whether two variables are conditionally independent given values for the other variables, log-linear models are useful. More commonly, research questions focus on effects of explanatory variables on a categorical response variable. Logistic regression models (see 7Logistic Regression) are then of particular interest. For binary (success-failure) response variables, they describe the logit, which is log[P(Y = )/P(Y = )], using
Distribution) play the central role that the normal does for quantitative data. Models for categorical data assuming the binomial or multinomial were unified with standard regression and 7analysis of variance (ANOVA) models for quantitative data assuming normality through the introduction by Nelder and Wedderburn () of the generalized linear model (GLM, see 7Generalized Linear Models). This very wide class of models can incorporate data assumed to come from any of a variety of standard distributions (such as the normal, binomial, and Poisson). The GLM relates a function of the mean (such as the log or logit of the mean) to explanatory variables with a linear predictor.
Contingency Tables Two categorical variables are independent if the probability of response in any particular category of one variable is the same for each category of the other variable. The most well-known result on two-way contingency tables is the test of the null hypothesis of independence, introduced by Karl Pearson in . If X and Y are two categorical variables with I and J categories respectively, then their cross-classification leads to a I × J table of observed frequencies n = (nij ). Under this hypothesis, the expected cell frequencies equal mij = nπ i⋅ π⋅j , i = , . . . , I, j = , . . . , J, where n is the total sample size (n = ∑i,j nij ) and π i⋅ (π⋅j ) is the ith row ( jth column) marginal of the underlying probabilities matrix π = (π ij ). Then the corresponding ˆ ij = npi⋅ p⋅j = maximum likelihood (ML) estimates equal m ni⋅ n⋅j , where pij denotes the sample proportion in cell (i, j). n The hypothesis of independence is tested through Pearson’s chi-squared statistic
χ =
ˆ ij ) ∑i,j (nij − m . ˆ ij m
()
log[P(Y = )/P(Y = )] = a + β x + β x + ⋯ + β p xp where Y is the binary response variable and x , . . . , xp the set of the explanatory variables. For a nominal response Y with J categories, the model simultaneously describes log[P(Y = )/P(Y = J)], log[P(Y = )/P(Y = J)], . . . , log[P(Y = J − )/P(Y = J)]. For ordinal responses, a popular model uses explanatory variables to predict a logit defined in terms of a cumulative probability (McCullagh ), log[P(Y ≤ j)/P(Y > j)], j = , , . . . , J − . For categorical data, the binomial (see 7Binomial Distribution) and multinomial distributions (see 7Multinomial
The P-value is the right-tail probability above the observed χ value. The distribution of χ under the null hypothesis is approximated by a χ(I−)(J−) , provided that the individual expected cell frequencies are not too small. When a contingency table has ordered row or column categories (ordinal variables), specialized methods can take advantage of that ordering. More generally, models can be formulated that are more complex than independence, and expected frequencies mij can be estimated under the constraint that the ˆ ij are the corresponding maximum likelimodel holds. If m hood estimates, then, to test the hypothesis that the model holds, we can use the Pearson statistic () or the statistic that results from the standard statistical approach of
Categorical Data Analysis
conducting a likelihood-ratio test, which has test statistic G = ∑ nij ln ( i,j
nij ). ˆ ij m
()
Independence between the classification variables X and Y (i.e., mij = nπ i⋅ π⋅j , for all i and j) can be expressed in terms of a log-linear model as log(mij ) = λ + λ iX + λ jY ,
i = , . . . , I, j = , . . . , J.
The more general model that allows association between the variables is log(mij ) = λ + λ iX + λ jY + λ ijXY ,
C
see the on-line manual of Thompson (). Bayesian analysis of categorical data can be carried out through WinBUGS (http://wlww.mrc-bsu.cam.ac.uk/bugs/winbugs /contents.shtml). The standard reference on log-linear models is Bishop et al. (). For logistic regression, Hosmer and Lemeshow () is popular. A more comprehensive book dealing with categorical data analysis using various types of models and analyses is Agresti (), with Agresti () focusing on ordinal data.
About the Author
i = , . . . , I, j = , . . . , J. () Log-linear models describe the way the categorical variables and their association influence the count in each cell of the contingency table. They can be considered as a discrete analogue of ANOVA. The two-factor interaction terms relate to odds ratios describing the association. Associations can be modeled through simpler association models. The simplest such model, the linear-by-linear association model, is relevant when both classification variables are ordinal. It replaces the interaction term λ ijXY by the product ϕµ i ν j , where µ i and ν j are known scores assigned to the row and column categories respectively. This model is
Professor Agresti is recipient of the first Herman Callaert Leadership Award in Statistical Education and Dissemination, Hasselt University, Diepenbeek, Belgium (), and Statistician of the Year award, Chicago chapter of American Statistical Association (–). He received an honorary doctorate from De Montfort University in the U.K. in . He has presented invited lectures and short courses for universities and companies in about countries. Professor Agresti is author or coauthor of five textbooks, including the internationally respected text “Categorical Data Analysis." Professor Kateri won the Myrto Lefkopoulou award for her Ph.D. thesis in Greece and has since published extensively on methods for categorical data.
log(mij ) = λ + λ iX + λ jY + ϕµ i ν j ,
Cross References
i = , . . . , I, j = , . . . , J. () More general models treat one or both sets of scores as parameters. The special case of square I × I contingency tables with the same categories for the rows and the columns occurs with matched-pairs data. For example, such tables occur in the study of rater agreement and in the analysis of social mobility. A condition of particular interest for such data is marginal homogeneity, that π i⋅ = π⋅i, i = , . . . , I. For the × case of binary matched pairs, the test comparing the margins using the chi-squared statistic (n − n ) /(n + n ) is called McNemar’s test. The models for two-way tables extend to higher dimensions. The various models available vary in terms of the complexity of the association and interaction structure.
Inference and Software Standard statistical packages, such as SAS, R, and SPSS, are well suited for analyzing categorical data, mainly using maximum likelihood for inference. For SAS, a variety of codes are presented and discussed in the Appendix of Agresti (), and see also Stokes et al. (). For R,
7Algebraic Statistics 7Association Measures for Nominal Categorical Variables 7Chi-Square Test: Analysis of Contingency Tables 7Data Analysis 7Exact Inference for Categorical Data 7Generalized Linear Models 7Logistic Regression 7Variation for Categorical Variables
References and Further Reading Agresti A () Categorical data analysis, nd edn. Wiley, New York Agresti A () Analysis of ordinal categorical data, nd edn. Wiley, New York Bishop YMM, Fienberg SE, Holland PW () Discrete multivariate analysis: theory and practice. MIT Press, Cambridge Goodman LA, Kruskal WH () Measures of association for cross classifications. Springer, New York Hosmer DW, Lemeshow S () Applied logistic regression, nd edn. Wiley, New York McCullagh P () Regression models for ordinal data (with discussion). J R Stat Soc B :– Nelder J, Wedderburn RWM () Generalized linear models. J R Stat Soc A :– Stokes ME, Davis CS, Koch GG () Categorical data analysis using the SAS system, nd edn. SAS Institute, Cary
C
C
Causal Diagrams
Thompson LA () R (and S-PLUS) manual to accompany Agresti’s Categorical data analysis (), nd edn. https://home.comcast.net/~lthompson/Splusdiscrete.pdf
Causal Diagrams Sander Greenland , Judea Pearl Professor University of California-Los Angeles, Los Angeles, CA, USA Professor, Director of Cognitive Systems Laboratory University of California-Los Angeles, Los Angeles, CA, USA
From their inception, causal systems models (more commonly known as structural-equations models) have been accompanied by graphical representations or path diagrams that provide compact summaries of qualitative assumptions made by the models. These diagrams can be reinterpreted as probability models, enabling use of graph theory in probabilistic inference, and allowing easy deduction of independence conditions implied by the assumptions. They can also be used as a formal tool for causal inference, such as predicting the effects of external interventions. Given that the diagram is correct, one can see whether the causal effects of interest (target effects, or causal estimands) can be estimated from available data, or what additional observations are needed to validly estimate those effects. One can also see how to represent the effects as familiar standardized effect measures. The present article gives an overview of: () components of causal graph theory; () probability interpretations of graphical models; and () methodologic implications of the causal and probability structures encoded in the graph, such as sources of bias and the data needed for their control.
Introduction From their inception in the early twentieth century, causal models (more commonly known as structural-equations models) were accompanied by graphical representations or path diagrams that provided compact summaries of qualitative assumptions made by the models. Figure provides a graph that would correspond to any system of five equations encoding these assumptions: . independence of A and B . direct dependence of C on A and B . direct dependence of E on A and C
. direct dependence of F on C and . direct dependence of D on B, C, and E The interpretation of “direct dependence” was kept rather informal and usually conveyed by causal intuition, for example, that the entire influence of A on F is “mediated” by C. By the s it was recognized that these diagrams could be reinterpreted formally as probability models, enabling use of graph theory in probabilistic inference, and allowing easy deduction of independence conditions implied by the assumptions (Pearl ). By the s it was further recognized that these diagrams could also be used as tools for guiding causal and counterfactual inference (Pearl , ; Pearl and Robins ; Spirtes et al. ) and for illustrating sources of bias and their remedy in empirical research (Greenland et al. ; Greenland , ; Robins ; Greenland and Brumback ; Cole and Hernán ; Hernán et al. ; Jewell ; Pearl ; Glymour and Greenland ). Given that the graph is correct, one can see whether the causal effects of interest (target effects, or causal estimands) can be estimated from available data, or what additional observations are needed to validly estimate those effects. One can also see how to represent the effects as familiar standardized effect measures. The present article gives an overview of: () components of causal graph theory; () probability interpretations of graphical models; and () methodologic implications of the causal and probability structures encoded in the graph, such as sources of bias and the data needed for their control. See 7Causation and Causal Inference for discussion of definitions of causation and statistical models for causal inference.
Graphical Models and Causal Diagrams Basics of Graph Theory As befitting a well developed mathematical topic, graph theory has an extensive terminology that, once mastered, provides access to a number of elegant results which may be used to model any system of relations. The term dependence in a graph, usually represented by connectivity, may refer to mathematical, causal, or statistical dependencies. The connectives joining variables in the graph are called arcs, edge, or links, and the variables are also called nodes or vertices. Two variables connected by an arc are adjacent or neighbors and arcs that meet at a variable are also adjacent. If the arc is an arrow, the tail (starting) variable is the parent and the head (ending) variable is the child. In causal diagrams, an arrow represents a “direct effect” of the parent on the child, although this effect is direct only relative
Causal Diagrams
C
share ancestors that are not shown in the graph; A ↔ B then means that there is an unspecified variable U with directed paths to both A and B (e.g., A ← U → B).
Interpretations of Graphs Causal Diagrams. Fig. E ← C → D is open, E → A → C ← B → D is closed
to a certain level of abstraction, in that the graph omits any variables that might mediate the effect. A variable that has no parent (such as A and B in Fig. ) is exogenous or external, or a root or source node, and is determined only by forces outside of the graph; otherwise it is endogenous or internal. A variable with no children (such as D in Fig. ) is a sink or terminal node. The set of all parents of a variable X (all variables at the tail of an arrow pointing into X) is denoted pa[X]; in Fig. , pa[D] = {B, C, E}. A path or chain is a sequence of adjacent arcs. A directed path is a path traced out entirely along arrows tail-to-head. If there is a directed path from X to Y, X is an ancestor of Y and Y is a descendant of X. In causal diagrams, directed paths represent causal pathways from the starting variable to the ending variable; a variable is thus often called a cause of its descendants and an effect of its ancestors. In a directed graph the only arcs are arrows, and in an acyclic graph there is no feedback loop (directed path from a variable back to itself). Therefore, a directed acyclic graph or DAG is a graph with only arrows for edges and no feedback loops (i.e., no variable is its own ancestor or its own descendant). A variable intercepts a path if it is in the path (but not at the ends); similarly, a set of variables S intercepts a path if it contains any variable intercepting the path. Variables that intercept directed paths are intermediates or mediators on the pathway. A variable is a collider on the path if the path enters and leaves the variable via arrowheads (a term suggested by the collision of the arrows at the variable). Note that being a collider is relative to a path; for example in Fig. , C is a collider on the path A → C ← B → D and a noncollider on the path A → C → D. Nonetheless, it is common to refer to a variable as a collider if it is a collider along any path (i.e., if it has more than one parent). A path is open or unblocked at noncolliders and closed or blocked at colliders; hence a path with no collider (like E ← C ← B → D) is open or active, while a path with a collider (like E ← A → C ← B → D) is closed or inactive. Some authors use a bidirectional arc (two-headed arrow, ↔) to represent the assumption that two variables
Depending on assumptions used in its construction, graphical relations may be given three distinct levels of interpretation: probabilistic, causal, and functional. We now briefly describe these levels, providing further details in later sections. The probabilistic interpretation requires the weakest set of assumptions. It treats the diagram as a carrier of conditional independencies constraints on the joint distribution of the variables in the graph. To serve in this capacity, the parents pa[X] of each variable X in the diagram are chosen so as to render X independent of all its nondescendants, given pa[X]. When this condition holds, we say that the the diagram is compatible with the joint distribution. In Fig. , for example, variable E is assumed to be independent of its nondescendants {B, D, F} given its parents pa[E] = {A, C}. We will see that compatibility implies many additional independencies (e.g., E and F are independent given C) that could be read from the diagram by tracing its paths. In real-life problems, compatibility arises if each parent–child family {X, pa[X]} represents a stochastic process by which nature determines the probability of the child X as a function of the parents pa[X], independently of values previously assigned to variables other than the parents. To use diagrams for causal inference, we must assume that the direction of the arrows correspond to the structure of the causal processes generating the data. More specifically, the graph becomes a causal diagram if it encodes the assumption that for each parent–child family, the conditional probability Pr(x∣pa[X]) would remain the same regardless of whether interventions take place on variables not involving {X, pa[X]}, even if they are ancestors or descendants of X. In Fig. , for example, the conditional probability P(C∣A, B) is assumed to remain invariant under manipulation of the consequences of C, i.e., E, F or D. A causal DAG represents a complete causal structure, in that all sources of causal dependence are explained by causal links; in particular, it is assumed that all common (shared) causes of variables in the graph are also in the graph, so that all exogenous variables (root nodes) are causally independent (although they may be unobserved). If we assume further that the arrows represent functional relationships, namely processes by which nature assigns a definite value to each internal node, the diagram can then be used to process counterfactual information and display independencies among potential outcomes
C
C
Causal Diagrams
(including counterfactual variables) (Pearl , Chap. ). We will describe such structural diagrams and potential outcomes below.
Control: Manipulation Versus Conditioning The word “control” is used throughout science, but with a variety of meanings that are important to distinguish. In experimental research, to control a variable C usually means to manipulate or set its value. In observational studies, however, to control C (or more precisely, to control for C) more often means to condition on C, usually by stratifying on C or by entering C in a regression model. The two processes are very different physically and have very different representations and implications (Pearl ; Greenland et al. ). If a variable X is influenced by a researcher, a realistic causal diagram would need an ancestor R of X to represent this influence. In the classical experimental case in which the researcher alone determines X, R and X would be identical. In human trials, however, R more often represents just an intention to treat (with the assigned level of X), leaving X to be influenced by other factors that affect compliance with the assigned treatment R. In either case, R might be affected by other variables in the graph. For example, if the researcher uses age to determine assignments (an age-biased allocation), age would be a parent of R. Ordinarily however R would be exogenous, as when R represents a randomized allocation. In contrast, by definition in an observational study there is no such variable R representing the researcher’s influence on X. Conditioning is often used as a substitute for experimental control, in the hopes that with sufficient conditioning, X will be independent of uncontrolled influences. Conditioning on a variable C closes open paths that pass through C. However, if C is a collider, conditioning on C opens paths that were blocked by C or by an ancestral collider A. In particular, conditioning on a variable may open a path even if it is not on the path, as with F in Figs. and . To illustrate conditioning in a graph, we will redraw the graph to surround conditioned variables with square brackets (conditioned variables are often circled instead). We may now graphically determine the status of paths after conditioning by regarding the path open at colliders that are bracketed or have bracketed descendants, open at unbracketed noncolliders, and closed elsewhere. Figure shows Fig. after conditioning on C, from which we see that the E − D paths E ← C ← B → D and E ← A → C → D have been blocked, but the path E ← A → C ← B → D has been opened. Were we to condition on F but not C, no open
Causal Diagrams. Fig. Conditional on C, E ← C → D is closed but E → A → C ← B → D is open
Causal Diagrams. Fig. Conditional on F, E ← C → D and E → A → C ← B → D are both open
path would be blocked, but the path E ← A → C ← B → D would again be opened. The opening of paths at conditioned colliders reflect the fact that we should expect two unconditionally independent causes A and B become dependent if we condition on their consequences, which in Fig. are C and F. To illustrate, suppose A and B are binary indicators (i.e., equal to or ), marginally independent, and C = A+B. Then among persons with C = , some will have A = , B = and some will have A = , B = (because other combinations produce C ≠ ). Thus when C = , A and B will exhibit perfect negative dependence: A = − B for all persons with C = . The distinction between manipulation and conditioning is brought to the fore when considering the notion of “holding a variable constant.” Conditioning on a variable X means that we choose to narrow the scope of discussion to those situations only where X attains a given value, regardless of how that value is attained. Manipulating X means that we physically intervene and set X to a given value, say X = x. The difference can be profound. For example, in cancer screening, conditioning on the absence of lighters and matches in the home lowers dramatically the probability of finding lung cancer, because restricting our attention to those who do not have these tools for smoking is tantamount to examining nonsmokers. In contrast, removing lighters and matches from people’s homes during the screening will not lower the probability of finding lung cancer, since any lung cancers present will be unaffected by this act. Likewise, conditional on a low barometer reading we will have a lower probability of rain than
Causal Diagrams
unconditionally, but setting the barometer to a low reading (e.g., by pushing its needle down) will have no effect on the weather.
Graphical Representation of Manipulation One way of representing manipulation in the graph is to simulate the act of setting X to a constant, or the immediate implications of that act. If prior to intervention the probability of X is influenced by its parents via P(x∣pa[X]), such influence no longer exists under an intervention that is made without reference to the parents or other variables. In that case, physically setting X at x dislodges X from the influence of its parents and subjects it to a new influence that keeps its value at X = x regardless of the values taken by other variables. This can be represented by cutting out all arrows pointing to X and thus creating a new graph, in which X is an exogenous (root) node, while keeping the rest of the graph (with its associated conditional probabilities) intact. For example, setting C to a constant in Fig. , will render E and D independent, because all E − D paths will be blocked by such intervention, including E ← A → C ← B → D, even though the latter path would be opened by conditioning on C. On the other hand, manipulating F but not C would leave all E − D paths intact, and the E − D association will therefore remain unaltered. Assuming the graph is correct, graphical representation of interventions by deleting arrows enables us to compute post-intervention distributions from preintervention distributions (Pearl , , ; Spirtes et al. ; Lauritzen ) for a wide variety of interventions, including those that have side effects or that are conditioned upon other variables in the graph (Pearl , pp. , ). Nonetheless, “holding X constant” does not always correspond to a physically feasible manipulation, not even conceptually. Consider systolic blood pressure (SBP) as a cause of stroke (Y). It is easy to “hold SBP constant” in the sense of conditioning on each of its observed values. But what does it mean to “hold SBP constant” in the manipulative sense? There is only one condition under which SBP is constant: Death, when it stays put at zero. Otherwise, SBP is fluctuating constantly in some strictly positive range in response to posture, activity, and so on. Furthermore, no one knows how to influence SBP except by interventions R which have side effects on stroke (directed paths from R to Y that do not pass through SBP). Yet these side effects vary dramatically with intervention (e.g., there are vast differences between exercise versus medication side effects). On the other hand, consider the problem of estimating the causal effect of SBP on the rate of blood flow in a given blood vessel. At this physiological level of discussion
C
we can talk about the effect on blood flow of “changing SBP from level s to level s′ ,” without specifying any mechanism for executing that change. We know from basic physics that the blood flow in a vessel depends on blood pressure, vessel diameter, blood viscosity, and so on; and we can ask what the blood flow would be if the blood pressure were to change from s to s′ while the other factors remained at their ambient values. Comparing the results from conditioning on SBP = s versus conditioning on SBP = s′ would not give us the desired answer because these conditioning events would entail different distributions for the causes (ancestors) of SBP, some of which might also affect those determinants of flow which we wish held constant when comparing. We may thus conclude that there are contexts in which it makes no practical sense to speak of “holding X constant” via manipulation. In these contexts, manipulation of a given variable X can only be represented realistically by an additional node R representing an actual intervention, which may have side effects other than those intended or desired. On the other hand, such an R node will be redundant if X itself is amenable to direct manipulation. For such an X, manipulation can be represented by removing the arrows ending in X which correspond to effects overridden by the manipulation (Pearl , , ; Spirtes et al. ; Lauritzen ). When X is completely randomized or held constant physically, this corresponds to removing all arrows into X. The phrase “holding X constant” may also be meaningful when X is not directly manipulable. In these cases, we may still be able to estimate a causal effect of X if we can find an instrumental variable Z (a variable that is associated with X but not with any uncontrolled confounding variable U, and Z has no effect on Y except through X). Although the operational meaning of these effects is not immediately apparent when direct manipulation of X free of side effects is not conceivable, estimation of these effects can help judge proposed interventions that affect Y via effects on X.
Separation The intuition of closing and opening paths by conditioning is captured by the concept of “separation” which will be defined next. We say that a path is blocked by a set S if the path contains either an arrow-emitting node that is in S, or a collider that is outside S and has no descendant in S. Two variables (or sets of variables) in the graph are d-separated (or just separated) by a set S if, after conditioning on S, there is no open path between them. Thus S d-separates X from Y if S blocks all paths from X to Y. In Fig. , {A, C} d-separates E from B, but {C} does
C
C
Causal Diagrams
not (because conditioning on C alone results in Fig. , in which E and B are connected via the open path A). In a causal DAG, pa[X] d-separates X from every variable that is not affected by X (i.e., not a descendant of X). This feature of DAGs is sometimes called the “Markov condition,” expressed by saying the parents of a variable “screen off ” the variable from everything but its effects. Thus in Fig. pa[E] = {A, C}, which d-separates E from B but not from D. In a probability graph, d-separation of X and Y by S implies that X and Y are independent given S in any distribution compatible with graph. In a causal diagram, d-separation of X and Y by S implies additionally that manipulation of X will not alter the distribution of Y if the variables in S are held constant physically (assuming this can be done). More generally, the distribution of Y will remain unaltered by manipulation of X if we can hold constant physically a set S that intercepts all directed paths from X to Y, even if S does not d-separate X and Y. This is so because only descendants of X can be affected by manipulation of X. In sharp contrast, conditioning on X may change the probabilities of X’s ancestors; hence the stronger condition of d-separation by S is required to insure that conditioning on X does not alter the distribution of Y given S.
Statistical Interpretations and Applications Earlier we defined the notion of compatibility between a joint probability distribution for the variables in a graph and the graph itself. It can be shown that compatibility is logically equivalent to requiring that two sets of variables are independent given S whenever S separates them in the graph. Moreover these conditional independencies constitute the only testable implications of a causal model specified by the diagram (Pearl , p. ). Thus, given compatibility, two sets of variables will be independent in the distribution if there is no open path between them in the graph. Many special results follow for distributions compatible with a DAG. For example, if in a DAG, X is not an ancestor of any variable in a set T, then T and X will be independent given pa[X]. A distribution compatible with a DAG thus can be reduced to a product of factors Pr(x∣pa[X]), with one factor for each variable X in the DAG; this is sometimes called the “Markov factorization” for the DAG. When X is a treatment, this condition implies the probability of treatment is fully determined by the parents of X, pa[X]. Algorithms are available for constructing DAGs that are compatible with a given distribution (Pearl , pp. –).
Some of the most important constraints imposed by a graphical model on a compatible distribution correspond to the independencies implied by absence of open paths; e.g., absence of an open path from A to B in Fig. constrains A and B to be marginally independent (i.e., independent if no stratification is done). Nonetheless, the converse does not hold; i.e., presence of an open path allows but does not imply dependency. Independence may arise through cancellation of dependencies; as a consequence even adjacent variables may be marginally independent; e.g., in Fig. , A and E could be marginally independent if the dependencies through paths A → E and A → C → E cancelled each other. The assumption of faithfulness, discussed below, is designed to exclude such possibilities.
Bias and Confounding Usually, the usage of terms like “bias,” “confounding” and related concepts refer to dependencies that reflect more than just the effect under study. To capture these notions in a causal graph, we say that an open path between X and Y is a biasing path if it is not a directed path. The association of X with Y is then unbiased for the effect of X on Y if the only open paths from X to Y are the directed paths. Similarly, the dependence of Y on X is unbiased given S if, after conditioning on S, the open paths between X and Y are exactly (only and all) the directed paths in the starting graph. In such a case we say S is sufficient to block bias in the X − Y dependence, and is minimally sufficient if no proper subset of S is sufficient. Informally, confounding is a source of bias arising from causes of Y that are associated with but not affected by X (see 7Confounding). Thus we say an open nondirected path from X to Y is a confounding path if it ends with an arrow into Y. Variables that intercept confounding paths between X and Y are confounders. If a confounding path is present, we say confounding is present and that the dependence of Y on X is confounded. If no confounding path is present we say the dependence is unconfounded, in which case the only open paths from X to Y through a parent of Y are directed paths. Similarly, the dependence of Y on X is unconfounded given S if, after conditioning on S, the only open paths between X and Y through a parent of Y are directed paths. An unconfounded dependency may still be biased due to nondirected open paths that do not end in an arrow into Y. These paths can be created when one conditions on a descendant of both X and Y, or a descendant of a variable intercepting a directed path from X to Y (Pearl , p. ). The resulting bias is called Berksonian bias, after its discoverer Joseph Berkson (Rothman et al. ). Most epidemiologists call this type of bias “selection bias” (Rothman et al. ) while computer scientists refer to
Causal Diagrams
it as “explaining away” (Pearl ). Nonetheless, some writers (especially in econometrics) use “selection bias” to refer to confounding, while others call any bias created by conditioning “selection bias”. Consider a set of variables S that contains no effect (descendant) of X or Y. S is sufficient to block confounding if the dependence of Y on X is unconfounded given S. “No confounding” thus corresponds to sufficiency of the empty set. A sufficient S is called minimally sufficient to block confounding if no proper subset of S is sufficient. The initial exclusion from S of descendants of X or Y in these definitions arises first, because conditioning on Xdescendants can easily block directed (causal) paths that are part of the effect of interest, and second, because conditioning on X or Y descendants can unblock paths that are not part of the X − Y effect, and thus create new bias. These considerations lead to a graphical criterion called the back-door criterion which identifies sets S that are sufficient to block bias in the X − Y dependence (Pearl , ). A back-door path from X to Y is a path that begins with a parent of X (i.e., leaves X from a “back door”) and ends at Y. A set S then satisfies the back-door criterion with respect to X and Y if S contains no descendant of X and there are no open back-door paths from X to Y after conditioning on S. In a unconditional DAG, the following properties hold (Pearl , ; Spirtes et al. ; Glymour and Greenland ): . . . . . .
All biasing paths are back-door paths. The dependence of Y on X is unbiased whenever there are no open back-door paths from X to Y. If X is exogenous, the dependence of any Y on X is unbiased. All confounders are ancestors of either x or of y. A back-door path is open if and only if it contains a common ancestor of X and Y. If S satisfies the back-door criterion, then S is sufficient to block X − Y confounding.
These conditions do not extend to conditional DAGs like Fig. . Also, although pa[X] always satisfies the backdoor criterion and hence is sufficient in a DAG, it may be far from minimal sufficient. For example, there is no confounding and hence no need for conditioning whenever X separates pa[X] from Y (i.e., whenever the only open paths from pa[X] to Y are through X). As a final caution, we note that the biases dealt with by the above concepts are only confounding and selection biases. To describe biases due to measurement error and model-form misspecification, further nodes representing
C
mismeasured or misspecifed variables must be introduced (Glymour and Greenland ).
Estimation of Causal Effects Suppose now we are interested in the effect of X on Y in a causal DAG, and we assume a probability model compatible with the DAG. Then, given a sufficient set S, the only source of association between X and Y within strata of S will be the directed paths from X to Y. Hence the net effect of X = x vs. X = x on Y when S = s is defined as Pr(y∣x , s) − Pr(y∣x , s), the difference in risks of Y = y at X = x and X = x . Alternatively one may use another effect measure such as the risk ratio Pr(y∣x , s)/Pr(y∣x , s). A standardized effect is a difference or ratio of weighted averages of these stratum-specific Pr(y∣x, s) over S, using a common weighting distribution. The latter definition can be generalized to include intermediate variables in S by allowing the weighting distribution to causally depend on X. Furthermore, given a set Z of intermediates along all directed paths from X to Y and identification of the X − Z and Z − Y effects, one can produce formulas for the X − Y effect as a function of the X − Z and Z − Y effects (“front-door adjustment” (Pearl , )). The above form of standardized effect is identical to the forms derived under other types of causal models, such as potential-outcome models (see 7Causation and Causal Inference). In those models, the outcome Y of each unit is replaced by a vector of outcomes Y containing components Yx , where Yx represents the outcome when X = x is the treatment given. When S is sufficient, some authors (Pearl ) go so far as to identify the Pr(y∣x, s) with the distribution of potential outcomes Yx given S, thereby creating a structural model for the potential outcomes. If the graph is based on functional rather than probabilistic relationships between parents and children, this identification can also model unit-based counterfactuals Yx (u) for any pair (X, Y), where u is a unit index or a vector of exogeneous variables characterizing the units. There have been objections to this identification on the grounds that not all variables in the graph can be manipulated, and that potential-outcome models do not apply to nonmanipulable variables. The objection loses force when X is an intervention variable, however. In that case, sufficiency of a set S implies that the marginal potential-outcome distribution Pr(Yx = y) equals ∑s Pr(y∣x, s)Pr(s), which is the risk of Y = y given X = x standardized to the S distribution. In fact, sufficiency of S implies the stronger condition of strong ignorability given S, which says that X and the vector Y of potential outcomes are independent given S. In particular, strong ignorability given S follows if S
C
C
Causal Diagrams
satisfies the back-door criterion, or if X is randomized given S. Nonetheless, for the equation Pr(Yx = y) = ∑s Pr(y∣x, s)Pr(s) it suffices that X be independent of each component potential outcome Yx given S, a condition sometimes called weak ignorability given S.
Identification of Effects and Biases To check sufficiency and identify minimally sufficient sets of variables given a graph of the causal structure, one need only see whether the open paths from X to Y after conditioning are exactly the directed paths from X to Y in the starting graph. Mental effort may then be shifted to evaluating the reasonableness of the causal independencies encoded by the graph, some of which are reflected in conditional independence relations. This property of graphical analysis facilitates the articulation of necessary background knowledge for estimating effects, and eases teaching of algebraically difficult identification conditions. As an example, spurious sample associations may arise if each variable affects selection into the study, even if those selection effects are independent. This phenomenon is a special case of the collider-stratification effect illustrated earlier. Its presence is easily seen by starting with a DAG that includes a selection indicator F = for those selected, otherwise, as well as the study variables, then noting that we are always forced to examine associations within the F = stratum (i.e., by definition, our observations stratify on selection). Thus, if selection (F) is affected by multiple causal pathways, we should expect selection to create or alter associations among the variables. Figure displays a situation common in randomized trials, in which the net effect of E on D is unconfounded, despite the presence of an uncontrolled cause U of D. Unfortunately, a common practice in health and social sciences is to stratify on (or otherwise adjust for) an intermediate variable F between a cause E and effect D, and then claim that the estimated (F-residual) association represents that portion of the effect of E on D not mediated through F. In Fig. this would be a claim that, upon stratifying on the collider F, the E − D association represents the direct effect of E on D. Figure however shows the graph conditional on F, in which we see that there is now an open path from E to D through U, and hence the residual E − D association is confounded for the direct effect of E on D. The E − D confounding by U in Fig. can be seen as arising from the confounding of the F − D association by U in Fig. . In a similar fashion, conditioning on C in Fig. opens the confounding path through A, C, and B as seen in Fig. ; this path can be seen as arising from the confounding of the C − E association by A and the C − D association by B in Fig. . In both examples, further stratification on
Causal Diagrams. Fig. E → F → D is open, E → F ← U → D is closed
Causal Diagrams. Fig. Conditional on F, E → F → D is closed but E → F ← U → D is open
either A or B blocks the created path and thus removes the new confounding. Bias from conditioning on a collider or its descendant has been called “collider bias” (Greenland ; Glymour and Greenland ). Starting from a DAG, there are two distinct forms of this bias: Confounding induced in the conditional graph (Figs. , , and ), and Berksonian bias from conditioning on an effect of X and Y. Both biases can in principle be removed by further conditioning on certain variables along the biasing paths from X to Y in the conditional graph. Nonetheless, the starting DAG will always display ancestors of X or Y that, if known, could be used remove confounding; in contrast, no variable need appear or even exist that could be used to remove Berksonian bias. Figure also provides a schematic for estimating the F − D effect, as in randomized trials in which E represents assignment to or encouragement toward treatment F. In this case E acts as an instrumental variable (or instrument), a variable associated with F such that every open path from E to D includes an arrow pointing into F (Pearl ; Greenland ; Glymour and Greenland ). Although the F − D effect is not generally estimable, using the instrument E one can put bounds on confounding of the F − D association, or use additional assumptions that render the effect of F on D estimable.
Questions of Discovery While deriving statistical implications of graphical models is uncontroversial, algorithms that claim to discover causal (graphical) structures from observational data have been
Causal Diagrams
subject to strong criticism (Freedman and Humphreys ; Robins and Wasserman ). A key assumption in certain “discovery” algorithms is a converse of compatibility called faithfulness Spirtes et al. . A compatible distribution is faithful to the graph (or stable Pearl ()) if for all X, Y, and S, X and Y are independent given S only when S separates X and Y (i.e., the distribution contains no independencies other than those implied by graphical separation). Faithfulness implies that minimal sufficient sets in the graph will also be minimal for consistent estimation of effects. Nonetheless, there are real examples of near cancellation (e.g., when confounding obscures a real effect), which make faithfulness questionable as a routine assumption. Fortunately, faithfulness is not needed for the uses of graphical models discussed here. Whether or not one assumes faithfulness, the generality of graphical models is purchased with limitations on their informativeness. Causal diagrams show whether the effects can be estimated from the given information, and can be extended to indicate effect direction when that is monotone VanderWeele and Robins ;. Nonetheless, the nonparametric nature of the graphs implies that parametric concepts like effect-measure modification (heterogeneity of arbitrary effect measures) cannot be displayed by the basic graphical theory. Similarly, the graphs may imply that several distinct conditionings are minimal sufficient (e.g., both {A, C} and {B, C} are sufficient for the ED effect in Fig. ), but offer no further guidance on which to use. Open paths may suggest the presence of an association, but that association may be negligible even if nonzero. Because association transmitted by an open path may become attenuated as the length of the path increases, there is often good reason to expect certain phenomena (such as the conditional E − D confounding shown in Figs. , and ) to be small in practical terms.
Further Readings Full technical details of causal diagrams and their relation to causal inference can be found in the books by Pearl () and Spirtes et al. (). A compact survey is given in Pearl (). Less technical reviews geared toward health scientists include Greenland et al. (), Greenland and Brumback (), and Glymour and Greenland ().
About the Authors For Dr. Greenland’s biography see the entry 7Confounding and Confounder Control. Dr. Pearl is Professor of Computer Science at the University of California in Los Angeles, and Director of UCLA’s Cognitive Systems Laboratory. He is one of
C
the pioneers of Bayesian networks and the probabilistic approach to artificial intelligence. He is a member National Academy of Engineering () and Corresponding Member, Spanish Academy of Engineering (). Professor Pearl was awarded the Lakatos Award, London School of Economics and Political Science for “an outstanding contribution to the philosophy of science” (); the ACM Allen Newell Award for “groundbreaking contributions that have changed the scientific world beyond computer science and engineering. Dr. Pearl made seminal contributions to the field of artificial intelligence that extend to philosophy, psychology, medicine, statistics, econometrics, epidemiology and social science.”; the Benjamin Franklin Medal in Computers and Cognitive Science for “creating the first general algorithms for computing and reasoning with uncertain evidence, allowing computers to uncover associations and causal connections hidden within millions of observations. His work has had a profound impact on artificial intelligence and statistics, and on the application of these fields to a wide range of problems in science and engineering” (). He has written three fundamental books in artificial intelligence: Heuristics: Intelligent Search Strategies for Computer Problem Solving (AddisonWesley, ), Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan-Kaufmann, ) and Causality: Models, Reasoning, and Inference (Cambridge University Press, ). Profeshhhhsor Pearl holds two Honorary Doctorates.
Cross References 7Causation and Causal Inference 7Confounding and Confounder Control 7Principles Underlying Econometric Estimators for Iden-
tifying Causal Effects 7Rubin Causal Model 7Structural Equation Models
References and Further Reading Cole S, Hernán MA () Fallibility in estimating direct effects. Int J Epidemiol :– Freedman DA, Humphreys P () Are there algorithms that discover causal structure? Synthese :– Glymour MM, Greenland S () Causal diagrams. Ch. . In: Rothman KJ, Greenland S, Lash TL (eds) Modern epidemiology, rd edn. Lippincott, Philadelphia Greenland S () An introduction to instrumental variables for epidemiologists. Int J Epidemiol :– (Erratum: , , ) Greenland S () Quantifying biases in causal models: classical confounding versus collider-stratification bias. Epidemiology :– Greenland S, Brumback BA () An overview of relations among causal modelling methods. Int J Epidemiol :–
hChh
C
Causation and Causal Inference
Greenland S, Pearl J, Robins JM () Causal diagrams for epidemiologic research. Epidemiology :– Hernán MA, Hernandez-Diaz S, Werler MM, Mitchell AA () Causal knowledge as a prerequisite for confounding evaluation. Am J Epidemiol :– Jewell NP () Statistics for epidemiology. Chapman and Hall/CRC Press, Boca Raton, Sect. . Lauritzen SL () Causal inference from graphical models. In: Cox DR, Kluppelberg C (eds) Complex stochastic systems. Chapman and Hall/CRC Press, Boca Raton, pp – Pearl J () Probabilistic reasoning in intelligent systems. Morgan Kaufmann, San Mateo Pearl J () Causal diagrams for empirical research (with discussion). Biometrika :– Pearl J () Causality. Cambridge University Press, New York. nd edition, Pearl J () Causal inference in statistics: an overview. Statist Surv :– Pearl J, Robins JM () Probabilistic evaluation of sequential plans from causal models with hidden variables. In: Proceedings of the eleventh conference annual conference on uncertainty in artificial intelligence (UAI-). Morgan Kaufmann, San Francisco, pp – Robins JM () Data, design, and background knowledge in etiologic inference. Epidemiology :– Robins JM, Wasserman L () On the impossibility of inferring causation from association without background knowledge. In: Glymour C, Cooper G (eds) Computation, causation, and discovery. AAAI Press/The MIT Press, Menlo Park/Cambridge, pp – Rothman KJ, Greenland S, Lash TL () Modern epidemiology, rd edn. Lippincott, Philadelphia Spirtes P, Glymour C, Scheines R () Causation, prediction, and search, nd edn. MIT Press, Cambridge VanderWeele, TJ, Robins JM () Signed directed acyclic graphs for causal inference. J R Stat Soc Ser B ():–
Causation and Causal Inference Sander Greenland Professor University of California-Los Angeles, Los Angeles, CA, USA
In the health sciences, definitions of cause and effect have not been tightly bound with methods for studying causation. Indeed,many approaches to causal inference require no definition, leaving users to imagine causality however they prefer. As Sir Austin Bradford Hill said in his famous article on causation: “I have no wish. . .to embark upon a philosophical discussion of the meaning of ‘causation”’ (Hill ). Without a formal definition of causation, an association is distinguished as causal only by having been identified as such based on external and largely subjectmatter considerations, such as those Hill put forth.
Nonetheless, beneath most treatments of causation in the health sciences, one may discern a class of definitions built around the ideas of counterfactuals or potential outcomes. These ideas have a very long history and form the foundation of most current statistical methods for causal inference. Thus, the present article will begin with these definitions and the methods they entail. It will then turn methods that explicitly presume no definition of causation but rather begin with an idea of what a causal association should look like (perhaps derived from subject-matter judgments, including consideration of possible counterfactuals), and employ statistical methods to estimate those associations.
Counterfactuals and Potential Outcomes Skeptical that induction in general and causal inference in particular could be given a sound logical basis, David Hume nonetheless captured the foundation of the potential-outcome approach when he wrote 7 We may define a cause to be an object, followed by another, . . . where, if the first object had not been, the second had never existed. (Hume , p. )
A key aspect of this view of causation is its counterfactual element: It refers to how a certain outcome event (the “second object,” or effect) would not have occurred if, contrary to fact, an earlier event (the “first object,” or cause) had not occurred. In this regard, it is no different from conventional statistics, which refers to samples that might have occurred, but did not. This counterfactual view of causation was adopted by numerous philosophers and scientists after Hume (e.g., Mill ; Fisher ; Cox ; Simon and Rescher ; MacMahon and Pugh ; Stalnaker ; Lewis ). The development of this view into a statistical theory with methods for causal inference is recounted by Rubin (), Greenland et al. (), Greenland (), and Pearl (). The earliest such theories were developed in the s by Fisher, Neyman, and others for the analysis of randomized experiments and are today widely recognized under the heading of potential-outcome models of causation (also known in engineering as destructive-testing models). Suppose we wish to study the effect of an intervention variable X on a subsequent outcome variable Y defined on an observational unit or a population; for example, X could be the daily dose regimen for a drug in a clinical trial, and Y could be survival time. Given X has potential values x , . . . , xJ (e.g., drug doses), we suppose that there is a list of potential outcomes y = (y(x ), . . . , y(xJ ))′ such that if X = xj then Y = y(xj ). The list y thus exhibits
Causation and Causal Inference
the correspondence between treatments, interventions, or actions (the X values) and outcomes or responses (the Y values) for the unit, and so is sometimes called a response schedule (Berk ). A simpler and common notation has y = (y , . . . , yJ )′ , with Yj denoting the random variable “outcome when treated with X = xj .” Under this model, assignment of a unit to a treatment level xj is a choice of which potential outcome y(xj ) from the list y to attempt to observe. It is ordinarily assumed that the assignments made for other units do not affect the outcomes of another unit, although there are extensions of the model to include between-unit interactions, as in contagious outcomes (Halloran and Struchiner ). Regardless of the X assignment, the remaining potential outcomes are treated as existing pre-treatment covariates on which data are missing (Rubin , ). Because at most one of the J potential outcomes is observed per unit, the remaining potential outcomes can be viewed as missing data, and causal inference can thus be seen as a special case of inference with missing data. To say that intervention xi causally affects Y relative to intervention xj means that y(xi ) ≠ y(xj ), i.e., X “matters” for Y for the unit. The sharp (or strong) null hypothesis is that y(x) is constant over x within units. This hypothesis states that changing X would not affect the Y of any unit, i.e., y(xi ) = y(xj ) for every unit and every xi and xj ; it forms the basis of exact 7permutation tests such as 7Fisher’s exact test (Greenland ). The effect of intervention xi relative to xj on a unit may be measured by the difference in potential outcomes y(xi ) − y(xj ). If the outcome is strictly positive (like life expectancy or mortality risk), it could instead be measured by the ratio y(xi )/y(xj ). Because we never observe two potential outcomes on a unit, we can only estimate population averages of the potential outcomes. We do this by observing average outcomes in differently exposed groups and substituting those observations for the average potential outcomes in the group of interest – a perilous process whenever the observed exposure groups are atypical of the population of interest with respect to other risk factors for the outcome (Maldonado and Greenland ) (see Confounding and Confounder Control). A more subtle problem is that only for difference measures will the population effect (the difference of average potential outcomes) equal the population average effect (the average difference of potential outcomes). Hence the average of the differences y(xi ) − y(xj ) in the population is often called the average causal effect (ACE) (Angrist et al. ). For some popular measures of effect, such as rate ratios and odds ratios, the population effect may not even equal any average of individual effects (Greenland , ; Greenland et al. ).
C
The theory extends to probabilistic outcomes by replacing the y(xj ) by probability functions pj (y) (Greenland ; Robins ; Greenland et al. ). The theory also extends to continuous X by allowing the potential-outcome list y to contain the potential outcome y(x) or px (y) for every possible value x of X. Both extensions are embodied in Pearl’s notation for intervention effects, in which px (y) becomes P(Y = y∣set[X = x]) or P(Y = y∣do[X = x]) (Pearl , ). Finally, the theory extends to complex longitudinal data structures by allowing the treatments to be different event histories or processes (Robins , ).
From Randomized to Observational Inference Potential outcomes were developed part of a design-based strategy for causal inference in which 7randomization provided the foundation for inference. Indeed, before the s, the model was often referred to as “the randomization model,” even though the causal concepts within it do not hinge on randomization (e.g., Wilk ; Copas ). It thus seems that the early strong linkage of potential outcomes to randomized designs deflected consideration of the model for observational research. In the s, however, a number of philosophers used counterfactuals to build general foundations for causal analysis (e.g., Simon and Rescher ; Stalnaker ; Lewis ). Similar informal ideas can be found among epidemiologists of the era (e.g., MacMahon and Pugh ), and conceptual models subsuming counterfactuals began to appear shortly thereafter (e.g., Miettinen ; Rothman ; Hamilton ). The didactic value of these models was quickly apparent in the clarification they brought to ideas of strength of effect, synergy, and antagonism (MacMahon and Pugh ; Rothman ; see also Rothman et al. , Chaps. and ). Most importantly, the models make clear distinctions between causal and statistical relations: Causal relations refer to relations of treatments to potential outcomes within treated units, whereas statistical relations refer associations of treatments with actual outcomes across units (Rothman et al. , Chap. ). Consequently, the models have aided in distinguishing confounding from collapsibility (Greenland and Robins ; Greenland et al. ), synergy from statistical interaction (Greenland and Poole ), and causation probabilities from attributable fractions (Greenland et al. ; Greenland and Robins ). The conceptual clarification also stimulated development of statistical methods for observational studies. Rubin (, ) and his colleagues extended statistical machinery based on potential outcomes from
C
C
Causation and Causal Inference
the experimental setting to observational data analysis, leading, for example, to propensity-scoring and inverseprobability-of-treatment methods for confounder adjustment (Rosenbaum ; Hirano et al. ), as well as new insights into analysis of trials with noncompliance (Angrist et al. ) and separation of direct and indirect effects (Robins and Greenland , ; Frangakis and Rubin ; Kaufman et al. ). In many cases, such insights have led to methodologic refinements and better-informed choices among existing methods. In the longitudinal-data setting, potential-outcome modeling has led to entirely new methodologies for analysis of time-varying covariates and outcomes, including g-estimation and marginal structural modeling (Robins , ; Robins et al. , , ). A serious caution arises, however, when it is not clear that the counterfactual values for X (treatments other than the actual one) represent physical possibilities or even unambiguous states of nature. A classic example is gender (biological sex). Although people speak freely of gender (male vs. female) as cause of heart disease, given a particular man, it is not clear what it would mean for that man to have been a woman instead. Do we mean that the man cross-dressed and lived with a female identity his entire life? Or that he received a sex-change operation after birth? Or that the zygote from which he developed had its male chromosome replaced by a female chromosome? Potential-outcome models bring to light such ambiguities in everyday causal language but do not resolve them (Greenland a; Hernán ). Some authors appear to insist that use of the models be restricted to situations in which ambiguities are resolved, so that X must represent an intervention variable, i.e., a precise choice among treatment actions or decisions (Holland ). Many applications do not meet this restriction, however, and some go so far as to confuse outcomes (Y) with treatments (X), which can lead to nonsense results. Examples include estimates of mortality after “cause removal,” e.g., removal of all lung-cancer deaths. Sensible interpretation of any effect estimate requires asking what intervention on a unit could have given the unit a value of X (here, lung-cancer death) other than the one that was observed, and what the side effects that intervention would have. One cannot remove all lung-cancer deaths by smoking cessation. A treatment with a % cure rate might do so but need not guarantee the same subsequent lifespan as if the cancer never occurred. If such questions cannot be given at least a speculative answer, the estimates of the impact of cause removal cannot be expected to provide valid information for intervention and policy purposes (Greenland a).
More sweeping criticisms of potential-outcome models are given by Dawid (), for example, that the distribution of the full potential-outcome vector Y (i.e., the joint distribution of the Y(x ), . . . , Y(xJ )) cannot be nonparametrically identified by randomized experiments. Nonetheless, as the discussants point out, the practical implication of these criticisms are not clear, because the marginal distributions of the separate potential outcomes Y(xj ) are nonparametrically identifiable, and known mechanisms of action may lead to identification of their joint distribution as well.
Canonical Inference Before the extension of potential outcomes to observational inference, the only systematic approach to causal inference in epidemiology was the informal comparison of observations to characteristics expected of causal relations. Perhaps, the most widely cited of such approach is based on Hill’s considerations (Hill ), which are discussed critically in numerous sources (e.g., Koepsell and Weiss ; Phillips and Goodman ; Rothman et al. , Chap. ) as well as by Hill himself. The canonical approach usually leaves terms like “cause” and “effect” as undefined concepts around which the self-evident canons are built, much like axioms are built around concepts like “set” and “is an element of ” in mathematics. Only proper temporal sequence (cause must precede effect) is a necessary condition for a cause–effect relation to hold. The remaining considerations are more akin to diagnostic symptoms or signs of causation – that is, they are properties an association is assumed more likely to exhibit if it is causal than if it is not (Hill ; Susser , ). Furthermore, some of these properties (like specificity and dose response) apply only under specific causal models (Weiss , ). Thus, the canonical approach makes causal inference more closely resemble clinical judgment than experimental science, although experimental evidence is listed among the considerations (Hill ; Rothman et al. , Chap. ; Susser ). Some of the considerations (such as temporal sequence, association, dose response or predicted gradient, and specificity) are empirical signs and thus subject to conventional statistical analysis. Others (such as plausibility) refer to prior belief, and thus (as with disease symptoms) require elicitation, the same process used to construct priors for Bayesian analysis. The canonical approach is widely accepted in health sciences, subject to many variations in detail. Nonetheless, it has been criticized for its incompleteness and informality, and the consequent poor fit it affords to the deductive
Causation and Causal Inference
or mathematical approaches familiar to classic science and statistics (Rothman et al. , Chap. ). Although there have been some interesting attempts to reinforce or reinterpret certain canons as empirical predictions of causal hypotheses (e.g., Susser ; Weed ; Weiss , ; Rosenbaum ), there is no generally accepted mapping of the entire canonical approach into a coherent statistical methodology; one simply uses standard statistical techniques to test whether empirical canons are violated. For example, if the causal hypothesis linking X to Y predicts a strictly increasing trend in Y with X, a test of this statistical prediction may serve as a statistical criterion for determining whether the hypothesis fails the dose-response canon. Such usage falls squarely in the falsificationist/frequentist tradition of the twentieth-century statistics, but leaves unanswered most of the policy questions that drive causal research; this gap led to the development of methodologic modeling or bias analysis.
Bias Analysis In the second half of the twentieth-century, a more rigorous approach to observational studies emerged in the wake of major policy controversies such as those concerning cigarette smoking and lung cancer (e.g., Cornfield et al. ). This approach begins with the idea that, conditional on some sufficient set of confounders Z, there is a population association or relation between X and Y that is the target of inference. In other words, the Z-stratified associations are presumed to accurately reflect the effect of X on Y in that population stratum, however “effect” may be defined. Estimates of this presumably causal association are then the effect estimates. Observational and analytic shortcomings bias or distort these estimates: Units may be selected for observation in a nonrandom fashion; stratifying on additional unmeasured covariates U may be essential for the X-Y association to approximate a causal effect; inappropriate covariates may be entered into the analysis; components of X or Y or Z may not be adequately measured; and so on. In methodologic modeling or bias analysis, one models these shortcomings. In effect, one attempts to model the design and execution of the study, including features (such as selection biases and measurement errors) beyond investigator control. The process is thus a natural extension to observational studies of the design-based paradigm in experimental and survey statistics. For further details, see BIAS MODELING or the overviews by Greenland (b, ).
C
Structural Equations and Causal Diagrams Paralleling the development of potential-outcome models, an entirely different approach causal analysis arose in observational research in economics and related fields. Like methodologic modeling, this structural-equations approach does not begin with a formal definition of cause and effect, but instead develops models to reflect assumed causal associations, from which empirical (and hence testable) associations may be derived. Like most of statistics before the s, structural-equations methods were largely limited to normal linear models to derive statistical inferences. Because these models bear no resemblance to typical epidemiologic data, this limitation may in part explain the near absence of structural equations from epidemiology, despite their ubiquity in social-science methodology. From their inception, however, causal system models have been accompanied by graphical representations or path diagrams that provided compact summaries of qualitative assumptions made by the structural model; see 7Causal Diagrams for a review.
Conclusion Different approaches to causal inference represent separate historical streams rather than distinct methodologies, and can be blended in various ways. The result of any modeling exercise is simply one more input to informal judgments about causal relations, which may be guided by canonical considerations. Insights and innovations in any approach can thus benefit the entire process of causal inference, especially when that process is seen as part of a larger context. Other traditions or approaches (some perhaps yet to be imagined) may contribute to the process. It thus seems safe to say that no one approach or blend is a complete solution to the problem of causal inference, and that the topic remains one rich with open problems and opportunities for innovation.
Acknowledgments Some of the above material is adapted from Greenland ().
About the Author For biography see the entry 7Confounding and Confounder Control.
Cross References 7Bias Analysis 7Causal Diagrams 7Complier-Average Causal Effect (CACE) Estimation
C
C
Causation and Causal Inference
7Confounding and Confounder Control 7Event History Analysis 7Forecasting Principles 7Interaction 7Misuse of Statistics 7Principles Underlying Econometric Estimators for Iden-
tifying Causal Effects 7Rubin Causal Model 7Simpson’s Paradox 7Spurious Correlation 7Structural Equation Models
References and Further Reading Angrist J, Imbens G, Rubin DB () Identification of causal effects using instrumental variables (with discussion). J Am Stat Assoc :– Berk R () Regression analysis: a constructive critique. Sage Publications, Thousand Oaks Copas JG () Randomization models for matched and unmatched × tables. Biometrika :– Cornfield J, Haenszel W, Hammond WC, Lilienfeld AM, Shimkin MB, Wynder EL () Smoking and lung cancer: recent evidence and a discussion of some questions. J Nat Cancer Inst :– Cox DR () The planning of experiments. Wiley, New York Dawid P () Causal inference without counterfactuals (with discussion). J Am Stat Assoc :– Fisher RA () The causes of human variability. Eugenics Rev :– Frangakis C, Rubin DB () Principal stratification in causal inference. Biometrics :– Greenland S () Interpretation and choice of effect measures in epidemiologic analysis. Am J Epidemiol :– Greenland S () On the logical justification of conditional tests for two-by-two contingency tables. Am Stat :– Greenland S () Absence of confounding does not correspond to collapsibility of the rate ratio or rate difference. Epidemiology :– Greenland S () Induction versus Popper: substance versus semantics. Int J Epidemiol :– Greenland S () The relation of the probability of causation to the relative risk and the doubling dose: a methodologic error that has become a social problem. Am J Publ Health : – Greenland S () An introduction to instrumental variables for epidemiologists. Int J Epidemiol :– (Erratum: Int J Epidemiol :) Greenland S () Quantifying biases in causal models: classical confounding versus collider-stratification bias. Epidemiology :– Greenland S () An overview of methods for causal inference from observational studies. In: Gelman A, Meng XL (eds) Applied Bayesian modeling and causal inference from an incomplete-data perspective. Wiley, New York, pp – Greenland S (a) Epidemiologic measures and policy formulation: lessons from potential outcomes (with discussion). Emerg Themes in Epidemiol (online journal), http://www.eteonline.com/content///
Greenland S (b) Multiple-bias modeling for observational studies (with discussion). J Roy Stat Soc, ser A :– Greenland S () Relaxation penalties and priors for plausible modeling of nonidentified bias sources. Stat Sci :– Greenland S, Brumback BA () An overview of relations among causal modelling methods. Int J Epidemiol :– Greenland S, Poole C () Invariants and noninvariants in the concept of interdependent effects. Scand J Work Environ Health :– Greenland S, Robins JM () Identifiability, exchangeability, and epidemiological confounding. Int J Epidemiol :– Greenland S, Robins JM () Epidemiology, justice, and the probability of causation. Jurimetrics :– Greenland S, Robins JM, Pearl J () Confounding and collapsibility in causal inference. Stat Sci :– Halloran ME, Struchiner CJ () Causal inference for infectious diseases. Epidemiology :– Hamilton MA () Choosing a parameter for × table or × × table analysis. Am J Epidemiol :– Hernán MA () Hypothetical interventions to define causal effects — afterthought or prerequisite? Am J Epidemiol : – Hill AB () The environment and disease: association or causation? Proc Roy Soc Med :– Hirano K, Imbens G, Ridder G () Efficient estimation of average treatment effects using the estimated propensity score. Econometrica :– Hume D () An enquiry concerning human understanding. reprint by Open Court Press, LaSalle Jewell NP () Statistics for epidemiology. Chapman & Hall/CRC Press, Boca Raton Kaufman JS, MacLehose R, Kaufman S () A further critique of the analytic strategy of adjusting for covariates to identify biologic mediation. Epidemiol Perspect Innov (online journal) : Lewis DK () Causation. J Philos :– MacMahon B, Pugh TF () Causes and entities of disease. In: Clark DW, MacMahon B (eds) Preventive medicine. Little Brown, Boston, pp – Maldonado G, Greenland S () Estimating causal effects (with discussion). Int J Epidemiol :– Miettinen OS () Standardization of risk ratios. Am J Epidemiol :– Mill JS () A system of logic, ratiocinative and inductive. reprint by Longman & Greens, London Morrison AS () Screening in chronic disease. Oxford, New York Pearl J () Causal diagrams for empirical research. Biometrika :– Pearl J () Causality, nd edn. Cambridge University Press, New York Phillips CV, Goodman K () The missed lessons of Sir Austin Bradford Hill. Epidemiologic Perspectives Innovations (online journal), http://www.epi- perspectives.com/content/// Robins JM () A graphical approach to the identification and estimation of causal parameters in mortality studies with sustained exposure periods. J Chronic Dis (suppl ):s–s Robins JM () Confidence intervals for causal parameters. Stat Med :– Robins JM () The control of confounding by intermediate variables. Stat Med :– Robins JM () Causal inference from complex longitudinal data. In: Berkane M (ed) Latent variable modeling and applications to
Censoring Methodology
causality. Lecture notes in statistics (). Springer, New York, pp – Robins JM () Structural nested failure time models. In: Armitage P, Colton T (eds) The encyclopedia of biostatistics. Wiley, New York, pp – Robins JM () Marginal structural models versus structural nested models as tools for causal inference. In: Halloran ME, Berry DA (eds) Statistical models in epidemiology. Springer, New York, pp – Robins JM () Data, design, and background knowledge in etiologic inference. Epidemiology :– Robins JM, Greenland S () The probability of causation under a stochastic model for individual risks. Biometrics : – Robins JM, Greenland S () Identifiability and exchangeability for direct and indirect effects. Epidemiology :– Robins JM, Greenland S () Adjusting for differential rates of PCP prophylaxis in high-versus low-dose AZT treatment arms in an AIDS randomized trial. J Am Stat Assoc : – Robins JM, Blevins D, Ritter G, Wulfson M () G-estimation of the effect of prophylaxis therapy for Pnemocystis carinii pneumonia on the survival of AIDS patients. Epidemiology :– (Errata: Epidemiology :) Robins JM, Greenland S, Hu FC () Estimation of the causal effect of a time-varying exposure on the marginal mean of a repeated binary outcome (with discussion). J Am Stat Assoc : – Robins JM, Hernán MA, Brumback BA () Marginal structural models and causal inference in epidemiology. Epidemiology :– Rosenbaum P () Observational studies, nd edn. Springer, New York Rothman KJ () Causes. Am J Epidemiol :– Rothman KJ, Greenland S, Lash TL () Modern epidemiology, rd edn. Lippincott, Philadelphia Rubin DB () Estimating causal effects of treatments in randomized and nonrandomized studies. J Educat Psychol : – Rubin DB () Bayesian inference for causal effects: the role of randomization. Ann Stat :– Rubin DB () Comment: Neyman () and causal inference in experiments and observational studies. Stat Sci : – Rubin DB () Practical implications of modes of statistical inference for causal effects and the critical role of the assignment mechanism. Biometrics :– Simon HA, Rescher N () Cause and counterfactual. Philos Sci :– Stalnaker RC () A theory of conditionals. In: Rescher N (ed) Studies in logical theory. Blackwell, Oxford Stone () The assumptions on which causal inference rest. J Roy Stat Soc, ser B :– Susser M () Falsification, verification and causal inference in epidemiology: reconsideration in light of Sir Karl Popper’s philosophy. In: Rothman KJ (ed) Causal inference. Epidemiology Resources, Boston, pp – Susser M () What is a cause and how do we know one? A grammar for pragmatic epidemiology. Am J Epidemiol : – Weed DL () On the logic of causal inference. Am J Epidemiol :–
C
Weiss NS () Inferring causal relationships: elaboration of the criterion of “dose-response.” Am J Epidemiol :– Weiss NS () Can “specificity” of an association be rehabilitated as a basis for supporting a causal hypothesis? Epidemiology :– Welch BL () On the z-test in randomized blocks and Latin squares. Biometrika :– Wilk MB () The randomization analysis of a generalized randomized block design. Biometrika :–
Censoring Methodology Ng Hon Keung Tony Associate Professor Southern Methodist University, Dallas, TX, USA
Basic Concepts on Censored Data In industrial and clinical experiments, there are many situations in which units (or subjects) are lost or removed from experimentation before the event of interest occurs. The experimenter may not always obtain complete information on the time to the event of interest for all experimental units or subjects. Data obtained from such experiments are called censored data. Censoring is one of the distinguishing features of lifetime data. Censoring can be either unintentional due to accidental breakage or an individual under study drops out or intentional in which the removal of units or subjects is pre-planned, or both. Censoring restricts our ability to observe the time-to-event and it is a source of difficulty in statistical analysis. Censoring can occur at either end (single censoring) or at both ends (double censoring). If the event of interest is only known to be occured before a certain time, it is called left censoring. The term “left censored” implies that the event of interest is to the left of the observed time point. The most common case of censoring is right censoring, in which the exact time to the event of interest is not observed and it is only known to be occured after a certain time. Different types of right censoring schemes are discussed in the subsequent section. For interval censoring, the event of interest is only known to be occurred in a given time interval. This type of data frequently comes from experiments where the items under test are not constantly monitored, for example, the patients in a clinical trial have periodic follow-up and events of interest occur in between two consecutive follow-ups. Note that left censoring is a special case of interval censoring where the starting time for the interval is zero.
C
C
Censoring Methodology
For life-testing experiments where the event of interest is the failure of the item on test, two common reasons for pre-planned censoring are saving the total time on test and reducing the cost associated with the experiment because failure implies unit’s destruction which can be costly. When budget and/or facility constraints are in place, suitable censoring scheme can be used to control the time spent and the cost of the experiment. Nevertheless, censored data usually will reduce the efficient of statistical inference compare to complete data. Therefore, it is desirable to develop censoring scheme which can balance between (i) total time spent for the experiment; (ii) number of units used in the experiment; and (iii) the efficient of statistical inference based on the results of the experiment.
censored sample. It is also called time-censoring since the experimental time is fixed. Note that the number of failures observed here is random and, in fact, has a Binomial(n, F(T; θ)) distribution. Figure shows a schematic representation of a Type-I censored life-test with m = . Inferential procedures based on Type-I censored samples have been discussed extensively in the literature; see, for example, Cohen () and Balakrishnan and Cohen (). Type-I censoring scheme has the advantage that the experimental time is controlled to be at most T while it has the disadvantage that the effective sample size can turn out to be a very small number (even equal to zero) so that usual statistical inference procedures will not be applicable or they will have low efficiency.
Different Types of Censoring Schemes Suppose n units are placed on a life-testing experiment. Further, suppose X , X , ⋯, Xn denote the lifetimes of these n units taken from a population with lifetime distribution function F(x; θ) and density function f (x; θ), where θ is an unknown parameter(s) of interest. Let X:n ≤ ⋯ ≤ Xn:n denote the corresponding ordered lifetimes observed from the life-test. Some commonly used censoring schemes are discussed in the following.
Type-I Censoring Suppose it is planned that the life-testing experiment will be terminated at a pre-fixed time T. Then, only the failures until time T will be observed. The data obtained from such a restrained life-test will be referred to as a Type-I
Type-II Censoring Suppose it is planned that the life-testing experiment will be terminated as soon as the mth (where m is pre-fixed) failure is observed. Then, only the first m failures out of n units under test will be observed. The data obtained from such a restrained life-test will be referred to as a TypeII censored sample. In contrast to Type-I censoring, the number of failures observed is fixed (viz., m) while the duration of the experiment is random (viz., Xm:n ). Figure shows a schematic representation of a Type-II censored life-test with m = . Inferential procedures based on TypeII censored samples have been discussed extensively in the
Withdrawn
Start
u
u
u
u
u
X :n
X :n
X :n
X :n
X :n
n − T (End)
Censoring Methodology. Fig. Schematic representation of a Type-I censored life-test
Withdrawn
Start
u
u
u
u
u
u
X :n
X :n
X :n
X :n
X :n
X :n
Censoring Methodology. Fig. Schematic representation of a Type-II censored life-test
n − u X :n (End)
Censoring Methodology
Withdrawn
t
R
(R ,R ,R ,R ) X ::n
Start
Withdrawn
t
Withdrawn
R
R t
(R ,R ,R ,R ) X ::n
(R ,R ,R ,R ) X ::n
C
Withdrawn
R t(R ,R ,R ,R ) X ::n (End)
Censoring Methodology. Fig. Schematic representation of a progressively Type-II censored life-test
literature; see, for example, Nelson (), Cohen (), and Balakrishnan and Cohen (). Type-II censoring scheme has the advantage that the number of observed failures is fixed to be m which ensure reasonable information is available for statistical inference. However, it has the disadvantage that the experimental time is random and it can be large.
Progressive Censoring Both the conventional Type-I and Type-II censoring schemes do not have the flexibility of allowing removal of units at points other than the terminal point of the experiment. This restricts our ability to observe extreme failures which may lead to inefficient statistical inference if we are interested in the behavior of the upper tail of the lifetime distribution. For this reason, a more general censoring scheme called progressive censoring has been introduced. The censored life-testing experiments described above can be extended to situations wherein censoring occurs in multiple stages. Data arising from such life-tests are referred to as progressively censored data. Naturally, progressive censoring can be introduced in both Type-I and Type-II forms. For example, a progressive Type-II censored lifetesting experiment will be carried out in the following manner. Prior to the experiment, a number m < n is determined and the censoring scheme (R , R , . . . , Rm ) with m
example, Balakrishnan and Aggarwala () and Balakrishnan () for excellent reviews on the literatures on this topic.
Hybrid Censoring As mentioned previously, both Type-I and Type-II censoring schemes have some shortcomings. To keep away from these shortcomings, hybrid censoring schemes combining Type-I and Type-II censoring schemes have been proposed. Specifically, if the experiment is terminated at T ∗ = min{Xm:n , T}, where m and T are pre-fixed prior to the experiment, then the censoring scheme is called Type-I hybrid censoring scheme; if the experiment is terminated at T ∗ = max{Xm:n , T}, then the censoring scheme is called Type-II hybrid censoring scheme. We can see that both Type-I and Type-II hybrid censoring schemes try to balance between the advantages and disadvantages of conventional Type-I and Type-II censoring schemes. Hybrid censoring schemes has been studied extensively in the literature, one may refer Epstein (), Draper and Guttman (), Gupta and Kundu (), and Childs et al. (, ), Kundu () for details. In recent years, the idea of hybrid censoring has been generalized to progressive censoring, for discussions on different types of hybrid progressive censoring schemes, see, for example, Kundu and Joarder (), Banerjee and Kundu (), Ng et al. () and Lin et al. ().
Rj > and ∑ Rj + m = n is specified. During the experj=
iment, j-th failure is observed and immediately after the failure, Rj functioning items are removed from the test. We denote the m completely observed (ordered) lifetimes by (R ,R ,...,R m ) Xj:m:n , j = , , . . . , m, which are the observed progressively Type-II right censored sample. Figure shows a schematic representation of a progressively Type-II censored life-test with m = . Notice that the conventional Type-II censoring scheme is a special case of a progressive Type-II censoring scheme when Ri = , for i = , . . . , m− and Rm = n−m. Similarly, progressive TypeI censoring scheme can be introduced in a similar manner. Inferential procedures based on progressively Type-II censored samples have been discussed in the literature; see, for
About the Author H. K. T. Ng is an Associate Professor in the Department of Statistical Science at Southern Methodist University. He received his Ph.D. degree in Mathematics () from McMaster University, Hamilton, Canada. He is an elected member of International Statistical Institute (ISI) and an elected senior member of Institute of Electrical and Electronics Engineers (IEEE). He is currently an Associate Editor of Communications in Statistics.
Cross References 7Astrostatistics 7Event History Analysis
C
C
Census
7Nonparametric Estimation Based on Incomplete Obser-
vations 7Ordered Statistical Data: Recent Developments 7Step-Stress Accelerated Life Tests 7Survival Data
References and Further Reading Balakrishnan N () Progressive censoring methodology: an appraisal (with discussions). Test :– Balakrishnan N, Aggarwala R () Progressive censoring: theory, methods and applications. Birkhäuser, Boston Balakrishnan N, Cohen AC () Order statistics and inference: estimation methods. Academic, San Diego Banerjee A, Kundu D () Inference based on Type-II hybrid censored data from a Weibull distribution. IEEE Trans Reliab :– Childs A, Chandrasekar B, Balakrishnan N, Kundu D () Exact likelihood inference based on Type-I and Type-II hybrid censored samples from the exponential distribution. Ann Inst Stat Math :– Childs A, Chandrasekar B, Balakrishnan N () Exact likelihood inference for an exponential parameter under progressive hybrid censoring schemes. In: Vonta F, Nikulin M, Limnios N, Huber-Carol C (eds) Statistical models and methods for biomedical and technical systems. Birkhäuser, Boston, pp – Cohen AC () Truncated and censored samples: theory and applications. Marcel Dekker, New York Draper N, Guttman I () Bayesian analysis of hybrid life tests with exponential failure times. Ann Inst Stat Math :– Epstein B () Truncated life tests in the exponential case. Ann Math Stat :– Gupta RD, Kundu D () Hybrid censoring schemes with exponential failure distribution. Commun Stat Theory Meth :– Kundu D () On hybrid censoring Weibull distribution. J Stat Plan Infer :– Kundu D, Joarder A () Analysis of Type-II progressively hybrid censored data. Comput Stat Data Anal :– Nelson W () Applied life data analysis. Wiley, New York Ng HKT, Kundu D, Chan PS (). Statistical analysis of exponential lifetimes under an adaptive Type-II progressive censoring scheme, Naval Research Logistics :–
Census Margo J. Anderson Professor of History and Urban Studies University of Wisconsin–Milwaukee, Milwaukee, WI, USA
Introduction A census usually refers to a complete count by a national government of the population, with the population further
defined by demographic, social or economic characteristics, for example, age, sex, ethnic background, marital status, and income. National governments also conduct other types of censuses, particularly of economic activity. An economic census collects information on the number and characteristics of farms, factories, mines, or businesses. Most countries of the world conduct population censuses at regular intervals. By comparing the results of successive censuses, analysts can see whether the population is growing, stable, or declining, both in the country as a whole and in particular geographic regions. They can also identify general trends in the characteristics of the population. Because censuses aim to count the entire population of a country, they are very expensive and elaborate administrative operations and thus are conducted relatively infrequently. The United States and the United Kingdom, for example, conduct a population census every years (a decennial census), and Canada conducts one every years (a quinquennial census). Economic censuses are generally conducted on a different schedule from the population census. Censuses of population usually try to count everyone in the country as of a fixed date, often known as Census Day. Generally, governments collect the information by sending a 7questionnaire in the mail or a census taker to every household or residential address in the country. The recipients are instructed to complete the questionnaire and send it back to the government, which processes the answers. Trained interviewers visit households that do not respond to the questionnaire and individuals without mail service, such as the homeless or those living in remote areas.
History Censuses have been taken since ancient times by emperors and kings trying to assess the size and strength of their realms. These early censuses were conducted sporadically, generally to levy taxes or for military conscription. Clay tablet fragments from ancient Babylon indicate that a census was taken there as early as BCE to estimate forthcoming tax revenues. The ancient Chinese, Hebrews, Egyptians, and Greeks also conducted censuses. However, enumerations did not take place at regular intervals until the Romans began to count of the population in the Republic and later the empire. Among the Romans the census was usually a count of the male population and assessment of property value. It was used mainly for drafting men into military service and for taxing property. After the fall of the Roman Empire in the fifth century CE, census taking disappeared for several hundred years in the West. The small feudal communities of the Middle
Census
Ages had neither the mechanisms nor the need for censuses. However, in William the Conqueror ordered the compilation of the census-like Domesday Book, a record of English landowners and their holdings. From the data given in this survey, which was made to determine revenues due to the king, historians have reconstructed the social and economic conditions of the times. The modern census dates from the seventeenth century, when European powers wanted to determine the success of their overseas colonies. Thus the British crown and the British Board of Trade ordered repeated counts of the colonial American population in the seventeenth and eighteenth centuries, starting in the s in Virginia. The first true census in modern times was taken in New France, France’s North American empire, beginning in . The rise of democratic governments resulted in a new feature of the census process: The census of the United States was the first to have its Constitution require a census and periodic reapportionment of its House of Representatives on the basis of the decennial census results. Sweden began to conduct censuses in the mid-eighteenth century, and England and Wales instituted a regular decennial census in . During the nineteenth century and the first half of the twentieth century, the practice of census taking spread throughout the world. India conducted its first national census in , under British rule. China’s first modern census, in , counted million people. The United Nations encourages all countries to conduct a population count through a census or population registration system. It also promotes adoption of uniform standards and census procedures. The United Nations Statistical Office compiles reports on worldwide population.
Uses of Census Information Governments use census information in almost all aspects of public policy. In some countries, the population census is used to determine the number of representatives each area within the country is legally entitled to elect to the national legislature. The Constitution of the United States, for example, provides that seats in the House of Representatives should be apportioned to the states according to the number of their inhabitants. Each decade, Congress uses the population count to determine how many seats each state should have in the House and in the electoral college, the body that nominally elects the president and vice president of the United States. This process is known as reapportionment. States frequently use population census figures as a basis for allocating delegates to the state legislatures and for redrawing district boundaries for seats in the House, in state legislatures, and in local legislative districts. In Canada, census population data are similarly used
C
to apportion seats among the provinces and territories in the House of Commons and to draw electoral districts. Governments at all levels – such as cities, counties, provinces, and states – find population census information of great value in planning public services because the census tells how many people of each age live in different areas. These governments use census data to determine how many children an educational system must serve, to allocate funds for public buildings such as schools and libraries, and to plan public transportation systems. They can also determine the best locations for new roads, bridges, police departments, fire departments, and services for the elderly. Besides governments, many others use census data. Private businesses analyze population and economic census data to determine where to locate new factories, shopping malls, or banks; to decide where to advertise particular products; or to compare their own production or sales against the rest of their industry. Community organizations use census information to develop social service programs and child-care centers. Censuses make a huge variety of general statistical information about society available to researchers, journalists, educators, and the general public.
Conducting a Census Most nations create a permanent national statistical agency to take the census. In the United States, the Bureau of the Census (Census Bureau), an agency of the Department of Commerce, conducts the national population census and most economic censuses. In Canada, the Census Division of Statistics Canada is responsible for taking censuses. Conducting a census involves four major stages. First, the census agency plans for the census and determines what information it will collect. Next, it collects the information by mailing questionnaires and conducting personal interviews. Then the agency processes and analyzes the data. Finally, the agency publishes the results to make them available to the public and other government agencies.
Planning the Census Census agencies must begin planning for a census years in advance. One of the most important tasks is to determine what questions will appear on the census questionnaire. Census agencies usually undertake a lengthy public review process to determine the questions to be asked. They conduct public meetings, consider letters and requests from the general public, and consult with other government agencies and special advisory committees. In the United States, census questions must be approved by Congress and
C
C
Census
the Office of Management and Budget. In Canada, questions must be approved by the governor-general on the recommendations of the Cabinet. The questions included on census forms vary from nation to nation depending on the country’s particular political and social history and current conditions. Most censuses request basic demographic information, such as the person’s name, age, sex, educational background, occupation, and marital status. Many censuses also include questions about a person’s race, ethnic or national origin, and religion. Further questions may ask the person’s place of birth; relationship to the head of the household; citizenship status; the individual’s or the family’s income; the type of dwelling the household occupies; and the language spoken in the household. Questions that are routine in one nation may be seen as quite controversial in another, depending on the history of the country. The United States census does not ask about religious affiliation because such a question is considered a violation of the First Amendment right to freedom of religion or an invasion of privacy. Other nations, such as India, do collect such information. Questions on the number of children born to a woman were quite controversial in China in recent years because of government efforts to limit families to having only one child. In the United States, asking a question on income was considered controversial in when it was first asked. It is no longer considered as objectionable. Questions change in response to public debate about the state of society. For example, Americans wanted to know which households had radios in , and the census introduced questions on housing quality in . Canadians have recently begun to ask census questions on disability status and on the unpaid work done in the home. Besides determining the content of the census, census agencies must make many other preparations. Staffing is a major concern for census agencies because censuses in most countries require a huge number of temporary workers to collect and process data. Consequently, census agencies must begin recruiting and training workers months or years in advance. For example, the U.S. Census Bureau had to fill , temporary, short-term positions to conduct the census. In order to hire and retain enough staff, it had to recruit nearly three million job applicants. The majority of temporary workers are hired to go door-to-door to interview households that do not respond to the census questionnaire. In some countries, government employees at a local level, such as schoolteachers, are asked to help conduct the count. Prior to any census, a census agency must develop an accurate list of addresses and maps to ensure that everyone is counted. The U.S. Census Bureau obtains addresses
primarily from the United States Postal Service and from previous census address lists. It also works closely with state, local, and tribal governments to compile accurate lists. Finally, census agencies often conduct an extensive marketing campaign before Census Day to remind the general population about the importance of responding to the census. This campaign may involve paid advertising, distributing materials by direct mail, promotional events, and encouraging media coverage of the census.
Collecting the Information Until relatively recently, population censuses were taken exclusively through personal interviews. The government sent enumerators (interviewers) to each household in the country. The enumerators asked the head of the household questions about each member of the household and entered the person’s responses on the census questionnaire. The enumerator then returned the responses to the government. Today, many censuses are conducted primarily through self-enumeration, which means that people complete their own census questionnaire. Selfenumeration reduces the cost of a census to the government because fewer enumerators are needed to conduct interviews. In addition, the procedure provides greater privacy to the public and generally improves the accuracy of responses, because household members can take more time to think over the questions and consult their personal records. Nevertheless, census operations still require hiring very large numbers of temporary enumerators to conduct address canvassing in advance of a mail census and to retrieve forms from non responding households and check on vacant units. Other nations continue to conduct censuses partially or totally through direct enumeration. Some, such as Turkey, require people to stay home on Census Day to await the census taker. Census agencies make a special effort to count people who may not receive a questionnaire by mail or who have no permanent address. For example, the U.S. Census Bureau sends census takers to interview people at homeless shelters, soup kitchens, mobile food vans, campgrounds, fairs, and carnivals. It consults with experts to find migrant and seasonal farmworkers. Finally, the agency distributes census questionnaires to people living in group quarters, such as college dormitories, nursing homes, hospitals, prisons and jails, halfway houses, youth hostels, convents and monasteries, and women’s shelters. The level of detail on the complete count census varies by country, particularly after the development of probability survey techniques in the s. In the United States, for example, until the census, most households received a “short form,” a brief set of questions on basic
Census
characteristics such as name, age, sex, racial or ethnic background, marital status, and relationship to the household head. But from the mid-twentieth century until , a smaller sample of households received the “long form,” with many additional detailed questions. These included questions about the individual’s educational background, income, occupation, language knowledge, veteran status, and disability status as well as housing-related questions about the value of the individual’s home, the number of rooms and bedrooms in it, and the year the structure was built. These “long form” questions have been collected in the American Community Survey since the early s, and thus are no longer asked on the U.S. Census in .
Processing and Analysis of Data For most of the th century in the United States and Canada, census data were tabulated and compiled by hand, without the aid of machines. Manual processing was very slow, and some figures were obsolete by the time they were published. The invention of mechanical tabulating devices in the late nineteenth century made processing of the data much faster and improved the accuracy of the results. For example, in , the U.S. Census Bureau will scan the data from + million paper questionnaires, and capture the responses using optical character recognition software. Once in electronic form, the data can be analyzed and turned into statistics. Unreadable or ambiguous responses are checked by census clerks and manually keyed into the computer.
Publication of Results U.S. and Canadian censuses publish only general statistical information and keep individual responses confidential. By law, the U.S. Census Bureau and Statistics Canada are prohibited from releasing individual responses to any other government agency or to any individual or business. Census workers in both countries must swear under oath that they will keep individual responses confidential. Employees who violate this policy face a monetary fine and possible prison term. If an individual’s personal data were not kept confidential, people might refuse to participate in the census for fear that their personal information would be made public or used by the government to track their activities. In the United States, individual census responses are stored at the National Archives. After years, the original forms are declassified and opened to the public. These original responses are frequently used by people researching the history of their families or constructing genealogies. In Canada, census responses from and later are stored at Statistics Canada. Microfilmed records
C
of census responses from and earlier are stored at the National Archives of Canada; these are the only individual census responses currently available for public use. Until the s, census agencies published their results in large volumes of numeric tables – sometimes numbering in the hundreds of volumes. Today, the majority of census data is distributed electronically, both in tabulated form, and through anonymized public use microdata samples.
Problems in Census Taking and Issues for the Future Censuses provide important information about the population of a country. But they can become embroiled in political or social controversy simply by reporting information. Complaints about the census generally involve concerns about the accuracy of the count, the propriety of particular questions, and the uses to which the data are put. All censuses contain errors of various kinds. Some people and addresses are missed. People may misunderstand a question or fail to answer all the questions. Census officials have developed elaborate procedures to catch and correct errors as the data are collected, but some errors remain. For example, the U.S. census missed . million people and mistakenly counted . million people, according to Census Bureau estimates. The latter figure included people counted more than once, fictitious people listed on forms, and fabrications by enumerators. Such errors undermine the credibility of the census as a mechanism for allocating seats in legislative bodies and government funds. In recent years, developments in statistical analysis have made it possible to measure the accuracy of censuses. Census results may be compared with population information from other sources, such as the records of births, deaths, and marriages in vital statistics. Census officials can also determine the level of accuracy of the count by conducting a second, sample count called a post-enumeration survey or post-censal survey. In this technique, census staff knock on the door of each housing unit in selected blocks around the country, regardless of whether the housing unit was on the master address list. The staff member determines whether the household was counted in the census. By comparing the results from this survey with the census records, census officials can estimate how many people from each geographic region were missed in the original census count. Some nations, such as Canada and Australia, have begun to adjust the census results for omissions and other errors. Concerns about the confidentiality of the census represent another source of data error. Censuses require public understanding, support, and cooperation to be successful. Concerns about government interference with private life
C
C
Central Limit Theorems
can prevent people from cooperating with what is essentially a voluntary counting process. People may be suspicious of giving information to a government agency or may object that particular census questions invade their privacy. When public trust is lacking, people may not participate. In some nations, past census controversies have led to the elimination of the national census. During World War II (–), for example, the German Nazi forces occupying The Netherlands used the country’s census records and population registration data to identify Jews for detention, removal, and extermination. This use ultimately undermined the legitimacy of the census after World War II. In The Netherlands, the legacy of the Nazi era was one of the major justifications to end census taking. The Netherlands took its last regular census in and now collects population information through other mechanisms. Many nations are currently exploring alternatives to or major modernizations of the traditional population census. France, for example, has recently implemented a continuous measurement population counting system. The United States is exploring the use of administrative records and electronic methods of data collection to replace the mail enumeration in .
About the Author Dr. Margo J. Anderson is Professor of History and Urban Studies, University of Wisconsin–Milwaukee. She specializes in the social history of the United States in the nineteenth and twentieth centuries. She was Chair, History Department (–). She was a member of the National Academy of Sciences’ Panel on Census Requirements for the Year and Beyond. Dr. Anderson was Vice President (), and President, Social Science History Association (). She is a Fellow, American Statistical Association (ASA) (), and was Chair, Social Statistics Section of ASA (). Currently, she is ASA Chair, Committee on Committees. She has authored and coauthored numerous papers and several books including, The American Census: A Social History (New Haven: Yale University Press, ), and Who Counts? The Politics of Census-Taking in Contemporary America (with Stephen E. Fienberg, Russell Sage, , revised and updated ), named as one of Choice Magazine’s Outstanding Academic Books of . She was Editor-in-Chief of the Encyclopedia of the U.S. Census (Washington, D.C.: CQ Press, ). Professor Anderson is widely regarded as the leading scholar on the history of the U.S. census, and has made
distinguished contributions to research in American social science.
Cross References 7African Population Censuses 7Demography 7Economic Statistics 7Federal Statistics in the United States, Some Challenges 7Population Projections 7Sample Survey Methods 7Simple Random Sample 7Small Area Estimation 7Statistical Publications, History of
References and Further Reading Anderson M () The American census: a social history. Yale University Press, New Haven Anderson M, Fienberg SE () Who counts? The politics of census taking in contemporary America, rev edn. Russell Sage Foundation, New York Desrosieres A. La Politique des Grands Nombres: Histoire de la Raison Statistique. Edition La Découverte, Paris (, The politics of large numbers: a history of statistical reasoning. Harvard University Press, Cambridge) Minnesota Population Center () Integrated public use microdata series. http://ipums.org. U.K. Office of National Statistics () Years of the Census. http://www.statistics.gov.uk/census/bicentenary/pdfs/ years.pdf U.N. Statistics Division () World Population and Housing Census Programme. http://unstats.un.org/unsd/demographic/ sources/census/_PHC/more.htm Ventresca M () When states count: institutional and political dynamics in modern census establishment, –. PhD dissertation, Stanford University, Stanford Worton DA () The Dominion bureau of statistics: a history of Canada’s central statistics office and its antecedents: –. McGill-Queens University Press, Kingston
Central Limit Theorems Julio M. Singer Professor, Head Universidade de São Paulo, São Paulo, Brazil
Introduction One of the objectives of statistical inference is to draw conclusions about some parameter, like the mean or the variance of a (possibly conceptual) population of interest based
Central Limit Theorems
on the information obtained in a sample conveniently selected therefrom. For practical purposes, estimates of these parameters must be coupled with statistical properties and except in the most simple cases, exact properties are difficult to obtain and one must rely on approximations. It is quite natural to expect estimators to be consistent, but it is even more important that their (usually mathematically complex) exact sampling distribution be adequately approximated by a simpler one, such as the normal or the χ distribution, for which tables or computational algorithms are available. Here we are not concerned with the convergence of the actual sequence of statistics {Tn } to some constant or random variable T as n → ∞, but with the convergence of the corresponding distribution functions {Gn } to some specific distribution function F. This is known as weak convergence and for simplicity, we write D
Tn Ð→ F. Although this is the weakest mode of stochastic convergence, it is very important for statistical applications, since the related limiting distribution function F may generally be employed in the construction of approximate confidence intervals for and significance tests about the parameters of interest. In this context, central limit theorems (CLT) are used to show that statistics expressed as sums of the underlying random variables, conveniently standardized, are asymptotically normally distributed, i.e., converge weakly to the normal distribution. They may be proved under different assumptions regarding the original distributions. The simplest CLT states that the (sampling) distribution of the sample mean of independent and identically distributed (i.i.d) random variables with finite second moments may be approximated by a normal distribution. Although the limiting distribution is continuous, the underlying distribution may even be discrete. CLT are also available for independent, but not identically distributed (e.g., with different means and variances) underlying random variables, provided some (relatively mild) assumptions hold for their moments. The Liapounov CLT and the Lindeberg-Feller CLT are useful examples. Further extensions cover cases of dependent random underlying ˇ variables; in particular, the Hájek-Sidak CLT is extremely useful in regression analysis, where as the sample size increases, the response variables form a triangular array in which for each row (i.e., for given n), they are independent but this is not true among rows (i.e., for different values of n). Extensions to cover cases where the underlying random variables have more complex (e.g., martingale-type) dependence structures are also available. When dealing with partial sum or empirical distributional processes, we
C
must go beyond the finite-dimensional case and assume some compactness conditions to obtain suitable results, wherein the so-called weak invariance principles play an important role.
C Different Versions of the Central Limit Theorem We now present (without proofs) the most commonly used versions of the CLT. Details and a list of related references may be obtained in Sen et al. (). Theorem (Classical CLT) Let {Xk , k ≥ } be a sequence of i.i.d. random variables such that . E(Xk ) = µ. . Var(Xk ) = σ < ∞. √ Also, let Zn = (Tn −nµ)/(σ n) where Tn = ∑nk= Xk . Then, D
Zn Ð→ N (, ). In practice, this result implies that for large n, the distribution of the sample mean X n = Tn /n may be approximated by a normal distribution with mean µ and variance σ /n. An interesting special case occurs when the underlying variables Xk have Bernoulli distributions with probability of success π. Here the expected value and the variance of Xk are π and π( − π), respectively. It follows that the largesample distribution of the sample proportion, pn = Tn /n may be approximated by a N [π, π( − π)/n] distribution. This result is known as the De Moivre–Laplace CLT. An extension of Theorem to cover the case of sums of independent, but not identically distributed random variables requires additional assumptions on the moments of the underling distributions. In this direction, we consider the following result. Theorem (Liapounov CLT) Let {Xk , k ≥ } be a sequence of independent random variables such that . E(Xk ) = µ k . (k)
. ν +δ = E(∣Xk − µ k ∣+δ ) < ∞, k ≥ for some < δ ≤ . Also let Tn = ∑nk= Xk , Var(Xk ) = σk , τ n = ∑nk= σk , Zn = −(+δ) (k) n (Tn − ∑nk= µ k ) /τ n and ρ n = τ n ∑k= ν +δ . Then, if D
limn→∞ ρ n = , it follows that Zn Ð→ N (, ). This as well as other versions of the CLT may also be extended to the multivariate case by referring to the Cramér-Wold Theorem, which essentially states that the asymptotic distribution of the multivariate statistic under
C
Central Limit Theorems
investigation may be obtained by showing that every linear combination of its components follows an asymptotic normal distribution. Given a sequence {X n , n ≥ } of random vectors in Rp , with mean vectors µ n and covariance
matrices Σ n , n ≥ , to show that n−/ ∑ni= (X i − µ i ) Ð→ Np (, Σ) with Σ = limn→∞ n− ∑ni= Σ i , one generally proceeds according to the following strategy: D
. Use one of the univariate CLT to show that for every fixed λ ∈ Rp , n−/ ∑ni= λ′ (X i − µ i ) Ð→ N (, γ ) with γ = limn→∞ n− λ′ (∑ni= Σ i ) λ. . Use the Cramér-Wold Theorem to complete the proof. D
finite scalars a < b, it follows that a necessary and sufficient D
condition for Zn Ð→ N (, ) is that τ n → ∞ as n → ∞. Up to this point we have devoted attention to the weak convergence of sequences of statistics {Tn , n ≥ } constructed from independent underlying random variables X , X , . . .. We consider now some extensions of the CLT where such restriction may be relaxed. The first of such extensions holds for sequences of (possibly dependent) random variables which may be structured as a double array of the form ⎛ X , X , ⋯, Xk ⎜ ⎜ ⎜ ⎜ X , X , ⋯, Xk ⎜ ⎜ ⎜ ⎜ ⋮ ⋮ ⋱ ⋮ ⎜ ⎜ ⎜ ⎝ Xn , Xn , ⋯, Xnkn
As an example we have: Theorem (Multivariate version of the Liapounov CLT) Let {X n , n ≥ } be a sequence of random vectors in Rp with mean vectors µ n and finite covariance matrices Σ n , n ≥ , such that max≤i≤n max≤j≤p E(∣Xij − µ ij ∣+δ ) < ∞ for some < δ < , and Σ = limn→∞ n− ∑ni= Σ i exists.
Then n−/ ∑ni= (X i − µ i ) Ð→ Np (, Σ). D
In the original formulation, Liapounov used δ = , but even (k) the existence of ν +δ , < δ ≤ is not a necessary condition, as we may see from the following theorem. Theorem (Lindeberg-Feller CLT) Let {Xk , k ≥ } be a sequence of independent random variables satisfying
where the Xnk are row-wise independent. The case where kn = n, n ≥ , is usually termed a triangular array of random variables. This result is very useful in the field of 7order statistics. Theorem (Double array CLT) Let the random variables {Ynk , ≤ k ≤ kn , n ≥ } where kn → ∞ as n → ∞ be such that for each n, {Ynk , ≤ k ≤ kn } are independent. Then . {Ynk , ≤ k ≤ kn , n ≥ } is an infinitesimal system of random variables, i.e., satisfies the UAN condition.
. E(Xk ) = µ k . . Var(Xk ) = σk < ∞.
D
Also, let Tn = ∑nk= Xk , τ n = ∑nk= σk and Zn = ∑nk= Ynk where Ynk = (Xk − µ k )/τ n and consider the following additional conditions: . Uniform asymptotic negligibility (UAN): max≤k≤n (σk /τ n ) → as n → ∞. D
. Asymptotic normality: Zn Ð→ N (, ). . Lindeberg-Feller (uniform integrability): ∀ε > ,
⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠
n ∑ E[(Xk − µ k ) I(∣Xk − µ k ∣ > ετ n )] → as n → ∞, τ n k=
where I(A) denotes the indicator function. Then, (A) and (B) hold simultaneously if and only if (C) holds. Condition (A) implies that the random variables Ynk are infinitesimal, i.e., that max≤k≤n P(∣Ynk ∣ > ε) → as n → ∞ for every ε > , or, in other words, that the random variables Ynk , ≤ k ≤ n, are uniformly in k, asymptotically in n, negligible. When the underlying random variables under consideration are bounded, i.e., when P(a ≤ Xk ≤ b) = for some
n . Zn = ∑kk= Ynk Ð→ N (, ).
hold simultaneously, if and only if, for every ε > , as n → ∞ the following two conditions hold kn
. ∑ P(∣Ynk ∣ > ε) → . k= kn
. ∑ { ∫
k= {∣y∣≤ε}
y dP(Ynk ≤ y)
−[ ∫
{∣y∣≤ε}
ydP(Ynk ≤ x)] } → .
Linear regression and related models pose special problems since the underlying random variables are not identically distributed and in many cases, the exact functional form of their distributions is not completely specified. Least-squares methods (see 7Least Squares) are attractive under these conditions, since they may be employed in a rather general setup. In this context, the following CLT specifies sufficient conditions on the explanatory variables such that the distributions of the least squares estimators of the regression parameters may be approximated by normal distributions.
C
Central Limit Theorems
Theorem (Hájek-Šidak CLT) Let {Yn , n ≥ } be a sequence of i.i.d. random variables with mean µ and finite variance σ ; let {xn , n ≥ } be a sequence of real vectors xn = (xn , . . . , xnn )′ . Then if Noether’s condition holds, i.e., if n
max [xni / ∑ xni ]→
≤i≤n
as
n → ∞,
i=
Loynes (), Brown (), Dvoretzky (), or McLeish (). Theorem (Martingale CLT) Consider a sequence {Xk , k ≥ } of random variables satisfying . E(Xk ) = . . E (Xk ) = σk < ∞. . E{Xk ∣X , . . . , Xk− } = , X = . Also let Tn = ∑nk= Xk , τ n = ∑nk= σk , vk E (Xk ∣X , . . . , Xk− ) and wn = ∑nk= vk . If
holds, it follows that n
n
Zn = [∑ xni (Yni − µ)] / [σ ∑ xni ] i=
=
/ D
Ð→ N (, ).
i=
As an illustration, consider the simple linear regression model (see 7Simple Linear Regression) yni = α + βxni + eni , i = , . . . , n,
P
. wn /τ n Ð→ as n → ∞. . ∀ε > , τ n− ∑nk= E[Xk I(∣Xk ∣ > ετ n )] → as n → ∞ (Lindeberg-Feller condition), then the sequence {Xk , k ≥ } is infinitesimal and Zn = D
Tn /τ n Ð→ N (, ).
where yni and xni represent observations of the response and explanatory variables, respectively, α and β are the parameters of interest and the eni correspond to uncorrelated random errors with mean and variance σ . The least squares estimators of β and α are respectively β̂n = n n α n = yn − ∑i= (xni − xn )(yni − yn )/ ∑i= (xni − xn ) and ̂ ̂ β n xn where xn and yn correspond to the sample means of the explanatory and response variables. Irrespectively of the form of underlying distribution of eni , we may use standard results to show that ̂ α n and β̂n are unbiased / ∑ni= (xni − xn ) ] and have variances given by σ [∑ni= xni
Note that the terms vk are random variables since they depend on X , . . . , Xk− ; condition (A) essentially states that all the information about the variability in the Xk is contained in X , . . . , Xk− . Also note that {Tn , n ≥ } is a zero mean martingale (See also 7Martingale Central Limit Theorem.)
and σ [∑ni= (xni − xn ) ] , respectively. Furthermore, the covariance between ̂ α n and β̂n is −σ xn / ∑ni= (xni − xn ) . When the underlying distribution of eni is normal, we may show that (̂ α n , β̂n ) follows a bivariate normal distribution. If Noether’s condition holds and both xn and n− ∑ni= (xni − xn ) converge to finite constants as n → ∞, we may use the ˇ Hájek-Sidak CLT and the Cramér-Wold Theorem to conclude that the same bivariate normal distribution specified above serves as an approximation of the true distribution of (̂ α n , β̂n ), whatever the form of the distribution of eni , provided that n is sufficiently large. The results may also be generalized to cover alternative estimators obtained by means of generalized and weighted least-squares procedures as well as via robust M-estimation procedures. They may also be extended to generalized linear and nonlinear models. Details may be obtained in Sen et al. (), for example. It is still possible to relax further the independence assumption on the underlying random variables. The following theorems constitute examples of CLT for dependent random variables having a martingale (or reverse martingale) structure. For further details, the reader is referred to
i.e., {Tk , k ≥ } is a zero mean reverse martingale. Assume that E (Tn ) < ∞ and let Yk = Tk − Tk+ , k ≥ , vk = E (Yk ∣Tk+ , Tk+ , . . . ) and wn = ∑∞ k=n vk . If
−
Theorem (Reverse Martingale CLT) Consider a sequence {Tk , k ≥ } of random variables such that E(Tn ∣Tn+ , Tn+ , . . .) = Tn+
and
E(Tn ) = ,
a.s.
. wn /E (wn ) Ð→ . ∞
. w− n ∑ E[Yk I(∣Yk ∣ > εwn )∣Tk+ , Tk+ , . . .] Ð→ , k=n
P
∞
ε > or w− n ∑ Yk Ð→ , a.s.
k=n
√ D it follows that Tn / E (wn ) Ð→ N (, ).
Rates of Convergence to Normality In the general context discussed above, a question of both theoretical and practical interest concerns the speed with which the convergence to the limiting normal distribution takes place. Although there are no simple answers to this question, the following result may be useful. Theorem (Berry-Esséen) Let {Xn , n ≥ } be a sequence of i.i.d. random variables with E(X ) = µ, Var(X ) = σ and suppose that E(∣X − µ∣+δ ) = ν +δ < ∞ for
C
C
Central Limit Theorems
some < δ ≤ . Also let Tn = ∑ni= Xi and F(n) (x) = √ P [(Tn − nµ)/ (σ n) ≤ x], x ∈ R. Then there exist a constant C such that ∆ n = sup ∣F(n) (x) − Φ(x)∣ ≤ C x∈R
ν +δ n−δ/ σ +δ
where Φ denotes the standard normal distribution function. The reader is referred to Feller () for details. Berry () proved the result for δ = and Esséen () showed that C ≥ .. Although the exact value of the constant C is not known, many authors have proposed upper bounds. In particular, van Beeck () showed that C ≤ . and more recently, Shevtsova () concluded that C ≤ .. The usefulness of the theorem, however, is limited, since the rates of convergence attained are not very sharp. Alternatively, the rates of convergence of the sequence of distribution functions F(n) to Φ or of the density functions f(n) (when they exist) to φ (the density function of the standard normal distribution) may be assessed by GramCharlier or Edgeworth expansions as discussed in Cramér (), for example. Although this second approach might offer a better insight to the problem of evaluating the rate of convergence to normality than that provided by the former, it requires the knowledge of the moments of the parent distribution and, thus, is less useful in practical applications.
Convergence of Moments Given that weak convergence has been established, a question of interest is whether the moments (e.g., mean and variance) of the statistics under investigation converge to the moments of the limiting distribution. Although the answer is negative in general, an important theorem, due to Cramér, indicates conditions under which the result is true. The reader is referred to Sen et al. () for details.
Asymptotic Distributions of Statistics not Expressible as Sums of Random Variables The Slustky theorem is a handy tool to prove weak convergence of statistics that may be expressed as the sum, product or ratio of two terms, the first known to converge weakly to some distribution and the second known to converge in probability to some constant. As an example, consider independent and identically distributed random variables Y , . . . , Yn with mean µ and variance σ . Since the corresponding sample standard deviation S converges in probability to σ and the distribution of Y may be approximated by a N (µ, σ /n) distribution, we may apply
Slutsky’s theorem to show that the large-sample distribu√ √ tion of n Y/S = ( n Y/σ) × (σ/S) may be approximated by a N (µ, ) distribution. This allows us to construct approximate confidence intervals for and tests of hypotheses about µ using the standard normal distribution. A similar approach may be employed to the Bernoulli example by noting that pn is a consistent estimator of π. An important application of Slutky’s Theorem relates to statistics that can be decomposed as a sum of a term for which some CLT holds and a term that converges in probability to . Assume, for example, that the variables Yi have a finite fourth central moment γ and write the sample variance as n
S = [n/(n − )] {n− ∑[(Yi − µ) − σ /n] i= n
+ [σ − ∑(Y − µ) ]} . i=
Since the first term within the {} brackets is known to converge weakly to a normal distribution by the CLT and the second term converges in probability to , we conclude that the distribution of S may be approximated by a N (σ , γ/n) distribution. This is the basis of the projection results suggested by Hoeffding () and extensively explored by Jureˇckova and Sen () to obtain large-sample properties of 7U-statistics as well as of more general classes of estimators. Another convenient technique to obtain the asymptotic distributions of many (smooth) functions of asymptotically normal statistics is the Delta-method: if g is a locally differentiable function of a statistic Tn whose distribution may be approximated (for large samples) by a N (µ, τ ) distribution, then the distribution of the statistic g(Tn ) may be approximated by a N {g(µ), [g ′ (µ)] τ } distribution, where g ′ (µ) denotes the first derivative of g computed at µ. Suppose that we are interested in estimating the odds of a failed versus pass response, i.e., π/( − π) in an exam based on a sample of n students. A straightforward application of the De Moivre Laplace CLT may be used to show that the estimator of π, namely, k/n, where k is the number of students that failed the exam, follows an approximate N [π, π( − π)/n] distribution. Taking g(x) = x/( − x), we may use the Delta-method to show that the distribution of the sample odds k/(n − k) may be approximated by a N {π/( − π), π/[n( − π) ]} distribution. This type of result has further applications in variance-stabilizing transformations used in cases (as the above example) where the variance of the original statistic depends on the parameter it is set to estimate.
Central Limit Theorems
For some important cases, like the Pearson χ -statistic or more general quadratic forms Q = Q(µ) = (Y − µ)t A(Y − µ) where Y is a p-dimensional random vector with mean vector µ and covariance matrix V and A is a pdimensional square matrix of full rank, the (multivariate) Delta-method may not be employed because the derivative of Q computed at µ is null. If A converges to an inverse of V, a useful result known as the Cochran theorem, states that the distribution of Q may be approximated by a χ instead of a normal distribution. In fact, the theorem holds even if A is not of full rank, but converges to a generalized inverse of V. This is important for applications in categorical data. The CLT also does not hold for extreme order statistics like the sample minimum or maximum; depending on some regularity conditions on the underlying random variables, the distribution of such statistics, conveniently normalized, may be approximated by one of three types of distributions, namely the extreme value distributions of the first, second or third type, which, in this context, are the only possible limiting distributions as shown by Gnedenko ().
Central Limit Theorems for Stochastic Processes Empirical distribution functions and 7order statistics have important applications in nonparametric regression models, resampling methods like the 7jackknife and bootstrap (see 7Bootstrap Methods), sequential testing as well as in Survival and Reliability analysis. In particular it serves as the basis for the well known goodness-of-fit KolmogorovSmirnov and Cramér-von Mises statistics and for L- and R-estimators like trimmed or Winsorized means. Given the sample observations Y , . . . , Yn assumed to follow some distribution function F and a real number y, the empirical distribution function is defined as n
Fn (y) = n− ∑ I(Yi ≤ y) i=
where I(Yi ≤ y) is an indicator function assuming the value if Yi ≤ y and , otherwise. It is intimately related to the order statistics, Yn: ≤ Yn: ≤ . . . ≤ Yn:n where Yn: is the smallest among Y , . . . , Yn , Yn: is the second smallest and so on. For each fixed sample, Fn is a distribution function when considered as a function of y. For every fixed y, when considered as a function of Y , . . . , Yn , Fn (y) is a random variable; in this context, since the I(Yi ≤ y), i = , . . . , n, are independent and identically distributed zero-one valued random variables, we may apply the classical CLT to conclude that for each fixed y the distribution of Fn (y) may
C
be approximated by a N {F(y), F(y)[−F(y)]/n} distribution provided that n is sufficiently large. In fact, using standard asymptotic results, we may show that given any finite number m of points y , . . . , ym , the distribution function of the vector [Fn (y ), . . . , Fn (ym )] may be approximated by a multivariate normal distribution function. This property is known as convergence of finite-dimensional distributions. On the other hand, Fn − F = {Fn (y) − F(y)∶ y ∈ R} is a random function defined on the set of real numbers, and, hence, to study its various properties we may need more than the results considered so far. Note that as the sample size n increases, so does the cardinality of the set of order statistics used to define the empirical distribution function and we may not be able to approximate this n-dimensional joint distribution by an m-dimensional one unless some further tightness or compactness conditions are imposed on the underlying distributions. This is the basis of the weak invariance principles necessary to show the convergence of empirical and other 7stochastic processes to Brownian bridge or Brownian motion processes. An outline of the rationale underlying these results follows. √ n[Gn (t) − t], t ∈ Let t = F(y) and Wn (t) = (, ) where Gn (t) = Fn [F − (t)] = Fn (y) with F − (x) = inf{y : F(y) > x}, so that {Wn (t), t ∈ (, )} is a stochastic process with E [Wn (t)] = and E [Wn (s)Wn (t)] = min(s, t) − st, ≤ s, t ≤ . Using the multivariate version of the CLT we may show that as n → ∞, for all m ≥ , given ≤ t ≤ . . . ≤ tm ≤ , the vector W nm = D
[Wn (t ), . . . , Wn (tm )] Ð→ [W (t ), . . . , W (tm )] = W m where W m follows a Nm (, Γ m ) distribution with Γ m denoting a positive definite matrix with elements min(ti , tj ) − ti tj , i, j = , . . . , m. Now, define a stochastic process {Z(t), t ∈ (, )} with independent and homogeneous increments such that, for every ≤ s < t ≤ , the difference Z(t) − Z(s) follows a N (, t − s) distribution. Then, it follows that E[Z(s)Z(t)] = min(s, t). This process is known as a standard Brownian motion or standard Wiener process. Furthermore, letting W (t) = Z(t)−tZ(), ≤ t ≤ , it follows that {W (t), t ∈ (, )} is also a Gaussian stochastic process such that E[W (t)] = and E[W (s)W (t)] = min(s, t) − st, ≤ s, t ≤ . Then for all m ≥ , given ≤ t ≤ . . . ≤ tm ≤ , the vector W m = [W (t ), . . . , W (tm )] also follows a Nm (, Γ m ) distribution. Since W () = W () = with probability , this process is called a tied down Wiener process or Brownian bridge. Using the Kolmogorov maximal inequality, we may show that {Wn (t), t ∈ (, )} is tight and referring to standard results in weak convergence of probability meaD
sures, we may conclude that {Wn (t), t ∈ (, )} Ð→
C
C
Chaotic Modelling
{W (t), t ∈ (, )}. Details and extensions to statistical functionals may be obtained in Jureˇckova and Sen () among others.
About the Author Dr. Julio da Motta Singer is Professor of Biostatistics and Head, Centre for Applied Statistics, University of São Paulo, Brazil. He has obtained his Ph.D. at the University of North Carolina in (advisor P.K. Sen). He has coauthored over refereed papers and several books including, Large Sample Methods in Statistics: An Introduction with Applications (with P.K. Sen, Chapman and Hall, ), and From finite sample to asymptotic methods in Statistics (with P.K. Sen and A.C. Pedroso-de-Lima, Cambridge University Press, ).
Cross References 7Almost Sure Convergence of Random Variables 7Approximations to Distributions 7Asymptotic Normality 7Asymptotic Relative Efficiency in Estimation 7Asymptotic Relative Efficiency in Testing 7Empirical Processes 7Limit Theorems of Probability Theory 7Martingale Central Limit Theorem 7Normal Distribution, Univariate 7Probability Theory: An Outline
References and Further Reading Billingsley P () Convergence of probability measures. Wiley, New York Berry AC () The accuracy of the Gaussian approximation to the sum of independent variates. Trans Am Math Soc :– Brown BM () Martingale central limit theorems. Ann Math Stat :– Chow YS, Teicher H () Probability theory: independence, interchangeability, martingales Springer, New York Cramér H () Mathematical methods of statistics. Princeton University Press, Princeton Dvoretzky A () Asymptotic normality for sums of dependent random variables. In: Proceedings of the sixth berkeley symposium on mathematical statistics and probability. University of California Press, Berkeley, vol , pp – Esséen CG () A moment inequality with an application to the central limit theorem. Skandinavisk Aktuarietidskrift : – Feller W () An introduction to probability theory and its applications, vol , nd edn. Wiley, New York Ferguson TS () A course in large sample theory. Chapman & Hall, London Gnedenko BV () Sur la distribution limite du terme maximum d’une série aléatoire. Ann Math :– Hoeffding W () A class of statistics with asymptotically normal distributions. Ann Math Stat :–
Jureˇcková J, Sen PK () Robust statistical procedures. Wiley, New York Lehmann EL () Elements of large sample theory. Springer, New York Loynes RM () An invariance principle for reversed martingales. Proc Am Math Soc :– McLeish DL () Dependent central limit theorems and invariance principles. Ann Probab :– Reiss RD () Approximate distributions of order statistics with applications to nonparametric statistics. Springer, New York Sen PK () Sequential nonparametrics: invariance principles and statistical inference. Wiley, New York Sen PK, Singer JM, Pedroso-de-Lima AC () From finite sample to asymptotic methods in statistics. Cambridge University Press, Cambridge Serfling RJ () Approximation theorems of mathematical statistics. Wiley, New York Shevtsova IG () Sharpening the upper bound of the absolute constant in the Berry–Esséen inequality. Theory Probab Appl :– van Beeck P () An application of Fourier methods to the problem of sharpening the Berry–Esséen inequality. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete : – van der Vaart AW () Asymptotic statistics. Cambridge University Press, New York
Chaotic Modelling Christos H. Skiadas Professor, Director of the Data Analysis and Forecasting Laboratory Technical University of Crete, Chania, Greece Chaotic modeling is a term used to express the representation of the state of a system or a process by using chaotic models or tools developed in the chaotic literature and the related scientific context. In the following we present the main elements of the chaotic modeling including chaotic terms, differential and difference equations and main theorems (Skiadas ). Chaos is a relatively new science mainly developed during last decades with the use of computers and supercomputers. It touches almost all the scientific fields. However, the basic elements can be found at the end of the nineteenth century and the attempts to solve the famous three-body problem by Henri Poincaré (). Although he succeeded to solve only the special case when the three bodies move in the same plane, he could explore the main characteristics of the general three-body problem and to see the unpredictability of the resulting paths in space. In other words he could realize the main characteristic of
Chaotic Modelling
C
differential equations with only one nonlinear term may express chaotic behavior. The Rössler system is the following (e, f and m are parameters): x˙ = −y − z, y˙ = x − ez, z˙ = f + xz − mz.
Chaotic Modelling. Fig. The Lorenz attractor (xyz view)
It can be verified that the number of chaotic parameters is equal to the number of the equations. Chemical chaotic oscillations where observed by Belousov () and later on by Zhabotinsky () when they where working with chemical autocatalytic reactions. The Nobel Prize in chemistry () was awarded to Prigogine for his work on dynamics of dissipative systems (see Prigogine ) including the mathematical representation of autocatalytic reactions. A simple autocatalytic reaction is expressed by the following set of three differential equations: x˙ = (
Chaotic Modelling. Fig. Autocatalytic attractor and chaotic oscillations
a chaotic process that very small changes in initial conditions have significant impact to the future states of a system. This was verified by Edwin Lorenz in with his work on modeling the atmospheric changes. He reduced the Navier-Stokes equations, used to express fluid flows, to a system of three nonlinear coupling differential equations and performed simulations in a computer trying to model the weather changes. He surprisingly found that the system was very sensitive to small changes of initial conditions thus making the forecasts of the future weather unpredictable. Famous are the forms of his simulated paths that look like a butterfly with open wings. The three-dimensional model which he proposed has the form (σ, r and b are parameters): x˙ = −σx + σy, y˙ = −xz + rx − y, z˙ = xy − bz. The famous Lorenz attractor also known as the butterfly attractor is illustrated in Fig. . Several years later Rössler () proposed a simpler three-dimensional model including only one nonlinear term thus verifying the assumption that a set of simple
xy + x − y + m) (k + z) − xy − x, y˙ = , z˙ = y − z +k e
This model is illustrated in Fig. ; the parameters set are: e = ., k = ., m = .. The use of computing power gave rise to the exploration of chaos in astronomy and astrophysics. A paper that influenced much the future developments of the chaotic applications was due to Hénon and Heiles in . They had predicted chaos in Hamiltonian systems that could apply to astronomy and astrophysics. Few years before George Contopoulos () had also found chaotic behavior when he explored the paths of stars in a galaxy. That it was most important was that they had shown that the computer experiments had much more to show than simply verify the results coming from the mathematical formulations. Hidden and unexplored scientific fields would emerge by the use of computers. It was found that chaos could emerge from a system of three or more differential equations with at list one nonlinear term. This comes from the Poincaré–Bendixson theorem which states that a two dimensional system of nonlinear equations may have a regular behavior. Another theorem is the famous KAM theorem from the initials of the names of Kolmogorov, Arnold and Moser. This theorem applies to dynamical systems and may explain the stability or not of these systems to small perturbations. It is interesting that the chaotic forms could be quite stable as it happens for vortex and tornados. However, the main scientific discovery on chaos came only in by Michel Feigenbaum when he found that the simple logistic map could produce a chaotic sequence. Feigenbaum tried a difference equation instead of the differential equations that where used in the previous works on chaos. That is different is that chaos can emerge even
C
C
Chaotic Modelling
from only one difference equation with at list one nonlinear term. This is because a difference equation defines a recurrence scheme which is a set of numerous equations in which every equation uses the outcomes from the preceding one. The complexity resulting from a nonlinear difference equation is large and it can be measured with a power law of the number of iterations. In the logistic model a mapping into itself is defined by the difference equation and gives rise to period doubling bifurcations and chaos for a specific range of the chaotic parameter. The logistic map is of the form: xn+ = bxn ( − xn ), where b is the chaotic parameter and xn is the chaotic function (see a (xn+ , xn ) diagram of the Logistic model in Fig. ; b = .). For the logistic map as also for other maps there exists the bifurcation diagram. This is a diagram, usually two dimensional, defining the bifurcation points with respect to the chaotic parameter or parameters (see Fig. ). The chaotic modeling has also to do with strange attractors by means forms in space that have a great detail and complexity. These forms can arise in nature and also can be simulated from chaotic equations. A very interesting future of a chaotic attractor is that for a variety of initial conditions the chaotic system leads the final results or solutions to a specific area, the strange or chaotic attractor. Chaos may also arise from a set of two or more difference equations with at least one nonlinear term. The most popular model is the Hénon () model given by: xn+ = yn + − axn , yn+ = bxn . The Jacobian determinant of this model is: RRR ∂xn+ RRR R ∂xn det J = RRRR RR ∂xn+ RRR RR ∂yn
∂yn+ ∂xn ∂yn+ ∂yn
RRR RRR RRR RRR = −b. RRR RRR
The system is stable for < b < . When b = the system is area preserving, but it is unstable.
Chaotic Modelling. Fig. The bifurcation diagram
Chaotic Modelling. Fig. A carpet-like form
An alternative of the Hénon map is provided by the following cosine model: xn+ = byn + a cos(xn ) − a + , yn+ = xn . This map provides a carpet-like form (see Fig. ) for b = − and a = −.. Very many cases in nature have to do with delays. This mathematically can be modeled by a delay differential or difference equation. Simpler is to use difference equations to express delay cases. An example is the transformation of the previous Hénon map to the corresponding delay difference equation of the form:
xn+ = bxn− + − axn . This delay differential equation has the same properties of the Hénon map. In general modeling delays leads to differential or difference equations which produce oscillations and may produce chaos for appropriate selection of the parameters. One of the first proposed chaotic models including delays is the famous Mackey-Glass () model regarding oscillation and chaos in physiological control systems. Ikeda found his famous attractor in (see Fig. ; parameters a = , b = ., c = . and d = ) when he was experimenting on the light transmitted by a ring cavity system. The equations’ set is: xn+ = a + b(xn cos(φn ) − yn sin(φn )),
Chaotic Modelling. Fig. The logistic model
yn+ = b(xn sin(φ n ) + yn cos(φn )),
Chaotic Modelling
C
of the Production Engineering and Management Department, TUC (–) and participated in many committees and served as a consultant in various firms while he directed several scientific and applied projects. He was a visiting fellow in the University of Exeter, UK () and Université Libre de Bruxelles, Belgium (–). He has been the guest-editor of the journals Communications in Statistics, Methodology and Computing in Applied Probability (MCAP) and Applied Stochastic Models and Data Analysis. He is a member of the Committee and Secretary of ASMDA International Society and was the Chair or cochair of the International Conferences: th ASMDA , th ASMDA . As a member of the Greek Statistical Institute he organised and co-chaired two Annual National Conferences of the Institute (th Chania and nd Chania ). Professor Skiadas contributions mainly are directed to the modeling and simulation of innovation diffusion and application of the findings in various fields as finance and insurance, energy consumption, forecasting and modeling. The related publications include more than papers and books, including Chaotic Modelling and Simulation: Analysis of Chaotic Models, Attractors and Forms (with Charilaos Skiadas, Chapman and Hall/CRC Press, ).
Chaotic Modelling. Fig. The Ikeda attractor
Chaotic Modelling. Fig. Chaotic rotating forms
Cross References 7Stochastic Modeling, Recent Advances in
d + xn + yn The last form of difference equations express a rotationtranslation phenomenon and can give very interesting chaotic forms (see Skiadas ). Figure illustrates such a case where the rotation angle follows an inverse low regardc c ing the distance r from the origin: φ n = √ = . where the rotation angle is: φ n = c −
xn + yn
r
A chaotic bulge is located in the central part followed by elliptic trajectories in the outer part (the parameters are: a = . and b = c = ). Other interesting aspects of chaotic modeling are found in numerous publications regarding control of chaos with applications in various fields. Chaotic mixing and chaotic advection have also studied with chaotic models as well as economic and social systems.
About the Author Christos H. Skiadas is the Founder and Director of the Data Analysis and Forecasting Laboratory, Technical University of Crete (TUC). He was Vice-Rector of the Technical University of Crete (–), Chairman
References and Further Reading ⌣
Belousov BP () Periodiqeski deistvuwa reakci i ee mehanizm. [A periodic reaction and its mechanism]. ⌣
Sbornik referatov po radiacionnoi medicine (Compilation of Abstracts on Radiation Medicine) : Contopoulos G () On the vertical motions of stars in a galaxy. Stockholm Ann (): Hénon M () A two-dimensional mapping with a strange attractor. Commun Math Phys :– Hénon M, Heiles C () The applicability of the third integral of motion: some numerical experiments. Astron J :– Feigenbaum MJ () Quantitative universality for a class of nonlinear transformations. J Stat Phys :– Ikeda K () Multiple-valued stationary state and its instability of the transmitted light by a ring cavity system. Opt Commun :– Lorenz EN () Deterministic nonperiodic flow. J Atmos Sci :– Mackey MC, Glass L () Oscillation and chaos in physiological control systems. Science :– Poincaré H () Sur les équations de la dynamique et le problème de trois corps. Acta Math :– Prigogine I () Thermodynamics of irreversible processes, nd edn. Interscience, New York Rössler OE () An equation for continuous chaos. Phys Lett A :–
C
C
Characteristic Functions
Skiadas CH, Skiadas C () Chaotic modeling and simulation: analysis of chaotic models attractors and forms. Taylor & Francis/CRC Press, London ⌣ Zhabotinsky AM () Periodiqeskii process
Characteristic Functions. Table Characteristic functions of some univariate probability distributions Density f (x)
Distribution
c.f. ϕ(t)
⌣
okisleni malonovoi kisloty rastvore (issledovanie kinetiki reakcii Belousova). [Periodic
Degenerate at c
processes of malonic acid oxidation in a liquid phase.] Biofizika [Biofizika] :–
Binomial
(nx)px ( − p)n−x
Poisson
e−λ λx!
eitc
Characteristic Functions Miljenko Huzak University of Zagreb, Zagreb, Croatia Characteristic functions play an outstanding role in the theory of probability and mathematical statistics (Ushakov ). The characteristic function (c.f.) of a probability distribution function (d.f.) is the Fourier–Stieltjes transform of the d.f. More precisely, if F is a probability d.f. on d-dimensional real space Rd (d ≥ ), then its c.f. is a complex function ϕ : Rd → C such that for any t = (t , . . . , td ) ∈ Rd , d
exp {−
Gamma
Γ(α)β α
Cauchy
α π(α +x )
(x−µ) } σ
eiµt−σ
t /
sin θt θt
x α− e−x/β
( − itβ)−α e−α∣t∣
Characteristic Functions. Table List of properties of characteristic functions ϕX (t) given by () (the list follows one from Ferguson ()) ()
ϕX (t) exists for all t ∈ Rd and is continuous.
()
ϕX (Ø) = and ∣ϕx (t)∣ ≤ for all t ∈ Rd .
()
For a scalar a, ϕaX (t) = ϕX (at).
()
For a matrix A and a vector c, τ ϕAX+c (t) = eit c ⋅ ϕX (Aτ t). For X and Y independent, ϕX+Y (t) = ϕX (t)ϕY (t). .
()
⎛d ⎞ + i ∫ sin ∑ tj xj dF(x , . . . , xd ), d R ⎝ j= ⎠
If E ∣X∣ < ∞, ϕX (t) exists and is continuous and . ϕX (Ø) = i E Xτ . ..
()
where the integrals are Lebesgue–Stieltjes integrals with respect to d.f. F. If X = (X , . . . , Xd ) is a d-dimensional random vector, then c.f. ϕ = ϕX associated to X is the c.f. of its d.f. F = FX . Hence ϕX (t) = E [ei ∑j= tj Xj ] , t = (t , . . . , td ) ∈ Rd .
√ π
θ
Rd
⎛d ⎞ = ∫ cos ∑ tj xj dF(x , . . . , xd ) d R ⎝ j= ⎠
σ
Symmetric uniform over (−θ, θ)
()
ϕ(t) = ∫ ei ∑j= tj xj dF(x , . . . , xd ) :=
exp{λ(eit − )}
x
Normal
(peit + − p)n
() ()
If E[∣X∣ ] < ∞, ϕX (t) exists and is continuous and .. ϕX (Ø) = − E[XXτ ]. If P(X = c) = for a vector c, ϕX (t) = eit c . τ
If X is normal r. vec. with µ = E X and cov(X) = Σ, ϕX (t) = exp {itτ µ − tτ Σt}.
d
()
Particularly, c.f. ϕ = ϕ X : R → C of a random variable (r.v.) X is equal to ϕ(t) = E[eitX ], t ∈ R. Examples of c.f.s of some r.v.s are in Table . C.f.s have many good properties (see Table ). One of the most important properties of c.f.s is that there is a oneto-one correspondence between d.f.s and their c.f.s, which is a consequence of the Lévy inversion formula (see Chow and Teicher or Feller ). Since it is usually simpler to manipulate with c.f.s than with corresponding d.f.s,
this property makes c.f.s useful in proving many theorems on probability distributions. For example, it can be proved that the components of a random vector X = (X , . . . , Xd ) are independent r.v.s if and only if (∀t , . . . , td ∈ R) ϕX (t , . . . , td ) = ϕ X (t ) ⋅ ϕ X (t ) . . . ϕ Xd (td ). Moreover, since for any independent r.v.s X , X , . . . , Xn , c.f. of their sum Sn = X + . . . + Xn is equal to the product of their c.f.s, to obtain the d.f. of Sn , it is usually easier
C
Chebyshev’s Inequality
to find the c.f. of their sum and to apply the Lévy inversion formula than to find the convolution of their d.f.s. Another very important property of c.f.s comes from the continuity theorem (see Chow and Teicher or Feller ): r.v.s Xn , n ≥ , with corresponding c.f.s ϕ n , n ≥ , converge in law to a r.v. X with c.f. ϕ if and only if c.f.s ϕ n , n ≥ , converge to ϕ pointwise. For example, this property makes proving 7central limit theorems easier if not only possible. C.f.s have been important tools in developing theories of infinite divisible and particularly stable distributions (e.g., see Feller ; Chow and Teicher ).
Indeed, choosing Y = ∣X − µ∣ and g(x) = x gives (). The proof of Markov’s inequality is very easy: For any t > , P(Y ≥ t) = ∫ {Y≥t} d P ≤ ∫
{Y≥t}
P(X − µ ≥ t) = P(X − µ + c ≥ t + c) ≤ =
References and Further Reading Chow YS, Teicher H () Probability theory, independence, interchangeabilty, martingales. nd edn. Springer-Verlag, New York Feller W () An introduction to probability theory and its applications, Vol , nd edn. Wiley, New York Ferguson TS () A course in large sample theory. Chapman & Hall, London Lukacs E () Characteristic functions. nd edn. Griffin, London Ushakov NG () Selected topics in characteristic functions. Brill Academic Publishers, Leiden
Chebyshev’s Inequality Gerold Alsmeyer Professor Institut für Mathematische Statistik, Münster, Germany
Chebyshev’s inequality is one of the most common inequalities used in probability theory to bound the tail probabilities of a random variable X having finite variance σ = VarX. It states that σ P(∣X − µ∣ ≥ t) ≤ t
for all t > ,
()
where µ = E X denotes the mean of X. Of course, the given bound is of use only if t is bigger than the standard deviation σ. Instead of proving () we will give a proof of the more general Markov’s inequality which states that for any nondecreasing function g : [, ∞) → [, ∞) and any nonnegative random variable Y P(Y ≥ t) ≤
E g(Y) g(t)
()
E(X − µ + c) (t + c)
σ + c . (t + c)
The right-hand side becomes minimal at c = σ /t giving the one-sided tail bound P(X − µ > t) ≤
σ + t
σ
for all t > ,
()
sometimes called Cantelli’s inequality. Although Chebyshev’s inequality may produce only a rather crude bound its advantage lies in the fact that it applies to any random variable with finite variance. Moreover, within the class of all such random variables the bound is indeed tight because, if X has a symmetric distribution on {−a, , a} with P(X = ±a) = /(a ) and P(X = ) = − /a for some a > , then µ = , σ = and P(∣X∣ ≥ a) = P(∣X∣ = a) = , a which means that equality holds in () for t = a. On the other hand, tighter bounds can be obtained when imposing additional conditions on the considered distributions. On such example is the following Vysoˇcanski˘ı-Petun¯ın inequality for random variables X with an unimodal distribution: P(∣X − µ∣ ≥ t) ≤
σ t
for all t >
√ / σ,
()
This improves () by a factor / for sufficiently large t. One of the most common applications of Chebyshev’s inequality is the weak law of large numbers (WLLN). Suppose we are given a sequence (Sn )n≥ of real-valued random variables with independent increments X , X , ... such that µ n := E Xn and σn := VarXn are finite for all n ≥ . Defining n
for all t > .
g(Y) E g(X) dP ≤ . g(t) g(t)
Plainly, () provides us with the same bound σ t − for the one-sided tail probability P(X − µ > t), but in this case an improvement is obtained by the following consideration: For any c ≥ , we infer from Markov’s inequality with g(x) = x
Cross References 7Probability on Compact Lie Groups
mn := E Sn = ∑ µ k k=
n
and
sn := VarSn = ∑ σk k=
C
C
Chemometrics
About the Author
and assuming Markov’s condition s lim n n→∞ n
=
()
we infer by making use of () that, for any є > , P (∣
Sn − mn s ∣ ≥ є) ≤ n → n є n
as n → ∞
Cross References
and therefore Sn − mn → n
in probability.
(WLLN)
This result applies particularly to the case of i.i.d. X , X , ... Then mn = nµ and sn = nσ where µ := E X and σ := VarX . In this case, Chebyshev’s inequality further gives, for all є, β > , that ∑ P (∣ n≥
Sn − nµ σ ∣ ≥ є log β n) ≤ ∑ < ∞ β n n≥ є n log n
and thus, by invoking the Borel-Cantelli lemma (see 7Borel–Cantelli Lemma and Its Generalizations), Sn − nµ n log β n
→
a.s. for all β >
()
This is not quite the strong law of large numbers (β = ) but gets close to it. In fact, in order for this to derive, a stronger variant of Chebyshev’s inequality, called Kolmogorov’s inequality, may be employed which states that sn for all t > ≤k≤n t under the same independence assumptions stated above for the WLLN. Notice the similarity to Chebyshev’s inequality in that only Sn − mn has been replaced with max≤k≤n (Sk − mk ) while retaining the bound. Let us finally note that, if X has mean µ, median m and finite variance σ , then the one-sided version () of Chebyshev’s inequality shows that and P(X − µ ≤ −σ) ≤ , P(X − µ ≥ σ) ≤ in other words, the median of X is always with one standard deviation of it mean. Bibliographical notes: () dates back to Chebyshev’s original work (Chebyshev ), but is nowadays found in any standard textbook on probability theory, like (Feller ). The latter contains also a proof of the one-sided version () which differs from the one given here. () for unimodal distributions is taken from Vysoˇcanski˘ı and Petun˘ın (), see also Sellke and Sellke (). For multivariate extensions of Chebyshev’s inequality see Olkin and Pratt () and Monhor (). P (max ∣Sk − mk ∣ ≥ t) ≤
Dr. Alsmeyer is Professor at the Department of Mathematics and Computer Science of the University of Münster and was the Chairman from till . He has written more than research articles and one book. He has supervised eight Ph.D. students. 7Borel–Cantelli Lemma and Its Generalizations 7Expected Value 7Laws of Large Numbers 7Standard Deviation
References and Further Reading Chebyshev P () Des valeurs moyennes. Liouville’s J Math Pure Appl :– Feller W () An introduction to probability theory and its applications, vol II, nd edn. Wiley, New York Monhor D () A Chebyshev inequality for multivariate normal distribution. Prob Eng Inform Sci ():– Olkin I, Pratt JW () A multivariate Tchebycheff inequality. Ann Math Stat :– Sellke TM, Sellke SH () Chebyshev inequalities for unimodal distributions. Am Stat ():– Vysoˇcanski˘ı DF, Petun˘ın J¯I () Proof of the σ rule for unimodal distributions. Teor Veroyatnost i Mat Stat :–
Chemometrics Rolf Sundberg Professor of Mathematical Statistics Stockholm University, Stockholm, Sweden The role of statistics in chemistry is over a century old, going back to the Guinness brewery chemist and experimenter Gosset, more well-known under the pseudonym “Student.” For his applications, he was in need of smallsample statistical methods. Until the s, chemistry methods and instruments were typically univariate, but in that decade analytical chemistry and some other branches of chemistry had to start handling data of multivariate character. For example, instead of measuring light intensity at only a single selected wavelength, instruments became available that could measure intensities at several different wavelengths at the same time. The instrumental capacity rapidly increased, and the multivariate spectral dimension soon exceeded the number of chemical samples analysed (the “n < p” problem). In parallel, other chemists worked with Quantitative Structure–Activity Relationships (QSAR), where they tried to explain and predict biological activity or similar properties of a molecule from
Chemometrics
a large number of structural physical-chemical characteristics of the molecule, but having an empirical data set of only a moderate number of different molecules. Generally, as soon as multivariate data are of high dimension, we must expect near collinearities among the variables, and when n < p, there are necessarily exact collinearities. These were some of the problems faced, long before statisticians got used to n < p in genomics, proteomics etc. This was the birth of the field of chemometrics, a name coined by Svante Wold to characterize these research and application activities. A standard definition of Chemometrics would be of type “The development and use of mathematical and statistical methods for applications in chemistry,” with more weight on statistical than mathematical. Another characterization, formulated by Wold, is that the aim of chemometrics is to provide methods for ● ● ●
How to get chemically relevant information out of measured chemical data How to get it into data How to represent and display this information
and that in order to achieve this, chemometrics is heavily dependent on statistics, mathematics and computer science. The first task is much concerned with analysis of dependencies and relationships (regression, calibration, discrimination, etc.) within a multivariate framework, because complex chemical systems are by necessity multidimensional. The second task is largely represented by experimental design, both classical and newer, where chemometrics has contributed the idea of design in latent factors (principal variates). For representation of highdimensional data, projection on a low-dimensional latent variable space is the principal tool. Using diagrams in latent factors from PCA or other dimension-reducing methods is also a way of displaying the information found. Another type of definition, often quoted, is that “Chemometrics is what chemometricians do.” This is not only to laugh at. A vital part of chemometrics is connected with chemistry, but the methods developed might be and are applied in quite different fields, where the data analysis problems are similar, such as metabolomics, food science, sensometrics, and image analysis. This could motivate to distinguish chemometrics and chemometric methods, where the latter could as well be described as statistical methods originally inspired by problems in chemistry. A statistician’s look at the contents of Journal of Chemometrics for the period – ( papers) showed that regression and calibration dominated, covering a third of the contents. Much of this was on regularized
C
regression methods, such as PCR (Principal Components Regression) and PLSR (Partial Least Squares Regression). Other statistical areas represented were multiway methods (where each observation is a matrix or an even higherdimensional array, see Smilde et al. ), classification (discrimination and clustering), multivariate statistical process control, and occasionally other areas, for example experimental design, wavelets, genetic algorithms. A difference between chemometrics and biometrics (7biostatistics) is that chemometricians are mostly chemists by principal education, with more or less of additional statistical education, whereas biometricians are typically statisticians by education. This has had several consequences for chemometrics. Statistical methods are sometimes reinvented. Metods are sometimes proposed without a theoretical underpinning. The popular method of partial least squares (see 7Partial Least Squares Regression Versus Other Methods) is a good such example, nowadays relatively well understood, but proposed and advocated as a computational algorithm, that was widely regarded with suspicion among statisticians. Thus there is often a role for theoretical statistical studies to achieve a deeper understanding of the chemometric methods and their properties, not least to reveal how various suggested methods relate to each others.
About the Author Professor Sundberg is Past President of the Swedish Statistical Association (–). He is an Elected member of the International Statistical Institute (). He was an Associate editor for Scandinavian Journal of Statistics (–). In he was awarded (with Marie Linder) the Kowalski prize for best theoretical paper in Journal of Chemometrics (–).
Cross References 7Sensometrics
References and Further Reading Regression and calibration Brown PJ () Measurement, regression, and calibration. Oxford University Press, Oxford Martens H, Næs T () Multivariate calibration. Wiley, Chichester Sundberg R () Multivariate calibration – direct and indirect regression methodology (with discussion). Scand J Stat : – Other fields of chemometrics Carlson R, Carlson JE () Design and optimization in organic synthesis, nd edn. Elsevier, Amsterdam Martens H, Martens M () Multivariate analysis of quality. An introduction. Wiley, Chichester Smilde A, Bro R, Geladi P () Multi-way analysis with applications in the chemical sciences. Wiley, Chichester
C
C
Chernoff Bound
Two journals are devoted to chemometrics, started in / Journal of Chemometrics. John Wiley & Sons. Chemometrics and Intelligent Laboratory Systems. Elsevier. There are several introductions to chemometrics written for chemists, not listed here. Not all of them are satisfactory in their more statistical parts.
Chernoff Bound Herman Chernoff Professor Emeritus Harvard University, Cambridge, MA, USA The Chernoff Bound, due to Herman Rubin, states that if X is the average of n independent observations on a random variable X with mean µ < a then, for all t, n
P(X > a) ≤ [E(et(X−a) )] . The proof which follows shortly is a simple application of the Markov inequality that states that for a positive random variable Y, P(Y ≥ b) ≤ E(Y)/b, for b > . The Chernoff bound was a step in the early development of the important field “Large Deviation Theory.” It became prominent among computer scientists because of its usefulness in Information Theory. The Markov inequality is derived from the fact that for b > , ∞
ydF(y) ≥ bP(Y ≥ b)
E(Y) = ∫ ydF(Y) ≥ ∫ b
where F is the cumulative distribution of Y. We observe that E(ent(X−a) ) = [E(e
t(X−a)
)]
n
and hence P(ent(X−a) ≥ ) is less than or equal to the bound. This implies the Chernoff bound for t > . For t ≤ the inequality is automatically satisfied because the bound is at least one. That follows form the fact that the 7moment generating function M(t) = E(etZ ) is convex with M() = , M ′ () = E(Z) and E(X − a) < . The prominence of the bound is due to a natural inclination to extend beyond its proper range of applicability. The Central Limit Theorem (see 7Central Limit Theorems), for which an informal statement is that X is approximately normally distributed with mean µ = E(X) and variance σ /n where σ is the standard deviation of X. For large deviations (see 7Large Deviations and Applications),
or many standard deviations from the mean, the theorem implies that the probability of exceeding a would approach zero, but a naive interpretation would state that this probability would be approximately exp(−na /)(πna)−/ and could be seriously wrong. In , for a special problem of testing a simple hypothesis versus a simple alternative using a statistic of the form X where X could take on a few integer values, I realized that the normal approximation was inappropriate. I derived (Chernoff ), for a > E(X), n− log P(X > a) → inf E(et(X−a) ) t
which was, as far as I know, the first application of Large Deviation Theory to Statistical Inference. This result was used to define a measure of information useful for experimental design and to show that the Kullback-Leibler information numbers (Kullback and Leibler ; Chernoff ) measure the exponential rate at which one error probability approaches zero when the other is held constant. At the time I was informed of Cramér’s () earlier elegant derivation of more encompassing results, using exponentially tilted distributions. Cramér dealt with deviations which were not limited to those of order square root of n standard deviations, but required a condition that excluded the case which I needed, where the range of X was a multiple of the integers. Blackwell and Hodges () later dealt with that case. One of my colleagues, Herman Rubin, claimed that he could derive my results more simply, and when I challenged him, he produced the upper bound that I included in my manuscript. At the time the proof seemed so trivial, that I did not mention that it was his. I made two serious errors. First, the inequality is stronger than the upper limit implied by my result, and therefore deserves mention of authorship even though the derivation is simple. Second, because I was primarily interested in the exponential rate at which the probability approached zero, it did not occur to me that this trivially derived bound could become prominent.
About the Author For biography see the entry 7Chernoff–Savage Theorem.
Cross References 7Central Limit Theorems 7Chebyshev’s Inequality 7Kullback-Leibler Divergence 7Large Deviations and Applications
Chernoff Faces
References and Further Reading Blackwell D, Hodges JL () The probability in the extreme tail of a convolution. Ann Math Stat :– Chernoff H () A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. Ann Math Stat :– Chernoff H () Large-sample theory: parametric case. Ann Math Stat :– Cramér H () Sus un nouveau théorème-limite de la théorie des probabilités. Acutalités Scientifiques et Industrielles, , Paris Kullback S, Leibler RA () On information and sufficiency. Ann Math Stat :–
Chernoff Faces Herman Chernoff Professor Emeritus Harvard University, Cambridge, MA, USA
The graphical representation of two dimensional variables is rather straightforward. Three dimensional variables presents more of a challenge, but dealing with higher dimensions is much more difficult. Two methods using profiles and stars suffers from a confusion of which variable is represented when the dimensionality is greater than six. The method called “Chernoff Faces” (Chernoff ) involves a computer program which draws a caricature of a face when given numbers between and . These numbers correspond to features of the face. Thus one may represent the length of the nose, another curvature of the mouth, and a third the size of the eyes. If we have dimensional data, we can adjoin constants to get points in dimensional space, each represented by a face. As the point moves in dimensional space the face changes. The method was developed in response to a problem in cluster analysis (see 7Cluster Analysis: An Introduction). There are many methods proposed to do clustering. It seems that an appropriate method should depend on the nature of the data, which is difficult to comprehend without visualization. The grouping of faces which look alike serves as a preliminary method of clustering and of recognizing which features are important in the clustering. In the two original applications of the method, the scientists involved claimed that the implementation was lucky because the features which were most important were represented respectively by the size of the eyes and the shape of the face, both of which are prominent features. I claimed that it did not matter which features were selected for the
C
various variables and challenged the scientists to select an alternative choice of features for the variables to degrade the effect of the faces. Their candidate choices had little degradation effect. To test the conjecture that the choice of variables would have no effect, Rizvi and I carried out an experiment (Chernoff and Rizvi ). Of course it is clear that the conjecture cannot be absolutely sound, since the position of the pupils in the eyes cannot be detected if the eyes are small and other features interact similarly. However we set up an experiment where subjects were supposed to cluster faces into two groups of approximately each. The faces were generated from two six dimensional 7multivariate normal distributions with means δ units apart, in Mahalanobis distance, and identity covariance matrix. These data were then subjected to a linear transformation to an -dimensional space, and feature selections were made at random. The subjects were given three clustering problem. For the first δ was so large that there was no problem recognizing the clusters. That was a practice problem to train the students in the experiment. For the other two problems two choices of δ were made to establish greater difficulty in separating the two distributions. The result of this experiment was that when the error rate in clustering varies from % to %, the typical random permutations could change the error rate by a proportion which decreases from % to %. Originally, Faces were designed to serve to understand which variables were important and which interacted with each other. Once such relations are understood, analytic methods could be used to probe further. In many applications, Faces could also be used to comprehend data where the roles of the various factors were well understood. For example, in business applications, a smiling face could indicate that some aspect of the business was doing well. With training of the users, such applications could be useful in providing a quick and easy comprehension of a moderately complicated system. For example, one could use a face to represent the many meters an airplane pilot watches, so that he could be alerted when the face begins to look strange. The method of stars could also serve such a function. Jacob () used faces to represent five particular scales of the Minnesota Multiphasic Personality Inventory (MMPI). The scales represented Hypochondriasis, Depression, Paranoia, Schizophrenia and Hypomania. Realizing that training a psychologist to recognize a smiling face as belonging to a depressed patient would be difficult, he developed an innovative approach to selecting features for the five scales. He presented a random selection of faces
C
C
Chernoff-Savage Theorem
to some psychologists and asked them to rate these faces on the MMPI scales. Then he used regression techniques to decide how the numerical values of an MMPI scale should be translated into features of the face, so that the face presented to a psychologist would resemble that of a person with those scaled values. This would facilitate the process of training psychologists to interpret the faces. The method of Faces handles many dimensions well. For more than variables, one could use a pair of faces. It does not deal so well with a large number of faces unless we have a time series in which they appear in succession. In that case they can be used to detect changes in characteristics of important complicated systems. It seems that face recognition among humans is handled by a different part of the brain than that handling other geometrical data and humans are sensitive to very small changes in faces. Also, it seems that cartoons and caricatures of faces are better remembered than realistic representations. Before the computer revolution, graphical representations, such as nomograms, could be used to substitute for accurate calculations. The Faces are unlikely to be useful for calculation purposes.
About the Author For biography see the entry 7Chernoff–Savage Theorem.
Cross References 7Cluster Analysis: An Introduction
References and Further Reading Chernoff H () The use of faces to represent points in k-dimensional space graphically. J Am Stat Assoc :– Chernoff H, Rizvi MH () Effect on classification error of random permutations of features in representing multivariate data by faces. J Am Stat Assoc :– Jacob RJK () Facial representation of multivariate data. In: Wang PCC (ed) Graphical representation of multivariate data. Academic, New York, pp –
the t-test for normal alternatives and more efficient for nonnormal alternatives. To be more precise, we assume that we have two large samples, of sizes m and n with N = m + n, from two distributions which are the same except for a translation parameter which differs by an amount δ. To test the hypothesis that δ = against one sided alternatives, we use a test statistic of the form N
TN = m− ∑ ENi zNi i=
where zNi is one or zero depending on whether the ith smallest of the N observations is from the first or the second sample. For example the Wilcoxon test is of the above form with ENi = i/N. It was more convenient to represent the test in the form ∞
TN = ∫
−∞
JN [HN (x)]dFm (x).
where Fm and Gn are the two sample cdf ’s, λ N = m/N and HN = λ N Fm + ( − λ N )Gn . These two forms are equivalent when ENi = JN (i/N). The proof of the conjecture required two arguments. One was the 7asymptotic normality of T when δ =/ . The Chernoff-Savage theorem (Chernoff and Savage ) establishes the asymptotic normality, under appropriate regularity conditions on JN , satisfied by c , using an argument where Fm and Gn are approximated by continuous time 7Gaussian Processes, and the errors due to the approximation are shown to be relatively small. The second argument required a variational result using the Pitman measure of local efficacy of the test of δ = , which may be calculated as a function of the underlying distribution. For distributions with variance , the efficiency of the test relative to the t-test is minimized with a value of for the normal distribution. It follows that the c test is as efficient as the t-test for normal translation alternatives and more efficient for nonnormal translation alternatives.
About the Author
Chernoff-Savage Theorem Herman Chernoff Professor Emeritus Harvard University, Cambridge, MA, USA Hodges and Lehmann () conjectured in that the nonparametric competitor to the t-test, the Fisher-YatesTerry-Hoeffding or c test (Terry ), was as efficient as
Dr. Herman Chernoff (born in New York City on July , ) is Professor Emeritus of Statistics at Harvard University and Emeritus Professor at M.I.T. He received a PhD in Applied Mathematics at Brown University in under the supervision of Abraham Wald (at Columbia University). Dr. Chernoff worked for the Cowles Commission at the University of Chicago and then spent three years in the Mathematics Department at the University of Illinois before joining the Department of Statistics at Stanford University in , where he remained for years.
Chi-Square Distribution
He moved to M.I.T. in , where he founded the Statistics Center. Since he has been in the Department of Statistics at Harvard. He retired from Harvard in . Professor Chernoff was President of the Institute of Mathematical Statistics (–) and is an Elected member of both the American Academy of Arts and Sciences and the National Academy of Sciences. He has been honored for his contributions in many ways. He is a recipient of the Townsend Harris Medal and Samuel S. Wilks Medal “for outstanding research in large sample theory and sequential analysis, for extensive service to scholarly societies and on government panels, for effectiveness and popularity as a teacher, and for his continuing impact on the theory of statistics and its applications in diverse disciplines” (). He was named Statistician of the Year, Boston Chapter of the ASA (). He holds four honorary doctorates. Professor Chernoff is the co-author, with Lincoln Moses, of a classic text, now a Dover Reprint, entitled Elementary Decision Theory. He is also the author of the SIAM monograph entitled Sequential Analysis and Optimal Design. The book Recent Advances in Statistics (MH Rizvi, J Rustagi and D Siegmund (Eds.), Academic Press, New York, ) published in honor of his th birthday in contained papers in the fields where his influence as a researcher and teacher has been strong: design and sequential analysis, optimization and control, nonparametrics, large sample theory and statistical graphics.
Cross References 7Asymptotic Normality 7Asymptotic Relative Efficiency in Testing
C References and Further Reading Chernoff H, Savage IR () Asymptotic normality and efficiency of certain nonparametric test statistics. Ann Math Stat :– Hodges JL, Lehmann EL () The efficiency of some nonparametric competitors to the t-test. Ann Math Stat :– Terry ME () Some rank order tests which are most powerful against specific parametric alternatives. Ann Math Stat : –
Chi-Square Distribution Miljenko Huzak University of Zagreb, Zagreb, Croatia
The chi-square distribution is one of the most important continuous probability distributions with many uses in statistical theory and inference. According to O. Sheynin (), Ernst Karl Abbe obtained it in , Maxwell formulated it for three degrees of freedom in , and Boltzman discovered the general expression in . Lancaster () ascertained that Bienaymé derived it as early as in . However, their derivations “had no impact on the progress of the mainstream statistics” (R. L. Plackett , p. )
df=1 df=2
0.4
df=3 df=5
0.3
df=7 0.2
df=10
0.1
0
2
4
6
7Gaussian Processes 7Student’s t-Tests 7Wilcoxon–Mann–Whitney Test 7Wilcoxon-Signed-Rank Test
0.5
0.0
C
8
10
12
14
Chi-Square Distribution. Fig. Densities of χ -distributions with , , , , , and degrees of freedom (df )
C
Chi-Square Goodness-of-Fit Tests: Drawbacks and Improvements
since chi-square is not only a distribution, but also a statistic and a test procedure, all of which arrived simultaneously in the seminal paper written by Karl Pearson in . Let n ≥ be a positive integer. We say that a random variable (r.v.) has χ (chi-square, χ is pronounced ki as in kind) distribution with n degrees of freedom (d.f.) if it is absolutely continuous with respect to the Lebesgue measure with density: ⎧ ⎪ ⎪ ⎪ ⎪ f (x) = ⎨ − ⎪ ⎪ ⎪ Γ ( n ) −n/ xn/− e−x/ ⎪ ⎩
if x ≤ if x >
where Γ denotes the Gamma function. Figure shows some of the densities. Hence, the χ -distribution (with n d.f.) is equal to the Γ - distribution with the parameters (n/, ), that is, with the mean and variance equal to n and n respectively. The χ -distribution is closely connected with the normal distribution. It turns out that the sample variance S of a random sample from a normally distributed population has, up to the constant, the χ – sample distribution. More precisely, if X , . . . , Xn are independent and identically distributed normal r.v.s with the population variance σ , then n− ⋅ S = ((X − X) + . . . + (Xn − X) ) σ σ = (X + X + ⋅ ⋅ ⋅ + Xn − nX ) σ is a χ -distributed r.v. with n − d.f. (see e.g., Shorack ). This is a consequence of a more general property of the normality (Feller ). For example, let X be an n-dimensional standard normal vector, that is, a random vector X = (X , . . . , Xn ) such that its components X , . . ., Xn are independent and normally distributed with mean and variance equal to and respectively. Then the square of the Euclidean norm of X, ∣X∣ = X + . . . + Xn , is χ -distributed with n d.f. If means of the components of X are non-zero, then ∣X∣ has non-central χ -distribution with n d.f. and non-centrality parameter equal to the square of the mean of X. In this generality, χ -distribution is the central χ -distribution, that is, a χ -distribution with non-centrality parameter equal to . In statistics, many test statistics have a χ or asymptotic χ -distribution. For example, goodness of fit χ -tests are based on the so-called Pearson’s χ -statistics or general χ -statistics that have, under appropriate null-hypothesis, asymptotic χ -distributions; The Friedman test statistic and likelihood ratio tests are also based on asymptotically χ -distributed test statistic (see Ferguson ). Generally, appropriately normalized quadratic forms of normal (and
asymptotic normal) statistics have χ (and asymptotic χ ) distributions. Non-central χ -distributions are used for calculating the power function of tests based on quadratic forms of normal or asymptotic normal statistics.
Cross References 7Categorical Data Analysis 7Chi-Square Goodness-of-Fit Tests: Drawbacks and
Improvements 7Chi-Square Test: Analysis of Contingency Tables 7Chi-Square Tests 7Continuity Correction 7Gamma Distribution 7Relationships Among Univariate Statistical Distributions 7Statistical Distributions: An Overview 7Tests for Homogeneity of Variance
References and Further Reading Feller W () An introduction to probability theory and its applications, vol , nd edn. Wiley, New York Ferguson TS () A course in large sample theory. Chapman & Hall, London Lancaster HO () Forerunners of the Pearson χ . Aust J Stat : – Pearson K () On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can reasonably be supposed to have arisen from random sampling. Philos Mag ():– Plackett RL () Karl Pearson and the Chi-squared test. Int Stat Rev ():– Sheynin OB () Studies in the history of probability and statistics. XXV. On the history of some statistical laws of distribution. Biometrika ():– Shorack GR () Probability for statisticians, Springer-Verlag, New York
Chi-Square Goodness-of-Fit Tests: Drawbacks and Improvements Vassiliy Voinov , Mikhail Nikulin Professor Kazakhstan Institute of Management, Economics and Strategic Research, Almaty, Kazakhstan Professor University of Victor Segalen, Bordeaux, France The famous chi-squared goodness-of-fit test was discovered by Karl Pearson in . If the partition of a sample space is such that observations are grouped over r disjoined
Chi-Square Goodness-of-Fit Tests: Drawbacks and Improvements
intervals ∆ i , and denoting ν i observed frequencies and npi (θ) expected that correspond to a multinomial scheme, the Pearson’s sum is written r
χ = Xn (θ) = ∑ i=
(ν i − npi (θ)) = VT (θ)V(θ), npi (θ)
()
where V(θ) is a vector with components vi (θ)=(ν i − npi (θ))(npi (θ))−/ , i=, . . . , r. If the number of observations n → ∞, the statistic () for a simple null hypothesis, specifying the true value of θ, will follow chi-squared probability distribution with r − degrees of freedom. Until , Pearson believed that the limit distribution of his chi-squared statistic would be the same if unknown parameters of the null hypothesis were replaced by estimates based on a sample (Stigler (), p. ). Stigler noted that this major error of Pearson “has left a positive and lasting impression upon the statistical world.” It would be better to rephrase this sentence as follows: “has left a positive (because it inspired the further development of the theory of chi-squared test) and lasting ‘negative’ impression”. Fisher () clearly showed that the number of degrees of freedom of the Pearson’s test must be reduced by the number of parameters estimated by a sample. The Fisher’s result is true if and only if parameters are estimated by grouped data (minimizing Pearson’s chi-squared sum, using multinomial maximum likelihood estimates (MLEs) for grouped data, or by any other asymptotically equivalent procedure). Nowadays, the Pearson’s test with unknown parameters replaced by grouped data estimates θˆn is known as Pearson-Fisher test Xn (θˆn ). Chernoff and Lehmann () showed that replacing unknown parameters in () by their maximum likelihood estimates based on nongrouped data would dramatically change the limit distribution. In this case, it will follow a distribution that in general depends on unknown parameters and, hence, cannot be used for testing. What is difficult to understand for those who apply chi-squared tests is that an estimate is a realization of a random variable with its own probability distribution and that a particular estimate can be too far from the actual unknown value of a parameter or parameters. This misunderstanding is rather typical for those who apply both parametric and non-parametric tests for compound hypotheses. Roy () extended Chernoff and Lehmann’s result to the case of random grouping intervals. Molinari () derived the limit distribution of Pearson’s sum if moment type estimates (MMEs) based on raw data are used. Like the case of MLEs it depends on unknown parameters.
C
Thus, a problem of deriving a test statistic, where limiting distribution will not depend on parameters, is aroused. Dahiya and Gurland () showed that for location and scale families with properly chosen random cells, the limit distribution of Pearson’s sum may not depend on unknown parameters but on the null hypothesis. Being distributionfree, such tests can be used in practice, but for each specific null distribution one has to evaluate corresponding critical values. So, two ways of constructing distributionfree Pearson’s type tests are to use proper estimates of unknown parameters (e.g., based on grouped data), or to use specially constructed grouping intervals. Another possible way is to modify the Pearson’s sum such that its limit probability distribution would not depend on unknowns. Nikulin (), using a very general theoretical approach (nowadays known as Wald’s method (see Moore )), solved the problem in full for any continuous probability distribution if one will use random cells based on predetermined probabilities to fall into a cell with random boundaries depending on efficient estimates (MLEs or best asymptotically normal (BAN) estimates) of unknown parameters. Rao and Robson (), using a much less general heuristic approach, confirmed the result of Nikulin for a particular case of exponential family of distributions. Formally their result fully coincides with that of Nikulin () Yn (θˆn ) = Xn (θˆn ) + VT (θˆn )B(J − Jg )− BT V(θˆn ), () where J and Jg = BT B are Fisher information matrices for non-grouped and grouped data correspondingly, and ∂p i (θ) , i = , . . . , r, B is a matrix with elements bij = √p(θ) ∂θ j i
j = , . . . , s. The statistic () can be presented also as (Moore and Spruill ()) Yn (θˆn ) = VT (θˆn )(I − BJ− BT )− V(θˆn ).
()
The statistic () or (), suggested first by Nikulin (a) for testing the normality, will be referred to subsequently as Nikulin-Rao-Robson (NRR) test. Nikulin () assumed that only asymptotically efficient estimates of unknown parameters (e.g., MLEs based on non-grouped data or BAN estimates) are used for testing. Singh (), Spruill (), and Lemeshko et al. () showed that the NRR test is asymptotically optimal in some sense. This optimality is not surprising because the second term of () depends on the difference between Fisher’s matrices for grouped and non-grouped data that possibly takes the information lost in full (Voinov ()). Dzhaparidze and Nikulin √ () generalized Fisher’s idea to improve any n- consistent estimator to make it asymptotically as efficient as
C
C
Chi-Square Goodness-of-Fit Tests: Drawbacks and Improvements
MLE. This gives the following way of chi-squared test modification: improve an estimator first and then use the NRR statistic. Since this way is not simple computationally, it is worth considering other modifications. At this point it is important to note that the NRR test is very suitable for describing censored data (Habib and Thomas ()). Dzhaparidze and Nikulin () proposed a modification of the standard Pearson’s statistic valid for any square root of n consistent estimate θ˜n of an unknown parameter Un (θ˜n ) = VT (θ˜n )B(BT B)− BT V(θ˜n ). This test (the DN test), like the asymptotically equivalent Pearson-Fisher one, is not powerful for equiprobable cells (McCulloch (), Voinov et al. ()) but it can be rather powerful if an alternative hypothesis is specified and one uses the Neyman-Pearson classes for data grouping. Having generalized the idea of Dzhaparidze and Nikulin (), Singh () suggested a generalization of the RRN test () √ valid for any n - consistent estimator θ˜n of an unknown parameter Qs (θ˜n ) = V∗T (θ˜n )(I − BJ− BT )− V∗ (θ˜n ), where V∗ (θ˜n ) = V(θ˜n ) − BJ− W(θ˜n ), and W(θ˜n ) = √
n n
∑ i=
∂ ln f (X i ,θ) ∣θ=θ˜n . ∂θ
A unified large-sample theory of general chi-squared statistics for tests of fit was developed by Moore and Spruill (). Moore (), based upon Wald’s approach, formulated a general recipe for constructing modified chi-squared tests for any square root of n consistent estimator that actually is a generalization of Nikulin’s idea. He was first to show that a resulting Wald’s quadratic form does not depend on the way of limit covariance matrix of generalized frequencies inverting. Hsuan and Robson () showed that a modified statistic will not be the same as () in the case of moment type estimates (MMEs) of unknown parameters. They succeeded in deriving the limit covariance matrix for generalized frequencies and proved the theorem that a corresponding Wald’s quadratic form will follow in the limit the chi-squared distribution. Hsuan and Robson provided the test statistic explicitly for the exponential family of distributions, when MMEs coincide with MLEs, thus confirming the already known result of Nikulin (). Hsuan and Robson have not derived the general modified test based on MMEs θ¯n explicitly. This was done later by Mirvaliev (). Taking into account the input of Hsuan and Robson, and Mirvaliev, this test will be referred to subsequently as the Hsuan-Robson-Mirvaliev (HRM) statistic Yn (θ¯n ) = Xn (θ¯n ) + Rn (θ¯n ) − Qn (θ¯n ).
()
Explicit expressions for quadratic forms Rn (θ¯n ) and Qn (θ¯n ) are given, e.g., in Voinov et al. (). The
approach, based on Wald’s transformation, was also used by Bol’shev and Mirvaliev (), Nikulin and Voinov (), Voinov and Nikulin (), and by Chichagov () for minimum variance unbiased estimators (MVUEs). It is important to mention two types of decompositions of classical and modified chi-squared tests. The first way decomposes a modified test on a sum of the classical Pearson’s test and a correcting term that makes the test chi-squared distributed being distribution free in the limit (Nikulin ()). A much more important decomposition was first suggested by McCulloch () (see also Mirvaliev ()). This is a decomposition of a test on a sum of the DN statistic and an additional quadratic form being asymptotically independent on the DN statistic. Denoting Wn (θ) = VT (θ)B(BT B)− BT V(θ) and Pn (θ) = VT (θ)B(J − Jg )− BT V(θ) the decomposition of the NRR statistic () in case of MLEs will be Yn (θˆn ) = ˆ + Pn (θˆn )) , where Un (θˆn ) is asymptotUn (θˆn ) + (Wn (θ) ˆ + Pn (θˆn )), and on Wn (θ). ˆ ically independent on (Wn (θ) ¯ The decomposition of the HRM statistic () is Yn (θ n ) = ¯ + Rn (θ¯n ) − Qn (θ¯n )) , where Un (θ¯n ) Un (θ¯n ) + (Wn (θ) ¯ + Rn (θ¯n ) − Qn is asymptotically independent on (Wn (θ) ¯ ¯ (θ n )), but is asymptotically correlated with Wn (θ). The decomposition of a modified chi-squared test on a sum of the DN statistic and an additional term is of importance because the DN test based on non-grouped data is asymptotically equivalent to the Pearson-Fisher’s (PF) statistic for grouped data. Hence, that additional term takes into account the Fisher’s information lost due to grouping. Later it was shown (Voinov et al. ()) that the DN part, like the PF test, is (for equiprobable cells, for example) insensitive to some alternative hypothesis in case of equiprobable cells (fixed or random) and would be sensitive to it for, e.g., non-equiprobable two Neyman-Pearson classes. For equiprobable cells this suggests using the difference between the modified statistic and the DN part that will be the most powerful statistic in case of equiprobable cells (McCulloch (), Voinov et al. ()). It became clear that the way of sample space partitioning essentially influences power of a test. Ronald Fisher () was the first to note that “in some cases it is possible to separate the contributions to χ made by the individual degrees of freedom, and so to test the separate components of a discrepancy.” Cochran () wrote “that the usual χ tests are often insensitive, and do not indicate significant results when the null hypothesis is actually false” and suggested to “use a single degree of freedom, or a group of degrees of freedom, from the total χ ,” to obtain more powerful and appropriate test. The
Chi-Square Goodness-of-Fit Tests: Drawbacks and Improvements
problem of implementing the idea of Fisher and Cochran was that decompositions of Pearson’s sum and modified test statistics were not known at that time. Anderson () (see also Boero et al. ()) was possibly the first who to decompose the Pearson’s χ for a simple null hypothesis into a sum of independent χ random variables in case of equiprobable grouping cells. A parametric decomposition of Pearson’s χ in case of non-equiprobable cells based on ideas of Mirvaliev () was obtained by Voinov et al. () in an explicit form. At the same time Voinov et al. () presented parametric decompositions of NRR and HRM statistics. Voinov () and Voinov and Pya () introduced vector-valued goodness-of-fit tests that, in some cases, can provide a gain in power for specified alternatives.
About the Authors Vassiliy Voinov is a Professor of the Operations Management and Information Systems Department, Kazakhstan Institute of Management, Economics and Strategic Research (KIMEP). He received his engineering diploma at Tomsk State University; Candidate of Science degree in Kazakh Academy of Science; Doctor of Science degree in Joint Institute for Nuclear Research, Dubna, Moscow region; Professor degree in Kazakh State Polytechnic University, and also in received a Professor’s degree in KIMEP. He has professional experience as an invited professor in statistics at the University Victor Segalen Bordeaux, France. Vassiliy Voinov participated in many international conferences and has more than research papers and books, including: Unbiased Estimators and Their Applications, Volume : Univariate Case, and Volume : Multivariate Case (with M. Nikulin, Kluwer Academic Publishers: Dordrecht, and ). Mikhail S. Nikulin (Nikouline) is Professor of Statistics at the University Victor Segalen, Bordeaux . He earned his doctorate in the Theory of Probability and Mathematical Statistics from the Steklov Mathematical Institute in Moscow in , under supervision of Professor L. N. Bol’shev. He was Dean of the the Faculty l’UFR “Sciences and Modélisation”, the University Victor Segalen (–) and Head of the Laboratory EA “Mathematical Statistics and its Applications” (–). Professor Nikulin has (co-)authored over papers, edited volumes and books, including A Guide to Chi-Squared Testing (with P.E. Greenwood, Wiley, ), Probability, Statistics and Modelling in Public Health (with D. Commenges, Springer, ), and was the editor of the volume Advances in Degradation Modeling: Applications to Reliability, Survival Analysis and Finance (with N. Balakrishnan, W. Kahle, N. Limnios and
C
C. Huber-Carol, Birkhäuser, ). His name is associated with several statistics terms: Dzhaparidze-Nikulin statistic, Rao-Robson-Nikulin statistic, Bagdonaviˇcius-Nikulin model, and Bagdonaviˇcius-Nikulin estimator.
Cross References 7Chi-Square Distribution 7Chi-Square Test: Analysis of Contingency Tables 7Chi-Square Tests
References and Further Reading Anderson G () Simple tests of distributional form. J Economet :– Boero G, Smith J, Wallis KF () The sensitivity of chi-squared goodness of-fit tests to the partitioning of data. Economet Rev :– Bol’shev LN, Mirvaliev M () Chi-square goodness-of-fit test for the Poisson, binomial, and negative binomial distributions. Theory Probab Appl :– (in Russian) Chernoff H, Lehmann EL () The use of maximum likelihood estimates in tests for goodness of fit. Ann Math Stat : – Chichagov VV () Unbiased estimates and chi-squared statistic for one-parameter exponential family. In: Statistical methods of estimation and hypotheses testing, vol . Perm State University, Perm, Russia, pp – Cohran G () Some methods for strengthening the common χ tests. Biometrics :– Dahiya RC, Gurland J () Pearson chi-squared test of fit with random intervals. Biometrica :– Dzhaparidze KO, Nikulin MS () On a modification of the standard statistic of Pearson. Theory Probab Appl : – Dzhaparidze KO, Nikulin MS () On evaluation of statistics of chi-square type tests. In: Problem of the theory of probability distributions, vol . Nauka, St. Petersburg, pp. – Fisher RA () The condition under which χ measures the discrepancy between observation and hypothesis. J R Stat Soc :– Fisher RA () Partition of χ into its components. In: Statistical methods for research workers. Oliver and Boyd, Edinburg Habib MG, Thomas DR () Chi-square goodness-of-fit tests for randomly censored data. Ann Stat :– Hsuan TA, Robson DS () The goodness-of-fit tests with moment type estimators. Commun Stat Theory Meth A:– Lemeshko BYu, Postovalov SN, Chimitiva EV () On the distribution and power of Nikulin’s chi-squared test. Ind Lab :– (in Russian) McCulloch CE () Relationships among some chi-squared goodness of fit statistics. Commun Stat Theory Meth :– Mirvaliev M () An investigation of generalized chi-squared type statistics. Academy of Sciences of the Republic of Uzbekistan, Tashkent, Doctoral dissertation Molinari L () Distribution of the chi-squared test in nonstandard situations. Biometrika :– Moore DS () Generalized inverses, Wald’s method and the construction of chisquared tests of fit. J Am Stat Assoc : –
C
C
Chi-Square Test: Analysis of Contingency Tables
Moore DS, Spruill MC () Unified large-sample theory of general chisquared statistics for tests of fit. Ann Stat :– Nikulin MS (a) Chi-square test for continuous distributions. Theory Probab Appl :– Nikulin MS (b) Chi-square test for continuous distributions with shift and scale parameters. Theory Probab Appl : – Nikulin MS, Voinov VG () A chi-square goodness-of-fit test for exponential distributions of the first order. Springer-Verlag Lect Notes Math :– Rao KC, Robson DS () A chi-squared statistic for goodnessof-fit tests within the exponential family. Commun Stat : – Roy AR () On χ statistics with variable intervals. Technical report N, Stanford University, Statistics Department Singh AC () On the optimality and a generalization of Rao– Robson’s statistic. Commun Stat Theory Meth , – Spruill MC () A comparison of chi-square goodness-of-fit tests based on approximate Bahadur slope. Ann Stat :– Stigler SM () Karl Pearson’s theoretical errors and the advances they inspired. Stat Sci :– Voinov V () On optimality of the Rao–Robson–Nikulin test. Ind Lab :– Voinov V () A decomposition of Pearson–Fisher and Dzhaparidze–Nikulin statistics and some ideas for a more powerful test construction. Commun Stat Theory Meth ():– Voinov V, Pya N () A note on vector-valued goodness-of-fit tests. Commun Stat ():– Voinov V, Nikulin MS, Pya N () Independently distributed in the limit components of some chi-squared tests. In: Skiadas CH (ed) Recent advances in stochastic modelling and data analysis. World Scientific, New Jersey Voinov V, Pya N, Alloyarova R () A comparative study of some modified chi-squared tests. Commun Stat Simulat Comput :–
Chi-Square Test: Analysis of Contingency Tables David C. Howell Professor Emeritus University of Vermont, Burlington, VT, USA
for the more common “contingency table,” in which categorization is across two or more dimensions. Voinov and Nikulin, this volume, discuss the controversy over the correct form for the goodness of fit test. This entry will focus on the lack of agreement about tests on contingency tables. In the Vermont State legislature approved a bill authorizing civil unions. The vote can be broken down by gender to produce the following table, with the expected frequencies given in parentheses. The expected frequencies are computed as Ri × Cj /N, where Ri and Cj represent row and column marginal totals and N is the grand total.
Vote Yes
No
Women
Men
(.)
(.)
(.)
Total
Total
(.)
The standard Pearson chi-square statistic is defined as χ = ∑ ∑
(Oij − Eij ) ( − .) = +⋯ Eij .
( − .) = . . where i and j index the rows and columns of the table. (For the goodness-of-fit test we simply drop the subscript j.) The resulting test statistic from the formula on the left is approximately distributed as χ on (r − )(c − ) degrees of freedom. The probability of χ ≥ . on df = ., so we can reject the null hypothesis that voting behavior is independent of gender. (Pearson originally misidentified the degrees of freedom, Fisher corrected him, though Pearson long refused to recognize the error, and Pearson and Fisher were enemies for the rest of their lives.) +
The term “chi-square” refers both to a statistical distribution and to a hypothesis testing procedure that produces a statistic that is approximately distributed as the 7chisquare distribution. In this entry the term is used in its second sense.
Likelihood Ratio Chi-Square
Pearson’s Chi-Square
G = ∑ Oij log (
The original chi-square test, often known as Pearson’s chisquare, dates from papers by Karl Pearson in the earlier s. The test serves both as a “goodness-of-fit” test, where the data are categorized along one dimension, and as a test
Pearson’s chi-square statistic is not the only chi-square test that we have. The likelihood ratio chi-square builds on the likelihood of the data under the null hypothesis relative to the maximum likelihood. It is defined as
+ ln ( = .
Oij ) = [ ln ( ) + ln ( ) Eij . .
) + ln ( )] . .
Chi-Square Test: Analysis of Contingency Tables
This result is slightly larger than the Pearson chi-square of .. One advantage of the likelihood ratio chi-square is that G for a large dimensional table can be neatly decomposed into smaller components. This cannot be done exactly with Pearson’s chi-square, and G is the usual statistic for log-linear analyses. As sample sizes increase the two chi-square statistics converge.
Small Expected Frequencies Probably no one would object to the use of the Pearson or likelihood ratio chi-square tests for our example. However, the chi-square statistic is only approximated by the chi-square distribution, and that approximation worsens with small expected frequencies. When we have very small expected frequencies, the possible values of the chi-square statistic are quite discrete. For example, for a table with only four observations in each row and column, the only possible values of chi-square are , , and . It should be clear that a continuous chi-square distribution is not a good match for a discrete distribution having only three values. The general rule is that the smallest expected frequency should be at least five. However Cochran (), who is generally considered the source of this rule, acknowledged that the number “” seems to be chosen arbitrarily. Yates proposed a correction to the formula for chisquare to bring it more in line with the true probability. However, given modern computing alternatives, Yates’ correction is much less necessary and should be replaced by more exact methods. For situations in which we do not satisfy Cochran’s rule about small expected frequencies, the obvious question concerns what we should do instead. This is an issue over which there has been considerable debate. One of the most common alternatives is Fisher’s Exact Test (see below), but even that is controversial for many designs.
Alternative Research Designs There are at least four different research designs that will lead to data forming a contingency table. One design assumes that all marginal totals are fixed. Fisher’s famous “tea-tasting” study had four cups of tea with milk added first and four with milk added second (row totals are fixed). The taster had to assign four cups to each guessed order of pouring, fixing the column totals. The underlying probability model is hypergeometric, and Fisher’s exact test () is ideally suited to this design and gives an exact probability. This test is reported by most software for × tables, though it is not restricted to the × case. Alternatively we could fix only one set of marginals, as in our earlier example. Every replication of that experiment would include women and men, although
C
the vote totals could vary. This design is exactly equivalent to comparing the proportion of “yes” votes for men and women, and it is based on the 7binomial distribution. The square of a z-test on proportions would be exactly equal to the resulting chi-square statistic. One alternative analysis for this design would be to generate all possible tables with those row marginals and compute the percentage of obtained chi-square statistics that are as extreme as the statistic obtained from the actual data. Alternatively, some authorities recommend the use of a mid-p value, which sums the probability of all tables less likely than the one we obtained, plus half of the probability of the table we actually obtained. For a different design, suppose that we had asked Vermont citizens to record their opinion on civil unions. In this case neither the Gender nor Vote totals would be fixed, only the total sample size. The underlying probability model would be multinomial. Pearson’s chi-square test would be appropriate, but a more exact test would be obtained by taking all possible tables (or, more likely, a very large number of randomly generated tables) with observations and calculating chi-square for each. Again the probability value would be the proportion of tables with more extreme outcomes than the actual table. And, again, we could compute a mid-p probability instead. Finally, suppose that we went into college classrooms and asked the students to vote. In this case not even the total sample size is fixed. The underlying probability model here is Poisson. Computer scripts written in R are available for each model with a fixed total sample size at http://www.uvm.edu /~dhowell/StatPages/chi-square-alternatives.html
Summary Based on a large number of studies of the analysis of contingency tables, the current recommendation would be to continue to use the standard Pearson chi-square test whenever the expected cell frequencies are sufficiently large. There seems to be no problem defining large as “at least .” With small expected frequencies 7Fisher’s Exact Test seems to perform well regardless of the sampling plan, but 7randomization tests adapted for the actual research design, as described above, will give a somewhat more exact solution. Recently Campbell () carried out a very large sampling study on × tables comparing different chi-square statistics under different sample sizes and different underlying designs. He found that across all sampling designs, a statistic suggested by Karl Pearson’s son Egon Pearson worked best in most situations. The statisN
. (For the justification for that tic is defined as χ N− adjustment see Campbell’s paper.) Campbell found that as
C
C
Chi-Square Tests
long as the smallest expected frequency was at least one, the adjusted chi-square held the Type I error rate at very nearly α. When the smallest expected frequency fell below , Fisher’s Exact Test did best.
The chi-square statistic on n degrees of freedom is defined as n (Y − µ) , χ n = ∑ zi = ∑ σ i=
About the Author
where zi is normally distributed with mean zero and standard deviation one (Winkler and Hayes , pp. –). If one were repeatedly to draw samples of one Y score from a normally distributed population, transform that score to a standard z score, and then square that z score, the resulting distribution of squared z scores would be a χ distribution on one degree of freedom. If one were repeatedly to draw samples of three scores, standardize, square, and sum them, the resulting distribution would be χ on three degrees of freedom. Because the χ statistic is so closely related to the normal distribution, it is also closely related to other statistics that are related to the normal distribution, such as t and F. One simple application of the χ statistic is to test the null hypothesis that the variance of a population has a specified value (Winkler and Hayes , pp. –; Wuensch ). From the definition of the sample variance,
David Howell is a Professor Emeritus (since ), and former chair of the Psychology department at the University of Vermont (–) and (–). Professor Howell’s primary area of research is in statistics and experimental methods. He has authored well known texts: Statistical Methods for Psychology (Wadsworth Publishing, th ed., ), Fundamental Statistics for Behavioral Sciences (Wadsworth Publishing, th ed., ), and is a coauthor (with Brian Everitt) of a four volume Encyclopedia of Statistics in Behavior Science (Wiley & Sons, ).
Cross References 7Chi-Square Distribution 7Chi-Square Goodness-of-Fit Tests: Drawbacks and
Improvements 7Chi-Square Tests
References and Further Reading Campbell I () Chi-squared and Fisher-Irwin tests of twoby-two tables with small sample recommendations. Stat Med :– Cochran WG () The χ test of goodness of fit. Ann Math Stat :– Fisher RA () The logic of inductive inference. J R Stat Soc : –
Chi-Square Tests Karl L. Wuensch Professor East Carolina University, Greenville, NC, USA
The χ statistic was developed by Karl Pearson () as a means to compare an obtained distribution of scores with a theoretical distribution of scores. While it is sometimes still employed as a univariate goodness of fit test, other statistics, such as the 7Kolmogorov–Smirnov test and, where the theoretical distribution is normal, the Shapiro– Wilk test, are now more often used for that purpose.
∑(Y − M)
, where Y is a score, M is the sample mean, s = N− and N is the sample size, the corrected sum of squares ∑(Y − M) = (N − )s . Substituting this expression for (N − )s
. ∑(Y − µ) in the defining formula yields χ = σ To test the hypothesis that an observed sample came from a population with a particular variance, one simply divides the sample sum of squares, (N − )s , by the hypothesized variance. The resulting χ is evaluated on N − degrees of freedom, with a two-tailed p value for nondirectional hypotheses and a one-tailed p for directional hypotheses. One can also compute a confidence interval for the population variance (Winkler and Hayes , pp. –; Wuensch ). For a ( − α)% confidence interval for the population variance, compute:
(N − )s b
and
(N − )s a
where a and b are the α/ and ( − α/) fractiles of the chi square distribution on (N − )df . It should be noted that these procedures are not very robust to their assumption that the population is normally distributed. When one states that he or she has conducted a “chisquare test,” that test is most often a “one-way chi-square test” or a “two-way chi-square test” (Howell , pp. –). The one-way test is a univariate goodness of fit test. For each of k groups one has an observed frequency (O) and a theoretical frequency (E), the latter being derived from the theoretical model being tested. The test
Chi-Square Tests
(O − E)
statistic is χ = ∑ on k− degrees of freedom. The E appropriate p value is one-tailed, upper-tailed, for nondirectional hypotheses. When k = , one should make a “correction for continuity”: (∣O − E∣ − .) . E The two-way chi-square test is employed to test the null hypothesis that two categorical variables are independent of one another. The data may be represented as an r×c contingency table, where r is the number of rows (levels of one categorical variable) and c is the number of columns (levels of the other categorical variable). For each cell in this table two frequencies are obtained, the observed frequency (O) and the expected frequency (E). The expected frequencies are those which would be expected given the marginal frequencies if the row variable and the column variable were independent of each other. These expected frequencies are easily calculated from the multiplication rule of probability under the assumption of independence. For each cell, the expected frequency is (Ri Cj /N), where Ri is the marginal total for all cells in the same row, Cj is the marginal total for all cells in the same column, and N is the total sample size. The χ is computed exactly as with the one-way chi-square and is evaluated on (r − )(c − ) degrees of freedom, with an upper-tailed p value for nondirectional hypotheses. Although statistical software often provides a χ with a correction for continuity when there are only two rows and two columns, almost always the uncorrected χ is more appropriate (Camilli and Hopkins ). It is not unusual to see the two-way chi-square inappropriately employed (Howell , pp. –). Most often this is a result of having counted some observations more than once or having not counted some observations at all. Each case should be counted once and only one. Statistical software will often provide a warning if one or more of the cells has a low expected frequency. The primary consequence of low expected frequencies is low power. Even with quite small expected frequencies, actual Type I error rates do not deviate much from the nominal level of alpha (Camilli and Hopkins ). The results of a two-way chi-square test are commonly accompanied by an estimate of the magnitude of the association between the two categorical variables. When the contingency table is × , an odds ratio and/or the phi coefficient (Pearson r between the two dichotomous variables) may be useful. With larger contingency tables Cramer’s phi statistic may be useful. The chi-square statistic is also employed in many other statistical procedures, only a few of which will be mentioned here. The Cochran-Mantel-Haenszel χ is employed χ = ∑
C
to test the hypothesis that there is no relationship between rows and columns when you average across two or more levels of a third variable. The Breslow-Day χ is employed to test the hypothesis that the odds ratios do not differ across levels of a third variable. Likelihood ratio chi-square is employed in the log-linear analysis of multidimensional contingency tables, where it can be employed to test the difference between two models, where one is nested within the other. Likewise, in 7logistic regression, chi-square can be employed to test the effect of removing one or more of the predictors from the model. In discriminant function analysis, chi-square may be employed to approximate the p value associated with the obtained value of Wilks’ lambda. A chi-square statistic can be employed to test the null hypothesis that k Pearson correlation coefficients are identical. Chi-square is also used to approximate the p value in the Kruskal-Wallis ANOVA and the Friedman ANOVA. Many more uses of the chi-square statistic could be cited.
About the Author Dr. Karl L. Wuensch is a Professor in the Department or Psychology, East Carolina University, Greenville, NC, U.S.A. He has authored or coauthored scholarly articles and chapters in books. Professor Wuensch has received several teaching awards, including the Board of Governors Award for Excellence in Teaching, the most prestigious teaching award in the University of North Carolina system.
Cross References 7Chi-Square Distribution 7Chi-Square Goodness-of-Fit Tests: Drawbacks and
Improvements 7Chi-Square Test: Analysis of Contingency Tables
References and Further Reading
Camilli G, Hopkins KD () Applicability of chi-square to × contingency tables with small expected cell frequencies. Psychol Bull :– Howell DC () Statistical methods for psychology, th edn. Cengage Wadsworth, Belmont Pearson K () On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh and Dublin Phil Mag J Sci Ser :–. Retrieved from www.economics.soton.ac.uk/staff/aldrich/.pdf Winkler RL, Hays WL () Statistics: probability, inference, and decision, nd edn. Holt Rinehart & Winston, New York Wuensch KL () Common univariate and bivariate applications of the chi-square distribution. Retrieved from http://core.ecu.edu/psyc/wuenschk/docs/Chi- square.doc
C
C
Clinical Trials, History of
Clinical Trials, History of Toshimitsu Hamasaki Associate Professor Osaka University Graduate School of Medicine, Osaka, Japan
In recent years, in addition to advances in methodology, the number of clinical trials conducted and published has greatly increased. Clinical trials, in particular, blinded, randomized, controlled comparative clinical trials, are widely recognized as the most scientific and reliable method for evaluating the effectiveness of therapies and promoting a culture of evidence-based medicine (Tukey ; Byar et al. ; Zelen ; Cowan ; Byar ; Royall ; Smith ). The first modern clinical trial is generally considered to be the treatment of pulmonary tuberculosis with streptomycin conducted by the UK Medical Research Council (MRC) and published in British Medical Journal in (MRC ; Pocock ; Ederer ; Day ). However, there is still some controversy surrounding this claim as some authors refer to the study with the common cold vaccine conducted by Diehl et al. () as the first modern trial (Hart , ; Gill ). The design of the streptomycin trial included blinding, 7randomization, and control groups as fundamental elements of the clinical trial. The trial included a total of patients from seven centers, who were assigned to either “streptomycin and bed-rest” (S case) or “bed-rest” (C case) groups, by a process involving a statistical series based on random sampling numbers drawn up for each sex and each center and sealed envelopes. The efficacy of streptomycin was evaluated based upon the examination of patient X-ray films by three experts consisting of one clinician and two radiologists. The decision of whether or not the treatment was effective was made by the majority based on independently reached conclusions by each expert, who were also blinded as to which treatment the patient had received. The streptomycin trial also included Sir Austin Bradford Hill who served as the trial statistician. Hill was recognized as the world’s leading medical statistician and popularized the use of statistical methods in clinical trials, and who also attempted to improve the quality of their implementation and evaluation by publishing a series of articles in The Lancet in (Hill et al. ). With the success of the streptomycin trial, the MRC and Hill continued with further blinded, randomized, controlled comparative clinical trials (Ederer ; Days
): for example, chemotherapy of pulmonary tuberculosis in young adults (MRC ), an antihistaminic drug in the prevention and treatment of the common cold (MRC ), the use of cortisone and aspirin in the treatment of early cases of rheumatoid arthritis (MRC , ), and an anticoagulant to treat cerebrovascular disease (Hill et al. ). In United States, the first randomized controlled trail started in and was the US National Institute of Health study of the adrenocorticotropic hormone, cortisone and aspirin in the treatment of rheumatic heart disease in children (Rheumatics Fever Working Party ). Presently, a huge number of randomized controlled clinical trials are being conducted worldwide, with the number of clinical trials steadily increasing each year. Although now commonplace, the fundamental elements of clinical trials, such as blinding, randomization, and control groups, did not just suddenly appear in the second quarter of the twentieth century. Evidence exists that a comparative concept for evaluating therapeutic efficacy with control groups has been known since ancient times (Ederer ; Day ). For example, Lilienfeld () and Slotki () cited the description of a nutritional experiment using a control group in the Book of Daniel from the Old Testament: 7 .: Among these were some from Judah: Daniel, Hananiah, Mishael and Azariah. . . .: But Daniel resolved not to defile himself with the royal food and wine, and he asked the chief official for permission not to defile himself this way. . . . .: Daniel then said to the guard whom the chief official had appointed over Daniel, Hananiah, Mishael and Azariah. .: Please test your servants for ten days; Give us nothing but vegetables to eat and water to drink. .: Then compare our appearance with that of the young men who eat the royal food, and treat your servants in accordance with what you see. .: So he agreed to this and tested them for ten days. .: At the end of the ten days they looked healthier and better nourished than any of the young men who ate the royal food. . So the guard took away their choice food and the wine they were to drink and gave them vegetables instead.
The above description is part of a story dating from approximately BC when Daniel was taken captive by the ruler of Babylonia, Nebuchadnezzar. In order to refrain from eating royal meals containing meat (perhaps pork) and wine offered by Nebuchadnezzar, Daniel proposed a comparative evaluation and was rewarded when his test group fared better than the royal food group. Although it is difficult to confirm the accuracy of the account, it is clear that the comparative concept already existed when the Book of Daniel was written around BC. In particular, it is
Clinical Trials, History of
remarkable that the passage from the Book of Daniel mentioned not only the choice of a control group but the use of a concurrent control group. Unfortunately, this fundamental concept was not widely practiced until the latter half of the twentieth century (Ederer ; Day ). Much later than the Book of Daniel, in the eighteenth and nineteenth centuries, there were some epoch-making clinical researches that formed the basis of the methodology used in current clinical trials. Before the modern clinical trial of the treatment of pulmonary tuberculosis with streptomycin mentioned above (Pocock ; Ederer ; Day ), the most famous historical example of a planned, controlled clinical trial involved six dietary treatments for scurvy on board a British ship. The trial was conducted by the ship’s surgeon, James Lind, who was appalled by the ravages of scurvy which had claimed the lives of three quarters of the crew during the circumnavigation of the world by British admiral, George Anson (Lind ; Bull ; Pocock ; Mosteller ; Ederer ; Day ). In , Lind conducted a comparative trial to establish the most promising “cure” for patients with scurvy using twelve individuals who had very similar symptoms on board the Salisbury. In addition to one common dietary supplement given to all of the patients, he assigned each of six pairs one of the following six dietary supplements: . . . . . .
Six spoonfuls of vinegar A half-pint of sea water A quart of cider Seventy-five drops of vitriol elixir Two oranges and one lemon Nutmeg
Those patients who received the two oranges and one lemon were cured within approximately days and were able to help nurse the other patients. Apart from the patients who improved somewhat after receiving the cider, Lind observed that the other remedies were ineffective. The reason for the success of Lind’s trial was likely due to his knowledge of previous work by James Lancaster (Purchas ), who had served three teaspoons of lemon juice each day to sailors suffering from scurvy during the first expedition to India sent by the East India Company in (Mosteller ). Unfortunately, however, the British Navy did not supply lemon juice to its sailors until , although conclusive results concerning the efficacy of such treatment had already been obtained much earlier (Bull ; Mosteller ). The use of statistical concepts in clinical trials was also advocated earlier than the streptomycin trials. For
C
example, Pierre Simon Laplace, a French mathematician and astronomer, mentioned the use of probability theory to determine the best treatment for the cure of a disease (Laplace ; Hill et al. ). Also, PierreCharles-Alexandre Louis, a French physician and pathologist, discussed the use of a “numerical method” for the assessment of treatments by constructing comparable groups of patients with similar degrees of a disease, i.e., to compare “like with like” (Louis ; Ederer ; Day ). Unfortunately, these suggestions were not earnestly acted upon until the streptomycin trial because in the eighteenth and nineteenth centuries, the investigators were more involved with the practice of medicine and less versed in the use of probability theory since saving patients’ life was considered more important rather than collecting data from the aspect of ethics (Bull ; Hill et al. ). Here, the history and development of clinical trials was very briefly traced. More detailed aspects of the history of clinical trials can be found in articles by Bull (), Armitage (, ), Lilienfeld (), Pocock (), Meinert (), Gail (), Ederer () and Day ().
About the Author Toshimitsu Hamasaki is Associate Professor of Department of Biomedical Statistics, Osaka University Graduate School of Medicine. He is the Elected member of International Statistical Institute. He has over peerreviewed publications roughly evenly split between applications and methods. He was awarded the Best Paper Prize (the Japanese Society of Computational Statistics, ) and Hida-Mizuno Prize (the Behaviormetric Society of Japan, ). He is the Editor-in-Chief of the Journal of the Japanese Society of Computational Statistics (–).
Cross References 7Biostatistics 7Clinical Trials: An Overview 7Clinical Trials: Some Aspects of Public Interest 7Design of Experiments: A Pattern of Progress 7Medical Research, Statistics in 7Medical Statistics
References and Further Reading Armitage P () History of randomised controlled trials. Lancet : Armitage P () Interim analysis in clinical trials. Stat Med : – Bayar DP, Simon RM, Friedewald WT, Schlesselman JJ, DeMets DL, Ellen-berg JH et al () Randomized clinical trials: perspective on some recent idea. New Engl J Med :– Bull JP () The historical development of the clinical trials. J Chron Dis :–
C
C
Clinical Trials: An Overview
Byar DP () Comment on “Ethics and statistics in randomized clinical trials” by R.M. Royall. Stat Sci :– Cowan DH () The ethics of trials of ineffective therapy. IRB: Rev Human Subjects Res :– Day S () The development of clinical trials. In: Machin D, Day S, Green S (eds) Textbook of clinical trials, nd edn. Wiley, Chichester, pp – Diehl HS, Baker AB, Cowan DW () Cold vaccines: an evaluation based on a controlled study. J Am Med Assoc : – Ederer F () Clinical trials, history of. In: Armitage P, Colton T (eds) Encyclopedia of biostatistics, nd edn. Wiley, Chichester, pp – Gail MH () Statistics in action. J Am Stat Assoc :– Gill DBEC () Early controlled trials. Brit Med J : Hart PD () History of randomised controlled trials. Lancet : Hart PD () Early controlled clinical trials. Brit Med J : – Hill AB, Marshall J, Shaw DA () A controlled clinical trial of long-term anticoagulant therapy in cerebrovascular disease. Q J Med :– Hill G, Forbes W, Kozak J, MacNeil I () Likelihood and clinical trials. J Clin Epidemiol :– Laplace PS () Théori analytique des probabilités. Courcier, Paris Lilienfeld AM () Ceteris paribus: the evaluation of the clinical trial. Bull Hist Med :– Lind J () A treatise of the scurvy. Sands Murray & Cochran, Edinburgh Louis PCA () The applicability of statistics to the practice of medicine. London Medical Gazette :– Medical Research Council () Streptomycin treatment of pulmonary tuberculosis. Brit Med J :– Medical Research Council () Clinical trials of antihistaminic drugs in the prevention and treatment of the common cold. Brit Med J :– Medical Research Council () Chemotherapy of pulmonary tuberculosis in young adults. Brit Med J :– Medical Research Council () A comparison of cortisone and aspirin in the treatment of early cases of rheumatoid arthritis I. Brit Med J :– Medical Research Council () A comparison of cortisone and aspirin in the treatment of early cases of rheumatoid arthritis II. Brit Med J :– Meinert CL () Clinical trials: design, conduct and analysis. Oxford University, New York Mosteller F () Innovation and evaluation. Science : – Pocock SJ () Clinical trials: a practical approach. Wiley, Chichester Purchas S () Hakluytus posthumus or purchas his pilgrimes: contayning a histrory of the world in sea voyages and lande travells by englishmen and others. James MacLehose & Sons, Glasgow (reprinted, ) Rheumatics Fever Working Party () The evaluation of rheumatic heart disease in children: five years report f a co-operative clinical trials of ACTH, cortisone, and aspirin. Circulation :– Royall RM () Ethics and statics in randomized clinical trials. Stat Sci :–
Slotki JJ () Daniel, Ezra, Nehemiah, Hebrew: text and english translation with introductions and commentary. Soncino Press, London Smith R () Fifty years of randomized controlled trials. Brit Med J : Tukey JW () Some thoughts on clinical trials, especially problems of multiplicity. Science :– Zelen M () A new design for randomized clinical trials. N Engl J Med :–
Clinical Trials: An Overview Hiroyuki Uesaka Osaka University, Osaka, Japan
A clinical trial is one type of clinical research where a procedure or drug is intentionally administered outside the realm of standard medical practice to human subjects with the aim of studying its effect on the human body. This includes medications, operations, psychotherapy, physiotherapy, rehabilitation, nursing, restricted diets, and the use of medical devices. The comparative study of two or more treatments, involving the random assignment of treatments to patients, is considered a clinical trial even if the study includes approved drugs or medical devices. This means that a clinical trial is an experiment which includes human subjects. It is necessary to distinguish clinical trials from observational studies which collect outcomes when executing a study treatment as an ordinary treatment. Since clinical trials include human subjects, the ethical aspects, i.e., the rights, safety and well-being of individual research subjects, should take precedence over all other interests at all stages, from the planning of clinical trials to the reporting of results. Such ethical principles for clinical research are in accordance with the Declaration of Helsinki, “Ethical Principles for Medical Research Involving Human Subjects,” issued by the World Medical Association (). In conducting a clinical trial, the study protocol should clearly describe the plan and content of the trial. The protocol must also be reviewed and approved by the ethics committee. Furthermore, the Declaration of Helsinki states: The protocol should contain a statement of the ethical considerations involved and indicate how the principles in the above declaration have been addressed. To protect the safety, well-being and rights of the human subjects participating in the trial, the Declaration indicates that potential subjects must be adequately informed of all
Clinical Trials: An Overview
relevant aspects of the study which include aims, methods, the anticipated benefits and potential risks of the study and any discomfort that participation may entail. And it states: The potential subjects must be informed of their right to refuse to participate in the study or to withdraw consent to participate at any time without reprisal. The voluntary agreement of a subject to participate after sufficient details have been provided is called informed consent. It is also recommended that the clinical trial be registered in a publicly accessible database, and the results from the trial should be made publicly available, regardless of whether the results are positive or negative. Clinical trials for a new drug application, when they are conducted in the EU, Japan and/or the United States of America, must meet the requirements of the “Good Clinical Practice” (GCP) guideline (ICH Steering Committee ). The GCP guideline is a unified standard provided by the Europe, Japan and the United States in the framework of the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use [http://www.ich.org/]. The GCP guideline provides protection for the safety, well-being and human rights of subjects in clinical trials in accordance with the Declaration of Helsinki. The GCP guideline also requires that people appointed by the sponsor, the socalled monitors, verify that the rights and well-being of all human subjects are being protected, that the reported trial data are accurate, complete, and verifiable from source documents, and that the conduct of the trial is in compliance with the currently approved protocol/amendment(s), with the GCP, and with the applicable regulatory requirement(s). This is referred to as trial monitoring in the GCP. In planning a clinical trial, a protocol must be prepared, including descriptions of the trial justification, trial objectives, study treatments, the population to be studied as defined by the study inclusion and exclusion criteria, test treatments and treatment procedures, observed variables and observation procedures, specification of variables to assess treatment effect, collection of safety information, prohibited concomitant medications, discontinuation criteria for individual subjects, the number of subjects planned to be enrolled and the justification for such, statistical methods to be employed, data collection, quality control and quality assurance (ICH Steering Committee ). A case report form (CRF) should be prepared as well. The CRF is a document designed to record all of the required information to be reported according to the protocol. After a trial, a so-called clinical study report is prepared (ICH Steering Committee ). This is a document which contains clinical and statistical descriptions
C
of the methods, rationale, results and analyzes of a specific clinical trial fully integrated into a single report. Clinical trials are conducted as collaborative activities involving many specialists, such as investigators, nurses, diagnostic testing specialists and other collaborators. Furthermore, regulators are involved in new drug applications. Therefore, the protocol, CRF and clinical study report should be clearly and accurately documented to be easily understood by those involved in the trial and by those who will make use of the trial results. Clinical trials can be classified into several types depending on various features (ICH Steering Committee ; ICH Steering Committee ). First, a trail can be controlled or uncontrolled, this being determined by the presence of a control group. A controlled trial is a trial to compare the study treatment(s) with a control treatment that is either the current standard treatment, best supportive care, placebo, or some other treatment; an uncontrolled trial involves giving the same treatment to all of the subjects participating in the trial. The second feature involves the objective of a trial, either exploratory or confirmatory. A clinical trial that aims to generate or identify a research topic, or provide information to determine the specifics of a trial method is called an exploratory trial. A confirmatory trial is defined as an adequately controlled trial where hypotheses which were derived from earlier research or theoretical considerations are stated in advance and evaluated. Furthermore, a confirmatory trial generally includes three types of comparisons: a superiority trial, a non-inferiority trial, and an equivalence trial. A superiority trial is used to show the superiority of a test treatment over a control. A non-inferiority trial is designed to show that the efficacy or safety of the study treatment is no worse than that of the control. An equivalence trial serves to demonstrate that the test treatment is neither better nor worse than the control. The third aspect involves distinguishing between a pragmatic and an explanatory trial (Gent and Sackett ; Schwartz and Lellouch ). The objective of a pragmatic trial is to confirm effectiveness of the test treatment for those subjects who are assigned to the test treatment. An explanatory trial serves to establish a biological action for the treatment. Finally, the fourth characteristic focuses on the difference between a single- and a multi-center trial. The single-center trial is conducted by a single investigator, and the multi-center trial is coconducted by multiple investigators at multiple study sites. Recently, many multi-center trials have been planned and conducted across not only a single country but also two or more countries. Such a multi-center trial is called a multinational trial.
C
C
Clinical Trials: An Overview
The clinical development of a new drug advances in stages (ICH Steering Committee ). A safety trial is executed first to determine the maximum dose that can be safely administered to a subject. In most safety trials of the first use of a new drug in humans, the subjects are healthy volunteers. Administration of the study treatment begins from a dosage expected to be safe enough for normal healthy volunteers, and then the dosage is increased in stages. The pharmacokinetic profile is usually examined in the same trial. Pharmacokinetics investigates the process of drug disposition which usually consists of absorption, distribution, metabolism and excretion. This stage is called Phase I. The next stage is to determine the dosage range that can be safely administered to patients and at which sufficient effectiveness can be expected. The dosage that will be used in clinical treatment as well as the dose intervals are also clarified at this stage. This is called the Phase II. In the third stage, the efficacy and safety of the study treatment is confirmed in the target patient population. This stage is referred to as Phase III. The dose and dosage regimen which are confirmed to be efficacious and safe in phase III are then submitted to the regulatory authority to obtain marketing authorization of the new drug. After marketing authorization is obtained, the drug becomes widely used for clinical treatment. This stage is called Phase IV. During the phase III trials many restrictions are imposed to ensure the safety of the participating subjects. These include the necessity of physical examinations, collection of patient anamneses, regulation of concomitant medications, and clearly defined test treatment administration periods. However, in phase IV such restrictions are relaxed and the approved study treatment can be used by various patients under diverse conditions. Therefore, because the number of patients who are administered the newly approved drug increases rapidly, with patients often using the drug for very long times according to their disease condition, there is a real concern about harmful effects that have not been anticipated. Therefore, an investigation to clarify the safety and effectiveness of the treatment in daily life, an observational study, a large-scale trial, or a long-term trial, is conducted. Moreover, a clinical trial to compare the newly approved drug with other medicines that have been approved for the same indication may also be conducted. The result of the trial should be scientifically valid. Clinical trial results are intended to be applied to a target population defined by inclusion and exclusion criteria for a given trial. The enrolled subjects should be a random sample from the target population so that the trial results can be applied to the target population. However a trial is conducted in a limited number of medical sites, and not all candidate subjects give informed consent. Therefore,
whether or not the trial result can be generalized to the target population will depend on the study protocol and the actual execution procedure. Accordingly, it is preferable to execute the trial in a variety of medical institutions with a wide range of patients corresponding to the diversity of the target population to improve the possibility of generalizing to the target population. A controlled trial usually estimates the difference in response to treatments between treatment groups. As described above, the clinical trial participants are not a random sample of the target population. Therefore the true mean difference in the study population (all subjects who participate in the trial) will be estimated. This is accomplished by dividing the study population into two or more treatment groups which are assigned to different treatments, and then comparing the means of response to treatment between groups. The estimated mean difference is usually different from this true value. When random allocation of treatment to subjects is used, it is assumed that the departure from the true difference is probabilistic or random error. However, there is the possibility of systematic error due to the execution procedure of the trial. This systematic error is called bias (ICH Steering Committee ). The execution of treatment, evaluation of results, and/or subjects’ reactions can be influenced if the people involved in a trial, such as investigators, relevant clinical staff or subjects, are aware of which treatment is assigned to subjects. Therefore, masking (blinding) and randomization are used to prevent participants from knowing which treatment is being allocated to which subjects. There are several levels of blinding: double-blind, single-blind, observer-blind and open-label. In a double-blind study neither the subjects, nor the investigator, nor any of the relevant clinical trial staff know who belongs to which treatment group. In a single-blind study only treatment assignments are unknown to the subjects or investigator and relevant clinical staff. In an observer-blind study treatment assignments are unknown to the observers who assess the subjects’ conditions. In an open-label study treatment assignments are known to both investigators and subjects. One of the typical methods of treatment assignment is to assign only one treatment to each subject, and then to compare the effects of the treatments between subject groups. This method is referred to as parallel group design. The other typical method is the cross-over design where one subject receives two or more treatments and an intrasubject comparison of treatments is done. It is necessary to select an appropriate design because bias can be caused by the design itself. A clinical trial is an experiment with human beings as subjects. It is preferable that the number of subjects be as small as possible to protect the rights, health and welfare
Clinical Trials: Some Aspects of Public Interest
of the subjects included in the trial. However, if the objective of the trial is not achieved, the reason for executing the trial is lost. Therefore, based on the estimated difference between the treatments, the trial should be designed to have sufficient precision to either detect such a difference if it truly exists, or to conclude that the difference is below a definite value based on concrete evidence. For this purpose, it is necessary to maintain high accuracy and precision in trials. To ensure the precision of a trial, it is important to consider the stratification of the study population, to make precise observations, and to secure a sufficient number of subjects. The objective of these trials is to estimate beneficial and adverse effects, and to confirm a hypothesis about the effect of the study treatment. Even if the effect size of the test treatment is assumed to be of a given size, the true effect size may be less than assumed. When the gap between the actual and the assumed value is large, the planned number of subjects might be insufficient and, in some cases, many more subjects than originally planned will be needed. In such cases, a sequential design (Jennison and Turnbull ) and a more advanced adaptive design (Bretz et al. ) would be proposed.
About the Author Dr. Hiroyuki Uesaka is a specially appointed Professor of The Center for Advanced Medical Engineering and Informatics, Osaka University, Suita, Japan. He has been working for pharmaceutical companies for about years as a statistical expert of clinical trials. He authored more than original statistical papers, and authored, coauthored and contributed to books written in Japanese (including Design and Analysis of Clinical Trials for Clinical Drug Development (Asakura-Shoten, ) and Handbook of Clinical Trials (jointly edited with Dr. Toshiro Tango, Asakura-Shoten, ).
Cross References 7Biopharmaceutical Research, Statistics in 7Biostatistics 7Causation and Causal Inference 7Clinical Trials, History of 7Clinical Trials: Some Aspects of Public Interest 7Design of Experiments: A Pattern of Progress 7Equivalence Testing 7Hazard Regression Models 7Medical Research, Statistics in 7Medical Statistics 7Randomization 7Statistics Targeted Clinical Trials Stratified and Personal-
ized Medicines
C
7Statistics: Controversies in Practice 7Statistics: Nelder’s view
References and Further Reading Bretz F, Koenig F, Brannath W, Glimm E, Posch M () Adaptive designs for confirmatory clinical trials. Stat Med :– Gent M, Sackett DL () The qualification and disqualification of patients and events in long-term cardiovascular clinical trials. Thromb Hemosta :– ICH Steering Committee () ICH harmonised tripartite guideline structure and content of clinical study reports. Recommended for adoption at Step of the ICH Process on November ICH Steering Committee () ICH Harmonized tripartite guideline. Guideline for good clinical practice. Recommended for adoption at step of the ICH process on May ICH Steering Committee () ICH Harmonized tripartite guideline. General considerations for clinical trials. Recommended for adoption at step of the ICH process on July ICH Steering Committee () ICH Harmonized tripartite guideline. Statistical principles for clinical trials. Recommended for adoption at step of the ICH process on February ICH Steering Committee () ICH Harmonized tripartite guideline. Recommended for adoption at step of the ICH process on July Jennison C, Turnbull BW () Group sequential methods with applications to clinical trials. Chapman & Hall/CRC, Boca Raton Schwartz D, Lellouch J () Explanatory and pragmatic attitudes in therapeutical trials. J Chron Dis : – World Medical Association () Declaration of Helsinki. Ethical principles for medical research involving human subjects . Amended by the th WMA General Assembly, Seoul, Korea , http://www.wma.net/
Clinical Trials: Some Aspects of Public Interest Jagdish N. Srivastava CNS Research Professor Emeritus Colorado State University, Fort Collins, CO, USA Medical experiments, often called “clinical trials,” are obviously extremely important for the human race. Here, we shall briefly talk, in layman’s language, about some important aspects of the same which are of great public interest.
Side Effect of Drugs The side effects of allopathic drugs are notorious; death is often included in the same. For degenerative diseases (as opposed to infectious diseases, as in epidemics) it is not clear to the author whether any serious effort is being made by the pharmaceutical companies to develop drugs
C
C
Clinical Trials: Some Aspects of Public Interest
which actually cure diseases; the trend seems to be at best to maintain people on drugs for a long time (even for the whole life). Mostly, people live under varying forms of painkiller-surgery regimes. However, in many institutions (for example, departments doing research on nutrition) there are people who are genuinely interested in finding cures, though often they do not possess the resources they need. Many things, considered true by the public, are not quite so. Consider preservatives and other food additives that are legal. They are found in varying quantities in most foods, and many people do not pay attention to this at all, and consume unknown amounts each day. The thought that they have no side effects is based on relatively (time-wise) small experiments and extrapolations there from. It is probably true that if some food with a particular preservative or a (combination of the same with others) is consumed, it may not have any noticeable effect within a short period. But, the worry that many thinkers have is whether consuming food (all the time ignoring preservatives that it may contain) will have a disastrous effect (like producing cancer, heart attack, diabetes, stroke, etc.) , , , or years earlier than it would have been expected for an additives-free diet. (The fact that, now, teenagers and young ones in their twenties are developing such diseases which, in an earlier age, were found mainly among seniors only, is alarming.) A full scale clinical trial (to study the long term effect of preservatives etc.) will take more than a century, and has not been done. Thus, extrapolations proclaiming that such additives are safe are based on guess work only, and are not necessarily scientifically sound. We live in an age when shelf life has become more important than human life. It is not even clear whether the damage done from side effects and the painkiller-surgery policies is limited to the increase in the periods of sickness of people, the intensities of such sickness, and the reduction in the age at death. The bigger question is whether there is an effect on the progeny, and for how many generations. We recall that in the processes of natural selection in the theory of evolution, only the fittest may survive. Clearly, for the human race, only the policy that promotes the good of the general public corresponds to being fit for survival.
Contradictory Statements by Opposing Camps of Medical Researchers Often, seemingly good scientists are found to be contradicting each other. For example, there may be a substance (say, an extract from some herbs) which may be claimed by some nature-cure scientists (based on their experiments) to positively affect some disease (relative to a placebo). However, some pharmaceutical scientists may claim that their experiments show that the drug is no better than
the placebo. This is to say that, often in such cases, a close look may reveal that the two sets of experiments are not referring to the same situation. To illustrate, the subjects (people, on whom an experiment is done) in the first group may be people who just contracted the disease, these people being randomly assigned the drug or the placebo. In the second case, the subjects may be people who have had the disease for some time and have been taking painkillers. Now, the herbal drug may be quite effective on a body which is in a more primeval and natural state, and yet not work well in a body which has been corrupted by the chemicals in the painkiller. Clearly, that would explain the discrepancy and support the use of the herbal drug soon after the disease begins, simultaneously discouraging the use of painkillers etc. whose primary effect is to temporarily fool the mind into thinking that one is feeling better. Thus, it is necessary to examine a clinical trial closely rather than take its results on face value.
Large Clinical Trials: Meta-analysis “Large” clinical trials are often touted as being very “informative.” To illustrate, take the simple case of comparing two drugs A and B with respect to a placebo C. Now, how effective a drug is for a person may depend upon his or her constitution. On some people, A may be the best, on some B, and on others, all the three may be essentially useless. For me, even though I may not know the reality, suppose the reality is that B would be very effective with little negative side effect, A would be only somewhat effective but with a large negative side effect, and the effect of C would be small (being somewhat positive or somewhat negative depending on environmental factors). Suppose a trial is done in Arizona, involving , patients randomly divided into three equal groups, the result being that A is effective in % (cases in its group), B in %, and C in % cases. Clearly, here, the drug A wins. But, for me, what is the value of this information? I really need to know which drug would be best for me. Now suppose a similar trial is done in Idaho and in California, the result for A, B, and C being %, %, %, and %, %, and % respectively in the two states. Does this help me in some way or does it simply add to the confusion? The drugs manufacturer, Mr. Gaines, would like “meta–analysis” (whose purpose is to combine the results in a legitimate and meaningful way), because his interest is in seeing the overall picture so that he can formulate an appropriate manufacturing policy for his company. However, the interest of the general public is different from that of Gaines, because each individual needs to know what is good for him or her personally. The individual’s interest, in a sense, runs counter to 7meta–analysis; he or she would be more interested in knowing what aspects of a person’s
Clinical Trials: Some Aspects of Public Interest
health make him or her more receptive to A or B. Instead of combining the data, more delineation needs to be done. In other words, one needs to connect the results with various features of the subjects and other related factors. Then we may gain knowledge not only on what proportion of subjects are positively affected by a drug, but what bodily features of people (or the food that they eat, or their lifestyle, or the environment around them, etc.) lead to this positive effect. For example, in the above (artificial) data, it seems that A is better in a warm climate and B in cold. Where the climate is mild, all of them do well, and many people may recover without much of drugs. If I have been more used to a cold climate, B may be more effective on me. With this knowledge, even though A may turn out to be much better than B in the area where I live, B may be better for me individually. (This leads us to the philosophy of statistical inference. Not only do we need to plan our experiments or investigations properly, we need to be careful in drawing inferences from the data obtained from them. According to the author, trying to find what a bunch of data “says” must involve in a relevant way the space of applications where such a finding will be made use of. Many scholars believe that, given a set of data, the “information” that the data contains is a fixed attribute of the data, and the purpose of inference is to bring out this attribute accurately. The author believes that the reason why the inference is sought (in particular, to what use or application the inference will be put) is also important, and should have a bearing on the inference drawn. This policy will give insight into the kind of information we need, what should receive more emphasis, etc. Clinical trials would really gain from this approach.)
Reducing Side Effects of Drugs Studies are usually done using a “loss function” which tells how much “loss” shall we incur by adopting each of a set of policies. For example, we may have many drugs, several possible doses of a drug per day, many possible durations of time over which a drug is to be continued, etc. For each combination of these factors, the “loss” may be “the total time of absence from work,” or “the total financial loss incurred because of sickness,” or “the amount of fever,” or “the blood pressure,” etc. If the loss function involves only one variable (like “blood pressure”), it is “uni-dimensional.” But if, many variables are involved simultaneously (like “blood pressure,” “fever,” “financial loss”), then it is called multi-dimensional. Usually, only one dimension is used or emphasized (like, “intensity of fever”). More theory needs to be developed on how to work with multi-dimensional loss functions.
C
Besides theory, we also need to develop good quantitative criteria for measuring “healthfulness.” There can be various sectors. For example, we can have one criterion for the sector of upper digestive track, one for the middle, one for the colon, one for the respiratory system, one for bone diseases, one for the joints, one for nerves, one for cancerous growth, and so on. For each sector, the corresponding criterion will provide a measure of how healthy that sector is. Suppose we decided to have such sectors. Then the loss function will be -dimensional. The drugs will be evaluated in each dimension, and the results will also be combined in various ways. The side effect of a drug in a particular sector will be caught more easily. When the drug is marketed, an assessment for each sector can be provided. Drugs with large effect in any sector can be rejected.
Experiments with Many Factors: Interactions We make some technical remarks here. A large part of the field of statistical design of multi-factorial scientific experiments is concerned with the simplistic situation when there are either no interactions or the set of non-negligible interactions is essentially known (though the values of these interactions and the main effects are not known). However, in medical experiments, we can have interactions of even very high orders. Thus, for the field of multifactor clinical trials, we have to go beyond Plackett–Burman designs, and orthogonal arrays of small strength (such as ). There is work available on search theory by the author and others, which would help. However, further work is needed in that field. Indeed, for vigorous full fledged research on how to cure diseases, the statistical theory of the design and analysis of multifactor multi-response experiments need to be developed much further beyond its current levels. However, the basics of the same are available in books such as Roy et al. (). For the reader who wishes to go deeper in the field of this article, some further references are provided below.
About the Author Jagdish N. Srivastava is now CNS Research Professor Emeritus at Colorado State University, where he worked during – as Full Professor, and where he also once held a joint appointment in Philosophy. He was born in Lucknow, India on June . He did his Ph.D. in (at the University of North Carolina, Chapel Hill, NC, USA). He was President (Indian Society of Agricultural Statistics, Session; International Indian Statistical Association (IISA) (–); Forum for Interdisciplinary Mathematics (–). He visited institutions all over the world. His honors include the Fellowship of the American Statistical Association, Institute of Mathematical Statistics,
C
C
Cluster Analysis: An Introduction
Institute of Combinatorial Mathematics and its Applications, IISA (Honorary), International Statistical Institute, and TWAS (The World Academy of Science for developing countries). He is the founder () and Editor-in-Chief of the monthly Journal of Statistical Planning and Inference. Two volumes containing papers by experts from all over the world were published in honor of his th birthday. He discovered, among other results, the “Srivastava Codes,” the “Srivastava Estimators,” and the Bose-Srivastava Algebras. He has been interested in, and has contributed to, Science and Spirituality, and is a major researcher in Consciousness and the “Foundations of Reality” (including, in particular, “Quantum Reality”).
Cross References 7Clinical Trials, History of 7Clinical Trials: An Overview 7Medical Research, Statistics in 7Medical Statistics 7Statistics Targeted Clinical Trials Stratified and Personal-
ized Medicines
References and Further Reading Roy SN, Gnanadesikan R, Srivastava JN () Analysis and design of quantitative multiresponse experiments. Pergamon, Oxford, England Srivastava JN () Modern factorial design theory for experimenters. In: Ghosh S (ed) Statistical design and analysis of industrial experiments. Marcel Dekker, New York, pp – Srivastava JN () A critique of some aspects of experimental designs. In: Ghosh S, Rao CR (eds) Handbook of statistics, vol . Elsevier, Amsterdam, pp – Srivastava JN () Foods, drugs and clinical trials: some outrageous issues. J Combinatorics Inf Syst Sci (–):– Srivastava JN () Some statistical issues concerning allopathic drugs for degenerative diseases. J Indian Soc Agric Stat : –
Cluster Analysis: An Introduction H. Charles Romesburg Professor Utah State University, Logan, UT, USA Cluster analysis is the generic name for a variety of mathematical methods for appraising similarities among a set of objects, where each object is described by measurements made on its attributes. The input to a cluster analysis is a data matrix having t columns, one for each object, and n rows, one for each attribute. The (i, j)th element of the data matrix is the measurement of the ith attribute for the jth
object. The output from a cluster analysis identifies groups of similar objects called clusters. A cluster may contain as few as one object, because an object is similar to itself. Applications of cluster analysis are widespread because the need to assess similarities and dissimilarities among objects is basic to fields as diverse as agriculture, geology, market research, medicine, sociology, and zoology. For example, a hydrologist considers as the objects a set of streams, and for attributes describes each stream with a list of water quality measures. A cluster analysis of the data matrix identifies clusters of streams. The streams within a given cluster are similar, and any stream in one cluster is dissimilar to any stream in another cluster. There are two types of cluster analysis. Hierarchical cluster analysis is the name of the collection of methods that produce a hierarchy of clusters in the form of a tree. The other type, nonhierarchical cluster analysis, is the name of the collection of methods that produce the number of clusters that the user specifies. For both types, computer software packages containing programs for the methods are available. Let us illustrate the main features of hierarchical cluster analysis with an example where the calculations can be done by hand because the data matrix is small, five objects and two attributes, consisting of made-up data:
Data matrix Object Attribute
To perform a hierarchical cluster analysis, we must specify: () a coefficient for assessing the similarity between any two objects, j and k; and () a clustering method for forming clusters. For the first, let us choose the “Euclidean distance coefficient,” ejk . The smaller its value is, the more similar objects j and k are. If the value is zero, they are identical, i.e., maximally similar. For our example with n = attributes, ejk is the distance between object j and object k computed with the Pythagorean theorem. And for the clustering method, let us choose the “UPGMA method,” standing for “unweighted pair-group method using arithmetic averages.” At the start of the cluster analysis, each object is considered to be a separate cluster. Thus with five objects, there are five clusters. For the five we compute the (−)/ =
Cluster Analysis: An Introduction
e = [( − ) + ( − ) ]
28.0
= ..
In this manner, we compute the other values and put the ten in a list, from smallest, indicating the most similar pair of clusters (objects), to largest, indicating the least similar pair: e = ., e = ., e = ., e = ., e = ., e = ., e = ., e = ., e = ., e = .. The two most similar clusters, and , head the list, as the Euclidean distance between them is the smallest. Therefore, Step Merge clusters and , giving , , (), and at the value of e = .. Next, for the four clusters – , , (), and – we obtain the ( − )/ = values of ejk . Three of these values are unchanged by the clustering at step and can be transcribed from the above list. The other three have to be computed according to the guiding principle of the UPGMA clustering method. It requires that we average the values of ejk between clusters, like this:
C
24.0 Euclidean distance, ejk
/
Step 4 20.0
16.0 Step 3
12.0
8.0 Step 2 Step 1
4.0
0.0
3
4
2 Object
1
5
Cluster Analysis: An Introduction. Fig. Tree showing the hierarchy of similarities between the five objects specified by the data matrix in the text
e() = / (e + e ) = / (. + .) = .; e() = / (e + e ) = / (. + .) = .;
e() = / (e + e ) = / (. + .) = .. So, the six ejk values listed in order of increasing distance are: e = ., e() = ., e = ., e = ., e() = ., e() = .. It follows that the two most similar clusters are and , since the Euclidean distance between them is the smallest. Therefore, Step Merge clusters and , giving , (), and () at the value of e = .. Before going to the next clustering step, we note that step left the distance between clusters and () unchanged at e() = .. The two remaining distances are calculated according to the UPGMA clustering method by averaging the values of ejk as follows: e() = / (e + e ) = / (. + .) = .; e()() = / (e + e + e + e ) = / (. + . + . + .) = .. The list of ejk in increasing distance is now: e() = ., e() = ., e()() = .. The two most similar clusters, and () head the list. Therefore, Step Merge clusters and (), giving () and () at the value of e() = ..
Tree
values of ejk . To demonstrate the calculation of one of these values, consider object and object . The Euclidean distance e is
C
At this point there are two clusters: () and (). The average Euclidean distance between them is: e()() = / (e + e + e + e + e + e ) = / (. + . + . + . + . + .) = .. The list of ejk has only one value: e()() = .. Therefore, Step Merge clusters () and (), giving () at the value of e()() = .. The calculations are finished. With each step, the tree (Fig. ) has been growing. It summarizes the clustering steps, e.g., showing that the branches containing cluster () and cluster join at an Euclidean distance value of .. The tree is a hierarchical ordering of similarities that begins at the tree’s bottom where each object is separate, its own cluster. As we move to higher levels of ejk , we become more tolerant and allow clusters to hold more than one object. When we reach the tree’s top we are completely tolerant of the differences between objects, and all objects are considered as one cluster. Suppose we took the five objects in the data matrix and plotted them on a graph with attribute as one axis and
C
Cluster Analysis: An Introduction
attribute as the other. We would see that the distances between the objects suggest clusters that nearly match those in the tree. However, real applications typically have many attributes, often more than a hundred. In such cases, the researcher cannot grasp the inter-object similarities by plotting the objects in the high-dimension attribute space. Yet cluster analysis will produce a tree that approximates the inter-object similarities. A tree is an old and intuitive way of showing a hierarchy. Witness the tree of life forms, the Linnaean classification system. At its bottom is the level of Species, at the next higher hierarchical level is the Genus, consecutively followed by levels of Order, Class, Phylum, and Kingdom. A widely practiced way of creating a classification of objects is to perform a hierarchical cluster analysis of the objects. On the tree, draw a line perpendicular across the tree’s axis, cutting it into branches, i.e., clusters. The objects in the clusters define the classes. Details may be found in Romesburg () and in Sneath and Sokal (). There are several general points to note about hierarchical cluster analysis: . There are various coefficients that can be used to assess the similarity between clusters. Of these, there are two types: dissimilarity coefficients and similarity coefficients. With a dissimilarity coefficient (as the Euclidean distance coefficient is), the smaller its value is, the more similar the two clusters are. Whereas with a similarity coefficient, the larger its value is, the more similar the two clusters are. An example of a similarity coefficient is the Pearson product moment correlation coefficient. But whether a dissimilarity coefficient or a similarity coefficient is used, a clustering method at each step merges the two clusters that are most similar. . Although the UPGMA clustering method (also called “average linkage clustering method”) is perhaps most often used in practice, there are other clustering methods. UPGMA forms clusters based on the average value of similarity between the two clusters being merged. Another is the SLINK clustering method, short for “single linkage” clustering method, and sometimes called “nearest neighbor” clustering method. When two clusters are joined by it, their similarity is that of their most similar pair of objects, one in each cluster. Another is the CLINK clustering method, short for “complete linkage” clustering method, and sometimes called “furthest neighbor” clustering method. When two clusters are joined by it, their similarity is that of the most dissimilar pair of objects, one in each cluster. Another is Ward’s clustering method, which assigns objects to clusters in such a way that a sum-of-squares index is minimized.
. The data in the data matrix may be measured on a continuous scale (e.g., temperature), an ordinal scale (e.g., people’s ranked preference for products), or on a nominal scale for unordered classes (e.g., people’s sex coded as = female, = male). For an illustration of nominal scale measurement, suppose a military researcher takes a set of aircraft as the objects, and for their attributes records whether or not an aircraft can perform various functions. If the jth aircraft is able to perform the ith function, the (i, j)th element of the data matrix is coded with a “”; if it is unable to perform the ith function, it is coded with a “.” In this way, the data matrix consists of zeroes and ones. A similarity coefficient, such as the one named “the simple matching coefficient,” gives a numerical value for the similarity between any two aircraft. The cluster analysis produces a tree which shows which of the aircraft are functionally similar (belong to the same cluster) and which are functionally dissimilar (belong to different clusters). . Whenever the attributes of the data matrix are measured on a continuous scale, it is sometimes desired to standardize the data matrix. Standardizing recasts the units of measurement of the attributes as dimensionless units. Then the cluster analysis is performed on the standardized data matrix rather than on the data matrix. There are several alternative ways of standardizing (Romesburg ). . Commercial software packages for performing hierarchical cluster analysis include SPSS, SAS, CLUSTAN, and STATISTICA. Of these, SPSS is representative, allowing the user a choice of about similarity/dissimilarity coefficients and seven clustering methods. . In the literature of cluster analysis, certain terms have synonyms. Among other names for the objects to be clustered are “cases,” “individuals,” “subjects,” “entities,” “observations,” “data units,” and “OTU’s” (for “operational taxonomic units”). Among other names for the attributes are “variables,” “features,” “descriptors,” “characters,” “characteristics,” and “properties.” And among other names for the tree are the “dendrogram” and the “phenogram.” In contrast to hierarchical cluster analysis, nonhierarchical cluster analysis includes those clustering methods that do not produce a tree. The software packages mentioned above have programs for nonhierarchical cluster analysis. Perhaps the most-used nonhierarchical method is K-means cluster analysis. For it, the user specifies k, the number of clusters wanted, where k is an integer less than t, the number of objects. Software programs for K-means
Cluster Sampling
cluster analysis usually differ a bit in their details, but they execute an iterative process to find clusters, which typically goes like this: To begin the first iteration, the program selects k objects from the data matrix and uses them as k cluster seeds. The selection is made so that the Euclidean distances between the cluster seeds is large, which helps insure that the seeds cover all regions of the attribute space in which the objects reside. Next, the program forms tentative clusters by sequentially assigning each remaining object to whichever cluster seed it is nearest to. As objects are assigned, the cluster seeds are recomputed and made to have the attribute values that are the average of those of the objects in the clusters. Hence, cluster seeds generally change as objects are tentatively assigned to clusters. When the first iteration is finished, the resulting cluster seeds are taken as the k initial seeds to start the second iteration. Then the process is repeated, sequentially assigning the objects to their nearest cluster seed, and updating the seeds as the process moves along. Finally, after a number of iterations, when the change in the cluster seeds is tolerably small from one iteration to the next, the program terminates. The k final clusters are composed of the objects associated with the k cluster seeds from the final iteration. We now turn to the question, “Which is the better method for finding clusters – hierarchical cluster analysis or nonhierarchical cluster analysis?” The answer depends. Broadly speaking, researchers like having a choice of a large variety of similarity/dissimilarity coefficients, and like having the similarities among clusters displayed as a hierarchy in the form of a tree – two features that hierarchical methods offer but nonhierarchical methods do not. However, for hierarchical methods the amount of computation increases exponentially with the number of objects. Whereas for nonhierarchical methods the amount of computation increases less than exponentially because the methods do not require the calculation of similarities between all pairs of objects. In any event, all of the software packages mentioned above can handle very large data matrices for hierarchical methods and for nonhierarchical methods. For instance, according to the literature that accompanies CLUSTAN’s hierarchical cluster analysis program, the limit to the size of a data matrix that at present can be processed in a reasonable time on a basic PC is in the neighborhood of , objects and , attributes. For more objects than that, CLUSTAN’s nonhierarchical K-means program can handle as many as a million objects. Books that provide detailed accounts of hierarchical cluster analysis and nonhierarchical cluster analysis
C
include those by Aldenderfer and Blashfield (), Everitt (), and Romesburg ().
About the Author Dr. H. Charles Romesburg is Professor of Environment and Society at Utah State University and holds degrees in Operations Research and Biostatistics, Nuclear Engineering, and Mechanical Engineering. He is the author of four books, including Cluster Analysis for Researchers (North Carolina: Lulu, ) and Best Research Practices (North Carolina: Lulu, ). He is an active and prolific researcher with numerous scientific articles to his credit. His publications in which he is the sole author have been cited more than , times. The Wildlife Society has awarded him its Wildlife Publication Award for his article “Wildlife Science: Gaining Reliable Knowledge.”
Cross References 7Data Analysis 7Distance Measures 7Fuzzy Logic in Statistical Data Analysis 7Hierarchical Clustering 7Multivariate Data Analysis: An Overview 7Multivariate Statistical Analysis 7Random Permutations and Partition Models
References and Further Reading Aldenderfer MS, Blashfield RK () Cluster analysis. Sage, Beverly Hills Everitt B () Cluster analysis. E. Arnold, London Romesburg HC () Cluster analysis for researchers. Lulu.com, North Carolina Sneath PHA, Sokal RR () Numerical taxonomy: the principles and practice of numerical classification. W. H. Freeman, San Francisco
Cluster Sampling Janusz Wywial Professor Katowice University of Economics, Katowice, Poland The quality of statistical inference is dependent not only on, for example, estimator construction but on the structure of a population and a sampling scheme too. For example, let the estimation of total wheat production in a population of farms be considered. The population of farms is divided into clusters corresponding to villages. This estimation can be based on the ordinary simple sample or on the cluster sample. Population units can be selected to the sample by means of
C
C
Cluster Sampling
several sampling schemes. The units (i.e., farms) can be selected to the ordinary sample, or clusters of the units (i.e., villages) can be drawn to the cluster sample. The accuracy of the estimation depends on the sampling scheme and on the intraclass spread of a variable under study (wheat production). When should the ordinary simple sample be used and when should the cluster one? Let us consider a fixed and finite population case. The fixed and finite population of the size N is denoted by Ω = {ω , . . . , ω N } , where ω k is an element (unit) of the population U. Let us assume that Ω is divided into G mutually disjoint clusters Ω k (k = , . . . , G) such that G
⋃ Ω k = Ω. The size of a cluster Ω k is denoted by Nk . k= G
So, ≤ Nk ≤ N and ⋃ Nk = N. Let U = {Ω , . . . , Ω G } k=
be the set of all clusters. The clusters are called units (elements) of the set U. The cluster sample is selected from the set U and it is denoted by s = {Ω i , . . . , Ω in } . The size of s is denoted by n, where ≤ n ≤ G. Let S be the set (space) of samples. The cluster sample is a random one if it is selected from U according to some sampling design denoted by P(s), where P(s) ≥ for s ∈ S and ∑P(s) = . s∈S
Let the inclusion probability of the first order be denoted by π k = ∑P(s), k = , . . . , G. A random sample is selected {s:k∈s}
from a population by means of the so-called sampling scheme, which fulfills the appropriate sampling design. It is well known that a sample can be selected according to previously determined inclusion probabilities of the first order without any explicit definition of the sampling design. This inference simplifies our practical research. Frequently, the inclusion probabilities are determined as proportional to cluster sizes, so π k ∝ Nk for k = , . . . , G. In general, π k ∝ xk , where xk > is the value of an auxiliary variable. Let us note that it is possible to show that the well-known systematic sampling design is a particular case of the cluster sampling design. Moreover, the cluster sampling design is a particular case of two (or more) stage sampling designs. In general, all known sampling designs and schemes can be applied to the cluster case. The examples of sampling designs and schemes are as follows: the simple cluster sample of fixed size n, drawn without replacement, is selected according to the following sampling design: P(s) = /( G ) g
for s ∈ S. The inclusion probability of the first order is g π k = . The sampling scheme fulfilling that sampling G design is as follows: The first element (unit) of the set U is selected to the sample with the probability /G, the next one with the probability /(G−), the kth element with the
probability /(G − k + ), and so on until the nth element of the sample. The sampling scheme selecting with replacement units to the sample of fixed size n is as follows: Each element of U is selected with probabilities equal to pk , where, for example, pk = xk /∑xi . So, elements are independently i∈U
drawn to the sample n times. In this case, the sampling design is defined in a straightforward manner. Particularly, if pk = /G for all k = , . . . , G, the simple cluster sample drawn with replacement is selected according to the sampling design P(s) = /Gn . In this case, each element of U is selected with the probability /G to the sample of size n. The so-called Poisson without replacement sampling scheme is as follows: The kth unit is selected with the probability pk , < pk ≤ , k = , . . . , G. In this case, the sample size is not fixed because ≤ n ≤ G. There exists the so-called conditional without replacement sampling design of a fixed sample size, but unfortunately its sampling schemes are complicated, see, for example, Tillé (). Additionally, let us note that the cluster sample can be useful in the case of estimating the population mean. It is well known that the precision of a population mean estimation, performed on the basis of the simple cluster sample, depends on the so-called intraclass (intracluster) correlation coefficient, see, for example, Hansen et al. () or Cochran (). Let us assume that sizes of clusters are the same and equal to M and N = GM. The ordinary variance of the variable is defined by v =
G ∑ ∑ (ykj − y) , where yk = N k= j∈Ω k
G ∑ ∑ ykj is the cluster sample. The intraclass and the N k= j∈Ω k
betweenclass variances are given by the expressions: vw = G G ∑ ∑ (ykj − yk ) and vb = ∑(¯yk − y¯) , G(M − ) k= j∈Ω k G − k= respectively, where yk = ∑ ykj . The intraclass correM j∈Ω k
lation coefficient is defined by the following expression: rI =
G ∑ ∑ ∑(yki − y) (ykj − y¯). The coefficient ρ N v k= i≠j∈Ω k
takes its value from the closed interval [−/(M − ), ]. The coefficient rI can be rewritten in the following forms: rI = (vb − vw /M) /v, rI = − vw /v or rI = (Mvm /v − )/ (M − ). The expressions lead to the conclusion that the intraclaas correlation coefficient is negative (positive) when the ordinary variance is smaller (larger) than the intraclass variance or equivalent if the ordinary variance is larger (smaller) than the betweenclass variance divided by M. Let us note that it is well known that the simple cluster sample mean is a more accurate estimator of the population mean than the simple sample mean when the
Collapsibility
intraclass correlation coefficient is negative. So, this leads to the conclusion that, if only possible, a population should be clustered in such a way that the intraclass correlation coefficient takes the smallest negative value. A more complicated case of unequal cluster sizes was considered, for example, by Konijn (). In this case, Särndal et al. () considered the so-called homogeneity coefficient, which is the function of the intraclass variance. On the basis of the cluster sample, not only the estimation of population parameters is performed but also testing statistical hypothesis, see, for example, Rao and Scott ().
Cross References 7Adaptive Sampling 7Intraclass Correlation Coefficient 7Multistage Sampling 7Sample Survey Methods 7Sampling From Finite Populations
References and Further Reading Cochran WG () Sampling techniques. Wiley, New York Hansen MH, Hurvitz WN, Madow WG () Sample survey methods and theory, vols I and II. Wiley, New York Konijn HS () Statistical theory of sample survey and analysis. North-Holland, Amsterdam Rao JNK, Scott A () The analysis of categorical data from complex sample surveys: chi-square tests of goodnees of fit and independence in two-way tables. J Am Stat Assoc (): – Särndal CE, Swensson B, Wretman J () Model assisted survey sampling. Springer, New York/Berlin/Heidelberg/London/ Paris/Tokyo/Hong Kong/Barcelona/Budapest Tillé Y () Sampling algorithms. Springer, New York
About the Author For biography see the entry 7Random Variable.
Cross References 7Semi-Variance in Finance 7Standard Deviation 7Variance
St¸epniak C () An effective characterization of Schur-convex functions with applications. J Convex Anal :–
Czesław St¸epniak Professor Maria Curie-Skłodowska University, Lublin, Poland University of Rzeszów, Rzeszów, Poland
Collapsibility
Coefficient of variation is a relative measure of dispersion and it may be considered in three different contexts: in probability, in a data set or in a sample. In the first context it refers to distribution of a random variable X and is defined by the ratio σ µ
Data series x = (x , . . . , xn ) corresponds to distribution of a random variable X taking values xi with probabilities pi = kni , for i = , . . . , n, where ki is the number of appearance of xi in the series. In this case the formula () remains valid if we replace µ by x = n ∑i xi and σ by √ (x − x) . n ∑ i If x = (x , . . . , xn ) is a sample from a population, then the coefficient may be treated as a potential estimator of the coefficient of variation v in the whole population. Since (x − x) is biased estimator of σ in order to elimin ∑ i nate this bias we use the sample coefficient of variance in the form s v= , () x √ where s = n− ∑(xi − x) . One can ask whether v is normalized, i.e., whether it takes values in the interval [, ]. In spite of that v is well defined providing x > it seems reasonable to restrict oneself to the nonnegative samples x, i.e., satisfying the condition xi ≥ for all i and ∑i xi > . Under this assumption the sample coefficient () of vari√ ance in the sample takes values in the interval [, n] and the lower and upper bound is attained. Therefore it is not normalized.
References and Further Reading
Coefficient of Variation
v=
C
()
√ where µ = EX and σ = E(X − EX) . It is well defined if EX > . Moreover it is scale-invariant in the sense that cX has the same v for all positive c.
Sander Greenland Professor University of California-Los Angeles, Los Angeles, CA, USA
Collapsibility in Contingency Tables Consider the I by J by K contingency table representing the joint distribution of three discrete variables X, Y, Z, the I by J marginal table representing the joint distribution of X and Y, and the set of conditional I by J subtables (strata) representing the joint distributions of X and Y within levels
C
C
Collapsibility
Collapsibility. Table Trivariate distribution with (a) strict collapsibility of Y∣X risk differences over Z, (b) collapsibility of Y∣X risk ratios when standardized over the Z margin, and (c) noncollapsibility of Y∣X odds ratios over Z. Table entries are cell probabilities Z=
a
Z=
Collapsed over Z
X=
X=
X=
X=
X=
X=
Y=
.
.
.
.
.
.
Y=
.
.
.
.
.
.
Risksa
.
.
.
.
.
.
Differences
.
.
.
Ratios
.
.
.
Odds ratios
.
.
.
Probabilities of Y = in column
of Z. A measure of association of X and Y is strictly collapsible across Z if it is constant across the strata (subtables) and this constant value equals the value obtained from the marginal table. Noncollapsibility (violation of collapsibility) is sometimes referred to as 7Simpson’s paradox, after a celebrated article by Simpson (). This phenomenon had however been discussed by earlier authors, including Yule (); see also Cohen and Nagel (). Some statisticians reserve the term Simpson’s paradox to refer to the special case of noncollapsibility in which the conditional and marginal associations are in opposite directions, as in Simpson’s numerical examples. Simpson’s algebra and discussion, however, dealt with the general case of inequality. The term “collapsibility” seems to have arisen in later work; see Bishop et al. (). Table provides some simple examples. The difference of probabilities that Y = (the risk difference) is strictly collapsible. Nonetheless, the ratio of probabilities that Y = (the risk ratio) is not strictly collapsible because the risk ratio varies across the Z strata, and the odds ratio is not at all collapsible because its marginal value does not equal the constant conditional (stratum-specific) value. Thus, collapsibility depends on the chosen measure of association. Now suppose that a measure is not constant across the strata, but that a particular summary of the conditional measures does equal the marginal measure. This summary is then said to be collapsible across Z. As an example, in Table the ratio of risks averaged over (standardized to) the marginal distribution of Z is Σ z P(Y = ∣X = , Z = z)P(Z = z)/Σ z P(Y = ∣X = , Z = z) P(Z = z) = {−.(.) + .(.)}/{−.(.) + .(.)} = .,
which is equal to the marginal (crude) risk ratio. Thus, the risk ratio in Table is collapsible under this particular weighting (standardization) scheme for the risks. Various tests of collapsibility and strict collapsibility have been developed (e.g., Whittemore ; Asmussen and Edwards ; Ducharme and LePage ; Greenland and Mickey ; Geng ) as well as generalizations to partial collapsibility. The literature on graphical probability models distinguishes other types of collapsibility; see Frydenberg (), Whittaker (, Sect. .) and Lauritzen (, Sect. .) for examples. Both definitions given above are special cases of parametric collapsibility (Whittaker ).
Collapsibility in Regression Models The above definition of strict collapsibility extends to regression contexts. Consider a generalized linear model (see 7Generalized Linear Models) for the regression of Y on three vectors w, x, z: g[E(Y∣w, x, z)] = α + wβ + xγ + zδ. This regression is said to be collapsible for β over z if β∗ = β in the regression omitting z, g[E(Y∣w, x)] = α ∗ + wβ∗ + xγ ∗ and is noncollapsible otherwise. Thus, if the regression is collapsible for β over Z and β is the parameter of interest, Z need not be measured to estimate β. If Z is measured, however, tests of β ∗ = β can be constructed (Hausman ; Clogg et al. ). The preceding definition generalizes the original contingency-table definition to arbitrary variables. There is a technical problem with the above regression definition,
Collapsibility
however: If the first (full) model is correct, it is unlikely that the second (reduced) regression will follow the given form; that is, most families of regression models are not closed under deletion of Z. For example, suppose Y is Bernoulli with mean p and g is the logit link function ln[p/( − p)], so that the full regression is first-order logistic. Then the reduced regression will not follow a first-order logistic model except in special cases. One way around this dilemma (and the fact that neither of the models is likely to be exactly correct) is to define the model parameters as the asymptotic means of the maximum-likelihood estimators. These means are well-defined and interpretable even if the models are not correct (White ). If the full model is correct, δ = implies collapsibility for β and γ over Z. Nonetheless, if neither β nor δ is zero, marginal independence of the regressors does not ensure collapsibility for β over Z except when g is the identity or log link (Gail et al. ; Gail ). Conversely, collapsibility can occur even if the regressors are associated (Whittemore ; Greenland et al. ). Thus, it is not generally correct to equate collapsibility over Z with simple independence conditions, although useful results are available for the important special cases of linear, log-linear, and logistic models (e.g., see Gail ; Wermuth , ; Robinson and Jewell ; Geng ; Guo and Geng ).
Confounding Versus Noncollapsibility Much of the statistics literature does not distinguish between the concept of confounding as a bias in effect estimation and the concept of noncollapsibility; for example, Becher () defines confounding as β∗ ≠ β in the regression models given above, in which case the elements of Z are called confounders. Similarly, Guo and Geng () define Z to be a nonconfounder if β ∗ = β. Nonetheless, confounding as defined in the causal-modeling literature (See 7Confounding) may occur with or without noncollapsibility, and noncollapsibility may occur with or without confounding; see Greenland (, ) and Greenland et al. () for examples. Mathematically identical conclusions have been reached by other authors, albeit with different terminology in which noncollapsibility is called “bias” and confounding corresponds to “covariate imbalance” (Gail ; Hauck et al. ).
About the Author For biography see the entry 7Confounding and Confounder Control.
C
Cross References 7Confounding and Confounder Control 7Simpson’s Paradox
References and Further Reading Asmussen S, Edwards D () Collapsibility and response variables in contingency tables. Biometrika :– Becher H () The concept of residual confounding in regression models and some applications. Stat Med :– Bishop YMM, Fienberg SE, Holland PW () Discrete multivariate analysis: theory and practice. MIT Press, Cambridge Clogg CC, Petkova E, Haritou A () Statistical methods for comparing regression coefficients between models (with discussion). Am J Sociol :– Cohen MR, Nagel E () An introduction to logic and the scientific method. Harcourt Brace, New York Ducharme GR, LePage Y () Testing collapsibility in contingency tables. J R Stat Soc Ser B :– Frydenberg M () Marginalization and collapsibility in graphical statistical models. Ann Stat :– Gail MH () Adjusting for covariates that have the same distribution in exposed and unexposed cohorts. In: Moolgavkar SH, Prentice RL (eds) Modern statistical methods in chronic disease epidemiology. Wiley, New York, pp – Gail MH, Wieand S, Piantadosi S () Biased estimates of treatment effect in randomized experiments with nonlinear regressions and omitted covariates. Biometrika :– Geng Z () Algorithm AS . Decomposability and collapsibility for log-linear models. J R Stat Soc Ser C :– Geng Z () Collapsibility of relative risk in contingency tables with a response variable. J R Stat Soc Ser B :– Greenland S () Interpretation and choice of effect measures in epidemiologic analyses. Am J Epidemiol :– Greenland S () Absence of confounding does not correspond to collapsibility of the rate ratio or rate difference. Epidemiology :– Greenland S, Mickey RM () Closed-form and dually consistent methods for inference on collapsibility in × × K and × J × K tables. J R Stat Soc Ser C :– Greenland S, Robins J, Pearl J () Confounding and collapsibility in causal inference. Stat Sci :– Guo J, Geng Z () Collapsibility of logistic regression coefficients. J R Stat Soc Ser B :– Hauck WW, Neuhas JM, Kalbfleisch JD, Anderson S () A consequence of omitted covariates when estimating odds ratios. J Clin Epidemiol :– Hausman J () Specification tests in econometrics. Econometrica :– Lauritzen SL () Graphical models. Clarendon, Oxford Neuhaus JM, Kalbfleisch JD, Hauck WW () A comparison of cluster-specific and population-averaged approaches for analyzing correlated binary data. Int Stat Rev :– Robinson LD, Jewell NP () Some surprising results about covariate adjustment in logistic regression. Int Stat Rev :– Simpson EH () The interpretation of interaction in contingency tables. J R Stat Soc Ser B :– Wermuth N () Parametric collapsibility and lack of moderating effects in contingency tables with a dichotomous response variable. J R Stat Soc Ser B :–
C
C
Comparability of Statistics
Wermuth N () Moderating effects of subgroups in linear models. Biometrika :– White HA () Estimation, inference, and specification analysis. Cambridge University Press, New York Whittaker J () Graphical models in applied multivariate statistics. Wiley, New York Whittemore AS () Collapsing multidimensional contingency tables. J R Stat Soc Ser B :– Yule GU () Notes on the theory of association of attributes in statistics. Biometrika :–
Comparability of Statistics Peter Hackl Professor Vienna University of Business and Economics, Vienna, Austria Comparison is the cognitive function that is basis for any measuring process and a frequent activity in everyday human life. Nowadays, comparisons of statistics over time and, even more demanding, cross-national and multilateral comparisons are a central element of economic and social analyzes. For politics and administration, for business and media, and for each citizen, comparisons are means to understand and assess the political, economic, and social processes of this world. This situation raises questions: Under which conditions are statistics comparable? Under which conditions is a comparison valid and leads to reliable results? What conclusions may be drawn and what are the risks implied by such conclusions?
Comparability: Definition and Assessment Rathsmann-Sponsel and Sponsel () describe “comparison” as a -digit relation f (P, S, Z, K, V, A, B); P represents the comparing person, S and Z describe the comparing situation and the purpose of comparison, respectively; vector K stands for a number of criteria, V for a number of procedures, and A and B represent the characteristics of the objects to be compared, respectively. This rather formal view of psychologists indicates the complexity of the interaction between the individual and the objects to be compared. More visible becomes this complexity when the definition is applied to real situations, e.g., comparing the employment rates of two countries. The employment rate is the number of persons aged – in employment as the share of the total population of the same age group. Obviously, the definition of
“being in employment” and the exact meaning of “persons aged –” are crucial for the value that is obtained for the employment rate. In addition, the statistical value is affected by the sampling design and other aspects of the data collection. In general, statistics are based on concepts and definitions, and the value of a statistic is the result of a complex measurement process; comparability is affected by all these factors and, consequently, a wide range of facts must be considered. Moreover, the relevance of these facts depends on the purpose of comparison, the comparing situation, and other aspects of the comparison process. E.g., if the comparison of the employment rates of two countries is the basis for a decision on the allocation of subsidies for structural development, comparability is a more serious issue than in the case where the result of the comparisons does not have such consequences. These – and many other – characteristics must be taken into consideration when assessing differences between two employment rates. Assessment of comparability has to take into account the multi-dimensionality of the conditions for comparability. Many aspects to be considered are qualitative, so that the corresponding dimensions cannot be measured on a metric scale. Moreover, important characteristics of the statistical products or the underlying measurement processes are often not available or uncertain. Hence, in general, it is not feasible to give a comprehensive picture of comparability by means of a few metric measures. Alternatives are ●
●
An indicator in form of a number between zero and one that indicates the degree of comparability, a one indicating perfect comparability. A rating of comparisons on an ordinal rating scale with a small number of points, a high value representing good comparability.
An example for a rating scale is the three point scale that is used for the “Overall assessment of accuracy and comparability” of indicators – such as the employment rate – within the Eurostat Quality Profiles; see Jouhette and Sproge (). This overall assessment is rated from “A” to “C”. Grad “A” indicates that ●
●
Data is collected from reliable sources applying high standards with regard to methodology/accuracy and is well documented in line with Eurostat metadata standard. The underlying data is collected on the basis of a common methodology for the European Union and,
Comparability of Statistics
●
where applicable, data for US and Japan can be considered comparable; major differences being assessed and documented. Data are comparable over time; impact of procedural or conceptual changes being documented.
This example illustrates: ● ●
●
●
That the rating process reduces a high-dimensional information to a single digit. Where the characteristics of the statistics to be compared, the underlying measurement processes, and also conditions of the comparison process are crucial input elements for the rating of comparability. That the rating outcome has only the character of a label which the user might trust but which only reflects – perhaps vaguely – the result of a complex and subjective assessment process. That the rating outcome may miss to give appropriate weight to aspects that are important for a certain user.
A rating of the comparability on an ordinal rating scale has the advantage that it allows an easy communication about comparability. The professional assessment of comparability requires: ● ●
An adequate competence of the scrutinizer A careful documentation of all characteristics of the statistical products that are relevant for assessing the comparability
Generally, the scrutinizer will be different from the producer of a statistical product. This certainly will be the case in respect of cross-national comparisons. The producer has to provide a comprehensive set of metadata that documents all characteristics which are relevant for assessing the comparability. The outcome of this exercise might be an indicator of the types that are described above. For the non-expert user, the assessment of the comparability of statistical products is hardly possible even when a well-designed set of all metadata is available that are relevant for assessing the comparability. Most users of the statistical products will have to rely on the professional assessment of the experts.
Comparability in Official Statistics The integration of societies and the globalization of economies have the consequence that not only comparisons over time but especially cross-regional comparisons of statistical products are of increasing interest and importance. Political planning and decisions of supranational bodies need information that encompasses all involved
C
nations. Multi-national enterprises and globally acting companies face the same problem. Of even higher relevance for the need of comparability is the fact that statistical products are more and more used for operational purposes. Within the European Union, the process of integration of the member states into a community of states requires political measures in many areas. National statistical indicators are the basis for allocating a part of the common budget to member states, for administrating the regional and structural funds, for assessing the national performances with respect to the pact for stability and growth, and for various other purposes. It is in particular the European version of the System of National Accounts (SNA) ESA that plays such an operational role in various administrative processes of the European Union. The Millennium Development Goals and the Kyoto Protocol are other examples for the use of statistical indicators in defining political aims and assessing the corresponding progress. In all these cases, the comparability of the relevant statistical products is a core issue. In the cross-national context, the responsibility for harmonizing cross-national concepts, definitions, and methodological aspects must be assigned to an authority with supra-national competence. Organizations like the UN, OECD, and Eurostat are engaged in the compilation of standards and the editing of recommendations, guidelines, handbooks, and training manuals, important means to harmonize statistical products and improve their comparability. Examples of standards are the Statistical Classifications of Economic Activities (ISIC) and the International Statistical Classification of Diseases (ICD). Principles and Recommendations for Population and Housing Censuses adopted by the Statistical Commission of the UN is an example for a standard methodology. Examples of standards on the European level are the NACE and CPA. Within the European Union, standards and methods are laid down in regulations which are legally binding for the member states. E.g., the ESA was approved as a Council Regulation in June and stipulates the member states to apply the SNA in a very concrete form. In working groups, experts from the member states are dealing with the preparation and implementation of such regulations; the harmonization of national statistical products is a central concern of these activities. The important role that is attributed to statistical comparability within the ESS is stressed by the fact that the European Statistics Code of Practice () contains Coherence and Comparability as one of its principles. The corresponding indicators refer mainly to national aspects but also to the European dimension.
C
C
Comparability of Statistics
To assess the comparability of statistical products, national reports are essential that provide metadata for all related aspects of the statistical product. Standard formats for the documentation of metadata have been suggested by the International Monetary Fund in form of the General Data Dissemination Standard (GDDS) and the Special Data Dissemination Standard (SDDS). It should be mentioned that standardizing concepts, definitions, and methods also has unfavorable effects; comparability has a price. An important means for improving harmonization are standards; however, they are never perfect and tend to get outdated over time. In particular the adaptation of standards to methodological progress might be a time-consuming task. Generally, standardization reduces flexibility and makes adaptations to new developments, especially of methodological alternatives, more difficult. This is especially true if standards are put into the form of regulations. It is even truer if such standards are implemented in order to ease the use for operational purposes, as it is the case in the ESS.
Conclusions Lack of comparability may lead to erroneous results when statistical products are compared. The need for crossnational comparability is even more pronounced if statistical results are used for operational purposes as it is the case, e.g., in the European Union. Hence, comparability is an important quality aspect of statistical products. It is affected by the involved concepts and definitions, the measurement processes, and comparability may also depend on conditions of the comparison. The producer of a statistical product has to care that the conditions of comparability are fulfilled to the highest extent possible. In the cross-national context, international organizations like 7Eurostat are fostering the compilation of standards for concepts and definitions and of principles and standards for methods and processes in order to harmonize statistical products and improve their cross-national comparability. For the assessment of comparability, a wide range of information is needed, as many aspects of the statistics to be compared but also of the purpose and conditions of the comparison have to be taken into account. No general rules are available that guarantee a valid assessment of comparability; only experts with profound knowledge can be expected to give a reliable assessment. For such an assessment, metadata which document all relevant characteristics are essential and have to be provided by the producer of the statistical product. For cross-national purposes, organizations like Eurostat have to care that the
relevant metadata are provided by the respective producers. The user, e.g., the consumer of an economic or social analysis, has to trust that the analysts and experts made use of the involved statistics responsively.
About the Author Dr. Peter Hackl was born in Linz, Austria. He is a Professor (since ) at the Department of Statistics, Vienna University of Business and Economics (Wirtschaftsuniversität); Head of the Division of Economic Statistics (since ). During the academic year / and during the summer term , he was visiting professor at the University of Iowa, Iowa City. He was President of the Austrian Statistical Society (Österreichische Statistische Gesellschaft), (–). He was ViceDean of Studies (–) at the Wirtschaftsuniversität, member and deputy chairman (–) of the Statistikrat, the advisory board of Statistics Austria, the Austrian National Statistical Office; Chairman of the Committee for Quality Assurance. He was Director General of Statistics Austria (–). Professor Hackl is Elected member of the International Statistical Institute (since ). He has authored/coauthored about articles in refereed journals in statistical theory and methods and applications of statistics. He was also editor of two books and has authored/coauthored books. He was awarded an Honorary Doctorate, Stockholm University of Economics ().
Cross References 7Economic Statistics 7Eurostat 7National Account Statistics
References and Further Reading European Commission () Communication from the Commission to the European Parliament and to the Council on the independence, integrity and accountability of the national and Community statistical authorities, Recommendation on the independence, integrity and accountability of the national and Community statistical authorities, COM () European Statistics Code of Practice () http://epp.eurostat. ec.europa.eu/portall/page/portal/quality/documents/code_ practice.pdf Eurostat () ESS handbook for quality reports. European Communities, Luxembourg Jouhette S, Sproge L () Quality profiles for structural indicators EU LFS based indicators. In: th CEIES seminar, Expert meeting statistics “Structural indicators”, Luxembourg, European Communities, pp – Rathsmann-Sponsel I, Sponsel R () Allgemeine Theorie und Praxis des Vergleichens und der Vergleichbarkeit. Internet
Complier-Average Causal Effect (CACE) Estimation
Publikation für allgemeine und integrative Psychotherapie, www.sgipt.org/wisms/verglbk.htm Nederpelt V, Peter WM () Checklist quality of statistical output. Statistics Netherlands, Den Haag/Heerlen
Complier-Average Causal Effect (CACE) Estimation Graham Dunn Professor of Biomedical Statistics and Head of the Health Methodology Research Group University of Manchester, Manchester, UK
Imagine a simple randomized controlled trial evaluating a psychotherapeutic intervention for depression. Participants are randomized to one of two treatment groups. The first (the control condition) comprises treatment and management as usual (TAU). Participants in the second group are offered a course of individual cognitive behavior therapy (CBT) in addition to TAU. The outcome of the trial is evaluated by assessing the severity of depression in all of the participants months after 7randomization. For simplicity, we assume there are no missing outcomes. We find a difference between the average outcomes for the two groups to be about four points on the depression rating scale, a difference that is of only borderline clinical significance. This difference of four points is the well-known intention-to-treat (ITT) effect – it is the estimated effect treatment allocation (i.e., offering the treatment). This we will call ITTALL . Participants in the control (TAU) condition did not get any access to CBT but we now discover that only about half of those offered psychotherapy actually took up the offer. Only % of the treatment group actually received CBT. So, it might be reasonable to now ask “What was the effect of receiving CBT?” or “What was the treatment effect in those who complied with the trial protocol (i.e., treatment allocation)?” Traditionally, trialists have been tempted to carry out what is called a “Per Protocol” analysis. This involves dropping the non-compliers from the treatment arm and comparing the average outcomes in the compliers with the average outcome in all of the controls. But this is not comparing like with like. There are likely to be selection effects (confounding) – those with a better (or worse) treatment-free prognosis might be more likely to turn up for their psychotherapy. The same criticism also applies to the abandonment of randomization altogether and comparing the average outcomes in those
C
who received treatment with those who did not (a mixture of controls and non-compliers) in a so-called “As treated” analysis. To obtain a valid estimate of the receipt of treatment we need to be able to compare the average of the outcomes in those who received CBT with the average of the outcomes of the control participants who would have received CBT had they been allocated to the treatment group. This is the Complier-Average Causal Effect (CACE) of treatment. How do we do this? The simplest approach is based on the realization that the ITT effect is attenuated estimate of the CACE, and that the amount of attenuation is simply the proportion of compliers (or would-be compliers) in the trial. The proportion of compliers (PC ) is simply estimated by the proportion of those allocated to the treatment group who actually receive CBT. We postulate that we have two (possibly hidden) classes of participant: Compliers and Non-compliers. Non-compliers receive no CBT irrespective of their treatment allocation. The intention-totreat effects in the Compliers and Non-compliers are ITTC (≡ CACE) and ITTNC , respectively. It should be clear to the reader that ITTALL = PC ITTC and ( − PC )ITTNC . To make use of this simple relationship, let’s now assume that treatment allocation in the Non-compliers has no impact on their outcome (i.e., does not affect the severity of their depression). This assumption is often referred to as an exclusion restriction. It follows that ITTALL = PC ITTC and that CACE = ITTC = ITTALL /PC So with % compliance, and estimated overall ITT effect of units on the depression scale, the CACE estimate is units – a result with much more promise to our clinicians. To get a standard error estimate we might apply a simple bootstrap (see 7Bootstrap Methods). Note that CACE estimation is not a means of conjuring up a treatment effect from nowhere – if the overall ITT effect is zero so will the CACE be. If the overall ITT effect is not statisticallysignificant, the CACE will not be statistically-significant. One last point: in a conventional treatment trial aiming to demonstrate efficacy, the ITT estimate will be conservative, but in a trial designed to demonstrate equivalence (or non-inferiority) it is the CACE estimate that will be the choice of the cautious analyst (we do not wish to confuse attenuation arising from non-compliance with differences in efficacy). Here we have illustrated the ideas with the simplest of examples. And here we have also made the derivation of the CACE estimate as simple as possible without any detailed reference to required assumptions. An analogous procedure was first used by Bloom () but its
C
C
Components of Statistics
formal properties were derived and compared with instrumental variable estimators of treatment effects by Angrist et al. (). Non-compliance usually has an implication for missing data – those that do not comply (or wouldbe Non-compliers) with their allocated treatment are also those who are less likely to turn up to have their outcome assessed. The links between CACE estimation and missing data models (assuming latent ignorability) are discussed by Frangakis and Rubin (). Generalizations of CACE methodology to estimation of treatment effects through the use of Principal Stratification have also been introduced by Frangakis and Rubin ().
About the Author For biography see the entry 7Psychiatry, Statistics in
Cross References 7Instrumental Variables 7Principles Underlying Econometric Estimators for Iden-
tifying Causal Effects
References and Further Reading Angrist JD, Imbens GW, Rubin DB () Indentification of causal effects using instrumental variables (with discussion). J Am Stat Assoc :– Bloom HS () Accounting for no-shows in experimental evaluation designs. Evaluation Rev :– Frangakis CE, Rubin DB () Addressing complications of intention-to-treat analysis in the combined presence of allor-none treatment-noncompliance and subsequent missing outcomes. Biometrika :– Frangakis CE, Rubin DB () Principal stratification in causal inference. Biometrics :–
Components of Statistics Clive William John Granger† Professor Emeritus University of California-San Diego, San Diego, CA, USA The two obvious subdivisions of statistics are: (a) Theoretical Statistics and (b) Practical Statistics. . The theoretical side is largely based on a mathematical development of probability theory (see 7Probability Theory: An Outline) applicable to data, particularly the asymptotic properties of estimates (see 7Properties of Estimators) which lead to powerful theorems such as the 7Central Limit Theorem. The aim is to put many practical approaches to data analysis (see
also 7Categorical Data Analysis; 7Multivariate Data Analysis: An Overview; 7Exploratory Data Analysis; 7Functional Data Analysis) on a sound theoretical foundation and to develop theorems about the properties of these approaches. The theories are usually based on a number of assumptions that may or may not hold in practice. . Practical statistics considers the analysis of data, how the data can be summarized in useful fashions, and how relationships between sets of data from different variables can be described and interpreted. The amount and the quality of the data (see 7Data Quality) that is available are essential features in this area. On occasions data may be badly constructed or terms may be missing which makes analysis more complicated. Descriptive statistics include means, variances, histograms, correlations, and estimates of quantiles, for example. There are now various types of statistics depending on the area of application. General statistics arose from considerations of gambling (see 7Statistics and Gambling), agriculture (see 7Agriculture, Statistics in; 7Analysis of Multivariate Agricultural Data), and health topics (see 7Medical research, Statistics in; 7Medical Statistics) but eventually a number of specialized areas arose when it was realized that these areas contained special types of data which required their own methods of analysis. Examples are: . Biometrics (see 7Biostatistics), from biological data which required different forms of measurement and associated tests. . 7Econometrics, for which 7Variables may or may not be related with a time gap; data can be in the form of 7Time Series (particularly in economies and finance) or in large panels (see 7Panel Data) in various parts of economics. The techniques developed over a wide range and the ideas have spread into other parts of statistics. . Psychometrics, where methods are required for the analysis of results from very specific types of experiments used in the area of psychology (see 7Psychology, Statistics in). Other major areas of application such as engineering, marketing, and meteorology generally use techniques derived from methods in the areas mentioned above, but all have developed some area-specific methods.
About the Author In Professor Granger was awarded the Nobel Memorial Prize in Economic Sciences (with Professor Robert
Composite Indicators
Engle) for methods of analyzing economic time series with common trends (cointegration). Granger was knighted in . Professor Granger had sent his contributed entry on June , excusing himself for not writing a bit longer, “Lexicon” kind of paper: “I have never written anything for a ‘Lexicon’ before and so have failed in my attempt to be helpful, but I do attach a page or so. I wish you good luck with your effort.” We are immensely thankful for his unselfish contribution to the prosperity of this project.
construction process because it has the greatest impact on the relevance of the indicator of the investigated phenomena. For that reason, the constructors team should include, besides the statisticians, who play the major role, the experts and stakeholders from the topic of the composite indicator. Due to the fact that many new multidimensional concepts do not have a generally agreed theoretical framework, transparency is essential in constructing credible indicators. ●
Cross References 7Statistics: An Overview 7Statistics: Nelder’s view
●
Composite Indicators Jože Rovan Associate Professor, Faculty of Economics University of Ljubljana, Ljubljana, Slovenia Definition: A composite indicator is formed when individual indicators are compiled into a single index, on the basis of an underlying model of the multidimensional concept that is being measured (OECD, Glossary of Statistical Terms). Multidimensional concepts like welfare, well-being, human development, environmental sustainability, industrial competitiveness, etc., cannot be adequately represented by individual indicators. For that reason, composite indicators are becoming increasingly acknowledged as a tool for summarizing complex and multidimensional issues. Composite indicators primarily arise in the following areas: economy, society, globalization, environment, innovation, and technology. A comprehensive list of indicators can be found at the following address: http:// composite-indicators.jrc.ec.europa.eu/FAQ.htm# List_of_Composite_Indicators_ The Handbook on Constructing Composite Indicators: Methodology and User Guide (OECD ) recommends a ten-step process of constructing composite indicators: ●
Theoretical framework is the starting point of the process of constructing composite indicators, defining individual indicators (e.g., variables) and their appropriate weights, reflecting the structure of the investigated multidimensional concept. This step is crucial in
C
●
●
●
●
●
Data selection should acquire analytically sound relevant indicators, having in mind their avaliability (country coverage, time, appropriate scale of measurement, etc.). Engagement of experts and stakeholders is recommended. Imputation of missing data (see 7Imputation) provides a complete dataset (single or. multiple). Inspection of presence of 7outliers in the dataset should not be omitted. Multivariate analysis reveals the structure of the considered dataset from two aspects: (a) units and (b) available individual indicators, using appropriate multivariate methods, e.g., 7principal component analysis, factor analysis (see 7Factor Analysis and Latent Variable Modelling), Cronbach coefficient alpha, cluster analysis (see 7Cluster Analysis: An Introduction), 7correspondence analysis, etc. These methods are able to reveal internally homogenous groups of countries or groups of indicators and interpret the results. Normalization procedures are used to achieve comparability of variables of the considered dataset, taking into account theoretical framework and the properties of the data. The robustness of normalization methods against possible 7outliers must be considered. Weighting and aggregation should take into account the theoretical framework and the properties of the data. The most frequently used aggregation form is a weighted linear aggregation rule applied to a set of variables (OECD ). Weights shoud reflect the relative importance of individual indicators in a construction of the particular composite indicator. Uncertainty and 7sensitivity analysis are necessary to evaluate robustness of composite indicators and to improve transparency, having in mind selection of indicators, data quality, imputation of missing data, data normalization, weighting, aggregation methods, etc. Back to the original data, to (a) reconsider the relationships between composite indicator and the original
C
C ●
●
Computational Statistics
data set and to identify the most influential indicators and (b) compare profiled performance of the considered units to reveal what is driving the composite indicator results, and in particular whether the composite indicator is overly dominated by a small number of indicators. Links to other indicators identify the relationships between the composite indicator (or its dimensions) and other individual or composite indicators. Visualization of results should attract audience, presenting composite indicators in a clear and accurate way.
Following the above-mentioned guidelines, the constructors of composite indicators should never forget that composite indicators should never be seen as a goal per se. They should be seen, instead, as a starting point for initiating discussion and attracting public interest and concern (Nardo et al. ). However, there is now general agreement about the usefulness of composite indicators: There is a strong belief among the constructors of composite indicators that such summary measures are meaningful and that they can capture the main characteristic of the investigated phenomena. On the other side, there is a scepticism among the critics of this approach, who believe that there is no need to go beyond an appropriate set of individual indicators. Their criticism is focused on the “arbitrary nature of the weighting process” (Sharpe ) in construction of the composite indicators.
Cross References 7Aggregation Schemes 7Imputation 7Multiple Imputation 7Multivariate Data Analysis: An Overview 7Multivariate Statistical Analysis 7Scales of Measurement 7Sensitivity Analysis
Munda G, Nardo M () Constructing consistent composite indicators: the issue of weights, EUR EN. Joint Research Centre, Ispra Nardo M, Saisana M, Saltelli A, Tarantola S () Tools for composite indicators building. european commission, EUR EN. Joint Research Centre, Ispra, Italy Saltelli A () Composite indicators between analysis and advocacy. Soc Indic Res :– Sharpe A () Literature review of frameworks for macroindicators. Centre for the Study of Living Standards, Ottawa, Canada
Computational Statistics Colin Rose Director Theoretical Research Institute, Sydney, NSW, Australia
What Is Computational Statistics? We define computational statistics to be: . . . ‘statistical methods/results that are enabled by using computational methods’. Having set forth a definition, it should be stressed, first, that names such as computational statistics and statistical computing are essentially semantic constructs that do not have any absolute or rigourous structure within the profession; second, that there are any number of competing definitions on offer. Some are unsatisfactory because they focus purely on data or graphical methods and exclude symbolic/exact methods; others are unsatisfactory because they place undue emphasis on ‘computationally-intensive methods’ or brute force, almost as if to exclude well-written efficient and elegant algorithms that might be computationally quite simple. Sometimes, the difficulty is not in the execution of an algorithm, but in writing the algorithm itself. Computational statistics can enable one: ●
References and Further Reading An information server on composite indicators and ranking systems (methods, case studies, events) http://compositeindicators.jrc.ec.europa.eu/FAQ.htm#List_of_Composite _Indicators Freudenberg M () Composite indicators of country performance: a critical assessment, OECD science, technology and industry working papers, OECD Publishing, / OECD, European Commission, Joint Research Centre () Handbook on constructing composite indicators: methodology and user guide. OECD Publishing OECD, Glossary of statistical terms (http://stats.oecd.org/ glossary/index.htm) OECD () Composite indicators of country performance: a critical assessment, DST/IND(), Paris
●
● ● ●
To work with arbitrary functional forms/distributions, rather than being restricted to traditional known textbook distributions. To simulate distributional properties of estimators and test statistics, even if closed-form solutions do not exist (computational inference rather than asymptotic inference). To compare statistical methods under different alternatives. To solve problems numerically, even if closed-form solutions are not possible or cannot be derived. To derive symbolic solutions to probability, moments, and distributional problems that may never have been solved before, and to do so essentially in real-time.
Computational Statistics
● ● ● ● ●
To explore multiple different models, rather than just one model. To explore potentially good or bad ideas via simulation in just a few seconds. To choose methods that are theoretically appropriate, rather than because they are mathematically tractable. To check symbolic/exact solutions using numerical methods. To bring to life theoretical models that previously were too complicated to evaluate . . .
Journals and Societies Important journals in the field include: ● ● ● ● ● ● ● ● ● ●
Combinatorics, Probability & Computing Communications in Statistics – Simulation and Computation Computational Statistics Computational Statistics and Data Analysis Journal of Computational and Graphical Statistics Journal of the Japanese Society of Computational Statistics Journal of Statistical Computation and Simulation Journal of Statistical Software SIAM Journal on Scientific Computing Statistics and Computing
Societies include: the International Association for Statistical Computing (IASC – a subsection of the ISI), the American Statistical Association (Statistical Computing Section), the Royal Statistical Society (Statistical Computing Section), and the Japanese Society of Computational Statistics (JSCS) . . . Computational statistics consists of three main areas, namely numerical, graphical and symbolic methods . . .
Numerical Methods The numerical approach is discussed in texts such as Gentle (), Givens and Hoeting (), and Martinez and Martinez (); for Bayesian methods, see Bolstad (). Numerical methods include: Monte Carlo studies to explore asymptotic properties or finite sample properties, pseudo-random number generation and sampling, parametric density estimation, non-parametric density estimation, 7bootstrap methods, statistical approaches to software errors, information retrieval, statistics of databases, high-dimensional data, temporal and spatial modeling, 7data mining, model mining, statistical learning, computational learning theory and optimisation etc. . . . While optimisation itself is an absolutely essential tool in the field, it is very much a field in its own right.
C
Graphical Methods Graphical methods are primarily concerned with either (a) viewing theoretical models and/or (b) viewing data/fitted models. In the case of theoretical models, one typically seeks to provide understanding by viewing one, two or three variables, or indeed even four dimensions (using -dimensional plots animated over time, translucent graphics etc.). Visualizing data is essential to data analysis and assessing fit; see, for instance, Chen et al. (). Special interest topics include smoothing techniques, kernel density estimation, multi-dimensional data visualization, clustering, exploratory data analysis, and a huge range of special statistical plot types. Modern computing power makes handling and interacting with large data sets with millions of values feasible . . . including live interactive manipulations.
Symbolic/Exact Methods The st century has brought with it a conceptually entirely new methodology: symbolic/exact methods. Recent texts applying the symbolic framework include Andrews and Stafford (), Rose and Smith (), and Drew et al. (). Traditional th century computer packages are based on numerical methods that tend to be designed much like a cookbook. That is, they consist of hundreds or even thousands of numerical recipes designed for specific cases. One function is written for one aspect of the Normal distribution, another for the LogNormal, etc. This works very well provided one stays within the constraints of the known common distributions, but unfortunately, it breaks down as soon as one moves outside the catered framework. It might work perfectly for random variable X, but not for X , nor exp(X), nor mixtures, nor truncations, nor reflections, nor folding, nor censoring, nor products, nor sums, nor . . . By contrast, symbolic/exact methods are built on top of computer algebra systems . . . programs such as Mathematica and Maple that understand algebra and mathematics. Accordingly, symbolic algorithms can provide exact general solutions . . . not just for specific distributions/models. Symbolic computational statistical packages include mathStatica (–, based on top of Mathematica) and APPL (based on top of Maple). Symbolic methods include: automated expectations for arbitrary distributions, probability, combinatorial probability, transformations of random variables, products of random variables, sums and differences of random variables, generating functions, inversion theorems, maxima/minima of random variables, symbolic and numerical maximum likelihood estimation (using exact methods),
C
C
Computational Statistics
curve fitting (using exact methods), non-parametric kernel density estimation (for arbitrary kernels), moment conversion formulae, component-mix and parameter-mix distributions, copulae, pseudo-random number generation for arbitrary distributions, decision theory, asymptotic expansions, 7order statistics (for identical and nonidentical parents), unbiased estimators (h-statistics, kstatistics, polykays), moments of moments, etc.
The Changing Notion of What is Computational Statistics Just or years ago, it was quite common for people working in computational statistics to write up their own code for almost everything they did. For example, the Handbook of Statistics : Computational Statistics (see Rao ) starts out Chapter by describing algorithms for sorting data. Today, of course, one would expect to find sorting functionality built into any software package one uses . . . indeed even into a word processor. And, of course, the ‘bar’ keeps on moving and evolving. Even in recent texts such as Gentle (), about half of the text (almost all of Part ) is devoted to computing techniques such as fixedand floating-point, numerical quadrature, numerical linear algebra, solving non-linear equations, optimisation etc., . . . techniques that Gentle et al. (, p. ) call “statistical computing” but which are really just computing. Such methods lie firmly within the domain of computational science and/or computational mathematics . . . they are now built into any decent modern statistical/mathematical software package . . . they take years of work to develop into a decent modern product, and they require tens of thousands of lines of code to be done properly . . . all of which means that it is extremely unlikely that any individual would write their own in today’s world. Today, one does not tend to build an airplane simply in order to take a flight. And yet many current texts are still firmly based in the older world of ‘roll your own’, devoting substantial space to routines that are (a) general mathematical tools such as numerical optimisation and (b) which are now standard functionality in modern packages used for computational statistics. While it is, of course, valuable to understand how such methods work (in particular so that one is aware of their limitations), and while such tools are absolutely imperative to carrying out the discipline of computational statistics (indeed, as a computer itself is) – these tools are now general mathematical tools and the days of building one’s own are essentially long gone.
Future Directions It is both interesting and tempting to suggest likely future directions.
(a) Software packages: At the present time, the computational statistics software market is catered for from two polar extremes. On the one hand, there are major general mathematical/computational languages such as Mathematica and Maple which provide best of breed general computational/numerical/graphical tools, and hundreds of high-level functional programming constructs to expand on same, but they are less than comprehensive in field-specific functionality. It seems likely such packages will further evolve by developing and growing tentacles into specific fields (such as statistics, combinatorics, finance, econometrics, biometrics etc.). At the other extreme, there exist narrow field-specific packages such as S-Plus, Gauss and R which provide considerable depth in field-specific functionality; in order to grow, these packages will likely need to broaden out to develop more general methods/general mathematical functions, up to the standard offered by the major packages. The software industry is nascent and evolving, and it will be interesting to see if the long-run equilibrium allows for both extremes to co-exist. Perhaps, all that is required is for a critical number of users to be reached in order for each eco-system to become self-sustaining. (b) Methodology: It seems likely that the field will see a continuing shift or growth from statistical inference to structural inference, . . . from data mining to model mining, . . . from asymptotic inference to computational inference. (c) Parallel computing: Multicore processors have already become mainstream, while, at the same time, the growth in CPU speeds appears to be stagnating. It seems likely then that parallel computing will become vastly more important in evolving computational statistics into the future. Future computational statistical software may also take advantage of GPUs (graphical processing units), though it should be cautioned that the latter are constrained in serious statistical work by the extremely poor numerical precision of current GPUs. (d) Symbolic methods: Symbolic methods are still somewhat in their infancy and show great promise as knowledge engines i.e., algorithms that can derive exact theoretical results for arbitrary random variables. (e) On knowledge and proof : Symbolic algorithms can derive solutions to problems that have never been posed before – they place enormous technological power into the hands of end-users. Of course, it is possible (though rare) that an error may occur (say in integration, or by mis-entering a model). In a sense,
Conditional Expectation and Probability
this is no different to traditional reference texts and journal papers which are also not infallible, and which are often surprisingly peppered with typographical or other errors. In this regard, the computational approach offers both greater exposure to danger, as well as the tools to avoid it. The “danger” is that it has become extremely easy to generate output in real-time. The sheer scale and volume of calculation has magnified, so that the average user is more likely to encounter an error, just as someone who drives a lot is more likely to encounter an accident. Proving that the computer’s output is actually correct can be very tricky, or impractical, or indeed impossible for the average practitioner to do, just as the very same practitioner will tend to accept a journal result at face value, without properly checking it, even if they could do so. The philosopher, Karl Popper, argued that the aim of science should not be to prove things, but to seek to refute them. Indeed, the advantage of the computational statistical approach is that it is often possible to check one’s work using two completely different methods: both numerical and symbolic. Here, numerical methods take on a new role of checking symbolic results. One can throw in some numbers in place of symbolic parameters, and one can then check if the solution obtained using symbolic methods (the exact theoretical solution) matches the solution obtained using numerical methods (typically, 7numerical integration or Monte Carlo methods, see 7Monte Carlo Methods in Statistics). If the numerical and symbolic solutions do not match, there is an obvious problem and we can generally immediately reject the theoretical solution (a la Popper). On the other hand, if the two approaches match up, we still do not have a proof of correctness . . . all we have is just one point of agreement in parameter space. We can repeat and repeat and repeat the exercise with different parameter values . . . and as we do so, we effectively build up, not an absolute proof in the traditional sense, but, appropriately for the statistics profession, an ever increasing degree of confidence . . . effectively a proof by probabilistic induction . . . that the theoretical solution is indeed correct. This is an extremely valuable (though time-consuming) skill to develop, not only when working with computers, but equally with textbooks and journal papers.
C
Cross References 7Bootstrap Asymptotics 7Bootstrap Methods 7Data Mining 7Monte Carlo Methods in Statistics 7Nonparametric Density Estimation 7Non-Uniform Random Variate Generations 7Numerical Integration 7Numerical Methods for Stochastic Differential
Equations 7R Language 7Statistical Software: An Overview 7Uniform Random Number Generators
References and Further Reading Andrews DF, Stafford JEH () Symbolic computation for statistical inference. Oxford University Press, New York Bolstad WM () Understanding computational Bayesian statistics. Wiley, USA Chen C, Härdle W, Unwin A () Handbook of data visualization. Springer, Berlin Drew JH, Evans DL, Glen AG, Leemis LM () Computational probability. Springer, New York Gentle JE () Computational statistics. Springer, New York Gentle JE, Härdle W, Mori Y (eds) () Handbook of computational statistics: concepts and methods. Springer, Berlin Givens GH, Hoeting JA () Computational statistics. Wiley, New Jersey Martinez WL, Martinez AR () Computational statistics handbook with MATLAB, nd edn. Chapman & Hall, New York mathStatica (–), www.mathStatica.com Rao CR () Handbook of statistics : computational statistics. Elsevier, Amsterdam Rose C, Smith MD () Mathematical statistics with Mathematica. Springer, New York
Conditional Expectation and Probability Takis Konstantopoulos Professor Heriot-Watt University, Edinburgh, UK
In its most elementary form, the conditional probability P(A∣B) of an event A given an event B is defined by
About the Author For biography see the entry 7Bivariate distributions.
P(A∣B) =
B(A ∩ B) , P(B)
C
C
Conditional Expectation and Probability
provided that P(B) ≠ . This is a well-motivated definition, compatible both with the frequency interpretation of probability as well as with elementary probability on countable spaces. An immediate consequence of the definition is 7Bayes’ theorem: if A , A , . . . , An are mutually disjoint events whose union has probability one, then P(A ∣B) = P(B∣A )P(A ) . ∑ni= P(B∣A i )P(A i ) Suppose now that X, Y are random variables taking values in finite sets. We define the conditional distribution of X given Y by ⎧ P(X=x,Y=y) ⎪ , ⎪ P(X = x∣Y = y) = ⎨ P(Y=y) ⎪ ⎪ ⎩,
if P(Y = y) ≠ if P(Y = y) = .
The latter choice, i.e., / interpreted as , is both physically motivated and mathematically desirable. The object P(X = x∣Y = y) is a probability in x (i.e., it sums up to over x) and a function of y. If X takes values in a set of real numbers then we can define the conditional expectation of X given Y = y by E(X∣Y = y) = ∑ xP(X = x∣Y = y),
()
x
where the summation extends over all possible values x of X. This is a function of y, say h(y) = E(X∣Y = y). We can then talk about the conditional expectation of X given Y as the random variable h(Y) obtained by substituting y by the random variable Y in the argument of h(y). From this definition the following important property of E(X∣Y) is easily derived: E[(X − E(X∣Y)) ⋅ g(Y)] = ,
()
for any random variable g(Y) which is a (deterministic) function of Y. One can easily generalise the above to countablyvalued random variables. However, defining conditional probability and expectation for general random variables cannot be done in the previous naive manner. One can mimic the definitions for random variables possessing density but this has two drawbacks: first, it is not easy to rigorously reconcile with the previous definitions; second, it is not easy to generalize. Instead, we resort to an axiomatic definition of conditional expectation, stemming directly from the fundamental property (). It can be easily verified that, in the earlier setup, there is only one function h(y) satisfying () for all functions g(y), and this h(y) is defined by (). The last observation leads us to the following definition: Let (Ω, F, P) be a probability space and X a positive random variable (i.e., a measurable function X : Ω → R+ ). Let G ⊂ F be another sigma-algebra. We say that E(X∣G)
is the conditional expectation of X given G if (a) E(X∣G) is G-measurable and (b) for all bounded G-measurable random variables G, we have E[XG] = E[E(X∣G)G].
()
Such an object exists and is almost surely unique. The latter means that if two random variables, H , H , say, satisfy E[XG] = E[Hi G], i = , , for all G then P(H = H ) = . (Such Hi are called versions of the conditional expectation.) Existence is immediate by the 7Radon–Nikodým theorem. Consider two measures on (Ω, G): the first one is P; the second one is E[X 1G ], G ∈ G (where 1G is defined as on G and on Ω/G). When P(G) = we have E[X 1G ] = and therefore the second measure is absolutely continuous with respect to the first. The Radon–Nikodým theorem ensures that the derivative (density) of the second measure with respect to the first exists and that it satisfies (). This observation and string of arguments is due to Kolmogorov (), and it is through this that modern Probability Theory was established as a mathematical discipline having a natural connection with Measure Theory. Having defined E[X∣G] for positive X we can define it for negative X by reversing signs and for general X via the formula E[X∣G] = E[max(X, )∣G] + E[min(X, )∣G], provided that wither E[max(X, )] < ∞ or E[min(X, )] > −∞. Given then two random variables X, Y (the first of which is real-valued, but the second may take values in fairly arbitrary spaces (such as a space of functions), we can define E[X∣Y] as E[X∣σ(Y)] where σ(Y) is the σ-algebra generated by Y. It can be seen that this is entirely compatible with the initial definition (). Passing on to conditional probability, observe that if A is an event, the expectation of 1A is precisely P(A). By analogy, we define P(A∣G) = E[1A ∣G]. For each event A ∈ F, this is a random variable, i.e., a measurable function of ω ∈ Ω which is defined almost surely uniquely (see explanation after formula ()). For a real-valued random variable X we define the conditional distribution function P(X ≤ x∣G) as E[1X≤x ∣G]. We would like this to be a right-continuous non-decreasing function of x. Since, for each x, P(X ≤ x∣G) is defined only almost surely, we need to show that we can, for each x, pick a version Hx of P(X ≤ x∣G) in a way that the probability of the event {Hx ≤ Hy if x ≤ y and limє↓ Hx+є = Hx } is one. This can be done and {Hx }x∈R is referred to as a regular conditional distribution function of X given G. Informally (and in practice) it is denoted as P(X ≤ x∣G). Regular conditional
Conditional Expectation and Probability
probabilities exist not only for real random variables X but also for random elements X taking values in a Borel space Kallenberg (). From this general viewpoint we can now go down again and verify everything we wish to have defined in an intuitive or informal manner. For instance, if (X, Y) is a pair of real-valued random variables having joint density f (x, y) then, letting fY (y) := ∫R f (x, y)dx, define ⎧ f (x,y) ⎪ , ⎪ h(x∣y) := ⎨ fY (y) ⎪ ⎪ ⎩,
if fY (y) ≠ if fY (y) =
.
x
Then the function ∫−∞ h(s∣Y)ds serves as a the regular conditional distribution of X given Y. We can also verify ∞ that E(max(X, )∣Y) = ∫ P(X > x∣Y)dx (in the sense that the right-hand side is a version of the left) and several other elementary formulae. It is important to mention an interpretation of E(X∣Y) as a projection. First recall the definition of projection in Euclidean space: Let x be a point (vector) in the space Rn and Π a hyperplane. We define the projection ̂ x of x onto Π as the unique element of Π which has minimal distance from x. Equivalently, the angle between x − ̂ x and any other vector g of Π must be ○ : this can be written as ⟨x−̂ x, g⟩ = , i.e., the standard inner product of x−̂ x and g is equal to . Next suppose that E[X ] < ∞. Then it can be seen that E[(X − E(X∣G)) ] = min E[(X − G) ], G
where the minimum is taken over all G-measurable random variables G with E[G ] < ∞. The defining property () then says that the inner product between X − E(X∣G) and any G is zero, just as in Euclidean space. Keeping the geometric meaning in mind, we can devise (prove and interpret) several properties of the conditional expectation. We mention one below. The tower property: If G ⊂ G are two sigma-algebras then E[E(X∣G )∣G ] = E[X∣G ]. The geometric meaning is as follows: If Π is a hyperplane (e.g., a plane in three dimensions) and Π a hyperplane contained in Π (e.g., a line on the plane) then we can find the projection onto Π by first projecting onto Π and then projecting the projection. The tower property holds for general random variables as long as conditional expectation can be defined, i.e., it does not require E[X ] < ∞. Another interpretation of it is as follows: if G , G represent states of knowledge (information, say) and G is wider than G (in the sense that G can be obtained from G ) then, in finding the conditional expectation of X given G , the
C
additional knowledge contained in G can be ignored. A particular form of this property is in the relation E[E(X∣G)] = E[X]. Another important property is that E[GX∣G] = GE[X∣G] if G is G-measurable. On the other hand, if Z is independent of (X, Y) then E[X∣Y, Z] = E[X∣Y]. For further properties, see Williams (). In particular, if X and Y are independent then E[X∣Y] = E[X], i.e., it is a constant. For normal random variables, the geometric picture completely characterizes what we can do with them. Recall that a random variable X is centred normal if it has finite variance σ and if for all constants a, b there is a constant c such that cX has the same distribution as aX ′ + bX ′′ where X ′ , X ′′ are independent copies of X. It follows that a + b = c and that X has density proportional to e−x /σ . We say that X is normal if X − E[X] is centred normal. We say that a collection of random variables {Xt }t∈T , with T being an arbitrary set, is (jointly) normal if for any t , . . . , tn , and any constants c , . . . , cn , the random variable c Xt +⋯+cn Xtn is normal. It follows that if {X, Y , . . . , Yk } are jointly normal then E[X∣Y , . . . , Yk ] = E[X∣σ(Y , . . . , Yk )] = a Y + ⋯ + ak Yk + b where the constants can be easily computed by (). The Kalman filter property says that if {X, Y , Y } are centred jointly normal such that Y and Y are independent then E[X∣Y , Y ] = E[X∣Y ] + E[X∣Y ]. The geometric interpretation of this is: to project a vector onto a plane defined by two orthogonal lines, we project to each line and then add the projections. The Kalman filter is one of the important applications of Probability to the fields of Signal Processing, Control, Estimation, and Inference (Catlin ). By the term conditioning in Probability we often mean an effective application of the tower property in order to define a probability measure or to compute the expectation of a functional. For example, if X , X , . . . are i.i.d. positive random variables and an N is a geometric random variable, say P(N = n) = α n− ( − α), n = , , . . ., then E[θ X +⋯+XN ] = E[E(θ X +⋯+XN ∣N)]. But E[θ X +⋯+XN ∣N = n] = E[θ X +⋯+Xn ] = (E[θ X ])n , by independence. Hence E[θ X +⋯+XN ∣N] = (E[θ X ])N and so E[θ X +⋯+XN ] = E[(E[θ X ])N ] = ( − α)/( − αE[θ X ]). Conditional expectation and probability are used in defining various classes of 7stochastic processes such as 7martingales and 7Markov chains (Williams ). Conditional probability is a fundamental object in 7Bayesian statistics (Williams ). Other applications are in the field of Financial Mathematics where the operation of taking conditional expectation of a future random variable with respect to the sigma-algebra of all events prior to the
C
C
Confidence Distributions
current time t plays a fundamental role. In fact, it can be said that the notion of conditioning, along with that of independence and coupling, are the cornerstones of modern probability theory and its widespread applications.
About the Author For biography see the entry 7Radon–Nikodým Theorem.
Cross References 7Bayes’ Theorem 7Bayesian Statistics 7Bivariate Distributions 7Expected Value 7Radon–Nikodým Theorem
References and Further Reading Catlin D () Estimation, control, and the discrete Kalman filter. Springer, New York Kallenberg O () Foundations of modern probability, nd edn. Springer, New York Kolmogorov A () Grundbegriffe der Wahrscheinlichkeitsrechnung. Julius Springer, Berlin (English translation by Chelsea, New York, ) Williams D () Probability with Martingales. Cambridge University Press, Cambridge Williams D () Weighing the odds: a course in probability and statistics. Cambridge University Press, Cambridge
Confidence Distributions William E. Strawderman Professor Rutgers University, Newark, NJ, USA
A Confidence Distribution (CD), H(X, θ), for a parameter is a function of the data, X, and the parameter in question, θ, such that: (a) for each data value X, H(X, .) is a (continuous) cumulative distribution function for the parameter, and (b) for the true parameter value, θ , H(., θ ) has a uniform distribution (see 7Uniform Distribution in Statistics) on the interval (,). The concept of CD has its historic roots in Fisher’s fiducial distribution (Fisher ), although, in its modern version, it is a strictly frequentist construct (Schweder and Hjort , ; Singh et al. , , and see also Efron ). The CD carries a great deal of information pertinent to a variety of frequentist inferences and may be used for the construction of confidence intervals, tests of hypotheses and point estimation.
For instance, the αth quantile of H(X, .), is the upper end of a ( − α) percent one sided confidence interval for θ, and also the interval formed by the sth and tth quantiles (s < t) is a (t − s) percent confidence interval. These properties indicate that a confidence distribution is, in a sense, a direct frequentist version of Fisher’s fiducial distribution. Similarly, a level α test of the one-sided hypothesis K : θ ≤ θ versus K : θ > θ is given by rejecting K when H(X, θ ) ≤ α, and an analogous result holds for testing K : θ ≥ θ versus K : θ < θ . Additionally, for testing the two sided hypothesis K : θ = θ versus K : θ ≠ θ , the rejection region {min(H(X, θ ), − H(X, θ ))} ≥ α gives an α level test. The CD may also be used in a natural ways to construct a point estimate of θ. Perhaps the most straightforward estimator is the median of H(X, .), which is median unbiased, and under mild conditions, consistent. Another obvious estimator, θ¯ =∫ θ(∂H(X,θ)/∂θ)dθ is also consistent under weak conditions. One particularly simple way to construct a CD is via a pivotal quantity, ψ(X, θ), a function of X and θ whose cumulative distribution function, G(.) under the true θ does not depend on θ. Then G(Ψ(X, θ)) is a CD provided Ψ(X, θ) is increasing in θ. Such quantities are easy to construct in invariant models such as location or scale models. Here is a prototypical example in a normal location model. Suppose Xi ∼ N(θ, ), for i = , ⋯, n, are iid. Then Ψ(X , ⋯, Xn , θ) = (X − θ) ∼ N(, /n) so that √ H(X , ⋯, Xn , θ) = Φ− ( n(θ − X)) is a CD. Another common construction is based on a series of one sided α-level tests of K : θ ≤ θ versus K : θ > θ . If the function P[θ , X] is a p-value for each value of θ , then typically P[θ , .] has a uniform distribution for each value of θ , and hence H(X, θ) = P[θ, X] is a CD. The above discussion can be extended naturally to include the notion of an asymptotic CD by replacing (b) above, with the requirement that H(., θ ) approaches a uniform distribution on (, ) weakly as the sample size approaches infinity, and dropping the continuity requirement in (a). Profile likelihoods (see, e.g., Efron ; Schweder and Hjort ; Singh et al. ), and Bootstrap Distributions (see Efron ; Singh et al. , ) are asymptotic CD’s under weak conditions. It can also be extended to include nuisance parameters. For example, in the case of a sample from a normal population with unknown mean and variance, the usual tpivot can be used to construct a CD for the mean, while the usual chi-square pivot can be used to construct a CD for the variance. See Schweder and Hjort (, ) or Singh et al. (, ), for more detailed discussion on construction,
Confidence Interval
properties and uses of CD’s. In particular Singh et al. () discusses the combination of information from independent sources via CD’s.
About the Author William E. Strawderman is Professor of Statistics and Biostatistics at Rutgers University, and past chair of the Department of Statistics. He is a Fellow of IMS and ASA and an elected member if ISI and has served on the councils of IMS and ISBA, as Chair of the ASA Section on Bayesian Statistics. He had been on the editorial boards of the Annals of Statistics, JASA, and the IMS Lecture Notes series. He has won the Distinguished Alumni Award in Science of the Graduate School at Rutgers University, and the ASA’s Youden Award in Interlaboratory Studies.
Cross References 7Bootstrap Methods 7Confidence Interval 7Data Depth 7Fiducial Inference
References and Further Reading Efron B () Bayes and likelihood calculations from confidence intervals. Biometrika :– Efron B () Empirical Bayes methods for combining likelihoods (with discussion), (). J Am Stat Assoc :– Fisher RA () Inverse Probability. Proc Camb Philos Soc : – Schweder T, Hjort NL () Confidence and likelihood. Scand J Stat :– Schweder T, Hjort NL () Frequentist analogues of priors and posteriors. In: Econometrics and the philosophy of economics. Princeton University Press, Princeton, NJ, pp – Singh K, Xie M, Strawderman WE () Combining information from independent sources through confidence distributions. Ann Stat :– Singh K, Xie M, Strawderman WE () Confidence distribution (CD)-distribution estimator of a parameter. In: Regina Liu, William Strawderman, and Cun-Hui Zhang (eds) Complex datasets and inverse problems. IMS Lecture Notes-Monograph Series, , pp –
Confidence Interval Michael Smithson Professor The Australian National University, Canberra, ACT, Australia
A ( − α)% confidence interval is an interval estimate around a population parameter θ that, under repeated random samples of size N, is expected to include θ’s true value
C
( − α)% of the time. The confidence interval thereby indicates the precision with which a population parameter is estimated by a sample statistic, given N and α. For many statistics there are also methods of constructing confidence regions, which are multivariate versions of simultaneous confidence intervals. The confidence level, ( − α)%, is chosen a priori. A two-sided confidence interval uses a lower limit L and upper limit U that each contain θ’s true value (−α/)% of the time, so that together they contain θ’s true value ( − α)% of the time. This interval often is written as [L, U], and sometimes writers combine a confidence level and interval by writing Pr(L ≤ θ ≤ U) = − α. In some applications, a one-sided confidence interval is used, primarily when only one limit has a sensible meaning or when interest is limited to bounding a parameter estimate from one side only. The confidence interval is said to be an inversion of its corresponding significance test because the ( − α)% confidence interval includes all hypothetical values of the population parameter that cannot be rejected by its associated significance test using a Type I error-rate criterion of α. In this respect, it provides more information than a significance test does. Confidence intervals become narrower with larger sample size and/or lower confidence levels. Narrower confidence intervals imply greater statistical power for the corresponding significance test, but the converse does not always hold. The limits L and U are derived from a sample statistic (often the sample estimate of θ) and a sampling distribution specifying a probability for each value that the sample statistic can take. Thus L and U also are sample statistics and will vary from one sample to another. This fact underscores a crucial point of interpretation regarding a confidence interval, namely that we cannot claim that a particular interval has a − α probability of containing the population parameter value. A widespread practice regarding two-sided confidence intervals is to place L and U so that α is evenly split between the lower and upper tails. This is often a matter of convention, but can be dictated by criteria that statisticians have used for determining the “best” possible confidence interval. One such criterion is simply narrowness. It is readily apparent, for instance, that if a sampling distribution is symmetric and unimodal then for high confidence levels the shortest ( − α)% confidence interval constructed from that distribution is one that allocates α/ to the tails outside of the lower and upper limits. Other criteria for evaluating confidence intervals are as follows. A ( − α)% confidence interval is exact if it can be expected to contain the relevant parameter’s true value (−α)% of the time. When approximate intervals
C
C
Confounding and Confounder Control
are used instead, if the rate of coverage is greater than ( − α)% then the interval is conservative; if the rate is less then the interval is liberal. The (−α)% interval that has the smallest probability of containing values other than the true parameter value is said to be uniformly most accurate. A confidence interval whose probability of including any value other than the parameter’s true value is less than or equal to ( − α)% is unbiased. Example Suppose that a standard IQ test has been administered to a random sample of N = adults from a large population with a sample mean of and standard deviation s = . We will construct a two-sided % confidence interval for the mean, µ. The limits U and L must have the property that, given a significance criterion of α, sample size of , mean of and standard deviation of , we could reject the hypotheses that µ > + U or µ < − L but not L ≤ µ ≤ U. The sampling distribution of the t-statistic defined by t =
X−µ is a t-distribution with df = N − = . serr
When df = the value tα/ = . standard-error units above the mean cuts α/ = . from the upper tail of this t-distribution, and likewise −tα/ = −. standard-error units below the mean cuts α/ = . from √ the lower tail. The sample standard error is serr = s/ N = .. So a t-distribution around U = + (.)(.) = . has . of its tail below , while a t-distribution around L = − (.)(.) = . has . of its tail above . These limits fulfill the above required property, so the % confidence interval for µ is [., .]. Thus, we cannot reject hypothetical values of µ that lie between . and ., using α = ..
Example (transforming one interval to obtain another) Cohen’s d for two independent samples is defined by δ = (µ − µ )/σp , where µ and µ are the means of two populations from which the samples have been drawn and σp is the population pooled standard deviation. This quantity has a noncentral t distribution with a noncentrality parameter ∆ = δ[N N /(N + N )]/ , where N and N are the sizes of the two samples. The sample t-statistic is the sample estimate of ∆. Suppose a two-condition between-subjects experiment with N = N = yields t() = .. Using an appropriate algorithm (Smithson ) we can find the % confidence interval for ∆, which is [., .]. Because δ and ∆ are monotonically related by δ = ∆/[N N /(N + N )]/ , we can obtain a % confidence interval for δ by applying this formula to the lower and upper limits of the interval for ∆. The sample estimate of δ is d = t/[N N /(N + N )]/ = ./. =
., and applying the same transformation to the limits of the interval for ∆ gives an interval of [., .] for δ.
About the Author Michael Smithson is a Professor in the Department of Psychology at The Australian National University in Canberra, and received his Ph.D. from the University of Oregon. He is the author of Confidence Intervals (), Statistics with Confidence (), Ignorance and Uncertainty (), and Fuzzy Set Analysis for the Behavioral and Social Sciences (), coauthor of Fuzzy Set Theory: Applications in the Social Sciences (), and coeditor of Uncertainty and Risk: Multidisciplinary Perspectives () and Resolving Social Dilemmas: Dynamic, Structural, and Intergroup Aspects (). His other publications include more than refereed journal articles and book chapters. His primary research interests are in judgment and decision making under uncertainty and quantitative methods for the social sciences.
Cross References 7Confidence Distributions 7Decision Trees for the Teaching of Statistical Estimation 7Effect Size 7Fuzzy Logic in Statistical Data Analysis 7Margin of Error 7Sample Size Determination 7Statistical Fallacies: Misconceptions, and Myths 7Statistical Inference 7Statistical Inference: An Overview
References and Further Reading Altman DG, Machin D, Bryant TN, Gardner MJ () Statistics with confidence: confidence intervals and statistical guidelines, nd edn. British Medical Journal Books, London Smithson M () Confidence intervals. Sage University Papers on Quantitative Applications in the Social Sciences, . Sage, Thousand Oaks
Confounding and Confounder Control Sander Greenland Professor University of California-Los Angeles, Los Angeles, CA, USA
Introduction The word confounding has been used to refer to at least three distinct concepts. In the oldest and most widespread usage, confounding is a source of bias in estimating causal
Confounding and Confounder Control
effects. This bias is sometimes informally described as a mixing of effects of extraneous factors (called confounders) with the effect of interest, and important in causal inference (see 7Causation and Causal Inference). This usage predominates in nonexperimental research, especially in epidemiology and sociology. In a second and more recent usage originating in statistics, confounding is a synonym for a change in an effect measure upon stratification or adjustment for extraneous factors (a phenomenon called noncollapsibility or Simpson’s paradox; see 7Simpson’s Paradox; Collapsibility). In a third usage, originating in the experimental-design literature, confounding refers to inseparability of main effects and interactions under a particular design (see 7Interaction). The three concepts are closely related and are not always distinguished from one another. In particular, the concepts of confounding as a bias in effect estimation and as noncollapsibility are often treated as equivalent, even though they are not. Only the first usage, confounding as a bias, will be described here; for more detailed coverage and comparisons of concepts see, Greenland et al. (a), Pearl (), and Greenland et al. ().
Confounding as a Bias in Effect Estimation In the first half of the nineteenth century, John Stuart Mill described the problem of confounding in causal inference; he acknowledged the seventeenth century scientist Francis Bacon as a forerunner in dealing with these issues (Mill , Chap. III). Mill listed a key requirement for an experiment intended to determine causal relations: 7 “...none of the circumstances [of the experiment] that we do know shall have effects susceptible of being confounded with those of the agents whose properties we wish to study” (emphasis added) (Mill , Chap. X).
In Mill’s time the word “experiment” referred to an observation in which some circumstances were under the control of the observer, as it still is used in ordinary English, rather than to the notion of a comparative trial. Nonetheless, Mill’s requirement suggests that a comparison is to be made between the outcome of our “experiment” (which is, essentially, an uncontrolled trial) and what we would expect the outcome to be if the agents we wish to study had been absent. If the outcome is not as one would expect in the absence of the study agents, then Mill’s requirement ensures that the unexpected outcome was not brought about by extraneous “circumstances” (factors). If, however, those circumstances do bring about the unexpected outcome, and that outcome is mistakenly attributed
C
to effects of the study agents, then the mistake is one of confounding (or confusion) of the extraneous effects with the agent effects. Much of the modern literature follows the same informal conceptualization given by Mill. Terminology is now more specific, with “treatment” used to refer to an agent administered by the investigator and “exposure” often used to denote an unmanipulated agent. The chief development beyond Mill is that the expectation for the outcome in the absence of the study exposure is now almost always explicitly derived from observation of a control group that is untreated or unexposed. Confounding typically occurs when natural or social forces or personal preferences affect whether a person ends up in the treated or control group, and these forces or preferences also affect the outcome variable. While such confounding is common in observational studies, it can also occur in randomized experiments when there are systematic improprieties in treatment allocation, administration, and compliance. A further and somewhat controversial point is that confounding (as per Mill’s original definition) can also occur in perfect randomized trials due to random differences between comparison groups (Fisher ; Rothman ); this problem will be discussed further below.
The Potential-Outcome Model of Confounding Various models of confounding have been proposed for use in statistical analyses. Perhaps the one closest to Mill’s concept is based on the potential-outcome or counterfactual model for causal effects (see 7Causation and Causal Inference). Suppose we wish to consider how a healthstatus (outcome) measure of a population would change in response to an intervention (population treatment). More precisely, suppose our objective is to determine the effect that applying a treatment x had or would have on an outcome measure µ relative to applying treatment x to a specific target population A. For example, this population could be a cohort of breast-cancer patients, treatment x could be a new hormone therapy, x could be a placebo therapy, and the measure µ could be the -year survival probability. The treatment x is sometimes called the index treatment; and x is sometimes called the control or reference treatment (which is often a standard or placebo treatment). The potential-outcome model posits that, in population A, µ will equal µ A if x is applied, µ A if x is applied; the causal effect of x relative to x is defined as the change from µ A to µ A , which might be measured as µ A − µ A , or if µ is strictly positive, µ A /µ A . If A is given treatment x ,
C
C
Confounding and Confounder Control
then µ will equal µ A and µ A will be observable, but µ A will be unobserved. Suppose now that µ B is the value of the outcome µ observed or estimated for a population B that was administered treatment x . If this population is used as a substitute for the unobserved experience of population A under treatment x , it is called the control or reference population. Confounding is said to be present if µ A ≠ µ B , for then there must be some difference between populations A and B other than treatment difference that is affecting µ. If confounding is present, a naive (crude) association measure obtained by substituting µ B for µ A in an effect measure will not equal the effect measure, and the association measure is said to be confounded. Consider µ A − µ B , which measures the association of treatments with outcomes across the populations. If µ A ≠ µ B , then µ A − µ B is said to be confounded for µ A − µ A , which measures the effect of treatment x on population A. Thus, to say an association measure µ A − µ B is confounded for an effect measure µ A − µ A is to say these two measures are not equal.
Dependence of Confounding on the Outcome Measure and the Population A noteworthy aspect of the potential-outcome model is that confounding depends on the outcome measure. For example, suppose populations A and B have a different -year survival probability µ under placebo treatment x ; that is, suppose µ B ≠ µ A so that µ A − µ B is confounded for the actual effect µ A − µ A of treatment on -year survival. It is then still possible that -year survival, υ, under the placebo would be identical in both populations; that is υ A could still equal υ B , so that υ A −υ B is not confounded for the actual effect of treatment on -year survival. Lest one think this situation unlikely, note that we should generally expect no confounding for -year survival, since no known treatment is likely to raise the -year survival probability of human patients above zero. Even though the presence of confounding is dependent on the chosen outcome measure, as defined above its presence does not depend on how the outcome is contrasted between treatment levels. For example, if Y is binary so that µ = E(Y) is the Bernoulli parameter or risk Pr(Y = ), then the risk difference µ A − µ B , risk ratio µ A /µ B , and odds ratio {µ A /( − µ A )}/{µ B /( − µ B )} are all confounded under exactly the same circumstances. In particular, and somewhat paradoxically, confounding may be absent even if the odds ratio changes upon covariate adjustment, i.e., even if the odds ratio is noncollapsible (Greenland and Robins ; Greenland et al. a, ; see 7Collapsibility).
A second noteworthy point is that confounding depends on the target population. The preceding example, with A as the target, had different -year survivals µ A and µ B for A and B under placebo therapy, and hence µ A − µ B was confounded for the effect µ A − µ A of treatment on population A. A lawyer or ethicist may also be interested in what effect the hormone treatment would have had on population B. Writing µ B for the (unobserved) outcome under treatment, this effect on B may be measured by µ B −µ B . Substituting µ A for the unobserved µ B yields µ A − µ B . This measure of association is confounded for µ B − µ B (the effect of treatment x on -year survival in population B) if and only if µ A ≠ µ B . Thus, the same measure of association, µ A −µ B , may be confounded for the effect of treatment on neither, one, or both of populations A and B, and may or may not be confounded for the effect of treatment on other targets such as the combined population A ∪ B.
Confounders (Confounding Factors) and Covariate Imbalance The potential-outcome model is that it invokes no explicit differences (imbalances) between populations A and B with respect to circumstances or covariates that might influence µ. (Greenland and Robins , ). Clearly, if µ A and µ B differ, then A and B must differ with respect to factors that influence µ. This observation has led some authors to define confounding as the presence of such covariate differences between the compared populations (Stone ). This is incorrect, however, because confounding is only a consequence of these covariate differences. In fact, A and B may differ profoundly with respect to covariates that influence µ, and yet confounding may be absent. In other words, a covariate difference between A and B is a necessary but not sufficient condition for confounding, as can be seen when the impact of covariate differences may balance each other out, leaving no confounding. Suppose now that populations A and B differ with respect to certain covariates, and that these differences have led to confounding of an association measure for the effect measure of interest. The responsible covariates are then termed confounders of the association measure. In the above example, with µ A − µ B confounded for the effect µ A − µ A , the factors responsible for the confounding (i.e., the factors that led to µ A ≠ µ B ) are the confounders. It can be deduced that a variable cannot be a confounder unless it can affect the outcome parameter µ within treatment groups and it is distributed differently among the compared populations (e.g., see Yule , who uses terms such as “fictitious association” rather than confounding). These two necessary conditions are sometimes
Confounding and Confounder Control
offered together as a definition of a confounder. Nonetheless, counterexamples show that the two conditions are not sufficient for a variable with more than two levels to be a confounder (Greenland et al. a). Note that the condition of affecting the outcome parameter is a causal assertion and thus relies on background knowledge for justification (Greenland and Robins ; Robins ; Pearl ).
Control of Confounding Prevention of Confounding An obvious way to avoid confounding is estimating µ A − µ A is to obtain a reference population B for which µ B is known to equal µ A . Such a population is sometimes said to be comparable to or exchangeable with A with respect to the outcome under the reference treatment. In practice, such a population may be difficult or impossible to find. Thus, an investigator may attempt to construct such a population, or to construct exchangeable index and reference populations. These constructions may be viewed as design-based methods for the control of confounding. Perhaps no approach is more effective for preventing confounding by a known factor than restriction. For example, gender imbalances cannot confound a study restricted to women. However, there are several drawbacks: restriction on enough factors can reduce the number of available subjects to unacceptably low levels, and may greatly reduce the generalizability of results as well. Matching the treatment populations on confounders overcomes these drawbacks, and, if successful, can be as effective as restriction. For example, gender imbalances cannot confound a study in which the compared groups have identical proportions of women. Unfortunately, differential losses to observation may undo the initial covariate balances produced by matching. Neither restriction nor matching prevents (although it may diminish) imbalances on unrestricted, unmatched, or unmeasured covariates. In contrast, 7randomization offers a means of dealing with confounding by covariates not accounted for by the design. It must be emphasized, however, that this solution is only probabilistic and subject to severe constraints in practice. Randomization is not always feasible or ethical, and many practical problems (such as differential loss and noncompliance) can lead to confounding in comparisons of the groups actually receiving treatments x and x . One somewhat controversial solution to noncompliance problems is intent-to-treat analysis, which defines the comparison groups A and B by treatment assigned rather than treatment received. Confounding may, however, affect even intent-to-treat analyses, and (contrary to widespread misperceptions) the bias in those analyses can
C
exaggerate the apparent treatment effect (Robins ). For example, the assignments may not always be random, as when blinding is insufficient to prevent the treatment providers from protocol violations. And, purely by bad luck, randomization may itself produce allocations with severe covariate imbalances between the groups (and consequent confounding), especially if the study size is small (Fisher ; Rothman ). Blocked (matched) randomization can help ensure that random imbalances on the blocking factors will not occur, but it does not guarantee balance of unblocked factors.
Adjustment for Confounding Design-based methods are often infeasible or insufficient to prevent confounding. Thus, there has been an enormous amount of work devoted to analytic adjustments for confounding. With a few exceptions, these methods are based on observed covariate distributions in the compared populations. Such methods can successfully control confounding only to the extent that enough confounders are adequately measured. Then, too, many methods employ parametric models at some stage, and their success may thus depend on the faithfulness of the model to reality. These issues cannot be covered in depth here, but a few basic points are worth noting. The simplest and most widely trusted methods of adjustment begin with stratification on confounders. A covariate cannot be responsible for confounding within internally homogeneous strata of the covariate. For example, gender imbalances cannot confound observations within a stratum composed solely of women. More generally, comparisons within strata cannot be confounded by a covariate that is unassociated with treatment within strata. This is so, whether the covariate was used to define the strata or not. Thus, one need not stratify on all confounders in order to control confounding; it suffices to stratify on a balancing score (such as a propensity score) that yields strata in which the confounders are unassociated with treatment. If one has accurate background information on relations among the confounders, one may use this information to identify sets of covariates statistically sufficient for adjustment, for example by using causal diagrams or conditional independence conditions (Pearl , ; Greenland et al. ab; Glymour and Greenland ). Nonetheless, if the stratification on the confounders is too coarse (e.g., because categories are too broadly defined), stratification may fail to adjust for much of the confounding by the adjustment variables. One of the most common adjustment approaches today is to enter suspected confounders into a model for the outcome parameter µ. For example, let µ be the mean (expectation) of an outcome variable of interest Y,
C
C
Confounding and Confounder Control
let X be the treatment variable of interest, and let Z be a suspected confounder of the X − Y relation. Adjustment for Z is often made by fitting a generalized-linear model (see 7Generalized Linear Models) g(µ) = g(α + βx + γz) or some variant, where g(µ) is a strictly increasing function such as the natural log ln(µ), as in loglinear modeling, or the logit function ln{µ/( − µ)}, as in 7logistic regression; the estimate of β that results is then taken as the Z-adjusted estimate of the X effect on g(µ). An oft-cited advantage of model-based adjustment is that it allows adjustment for more variables and in finer detail than stratification. If however the form of the fitted model cannot adapt well to the true dependence of Y on X and Z, such model-based adjustments may fail to adjust for confounding by Z. For example, suppose Z is symmetrically distributed around zero within X levels, and the true dependence is g(µ) = g(α + βx + γz ); then using the model g(µ) = g(α + βx + γz) will produce little or no adjustment for Z. Similar failures can arise in adjustments based on models for treatment probability (propensity scores). Such failures can be minimized or avoided by using reasonably flexible models, by carefully checking each fitted model against the data, and by combining treatment-probability and outcome models to produce doubly robust effect estimators (Hirano et al. ; Bang and Robins ). Finally, if (as is often done) a variable used for adjustment is not a confounder, bias may be introduced by the adjustment (Greenland and Neutra ; Greenland et al. b; Hernán et al. ; Pearl ). The form of this bias often parallels selection bias familiar to epidemiologists, and tends to be especially severe if the variable is affected by both the treatment and the outcome under study, as in classic Berksonian bias (Greenland ). In some but not all cases the resulting bias is a form of confounding within strata of the covariate (Greenland et al. b); adjustment for covariates affected by treatment can produce such confounding, even in randomized trials (Cox , Chap. ; Greenland ).
Confounded Mechanisms Versus Confounded Assignments If the mechanism by which the observational units come to have a particular treatment is independent of the potential outcomes of the units, the mechanism is sometimes described as unconfounded or unbiased for µ (Rubin ; Stone ); otherwise the mechanism is confounded or biased. Randomization is the main practical example of such a mechanism. Graphical models (see 7Causal Diagrams) provide an elegant algorithm for checking whether the graphed mechanism is unconfounded within strata
of covariates (Pearl , ; Greenland et al. b; Glymour and Greenland ). Note however that in typical epidemiologic usage the term “confounded” refers to the result of a single assignment (the study group actually observed), not the behavior of the mechanism. Thus an unconfounded mechanism can by chance produce confounded assignments. The latter fact resolves a controversy about adjustment for baseline (pre-treatment) covariates in randomized trials. Although Fisher asserted that randomized comparisons were “unbiased,” he also pointed out that particular assignments could be confounded in the single-trial sense used in epidemiology; see Fisher (, p. ). Resolution comes from noting that Fisher’s use of the word “unbiased” referred to the design and corresponds to an unconfounded assignment mechanism; it was not meant to guide analysis of a given trial (which has a particular assignment). Once the trial is underway and the actual treatment allocation is completed, the unadjusted treatment-effect estimate will be biased conditional on the observed allocation if the baseline covariate is associated with treatment in the allocation and the covariate affects the outcome; this bias can be removed by adjustment for the covariate (Rothman ; Greenland and Robins , ; Greenland et al. a).
Confounder Selection An essential first step in the control of confounding is to identify which variables among those measured satisfied the minimal necessary conditions to be a confounder. This implies among other things that the variables cannot be affected by exposure or outcome; it thus excludes intermediate variables and effects of exposure and disease, whose control could introduce Berksonian bias. This initial screening is primarily a subject-matter decision that requires consideration of the causal ordering of the variables. Relatively safe candidate confounders will be “pre-treatment” covariates (those occurring before treatment or exposure), which have the advantage that they cannot be intermediates or effects of exposure and outcome. Exceptions occur in which control of certain pre-treatment variables introduce bias (Pearl , ; Greenland et al. b), although the bias so introduced may be much less than the confounding removed (Greenland ). Variables that pass the initial causal screening are sometimes called “potential confounders.” Once these are identified, the question arises as to which must be used for adjustment. A common but unjustified strategy is to select confounders to control based on a test (usually a significance test) of each confounder’s association with the treatment X (a test of imbalance) or with the outcome Y, e.g.,
Confounding and Confounder Control
using stepwise regression. Suppose Z is a pre-treatment covariate (potential confounder). The strategy of testing the Z association with X arises from a confusion of two distinct inferential problems: .
.
Do the treated (X = ) evince larger differences from the untreated (X = ) with respect to Z than one should expect from a random (or unconfounded) assignment mechanism? Should we control for Z to estimate the treatment effect?
A test of the X − Z association addresses question (a), but not (b). For (b), the “large-sample” answer is that control is advisable, regardless of whether the X − Z association is random. This is because an imbalance produces bias conditional on the observed imbalance, even if the imbalance derived from random variation. The mistake of significance testing for confounding lies in thinking that one can ignore an imbalance if it is from random variation. Random assignment only guarantees valid performance of statistics over all possible treatment allocations. It does not however guarantee validity conditional on the observed Z imbalance, even though any such imbalance must be random in a randomized trial. Thus the X − Z test addresses a real question (one relevant to a field methodologist studying determinants of response/treatment), but is irrelevant to the second question (b) (Greenland and Neutra ; Robins and Morgenstern ; Greenland et al. a). The case of testing the Z association with Y devolves in part to whether one trusts prior (subject-matter) knowledge that Z affects Y (or is a proxy for a cause of Y) more than the results of a significance test in one’s own limited data. There are many examples in which a well-known risk factor exhibits the expected association with Y in the data, but for no more than chance reasons or sample-size limitations, that association fails to reach conventional levels of “significance” (e.g., Greenland and Neutra ). In such cases there is a demonstrable statistical advantage to controlling Z, thus allowing subject-matter knowledge to over-ride nonsignificance (Robins and Morgenstern ). Another problematic strategy is to select a potential confounder Z for control based on how much the effect estimate changes when Z is controlled. Like the testing methods described above, it also lacks formal justification and can exhibit poor performance in practice (Maldonado and Greenland ). The strategy can also mislead if the treatment affects a high proportion of subjects and one uses a “noncollapsible” effect measure (one that changes upon stratification even if no confounding is present), such as an odds ratio or rate ratio (Greenland and Robins ; Greenland ; Greenland et al. a).
C
In practice, there may be too many variables to control using conventional methods, so the issue of confounder selection may seem pressing. Nonetheless, hierarchicalBayesian or other shrinkage methods may be applied instead. These methods adjust for all the measured confounders by estimating the confounder effects using a prior distribution for those effects. See Greenland (, ) for details. Some of these methods (e.g., the Lasso; Tibshirani ) may drop certain variables entirely, and thus in effect result in confounder selection; unlike significancetesting based selection, however, this selection has a justification in statistical theory.
About the Author Dr. Greenland is Professor of Epidemiology, UCLA School of Public Health, and Professor of Statistics, UCLA College of Letters and Science. He has published over scientific papers, two of which have been cited over times. Professor Greenland is a coeditor (with Kenneth J.h Rothman) of the highly influential text in the field of epidemiology, Modern Epidemiology. This book has received the highest number of citations among all texts and papers in the field, over , (M. Porta et al. ). Book citations: influence of epidemiologic thought in the academich community, Rev Saúde Púbica, , p. ; Lippincott-Raven . Currently, he is an Associate editor for Statistics in Medicine. He was Chair, Section in Epidemiology, American Statistical Association (–). He is Chartered Statistician and Fellow, Royal Statistical Society (), and a Fellow American Statistical Association ().
Cross References 7Bias Analysis 7Causal Diagrams 7Causation and Causal Inference 7Collapsibility 7Exchangeability 7Interaction 7Simpson’s Paradox 7Statistical Methods in Epidemiology
References and Further Reading Bang H, Robins J () Doubly robust estimation in missing data and causal inference models. Biometrics :– Cox DR () Planning of experiments. Wiley, New York Fisher RA () The Design of experiments. Oliver & Boyd, Edinburgh
C
C
Contagious Distributions
Glymour MM, Greenland S () Causal diagrams. In: Rothman KJ, Greenland S, Lash TL (eds) Modern epidemiology, rd edn. Lippincott, Philadelphia, pp – Greenland S () Absence of confounding does not correspond to collapsibility of the rate ratio or rate difference. Epidemiology :– Greenland S () When should epidemiologic regressions use random coefficients? Biometrics :– Greenland S () Quantifying biases in causal models. Epidemiology :– Greenland S () Variable selection and shrinkage in the control of multiple confounders. Am J Epidemiol :–. Erratum Greenland S, Neutra RR () Control of confounding in the assessment of medical technology. Int J Epidemiol : – Greenland S, Robins JM () Identifiability, exchangeability, and epidemiological confounding. Int J Epidemiol :– Greenland S, Robins JM () Identifiability, exchangeability, and confounding revisited. Epidemiol Perspect Innov (online journal) : Greenland S, Robins JM, Pearl J (a) Confounding and collapsibility in causal inference. Stat Sci :– Greenland S, Pearl J, Robins JM (b) Causal diagrams for epidemiologic research. Epidemiology :– Greenland S, Rothman KJ, Lash TL () Measures of effect and measures of association. In: Rothman KJ, Greenland S, Lash TL (eds) Modern epidemiology, rd edn. Lippincott, Philadelphia, pp – Hernán M, Hernandez-Diaz S, Werler MM, Mitchell AA () Causal knowledge as a prerequisite for confounding evaluation. Am J Epidemiol :– Hirano K, Imbens G, Ridder G () Efficient estimation of average treatment effects using the estimated propensity score. Econometrica :– Maldonado G, Greenland S () A simulation study of confounder-selection strategies. Am J Epidemiol : – Mill JS () A system of logic, ratiocinative and inductive. Reprinted by Longmans. Green & Company, London, Pearl J () Causal diagrams for empirical research. Biometrika :– Pearl J () Causality, nd edn. Cambridge University Press, New York Robins JM () Correction for non-compliance in equivalence trials. Stat Med :– Robins JM () Data, design, and background knowledge in etiologic inference. Epidemiology :– Robins JM, Morgenstern H () The foundations of confounding in epidemiology. Comput Math Appl :– Rothman KJ () Epidemiologic methods in clinical trials. Cancer :– Rubin DB () Practical implications of modes of statistical inference for causal effects and the critical role of the assignment mechanism. Biometrics :– Stone R () The assumptions on which causal inference rest. J R Stat Soc B :– Tibshirani R () Regression shrinkage and selection via the Lasso. J R Stat Soc B :– Yule GU () Notes on the theory of association of attributes in statistics. Biometrika :–
Contagious Distributions Seng Huat Ong , Choung Min Ng Professor University of Malaya, Kuala Lumpur, Malaysia The term contagious distribution was apparently first used by Neyman () for a discrete distribution that exhibits clustering or contagious effect. The classical Neyman Type A distribution is one well-known example. However, contagious distributions are used nowadays to describe a plethora of distributions, many of which possess complicated probability distribution expressed in terms of special functions (see, for instance, Johnson et al. ). It is instructive to give an account of the derivation of the Neyman Type A distribution as developed by Neyman () in his paper “On a new class of contagious distributions applicable in entomology and bacteriology.” Neyman wanted to model the distribution of larvae on plots in a field. He assumed that the number of clusters of eggs per unit area, N, followed a Poisson distribution with mean θ denoted by Poi(θ), while the number of larvae developing from the egg clusters Xi , i = , , . . . , N is distributed as another Poisson distribution Poi (ϕ). Mathematically, this may be expressed as follows: SN = X + X + . . . + XN where SN is the total number of larvae per unit area. The distribution of SN is then a Neyman Type A. The above model is known as a true contagion model, where the occurrence of “a favourable event depends on the occurrence of the previous favorable events” (Gurland ). Among other terms used for the distribution arising from this model are generalized, clustered, stopped, or stopped-sum distribution (see Douglas ; Johnson et al. ). It is convenient to represent the distribution of SN concisely by SN ∼ U ∨ U , which reads SN distribution is a U distribution generalized by a U distribution. As an example, the Neyman Type A distribution for SN , above, is Neyman Type A ∼ Poi (θ) ∨ Poi (ϕ). In addition, the probability generating function (pgf) of SN distribution is E[z SN ] = g (g (z)), where gi (z) is the pgf for the corresponding Ui , i = , distribution.
Contagious Distributions
Next, using the same example as above but instead, the number of larvae per unit area is now considered to be distributed as a Poisson distribution, Poi (kθ), where due to heterogeneity, the mean number of eggs that hatched into larvae is assumed to vary with k following a Poisson distribution Poi (ϕ). The distribution for the number of larvae per unit area is again a Neyman Type A. The model that gives rise to a distribution as in the preceding formulation is known as an apparent contagion model. Generally, this distribution arises when a parameter of a U distribution is a random variable Ω that follows a U distribution of its own. This type of distribution is also known as a mixed, mixture, or compound distribution (see Ord ; Johnson et al. ). A compound (mixture) distribution can be represented by U ∧ U , Ω
which means that the U distribution is compounded by U distribution (the distribution of Ω). U is known as the compounding (mixing) distribution. Thus, the Neyman Type A distribution formulated through compounding is represented by Neyman Type A ∼ Poi (kθ) ∧ Poi (ϕ) k
The pgf of a compound (mixture) distribution is ∫ g ( z∣ ω) dF (ω) ω
where g (.) is the pgf for the U and F (.) is the cumulative distribution function for U . The class of mixed Poisson distributions is a well-known class of compound distributions that has found applications in many areas of study including biology, sociology, and medicine. Note that both given examples of contagion models lead to the Neyman Type A distribution. The relationship between the generalized and compound distributions is given by the following theorem: Theorem (Gurland ) Let U be a random variable with pgf [h (z)]θ , where θ is a given parameter. Suppose now θ is regarded as a random variable. Then, whatever be U U ∧ U ∼ U ∨ U .
()
This relation shows that it may not be possible to distinguish between the two types of contagion directly from the data (Gurland ). Contagious distributions have been studied by many researchers including Feller (), Skellam (), Beall and Rescia (), Gurland (), Hinz and Gurland (), Khatri (), and Hill (), creating a rich
C
literature in this field. The readers are referred to Ord (, p. ) for a list of generalized and compound Poisson distributions such as Polya-Aeppli, negative binomial, and Hermite distributions. Other references for generalized and compound distributions can be found in Douglas (, Chaps. and ) and Johnson et al. (, Chaps. and ). These references also describe statistical inference for the contagious distributions. Recent review articles on this subject are Gupta and Ong () and Karlis and Xekalaki ().
Cross References 7Mixture Models 7Multivariate Statistical Distributions
References and Further Reading Beall G, Rescia RR () A generalization of Neyman’s contagious distributions. Biometrics :– Douglas JB () Analysis with standard contagious distributions. International Co-operative Publishing House, Burtonsville Feller W () On a general class of “contagious” distributions. Ann Math Stat :– Gupta RC, Ong SH () Analysis of long-tailed count data by Poisson mixtures. Commun Stat ():– Gurland J () Some interrelations among compound and generalized distributions. Biometrika :– Gurland J () A generalized class of contagious distributions. Biometrics :– Gurland J () Some applications of the negative binomial and other contagious distributions. Am J Public Health :– Hill DH () Response and sequencing errors in surveys: a discrete contagious regression analysis. J Am Stat Assoc :– Hinz P, Gurland J () A test of fit for the negative binomial and other contagious distributions. J Am Stat Assoc : – Johnson NL, Kemp AW, Kotz S () Univariate discrete distributions, rd edn. Wiley, Hoboken Karlis D, Xekalaki E () Mixed Poisson distributions. Int Stat Rev :– Khatri CG () On multivariate contagious distributions. Sankhya :– Neyman J () On a new class of “contagious” distributions, applicable in entomology and bacteriology. Ann Math Stat : – Ord JK () Families of frequency distributions. Griffin, London Skellam JG () Studies in statistical ecology: I. spatial pattern. Biometrika :–
C
C
Continuity Correction
Continuity Correction Rabi Bhattacharya Professor of Mathematics The University of Arizona, Tucson, AZ, USA
random variables X , . . . , Xn , with lattice span , mean µ, variance σ , and finite moments of order at least four. The distribution function Fn (x) of n−/ (X + . . . + Xn ) may then be approximated by the 7Edgeworth expansion (See Bhattacharya and Ranga , p. , or Gnedenko and Kolmogorov , p. ) Fn (x) = Φ σ (x) − n− S (nµ + n x) ϕ σ (x)
According to the central limit theorem (CLT) (see 7Central Limit Theorems), the distribution function Fn of a normalized sum n−/ (X + . . . + Xn ) of n independent random variables X , . . . , Xn , having a common distribution with mean zero and variance σ > , converges to the distribution function Φ σ of the normal distribution with mean zero and variance σ , as n → ∞. We will write Φ for Φ for the case σ = . The densities of Φ σ and Φ are denoted by ϕ σ and ϕ, respectively. In the case Xj′ s are discrete, Fn has jumps and the normal approximation is not very good when n is not sufficiently large. This is a problem which most commonly occurs in statistical tests and estimation involving the normal approximation to the binomial and, in its multi-dimensional version, in Pearson’s frequency 7chi-square tests, or in tests for association in categorical data. Applying the CLT to a binomial random variable T with distribution B(n, p), with mean np and variance npq(q = − p), the normal approximation is given, for integers ≤ a ≤ b ≤ n, by √ √ P(a ≤ T ≤ b) ≈ Φ ((b − np)/ npq) −Φ ((a − np)/ npq). () Here ≈ indicates that the difference between its two sides goes to zero as n → ∞. In particular, when a = b, the binomial probability P(T = b) = Cbn pb qn−b is approximated by zero. This error is substantial if n is not very large. One way to improve the approximation is to think graphically of each integer value b of T being uniformly spread over the interval [b − , b + ]. This is the so called histogram approximation, and leads to the continuity correction given by replacing {a ≤ T ≤ b} by {a − ≤ T ≤ b + } √ ≤ T ≤ b + ) ≈ Φ ((b + − np) / npq) √ − Φ ((a − − np) / npq). ()
P (a −
To give an idea of the improvement due to this correction, let n = , p = .. Then P(T ≤ ) = ., whereas the approximation () gives a probability Φ(−.) = ., and the continuity correction () yields Φ(−.) = .. Analogous continuity corrections apply to the Poisson distribution with a large mean. For a precise mathematical justification of the continuity correction consider, in general, i.i.d. integer-valued
+ n− µ /(σ )( − x /σ )ϕ σ (x) + O(n− ), ()
where S (y) is the right continuous periodic function y − (mod ) which vanishes when y = . Thus, when a is an √ integer and x = (a − nµ)/ n, replacing a by a + (or a − ) on the right side of () gets rid of the discontinuous term involving S . Consider next the continuity correction for the (MannWhitney-)Wilcoxon two sample test (see 7Wilcoxon–Mann– Whitney Test). Here one wants to test nonparametrically if one distribution G is stochastically larger than another distribution F, with distribution functions G(.), F(.). Then the null hypothesis is H : F(x) = G(x) for all x, and the alternative is H : G(x) ≤ F(x) for all x, with strict inequality for some x. The test is based on independent random samples X , . . . , Xm and Y , . . . , Yn from the two unknown continuous distributions F and G, respectively. The test statistic is Ws = the sum of the ranks of the Yj′ s in the combined sample of m + n Xi′ s and Yj′ s. The test rejects H if Ws ≥ c, where c is chosen such that the probability of rejection under H is a given level α. It is known (see Lehmann , pp. –) that Ws is asymptotically normal and E(Ws ) = n(m + n + ), Var(Ws ) = mn(m + n + )/. Since Ws is integer-valued, the continuity correction yields P(Ws ≥ c∣H ) = P (Ws ≥ c − ∣H ) ≈ − Φ(z), () √ where z = (c − − n(m + n + )) / mn(m + n + )/. As an example, let m = , n = , c = . Then P(Ws ≥ ∣ H ) = ., and its normal approximation is − Φ(.) = .. The continuity correction yields the better approximation P(Ws ≥ ∣ H ) = P(Ws ≥ . ∣ H ) ≈ − Φ(.) = .. The continuity correction is also often used in × contingency tables for testing for association between two categories. It is simplest to think of this as a two-sample problem for comparing two proportions p , p of individuals with a certain characteristic (e.g., smokers) in two populations (e.g., men and women), based on two independent random samples of sizes n , n from the two populations, with n = n + n . Let r , r be the numbers in the samples possessing the characteristic. Suppose first that we
Control Charts
wish to test H : p = p , against H : p < p . Consider the test which rejects H , in favor of H , if r ≥ c(r), where r = r + r , and c(r) is chosen so that the conditional probability (under H ) of r ≥ c(r), given r + r = r, is α. This is the uniformly most powerful unbiased (UMPU) test of its size (See Lehmann , pp. –, or Kendall and Stuart , pp. –). The conditional distribution of r , given r + r = r, is multinomial, and the test using it is called Fisher’s exact test. On the other hand, if ni pi ≥ and ni ( − pi ) ≥ (i = , ), the normal approximation is generally used to reject H . Note that the (conditional) expectation and variance of r are n r/n and n n r(n − r)/[n (n − )], respectively (See Lehmann , p. ). The normalized statistic t is then √ t = [r − n r/n]/ n n r(n − r)/[n (n − )],
()
and H is rejected when t exceeds z−α , the ( − α)th quantile of Φ. For the continuity correction, one subtracts from the numerator in (), and rejects H if this adjusted t exceeds z−α . Against the two-sided alternative H : p ≠ p , Fisher’s UMPU test rejects H if r is either too large or too small. The corresponding continuity corrected t rejects H if either the adjusted t, obtained by subtracting from the numerator in (), exceeds z−α/ , or if the t adjusted by adding to the numerator in () is smaller than−z−α/ . This may be compactly expressed as Reject H if V ≡ (n − ) [∣ r n − r n ∣ − n] / (n n r(n − r)) > χ −α (),
()
where χ −α () is the ( − α)th quantile of the 7chi-square distribution with one degree of freedom. This two-sided continuity correction was originally proposed by F. Yates in , and it is known as Yates’ correction. For numerical improvements due to the continuity corrections above, we refer to Kendall and Stuart (, pp. –) and Lehmann (, pp. –). For a critique, see Connover (). If the sampling of n units is done at random from a population with two categories (men and women), then the UMPU test is still the same as Fisher’s test above, conditioned on fixed marginals n ,(and, therefore, n ) and r. Finally, extensive numerical computations in Bhattacharya and Chan () show that the chisquare approximation to the distribution of Pearson’s frequency chi-square statistic is reasonably good for degrees of freedom and , even in cases of small sample sizes, extreme asymmetry, and values of expected cell frequencies much smaller than . One theoretical justification for this may be found in the classic work of Esseen (), which shows
C
that the error of chisquare approximation is O(n−d/(d+) ) for degrees of freedom d.
Acknowledgments The author acknowledges support from the NSF grant DMS .
About the Author For biography see the entry 7Random Walk.
Cross References 7Binomial Distribution 7Chi-Square Test: Analysis of Contingency Tables 7Wilcoxon–Mann–Whitney Test
References and Further Reading Bhattacharya RN, Chan NH () Comparisons of chisquare, Edgeworth expansions and bootstrap approximations to the distribution of the frequency chisquare. Sankhya Ser A :– Bhattacharya RN, Ranga Rao R () Normal approximation and asymptotic expansions. Wiley, New York Connover WJ () Some reasons for not using Yates’ continuity correction on × contingency tables. J Am Stat Assoc :– Esseen CG () Fourier analysis of distribution functions: a mathematical study of the Laplace–Gaussian law. Acta Math :– Gnedenko BV, Kolmogorov AN () Limit distributions of sums of independent random variables. English translation by K.L. Chung, Reading Kendall MG, Stuart A () The advanced theory of statistics, vol , rd edn. Griffin, London Lehmann EL () Testing statistical hypotheses. Wiley, New York Lehmann EL () Nonparametrics: statistical methods based on ranks. (With the special assistance of D’Abrera, H.J.M.). Holden-Day, Oakland
Control Charts Alberto Luceño Professor University of Cantabria, Santander, Spain
Introduction A control chart is a graphical statistical device used to monitor the performance of a repetitive process. Control charts were introduced by Shewhart in the s while working for Western Electric and Bell Labs and, since then, they have been routinely used in Statistical Process Control (SPC). According to Shewhart, control charts are useful to define the standard to be attained for a process, to help
C
C
Control Charts
attaining that standard, and to judge whether that standard has been reached.
Variability and Its Causes Any manufacturing or business process shows some degree of variability. This is obviously true when little effort has been made to try to keep the process stable around a target, but it continues to be true even when a lot of effort has already been dedicated to stabilize the process. In other words, the amount of variability can be reduced (as measured, for example, by the output standard deviation), but cannot be eliminated completely. Therefore, some knowledge about the types of variability that can be encountered in practice and the causes of this variability is necessary. Concerning the types of variability, one must recognize at least the difference between stationary and nonstationary behavior, the former being desirable, the latter undesirable. A stationary process has fixed mean, variance and probability distribution, so that it is difficult (if not impossible) to perfectly attain this desirable state in practice. A non-stationary process does not have fixed mean, variance or probability distribution, so that its future behavior is unpredictable. Moreover, any natural process, when left to itself, tends to be non-stationary, sometimes in the long run, but most often in the short run. Consequently, some control effort is almost always necessary to, at least, induce stationarity in the process. Control charts are useful for this purpose. Concerning the causes of variability, the most obvious facts are that there are a lot of causes, that many of them are unknown, and, consequently, that they are difficult to classify. Nevertheless, Shewhart suggested that it is conceptually useful to classify the causes of variability in two groups: common causes and special causes. Common causes are those that are still present when the process has been brought to a satisfactory stationary state of control; they can be described as chance variation, because the observed variation is the sum of many small effects having different causes. Special causes are those that have larger effects and, hence, have the potential to send the process out of control; hopefully, they can eventually be discovered (assigned) and permanently removed from the system. Control charts are useful tools to detect the presence of special causes of variation worthy of removal. They do so by modelling the likely performance of a process under the influence of the common causes of variation, so that the unexpected behavior (and possible non-stationarity) of the process caused by the emergence of a special cause at any time can be detected efficiently.
Shewhart Charts When Shewhart presented his control charts, he did not claim any mathematical or statistical optimality for such charts, but he did demonstrate that the cost of controlling a process could often be reduced by using control charts. Consequently, Shewhart control charts are much more justifiable for their practical benefits than for their theoretical properties.
A Basic Chart Bearing this in mind, a Shewhart control chart for a measurable quality characteristic is constructed in the following way. () Select the frequency of sampling and the sample size; e.g., take n = observations every h. () Cal¯ t for every time interval t (e.g., culate the sample average X ¯ t versus t for all the values of t at hand. every h) and plot X By doing so, one obtains a run chart. () Add a center line (CL) to the run chart. The ordinate of this horizontal line can be a target value for the quality characteristic, a historical mean of past observations, or simply the mean of the observations at hand. () Add an upper control limit (UCL) and a lower control limit (LCL). These horizontal lines are usually situated symmetrically around the CL and at a distance of three times the standard deviation of the statistics plotted in the run chart (e.g., three times the ¯ t ). standard deviation of X This chart is used at every time interval t to take the decision of whether the process should be considered to be in a state of economic control or not. The usual decision rule is: () Decide that the process stays in control at time ¯ t ) lies between the UCL and the t if the plotted statistics (X LCL, and continue plotting. () Declare an out of control situation otherwise; in this case, a search for an assignable cause should be started, which hopefully will eventually lead to the identification of this cause and its permanent removal from the system. This type of control procedure is sometimes called process monitoring, or process surveillance, and is a part of SPC. Figure shows a Shewhart chart ¯ t having mean µ = for a random sample of values of X and standard deviation σ = . The chart does not show any alarm.
Some Modifications of the Basic Chart Under certain theoretical assumptions, the basic chart can claim some type of optimality. However, it may not be completely satisfactory in practice. Consequently, the form of the basic chart and how it is used can be modified in many different ways. For example, the control limits could not be symmetrically placed around the CL or could not necessarily lie at three standard deviations from the CL.
Control Charts
¯ chart, previously described, it in addition to the basic X is customary to simultaneously run a chart to control the range (R-chart) or the standard deviation (S-chart) of the observations taken every time interval t. Similarly, when the quality characteristic is not measurable, one can use a p-chart or an np-chart to control the fraction nonconforming for each time interval t, or a c-chart or a u-chart to control the total numbers (counts) of nonconforming items for each period t.
Warning limits situated at two standard deviations from the CL could also be plotted. Lines at one standard deviation from the CL could be added. The decision rule could correspondingly be modified using, for example, the so called Western Electric rules, etc. The usefulness of these modifications of the basic chart should be judged, in each particular application, on the bases of the economical or practical advantages they provide. In doing so, the costs of declaring that the process is in control when in fact is not, and vice versa, usually play a role. The elapsed time since the process starts to be out of control until this state is detected can also play a role (true alarm), as well as the time between consecutive declarations of out of control situations when the process stays in control (false alarm rate). These elapsed random times are usually called run lengths (RLs) and their means are called average run lengths (ARLs). Clearly, the frequency distribution (or probability distribution) of the RL will depend on whether the process is in control (RL for false alarms) or out of control (RL for true alarms), and the ARL for false alarms should be much larger than the ARL for true alarms.
Some Other Types of Control Charts Basic Shewhart charts are useful to detect relatively large and sporadic deviations from the state of control. However, the control of a process may be jeopardized also by small but persistent deviations from the state of control. The Western Electric rules may be considered as one of many attempts to tackle this problem. However, a more formal approach was suggested by Page (, ) by introducing the cumulative sum (CUSUM) charts. Moreover, the introduction of the exponentially weighted moving average (EWMA) charts provided an alternative procedure. More recently, cumulative score (CUSCORE) charts, specialized in detecting particular types of deviation from the state of control, have also been suggested (e.g., by Box and Ramírez ; Box and Luceño ; Box et al. ).
Some More Basic Charts Control of the mean of a measurable quality characteristic is important, but a process can also be out of control because of excessive variation around its mean. Therefore,
A Shewhart control chart UCL
56
54
52 CL
50
48
46 LCL
44 5
10
15
Control Charts. Fig. An example of a Shewhart chart
20
C
25
30
35
40
45
50
C
C
Control Charts
A one−sided CUSUM chart 9 8
Decision interval
7 6 5 4 3 2 1 0 5
10
15
20
25
30
35
40
45
50
Control Charts. Fig. An example of a one-sided CUSUM chart for the same data as in Fig.
CUSUM Charts To be able to efficiently detect small persistent deviations from target occurring before and at period t, some use of recent observations is necessary. CUSUM charts do so by using the following statistics:
+
EWMA Charts EWMA charts use recent data in a different way than CUSUM charts. The EWMA statistic is
S+t = max [S+t− + (X¯ t − k+ ); ] ; ¯ t − k− ); ] ; S−t = max [S−t− + (−X
σ, for the sample used in Fig. . This chart produces an alarm at t = .
()
˜ t = ( − λ)X˜ t− + λX¯ t , X
()
−
where k and k are called reference values. The process is considered to be in control until the period t at which one of the inequalities S+t > h+ or S−t > h− becomes true, where h+ and h− are called decision intervals. At this time, an alarm is declared, and the search for a special cause (or assignable cause, in Deming’s words) should begin. The reference values and decision intervals of the chart are often chosen in the light of the theoretical ARLs that they produce when the process is on target and when the process is out of target by an amount of D times the √ standard deviation of Xt (or, equivalently, D n times the ¯ t ). standard deviation of X If only one of the statistics in () is used, the CUSUM chart is called one-sided; if both are used, the CUSUM is called two-sided. The theoretical evaluation of the run length distributions for two-sided CUSUM charts is considerably more difficult than for their one-sided counterparts. Figure shows a one-sided CUSUM chart based on S+t , with reference value µ + .σ and decision interval at
where < λ < , but most often . ≤ λ ≤ .. The EWMA statistic at time t is an average of all observations taken at time t and before, in which each observation receives a weight that decreases exponentially with its age. In other words, Eq. () can be written as ˜ t = λ[X ¯ t + ( − λ)X¯ t− + ( − λ) X ¯ t− + ⋯ ]. X
()
The smaller the value of λ, the smoother the chart. The process is usually considered to be in control until ˜ t ∣ reaches three times the standard the period t at which ∣X deviation of the EWMA statistic X˜ t . It can be shown that ˜ t is the product of the variance of X ¯ t by a the variance of X factor λ[ − ( − λ)t ]/( − λ), where t = is the origin of the chart. When an alarm is triggered, the search for a special cause should start. Information about the above mentioned charts and many possible variants can be found in the bibliography that follows.
Convergence of Random Variables
About the Author Professor Luceño was awarded Brumbaugh Award of the American Society for Quality jointly with Professor George E.P. Box. He is a co-author (with G.E.P. Box) of the well known text Statistical Control By Monitoring and Feedback Adjustment (John Wiley & Sons, ), and (with G.E.P. Box and M.A. Paniagua-Quiñones) ñ (John Wiley & Sons, ). He is currently Associate Editor of Quality Technology and Quantitative Management, and Quality Engineering.
Cross References 7Acceptance Sampling 7Industrial Statistics 7Multivariate Statistical Process Control 7Six Sigma 7Statistical Quality Control 7Statistical Quality Control: Recent Advances
References and Further Reading Box GEP, Luceño A () Statistical control by monitoring and feedback adjustment. Wiley, New York Box GEP, Ramírez JG () Cumulative score charts. Qual reliab Eng Int :– Box GEP, Luceño A, Paniagua-Quiñones MA () Statistical control by monitoring and adjustment, nd edn. Wiley, New York Deming WE () Out of the crisis. Massachusetts Institute of Technology, Center for Advanced Engineering Studies, Cambridge Khattree R, Rao CR (eds) () Handbook of statistics : statistics in industry. Elsevier, Amsterdam Luceño A, Cofiño AS () The random intrinsic fast initial response of two-sided CUSUM charts. Test :– Luceño A, Puig-Pey J () Evaluation of the run-length probability distribution for CUSUM charts: assessing chart performance. Technometrics :– Montgomery DC () Introduction to statistical quality control, th edn. Wiley, New York NIST/SEMATECH () e-Handbook of statistical methods. http://www.itl.nist.gov/div/handbook/ Page ES () Continuous inspection schemes. Biometrika : – Page ES () On problems in which a change in a parameter occurs at an unknown point. Biometrika :– Ryan TP () Statistical methods for quality improvement. Wiley, New York Ruggery F, Kenetts RS, Faltin FW (eds) () Encyclopedia of statistics in quality and reliability. Wiley, New York Shewhart WA () Economic control of quality of manufacturing product. Van Nostrand Reinhold, Princeton, NJ. Republished by Quality Press, Milwaukee, Western Electronic Company () Statistical quality control handbook. Western Electric Corporation, Indianapolis
C
Convergence of Random Variables Pedro J. Rodríguez Esquerdo Professor, Head University of Puerto Rico, San Juan, Puerto Rico
Introduction The convergence of a sequence of random variables (RVs) is of central importance in probability theory and in statistics. In probability, it is often desired to understand the long term behavior of, for example, the relative frequency of an event, does it converge to a number? In what sense does it converge? In statistics, a given estimator often has the property that for large samples the values it takes are distributed around and are close to the value of the desired parameter. In many situations the distribution of this estimator can be approximated by a well known distribution, which can simplify the analysis. Thus it is necessary to understand the types of convergence of such sequences, and conditions under which they occur. Four modes of convergence are presented here. . Weak convergence, also called convergence in distribution or convergence in law, refers to the conditions under which a sequence of distribution functions converges to a cumulative distribution function (cdf). . A second mode is convergence in probability, which studies the limiting behavior of the sequence of probabilities that for each n, a RV deviates by more than a given quantity from a limiting RV. . Convergence with probability one, or almost sure convergence, studies the conditions under which the probability of a set of points in the sample space for which a sequence of RVs converges to another RV is equal to one. . Convergence in the rth mean refers to the convergence of a sequence of expected values. As it is to be expected, there are some relations between the different modes of converge. The results here are explained, for their formal proof, the reader is referred to the included references. In general, the RVs {Xn } cannot be assumed to be independent or identically distributed. For each value of the subscript n, the distribution of Xn may change (Casella and Berger ). In many cases, however, the sequence of dfs converge to another df. A large amount of literature exists on the convergence of random variables. An excellent reference for understanding the definitions and relations is Rohatgi (). For
C
C
Convergence of Random Variables
a discussion of some of these modes of convergence and as they apply to statistics, see Casella and Berger (). Chow and Teicher (), Loeve () and Dudley () present a more formal and general approach to the concept of convergence of random variables. In this paper, the notation {Xn } is used to represent the sequence X , X , X , . . .
Convergence in Probability Let {Xn } be a sequence of RVs defined on a sample space (Ω, F, P). The sequence {Xn } is said to converge in probp
ability to a RV X, denoted by Xn Ð→ X, if for every real number ε > , lim P{∣Xn − X∣ > ε} = .
n→∞
Convergence in Distribution Let {Xn } be a sequence of RVs defined on a sample space (Ω, F, P), and let {Fn } be the corresponding sequence of cdfs. Let X be a RV with cdf F. The sequence {Xn } is said to converge in distribution to X if lim Fn (x) = F(x) at every n→∞
point where F(x) is continuous. This type of convergence is sometimes also called convergence in law and denoted L
Xn Ð→ X. A sequence of distribution functions does not have to converge, and when it does: .
.
.
.
.
The limiting function does not have to be a cdf itself. Consider the sequence given by Fn (x) = if x < n and Fn (x) = if x ≥ n; n = , , . . . Then, at each real value x, Fn (x) → , which is not a cdf. Convergence in distribution does not imply that the sequence of moments converges. For n = , , . . . , consider a sequence of cdfs {Fn } defined by Fn (x) = , if x < ; Fn (x) = − /n, for ≤ x < n; and Fn (x) = for x ≥ n. The sequence of cdfs converges to the cdf F(x) = for x ≥ , and F(x) = otherwise. For each n, the cdf Fn corresponds to a discrete RV Xn that has probability function (pf) given by P{Xn = } = − /n and P{Xn = n} = /n. The limiting cdf F, corresponds to a RV X with pf P(X = ) = . For k ≥ , the kth moment of Xn is E (Xnk ) = ( − /n) + nk (/n) = nk− . Finally, E(X k ) = , so that E (Xnk ) does not converge to E(X k ). Convergence in distribution does not imply convergence of their pfs or probability density functions (pdfs). Let a and b be fixed real numbers, and {Xn } a sequence of RVs with pfs given by P{Xn = x} = for x = b + a/n and P{Xn = x} = otherwise. None of the pfs assigns any probability to the point x = b. Then P{Xn = x} → , which is not a pf, but the sequence of cdfs {Fn } of the RVs Xn converges to a cdf, F(x) = for x ≥ b and F(x) = otherwise. For integer valued RVs, its sequence of pfs converges to another pf if and only if the corresponding sequence of RVs converges in distribution. If a sequence of RVs {Xn } converges in distribution to X and c is a real constant, then {Xn + c}, and {cXn } converge in distribution to {X + c}, and {cX}, respectively.
Convergence in probability of {Xn } to the RV X refers to the convergence of a sequence of probabilities, real numbers to . It means that the probability that the distance between Xn and X is larger than ε > tends to as the n increases to infinity. It does not mean that given ε > , we can find N such that ∣Xn − X∣ < ε for all n ≥ N. Convergence in probability, behaves in many respects as one would expect with respect to common arithmetic operations and under continuous transformations. The following results hold (Rohatgi ): p
p
. Xn Ð→ X if and only if Xn − X Ð→ . p
p
. If Xn Ð→ X and Xn Ð→ Y, then P{X = Y} = . p
p
. If Xn Ð→ X, then Xn − Xm Ð→ , as n, m → ∞. p p p . If Xn Ð→ X and Yn Ð→ Y, then Xn + Yn Ð→ X + Y, p and Xn − Yn Ð→ X − Y. p p . If Xn Ð→ X and k is a real constant then kXn Ð→ kX. p p . If Xn Ð→ k then Xn Ð→ k . p p . If Xn Ð→ a and Yn Ð→ b; a, b real constants, then p Xn Yn Ð→ ab. p p . If Xn Ð→ then /Xn Ð→ . p
p
. If Xn Ð→ a and Yn Ð→ b; a, b real constants, b ≠ , p then Xn /Yn Ð→ a/b. p
p
. If Xn Ð→ X and Y is a RV then Xn Y Ð→ XY. p p p . If Xn Ð→ X and Yn Ð→ Y, then Xn Yn Ð→ XY. . Convergence in probability is stronger than converp
gence in distribution; that is, if Xn Ð→ X then L
Xn Ð→ X. . Let k be a real number, then convergence in distribution to k implies convergence in probability to k, that L
p
is, if Xn Ð→ k then Xn Ð→ k. . In general, convergence in distribution does not imply convergence in probability. For an example, consider the identically distributed RVs X, X , X , . . . with sample space {, }, such that for every n, P(Xn = , X = ) = P(Xn = , X = ) = and P(Xn = , X = ) = P(Xn = , X = ) = / . Because X, Xn , are L
identically distributed RVs, Xn Ð→ X, but P{∣Xn − X∣ >/ } ≥ P{∣Xn − X∣ = } = ≠ . (Rohatgi ).
Convergence of Random Variables
C
0.400 x/n 0.350
C
0.300
(0.25 + ε)
(0.25 + ε)
0.250 (0.25 – ε)
0.200
(0.25 – ε)
0.150
0.100 500
1,000
1,500
2,000
n
Ten series of , trials each, of a Binomial (, .) RV X were simulated. The ratio of the running total of successes x, to the number of trials n is plotted for each series. For X/n to converge in probability to . implies for this experiment, that as n increases, for fixed ε, the probability of observing a series outside the interval (. − ε, . + ε), will decrease to zero. It does not mean there is a value N such that all the series that we can possibly simulate n will be found inside the interval for all n > N.
Convergence of Random Variables. Fig. Illustration of convergence in probability
. Convergence in probability does not imply that the kth p
moments converge, that is, Xn Ð→ X does not imply that E (Xnk ) → E(X k ) for any integer k > . This is illustrated by the example in () above.
Almost Sure Convergence Let {Xn } be a sequence of RVs defined on a sample space (Ω, F, P). The sequence {Xn } is said to converge to X with as probability one or almost surely, denoted Xn Ð→ X if P ( lim Xn = X) = .
Figure illustrates the concept of convergence in probability for series of sample means of RVs from a Binomial(, .) distribution. The following results further relate convergence in distribution and convergence in probability. Let {Xn , Yn }, n = , , . . . be a sequence of pairs of random variables, and let c be a real number. L
p
L
. If ∣Xn − Yn ∣ Ð→ and Yn Ð→ Y, then Xn Ð→ Y. L
L
p
. If Xn Ð→ X and Yn Ð→ c, then Xn + Yn Ð→ X + c. This is also true for the difference Xn − Yn . p L L . If Xn Ð→ X and Yn Ð→ c then Xn Yn Ð→ cX (for p c ≠ ) and Xn Yn Ð→ (for c = ). L
p
L
. If Xn Ð→ X and Yn Ð→ c then Xn /Yn Ð→ X/c (for c ≠ ).
n→∞
Almost sure convergence of a sequence of RVs {Xn } to an RV X, means that the probability of the event {ω; lim Xn (ω) = X(ω)} is one (see also 7Almost Sure n→∞
Convergence of Random Variables). That is, the set of all points ω in the sample space Ω, where Xn (ω) converges to X(ω), has probability one. It is not required that the sequence of functions {Xx (ω)} converge to the function X(ω) pointwise, for all ω in the sample space, only that the set of such ω has probability one. . Convergence almost surely implies convergence in probability. If the sequence of random variables {Xn } converges almost surely to X then it converges in probability to X.
C
Convergence of Random Variables
. Skorokhod’s representation theorem shows that if a sequence of RVs {Xn } converges in distribution to an RV X, then there exists a sequence of random variables {Yn }, identically distributed as {Xn } such that {Yn } converges almost surely to a RV Y, which itself is identically distributed as X (Dudley ). . Continuity preserves convergence in distribution, in probability, and almost sure convergence. If Xn converges in any of these modes to X, and f is a continuous function defined on the real numbers, then f (Xn ) converges in the same mode to f (X). . If {Xn } is a strictly decreasing sequence of positive random variables, such that Xn converges in probability to , then Xn converges almost surely to . . Convergence in probability does not imply convergence almost surely. Consider (Casella and Berger ) the sample space given by the interval [, ], and the uniform probability distribution. Consider the RV X(ω) = ω and let {Xn } be defined by X (ω) = ω + I[,] (ω),
X (ω) = ω + I[,/] (ω),
X (ω) = ω + I[/,] (ω), X (ω) = ω + I[/,/] (ω),
X (ω) = ω + I[,/] (ω),
P(Xn = ) = − and P(Xn = n) = for r > . n n (Rohatgi ).
About the Author Dr. Pedro J. Rodríguez Esquerdo is Professor at the Department of Mathematics, College of Natural Sciences and is currently Professor and Head, Institute of Statistics and Computer Information Science, College of Business Administration, both at the University of Puerto Rico, Rio Piedras. He is past Associate Dean for Academic Affairs at the University of Puerto Rico (–). Professor Rodríguez Esquerdo has served on several advisory boards, including the Advisory Board of Gauss Research Laboratory (–) in San Juan, Puerto Rico. He has been a consultant on education, technology, statistics, mathematics, and intellectual property law. He designed and maintains his statistics education web site www.eduesta.org since . Prof. Rodríguez Esquerdo coauthored a book with professors Ana Helvia Quintero and Gloria Vega, Estadística Descriptiva (Publicaciones Puertorriqueñas, ), and is currently participating in a distance learning project in applied mathematics.
X (ω) = ω + I[/,] (ω),
and so on. Here IA (ω) is the indicator function of the set A. Then {Xn } converges in probability to X, but does not converge almost surely since the value Xn (ω) alternates between ω and ω + infinitely often.
Convergence in the rth Mean Definition Let {Xn } be a sequence of RVs defined on a samplespace(Ω, F, P)Thesequence{Xn }issaidtoconverge to X in the rth mean, r ≥ , if E(∣Xn∣r ) < ∞, E(∣X∣r ) < ∞ and lim E(∣Xn − X∣r ) = . n→∞
. When r = we say that {Xn } converges in the mean, while for r = , we say that {Xn } converges in the mean square. . If a sequence {Xn } converges in the rth mean, and s < r, then {Xn } converges in the sth mean. For example, convergence in the mean square implies convergence in the mean. This means that if the variances of a sequence converge, so do the means. . Convergence in the rth mean implies convergence in probability, if {Xn } converges in the rth mean to X, then {Xn } converges in probability to X. However, the converse is not true. For an example, consider the sequence {Xn } with probability function defined by
Cross References 7Almost Sure Convergence of Random Variables 7Binomial Distribution 7Central Limit Theorems 7Ergodic Theorem 7Glivenko-Cantelli Theorems 7Laws of Large Numbers 7Limit Theorems of Probability Theory 7Probability Theory: An Outline 7Random Variable 7Strong Approximations in Probability and Statistics 7Uniform Distribution in Statistics 7Weak Convergence of Probability Measures
References and Further Reading Casella G, Berger RL () Statistical inference. Duxbury, Pacific Grove Chow Y, Teicher H () Probability theory: independence, interchangeability, martingales, rd edn. Springer, New York Dudley RM () Real analysis and probability. Chapman & Hall, New York Loeve M () Probability theory, vol I, th edn. Springer, New York Rohatgi VK () An introduction to probability theory and mathematical statistics. Wiley, New York
Cook’s Distance
Cook’s Distance R. Dennis Cook Professor University of Minnesota, Minneapolis, MN, USA
Introduction Prior to there was little awareness within statistics or the applied sciences generally that a single observation can influence a statistical analysis to a point where inferences drawn with the observation included can be diametrically opposed to those drawn without the observation. The recognition that such influential observations do occur with notable frequency began with the publication of Cook’s Distance, which is a means to assess the influence of individual observations on the estimated coefficients in a linear regression analysis (Cook ). Today the detection of influential observations is widely acknowledged as an important part of any statistical analysis and Cook’s distance is a mainstay in linear regression analysis. Generalizations of Cook’s distance and of the underlying ideas have been developed for application in diverse statistical contexts. Extensions of Cook’s distance for linear regression along with a discussion of surrounding methodology were presented by Cook and Weisberg (). Cook’s distance and its direct extensions are based on the idea of contrasting the results of an analysis with and without an observation. Implementation of this idea beyond linear and 7generalized linear models can be problematic. For these applications the related concept of local influence (Cook ) is used to study the touchiness of an analysis to local perturbations in the model or the data. Local influence analysis continues to be an area of active investigation (see, for example, Zhu et al. ).
Cook’s Distance Consider the linear regression of a response variable Y on p predictors X , . . . , Xp represented by the model Yi = β + β Xi + ⋯ + β p Xip + ε i , where i = , . . . , n indexes observations, the β’s are the regression coefficients and ε is an error that is independent of the predictors and has mean and constant variance σ . This classic model can be represented conveniently in matrix terms as Y = Xβ + ε. Here, Y = (Yi ) is the n × vector of responses, X = (Xij ) is the n × (p + ) matrix of predictor values Xij , including a constant column to account for the intercept β , and ε = (ε i ) is the n × vector
C
of errors. For clarity, the ith response Yi in combination with its associated values of the predictors Xi , . . . , Xip is called the ith case. Let ̂ β denote the ordinary least squares (OLS) estimator of the coefficient vector β based on the full data and let β (i) denote the OLS estimator based on the data after removing the ith case. Let s denote estimator of σ based on the OLS fit of the full dataset – s the residual sum of squares divided by (n − p − ). Cook () proposed to assess the influence of the ith case on ̂ β by using a statistic Di , which subsequently became known as Cook’s distance, that can be expressed in three equivalent ways: Di =
(̂ β −̂ β (i) )T XT X(̂ β−̂ β (i) ) (p + )s
()
=
̂−Y ̂(i) )T (Y ̂−Y ̂(i) ) (Y (p + )s
()
=
ri hi × . p + − hi
()
The first form () shows that Cook’s distance measures the difference between ̂ β and ̂ β (i) using the inverse of the contours of the estimated covariance matrix s (XT X)− of ̂ β and scaling by the number of terms (p + ) in the model. The second form shows that Cook’s distance can be viewed also as the squared length of the difference between the n× ̂ = X̂ vector of fitted values Y β based on the full data and ̂(i) = X̂ the n × vector of fitted values Y β (i) when β is estimated without the ith case. The final form () shows the general characteristics of cases with relatively large values of Di . The ith leverage hi , ≤ hi ≤ , is the ith diagonal of the projection matrix ̂ = HY. It meaH = X(XT X)− X that puts the “hat” on Y, Y sures how far the predictor values Xi = (Xi , . . . , Xip )T for the ith case are from the average predictor value X. If Xi is far from X then the ith case will have substantial pull on the fit, hi will be near its upper bound of , and the second factor of () will be very large. Consequently, Di will be large unless the first factor in () is small enough to compensate. The second factor tells us about the leverage or pull that Xi has on the fitted model, but it does not depend on the response and thus says nothing about the actual fit of the ith case. That goodness of fit information is provided by ri in first factor of (): ri is the Studentized residual for the ith √ case – the ordinary residual for the ith case divided by s − hi . The squared Studentized residual ri will be large when Yi does not fit the model and thus can be regarded as an outlier, but it says nothing about leverage. In short, the
C
C
Copulas
first factor gives information on the goodness of the fit of Yi , but it says nothing about leverage, while the second factor gives the leverage information but says nothing about goodness of fit. When multiplied, these factors combine to give a measure of the influence of the ith case. The Studentized residual ri is a common statistic for testing the hypothesis that Yi is not an outlier. That test is most powerful when hi is small, so Xi is near X, and least powerful when hi is relatively large. However, leverage or pull is weakest when hi is small and strongest when hi is large. In other words, the ability to detect 7outliers is strongest where the outliers tend to be the least influential and weakest where the outliers tend to be the most influential. This gives another reason why influence assessment can be crucial in an analysis. Cook’s distance is not a test statistic and should not by itself be used to accept cases or reject cases. It may indicate an anomalous case that is extramural to the experimental protocol or it may indicate the most important case in the analysis, one that points to a relevant phenomenon not reflected by the other data. Cook’s distance does not distinguish these possibilities.
About the Author Dennis Cook is Full Professor, School of Statistics, University of Minnesota. He served a ten-year term as Chair of the Department of Applied Statistics, and a three-year term as Director of the Statistical Center, both at the University of Minnesota. He has served as Associate Editor of the Journal of the American Statistical Association (–; –; –), The Journal of Quality Technology, Biometrika (–), Journal of the Royal Statistical Society, Seies B (–) and Statistica Sinica (–). He is a three-time recipient of the Jack Youden Prize for Best Expository Paper in Technometrics as well as the Frank Wilcoxon Award for Best Technical Paper. He received the COPSS Fisher Lecture and Award. He is a Fellow of ASA and IMS, and an elected member of the ISI.
Cross References 7Influential Observations 7Regression Diagnostics 7Robust Regression Estimation in Generalized Linear
Models 7Simple Linear Regression
References and Further Reading Illustration The data that provided the original motivation for the development of Cook’s distance came from an experiment on the absorption of a drug by rat livers. Nineteen rats were given various doses of the drug and, after a fixed waiting time, the rats were sacrificed and the percentage Y of the dose absorbed by the liver was measured. The predictors were dose, body weight and liver weight. The largest absolute Studentized residual is max ∣ri ∣ = ., which is unremarkable when adjusting for multiple testing. The case with the largest leverage . has a modest Studentized residual of ., but a relatively large Cook’s distance of . – the second largest Cook’s distance is .. Body weight and dose have significant effects in the analysis of the full data, but there are no significant effects after the influential case is removed. It is always prudent to study the impact of cases with relatively large values of Di and all case for which Di > .. The most influential case in this analysis fits both of these criteria. The rat data are discussed in Cook and Weisberg () and available from the accompanying software.
Acknowledgments Research for this article was supported in part by National Science Foundation Grant DMS-.
Cook RD () Detection of influential observations in linear regression. Technometrics :–. Reprinted in under the same title for the Technometrics Special th Anniversary Issue , – Cook RD () Assessment of local influence (with discussion). J R Stat Soc Ser B :– Cook RD, Weisberg S () Residuals and influence in regression. Chapman & Hall, London/New York. This book is available online without charge from the University of Minnesota Digital Conservancy: http://purl.umn.edu/ Cook RD, Weisberg S () Applied regression including computing and graphics. Wiley, New York Zhu H, Ibrahim JG, Lee S, Zhang H () Perturbation selection and influence measures in local influence analysis. Ann Stat :–
Copulas Carlo Sempi Professor, Dean of the Faculty of Mathematical, Physical and Natural Sciences Università del Salento, Lecce, Italy Copulas were introduced by Sklar in (Sklar ). In a statistical model they capture the dependence structure of the random variables involved, whatever the distribution
Copulas
functions of the single random variables. They also allow the construction of families of bivariate or multivariate distributions. The definition of the notion of copula relies on those of d-box (Definition ) and of H-volume (Definition ). Here, and in the following, we put I := [, ].
Wd (x , . . . , xd ) : = max{, x + ⋅ ⋅ ⋅ + xd − d + } Md (x , . . . , xd ) : = min{x , . . . , xd }.
[a, b] = ∏ [aj , bj ], j= d
Definition For a function H : R → R, the H-volume VH of the d-box [a, b] is defined by VH ([a, b]) := ∑ sign(v) H(v),
d
Π d (x , . . . , xd ) := ∏ xj . j=
●
●
v
where the sum is taken over the d vertices v of the box [a, b]; here ⎧ ⎪ if vj = aj for an even number of indices, ⎪, sign(v) = ⎨ ⎪ ⎪ ⎩−, if vj = aj for an odd number of indices. A function Cd : Id → I is a d-copula if
(a) Cd (x , x , . . . , xd ) = , if xj = for at least one index j ∈ {, , . . . , d}; (b) when all the arguments of Cd are equal to , but for the j-th one, then Cd (, . . . , , xj , , . . . , ) = xj ;
●
Π d is the distribution function of the random vector U = (U , U , . . . , Ud ) whose components are independent and uniformly distributed on I. Md is the distribution function of the vector U = (U , U , . . . , Ud ) whose components are uniformly distributed on I and such that U = U = ⋅ ⋅ ⋅ = Ud almost surely. W is the distribution function of the vector U = (U , U ) whose components are uniformly distributed on I and such that U = − U almost surely.
The importance of copulas for the applications in statistics stems from Sklar’s theorem. Theorem (Sklar ) Let H be a d-dimensional distribution function with margins F , F ,…, Fd , and let Aj denote the range of Fj , Aj := Fj (R) ( j = , , . . . , d). Then there exists a d-copula C, uniquely defined on A × A × ⋅ ⋅ ⋅ × Ad , d
(c) the VCd -volume of every d-box [a, b] is positive, VCd ([a, b]) ≥ . The set of d-copulas (d ≥ ) is denoted by Cd ; in particular, the set of (bivariate) copulas is denoted by C . Property (c) is usually referred to as the “d-increasing property of a d-copula”. Thus every copula is the restriction to the unit cube Id of a distribution function that concentrates all the probability mass on Id and which has uniform margins (and this may also serve as an equivalent definition). It is possible to show that Cd is a compact set in the set of all continuous functions from Id into I equipped with the product topology, which corresponds to the topology of pointwise convergence. Moreover, in Cd pointwise and uniform convergence are equivalent. Every d-copula satisfies the Fréchet–Hoeffding bounds: for all x ,…, xd in I, one has Wd (x , . . . , xd ) ≤ C(x , . . . , xd ) ≤ Md (x , . . . , xd ),
C
Also relevant is the “independence copula”
While Π d and Md are copulas for every d ≥ , Wd is a copula only for d = , although the lower bound provided by () is the best possible.
d
()
where
Definition Let a = (a , a , . . . , ad ) and b = (b , d b , . . . , bd ) be two points in R , with ≤ aj ≤ bj ≤ (j ∈ {, , . . . , d}); the d-box [a, b] is the cartesian product
Definition
C
such that, for all (x , x , . . . , xd ) ∈ R , H(x , x , . . . , xd ) = C (F (t ), F (t ), . . . , Fd (td )) .
()
Conversely, if F , F ,…, Fd are distribution functions, and d
if C is any d-copula, then the function H : R → I defined by () is a d-dimensional distribution function with margins F , F ,…, Fd . For a compact and elegant proof of this result see (Rüschendorf ). The second (“converse”) part of Sklar’s theorem is especially important in the construction of statistical models, since it allows to proceed in two separate steps: ●
●
Choose the one-dimensional distribution functions F , F , … , Fd that describe the behavior of the individual statistical quantities (random variables) X , X , … , Xd that appear in the model. Fit these in () by means of a copula C that captures the dependence relations among X , X , … , Xd .
C
Copulas
These two steps are independent in the sense that, once a copula C has been chosen, any choice of the distribution functions F , F ,…, Fd is possible. It should be stressed that the copula whose existence is established in Sklar’s theorem is uniquely defined only when the distribution functions have no discrete component; otherwise, there are, in general, several copulas that coincide on A × A × ⋅ ⋅ ⋅ × Ad and which satisfy (). This lack of uniqueness may have important consequences when dealing with the copula of random variables (see, e.g., (Marshall )). The introduction of copulas in the statistical literature has allowed an easier way to construct models by proceeding in two separate steps: (i) the specification of the marginal laws of the random variables involved and (ii) the introduction of a copula that describes the dependence structure among these variables. In many applications (mainly in Engineering) this has allowed to avoid the mathematically elegant and easy-to-deal, but usually unjustified, assumption of independence. In view of possible applications, it is important to have at one’s disposal a stock of copulas. Many families of bivariate copulas can be found in the books by Nelsen (), by Balakrishnan and Lai () and Jaworski et al. (). Here we quote only the gaussian, the meta-elliptical (Fang et al. ) and the extreme-value copulas (Ghoudi et al. ). A popular family of copulas is provided by the Archimedean copulas, which, in the two-dimensional case, are represented in the form Cφ (s, t) = φ[−] (φ(s) + φ(t)) , where the generator φ : [, ] → [, +∞] is continuous, strictly decreasing, convex and φ() = , and φ[−] is the pseudo-inverse of φ, defined by φ[−] (t) := φ− (t), for t ∈ [, φ()], and by , for t ∈ [φ(), +∞]. These copulas depend on a function of a single variable, the generator φ; as a consequence, the statistical properties of a pair of random variables having Cφ as their copula are easily computed in terms of φ (Genest and MacKay ; Nelsen ). For the multivariate case the reader is referred to the paper by McNeil and Neˇslehov´a (), where the generators of a such a copula are completely characterized. Notice, however, that the choice of a symmetric copula, in particular of an Archimedean one, means that the random variables involved are exchangeable, if they have the same distribution. The effort to avoid this limitation motivates the recent great interest in the construction of nonsymmetric copulas (see, e.g., Liebscher ()). It must also be mentioned that many methods of construction for copulas have been introduced; here we mention
● ● ● ●
● ● ●
Ordinal sums (Mesiar and Sempi ); Shuffles of Min (Mikusi´nski et al. ) and its generalization to an arbitrary copula (Durante et al. ); The ∗-product (Darsow et al. ) and its generalization (Durante et al. a); Transformations of copulas, Ch (u, v) := h[−] (C(h(u), h(v))), where the function h : I → I is concave (Durante and Sempi ); Splicing of symmetric copulas (Durante et al. b; Nelsen et al. ); Patchwork copulas (De Boets and De Meyer ; Durante et al. ); Gluing of copulas (Siburg and Stoimenov ).
A strong motivation for the development of much of copula theory in recent years has come from their applications in Mathematical Finance (see, e.g., (Embrechts et al. ), in Actuarial Science (Free and Valdez ), and in Hydrology (see, e.g., (Genest and Favre ; Salvadori et al. )).
About the Author Carlo Sempi received his Ph.D. in Applied Mathematics in , University of Waterloo, Canada (his advisor was Professor Bruno Forte). He was Chairman of the Department of Mathematics, Università del Salento (–). Currently, he is Faculty of Mathematical, Physical and Natural Sciences, University of Salento. Professor Sempi has (co-)authored about refereed papers; many of these papers are on Copulas (some written with the leading authorities on this subject (including Abe Sklar and Roger Nelsen). He was the organizer of the conference “Meeting Copulæ: the th anniversary,” Lecce, June .
Cross References 7Bivariate Distributions 7Copulas in Finance 7Copulas: Distribution Functions and Simulation 7Measures of Dependence 7Multivariate Statistical Distributions 7Multivariate Statistical Simulation 7Non-Uniform Random Variate Generations 7Quantitative Risk Management 7Statistical Modeling of Financial Markets
References and Further Reading Balakrishnan N, Lai C-D () Continuous bivariate distributions, nd edn. Springer, New York Darsow W, Nguyen B, Olsen ET () Copulas and Markov processes. Illinois J Math :– De Baets B, De Meyer H () Orthogonal grid constructions of copulas. IEEE Trans Fuzzy Syst :–
Copulas in Finance
Durante F, Sempi C () Copula and semicopula transforms. Int J Math Math Sci :– Durante F, Klement EP, Quesada-Molina JJ (a) Remarks on two product-like constructions for copulas. Kybernetika (Prague) :– Durante F, Kolesárová A, Mesiar R, Sempi C (b) Copulas with given diagonal sections: novel constructions and applications. Int J Uncertain Fuzziness Knowledge-Based Syst : – Durante F, Rodríguez Lallena JA, Úbeda Flores M () New constructions of diagonal patchwork copulas. Inf Sci :– Durante F, Sarkoci P, Sempi C () Shuffles of copulas. J Math Anal Appl :– Embrechts P, Lindskog F, McNeil A () Modelling dependence with copulas and applications to risk management. In: Rachev S (ed) Handbook of heavy tailed distributions in finance, Chapter . Elsevier, Amsterdam, pp – Fang HB, Fang KT, Kotz S () The meta-elliptical distributions with given marginals. J Multivariate Anal :– Free EW, Valdez EA () Understanding relationships using copulas. N Am J Actuar :– Genest C, Favre AC () Everything you always wanted to know about copula modeling but were afraid to ask. J Hydrol Eng ():– Genest C, MacKay J () The joy of copulas: bivariate distributions with uniform marginals. Am Stat :– Ghoudi K, Khoudraji A, Rivest L-P () Propriétés statistiques des copules de valeurs extrêmes bidimensionnelles. Canad J Stat :– Jaworski P, Durante F, Härdle W, Rychlik T (eds) () Copula theory and its applications. Springer, Berlin Liebscher E () Construction of asymmetric multivariate copulas. J Multivariate Anal :– Marshall AW () Copulas, marginals, and joint distributions, In: Rüschendorf L, Schweizer B, Taylor MD (eds) Distributions with fixed marginals and related topics, Institute of Mathematical Statistics, Lecture Notes – Monograph Series vol , Hayward, pp – McNeil AJ Nešlehová J () Multivariate Archimedean copulas, d-monotone functions and l -norm symmetric distributions. Ann Stat :– Mesiar R, Sempi C () Ordinal sums and idempotents of copulas. Mediterr J Math :– Mikusi n´ ski P, Sherwood H, Taylor MD () Shuffles of min. Stochastica :– Nelsen RB () An introduction to copulas, Lecture Notes in Statistics , nd edn. Springer, New York Nelsen RB, Quesada Molina JJ, Rodríguez Lallena JA, Úbeda Flores M () On the construction of copula and quasicopulas with given diagonal sections. Insurance Math Econ :– Rüschendorf L () On the distributional transform, Sklar’s theorem, and the empirical copula process. J Statist Plan Inference :– Salvadori G, De Michele C, Kottegoda NT, Rosso R () Extremes in nature. An approach using copulas, Water Science and Technology Library, vol . Springer, Dordrecht Siburg KF, Stoimenov PA () Gluing copulas. Commun Stat Theory and Methods :– Sklar A () Fonctions de répartition à n dimensions et leurs marges. Publ Inst Stat Univ Paris :–
C
Copulas in Finance Cherubini Umberto Associate Professor of Mathematical Finance, MatematES University of Bologna, Bologna, Italy
Introduction Correlation trading denotes the trading activity aimed at exploiting changes in correlation or more generally in the dependence structure of assets or risk factors. Likewise, correlation risk is defined as the exposure to losses triggered by changes in correlation. The copula function technique, which enables analyzing the dependence structure of a joint distribution independently from the marginal distributions, is the ideal tool to assess the impact of changes in market comovements on the prices of assets and the amount of risk in a financial position. As far as the prices of assets are concerned, copula functions enable pricing multivariate products consistently with the prices of univariate products. As for risk management, copula functions enable assessing the degree of diversification in a financial portfolio as well as the sensitivity of risk measures to changes in the dependence structure of risk factors. The concept of consistency between univariate and multivariate prices and risk factors is very similar, and actually parallel, to the problem of compatibility between multivariate probability distributions and distribution of lower dimensions. In finance, this concept is endowed with a very practical content, since it enables designing strategies involving univariate and multivariate products with the aim of exploiting changes in correlation.
Copulas and Spatial Dependence in Finance Most of the applications of copula functions in finance are limited to multivariate problems in a cross-sectional sense (as econometricians are used to saying), or in a spatial sense (as statisticians prefer). In other words, almost all the applications have to do with the dependence structure of different variables (prices or losses in the case of finance) at the same date. The literature on applications like these is too large to be quoted here in detail, and we refer the reader to the bibliography below and to those in Bouyé et al. () and Cherubini et al. () for more details.
Pricing Applications Standard asset pricing theory is based on the requirement that the prices of financial products must be such that no arbitrage opportunities can be exploited, meaning that no financial strategy can be built yielding positive return
C
C
Copulas in Finance
with no risk. The price consistent with absence of arbitrage opportunities is also known as the “fair value” of the product. The fundamental theorem of finance states that this amounts to assert that there must exist a probability measure, called risk-neutral measure, under which the expected future returns on each and every asset must be zero, or, which is the same, that the prices of financial assets must be endowed with the martingale property, when measured with that probability measure. Then, the price of each asset promising some payoff in the future must be the expected value with respect to the same probability measure. This implies that if the payoff is a function of one risk factor only, the price is the expected value with respect to a univariate probability measure. If the payoff is instead a function of more than one variable, then it must be computed by taking expectations with respect to the joint distribution of the risk factors. Notice that this implies that there must be a relationship of price consistency between the prices of univariate and multivariate products, and more generally there must be compatibility relationships (in the proper meaning of the term in statistics) among prices. This is particularly true for derivative products promising some payments contingent on a multivariate function of several assets. These are the so-called basket derivative products, which are mainly designed on common equity stock (equity derivatives), or insurance policies against default of a set of counterparties (credit derivatives). The same structure may be used for products linked to commodities or actuarial risks. There are also products called “hybrids” that include different risk factors (such as market risk, i.e., the risk of market movements and default of some obligors) in the same product. For the sake of illustration, we provide here two standard examples of basket equity and credit derivatives: Example (Altiplano Note) These are so-called digital products, that is, paying a fixed sum if some event takes place at a given future date T. Typically, the event is defined as a set of stocks or market indexes, and the product pays the fixed sum if all of them are above some given level, typically specified as a percentage of the initial level. The price of this product is of course the joint risk-neutral probability that all the assets be above a specified level at time T : Q(S (T) > K , S (T) > K , . . . , Sm (T) > Km ), where Ki are the levels (so-called strike prices). Consider now that we can actually estimate the marginal distributions from the option markets, so that we can price each Qi (Si (T) > Ki ). As a result, the only reason why one wants to invest in the multivariate digital product above instead of on a set of univariate ones is to exploit changes in correlation among the assets. To put it in other terms, the value
of a multivariate product can increase even if the prices of all the univariate products remain unchanged, and this may occur if the correlation increases. Copula functions are ideal tools to single out this effect. Example (Collateralized Debt Obligation (CDO)) Today it is possible to invest in portfolios of credit derivatives. In nontechnical terms, we can buy and sell insurance (“protection” is the term in market jargon) on the first x% losses on defaults of a set of obligors (called “names”). This product is called − x% equity tranche of a portfolio of credit losses. For the sake of simplicity assume names and a –% equity tranche, and assume that in case of default, each loss is equal to . So, this tranche pays insurance the first time a default occurs (it is also called a first-to-default protection). Again, we can easily recover the univariate probabilities of default from other products, namely the so-called credit default swap (CDS) market. So, we can price the protection for every single name in the basket. The price of the first-todefault must then be compatible with such prices. In fact, with respect to such prices, the multivariate product is different only because it allows to invest in correlation. Again, the equity tranche can increase in value even though the values of single-insurance CDS for all the names remain constant, provided that the correlation of defaults increase. Even in this case, copula functions provide the ideal tool to evaluate and trade the degree of dependence of the events of default.
Risk Management The risk manager faces the problem of measuring the exposure of the position to different risk factors. In the standard practice, he transforms the financial positions in the different assets and markets into a set of exposures (buckets, in jargon) to a set of risk factors (mapping process). The problem is then to estimate the joint distribution of losses L , L , L , . . . , Lk , on these exposures and define a risk measure on this distribution. Typical measures are Value-at-Risk (VaR) and Expected Shortfall (ES) defined as VaR(Li ) ≡ inf(x : Hi (Li ) > −α)
ES ≡ E(Li ∣Li ≥ VaR)
where Hi (.) is the marginal probability distribution of loss Li . The risk measure of the overall portfolio will analogously be k
k
VaR( ∑ Li ) ≡ inf (x : H( ∑ Li ) > − α) i=
i= k
ES ≡ E( ∑ Li ∣Li ≥ VaR) i=
where H(.) is now the probability distribution of the sum of losses. It is clear that the relationship between
Copulas in Finance
univariate and multivariate risk measures is determined by the dependence structure linking the losses themselves. Again, copula functions are used to merge these risk measures together. Actually, if the max(. , .) instead of the sum were used as the aggregation operator, the risk measure would use the copula function itself as the aggregated distribution of losses.
Copula Pricing and Arbitrage Relationships Using copula functions is very easy to recover arbitrage relationships (i.e., consistency, or compatibility relationships) among prices of multivariate assets. These relationships directly stem from links between copula functions representing the joint distribution of a set of events and those representing the joint distribution of the complement sets. A survival copula is defined as Q(S > K , S > K , . . . , Sm > Km) = C(− u , − u , . . . , −um) The relevance of this relationship in finance is clear because it enforces the so-called put-call parity relationships. These relationships establish a consistency link between the price of products paying out if all the events in a set take place and products paying out if none of them take place. Going back to the Altiplano note above, we may provide a straightforward check of this principle. Example (Put-Call Parity of Altiplano Notes) Assume an Altiplano Note like that in Example , with the only difference that the fixed sum is paid if all the assets Si are below (instead of above) the same predefined thresholds Ki . Clearly, the value of the product will be Q(S (T) ≤ K , S (T) ≤ K , . . . , Sm (T) ≤ Km ). Given the marginal distributions, the dependence structure of this product, which could be called put, or bearish, Altiplano should be represented by a copula, while the price of the call or bullish Altiplano in Example should be computed using the survival copula. It can be proved that if this is not the case, one could exploit arbitrage profits (see Cherubini and Luciano ; Cherubini and Romagnoli ).
Copulas and Temporal Dependence in Finance So far, we have described correlation in a spatial setting. The flaw of this approach, and of copula applications to finance in general, is that no consistency link is specified, among prices with the same underlying risk factors, but payoffs at different times. We provide three examples here, two of which extend the equity and credit products cases presented above, while the third one refers to a problem arising in risk management applications. Research on this
C
topic, as far as applications to finance are concerned, is at an early stage, and is somewhat covered in the reference bibliography below. Example (Barrier Altiplano Note) Assume an Altiplano Note with a single asset, but paying a fixed sum at the final date if the price of that asset S remains above a given threshold K on a set of different dates {t , t , t , . . . , tn }. This product can be considered multivariate just like that in Example , by simply substituting the concept of temporal dependence for that of spatial dependence. Again, copula functions can be used to single out the impact of changes in temporal dependence on the price of the product. For some of these products, it is not uncommon to find the so-called memory feature, according to which the payoff is paid for all the dates in the set at the first time that the asset is above the threshold. Example (Standard Collateralized Debt Obligations) (CDX, iTraxx) In the market there exist CDO contracts, like those described in Example above, whose terms are standardized, so that they may be of interest for a large set of investors. These products include “names” representative of a whole market (CDX for the US and iTraxx for Europe), and on these markets people may trade tranches buying and selling protection on –%, –%, and so on, according to a schedule, which is also standardized. So, for example, you may buy protection against default of the first % of the same names, but for a time horizon of or years (the standard maturities are , , and years). For sure you will pay more for the years insurance than for the years insurance on the same risk. How much more will depend on the relationship between the losses which you may incur in the first years and those that you may face in the remaining years. Clearly, temporal dependence cannot be avoided in this case and it is crucial in order to determine a consistency relationship between the price of insurance against losses on a term of years and those on a term of years. This consistency relationship is known as the term structure of CDX (or iTraxx) premia. Example (Temporal aggregation of risk measures) We may also think of a very straightforward problem of temporal dependence in risk management, which arises whenever we want to compute the distribution of losses over different time horizons. An instance in which this problem emerges is when one wants to apply risk measures to compare the performance of managed funds over different investment horizons. The same problem arises whenever we have to establish a dependence structure between risk factors that are measured with different time frequencies.
C
C
Copulas in Finance
To take the typical example, assume you want to study the dependence structure between market risk and credit risk in a portfolio. The risk measures of market risk are typically computed on losses referred to a period of or days, while credit risk losses are measured in periods of months. Before linking the measures, one has to modify the time horizon of one of the two in order to match that of the other one. The typical “square root rule” used in the market obviously rests on the assumption of independent losses with Gaussian distribution, but this is clearly a coarse approximation of reality.
Financial Prices Dynamics and Copulas The need to extend copulas to provide a representation of both spatial and temporal dynamics of financial prices and risk factors has led to the rediscovery of the relationship between 7copulas and the Markov process (see 7Markov Processes) that was first investigated by Darsow et al. (). Actually, even though the Markov assumption may seem restrictive for general applications, it turns out to be consistent with the standard Efficient Market Hypothesis paradigm. This hypothesis postulates that all available information must be embedded in the prices of assets, so that price innovations must be unpredictable. This leads to models of asset prices driven by independent increments, which are Markovian by construction. For these reasons, this approach was rediscovered both for pricing and financial econometrics applications (Cherubini et al. , ; Cherubini and Romagnoli ; Ibragimov ; Chen ). We illustrate here the basic result going back to Darsow et al. () with application to asset prices. We assume a set of {S , S , . . . , Sm } assets and a set of {t , t , t , . . . , tn } dates, and a filtered probability space generated by the j prices and satisfying the usual conditions. Denote Si the price of asset i at time j. First, define the product of two copulas as
A ∗ B(u, v) ≡ ∫
∂A(u, t) ∂B(t, v) dt ∂t ∂t
and the extended concept of “star-product” as A ∗ B(u , u , . . . , um+n− ) um
∂A(u , u , . . . , um− , t) ∂t ∂B(t, um+ , um+ , . . . , um+n− ) × dt ∂t
≡∫
Now, Darsow et al. proved that a stochastic process Si is a first order 7Markov chain if and only if there exists a
j,j+
set of bivariate copula functions Ti , j = , , . . . , n, such that the dependence among {Si , Si , . . . , Sni } can be written as j
j
Gi (ui , ui , . . . , ui ) = Ti, (ui , ui ) j−,j
∗Ti, (ui , ui ) . . . ∗ Ti
(ui , ui )
The result was extended to general Markov processes of order k by Ibragimov (). Within this framework, Cherubini et al. () provided a characterization of processes with independent increments. The idea is to represent the price Sj (or its logarithm) as Sj− + Y j . Assume that the dependence structure between Sj− and Y j is represented by copula C(u, v). Then, the dependence between Sj− and Sj may be written as u
T
j−,j
− − (u, v) = ∫ D C(w, FY (FS,j (v) − FS,j− (w)))dw
where D represents partial derivative with respect to the first variable, FY (.) denotes the probability distribution of the increment, and the distribution FS,k (.) the probability distribution of Sk . The probability distribution of Sk is obtained by taking the marginal
− F(Sj ≤ s) = T j−,j (, v) = ∫ D C(w, FY (s − FS,j− (w)))dw
This is a sort of extension of the concept of convolution to the case in which the variables in the sum are not independent. Of course, the case of independent increments is readily obtained by setting C(u, v) = uv. The copula linking Sj− and Sj becomes in this case u
− − (w))dw T j−,j (u, v) = ∫ FY (FS,j (v) − FS,j−
A well-known special case is u
T j−,j (u, v) = ∫ Φ(Φ− (v) − Φ− (w))dw
with Φ(x) the standard normal distribution, which yields the dependence structure of a Brownian motion (see 7Brownian Motion and Diffusions) upon appropriate standardization. As for pricing applications, Cherubini et al. () applied the framework to temporal dependence of losses and the term structure of CDX premia, and Cherubini and Romagnoli () exploited the model to price barrier Altiplanos. This stream of literature, which applies copulas to modeling stochastic processes in discrete time, casts a bridge to a parallel approach, that directly applies copulas to model dependence among
Copulas: Distribution Functions and Simulation
stochastic processes in continuous time: this is the socalled Lévy copula approach (Kallsen and Tankov ). Both these approaches aim at overcoming the major flaw of copula functions as a static tool and unification of them represents the paramount frontier issue in this important and promising field of research.
About the Author Umberto Cherubini is Associate Professor of Mathematical Finance at the University of Bologna. He is the author or coauthor of about papers and is the coauthor of books, of which two are in Italian and Copula Methods in Finance (with E. Luciano and W. Vecchiato, ), Structured Finance: The Object Oriented Approach (with G. Della Lunga, ), and Fourier Methods in Finance (with G. Della Lunga, S. Mulinacci and P. Rossi, ), all with John Wiley, Finance Series. Chichester, UK.
Cross References 7Copulas 7Copulas: Distribution Functions and Simulation 7Quantitative Risk Management 7Statistical Modeling of Financial Markets
References and Further Reading Bouyé E, Durrleman V, Nikeghbali A, Riboulet G, Roncalli T () Copulas for finance: a reading guide and some applications. Groupe de Recherche, Opérationnelle, Crédit Lyonnais. Working paper Chen X, Wu SB, Yi Y () Efficient estimation of copula-based semiparametric Markov models. Ann Stat (B):– Cherubini U, Luciano E () Bivariate option pricing with copulas. Appl Math Finance :– Cherubini U, Romagnoli S () Computing the volume of n-dimensional copulas. Appl Math Finance ():– Cherubini U, Romagnoli S () The dependence structure of running maxima and minima: results and option pricing applications. Mathematical Finance ():– Cherubini U, Luciano E, Vecchiato W () Copula methods in finance. Wiley Finance Series, Chichester Cherubini U, Mulinacci S, Romagnoli S () A copula-based model of the term structure of CDO tranches. In: Hardle WK, Hautsch N, Overbeck L (eds) Applied quantitative finance. Springer, Berlin, pp – Cherubini U, Mulinacci S, Romagnoli S () A copula-based model of speculative price dynamics. University of Bologna, Berlin Darsow WF, Nguyen B, Olsen ET () Copulas and Markov processes. Illinois J Math :– Embrechts P, Lindskog F, McNeil AJ () Modelling dependence with copulas with applications to risk management. In: Rachev S (ed) A handbook of heavy tailed distributions in finance. Elsevier, Amsterdam, pp – Gregory J, Laurent JP () Basket defaults, CDOs and factor copulas. J Risk ():– Ibragimov R () Copula based characterization and modeling for time series. Economet Theory :–
C
Kallsen J, Tankov P () Characterization of dependence of multidimensional Lévy processes using Lévy copulas. J Multivariate Anal :– Li D () On default correlation: a copula function approach. J Fixed Income :– Patton A () Modelling asymmetric exchange rate dependence. Int Econ Rev ():– Rosemberg J () Non parametric pricing of multivariate contingent claims federal reserve bank of New York staff reports, n. Van Der Goorberg R, Genest C, Werker B () Bivariate option pricing using dynamic copula models. Insur Math Econ :–
Copulas: Distribution Functions and Simulation Pranesh Kumar Professor University of Northern British Columbia, Prince George, BC, Canada
Introduction In multivariate data modelling for an understanding of stochastic dependence the notion of correlation has been central. Although correlation is one of the omnipresent concepts in statistical theory, it is also one of the most misunderstood concepts. The confusion may arise from the literary meaning of the word to cover any notion of dependence. From mathematics point of view, correlation is only one particular measure of stochastic dependence. It is the canonical measure in the world of 7multivariate normal distributions and in general for spherical and elliptical distributions. However empirical research in many applications indicates that the distributions of the real world seldom belong to this class. We collect and present ideas of copula functions with applications in statistical probability distributions and simulation.
Dependence We denote by (X, Y) a pair of real-valued nondegenerate random variables with finite variances σx and σy respectively. The correlation coefficient between X and Y is the σ standardized covariance σxy , i.e., ρ = σxxyσy , ρ ∈ [−, ]. The correlation coefficient is a measure of linear depen-hh dence only. In case of independent random variables, correlation is zero. Embrechts, McNeil and Straumann () have discussed that in case of imperfect linear dependence, i.e., − < ρ < , misinterpretations of correlation are possible. Correlation is not ideal for a dependence measure and
C
C
Copulas: Distribution Functions and Simulation
causes problems when there are heavy-tailed distributions. Independence of two random variables implies they are uncorrelated but zero correlation does not in general imply independence. Correlation is not invariant under strictly increasing linear transformations. Invariance property is desirable for the statistical estimation and significance testing purposes. Further correlation is sensitive to 7outliers in the data set. The popularity of linear correlation and correlation based models is primarily because it is often straightforward to calculate and manipulate them under algebraic operations. For many 7bivariate distributions it is simple to calculate variances and covariances and hence the correlation coefficient. Another reason for the popularity of correlation is that it is a natural measure of dependence in multivariate normal distributions and more generally in multivariate spherical and elliptical distributions. Some examples of densities in the spherical class are those of the multivariate t-distribution and the 7logistic distribution. Another class of dependence measures is rank correlations. They are defined to study relationships between different rankings on the same set of items. Rank correlation measures the correspondence between two rankings and assess their significance. Two commonly used measures of concordance are Spearman’s rank correlation (ρ s ) and Kendall’s rank correlation (τ). Assuming random variables X and Y have distribution functions F and F and joint distribution function F, Spearman’s rank correlation ρ s = ρ(F (X), F (Y)) where ρ is the linear correlation coefficient. If (X , Y ) and (X , Y ) are two independent pairs of random variables from the distribution function F, then the Kendall’s rank correlation is τ = Pr [(X − X )(Y − Y ) > ]−Pr [(X − X )(Y − Y ) < ]. The main advantage of rank correlations over ordinary linear correlation is that they are invariant under monotonic transformations. However rank correlations do not lend themselves to the same elegant variance–covariance manipulations as linear correlation does since they are not moment-based. A measure of dependence like linear correlation summarizes the dependence structure of two random variables in a single number. Scarsini () has detailed properties of copula based concordance measures. Another excellent discussion of dependence measures is by Embrecht et al. (). Let D(X, Y) be a measure of dependence which assigns a real number to any real-valued pair of random variables (X, Y). Then dependence measure D(X, Y) is desired to have properties: (i) Symmetry: D(X, Y)= D(Y, X); (ii) Normalization: − ≤ D(X, Y) ≤ + ; (iii) Comonotonic or Countermonotonic: The notion of comonotonicity in probability theory is
that a random vector is comonotonic if and only if all marginals are non-decreasing functions (or nonincreasing functions) of the same random variable. A measure D(X, Y) is comonotonic if D(X, Y) = ⇐⇒ X, Y or countermonotonic if D(X, Y) = − ⇐⇒ X, Y; (iv) For a transformation T strictly monotonic on the range of X, D(T(X), Y) = D(X, Y), T(X) increasing or D(T(X), Y) = −D(X, Y), T(X) decreasing. Linear correlation ρ satisfies properties (i) and (ii) only. Rank correlations fulfill properties (i)–(iv) for continuous random variables X and Y. Another desirable property is: (v) D(X, Y) = ⇐⇒ X, Y ( Independent). However it contradicts property (iv). There is no dependence measure satisfying properties (iv) and (v). If we desire property (v), we should consider dependence measure ≤ D∗ (X, Y) ≤ + . The disadvantage of all such dependence measures D∗ (X, Y) is that they can not differentiate between positive and negative dependence (Kimeldorf and Sampson ; Tjøstheim ).
Copulas 7Copulas have recently emerged as a means of describ-
ing joint distributions with uniform margins and a tool for simulating data. They express joint structure among random variables with any marginal distributions. With a copula we can separate the joint distribution into marginal distributions of each variable. Another advantage is that the conditional distributions can be readily expressed using the copula. An excellent introduction of copulas is presented in Joe () and Nelsen (). Sklar’s theorem () states that any multivariate distribution can be expressed as the k-copula function C(u , . . . , ui , . . . , uk ) evaluated at each of the marginal distributions. Copula is not unique unless the marginal distributions are continuous. Using probability integral transform, each continuous marginal Ui = Fi (xi ) has a uniform distribution (see 7Uniform Distribution in Statistics) on I ∈[, ] where Fi (xi ) is the cumulative integral of fi (xi ) for the random variable Xi ∈ (−∞, ∞) . The k-dimensional probability distribution function F has a unique copula representation F(x , x , . . . , xk ) = C(F (x ), F (x ), . . . , Fk (xk )) = C(u , u , . . . , uk ). The joint probability density function is written as f (x , x , . . . , xk ) = Π ki= fi (xi ) × c(F (x ), F (x ), . . . , Fk (xk )) where fi (xi ) is each marginal density and coupling is provided by copula density c(u , u , . . . , uk ) = ∂ k C(u , u , . . . , uk )/∂u ∂u . . . ∂uk if it exists. In case of independent random variables, copula density c(u , u , . . . , uk ) is identically equal to one. The importance of the above equation f (x , x , . . . , xk )
Copulas: Distribution Functions and Simulation
is that the independent portion expressed as the product of the marginals can be separated from the function c(u , u , . . . , uk ) describing the dependence structure or shape. The dependence structure summarized by a copula is invariant under increasing and continuous transformations of the marginals. This means that suppose we have a probability model for dependent insurance losses of various kinds. If our interest now lies in modelling the logarithm of these losses, the copula will not change, only the marginal distributions will change. The simplest copula is independent copula Π := C(u , u , . . . , uk ) = u u . . . uk with uniform density functions for independent random variables. Another copula example is the Farlie–Gumbel–Morgenstern (FGM) bivariate copula. The general system of FGM bivariate distributions is given by F(x , x ) = F (x ) × F (x ) [ + ρ ( − F (x )) ( − F (x ))] and copula associated with this distribution is a FGM bivariate copula C(u, v) = uv [ + ρ ( − u) ( − v)]. A widely used class of copulas is Archimedean copulas which has a simple form and models a variety of dependence structures. Most of the Archimedean copulas have closed-form solutions. To define an Archimedean copula, let ϕ be a continuous strictly decreasing convex function from [, ] to [, ∞] such that ϕ() = and ϕ() = ∞ . Let ϕ− be the pseudo inverse of ϕ. Then a k-dimensional Archimedean copula is C(u , u , . . . , uk ) = ϕ− [ϕ(u ) + . . . + ϕ(uk )]. The function ϕ is known as a generator function. Thus any generator function satisfying ϕ() = ; limx→ ϕ(x) = ∞; ϕ′ (x) < ; ϕ′′ (x) > will result in an Archimedean copula. For an example, generator function ϕ(t) = (t −θ − ) /θ, θ ∈ [−, ∞)/{} results in the bivariate Clayton copula C(u , u ) = max [(u−θ + u−θ − )
−/θ
, ]. The copula
parameter θ controls the amount of dependence between X and X . The Frécht–Hoeffding bounds for copulas: The lower bound for k-variate copula is W(u , u , . . . , uk ) := max {− n + ∑ki= ui , } ≤ C(u , u , . . . , uk ). The upper bound for k-variate copula is C(u , u , . . . , uk ) ≤ mini∈{,,...,k} ui =: M(u , u , . . . , uk ). For all copulas, the inequality W(u , . . ., uk )≤ C(u , . . . , uk ) ≤ M(u , . . . , uk ) is satisfied. This inequality is well known as the Frécht-Hoeffding bounds for copulas. Further, W and M are copulas themselves. It may be noted that the Frécht-Hoeffding lower bound is not a copula in dimension k > . Copulas M, W and Π have important statistical interpretations (Nelson, ). Given a pair of continuous random variables (X , X ), (i) copula of (X , X ) is M(u , u ) if and only if each of X and X is almost surely increasing function of the other; (ii) copula of (X , X ) is W(u , u ) if and only if each of X and X
C
is almost surely decreasing function of the other and (iii) copula of (X , X ) is Π(u , u ) = u u if and only if X and X are independent. Three famous measures of concordance Kendall’s τ, Spearman’s ρ s and Gini’s index γ could be expressed in terms of copulas (Schweizer and Wolff ) τ = ∫ ∫I C(u , u ) dC(u , u ) − , ρ s = ∫ ∫I u u dC(u , u ) − and γ = ∫ ∫I (∣u + u − ∣ − ∣u − u ∣) dC(u , u ). It may however be noted that the linear correlation coefficient ρ cannot be expressed in terms of copula. The tail dependence indexes of a multivariate distribution describe the amount of dependence in the upper right tail or lower left tail of the distribution and can be used to analyze the dependence among extremal random events. Tail dependence describes the limiting proportion that one margin exceeds a certain threshold given that the other margin has already exceeded that threshold. Joe () defines tail dependence: If a bivariate copula C(u , u ) is exists, then C(u , u ) such that λ U := limu→ [−u+C(u,u)] (−u) has upper tail dependence for λ U ∈ (, ] and no upper tail dependence for λ U = . Similarly lower tail depen. dence in terms of copula is defined λ L := limu→ C(u,u) u Copula has lower tail dependence for λ L ∈ (, ] and no lower tail dependence for λ L = . This measure is extensively used in extreme value theory. It is the probability that one variable is extreme given that other is extreme. Tail measures are copula-based and copula is related to the full distribution via quantile transformations, i.e., C(u , u ) = F (F− (u ), F− (u )) for all u , u ∈ (, ) in the bivariate case.
Simulation Simulation in statistics has a pivotal role in replicating and analysing data. Copulas can be applied in simulation and Monte Carlo studies. Johnson () discusses methods to generate a sample from a given joint distribution. One such method is a recursive simulation using the univariate conditional distributions. The conditional distribution of Ui given first i − components is i− (u ,...,u i ) ∂ i− C i− (u ,...,u i− ) / ∂u ...∂ui− . For k ≥ , Ci (ui ∣u , . . . , ui− ) = ∂ ∂uCi...∂u i− procedure is as follows: (i) Simulate a random number u from Uniform (,); (ii) Simulate value u from the conditional distribution C (u ∣u );(iii) Continue in this way; (iv) Simulate a value uk from Ck (uk ∣u , . . . , uk− ). We list some important contributions in the area of copulas under the reference section.
Acknowledgments Author wish to thank the referee for the critical review and useful suggestions on the earlier draft. This work was
C
C
Cornish–Fisher Expansions
supported by the author’s discovery grant from the Natural Sciences and Engineering Research Council of Canada (NSERC) which is duly acknowledged.
About the Author Dr. Pranesh Kumar is Professor of Statistics in the University of Northern British Columbia, Prince George, BC, Canada. He has held several international positions in the past: Professor and Head, Department of Statistics, University of Transkei, South Africa; Associate Professor, Bilkent University, Ankara, Turkey; Associate Professor, University of Dar-es-Salaam, Tanzania; Visiting Professor, University of Rome, Italy; Visiting Senior Researcher at the Memorial University of Newfoundland, Canada; Associate Professor, Indian Agricultural Statistics Research Institute, New Delhi. Dr. Kumar has published his research in many prestigious professional journals. He holds membership in several professional international societies including the International Statistical Institute. Dr. Kumar have membership of the editorial boards of the Journal of Applied Mathematics and Analysis, Journal of Mathematics Research, Journal of the Indian Society of Agricultural Statistics and JNANABHA which has reciprocity agreement with the American Mathematical Society. He has (co-)authored about refereed papers.
Cross References 7Copulas 7Copulas in Finance 7Multivariate Statistical Distributions 7Multivariate Statistical Simulation
References and Further Reading Clayton DG () A model for association in bivariate life tables and its application in epidemiological studies of familial tendency in chronic disease incidence. Biometrika : – Cuadras CM, Fortiana J, Rodríguez Lallena JA () Distributions with given marginals and statistical modelling. Kluwer, Dodrecht Embrechts P, McNeil A, Straumann D () Correlation and dependence in risk management: properties and pitfalls. Risk ():– Fang K-T, Kotz S, Ng K-W () Symmetric multivariate and related distributions. Chapman & Hall, London Frank MJ () On the simultaneous associativity of F(x, y) and x + y − F(x, y). Aequationes Mathematicae :– Frécht M () Sue les tableaux de corrélation dont les marges son données. Ann Univ Lyon Sect A :– Genest C () Frank’s family of bivariate distributions. Biometrika :– Genest C, Mackay J () The joy of copulas: bivariate distributions with uniform marginals. Am Stat :–
Genest C, Rivest L () Statistical inference procedures for bivariate Archimedean copulas. J Am Stat Assoc :– Genest C, Ghoudi K, Rivest L () A semi-parametric estimation procedure of dependence parameters in multivariate families of distributions. Biometrika :– Gumbel EJ () Bivariate exponential distributions. J Am Stat Assoc :– Hougaard P () A class of multivariate failure time distributions. Biometrika :– Hutchinson TP, Lai CD () Continuous bivariate distributions emphasizing applications. Rumsby Scientific, Adelaide, South Australia Joe H () Multivariate models and dependent concepts. Chapman & Hall, New York Johnson ME () Multivariate statistical simulation. Wiley, New York Kimeldorf G, Sampson AR () Monotone dependence. Ann Stat :– Marshall AW, Olkin I () Families of multivariate distributions. J Am Stat Assoc :– Nelsen R () An introduction to copulas. Springer, New York Nelsen RB, Quesada Molina JJ, Rodríguez Lallena JA, Úbeda Flores M () Bounds on bivariate distribution functions with given margins and measures of association. Commun Stat Theory Meth :– Scarsini M () On measures of concordance. Stochastica :– Schweizer B, Sklar A () Probabilistic metric spaces. North Holland, New York Schweizer B, Wolff E () On nonparametric measures of dependence for random variables. Ann Stat :– Sklar A () Fonctions de répartition án dimensional et leurs marges. Publ Inst Stat Univ Paris :– Tjøstheim D () Measures of dependence and tests of independence. Statistics :–
Cornish–Fisher Expansions Vladimir V. Ulyanov Professor Lomonosov Moscow State University, Moscow, Russia
Introduction In statistical inference it is of fundamental importance to obtain the sampling distribution of statistics. However, we often encounter situations where the exact distribution cannot be obtained in closed form, or even if it is obtained, it might be of little use because of its complexity. One practical way of getting around the problem is to provide reasonable approximations of the distribution function and its quantiles, along with extra information on their possible errors. This can be accomplished with the help of Edgeworth and Cornish–Fisher expansions. Recently, interest in Cornish–Fisher expansions has increased
Cornish–Fisher Expansions
because of intensive study of VaR (Value at Risk) models in financial mathematics and financial risk management (see Jaschke ()).
∞
x=u−∑
F − (u) = inf{x; F(x) > u}.
Fn (x) = Gk,n (x) + O(є k )
with
Gk,n (x) = G(x) + {єa (x) + ⋅ + є k− ak− (x)}g(x),
r=
r− {−[g(u)]− du } [{zn (u)}r /g(u)] , () r!
where du = d/du and zn (u) = Fn (u) − G(u). A similar relation can be written for u as a function of x. In many statistical applications, Fn (x) is known to have an asymptotic expansion of the form Fn (x) = G(x) + g(x) [n−a p (x) + n−a p (x) + ⋯] , where pr (x) may be polynomials in x and a = / or . Then the formulas () can be written as ∞
x=u−∑
This is a right-continuous nondecreasing function defined on the interval (,) and F(x ) ≥ u if x = F − (u ). Let Fn (x) be a sequence of distribution functions and let each Fn admit the 7Edgeworth expansion (EE) in the powers of є = n−/ or n− :
r=
d(r) {gn (u)}r , r!
()
where qn (u) = n−a p (u) + n−a p (u) + ⋯, m(x) = −g ′ (x)/g(x), d() = the identity operator, d(r) = {m(u) − du }{m(u) − du }⋯{(r − )m(u) − du },
()
where g(x) is the density function of the limiting distribution function G(x). An important approach to the problem of approximating the quantiles of Fn is to use their asymptotic relation to those of G’s. Let x and u be the corresponding quantiles of Fn and G, respectively. Then we have Fn (x) = G(u). () Write x(u) and u(x) to denote the solutions of () for x in terms of u and u in terms of x, respectively [i.e. u(x) = G− (Fn (x)) and x(u) = Fn− (G(u))]. Then we can use the EE () to obtain formal solutions x(u) and u(x) in the form x(u) = u + єb (u) + є b (u) + ⋯
()
u(x) = x + єc (x) + є c (x) + ⋯.
()
and Cornish and Fisher () and Fisher and Cornish () obtained the first few terms of these expansions when G is the standard normal distribution function (i.e., G = Φ). We call both () and () the Cornish–Fisher expansions, (CFE). Concerning CFE for random variables obeying limit laws from the family of Pearson distributions see Bol’shev (). Hill and Davis () gave a general algorithm for obtaining each term of CFE when G is an analytic function:
Theorem Assume that the distribution function G is analytic. Then the following relation for x and u satisfying Fn (x) = G(u) holds:
Expansion Formulas Let X be a univariate random variable with a continuous distribution function F. For α : < α < , there exists x such that F(x) = α, which is called the (lower) α% point of F. If F is strictly increasing, the inverse function F − (⋅) is well defined and the α% point is uniquely determined. We also speak of “quantiles” without reference to particular values of α meaning the values given by F − (⋅). Even in the general case, when F(x) is not necessarily continuous nor is it strictly increasing, we can define its inverse function by the formula
C
r = , , . . . The rth term in () is of order O(n−ra ). It is a tedious process to rewrite () in the form of () and to express the adjustment terms bk (u) directly in terms of the cumulants (see Hill and Davis ()). Lee and Lin developed a recurrence formula for bk (u), which is implemented in the algorithm AS (see Lee and Lin (, )). Usually the CFE are applied in the following form with k = , , or : k−
xk (u) = u + ∑ є j bj (u) + O(є k ),
()
j=
In order to find the explicit expressions for b (u) and b (u) we substitute () with k = to () and using () we have Fn (x) = Fn (u + єb + є b + ⋯) = G(u + єb + є b ) + g(u + єb + є b ) × {єa (u + єb ) + є a (u)} + O(є ). By Taylor’s expansions for G, g, and a , we obtain Fn (x) =G(u) + єg(u){b + a (u)} ′ ′ + є [g(u)b + g (u)b + g(u)a (u)b +g(u)a (u) + g ′ (u)b a (u)] + O(є ),
C
C
Cornish–Fisher Expansions
which should be G(u). Therefore, b = −a (u), ′ ′ b = {g (u)/g(u)}a (u) − a (u) + a (u)a (u). An application of general formulas () in the case of normal limit distribution see the entry 7Edgeworth Expansion. Suppose that fγ [Gf (x) + Gf + (x) + Gf + (x)] n f j − + ∑(−) cj Gf +j (x) + o(n ), n j=
Fn (x) = Gf (x) +
where Gf (x) = Pr {χ f ≤ x}; that is, the distribution function of the 7chi-square distribution with f degrees of freedom, γ is a constant, and cj are constants such that j ∑j= (−) cj = . Then G(u) = Gf (u), g(u) = gf (u) = [Γ( f /) m(u) = −gf′ (u)/gf (u) =
f / −
] u
f /−
f − ( − ) . u
u(u − f − u [c u − n( f + ) n ( f + )( f + )( f + ) + (c − c )( f + )u + (c − c ) ( f + )( f + )u c ( f + )( f + )( f + )] + o(n− ).
Therefore, we obtain x = u − qn (u)−[γ u(u − f − )/{n (f + ) }] × {u − (f + )u + (f + ) } + o(n− ). The upper and lower bounds for the quantiles x = x(u) and u = u(x), satisfying the equation (), i.e. xn (u) ≥ x(u) ≥ x¯ n (u),
Fn (xα ) = Gk,n (x∗α ) = − α. The approximate quantile x∗α based on the Edgeworth expansion is available in numerical form but cannot be expressed in explicit form. Suppose that the remainder term, Rk,n (x) = Fn (x) − Gk,n (x), is such that ∣Rk,n ∣ ≤ є n Ck . Then ∣Fn (x∗α ) − ( − α)∣ = ∣Fn (x∗α ) − Gk,n (x∗α )∣ ≤ є n Ck . This gives an error bound for the absolute differences between the probabilities based on the true quantiles and their approximations. The other validity of the CFE was obtained by considering the distribution function F˜ k,n of k−
˜ = U + ∑ є j bj (U), X
exp(−u/),
Thus, we can write qn (u) = −
Let xα and x∗α be the upper α% points of Fn and Gk,n from (), respectively; that is, they satisfy
un (x) ≥ u(x) ≥ u¯ n (x)
were obtained for some special distributions by Wallace ().
Validity of Cornish–Fisher Expansions In applications, the CFE are usually used in the form (). It is necessary to remember that the approximations for αquantiles provided by the CFE (i) become less and less reliable for α → and α → ; (ii) do not necessarily improve (converge) for a fixed Fn and increasing order of approximation k.
j=
where U is the standard normal variable. Takeuchi and Takemura () showed that if ∣Fn (x) − Gk,n (x)∣ = o(є k− ), then ∣Fn (x) − F˜ k,n ∣ = o(є k− ).
Function of Sample Mean Usually the conditions that are sufficient for validity of EE are sufficient as well for validity of CFE. Under the conditions of section “7Function of Sample Means” in the entry 7Edgeworth Expansion and in its notation we have (see Hall ()): RRR k− b (u ) RRR α R j sup RRRRRxα − uα − ∑ j/ RRRR = o ( (k−)/ ) , n RRR є<α<−є RR j= n R √ ¯ ≤ x) > α}, uα = where xα = inf{x; Pr( nH(Y)/σ − Φ (α), є is any constant in (,/) and b′j s are polynomials depending on Q′j s.
Error Bounds It is possible to get error bounds for approximation given by the CFE provided we have error bounds for EE. For simplicity, we give error bounds for the first-order CFE (see Chap. in Fujikoshi et al. ()): Theorem we have
Suppose that for the distribution function of U F(x) ≡ Pr{U ≤ x} = G(x) + R (x),
where for remainder term R (x) there exists a constant c such that ∣R (x)∣ ≤ d ≡ c є.
Correlation Coefficient
Let xα and uα be the upper α% points of F and G, respectively; that is, Pf{U ≤ xα } = G(uα ) = − α.
C
Takeuchi K, Takemura A () Some results on univariate and multivariate Cornish–Fisher expansion: algebraic properties and validity, Sankhy¯a A :– Wallace DL () Bounds on normal approximations to Student’s and the chisquare distributions. Ann Math Stat :–
C
Then, for any α such that > α > d and > α + d : . uα+d ≤ xα ≤ uα−d . . ∣xα − uα ∣ ≤ d /g(u() ), where g(u ) =
min
u∈[u α+d ,u α−d ]
g(u).
About the Author DSc Vladimir V. Ulyanov is Professor, Department of Mathematical Statistics, Faculty of Computational Mathematics and Cybernetics at Lomonosov Moscow State University. He has received the State Prize of the USSR for Young Scientists (), Alexander von Humboldt Research Fellowship, Germany (–), JSPS Research Fellowship, Japan (, ). He was visiting Professor/Researcher at Bielefeld University, Germany, University of Leiden, Université de Paris V, University of Hong Kong, Institute of Statistical Mathematics in Tokyo, National University of Singapore, the University of Melbourne etc. He is a member of the Bernoulli Society. Professor Ulyanov is the author of more than journal articles and a book Multivariate Statistics: HighDimensional and Large-Sample Approximations (with Y. Fujikoshi and R. Shimizu, John Wiley and Sons, ).
Cross References 7Edgeworth Expansion 7Multivariate Statistical Distributions
References and Further Reading Bol’shev LN () Asymptotically Pearson transformations. Theor Probab Appl :– Cornish EA Fisher RA () Moments and cumulants in the specification of distributions. Rev Inst Int Stat :– Fisher RA, Cornish EA () The percentile points of distributions having known cumulants. J Am Stat Assoc :– Fujikoshi Y, Ulyanov VV, Shimizu R () Multivariate statistics : high-dimensional and large-sample approximations. Wiley Series in Probability and Statistics. Wiley, Hoboken Hall P () The bootstrap and Edgeworth expansion. Springer, New York Hill GW, Davis AW () Generalized asymptotic expansions of Cornish–Fisher type. Ann Math Stat :– Jaschke S () The Cornish–Fisher-expansion in the context of delta-gamma-normal approximations. J Risk ():– Lee YS, Lin TK () Higher-order Cornish–Fisher expansion. Appl Stat :– Lee YS, Lin TK () Correction to algorithm AS: higher-order Cornish–Fisher expansion. Appl Stat :–
Correlation Coefficient Nitis Mukhopadhyay Professor University of Connecticut-Storrs, Storrs, CT, USA
Introduction A covariance term loosely aims at capturing some essence of joint dependence between two random variables. A correlation coefficient is nothing more than an appropriately scaled version of the covariance. Section “Population Correlation Coefficient” introduces the concepts of a covariance and the population correlation coefficient. Section “Correlation Coefficient and Independence” highlights some connections between the correlation coefficient, independence, and dependence. Section “A Sample Correlation Coefficient” summarizes the notion of a sample correlation coefficient and its distribution, both exact and large-sample approximation, due to Fisher (; ). Section “Partial Correlations” gives a brief summary of the concept of partial correlation coefficients.
Population Correlation Coefficient A covariance term tries to capture a sense of joint dependence between two real valued random variables. A correlation coefficient, however, is an appropriately scaled version of a covariance. Definition The covariance between two random variables X and X , denoted by Cov(X , X ), is defined as Cov(X , X ) = E [(X − µ )(X − µ )] or equivalently E [X X ] − µ µ , where µ i = E(Xi ), i = , and E [X X ] , µ , µ are assumed finite. Definition The correlation coefficient between two random variables X and X , denoted by ρ X ,X , is defined as ρ X ,X =
Cov(X , X ) , σ σ
C
Correlation Coefficient
whenever one has < σ = V(X ) < ∞ and < σ = V(X ) < ∞. One may note that we do not explicitly assume −∞ < Cov(X , X ) < ∞. In view of the assumption < σ , σ < ∞, one can indeed claim the finiteness of Cov(X , X ) by appealing to Cauchy–Schwartz inequality. It should also be clear that Cov(X , X ) = Cov(X , X ) and Cov(X , X ) = V(X ), as long as those terms are finite. Two random variables X , X are respectively called negatively correlated, uncorrelated, or positively correlated if and only if ρ X ,X is negative, zero or positive. Theorem Consider random variables X and X and assume that < V(X ), V(X ) < ∞. Then, we have the following results: . Let Yi = ci + di Xi w.p. where −∞ < ci < ∞ and < di < ∞ are fixed numbers, i = , . Then, ρ Y ,Y = ρ X ,X . . ∣ρ X ,X ∣ ≤ , the equality holds if and only if X = a + bX w.p. for some real numbers a and b. More details can be found from Mukhopadhyay (, Sect. .).
Correlation Coefficient and Independence If ρ X ,X is finite and X , X are independent, then ρ X ,X = . Its converse is not necessarily true. In general, ρ X ,X = may not imply independence between X , X . An example follows. Example Let X be N(, ) and X = X . Then, Cov(X , X ) = , and surely < V(X ), V(X ) < ∞, so that ρ X ,X = . But, X and X are dependent variables. More details can be found from Mukhopadhyay (, Sect. .). The recent article of Mukhopadhyay () is relevant here. Now, we state an important result which clarifies the role of zero correlation in a bivariate normal distribution. Theorem Suppose that (X , X ) has the N (µ , µ , σ , σ , ρ) distribution where −∞ < µ , µ < ∞, < σ , σ < ∞ and − < ρ(= ρ X ,X ) < . Then, X and X are independent if and only if ρ = . Example A zero correlation coefficient implies independence not merely in the case of a bivariate normal distribution. Consider random variables X and X whose joint probability distribution puts mass only at four points (, ), (, ), (, ), and (, ). Now, if Cov(X , X ) = , then X and X must be independent.
A Sample Correlation Coefficient We focus on a bivariate normal distribution. Let (X , Y ), . . . , (Xn , Yn ) be iid N (µ , µ , σ , σ , ρ) where −∞ < µ , µ < ∞, < σ , σ < ∞ and − < ρ < , n ≥ . Let us denote X = n− Σ ni= Xi S = (n − )− Σ ni= (Xi − X)
Y = n− Σ ni= Yi S = (n − )− Σ ni= (Yi − Y)
S = (n − )− Σ ni= (Xi − X)(Yi − Y) r = S /(S S ).
Here, r is called the Pearson (or sample) correlation coefficient. This r is customarily used to estimate ρ. The probability distribution of r is complicated, particularly when ρ ≠ . But, even without explicitly writing the pdf of r, it is simple enough to see that the distribution of r can not involve µ , µ , σ and σ . Francis Galton introduced a numerical measure, r, which he termed “reversion” in a lecture at the Royal Statistical Society on February , and later called “regression.” The term “cor-relation” or “correlation” probably appeared first in Galton’s paper to the Royal Statistical Society on December , . At that time, “correlation” was defined in terms of deviations from the median instead of the mean. Karl Pearson gave the definition and calculation of correlation r in . In , Pearson and his collaborators discovered that the standard deviation of r happened √ to be ( − ρ )/ n when n was large. “Student” derived the “probable error of a correlation coefficient” in . Soper () gave large-sample approximations for the mean and variance of r which performed better than those proposed earlier by Pearson. Refer to DasGupta () for more historical details. The unsolved problem of finding the exact pdf of r for normal variates came to R. A. Fisher’s attention via Soper’s paper. The pdf of r was published in the year by Fisher for all values of ρ ∈ (−, ). Fisher, at the age of , brilliantly exploited the n-dimensional geometry to come up with the solution, reputedly within one week. Fisher’s genius immediately came into limelight. Following the publication of Fisher’s results, however, Karl Pearson set up a major cooperative study of the correlation. One will notice that in the team formed for this cooperative project (Soper et al. ) studying the distribution of the sample correlation coefficient, the young Fisher was not included. This happened in spite of the fact that Fisher was right there and he already earned quite some fame. Fisher felt hurt as he was left out of this project. One thing led to another. RA. Fisher and Karl Pearson continued criticizing each other even more as each held on to his own philosophical stand.
Correlation Coefficient
We will merely state the pdf of r when ρ = . This pdf is given by f (r) = c ( − r ) (n−) for − < r < ,
√ − where c = Γ ( (n − )) { π Γ ( (n − ))} for n ≥ . Using a simple transformation technique, one can easily derive the following result: r(n − )/ ( − r )−/ has the Student’s t distribution with (n − ) degrees of freedom when ρ = . Fisher’s geometric approach () also included the exact pdf of r in the form of an infinite power series for all values of ρ ≠ . One may also look at Rao (, pp. –) for a non-geometric approach.
+ρ
C
′
since with g(ρ) = log { −ρ } , one has g (ρ) = −ρ . That is, for large n, we should consider the following pivot: √ n[U − ξ], which is approximately N(, ) for large n.
These transformations can be equivalently stated as −
−
U = tanh (r) and ξ = tanh (ρ), which are referred to as Fisher’s Z transformations introduced in . Fisher obtained the first four moments of tanh− (r) which were later updated by Gayen (). It turns out that the variance of tanh− (r) is approximated better by n− rather than n when n is moderately large. Hence, in many applications, one uses an alternate pivot (for n > ): √ n − [tanh− (r)− tanh− (ρ)] , which is approximately N(, ),
Large-Sample Distribution But, now suppose that one wishes to construct an approximate ( − α)% confidence interval for ρ, < α < . In this case, one needs to work with the non-null distribution of r. We mentioned earlier that the exact distribution of r, when ρ ≠ , was found with an ingenious geometric technique by Fisher (). That exact distribution being very complicated, Fisher () proceeded to derive the following asymptotic distribution when ρ ≠ : √ £ n(r − ρ) → N (, ( − ρ ) ) as n → ∞. For a proof, one may look at Sen and Singer (, pp. –) among other sources. One should realize that a variance stabilizing transformation may be useful here. We may invoke MannWald Theorem (see Mukhopadhyay , pp. –) by requiring a suitable function g(.) such that the asymptotic √ variance of n [g(r) − g(ρ)] becomes free from ρ. That is, we want to have: ′
g (ρ)( − ρ ) = k, a constant. So, g(ρ) = k ∫
dρ. (−ρ )
+ g(ρ) = k ∫ { −ρ
Hence, we rewrite
} dρ +ρ
+ρ
= k log { −ρ } + constant.
It is clear that we should look at the transformations: +ρ +r log { } and ξ = log { }, −r −ρ √ and consider the asymptotic distribution of n[U − ξ]. Now, we can claim that U=
√ £ n[U − ξ] → N(, ) as n → ∞,
for large n whatever be ρ, − < ρ < . For large n, one customarily uses Fisher’s Z transformations to come up with an approximate ( − α)% confidence interval for ρ. Also, to test a null hypothesis H : ρ = ρ , for large n, one uses the test statistic √ Zcalc = n − [tanh− (r) − tanh− (ρ )] and comes up with an approximate level α test against an appropriate alternative hypothesis. These are customarily used in all areas of statistical science whether the parent population is bivariate normal or not.
Partial Correlations Suppose that in general X = (X , . . . , Xp ) has a p-dimensional probability distribution with all pairwise correlations finite. Now, ρ Xi ,Xj will simply denote the correlation coefficient between Xi , Xj based on their joint bivariate distribution derived from the distribution of X, for any i ≠ j = , . . . , p. Next, ρ Xi ,Xj .Xk is simply the correlation coefficient between Xi , Xj based on their joint bivariate conditional distribution given Xk that is derived from the distribution of X, for any i ≠ j ≠ k = , . . . , p. Similarly, ρ Xi ,Xj .Xk ,Xl is simply the correlation coefficient between the pair of random variables Xi , Xj based on their joint bivariate conditional distribution given Xk , Xl derived from the distribution of X, for any i ≠ j ≠ k ≠ l = , . . . , p. Clearly, one may continue further like this. Such correlation coefficients ρ Xi ,Xj .Xk , ρ Xi ,Xj .Xk ,Xl are referred to as partial correlation coefficients. Partial correlation coefficients have important implications in multiple linear regression analysis. One may refer to Ravishanker and Dey (, pp. –) among other sources.
C
C
Correspondence Analysis
About the Author For biography see the entry 7Sequential Sampling.
Cross References 7Autocorrelation in Regression 7Intraclass Correlation Coefficient 7Kendall’s Tau 7Measures of Dependence 7Rank Transformations 7Spurious Correlation 7Tests of Independence 7Weighted Correlation
References and Further Reading DasGupta S () Distributions of the correlation coefficient. In: Fienberg SE, Hinkley DV (eds) R. A. Fisher: an appreciation. Springer, New York, pp – Fisher RA () Frequency distribution of the values of the correlation coefficients in samples from an indefinitely large population. Biometrika :– Fisher RA () Theory of statistical estimation. Proc Camb Phil Soc :– Gayen AK () The frequency distribution of the product moment correlation coefficient in random samples of any size drawn from non-normal universes. Biometrika :– Mukhopadhyay N () Probability and statistical inference. Marcel Dekker, New York Mukhopadhyay N () When finiteness matters: Counterexamples to notions of covariance, correlation, and independence. The Amer. Statistician, in press Rao CR () Linear statistical inference and its applications, nd edn. Wiley, New York Ravishanker N, Dey DK () A first course in linear model theory. Chapman & Hall/CRC Press, Boca Raton Sen PK, Singer JO () Large sample methods in statistics. Chapman & Hall, New York Soper HE () On the probable error of the correlation coefficient to a second approximation. Biometrika :– Soper HE, Young AW, Cave BM, Lee A, Pearson K () On the distribution of the correlation coefficient in small samples. A cooperative study. Biometrika :– “Student” (Gosset WS) () The probable error of a mean. Biometrika :–
Correspondence Analysis Jörg Blasius Professor University of Bonn, Bonn, Germany Correspondence analysis (CA) has been developed in the s in France by Jean-Paul Benzécri and his collaborators; it is the central part of the French “Analyse des Données,” or in English, geometric data analysis
(cf. Benzécri et al. ; Greenacre , ; Lebart et al. ; Le Roux and Rouanet ). The method can be applied to any data table with nonnegative entries. The main objective of CA is to display rows and columns of data tables in two-dimensional spaces, called “maps.” This kind of data description via visualization reflects a way of thinking that is typical for the social sciences in France, especially associated with the name of Pierre Bourdieu, and of many statisticians in the s and s in France, who at that time published almost only in French. The philosophy behind their work can be expressed by the famous quotation of Jean-Paul Benzécri who pointed out that “The model must follow the data, and not the other way around.” Instead of limiting the data to restrictive and subjectively formulated statistical models, they show the importance of the data and of the features in the data themselves. The discussion outside of France started with the textbooks by Greenacre () and Lebart et al. (). CA translates deviations from the independence model in a contingency table into distances as the following brief introduction shows. In the simple case, there is a two-way table N with I rows and J columns. In cases where the data are from survey research, the cells nij of N contain the frequencies of a bivariate cross-tabulation of two variables, with ∑nij = n. Dividing nij by the sample size n ij
provides the percentages of the total pij , or, for the entire table, with the (I × J) correspondence matrix P. Thereby, r = P is the vector of the “row masses,” or the “average column profile” with elements ri = ni+ /n, and c = PT is the vector of “column masses” or the “average row profile” with elements cj = n+j /n; Dr and Dc are the diagonal matrices of the row and column masses, respectively. The matrix of row profiles can be defined as the rows of the correspondence matrix P divided by their respective row masses, D− r P; for the matrix of columns profiles yields PD− c . As a measure of similarity between two row profiles (or between two column profiles, respectively), a weighted Euclidian or chi-square distance in the metric D− r (or, D− c , respectively) is used. For chi-square calculations, the weighted deviations from independence over all cells of the contingency table are used. For each cell, the unweighted deviation of the observed from the expected value can be calculated by (nij − nˆ ij ), with nˆ ij = (ni+ × n+j )/n. Dividing (nij − nˆ ij ) by n provides with (pij − ri cj ), or, in matrix notation, (P − rcT ), with the unweighted deviations from the independence model for the entire table. To fulfill the chi-square statistic, this matrix is weighted by the product of the square root of the row and column masses to give the standardized residuals sij = (pij − √ ri cj )/ ri cj , or in matrix notation, the (I × J) matrix of
Correspondence Analysis −/
−/
standardized residuals S = Dr (P − rcT )Dc . The similarity to chi-square analysis and total inertia as a measure for the variation in the data table, which is defined as ∑sij = ij
I J (p − r c ) χ ij i j = ∑∑ , becomes apparent. Applyn r c i j i= j=
ing singular value decomposition to S results in SVD(S) = UΓV T , where Γ is a diagonal matrix with singular values in descending order γ ≥ γ ≥ . . . ≥ γ S > , with S = rank of S. The columns from U are the left singular vectors, the columns from V are the right singular vectors, with UT U = VT V = I. The connection between SVD as used in CA and the well-known canonical decomposition is shown by ST S = VΓUT UΓVT = VΓ VT = VΛV T , with SST = UΓVT VΓUT = UΓ UT = UΛU T , and λ ≥ λ ≥ . . . ≥ λ S > ; χ /n = ∑ λ s = total inertia, since trace (SST ) = s
trace (ST S) = trace (Γ ) = trace (Λ). As in 7principal component analysis (PCA), the first axis is chosen to explain the maximum variation in the data; the second axis captures the maximum of the remaining variation, and so on. Again, analogous to PCA, it is possible to interpret the variable categories in relation to the axes, which can be considered the latent variables. And furthermore, as in PCA and other data reduction methods, only the s major components are used for interpretation. The number of interpretable dimensions depends on criteria such as the eigenvalue criteria, theory (how many latent variables can be substantively interpreted), or the scree test (for more details, see Blasius ). −/ For the graphical representation, we use F = Dr UΓ providing the principal coordinates of the rows, and −/ G = Dc VΓ providing the principal coordinates of the columns (for further details see Greenacre , ). The maps drawn on the basis of principal coordinates are called “symmetric maps.” In the full space, the distances between the rows and the distances between the columns can be interpreted as Euclidian distances, whereas the distances between the rows and the columns are not defined. As in PCA, the input data can be factorized. Understanding correspondence analysis as a model (see, e.g., van der Heijden et al. , ), the row and column coordinates can be used for recomputing the input data. Adding the latent variables successively models the deviations from independency. This is similar to the loglinear model and other modeling approaches such as the latent class model or the log-multiplicative model (see, e.g., van der Heijden et al. , ; Goodman ). In loglinear analysis, for example, these deviations are modeled by using higher-order interaction effects; in correspondence analysis latent variables are used. For any cell yields
S
S
s=
s=
C
nij = nri cj ( + ∑ fis gjs /γ s ), or pij = ri cj ( + ∑ fis gjs /γ s ), and in matrix notation P = rcT + Dr FΓ− GT Dc . The left part of the equation reflects the independence model and the right part, the modeling from independency by including the S factors in successive order. Including all factors in the model fully reconstructs the original data table N. The interpretation of CA is similar to the one of PCA, both methods provide eigenvalues and their explained variances, factor loadings, and factor values. While PCA is restricted to metric data, CA can be applied to any kind of data table with nonnegative entries, among others, to indicator and Burt matrices – in these two cases the method is called multiple correspondence analysis (MCA). Whereas simple correspondence analysis is applied to a single contingency table or to a stacked table, MCA uses the same algorithm to an indicator or a Burt matrix. In the case of survey research, input data to simple CA is usually a matrix of raw frequencies of one or more contingency tables. In this context, there is usually one variable to be described, for example, preference for a political party, and one or more describing variables, for example, educational level and other sociodemographic indicators such as age groups, gender, and income groups. The number of variables can be quite high, apart from theoretical considerations there is no real limitation by the method. In the given case, each of the describing variables is crosstabulated with the variable to be described in order to investigate the importance of this association. Concatenating, or stacking the tables before applying CA allows to visualize and interpret several relationships of “preferred political party” with the sociodemographic indicators in the same map. Applying CA to the indicator matrix Z (=MCA), the table of input data has as many rows as there are respondents, and as many columns as there are response alternatives in all variables included in the analysis. A “” in a given row indicates the respondent who chose that specific response category; otherwise there is a “” for “specific response category not chosen.” Considering all categories of all variables provides row sums that are constant and equal to the number of variables, the column sums reflect the marginals. An alternative to the indicator matrix as input to MCA is the Burt matrix B. This matrix can either be generated by cross-tabulating all variables by all variables, including the cross-tabulations of the variables by themselves, and stacking them row- and column-wise. Further, B can be computed by multiplying the transposed indicator matrix by itself, that is B = ZT Z. The solutions from Z can be directly converted to those of B by rescaling the solution; for example, the squared eigenvalues of Z
C
C
Correspondence Analysis
are equal to those of B. As it is true for PCA, MCA contains all first-order interaction effects, the method can be understood as a generalization of PCA to categorical data. Taking a two-way contingency table with I = rows, J = columns, and n = cases as an example, input data of the simple CA would be the frequencies of the ( × ) cross-table. Turning to MCA, input data is an indicator matrix with rows (the number of cases) and columns (the number of variable categories). MCA is also known under the names “homogeneity analysis” (see Gifi ; Heiser and Meulman ), “dual scaling” (Nishisato , ), and “quantification of qualitative data III” (Hayashi ); CA procedures are available in all major statistic packages as well as in R (Greenacre and Nenadi´c ). For details regarding the history of CA and related methods, we refer to Nishisato (, Chap. ). CA employs the concept of inertia: the farther the categories are from the centroid along a given axis (squared distances) and the higher their masses (their marginals), the more the categories determine the geometric orientation of that axis. In the graphical solution, the locations of all variable categories can be compared to each other (except in simple CA and using symmetric maps, in this case the distances between rows and columns are not defined), short distances imply high similarities and long distances imply high dissimilarities. For all dimensions, CA supplies principal inertias that can be interpreted as canonical correlation coefficients (they are the singular values of the solution, i.e., the square roots of the eigenvalues), correlation coefficients between the item categories and the latent variables as well as scores for all item categories and all respondents. There are several extensions of simple CA and MCA. With respect to the Burt matrix B, it is apparent that most of the variation in this super matrix is caused by the main diagonal blocks. These sub-matrices contain the cross-tabulations of the variables by themselves; the main diagonal elements of them contain the marginals of the variables while their off-diagonal elements are equal to zero. Excluding this variation in an iterative procedure and visualizing the variation of the off-diagonal blocks of B only is the objective of joint correspondence analysis. The aim of subset correspondence analysis is to concentrate on some response categories only, while excluding others from the solution. For example, applying subset MCA to a set of variables, the structure of non-substantive responses (“don’t know,” “no answer”) can be analyzed separately, or these responses can be excluded from the solution while concentrating on the substantive responses. Variables can also be included in the model as supplementary or passive ones; in this case they do not have any impact on the
geometric orientation of the axes but they can be interpreted together with the active variables. CA can not only be applied to single and stacked contingency tables or to indicator matrix, it can also be used to analyze rank and metric data, multiple responses, or squared tables. The statistical background and examples of these kinds of data can be found in the textbook of Greenacre () as well as in the readers of Greenacre and Blasius (, ), and Blasius and Greenacre ().
About the Author Jörg Blasius is the President of RC (Research Committee of Logic and Methodology in Sociology) at ISA (International Sociological Association) (–). Together with Michael Greenacre (Barcelona) he founded CARME (Correspondence Analysis and Related Methods Network) and edited three books on Correspondence Analysis, in June they had a special issue on this topic in Computational Statistics and Data Analysis (further coeditors: Patrick Groenen and Michel van de Velden).
Cross References 7Data Analysis 7Distance Measures 7Multivariate Data Analysis: An Overview
References and Further Reading Benzécri JP et al () L’analyse des Données. L’analyse des Correspondances. Dunod, Paris Blasius J () Correspondence analysis in social science research. In: Greenacre M, Blasius J (eds) Correspondence analysis in the social sciences. Recent developments and applications. Academic, London, pp – Blasius J, Greenacre M (eds) () Visualization of categorical data. Academic, London Gifi A () Nonlinear multivariate analysis. Wiley, Chichester Goodman LA () Measures, models, and graphical display in the analysis of cross-classified data (with discussion). J Am Stat Assoc :– Greenacre MJ () Theory and applications of correspondence analysis. Academic, London Greenacre MJ () Correspondence analysis in practice. Chapman & Hall, Boca Raton Greenacre MJ, Blasius J (eds) () Correspondence analysis in the social sciences. Recent developments and applications. Academic, London Greenacre MJ, Blasius J (eds) () Multiple correspondence analysis and related methods. Chapman & Hall, Boca Raton Greenacre MJ, Oleg Nenadi´c () Computation of multiple correspondence analysis, with code in R. In: Greenacre M, Blasius J (eds) Multiple correspondence analysis and related methods. Chapman & Hall, Boca Raton, pp – Hayashi C () Multidimensional quantification – with the applications to the analysis of social phenomena. Ann Inst Stat Math :–
Cp Statistic
Heiser WJ, Meulman JJ () Homogeneity analysis: exploring the distribution of variables and their nonlinear relationships. In: Greenacre M, Blasius J (eds) Correspondence analysis in the social sciences. Recent developments and applications. Academic, London, pp – Lebart L, Morineau A, Warwick KM () Multivariate descriptive statistical analysis: correspondence analysis and related techniques for large matrices. Wiley, New York Le Roux B, Rouanet H () Geometric data analysis. North Holland, Amsterdam Nishisato S () Analysis of categorical data: dual scaling and its applications. University of Toronto Press, Toronto Nishisato S () Multidimensional nonlinear descriptive analysis. Chapman & Hall, Boca Raton Van der Heijden PGM, de Falguerolles A, de Leeuw J () A combined approach to contingency table analysis using correspondence analysis and loglinear analysis. Appl Stat :– Van der Heijden PGM, Mooijaart A, Takane Y () Correspondence analysis and contingency table models. In: Greenacre M, Blasius J (eds) Correspondence analysis in the social sciences. Recent developments and applications. Academic, London, pp –
C
Under the usual assumptions, that the vector of observations y equals ν + z where ν is the vector of true means, and the z’s are independent with mean zero and constant variance σ , s Cp is an unbiased estimate of σ E(JP ) where JP is ∣νˆP − ν∣ , and where νˆP is the estimate of ν that is obtained by fitting the P model. Thus JP is a measure of the adequacy for prediction of the P model. This result holds even when the true model ν is not expressible in terms of the available regressors. However in this case we cannot use the residual sum of squares from the full (K) model as an estimate of σ . The Cp statistic is often used to guide selection of a subset-model, but this cannot be recommended; while for each P separately, Cp gives an unbiased estimate of the scaled mean-square error for that subset, this is not true if the subset is chosen to minimise Cp . In fact this approach can lead to worse results than are obtained by simply fitting all available regressors. In a paper, Mallows has attempted to quantify this effect. The Cp statistic is similar to 7Akaike’s Information criterion.
Cp Statistic Colin Mallows Basking Ridge, Avayalabs, NJ, USA The Cp statistic was invented by C. Mallows in . It facilitates the comparison of many subset-regression models, by giving for each model an unbiased estimate of the (scaled) total mean-square-error for that model. There is an associated graphical technique called the “Cp plot” in which values of Cp (one for each subset of regressors) are plotted against p. The problem in choosing a subset-regression model for predicting a response is that including too many unnecessary terms will add to the variance of the predictions, while including too few will result in biased predictions. In more detail, if we have n observations, and k regressors are available (possibly including a constant term), let P denote some subset of these. (Usually if a constant term is to be considered, this will appear in each subset). Let p be the number of regressors in the subset P. Then Cp (for the P-subset model) is defined to be RSSP − n + p s where RSSP is the residual sum of squares for this P-model, and s is an estimate of the residual variance when all relevant terms are included in the model. Usually this is taken to be RSSK where K is the set of all available regressors. Cp =
About the Author Colin L. Mallows spent years at AT&T Bell Labs and one of its descendants, AT&T Labs. Since retiring he has been a consultant at another descendant, Avaya Labs. He is a Fellow of the American Statistical Association, the Institute of Mathematical Statistics, and the Royal Statistical Society. He has served on several committees of the IMS and ASA. He was an Associate editor of JASA (–). He has been COPSS Fisher Lecturer and ASA Deming Lecturer, and has received the ASA Wilks Medal. He has written over papers, and has edited two books.
Cross References 7Akaike’s Information Criterion 7General Linear Models
References and Further Reading Daniel C, Wood FS () Fitting equations to data, rev edn. Wiley, New York Gorman JW, Toman RJ () Selection of variables for fitting equations to data. Technometrics :– Mallows CL () Some comments on Cp. Technometrics :– Mallows CL () More comments on Cp. Technometrics :–
C
C
Cramér–Rao Inequality
Cramér–Rao Inequality Maarten Jansen , Gerda Claeskens Université libre de Bruxelles, Brussels, Belgium K. U. Leuven, Leuven, Belgium
The Cramér–Rao Lower Bound The Cramér–Rao inequality gives a lower bound for the variance of an unbiased estimator of a parameter. It is named after work by Cramér () and Rao (). The inequality and the corresponding lower bound in the inequality are stated for various situations. We will start with the case of a scalar parameter and independent and identically distributed random variables X , . . . , Xn , with the same distribution as X. Denote X = (X , . . . , Xn ) and denote the common probability mass function or probability density function of X at a value x by f (x; θ) where θ ∈ Θ, which is a subset of the real line R and x ∈ R. Denote the support of X by R, that is, R = {x : f (x; θ) > }.
Assumptions ∂ log f (x; θ) exists for all θ ∈ . The partial derivative ∂θ Θ and all x ∈ R and it is finite. This is equivalent to stating that the Fisher information value IX (θ) = ∂ log f (X; θ)) ] is well defined, for all θ ∈ Θ. E [( ∂θ . The order of integration and differentiation is inter∂ changeable in ∫ ∂θ log f (x; θ)dx. If the support of X, that is, the set R, is finite, then the interchangeability is equivalent with the condition that the support does not depend on θ. A counter-example on uniformly distributed random variables is elaborated below.
The Cramér–Rao inequality Under assumptions (i) and (ii), if θˆ = g(X) is an unbiased ˆ = θ, then estimator of θ, this means that E[θ] ˆ ≥ / [n ⋅ IX (θ)] . var(θ) The lower bound in this inequality is called the Cramér– Rao lower bound. The proof starts by realizing that the correlation of the n ∂ score V = ∂θ ∑i= log fX (Xi ; θ) and the unbiased estimator θˆ is bounded above by . This implies that (var(V) ⋅ ˆ / ≥ cov(V, θ). ˆ The assumptions are needed to var(θ)) prove that the expected score E(V) is zero. This implies ˆ = , from which the stated that the covariance cov(V, θ) inequality readily follows.
A second version of the Cramér–Rao inequality holds if we estimate a functional κ = H(θ). Under assumptions (i) and (ii), if X is a sample vector of independent observations from random variable X with density function f (x; θ) and κˆ = h(X) is an unbiased estimator of H(θ), such that exists and is finite for all θ, then the first derivative dH(θ) dθ var(κˆ ) ≥ [
dH(θ) ] / [n ⋅ IX (θ)] . dθ
Similar versions of the inequality can be phrased for observations that are independent but not identically distributed. In the case of a vector parameter θ, the variance of ˆ is replaced by the the single parameter estimator var(θ) covariance matrix of the estimator vector Σ̂θ . This matrix is bounded by a matrix expression containing the inverse of the Fisher information matrix, where bounded means that the difference between the covariance matrix and its “upper bound” is a negative semidefinite matrix. The Cramér–Rao inequality is important because it states what the best attainable variance is for unbiased estimators. Estimators that actually attain this lower bound are called efficient. It can be shown that maximum likelihood estimators asymptotically reach this lower bound, hence are asymptotically efficient.
Cramér–Rao and UMVUE If X is a sample vector of independent observations from the random variable X with density function fX (x; θ) and ˆ = θˆ = g(X) is an unbiased estimator of θ, then var(θ) ˆ / [n ⋅ IX (θ)] ⇔ θ = aV + b with probability one, where V is the score and a and b are some constants. This follows from the proof of the Cramér–Rao inequality: the lower bounded is reached if the correlation between the score ˆ and the estimator is one. This implies that var ( σVV + σθˆ ) θ
ˆ
= ⇒ σVV + σθˆ = c almost surely for some constant c. We θ here used the notation σX to denote the standard deviation of a random variable X. The coefficients a and b may depend on θ, but θˆ should be observable without knowing θ. If a and b exist such that θˆ is unbiased and observable, then θˆ has the smallest possible variance among all unbiased estimators: it is then certainly the uniformly minimum variance unbiased estimator (UMVUE). It may, however, be well possible that no a and b can be found. In that case, the UMVUE, if it exists, does not reach the Cramér-Rao lower bound. In that case, the notion of sufficiency can be used to find such UMVUE.
Cramér–Von Mises Statistics for Discrete Distributions
Counter example: estimators for the upperbound of uniform data Let X ∼ unif[, a], so fX (x) = a I( ≤ x ≤ a), where I(c ≤ x ≤ d) is the indicator function of the interval [c, d]. We want to estimate a. The maximum likelihood estimator (MLE) is aˆ MLE = max Xi , which is biased. Define i=,...,n
n aˆ MLE , which is unbiased. The method of moments aˆ u = n− leads to an estimator aˆ MME = X, which is also unbiased. ∂ log fX (Xi ; a) = − a . This is a conThe score is Vi = ∂a stant (so, not a random variable), whose expected value is of course not zero. This is because the partial derivative and expectation cannot be interchanged, as the boundary of the support of X depends on a. As a consequence, the Cramér–Rao lower bound is not valid here. We can verify n that var(ˆaMLE ) = (n+)(n+) au ) = n(n+) a . a and var(ˆ This is (for n → ∞) one order of magnitude smaller than var(ˆaMME ) = n a and also one order of magnitude smaller than what you would expect for an unbiased estimator if the Cramér–Rao inequality would hold.
A Bayesian Cramér–Rao Bound It should be noted that biased estimators can have variances below the Cramér–Rao lower bound. Even the MSE (mean squared error), which equals the sum of the variance and the squared bias can be lower than the Cramér–Rao lower bound (and hence lower than any unbiased estimator could attain). A notable example in this respect is Stein’s phenomenon on shrinkage rules (Efron and Morris ). In practice, large classes of estimators, for example most nonparametric estimators, are biased. An inequality that is valid for biased or unbiased estimators is due to van Trees (, p. ), see also Gill and Levit () who developed multivariate versions of the inequality. We assume that the parameter space Θ is a closed interval on the real line and denote by g some probability distribution on Θ with density λ(θ) with respect to the Lebesgue measure. This is where the Bayesian flavor enters. The θ is now treated as a random variable with density λ. We assume that λ and f (x; ⋅) are absolutely continuous and that λ converges to zero at the endpoints of the interval ∂ log f (X; θ)] = . We Θ. Moreover we assume that E[ ∂θ denote I(λ) = E[{log λ(θ)} ] and have that E[IX (θ)] = ˆ it holds ∫ IX (θ)g(θ)dθ. Then, for an estimator θˆ = θ(X), that . E[{θˆ − θ} ] ≥ E[IX (θ)] + I(λ) A second form of this inequality is obtained for functionals κ = H(θ). Under the above assumptions, for an
C
estimator κˆ = h(X) of H(θ), such that the first derivative dH(θ) exists and is finite for all θ, dθ E[{κˆ − H(θ)} ] ≥
d H(θ)]} {E[ dθ . E[IX (θ)] + I(λ)
About the Authors For the biography of Maarten Jansen see the entry 7Nonparametric Estimation. For the biography of Gerda Claeskens see the entry 7Model Selection.
Cross References 7Estimation 7Minimum Variance Unbiased 7Sufficient Statistical Information 7Unbiased Estimators and Their Applications 7Uniform Distribution in Statistics
References and Further Reading Cramér H () Mathematical methods of statistics. Princeton University Press, Princeton Efron B, Morris C () Stein’s paradox in statistics. Scient Am :– Gill RD, Levit BY () Applications of the van Trees inequality: a Bayesian Cramér–Rao bound. Bernoulli (–): – Rao C () Information and the accuracy attainable in the estimation of statistical parameters. Bull Calcutta Math Soc :– van Trees HL () Detection, estimation and modulation theory: part I. Wiley, New York
Cramér–Von Mises Statistics for Discrete Distributions Michael A. Stephens Professor Emeritus Simon Fraser University, Burnaby, B.C., Canada
Introduction Cramér–von Mises statistics are well established for testing fit to continuous distributions; see Anderson () and Stephens (), both articles in this encyclopedia. In this paper, the corresponding statistics for testing discrete distributions will be described. Consider a discrete distribution with k cells labeled , , . . . , k, and with probability pi of falling into cell i. Suppose n independent observations are given; let oi be the observed number of observations and ei = npi be the j j expected number in cell i. Let Sj = ∑i= oi and Tj = ∑i= ei .
C
C
Cramér–Von Mises Statistics for Discrete Distributions
Then Sj /n and Hj = Tj /n are the cumulated histograms of observed and expected values and correspond to the empirical distribution function Fn (z) and the cumulative distribution function F(.) for continuous distributions. Suppose Zj = Sj − Tj , j = , , . . . , k; the weighted mean of the Zi is Z¯ = ∑kj= Zj tj , where tj = (pj + pj+ )/, with pk+ = p . The modified Cramér–von Mises statistics are then defined as follows: k
Wd = n− ∑ Zj tj ;
()
j= k
¯ tj ; Ud = n− ∑(Zj − Z)
()
j= k
Ad = n− ∑ Zj tj /{Hj ( − Hj )}.
()
j=
note that Zk = in these summations, so that the last term in Wd is zero. The last term in Ad is of the form /, and is set equal to zero. The well-known Pearson χ statistic is
photograph, the histogram of cells with light to dark observations might be recorded, or vice versa. . The statistic Ud is intended for use with a discrete distribution around a circle, since its value does not change with different choices of origin; this is why pk+ is set equal to p .
Matrix Formulation To obtain asymptotic distributions it is convenient to put the above definitions into matrix notation. Let a prime, e.g., Z′ , denote the transpose of a vector or matrix. Let I be the k × k identity matrix, and let p′ be the × k vector (p , p , . . . , pk ). Suppose D is the k × k diagonal matrix whose j-th diagonal entry is pj , j = , . . . , k and let E be the diagonal matrix with diagonal entries tj , and K be the diagonal matrix whose (j, j)-th element is K jj = /{Hj ( − Hj )}, j = , . . . , k − and K kk = . Let oi and ei be arranged into column vectors o, e (so that, for example, the j-th component of o is oj , j = , . . . , k). Then Z = Ad, where d = o − e and A is the k × k partial- sum matrix
k
χ = ∑(oi − ei ) /ei .
⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ A=⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝
i=
Statistics corresponding to the Kolmogorov–Smirnov statistics (see 7Kolmogorov-Smirnov Test) for continuous observations are √ √ − D+d = max(Zj )/ n, Dd = max(−Zj )/ n, j j √ Dd = max ∣Zj ∣/ n. j
Comments on the Definitions . Several authors have examined distributions of the Kolmogorov–Smirnov family, see Pettitt and Stephens () and Stephens () for tables and references. In general, for continuous data, the Kolmogorov– Smirnov statistic is less powerful as an omnibus test statistic than the Cramér–von Mises family; limited Monte Carlo studies suggest that this holds also for Dd . . The Cramér–von Mises and Kolmogorov–Smirnov statistics take into account the order of the cells, in contrast to the Pearson χ statistic. . Use of tj in these definitions ensures that the value of the statistic does not change if the cells are labelled in reverse order. For instance, in testing the 7binomial distribution, one statistician might record the histogram of successes, and another the histogram of failures; or in a test involving categorical data such as the tones of a
...
...
...
⋮
⋮
⋮
⋱
...
⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ . ⎟ ⎟ ⎟ ⎟ ⋮ ⎟ ⎟ ⎟ ⎠
The definitions become Wd = Z′ EZ/n, Ud Ad
′
() ′
′
= Z (I − E )E(I − E)Z/n, ′
() ()
= Z EKZ/n, ′
X = (d′ D− d)/n = Z′ A− D− A− Z/n.
()
Asymptotic Theory All Parameters Known
All four statistics above are of the general form S = Y ′ MY, √ where Y = Z/ n and M is symmetric. For Wd , M = E, for Ud , M = (I − E′ )E(I − ′ E), and for Ad , M = EK. Also Y has mean . Suppose its covariance matrix is Σ y , to be found below; then S may be written k−
S = Y ′ MY = ∑ λ i (w′i Y) , i=
()
Cramér–Von Mises Statistics for Discrete Distributions
where λ i are the k − non-zero eigenvalues of MΣ y and wi are the corresponding eigenvectors, normalized so that w′i Σ y wj = δ ij where δ ij is if i = j and otherwise. As n → ∞, the si tend to standard normal, and they are independent; the limiting distribution of S is that of S∞ where k−
sinf S∞ = ∑ λ i si
()
i=
which is a sum of independent weighted χ variables. √ √ Recall that Y = Z/ n = Ad/ n; its covariance Σ y is found as follows. Calculate the k × k matrix Σ = D − pp′ ;
()
√ this is the covariance matrix of (o − e)/ n. Then Σ y = ′ AΣ A , with entries Σ y,ij = min(Hi , Hj ) − Hi Hj . For the appropriate M for the statistic required, the eigenvalues λ i , i = , ⋯, k of MΣ y are used in () to obtain the limiting distribution of the statistic. The limiting distributions have been examined in detail in Choulakian et al. ().
Parameters Unknown Cramér–von Mises statistics when the tested distribution contains unknown parameters θ i have been investigated by Lockhart et al. (). The θ i must be estimated efficiently, for example by maximum likelihood (ML). Suppose θ = (θ , θ , . . . , θ m )′ is the vector of m parameters. The log-likelihood is (omitting irrelevant constants) k
L∗ = ∑ oi log pi , i=
and pi contains the unknown parameters. The ML estimation consists of solving the m equations k oi ∂pi ∂L∗ =∑ = , ∂θ j i= pi ∂θ j
for j = , . . . , m. Let θˆ be the ML estimate of θ, let pˆ be the estimate ˆ and let ˆe be the estimated vector of p, evaluated using θ, of expected values in the cells, with components ˆej = npˆ j . ˆ Then let dˆ = (o − ˆe) and Zˆ = Ad. Define a k by m matrix B with entries Bi,j = ∂pi /∂θ j for i = , . . . , k and j = , . . . , m. The matrix B′ D− B is the Fisher Information matrix for the parameter estimates. Define V = (B′ D− B)− . The asymptotic covariance of θˆ ˆ √n is Σ d = Σ − BVB′ , is then V/n, the covariance of d/
C
√ ˆ n= where Σ is defined in (), and the covariance of Z/ √ ˆ n = Yˆ is Ad/ Σ u = AΣ d A′ . Then, as in the previous section, where parameters were known, the weights λ i in the asymptotic distribution () are the k eigenvalues of MΣ u for the appropriate M for the statistic required. In practice, in order to calculate the statistics, using (–), the various vectors and matrices must be replaced by their estimates where necessary. For example, let matrix ˆ be D with p replaced by pˆ and similarly obtain B, ˆ E, ˆ V, ˆ Kˆ D ˆ using estimates in an obvious way. The eigenvalues and Σ ˆ u and M. ˆ will also be found using the estimated matrices Σ Consistent estimates of the λ i will be obtained and () used to find the estimated asymptotic distribution. Thus the steps are : ˆ = (Bˆ ′ D ˆ − B) ˆ − . . Calculate V ˆd = Σ ˆ − Bˆ V ˆ u = AΣ ˆ d A′ . ˆ Bˆ ′ and Σ . Calculate Σ ˆ be the estimate of . For the statistic required, let M the appropriate M. Find the k − eigenvalues of ˆ u , or (equivalently) those of the symmetric matrix ˆΣ M / ˆ uM ˆ Σ ˆ / and use them in () to obtain the asympM totic distribution. For practical purposes, percentage points of S∞ using exact or estimated λs, can be used for the distributions of the statistics for finite n; this has been verified by many Monte Carlo studies. One therefore needs good approximate points in the upper tail of S∞ ; these can be found from the percentage points of S, where S has the distribution a + bχ p , and the a, b, p are chosen so that the first three cumulants of S match those of S∞ in (). These cumuj lants are κ j = j− (j − )! ∑k− i= λ i . In particular, the mean κ k− k− is ∑i= λ i , the variance κ is ∑k− i= λ i and κ is ∑i= λ i . Then for the S approximation, b = κ /(κ ), p = κ /κ , and a = κ − bp. This approximation is generally accurate in the upper tail, at levels α < .. More accurate points can be obtained by the method of Imhof ().
About the Author Michael A. Stephens is Professor Emeritus of Mathematics and Statistics at Simon Fraser University in Burnaby, British Columbia, Canada. Prior to that he taught at several universities including McGill, Nottingham, McMaster, and Toronto, and was a visiting professor at Stanford, Wisconsin-Madison, and Grenoble. He has (co-)authored over papers on the analysis of directional data, continuous proportions, curve-fitting, and tests of fit. Professor Stephens was President of the Statistical Society of Canada in . He is a Fellow of the Royal Statistical Society, and
C
C
Cross Classified and Multiple Membership Multilevel Models
his honors include membership in the International Statistical Institute, and fellowships of the American Statistical Association and the Institute of Mathematical Statistics. Dr. Stephens received the B.Sc. degree () from Bristol University and A.M. degree () in physics from Harvard University, where he was the first Frank Knox Fellow, and Ph.D. degree () from the University of Toronto. In he was awarded the Gold Medal, Statistical Society of Canada for two main areas of research: analysis of directional data, and statistical theory and methods associated with goodness of fit.
Cross References 7Anderson-Darling Tests of Goodness-of-Fit 7Exact Goodness-of-Fit Tests Based on Sufficiency 7Kolmogorov-Smirnov Test 7Tests of Fit Based on The Empirical Distribution Func-
tion
References and Further Reading Anderson TW () Anderson–Darling tests of goodness-of-fit. Article in this encyclopedia Anderson TW, Darling DA () Asymptotic theory of certain goodness of fit criteria based on stochastic processes. Ann Math Stat :– Choulakian V, Lockhart RA, Stephens MA () Cramer–von Mises Tests for discrete distributions. Can J Stat :– Darling DA () The Cramér–Smirov test in the parametric case. Ann Math Stat :– Imhof JP () Computing the distribution of quadratic forms in normal variables. Biometrika :– Lockhart RA, Spinelli JJ, Stephens MA () Cramér–von Mises statistics for discrete distributions with unknown parameters. Can J Stat :–() Pettitt AN, Stephens MA () The Kolmogorov–Smirov test for discrete and grouped data. Technometrics ():– Stephens MA () Asymptotic results for goodness-of-fit statistics with unknown parameters. Ann Stat :– Stephens MA () Tests based on EDF statistics. In: D’Agostino R, Stephens MA (eds) Chap. in Goodness-of-fit techniques. Marcel Dekker, New York Stephens MA () EDF tests of fit. Article in this encyclopedia
Cross Classified and Multiple Membership Multilevel Models Harvey Goldstein Professor of Social Statistics University of Bristol, Bristol, UK
Hierarchically Structured Data Interesting real life data rarely conform to classical text book assumptions about data structures. Traditionally
these assumptions are about observations that can be modelled with independently, and typically identically, distributed “error” terms. More often than not, however, the populations that generate data samples have complex structures where measurements on data units are not mutually independent, but depend on each other through complex structural relationships. For example, a household survey of voting preferences will typically show variation among households and voting constituencies (constituencies and households differ on average in their political preferences). This implies that the replies from individual respondents within a household or constituency will be more alike than replies from individuals in the population at large. Another example of such “hierarchically structured data” would be measurements on students in different schools, where, for example, schools differ in terms of the average attainments of their students. In epidemiology we would expect to find differences in such things as fertility and disease rates across geographical and administrative areas. Techniques for modelling such data have come to be known as “multilevel” or “hierarchical data” models and basic descriptions of these are dealt with in other articles (see 7Multilevel Analysis). In the present article we shall consider two particular extensions to the basic multilevel model that allow us to fit structures that have considerable complexity and are quite commonly found, especially in the social and medical sciences.
The Basic Multilevel Model A simple multilevel model for hierarchical data structures with normally distributed responses can be written as: iid
iid
yij = β + β xij + uj + eij , uj ∼ N(, σu ), eij ∼ N(, σe ). () This might be applied to a sample, say, of school students where i indexes students (level ), who are grouped within schools (level ). The response y might be an attainment measure and x a predictor such as a prior test score. Often referred to as a “variance components” model this may be extended in a number of ways to better fit a data set. For example, we may introduce further covariates and we may allow the coefficients of such covariates to vary at level , so that, say, β , may vary from school to school. Another possibility is to allow the level variance to depend on a set of explanatory variables, so that, for example, we can allow the variance between male students to be different from that for female students. We can have several responses that are correlated leading to a multivariate model, and we can consider non-normal responses, such as binary ones, in order to fit generalised linear multilevel models. We can also have several further levels; for example schools may be
Cross Classified and Multiple Membership Multilevel Models
grouped within school boards or authorities, so yielding a three level structure. Goldstein () provides further details and discusses estimation methods.
Cross Classified Structures The above only describes purely hierarchical models. In practice, however, data structures are often more complicated. Consider an educational example where students move through both their primary and secondary education with the response being attainment at the end of secondary school. For any given primary school, students will generally move to different secondary schools, and any given secondary school will draw students from a number of primary schools. We therefore have a cross classification of primary by secondary schools where each cell of the classification will be populated by students (some may be empty). When we model such a structure we have a contribution to the response that is the sum of an effect from the primary and an effect from the secondary school attended by a student. A basic, variance components, cross classified model may be written as ()
()
()
yi = β + β xi + uprimary school(i) + usecondary school(i) ()
+ustudent(i) () ()
iid
usecondary school(i) ∼ N(, σu() ) ()
members of more than one friendship group. In an educational system students may attend more than one school over time. In all such cases we shall assume that for each higher level unit to which a lower level unit belongs there is a known weight (summing to . for each lower level unit), which represents, for example, the amount of time spent in the higher level unit. The choice of weights may be important but is beyond the scope of this article. For more details about choosing weights see Goldstein et al. (). Using the general notation we used for cross classifications we can write a basic variance components multiple membership model as ()
yi
() ()
()
= β + β xi + ∑ wi,j u(j) + ui j∈school(i)
() u(j) ∼
N(, σu() ), () ∑ wi,j j∈school(i)
()
u(i) ∼N(, σu() )
This assumes that the total contribution from the level units (schools) is a weighted sum over all the units of which the level unit has been a member. Thus, for example, if every student spends half their time in one school and half their time in another (randomly selected) then the variance at level will be ()
()
iid
ustudent(i) ∼ N(, σu() ), i = , . . . , N.
We have changed the notation to make it more general and flexible. The superscript refers to the set of units, or classification; being students, primary school and secondary school. Model () thus assumes that there are separate, additive, contributions from the primary and the secondary school attended. As with the simple hierarchical model we can extend () in several ways by introducing random coefficients, complex variance structures and further cross classifications and levels. There are many examples where cross classified structures are important. Thus, for example, students will generally be grouped by the neighborhood where they live and this will constitute a further classification. In a repeated measures study where there is a sample of subjects and a set of raters or measurers, if the subjects are rated by different people at each occasion we would have a cross classification of subjects by raters.
Multiple Membership Structures In many circumstances units can be members of more than one higher level unit at the same time. An example is friendship patterns where at any time individuals can be
()
= .
()
var(.uj + .uj ) = σu() /.
iid
uprimary school(i) ∼ N(, σu() ),
C
()
Thus, a failure to account for the multiple membership of higher level units in this case will lead us to treat the esti mate of the level variance, σu() / as if it were a consistent estimate of the true level variance σu() . More generally, ignoring a multiple membership structure will lead to an underestimation of the higher level variance. Finally, we can combine cross classified and multiple membership structures within a single model and this allows us to handle very complex structures. An example where the response is a binary variable is given in Goldstein (, Chap. ). It is possible to use maximum likelihood estimation for these models, but apart from small scale datasets, MCMC estimation is more efficient and flexible. The MLwiN software package (Rasbash et al. ; Browne . http://www.cmm.bristol.ac.uk) is able to fit these models.
About the Author Professor Goldstein is a chartered statistician, has been editor of the Royal Statistical Society’s Journal, Series A, a member of the Society’s Council and was awarded the Society’s Guy medal in silver in . He was elected a member of the International Statistical Institute in , and a Fellow of the British Academy in . He was awarded an honorary doctorate by the Open University in .
C
C
Cross-Covariance Operators
His most important research interest is in the methodology of multilevel modelling. He has had research funding for this since and has been involved in the production (with Jon Rasbash and William Browne) of a widely used software package (MLwiN) and made a number of theoretical developments. These include multiple membership models, multilevel structural equation models and more recently the use of multilevel multivariate latent normal models and especially their application to missing data problems. The major text on multilevel modelling is his book Multilevel Statistical Models (New York, Wiley, Fourth Edition, ). He has also written extensively on the use of statistical modelling techniques in the construction and analysis of educational tests. The implications of adopting such models have been explored in a series of papers since .
Cross References 7Multilevel Analysis 7Psychology, Statistics in
References and Further Reading Browne WJ () MCMC estimation in MLwiN. Version .. Bristol, Centre for Multilevel Modelling, University of Bristol Goldstein H () Multilevel statistical models, th edn. New York, Wiley Goldstein H, Burgess S, McConell B () Modelling the effect of pupil mobility on school differences in educational achievement. J R Stat Soc Ser A ():– Rasbash J, Steele F, Browne W, Goldstein H () A user’s guide to MLwiN version .. Bristol, Centre for Multilevel Modelling, University of Bristol
the cross-covariance operator of a joint measure, and providing new results for actually computing likelihood ratios for joint measures. Applications to date have included results on absolute continuity of probability measures, mutual information for pairs of 7stochastic processes, and analysis of information capacity for communication channels. More recently, there has been interest in this topic by researchers in machine learning, who have applied theory from Baker () in a number of interesting publications (e.g., Fukumizu et al. , ; Gretton et al. ). The joint measures to be discussed are probability measures on the product of two real separable Hilbert spaces, H and H , with Borel sigma fields θ . and θ . Denote the inner products by < , > on H and < , > on H . H x H is then a real separable Hilbert space under the inner product defined by <(x,u), (v,y)> = <x,v> +
. Next, introduce a joint measure π on the measurable space (H x H , θ x θ ). Only strong second-order probability measures will be considered: those joint measures ∂ such that E∂ , ∣∣∣(x,y)∣∣∣ = ∫H xH (∣∣x∣∣ +∣∣y∣∣ )d∂ (x,y) is finite. All Gaussian measures on H xH are strong second order, as are their projections on H and H . From the measure π one has projections π i on (Hi , θ i ), i = , . Let m and m denote the mean elements and R and R the covariance operators of π and π . The first result of note is the definition and properties of the cross-covariance operator for the joint measure π . Denoting that operator by C , it is defined for all (u,v) in H xH by <x − m, u> dπ (x, y).
= ∫ H xH
Theorem C has representation C = R VR , V: H → H a unique linear operator having ∣∣V ∣∣ ≤ and P VP = V, Pi the projection of Hi onto the closure of range(Ri ). ⊡ /
Cross-Covariance Operators Charles R. Baker Professor Emeritus University of North Carolina, Chapel Hill, NC, USA
/
Next, we turn to the definition and properties of the covariance operator R of π . By direct computation (Baker ), one can show that this operator is defined on every element (u,v) in H xH by ∗ R (u, v) = (R u + C v, R v + C u)
This article will initially treat joint probability measures and their associated cross-covariance operators. Subsequently, attention will be shifted to three examples of problems on capacity of information channels. Cross-covariance operators were introduced in Baker () as a tool in solving a basic problem in information theory, and treated more extensively in Baker (). Related results are in Gualtierotti () and Fortet (). The empahsis in Baker () was in two directions: showing the added power of analysis obtained by introducing
∗ = (R ⊗ R )(u, v) + (C ⊗ C )(u, v).
We now give a result that illustrates both similarity and difference between a joint measure and the usual measure.as defined on one of the spaces H or H . We define a self-adjoint operator V in H xH by V(u,v) =(Vv, V∗ u) = (V∗ ⊗V)(u,v) and denote by I the identity operator in H xH ; it is shown in Baker () that ∣∣V∣∣ ≤ and that the non-zero eigenvalues of VV∗ are squares of the non-zero eigenvalues of V.
Cross-Covariance Operators
Theorem The covariance operator R of the measure / / π on H xH has representation R = R⊗ (I+V)R⊗ , where R⊗ is the covariance operator of the product measure π ⊗ π . If π is Gaussian, then π and π ⊗ π are mutually absolutely continuous if and only if V is Hilbert-Schmidt with ∣∣V∣∣ < , and otherwise orthogonal.⊡ The preceding results give some of the basic properties of the covariance operator of a joint measure, and it is seen that the cross-covariance operator is an essential component in the definition and properties of the covariance operator. In Baker (), considerable attention is given to Gaussian joint measures. However, it should be noted that the definition of the cross-covariance operator and its relation to the covariance operator hold for any strong-second order joint probability measure When the joint measure at hand is not Gaussian, one still has the cross-covariance operator available as well as the mean and the covariance operator. These functions can frequently be estimated from data and used to develop suboptimum but effective operations using (for example) second moment criteria. We now turn to a brief introduction to three probllems on the capacity of a Gaussian channel without feedback (Baker , ; Baker and Chao a, b). The cited papers provide examples of the use of results from Baker () in applications to information theory. The definition of the channel capacity is as follows. We have a joint measure π S,AS+N where S is the actual signal, AS is the transmitted coded signal (from a measurable space (Ω, Θ)) and AS+N is the received waveform of signal+noise from a measurable space (Ψ, Γ). The (average) mutual information will be finite if π S,AS+N is absolutely continuous with respect to its product measure π S⊗AS+N , and its value is then given by log[(dπ S,AS+N /dπ S⊗AS+N )(x, y)]dπ S,AS+N (x, y).
∫ ΩxΨ
The transmitted signal AS and the received AS+N can vary with choices by the coder (and the jammer in the third example below), and the channel capacity is the supremum of the mutual information over all admissible S and AS+N.pairs. In each case, the transmitted signal has a constraint given in terms of the ambient noise process. When the constraint on the transmitted signal is given in terms of the channel noise covariance, one says that the channel is “matched” (coder constraint is matched to the channel noise covariance) (Baker ). The second type of channel is “mismatched” (the signal constraint is not given in terms of the channel noise covariance) (Baker ). The third class is the jamming channel without feedback,
C
wherein the noise in the channel consists of a known Gaussian ambient noise (nature’s contribution) plus an independent noise that is under the control of a hostile jammer (Baker and Chao a, b). In this channel, there is a constraint on the jammer’s noise as well as one on the coder’s.transmitted signal. In the jamming channel, the jammer has no constraints on the choice of the probability distributions of the noise at his command. However, it is known (Ihara ) that if the channel noise due to nature is Gaussian, then the information capacity is minimized by the jammer choosing (among all processes satisfying the constraints) a Gaussian process. Thus, the original problem becomes a problem involving an ambient Gaussian noise (which is used to calculate the coder’s constraint) and an independent Gaussian process (jamming) giving the covariance constraint that the jammer uses.
Cross References 7Canonical Analysis and Measures of Association 7Measure Theory in Probability 7Statistical Signal Processing 7Statistical View of Information Theory
References and Further Reading Baker CR () Mutual information for Gaussian processes. SIAM J Appl Math ():– Baker CR () Joint measures and cross-covariance operators. Trans Am Math Soc :– Baker CR () Capacity of the Gaussian channel without feedback. Inf Cont :– Baker CR () Capacity of the mismatched Gaussian channel. IEEE Trans Inf Theory :– Baker CR, Chao IF (a) Information capacity of channels with partially unknown noise. I. Finite-dimensional channels. SIAM J Appl Math :– Baker CR, Chao IF (b) Information capacity of channels with partially unknown noise. II. Infinite-dimensional channels. SIAM J Cont Optimization :– Fortet RM () Vecteurs, fonctions et distributions aleatoires dans les espaces de Hilbert. Hermes, Paris (see pp. ff.) Fukumizu K, Bach FR, Jordan MJ () Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces. J Mach Learning Res :– Fukumizu K, Bach FR, Jordan MJ () Kernel dimensionality reduction in regression. Ann Stat :– Gretton A, Bousquet O, Smola AJ, Schölkopf B () Measuring statistical dependence with Hilbert–Schmidt norms. MPI Technical Report, Max Planck Institute for Biological Cybernetics. Tübingen, Germany’ Report Gualtierotti AF () On cross-covariance operators. SIAM J Appl Math ():– Ihara S () On the capacity of channels with additive non-Gaussian noise. Inf Cont :–
C
D Data Analysis Alfredo Rizzi Professor Sapienza Università di Rome, Rome, Italy
The Development of Data Analysis Data analysis began to be developed for use in statistical methodology in the early s. Today, it includes a number of techniques which allow an acceptable synthesis of information collected from n statistical units or objects which are each characterized by p qualitative or quantitative variables. The basics of certain data analysis techniques were established in the last century, principal components at the beginning of the s, factor analysis in the s, and automated classification methods in the s. During the same period, statistical methodology also saw development in statistical inference (classic and Bayesian) and in studies of interpretative models of complex phenomena (linear, loglinear, generalized linear, analysis of time series such as ARMA, ARIMA, autoregressive, etc.). Data analysis has made use and at times stimulated the development of two important “tools”: the language of matrices and appropriate computer software requiring hardware with large memory capacity and very fast access to information. The development of data analysis has been stimulated by the operational needs of its application in various sectors, in particular, business and the social sciences. For example, automatic classification methods are interesting applications used for classifying clients and market segments and for establishing homogeneity between company units and territorial administrative divisions. These methods are also applied in medicine for pattern recognition (see 7Pattern Recognition, Aspects of and 7Statistical Pattern Recognition Principles), for the automatic assessment of words used by speakers in different situations, as well as in many other sectors. Cluster analysis (see 7Cluster Analysis: An Introduction), when used in sample surveys of a population with known structural characteristics, allows one to reduce the variability within an equal sample
size, thereby increasing precision. Multiway matrix analysis is especially interesting when seeking trends in a single location at different times or in different locations at the same time. Methods such as 7multidimensional scaling allow us to represent a set of n objects belonging to a space with p > dimensions in spaces of two or three dimensions, so that the distortion of the matrix of the distance between points is minimal. 7Principal component analysis has also been the subject of in-depth studies which have opened the way for its use in a wide variety of fields. 7Correspondence analysis, introduced by Benzécri at the end of the s, is considered to be the acme of data analysis. The French school has, in some way, introduced this new sector of statistics to scholars and has carried out research in social sciences, economics, 7demography, and business using methodologies which have proved to be very useful in many research situations. Data analysis uses not only the matrix language and advanced software mentioned above but also methods of mathematical optimization, the theory of eigenvalues (see 7Eigenvalue, Eigenvector and Eigenspace) which are of great importance in many analyses, certain results from vector analysis, the theory of graphs and operations research along with some theorems of mathematical analysis, and the theory of linear spaces and fuzzy sets. Important and interesting contributions to classification have been made by scientists studying mathematical optimization methods, who have introduced rigorous algorithms to establish the partitions of a set of n objects characterized by p variables, which constitute the statistical population or group of reference. The studies carried out by Minkowski and Fréchet and other mathematicians on metrics are applicable to the realm of data analysis. Euclidean metrics is one of many but is often preferred by certain software for reasons of tradition. The L norm, for example, and the geometry based on it is, in many research situations, better suited than other metrics to represent complex phenomena in various fields. But metrics in the traditional mathematical sense have proved completely inadequate for the applications.
Miodrag Lovric (ed.), International Encyclopedia of Statistical Science, DOI ./----, © Springer-Verlag Berlin Heidelberg
D
Data Analysis
Therefore, these interesting studies of distance and similarity indices and, most importantly, of ultra-metrics have become an important part of data analysis. It is also of great importance in data analysis to be very careful about conclusions drawn from surveys which have been carried out using methodologies which are not justified by the hypotheses on which the research is based. The great progress and widespread availability of hardware and software has made it possible for those who are not students or specialists in the field to carry out research. This may lead to data processing done without in-depth knowledge of the methodology, techniques, or parametrization used; so, the default settings are chosen by the software without an understanding of the meaning of many statistical parameters which allow an a posteriori evaluation of the goodness of the data model adopted. The data coding step is of fundamental importance for the evaluation of any outcome although it is often given less attention than needed by those preferring to concentrate on the mathematical aspects of the analysis. Inferential problems also arise in data analysis. Data analysis has, in the first stage, been viewed only as a methodology for synthesizing information with any probabilistic or inferential approach excluded. Attempts to apply classical inferential methodologies requiring strong distribution hypotheses, especially in the multidimensional field, have not led to any worthwhile outcome. In some techniques – such as principal components and factor analysis – inferential aspects were considered to be of primary importance and were the subject of wide studies during the first half of the last century. As is often the case in scientific research, techniques used in statistical induction have proved to be unsuited to solving inferential problems in data analysis. Furthermore, in this field, induction has a wider meaning that is merely passing from “the sample to the population,” both in terms of an estimate of the parameters and of a verification of the hypotheses. The themes of fundamental importance here are those concerning the stability of results in terms of knowing to what extent the relations found between individuals and/or between variables can be considered valid. The relations between variables – linear and nonlinear – are often very strong. The verification of the hypotheses and the estimation of the parameters are not specific interests. One is nearly always concerned with data gathered from populations of a finite number of individuals, which makes the stability of the results extremely important. In sampling, on the other hand, it is important to carefully consider what the expected outcome is and what the meaning of the extension is. We are unaware of any studies of the
sampling plans capable of guaranteeing an acceptable level of representation of the samples extracted. This is also a completely new field of research regarding the methodological instruments usable in this important field.
The Techniques of Data Analysis The expression “data analysis” (analyse des données in French) or multidimensional data analysis (MDA) originated from the methodological approach first used in France in the late s (Benzécri ; Bertier and Bouroche ; Caillez and Pages , etc.). It includes two groups of multivariate statistical methods: classification methods or automatic clustering and linear data analysis and specifically principal component analysis, canonic correlation analysis, both simple and multiple correspondence analysis, and multidimensional scaling. During the s, MDA spreaded to other countries of Europe and was established as an autonomous branch of statistics. This created the conditions for new developments in both methodology and in applications, which characterize recent developments in multidimensional data analysis. There were important contributions in this area from Italian statisticians. In addition to those mentioned above, there are other methods available for multivariate statistical analysis: regression, 7analysis of variance, discriminant analysis (see 7Discriminant Analysis: An Overview and 7Discriminant Analysis: Issues and Problems), common factor analysis, etc.,inwhichprobabilitiesandinferentialaspectsaretreated. These may be called validation analyses for assumptions formulated about the multidimensional data set. Inferential and validation aspects come into play in MDA and can also be introduced, with some conditions, into the methods. In general, the various methods of 7multivariate statistical analysis have different ways, depending on the purpose, to analyze, describe, and synthesize the relations between statistical characteristics and statistical units or cases. It is useful to this purpose to divide the different methods of multivariate statistical analysis according to the role of the variables that come into play in the analysis. In the original MDA approach, the methods used for multivariate statistical analysis could be divided into: Symmetric Analyses: with variables of the same role (interdependence). The principals are: ● ● ● ● ● ●
Cluster analysis Principal component analysis Canonic correlation analysis Factor analysis of correspondences Common factor analysis Symmetric multidimensional scaling
Data Analysis
Asymmetric Analyses: where some variables are explicative, independent attributes of other variables, dependent and endowed with attributes or where pairs of variables or units are ranked as in: ● ● ● ● ●
Regression Analysis of variance Discriminant analysis Nonsymmetrical multidimensional scaling Path analysis
In the context of recent developments in MDA, such a split is no longer sharp because asymmetric variants have been introduced in all the main methods of data analysis. For example, in principal component analysis, the supplemental or passive variables can be treated as variables in a reference subspace instead of being handled as supplemental elements in the plots of the principal axes without having been used in the actual computations. Asymmetric variants have also been introduced in simple and multiple correspondence analysis. Multiway data analysis concerning particular extensions of the methods of data analysis is linked to asymmetric data analysis because of the different role it assigns to variables. Instead of the traditional data classification according the two criteria (statistical units x variables), these methods are applied, with different approaches, to data classified according to three or more criteria or statistical ways (statistical units x variables x occurrences). The combination of multivariate statistical methods and the automatic data processing necessary for this type of analysis is what brought about MDA. MDA is an important tool of 7data mining. One can consider data mining to be a procedure that starts from elementary data in a Data Warehouse to arrive at a decision. One can consider data mining to be the heir of artificial intelligence and expert systems. In each case, MDA is the basis for discovering data structure.
Three-Way Data Analysis Since the last decades of the twentieth century, there has been growing attention paid to the analysis of phenomena characterized by a set X of nkr variables, observed on a set of n units on r occasions (different times, places, etc.). These phenomena must be distinguished from the classical multivariate and are referred to as multivariatemultioccasion. The set X is called a three-way data set because its elements can be classified according to modes: units, variables, and occasions. The data structure associated with this data set is a three-indices or three-way array. The most widely collected three-way data set occurs when modes are units and variables and occasions are
D
different times. Here, we will especially refer to this data type. The repetition in time of the observation of the units allows us to evaluate the dynamics of the phenomenon differently from the classical case of a multivariate or cross-sectional (two-way) data set. There are several major advantages over a conventional cross-sectional or time series data set in using these, so-called three-way longitudinal data. The researcher has a larger number of data which increases the degree of freedom and reduces collinearity among explanatory variables and the possibility of making inferences about the dynamics of change from cross-sectional evidence. The three-way longitudinal data set may be given by: ●
●
●
●
Repeated recurring surveys with no overlapping units. That is, a survey organization repeats a survey on a defined topic, generally at regular time intervals. No overlaps of the sample units are required at different times. Examples of these surveys are given by the repeated analyses made by Central Bureau of Statistics in most countries. Repeated surveys with partially overlapping units. These surveys are also repeated at regular intervals. The survey design includes rotating units to allow variance reduction, i.e., the units are included in the analysis a number of times, then rotated out of the survey. Longitudinal surveys with no rotation of units. A set of units is followed over time with a survey designed with this aim. In the economic field, these collected data are called 7panel data. An example of this type is the Survey of Income. Longitudinal surveys with rotation. A group of units is followed for a period of time, so that new units are introduced and followed over time. These are repeated surveys with a partially overlapping units. An example of this type is the Monthly Retail Trade Survey.
In longitudinal surveys, it is generally the same variable which is observed on the units to allow comparison over time. Examples of the applications of the analyses of threeway data sets are found in many different fields, including chemometrics∗ , biology, economics. Different objectives can be pursued in analyzing threeway data sets: . To synthesize a three-way data set . To study the multiway variability of a three-way data set . To study the interrelationships between sets of elements of X: (a) Between sets of variates
D
D .
.
.
.
Data Depth
(b) Between sets of units (c) Between sets of occasions To define virtual modes or derivational modes for the three sets of modes of X: (a) Virtual units (b) Factors or latent variables (c) Virtual occasions To analyze individual differences in the judgments of different sets dissimilarities: (a) Three-way scaling of a replicated set of dissimilarities (replicated multidimensional scaling): metric and nonmetric Euclidean models (b) Weighted multidimensional scaling: metric and nonmetric weighted Euclidean models To simultaneously classify different sets of the elements of X: (a) Clustering of units (b) Clustering of variables (c) Clustering of occasions To give a linear or nonlinear model of X
About the Author Professor Alfredo Rizzi is past President of the Società Italiana di Statistica (–). He was Head of the Department of Statistica, Probabilità e Statistiche Applicate, Sapienza università di Roma (–). He was full professor of Teoria dell’inferenza statistica (– ), Facoltà di Scienze statistiche, Sapienza università di Roma. He was an elected member of the Italian Consiglio Nazionale delle Ricerche (CNR) (–). Professor Rizzi was a member of the Italian council of Istituto Nazionale di Statistica (–). He was the chairman of the Scientific and Organizing Committee of the th and th Symposia in Computational Statistics (Compstat) held in Rome in and , and the th conference of the International Federation of Classification Societies (IFCS), held in Rome in . Dr Rizzi has authored and co-authored more than papers and books including Inferenza Statistica (Utet, ).
Cross References 7Analysis of Areal and Spatial Interaction Data 7Analysis of Multivariate Agricultural Data 7Categorical Data Analysis 7Exploratory Data Analysis 7Functional Data Analysis 7Fuzzy Logic in Statistical Data Analysis 7Interactive and Dynamic Statistical Graphics 7Multi-Party Inference and Uncongeniality 7Multivariate Data Analysis: An Overview 7Statistical Analysis of Longitudinal and Correlated Data
7Statistical Fallacies: Misconceptions, and Myths 7Statistics Education
References and Further Reading Agresti A () An Introduction to categorical data analysis, nd edn. Wiley, Hoboken Benzécri JP () L’analyse des données, Tome : Taxinomie; Tome II Analyse des correspondances. Dunod, Paris Bertier P, Bouroche JM () Analyse des données, multidimensionel, Ed. P.U.F, France Billard L, Diday E () Symbolic data analysis. Conceptual statistics and data mining. Wiley, New York Caillez F, Pages JP () Introduction á l’analyse des données, smash, Paris Chatfield C, Collins AJ () Introduction to multivariate analysis. Chapman & Hall, London Greenacre MJ () Theory and applications of correspondence analysis. Academic, London Lebart L, Morineau A, Piron M () Statistique exploratoire multidimensionelle. Dunod, Paris
Data Depth Reza Modarres Chair and Professor of Statistics The George Washington University, Washington, D.C., USA Data depth provides a systematic approach to order multivariate observations in Rd . A data depth function is any function D(t; F) that measures the closeness or centrality of a point t ∈ Rd with respect to a distribution function F. Thus, a depth function assigns to each x ∈ Rd a nonnegative score as its center-outward depth with respect to F. Observations close to the center of F receive high ranks whereas peripheral observations receive low ranks. Hotelling () characterized the univariate median as the point which minimizes the maximum number of observations on one of its sides. That is, D(t; F) = min (F(t), − F(t)). This notion was generalized by Tukey (), giving rise to the definition of half-space or Tukey’s multivariate depth (Donoho ). The seminal work of Oja () and Liu () has renewed interest in data depth functions. Over the past two decades, various new notions of data depth have emerged as powerful explanatory and inferential tools for nonparametric multivariate analysis. Zuo and Serfling () proposed a formal framework for statistical depth functions based on the following four properties. Let F be the class of distributions on the Borel set on Rd and FX be the distribution function of a given random
Data Depth
vector X. Let D : Rd × F z→ R be bounded, nonnegative, and satisfy the following four conditions: ● ● ● ●
Affine invariance: D(At + b; FAX+b ) = D(t; FX ) holds for any random vector X, any d × d nonsingular matrix A, and any vectors t and b in Rd ; Maximality at center: D(θ; F) = supt∈Rd D(t; F) holds for any F ∈ F with center θ F (a point of symmetry); Monotonicity relative to deepest point: For any F ∈ F with center θ, D(t; F) ≤ D(θ + α(t − θ); F) holds for α ∈ [, ]; and Vanishing at infinity: D(t; F) → as ∥ t ∥→ ∞, for each F ∈ F.
A sample version of D(⋅; F) is denoted by D(⋅; Fm ), where Fm is the sample or empirical distribution function. Several novel depth functions have appeared in the literature, including Mahalanobis (MD) (Mahalanobis ), half-space (HSD) or Tukey’s depth (Tukey ), convex hull peeling (CHPD) (Mosler ), simplicial volume (SVD) or Oja’s depth (Oja ), majority (MJD) (Singh ), simplicial (SD) (Liu ), spatial (SPD) (Vardi and Zhang ), projection (PD) (Stahel ; Donoho, ), zonoid (ZD) (Koshevoy and Mosler ; Mosler, ); spherical (SPHD) (Elmore et al. ); and triangle (Liu and Modarres ) depth functions. Some of the above data depth functions are described below. Let F be a multivariate distribution function in Rd , d ≥ and X = {x , . . . , xm } be a random sample from F. The Mahalanobis depth MD(t; F) is defined as −
MD(t; F) = [ + (t − µ F )Σ− F (t − µ F )] , where µ F and Σ F are the mean vector and dispersion matrix of F, respectively. The sample version of MD(t; F) is obtained by replacing µ F and Σ F with their sample estimates. The half-space depth HSD(t; F) is defined as HSD(t; F) = inf {P(H) : H is a closed half-space in Rd H
and t ∈ H} = inf PF ({X : uT X ≥ uT t}) = inf
∥u∥=
u∈R d
PF ({X : u X ≥ u t}). T
T
The sample version of HSD(t; F) is HSD(t; Fm ) = min∥u∥= #{i : uT xi ≥ uT t}. m The simplicial depth SD(t; F) is defined as SD(t; F) = PF {t ∈ S [X , . . . , Xd+ ]} where S [X , . . . , Xd+ ] is a closed simplex formed by (d + ) random observations from F. The sample version of SD(t; F) is obtained by replacing F by Fm , or alternatively, by computing the fraction of the
D
sample random simplices containing the point t. The projection depth PD(t; F) (Zuo ) is defined as PD(t; F) = ( + O(t; F))− where O(t; F) is a measure of outlyingness of a point t w.r.t. F. For example, the outlyingness of a point t can be defined as the maximum outlyingness of t with respect to the univariate median in any one dimensional projection, that is, ∣uT t − Med(uT X)∣ O(t; F) = sup T ∥u∥= MAD((u X)) where Med(⋅) denotes the univariate median, MAD(⋅) denotes the univariate median absolute deviation; i.e., MAD(Y) = Med(∣Y − Med(Y)∣), and ∥⋅∥ is the Euclidean norm. The spatial depth SPD(t; F) or L depth is defined as SPD(t; F) = − ∥EF (
t−X )∥ . ∥t − X∥
The spherical depth SPHD(t; F) is defined as the probability that t is contained in the unique, closed random hypersphere, S(X , X ), formed by two random points X and X , which are i.i.d with c.d.f. F . That is, SPHD(t; F) = P(t ∈ S(X , X )). The sample version of the spherical depth at a point t is SPHD(t; Fm ) = m ∑m i<j I[t ∈ ()
S(xi , xj )]. The empirical convex hull peeling depth CHPD(t; Fm ) at t w.r.t. the sample X = {x , . . . , xm } is defined as CHPD(t, Fm ) = min{k : t ∈ Ek }, if such a k exists; and zero, otherwise, where Ek denotes the level k convex layer to which t belongs. To obtain a convex layer at level k, one initially construct the smallest convex hull which enclose all samples points {x , . . . , xm } and t. The points on the perimeter are designated the first convex layer and removed. The convex hull for the remaining points is now constructed; the points on the perimeter constitute the second layer. This process is repeated, and a sequence of nested convex layers is formed. The empirical zonoid depth ZD(t; Fm ) at t w.r.t. the sample X = {x , . . . , xm } is defined as ZD(t, Fm ) = sup{α : t ∈ Dα (x , . . . , xm )} where Dα (x , . . . , xm ) = m {∑ m ≤ λ i , αλ i ≤ n , for all i}. i= λ i xi : ∑i= λ i = , For a continuous distribution F on Rd , let L(X , X ) denote the hyper-lens defined by the intersection of two identical closed hyper-spheres centered at X and X , respectively, with radius ∥X − X ∥. The hyper-lens of any two i.i.d. random vectors X and X in Rd is defined as L(X , X ) = B(X , ∥X − X ∥) ∩ B(X , ∥X − X ∥) where B(c, r) is the closed ball of radius r centered at c. The triangle depth function TD(t; F) for a vector t ∈ Rd , with
D
D
Data Mining
respect to a distribution F on Rd , is the probability that t is contained in the random hyper-lens L(X , X ). That is, TD(t; F) = P(t ∈ L(X , X )) where X and X are i.i.d with c.d.f. F. Let X , . . . , Xm be an i.i.d. random sample with c.d.f. F. The sample (empirical) version of TD(t; F) is the proportion of L(Xi , Xj ), ≤ i < j ≤ m, that contain t. That is, TD(t; Fm ) = m ∑m i<j I[t ∈ L(xi , xj )]. ()
This is due to two main reasons: on one side the development of automatic methods of data acquisition, and on the other side the progress of storage technology producing the fall of related costs. This new environment involves all areas of human endeavor. ●
About the Author For biography Dependence.
see
the
entry
7Measures
of
Cross References 7Multivariate Outliers 7Multivariate Rank Procedures: Perspectives and
Prospectives 7Statistical Quality Control: Recent Advances
References and Further Reading Donoho DL () Breakdown properties of multivariate location estimators. PhD qualifying paper, Department of Statistics, Harvard University Elmore RT, Hettmansperger TP, Xuan F () Spherical data depth and a multivariate median. In: Liu R, Serfling R, Souvaine D (eds) Proceedings of data depth: robust multivariate analysis, computational geometry and applications, pp – Hotelling H () Stability in competition. Econ J :– Koshevoy G, Mosler K () Zonoid trimming for multivariate distributions. Ann Stat :– Liu RY () On a notion of data depth based on random simplices. Ann Stat :– Liu Z, Modarres R () Triangle data depth. Technical report. Department of Statistics, George Washington University Mahalanobis PC () On the generalized distance in statistics. Proc Natl Inst Sci India :– Mosler K () Multivariate dispersion, central regions and depth. Lecture notes in statistics. Springer, Berlin Oja H () Descriptive statistics for multivariate distributions. Stat Probab Lett :– Singh K () Majority depth. Technical report. Rutgers University Stahel WA () Robust estimation: infinitesimal optimality and covariance matrix estimators. PhD thesis, ETH, Zurich (in German)
●
●
●
Data Mining Bruno Scarpa University of Padua, Padua, Italy
In recent decades technological innovation has made the availability of large and sometimes huge amounts of information on a phenomenon of interest simple and cheap.
●
Every month a supermarket chain releases millions of receipts, one for each shopping trolley checking out. The content of each trolley summarizes the needs, the propensities and the economic behavior of the customer that selected it. The collection of all these shopping lists forms an important information base for the supermarket in order to decide the sales and purchases politics. Such an analysis becomes even more interesting when each shopping list is connected with the customers loyalty cards, allowing to follow the single client behavior by recording the purchases sequences; a similar problem arises in analyzing the usage of credit cards. Telecommunication companies generate every day data on millions of phone calls and of other services. Companies are interested in analyzing the customers behavior in order to measure chances of up sell and cross sell, and to identify potential churners (customers with high propensity to move to others operators). The growth of the World Wide Web in the last years, and its success and usage in research, business and daily life, makes it the largest publicly accessible data source in the world. Analysts often need to extract knowledge from this huge source of available data. Search engines need to identify the small part of the documents that are related to each single query; this operation is complicated by a number of elements: (a) the total size of the set of documents is amazing, (b) data are not collected in a structured form, such as a well organized data base, (c) inside each single document the elements related with the specific query are not in a pre-specified position either with respect to the entire document or among themselves. In scientific research huge amount of data are also available, for example in microbiology for the study of DNA structure. The analysis of sequences of portions of DNA produces very large tables, called “DNA microarray”, where each column represents a sequence of some thousands of numerical values associated to DNA of an individual and each row represents different individuals. The goal is, for example, to connect the configuration of these sequences with the occurrence of some disease. The set of physical, chemical or other measurements finalized to examining the Earth’s climate is becoming massive. Even the simple structured organization of
Data Mining
●
such data is problematic as much as the analysis to summarize all these pieces of information. Digital images of the sky will create massive data sets of observational data regarding many different features of hundreds of millions of sky objects. Astronomers are often interested in finding some algorithm that can perform as well as human experts in classifying stars and galaxies.
These are only few examples, but from all of these (and from many other that could be provided), it is clear that nowadays the analysis of huge amount of data is often required in order to solve real world problems. Specific statistical tools are required in order to extract important and useful knowledge from such an abundant amount of information. In fact, from one side the large number of cases are difficult to visualize effectively and, on the other side, as dimensionality increases, it becomes more difficult to describe data. Also, as we have seen in some of the examples above, data may not assume the simple structure of a database and sometimes they are even collected from streaming (e.g., for recording electricity usage). The exploration and analysis of huge amount of data is often called “data mining” recalling the extraction (of relevant knowledge) from a mine of data: 7 Data mining is the activity of graphical and numerical analysis of large observational data sets or streaming data in order to find useful knowledge for the data owner.
“Useful knowledge” in this context is quite generic, since very often it is not a priori specified what is the object of interest, which is in general acquired by “mining” the data. From such a definition, it is clear that data mining is an interdisciplinary exercise where statistics, database technology, machine learning, pattern recognition (see 7Pattern Recognition, Aspects of and 7Statistical Pattern Recognition Principles), artificial intelligence and visualization play a role (Hand et al. ). Therefore, data mining requires an understanding of both statistical and computational issues. Looking at it as a statistician, (e.g., Azzalini and Scarpa ) it is worthwhile outlining some statistical specificities of data mining: ●
The size of the data is huge. When many variables are available, the problem of the curse of dimensionality appears: the number of unit cells in a space as the number of variables increases linearly has an exponential growth rate, so that, in high dimensional spaces, “nearest” points may be very far away. Also, when the number of cases is of the order of millions, computational issues need to be considered. “Every time the
●
●
● ●
●
D
amount of data increases by a factor of ten, we should totally rethink how we analyze it” (Friedman ). The observational context: data are often not collected from an experiment, but purely observational. They just “exists.” Often they have been collected by convenience or opportunity, rather than randomly. Data used for the analysis have often been collected for some other purpose (e.g., they may have been collected in order to maintain an up-to-date record of all the transactions in a bank). Data may be dirty, corrupted, contaminated, missing and need to be preprocessed in order to be used. Sampling and population issues: the huge amount of data can be tackled by sampling, if the aim is modeling, but not necessarily so if the aim is pattern detection. When data sets are entire populations (e.g., the entire customer base of a company), the standard statistical notion of inference has no relevance. “If you torture data long enough, Nature will always confess” (Coase ), with so large data sets it would be easy to find a model that will fit the data well, but often this model will not really describe the phenomenon of interest. In this context the trade off between bias and variance needs to be managed.
In summary, while data mining does overlap considerably with the standard exploratory data analysis techniques of statistics, it also runs into new problems, many of which are consequences of size and the non-traditional nature of the data sets involved. Using a machine learning classification, data mining tasks may be divided into two groups, the supervised learning problems, “supervised” because of the presence of an outcome variable to guide the learning process, and the unsupervised learning problems, (also said in a more statistical language internal analysis problems) where we observe only the features and have no measurements of the outcome. Often the aim of supervised learning is to build a model that will permit the prediction of the value of one variable from known values of other variables. If the variables being predicted (called dependent variables or outputs or responses) are categorical, the problem is termed classification, while if they are quantitative the problem is called regression. A large number of methods have been developed in statistics and machine learning to tackle these problems, from simple parametric models such as 7generalized linear models (McCullagh and Nelder ) to nonparametric structures such as generalized additive models (Hastie and Tibshirani ), 7neural networks (e.g., Ripley ), support vector machines (Vapnik ), classification and regression trees (Breiman et al. )
D
D
Data Mining
and their resampling versions such as bagging (Breiman ), boosting (Freund and Schapire ) or random forests (Breiman a) and many others that have been developed in recent years. Unsupervised learning is a class of problems in which one seeks to determine how the data are organized, directly inferring the properties of the density of a multivariate variable without the help of a “teacher” providing correct answers. Classical statistics and machine learning literature (e.g., Kaufman and Rousseeuw ; Ripley ) provide many clustering algorithms with the aim of grouping or segmenting a collection of objects into subsets or “clusters,” such that those within each cluster are more closely related to one another than objects assigned to different clusters. Another family of algorithmic techniques based on association rules (e.g., Hastie, Tibshirani and Friedman ) is related to the task of finding combinations of items that occur frequently in transaction databases. Other goals of interest, in this context, are on detecting patterns among data, like spotting fraudulent behavior among customers of a company, or detecting unusual stars or galaxies in astronomical work in order to identify previously unknown phenomena. A particular case of this pattern detection is when a pattern of interest is already known by the user and he is interested in finding similar patterns in the data set. This task is very common in text and image data sets, where the pattern can be a set of keywords, and the user may wish to find relevant documents within a large set of possibly important documents, or the user may have a sample image, and wish to find similar images from a large set of images. Most of the methods included in data mining are often considered as automatic tools that do not require any human intervention. In particular many algorithms generated in the machine learning context, dealing with really huge amount of data with, sometimes, very complicated computational structure, and seems to present an objective advantage by working without any human intervention. However, from empirical experience, it seems that an analysis driven by a thinking brain is much more powerful. In fact the necessity to “understand the problem” characterizes the style of statistical data mining, in all the phases of the analysis from the data preparation to the interpretation of the results. Even when black box algorithms are used, one can never hope to solve every problem by using software on a powerful computer just by “pushing a button”. A deep knowledge of the nature of the tools used and of how the used methods work is crucial for, at least, three broad reasons (Azzalini and Scarpa ): . Understanding the characteristics of the tools is crucial in choosing the right method.
. The same mastery is required in order to correctly interpret the results of the algorithms. . Some ability in the computational and algorithmic aspects is very useful in order to better evaluate the computer output, considering also its reliability. The paper by Breiman (b), with the discussion that it generated, describes quite well the specificities of the type of statisticians needed to tackle these problems.
About the Author Dr. Bruno Scarpa joined the Department of Statistical Sciences, University of Padova, in , after four years being at the Department of Applied Statistics at the University of Pavia. Previously, he used to work as a manager for some telecommunication companies where he was in charged of activities of customer profiling, churn analysis, CRM and data mining services for marketing. He is author of several scientific papers and jointly with Adelchi Azzalini is author of the book Data Mining and Data Analysis (published in by Springer in Italian and soon will be published in English).
Cross References 7Business Forecasting Methods 7Data Mining Time Series Data 7Preprocessing in Data Mining 7Statistics: An Overview
References and Further Reading Azzalini A, Scarpa B () Data analysis and data mining. Oxford University Press, New York Breiman L, Friedman JH, Olshen RA, Stone CJ () Classification and regression trees. Wadsworth International, Monterey, CA Breiman L () Bagging predictors. Mach Learn :– Breiman L (a) Random forests. Mach Learn :– Breiman L (b) Statistical modeling: the two cultures (with discussion). Stat Sci :– Coase RH () How should economists choose? American Enterprise Institute for Public Policy Research, Washington, DC Friedman JH () Data mining and statistics: what’s the connection? In: Proceedings of the th symposium on the interface: computing science and statistics. Houston, TX, May . Available at http://www- stat.stanford.edu/~jhf/ftp/dm- stat.pdf Freund Y, Schapire R () Experiments with a new boosting algorithm. In: Machine learning: proceedings of the thirteenth international conference, Morgan Kauffman, San Francisco, pp – Hand D, Mannila H, Smyth P () Principles of data mining. MIT Press, Cambridge Hastie T, Tibshirani R () Generalized additive models. Chapman and Hall, London Hastie T, Tibshirani R, Friedman JH () The elements of statistical learning: data mining, inference, and prediction. Springer, New York Kaufman L, Rousseeuw PJ () Finding groups in data: an introduction to cluster analysis. Wiley, New York
Data Mining Time Series Data
McCullagh P, Nelder JA () Generalized linear models. Chapman and Hall, London Ripley BD () Pattern recognition and neural networks. Cambridge University Press, Cambridge Vapnik V () The nature of statistical learning theory. Springer, New York
●
●
Data Mining Time Series Data Eamonn J. Keogh Professor, University Scholar University of California-Riverside, Riverside, CA, USA Time series data is ubiquitous; large volumes of time series data are routinely created in medical and biological domains, examples include gene expression data (Aach and Church ), electrocardiograms, electroencephalograms, gait analysis, growth development charts etc. Although statisticians have worked with time series for more than a century, many of their techniques hold little utility for researchers working with massive time series databases (for reasons discussed below). Below are the major task considered by the time series data mining community: ●
●
● ● ●
●
Indexing (Query by Content): Given a query time series Q, and some similarity/dissimilarity measure D(Q,C), find the most similar time series in database DB (Chakrabarti et al. ; Faloutsos et al. ; Kahveci and Singh ; Popivanov and Miller ). Clustering: Find natural groupings of the time series in database DB under some similarity/dissimilarity measure D(Q,C) (Aach and Church ; Debregeas and Hebrail ; Kalpakis et al. ; Keogh and Pazzani ). Classification: Given an unlabeled time series Q, assign it to one of two or more predefined classes (Geurts ; Keogh and Pazzani ). Prediction (Forecasting): Given a time series Q containing n datapoints, predict the value at time n + . Association Detection: Given two or more time series, find relationships between them. Such relationships may or may not be casual and may or may not exist for the entire duration of the time series. Summarization: Given a time series Q containing n datapoints where n is an extremely large number, create a (possibly graphic) approximation of Q which retains
D
its essential features but fits on a single page, computer screen etc. (Indyk et al. ; van Wijk and van Selow ). Anomaly Detection (Interestingness Detection): Given a time series Q, assumed to be normal, and an unannotated time series R. Find all sections of R which contain anomalies or “surprising/interesting/unexpected” occurrences (Guralnik and Srivastava ; Keogh et al. ; Shahabi et al. ). Segmentation: Given a time series Q containing n datapoints, construct a model Q, from K piecewise segments (K << n) such that Q closely approximates Q (Keogh and Pazzani ).
Note that indexing and clustering make explicit use of a distance measure, and many approaches to classification, prediction, association detection, summarization and anomaly detection make implicit use of a distance measure. For this reason, there has been significant research on finding the “best” distance measure for time series (Keogh and Kasetty ). Recent work, however, suggests that the simple Euclidean distance is surprisingly competitive (Ding et al. ). It is interesting to note that with the exception of indexing, research into the tasks enumerated above predate not only the decade old interest in 7data mining, but computing itself. What then, are the essential differences between the classic, and the data mining versions of these problems? The key difference is simply one of size and scalability; time series data miners routinely encounter datasets that are gigabytes in size. As a simple motivating example, consider hierarchal clustering. The technique has a long history, and well-documented utility. If however, we wish to hierarchically cluster a mere million items, we would need to construct a matrix with cells, well beyond the abilities of the average computer for many years to come. A data mining approach to clustering time series, in contrast, must explicitly consider the scalability of the algorithm (Kalpakis et al. ). In addition to the large volume of data, it is often the case that each individual time series has a very high dimensionality (Chakrabarti et al. ). Whereas classic algorithms assume a relatively low dimensionality (e.g., a few measurements such as “height, weight, blood sugar etc.”), time series data mining algorithms must be able to deal with dimensionalities in the hundreds and thousands. The problems created by high dimensional data are more than mere computation time considerations, the very meanings of normally intuitive terms such as “similar to” and “cluster forming” become unclear in high dimensional space. The reason is that as dimensionality increases, all objects
D
D
Data Mining Time Series Data
become essentially equidistant to each other, and thus classification and clustering lose their meaning. This surprising result is known as the “curse of dimensionality” and has been the subject of extensive research (Aggarwal et al. ). The key insight that allows meaningful time series data mining is that although the actual dimensionality may be high, the intrinsic dimensionality is typically much lower. For this reason, virtually all time series data mining algorithms avoid operating on the original “raw” data, instead they consider some higher-level representation or abstraction of the data.
Time Series Representations As noted above, time series datasets are typically very large, for example, just h of electroencephalogram data can require in excess of a gigabyte of storage. This is a problem because for almost all data mining tasks, most of the execution time spent by algorithm is used simply to move data from disk into main memory. This is acknowledged as the major bottleneck in data mining, because many naïve algorithms require multiple accesses of the data. As a simple example, imagine we are attempting to do k-means clustering of a dataset that does not fit into main memory. In this case, every iteration of the algorithm will require that data in main memory to be swapped. This will result in an algorithm that is thousands of times slower than the main memory case. With this in mind, a generic framework for time series data mining has emerged. The basic idea is summarized in (Table ). It should be clear that the utility of this framework depends heavily on the quality of the approximation created in Step . If the approximation is very faithful to the original data, then the solution obtained in main memory is likely to be the same, or very close to, the solution we would have obtained on the original data. The handful of disk accesses made in Step to confirm or slightly modify
Data Mining Time Series Data. Table A generic time series data mining approach . Create an approximation of the data, which will fit in main memory, yet retains the essential features of interest . Approximately solve the problem at hand in main memory . Make (hopefully very few) accesses to the original data on disk to confirm the solution obtained in Step , or to modify the solution so it agrees with the solution we would have obtained on the original data
the solution will be inconsequential compared to the number of disks accesses required if we had worked on the original data. With this in mind, there has been a huge interest in approximate representation of time series. Figure illustrates a hierarchy of every major representation proposed in the literature. Given the plethora of different representations, it is natural to ask which is best. Recall that the more faithful the approximation, the less clarification disks accesses we will need to make in Step of Table . There have been many attempts to answer the question of which is the best representation, with proponents advocating their favorite technique (Chakrabarti et al. ; Faloutsos et al. ; Popivanov and Miller ; Rafiei and Mendelzon ). The literature abounds with mutually contradictory statements such as “Several wavelets outperform the . . . DFT” (Popivanov and Miller ), “DFT-based and DWT-based techniques yield comparable results” (Wu et al. ), and “Haar wavelets perform. . . better that DFT” (Kahveci and Singh ). However an extensive empirical comparison on diverse datasets suggests that while some datasets favor a particular approach, overall there is little difference between the various approaches in terms of their ability to approximate the data (Ding et al. ; Keogh and Kasetty ). There are however, other important differences in the usability of each approach (Chakrabarti et al. ). We will consider some representative examples of strengths and weaknesses below. The wavelet transform is often touted as an ideal representation for time series data mining, because the first few wavelet coefficients contain information about the overall shape of the sequence while the higher order coefficients contain information about localized trends (Popivanov and Miller ; Shahabi et al. ). This multiresolution property can be exploited by some algorithms, and contrasts with the Fourier representation in which every coefficient represents a contribution to the global trend (Faloutsos et al. ; Rafiei and Mendelzon ). However wavelets do have several drawbacks as a data mining representation. They are only defined for data whose length is an integer power of two. In contrast, the Piecewise Constant Approximation suggested by (Yi and Faloutsos ), has exactly the fidelity of resolution of as the Haar wavelet, but is defined for arbitrary length time series. In addition, it has several other useful properties such as the ability to support several different distance measures (Yi and Faloutsos ), and the ability to be calculated in an incremental fashion as the data arrives (Chakrabarti et al. ). Choosing the right representation for the task
Data Mining Time Series Data
D
Time series representations Data adaptive
Sorted coefficients
Non data adaptive
Singular Symbolic Piecewise value polynomial decomposition
Piecewise linear approximation
Adaptive Natural piecewise language constant approximation Interpretation Regression
Trees
Strings
Wavelets
Random mappings
Orthonormal Bi-Orthonormal Discrete
fourier transform
Haar Daubechies Coiflets dbn n > 1
Piecewise aggregate approximation
Spectral
Discrete cosine transform
Symlets
Data Mining Time Series Data. Fig. A hierarchy of time series representations
at hand is the key step in any time series data-mining endeavor. The points above only serve as a sample of the issues that must be addressed.
indexing with a lower-bounding distance measure, and LB_Keogh, a tool for lower bounding various time series distance measure, that makes retrieval of time-warped time series feasible even for large data sets.
Readings The field of time series data mining is relatively new, and ever changing. Because of the length of journal publication delays, the most interesting and useful work tends to appear in top-tier conference proceedings. Interested readers are urged to consult the latest proceedings of the major conferences in the field. These include the ACM Knowledge Discovery in Data and Data Mining, IEEE International Conference on Data Mining and the IEEE International Conference on Data Engineering.
About the Author Eamonn J. Keogh received his Ph.D. in Computer Science in , UC Irvine. His research areas include data mining, machine learning and information retrieval, specializing in techniques for solving similarity and indexing problems in time-series datasets. Dr. Keogh is a prolific author in data mining/database conferences. He has (co)authored more than papers. His papers on time series data mining have been referenced well over , times (his H-index is ). He received the IEEE ICDM best paper award, SIGMOD best paper award, and runner up best paper award in KDD . He has made some important contributions in data mining time series data, including: Symbolic Aggregate approXimation (SEX) (with Jessica Lin, ), the first symbolic representation for time series that allows for dimensionality reduction and
Cross References 7Business Forecasting Methods 7Data Mining 7Distance Measures 7Hierarchical Clustering 7Time Series
References and Further Reading Aach J, Church G () Aligning gene expression time series with time warping algorithms. Bioinformatics : – Aggarwal C, Hinneburg A, Keim DA () On the surprising behavior of distance metrics in high dimensional space. In: Proceedings of the th international conference on database theory, London, UK, – Jan , pp – Chakrabarti K, Keogh E, Pazzani M, Mehrotra S () Locally adaptive dimensionality reduction for indexing large time series databases. ACM T Database Syst ():– Ding H, Trajcevski G, Scheuermann P, Wang X, Keogh E () Querying and mining of time series data: experimental comparison of representations and distance measures VLDB Debregeas A, Hebrail G () Interactive interpretation of kohonen maps applied to curves. In: Proceedings of the th Int’l conference of knowledge discovery and data mining, New York, NY, – Aug , pp – Faloutsos C, Ranganathan M, Manolopoulos Y () Fast subsequence matching in time-series databases. In: Proceedings of the ACM SIGMOD Int’l conference on management of data, Minneapolis, MN, – May , pp –
D
D
Data Privacy and Confidentiality
Geurts P () Pattern extraction for time series classification. In: Proceedings of principles of data mining and knowledge discovery, th European conference, Freiburg, Germany, – Sept , pp – Guralnik V, Srivastava J () Event detection from time series data. In: Proceedings of the th ACM SIGKDD Int’l conference on knowledge discovery and data mining, San Diego, CA, – Aug , pp – Indyk P, Koudas N, Muthukrishnan S () Identifying representative trends in massive time series data sets using sketches. In: Proceedings of the th Int’l conference on very large data bases, Cairo, Egypt, – Sept , pp – Kahveci T, Singh A () Variable length queries for time series data. In: Proceedings of the th Int’l conference on data engineering, Heidelberg, Germany, – Apr , pp – Kalpakis K, Gada D, Puttagunta V () Distance measures for effective clustering of ARIMA time-series. In: Proceedings of the IEEE Int’l conference on data mining, San Jose, CA, Nov –Dec , pp – Keogh E, Lonardi S, Chiu W () Finding surprising patterns in a time series database in linear time and space. In: The th ACM SIGKDD international conference on knowledge discovery and data mining, Edmonton, Alberta, Canada, – July , pp – Keogh E, Pazzani M () An enhanced representation of time series which allows fast and accurate classification, clustering and relevance feedback. In: Proceedings of the th Int’l conference on knowledge discovery and data mining, New York, NY, – Aug , pp – Keogh E, Kasetty S () On the need for time series data mining benchmarks: a survey and empirical demonstration. In: The th ACM SIGKDD international conference on knowledge discovery and data mining, Edmonton, Alberta, Canada, – July , pp – Popivanov I, Miller RJ () Similarity search over time series data using wavelets. In: Proceedings of the th Int’l conference on data engineering, San Jose, CA, Feb –Mar , pp – Rafiei D, Mendelzon AO () Efficient retrieval of similar time sequences using DFT. In: Proceedings of the th Int’l conference on foundations of data organization and algorithms, Kobe, Japan, – Nov Shahabi C, Tian X, Zhao W () TSA-tree: a wavelet based approach to improve the efficiency of multi-level surprise and trend queries. In: Proceedings of the th Int’l conference on scientific and statistical database management, Berlin, Germany, – Jul , pp – van Wijk JJ, van Selow E () Cluster and calendar-based visualization of time series data. In: Proceedings IEEE symposium on information visualization, IEEE Computer Society, – October , pp – Wu Y, Agrawal D, El Abbadi A () A comparison of DFT and DWT based similarity search in time-series databases. In: Proceedings of the th ACM CIKM Int’l conference on information and knowledge management, McLean, VA, – Nov , pp – Yi B, Faloutsos C () Fast time sequence indexing for arbitrary lp norms. In: Proceedings of the th Int’l conference on very large databases, Cairo, Egypt, – Sept , pp –
Data Privacy and Confidentiality Stephen E. Fienberg , Aleksandra B. Slavkovi´c Maurice Falk University Professor Carnegie Mellon University, Pittsburgh, PA, USA Associate Professor The Pennsylvania State University, University Park, PA, USA
Introduction Data privacy is an overarching concern in modern society, as government and non-government agencies alike collect, archive, and release increasing amounts of potentially sensitive personal data. Data owners or stewards, in the case of statistical agencies, often critically evaluate both the type of data that they make publicly available and the format of the data product releases. The statistical challenge is to discover how to release important characteristics of existing databases without compromising the privacy of those whose data they contain. Modern databases, however, pose new privacy problems due to the types of information they hold and their size. In addition to traditional types of information contained in censuses (see 7Census), surveys, and medical and public health studies, contemporary information repositories store social network data (e.g., cell phone and Facebook data), product preferences (e.g., from commercial vendors), web search data, and other statistical information that was hitherto unavailable in digital format. The information in modern databases is also more commercially exploitable than pure census data (e.g., credit cards, purchase histories, medical history, mobile device locations). As the amount of data in the public realm accumulates and record-linkage methodologies improve, the threat to confidentiality and privacy magnifies. Repeated database breaches demonstrate that removing obviously identifying attributes such as names is insufficient to protect privacy (e.g., Narayanan and Shmatikov ; Backstrom et al. ). Even supposed anonymization techniques can leak sensitive information when the intruder has modest partial knowledge about the data from external sources (e.g., Coull et al. ; Ganta et al. ). Two rich traditions of investigating data confidentiality have developed within the scientific community, one in statistics, e.g., see Federal Committee on Statistical Methodology (), and the other more recently in computer science. Both tackle the fundamental tradeoff between utility and privacy, but in essentially different ways. In statistics the focus has been on the trade-off
Data Privacy and Confidentiality
between disclosure risk and data utility while in computer science the focus has been on algorithmic aspects of the problem and more recently on rigorous definitions of privacy that may also allow for utility. A special issue of the Journal of Privacy and Confidentiality features survey articles describing the current approaches and open research problems (http://repository.emu.edu/jpe/).
Statistical Disclosure Limitation Statistical disclosure limitation (SDL) applies and develops statistical tools for limiting releases of sensitive information from statistical databases while allowing for valid statistical inference. Emanating from official statistics involving censuses and large-scale national surveys, modern SDL emphasizes statistical inference as the main yardstick of utility; e.g., given a statistical model, how well can one “anonymize” the data and still carry out valid model estimation and assessment? The goal of disclosure limitation is to examine and manage a trade-off between data utility and disclosure risk (e.g., Doyle et al. ; Trottini and Fienberg ; Duncan and Stokes ). Data utility is a measure of usefulness of a dataset for an intended analyst. Disclosure risk measures the degree to which a dataset and its released statistics reveal sensitive information. This notion is probabilistic; as released data accumulate in the public domain, the probability of uniquely identifying members of the population increases. Starting with Dalenius () and continuing to modern work with connections to algebraic geometry, multivariate analysis, optimization, and the generation of synthetic data, SDL has a vast literature with primary applications to social sciences and health data. In general, SDL methods introduce bias and variance to data in order to minimize identity and attribute disclosure while trying to retain sufficient information needed for proper statistical inference. Data masking involves transforming an n × p (cases by variables) original data matrix Z through preand post-multiplication and the possible addition of noise. That is, Z = AZB + C, where A is a matrix that operates on the n cases, B is a matrix that operates on the p variables, and C is a matrix that adds noise. Matrix masking can be applied to either microdata or table of counts, and includes a variety of standard approaches to disclosure limitation (see 7Statistical Approaches to protecting confidentiality in public use data): (a) recodings such as (e.g., rounding and thresholding), (b) cell suppression, (c) data swapping, and (d) perturbation. More recent methods include sampling which involves releasing subsets of data or variables – deleting rows that is columns of Z, and simulations such as adding rows to Z (e.g., see Reiter for generation of synthetic Microdata, Slavkovic and Lee
D
for creation of synthetic tables, and Dobra et al. for partial information release, including optimization bounds on contingency tables).
Computer Science Approaches Closely related to SDL techniques are privacy-preserving data mining (PPDM) methods that aim to construct efficient data mining algorithms (see 7Data Mining) while maintaining privacy with the emphasis on automation and scalability of the anonymization process. Currently proposed methods can be roughly grouped in (a) randomization methods (e.g., multiplicative perturbations, data swapping), (b) group based anonymization, (c) distributed PPDM, and (d) privacy-preservation of application rules (e.g., see Fienberg and Slavkovi´c for association rule hiding). For an overview of the PPDM models and algorithms, and related issues see Aggarwal and Yu (). Research emerging from cryptography has emphasized that privacy and security are properties of the anonymization algorithm, rather than of a particular output. This perspective allows for rigorous definitions and proofs of privacy. In particular, a concept called differential privacy (e.g., Dwork et al. ) provides rigorous guarantees no matter what external information is available to an intruder. The algorithmic perspective also leads to utility being defined in terms of functional approximation: given a database x and a function f , how well does the released information allow one to approximate f (x). Some recent developments aim to establish connections between differential privacy and traditional statistical inference; e.g., see Smith () for private parametric estimation and Wasserman and Zhou () for approximation of smooth densities. In some cases, the theory has sufficiently advanced for the development of concrete methodological guidelines; e.g., related to the additive perturbation techniques of Dwork et al. (), Vu and Slavkovi´c () have begun to develop rules for sample size calculations and statistical 7power analysis. Machanavajjhala et al. () describe the first application of privacy tools to large-scale public-use databases by creating differentially private synthetic data based on Bayesian posterior predictive distributions. Another more recent promising approach uses cryptographic protocols for distributing privacy-preserving algorithms for valid statistical analysis among a group of servers (or data owners) so as to avoid pooling data in any single location. Using ideas from secure multiparty computation (Lindell and Pinkas ), statisticians have been working on variations of secure protocols for generalized linear models analysis (e.g., Ghosh et al. ).
D
D
Data Privacy and Confidentiality
Future Directions Protecting privacy in statistical databases is an increasingly challenging problem. Any useful statistics extracted from a database must reveal some information about the individuals whose data are included. Despite an increasing focus on large and often sparse data sets, the evaluation of disclosure risk and utility of statistical results often involves human-guided tuning of parameters and context-specific notions of privacy. New policies and data privacy research calls for methodology that achieves maximum utility, minimal risk and transparency. Current trends aim at integration of the computationally focused, rigorous definitions of privacy and cryptographic protocols emanating from computer science with notions of utility from statistics.
Acknowledgments Supported in part by National Science Foundation grant DMS- and U.S. Army Research Office Contract WNF--- to Carnegie Mellon University. Supported in part by National Science Foundation grant SES- to the Department of Statistics, Pennsylvania State University.
About the Authors Stephen E. Fienberg is Maurice Falk University Professor of Statistics and Social Science at Carnegie Mellon University, with appointments in the Department of Statistics, the Machine Learning Department, Cylab, and i-Lab. Professor Fienberg is Past President of the Institute of Mathematical Statistics (–), and Past President of the International Society for Bayesian Analysis (–) and he has been Vice President of the American Statistical Association. He is an Elected member of the U.S. National Academy of Sciences () and co-chair of its Report Review Committee (–). He is a fellow of the American Academy of Arts and Sciences (), the American Academy of Political and Social Science (), and the Royal Society of Canada (). Professor Fienberg is an Editor of the Annals of Applied Statistics (–), and former editor of the Journal of the American Statistical Association. He was Co-Founder of the Journal of Privacy and Confidentiality and Chance. He has received the Wilks Award () and the Founders Award () from the American Statistical Association, and the Lise Manchester Award from the Statistical Society of Canada (). Aleksandra Slavkovic received her Ph.D. from Carnegie Mellon University in . She is currently an Associate Professor of Statistics, with appointments at the Department of Statistics and the Institute for CyberScience at Penn State University, University Park, and at the Department of Public Health Sciences, Pennsylvania State College
of Medicine, Hershey. She is currently serving as an Associate Editor of the Annals of Applied Statistics, Journal of Privacy and Confidentiality and Journal of Statistical Computation and Simulation. Dr. Slavkovic has served as a consultant to the National Academy of Sciences/National Research Council Committee to Review the Scientific Evidence of Polygraph in and part of . In , she received and honorable mention for the best student paper from the Committee on Statisticians in Defense and National Security of the American Statistical Association.
Cross References 7Census 7Federal Statistics in the United States, Some Challenges 7Multi-Party Inference and Uncongeniality 7Statistical Approaches to Protecting Confidentiality in
Public Use Data
References and Further Reading Aggarwal C, Yu P () Privacy-preserving data mining: models and algorithms. Springer, New York Backstrom L, Dwork C, Kleinberg J () Wherefore art thou rx? Anonymized social networks, hidden patterns, and structural steganography. In: Proceedings of th international World Wide Web conference, Banff, Alberta, Canada Coull S, Wright C, Monrose F, Keromytis A, Reiter M () Taming the devil: techniques for evaluating anonymized network data. In: Fifteenth annual Network & Distributed System Security Symposium (NDSS), San Diego, CA, USA Dalenius T () Towards a methodology for statistical disclosure control. Stat Tidskr :– Dobra A, Fienberg S, Rinaldo A, Slavkovi´c A, Zhou Y () Algebraic statistics and contingency table problems: log-linear models, likelihood estimation, and disclosure limitation. In: Putinar M, Sullivant S (eds) Emerging applications of algebraic geometry. IMA series in applied mathematics. Springer, New York, pp – Doyle P, Lane J, Theeuwes J, Zayatz L () Confidentiality, disclosure and data access. Theory and applications for statistical agencies. Elsevier, New York Duncan G, Stokes L () Data masking for disclosure limitation. Wiley Interdiscip Rev: Comput Stat ():– Dwork C, McSherry F, Nissim K, Smith A, March () Calibrating noise to sensitivity in private data analysis. Proceedings of The theory of cryptography conference (TCC ). Springer-Verlag LNCS ():–. New York, NY Federal Committee on Statistical Methodology () Report on statistical disclosure limitation methodology. Statistical policy working paper . http://ntl.bts.gov/docs/wp.html Fienberg S, Slavkovi´c A () Preserving the confidentiality of categorical statistical data bases when releasing information for association rules. Data Mining and Knowledge Discovery, ():– Ganta SR, Kasiviswanathan SP, Smith A () Composition attacks and auxiliary information in data privacy. In: Proceedings of SIG-KDD, pp –
Data Quality (Poor Quality Data: The Fly in the Data Analytics Ointment)
Ghosh J, Reiter J, Karr A () Secure computation with horizontally partitioned data using adaptive regression splines. Comput Stat & Data Anal, ():– Lindell Y, Pinkas B () Secure multiparty computation for privacy-preserving data mining. J Priv and Confidentiality, :– Machanavajjhala A, Kifer D, Abowd J, Gehrke J, Vilhuber L () Privacy: Theory meets practice on the map. In: IEEE th international conference on data engineering, pp – Narayanan A, Shmatikov V () How to break anonymity of the netflix prize dataset. ArXiV report cs.CR/, http://arxiv.org Reiter J () Releasing multiply imputed, synthetic public use microdata: An illustration and empirical study. J R Stat Soc Ser A ():– Slavkovic AB, Lee J () Synthetic two-way contingency tables that preserve conditional frequencies. Stat Meth, ():– Smith A () Efficient, differentially private point estimators. Preprint arXiv:.v Trottini M, Fienberg S () Modelling user uncertainty for disclosure risk and data utility. Int J of Uncert, Fuzzi and Know-Based Sys ():– Vu D, Slavkovi´c A () Differential privacy for clinical trial data: preliminary evaluations. In: ICDMW ’: Proceedings of the IEEE international conference on data mining workshops, Miami, FL, USA. IEEE Computer Society, Washington, DC, pp – Wasserman L, Zhou S () A statistical framework for differential privacy. URL http://www.citebase.org/abstract?id=oai:arXiv. org:.
Data Quality (Poor Quality Data: The Fly in the Data Analytics Ointment) Frank M. Guess , Thomas C. Redman Professor University of Tennessee, Knoxville, TN, USA President of Navesink Consulting Group Navesink Consulting Group, Rumson, NJ, USA
Introduction and Summary In recent years a powerful combination of database technologies, data mining techniques (see 7Data Mining) and analytics software have created vast new opportunities for data analysts and statisticians. For example, corporations have duly stored the results of their customer transactions in corporate databases for over a generation. There are, quite literally millions of records. Massively parallel engines can examine these data in heretofore unimagined ways. The potentials to understand customer profitability, develop better understandings of customers’ past
D
needs and predict future ones, and to use those insights to develop new product niches are enormous. Yet all is not well in the world of data analytics. Unlocking the mysteries data have to offer is difficult at best. And putting the discoveries to work can be even harder. One major reason is poor quality data. Bad data camouflage the hidden nuggets in data or, worse, send an analysis in the wrong direction altogether. Some years ago George Box observed that “. . .all models are wrong, but some are useful”. Compare Box () and see Box and Draper (). Many decision makers seem to intuitively grasp the “all models are wrong” portion and so are skeptical of data mining. Poor data quality exacerbates the problem. Indeed most decision makers are well aware that poor data cause problems in operations. How can things be any better when bad data are combined with incorrect models?! What is a poor data analyst to do? Our advice: “Think and act like an experimenter!” Since time immemorial, experimenters have understood the importance of highquality data – data that are relevant to the subject area, are clearly defined, have minimal or no bias and high precision. And they invest considerable time and energy to achieve this goal. The following summarizes some of these investments and how they apply to analytics. Experimenters plan and design their experiments carefully: Perhaps because their data are so dear, experimenters take time to understand what is already known and to carefully define their objectives, the populations and sub-populations of interest, and the hypotheses of primary interest. In contrast, it is too easy to turn an analytics tool loose, in hope that the computer can make interesting discoveries on its own. Much is lost, especially the abilities to interpret results and identify spurious correlations. So data analysts are well-advised to narrow their foci and develop working knowledge of their subject areas. Then they should aim their analytic tools rather more like a high-powered rifle than a shotgun. Some may object that doing so limits opportunities for serendipity. In our experience just the opposite is true. Serendipitous discoveries are more likely when the current level of understanding does not explain the data and piques the experimenter’s curiosity. That cannot happen without adequate background. Similarly, experimenters design their experiments, seeking exactly the right data. Data analysts should be just as persistent in creating or searching for the data that best meets their needs, not the data that are easy to find. They should confirm interesting results based on large-scale analyzes with small, designed studies. Experimenters get very close to their data: First, they develop deep understanding of their data collection
D
D
Decision Theory: An Introduction
devices. Analysts can do the same thing by working backwards, identifying and understanding the business processes that created their data. Business people are often more casual than experimenters about data definition, so it is critical that data analysts understand the nuance in their data. Second, experimenters build controls into their data collection processes. The simplest example is calibrating their equipment. Data analysts do not own these processes, but they must understand existing controls and recommend others where needed to prevent poor data, whenever possible. Third, experimenters search for and eliminate 7outliers. When one of us (Redman) worked at Bell Labs in the s and s, we used the expression “rinse, wash, scrub” for increasing efforts to identify and eliminate suspect data. It was intense, manual effort, certainly not feasible for enormous databases. But data analysts can certainly rinse, wash, and scrub a small sample. Doing so provides a basis for evaluating the quality of the entire database. And if the results of a large analysis are confirmed on the scrubbed validating subset, one can proceed with greater confidence. Experimenters are transparent in discussing strengths and weaknesses in their data collection processes: Data analysts must do so as well. It is the only way to understand the limitations on what they’ve learned. Experimenters recognize they are part of an ongoing process: For an experimenter, there is no “ultimate experiment.” A good experiment increases the body of knowledge (even if by saying “there’s nothing of interest here”) and leads to another experiment. Most business data analysts are not engaged in formal science, per se. But they may well be part of an end-to-end innovation process. Such data analysts should develop an understanding of where they fit and how they can make product developers, marketers, and others more effective. (Note: if your organization doesn’t have such a process, data analysts are well-advised to behave as though it did.) Final Remarks: Taken together, relatively simple actions, quite natural to experimenters, can assist data analysts to better understand their data, know how good their data really are, and improve their overall effectiveness. For learning more about the area of data quality, we recommend these books, among many out there, English (), Huang et al. (), and Redman (, ).
Cross References 7Fraud in Statistics 7Misuse of Statistics 7Multi-Party Inference and Uncongeniality
References and Further Reading Box GEP () Science and statistics. J Am Stat Assoc :– Box GEP, Draper NR () Empirical model-building and response surfaces. Wiley, New York English LP () Improving data warehouse and business information quality: methods for reducing costs and increasing profits. Wiley, New York Huang K-T, Lee YL, Wang RY () Quality information and knowledge. Prentice Hall, New York Redman TC () Data quality: the field guide. ButterworthHeinemann Digital Press, Boston, MA Redman TC () Data driven: profiting from your most important business asset. Harvard Business School Press, Boston, MA
Decision Theory: An Introduction Martin Peterson Associate Professor Eindhoven University of Technology, Eindhoven, Netherlands Decision theory (see also 7Decision Theory: An Overview) is the theory of rational decision making. This is an interdisciplinary field to which philosophers, economists, psychologists, computer scientists and statisticians contribute their expertise. It is common to distinguish between normative and descriptive decision theory. Normative decision theory seeks to yield prescriptions about what decision makers are rationally required – or ought – to do. Descriptive decision theories seek to explain and predict how people actually make decisions. Descriptive decision theory is thus an empirical discipline, which has its roots in experimental psychology. Descriptive and normative decision theory are, thus, two separate fields of inquiry, which may or may not be studied independently of each other. In decision theory, a decision problem is situation in which a decision maker, (a person, a company, or a society) chooses what to do from a set of alternative acts, where the outcome of the decision depends on which state of the world turns out to be the actual one. Decision problem are classified as decisions under risk or ignorance (or uncertainty) depending on the information available to the agent at the time at which he makes his choice. In decisions under risk the decision maker knows the probability of the possible outcomes, whereas in decisions under ignorance the probabilities are either unknown or non-existent. The term uncertainty is either used as a synonym for ignorance, or as a broader term referring to both risk and ignorance. Let us consider an example. Suppose that you are thinking about taking out insurance against theft of your
Decision Theory: An Introduction
new sports car. Perhaps it costs $ to take out insurance on a car worth $,, and you ask: Is it worth it? In case you know that the probability that the car is stolen is, say, in ,, then you are clearly facing a decision under risk. However, in case you are not able to assess the probability of theft then you are on the contrary facing a decision under uncertainty or ignorance. The most widely applied decision rule for making decisions under risk is to apply the principle of maximizing expected value (or utility). According to this decision rule, the total value of an act equals the sum of the values of its possible outcomes weighted by the probability for each outcome. Hence, the expected values of not taking out fire insurance is /, times the value of losing $,, whereas the expected value of taking out insurance equals the value of losing $ (since the insurance guarantees that you will suffer no financial loss in case of a fire). Modern decision theory is dominated by attempts to axiomatise the principles of rational decision making, and in particular the principle of maximizing expected utility. (The term “utility” refers to a technically precise notion of value.) The first axiomatisation was presented by Ramsey in his paper Truth and Probability, written in but published posthumously in . Ramsey was a philosopher working at Cambridge together with Russell, Moore, and Wittgenstein. In his paper, Ramsey proposed a set of eight axioms for how rational decision makers ought to choose among uncertain prospects. He pointed out that every decision maker behaving in accordance with these axioms will act in a way that is compatible with the principle of maximizing expected value, by implicitly assigning numerical probabilities and values to outcomes. However, it does not follow that the decision maker’s choices were actually trigged by these implicit probabilities and utilities. This way of thinking about rational decision making is very influential in the modern literature, and similar ideas were put forward by Savage in The Foundations of Statistics about two decades later. Another important point of departure in modern decision theory is von Neumann and Morgenstern’s book Theory of Games and Economic Behavior. Von Neumann and Morgenstern showed how a linear measure of value for outcomes (i.e., a utility function) can be generated from a set of axioms dealing with how rational decision makers ought to choose among lotteries. For von Neumann and Morgenstern a lottery is a probabilistic mixture of outcomes; for example, “a fifty-fifty chance of winning either $ or a trip to London” is a lottery. They showed that every decision maker behaving in accordance with their axioms implicitly behaves in accordance with the principle of maximizing expected utility, and implicitly assigns
D
numerical utilities to outcomes. The main difference compared to Ramsey’s axiomatisation is that von Neumann and Morgenstern presented no novel theory of probability. The use of the principle of maximizing expected utility has been criticized by some decision theorists. The most famous counter argument was proposed by Allais in the s. Consider the following lotteries in which exactly one winning ticket will be drawn at random.
Ticket no. Ticket no. – Ticket no. – Gamble
$ Million
$ Million
$ Million
Gamble
$
$ Million
$ Million
Gamble
$ Million
$ Million
$
$ Million
$
Gamble $
In a choice between Gamble and Gamble it seems reasonable to choose Gamble since it gives the decision maker $ Million for sure, whereas in a choice between Gamble and Gamble many people would feel that it makes sense to trade a ten-in-hundred chance of getting $ Million, against a one-in-hundred risk of getting nothing, and consequently choose Gamble . Several empirical studies have confirmed that most people reason in this way. However, no matter what utility one assigns to money, the principle of maximizing expected utility recommends that the decision maker prefers Gamble to Gamble if and only if Gamble is preferred to Gamble . There is simply no utility function such that the principle of maximizing utility is consistent with a preference for Gamble to Gamble and a preference for Gamble to Gamble . To see why this is so, we calculate the difference in expected utility between the two pairs of gambles. Note that the probability that ticket will be drawn is ., and the probability that one of tickets numbered – will be drawn is .; hence, the probability that one of tickets numbered – will be drawn is .. This gives the following equations: u (G) − u(G) = u (M) − [.u (M) + .u (M) + .u (M)] = .u (M) − [.u() + .u(M)]
()
u (G) − u (G) = [.u (M) + .u ()] − [.u (M) + .u (M)] = .u (M) − [.u () + .u (M)] ()
D
D
Decision Theory: An Introduction
Equations and show that the difference in expected utility between G and G is precisely the same as the difference between G and G. Hence, no matter what the decision maker’s utility for money is, it is impossible to simultaneously prefer G to G and to prefer G to G without violating the expected utility principle. However, since many people who have thought very hard about this example still feel it would be rational to stick to the problematic preference pattern described above, there seems to be something wrong with the expected utility principle. Let us now move on to the other type of decision problems mentioned above, viz., decisions under ignorance (or uncertainty). There is no single decision rule for decision making under ignorance that is currently widely accepted by decision theorists. However, the maximin rule and the principle of insufficient reason are two of the most influential rules, which have been widely discussed in the literature. The maximin rule, famously adopted by Rawls, focuses on the worst possible outcome of each alternative. According to this principle, one should maximize the minimal value obtainable with each act. If the worst possible outcome of one alternative is better than that of another, then the former should be chosen. According to the principle of insufficient reason, adopted by e.g., Bernoulli and Laplace, it holds that if one has no reason to think that one state of the world is more probable than another, then all states should be assigned equal probability. A well-known objection to the principle of insufficient reason is that that it seems completely arbitrary to infer that all states are equally probable. If one has no reason to think that one state is more probable than another, it seems strange to conclude anything at all about probabilities. Or, alternatively put, if one has no reason to think that some state is twice as probable as another, why not then reason as if that state is twice as probable as the other? Every possible distribution of probabilities seems to be equally justified. However, not every decision problem can be classified as being either a decision under risk or a decision under ignorance. A major sub-field of modern decision theory is so-called multi-attribute approaches to decision theory. The difference between single- and multi-attribute approaches is that in a single-attribute approach, all outcomes are compared on a single utility scale. For example, in a decision between saving a group of fishermen from a sinking ship at a cost of one million dollars or letting the fishermen die and save the money, the value of a human life will be directly compared with monetary outcomes on a single scale. However, many authors think that such direct comparisons between the value of a human life and
money makes no sense – i.e., that human life and money are incommensurable. The multi-attribute approach seeks to avoid the criticism that money and human welfare are incommensurable by giving up the assumption that all outcomes have to be compared on a common scale. In a multi-attribute approach, each type of attribute is measured in the unit deemed to be most suitable for that attribute. Perhaps money is the right unit to use for measuring financial costs, whereas the number of lives saved is the right unit to use for measuring human welfare. The total value of an alternative is thereafter be determined by aggregating the attributes, e.g., money and lives, into an overall ranking of the available alternatives. Here is an example. Mary has somehow divided the relevant objectives of her decision problem into a list of attributes. For illustrative purposes, we assume that the attributes are (a) the number of lives saved, (b) the financial aspects of the decision, (c) the political implications of the decision, and (d) the legal aspects of the decision. Now, to make a decision, Mary has to gather information about the degree to which each attribute can be realized by each alternative. Consider the following table, in which we list four attributes and three alternatives. Attribute Attribute Attribute Attribute Alt. a
Alt. a
Alt. a
The numbers represent the degree to which each attribute is fulfilled by the corresponding alternative. For example, in the leftmost column the numbers show that the second alternative fulfills the first attribute to a higher degree than the first alternative, and so on. So far the ranking is ordinal, so nothing follows about the “distance” in value between the numbers. However, in many applications of the multi-attribute approach it is of course natural to assume that the numbers represent more than an ordinal ranking. The number of people saved from a sinking ship can, for instance, be measured on a ratio scale. This also holds true of the amount of money saved by not rescuing the fishermen. In this case, nothing prevents the advocate of the multi-attribute approach to use a ratio or interval scale if one so wishes. Several criteria have been proposed for choosing among alternatives with multiple attributes. It is common to distinguish between additive and non-additive criteria. Additive criteria assign weights to each attribute, and rank
Decision Theory: An Overview
alternatives according to the weighted sum calculated by multiplying the weight of each attribute with its value. The weights are real numbers between zero and one, which together sum up to one. Obviously, this type of criterion makes sense only if the degree to which each alternative satisfies any given attribute can be represented at least on an interval scale, i.e., if it makes sense to measure value in quantitative terms. Let us, for the sake of the argument, suppose that this is the case for the numbers in table above, and suppose that all attributes are assigned equal weights, i.e., /. This implies that the value of alternative a is / ⋅ + / ⋅ + / ⋅ + / ⋅ = /. Analogous calculations show that the value of a is , while that of a is also . Since we defined the ranking by stipulating that a higher number is better than a lower, it follows that a and a are better than a . Another major sub-field of contemporary decision theory is social choice theory. Social choice theory seeks to analyze collective decision problems: How should a group aggregate the preferences of its individual members into a joint preference ordering? In this context, a group could be any constellation of individuals, such as a married couple, a number of friends, the members of a club, the citizens of a state, or even all conscious beings in universe. A social choice problem is any decision problem faced by a group, in which each individual is willing to state at least ordinal preferences over outcomes. Once all individuals have stated such ordinal preferences we have a set of individual preference orderings. The challenge faced by the social decision theorist is to somehow combine the individual preference ordering into a social preference ordering, that is, a preference ordering that reflects the preferences of the group. A social state is the state of the world that includes everything that individuals care about, and the term social welfare function (SWF) refers to any decision rule that aggregates a set of individual preference orderings over social states into a social preference ordering over those states. The majority rule used in democratic elections is an example of a SWF. The most famous technical result in social choice theory is Arrow’s impossibility theorem, according to which there is no SWF that meets a set of relatively weak normative conditions. A natural interpretation is that social decisions can never be rationally justified, simply because every possible mechanism for generating a social preference ordering – including the majority rule – is certain to violate at least one of Arrow’s conditions. This result received massive attention in academic circles, and in the s and s, many people took the theorem to prove that “democracy is impossible”. However, the present view is that the situation is not that bad. By giving up or modifying
D
some of Arrow’s conditions one can formulate coherent SWFs that are not vulnerable to his impossibility result. Today, the theorem is interesting mainly because it opened up an entirely new field of inquiry.
Cross References 7Bayesian Statistics 7Decision Theory: An Overview 7Imprecise Probability 7Loss Function 7Multicriteria Decision Analysis 7Multiple Statistical Decision Theory 7Philosophical Foundations of Statistics 7Statistics and Gambling
References and Further Reading Allais M () Le comportement de l’homme rationnel devant le risque: critique des postulates et axiomes de l’ecole Américaine. Econometrica :– Arrow KJ () Social choice and individual values, nd edn (). Wiley, New York Kahneman D, Tversky A () Prospect theory: an analysis of decisions under risk. Econometrica :– Peterson M () An introduction to decision theory. Cambridge University Press, Cambridge Ramsey FP () Truth and Probability, Ramsey, . In: Braithwaite RB (ed) The foundations of mathematics and other logical essays, Ch. VII. Kegan Paul, London, pp – Savage LJ () The foundations of statistics, nd edn (, Dover). Wiley, New York Trench, Trubner & Co., Harcourt, Brace and Company, New York von Neumann J, Morgenstern O () Theory of games and economic behavior, nd edn. Princeton University Press (st edn without utility theory)
Decision Theory: An Overview Sven Ove Hansson Professor and Head Royal Institute of Technology, Stockholm, Sweden
Decision theory (see also 7Decision Theory: An Introduction) is the systematic study of goal-directed behavior under conditions when different courses of action (options) can be chosen. The focus in decision theory is usually on the outcome of decisions as judged by pre-determined criteria or, in other words, on means-ends rationality. Decision theory has developed since the middle of the twentieth century through contributions from several academic disciplines. In this overview over fundamental decision theory, the
D
D
Decision Theory: An Overview
focus will be on how decisions are represented and on decision rules intended to provide guidance for decisionmaking. Finally two paradoxes will be presented in order to exemplify the types of issues that are discussed in modern decision theory.
The Representation of Decisions The standard representation of a decision problem requires that we specify the alternatives available to the decisionmaker, the possible outcomes of the decision, the values of these outcomes, and the factors that have an influence on the outcome.
Alternatives To decide means to choose among different alternatives (options). In some decision problems, the set of alternatives is open in the sense that new alternatives can be invented or discovered by the decision-maker. A typical example is your decision how to spend tomorrow evening. In other decision problems, the set of alternatives is closed so that no new alternatives can be added. Your decision how to vote in the upcoming elections will probably be an example of this. There will be a limited number of alternatives (candidates or parties) between which you can choose. In real life, many if not most decisions come with an open set of alternatives. In decision theory, however, alternative sets are commonly assumed to be closed. The reason for this is that closed decision problems are much more accessible to theoretical treatment. If the alternative set is open, a definitive solution to a decision problem is not in general available. In informal deliberations about decisions, we often refer to alternatives that can be combined with each other. Hence, when deciding how to spend tomorrow evening you may begin by choosing between eating out and going to the cinema, but end up deciding to do both. In decision theory, the alternatives are assumed to be mutually exclusive, i.e., no two of them can both be realized. However, this difference is not very important since you can always convert a set of compatible alternatives to a set of mutually exclusive ones. This is done by listing all the possible combinations (in this example: eating out and not going to the cinema, eating out and going to the cinema, etc.).
They may depend, for instance, on the actions of other persons and various natural events. In decision theory, it is common to summarize the various unknown extraneous factors into a number of cases, called states of nature. The states of nature include decisions by other persons. This is a major difference between decision theory and game theory. In game theory, decisions by several persons that may compete or cooperate are treated on a par with each other in the formal representation. In decision theory, the focus is on one decision-maker, and the actions and choices of others are treated differently, namely in the same way as natural events. As an example, consider a young boy, Peter, who makes up his mind whether or not to go to the local soccer ground to see if there is any soccer going on that he can join. The effect of that decision depends on whether there are any soccer players present. In decision theory, this situation is described in terms of two states of nature, “players present” and “no players present.”
Outcomes The possible outcomes of a decision are determined by the combined effects of a chosen alternative and the state of nature that materializes. Hence, if Peter goes to the soccer ground and there are no players present, then the outcome can be summarized as “walk and no soccer,” if he goes and there are players present then the outcome is “walk and soccer,” and if he does not go then the outcome is “no walk and no soccer.”
Decision Matrices The alternatives, the states of nature, and the resulting outcomes in a decision can be represented in a decision matrix. A decision matrix is a table in which the alternatives are represented by rows and the states of nature by columns. For each alternative and each state of nature, the decision matrix assigns an outcome (such as “walk, no soccer” in our example). The decision matrix for Peter’s decision is as follows:
No soccer players Soccer players Go to soccer ground Walk, no soccer Stay home
Walk, soccer
No walk, no soccer No walk, no soccer
States of Nature The effects of a decision depend not only on what choice the decision-maker makes, but also on various factors beyond the decision-maker’s control. Some of these extraneous factors constitute background information that the decision-maker has access to. Others are unknown.
Such a matrix provides a clear presentation of the decision, but it does not contain all the information that the decision-maker needs to make the decision. The most important missing information concerns how the outcomes are valued.
D
Decision Theory: An Overview
Value Representation When we make decisions, or choose among options, we try to obtain as good an outcome as possible, according to some standard of what is good or bad. The choice of a value-standard for decision-making is usually not considered to fall within the subject matter of decision theory. Instead, decision theory assumes that such a standard is available from other sources, perhaps from moral philosophy. There are two major ways to express our evaluations of outcomes. One of these is relational representation. It is expressed in terms of the three comparative value notions, namely “better than” (strong preference, >), “equal in value to” (indifference, ≡), and “at least as good as” (weak preference, ≥). These three notions are interconnected according to the following two rules: . A is better than B if and only if A is at least as good as B but B is not at least as good as A. (A > B if and only if A ≥ B and not B ≥ A.) . A is equally good as B if and only if A is at least as good as B and B is at least as good as A. (A ≡ B if and only if A ≥ B and B ≥ A.) The other major method to express our evaluations of outcomes is numerical representation. It consists in assigning numbers to the possible outcomes; such that an outcome has a higher number than another if and only if it is preferred to it. In an economic context, willingness to pay is often used as a measure of value. If a person is prepared to pay, say $ for a certain used car and $ for another, then these sums can be used to express her (economic) valuation of the two vehicles. However, not all values are monetary. According to some moral theorists, all values can instead be reduced to one unit of measurement, utility. This entity may or may not be identified with units of human happiness. According to utilitarian moral theory, decision-makers should, at least in principle, always (try to) maximize total utility. Decision theorists often use numerical values as abstract tools in the analysis of decisions. These values may be taken to represent utilities, but only in a rather abstract sense since they are not based on any method to measure utilities. Once we have a numerical representation of value, we can replace the verbal descriptions of outcomes in a decision matrix by these values. In our example, suppose that Peter likes to play soccer but does not like walking to the soccer ground and back home. Then his utilities may be representable as follows:
No soccer players
Soccer players
Go to soccer ground
Stay home
Mainstream decision theory is almost exclusively devoted to problems that can be expressed in matrices of this type, utility matrices (payoff matrices).
Probability or Uncertainty Decisions are often categorized according to how much the decision-maker knows beforehand about what state of nature will in fact take place. In an extreme case, the decision-maker knows for sure which state of nature will obtain. If, in the above example, Peter knows with certainty that there are players at the soccer ground, then this makes his decision very simple. The same applies if he knows that there are no players. Cases like these, when only one state of nature needs to be taken into account, are called decision-making under certainty. Non-certainty is usually divided into two categories, called risk and uncertainty. A decision is made under risk if it is based on exact probabilities that have been assigned to the relevant states of nature; otherwise it is made under uncertainty. Decisions at the roulette table are good examples of decisions under risk since the probabilities are known (although some players do not pay much attention to them). A decision whether to marry is a good example of a decision under uncertainty. There is no way to determine the probability that a marriage will be successful, and presumably few prospective brides or grooms would wish to base their decision on precise probability estimates of marital success or failure. In some cases, we do not even have a full list of the relevant states of affairs. Hence, decisions on the introduction of a new technology have to be made without full knowledge of the possible future social states in which the new technology will be used. Such cases are referred to as decision-making under great uncertainty, or ignorance. This adds up to the following scale of knowledge situations in decision problems: Certainty
It is known what states of nature will occur
Risk
The states of nature and their probabilities are known
Uncertainty
The states of nature are known but not their probabilities
Great uncertainty, ignorance
Not even the states of nature are known
D
D
Decision Theory: An Overview
The probabilities referred to in decision theory may be either objective or subjective. In some applications, reliable estimates of probabilities can be based on empirically known frequencies. As one example, death rates at high exposures to asbestos are known from epidemiological studies. In most cases, however, the basis for probability estimates is much less secure. This applies for instance to failures of a new, as yet untried technological device. In such cases we have to resort to subjective estimates of the objective probabilities. Some decision theorists deny the existence of true objective probabilities and regard all probabilities as expressions of degrees of belief, which are of course strictly subjective. In cases when exact probabilities are not known, uncertainty can be expressed with various, more complex measures. Binary measures: The probability values are divided into two groups, possible and impossible values (or attention-worthy and negligible values). Usually, the former form a single interval. Then the uncertainty can be expressed in terms of an interval, for instance: “The probability of a nuclear war in the next thirty years is between and per cent.” Multivalued measures: A numerical measure is used to distribute plausibility over the possible probability values. This measure may (but need not) be a (second-order) probability measure. Then, instead of just saying that the probability is between % and %, we can say that there is a % probability that the probability is between % and %, a % probability that it is between % and %, etc. Robustness measures: The more certain we are about a probability, the less willing we are to change our estimate of it. Therefore, willingness to change one’s estimate of a probability when new information arrives can be used as a measure of uncertainty.
Decision Rules
The maximin rule has often, and quite accurately, been described as a cautious rule. It has also been described as pessimistic, but that is an unfortunate terminology, since caution and pessimism are quite independent of each other. As an example of the maximin rule, consider the following variant of the soccer example from above:
No soccer players Soccer players Go to soccer ground Walk, no soccer Stay home
Walk, soccer
No walk, no soccer No walk, no soccer
The preferences are: Walk, soccer is better than No walk, no soccer is better than Walk, no soccer The security level of Stay home is “no walk, no soccer” whereas that of Go to soccer ground is “walk, no soccer”. Since the former is better than the latter, in order to maximize the security level, Peter would have to stay at home. Consequently, this is what the maximin rule recommends him to do. Even though the maximin rule can be applied to relational value information as above, it is easier to apply if the value information is presented in numerical form. Again, consider the following utility matrix:
No soccer players
Soccer players
Go to soccer ground
Stay home
Decision theorists have developed a series of decision rules, intended to ensure that decisions are made in a systematic and rational way.
Here, the security level of Stay home is whereas that of Go to soccer ground is . Since is larger than , the maximin rule recommends Peter to stay at home, just as in the relational presentation of the same example.
The Maximin Rule
The Maximax Rule
Among the decision rules that are applicable without numerical information, the maximin rule is probably the most important one. For each alternative, we define its security level as the worst possible outcome with that alternative. The maximin rule urges us to choose the alternative that has the highest security level. In other words, we maximize the minimal outcome.
The best level that we can at all obtain if we choose a certain alternative is called its hope level. According to the maxi-max rule, we should choose the alternative whose hope level (best possible outcome) is best. Just like the maxi-min rule, the maxi-max rule can be applied even if we only have relational (non-numerical) value information. Consider again the soccer example. The hope level of
D
Decision Theory: An Overview
Stay home is “no walk, no soccer,” and that of Go to soccer ground is “walk, soccer” that Peter values higher. Hence, the maximax rule urges Peter to go to the soccer ground. Similarly, in the numerical representation, Stay home has the hope level and Go to soccer ground has ; hence again Peter is advised to go to the soccer ground. The maximax rule has seldom been promoted. Contrary to the maximin rule, it is often conceived as irrational or as an expression of wishful thinking. It is indeed hardly commendable as a general rule for decision-making. However, in certain subareas of life, taking chances may be beneficial, and in such areas behavior corresponding to the maximax rule may not be irrational. Life would probably be much duller unless at least some of us were maximaxers on at least some occasions.
The Cautiousness Index There is an obvious need for a decision criterion that does not force us into the extreme cautiousness of the maximin rule or the extreme incautiousness of the maximax rule. A middle road is available, but only if we have access to numerical information. We can then calculate a weighted average between the security level and the hope level, and use this weighted average to rank the alternatives. Let us again consider the numerical presentation of the soccer example:
No soccer players
Soccer players
Go to soccer ground
Stay home
For each alternative A, let min(A) be its security level and max(A) its hope level. In our example, min(Go to soccer ground) = and max(Go to soccer ground) = . If we choose to assign equal weight to the security level and the hope level, then the weighted value of Go to soccer ground is . × + . × = . Since min(Stay home) = max(Stay home) = , the weighted average value of Stay home is . Hence, with these relative weights, Peter is recommended to go to the soccer ground. More generally speaking, each alternative A is assigned a value according to the following formula: α × min (A) + ( − α) × max (A) If α = , then this rule reduces to the maximin criterion and if α = , then it reduces to the maximax criterion. The index α is often called the Hurwicz α index, after economist Leonid Hurwicz who proposed it in . It is also often
called the optimism-pessimism index, but the latter terminology should be avoided since the index represents the degree of (un)cautiousness rather than that of optimism. It can more appropriately be called the cautiousness index.
Minimax Regret Utility information also allows for another decision criterion that puts focus on how an outcome differs from other outcomes that might have been obtained under the same state of affairs, if the decision-maker had chosen another alternative. In our example, if Peter stays home and there are players at the soccer ground, then he has made a loss that may give rise to considerable regret. If he goes to the soccer ground and there is no one there to play with him, then he has also made a loss, but a smaller one. The decision rule based on these considerations is usually called the minimax regret criterion. It also has other names, such as minimax risk, minimax loss, and minimax. In this decision rule the degree of regret is measured as the difference between the utility obtained and the highest utility level that could have been obtained (in the same state of the world) if another alternative had been chosen. A regret matrix can quite easily be derived from a utility matrix: Replace each entry by the number obtained by subtracting it from the highest utility in its column. In our example, the regret matrix will be as follows: No soccer players
Soccer players
Go to soccer ground
Stay home
The minimax regret criterion advices the decisionmaker to choose the option with the lowest maximal regret (to minimize maximal regret). In this case it recommends Peter to go to the soccer ground. Just like the maximin rule, the minimax regret rule can be described as cautious, but they apply cautiousness to different aspects of the decision (the value of the actual outcome respectively its regrettableness). As this example shows, they do not always yield the same recommendation.
Expected Utility None of the above decision rules requires or makes use of probabilistic information. When probabilities are available, the dominating approach is to maximize expected utility (EU). Then to each alternative is assigned a weighted average of its utility values under the different states of nature, with the probabilities of these states used as weights.
D
D
Decision Theory: An Overview
In the above example, suppose that based on previous experience Peter believes the probability to be . that there are players at the soccer ground. We can enter the probabilistic information into the column headings of the utility matrix as follows: No soccer players Probability .
Soccer players Probability .
Go to soccer ground
Stay home
The expected (probability-weighted) utility of going to the soccer ground is . × + . × = , and that of staying at home is . × + . × = . If Peter wants to maximize expected utility then he should, in this case, go to the soccer ground. Obviously, the recommendation would be different with other probabilities. For a general formula representing expected utility, let there be n outcomes, to each of which is associated a utility and a probability. The outcomes are numbered, so that the first outcome has utility u and probability p , the second has utility u and probability p , etc. Then the expected utility is defined as follows: p × u + p × u + . . . + pn × un Expected utility maximization based on subjective probabilities is commonly called Bayesian decision theory, or Bayesianism. (The name derives from Thomas Bayes, – , who provided much of the mathematical foundations for probability theory). According to Bayesianism, a rational decision-maker should have a complete set of probabilistic beliefs (or at least behave as if she had one) and all her decisions should take the form of choosing the option with the highest expected utility.
Two Paradoxes of Decision Theory Much of the modern discussion on decision theory has been driven by the presentation of paradoxes, i.e., situations in which decision-making criteria that seem to epitomize rationality nevertheless give rise to decisions that are contrary to most people’s intuitions. The following two decision paradoxes serve to exemplify the kinds of philosophical problems that are discussed in the decisiontheoretical research literature.
latter two colors is unknown. A ball is going to be drawn at random from the urn. Before that is done you are offered bets by two persons. Anne offers you to bet either on red or on black. If you bet on red, then you will receive e if the drawn ball is red and nothing if it is either black or yellow. Similarly, if you bet on black, then you will get e if the ball is black, and nothing if it is red or yellow. Betty offers you to bet either on red-or-yellow or on black-or-yellow. If you bet on red-or-yellow, then you will get e if the drawn ball is either red or yellow, but nothing if it is black. If you bet on black-or-yellow, then you will get e if the drawn ball is either black or yellow, but nothing if it is red. Most people, it turns out, prefer betting red to betting black, but they prefer betting black-or-yellow to betting red-or-yellow. It is fairly easy to show that this pattern is at variance with expected utility maximization, i.e., there is no way to assign utilities that would make this pattern compatible with the maximization of expected utility. Ellsberg’s own conclusion was that decision-making must take into account factors not covered by probabilities and utilities, in particular the degree of uncertainty of the various probability estimates. Another problem with this pattern is that it violates the sure-thing principle that is a much acclaimed rationality criterion for decisions. To introduce the principle, let A and B be two alternatives, and let S be a state of nature such that the outcome of A in S is the same as that of B. In other words, the outcome in case of S is a “sure thing,” not influenced by the choice between A and B. The sure-thing principle says that if the “sure thing” (i.e., the common outcome in case of S) is changed, but nothing else is changed, then the choice between A and B is not affected. As an example, suppose that a whimsical host wants to choose a dessert by tossing a coin. You are invited to choose between alternatives A and B. In alternative A, you will have fruit in case of heads and nothing in case of tails. In alternative B you will have pie in case of heads and nothing in case of tails. The decision matrix is as follows:
Heads
Tails
A
Fruit
Nothing
B
Pie
Nothing
Ellsberg’s Paradox Daniel Ellsberg has presented the following decision problem: We have an urn that contains red ball and balls that are either black or yellow. The distribution between the
When you have made up your mind and announced which of the two alternatives you prefer, the whimsical host suddenly remembers that he has some ice cream, and
Decision Theory: An Overview
changes the options so that the decision matrix is now as follows:
Heads
Tails
A
Fruit
Ice cream
B
Pie
Ice cream
Since only a “sure thing” (an outcome that is common to the two alternatives) has changed between the two decision problems, the sure-thing principle demands that you do not change your choice between A and B when the decision problem is revised in this fashion. If, for instance, you chose alternative A in the first decision problem, then you are bound to do so in the second problem as well. In this example, the sure-thing principle appears rational enough, and it would seem natural to endorse it as a general principle for decision-making. Ellsberg’s paradox shows that is not quite as self-evident as it may seem to be at first sight.
Newcomb’s Paradox The following paradox was proposed by the physicist William Newcomb: In front of you are two boxes. One of them is transparent, and you can see that it contains $,. The other is covered, so that you cannot see its contents. It contains either $,, or nothing. You have two options to choose between. One is to take both boxes, and the other is to take only the covered box. A predictor who has infallible (or almost infallible) knowledge about your psyche has put the million in the covered box if he predicted that you will only take that box. Otherwise, it is empty. Let us apply maximized expected utility to the problem. If you decide to take both boxes, then the predictor has almost certainly foreseen this and put nothing in the covered box. Your gain is $,. If, on the other hand, you decide to take only one box, then the predictor has foreseen this and put the million in the box, so that your gain is $,,. In other words, maximization of expected utility urges you to take only the covered box. There is, however, another plausible approach to the problem that leads to a different conclusion. If the predictor has put nothing in the covered box, then it is better to take both boxes than to take only one, since you will gain $, instead of nothing. If he has put the million in the box, then too it is better to take both boxes, since you will gain $,, instead of $,,. Thus, taking both boxes is better in all states of nature. (It is a dominating
D
option.) It seems to follow that you should take both boxes, contrary to the rule of maximizing expected utilities. The two-box strategy in Newcomb’s problem maximizes the “real gain” of having chosen an option, whereas the one-box strategy maximizes the “news value” of having chosen an option. The very fact that a certain decision has been made in a certain way changes the probabilities that have to be taken into account in that decision. In causal decision theory, expected utility calculations are modified so that they refer to real value rather than news value. This is done by replacing standard probabilities by some formal means for evaluating the causal implications of the different options. Since there are several competing philosophical views of causality, there are also several formulations of causal decision theory.
About the Author Sven Ove Hansson is professor in philosophy and Head of the division of Philosophy, Royal Institute of Technology, Stockholm. He is Editor-in-Chief of Theoria. His research areas include value theory, decision theory, epistemology, belief dynamics and the philosophy of probability. He is the author of well over articles in refereed journals. His books include A Textbook of Belief Dynamics. Theory Change and Database Updating (Kluwer ) and The Structures of Values and Norms (CUP ). He is a member of the Royal Swedish Academy of Engineering Sciences (IVA).
Cross References 7Bayesian Statistics 7Decision Theory: An Introduction 7Imprecise Probability 7Loss Function 7Multicriteria Decision Analysis 7Multiple Statistical Decision Theory 7Philosophical Foundations of Statistics 7Statistical Inference for Quantum Systems 7Statistics and Gambling
References and Further Reading Gärdenfors P, Sahlin NE (eds) () Decision, probability, and utility. Cambridge University Press, Cambridge Hansson SO () Decision theory. A brief introduction. http://www.infra.kth.se/~soh/decisiontheory.pdf Luce RD, Raiffa H () Games and decisions: introduction and critical survey. Wiley, New York Resnik MD () Choices – an introduction to decision theory. University of Minnesota Press, Minneapolis
D
D
Decision Trees for the Teaching of Statistical Estimation
Decision Trees for the Teaching of Statistical Estimation Carlos Eduardo Valdivieso Taborga Professor of Statistics Universidad Privada Boliviana, Cochabamba, Bolivia
Introduction In many experiments or research, researchers want to know whether the difference in means, proportions, or variances between two populations is significant, or sometimes if the average, proportion, or variance are near to a standard value. They also will be interested in an interval which is expected to contain the value of the difference in means, proportions, and variances of the two populations, or the average, proportion, or variance of a population. In all these cases, researchers have to rely on statistical estimation and choose the most appropriate procedure. Since many different confidence intervals have been proposed in the literature, the choice can sometimes be difficult and confusing, especially for students. In this chapter we will show how decision trees can help in finding the most appropriate 7confidence interval.
is called the confidence level, or sometimes confidence coefficient (and is typically taken to be . or .).
Decision Trees for Choosing the Proper Confidence Interval Because in statistics there are many different confidence interval expressions we’ll attempt to show here how one can choose the appropriate one, in other words, the one that is best suited for the data at hand. Choice of the interval depends on several conditions: ● ● ●
●
Which population parameter has to be estimated? How many populations are under investigation? With one population, does the sample come from a normal population or not, do we know the population variance or not, and also is the sample size small (less than ) or large? In the case of two populations, are the samples independent or dependent and are population variances equal or not?
To facilitate making the proper decision regarding an adequate confidence interval a researcher may use the decision trees, discussed in the next section. Their use is simple; however, we will explain the procedure and give an example to provide more clarity.
Types of Estimations An estimator is a statistic of a sample that is used to estimate a population parameter such as mean, proportion, or variance. An estimate is a specific value of the observed statistics. In real life the parameter values of the populations studied are unknown. To obtain the best achievable information for the parameters we rely on the sample data and apply a procedure that is called statistical estimation. For this purpose we can use one of the following: ●
Point estimator as is a single value that estimates the value of the parameter. Two estimators are, for exam-
Example A new filtering device is installed at a chemical plant. Random samples yield the following information of percentage of impurity before and after installation:
Sample/Statistics
Mean
Variance
Sample size
Before
.
.
After
.
.
n
ple, the sample mean: x¯ =
∑ xi i=
n
(as estimator for the n
∑ x i −n¯x
●
population mean), and sample variance: s = i= n− (as estimator for the population variance). Interval estimation, which provides a numerical range in which it is intended to find the value of the parameter under study.
One disadvantage of point estimators is that they do not provide information on how close they are to the true value of the parameter. To incorporate some measure of the accuracy of an estimate, we determine a range (confidence interval). This range includes the value of the parameter with a certain predetermined probability − α, which
(a) Establish a way to estimate if the impurity has the same variability before and after installing the new filter device. Use α = .. (b) Does the filtering device have reduced the average impurity significantly? Use α = .. (c) If the average percentage of impurities allowed in the chemical plant is %, is the goal reached? (a) From Fig. , we get the following information: ● ●
How many populations are being investigated? Since we have a before-and-after case, obviously there are two populations. What is the keyword? Comparison of variances.
Decision Trees for the Teaching of Statistical Estimation
D
Mean (Figure 2) Average mean Variation Variability
One
Variance
D
Fraction percentage Proportion
Two
Comparison of averages Comparison of variances
How many populations are being investigated?
Comparison of fractions What is the keyword?
Mean differences (Figures 3)
Variance ratios
Difference of proportions
Decision Trees for the Teaching of Statistical Estimation. Fig. Decision tree for choosing the proper confidence interval
Use the following confidence interval: Any n
Yes Large n >30 Yes
No n <30
Small
Large n >30 No
Yes
n <30
Is the population normally distributed?
Small
Large No Do you know the population variance?
n >30
n <30
Small
There is no confidence interval
What is the sample size?
Decision Trees for the Teaching of Statistical Estimation. Fig. Decision tree for choosing the proper confidence interval for the mean of the population
D
Decision Trees for the Teaching of Statistical Estimation
No
Any n
No
Large
No
No
Yes Yes
Yes Yes
n >30 Yes
No
Yes Do you know population variances?
n<30 Small
What are the sample sizes?
No Are the samples paired?
No Are the variances of the populations are same?
Decision Trees for the Teaching of Statistical Estimation. Fig. Decision tree for choosing the proper confidence interval for the difference of means of normal populations
●
Therefore, we choose the interval for the variance ratio P(
s σ s ≤ ≤ ) = −α. s Fα/,n −,n − σ s F−α/,n −,n −
Replacing the sample information and critical points of the F distribution we obtain: P(
. σ . ≤ ≤ ) . . σ . .
= % P (. ≤
σ ≤ .) = %. σ
We conclude that there is no significant difference between the impurity variability before and after installing the new filter device (since the interval contains ). (b) From Fig. , we acquire answers to the following questions: ● ● ● ● ●
How many populations are being investigated? Two populations, with the same argument as above. What is the keyword? Comparison of averages. We will make use of Fig. . Do we know population variances? No. What are the sample sizes? Less than . Are the variances of the populations the same? Yes (Result of part a).
Therefore, the appropriate confidence interval is for mean difference: √ ⎛ P (x − x ) − tα/,v sp + ≤ µ − µ ≤ (x − x ) ⎝ n n √ ⎞ +tα/,v sp + =−α n n ⎠
with √ sp =
(n − ) s + (n − )s n + n −
By including the sample information and critical points of the t distribution we obtain √ ( − ). + ( − ). sp = = .. +− √ ⎛ P (. − .) − (.). + ≤ µ − µ ≤ ⎝ √ ⎞ (. − .) + (.). + = . ⎠ P(−. ≤ µ − µ ≤ .) = . The filtering device has not been effective because the average impurity before and after is the same (since the confidence interval encompasses ).
Degradation Models in Reliability and Survival Analysis
(c) From Fig. , we get the following information: ● How many populations are being investigated? Since only want to see if the new device has helped to achieve the goal: one. ● What is the keyword? Mean. We will use Fig. . ● Is the population normally distributed? Let us assume that the sample came from a normal population. ● Do we know the population variance? No. ● What is the sample size? Small (less than ). Thus, the appropriate confidence interval for the mean is: s s P (x − tα/,n− √ ≤ µ ≤ x + tα/,n− √ ) n n =−α Replacing the sample information and critical points of the t distribution: . . P (. − . √ ≤ µ ≤ . + . √ ) = % P(. ≤ µ ≤ .) = % The impurity is within the limit allowed by the plant.
References and Further Reading Freund JE, Simon GA () Estadística Elemental, Octava Edición. Prentice Hall, México Levin RI, Rubin DS () Estadística para Administradores, Sexta Edición. Prentice Hall Hispanoamericana S.A., México Mason R, Lind D () Estadística para Administración y Economía, Séptima Edición. Alfaomega, México Mendenhal W () Estadística para Administradores, Segunda Edición. Grupo Editorial Iberoamérica, México Miller I, Freund J, Jonson R () Probabilidad y Estadística para Ingenieros, Cuarta Edición. Editorial Prentice Hall Hispanoamericana S.A
D
units are rare and other information should be used in addition to censored failure time data. One way of obtaining complementary reliability information is to use higher levels of experimental factors or covariates to increase the number of failures and, hence, to obtain reliability information quickly. The accelerated life testing (ALT, see 7Accelerated Lifetime Testing) of technical, biological or biotechnical systems is an important practical method of estimation of the reliability of new systems without having to wait through the operating life of them. It is evident that the extrapolating reliability from ALT always carries the risk that the accelerating stresses do not properly excite the failure mechanism which dominates at operating (usual, normal, standard) stresses. Another way of obtaining this complementary reliability information is to measure some parameters (covariates) that characterize the aging and the degradation of the product in time. In analysis of longevity of highly reliable complex industrial or biological systems, the degradation processes provide additional information about the aging, degradation, fatigue, internal wear and deterioration of systems, and from this point of view the degradation data are really a very rich source of additional information and often offer many advantages over failure time data. Degradation is the natural response for some tests, and it is also natural that with degradation data it is possible to make useful reliability and statistical inference, even with no failures. It is evident that it may be difficult or costly to collect degradation measures from some components or materials. Sometimes it is possible to apply the expert’s estimation of the level of degradation or fatigue, etc. Sometimes it is possible to construct degradation models in which degradation is measured without errors, but there are also situations when the degradation data are obtained with measurement errors.
Dynamic Regression Models
Degradation Models in Reliability and Survival Analysis Mikhail Nikulin Professor Université Victor Segalen, Bordeaux, France
Introduction Traditionally the failure time data are usually used for product reliability estimation. Failures of highly reliable
Statistical inference from ALT is possible if failure time regression models relating failure time distribution with explanatory variables (covariates, stresses), influencing the reliability are well chosen. Statistical inference from failure time-degradation data with covariates needs even more complicated models relating failure time distribution not only with external but also with internal explanatory variables (degradation, wear) which explain the state of units before the failures. In the last case models for degradation process distribution are needed, too. Here we discuss only several most used failure time-degradation regression models. Denote by T the random time-to-failure of a unit (or system). We say also that T is the time of hard or traumatic
D
D
Degradation Models in Reliability and Survival Analysis
failure. Let S be the survival function and λ be the hazard rate of T: S(t) = P{T > t}, λ(t) = lim P{t < T ≤ t + h∣T > t}, h→ h ()
where
Λ(t) = ∫
t
It implies that λ(s)ds
f (t∣x) = S− [S(t∣x)∣x ] ,
is the cumulative hazard function of T. Denote F(⋅) = − S(⋅) the cumulative distribution function of T. In Survival Analysis and Reliability the models are often formulated in terms of cumulative hazard and hazard rate functions. The most common shapes of hazard rates are monotone, ∪- shaped or ∩- shaped, see, for example, Meeker and Escobar (), Lawless (), Zacks (). Let suppose that any explanatory variable is a deterministic time function x = x(⋅) = (x (⋅), . . . , xm (⋅))T : [, ∞[→ B ∈ Rm , which is a vector of covariates itself or a realization of a stochastic process X(⋅), which is called also the covariate process, X(⋅) = (X (⋅), . . . , Xm (⋅))T . We denote by E the set of all possible (admissible) covariates. We do not discuss here the questions of choice of Xi and m, but they are very important for the organization (design) of the experiments and for statistical inference. The covariates can be interpreted as the control, so we may consider models of aging in terms of the optimal control theory. We may say also that we consider statistical modeling with dynamic design or in dynamic environments. In accordance with () the survival, the hazard rate, the cumulative hazard and the distribution functions given covariate x are: S(t∣x) = P (T > t∣x(s); ≤ s ≤ t) , S′ (t∣x) , λ(t∣x) = − S(t∣x) Λ(t∣x) = − ln [S(t∣x)] ,
()
F(t∣x) = P (T ≤ t∣x(s); ≤ s ≤ t) = − S(t∣x),
S(t∣x) = P{T > t∣x(s); ≤ s ≤ t} = P{T > f (t∣x)∣x (s); ≤ s ≤ f (t∣x)} = S( f (t∣x)∣x ).
from where it follows that S(⋅) can be written as S(t) = e−Λ(t) ,
unit used under the stress x would survive till the moment t is equal to the probability that a unit used under the stress x would survive till the moment f (t∣x):
x ∈ E,
from where one can see their dependence on the lifehistory up to time t. On any family E of admissible stresses, we may consider a class {S(⋅∣x), x ∈ E} of survival functions which could be very rich. Failure is affected by time-dependent covariates (loads, stress, usage rate) which describe heterogeneous and dynamic operating conditions. We say that the time f (t∣x) under the stress x is equivalent to the time t under the stress x if the probability that a
x ∈ E.
()
Let x and y be two admissible stresses: x, y ∈ E. We say that a stress y is accelerated with respect to x, if S(t∣x) ≥ S(t∣y), ∀ t ≥ . The accelerated failure time (AFT) model is more adapted for failure time regression analysis, see Meeker and Escobar (), Bagdonaviˇcius and Nikulin (), Lawless (), Nelson (). We say that AFT model holds on E if there exists a positive continuous function r : E → R such that for any x ∈ E the survival and the cumulative hazard functions under a covariate realization x are given by formulas: S(t∣x) = G (∫
t
Λ(t∣x) = Λ (∫
r [x(s)] ds) t
r [x(s)] ds) ,
and x ∈ E,
()
respectively, where G(t) = S(t∣x ), Λ (t) = − ln G(t), x is a given (usual) stress, x ∈ E. The function r changes locally the time scale. From the definition of f (t∣x) (cf. ()) it follows that for the AFT model on E ∂f (t∣x) = r(x(t)) ∂t at the continuity points of r[x(⋅)]. ()
f (t∣x) = ∫
t
r [x(s)] ds,
hence
Note that the model can be considered as parametric (r and G belong to parametric families) semiparametric (one of these functions is unknown, other belongs to a parametric family) or nonparametric (both are unknown).
Modeling of Degradation Process In reliability there is considerable interest in modeling covariate processes Z(t), t ≥ , with some properties, depending on the phenomena in consideration, which describe the real process of wear or the usage history up to time t, indicate the level of fatigue, degradation, and deterioration of a system, and may influence the rate of degradation, risk of failure, and reliability of the system. Statistical modeling of observed degradation processes can help to understand different real physical, chemical, medical, biological, physiological, or social degradation processes of aging. Information about real degradation processes help us to construct degradation models, which permit us to
Degradation Models in Reliability and Survival Analysis
predict the cumulative damage, and so on. According to the principal idea of degradation models, a soft failure is observed if the degradation process reaches a critical threshold z . A soft failure caused by degradation occurs when Z(t), t ≥ , reaches the value z . The moment T of soft failure is defined by relation
T = sup{t : Z(t) < z } = inf{t : Z(t) ≥ z }.
()
So it is reasonable to construct the so-called degradation models, based on some suppositions about the mathematical properties of the degradation process Z(t), t ≥ , in accordance with observed longitudinal data in the experiment. Both considered methods may be combined to construct the so-called joint degradation models. For this, it is enough to define a failure of the system when its degradation (internal wear) reaches a critical value or a traumatic event occurs. The joint degradation models form the class of models with competing risk, since for any item we consider two competing causes of failure: degradation, reaching a threshold (soft failure), and occurrence of a hard or traumatic failure. Let T be the moment of traumatic failure of a unit. In the considered class of models, the failure time τ is the minimum of the moments of traumatic and soft failures, τ = min(T , T) = T ∧ T. These models are also called degradation-threshold-shock models. For such models the degradation process Z(t) can be considered as an additional time-depending covariable, which describes the process of wear or the usage history up to time t. The degradation models with covariates are used to estimate reliability when the environment is dynamic (see Singpurwalla ; Bagdonaviˇcius and Nikulin ). The covariates cannot be controlled by an experimenter in such a case. For example, the tire wear rate depends on the quality of roads, the temperature, and other factors. In Meeker and Escobar () the use of so-called path models is proposed for construction the degradation process. The authors describe many different applications and models for accelerated degradation and Arrhenius analysis for data involving a destructive testing. Meeker and Escobar () used convex and concave degradation models to study the growth of fatigue cracks, the degradation of components in electronic circuits. The most applied 7stochastic processes describing degradation are general path models and time scaled stochastic processes with stationary and independent increments such as the gamma process, shock processes and Wiener process with a drift.
D
Example General degradation path model: The general degradation path model was considered by Meeker and Escobar (), according to which Z(t) = g(t, A, θ)
()
where A = (A , . . . , Ar ) is a finite dimensional random vector with positive components and the distribution function π of A, θ is a finite dimensional non-random parameter, and g is a specified continuously differentiable in t function, which increases from to +∞ when t increases from to +∞. The form of the function g is suggested by the form of individual degradation curves, obtained from the experiments. For example, g(t, a) = t/a (linear degradation, r = ) or g(t, a) = (t/a )a (convex or concave degradation, r = ). It is also supposed that, for each t > , the degradation path (g(s, a) ∣ < s ≤ t) determines in the unique way the value a of the random vector A. Degradation in these models is modeled by the process Z(t, A), where t is time and A is some possibly multidimensional random variable. The linear degradation models are used often to study the increase in a resistance measurement over time, and the convex degradation models and concave degradation models are used to study the growth of fatigue cracks. In many different applications and models for accelerated degradation for data involving a destructive testing are described. Influence of covariates on degradation is also modeled in Bagdonaviˇcius and Nikulin (), Doksum and Normand (), Padgett and Tomlinson (), etc., to estimate reliability when the environment is dynamic. The semiparametric analysis of several new degradation and failure time regression models without and with covariables is described in Bagdonaviˇcius and Nikulin (, ), Yashin (), Bagdonaviˇcius et al. (), and Zacks (). The degradation under the covariate x is modeled by Z(t∣x) = g( f (t, x, β), A),
m(t∣x) = Eg( f (t, x, β), A).
About the methods of estimation one can see also in Meeker and Escobar (). Example Time scaled gamma process: Z(t) = σ γ(t), where γ(t) is a process with independent increments such that for any fixed t > γ(t) ∼ G(, ν(t)),
ν(t) =
m(t) , σ
i.e., γ(t) has the 7gamma distribution with the density pγ(t) (x) =
x ν(t)− −x e , Γ(ν(t))
x ≥ ,
D
D
Degradation Models in Reliability and Survival Analysis
where m(t) is an increasing function. Then
Z(t∣x) = σ γ( f (t, x, β)). The mean degradation and the covariances under the covariate x are m(t∣x) = E(Z(t∣x)) = m ( f (t, x, β)) , Cov(Z(s∣x), Z(t∣x)) = σ m( f (s ∧ t, x, β)). Bagdonaviˇcius and Nikulin () considered estimation from failure time-degradation data, Lawless (), considered estimation from degradation data. Example Time scaled Wiener process with a drift: Z(t) = m(t) + σW(m(t)), where W denotes the standard Wiener motion, i.e., a process with independent increments such that W(t) ∼ N(, t). Then Z(t∣x) = m( f (t, x, β)) + σW(m( f (t, x, β))). The mean degradation and the covariances under the covariate x are m(t∣x) = m ( f (t, x, β)) ,
Cov(Z(s∣x), Z(t∣x))
= σ m( f (s ∧ t, x, β)). Doksum and Normand (), Lehmann (, , ) Whitmore and Schenkelberg () considered estimation from degradation data. Example Shock processes: Assume that degradation results from shocks, each of them leading to an increment of degradation. Let Tn , (n ≥ ) be the time of the nth shock and Xn the nth increment of the degradation level. Denote by N(t) the number of shocks in the interval [, t]. Set X = . The degradation process is given by ∞
N(t)
n=
n=
Z(t) = ∑ {Tn ≤ t}Xn = ∑ Xn . Kahle and Wendt () model Tn as the moments of transition of the doubly stochastic Poisson process, i.e., they suppose that the distribution of the number of shocks up to time t is given by P{N(t) = k} = E {
(Yη(t))k exp{−Yη(t)}} , k!
where η(t) is a deterministic function and Y is a nonnegative random variable with finite expectation. If Y is non-random, N is non-homogenous Poisson process, in particular, when η(t) = λt, N is homogenous Poisson process. If η(t) = t, then N is a mixed Poisson process. Other models for η may be used, for example, η(t) = t α , α > . The random variable Y is taken from some parametric class of distributions. Wendt and Kahle (),
Kahle and Wendt () considered parametric estimation from degradation data. Lehmann (, ) considered estimation from failure time-degradation data. Example Degradation model with noise: suppose that the lifetime of a unit is determined by the degradation process Z(t) and the moment of its potential traumatic failure is T. Denote by T the moment at which the degradation attains some critical value z . Then the moment of the unit’s failure is τ = T ∧ T. To model the degradation-failure time process, we suppose that the real degradation process is modeled by the general path model: Zr (t) = g(t, A)
()
where A = (A , . . . , Ar ) is a random positive vector, and g is continuously differentiable increasing in t function. As remarked earlier, the real degradation process Zr (t) is often not observed, and we have to measure (to estimate) it. In this case, the observed degradation process Z(t) is different from the real degradation process Zr (t). We supposed that we have the degradation model with noise according to which the observed degradation process is Z(t) = Zr (t)U(t),
t>
()
where ln U(t) = V(t) = σW(c(t)), W is the standard Wiener process independent of A, and c is a specified continuous increasing function, with c() = . For any t > the median of U(t) is . Using this model it is easy to construct a degradation model with noise. An important class of models based on degradation processes was developed recently by Bagdonaviˇcius et al. (). Wulfsohn and Tsiatis () proposed considered the so-called joint model for survival and longitudinal data measured with error, given by λ T (t∣A) = λ (t)e β(A +A t)
()
where A = (A , A ) follows bivariate normal distribution. On the other hand Bagdonaviˇcius and Nikulin () proposed the model in terms of a conditional survival function of T given the real degradation process: ST (t∣A) = P{T > t∣g(s, A), ≤ s ≤ t} = exp {− ∫
t
λ (s, θ)λ(g(s, A))ds}
()
where λ is the unknown intensity function, λ (s, θ) is from a parametric family of hazard functions. The distribution of A is not specified. This model states that the conditional hazard rate λ T (t∣A) at the moment t given the degradation
Degrees of Freedom
g(s, A), ≤ s ≤ t, has the multiplicative form as in the famous Cox model: λ T (t∣A) = λ (s, θ)λ(g(s, A))
()
If for example, λ (s, θ) = ( + t)θ or λ (s, θ) = etθ , then θ = corresponds to the case when the hazard rate at any moment t is a function of the degradation level at this moment. One can note that in the second model the function λ, characterizing the influence of degradation on the hazard rate, is nonparametric.
About the Author For biography see the entry 7Chi-Square Goodness-of-Fit Tests: Drawbacks and Improvements.
Cross References 7Accelerated Lifetime Testing 7Modeling Survival Data 7Parametric and Nonparametric Reliability Analysis 7Survival Data
References and Further Reading Bagdonaviˇcius V, Nikulin M () Estimation in degradation models with explanatory variables. Lifetime Data Anal :– Bagdonaviˇcius V, Nikulin M () Accelerated life models. Chapman & Hall/CRC, Boca Raton Bagdonaviˇcius V, Nikulin M () Semiparametric analysis of degradation and failure time data with covariates. In: Nikulin M, Balakrishnan N, Mesbah M, Limnios N (eds) Parametric and semiparametric models with applications to reliability, survival analysis, and quality of life. Birkhauser, Boston, pp – Bagdonaviˇcius V, Bikelis A, Kazakeviˇcius V, Nikulin M () Non-parametric estimation from simultaneous renewal-failuredegradation data with competing risks. J Stat Plan Inference :– Doksum KA, Normand SLT () Gaussian models for degradation processes – part I: methods for the analysis of biomarker data. Lifetime Data Anal :– Kahle W, Wendt H () Statistical analysis of some parametric degradation models. In: Nikulin M, Commenges D, Huber C (eds) Probability, statistics and modelling in public health. Springer, New York, pp – Lawless JF () Statistical models and methods for lifetime data. Wiley, New York Lehmann A () A Wiener process based model for failure and degradation data in dynamic environments. Drexdner Schriften zur Mathemat Stochastik :– Lehmann A () On degradation-failure models for repairable items. In: Nikulin M, Balakrishnan N, Mesbah M, Limnios N (eds) Parametric and semiparametric models with applications to reliability, survival analysis, and quality of life. Birkhauser, Boston, pp –
D
Lehmann A () Joint models for degradation and failure time data. In: Proceedings of the internationaol workshop “Statistical modelling and inference in life Sciences”, September –, Potsdam, pp. – Lehmann A () Degradation-threshold-shock models. In: Nikulin M, Commenges D, Huber C (eds) Probability, statistics and modelling in public health. Springer, New York, pp – Meeker W, Escobar L () Statistical methods for reliability data. Wiley, New York Nelson WB () Accelerated testing: Statistical models, test plans, and data analysis, nd edn. Wiley, NY Nikulin M, Limnios N, Balakrishnan N, Kahle W, Huber C (eds) () Advances in degradation modeling, Birkhauser: Boston Padgett WJ, Tomlinson MA () Accelerated degradation models for failure based on geometric Brownian motion and gamma processes. Lifetime Data Anal :– Singpurwalla N () Survival in dynamic environments. Stat Sci :– Wendt H, Kahle W () On parametric estimation for a positiondependent marking of a doubly stochastic Poisson process. In: Nikulin M, Balakrishnan N, Mesbah M, Limnios N (eds) Parametric and semiparametric models with applications to reliability, survival analysis, and quality of life. Birkhauser, Boston, pp – Whitmore GA, Schenkelberg F () Modelling accelerated degradation data using Wiener diffusion with a time scale transformation. Lifetime Data Anal :– Wulfsohn M, Tsiatis A () A joint model for survival and longitudinal data measured with error. Biometrics : – Yashin AI () Semiparametric models in the studies of aging and longevity. In: Nikulin M, Balakrishnan N, Limnios N, Mesbah M (eds) Parametric and semiparametric models with applications for reliability, survival analysis, and quality of life. Birkhauser, Boston, pp – Zacks Sh () Failure distributions associated with general compound renewal damage processes. In: Antonov V, Huber C, Nikulin M, Polischook V (eds) Longevity, aging and degradation models in reliability, public health, medicine and biology, vol . St. Petersburg State Polytechnical University, St. Petersburg, pp –
Degrees of Freedom Chong Ho Yu Arizona State University, Tempe, AZ, USA
Many elementary statistics textbooks introduce the concept of degrees of freedom (df ) in terms of the number scores that are “free to vary.” However, this explanation cannot clearly show the purpose of df . There are many other approaches to present the concept of degrees of freedom. Two of the most meaningful ways are to illustrate df
D
D
Degrees of Freedom
Plot
in terms of sample size and dimensionality. Both represent the number of pieces of useful information. 10
DF in Terms of Sample Size Toothaker () explained df as the number of independent components minus the number of parameters estimated. This approach is based upon the definition provided by Walker (): the number of observations minus the number of necessary relations, which is obtainable from the observations (df = n − r). Although Good () criticized that Walker’s approach is not obvious in the meaning of necessary relations, the number of necessary relationships is indeed intuitive when there are just a few variables. “Necessary relationship” can be defined as the relationship between a dependent variable (Y) and each independent variable (X) in the research. Please note that this illustration is simplified for conceptual clarity. Although Walker regards the preceding equation as a universal rule, df = n − r might not be applicable to all situations.
8
6
4
2 0
2
4
6
8
Degrees of Freedom. Fig. No degree of freedom with one datum point Plot 10
8
No Degree of Freedom and Effective Sample Figure shows that there is one relationship under investigation (r = ) when there are two variables. In the scatterplot there is only one datum point. The analyst cannot do any estimation of the regression line because the line can go in any direction, as shown in Fig. . In other words, there isn’t any useful information. When the degree of freedom is zero (df = n − r = − = ), it is impossible to affirm or reject the model. In this sense, the data have no “freedom” to vary. Bluntly stated, one subject is basically useless, and obviously, df defines the effective sample size (Eisenhauer ). The effective sample size is smaller than the actual sample size when df is taken into account.
Perfect Fitting and Overfitting In order to plot a regression line, one must have at least two data points as indicated in Fig. . In this case, there is one degree of freedom for estimation (n − = , where n = ). When there are two data points only, one can always join them to be a straight regression line and get a perfect correlation (r = .). This “perfect-fit” results from the lack of useful information. Since the data do not have much “freedom” to vary, no alternate models could be explored. In addition, when there are too many variables in a regression model, i.e., the number of parameters to be estimated is larger than the number of observations, this model is said to lacking degrees of freedom and thus is over-fit.
6
4
2 0
2
4
6
8
Degrees of Freedom. Fig. Perfect fit with two data points
DF in Terms of Dimensions and Parameters In this section, degrees of freedom are illustrated in terms of dimensionality and parameters. According to Good (), degrees of freedom can be expressed as D(K) − D(H), whereas D(K) = the dimensionality of a broader hypothesis, such as a full model in regression D(H) = the dimensionality of the null hypothesis, such as a restricted or null model. In Fig. vectors (variables) in hyperspace (Saville and Wood ; Wickens ) are used for illustrating a regression model. It is important to point out that the illustration is only a metaphor to make comprehension easier. Vectors do not behave literally as shown. In hyperspace, Vector Y represents the dimension of the outcome variable
Degrees of Freedom in Statistical Inference
X1b1
D
Y
variables. Take the -observation and -variable case as an example. In this case, df = − − = . Readers who are interested in df are referred to the online tutorial posted on http://www.creative-wisdom.com/pub/df/default.htm
^
About the Author
Y X 2b2
Chong Ho Yu is the Director of Testing, Evaluation, Assesment, and Research group at Applied Learning Technologies Institute, ASU.
Degrees of Freedom. Fig. Vectors in hyperspace
Cross References whereas Vector Xβ and Xβ denote the dimensions of the predictors with two estimated parameters (βs). Vector Yˆ is the dimension of Y expected and cos(θ) equals R (relationship among X, X, and Y). In this example the intercept is ignored. What is (are) the degree(s) of freedom when there is one variable (vector) in a regression model? First, we need to find out the number of parameter(s) in a one-predictor model. Since only one predictor is present, there is only one beta weight to be estimated. The answer is straightforward: There is one parameter to be estimated. How about a null model? In a null model, the number of parameters is set to zero. The expected Y score is equal to the mean of Y and there is no beta weight to be estimated. Based upon df = D(K)−D(H), when there is only one predictor, the degree of freedom is just one (− = ). It means that there is only one piece of useful information for estimation. In this case, the model is not well-supported. As you notice, a -predictor model (df = − = ) is bettersupported than the -predictor model (df = − = ). When the number of orthogonal vectors increases, we have more pieces of independent information to predict Y and the model tends to be more stable. In short, the degree of freedom can be defined in the context of dimensionality, which conveys the amount of useful information. However, it is important to note that some regression methods, such as ridge regression (see 7Ridge and Surrogate Ridge Regressions), linear smoothers and 7smoothing splines are not based on 7least squares, and so df defined in terms of dimensionality is not applicable to these modeling.
Putting Both Together The above illustrations (Walker’s df and Good’s df ) compartmentalize df in terms of sample size and df in terms of dimensionality (variables). However, observations (n) and parameters (k), in the context of df , must be taken into consideration together. For instance, in regression, the working definition of degrees of freedom involves the information of both observations and dimensionality: df = n − k − whereas n = sample size and k = the number of
7Degrees of Freedom in Statistical Inference
References and Further Reading Eisenhauer JG () Degrees of freedom. Teach Stat ():– Good IJ () What are degrees of freedom? Am Stat :– Saville D, Wood GR () Statistical methods: the geometric approach. Springer, New York Toothaker LE, Miller L () Introductory statistics for the behavioral sciences, nd edn. Brooks/Cole, Pacific Grove Walker HW () Degrees of freedom. J Educ Psychol :– Wickens T () The geometry of multivariate statistics. Lawrence Erlbaum, Hillsdale
Degrees of Freedom in Statistical Inference Joseph G. Eisenhauer Professor and Dean University of Detroit Mercy, Detroit, MI, USA The term degrees of freedom refers to the number of items that can be freely varied in calculating a statistic without violating any constraints. Such items typically include observations, categories of data, frequencies, or independent variables. Because the estimation of parameters imposes constraints on a data set, a degree of freedom is generally sacrificed for each parameter that must be estimated from sample data before the desired statistic can be calculated. Consider first a simple case. A sample of size n initially has n degrees of freedom, in the sense that any or all of the observations could be freely discarded and replaced by others drawn from the population. However, once their sum has been calculated, only n − observations are free to vary, the final one being determined by default. Equivalently, once the sample mean x¯ has been calculated, n deviations from the mean can be found but they must sum to zero, so only n − deviations are free to vary. In letters to W.S. Gosset around , Sir Ronald Fisher first
D
D
Degrees of Freedom in Statistical Inference
showed that if the deviations are squared and averaged over n − rather than n, the resulting sample variance s is an unbiased estimator of the population variance σ . This basic notion recurs throughout inferential statistics. With a normal distribution, testing a hypothesis regarding σ proceeds by way of a chi-square statistic, χ = (n − )s /σ . Notice that E(s ) = σ implies E(χ ) = n − . As this suggests, there exists an entire family of chisquare distributions, the mean of any one of which is its degrees of freedom. Moreover, because χ takes only nonnegative values, the distribution is positively skewed when its mean is near zero, and it becomes increasingly symmetric as the degrees of freedom rise. Consequently, the probability that χ exceeds a given value increases with the degrees of freedom. Put differently, the critical value for a chi-square test at a given level of significance increases with its degrees of freedom. Student’s t statistic for testing a normal population mean likewise has the n − degrees of freedom from the sample variance. As the degrees of freedom rise, the shape of the t distribution approaches that of the standard normal; its variance decreases, and thus, the probability that t exceeds a given value diminishes. To compare the means from two normal populations with σ = σ , the sample variances are pooled and the resulting t statistic then has (n − ) + (n − ) degrees of freedom. For normal populations with σ ≠ σ , Satterthwaite () √ proposed the approximation t = (¯x − x¯ )/ a + a with (a + a ) / [(a /v ) + (a /v )] degrees of freedom, where ai = si /ni and vi = ni − ; in this case, the degrees of freedom will generally not be an integer. To test for equal variances, the F ratio (so named for Fisher), F = s /s , is used with n − numerator degrees and n − denominator degrees of freedom. If F is significantly larger or smaller than unity, the null hypothesis of equal variances can be rejected. Of course, because the designation of populations and is arbitrary, we could just as well invert the F ratio and reverse its degrees of freedom. Therefore, commonly available tables of the F distribution report only the right-hand critical value for an F test or confidence interval; to obtain the left-hand critical value, the numerator and denominator degrees of freedom are reversed, and the resulting table value is inverted. The decomposition of a sample variance into explained and unexplained components via 7analysis of variance (ANOVA) also relies on degrees of freedom. The sum of squared deviations from the sample mean, or sum of squares total (SST), has n − degrees of freedom as explained above. Similarly, with k treatment categories, the sum of squares due to treatment (SSTR) has k − degrees, and the residual, the sum of squares due to error (SSE), has (n − ) − (k − ) = (n − k) degrees of
freedom. Dividing the sums of squares by their respective degrees of freedom, the mean square due to treatment is MSTR = SSTR/(k − ) and the mean square due to error is MSE = SSE/(n − k). Because each of these is a type of variance, we compare them using the test statistic F = MSTR/MSE with k− numerator degrees and n−k denominator degrees of freedom. A related application involves measuring the strength, or explanatory power, of a multiple linear regression, where k is reinterpreted as the number of regression coefficients (including the intercept) and SSTR denotes the sum of squares due to regression. Increasing the number of independent variables (k − ) tends to alter the composition of SST by raising SSTR and reducing SSE, though the increase in explained variation may be somewhat spurious; the coefficient of determination, R = − (SSE/SST), can thereby become inflated. The customary correction is to divide SSE and SST by their respective degrees of freedom, yielding the adjusted coefficient of determination R¯ = − {[SSE/(n − k)]/[SST/(n − )]} as the measure of explanatory power. Algebraically, a system of j simultaneous equations in k unknowns has k − j degrees of freedom. Consider, for example, a goodness-of-fit test to determine whether the variable x, which takes the values , , . . . (k − ), follows a truncated Poisson distribution with a given mean. There are k expected frequencies, but since they must sum to the sample size n, one can be determined from the others. Thus, only k − are free to vary, i.e., there are k − degrees of freedom. Moreover, if it is necessary to first estimate the population mean from the sample data in order to obtain the Poisson distribution, an extra constraint is imposed, leaving k − degrees of freedom; see Eisenhauer () for a numerical illustration. By the same reasoning, a goodness-of-fit test for a normal distribution segmented into k intervals has k − degrees of freedom if the mean and standard deviation of the population are known; but if both parameters need to be estimated, two more degrees are sacrificed and k − degrees of freedom remain. Fisher () originally coined the phrase degrees of freedom in correcting Karl Pearson’s test of independence. A contingency table consisting of r rows and c columns into which frequencies are entered has rc cells, but given row and column totals, the entries for the final row and final column – totaling r + c − cells – can be determined from the others. Thus, there are only rc−(r+c−) = (r−)(c−) degrees of freedom for the chi-square test statistic.
About the Author Dr. Joseph G. Eisenhauer is Dean of the College of Business Administration at the University of Detroit Mercy. His prior positions include Professor and Chair of Economics
Demographic Analysis: A Stochastic Approach
at Wright State University, and Professor of Economics and Finance at Canisius College. He has been a postdoctoral fellow at the University of Pennsylvania, a Visiting Scholar at the Catholic University of America, and a Visiting Professor at the Universita di Roma, La Sapienza, Italy. He is a past President and Distinguished Fellow of the New York State Economics Association, and has published more than articles and reviews in scholarly journals. He recently won the C. Oswald George Prize () for the best article, “Degrees of Freedom” in Teaching Statistics.
Cross References 7Degrees of Freedom
References and Further Reading Eisenhauer JG () Degrees of freedom. Teach Stat ():– Fisher RA () On the interpretation of χ from contingency tables and the calculation of P. J Roy Stat Soc ():– Satterthwaite FE () An approximate distribution of estimates of variance components. Biometrics Bull ():–
Demographic Analysis: A Stochastic Approach Krishnan Namboodiri Professor Emeritus Ohio State University, Columbus, OH, USA
Introduction Demographers study population dynamics: changes in population size and structure resulting from fertility (childbearing performance), mortality (deaths), and spatial and social mobility. The focus may be the world population or a part of it, such as the residents of a country or the patients of a hospital. Giving birth, dying, shifting usual place of residence, and trait changes (e.g., getting married) are called events. Each event involves transition from one “state” to another (e.g., from never-married state to married state). A person is said to be “at risk” or “exposed to the risk” of experiencing an event, if for that person the probability of that experience is greater than zero. The traits influencing the probability of experiencing an event are called the risk factors of that event (e.g., high blood pressure, in the case of ischemic heart disease). Demographic data are based on censuses (see 7Census), sample surveys, and information reported to offices set up for continuously recording demographic events. Some
D
observational studies can be viewed as random experiments. For an individual selected at random from a population at time t, the value of the variable yt+θ , denoting whether that individual will be alive as of a subsequent moment t + θ is unpredictable. This unpredictability of the value of yt+θ qualifies the observational study involving observations at times t and t + θ to be considered as a random experiment and yt+θ as a random variable, defined by a set of possible values it may take (e.g., if alive at time t + θ, and , otherwise) and an associated probability function (Kendall and Buckland ). The interval between a fixed date and a subsequent event is a random variable, in the above-mentioned sense. The term rate is used in 7demography for the number of events (e.g., deaths) expressed per unit of some other quantity, such as person-years at risk (often expressed per ,). For example, the crude death rate (annual number of deaths expressed per , mid-year population) in Japan in was . The mid-year population in such calculations is an approximation to the sum of the personyears lived by the members of the population involved during the specified year. Death rates calculated for subpopulations, homogeneous, to some degree, with respect to one or more relevant risk factors are called specific death rates. Examples are age-specific and age-sex specific death rates. A life-table (see 7Life Table) shows the life-and-death history of a group of persons, called a cohort, born at the same time (e.g., a year), as the cohort members survive to successive ages or die in the intervals, subject to the mortality conditions portrayed in a schedule of age-specific death rates. An account of the origin, nature, and uses of life tables is available in P. R. Cox (). Life tables have become powerful tools for the analysis of non-renewable (non-repeatable) processes. If a repeatable process, such as giving births, can be split into its non-renewable components (e.g., births by birth order) then each component can be studied, using the life-table method. The term: survival analysis is applied to the study of non-renewable processes, in general. Associated with the survival rate is the hazard rate, representing the instantaneous rate of failure (to survive). Hazard rate corresponds to the instantaneous death rate or force of mortality, as used in connection with life tables.
Macro-Level Focus A great deal of demographic research is linked directly or indirectly to model construction and validation, viewing observations as outcomes of random experiments. Birthand-death process (see Kendall ; Bhat ), is a continuous time, integer valued, counting process, in which
D
D
Demographic Analysis: A Stochastic Approach
population size at time t, remains constant, increases by one unit (a birth), or decreases by unit (a death), over the period: t to t + ∆t. Time-trend in population size is studied using branching processes, in a simple version of which, each member of each generation produces offspring, in accordance with a fixed probability law common to all members (see, e.g., Grimmett and Stirzaker for a discussion of simple as well as complex models of branching processes). The logistic process for population growth of the “birth-and-death” type views the instantaneous rates of birth and death for each individual alive at a given moment as linear functions of population size (see Brillinger ; Goel and Dyn ; Mollison ). For compositional analysis, one may apply an appropriate log-ratio transformation to the composition of interest, and treat the resulting values as a random vector from a multivariate normal distribution (see Aitchison ; Namboodiri ). Using the component model (see Keyfitz ) of population projection, one obtains internally consistent estimates of the size and age-sex composition of populations as of future years by combining hypothesized patterns of change in fertility, mortality, and migration. On the basis of such projections, issues such as the following can be examined: () Reduction in population growth rate resulting from the elimination of deaths due to a specific cause, e.g., heart disease; () Relative impact on the age-composition, in the long-run, of different combinations of populationchange components (e.g., fertility and mortality); and () tendency of populations to “forget” the past features (e.g., age composition) if the components of population dynamics were to continue to operate without change over a sufficiently long time. To estimate and communicate the uncertainty of population projections, the practitioners have been combining “high,” “medium,” and “low” scenarios for the components of population change in various ways (e.g., “high” fertility combined with “low” mortality to produce “high” population projection) to show different possibilities regarding future population size and composition. Since such demonstrations of uncertainties have no probabilistic interpretations, Lee and Tuljapurkar, among others, have pioneered efforts to develop and popularize the use of stochastic population projections (see Lee ). Lee and Tuljapurkar () demonstrated, for example, how to forecast births and deaths, from time-series analyses of fertility and mortality data for the United States, and then combine the results with deterministically estimated migration to forecast population size and composition. They used in the demonstration, products of stochastic matrices. Comparison of the simple non-stochastic trend model: yt = β + β (t) + et , with the stochastic (random-walk
with a drift) model: yt = α + yt− + et , where et ’s are NID(, σe ) for all t, shows that even when the error terms have equal variance (σe ) in the two models, the prediction intervals for the latter are wider than those of the former: For a forecast horizon H, the variance of the forecast error (the departure of the forecast from the actual) in the case of yt = β + β (t) + et is σe , while the corresponding quantity is Hσe , in the case of yt = α + yt− + et .
Micro-Level Processes At the micro level, one focuses on events (such as giving birth to the first child, dying, recovering from illness, and so on) experienced by individuals. In event histories, points of time at which transitions occur (e.g., from not in labor force to employed) are represented by a sequence of non-negative random variables: (T , T , . . .), and the differences: Vk = Tk − Tk− , k = , , . . . , are commonly referred to as waiting times. Comprehensive discussions of waiting times are available, for example, in: Cleves et al. (); Collett (); Elandt-Johnson and Johnson (/); and Lawless (/). D. R. Cox () introduced, what has come to be known as, the proportional hazards model: λ(t) = λ (t)ψ(z , z , . . . , zk ), where “t” represents time, and the multiplier, ψ(z , z , . . . , zk ), is positive and timeindependent. A special form of the model is: λ(t) = λ (t) exp(Σβ j zj ), in which {β j } are unknown regression coefficients. An important feature of waiting time is heterogeneity (variation among individuals) in the hazard rate (see Sheps and Menken ; Vaupel et al. ; Heckman and Singer ). Heterogeneity is incorporated often as a multiplier in the Cox proportional hazards model. For example, the hazard function for the ith individual may be specified as: λ (t)ν i exp(Σ j β j Zij ), representing an individual-specific, unobserved heterogeneity factor by ν i . Vaupel et al. () called such models: “frailty” models (see 7Frailty Model). Heckman and Singer () suggested the specification of the unobserved heterogeneity factor in λ(t) = λ (t)ν i exp(Σ j β j Zij ), as a K-category discrete random variable. Thus the ith individual is presumed to belong to one of K groups. The value of K is determined empirically so as to maximize the likelihood of the sample on hand, under a specified (e.g., the exponential or Weibull) form for λ (t). In the presence of heterogeneity, inference becomes sensitive to the form assumed for the hazard function (see, e.g., Trussell and Richards ). As Sheps and Perin () and Menken (), among others, have pointed out, simplified models, unrealistic though they may be, have proved useful in gaining insights such as that a highly effective contraceptive used by a rather small proportion of a population reduces birth rates more
Demographic Analysis: A Stochastic Approach
than does a less effective contraceptive used by a large proportion of the population. Some fertility researchers have been modeling parts rather than the whole of the reproductive process. The components of birth intervals have been examined, with emphasis on the physiological and behavioral determinants of fertility (see Leridon ). Another focus has been abortions, induced and spontaneous (see Abramson ; Potter et al. ; Michels and Willett ). Fecundability investigations have been yet another focus (see Menken ; Wood et al. ). Menken () alerts researchers to the impossibility of reliably estimating fecundability from survey data. The North Carolina Fertility Study referred to in Dunson and Zhou () is of interest in this connection: In that study couples were followed up from the time they discontinued birth control in order to attempt pregnancy. The enrolled couples provided base-line data and then information regarding ovulation in each menstrual cycle, day-by-day reports on intercourse, first morning urine samples, and the like. Dunson and Zhou present a Bayesian Model and Wood et al. () present a multistate model for the analysis of fecundability and sterility. To deal with problems too complex to be addressed using analytic models, researchers have frequently been adopting the simulation strategy, involving computerbased sampling and analysis at the disaggregated (e.g., individual) level. See, for example, the study of () kinshipresources for the elderly by Murphy (); and Wachter (); () female family-headship by Moffit and Rendall (); () AIDs and the elderly by Wachter et al. (); and () the impact of heterogeneity on the dynamics of mortality by Vaupel and Yashin (); and Vaupel et al. (). Questions such as the following arise: Is it possible to reproduce by simulation the world-population dynamics, detailing the changes in the demographiceconomic-spatial-social DESS) complex, over the period, say: –? Obviously, in order to accomplish such a feat, one has to have a detailed causal model of the observed changes to be simulated. As of now no satisfactory model of that kind is available. Thinking along such lines demographers might begin to view micro-simulation as a challenge and an opportunity to delve into the details of population dynamics.
About the Author Dr. Krishnan Namboodiri was Robert Lazarus Professor of Population Studies at the Ohio State University, Columbus, Ohio, USA, (–) and has been Professor Emeritus at the same institution since . Before joining the Ohio State University, he was Assistant Professor, Associate Professor, Professor, and Chairman, Department of Sociology,
D
University of North Carolina at Chapel Hill, USA, (– ); Reader in Demography, University of Kerala, India, (–). Dr. Namboodiri was Editor of Demography (–), and Associate Editor of of a number of professional journals such as Mathematical Population Studies (–). He has authored or co-authored over publications including books. He is a Fellow of the American Statistical Association, and is a recipient of honors such as Lifetime Achievement Award from Kerala University, and has been consultant from time to time to Ford Foundation, World Bank, United Nations, and other organizations.
Cross References 7Demography 7Life Table 7Population Projections
References and Further Reading Abramson FD () High foetal mortality and birth intervals. Popul Stud :– Aitchison J () The statistical analysis of compositional data. Chapman and Hall, London Bhat UN () Elements of applied stochastic processes, nd edn. Wiley, New York Brillinger DR () Some aspects of modern population mathematics. Can J Stat :– Cleves M, Gould WG, Gutierrez RG () An introduction to survival analysis using stata, Revised edn. Stata Press, College Station Collett D () Modeling survival data in medical research, nd edn. Chapman and Hall/CRC, Boca Raton Cox DR () Regression models and life tables (with discussion). J Roy Stat Soc B :– Cox PR () Population trends, vols I–II. Her Majesty’s Stationary Office, London Dunson DB, Zhou H () A Bayesian model for fecundability and sterility. J Am Stat Assoc ():– Elandt-Johnson R, Johnson N (/) Survival models and data analysis. Wiley, New York Goel NS, Dyn NR () Stochastic models in biology. Academic Press, New York Grimmett GR, Stirzaker DR () Probability and random processes, nd edn. Clarendon Press, Oxford Heckman JJ, Singer B () Population heterogeneity in demographic models. In: Land KC, Rogers A (eds) Multidimensional mathematical demography. Academic, New York, pp – Kendall DG () A generalized birth and death process. Ann Math Stat :– Kendall MG, Buckland WR () Dictionary of statistical terms, rd edn. Hafner, New York Keytz N () Models. Demography :– Lawless JF (/) Statistical models and methods for lifetime data, nd edn. Wiley, New York Lee RD () Quantifying our ignorance: stochastic forecasts of population and public budgets. In: Waite LJ (ed) Aging, health, and public policy: demographic and economic perspectives. A special supplement to vol of population and development review, pp –
D
D
Demography
Lee RD, Tuljapurkar S () Stochastic population projections for the United States beyond high, medium, and low. J Am Stat Assoc ():– Leridon H () Human fertility: the basic components (trans: Helzner JF). University of Chicago Press, Chicago Menken J () Biometric models of fertility. Soc Forces :– Michels KB, Willett WC () Does induced or spontaneous abortion affect the risk of cancer? Epidemiology :– Moffit RA, Rendall MS () Cohort trends in the lifetime distribution of family headship in the United States, –. Demography :– Mollison D (ed) () Epidemic models: their structure and relation to data. Cambridge University Press, London Murphy M () Tracing very long-term kinship networks Using SOCSIM. Demogr Res :– Namboodiri K () Demographic analysis: a stochastic approach. Academic, San Diego/New York Potter RG, Ford K, Boots B () Competition between spontaneous and induced abortions. Demography :– Sheps MC, Menken J () Mathematical models of conception and birth. University of Chicago Press, Chicago Sheps MC, Perin EB () Changes in birth rates as a function of contraceptive effectiveness: some applications of a stochastic model. Am J Public Health :– Trussell TJ, Richards T () Correcting for unobserved heterogeneity in hazard models: an application of the Heckman-Singer model for demographic data. In: Tuma NB (ed) Sociological methodology. Jossey-Basspp, San Francisco, pp – Vaupel JW, Manton KG, Stallard E () The impact of heterogeneity in individual frailty on the dynamics of mortality. Demography :– Vaupel JW, Yashin AJ () Heterogeneity ruses: some surprising effects of selection in population dynamics. Am Stat :– Wachter KW () Kinship resources for the elderly. Philos T Roy Soc B :– Wachter KW, Knodel JE, Vanlandingham M () AIDs and the elderly of Thailand: projecting familial impacts. Demography :– Wood JW, Holman DJ, Yashin AI, Peterson RJ, Weinstein M, Chang MC () A multistate model of fecundability and sterility. Demography :–
Demography Jan M. Hoem Professor, Director Emeritus Max Planck Institute for Demographic Research, Rostock, Germany
The Character of the Field of Demography The Topics of Investigation Demography is the statistical study of human populations, including their size, composition, and geographical
distribution, and of the processes of change in these elements. Demographers focus on childbearing (fertility), death (mortality), and geographical moves (migration), but they also cover other processes, like the formation and dissolution of (marital and nonmarital) unions, the transition out of the parental home, and other transitions relevant to population structure and population trends. As a field of inquiry, Demography has roots going back to John Graunt’s famous study of the bills of mortality () and to T. R. Malthus’s essay on the principle of population (). For an account of how the endeavors of demographers have developed into a discipline, see Hodgson (), Szreter (), and Caldwell, ). For more extensive accounts about the field of demography, see Rosenzweig and Stark () and Demeny and McNicoll ().
The Discipline Like all academic disciplines, demographers have formed national, regional, and international societies, such as the Population Association of America, the European Association for Population Studies, the International Union for the Scientific Study of Population, and many others. Its oldest professional journals are publishing their th volumes or so in (Population Studies, Population, Genus), and there are a large number of younger journals (Demography, the Population and Development Review, the European Journal of Population, Demographic Research, Journal of Population Research, Mathematical Population Studies, Journal of Population Economics and so on), some of which are recent start-ups (like the Romanian Journal of Population Studies). Demography courses are given by some universities, often as a part of studies in other disciplines, such as geography, sociology, or statistics. More extensive teaching is organized through demographic research centers like INED (Institut National des Études Démographiques in Paris), NIDI (the Netherlands Interdisciplinary Demographic Institute in The Hague), and MPIDR (the Max Planck Institute for Demographic Research in Rostock), sometimes as a joint venture of demographic centers and university departments across Europe (like the European Doctoral School in Demography). Teaching, research, and instrumental advice are also offered by demographic centers established by the United Nations and others (e.g., the Population Council). North America has its own demographic centers. Ideally, teaching programs reflect the multidisciplinarity of demography (Caselli ).
Demographic Data Since the beginning of public data collection, demographers have made use of several sources of official statistics. For instance, status data are collected in decennial
Demography
D
censuses, typically to provide information about population composition and about the size of local populations or other groups at census time. Vital statistics (i.e., data on births, deaths, migration, marriages, etc.) are published annually and normally provide aggregate data for an entire population or for large population units. Individuallevel (micro) data are mostly collected in special sample surveys organized at the national (or sub-national) level or designed to provide internationally comparable individual-level data (the Fertility and Family Surveys, the Generations and Gender Surveys, the Demographic and Health Surveys, and so on). In the Nordic countries and in a few other countries, individual-level demographic data are organized in a continuous-register system and made available for research. Several other countries or organizations have started to make registers of specific events available for research (such as a birth register, pension registers, migration registers). Recently international databases have been established for scientific use, such as the Human Mortality Database and the Human Fertility Database, both at MPIDR, or the IPUMS-data (International Public Use Microdata Series at the Minnesota Population Center). For further insights about demographic data regimes, see Haines ().
and fertility decline, and why the pattern of the latter differs between societies, it is natural to resort to economics, health science, and perhaps anthropology. The following examples provide selected glimpses of some overlaps between disciplines: There is constantly a mutual enrichment between demographic studies of family dynamics and sociological theory about the family. A deep investigation of policy effects in fertility trends is unthinkable without the insight and knowledge of political scientists, and conversely the latter can benefit from demographic statisticians’ understanding of measurement issues (Frejka et al. , Overview Chapter ). Studies of mortality differentials gain from a sociological understanding of class differences. Demographic items are central to some recent theories about violent conflicts internal to populations. Demographers use elements from spatial theory in their studies of childbearing patterns among immigrants. Patterns of union formation contain important signals about the integration of minority groups, including immigrants (Basu and Aaby ), signals that should be useful to anthropologists. On the other hand, the latter are not always happy about the lack of attention paid to anthropological insight in classical demographic explanations of fertility trends in developing countries (Greenhalgh ).
Population Forecasts
Demographic Methodology
Producing population forecasts is a practical activity that many demographers are engaged in. Colleagues in other disciplines often resort to population forecasts produced by demographers.
Stocks and Flows; Timing and Quantum
Demography at the Crossroad of many Disciplines As a field of endeavor, demography is highly interdisciplinary. Today we would characterize as demography much of the early activity of academic statisticians, and there is a residue of common interest down to the present time. Historically, demography has also been allied particularly closely with actuarial mathematics, epidemiology, sociology, economics, geography, and history. More recently it has developed links with biology, political science, and anthropology. The overlap between scientific fields is often so great that individual scientists may find it difficult to pledge their allegiance solely to demography or to a neighboring discipline; they often see themselves as members of both. Even what initially looks like a purely demographic theory may really need an underpinning in other disciplines. One case in point is an understanding of the role that a reduction of mortality at young ages has in initiating a decline in fertility. To explain why fertility declines, why there is a link between mortality decline
Demography has a distinctive emphasis on stocks and flows. Conceptually, a population is subdivided into a number of subgroups between which individuals move. The subgroups may represent life statuses, like “married,” “divorced,” “childless,” “at parity ” (= mothers/fathers with one child), and individuals are seen as constantly exposed to the risk of transition from one subgroup (or status) into another. Deaths are counted among the flows, and a 7life table can be seen as a tabular representation of attrition from the stock of the living at the various ages. The underlying risks determine the timing of the flows, and the percentage that ultimately participates in a flow is called its quantum.
Data Cleaning and Careful Description Cleaning the data of incorrect and inaccurate records is part of a demographer’s preparation for analysis (Booth ). Demographers have always put great store on bringing out the facts, and this continues to be important in demographic investigations, while deeper explanation is often left to members of other disciplines despite any aspiration demographers may have toward the use (and development) of substantive theory. In this vein, demographers have a long tradition of careful description of empirical
D
D
Demography
regularities, mostly based on macro-data, but lately also covering patterns of individual-level behavior.
New Statistical Methods To analyze demographic (vital) processes as dependent “variables,” demographers have developed or adopted a number of statistical methods also used in other disciplines, notably public health and epidemiology. One can get an overview of the range of such methods as seen by demographers by consulting the various entries under ‘demographic techniques’ in Smelser and Baltes (). Entries under ‘population dynamics’ in the same source cover stable population theory, which is much used in demography. It is essentially based on extensions of branching-process theory. The advent of event-history techniques has recently induced a change in the way many demographers approach their investigations. Demographers have increasingly turned to individual-level analyses of multi-process models and to multi-level modeling. The latter has opened the way for the inclusion of population-level features as determinants in investigations of individual behavior. It has also become possible to incorporate a feature like unobserved heterogeneity and thus to pick up selectivity in demographic behavior. These new possibilities have strengthened the tendency to use probabilistic thinking in demographic investigations and thus to make the field more recognizable to mathematical statisticians.
Transition Rates of Type and Type To give an impression of how demographic methodology fits into the world of statistical theory but have a flavor of its own, let us first note that just like in any application of event-history analysis in a transition model with a piecewise-constant hazard specification, a count of event occurrences {Dj } and exposures {Rj } (time-units of exposure) for the various subgroups {j} of a population can be seen as a statistically sufficient set of statistics for the model risk parameters {µ j }. In such a situation, the occurrence/exposure rate µˆ j = Dj /Rj is a maximum-likelihood estimator of the parameter µ j , and demographers would call it a rate of the first kind. Suppose, for instance, that the subscript j stands for a combination of (discrete) age x attained and civil status c, where c = stands for “never married,” say. Then µ x, can be the “risk” of entering first marriage at age x (for someone who has never been married before that age) and the first-marriage rate of the first kind will be µˆ x, = Dx, /Rx, . When the occurrences can be fully sub-specified but the population cannot be subdivided by marital status but only by age attained, say, a first-marriage rate µ∗x = Dx, /Rx of the second kind can still
be computed for a given birth cohort, where Rx = ∑Rx,c c
is an aggregate over all marital statuses c. (Note that µ∗x is not an occurrence/exposure rate, because the denominator Rx includes the person-years ∑Rx,c also of individuals c≠
who are not under risk of first marriage in addition to the exposures Rx, for those who are under the risk of first marriage.) Demographers put rates of the second kind to uses of their own, but to a statistician, the main advantage of such rates is probably that in cohort data they can be converted into estimates rates of the first kind by means of the following formula: ⎛ ⎞ µˆˆ x, = µ∗x / − ∑ µ∗y . ⎝ y<x ⎠ For a proof, see the argument by Calot () leading up to his formula (). (Note that all quantities in this formula refer to the same birth cohort. No corresponding conversion formula seems to exist for period data.)
About the Author Dr. Jan Hoem is a Past President of the Scandinavian Demographic Society (–). He was Professor of Insurance Mathematics at Copenhagen University (–), Professor of Statistical Demography, Stockholm (–), where he is Guest Professor since . He was Director of the Max Planck Institute for Demographic Research (–), and is Director Emeritus since . Dr. Hoem was Section Editor for Demography of the International Encyclopedia of the Social and Behavioral Sciences (–). He was Editor of the electronic journal Demographic Research (–). Professor Hoem is Laureate of the International Union for the Scientific Study of Population (). He has authored and co-authored well over a hundred publications. He has initiated the use of Markov chains in life insurance mathematics, and has been instrumental in the development of event-history analysis in demography.
Cross References 7Census 7Demographic Analysis: A Stochastic Approach 7Life Table 7Population Projections 7Sociology, Statistics in
References and Further Reading Basu A, Aaby P () The methods and the uses of anthropological demography. Clarendon Press, Oxford Booth H () Demographic techniques: data adjustment and correction. In: Smelser N, Baltes P (eds) International encyclopedia
Density Ratio Model
of the social and behavioral sciences, vol . Pergamon, Oxford, pp – Caldwell JC () Demography, history of. In: Demeny P, McNicoll G (eds) Encyclopedia of population, vol . Macmillan Reference USA/Thomson Gale, New York, pp – Calot G () Demographic techniques: rates of the first and second kind. In: Smelser N, Baltes P (eds) International encyclopedia of the social and behavioral sciences, vol . Pergamon, Oxford, pp – Caselli G () Teaching demography in the early st century. Genus (special issue):– Demeny P, McNicoll G (eds) () Encyclopedia of population. Macmillan Reference USA/Thomson Gale, New York Frejka T, Sobotka T, Hoem JM, Toulemon L () Childbearing trends and policies in Europe. Demogr Res , Art. – Greenhalgh S () The social construction of population science: an intellectual, institutional, and political history of twentieth century demography. Soc Comp Study Soc Hist :– Haines MR () Demographic data regimes. In: Smelser N, Baltes P (eds) International encyclopedia of the social and behavioral sciences, vol . Pergamon, Oxford, pp – Hodgson D () Demography, twentieth-century. In: smelser N, Baltes P (eds) International encyclopedia of the social and behavioral Sciences, vol . Pergaman, Oxford, PP – Rosenzweig, Mark R, Oded Stark (eds) () Handbook of population and family economics, Amsterdom, Elsevier Smelser, Neil J, Paul B, Baltes (eds) () International Encyclopedia of the Social and behavioral sciences, Elsevier Szsreter, Simon () Demography, history of. In: Smelser N, Baltes P (eds) International encyclopedia of the social and Behavioral Sciences, vol . Pergamon, Oxford, pp –
Density Ratio Model Konstantinos Fokianos Associate Professor University of Cyprus, Nicosia, Cyprus
The problem of comparing two (or more samples) appears in several and diverse applications. The parametric theory resolves the problem by appealing to the well-known t-test. To carry out the t-test both samples are assumed to be normally distributed with common unknown variance and unknown means. The two-sample t-test enjoys several optimality properties, for instance, it is uniformly the most powerful unbiased test. Occasionally some (or all) of the needed assumptions fail; for instance, when there exists a group of observations with skewed distribution, then both assumptions of normality and equality of variances do not hold true. Hence, application of the ordinary two-sample t-test is questionable. The problem is usually bypassed after a suitable transformation but the comparison needs to be carried out in the transformed
D
scale. Alternatively, we can appeal to the nonparametric theory, which approaches the problem of comparing two samples by the so-called Mann–Whitney–Wilcoxon test (see 7Wilcoxon–Mann–Whitney Test). We consider a quite different approach to the twosample comparison problem. The methodology is relatively new and depends on the so-called density ratio model for semiparametric comparison of two samples. To be more specific, assume that X , . . . , Xn
∼
f (x)
Xn + , . . . , Xn
∼
f (x) = exp (α + βh(x)) f (x).
()
where fi (x), i = , are probability densities, h is a known function, and α, β are two unknown parameters. In principle, h(x) can be multivariate but we assume for simplicity that it is a univariate function. Model () is motivated by means of the standard 7logistic regression and the equivalence between prospective and retrospective sampling, Prentice and Pyke (). Suppose that Y is a binary response variable and let X be a covariate. The simple logistic regression model is of the form P[Y = ∣ X] =
exp(α ⋆ + βh(x)) , + exp(α ⋆ + βh(x))
()
where α ⋆ and β are scalar parameters. Notice that the marginal distribution of X is left completely unspecified. Assume that X , . . . , Xn is a random sample from F(x ∣ Y = ). Independent of the Xi , assume that Xn + , . . . , Xn is a random sample from F(x ∣ Y = ), and let n = n − n . Put π = P(Y = ) = − P(Y = ) and assume that f (x ∣ Y = i) = dF(x ∣ Y = i)/dx exists and represents the conditional density function of X given Y = i for i = , . A straightforward application of 7Bayes’ theorem shows that f (x ∣ Y = ) = exp(α + βh(x)) f (x ∣ Y = ) with α = α ∗ + log[( − π)/π]. In other words, model () is equivalent to () with α = α ∗ + log[( − π)/π]. We refer to () as the density ratio model since it specifies a parametric function of the log likelihood ratio of two densities without assuming any specific form about them. Hence, it is a semiparametric model and it is easy to see that, under the hypothesis β = , both of the distributions are identical. In other words, both density functions are assumed unknown but are related, however, through an exponential tilt–or distortion–which determines the difference between them. Notice that model () is quite general and includes examples such as the exponential and
D
D
Design for Six Sigma
partial exponential families of distributions. Model () provides a compromise between the fully parametric and nonparametric approaches to the problem of testing equality of two distribution, see Qin et al. (), Kedem et al. (), and Fokianos et al. (), among others, for applications of the density ratio model to real data problems. It is also important to note that () is a biased sampling model with weights depending upon parameters. Inference regarding biased sampling models has been discussed by Vardi (, ), Gill et al. (), and Bickel et al. (), in the case of completely known weight functions, while Qin and Zhang (), Qin (), Gilbert et al. (), Gilbert (), and Fokianos et al. () consider weight functions unknown up to a parameter. It is easy to see that () generalizes to the m–samples comparison problem. Consider m unknown densities that are related by an exponential tilt of the following form X , . . . , Xn ∼ f (x) = exp (α + β h(x)) fm (x), X , . . . , Xn ∼ f (x) = exp (α + β h(x)) fm (x), ... ... ...
()
Xm , . . . , Xmnm ∼ fm (x), where the notation follows (). Estimation of β , . . . , β m− as well as inference regarding the cumulative distribution functions that correspond to f (.), . . . , fm (.) has been considered by Fokianos et al. (), who also propose some test statistics for the hypotheses β = . . . = β m− = , that is, all the samples are identically distributed. In this sense, model () is also referred to as a semiparametric one-way ANOVA. In conclusion, the density ratio model for two and m samples avoids the normal theory by specifying that the log ratio of two unknown densities is of some parametric form. Hence, it provides another way of testing the equality of several distributions without resorting to transformations or any other techniques. The last comment is particularly useful since there are examples of data that show that populations follow skewed distributions and therefore classical estimation theory might yield questionable results. The suggested model accommodates skewed data and provides desirable results such as consistent estimators of means, test statistics, and so on.
About the Author Konstantinos Fokianos received his Ph.D. in Statistics from the University of Maryland at College Park in and is currently an Associate Professor with the Department of Mathematics and Statistics, University of Cyprus. His research interests include time series, bioinformatics, and semiparametric statistical inference. He
has authored/coauthored papers and the book Regression Models for Time Series Analysis (Wiley, with B. Kedem). He has been an elected member of the International Statistical Institute since . He serves as an associate editor for the Journal of Environmental Statistics, Statistics and Probability Letters and Computational Statistics & Data Analysis.
Cross References 7Exponential Family Models 7Logistic Regression 7Student’s t-Tests
References and Further Reading Bickel PJ, Klaassen CAJ, Ritov Y, Wellner JA () Efficient and adaptive estimation for semiparametric models. SpringerVerlag, New York. Reprint of the original Fokianos K, Kedem B, Qin J, Short DA () A semiparametric approach to the one-way layout. Technometrics :– Fokianos K, Sarrou I, Pashalidis I () A two-sample model for the comparison of radiation doses. Chemom Intell Lab Syst :– Gilbert PB () Large sample theory of maximum likelihood estimates in semiparametric biased sampling models. Ann Stat :– Gilbert PB, Lele SR, Vardi Y () Maximum likelihood estimation in semiparametric selection bias models with application to AIDS vaccine trials. Biometrika :– Gill RD, Vardi Y, Wellner JA () Large sample theory of empirical distributions in biased sampling models. Ann Stat :– Kedem B, Wolff D, Fokianos K () Statistical comparisons of algorithms. IEEE Trans Instrum Meas :– Prentice RL, Pyke R () Logistic disease incidence models and case–control studies. Biometrika :– Qin J () Inferences for case–control data and semiparametric two-sample density ratio models. Biometrika :– Qin J, Zhang B () A goodness of fit test for the logistic regression model based on case–control data. Biometrika :– Qin J, Barwick M, Ashbolt R, Dwyer T () Quantifying the change of melanoma incidence by Breslow thickness. Biometrics :– Vardi Y () Nonparametric estimation in the presence of length bias. The Annals of Statistics, :– Vardi Y () Empirical distribution in selection bias models. The Annals of Statistics, :–
Design for Six Sigma Rick L. Edgeman Professor and Chair & Six Sigma Black Belt University of Idaho, Moscow, ID, USA 7Six Sigma can be defined as a highly structured strategy
for acquiring, assessing, and applying customer, competitor, and enterprise intelligence in order to produce superior product, system or enterprise innovation and designs
Design for Six Sigma
(Klefsjö et al. ). Six Sigma originated approximately three decades ago as a means of generating near-perfect products via focus on associated manufacturing processes and while initially applied almost exclusively in manufacturing environments, its inherent sensibilities and organization facilitated migration to service operations. Similarly, while Six Sigma was at the outset used to generate significant innovation in and improvement of existing products, those same sensibilities led to its adaptation to new product and process design environments and it is on use of Six Sigma in design applications that the present contribution is focused. There is a distinction between using Six Sigma principles in design or innovation applications versus a process operating at a level of six sigma. In terms of performance, a process operating at a “true” six sigma level produces an average of only . defects per million opportunities (DPMO) for defects where this figure is associated with a process with a standard deviation spread between lower and upper specification limits, but wherein the . DPMO figure is based on allowance for a . standard deviation non-centrality factor or shift away from “perfect centering” so that, in essence, one specification limit is . standard deviations away from the targeted or ideal performance level whereas the other specification limit is . standard deviations away from that performance level. In practice, of course, a process may operate (typically) at lower or (rarely) higher sigma levels – that is, with less or more spread in standard deviation units between specification limits. Herein we are not focused on the “sigma level” per se, but rather on the design approach leading to the process or product in question that operates at some sigma level. This approach is called Design for Six Sigma (DFSS) and is conducive to higher sigma levels – that is, nearer to perfect results – while at the same time aligning with customer demands and desires or, in more customary language, the Voice of the Customer (VOC) expressed as “customer needs and wants”. While multiple DFSS approaches exist, a few similar ones dominate the application arena with the two most prevalent ones being referred to as I DOV (Innovation, Invention, Design, Optimization and Verification) and DMADV (Define, Measure, Analyze, Design, Verify) Algorithm with DMADV being more commonly applied of the two and hence emphasized herein. It should be emphasized that whichever DFSS approach is used, whether DMADV, I DOV, or another, that the approach provides freedom within structure rather than rigidity. That is to say that each phase in the chosen approach has a particular intent, that the phases are generally sequential and linked, and that together they are
D
complete, but that within a given phase many and differing tools and techniques can be brought to bear with those used potentially differing substantially from one application to the next. Turning now to the DMADV approach to DFSS we can provide the following brief descriptions of each portion of the algorithm. Define (D): A primary goal of the Define phase of DFSS is to acquire and access the VOC and subsequently align goals for the product, process, or service with the VOC. We note here that the customers considered should be both internal and external ones, as applicable. Among methods for acquiring the VOC are focus groups, sample surveys, and examination of customer complaints. Another method of value is to directly observe customer use of similar to products, processes or services so that unspoken, more implicit information can be gathered. In goal-setting it is recommended that these be so-called “SMART” goals, where SMART is an acronym for Specific, Measurable, Attainable, Relevant, and Time-bound. Further, while these goals should be attainable, they should not be “easily” attained, but should rather represent stretch goals, ones that are more likely to position the product, process, or service at the leading edge. Measure (M): In the DFSS context this requires that we measure and match performance to customer requirements. Fundamentally this is a quantification of the VOC and the alignment of this quantification with organizational and management goals. Analyze (A): This phase demands that the design for any existing relevant product, process or service be analyzed and assessed to determine its suitability, performance, error or defect sources, and any corrective or innovative actions that may be taken. Various tools are of potential value in this phase, including Design Failure Modes and Effects Analysis (DFMEA), a tool whose name belies its intent. Other useful tools and methods include Concept Generation and Selection, and the Theory of Inventive Problem Solving (TRIZ). Design (D): In this phase the array of corrective or innovative actions identified in the analyze phase are embedded in the design and subsequent deployment of new processes required to activate the VOC while simultaneously fulfilling organizational and management goals. While various tools may be of value here, a few of the more advanced approaches that are useful include many from experimental design and response surface analysis (Myers et al. ) along with more rigorous quality oriented approaches such as Quality Function Deployment (QFD). As a way of relating and integrating these latter approaches various customer needs and wants (the VOC) that are critical to QFD and its so-called House of Quality
D
D
Design of Experiments: A Pattern of Progress
(HOQ) can be regarded as response variables (Y’s) whose selected or joint optimization is attained through deployment of identified product or process design attributes are controllable variables X , X , . . . XP so that we have Y = f(X , X , . . . XP ) + ε Where the optimal combination of settings of X , X , . . . XP – called “engineering attributes” in the parlance of QFD – can be determined through use of, e.g., response surface methods, steepest ascent methods, and evolutionary operations or EVOP (Myers et al. ). It is important to note that it is not sufficient to simply identify the optimal combination of these variables as it is their specific means of deployment – the process – that ultimately actualizes the VOC. Verify (V): In the Verify phase the objective is to assess performance of the design via such means as prototyping, simulation, or direct observation of the designed product or process in use prior to marketplace deployment. In this way design performance is verified. From the description of DMADV it is easily concluded that a variety of statistical and other methods can be used to support its effectiveness and a few of these have been suggested herein. That said, it should be evident that the specific methods applied are almost boundless, being limited only as they are primarily by the knowledge and imagination of the design team. In all DMADV offers a logical and highly structured, yet versatile approach to product, process, or service design.
About the Author Rick Edgeman in Professor and Chair of Statistics at the University of ldaho and Professor in the Aarhus University (Denmark) Summer University and also serves on the Advisory Board of Hamdan Bin Mohammed E-University of Dubai. He has more than publications in leading journals to his credit with much of his work in the areas of Six Sigma, Sustainability, Innovation, Quality Management, Leadership, and Statistical Applications in Quality and Reliablity Engineering. In he was cited in Quality Progress as one of Voices of Quality for the st Century, one of only six academics worldwide so identified.
Cross References 7Business Statistics 7Industrial Statistics 7SIPOC and COPIS: Business Flow–Business Optimiza-
tion Connection in a Six Sigma Context 7Six Sigma
References and Further Reading Klefsjö B, Bergquist B, Edgeman R () Six sigma and total quality management: different day, same soup? Six Sigma & Competitive Advantage ():– Myers RH, Montgomery DC, Anderson-Cook CM () Response surface methodology: process and product optimization using designed experiments, rd edn. Wiley, New York
Design of Experiments: A Pattern of Progress D. R. Cox Honorary Fellow Nuffield College, Oxford, UK The article Experimental Design, Introduction to (Hinkelman ) sets out the basic statistical principles of experimental design. This supplementary note comments on the historical development of the subject. Careful experimentation has a long history, perhaps especially in the physical sciences. There is, however, a long history also of experimentation in fields as diverse as agriculture and clinical medicine. The first systematic discussion of experimental design in the presence of substantial haphazard variation seems to be that of Fisher (), later developed in his book (Fisher ). He set out four principles: ● ● ● ●
error control, for example by some form of matching to compare like with like independent replication to improve precision and allow its estimation randomization to achieve a number of aims, notably avoidance of selection biases factorial design to improve the efficiency of experimentation and to allow the exploration of interactions.
The relative importance of these four principles varies between subject-matter fields. This accounts to some extent for differences in how the subject has developed when directed toward, say, agricultural field trials as contrasted with some other fields of application. In the period up to these ideas were extensively developed, notably by Fisher’s friend and colleague, F. Yates, whose monograph (Yates ) is a little known masterpiece of the statistical literature. The focus and impact of this work was primarily but by no means exclusively agricultural. In the ’s a strand of new ideas entered from the chemical process industries notably with the work of G.E.P.
Designs for Generalized Linear Models
Box and his associates; see, for example, Box and Draper (). The differences were not so much that factors in a factorial experiment mostly had quantitative levels as that emphasis shifted from the estimation of factorial effects to the response surface of mean outcome considered as a smooth function of the factor level. This led to a richer family of designs and to an emphasis on exploring the form of the response surface in the neighborhood of an optimum. In some of the applications error was relatively small and experiments could be completed quite quickly allowing for a developing sequence of designs. Carefully designed clinical trials have a long history but the period from about onward saw their application and development on a large scale. Here typically there a very small number of possible treatments, often just two, and error is large, so that substantial replication is essential. Avoidance of various forms of selection bias by concealment achieved by 7randomization is often crucial. See Piantadosi () for a systematic account. Two more recent areas of development are applications to matters of public policy, for example in education and criminology. Here an issue concerns the extent to which a parallel with randomized clinical trials is appropriate. The second and very different application concerns the systematic sensitivity analysis of complex computer models involving many adjustable parameters and computationally highly intensive individual runs. Models of climate change are an example. In all these areas choice of a specific design in a particular context typically involves largely qualitative considerations of balancing the primary requirements of achieving precision and clarity and security of interpretation with practical constraints that are always present, although taking different forms in different fields. There may also be a number of distinct somewhat conflicting objectives. This often makes formal theoretical analysis of design choice difficult. Nevertheless a formal theory of design choice as an optimization problems has appeal both in the sense of showing the formal optimality of standard designs in specific circumstances and in guiding how to proceed in unusual situations, as for example there are specific technical constraints on the factor combinations that may be studied in a complex factorial experiment. A landmark result in such a theory of optimal design is the General Equivalence Theorem of Kiefer and Wolfowitz ().
For biography see the entry 7Survival Data.
Cross References 7Clinical Trials: An Overview 7Experimental Design: An Introduction 7Factorial Experiments 7Optimum Experimental Design 7Randomization 7Statistical Design of Experiments (DOE) 7Uniform Experimental Design
D
References and Further Reading Box GEP, Draper NR () Empirical model building and response surfaces. Wiley, New York Fisher RA () The arrangement of field experiments. J Minist Agric :– Fisher RA () The design of experiments. Oliver and Boyd, Edinburgh (and subsequent editions) Hinkelman K () This publication Kiefer J, Wolfowitz J () Optimal designs in regression problems. Ann Math Stat :– Piantadosi S () Clinical trials. Wiley, New York Yates F () The design and analysis of factorial experiments. Imperial Bureau of Soil Science, Harpenden
Designs for Generalized Linear Models André I. Khuri Professor Emeritus University of Florida, Gainesville, FL, USA
Introduction 7Generalized linear models (GLMs) represent an exten-
sion of the class of linear models. They are used to fit models in general situations where the response data under consideration can be discrete or continuous. Thus, GLMs provide a unified approach for the analysis of such data. Nelder and Wedderburn () are credited for having introduced these models. Three components are needed to define GLMs. These components are: (a) The response data are values of a random variable, denoted by y, whose distribution is of the exponential type. Its density function (or probability mass function for a discrete distribution) is of the form ℓ(y, θ, ϕ) = exp [
About the Author
D
θy − b(θ) + c(y, ϕ)] , a(ϕ)
()
where a(.), b(.), and c(.) are known functions, θ is a so-called canonical parameter, and ϕ is a dispersion parameter.
D
Designs for Generalized Linear Models
(b) The mean response, µ, is related to the so-called linear predictor, η, through a function, denoted by h, such that µ = h(η), which is assumed to be monotone and differentiable. The inverse function of h, denoted by g, is called the link function. Thus, η = g(µ).
()
The value of the mean response at a point, x = (x , x , . . . , xk )′ , in a k-dimensional Euclidean space is denoted by µ(x), and the corresponding value of η is denoted by η(x). Here, x , x , . . . , xk represent control variables that affect the response variable y. (c) The linear predictor is represented in terms of a linear model of the form ′
η(x) = f (x)β,
x ∈ R,
()
where R is a certain region of interest in the kdimensional space, f (x) is a known function of x, and β is a vector of p unknown parameters. It follows that the mean response value at x is given by µ(x) = h[η(x)] = h[ f ′ (x)β].
()
Given a set of n independent observations on the response y, namely y , y , . . . , yn , at n distinct locations in R, an estimate of β in () is obtained by using the method of maximum likelihood. This method maximizes the likelihood function given by
Using (), the variance of ηˆ (x) is approximately equal to Var[ηˆ (x)] ≈ f ′ (x)(X ′ W − X)− f (x).
()
The mean response, µ(x), in () can then be estimated by using the expression, ˆ ˆ µ(x) = h[ f ′ (x)β].
()
This estimate is called the predicted response at x. From () ˆ and () it follows that the variance of µ(x) is approximately equal to ˆ Var[ µ(x)] ≈ {h′ [η(x)]} f ′ (x)(X ′ W − X)− f (x), () where h′ [η(x)] is the derivative of h with respect to η evaluated at x. Formula () results from taking the variance of the first-order Taylor’s series approximation of h[ηˆ (x)] in a neighborhood of η(x). The expression on the right-hand side of () is called the prediction variance. ˆ It should be noted that µ(x) is a biased estimator of ˆ µ(x). A measure of closeness of µ(x) to µ(x) is given by its mean-squared error, namely, ˆ ˆ MSE[ µ(x)] = E[ µ(x) − µ(x)] ,
n
L(θ, ϕ, y) = ∏ ℓ(yi , θ i , ϕ),
In formula (), ⊕ is the direct sum notation, g ′ (µ i ) is the derivative of g(µ) with respect to µ evaluated at µ i , and b′′ (θ i ) is the second derivative of b(θ) with respect to θ evaluated at θ i (i = , , . . . , n). An estimate of the linear predictor, η(x), in () is given by ˆ ηˆ (x) = f ′ (x)β. ()
()
which can be partitioned as
i=
with respect to β, where θ i is a canonical parameter corresponding to yi (i = , , . . . , n), θ = (θ , θ , . . . , θ n )′ , and y = (y , y , . . . , yn )′ . The dispersion parameter, ϕ, is considered to have a fixed value that does not change over the values of yi (i = , , . . . , n). The maximum likelihood ˆ Details about the estimate (MLE) of β is denoted by β. ˆ computation of β can be found in McCullagh and Nelder (, Chap. ), McCulloch and Searle (, Chap. ), and Dobson (, Chap. ). The variance–covariance matrix of βˆ is approximately equal to (see, for example, McCulloch and Searle :) ˆ ≈ (X ′ W − X)− , Var(β)
n
i=
()
ˆ The second expression, E[ µ(x)] − µ(x), on the right-hand side of () is called the bias associated with estimating ˆ µ(x), and is denoted by Bias[ µ(x)]. We thus have ˆ ˆ ˆ MSE[ µ(x)] = Var[ µ(x)] + {Bias[ µ(x)]} .
()
This is called the mean-squared error of prediction (MSEP) evaluated at x. A second-order approximation of ˆ Bias[ µ(x)] is described in Robinson and Khuri ().
Choice of Design for GLMs ()
where X is the model matrix for the linear predictor in () based on the design matrix used to generate the response data, y , y , . . . , yn , and W is the diagonal matrix, W = ⊕{[g ′ (µ i )] a(ϕ)b′′ (θ i )}.
ˆ ˆ ˆ MSE[ µ(x)] = Var[ µ(x)] + {E[ µ(x)] − µ(x)} .
()
By a choice of design, it is meant the determination of the settings of the control variables, x , x , . . . , xk , that yield an estimated (or predicted) response with desirable properties. Desirability is assessed by having small values for the prediction variance in (), or small values for the MSEP in (). Thus, we can have designs that minimize the prediction variance, or designs that minimize the MSEP. The
Designs for Generalized Linear Models
former designs utilize criteria similar to those used in linear models, such as D-optimality and G-optimality, and are therefore referred to as variance-related criteria. However, because of the bias in estimating µ(x), it would be more appropriate to adopt a design criterion based on minimizing the MSEP in ().
The Design Dependence Problem One problem that faces the actual construction of a design for GLMs is the dependence of the design on β, the vector of unknown parameters in the linear predictor in (). This is true since η(x), and hence µ(x), depends on β. Consequently, the elements of the matrix W in (), which is used in the formulation of both the prediction variance and the MSEP, also depend on β. Thus, to minimize any design criterion function, some knowledge of β is required. This is quite undesirable since the purpose of any design is to estimate β in order to estimate the mean response µ(x). Common approaches to solving this design dependence problem include the following: (a) The specification of “guessed”, or initial, values of the unknown parameters involved. These values are used in the determination of the so-called locally-optimal design. Some references that discuss this approach include those by Mathew and Sinha (), Wu (), and Sitter and Wu (). (b) The sequential approach which starts by using initial values for the unknown parameters. The design derived from these values is then utilized to obtain estimates of the parameters which are then used as updated values leading to another design, and so on. Sequential designs were proposed by Wu (), Sitter and Forbes (), and Sitter and Wu (). (c) The Bayesian approach which assumes a prior distribution on the elements of β. This distribution is then incorporated into an appropriate design criterion by integrating it over the prior distribution. This approach was discussed by several authors. See, for example, Chaloner and Verdinelli () and Atkinson and Haines (). (d) The use of the quantile dispersion graphs approach. This more recent approach considers the MSEP in () as a criterion for comparing designs rather than selecting an optimal design. More specifically, quantiles of ˆ MSE[ µ(x)] values in () are obtained on several concentric surfaces inside a region of interest, R, which is a subset of the k-dimensional Euclidean space (recall that k is the number of control variables in the linear predictor model in ()). Let Rν denote the surface of a
D
region obtained by reducing R using a shrinkage factor, ν ( < ν ≤ ). Furthermore for a given design, D, let QD (p, β, ν) denote the pth quantile of the distribution ˆ of the values of MSE[ µ(x)] on Rν . Several concentric surfaces can be so obtained by varying the values of ν. In order to assess the dependence of the design D on β, a certain parameter space, C, to which β is assumed to belong, is specified. Then, the minimum and maximum values of QD (p, β, ν) with respect to β are computed over the parameter space C. This results in the following extrema of QD (p, β, ν) for each ν and a given p: Qmin D (p, ν) = min{QD (p, β, ν)} β∈C
Qmax D (p, ν) = max{QD (p, β, ν)}. β∈C
Plotting these values against p produces the socalled quantile dispersion graphs (QDGs) of the MSEP over the surface Rν for the design D. By repeating this process using several values of ν we obtain plots that depict the prediction capability of the design D throughout the region R. Several plots can be so constructed for each of several candidate designs for the model in (). A preferred design is one that has small max values of Qmin D and QD over the range of p ( ≤ p ≤ ). More details concerning this approach with examples can be found in Khuri and Mukhopadhyay (). The approach itself was first introduced in Robinson and Khuri (). The general problem of design dependence along with the aforementioned four approaches were discussed in detail by Khuri et al. ().
About the Author For biography see the entry 7Response Surface Methodology.
Cross References 7Generalized Linear Models 7Optimum Experimental Design
References and Further Reading Atkinson AC, Haines LM () Designs for nonlinear and generalized linear models. In: Ghosh S, Rao CR (eds) Design and analysis of experiments. North-Holland, Amsterdam, pp – Chaloner K, Verdinelli I () Bayesian experimental design: a review. Stat Sci :– Dobson AJ () An introduction to generalized linear models, nd edn. Chapman & Hall/CRC, Boca Raton Khuri AI, Mukhopadhyay S () GLM designs: the dependence on unknown parameters dilemma. In: Khuri AI (ed)
D
D
Detecting Outliers in Time Series Using Simulation
Response surface methodology and related topics. World Scientific, Singapore, pp – Khuri AI, Mukherjee B, Sinha BK, Ghosh M () Designs issues for generalized linear models: a review. Stat Sci :– Mathew T, Sinha BK () Optimal designs for binary data under logistic regression. J Stat Plan Inference :– McCullagh P, Nelder JA () Generalized linear models, nd edn. Chapman & Hall, London McCulloch CE, Searle SR () Generalized, linear and mixed models. Wiley, New York Nelder JA, Wedderburn RWM () Generalized linear models. J R Stat Soc A :– Robinson KS, Khuri AI () Quantile dispersion graphs for evaluating and comparing designs for logistic regression models. Comput Stat Data Anal :– Sitter RR, Forbes BE () Optimal two-stage designs for binary response experiments. Stat Sin :– Sitter RR, Wu CFJ () Optimal designs for binary response experiments: Fieller, D, and A criteria. Scand J Stat :– Sitter RR, Wu CFJ () Two-stage design of quantal response studies. Biometrics :– Wu CFJ () Efficient sequential designs with binary data. J Am Stat Assoc :– Wu CFJ () Optimal design for percentile estimation of a quantal response curve. In: Dodge Y, Fedorov VV, Wynn HP (eds) Optimal design and analysis of experiments. North–Holland, Amsterdam, pp –
Detecting Outliers in Time Series Using Simulation Abdalla M. El-Habil Head of the Department of Applied Statistics, Faculty of Economics and Administrative Sciences Al-Azhar University, Gaza, Palestine
Introduction 7Outliers have recently been studied more in the statistical time series literature and this interest is also growing in econometrics. Usually, time series outliers are informally defined as somehow unexpected or surprising values in relation to the rest of the series. Data of potential value in the formulation of public and private policy frequently occur in the form of time series. Most time series data are observational in nature. In addition to possible gross errors, time series data are often subject to the influence of some uncontrolled or unexpected interventions, for example, implementations of a new regulation, major changes in political or economic policy, or occurrence of a disaster. Consequently, discordant observations and various types of structural changes occur frequently in time series data. Whereas the usual time series
model is designed to grasp the homogeneous memory pattern of a time series, the presence of outliers, depending on their nature, may have a moderate to substantial impact on the effectiveness of the standard methodology for time series analysis with respect to model identification, estimation, and forecasting. Therefore, there is a clear need to have available methods to detect or accommodate them. Simulation data are derived from a sequence of pseudorandom numbers. These pseudorandom numbers are created by a random number generator. The generator requires an initial seed value from which to generate its first value. The random number generator creates both a random number and a new seed for the next value. The SIMULATE paragraph in the Scientific Computing Associate Corporation (SCA) program may be used to estimate an ARIMA model or a transfer function model. The use of the SIMULATE paragraph for the estimation of a transfer function model is identical as its use for the estimation of an ARIMA model, except for the presence of input series. The SIMULATE paragraph will first generate a noise sequence using a pseudorandom number generator. This sequence is then used according to a transfer function model specified lately using the TSMODEL paragraph.
Detecting Outliers of a Simulated AR() Time Series To facilitate our understanding of detecting outliers and their effects, for example, on the values of a simulated AR() process, we will assume that the constant of the proposed model is equal to zero. For this purpose, observations are simulated from the model zt = [/( − .B)]at with σa = .. The data are shown in Fig. . To illustrate, for example, the effect of an AO on the base AR() model, we include an AO at time t = with ωA = (the value ωA represents the amount of deviation from the “true” value of ZT). The new shape of data is shown in Fig. . By using the SCA program, only AO has been detected at t = , and we obtain the estimation results for an AR() fit of the simulated AR() process as the following:
φ estimate
S. E. of φ estimate
σa estimate
Without outlier
.
.
.
AO at time t =
.
.
.
Case
From the table, with the additive outlier at time t = , we can see that the parameter estimate is decreased
D
Detecting Outliers in Time Series Using Simulation 3.4
We select only the last values of xdata, ydata, and zdata to ensure that any potential irregularities in the beginning of the recursive computation of values are eliminated. By using the SCA program, we estimated the model
ZDATA
1.8 –0.2 –1.4
zdata = . + (ω)xdata + (ω)ydata + at,
–3.0 12
24
36
48 60 Time
72
84
96
Detecting Outliers in Time Series Using Simulation. Fig. Zdata
AO has been detected at t = , , and TC at t = . We obtain estimation results for a single-equation transfer function model (with two-input variables) fit of the simulated single-equation transfer function model (with two-input variables) process as the following:
6.6 4.2 ZDATA
Case
1.8
–0.6 –3.0 12
24
36
48 60 Time
72
84
96
Detecting Outliers in Time Series Using Simulation. Fig. Zdata
by approximately ., the estimated residual variance is inflated, and in consequence the prediction intervals can be too wide. In turn, it will affect the model identification, estimation, and forecasting.
Simulation of a Single-Equation Transfer Function Model (with Two-Input Variables) To detect outliers and study their effects on the values of a simulated single-equation transfer function model (with two-input variables), observations are simulated from the model zdata = . + (.)xdata + (.)ydata + at, where the model of xdata is ( − .B)xdata = . + at, and the model of ydata is ( − .B)ydata = . + ( − .B)at, with σa = ..
S. E. of S. E. of Estimate estimate Estimate estimate Estimate of ω of ω of ω of ω of σa
Without outlier
.
.
.
.
.
AO at t = , , and TC at t =
.
.
.
.
.
As we see from the table, the parameter estimates are moderately changed, and the estimated residual variance is inflated. Thus, the presence of those extraordinary events could easily mislead the conventional time series analysis.
Simulation of Simultaneous Transfer Function (STF) Model In order to detect outliers and study their effects on the values of a simulated simultaneous transfer function model, observations are simulated from the models Zdata = . + ( − .B)at, Zdata = . + ( − .B)at, with σa = .. By using the SCA program, we estimated the two models simultaneously using the STFMODEL JOINTMDL paragraph. TC has been detected at t = and IO at t = . We get estimation results for simultaneous transfer
D
D
Detection of Turning Points in Business Cycles
function model fit of the simulated simultaneous transfer function process as the following: Estimation results for the simultaneous transfer function fit of the simulated transfer function process (first model)
Case
S. E. of S. E. of Estimate estimate Estimate estimate of zdata of zdata of zdata of zdata
Without outlier
.
.
.
.
TC at t =
.
.
.
.
Palestine. He is Past Dean of the Faculty of Economics & Admin. Sciences (–). He is a member of The International Association for Statistical Education and The International Biometrics Society. He is a member of The Consultative Council of Professional Statistics for Palestine. He was a Chairman of the Palestinian Economists and Statisticians Association (–).
Cross References 7Multivariate Outliers 7Outliers 7Time Series
References and Further Reading Estimation results for the simultaneous transfer function fit of the simulated transfer function process (second model)
Case Without outlier TC at t =
S. E. of S. E. of Estimate estimate Estimate estimate of zdata of zdata of zdata of zdata .
.
.
.
−.
.
.
.
As we see from the above two tables, the parameters estimates are changed, and the estimated residual variance is inflated. So, those outliers could easily mislead the conventional time series analysis.
Summary In this entry, simulations for detecting outliers and studying their effects on the values of AR() time series, transfer function model with one-input variable, transfer function model with two-input variables processes, and simultaneous transfer function (STF) are conducted using the STFMODEL JOINTMDL paragraph in the Scientific Computing Associate Corporation (SCA) program. The conclusion, which we come up with, is that the presence of outliers, depending on their nature, may have a moderate to substantial impact on the effectiveness of the standard methodology for time series analysis with respect to model identification, estimation, and forecasting.
About the Author Dr. Abdalla El-Habil is Professor of Statistics, and Head of the Department of Applied Statistics at the Faculty of Economics & Admin. Sciences, Al-Azhar University – Gaza,
Box GEP, Jenkins GM () Time series analysis: forecasting and control. Holden Day, San Francisco Chen C, Liu LM () Joint estimation of model parameters and outlier effects in time series. J Am Stat Assoc :– Lon-Mu L, Hudak GB (–) Forecasting and time series analysis using the SCA Statistical System, vols and . Scientific Computing Associates, Dekalb Tsay RS () Time series model specification in the presence of outliers. J Am Stat Assoc :–
Detection of Turning Points in Business Cycles Marianne Frisén Professor, Statistical Research Unit University of Gothenburg, Gothenburg, Sweden
Turns in Business Cycles A turn in a business cycle is a change from a phase of expansion to one of recession (or vice versa). Both government and industry need to have systems for predicting the future state of the economy, for example in order to timely predict the shift from a period of expansion to one of recession. Warnings of a turn can be given by using information from one or several time series that are leading in relation to the actual business cycle. A system for detecting the turning points of a leading indicator can give us early indications on the future behavior of the business cycle. As pointed out for example by Diebold and Rudebusch (), Kim and Nelson () and Birchenhall et al. (), two distinct but related approaches to the characterisation and dating of the business cycle can be discerned. One approach emphasizes the common movements of several variables. This approach is pursued for example by Stock and Watson (). The other approach, the
Detection of Turning Points in Business Cycles
regime shift, is the one pursued in the works by Neftci (), Diebold and Rudebusch (), Hamilton (), Jun and Joo (), Lahiri and Wang (), Layton (), Birchenhall et al. (), Koskinen and Öller () and Andersson et al. (, ). An important issue is which characteristic of the leading index best predicts a turn in the business cycle. The question remains whether the most useful predictor is the level, as in Birchenhall et al. (), the transition and level, as in Hamilton () and Koskinen and Öller (), or the transition and change in monotonicity, as in Andersson et al. (, ).
The Detection Problem There is a need for methods for early warning, i.e., methods for the timely detection of a regime shift in a leading index. For reviews and general discussions on the importance of timeliness in the detection of turning points, see for example Neftci (), Zarnowitz and Moore (), Hackl and Westlund (), Zellner et al. (), Li and Dorfman () and Layton and Katsuura (). The inference situation can be described as one of 7surveillance, since a time series is continuously observed with the goal of detecting the turning point in the underlying process as soon as possible. Repeated decisions are made, the sample size is increasing and no null hypothesis is ever accepted. Thus, the inference situation is different from that where we have a fixed number of observations. The aim of statistical surveillance is the timely detection of important changes in the process that generates the data. A process X (a leading economic indicator) is under surveillance, where X is often measured monthly or quarterly. Based on the available observations, we decide whether the observations made so far indicate a turn. Thus at every decision time s, we use the alarm system to decide whether there has been a turn or not. This can be formulated as discriminating between two events at each decision time: D = “the turn has not occurred yet” and C = “the turn has occurred”.
Models An important question is which assumptions can be made about the process. Economic time series often exhibit seasonal variation. Unfortunately, most data-driven filters can seriously alter the turning point times. Autocorrelation can be a problem when the sampling intervals are short (Luceno and Box ()). When this is expected to be a problem, methods that adjust for autocorrelation should be used. Ivanova et al. () argue that the
D
effect of the autoregressive parameters will largely be captured by the probabilities of remaining in the current state. Many macroeconomic variables can be characterized as cyclical movements around a trend. In order to distinguish the movements and make the time series stationary in relation to the cycle it is sometimes judged necessary to adjust for the trend. Adjusting for the trend by data transformation may result in a distortion of the characteristics of the original series. Canova () points out that the trend may interact with the cyclical component, and therefore it may be difficult to isolate. In the common movement approach, a business cycle is characterized as the cyclical movement of several economic activities. This is one example of how multivariate data are used. The common movement approach demonstrates that important information is contained in the relation between the turns of various indices. This information can be utilized either by transforming the problem to a univariate one (by using a composite index of leading indicators) or by applying a special method for surveillance of multivariate data. See for example Ryan (), Frisén () and Frisén et al. (). When estimating the parameters of the monitoring system, historical data are often used. The user of a system for on-line detection is faced with the paradox that the parameters in the surveillance system may be estimated using previous data, which means that it is assumed that previous patterns will repeat themselves. However, the aim of the surveillance method is to detect changes, and by estimating parameters from previous data, the ability to detect changes in the current cycle might be diminished. Sarlan () examines the change in intensity and duration of US business cycles and concludes that the modern business cycle is different from the historical one. The approach described by Andersson et al. (, ) of a non-parametric method may be preferred in order to avoid the risk of misleading results. The nonparametric method does not assume that all phases are of the same type or have the same level and parametric shape. In practice, this varies a lot. At time τ there is a turn in the expected value, µ. Different assumptions can be made about µ, conditional on D (expansion) and C (recession), and at a turn. Instead of assuming that the parametric shape is known, we can use monotonicity restrictions to define µ under C and D. Then the aim (at the detection of a peak) is to discriminate between the following two events: D(s) : µ() ≤ . . . ≤ µ(s) and C(s): “µ() ≤ . . . ≤ µ(τ − ) and µ(τ − ) ≥ µ(τ) ≥ . . . ≥ µ(s)”. The monotonicity restrictions for a trough are the opposite. In such situations the exact parametric shape of µ is unknown. We only know that µ is monotonic within each phase.
D
D
Detection of Turning Points in Business Cycles
Methods In recent years, methods based on likelihood have been in focus. The performance of three methods for turning point detection in leading indicators is compared in detail by Andersson et al. (, ). All three methods are based on likelihood, but there are differences in model specifications, the amount of information used and parameter estimation. The Hidden Markov Model (HMM) is suggested for business cycle modeling for example by Hamilton (), Lahiri and Moore (), Lahiri and Wang (), Layton (), Gregoir and Lenglart () and Koskinen and Öller (). The HMlin method is based on regime switching and has a model which is piecewise linear. In many ways it is similar to, for example, the method presented by Koskinen and Öller (). The SRlin method is based on the Shiryaev-Roberts (SR) technique under the assumption of a piecewise linear model. The SRnp method is a generalized version of the SR method. It is a non-parametric version of the SRlin method with no parametric assumption on the shape of the curve. It uses only the monotonicity change and not the level. The safe way of the SRnp method was recommended since there is no major loss of efficiency. Andersson et al. (, ) illustrated the methods by monitoring a period of Swedish industrial production. Evaluation measures that reflect timeliness were used.
About the Author For biography see the entry 7Surveillance.
Cross References 7Business Statistics 7Economic Statistics 7Forecasting: An Overview 7Surveillance
References and Further Reading Andersson E, Bock D, Frisén M () Statistical surveillance of cyclical processes with application to turns in business cycles. J Forecasting :– Andersson E, Bock D, Frisén M () Some statistical aspects on methods for detection of turning points in business cycles. J Appl Stat :– Birchenhall CR, Jessen H, Osborn DR, Simpson P () Predicting us business-cycle regimes. J Bus Econ Stat :– Canova F () Does detrending matter for the determination of the reference cycle and the selection of turning points? Econ J :– Diebold FX, Rudebusch GD () Scoring the leading indicators. J Bus :– Diebold FX, Rudebusch GD () Measuring business cycles: a modern perspective. Rev Econ Stat :–
Frisén M () Principles for multivariate surveillance. In: Lenz H-J, Wilrich P-T, Schmid W (eds) Frontiers in statistical quality control , pp – Frisén M, Andersson E, Schiöler L () Evaluation of multivariate surveillance. J Appl Stat : Gregoir S, Lenglart F () Measuring the probability of a business cycle turning point by using a multivariate qualitative hidden markov model. J Forecasting :– Hackl P, Westlund AH () Statistical analysis of “structural change”, Empirical Econ :– Hamilton JD () A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica :– Ivanova D, Lahiri K, Seitz F () Interest rate spreads as predictors of german inflation and business cycles. Int J Forecasting :– Jun DB, Joo YJ () Predicting turning points in business cycles by detection of slope changes in the leading composite index. J Forecasting :– Kim C-J, Nelson CR () Business cycle turning points, a new coincident index, and tests of duration dependence based on a dynamic factor model with regime switching. Rev Econ Stat :– Koskinen L, Öller L-E () A classifying procedure for signalling turning points. J Forecasting :– Lahiri K, Moore G () Leading economic indicators: new approaches and forecasting record. Cambridge University Press, Cambridge Lahiri K, Wang JG () Predicting cyclical turning points with a leading index in a markov switching model. J Forecasting :– Layton AP () Dating and predicting phase changes in the U.S. business cycle. Int J Forecasting :– Layton AP () A further test of the influence of leading indicators on the probability of us business cycle phase shifts. Int J Forecasting :– Layton AP, Katsuura M () Comparison of regime switching, probit and logit models in dating and forecasting us business cycles. Int J Forecasting :– Li DT, Dorfman JH () Predicting turning points through the integration of multiple models. J Bus Econ Stat :– Luceno A, Box GEP () Influence of the sampling interval, decision limit and autocorrelation on the average run length in cusum charts. J Appl Stat :– Neftci S () Optimal prediction of cyclical downturns. J Econ Dynam Control :– Ryan TP () Statistical methods for quality improvement. Wiley, New York Sarlan H () Cyclical aspects of business cycle turning points. Int J Forecasting :– Stock JH, Watson MW () A procedure for predicting recessions with leading indicators: econometric issues and recent experience. In: Stock JH, Watson MW (eds) Business cycles, indicators and forecasting, vol . University of Chicago Press, Chicago, pp – Zarnowitz V, Moore GH () Sequential signals of recessions and recovery. J Bus :– Zellner A, Hong C, Min C-K () Forecasting turning points in international output growth rates using Bayesian exponentially weighted autoregression, time-varying parameter, and pooling techniques. J Econometrics :–
Dickey-Fuller Tests
Dickey-Fuller Tests David A. Dickey William Neal Reynolds Professor North Carolina State University, Raleigh, NC, USA One of the most basic and useful of the time series models is the order ( lag) autoregressive model, denoted AR() and given by Yt − µ = ρ(Yt− − µ)+et where Yt is the observation at time t, µ is the long run mean of the time series and et is an independent sequence of random variables. We use this venerable model to illustrate the Dickey–Fuller test then mention that the results extend to a broader collection of models. When written as Yt = µ( − ρ) + ρYt− + et , or more convincingly as Yt = λ + ρYt− + et , with e independent and identically distributed as N(, σ ), the AR() model looks like a regression with errors satisfying the usual assumptions. Indeed the least squares estimators of the coefficients are asymptotically unbiased and normally distributed under one key condition, namely that the true ρ satisfies ∣ρ∣ < . It appears that this assumption is quite often violated. Many prominent time series appear to have ρ = , in which case Yt − µ = ρ(Yt− − µ) + et becomes Yt = Yt− + et or Yt − Yt− = et . That is to say there are many series whose first differences Yt −Yt− seem to form a sequence of independent shocks, as the e’s are often called. For estimation of the parameters, only mild assumptions on e are required. Normality is not necessary as long as the sample size is reasonable and ∣ρ∣ < . Two things are worth noting. The first is that the long term mean µ has dropped from this equation, that is, there is no tendency to move toward a long term mean, no mean reversion. A series that is high at time t is just as likely to move up as to move down. There is nothing to be gained by assessing the distance from the historic mean. The second point is a recommendation to write the model as Yt − µ = ρ(Yt− − µ) + et . The representation Yt = λ + ρYt− + et masks the relationship λ = µ( − ρ). The uninformed analyst may not realize that the intercept disappears when ρ = . The case when ∣ρ∣ < falls into the category of stationary autoregressive models whereas the model with ρ = is an example of a nonstationary model. For reasons discussed later, the ρ = case is also called a unit root model and sometimes an integrated model. There are many series of economic interest for which a test of the hypothesis H : ρ = is desired. The interest in such tests can be explained with a hypothetical scenario. Suppose an autoregressive order model Yt = . + .(Yt− −.)+et describes the price of a stock over some
D
time index t. Now if the current price is Y = . then the expected value of the next observation is . + .(. − .) + = ., closer than the current . to the mean . Likewise we’d predict a rise to ., were we currently at ., that is, the strategy of selling high and buying low would make money in the long run. The ability to forecast the direction of the stock market would cast doubt on the assumption that it was a quickly responding market with fully informed participants so we think that . is an estimate of ρ = and expect that the null hypothesis will not be rejected. In contrast, we might expect the ratio of short term to long term bond yields or interest rates to be stable, that is, we expect some long term equilibrium ratio and we expect movement of the ratio toward that number in the long run. Here we expect to reject H : ρ = in favor of H : ρ < . On a logarithmic scale, we want to test Yt = log(short term rate)−log(long term rate) for stationarity. When two individual series are nonstationary but their difference is stationary, the series are said to be “cointegrated.” Some of the most interesting hypothesis tests in economics concern constructed variables, as does this test for cointegration. Having established an interest in testing the hypothesis H : ρ = , we return to the statement that the least squares estimator is not normal under this hypothesis. How can this be when the errors satisfy all the usual regression assumptions? The answer lies in the assumption on the independent (X) variables. In usual regression theory, these X variables are assumed fixed and measured without error. Extensions to random X cases typically involve a conditioning argument, that is, we consider what would happen if we only looked at repeated samples that have the same X values that we observed. There is no way to make that argument here, where the X variables are just the Y (dependent) variables at a different time. When ρ = and µ is known to be , the theory of weak convergence as in Billingsley () implies that the least squares estimator ρˆ satisfies a limit in terms of a standard Wiener process W(t) : n
n( ρˆ − ) =
∑ Yt− et
t= n
∑ Yt−
t=
L
−Ð→
W () − . W (t)dt ∫
This is known as the “normalized bias” statistic. This is an appealing mathematical representation, but is nevertheless a random quantity. Knowing this expression does not lead directly to a distribution and resulting table of critical values for testing. With simulation, empirical percentiles for various sample sizes n can be computed, however it is not clear how large an n will get the simulations close
D
D
Dickey-Fuller Tests
enough to the limit so that no further n values need be considered. In addition to running rather large simulations as just mentioned, Dickey and Fuller () calculated the eigenvalues of the limits of the numerator and denominator quadratic forms in the above normalized bias and from those directly simulated the limit distribution. They were then able to see how large an n was required to get close to that limit. This resulted in critical values for the normalized bias n( ρˆ − ) for finite n and the limit. Statistical programs that do least squares regression also produce t tests. Dickey and Fuller looked at the t statistic associated with the mean regression. Its limit is the same as that −/ n
n
of (σ ∑ Yt− ) t=
∑ Yt− et . They gave percentiles for that
t=
statistic as well and called it τ to indicate that it did not behave like the usual t, even in the limit. At this point two distributions, one for the normalized bias and one for τ have been discussed, but the underlying assumption that µ is makes this result of little practical interest. Suppose you have a stationary (∣ρ∣ < ) AR() time series whose values range between and . With no intercept in the model, the estimator n
n
∑ Y t− Y t t= n
∑ Y t−
t=
≈ − .
∑ (Y t −Y t− ) t=
, where the sum of squared
n
∑ Y t−
t=
differences can be no greater than (n − )( ) and the denominator is no less than (n − )( ) so the estimate of ρ is between . and . Simulations (Dickey ) verify that, as the algebra suggests, stationary series with positive means give tests that rarely reject the (false) null hypothesis when no intercept is included in the regression. It thus becomes of interest to study the mean adjusted n
estimator n( ρˆ µ − ) =
∑ (Y t− −Y)e t t= n
∑ (Y t−
−Y)
n
, where Y = ∑
t=
Yt , n
t=
and its associated studentized statistic τ µ . An asymptotically equivalent estimator and τ µ statistic are obtained by regressing Yt on an intercept and Yt− . Dickey and Fuller show that the addition of an intercept changes even the asymptotic distribution rather dramatically. This can be demonstrated either through their quadratic form approach or the Wiener Process limit representation. Applying the same logic to a series that appears to be moving upward or downward at a rather steady long run rate, a fair test for “trend stationarity” as the alternative hypothesis is called, one that gives a chance to both the null and alternative hypothesis, must use a model that can capture the trend under both hypotheses. A logical candidate is a model with linear trend and AR() error. The model can be expressed in two ways: Yt − α − βt = ρ(Yt− − α − β(t − )) + et
or Yt = α + βt + Zt ,
Zt = ρZt− + et .
The first of these models most easily demonstrates that under our null hypothesis ρ = , we have Yt = β + Yt− + et a random walk with drift β. The former slope is now called a drift and algebraically appears as an intercept term but in reality, it represents the long term monotone trend regardless of the ρ value. It is, after all, the mean change in Yt per unit change in t. Its inclusion accomplishes our purpose of allowing both the null and alternative model to capture the long term trend, thus separating the alternative hypothesis case of stationary errors around a linear trend from the null hypothesis case of random walk with drift. The alternative hypothesis defines the concept of trend stationarity. The model can be fit by regressing Yt on an intercept, t, and Yt− or, in an asymptotically equivalent way, by regressing Yt on and t, obtaining residuals rt , and then regressing rt on rt− with no intercept. This second approach is motivated by the second form of the model above. As with the mean adjusted case, this results in further changes in the distributions of the estimate and its studentized statistic τ τ , even asymptotically. Since this last model subsumes the others, why not always use an intercept and time trend? The answer lies in a substantial loss of power when unnecessary parameters are added to the model. We now summarize a few numerical results from the literature. Tables of critical values for the normalized bias and studentized statistics are developed in Dickey and Fuller (, ) and reported in detail in Fuller (). The % and % critical values from Fuller’s text are reported below for the smallest tabulated sample size n = and for the limit distribution. The studentized statistics (top rows of Table ) have the more stable percentiles over different sample sizes, especially at the % level and are more often used in practice. Dickey (, Table .) shows what happens when there are nonzero intercept or trend parameters in the generated data that are not in the model. For even modest values of these omitted parameters, there are almost no rejections of the (false) null hypothesis when ∣ρ∣ < . In Table . of that paper we find empirical powers for replicates of these tests for data generated by Yt = ρYt− + et , t = , , . . . , . Substantial loss of power results when unneeded intercept or trend parameters are added to the model as in columns ,,, and of Table , which displays a few entries from the power study. Clearly inclusion of either too many or too few deterministic terms in the model can cause power loss. Failing
Dickey-Fuller Tests
Dickey-Fuller Tests. Table Left tailed unit root test percentiles %,
%,
%,
%,
Model
n =
limit
n =
limit
No intercept τ
−.
−.
−.
−.
Intercept τ µ
−.
−.
−.
−.
Linear trend τ τ
−.
−.
−.
−.
No intercept n(̂ ρ − )
−.
−.
−.
−.
Intercept n(̂ ρ µ − )
−.
−.
−.
−.
Linear trend n(̂ ρ τ − )
−.
−.
−.
.
Dickey-Fuller Tests. Table Some empirical powers n(̂ ρ − ) n(̂ ρ µ − ) n(̂ ρ τ − )
True ρ
τ
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
τµ
ττ
to reject unit roots when a model without trend is fit to trending data may do less damage than rejecting. Forecasting future values to be the same as the present value may be less damaging than giving mean reverting forecasts that go in the opposite direction of the trend. The autoregressive order model is common but is not rich enough to fit all time series. Consider an autoregressive model of order , Yt − µ = .(Yt− − µ) − .(Yt− − µ) + .(Yt− − µ) + et or, written in terms of the backshift operator B(Yt ) = Yt− , ( − .B + .B − .B )(Yt − µ) = et Because − . + . − . = and µ is constant over time, µ drops out of the equation. The factored form ( + .B)( − .B)( − B)(Yt − µ) = et , shows that the first differences, ( − B)(Yt − µ) = Yt − Yt− , form a stationary process ( + .B)( − .B)(Yt − Yt− ) = et . The characteristic polynomial is the backshift polynomial with B considered an algebraic variable. An autoregressive series is stationary if the roots of its characteristic polynomial all exceed in magnitude. The order autoregressive example
D
here has a root , a “unit root.” It is therefore nonstationary but its differences form a stationary (roots /. = and −/. = −.) autoregressive order series. Dickey and Fuller proved that if the first difference Yt − Yt− of a unit root autoregressive process is regressed on the lagged level Yt− of the process (and possibly an intercept and trend) and enough lagged differences to produce the appropriate lag structure, the studentized statistic on the lag level term will have the same limit distribution as in the autoregressive order case. The tables discussed above can thus be used. In contrast, the distribution of the normalized bias is altered by the presence of the lagged differences which is one reason the studentized statistics are preferred despite the power advantage of the normalized bias test in the simplest AR() cases. The characteristic polynomial here has the form (−a B)(−a B)(−a B) with roots /ai , i = , , . If (and only if) B = is a root, we will have (−a )(−a )(−a ) = . These expressions are algebraically equivalent: (I) (II) (III)
( − a B)( − a B)( − a B)(Yt − µ) = et Yt − µ = (a + a + a )(Yt− − µ) − (a a + a a + a a )(Yt− − µ) + a a a (Yt− − µ) + et Yt −Yt− = −(−(a +a +a )+(a a +a a +a a )− a a a )(Yt− − µ) + (a a + a a + a a − a a a ) (Yt− − Yt− ) + a a a (Yt− − Yt− ) + et
Expression (III) is the motivation for the regression approach. The coefficient on (Yt− − µ) is −( − a ) ( − a )( − a ), the negative of the characteristic polynomial evaluated at B = so a unit root (any ai = ) makes this coefficient and removes µ from the model. There is, then, no mean reversion. The lagged differences are known as augmenting terms and the resulting tests are known as ADF or augmented Dickey Fuller tests. Focusing on one of the three factors in (−a )(−a )(−a ), clearly it is multiplied by the other two. Under the null hypothesis that one ai value is , the coefficient on Yt− in (III) is an estimate of but is scaled by the other two factors. This affects the distribution of the coefficient, but not that of its τ statistic. As in any regression, division of the estimate by its standard error makes τ a self normalizing statistic. Some estimate of those other two factors is needed to scale the normalized bias so that it is asymptotically equivalent to the AR() case. Other results worth mentioning include the work of S. E. Said (Said and Dickey ). Said showed that even if a time series contained an invertible moving average part, one could use an augmented autoregressive model. Given a number of augmenting lagged differences that increases with n at the proper rate, the studentized statistic has the same limit distribution as in the autoregressive
D
D
Discriminant Analysis: An Overview
order model. One can simply fit a model with augmenting lagged differences to test for unit roots. Park and Fuller () use a weighted regression resulting in considerable power improvement. Elliott et al. () incorporate a generalized least squares trend and intercept estimator that results in a test with similar power increase. Pantula et al. () compare a test of Gonzalez-Farias, based on maximization of the exact stationary likelihood, and these other two. They all have competitive power. Phillips and his students have made many contributions in the area. Among them, Phillips and Perron () suggest a different adjustment for the higher order models and show that the tables described herein can be used for the test under weak assumptions on the errors.
Pantula SG, Gonzalez-Farias G, Fuller WA () A comparison of unit root criteria. J Bus Econ Stat :– Park HJ, Fuller WA () Alternative estimators and unit root tests for the autoregressive process. J Time Ser Anal :– Phillips PCB, Perron P () Testing for a unit root in time series regression. Biometrika :– Said SE, Dickey DA () Testing for unit roots in autoregressivemoving average models of unknown order. Biometrika :–
Discriminant Analysis: An Overview
About the Author
Barry J. Babin Max P. Watson, Jr. Professor and Department Head Louisiana Tech University, Ruston, LA, USA
Professor David Dickey, with Wayne Fuller, developed a Unit Root Test, in . His paper “opened-up a new way research agenda in time series econometrics, the investigation and identification of nonstationary processes” (Lex Oxley, The “Top ” Papers in Econometrics, –). He was Program Chair, Business and Economics section American Statistical Association (). He is an elected Fellow () of the American Statistical Association. In addition to writing four books, Dr. Dickey has been published in numerous papers and has given over presentations at a variety of professional events and organizations. He has also been recognized as a member of the Academy of Outstanding Teachers at North Carolina State University.
Discriminant analysis is a multivariate technique that helps the user explain group membership as a function of multiple independent variables. In particular, discriminant analysis is appropriate when the user is faced with a situation involving a categorical (nominal or ordinal) dependent variable and independent variables that are primarily continuous (interval or ratio) although categorical independent variables can be included under some conditions. This would typically involve dummy coding of the categorical variables. In many cases, a dichotomous dependent variable is employed although it can be multichotomous. The variate takes the familiar form: V = w x + w x + . . . + wi xi + ei
Cross References 7Autocorrelation in Regression 7Bayesian Approach of the Unit Root Test 7Seasonal Integration and Cointegration in Economic
Time Series
References and Further Reading Billingsley P () Convergence of probability measures. Wiley, New York Dickey DA, Fuller WA () Distribution of the estimators for autoregressive time series with a unit root. J Am Stat Assoc :– Dickey DA, Fuller WA () Likelihood ratio statistics for autoregressive time series with a unit root. Econometrica :– Dickey DA () Power of unit root tests. In: Proceedings of the business and economic statistics section. American Statistical Association, Philadelphia, Washington, DC, pp – Elliott G, Rothenberg TJ, Stock JH () Efficient tests for an autoregressive unit root. Econometrica :– Fuller WA () Introduction to statistical time series. Wiley, New York
where x represents the independent variables, w represents an empirically derived parameter estimate that weights the extent to which the independent variable determines group membership, and V represents the resulting variate value which determines group membership. The variate takes on the form of a Z score. This metric value is ultimately the key indicator of which group an observation should belong to based on the corresponding set of independent variable values. Groups are formed based on these Z-values by determining a cutting score which is used to separate observations into two groups. Discriminant analysis applies the fundamental logic that observations that produce similar Z-values (based on values for the set of independent variables) should be grouped together and observations with dissimilar Z-values are different and should not be grouped together. Thus, for a dichotomous dependent variable, the cutting score can be thought of as: ZCS =
NA ZB + NB ZA NA + NB
Discriminant Analysis: An Overview
Where, Zcs is the cutting score and N represents the sample size in groups A and B, respectively. ZA and ZB represent the centroids of groups A and B formed by splitting all observations into groups using the cutting score. Considering the dichotomous case, the observed values for each independent variable can be used to predict whether each observed value for a dependent variable is a “” or “.” The values for the independent variables are multiplied by derived parameter estimates just as in multiple regression. In other words, the analysis determines which group an observation should belong to based upon the calculated value? A classification matrix is created which shows how accurately the model can actually place these observations into categories. This matrix is formed by taking a conditional frequency of predicted group membership against actual observed group membership. The contingency table takes a form as follows:
Actual Group
Predicted Group Membership
Membership
Group
Group
Group
Correctly classified
Incorrectly classified
Group
Incorrectly classified
Correctly classified
The percent of observations correctly classified becomes one measure of a discriminant function’s effectiveness. Consider that if group had two times as many respondents as group , correct classification could be percent simply by placing all observations into group . Therefore, the discriminant analysis should be able to produce better than percent correct classifications if it is to be useful. In fact, a t-test is available to determine the significance level for classification accuracy: t= √
p−g g(. − g) N
Where, p = percent correctly classified, N = sample size, and g represents the proportion of observations actually observed in the largest group (. in the example above and . if group sizes are equal, for instance). The pattern of discriminant weights or discriminant loadings determines which variables most predict or explain group membership (i.e., via the Z-values). Discriminant loadings are sometimes called structure correlations and they are directly analogous to factor loadings in the multivariate technique known as factor analysis (see 7Factor Analysis and Latent Variable Modeling). They represent the correlation between an observation and the
D
discriminant function (i.e., variate). Thus, the relative magnitude of a loading shows how influential a variable is to the grouping variable. If causal terms are appropriate, it would show relatively speaking, how much a variable caused observations to be in one group or another. The number of discriminant factors that can be recovered is equal to one less than the number of groups so that for a dichotomous dependent variable, only one discriminant function, or factor, is possible. In the three group case two factors are possible. Thus, two key results are () how accurately a discriminant function can classify observations and () which variables are most responsible for the classifying. As the cross-classification suggests that the model predicts membership at better than a chance rate, the discriminant function has validity. As loadings are greater in magnitude, they are predictive of group membership. In a very real way, the mathematics of discriminant analysis mirrors that of MANOVA, another multivariate technique. Both MANOVA and discriminant analysis will each produce a Wilkes Λ that indicates whether or not the observed group membership accounts for differences in the means of the independent variables. Discriminant validity is often used in combination with other techniques like cluster analysis (see 7Cluster Analysis: An Introduction) and also is conceptually similar to techniques like 7logistic regression that also predict categorical dependent variables. Here, a simple illustration of discriminant analysis is described. The observations represent records of academic journal submissions forming a sample of the larger population of submissions. The dependent variable is whether or not the submission was rejected. Thus, there are two groups of observations. One hundred and eighteen papers were rejected and were not rejected. Group membership constitutes the dependent variable and was coded = Rejected and = Not Rejected. The independent variables represent the paper’s review scores based on: (a) (b) (c) (d) (e) (f)
topical relevance, theoretical development, research approach, analytical competence, writing quality, contribution potential.
Each independent variable was scored on a point scale with higher scores being better. Here is a brief overview of the results. The Wilkes’ Λ is . which suggests the means for the independent variables differ significantly based on group membership (p < .). The resulting cross-classification matrix is shown below:
D
D
Discriminant Analysis: Issues and Problems
Predicted Group Membership Actual Group Membership Rejected Not Rejected Rejected Not Rejected
of observations, or . percent, are correctly classified. This can be compared to the percent of observations that would be correctly classified if all observations were predicted to be rejected. The discriminant analysis improves predictability significantly over chance (p < .). Additionally, the discriminant loadings are as follows:
Variable
Function
Theoretical development
.
Research methods
.
Writing quality
.
Analytical competence
.
Contribution potential
.
Topical relevance
.
Thus, theoretical development is the most influential independent variable helping explain why papers get rejected from this particular journal. In contrast, topical relevance does very little to distinguish papers that are rejected from others. Perhaps the reason for this is that most submissions receive similar high scores for relevance. Practically, the results suggest that stronger theory is a route to avoiding rejection. As you can see, discriminant analysis can be a very useful tool for explaining why observations end up in one group or another (Hair et al. ). Some applications of discriminant analysis include a study predicting cohort group membership as the dependent variable with personal value scores as a independent variables (Noble and Schewe ), a study trying to explain differences between small firms and firms that grow into large firms (Culpan ), and a study explaining groups of consumers based on how they respond to poor performance in a retail setting. This idea involves explaining why someone might be an irate consumer, an active consumer or a passive consumer, for instance, using a variety of demographic, lifestyle and situational independent variables (Singh ).
About the Author Professor Babin is Past President, The Academy of Marketing Science (–) and Past President of The Society for Marketing Advances (–). He is a Distinguished Fellow of the Academy of Marketing Science and the Society for Marketing Advances. He was awarded the Omerre DeSerres Award for Outstanding Research in Retailing. Dr. Babin is a co-author (with J Hair, R. Anderson, R. Tatham, and W. Black) of the well known text Multivariate Data Analysis (Prentice Hall).
Cross References 7Canonical Correlation Analysis 7Discriminant Analysis: Issues and Problems 7Multivariate Analysis of Variance (MANOVA) 7Multivariate Data Analysis: An Overview 7Multivariate Statistical Analysis 7Statistics: An Overview
References and Further Reading Culpan R () Export behavior of firms: relevance of firm size. J Bus Res :– (March) Hair J, Black WC, Babin BJ, Anderson R () Multivariate data analysis, th edn. Prentice Hall, Upper Saddle River, NJ Noble SM, Schewe CD () Cohort segmentation: an exploration of its validity. J Bus Res :– (December) Singh J () A typology of consumer dissatisfaction styles. J Retail :– (Spring)
Discriminant Analysis: Issues and Problems Carl J. Huberty Professor Emeritus University of Georgia, Athens, GA, USA
Introduction It was in the mid s when Sir Ronald A. Fisher (– ) formally introduced the notion of “discriminant analysis” (DA) in writing. His introduction involved prediction of group membership in a two-group context – a predictive discriminant analysis (PDA). The notion of “discriminant analysis” became of interest to researchers in various areas of study in the s and s (e.g., Cooley and Lohnes ). That is when the variant which may be termed “descriptive discriminant analysis” (DDA) “caught on.”
PDA Versus DDA The mixing of PDA and DDA is fairly common in many books and journal articles. The distinction of PDA from
Discriminant Analysis: Issues and Problems
D
Discriminant Analysis: Issues and Problems. Table PDA versus DDA PDA
DDA
Prediction of group membership
Description of group separation
Predictor(s)
Response variables
Grouping variable(s)
Criterion(ia)
Grouping variable
Response variables
Response variable set
Hodgepodge
System
Response variable composite
LCF/QCF
LDF
Number of composites
k
min(p, k − )
Equality of covariance matrices
Yes
Yes
MANOVA
No
Yes
Variable construct(s)
No
Yes(!)
Response variable deletion
Yes (!)
Maybe
Response variable ordering
Yes
Yes
Criterion for variable deletion/ordering
Classification accuracy
Group separation
Research purpose
Practical/theoretical
Theoretical
Research context Variable roles
Preliminary analysis concerns
Analysis aspects of typical concern
DDA is, to the current writer, fairly important for description, interpretation, and reporting purposes. A PDA is used to determine a “rule” for predicting membership in one of k groups of analysis units based on measures on p predictor variables. The rule, then, consists of k composites of the p predictors. These composites may be of linear form (if the k p × p covariance matrices are judged to be approximately equal) or of quadratic form (if the k covariance matrices are clearly unequal). For the former, one has k linear classification functions (LCFs). For the latter, one has k quadratic classification functions (QCFs). For details, see Huberty and Olejnik (, Chap. ). A DDA is considered in the context of comparing k group mean vectors of the p outcome variables. A DDA would be applicable when a 7multivariate analysis of variance (MANOVA) is conducted and the k mean vectors of the p outcome variables are judged to be unequal. [In a MANOVA context, it is assumed that the k covariance matrices are “in the same ballpark.” See Huberty and Olejnik (, Chap. ).] One obtains a set of min(p, k − ) linear discriminant functions (LDFs). These linear composites comprise the essence of a DDA. [Such composites are analogous to “factors” in a factor analysis.] An issue/problem in DA is the lack of distinction between PDA and DDA. This lack results in the misuse
of terms, and some misinterpretations of computer output. [The computer output often leads to misuse of terms.] See Table for a summary of PDA versus DDA. In sum, it is classification (PDA) versus separation (DDA).
Terms It should be clear from the stated purpose of an empirical study involving k ≥ groups of analysis units and p ≥ attributes/variables pertaining to the units, whether one is interested in group membership prediction or in group comparison description. For the former analysis (PDA), one would be interested in prediction, and, therefore, related term use should be utilized. Similarly for applications of DDA. Issues in term use in PDA and in DDA are reviewed by Huberty and Olejnik (, Chap. ). More specifically, proper use of terms in reporting the use, and results, of a PDA and DDA is given by Huberty and Olejnik (, Chaps. and ). Also, reliance on terms found in computer output is not strongly recommended.
More Issues and Problems Five issues in PDA are discussed by Huberty and Olejnik (, Chap. ): () linear versus quadratic classification
D
D
Dispersion Models
rules; () nonnormal classification rules; () prior probabilities; () misclassification costs; and () hit-rate estimation. Also discussed in that chapter are three issues in DDA: () stepwise analyses (yuk!); () standardized LDF weights versus structure r’s; and () data-based structure. All of these issues involve the use of researcher judgment – of course, consultation with respected “experts” would help in making some judgment calls. There are, also, some additional problems in the use of both PDA and DDA – see Huberty and Olejnik (, Chap. ). Three of these problems involve missing data, outliers, and misclassification costs. A detailed discussion of these problems will not be presented herein. With regard to the first two problems, a recommendation is to “look at your data” (via graphs).
About the Author Dr. C. J. Huberty is a Professor Emeritus, Department of Research, Measurement, and Statistics, University of Georgia, USA. He served UGA from to , and was Department Head (–). He was given the College of Education Faculty Research Award in , and the Career Center Award in . He was also given recognition by being nominated for national academic awards. In , he was recognized as a Fulbright Senior Specialist. He was an ASA member for years. He authored two books: Applied Discriminant Analysis (Wiley ) and Applied MANOVA and Discriminant Analysis (Wiley, , with Stephen Olejnik). He was the sole/senior author of invited book chapters, journal articles, book reviews, and papers presented at professional meetings. He conducted workshops on multivariate statistics at professional meetings in Australia, Belgium, Egypt, and USA. He served as a statistical consultant for grants. Dr. Huberty served on journal editorial boards, including the founding board of the Journal of Educational Statistics. Finally, he sponsored academic visitors from Canada (), Egypt (), Iran (), Italy (), Japan (), South Korea (), St. Lucia (), and Sweden ().
Cross References 7Canonical Correlation Analysis 7Discriminant Analysis: An Overview 7Multivariate Analysis of Variance (MANOVA) 7Multivariate Data Analysis: An Overview 7Multivariate Statistical Analysis
References and Further Reading Cooley WW, Lohnes PR () Multivariate procedures for the behavioral sciences. Wiley, New York
Huberty CJ () Discriminant analysis. In: Meij J (ed) Dealing with the data flood. Study Centre for Technology Trends, The Netherlands, pp – Huberty CJ () Discriminant analysis. In: Everitt BS, Howell DC (eds) Encyclopedia of statistics in behavioral science, vol . Wiley, Chichester, pp – Huberty CJ, Hussein MH () Some problems in reporting use of discriminant analyses. J Exp Educ :– Huberty CJ, Olejnik S () Applied MANOVA and discriminant analysis. Wiley, Hoboken
Dispersion Models Bent Jørgensen Professor University of Southern Denmark, Odense M, Denmark
Introduction A dispersion model, denoted Y ∼ DM(µ, σ ), is a twoparameter family of distributions with probability density functions on R of the form () f (y; µ, σ ) = a(y; σ ) exp {− d(y; µ)}. σ Here µ and σ are real parameters with domain (µ, σ ) ∈ Ω × R+ (Ω being an interval), called the position and dispersion parameters, respectively. Also a and d are suitable functions such that () is a probability density function for all parameter values. In particular, d is assumed to be a unit deviance, satisfying d(µ; µ) = for µ ∈ Ω and d(y; µ) > for y ≠ µ. Dispersion models were introduced by Sweeting () and Jørgensen (; b) who extended the analysis of deviance for 7generalized linear models in the sense of Nelder and Wedderburn () to non-linear regression models with error distribution DM(µ, σ ). In many cases, the unit deviance d is regular, meaning that it is twice continuously differentiable and ∂ d/∂µ (µ; µ) > for µ ∈ Ω. We then define the unit variance function for µ ∈ Ω by V(µ) =
∂ d (µ; µ) ∂ µ
.
()
For example, the normal distribution N(µ, σ ) corresponds to the unit deviance d(y; µ) = (y − µ) on R , with unit variance function V(µ) ≡ and a(y; σ ) = −/ (πσ ) . Note that for < ρ < the unit deviance d(y; µ) = ∣y − µ∣ ρ , with a(y; σ ) constant (not depending on y) provides an example of a dispersion model where d is not regular.
Dispersion Models
The renormalized saddlepoint approximation for a dispersion model with regular unit deviance d is defined for y ∈ Ω by f (y; µ, σ ) ∼ a (µ, σ )V − (y) exp {−
d(y; µ)}, () σ
where a (µ, σ ) is a normalizing constant, making the right-hand side of () a probability density function on Ω with respect to Lebesgue measure. The ordinary saddlepoint approximation is obtained by the asymptotic approx−/ imation a (µ, σ ) ∼ (πσ ) as σ ↓ , and correspondingly, the distribution DM(µ, σ ) is asymptotically normal N {µ, σ V(µ)} for σ small, so that µ and σ V(µ) are the asymptotic mean and variance of Y, respectively. In this way, dispersion models have properties that resemble those of the normal distribution. In particular the unit deviance d(y; µ) is often a measure of squared distance between y and µ. There are two main classes of dispersion models, namely proper dispersion models (PD) and reproductive exponential dispersion models (ED), as defined below. The class of additive exponential dispersion models (ED∗ ) is not of the dispersion model form (), but is closely related to reproductive exponential dispersion models. These three types of models cover a comprehensive range of nonnormal distributions, and as shown in Table they include many standard statistical families as special cases. The exponential and Poisson families are examples of natural exponential families, which correspond to exponential dispersion models with σ = .
Proper Dispersion Models Jørgensen (a) proposed a special case of (), called a proper dispersion model, which corresponds to a density of the form f (y; µ, σ ) = a(σ )V − (y) exp {−
d(y; µ)}, σ
()
where d is a given regular unit deviance with unit variance function V(µ). We denote this model by Y ∼ PD(µ, σ ). Note that the unit deviance d characterizes the proper dispersion model (), because the unit variance function () is a function of d, while a(σ ), in turn, is a normalizing constant. It also follows that the normalizing constant of () is a (µ, σ ) = a(σ ), making the the two sides of () identical in this case. Conditions under which a given unit deviance gives rise to a proper dispersion model, and the connection with exactness of Barndorff-Nielsen’s p∗ -formula, are discussed by Jørgensen (b, Chap. ). Many proper dispersion models are transformation models when σ is known. For example, unit deviances
D
of the form d(y; µ) = h(y − µ) on Ω = R, for a suitable function h, give rise to so-called location-dispersion models, where the unit variance function is constant. A specific example is given by the unit deviance d(y; µ) = log { + (y − µ) } , which corresponds to the Student t location family. Another important case corresponds to unit deviances of the form d(y; µ) = h(y/µ) on Ω = R+ , which give rise to so-called scale-dispersion models. For example, Jørgensen (b, Chap. ) showed that the generalized 7inverse Gaussian distribution is a family of scale-dispersion models with unit deviances µ y µ d β (y; µ) = β log + ( + β) + ( − β) − , () y µ y for y, µ > , where β ∈ [−, ] is an additional parameter. In particular, the values β = ± correspond to the gamma and reciprocal gamma families, respectively, and β = to the one-dimensional hyperboloid distribution. The von Mises distribution, corresponding to the unit deviance d(y; µ) = { − cos (y − µ)} on Ω = (, π) is also a transformation model when σ is known, corresponding to the group of additions modulo π. The class of simplex distributions of Barndorff-Nielsen () contains several proper dispersion models as special cases. The simplest example is given by the probability density function −/ f (y; µ, σ ) = √ {y( − y)} σ π (y − µ) }, () × exp {− σ y( − y)µ ( − µ) where y, µ ∈ (, ) and σ > . The transformed Leipnik distribution, defined by the unit deviance d(y; µ) = log { +
(y − µ) } for y, µ ∈ (, ) y( − y)
provides a further examples of proper dispersion models. This and the simplex distribution are examples of proper dispersion models that are not transformation models when σ is known.
Exponential Dispersion Models Exponential dispersion models are two-parameter extensions of natural exponential families. A natural exponential family has densities of the form f (x; θ) = c(x) exp {θx − κ(θ)},
()
with respect to a suitable measure ν, usually Lebesgue or counting measure. Here c(x) is a given function and κ(θ) = log ∫ c(x)eθx ν(dx) is the corresponding cumulant function. The canonical parameter θ has domain Θ; the
D
D
Dispersion Models
Dispersion Models. Table Main examples of univariate dispersion models Data type
Support
Examples
Type
Real data
R
Normal, generalized hyperbolic secant
ED
Positive data
R+
Exponential, gamma, inverse Gaussian
ED
Positive data with zeros
R = [, ∞)
Compound Poisson distribution
ED
Proportions
(, )
Simplex, Leipnik distributions
PD
Directional data
[, π)
von Mises distribution
PD
Count data
N = {, , , . . . }
Poisson, negative binomial distributions
ED∗
Binomial count data
{, , . . . , m}
Binomial distribution
ED∗
largest interval for which κ(θ) is finite. The mean of a random variable X with distribution () is µ = τ(θ), where τ(θ) = κ ′ (θ) (a one-to-one function of θ), and the domain for µ is Ω = τ(intΘ), where intΘ denotes the interior of the domain Θ. If necessary, we define the value of µ by continuity at boundary points of Θ. Examples of natural exponential families include the exponential, geometric, Poisson, and logarithmic families. The variance of X is V(µ) = τ ′ {τ − (µ)}, where V is known as the variance function of the family (). This function characterizes () among all natural exponential families, see for example Morris (), and V is an important convergence and characterization tool for natural exponential families, see, e.g., Jørgensen (b, Chap. ). An additive exponential dispersion model is a twoparameter extension of natural exponential families with densities of the form f ∗ (x; θ, λ) = c∗ (x; λ) exp {θx − λκ(θ)},
()
where the index parameter λ has domain Λ consisting of those values of λ > for which λκ(θ) is a cumulant function of some function c∗ (x; λ). The mean and variance of a random variable X with distribution () are λµ and λV(µ), respectively. Many discrete distributions are additive exponential dispersion models, for example the binomial and negative binomial families. An additive exponential dispersion model is denoted X ∼ ED∗ (µ, λ). A reproductive exponential dispersion model is defined by applying the transformation Y = X/λ to (), known as the duality transformation. The reproductive form is denoted Y ∼ ED(µ, σ ), corresponding to densities of the form f (y; θ, λ) = c(y; λ) exp[λ{θy − κ(θ)}]
()
for a suitable function c(y; λ). Here µ (with domain Ω) is the mean of Y, σ = /λ is the dispersion parameter, and the variance of Y is σ V(µ). Reproductive exponential dispersion models were proposed by Tweedie (), and again by Nelder and Wedderburn () as the class of error distributions for generalized linear models. The inverse duality transformation X = λY takes () back into the form () so that each exponential dispersion model has, in principle, an additive as well as a reproductive form. Both () and () are natural exponential families when the index parameter λ is known. To see the connection with dispersion models as defined by (), we introduce the unit deviance corresponding to (), d(y; µ) = [sup {θy − κ(θ)} − yτ − (µ) + κ {τ − (µ)}], θ
() which is a Kullback-Leibler distance, see Hastie (). Defining a(y; σ ) = c(y; σ − ) exp [sup {θy − κ(θ)}], θ
we may write the density () in the dispersion model form (). Consequently, reproductive exponential dispersion models form a sub-class of dispersion models, whereas additive exponential dispersion models are not in general of the form (). The overlap between exponential and proper dispersion models is small; in fact the normal, gamma and inverse Gaussian families are the only examples of exponential dispersion models that are also proper dispersion models. An additive exponential dispersion model satisfies a convolution formula, defined as follows. If X , . . . , Xn are independent random variables, and Xi ∼ ED∗ (µ, λ i ) with
D
Dispersion Models
λ i ∈ Λ for i = , . . . , n, then the distribution of X+ = X + . . . + Xn is X+ ∼ ED∗ (µ, λ + . . . + λ n ),
()
where in fact λ + ⋯ + λ n ∈ Λ. This is called the additive property of an additive exponential dispersion model. An additive exponential dispersion model ED∗ (µ, λ) gives rise to a stochastic process {X(t) : t ∈ Λ ∪ {}} with stationary and independent increments, called an additive process. This process is defined by assuming that X() = , along with the following distribution of the increments: X(t + s) − X(t) ∼ ED∗ (µ, s), for s, t ∈ Λ. An additive process with time domain Λ = N (the positive integers) is a random walk, the simplest example being the Bernoulli process. An additive process with time domain Λ = R+ (which requires the distribution to be infinitely divisible), is a Lévy process (see 7Lévy Processes), including examples such as Brownian motion with drift and the Poisson process (see 7Poisson Processes). Note, in particular, that the average of the process over the interval (, t) has distribution X(t)/t ∼ ED(µ, t − ), which follows by an application of the duality transformation. A reproductive exponential dispersion model ED(µ, σ ) satisfies the following reproductive property, which follows as a corollary to (). Assume that Y , . . . , Yn are independent and Yi ∼ ED (µ, σ /wi ) for i = , . . . , n, where w , . . . , wn are positive weights such that wi /σ ∈ Λ for all i. Then, with w+ = w + . . . + wn , n
Dispersion Models. Table Summary of Tweedie exponential dispersion models Distribution family
p
Support Ω
Extreme stable exponential family
p<
R
R+ R
Normal distribution
p=
R
R
Poisson distribution
p=
N
R+ R
Compound Poisson distribution
< p < R
R+ R−
Gamma distribution
p=
R+
R+ R−
Positive stable exponential family
p>
R+
R+ −R
Inverse Gaussian distribution
p=
R+
R+ −R
Extreme stable exponential family
p=∞
R
R
Θ
R
R−
Notation: −R = (−∞, ]
family Twp (µ, σ ) is given by dp (y; µ) = [
{max(y, )}−p yµ −p µ −p − + ], ( − p)( − p) −p −p
where y belongs to the support, see Table . The Tweedie models are characterized by the following scaling property: b Twp (µ, σ ) ∼ Twp (bµ, b−p σ ).
()
σ ). ∑ wi Yi ∼ ED (µ, w+ i= w+
()
Tweedie Exponential Dispersion Models The class of Tweedie models, denoted Twp (µ, σ ), consist of reproductive exponential dispersion models corresponding to the unit variance functions V(µ) = µ p , where p is a parameter with domain (−∞, ] ∪ [, ∞] and Ω is defined in Table . Here we let p = ∞ correspond to the variance function V(µ) = e β µ for some β ∈ R. These models were introduced independently by Tweedie (), Morris (), Hougaard () and Bar-Lev and Enis (). As shown in Table , the Tweedie class contains several wellknown families of distributions. For p < or p > , the Tweedie models are natural exponential families generated by extreme or positive stable distributions with index α = (p − )/(p − ). For p ≠ , , , the unit deviance of the
for all b > , see Jørgensen et al. () and Jørgensen (b, Chap. ). The former authors showed that the scaling property implies a kind of central limit theorem (see 7Central Limit Theorems) for exponential dispersion models, where Tweedie models appear as limiting distributions. For < p < , the Tweedie models are compound Poisson distributions, which are continuous non-negative distributions with an atom at zero. Such models are useful for measurements of for example precipitation, where wet periods show positive amounts, while dry periods are recorded as zeros. Similarly, the total claim on an insurance policy over a fixed time interval (Renshaw ; Jørgensen and Souza ), may be either positive if claims were made in the period or zero if no claims were made. Other values of p (except p = ) correspond to continuous distributions with support either R+ or R.
D
D
Dispersion Models
Let us apply the inverse duality transformation X = λY to the Tweedie variable Y ∼ Twp (µ, /λ), in order to obtain the additive form of the Tweedie distribution. By the scaling property () this gives X ∼ Twp (λµ, λ −p ), which shows that the Tweedie exponential dispersion models have an additive as well as a reproductive form. The additive form of the Tweedie model gives rise to a class of additive processes {X(t) : t ≥ }, defined by the following distribution of the increments: X(t + s) − X(t) ∼ Twp (sµ, s−p ) , for s, t > . This class of Hougaard Lévy processes was introduced by Jørgensen () and Lee and Whitmore (). Brownian motion (see 7Brownian Motion and Diffusions) with drift (p = ) and the Poisson process (p = ) mentioned above are examples of Hougaard processes, and other familiar examples include the gamma process (p = ) and the inverse Gaussian process (p = ). The Hougaard processes corresponding to < p < are compound Poisson processes with gamma distributed jumps.
Multivariate Dispersion Models One way to generalize () to the multivariate case is to consider a probability density function of the form d(y; µ)} for y ∈ Rk , σ () where (µ, σ ) ∈ Ω × R+ , and Ω is now a domain in Rk . Here a is a suitable function such that () is a probability density function, and d is again a unit deviance satisfying d(µ; µ) = for µ ∈ Ω and d(y; µ) > for y ≠ µ. The multivariate exponential dispersion models of Jørgensen (; a) provide examples of (). Other examples include the multivariate von Mises-Fisher distribution, the multivariate hyperboloid distribution (see Jensen ), and some special cases of the multivariate simplex distributions of Barndorff-Nielsen (). A more flexible definition of multivariate dispersion models is obtained by considering models of the form f (y; µ, Σ) = a(y; Σ) exp [− g {r⊺ (y; µ)Σ− r(y; µ)}] , () where Σ is a symmetric positive-definite k × k matrix, g is an increasing function satisfying g() = and g ′ () > , and r(y; µ) is a suitably defined k-vector of residuals satisfying r(µ; µ) = for µ ∈ Ω, see Jørgensen and Lauritzen (). An example is the 7multivariate normal distributions Nk (µ, Σ), which is obtained for r(y; µ) = y − µ and g the identity function. Further examples include the multivariate Laplace and t distributions, along with the class of f (y; µ, σ ) = a(y; σ ) exp {−
elliptically contoured distributions of Fang (). Further generalizations and examples may be found in Jørgensen and Lauritzen (), see also Jørgensen and Rajeswaran ().
About the Author Bent Jørgensen is Professor of Mathematical Statistics at the University of Southern Denmark, Odense. He has also held positions at the Institute of Pure and Applied Mathematics, Rio de Janeiro, and the University of British Columbia, Vancouver. He is an Elected member of the International Statistical Institute. Professor Jørgensen has authored or co-authored more than papers and three books, including The Theory of Dispersion Models (Chapman & Hall, ).
Cross References 7Exponential Family Models 7Generalized Linear Models 7Saddlepoint Approximations
References and Further Reading Bar-Lev SK, Enis P () Reproducibility and natural exponential families with power variance functions. Ann Stat :– Barndorff-Nielsen OE, Jørgensen B () Some parametric models on the simplex. J Multivar Anal :– Fang K-T () Elliptically contoured distributions. In: Kotz S, Read CB, Banks DL (eds) Encyclopedia of statistical sciences, update vol . Wiley, New York, pp – Hastie T () A closer look at the deviance. Am Stat :– Hougaard P () Survival models for heterogeneous populations derived from stable distributions. Biometrika :– Jensen JL () On the hyperboloid distribution. Scand J Stat : – Jørgensen B () Maximum likelihood estimation and largesample inference for generalized linear and nonlinear regression models. Biometrika :– Jørgensen B () Some properties of exponential dispersion models. Scand J Stat :– Jørgensen B (a) Exponential dispersion models (with discussion). J R Stat Soc Ser B :– Jørgensen B (b) Small-dispersion asymptotics. Braz J Probab Stat :– Jørgensen B () Exponential dispersion models and extensions: a review. Int Stat Rev :– Jørgensen B (a) Proper dispersion models (with discussion). Braz J Probab Stat :– Jørgensen B (b) The theory of dispersion models. Chapman & Hall, London Jørgensen B, Lauritzen SL () Multivariate dispersion models. J Multivar Anal :– Jørgensen B, Rajeswaran J () A generalization of Hotelling’s T. Commun Stat – Theory and Methods :– Jørgensen B, Souza MP () Fitting Tweedie’scompound Poisson model to insurance claims data. Scand Actuarial J – Jørgensen B, Martínez JR, Tsao M () Asymptotic behaviour of the variance function. Scand J Statist :–
Distance Measures
Lee M-LT, Whitmore GA () Stochastic processes directed by randomized time. J Appl Probab :– Morris CN () Models for positive data with good convolution properties. Memo no. . Rand Corporation, California Morris, CN () Natural exponential families with quadratic variance functions. Ann Stat :– Nelder JA, Wedderburn RWM () Generalized linear models. J R Stat Soc Ser A :– Renshaw AE () An application of exponential dispersion models in premium rating. Astin Bull :– Sweeting TJ () Scale parameters: a Bayesian treatment. J R Stat Soc Ser B :– Tweedie MCK () Functions of a statistical variate with given means, with special reference to Laplacian distributions. Proc Camb Phil Soc :– Tweedie MCK () An index which distinguishes between some important exponential families. In: Ghosh JK, Roy J (eds) Statistics: applications and new directions. Proceedings of the Indian Statistical Institute Golden Jubilee International Conference. Indian Statistical Institute, Calcutta, pp –
Distance Measures Bojana Dalbelo Basic Professor, Faculty of Electrical Engineering and Computing University of Zagreb, Zagreb, Croatia Distance is a key concept in many statistical and pattern recognition methods (e.g., clustering, 7multidimensional scaling, 7correspondence analysis, 7principal component analysis, k-nearest neighbors, etc.). From the mathematical point of view, any set X (whose elements are called points) is said to be a metric space (Rudin ) if for any two points a, b ∈ X there is an associated real number d(a, b) called distance if the following properties hold true: . . . .
d(a, b) ≥ , (non-negativity) d(a, b) = ⇐⇒ a = b (definiteness) d(a, b) = d(b, a) (symmetry) d(a, c) ≤ d(a, b) + d(b, c), for any b ∈ X (triangle inequality)
Any function with these properties is called a distance function, a distance measure, or a metric. An example of the distance measure in metric space Rn is the so-called Minkowski distance (Ln ) defined by: ¿ n Á n À dn (a, b) = Á ∑(ai − bi )n , i=
where a and b are vectors, elements of Rn .
D
The Minkowski metric for n = is called the Manhattan or the Cityblock distance (L ) and is given by: n
d (a, b) = ∑ ∣ai − bi ∣. i=
For binary vectors (whose components are restricted to values or ) this metric is called the Hamming distance. A special case of the Minkowski metric for the n = metric is called the Euclidian distance (L ) and it is the most popular metric: ¿ Án À∑(ai − bi ) . d (a, b) = Á i=
For n → ∞, we have the distance (L∞ ) defined by: d∞ (a, b) = max≤i≤n {∣ai − bi ∣}. As an example, consider a set of points x in R that have a constant distance r from the origin, that is, (, ) point in R . Then, a set of points x ∈ R having the property d (, x) = r is a circle with radius r and origin (, ); a set of points x ∈ R having d (, x) = r is a “diamond” having vertices at (r, ), (, r), (−r, ) and (, −r); and a set of points x ∈ R having the property d∞ (, x) = r is a square having vertices at (r, r), (−r, r), (−r, −r), (r, −r). Although the Euclidean distance is the most widely used, it is not an appropriate measure when the statistical properties of variables (attributes of the items) are being explicitly considered because it assumes that the variables are uncorrelated. Also, if we measure the distance between two items, the Euclidean distance could change with a change in the scale of different variables. For that purpose, the statistical distance is used, since it is not affected by a change in scale. An example of such scale-invariant distance measures is the squared (weighted) Euclidean distance for standardized data and the Mahalanobis distance. The squared Euclidean distance for standardized data is weighted by /si , where si is the standard deviation of the ith variable: n
d (a, b) = ∑ i=
(ai − bi ) . si
This distance is also called the Karl Pearson distance. Another example of a weighted Euclidean distance is the chi-squared distance used in the 7correspondence analysis. A very important statistical measure that is scale invariant is the Mahalanobis distance (Mardia and Kent ), defined by: dMahalanobis (a, b) = (a − b)′ Σ− (a − b),
D
D
Distance Sampling
where a and b are two multivariate observations, Σ− is the inverse of the variance-covariance matrix and (a − b)′ is the transpose of vector (a − b). The Mahalanobis distance is designed to take into account the correlation between all variables (attributes) of the observations under consideration. For uncorrelated variables, the Mahalanobis distance reduces to the Euclidean distance for standardized data. As an example, consider a set of points x in R that have the constant distance r from the origin, that is, (, ). Then, the set of points having the property dMahalanobis (, x) = r is an ellipse. The Mahalanobis distance is a positive definite quadratic form x′ Ax, where the matrix A = Σ− . Distance measures or metrics are members of a broader concept called similarity measures (or dissimilarity measures) (Theodoridis and Koutroumbas ) that measure likeness (or affinity) in the case of the similarity measure, or difference (or lack of affinity) in the case of dissimilarity between objects. Similarity measures can be converted to dissimilarity measures using a monotone decreasing transformation and vice versa. The main difference between metrics and broader concepts of similarity/dissimilarity measures is that some of the properties ()–() do not hold for similarity/dissimilarity measures. For example, definiteness, or the triangle inequality, usually do not hold for similarity/dissimilarity measures. The cosine similarity and the Pearson’s product moment coefficient are two similarity measures that are not metric. The cosine similarity is the cosine of an angle between the vectors x and y from Rn and is given by: s(x, y) =
x′ y , ∥x∥∥y∥
where ∥x∥ and ∥y∥ are norms of the vectors x and y. This measure is very popular in information retrieval and textmining applications. In statistical analysis (especially when applied to ecology, natural language processing, social sciences, etc.) there are often cases in which similarity or the distance between two items (e.g., sets, binary vectors) is based on two-way contingency tables with elements a, b, c, and d, where a represents the number of elements (attribute values, variables values) present in both items, b is the number of elements present in the first but absent in the second item, c is the number of elements present in the second but absent in the first item, and d is number of elements absent simultaneously in both items. The numbers a, b, c, and d can be defined as properties of two sets or two binary vectors.
Similarity coefficients (Theodoridis and Koutroumbas ) or associations measures can be defined as a combination of numbers a, b, c, and d. Examples of associations measures are: Simple matching coefficient Dice coefficient Jaccard (or Tanimoto) coefficient
(a + d)/n, a/(a + b + c), a/(a + b + c).
Although association measures, similarity measures, and correlation coefficients are not metric, they are applicable in the analysis where they are consistent with the objective of the study and where they have meaningful interpretation (Sharma ).
Cross References 7Cook’s Distance 7Data Mining Time Series Data 7Entropy and Cross Entropy as Diversity and Distance
Measures 7Multidimensional Scaling: An Introduction 7Multivariate Outliers 7Statistical Natural Language Processing 7Statistics on Ranked Lists
References and Further Reading Mardia KV, Kent JT, Bibby JM () Multivariate analysis. Academic, New York Rudin W () Principles of mathematical analysis, rd edn. McGraw Hill, New York Sharma SC () Applied multivariate techniques. Wiley, New York Theodoridis S, Koutroumbas K () Pattern recognition, th edn. Elsevier
Distance Sampling Tiago A. Marques, , Stephen T. Buckland , David L. Borchers , Eric A. Rexstad , Len Thomas University of St Andrews, St Andrews, UK Faculdade de Ciências da Universidade de Lisboa, Lisboa, Portugal
Distance sampling is a widely used methodology for estimating animal density or abundance. Its name derives from the fact that the information used for inference are the recorded distances to objects of interest, usually animals, obtained by surveying lines or points. The methods
Distance Sampling
are also particularly suited to plants or immotile objects, as the assumptions involved (see below for details) are more easily met. In the case of lines the perpendicular distances to detected animals are recorded, while in the case of points the radial distances from the point to detected animals are recorded. A key underlying concept is the detection function, usually denoted g(y) (here y represents either a perpendicular distance from the line or a radial distance from the point). This represents the probability of detecting an animal of interest, given that it is at a distance y from the transect. This function is closely related to the probability density function (pdf) of the detected distances, f (y), as g(y)π(y) , () f (y) = w ∫ g(y)π(y)dy where π(y) is the distribution of distances available for detection and w is a truncation distance, beyond which distances are not considered in the analysis. The above pdf provides the basis of a likelihood from which the parameters of the detection function can be estimated. An important and often overlooked consideration is that π(y) is assumed known. This is enforced by design, as the random placement of transects, independently of the animal population, leads to a distribution which is uniform in the case of line transects and triangular in the case of point transects (Buckland et al. ). Given the n distances to detected animals, density can be estimated by ˆ ˆ = nf () () D L in the case of line transects with total transect length L, where f () is the estimated pdf evaluated at zero distance, and by ˆ ˆ = nh() () D kπ in the case of k point transects, where h() is the slope of the estimated pdf evaluated at zero distance (Buckland et al. ). This is a useful result because we can then use all the statistical tools that are available to estimate a pdf in order to obtain density estimates. So one can consider plausible candidate models for the detection function and then use standard maximum likelihood to obtain estimates for the corresponding parameters and therefore density estimates. The most common software to analyze distance sampling data, Distance (Thomas et al. ), uses the semi-parametric key+series adjustment formulation from Buckland (), in which a number of parametric models are considered as a first approximation and then some expansion series terms are added to improve the fit to the
D
data. Standard model selection tools and goodness-of-fit tests are available for assisting in 7model selection. Variance estimates can be obtained using a delta method approximation to combine the individual variances of the random components in the formulas above (i.e., n and either f () or h() ; for details on obtaining each component variance, see Buckland et al. ). In some of the more complex scenarios, one must use resampling methods based on the non-parametric bootstrap, which are also available in the software. Given a sufficiently large number of transects randomly allocated independently of the population of interest, estimators are asymptotically unbiased if () all animals on the transect are detected, i.e., g() = , () sampling is an instantaneous process (typically it is enough if animal movement is slow relative to the observer movement), and () distances are measured without error. See Buckland et al. () for discussion of assumptions. Other assumptions, for example that all detections are independent events, are strictly required as the methods are based on maximum likelihood, but the methods are extraordinarily robust to their failure (Buckland ). Failure of the g() = assumption leads to underestimation of density. Violation of the movement and measurement error assumption have similar consequences. Underestimation of distances and undetected responsive movement toward the observers lead to overestimation of density, and overestimation of distances and undetected movement away from the observer lead to underestimation of density. Random movement and random measurement error usually leads to overestimation of density. Naturally the bias depends on the extent to which the assumptions are violated. Most of the current research in the field is aimed at relaxing or avoiding the need for such assumptions. As there are no free lunches in statistics, these come at the expense of more elaborate methods, additional data demands and additional assumptions. Further details about conventional distance sampling, including dealing with clustered populations, cue counting methods and field methods aspects, can be found in Buckland et al. (), while advanced methods, including the use of multiple covariates in the detection function, double platform methods for when g() < , spatial models, automated survey design, and many other specialized topics, are covered in Buckland et al. ().
About the Authors The authors have a wide experience in developing and applying methods for estimation of animal abundance. They have published collectively around one hundred papers in distance sampling methods and applications,
D
D
Distributions of Order k
and are actively involved in dissemination of distance sampling methods through international workshops and consultancy. Three of them are also co-authors and editors of the following key books on distance sampling: Introduction to Distance Sampling: Estimating Abundance of Biological Populations (Oxford University Press, ) and Advanced Distance Sampling: Estimating Abundance of Biological Populations (Oxford University Press, ).
observed that a typical element of the event {Tk = x} is an arrangement a a . . . ax +x +⋅⋅⋅+xk SS . . . S , ´¹¹ ¹ ¹ ¸¹¹ ¹ ¹ ¶
()
k
such that x of the a’s are E = F, x of the a’s are E = SF, . . . , xk of the a’s are Ek = SS . . . S F, and proceeded ´¹¹ ¹ ¹ ¸¹¹ ¹ ¹ ¶ k−
Cross References 7Statistical Ecology 7Statistical Inference in Ecology
References and Further Reading Buckland ST () Fitting density functions with polynomials. Appl Stat :– Buckland ST () Point transect surveys for songbirds: robust methodologies. The Auk :– Buckland ST, Anderson DR, Burnham KP, Laake JL, Borchers DL, Thomas L () Introduction to distance sampling: estimating abundance of biological populations. Oxford University Press, Oxford Buckland ST, Anderson DR, Burnham KP, Laake JL, Borchers DL, Thomas L () Advanced distance sampling. Oxford University Press, Oxford Thomas L, Buckland ST, Rexstad R, Laake L, Strindberg S, Hedley S, Bishop J, Marques TA, Burnham KP () Distace software: design and analysis of distance sampling surveys for estimating population size. J App Ecol :–
Distributions of Order k Andreas N. Philippou , Demetrios L. Antzoulakos Professor of Probability and Statistics University of Patras, Patras, Greece Associate Professor University of Piraeus, Piraeus, Greece
The distributions of order k are infinite families of probability distributions indexed by a positive integer k, which reduce to the respective classical probability distributions for k = , and they have many applications. We presently discuss briefly the geometric, negative binomial, Poisson, logarithmic series and binomial distributions of order k.
Geometric Distribution of Order k Denote by Tk the number of independent Bernoulli trials with success (S) and failure (F) probabilities p and q = − p ( < p < ), respectively, until the occurrence of the kth consecutive success. Philippou and Muwafi ()
to obtain the following exact formula for the probability mass function (pmf) of Tk , namely, f (x) = P(Tk = x) = px ∑ (
x + x + ⋅ ⋅ ⋅ + xk q )( ) p x , x , . . . , xk
x +x +⋅⋅⋅+x k
, x ≥ k, ()
where the summation is taken over all non-negative integers x , x , . . . , xk satisfying the condition x + x + ⋅ ⋅ ⋅ + kxk = x − k. Alternative simpler formulas have been derived. The following recurrence for example, due to Philippou and Makri (), is very efficient for computations f (x) = f (x − ) − qpk f (x − − k),
x > k
()
with initial conditions f (k) = pk and f (x) = qpk for k < x ≤ k. Furthermore, it shows that f (x) attains its maximum pk for x = k, followed by a plateau of height qpk for x = k + , k + , . . . , k, and decreases monotonically to for x ≥ k + . Philippou et al. () employed the transformation xi = mi ( ≤ mi ≤ k) and x = m + ∑ki= (i − )mi to show that ∑∞ x=k f (x) = (and hence f (x) is a proper pmf). They named the distribution of Tk geometric distribution of order k with parameter p and denoted it by Gk (p), since for k = it reduces to the classical geometric distribution with pmf f (x) = qx− p (x ≥ ). It follows from (), by means of the above transformation and the multinomial theorem, that the probability generating function (pgf) of Tk is given by ∞
ϕ k (w) = ∑ xw f (x) = x=k
pk wk ( − pw) , − w + qpk wk+
∣ w ∣≤ .
()
The mean and variance of Tk readily follow from its pgf and they are given by − (k + )qpk − pk+ . (qpk ) () A different derivation of () was first given by Feller (), who used the method of partial fractions on ϕ k (w) to E(Tk ) =
− pk , qpk
Var(Tk ) =
D
Distributions of Order k
() observed that the moment estimator pˆ of p is the
derive the surprisingly good approximation − pw P(Tk > x) ≃ , (k + − kw )q w x+
unique admissible root of the equation ()
where w is the unique positive root of pk wk /ϕ k (w). It is well known that the negative binomial distribution arises as the distribution of the sum of r independent rv’s distributed identically as geometric. This fact motivated the genesis of the following distribution of order k.
r(−p k ) (−p)p k
= X. In
r+(r +rX)/ X
particular, for k = , pˆ = and it is consistent for p. The fact that the Poisson and the logarithmic series distributions arise as appropriate limits of the negative binomial distribution, motivated the following.
Poisson and Logarithmic Series Distributions of Order k
Negative Binomial Distribution of Order k Let X , X , . . . , Xr be independent rv’s distributed identically as Gk (p) and set Tr,k = ∑ri= Xi . Then, relation () implies ∞
Let X be a rv distributed as NBk (r, p) and set Yr = X − rk. (a) Assume that q → and rq → λ (> ) as r → ∞. Then, for y = , , . . . , lim P(Yr = y) = e−kλ ∑
ϕ r,k (w) = ∑ xw P(Tr,k = x)
r→∞
x=rk
λ y +y +⋅⋅⋅+yk = fY (y), y !y ! . . . yk !
()
r
pk wk ( − pw) ) , =( − w + qpk wk+
∣ w ∣≤ .
()
Expanding ϕ r,k (w) in a series around , Philippou et al. () obtained the pmf of Tr,k as P(Tr,k = x) = px ∑ ( q ×( ) p
x + x + ⋅ ⋅ ⋅ + xk + r − ) x , x , . . . , xk , r −
where the summation is taken over all non-negative integers y , y , . . . , yk satisfying the condition y + y + ⋅ ⋅ ⋅ + kyk = y. The distribution of Y has been named Poisson distribution of order k with parameter λ. (b) Assume that r is positive and real. Then, for z = , , . . . ,
x +x +⋅⋅⋅+x k
,
x ≥ rk,
() lim P(Yr = z ∣ Yr ≥ ) = αpz ∑ r→
where the summation is taken over all non-negative integers x , x , . . . , xk satisfying the condition x + x + ⋅ ⋅ ⋅ + kxk = x − rk. They called the distribution of Tr,k negative binomial distribution of order k with parameter vector (r, p) and denoted it by NBk (r, p), since for k = it reduces to the negative binomial distribution with pmf )pr qx−r (x ≥ r). For r = , it reduces f (x) = (x− r− to the geometric distribution of order k. Obviously, Tr,k denotes the waiting time for the occurrence of the rth nonoverlapping success run of length k. Alternative formulas, simpler than (), have been derived by Godbole () (see also Antzoulakos and Philippou ()) and Philippou and Georghiou (). The latter authors established also the following efficient recurrence g(x) = P(Tr,k
q/p k = x) = ∑[x − rk + i(r − )] x − rk i= × pi g(x − i),
x ≥ rk + ,
()
with g(x) = for ≤ x ≤ rk − and f (rk) = prk , which recaptures () for r = . Let X , X , . . . , Xn be independent rv’s distributed identically as NBk (r, p), and set X = n ∑ni= Xi . Philippou
q ×( ) p
(z + z + ⋅ ⋅ ⋅ + zk − )! z !z ! . . . zk !
z +z +⋅⋅⋅+z k
= fZ (z),
()
where α = −/k log p, and the summation is taken over all non-negative integers z , z , . . . , zk satisfying the condition z + z + ⋅ ⋅ ⋅ + kzk = z. This result is due to Aki et al. () who extended the definition of NBk (r, p) to positive real r. They called the distribution of Z logarithmic series distribution of order k with parameter p. We end this note with a few words on the binomial distribution of order k.
Binomial Distribution of Order k Let Nn,k denote the number of non-overlapping success runs of length k in n (≥ ) independent trials with success probability p ( < p < ). The 7asymptotic normality of a normalized version of Nn,k was established by von Mises (see Feller :), where a simpler proof is presented). Its exact pmf was obtained by Hirano () and Philippou and Makri (), who named the distribution binomial distribution of order k with parameter vector (n, p). They
D
D
Diversity
found that, for x = , , . . . , [n/k],
Cross References 7Binomial Distribution 7Geometric and Negative Binomial Distributions
x + x + ⋅ ⋅ ⋅ + xk + x n )p = x) = ∑ ∑ ( x , x , . . . , xk , x i= k−
P(Nn,k
q ×( ) p
References and Further Reading
x +x +⋅⋅⋅+x k
,
()
where [u] denotes the greatest integer in u and the inner summation is taken over all non-negative integers x , x , . . . , xk satisfying the condition x + x + ⋅ ⋅ ⋅ + kxk + i = n − kx. The following exact formula for the mth descending factorial moment µ ′m of Nn,k was established by Antzoulakos and Chadjiconstantinidis (), µ′m = m!pkm
min(n−km,m)
∑ j=
[
×
n−km−j ] k
∑ ( i=
m (−)j ( )pj j
m + i − m + n − k(m + i) ki )p . )( m i
()
The above results for the pmf ’s of Tk and Nn,k give exact formulas for the reliability of a consecutive-k-out-of-n:F system, which is very important in Engineering. For more on distributions of order k and their applications, we refer to Balakrishnan and Koutras ().
About the Authors Professor Andreas Philippou is a member of the Greek Statistical Authority and he has been teaching statistics and probability at the University of Patras since . He has also taught earlier at the University of Texas at El Paso and the American University of Beirut. He wrote papers, mainly in statistics and probability, and edited books on Fibonacci numbers and their applications. He served as Vice-President of Hellenic Aerospace Industry (– ), Vice-Rector of the University of Patras (–), and Minister of Education (–), Member of the House of Representatives (–) and Member of the Tax Tribunal (–) of the Republic of Cyprus. Professor Philippou is Grande Ufficiale of Italy and Honorary President of the Mathematical Association of Cyprus. He is a reviewer of more than papers and books (including papers written by Paul Erd˝os, Peter Hall, C. R. Rao, and N. Balakrishnan). Dr. Demetrios Antzoulakos is Associate Professor at the University of Piraeus where he has been teaching since . He has also taught earlier at the University of Crete. He wrote research papers mainly on the topics of statistical process control and distribution theory of runs and patterns.
Aki S, Kuboki H, Hirano K () On discrete distributions of order k. Ann Inst Stat Math :– Antzoulakos DL, Chadjiconstantinidis S () Distributions of numbers of success runs of fixed length in Markov dependent trials. Ann Inst Stat Math :– Antzoulakos DL, Philippou AN () Multivariate Pascal polynomials of order k with probability applications. In: Howard FT (ed) Applications of Fibonacci numbers, vol . Kluwer, Dordrecht, pp – Balakrishnan N, Koutras MV () Runs and scans with applications. Wiley, New York Feller W () An introduction to probability theory and its applications, vol I, rd edn. Wiley, New York Godbole AP () Specific formulae for some success run distributions. Stat Probab Lett :– Hirano K () Some properties of the distributions of order k. In: Philippou AN et al (eds) Fibonacci numbers and their applications. Reidel, Dordrecht, pp – Philippou AN () The negative binomial distribution of order k and some of its properties. Biom J :– Philippou AN, Georghiou C () Convolutions of Fibonacci-type polynomials of order k and the negative binomial distributions of order k. Fibonacci Q :– Philippou AN, Makri FS () Longest success runs and Fibonaccitype polynomials. Fibonacci Q :– Philippou AN, Makri FS () Successes, runs and longest runs. Stat Probab Lett :– Philippou AN, Muwafi AA () Waiting for the kth consecutive success and the Fibonacci sequence of order k. Fibonacci Q :– Philippou AN, Georghiou C, Philippou GN () A generalized geometric distribution and some of its properties. Stat Probab Lett :–
Diversity Tarald O. Kvålseth Professor Emeritus University of Minnesota, Minneapolis, MN, USA
Introduction Diversity is a concept that appears in various fields of study. In the most general situation involving a set of s exhaustive and mutually exclusive events with probabilities (or proportions) pi (i = , . . . , s), diversity is an attribute that depends on s and all the individual pi ’s. In particular, diversity is typically considered to increase with increasing s and with increasing evenness (uniformity) among the pi ‘s.
Diversity
For any given s, diversity is considered to be minimum for the degenerate distribution Ps and maximum for the completely even (uniform) distribution Ps defined by Ps = (, . . . , , , , . . . , ), Ps = (/s, . . . , /s).
()
As a clarification of these concepts and their measurement, consider a generic measure of diversity D that takes on the value D(Ps ) for the probability distribution Ps = (p , . . . , ps ) with D (Ps ) ≤ D (Ps ) ≤ D (Ps ) Ps
()
Ps
for the and in (). Furthermore, define the normed form of D(Ps ) as D∗ (Ps ) =
D(Ps ) − D (Ps ) D (Ps ) − D (Ps )
∈ [, ]
()
which effectively controls for s. From (), D(Ps ) = D (Ps ) + D∗ (Ps ) [D (Ps ) − D (Ps )]
()
showing that D(Ps ) is an increasing function of both the evenness (uniformity) D∗ (Ps ) and of D(/s, . . . , /s), which, for an appropriate measure D, is strictly increasing in s. Numerous measures of diversity D and evenness D∗ have been proposed. This is especially the case in biology (and ecology) where such measures are frequently used. In such applications, the concern is with a sample or a population of s different species, with pi being the probability of the event that a randomly selected specimen belongs to the ith species for i = , . . . , s. The Ps = (p , . . . , ps ) is then usually referred to as the species abundance distribution and s is often called the species richness.
Commonly Used Measures The most widely used diversity measures appear to be the following: D (Ps ) = s, D (Ps ) =
() s
− ∑ pi , i= s
D (Ps ) = − ∑ pi log pi .
() ()
i=
where both base − and base −e (natural) logarithms are being used. The D is most often referred to as the Simpson index, after Simpson (), although it is sometimes called the Gini index (e.g., Upton and Cook ). However, in some historical comments about D and − D , Good () suggested that “it is unjust to associate ρ with any one person” (where ρ = − D ). The measure of D is the Shannon’s entropy (Shannon ).
D
While D has very limited information content, both D and D can be seen to have a number of desirable properties. In particular, for both i = and i = , Di is (a) zero-indifferent (expansible), i.e., Di (p , . . . , ps , ) = Di (pi , . . . , ps ), (b) permutation symmetric in the pi ’s, (c) continuous in the pi ’s and (d) strictly Schur-concave in Ps (Marshall and Olkin , ch. ) and also strictly concave (in the usual sense). Also, with Di (Ps ) = for both i = and i = in ()–(), Di (Ps ) = D∗i (Ps )Di (Ps ) , i = ,
()
with the evenness measure D∗i (Ps ) for i = , (Magurran , ch. ) and showing that Di (Ps ) increases with both evenness and s since D (Ps ) = − /s and D (Ps ) = log s. The D in () has the advantage of having a reasonable probabilistic meaning, which helps with interpretation of its numerical values. In terms of a statistical experiment with s possible events whose probability distribution is Ps = (p , . . . , ps ), D (Ps ) is simply the probability that two independent repetitions of the experiment result in two different events occurring. In the case of a biological sample or population, D (Ps ) is the probability that two specimen, selected at random with replacement, belong to different species. By contrast, the D in () has no such convenient probability interpretation. As a measure of diversity, this 7entropy has no real or important, theoretical or practical, advantage over D . Nevertheless, and for no good reason, the D , often referred to as the Shannon index, remains a most popular diversity measure, causing some to lament that “Simpson’s index remains inexplicably less popular than the Shannon index” (Magurran : ).
Cardinal Measures Various other diversity measures and their corresponding evenness measures have been suggested (e.g., Magurran ). Some of these are cardinal measures or so-called numbers equivalent measures. Such a measure ranges in value from for the distribution Ps to s for the distribution Ps in (). For some diversity measure D, the numbers equivalent De is defined as being the number of elements in an equiprobable (completely even or uniform) distribution for which the value of D equals D(Ps ) for any given Ps = (p , . . . , ps ), i.e., D(/De (Ps ), . . . , /De (Ps )) = D(Ps )
()
where De (Ps ) is not necessarily an integer number, although this expression can only be strictly true in the integer case.
D
D
Diversity
From (), (), and (), the numbers equivalent of D and D (with base −e logarithms) are readily seen to be = − D (Ps )
D (Ps ) = De (Ps ) =
s
,
∑pi
Families of Measures Parameterized families of diversity measures have also been proposed and of which some of the above measures are special members. Hill () proposed the following inverse self-weighted arithmetic mean of order β:
i=
D (Ps ) = De (Ps ) = e
D (P s )
.
() s
It is clear from () that, for the distributions in (), D (Ps ) = D (Ps ) = and D (Ps ) = D (Ps ) = s. Also, by subtracting from D (Ps ), one gets the following measure (Kvålseth ):
D (Ps ) =
s
−
which has a statistical odds interpretation equivalent to the probability interpretation of D (Ps ): it is the odds that different events will occur during two independent repetitions of a statistical experiment. Or, in the case when two specimen are randomly selected with replacement from a biological community (sample or population), D (Ps ) is the odds that the specimen will be of different species. An interesting and rather intuitive diversity measure can be formulated by simply adjusting s for the absolute differences between the pi ’s as follows: D (Ps ) = s − ∑ ∑ ∣pi − pj ∣
()
≤i<j≤s
where, for the distribution in (), D (Ps ) = and D (Ps ) = s. Since the summation term in () can be shown to be strictly Schur-convex (Marshall and Olkin : , ), D (Ps ) is strictly Schur-concave in Ps . If the pi ’s are ordered such that p[] ≥ p[] ≥ . . . ≥ p[s] and since s
s
∑ ∑ ∣pi − pj ∣ = ∑ ∑ (p[i] − p[ j] ) = ∑ (s + − i)p[i] i= j=i+
)
, −∞ < β < ∞
()
i=
(β = α − in Hill’s notation). Patil and Taillie () proposed s
i=
≤i<j≤s
β+
()
∑pi
s−
−/β
D(β) (Ps ) = (∑ pi
i=
the D (Ps ) in () can also be expressed as
D(β) (Ps ) = ( − ∑ pi
()
From () to () and (), it is seen that D = D(−) + , D = D() , and D = D() (in the limiting sense of β → and with natural logarithm in ()). Also from () and (), D = D() and D = D() (in the limit as β → ). It can also be seen from (), () and () that D(β) is simply the numbers equivalent of D(β) .
Statistical Inferences Consider now that the pi ’s are multinomial sample estimates (and estimators) pi = ni /N for i = , . . . , s and with s
sample size N = ∑ni . One may then be interested in maki=
ing statistical inferences about the population value D(Π s ) of some diversity measure D for the corresponding population distribution Π s = (π , . . . , π s ). In particular, besides the estimate D(Ps ), one may want to construct a confidence interval for D(Π s ). When N is reasonably large and D(Π s ) is differentiable, it follows from the delta method Agresti () that D(Ps ) has approximately a normal distribution with mean D(Π s ) and estimated variance
() σˆD =
i=
The D in ()–() has been discovered and re-discovered as a potential diversity measure (Carmargo ; Kvålseth ; Patil and Taillie ). While Carmargo () proposed D (Ps )/s ∈ [/s, ] as an evenness index for which the lower bound depends on s, a preferred measure of evenness would seem from () to be
)/β, − ≤ β < ∞.
i=
s
D (Ps ) = ∑ ip[i] − .
β+
⎡ s ⎤ s ⎥ ⎢ ⎢∑ pi ϕˆDi − (∑ pi ϕˆDi ) ⎥ ⎢ ⎥ N ⎢ i= ⎥ i= ⎣ ⎦
()
where ∂D(Π s ) ϕˆDi = , i = , . . . , s ∣ ∂π i π i =pi
()
s
D∗ (Ps ) =
∑ip[i] i=
s−
∈ [, ].
()
i.e., ϕˆDi is the partial derivative of D(Π s ) with respect to π i , which is then replaced with pi , for i = , . . . , s.
D
Divisible Statistics
From ()–() and for the Di in (), (), (), (), and (), the following estimated variances are obtained: ⎡ s ⎤ s ⎥ ⎢ ⎢∑ pi − (∑ pi ) ⎥, ⎢ ⎥ N ⎢ i= ⎥ i= ⎣ ⎦ ⎡ s ⎤ s ⎥ ⎢ = ⎢ ∑ pi (log pi ) − (∑ pi log pi ) ⎥ ⎢ ⎥, N ⎢ i= ⎥ i= ⎣ ⎦ ⎡ s ⎤ s ⎢ ⎥ ⎥ ⎢ = ⎢ ∑ p i − (∑ p i ) ⎥ , s ⎢ ⎥ i= i= ⎦ N(∑pi ) ⎣
σˆD =
()
σˆD
()
σˆD
()
Magurran AE () Measuring biological diversity. Blackwell, Oxford Marshall AW, Olkin I () Inequalities: theory of majorization and its applications. Academic, San Diego Patil GP, Taillie C () Diversity as a concept and its measurement. J Am Stat Assoc :– Shannon CE () The mathematical theory of communication. Bell Syst Tech J :–, – Simpson EH () Measure of diversity. Nature : Upton G, Cook I () Oxford dictionary of statistics. Oxford University Press, Oxford
i=
σˆD
= [D (Ps )] σˆD , σˆD = σˆD ,
()
⎡ s ⎤ s ⎥ ⎢ ⎢∑ i p[i] − (∑ ip[i] ) ⎥ () ⎥ N⎢ ⎢ i= ⎥ i= ⎣ ⎦ where D in () and () and hence () and D in () are assumed to be based on natural logarithms. For example, consider the multinomial frequencies ni = , , , for which D (P ) = ., and = .. Consequently, an approximate from (), σˆD %√confidence interval for D (Π ) becomes . ± . ., or (., .). Similarly, from () and (), D (P ) = . and σˆD = . so that an approximate % confidence interval for D (Π ) is . ± √ . . or (., .). σˆD =
Divisible Statistics Estate V. Khmaladze Professor Victoria University of Wellington, Wellington, New Zealand Suppose ν in , i = , . . . , M, are frequencies of M disjoint events in a sample of size n. Suppose the probabilities of these events are pi , i = , . . . , M. The statistics of the form M
∑ g(ν in , npi )
or
i=
About the Author For biography see the entry 7Entropy.
Cross References 7Entropy 7Entropy and Cross Entropy as Diversity and Distance
Measures 7Information Theory and Statistics 7Lorenz Curve 7Measurement of Uncertainty 7Statistical View of Information Theory
are called divisible or additively divisible statistics. The total space was “divided” into disjoint events, then frequencies of these events were treated, so to say, “individually” and only then an aggregated sum was formed – hence the term. It will be more justified when we start speaking about finer and finer divisions of the space. Examples of divisible statistics are easy to find. To some extent, to attribute to them the name of divisible statistics is similar to the discovery that we speak prose, as statisticians could very well have used and studied them without paying attention or knowing that these were divisible statistics. In the following examples,
References and Further Reading Agresti A () Categorical data analysis, nd edn. Wiley, Hoboken Carmargo JA () Most dominance increase with the number of subordinate species in competitive interactions? J Theor Biol :– Good IJ () Comment to paper by Patil and Taillie. J Am Stat Assoc :– Hill MO () Diversity and evenness: a unifying notation and its consequences. Ecology :– Kvålseth TO () Note on biological diversity, evenness, and homogeneity measures. Oikos :– Kvålseth TO () On difference – based summary measures. Percept Mot Skills :–
M ν in − npi , npi ) ∑h( √ npi i=
M
XnM =∑ i=
(ν in − npi ) , npi
M
LnM = ∑ ν in ln i=
ν in , npi
M
µ n = ∑ I{ν in > }, i=
where I{A} denotes the indicator function of the event A, the reader will easily recognize the classical chi-square goodness of fit statistic, the maximum likelihood statistic for the 7multinomial distribution and what is often called an “empirical vocabulary” – the number of different events
D
D
Divisible Statistics
in a sample. Versions of this latter statistic are often used in problems of statistical diversity, cryptology and analysis of texts:
while the probability on the left side is obviously of a global nature. In certain problems limit theorems for the process of partial sums
M
µ n () = ∑ I{ν in = },
m
Zn (m) = ∑ g(ν in , npi ), m = , , . . . , M,
i=
i=
M
µ n (k) = ∑ I{ν in = k}, k = , , . . . . i=
The classical theory studies the asymptotic behavior of divisible statistics when the sample size n → ∞, but the number of different events M stays fixed. In many problems, however, a different setting is necessary: not only n → ∞, but also M → ∞ at the same time, so that pi , i = , . . . , M, form a triangular array of an increasing number of diminishing probabilities. Indeed, in statistical analysis of random number generators often no less than M = different events are identified and samples of n = – are generated. In analysis of a corpus of a language, the number of different words is often of the order M = – , while the size of texts analyzed is about n = – word-usages (see, e.g., Baayen ; Simpson et al. ). As a rule, however, one cannot access this asymptotic range by using the classical results first and then letting M tend to ∞. For example, although it is true that , n → ∞, →d χ M− XnM
and
− (M − ) χ M− →d N(, ), M → ∞, √ (M − ) where χ M− is a chi-square random variable with M − degrees of freedom and N(, ) is a standard normal random variable, for most cases it is not true that XnM − (M − ) →d N(, ) √ (M − )
if n, M → ∞ simultaneously and n/M → const. One of the frequently used tools to study the asymptotic behavior of divisible statistics is the so-called Poissonization (see, e.g., Ivchenko and Medvedev ; Morris ): if sample size N is Poisson(n) random variable then frequencies ν iN , i = , . . . , M, become independent Poisson(npi ) random variables and for Bn ⊂ R M
P {∑ g(ν in , npi ) ∈ Bn } = i=
P {∑ M i= g(ν iN , npi ) ∈ Bn , N = n} . P{N = n}
In the numerator of the right side there is now a sum of independent random variables, which is convenient. Some trouble is, however, that the probability of N = n tends to and some sort of a local limit theorem must be used,
are needed and the use of Poissonization above, again, becomes technically unpleasant. Alternative approach, demonstrated in Khmaladze (), suggests the use of an “additional” object – the filtration {Hmn }M m= , where each σ-algebra Hmn is generated by the first m frequencies ν n , . . . , ν mn . The gain here lies in the following: in functional limit theorems for the resulting semi-martingale {Zn (m), Hmn }M m= , conditional distribution of ν m+,n given Hmn plays a central role (see, e.g., Liptzer and Shiryaev ; Rebolledo ); however, this distribution is extremely simple – just binomial distribution b(⋅, n˜ m+ , p˜ m+ ) with parameters m
n˜ m+ = n − ∑ ν in
m
and
p˜ m+ = pm+ / ( − ∑ pi ) .
i=
i=
This allows to prove the functional limit theorem, that is, convergence to a Brownian motion (see 7Brownian Motion and Diffusions), for the martingale part of {Zn (m), Hmn }M m= in a simple and natural way. True that if max pi → , while n, M → ∞, the mutual dependence of multinomial frequencies ν in , i = , . . . , M, decreases, but there is an increasing number of them. Therefore the process {Zn (m), Hmn }M m= should not be expected to converge to a process with independent increments. Centered and normalized in a usual way and in time t = m/M this process converges to a Gaussian semi-martingale, W(t) + K(t),
t ∈ [, ],
where W is a Brownian motion and the compensator K, which accounts for the above mentioned dependence, is a process with differentiable trajectories. This result, and its extensions, was established in Khmaladze () under an assumption that all frequencies ν in are asymptotically Poisson random variables. A very interesting case arises when some of the frequencies can be asymptotically Gaussian, that is, some probabilities in our triangular array do not decrease sufficiently quickly. Indeed, in a typical corpus of a contemporary language, with word-count of order , some words will occur tens of thousands of times, while many others only a few hundred times, while still many more will occur only a few times.
Dummy Variables
In this mixed case even an asymptotic expression for the expected values ν in − npi E∑h( √ , npi ) npi i= M
becomes an object of research. In particular, asymptotic expressions for Eµ n (k) may reveal the presence or absence of well known limiting expressions like Zipf ’s law or Karlin–Rouault law (see, e.g., Baayen ; Karlin ; Rouault ). Also the central limit theorem (see 7Central Limit Theorems) for the mixed case is yet to be established. At the moment of writing this article some work on this was in progress (e.g., Khmaladze ; Kvizhinadze and Wu ).
About the Author For biography see the entry 7Testing Exponentiality of Distribution.
7Brownian Motion and Diffusions 7Chi-Square Tests 7Multinomial Distribution 7Poisson Distribution and Its Application in Statistics 7Random Walk 7Uniform Random Number Generators
References and Further Reading Baayen RH () Word frequency distributions. Kluwer, Dordrecht Ivchenko GI, Medvedev YuI () Decomposable statistics and hypothesis testing for grouped data. Theory Probab Appl :– Karlin S () Central limit theorems for certain infinite urn schemes. Indiana Univ Math J :– Khmaladze EV () Martingale limit theorems for divisible statistics. Theory Probab Appl :– Khmaladze EV () Diversity of responses in questionnaires and similar objects. MSOR Research Report –, VUW Kvizhinadze G, Wu H () Diversity of responses in general questionnaires, Stat Prob Lett :– Liptzer RSh, Shiryaev AN () Functional central limit theorem for semimartingales. Theory Probab Appl :– Morris C () Central limit theorem for multinomial sums. Ann Stat :– Rebolledo R () Central limiit theorem for local martingales. Z Warsch Verw Geb :– Rouault A () Loi de Zipf et sources markoviennes. Ann Inst H Poincar´e :– Simpson RC, Briggs SL, Ovens J, Swales JM () The Michigan Corpus of academic spoken English. The Regents of the University of Michigan, Ann Arbor
Dummy Variables Nataša Erjavec Professor, Faculty of Economics University of Zagreb, Zagreb, Croatia A dummy variable is a binary variable that can take only two values, and . It is often used in the regression model to incorporate qualitative (categorical) explanatory variables, such as gender (male or female), marital status (single or married), union membership (yes or no), etc. In the above examples, the dummy variable can be defined to take the value one if the observation relates to a male and zero for a female (or vice versa), one for single and zero if married, one for a union member and zero if a nonmember. The dummy variable can then be used in a standard regression model like any other regressor (Green ). As an example, if we are interested whether earnings are subject to gender discrimination, the simple regression model can be defined as Yi = α + βXi + γDi + ε i
Cross References
D
()
in which the regressand (Y) are earnings and the regressor (X) is the working hours of an individual. In the model, one dummy variable is introduced that takes the value one (Di = ) if the ith individual is male and zero (Di = ) if the individual is female. The parameter γ in Eq. () represents the difference between the expected earnings of men and women who work the same hours. In general, the qualitative (categorical) variable can have k classifications (categories). The examples are ethnicity (i.e., Hispanic, black, or white), region (north, south, east, and west) in a study of salary levels in a sample of highly educated employees or the effects of company size (small, medium, or large) on wages. In such cases, in the standard regression model (k − ) dummy variables are required, one less than the number of classifications. In this way, the dummy trap (perfect 7multicollinearity when the model cannot be estimated) is avoided. For example, when analyzing the effects of company size on wages in which there are three types of companies (small, medium, and large), k = , two dummy variables are required in the regression model. Thus, the regression model () can be extended to Yi = α + βXi + γDMi + δDLi + ε i
()
where the dummy variable DMi takes the value of one if the ith individual works in a medium size company and zero otherwise, while DLi takes the value of one if the ith individual works in the large company and zero otherwise. The small company is taken as a reference category to
D
D
Durbin–Watson Test
which Eq. () applies. The dummy variable for the reference category is not included. In Eq. (), the parameter γ now represents the expected extra earnings of an individual who works in medium size company relative to the earnings of an individual in the reference category, i.e., in a small company. Dummy variables have many applications. For example, with dummy variables one can treat seasonality in the data or capture the effect of 7outliers (e.g., abrupt shifts in data). Furthermore, in the regression model, a dummy variable can be used as a regressor variable for coefficient stability tests, for obtaining predictions or for imposing cross-equation constraints (Maddala ). On the other hand, a dummy variable as a regressand variable is used in the linear probability model, logit model, and probit model. For more details on the topic, readers may refer to the list of references.
Cross References 7Linear Regression Models
References and Further Reading Green WH () Econometric analysis, th edn. Prentice Hall, Upper Saddle River Maddala GS () Introduction to econometrics, rd edn. Wiley, Chichester
variables) with mean zero and equal variances. In the case of uncorrelated disturbances it is known from the 7GaussMarkov-Theorem that the ordinary least squares estimator βˆ for β, where βˆ = (X ′ X)− X ′ y is best linear unbiased (BLUE) for β. In a time-series context, however, one often suspects that the components ut of u might be correlated with each other, so it is of much interest to test whether the latter is indeed the case. Due to the vastness of the alternative, there is no hope that a uniformly most powerful test exists (Anderson ) and one has to focus on more restricted alternatives. The one investigated by Durbin and Watson (, , ) stipulates that the ut ’s follow a stationary first order autoregressive process ut = ρut− + є t
(∣ρ∣ < , є i ∼ iid),
t = , . . . , T
()
so the correlation matrix V of u is given by ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ρ ⎢ V=⎢ ⎢ ⎢ ⋮ ⎢ ⎢ ⎢ T− ⎢ρ ⎣
ρ ⋮
⋱
ρ t−
⎤ ρ T− ⎥ ⎥ ⎥ ⎥ ρ T− ⎥ ⎥ ⎥ ⎥ ⋮ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦
()
and the null hypothesis boils down to H : ρ = . The DW test statistic is T
d=
∑ (uˆt − uˆ t− )
t=
∑ uˆ t
Durbin–Watson Test
uˆ ′ Aˆu , uˆ ′ uˆ
()
t=
Walter Krämer Professor and Chairman Technische Universität Dortmund, Dortmund, Germany
The Durbin–Watson test is arguably, next to the method of 7least squares, the most widely applied procedure in all of statistics; it is routinely provided by most software packages and almost automatically applied in the analysis of economic time series when a researcher is fitting a linear regression model (see 7Linear Regression Models) y = Xβ + u,
T
=
()
where y (T × ) and X (T × K, nonstochastic, rank K) denote the vector of observations of the dependent and the matrix of observations of K independent (regressor-) variables, respectively, β is a K× vector of unknown regression coefficients to be estimated, and u (T × ) is an unobservable vector of stochastic errors (disturbances, latent
where uˆ = y − X βˆ and ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ − ⎢ ⎢ ⎢ A=⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣
−
⋱
⋱
⋱
⋱
⋱
−
⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥. ⎥ ⎥ ⎥ − ⎥ ⎥ ⎥ ⎥ ⎥ ⎦
()
It is easily checked that ≤ d ≤. With positive serial correlation among the ut ’s, neighboring ut ’s and thus uˆ t ’s will tend to be close to each other, i.e., d will tend to be small. On the other hand, when ρ is negative, d will be large. A plausible two-sided test therefore rejects H whenever d moves too far from , the mid-point of its range. One-sided tests will reject H whenever d is too small (H : ρ > ) or too large (H : ρ < ).
Durbin–Watson Test
Before the era of modern computers, the usefulness of this rule was compromised by the dependence of the null distribution on the regressor matrix X. Given T and K and selected significance levels, Durbin and Watson provided upper and lower bounds for the critical values; no conclusion was possible for a test statistic in between (the “inconclusive range”). Today, exact prob-values are easily available via some variant of the Imhof-algorithm (). Also, there exist competitors like the Breusch ()– Godfrey () test which do not require fixed regressors or AR()-alternatives. The theoretical foundation for the DW test dates back to Anderson (). He showed that whenever the disturbance vector u has density f (u) = K exp [−
(( + ρ )u′ u − ρu′ ϕu)] , σ
()
where K is some constant and ϕ is a symmetric T × T matrix, a uniformly most powerful test of H : ρ = against H : ρ > is given by uˆ ′ ϕˆu > k, () uˆ ′ uˆ provided the columns of X are linear combinations of eigenvectors of ϕ. This result is linked to the DW test as follows: The inverse of Cov(u) is
− V σu
⎡ ⎢ ⎢ ⎢ ⎢ ⎢ −ρ ⎢ ⎢ ⎢ = ⎢ σє ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ =
[( + σє
−ρ + ρ
⋱
⋱
⋱
⋱
⋱ + ρ −ρ
⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ −ρ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦
ρ )I − ρϕ − ρ(ρ − )C],
where ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ϕ= ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣
⋱
⋱
⋱ ⋱
⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⋱ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦
()
D
and C= diag(,,…,,). Thus when we neglect the term ρ( − ρ)C, the density of u is of the form (), and from Anderson’s result, an UMP rejection region against H : ρ > is obtained from large values of uˆ ′ ϕˆu/ˆu′ uˆ . Since A = I − ϕ, this is equivalent to rejecting whenever ′ d = uˆuˆ ′Auˆuˆ is too small. A crucial condition for the approximate optimality of the DW test is that the column space of X be spanned by eigenvectors of ϕ. Much less is known about the power of the DW test when this condition fails. In fact, the power of the DW test can even drop to zero for certain regressors (Krämer ). This is so because for regressions without an intercept, d tends to some constant d¯ as ρ → . If d¯ is less than the critical level d⋆ corresponding to the given X matrix and significance level, the limiting power of the DW test is , otherwise it is zero (neglecting the possibility that d¯ = d⋆ ).
About the Author For biography see the entry 7Statistical Fallacies.
Cross References 7Approximations to Distributions 7Autocorrelation in Regression 7Best Linear Unbiased Estimation in Linear Models 7Gauss-Markov Theorem 7Least Squares 7Time Series Regression
References and Further Reading Anderson TW () On the theory of testing serial correlation. Skand Aktuarietidskr :– Breusch TS () Testing for autocorrelation in dynamic linear models. Aust Econ Pap :– Durbin J, Watson GS () Testing for serial correlation in least sqares regression I. Biometrika :– Durbin J, Watson GS () Testing for serial correlation in least sqares regression II. Biometrika :– Durbin J, Watson GS () Testing for serial correlation in least sqares regression III. Biometrika :– Godfrey LG () Testing against general autoregressive and moving average error models when the regressors include lagged dependent variables. Econometrica :– Imhof JP () Computing the distribution of quadratic forms in normal variables. Biometrika :– Krämer W () The power of the Durbin-Watson test for regression without an intercept. J Econom :–
D
E Econometrics Peter Kennedy Professor Emeritus Simon Fraser University, Burnaby, B.C., Canada
Several definitions of econometrics exist, a popular example being “Econometrics is the study of the application of statistical methods to the analysis of economic phenomena.” The variety of definitions is due to econometricians wearing many different hats. First, and foremost, they are economists, capable of utilizing economic theory to improve their empirical analyses of the problems they address. At times they are mathematicians, formulating economic theory in ways that make it appropriate for statistical testing. At times they are accountants, concerned with the problem of finding and collecting economic data and relating theoretical economic variables to observable ones. At times they are applied statisticians, spending hours with the computer trying to estimate economic relationships or predict economic events. And at times they are theoretical statisticians, applying their skills to the development of statistical techniques appropriate to the empirical problems characterizing the science of economics. It is to the last of these roles that the term “econometric theory” applies, and it is on this aspect of econometrics that most textbooks on the subject focus. The workhorse of econometrics is the linear regression model (see 7Linear Regression Models), extended in a wide variety of nonlinear ways. So, for example, the log of wages might be regressed on explanatory variables such as years of education and a dummy for gender, with interest focusing on the slope of years of education, reflecting the return to investing in education, and the slope of the gender dummy, measuring discrimination in the labor market. Because this kind of empirical analysis is an important element of traditional statistics, one might naturally ask “How does econometrics differ from statistics?” There are two main differences between econometrics and statistics. The first stems from the fact that most economic data come from the real world rather than from controlled experiments, forcing econometricians to develop special
techniques to deal with the unique statistical problems that accompany such data. The second difference is that econometricians believe that economic data reflect strategic behavior by the individuals and firms being observed, and so they employ models of human behavior to structure their data analyses. Statisticians are less willing to impose this kind of structure, mainly because doing so usually is not fully consistent with the data. Econometricians ignore such inconsistencies, so long as they are not gross, to enable them to address issues of interest. Some examples can illustrate these differences, the first four below resulting in Nobel prizes for econometricians. . Other things equal, females with young children earn higher wages than females without young children. Why? Females with children only appear in the labor market if their wage is large enough to entice them away from being a homemaker; this means that a sample of female wage earners is not a random sample of potential female workers – other things equal, lowwage earners are under-represented. This self-selection phenomenon causes bias in estimates of the wage equation, a bias that is of consequence because it narrows the estimated discrimination gap between male and female wages. . Economics is all about people making choices in a world of scarcity. People make choices (and thereby produce data for the econometrician) by maximizing an objective function. How do people choose which transportation mode to use to commute to work? Economic theory suggested addressing this question using the random utility model in which people choose on the basis of which option provides them with the greatest utility. Viewing the problem this way greatly enhanced the development of multinomial logit/probit analyses, resulting in markedly superior predictions of the ridership of the San Francisco bay area rapid transit system, then under construction. . For many years, in time series data it was thought that it did not make sense to employ levels and first differenced data in the same specification, and statisticians were careful to avoid doing this. But economic theory suggested that a certain combination of levels
Miodrag Lovric (ed.), International Encyclopedia of Statistical Science, DOI ./----, © Springer-Verlag Berlin Heidelberg
E .
.
.
.
Econometrics: A Failed Science?
data, representing an economic equilibrium position, could be compatible with differenced data. This concept of cointegration has changed forever the way in which time series analysis is undertaken. Why would we ever be interested in how volatile some variable will be in the future? When pricing financial instruments such as options such information is crucial. An econometrician developed a way of addressing this problem, creating a major statistical industry, referred to as ARCH, autoregressive conditional heteroskedasticity. Supply and demand curves are at the heart of economic analysis. When we regress quantity on price how do we know if the result is an estimated supply curve, an estimated demand curve, or garbage (a linear combination of the two)? To some the solution to this identification problem in the late s marked the beginning of econometrics as a separate discipline. (But to others the founding of the Econometric Society and its journal Econometrica in the early s marks the beginning of econometrics.) The interaction of supply and demand illustrates another special feature of econometric work. Suppose we are estimating a demand curve and suppose amount demanded changes due to a bump in the error term in the demand equation. This shifts the demand curve, changes the intersection of the supply and demand curves, and so changes price. A regression of quantity on price produces biased estimates because the explanatory variable price is correlated with the error term in the demand equation. This dilemma of simultaneous equation bias is widespread, occurring whenever variables feed back to one another. Instrumental variable estimation, developed by econometricians, is a way of avoiding bias in this context. When a sporting event is sold out, we don’t know the actual demand for tickets; all we know is that the demand was at least as big as the venue’s seating capacity. When estimating the determinants of demand what should we do with the observations on sold-out games? Estimation in the context of such limited dependent variables is another example of special estimation procedures developed by econometricians.
For a much more detailed overview of econometrics, including discussion of its historical development, see Geweke et al. (). For both students and practitioners, Kennedy () is a very popular source of information on econometrics, offering intuition, skepticism, insights, humor, and practical advice. Section ., beginning on p. , offers general commentary on “What is Econometrics?” and provides further references.
About the Author Peter Kennedy received his BA from Queen’s in and Ph.D. from Wisconsin in . He is Professor Emeritus, Simon Fraser University. He held visiting positions at Cornell, Wisconsin, the London School of Economics, Singapore, Deakin, Cape Town, Canterbury, Curtin, Adelaide, Otago, and EERC (Ukraine). Peter Kennedy is best known as the author of A Guide to Econometrics (sixth edition, , Wiley-Blackwell). He is the recipient of four awards for excellence in teaching, and one award for excellence in research. Upon retiring from Simon Fraser University in , he stepped down as an associate editor of the International Journal of Forecasting and of Economics Bulletin, but continues to edit the research section of the Journal of Economic Education. “Kennedy’s Guide is a handbook of important concepts, techniques, and mathematical results that are developed at length in the standard textbooks but for which insights, interpretations, and intuition are missing. The popularity of his earlier three editions and their presence on students’ shelves around the world documents the usefulness of Kennedy’s ability to communicate the essence of econometrics without losing the reader in technical details” (Becker W. E., The Journal of Economic Education, , , p. .).
Cross References 7Components of Statistics 7Econometrics: A Failed Science? 7Economic Statistics 7Linear Regression Models 7Principles Underlying Econometric Estimators for
Identifying Causal Effects
References and Further Reading Geweke JK, Horowitz JL, Pesaran MH () Econometrics: a bird’s eye view. In: Blume L, Durlauf S (eds) The new palgrave dictionary of economics, nd edn. Macmillan, London Kennedy PE () A guide to econometrics, th edn. Blackwell, Oxford
Econometrics: A Failed Science? Jan Kmenta Professor Emeritus University of Michigan, Ann Arbor, MI, USA Center for Economic Research and Graduate Education, Prague, Czech Republic The official beginning of 7econometrics can be traced to the first issue of Econometrica (), the journal
Econometrics: A Failed Science?
dedicated to the “advancement of economic theory in its relation to statistics and mathematics.” This makes it clear that the development of economic theory as a discursive process is to be supplemented by quantitative analysis and that econometrics is a part of economics. The earliest major attempts at specifying and estimating economic relationships include the famous – and to this day used – Cobb-Douglas production function (), the first macro-econometric models of Jan Tinbergen ( and ), and various unsuccessful attempts at estimating supply and demand functions in the s. The first systematic development of the mathematical formulation of economic theory, without which rigorous research in econometrics could not exist, is due to Paul Samuelson’s Foundations of Economic Analysis (). The next step was the development of econometric methods largely due to the work of the researchers at the Cowles Commission at the University of Chicago (–). The resulting series of monographs included a solution to the identification problem that allowed separate estimations of supply and demand functions. An irony of fate is that the solution of the identification problem was provided already by Tinbergen in an overlooked article written in German and published in Vienna in . The Cowles Commission monographs laid foundation to the formulation and estimation of simultaneous equation models that swamped the econometric literature in the s and s. All of this led W.C. Mitchell, one of the founders of the National Bureau for Economic Research (NBER), to declare in that “we must expect a recasting of old problems into new forms amenable to statistical attack.” The monographs of Cowles Commission would not have made an impact on the profession of economics and would not have led to the ensuing “scientific revolution” in economics – had they not given rise to a number of early popularizers who translated the sophisticated and technical language of the monographs into a language understandable to economists. Prominent among these were Gerhard Tintner, Henri Theil, Lawrence Klein, and Denis Sargan who carried the torch of econometric enlightenment in the s and s. The subsequent two decades were marked by the introduction of the computer technology that enabled implementation of Monte Carlo experiments, formulation and estimation of large macro-econometric models, handling of large micro-data sets, and 7numerical integration of Bayesian inference. Indeed, at a conference in Princeton in computers were reported to have been of only moderate influence in various disciplines except economics. Of course, the use of computers required an accessory development of software. In this context a breakthrough was made by Harry Eisenpress who in compiled a
E
program for computing maximum likelihood estimates of the coefficients of simultaneous equation models. The introduction of computer technology really helped econometrics to take off. The macro-econometric modeling efforts were crowned by the Nobel Prize awarded to Lawrence Klein in , the upsurge of microeconometrics culminated with the award of the Nobel Prize to James Heckman and Daniel McFadden in , and the modern “time series revolution” was blessed by the award of the Nobel Prize to Robert Engle and Clive Granger in . However, the golden years of econometrics were not without criticism. Articles and books with ominous sounding titles made an appearance that did not go unnoticed in the profession. Following are some of the titles appearing in the literature: “Econometrics: Alchemy or Science?” (Hendry ) “Lets Take the ‘Con’ out of Econometrics” (Leamer ) “Data Mining” (Lovell ) “What Will Take the Con out of Econometrics” (McAleer et al. ) “The Foundations of Econometrics: Are There Any?” (Swamy et al. ) Economics in Disarray (wiles and Routh ) The Death of Economics (Ormerod ) There were several bases for these criticisms, one of which was the poor specification and forecasting performance of econometric models. The sharpest criticism came from Sims () who declared that “among academic macro-economists the conventional methods (of macroeconometric modeling), have not just been attacked, they have been discredited.” Sims’ response to the challenge was the introduction of the vector-autoregressive (VAR) model formulation (see 7Vector Autoregressive Models). Another line of criticism, coming from David Hendry, was about estimation being the focus of econometric practice. According to Hendry (), “what should have been a relatively minor aspect of the subject, namely estimation, has been accorded the centre of the stage…the three golden rules of econometrics are test, test, test !” The common practice of “data mining,” (see 7Data Mining) namely trying different model formulations until the estimated parameters conform to the pre-judgment of the investigator, also did not escape criticism. A sarcastic but fitting description of this practice is due to Leamer () who noted that “if you torture data long enough, Nature will confess.” Hendry () provided further elaboration of this point by a cute example, “Econometricians have found their Philosopher’s stone. It is called regression analysis and it is used for transforming data into ‘significant results’.
E
E
Econometrics: A Failed Science?
Deception is easily practiced.” As an example, Hendry came up with his own ‘theory of inflation’ in which the price level is a function of a variable C(t) and an auto regressive disturbance, where C(t) is cumulative rainfall in the UK. The fit was spectacular, even better than if C(t) was replaced by the quantity of money. Perhaps the most serious criticism was leveled against the alleged meager contribution of econometrics to economic knowledge. The earliest criticism on this count came from Leontief () who in his presidential address declared that, “in no other field of empirical inquiry has so massive and sophisticated statistical machinery been used with such indifferent results.” Blaug () elaborated on this theme by stating that, “empirical work that fails utterly to discriminate between competing explanations quickly degenerates into a sort of mindless instrumentalism and it is not too much to say that the bulk of empirical work in modern economics is guilty on that score.” The “time series revolution” in econometric practice, initiated by Engle and Granger in the s, does not help in the direction of expanding economic knowledge either. On the contrary the mechanistic modeling of time series analysis pushes econometrics away from economics. This “revolution” is based on the contention – not rejected by statistical tests – that many economic variables have infinite variances, that is, that they grow without limit in time, and thus observed relationships maybe purely fictitious. The consoling fact is that while the variables themselves have infinite variances, typically the linear combinations of their first differences may well have variances that are finite. However, there are some serious problems with the way time series analysis is presented in the text books and applied in research. With respect to the former, the standard approach of teaching time series analysis ignores completely the classical econometrics, whose basic foundation is the regression model. A logical way would be to explain which of the assumptions of the classical regression model are likely to be violated when dealing with economic time series data. This would then be followed by discussion of the undesirable consequences of this on the properties of the least squares estimators and what to do about it. As it is, there is a complete disassociation between classical econometrics and time series econometrics. A lack of connection between time series analysis and economics as such is even more remarkable. The standard explanation of the long-run path of any economic variables as being purely determined by the passage of time plus a stochastic disturbance, with no reference to economic factors, is “primitive beyond words” (Kocenda and Cerny ). As for the so-called “co-integration analysis,”
uncovering that certain economic variables move together in stable relationships and looking for their economic interpretation is in contrast to the traditional deductive approach in which theory comes first and is followed by testing. A final bone of contention regarding econometrics are the common results of empirical research that conflict with each other. This was pointedly illustrated by Leamer () in his comments on the study of the deterrent of capital punishment …“there was a great outpouring of papers that showed that the results depend on (i) which variables are included; (ii) which observations are included; (iii) how simultaneity is treated, etc.” The source of failure of econometrics to satisfy its critics is undoubtedly due to, “a huge credibility gap between economic theory, empirical evidence and policy prescriptions” (Phillips ). The fault is on both sides; on the side of economic theorists as well as on the side of applied econometricians. Most economic theorists do not allow for the stochastic nature of the world, and most applied researchers tend to be very casual about the stochastic specifications of their models. Most of the work of economic theorists deals with deterministic models, while most of the applied workers treat the stochastic disturbance in their models as an afterthought barely worth mentioning. There are exceptions, of course, and they should be appreciated. In the field of micro-economics, the shining example of stochastic consciousness are the labor economists, while in the field of macro-economics the orchid goes to the dynamic stochastic general equilibrium models (DSGE), the modern successors of the simultaneous equation models of yesterday. On the whole, though, the gap between economic theory and empirical research is daunting. To rescue the reputation of econometrics and to “advance economic theory in relation to statistics and mathematics” it is, in my opinion, absolutely necessary for economic theory and econometrics to merge. There should be no separation between economic theory and econometrics as it by and large exists today.
About the Author Jan Kmenta received his B.Ec with First Class Honors from the University of Sydney, Australia in and graduated from Stanford University with a Ph.D. in Economics with a minor in Statistics in . He is a Fellow of the American Statistical Association since and a Fellow of the Econometric Society since . Professor Kmenta has received the Alexander von Humboldt Foundation Award
Economic Growth and Well-Being: Statistical Perspective
for Senior U.S. Scientists (–). He was awarded an Honorary Doctorate from the University of Saarland, Germany (), and in he was the first recipient of the Karel Engliš Medal from the Academy of Sciences of the Czech Republic. In he received an annual award from the Czech Society of Economic for long-term contribution to the development of teaching of economics in the Czech Republic. Professor Kmenta has published a number of articles in prominent economic and statistical journals. He was ranked as th among all economists ranked by the total number of citations (–). Professor Kmenta is well known as the author of the internationally respected text The Elements of Econometrics (University of Michigan Press, nd edition, ), that was translated into several foreign languages.
Cross References 7Econometrics 7Linear Regression Models 7Seasonal Integration and Cointegration in Economic
Time Series 7Time Series 7Vector Autoregressive Models
References and Further Reading Blaug M () The methodology of economics. Cambridge University Press, Cambridge Hendry DF () Econometrics: alchemy or science? Economica :– Kocenda G, Cerny A () Elements of time series econometrics. Charles University Press, Prague Leamer EE () Lets take the ‘Con’ Out of econometrics. Am Econ Rev :– Leamer EE () Sensitivity analysis would help. Am Econ Rev :– Leontief W () Theoretical assumptions and nonobserved facts. Am Econ Rev :– Lovell MC () Data mining. Rev Econ Stats :– McAleer M et al () What will take the con out of econometrics. Am Econ Rev :– Mitchell WC () The backward art of spending money. Augustus M. Kelley, New York Ormerod P () The death of economics. Wiley, New York Phillips PCB () Reflections on econometric methodology. Econ Rec :– Sims C () Macroeconomics and reality. Econometrica : – Swamy PAVB et al () The foundations of econometrics: are there any? Econometric Reviews :– Tinbergen J () Bestimmung und Deutung von Angebotskurven: Ein Beispiel. Zeitschrift für Nat, Vienna () Wiles P, Routh G () Economics in disarray. Basil Blackwell, New York
E
Economic Growth and Well-Being: Statistical Perspective Zarylbek I. Kudabaev Chairman of the Statistical Society of Kyrgyz Republic, Professor, Head of Economic Department American University of Central Asia, Bishkek, Kyrgyz Republic
E 7 There is a clear case for complementing GDP with statistics covering the other economic, social and environmental issues, on which people’s well-being critically depends. European Commission ()
Access to well-developed, cross-cutting, high-quality shared information is a basic need of an effective democracy. The more information people have about economic and social conditions and trends in their countries, the better able they are to demand from politicians policies that improve economic growth and development. Another issue is the public’s trust in its government. If people don’t believe the figures that their official statistical bodies produce, they are likely to lose trust in government as a whole, which can have a negative impact on the democratic process. So, the challenge is avoiding a gap between the official statistics on economic performance and the public’s perception of their own living conditions. Such a gap has been evident over a long period in many countries around the world, but it is especially the case for the developing countries. Possible reasons for the gap between the official statistics and people’s perceptions are misuse of certain statistics or the poor quality of official statistics.
Misuse of Certain Statistics Statistical data produced for one purpose may not be appropriate for other purposes. A good example is gross domestic product (GDP), which tells us a nation’s total income and the total expenditure on its output of goods and services and mainly is a measure of market production. GDP is not well suited as a metric of well-being, but it has been increasingly used for this purpose. GDP fails to capture such factors as unpaid work of households, distribution of income among the population groups, and depletion of resources, which creates a problem in using GDP (or GDP per capita) as an indicator of well-being. For example, if inequality in income distribution among population groups increases enough relative to the increase in average GDP per capita, then most people will be worse off even though average income is increasing. Or, if the number of cars drastically increases, then GDP goes up, but it
E
Economic Growth and Well-Being: Statistical Perspective
also leads to traffic jams and an increase in air pollution. In these examples, from the public’s perspective, an increase in GDP does not improve their quality of life at all. For these reasons the original architects of the United Nations System of National Accounts (SNA) knew that well-being could not be measured by national income alone.
Low Quality of Official Statistics Developing countries are more likely to produce official statistics of lower quality. Due to economic difficulties, developing countries struggle with making significant investments to develop statistical systems that are able to produce, using internationally recognized methodologies, a whole set of necessary indicators of economic and social progress. The quality of official statistics might also be low because existing methodologies are imperfect. For example, consider how well statisticians can measure a nonobserved, or shadow, economy. The economy of any country consist of two parts, observed and non-observed. The share of non-observed (shadow) economy in the GDP can be significant, especially in developing countries (up to % and higher). Also, it is well known that, in developing countries, many city-dwellers and internal migrants from rural areas are able to survive only because of their employment in the non-observed economy. Therefore, it is not possible to ignore a measure of non-observed economy in terms of both economic growth and development (well-being). It has to be separately estimated (measured) and then included in the country’s GDP. There are several methodologies, based on different approaches, for measurement of the non-observed economy: conducting surveys of households on production of goods and services in the non-observed economy, labor force surveys, analysis of cash operations, analysis of electricity supply, and so on. But the problem is that all of the existing methodologies for measurement of the non-observed economy are not as good as those for the observed economy, and they should be improved significantly. The same criticism can applied to the statistical data on services, which should be improved as well.
What Should Be Done? Statistical indicators for economic performance, although they are well developed, need to be improved in order to provide accurate monitoring of the evolution of modern economies (Stiglitz et al. ). Evolution toward a more complex economy means the growth of the share of services and production of a more complex quality of goods. The “Going beyond GDP” strategy is very promising as a better measure of societal well-being, quality of life, and
progress. “Going beyond GDP” does not mean dismissing GDP and production measures. It means complementing GDP with statistics on which people’s well-being critically depends. Well-being is multidimensional and includes not just economic but also social, environmental, and governance, and, as mentioned in (Stiglitz et al. ), the following dimensions should be considered simultaneously: . Material living standards (income, consumption, and wealth) . Health . Education . Personal activities including work . Political voice and governance . Social connections and relationships . Environment (present and future conditions) . Insecurity, of an economic as well as a physical nature Real implementation of the “Going beyond GDP” strategy is a huge challenge for statisticians, and it likely requires a shift in emphasis from measuring economic production to measuring people’s well-being. Ideally, national accounts of well-being should be developed for the successful monitoring of social progress, as is already done for monitoring of economic progress (Michaelson et al. ). Forty-eight indicators of the Millennium Development Goals (MDG), in combination with the indicators of SNA, could be a good basis for construction of the national accounts of well-being. How can developing countries improve their statistical products? International organizations and well-developed countries could play a key role. Strengthening the statistical capacities should become an important part of their programs of support for developing countries.
About the Author Professor Zarylbek I. Kudabaev, Doctor of Economics and Ph.D in Physics. He is a Professor and Head of the Economics Department, American University of Central Asia (AUCA) in Bishkek (Kyrgyz Republic). He is Past Chairman (–) of the National Statistical Committee of the Kyrgyz Republic. He is a Chairman of the Statistical Society of Kyrgyz Republic. He was a member of Steering Committee of the International Consortium “Partnership in Statistics for Development in the st Century” (PARIS, –). In he was a Chairman of the Council of the Heads of Statistical Services of CIS countries. He is National Coordinator for subscribing the Kyrgyz Republic to the International Monetary Fund’s (IMF) General Data Dissemination Standard () and IMF’s Special Data Dissemination Standard (). Professor
Economic Statistics
Kudabaev is author of monographs and more than scientific articles on statistics and economic development.
Cross References 7Economic Statistics 7Measurement of Economic Progress
References and Further Reading European Commission () GDP and beyond: measuring progress in a changing world, communication from the Commission to the Council and the European Parliament, Brussels, COM () final Michaelson J, Abdallah S, Steuer N, Thompson S, Marks N, Aked J, Cordon C, Potts R () National accounts of well-being: bringing real wealth onto the balance sheet. The New Economics Foundation, London (available at www.neweconomics.org) Stiglitz JE, Sen A, Fitoussi J-P () Report by the commission on the measurement of economic performance and social progress. (available at www.stiglitz- sen- fitoussi.fr)
Economic Statistics Yasuto Yoshizoe Professor, President of the Japan Statistical Society Aoyama Gakuin University, Tokyo, Japan
Introduction In this article, we regard economic statistics as a branch of applied statistics which deals with the following topics: () collection of statistics on socioeconomic conditions, () compilation of surveys and registered records to produce various economic indicators, such as consumer price index or GDP, () evaluation of economic indicators from the viewpoint of reliability. Economic statistics is closely related to official statistics, since most of the statistics of the society and economy are provided by official organizations like national statistical offices or intergovernmental organizations such as United Nations, OECD, and World Bank. On the other hand, economic statistics is different from econometrics in a narrow sense. Typically, objective of econometrics lies in developing the theory and its applications to various economic data. In contrast, economic statistics places more emphasis on quality of data before applying sophisticated methods to analyze them. In other words, we are more interested in appropriate interpretation of the data paying attention to their detailed characteristics. Here, we describe some typical issues from economic statistics: censuses, sample surveys, index numbers, system of national accounts, followed by an illustrative example.
E
Censuses – Complete Enumeration Consumers and producers consist of two major components in economics. Correspondingly, two fundamental information of economic statistics are () population and households, and () firms and establishments. U.S. Census Bureau, http://www.census.gov/econ/, provides definitions of firms and establishments among other related concepts. To construct reliable statistics on these subjects, the complete enumeration is required. A 7census is the procedure of collecting information about all members of a population. In many countries, population censuses are carried out periodically. In contrast, information of firms and establishments are obtained either by statistical surveys (so-called economic censuses) or by registered information collected through some legal requirement. The complete enumeration is necessary to provide accurate information for small areas. In planning the location of elementary schools, hospitals or care agencies for elderly people, local governments need information for small areas. If the censuses or registration records provide accurate information, local governments can depend on them in making important policies. Another role of the census is that it provides a list of households or firms/establishments. Such a list is used as the sampling frame, and it plays an essential role in many sampling surveys described in the next section. In this sense, inaccurate censuses, or administrative records that are out of date, reduces the reliability of sample surveys, and eventually increase social costs. For household surveys, efficient sampling designs, such as stratified sampling, become difficult if accurate sampling frame is not available. The inaccurate sampling frame can sometimes cause extremely biased estimates for business surveys due to frequent entry/exit behavior of firms/establishments. An example is given later.
Sample Surveys Since economic conditions vary dynamically, we need monthly or quarterly indicators, such as consumer price index or industrial production index. These statistics have to be compiled from sample surveys on households and/or establishments. Therefore, the accuracy of sampling survey is crucial in maintaining reliability of economic indicators. Major factors that affect performance of the estimators are: sampling design (e.g., simple random sampling, stratified sampling, cluster sampling, etc.) and estimation methods (e.g., linear estimator, ratio estimator, regression estimator, etc.). See Cochran (), Groves et al. (), Lesser and Kalsbeek (), and Särndal et al. () for detail.
E
E
Economic Statistics
The auxiliary information available from censuses are used to set up appropriate strata, or to apply a ratio estimation, resulting in more efficient and inexpensive surveys. An example of ratio estimator is to use the size of population who are older than years (Y) to estimate the number of employees (X). Let the sample means are x¯ and y¯ in a sample of size n from a population of size N. In case of simˆ = N x¯ . On ple random sampling, the linear estimator is X the other hand, if population Y is obtained from a census and additional registration records, then the ratio estimaˆ r = Y(¯x/¯y) can be applied. Since X and Y tor defined by X ˆ r has a smaller variance than X. ˆ are correlated, X More important and difficult task is that we need to evaluate the non-sampling errors, such as nonresponse caused by many reasons, possible bias from a deteriorated sampling frame, etc. In the example above, the ratio estimator may become worse if Y is incorrectly measured. As for business surveys, about % of establishments start business and another % shut down annually in case of Japan. It is clear that maintaining accurate list of establishments is vital for high quality of economic statistics.
Index Numbers 7Index numbers are widely used to compare levels and
evaluate changes of economic variables over time or among regions (see Schultze and Mackie ; ILO ). Typically, the base value equals , but it can be other values. For example, the real GDP can be regarded as a quantity index, if we choose its base value as the nominal GDP of the base year. Most often, index numbers are used to compare economic variables, such as unemployment or prices, over time. In some cases, however, index numbers may compare geographic areas at a specific time. Such an example is the purchasing power parity, which is an international comparison of prices among countries. As for price index and quantity index, the most wellknown formulas are Laspeyres, Paasche, and Fisher. If we deal with i = , . . . , n commodities (goods or services), we need to collect data of prices pit and quantities qit at time t. Then the three price indexes (base period is t = ) are defined as Laspeyres: PL = ∑ pit qi /√ ∑ pi qi , Paasche: PP = ∑ pit qit /∑ pi qit , and Fisher: PF = PL PP . Similarly, quantity indexes are defined by interchanging p and q in the definitions. These formulas are, however, too superficial for practical purposes. In calculating consumer price index (CPI), national statistical offices of developed countries adopt much more complicated methods using vast amount of related data. The price of a specific merchandize (a pack
of milk, for example) varies among stores and areas. Moreover, there are several kinds of milk that consumers purchase in a given period. The observed prices are averaged by some method before the price is used as pit in an index formula like Laspeyres. Hence, we should review and evaluate the appropriateness of the specific method employed. The judgment also depends on the data collection procedure of prices and weights, which varies among national statistical offices. Since CPI has a rigorous background in economic theory, there is a good amount of reference, including the definition of the true index number. Helped by the theory, some statistical offices provide, although approximately, the superlative price index, which has enhanced the reliability of CPI. In other applications of index numbers, however, the interpretation cannot be so exact, and the quality of an index is judged by coverage of commodities and reliability of data sources used in calculating the index. For example, the Bank of Japan publishes a monthly price index called Corporate Goods Price Index (CGPI). It is a Laspeyres price index, but the prices are mixtures of producers prices and wholesale prices. Thus, despite the title, experienced economists do not regard summary CGPI (for all commodities) as a price index but an indicator of business activities. On the other hand, since the data collection process guarantees high reliability of CGPI data, collected prices are quite important for many official statisticians to calculate other statistics involving prices.
System of National Accounts The purpose of the United Nations System of National Accounts (SNA) is to provide an integrated accounts of significant economic activities to make international or temporal comparisons possible (Lequiller and Blades ). The most important implication of SNA to economic statistics is that the system provides the conceptual and actual basis to judge and achieve coherence among all statistical sources available in a country. In this sense, national accounts are the core of modern economic statistics. Since the data from national accounts are so rich and full, it is not easy for economists to understand the whole system. To make things more complicated, some important changes have been made in the new version called SNA followed by a revision (SNA ). The difference from the earlier version becomes quite important for drawing meaningful analytical conclusions when we are interested in making inflation forecasts or in assessing economic growth capabilities.
Economic Statistics
Rigorously speaking, the national accounts system adopted by national statistical offices differ in some extent from country to country. Therefore, we need to make sure whether the differential in economic growth observed in the s between the United States and the European Union is real or not. The answer is that it just reflected the different treatment of prices of the products for information and communication technology (ICT), where rapid technical changes have been taking place. The quality of national accounts depends heavily on the statistical system of the country. To be more specific, GDP and other national accounts are not the result of a single survey, but the result of combining a mixture of various data from many sources. Since it is practically impossible for national accounts to cover all units in a country, a significant number of adjustments has to be made. In the process of adjusting those data to achieve coherence with SNA, quite often, some obscure methods are applied. Hence, it is quite difficult to obtain a formal assessment on the accuracy of the GDP, and an attempt to construct an interval estimation is almost meaningless. National accounts should be regarded at best as approximations. Another current issue is increasing publication of quarterly accounts in many countries. One of the essential objectives of economic statistics is to provide appropriate information to policy-makers at the right moment. In
particular, it is requested to provide the newest information regarding the business cycle. In this context, traditional annual national accounts are not useful, and the demand for quarterly accounts has been increasing. The availability and the reliability of the statistics or administrative records is the key to reliable quarterly GDP, but in some countries, even monthly GDP are compiled by national statistical offices or by private research institutions. Today, it is well-known that GDP is not a single measure of well-being, and it has many drawbacks. Yet, GDP remains to be the most important economic statistics as it defines a standard. To conclude, we should be aware of the usefulness and limitations of GDP and other economic statistics.
An Example In this section, we describe some issues in the statistics of corporation. Refer to Yoshizoe et al. () for a more detailed description. The Japanese Ministry of Finance publishes Financial Statement Statistics of Corporations by Industry consisting of Annual Survey and Quarterly Survey, whose accuracy is vital for the government to judge the current economic situation. The large-scale corporations are quite few in number (the population ratio is .%), but their influence is predominant (.% in sales, .% in profit, and .%
Fixed assets per coproration (1971 – 1990, all size)
million yen 1,000
900
800
700
Economic Statistics. Fig. All corporations
1990
1989
1988
1987
1985
1986
1984
1983
1982
1981
1980
1978
1979
1977
1976
1975
1974
1973
1971
1972
600
500
E
E
E
Economic Statistics
Fixed assets per coproration (1971 – 1990,large-scale)
million yen 6,000 5,500 5000 4,500 4,000 3,500 3,000
1990
1989
1987
1988
1986
1985
1984
1983
1982
1981
1979
1980
1978
1977
1975
1976
1974
1971
2,000
1973
2,500
1972
Economic Statistics. Fig. Large-scale corporations
in fixed assets, in ). Nevertheless, trends of small-scale corporations are also important, especially in economic forecasts. One of the problems is that some statistics related to small-scale corporations occasionally show unnatural patterns. Figures and show “fixed assets per corporation” of all corporations and those of large-scale corporations, respectively. Figure indicates considerable drops each year from January–March period to April–June period. On the other hand, Fig. for large-scale corporations (enumerated completely) does not show noticeable drops. We found out that the gaps in Fig. was caused by the sampling design. Since the response burden is rather heavy for small corporations, sample corporations are replaced after a year. Thus, newly sampled corporations start to respond to the questionnaires in April–June period. During the survey period (one year), some corporations go out of business and drop out of the survey. On the other hand, newly established corporations which start operating in that year cannot be included in the survey until the new list of corporations becomes available. The fact that these new entrants are usually smaller in size means that in Quarterly Survey, the corporations that survive and respond throughout the survey period tend to have better business performance than average. This in turn implies that, in the
sample, indicators such as fixed capital per corporation will gradually have upward bias from April–June to January– March in the next year. When corporations are selected from the new list which becomes available in April, the new sample represent the population more appropriately. Thus, the sampling design explains a major portion of the gaps in the statistics. We also examined the way corporations provide financial statements as another possible source of the gap. While large-scale corporations prepare quarterly statements, most small-scale corporations only prepare annual statements reflecting the difference of legal requirement. Moreover, it is often said that interim financial results are systematically different from the final financial statement even for large corporations. If we look at Fig. carefully, we can find occasional gaps from January–March to April– June. The difference between provisional settlement of account and the final statement explains the gaps for larger corporations, hence a similar systematic behavior may partially explain the gaps of small-scale corporations. Further analyses and possible solutions for the gap problem can be found in the reference given above.
About the Author Professor Yoshizoe is President, Japan Statistical Society (-present).
Edgeworth Expansion
E
7Business Statistics 7Composite Indicators 7Econometrics 7Index Numbers 7National Account Statistics 7Social Statistics
χ(t, u) into a power series of u and then, by setting u = , obtained another formal expansion that is equivalent to a rearrangement of the Chebyshev expansion. Although the two are equivalent in the sense of formal expansion, the Edgeworth expansion (EE) provides an optimal approximation when only a finite number of moments of Xn exist.
References and Further Reading
Formal Expansion
Cochran WG () Sampling techniques, rd edn. Wiley, New York Groves RM et al () Survey methodology. Wiley, New York International Labour Organization () Consumer price index manual: theory and practice. International Labour Office, Geneva Lequiller F, Blades D () Understanding national accounts. OECD, Paris Lesser JT, Kalsbeek WD () Nonsampling error in surveys. Wiley, New York Särndal C-E, Swensson B, Wretman J () Model assisted survey sampling. Springer, New York Schultze CL, Mackie C (eds) () At what price? Conceptualizing and measuring cost-of-living indexes. National Academy Press, Washington, DC Yoshizoe Y et al () Correcting non-sampling errors in Financial statement statistics of Japanese Ministry of Finance. Proceedings of the th meeting of the International Statistical Institute in Lisbon, Portugal. http://www.yoshizoe- stat.jp/english/ isi_yoshizoe.pdf
Suppose that all moments of Xn exist and additionally all cumulants (also called semi-invariants) ν nk of Xn exist. In this case, the characteristic function of Xn can be written as
Cross References
∞
ψ n (t) = exp (∑ r=
∞ ν nr (it)r ν nr (it)r ) = ϕ(t) exp (∑ ), r! r! r=
where ϕ(t) = exp (−iµt −
t σ ),
µ = ν n and σ = ν n . In
applications, the parameters µ and σ are independent of n and ν nr , r ≥ , has the orderr r− of n−(r−)/ . ∞ ν nr (it) u ) into a power series in u: Expand exp (∑r= r! ∞
χ(t, u) = exp (∑ r=
∞ ν nr (it)r ur− ) = + ∑ pnj (it)uj , () r! j=
where pnj (x) is a polynomial with powers of x ranging between j + and j and coefficients depending on the cumulants ν nr , r = , , . . . , j + . Some examples: (it) ν n (it) ν n (it) ν n , pn (it) = + , pn (it) (it) ν n (it) ν n ν n + . =
pn (it) =
Edgeworth Expansion Zhidong Bai Professor Northeast Normal University, Changchun, China
Introduction In many statistical problems, for example, hypothesis testing and confidence region estimation, it is necessary to know the distribution functions of statistics of interest. When they are not exactly known, one might like to have an approximation of the unknown distribution as accurately as possible. One of the possibilities is the normal approximation. However, it is proved that normal approximation has an unimprovable rate of O(n−/ ) . To obtain a more accurate approximation for the distribution function, Fn , of a random variable Xn , Chebyshev () expanded the function ψ(t)/ϕ(t) (definition given below) into a power series of (it) and obtained a formal asymptotic expansion of Fn . Edgeworth (, ) expanded
Differentiating r times with respect to x both sides of the formula π ∫ e−itx e−t / dt = √π e−x / ≡ φ(x), the density of the standard normal variable, we have dr (it)r e−itx−t / dt = (−)r r φ(x) = Hr (x)φ(x), ∫ π dx
where Hr is the r-th Chebyshev–Hermitian polynomial. From this, one can easily derive that the inverse Fourier transform of (it)r exp(itµ − t σ /) is σ −r Hr ((x − µ)/σ) φ ((x − µ)/σ). j Write pnj (it) = ∑r=j+ bnr (it)r . By noting that ∞ ⎛ ⎞ ψ n (t) = ϕ(t) + ∑ pnj (it) , ⎝ j= ⎠
the distribution function of Xn can be written as Fn (x) = Φ (
∞ x−µ x−µ x−µ ) − ∑ Qnj ( )φ( ), σ σ σ j=
E
E
Edgeworth Expansion
where Qnj (x) = ∑r=j+ bnr σ −r Hr− (x). This is called the formal Edgeworth expansion (FEE) of Fn (x). The first success of rigorous theory was done by Cramér (). j
Validity of Edgeworth Expansion The FEE may not be valid in three senses: () in real applications, not all orders of moments exist and thus the FEE is not well-defined; () even if all terms of FEE are welldefined, the series may not converge; and () the equality may not be true even if the series converges. To this end, one needs to establish a valid approximation when only the first k (k ≥ ) moments are finite. In this case, all terms Qj , j ≤ k − in the FEE are well-defined and thus, our task reduces to establishing error bounds for ⎤ ⎡ k− ⎢ x−µ ⎥ x−µ x−µ ⎥. ) − ) φ ( ) Q ( Φ ( Rn (x) = Fn (x)−⎢ ∑ j ⎥ ⎢ σ σ σ ⎥ ⎢ j= ⎦ ⎣ () If the error bound for Rn (x) is given by the form Kg(n, k), the bound is then called a uniform estimation; if the bound is of the form Kg(n, k)( + ∣x∣)−k , the bound is then called a nonuniform estimation, where K is an absolute constant and g(n, k) → describing the order of the convergence. The quantity in the brackets is called the (k − )-term EE of Fn (x) or Xn .
Sum of Independent Random Variables Suppose that for each n, {Xnj , j = , , . . . , n} is a sequence of random variables with means , variances σnj , finite k-th moments and 7characteristic functions vnj (t) satisfying lim sup sup ∣t∣→∞
n
n ∑ ∣vnj (t)∣ < n j=
()
(called the Cramér condition), then for the distribution n function Fn (x) of Sn = B− n ∑j= Xnj , we have n k+ ⎛ ∑nk= E∣Znjx ∣k ∑j= E∣Wnjx ∣ ⎞ , + ∣Rn (x)∣ ≤ B k+ ⎠ ⎝ Bkn ( + ∣x∣)k Bk+ n ( + ∣x∣)
where Bn = ∑nj= σnj , Znjx = Xnj I(∣Xnj ∣ > Bn ( + ∣x∣) and Wnjx = Xnj I(∣Xnj ∣ ≤ Bn ( + ∣x∣). If the r-cumulants of Xnj is denoted by ν jr , then the cumulants of Sn is given by µ = , n σ = and for r ≥ , ν nr = B−r n ∑j= ν jr . When the random variables are independent and identically distributed (IID) with mean and variance σ and r-th cumulant ν r , we have ν nr = n−(r−)/ ν r /σ r and thus Qnj (x) = n−j/ qj (x) the EE of Fn .
Function of Sample Means Suppose that {Y , . . . , Yn } is a sequence of iid random m-vectors with mean vector µ, covariance matrix Σ and
finite k-th moment. Assume the function H defined on Rm is k− times continuously differentiable in a neighborhood ¯ one has of µ. By the Taylor expansion of H(Y), k− √ √ ¯ − H(µ)) = ∑ n ∑ li ,...,im n (H(Y) r= r! i +⋯+i m =r m
i ∏ ij !(Y¯ (j) − µ(j) ) j + Rk ,
()
j=
where li ,...,im =
∂ r H( µ ) ⋯∂ µ i m ∂ µ () (m) i
, Y¯ (j) and µ(j) are the j-th
¯ and µ, respectively. If the cumulants are components of Y defined by those of the first term on the right-hand side of √ ¯ − H(µ)) is Eq. (), then the (k − )-term EE for n(H(Y) −(k−)/ valid with a uniform bound o(n ) if σ > and the Cramélr condition lim sup ∣Eeit Y ∣ < , ′
∥t∥→∞
()
holds, where σ = ℓ′ Σℓ and the j-th component of ℓ is ∂H( µ ) . ∂ µj
Partial Cramér Condition Condition () may not hold when Y contains a discrete component and the moment condition may not be minimal when some component of ℓ is . To this end, we have the following theorem. √ ¯ − H(µ)) Theorem The (k − )-term EE of n (H(Y) −(k−)/ is valid with a uniform bound o(n ) if the following conditions hold: () The following partial Cramér condition holds lim sup E∣E(eitY() ∣Y() , . . . , Y(m) )∣ < . ∣t∣→∞
() There is an integer p < m such that ℓ ≠ and l,...,,ip+ ,...,im = , provided ip+ + ⋯ + im ≥ , where ℓ is the first component of ℓ. () E∣Y(j) ∣k < ∞ for j = , . . . , p and E∣Y(j) ∣k/ < ∞.
Cornish–Fisher EE For a constant α ∈ (, ), denote by ξ = ξ n,α and z = zα z−µ the quantiles of Fn and Φ ( σ ), respectively. In view of the EE of Fn , one may expect ξ can be approximated by z of the form k−
ξ = z + ∑ η n,j n−j/ + o(n−(k−)/ ),
()
j=
which is called the Cornish–Fisher EE of quantiles. Indeed, the coefficients η n,j can be determined by submitting ()
Effect Modification and Biological Interaction
into the valid EE of Fn , that is, Fn (ξ) = Φ (
ξ−µ ) − ∑k− j= σ
Qn,j (
ξ−µ ξ−µ ) φ ( σ ) + Rn . σ
As an example, for the iid case with µ = and σ = , we give expressions of the first three η j as follows ν ν ν (z − ), η = (z − z) − (z − z) ν ν ν (z − z + ) − (z − z + z) η = ν + (z − z + ). η =
About the Author “Zhidong has a legendary life. He was adopted into a poor peasant family at birth. He spent his childhood during the Chinese resistance war against Japan.” (Advances in statistics: proceedings of the conference in honor of Professor Zhidong Bai on His th Birthday, Eds. Zehua Chen, Jin-ting Zhang, Feifang Hu, World Scientific Publishing Company, , Preface). Zhidong Bai has started his illustrious career as a truck driver’s team leader during the Cultural Revolution in China (–), but he managed to become a Professor of statistics, North East Normal University, China, and Department of Statistics and Applied Probability, National University of Singapore. He is a Fellow of the Institute of Mathematical Statistics (), and Fellow of the Third World Academy of Science (). Dr. Bai was the Editor of the Journal of Multivariate Analysis (–), and Associate editor of the Journal of Statistical Planning and Inference (–) and Statistica Sinica (–). Currently, he is the Associate editor of the Journal of Probability and Statistics and Sankya. He has published seven papers related to Edgeworth expansion (he was the first who proposed the concept of Partial Cramér condition, and published a series of papers in Ann. Statist., JMVA and Sankya to solve a series of problems), and over refereed papers in other areas (including papers with Professor C.R. Rao). He has also (co)authored three books, including Ranked set sampling. Theory and applications (with Chen, Z.H. and Sinha, B.K., Springer-Verlag, New York, ). In , the Conference in Honor of Professor Zhidong Bai on his th Birthday was held at the National University of Singapore. “Professor Bai represents a special generation who experienced the most difficult period in China but never gave up their quest for science and eventually excelled in the world arena. This special issue is not only to honor Professor Bai for his great achievements but also to inspire the young generation to devote themselves to the promotion
E
of Statistics in China as China enters into a great new era.” (Loc. cit.) Zhidong Bai is a professor of the School of Mathematics and Statistics at Northeast Normal University and Department of Statistics and Applied Probability at National University of Singapore. He is a Fellow of the Third World Academy of Sciences and a Fellow of the Institute of Mathematical Statistics.
Cross References 7Approximations for Densities of Sufficient Estimators 7Approximations to Distributions 7Cornish-Fisher Expansions 7Multivariate Statistical Distributions
References and Further Reading Babu GJ, Bai ZD () Edgeworth expansions of a function of sample means under minimal moment conditions and partial Cramèr’s condition. Sankhya Ser A ():– Bai ZD, Rao CR () Edgeworth expansion of a function of sample means. Ann Stat ():– Bai ZD, Zhao LC () Asymptotic expansions of sums of independent random variables. Sci Sin ():– Battacharya RN, Ghosh JK () On the validaty of the formal Edgeworth expansion. Ann Stat ():– Chebyshev PL () Sur deux theoremes relatifs aux probabilites. Acta Math :– Cramér H () On the composition of elementary errors. Skand Aktuarietidskr :–, – Edgeworth FY () The law of error. Trans Camb Philos Soc :– , – Edgeworth FY () On the representatin of a statistical frequency by a series. J R Stat Soc :– Fisher SRA, Cornish EA () The percentile points of distributions having known cumulants. Technometrics :– Hall P () Edgeworth expansion for Student’s t statistic under minimal moment conditions. Ann Probab ():– Petrov VV () Sums of independent random variables (trans from Russian: Brown AA). Springer, New York/Heidelberg
Effect Modification and Biological Interaction Sander Greenland Professor University of California-Los Angeles, Los Angeles, CA, USA
The term effect modification has been applied to two distinct phenomena. For the first phenomenon, effect modification simply means that some chosen measure of effect
E
E
Effect Modification and Biological Interaction
varies across levels of background variables. This phenomenon is thus more precisely termed effect-measure modification, and in the statistics literature is more often termed heterogeneity or “interaction” (Greenland et al. ), although “interaction” is more often used as a synonym for a product term in a regression model (see Interaction). For the second phenomenon, effect modification means that the mechanism of effect differs with background variables, which is known in the biomedical literature as dependent action or (again) “interaction.” The two phenomena are often confused, as reflected by the use of the same terms (“effect modification” and “interaction”) for both. In fact they have only limited points of contact.
Effect-Measure Modification (Heterogeneity of Effect) To make the concepts and distinctions precise, suppose we are studying the effects that changes in a variable X will have on a subsequent variable Y, in the presence of a background variable Z that precedes X and Y. For example, X might be treatment level such as dose or treatment arm, Y might be a health outcome variable such as life expectancy following treatment, and Z might be sex ( = female, = male). To measure effects, write Yx for the outcome one would have if administered treatment level x of X; for example, if X = for active treatment, X = for placebo, then Y is the outcome a subject will have if X = is administered, and Y is the outcome a subject will have if X = is administered. The Yx are often called potential outcomes (see Causation and causal inference). One measure of the effect of changing X from to on the outcome is the difference Y − Y ; for example, if Y were life expectancy, Y − Y would be the change in life expectancy. If this difference varied with sex in a systematic fashion, one could say that the difference was modified by sex, or that there was heterogeneity of the difference across sex. Another common measure of effect is the ratio Y /Y ; if this ratio varied with sex in a systematic fashion, one could say that the ratio was modified by sex. For purely algebraic reasons, two measures may be modified in very different ways by the same variable. Furthermore, if both X and Z affect Y, absence of modification of the difference implies modification of the ratio, and viceversa. For example, suppose for the subjects under study Y = and Y = for all the males, but Y = and Y = for all the females. Then Y − Y = for males but Y − Y = for females, so there is -year modification of the difference measure by sex. But suppose we measured the effects by expectancy ratios Y /Y , instead of differences. Then Y /Y = / = for males and
Y /Y = / = for females as well, so there is no modification of the ratio measure by sex. Consider next an example in which Y = and Y = for males, and Y = and Y = for females. Then Y − Y = for both males and females, so there is no modification of the difference by sex. But Y /Y = / = for males and Y /Y = / = . for females, so there is modification of the ratio by sex. Finally, suppose Y = and Y = for males, and Y = and Y = for females. Then Y − Y = for males and Y − Y = for females, so the Y-difference is smaller among males than among females. But Y /Y = / = for males and Y /Y = / = . for females, so the Y-ratio is larger among males than among females. Thus, modification can be in the opposite direction for different measures of effect.
Biological Interaction The preceding examples show that one should not in general equate the presence or absence of effect-measure modification to the presence or absence of interactions in the biological (mechanistic) sense, because effectmeasure modification depends entirely on what measure one chooses to examine, whereas the mechanism is the same regardless of that choice. Nonetheless, it is possible to formulate mechanisms of action that imply homogeneity (no modification) of a particular measure. For such a mechanism, the observation of heterogeneity in that measure can be taken as evidence against the mechanism (assuming of course that the observations are valid). It would be fallacious however to infer the mechanism is correct if homogeneity was observed, because many other mechanisms (some unimagined) would imply the observation. A classic example is the simple “independent action” model for the effect of X and Z on Y, in which subjects affected by changes in X are disjoint from subjects affected by changes in Z (Greenland and Poole ; Weinberg ). This model implies homogeneity (absence of modification by Z) of the average X effect on Y when that effect is measured by the difference in the average Y. In particular, suppose Y is a disease indicator ( if disease occurs, if not). Then the average of Y is the proportion getting disease (the incidence proportion, often called the risk) and the average Y difference is the risk difference. Thus, in this context, the independent action model implies that the risk difference for the effect of X on Y will be constant across levels of Z; in other words, the risk difference will be homogeneous across Z, or unmodified by Z. If both X and Z have effects, this homogeneity of the difference forces ratio measures of the effect of X on Y
Effect Modification and Biological Interaction
to be heterogeneous across Z. When additional factors are present in the model (such as confounders) homogeneity of the risk differences can also lead to heterogeneity of the excess risk ratios (Greenland a). Thus, under the simple independent action model, the independence of the X and Z effects will cause the measures other than the risk difference to be heterogeneous, or modified, across Z. Biological models for the mechanism of X and Z interactions can lead to other patterns. For example, certain multistage models in which X and Z act at completely separate stages of a multistage mechanism can lead to homogeneity of ratios rather than differences, as well as particular dose–response patterns. Special caution is needed when interpreting observed patterns, however, because converse relations do not hold: Different plausible biological models may imply identical patterns of the effect measures (Moolgavkar ).
E
in Y across levels of X (Greenland and Poole ; Greenland b). The converse is false, however: Homogeneity of the difference measures (e.g., lack of modification of the risk difference) does not imply absence of synergy or antagonism, because such homogeneity can arise through other means (e.g., averaging out of the synergistic and antagonistic effects across the population being examined) (Greenland et al. ; Greenland and Poole ). A more restrictive set of definitions is based on the sufficient-component cause model of causation (Greenland et al. ; Rothman ; VanderWeele and Robins ). Here, for two binary indicators X and Z, synergism of the effects of X = and Z = is defined as the presence of X = and Z = in the same sufficient cause of Y = , i.e., the sufficient cause cannot act without both X = and Z = . Similarly, antagonism of the X = effect by Z = is defined as the presence of X = and Z = in the same sufficient cause of Y = . These definitions are also coding dependent.
Synergism and Antagonism Taking the simple independent-action model as a baseline, one may offer the following dependent-action definitions for an outcome indicator Y as a function of the causal antecedents X and Z. Synergism of X = and Z = in causing Y = is defined as necessity and sufficiency of X = and Z = for causing Y = , i.e., Y = if and only if X = and Z = . We also may say that Y = in a given individual would be a synergistic response to X = and Z = if Y = would have occurred instead if either X = or Z = or both. In potential-outcome notation where Yxz is an individual’s outcome when X = x and Z = z, this definition says synergistic responders have Y = and Y = Y = Y = . Antagonism of X = by Z = in causing Y = is defined as necessity and sufficiency of Z = in order for X = to cause Y = . This definition says antagonistic responders to X or Z have Y = or Y = or both, and Y = Y = . With these definitions, synergism and antagonism are not logically distinct concepts, but depend on the coding of X and Z. For example, switching the labels of “exposed” and “unexposed” for one factor can change apparent synergy to apparent antagonism, and vice-versa (Greenland et al. ; Greenland and Poole ). The only labelinvariant property is whether the effect of X on a given person is altered by the level of Z, i.e., the action of X is dependent on Z. If so, by definition we have biological interaction. Absence of any synergistic or antagonistic interaction among levels of X and Z implies homogeneity (absence of modification by Z) of the average X effect across levels of Z when the X effect is measured by the differences
Extensions to Continuous Outcomes The use of indicators in the above definitions may appear restrictive but is not. For example, to subsume a continuous outcome T such as death time, we may define Yt as the indicator for T ≤ t and apply the above definitions to each Yt . Similar devices can be applied to incorporate continuous exposure variables (Greenland b). The resulting set of indicators is of course unwieldy, and in application has to be simplified by modeling constraints (e.g., proportional hazards for T).
About the Author For biography see the entry 7Confounding and Confounder Control.
Cross References 7Interaction
References and Further Reading Greenland S (a) Additive-risk versus additive relative-risk models. Epidemiology :– Greenland S (b) Basic problems in interaction assessment. Environ Health Perspect (Suppl ):– Greenland S, Poole C () Invariants and noninvariants in the concept of interdependent effects. Scand J Work Environ Health :– Greenland S, Lash TL, Rothman KJ () Concepts of interaction. In: Modern epidemiology (Chapter ), rd edn. Lippincott, Philadelphia, pp – Moolgavkar S () Carcinogenesis modeling: from molecular biology to epidemiology. Annu Rev Public Health :– Rothman KJ () Causes. Am J Epidemiol :–
E
E
Effect Size
VanderWeele TJ, Robins JM () The identification of synergism in the sufficient-component cause framework. Epidemiology :– Weinberg CR () Applicability of simple independent-action model to epidemiologic studies involving two factors and a dichotomous outcome. Am J Epidemiol :–
Effect Size Shlomo Sawilowsky , Jack Sawilowsky , Robert J. Grissom Wayne State University, Detroit, MI, USA San Francisco State University, San Francisco, CA, USA In general, an effect size (ES) measures the extent parameters differ or variables are related. Effect-size methodology is barely out of its infancy and yet the effect size has already been proclaimed as the statistical coin of the realm for the st century. It was primarily due to the diligence of Cohen (, , , ) that the role and importance of the ES has attained great prominence in 7power analysis, statistical analysis, and product and program evaluation. The raw score difference may be a naïve ES measure in the context of a two independent samples layout consisting of a treatment and control group when the intervention is modeled as shift in location. It cannot always be prudently used to assess the magnitude of an intervention, or for that matter even a naturally occurring phenomenon, if arbitrary measures of the dependent variable (e.g., Minnesota Multiphasic Personality Inventory subscale, Tennessee Self Concept Scale) are used as a stand-in for latent variables, as opposed to inherently meaningful dependent variables (e.g., weight in pounds or kilograms, length of hospital stay). The two primary parametric ES families are () d, a standardized difference between group’s means, and () r , a measure of the proportion of variance in a variable that is attributable to the variability of another variable. (The Pearson correlation, r, is an ES measure of bivariate association.) Kirk () listed effect size indices among the d and r families, and the list has grown since then. Cohen’s () d estimator of population d is the difference between two sample means divided by the pooled estimate of the assumed common population standard deviation: XT − XC d= sPooled where X T = mean of the treatment group, X C = mean of the control group, and sPooled is the pooled standard deviation.
In the case of two samples of equal size the point-biserial correlation (rPB ) can be approximately obtained from d: rPBX X = √
d d +
,
where X is a truly dichotomous independent variable (e.g., treatment group membership or gender) and X is a continuous dependent variable. Although his warning to be flexible about such very general designations in various areas of research is widely ignored, Cohen provided rule of thumb descriptors of the magnitude of d: d ≤ . = small, d = . = medium, and d ≥ . = large. Sawilowsky () defined d ≥ . = very large. Glass’ ∆ (Glass et al. ) and Hedges’ g (Hedges and Olkin, ) are indices that are related to d. They are obtained by substituting one group’s standard deviation for the estimate based on the pooled variance to counter heterogeneity of variance (the ∆ES), or substituting each (ni − ) for each ni , in the denominator when pooling variances for a better estimate of the assumed common population variance (the g ES). When sample sizes reach n = n ≈ these three indices tend to converge at the second decimal. A benefit of standardizing the mean difference via any of these three methods is it permits a 7meta-analysis of ESs from different studies that used the same two levels of an independent variable, but different measures of the same latent dependent variable. In terms of association, r is called the sample coefficient of determination. (R , preferred by some textbook authors, refers to a hypothetical population parameter; see King .) A related statistic, radj , is used in multiple regression to adjust for the number of variables in the model. Some caution is necessary in its use because squaring r, which can be positive or negative, yields a directionless r , and limitations inherent in r apply to r (e.g., no causality is implied). To ensure correct interpretation of r , r (with its sign) should also be reported. An approximate value of r can be obtained from t: r =
t
t , + df
()
where t = obtained value of the Student t statistic and df = degrees of freedom. In terms of the two variable general linear model, r = ηˆ SSBetween = , SSTotal
E
Effect Size
where SSBetween = sum of squares between groups and SSTotal = total sum of squares. (ηˆ is a sample estimate of the population η , the proportion of explained variance.) The ηˆ ES and its bias-adjusted counterpart, ω , can be extended to one-way and factorial ANOVA and MANOVA. According to Cohen, a general rule of thumb is r ≤ . = small, . < r ≤ . = medium, and r > . = large. Alternately, for rPB or ηˆ , . = small, . = medium, and . = large. Attacks against hypothesis testing in the last quarter of the th century were accelerated with the emergence of the ES. For a response, see Sawilowsky (a). However, considerable controversy did arise regarding emerging ES reporting requirements in scholarly journals. For example, Thompson (:, :) recommended the ES “can and should be reported and interpreted in all studies, regardless of whether or not statistical tests are reported” (), and “even [for] nonstatistically significant effects” (). This advice was initially, albeit lukewarmly, endorsed by the American Psychological Association (Wilkinson and APA ; APA ). An invited debate on this topic was published in the Journal of Modern Applied Statistical Methods in (see, e.g., Sawilowsky b, c) where it was argued that the ES should only be reported in the presence of a statistically significant hypothesis test. That is, the evidence against a null hypothesis must first be shown to be statistically significant before it makes sense to quantify the ES. Indeed, the typical null hypothesis implies that an ES = . Subsequently, the APA () recanted its support for publishing effect sizes in the absence of a statistically significant test result by at least removing active language supporting Thompson’s recommendation. Thus, the words of Cohen prevailed: “The null hypothesis always means the effect size is zero...[but] when the null hypothesis is false, it is false to some specific degree, i.e., the effect size (ES) is some specific nonzero value in the population” (Cohen :). (In more current usage “nil hypothesis” would substitute for “null hypothesis”.) Nevertheless, this controversy persists. A primary reason is due to a misunderstanding attributable to r being commonly interpreted via r , even though no statistical test has been conducted. For example, suppose rX X = .. Then, r = ., indicating % of the variability in the X scores are attributable to the X variable. Similarly, if rX X = ., then r = ., or %. Using r as the ES measure of the sample data, the X variable is shown to be sixteen (
. = ) times more effective in attributing .
variance in the X scores than the X variable, even though no statistical test has been conducted. Rearranging (), for r ≠ ±, Student’s t is expressed in terms of r as: rX X t= √ . ( − rX X )( + rX X )/df Student’s t can also be expressed in terms of rPB as: t = rPBX X
¿ Á Á À
E
N − , ( − rPBX X )( + rPBX X )
where N = number of scores, and rPBX X ≠ ±. In either case it is erroneously believed that inherent in calculating a Pearson correlation for descriptive purposes a hypothesis test result, Student’s t, has likewise been obtained. This is not correct, because r (or rPB ) is only a measure of co-relationship, whereas t determines the statistical significance of that correlation. The explained amount of variance in the dependent variable, r , is only a function of the size of r, whereas t is obtained based on the size of r as well as the size of the df . “Although t and r are related, they are measuring different things” (Gravetter and Wailnau :). Hence, knowing only the magnitude of r is insufficient to inform statistical significance. The converse is also true. Conducting a Student’s t test informs statistical significance, but does not convey the magnitude of an ES. Also, the result of an experiment might be statistically significant, but the ES might be too small to attain practical significance (e.g., no or insufficient clinical benefit). Additional caution is in order in interpreting the magnitude of an ES. In terms of the explained proportion of variance ES, consider the Abelson () paradox. The r associated with batting average and the outcome of Amer
ican baseball games was found to be ., not quite of %, although it is universally accepted that batting average is the single most important characteristic of a successful baseball player. Therefore, heed Cohen’s (:) warning, “The next time you read that ‘only X% of the variance is accounted for’ remember Abelson’s Paradox.” Similarly, in terms of the strength of an effect in an ANOVA layout, consider the Sawilowsky (a) paradox. The Michelson-Morley interferometer experiment was designed to detect the luminiferous ether. An effect of kilometers/second (km/s) was hypothesized, but obtained results were only –. km/s, with a paltry ES of ηˆ = .. What is commonly referred to as the most famous null result in physics was not km/s as might be imagined. Instead, it was actually more than ,
E
Effect Size
miles/h, a speed that exceeds Earth’s satellite orbital velo
city! The next time you read Eta squared was only “ of X %” remember Sawilowsky’s Paradox. Effect sizes developed to accompany parametric hypothesis tests have parametric assumptions (e.g., homoscedasticity, normality), which must not be overlooked. There have been advances in developing nonparametric ESs to accompany nonparametric tests. For example, see Hedges and Olkin (, ), Kraemer and Andrews (), and Newcombe (). Emphasizing confidence intervals for ESs, Grissom and Kim () discussed traditional and robust parametric ESs for contingency tables with nominal or ordinal outcome categories (phi, risk difference, relative risk, odds ratio, cumulative odds ratio, and generalized odds ratio), and the nonparametric stochastic superiority ES for continuous or ordinal data: Pr(Y > Y ), where Y and Y are randomly sampled scores from two different populations. For further reading see Algina et al. (), Baugh (), Bird (), Cohen (, , , ), Cooper and Findley (), Cortina and Nouri (), Dwyer (), Fern and Monroe (), Fleiss (), Fowler (), Grissom and Kim (), Hedges (), Hedges and Olkin (), Huberty (), Hunter and Schmidt (), Keselman (), Levin and Robinson (), Murray and Myors (), O’Grady (), Olejnik and Algina (, ), Ozer (), Parker (), Preece (), Prentice and Miller (), Richardson (), Ronis (), Rosenthal and Rubin (, ), Rosenthal et al. (), Rosnow and Rosenthal (), Sink and Stroh (), Vaughan and Corballis (), Volker (), and Wilcox and Muska ().
About the Authors Biography of Shlomo Sawilowsky is in 7Frequentist hypothesis testing: A Defence. Jack Sawilowsky, M. Ed., is a doctoral student in Evaluation and Research at Wayne State University. He is an Editorial Assistant for the Journal of Modern Applied Statistical Methods. Dr. Robert J. Grissom is Professor Emeritus and Adjunct Professor, Department of Psychology, San Francisco State University, San Francisco, California, retiring early from teaching to focus on research and writing on effect sizes. He was a cofounder and Coordinator of the Graduate Program in Psychological Research. He received his doctorate in Experimental Psychology from Princeton University. He has authored and co-authored articles in edited books, Science magazine, and in methodological journals, where he is a regular reviewer on effect sizes. He
is the senior author, with Dr. John J. Kim, of Effect Sizes for Research: A Broad Practical Approach (), published by Erlbaum; second edition scheduled for publication by Routledge in .
Cross References 7Power Analysis 7Psychology, Statistics in 7Significance Tests: A Critique 7Statistical Fallacies: Misconceptions, and Myths 7Statistics: Controversies in Practice
References and Further Reading Abelson RP () A variance explanation paradox: When a little is a lot. Psychol Bull :– Algina J, Keselman HJ, Penfield RD () Confidence interval coverage for Cohen’s effect size statistic. Educ Psychol Meas ():– American Psychological Association () Publication manual, th edn. American Psychological Association, Washington, DC American Psychological Association () Publication manual, th edn. American Psychological Association, Washington, DC Baugh F () Correcting effect sizes for score reliability: a reminder that measurement and substantive issues are linked inextricably. Educ Psychol Meas :– Bird KD () Confidence intervals for effect sizes in analysis of variance. Educ Psychol Meas :– Cohen J () The statistical power of abnormal-social psychological research: a review. J Abnorm Soc Psychol :– Cohen J () Statistical power analysis for the behavioral sciences. Academic, San Diego Cohen J () Eta-squared and partial eta-squared in fixed factor ANOVA designs. Educ Psychol Meas :– Cohen J () Statistical power analysis for the behavioral sciences, rev. edn. Academic, San Diego Cohen J () Statistical power analysis for the behavioral sciences, nd edn. Erlbaum, Hillsdale Cohen J () Things I have learned (so far). Am Psychol :– Cohen J () A power primer. Psychol Bull :– Cohen J () The earth is round (p < .). Am Psychol :– Cooper HM, Findley M () Expect effect sizes: estimates for statistical power analysis in social psychology. Personal Soc Psychol Bull :– Cortina JM, Nouri H () Effect sizes for ANOVA designs. Sage, Thousand Oaks Dwyer JH () Analysis of variance and the magnitude of effects: a general approach. Psychol Bull :– Fern FE, Monroe KB () Effect-size estimates: issues and problems in interpretation. J Consum Res :– Fleiss JL () Measures of effect size for categorical data. In: Cooper H, Hedges LV (eds) The handbook of research synthesis. Russell Sage, New York, pp – Fowler RL () A general method for comparing effect magnitudes in ANOVA designs. Educ Psychol Meas :– Glass GV, McGaw B, Smith ML () Meta-analysis in social research. Sage, Thousand Oaks
Eigenvalue, Eigenvector and Eigenspace
Gravetter FJ, Wallnau LB () Essentials of statistics for the behavioral sciences. th edn. Wadsworth/Thompson Learning, Belmont Grissom RJ, Kim JJ () Review of assumptions and problems in the appropriate conceptualization of effect size. Psychol Methods :– Grissom RJ, Kim JJ () Effect sizes for research: A broad practical approach. Erlbaum, Mahwah Hedges L () Estimation of effect size from a series of independent experiments. Psychol Bull :– Hedges L, Olkin I () Nonparametric estimators of effect size in meta-analysis. Psychol Bull :– Hedges L, Olkin I () Statistical methods for meta-analysis. Academic, New York Huberty CJ () A history of effect size indices. Educ Psychol Meas :– Hunter JE, Schmidt FL () Methods of meta-analysis: correcting error and bias in research findings. Sage, Newbury Park Keselman H () A Monte Carlo investigation of three estimates of treatment magnitude: epsilon squared, eta squared, and omega squared. Can Psychol Rev :– King G () How not to lie with statistics: avoiding common mistakes in quantitative political science. Am J Polit Sci (): – Kirk RE () Practical significance: a concept whose time has come. Educ Psychol Meas :– Kraemer HC, Andrews G () A non-parametric technique for meta-analysis effect size calculation. Psychol Bull : – Levin JR, Robinson DH () The trouble with interpreting statistically nonsignificant effect sizes in single-study investigations. J Mod Appl Stat Methods :– Murray LW, Myors B () How significant is a significant difference? Problems with the measurement of magnitude of effect. J Couns Psychol :– Newcombe RG () Confidence intervals for an effect size measure based on the Mann–Whitney statistic. Stat Med (): – O’Grady KE () Measures of explained variance: cautions and limitations. Psychol Bull :– Olejnik S, Algina J () Measures of effect size for comparative studies: application, interpretations, and limitations. Contemp Educ Psychol :– Olejnik S, Algina J () Generalized eta and omega squared statistics: measures of effect size for some common research designs. Psychol Methods :– Ozer DJ () Correlation and the coefficient of determination. Psychol Bull :– Parker S () The “difference of means” may not be the “effect size”. Am Psychol :– Preece PFW () A measure of experimental effect size based on success rates. Educ Psychol Meas :– Prentice DA, Miller DT () When small effects are impressive. Psychol Bull :– Richardson JTE () Measures of effect size. Behav Res Methods: Instrum Comput :– Ronis DL () Comparing the magnitude of effects in ANOVA designs. Educ Psychol Meas :– Rosenthal R, Rubin DB () A simple, general purpose display of magnitude of experimental effect. J Educ Psychol : –
E
Rosenthal R, Rubin DB () The counternull value of an effect size: a new statistic. Psychol Sci :– Rosnow RL, Rosenthal R () Assessing the effect size of outcome research. In: Nezu AM, Nezu CM (eds) Evidence-based outcome research. Oxford University Press, Oxford, pp – Rosenthal R, Rosnow RL, Rubin DB () Contrasts and effect sizes in behavioral research: a correlational approach. Cambridge University Press, United Kingdom Sawilowsky S () Robust and power analysis of the × × ANOVA, rank transformation, random normal scores, and expected normal scores transformation tests. Unpublished doctoral dissertation, University of South Florida Sawilowsky S (a) Deconstructing arguments from the case against hypothesis testing. J Mod Appl Stat Methods (): – Sawilowsky S (b) You think you’ve got trivials? J Mod Appl Stat Methods ():– Sawilowsky S (c) Trivials: the birth, sale, and final production of meta-analysis. J Mod Appl Stat Methods ():– Sink CA, Stroh HR () Practical significance: the use of effect sizes in school counseling research. Prof Sch Couns ():– Thompson B () AERA editorial policies regarding statistical significance testing: three suggested reforms. Educ Res :– Thompson B () Five methodology errors in educational research: a pantheon of statistical significance and other faux pas. In: Thompson B (ed) Advances in social science methodology, vol . Sage, Thousand Oaks, CA, pp – Vaughan GM, Corballis MC () Beyond tests of significance: estimating strength of effects in selected ANOVA designs. Psychol Bull :– Volker MA () Reporting effect size estimates in school psychology research. Psychol Sch ():– Wilcox RR, Muska J () Measuring effect size: a non-parametric analog of ω . Br J Math Stat Psychol :– Wilkinson L, the American Psychological Association () Task force on statistical inference. Statistical methods in psychology journals: guidelines and explanations. Am Psychol : –
Eigenvalue, Eigenvector and Eigenspace Pui Lam Leung Associate Professor, Director The Chinese University of Hong Kong, Hong Kong, China Let A be a n × n matrix and λ be a scalar. The characteristic equation of A is defined as det(A − λIn ) = ,
()
where In is an identity matrix of order n and det denotes the determinant. The values of λ that satisfy Eq. () are called the characteristic roots or eigenvalues of A. In general, there are n eigenvalues of A. If λ i is an eigenvalue, then the matrix
E
E
Eigenvalue, Eigenvector and Eigenspace
(A − λ i In ) is singular and there exist a non-zero vector ν i such that (A − λ i In )ν i = . Equivalently, Aν i = λ i ν i .
()
ν i is called the eigenvector corresponds to the eigenvalue λ i . Geometrically, A represents a linear transformation in n-dimensional space. ν i is the direction remains unchanged or invariant under this transformation. In general, there are exactly n eigenvalues of A. These eigenvalues are not necessarily distinct and may be real or complex. Note that if ν i is the eigenvector corresponds to λ i , then any scalar multiple of ν i is also an eigenvector of A. There may be more than one eigenvector correspond to the same eigenvalue λ i , any linear combination of these eigenvectors will also be an eigenvector. The linear subspace spanned by the set of eigenvectors corresponds to the same eigenvalue together with the zero vector is called an eigenspace. In statistics, we often deal with real symmetric matrices. Many test statistics in Multivariate linear model are functions of eigenvalues and eigenvectors of certain matrices. Some basic results concerning eigenvalues and eigenvectors are listed as follow: . If A is a real symmetric matrix then its eigenvalues are all real. . If A is a n × n matrix, then the eigenvalues of A and its transpose A′ are the same. . If A and B are n×n matrices and A is non-singular, then the eigenvalues of AB and BA are equal. . If λ is an eigenvalue of A, then λ k is an eigenvalue of Ak , where k is any positive integer. . If λ is an eigenvalue of a non-singular matrix A, then λ− is an eigenvalue of A− . . If A is an orthogonal matrix, (i.e., AA′ = I), then all its eigenvalues have absolute value . . If A is symmetric, then A is idempotent (i.e., A = A) if and only if its eigenvalues take values s and s only. . If A is a positive (non-negative) definite matrix, then all its eigenvalues are positive (non-negative). . If A is a real symmetric matrix and λ i and λ j are two distinct eigenvalues of A, then the two corresponding vectors ν i and ν j are orthogonal. . Let A be a n × n real symmetric matrix and H = (ν , . . . , ν n ) be a n × n matrix whose ith column is the eigenvectors ν i of A corresponds to the eigenvalues λ i , then H ′ AH = D = diag(λ , . . . , λ n ), () where D is a diagonal matrix whose diagonal elements are λ i . Furthermore, Eq. () can be rewritten as A = HDH ′ = λ ν ν′ + ⋯ + λ n ν n ν ′n .
()
Equation () is known as the diagonalization of A while Eq. () is known as the spectral decomposition of A.
Extrema of Quadratic Form There is an important theorem which is very useful in Multivariate analysis concerning the minimum and maximum of quadratic form. Theorem Let A be a n × n positive definite matrix has the ordered eigenvalues λ ≥ ⋯ ≥ λ n > and the corresponding eigenvectors are ν , . . . , ν n and c is a n × vector. Then . maxc≠
c′ Ac c′ c
= λ and the maximum is attained at c = ν .
= λ n and the minimum is attained at . minc≠ cc′Ac c c = νn . ′ = λ k+ and the maximum is attained . maxcν ,...,ν k cc′Ac c at c = ν k+ for k = , , . . . , n − . ( cν , . . . , ν k means c is orthogonal to ν , . . . , ν k ). ′
This theorem is important since it provides the theory behind 7principal component analysis and 7canonical correlation analysis. In principal component analysis, we are looking for a linear transformation Y = α ′ X of the original variables X such that the variance of Y is maximum. In canonical correlation analysis, we are looking for a pair of linear transformation U = α ′ X and V = β ′ Y of the original variables X and Y such that the correlation between U and V is maximum.
Software Standard mathematical and statistical packages, such as MATLAB and R, have built-in function to compute eigenvalues and eigenvectors. The eigenvalues are usually output in descending order and the corresponding eigenvectors are normalized to unit length.
Cross References 7Correspondence Analysis 7Multivariate Technique: Robustness 7Principal Component Analysis 7Random Matrix Theory
References and Further Reading Anderson TW () An introduction to multivariate statistical analysis, rd edn. Wiley, New York Graybill FA () Matrices with applications in statistics, nd edn. Wadsworth, Belmont Muirhead RJ () Aspects of multivariate statistical theory. Wiley, New York Rao CR () Linear statistical inference and its applications, nd edn. Wiley, New York
E
Empirical Likelihood Approach to Inference from Sample Survey Data
Empirical Likelihood Approach to Inference from Sample Survey Data J. N. K. Rao Professor Emeritus and Distinguished Research Professor Carleton University, Ottawa, ON, Canada
Introduction Traditional sample survey theory largely focused on descriptive parameters, such as finite population means, totals and quantiles, from samples selected according to efficient probability sampling designs (subject to cost constraints). Associated inferences consist of point estimation, standard errors of estimators and large-sample confidence intervals based on normal approximations. Inferences, based on the known probability distribution induced by the sampling design with the population item values held fixed but unknown, are essentially non-parametric. Standard text books on sampling (e.g., Cochran ) provide excellent accounts of the traditional design-based approach. Attempts were also made to integrate sampling theory with mainstream statistical inference based on likelihood functions, but retaining the non-parametric set up. Here we provide a brief account of some methods for sample survey data, based on non-parametric likelihoods. In particular, we focus on the “empirical likelihood” approach which was first introduced in the context of survey sampling by Hartley and Rao () under the name “scale-load approach.” Twenty years later, Owen () introduced it in the main stream statistical inference, under the name “empirical likelihood.” He developed a unified theory for the standard case of a random sample of independent and identically distributed (IID) variables, and demonstrated several advantages of the approach. In particular, the shape and orientation of empirical likelihood (EL) confidence intervals are determined entirely by the data, and the intervals are range preserving and transformation respecting, unlike the intervals based on normal approximations. An extensive account of EL, including various extensions and applications, is given in the excellent book by Owen ().
Scale-Load Approach Suppose that the finite population U consists of N units labeled i = , . . . , N with associated item values yi . A sample of units, s, is selected from U with specified probability p(s). Godambe () obtained the
non-parametric likelihood function based on the full sample data {(i, yi ), i ∈ s} and showed that it is noninformative in the sense that all possible non-observed values yi , i ∈/ s lead to the same likelihood. This difficulty arises because of the distinctness of labels i associated with the sample data that makes the sample unique. To get around this difficulty, Hartley and Rao () suggested data reduction by ignoring some aspects of the data to make the sample non-unique and in turn make the associated likelihood informative. The reduction of sample data is not unique and it depends on the situation at hand. A basic feature of the Hartley–Rao (HR) approach is a specific representation of the finite population, assuming that the possible values of the variable y is a finite set of scale points y∗ , . . . , y∗T for some finite T. The associated population frequencies or “scale loads” are denoted by N∗ , . . . , NT∗ and the population parameters can be expressed in terms of the scale loads; for example, the population mean Y can be expressed as Y¯ = ∑p∗t y∗t where p∗t = Nt∗ /N and ∑p∗t = t
t
. Under simple random sampling without replacement, the sample labels i may be suppressed from the sample data in the absence of information relating the label to the corresponding item value, and the resulting nonparametric likelihood based on the reduced data {yi , i ∈ s} is given by the multivariate hyper-geometric distribution that depends on the scale loads Nt∗ and the associated sample scale loads n∗t . Thus the likelihood function based on the reduced sample data is informative, unlike the likelihood based on the full sample data. If the sampling fraction n/N is negligible, then the hyper-geometric likelihood may be approximated by the multinomial likelihood with probabilities p∗t and associated sample frequencies n∗t so that the log-likelihood function is given by l(p∗ ) = ∑n∗t log (p∗t ) where the sum is over the observed non-zero sample scale loads. The resulting maximum likelihood estimator (MLE) of the mean Y is the sample mean y = ∑pˆ ∗t y∗t = n− ∑yi , where i∈s
pˆ ∗t = n∗t /n. Hartley and Rao () also considered ML estimation when the population mean X of an auxiliary variable x is known and showed that the MLE is asymptotically equal to the customary linear regression estimator of Y. The Hartley–Rao (HR) approach readily extends to stratified simple random sampling by retaining the strata labels h to reflect known strata differences and then regarding each stratum as a separate population and applying the scale-load approach to each stratum separately. The traditional stratified mean is the MLE under this approach.
Empirical Likelihood: IID Case In the case of IID observations y , . . . , yn from some distribution F(.), Owen () obtained a non-parametric
E
E
Empirical Likelihood Approach to Inference from Sample Survey Data
(or empirical) likelihood function which puts masses pi = Pr(y = yi ) at the sample points yi and the log-EL function is given by l(p) = ∑log(pi ) which is equivalent i
to the HR scale-load log-likelihood function. Maximizing l(p) subject to pi > and ∑pi = gives maximum empirical likelihood (MEL) estimators of pi and the mean µ = E(y) as pˆ i = /n and µˆ = ∑pˆ i yi . Chen and Qin () studied EL in the sampling context, assuming simple random sampling, and considered parameters of the form θ = N − ∑g(yi ) for specified functions g(.) and known i∈U
mixture distributions for the variable y were used because they performed better than the normal approximationbased intervals in terms of coverage. On the other hand, the EL intervals compared favorably to the parametric intervals when the assumed parametric distribution holds, and performed better than the parametric intervals under deviations from the assumed parametric distribution, by providing non-coverage rate below the lower bound on Y closer to the nominal value and also larger lower bound. Typically, government agencies use the lower bound for the collection of excess amounts claimed.
auxiliary information of the form N − ∑w(xi ) = . In the i∈U
special case of g(y) = y and w(x) = x − X, their results are equivalent to those of HR for estimating the mean Y. By letting g(yi ) be the indicator function I(yi ≤ t), the MEL estimator of the population distribution funcˆ tion is obtained as F(t) = ∑pˆ i I(yi ≤ t), where pˆ i is the i∈s
maximum EL estimator of pi obtained by maximizing the log-EL function subject to the above restrictions on the pi and the additional restriction ∑pi w(xi ) = . The estimai∈s
ˆ tor F(t) is non-decreasing and hence can be used to obtain estimators of population quantiles and other functionals, in particular the population median. The EL approach can provide non-parametric confidence intervals on parameters of interest, similar to the classical parametric likelihood ratio intervals. For example, the EL intervals on the mean µ for the IID case are obtained by first deriving the profile EL ratio function R(µ) and then treating r(µ) = − log R(µ) as χ , a chi-squared variable with one degree of freedom, where R(µ) is the maximum of ∏(npi ) subject to the previous restrictions on the pi and the additional constraint ∑pi yi = µ. The ( − α)-level EL interval is given by {µ∣r(µ) ≤ χ (α)} , where χ (α) is the upper α-point of the distribution of χ . This result is based on the fact that r(µ) is approximately distributed as χ in large samples. As noted earlier, the shape and orientation of EL intervals are determined entirely by the data, and the intervals are range preserving and transformation respecting. Moreover, unlike the traditional normal approximation intervals the EL intervals do not require the evaluation of standard errors of estimators in the IID case and are particularly useful if balanced tail error rates are considered desirable. An important application of EL intervals in audit sampling is reported by Chen et al. (). Populations containing many zero item values are encountered in audit sampling, where yi denotes the amount of money owed to a government agency and Y is the average amount of excessive claims. In some previous work on audit sampling, parametric likelihood ratio intervals based on parametric
Pseudo-EL: Complex Surveys The HR scale load approach or the EL approach does not readily extend to general sampling design involving unequal probability sampling. To circumvent this difficulty, a pseudo-EL approach has been proposed (Chen and Sitter ) by regarding the finite population as a random sample from an infinite super-population. This in turn leads to the “census” log-EL function lN (p) = ∑log(pi ) i∈U
which is simply the finite population total of the log(pi ). From standard sampling theory, an unbiased estimator of the finite population total lN (p) is given by ˆl(p) = ∑di log(pi ), where di = /π i is the design weight associi∈s
ated with unit i and π i is the associated probability of inclusion in the sample. The pseudo empirical log-likelihood function is defined as ˆl(p) and then one can proceed as in section “7Scale-Load Approach” to get point estimators of finite population parameters. For example, in the absence of auxiliary information, the pseudo-MEL estimator of the finite population mean Y is given by the well-known ratio −
estimator (∑di ) (∑di yi ). Wu () provided algoi∈s
i∈s
rithms and R codes for computing the pseudo-MEL estimators. Interval estimation, however, runs into difficulties under this approach because the profile pseudo-EL ratio function obtained from the pseudo-EL does not lead to asymptotic χ distribution in large samples. To get around this difficulty, Wu and Rao () proposed an alternative pseudo empirical log-likelihood function. It is given by ˆl mod (p) = n∗ ∑d˜ i log(pi ), where d˜ i = di /∑dk are the i∈s
k∈s
normalized design weights and n∗ is the “effective sample size” defined as the ratio of the sample size n and the estimated design effect; the latter is taken as the ratio of the estimated variance of the point estimator under the specified design and under simple random sampling. This design effect depends on the parameter of interest. For simple random sampling with replacement, ˆl mod (p) reduces to the usual empirical log-likelihood function.
Empirical Processes
The pseudo-MEL estimator under the modified function remains the same as the estimator obtained from the original pseudo-EL function. But the modification leads to a different pseudo-EL ratio function that leads to asymptotic χ distribution and hence the resulting intervals can be used. Wu and Rao () proposed a bootstrap calibration method that bypasses the need of effective sample size and the resulting pseudo-EL intervals are asymptotically valid and performed well in simulation studies. In a recent application, the modified pseudo-EL method, based on the χ approximation, was applied to adaptive cluster sampling which is often used to sample populations that are sparse but clustered (Salehi et al. ). Simulation results showed that the resulting confidence intervals perform well in terms of coverage whereas the traditional normal approximation-based intervals perform poorly in finite samples because the distribution of the estimator under 7adaptive sampling is highly skewed. Various extensions of the EL method for survey data have been reported, including multiple frame surveys and imputation for missing data (see 7Imputation). We refer the reader to a recent review paper (Rao and Wu ) on empirical likelihood methods for survey data.
About the Author Dr. J. N. K. Rao is Distinguished Research Professor at Carleton University, Ottawa, Canada and a long term Consultant to Statistics Canada and a Member of Statistics Canada’s Advisory Committee on Methodology. “He published his first research work in sampling at the age of twenty, four years before he received his PhD in . Very quickly Jon became one of the leading researchers in survey sampling and has remained one of the field’s leaders to this day” (David Bellhouse, J.N.K. Rao: An Appreciation of his work, SSC Annual Meeting, June , Proceedings of the Survey Methods Section, p. ). Professor Rao received the prestigious Waksberg Award for survey methodology and the Gold Medal of Statistical Society of Canada in recognition of “fundamental research achievements, especially in the theory and practice of surveys.” Professor Rao is a Fellow of the American Statistical Association and the Institute of Mathematical Statistics. He was elected as a Fellow of the Royal Society of Canada for “fundamental contributions to survey sampling, from its foundational principles to complex survey design and analysis. These contributions have influenced the way surveys are conducted and analyzed.” His paper with Alastair Scott, published in the Journal of American Statistical Association, was selected as one of landmark papers in the history of survey sampling for the centenary volume of the International Association of
E
Survey Statisticians. Currently, he is Associate editor, Survey Methodology (–). He received Honorary Doctor of Mathematics degree in from the University of Waterloo. Professor Rao is the author of the well known book Small Area Estimation (Wiley-Interscience, ).
Cross References 7Likelihood 7Sample Survey Methods
References and Further Reading Chen J, Qin J () Empirical likelihood estimation for finite population and the effective usage of auxiliary information. Biometrika :– Chen J, Sitter RR () A pseudo empirical likelihood approach to the effective use of auxiliary information in complex surveys. Stat Sin :– Chen J, Chen SY, Rao JNK () Empirical likelihood confidence intervals for the mean of a population containing many zero values. Can J Stat :– Cochran WG () Sampling techniques, rd edn. Wiley, New York Godambe VP () A unified theory of sampling from finite populations. J R Stat Soc Ser B :– Hartley HO, Rao JNK () A new estimation theory for sample surveys. Biometrika :– Owen AB () Empirical likelihood ratio confidence intervals for a single functional. Biometrika :– Owen AB () Empirical likelihood. Chapman and Hall, New York Rao JNK, Wu C () Empirical likelihood methods. In: Pfeffermann D, Rao CR (eds) Handbook of statistics – sample surveys: inference and analysis, vol B. North-Holland, The Netherlands, pp – Salehi MM, Mohammadi M, Rao JNK, Berger YG () Empirical likelihood confidence intervals for adaptive cluster sampling. Environ Ecol Stat :– Wu C () Algorithms and R codes for the pseudo empirical likelihood methods in survey sampling. Surv Methodol : – Wu C, Rao JNK () Pseudo-empirical likelihood ratio confidence intervals. Can J Stat :– Wu C, Rao JNK () Bootstrap procedures for the pseudo empirical likelihood method in sample surveys. Stat Prob Letters :–
Empirical Processes Evarist Giné Professor University of Connecticut, Storrs, CT, USA A basic question in Statistics is how well does the frequency of an event approach its probability when the number of repetitions of an experiment with “statistical regularity”
E
E
Empirical Processes
increases indefinitely, or how well does the average of the values of a function at the observed outcomes approach its expected value. “How well” of course may have different meanings. Empirical process theory has its origin on this question. The counterpart in the “probability model” of averaging the values of a function f at the outcomes of n repetitions of an experiment, is just Pn f = n− ∑ni= f (Xi ), where Xi , the data, are independent, identically distributed random variables with common probability law P. The variables Xi need not be real but may take values in any measurable space (S, S). The random measure Pn = n− ∑ni= δ Xi , where δ x denotes unit mass at x (the Dirac delta) is the “empirical measure,” and Pn f = ∫ fdPn , assigns to each function its average over the values Xi (ω), and in particular to each set A the frequency of the occurrences Xi (ω) ∈ A. This random measure is often understood as a stochastic process indexed by f ∈ F, F being a collection of measurable functions (which can in particular be indicators). The object of empirical process theory is to study the properties of the approximation of Pf by Pn f , uniformly in F, mainly, probabilistic limit theorems for the processes {(Pn − P)f : f ∈ F} and probability estimates of the random quantity ∥Pn − P∥F := sup ∣Pn f − Pf ∣. f ∈F
This program has a long history. It starts with Bernoulli and de Moivre in the early s: they studied the approximation of the probability of an event, PA, by its frequency, Pn A, thus obtaining respectively the Bernoulli law of large numbers and the de Moivre(-Laplace) normal approximation of the binomial law. Next, Glivenko and Cantelli, in the s showed, for Xi real valued, that Fn (x) = Pn (−∞, x] converges uniformly almost surely to the “true” distribution function F(x) = P(−∞, x], and this was the first “uniform” law of large numbers for the empirical measure. Kolmogorov in the √ s gave the limiting distribution of n supx∈R ∣Fn (x) − F(x)∣, and Dvoretzky–Kiefer–Wolfowitz in the s gave an inequality of the right order, that applies for all values of n, for Pr {supx∈R ∣Fn (x) − F(x)∣ > ε}, namely, this is less than or equal to e−nε (where the constant in front of the exponent was determined by Pascal Pascal Massart (), cannot be improved, and neither can the exponent). The theorem of Kolmogorov is a corollary of Donsker’s theorem, who viewed the empirical process as a random function and proved for it a central limit theorem (see 7Central Limit Theorems), that is, a limit theorem for its probability law defined on function space (in this case, the space of right continuous functions with left limits, D(−∞, ∞), with a separable metric on it, the Skorokhod metric, with the law of a Gaussian process (the P-Brownian
bridge) as limit). Then, by the “continuous mapping √ theorem” H( n(Fn − F)) converges in law to H(GF ) for all functionals H continuous on D, and Kolmogorov’s theorem follows by taking H( f ) = supx∈R ∣ f (x)∣. [Actually, Donsker’s theorem is a result of efforts by at least Kolmogorov, Skorokhod, Doob, Donsker and, in several dimensions, Dudley]. This was mostly done in the mid s. Another important area on what we may call the classical empirical process theory is that of the “strong approximation” in a suitable probability space of the empirical process for the uniform distribution (see 7Uniform Distribution in Statistics) on [, ] by Brownian bridges, in the supremum norm (“Hungarian constructions”), very useful e.g., in the study of limit theorem of complicated functions of the empirical process (Komlos et al. ). Other results include laws of the iterated logarithm and large deviation principles (Sanov’s theorem). The book to consult on the classical theory of empirical processes is Shorack and Wellner’s (). The Glivenko–Cantelli, Donsker, and Dvoretzky– Kiefer–Wolfowitz results for the empirical distribution function became the models for the striking generalizations that constitute modern empirical process theory. The ˇ first is the Vapnik–Cervonenkis (V–C) () law of large numbers. These authors gave very sharp combinatorial, non-random conditions on classes of sets C (classes of functions F in a subsequent article in ) for the empirical measure to converge to the probability law of the observations uniformly over the sets in C (or the functions in F) almost surely, and necessary and sufficient random conditions on these classes for uniformity to take place. A little earlier, Blum and DeHardt had given other also very sharp general conditions (bracketing entropy) for the same result to hold over classes of functions in Euclidean space, ˇ see DeHardt (). Vapnik–Cervonenkis also applied their work to pattern recognition (see 7Pattern Recognition, Aspects of and 7Statistical Pattern Recognition Principles). The theory greatly gained in scope and applicability with the work of Dudley (), where Donsker’s central limit theorem (also called invariance principle) was ˇ greatly generalized in the spirit of Vapnik–Cervonenkis and Blum–deHardt by proving central limit theorems for empirical processes that hold uniformly in f ∈ F and by giving concrete meaningful examples of such classes, like classes of sets with differentiable boundaries, or positivity parts of finite dimensional sets of functions; measurability problems had to be overcome, and, more importantly, a relationship had to be provided between the V–C combinatorial quantities and metric entropy, which allowed proving asymptotic equicontinuity
Empirical Processes
conditions (hence, “uniform tightness”) for the empirical process over large classes of functions in a way reminiscent of how Dudley proved his famous entropy bound for 7Gaussian processes. ˇ The articles of Vapnik and Cervonenkis and Dudley just mentioned originated the field of modern empirical processes. We now review some of its main developments. The most general form of the central limit theorem under bracketing conditions was obtained by Ossiander (). The connection between empirical processes and Gaussian and sub-Gaussian processes and empirical processes was made more explicit by means of random sign and Gaussian randomization (Pollard ; Koltchinskii and, more extensively, in Giné and Zinn (), where, moreover, the central limit theorem for classes of sets analogue to the V–C law of large numbers is proved; its conditions were proved later, in , to be necessary by M. Talagrand, who continued some aspects of the work of Giné and Zinn and also gave a “structural” necessary and sufficient condition for the central limit theorem). Talagrand () also has the last word on Glivenko–Cantelli type theorems. The limit theory contains as well laws of the iterated logarithm, theorems on uniform and universal Donsker classes (very convenient for applications since in Statistics one does not know the law of the data; see Dudley ; Giné and Zinn ; Sheehy and Wellner and for the uniform in P Glvienko–Cantelli theorem, Dudley et al. ) and the bootstrap of empirical processes (Giné and Zinn ) among other types of results. Particular mention should be made of exponential inequalities for empirical processes, which may be seen as Bernstein or Prokhorov exponential inequalities uniform over large collections of sums of independent random variables, indexed by functions. For the special but conspicˇ uous Vapnik–Cervonekis classes of functions, Alexander () and Massart () had versions of such inequalities in the mid s, but Talagrand () obtained a striking, completely general such inequality, just in terms of the supremum of ∣f (X)∣ over F and of E supf ∈F ∑ni= f (Xi ). The constants in his inequality are not specific, but in a series of articles, P. Massart, T. Klein and E. Rio, and O. Bousquet obtained the best constants. This inequality has applications in many areas of Statistics and, in this author’s view, is a true landmark. Both, limit theorems and exponential inequalities have beenextendedtoU-processes,notnecessarilyinthegreatest generality. See Arcones and Giné () and Major (). Modern empirical process theory has already had and is having a deep impact in pattern recognition and learning theory, M-estimation, density estimation, and quite generally, asymptotic and non-parametric statistics. This will not be reviewed in this article.
E
The books to consult on modern empirical process theory are those of Pollard (), van der Vaart and Wellner (), Dudley (); for U-processes, de la Peña and Giné (); and for Talagrand’s and other inequalities for empirical processes, Ledoux ().
About the Author Dr. Evarist Giné is a Professor of Mathematics and (by courtesy) of Statistics at the University of Connecticut. He is an Elected member of the International Statistical Institute, a Fellow of the Institute of Mathematical Statistics and a corresponding member of the Institut d’Estudis Catalans. He has authored/coauthored more than articles in Probability and Mathematical Statistics, as well as two books: The Central Limit Theorem for Real and Banach Valued Random Variables (with A. Araujo, Wiley, ), and Decoupling, from Dependence to Independence (with V. de la Peña, Springer, ). He has organized several meetings and special sessions in Probability and Mathematical Statistics, in particular he was the Program Chair of the th World Congress of the Bernoulli Society/IMS Annual Meeting. He has been Associate editor of Annals of Probability, Stochastic Processes and Applications, and Revista Matemática Hispano-Americana, and is presently Associate editor of Bernoulli, Journal of Theoretical Probability, Electronic Journal and Electronic Communications in Probability, TEST and Publicacions Matemàtiques.
Cross References 7Exact Goodness-of-Fit Tests Based on Sufficiency 7Khmaladze Transformation 7Strong Approximations in Probability and Statistics 7Testing Exponentiality of Distribution
References and Further Reading Alexander K () Probability inequalities for empirical processes and a law of the iterated logarithm. Ann Probab :– Arcones A, Giné E () Limit theorems for U-processes. Ann Probab :– DeHardt J () Generalizations of the Glivenko–Cantelli theorem. Ann Math Stat :– de la Peña V, Giné E () Decoupling, from dependence to independence. Springer, New York Donsker MD () Justification and extension of Doob’s heuristic approach to the Kolmogorov–Smirnov theorems. Ann Math Stat :– Dudley RM () Central limit theorems for empirical measures. Ann Probab :– Dudley RM () Universal Donsker classes and metric entropy. Ann Probab :– Dudley RM () Uniform central limit theorems. Cambridge University Press, Cambridge Dudley RM, Giné E, Zinn J () Uniform and universal Glivenko– Cantelli classes. J Theoret Probab :–
E
E
Entropy n
Giné E, Zinn J () Some limit theorems for empirical processes. Ann Probab :– Giné E, Zinn J () Bootstrapping general empirical measures. Ann Probab :– Giné E, Zinn J () Gaussian characterization of uniform Donsker classes of functions. Ann Probab :– Koltchinskii VI () Functional limit theorems and empirical entropy, I and II. Teor Veroyatn Mat Statist :–, bf – Komlós J, Major P, Tusnády G () An approximation of partial sums of independent RV’s and the sample DF. I. Z. Wahrsch Verw Gebiete :– Ledoux M () The concentration of measure phenomenon. American mathematical society, Providence, Rhode Island Major P () An estimate on the supremum of a nice class of stochastic integrals and U-statistics. Probab Theory Relat Fields :– Massart P () Rates of convergence in the central limit theorem for empirical processes. Ann Inst H Poincaré Probab Stat :– Massart P () The tight constant in the Dvoretzky-KieferWolfowitz inequality. Ann Probab :– Ossiander M () A central limit theorem under metric entropy with L bracketing. Ann Probab :– Pollard D () Limit theorem for empirical processes. Z Warsch Verb Gebiete :– Pollard D () Convergence of Stochastic Processes. Springer, New York Sheehy A, Wellner JA () Uniform Donsker classes of functions. Ann Probab :– Shorack G, Wellner J () Empirical Processes with Applications to Statistics. Wiley, New York Talagrand M (a) The Glivenko–Cantelli problem. Ann Probab :– Talagrand M (b) Donsker classes and random geometry. Ann Probab :– Talagrand M () Donsker classes of sets. Probab Theory Relat Fields :– Talagrand M () New concentration inequalities in product spaces. Invent Math :– van der Vaart A, Wellner J () Weak convergence and empirical processes: with applications to statistics. Springer, New York ˇ Vapnik VN, Cervonenkis AJa () On the uniform convergence of relative frequencies of events to their probabilities. Theory Probab Appl :– ˇ Vapnik VN, Cervonenkis AJa () Necessary and sufficient conditions for the convergence of means to their expectations. Theory Probab Appl :–
∑pi = . This distribution may be associated with a set of i=
mutually exclusive and exhaustive events {E , . . . , En } or with a random variable X taking on a set of n discrete values with probabilities pi (i = , . . . , n). For such a distribution, the classical definition of entropy is given by n
H(Pn ) = − ∑ pi log pi
()
i=
and generally referred to as Shannon’s entropy after Shannon (Shannon ). Two different bases are typically used for the logarithm in (). In the field of information theory, in whichtheentropyisoffundamentalimportance,thebase− logarithm is the traditional choice, with the unit of measurement of entropy then being a bit. If the base −e (natural logarithm) is used, which is mathematically more convenient to work with, the unit of measurement is sometimes called a nat. Also, log = is the usual convention for (). Most generally, the entropy in () is considered to be a measure of the amount of randomness in a system or of a set of events or of a random variable. Alternatively, the entropy is considered to express the amount of uncertainty, surprise, or information content of a set of events or of a random variable. In information theory, the pi ’s in () may be those of a set of messages (events) or those of a random variable X taking on the value i for the ith symbol of an alphabet for i = , . . . , n. In thermodynamics and statistical mechanics, where entropy plays an important role, the pi in () is the probability of a system being in the ith quantum state with i = , . . . , n. Besides such fundamentally important applications, the entropy in () has proved to be remarkably versatile as a measure of a variety of attributes in different fields of study, ranging from psychology (e.g., Garner ) to ecology (e.g., Magurran ). The wide range of applications of the entropy formula is due in part to its desirable properties and extensions and partly perhaps because of the intrigue held by the entropy concept.
Properties of H
Entropy Tarald O. Kvålseth Professor Emeritus University of Minnesota, Minneapolis, MN, USA
Introduction and Definition Consider that Pn = (p , . . . , pn ), or {pi }, is a finite discrete probability distribution with pi ≥ for i = , . . . , n and
Some of the most important properties of the entropy function H in () for any Pn = (p , . . . , pn ) may be outlined as follows: (P) H is continuous in all its arguments p , . . . , pn . (P) H is (permutation) symmetric in the pi (i = , . . . , n). (P) H is zero-indifferent (expansible), i.e., H(p , . . . , pn , ) = H(p , . . . , pn ). (P) H(, . . . , , , , . . . , ) = and H(/n, . . . , /n) is strictly increasing in n.
Entropy
(P) H is strictly Schur-concave in Pn and also strictly concave (in the usual or Jensen sense). These properties are all readily apparent from the definition in () and would seem to be reasonable for a measure of randomness. The strict Schur-concavity property ensures that H(Pn ) increases strictly as the pi ’s become increasingly equal (even or uniform). Stated more precisely in terms of majorization theory (Marshall and Olkin ), if the probability distribution Pn = (p , . . . , pn ) is majorized by the distribution Qn = (q , . . . , qn ), denoted as Pn ≺ Qn , then, because of the strict Schur-concavity of H, H(Pn ) ≥ H(Qn ) with strict inequality unless Pn is a permutation of Qn . From the majorization Pn = (/n, . . . , /n) ≺ Pn = (p , . . . , pn ) ≺ Pn = (, . . . , , , , . . . , )
()
together with the strict Schur-concavity of H, it follows that H (Pn )
H (Pn )
≤ H(Pn ) ≤
()
with equalities if, and only if, Pn = Pn or Pn = Pn (Marshall and Olkin : , , ). Another property of H is that of additivity. Thus, consider the case of two statistical experiments with their respective events, or, consider the pair of random variables (X, Y) with the joint probability distribution {pij } and the marginal distributions {pi+ } and {p+j } for X J
and Y, respectively, with pi+ = ∑pij for all i and p+j = I
I
J
j= I
J
∑pij for all j and with ∑∑pij = ∑pi+ = ∑p+j = . Then, i=
i= j=
i=
j=
the following property follows from (): (P) H is additive, i.e., H({pij }) = H({pi+ p+j }) = H({pi+ }) + H({p+j }) () under independence i.e., when the two experiments or when X and Y are independent. For the case when pi+ = /I and p+j = /J for all i and j, () reduces to H ({ }) = H ({ }) + H ({ }) IJ I J
()
which is sometimes referred to as the weak additivity property. Two famous inequalities involving the entropy function H in () can be expressed as follows: (P) For any two probability distributions Pn = (p , . . . , pn ) and Qn = (q , . . . , qn ), n
n
− ∑ pi log pi ≤ − ∑ pi log qi i=
i=
which is referred to as Shannon’s fundamental inequality and is sometimes called the Gibbs’ inequality. (P) As a consequence of (), H({pij }) ≤ H({pi+ }) + H({p+j })
()
which is called the subadditivity property of H and of which () is the only equality case, which occurs under independence when pij = pi+ p+j for all i and j. The entropy in (), as well as other entropy measures, may be derived by using some of these properties as axioms or postulates. Such axiomatic characterization approach starts off with a set of axioms specifying some properties that a measure should possess and then showing that a particular function satisfies those axioms (e.g., Aczél and Daróczy ; Mathai and Rathie ).
Cross-Entropy (Divergence) The inequality in () can also be expressed as n
D(Pn : Qn ) = ∑ pi log ( i=
pi )≥ qi
()
where D(Pn : Qn ) is known as the cross-entropy, but is more commonly called the divergence of Pn from Qn . Specifically, it is called the 7Kullback–Leibler divergence or information (Kullback and Leibler ). Since D(Pn : Qn ) is not symmetric in Pn and Qn , the term directed divergence is sometimes used. It differs from the conditional I
J
i=
j=
entropy defined as ∑pi+ [−∑(pij /pi+ ) log(pij /pi+ )] . To avoid mathematical difficulties with (), the convention log = is used together with the assumption that qi = whenever pi = . From (), D(Pn : Qn ) = if, and only if, Pn = Qn . It can also easily be seen that D(Pn : Qn ) is strictly convex in Pn for any given Qn and strictly convex in Qn for any given Pn . The D(Pn : Qn ) in () and various other measures and generalizations of divergence (or “distance”) are important in statistical inferences (see Pardo for a recent and extensive treatment of this topic). It is recognized from the form of the expression in () that it is related to the likelihood-ratio chi-square statistic G for goodness-of-fit tests. In particular, if the pi ’s are multinomial sample probabilities (proportions) pi = ni /N for i = , . . . , n with n
sample size N = ∑ni and if Π n = (π , . . . , π n ) is the i=
corresponding hypothesized distribution, with mi = Nπ i for i = , . . . , n, then n
()
E
G = ND(Pn : Π n ) = ∑ ni log ( i=
ni ) mi
()
E
E
Entropy
which has the asymptotic 7chi-square distribution under the null hypothesis involving the Π n - distribution and with the proper degrees of freedom (n − if Π n involves no unknown parameters and n − s − if Π n involves s unknown parameters that are efficiently estimated). One particular divergence (cross-entropy) is that from the joint distribution {pij } for the random variables X and Y to their independence distribution {pi+ p+j }, which, from () is given by I
J
D({pij } : {pi+ p+j }) = ∑ ∑ pij log ( i= j=
pij ) pi+ p+j
()
and is known as the mutual information between X and Y, but is also called transinformation. This measure reflects in a sense the statistical dependence between X and Y, with D({pij } : {pi+ p+j }) = if, and only if, X and Y are independent. It can also clearly be expressed in terms of individual entropies from () as D({pij } : {pi+ p+j }) = H({pi+ }) + H({p+j }) − H({pij }) () which can also be seen to be non-negative from Property (P) as expressed in (). In the case when X and Y are continuous random variables with the joint probability density function h(x, y) and the marginal probability density functions f (x) and g(y), the equivalent of () becomes D(h(x, y) : f (x)g(y)) = ∫
∞
−∞
∞
∫
−∞
h(x, y)
h(x, y) dxdy log f (x)g(y) ∞
= −∫
f (x) log f (x)dx
−∞ ∞
+∫ log
−∞
∫
HHCα (Pn ) =
−α −
n
(∑ piα − ) , α > , α ≠
()
i=
of which the entropy in () with base − logarithm is the limiting case as α → . If − α is used as the denominator in (), as is often done, then () with base −e (natural) logarithm would be the limiting case of () as α → . There have been numerous other generalized entropy formulations (e.g., Arndt , Ch. ; Kapur ; Pardo , Ch. ). In order to include the great majority of those formulations within a single parameterized generalization, one needs a four-parameter generalized entropy as introduced by Kvålseth (Kvålseth , ), i.e., β ⎤ ⎡ n ⎥ ⎢⎛ pα ⎞ ⎥ ⎢ ∑ i ⎥ ⎢⎜ i= ⎟ λ ⎢ Hα βδ (Pn ) = λ ⎢⎜ n ⎟ − ⎥ ⎥ ⎟ ⎜ ⎥ ⎢ ⎥ ⎢⎝ ∑piδ ⎠ ⎥ ⎢ i= ⎦ ⎣
< α < ≤ δ, βλ > ; or, ≤ δ ≤ < α, βλ <
(a)
(b)
∞
−∞
h(x, y)
h(x, y) dxdy g(y)
()
provided, of course, that the integrals exist. The first rightside term in () is the continuous analog of the discrete entropy in (). However, this continuous entropy may be arbitrarily large, positive or negative.
Parameterized Generalizations
n log ∑ piα , α > , α ≠ −α i=
with the parameter restriction in (b) necessary for the entropy in (a) to be non-negative and strictly Schurconcave. In order for the expression in (a) to be nonnegative, it is found to be sufficient that (δ − α) βλ > for any real-valued α, β, δ, and λ (using the convention c = for all real c), but (b) is found to be required for strict Schur-concavity. For the particular case when, for instance, λ = λ = [( − α)β]− and δ = , it follows from (a) that n
A number of generalizations of the entropy in () have been proposed by introducing one or more arbitrary realvalued parameters. The best known such generalization is the Rényi entropy of order α (Rényi ) defined by HRα (Pn ) =
with α being an arbitrary parameter. Rényi () defined α ∈ (−∞, ∞), but α > is used in () to avoid mathematical difficulties when some pi = . While he used only the base − logarithm, the natural logarithm could obviously also be used in (). Rényi’s entropy has many of the same properties as Shannon’s entropy in (), including the additivity in property (P). In the limit as α → , and using l’Hôpital’s rule, () reduces to () with base − logarithm, showing that Shannon’s entropy is a special case of Rényi’s entropy. The second early parameterized generalization of Shannon’s entropy was that of Havrda and Charvát () and given by
()
⎛ ∑ piα ⎞ ⎜ i= ⎟ λ λ ⎟,α > log ⎜ Hα (Pn ) = lim Hα β (Pn ) = β→ −α e⎜ n ⎟ p ∑ i ⎝ ⎠ i= () which is Rényi’s entropy of order α for a possibly incomplete n
distribution when ∑ pi ≤ (Rényi ). The equivalent i=
Entropy
form of Shannon’s entropy becomes λ λ H (Pn ) = lim Hα (Pn ) = α→
−
n
∑ pi loge pi
n
∑ pi
()
i=
i= n
for ∑pi ≤ . L’Hôpital’s rule is used for the limits in i=
()–(). If, instead of setting λ = λ = [( − α)β]− , λ = λ = ((−α)β − )− is used in (a), then the special cases in ()–() would involve base − logarithms. Two very simple members of () are the quadratic entropy n
n
n
i=
i=
i=
− ∑pi and the cubic entropy − ∑pi for ∑pi = (Kapur and Kesavan : –). Note that, because n
of the term ∑piδ in (), this general formulation also i= n
includes incomplete distributions (∑pi < ) and hence i=
also the entropy of a single event. Thus, since the generalized entropy, or four-parameter family of entropies, in () is zero-indifferent (see also Property (P)), the entropy of a single event of probability p becomes Hαλ βδ (p, , . . . , ) = λ(p(α−δ)β − ) subject to the parameter restriction in (b) and with two simple particular cases being − p (e.g., when λ = , α = , δ = β = ) and the corresponding odds /p − . Various parameterized generalizations have also been proposed for additive and non-additive directed divergence measures. Arndt (), Kapur (), and Pardo () provide overviews of such generalizations. While such generalized entropy formulations provide interesting mathematical exercises and explorations, the practical utility of such efforts seem to be somewhat limited. Unless the added flexibility of such generalizations prove useful in particular situations, Shannon’s entropy in () and () is the entropy of choice.
About the Author Dr. Tarald O. Kvålseth (Ph.D. in Industrial Engineering and Operations Research, University of California, Berkely, ) is Professor Emeritus in mechanical engineering, industrial engineering, and biomedical engineering at the University of Minnesota, Minneapolis, USA, where he also served as Director of the Industrial Engineering Division. Before joining the University of Minnesota, he was first on the faculty at Georgia Institute of Technology, USA, then at the Norwegian Institute of Technology where he was Head of Industrial Management. He is the author of numerous papers and editor of three books on ergonomics and has been a member of various
E
organizations, including the American Statistical Association, The Mathematical Association of America, the Institute of Ergonomics & Human Factors, and is a Fellow of the American Association for the Advancement of Science. He has served as chair and committee member of several international conferences and been a member of editorial boards and association boards, including Chair of the Nordic Ergonomics Committee and Vice President of the International Ergonomics Association. He is listed in Marquis Who’s Who, including Who’s Who in in Science and Engineering, Who’s Who in America, and Who’s Who in the World.
Cross References 7Bayesian Statistics 7Diversity 7Entropy and Cross Entropy as Diversity and Distance
Measures 7Kullback-Leibler Divergence 7Maximum Entropy Method for Estimation of Missing Data 7Measurement of Uncertainty 7Radon–Nikodým Theorem 7Statistical View of Information Theory
References and Further Reading Aczél J, Daróczy Z () On measures of information and their characterization. Academic, New York Arndt C () Information measures: information and its description in science and engineering. Springer, Berlin Garner WR () Uncertainty and structure as psychological concepts. Wiley, New York Havrda J, Charvát F () Quantification method of classification processes. Kybernetika :– Kapur JN () Measures of information and their applications. Wiley, New York Kapur JN, Kesavan HK () Entropy optimization principles with applications. Academic, San Diego Kullback S, Leibler RA () On information and sufficiency. Ann Math Stat :– Kvålseth TO () On generalized information measures of human performance. Percept Mot Skills :– Kvålseth TO () Correction of a generalized information measure. Percept Mot Skills :– Magurran AE () Measuring biological diversity. Blackwell, Oxford Marshall AW, Olkin I () Inequalities: theory of majorization and its applications. Academic, San Diego Mathai AM, Rathie PN () Basic concepts in information theory and statistics. Wiley Eastern, New Delhi Pardo L () Statisctical inferences based on divergence measures. Chapman & Hall/CRC, Boca Raton R´enyi () On measures of entropy and information. In: Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, Berkeley, vol , pp – Shannon CE () A mathematical theory of communication. Bell Syst Tech J :–, –
E
E
Entropy and Cross Entropy as Diversity and Distance Measures
In this paper, a general discussion of entropy and cross entropy measures, their characterizations, and applications to problems in statistics are given.
Entropy and Cross Entropy as Diversity and Distance Measures C. R. Rao Distinguished Professor Emeritus and Adviser C. R. RAO AIMCS, Hyderabad, India Eberly Professor Emeritus in Statistics Pennsylvania State University, University Park, PA, USA 7 We view this pile of data as an asset to be learned from. The bigger the pile, the better – if you have the tools to analyze it, to synthesize it, and make yourself more and more creative. Britt Mayo
Entropy Functional We state some general postulates governing an entropy function H on P, conceived of as a measure of diversity (or randomness or uncertainty in prediction) as discussed in Rao (a, ). A : H(p) ≥
∀ p ∈ P and H(p) = iff p is degenerate ()
A : H(λp + µq) − λH(p) − µH(q) = J(p, q : λ, µ) ≥ () ∀ p, q ∈ P, λ ≥ , µ ≥ , λ + µ = , and = iff p = q
Introduction Let P be a convex set of probability distributions defined on a measurable space (X , B). The classical definition of entropy as a measure of randomness in p ∈ P is what is known as Shannon entropy n
H(p) = − ∑ pi log pi
()
i=
in the case of a 7multinomial distribution in n classes with class probabilities, p , . . . , pn , and H(p) = − ∫ p log p dv
()
in the case of a continuous distribution. These measures were used by Boltzman () and Gibbs (–) in describing some equilibrium states in thermodynamics and Shannon () in information theory. The function () has been adopted by ecologists as a diversity measure in discussing relative abundance of different species of animals or plants in a locality. Some early references are Pielou (), Patil and Taillie (), and Rao (a, b, c). While H(p) is defined on P, there is another function C(q∣p), defined on P × P, called cross entropy (CE), not necessarily symmetric in p and q, designed to examine how close a surrogate model q is to a true model p. A well-known CE is Kullback and Leibler () divergence measure n pi () C(q∣p) = ∑ pi log qi i= in the discrete case of n class multinomial distributions, and p C(q∣p) = ∫ p log dv () q in the case of continuous distributions.
While A requires H to be a non-negative function, A implies that uncertainty increases if we contaminate p with another member q, i.e., H(p) is a strictly concave function on P. In a paper titled Entropy in this volume, Kvålseth gives a number of postulates governing H in addition to A and A and discusses the consequences of each postulate. One important difference is the additional postulate requiring symmetry of H in () with respect to p , . . . , pn . Some of the consequences of the postulate A as stated in () are as follows: . In the class of multinomial distributions, H(p) attains the maximum when all class probabilities are equal if H is symmetric in p , . . . , pn . . Dalton and Pielou used the following condition in characterizing a diversity measure: H(p , . . . , pi , . . . , pj , . . . , pn ) ≤ H(p , . . . , pi + δ, . . . , pj − δ, . . . , pn ) if pi < pi + δ ≤ pj − δ < pj . This is implied by A if H is symmetric in p , . . . , pn . Some examples of entropy functions in the case of multinomial distributions with p , . . . , pn as class probabilities are: . . . . .
−∑pi log pi , Shannon entropy − ∑pi , Gini-Simpson entropy ( − α)− log ∑piα , α > , α ≠ , Rényi entropy (α − )− ( − ∑piα ) , Havrda and Charvát entropy ∑∑dij pi pj , Rao’s () quadratic entropy where d = . . . = dkk and the (k − ) × (k − ) matrix (dik + djk − dij − dkk ) is non-negative definite.
Entropy and Cross Entropy as Diversity and Distance Measures
Properties of some of these entropy functions are discussed in Kvålseth in this volume. In the continuous case, some of the entropy functions are: .
− ∫ p log p dv, Shannon entropy, which may be negative for some p. For example, if p = for all x in (, /), = −∫
. .
/
− ∫ p log p dv
log dv = − log <
( − α)− log ∫ pα dv, α > , α ≠ , Rényi entropy ∫ k(x, y)p(x)p(y)dvx dvy , Rao’s () quadratic entropy where k is a conditionally negative definite kernel, i.e., n
n
∑ ∑ k(xi , xj )ai aj < i= j=
for any n and x , . . . , xn and a , . . . , an such that ∑ai = . Rao’s quadratic entropy in the case of multinomial distributions is a function of both the class probabilities and possible differences in species in other aspects. It may not be a symmetric function of class probabilities. For a discussion on the use of Rao’s quadratic entropy as an appropriate tool in the ecological studies, reference may be made to papers by Pavoine et al. (), Ricotta and Szeidl () and Zoltán (). It may be noted that Rao’s quadratic entropy reduces to the expression for the variance of the distribution p if k (x, y) = (x − y) . Maxwell and Boltzmann obtained what is known as Maxwell-Boltzmann distribution of elementary particles (such as molecules) by maximizing Shannon entropy function, as in . above, subject to a restriction on p. Rao (, pp. –) obtained the corresponding model by maximizing Rényi’s entropy, as in . above, subject to the same restriction on p. It would be of interest to try other entropy functions and compare different models.
ANODIV R. A. Fisher introduced the method of Analysis of Variance (ANOVA) for partitioning the variance of a set of measurements into several components such as “between” and “within” populations, “first order and higher order interactions” of factors, etc. Can a similar analysis be done with other measures of variability such as the mean deviation in the case of quantitative variables and a measure of diversity in the case of qualitative measurements? For instance we may have n populations of individuals and each individual in a population is scored for one of k possible skin colors. We may ask what is the average diversity of skin
E
color within populations? What is the difference in skin color between populations as a whole? The key to this lies in the choice of a suitable measure of diversity whether the measurements involved are qualitative or quantitative satisfying the postulate A k
H(λ l p + ⋅ ⋅ ⋅ + λ k pk ) − ∑ λ i H(pi ) = J ({pi }, {λ i }) ≥ i=
() where the first term is the diversity of individuals in the mixed population, the second term is average diversity within populations and the difference is attributed to diversity between populations. The function J in () is called Jensen difference of order , and H satisfying () as a diversity measure of order . Using () we have one-way ANODIV (Analysis of Diversity) as in Table . where P. = ∑ λ i pi . In practice, we estimate H(P. ) and H(Pi ) based on observed data and λ i as proportional to sample size ni of observations from the ith population. The ratio G = B/T, has been termed as genetic index of diversity in some population studies to interpret differences between populations as in Lewontin (), Nei (), and Rao (b). It may be noted that G depends on the choice of a suitable measure of diversity relevant to problem under consideration. For some applications reference may be made to Rao (a, b, b). Now, we investigate the condition on H(P) for carrying out a two-way ANODIV. Let us consider populations denoted by Pij , i = , . . . , r and j = , . . . , s, where the first index i refers to a locality and the second index j to a specific community. Further, let λ i µ j be the relative strength of individuals in locality i and community j, where λ i and µ j are all non-negative and ∑ λ i = ∑ µ j = . We may ask: how are the genetic contributions of individuals between localities and between communities as a whole different? Is the magnitude of genetic differences between the communities different in different localities? Such questions concerning the rs populations can be answered by a two-way ANODIV as in Table .
Entropy and Cross Entropy as Diversity and Distance Measures. Table One-way ANODIV for k populations Due to
Diversity
Between populations (B)
J ({pi }, {λi })
Within populations
(W)
∑ λi H(pi )
Total
(T)
H(P. )
E
E
Entropy and Cross Entropy as Diversity and Distance Measures
Entropy and Cross Entropy as Diversity and Distance Measures. Table Two-way ANODIV Due to
functions can be used only for two-way ANODIV but not for higher order ANODIV. In the case of Rao’s quadratic entropy, Jensen differences of all orders are convex, so that ANODIV can be carried out for any order classified data. It is shown by Lau () that an entropy with this property is necessarily Rao’s Quadratic Entropy. Asymptotic distribution of ratios of entries in the ANODIV table, which can be used for tests of hypotheses as in ANOVA, can be obtained by bootstrapping as illustrated in Liu and Rao ().
Diversity r
Localities
(L)
H(P.. ) − ∑ λi H(Pi. ) i= s
Communities
(C)
H(P.. ) − ∑ µj H(P.j ) j=
Interaction
(LC)
* by subtraction
Between populations (B)
H(P.. ) − ∑ λi µj H(Pij )
Within populations
(W)
∑∑ λi µj H(Pij )
Total
(T)
H(P.. )
Entropy Differential Metric
In Table , P.. = ∑∑ λ i µ j Pij , Pi. = ∑ µ j Pij , P.j = ∑ λ i Pij . What are the conditions under which the entries in Table are non-negative? For B, L, and C to be non-negative, the function H(⋅) defined on P should be strictly concave. For the interaction LC to be non-negative (LC) = − [H(P.. ) − λ i H(Pi. )] + ∑ µ j [H(P.j ) − λ i H(Pij )] = −J ({Pi. } : {λ i }) + ∑ µ j J ({Pij } : {λ i }) = J ({Pij } : {λ i µ j }) ≥
()
or J as defined in () on P r is convex. We represent this condition as C on a diversity measure. If C and C hold, we call such a diversity measure as of order . Note the (LC) can also be expressed as
Using Fisher Information matrix, Rao () introduced a quadratic differential metric, known as Fisher-Rao metric, over the parameter space of a family of probability distributions and proposed the geodesic distance (Rao distance) induced by the metric as a measure of dissimilarity between probability distributions. Burbea and Rao (c) introduced ϕ-entropy functional Hϕ (p) = − ∫ ϕ[p(x)]dv
and derived the quadratic differential metric as the Hessian of ϕ, assuming a parametric family of probability densities p(x) = p(x, θ , . . . , θ n ). This opens up a wide variety of differential metrics. For instance, the choice of Shannon entropy with ϕ(p) = p log p
(LC) = − [H(P.. ) − ∑ µ j H(Pij )] + ∑ λ i [H(Pi. ) −∑ µ j H(Pij )] = −J ({P.j } : {µ j }) + ∑ λ i J ({Pij } : {µ j }) () or J as a function on P s is convex. We can recursively define higher order Jensen differences J from J , J from J and so on, and call H(⋅) for which J , J , . . . , Ji− are convex as the ith order diversity measure. With such a measure we can carry out ANODIV for i-way classified data. A diversity measure for which Jensen differences of all orders are convex is called a perfect diversity measure. Burbea and Rao (a, b, c) have shown that Shannon entropy satisfies the conditions C , C , and C , but not C , C , . . . The Havrda and Charvát entropy satisfies C , C , and C for α in the range (, ] when k ≥ and for α in the range [, ] ∪ (, /) when k = and C , C , . . . do not hold except when α = in which case it reduces to Gini-Simpson index. Rényi’s entropy satisfies C , C , and C only for α in (, ). Most of the well known entropy
()
()
in () gives Fisher-Rao metric. The reader is referred to Burbea and Rao (b, c) for the details. Maybank () has written a number of papers on the application of Fisher-Rao metric, Rao distance and the associated Rao Measure. The possibility of using Burbea-Rao metric, which offers wider possibilities may be explored.
Cross Entropy Characterization There are situations where the true probability distribution p is not known but we us a surrogate distribution q for an analysis of data arising from p, or p is known but it is easy to generate observations from q to estimate some quantities such as the probability of large deviations in p, by a technique known in statistics as importance sampling as explained in section “7Some Applications of CE”. See also Rubinstein and Kroese (). In such a case we need to select q from Q, a chosen family of distributions, such that q is close to an optimum distribution, using a suitable measure of closeness. One such measure, called
Entropy and Cross Entropy as Diversity and Distance Measures
cross entropy (CE), used in some problems is KullbackLeibler () measure of divergence p C(q∣p) = ∫ p log dv. q
()
We suggest a few postulates for the choice of CE: B : C(q∣p) ≥ ∀ p, q ∈ P and C(q∣p) = only if p = q. () B : C(q∣λp + µq) ≤ C(q∣p), λ ≥ , µ ≥ , λ + µ = . () The postulate B is a natural requirement as the mixture λp + µq has some component of q which brings q closer to λp + µq. There are several ways of constructing CEs depending on its use in solving a given problem. A loss function approach is as follows: let p be true probability density and we wish to find q such that ∫ l(p(x), q(x))p(x)dx
()
p(x) q(x)
Some Applications of CE A comprehensive account of the use of CE in estimating probability of large deviations and a variety of stochastic and non-stochastic optimization problems is given in Rubinstein and Kroese (). An example of how cross entropy is used is as follows. Suppose the problem is that of estimating γ = Ep [Φ(x)] , the expectation of a function Φ(x) with respect to a given probability distribution p(x). For instance, if we want to find the probability of x ≥ a, as in the problem of large deviations. we can express it as the expectation of the function Ix≥a , where I is the indicator function. A general Monte Carlo technique of estimating γ is to draw a sample x , . . . , xn from p(x) and estimate γ by γˆ = n− ∑ Φ(xi ).
p C(q∣p) = ∫ p log dv. q
p(x) ∫ Φ(x)p(x) dx = ∫ Φ(x) q(x) q(x) dx
[p(x) − q(x)] p(x)q(x)
()
()
()
which has some interesting properties. A general version of CE, known as Csizar generalized measure, is q ∫ l ( p ) pdv where
l ≥ , l() = and l′′ () > .
γ = Eq [Φ(x)
p(x) ] q(x)
()
C(q∣p) = lim
λ→
H(q + λ(p − q)) − H(q) + H(q) − H(p). λ ()
()
The best choice of q which reduces the variance of γˆ to zero is Φ(x)p(x) q∗ (x) = . () γ However, the solution depends on the unknown γ. An alternative is to choose a family of sampling distributions indexed by a number of parameters q(x, θ), θ = (θ , . . . , θ s )
()
and estimate θ by minimizing C[q(x, θ)∣q∗ (x)] with respect to θ. If we are using KL divergence measure, the problem reduces to max ∫ q∗ (x) log q(x, θ)dv θ
Rao and Nayak () gave a general method of deriving a CE from a given entropy function H(p) as
()
we may draw a sample (x′ , . . . , x′n ) from q, known as an importance sampling distribution, and estimate γ by γˆ = n− ∑ Φ (x′i ) p (x′i ) /q (x′i ).
introduced by Rao () leads to (p − q) p pdv = ∫ ∫ q dv − , pq
()
and
The choice l(p(x), q(x)) =
()
Observing that
()
leads to 7Kullback-Leibler divergence measure
The choice of H(p) as Shannon entropy in () leads to (), the Kullback-Leibler divergence measure.
is a minimum, where l is a measure of difference between the likelihoods p(x) and q(x). The choice l(p(x), q(x)) = log
E
()
which can be solved analytically or by maximizing with respect to the stochastic counterpart n
q∗ (xi )
i=
q(xi , θ)
∑
log q(x, θ)
()
E
E
Entropy and Cross Entropy as Diversity and Distance Measures
with respect to θ, where, x , . . . , xn are observations drawn from q(x, θ), choosing a fixed value θ of θ. Use of () as CE reduces the variance of the estimate (). The CE method can also be used to find the maximum or minimum of a mathematical function f (x) defined over a region R, subject to some equality and inequality constraints as in the Travelling Salesman, pattern recognition (see 7Pattern Recognition, Aspects of and 7Statistical Pattern Recognition Principles), and clustering problems. The problem is converted to a stochastic problem by choosing a probability distribution p(x) over the given region R and a constant γ, and applying the algorithm used in the problem of large deviations, Prob [ f (x) > γ]. The probability distribution p(x) and the constant γ are altered step by step till the large deviation probability becomes negligibly small. For details, the reader is referred to Rubinstein and Kroese ().
Epilogue In his paper in this volume on The Future of Statistics, Efron mentions that the nature of statistical methods is constantly changing with the availability of large data sets and increase in computing power. Early development of statistical theory and practice during the first half of the last century was based on parametric models for observed small data sets, and a set of prescribed rules for testing given hypotheses and estimating parameters of the chosen model. A critical discussion of these methods is given in the paper, Has statistics a future? If so, in what form? (with discussion), by Rao (). Efron’s bootstrap (see 7Bootstrap Methods) and new computer intensive methods such as boosting, bagging, CART, Lasso, Lars, projection pursuit, machine learning methods as in training 7neural networks, and random forests, have put statistical methodology in a different perspective without use of specific models for data. To these may be added the CE approach described in this paper, which provides a comprehensive computational algorithm for solving a variety of statistical problems, such as estimation of large deviations (see 7Large Deviations and Applications), pattern recognition and clustering, DNA sequence alignment and non-stochastic optimization problems such as The Travelling Salesman Problem. I would like to mention that the aim of statistical analysis is not just to answer specific questions posed by the customer, but to find hidden patterns in given data, described as 7exploratory data analysis emphasized by Tukey, or 7data mining or bottom-up analysis to use modern terminology. Such an analysis will enable us to frame new hypotheses leading to possible expansion of knowledge.
Some of these hypotheses could be tested with the existing data and some of them may need further acquisition of data. For further discussion on the need of statistics in solving problems of the real world, development of statistical methodology in view of technological advances in data collection, availability of large data sets, and emergence of new disciplines like 7bioinformatics raising new problems for data analysis, and training of statisticians, reference may be made to Efron’s paper in this volume and Rao ().
About the Author Calyampudi Radhakrishna Rao (born on September , , in India) is among the most important statisticians in the past years. He was the eighth in a family of children. Following the usual custom, he was named after the God Krishna, who was also the eighth child of his parents. Rao received MA degree in mathematics from Andhra University () and MA degree in statistics from Calcutta University () with a first class, first rank and a gold medal. He started working in the Indian Statistical Institute (ISI), at Calcutta in . In while giving a course on estimation at the Calcutta University to the senior students of the master’s class a student asked him whether a result similar to the Fisher’s information inequality for the asymptotic variance of a consistent estimate exists in small samples. During the same night Rao discovered the result that is today widely known as Cramér-Rao inequality. He obtained his Ph.D. in from the King’s College, Cambridge University, UK, with R. A. Fisher as his thesis advisor (Fisher requested that Rao choose a research problem for his Ph.D. thesis himself). On his return to India Rao was appointed at the ISI as Professor at the age of . In India he held several important positions, as the Director of the ISI (–), Jawaharlal Nehru Professor and National Professor. He was the founder of Indian Econometric Society and the Indian Society for Medical Statistics. Rao took mandatory retirement at the age of and moved to USA where he was appointed as a professor at the University of Pittsburgh (–). In he moved to the Pennsylvania State University accepting the newly established Chaired Professorship as Eberly Professor of Statistics (–). Since he is Eberly Professor Emeritus of Statistics. Professor Rao has been the President of the International Biometric Society (–), Institute of Mathematical Statistics (–) and International Statistical Institute (–). He has (co-)authored about research papers, books, including Linear Statistical Inference and its Applications (John Wiley ) and Statistics and Truth: Putting Chance to Work (World Scientific Publishing, ), both translated into six languages. Professor Rao
Entropy and Cross Entropy as Diversity and Distance Measures
is the only living statistician whose two single-authored papers (Information and the accuracy attainable in the estimation of statistical parameters and Large Sample Tests of Statistical Hypotheses Concerning Several Parameters with Applications to Problems in Estimation) which had a high impact on the development of statistical theory were included in the book Breakthroughs in Statistics (Eds. S. Kotz and N. L. Johnson, Springer, /). His name is associated with many statistical terms like: Cramér-Rao inequality, Rao-Blackwellization, Rao’s Score Test, FisherRao and Rao Theorems on second order efficiency of an estimator, Rao’s U test, Fisher-Rao metric, and Rao distance. Other technical terms bearing his name appearing in specialized books are Rao’s Quadratic Entropy, Cross Entropy, Rao’s Paradoxes, Rao-Rubin, Lau-Rao, Lau-RaoShanbhag and Kagan-Linnik-Rao theorems on characterization of probability distributions. For his pioneering contributions to statistical theory and applications Professor Rao has won numerous awards and honors. Just to mention a few, he is a Fellow of Royal Society, UK (), a member of the Academy of Arts and Science (), and National Academy of Science (), US, and a member of three National Academies in India. Professor Rao was awarded numerous medals, including: Guy Medal in Silver of the Royal Statistical Society (), Jagdish Chandra Bose Gold Medal (), Samuel S. Wilks Memorial Medal (), Mahalanobis Birth Centenary Gold Medal (), Army Wilks Award (). On June , , he received the National Medal of Science, the highest scientific honor by the United States. Professor Rao has also received other prestigious awards, such as: Padma Vibhushan award (), International Mahalanobis Prize (), and India Science Award (). For his outstanding achievements, C.R. Rao has been honored with the establishment of an institute named after him: C.R. Rao Advanced Institute for Mathematics, Statistics and Computer Science (C.R. Rao AIMSCS), located in the campus of the University of Hyderabad, India (http://aimscs.org). Nine special issues of several leading international journals were published (–) and six international conferences were organized in honor of Professor Rao (–). He was the doctoral thesis advisor for over students who in turn produced more than Ph.D.’s. Professor Rao holds honorary doctorates from universities in countries spanning six continents. “The first half of the century was the golden age of statistical theory, during which our field grew from ad hoc origins, similar to the current state of computer science, into a firmly grounded mathematical science. Men of the intellectual caliber of Fisher, Neyman, Pearson, Hotelling, Wald, Cramér and Rao were needed to bring statistical
E
theory to maturity.” (Bradley Efron (). The Statistical Century, Royal Statistical Society News, , , –.)
Cross References 7Bayesian Statistics 7Diversity 7Entropy 7Kullback-Leibler Divergence 7Maximum Entropy Method for Estimation of Missing
Data 7Measurement of Uncertainty 7Radon–Nikodým Theorem 7Statistical View of Information Theory
References and Further Reading Boltzmann, Ludwig () Further Studies on the Thermal Equilibrium of Gas Molecules (Weitere Studien über das Wärmegleichgewicht unter Gasmolekülen), In: Sitzungsberichte der Akademie der Wissenschaften, Mathematische-Naturwissenschaftliche Klasse (pp. –), Bd. , Dritte Heft, Zweite Abteilung, Vienna, Gerold Burbea J, Rao CR (a) On the convexity of divergence measures based on entropy functions. IEEE Trans Inf Theory :– Burbea J, Rao CR (b) On the convexity of higher order Jensen differences based on entropy functions. IEEE Trans Inf Theory :– Burbea J, Rao CR (c) Entropy differential metric distance and divergence measures in probability spaces: a unified approach. J Multivariate Anal :– Burbea J, Rao CR () Differential metrics in probability spaces. Probab Math Stat :– Efron, B () The future of statistics, this volume Friedman JH, Popescu BE () Importance sampled learning ensembles. Technical Report, Stanford University Gibbs JW (–) On the equilibrium of heterogeneous substances. Connecticut Academy of Sciences. (Reprinted in The Scientific Papers of J. Willard Gibbs. Dover, New York ()) Kullback S, Leibler RA () On information and sufficiency. Ann Math Stat :– Kvålseth TO () Entropy, this volume Lewontin RC () The apportionment of human diversity. Evol Biol :– Liu ZJ, Rao CR () Asymptotic distribution of statistics based on quadratic entropy and bootstrapping. J Stat Plan Infer :– Maybank SJ () The Fisher-Rao metric. Math Today :– Nei M () Analysis of gene diversity in subdivided populations. Proc Nat Acad Sci :– Pavoine S, Ollier S, Pontier D () Measuring diversity with Rao’s quadratic entropy: are any dissimilarities suitable? Theor Popul Biol :– Patil GP, Taillie C () Diversity as a concept and its measurement. J Am Stat Assoc :– Pielou EC () Ecological diversity. Wiley, New York Rao CR () Information and accuracy attainable in the estimation of statistical parameters. Bull Calcutta Math Soc :– Rao CR () Linear statistical inference and its applications. Wiley, New York, pp –
E
E
Environmental Monitoring, Statistics Role in
Rao CR (a) Diversity and dissimilarity coefficients: a unified approach. Theor Popul Biol :– Rao CR (b) Diversity, its measurement, decomposition, apportionment and analysis. Sankhya :– Rao CR (c) Gini-Simpson index of diversity: a characterization, generalization and applications. Utilitas Math :– Rao CR (a) Convexity properties of entropy functions and analysis of diversity. In: Tong YL (ed) Inequalities in statistics and probability, IMS Lecture notes, vol , IMS, Hayward, pp – Rao CR (b) Use of diversity and distance measures in the analysis of qualitative data. In: Vark GN, Howels WW (eds) Multivariate statistical methods in physical anthropology: a review of recent advances and current developments, D. Reidel Pub. Co., Boston, pp – Rao CR () Rao’s axiomatization of diversity measures. In: Kotz S, Johnson NL (eds) Encyclopedia of statistical sciences, vol . Wiley, New York, pp – Rao CR () Has statistics a future? If so in what form? (with discussion). In: Gulati C, Lin Y-X, Mishra S, Rayner J (eds) Advances in statistics, combinatorics and related areas, World Scientific, Singapore, pp – Rao CR () Cross entropy with applications to problems of large deviations, combinatorial optimization and machine learning, University at Buffalo, Lecture delivered at the Bio Stat Dept Rao CR, Nayak T () Cross entropy, dissimilarity measures and quadratic entropy. IEEE T Inform Theory :– Ricotta C, Szeidl L () Towards a unifying approach to diversity measures: bridging the gap between Shannon entropy and Rao’s quadratic entropy. Theor Popul Bio :– Rubinstein RV, Kroese DP () The cross entropy method: a unified approach to combinatorial optimization, monte carlo simulation and machine learning. Springer, New York Shannon CE () A mathematical theory of communication. Bell Syst Tech J :–, – Zoltán BD () Rao’s quadratic entropy as a measure of functional diversity based on multiple traits. J Veg Sci :–
Environmental Monitoring, Statistics Role in Jennifer Brown President of the New Zealand Statistics Association, Head University of Canterbury, Christchurch, New Zealand
Environmental monitoring is conducted to provide information on status, and changes in status, of an environmental system. Often monitoring is associated with an impact, such as a proposed land development activity or rehabilitation of a habitat. At other times, environmental monitoring is conducted to assess the success (or failure) of a new environment management strategy or change in strategy. Monitoring can also be carried out to provide
information on the overall status of a land or water area of special interest in fields such as biodiversity, the welfare of an endangered species or the abundance of a pest species. In all these examples, the common theme is that monitoring is conducted to provide information. This information may be used in reports and articles that are created to bring about change in management. Typically, such information is numerical – a summary statistic, or a set of data – or some type of numerical measure. This is the role of statistics – statistics is the process used to collect and summarize data to provide relevant information for environmental monitoring. Some underlying principles apply to information collected from environmental monitoring. The first is that in any monitoring design, the aims and objectives need to be clearly stated, both in a temporal and a spatial scale. The most successful monitoring programmes have aims and objectives that can be quantified to guide development of the survey design and data analysis (Gilbert ). The survey design for the monitoring programme should specify how information is to be collected. Various survey designs can be used, and the important criterion is that they should provide a sample that is representative of the population and provide information relevant to the survey objective. The population can be considered to be an area of land or water that has fixed and delineated boundaries. A reserve or national park is an example of such a population. Other populations may be a species of interest, e.g., a bird population. In the example of a bird species, the population may be finite, although of unknown size, but the spatial boundaries may be unknown if the birds are highly mobile. In other applications, defining the population may be very difficult. For example, when monitoring the impact of a new industrial development in a rural area, delineating the area beyond which there is unlikely to be an effect may be very difficult. Sample designs that use an element of probability (probability sampling) are recommended so that sample survey theory can be used to estimate sample precision (Thompson ). Examples of designs are simple random sampling, stratified sampling, grid-based sampling and spatially balanced designs (Stevens and Olsen ). Statistics is used when a modeling approach is needed to describe the environmental system or for measuring the size of an environmental effect. Examples of an environmental effect are the changes (measured in some way) in environmental status of an area over time, or difference in environmental quality among sites receiving different management treatments (Manly ).
Equivalence Testing
The very nature of environmental monitoring is that data are collected over time to allow assessment of change. A special type of statistical analysis is used for such data to account for the temporal correlation. Repeated measures analysis and longitudinal analysis are two terms used to describe different analyses for when environmental samples are collected from the same population over time (Manly , Diggle et al. ). A consideration in surveys that are repeated over time is whether the same sites should be visited on each survey occasion or whether new sites should be selected and visited at each occasion. Designs that combine aspects of both are often used, where on each survey occasion, a mix of previous and new sites are surveyed (See, for example, Skalski ). A feature of environmental monitoring that is designed to detect any impact resulting from an event is that information on the environment in the absence of the impact needs to be collected. An environmental impact describes events such as a new industrial development or the implementation of a new management strategy to protect an endangered species. The statistical term for these designs is before/after control/impact (BACI) (Underwood ). In a BACI study, information is collected before the impact both at the control sites and at the site(s) of the future impact. Control sites are those that will not be affected by the impact event. Information is then collected from control sites and from the actual impact sites. This design provides information on the environment in the absence of the impact temporally (i.e., before the impact) and spatially (i.e., the control sites). The BACI statistical analysis considers whether the difference between the control and impact sites increases or decreases after the impact event. The analysis provides a way of quantifying this change in differences. Clearly, for some events, the complete BACI design cannot be used, e.g., unplanned impacts such as oil spills. A growing area of statistics is spatial analysis and the use of geographic information systems (GIS) to map and describe environmental systems. Environmental monitoring can be designed to provide information on geographic changes in spatial patterns and distributions. Often maps and other well-designed graphical displays can be very informative. A final comment must be made about the logistical support for environmental monitoring. Monitoring programmes rely on (and often assume) data being collected accurately in the field. Whether this is true or not depends heavily on whether the field team have appropriate training and support. It is very important to have consistency in data collection protocols to ensure that any observed variation in summary statistics over time is a result of changes in the environmental system and not
E
simply changes in field staff or their ability. Other considerations are that data need to be recorded accurately and stored in an accessible way. Statistics plays a vital role in environmental monitoring because the essential ingredient to monitoring is data. A well planned, designed and executed monitoring programme can provide a wealth of information to guide future management and help ensure we maintain healthy environments.
About the Author Professor Brown is President of the New Zealand Statistical Association, a position she has held since . She is the Head of the Department of Mathematics and Statistics at University of Canterbury. Professor Brown is currently an Associate Editor of Journal of Agricultural, Biological and Environmental Statistics, of the Australian and New Zealand Journal of Statistics, and of the International Journal of Ecological Economics and Statistics.
Cross References 7Geostatistics and Kriging Predictors 7Mathematical and Statistical Modeling of Global
Warming 7Spatial Statistics 7Statistics of Extremes
References and Further Reading Diggle PJ, Heagerty PJ, Liang KY, Zeger SL () Analysis of longitudinal data, nd edn. Oxford University Press, Oxford Gilbert RO () Statistical methods for environmental pollution monitoring. Wiley, New York Manly BFJ () Statistics for environmental science and management, nd edn. Chapman & Hall/CRC Skalski JR () A design for long-term status and trend. J Environ Manage :– Stevens DL Jr, Olsen AR () Spatially balanced sampling of natural resources. J Am Stat Assoc :– Thompson SK () Sampling, nd edn. Wiley, New York Underwood AJ () On beyond BACI: sampling designs that might reliably detect environmental disturbances. Ecol Appl :–
Equivalence Testing Dieter Hauschke Professor University Medical Centre Freiburg, Freiburg, Germany Many clinical trials have the objective of showing equivalence between two treatments, usually a test treatment
E
E
Equivalence Testing
√ under development and an existing reference treatment. In such studies the aim is no longer to detect a difference, as in the case when comparing a new treatment with placebo, but to demonstrate that the two active treatments are equivalent within a priori stipulated acceptance limits. Let YT and YR designate the primary clinical outcome of interest for the test and reference treatment, respectively. A two-sample situation is considered where it is assumed that the outcomes are mutually independent and normally distributed with unknown but common variance σ , YTj ∼ N(µ T , σ ), j = , . . . , n , and YRj ∼ N(µ R , σ ), j = , . . . , n . For equivalence testing it is reasonable to assume that the signs of the corresponding population means µ T and µ R are the same and, without loss of generality, positive. Let the interval (δ , δ ), δ < < δ , denote the pre-specified equivalence range, so that the corresponding test problem can be formulated as follows: H : µ T − µ R ≤ δ or µ T − µ R ≥ δ
vs
H : δ < µ T − µ R < δ .
and hence to the inclusion of the two-sided ( − α)% confidence interval for µ T − µ R in the equivalence range √ + , YT − YR [Y T − Y R − t−α,n +n − σˆ n n √ + t−α,n +n − σˆ + ] ⊂ (δ , δ ). n n In clinical practice the equivalence limits δ and δ are often expressed as fractions of the unknown reference mean µ R ≠ , i.e., δ = f µ R and δ = f µ R , − < f < < f . For example f = −f = −. corresponds to the common ±% criterion. The test problem for equivalence can then be formulated as: µT µT µT ≤ θ or ≥ θ vs H : θ < < θ, H : µR µR µR where (θ , θ ), θ = + f , θ = + f , < θ < < θ , is the corresponding equivalence range for the ratio of the expected means µ T and µ R . The null hypothesis can be rejected in favor of equivalence, if
A split of the above two-sided test problem into two onesided test problems (Schuirmann ) results in H : µ T − µ R ≤ δ
vs
Tθ =
H : µ T − µ R > δ
vs
Tδ =
Y T − Y R − δ > t−α,n +n − and √ σˆ + n
Tδ =
Y T − Y R − δ < −t−α,n +n − , √ σˆ n + n
⎛ ⎞ ∑ (YTj − Y T ) + ∑ (YRj − Y R ) n + n − ⎝ j= ⎠ j= n
is the pooled estimator of σ . This is equivalent to Y T − Y R − t−α,n +n − σˆ
n
Y T − θY R < −t−α,n +n − . √ θ σˆ + n n
Hauschke et al. () have shown that rejection of H by the two tests Tθ and Tθ each at level α is equivalent to inclusion of the ( − α)% confidence interval for µ T /µ R , given by Fieller (), in the equivalence range (θ , θ ), with
[θ l , θ u ] ⊂ (θ , θ ) and Y R > aR ,
n
where Y T and Y R denote the corresponding sample means, t−α,n +n − is the ( − α) quantile of the central Student’s t-distribution with n + n − degrees of freedom and σˆ =
Tθ =
H : µ T − µ R < δ .
According to the intersection–union principle (Berger and Hsu ), H is rejected at significance level α in favor of H if both hypotheses H and H are rejected at significance level α:
Y T − θY R > t−α,n +n − and √ θ ˆσ + n
and H : µ T − µ R ≥ δ
+ < δ , n n
Y T − Y R + t−α,n +n − σˆ
where θl = θu =
n
aT =
YTYR −
√
aR Y T + aT Y R − aT aR
YTYR +
√
Y R − aR
aR Y T + aT Y R − aT aR
Y R − aR σˆ t−α,n +n − , n
,
aR =
,
σˆ t−α,n +n − . n
Note that the condition Y R > aR implies that µ R ≠ √
+ > δ and n n
YR >
∣Y R ∣ σˆ t−α,n +n − ⇔ √ > t−α,n +n − . n σˆ n
Ergodic Theorem
It should be noted that in clinical trials, a significance level of α = . is required for equivalence testing and this refers to the calculation of two-sided % confidence intervals (CPMP ). Hence, equivalence can be concluded at level α = . if the corresponding two one-sided test problems can be rejected each at level α = ..
E
Ergodic Theorem Steven P. Lalley Professor University of Chicago, Chicago, IL, USA
Birkhoff’s Ergodic Theorem About the Author Dr. Dieter Hauschke is a Professor in Biometry, Department of Statistics, University of Dortmund, Germany. Currently, he is working as a scientist at the Institute for Medical Biometry and Medical Informatics, University Medical Center Freiburg, Germany. He was a member of the Advisory Council of the International Biometric Society – German Region (–) and council member of the International Biometric Society (–). He is incoming Head (–), Head (–) of the committee biometry within German Association of Medical Informatics, Biometry and Epidemiology. He is member of the Editorial Board of the International Journal of Clinical Pharmacology and Therapeutics and he was Associate Editor of the Biometrical Journal (–). Since , he is statistical consultant of the Drug Commission of the German Medical Association and of the Treat-NMD Neuromuscular Network. Prof. Hauschke is author or co-author of more than research articles and book-chapters. He is also co-author of the textbook Bioequivalence in Drug Development: Methods and Applications (with V. Steinijans and I. Pigeot, Wiley, ).
Cross Reference 7Clinical Trials: An Overview 7Pharmaceutical Statistics: Bioequivalence 7Significance Testing: An Overview
References and Further Reading Berger RL, Hsu JC () Bioequivalence trials, intersection union tests and equivalence confidence sets. Stat Sci : – Committee for Proprietary Medicinal Products () Points to consider on switching between superiority and non-inferiority. EMEA, London Fieller E () Some problems in interval estimation. J R Statist Soc B :– Hauschke D, Kieser M, Diletti E, Burke M () Sample size determination for proving equivalence based on the ratio of two means for normally distributed data. Stat Med :– Schuirmann DJ () A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. J Pharmacokinet Biopharm : –
Birkhoff ’s theorem (see Birkhoff ) extends the strong law of large numbers to stationary processes. The theorem is most easily formulated in terms of measure-preserving transformations: If (Ω, F, P) is a probability space then a measurable transformation T : Ω → Ω is measurepreserving if EX = EX ○ T for every bounded random variable X defined on Ω. In this case P is said to be Tinvariant. Let T be a measure-preserving transformation of a probability space (Ω, F, P); then a random variable X defined on (Ω, F) is T-invariant if X = X ○ T almost surely, and an event F ∈ F is T-invariant if its indicator function is. The collection of all T-invariant events F ∈ F is a σ-algebra, denoted by I. The measure-preserving transformation T is ergodic if the only bounded, T-invariant random variables are almost surely constant. Denote by T i the ith iterate of T, that is, T i+ = T ○ T i . Birkhoff ’s Theorem. Let T be a measure-preserving transformation of a probability space (Ω, F, P). Then for every integrable random variable X defined on (Ω, F), n i ∑ X ○ T = E(X ∣ I) n→∞ n i= lim
almost surely.
()
If T is ergodic, then E(X∣I) = EX almost surely, so in this case the sample averages converge almost surely to the expectation EX. It is not difficult to see that if T is a measure-preserving transformation of a probability space (Ω, F, P) then for every random variable X defined on (Ω, F) the sequence Xn := X ○ T n is a stationary sequence, that is, the joint distribution of X , X , . . . is the same as that of X , X , . . . . Conversely, if X , X , . . . is a stationary sequence of real random variables defined on some probability space and if µ is the joint distribution of X , X , . . . , viewed as a random element of R∞ , then the forward shift operator σ is a measure-preserving transformation of the probability space (R∞ , B∞ , µ). Birkhoff ’s theorem implies that if this shift is ergodic – in which case the stationary sequence X , X , . . . is said to be ergodic – then the sequence X , X , . . . obeys the strong law of large numbers. Kolmogorov’s − Law implies that if the sequence X , X , . . . consists of i.i.d. random variables then it is ergodic, when viewed as a stationary sequence. Thus, the
E
E
Ergodic Theorem
usual strong law of large numbers for sample averages of independent, identically distributed random variables is a corollary of Birkhoff ’s theorem. In fact, Birkhoff ’s theorem implies much more: If the sequence X , X , . . . is stationary and ergodic, then for any Borel measurable function F : R∞ → R the sequence Yn := F(Xn , Xn+ , . . . ) is stationary and ergodic, and therefore obeys the strong law of large numbers provided the first moment ∣EY ∣ is finite.
Consequences of Birkhoff’s Theorem Ergodic Markov Chains An immediate consequence of Birkhoff ’s theorem is a strong law of large numbers for additive functionals of ergodic Markov chains. A Markov chain (see 7Markov Chains) Xn on a finite or denumerable state space X is said to be ergodic if all states communicate and all states are positive recurrent. Each ergodic Markov chain has a unique stationary probability distribution. If the initial state X is distributed according to the stationary distribution then the sequence X , X , X,... is stationary and ergodic, and so the law of large numbers for real-valued functionals F(Xn ) follows directly. Moreover, even if the Markov chain is started in an initial distribution ν other than the stationary distribution π, the strong law of large numbers must hold, because the probability measure P ν governing the non-stationary chain is absolutely continuous relative to the probability measure P π governing the stationary chain. See Revuz () or Meyn and Tweedie () for similar results concerning Markov chains on non-denumerable state spaces.
Birkhoff’s Theorem and Dynamical Systems Birkhoff ’s theorem has important implications for deterministic dynamical systems, especially those governed by Hamilton’s equations where the Hamiltonian has no explicit time dependence (see Arnold and Avez ). According to a fundamental theorem of Liouville, if {Φ t }t≥ is the phase flow of a Hamiltonian system on the phase space RN then for each t > Lebesgue measure on RN is Φ t -invariant. Lebesgue measure cannot be renormalized to be a probability measure; however, if the level surfaces H = E of the Hamiltonian are compact, as is often the case, then Lebesgue measure induces on each energy surface a Φ t -invariant probability measure, called the Liouville measure. The ergodic hypothesis of Boltzmann asserts that the Liouville measure-preserving transformation Φ t is ergodic. When true this implies, by Birkhoff ’s theorem, that “time averages equal space averages.” To date
the ergodic hypothesis has been established only for some very special Hamiltonian systems.
The Shannon–MacMillan–Breiman Theorem Let {Xn }n∈Z be a two-sided stationary process valued in a finite alphabet A. For each finite sequence x x ⋯xn in A, denote by p(x x ⋯xn ) the probability that Xi = xi for each index i ∈ [, n]. Similarly, denote by p(x ∣x− x− ⋯x−n ) the conditional probability that X = x given that Xi = xi for every −n ≤ i ≤ −. The martingale convergence theorem implies that for almost every sequence x− x− ⋯ (relative to the joint distribution of the process {Xn }n∈Z ) the limit p(x ∣x− x− ⋯ ) := lim p(x ∣x− x− ⋯x−n ) n→∞
()
exists. The Kolmogorov–Sinai entropy of the stationary process {Xn }n∈Z is defined to be h := −E log p(X ∣X−X− ⋯ ).
()
Shannon-MacMillan-Breiman Theorem. If the stationary process {Xn }n∈Z is ergodic then with probability one, lim p(X X ⋯Xn )/n = e−h
n→∞
()
This can be deduced from Birkhoff ’s theorem, which implies directly that with probability one n ∑ log p(Xk ∣Xk− Xk− ⋯ ) = −h. n→∞ n k= lim
()
The Shannon–MacMillan–Breiman theorem is of fundamental importance in information theory, for it implies that entropy limits the “compressibility” of a “message” generated by a stationary, ergodic source. See Shannon and Weaver () and Cover and Thomas () for further information.
Kingman’s Subadditive Ergodic Theorem In the years since Birkhoff ’s paper, dozens of extensions, generalizations, and other progeny of Birkhoff ’s theorem have been discovered. See Krengel () for an excellent review. One of these subsequent extensions has proved to be of singular importance: this is Kingman’s subadditive ergodic theorem (see Kingman ). Let T be an ergodic, measure-preserving transformation of a probability space (Ω, F, P). A double array {Sk,m }≤k≤m of real random variables defined on (Ω, F) is called a subadditive process relative to T if Sk,m ○ T = Sk+,m+ , Sk,n ≤ Sk,m + Sm,n ,
()
and
()
γ := inf n ES,n > −∞.
()
−
n≥
Erlang’s Formulas i Observe that if Sk,m = ∑m i=k+ X ○ T where X satisfies the hypotheses of Birkhoff ’s theorem then {Sk,m } is a subadditive process.
Kingman’s Theorem. If {Sk,m } is a subadditive process relative to an ergodic, measure-preserving transformation T of a probability space (Ω, F, P), then lim S,n /n = γ
n→∞
µ − almost surely.
()
Kingman’s theorem has myriad uses in the study of percolation processes and interacting particle systems. See Liggett () for some of the basic applications. Subadditive processes also arise naturally in connection with random walks (see 7Random Walk) on groups and semigroups. A right random walk on a semigroup G started at the identity X = is the sequence Xn of partial products of a sequence ξ , ξ , . . . of i.i.d. G-valued random variables, with multiplication on the right: Xn = ξ ξ ⋯ξ n .
()
The special case where G is a matrix group or semigroup is of particular importance. Let d be an invariant metric on G, that is, a metric such that d(x, y) = d(xz, yz) for all x, y, z ∈ G. For instance, if G is a discrete, finitely generated group then one might choose d to be the natural distance in the Cayley graph; if G is a matrix group then one might take d to be the distance induced by the Riemannian metric on G. In any case, the process Sk,m := d(ξ k+ ξ k+ ⋯ξ m , ) is subadditive, and so Kingman’s theorem implies that d(Xn , )/n converges almost surely to a constant γ (possibly +∞). Similarly, if G = Md is the semigroup of all d × d real matrices and ∥g∥ denotes the usual matrix norm of a matrix g ∈ G, then
E
de Paris VI. He is a former Editor of the Annals of Probability, and is a Fellow of the Institute of Mathematical Statistics.
Cross References 7Almost Sure Convergence of Random Variables 7Glivenko-Cantelli Theorems 7Random Matrix Theory 7Random Walk
References and Further Reading Arnold VI, Avez A () Problèmes ergodiques de la mécanique classique. Monographies Internationales de Mathématiques Modernes, . Éditeur. Gauthier-Villars, Paris Birkhoff GD () Proof of the ergodic theorem. Proc Natl Acad Sci USA :– Bougerol P, Lacroix J () Products of random matrices with applications to Schrödinger operators. Progress in probability and statistics, vol . Birkhäuser Boston, Boston Cover TM, Thomas JA () Elements of information theory, nd edn. Wiley-Interscience, Hoboken Furstenberg H, Kesten H () Products of random matrices. Ann Math Stat :– Kingman JFC () Subadditive ergodic theory. Ann Probab :– . With discussion by Burkholder DL, Daryl Daley, Kesten H, Ney P, Frank Spitzer and Hammersley JM, and a reply by the author Krengel U () Ergodic theorems. de Gruyter studies in mathematics, vol . Walter de Gruyter, Berlin. With a supplement by Antoine Brunel Liggett TM () Interacting particle systems. Grundlehren der Mathematischen Wissenschaften [Fundamental principles of mathematical sciences], vol . Springer, New York Meyn S, Tweedie RL () Markov chains and stochastic stability, nd edn. Cambridge University Press, Cambridge. With a prologue by Peter W. Glynn
Sk,m := log ∥ξ k+ ξ k+ ⋯ξ m ∥ is subadditive, and so Kingman’s theorem implies that if log ∥ξ ∥ has finite first moment then lim ∥Xn ∥/n = eγ
n→∞
()
almost surely. This is the celebrated Furstenberg–Kesten theorem (Furstenberg and Kesten ) of random matrix theory. See Bougerol and Lacroix () for further development of the theory of random matrix products.
About the Author Steven Lalley is a Professor of Statistics and Mathematics at the University of Chicago. He has also held positions at Columbia University, Purdue University, and l’Universite
Erlang’s Formulas M. F. Ramalhoto Professor Technical University of Lisbon, Lisbon, Portugal Mathematical calculations on probabilities have been put in writing mainly since the seventeenth century with the work of mathematicians like Fermat (–), Pascal (–), Huygens (–), Bernoulli (–), Moivre (–), Laplace (–), Gauss (– ), Poisson (–), and Tchébychev (–). In the twentieth century the work of Markov, Liapounov,
E
E
Erlang’s Formulas
Khintchine, Kolmogorov, Palm, Wiener, Fisher, Doob, Neumann, Cramer, Takács, Itó, and Santaló, among many others, on probability foundations, 7stochastic processes, and mathematical statistics provided the basis for what it is called here “Stochastics.” There are two stochastic processes that are fundamental, and occur over and over again, often in surprising ways. The deepest results in “Stochastics” seem to be concerned with their interplay. One, the Wiener process (the Wiener model of Brownian motion, see 7Brownian Motion and Diffusions), has been the subject of many books. The other, the Poisson process (the simplest renewal process, i.e., exponential inter-renewal times), has been comparatively a bit more neglected. Kingman () redresses the balance and provides an enjoyable and clearly written introduction to the structure and properties of 7Poisson processes in one and more dimensions (the book simultaneously addresses the beginner and the expert). When, in (June ) in Boston, Alexander Graham Bell accidentally discovered the possibility of telephone communication he could hardly foresee that by this discovery he had also created the inspiration for 7queueing theory. In fact, the pioneer work on queueing theory is in Erlang (). A. K. Erlang was a Danish engineer and mathematician dealing with the problems and worries of the telephone communication system of the time, whose fundamental papers appeared between and . He was responsible for the concept of statistical equilibrium, for the introduction of the so-called balance-of-state equations, and for the first consideration of the optimization of a queueing system. Rapid progress was made when the use of queueing theory spread out to many other fields in addition to telephone theory, and mathematicians like William Feller and David Kendall, among many others, became interested in the mathematics of queues. It would be difficult to give a brief outline of the development of the subject with a proper assignment of credits. Many of the most meritorious papers responsible for new methods are now rarely mentioned in the literature, in spite of the progress that they initiated; the D. V. Lindley’s integral equation of the single-server queueing system, published in (), is an example. Queueing theory (including queueing networks and complex particle systems) is one of the most relevant branches of “Stochastics.” Indeed, some of the today’s most significant problems can be reduced to resource allocation and resource sharing. When the customers are human beings, very often patterns of human behavior and responses have to be brought into the mathematical analysis of the corresponding queueing systems. For instance, Little’s law, a celebrated queueing property, does not necessarily hold due to the non-linear human’s
perception of waiting. Particularly in the queueing systems of the service industry, issues linked to quality improvement and innovation are now more than ever very relevant; an introduction to these issues can be founded in Ramalhoto (). The power of the structure and the mathematical properties of the Poisson process (a few key results often produce surprising consequences) have been recognized in the queueing theory since its foundation in Erlang ().
First Erlang Formula In , A.K. Erlang (who also laid the foundations of modern teletraffic theory) from the analysis of the concept of statistical equilibrium obtained his famous formula (one of the most used formulas in practice even today) for the loss probability of the M/G/r/r queueing system (i.e., Poisson arrival process, arbitrary service time distribution, r servers and zero waiting positions) in steady state: −
r
B(r, rρ) = [(rρ)r /r!] [∑(rρ)i /i!] ; r ∈ N, ρ ∈ R+ () i=
where N and R+ represent the natural numbers and the positive real numbers, respectively. Here, rρ = λE[S] is the offered load, ρ is the intensity of traffic, and E[S] = µ − is the mean service time. The problem considered by Erlang can be phrased as follows. Calls arrive at a link as a Poisson process of rate λ. The link comprises r circuits, and a call is blocked and lost if all r circuits are occupied. Otherwise, the call is accepted and occupies a single circuit for the holding period of the call. Call holding periods are independent of each other and of arrival times, and are identically distributed with unit mean. This formula gives the proportion of calls that are lost in the M/G/r/r queueing system in steady state. It is called the st Erlang formula, the Erlang’s loss formula, or Erlang B formula. In several telecommunication studies the need arose to extend the definition of the st Erlang formula to non-integral values of r and to evaluate the derivatives of B(r, rρ) with respect to r and ρ. The most commonly used extension of the st Erlang formula is the analytic continuation B(r, a)− = ∫
+∞
e−u ( − u/a)r du
()
where a = rρ (offered load). This is known as the “continued st Erlang function.” Jagerman () presents improved computations of the st Erlang formula and its derivatives, with practical computer programs for immediate application.
Erlang’s Formulas
Second Erlang Formula
r−
C(r, rρ) = [(rρ)r /r!( − ρ)] [ ∑(rρ)i /i! i= −
+ (rρ) /r!( − ρ)] ; r ∈ N, < ρ < . () r
It gives the proportion of customers (calls) that are delayed in the M/M/r queueing system in steady state. Unlike the st Erlang formula, the nd Erlang formula is not valid for an arbitrary service time distribution. From () and (), it is clear that the st and nd Erlang formulas are related as follows: −
C(r, rρ) = [ρ + ( − ρ)B− (r, rρ)] ; r ∈ N, < ρ < . () Further results and generalizations of the st and nd Erlang formulas may be found, for example, in the proceedings of international telecommunication and teletraffic conferences.
Third Erlang Formula Let Cd (r, rρ) mean the probability that an arbitrary customer on its arrival finds r or more customers in the M/M/r/r + d queueing system (i.e., Poisson arrivals, exponential service time distribution, r servers and d waiting positions) in steady state. That is to say, Cd (r, rρ) is the stationary probability that an arriving customer is blocked (i.e., not immediately served). It gives the proportion of customers that are delayed or lost in the M/M/r/r + d queueing system in steady state. It is easy to show that Cd (r, rρ) can be written as follows: d
i
i= d
−
[∑(rρ) /i! + ((rρ)r /r!) ∑ ρ i ] ; i=
i
r ∈ N, d ∈ N , ρ ∈ R+ .
− − ⎤− ⎡ d d ⎥ ⎢ − i i ⎥ ; B (r, rρ) ( ρ ) + − ( Cd (r, rρ) = ⎢ ρ ) ∑ ∑ ⎥ ⎢ ⎥ ⎢ i= i= ⎦ ⎣ + r ∈ N, d ∈ N , ρ ∈ R . ()
The relation () shows the ability of the rd Erlang formula to make use of the results available for the st Erlang formula; namely through the relation () it may be extended to non-integral values of r and d (i.e., the “continued rd Erlang function”). Monotonicity and convexity properties of the rd Erlang formula in terms of its parameters r, d, ρ, and rρ (offered load) are also easy to obtain. All these properties are useful in the study of the probabilistic behavior of the M/M/r/r + d queueing system (also called Markovian multi-server finite-capacity queue or multi-server Erlang loss-delay queue). Mainly because most of the relevant system’s characteristics can be rewritten as very simple functions of the rd Erlang formula. For instance, it is easy to show that in steady state both the loss probability and the delay probability depend on r only through the rd Erlang formula.
Exact Decomposition Formulas for the Multi-Server Erlang Loss-Delay Queue For the M/M/r/r + d queueing system in steady state each ′ random variable (r. v.) Nr,d (the number of customers waiting in the queue), NSr,d , (the number of occupied servers), Nr,d (the number of customers in the system; waiting or ′ being served), Tr,d (the waiting time in the queue), and Tr,d (the total sojourn time in the system; waiting or being served) is distributed as the sum of two r. v.’s. weighted by the rd Erlang formula, for r ∈ N, d ∈ N (where the symbol ∼ means “distributed as”). ′ ∼ ( − Cd (r, rρ))O + Cd (r, rρ)N,d− , Nr,d
()
where P( = o) = (i.e., a degenerated r. v.) and N,d− is the number of customers in the M/M//+(d−) queueing system in steady state with the same ρ (which has a truncated geometric distribution with parameters d and − ρ, denoted here by the symbol G(d, − ρ)).
Cd (r, rρ) = [((rρ) /r!) ∑ ρ ] × r−
be rewritten in terms of the st Erlang formula as follows:
The nd Erlang formula (also called the Erlang’s delay formula or Erlang C formula), first published by Erlang in , is defined in the M/M/r queueing system (i.e., Poisson arrivals, exponential service time, r servers and infinite waiting positions) in steady state as follows:
r
E
i=
()
From () and () it is clear that when d = , Cd (r, rρ) coincides with B(r, rρ), the st Erlang formula. From () and () it is clear that the limit of Cd (r, rρ) as d goes to infinity is the nd Erlang formula (which is defined only for < ρ < ). Therefore, it is reasonable to call Cd (r, rρ) the rd Erlang formula (or Erlang’s loss-delay formula). From () and () it is clear that the rd Erlang formula can
NSr,d ∼ ( − Cd (r, rρ))(X∣X < r) + Cd (r, rρ)R, () where P(R = r) = and X is a Poisson r. v. of parameter rρ. Nr,d ∼ ( − Cd (r, rρ))(X∣X < r) + Cd (r, rρ)(R + N,d− ), () ′ Tr,d ∼ ( − Cd− (r, rρ))O + Cd− (r, rρ)Ed ,
()
E
E
Erlang’s Formulas
where Ed is a “generalized” Erlang r. v. with parameters G(d − , − ρ) and µ, respectively. Tr,d ∼ ( − Cd− (r, rρ))Sr + Cd− (r, rρ)(Ed + Sr ),
()
where Sr is the service time distribution of each server. The proof is essentially based on rewriting the respective distributions (that can be found in Gross and Harris (, pp. -) or Gross et al. ()) in terms of the rd Erlang formula. Therefore, in the steady state, the probabilistic behavior of the M/M/r/r + d queueing system can be obtained in terms of the corresponding probabilistic behavior of the M/M/r/r queueing system and of the M/M// + (d − ) queueing system, respectively; and its closeness to each one of these two queueing systems is measured by the rd Erlang formula. Taking limits as d goes to infinity in () to () (for < ρ < ) similar results follow for the M/M/r queueing system, in the steady state, in terms of the M/M/r/r queueing system, the M/M/ queueing systems and the nd Erlang formula, respectively.
Further Models Due to their analytical tractability the queueing systems M/M/∞, M/M/, and M/M// + d are by far the most extensively studied queueing systems in the steady state as well as in the time-dependent regime. Moreover, the use of the M/M/∞ queueing system to approximate the M/M/r/r queueing system (and their generalized cases for arbitrary service time distributions) has been studied by several authors, see, for instance, Ramalhoto () and the references therein. An extension of the decomposition formulas for the M/M/r/r + d queueing system with constant retrial rate is provided in Ramalhoto and Gomez-Corral (). This type of queueing system (which, in fact, is a queueing network) is characterized by the feature that if on its arrival the customer finds all servers and waiting positions occupied the customer leaves the service area and enters an orbit, i.e., the retrial group. In the case r + d < , analytical results were obtained exactly in the same way, as before, by rewriting the corresponding distribution function in terms of the probability of entering the orbit. It is shown that the distributions of the “number of customers in orbit” (i.e., the orbit size) and the “total number of servers and waiting positions occupied” can be expressed also as mixtures of two r. v.’s weighted by the probability of entering the orbit (which in the retrial case corresponds to the rd Erlang formula). Another relevant property of these novel formulas is that they display the exact influence of the M/M/r/r queue (which is well approximated by the M/M/∞) and the probability of entering the orbit (in the retrial case) which somehow captures the effect of traffic intensity. Because the
physics of the decomposition structures presented do not seem to have much to do with the values of r and d themselves empirical decomposition formulas (of similar kind of the r + d < case) were proposed and heuristically explored. Based on them, numerical results for the stationary distribution of the orbit size were obtained for the queueing systems M/M// + , M/M//, M/M// + , and M/M// with constant retrial, respectively. A similar methodology may be applicable to more complex types of retrials The idea behind this decomposition concept is intuitive and simple to understand and use. And it seems to be suitable for obtaining bounds and other approximations for time-dependent regime and non-Markovian queueing systems as well as for new simulation strategies in queueing systems.
About the Author M. F. Ramalhoto (Maria Fernanda Neto Ramalhoto) is Professor of Probability and Statistics, Department of Mathematics, at Instituto Superior Técnico (IST) of the Technical University of Lisbon since . She was ViceRector of the Portuguese Open University (–). She was a Member of the Executive Council of IST (– ) and the ERASMUS Institutional Coordinator at IST (–). She is a Fellow of the “Société Européenne pour la Formation des Engénieurs”, and Associate Editor of the European Journal for Engineering Education. She was Visiting Professor in several universities worldwide, including MIT, Berkeley and Princeton, and Programme Committee Member of several international conferences in Europe, Israel and India. She was a Co-Founder and a Vice-President of the European Network for Business and Industrial Statistics (which now has over a thousand members and a successful annual conference). She was Expert/Evaluator for the Fifth and Sixth Frameworks of the European Union. She is Expert for the Seventh Framework. She is author and co-author (with colleagues from the US, UK, Germany, Italy, and Spain) of research papers in stochastic processes, statistics, quality control, and reliability. She has been Associate Editor of the international journal Quality Technology and Quantitative Management since its foundation.
Cross References 7Queueing Theory
References and Further Reading Erlang AK () The theory of probabilities and telephone conversations. Nyt Tidsskrift Matematik B :–
Estimation
Erlang AK () Solution of some problems in the theory of probabilities of significance in automatic telephone exchanges. Electroteknikeren (Danish) :– (English translation in the PO Elec Eng J :– (–)) Gross D, Harris CM () Fundamentals of queueing theory, nd edn. Wiley, New York Gross D, Shortle JF, Thompson JM, Harris CM () Fundamentals of queueing theory, th edn. Wiley, New York Jagerman DL () Methods in traffic calculations. AT&T Bell Labs Tech J :– Kingman JFC () Poisson processes. Oxford studies in probability . Clarendon Press, Oxford Lindley DV () The theory of queues with a single server. Proc Camb Philos Soc :– Ramalhoto MF () The infinite server queue and heuristic approximations to the multi-server with and without retrials. TOP :– (December) Ramalhoto MF () Stochastic modelling for quality improvement in processes. In: Park SH, Geoffrey Vining G (eds) Statistical process monitoring and optimization. Marcel Dekker, New York (Chapter ), pp – Ramalhoto MF, Gomez-Corral A () Some decomposition formulae for M/M/r/r + d queues with constant retrial rate. Commun Stat-Stoch Models ():–
Estimation Nancy Reid Professor University of Toronto, Toronto, ON, Canada
Introduction Statistical models involve unknown quantities, usually called parameters of the model, and inference about these parameters provides, in principle, understanding about plausible data-generating mechanisms. Inference about these parameters is also a first step in assessing the adequacy of the proposed model, in predicting future patterns likely to be observed from this model, and as a basis for making decisions. Estimation is the process of using the data to make conclusions about the values of unknown quantities in a statistical model. The theory of point estimation is concerned with a narrower problem: given a parametric model f (y; θ), that is a density on a sample space Y, with unknown parameter θ taking values in a parameter space Θ, and a set of observations from this model, how do we use these observations to provide a good guess for the true value of θ? In 7nonparametric estimation, Θ is an infinite-dimensional space, for example the space of all distribution functions on R; in parametric estimation Θ is usually a subspace of
E
a well-behaved parameter space, such as Rp for fixed and known dimension p. A point estimate of a parameter or a function θ is not often very useful without an associated statement about the accuracy of the point estimate, usually provided as an estimated standard error. More formally, the theory of interval estimation aims to estimate a set of plausible values of θ consistent with the data. In the early development of statistical science, methods for constructing a point estimate were often developed in the context of particular problems, and some statistical properties of the resulting point estimate were then studied. This led to a number of strategies for point estimation, including the method of moments, the method of 7least squares, and, with Fisher (), the method of maximum likelihood estimation; see Aldridge (). Least squares estimates were used in the early th century in problems in astronomy (Stigler, ). The method of moments was very popular in the early th century, but except in very complex problems has been superseded by maximum likelihood estimation. A method for estimation is evaluated by assessment of its statistical properties. Historically, there developed a very large literature on deciding which properties of an estimation method were most relevant, and which methods of estimation satisfied those properties. Some properties commonly proposed for “good” estimators are: unbiased, admissible, minimax, minimum variance, minimum mean-squared error, equivariant, consistent, efficient, asymptotically unbiased, asymptotically optimal, and robust. In the s and s statistical theory emphasized optimality of various methods of point estimation according to some criterion to be specified, such as variance, mean squared error, or risk. The conditions for optimal estimation, by any criterion, are rarely satisfied in a wide enough range of models to inform statistical practice, so attention turned to asymptotic optimality. The maximum likelihood estimator was shown to be consistent and asymptotically efficient, and is now generally computable with standard software, so is usually the first choice of estimation method, at least in parametric families of models. In the s a theory of robust point estimation was developed, to formalize the notion that a “good” method of estimation should be stable under perturbations of the assumed model. Theory and methods for nonparametric estimation of a wide variety of functional, or infinite-dimensional, parameters has developed very intensively over the past twenty years and have a prominent role in theoretical and methodological statistics. In this note however we will concentrate on finite-dimensional parametric estimation. The classic
E
E
Estimation
text on the theory of point estimation is Lehmann (), subsequently revised in Lehmann and Casella (). Casella and Berger () and Knight () provide excellent treatments at a somewhat less advanced level. The exposition here draws heavily on Chapter of Davison () and Chapter of Cox and Hinkley ().
Defining Point Estimators
which is an empirical version of the derivative of D( f , g) with respect to θ. In machine learning applications maximum likelihood estimators are often derived from this point of view, and D is referred to as the negative logentropy loss function. The method of moments defines point estimators by equating the theoretical moments of the presumed model to the observed sample moments. These estimators have been largely superseded by maximum likelihood estimators, but are often useful starting points, especially in rather complex models defined, for example, for stochastic systems modeled through sets of partial differential equations. The other class of models for which method of moments estimators are sometimes used is in estimating components of variance in normal theory linear models.
Most methods of points estimation are based on minimizing some notion of distance between the data and some aspect of the fitted model. For example, least squares estimators are defined to minimize ∑ni= {yi − µ i (θ)} , where we assume that E(yi ) = µ i (θ) has a known form, for example xiT θ for a given p-vector xiT . Least absolute deviation estimators minimize ∑ni= ∣yi − µ i (θ)∣. There may be other unknown parameters in the model; for example if yi is normally distributed with mean µ i (θ) the variance may also be unknown, and a suitable estimator for the variance is usually taken to be the residual mean square Σ{yi − ˆ /(n − p), where θˆ is the estimate of the paramµ i (θ)} eters in the mean, and θ is assumed to have dimension p. This estimator of σ is unbiased, as defined below, and widely used, but not optimal under conventional distance measures. Maximum likelihood estimators are defined to maximize the probability of the observed data, or equivalently to minimize − log f (y; θ). They are defined via
Example Suppose the vector of observations y is taken in k groups of size n, and modeled as
L(θˆML ; y) = sup L(θ; y),
i= j= k n
yij = µ + bi + є ij ,
j = , . . . , n; i = i, . . . , k,
where we assume that bi ∼ N (, σb ), є ij ∼ N(, σ ), and the b’s and є’s are mutually independent. The analysis of between and within group sums of squares leads to the following two statistics: k
n
SSwithin = ∑ ∑ (yij − y¯ i. ) , SSbetween = ∑ ∑ (¯yi. − y¯ .. ) ,
θ
i= j=
where L(θ; y) ∝ f (y; θ) is the likelihood function. In models with smooth differentiable likelihoods that have just one maximum, ∇θ ℓ(θˆML ; y) = ,
()
where ℓ(θ; y) = log L(θ; y). The maximum likelihood estimator can also be defined as an estimate of the value that minimizes the 7Kullback–Leibler divergence between the parametric model and the true density for y: D(f , g) = ∫ log {
g(y) } g(y)dy, f (y; θ)
where we temporarily assume that the true density is g(⋅) but we are fitting the parametric model f (y; θ). In the case of independent observations y , . . . , yn , the score equation () can be expressed as n
n− ∑(∂/∂θ) log f (yi ; θˆML ) = , i=
where y¯ indicates averaging over the appropriate subscript. It is not difficult to show that E(SSwithin ) = k(n − )σ , E(SSbetween ) = (k − ) (σ + nσb ), which leads to simple method of moments estimators of σ and σb . If there is very little sample variation among groups, then the estimate of σb could be negative, as method of moments does not automatically obey constraints in the parameter space. This example can be extended to much more complex structure, with several levels of variation, and possible dependence among some of the random effects. Point estimators can also be derived from a Bayesian point of view, although in a fully Bayesian context it is not necessary and may not even be wise to summarize the posterior distribution of the parameters by a point estimate. However, it is relatively straightforward to proceed, assuming the availability of a prior density π(θ), to construct the
Estimation
posterior density π(θ ∣ y) = L(θ; y)π(θ)/ ∫ L(θ; y)π(θ)dθ, and then to compute the posterior mode, θˆ π , which maximizes π(θ ∣ y), or the posterior median, or mean, or some other summary statistic. In most practical applications there will be much more information in the data than in the prior, and θˆ π will not be very different from the maximum likelihood estimator. This can be made more precise by considering asymptotic properties of Bayesian posterior distributions, under the sampling model. Empirical Bayes estimators can be obtained by using the data to estimate the parameters in the prior distribution, as well as the parameters in the model. A class of estimators often called shrinkage estimators can be derived by minimizing a 7loss function, but imposing a penalty on the resulting estimator of some form. The simplest example is the ridge regression estimator (see 7Ridge and Surrogate Ridge Regressions) associated with 7least squares. Example Suppose y is an n × vector that follows a normal distribution with mean Xβ and covariance matrix σ I, where X is an n × p matrix and β is a p × vector of regression coefficients of interest. The least squares estimator of β minimizes (y − Xβ)T (y − Xβ), with solution, if X is of full column rank, βˆ = (X T X)− X T y. If the least squares problems is changed to min(y − Xβ)T (y − Xβ), β
E
a large number of parameters, and then to shrink the polynomial coefficients. Empirical Bayes estimators are also usually shrinkage estimators; typically shrinking to some central data value, such as the sample mean, rather than a particular point of interest, such as , in the parameter space.
Evaluating Point Estimators It is conventional to refer to a point estimate as the value computed from a given set of data, and the point estimator as the function of the observations that will be used in practice. Many texts distinguish these two ideas by using upper case letters for random variables, and lower case letters for observed values of these random variables. Example Suppose we assume that we have a sample y = (y , . . . , yn ) of independent observations of a random vector Y = (Y , . . . , Yn ) with joint density f (y; µ, σ ) = √
exp {− ∑(yi − µ) }. () (π)n σ n σ
The sample mean µˆ = (/n)Σyi is an estimate of µ, and the estimator µˆ = (/n)ΣYi has a distribution determined by ˆ = µ and var( µ) ˆ = σ /n. Note that (); for example E( µ) the notation for µˆ does not distinguish between the estimate and the estimator, but this will usually be clear from the context. Suppose more generally that we have a vector of observations y of a random vector Y from a parametric model with density f (y; θ) where y ∈ Rn and θ ∈ Rp .
subject to β T β ≤ t, ˆ Definition An estimator θˆ = θ(y) is unbiased for θ if
then the solution, which also minimizes the penalized sum of squares
ˆ E{θ(Y)} = θ.
(y − Xβ)T (y − Xβ) + λβ T β,
The estimator is a minimum variance unbiased estimator ˜ we have if, for any other unbiased estimator θ,
βˆR = (X T X + λI)− X T y; the individual components βˆj are shrunk towards zero. This is sometimes used if number of components of β is quite large relative to n, as in most settings this would suggest that the individual components cannot all be well determined from the data. There are a great many extensions of this idea, especially in the literature on fitting “smooth” functions, rather than well specified parametric functions, to data. For example, we might replace Xβ by m (x ) + ⋅ ⋅ ⋅ + mp (xp ), without specifying very much about the form of mj . One approach to estimating these functions is to model them as piecewise polynomial, with
ˆ ≤ var(θ). ˜ var(θ)
is
The expectation and variance are calculated under the model f (y; θ). When θ is a vector, var(θ) is a matrix, and ˆ ≤ the variance condition above is generalized to aT var(θ)a ˜ for all real vectors a of length p. Minimum variaT var(θ)a ance unbiased estimators rarely exist; an exception is in regular exponential family models, where the minimal sufficient statistic is a minimum variance unbiased estimator of its expectation. An estimator θˆ is (weakly) consistent for θ if it converges to θ in probability, under the model f (y; θ). A consistent estimator θˆ is asymptotically efficient if the
E
E
Estimation
asymptotic variance of its limiting distribution is as small as possible. The 7Cramér–Rao inequality establishes that maximum likelihood estimators in regular models are asymptotically efficient, and the lower bound on the asymptotic variance for any consistent estimator is i− (θ), where i(θ) is the expected Fisher information in the model, E{−∂ ℓ(θ; Y)/∂θ }. Bayesian treatments of point estimation often emphasize that estimates based on posterior distributions obtained via proper priors are admissible, meaning that no non-Bayesian estimators have smaller Bayes risk. Bayes risk is defined with respect to a loss function, so depends in an intrinsic way on the units of measurement for the parameter. Admissible estimators are sometimes useful in decision theoretic contexts, but not often in scientific work. In machine learning contexts where there is the possibility to fit rather complex models to large data sets, performance is often measured by mean-squared error, as biased point estimators such as the shrinkage estimator mentioned above are widely used. The phrase “bias-variance tradeoff ” refers to the fact that a biased point estimator can have smaller variance than an unbiased estimator, but that bias and variance cannot be simultaneously minimized. Estimates of functional parameters, such as densities or regression functions, are often compared by means of integrated mean-squared error. Example Suppose Y is a sample of independent, identically distributed observations each from an unknown density f (⋅), and the goal is to estimate this density. The kernel density estimator of f is defined as y − Yi n ), fˆ(y) = ∑K ( nh i= h where h > is a tuning parameter and K(⋅) is a kernel function, usually a symmetric probability density function, such as the density for a N(, ) distribution. The tuning parameter h controls the smoothness of the estimated function fˆ(⋅), by controlling the number of sample points that enter the average in estimating f (y). It can be shown that the squared bias of fˆ at a single point y is O(h ), and the variance is O(nh)− , so the optimal bandwidth can be shown to be h ∝ n−/ , with the constant of proportionality depending on the point y and the kernel function K(⋅). Robust estimators are defined, following Huber (), to be stable under perturbations of the postulated model. From a data analytic point of view, robust estimators are not much affected by 7outliers, or extremely unusual
data points. The motivation for this is that such outliers may be coding mistakes, or may be irrelevant for the inference desired. However this varies widely by application area. The theory of estimating equations considers the primary object of interest as the equation defining the point estimator, rather than the point estimator itself. For an independent, identically distributed sample y = (y , . . . , yn ), from a model with parameter θ, an estimating equation for θ is given by n
∑ g(yi ; θ) = , i=
for a function g(⋅) to be chosen. An estimating function is unbiased if E{g(Y; θ)} = , and it would be rare to start with a biased estimating function, as the resulting estimator would not be consistent for θ without further modification. The maximum likelihood estimator is, in regular models, defined by the estimating equation with g(y; θ) = (∂/∂θ) log f (y; θ), but more general classes of estimating equations arise naturally in studies of robustness; see Davison (, Ch. ).
About the Author For biography see the entry 7Likelihood.
Cross References 7Asymptotic Relative Efficiency in Estimation 7Bayesian vs. Classical Point Estimation: A Comparative
Overview 7Best Linear Unbiased Estimation in Linear Models 7Cramér–Rao Inequality 7Estimation: An Overview 7Methods of Moments Estimation 7Nonparametric Estimation 7Properties of Estimators 7Robust Regression Estimation in Generalized Linear Models 7Statistical Inference 7Statistical Inference: An Overview 7Sufficient Statistical Information
References and Further Reading Aldridge J () R.A. Fisher and the making of maximum likelihood –. Stat Sci :– Casella G, Berger RL () Statistical inference. Pacific Grove, California Cox DR, Hinkley DV () Theoretical statistics. Chapman and Hall, London Davison AC () Statistical models. Cambridge University Press, Cambridge
Estimation Problems for Random Fields
Fisher RA () On the mathematical foundations of theoretical statistics. Phil Trans R Soc Lond A :– Huber PJ () Robust statistics. Wiley, New York Knight K () Mathematical statistics. Chapman and Hall, London Lehmann EL () Theory of point estimation. Wiley, New York Lehmann EL, Casella G () Theory of point estimation, nd edn. Springer-Verlag, New York Stigler SM () Gauss and the invention of least squares. Ann Statist :–
Estimation Problems for Random Fields Mikhail P. Moklyachuk Professor Kyiv National Taras Shevchenko University, Kyiv, Ukraine Estimation problems for random fields X(t), t ∈ Rn (estimation of the unknown mathematical expectation, estimation of the correlation function, estimation of regression parameters, extrapolation, interpolation, filtering, etc) are similar to the corresponding problems for 7stochastic processes (random fields of dimension ). Complications usually are caused by the form of domain of points {tj } = D ⊂ Rn , where observations {X(tj )} are given, and by the dimension of the field. The complications can be overcome by considering specific domains of observations and particular classes of random fields. Say in the domain D ⊂ Rn there are given observations of the random field
E
on the sphere Sn (r) = {x ∈ Rn , ∥x∥ = r} is of the form X(r, u)mn (du), n ≥ , θˆ = ω n ∫Sn(r) where mn (du) is the Lebesgue measure on the sphere Sn (r), ω n is the square of the surface of the sphere, (r, u) are spherical coordinates of the point t ∈ Rn . Consider the extrapolation problem. . Observations of the mean-square continuous homogeneous and isotropic random field X(t), t ∈ Rn are given on the sphere Sn (r) = {x ∈ Rn , ∥x∥ = r}. The problem is to ˆ determine the optimal mean-square linear estimate X(s) of the unknown value X(s), s ∉ Sn (r), of the random field. It follows from the spectral representation of the field that this estimate is of the form ∞ h(m,n)
l ˆ X(s) = ∑ ∑ cm (s) ∫ m=
l=
∞ Jm+(n−)/(rλ) l (dλ), Zm (rλ)(n−)/
l where coefficients cm (s) are determined by a special algorithm (Yadrenko ). For practical purposes it is more convenient to have a formula where observations X(t), t ∈ Sn (r), are used directly. The composition theorem for spherical harmonics gives us this opportunity. We can write
ˆ X(s) =∫
S n (r)
c(s, t)X(t)dmn (t),
where the function c(s, t) is determined by the spectral function Φ(λ) of the field X(t) (Yadrenko ). . An isotropic random field X(t), t = (r, u) ∈ Rn is observed in the sphere VR = {x ∈ Rn , ∥x∥ ≤ R}. The optiˆ mal liner estimate X(s) of the unknown value X(s), s = (ρ, v) ∉ VR , of the field has the form
q
X (t) = ∑ θ, gi (t) + Y(t), i=
where gi (t), i = , . . . , q, are known non-random functions, θ i , i = , . . . , q, are unknown parameters, and Y(t) is a random field with EY(t) = . The problem is to estimate the regression parameters θ i , i = , . . . , q. This problem includes as a particular case (q = , g (t) = ), the problem of estimation of the unknown mathematical expectation. Linear unbiased least squares estimates of the regression parameters can be found by solving the corresponding linear algebraic equations or linear integral equations determined with the help of the correlation function. For the class of isotropic random fields, formulas for estimates of the regression parameters are proposed by M. I. Yadrenko (Yadrenko ). For example, the estimate θˆ of the unknown mathematical expectation θ of an isotropic random field X(t) = X(r, u) from observations
ˆ X(s) =∫
C(s, t)X(t)dmn (t), VR ∞ h(m,n)
l C(s, t) = ∑ ∑ cm (r)Slin (u), m=
l=
l where coefficients cm (r) are determined via special integral equations R
l (r)r n− dr, bm (ρ, q)Slm (v) =∫ bm (r, q)cm
l = , , . . . , h(m, n),
m = , , . . . ;
q ∈ [, R].
If, for example, X(t), t = (r, u), is an isotropic random field where bm (r, q) = a∣m∣ exp{−β∣r − q∣}, then it is easy to ˆ see that X(ρ, v) = exp{−β∣ρ − R∣}X(R, v), v ∈ Sn . For methods of solutions of other estimation problems for random fields (extrapolation, interpolation, filtering, etc.) see Grenander (), Moklyachuk (), Ramm
E
E
Estimation: An Overview
(), Ripley (), Rozanov (), Yadrenko (), and Yaglom ().
About the Author For biography see the entry 7Random Field.
Cross References 7Estimation 7Random Field
References and Further Reading Grenander U () Abstract inference. Wiley series in probability and mathematical statistics. Wiley, New York Moklyachuk MP () Robust estimates of functionals of stochastic processes. Vydavnycho-Poligrafichny˘ı Tsentr, Kyïvsky˘ı Universytet, Kyïv Ramm AG () Random fields estimation. World Scientific, Hackensack, NJ Ripley BD () Spatial statistics. Wiley series in probability and mathematical statistics. Wiley, New York Rozanov YA () Markov random fields. Springer-Verlag, New York Yadrenko MI () Spectral theory of random fields. Translation series in mathematics and engineering. (Optimization Software Inc., New York) Springer-Verlag, New York Yaglom AM () Correlation theory of stationary and related random functions, volume I and II. Springer series in statistics. Springer-Verlag, New York
A statistical model is a collection of probability measures P. If the true distribution is in the model class P, we call the model well-specified. In many situations, it is useful to parametrize the distributions in P, i.e., to write P := {Pθ : θ ∈ Θ}, where Θ is the parameter space. The parameter of interest is some function of θ, say γ := g(θ) ∈ Γ. Example . Case (i) Suppose X is known to be normally distributed with unknown mean µ and unknown variance σ (we write this as X ∼ N (µ, σ )). The parameter space is then the collection of -dimensional parameters θ := (µ, σ ). If one is only interested in estimating the mean µ, one lets g(µ, σ ) = µ. The second parameter σ is then called a nuisance parameter. Case (ii) If actually the distribution of X is completely unknown, we may take P := {all distributions on R}. We can let P itself be the parameter space, and the parameter of interest is written as function g(P) of the probability measure P. For example, with µ = EX being the parameter interest, we have g(P) := ∫ xdP(x).
How to Construct Estimators?
Estimation: An Overview Sara van de Geer Professor ETH Zürich, Zürich, Switzerland
Introduction Let P be a probability distribution on some space X . A random sample from P is a collection of independent random variables X , . . . , Xn , all with the same distribution P. We then also call X , . . . , Xn independent copies of a population random variable X, where X has distribution P. In statistics, the probability measure P is not known, and the aim is to estimate aspects of P using the observed sample X , . . . , Xn . Formally, an estimator, say T, is any given known function of the data, i.e., T = T(X , . . . , Xn ). Example . Suppose the observations are real-valued, i.e., the sample space X is the real line R. An estimator of the population mean µ := EX is the sample mean µˆ := n ∑i= Xi /n.
From a mathematical point of view, the construction of an estimator as function of the data X , . . . , Xn can be based on loss and risk. Let g(θ) be the parameter of interest, and T = T(X , . . . , Xn ) be an estimator. With this estimator, we associate the loss L(g(θ), T), where L is a given 7loss function. As T is a random variable, the loss is generally random as well. The risk of the estimator T is now defined as R(θ, T) := Eθ L(g(θ), T), where the expectation Eθ is with respect to the probability measure of an i.id. sample X , . . . , Xn from Pθ . When the parameter of interest is real-valued, an important special case is quadratic loss L(g(θ), T) = ∣T − g(θ)∣ . The risk is then the mean square error MSEθ (T) = Eθ ∣T − g(θ)∣ . The mean square error can be separated into a squared bias term and a variance term MSEθ (T) = biasθ (T) + varθ (T),
Estimation: An Overview
where biasθ (T) := Eθ T − g(θ), and varθ (T) := Eθ T − (Eθ T) . An estimator T is called unbiased if Eθ T = g(θ), ∀ θ. It is called uniform minimum variance unbiased (UMVU) if it has the smallest variance among all unbiased estimators, for all values of the parameter. Once the loss function L(g(θ), T) is specified, one may try to find the estimator T which has smallest risk R(θ, T). However, the risk depends on θ, which is unknown! This generally means that estimators are not directly comparable. One way to overcome this problem is to require the estimator to be the best one in the worst possible case: min
estimators T
max R(θ, T). θ∈Θ
This is called the minimax approach. Another way to treat the dependence on θ is to assign a priori weights to the various parameter values. This is called the Bayesian approach. Formally, let Π be some probability measure on Θ, the so-called prior. The Bayes’ risk of the estimator T is then defined as R(T) := ∫ R(ϑ, T)dΠ(ϑ). In other words, the unknown parameter is integrated out. The minimizer Tbayes of R(T) over all estimators T = T(X , . . . , Xn ) is called Bayes’ estimator. The (minimax, Bayes) risk generally heavily depends on the model class P. This means that if the model is misspecified, the estimator may not have the optimality properties one hoped for. A related problem is the robustness of an estimator: how much does it change if some of the data points in the sample are perturbed? One often decides to stay on the conservative side, i.e. instead of believing in the model and aiming at mathematical optimality, one prefers to be sure to be doing something reasonable. And, last but not least, there may be practical cost considerations that prevent one from applying a particular estimator. Here, cost can also be computation time, e.g., if one needs online answers. The asymptotic approach is to try to construct estimators that are “approximately” “optimal” under mild assumptions. For example, unbiased estimators often do not even exist. However, for large sample size n, many estimators are “approximately” unbiased. One then looks for the one with the smallest asymptotic variance, i.e., the highest asymptotic efficiency. Rigorous asymptotic theory
E
can be mathematically involved, as a certain uniformity in the parameter θ is required.
The Plug in Principle In this section, we assume that the sample size allows for an asymptotic approach, i.e., that n is large enough to base ideas on e.g. 7laws of large numbers and 7central limit theorems. We then see X , . . . , Xn as the first n of an infinite sequence. The plug in principle is mainly based on the law of large numbers. Consider for example a subset A ⊂ X . The probability of this set is P(X ∈ A), or shortly P(A). As, by the law of large numbers, probabilities can be approximated by frequencies, a sensible estimator of P(A) is the proportion of observations that fall into A: Pˆ n (A) := #{i ∈ {, . . . , n} : Xi ∈ A}. n Note that Pˆ n corresponds to a probability measure that puts mass /n at each observation. One calls Pˆ n the empirical distribution. Consider now a parameter of interest γ = g(θ). We write it as γ := Q(P), where Q is a given function on the probability measures P ∈ P. In other words, g(θ) = Q(Pθ ). The plug in principle is now to use the estimator γˆn := Q(Pˆ n ), i.e., to replace the unknown probability measure P by the empirical measure. We remark however that is may happen that Q(Pˆ n ) is not defined (as generally Pˆ n is not in the model class P). In that case, one takes γˆn := Qn (Pˆ n ), where Qn (P) ≈ Q(P). Example . Suppose X ∈ R. Let µ = EX = ∫ xdP(x) be the parameter of interest. The sample mean is µˆ n = ∫ xdPˆ n (x) =
n ∑ Xi . n i=
Example . Let the sample space again be the real line. The distribution function of X is F(x) := P(X ≤ x),
x ∈ R.
The empirical distribution function is Fˆ n (x) = #{i ∈ {, . . . , n} : Xi ≤ x}. n
E
E
Estimation: An Overview
The Glivenko–Cantelli Theorem tells us that the law of large numbers hold, uniformly in x (In fact, by Donsker’s Theorem, √ n(Fˆ n − F) converges in distribution to B ○ F, where B is a Brownian bridge.) sup ∣Fˆ n (x) − F(x)∣ → ,
n → ∞,
x
with probability one. Suppose we know that F has a density f with respect to Lebesgue measure, and that we aim at estimating f at a fixed point x. Then F(x + h) − F(x − h) . h→ h
When Θ is finite-dimensional, say Θ ⊂ Rp , then under general regularity conditions, the MLE is approximately unbiased and each component has the smallest asymptotic √ variance. In fact, then n(θˆn − θ) is approximately normally distributed, with mean zero and covariance matrix I(θ)− , where I(θ) is the p × p Fisher information matrix I(θ) = Eθ sθ (X)sTθ (X), with sθ the score function sθ :=
Q(F) = f (x) = lim
Note that Q(Fˆ n ) does not make sense in this case. Instead, we take Fˆ n (x + hn ) − Fˆ n (x − hn ) , Qn (Fˆ n ) := hn where the bandwidth hn is “small.” The choice of hn can theoretically be based on a trade-off between bias and variance.
∂ log pθ . ∂θ
Moreover, I(θ) can be approximated by the matrix of second derivatives −
n ∂ . ∑ log p ϑ (Xi )/n∣ T ∂ϑ∂ϑ i= ϑ= θˆ n
These results however are not for free as they do rely on regularity conditions. The MLE can be seen as plug in estimator: θˆn = arg max ∫ log p ϑ dPˆ n ,
Maximum Likelihood
ϑ
Suppose that P = {Pθ : θ ∈ Θ} is a dominated family, i.e., there exists a σ-finite dominating measure ν, such that the densities dP θ∈Θ pθ := θ , dν exist. The maximum likelihood estimator (MLE) θˆn of θ is then defined as
and
θ = arg max ∫ log p ϑ dPθ . ϑ
M-Estimators An M-estimator is of the form γˆn := arg min
n
θˆn := arg max ∑ p ϑ (Xi ), ϑ∈Θ i=
where “arg” is “argument,” i.e, the location where the maximum is achieved. It is to be checked in particular cases that the maximum actually exists. Example . Suppose the densities w.r.t. Lebesgue measure are pθ (x) = θ( + x)−θ+ , x > , where θ ∈ Θ = (, ∞). Then log p ϑ (x) = log ϑ − (ϑ + ) log( + x). We denote the derivative w.r.t. ϑ by s ϑ (x) :=
d log p ϑ (x) = − log( + x). dϑ ϑ
We put the derivative ∑ni= s ϑ (Xi ) to zero: n n − ∑ log( + Xi ) = . θˆn i=
This gives θˆn = / [∑ni= log( + Xi )/n].
c∈Γ
n ∑ ρ c (Xi ). n i=
Here, for each c ∈ Γ, ρ c : X → R, c ∈ Γ is a given function, called a loss function (generally to be distinguished from the loss function considered in Section 7“How to Construct Estimators?”). The M-estimator targets at the parameter γ := arg min Eρ c (X). c∈Γ
This is indicated by the plug-in principle: the mean Eρ c (X) is approximated by the average ∑ni= ρ c (Xi )/n. Example . Let X = R and also Γ = R. The sample mean µˆ = ∑ni= Xi /n is an M-estimator with ρ c (x) = (x − c) , the squared error. In other words, the sample mean is the least squares estimator. In the same spirit, the sample median (the middle observation) can be seen as an M-estimator with ρ c (x) = ∣x−c∣. More generally, the empirical (−α)-quantile Fˆ n− ( − α) is an M-estimator with ρ c (x) = ρ(x − c), ρ(x) = α∣x∣{x > } + ( − α)∣x∣{x < }. Example . The MLE is an M-estimator, with Γ = Θ, and ρ ϑ = − log p ϑ .
Estimation: An Overview
Non-Parametric and High-Dimensional Problems The non-parametric case is the situation where the parameter space Θ is infinite-dimensional. Closely related is the case where Θ ⊂ Rp with p > n (or p ≫ n). Construction of estimators in high-dimensional or nonparametric models often - but not always - requires some complexity regularization. This is because a famous statistical rule-ofthumb, which says that the number of parameters should not exceed the number of observations, is violated. Example . Consider a linear regression model (see 7Linear Regression Models) p
Y = ∑ Zj β j + є, j=
where Y is a real-valued response variable, Z , . . . , Zp are p co-variables, and where є is mean-zero measurement error. The coefficients β = (β , . . . , β p )T form a p-dimensional unknown parameter. Let Z = (Z , . . . , Zp ) be the pdimensional co-variable. Suppose we observe n independent copies {(Yi , Zi )}ni= of (Y, Z). Ideally, we would like n ≫ p, in order to estimate the p parameters. If however p ≥ n, one generally needs a regularization penalty. For example, when one believes that only a few of the variables are relevant, one may use the so-called Lasso p n βˆn = arg min ∑ ∣Yi − (Zβ)i ∣ + λ n ∑ ∣β j ∣, β n i= j=
where λ n > is a tuning parameter. Note that we used the standard quadratic loss function here. Other choices may also be applied, for instance quantile regression: p n βˆn = arg min ∑ ρ(Yi − (Zβ)i ) + λ n ∑ ∣β j ∣, β n i= j=
E
where K is a given kernel. Another possibility is to use a regularized M-estimator. For example, when X = [, ], one may consider using fˆn = arg min { ∫ f
n
f (x)dx − ∑ f(Xi )/n i=
+ λn ∫
∣¨f(x)∣ dx},
where λ n is again a tuning parameter. Regularization is not always necessary, it can often be replaced by qualitative constraints such as monotonicity or convexity. Example . Suppose X has density f with respect to Lebesgue measure, and suppose we know that f is increasing. Let fˆn be the Grenander estimator, i.e., the maximum likelihood estimator fˆn = arg max f increasing
n ∑ log f(Xi ). n i=
Then under moderate conditions, fˆn is a good approximation of f for large sample size n.
Further Reading Estimation is a very broad area is statistics. Classical books covering this area are Bickel and Doksum () and Rice (), and for Bayesian methods, Berger (). The details for the asymptotic approach (also non- and semiparametric) are in van der Vaart (). In van de Geer () one finds a treatment of M-estimators, mostly in a nonparametric context. One can read more about density estimation in Wand and Jones (), and Silverman (). The Lasso is explained in Hastie et al. (), and the estimator of a monotone density is handled in Grenander ().
where, as in Example ρ(x) = α∣x∣{x > } + ( − α)∣x∣{x < }. Example . The estimator Fˆ n (x + hn ) − Fˆ n (x − hn ) , fˆn (x) := hn considered in Example is a nonparametric estimator of the density f at the point x. In this case, the bandwidth hn is the tuning parameter. More generally, one may apply kernel estimators n x − Xi fˆn (x) = ), ∑K ( nhn i= hn
About the Author Dr. Sara Ana van de Geer obtained her PhD in under supervision of Prof. R.D. Gill and Prof. W.R. van Zwet. She is Full Professor at the ETH Zürich (since ). She was Chair of Probability and Mathematical Statistics, Leiden University (–), and currently is Chair of the Seminar for Statistics, ETH Zürich. Dr. van de Geer is Elected Fellow of the Institute of Mathematical Statistics, Elected member of the International Statistical Institute, and Correspondent of the Dutch Royal Academy of Sciences. She was Associate Editor of Bernoulli (–), Annals of Statistics (–), Statistica Sinica (–), and Statistica Neerlandica (–). Currently, she is Editor
E
E
Eurostat
of Collections Volume (Springer, ) and Associate Editor of The Scandinavian Journal of Statistics (–). Professor van de Geer has (co-)authored over papers and three books, including Empirical Processes in M-Estimation (Cambridge University Press, ). She was awarded the Medallion Lecturer at the Joint Statistical Meetings, San Francisco (August ) and the International Statistical Institute Award.
Cross References 7Absolute Penalty Estimation 7Advantages of Bayesian Structuring: Estimating Ranks
and Histograms 7Approximations for Densities of Sufficient Estimators 7Asymptotic Relative Efficiency in Estimation 7Bayesian vs. Classical Point Estimation: A Comparative Overview 7Best Linear Unbiased Estimation in Linear Models 7Complier-Average Causal Effect (CACE) Estimation 7Decision Trees for the Teaching of Statistical Estimation 7Estimation 7Estimation Problems for Random Fields 7Hazard Ratio Estimator 7Hodges-Lehmann Estimators 7Horvitz–Thompson Estimator 7James-Stein Estimator 7Kaplan-Meier Estimator 7Maximum Entropy Method for Estimation of Missing Data 7Methods of Moments Estimation 7Nonparametric Density Estimation 7Nonparametric Estimation 7Nonparametric Estimation Based on Incomplete Observations 7Optimal Designs for Estimating Slopes 7Optimal Shrinkage Estimation 7Optimal Shrinkage Preliminary Test Estimation 7Principles Underlying Econometric Estimators for Identifying Causal Effects 7Properties of Estimators 7Robust Regression Estimation in Generalized Linear Models 7Small Area Estimation 7Statistical Estimation of Actuarial Risk Measures for Heavy-Tailed Claim Amounts 7Statistical Inference 7Statistical Inference: An Overview 7Target Estimation: A New Approach to Parametric Estimation 7Trend Estimation 7Unbiased Estimators and Their Applications
References and Further Reading Berger JO () Statistical decision theory and Bayesian analysis. Springer, New York Bickel PJ, Doksum KA () Mathematical statistics, vol . Prentice Hall, Upper Saddle River, NJ Grenander U () Abstract inference. Wiley, New York Hastie T, Tibshirani R, Friedman J () The elements of statistical learning. Springer, New York Rice JA () Mathematical statistics and data analysis. Duxbury Press, Belmont, CA Silverman BW () Density estimation for statistics and data analysis. Chapman & Hall/CRC, Boca Raton, FL van de Geer S () Empirical processes in M-estimation. Cambridge University Press, Cambridge, UK van der Vaart AW () Asymptotic statistics. Cambridge University Press, Cambridge, UK Wand MP, Jones MC () Kernel smoothing. Chapman & Hall/CRC, Boca Raton, FL
Eurostat Walter J. Radermacher Chief Statistician of the European Union and Director-General of Eurostat Luxembourg
Eurostat is the Statistical Office of the European Union. Established in and based in Luxembourg, it is part of the European Commission. Its role is to provide the European Union with high-quality statistics that enable comparisons between countries and regions.
Responsibilities Eurostat has two main areas of responsibility. Firstly, in close cooperation with the national statistical authorities of the EU Member States, it develops harmonized methods for the production of statistics. Secondly, Eurostat calculates and publishes total figures, aggregates, for the European Union and the euro area. As Eurostat does not conduct its own surveys in EU Member States, statistics from specific areas are transmitted to Eurostat by individual countries. The methodology applied by EU Member States in the data collection as well as the delivery deadlines for transmission to Eurostat are regulated to a large degree by EU legislation. Eurostat’s statistics enable straightforward comparisons between European countries and regions and are crucial for politicians, economists and business people active at European, country and local government levels.
Eurostat
Increasingly, statistics made available by Eurostat are also consulted by members of the European public and media. The Eurostat website gives access to a huge selection of statistical information. Over million figures are available in its data base, around , Eurostat publications; many of them in English, German and French, and , tables can be accessed online and free of charge.
Internal Organization Eurostat is part of the European Commission. It is headed by Director-General Walter Radermacher, the Chief Statistician of the European Union. Eurostat reports to the European Commissioner for Economic and Monetary Affairs. In , Eurostat had staff. It is divided into seven directorates. They are: ● ● ● ● ● ●
●
Cooperation in the European Statistical System; Resources Quality, methodology and information systems National and European Accounts External cooperation, communication and key indicators Sectoral and regional statistics (agriculture, fisheries, environment, transport, energy and regional statistics) Social and information society statistics (population, labor market, living conditions and social protection, education, science, culture, health and food safety, crime, information society and tourism statistics) Business statistics (business statistics, international trade and price statistics)
European Statistical System Eurostat does not work alone. Together with the national statistical authorities of the EU- Member States (national statistical institutes, or NSIs, ministries, central banks etc.) it forms the European Statistical System (ESS). Its legal basis is the Regulation on European Statistics which came into force in April , replacing older legislation. The main decision-making body of the ESS is the European Statistical System Committee (ESSC), chaired by the Director-General of Eurostat. The ESSC is made up of Heads of the NSIs. It develops the ESS strategy, in accordance with the so-called European Statistics Code of Practice from . In , the European Statistical Governance Advisory Board, or ESGAB, was created. Its role is to enhance the professional independence, integrity and accountability of the European Statistical System as well as to improve the quality of European statistics.
E
Also created in , the European Statistical Advisory Committee (ESAC) consists of representatives of institutional and private statistics users, who give their opinion on various statistical projects developed by the ESS.
Short History The origins of Eurostat go back to the early s, when a “statistical service” of the Coal and Steel Community was created in , charged with conducting “a continuous study of market and price trends.” Over the next few years the responsibilities of the Statistical Division, renamed in , diversified and, when the European Community was founded in , it became a separate Directorate-General of the European Commission. Eurostat received its present name in . Today, Eurostat’s key role is to supply the EU institutions and policy makers as well as the research community and all European citizens with a wide range of statistics, instrumental for planning, development and implementation of joint European Community policies.
For More Information Eurostat website: http://epp.eurostat.ec.europa.eu/portal/ page/portal/eurostat/home
About the Author Walter Radermacher was born in Walheim (near Aachen) on June . From to he studied business administration in Aachen and in Münster. After obtaining the academic degree of Diplom-Kaufmann (economist), he worked as member of academic staff at the University of Münster from to . At the beginning of , Walter Radermacher joined the Federal Statistical Office. Mr Radermacher has held a wide variety of posts at the German Statistical Office during his thirty-year career there, notably in the fields of environmental and economic statistics. He has been Vice-President of the German Federal Statistical Office (–), and President since . During the German Presidency, he was Chair of the Council’s working group on statistics. He has also chaired the UN Committee on Environmental–Economic Accounting. During his career, Mr. Radermacher has had teaching assignments in statistics and environmental economy at the specialised college of higher education (Fachhochschule) in Wiesbaden and the University of Lüneburg in Germany. As of , he was appointed Director General of Eurostat and chief statistician of the European Union.
E
E
Event History Analysis
Cross References 7Business Surveys 7Comparability of Statistics 7Integrated Statistical Databases 7National Account Statistics 7Statistical Publications, History of
Event History Analysis Hans-Peter Blossfeld Professor Otto-Friedrich-Universität Bamberg, Bamberg, Germany
Introduction Event history analysis studies a collection of units, each moving among a finite (usually small) number of states. An example is individuals who are moving from unemployment to employment. In this article various examples of event history applications are presented, the concepts and advantages of event history analysis are demonstrated, and practical complications are addressed.
Examples of Continuous-Time, Discrete-State Processes Event history analysis is the study of processes that are characterized in the following general way: () there is a collection of units (which may be individuals, organizations, societies, etc.), each moving among a finite (usually small) number of states; () these changes (or events) may occur at any point in time (i.e., they are not restricted to predetermined points in time); and () there are timeconstant and/or time-varying factors influencing the timing of events. Examples are workers who move between unemployment and employment; men and women who enter into consensual unions or marriages; companies that are founded or closed down; governments that break down; people who are mobile between different regions or nation states; consumers who switch from one brand to another; prisoners who are released and commit another crime; people who show certain types of verbal and non-verbal behaviors in social interactions; students who drop out of school; incidences of racial and ethnic confrontation, protest, riot, and attack; people who show signs of psychoses or neuroses; patients who switch between the states “healthy” and “diseased,” and so on. In event history analysis, the central characteristics of the underlying stochastic process are mirrored in the specific way theoretical and
mathematical models are built, data are collected, and the estimation and evaluation of models is done.
Different Observation Plans In event history analysis the special importance of broader research design issues has been stressed. In particular, different observation plans have been used to collect information on the continuous-time, discrete-state substantive process. These various schemes produce different types of data that constrain the statistical analysis in different ways: cross-sectional data, event count data, event sequence data, 7panel data, and event history data. The cross-sectional observation design is still the most common form of data collection in many fields. A cross-sectional sample is only a “snapshot” taken on the continuous-time, discrete-state substantive process (e.g., the jobs of people at the time of the interview). Event history analysis has shown that one must be very cautious in drawing inferences about explanatory variables on the basis of such data because, implicitly or explicitly, social researchers have to assume that the process under study is in some kind of equilibrium. Equilibrium means that the state probabilities are fairly trendless and the relationships among the variables are quite stable over time. Therefore, an equilibrium of the process requires that the inflows to and the outflows from each of the discrete states be equal over time to a large extent. This is a strong assumption that is not very often justified in the social and behavioral sciences (see Blossfeld and Rohwer ; Blossfeld et al. ). A comparatively rare type of data on changes of the process are event count data. They record the number of different types of events for each unit in an interval (e.g., the number of racial and ethnic confrontations, protests, riots, or attacks in a period of years). Event sequence data provide even more information. They record the sequence of specific states occupied by each unit over time. Today, the temporal data most often available to the scientist are panel data. They provide information on the same units of analysis at a series of discrete points in time. In other words, there is only information on the states of the units at predetermined survey points and the course of the process between the survey points remains unknown. Panel data certainly contain more information than crosssectional data, but involve well-known distortions created by the method itself (e.g., panel bias, attrition of sample). In addition, causal inferences based on panel approaches are much more complicated than has been generally acknowledged (Blossfeld and Rohwer ; Blossfeld et al. ). With regard to the continuous-time, discrete-state substantive process, panel analysis is particularly sensitive
Event History Analysis
to the length of the time intervals between the waves relative to the speed of the process. They can be too short, so that very few state transitions will be observed, or too long, so that it is difficult to establish a time-order between the events. The event oriented observation design records the complete sequence of states occupied by each unit and the timing of changes among these states. For example, an event history of job mobility consists of more or less detailed information about each of the jobs and the exact beginning and ending dates of each job. Thus this is the most complete information one can get on the continuous-time, discrete-state substantive process. Such event history data are often collected retrospectively with life history studies. Life history studies are normally cheaper than panel studies and have the advantage that they code the data into one framework of codes and meaning. But retrospective studies also suffer from several limitations (see Blossfeld and Rohwer ; Blossfeld et al. ). In particular, data concerning motivational, attitudinal, cognitive, or affective states are difficult (or even impossible) to collect retrospectively because the respondents can hardly recall the timing of changes in these states accurately. Also the retrospective collection of behavioral data has a high potential for bias because of its strong reliance on autobiographic memory. To reduce these problems of data collection, modern panel studies (e.g., the PSID in the US, the BHPS in the UK, or the SOEP in Germany) use a mixed design that provides traditional panel data and retrospectively collects event history data for the period before the first panel wave and between the successive panel waves.
Event History Analysis Event history analysis, originally developed as independent applications of mathematical probability theory in 7demography (e.g., the classical life table analysis and the product-limit estimator), reliability engineering, and 7biostatistics (e.g., the path-breaking semiparametric regression model for 7survival data), has gained increasing importance in the social and behavioral sciences (see Tuma and Hannan ; Blossfeld et al. ; Lancaster ; Blossfeld and Rohwer ; Blossfeld et al. ) since the s. Due to the development and application of these methods in various disciplines, the terminology is normally not easily accessible to the user. Central definitions are therefore given first.
Basic Terminology Event history analysis studies transitions across a set of discrete states, including the length of time intervals between entry to and exit from specific states. The basic analytical
E
framework is a state space and a time axis. The choice of the time axis or clock (e.g., age, experience, and marriage duration) used in the analysis must be based on theoretical considerations and affects the statistical model. Dependent on practical and theoretical reasons, there are event history methods using a discrete- (Yamaguchi ; Vermunt ) or continuous-time axis (Tuma and Hannan ; Blossfeld et al. ; Lancaster ; Courgeau and Lelièvre ; Blossfeld and Rohwer ; Blossfeld et al. ). Discretetime event history models are, however, special cases of continuous-time models, and are therefore not further discussed here. An episode, spell, waiting time, or duration – terms that are used interchangeably in the literature – is the time span a unit of analysis (e.g., an individual) spends in a specific state. The states are discrete and usually small in number. The definition of a set of possible states, called the state space, is also dependent on substantive considerations. Thus, a careful, theoretically driven choice of the time axis and design of state space are crucial because they are often serious sources of misspecification. The most restricted event history model is based on a process with only a single episode and two states (an origin state j and a destination state k). An example may be the duration of first marriage until the end of the marriage, for whatever reason (separation, divorce, or death). In the single episode case each unit of analysis that entered into the origin state is represented by one episode. Event history models are called multistate models, if more than one destination state exists. Models for the special case with a single origin state but two or more destination states are also called models with competing events or risks. For example, a housewife might become “unemployed” (meaning entering into the state “looking for work”), or start being “full-time” or “part-time employed.” If more than one event is possible (i.e., if there are repeated events or transitions over the observation period), the term multiepisode modes is used. For example, an employment career normally consists of a series of job shifts.
Censoring Observations of event histories are often censored. Censoring occurs when the information about the duration in the origin state is incompletely recorded. An episode is fully censored on the left, if starting and ending times of a spell are located before the beginning of an observation period (e.g., before the first panel wave) and it is partially censored on the left, if the length of time a unit has already spent in the origin state is unknown. This is typically the case in panel studies, if individuals’ job episodes at the time of a first panel wave are known but no further information about the history of the job is collected. Left censoring is
E
E
Event History Analysis
normally a difficult problem because it is not possible to take the effects of the unknown episodes into account. It is only without problems if the assumption of a Markov process (see 7Markov Processes) is justified (i.e., if the transition rates do not depend on the duration in the origin state). The usual kind of censoring, however, is right censoring. In this case the end of the episode is not observed but the observation of the episode is terminated at an arbitrary point in time. This type of censoring typically occurs in life course studies at the time of the retrospective interviews or in panel studies at the time of the last panel wave. Because the timing of the end of the observation window is normally determined independently from the substantive process under study, this type of right censoring is unproblematic and can easily be handled with event history methods. Finally, episodes might be completely censored on the right. In other words, entry into and exit from the duration occurs after the observation period. This type of censoring normally happens in retrospective life history studies in which individuals of various birth cohorts can only be observed over very different spans of life. To avoid sample selection bias, such models have to take into account variables controlling for the selection, for example, by including birth cohort dummy variables (see Yamaguchi ).
The Transition Rate The central concept of event history analysis is the transition rate. Because of the various origins of event history analysis in the different disciplines, the transition rate is also called the hazard rate, intensity rate, failure rate, transition intensity, risk function, or mortality rate: r (t) = lim Pr (t ≤ T < t ′ ∣T ≥ t) / (t′ − t). ′ t →t
The transition rate provides a local, time-related description of how the process evolves over time. It can be interpreted as the propensity (or intensity) to change from an origin state to a destination state, at time t. But one should note that this propensity is defined in relation to a risk set (T ≥ t) at t, i.e., the set of units that still can experience the event because they have not yet had the event before t.
Statistical Models The central idea in event history analysis is to make the transition rate, which describes a process evolving in time, dependent on time (t) and on a set of covariates, x: r (t) = g (t, x) .
The causal interpretation of the transition rate requires that we take the temporal order in which the processes evolve very seriously. In other words, at any given point in time t, the transition rate r(t) can be made dependent on conditions that happened to occur in the past (i.e., before t), but not on what is the case at t or in the future after t. There are several possibilities to specify the functional relationship g(.) (see Blossfeld and Rohwer ; Blossfeld et al. ). For example, the exponential model, which normally serves a baseline model, assumes that the transition rate can vary with different constellations of covariates x, but that the rates are time constant. The piecewise-constant exponential model allows the transition rate to vary across fixed time periods with period-specific effects of covariates. There are also different parametric models that are based on specific shapes of the time dependence (e.g., the Gompertz-Makeham, Weibull, Sickle, Log-logistic, Lognormal, or Gamma model). If the time shape of the baseline hazard rate is unspecified, and only possible effects of covariates x are estimated, the model is called a semiparametric or partial likelihood model. Finally, there are also models of unobserved heterogeneity in which the transition rate can be made dependent on the observed covariates x, the duration t, and a stochastic error term.
Parallel and Interdependent Processes The most important scientific progress permitted by event history analysis is based on the opportunity to include explicitly measured time-varying covariates in transition rate models (see Blossfeld and Rohwer ; Blossfeld et al. ). These covariates can change their values over process time in an analysis. Time-varying covariates can be qualitative or quantitative, and may stay constant for finite periods of time or change continuously. From a substantive point of view, time-varying covariates can be conceptualized as observations of the sample path of parallel processes. These processes can operate at different levels. In sociology, for example, () there can be parallel processes at the level of the individual’s different domains of life (e.g., one may ask how upward and downward moves in an individual’s job career influence his/her family trajectory), () there may be parallel processes at the level of some few individuals interacting with each other (e.g., one might study the effect of the career of the husband on his wife’s labor force participation), () there may be parallel processes at the intermediate level (e.g., one can analyze how organizational growth influences career advancement or changing household structure determines women’s labor force participation), () there may be parallel processes at the macro level (e.g., one may be interested in the effect of changes in the business cycle on family formation or career
Event History Analysis
advancement), and () there may be any combination of such processes of type () to (). For example, in the study of life course, cohort, and period effects, time-dependent covariates at different levels (see below) must be included simultaneously (7multilevel analysis). In dealing with such systems of parallel processes, the issue of reverse causation is often addressed in the methodological literature (see, e.g., Tuma and Hannan ; Blossfeld et al. ; Yamaguchi ). Reverse causation refers to the (direct or indirect) effect of the dependent process on the independent covariate process(es). Reverse causation is often seen as a problem because the effect of a time-dependent covariate on the transition rate is confounded with a feedback effect of the dependent process on the values of the time-dependent covariate. However, Blossfeld and Rohwer () have developed a causal approach to the analysis of interdependent processes that also works in the case of interdependence. For example, if two interdependent processes, Yt A and Yt B , are given, a change in Yt A at any (specific) point in time t′ may be modeled as being dependent on the history of both processes up to, but not including t ′ . Or stated in another way: What happens with Yt A at any point in time t ′ is conditionally independent of what happens with Yt B at t′ , conditional on the history of the joint process Yt = (Yt A , Yt B ) up to, but not including, t (“principle of conditional independence”). Of course, the same reasoning can be applied if one focuses on YtB instead of YtA as the “dependent variable.” Beginning with a transition rate model for the joint process, Yt = (YtA , YtB ), and assuming the principle of conditional independence, the likelihood for this model can then be factorized into a product of the likelihood’s for two separate models: a transition rate model for Yt A , which is dependent on Yt B as a time-dependent covariate, and a transition rate model for Yt B , which is dependent on Yt A as a time-dependent covariate. From a technical point of view, there is therefore no need to distinguish between defined, ancillary, and internal covariates because all of these time-varying covariate types can be treated in the estimation procedure. Estimating the effects of time-varying processes on the transition rate can easily be achieved by applying the method of episode splitting (see Blossfeld and Rohwer ; Blossfeld et al. ).
Unobserved Heterogeneity Unfortunately, researchers are not always able to include all the important covariates into an event history analysis. These unobserved differences between subpopulations can lead to apparent time dependence at the population level and additional identification problems. There exists a fairly large literature on misspecified models in general.
E
In particular, there have been several proposals to deal with unobserved heterogeneity in transition rate models (so-called frailty models) (see, e.g., Blossfeld and Rohwer ; Blossfeld et al. ).
About the Author Dr. Hans-Peter Blossfeld is Professor of Sociology at Bamberg University. He is also Director of the Institute of Longitudinal Studies in Education (Institut für bildungswissenschaftliche Längsschnittstudien – INBIL), Principal Investigator of the National Educational Panel Study (NEPS), and Director of the State Institute for Family Research at Bamberg University (Staatsinstitut für Familienforschung - ifb). He is an Elected member of the German Academy of Sciences (Leopoldina), the Bavarian Academy of Sciences and the Berlin-Brandenburg Academy of Sciences. He is also a Elected member of the European Academy of Sociology, London, and Elected Chairman of the European Consortium of Sociological Research (ECSR). He serves as member of the Steering Committees of the European Science Foundation (ESF) Programes “Quantitative Methods in the Social Sciences II” and “TransEurope”. He is also member of the ESRC National Strategy Committee on Longitudinal Studies in the UK. He has authored and co-authored more than papers, mostly in refereed journals, and has written as well as edited books, including Techniques of Event History Modeling: New Approaches to Causal Analysis (with G. Rohwer, Lawrence Erlbaum Associates, ). Professor Blossfeld has received the Descartes award for his Globalife project (European Commission, ). He is Editor of the following journals: European Sociological Review, Zeitschrift für Erziehungswissenschaft, and Zeitschrift für Familienforschung. Currently, he is also an Associate editor of International Sociology.
Cross References 7Censoring Methodology 7Demography 7Frailty Model 7Survival Data
References and Further Reading Blossfeld H-P, Hamerle A, Mayer KU () Event history analysis. Lawrence Erlbaum Associates, Hillsdale, NJ Blossfeld H-P, Rohwer G () Techniques of event history modeling. New approaches to causal analysis. Lawrence Erlbaum Associates, Mahwah, NJ/London Blossfeld H-P, Golsch K, Rohwer G () Event history analysis with stata. Lawrence Erlbaum Associates, Mahwah, NJ/London Courgeau D, Lelièvre E () Event history analysis in demography. Clarendon, Oxford
E
E
Exact Goodness-of-Fit Tests Based on Sufficiency
Lancaster T () The econometric analysis of transition data. Cambridge University Press, Cambridge Tuma NB, Hannan MT () Social dynamics. Models and methods. Academic, New York Vermunt JK () Log-linear models for event histories. Sage, Newbury Park, CA Yamaguchi K () Event history analysis. Sage, Newbury Park, CA
These may be described in the (, ) transformed range as: Dn = supu ∣η n (u)∣ Wn = ∫ A n = ∫
Exact Goodness-of-Fit Tests Based on Sufficiency Federico J. O’Reilly, Leticia Gracia-Medrano Professors IIMAS, UNAM, México City, México
(η n (u)) du
(η n (u))
du, u( − u)
having well known computable formulas in terms of the u(i) ’s, the now ordered transformed ui ’s, from smallest to largest, with u() = and u(n+) = , if needed. Dn = max{D+ n , D− n }, D+ n = maxi {Gn (u(i) ) − u(i) } ,
D− n = maxi {u(i) − Gn (u(i−) )}, Gn (u(i) ) = i/n, n
W n = ∑[u(i) − (i − )/(n)] + /(n), i=
Introduction and Simple Case In the univariate continuous goodness-of-fit problem, the probability integral transformation (PIT) is used to transform the sample observations into the unit interval when the distribution to be tested is completely specified (the simple case); thus reducing the problem to that of testing uniformity of the transforms. The asymptotic theory for the induced empirical process in (,) is well known and so, that of functionals of it, including the so called EDF tests that are based on the Empirical Distribution Function (see 7Tests of fit based on the empirical distribution function). The empirical process (see 7Empirical Processes), labeled with the original x is the process √ ψ n (x) = n{Fn (x) − F (x)}, for x ∈ R, with Fn being the empirical distribution function constructed from the observed sample x , ⋯, xn and with F the known distribution function to be tested. This process is in a : correspondence with the corresponding process labeled with u ∈ (, ), √ η n (u) = n{Gn (u) − u}, with u = F (x), where Gn stands for the empirical distribution function of the transformed sample u , ⋯, un with the ui ’s given by ui = F (xi ). Functionals of the empirical process, that appeared in literature (mostly) for the continuous goodness-offit problem are Kolmogorov’s statistic, Dn , Cramér-von Mises, W n , Anderson-Darling’s, A n and others like Kuiper’s Vn , Watson’s U n , and Pearson’s chi square, χ .
An = −n −
n ∑(i − )[log(u(i) ) + log( − u(n+−i) )], n i=
with similar expressions for the other EDF tests (like Kuiper’s Vn = D+ n + D− n ). If the continuous distribution to be tested, has unknown parameters, the established procedure (using asymptotics) is to estimate first and substitute them in place of the parameters before employing the PIT, to transform to the unit interval. Under quite general conditions, asymptotics for the resulting empirical process in (, ) have been reported in literature for many distributions (see Durbin ; D’Agostino and Stephens and references given there). If the distribution is discrete, the above approach using asymptotics, has not been followed much. To start, in the simple case, the PIT does not yield uniform transforms. There are however several proposed tests for the discrete case; some based on the empirical characteristic function, others on the empirical probability generating function and others. See for example Kocherlakota and Kocherlakota (), Rueda et al. (), Nakamura and Pérez-Abreu (), Rueda and O’Reilly (), Gurtler and Henze () and references therein. In Spinelli and Stephens () discrete versions of EDF tests are studied. All these procedures, yield tests related to the “observed and expected counts.” Further mention of the discrete case is made only for “on site” evaluation of 7p-values. With fast computers available now, the simple goodnessof-fit problem, either continuous or discrete, may be solved by computing the p-value of the selected statistic to test the fit, simply by simulating many independent samples from
Exact Goodness-of-Fit Tests Based on Sufficiency
the null hypothetical distribution. This is done by observing first the value of the test statistic with the initial sample and then looking the corresponding simulated values of the test statistic, then finding the proportion of simulated values exceeding the initial value of the test statistic. That is the p-value. This procedure is easier than having to use tables, and possible finite-n corrections. Moreover, this simulation procedure applies to the continuous as well as the discrete case, because the inverse of the PIT (IPIT) is a proper generator for any distribution. The IPIT maps a uniformly distributed random variable U with the “inverse” of a distribution function F − yielding a new random variable X = F − (U), whose distribution function is precisely, F. With the inverse F − (u), say, taken as the supremum of the x-values such that F(x) ≤ u. So in any simple goodness-of-fit problem (continuous or discrete), given x = (x , x , ⋯, xn ), a sample of independent identically distributed random variables with common distribution function F, to test the null hypothesis, H : F = F , with F totally known, one proceeds as follows: For Sn (x) = Sn (x , ⋯, xn ) the test statistic to be used for testing H , evaluated at the observed sample, simulate from the null distribution F , say , samples (xj , j = , ⋯, ,) of the same size n and with each, compute Sn (xj ). Finally look at the proportion of simulated values (the Sn (xj )’s) exceeding Sn (x). This is the p-value obtained with the , simulations.
Composite Case: Group Model Many “composite” goodness-of-fit problems may be tackled with a strikingly similar Monte-Carlo simulation. Those problems arise when the null hypotheses corresponds to a parametric family with a group structure, where the test statistic Sn is invariant; typically, most families with location and/or scale parameters only. It was noticed that when finding the limiting distribution of the “empirical process with estimated parameters,” the limiting process was a zero mean Gaussian process, whose covariance function did not depend on the parameter values (generally depends on the particular family to be tested). This result allowed the use of the limiting distribution for the EDF test statistic; perhaps using some finite-n correction (see e.g., Chap. in D’Agostino and Stephens ). This is the case of most location/scale families.
E
Under a group model, not only the asymptotic, but the finite-n distributions of invariant tests Sn are independent of the value of the parameters, so one may simply assign any arbitrary value to them, and simulate tens of thousands of samples and repeat as in the simple case to evaluate an exact p-value in situ.
Composite Case: Non-Group Model There are important parametric families where there is no group structure, and limiting distributions depend also on the parameter values. These include those where a “shape” parameter is present, as in the case of the 7gamma distribution, the inverse-Gaussian and is the case of most discrete distributions with unknown parameters. For these, an attempt has been made to compute not the p-value, since in general that quantity depends on the parameter, but rather use the conditional distribution of the test statistic Sn given some suitable sufficient statistic to compute a conditional p-value. This is explained next. Assume that in the setting H : F(⋅) = F (⋅; θ), where θ ∈ Θ, Tn is the minimal sufficient statistic. Not being a group model the test statistic Sn will have a distribution, under H , that will in general depend on θ. But having Tn sufficient means that the conditional distribution of Sn given Tn , will not depend on θ, so in order to base the procedure in a simulation, it will be enough to have a simulator of samples which are conditionally to Tn , independent identically distributed. One way to simulate in that fashion appears in O’Reilly and Gracia-Medrano (), and is illustrated for the 7inverse Gaussian distribution. In González-Barrios et al. (), the results provide means to simulate for the Poisson, binomial and negative binomial, with unknown parameters (besides the test derived there that uses enumeration for the conditional distribution). In Lockhart et al. () a simulator is provided for the two parameter gamma distribution. The first reference bases the simulator on the explicit use of the Rao-Blackwell estimator of the distribution F (⋅; θ). The third reference uses the Gibbs sampler to achieve an exact simulation method. The inverse-Gaussian simulator is sketched to illustrate. Let the null hypothesis correspond to the inverseGaussian distribution, F (x; µ, λ). Denote the Rao-Blackwell distribution estimate, based on the sample x, by F˜ n (x), which is a function of the minimal sufficient statistic, Tn (x) = (∑ xi , ∑ ( )) , x i i i and exists for n > . Its expression appears (corrected) in Chhikara and Folks ().
E
E
Exact Inference for Categorical Data
Observe that the statistic Tn has the property of double transitivity, which means that the pair (Tn , Xn ) is in a : correspondence with (Tn− , Xn ). It implies in particular that knowing the value of the statistic with a sample of size n and knowledge of, say, the last observation, one can find the value of the statistic with the corresponding smaller sample of size (n − ). After observing the sample, denote by s the value of the goodness of fit test and by tn the value of the minimal sufficient statistic; that is, Sn (x) = s and Tn (x) = tn . In order to simulate, conditionally to Tn = tn , a conditional sample (denoted a ∗-sample) proceed as follows: Obtain randomly un from the U(, ) distribution and ⋆ set x⋆ n = F˜ n− (un ), then compute tn− from the knowledge of tn and taking away form its evaluation the (imposed) ⋆ n-th value x⋆ n , i.e., obtain tn− from the pair (tn , x⋆ n ). Next with an independently selected un− from the U(, ), use ⋆ the Rao-Blackwell estimate given tn− and define x⋆ n− = − ⋆ ⋆ F˜ n− (un− ). Continue finding tn− from (tn− , x⋆ n− ), then ⋆ looking at the Rao-Blackwell estimate given tn− , use its ⋆ inverse to get x n− with an independently selected un− ; and so on until finding x⋆ . Finally, get x⋆ and x⋆ as the solution to yield the same value for Tn when evaluated at the new ∗-sample (that is tn ). The procedure is repeated for tens of thousands of times getting tens of thousands of ∗-samples which in turn produces the desired simulated values of the statistic Sn . The proportion of simulated values that exceed the original s is the conditional p-value, and is exact. With , ∗-samples and n = , evaluation of the p-value takes around s on a conventional PC.
Parametric Bootstrap Simulations based on parametric bootstrap to get a p-value, provide an exact procedure in the case of a group model. In the non-group model, bootstrapping is not theoretically exact but very good. Ongoing research is being done on the closeness of the conditional p-value and the one found with bootstrap (see 7Bootstrap Methods). Articles where bootstrap has been proposed to evaluate the p-value for particular distributions, show that for large n, the procedure provides a very good approximation.
About the Author Professor O’Reilly is past President of the Mexican Statistical Association (–), and former Director of the “Instituto de Investigaciones en Matematicas Aplicadas y en Sistemas” (IIMAS, –). He is member of the Mexican Academy of Science and also a member of the ISI.
Cross References 7Anderson-Darling Tests of Goodness-of-Fit 7Bootstrap Methods 7Chi-Square Goodness-of-Fit Tests: Drawbacks and
Improvements 7Chi-Square Test: Analysis of Contingency Tables 7Cramér-Von Mises Statistics for Discrete Distributions 7Jarque-Bera Test 7Kolmogorov-Smirnov Test 7Tests of Fit Based on The Empirical Distribution Function
References and Further Reading Chhikara RS, Folks JL () The inverse Gaussian distribution as a lifetime model. Technometrics :– D’Agostino RB, Stephens MA (eds) () Goodness of fit techniques. Marcel Decker, New York Durbin J () Distribution theory for tests based on the sample distribution function. In: Regional Conference Series in Applied Mathematics, vol . SIAM, Philadelphia González-Barrios JM, O’Reilly F, Rueda R () Goodness of fit for discrete random variables using the conditional density. Metrika :– Gurtler N, Henze N () Recent and classical goodness of fit tests for the Poisson distribution. J Stat Plan Infer :– Kocherlakota S, Kocherlakota K () Goodness of fit tests for discrete distributions. Commun Stat Theor Meth :– Lockhart RA, O’Reilly F, Stephens MA () Use of the Gibbs sampler to obtain conditional tests, with applications. Biometrika :– Nakamura M, Pérez-Abreu V () Empirical probability generating function. An overview. Insur Math Econ :– O’Reilly F, Gracia-Medrano L () On the conditional distribution of goodness-of-fit-tests. Commun Stat Theor Meth :– Rueda R, Pérez-Abreu V, O’Reilly F () Goodness of fit for the Poisson distribution based on the probability generating function. Commun Stat Theor Meth ():– Rueda R, O’Reilly F () Tests of fit for discrete distributions based on the probability generating function. Commun Stat Sim Comp ():– Spinelli J, Stephens MA () Cramér-von Mises tests of fit for the Poisson distribution. Can J Stat ():–
Exact Inference for Categorical Data Roshini Sooriyarachchi Professor University of Colombo, Colombo, Sri Lanka In practice statistical inference for categorical data is based mainly on large sample approximations of test statistics.
E
Exact Inference for Categorical Data
This asymptotic theory is valid only if the sample sizes are reasonably large and well balanced across populations. To make valid inferences in the presence of small, sparse or unbalanced data, exact p-values and confidence intervals, based on the permutational distribution of the test statistic (Good ) needs to be computed. The most common approach to exact inference for categorical data has been a conditional one (Agresti, ) thus this approach is mainly discussed here. This utilizes the distribution of the sufficient statistic for the parameter of interest, conditional on the sufficient statistics for the other model parameters (Agresti ). The sufficiency principle used here is explained in more detail in Mehta ().
Settings and Notation The rest of the article presents a variety of exact methods for categorical data. The required settings and notation are defined in this section. Consider a two-way contingency table having I rows and J columns cross–classifying a row variable X and a column variable Y. Let {nij } correspond to the cell counts ni+ = ∑nij , n+j = ∑nij , and j
by θ =
P(n = k ∣ n, n+ , n+ ; θ) =
. Full multinomial sampling scheme where the cell counts {nij } have a 7multinomial distribution generated by n independent trials with IJ cell probabilities {π ij }. . Product multinomial sampling where counts {nij } in row I have a multinomial distribution for all j, with counts in different rows being independent. Here ni+ = ∑nij , i = , . . . , I are fixed.
( n+ )( n − n+ )θ k k n+ − k . () n n − n+ )θ u ∑( + )( u n+ − u u
Under the null hypothesis of independence (θ = ) the conditional distribution given in () is hypergeometric. To test H : θ = versus H : θ > the p-value is given by P = ∑P(t ∣ n, n+ , n+ ; θ = ) where S = {t ; t ≥ n }. S
This test is called the 7Fisher’s Exact test (Fisher, ).
I × J Tables For I × J tables, statistical independence of X and Y (H ) corresponds to the log-linear model (Bishop et al., ) log(mij ) = µ + λ i X + λ j Y .
n = ∑nij . Agresti () and Mehta () list three i,j
π /( − π ) . π /( − π )
Several authors (Mehta, ; Agresti, , ) showed that
i
sampling schemes (settings) that can give rise to crossclassified categorical data. These schemes are namely:
i,j
is random. As Agresti () indicates that all three sampling models lead to the same inference, for the sake of simplicity the following results are based on sampling scheme (). Under this scheme, the conditional probability of belonging to the jth column given that observation is in the ith row is denoted by π j∣i = P(Y = j∣X = i) for i = , . . . , I and j = , . . . , J.
()
To test H , model () has to be compared with the saturated model log(mij ) = µ + λ i X + λ j Y + λ ij XY .
()
Cornfield () showed that the distribution of {nij } given {ni+ } and {n+j } depends only on the odds ratios (α ij ) such that I− J−
∏∏α ij
P(nij ∣ {ni+ }, {n+j }; α ij ) =
n ij
i= j=
......
I− J−
()
∏∏nij !
j
. Poisson sampling, where {nij } have a Poisson distribution with expected value E(nij ) = mij where n = ∑nij
i= j=
where α ij =
π ij /π IJ . π iJ /π Ij
Under H , α ij = for all i and j thus the conditional distribution of nij given {ni+ } and {n+j } under H is multiple hypergeometric and is given by (∏ni+ !) (∏n+j !) P(nij ∣ {ni+ }, {n+j }) =
i
j
.
n!∏∏nij !
× Table Let π and π denote “success” probabilities associated with row and row respectively. Then the odds ratio is given
i= j=
The methods for I × J tables can be extended to multiway tables (Agresti, ).
E
E
Exchangeability
Exact Inference for Logistic Models Consider the case of a single binary response variable (Y) and several explanatory variables X , . . . , Xk . In this situation a logistic model (Hosmer and Lemeshow ()) is preferred to a log-linear model. This model is expressed as log (
k πi ) = ∑ β j xij where π i is the probability of − πi j=
belonging to the research category of interest for the ith subject, xi , . . . , xik are the values of the k explanatory variables observed for the ith subject and xi = for all i. When {yi } are independent Bernoulli outcomes, Cox () showed that the sufficient statistics for β j are Tj = ∑yi xij ; j = , . . . , k and illustrated how to conduct infer-
Cornfield J () A statistical problem arising from retrospective studies. Proceedings of the rd Berkeley Symposium on Mathematical Statistics and Probability, vol . University of California, Berkeley, – Cox DR () Analysis of binary data. Chapman and Hall, Boca Raton, FL Fisher RA () The logic of inductive inference. J R Stat Soc Ser A Stat Soc :– Good P () Permutation tests. Springer Verlag, New York Hosmer DW, Lemeshow S () Applied logistic regression, nd edn. Wiley, New York Mehta CR () Exact inference for categorical data. Encycl Biostat :–
i
ence for β j using the conditional distribution of Tj given {Ti , i ≠ j}.
Other Topics and Available Software Agresti () discusses less common topics such as exact confidence intervals, exact goodness of fit tests, controversies regarding the exact conditional approach, exact unconditional approach and Bayesian approach. Software support for exact inference for categorical data are available in SAS, Stata, R, SPSS, StatXact and LogXact.
About the Author Dr Sooriyarachchi is Professor of Statistics at the University of Colombo Sri Lanka. She has written several papers both in international and local journals on methodological developments for categorical data. She has received several awards (Presidential award and University of Colombo Award). She has been a visiting Research Fellow at the University of Reading UK for three years.
Cross References 7Categorical Data Analysis 7Fisher Exact Test 7Statistical Inference 7Variation for Categorical Variables
References and Further Reading Agresti A () A survey of exact inference for contingency tables. Stat Sci ():– Agresti A () Exact inference for categorical data: recent advances and continuing controversies. Stat Med :– Bishop YMM, Feinberg SE, Holland PW () Discrete multivariate analysis: theory and practice. MIT press, Cambridge, MA
Exchangeability Sander Greenland , David Draper Professor University of California-Los Angeles, Los Angeles, CA, USA Professor University of California-Santa Cruz, Santa Cruz, CA, USA
In probability theory, the random variables Y , . . . , YN are said to be exchangeable (or permutable or symmetric) if their joint distribution F(y , . . . , yN ) is a symmetric function; that is, if F is invariant under permutation of its arguments, so that F(z , . . . , zN ) = F(y , . . . , yN ) whenever z , . . . , zN is a permutation of y , . . . , yN . There is a related epidemiologic usage described in the article on 7confounding. Exchangeable random variables are identically distributed, and iid variables are exchangeable. In many ways, sequences of exchangeable random variables play a role in subjective Bayesian theory analogous to that played by independent identically distributed (iid) sequences in frequentist theory. In particular, the assumption that a sequence of random variables is exchangeable allows the development of inductive statistical procedures for inference from observed to unobserved members of the sequence (Bernardo and Smith ; De Finetti , ; Draper ; Draper et al. ; Lindley and Novick ). Now suppose that Y , . . . , YN are iid given an unknown parameter θ that indexes their joint distribution. Such variables will not be unconditionally independent when θ is a random variable, but will be exchangeable. Suppose, for example, Y , . . . , YN have a joint density. The
Exchangeability
unconditional density of Y , . . . , YN will be f (y , . . . , yN ) = ∫ f (y , . . . , yN ∣θ)dF(θ) θ
= ∫ ∏ f (yi ∣θ)dF(θ). i
Exchangeability of Y , . . . , YN follows from the identity of the marginal densities in the product. However, given that these densities depend on θ, the integral and product cannot be interchanged, so that f (y , . . . , yN ) ≠ ∏f (yi ). We thus have that a mixture of iid i
sequences is an exchangeable sequence, but not iid except in trivial cases. One consequence of this result is that the usual procedures for generating a sequence Y , . . . , YN of iid random variables for inference on an unknown parameter (such as Bernoulli trials of binary data with unknown success probability) generate an exchangeable sequence when the parameter is generated randomly and the sequence is considered unconditionally. From a Bayesian perspective, this means that, when your uncertainty about the parameter is integrated with your uncertainty about the realizations of Y , . . . , YN , the latter are (for you) exchangeable but dependent even if the realizations are physically independent. This subjective dependence is immediately clear if you consider (say) a sequence of iid (but possibly biased) coin tosses, with Yi the indicator of heads on toss i. Then, starting from a uniform prior distribution for the chance of heads θ, you should have Pr(Y = ) =/ before seeing any toss, but Pr(Y = ∣Yi = for i = , . . . , ) = . after seeing the first tosses come up heads (Good ). A generalization important for statistical modeling is partial or conditional exchangeability (De Finetti, , ). For example, suppose that the sequence Y , . . . , YN is partitioned into disjoint subsequences. Then the sequence is said to be partially exchangeable given the partition if each subsequence can be permuted without changing the joint distribution. For example, if the Yi represent survival times within a cohort of male stroke patients, then a judgment of unconditional exchangeability of the Yi would be unreasonable if the patient ages were known, because age is a known predictor of survival time. Nonetheless, one might regard the survival times as partially exchangeable, given age, if no further prognostically relevant partitioning was possible based on the available data. While exchangeability is weaker than iid, finite subsequences of an infinite exchangeable sequence of Bernoulli
E
(binary) variates have representations as mixtures of iid Bernoulli sequences – a partial converse of the fact that any mixture of iid sequences is an exchangeable sequence. More precisely, suppose that Y , Y , . . . is an infinite sequence of exchangeable Bernoulli variates (that is, every finite subsequence of the sequence is exchangeable), and that θ is the limit of (Y + ⋯ + Yn )/n as n goes to infinity. De Finetti [, Chapter ] showed that there exists a distribution function P(θ) for θ such that, for all n, Pr(Y = y , . . . , Yn = yn ) ≡ Pr(y , . . . , yn ) = ∫ θ s ( − θ)n−s dP(θ), θ
where s = y + ⋯ + yn . Many Bayesian statisticians find this theorem helpful, because it partially specifies the form of the predictive probability Pr(y , . . . , yn ) when Y , . . . , Yn can be modeled as a subsequence of an infinite exchangeable sequence. In this representation, P(θ) is recognizable as a prior distribution for θ, a distribution that may be developed from what is known about θ before the Yi are observed. As noted in (Freedman ), however, the strength of the theorem’s conclusion is easy to overstate: it does not imply that all binary data must be analyzed using the representation; it merely says that if you judge Y , Y , . . . to be an exchangeable sequence, then there is a prior P(θ) that allows you to use to specify Pr(y , . . . , yn ) as a mixture of iid variables over that prior. Finite versions of De Finetti’s theorem exist (Diaconis and Freedman ). If Y , . . . , Yn is the start of an exchangeable Bernoulli sequence Y , . . . , YN and n/N is small enough, then Pr(y , . . . , yn ) may be approximately represented by the mixture in the theorem, with the approximation improving as n/N approaches zero. There are further generalizations to exchangeable sequences of polytomous variates, as well as to exchangeable sequences of continuous variates (Diaconis and Freedman ). The latter generalization requires a prior distribution on the space of continuous distributions, however, which can be much harder to specify than a prior for a vector of multinomial parameters, and which may lead to intractable computational problems (Draper ).
About the Authors For Dr. Greenland’s biography see the entry 7Confounding and Confounder Control. David Draper is a Professor of Statistics in the Department of Applied Mathematics and Statistics (in the Baskin School of Engineering) at the University of California, Santa Cruz (UCSC). He was President of the International
E
E
Expected Value
Society for Bayesian Analysis in . From to he served as the Founding Chair of the Applied Mathematics and Statistics Department at UCSC. He is a Fellow of the American Association for the Advancement of Science, the American Statistical Association, the Institute of Mathematical Statistics and the Royal Statistical Society. He is the author or co-author of contributions to the research literature. Since he has given invited, special invited or plenary talks at major research conferences and leading statistics departments in countries worldwide. He has been an Associate Editor for leading journals (including Journal of the American Statistical Association and the Journal of the Royal Statistical Society), and has organized or co-organized international research conferences.
Cross References 7Bayesian Statistics 7Confounding and Confounder Control
References and Further Reading Bernardo JM, Smith AFM () Bayesian theory. Wiley, New York De Finetti B () Foresight: its logical laws, its subjective sources. In: Kyburg HE, Smokler HE (eds) Studies in subjective probability (reprinted in ). Wiley, New York De Finetti B () Theory of probability (two vols). Wiley, New York Diaconis P, Freedman DA () Finite exchangeable sequences. Ann Probab :– Draper D () Assessment and propagation of model uncertainty (with discussion). J R Stat Soc Ser B :– Draper D, Hodges J, Mallows C, Pregibon D (). Exchangeability and data analysis (with discussion). J R Stat Soc Ser A : –